Visual-Semantic Embedding Model Informed by Structured Knowledge

Published in The 9th European Starting AI Researchers’ Symposium (STAIRS@ECAI2020), 2020

This study proposes a novel approach to improve a visual-semantic embedding model by incorporating concept representations captured from an external structured knowledge base. We investigate its performance on image classification under both standard and zero-shot settings. We propose two novel evaluation frameworks to analyse classification errors with respect to the class hierarchy indicated by the knowledge base. The approach is tested using the ILSVRC 2012 image dataset and a WordNet knowledge base. With respect to both standard and zero-shot image classification, our approach shows superior performance compared with the original approach, which uses word embeddings. We discuss reasons for this improvement and future work.

Download paper here