Image-Text Embedding with Hierarchical Knowledge for Cross-Modal Retrieval

Abstract

Heterogeneous data embedding is a process of mapping different kinds of data into a common vector space of a certain dimension. Image-text embedding also means mapping image and text data that have completely different characteristics into a common vector space. In this paper, we propose an image-text embedding method using hierarchical knowledge such as coarse and fine labels of text data. The proposed method improves the training efficiency of the embedding model by fixing the coarse label vectors. In addition, the loss function is designed by arbitrarily selecting the negative sample from the fine labels having a hierarchical relationship with the coarse label, so that the difference between the vectors of the fine labels which have same coarse label becomes larger. So, when the images that are visual data is mapped into a common vector space, the semantic of images becomes clear. Experimental results show that embedding with hierarchical knowledge has been successfully performed using the proposed methodology and that cross-modal retrieval can be efficiently performed through embedding model.