Please use this identifier to cite or link to this item:
Title: Incorporating concreteness in multi-modal language models with curriculum learning
Authors: Sezerer, Erhan
Tekir, Selma
Keywords: Multi-modal dataset
Wikimedia Commons
Multi-modal language model
Curriculum learning
Publisher: MDPI
Abstract: Over the last few years, there has been an increase in the studies that consider experiential (visual) information by building multi-modal language models and representations. It is shown by several studies that language acquisition in humans starts with learning concrete concepts through images and then continues with learning abstract ideas through the text. In this work, the curriculum learning method is used to teach the model concrete/abstract concepts through images and their corresponding captions to accomplish multi-modal language modeling/representation. We use the BERT and Resnet-152 models on each modality and combine them using attentive pooling to perform pre-training on the newly constructed dataset, which is collected from the Wikimedia Commons based on concrete/abstract words. To show the performance of the proposed model, downstream tasks and ablation studies are performed. The contribution of this work is two-fold: A new dataset is constructed from Wikimedia Commons based on concrete/abstract words, and a new multi-modal pre-training approach based on curriculum learning is proposed. The results show that the proposed multi-modal pre-training approach contributes to the success of the model.
ISSN: 2076-3417
Appears in Collections:Computer Engineering / Bilgisayar Mühendisliği
Scopus İndeksli Yayınlar Koleksiyonu / Scopus Indexed Publications Collection
WoS İndeksli Yayınlar Koleksiyonu / WoS Indexed Publications Collection

Files in This Item:
File SizeFormat 
applsci-11-08241.pdf1 MBAdobe PDFView/Open
Show full item record

CORE Recommender


checked on Jul 12, 2024


checked on Jul 13, 2024

Page view(s)

checked on Jul 15, 2024


checked on Jul 15, 2024

Google ScholarTM



Items in GCRIS Repository are protected by copyright, with all rights reserved, unless otherwise indicated.