On The Automatic Image Captioning Task In Italian: A Human-Centric Approach

Automatic Image Captioning (AIC) combines two distinct machine learning disciplines, Computer Vision (CV) and Natural Language Processing (NLP), forming a challenging research landscape. Much research has been carried out for AIC in the English language facilitated by the availability of English language datasets. However, this is not true for other languages. The aim of this work is to investigate existing AIC models' compatibility for the Italian language. The scope of this research will be aimed to reduced complexity, improved efficiency, and readily trainable models on smaller datasets while maintaining an acceptable level of caption quality. The popular Show and Tell encoder-decoder model was selected and trained on subsets of the MSCOCO-it dataset of 10k, 20k, and 30k with and without human validated captions, and then evaluated with BLEU-3/4, METEOR, ROUGE-L, and CIDEr. The model was also tested on unseen images. The evaluation results were not promising and were below 50%. We intend to investigate multimodal augmentation techniques for image-caption datasets. To enhance interpretability, explainable AI (XAI) methods such as attention visualization and saliency mapping are to be employed to reveal which image regions and linguistic features influenced caption generation. Furthermore, a gender-aware evaluation is planned to be researched and introduced, assessing whether generated captions reproduce gender bias present in training data. This dual focus on explainability and inclusivity strengthens trust in AIC systems while supporting fairer and more transparent applications.