Generating diverse and meaningful captions: Unsupervised specificity optimization for image captioning

Annika Lindh; Robert J. Ross; Abhijit Mahalunkar; Giancarlo Salton; John D. Kelleher

doi:10.1007/978-3-030-01418-6_18

Generating diverse and meaningful captions: Unsupervised specificity optimization for image captioning

Annika Lindh, Robert J. Ross, Abhijit Mahalunkar, Giancarlo Salton, John D. Kelleher

School of Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Image Captioning is a task that requires models to acquire a multimodal understanding of the world and to express this understanding in natural language text. While the state-of-the-art for this task has rapidly improved in terms of n-gram metrics, these models tend to output the same generic captions for similar images. In this work, we address this limitation and train a model that generates more diverse and specific captions through an unsupervised training approach that incorporates a learning signal from an Image Retrieval model. We summarize previous results and improve the state-of-the-art on caption diversity and novelty. We make our source code publicly available online (https://github.com/AnnikaLindh/Diverse_and_Specific_Image_Captioning).

Original language	English
Title of host publication	Artificial Neural Networks and Machine Learning – ICANN 2018 - 27th International Conference on Artificial Neural Networks, 2018, Proceedings
Editors	Vera Kurkova, Barbara Hammer, Yannis Manolopoulos, Lazaros Iliadis, Ilias Maglogiannis
Publisher	Springer Verlag
Pages	176-187
Number of pages	12
ISBN (Print)	9783030014179
DOIs	https://doi.org/10.1007/978-3-030-01418-6_18
Publication status	Published - 2018
Event	27th International Conference on Artificial Neural Networks, ICANN 2018 - Rhodes, Greece Duration: 4 Oct 2018 → 7 Oct 2018

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	11139 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	27th International Conference on Artificial Neural Networks, ICANN 2018
Country/Territory	Greece
City	Rhodes
Period	4/10/18 → 7/10/18

Keywords

Computer vision
Contrastive learning
Deep learning
Diversity
Image captioning
Image retrieval
MS COCO
Machine learning
Multimodal training
Natural language generation
Natural language processing
Neural networks
Specificity

Access to Document

10.1007/978-3-030-01418-6_18

https://arrow.tudublin.ie/airccon/13Licence: CC BY-NC-SA

Cite this

Lindh, A., Ross, R. J., Mahalunkar, A., Salton, G., & Kelleher, J. D. (2018). Generating diverse and meaningful captions: Unsupervised specificity optimization for image captioning. In V. Kurkova, B. Hammer, Y. Manolopoulos, L. Iliadis, & I. Maglogiannis (Eds.), Artificial Neural Networks and Machine Learning – ICANN 2018 - 27th International Conference on Artificial Neural Networks, 2018, Proceedings (pp. 176-187). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11139 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-030-01418-6_18

Lindh, Annika ; Ross, Robert J. ; Mahalunkar, Abhijit et al. / Generating diverse and meaningful captions : Unsupervised specificity optimization for image captioning. Artificial Neural Networks and Machine Learning – ICANN 2018 - 27th International Conference on Artificial Neural Networks, 2018, Proceedings. editor / Vera Kurkova ; Barbara Hammer ; Yannis Manolopoulos ; Lazaros Iliadis ; Ilias Maglogiannis. Springer Verlag, 2018. pp. 176-187 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{8f06f2cc794d4bee8793edd1f6776d99,

title = "Generating diverse and meaningful captions: Unsupervised specificity optimization for image captioning",

abstract = "Image Captioning is a task that requires models to acquire a multimodal understanding of the world and to express this understanding in natural language text. While the state-of-the-art for this task has rapidly improved in terms of n-gram metrics, these models tend to output the same generic captions for similar images. In this work, we address this limitation and train a model that generates more diverse and specific captions through an unsupervised training approach that incorporates a learning signal from an Image Retrieval model. We summarize previous results and improve the state-of-the-art on caption diversity and novelty. We make our source code publicly available online (https://github.com/AnnikaLindh/Diverse_and_Specific_Image_Captioning).",

keywords = "Computer vision, Contrastive learning, Deep learning, Diversity, Image captioning, Image retrieval, MS COCO, Machine learning, Multimodal training, Natural language generation, Natural language processing, Neural networks, Specificity",

author = "Annika Lindh and Ross, {Robert J.} and Abhijit Mahalunkar and Giancarlo Salton and Kelleher, {John D.}",

note = "Publisher Copyright: {\textcopyright} Springer Nature Switzerland AG 2018.; 27th International Conference on Artificial Neural Networks, ICANN 2018 ; Conference date: 04-10-2018 Through 07-10-2018",

year = "2018",

doi = "10.1007/978-3-030-01418-6_18",

language = "English",

isbn = "9783030014179",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "176--187",

editor = "Vera Kurkova and Barbara Hammer and Yannis Manolopoulos and Lazaros Iliadis and Ilias Maglogiannis",

booktitle = "Artificial Neural Networks and Machine Learning – ICANN 2018 - 27th International Conference on Artificial Neural Networks, 2018, Proceedings",

address = "Germany",

}

Lindh, A, Ross, RJ, Mahalunkar, A, Salton, G & Kelleher, JD 2018, Generating diverse and meaningful captions: Unsupervised specificity optimization for image captioning. in V Kurkova, B Hammer, Y Manolopoulos, L Iliadis & I Maglogiannis (eds), Artificial Neural Networks and Machine Learning – ICANN 2018 - 27th International Conference on Artificial Neural Networks, 2018, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11139 LNCS, Springer Verlag, pp. 176-187, 27th International Conference on Artificial Neural Networks, ICANN 2018, Rhodes, Greece, 4/10/18. https://doi.org/10.1007/978-3-030-01418-6_18

Generating diverse and meaningful captions: Unsupervised specificity optimization for image captioning. / Lindh, Annika; Ross, Robert J.; Mahalunkar, Abhijit et al.
Artificial Neural Networks and Machine Learning – ICANN 2018 - 27th International Conference on Artificial Neural Networks, 2018, Proceedings. ed. / Vera Kurkova; Barbara Hammer; Yannis Manolopoulos; Lazaros Iliadis; Ilias Maglogiannis. Springer Verlag, 2018. p. 176-187 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11139 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Generating diverse and meaningful captions

T2 - 27th International Conference on Artificial Neural Networks, ICANN 2018

AU - Lindh, Annika

AU - Ross, Robert J.

AU - Mahalunkar, Abhijit

AU - Salton, Giancarlo

AU - Kelleher, John D.

N1 - Publisher Copyright: © Springer Nature Switzerland AG 2018.

PY - 2018

Y1 - 2018

N2 - Image Captioning is a task that requires models to acquire a multimodal understanding of the world and to express this understanding in natural language text. While the state-of-the-art for this task has rapidly improved in terms of n-gram metrics, these models tend to output the same generic captions for similar images. In this work, we address this limitation and train a model that generates more diverse and specific captions through an unsupervised training approach that incorporates a learning signal from an Image Retrieval model. We summarize previous results and improve the state-of-the-art on caption diversity and novelty. We make our source code publicly available online (https://github.com/AnnikaLindh/Diverse_and_Specific_Image_Captioning).

AB - Image Captioning is a task that requires models to acquire a multimodal understanding of the world and to express this understanding in natural language text. While the state-of-the-art for this task has rapidly improved in terms of n-gram metrics, these models tend to output the same generic captions for similar images. In this work, we address this limitation and train a model that generates more diverse and specific captions through an unsupervised training approach that incorporates a learning signal from an Image Retrieval model. We summarize previous results and improve the state-of-the-art on caption diversity and novelty. We make our source code publicly available online (https://github.com/AnnikaLindh/Diverse_and_Specific_Image_Captioning).

KW - Computer vision

KW - Contrastive learning

KW - Deep learning

KW - Diversity

KW - Image captioning

KW - Image retrieval

KW - MS COCO

KW - Machine learning

KW - Multimodal training

KW - Natural language generation

KW - Natural language processing

KW - Neural networks

KW - Specificity

UR - http://www.scopus.com/inward/record.url?scp=85054843258&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-01418-6_18

DO - 10.1007/978-3-030-01418-6_18

M3 - Conference contribution

AN - SCOPUS:85054843258

SN - 9783030014179

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 176

EP - 187

BT - Artificial Neural Networks and Machine Learning – ICANN 2018 - 27th International Conference on Artificial Neural Networks, 2018, Proceedings

A2 - Kurkova, Vera

A2 - Hammer, Barbara

A2 - Manolopoulos, Yannis

A2 - Iliadis, Lazaros

A2 - Maglogiannis, Ilias

PB - Springer Verlag

Y2 - 4 October 2018 through 7 October 2018

ER -

Lindh A, Ross RJ, Mahalunkar A, Salton G, Kelleher JD. Generating diverse and meaningful captions: Unsupervised specificity optimization for image captioning. In Kurkova V, Hammer B, Manolopoulos Y, Iliadis L, Maglogiannis I, editors, Artificial Neural Networks and Machine Learning – ICANN 2018 - 27th International Conference on Artificial Neural Networks, 2018, Proceedings. Springer Verlag. 2018. p. 176-187. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-01418-6_18

Generating diverse and meaningful captions: Unsupervised specificity optimization for image captioning

Abstract

Publication series

Conference

Keywords

Access to Document

Fingerprint

Cite this