Beyond Meme Templates: Limitations of Visual Similarity Measures in Meme Matching

The analysis of Internet memes allows researchers to gain insights into contemporary digital culture. These engaging user-generated content are characterised by shared visual elements also found in other memes. Matching instances of memes via these elements, meme matching, is the basis of a wealth of meme analysis approaches. However, most existing methods assume that every meme consists of a shared visual background, called a template, with some overlaid text, excluding the many memes that are not template-based, and limiting the effectiveness of automated meme analysis. For example, this assumption leads to approaches that would not be effective in linking memes to contemporary web-based meme dictionaries. In this work, we introduce a broader formulation of meme matching that extends beyond template matching. We show that conventional similarity measures, including a novel segment-wise computation of the similarity measures, excel at matching templatebased memes but fall short when applied to non-template-based meme formats. Notably, the segment-wise measures were found to consistently outperform the whole-image counterparts on matching non-template-based memes. Finally, we explore prompting of a pretrained Multimodal Large Language Model as an approach to meme matching. Collectively, our results show that both similarity- and prompting-based approaches struggle to accurately match memes via shared visual elements, not just background templates. We show that meme matching remains an open challenge requiring more sophisticated techniques than those currently employed for template-based matching.