Talk by Minh Duc Bui

25 February 2025, by DS Group

This week, Duc will give a talk at our lab. We are very lucky to have him present his current work.

Title: Multimodal, Multilingual, and Multicultural Hate Speech Detection with Vision–Language Models

Abstract: The talk will primarily focus on our recent work on Multicultural Hate Speech Detection using Vision–Language Models. If time permits, we will also discuss our ongoing research on "Generalization Across Measurement Systems: LLMs Require More Test-Time Compute for Underrepresented Cultures."

(1) Multicultural Hate Speech Detection: Hate speech moderation on global platforms poses unique challenges due to the multimodal and multilingual nature of content, along with the varying cultural perceptions. How well do current vision-language models (VLMs) navigate these nuances? To investigate this, we create the first multimodal and multilingual parallel hate speech dataset, annotated by a multicultural set of annotators, called Multi^3Hate. It contains 300 parallel meme samples across 5 languages: English, German, Spanish, Hindi, and Mandarin. We demonstrate that cultural background significantly affects multimodal hate speech annotation in our dataset. [...] We then conduct experiments with 5 large VLMs in a zero-shot setting, finding that these models align more closely with annotations from the US than with those from other cultures, even when the memes and prompts are presented in the dominant language of the other culture.

(2) On Generalization across Measurement Systems: Measurement systems (e.g., currencies) differ across cultures, but the conversions between them are well defined so that humans can state facts using any measurement system of their choice. Being available to users from diverse cultural backgrounds, large language models (LLMs) should also be able to provide accurate information irrespective of the measurement system at hand. [...] Our findings show that LLMs default to the measurement system predominantly used in the data. Additionally, we observe considerable instability and variance in performance across different measurement systems. While this instability can in part be mitigated by employing reasoning methods such as chain-of-thought (CoT), this implies longer responses and thereby significantly increases test-time compute (and inference costs), marginalizing users from cultural backgrounds that use underrepresented measurement systems.

Who: Minh Duc Bui is a PhD student at JGU Mainz, supervised by Katharina von der Wense. His research explores cross-cultural NLP, focusing on analyzing and developing language models that account for cultural differences and biases. He holds a bachelor's degree in Mathematics in Business and Economics and a master's in Data Science from the University of Mannheim, where he worked on multilingual NLP under the supervision of Goran Glavaš. Before his PhD, he was a data scientist in the autonomous driving industry in Stuttgart.