Understanding Cultural Alignment in Multilingual LLMs via Natural Debate Statements
Abstract
AbstractIn this work we investigate the sociocultural values learned by large language models (LLMs). We introduce CultureProbe, a novel open-access dataset constructed from natural debate statements using a multi-step methodology. The dataset is synthetically annotated with quantized sociocultural norms according to the Hofstede cultural dimensions. We verify the accuracy of the synthetic annotations using a crowdsourced human annotation experiment conducted on a representative sample. We conduct a comparative analysis between two groups of LLMs developed in different countries (U.S. and China), and use as a comparative baseline patterns observed in human measurements. Using this new dataset and the analysis above, we found that culturally-distinct LLMs largely reflect the values and norms of the countries in which they were developed, highlighting their inability to adapt to the sociocultural backgrounds of their users.