Benchmarking Machine Translation on Chinese Social Media Texts
Abstract
AbstractThe prevalence of rapidly evolving slang, neologisms, and highly stylized expressions in informal user-generated text, particularly on Chinese social media, poses significant challenges for Machine Translation (MT) benchmarking. Specifically, we identify two primary obstacles: (1) data scarcity, as high-quality parallel data requires bilingual annotators familiar with platform-specific slang, and stylistic cues; and (2) metric limitations, where traditional evaluators like COMET often fail to capture stylistic fidelity and non-standard expressions. To bridge these gaps, we first introduce CSM-MTBench, an expert-curated benchmark covering five Chinese→foreign language directions and comprising 10,915 total instances, organized into two subsets: **Fun Posts**, featuring context-rich content with frequent slang and neologisms, and **Social Snippets**, focusing on concise, emotion- and style-driven expressions. Next, we propose tailored evaluation approaches for each subset, including slang translation success rate and style-preservation metrics combining embeddings and LLM-asa-judge evaluation. Experiments on over 20 models reveal substantial variation in how current MT systems handle semantic fidelity, slang translation, and stylistic nuances. We further look into model behaviors through prompt sensitivity analysis and qualitative case studies, highlighting the challenges of translating these informal expressions. CSM-MTBench thus serves as a rigorous testbed for advancing MT systems capable of mastering real-world Chinese social media texts.