Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Multi-Model Framework for Sentiment Annotation: A Use Case of Multilingual Wikipedia

AbstractIn this work, we investigate whether data annotation can be automated using state-of-the-art large language models (LLMs), while maintaining high consistency and reliability. We propose a multi-model, multi-run framework in which several LLMs act as annotators and their outputs are aggregated using a strict consensus criterion to improve annotation robustness and reduce model-specific biases.The proposed approach is evaluated on a multilingual sentiment analysis task using lead sections of Wikipedia articles covering diverse topics in five languages. The resulting annotations are assessed using intra-model consistency, cross-model agreement, and standard inter-annotator agreement metrics. Additionally, we conducted human evaluation.The described framework enables scalable and reproducible data annotation, and the resulting dataset is publicly released to support further research.

More from EMNLP 2026

All EMNLP 2026 papers →