Multi-Model Framework for Sentiment Annotation: A Use Case of Multilingual Wikipedia
Abstract
AbstractIn this work, we investigate whether data annotation can be automated using state-of-the-art large language models (LLMs), while maintaining high consistency and reliability. We propose a multi-model, multi-run framework in which several LLMs act as annotators and their outputs are aggregated using a strict consensus criterion to improve annotation robustness and reduce model-specific biases.The proposed approach is evaluated on a multilingual sentiment analysis task using lead sections of Wikipedia articles covering diverse topics in five languages. The resulting annotations are assessed using intra-model consistency, cross-model agreement, and standard inter-annotator agreement metrics. Additionally, we conducted human evaluation.The described framework enables scalable and reproducible data annotation, and the resulting dataset is publicly released to support further research.