Skip to content
conftrace_
2026 EMNLP EMNLP 2026

Large Language Models as Automatic Annotators and Annotation Adjudicators for Fine-Grained Opinion Analysis

AbstractFine-grained opinion analysis enables a detailed understanding of expressed sentiments and their corresponding targets. However, annotating domain-specific datasets requires substantial human effort and financial resources. To address this limitation, we investigate the feasibility of using large language models (LLMs) as automatic annotators for fine-grained opinion analysis. We use a declarative annotation pipeline to reduce prompt engineering when identifying fine-grained opinion spans, and compare rule-based voting against an LLM-based adjudicator for reconciling conflicting predictions. We trial the pipeline across different model sizes on the Aspect Sentiment Triplet Extraction (ASTE) and Aspect-Category-Opinion-Sentiment (ACOS) tasks. Our findings indicate a critical performance bifurcation: LLMs are reliable at the span level yet struggle to faithfully reproduce the relational structures connecting those spans. The better aggregation strategy depends on this complexity: voting suffices for triplets, while quadruples require an adjudicator that re-reads the source. Treating annotator disagreement as an informative signal rather than noise, we further show that agreement between LLM annotators is a calibrated predictor of correctness and can be used to delegate cases with high annotator disagreement to humans.

More from EMNLP 2026

All EMNLP 2026 papers →