Skip to content
conftrace_
2026 EMNLP EMNLP 2026

SCORE: Specificity, Context Utilization, Robustness, and Relevance for Reference-Free LLM Evaluation

AbstractLarge language models (LLMs) are increasingly used to support question answering and decision-making in high-stakes, domain-specific settings such as natural hazard response and infrastructure planning, where effective answers must convey fine-grained, decision-critical details. However, existing evaluation frameworks for retrieval-augmented generation (RAG) and open-ended question answering primarily rely on surface-level similarity, factual consistency, or semantic relevance, and often fail to assess whether responses provide the specific information required for domain-sensitive decisions. To address this gap, we propose a multi-dimensional, reference-free evaluation framework that assesses LLM outputs along four complementary dimensions: specificity, context utilization, robustness to paraphrasing and semantic perturbations, and answer relevance. We introduce ClimateRiskQA, a curated dataset of 1,412 domain-specific question–answer pairs spanning 40 professional roles and seven natural hazard types, and further evaluate on PubMedQA to examine cross-domain generalization. We also conduct human evaluation to assess inter-annotator agreement and alignment between model outputs and human judgments, which highlights the inherent subjectivity of open-ended, domain-specific evaluation. Our results show that no single metric sufficiently captures answer quality in isolation and demonstrate the need for structured, multi-metric evaluation frameworks when deploying LLMs in high-stakes applications.

More from EMNLP 2026

All EMNLP 2026 papers →