Skip to content
conftrace_
2026 EMNLP EMNLP 2026

When Targeted Grounded Gains Hide Complementary Abstraction Regression: DION-Bench and a Conjunctive Evaluation Protocol

AbstractPost-training with internal feedback can improve a targeted knowledge-grounded behavior while silently degrading complementary abstraction and logical-consistency capabilities. We propose a conjunctive evaluation protocol that accepts a method only when a designated target improves and complementary probes remain non-inferior (with a stricter secondary criterion requiring complementary equivalence), while also reporting retention and predictive drift. We instantiate the protocol in DION-Bench, combining two lexical/ontological probes (same-category exceptions; abstraction recovery) with a synthetic SAT-checked logical-consistency probe. On Qwen3-8B-Base, a task-specific self-judged DPO adaptation improves OntoNeg-WN by 5.27 percentage points but reduces ConceptShift by 37.50 points—despite humans reaching 82.7% and frontier LLMs 44–73% on OntoNeg—whereas mean change across five external accuracy suites is only −1.70 points. A development-tuned decoding rule partially mitigates ConceptShift regression to 12.93/14.60 points on the original/holdout splits while retaining most of the target gain. These results show that faithful post-training claims require joint reporting of gain, complementary abstraction regression, retention, drift, and equivalence—not proof that a single shared knowledge item degraded.

More from EMNLP 2026

All EMNLP 2026 papers →