Telling Incomplete Truths: Measuring Omission-Induced Misleadingness in Aligned Language Models
Abstract
AbstractAligned language models are evaluated for whether they state falsehoods and whether users prefer their answers. Neither axis asks whether an answer carries what a reader needs to decide. We name that failure mode Pragmatic Reticence and build an instrument to measure it. A generator answers a passage-and-question prompt in free form, never seeing the answer options, and a frozen reader, never seeing the passage, decides from that answer alone. A natural language inference judge scores whether the answer conveys the passage’s decision-critical sentence. Across 47 generators the median non-reasoning model reaches a critical-fact recall of 0.28, while every reasoning-tuned checkpoint exceeds 0.74 on the whole response. Supervised fine-tuning and preference optimization do not close that gap. For models that emit a reasoning trace, the recalled fact usually sits inside the trace rather than in the answer that follows it, so a deployment that hides the trace hides the recall. In those models a trace-free prompt restores it, so the omission is a format effect. An answer can be true, preferred, and still not enough to act on.