Evaluating Communicative Belief Updates in Large Language Models via Implicature Recognition and Cancellation
Abstract
AbstractHuman language is driven by unspoken beliefs and belief updates which are thus of critical importance for successful interactions between large language models (LLMs) and their users. In this paper, we evaluate the ability of LLMs to recognize unspoken beliefs made through implicatures and to recognize their updates through implicature cancellation: the pragmatic phenomenon whereby an utterance’s implied meaning is weakened or negated. We create the first expert-annotated implicature cancellation dataset, ImplicatureX, crowdsourced for human judgements of implicatures and their corresponding cancellations. We find that LLM belief update understanding lags behind that of humans, especially in more naturally-occurring scenarios. Additional control experiments suggest that successes in LLM belief updates may in part stem from a reliance on prior beliefs, and that failures in belief updates may depend on their type and on their form. Overall, our study suggests that current LLMs have not yet reached human-level understanding of communicative belief updates.