Answer or Compute? Direct Inference vs. Code Generation for LLM Game-State Understanding in RTS Games
Abstract
AbstractA large language model (LLM) can understand a game state in two ways: answer questions about it directly, or write a program that computes the answer. We compare these modalities on three real-time-strategy tasks—factual extraction, strategic assessment, and win prediction—holding states, questions, and models fixed. On mechanical questions, code wins decisively: hit-point (HP) aggregation rises from 30% to 100%, and the direct-mode failures are mostly a refusal to declare ties, not bad arithmetic. On judgment tasks, code seemed to win as well, until a trivial baseline exposed why: on snapshots from bot games the material leader always wins, so sign(ΣHP) alone scores 100%. On contested states built to remove this artifact, no arm—direct, chain-of-thought, or code-based—reliably beats chance (40 tests, Holm-corrected), while a simple unit-count cue reaches 72%. Code makes LLMs reliable aggregators; it does not make them judges. We contribute the shortcut audit and hard-set construction that judgment benchmarks built on game traces should adopt.