Skip to content
conftrace_
2026 AACL AACL 2026

Prompt Rewrites Are Semantic Actions: Testing Whether Language Models Obey Typed Operator Laws

AbstractPrompt rewrites can prescribe a change in an answer, not merely a change in wording. We introduce ORBIT-SIT, a framework that assigns each rewrite an explicit action on a task’s output space and separately measures correctness against transformed gold answers and consistency with transformed model predictions. Across eight operators, five datasets, and three instruction-tuned models, high task accuracy does not ensure compliance: Phi-4-mini achieves 90.7% accuracy on SST-2 but only 16.4% under a label-swap instruction. Deterministic validation and a three-annotator audit check the operator contracts. Non-deceptive binary controls retain a 17.8-percentage-point gap, and separating inference from transformation improves compliance, implicating joint execution rather than refusal alone. OrbitAlign, a quantized low-rank adaptation objective, reduces gold-grounded risk by 25.4 points on a 400-item BoolQ repair pool. The recipe lowers risk in 39 of 40 cells on two additional backbones, and same-action transfer across binary tasks reduces risk by 12.9–15.1 points. The binary tasks provide the primary evidence; structured tasks remain extensibility diagnostics. ORBIT-SIT makes declared prompt actions measurable and partly repairable without equating compliance with general understanding.

More from AACL 2026

All AACL 2026 papers →