LoCoDec: Robust Decoding for LLM-based Sequence-Structured Prediction
Abstract
AbstractMany Natural Language Processing (NLP) tasks, including information extraction, dialogue understanding, and sentiment analysis, can be formulated as structured prediction over sequences or spans. Although Large Language Model (LLM)-based methods have achieved strong performance on standard benchmarks for such tasks, their robustness under natural sentence-level perturbations and their controllability in producing valid structures remain insufficiently studied. In this paper, we propose Locality-aware Constrained Decoding (LoCoDec), a method for LLM-based structured prediction over sequences and spans that formulates decoding as constrained token paraphrasing. LoCoDec combines neighborhood context, which reduces long-text noise, with structural decoding constraints that enforce source-faithful and well-formed outputs. Experiments on sixteen benchmarks of six tasks show that LoCoDec remains competitive in S-F1 and P-F1, while achieving the best R-F1 on all datasets. Ablation results show that removing neighborhood context reduces R-F1 by up to 16.23 points. Overall, LoCoDec improves the robustness and structural consistency of LLM-based structured prediction.