Skip to content
conftrace_
2026 EMNLP EMNLP 2026

OLIVIA: Online Learning via Inference-time Action Adaptation

AbstractLarge language models (LLMs) tool-use policies select actions from a finite candidate set at each reasoning step. We observe that this action-selection interface is already a linear contextual scorer: under a restricted output vocabulary, the LM head computes per-action scores from the last-layer hidden state, exactly the structure assumed by linear contextual bandits. Existing online adaptation methods for LLMs operate indirectly through prompts or retrieved memories, leaving this decision layer fixed and providing no explicit mechanism for action-level credit assignment or uncertainty-aware exploration. We propose OLIVIA, a lightweight online adaptation layer that replaces the frozen action-selection head with a per-action contextual linear bandit, selects actions via LinUCB exploration on the LLM’s hidden state and updates only its per-action sufficient statistics from step-level feedback via a rank-one Sherman–Morrison rule. Without backbone fine-tuning and additional LLM calls per step, OLIVIA adds negligible overhead relative to LLM inference. Across four tool-use benchmarks ToolBench, TaskBench, TaskBench-MM, BFCL and two backbones Qwen3-4B, Mistral-7B, OLIVIA outperforms prompt-based online adaptation and contextual-bandit variants.

More from EMNLP 2026

All EMNLP 2026 papers →