Skip to content
conftrace_
2026 ICML ICML 2026

RealtimeTool: Parallel Decoding for Real-Time LLM Function Calling

LLM-based function calling enables intelligent agents to interact with external tools and environments, yet autoregressive decoding imposes a fundamental latency bottleneck that limits real-time applications such as embodied intelligence, game AI, and interactive avatars (e.g., 10 Hz control frequency). We observe that function calling differs fundamentally from free-form text generation: structured outputs exhibit substantial token redundancy (delimiters, parameter names) and weak causal dependencies among arguments—two properties that must be exploited jointly to achieve real-time performance. We present RealtimeTool, which introduces special tokens that serve a dual role: compressing low-entropy tokens (4–6$\times$ reduction) while acting as mode selectors that enable independent parallel generation of function name and arguments. This synergistic design achieves 3–6$\times$ end-to-end speedup (up to 9.6$\times$) with only +8.2% parallelization overhead, while maintaining competitive or improved accuracy across five benchmarks on Qwen-series models (0.5B–14B). With quantization on a consumer-grade GPU, RealtimeTool reaches 61.2 ms P50 latency at 4B scale—enabling 16 Hz real-time control and bridging the gap between LLM function calling and latency-critical real-world deployment.

More from ICML 2026

All ICML 2026 papers →