Skip to content
conftrace_
2026 EMNLP EMNLP 2026

From Monolithic Prompts to Scaffolded Execution for Memory Agents with Small Language Models

AbstractPersonal memory agents process sensitive user data, making on-device deployment with small language models (SLMs) attractive. However, existing memory-agent prompts are often monolithic specifications written for high-capacity LLMs, overloading a single inference with many requirements while leaving necessary intermediate operations implicit. This mismatch creates two bottlenecks for SLMs: prompt complexity and logical incompleteness. We introduce CAST (Compiling Agent Specifications into Scaffolded Tasks), a framework that converts a monolithic memory-agent prompt into a scaffolded execution procedure. CAST segments the original prompt into fixed, semantics-preserving main tasks, then synthesizes and places auxiliary scaffolding tasks that produce reusable intermediate artifacts; a trajectory-feedback step then adapts these scaffolds to the target SLM. At inference time, only the target SLM executes the compiled procedure. Across two memory-agent frameworks on LoCoMo and MemoryAgentBench, CAST outperforms monolithic prompting, prompt optimization, and reasoning-based baselines, while offering a stronger quality-latency trade-off than unconstrained reasoning. The results show that adapting memory agents to SLMs requires restructuring execution, not merely rewriting prompts.

More from EMNLP 2026

All EMNLP 2026 papers →