ExpRL: Unified Non-Parametric and Parametric Experience Learning for LLM Agent Evolution
Abstract
AbstractLarge language model (LLM)-based agents solve complex long-horizon tasks through iterative interaction with environments, where effectively learning from experience under sparse or delayed feedback remains a key challenge. Existing approaches typically rely on either non-parametric memory-based experience reusing or parametric reinforcement learning, each with inherent limitations when used in isolation. To this end, we propose **ExpRL**, a unified experience learning framework that bridges non-parametric memory and parametric policy optimization for LLM agents. ExpRL induces abstracted success and failure experiences from on-policy interaction trajectories, stores them in an external memory, and integrates experience acquisition and utilization directly into the reinforcement learning process via an agentic experience acquisition mechanism. This design enables memory and policy to co-evolve and mutually reinforce each other, effectively unifying online policy learning with offline experience reuse. Experiments on ALFWorld and WebShop demonstrate that ExpRL achieves an average performance improvement of **13.0** over GRPO, with improved learning efficiency and stability. We further show that the induced experience memory generalizes across different models and can be directly transferred to stronger LLMs without additional training.