Ability-Aligned Skill Evolution for LLM Agents via Reinforcement Learning
Abstract
AbstractLarge Language Model agents augmented with external skill banks have demonstrated remarkable performance on sequential decision-making tasks. However, existing methods update skills in an undirected, unverified manner and prematurely penalize agents for failing to utilize immature skills. We introduce a structured cognitive representation that decomposes both general and task-specific skills into four dimensions—D-PLAN, D-TRACK, D-EXEC, and D-GUARD—enabling precise failure attribution. Building upon this representation, we propose A2SE, an ability-aligned skill evolution framework that synchronizes skill bank updates with the agent’s learning trajectory. Specifically, A2SE concentrates evolution on the identified bottleneck dimension, safeguards updates via a causal verification mechanism with per-task-type rollback, and employs a competence-aligned skill reward to alleviate premature penalization. Extensive experiments on ALFWorld and WebShop demonstrate that A2SE achieves 11.0% and 7.1% improvements over the strongest baselines, respectively, with ablation studies confirming the necessity of each proposed component.