2012
NeurIPS
NeurIPS 2012
Regularized Off-Policy TD-Learning
Abstract
We present a novel $l_1$ regularized off-policy convergent TD-learning method (termed RO-TD), which is able to learn sparse representations of value functions with low computational complexity. The algorithmic framework underlying RO-TD integrates two key ideas: off-policy convergent gradient TD methods, such as TDC, and a convex-concave saddle-point formulation of non-smooth convex optimization, which enables first-order solvers and feature selection using online convex regularization. A detailed theoretical and experimental analysis of RO-TD is presented. A variety of experiments are presented to illustrate the off-policy convergence, sparse feature selection capability and low computational cost of the RO-TD algorithm.
🌉
Interdisciplinary Bridge
- Artificial Intelligence and Reinforcement Learning
📈
Trend Setter
- Reinforcement Learning
🐣
Hot Topic Early Bird
- value function
Authors
Topics
Machine Learning > Core Methods > Representation Learning
Machine Learning > Optimization & Theory > Optimization
Reinforcement Learning
Reinforcement Learning > Methods > Deep RL
Reinforcement Learning > Methods > Offline RL
Machine Learning > Learning Types > Reinforcement Learning
Machine Learning > Optimization & Theory > Sparse Optimization
Machine Learning > Learning Types > Off-Policy Learning
Keywords
reinforcement learning
temporal difference learning
convex optimization
feature selection
sparse representation
l1 regularization
sparse feature selection
value function
value function approximation
off-policy learning
off-policy reinforcement learning
off-policy td-learning
convex-concave saddle-point
first-order solvers