Safe Exploration in Reinforcement Learning: Training Backup Control Barrier Functions with Zero Training-Time Safety Violations

This paper introduces the Reinforcement Learning Backup Shield (RLBUS), a framework that guarantees safe exploration in reinforcement learning (RL) by incorporating Backup Control Barrier Functions (BCBFs). RLBUS synthesizes an implicit control forward invariant subset of the safe set using multiple backup policies, ensuring safety in the presence of input constraints. While traditional BCBFs often yield conservative control forward invariant sets due to the design of backup controllers, RLBUS addresses this limitation by leveraging model-free RL to train an additional backup policy, enlarging the identified forward invariant subset of the safe set. This approach enables safe exploration of larger regions of the state space with zero safety violations during training. The effectiveness of RLBUS is demonstrated on an inverted pendulum example, where the expanded invariant set facilitates safe exploration over a broader state space, enhancing performance without compromising safety.