A Constrained Deep Reinforcement Learning Approach for Charging Scheduling of a Battery Swapping Station

Battery swapping station (BSS) can provide fast battery swapping and flexible battery charging in off-peak hours, it is thus beneficial for electric vehicles (EVs) and power grid in terms of battery life extension and power grid regulation. However, this increases the charging scheduling complexity in BSS since the batteries are not necessarily required to be charged immediately as they arrived. This problem becomes challenging in the presence of nonlinear battery charging characteristics and demand/supply uncertainties. Since it is difficult for the traditional learning-based methods to deal with the constraints caused by nonlinear charging characteristics, low sampling efficiency and unstable training issues can occur. In order to solve these issues, we present a novel deep reinforcement learning (DRL) approach. In contrast to the traditional approaches where the battery charging characteristic is simplified to a constant-current or constant-power process, we propose an equivalent circuit model (ECM) to capture the nonlinear charging characteristics. In ECM, the battery’s open-circuit voltage (OCV) is a function of its state of charge (SoC), as a result, the upper bound of the charging/discharging power of battery is influenced by its SoC. Then we construct a constrained Markov decision process (CMDP) model and propose a Beta distribution-based DRL approach with a continuous action mask (AM) to improve the sampling efficiency and consistency of the training process. Numerical experiments show that our new approach can provide better results in terms of operation cost and quality of service (QoS) in comparison with other state-of-the-art DRL methods.