Hierarchical reinforcement learning for kinematic control tasks with parameterized action spaces

Most existing reinforcement learning (RL) algorithms are solely applied to scenarios with pure discrete action space or pure continuous action space. However, in certain real-world kinematic control tasks that involve robot control based on kinematic properties, the action space is parameterized, wherein actions are represented by a fusion of discrete actions and continuous parameters. In this paper, we propose a hierarchical RL architecture designed specifically for handling parameterized action spaces. Our architecture consists of two levels, the higher level (discrete actor network) selects the discrete action and the lower level (continuous actor networks) determines the corresponding continuous parameters. These components work in tandem to generate an action-parameters vector to interact with the environment. Both the higher and lower levels share the rewards of environmental feedback and the critic networks to update the network weights. The soft actor critic and deep deterministic policy gradient algorithms are adopted to update higher-level and lower-level policies, respectively. Through simulation experiments conducted on different kinematic control tasks with parameterized action spaces, we demonstrate the effectiveness of our proposed algorithm.