Hierarchical Policy Optimization for Cooperative Multi-Agent Reinforcement Learning

To handle the non-stationarity of the environment and the curse of dimensionality issues in multi-agent reinforcement learning, gathering information through communication is a critical part. Existing frameworks have proposed centralized or distributed structures to deal with the problem. However, they either have problems of robustness or problems of high communication costs. This paper adopts a hierarchical zeroth-order policy optimization (HZOPO) algorithm for cooperative multi-agent reinforcement learning (MARL) problems. The agents are divided into different groups with high-and low-level entities. A hierarchical communication structure is implemented to reach global consensus. It is shown that the HZOPO algorithm can balance both convergence and communication efficiency in cooperative MARL environments. The convergence of the algorithm is also proved.