Hierarchical Twin-Delayed Policy Gradient Reinforcement Learning for Intelligent Cooperative Control of Aircraft

To address the modeling and coordination challenges in intelligent cooperative control of aircraft caused by large-scale systems, complex environments, and resource constraints, this study proposes an intelligent cooperative control method by establishing a hierarchical multi-agent decision-making architecture with the goal of improving decision-making algorithm efficiency. First, aircraft is modeled as an intelligent agent to establish a cooperative control framework. Second, a partially observable Markov decision process (POMDP) model is employed to handle incomplete observation information. Then, to tackle the issues of dynamic game environments and high learning costs, a hierarchical twin-delayed policy gradient reinforcement learning method based on centralized training with decentralized execution is proposed, which effectively combines model-based and model-free mechanisms to leverage existing game environment evolution models. Finally, under the hierarchical decision-making framework, simulations of typical multi-aircraft game scenarios and thousands of multi-scenario tests are conducted. The results demonstrate that the proposed method successfully resolves multi-aircraft cooperative control problem. Compared to the multi-agent reinforcement learning algorithms MAPPO and QMIX, the training time is reduced by 51.03% and 79.03%, algorithm efficiency (cumulative reward) is improved by 37.51% and 58.73%, and evasion maneuver success rate is increased by 17.63% and 39.79%, respectively.