A hierarchical partially observable Markov decision process framework for tunnel boring machine trajectory navigation

The purpose of Tunnel Boring Machine (TBM) trajectory navigation is the precise excavation along a designed trajectory guided by flexible control strategy under uncertain underground conditions. The main challenge lies in determining appropriate thrust force to balance the geotechnical loads at the excavation face, and drive the TBM along the current optimal excavation path. Owing to the impracticability of directly measuring geotechnical information, the TBM trajectory navigation problem is essentially a Partially Observable Markov Decision Process (POMDP). To address it, a hierarchical POMDP framework is proposed to reveal the temporal and causal relationships between unobservable geotechnical effects and observable motion states. Within this framework, the optimal path planning for TBM movement is achieved using Reinforcement Learning (RL), based on the observable TBM position and orientation states. According to the planned path, the temporal relationship of observable states is modeled using a Gated Recurrent Unit (GRU) network, and combined with preliminary geotechnical data to support the final thrust decision-making. A TBM Patch-based Contact-Driven Motion (PCDM) model is established to simulate the motion process and force conditions, serving as the environment for interaction with the RL agent. Experimenting with various construction tasks validated the path planning, revealing average position and angle errors of 0.007 m and 0.415°, respectively. Additionally, using diverse geotechnical conditions as an illustrative case, the approach achieved a 96.07 % success rate in predicting thrust force variations. This showcased the effectiveness in forecasting TBM thrust during operation, with further confirmation through fine-tuning using real-world data to validate its generalization ability.