Learning Natural Pedestrian Policies via Human Demonstration for Human-Vehicle Interaction

Existing simulators still struggle to reproduce realistic human-vehicle interaction scenarios due to the lack of natural representation of human pedestrian movement. While VR-based human demonstration enables capturing realistic pedestrian experiences, it remains challenging to convert these sparse VR observations into scalable learning strategies for natural pedestrian behaviors. Our method utilizes a hierarchical spatio-temporal diffusion model to reconstruct full-body pedestrian poses from sparse VR demonstration signals, with its backbone network alternately encoding temporal and spatial dimensions. To improve the consistency of estimated posture, we incorporate concatenated conditional signals and repeated time-step embeddings. Furthermore, to bridge the gap between demonstrations and imitative behavior, we incorporate generative adversarial imitation learning (GAIL) to learn physically-constrained policies, enabling virtual avatars to imitate realistic behaviors. Experimental results on the AMASS dataset show our method significantly outperforms existing approaches in both position and rotation accuracy. We further validate our framework in a human-vehicle interaction environment, where pedestrian avatars reconstruct VR users’ demonstrated behaviors, exhibiting diverse and physically plausible actions such as walking, jumping, and crouching, highlighting the potential for more realistic human-vehicle interaction scenarios.