Audio-to-3D: One-shot talking face generation with disentangled latent codes and diffusion control

Audio-driven talking face generation has a wide range of applications and has seen significant progress in recent years. However, existing methods remain limited in their ability to maintain consistent identity and accurate audio-lip synchronization when generating novel-view videos from a single image. In this paper, we propose Audio-to-3D (A-to-3), a novel framework for generating 3D talking faces from a single image guided by audio. To extract 3D facial features, we introduce a 3D Facial Decoupling Encoder (3DE) that efficiently disentangles lightweight codes for identity, expression, and pose. We further present the Audio-driven Hyper Diffusion Model (AHDM), a modified diffusion-based model designed to capture realistic 3D facial motion while preserving identity consistency. The resulting facial codes are decoded into tri-planes and rendered at high resolution. Our approach is trained on monocular video data and supports arbitrary audio inputs, enabling the generation of high-quality, multi-identity talking face videos with controllable viewpoints. Extensive experiments validate the effectiveness of A-to-3 in one-shot talking face generation.