S²NeRF: Neural Radiance Fields Training with Sparse Points and Sparse Views

Neural volume rendering methods, especially NeRF, have demonstrated remarkable performance in novel view synthesis. However, NeRF relies solely on image data and lacks explicit geometric information, necessitating a large number of posed images and a computationally intensive ray sampling strategy to learn accurate scene representations. This poses challenges and may result in incomplete or locally optimal scene geometry when views are sparse or incomplete, as the limited views may not provide sufficient constraints to determine a unique geometry solution for complex scenes. Meanwhile, sparse point clouds provide an attractive source of scene information, especially for geometry, to complement images in neural scene representations, particularly when input views are sparse. To overcome these limitations, we propose S²NeRF, a novel Neural Radiance Field that simultaneously incorporates features from both point clouds and images for volume rendering. Specifically, S²NeRF extracts patch-wise point features from point clouds and ray-wise image features from adjacent views. Then the scene feature volume is constructed by implicitly fusing these point and image features through self-attention. Finally, the volume feature is utilized to render novel views of the scene. Experimental results on the challenging TartanAir dataset demonstrate that, thanks to the integration of feature volume from point clouds and images, S²NeRF achieves state-of-the-art performance in novel view synthesis.