Misaligned RGB-Depth Boundary Identification and Correction for Depth Image Recovery

Raw depth images generally contain a large number of erroneous pixels near object boundaries due to the limitation of depth sensors. It induces misalignment of object boundaries between RGB and depth pairs. Most existing methods do not explicitly study such RGB-Depth misalignment problem. Thereby, depth boundaries cannot be accurately recovered. In this paper, a simple yet effective model is developed based on the guided filter (GF) to identify misaligned object boundaries of a raw depth image. Using GF to filter a raw depth image with the guidance of a reference RGB image, structure of the RGB image can be progressively transferred to filtered depth images as the window size of GF increases. Therefore, misaligned object boundaries in raw depth image can be identified from residuals of filtered depth images from large-size and small-size GFs. The model is embedded into Markov random field to correct misaligned object boundaries. It is restricted in fixed-width regions around depth boundaries to avoid texture-copy artifacts. The optimization problem is solved efficiently in an iterative way. Quantitative and visual results on three RGB-Depth datasets verify that the proposed method achieves the best results compared with recent optimization-based or learning-based baselines. In addition, the proposed method is effectively applied in no-reference depth quality assessment, depth super-resolution, and depth estimation enhancement.