End-to-end learning of self-rectification and self-supervised disparity prediction for stereo vision

Stereo rectification and stereo matching are two critical components for the practical application of stereo vision systems. Previous studies treat them as two individual issues. For stereo rectification, various traditional algorithms are proposed to estimate homography transformations, but the performance and the efficiency are unsatisfactory for real-time deployment. For stereo matching, disparity accuracy has been largely improved by learning based methods. However, the input data of all previous stereo networks are assumed to be a pair of offline pre-rectified images, making them invalidate for accurate matching when the stereo vision system suffers from mechanical misalignment due to external collisions or temperature variations. In this paper, we optimize these two components jointly and propose an end-to-end learning framework to achieve online self-rectification and self-supervised disparity prediction simultaneously. The overall network contains two cascaded subnetworks which enable stereo rectification and stereo matching sequentially for a pair of unrectified images. The experimental results are evaluated on both publicly available datasets and realistic scenarios. Evaluation results demonstrate that, the proposed network produces state-of-the-art results for self-rectification in terms of computation accuracy and speed, and also produces competitive disparity results with previous self-supervised methods. Therefore, the proposed design provides a more practical and efficient solution for stereo vision systems deployed on mobile platforms.