单幅图像去雨数据集和深度学习算法的联合评估与展望

The visual quality of captured images in rainy weather conditions is constrained to the outdoor vision degradation. It is essential to design relevant rain removal algorithms. Due to the lack of temporal information, single image rain removal is challenging compared to video-based rain removal. The target of single image rain removal analysis is to restore the rain-removed background image from the corresponding rain-affected image. Current deep learning based vision tasks construct diverse data-driven frameworks like single image rain removal task via multiple network modules. Current research tasks are focusing on the quality of datasets, the design of single image deraining algorithms, the subsequent high-level vision tasks, and the design of performance evaluation metrics. Specifically, the quality of rain datasets largely affects the performance of deep learning based single image deraining methods, since the generalization ability of deep single image rain removal is highly related to the domain gap between synthesized training dataset and real testing dataset. Besides, rain removal plays an important preprocessing role in outdoor visual tasks because its result would affect the performance of the subsequent visual task. Additionally, the design of image quality assessment (IQA) metrics is quite important for the fair quantitative analysis of human perception of image quality in general image restoration tasks. We conducted critical literature review for deep learning based single image rain removal from the four aspects as mentioned below: 1) dataset generation in rain weather conditions; 2) representative deep neural network based single image rain removal algorithms; 3) the research of the downstream high-level task in rainy days and 4) performance metrics for evaluating single image rain removal algorithms. Specifically, in terms of the generation manners, the current rain image datasets are roughly divided into four categories as following: 1) synthesizing rain streaks based on photo-realistic rendering technique and then adding them on clear images based on simple physical model; 2) constructing rain images based on complex physical model via manual parameters setting; 3) generating rain images based on generative adversarial network (GAN); 4) collecting paired rain-free/rain-affected images by shooting different scenarios and adjusting camera parameters. We reviewed the download links of the existing representative rain image datasets. For deep learning based single image rain removal methods, we review the supervised and semi-/unsupervised rain removal methods for single task and joint tasks in terms of task scenarios, learning mechanisms and network design. Here, single task is relevant to rain drop removal, rain streak, rain fog, heavy rain; and the integrated analyses of removal of rain drop and rain streak, or multiple noises removal. Furthermore, we overview the construction manners of representative network architectures, including simplified convolutional neural networks based (CNNs-based) multi-branches architecture, GAN-based mechanism, recurrent and multi-stage framework, multi-scale architecture, the integration of encoder-decoder modules, attention mechanism or transformer based module as well as model-driven and data-driven learning manners. Since the implicit or explicit embedding of domain knowledge can promote network construction, we provide a detailed survey in the context of the relationship between rain removal methods and domain knowledge. We illustrated that the domain knowledge and the learning of benched networks has the potential to improve the generalization performance of single image rain removal algorithm further. Based on the real high-level outdoor vision tasks in rain weather, it would be meaningful to use the joint processing strategies of low-level and high-level tasks and the customized construction of rainy datasets. Meanwhile, we reviewed and clarified some related literatures of high-level computer vision tasks and comprehensively analyzed the performance evaluation metrics in the context of full-reference metrics and non-reference metrics. We analyzed the potential challenges of single image rain removal further in the context of feasible benchmark datasets construction, future fair evaluation metrics designing, and the optimized integration of rain removal and high-level vision tasks.