MMMNet: An End-to-End Multi-Task Deep Convolution Neural Network with Multi-Scale and Multi-Hierarchy Fusion for Blind Image Quality Assessment

As the evaluation of image quality depends on the human visual system (HVS), many existing image quality assessment (IQA) methods focus on modeling the HVS to account for subjective perception. The visual attention of the HVS makes humans more sensitive to distortion on the attended regions than on regions which are not the focus of attention. Therefore, we propose an end-to-end multi-task deep convolution neural network with multi-scale and multi-hierarchy fusion (MMMNet), in which the IQA and saliency subtasks are jointly optimized to improve saliency-guided IQA performance. Particularly, the incorporation of saliency information is achieved by fusing saliency features with IQA features hierarchically to progressively improve the IQA features over network depth. A multi-scale feature extraction module (MSFE) is proposed to provide effective saliency features for the IQA network. Based on the saliency fusion, MMMNet introduces an auxiliary saliency task, achieving the multi-task learning to improve the generalization of the IQA task. Experimental results show that MMMNet achieves state-of-the-art performance and strong generalization ability on IQA databases.