Combining local and global hypotheses in deep neural network for multi-label image classification

Multi-label image classification is a challenging problem in computer vision. Motivated by the recent development in image classification performance using Deep Neural Networks, in this work, we propose a flexible deep Convolutional Neural Network (CNN) framework, called Local-Global-CNN (LGC), to improve multi-label image classification performance. LGC consists of firstly a local level multi-label classifier which takes object segment hypotheses as inputs to a local CNN. The output results of these local hypotheses are aggregated together with max-pooling and then re-weighted to consider the label co-occurrence or interdependencies information by using a graphical model in the label space. LGC also utilizes a global CNN that is trained by multi-label images to directly predict the multiple labels from the input. The predictions of local and global level classifiers are finally fused together to obtain MAP estimation of the final multi-label prediction. The above LGC framework could benefit from a pre-train process with a large-scale single-label image dataset, e.g., ImageNet. Experimental results have shown that the proposed framework could achieve promising performance on Pascal VOC2007 and VOC2012 multi-label image dataset.