Construction of Deep ReLU Nets for Spatially Sparse Learning

Training an interpretable deep net to embody its theoretical advantages is difficult but extremely important in the community of machine learning. In this article, noticing the importance of spatial sparseness in signal and image processing, we develop a constructive approach to generate a deep net to capture the spatial sparseness feature. We conduct both theoretical analysis and numerical verifications to show the power of the constructive approach. Theoretically, we prove that the constructive approach can yield a deep net estimate that achieves the optimal generalization error bounds in the framework of learning theory. Numerically, we show that the constructive approach is essentially better than shallow learning in the sense that it provides better prediction accuracy with less training time.