Incremental projection vector machine: A one-stage learning algorithm for high-dimension large-sample dataset

Dimension reduction has been widely employed to deal with the curse of dimensionality before training supervised learning such as neural network and this framework combining dimension reduction and supervised learning algorithms is called as two-stage approach. However during the process of this approach, the system has to store original data and pre-process data simultaneously which will increase the complexity and re-compute the SVD when the new data arrive. To address the above problems, this paper proposes a novel learning algorithm for high-dimension large-scale data, by combining a new incremental dimension reduction with feed-forward neural network training simultaneously, called Incremental Projection Vector Machine (IPVM). With new samples arriving, instead of re-computing the full rank SVD of the whole dataset, an incremental method is applied to update the original SVD. It is suitable for high-dimension large-sample data for the singular vectors are updated incrementally. Experimental results showed that the proposed one-stage algorithm IPVM was faster than two-stage learning approach such as SVD+BP and SVD+ELM, and performed better than conventional supervised algorithms.