Video object segmentation by motion-based sequential feature clustering

Segmentation of video foreground objects from background has many important applications, such as human computer interaction, video compression, multimedia content editing and manipulation. Most existing methods work on image pixels or color segments which are computationally expensive. Some methods require extensive manual inputs, static cameras, and/or rigid scenes. In this paper we propose a fully automatic foreground segmentation method based on sequential clustering of sparse image features. The sparseness makes the method computationally efficient. We use both edge and corner points extracted from each video frame. A joint spatio-temporal linear regression method is developed to compute sparse motion layers of M consecutive frames jointly under the temporal consistency constraint. Once the sparse motion layers have been identified for each frame, the corresponding dense motion layers are created using the Markov Random Field (MRF) model. The MRF model assigns the rest of the image pixels to the motion layers by considering both the color attributes and the spatial relations between each pixel and its surrounding edge/corner points. Experimental evaluations on videos taken by webcams show the effectiveness of the proposed method.