Video object segmentation by integrating trajectories from points and regions

We describe a novel video object segmentation system based on a conditional random field model with high-order term which is capable of capturing longer-range spatial and temporal grouping information. Our system is able to segment different moving objects effectively from complex background due to integrating the complementary properties of trajectories from points and regions. Although point and region trajectories have already been used in video object segmentation, their complementary properties have not been well investigated. In this paper, we propose an ingenious scheme to transfer the labels of sparse point trajectories to region trajectories. Especially, for region trajectories with few texture, this scheme can automatically predict their label probabilities by using a Gaussian mixture model of appearance and motion given the labels of point trajectories. Meanwhile, we design a reliability measurement for region trajectories based on shape consistency, which helps us to design robust high-order potentials for spatially overlapping region trajectories. Our region trajectories are extracted from hierarchical image over-segmentation, and hence they can capture meaningful regions over time. Additionally, our approach is a streaming process, in which object labels are propagated over a video.