Real-time multiple object tracking and anomaly detection

In this paper we describe a real time video surveillance system which is capable of tracking multiple objects simultaneously and detecting violations. The number of objects is unknown and varies during tracking. Based on preliminary results of object detection in each image which may have missing and/or false detections, the multiple object tracking algorithm keeps a graph structure where it maintains multiple hypotheses about the number and the trajectories of the objects in the video. The image information drives the process of extending and pruning the graph, and determines the best hypothesis to explain the video. While the image-based object detection makes a local decision, the tracking process confirms and validates the detection through time, therefore, it can be regarded as temporal detection which makes a global decision across time. The multiple object tracking algorithm gives feedbacks which are predictions of object locations to the object detection module. Therefore, the algorithm integrates object detection and tracking tightly. The multiple object tracking algorithm is reliable to deal with occlusions, irregular object motions, changing appearances by postponing the decision of object trajectories until sufficient information is accumulated over time. The most possible hypothesis provides the multiple object tracking result which is used to accomplish anomaly detection. The trajectories generated by the tracking algorithm provide information of object identifications, motion histories, timing at sensitive areas and object interactions. Therefore, we are capable of detecting abnormal behaviors which are related to object counting, interaction, motion and timing. Since objects are tracked all the time, the anomaly detection module can also provide a scheme to report traffic information. The system has been running at a few access control areas for more than eighteen months. Experimental results on human tracking are presented and applications to anomaly detection are described.