Human action recognition with contextual constraints using a RGB-D sensor

Recognition of human actions using a vision based approach is a challenging task. To improve the action recognition performance, we proposed a hierarchical probabilistic model based framework which not only models the dynamics of the actions but also considers contextual constraints in terms of object/action correlation and action sequential constraints. By considering the action/object correlation, it is possible to recognize actions which are either too subtle to perceive or too hard to recognize using motion features only. On the other hand, with the action sequential constraints, the recognition accuracy can be further improved. In the proposed approach, first, the dynamics of an action is modeled using Hidden Markov Models (HMMs). Then, a Bayesian network is adopted to model the object constraints for the low-level action recognition. Finally, a high-level HMM is created to model the sequential constraints which refines the decision from the Bayesian model. Our approach was evaluated through experiments using a single RGB-D camera, which provides data of both the human gesture and manipulated objects. The experimental results show that the proposed approach can recognize human actions effectively.