Object affordances graph network for action recognition

Human actions often involve interactions with objects, and such action possibilities of objects were termed “affordances” in human-computer interaction (HCI) literature. To facilitate action recognition with object affordances, we propose the Object Affordances Graph (OAG), which cast human-object interaction cues into video representations via an iterative refinement procedure. With the spatio-temporal co-occurrences between human and objects captured, the Object Affordances Graph Network (OAGN) is subsequently proposed. To provide a fair evaluation of the role that object affordances could play on human action recognition, we have assembled a new dataset with additional annotated object bounding-boxes to account for human-object interactions. Multiple experiments on this proposed Object-Charades dataset verify the value of including object affordances in human action recognition, specifically via the proposed OAGN, which outperforms existing state-of-the-art affordance-less action recognition methods.