Relationship detection for manipulation in object stacking scene with fully connected CRF

Relationship detection for manipulation in object stacking scenes is an actively studied problem in recent years. Since the stacked objects may occlude and block each other, it is necessary to model relationships among them for safety. Almost all existing solutions to this problem model object relationships separately and ignore their dependencies on each other. In this paper, we propose a fully connected Conditional Random Field (CRF) to impose global constraints on the object stacking scenes. Since the inference of fully connected CRF is usually intractable, we propose efficient methods for the exact inference and the variational inference to solve the proposed CRFs. Furthermore, the variational inference is implemented as an RNN for end-to-end training. Experimental results on Visual Manipulation Relationship Dataset VMRD and REGRAD suggest that the proposed CRF can, with a clear margin, improve the accuracy of manipulation relationship detection in object stacking scenes. Finally, the proposed models are also deployed on a physical robot to build a relation-aware grasping system. Results demonstrate that our algorithms can be generalized to real-world scenes well and help the robot perform safe grasping in object stacking scenes.