Visual Manipulation Relationship Detection based on Gated Graph Neural Network for Robotic Grasping

Exploring the relationship among objects and giving the correct operation sequence is vital for robotic manipulation. However, most previous algorithms only model the relationship between pairs of objects independently, ignoring the interaction effect between them, which may generate redundant or missing relations in complex scenes, such as multi-object stacking and partial occlusion. To solve this problem, a Gated Graph Neural Network (GGNN) is designed for visual manipulation relationship detection, which can help robots detect targets in complex scenes and obtain the appropriate grasping order. Firstly, the robot extracts feature from the input image and estimate object categories. Then GGNN is used to effectively capture the dependencies between objects in the whole scene, update the relevant features, and output the grasping sequence. In addition, by embedding positional encoding into pair object features, accurate context information is obtained to reduce the adverse effects of complex scenes. Finally, the constructed algorithm is applied to the physical robot for grasping. Experiment results on the Visual Manipulation Relationship Dataset (VMRD) and the large-scale relational grasp dataset named REGRAD show that our method significantly improves the accuracy of relationship detection in complex scenes, and can be well generalized in the real world.