Large and small-scale models’ fusion-driven proactive robotic manipulation control for human-robot collaborative assembly in industry 5.0

Human-robot collaborative (HRC) assembly has been popular by combining human creativity and dexterity with robotic precision for higher assembly efficiency and resilience in industry 5.0. Nevertheless, current HRC assembly systems rely predefined codes, limiting robot adaptability to dynamic and unstructured assembly environments. To bridge the gap, this paper proposes a novel proactive robotic manipulation control method for HRC assembly, which fully utilizes large-scale model (LSM) in cognitive computing and reasoning for dynamic robotic control path planning, and small-scale models (SSMs) in efficiently computing for dynamic robotic control demand perception and control constraints verification. Specifically, LSM, namely ChatGPT 4o, is deployed on the cloud to proactively generate robotic control constraints according to the robotic control demand derived from SSMs on the edge. Here, two kinds of SSMs are developed, including robotic control demands perception model and robotic control constraints verification model. For robotic control demands perception, an ensemble encoder model is proposed for ongoing human assembly action detection, on which a vision model and fine-tuned assembly instruction generation model are designed for assembly manipulation keypoints image and robot control instruction generation, serving as the input for LSM. For robotic control constraints verification, a digital twin model is used to verify the control constraints derived from LSM, where verified constraints are used for robotic control during assembly process. Finally, the feasibility and effectiveness of the proposed approach are demonstrated through experiments on an HRC assembly process, where over 99 % accuracy for human assembly action detection and 80 % task execution accuracy are conducted.