This project investigates two human-robot interaction techniques that enable non-roboticist users to remotely convey intent to a robotic manipulator via a digital duplicate (a simplified virtual representation informed by the digital twin concept). We introduce a proof-of-concept system for remote human-robot collaboration in a pick-and-place task, wherein users convey their intent through embodied interaction. This can take the form of direct task demonstration (e.g., physically replicating a pick-and-place action on the digital duplicate) or high-level goal specification (e.g., "Pick the wristwatch in the scene and place it at location B"), requiring the robot to autonomously localise the referenced object via onboard perception. The system integrates a real-time object detection pipeline deployed on an edge AI device (Raspberry Pi) and employs a Dobot Magician robotic arm to perform the user-instructed tasks. The physical state and actions of the robot are synchronised with the digital duplicate, providing continuous visual feedback through a virtual reality headset. This architecture supports intuitive, intent-based human-robot interaction while abstracting low-level control complexities. Future directions include augmenting the digital interface with egocentric video streaming to enhance situational awareness and enable more immersive, multimodal interaction modalities.
Madhawa Perera
ImmersiveEnvs