3D helping hands: a gesture based MR system for remote collaboration

There is currently a strong need for collaborative systems with which two or more participants interact over a distance on a task involving tangible artifacts (e.g., a machine, a patient, a tool). The present paper focuses on the specific category of remotecollaboration systems where hand gestures are used by a remote helper to assist a physically distant worker to perform manual tasks. Existing systems use a combination of video capturing, 2D monitors or 2D projectors, however displaying a video of the remote workspace and allowing helpers to gesture over the video does not provide helpers with sufficient understanding of the spatial relationships between remote objects and between their hands and the remote objects. In this paper we introduce our tele-presence Mixed Reality system for remote collaboration on physical tasks based on real-time capture and rendering of the remote workspace and of the helper’s hands. We improve on previous 2D systems introducing 3D capturing and rendering, and exploiting the possibility offered by the use of real 3D data to increase the feeling of immersion offered by the system using head tracking, stereoscopic rendering, inter-occlusion handling and virtual shadowing. We performed initial usability test of our system to verify if users are satisfied with the spatial awareness the system provides.