Volumetric Hand Reconstruction and Tracking to Support Non-Verbal Communication in Collaborative Virtual Environments

Abstract

The success of future teleconferencing solutions will depend on their ability to support a wide range of expressions and interactions. Collaborative Virtual Environments (CVEs) represent a class of promising technologies to achieve this. However, in today’s office situations CVE applications are usually quite limited due to the lack of appropriate interface support for gestural communication and gesture-based interaction with virtual artefacts. These tasks require the development of efficient hand tracking or hand reconstruction solutions and their integration into affordable desktop teleconferencing environments.

A new vision-based tabletop interface for non-obtrusive volumetric reconstruction and tracking of hands and its integration into a CVE is presented. The application of a unified probabilistic approach for reconstruction, tracking, and hand appearance capture thereby results in an almost configuration-free system design which can even cope with uncontrolled background scenes. The proposed interface embeds users of 3D virtual teleconferencing applications into a common world frame and thereby introduces hands as an additional non-verbal communication and interaction channel. The integration is based on a new probabilistic Shape from Silhouette algorithm which employs multiple cameras to interactively reconstruct a user’s body volume in a desk-based working environment. The reconstructed body volume serves as data source for hand and articulated upper body tracking and is employed alongside with tracked hand locations to interactively compute polygonal mesh descriptions of hand volumes.

A reference implementation of the proposed teleconferencing solution is introduced which is exclusively based on off-the-shelf hardware components that facilitate a wide range of cost-efficient applications. Compliance with real-time constraints is thereby achieved through a massive parallel algorithmic design and Graphics Processing Unit-based (GPU) implementation of reconstruction and tracking components. As a proof of concept finally also an empirical study is presented which evaluates the developed CVE with respect to refined communication and collaboration quality. The findings of this thesis contribute to a new area of human-computer-interface and teleconferencing research by developing tools that support the consolidation of local and remote conference situations.