To enable real-time interactive dance choreography in tele-immersive environments, multiple challenges need to be overcome. Some of the challenges have been solved, others are still in progress.
One of the major challenges is to accomplish 3D real-time video reconstruction. The 3D reconstruction algorithms are still lacking behind the 2D video capabilities in real-time performance due to their algorithmic complexity as well as inefficient implementation. A second major challenge is the robust calibration of 3D cameras under different light conditions, mobility and other environmental changes that happen in TI sites.
The current solutions in the 3D real-time reconstruction space currently rely on an image-based trinocular stereo algorithm7 to calculate the depth information for each pixel. The output from one 3D camera cluster corresponds to one stream of 3D frames containing both color and depth information. All cameras are synchronized via hotwires to take shots simultaneously. They are calibrated with a self-calibration algorithm by Svoboda et al.8 Figure 3 shows some of the challenges and solutions/results in the 3D vision space.
One of the major challenges in TI networking is the bandwidth management and appropriate semantic networking protocols to provide Quality of Service (QoS) on top of the best effort heterogeneous Gigabit Ethernet/Internet 2 networks.
The current networking solution relies on our semantic-aware, session data protocol within the service middleware layer that runs on top of the TCP/IP protocol stack. It executes 3D video stream compression as well as end-to-end multi-tier streaming using bandwidth estimation, context-aware bandwidth allocation and rate adaptation. The service middleware layer provides the control plane functionalities for the bandwidth management, such as the multi-stream coordination and adaptation algorithms, depending on the end-to-end Internet 2 bandwidth availability and user view selections. Furthermore, the control functions include display setup with selection of scene views and screen layouts at the viewer site. Figure 4 shows some of the networking challenges and solutions/results in the semantic networking space.
