Hybrid Extended reAliTy

Hybrid Extended reAliTy (HEAT) is born to pave the way for the next-generation distributed experiences by addressing major challenges to make those experiences that up to now could only be in our imagination: being realistically immersed (holo-ported) within real captured omnidirectional and navigable hyper-realistic 3D spaces, feeling their atmosphere, and sharing these experiences with others, regardless of their location. The aim of this proposal is to integrate immersive media technologies such as point cloud/holographic imaging, multi-sensorial media, Social VR in a multi-user, feedback-enabled communication system to provide the construction of compelling context-aware and embodied experiences for innovative hybrid XR applications, where remote users can experience a real captured environment through immersive VR, while in presence users can visualise and interact with the holograms of remote users integrated in the real environment through holographic rendering.The system will aim at facilitating the exploitation of agile (multi-sensory) 3D data acquisition techniques, enhancing performance while reducing technology costs. It will create a scalable communication pipeline embodying either encapsulation of different media from classical audio-video to multisensorial to holographic video or a combination of them, providing means for efficient and scalable encoding, processing, storage, (real-time) streaming and rendering. In such a way, the system will allow users to provide/exploit the features according to the acquisition/rendering system available.The project will provide well designed and fully tested scenarios in real-world environments for enhanced XR experiences: a blended learning, a modern theatre act, a music festival and an opera show. All pilot actions will ensure that GDPR and ethics are addressed for end-users (privacy and ethics by design methodology).