Immersive Experience (IE)

Immersive Experience (IE) and related terms like Extended Reality (XR) – an umbrella designation that encompasses Virtual, Augmented and Mixed Reality (AR/VR/MR) – or the concept of the Metaverse, have all been a fixed feature of IT analysts’ top tech trend rankings for the past few years (Metaverse came in at number nine in Gartner’s Top 10 Strategic Technology Trends for 2023). These disruptive technologies have the potential to change the way in which people perceive and interact with the digital world, and even the real world, when augmented and immersive experiences or devices are available. Since 2018, Φ-lab has invested in IE, both through projects and in-house activities, and hosts an ever-growing set of VR installations, some of which are permanently available to visitors in Φ-lab’s Archimedes mini-theatre.

These installations address two main goals, that of advanced EO visualisation applications and that of outreach, not only for Φ-lab but also for EOP and ESA in general. Many of these installations are frequently shipped to and used to support events in ESRIN and around Europe where Φ-lab might be present (Living Planet Symposium, Φ-Week, European Researcher’s Night, EGU, SLUSH, Web Summit, Big Data from Space, WeMakeFuture, etc.).

Some of the main Φ-lab IE initiatives are as follows:

• The NORCE GeoViz system, which allows the user to navigate and explore a 3D globe of the Earth as well as fly to other planets/satellites (e.g. the Moon). ESA EO promotion and outreach materials, consisting of processed EO satellite imagery as well as infographics and videos can be explored via this virtual environment. Users can choose which locations to explore and, for each of those locations, which EO images/products to examine (typical products are Sentinel-2 band combinations, Sentinel-3 scenes and Sentinel-1 derived products such as interferograms, coherence and intensity). GeoViz is also able to display 3D models (e.g. of spacecraft, buildings), point clouds (e.g. of cities, sites, inSAR showing soil subsidence) and laser scans (e.g. of plantations). It is also possible to visualise hyperspectral data: hyperspectral datasets can be loaded and users can browse through 200 bands or more in a matter of seconds using convenient VR controller buttons
• The Digital Twin ESRIN prototype, which allows navigating two different 3D models of the ESRIN site, of different provenances, but both generated from drone imagery through the use of photogrammetry techniques. The model is enriched with data from two IoT-enabled in-situ data sources, one measuring air-quality (another in-house Φ-lab project) and one measuring vegetation health (commercial device normally used in industrial agriculture). The data is retrieved in real-time from these IoT devices and displayed to the VR user
• The Digital Twin Φ-lab prototype, which allows the user to explore a very high-quality representation of the inside of the Φ-lab, generated thanks to internal 3D scanning techniques and further post-processing to enhance the visual quality of the experience. A promotional video containing interviews with Φ-Explore Fellows describing their research as well as InCubed staff describing their activities, can be played inside the VR application
• The Φ-lab in the Metaverse prototype, a customisation of BIT’s PrometheusXR system, allowing the exploration of a futuristic three-storey building and the interaction with Φ-lab content loaded around its several locations. The virtual facilities include a welcome reception (with assistant avatars), a gallery (general purpose but in this case used for showcasing Φ-lab Explore office fellow research in poster-like fashion, with links to online resources hosted in Φ-lab’s AI4EO Knowledge Management Tool), a small auditorium with a stage and large screens, and finally a 1-1 private meeting room. Large screens all throughout the experience display InCubed promotional videos with spatial audio. This experience is the only one in the available suite that is multi-player. Two or more users, each with their own headset, can connect simultaneously to a server and share the experience