Hybrid Worlds: The Next Frontier in Virtual Reality

Hybrid worlds are the next level in terms of augmented reality and mixed reality. The creation of hybrid worlds is about mixing, in a credible way, the real world with the virtual world.  Mixed reality, as it exists today, is not enough for all applications, that is, the level of integration is still not enough for some applications. The goal is to add virtual objects to the real world coherently as if these objects were part of the real world.

Hybrid worlds: a matter of coherence

To speak coherently is to take into account the context: it is to create an experience taking into account information such as the where? when? and with whom? This information is often not taken into account when creating a mixed-reality application.

This coherence must exist at the visual level (geometric, lighting and context) and also at the multi-sensory level (the various senses: hearing, smell, touch, sight, taste).

Example of geometric coherence vs. coherence of illumination (non – (left) vs coherent (right) illumination

How to achieve a coherent hybrid world?

To reach the coherent hybrid world it is necessary to overcome some technological challenges:

  1. Scene understanding – realizing which objects are there, what is the correct geometry of objects, to fully integrate them.
  2. Scene registering – fine-tune the accuracy so that the objects are perfectly aligned with the real world.
  3. Consistent illumination – improving techniques to obtain consistent lighting.
  4. Multi-sensoriality – increase the exploration of multi-sensoriality at the level of augmented reality.
  5. Natural and haptic interfaces – interacting with virtual objects in the same way as with the real world, to have the same interaction, without distinguishing what is real and virtual.


Challenges to the creation of hybrid worlds

Example of information overload

We want these new hybrid worlds to be indistinguishable from interaction with the real world even though they have overlayed digital elements.

This is where CCG researchers enter the scene, to achieve natural interaction with the digital world.

To do so, one must respond to four challenges, to join digital worlds and superimpose them on real worlds.

  1. The constancy of illumination and color – given that natural objects suffer color change throughout the day, mainly exposed to natural light, it is necessary to achieve this also with digital objects.
  2. Depth perception – continue to see the 3D world but in a 2D, bidimensional plane, without visual discomfort.
  3. Fidelity in multi-sensoriality – how to bring the different sensory areas reliably, we need all the senses to interact with the environment, the more sensory systems we join, the more immersive the environment.
  4. Information overload – avoid the overuse of technologies, the information apparatus upon the person.

“Hybrid Worlds: The Next Frontier in Virtual Reality” is a presentation by Luis Magalhães, CVIG – Computer Vision Interaction and Graphics’ Scientific Coordinator, and by Carlos Silva, PIU – Perception Interaction and Usability Development Coordinator.