Facebook researchers Reality Labs have today released a new work showing a prototype headset with external screens to represent the user’s eyes to other people outside the headphones. The goal is to allow eye contact between the headset user and other people in order to make it less uncomfortable while wearing headphones and communicating with someone in the same room.
One of my favorite things to do an Oculus Quest demonstration to someone for the first time is to put on the headphones, activate their “step view” (which allows me to see the world outside of the headphones) and then go up and shake their hand to clearly reveal that I can see them. Since Quest cameras are on all four corners of the visor, it’s not easy to imagine that there would be any way for the user to see “through” the headphones, so the result on the outside looks a bit magical. . Then I put the headphones on the person and let them see what they could see from inside.
But this fun little demonstration also reveals a problem. While it is easy for the person on the headset to see people outside the headset, it is not clear to people outside the headset when the person looking at the headset instead of looking at a completely different virtual world).
Eye contact is clearly a huge factor in face-to-face communication; it helps us assess whether someone is paying attention to the conversation, how they are feeling, and even if they have something to say, want to change the subject, or leave the conversation altogether. Trying to talk to someone who can’t see their eyes is awkward and uncomfortable, especially because it robs us of our ingrained ability to detect such intentions.
But as virtual reality headsets become thinner and more comfortable (and it becomes easier to use passthrough to have a conversation with someone nearby than to take off the headphones completely), this will become a growing problem.
Facebook Reality Labs researchers have found a high-tech solution to the problem. Using light field screens mounted on the outside of a VR headset, the prototype system called “reverse passthrough” aims to show a representation of the user’s eyes with precision and depth of direction.
In an article published this week in SIGGRAPH 2021, Facebook researchers Reality Labs, Nathan Matsuda, Joel Hegland and Douglas Lanman, detailed the system. While to outside observers it looks like the headset is very thick but transparent enough to see your eyes, the apparent depth is an illusion created by a light field screen on the outside of the headset.
If it were a typical screen, the user’s eyes would appear to float far away from their face, which might be a more awkward image than not being able to see them. Below, researcher Nathan Matsuda shows the system without eyes (left), with eyes but without depth (middle) and with eyes i depth (right).
The light field screen (in this case a screen that uses an array of microlenses), allows multiple observers to see the correct depth signals, regardless of the angle at which they are located.
What observers see is not a real image of the user’s eyes. In contrast, eye tracking data is applied to a 3D model of the user’s face, which means that this technique would be limited by the realism of the model and the ease of acquisition for each individual.
Of course, Facebook has also been doing some really impressive work on this front with its Avatar Codec project. The researchers mocked an example of an Avatar Codec that was used for the reverse step function (above), which looks even better, but the resolution is still clearly a limiting factor, which the researchers believe will be overcome. in time.
Michael Abrash, chief scientist at Facebook Reality Labs, admits he didn’t think there was much merit to the idea of the reverse step until researchers demonstrated the concept.
“My first reaction was that it was kind of a ridiculous idea, at most a novelty,” Abrash said in a post about the play. “But I’m not telling researchers what to do, because you don’t get innovation without the freedom to try new things, and that’s good, because now it’s clear it’s a unique idea with a genuine promise.”
– – – – –
It might seem like a lot of work and extra hardware to solve a problem that is not really a problem if you just wear RA headphones. After all, most AR headphones are built with transparent optics from the outset, and being able to see the user’s eyes is an important advantage when interacting with other people while wearing the device. .
But even then, RA headphones can suffer from a “glare” that obstructs the eye’s vision from the outside, sometimes severely, depending on the optics and angle of the viewfinder.
AR headphones also have other limitations that are not a problem in virtual reality headphones, such as a limited field of view and lack of full opacity control. Depending on the use case, a future and lightweight VR headset with a very convincing reverse step system could be preferred over an AR headset with transparent optics.