Facebook Reality Labs wants to help people see your eyes while you’re in virtual reality, even if the results are between a bit unsettling and a nightmare. Earlier this week, FRL released a paper on “the reverse realization of virtual reality,” a recipe for making virtual reality headsets less physically insulating. The researchers devised a method to move your face to the front of the headphones, although they point out that it is still firmly experimental.
“Passthrough VR” refers to a feature that displays a live video channel from the headset’s cameras, allowing users to see the real world while still carrying the device. Facebook’s Oculus Quest platform, for example, shows users a step feed when they step out of the limits of their VR space. It is useful for quickly abandoning virtual reality and can also enable an augmented reality form by adding virtual objects to the camera’s power supply. But, as FRL points out, people surrounding a headset user can’t make eye contact, even if the user can see them perfectly. This is awkward if viewers are used to seeing the exposed face of their friend or co-worker.
FRL scientist Nathan Matsuda decided to change that. A blog post explains that Matsuda started in 2019, when he mounted a 3D screen on an Oculus Rift S headset. The screen showed a virtual representation of his upper face and custom-made eye tracking cameras captured wherever he went. Matsuda was staring, so his avatar’s eyes could point in the same direction. The result was basically that Matsuda was carrying a telepresence tablet that showed a copy of her own face, which is arguably so awkward, but with a more intriguingly postmodern twist.
According to the blog post, the chief scientist of the FRL, Michael Abrash, is very understandable, who did not find the idea very practical. “My first reaction was that it was kind of a ridiculous idea, at most a novelty,” he points out. “But I don’t tell researchers what to do, because you don’t get innovation without the freedom to try new things.”
Matsuda ran with the concept and, for the next two years, led a team to develop a smarter design. The team’s prototype headset, which it unveiled ahead of next week’s SIGGRAPH conference, adds a stack of lenses and cameras to a standard VR headset display. Stereo cameras capture an image of the face and eyes inside the headphones and their movement is mapped into a digital model of the face. The image is then projected onto an external light field screen. This screen creates the illusion of looking through the lenses of thick glasses and seeing a pair of eyes, even though you actually still see a real-time animated copy. If the user returns to full RV, the screen can be left blank to indicate that he is no longer involved with the outside world.
The result is a pair of octagonal glasses that would be seen at home in a Terry Gilliam movie. FRL used a simple rendering of a virtual face, but also displayed the system with its most realistic Codec avatars, as seen below.
FRL recognizes that the individual components of the system are not revolutionary. HTC already has a facial tracking add-on for its Vive Pro headphones; maps the movement to an avatar within VR, not to an outward-facing screen, but the principle is similar. This week’s article focuses on the potential of light field screens and the system’s opportunities for better in-person social interactions.
HoloLens style projection glasses theoretically leave your face much clearer than passthrough screens, although many of these glasses have darkened lenses and Path to virtual reality notes, light projected on transparent lenses can also block your eyesight. But, according to companies like Apple, experimenting with passthrough designs, new Facebook research shows that the solid screen is not necessarily a barrier to eye contact … in a way.