Better watch what you’re looking at, AI can reconstruct it in 3D


In the project “Seeing the World through Your Eyes,” researchers at the University of Maryland, College Park, show that the reflections of the human eye can be used to reconstruct 3D scenes. This, they say, is an “underappreciated source of information about what the world around us looks like”.

The paper uses a NeRF-based method to reconstruct a 3D scene from a reflection in the eye. This sounds straightforward, but is actually a complicated process that involves several factors, including accurately determining the direction of gaze and distinguishing reflections from patterns in the iris, the researchers say.

Links sieht man eine Iris mit Spiegelung, rechts davon eine 3D-Rekonstruktion dieser Spiegelung.
In the “Seeing the World through Your Eyes” project, researchers show that the reflections of the human eye can be used to reconstruct 3D scenes. | Image: Alzayer, Zhang et al.

Calculating the outside world through the cornea

To calculate the outside world, the team uses the geometry of the cornea, which is fairly uniform in healthy adults. Based on the size of the cornea in an image, they can estimate the position and orientation of the eye.

An important aspect is optimizing the detection of the cornea’s position, which is used to refine the initial position estimate for each image. This technique has proven to be critical to the robustness of the method.


The structure of the reconstruction method. | Image: Alzayer, Zhang et al.

To separate the reflection of the outside world from the iris pattern, the team used a modified training framework from Nerfstudio. They had NeRF learn both the reflected 3D scene and the iris pattern, and simultaneously trained the system to separate the two elements.

For iris detection, the system was given the general radially symmetric shape as a default and the assumption that the reflection changes from different perspectives but the texture of the iris remains the same.

Reconstructing a 3D scene from the reflection in the eye of a portrait photo. | Video: Alzayer, Zhang et al.

Bild im Fotolabor, Mann schaut in Kamera, auf dem Tisch vor ihm liegen zwei Plüschtiere, die Lampen leuchten die Szene hell aus.
The photo setup under laboratory conditions. | Image: Alzayer, Zhang et al.

Reconstruction of a 3D scene from the reflection in the eye of a portrait photograph. | Video: Alzayer, Zhang et al.

3D nerf with only one camera

While typical NeRF reconstructions require multiple camera views of the object being reconstructed, the researchers simply moved the person through the camera’s field of view. The reflections in the eye, which change with even minimal movement, provided the different perspectives needed to reconstruct the 3D scene, even though only one camera saw the person.


Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top