How VR Eye Tracking Works

What eye tracking is

VR eye tracking is the subsystem inside a headset that measures where each of your eyes is pointing. It works by illuminating each eye with infrared light and imaging the result with small cameras aimed inward at your face, then solving for gaze direction from the geometry of the pupil and the light reflecting off the cornea. Headsets use that gaze for two jobs: as an input signal, so you can select something just by looking at it, and as a rendering hint, so the GPU can spend its budget where you are actually looking instead of across the whole frame.

The second job is the one that matters most to anyone working in graphics. Human vision only resolves fine detail in the fovea, a small central region of the retina covering a few degrees of view, and everything outside it is low resolution by design. A headset can lean on that asymmetry, but only if it knows where the fovea is pointing at any given moment.

How a headset measures your gaze

Eye tracking is a light and camera problem before it is a machine learning problem. Near infrared LEDs mounted around the lenses illuminate each eye, and small cameras, usually one or two per eye, capture images of it. Each frame contains two features that matter: the dark pupil in the middle, and a bright spot where the illumination bounces off the cornea, which the industry calls the glint or the corneal reflection. The vector between them shifts as your eye rotates even when your head is still, so tracking the pupil center against the corneal reflection is what converts a flat camera image into a direction. That method is called pupil center corneal reflection, or PCCR, and it is how the overwhelming majority of video based eye trackers work.

Two details set the quality. The first is how the eye is lit. If the light source sits on the optical axis of the camera, the retina retroreflects it and the pupil comes back bright, the same effect as red eye in a flash photo. If the source is offset, the pupil stays dark. Both are used in practice, and the choice trades contrast against robustness across skin tones and lighting conditions. The second is the processing model: software fits a 3D model of the eye to the image and locates both the pupil and the corneal reflection inside it. That model is why a per user calibration step is nearly universal. You look at a sequence of targets, and the system learns your eye geometry. Tobii, whose trackers Sony used for PS VR2, describes this same pipeline for its own hardware.

Speed matters too. Research grade trackers sample from 30 Hz up to about 1200 Hz, while headset implementations run their eye cameras synchronised to the display frame rate instead. Precision is a separate axis from speed: inexpensive trackers land around 1 degree of visual angle, and high end units reach roughly 0.1 degree.

The cameras look inward at your eyes rather than outward at the room, but it is the same sensor philosophy either way. How passthrough works covers the outward facing half of that suite, where the headset rebuilds the world from cameras instead of reading your gaze from them.

From a camera image to a gaze vector

Raw eye images never leave the system. What an app receives is a pose: an origin, a direction, and in Meta’s implementation a confidence value for each eye. Two OpenXR extensions cover the use cases. XR_EXT_eye_gaze_interaction is the vendor neutral one, exposing a combined gaze pose, and it is what Varjo, HTC and others implement. XR_FB_eye_tracking_social is Meta’s addition for social applications, returning gaze direction and position per eye plus that confidence float. On Meta hardware the API also requires an explicit eye tracking permission that the user grants, and a headset setting lets them switch tracking off entirely.

Timing is part of the contract. Callers request gaze at the predicted display time rather than the current instant, so the runtime has to model where the eye is heading, not only where it has been.

Why eye tracking matters: foveated rendering

Foveated rendering is the reason eye tracking went from a research curiosity to a shipping feature. Fixed foveated rendering already exploits the fovea: it renders the center of the lens at full resolution and the edges at less, with no tracking at all, and it has been available on Meta headsets since the Oculus Go era. Eye tracked foveated rendering adds the tracking so the sharp region moves with your gaze, which allows a more aggressive foveation map at the same perceived image quality.

The savings are concrete. At foveation level 1 the periphery is rendered with 4x fewer pixels, and at level 3 mostly with 16x fewer. In Meta’s own test application, fixed foveation saved between 26 and 36 percent of GPU time at default resolution, while eye tracked foveation saved between 33 and 45 percent. At 1.5x the default resolution the ranges rose to 34 to 43 percent and 36 to 52 percent respectively. Sony claims a larger gap on PS VR2, roughly 60 percent for fixed foveation and around 72 percent for the eye tracked version. The often quoted theoretical ceiling of about 20x assumes perfect tracking with zero latency, which no shipping headset has.

Meta’s account of its own pipeline shows why latency is the hard part. The eye cameras and their processing run at the display frame rate and are synchronised to the renderer, and the company offsets the eye camera capture time to shrink the gap between a fresh gaze sample and the updated foveation map. When that timing slips, the full resolution patch lands where you were looking a frame or two ago, and the artifact shows up as shimmering or swimming near the edges of the sharp region.

After early generations treated it as an optimization, eye tracking is now a defining feature of new hardware. Valve’s Steam Frame runs its eye tracking whenever you stream PC VR from a desktop, using it to encode the video stream at higher resolution where you are looking, an approach the company calls foveated streaming. Games are already tuning for that: No Man’s Sky’s Cosmos update shipped eye tracked dynamic foveated rendering on SteamVR ahead of the hardware.

Which headsets have eye tracking

• Meta Quest Pro: Meta’s first headset with eye tracking, and the one its Unity documentation still names for eye tracked foveated rendering. In Unity that path requires Vulkan with Multiview stereo rendering, and it stays off if the user declines the eye tracking permission.
• Meta Quest 3 and Quest 3S: no eye tracking. Quest 3 pairs two RGB passthrough cameras with four tracking cameras and a depth projector, which is a passthrough play rather than a gaze one. Quest 3 vs Steam Frame breaks down what each headset’s sensor suite buys it.
• PlayStation VR2: one infrared camera per eye dedicated to gaze, alongside four embedded cameras for headset and controller tracking, with Tobii providing the eye tracking technology.
• Apple Vision Pro: four eye tracking cameras, plus a TrueDepth camera and a LiDAR scanner. Apple describes the mechanism as an interconnected system of LEDs and infrared cameras projecting invisible light patterns onto each eye.
• Valve Steam Frame: built in eye tracking with infrared illuminators, used to drive foveated streaming over a wireless link rather than interface control.
• Others: Varjo headsets, the HTC Vive Pro Eye and Focus Vision, Pimax Crystal and Bigscreen Beyond 2e all expose gaze through the same cross vendor OpenXR extension.

The catch

Four things bite in practice.

Latency. A gaze sample is always slightly stale, and foveated rendering spends its savings on a prediction. The gap on Quest Pro’s first generation pipeline is on the order of 50 milliseconds end to end, which is why the realistic gain is a slice rather than an order of magnitude. Newer pipelines tighten it by syncing capture to the frame.

Calibration and fit. Tracking depends on a model of your own eye geometry, so it needs an initial calibration and a headset that sits the same way every time. Glasses, eyelashes and a shifted fit all degrade it, and changes in pupil size can drag the measured pupil center away from its true position, an error researchers call the pupil size artifact.

Privacy. This is where platforms diverge sharply, and it feeds straight into what developers can build. Apple keeps eye input on device, does not share where you are looking with apps or websites or Apple itself, and only tells an app what you selected. Meta takes the opposite route, exposing gaze to apps behind a permission prompt, with a system setting that lets the user turn tracking off. Where tracking is absent or disabled, apps fall back to fixed foveation.

Gaze is a pointer, not a click. Looking at something and acting on it are separate events, so interfaces pair gaze with a confirmation gesture. Vision Pro makes a look plus a finger tap the core interaction, and gaze plus a hand gesture is the same division of labour behind Meta’s hands-first VR Glasses, where the eyes pick the target and the hands commit.

Sources

• Tobii: How do eye trackers work?
• Wikipedia: Eye tracking
• Wikipedia: Foveated rendering
• Meta Horizon OS Developers: Eye Tracked Foveated Rendering
• Meta Horizon OS Developers: Save GPU with eye tracked foveated rendering
• UploadVR: Here’s The Exact Performance Benefit Of Foveated Rendering On Quest Pro
• UploadVR: Steam Frame Hands-On Impressions
• PlayStation.Blog: PlayStation VR2, the ultimate FAQ
• Apple: Apple Vision Pro technical specifications
• Apple: Eyes, Hands & Privacy
• Meta: Eye Tracking in Movement SDK for OpenXR
• Varjo: Eye tracking with Unreal Engine 5
• iMotions: What is eye tracking and how does it work?