ExplorXR
passthroughmixed-realitycamerasvr-hardwarequest-3steam-frameapple-vision-procomputer-vision

How Passthrough Works in VR Headsets

By Toni Tan · · 11 min read

What passthrough is

Passthrough is the live view of your physical surroundings that an XR headset draws on its own displays, built from outward facing cameras instead of transparent optics. The headset captures the room, corrects the image for the fact that its cameras are not where your eyes are, and hands the result to the display compositor, which blends it with whatever the running app has rendered. It is the layer that makes mixed reality possible: virtual objects can sit on your real desk, hide behind your real couch, or be occluded by your real hand.

There are two ways to see the world through a headset, and they get confused constantly.

Video see-through is passthrough. The headset is opaque, cameras capture the world, and you look at screens. Quest 3, Quest 3S, Steam Frame, and Apple Vision Pro all work this way.

Optical see-through is what AR glasses do. You look through a transparent waveguide and the display projects light into it. Snap’s Specs use a see-through waveguide with a 51 degree field of view and quote 7 milliseconds of motion to photon latency. Nothing has to be reconstructed, so there is no reprojection error and no camera pipeline, but the display cannot fully block the world behind it, which is why solid, opaque virtual objects are hard on that hardware.

The pipeline, stage by stage

1. Capture. Two or more wide angle cameras facing away from your face grab the world. Quest 3 and Quest 3S use two forward facing RGB cameras. Apple Vision Pro lists two high resolution main cameras alongside six world facing tracking cameras, four eye tracking cameras, and a LiDAR scanner. Steam Frame’s built in cameras are monochrome. See also Steam Frame color passthrough.

2. Undistort and rectify. Wide angle lenses bend straight lines, so each frame is flattened into a rectilinear image and the two cameras are matched to a common geometry. Without this step, straight edges in your room would bow.

3. Estimate depth. This is the step that decides whether passthrough feels solid or nauseating. Because the cameras sit a few centimeters in front of your eyes, a direct camera feed would make the world appear displaced from where your hands are. Correcting that requires knowing how far away everything in view is. Quest 2 and Quest Pro estimated depth from computer vision, and Quest Pro fused several black and white cameras with color from a single RGB camera, which meant its color image was monoscopic and carried visible color fringing. Quest 3 replaced that with two RGB cameras for true stereo color plus a depth projector. Quest 3S keeps the color cameras but drops the projector, relying on two infrared flood LEDs instead, which is why it tracks better in the dark but has weaker room meshing.

4. Reproject to the eye. With a depth map in hand, each camera image is warped to the viewpoint your eye actually has. A 2023 Meta Reality Labs paper presented at SIGGRAPH states the problem plainly: even with cameras directly in front of the eyes there is still an axial offset from the thickness of the headset, and streaming those images to the user produces visual displacement. The standard fix is depth estimation plus reprojection, and that fix inherits every depth error. Errors concentrate at occlusion boundaries and on near field objects, which is exactly where Quest 2 and Quest Pro passthrough used to smear and warp.

5. Composite. Apps never receive the raw camera feed. Meta’s developer documentation describes passthrough as rendered by a dedicated service into its own layer: an app creates a passthrough layer, and the XR compositor replaces that layer with the real camera rendition. Apps can style it, with hooks for colorizing the feed, highlighting edges, adjusting contrast, and posterizing the grayscale values, but they cannot read its pixels.

6. Display. Late stage reprojection re-warps the finished frame for the head pose measured moments before scanout. This is what keeps the virtual layer glued to the real layer while your head moves, and it is the same machinery that keeps rendered VR frames stable.

What the numbers mean

Vendor figures for passthrough, per eye:

  • Quest 2: grayscale, 0.3 megapixels, 4 PPD.
  • Quest Pro: colorized grayscale, 1.3 megapixels.
  • Steam Frame: grayscale with a configurable tint, 1.3 megapixels.
  • Quest 3: true color, 4 megapixels, 18 PPD.
  • Steam Frame with the Arcturus Vision Camera: true color, 6.1 megapixels.
  • Apple Vision Pro: true color, 6.5 megapixels.

Megapixels are not the whole story, but they track closely with how much detail survives. Reading a phone screen, picking out a small object on a table, or decoding a QR code all depend on this number before a single line of app code runs. Meta describes Quest 3 as having three times the color passthrough pixels of Quest Pro, and the jump from Quest 2’s 4 PPD black and white feed to Quest 3’s 18 PPD color feed is the single biggest generational change in the category. Steam Frame’s built in passthrough sits between Quest 2 and Quest 3 for sharpness and has no color at all until you add the $149 Arcturus Vision Camera, a 10 gram module on the PCIe expansion port that uses dual 32 megapixel Sony RGB sensors, runs 2464x2464 per eye at up to 72Hz by default, and can drop to 1232x1232 per eye at up to 144Hz in an experimental mode. See also Quest 3 vs Steam Frame.

Latency is what decides comfort

Apple lists 12 milliseconds of photon to photon latency for Vision Pro’s R1 chip, its figure for the delay between the world in front of you and the pixels on the displays. Snap quotes 7 milliseconds of motion to photon latency for Specs. Those are vendor numbers from two different architectures and two different measurements, so treat them as directional rather than directly comparable. What both companies are chasing is the threshold where your inner ear and your eyes stop disagreeing.

For developers, the number that matters is different from the number the wearer feels. Meta’s Passthrough Camera API documentation lists 20 to 40 milliseconds of image capture latency, a 60Hz data rate, roughly 45MB of memory overhead, 1 to 2 percent GPU overhead per streamed camera, and a YUV420 internal format. UploadVR reported the earlier experimental release at up to 1280x960 per camera and 30FPS with a stated 40 to 60 millisecond latency, and noted it is not suitable for tracking fast moving objects or reading small text. That stream is a delayed, lower rate copy of what the user sees. The wearer’s own view is composited by the system inside a much tighter budget, which at 90Hz means 11.1 milliseconds for an entire frame including your app’s rendering.

What developers can and cannot access

By default, none of the raw pixels. What an app receives are derived representations built by the system: hand and body tracking, a scene mesh of the room, depth data for occlusion, and persistent spatial anchors. Meta’s mixed reality documentation frames this as the point of the platform, and it is why most mixed reality apps never touch a camera.

Since Horizon OS v74, Quest 3 and Quest 3S can expose the forward facing cameras through the Passthrough Camera API, built on Android’s Camera2 API. It requires a camera permission grant, it only works while passthrough is enabled, and it tops out at 1280x1280 per camera from Horizon OS v83 onward, up from 1280x960. Meta’s documented use cases are exactly what you would expect from a computer vision feed: custom models that recognize specific objects, apps that check whether a physical task was completed, and lighting estimation so virtual materials respond to the room. In Unity the cameras arrive through WebCamTexture, which can only deliver one camera at a time. Because the foundation is Camera2, the same approach carries over to Android XR headsets with a different permission request.

Why this matters beyond the view

Passthrough quality sets the ceiling for what mixed reality apps can attempt. An app that needs to read a label, recognize equipment, or place a virtual object flush against a real wall is limited first by how good the camera image is, and only second by how clever the software is.

It is also a compute cost. Capturing, correcting, depth estimating, and compositing every frame consumes the same silicon and thermal budget that would otherwise go into rendering, which is part of why standalone headsets are always fighting for milliseconds. That pressure is the same one driving the argument for moving heavy work off the headset entirely, whether to a PC or a companion device. Background: getting compute off the head.

Capture is the other half of the story. The same cameras that show you the world can record it. Vision Pro captures spatial photos and video, Horizon OS v78 improved the Passthrough Camera API image quality enough that UploadVR called Quest 3 a viable entry level 3D camera, and the Arcturus module turns Steam Frame into a tracked stereoscopic capture device. Those stereo frames are also the raw material for reconstruction pipelines, the same family of data behind Gaussian Splatting workflows and neural rendering, which means passthrough hardware is quietly doubling as a 3D scanning rig. Background: Gaussian splatting.

Finally, there is the social constraint. Headset cameras always run permission gated, and smart glasses signal recording with an LED. Meta plans to launch a camera free range of smart glasses at Connect 2026 in response to privacy backlash, which is a reminder that what limits this technology is not only silicon.

Limitations

  • Depth failures are visible failures. Low texture walls, mirrors, and glass are where the estimate breaks and virtual objects start floating or vanishing.
  • Near field warping. Whatever you hold close to the visor is the hardest thing to reproject correctly.
  • Low light. Infrared emitters keep head and hand tracking alive in a dark room, but they do not restore color quality to the passthrough image.
  • Camera hardware costs weight, money, and battery, which is why cheap headsets and light glasses make different tradeoffs.
  • Latency has a comfort floor. Past a certain mismatch between head motion and visual feedback, some people feel it as motion sickness.

None of that is a reason to avoid passthrough. It is a reason to know what you are looking at when a demo looks stunning on a bright afternoon and falls apart in a dim hallway.

Sources

Frequently asked questions

What is passthrough on a VR headset?
Passthrough is the live view of your physical surroundings that a headset draws on its own displays, built from outward facing cameras rather than transparent optics. The headset captures the room, corrects the image for the fact that its cameras are not where your eyes are, and hands the result to the compositor, which blends it with whatever the app has rendered. It is the layer that makes mixed reality possible.
Is Quest 3 passthrough in color?
Yes. Quest 3 uses two RGB cameras for true color stereo passthrough at 18 PPD, roughly 4 megapixels per eye, and adds a depth projector so virtual objects can be placed against real surfaces more accurately. Quest 3S is also full color at 18 PPD, but it has no depth projector and uses two infrared flood LEDs instead.
Why does passthrough look warped when I move my hands close to my face?
The cameras sit in front of your eyes, so the image they capture is not the image your eyes would see. The system has to estimate depth and reproject that image to your eye position, and near field objects are the hardest case for the depth estimate. That is why hands and objects held close to the visor are where warping and smeared edges show up first.
Can developers access the passthrough cameras on Quest 3?
Yes, with user permission. The Passthrough Camera API exposes the forward facing cameras through Android's Camera2 API on Quest 3 and Quest 3S, at up to 1280x1280 per camera, a 60Hz data rate, and 20 to 40 milliseconds of image capture latency according to Meta's developer documentation. It only works while passthrough is enabled.
Does Steam Frame have color passthrough?
Not out of the box. Steam Frame's built in passthrough is monochrome with a configurable tint at roughly 1.3 megapixels per eye, sharper than Quest 2 but blurrier than Quest 3. Color comes from the $149 Arcturus Vision Camera, which attaches to the headset's PCIe expansion port and uses dual 32 megapixel sensors.
How is passthrough different from AR glasses you see through?
Passthrough headsets are opaque and rebuild the world from cameras, which costs camera hardware, compute, and latency but lets virtual objects occlude real ones. Optical see-through glasses such as Snap's Specs use a transparent waveguide so you look at the real world directly, which is lighter and lower latency but cannot fully block out what is behind the display.

Related reading

How VR Eye Tracking Works

VR eye tracking fires infrared light at your eyes, reads the pupil and the corneal reflection, and turns that into a gaze vector. Here is the full pipeline.

XR Daily: A $150 Camera Fix for Steam Frame

Arcturus gives Steam Frame true passthrough for $150, Meta opens a $1M hands-first contest, Hologram avatars get a hands-on, and Walkabout cuts staff again.

XR Daily: Beat Saber Retrofits for Hands

Beat Saber rebuilds itself around hand tracking, Dungeons of Eternity brings Half-Life: Alyx style lighting tricks to Quest, and 177 scanned materials go free.