OpenXR SteamVR Passthrough API Layer

OpenXR API layer for providing camera passthrough (Mixed Reality) support to SteamVR.

This layer allows OpenXR applications that use the OpenXR passthrough feature to enable it when using the SteamVR runtime. It also allows forcing several different passthrough modes onto applications that themselves do not support it.

The SteamVR runtime itself does not currently support any OpenXR passthrough features, but provides access to the camera video feeds and projection data through the proprietary OpenVR interface. This layer acts as a compositor in-between the application and runtime, retrieves the passthrough data from either OpenVR or a USB camera, and renders it on the frames submitted by the application before passing them on to the runtime.

Please report any issues! Any comments and suggestions are also appreciated.

DISCLAIMER

This is an experimental release. Please be careful when using passthrough.

This software is distributed as-is, without any warranties or conditions of any kind. Use at your own risk!

Using the 3D stereo mode may induce heavy flickering on the display. Exercise caution if you are photosensitive.

Hardware/software support

See Wiki Compatibility Page

Features

• Supports both application selectable environment blend modes in the OpenXR core specification: Alpha Blended and Additive.
• Configuration menu available in the SteamVR dashboard.
• User adjustable color parameters and opacity.
• Override mode for applying passthrough to applications that do not support it. The passthrough view can be blended either additively, using chroma keying, using alpha, or using depth.
• The floor projection height can be shifted up to get correct projection on an horizontal surface such as a desk.
• Supports 3D stereo reconstruction to estimate projection depth, using OpenCV. Includes support for Weighted Least Squares disparity filtering, and Fast Bilateral Solver filtering.
• Supports custom fisheye lens rectification instead of using the OpenVR pre-rectified output.
• Supports compositing the passthrough based on scene depth, for applications that supply depth buffers.
• Basic support for USB camera input. This can be used alone or in conjunction with the depth provided by a stereoscopic HMD camera. The camera can either be attached to a tracked SteamVR device, or be set up in a static position. Manual calibration is required.
• Supports several OpenXR vendor extensions, including partial support for XR_FB_passthrough.
• Basic temporal filtering available for both the camera images and stereo reconstruction depth.

Limitations

• Only the SteamVR runtime is supported.
• Only HMD cameras that provide the passthrough camera feed to SteamVR are supported. If the SteamVR Room View does not work, the HMD camera will not be accessible.
• OpenVR applications are not supported.
• The depth reconstruction from the 3D Room View is not supported. It is not currently accessible to developers. A custom depth reconstruction is used instead.
• The passthrough view has higher latency than the SteamVR compositor.
• The Vulkan renderer interop does not support the the old XR_KHR_vulkan_enable extension.
• Depth blending requires the application to submit depth buffers using the XR_KHR_composition_layer_depth extension.
• USB webcam frames can not be accurately timed. This will cause hitching in the image, especially if the frame rate jitters.
• Applications using the XR_FB_passthrough extension may or may not work.

Installation

1. Download and install the Visual Studio C++ Redistributable (64 bit)
2. Download the API layer release from the GitHub Releases page, and extract the files to the location you want to keep them in.
3. Run the passthrough-setup.exe utility, and select the Install option to install the API layer. Note that if you want to move the files, you will need to run the utility again.
4. If you want to disable or uninstall the API layer, run the passthrough-setup.exe utility and select the Uninstall option.

Usage

Starting an OpenXR application will automatically start the API layer. If the application natively supports passthrough, the API layer will by default notify the application that additional environment blend modes and vendor features are available. If the application does not support any OpenXR passthrough features, it is still possible to enable limited passthrough modes (see below).

While an application is running, the SteamVR dashboard will have an additional button in the bottom left, pressing it will open the settings overlay.

The options under the OpenXR Core allow setting what passthrough modes are available for the application to use, as well as what mode it should prefer. Some applications may automatically switch to the preferred mode even though they don't support passthrough.

The options under Overrides allow forcing the passthrough mode regardless of whether the application has support for passthrough. Note that the Alpha Blend mode will show nothing unless the application submits alpha channels to the compositor.

The Additive mode will blend the passthrough on top of the view.

The Opaque mode will replace the application view with the passthrough.

The Masked mode allows setting a chroma key color that gets replaced with the passthrough view, as well as range for how similar colors get replaced, and a smoothness of the transition.

The settings can also be edited from %APPDATA%\OpenXR SteamVR Passthrough\config.ini

See the project Wiki for more information.

Building from source

The following are required:

• Visual Studio 2022
• The MSVC build tools, and the Windows 10 SDK (installed via the Visual Studio Installer as "Desktop development with C++").
• Python 3 interpreter (installed via the Visual Studio Installer or externally available in your PATH).
• OpenXR SDK Source (Included as Git submodule)
• OpenXR SDK (Included as Git submodule)
• OpenVR (Included as Git submodule, the project is setup for static linking by default - requires custom source build)
• LodePNG (Included as Git submodule)
• SimpleINI (Included as Git submodule)
• Dear ImGui (Included as Git submodule)
• Vulkan SDK (Uses the VULKAN_SDK environment variable)
• OpenCV 4.10.0 (The project is setup for static linking by default - requires custom source build)
• OpenCV-Contrib (The ximgproc module needs to be built along with OpenCV for WLS and FBS filtering support.)

Notes

Based on OpenXR-Layer-Template