openxr-sys layer api relies on xrEnumerateApiLayerProperties which is not available in all runtimes

[TobiSchluter]

For background, please check here https://fredemmott.com/blog/2024/11/25/best-practices-for-openxr-api-layers.html

Loading an API layer built with openxrs fails when using the SteamVR OpenXR runtimes as well as the runtime on HTC Android headsets. This is because these runtimes don't provide xrEnumerateApiLayerProperties (the link has some info on that). The function is not actually used by openxr-sys, just stored, so this seems overly restrictive.

We work around this by hooking into GetInstanceProcAddr and providing a dummy implementation of xrEnumerateApiLayerProperties if the runtime doesn't provide it, but it would be nice to not have to do that.

[Ralith]

Usage from layer implementations isn't something I've thought much about, though I'm open to merging changes to streamline it. Can you elaborate on how you're using openxrs and where the xrEnumerateApiLayerProperties requirement arises from?

[TobiSchluter]

Thank you for the quick reply. I figured that layer usage is not all that common, so we're grateful that it works as well as it does.

I think the runtimes that don't work all have the same code underlying them, as SteamVR and HTC's code very likely share the same origins. It crashes like this

I call this code from my create_api_layer implementation like so

    let entry = match next_info.next_get_instance_proc_addr {
        Some(next_get_instance_proc_addr) => {
            match openxr::Entry::from_get_instance_proc_addr(next_get_instance_proc_addr) {
                Ok(entry) => entry,
                Err(result) => return result,
            }
        }
        None => return xr::Result::ERROR_INITIALIZATION_FAILED,
    };

Where next_info is derived from instance_info: *const xr::InstanceCreateInfo in the documented way.

The issue is that openxrs's code requires xrEnumerateApiLayerProperties even though it's never used -- and not provided by the runtimes. I think the reason for all this is that layers don't link to a loader -- and for everything that links to a loader, the Api layer properties are gathered and enumerated by the loader, not the runtime, and so the function seems fairly useless unless one is concerned with the layers themselves.

[TobiSchluter]

ps I should have said that it panics further down the call chain from my screenshot, the actual crash is in

unsafe fn get_instance_proc_addr_helper(
    get_instance_proc_addr: sys::pfn::GetInstanceProcAddr,
    instance: sys::Instance,
    name: &CStr,
) -> Result<unsafe extern "system" fn()> {
    let mut f = None;
    cvt((get_instance_proc_addr)(instance, name.as_ptr(), &mut f))?;
    Ok(f.unwrap())
}

where cvt returns an Err

[Ralith]

Thanks for the details. While this function isn't used internally, it is public and may be used by downstream application code. To cover both use cases, we could provide our own stub implementation as a fallback for when dynamically loading that function fails. That should be much simpler than wrapping GetInstanceProcAddr downstream. How's that sound?

[TobiSchluter]

Yes, that seems like the safest way of doing this, also because the dummy doesn't have to keep track of the instance -- which the downstream workaround has to do.

[Ralith]

Do we need the same mitigation for the Instance function pointer table, or just Entry?

[TobiSchluter]

I don't think so. I went back a t the code for a minute to figure out error cases that I'm missing, but didn't find any. My application works fine with the runtime, even though the function isn't there. Loading my layer with only the Entry table worked around works. But I may be missing the point of your question.

[TobiSchluter]

Sorry, my colleague who works on the mobile headsets points out to me that they also needed to fake xrEnumerateInstanceExtensionProperties in the same way. Maybe that's what you meant?

[TobiSchluter]

So we did some further experiments, and we found that the SteamVR runtime has another bug. When get_instance_proc_addr is called with a NULL instance (as it should be during startup), it throws an assertion:

Wed Jan 15 2025 16:57:07.838 - ASSERT: "sxr_xrGetInstanceProcAddr: Invalid CSxrInstance handle: 0
" at c:\buildslave\steamvr_rel_hotfix_win64\build\src\vrclient\openxr_entrypoints.cpp:907.

I'm mentioning this, because this would render the fix moot. But let me qualify that.

Why did we not notice this before? Because Unreal engine, our main platform, catches the related exception somehow and manages to continue if we also provide the workaround described further up in this issue! This means that a fix to opensxrs would still benefit all Unreal projects.

It must be a known bug in SteamVR, old enough for Unreal to take care of. We'll try with a newer version of SteamVR tomorrow, but this has become an amazing rabbit hole built out of quicksand!

[TobiSchluter]

We confirmed that recent SteamVR versions fix the assertion. That is to say that with the xrEnumerateApiLayerProperties and xrEnumerateInstanceExtensionProperties workarounds, API layers in SteamVR will work.

[Ralith]

Sorry, my colleague who works on the mobile headsets points out to me that they also needed to fake xrEnumerateInstanceExtensionProperties in the same way. Maybe that's what you meant?

My question is whether you need to create an Instance function pointer table to build a layer, and if so, whether that needs the same mitigation applied, since it also tries to load xrEnumerateInstanceExtensionProperties .

[TobiSchluter]

That's a good catch! We just stepped through the code in the debugger to be perfectly sure, and indeed, it also tries to find the function when building the Instance jump table. It is also not provided by the runtime on this second attempt but instead goes through the fallback that we installed, and openxrs crashes if we only enable the workaround for the case where Instance == nullptr, i.e. when setting up the Entry table.