Question about rendering optimization in the released app

[daminoworld]

Hi there!
First of all, thank you so much for this amazing library!
I've been learning about 3D Gaussian Splatting by using MetalSplatter
with PLY files converted via ml-sharp, and this open-source project
has been incredibly helpful for my studies.

I have a question that I hope you don't mind me asking:
When I load my PLY files (~1.18M splats) using an app I built based on
this library, the Vision Pro fan spins up and there's noticeable
flickering during rendering, especially when moving my head.

However, when I load the exact same PLY file in your released
MetalSplatter app from the App Store, it renders beautifully -
no fan noise, no flickering, and smooth interaction with gestures.

I've tried various approaches to fix this (adjusting sortTimeout,
deviceAnchor handling, etc.) but haven't been able to match the
performance of your released app.

I was wondering if you could share some insight on how you optimized
the rendering in the released version? The visual quality seems the same,
so I'm curious if you're using a different format (like SPZ perhaps?)
or some other technique.

Any guidance would be greatly appreciated! 🙏
Thank you again for your work on this project!

[scier]

Glad the library's been helpful thus far! The App Store app is using the same library and loading path with no additional optimizations. So, the performance difference may be straightforward to fix:

• Are you building your app in Release (vs. Debug) mode?
• Are you running the app without the debugger attached (i.e. straight from the Home Screen after hitting Stop in Xcode)?
• Do you have highQualityDepth enabled? (This actually slows down actual frame rendering by ~~5% depending on scene, but makes the Vision Pro experience much smoother)

If the answer to each is 'yes', then the issue must be something subtler. How is the performance with the sample app (running under the same Release, no-debugger conditions)?

[daminoworld]

Thank you so much for your quick and helpful response! 🙏

I tested in Release mode without the debugger attached, and I'm happy to
report that the periodic flickering issue has been resolved. thank you for that!

However, I'm still experiencing an issue where the splat model shakes
significantly when moving the camera/head, and sometimes the entire scene
disappears momentarily before re-rendering.

I tested the same PLY file (~1.1M splats) in the Released app, and it only
showed minor wobble during head movement — nothing like the complete
scene disappearance I'm seeing in my app.

I'm currently investigating this on my end, but if you have any insights
or suggestions about what might cause this behavior, I would really
appreciate it!

[scier]

Does the "shaking" look like the foreground is bouncing rapidly back and forth when you head moves (and more with more movement)? That sounds like the effect that led me to develop the "high quality depth", so I'd again suggest double-checking that you're leaving that enabled. The root cause of that flickering was actually the Vision Pro's frame extrapolation going wonky because it was using the (poor) depth produced by default when one renders splats.

But the scene disappearance sounds like something deeper. Perhaps it's actually dropping too many frames and the Vision Pro is giving up. I'd first make sure that the sample app isn't doing the same thing (if it is, then it's not your app). If it isn't, then Instruments is your best bet to see what's going on (maybe the Metal System Trace instrument?)

One other possibility: the MetalSplatter app is using a slightly older version of the MetalSplatter library (I haven't submitted it with the latest library, but will soon) and isn't using the 26 SDK; so the problem could actually be an issue with the latest MetalSplatter library, or and issue when building with the iOS 26 SDK. If you aren't able to make any headway, we could investigate this via a TestFlight of a new MetalSplatter app build once I have one available.

[daminoworld]

Hi @scier — sorry for the late reply; I’ve been tied up with other work 🥲

Thanks again for the detailed guidance. I’ve been testing and profiling more, and here’s a consolidated update.

Environment / setup

• Vision Pro (M2), visionOS 26.2
• Release build, no debugger attached
• highQualityDepth = true
• Command buffer per frame = 1 (confirmed in Metal System Trace)
• Using queryDrawable() single drawable (aligned with SampleApp)

Behavior

• “Shaking” still happens when moving the head. Throttling sort reduces the frequency, but large head movement still causes the scene to disappear briefly and then re-render.
• The effect is worse on larger PLYs (1.4M splats vs 1.15M).

Metal System Trace summary

• Display VSYNC is stable at 90fps (11.11ms).
• Average FPS ≈ 43; median frame time ≈ 33ms (most frames in 33–50ms bucket → 30fps fallback).
• Command Buffer submission rate ≈ 30.4/s.
• CPU encoder time is tiny (~0.058ms avg), so CPU encoding is not the bottleneck.
• GPU driver-level command buffer time frequently 55–91ms (sometimes >100ms).
• A few severe stalls (e.g. an 8.2s gap, hundreds of missed VSYNCs).
• Thermal state transitions to “Fair” after ~33s.

Interpretation

• The bottleneck appears to be GPU work (sort + splat rendering), not CPU or draw submission.
• Large head motion + long sort time often causes frame drops, and that’s likely when the scene “disappears”.

Changes tried

• Pre-allocating index buffers and triangle index buffer.
• Switching PLY load to SplatIO + SplatChunk to avoid repeated buffer reallocations.
• Sort throttling (position/angle threshold + minimum interval + avoid queuing while a sort is running).
• World-locked model transform (fixed to initial device anchor).

Open questions

Could the released MetalSplatter app (older library / pre‑26 SDK) be avoiding some GPU cost in the sort/render path?
Is there any known change in the latest MetalSplatter or iOS/visionOS 26 SDK that increases GPU cost or sorting overhead?
In the released app, when do you actually trigger a re‑sort? Is it every frame, or only after certain camera deltas/intervals? I see the flicker mostly when a re‑sort kicks in, so understanding that trigger would really help.
Thanks again for all your help!