Post config and output status for each tested server

[kennylevinsen]

For each of the benchmarks it would be helpful if the exact configuration was included with their benchmarks.

There is for example some uncertainty as to whether all the tests were run with VRR enabled. The sway jitter might suggests this was not the case. In addition, for sway setting a positive output max_render_time will ensure that frame callbacks are emitted before any render tasks, which lowers variability.

[killown]

I agree that FIFO mode + max_render_time does what you say but Mailbox is designed to never block. Adding a "Max Render Time" turns it into a fragile FIFO, forcing back-pressure onto an architecture meant to avoid it. Metrics may improve, but frame presentation is unlikely to.

I need to implement a way to measure frame presentation so accurately that not even (mailbox) Max Render Time could lead to fake metrics results.

All tests were VRR-enabled, verified manually. I'll update them with the compositor configuration and test details.

[kennylevinsen]

So far the primary finding is that benchmark numbers and overall user experience improve a lot if we wait for the acquire point to be signaled (i.e., rendering to complete) before looking at the buffer, instead of passing the acquire point along to composition/KMS. This technically worsens latency a little bit (instead of driver just using the frame when ready it has to wake us up first), but it means we can keep rendering smoothly by ignoring cube-simulator entirely until it has something that is ready to display. If you're curious, see https://gitlab.freedesktop.org/wlroots/wlroots/-/merge_requests/5289.

This is similar to what Jay was doing, and in my test makes all the numbers and perceived user experience beween the two equivalent. Quite a big UX impact tbh.

I need to implement a way to measure frame presentation so accurately that not even (mailbox) Max Render Time could lead to fake metrics results.

Sorry, but max render time is not "fake metric results". It is expected frame scheduling behavior to improve presentation timing: the later we composite and commit, the fresher the frame and lower perceived latency. Our standard behavior is to commit it as early as possible, maximizing headroom but at the cost of freshness.

It does not turn anything into a "fragile FIFO" - it's nothing more than a delay. Jay does this automatically based on current render performance, sway lets the user pick the time offset.

Anyway, there's a lot of things that affect presentation and performance, hence why config and setup details are important. Otherwise the numbers are incomparable.

[killown]

So far the primary finding is that benchmark numbers and overall user experience improve a lot if we wait for the acquire point to be signaled (i.e., rendering to complete) before looking at the buffer, instead of passing the acquire point along to composition/KMS. This technically worsens latency a little bit (instead of driver just using the frame when ready it has to wake us up first), but it means we can keep rendering smoothly by ignoring cube-simulator entirely until it has something that is ready to display. If you're curious, see https://gitlab.freedesktop.org/wlroots/wlroots/-/merge_requests/5289.

This is similar to what Jay was doing, and in my test makes all the numbers and perceived user experience beween the two equivalent. Quite a big UX impact tbh.

I need to implement a way to measure frame presentation so accurately that not even (mailbox) Max Render Time could lead to fake metrics results.

Sorry, but max render time is not "fake metric results". It is expected frame scheduling behavior to improve presentation timing: the later we composite and commit, the fresher the frame and lower perceived latency. Our standard behavior is to commit it as early as possible, maximizing headroom but at the cost of freshness.

It does not turn anything into a "fragile FIFO" - it's nothing more than a delay. Jay does this automatically based on current render performance, sway lets the user pick the time offset.

Anyway, there's a lot of things that affect presentation and performance, hence why config and setup details are important. Otherwise the numbers are incomparable.

The cube simulator's metric results are unreliable with a max render time of 1, making it unsuitable for mailbox mode when testing wlroots compositors.

Sway

Niri

I'll test your PR soon. Regarding the mailbox: JIT and MISSED metrics remain at zero regardless the amount of cubes in mailbox mode with max render time set to 1.

[killown]

Your pr https://gitlab.freedesktop.org/wlroots/wlroots/-/merge_requests/5289 fixed the issue !!

[kennylevinsen]

I'll test your PR soon. Regarding the mailbox: JIT and MISSED metrics remain at zero regardless the amount of cubes in mailbox mode with max render time set to 1.

If that messes up the metrics it would mainly make me a bit suspicious of the measurements themselves. However, those values certainly do grow on my machine with higher cube counts, they just stay near zero at loads that match display refresh rate.

The primary difference from the client perspective is that by committing much closer to the deadline, the client might end up providing us a newer, fresher buffer just in time for commit, but the interval between presenting buffers, and the ratio of presented and discarded buffers will remain the same.

That is, unless you set max_render_time so low that we miss every single deadline, in which case you'll see your output run at half refresh rate. In my case, with light load the needed max_render_time value is 3-5ms.

[killown]

I'll test your PR soon. Regarding the mailbox: JIT and MISSED metrics remain at zero regardless the amount of cubes in mailbox mode with max render time set to 1.

If that messes up the metrics it would mainly make me a bit suspicious of the measurements themselves. However, those values certainly do grow on my machine with higher cube counts, they just stay near zero at loads that match display refresh rate.

The primary difference from the client perspective is that by committing much closer to the deadline, the client might end up providing us a newer, fresher buffer just in time for commit, but the interval between presenting buffers, and the ratio of presented and discarded buffers will remain the same.

That is, unless you set max_render_time so low that we miss every single deadline, in which case you'll see your output run at half refresh rate. In my case, with light load the needed max_render_time value is 3-5ms.

So you are right, the render time limit is likely set too low and causing the wrong metrics.

It will also take time to update the benchmarks until some compositors start using your wlroots changes.

Thanks for your reports, really helped!