[metal backend] fix buffer overruns

[gcsmith]

Signature 1:
(this is likely the same root cause as rt64/rt64#218)

=================================================================
==40213==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6070001a7088 at pc 0x0001138ad90c bp 0x00017149e8d0 sp 0x00017149e080
READ of size 80 at 0x6070001a7088 thread T49
    #0 0x0001138ad908 in memcpy+0x480 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x51908)
    #1 0x00012ef4e814 in -[AGXG16XFamilyRenderContext setVertexBytes:length:atIndex:]+0x7c (AGXMetalG16X:arm64+0x726814)
    #2 0x00010f8222e0 in plume::MetalCommandList::checkForUpdatesInGraphicsState() plume_metal.cpp:3417
    #3 0x00010f82266c in plume::MetalCommandList::drawIndexedInstanced(unsigned int, unsigned int, unsigned int, int, unsigned int) plume_metal.cpp:2451
    #4 0x0001006d3b10 in recompui::RmlRenderInterface_RT64_impl::RenderGeometry(Rml::Vertex*, int, int*, int, unsigned long, Rml::Vector2<float> const&) ui_renderer.cpp:403
    ...

The solution is to alignUp() the buffer size so that pushConstants[rangeIndex].data.size() matches pushConstants[rangeIndex].size. Otherwise, setVertexBytes() reads beyond the end of the buffer.

Signature 2:

=================================================================
==40019==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x612000056590 at pc 0x000100edce20 bp 0x00016ef71510 sp 0x00016ef71508
READ of size 4 at 0x612000056590 thread T0
    #0 0x000100edce1c in plume::MetalCommandList::barriers(unsigned int, plume::RenderBufferBarrier const*, unsigned int, plume::RenderTextureBarrier const*, unsigned int) plume_metal.cpp:2405
    #1 0x000100e923d8 in plume::RenderCommandList::barriers(unsigned int, plume::RenderTextureBarrier const&) plume_render_interface.h:160
    #2 0x000100e95ca8 in plume::render(plume::CubeContext&) main.cpp:470
    #3 0x000100e9ef20 in plume::CubeExample(plume::RenderInterface*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) main.cpp:559
    #4 0x000100e9f988 in main main.cpp:612

There is an invalid cast from an instance of MetalDrawable (fetched via MetalSwapChain::getTexture()) to MetalTexture when passed to MetalCommandList::barriers().

struct MetalDrawable : ExtendedRenderTexture { ... }
struct MetalTexture : ExtendedRenderTexture { ... }
...

// textureBarrier.texture is MetalDrawable, which is not subclassed by MetalTexture
MetalTexture *interfaceTexture = static_cast<MetalTexture *>(textureBarrier.texture);

They share the same parent class ExtendedRenderTexture, but MetalTexture is not a MetalDrawable so when we access the member interfaceTexture->barrierStages we are again reading an OOB address when interfaceTexture == &drawables[2] (where drawables has 3 items), and for indices 0 and 1 we alias the wrong data.

My change fixes the invalid access by casting to the parent class (and moving the barrierStages state up to ExtendedRenderTexture for that to work), but it wasn't clear to me what the original intent was (e.g. was MetalTexture supposed to extend MetalDrawable?). This fixes the bug and the MM recomp now runs without crashing, but I'm not sure if this fix is architecturally correct.

[dcvz]

This is the right approach, I think it was just a failure on our side. ExtendedRenderTexture is any renderable texture which may be a drawable or not.

[dcvz]

Thanks for looking into this!

[squidbus]

LGTM other than the comment already left.

[gcsmith]

Committed the suggested change. Thanks for reviewing.

[dcvz]

Thanks again @gcsmith !