ggml : remove KQ mask padding

[ggerganov]

target #16148

Gauging what would it take to remove the KQ mask padding along the batch dimension (ne31). Removing this padding would simplify the graph building logic and will reduce the amount of memory that we allocate and transfer for KQ masks.

• Metal (after 46c338f)
• CUDA ?
• Vulkan (after vulkan: in flash attention, bounds check against nem1 (don't rely on GGML_KQ_MASK_PAD) #16316)

[jeffbolznv]

This will require some more changes to the Vulkan backend.

[jeffbolznv]

#16316 makes Vulkan handle this.

[slaren]

Wouldn't this cause the tensor shape to change in every evaluation, and break graph reuse and CUDA graphs?

[ggerganov]

Wouldn't this cause the tensor shape to change in every evaluation, and break graph reuse and CUDA graphs?

It shouldn't - this is the padding along the batch dimension (src[3]->ne[1]). The padding along the context dimension (src[3]->ne[0]) is relevant for having constant graph shapes. It will remain.