[Vulkan] Reduce descriptor set churn - batch/reuse descriptor sets

[goniz]

Problem

dispatch_with_spec() allocates/reuses a descriptor set, writes descriptors with vkUpdateDescriptorSets, binds it, dispatches, then defers reclamation by epoch. This high-frequency driver work hurts small kernels, fused graphs, and attention-heavy workloads.

Reference

ggml-vulkan is much more aggressive about bulk descriptor-set handling and queue-side reuse patterns.

Tasks

• Preallocate descriptor sets per layout in large batches
• Use reset/recycle semantics rather than fine-grained free behavior
• Consider descriptor buffer / push-descriptor paths on capable drivers

Related

See Tier 1 item 4 in performance analysis report.