Have there been efforts regarding a cblas_sgemm_strided_batched API?

[markusheimerl]

See https://github.com/markusheimerl/attention/blob/1316628d97b4a626725607c0a9973f5206e7277b/attention.c#L208

I would like to ask whether there have been any efforts to introduce a cblas_sgemm_strided_batched API in OpenBLAS.

A strided batched API would make it possible to parallelize along the batch dimension in scenarios where the core computation is fundamentally two-dimensional, and the batch dimension effectively introduces a third axis.

At the moment, I am working around this by placing an OpenMP parallel for loop around the batch dimension. However, this requires setting OpenBLAS threads to 1 and OpenMP threads to 4 (instead of simply setting OpenBLAS threads to 4), which feels like an unnecessary and somewhat hacky configuration.

An official API for strided batched GEMM would simplify this workflow significantly and provide a cleaner, more robust solution.

[martin-frbg]

No, this has not been implemented so far - and neither has strided GEMM per se (#2289).
You can find the list of available BLAS extensions in https://github.com/OpenMathLib/OpenBLAS/blob/develop/docs/extensions.md (though this usually gets updated with the respective release only, so as to avoid confusion)

[Mousius]

@martin-frbg couldn't we use cblas_sgemm_batch with pointers at strides?

[Mousius]

PyTorch does something similar with MKL's batch GEMM: https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/CPUBlas.cpp#L627-L640

[martin-frbg]

Probably, but I haven't looked yet.

[markusheimerl]

Is there still a development branch with an unfinished implementation? #2695 (comment)

[martin-frbg]

Is there still a development branch with an unfinished implementation? #2695 (comment)

Not that I'm aware of - but I implemented "plain" cblas_?gemm_batch in early summer of last year, lives in interface/gemm_batch.c. Only the strided implementation is missing

[Mousius]

I was considering if we could add ?gemm_batch to the BLAS interface as well, for the strided use case in PyTorch 😸 That's why it was front of my mind.

[martin-frbg]

I was considering if we could add ?gemm_batch to the BLAS interface as well, for the strided use case in PyTorch 😸 That's why it was front of my mind.

TBH I'm actually not sure right now why I only implemented the CBLAS interface, apart from the usual time constraints and the fact that I was plundering xianyi's abandoned small-matrices branch (#4720). It may well have been something simple like not being sure about the interface specification or having only two brain cells.

[martin-frbg]

quick hack in #5449 now - but still without the strided case and compile-tested only