Discussion: Some examples for consideration. #160
Description
In the ThunderKittens arXiv paper, they implemented a series of hardware-aware algorithms and designed experiments. I believe some of these algorithms are also meaningful and could serve as references for us.
The ThunderKittens experiments are divided into the following sections:
- Baseline: GEMM and Attention. Among them, FlashAttention has implementations for forward, backward, as well as causal, non-causal, and grouped query attention.
- Linear Attention: Flash Linear Attention.
- State space models: Mamba-2.
- Long Convolution: FlashFFTConv.
- Others: Fused dropout-residual-norm and Rotary Encoding.
Some kernels can be used in different fields, for example, Long Convoluntion can be applied in CNNs, while RoPE and DropOut can be utilized in LLM. This demonstrates the generality across different fields, seem worth considering?