Implementation of three different type of LLMs that are GPT2 , Mamba and xLSTM.
Some examples of Mamba models are Mistral's Codestral Falcon series, xLSTM's model from NXAI.
All models were trained on Kaparthy's educational example on Shakespeare text.
Model implementation was done and partially inspired by :
- GPT2 : https://github.com/karpathy/nanoGPT
- MAMBA : https://github.com/alxndrTL/mamba.py
- xLSTM : https://github.com/NX-AI/xlstm
Please run, the following command to install relevant dependencies:
pip install .To prepare the dataset, please run (which is the same command as in nanoGPT).
python data/shakespeare_char/prepare.pyYou may want to run small LLM to generate Shakespeare-like text, please check out the notebook notebooks/shakespeare_char.ipynb that trains GPT, Mamba and xLSTM models from scratch.