PEFT Showdown — SmolLM2-1.7B on Medical Q&A

Comparing 6 parameter-efficient fine-tuning methods on the same model, dataset, and hardware. Built to answer one question: which method should you actually use?

Setup

• Model: HuggingFaceTB/SmolLM2-1.7B (base)
• Dataset: medalpaca/medical_meadow_medqa (10,178 samples)
• Hardware: Kaggle T4 GPU (free tier)
• Framework: HuggingFace PEFT + TRL

Methods compared

Method	Trainable params	% of model
LoRA	3.1M	0.18%
QLoRA	3.1M	0.35%
DoRA	3.34M	0.19%
VeRA	0.29M	0.02%
AdaLoRA	4.72M	0.28%
IA3	0.29M	0.02%

Key findings

• IA3 matched LoRA quality with 10x fewer parameters
• VeRA hit a hard ceiling — only 9.5% loss improvement
• AdaLoRA's 60% loss drop is misleading due to SVD warmup
• QLoRA is the honest default for free GPU users
• DoRA consistently achieved the best final loss

Results

Run it yourself

Each notebook is self-contained and runs on free Kaggle T4. Just open and run top to bottom.

Phase 2 — coming soon

Testing safety alignment degradation across all 6 methods after fine-tuning on benign data.

Citation

If you use this work please star the repo and link back.