v0.2.9

News: The vLLM project has fully adopted AutoAWQ

It is no secret that maintaining a project such as AutoAWQ that has 2+ million downloads, 7000+ models on Huggingface, and 2.1k stars is hard for a solo developer who is doing this in their free time.

Important Notice:

• AutoAWQ is officially deprecated and will no longer be maintained.
• The last tested configuration used Torch 2.6.0 and Transformers 4.51.3.
• If future versions of Transformers break AutoAWQ compatibility, please report the issue to the Transformers project.

Alternative:

• AutoAWQ has been adopted by the vLLM Project: https://github.com/vllm-project/llm-compressor
• MLX-LM now supports AWQ for Mac devices: http://github.com/ml-explore/mlx-lm

For further inquiries, feel free to reach out:

• X: https://x.com/casper_hansen_
• LinkedIn: https://www.linkedin.com/in/casper-hansen-804005170/

What's Changed

• Add Qwen2.5-VL by @seungwoos in #706
• fix setup by @jiqing-feng in #715
• Fixed issue with the slow implementation warning by @Egor-Krivov in #711
• fix(cache): remove k dim in cache by @neurowelt in #718
• fix dtype mismatch by @jiqing-feng in #740
• NEWS by @casper-hansen in #750
• Added Qwen 3 support. by @zju-stu-lizheng in #751
• assertion for non-activated experts in MoE by @casper-hansen in #755
• update scripts by @casper-hansen in #756
• Add Qwen2.5-Omni support. by @tiger-of-shawn in #759
• the final piece by @casper-hansen in #760
• the final piece (v2) by @casper-hansen in #761

New Contributors

• @seungwoos made their first contribution in #706
• @neurowelt made their first contribution in #718
• @zju-stu-lizheng made their first contribution in #751
• @tiger-of-shawn made their first contribution in #759

Full Changelog: v0.2.8...v0.2.9

v0.2.8

What's Changed

• Minimum of torch 2.2.0 during build by @casper-hansen in #649
• post release 1 by @casper-hansen in #650
• Post release 2 - All additional packages goes into extras by @casper-hansen in #653
• Fused attention: Switch to Flash Decoding by @casper-hansen in #656
• Add EXAONE support by @lgai-exaone in #651
• fix exaone import by @casper-hansen in #659
• fix "Expected all tensors to be on the same device" by @casper-hansen in #664
• multi-gpu fix by @casper-hansen in #668
• Fix missing embed_tokens by @casper-hansen in #671
• install hub main branch by @casper-hansen in #672
• pin huggingface_hub by @casper-hansen in #673
• bump to 0.2.7.post3 by @casper-hansen in #674
• improve type hinting and fix use_cache by @casper-hansen in #680
• fix bug when using FSDP by @kaixuanliu in #696
• Added DeepSeek V3 support. by @LagPixelLOL in #688
• add ability to define torch_dtype by @casper-hansen in #697
• automatically load dtype from config by @casper-hansen in #698
• Enable triton on XPU devices by @Egor-Krivov in #695
• fix workflow build by @casper-hansen in #699
• bump to 028 by @casper-hansen in #700

New Contributors

• @lgai-exaone made their first contribution in #651
• @kaixuanliu made their first contribution in #696
• @LagPixelLOL made their first contribution in #688
• @Egor-Krivov made their first contribution in #695

Full Changelog: v0.2.7...v0.2.8

v0.2.7.post3

What's Changed

• Fused attention: Switch to Flash Decoding by @casper-hansen in #656
• Add EXAONE support by @lgai-exaone in #651
• fix exaone import by @casper-hansen in #659
• fix "Expected all tensors to be on the same device" by @casper-hansen in #664
• multi-gpu fix by @casper-hansen in #668
• Fix missing embed_tokens by @casper-hansen in #671
• install hub main branch by @casper-hansen in #672
• pin huggingface_hub by @casper-hansen in #673
• bump to 0.2.7.post3 by @casper-hansen in #674

New Contributors

• @lgai-exaone made their first contribution in #651

Full Changelog: v0.2.7.post2...v0.2.7.post3

v0.2.7.post2

What's Changed

• Post release 2 - All additional packages goes into extras by @casper-hansen in #653

Full Changelog: v0.2.7.post1...v0.2.7.post2

v0.2.7.post1

What's Changed

• Minimum of torch 2.2.0 during build by @casper-hansen in #649
• post release 1 by @casper-hansen in #650

Full Changelog: v0.2.7...v0.2.7.post1

v0.2.7

What's Changed

• fix: pass rope_theta argument when initializing LlamaLikeBlock for models like qwen2, mistral, etc. by @Shuai-Xie in #568
• Add Gemma2 support. by @radi-cho in #562
• ignore onnx in ignore_patterns by @casper-hansen in #570
• Add Internlm2 support by @Crystalcareai in #576
• quantization fails with old datasets by @stas00 in #593
• doc: replace a broken example with a working one by @stas00 in #595
• Implement NO_KERNELS flag and update torch requirement by @devin-ai-integration in #582
• AWQ Triton kernels. Make autoawq-kernels optional. by @casper-hansen in #608
• device_map defaults to auto by @casper-hansen in #607
• Let installed PyTorch decide required version number by @wasertech in #573
• Replace itrex qbits to ipex woq linear by @jiqing-feng in #549
• enable awq ipex linear in transformers by @jiqing-feng in #610
• fix for "two devices" issue due to RoPE changes by @davedgd in #630
• add qwen2vl support by @kq-chen in #599
• Add support for Phi-3-vision series model by @Isotr0py in #596
• support minicpm3.0 by @LDLINGLINGLING in #605
• Enable Intel GPU path and lora finetune and change examples to support different devices by @jiqing-feng in #631
• Replace custom sharding with save_torch_state_dict from huggingface_hub by @casper-hansen in #644
• New release (0.2.7) + Fix build by @casper-hansen in #647
• Only build once by @casper-hansen in #648

New Contributors

• @Shuai-Xie made their first contribution in #568
• @radi-cho made their first contribution in #562
• @Crystalcareai made their first contribution in #576
• @stas00 made their first contribution in #593
• @wasertech made their first contribution in #573
• @jiqing-feng made their first contribution in #549
• @davedgd made their first contribution in #630
• @kq-chen made their first contribution in #599

Full Changelog: v0.2.6...v0.2.7

v0.2.6

What's Changed

• Cohere Support by @TechxGenus in #457
• Add phi3 support by @pprp in #481
• Support Weight-Only quantization on CPU device with QBits backend by @PenghuiCheng in #437
• Fix typo by @wanyaworld in #486
• Add updates + sponsorship by @casper-hansen in #495
• Update README.md by @casper-hansen in #497
• Update doc by @imba-tjd in #499
• add support for Openbmb/MiniCPM by @LDLINGLINGLING in #504
• Update RunPod support by @casper-hansen in #514
• add deepseek v2 support by @TechxGenus in #508
• nan problem of Qwen2-72B quantization by @baoyf4244 in #519
• Qwen nan fix by @baoyf4244 in #522
• fix deepseek v2 input feat by @TechxGenus in #524
• Batched quantization by @casper-hansen in #516
• Fix step size when computing clipping by @casper-hansen in #531
• Pin torch version to 2.3.1 by @devin-ai-integration in #542
• Revert "Pin torch version to 2.3.1 (#542)" by @casper-hansen in #547
• CLI example + Runpod launch script by @casper-hansen in #548
• Print warning if AutoAWQ cannot load extensions by @casper-hansen in #515
• Remove progress bars by @casper-hansen in #550
• Add test for chunked methods by @casper-hansen in #551
• Llama with inputs_embeds only(LLava-v1.5 bug fixed) and Llava-v1.6 Support by @WanBenLe in #471
• Better CLI + RunPod Script by @casper-hansen in #552
• Release 026 by @casper-hansen in #546
• pin torch==2.3.1 by @casper-hansen in #554
• Remove ROCm build and only build for PyPi by @casper-hansen in #555

New Contributors

• @pprp made their first contribution in #481
• @PenghuiCheng made their first contribution in #437
• @wanyaworld made their first contribution in #486
• @imba-tjd made their first contribution in #499
• @LDLINGLINGLING made their first contribution in #504
• @baoyf4244 made their first contribution in #519
• @devin-ai-integration made their first contribution in #542
• @WanBenLe made their first contribution in #471

Full Changelog: v0.2.5...v0.2.6

v0.2.5

What's Changed

• Fix fused models for tf >= 4.39 by @TechxGenus in #418
• FIX: Add safe guards for static cache + llama on transformers latest by @younesbelkada in #401
• Pin: lm_eval==0.4.1 by @casper-hansen in #426
• Implement apply_clip argument to quantize() by @casper-hansen in #427
• Workaround: illegal memory access by @casper-hansen in #421
• Add download_kwargs for load model (#302) by @Roshiago in #399
• add starcoder2 support by @shaonianyr in #406
• Add StableLM support by @Isotr0py in #410
• Fix starcoder2 fused norm by @TechxGenus in #442
• Update generate example to llama 3 by @casper-hansen in #448
• [BUG] Fix github action documentation build by @suparious in #449
• Fix path by @casper-hansen in #451
• FIX: 'awq_ext' is not defined error by @younesbelkada in #465
• FIX: Fix multiple generations for new HF cache format by @younesbelkada in #444
• support max_memory to specify mem usage for each GPU by @laoda513 in #460
• Bump to 0.2.5 by @casper-hansen in #468

New Contributors

• @Roshiago made their first contribution in #399
• @shaonianyr made their first contribution in #406
• @Isotr0py made their first contribution in #410
• @suparious made their first contribution in #449
• @laoda513 made their first contribution in #460

Full Changelog: v0.2.4...v0.2.5

v0.2.4

What's Changed

• Add Gemma Support by @TechxGenus in #393
• Pin transformers>=4.35.0,<=4.38.2 by @casper-hansen in #408
• Bump to v0.2.4 by @casper-hansen in #409

New Contributors

• @TechxGenus made their first contribution in #393

Full Changelog: v0.2.3...v0.2.4

v0.2.3

What's Changed

• New optimized kernels by @casper-hansen in #365
• Fix double bias by @casper-hansen in #383
• x_max -> x_mean and w_max -> w_mean name changes and some comments by @OscarSavolainenDR in #378

New Contributors

• @OscarSavolainenDR made their first contribution in #378

Full Changelog: v0.2.2...v0.2.3