Build PERF for HQQ vs. Affine

[namgyu-youn]

Summary:
Build a benchmark module for affine (base) vs. HQQ. Both workflows don't require calibration flow, which means there is no need to compute element-wise ops. The HQQ unit test is added because there was only an e2e test.

Test Plan:

test/quantization/test_quant_primitives.py
benchmarks/benchmark_intx.py

PERF result:

> python benchmarks/benchmark_intx.py --model_id Qwen/Qwen3-14B --limit 50

Method             Size(GB)   Comp     Quant(s)   Fwd(ms)    Tok/s      Mem(GB)    Accuracy
----------------------------------------------------------------------------------------------
INT8-INT4          9.465      1.73     0.8        404.58     3.5        18.87      0.88
INT8-INT4-HQQ      9.465      1.73     12.6       403.42     3.5        26.35      0.91

• Size(GB): Model size after quantization
• Comp: Compression ratio for model size (vs. origin)
• Quant(s): Quantization overhead
• Fwd(ms): Forward pass
• Tok/s (i.e., throughout): Real token generation speed, involving dequantization
• Mem(GB): Memory consumption while quantization
• Accuracy (i.e., ppl): Accuracy for GSM8K task

Related Issue/PR: #3156

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/3403

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 2 Active SEVs

There are 2 currently active SEVs. If your PR is affected, please view them below:

• Dr CI is temporarily not working due to API fairewall
• Pytorch CI is partially paused for the time being (updated 11/27)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[pytorch-bot]

❌ 🤖 pytorchbot command failed:

Got EOF while in a quoted string```
Try `@pytorchbot --help` for more info.

[namgyu-youn]

@pytorchbot label "topic: performance"

[namgyu-youn]

This is updated because the GSM8K task is CoT reasoning (long-context)

[namgyu-youn]

cc @metascroy can you take a look into this? I will move to E2E support for SINQ after this PR as we talked with #3156 (comment); we can add SINQ to this e2e benchmark I feel.

[metascroy]

Thanks! The unit test looks good to me. I'll let @jainapurva review the benchmark script, as I'm less familiar with that part of the codebase and if there are coding standards that are usually followed.

[jerryzh168]

intx is targeting mobile right? so testing performance on server doesn't seem to be helpful?

[namgyu-youn]

intx is targeting mobile right? so testing performance on server doesn't seem to be helpful?

@jerryzh168 Even though we don't test in AP device, testing performance would be helpful because there are only outdated CUDA benchmarks for HQQ:

• https://github.com/pytorch/ao/blob/main/benchmarks/benchmark_hqq.py
• https://github.com/pytorch/ao/blob/main/benchmarks/benchmark_uintx.py

. For the future plan, I only used up-to-date functions like TransformerEvalWrapper and version=2 in this PR. Ideally, we can go with comprehensive PERF for calibration-free (e.g., basic affine, HQQ) vs. calibration-based (e.g., AWQ, SmoothQuant) or something I guess.