Can you provide a script that can reproduce the results of the paper?

[xuzukang]

I can't reproduce the results of the paper according to your configuration. Could you please provide a script that can directly reproduce the results of the paper?
I got very poor results. [wikitext2 ppl: 77.4593]

[YangWang92]

can you show us your configurations? Espicially the number of centroids and vector lengths? And have you verified the open sourced models' results?

[YangWang92]

I’ve been really busy lately — I’ll take a look over the weekend.

[xuzukang]

run_vptq.py --model_name /Llama-2-13b-hf --output_dir outputs/llama2_13b --vector_lens -1 6 --group_num 1 --num_centroids -1 4096 --num_res_centroids -1 -1 --npercent 0 --blocksize 128 --new_eval --seq_len 2048 --kmeans_mode hessian --num_gpus 1 --enable_perm --enable_norm --save_model --save_packed_model --hessian_path /hess/Hessians-Llama-2-13b-6144/ --inv_hessian_path /invhess/Hessians-Llama-2-13b-6144 --ktol 1e-5 --kiter 100

[YangWang92]

--inv_hessian_path /invhess/Hessians-Llama-2-13b-6144 -> Have you checked the Inv hessian? I think you should generate inv hessian from Hessians-Llama-2-13b-6144, by this script https://gist.github.com/YangWang92/ec98a86c3a33c573b601cf4348d0a0e7

[YangWang92]

Hi @xuzukang, can you tell me how you get "inv_hessian"? Thanks!

[xuzukang]

--inv_hessian_path /invhess/Hessians-Llama-2-13b-6144 -> Have you checked the Inv hessian? I think you should generate inv hessian from Hessians-Llama-2-13b-6144, by this script https://gist.github.com/YangWang92/ec98a86c3a33c573b601cf4348d0a0e7

Yes, my inv_hessian was generated using this script.
I used this script to obtain the Hessian matrix：https://github.com/Cornell-RelaxML/quip-sharp/blob/main/quantize_llama/hessian_offline_llama.py

[YangWang92]

Hi @xuzukang, BTW how many samples do you use in your hessiam matrix collection? hessian_offline_llama.py used 256 samples from redpajama-1T-sample. But in their papaers, they use 6144 samples, and you can download these hessians from here. https://huggingface.co/relaxml/Hessians-Llama-2-13b-6144

[fbatf]

Hi， @YangWang92 ，I also can't reproduce the results of the paper according to this configuration:
python3 run_vptq.py --model_name /data3/lowbit/yxy/llm/Meta-Llama-3.1-8B-Instruct --output_dir outputs/Meta-Llama-3.1-8B-Instruct/ --vector_lens -1 12 -1 --group_num 1 --num_centroids -1 65536 -1 --num_res_centroids -1 4096 -1 --npercent 0 --blocksize 128 --new_eval --seq_len 8192 --kmeans_mode hessian --num_gpus 8 --save_model --save_packed_model --hessian_path /data3/lowbit/yxy/llm/Hessians-Llama-31-8B-Instruct-6144-8k --inv_hessian_path /data3/lowbit/yxy/llm/InvHessians-Llama-31-8B-Instruct-6144-8k --enable_perm --enable_norm --ktol 1e-5 --kiter 200
I download hessian from https://huggingface.co/VPTQ-community/InvHessians-Llama-31-8B-Instruct-6144-8k/ and VPTQ-community/Hessians-Llama-31-8B-Instruct-6144-8k

when I set the enable_norm=true the loss comes to 29 and when I set the enable_norm=false the loss come to 12,so I test different configuration and find the same issue .Is here some problem in the algoritm branch?

I am little confused adout this:

if self.enable_norm:

tile_error = tile_error * self.quantizer.weight_scale.unsqueeze(self.norm_dim) + \

self.quantizer.weight_bias.unsqueeze(self.norm_dim)

tile_error = tile_error.matmul(tile_inv_hessian)

Losses1[:,i:i+step] = err1.matmul((w-q).T())

[(norm_tile_weight - norm_tile_qweight) * S + B] * H^-1 * H^-1 * (w-q).T()

inv_tile_error = tile_error.matmul(block_inv_hessian[k:k + self.step, k + self.step:])

if self.enable_norm:

inv_tile_error = (inv_tile_error - self.quantizer.weight_bias.unsqueeze(self.norm_dim)) / \

self.quantizer.weight_scale.unsqueeze(self.norm_dim)

tile_error = (tile_error - self.quantizer.weight_bias.unsqueeze(self.norm_dim)) / \

self.quantizer.weight_scale.unsqueeze(self.norm_dim)

you use tile_error=(qw-w)tile_inv_hessian
but I think maybe tile_error=((((qwthis.scale+this.bias)-(w*this.scale+this.bias))*tile_inv_hessian)-other.bias)/other.scale
this.bias and this.scale maybe this channel's scale parameters
other.bias and other.scale maybe from the channel that you add the error to
I am not sure if my thought is right but the if self.enable_norm code needs be implemented .
I run the commented-out code and it has some problems

[fbatf]

this code is from def vptq in vptq.py

[YangWang92]

Sorry about the late reply, I'm quite busy during these days. Let me check the replies. A quick explain is that the open sourced code is refactored a lot and removed some experimential codes.

[xuzukang]

Many thanks to the contributions of VPTQ. I'm really looking forward to VPTQ providing a script that can reproduce the results of the paper report.