Question about result reproduction

[searchthe]

I tried the quantization configuration in #4 and #5, i.e. v6k4096, but could not reproduce the similar results in the llama2 13B model. I also used the script configuration in the readme, i.e. v8k65536res256, but could only get poor ppl results. How can I reproduce the unfinetuned results in Table 10?

Here are the scripts and results of the algorithm branch I used:

python run_vptq.py --model_name /Llama-2-13b-hf --output_dir outputs/llama2_13b_v6k4096 --vector_lens -1 6 --group_num 1 --num_centroids -1 4096 --num_res_centroids -1 -1 --npercent 0 --blocksize 128 --new_eval --seq_len 4096 --kmeans_mode hessian --num_gpus 1 --enable_perm --enable_norm --save_model --save_packed_model --hessian_path /hess/llama2_13b_6144/ --inv_hessian_path /invhess/llama2_13b_6144/ --ktol 1e-5 --kiter 100

results:
"wikitext2": 253.8889923095703,
"c4-new": 180.23741149902344

python run_vptq.py --model_name /Llama-2-13b-hf --output_dir outputs/llama2_13b_v8k65536r256 --vector_lens -1 8 --group_num 1 --num_centroids -1 65536 --num_res_centroids -1 -256 --npercent 0 --blocksize 128 --new_eval --seq_len 4096 --kmeans_mode hessian --num_gpus 1 --enable_perm --enable_norm --save_model --save_packed_model --hessian_path /hess/llama2_13b_6144/ --inv_hessian_path /invhess/llama2_13b_6144/ --ktol 1e-5 --kiter 100

results:
"wikitext2": 71.66665649414062,
"c4-new": 64.28524017333984

Thanks and look forward to your answer.

[YangWang92]

hmmm, let me check, these might be something wrong in the configuration or the open sourced algorithm codes. Can you check the results from here? https://huggingface.co/collections/VPTQ-community/reproduced-vptq-tech-report-baseline-66fbf1dffe741cc9e93ecf04 oh, some configurations may not updated in this collection, because we have updated our inference codes.

[searchthe]

hmmm, let me check, these might be something wrong in the configuration or the open sourced algorithm codes. Can you check the results from here? https://huggingface.co/collections/VPTQ-community/reproduced-vptq-tech-report-baseline-66fbf1dffe741cc9e93ecf04 oh, some configurations may not updated in this collection, because we have updated our inference codes.

I tried "Llama-2-13b-hf-v6-k4096-0-woft" and got ppl results close to the paper:
wikitext2: 6.6192216873168945
c4-new: 8.55883502960205

[YangWang92]

hmmm, let me check, these might be something wrong in the configuration or the open sourced algorithm codes. Can you check the results from here? https://huggingface.co/collections/VPTQ-community/reproduced-vptq-tech-report-baseline-66fbf1dffe741cc9e93ecf04 oh, some configurations may not updated in this collection, because we have updated our inference codes.

I tried "Llama-2-13b-hf-v6-k4096-0-woft" and got ppl results close to the paper: wikitext2: 6.6192216873168945 c4-new: 8.55883502960205

We may have some bugs in the current (open sourced) codes, or have you checked the hessian files?

[searchthe]

hmmm, let me check, these might be something wrong in the configuration or the open sourced algorithm codes. Can you check the results from here? https://huggingface.co/collections/VPTQ-community/reproduced-vptq-tech-report-baseline-66fbf1dffe741cc9e93ecf04 oh, some configurations may not updated in this collection, because we have updated our inference codes.

I tried "Llama-2-13b-hf-v6-k4096-0-woft" and got ppl results close to the paper: wikitext2: 6.6192216873168945 c4-new: 8.55883502960205

We may have some bugs in the current (open sourced) codes, or have you checked the hessian files?

I directly used the following code to obtain hessian and invhess, will this cause any problems?

https://github.com/Cornell-RelaxML/quip-sharp/blob/main/quantize_llama/hessian_offline_llama.py
https://gist.github.com/YangWang92/ec98a86c3a33c573b601cf4348d0a0e7

[YangWang92]

hmmm, let me check, these might be something wrong in the configuration or the open sourced algorithm codes. Can you check the results from here? https://huggingface.co/collections/VPTQ-community/reproduced-vptq-tech-report-baseline-66fbf1dffe741cc9e93ecf04 oh, some configurations may not updated in this collection, because we have updated our inference codes.

I tried "Llama-2-13b-hf-v6-k4096-0-woft" and got ppl results close to the paper: wikitext2: 6.6192216873168945 c4-new: 8.55883502960205

We may have some bugs in the current (open sourced) codes, or have you checked the hessian files?

I directly used the following code to obtain hessian and invhess, will this cause any problems?

https://github.com/Cornell-RelaxML/quip-sharp/blob/main/quantize_llama/hessian_offline_llama.py https://gist.github.com/YangWang92/ec98a86c3a33c573b601cf4348d0a0e7

the number of sequences and datasets are important, and you can download from here https://huggingface.co/relaxml to use their hessians.

[YangWang92]

https://huggingface.co/relaxml/Hessians-Llama-2-13b-chat-6144 and https://huggingface.co/relaxml/Hessians-Llama-2-13b-6144 for example

[YangWang92]

Hi @searchthe, you can reopen the issue if you have more questions. Thanks!

[searchthe]

Sorry it took so long to download hessian, I used hessian from relaxml , and processed invhess with invhessian.py , still got poor ppl results:

wikitext2: 145.5493927001953
c4-new: 135.51992797851562

scripts:
run_vptq.py --model_name /Llama-2-13b-hf --output_dir outputs/llama2_13b --vector_lens -1 6 --group_num 1 --num_centroids -1 4096 --num_res_centroids -1 -1 --npercent 0 --blocksize 128 --new_eval --seq_len 2048 --kmeans_mode hessian --num_gpus 1 --enable_perm --enable_norm --save_model --save_packed_model --hessian_path /hess/Hessians-Llama-2-13b-6144/ --inv_hessian_path /invhess/Hessians-Llama-2-13b-6144 --ktol 1e-5 --kiter 100

[searchthe]

By the way, I don't have permission to reopen this issue

[YangWang92]

Sorry it took so long to download hessian, I used hessian from relaxml , and processed invhess with invhessian.py , still got poor ppl results:

wikitext2: 145.5493927001953 c4-new: 135.51992797851562

scripts: run_vptq.py --model_name /Llama-2-13b-hf --output_dir outputs/llama2_13b --vector_lens -1 6 --group_num 1 --num_centroids -1 4096 --num_res_centroids -1 -1 --npercent 0 --blocksize 128 --new_eval --seq_len 2048 --kmeans_mode hessian --num_gpus 1 --enable_perm --enable_norm --save_model --save_packed_model --hessian_path /hess/Hessians-Llama-2-13b-6144/ --inv_hessian_path /invhess/Hessians-Llama-2-13b-6144 --ktol 1e-5 --kiter 100

let me rerun the experiment

[PingchengDong]

Hi, I also met the same reproduction issues when applying VPTQ on Llama2-7B. I firstly generated the hessians and inv hessians from hessian_collector.py and inv_hessian.py with 4096 ctx_size and 128 dev_size with the other args set as default. Then, I ran the codes with the below scripts from the algorithm branch:

python run_vptq.py
        --model_name /home/gpu2-user4/llm/llama/Llama-2-7b-hf
        --output_dir outputs/Llama-2-7B-v8-k65536-r256-seq4096/
        --vector_lens -1 8
        --group_num 1
        --num_centroids -1 65536
        --num_res_centroids -1 256
        --npercent 0
        --blocksize 128
        --new_eval
        --seq_len 4096
        --kmeans_mode hessian
        --num_gpus 4
        --enable_perm
        --enable_norm
        --save_model
        --save_packed_model
        --hessian_path ./hessians/llama2_7b_4096seq_128sample
        --inv_hessian_path ./hessians/llama2_7b_4096seq_128sample_inv
        --ktol 1e-5 --kiter 100

The results are:

wikitext2: 263.9034729003906
c4-new: 161.25343322753906

The results from the VPTQ-community are normal, but I want to try VPTQ on some customized LLMs, could you please help check this issue? Thanks.

[YangWang92]

@searchthe you can checkout this issue, I guess you both have the similar questions. #191