Is the checkpoint of TILDEv2 trained w/o expansion available?

[kwang2049]

Hi,

Is the checkpoint of TILDEv2 trained w/o expansion available? If so, could you please also upload it to the HF-hub? Thanks.

[ArvinZhuang]

Hi @kwang2049, thanks for raising the issue.
Now all the model checkpoints in the TILDEv2 paper are available on HF-hub: https://huggingface.co/ielab
You should be able to use them in the v2 indexing script by setting --ckpt_path_or_name ielab/model_name

let me know if they work for you.

[kwang2049]

Hi @ArvinZhuang, thanks for your immediate response!

I am trying reproducing TILDEv2 (re-ranking on TREC-DL 2019 without document expansion). Here is link to the whole process of the reproduction with colab notebook: https://colab.research.google.com/drive/1VpF1gsq1dCcNaD_LrPf1jWf19LKXciZ9?usp=sharing.

For evaluation convenience (blocks 1-5 in the notebook), I extracted all the document IDs from your run.trec2019-bm25.res results, which results in 42804 different douments.

Then based on collection, I ran the indexingv2.py, inferencev2.py and trec_eval for the checkpoint ielab/TILDEv2-noExp. However, here is what I got for the final evaluation results:

map                   	all	0.4187
ndcg_cut_10           	all	0.6058

which seems different from the reported scores in the paper (I think it is in Table 3 and the reported scores are 0.613 and 0.417 for nDCG@10 and MAP, resp.).

Could you please help me and tell me what might be the issue? I just want to make sure my reproduction is correct:).

Thanks!

[ArvinZhuang]

Hi @kwang2049 , can you try to re-rank this bm25 run file? Maybe I uploaded the wrong bm25 file that I used to re-rank in the paper.
bm25-top1000-dl2019-pass.txt

Also, slightly different results are expected as we do notice different environments (Linux, macOS, windows, Colab, etc) will impact results for some reason. The scores in my readme example that I reproduced were also slightly different (a little bit higher) than what I reported in the paper...

[kwang2049]

Hi @kwang2049 , can you try to re-rank this bm25 run file? Maybe I uploaded the wrong bm25 file that I used to re-rank in the paper. bm25-top1000-dl2019-pass.txt

Also, slightly different results are expected as we do notice different environments (Linux, macOS, windows, Colab, etc) will impact results for some reason. The scores in my readme example that I reproduced were also slightly different (a little bit higher) than what I reported in the paper...

Thanks for your help! I have just tried the new bm25 retrieval results and here is the link to the colab: https://colab.research.google.com/drive/1VpF1gsq1dCcNaD_LrPf1jWf19LKXciZ9?usp=sharing

This time the evaluation results:

map                   	all	0.4187
ndcg_cut_10           	all	0.6091

It seems to be still slightly different, especially for nDCG@10. It might not be the issue of different enviroments🤔, since I got identical results when I ran the same evaluation process on another machine.

Anyway, could you please tell me whether my reproduction code is correct? I do not really care about the absolute scores but just want to make sure I am doing the right way:)

[ArvinZhuang]

Your script looks right to me @kwang2049, there must be a small mismatch between the refactored code in this repo and my original experiment code... After a quick investigation, I think this may due to the stopwords list mismatch or different max padding lengths. Anyway here is my original run file that can get the same score in the paper if you need it :) TILDEv2_rerank_BM25_top1000_dl2019.txt