This replication for lce-loss is based on Capreolus toolkit.
- Prerequisites: Python 3.7+ and Java 11. See the installation instructions
- Clone capreolus toolkit: git clone https://github.com/crystina-z/capreolus.git -b feature/eval+ptmaxp
- Run an experiment:
python run.py --config_path yaml_config_neg/yaml_filename
Here are the three tables we stated and the corresponding yaml file for each row
| Baseline - HN+FirstStage - Loss - n | MRR@10 | yaml_file_name |
|---|---|---|
| (1) monoELECTRABase-BM25-CE-1 | 0.378 | test_line4_nneg_1_bm25_ce.yaml |
| (2) monoELECTRABase-BM25-HG-1 | 0.379 | test_line5_nneg_1_bm25_hg.yaml |
| (3) monoELECTRABase-BM25-LCE-1 | 0.378 | test_line24_nneg_1_bm25_lce.yaml |
| (4) monoELECTRABase-BM25-LCE-7 | 0.391 | test_line24_nneg_1_bm25_lce.yaml |
| (5) monoELECTRABase-TCT-ColBERTv2-CE-1 | 0.365 | test_line26_nneg_1_tct_ce.yaml |
| (6) monoELECTRABase-TCT-ColBERTv2-HG-1 | 0.375 | test_line27_nneg_1_tct_hg.yaml |
| (7) monoELECTRABase-TCT-ColBERTv2-LCE-1 | 0.394 | test_line16_nneg_1_tct_lce.yaml |
| (8) monoELECTRABase-TCT-ColBERTv2-LCE-7 | 0.401 | test_line11_nneg_7_tct_lce.yaml |
Please note that in table 2 both a and c, b and d are trained with the same yaml file but inference on different ones.
| HN-FirstStage | MRR@10 | yaml_file_name |
|---|---|---|
| (a) BM25-BM25 | 0.391 | test_line10_nneg_7_bm25_lce.yaml |
| (b) TCT-ColBERTv2-BM25 | 0.389 | test_line11_nneg_7_tct_lce.yaml |
| (c) BM25-TCT-ColBERTv2 | 0.402 | test_line10_nneg_7_bm25_lce.yaml |
| (d) TCT-ColBERTv2-TCT-ColBERTv2 | 0.401 | test_line11_nneg_7_tct_lce.yaml |
| Group-size | MRR@10 | yaml_file_name |
|---|---|---|
| (a-1) 2 | 0.393 | test_line26_nneg_1_tct_ce.yaml |
| (a-2) 4 | 0.400 | test_line17_nneg_3_tct_lce.yaml |
| (a-3) 8 | 0.401 | test_line11_nneg_7_tct_lce.yaml |
| (a-4) 16 | 0.408 | test_line19_nneg_15_tct_lce.yaml |
| (a-5) 32 | 0.414 | test_line20_nneg_31_tct_lce.yaml |
| (b-1) 2 | 0.381 | test_table3_nneg_1-bm25-train.yaml |
| (b-2) 4 | 0.397 | test_table3_nneg_3-bm25-train.yaml |
| (b-3) 8 | 0.402 | test_table3_nneg_7-bm25-train.yaml |
| (b-4) 16 | 0.403 | test_table3_nneg_15-bm25-train.yaml |
| (b-5) 32 | 0.407 | test_table3_nneg_31-bm25-train.yaml |