Duplicates in Qrels yield incorrect nDCG scores

[lgienapp]

When duplicated rows are present in qrel data, the ideal ranking of nDCG is severely overestimated, since the duplicates are treated as separate docs (and thus positive scores) when sorting. This inflates the iDCG score and causes incorrect results. MWE with TREC-DL20 data attached.

import trectools
import ir_datasets

qrels_df = (
    pd.DataFrame(ir_datasets.load("msmarco-passage/trec-dl-2020/judged").qrels_iter())
    .loc[:, ["query_id", "doc_id", "relevance"]]
    .rename({"query_id": "query", "doc_id": "docid", "relevance": "rel"}, axis="columns")
)

qrels = trectools.TrecQrel()
qrels.qrels_data = qrels_df

run = trectools.TrecRun("../data/external/dl20-passages-runs/input.1.gz") # replace with your local path if needed; randomly chosen run

print("Normal Qrels: ", trectools.TrecEval(run, qrels).get_ndcg())

qrels.qrels_data = pd.concat([qrels_df, qrels_df])

print("Duplicated Qrels: ", trectools.TrecEval(runs[0], qrels).get_ndcg())

Normal Qrels: 0.18245952802407694
Duplicated Qrels: 0.006991538267262153

[hscells]

Ideally, we should take the trec_eval approach and simply return an error when duplicate docids are present. Perhaps we could simply add a check in this function that would check for duplicates?

trectools/trectools/trec_qrel.py

Lines 76 to 98 in 8a896de

	def read_qrel(self, filename, qrels_header=None):
	# Replace with default argument for qrel_header
	if qrels_header is None:
	qrels_header = ["query", "q0", "docid", "rel"]
	# Set filename
	self.filename = filename
	# Read data from file
	self.qrels_data = pd.read_csv(filename, sep="\s+", names=qrels_header)
	# Enforce string type on docid column (if present)
	if "docid" in self.qrels_data:
	self.qrels_data["docid"] = self.qrels_data["docid"].astype(str)
	# Enforce string type on q0 column (if present)
	if "q0" in self.qrels_data:
	self.qrels_data["q0"] = self.qrels_data["q0"].astype(str)
	# Enforce string type on query column (if present)
	if "query" in self.qrels_data:
	self.qrels_data["query"] = self.qrels_data["query"].astype(str)
	# Removes the files that were not judged:
	self.qrels_data = self.qrels_data[self.qrels_data["rel"] >= 0]

Looking more deeply, probably qrels_data should not be assignable either, rather we should prefer that a dataframe is passed to a function which can also perform this check, perhaps a new class method called set_qrels_data? We could do this without breaking changes to the API by moving qrels-data to a function and annotating it with @property? (since assigning to qrels_data is likely something that people would do very rarely)