Add graded rank-biased precision (GRBP)

[sushobhan2024]

Extend the existing RBP metric to handle graded relevance judgements (non-binary r_i ∈ [0, 1]) as described in Rank-Biased Precision for Measurement of Retrieval Effectiveness . Graded RBP reflects fractional utility rather than binary success, but uses the same weighting and persistence model.

Proposed Implementation

• Add a new class GradedRBP, inheriting from RBP
• Reads a relevance field (default: grade) from the test ItemList
• Scale grades to a unit range [0, 1]
• Default grade value for unseen items = 0.25
• If grade field is not found, defaults to binary RBP

Questions

Should we assume the grade field is already scaled to [0, 1]? If not, we can apply scaling. So, should scaling be optional here? Should we divide by the max grade value for scaling or is there a different method?

[mdekstrand]

• I think we should require grade to be pre-scaled. The metric will not have sufficient information to properly auto-scale grades — the maximum grade in a given item list is not necessarily the maximum grade that could be assigned.
• If the grade field is not defined in the item list, perhaps it should be an error rather than silently converting to binary RBP.
• Why should the class inherit from RBP? It's usually best to have shallow inheritance hierarchies unless there is a specific reason to inherit.
• I am indifferent as to whether we create a new class, or make RBP take an optional grade field name (which would be None to use binary relevance); probably whichever version has less code repetition. If code repetition is the reason to make GradedRBP extend RBP, then it would be better to just make the class configurable.

[sushobhan2024]

Making RBP configurable sounds better design.