A retrieval ranking score needs its relevance judgments

Separate relevance judgments from retrieval scores

A search system returns a neat descending score column and reports high quality. Its own scores establish the order, but they do not supply the reference relevance labels needed to evaluate that order. Before accepting a ranking measure, ask who judged which query-document pairs, under what task and with what unresolved cases. A document can rank highly without answering the user’s actual question.

Scikit-learn’s NDCG definition discounts supplied relevance scores according to predicted rank and normalizes by the ideal ranking. Its interface exposes a cutoff and tie handling. These are part of the method, rather than cosmetic reporting options. The gain mapping, candidate universe and query aggregation must likewise be fixed before comparing receiving search configurations.

Three judged documents make the arithmetic inspectable

This hypothetical fixture contains one query and exactly three candidate documents, all judged in advance on a stipulated 0–3 relevance scale. Their returned order has relevance 3, 0, 2. Use the raw relevance number as linear gain and a discount of 1/log₂(rank+1). There are no tied retrieval scores, unknown judgments or unequal query weights in this example.

The returned DCG at three is 3/log₂(2)+0/log₂(3)+2/log₂(4)=4. The ideal order is 3, 2, 0, with DCG 3+2/log₂(3), approximately 4.261860. Dividing gives NDCG@3 approximately 0.938557. The author independently computed this arithmetic locally; it is not an executed search benchmark or a score for any available company archive.

A cutoff changes which ordering defect counts

For this hypothetical ranking, at cutoff one the returned top document already has the highest grade, so the normalized result is 1. At cutoff two, the useful grade-2 document is still missing from the returned prefix: DCG@2 is 3 against the same approximately 4.261860 ideal, giving about 0.703918. At three it reappears, producing the higher 0.938557 result. These three numbers describe one fixed ranking under different cutoffs.

Choose a cutoff that corresponds to the inspected retrieval stage and use. A user who sees only two results does not experience the third position as though it were second. Keep gain convention explicit as well: replacing raw grades with an exponential transform creates another metric calculation. This article’s arithmetic deliberately uses linear gain; it should not be copied into an implementation with another convention under an unchanged label.

Do not relabel an unjudged document as irrelevant

The historical TREC 2020 Deep Learning guidelines describe depth pools, query subsets judged by NIST assessors and graded labels used for NDCG. They distinguish that evidence from sparse labels supporting a secondary MRR measure. The lesson is to retain how reference coverage was built. A benchmark’s judgment procedure is part of what its result can support.

For an operational archive, keep three states distinct: judged relevant, judged irrelevant and unjudged. If an evaluator maps unjudged pairs to zero for a specified computation, disclose that mapping and the affected count. The resulting score describes that procedure; it does not prove a human found every zero-valued item irrelevant. Compare systems on a defensible common judgment basis before attributing a difference to retrieval quality.

The complete three-document fixture has no such gap. A real review should record reference task, assessors or adjudication procedure, candidate identifiers, grades, judgment version and unjudged fraction at each important cutoff. A score with missing reference coverage may call for more targeted judging rather than a different search algorithm.

Keep ranking quality within its evidence boundary

The completed example accepts only the stated ranking calculations and shows why cutoff one hides the second-position defect. Save the query, candidate set, judgments, returned scores/order, gain mapping, discount, cutoff, tie policy and implementation version. For several queries, retain per-query scores and the declared aggregation so an average cannot conceal which tasks were unsupported.

NDCG does not establish that a generated answer preserves a manual’s exception, that a user may retrieve the document or that an unsupported action is safe. Those require their own context and access checks. Use the ranking fixture to diagnose ordering, then inspect the permitted passages and answer behavior for the actual task. That sequence leaves a receiving reviewer with a reproducible ranking claim and its remaining limits.