Key takeaway
The same agreement percentage can produce different kappas when class marginals change. Neither agreement measure establishes that the labels are correct.
Ask what the agreement percentage leaves out
Two annotation reports both say that reviewers agreed on 90% of cases. One distributes its labels fairly evenly; the other labels nearly every case positive. Before treating the reports as equivalent, retain the joint assignments and the amount of agreement expected under the chosen statistic’s marginal-label model. A percentage alone discards that structure.
Scikit-learn defines Cohen’s kappa as (pₒ−pₑ)/(1−pₑ). Here pₒ is observed agreement, while pₑ is calculated from each annotator’s empirical class proportions. This is a declared statistical baseline, not an observation that the annotators actually labelled cases at random. Record that interpretation with the score.
Two twenty-case tables retain the same raw agreement
These hypothetical fixtures concern two raters assigning Yes or No to exactly twenty invented cases. In the balanced fixture, joint counts are nine Yes/Yes, one Yes/No, one No/Yes and nine No/No. Each rater uses Yes on half the cases. Observed agreement is 18/20=0.9 and the marginal baseline is 0.5×0.5+0.5×0.5=0.5, giving kappa 0.8.
In the positive-heavy fixture, the corresponding counts are 18,1,1,0. Each rater uses Yes on 19/20 cases. Observed agreement is still 0.9, but pₑ=0.95²+0.05²=0.905, giving (0.9−0.905)/(1−0.905)≈−0.052632. The author independently recomputed these synthetic quantities. There is no reference truth, observed annotation campaign or accepted dataset in either fixture.
| Synthetic fixture | Counts [YY, YN; NY, NN] | Raw agreement | Marginal baseline pₑ | Cohen’s κ |
|---|---|---|---|---|
| Balanced | [9, 1; 1, 9] | 0.90 | 0.50 | 0.80 |
| Positive-heavy | [18, 1; 1, 0] | 0.90 | 0.905 | ≈−0.052632 |
Preserve the table and its orientation
Statsmodels accepts a square contingency table with one rater in rows and the other in columns. Save the category order and joint counts, rather than only a scalar copied into a report. The calculation should be reconstructible even if the display order changes. Also identify whether the result uses ordinary unweighted kappa or a weighting convention for ordered labels.
The worked example is binary and unweighted. It does not select a disagreement penalty for an ordinal severity scale or aggregate more than two raters. A change in category set, missing-case handling or weighting creates a different calculation. Keep excluded and unassigned cases in the coverage record before a reviewer compares two published percentages.
Use the difference to inspect the labelling task
The negative kappa in the second fixture follows from a constructed baseline slightly above its observed agreement. It is not proof that every positive label is wrong, that the reviewers deliberately disagreed, or that the first archive is commercially superior. Review class prevalence, case selection and disagreement types alongside the statistic. Do not rebalance an evaluation merely to obtain a more attractive coefficient.
Agreement and correctness require different evidence. Two annotators can consistently follow an incorrect definition, or use the same incomplete source. Preserve the written label dictionary, independent assignments, source evidence and adjudication history. A reference-label review can change the decision even when both agreement measures look favourable.
Return an annotation claim someone can reproduce
For the two illustrations, the supported result is 90% agreement under two different marginal structures, with different kappas. There is no universal passing threshold in this article. The receiving note should identify the case population, two raters, label set, full table, missingness policy, weight convention and implementation, then state what review remains.
An operational follow-up can examine the actual disagreement cells before expanding annotation work. If the label definition changes, version it and identify which cases need reassessment. This leaves a buyer with a specific consistency question and preserves uncertainty about truth. It supplies no model-performance result, promised improvement or authority to distribute the labelled records.