Cohen's Kappa Calculator
Calculate Cohen's Kappa, a measure of inter-rater or model-vs-label agreement that corrects for chance.
Inputs
Fraction of items where the two raters/model and label agreed.
Fraction of agreement expected if labels were assigned randomly, based on marginal class frequencies.
Cohen's Kappa
0.7000
Observed Agreement
0.85%
Expected Agreement (Chance)
0.50%
Step by step
Kappa: (p_o − p_e) / (1 − p_e)
(0.85 − 0.5) ÷ (1 − 0.5)
= 0.7000
How it works
Cohen's Kappa measures agreement between two raters (or a model's predictions and true labels) while correcting for the agreement expected by chance alone: κ = (p_o − p_e) / (1 − p_e), where p_o is observed agreement and p_e is the agreement expected under random labeling given the class distributions. Kappa = 1 means perfect agreement, 0 means agreement no better than chance, and negative values mean agreement worse than chance. It's especially useful over raw accuracy when classes are imbalanced, since chance agreement is high in that case.
Formula
kappa = (p_o - p_e) / (1 - p_e)
- p_o
- Observed agreement fraction
- p_e
- Expected agreement by chance
Frequently Asked Questions
How is expected agreement (p_e) calculated?
p_e is computed from the marginal probabilities of each class for both raters: for each class, multiply its frequency under rater A by its frequency under rater B, then sum across all classes — this gives the agreement rate expected if labels were assigned independently at random.
What kappa value indicates good agreement?
A common rule of thumb (Landis & Koch) treats 0.61-0.80 as 'substantial' agreement and above 0.80 as 'almost perfect', though acceptable thresholds vary by field and application.
Why not just use raw percent agreement?
Raw percent agreement doesn't account for the fact that some agreement would happen purely by chance, especially when one class is much more common — Kappa corrects for this baseline, giving a fairer measure of true agreement.