Calculate Cohen's Kappa to measure inter-rater agreement corrected for chance.
Cohen's Kappa measures agreement between two raters (or a model's predictions and true labels) while correcting for the agreement expected by chance alone: κ = (p_o − p_e) / (1 − p_e), where p_o is observed agreement and p_e is the agreement expected under random labeling given the class distributions. Kappa = 1 means perfect agreement, 0 means agreement no better than chance, and negative values mean agreement worse than chance. It's especially useful over raw accuracy when classes are imbalanced, since chance agreement is high in that case.
kappa = (p_o - p_e) / (1 - p_e)
p_e is computed from the marginal probabilities of each class for both raters: for each class, multiply its frequency under rater A by its frequency under rater B, then sum across all classes — this gives the agreement rate expected if labels were assigned independently at random.
A common rule of thumb (Landis & Koch) treats 0.61-0.80 as 'substantial' agreement and above 0.80 as 'almost perfect', though acceptable thresholds vary by field and application.
Raw percent agreement doesn't account for the fact that some agreement would happen purely by chance, especially when one class is much more common — Kappa corrects for this baseline, giving a fairer measure of true agreement.