Inter Annotator Agreement Calculator

Work out inter annotator agreement instantly with clear inputs, formula shown and shareable results.

About Inter Annotator Agreement Calculator

Raw agreement flatters annotators on skewed tasks: if 90 percent of items are negative, two people guessing would agree most of the time. Cohen's kappa subtracts the chance agreement and rescales, so it measures how much of the achievable agreement above chance was actually reached. Values below 0.4 mean the guidelines, not the annotators, need fixing.

Formula

Cohen's kappa

kappa = (Po - Pe) / (1 - Pe)

Po: Observed agreement
Pe: Agreement expected by chance

Worked example

With the default values below, Inter Annotator Agreement Calculator returns Cohen's kappa of 0.6889. Change any input above and every figure here is recalculated in your browser.

Inputs used

Observed agreementaccepts 0.01–100 %: 86 %
Agreement expected by chanceaccepts 0.01–99.9 %: 55 %
Items double-labelledminimum 1: 500

Results

Cohen's kappa: 0.6889
Agreement above chance: 31.00%
Items the annotators disagreed on: 70
Landis and Koch band: Substantial

Step by step

1Values used
2Cohen's kappa
3Cohen's kappa= 0.6889
4Agreement above chance= 31.00
5Items the annotators disagreed on= 70
6Landis and Koch band= Substantial

Assumptions and limits

The result is derived only from the 3 inputs listed above. Anything not among them is not part of the calculation.
In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
Figures are rounded for display to 4 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.

How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.

Frequently Asked Questions

What kappa should I require before scaling up?

Aim for 0.7 or above on a pilot batch. Below that, disagreements will end up in the training labels as noise and cap the achievable model accuracy.

Which coefficient should I use for more than two annotators?

Fleiss' kappa for multiple raters and nominal labels, or Krippendorff's alpha when raters differ per item or the labels are ordinal.