Inter Rater Reliability Calculator

Work out inter rater reliability instantly with clear inputs, formula shown and shareable results.

About Inter Rater Reliability Calculator

Cohen's kappa corrects raw agreement for the agreement expected by chance: (po - pe)/(1 - pe). With 45 and 30 agreements against 25 disagreements, observed agreement is 75% but chance agreement is 51%, giving kappa 0.490 - only moderate. Raw agreement always flatters, which is why kappa is the reporting standard.

Formulas

Cohen's kappa

kappa = (observed - expected) / (1 - expected)

Expected agreement

pe = sum of the products of the marginal proportions

Worked example

With the default values below, Inter Rater Reliability Calculator returns Cohen's Kappa of 0.4898. Change any input above and every figure here is recalculated in your browser.

Inputs used

Both raters said yesminimum 0: 45
Both raters said nominimum 0: 30
Only the first rater said yesminimum 0: 10
Only the second rater said yesminimum 0: 15

Results

Cohen's Kappa: 0.4898
Observed Agreement: 75.00%
Agreement Expected By Chance: 51.00%
Landis And Koch Interpretation: Moderate agreement

Step by step

1Values used
2Cohen's kappa
3Expected agreement
4Cohen's Kappa= 0.4898
5Observed Agreement= 75.00
6Agreement Expected By Chance= 51.00
7Landis And Koch Interpretation= Moderate agreement

Assumptions and limits

The result is derived only from the 4 inputs listed above. Anything not among them is not part of the calculation.
In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
Figures are rounded for display to 4 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.

How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.

Frequently Asked Questions

Why can kappa be low when agreement looks high?

Because when one category dominates, chance agreement is already high. 90% agreement on a 95% prevalent category can give a kappa near zero.

What kappa is acceptable?

Above 0.61 is usually considered substantial and 0.81 almost perfect, though clinical decisions often demand 0.80 or better.