Calculating Kappa
- Physician A finds that 30 of the patients have swollen knees.
- Physician B finds 40 of the patients have swollen knees.
- The physicians agreed that 25 of the patients had swollen knees, and that 55 patients did not (leaving 15 patients where the doctors disagreed).
The Kappa statistic is calculated using the following formula:
| Observed agreement - chance agreement
1-chance agreement
|
First fill in your 2 X 2 table as follows:
| Physician A |
|||
Physician B |
Yes | No |
|
| Yes | 25 | 15 | |
| No | 5 | 55 |
|
The observed percentage agreement is: (a + d) / N
= (25 + 55) / 100
= 0.8
To calculate the chance agreement, note that Physician A found 30 / 100 patients to have swollen knees and 70/100 to not have swollen knees. Thus, Physician A said ‘yes’ 30% of the time. Physician B said ‘yes’ 40% of the time. Thus, the probability that both of them said ‘yes’ to swollen knees was 0.3 x 0.4 = 0.12. The probability that both physicians said ‘no’ to swollen knees was 0.7 x 0.6 = 0.42%. The overall probability of chance agreement is 0.12 + 0.42 = 0.54.
Thus the Kappa would be:
| Kappa = |
0.8 – 0.54 |
| 0.46 | |
| Kappa= | 0.57 |
A kappa value of 0.57 indicates moderate to good agreement between observers.