How do we measure reliability?

In order to assess the reliability of a nominal observation, like whether a patient has a specific physical exam finding, we use a statistic called Cohen’s kappa (often known as simply “kappa”). This statistic represents how much more agreement there is between two raters than we would expect to see by chance alone.

 It is not necessary for you to know how to calculate it, but it is shown here to increase your understanding.