named statistical measure assessing reliability agreement fixed number assigning ratings number items classifying items contrasts work assessing agreement measure degree agreement classification expected chance generally agreed measure significance guidelines given nominal scale ratings version available ordered ratings introduction scott statistical measure inter reliability related appropriate certain instances scott work works number giving ratings fixed number items interpreted expressing extent observed agreement exceeds expected ratings completely randomly important note assumes rated set items specifically allows fixed number different items rated different individuals item rated item rated agreement thought follows fixed number people assign numerical ratings number items measure consistent ratings formula defined formula factor formula gives degree agreement chance formula gives degree agreement actually achieved chance complete agreement formula agreement expected chance formula example use following consider fourteen asked look patients psychiatrist gives possibly patient compiled matrix matrix example degree agreement level agreement expected chance let total number subjects let number ratings subject let number categories assignments subjects categories let represent number assigned subject category proportion assignments category formula formula extent agree subject pairs agreement relative number possible pairs formula formula formula formula mean formula formula formula formula formula formula formula worked example following example fourteen formula assign subjects formula total categories formula categories presented columns subjects presented rows cell lists number assigned indicated row subject indicated column category data table right formula formula formula sum cells sum formula calculations value formula proportion assignments formula formula formula category example taking column formula taking second row formula order formula need know sum formula formula sheet formula formula formula interpretation gave following table interpreting formula values table means universally accepted supplied evidence support basing instead personal opinion noted guidelines harmful helpful number categories subjects affect magnitude value higher fewer categories
Fleiss' kappa