Statistical Inference for G -indices of Agreement

Abstract

The limitations of Cohen’s κ are reviewed and an alternative G-index is recommended for assessing nominal-scale agreement. Maximum likelihood estimates, standard errors, and confidence intervals for a two-rater G-index are derived for one-group and two-group designs. A new G-index of agreement for multirater designs is proposed. Statistical inference methods for some important special cases of the multirater design also are derived. G-index meta-analysis methods are proposed and can be used to combine and compare agreement across two or more populations. Closed-form sample-size formulas to achieve desired confidence interval precision are proposed for two-rater and multirater designs. R functions are given for all results.

1. Introduction

The assignment of mutually exclusive nominal ratings to a set of objects by different raters is a very common activity in many fields of research. For example, suppose two deliberately selected raters assign every object in a population of objects to one and only one of r clearly defined nominal categories. The ratings can then be summarized in an r × r contingency table of population proportions, where ${\text{π}}_{ij}$ is the population proportion of objects assigned to category i (i = 1 to r) by Rater 1 and category j (j = 1 to r) by Rater 2, ${\text{π}}_{i+}$ is the population proportion of objects assigned to category i (i = 1 to r) by Rater 1, and ${\text{π}}_{+i}$ is the population proportion of objects assigned to category i (i = 1 to r) by Rater 2.

If the population of objects is large or if the rating process is time-consuming or costly, it may be necessary to obtain a random sample of n objects from the population and have each rater classify the sample of n objects. When two raters each classify a random sample of n objects, the resulting classifications can be summarized into an r × r table of observed frequency counts, where $f_{ij}$ is the number of sample objects assigned to category i (i = 1 to r) by Rater 1 and category j (j = 1 to r) by Rater 2, $f_{i+}$ is the number of sample objects assigned to category i (i = 1 to r) by Rater 1, and $f_{+i}$ is the number of sample objects assigned to category i (i = 1 to r) by Rater 2. The observed frequency counts can be used to estimate the population proportions in the r × r contingency table. Specifically, the maximum likelihood estimates of ${\text{π}}_{ij}$ , ${\text{π}}_{i+}$ , and ${\text{π}}_{+j}$ are $f_{ij}/n$ , $f_{i+}/n$ , and $f_{+i}/n$ , respectively.

In the two-rater design, one measure of agreement is the proportion of objects that are assigned to the same nominal category by both raters. This proportion is ${\text{π}}_A={\displaystyle \sum }_{i=1}^r{\text{π}}_{ii}$ and its maximum likelihood estimate is ${\widehat{\text{π}}}_A={\displaystyle \sum }_{i=1}^r{f}_{ii}/n$ .

2. Chance-Corrected Measures of Agreement

As a measure of agreement, ${\text{π}}_A$ exaggerates the degree of agreement between two raters because ${\text{π}}_{ii}$ can be greater than 0 even if both raters classify the objects in a purely random manner. A general chance-corrected measure of agreement (Scott, 1955) is defined as

$\kappa =\frac{{\text{π}}_A-{\text{ π}}_R}{1-{\text{ π}}_R},$

1

where ${\text{π}}_R$ is the proportion of agreements that would be expected if the two raters had assigned the classifications independently and at random. The maximum value of $\text{κ}$ is 1, representing perfect agreement, and the minimum value of $\text{κ}$ is $-{\text{π}}_R/\left(1 - {\text{π}}_R\right)$ .

Different definitions of ${\text{π}}_R$ define different chance-corrected measures of agreement. Scott (1955) sets ${\text{π}}_R={\displaystyle \sum }_{i=1}^r{\text{π}}_i{ }^2$ , Bennett et al. (1954) set ${\text{π}}_R$ = 1/r, and Gwet (2008) sets ${\text{π}}_R$ = ${\displaystyle \sum }_{i=1}^r{\text{π}}_i\left(1-{\text{π}}_i\right)/\left(r-1\right)$ , where ${\text{π}}_i=\left({\text{π}}_{+i}+{\text{π}}_{i+}\right)/2$ . The chance-corrected measure of agreement proposed by Krippendorff (1970) also sets ${\text{π}}_R={\displaystyle \sum }_{i=1}^r{\text{π}}_i{ }^2$ but makes a small-sample adjustment to the estimate of ${\text{π}}_A$ . For r = 2, Scott’s measure of agreement is also called intraclass κ (Shoukri, 2011). The most well-known and widely used chance-corrected measure of agreement in the social sciences is Cohen’s κ (Cohen, 1960), where ${\text{π}}_R={\displaystyle \sum }_{i=1}^r{\text{π}}_{+i}{\text{π}}_{i+}$ .

The population value of Cohen’s κ is denoted here as ${\text{κ}}_c$ . A maximum likelihood estimate of ${\text{κ}}_c$ , denoted as ${\widehat{\text{κ}}}_c$ , is obtained by replacing ${\text{π}}_A$ with ${\widehat{\text{π}}}_A={\displaystyle \sum }_{i=1}^r\frac{f_{ii}}{n}$ and replacing ${\text{π}}_R$ with ${\widehat{\text{π}}}_R=\frac{{{\displaystyle \sum }}_{i=1}^r{f}_{+i}{f}_{i+}}{n^2}$ in Equation 1. An approximate large-sample standard error of ${\widehat{\text{κ}}}_c$ was derived by Fleiss et al. (1969), which can be used to construct a Wald confidence interval for ${\text{κ}}_c$ .

SAS (SAS Institute, 2011) and Stata (StataCorp, 2019) software compute the Wald confidence interval for ${\text{κ}}_c$ . The cohen.kappa function in the psych R package (Revelle, 2022) also computes the Wald confidence interval for ${\text{κ}}_c$ . The Wald confidence interval for Cohen’s κ ( ${\text{κ}}_c$ ) is known to have poor performance characteristics (Blackman & Koval, 2000; Flack, 1987). Although several alternative confidence interval estimation methods for ${\text{κ}}_c$ have been proposed (see Lee & Tu, 1994), bootstrap confidence intervals for ${\text{κ}}_c$ appear to be the best option (Klar et al., 2002). Lee and Tu (1994) found that a sample of at least n = 100 objects must be rated before the Wald confidence interval for ${\text{κ}}_c$ can be expected to perform properly. The simulation results of Klar et al. (2002) suggest that the bootstrap confidence interval for ${\text{κ}}_c$ should not be used for n < 35.

Cohen’s κ is widely used but has several limitations. Feinstein and Cicchetti (1990) explain how ${\widehat{\text{κ}}}_c$ can have a very low value even when ${\widehat{\text{π}}}_A$ is close to 1. For example, with $f_{11}$ = 97, $f_{12}$ = 0, $f_{21}$ = 2, and $f_{22}$ = 1, there is near perfect agreement between the two raters but ${\widehat{\text{κ}}}_c$ = .492. With $f_{11}$ = 80, $f_{12}$ = 20, $f_{21}$ = 0, and $f_{22}$ = 0, there is high agreement between the two raters but ${\widehat{\text{κ}}}_c$ = 0. Warrens (2010) shows that ${\widehat{\text{κ}}}_c$ is paradoxically larger when ${\text{π}}_{i+}\ne {\text{π}}_{+i}$ . For example, ${\widehat{\text{κ}}}_c$ = .40 with $f_{11}$ = 35, $f_{12}$ = 15, $f_{21}$ = 15, and $f_{22}$ = 35, where ${\text{π}}_{i+}= {\text{π}}_{+i}$ , but ${\widehat{\text{κ}}}_c$ paradoxically increases to .45 with $f_{11}$ = 35, $f_{12}$ = 0, $f_{21}$ = 30, and $f_{22}$ = 35, where ${\widehat{\text{π}}}_A$ is unchanged but ${\text{π}}_{i+}\ne {\text{π}}_{+i}$ .

Cohen’s definition of ${\text{π}}_R={\displaystyle \sum }_{i=1}^r{\text{π}}_{+i}{\text{π}}_{i+}$ is perhaps its most serious weakness as a measure of nominal agreement. This definition implies that Rater 1 would assign objects to category i with probability ${\text{π}}_{+i}$ and Rater 2 would assign subjects to category i with probability ${\text{π}}_{i+}$ if both raters were simply guessing. However, ${\text{π}}_R={\displaystyle \sum }_{i=1}^r{\text{π}}_{+i}{\text{π}}_{i+}$ is estimated using sample data, where raters typically are not guessing. For example, if r = 3 and Rater 1 classifies 20% of the objects in category 1, 45% in category 2, and 35% in category 3, Cohen’s κ assumes that these exact same marginal proportions would be obtained if Rater 1 was simply guessing. Green (1981) argued that ${\text{π}}_R={\displaystyle \sum }_{i=1}^r{\text{π}}_{+i}{\text{π}}_{i+}$ could be justified in the arguably unrealistic situation where a rater would “guess at a rate equivalent to the proportion of time they have determined the presence of a characteristic when they were not guessing.” Except for the chance-corrected measure of agreement proposed by Bennett et al. (1954), all of the other measures of agreement also use an estimate of ${\text{π}}_R$ in situations where raters are typically not guessing. Furthermore, estimating ${\text{π}}_R={\displaystyle \sum }_{i=1}^r{\text{π}}_{+i}{\text{π}}_{i+}$ introduces another source of sampling variability into ${\widehat{\text{κ}}}_c$ , which degrades the small-sample performance of the Wald confidence interval for ${\text{κ}}_c$ .

For r = 2, Byrt et al. (1993) define prevalence as ${\text{π}}_{11}- {\text{π}}_{22}$ and bias as ${\text{π}}_{i+}-{\text{π}}_{+i}$ and then show that a bias adjustment to ${\text{κ}}_c$ is equal to the intraclass κ. They also show that a bias adjustment combined with a prevalence adjustment to ${\text{κ}}_c$ is equal to the Bennett–Alpert–Goldstein index. For r = 2, Blackman and Koval (1993) show that intraclass κ is a large-sample approximation to an intraclass reliability of a single rater from a one-way analysis of variance (ANOVA), while Cohen’s κ is a large-sample approximation to an intraclass reliability of a single rater from a two-way ANOVA. In an interrater reliability study, the intraclass reliability coefficient describes the reliability of a single rater assuming parallel measurements (McDonald, 1999). Parallel measurements are assumed to be homoscedastic, and this assumption is violated in the case of r = 2 if ${\text{π}}_{i+}\left(1-{\text{π}}_{i+}\right)\ne {\text{ π}}_{+i}\left(1-{\text{π}}_{+i}\right)$ . Although Cohen’s κ is arguably an inappropriate measure of interrater agreement, it is an appropriate measure of interrater reliability in the special case of r = 2 and homoscedasticity. Maclure and Willett (1987) argue that ${\widehat{\text{κ}}}_c$ should not be used with r > 2.

Some of the controversy and debate regarding interrater agreement stems from a failure to distinguish between interrater agreement and interrater reliability (Kottner & Stiener, 2011). One of the claimed limitations of ${\text{κ}}_c$ is the attenuation that occurs in a 2 × 2 table when ${\text{π}}_{+1}$ or ${\text{π}}_{1+}$ is close to 1 or 0. Although this attenuation of ${\widehat{\text{κ}}}_c$ is an inappropriate characteristic for a measure of interrater agreement, it is a perfectly appropriate characteristic for a measure of interrater reliability because reliability cannot be large if there is little variability in the ratings, and the variability of the dichotomous ratings will be small when ${\widehat{\text{π}}}_{+1}$ or ${\widehat{\text{π}}}_{1+}$ is close to 1 or 0. The problem of assessing interrater agreement rather than interrater reliability is addressed here.

3. G-index for Two Raters

As noted above, the chance-corrected measure of agreement proposed by Bennett et al. (1954) sets ${\text{π}}_R$ = 1/r. This value for ${\text{π}}_R$ assumes that two independent raters, if they were simply guessing, would select one of the r categories with probability 1/r, so that the joint probabilities in the r × r contingency table under random and independent ratings is 1/r ². The sum of the r joint agreement probabilities gives ${\text{π}}_R$ = 1/r. This definition of ${\text{π}}_R$ is a sensible assumption for random nominal-scale classifications and is consistent with signal detection theory (Wickens, 2002, p. 95) when choosing among r alternatives under a pure noise condition. Setting ${\text{π}}_R$ = 1/r is also consistent with the conceptualization of chance agreement proposed by Lawlis and Lu (1972), Maxwell (1977), and Grove et al. (1981). Hayes and Krippendorff (2007) criticize using ${\text{π}}_R$ = 1/r. They argue that any category that is unused by both raters will “inflate” the Bennet–Alpert–Goldstein coefficient but not Cohen’s κ. However, a category that is unused by both raters indicates perfect agreement for that category and it is appropriate that the Bennet–Alpert–Goldstein coefficient reflects this agreement.

The population chance-corrected measure of agreement proposed by Bennett et al. (1954) and Brennan and Prediger (1981), and referred to by Holley and Guilford (1964) as a G-index of agreement, will be denoted here as ${\text{κ}}_G$ . Setting ${\text{π}}_R$ = 1/r in Equation 1 gives

${\text{κ}}_G=\frac{{\text{π}}_A-1/r}{1-1/r}==(r{\text{π}}_A -1)/\left(r-1\right),$

2

and has a possible range of −1/(r – 1) to 1. The maximum likelihood estimate of ${\text{κ}}_G$ is

${\widehat{\kappa }}_G=(r{\widehat{\text{π}}}_A -1)/\left(r-1\right),$

3

where ${\widehat{\text{π}}}_A={\displaystyle \sum }_{i=1}^r{f}_{ii}/n$ is the unbiased maximum likelihood estimate of ${\text{π}}_A$ . The estimator of ${\text{κ}}_G$ is an unbiased maximum likelihood estimator because it is a linear function of the unbiased maximum likelihood estimator of ${\text{π}}_A$ . An approximate standard error for ${\widehat{\text{κ}}}_G$ given below

$\text{SE(}{\widehat{\kappa }}_G)=\left[r/\left(r-1\right)\right]\sqrt{{\widehat{\text{π}}}_A\left(1-{\widehat{\text{π}}}_A\right)/n},$

4

which is a linear function of the standard error of ${\widehat{\text{π}}}_A$ . In the above example where $f_{11}$ = 97, $f_{12}$ = 0, $f_{21}$ = 2, and $f_{22}$ = 1, the estimate of ${\text{κ}}_G$ (Equation 3) is .96 and appropriately describes the near perfect agreement between the two raters.

The following 100(1 − $\text{α}$ )% adjusted Wald confidence interval for ${\text{κ}}_G$ is proposed here

$\left[r/\left(r-1\right)\right][{\widehat{\text{π}}}_A^{\text{*}}\pm z_{\alpha /2}\sqrt{{\widehat{\text{π}}}_A^{\text{*}}\left(1-{\widehat{\text{π}}}_A^{\text{*}}\right)/\left(n+4\right)}]-1/\left(r-1\right),$

5

where ${\widehat{\text{π}}}_A^{*}=\left(f_A+2\right)/\left(n+4\right)$ and f_A = ${\displaystyle \sum }_{i=1}^r{f}_{ii}$ . The ci.qrater R function in the Online Supplementary Material computes Equations 3 –5.

Although a point estimate of ${\text{κ}}_G$ was proposed decades ago (Bennett et al., 1954), little progress has been made in terms of statistical inference for ${\text{κ}}_G$ . For the special case of r = 2, a standard error of ${\widehat{\text{κ}}}_G$ is given in Shoukri (2011) as

$(1-{\widehat{\text{κ}}}_G^2)/n,$

6

which can be algebraically reexpressed in the form of Equation 4. For r = 2, this standard error could be used to compute the following 100(1 − $\text{α}$ )% Wald confidence interval for ${\text{κ}}_G$

${\widehat{\kappa }}_G\pm z_{\text{α}/2}\text{SE}({\widehat{\text{κ}}}_G).$

7

It can be shown that Equation 7 can be expressed in the form of Equation 5, where the interval estimate in brackets is the traditional Wald confidence interval for a population proportion. Agresti and Coull (1998) showed that the traditional Wald confidence interval has poor performance characteristics under realistic conditions and they proposed an adjusted Wald confidence interval. With sample sizes as small as n = 10, the 95% Agresti–Coull confidence interval had an average coverage probability close to .95 and a worst-case coverage probability no less than .92 across the entire range of possible population proportion values (Agresti & Coull, 1998). The interval estimate in brackets of Equation 7 is the Agresti–Coull confidence interval, and hence, Equation 5 inherits all of its performance characteristics because ${\text{κ}}_G$ is a linear function of ${\text{π}}_A$ . The “exact” Clopper–Pearson confidence interval for ${\text{π}}_A$ also could be used in place of the Agresti–Coull confidence interval in Equation 5 (see Section 10). Note that Cohen’s κ cannot be expressed solely in terms of ${\text{π}}_A$ and hence the Agresti–Coull or Clopper–Pearson confidence intervals cannot be used to obtain a confidence interval for Cohen’s κ.

4. Comparing Agreement in Two-Group Designs

Assessing interrater agreement from two independent groups can answer a wide variety of interesting research questions. The two-group design can be experimental or nonexperimental. A two-group nonexperimental design could consist of two types of randomly sampled objects (e.g., male vs. female students) that are classified into r categories by the same two raters. In a two-group experimental design, a single sample of objects is randomly divided into two groups and each group could be rated by different types of raters (e.g., expert vs. novice) or under differing rating conditions (e.g., complete vs. incomplete case files). Methods for comparing two or more independent intraclass κ values have been proposed by Donner et al. (1996) and Donner and Zou (2002).

Let ${\text{κ}}_{Gj}$ denote the population value of the G-index that will be estimated from subpopulation j in a two-group nonexperimental design or from condition j in a two-group experimental design. It is easy to show that the difference ${\text{κ}}_{G1}- {\text{κ}}_{G2}$ can be expressed as $\left[r/\left(r-1\right)\right]({\text{π}}_{A1}-{\text{π}}_{A2})$ , where ${\text{π}}_{Aj}$ is the population proportion of agreements that will be estimated in group j. The following 100(1 − $\text{α}$ )% adjusted Wald confidence interval for ${\text{κ}}_{G1}- {\text{κ}}_{G2}$ is proposed here

$\left[r/\left(r-1\right)\right]\left[{\widehat{\text{π}}}_{A1}^{\text{*}}-{\widehat{\text{π}}}_{A2}^{\text{*}}\pm z_{\text{α}/2}\sqrt{\frac{{\widehat{\text{π}}}_{A1}^{\text{*}}\left(1-{\widehat{\text{π}}}_{A1}^{\text{*}}\right)}{n_1+2}+\frac{{\widehat{\text{π}}}_{A2}^{\text{*}}\left(1-{\widehat{\text{π}}}_{A2}^{\text{*}}\right)}{n_2+2}}\right],$

8

where ${\widehat{\text{π}}}_{Aj}^{*}=\left(f_{Aj}+1\right)/\left(n_j+2\right)$ . Note that the confidence interval in brackets is the adjusted Wald confidence interval for a difference between two independent proportions developed by Agresti and Caffo (2000), which has been shown to have excellent small-sample properties under a wide range of conditions. With sample sizes as small as 10 per group, the 95% Agresti–Caffo confidence interval had an average coverage probability close to .95 and a worst-case coverage probability no less than .92 in the 10,000 conditions they considered (Agresti & Caffo, 2000). The ci.diff R function in the Online Supplementary Material computes Equation 8.

5. Multistudy Designs

A small sample of objects might be necessary if the ratings are costly or time-consuming. However, a confidence interval for ${\text{κ}}_G$ might be uselessly wide if the sample size is too small. One way to obtain a more accurate estimate of ${\text{κ}}_G$ is to statistically combine agreement estimates from two or more independent studies. Combining parameter estimates from two or more studies is called a meta-analysis, and Vacha-Haase (1998) referred to a meta-analysis of reliability estimates as reliability generalization. Bonett (2010) developed statistical methods for combining and comparing Cronbach’s α reliability coefficients from two or more studies. The logic and rationale for reliability generalization also apply to interrater agreement with the goal of obtaining a more precise and generalizable estimate of agreement and also to assess the degree to which an agreement index might vary across different types of raters, different rating conditions, or different types of rated objects.

Sun (2011) showed how the classical fixed-effect and random-effects meta-analysis methods (see Borenstein et al., 2009) can be used to obtain a confidence interval for an average of two or more population Cohen κ values. The varying-coefficient meta-analysis method (Bonett & Price, 2015) is used here to obtain a confidence interval for an average of G-index values from multiple populations. The varying-coefficient model does not require the unrealistic assumptions of the classical fixed-effect and random-effects meta-analysis methods. Unlike the traditional fixed-effect model, the varying-coefficient model does not assume effect-size homogeneity, and unlike the random-effects model, the varying-coefficient model does not assume randomly selected studies or effect size standard errors that are uncorrelated with the effect sizes (see Bonett & Price, 2015, for more details).

Let ${\text{κ}}_{Gj}$ represent the population G-index value that has been estimated in study j (j = 1 to m). The following 100(1 − $\text{α}$ )% adjusted Wald confidence interval for ${\displaystyle \sum }_{j=1}^m{\text{κ}}_{Gj}/m$ is proposed here

${\displaystyle \sum }_{j=1}^m{\widehat{\text{κ}}}_{Gj}^{\text{ *}}/m\pm z_{\text{α}/2}\sqrt{{\displaystyle \sum }_{j=1}^m\text{SE}{\left({\widehat{\text{κ}}}_{Gj}^{\text{ *}}\right)}^2/m^2},$

9

where ${\widehat{\text{κ}}}_{Gj}^{\text{ *}}=(r{\widehat{\text{π}}}_{Aj}^{\text{*}}-1)/\left(r-1\right)$ , $\text{SE}\left({\widehat{\text{κ}}}_{Gj}^{\text{ *}}\right)=\left[r/\left(r-1\right)\right]\sqrt{\frac{{\widehat{\text{π}}}_{Aj}^{\text{*}}\left(1-{\widehat{\text{π }}}_{Aj}^{\text{*}}\right)}{n_j+4/m}}$ , and ${\widehat{\text{π}}}_{Aj}^{\text{*}}=\left(f_{Aj}+2/m\right)/\left(n_j+4/m\right)$ . Note that ${\displaystyle \sum }_{j=1}^m{\text{κ}}_{Gj}/m=\left[r/\left(r-1\right)\right]{\displaystyle \sum }_{j=1}^m{\text{π}}_{Aj}/m-m/\left(r-1\right)$ , and hence, the end points of Equation 9 are linear functions of the end points of the adjusted Wald confidence interval proposed by Price and Bonett (2004) for a linear function of independent proportions. The Price–Bonett confidence interval is a generalization of the Agesti–Coull confidence interval and has been shown to have excellent small-sample properties under a wide range of conditions. In meta-analysis applications with m = 5, 15, or 30 studies and sample sizes as small as 10 per group, the 95% Price–Bonett confidence interval had an average coverage probability close to .95 and a worst-case coverage probability no less than .938 in the 13,500 conditions they considered (Price & Bonett, 2004). The ci.meta R function in the Online Supplementary Material computes Equation 9 using the sample sizes and number of agreements from each study.

Important differences in ${\text{κ}}_{Gj}$ values across the m populations could be due to differences in rating conditions, rater characteristics, or characteristics of the rated objects. A linear contrast of ${\text{κ}}_{Gj}$ values can be expressed as ${\displaystyle \sum }_{j=1}^m{h}_j{\text{κ}}_{Gj}$ , where the h_j values are specified by the researcher and ${\displaystyle \sum }_{j=1}^m{h}_j$ = 0. For example, in a meta-analysis of m = 5 comparable interrater agreement studies where the first three studies rated the behavior of high school students and the last two studies rated the behavior of college students, the researcher might want to estimate $\left({\text{κ}}_{G1}+{\text{κ}}_{G2}+{\text{κ}}_{G3}\right)/3-({\text{κ}}_{G4}+{\text{κ}}_{G5})/2$ . This linear contrast can be specified with contrast coefficients h ₁= 1/3, h ₂ = 1/3, h ₃ = 1/3, h ₄ = −1/2, and h ₅ = −1/2.

The following 100(1 − $\text{α}$ )% adjusted Wald confidence interval for ${\displaystyle \sum }_{j=1}^m{h}_j{\text{κ}}_{Gj}$ is proposed here

${\displaystyle \sum }_{j=1}^m{h}_j{\widehat{\text{κ}}}_{Gj}^{\text{ *}}\pm z_{\text{α}/2}\sqrt{{\displaystyle \sum }_{j=1}^m{h}_j^2\text{SE}{\left({\widehat{\text{κ}}}_{Gj}^{\text{ *}}\right)}^2},$

10

where ${\widehat{\text{κ}}}_{Gj}^{ *}$ and $\text{SE}\left({\widehat{\text{κ}}}_{Gj}^{ *}\right)$ are defined above with m set equal to the number of nonzero h_j values as recommended by Price and Bonett (2004). Note that ${\displaystyle \sum }_{j=1}^m{h}_j{\text{κ}}_{Gj}=\left[r/\left(r-1\right)\right]{\displaystyle \sum }_{j=1}^m{h}_j{\text{π}}_{Aj}$ , and hence, the end points of Equation 10 are linear functions of the end points of the adjusted Wald confidence interval proposed by Price and Bonett (2004) for a linear function of independent proportions. The Price–Bonett confidence interval has been shown to have excellent performance characteristics. With sample sizes as small as 20 per group, the 95% Price–Bonett confidence interval had an average coverage probability close to .95 and a worst-case coverage probability no less than .920 across 10,000 population proportion values and several different types of linear contrasts (Price & Bonett, 2004). The ci.contrast R function in the Online Supplementary Material computes Equation 10.

6. Multirater Designs

If q different and deliberately selected raters each classify a random sample of objects into r categories, the results can be summarized in a r^q contingency table. A G-index of agreement for any two raters can be computed by collapsing the r^q table into an r × r table for the two raters of interest and then applying Equations 3 –5.

A G-index of agreement among all q raters is defined here as

${\text{κ}}_{G\left(q\right)}=\frac{{\text{π}}_A-{\text{ π}}_R}{1-{\text{ π}}_R}=\frac{{\text{π}}_A-1/r^{q-1}}{1-1/r^{q-1}}=(r^{q-1}{\text{π}}_A-1)/(r^{q-1}-1),$

11

where ${\text{π}}_A={\displaystyle \sum }_{i=1}^r{\text{π}}_{ii\dots i}$ is the probability of unanimous agreement among the q raters and ${\text{π}}_{ii\dots i}$ is the probability of unanimous agreement among the q raters for category i. Assuming random and independent ratings, the joint probability in the r^q contingency table is $\frac{1}{r^q}$ , so that the probability of random agreement in any of the r categories is ${\text{π}}_R={\displaystyle \sum }_{i=1}^{r}1/r^q=1/r^{q-1}$ . Note that ${\text{π}}_A$ satisfies the definition of agreement among multiple raters given by Hubert (1977).

The maximum likelihood estimate of ${\text{κ}}_{G\left(q\right)}$ is

${\widehat{\text{κ}}}_{G\left(q\right)}=(r^{q-1}{\widehat{\text{π}}}_A-1)/(r^{q-1}-1),$

12

where ${\widehat{\text{π}}}_A={\displaystyle \sum }_{i=1}^r{f}_{ii\dots i}/n$ is the maximum likelihood estimate of ${\text{π}}_A$ . The approximate standard error for ${\widehat{\text{κ}}}_{G\left(q\right)}$ given below is a function of the variance of ${\widehat{\text{π}}}_A$

${\text{SE}(\widehat{\text{κ}}}_{G\left(q\right)})=[r^{q-1}/(r^{q-1}-1)]\sqrt{{\widehat{\text{π}}}_A\left(1-{\widehat{\text{π}}}_A\right)/n}.$

13

The following 100(1 − $\text{α}$ )% adjusted Wald confidence interval for ${\text{κ}}_{G\left(q\right)}$ is proposed here

$[r^{q-1}/(r^{q-1}-1)][{\widehat{\text{π}}}_A^{\text{*}}\pm z_{\alpha /2}\sqrt{{\widehat{\text{π}}}_A^{\text{*}}\left(1-{\widehat{\text{π}}}_A^{\text{*}}\right)/\left(n+4\right)}]-1/(r^{q-1}-1),$

14

where ${\widehat{\text{π}}}_A^{*}=\left(f_A+2\right)/\left(n+4\right)$ and f_A = ${\displaystyle \sum }_{i=1}^r{f}_{ii\dots i}$ . Like Equation 5, Equation 14 uses the Agresti–Coull adjusted Wald confidence interval for a single population proportion. The ci.qrater R function in the Online Supplementary Material computes Equation 14.

6.1. Three-Rater Design (r = 2)

Consider a q = 3 rater design with r = 2. The three pairwise G-indices are denoted as ${\text{κ}}_{G\left\{1,2\right\}}$ , ${\text{κ}}_{G\left\{1,3\right\}}$ , and ${\text{κ}}_{G\left\{2,3\right\}}$ . The average of all pairwise measures of agreement is an alternative multirater measure of agreement proposed by Hubert (1977). It can be shown for the special case of q = 3, using straightforward but tedious algebra, that ${\text{κ}}_{G\left(3\right)} =({\text{κ}}_{G\left\{1,2\right\}}+{\text{κ}}_{G\left\{1,3\right\}}+{\text{κ}}_{G\left\{2,3\right\}})/3$ .

In some applications, it will be informative to compare ${\text{κ}}_{G\left\{1,2\right\}}$ , ${\text{κ}}_{G\left\{1,3\right\}}$ , and ${\text{κ}}_{G\left\{2,3\right\}}$ . For example, if Raters 1 and 2 are novices and Rater 3 is an expert, confidence intervals for ${\text{κ}}_{G\left\{1,2\right\}}-{\text{κ}}_{G\left\{1,3\right\}}$ , ${\text{κ}}_{G\left\{1,2\right\}}-{\text{κ}}_{G\left\{2,3\right\}}$ , and ${\text{κ}}_{G\left\{1,3\right\}}-{\text{κ}}_{G\left\{2,3\right\}}$ will provide information regarding the direction and the magnitude of these pairwise differences. The three G-indices in a three-rater design with r = 2 can be expressed as

${\text{κ}}_{G\left\{1,2\right\}}=2\left({\text{π}}_{111}+{\text{ π}}_{112}+{\text{π}}_{221}+{\text{π}}_{222}\right)-1,$

15

${\text{κ}}_{G\left\{1,3\right\}}=2\left({\text{π}}_{111}+ {\text{π}}_{121}+{\text{π}}_{212}+{\text{π}}_{222}\right)-1,$

16

${\text{κ}}_{G\left\{2,3\right\}}=2\left({\text{π}}_{111}+ {\text{π}}_{211}+{\text{π}}_{122}+{\text{π}}_{222}\right)-1,$

17

and it follows that the pairwise differences in these G-indices can be expressed as

${\text{κ}}_{G\left\{1,2\right\}}-{\text{κ}}_{G\left\{1,3\right\}}=2\left({\text{π}}_{112}+{\text{π}}_{221}\right)-2({\text{π}}_{121}+{\text{π}}_{212}),$

18

${\text{κ}}_{G\left\{1,2\right\}}-{\text{κ}}_{G\left\{2,3\right\}}=2\left({\text{π}}_{112}+{\text{π}}_{221}\right)-2({\text{π}}_{211}+{\text{π}}_{122}),$

19

${\text{κ}}_{G\left\{1,3\right\}}-{\text{κ}}_{G\left\{2,3\right\}}=2\left({\text{π}}_{121}+{\text{π}}_{212}\right)-2({\text{π}}_{211}+{\text{π}}_{122}).$

20

Maximum likelihood estimates of ${\text{κ}}_{G\left\{1,2\right\}}$ , ${\text{κ}}_{G\left\{1,3\right\}}$ , and ${\text{κ}}_{G\left\{2,3\right\}}$ and the pairwise differences in G-indices are obtained by replacing ${\text{π}}_{ijk}$ in Equations 15 –20 with the maximum likelihood estimate ${\widehat{\text{π}}}_{ijk}$ = $f_{ijk}/n$ .

An approximate standard error of each pairwise difference is derived by first collapsing the 2³ table of multinomial proportions into three mutually exclusive categories. For example, from Equation 18, the three categories needed to assess ${\text{κ}}_{G\left\{1,2\right\}}-{\text{κ}}_{G\left\{1,3\right\}}$ have probabilities of ${\text{π}}_1={\text{π}}_{112}+{\text{π}}_{221}$ , ${\text{π}}_2={\text{π}}_{121}+{\text{π}}_{212}$ , and ${\text{π}}_3=1-({\text{π}}_1+{\text{π}}_2)$ . Using the variances and covariances of a multinomial distribution (Bishop et al., 1975, p. 442), an approximate standard error of ${\widehat{\text{κ}}}_{G\left\{1,2\right\}}-{\widehat{\text{κ}}}_{G\left\{1,3\right\}}$ is

${\text{SE}(\widehat{\text{κ}}}_{G\left\{1,2\right\}}-{\widehat{\text{κ}}}_{G\left\{1,3\right\}})=\sqrt{4\left\{{\widehat{\text{π}}}_1+{\widehat{\text{π}}}_2-{\left({\widehat{\text{π}}}_1-{\widehat{\text{π}}}_2\right)}^2\right\}/n},$

21

where ${\widehat{\text{π}}}_1=(f_{112}+f_{221})/n$ and ${\widehat{\text{π}}}_2=(f_{121}+f_{212})/n$ . An approximate standard error of ${\widehat{\text{κ}}}_{G\left\{1,2\right\}}-{\widehat{\text{κ}}}_{G\left\{2,3\right\}}$ is given by Equation 21 with ${\widehat{\text{π}}}_1=(f_{112}+f_{221})/n$ and ${\widehat{\text{π}}}_2=(f_{211}+f_{122})/n$ , and an approximate standard error of ${\widehat{\text{κ}}}_{G\left\{1,3\right\}}-{\widehat{\text{κ}}}_{G\left\{2,3\right\}}$ is given by Equation 21 with ${\widehat{\text{π}}}_1=(f_{121}+f_{212})/n$ and ${\widehat{\text{π}}}_2=(f_{211}+f_{122})/n$ .

A 100(1 − $\text{α}$ )% adjusted Wald confidence interval for ${\text{κ}}_{G\left\{1,2\right\}}-{\text{κ}}_{G\left\{1,3\right\}}$ is

$2[{\widehat{\text{π}}}_1^{\text{*}}-{\widehat{\text{π}}}_2^{\text{*}}\pm z_{\text{α}/2}\sqrt{\left\{{\widehat{\text{π}}}_1^{\text{*}}+{\widehat{\text{π}}}_2^{\text{*}}-{\left({\widehat{\text{π}}}_1^{\text{*}}-{\widehat{\text{π}}}_2^{\text{*}}\right)}^2\right\}/\left(n+2\right)}],$

22

where ${\widehat{\text{π}}}_1^{\text{*}}=(f_{112}+f_{221}+1)/\left(n+2\right)$ and ${\widehat{\text{π}}}_2^{\text{*}}=(f_{121}+f_{212}+1)/\left(n+2\right)$ . The confidence interval for ${\text{κ}}_{G\left\{1,2\right\}}-{\text{κ}}_{G\left\{2,3\right\}}$ is given by Equation 22 with ${\widehat{\text{π}}}_1^{\text{*}}=(f_{112}+f_{221}+1)/\left(n+2\right)$ and ${\widehat{\text{π}}}_2^{\text{*}}=(f_{121}+f_{212}+1)/\left(n+2\right)$ , and the confidence interval for ${\text{κ}}_{G\left\{1,2\right\}}-{\text{κ}}_{G\left\{2,3\right\}}$ is given by Equation 22 with ${\widehat{\text{π}}}_1^{\text{*}}=(f_{121}+f_{212}+1)/\left(n+2\right)$ and ${\widehat{\text{π}}}_2^{\text{*}}=(f_{211}+f_{122}+1)/\left(n+2\right)$ . The adjusted Wald confidence interval in brackets of Equation 22 was developed by Bonett and Price (2012) and was shown to have excellent small-sample properties under a wide range of conditions. With sample sizes as small as n = 15, the 95% Bonett–Price confidence interval had an average coverage probability that was slightly greater than .95 and a worst-case coverage probability no less than .910 in the 25,000 conditions they considered (Bonett & Price, 2012). The agree.3rater R function in the Online Supplementary Material computes Equation 22 for all three pairs of raters and also computes Equation 14 for ${\text{κ}}_{G\left(3\right)}$ .

6.2. Four-Rater Design (r = 2)

The case of q = 4 raters with r = 2 has received special attention in the literature (Banerjee et al., 1999; Donner et al., 2000; McKenzie et al., 1996; Williamson & Manatunga, 1997) because some of the research questions that can be answered using the two-group design described previously might be answered more economically using a one-group design with four raters. The four-rater design can be used to compare the agreement between two different types of raters, such as two novice raters and two expert raters or two male raters and two female raters.

A G-index for any two raters in a four-rater design can be computed by collapsing the 2⁴ table into a 2 × 2 table for any two raters of interest and then applying Equations 3 –5. The focus here will be the comparison of agreement between the first two raters ( ${\text{κ}}_{G\left\{1,2\right\}}$ ) with the last two raters ( ${\text{κ}}_{G\left\{3,4\right\}}$ ). These two G-indices in a four-rater design can be expressed as

${\text{κ}}_{G\left\{1,2\right\}}=2\left({\text{π}}_{1111}+{\text{ π}}_{1121}+{\text{π}}_{1112}+{\text{π}}_{1122}+{\text{π}}_{2211}+{\text{ π}}_{2221}+{\text{π}}_{2212}+{\text{π}}_{2222}\right)-1,$

23

${\text{κ}}_{G\left\{3,4\right\}}=2\left({\text{π}}_{1111}+{\text{ π}}_{1211}+{\text{π}}_{2111}+{\text{π}}_{2211}+{\text{ π}}_{1122}+{\text{ π}}_{1222}+{\text{π}}_{2122}+{\text{π}}_{2222}\right)-1,$

24

and it follows that the difference between these two G-indices can be expressed as

${\text{κ}}_{G\left\{1,2\right\}}-{\text{κ}}_{G\left\{3,4\right\}}=2\left({\text{π}}_1-{\text{ π}}_2\right),$

25

where ${\text{π}}_1$ = ${\text{π}}_{1121}$ + ${\text{π}}_{1112}$ + ${\text{π}}_{2221}$ + ${\text{π}}_{2212}$ and ${\text{π}}_2$ = ${\text{π}}_{1211}$ + ${\text{π}}_{2111}$ + ${\text{π}}_{1222}$ + ${\text{π}}_{2122}$ . The maximum likelihood estimate of ${\text{κ}}_{G\left\{1,2\right\}}$ − ${\text{κ}}_{G\left\{3,4\right\}}$ is obtained by replacing the population proportions in Equations 23 and 24 with their maximum likelihood estimates.

Applying the same approach used to derive Equation 22 gives the following approximate standard error of ${\widehat{\text{κ}}}_{G\left\{1,2\right\}}$ – ${\widehat{\text{κ}}}_{G\left\{3,4\right\}}$

${\text{SE}(\text{κ}^}_{G\left\{1,2\right\}}-{\widehat{\text{κ}}}_{G\left\{3,4\right\}})=\sqrt{4\left\{{\widehat{\text{π}}}_1+{\widehat{\text{π}}}_2-{\left({\widehat{\text{π}}}_1-{\widehat{\text{π}}}_2\right)}^2\right\}/n},$

26

and the following 100(1 − $\text{α}$ )% adjusted Wald confidence interval for ${\text{κ}}_{G\left\{1,2\right\}}$ − ${\text{κ}}_{G\left\{3,4\right\}}$

$2[{\widehat{\text{π}}}_2^{\text{*}}-{\widehat{\text{π}}}_2^{\text{*}}\pm z_{\text{α}/2}\sqrt{\left\{{\widehat{\text{π}}}_1^{\text{*}}+{\widehat{\text{π}}}_2^{\text{*}}-{\left({\widehat{\text{π}}}_1^{\text{*}}-{\widehat{\text{π}}}_2^{\text{*}}\right)}^2\right\}/\left(n+2\right)}],$

27

where ${\widehat{\text{π}}}_{\text{1}}$ and ${\widehat{\text{π}}}_{\text{2}}$ are the maximum-likelihood estimates, ${\widehat{\text{π}}}_1^{\text{*}}=(f_{1121}$ + $f_{1112}$ + $f_{2221}$ + $f_{2212}+1)/\left(n+2\right)$ and ${\widehat{\text{π}}}_{\text{2}}^{*}=(f_{1211}$ + $f_{2111}$ + $f_{1222}$ + $f_{2122}+1)/\left(n+2\right)$ . Note that the adjusted Wald confidence interval in brackets is a Bonett–Price confidence described in Equation 22. The ci.4rater R function in the Online Supplementary Material computes the maximum likelihood estimate and confidence interval for ${\text{κ}}_{G\left\{1,2\right\}}$ − ${\text{κ}}_{G\left\{3,4\right\}}$ requiring only the sample size, $f_1$ = $f_{1121}$ + $f_{1112}$ + $f_{2221}$ + $f_{2212}$ , and $f_2=f_{1211}$ + $f_{2111}$ + $f_{1222}$ + $f_{2122}$ as input.

The above results for q = 3 and q = 4 raters can be applied to designs with q $\ge$ 4 raters by collapsing a 2 ^q table into a 2³ table if the comparison involves three raters or a 2⁴ table if the comparison involves four raters. For example, with q = 5, an estimate and confidence interval for ${\text{κ}}_{G\left\{1,3\right\}}$ − ${\text{κ}}_{G\left\{3,4\right\}}$ is computed from a 2³ table for Raters 1, 3, and 4, and an estimate and confidence interval for ${\text{κ}}_{G\left\{1,3\right\}}$ − ${\text{κ}}_{G\left\{4,5\right\}}$ is computed from a 2⁴ table for Raters 1, 3, 4, and 5.

7. Benchmark G-Index Values

When reporting the numerical results for a measure of agreement, it is common to also provide a verbal description of the strength of agreement. Altman (1991), Fleiss et al. (2003), and Landis and Koch (1977) have each suggested their own benchmark verbal descriptions for different point estimates of Cohen’s κ. Landis and Koch (1977) suggested that a Cohen κ value within one of the following six ranges < 0, 0–.20, .21–.40, .41–.60, .61–.80, or .81–1.0 represents a “poor,” “slight,” “fair,” “moderate,” “substantial,” or “almost perfect” agreement, respectively. The Landis–Koch benchmarks are frequently used to describe a point estimate of an agreement index. This practice is misleading because point estimates contain a sampling error of unknown magnitude and direction. Benchmark descriptions should be applied to an interval estimate rather than a point estimate. Although the Landis–Koch benchmark scale is appealing because of its six very specific descriptive categories, a confidence interval is likely to cover two or more of these categories unless n is large. Fleiss et al. (2003) proposed a three category scale for Cohen’s κ where values below .4 represent “poor” agreement, values between .40 and .75 represent “good” agreement, and values greater than .75 represent “excellent” agreement. Some researchers might find the Fleiss scale to be too crude for their purposes.

A four category benchmark scale is proposed here for the G-index. A G-index value within one of the four ranges < .25, .25–.50, .51–.75, and .76–1.0 could be described as “poor,” “fair,” “good,” and “excellent,” respectively. A confidence interval is less likely to include more than two descriptive categories in a four category scale than a six category scale. For example, a confidence interval of [.581, .824] would be described as “moderate, substantial, or almost perfect agreement” using the Landis–Koch scale and would be described as “good or excellent agreement” using the proposed scale.

8. Sample-Size Planning

Sample-size planning is one of the most important components in the design of an interrater agreement study. If the number of objects sampled is too small, the confidence interval for the population agreement index could be uselessly wide. Several methods to approximate the sample-size requirement when assessing Cohen’s κ have been proposed (Bujang & Baharun, 2017; Cantor, 1996; Donner & Eliasziw, 1992; Flack et al., 1988). The available sample-size methods for Cohen’s κ are of limited value because they require assumptions about the classification probabilities of each rater in addition to the value for ${\text{κ}}_c$ . Closed-form sample-size formulas are derived here that approximate the required sample size to obtain $100\left(1-\text{α}\right)\%$ confidence intervals for ${\text{κ}}_G$ , ${\text{κ}}_{G1}- {\text{κ}}_{G2},$ and ${\text{κ}}_{G\left(q\right)}$ with desired precision. These closed-form formulas are particularly useful because they do not require assumptions about the classification probabilities of each rater.

Larger sample sizes give narrower confidence intervals, and it is possible to approximate the sample size that will give the desired width (w) of a confidence interval for a specified level of confidence. The sample size needed to obtain a $100\left(1-\text{α}\right)\%$ confidence interval for ${\text{κ}}_G$ (Equation 5) having a desired width (upper limit minus lower limit) equal to w is approximately

$n^{\prime }=4\left({\tilde{\kappa }}_G+\frac{1}{r-1}\right)\left(1-{\tilde{\kappa }}_G\right){\left(z_{\text{α}/2}/w\right)}^2,$

28

where ${\widetilde{\text{κ}}}_G$ is a planning value of ${\text{κ}}_G$ . Equation 28 was derived from Equation 4, where 2 $z_{\text{α/2}}$ SE( ${\widehat{\text{κ}}}_G)$ is the approximate width of Equation 5. Setting this width to w, solving for n, and then replacing ${\widehat{\text{κ}}}_G$ with ${\widetilde{\text{κ}}}_G$ gives Equation 28. A planning value of ${\text{κ}}_G$ is obtained from expert opinion, pilot studies, or previously published research. Setting ${\widetilde{\text{κ}}}_G$ = (r – 2)/[2(r – 1)] maximizes Equation 28 and is useful in applications where no prior information about ${\text{κ}}_G$ is available.

The width of Equation 5 tends to be greater than 2 $z_{\text{α}/2}$ SE( ${\widehat{\text{κ}}}_G)$ , especially in small samples or if ${\widehat{\text{π}}}_A$ is close to 0 or 1, and hence, n′ tends to understate the required sample size. Following the general approach of Bonett and Wright (2000), a more accurate sample-size approximation is

$n=n\text{’(}{w_0/w)}^2,$

29

where $w_0$ is the width of Equation 5 computed using n = n′ and the value of ${\widehat{\text{π}}}_A^{*}$ implied by ${\widetilde{\text{κ}}}_G$ . The size.ci.qrater R function in the Online Supplementary Material computes Equation 28 and then adjusts the result using Equation 29.

The sample size per group needed to obtain a $100\left(1-\text{α}\right)\%$ confidence interval for ${\text{κ}}_{G1}-{\text{κ}}_{G2}$ (Equation 8) in a two-group design having a desired width of w is approximately

$n\text{’}=4v{\left(z_{\text{α}/2}/w\right)}^2,$

30

where v = $\left({\widetilde{\text{κ}}}_{G1}+\frac{1}{r - 1}\right)\left(1- {\widetilde{\text{κ}}}_{G1}\right)+\left({\widetilde{\text{κ}}}_{G2}+\frac{1}{r - 1}\right)\left(1- {\widetilde{\text{κ}}}_{G2}\right)$ and ${\widetilde{\text{κ}}}_{Gj}$ is a planning value of ${\text{κ}}_{Gj}$ . If no prior information is available, ${\widetilde{\text{κ}}}_{G1}$ and ${\widetilde{\text{κ}}}_{G2}$ can be set to (r − 2)/[2(r − 1)], which maximizes Equation 30. Equation 30 was derived by setting 2 $z_{\text{α}/2}\sqrt{\text{SE}{\left({\widehat{\text{κ}}}_{G1}\right)}^2+\text{SE}{\left({\widehat{\text{κ}}}_{G2}\right)}^2}$ equal to w, solving for n, and replacing ${\widehat{\text{κ}}}_{Gj}$ with ${\widetilde{\text{κ}}}_{Gj}$ .

The width of Equation 8 tends to be greater than 2 $z_{\text{α}/2}\sqrt{\text{SE}{\left({\widehat{\text{κ}}}_{G1}\right)}^2+\text{SE}{\left({\widehat{\text{κ}}}_{G2}\right)}^2}$ , especially in small samples or if either ${\widehat{\text{π}}}_{A1}$ or ${\widehat{\text{π}}}_{A2}$ is close to 0 or 1, and hence, $n\text{'}$ tends to understate the required sample size per group. The size.ci.diff R function in the Online Supplementary Material computes Equation 30 and then adjusts the result using Equation 29.

The sample size needed to obtain a $100\left(1-\text{α}\right)\%$ confidence interval for ${\text{κ}}_{G\left(q\right)}$ (Equation 14) having a desired width of w is approximately

$n\text{’}=4\left({\widetilde{\text{κ}}}_{G\left(q\right)}+\frac{1}{r^{q-1}-1}\right)\left(1-{\widetilde{\text{ κ}}}_{G\left(q\right)}\right){\left(z_{\text{α}/2}/w\right)}^2,$

31

where ${\widetilde{\text{κ}}}_{G\left(q\right)}$ is a planning value of ${\text{κ}}_{G\left(q\right)}$ . Equation 31 was derived from Equation 13, where 2 $z_{\text{α/}2}$ SE( ${\widehat{\text{κ}}}_{G\left(q\right)})$ is the approximate width of Equation 14. Setting this width to w, solving for n, and then replacing ${\widehat{\text{κ}}}_{G\left(q\right)}$ with ${\widetilde{\text{κ}}}_{G\left(q\right)}$ gives Equation 31. Setting ${\widetilde{\text{κ}}}_{G\left(q\right)}$ = ( $r^{q-1}$ – 2)/[2( $r^{q-1}$ – 1)] maximizes Equation 31 and is useful in applications where no prior information about ${\text{κ}}_{G\left(q\right)}$ is available.

The width of Equation 14 tends to be greater than 2 $z_{\text{α}/2}$ SE( ${\widehat{\text{κ}}}_{G\left(q\right)})$ , especially in small samples or if ${\widehat{\text{π}}}_A$ is close to 0 or 1, and hence, n′ tends to understate the required sample size. The size.ci.qrater R function in the Online Supplementary Material computes Equation 31 and then adjusts the result using Equation 29.

9. Illustrative Examples

The Online Supplementary Material contains R functions that will compute confidence intervals for (1) ${\text{κ}}_G$ (Equation 5), (2) ${\text{κ}}_{G1}-{\text{κ}}_{G2}$ in two-group designs (Equation 8), (3) ${\displaystyle \sum }_{j=1}^m{\text{κ}}_{Gj}/m$ in meta-analysis applications (Equation 9), (4) a general linear contrast of ${\text{κ}}_{Gj}$ in multiple group designs (Equation 10), (5) ${\text{κ}}_{G\left(q\right)}$ in multirater designs (Equation 14), and (6) pairwise comparisons in three-rater (Equation 22) and four-rater designs (Equation 27). The Online Supplementary Material also contains R functions that will compute the sample-size requirements to (1) estimate ${\text{κ}}_G$ with desired precision (Equation 28), (2) estimate ${\text{κ}}_{G1}-{\text{κ}}_{G2}$ with desired precision (Equation 30), and (3) estimate ${\text{κ}}_{G\left(q\right)}$ with desired precision (Equation 31). The following examples are adapted from the author’s statistical consulting files.

9.1. Two Raters in One-Group Design

Two research assistants classified a random sample of n = 90 open-ended questionnaire responses into r = 3 predetermined categories, and 82 of the 90 responses were classified into the same categories by both research assistants. The command ci.qrater(.05, 90, 82, 3) computes Equations 3 –5 and returns a point estimate for ${\text{κ}}_G$ of .867 with a 95% confidence interval of [.747, .934].

9.2. Two Raters in Two-Group Design

Two school psychologists evaluated a random sample of n ₁ = 75 case files for boys and a random sample of n ₂ = 60 case files for girls. Each psychologist rated each child as having or not having (r = 2) attention deficit hyperactivity disorder symptoms. The psychologist ratings agreed in 70 of the 75 cases for the boys and in 45 of the 60 cases for the girls. The command ci.diff(.05, 75, 70, 60, 45, 2) computes Equation 8 and returns a 95% confidence interval for the difference in agreement for boys and girls of [.112, .609]. This function also returns point estimates of .867 and .500 with 95% confidence intervals of [.697, .948] and [.252, .685] for boys and girls, respectively. In this example where the two ratings are dichotomous, Cohen’s κ and intraclass κ are questionable measures of interrater agreement, but they are valid measures of interrater reliability. To obtain point and interval estimates of Cohen’s κ and intraclass κ, the frequency counts in the two 2 × 2 contingency tables for boys and girls are required. Suppose the frequency counts for boys are $f_{11}$ = 65, $f_{12}$ = 4, $f_{21}$ = 1, and $f_{22}$ = 5. The point estimates are .631 and .630, and the 95% Wald confidence intervals are [.336, .926] and [.331, .928] for Cohen’s κ and intraclass κ, respectively. Suppose the frequency counts for girls are $f_{11}$ = 35, $f_{12}$ = 8, $f_{21}$ = 7, and $f_{22}$ = 10. The point estimates are .395 and .395, and the 95% Wald confidence intervals are [.142, .649] and [.141, .649] for Cohen’s κ and intraclass κ, respectively.

9.3. Meta-Analysis of Two-Rater Studies

Suppose two published studies used two raters to assess interrater agreement for the absence or presence (r = 2) of gender stereotype behavior in educational children’s TV shows. The first published study used a random sample of n ₁ = 50 episodes and reported agreement in 41 of the 50 episodes. The second published study used a random sample of n ₂ = 70 episodes and reported agreement in 58 of the 70 episodes. The three commands f = c(41, 58), n = c(50, 70), and ci.meta(.05, f, n, 2) computes Equation 9 and returns a point estimate for ( ${\text{κ}}_{G1}+{\text{κ}}_{G2})/2$ of .648 with a 95% confidence interval of [.488, .766].

9.4. Linear Contrast in a Three-Study Design

Suppose the above meta-analysis included a third published study that sampled episodes of noneducational children’s TV shows. The third study used a random sample of n ₃ = 90 episodes and reported agreement in 85 of the 90 episodes. The four commands f = c(41, 58, 85), n = c(50, 70, 90), h = c(−.5, −.5, 1), and ci.contrast(.05, f, n, h, 2) compute Equation 10 and return a point estimate for ${\text{κ}}_{G3}$ – ( ${\text{κ}}_{G1}+{\text{κ}}_{G2})/2$ of .240 with a 95% confidence interval of [.071, .412]. This result suggests that interrater agreement is greater for noneducational than educational TV shows.

9.5. Four-Rater Design

Four parole officers (q = 4) evaluated the parole application files of 100 prisoners, and there was a unanimous grant or deny (r = 2) agreement in 87 of the 100 files. The command ci.qrater(.05, 100, 87, 4, 2) computes Equations 12–14 and returns a point estimate for ${\text{κ}}_{G\left(4\right)}$ of .851 with a 95% confidence interval of [.758, .912].

9.6. Three-Rater Design

A school psychologist (Rater 1), a teacher (Rater 2), and a principal (Rater 3) evaluated a sample of 300 high school students with disciplinary problems and gave a suspension or a nonsuspension recommendation (r = 2) for each student. The ci.3rater function requires a vector of the eight frequency counts in the 2³ contingency table (see comments in the ci.3rater function about how to order the frequencies). In this example, suppose the frequency counts are 100, 6, 4, 40, 20, 1, 9, and 120. The two commands f = c(100, 6, 4, 40, 20, 1, 9, 120) and ci.3rater(.05, f) compute Equations 21 and 22 and return 95% confidence intervals for ${\text{κ}}_{G\left\{1,2\right\}}$ − ${\text{κ}}_{G\left\{1,3\right\}}$ , ${\text{κ}}_{G\left\{1,2\right\}}$ − ${\text{κ}}_{G\left\{2,3\right\}}$ , and ${\text{κ}}_{G\left\{1,3\right\}}$ − ${\text{κ}}_{G\left\{2,3\right\}}$ of [.006, .127], [−.407, −.189], and [−.462, −.266], respectively. These results indicate that the agreement between the school psychologist and teacher is greater than the agreement between the school psychologist and principal. The results also indicate that the agreement between the teacher and the principal is greater than the agreement between the school psychologist and teacher as well as the agreement between the school psychologist and principal.

9.7. Comparison of Two-Rater Agreement in Four-Rater Design

Two graduate students (Raters 1 and 2) and two undergraduate students (Raters 3 and 4) were trained to code open-ended responses for the absence or presence (r = 2) of a particular ideological theme in newspaper articles. Suppose these four raters coded 300 articles and frequency counts of f ₁ = 78 and f ₂ = 52 are extracted from the 2⁴ table (see Equation 27 for definition of f ₁ and f ₂). The command ci.4rater(.05, 300, 78, 52) computes Equations 26 and 27 and returns a point estimate ${\text{κ}}_{G\left\{1,2\right\}}$ − ${\text{κ}}_{G\left\{3,4\right\}}$ of .173 with a 95% confidence interval of [.024, .320].

9.8. Sample Size Requirements for Two-Rater and Three-Rater Designs

A proposed study will use two master teachers (q = 2) to provide dichotomous ratings (meets expectations or needs improvement) for a sample of student teachers based on classroom observations. Setting ${\widetilde{\text{κ}}}_G$ = .90, r = 2, $\text{α}$ = .05, and a desired confidence interval width of .25, size.qrater(.05, .9, 2, .25, 2) computes Equations 28 and 29 and returns a sample-size requirement of 71 student teachers to be rated by two master teachers. The researcher is also considering using three master teachers to rate each student. Setting ${\widetilde{\text{κ}}}_G$ = .90, r = 2, $\text{α}$ = .05, and a desired confidence interval width of .25, size.qrater(.05, .9, 2, .25, 3) computes Equations 31 and 29 and returns a sample-size requirement of 42 student teachers to be rated by three master teachers.

9.9. Sample Size Requirement for Two-Group Design

The interrater reliability for two expert raters will be compared with the interrater reliability of two novice raters. The expert raters will classify one random sample of newspaper articles regarding educational reform into three different categories. The novice raters will perform the same task using another random sample of newspaper articles. Setting ${\widetilde{\text{κ}}}_{G1}$ = .80, ${\widetilde{\text{κ}}}_{G2}$ = .7, r = 3, $\text{α}$ = .05, and a desired confidence interval width of .30, size.diff(.05, .8, .7, 3, .3) computes Equations 30 and 29 and returns a sample-size requirement of 107 per group. The two expert raters should evaluate one random sample of 107 newspaper articles and the two novice raters should evaluate a second random sample of 107 newspaper articles.

10. Conclusion

The G-index of agreement is an attractive alternative to Cohen’s κ for the assessment of nominal-scale agreement. The new confidence interval and sample size methods presented here parallel the methods developed for Cohen’s κ over the last 50 years. The R functions in the Online Supplementary Material can be used to apply the new methods presented here for the G-index of agreement, so that researchers will now be able to perform the same types of inferential and sample-size analyses that are currently available for Cohen’s κ.

All of the proposed confidence intervals are based on adjusted Wald confidence intervals, which have been shown to have excellent performance characteristics in terms of expected coverage probability, worst-case coverage probability, and expected confidence interval width (Agresti & Caffo, 2000; Agresti & Coull, 1998; Bonett & Price, 2012; Price & Bonett, 2004). Newcombe (2013) describes alternatives to the adjusted Wald confidence intervals that have different performance characteristics that might be more desirable in certain applications. For example, the Clopper–Pearson confidence interval for a single population proportion tends to be substantially wider than the adjusted Wald interval but it has a worst-case coverage probability that is guaranteed to be no less than 1 − $\text{α}$ . If worst-case coverage probability is the primary concern, then the adjusted Wald confidence intervals (the terms in brackets) in Equations 5 and 14 could be replaced with Clopper–Pearson confidence intervals. If any current or newly developed confidence interval for a single proportion, a difference of independent proportions, a difference of paired proportions, or a linear function of proportions is considered to be more appropriate than an adjusted Wald confidence, then that confidence interval can be used in place of the adjusted Wald intervals used here.

The results in Sections 3, 4, 5, and 6 are general for r $\ge$ 2, but the results in Sections 6.1 and 6.2 are limited to r = 2. Future research could extend the results in Sections 6.1 and 6.2 to the general case of r $\ge$ 2.

Supplemental Material

Supplemental Material, sj-docx-1-jeb-10.3102_10769986221088561 - Statistical Inference for G-indices of Agreement

Supplemental Material, sj-docx-1-jeb-10.3102_10769986221088561 for Statistical Inference for G-indices of Agreement by Douglas G. Bonett in Journal of Educational and Behavioral Statistics