Confidence Intervals and F Tests for Intraclass Correlation Coefficients Based on Three-Way Mixed Effects Models

Abstract

Intraclass correlation coefficients (ICCs) are commonly used in behavioral measurement and psychometrics when a researcher is interested in the relationship among variables of a common class. The formulas for deriving ICCs, or generalizability coefficients, vary depending on which models are specified. This article gives the equations for computing confidence intervals and conducting F tests for three-way mixed models with one fixed facet, extending the work on random effects models produced by Wong and McGraw. It provides the formula for finding the confidence interval for generalizability coefficients in the situations in which ${{\mathrm{n}}_{\mathrm{i}}}^{\prime }$ is not necessarily equal to n_i, where n_i and ${{\mathrm{n}}_{\mathrm{i}}}^{\prime }$ are the number of levels of facet i in generalizability and decision studies, respectively. The authors provide the formulas for finding ${{\mathrm{n}}_{\mathrm{i}}}^{\prime }$ when the researcher wishes to maintain a minimum desired generalizability coefficient. Finally, the authors illustrate the analytic differences among four models using one hypothetical data set and then use real data from student course evaluations to further illustrate one of the four models.