This study examines adverse consequences of using hierarchical linear modeling (HLM) that ignores rater effects to analyze ratings collected by multiple raters in longitudinal research. The most severe consequence of using HLM ignoring rater effects is the biased estimation of Levels 1 and 2 fixed effects and potentially incorrect significance tests about them. A cross-classified random effects model (CCREM) is proposed as an alternative to HLM. A Monte Carlo study and an empirical evaluation confirm that CCREM performs better than does HLM in dealing with rater effects. Strengths, limitations, and implications of the study are discussed.
BerkR. A. (2004). Regression analysis: A constructive critique. Thousand Oaks, CA: Sage.
2.
CicchettiD. V. (1991). The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation. Behavioral and Brain Sciences, 14, 119-135.
3.
CronbachL. J.GleserG. G.NandaH.RajaratnamN. (1972). The dependability of behavioral measurement: Theory of generalizability of scores and profiles. New York, NY: Wiley.
4.
DiggleP. J. (1990). Time series: A biostatistical introduction. New York, NY: Oxford University Press.
5.
EngelhardG.GordonB. (2000). Setting and evaluating performance standards for high stakes writing assessments. In WilsonM.EngelhardG.Jr. (Eds.), Object measurement: Theory into practice, Volume 5 (pp. 3-14). Stamford, CT: Ablex.
GuoS.BollenK. A. (2013). Research using longitudinal ratings collected by multiple raters: One methodological problem and approaches to its solution. Social Work Research. Advance online publication. doi:10.1093/swr/svs029
8.
GuoS.HusseyD. (1999). Analyzing longitudinal rating data: A three-level hierarchical linear model. Social Work Research, 23, 258-269.
Institute of Education Sciences. (2010). Efficacy of schoolwide programs to promote social and character development and reduce problem behavior in elementary school children. Retrieved from http://ies.ed.gov/ncer/pubs/20112001/index.asp
11.
JayasingheU. W.MarshH. W.BondN. (2003). A multilevel cross-classified modeling approach to peer review of grant proposals: The effects of assessor and researcher attributes on assessor ratings. Journal of the Royal Statistical Society: Series A, 166, 279-300.
12.
LairdN.WareJ. (1982). Random-effects models for longitudinal data. Biometrics, 38, 963-974.
13.
LiangK.ZegerS. (1986). Longitudinal data analysis using generalized linear models. Biometrika, 73, 13-22.
14.
LindleyD. V.SmithA. F. M. (1972). Bayes estimates for the linear model. Journal of the Royal Statistical Society, Series B, 34, 1-41.
15.
LittellR. C.MillikenG. A.StroupW. W.WolfingerR. D.SchabenbergerO. (2006). SAS system for mixed models (2nd ed.). Cary, NC: SAS Institute.
16.
LuoW.KwokO. (2009). The impact of ignoring a crossed factor in analyzing cross-classified data. Multivariate Behavioral Research, 44, 182-212.
17.
MacgowanM. J.NashJ. K.FraserM. W. (2002). The Carolina Child Checklist of risk and protective factors for aggression. Research on Social Work Practice, 12, 253-276. doi:10.1177/104973150201200204
18.
MeyersJ. L.BeretvasS. N. (2006). The impact of inappropriate modeling of cross-classified data structures. Multivariate Behavioral Research, 41, 473-497.
19.
PatzR. J.JunkerB. W.JohnsonM. S.MarianoL. T. (2002). The hierarchical rater model for rated test items and its application to large-scale educational assessment data. Journal of Educational and Behavioral Statistics, 27, 341-384.
20.
RaudenbushS. W. (1993). A crossed random effects model for unbalanced data with applications in cross-sectional and longitudinal research. Journal of Educational Statistics, 18, 321-349.
21.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.
ShavelsonR. J.WebbN. M. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage.
25.
ShiY.LeiteW.AlginaJ. (2010). The impact of omitting the interaction between crossed factors in cross-classified random effects modeling. British Journal of Mathematical and Statistical Psychology, 63, 1-15.
26.
WilletJ. B.SayerA. G. (1994). Using covariance structure analysis to detect correlates and predictors of individual change over time. Psychological Bulletin, 116, 363-381.
27.
WilsonH. G. (1988). Parameter estimation for peer grading under incomplete design. Educational and Psychological Measurement, 48, 69-81.
