Sparse rating designs, where each examinee’s performance is scored by a small proportion of raters, are prevalent in practical performance assessments. However, relatively little research has focused on the degree to which different analytic techniques alert researchers to rater effects in such designs. We used a simulation study to compare the information provided by two popular approaches: Generalizability theory (G theory) and Many-Facet Rasch (MFR) measurement. In previous comparisons, researchers used complete data that were not simulated—thus limiting their ability to manipulate characteristics such as rater effects, and to understand the impact of incomplete data on the results. Both approaches provided information about rating quality in sparse designs, but the MFR approach highlighted rater effects related to centrality and bias more readily than G theory.
American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (2014). Standards for educational and psychological testing. AERA.
2.
AndrichD. A. (1978). Application of a psychometric rating model to ordered categories which are scored with successive integers. Applied Psychological Measurement, 2(4), 581–594. https://doi.org/10.1177/014662167800200413
3.
BairdJ.HayesM.JohnsonR.JohnsonS.LamprianouI. (2013). Marker effects and examination reliability: A comparative exploration from the perspectives of Generalizability theory, Rasch modelling and multilevel modelling (Ofqual/13/5261). Office of Qualifications and Examinations Regulation.
4.
BrennanR. L. (2000). Performance assessments from the perspective of generalizability theory. Applied Psychological Measurement, 24(4), 339–353. https://doi.org/10.1177/01466210022031796
5.
BrennanR. L. (2001). Generalizability theory. Springer-Verlag.
6.
ChiuC. W. T.WolfeE. W. (2002). A method for analyzing sparse data matrices in the generalizability theory framework. Applied Psychological Measurement, 26(3), 321–338. https://doi.org/10.1177/0146621602026003006
7.
EckesT. (2015). Introduction to many-facet Rasch measurement: Analyzing and evaluating rater-mediated assessments (2nd ed.). Peter Lang.
8.
EngelhardG. (2002). Monitoring raters in performance assessments. In TindalG.HaladynaG. (Eds.), Large-scale assessment programs for ALL students: Development, implementation, and analysis (pp. 261–287). Erlbaum.
9.
EngelhardG.WindS. A. (2018). Invariant measurement with raters and rating scales: Rasch models for rater-mediated assessments. Taylor & Francis.
10.
HillH. C.CharalambousC. Y.KraftM. A. (2012). When rater reliability is not enough: Teacher observation systems and a case for the generalizability study. Educational Researcher, 41(2), 56–64. https://doi.org/10.3102/0013189X12437203
11.
HoytW. T. (2000). Rater bias in psychological research: When is it a problem and what can we do about it?Psychological Methods, 5(1), 64.
12.
IramaneeratC.SmithE. V.SmithR. M. (2008). An introduction to Rasch measurement. In OsborneJ. W. (Ed.), Best practices in quantitative methods. Sage Publications, Inc.
13.
JohnsonR. L.PennyJ. A.GordonB. (2009). Assessing performance: Designing, scoring, and validating performance tasks. The Guilford Press.
14.
KimS. C.WilsonM. (2010). A comparative analysis of the ratings in performance assessment using generalizability theory and the Many-Facet Rasch model. In GarnerM. L.EngelhardJrG.WilsonM. (Eds.), Advances in Rasch measurement (1). JAM Press.
15.
LinC.-K. (2017). Working with sparse data in rated language tests: Generalizability theory applications. Language Testing, 34(2), 271–289. https://doi.org/10.1177/0265532216638890
16.
LinacreJ. M. (1989). Many-Facet Rasch measurement. MESA Press.
LinacreJ. M. (1996). Generalizability theory and Many-Facet Rasch measurement. In EngelhardG.WilsonM. (Eds.), Objective measurement: Theory into practice (3, pp. 85–98). Ablex.
19.
LinacreJ. M. (2015). Facets Rasch Measurement (3.71.4). Winsteps.com.
20.
LumleyT.McNamaraT. F. (1995). Rater characteristics and rater bias: Implications for training. Language Testing, 1995(12), 54–71.
21.
LynchB. K.McNamaraT. F. (1998). Using G-theory and Many-Facet Rasch measurement in the development of performance assessments of the ESL speaking skills of immigrants. Language Testing, 15(2), 158–180. https://doi.org/10.1177/026553229801500202
22.
MacMillanP. D. (2000). Classical, generalizability, and multifaceted Rasch detection of interrater variability in large, sparse data sets. The Journal of Experimental Education, 68(2), 167–190. https://doi.org/10.1080/00220970009598501
23.
McEwenM. R. (2015). Development of a software prototype for generating and classifying incomplete Many-Facet Rasch Model rating designs. Unpublished doctoral project, Department of Instructional Psychology and Technology, Brigham Young University, Provo, UT. [Available from the BYU Library in electronic form.]
24.
McEwenM. (2018). The effects of incomplete rating designs on results from Many-facets Rasch Model analyses. Unpublished PhD. Dissertation, Brigham Young University, Provo, UT.
MyfordC. M.WolfeE. W. (2003). Detecting and measuring rater effects using Many-Facet Rasch measurement: Part I. Journal of Applied Measurement, 4(4), 386–422.
27.
MyfordC. M.WolfeE. W. (2004). Detecting and measuring rater effects using Many-Facet Rasch measurement: Part II. Journal of Applied Measurement, 5(2), 189–227.
28.
RajuN. S. (1988). The area between two item characteristic curves. Psychometrika, 53(4), 495–502.
29.
R Core Team. (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/
30.
SchumackerR. E. (1999). Many-facet Rasch Analysis with crossed, nested, and mixed designs. Journal of Outcome Measurement, 3(4), 323–338.
31.
ShavelsonR. J.WebbN. M. (1991). Generalizability theory: A primer. Sage Publications, Inc.
32.
SudweeksR. R.ReeveS.BradshawW. S. (2004). A comparison of generalizability theory and many-facet Rasch measurement in an analysis of college sophomore writing. Assessing Writing, 9(3), 239–261. https://doi.org/10.1016/j.asw.2004.11.001
33.
WindS. A.JonesE. (2018). Exploring the influence of range restrictions on connectivity in sparse assessment networks: An illustration and exploration within the context of classroom observations. Journal of Educational Measurement, 55(2), 217–241. https://doi.org/10.1111/jedm.12173
34.
WindS. A.JonesE. (2019). The effects of incomplete rating designs in combination with rater effects. Journal of Educational Measurement, 56(1), 76–100. https://doi.org/10.1111/jedm.12201
35.
WolfeE. W.McVayA. (2012). Application of latent trait models to identifying substantively interesting raters. Educational Measurement: Issues and Practice, 31(3), 31–37. https://doi.org/10.1111/j.1745-3992.2012.00241.x
36.
WolfeE. W.SongT. (2015). Comparison of models and indices for detecting rater centrality. Journal of Applied Measurement, 16(3), 228–241.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
