It is common practice for assessment programs to organize qualifying sessions during which the raters (often known as “markers” or “judges”) demonstrate their consistency before operational rating commences. Because of the high-stakes nature of many rating activities, the research community tends to continuously explore new methods to analyze rating data. We used simulated and empirical data from two high-stakes language assessments, to propose a new approach, based on social network analysis and exponential graph models, to evaluate the readiness of a group of raters for operational rating. The results of this innovative approach are compared with the results of a Rasch analysis, which is a well-established approach for the analysis of such data. We also demonstrate how the new approach can be practically used to investigate important research questions such as whether rater severity is stable across rating tasks. The merits of the new approach, and the consequences for practice are discussed.
BairdJ.HayesM.JohnsonR.JohnsonS.LamprianouI. (2013). Marker effects and examination reliability: A comparative exploration from the perspectives of generalizability theory, Rasch modelling and multilevel modelling. Report to The Office of Qualifications and Examinations Regulation (Ofqual), UK.
2.
BanerjiM. (1999). Validation of scores/measures from a K-2 developmental assessment in mathematics. Educational and Psychological Measurement, 59, 694-715.
3.
BarrettS. (2001). The impact of training on rater variability. International Education Journal, 2, 49-58.
BlockP. (2015). Reciprocity, transitivity, and the mysterious three-cycle. Social Networks, 40, 163-173. doi:10.1016/j.socnet.2014.10.005
6.
BojanowskiM.CortenR. (2014). Measuring segregation in social networks. Social Networks, 39, 14-32. doi:10.1016/j.socnet.2014.04.001
7.
BonkW. J.OckeyG. J. (2003). A many-facet Rasch analysis of the second language group oral discussion task. Language Testing, 20, 89-110.
8.
BraunH. I. (1988). Understanding score reliability: Experiments in calibrating essay readers. Journal of Educational Statistics, 13, 1-18.
9.
BrennanR. L. (2001). Generalizability theory. New York, NY: Springer-Verlag.
10.
CongdonP. J.McQueenJ. (2000). The stability of rater severity in large-scale assessment programs. Journal of Educational Measurement, 37, 163-178.
11.
CushingS. W. (1998). Using FACETS to model rater training effects. Language Testing, 15, 263-287.
12.
CushingS. W. (1999). Investigating rater/prompt interactions in writing assessment: Quantitative and qualitative approaches. Assessing Writing, 6, 145-178.
13.
EckesT. (2005). Examining rater effects in TestDaF writing and speaking performance assessments: a many-facet Rasch analysis. Language Assessment Quarterly, 2, 197-221.
14.
EngelhardG.Jr. (1992). The measurement of writing competence with a many-faceted Rasch model. Applied Measurement in Education, 5, 171-191.
15.
EngelhardG.Jr. (1994). Examining rater errors in the assessment of written composition with a many-faceted Rasch model. Journal of Educational Measurement, 31, 93-112.
16.
EngelhardG.Jr. (1997). Constructing rater and task banks for performance assessments. Journal of Outcome Measurement, 1, 19-33.
17.
EngelhardG.Jr. (2002). Monitoring raters in performance assessments. In TindalG.HaladynaT. M. (Eds.), Large scale assessments for all students: Validity, technical adequacy, and implementation (pp. 261-288). Mahwah, NJ: Lawrence Erlbaum.
18.
EngelhardG.Jr. (2013). Invariant measurement: Using Rasch models in the social, behavioral, and health sciences. New York, NY: Routledge.
19.
FreedmanS. W. (1981). Influences on evaluators of expository essays: beyond the text. Research in the Teaching of English, 15, 245-255.
20.
FosterJ. G.FosterD. V.GrassbergerP.PaczuskiM. (2010). Edge direction and the structure of networks. Proceedings of the National Academy of Sciences of the U S A, 107, 10815-10820.
21.
GarnerM.EngelhardG. (2009). Using paired comparison matrices to estimate parameters of the partial credit Rasch measurement model for rater-mediated assessments. Journal of Applied Measurement, 10, 30-41.
GoodreauS. (2007). Advances in exponential random graph (p*) models applied to a large social network. Social Networks, 29, 231-248.
24.
HarrisJ. K. (2014). An introduction to exponential random graph modeling. Thousand Oaks, CA: Sage.
25.
HeQ.AnwyllS.GlanvilleM.DeavallA. (2013). An investigation of the reliability of marking of the Key Stage 2 National Curriculum English writing tests in England. Educational Research, 55, 393-410.
26.
HoskensM.WilsonM. (2001). Real-time feedback on rater drift in constructed-response items: An example from the Golden State Examination. Journal of Educational Measurement, 38, 121-145.
27.
HunterD. R.HandcockM. S.ButtsC. T.GoodreauS. M.MorrisM. (2008). ergm: A package to fit, simulate and diagnose exponential-family models for networks. Journal of Statistical Software, 24(3), 1-29.
28.
LamprianouI. (2004). Marking quality assurance procedures for large-scale high-stakes assessment (Not in public domain), National Assessment Agency, UK.
29.
LamprianouI. (2006). The stability of marker characteristics across tests of the same subject and across subjects. Journal of Applied Measurement, 7, 192-200.
30.
LamprianouI. (2008). High-stakes tests with self-selected essay questions: Addressing issues of fairness. International Journal of Testing, 18, 55-89.
31.
LamprianouI.BoyleB. (2004). Accuracy of measurement in the context of mathematics National Curriculum tests in England for ethnic minority pupils and pupils who speak English as an additional language. Journal of Educational Measurement, 41, 239-260.
32.
LeckieG.BairdJ. (2011) Rater effects on essay scoring: A multilevel analysis of severity drift, central tendency, and rater experience. Journal of Educational Measurement, 48, 399-418.
LunzM. E.WrightB. D.LinacreJ. M. (1990). Measuring the impact of judge severity on examination scores. Applied Measurement in Education, 3, 331-345.
35.
LunzM. E.StahlJ. A.WrightB. D. (1991). The invariance of judge severity calibrations. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.
36.
MastersG. N. (2002). Fair and meaningful measures? A review of examination procedures in the NSW Higher School Certificate. Camberwell, Victoria, Australia: ACER Press.
37.
McPhersonM.Smith-LovinL.CookJ. M. (2001). Birds of a feather: Homophily in social networks. Annual Review of Sociology, 27, 415-444.
38.
MyfordC. M.WolfeE. W. (2002). When raters disagree, then what: Examining a third-rating discrepancy resolution procedure and its utility for identifying unusual patterns of ratings. Journal of Applied Measurement, 3, 300-324.
39.
RaeG.HylandP. (2001). Generalizability and classical test theory analyses of Koppitz’s Scoring System for human figure drawings. British Journal of Educational Psychology, 71, 369-382.
40.
RaschG. (1960). Probabilistic models for some intelligence and achievement tests. Copenhagen, Denmark: Danish Institute for Educational Research. (Expanded edition, 1980. Chicago, IL: University of Chicago Press)
41.
RobinsG.PattisonP.KalishY.LusherD. (2007). An introduction to exponential random graph (p*) models for social networks. Social Networks, 29, 173-191.
42.
Royal-DawsonL.BairdJ.-A. (2009). Is teaching experience necessary for reliable scoring of extended English questions?Educational Measurement: Issues and Practice, 28(2) 2-8.
43.
SaalF. E.DowneyR. G.LaheyM. A. (1980). Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin, 88, 413-428.
44.
SmithR. M. (1991). The distributional properties of Rasch item fit statistics. Educational and Psychological Measurement, 51, 541-565.
45.
SmithR. M. (2000). Fit analysis in latent trait measurement models. Journal of Applied Measurement, 1, 199-218.
SnijdersT. A.Van de BuntG. G.SteglichC. E. (2010). Introduction to stochastic actor-based models for network dynamics. Social Networks, 32, 44-60.
48.
StahlJ. A.LunzM. E. (1991). Judge performance reports: Media and message. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
49.
WangJ.EngelhardG.Jr.WolfeE. W. (2016). Evaluating rater accuracy in rater-mediated assessments using an unfolding model. Educational and Psychological Measurement, 76, 1005-1025. doi:10.1177/0013164415621606
50.
WangW.-C.ChenC.-T. (2005). Item parameter recovery, standard error estimates, and fit statistics of the Winsteps program for the family of Rasch models. Educational and Psychological Measurement, 65, 376-404. doi:10.1177/0013164404268673
51.
WeigleS. (1998). Using FACETS to model rater training effects. Language Testing, 15, 263-287.
52.
WilliamsO.Del GenioC. I. (2014). Degree correlations in directed scale-free networks. PLoS One, 9:e110121.
53.
WisemanC. S. (2012). Rater effects: Ego engagement in rater decision-making. Assessing Writing, 17, 150-173.
WrightB. D.StoneM. H. (1979). Best test design. Chicago, IL: MESA Press.
57.
WuS. M.TanS. (2016). Managing rater effects through the use of FACETS analysis: The case of a university placement test. Higher Education Research & Development, 35, 380-394. doi:10.1080/07294360.2015.1087381
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
