We used multivariate generalizability theory to examine the reliability of hand-scoring and automated essay scoring (AES) and to identify how these scoring methods could be used in conjunction to optimize writing assessment. Students (n = 113) included subsamples of struggling writers and non-struggling writers in Grades 3–5 drawn from a larger study. Students wrote six essays across three genres. All essays were hand-scored by four raters and an AES system called Project Essay Grade (PEG). Both scoring methods were highly reliable, but PEG was more reliable for non-struggling students, while hand-scoring was more reliable for struggling students. We provide recommendations regarding ways of optimizing writing assessment and blending hand-scoring with AES.
BridgemanB. (2013). Human ratings and automated essay evaluation. In ShermisM. D.BursteinJ. (Eds.), Handbook of automated essay evaluation: Current applications and new directions (pp. 221–232). New York, NY: Routledge.
10.
ChenE.NiemiD.WangJ.WangH.MirochaJ. (2007). Examining the generalizability of direct writing assessment tasks. CSE Technical report 718. National Center for Research on Evaluation, Standards, and Student (CRESST).
11.
CoeM.HanitaM.NishiokaV.SmileyR. (2011). An investigation of the impact of the 6+1 Trait Writing model on Grade 5 student writing achievement (NCEE 2012–4010). National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
12.
CronbachL. J.GelserG. C.NandaH.RajaratnamN. (1972). The dependability of behavioral measurements. Wiley.
13.
CronbachL. J.RajaratnamN.GelserG. C. (1963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16, 137–163.
14.
DeaneP. (2013). On the relation between automated essay scoring and modern views of the writing construct. Assessing Writing, 18, 7–24.
15.
EckesT. (2011). Introduction to many-facet Rasch measurement: Analyzing and evaluating rater-mediated assessments. Peter Lang.
16.
EtmanskieJ. M.PartanenM.SiegelL. S. (2016). A longitudinal examination of the persistence of late emerging reading disabilities. Journal of Learning Disabilities, 49, 21–35. http://dx.doi.org/10.1177/0022219414522706
17.
GansleK. A.VanDerHeydenA. M.NoellG. H.ResetarJ. L.WilliamsK. L. (2006). The technical adequacy of curriculum-based and rating-based measures of written expression for elementary school students. School Psychology Review, 35(4), 435-450. DOI: 10.1080/02796015.2006.12087977
GrahamS.CapizziA.HarrisK.HebertM.MorphyP. (2014). Teaching writing to middle school students: A national survey. Reading and Writing, 27, 1015-1042. https://doi.org/10.1007/s11145-013-9495-7
21.
GrahamS.HarrisK. R. (2015). Common Core State Standards and writing: Introduction to the special issue. Elementary School Journal, 115, 457–463. https://doi.org/10.1086/681963
22.
GrahamS.HarrisK.HebertM. A. (2011a). Informing writing: The benefits of formative assessment. A Carnegie Corporation Time to Act report. Alliance for Excellent Education.
23.
GrahamS.HarrisK. R.HebertM. (2011b). It is more than just the message: Presentation effects in scoring writing. Focus on Exceptional Children, 44(4), 1–12.
24.
GrahamS.HebertM.HarrisK. R. (2015). Formative assessment and writing. Elementary School Journal, 115, 523–547.
25.
GrahamS.HebertM.SandbankM. P.HarrisK. R. (2016). Assessing the writing achievement of young struggling writers: Application of generalizability theory. Learning Disability Quarterly, 39, 72–82.
26.
HallgrenK. A. (2012). Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in Quantitative Methods for Psychology, 8, 25–34. https://doi.org/10.20982/tqmp.08.1.p023
27.
HigginsD.HeilmanM. (2014). Managing what we can measure: Quantifying the susceptibility of automated scoring systems to gaming behavior. Educational Measurement: Issues and Practice, 33(3), 36–46. https://doi.org/10.1111/emip.12036
28.
HoangG. T. L.KunnanA. J. (2016) Automated essay evaluation for English language learners: A case study of MY access. Language Assessment Quarterly, 13(4), 359–376. https://doi.org/10.1080/15434303.2016.1230121
29.
In’namiY.KoizumiR. (2016). Task and rater effects in L2 speaking and writing: A synthesis of generalizability studies. Language Testing, 33(3), 341–366. https://doi.org/10.1177/0265532215587390
30.
JohnsonA. C.WilsonJ.RoscoeR. D. (2017). College student perceptions of writing errors, text quality, and author characteristics. Assessing Writing, 34, 72–87.
31.
JohnsonR. L.PennyJ.FisherS.KuhsT. (2003). Score resolution: An investigation of the reliability and validity of resolved scores. Applied Measurement in Education, 16, 299–322. https://doi.org/10.1207/S15324818AME1604_3
32.
KimY. G.SchatschneiderC.WanzekJ.GatlinB.Al OtaibaS. (2017). Writing evaluation: Rater and task effects on the reliability of writing scores for children in Grades 3 and 4. Reading and Writing, 30, 1287–1310.
LiH. (2017). Narrative and expository genre effects on students, raters, and performance criteria. Assessing Writing, 31, 113–125. https://doi.org/10.1016/j.asw.2016.08.006
35.
LottridgeS. M.SchulzE. M.MitzelH. C. (2013). Using automated scoring to monitor reading performance and detect reader drift in essay scoring. In ShermisM. D.BursteinJ. (Eds.), Handbook of automated essay evaluation (pp. 233–250). Routledge.
36.
National Center for Education Statistics. (2012). The Nation’s Report Card: Writing 2011 (NCES 2012–470). Institute of Education Sciences, U.S. Department of Education.
37.
National Commission on Writing. (2003). The neglected R: The need for a writing revolution. College Entrance Examination Board. www.collegeboard.com
38.
NunnallyJ. C. (1967). Psychometric theory. McGraw-Hill.
39.
Osborn PoppS. E.RyanJ. M.ThompsonM. S. (2009). The critical role of anchor paper selection in writing assessment. Applied Measurement in Education, 22, 225–271. https://doi.org/10.1080/08957340902984026
40.
PageE. B. (2003). Project Essay Grade: PEG. In ShermisM. D.BursteinJ. C. (Eds.), Automated essay scoring: A cross-disciplinary perspective (pp. 43–54). Erlbaum.
41.
PalermoC.BunchM.RidgeK. (2019). Scoring stability in a large-scale assessment program: A longitudinal analysis of leniency/severity effects. Journal of Educational Measurement, 56(3), 626–652. https://doi.org/10.1111/jedm.12228
42.
PennyJ.JohnsonR. L.GordonB. (2000a). The effect of rating augmentation on inter-rater reliability: An empirical study of a holistic rubric. Assessing Writing, 7, 143–164. https://doi.org/10.1016/S1075-2935(00)00012-X
43.
PennyJ.JohnsonR. J.GordonB. (2000b). Using rating augmentation to expand the scale of an analytic rubric. Journal of Experimental Education, 68, 269–287. https://doi.org/10.1080/00220970009600096
44.
PerskyH. R.DaaneM. C.JinY. (2002). The nation’s report card: Writing 2002. (NCES 2003–529). National Center for Education Statistics, Institute of Education Sciences. U.S. Department for Education.
45.
PhilippakosZ. A. T.FitzPatrickE. (2018). A proposed tiered model of assessment in writing instruction: Supporting all student-writers. Insights Into Learning Disabilities, 15, 149–173.
46.
PowersD. E.EscofferyD. S.DuchnowskiM. P. (2015). Validating automated essay scoring: A (modest) refinement of the “gold standard.”Applied Measurement in Education, 28, 130–142. https://psycnet.apa.org/doi/10.1080/08957347.2014.1002920
47.
RaczynskiK.CohenA. (2018). Appraising the scoring performance of automated essay scoring systems—some additional considerations: Which essays? Which human raters? Which scores?Applied Measurement in Education, 31, 233–244. https://doi.org/10.1080/08957347.2018.1464449
48.
SchoonenR. (2005). Generalizability of writing scores: An application of structural equation modeling. Language Testing, 22, 1–30.
49.
SchoonenR. (2012). The validity and generalizability of writing scores: The effect of rater, task and language. In Van SteendamE.TillemaM.RijlaarsdamG.Van den BerghH. (Eds.), Measuring writing: Recent insights into theory, methodology and practices (pp. 1–22). Brill.
50.
ShanahanT. (2015). Common Core State Standards: A new role for writing. Elementary School Journal, 115, 464–479. https://doi.org/10.1086/681130
51.
ShavelsonR. J.WebbN. M. (1991). Generalizability theory: A primer. Sage.
52.
ShermisM. D. (2014). State-of-the-art automated essay scoring: Competition, results, and future directions from a United States demonstration. Assessing Writing, 20, 53–76. https://doi.org/10.1016/j.asw.2013.04.001
53.
ShroutP. E.FleissJ. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428. https://doi.org/10.1037/0033-2909.86.2.420
54.
SinharayS.ZhangM.DeaneP. (2019). Prediction of essay scores from writing process and product features using data mining methods. Applied Measurement in Education, 32, 116–137. https://doi.org/10.1080/08957347.2019.1577245
55.
SpeeceD. L.RitcheyK. D. (2005). A longitudinal study of the development of oral reading fluency in young children at risk for reading failure. Journal of Learning Disabilities, 38, 387–399. http://dx.doi.org/10.1177/00222194050380050201
56.
WangE. L.MatusmuraL. C.CorrentiR.LitmanD.ZhangH.HoweE.MagoodaA.QuintanaR. (2020). eRevis(ing): Students’ revision of text evidence use in an automated writing evaluation system. Assessing Writing, 44. https://doi.org/10.1016/j.asw.2020.100449
57.
WebbN. M.RowleyG. L.ShavelsonR. J. (1988). Using generalizability theory in counseling and development. Measurement and Evaluation in Counseling and Development, 21, 81–90. http://dx.doi.org/10.1080/07481756.1988.12022886
58.
WebbN.ShavelsonR. (2005). Generalizability theory: Overview. In EverittB. S.HowellD. C. (Eds.), Encyclopedia of statistics in behavioral science (Vol. 2, pp. 717–719). John Wiley.
59.
WilsonJ. (2018). Universal screening with automated essay scoring: Evaluating classification accuracy in Grades 3 and 4. Journal of School Psychology, 68, 19–37. https://doi.org/10.1016/j.jsp.2017.12.005
60.
WilsonJ.ChenD.SandbankM. P.HebertM. (2019). Generalizability of automated scores of writing quality in Grades 3–5. Journal of Educational Psychology, 111, 619–640. https://doi.org/10.1037/edu0000311
61.
WilsonJ.RodriguesJ. (2020). Classification accuracy and efficiency of writing screening using automated essay scoring. Journal of School Psychology, 82, 123–140. https://doi.org/10.1016/j.jsp.2020.08.008
62.
WilsonJ.RoscoeR.AhmedY. (2017). Automated formative writing assessment using a levels of language framework. Assessing Writing, 34, 16–36.
63.
WindS. A. (2018). Examining the impacts of rater effects in performance assessments. Applied Psychological Measurement, 43, 159–171. https://doi.org/10.1177%2F0146621618789391
64.
WindS. A. (2020). Exploring the impact of rater effects on person fit in rater-mediated. Assessments. Assessments. Educational Measurement: Issues and Practice, 39(4), 76–94. https://doi.org/10.1111/emip.12354
65.
WindS. A.WalkerA. A. (2019). Exploring the correspondence between traditional score resolution methods and person fit indices in rater-mediated writing assessments. Assessing Writing, 39, 25–38. https://doi.org/10.1016/j.asw.2018.12.002
66.
WindS. A.WolfeE. W.EngelhardG. E.FoltzP.RosensteinM. (2017). The influence of rater effects in training sets on the psychometric quality of automated scoring for writing assessments. International Journal of Testing, 18, 27–49. https://doi.org/10.1080/15305058.2017.1361426
67.
YamanishiH. (2005). Ippanka kanousei riron wo mochiita kokosei no jiyueisakubun hyoka no kento [Using Generalizability Theory in the evaluation of L2 writing]. JALT Journal, 27, 169–185.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
