Restricted accessResearch articleFirst published online 2024-4
Triangulating natural language processing (NLP)-based analysis of rater comments and many-facet Rasch measurement (MFRM): An innovative approach to investigating raters’ application of rating scales in writing assessment
Rater comments tend to be qualitatively analyzed to indicate raters’ application of rating scales. This study applied natural language processing (NLP) techniques to quantify meaningful, behavioral information from a corpus of rater comments and triangulated that information with a many-facet Rasch measurement (MFRM) analysis of rater scores. The data consisted of ratings on 987 essays by 36 raters (a total of 3948 analytic scores and 1974 rater comments) on a post-admission English Placement Test (EPT) at a large US university. We computed a set of comment-based features based on the analytic components and evaluative language the raters used to infer whether raters were aligned to the scale. For data triangulation, we performed correlation analyses between the MFRM measures of rater performance and the comment-based measures. Although the EPT raters showed overall satisfactory performance, we found meaningful associations between rater comments and performance features. In particular, raters with higher precision and fit to what the Rasch model predicts used more analytic components and used evaluative language more similar to the scale descriptors. These findings suggest that NLP techniques have the potential to help language testers analyze rater comments and understand rater behavior.
BakerB. A. (2012). Individual differences in rater decision-making style: An exploratory mixed-methods study. Language Assessment Quarterly, 9(3), 225–248. https://doi.org/10.1080/15434303.2011.637262
2.
BallardL. (2017). The effects of primacy on rater cognition: An eye-tracking study (Publication No. 1897079271) [Doctoral dissertation, Michigan State University]. ProQuest Dissertations and Theses Global.
BarkaouiK. (2010). Variability in ESL essay rating processes: The role of the rating scale and rater experience. Language Assessment Quarterly, 7(1), 54–74. https://doi.org/10.1080/15434300903464418
CronbachL. J. (1995). Giving method variance its due. In ShroutP. E.FiskeS. T. (Eds.), Personality research, methods, and theory: A Festschrift in honor of Donald Fiske (pp. 145–157). Lawrence Erlbaum Associates. https://doi.org/10.4324/9781315806815-10
EckesT. (2005). Examining rater effects in TestDaF writing and speaking performance assessments: A many-facet Rasch analysis. Language Assessment Quarterly, 2(3), 197–221. https://doi.org/10.1207/s15434311laq0203_2
11.
GreeneJ. C.CaracelliV. J.GrahamW. F. (1989). Toward a conceptual framework for mixed-method evaluation design. Educational Evaluation and Policy Analysis, 11, 255–274. https://doi.org/10.3102/01623737011003255
12.
IshaqA.AsgharS.GillaniS. A. (2020). Aspect-based sentiment analysis using a hybridized approach based on CNN and GA. IEEE Access, 8, 135499–135512. https://doi.org/10.1109/ACCESS.2020.3011802
JurafskyD.MartinJ. H. (2000). Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition. Prentice Hall.
15.
KimH. J. (2015). A qualitative analysis of rater behavior on an L2 speaking assessment. Language Assessment Quarterly, 12(3), 239–261. https://doi.org/10.1080/15434303.2015.1049353
16.
KimY. H. (2009). An investigation into native and non-native teachers’ judgments of oral English performance: A mixed methods approach. Language Testing, 26(2), 187–217. https://doi.org/10.1177/0265532208101010
17.
KiritchenkoS.ZhuX.CherryC.MohammadS. (2014). NRC-Canada-2014: Detecting aspects and sentiment in customer reviews. In NakovP.ZeschT. (Eds.), Proceedings of the 8th international workshop on semantic Evaluation (SemEval 2014) (pp. 437–442). ACL Anthology. https://doi.org/10.3115/v1/S14-2076
18.
KjellO. N.KjellK.GarciaD.SikströmS. (2019). Semantic measures: Using natural language processing to measure, differentiate, and describe psychological constructs. Psychological Methods, 24(1), 92–115. https://doi.org/10.1037/met0000191
LinacreJ. M. (2002a). Optimizing rating scale category effectiveness. Journal of Applied Measurement, 3(1), 85–106.
21.
LinacreJ. M. (2002b). What do infit and outfit, mean-square and standardized mean?Rasch Measurement Transactions, 16, 878. https://www.rasch.org/rmt/rmt162f.htm
MessickS. (1989). Validity. In LinnR. L. (Ed.), Educational measurement (pp. 13–104). American Council on Education and Macmillan.
27.
MyfordC. M.WolfeE. W. (2003). Detecting and measuring rater effects using many-facet Rasch measurement: Part I. Journal of Applied Measurement, 4(4), 386–422.
28.
PennebakerJ. W.MehlM. R.NiederhofferK. G. (2003). Psychological aspects of natural language use: Our words, our selves. Annual Review of Psychology, 54, 547–577. https://doi.org/10.1146/annurev.psych.54.101601.145041
29.
RenQ.ZhengY.GuoG.HuY. (2019). Resource recommendation algorithm based on text semantics and sentiment analysis. In BrugaliD.SheuC. Y.SicilianoB.TsaiJ. P. (Eds.), 2019 third IEEE international conference on robotic computing (IRC) (pp. 363–368). https://doi.org/10.1109/IRC.2019.00065
30.
SaipechP.SeresangtakulP. (2018, August14–17). Automatic Thai subjective examination using cosine similarity. In KuriyamaS.LestariD. P.KhaderA. T.RasmequanS. (Eds.), 2018 5th IEEE international conference on advanced informatics: Concept theory and applications (ICAICTA) (pp. 214–218). IEEE. https://doi.org/10.1109/ICAICTA.2018.8541276
SinghalA. (2001). Modern information retrieval: A brief overview. IEEE Data Engineering Bulletin, 24(4), 35–43.
33.
SocherR.PerelyginA.WuJ.ChuangJ.ManningC. D.NgA. Y.PottsC. (2013). Recursive deep models for semantic compositionality over a sentiment treebank. In YarowskyD.BaldwinT.KorhonenA.LivescuK.BethardS. (Eds.), Proceedings of the 2013 conference on empirical methods in natural language processing (pp. 1631–1642). ACL Anthology.
34.
TraceJ.JanssenG.MeierV. (2017). Measuring the impact of rater negotiation in writing performance assessment. Language Testing, 34(1), 3–22. https://doi.org/10.1177/0265532215594830
35.
VasilievY. (2020). Natural language processing with Python and spaCy: A practical introduction. No Starch Press.
36.
WangJ.EngelhardG.JrRaczynskiK.SongT.WolfeE. W. (2017). Evaluating rater accuracy and perception for integrated writing assessments using a mixed-methods approach. Assessing Writing, 33, 36–47. https://doi.org/10.1016/j.asw.2017.03.003
WindS. A. (2020). Do raters use rating scale categories consistently across analytic rubric domains in writing assessment?Assessing Writing, 43, 100416. https://doi.org/10.1016/j.asw.2019.100416
39.
WinkeP.LimH. (2015). ESL essay raters’ cognitive processes in applying the Jacobs et al. Rubric: An eye-movement study. Assessing Writing, 25, 38–54. https://doi.org/10.1016/j.asw.2015.05.002
40.
YanX. (2014). An examination of rater performance on a local oral English proficiency test: A mixed-methods approach. Language Testing, 31(4), 501–527. https://doi.org/10.1177/0265532214536171
41.
ZhangY.ElderC. (2011). Judgments of oral proficiency by non-native and native English speaking teacher raters: Competing or complementary constructs?Language Testing, 28(1), 31–50. https://doi.org/10.1177/0265532209360
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
