Automatic coding of short text responses opens new doors in assessment. We implemented and integrated baseline methods of natural language processing and statistical modelling by means of software components that are available under open licenses. The accuracy of automatic text coding is demonstrated by using data collected in the Programme for International Student Assessment (PISA) 2012 in Germany. Free text responses of 10 items with responses in total were analyzed. We further examined the effect of different methods, parameter values, and sample sizes on performance of the implemented system. The system reached fair to good up to excellent agreement with human codings Especially items that are solved by naming specific semantic concepts appeared properly coded. The system performed equally well with and somewhat poorer but still acceptable down to Based on our findings, we discuss potential innovations for assessment that are enabled by automatic coding of short text responses.
AndroutsopoulosI.MalakasiotisP. (2010). A survey of paraphrasing and textual entailment methods. Journal of Artificial Intelligence Research,38, 135-187. doi:10.1613/jair.2985
2.
BärD.ZeschT.GurevychI. (2013). DKPro Similarity: An open source framework for text similarity. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations (pp. 121-126). Sofia, Bulgaria: Association for Computational Linguistics.
3.
BejarI. I. (2012). Rater cognition: Implications for validity. Educational Measurement: Issues and Practice, 31(3), 2-9. doi:10.1111/j.1745-3992.2012.00238.x
4.
BirenbaumM.TatsuokaK. K. (1987). Open-ended versus multiple-choice response formats: It does make a difference for diagnostic purposes. Applied Psychological Measurement,11, 385-395. doi:10.1177/014662168701100404
5.
BorraS.Di CiaccioA. (2010). Measuring the prediction error. A comparison of cross-validation, bootstrap and covariance penalty methods. Computational Statistics & Data Analysis,54, 2976-2989. doi:10.1016/j.csda.2010.03.004
6.
BridgemanB. (1991). A comparison of open-ended and multiple-choice question formats for the quantitative section of the Graduate Record Examinations General Test. ETS Research Report Series, 1991(2), 1-25. doi:10.1002/j.2333-8504.1991.tb01402.x
7.
BridgemanB.TrapaniC.AttaliY. (2012). Comparison of human and machine scoring of essays: Differences by gender, ethnicity, and country. Applied Measurement in Education, 25(1), 27-40. doi:10.1080/08957347.2012.635502
8.
BurrowsS.GurevychI.SteinB. (2014). The eras and trends of automatic short answer grading. International Journal of Artificial Intelligence in Education, 25(1), 60-117.
9.
CarlsonS. S.WardW. C. (1988). A new look at formulating hypotheses items. Princeton, NJ: Educational Testing Service. doi:10.1002/j.2330-8516.1988.tb00268.x
10.
DeerwesterS.DumaisS. T.FurnasG. W.LandauerT. K.HarshmanR. (1990). Indexing by latent semantic analysis. Journal of the American Society for Information Science,41, 391-407. doi:10.1002/(SICI)1097-4571(199009)41:6<391::AID-ASI1 >3.0.CO; 2-9
DumaisS. T. (1991). Improving the retrieval of information from external sources. Behavior Research Methods, Instruments, & Computers,23, 229-236. doi:10.3758/BF03203370
13.
FerrucciD.LallyA. (2004). UIMA: An architectural approach to unstructured information processing in the corporate research environment. Natural Language Engineering,10, 327-348. doi:10.1017/S1351324904003523
14.
FisherR. A. (1915). Frequency distribution of the values of the correlation coefficient in samples of an indefinitely large population. Biometrika,10, 507-521. doi:10.1093/biomet/10.4.507
15.
FisherR. A. (1921). On the probable error of a coefficient of correlation deduced from a small sample. Metron,1, 3-32.
16.
FleissJ. L. (1981). The measurement of interrater agreement. In FleissJ. L.LevinB.PaikM. C. (Eds.), Statistical methods for rates and proportions (pp. 598-626). New York, NY: John Wiley.
17.
GabrilovichE.MarkovitchS. (2007). Computing semantic relatedness using Wikipedia-based explicit semantic analysis. In Proceedings of the 20th International Joint Conference on Artificial Intelligence (pp. 1606-1611).
18.
GabrilovichE.MarkovitchS. (2009). Wikipedia-based semantic interpretation for natural language processing. Journal of Artificial Intelligence Research,34, 443-498.
19.
GurevychI.MühlhäuserM.MüllerC.SteimleJ.WeimerM.ZeschT. (2007). Darmstadt Knowledge Processing Repository based on UIMA. In Proceedings of the First Workshop on Unstructured Information Management Architecture at Biannual Conference of the Society for Computational Linguistics and Language Technology.
20.
JurgensD.StevensK. (2010). The s-space package: An open source package for word space models. In Proceedings of the ACL 2010 System Demonstrations (pp. 30-35). Uppsala, Sweden: Association for Computational Linguistics.
21.
LandauerT. K. (2011). LSA as a theory of meaning. In LandauerT. K.McNamaraD. S.DennisS.KintschW. (Eds.), Handbook of latent semantic analysis (pp. 3-34). Mahwah, NJ: Erlbaum.
MartinD. I.BerryM. W. (2011). Mathematical foundations behind latent semantic analysis. In LandauerT. K.McNamaraD. S.DennisS.KintschW. (Eds.), Handbook of latent semantic analysis (pp. 35-55). Mahwah, NJ: Erlbaum.
24.
MillisK.MaglianoJ.Wiemer-HastingsK.TodaroS.McNamaraD. S. (2011). Assessing and improving comprehension with latent semantic analysis. In LandauerT. K.McNamaraD. S.DennisS.KintschW. (Eds.), Handbook of latent semantic analysis (pp. 207-225). Mahwah, NJ: Erlbaum.
25.
Organisation for Economic Cooperation and Development. (2013). PISA 2012 assessment and analytical framework: Mathematics, reading, science, problem solving and financial literacy. doi:10.1787/9789264190511-en.
26.
Organisation for Economic Cooperation and Development. (2014). PISA 2012 results: What students know and can do: Student performance in mathematics, reading and science (Vol. 1, Rev. ed.). doi:10.1787/9789264201118-en.
PrenzelM.SälzerC.KliemeE.KöllerO. (2013). PISA 2012: Fortschritte und Herausforderungen in Deutschland. Münster, Germany: Waxmann.
29.
R Core Team. (2014). R: A language and environment for statistical computing. Retrieved from http://www.R-project.org/
30.
RaschD.KubingerK. D.YanagidaT. (2011). Statistics in psychology using R and SPSS. Chichester, England: John Wiley. doi:10.1002/9781119979630
31.
RauchD. P.HartigJ. (2010). Multiple-choice versus open-ended response formats of reading test items: A two-dimensional IRT analysis. Psychological Test and Assessment Modeling,52, 354-379.
32.
RuppA. A.FerneT.ChoiH. (2006). How assessing reading comprehension with multiple-choice questions shapes the construct: A cognitive processing perspective. Language Testing,23, 441-474. doi:10.1191/0265532206lt337oa
33.
SaltonG.BuckleyC. (1990). Improving retrieval performance by relevance feedback. Journal of the American Society for Information Science,41, 288-297. doi:10.1002/(SICI)1097-4571(199006)41:4<288::AID-ASI8 >3.0.CO;2-H
34.
SzarvasG.ZeschT.GurevychI. (2011). Combining heterogeneous knowledge resources for improved distributional semantic models. Computational Linguistics and Intelligent Text Processing,6608, 289-303. doi:10.1007/978-3-642-19400-9\textunderscore23
WardJ. H. J. (1963). Hierarchical grouping to optimize an objective function. American Statistical Association Journal,58, 236-244. doi:10.1080/01621459.1963.10500845
37.
WittenI. H.FrankE.HallM. A. (2011). Data mining: Practical machine learning tools and techniques (3rd ed.). Burlington, MA: Morgan Kaufmann. doi:10.1016/B978-0-12-374856-0.00023-7
38.
ZeschT.MüllerC.GurevychI. (2008). Extracting lexical semantic knowledge from Wikipedia and Wiktionary. In Proceedings of the 6th International Conference on Language Resources and Evaluation (pp. 1646-1652).
