A latent semantic analysis-based automated summary assessment is described; this automated system is applied to a real learning from text task in a Distance Education context. We comment on the use of automated content, plagiarism, text coherence measures, and word weights average and their impact on predicting human judges summary scoring. A first regression analysis showed the independence of interparagraph coherence with respect to superficial text variables, advising its inclusion in a general regression model, along with content, plagiarism measures. The final regression model explains a considerable degree of variability in human judgment of summaries. Finally, we discuss several methodological implications and further applications of the automated summary scoring technique developed in this study.
Barrón-CedeñoA.RossoP. (2009) On automatic plagiarism detection based on n-grams comparison. In: Boughanem (eds) Advances in Information Retrieval, ECIR 2009, LNCS 5478, Berlin/Heidelberg, Germany: Springer-Verlag, pp. 696–700.
2.
BellissensC.JeuniauxP.DuranN. D.McNamaraD. S. (2010) A text relatedness and dependency computational model: Using latent semantic analysis and coh-metrix to predict self-explanation quality. Studia Informatica Universalis8: 85–125.
3.
BereiterC.ScardamaliaM. (1984) Information processing demand of text composition. In: MandlH.SteinN.TrabassoC. (eds) Learning and comprehension of text, Hillsdale, NJ: LEA, pp. 407–428.
4.
BestgenY.LoriesG.ThewissenJ. (2010) Using latent semantic analysis to measure coherence in essays by foreign language learners? In: BolascoS.ChiariI.GiulianoL. (eds) Proceedings of 10th international conferences journée d’analyse statistique des données textuelles, (JADT2010), 9–11 July 2010, Rome.
5.
BrownA. L.DayJ. D. (1983) Macrorules for summarizing texts: The development of expertise. Journal of Verbal Learning and Verbal Behavior22(1): 1–14.
6.
CaiZ.McNamaraD. S.LouwerseM. M.HuX.RoweM. P.GraesserA. C. (2004) NLS: A non-latent similarity algorithm. In: ForbusK. D.GentnerD.RegierT. (eds) Proceedings of the 26th annual conference of the cognitive science society, Mahwah, NJ: LEA, pp. 180–185.
7.
ČeškaZ. (2010) Automatic plagiarism detection based on latent semantic analysis: Theory and practice, Saarbrüecken, Germany: VDM Verlag Dr. Müller, pp. 101–110.
8.
ČeškaZ.HanákI.TesařR. (2007) Teraman: A tool for N-gram extraction from large datasets. ICCP 2007, New York, NY: IEEE, pp. 209–216.
9.
DeerwesterS.DumaisS. T.FurnasG. W.LandauerT. K.HarshmanR. (1990) Indexing by latent semantic analysis. Journal of the American Society for Information Science41: 391–407.
10.
DefeldreA. C. (2005) Inadvertent plagiarism in everyday life. Applied Cognitive Psychology19: 1033–1040.
11.
FoltzP. W. (2007) Discourse coherence and LSA. In: LandauerT.McNamaraD.SimonD.KintschW. (eds) Handbook of latent semantic analysis, Mahwah, NJ: Lawrence Erlbaum Associates.
12.
FoltzP. W.KintschW.LandauerT. K. (1998) The measurement of textual coherence with latent semantic analysis. Discourse Processes25: 285–307.
13.
FoltzP. W.LahamD.LandauerT. K. (1999) Automated essay scoring: Applications to educational technology. World Conference on Educational Multimedia, Hypermedia and Telecommunications (ED-MEDIA)1999(1): 939–944.
14.
FoltzP. W.LochbaumK. E.RosensteinM. B. (2011) Analysis of student writing for a large scale implementation of formative assessment. Paper presented at the National Council for Measurement in Education, New Orleans, LA.
15.
GraesserA. C.MillisK. N.ZwaanR. A. (1997) Discourse comprehension. Annual Review of Psychology48: 163–189.
16.
GraesserA. C.PenumatsaP.VenturaM.CaiZ.HuX. (2007) Using LSA in AutoTutor: Learning through mixed-initiative dialogue in natural language. In: LandauerT. K.McNamaraD.DennisS.KintschW. (eds) The handbook of latent semantic analysis, Mahwah, NJ: Erlbaum, pp. 243–262.
17.
GraesserA. C.SingerM.TrabassoT. (1994) Constructing inferences during narrative text comprehension. Psychological Review101: 371–395.
18.
GraesserA. C.Wiemer-HastingsP.Wiemer-HastingsK.HarterD.PersonN.The Tutoring Research Group (2000) Using latent semantic analysis to evaluate the contributions of students in AutoTutor. Interactive Learning Environments8: 129–148.
19.
HaleyD.Applying latent semantic analysis to computer assisted assessment in the computer science domain: A framework, a tool, and an evaluation (Doctoral dissertation).2009Buckinghamshire, UK: The Open University.
20.
HuX.CaiZ.Wiemer-HastingsP.GraesserA. C.McNamaraD. S. (2007) Strengths, limitations, and extensions of LSA. In: LandauerT. K.McNamaraD. S.DennisS.KintschW. (eds) The handbook of latent semantic analysis, Mahwah, NJ: Erlbaum, pp. 401–426.
21.
Jorge-BotanaG.LeónJ. A.OlmosR.EscuderoI. (2010) Latent semantic analysis parameters for essay evaluation using small-scale corpora. Journal of Quantitative Linguistics17(1): 1–29.
22.
Jorge-BotanaG.OlmosR.BarrosoA. (2013, July) Gallito 2.0: A natural language processing tool to support research on discourse. Proceedings of the 13th Annual Meeting of the Society for Text and Discourse, Valencia, Spain.
23.
Jorge-BotanaG.OlmosR.BarrosoA. (2015) Call routing based on a combination of the construction-integration model and latent semantic analysis: A full system. Informatica.
24.
KakkonenT.SutinenE. (2011) Essay aid: Towards a semi-automatic system for assessing student texts. International Journal of Continuing Engineering Education and Life-Long Learning21(2–3): 119–139.
25.
KintschE.CaccamiseD.FranzkeM.JohnsonN.DooleyS. (2007) Summary street: Computer-guided summary writing. In: LandauerT. K.McNamaraD.DennisS.KintschW. (eds) The handbook of latent semantic analysis, Mahwah, NJ: Erlbaum, pp. 263–277.
26.
KintschW. (1998) Comprehension: A paradigm for cognition, New York, NY: Cambridge University Press.
27.
KintschW. (2002) On the notions of theme and topic in psychological process models of text comprehension. In: LouwerseM.van PeerW. (eds) Thematics: Interdisciplinary studies, Amsterdam, the Netherlands: Benjamins, pp. 157–170.
28.
KintschW.MangalathP. (2011) The construction of meaning. Topics in Cognitive Science3: 346–370.
29.
KurbyC. A.Wiemer–HastingsK.GanduriN.MaglianoJ. P.MillisK. K.McNamaraD. S. (2003) Computerizing reading training: Evaluation of a latent semantic analysis space for science text. Behavior Research Methods, Instruments and Computers35: 244–250.
30.
LandauerT. K. (2002) On the computational basis of cognition: Arguments from LSA. In: RossB. H. (ed.) The psychology of learning and motivation, New York, NY: Academic Press, pp. 43–84.
31.
LandauerT. K. (2003) Automatic essay assessment. Assessment in Education10(3): 295–308.
32.
LandauerT. K. (2007) LSA as a theory of meaning. In: LandauerT. K.McNamaraD.DennisS.KintschW. (eds) Handbook of latent semantic analysis, New York, NY and London, England: Routledge-Taylor & Francis Group, pp. 3–34.
33.
LandauerT. K.DumaisS. T. (1997) A solution to Plato’s problem: The latent semantic analysis theory of the acquisition, induction and representation of knowledge. Psychological Review104: 211–240.
34.
LandauerT. K.DumaisS. T. (2008) Latent semantic analysis. Scholarpedia3(11): 4356.
LandauerT. K.LahamD.FoltzP. W. (1998) Learning human-like knowledge by singular value decomposition: A progress report. In: JordanM. I.KearnsM. J.SollaS. A. (eds) Advances in neural information processing systemsVol. 10, Cambridge, MA: MIT Press, pp. 45–51.
37.
LeónJ. A.OlmosR.EscuderoI.CañasJ. J.SalmerónL. (2006) Assessing short summaries with human judgments procedure and latent semantic analysis in narrative and expository texts. Behavior Research Methods, Instruments and Computers38(4): 616–627.
38.
LorchR. F.O’BrienE. J. (1995) Sources of coherence in reading, Hillsdale, NJ: Erlbaum.
39.
MaglianoJ. P.GraesserA. C. (2012) Computer-based assessment of student-constructed responses. Behavior Research Methods44: 608–621. doi:10.3758/s13428-012-0211-3.
McCowanI.MooreD.DinesJ.Gatica-PérezD.FlynnM.WellnerP.BourlanrH. (2004) On the use of information retrieval measures for speech recognition evaluation, IDIAP-RR73, Martigny, Switzerland: IDIAP.
42.
McNamaraD. S.BoonthumC.LevinsteinI. B.MillisK. (2007) Evaluating self-explanations in iSTART: Comparing word-based and LSA algorithms. In: LandauerT. K.McNamaraD.DennisS.KintschW. (eds) The handbook of latent semantic analysis, Mahwah, NJ: Erlbaum, pp. 227–242.
43.
McNamaraD. S.CaiZ.LouwerseM. M. (2007) Optimizing LSA measures of cohesion. In: LandauerT.McNamaraD.SimonD.KintschW. (eds) Handbook of latent semantic analysis, Mahwah, NJ: Lawrence Erlbaum Associates.
44.
McNamaraD. S.LevinsteinI. B.BoonthumC. (2004) iSTART: Interactive strategy training for active reading and thinking. Behavior Research Methods, Instruments, and Computers36: 222–233.
45.
McNamaraT. (1996) Measuring second language performance, Harlow, Essex, England: Addison Wesley Longman Ltd.
46.
MillisK. K.KimH. J. J.TodaroS.MaglianoJ. P.Wiemer-HastingsK.McNamaraD. S. (2004) Identifying reading strategies using latent semantic analysis: Comparing semantic benchmarks. Behavior Research Methods, Instruments and Computers36: 213–221.
47.
Monjurul IslamM.Latiful HoqueA. S. M. (2012) Automated essay scoring using generalized latent semantic analysis. Journal of Computers7(3): 616–626.
48.
MulderG.SandersT. J. M. (2012) Causal coherence relations and levels of discourse representation. Discourse Processes49: 501–522.
49.
NakovP. (2000, August) Getting better results with latent semantic indexing. Paper presented at the Students Presentations at the European Summer School in Logic Language and Information (ESSLLI’00) (pp. 156–166), Birmingham, England.
50.
NakovP.PopovaA.MateevP. (2001) Weight functions impact on LSA performance. Paper presented at Recent Advances in Natural Language Processing, – RANLP 2001, Tzigov Chark, Bulgaria.
51.
OldeB. A.FranceschettiD. R.KarnavatGraesserA. C.The Tutoring Research Group (2002) The right stuff: Do you need to sanitize your corpus when using latent semantic analysis? In: GrayW.SchunnC. D. (eds) Proceedings of the 24th annual conference of the cognitive science society, Mahwah, NJ: Lawrence Erlbaum Associates, pp. 708–713.
52.
OlmosR.LeónJ. A.Jorge-BotanaG.EscuderoI. (2009) New algorithms assessing short summaries in expository texts using latent semantic analysis. Behaviour Research Methods, Instruments, and Computers41: 944–950.
53.
PalincsarA. S.BrownA. L. (1984) Reciprocal teaching of comprehension fostering and comprehension-monitoring activities. Cognition & Instruction1: 117–175.
54.
RehderB.SchreinerM. E.WolfeB. W.LahamD.LandauerT. K.KintschW. (1998) Using latent semantic analysis to assess knowledge: Some technical considerations. Discourse Processes25: 337–354.
55.
SandersT. J. M.SpoorenW. P. M.NoordmanL. G. M. (1993) Coherence relations in a cognitive theory of discourse representation. Cognitive Linguistics8: 93–133.
56.
SantrockJ. W. (2012) Psicología de la Educación [Educational psychology], 4th ed. Madrid: McGraw-Hill UNED.
57.
ShermisM. D.BursteinJ.HigginsD.ZechnerK. (2010) Automated essay scoring: Writing assessment and instruction. In: BakerE.McGawB.PetersenN. S. (eds) International encyclopedia of educationVol. 4, Oxford, UK: Elsevier, pp. 20–26.
58.
SrihariS.CollinsJ.ShihariR.SrinivasanH.ShettyS.Brutt-GrifferJ. (2008) Automatic scoring of short handwritten essays in reading comprehension tests. Artificial Intelligence172: 300–324.
59.
Wade-SteinD.KintschE. (2004) Summary street: Interactive computer support for writing. Cognition and Instruction22: 333–362.
60.
Wiemer-HastingsP.Wiemer-HastingsK.GraesserA. (1999) Improving an intelligent tutor’s comprehension of students with latent semantic analysis. In: LajoieS. P.VivetM. (eds) Artificial intelligence in education, Amsterdam: IOS Press, pp. 535–542.
61.
WolfeM. B. W.GoldmanS. R. (2003) Use of latent semantic analysis for predicting psychological phenomena: Two issues and proposed solutions. Behavior Research Methods, Instruments, and Computers35: , 22–31.
62.
WolfeM. B. W.MaglianoJ. P.LarsenB. (2005) Causal and semantic relatedness in discourse understanding and representation. Discourse Processes39: 165–187.
