Personalized learning technologies such as automated reading tutors have been proposed as a way to help struggling readers acquire needed skills while simultaneously encouraging engaged, sustained reading of entire books. This article investigates a key step in the development of such technologies: translating an entire novel into a sequence of nonoverlapping reading passages, so that valid feedback about students’ performances can be generated at the end of each tutoring session. A critical reliability issue that may be introduced at this step is identified, an approach for addressing that issue is proposed, and an evaluation conducted with respect to three popular middle-grade novels is presented. The findings suggest that our new approach provides more reliable evidence about students’ abilities and thus may be more appropriate for use when building automated reading tutors.
Common Core State Standards Initiative (2010, June) Common core state standards for English language arts & literacy in history/social studies, science and technical subjects, Washington, DC: CCSSO & National Governors Association.
2.
Beigman KlebanovB.LoukinaA.SabatiniJ.O’ReillyT. (2017) Continuous fluency tracking and the challenges of varying text complexity. Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications (pp. 22–32), Copenhagen, Denmark: Association for Computational Linguistics.
3.
BiberD.ConradS.ReppenR.ByrdP.HeltM.ClarkV.UrzuaA. (2004) Representing language use in the university: Analysis of the TOEFL 2000 spoken and written academic language corpus (TOEFL Monograph Series, MS-25), Princeton, NJ: Educational Testing Service.
4.
BrysbaertM.BiemillerA. (2017) Test-based age-of-acquisition norms for 44 thousand English word meanings. Behavior Research Methods49(4): 1520–1523.
5.
CarverR. P. (1985) Measuring readability using DRP units. Journal of Literacy Research17(4): 303–316.
6.
CoeR. (2009) Unobserved but not unimportant: The effects of unmeasured variables on causal attributions. Effective Education1(2): 101–122.
7.
ColtheartM. (1981) The MRC psycholinguistic database. Quarterly Journal of Experimental Psychology33A: 497–505.
8.
ComptonD.AppletonA.HospM. (2004) Exploring the relationship between text-leveling systems and reading accuracy and fluency in second-grade students who are average and poor decoders. Learning Disabilities Research19(3): 176–184.
9.
DeaneP.SheehanK. M.SabatiniJ. P.FutagiY.KostinI. (2006) Differences in text structure and its implications for the assessment of struggling readers. Scientific Studies of Reading10(3): 257–275.
10.
DentonC. A.BarthA. E.FletcherJ. M.WexlerJ.VaughnS.CirinoP. T.FrancisD. J. (2011) The relations among oral and silent reading fluency and comprehension in middle school: Implications for identification and instruction of students with reading disabilities. Scientific Studies of Reading15(2): 109–135.
11.
FisherD.FreyN.LappD. (2012) Text complexity: Raising rigor in reading, Newark, DE: International Reading Association.
12.
FryE. (1990) A readability formula for short passages. Journal of Reading33: 594–597.
13.
GorinJ. S. (2007) Test design with cognition in mind. Educational Measurement: Issues and Practice24(4): 21–35.
14.
GraesserA. C.McNamaraD. S.KulikowichJ. M. (2011) Coh-Metrix: Providing multilevel analyses of text characteristics. Educational Researcher40(5): 223–234.
15.
HiebertE. H.MesmerH. A. (2013) Upping the ante of text complexity in the Common Core State Standards: Examining its potential impact on young readers. Educational Researcher42(1): 44–51.
16.
Kantor, A., Cerňak, M., Havelka, J., Huber, S., Kleindienst, J., & Gonzalez, D. B. (2012). Reading companion: The technical and social design of an automated reading tutor. Proceedings of the 3rd Workshop on Child, Computer and Interaction (pp. 53–59). Portland, OR: International Speech Communication Association (ISCA).
17.
KidwellP.LebanonG.Collins-ThompsonK. (2010) Statistical estimation of word acquisition with application to readability prediction. Journal of the American Statistical Association106: 21–30.
18.
KincaidJ. P.FishburneR. P.RogersR. L.ChissomB. S. (1975) Derivation of new readability formulas (automated readability index, Fog count and Flesch reading ease formula) for Navy enlisted personnel (Research Branch Report 8-75), Memphis, TN: Naval Air Station.
19.
LiX.DengL.JuY.AceroA. (2008) Automatic children’s reading tutor on hand-held devices. Proceedings of Interspeech 2008 (pp. 1733–1736), Brisbane, Australia: International Speech Communication Association.
20.
LoukinaA.LiceraldeV. R.Beigman KlebanovB. (2018) Towards understanding text factors in reading. Proceedings of NAACL-HLT (pp. 2143–2154), New Orleans, LA: Association for Computational Linguistics.
21.
Loukina, A., Sabatini, J., O’Reilly, T., & Beigman-Klebanov, B. (2017, July). Exploring variation in text features in Harry Potter. Presented at the 27th Annual Meeting of the Society for Text and Discourse, Philadelphia, PA.
22.
MislevyR. J.AlmondR. J.LukasJ. F. (2003) A brief introduction to evidence-centered design (Research Report No. RR-03-16), Princeton, NJ: Educational Testing Service.
23.
MostowJ.Nelson-TaylorJ.BeckJ. E. (2013) Computer-guided oral reading versus independent practice: Comparison of sustained silent reading to an automated reading tutor that listens. Journal of Educational Computing Research49(2): 249–276.
24.
Napolitano, D., Sheehan, K. M., & Mundkowski, R. (2015, June). Online readability and text complexity analysis with TextEvaluator. Presented at the North American Chapter of the Association of Computational Linguistics—Human Language Technologies (NAACL HLT 2015), Denver, CO.
25.
National Institute of Child Health and Development. (2000). Report of the National Reading Panel: Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction (NIH Publication No. 00-4769). Washington, DC: U.S. Government Printing Office. Retrieved from https://www.nichd.nih.gov/publications/pubs/nrp/pages/smallbook.aspx.
26.
NelsonJ.PerfettiC.LibenD.LibenM. (2012) Measures of text difficulty: Testing their predictive value for grade levels and student performance, Washington, DC: Council of Chief State School Officers and National Governors Association.
27.
Pitler, E., & Nenkova, A. (2008). Revisiting readability: A unified framework for predicting text quality. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing (pp. 186–195). Stroudsburg, PA: Association for Computational Linguistics.
28.
ProencaJ.LopesC.TjalveM.StolckeA.CandeiasS.PerdigaoF. (2017) Automatic evaluation of reading aloud performance in children. Speech Communication94: 1–14.
29.
RaschG. (1960) Probabilistic models for some intelligence and attainment tests, Copenhagen, Denmark: Nielsen & Lydiche.
30.
SheehanK. M. (1997) A tree-based approach to proficiency scaling and diagnostic assessment. Journal of Educational Measurement34: 333–352.
31.
SheehanK. M.FlorM.NapolitanoD. (2013) A two-stage approach for generating unbiased estimates of text complexity. In: RelloL.SaggionH.Baeza-YatesR. (eds) Proceedings of the 2nd Workshop on Natural Language Processing for Improving Textual Accessibility, Stroudsburg, PA: Association for Computational Linguistics, pp. 49–58.
32.
Sheehan, K. M., & Ginther, A. (2001, April). What do passage-based multiple-choice verbal reasoning items really measure? An analysis of the cognitive skills underlying performance on TOEFL reading comprehension items. Paper presented at the National Council on Measurement in Education (NCME), Seattle, WA.
33.
Sheehan, K. M., Kostin, I., & Futagi, Y. (2007). SourceFinder: A construct-driven approach for locating appropriately targeted reading comprehension source texts. Proceedings of the 2007 Workshop on Speech and Language Technology in Education (pp. 80–83). Farmington, PA: International Speech Communication Association (ISCA).
34.
SheehanK. M.KostinI.NapolitanoD.FlorM. (2014) The TextEvaluator Tool: Helping teachers and test developers select texts for use in instruction and assessment. Elementary School Journal115(2): 184–209.
35.
Shermis, M. D., & Burstein, J. (Eds). (2013). Handbook of automated essay scoring evaluation: Current applications and new directions. New York, NY: Routledge.
36.
StennerA. J.BurdickH.SanfordE. E.BurdickD. S. (2006) How accurate are Lexile measures?Journal of Applied Measurement7: 307–322.
37.
Vajjala, S., & Meurers, D. (2012). On improving the accuracy of readability classification using insights from second language acquisition. In Proceedings of the 7th Workshop on Building Educational Applications Using NLP (pp. 163–173). Stroudsburg, PA: Association for Computational Linguistics.
38.
YngveV. H. (1960) A model and hypothesis for language structure. Proceedings of the American Philosophical Society104: 444–466.
39.
ZwickR. (1991) Statistical and psychometric issues in the measurement of educational achievement trends: Examples from the National Assessment of Educational Progress. Journal of Educational Statistics17(2): 205–218.
