Stealth assessment has been successfully embedded in educational games to measure students’ learning in an unobtrusive and supportive way. This study explored the possibility of applying stealth assessment in a digital reading platform and sought to identify potential in-system indicators of students’ digital learning outcomes. Utilizing the user log data from third- to fifth-grade students (n = 573) who read an interactive Word Knowledge E-Book, we examined various user log variables and their associations with word knowledge and strategic reading outcomes. Descriptive analysis provided a depiction of the real-time reading processes and behaviors in which students engaged while digitally reading. Multiple regression analysis with classroom fixed effects demonstrated that user log variables relevant to answering questions and making decisions (i.e., percentage of embedded questions answered correctly; number of attempts to answer the questions; and making implausible decisions) were significantly associated with students’ word knowledge and strategic reading outcomes. Variables indicating reading time and frequency, however, were not significantly associated with these outcomes. This study highlights the potential of interactive e-books as another digital learning environment to establish stealth assessment, which may allow researchers and educators to track students’ reading processes and predict reading outcomes while supporting digital learning.
AkçapınarG.HasnineM. N.MajumdarR.FlanaganB.OgataH. (2019). Developing an early-warning system for spotting at-risk students by using eBook interaction logs. Smart Learning Environments, 6(1), Article 4.
2.
AskinadzeA.LiebeckM.ConradS. (2018, November). Predicting Student Test Performance based on Time Series Data of eBook Reader Behavior Using the Cluster-Distance Space Transformation [Conference session]. International Conference on Computers in Education (ICCE2018): Learning Analytics Workshop Joint Activity, (pp. 26–30). Manila, Philppines.
3.
BakerR.XuD.ParkJ.YuR.LiQ.CungB.SmythP. (2020). The benefits and caveats of using clickstream data to understand student self-regulatory behaviors: Opening the black box of learning processes. International Journal of Educational Technology in Higher Education, 17, 1–24.
4.
BakerR. S.CorbettA. T.AlevenV. (2008). More accurate student modeling through contextual estimation of slip and guess probabilities in Bayesian knowledge tracing. In (WoolfBAïmeurENkambouRLajoieS (Eds.), International Conference on Intelligent Tutoring Systems, (pp. 406-415). Springer.
5.
BakerR. S.CorbettA. T.KoedingerK. R. (2004, August). Detecting student misuse of intelligent tutoring systems. In GoosG.HartmanisJ.LeeuwenJ. V (Eds.), International Conference on Intelligent Tutoring Systems (pp. 531–540). Springer, Berlin, Heidelberg.
6.
BarthA. E.BarnesM.FrancisD.VaughnS.YorkM. (2015). Inferential processing among adequate and struggling adolescent comprehenders and relations to reading comprehension. Reading and Writing, 28(5), 587–609.
7.
BoteanuA.ChernovaS.NunezD.BreazealC. (2016). Fostering parent–child dialog through automated discussion suggestions. User Modeling and User-Adapted Interaction, 26(5), 393–423.
8.
BryanG.FawsonP. C.ReutzelD. R. (2003). Sustained silent reading: Exploring the value of literature discussion with three non-engaged readers. Literacy Research and Instruction, 43(1), 47–73.
9.
CainK.OakhillJ.BryantP. (2004). Children’s reading comprehension ability: Concurrent prediction by working memory, verbal ability, and component skills. Journal of Educational Psychology, 96, 31–42.
10.
CarverR. P. (1992). Reading rate: Theory, research, and practical implications. Journal of Reading, 36(2), 84–95.
11.
ChappiusS.ChappiusJ. (2008). The best value in formative assessment. Educational Leadership, 65(5), 14–19.
12.
CiampaK. (2012). ICANREAD: The effects of an online reading program on grade 1 students’ engagement and comprehension strategy use. Journal of Research on Technology in Education, 45(1), 27–59.
13.
ConnorC. M. (2016). A lattice model of the development of reading comprehension. Child Development Perspectives, 10(4), 269–274.
14.
ConnorC. M.DayS. L.ZargarE.WoodT. S.TaylorK. S.JonesM. R.HwangJ. K. (2019). Building word knowledge, learning strategies, and metacognition with the Word-Knowledge e-Book. Computers & Education, 128, 284–311.
15.
De JongM. T.BusA. G. (2004). The efficacy of electronic books in fostering kindergarten children’s emergent story understanding. Reading Research Quarterly, 39(4), 378–393.
16.
DentonC. A.EnosM.YorkM. J.FrancisD. J.BarnesM. A.KuleszP. A.. . . CarterS. (2015). Text-processing differences in adolescent adequate and poor comprehenders reading accessible and challenging narrative and informational text. Reading Research Quarterly, 50(4), 393–416.
17.
EllemanA. M.LindoE. J.MorphyP.ComptonD. L. (2009). The impact of vocabulary instruction on passage level comprehension of school age children: A meta-analysis. Journal of Research on Educational Effectiveness, 2(1), 1–44.
18.
GoldhammerF.NaumannJ.StelterA.TóthK.RölkeH.KliemeE. (2014). The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment. Journal of Educational Psychology, 106(3), 608–626.
HulstijnJ. H.HollanderM.GreidanusT. (1996). Incidental vocabulary learning by advanced foreign language students: The influence of marginal glosses, dictionary use, and reoccurrence of unknown words. The Modern Language Journal, 80(3), 327–339.
21.
KimY. S. G. (2017). Multicomponent view of vocabulary acquisition: An investigation with primary grade children. Journal of Experimental Child Psychology, 162, 120–133.
22.
LauferB.HulstijnJ. (2001). Incidental vocabulary acquisition in a second language: The construct of task-induced involvement. Applied Linguistics, 22(1), 1–26.
23.
MarinakB. A.GambrellL. B. (2010). Reading motivation: Exploring the elementary gender gap. Literacy Research and Instruction, 49(2), 129–141.
24.
McEneaneyJ. E. (2006). Agent-based literacy theory. Reading Research Quarterly, 41(3), 352–371.
25.
McNamaraD. S. (Ed.). (2007). Reading comprehension strategies: Theory, interventions, and technologies. Lawrence Erlbaum.
26.
MislevyR. J.SteinbergL. S.AlmondR. G. (2003). On the structure of educational assessment. Measurement: Interdisciplinary Research and Perspective, 1(1), 3–62.
27.
PellegrinoJ. W.ChudowskyN.GlaserR. (2001). Knowing what students know: The science and design of educational assessment. National Academy Press.
28.
PerfettiC. A.HartL. (2002). The lexical quality hypothesis. Precursors of functional literacy, 11, 67–86.
29.
QuinnJ. M.WagnerR. K.PetscherY.RobertsG.MenzelA. J.SchatschneiderC. (2020). Differential codevelopment of vocabulary knowledge and reading comprehension for students with and without learning disabilities. Journal of Educational Psychology, 112(3), 608–627.
30.
RicciC. M.BealC. R. (2002). The effect of interactive media on children’s story memory. Journal of Educational Psychology, 94(1), 138–144.
31.
ShuteV. J.KeF. (2012). Games, learning, and assessment. In IfenthalerD.EseryelD.GeX. (Eds.), Assessment in game-based learning (pp. 43–58). Springer.
32.
ShuteV. J.KimY. J. (2014). Formative and stealth assessment. In SpectorM.MerrillM.D.ElenJ.BishopM. J. (Eds.), Handbook of research on educational communications and technology (pp. 311–321). Springer.
33.
ShuteV. J.MooreG. R. (2017). Consistency and validity in game-based stealth assessment. In JiaoHLissitzR.W (Eds.), Technology enhanced innovative assessment: Development, modeling, and scoring from an interdisciplinary perspective (pp. 31–51). Information Age.
34.
ShuteV. J.VenturaM. (2013). Stealth assessment: Measuring and supporting learning in video games. MIT Press.
35.
ShuteV. J.WangL.GreiffS.ZhaoW.MooreG. (2016). Measuring problem solving skills via stealth assessment in an engaging video game. Computers in Human Behavior, 63, 106–117.
36.
SymondsK. W. (2004). After the test: Closing the achievement gaps with data. Learning Point.
37.
ToppingK. (2018). Implementation fidelity in computerised assessment of book reading. Computers & Education, 116, 176–190. https://doi.org/j.compedu.2017.09.009
38.
TorgesenJ. K.RashotteC. A.AlexanderA. (2001). Principles of fluency instruction in reading: Relationships with established empirical outcomes. In WolfM. (Ed.), Dyslexia, fluency, and the brain (pp. 333–355). York Press.
39.
WalkerE.AdamsA.RestrepoM. A.FialkoS.GlenbergA. M. (2017). When (and how) interacting with technology-enhanced storybooks helps dual language learners. Translational Issues in Psychological Science, 3(1), 66–79.
40.
WiliamD.ThompsonM. (2007). Integrating assessment with instruction: What will it take to make it work? In DwyerC. A. (Ed.), The future of assessment: Shaping teaching and learning (pp. 53–82). Lawrence Erlbaum.
41.
XuY.YauJ.ReichS. (in print). Press, swipe, and read: Do interactive features facilitate engagement and learning with e-books?Journal of Computer Assisted Learning.
42.
ZargarE.AdamsA. M.ConnorC. M. (2020). The relations between children’s comprehension monitoring and their reading comprehension and vocabulary knowledge: An eye-movement study. Reading and Writing, 33(3), 511–545.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
