One of the most common technology-enhanced items used in large-scale K-12 testing programs is the drag-and-drop response interaction. The main research questions in this study are: (a) Does adding a drag-and-drop interface to an online test affect the accuracy of student performance? (b) Does adding a drag-and-drop interface to an online test affect the speed of student performance? In three different experiments involving fourth, sixth, and eighth graders, respectively; students answered reading comprehension questions presented in conventional (i.e., paper-based design) or drag-and-drop formats. The tests consisted of four-sentence ordering items in Experiment 1, four graphic organizer items in Experiment 2, and two cloze tests and two graphic organizer items in Experiment 3. The conventional and drag & drop groups were compared on test performance (i.e., accuracy) and efficiency (i.e., response time and number of mouse clicks). Across the three experiments, the conventional and drag & drop groups did not differ in mean performance, but the drag & drop group responded more efficiently than the conventional group (faster response time, d = 0.62, and fewer mouse clicks, d = 1.13).
AckermanP. L.KanferR. (2009).
Test length and cognitive fatigue: An empirical examination of effects on performance and test-taker reactions. Journal of Experimental Psychology. Applied,
15(2), 163–181. https://doi.org/10.1037/a0015719
2.
AndersonL. W.KrathwohlD.AirasianP.CruikshankK. A.MayerR. E.PintrichP., Raths, J., & WittrockM. C. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives.Complete edition. Allyn & Bacon.
3.
ArslanB.JiangY.KeehnerM.GongT.KatzI. R.YanF. (2020).
The effect of drag-and-drop item features on test-taker performance and response strategies. Educational Measurement: Issues and Practice,
39(2), 96–106. https://doi.org/10.1111/emip.12326
4.
BarnardJ. J. (2018).
From simulation to implementation: Two CAT case studies. Practical Assessment, Research & Evaluation,
23(14), 1–8. https://doi.org/10.7275/bwvg-d091
5.
BarryC. L.HorstS. J.FinneyS. J.BrownA. R.KoppJ. P. (2010).
Do examinees have similar test-taking effort? A high-stakes question for low-stakes testing. International Journal of Testing,
10(4), 342–363. https://doi.org/10.1080/15305058.2010.508569
6.
BertlingM.WeeksJ. P. (2018).
Using response time data to reduce testing time in cognitive tests. Psychological Assessment,
30(3), 328–338. https://doi.org/10.1037/pas0000466
7.
BloomB. S.EngelhartM. D.FurstE. J.HillW. H.KrathwohlD. R. (1956). Taxonomy of educational objectives: Book 1: Cognitive domain.
David McKay Company.
8.
BryantW. (2017).
Developing a strategy for using technology-enhanced items in large-scale standardized tests. Practical Assessment, Research & Evaluation,
22(1), 1–10. https://doi.org/10.7275/70yb-dj34
9.
CrabtreeA. R. (2016). Psychometric properties of technology-enhanced item formats: An evaluation of construct validity and technical characteristics.
University of Iowa. https://doi.org/https://doi.org/10.17077/etd.922fbj4d
10.
De BoeckP.ChenH.DavisonM. (2017).
Spontaneous and imposed speed of cognitive test responses. The British Journal of Mathematical and Statistical Psychology,
70(2), 225–237. https://doi.org/10.1111/bmsp.12094
11.
GreiffS.SchererR.KirschnerP. A. (2017).
Some critical reflections on the special issue: Current innovations in computer-based assessments. Computers in Human Behavior,
76, 715–718. https://doi.org/10.1016/j.chb.2017.08.019
KongX.DavisL. L.McBrideY.MorrisonK. (2018).
Response time differences between computers and tablets. Applied Measurement in Education,
31(1), 17–29. https://doi.org/10.1080/08957347.2017.1391261
14.
MayerR. E. (2009). Multimedia learning (2nd ed.).
Cambridge University Press.
15.
MayerR. E. (2011). Applying the science of learning.
Pearson/Allyn & Bacon.
16.
MayerR. E. (2014). Cognitive theory of multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed., pp. 43–71). Cambridge University Press.
17.
McCrayG.BrunfautT. (2018).
Investigating the construct measured by banked gap-fill items: Evidence from eye-tracking. Language Testing,
35(1), 51–73. https://doi.org/10.1177/0265532216677105
18.
MedinaA.GajardoA. M. (2009). Prueba de comprensión lectora y producción de textos (CL-PT): Kinder a 4to año básico [Reading comprehension and text production test (CL-PT): Kindergarten to 4th grade].
Ediciones Universidad Católica de Chile.
19.
National Research Council. (2001). Knowing what students know: The science and design of educational assessment. (J. W. Pellegrino, N. Chudowsky, & R. Glaser, Eds.). National Academies Press. https://doi.org/10.17226/10019
20.
NielsenJ. (2000). Designing web usability.
New Riders Publishing.
21.
ParshallC. G.GuilleR. A. (2016). Managing ongoing changes to the test: Agile strategies for continuous innovation. In F. Drasgow (Ed.), Technology and testing: Improving educational and psychological measurement (pp. 1–22). Taylor & Francis. https://doi.org/10.4324/9781315871493
22.
PonceH. R.MayerR. E.SitthiworachartJ.LópezM. J. (2020).
Effects on response time and accuracy of technology-enhanced cloze tests: An eye-tracking study. Educational Technology Research and Development,
68(5), 2033–2053. https://doi.org/10.1007/s11423-020-09740-1
23.
QianH.WooA.DoyoungK. (2018). Exploring the psychometric properties of innovative items in computerized adaptive testing. In JiaoH.LissitzR. W. (Eds.), Technology enhanced innovative assessment: Development, modeling, and scoring from an interdisciplinary perspective (pp. 95–116).
Information Age Publishing.
24.
RaynerK. (1998).
Eye movements in reading and information processing: 20 years of research. Psychological Bulletin,
124(3), 372–422. https://doi.org/10.1037/0033-2909.124.3.372
25.
RaynerK.PollatsekA.AshbyJ.CliftonC.Jr., (2012). Psychology of reading (2nd ed.).
Psychology Press.
26.
RussellM.MoncaleanoS. (2019).
Examining the use and construct fidelity of technology-enhanced items employed by K-12 testing programs. Educational Assessment,
24(4), 286–219. https://doi.org/10.1080/10627197.2019.1670055
27.
SaßS.SchütteK.LindnerM. A. (2017).
Test-takers’ eye movements: Effects of integration aids and types of graphical representations. Computers & Education,
109, 85–97. https://doi.org/10.1016/j.compedu.2017.02.007
28.
SaßS.WittwerJ.SenkbeilM.KöllerO. (2012).
Pictures in test items: Effects on response time and response correctness. Applied Cognitive Psychology,
26(1), 70–81. https://doi.org/10.1002/acp.1798
29.
ShneidermanB.PlaisantC.CohenM.JacobsS. M.ElmqvistN.DiakopoulosN. (2017). Designing the user interface: Strategies for effective human-computer interaction (6th ed.).
Pearson.
30.
SireciS. G.ZeniskyA. L. (2006). Innovative item formats in computer-based testing: In pursuit of improved construct representation. In S. Downing & T. Haladyna (Eds.), Handbook of test development (pp. 329–347). Lawrence Erlbaum Associates. https://doi.org/10.4324/9780203874776.ch14
31.
SpeerA. B.KingB. S.GrossenbacherM. (2016).
Applicant reactions as a function of test length: Is there reason to fret over using longer tests?Journal of Personnel Psychology,
15(1), 15–24. https://doi.org/10.1027/1866-5888/a000145
32.
SwellerJ.AyresP. L.KalyugaS. (2011). Cognitive load theory. Explorations in the learning sciences, instructional systems and performance technologies.
Springer.
33.
TullisT.AlbertW. (2013). Measuring the user experience: Collecting, analyzing, and presenting usability metrics (2nd ed.).
Elsevier.
34.
WanL.HenlyG. A. (2012).
Measurement properties of two innovative item formats in a computer-based test. Applied Measurement in Education,
25(1), 58–78. https://doi.org/10.1080/08957347.2012.635507
35.
WangS.JiaoH.YoungM. J.BrooksT.OlsonJ. (2008).
Comparability of computer-based and paper-and-pencil testing in K–12 reading assessments: A meta-analysis of testing mode effects. Educational and Psychological Measurement,
68(1), 5–24. https://doi.org/10.1177/0013164407305592
36.
WuH. K.KuoC. Y.JenT. H.HsuY. S. (2015).
What makes an item more difficult? Effects of modality and type of visual information in a computer-based assessment of scientific inquiry abilities. Computers & Education,
85, 35–48. https://doi.org/10.1016/j.compedu.2015.01.007
