Although many studies have been carried out on the psychometric aspects of computerized adaptive testing (CAT), its psychological aspects are less researched. Early studies claimed that CAT can be more motivating and induce less anxiety than traditional fixed-item tests (FIT). The purpose of this systematic review and meta-analysis was to gain a comprehensive understanding of the effects of CAT on motivation and anxiety in comparison to traditional fixed-item testing. Seven databases were examined. Articles were eligible if they employed an empirical study containing a direct comparison between CAT and FIT. Meta-analytical results showed no overall effect of test type on anxiety and motivation when comparing CAT with FIT (k = 11, g+ = 0.06, p = .28). However, easier CAT had positive effect compared with FIT (k = 2, g+ = .22, p < .001). Certain modifications in CAT administration can provide positive psychological effects for test-takers.
BakkerA.CaiJ.EnglishL.KaiserG.MesaV.Van DoorenW. (2019). Beyond small, medium, or large: Points of consideration when interpreting effect sizes. Educational Studies in Mathematics, 102(1), 1–8. https://doi.org/10.1007/s10649-019-09908-4
3.
BergstromB. A.LunzM. E.GershonR. C. (1992). Altering the difficulty in computer adaptive testing. Applied Measurement in Education, 5, 137–149.
4.
BetzN. E. (1977). Effects of immediate knowledge of results and adaptive testing on ability test performance. Applied Psychological Measurement, 1, 259–266.
5.
BetzN. E.WeissD. J. (1976). Psychological effects of immediate knowledge of results and adaptive ability testing (Research Report, 76-4). University of Minnesota, Department of Psychology, Psychometric Methods Program, Computerized Adaptive Testing Laboratory.
6.
BorensteinM.HedgesL. V.HigginsJ. P. T.RothsteinH. R. (2011). Introduction to meta-analysis. Wiley.
BorensteinM.HigginsJ. P.HedgesL. V.RothsteinH. R. (2017). Basics of meta-analysis: I2 is not an absolute measure of heterogeneity. Research Synthesis Methods, 8(1), 5–18. https://doi.org/10.1002/jrsm.1230
9.
ChoiS. W.GradyM. W.DoddB. G. (2011). A new stopping rule for computerized adaptive testing. Educational and Psychological Measurement, 71(1), 37–53. https://doi.org/10.1177/0013164410387338
10.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum.
11.
CuiZ.LiuC.HeY.ChenH. (2018). Evaluation of a new method for providing full review opportunities in computerized adaptive testing—Computerized adaptive testing with salt. Journal of Educational Measurement, 55(4), 582–594. https://doi.org/10.1111/jedm.12193
12.
DaveyJ.TurnerR. M.ClarkeM. J.HigginsJ. P. (2011). Characteristics of meta-analyses and their component studies in the Cochrane Database of Systematic Reviews: A cross-sectional, descriptive analysis. BMC Medical Research Methodology, 11(1), 1–11. https://doi.org/10.1186/1471-2288-11-160
13.
EggerM.SmithG. D.SchneiderM.MinderC. (1997). Bias in meta-analysis detected by a simple, graphical test. The BMJ, 315(7109), 629–634. https://doi.org/10.1136/bmj.315.7109.629
14.
FinkelmanM. D.KimW.WeissmanA.CookR. J.VeldkampB. P.BarnardJ.van der LindenW. J.Ramón BarradaJ.MeadA. D.BeckerK. A.ReckaseM. D.DoddB. G.RileyB.EggenT.WalterO. B.FreyA.WangW.-C.HanK. T.WiseS. L. (2014). Cognitive diagnostic models and computerized adaptive testing: Two new item-selection methods that incorporate response times. Journal of Computerized Adaptive Testing, 2(3), 59–76. https://doi.org/10.7333/1412-0204059
15.
FlensG.SmitsN.CarlierI.van HemertA. M.de BeursE. (2016). Simulating computer adaptive testing with the mood and anxiety symptom questionnaire. Psychological Assessment, 28(8), 953–962. https://doi.org/10.1037/pas0000240
16.
FrittsB. E.MarszalekJ. M. (2010). Computerized adaptive testing, anxiety levels, and gender differences. Social Psychology of Education, 13(3), 441–458. https://doi.org/10.1007/s11218-010-9113-3
17.
GreenB. F.BockR. D.HumphreysL. G.LinnR. L.ReckaseM. D. (1984). Technical guidelines for assessing computerized adaptive tests. Journal of Educational Measurement, 21(4), 347–360. https://doi.org/10.1111/j.1745-3984.1984.tb01039.x
18.
HanK. T. (2013). Item pocket method to allow response review and change in computerized adaptive testing. Applied Psychological Measurement, 37(4), 259–275. https://doi.org/10.1177/0146621612473638
19.
HäuslerJ.SommerM. (2008). The effect of success probability on test economy and self-confidence in computerized adaptive tests. Psychology Science Quarterly, 50(1), 75–87.
HigginsJ.ThomasJ.ChandlerJ.CumpstonM.LiT.PageM.WelchV. (Eds.). (2021). Cochrane handbook for systematic reviews of interventions (version 6). Cochrane. https://www.training.cochrane.org/handbook
22.
HongQ.PluyeP.FàbreguesS.BartlettG.BoardmanF.CargoM.DagenaisP.GagnonM.-P.GriffithsF.NicolauB.O’CathainA.RousseauM.-C.VedelI. (2018). Mixed Methods Appraisal Tool (MMAT), version 2018. Registration of Copyright (#1148552), Canadian Intellectual Property Office, Industry Canada.
23.
KimJ.McLeanJ. E. (1995, April). The influence of examinee test-taking behavior motivation in computerized adaptive testing [Paper presentation]. Annual meeting of the National Council on Measurement in Education, San Francisco, CA, United States.
24.
KimuraT. (2017). The impacts of computer adaptive testing from different perspectives. Journal of Educational Evaluation for Health Professions, 14, 12. https://doi.org/10.3352/jeehp.2017.14.12
25.
KingsburyG.WiseS. (2000). Practical issues in developing and maintaining a computerized adaptive testing program. Psicológica: Revista de Metodología y Psicología Experimental, 21(1), 135–156.
LeeH. Y.DoddB. G. (2012). Comparison of exposure controls, item pool characteristics, and population distributions for CAT using the partial credit model. Educational and Psychological Measurement, 72(1), 159–175. https://doi.org/10.1177/0013164411411296
LingG.AttaliY.FinnB.StoneE. A. (2017). Is a computerized adaptive test more motivating than a fixed-item test?Applied Psychological Measurement, 41(7), 495–511. https://doi.org/10.1177/0146621617707556
31.
LunzM. E.BergstromB. A.GershonR. C. (1994). Computer adaptive testing. International Journal of Educational Research, 21(6), 623–634. https://doi.org/10.1016/0883–0355(94)90015-9
32.
MartinA. J.LazendicG. (2018). Computer-adaptive testing: Implications for students’ achievement, motivation, engagement, and subjective test experience. Journal of Educational Psychology, 110(1), 27–45. https://doi.org/10.1037/edu0000205
33.
MeadA. D.DrasgowF. (1993). Equivalence of computerized and paper-and-pencil cognitive ability tests : A equivalence of computerized and paper-and-pencil cognitive ability tests : A meta-analysis. Psychological Bulletin, 114(3), 449–458. https://doi.org/10.1037/0033-2909.114.3.449
34.
Mohd AliS.NorfarahN.Ilya SyazwaniJ. I.Mohd ErfyI. (2019). The effect of computerized-adaptive test on reducing anxiety towards math test for polytechnic students. Journal of Technical Education and Training, 11(4), 27–35. https://doi.org/10.30880/jtet.2019.11.04.004
35.
OleaJ.RevueltaJ.XiménezM.AbadF. (2000). Psychometric and psychological effects of review on computerized fixed and adaptive test. Psicológica: Revista de Metodología y Psicología Experimental, 21(1), 157–174.
36.
OrtnerT. M.CaspersJ. (2011). Consequences of test anxiety on adaptive versus fixed item testing. European Journal of Psychological Assessment, 27(3), 157–163. https://doi.org/10.1027/1015-5759/a000062
37.
OrtnerT. M.WeißkopfE.KochT. (2014). I will probably fail: Higher ability students’ motivational experiences during adaptive achievement testing. European Journal of Psychological Assessment, 30(1), 48–56. https://doi.org/10.1027/1015-5759/a000168
38.
PageM. J.McKenzieJ. E.BossuytP. M.BoutronI.HoffmannT. C.MulrowC. D.ShamseerL.TetzlaffJ. M.AklE. A.BrennanS. E.ChouR.GlanvilleJ.GrimshawJ. M.HróbjartssonA.LaluM. M.LiT.LoderE. W.Mayo-WilsonE.McDonaldS.. . . MoherD. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. The BMJ, 372, Article n71. https://doi.org/10.1136/bmj.n71
39.
PitkinA. K.VispoelW. P. (2001). Differences between self-adapted and computerized adaptive tests: A meta-analysis. Journal of Educational Measurement, 38(3), 235–247. https://doi.org/10.1111/j.1745–3984.2001.tb01125.x
RevickiD. A.CellaD. F. (1997). Health status assessment for the twenty-first century: Item response theory, item banking and computer adaptive testing. Quality of Life Research, 6(6), 595–600. https://doi.org/10.1023/A:1018420418455
42.
RevueltaJ.XiménezM. C.OleaJ. (2003). Psychometric and psychological effects of item selection and review on computerized testing. Educational and Psychological Measurement, 63(5), 791–808. https://doi.org/10.1177/0013164403251282
43.
RobbinsS. B.LeH.DavisD.LauverK.LangleyR.CarlstromA. (2004). Do psychosocial and study skill factors predict college outcomes? A meta-analysis. Psychological Bulletin, 130(2), 261–288. https://doi.org/10.1037/0033-2909.130.2.261
44.
SpeilbergerC. D. (Ed.). (1972). Anxiety: Current trends in theory and research (Vols. 1–2). Academic Press.
45.
SterneJ. A.SuttonA. J.IoannidisJ. P.TerrinN.JonesD. R.LauJ.CarpenterJ.RückerG.HarbordR. M.SchmidC. H.TetzlaffJ.DeeksJ. J.PetersJ.MacaskillP.SchwarzerG.DuvalS.AltmanD. G.MoherD.HigginsJ. P. (2011). Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. The BMJ, 343, Article d4002. https://doi.org/10.1136/bmj.d4002
46.
TabachnickB. G.FidellL. S. (2013). Using multivariate statistics (6th ed.). Pearson.
47.
TanQ.CaiY.LiQ.ZhangY.TuD. (2018). Development and validation of an item bank for depression screening in the Chinese population using computer adaptive testing: A simulation study. Frontiers in Psychology, 9(JUL), Article 1225. https://doi.org/10.3389/fpsyg.2018.01225
48.
TonidandelS.QuiñonesM. A. (2000). Psychological reactions to adaptive testing. International Journal of Selection and Assessment, 8(1), 7–15. https://doi.org/10.1111/1468–2389.00126
49.
TonidandelS.QuiñonesM. A.AdamsA. A. (2002). Computer-adaptive testing: The impact of test characteristics on perceived performance and test takers’ reactions. Journal of Applied Psychology, 87(2), 320–332. https://doi.org/10.1037/0021–9010.87.2.320
50.
van der LindenW. J. (2005). A comparison of item-selection methods for adaptive tests with content constraints. Journal of Educational Measurement, 42(3), 283–302. https://doi.org/10.1111/j.1745-3984.2005.00015.x
van der LindenW. J.PashleyP. J. (2000). Item selection and ability estimation in adaptive testing. In van der LindenW. J.GlasC. A. W. (Eds.), Computerized adaptive testing: Theory and practice (pp. 1–25). Springer. https://doi.org/10.1007/0-306-47531-6_1
53.
VispoelW. P.HendricksonA. B.BleilerT. (2000). Limiting answer review and change on computerized adaptive vocabulary tests: Psychometric and attitudinal results. Journal of Educational Measurement, 37(1), 21–38. https://doi.org/10.1111/J.1745-3984.2000.TB01074.X
54.
WainerH. (2000). Computerized adaptive testing: A primer (2nd ed.). Lawrence Erlbaum.
55.
WainerH.EignorD. (2000). Caveats, pitfalls, and unexpected consequences of implementing large scale computerized testing. In WainerH. (Ed.), Computerized adaptive testing: A primer (pp. 271–299). Lawrence Erlbaum.
56.
WeissD. J. (1982). Improving measurement quality and efficiency with adaptive testing. Applied Psychological Measurement, 6, 473–492.
57.
WeissD. J.BetzN. E. (1973). Ability measurement: Conventional or adaptive? (Research Report 73-1). Department of Psychology, Psychometric Methods Program.
58.
WigfieldA.EcclesJ. S. (2000). Expectancy-value theory of achievement motivation. Contemporary Educational Psychology, 25, 68–81.
59.
WiseS. L. (2014, January). The utility of adaptive testing in addressing the problem of unmotivated examinees the utility of adaptive testing in addressing the problem of unmotivated examinees. Journal of Computerized Adaptive Testing, 2(1). https://doi.org/10.7333/1401-0201001
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
