Applicability and Efficiency of a Computerized Adaptive Test for the Washington Assessment of the Risks and Needs of Students

Abstract

The Washington Assessment of the Risks and Needs of Students (WARNS) is a computer-based assessment created to help courts, schools, and youth service providers determine an adolescent’s risks and needs that may lead to truancy, drop out, or delinquency from school. Users are advised to consider the WARNS total score to work with youth. A total score estimate based on fewer items than the full item set may result in less respondent burden, administration time, and fatigue, while not hindering accurate decisions. This simulation study examined the applicability and efficiency of a computerized adaptive test (CAT) to estimate a WARNS total score under a unidimensional item response theory model. The results demonstrate that the CAT provides an accurate estimate of students’ risks and needs and reduces the number of items administered for each examinee compared with the existing version. Future directions and limitations of CAT development with the WARNS are discussed.

Keywords

computerized adaptive test item response theory risk assessment screening WARNS

Get full access to this article

View all access options for this article.

References

Akaike

(1974). A new look at the statistical model identification. In Parzen

Tanabe

Kitagawa

(Eds.), Selected papers of Hirotugu Akaike: Springer series in statistics (perspectives in statistics) (pp. 215-222). Springer. https://doi.org/10.1007/978-1-4612-1694-0_16

Alpizar

French

B. F.

T. T.

(2020). Equivalence testing of a youth risk and needs assessment. Journal of Psychoeducational Assessment, 8(8), 1046-1051. https://doi.org/10.1177/0734282920930892

Anderson

Kahn

J. D.

Tindal

(2017). Exploring the robustness of a unidimensional item response theory model with empirically multidimensional data. Applied Measurement in Education, 30(3), 163-177. https://doi.org/10.1080/08957347.2017.1316277

Andrich

(1978). A rating formulation for ordered response categories. Psychometrika, 43(4), 561-573. https://doi.org/10.1007/bf02293814

Beck

A. T.

Kovacs

Weissman

(1979). Assessment of suicidal intention: Scale for suicide ideation [Database record]. APA PsycTests. https://doi.org/10.1037/t01299-000

Beuk

C. H.

(1984). A method for reaching a compromise between absolute and relative standards in examinations. Journal of Educational Measurement, 21(2), 147-152. https://doi.org/10.1111/j.1745-3984.1984.tb00226.x

Bock

R. D.

Mislevy

R. J.

(1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6(4), 431-444. https://doi.org/10.1177/014662168200600405

Bulut

Kan

(2012). Application of computerized adaptive testing to entrance examination for graduate studies in Turkey. Eurasian Journal of Educational Research, 12(49), 61-80. https://eric.ed.gov/?id=EJ1059924

Butler

S. F.

Black

R. A.

McCaffrey

S. A.

Ainscough

Doucette

A. M.

(2017). A computer adaptive testing version of the Addiction Severity Index-Multimedia Version (ASI–MV): The addiction severity CAT. Psychology of Addictive Behaviors, 31(3), 265-275. https://doi.org/10.1037/adb0000256

10.

Cai

Thissen

du Toit

S. H. C.

(2011). IRTPRO for Windows [Computer software]. Scientific Software International.

11.

De Beurs

D. P.

De Vries

A. L.

De Groot

M. H.

De Keijser

Kerkhof

A. J

. (2014). Applying computer adaptive testing to optimize online assessment of suicidal behavior: A simulation study. Journal of Medical Internet Research, 16(9), e207. https://doi.org/10.2196/jmir.3511

12.

De Matos

M. G.

Wainwright

Brebels

Craciun

Gabrhelík

Schjodt

B. H.

Plantade-Gipch

Poštuvan

Stojadinovic

Richards

. (2019). Looking ahead. European Psychologist, 24(4), 337-348. https://doi.org/10.1027/1016-9040/a000362

13.

Embretson

S. E.

Reise

S. P.

(2013). Item response theory for psychologist. Psychology Press.

14.

Feinberg

R. A.

Rubright

J. D.

(2016). Conducting simulation studies in psychometrics. Educational Measurement, 35(2), 36-49. https://doi.org/10.1111/emip.12111

15.

Flens

Smits

Carlier

Van Hemert

A. M.

De Beurs

(2016). Simulating computer adaptive testing with the mood and anxiety symptom questionnaire. Psychological Assessment, 28(8), 953-962. https://doi.org/10.1037/pas0000240

16.

French

B. F.

T. T.

(2020). Differential item functioning of a truancy assessment. Journal of Psychoeducational Assessment, 38(5), 642-648. https://doi.org/10.1177/0734282919863215

17.

George

Coker

French

Strand

McBride

McCurley

(2015). Washington assessment of the risks and needs of students: WARNS user manual. Center for Court Research, Administrative Office of the Courts.

18.

Gibbons

R. D.

Weiss

D. J.

Pilkonis

P. A.

Frank

Moore

Kim

J. B.

Kupfer

D. J.

(2012). Development of a computerized adaptive test for depression. Archives of General Psychiatry, 69(11), 1104-1112. https://doi.org/10.1001/archgenpsychiatry.2012.14

19.

Gotch

C. M.

French

B. F.

(2020). A validation trajectory for the Washington assessment of risks and needs of students. Educational Assessment, 25(1), 65-82. https://doi.org/10.1080/10627197.2019.1702462

20.

Hoge

R. D.

Andrews

D. A.

(2002). Youth level of service/case management inventory: User’s manual. Multi-Health Systems.

21.

Hol

A. M.

Vorst

H. C.

Mellenbergh

G. J.

(2005). A randomized experiment to compare conventional, computerized, and computerized adaptive administration of ordinal Polytomous attitude items. Applied Psychological Measurement, 29(3), 159-183. https://doi.org/10.1177/0146621604271268

22.

Hol

A. M.

Vorst

H. C.

Mellenbergh

G. J.

(2007). Computerized adaptive testing for polytomous motivation items: Administration mode effects and a comparison with short forms. Applied Psychological Measurement, 31(5), 412-429. https://doi.org/10.1177/0146621606297314

23.

Hol

A. M.

Vorst

H. C.

Mellenbergh

G. J.

(2008). Computerized adaptive testing of personality traits. Zeitschrift für Psychologie/Journal of Psychology, 216(1), 12-21. https://doi.org/10.1027/0044-3409.216.1.12

24.

Iverson

French

B. F.

Strand

P. S.

Gotch

C. M.

McCurley

(2018). Understanding school truancy: Risk–need latent profiles of adolescents. Assessment, 25(8), 978-987. https://doi.org/10.1177/1073191116672329

25.

Latimer

Meade

Tennant

(2014). Development of item bank to measure deliberate self-harm behaviours: Facilitating tailored scales and computer adaptive testing for specific research and clinical purposes. Psychiatry Research, 217(3), 240-247. https://doi.org/10.1016/j.psychres.2014.03.015

26.

Lian

Cai

(2020). Developing and validating an item bank for alcohol use disorder screening in the Chinese population by using the computerized adaptive testing. Frontiers in Psychology, 11, 1652. https://doi.org/10.3389/fpsyg.2020.01652

27.

Linden

W. J.

Glas

C. A.

(2007). Computerized adaptive testing: Theory and practice. Springer Science & Business Media.

28.

Magis

Barrada

J. R.

(2017). Computerized adaptive testing with R: Recent updates of the package catR. Journal of Statistical Software, 76(Code Snippet 1). https://doi.org/10.18637/jss.v076.c01

29.

Magis

Raîche

(2012). Random generation of response patterns under computerized adaptive testing with the R package catR. Journal of Statistical Software, 48(8). https://doi.org/10.18637/jss.v048.i08

30.

Magis

Yan

Davier

A. A.

(2017). Computerized adaptive and multistage testing with R: using packages catR and mstR. Springer.

31.

Masters

G. N.

(1982). A Rasch model for partial credit scoring. Psychometrika, 47(2), 149-174. https://doi.org/10.1007/bf02296272

32.

McClarty

K. L.

(2006). A feasibility study of a computerized adaptive test of the international personality item pool NEO [Doctoral dissertation, The University of Texas at Austin]. https://repositories.lib.utexas.edu/handle/2152/2576

33.

McDonald

R. P.

(1999). Test theory: A unified treatment. Psychology Press.

34.

Muraki

(1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16(2), 159-176. https://doi.org/10.1177/014662169201600206

35.

Nandakumar

(1991). Traditional dimensionality versus essential dimensionality. Journal of Educational Measurement, 28(2), 99-117. https://doi.org/10.1111/j.1745-3984.1991.tb00347.x

36.

Rahdert

(1991). Problem-oriented screening instrument for teenagers: The adolescent assessment referral system manual. National Institute on Drug Abuse.

37.

Rodriguez

Reise

S. P.

Haviland

M. G.

(2016). Evaluating bifactor models: Calculating and interpreting statistical indices. Psychological Methods, 21(2), 137-150. https://doi.org/10.1037/met0000045

38.

Samejima

(1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika, 34(1), 1-97. https://doi.org/10.1007/BF03372160

39.

Schwarz

(1978). Estimating the dimension of a model. Annals of Statistics, 6(2), 461-464. https://doi.org/10.1214/aos/1176344136

40.

Sireci

S. G.

Thissen

Wainer

(1991). On the reliability of testlet-based tests. ETS Research Report Series, 1991(1), i-15. https://doi.org/10.1002/j.2333-8504.1991.tb01389.x

41.

Stout

W. F.

(1990). A new item response theory modeling approach with applications to unidimensionality assessment and ability estimation. Psychometrika, 55(2), 293-325. https://doi.org/10.1007/BF02295289

42.

Strand

P. S.

Gotch

C. M.

French

B. F.

Beaver

J. L.

(2019). Factor structure and invariance of an adolescent risks and needs assessment. Assessment, 26(6), 1105-1116. https://doi.org/10.1177/1073191117706021

43.

Sunderland

Batterham

P. J.

Calear

A. L.

Carragher

(2017). The development and validation of static and adaptive screeners to measure the severity of panic disorder, social anxiety disorder, and obsessive compulsive disorder. International Journal of Methods in Psychiatric Research, 26(4), e1561. https://doi.org/10.1002/mpr.1561

44.

Tan

Cai

Zhang

(2018). Development and validation of an item bank for depression screening in the Chinese population using computer adaptive testing: A simulation study. Frontiers in Psychology, 9. https://doi.org/10.3389/fpsyg.2018.01225

45.

Thimm

J. C.

(2020). The Norwegian computerized adaptive test of personality disorder–static form (CAT-PD-SF): Reliability, factor structure, and relationships with personality functioning. Assessment, 27(3), 585-595. https://doi.org/10.1177/1073191117749296

46.

Waller

N. G.

Reise

S. P.

(1989). Computerized adaptive personality assessment: An illustration with the absorption scale. Journal of Personality and Social Psychology, 57(6), 1051-1058. https://doi.org/10.1037//0022-3514.57.6.1051

47.

Walter

O. B.

Becker

Bjorner

J. B.

Fliege

Klapp

B. F.

Rose

(2007). Development and evaluation of a computer adaptive test for “Anxiety” (Anxiety-CAT). Quality of Life Research, 16(Suppl. 1), 143-155. https://doi.org/10.1007/s11136-007-9191-7

48.

Walter

O. B.

Holling

(2008). Transitioning from fixed-length questionnaires to computer-adaptive versions. Zeitschrift für Psychologie/Journal of Psychology, 216(1), 22-28. https://doi.org/10.1027/0044-3409.216.1.22

49.

Zheng

Chang

(2013). Content-balancing strategy in bifactor computerized adaptive patient-reported outcome measurement. Quality of Life Research, 22(3), 491-499. https://doi.org/10.1007/s11136-012-0179-6