Using Interval Likelihood Ratios in Gated Screening: A Direct Replication Study

Abstract

This direct replication study compared the use of dichotomized likelihood ratios and interval likelihood ratios, derived using a prior sample of students, for predicting math risk in middle school. Data from the prior year state test and the Measures of Academic Progress were analyzed to evaluate differences in the efficiency and diagnostic accuracy of gated screening decisions. Post-test probabilities were interpreted using a threshold decision-making model to classify student risk during screening. Using interval likelihood ratios led to fewer students requiring additional testing after the first gate. But, when interval likelihood ratios were used, three tests were required to classify 6th- and 7th-grade students as at-risk or not at-risk. Only two tests were needed to classify students as at-risk or not at-risk when dichotomized likelihood ratios were used. Acceptable sensitivity and specificity estimates were obtained, regardless of the type of likelihood ratios used to estimate post-test probabilities. When predicting academic risk, interval likelihood ratios may be best reserved for situations where at least three successive tests are available to be used in a gated screening model.

Keywords

replication gated universal screening math middle school

Get full access to this article

View all access options for this article.

References

Brown

M. D.

Reeves

M. J.

(2003). Interval likelihood ratios: Another advantage for the evidence-based diagnostician. Annals of Emergency Medicine, 42(2), 292–297. https://doi.org/10.1067/mem.2003.274

Canivez

G. L.

(2019). Evidence-based assessment for school psychology: Research, training, and clinical practice. Contemporary School Psychology, 232(2), 194–200. https://doi.org/10.1007/s40688-019-00238-z

Deeks

J. J.

Altman

D. G.

(2004). Diagnostic tests 4: Likelihood ratios. British Medical Journal, 329, 168–169. https://doi.org/10.1136/bmj.329.7458.168

Enders

C. K.

(2010). Applied missing data analysis. Guilford.

Gallagher

E. J.

(1998). Clinical utility of likelihood ratios. Annals of Emergency Medicine, 31(3), 391–397. https://doi.org/10.1016/S0196-0644(98)70352-X

Gersten

Beckman

Clark

Foegen

Marsh

Star

J. R.

Witzel

(2009). Assisting students struggling with mathematics: Response to Intervention (RtI) for elementary and middle schools (NCEE 2009-4060). https://ies.ed.gov/ncee/wwc/PracticeGuide/2

IBM Corp. (2017). IBM SPSS Statistics for Windows (Version 25). Author.

Jenkins

J. R.

Hudson

R. F.

Johnson

E. S.

(2007). Screening for at-risk readers in a response to intervention framework. School Psychology Review, 36(4), 582–600. https://doi.org/10.1080/02796015.2007.12087919

Klingbeil

D. A.

Van Norman

E. R.

Nelson

P. M.

Birr

(2019). Interval likelihood ratios: Applications for gated screening in schools. Journal of School Psychology, 76, 107–123. https://doi.org/10.1016/j.jsp.2019.07.016

10.

Meehl

P. E.

Rosen

(1955). Antecedent probability and the efficiency of psychometric signs, patterns, or cutting scores. Psychological Bulletin, 52(3), 194–216. https://doi.org/10.1037/h0048070

11.

Nelson

P. M.

Van Norman

E. R.

VanDerHeyden

(2017). Reduce, reuse, recycle: The longitudinal value of local cut scores using state test data. Journal of Psychoeducational Assessment, 35(7), 683–694. https://doi.org/10.1177/0734282916658567

12.

Northwest Evaluation Association. (2014). Technical manual for measures of academic progress and measures of academic progress for primary grades.

13.

Northwest Evaluation Association. (2017). Linking the Wisconsin forward exam to the NWEA MAP tests. https://www.nwea.org/content/uploads/2017/02/WI-MAP-Growth-Linking-Study-JAN2017.pdf

14.

Pauker

S. G.

Kassirer

J. P.

(1980). The threshold approach to clinical decision-making. New England Journal of Medicine, 302(20), 1109–1116. https://doi.org/10.1056/nejm198005153022003

15.

Pendergast

L. L.

Youngstrom

E. A.

Ruan-Iu

Beysolow

(2018). The nomogram: A decision-making tool for practitioners using multitiered systems of support. School Psychology Review, 47(4), 345–359.

16.

R Core Team. (2020). R: A language and environment for statistical computing. Author. https://www.R-project.org/

17.

Sonis

(1999). How to use and interpret likelihood ratios. Family Medicine, 31(6), 432–437.

18.

VanDerHeyden

A. M.

(2011). Technical adequacy of response to intervention decisions. Exceptional Children, 77(3), 335–350. https://doi.org/10.1177/001440291107700305

19.

VanDerHeyden

A. M.

(2013). Universal screening may not be for everyone: Using a threshold model as a smarter way to determine risk. School Psychology Review, 42(4), 402–414. https://doi.org/10.1080/02796015.2013.12087462

20.

VanDerHeyden

A. M.

Burns

M. K.

(2018). Improving decision making in school psychology: Making a difference in the lives of students, not just a predication about their lives. School Psychology Review, 47(4), 385–395. https://doi.org/10.17105/SPR-2018-0042.V47-4

21.

VanDerHeyden

A. M.

Codding

R. S.

Martin

(2017). Relative value of common screening measures in mathematics. School Psychology Review, 46(1), 65–87. https://doi.org/10.17105/SPR46-1.65-87

22.

Van Norman

E. R.

Klingbeil

D. A.

Nelson

P. M

. (2017). Post-test probabilities: An empirical demonstration of their use in evaluating the performance of universal screening measures across settings. School Psychology Review, 46(4), 349–362. https://doi.org/10.17105/SPR-2017-0046.V46-4

23.

Wisconsin Department of Public Instruction. (2018). Wisconsin Forward Exam Spring 2018 Technical Report. https://dpi.wi.gov/sites/default/files/imce/assessment/pdf/Forward_Exam_Tech_Report_2018.pdf

24.

Youngstrom

E. A.

(2013). Future directions in psychological Assessment: Combining evidence-based medicine innovations with psychology’s historical strengths to enhance utility. Journal of Child Clinical and Adolescent Psychology, 42(1), 139–159. https://doi.org/10.1080/15374416.2012.736358

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.35 MB