Diagnostic Accuracy of Multivariate Universal Screening Procedures for Reading in Upper Elementary Grades

Abstract

We examined the diagnostic accuracy and efficiency of three approaches to universal screening for reading difficulties using retrospective data from 1,307 students in Grades 3 through 5. School staff collected screening data using the Measures of Academic Progress (MAP), a curriculum-based measure (CBM), and running records (RR). The criterion measure was a high-stakes state accountability test aligned with the Common Core State Standards. We examined the diagnostic accuracy of the tests in isolation, as multivariate batteries, and via a simulated gated-screening approach. CBM and RR data resulted in unacceptable diagnostic accuracy across all three grades. In the fourth grade, the MAP alone resulted in the best balance of sensitivity and specificity. Among third- and fifth-grade students, the multivariate combination of MAP and CBM demonstrated the best balance between diagnostic accuracy and efficiency. Gated-screening increased specificity but lowered sensitivity. Results highlight the need for population-specific considerations in universal screening.

Keywords

identification assessment reading elementary school(s)

Get full access to this article

View all access options for this article.

References

Baker

D. L.

Biancarosa

Park

B. J.

Bousselot

Smith

Baker

S. K.

. . . Tindal

(2015). Validity of CBM measures of oral reading fluency and reading comprehension on high-stakes reading assessments in Grades 7 and 8. Reading and Writing, 28, 57–105. doi:10.1007/s11145-014-9505-4

Bossuyt

P. M.

Reitsma

J. B.

Bruns

D. E.

Gatsonis

C. A.

Glasziou

P. P.

Irwig

L. M.

. . . de Vet

H. C. W.

(2003). Toward complete and accurate reporting of studies of diagnostic accuracy: The STARD initiative. American Journal of Clinical Pathology, 119, 18–22. doi:10.1092/8exccm6yr1thubaf

Catts

H. W.

Fey

M. E.

Zhang

Tomblin

J. B.

(2001). Estimating the risk of future reading difficulties in kindergarten children: Research-based model and its clinical implementation. Language, Speech, and Hearing Services in Schools, 32, 38–50. doi:10.1044/0161-1461

Clemens

N. H.

Keller-Margulis

M. A.

Scholten

Yoon

(2016). Screening assessment within a multi-tiered system of support: Current practices, advances, and next steps. In Jimerson

S. R.

Burns

M. K.

VanDerHeyden

A. M.

(Eds.), Handbook of response to intervention: The science and practice of multi-tiered systems of support (pp. 187–213). New York, NY: Springer.

Compton

D. L.

Fuchs

L. S.

Bouton

Gilbert

J. K.

Barquero

L. A.

. . . Crouch

R. C.

(2010). Selecting at-risk first-grade readers for early intervention: Eliminating false positives and exploring the promise of a two-stage gated screening process. Journal of Educational Psychology, 102, 327–340. doi:10.1037/a0018448

Compton

D. L.

Fuchs

L. S.

Bryant

J. D.

(2006). Selecting at risk readers in first grade for early intervention: A two-year longitudinal study of decision rules and procedures. Journal of Educational Psychology, 98, 394–409. doi:10.1037/0022-0663.98.2.394

Decker

D. M.

Hixson

M. D.

Shaw

Johnson

(2014). Classification accuracy of oral reading fluency and maze in predicting performance on large-scale reading assessments. Psychology in the Schools, 51, 625–635.

Deeks

J. J.

Altman

D. G.

(2004). Diagnostic tests 4: Likelihood ratios. British Medical Journal, 429, 168–169. doi: 10.1136/bmj.329.7458.168

Denton

C. A.

Barth

A. E.

Fletcher

J. M.

Wexler

Vaughn

Cirino

P. T.

. . . Francis

D. J.

(2011). The relations among oral and silent reading fluency and comprehension in middle school: Implications for identification and instruction of students with reading difficulties. Scientific Studies of Reading, 15, 109–135. doi:10.1080/10888431003623546

10.

Espin

Wallace

Lembke

Campbell

Long

J. D.

(2010). Creating a progress-monitoring system in reading for middle-school students: tracking progress toward meeting high-stakes standards. Learning Disabilities Research & Practice, 25, 60–75.

11.

Fuchs

L. S.

Compton

D. L.

(2012). Smart RTI: A next-generation approach to multilevel prevention. Exceptional Children, 78, 263–279. doi:10.1177/001440291207800301

12.

Fuchs

L. S.

Fuchs

Compton

D. L.

(2010). Rethinking response to intervention at middle and high school. School Psychology Review, 39, 22–28.

13.

Fuchs

L. S.

Vaughn

(2012). Responsiveness-to-intervention: A decade later. Journal of Learning Disabilities, 45, 195–203. doi:10.1177/0022219412442150

14.

Glover

T. A.

Albers

C. A.

(2007). Considerations for evaluating universal screening assessments. Journal of School Psychology, 45, 117–135. doi:10.1016/j.jsp.2006.05.005

15.

Harper

Reeves

(1999). Reporting of precision of estimates for diagnostic accuracy: A review. British Medical Journal, 318, 1322–1323.

16.

Jenkins

J. R.

Hudson

R. F.

Johnson

E. S.

(2007). Screening for service delivery in an RTI framework: Candidate measures. School Psychology Review, 36, 560–582.

17.

Jenkins

J. R.

Jewell

(1993). Examining the validity of two measures for formative teaching: Reading aloud and maze. Exceptional Children, 59, 421–432. doi:10.1177/001440299305900505

18.

Jenkins

J. R.

Schiller

Blackorby

Thayer

S. K.

Tilly

W. D.

(2013). Responsiveness to intervention in reading: Architecture and practices. Learning Disability Quarterly, 36, 36–46. doi:10.1177/0731948712464963

19.

Johnson

E. S.

Jenkins

J. R.

Petscher

Catts

H. W.

(2009). How can we improve the accuracy of screening instruments? Learning Disabilities Research & Practice, 24, 174–185.

20.

Kilgus

S. P.

Methe

S. A.

Maggin

D. M.

Tomasula

J. L.

(2014). Curriculum-based measurement of oral reading (R-CBM): A diagnostic test accuracy meta-analysis of evidence supporting use in universal screening. Journal of School Psychology, 52, 377–405. doi:10.1016/j.jsp.2014.06.002

21.

Lopez-Raton

Rodriguez-Alvarez

M. X.

Suarez

C. C.

Sampedro

F. G.

(2014). OptimalCutpoints: An R package for selecting optimal cutpoints in diagnostic tests. Journal of Statistical Software, 61, 1–36.

22.

McGlinchey

M. T.

Hixson

M. D.

(2004). Using curriculum-based measurement to predict performance on state assessments in reading. School Psychology Review, 33, 193–203.

23.

Mellard

McKnight

Woods

(2009). Response to intervention screening and progress-monitoring practices in 41 local schools. Learning Disabilities Research & Practice, 24, 186–195. doi:10.1111/j.1540-5826.2009.00292.x

24.

National Governors Association Center for Best Practices & Council of Chief State School Officers. (2010). Common Core State Standards for mathematics. Washington, DC: Author.

25.

Nelson

P. M.

Van Norman

E. R.

Lackner

S. K.

(2016). A comparison of methods to screen middle school students for reading and math difficulties. School Psychology Review, 45, 327–342.

26.

Nese

J. F.

Park

B. J.

Alonzo

Tindal

(2011). Applied curriculum-based measurement as a predictor of high-stakes assessment: Implications for researchers and teachers. The Elementary School Journal, 111, 608–624. doi:10.1086/659034

27.

Northwest Evaluation Association. (2011). Technical manual for measures of academic progress (MAP) and measures of academic progress for primary grades (MPG). Portland, OR: Author.

28.

Northwest Evaluation Association. (2015). Smarter balanced preliminary performance levels. Portland, OR: Author.

29.

Pearson Inc. (2011). AIMSweb state prediction user’s guide. Bloomington, MN: Author.

30.

Pearson Inc. (2012). AIMSweb technical manual. Bloomington, MN: Author.

31.

Prewett

Mellard

D. F.

Deshler

D. D.

Allen

Alexander

Stern

(2012). Response to intervention in middle schools: Practices and outcomes. Learning Disabilities Research & Practice, 27, 136–147.

32.

R Core Team. (2015). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Available from http://www.R-project.org/

33.

Shapiro

E. S.

Solari

Petscher

(2008). Use of a measure of reading comprehension to enhance prediction on the state high stakes assessment. Learning and Individual Differences, 18, 316–328. doi:10.1016/j.lindif.2008.03.002

34.

Silberglitt

Burns

M. K.

Madyun

N. H.

Lail

K. E.

(2006). Relationship of reading fluency assessment data with state accountability test scores: A longitudinal comparison of grade levels. Psychology in the Schools, 43, 527–535.

35.

Smarter Balanced Assessment Consortium. (2016). 2014–2015 technical report. Los Angeles, CA: Author.

36.

Speece

D. L.

Ritchey

K. D.

Silverman

Schatschneider

Walker

C. Y.

Andrusik

K. N.

(2010). Identifying children in middle childhood who are at risk for reading problems. School Psychology Review, 39, 258–276.

37.

Stage

S. A.

Jacobsen

M. D.

(2001). Predicting student success on a state-mandated performance-based assessment using oral reading fluency. School Psychology Review, 30, 407–419.

38.

Stevenson

N. A.

Reed

D. K.

Tighe

E. L.

(2016). Examining potential bias in screening measures for middle school students by special education and low socioeconomic status subgroups. Psychology in the Schools, 53, 533–547. doi:10.1002/pits.21919

39.

Teachers College Reading and Writing Project. (2012). Benchmark reading levels and marking period assessments. Retrieved from http://readingandwritingproject.org/resources/assessments/running-records

40.

Teachers College Reading and Writing Project. (2014). Teacher resources and guidebook for levels A-K reading level assessments. Retrieved from http://readingandwritingproject.org/resources/assessments/running-records

41.

Thum

Y. M.

Hauser

C. H.

(2015). NWEA 2015 MAP norms for student and school achievement status and growth. Portland, OR: Northwest Evaluation Association.

42.

VanDerHeyden

A. M.

(2011). Technical adequacy of RtI decisions. Exceptional Children, 77, 335–350. doi:10.1177/001440291107700305

43.

VanDerHeyden

A. M.

(2013). Universal screening may not be for everyone: Using a threshold model as a smarter way to determine risk. School Psychology Review, 42, 402–414.

44.

Van Norman

E. R.

Nelson

P. M.

Klingbeil

D. A

. (2016). Single measure and gated screening approaches for identifying students at-risk for academic problems: Implications for sensitivity and specificity. School Psychology Quarterly. Advance online publication. doi:10.1037/spq0000177

45.

Yovanoff

Duesbery

Alonzo

Tindal

(2005). Grade-level invariance of a theoretical causal structure predicting reading comprehension with vocabulary and oral reading fluency. Educational Measurement, 24, 4–12. doi:10.1111/j.1745-3992.2005.00014.x