Assessing the Classification Accuracy of Early Numeracy Curriculum-Based Measures Using Receiver Operating Characteristic Curve Analysis

Abstract

Receiver operating characteristic curve (ROC) analysis was used to investigate the ability of early numeracy curriculum-based measures (EN-CBM) administered in preschool to predict performance below the 25th and 40th percentiles on a quantity discrimination measure in kindergarten. Areas under the curve derived from a sample of 279 students ranged from .65 to .82. Optimal cut scores were associated with sensitivity ranging from .44 to .79, specificity ranging from .49 to .91, and classification accuracy ranging from .60 to .79. Among a validation sample of 140 students, sensitivity ranged from .36 to .83, specificity ranged from .46 to .89, and classification accuracy ranged from .57 to .79. Results indicate that EN-CBM administered in preschool predicts math performance in kindergarten and can be a valuable component of an evidence-based decision-making process.

Keywords

early numeracy curriculum-based measurement

Get full access to this article

View all access options for this article.

References

Baglici

S. P.

Codding

Tryon

(2010). Extending the research on the Tests of Early Numeracy: Longitudinal analyses over two school years. Assessment for Effective Intervention, 35, 89–102. doi:10.1177/1534508409346053

Barnett

D. W.

VanDerHeyden

A. M.

Witt

J. C.

(2007). Achieving science-based practice through response to intervention: What it might look like in preschools. Journal of Educational and Psychological Consultation, 17, 31–54. doi:10.1080/10474410709336589

Betts

Pickart

Heistad

(2009). Construct and predictive validity evidence for curriculum-based measures of early literacy and numeracy skills in kindergarten. Journal of Psychoeducational Assessment, 27, 83–95. doi:10.1177/0734282908323398

Black

Wiliam

(1998). Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan, 80, 139–144.

Chard

D. J.

Clarke

Baker

Otterstedt

Braun

Katz

(2005). Using measures of number sense to screen for difficulties in mathematics: Preliminary findings. Assessment for Effective Intervention, 30, 3–14. doi:10.1177/073724770503000202

Clarke

Baker

Smolkowski

Chard

D. J.

(2008). An analysis of early numeracy curriculum-based measurement: Examining the role of growth in student outcomes. Remedial and Special Education, 29, 46–57. doi:10.1177/0741932507309694

Clarke

Nese

J. F. T.

Alonzo

Smith

J. L. M.

Tindal

Kame’enui

E. J.

Baker

S. K.

(2011). Classification accuracy of easyCBM first-grade mathematics measures: Findings and implications for the field. Assessment for Effective Intervention, 36, 243–255. doi:10.1177/1534508411414153

Clarke

Shinn

M. R.

(2002). Test of Early Numeracy (TEN): Administration and scoring of AIMSweb early numeracy measures for use with AIMSweb. Eden Prairie, MN: AIMSweb.

Clarke

Shinn

M. R.

(2004). A preliminary investigation into the identification and development of early mathematics curriculum-based measurement. School Psychology Review, 33, 234–248.

10.

Clements

D. H.

Sarama

(2009). Learning and teaching early math: The learning trajectories approach. New York, NY: Routledge.

11.

Compton

D. L.

Fuchs

L. S.

Bryant

J. D.

(2006). Selecting at-risk readers in first grade for early intervention: A two-year longitudinal study of decision rules and procedures. Journal of Educational Psychology, 98, 394–409. doi:10.1037/0022-0663.98.2.394

12.

Duncan

G. J.

Dowsett

C. J.

Claessens

Magnuson

Huston

A. C.

Klebanov

. . . Japel

(2007). School readiness and later achievement. Developmental Psychology, 43, 1428–1446. doi:10.1037/0012-1649.43.6.1428

13.

Floyd

R. G.

Hojnoski

Key

(2006). Preliminary evidence of the technical adequacy of the preschool numeracy indicators. School Psychology Review, 35, 627–644.

14.

Foegen

Jiban

Deno

(2007). Progress monitoring measures in mathematics: A review of the literature. The Journal of Special Education, 41, 121–139. doi:10.1177/00224669070410020101

15.

Fuchs

L. S.

Fuchs

Compton

D. L.

Bryant

J. D.

Hamlett

C. L.

Seethaler

P. M.

(2007). Mathematics screening and progress monitoring at first grade: Implications for responsiveness to intervention. Exceptional Children, 73, 311–330.

16.

Gersten

Chard

(1999). Number sense: Rethinking arithmetic instruction for students with mathematical disabilities. The Journal of Special Education, 33, 18–28. doi:10.1177/002246699903300102

17.

Ginsburg

Baroody

(2003). Test of early mathematics ability: Third edition. Austin, TX: Pro-Ed.

18.

Greenwood

C. R.

Carta

J. J.

McConnell

(2011). Advances in measurement for universal screening and individual progress monitoring of young children. Journal of Early Intervention, 33, 254–267. doi:10.1177/1053815111428467

19.

Greenwood

C. R.

Carta

J. J.

Walker

Hughes

Weathers

(2006). Preliminary investigations of the application of the Early Communication Indicator (ECI) for infants and toddlers. Journal of Early Intervention, 28, 178–196. doi:10.1177/105381510602800306

20.

Greenwood

C. R.

Walker

Buzhardt

(2010). The early communication indicator for infants and toddlers: Early Head Start growth norms from two states. Journal of Early Intervention, 32, 310–334. doi:10.1177/1053815110392335

21.

Hanley

J. A.

McNeil

B. J.

(1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143, 29–36.

22.

Hintze

J. M.

Ryan

A. L.

Stoner

(2003). Concurrent validity and diagnostic accuracy of the Dynamic Indicators of Basic Early Literacy Skills and the Comprehensive Test of Phonological Processing. School Psychology Review, 32, 541–556.

23.

Hintze

J. M.

Silberglitt

(2005). A longitudinal examination of the diagnostic accuracy and predictive validity of R-CBM and high-stakes testing. School Psychology Review, 34, 372–386.

24.

Hojnoski

R. L.

Floyd

R. G.

(2013). Individual growth and development indicators of early numeracy (IGDIs-EN). Minneapolis, MN: Early Learning Labs. Available from http://www.myigdis.com

25.

Hojnoski

R. L.

Silberglitt

Floyd

R. G.

(2009). Sensitivity to growth over time of the preschool numeracy indicators with a sample of preschoolers in head start. School Psychology Review, 38, 402–418.

26.

Howell

S. C.

Kemp

C. R.

(2010). Assessing preschool number sense: Skills demonstrated by children prior to school entry. Educational Psychology, 30, 411–429. doi:10.1080/01443411003695410

27.

IBM Corp. (2013). IBM SPSS Statistics for Windows (Version 22.0). Armonk, NY: Author.

28.

Jenkins

J. R.

Hudson

R. F.

Johnson

E. S.

(2007). Screening for at-risk readers in a response to intervention framework. School Psychology Review, 36, 582–600.

29.

Johnson

E. S.

Jenkins

J. R.

Petscher

(2010). Improving the accuracy of a direct route screening process. Assessment for Effective Intervention, 35, 131–140. doi:10.1177/1534508409348375.

30.

Jordan

N. C.

Kaplan

Ramineni

Locuniak

M. N.

(2009). Early math matters: Kindergarten number competence and later mathematics outcomes. Developmental Psychology, 45, 850–867. doi:10.1037/a0014939

31.

Kettler

R. J.

Albers

C. A.

(2013). Predictive validity of curriculum-based measurement and teacher ratings of academic achievement. Journal of School Psychology, 51, 499–515. doi:10.1016/j.jsp.2013.02.004

32.

Lei

DiPerna

J. C.

Morgan

P. L.

(2009). Developing short forms of the EARLI numeracy measures: Comparison of item selection methods. Educational and Psychological Measurement, 69, 825–842. doi:10.1177/0013164409332215

33.

Lembke

Foegen

(2009). Identifying early numeracy indicators for kindergarten and first-grade students. Learning Disabilities Research & Practice, 24, 12–20. doi:10.1111/j.1540-5826.2008.01273.x

34.

McConnell

Bradfield

Wackerle-Hollman

Rodriguez

(2012). Individual growth and development indicators of early literacy (IGDIs-EL) (2nd ed.). St. Paul, MN: Early Learning Labs.

35.

McFall

R. M.

Treat

T. A.

(1999). Quantifying the information value of clinical assessments with signal detection theory. Annual Review of Psychology, 50, 215–241. doi:10.1146/annurev.psych.50.1.215

36.

Methe

S. A.

Begeny

J. C.

Leary

L. L.

(2011). Development of conceptually focused early numeracy skill indicators. Assessment for Effective Intervention, 36, 230–242. doi:10.1177/1534508411414150

37.

Methe

S. A.

Hintze

J. M.

Floyd

R. G.

(2008). Validation and decision accuracy of early numeracy skill indicators. School Psychology Review, 37, 359–373.

38.

Methe

S. A.

Hojnoski

Clarke

Owens

B. B.

Lilley

P. K.

Politylo

B. C.

. . . Marcotte

A. M.

(2011). Innovations and future directions for early numeracy curriculum-based measurement: Commentary on the special series. Assessment for Effective Intervention, 36, 200–209. doi:10.1177/1534508411414154

39.

Missall

K. N.

Mercer

S. H.

Martinez

R. S.

Casebeer

(2012). Concurrent and longitudinal patterns and trends in performance on early numeracy curriculum-based measures in kindergarten through third grade. Assessment for Effective Intervention, 37, 95–106. doi:10.1177/1534508411430322

40.

National Mathematics Advisory Panel. (2008). Foundations for success: The final report of the National Mathematics Advisory Panel. Washington, DC: U.S. Department of Education.

41.

Obuchowski

N. A.

(2005). Fundamentals of clinical research for radiologists: ROC analysis. American Journal of Roentgenology, 184, 364–372. doi:10.2214/ajr.184.2.01840364

42.

Polignano

J. C.

Hojnoski

R. L.

(2012). Preliminary evidence of the technical adequacy of additional curriculum-based measures for preschool mathematics. Assessment for Effective Intervention, 37, 70–83. doi:10.1177/1534508411430323

43.

Purpura

D. J.

Hume

L. E.

Sims

D. M.

Lonigan

C. J.

(2011). Early literacy and early numeracy: The value of including early literacy skills in the prediction of numeracy development. Journal of Experimental Child Psychology, 110, 647–658. doi:10.1016/j.jecp.2011.07.004

44.

Purpura

D. J.

Reid

E. E.

Eiland

M. D.

Baroody

A. J.

(2015). Using a brief preschool early numeracy skills screener to identify young children with mathematics difficulties. School Psychology Review, 44, 41–59.

45.

Reid

E. E.

Morgan

P. L.

DiPerna

J. C.

Lei

P. W.

(2006). Development of measures to assess young children’s early academic skills: Preliminary findings from a Head Start-university partnership. Insights on Learning Disabilities, 3(2), 25–38.

46.

Speece

Mills

Ritchey

Hillman

(2003). Initial evidence that letter fluency tasks are valid indicators of early reading skill. The Journal of Special Education, 36, 223–233. doi:10.1177/002246690303600403

47.

VanDerHeyden

A. M.

Broussard

Cooley

(2006). Further development of measures of early math performance for preschoolers. Journal of School Psychology, 44, 533–553. doi:10.1016/j.jsp.2006.07.003

48.

VanDerHeyden

A. M.

Broussard

Fabre

Stanley

Legendre

Creppell

(2004). Development and validation of curriculum-based measures of math performance for preschool children. Journal of Early Intervention, 27, 27–41. doi:10.1177/105381510402700103

49.

VanDerHeyden

A. M.

Snyder

(2006). Integrating frameworks from early childhood intervention and school psychology to accelerate growth for all young children. School Psychology Review, 35, 519–534.

50.

Woodcock

R. W.

Johnson

M. B.

(1989). Woodcock Johnson Psychoeducational Battery-Revised. Allen, TX: DLM Teaching Resources.

51.

Woodcock

R. W.

McGrew

K. S.

Mather

(2001). Woodcock-Johnson III tests of achievement. Itasca, IL: Riverside.

52.

Lei

DiPerna

J. C.

Morgan

P. L.

Reid

E. E.

(2015). Identifying differences in early mathematical skills among children in Head Start. International Journal of Science and Mathematics Education, 13, 1403–1423. doi:10.1007/s10763-014-9552-y

53.

Youngstrom

E. A.

(2014). A primer on receiver operating characteristic analysis and diagnostic efficiency statistics for pediatric psychology: We are ready to ROC. Journal of Pediatric Psychology, 39, 204–221. doi:10.1093/jpepsy/jst062