Evaluating Procedures for Reducing Measurement Error in Math Curriculum-Based Measurement Probes

Abstract

At present, it is unclear whether math curriculum-based measurement (M-CBM) procedures provide a dependable measure of student progress in math computation because support for its technical properties is based largely upon a body of correlational research. Recent investigations into the dependability of M-CBM scores have found that evaluating probe design procedures is necessary to estimate the contribution of probe-related factors to variance in M-CBM scores. To extend this research, the present study compared a set of commercially available probes (i.e., AIMSweb) with an experimental set that was constructed to enhance consistency in item content across alternative forms. Two probes were randomly selected from each set and administered to 43 students in fifth grade. Generalizability analyses indicated that the experimental probes generated scores with less error variance than those of the AIMSweb probes. Follow-up analyses indicated that fewer administrations of the experimental probes were necessary to attain adequate reliability for relative decisions (i.e., >.80) and high reliability for absolute decisions (i.e., >.90). Results support prior studies and suggest that error variance is associated with M-CBM probe content and development procedures.

Keywords

curriculum-based measurement math achievement assessment

Get full access to this article

View all access options for this article.

References

Allinder

R. M.

Fuchs

L. S.

Fuchs

Hamlett

C. L.

(1992). Effects of summer break on math and spelling performance as a function of grade level. The Elementary School Journal, 92, 451–460. Retrieved from http://dx.doi.org/10.1086/461701

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association.

American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed., text rev.). Washington, DC: Author.

Batsche

G. M.

Curtis

M. J.

Dorman

Castillo

J. M.

Porter

L. J.

(2007). The Florida problem-solving/response to intervention model: Implementing a statewide initiative. In Jimerson

S. R.

Burns

M. K.

VanDerHeyden

A. M.

(Eds.), Handbook of response to intervention (pp. 378–395). New York, NY: Springer. Retrieved from http://dx.doi.org/10.1007/978-0-387-49053-3_28

Betts

(2010). Expanding school psychology’s horizons for understanding and conducting research: Commentary on the special series on statistical analysis. Journal of School Psychology, 48, 1–4. Retrieved from http://dx.doi.org/10.1016/j.jsp.2009.11.001

Betts

Pickart

Heistad

(2009). An investigation of the psychometric evidence of CBM-R passage equivalence: Utility of readability statistics and equating for alternate forms. Journal of School Psychology, 47, 1–17. Retrieved from http://dx.doi.org/10.1016/j.jsp.2008.09.001

Brennan

R. L.

(2001). Generalizability theory. New York, NY: Springer. Retrieved from http://dx.doi.org/10.1007/978-1-4757-3456-0

Bricker

Yovanoff

Capt

Allen

(2003). Use of a curriculum-based measure to corroborate eligibility decisions. Journal of Early Intervention, 26, 20–30. Retrieved from http://dx.doi.org/10.1177/105381510302600102

Briesch

A. M.

Swaminathan

Welsh

Chafouleas

S. M.

(2014). Generalizability theory: A practical guide to study design, implementation, and interpretation. Journal of School Psychology, 52, 13–35. Retrieved from http://dx.doi.org/10.1016/j.jsp.2013.11.008

10.

Burns

M. K.

(2012). Assessment research and school psychology: Introduction to the special series. School Psychology Review, 41, 243–245.

11.

Christ

T. J.

Ardoin

S. P.

(2009). Curriculum-based measurement of oral reading: Passage equivalence and probe-set development. Journal of School Psychology, 47, 55–75. Retrieved from http://dx.doi.org/10.1016/j.jsp.2008.09.004

12.

Christ

T. J.

Johnson-Gros

K. N.

Hintze

J. M.

(2005). An examination of alternate assessment durations when assessing multiple-skill computational fluency: The generalizability and dependability of curriculum-based outcomes within the context of educational decisions. Psychology in the Schools, 42, 615–622. Retrieved from http://dx.doi.org/10.1002/pits.20107

13.

Christ

T. J.

Scullin

Tolbize

Jiban

C. L.

(2008). Implications of recent research: Curriculum-based measurement of math computation. Assessment for Effective Intervention, 33, 198–205. Retrieved from http://dx.doi.org/10.1177/1534508407313480

14.

Christ

T. J.

Vining

(2006). Curriculum-based measurement procedures to develop multiple-skill math computation probes: Evaluation of random and stratified stimulus-set arrangements. School Psychology Review, 35, 387–400.

15.

Cooney

J. B.

Swanson

H. L.

Ladd

S. F.

(1988). Acquisition of mental multiplication skill: Evidence for the transition between counting and retrieval strategies. Cognition and Instruction, 5, 323–345. Retrieved from http://dx.doi.org/10.1207/s1532690xci0504_5

16.

Crick

J. E.

Brennan

R. L.

(1983). Manual for GENOVA: A generalized analysis of variance system. Iowa City, IA: The American College Testing Program.

17.

Crocker

Algina

(1986). Introduction to classical and modern test theory. Orlando, FL: Holt, Rinehart, and Winston.

18.

Deno

S. L.

(2003). Developments in curriculum-based measurement. The Journal of Special Education, 37, 184–192. Retrieved from http://dx.doi.org/10.1177/00224669030370030801

19.

Deno

S. L.

Mirkin

P. K.

(1977). Data-based program modification: A manual. Reston, VA: Council for Exceptional Children.

20.

Dirks

Spyer

van Lieshout

E. C.

de Sonneville

(2008). Prevalence of combined reading and arithmetic disabilities. Journal of Learning Disabilities, 41, 460–473. Retrieved from http://dx.doi.org/10.1177/0022219408321128

21.

Downing

S. M.

(2006). Twelve steps for effective test development. In Downing

S. M.

Haladyna

T. M.

(Eds.), Handbook of test development (pp. 3–25). Mahwah, NJ: Lawrence Erlbaum.

22.

Foegen

Jiban

Deno

S. L.

(2007). Progress monitoring measures in math: A review of the literature. Journal of Special Education, 41, 121–139. Retrieved from http://dx.doi.org/10.1177/00224669070410020101

23.

Foegen

Olson

J. R.

Impecoven-Lind

(2008). Developing progress monitoring measures for secondary math: An illustration in algebra. Assessment for Effective Intervention, 33, 240–249. Retrieved from http://dx.doi.org/10.1177/1534508407313489

24.

Fuchs

L. S.

(2003). Assessing intervention responsiveness: Conceptual and technical issues. Learning Disabilities Research & Practice, 18, 172–186. Retrieved from http://dx.doi.org/10.1111/1540-5826.00073

25.

Fuchs

L. S.

(2004). The past, present, and future of curriculum based measurement research. School Psychology Review, 33, 188–192.

26.

Fuchs

L. S.

Deno

S. L.

(1991). Paradigmatic distinctions between instructionally relevant measurement models. Exceptional Children, 57, 488–500.

27.

Fuchs

L. S.

Fuchs

(1986). Effects of systematic formative evaluation: A meta-analysis. Exceptional Children, 53, 199–208.

28.

Fuchs

L. S.

Fuchs

Compton

D. L.

Bryant

J. D.

Hamlett

C. L.

Seethaler

P. M.

(2007). Math screening and progress monitoring at first grade: Implications for responsiveness to intervention. Exceptional Children, 73, 311–330. Retrieved from http://dx.doi.org/10.1177/001440290707300303

29.

Fuchs

L. S.

Fuchs

Hamlett

C. L.

Walz

Germann

(1993). Formative evaluation of academic progress: How much growth can we expect? School Psychology Review, 22, 27–48.

30.

Fuchs

L. S.

Fuchs

Phillips

Hamlett

Karns

(1995). Acquisition and transfer effects of classwide peer-assisted learning strategies in math for students with varying learning histories. School Psychology Review, 24, 604–620.

31.

Fuchs

L. S.

Fuchs

Powell

S. R.

Seethaler

P. M.

Cirino

P. T.

Fletcher

J. M.

(2008). Intensive intervention for students with math disabilities: Seven principles of effective practice. Learning Disability Quarterly, 31, 79–92.

32.

Fuchs

L. S.

Hamlett

C. L.

Fuchs

(1998). Monitoring basic skills progress: Basic math computation. Austin, TX: Pro-Ed.

33.

Geary

D. C.

(2004). Math and learning disabilities. Journal of Learning Disabilities, 37, 4–15. Retrieved from http://dx.doi.org/10.1177/00222194040370010201

34.

Geary

D. C.

Hoard

M. K.

Nugent

Bailey

D. H.

(2012). Mathematical cognition deficits in children with learning disabilities and persistent low achievement: A five-year prospective study. Journal of Educational Psychology, 104, 206–233. Retrieved from http://dx.doi.org/10.1037/a0025398

35.

Gersten

Beckmann

Clarke

Foegen

March

Star

J. R.

Witzel

(2009). Assisting students struggling with mathematics: Response to Intervention (RtI) for elementary and middle schools (Practice Guide Report). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

36.

Halfon

Houtrow

Larson

Newacheck

P. W.

(2012). The changing landscape of disability in childhood. The Future of Children, 22, 13–42. Retrieved from http://dx.doi.org/10.1353/foc.2012.0004

37.

Hauser

M. D.

Chomsky

Fitch

W. T.

(2002). The faculty of language: What is it, who has it, and how did it evolve? Science, 298, 1569–1579. Retrieved from http://dx.doi.org/10.1126/science.298.5598.1569

38.

Hintze

J. M.

Christ

T. J.

Keller

L. A.

(2002). The generalizability of CBM survey-level math assessments: Just how many samples do we need? School Psychology Review, 31, 514–528.

39.

Hosp

J. L.

(2012). Using assessment data to make decisions about teaching and learning. In Harris

K. R.

Graham

Urdan

(Eds.), APA educational psychology handbook: Application to learning and teaching (Vol. 3, pp. 87–109). Washington, DC: American Psychological Association.

40.

Keller-Margulis

Shapiro

E. S.

Hintze

J. M.

(2008). Long-term diagnostic accuracy of curriculum-based measures in reading and math. School Psychology Review, 37, 374–390.

41.

Lei

Smith

Suen

(2007). The use of generalizability theory to estimate data reliability in single-subject observational research. Psychology in the Schools, 44, 433–439. Retrieved from http://dx.doi.org/10.1002/pits.20235

42.

Lembke

Stecker

(2007). Curriculum-based measurement in math: An evidence-based formative assessment procedure. Portsmouth, NH: RMC Research Corporation, Center on Instruction.

43.

Linn

R. L.

(2006). The standards for educational and psychological testing: Guidance in test development. In Downing

S. M.

Haladyna

T. M.

(Eds.), Handbook of test development (pp. 27–38). Mahwah, NJ: Lawrence Erlbaum.

44.

Marston

D. B.

(1989). A curriculum-based measurement approach to assessing academic performance: What it is and why do it. In Shinn

M. R.

(Ed.), Curriculum-based measurement: Assessing special children (pp. 18–78). New York, NY: Guilford Press.

45.

Mazzocco

M. M. M.

(2007). Defining and differentiating mathematical learning disabilities and difficulties. In Berch

D. B.

Mazzocco

M. M. M.

(Eds.), Why is math so hard for some children? The nature and origins of mathematical learning difficulties and disabilities (pp. 7–27). Baltimore, MD: Paul H. Brookes.

46.

National Governors Association Center for Best Practices & Council of Chief State School Officers. (2010). Common core state standards for math. Washington, DC: Author.

47.

Seethaler

P. M.

Fuchs

L. S.

(2011). Using curriculum-based measurement to monitor kindergarteners’ math development. Assessment for Effective Intervention, 36, 219–229. Retrieved from http://dx.doi.org/10.1177/1534508411413566

48.

Shalev

R. S.

(2007). Prevalence of developmental dyscalculia. In Berch

D. B.

Mazzocco

M. M. M.

(Eds.), Why is math so hard for some children? The nature and origins of mathematical learning difficulties and disabilities (pp. 49–60). Baltimore, MD: Paul H. Brookes.

49.

Shapiro

E. S.

(2008). From research to practice: Promoting academic competence for underserved students. School Psychology Review, 37, 46–51.

50.

Shapiro

E. S.

Edwards

Zigmond

(2005). Progress monitoring of math among students with learning disabilities. Assessment for Effective Intervention, 30, 15–32. Retrieved from http://dx.doi.org/10.1177/073724770503000203

51.

Shinn

M. R.

(1989). Curriculum-based measurement: Assessing special children. New York, NY: Guilford Press.

52.

Shinn

M. R.

(2004). Administration and scoring of math computation curriculum-based measurement (M-CBM) and math fact probes for use with AIMSweb. Available from www.aimsweb.com

53.

Smith

P. L.

(1981). Gaining accuracy in generalizability theory: Using multiple designs. Journal of Educational Measurement, 18, 147–154. Retrieved from http://dx.doi.org/10.1111/j.1745-3984.1981.tb00849.x

54.

Stecker

P. M.

Fuchs

L. S.

(2000). Effecting superior achievement using curriculum-based measurement: The importance of individual progress monitoring. Learning Disabilities Research & Practice, 15, 128–134. Retrieved from http://dx.doi.org/10.1207/SLDRP1503_2

55.

Thurber

R. S.

Shinn

M. R.

Smolkowski

(2002). What is measured in math tests? Construct validity of curriculum-based math measures. School Psychology Review, 31, 498–513.

56.

VanDerHeyden

A. M.

(2011). Technical adequacy of response to intervention decisions. Exceptional Children, 77, 335–350. Retrieved from http://dx.doi.org/10.1177/001440291107700305

57.

VanDerHeyden

A. M.

Burns

M. K.

(2005). Using curriculum-based assessment and curriculum-based measurement to guide elementary math instruction: Effect on individual and group accountability scores. Assessment for Effective Intervention, 30, 15–31. Retrieved from http://dx.doi.org/10.1177/073724770503000302

58.

Vanderwood

M. L.

Linklater

Healy

(2008). Predictive accuracy of nonsense word fluency for English language learners. School Psychology Review, 37, 5.

59.

Volpe

R. J.

Briesch

A. M.

(2012). An investigation of the generalizability and dependability of single item and multiple item Direct Behavior Rating scales. School Psychology Review, 41, 246–261.

60.

Ysseldyke

J. E.

Vanderwood

M. L.

Shriner

(1997). Changes over the past decade in special education referral to placement probability: An incredibly reliable practice. Assessment for Effective Intervention, 23, 193–201.