The Influence of Measurement Error Associated With Oral Reading Progress Monitoring Measures on the Consistency and Accuracy of Nonparametric Single-Case Design Effect Size Outcomes

Abstract

Researchers and practitioners frequently use curriculum-based measures of reading (CBM-R) within single-case design (SCD) frameworks to evaluate the effects of reading interventions with individual students. Effect sizes (ESs) developed specifically for SCDs are often used as a supplement to visual analysis to gauge treatment effects. The degree to which measurement error associated with academic measures like CBM-R influences said ESs has not been fully explored. We used simulation methodology to evaluate how common magnitudes of error influenced the consistency and accuracy of outcomes from two nonparametric SCD ESs, percentage of data exceeding baseline trend and TauU. After accounting for other data characteristics, measurement error accounted for a statistically and practically significant amount of variance in the consistency and accuracy of outcomes from both ESs. This article suggests that the psychometric properties of academic measures are important to consider when interpreting ESs from SCDs.

Keywords

curriculum-based measurement single-case design effect sizes

Get full access to this article

View all access options for this article.

References

Albers

C. A.

Hoffman

(2012). Using flashcard drill methods and self-graphing procedures to improve the reading performance of English language learners. Journal of Applied School Psychology, 28, 367–388.

Allison

D. B.

Gorman

B. S.

(1993). Calculating effect sizes for meta-analysis: The case of the single case. Behavior Research and Therapy, 31, 621–631.

Ardoin

S. P.

Christ

T. J.

(2009). Curriculum-based measurement of oral reading: Standard errors associated with progress monitoring outcomes from dibels, aimsweb, and an experimental passage set. School Psychology Review, 38, 266–283.

Ardoin

S. P.

Christ

T. J.

Morena

L. S.

Cormier

D. C.

Klingbeil

D. A.

(2013). A systematic review and summarization of the recommendations and research surrounding curriculum-based measurement of oral reading fluency (cbm-r) decision rules. Journal of School Psychology, 51, 1–18. doi:10.1016/j.jsp.2012.09.004

Ardoin

S. P.

Roof

C. M.

Klubnick

Carfolite

(2008). Evaluating curriculum-based measurement from a behavioral assessment perspective. The Behavior Analyst Today, 9, 36–49.

Barnett

D. W.

Daly

E. J.

III. Jones

K. M.

Lentz

E. F.

(2004). Response to intervention: Empirically based special service decisions from single-case designs of increasing and decreasing intensity. The Journal of Special Education, 38, 66–79. doi:10.1177/00224669040380020101

Betts

Pickart

Heistad

(2009). An investigation of the psychometric evidence of cbm-r passage equivalence: Utility of readiability statistics and equating for alternate forms. Journal of School Psychology, 47, 1–17.

Bottai

Cai

McKeown

R. E.

(2010). Logistic quantile regression for bounded outcomes. Statistics in Medicine, 29, 309–317. doi:10.1002/sim.3781

Carnine

(1995). Trustworthiness, useability, and accessibility of educational research. Journal of Behavioral Education, 5, 251–258. doi:10.1007/BF02110314

10.

Christ

T. J.

(2006). Short-term estimates of growth using curriculum-based measurement of oral reading fluency: Estimating standard error of the slope to construct confidence intervals. School Psychology Review, 35, 128–133.

11.

Christ

T. J.

Ardoin

S. P.

(2009). Curriculum-based measurement of oral reading: Passage equivalence and probe-set development. Journal of School Psychology, 47, 55–75. doi:10.1016/j.jsp.2008.09.004

12.

Christ

T. J.

Silberglitt

(2007). Estimates of the standard error of measurement for curriculum-based measures of oral reading fluency. School Psychology Review, 36, 130–146.

13.

Christ

T. J.

Van Norman

E. R.

Nelson

P. M.

(2016). Foundations of fluency-based assessments in behavioral and psychometric paradigms In Cummings

K. D.

Petscher

(Eds.), The fluency construct: Curriculum-based measurement concepts and applications (pp. 143–163). New York, NY: Springer.

14.

Christ

T. J.

White

M. J.

Ardoin

S. P.

Eckert

T. L.

(2013). Curriculum based measurement of reading: Consistency and validity across best, fastest, and question reading conditions. School Psychology Review, 42, 415–436.

15.

Christ

T. J.

Zopluoglu

Monaghen

B. D.

Van Norman

E. R.

(2013). Curriculum-based measurement of oral reading: Multi-study evaluation of schedule, duration, and dataset quality on progress monitoring outcomes. Journal of School Psychology, 51, 19–57. doi:10.1016/j.jsp.2012.11.001

16.

Colon

E. P.

Kranzler

J. H.

(2006). Effect of instructions on curriculum-based measurement of reading. Journal of Psychoeducational Assessment, 24, 318–328. doi:10.1177/0734282906287830

17.

Cone

J. D.

(1977). The relevance of reliability and validity for behavioral assessment. Behavior Therapy, 8, 411–426.

18.

Cronbach

L. J.

Glesser

G. C.

(1964). The signal/noise ratio in the comparison of reliability coefficients. Educational and Psychological Measurement, 24, 467–480.

19.

Cronbach

L. J.

Gleser

G. C.

Nanda

Rajaratnam

(1972). The dependability of behavioral measurements. New York, NY: Wiley.

20.

Cummings

K. D.

Biancarosa

Schaper

Reed

D. K.

(2014). Examiner error in curriculum-based measurement of oral reading. Journal of School Psychology, 52, 361–375. doi:10.1016/j.jsp.2014.05.007

21.

Cummings

K. D.

Park

Bauer Schaper

H. A.

(2013). Form effects on dibels next oral reading fluency progress monitoring passages. Assessment for Effective Intervention, 38, 91–104. doi:10.1177/1534508412447010

22.

Cummings

K. D.

Stoolmiller

M. L.

Baker

S. K.

Fien

Kame’enui

E. J.

(2015). Using school-level student achievement to engage in formative evaluation: Comparative school-level rates of oral reading fluency growth conditioned by initial skill for second grade students. Reading and Writing, 28, 105–130. doi:10.1007/s11145-014-9512-5

23.

Deno

S. L.

(1985). Curriculum-based measurement: The emerging alternative. Exceptional Children, 52, 219–232.

24.

Deno

S. L.

(1986). Formative evaluation of individual student programs: A new role for school psychologists. School Psychology Review, 15, 358–374.

25.

Deno

S. L.

(1990). Individual differences and individual difference: The essential difference of special education. Journal of Special Education, 24, 160–173.

26.

Deno

S. L.

(2003). Developments in curriculum-based measurement. Journal of Special Education, 37, 184–192.

27.

Deno

S. L.

Fuchs

L. S.

Marston

Shin

(2001). Using curriculum-based measurement to establish growth standards for students with learning disabilities. School Psychology Review, 30, 507–524.

28.

Derr

T. F.

Shapiro

E. S.

(1989). A behavioral evaluation of curriculum-based assessment of reading. Journal of Psychoeducational Assessment, 7, 148–160. doi:10.1177/073428298900700205

29.

Every Student Succeeds Act of 2015, Pub. L. No. 114-95 § 114 Stat. 1177 (2015).

30.

Francis

D. J.

Santi

K. L.

Barr

Fletcher

J. M.

Varisco

Foorman

B. R.

(2009). Form effects on the estimation of students’ oral reading fluency using dibels. Journal of School Psychology, 46, 315–342.

31.

Fuchs

L. S.

Stecker

P. M.

(2010). The blurring of special education in a new continuum of general education placements and services. Exceptional Children, 76, 301–323.

32.

Fuchs

L. S.

Fuchs

Hosp

M. K.

Jenkins

J. R.

(2001). Oral reading fluency as an indicator of reading competence: A theoretical, empirical, and historical analysis. Scientific Studies of Reading, 5, 239–256. doi:10.1207/S1532799XSSR0503_3

33.

Fuchs

L. S.

Fuchs

Speece

D. L.

(2002). Treatment validity as a unifying construct for learning disabilities. Learning Disability Quarterly, 25, 33–45.

34.

Gast

D. L.

(2014). General factors in measurement and evaluation. In Gast

D. L.

Ledford

J. R.

(Eds.), Single case research methodology: Applications in special education and behavioral science (pp. 85–104). New York, NY: Routledge.

35.

Good

R. H.

Shinn

M. R.

(1990). Forecasting accuracy of slope estimates for reading curriculum-based measurement: Empirical evidence. Behavioral Assessment, 12, 179–193.

36.

Hedges

L. V.

(1981). Distribution theory for glass’s estimator of effect size and related estimators. Journal of Educational Statistics, 6, 107–128. doi:10.2307/1164588

37.

Hintze

J. M.

Christ

T. J.

(2004). An examination of variability as a function of passage variance in CBM progress monitoring. School Psychology Review, 33, 204–217.

38.

Horner

R. H.

Swaminathan

Sugai

Smolkowski

(2012). Considerations for the systematic analysis and use of single-case research. Education and Treatment of Children, 35, 269–290.

39.

Hua

Hendrickson

J. M.

Therrien

W. J.

Woods-Groves

Ries

P. S.

Shaw

J. J.

(2012). Effects of combined reading and question generation on reading fluency and comprehension of three young adults with autism and intellectual disability. Focus on Autism and Other Developmental Disabilities, 27, 135–146.

40.

Hunter

J. E.

Schmidt

F. L.

(2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: SAGE.

41.

Kazdin

A. E.

(2011). Single-case research designs: Methods for clinical and applied settings (2nd ed.). New York, NY: Oxford University Press.

42.

Kilgus

S. P.

Methe

S. A.

Maggin

D. M.

Tomasula

J. L.

(2014). Curriculum-based measurement of oral reading (R-CBM): A diagnostic test accuracy meta-analysis of evidence supporting use in universal screening. Journal of School Psychology, 52, 377–405. doi:10.1016/j.jsp.2014.06.002

43.

Klingbeil

D. A.

Van Norman

E. R.

McLendon

K. E.

Ross

S. G.

Begeny

J. C.

(2017). Evaluating tau-u with oral reading fluency data and the impact of measurement error. Manuscript submitted for publication.

44.

Kratochwill

T. R.

Hitchcock

Horner

R. H.

Levin

J. R.

Odom

S. L.

Rindskopf

D. M.

Shadish

W. R.

(2010). Single-case design technical documentation. Retrieved from https://ies.ed.gov/ncee/wwc/Docs/ReferenceResources/wwc_scd.pdf

45.

Kratochwill

T. R.

Levin

J. R.

(2014). Meta-and statistical analysis of single-case intervention research data: Quantitative gifts and a wish list. Journal of School Psychology, 52, 231–235. doi:10.1016/j.jsp.2014.01.003

46.

Lieberman

R. G.

Yoder

P. J.

Reichow

Wolery

(2010). Visual analysis of multiple baseline across participants graphs when change is delayed. School Psychology Quarterly, 25, 28–44. doi:10.1037/a0018600

47.

Cooke

N. L.

Starling

A. L. P.

(2011). Using a repeated reading program to improve generalization of oral reading fluency. Education & Treatment of Children, 34, 115–140.

48.

H. H.

(2006). An alternative method for quantitative synthesis of single-subject research: Percentage of data points exceeding the median. Behavior Modification, 30, 598–617.

49.

Maggin

D. M.

Swaminathan

Rogers

H. J.

O’Keeffe

B. V.

Sugai

Horner

R. H.

(2011). A generalized least squares regression approach for computing effect sizes in single-case research: Application examples. Journal of School Psychology, 49, 301–321. doi:10.1016/j.jsp.2011.03.004

50.

Moeyaert

Ferron

J. M.

Beretvas

Van den Noortgate

(2014). From a single-level analysis to multilevel analysis of single-case experimental designs. Journal of School Psychology, 52, 191–211. doi:10.1016/j.jsp.2013.11.003

51.

Musti-Rao

Plati

(2015). Using an iPad® app to improve sight word reading fluency for at-risk first graders. Remedial and Special Education, 36, 154–166.

52.

National Reading Panel. (2000). Report of the National Reading Panel: Teaching children to read—An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction. Washington, DC: National Institute of Child Health and Human Development.

53.

Ninci

Vannest

K. J.

Wilson

Zhang

(2015). Interrater agreement between visual analysts of single-case data: A meta-analysis. Behavior Modification, 39, 510–541.

54.

No Child Left Behind Act of 2001, P.L. 107-110, 20 U.S.C. § 6319 (2002).

55.

O’Keeffe

B. V.

Bundock

Kladis

K. L.

Yan

Nelson

(2017). Variability in DIBELS Next progress monitoring measures for students at risk for reading difficulties. Remedial and Special Education, 38, 272–283. doi:10.1177/0741932517713310

56.

Parker

R. I.

Brossart

D. F.

Callicott

K. J.

Long

J. R.

Garcia

Alba

Baugh

F. G.

Sullivan

J. R.

(2005). Effect sizes in single-case research: How large is large? School Psychology Review, 34, 116–132.

57.

Parker

R. I.

Vannest

K. J.

Davis

J. L.

(2011). Effect size in single-case research: A review of nine nonoverlap techniques. Behavior Modification, 35, 303–322. doi:10.1177/0145445511399147

58.

Parker

R. I.

Vannest

K. J.

Davis

J. L.

Sauber

S. B.

(2011). Combining nonoverlap and trend for single-case research: Tau-u. Behavior Therapy, 42, 284–299.

59.

Pinheiro

Bates

DebRoy

Sarkar

Core, R. Team. (2016). nlme: Linear and nonlinear mixed effects models. Retrieved from http://CRAN.R-project.org/package=nlme

60.

Poncy

B. C.

Skinner

C. H.

Axtell

P. K.

(2005). An investigation of the reliability and standard error of measurement of words read correctly per minute using curriculum-based measurement. Journal of Psychoeducational Assessment, 23, 326–338. doi:10.1177/073428290502300403

61.

R Core Team. (2016). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org

62.

Reschly

A. L.

Busch

T. W.

Betts

Deno

S. L.

Long

J. D.

(2009). Curriculum-based measurement oral reading as an indicator of reading achievement: A meta-analysis of the correlational evidence. Journal of School Psychology, 47, 427–469.

63.

Rohatgi

(2016). Webplotdigitizer ver.3.10. Retrieved from http://arohatgi.info/WebPlotDigitizer

64.

Ross

S. G.

Begeny

J. C.

(2014). Single-case effect size calculation: Comparing regression and non-parametric approaches across previously published reading intervention data sets. Journal of School Psychology, 52, 419–431. doi:10.1016/j.jsp.2014.06.003

65.

Schmidt

F. L.

Hunter

J. E.

(2015). Methods of meta-analysis: Correcting error and bias in research findings. Thousand Oaks, CA: SAGE.

66.

Shadish

W. R.

(2014). Statistical analyses of single-case designs: The shape of things to come. Current Directions in Psychological Science, 23, 139–146. doi:10.1177/0963721414524773

67.

Smith

J. D.

(2012). Single-case experimental designs: A systematic review of published research and current standards. Psychological Methods, 17, 510–550. doi:10.1037/a0029312

68.

Vannest

K. J.

Davis

J. L.

Parker

R. I.

(2013). Single case research in schools: Practical guidelines for school-based professionals. New York, NY: Routledge.

69.

Vasquez

III Slocum

T. A.

(2012). Evaluation of synchronous online tutoring for students at risk of reading failure. Exceptional Children, 78, 221–235.

70.

Walcott

Marett

Hessel

A. B.

(2014). Effectiveness of a computer-assisted intervention for young children with attention reading problems. Journal of Applied School Psychology, 30, 83–106.

71.

White

O. R.

Haring

N. G.

(1980). Exceptional teaching: A multimedia training package. Columbus, OH: Merrill.

72.

Wolery

(2013). A commentary: Single-case design technical document of the what works clearinghouse. Remedial and Special Education, 34, 39–43. doi:10.1177/0741932512468038

73.

Wolery

Busick

Reichow

Barton

(2010). Comparison of overlap methods for quantitatively synthesizing single-subject data. Journal of Special Education, 44, 18–28.