False Discovery Rates When Engaging in Skill Versus Performance Deficit Analysis for Academic Instructional Planning

Abstract

Brief experimental analysis (BEA) is a well-researched approach to conducting problem analysis, where potential interventions are pilot tested using a single-subject alternating treatment design. However, its brevity may lead to a high frequency of decision-making errors, particularly in situations where one tested condition is rarely optimal for students (i.e., the base rate). The current study explored the accuracy of a specific variant of BEA, skill versus performance deficit analysis (SPA), across different variations of the basic BEA design, score difference thresholds, and reading and math curriculum-based measurements (CBMs). Findings indicate that the ABAB design provides a reasonable control of such error rates when using reading CBM, whereas subtraction CBM required the use of an ABABAB design. Such error rates could not be controlled, regardless of design, when using multiplication CBM. Implications for best practice in the use of BEA are discussed.

Keywords

curriculum-based measurement functional analysis statistical methods

Get full access to this article

View all access options for this article.

References

Andersen

M. N.

Daly

E. J.

III Young

N. D.

(2013). Examination of a one-trial brief experimental analysis to identify reading fluency interventions. Psychology in the Schools, 50, 403–414. doi:10.1002/pits.21682

Bar-Hillel

Fischoff

(1981). When do base rates affect predictions? Journal of Personality and Social Psychology, 41, 671–680.

Barlow

D. H.

Hayes

S. C.

(1979). Alternating treatment design: One strategy for comparing the effects of two treatments in a single subject. Journal of Applied Behavior Analysis, 12, 199–210. doi:10.1901/jaba.1979.12-199

Bergan

J. R.

Kratochwill

T. R.

(1990). Behavioral Consultation and Therapy. New York: Plenum. doi:10.1007/978-1-4899-7288-0

Burns

M. K.

Ganuza

Z. M.

London

R. M.

(2009). Brief experimental analysis of written letter formation: Single-case demonstration. Journal of Behavioral Education, 18, 2–34.

Burns

M. K.

Taylor

C. N.

Warmbold-Brann

K. L.

Preast

J. L.

Hosp

J. L.

Ford

J. W.

(2017). Empirical synthesis of the effect of standard error of the measurement on decisions within brief experimental analyses of reading fluency. Psychology in the Schools, 54, 640–654.

Christ

T. J.

Monaghen

B. D.

Zopluoglu

Van Norman

E. R.

(2012). Curriculum-based measurement of oral reading: Evaluation of growth estimates derived with pre-post assessment methods. Assessment for Effective Intervention, 38, 139–153. doi:10.1177/1534508412456417

Christ

T. J.

Silberglitt

(2007). Estimates of the standard error of measurement for curriculum-based measures of oral reading fluency. School Psychology Review, 36, 130–146.

Christ

T. J.

Zopluoglu

Monaghen

B. D.

Van Norman

E. R.

(2013). Curriculum-based measurement of oral reading: Multi-study evaluation of schedule, duration, and dataset quality on progress monitoring outcomes. Journal of School Psychology, 51, 19–57.

10.

Codding

R. S.

Beglici

Gottesman

Johnson

Kert

A. S.

Lebeouf

(2009). Selecting intervention strategies: Using brief experimental analysis for mathematics problems. Journal of Applied School Psychology, 25, 146–168.

11.

Daly

E. J.

III Martens

B. K.

Dool

E. L.

Hintze

J. M

. (1998). Using brief functional analysis to select interventions for oral reading. Journal of Behavioral Education, 8, 203–218.

12.

Daly

E. J.

III Witt

J. C.

Martens

B. K.

Dool

E. J.

(1997). A model for conducting a functional analysis of academic performance problems. School Psychology Review, 26, 554–574.

13.

Dawes

J. M.

Poncy

B. C.

Solomon

B. G.

Duhon

G. J.

(2018). A comparison of three variations of cover, copy, and compare procedures to increase multiplication fact performance. Manuscript in preparation.

14.

Duhon

G. J.

Noell

G. H.

Witt

J. C.

Freeland

J. T.

Dufrene

B. A.

Gilbertson

D. N.

(2004). Identifying academic skill and performance deficits: The experimental analysis of brief assessments of academic skills. School Psychology Review, 33, 429–443.

15.

Duhon

G. J.

Poncy

B. C.

(2015). The effectiveness of two explicit timing interventions to increase subtraction fluency. Unpublished dataset.

16.

Dynamic Measurement Group. (2013). DIBELS Next technical manual. Eugene, OR: Author.

17.

Eckert

T. L.

Ardoin

S. P.

Daisey

D. M.

Scarola

M. D.

(2000). Empirically evaluating the effectiveness of reading interventions: The use of brief experimental analysis and single case designs. Psychology in the Schools, 37, 463–473.

18.

Fuchs

Mock

Morgan

P. L.

Young

C. L.

(2003). Responsiveness-to-intervention: Definitions, evidence, and implications for the learning disabilities construct. Learning Disabilities Research & Practice, 18, 157–171.

19.

Gilbertson

Witt

Duhon

Dufrene

(2008). Using brief assessments to select math fluency and on-task behavior interventions: An investigation of treatment utility. Education & Treatment of Children, 31, 167–181.

20.

Harding

Wacker

D. P.

Cooper

L. J.

Millard

Jensen-Kovalan

(1994). Brief hierarchical assessment of potential treatment components with children in an outpatient clinic. Journal of Applied Behavior Analysis, 27, 291–300.

21.

Johnston

J. M.

Pennypacker

H. S.

(2008). Strategies and tactics of behavioral research (3rd ed.). New York, NY: Routledge.

22.

Jones

K. M.

Wickstrom

K. F.

(2002). Done in sixty seconds: Further analysis of the brief assessment model for academic problems. School Psychology Review, 31, 554–568.

23.

Jones

K. M.

Wickstrom

K. F.

Daly

E. J.

(2009). Best practices in brief assessment of reading concerns. In Thomas

Grimes

(Eds.), Best practices in school psychology (5th ed., pp. 1163–1180). Bethesda, MD: National Association of School Psychologists.

24.

Kahneman

Tversky

(1973). On the psychology of prediction. Psychological Review, 80, 237–251.

25.

Kilgus

S. P.

Methe

S. A.

Maggin

D. M.

Tomasula

J. L.

(2014). Curriculum-based measurement of oral reading (R-CBM): A diagnostic test accuracy meta-analysis of evidence supporting use in universal screening. Journal of School Psychology, 52, 377–405.

26.

Little

S. G.

Akin-Little

(2014). Academic assessment and intervention. New York, NY: Taylor & Francis.

27.

McComas

J. J.

Wacker

D. P.

Coopre

L. J.

Asmus

J. M.

Richman

Stoner

(1996). Brief experimental analysis of stimulus prompts for accurate responding on academic tasks in an outpatient clinic. Journal of Applied Behavior Analysis, 29, 397–401. doi:10.1901/jaba.1996.29-397

28.

Meehl

P. E.

Rosen

A. R.

(1955). Antecedent probability and the efficiency of psychometric signs, patterns, or cutting scores. Psychological Bulletin, 52, 194–216.

29.

Mercer

S. H.

Harpole

L. L.

Mitchell

R. R.

McLemore

Hardy

(2012). The impact of probe variability on brief experimental analysis of reading skills. School Psychology Quarterly, 27, 223–235.

30.

Noell

G. H.

Freeland

J. T.

Witt

J. C.

Gansle

K. A

. (2001). Using brief assessments to identify effective interventions for individual students. Journal of School Psychology, 39, 335–355.

31.

Parker

D. C.

Dickey

B. N.

Burns

M. K.

McMaster

K. L.

(2012). An application of brief experimental analysis with early writing. Journal of Behavioral Education, 21, 329–349.

32.

Poncy

B. C.

Skinner

C. H.

Axtell

P. K.

(2005). An investigation of the reliability and standard error of measurement of words read correctly per minute using curriculum-based measurement. Journal of Psychoeducational Assessment, 23, 326–338.

33.

Reisener

C. D.

Dufrene

B. A.

Clark

C. R.

Olmi

D. J.

Tingstrom

D. H.

(2016). Selecting effective interventions to increase math computation fluency via brief experimental analysis. Psychology in the Schools, 53, 39–57.

34.

Seethaler

P. M.

Fuchs

L. S.

(2011). Using curriculum-based measurement to monitor kindergarteners’ mathematics development. Assessment for Effective Intervention, 36, 219–229.

35.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi-experimental designs for generalized causal inference. Belmont, CA: Wadsworth, Cengage Learning.

36.

Shapiro

E. S.

Edwards

Zigmond

(2005). Progress monitoring of mathematics among students with learning disabilities. Assessment for Effective Intervention, 30, 15–32.

37.

Skinner

C. H.

(1998). Preventing academic skill deficits. In Watson

T. S.

Gresham

F. M.

(Eds.), Handbook of child behavior therapy (pp. 61–79). New York, NY: Springer.

38.

Stuebing

K. K.

Fletcher

J. M.

Branum-Martin

Francis

D. J.

(2012). Evaluation of the technical adequacy of three methods for identifying specific learning disabilities based on cognitive discrepancies. School Psychology Review, 41, 3–22.

39.

Tilly

W. D.

III . (2002). Best practices in school psychology as a problem-solving enterprise. In Thomas

Grimes

(Eds.), Best practices in school psychology (4th ed., pp. 21–36). Bethesda, MD: National Association of School Psychologists.

40.

VanDerHeyden

A. M.

(2011). Technical adequacy of response to intervention decisions. Exceptional Children, 77, 335–350.

41.

VanDerHeyden

A. M.

Witt

J. C.

(2005). Quantifying context in assessment: Capturing the effect of base rates on teacher referral and a problem-solving model of identification. School Psychology Review, 34, 161–183.

42.

VanDerHeyden

A. M.

Witt

J. C.

Gilbertson

(2007). A multi-year evaluation of the effects of response to intervention (RTI) model on identification of children for special education. Journal of School Psychology, 45, 225–256.

43.

VanDerHeyden

A. M.

Witt

J. C.

Naquin

(2003). Development and validation of a process for screening referrals to special education. School Psychology Review, 32, 204–227.

44.

Van Norman

E. R.

Christ

T. J

. (2016). Curriculum-based measurement of reading: Accuracy of recommendations from three-point decision rules. School Psychology Review, 45, 296–309.

45.

Van Norman

E. R.

Christ

T. J.

Newell

K. W

. (2017). Curriculum-based measurement of reading progress monitoring: The importance of growth magnitude and goal setting in decision making. School Psychology Review, 46, 320–328.

46.

Van Norman

E. R.

Klingbeil

D. A.

Nelson

. (2017). Posttest probabilities: An empirical demonstration of their use in evaluating the performance of universal screening measures across settings. School Psychology Review, 46, 349–362.

47.

What Works Clearinghouse. (2017). Procedures and standards handbook version 4.0. Washington, DC: Institute for Education Sciences.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.47 MB