A Description of Missing Data in Single-Case Experimental Designs Studies and an Evaluation of Single Imputation Methods

Abstract

Missing data is inevitable in single-case experimental designs (SCEDs) studies due to repeated measures over a period of time. Despite this fact, SCEDs implementers such as researchers, teachers, clinicians, and school psychologists usually ignore missing data in their studies. Performing analyses without considering missing data in an intervention study using SCEDs or a meta-analysis study including SCEDs studies in a topic can lead to biased results and affect the validity of individual or overall results. In addition, missingness can undermine the generalizability of SCEDs studies. Considering these drawbacks, this study aims to give descriptive and advisory information to SCEDs practitioners and researchers about missing data in single-case data. To accomplish this task, the study presents information about missing data mechanisms, item level and unit level missing data, planned missing data designs, drawbacks of ignoring missing data in SCEDs, and missing data handling methods. Since single imputation methods among missing data handling methods do not require complicated statistical knowledge, are easy to use, and hence are more likely to be used by practitioners and researchers, the present study evaluates single imputation methods in terms of intervention effect sizes and missing data rates by using a real and hypothetical data sample. This study encourages SCEDs implementers, and also meta-analysts to use some of the single imputation methods to increase the generalizability and validity of the study results in case they encounter missing data in their studies.

Keywords

time-series data single-case experimental designs missing data missing data mechanisms single imputation methods

Get full access to this article

View all access options for this article.

References

Aydin

(2023a). Rise of single-case experimental designs: A historical overview of the necessity of single-case methodology. Neuropsychological Rehabilitation. Advanced online publication. https://doi.org/10.1080/09602011.2023.2181191

Aydin

(2023b). Characteristics of missing data in single-case experimental designs: An investigation of published data. Behavior Modification. Advanced online publication. https://doi.org/10.1177/01454455231212265

Aydin

Tanious

(2022). Performance criteria-based effect size (PCES) measurement of single-case experimental designs: A real-world data study. Journal of Applied Behavior Analysis, 55(3), 891–918. https://doi.org/10.1002/jaba.928

Barker

J. B.

Mellalieu

S. D.

McCarthy

P. J.

Jones

M. V.

Moran

(2013). A review of single-case research in sport psychology 1997–2012: Research trends and future directions. Journal of Applied Sport Psychology, 25(1), 4–32. https://doi.org/10.1080/10413200.2012.709579

Bennett

D. A.

(2001). How can I deal with missing data in my study? Australian and New Zealand Journal of Public Health, 25(5), 464–469. https://doi.org/10.1111/j.1467-842X.2001.tb00294.x

Brossart

D. F.

Laird

V. C.

Armstrong

T. W.

(2018). Interpreting Kendall’s Tau and Tau-U for single-case experimental designs. Cogent Psychology, 5(1), 1–26. https://doi.org/10.1080/23311908.2018.1518687

Chen

L. T.

Feng

P. J.

Peng

C. Y. J.

(2020). Dealing with missing data by EM in single-case studies. Behavior Research Methods, 52, 131–150. https://doi.org/10.3758/s13428-019-01210-8

Collins

L. M.

Schafer

J. L.

Kam

C. M.

(2001). A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6(4), 330–351. https://psycnet.apa.org/doi/10.1037/1082-989X.6.4.330

Cook

B. G.

Lloyd

J. W.

Mellor

Nosek

B. A.

Therrien

W. J.

(2018). Promoting open science to increase the trustworthiness of evidence in special education. Exceptional Children, 85(1), 104–118. https://doi.org/10.1177/0014402918793138.

10.

Davidson

S. J.

O’Connor

R. E.

(2019). An intervention using morphology to derive word meanings for English language learners. Journal of Applied Behavior Analysis, 52(2), 394–407. https://doi.org/10.1002/jaba.539

11.

T. K.

Onghena

(2022). The randomized marker method for single-case randomization tests: Handling data missing at random and data missing not at random. Behavior Research Methods. Advanced online publication. https://doi.org/10.3758/s13428-021-01781-5

12.

T. K.

Michiels

Tanious

Onghena

(2020). Handling missing data in randomization tests for single-case experiments: A simulation study. Behavior Research Methods, 52, 1355–1370. https://doi.org/10.3758/s13428-019-01320-3

13.

Dempster

A. P.

Laird

N. M.

Rubin

D. B.

(1977). Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B, 39(1), 1–22. https://doi.org/10.1111/j.2517-6161.1977.tb01600.x

14.

Dong

Peng

C. Y. J.

(2013). Principled missing data methods for researchers. SpringerPlus, 2, 1–17. https://doi.org/10.1186/2193-1801-2-222

15.

Edgington

E. S.

Onghena

(2007). Randomization tests (4th ed.). Chapman & Hall/CRC.

16.

Horner

R. H.

Carr

E. G.

Halle

McGee

Odom

Wolery

(2005). The use of single- subject research to identify evidence-based practice in special education. Exceptional Children, 71(2), 165–179. https://doi.org/10.1177/001440290507100203

17.

Huitema

B. E.

McKean

J. W.

(1991). Autocorrelation estimation and inference with small samples. Psychological Bulletin, 110(2), 291–304. https://psycnet.apa.org/doi/10.1037/0033-2909.110.2.291

18.

Ibrahim

J. G.

Chen

M. H.

Lipsitz

S. R.

Herring

A. H.

(2005). Missing-data methods for generalized linear models: A comparative review. Journal of the American Statistical Association, 100(469), 332–346. https://doi.org/10.1198/016214504000001844

19.

Jamshidi

Heyvaert

Declercq

Fernandez-Castilla

Ferron

J. M.

Moeyaert

Beretvas

S. N.

Onghena

Van den Noortgate

(2018). Methodological quality of meta-analyses of single-case experimental studies. Research in Developmental Disabilities, 79, 97–115. https://doi.org/10.1016/j.ridd.2017.12.016

20.

Jamshidi

Heyvaert

Declercq

Fernández-Castilla

Ferron

J. M.

Moeyaert

Beretvas

S. N.

Onghena

Van den Noortgate

(2022). A systematic review of single-case experimental design meta-analyses: Characteristics of study designs, data, and analyses. Evidence-Based Communication Assessment and Intervention. Advance online publication. https://doi.org/10.1080/17489539.2022.2089334

21.

Kazdin

A. E.

(2010). Single-case research designs: Methods for clinical and applied settings (2nd ed.). Oxford University Press.

22.

Kazdin

A. E.

(2021). Single-case experimental designs: Characteristics, changes, and challenges. Journal of the Experimental Analysis of Behavior, 115(1), 56–85. https://doi.org/10.1002/jeab.638

23.

Krasny-Pacini

Evans

(2018). Single-case experimental designs to assess intervention effectiveness in rehabilitation: A practical guide. Annals of Physical and Rehabilitation Medicine, 61(3), 164–179. https://doi.org/10.1016/j.rehab.2017.12.002

24.

Kratochwill

T. R.

(Ed.). (2013). Single subject research: Strategies for evaluating change. Academic Press.

25.

Kratochwill

T. R.

Hitchcock

Horner

R. H.

Levin

J. R.

Odom

S. L.

Rindskopf

Shadish

W. R. M.

(2010). Single case designs technical documentation. https://files.eric.ed.gov/fulltext/ED510743.pdf

26.

Lane

J. D.

Gast

D. L.

(2014). Visual analysis in single case experimental design studies: Brief review and guidelines. Neuropsychological Rehabilitation, 24(3–4), 445–463.

27.

Lang

K. M.

Little

T. D.

(2018). Principled missing data treatments. Prevention Science, 19(3), 284–294. https://doi.org/10.1007/s11121-016-0644-5

28.

Lillie

M. A.

Tiger

J. H.

(2019). Acquisition and generative responding following print-to-braille construction response training with sighted learners. Journal of Applied Behavior Analysis, 52(1), 286–298. https://doi.org/10.1002/jaba.516

29.

Little

R. J. A.

D’Agostino

Cohen

M. L.

Dickersin

Emerson

S. S.

Farrar

J. T.

Frangakis

Hogan

J. W.

Molenberghs

Murphy

S. A.

Neaton

J. D.

Rotnitzky

Scharfstein

Shih

W. J.

Siegel

Stern

(2012). The prevention and treatment of missing data in clinical trials. New England Journal of Medicine, 367(14), 1355–1360. https://doi.org/10.1056/NEJMsr1203730

30.

Little

R. J. A.

Rubin

D. B.

(2002). Statistical analysis with missing data (2nd ed.). John Wiley & Sons, Inc.

31.

Little

T. D.

Jorgensen

T. D.

Lang

K. M.

Moore

E. W. G.

(2014). On the joys of missing data. Journal of Pediatric Psychology, 39(2), 151–162. https://doi.org/10.1093/jpepsy/jst048

32.

Manolov

(2023). Does the choice of a linear trend-assessment technique matter in the context of single-case data? Behavior Research Methods. Advanced online publication. https://doi.org/10.3758/s13428-022-02013-0

33.

Manolov

Lebrault

Krasny-Pacini

(2023). How to assess and take into account trend in single-case experimental design data. Neuropsychological Rehabilitation. Advanced online publication. https://doi.org/10.1080/09602011.2023.2190129

34.

Manolov

Moeyaert

(2017). Recommendations for choosing single-case data analytical techniques. Behavior Therapy, 48(1), 97–114. https://doi.org/10.1016/j.beth.2016.04.008.

35.

Manolov

Moeyaert

Fingerhut

(2022). A priori justification for effect measures in single-case experimental designs. Perspectives on Behavior Science, 45(1), 153–186. https://doi.org/10.1007/s40614-021-00282-2

36.

Mirzaei

Carter

S. R.

Patanwala

A. E.

Schneider

C. R.

(2022). Missing data in surveys: Key concepts, approaches, and applications. Research in Social and Administrative Pharmacy, 18(2), 2308–2316. https://doi.org/10.1016/j.sapharm.2021.03.009

37.

Moeyaert

Yang

Kim

(2023). Characteristics of moderators in meta-analyses of single-case experimental design studies. Behavior Modification, 47(6), 1510–1545. https://doi.org/10.1177/01454455211002111

38.

Parker

R. I.

Vannest

K. J.

Brown

(2009). The improvement rate difference for single-case research. Exceptional Children, 75(2), 135–150. https://doi.org/10.1177/001440290907500201

39.

Parker

R. I.

Vannest

K. J.

Davis

J. L.

Sauber

S. B.

(2011). Combining nonoverlap and trend for single-case research: Tau-U. Behavior Therapy, 42(2), 284–299. https://doi.org/10.1016/j.beth.2010.08.006

40.

Peng

C. Y. J.

Chen

L. T.

(2018). Handling missing data in single-case studies. Journal of Modern Applied Statistical Methods, 17(1), 1–35. https://doi.org/10.22237/jmasm/1525133280

41.

Peng

C. Y. J.

Chen

L. T.

(2021). Assessing intervention effects in the presence of missing scores. Education Sciences, 11(2), 1–20. https://doi.org/10.3390/educsci11020076

42.

Riley-Tillman

T. C.

Burns

M. K.

Kilgus

S. P.

(2020). Evaluating educational interventions: Single-case design for measuring response to intervention (2nd ed.). The Guildford Press.

43.

Rubin

D. B.

(1978). Multiple imputations in sample surveys-a phe- nomenological Bayesian approach to nonresponse. Proceedings of the Survey Research Methods Section of the American Statistical Association, 1, 20–34.

44.

Rubin

D. B.

(1987). Multiple imputation for nonresponse in surveys. Wiley.

45.

Schafer

J. L.

(1999). Multiple imputation: A primer. Statistical Methods in Medical Research, 8(1), 3–15. https://doi.org/10.1177/096228029900800102

46.

Schafer

J. L.

Graham

J. W.

(2002). Missing data: our view of the state of the art. Psychological Methods, 7(2), 147–177. https://doi.org/10.1037/1082-989X.7.2.147

47.

Schlomer

G. L.

Bauman

Card

N. A.

(2010). Best practices for missing data management in counseling psychology. Journal of Counseling Psychology, 57(1), 1–10. https://psycnet.apa.org/doi/10.1037/a0018082

48.

Scruggs

T. E.

Mastropieri

M. A.

Casto

(1987). The quantitative synthesis of single-subject research methodology and validation. Remedial and Special Education, 8(2), 24–33. https://doi.org/10.1177/074193258700800206

49.

Shadish

W. R.

Hedges

L. V.

Pustejovsky

J. E.

(2014). Analysis and meta-analysis of single-case designs with a standardized mean difference statistic: A primer and applications. Journal of School Psychology, 52(2), 123–147. https://doi.org/10.1016/j.jsp.2013.11.005

50.

Shadish

W. R.

Kyse

E. N.

Rindskopf

D. M.

(2013). Analyzing data from single-case designs using multilevel models: New applications and some agenda items for future research. Psychological Methods, 18(3), 385–405. https://doi.org/10.1037/a0032964

51.

Shadish

W. R.

Sullivan

K. J.

(2011). Characteristics of single-case designs used to assess intervention effects in 2008. Behavior Research Methods, 43, 971–980. https://doi.org/10.3758/s13428-011-0111-y.

52.

Smith

J. D.

(2012). Single-case experimental designs: A systematic review of published research and current standards. Psychological Methods, 17(4), 510–550. https://doi.org/10.1037/a0029312.

53.

Smith

J. D.

Borckardt

J. J.

Nash

M. R.

(2012). Inferential precision in single-case time-series data streams: How well does the EM procedure perform when missing observations occur in autocorrelated data? Behavior Therapy, 43(3), 679–685. https://doi.org/10.1016/j.beth.2011.10.001

54.

Somer

Gische

Miočević

(2022). Methods for modeling autocorrelation and handling missing data in mediation analysis in single case experimental designs (SCEDs). Evaluation & the Health Professions, 45(1), 36–53. https://doi.org/10.1177/01632787211071136

55.

Sterner

W. R.

(2011). What is missing in counseling research? Reporting missing data. Journal of Counseling & Development, 89(1), 56–62. https://doi.org/10.1002/j.1556-6678.2011.tb00060.x

56.

Tanious

Onghena

(2021). A systematic review of applied single-case research published between 2016 and 2018: Study designs, randomization, data aspects, and data analysis. Behavior Research Methods, 53, 1371–1384. https://doi.org/10.3758/s13428-020-01502-4

57.

Tate

R. L.

Perdices

(2019). Single-case experimental designs for clinical research and neurorehabilitation settings: Planning, conduct, analysis, and reporting. Routledge.

58.

Vannest

K. J.

Parker

R. I.

Gonen

Adiguzel

(2016). Single case research: Web-based calculators for SCR analysis (Version 2.0) [Web-based application]. http://singlecaseresearch.org

59.

Velicer

W. F.

Colby

S. M.

(2005). A comparison of missing-data procedures for ARIMA time-series analysis. Educational and Psychological Measurement, 65(4), 596–615. https://doi.org/10.1177/0013164404272502

60.

What Works Clearinghouse. (2022). Procedures and standards handbook (Version 5.0). https://ies.ed.gov/ncee/wwc/

61.

Wilkinson

(1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54(8), 594–604. https://psycnet.apa.org/doi/10.1037/0003-066X.54.8.594

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.01 MB