Statistical Analysis of Single Case Studies in the Clinical Setting: The Example of Methylphenidate Trials in Children with Attention-Deficit Hyperactivity Disorder

Abstract

To develop guidelines for conducting statistically valid clinical n-of-1 (single-subject) trials, we compared actual clinical decisions with statistically based decisions about treatment efficacy. A series of n-of-1 trials involving methylphenidate (MPH) and placebo were conducted for clinical purposes on 11 hospitalized children and adolescents with attention-deficit hyperactivity disorder (ADHD). The original clinical data were retrospectively analyzed using standard statistical methods for n-of-1 trial data. Statistically based decisions were compared to decisions made by five child psychiatrists after inspection of the clinical data in two types of graphic display. In this study, clinicians' treatment decisions appeared to be based on the size of treatment effects rather than on statistical significance. Clinicians' decisions were frequently made in response to statistical artifacts before the collection of sufficient data to permit a sound decision. Data interpretation by graphic inspection resulted in a greater likelihood of falsely accepting a treatment as effective than would be generally acceptable by statistical standards. This may reflect clinician bias toward MPH treatment or to low intuitive weighting of the risks (versus benefits) of MPH in this population. Most raters appeared to intuitively accept making an erroneous judgment that a treatment was effective in about one out of five trials. It is unlikely that they were aware that this is what they were doing. Although the n-of-1 trial is useful for evaluating the effectiveness of MPH in patients with ADHD, several modifications to the standard clinical methodology are proposed to promote reliable, unbiased interpretation of the effects of medication treatments. Clinicians who apply n-of-1 trials to evaluate treatment efficacy in individual patients should be aware of their own level of acceptable risk of falsely accepting a treatment as effective when deciding upon a method of data interpretation and decision-making.