The central roles of science in the field of remedial and special education are to (a) identify basic laws of nature and (b) apply those laws in the design of practices that achieve socially valued outcomes. The scientific process is designed to allow demonstration of specific (typically positive) outcomes, and to assist in the attribution of those outcomes to controlled variables. Although growing recognition is being given to the importance of replication in this process, equal consideration should be given to the function of publishing studies that document negative (or null) results. In this manuscript, we outline the features of negative results in educational and psychological single-case intervention research. We also discuss the assessment, methodological, and statistical dimensions of negative results that should be considered when reporting negative results. The importance of replication studies (direct, systematic, and clinical) is also discussed within the context of negative-results research.
AtkinsonD. R.FurlongM. J.WampoldB. E. (1982). Statistical significance, reviewer evaluations, and the scientific process: Is there a (statistically) significant relationship?Journal of Counseling Psychology, 29, 189–194. doi:10.1037/0022-0167.29.2.189
2.
BarlowD. H. (2010). Negative effects from psychological treatments: A perspective. American Psychologist, 65, 13–20.
3.
BarlowD. H.HersenM. (1984). Single case experimental designs: Strategies for studying behavior change (2nd ed.). New York, NY: Pergamon Press.
BartonE. E.ReichowB.SchnitzA.SmithI. C.SherlockD. (2015). A systematic review of sensory-based treatments for children with disabilities. Research in Developmental Disabilities, 37, 64–80. doi:10.1016/j.ridd.2014.11.006
6.
BrachtG. H.GlassG. V. (1968). The external validity of experiments. American Educational Research Journal, 5, 437–474.
7.
CampbellD. T.FiskeD. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105.
8.
CookB. G.TherrienW. J. (2017). Null effects and publication bias in special education research. Behavioral Disorders, 42, 149–158. doi:10.1177/0198742917709473
9.
DavesR. M. (1994). House of cards: Psychology and psychotherapy built on myth. New York, NY: The Free Press.
10.
DishionT. J.McCordJ.PoulinF. (1999). When interventions harm: Peer groups and problem behavior. American Psychologist, 54, 755–764.
11.
DriessenE.HollonS. D.BocktingC. L.CuijpersP.TurnerE. H. (2015). Does publication bias inflate the apparent efficacy of psychological treatment for major depressive disorder? A systematic review and meta-analysis of US National Institutes of Health-funded trials. PLoS ONE, 10(9), Article e0137864.
12.
EngberD. (2013, April24). Run? Walk? Exercise benefit may come down to duration. Arizona Daily Star, A. 19.
13.
FanelliD. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891–904. doi:10.1007/s11192-011-0494-7
14.
FergusonC. J.BrannickM. T. (2012). Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17, 120–128.
15.
FergusonC. J.HeeneM. (2012). A vast graveyard of undead theories: Publication bias and psychological science’s aversion to the null. Perspectives on Psychological Science, 7, 555–561.
16.
Fertel-DalyD.BedellG.HinojosaJ. (2001). Effects of a weighted vest on attention to task and self-stimulatory behaviors in preschoolers with pervasive developmental disorders. American Journal of Occupational Therapy, 55, 629–640.
17.
FlayB. R.BiglanA.BoruchR. F.CastroF. G.GottfredsonD.KellamS.. . . JiP. (2005). Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Prevention Science, 6, 151–175.
18.
FrancisG. (2012). The psychology of replication and replication in psychology. Perspectives on Psychological Science, 7, 585–594.
19.
FrancoA.MalhotraN.SimonovitsG. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345, 1502–1505. doi:10.1126/science.1255484
20.
GageN. A.CookB. G.ReichowB. (2017). Publication bias in special education meta-analyses. Exceptional Children, 83, 428–445.
21.
GlassG. V.SmithM. L. (1978). “An exercise in mega-silliness”: Reply. American Psychologist, 33, 517–519.
22.
GreenwaldA. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1–20.
23.
HersenM.BarlowD. H. (1976). Single-case experimental designs: Strategies for studying behavior change. New York, NY: Pergamon Press.
24.
HornerR. H.CarrE. G.HalleJ.McGeeG.OdomS.WoleryM. (2005). The use of single subject research to identify evidence-based practice in special education. Exceptional Children, 71, 165–179.
25.
HornerR. H.KratochwillR. T. (2011). Using single-case research to identify evidence-based practices: Some brief reflections. Savage Controversies, 5, 2–5.
26.
IngramS.Lewis-PalmerG.SugaiG. (2005). Function-based intervention planning: Comparing the effectiveness of FBA indicated and contra-indicated intervention plans. Journal of Positive Behavior Interventions, 7, 224–236.
IoannidisJ. P. A. (2005). Why most published research findings are false. PLOS Medicine, 2, 696–701.
30.
KazdinA. E. (1998). Research design in clinical psychology (3rd ed.). Boston, MA: Allyn & Bacon.
31.
KazdinA. E. (2011). Single-case research designs: Methods for clinical and applied settings (2nd ed.). New York, NY: Oxford University Press.
32.
KilgusS. P.Riley-TillmanT. C.KratochwillT. R. (2016). Intervention research with a theory driven single-case design research cycle. School Psychology Review, 45, 477–498.
33.
KiuharaS. A.KratochwillT. R.PullenP. C. (2017). Designing robust single-case design experimental studies. In KauffmanJ. M.HallahanD. P.PullenP. C. (Eds.), Handbook of special education (2nd ed., pp. 116–136). New York: Routledge.
34.
KratochwillT. R.HitchcockJ.HornerR. H.LevinJ. R.OdomS. L.RindskopfD. M.ShadishW. R. (2010). Single case designs technical documentation. Retrieved from http://ies.ed.gov/ncee/wwc/pdf/wwc_scd.pdf
35.
KratochwillT. R.HitchcockJ.HornerR. H.LevinJ. R.OdomS. L.RindskopfD. M.ShadishW. R. (2013). Single-case designs technical documentation. Remedial and Special Education, 34, 26–38.
36.
KratochwillT. R.LevinJ. R.HornerR. H.SwobodaC. (2014). Visual analysis of single-case intervention research: Conceptual and methodological considerations. In KratochwillT. R.LevinJ. R. (Eds.), Single-case intervention research: Methodological and statistical advances. Washington, DC: American Psychological Association.
37.
KratochwillT. R.StoiberK. C.GutkinT. B. (2000). Empirically supported interventions in school psychology: The role of negative results in outcome research. Psychology in the Schools, 37, 399–413.
38.
KupfersmidJ. (1988). Improving what is published: A model in search of an editor. American Psychologist, 43, 635–642.
39.
LevinJ. R. (1985). Some methodological and statistical “bugs” in research on children’s learning. In PressleyM.BrainerdC. J. (Eds.), Cognitive learning and memory in children (pp. 204–233). New York, NY: Springer-Verlag.
40.
LevinJ. R. (1989). A transfer-appropriate-processing perspective of pictures in prose. In MandlH.LevinJ. R. (Eds.), Knowledge acquisition from text and pictures (pp. 83–100). Amsterdam, The Netherland: Elsevier.
41.
LindsayD. S. (2017). Preregistered Direct Replications in. Psychological Science. Psychological Science, 28, 1191–1192.
42.
LykkenD. T. (1968). Statistical significance in psychological research. Psychological Bulletin, 70, 151–159.
43.
MagginD. M.BrieschA. M.ChafouleasS. M. (2013). An application of the What Works Clearinghouse Standards for evaluating single-subject research: Synthesis of the self-management literature base. Remedial and Special Education, 34, 44–58.
44.
MagginD. M.ChafouleasS. M.GoddardK. M.JohnsonA. H. (2011). A systematic evaluation of token economies as a classroom management tool for students with challenging behavior. Journal of School Psychology, 49, 529–554.
45.
MahoneyM. (1977). Publication prejudices: An experimental study of confirmatory bias on the peer review system. Cognitive Therapy and Research, 1, 161–175.
46.
MaxwellS. E.KelleyK. (2011). Ethics and sample size planning. In PanterA. T.SterbaS. K. (Eds.), Handbook of ethics in quantitative methodology (pp. 159–184). New York, NY: Routledge.
47.
NeuliepJ. W.CrandallR. (1993). Reviewer bias against replication research. Journal of Social Behavior and Personality, 8(6), 21–29.
48.
NosekB. A.SpiesJ. R.MotylM. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publish ability. Perspectives on Psychological Science, 7, 615–631.
49.
OglesB. M.LambertM. J.MastersK. S. (1996). Assessing outcome in clinical practice. Boston, MA: Allyn & Bacon.
50.
OlsonC. M.RennieD.CookD.DickersinK.FlanaginA.HoganJ. W.. . . PaceB. (2002). Publication bias in editorial decision making. Journal of the American Medical Association, 287, 2825–2828. doi:10.1001/jama.287.21.2825
51.
Open Science Collaboration. (2012). An open, large scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7, 657–660.
52.
PashlerH.WagenmakersE. J. (2012). Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence?Perspectives on Psychological Science, 7, 528–530.
53.
PolaninJ. R.Tanner-SmithE. E.HennessyE. A. (2016). Estimating the difference between published and unpublished effect sizes a meta-review. Review of Educational Research, 86, 207–236. doi:10.3102/0034654315582067
54.
ReichowB.BartonE. E.SewellJ. N.GoodL.WoleryM. (2010). Effects of weighted vests on the engagement of children with developmental delays and autism. Focus on Autism and Other Developmental Disabilities, 25, 3–11.
55.
RosenthalR. (1966). Experimenter effects in behavioral research. New York, NY: Appleton Century Crofts.
56.
RosenthalR. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638–641. doi:10.1037/0033-2909.86.3.638
57.
SeftorN. (2016). What does it mean when a study finds no effects? (REL 2017–265). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. Retrieved from http://ies.ed.gov/ncee/edlabs
58.
ShadishW. R.CookT. D.CampbellD. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
59.
ShadishW. R.DohertyM.MontgomeryL. M. (1989). How many studies are in the file drawer? An estimate from the family marital psychotherapy literature. Clinical Psychology Review, 9, 589–603.
60.
ShadishW. R.ZelinskyN. A. M.VeveaJ. L.KratochwillT. R. (2016). Survey of publication preferences of single-case design researchers when treatments have small or large effects. Journal of Applied Behavior Analysis, 49, 656–673. doi:10.1002/jaba.308
61.
ShamE.SmithT. (2014). Publication bias in studies of an applied behavior-analytic intervention: An initial analysis. Journal of Applied Behavior Analysis, 47, 663–678.
62.
SimmonsJ. P.NelsonL. D.SimonsohnU. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.
63.
SongF.EasterwoodA.GilbodyS.DuleyL.SuttonA. J. (2000). Publication and other selection biases in systematic reviews. Health Technology Assessment, 4, 1–115.
64.
SuttonA. J.DuvalS. J.TweedieR. L.AbramsK. R.JonesD. R. (2000). Empirical assessment of effect of publication bias on meta-analyses. British Medical Journal, 320, 1574–1577.
65.
TincaniM.TraversJ. (IN PRESS). Publishing single-case research design studies that do not demonstrate experimental control. Remedial and Special Education. Advance online publication. doi:10.1177/074193251769744.
66.
TraversJ. C.CookB. G.TherrienW. J.CoyneM. D. (2016). Replication research and special education. Remedial and Special Education, 37, 1–10. doi:10.1177/0741932516648462
67.
WalsterW.ClearyT. A. (1970). A proposal for a new editorial policy in the social sciences. American Statistician, 24(2), 16–19.
68.
WilliamsP. T. (2013, April). Greater weight loss from running than walking during a 6.2-yr prospective follow-up. Medicine & Science in Sports & Exercise, 45, 706–713.
69.
WilliamsP. T.ThompsonP. D. (2013). Walking versus running for hypertension, cholesterol, and diabetes mellitus risk reduction. Artheriosclerosis, Thrombosis, and Vascular Biology, 33, 1085–1091.
70.
WinermanL. (2013). Interesting results: Can they be replicated?APA Monitor, 44(2), 38.
