Although many researchers have discussed replication as a means to facilitate self-correcting science, in this article, we identify meta-analyses and evaluating the validity of correlational and causal inferences as additional processes crucial to self-correction. We argue that researchers have a duty to describe sampling decisions they make; without such descriptions, self-correction becomes difficult, if not impossible. We developed the Replicability and Meta-Analytic Suitability Inventory (RAMSI) to evaluate the descriptive adequacy of a sample of studies taken from current psychological literature. Authors described only about 30% of the sampling decisions necessary for self-correcting science. We suggest that a modified RAMSI can be used by authors to guide their written reports and by reviewers to inform editorial recommendations. Finally, we claim that when researchers do not describe their sampling decisions, both readers and reviewers may assume that those decisions do not matter to the outcome of the study, do not affect inferences made from the research findings, do not inhibit inclusion in meta-analyses, and do not inhibit replicability of the study. If these assumptions are in error, as they often are, and the neglected decisions are relevant, then the neglect may create a good deal of mischief in the field.
*AlcockN. (1996). Factors affecting the assessment of postoperative pain: A literature review. Journal of Advanced Nursing, 24, 1144–1151.
2.
Alsheikh-AliA. A.QureshiW.Al-MallahM. H.IoannidisJ. P. A. (2011). Public availability of published research data in high-impact journals. PLoS ONE, 6(9), e24357. doi:10.1371/journal.pone.0024357
3.
American Psychological Association. (2009). Publication Manual of the American Psychological Association (6th ed.). Washington, DC: Author.
4.
AsendorpfJ. B.ConnerM.De FruytF.De HouwerJ.DenissenJ. J. A.FiedlerK.. . . WichertsJ. M. (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108–119. doi:10.1002/per.1919
5.
AvivA. L.ZelenskiJ. M.RalloL.LarsenR. J. (2002). Who comes when: Personality differences in early and late participation in a university subject pool. Personality and Individual Differences, 33, 487–496.
6.
*BaddeleyA. D.LongmanD. J. A. (1978). The influence of length and frequency of training session on rate of learning to type. Ergonomics, 21, 627–635.
7.
*BakerM. D.ManerJ. K. (2009). Male risk-taking as a context-sensitive signaling device. Journal of Experimental Social Psychology, 45, 1136–1139.
8.
BarghJ. A. (2014). Our unconscious mind: Unconscious impulses and desires impel what we think and do in ways Freud never dreamed of. Scientific American, 310, 30–37.
9.
BarghJ. A.ChenM.BurrowsL. (1996). Automaticity of social behavior: Direct effects of trait construct and stereotype activation on action. Journal of Personality and Social Psychology, 71, 230–244.
10.
BeggC.ChoM.EastwoodS.HortonR.MoherD.OlkinI.. . . StroupD. F. (1996). Improving the quality of reporting of randomized controlled trials: The CONSORT statement. Journal of the American Medical Association, 276, 637–639.
11.
*BernsteinL. (1956). The examiner as inhibiting factor in clinical testing. Journal of Consulting Psychology, 20(4), 287–290.
12.
*BierenbaumH.NicholsM. P.SchwartzA. J. (1976). Effects of varying session length and frequency in brief emotive psychotherapy. Journal of Consulting and Clinical Psychology, 44, 790–798.
13.
BordensK. S.AbbottB. B. (2008). Research design and methods: A process approach (7th ed.). Boston, MA: McGraw Hill.
14.
*CampbellT. S.HolderM. D.FranceC. R. (2006). The effects of experimenter status and cardiovascular reactivity on pain reports. Pain, 125, 264–269.
15.
CassidyR. N.KangasB. D. (2014). Impulsive students articulate later: Delay discounting in a research subject pool. The Experimental Analysis of Human Behavior Bulletin, 30, 1–5.
16.
CesarioJ.PlaksJ. E.HigginsE. T. (2006). Automatic social behavior as motivated preparation to interact. Journal of Personality and Social Psychology, 90, 893–910. doi:10.1037/0022-3514.90.6.893
17.
*ChapmanL. J.ChapmanJ. P. (1978). The measurement of differential deficit. Journal of Psychiatric Research, 14, 303–311.
18.
*CohenD.Atun-EinyO.ScherA. (2012). Seasonal effects on infants’ sleep regulation: A preliminary study of Mediterranean climate. Chronobiology International, 29, 1352–1357.
19.
CohenM. R.NagelE. (1934). An introduction to logic and scientific method. New York, NY: Harcourt, Brace.
20.
*CooperH.BaumgardnerA. H.StrathmanA. (1991). Do students with different characteristics take part in psychology experiments at different times of the semester?Journal of Personality, 59, 109–127. doi:10.1111/j.1467-6494.1991.tb00770.x
21.
*CurtisH. S.WolfE. (1951). The influence of the sex of the examiner on the prediction of sex responses on the Roschach. American Psychologist, 6, 345–346.
22.
*DavisD. W. (1997). The direction of race of interviewer effects among African-Americans: Donning the black mask. American Journal of Political Science, 41, 309–322.
23.
Des JarlaisD. C.LylesC.CrepazN., & the TREND Group. (2004). Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: The trend statement. American Journal of Public Health, 94, 361–366.
24.
*deTurckM. A.MillerG. R. (1990). Training observers to detect deception: Effects of self-monitoring and rehearsal. Human Communication Research, 16, 603–620.
25.
de VriesE. N.PrinsH. A.CrollaR. M.den OuterA. J.van AndelG.van HeldenS. H., . . . SURPASS Collaborative Group (2010). Effect of a comprehensive surgical safety system on patient outcomes. New England Journal of Medicine, 363, 1928–1937. doi:10.1056/NEJMsa0911535
26.
DoyenS.KleinO.PinchonC.CleeremansA. (2012). Behavioral priming: It’s all in the mind, but whose mind?PLoS ONE, 7(1), e29081. doi:10.1371/journal.pone.0029081
27.
*DwanK.AltmanD. G.ArnaizJ. A.BloomJ.ChanA.CroninE.. . . WilliamsonP. R. (2008). Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS Clinical Trials, 3(8), e3081. doi:10.1371/journal.pone.0003081
28.
EichE. (2014). Business not as usual. Psychological Science, 25(3), 3–6. doi:10.1177/0956797613512465
29.
*EpleyN.HuffC. (1998). Suspicion, affective response, and educational benefit as a result of deception in psychology research. Personality and Social Psychology Bulletin, 24, 759–768.
30.
EysenckH. J. (1994). Meta-analysis and its problems. British Medical Journal, 309, 789–792.
31.
FergusonC. J.BrannickM. T. (2012). Publication bias in psychological science: Prevalence, methods for controlling and implications for the use of meta-analyses. Psychological Methods, 17(1), 120–128.
32.
FiedlerK. (2011). Voodoo correlations are everywhere—Not only in neuroscience. Perspectives on Psychological Science, 6, 163–171. doi:10.1177/1745691611400237
33.
*FucciD.PetrosinoL.SloaneN.CantrellJ. (1981). Source variation on lingual vibrotactile thresholds: I. The influence of experimenter experience. Bulletin of the Psychonomic Society, 17, 231–232.
34.
*GambozN.RussoR.FoxE. (2002). Age differences and the identity negative priming effect: An updated meta-analysis. Psychology and Aging, 17, 525–530.
35.
*GarbH. (1998). Studying the clinician. Washington, DC: American Psychological Association.
36.
*GiacomoniC.DaviesP. (2013). Effect of room characteristics on perception of low-amplitude sonic booms heard indoors. Proceedings of Meetings on Acoustic, 19, 040050. Retrieved from http://asadl.org/poma/resource/1/pmarcw/v19/i1/p040050_s1
37.
GlassG. V. (1976). Primary, secondary, and meta-analysis of research. The Educational Researcher, 10, 3–8.
38.
GoodwinC. J.GoodwinK. A. (2013). Research in psychology: Methods and design (7th ed.). Hoboken, NJ: John Wiley & Sons.
39.
GrantS.Mayo-WilsonE.HopewellS.MacdonaldG.HoherD.MontgomeryP. (2013). Developing a reporting guideline for social and psychological intervention trials. Journal of Experimental Criminology, 9, 355–367. doi:10.1007/s11292-013-9180-5
40.
*HarrisS. (1971). Influence of subject and experimenter sex in psychological research. Journal of Consulting and Clinical Psychology, 37, 291–294.
*HsiehA. Y.TrippD. A.JiL. (2011). The influence of ethnic concordance and discordance on verbal reports and nonverbal behaviors of pain. Pain, 152, 2016–2022.
43.
HullJ.SloneL.MetayerK.MatthewsA. (2002). The nonconsciousness of self-consciousness. Journal of Personality and Social Psychology, 83, 406–424.
44.
*ImerajL.AntropI.RoeyersH.SwansonJ.DeschepperE.BalS.DeboutteD. (2012). Time-of-day effects in arousal: Disrupted diurnal cortisol profiles in children with ADHD. Journal of Child Psychology and Psychiatry, 53, 782–789. doi:10.1111/j.1469-7610.2012.0256.x
45.
IoannidisJ. P. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. doi:10.1371/journal.pmed.0020124
46.
IoannidisJ. P. (2008). Interpretation of tests of heterogeneity and bias in meta-analysis. Journal of Evaluation in Clinical Practice, 14, 951–957. doi:10.1111/j.1365-2753.2008.00986.x
47.
IoannidisJ. P. (2012). Why science is not necessarily self-correcting. Perspectives on Psychological Science, 7, 645–654. doi:10.1177/1745691612464056
*KallaiI.BarkeA.VossU. (2004). The effects of experimenter characteristics on pain reports in women and men. Pain, 112, 142–147.
50.
*KirkhamJ. J.DwanK. M.AltmanD. G.GambleC.DoddS.SmythR.WilliamsonP. R. (2010). The impact of outcome reporting bias in randomized controlled trials on a cohort of systematic reviews. British Medical Journal, 340, c365. doi:10.1136/bmj.c365
51.
*Klainin-YobasP.ChoM. A. A.CreedyD. (2012). Efficacy of mindfulness-based interventions on depressive symptoms among people with mental disorders: A meta-analysis. International Journal of Nursing Studies, 49, 109–121.
52.
KlatzkyR. L.CreswellJ. D. (2014). An intersensory interaction account of priming effects—and their absence. Perspectives on Psychological Science, 9, 48–58. doi:10.11/145691613514468
53.
KleinR. A.RatliffK. A.VianelloM.AdamsR. B.Jr.BahníkŠ.BernsteinM. J.BocianK.. . . NosekB. A. (2014). Investigating variation in replicability: A “many labs” replication project. Social Psychology, 45, 142–152. doi:10.1027/a000001
54.
LeBelE. P.BorsbookD.Giner-SorollaR.HasselmanF.PetersK.RatliffK. A.SmithC. T. (2013). PsychoDisclosure.org: Grassroots support for reforming reporting standards in psychology. Perspectives on Psychological Science, 8, 424–432. doi:10.1177/1745691613491437
55.
LeBelE. P.PetersK. R. (2011). Fearing the future of empirical psychology: Bem’s (2011) evidence as a case study of deficiencies in modal research practice. Review of General Psychology, 15, 371–379. doi:10.1037/a002525172
56.
*LewM. B. (1982). Child and adult experimenters: Some differential effects. Child Study Journal, 12(4), 223–235.
57.
LilienfeldS. O.LynnS. J.NamyL.WoolfN. (2009). Psychology: From inquiry to understanding. Boston, MA: Allyn & Bacon.
58.
*LittleL. M.CurranJ. P.GilbertF. S. (1977). The importance of subject recruitment procedures in therapy analogue studies on heterosexual-social anxiety. Behavior Therapy, 8, 24–29.
59.
*LombardoJ. P.TocciM. E. (1979). Attribution of positive and negative characteristics of instructors as a function of attractiveness and sex of instructor and sex of subject. Perceptual & Motor Skills, 48, 491–494. doi:10.2466/pms.1979.48.2.491
60.
*LowreyP. E. (1993). The Assessment Center: An Examination of the Effects of Assessor Characteristics on Assessor Scores. Public Personnel Management, 22, 487–501.
61.
*MacobyE. (1974). The psychology of sex differences vol. I: Text. Stanford, CA: Stanford University Press.
62.
*MatsumotoD. (1993). Ethnic differences in affect intensity, emotional judgments, display rule attitudes, and self-reported emotional expression in an American sample. Motivation and Emotion, 17, 107–123.
63.
*MayC. P.HasherL.FoongN. (2005). Implicit memory, age, and time of day: Paradoxical priming effects. Psychological Science, 16, 96–100. doi:10.1111/j.0956-7976.2005.00788.x
64.
McShaneB. B.BöckentholtU. (2014). You cannot step into the same river twice: When power analyses are optimistic. Perspectives in Psychological Science, 9, 612–625.
65.
MillJ. S. (1843). A system of logic, ratiocinative and inductive: Being a connected view of the principles of evidence and the methods of scientific investigation (Vols. 1–3). London, England: John W. Parker, West Strand.
66.
*MintzerE.HalpernJ. (1980). Effect of sex of therapist and client on therapists’ attitudes toward assertiveness problems. Journal of Clinical Psychology, 36, 704–708. doi:10.1002/1097-4679(198007)36:3<704::AID-JCLP2270360317>3.0.CO;2-F
67.
*MishraS. P. (1980). The influence of examiners’ ethnic attributes on intelligence test scores. Psychology in the Schools, 17, 117–122.
68.
MoherD.LiberatiA.TetzlaffJ.AltmanD. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA Statement. PLoS Medicine, 6(6), e1000097. doi:10.1371/journal.pmed1000097
*PaulusP. B.AnnisA. B.SetaJ. J.SchkadeJ. K.MatthewsR. W. (1976). Density does affect task performance. Journal of Personality and Social Psychology, 34, 248–253. doi:10.1037/0022-3514.34.2.248
72.
*PereiraM.AustrinH. R. (1980). Locus of control and status of the experimenter as predictors of suggestibility. Clinical and Experimental Hypnosis, 28, 367–374. doi:10.1080/00207148008409865
ReidpathD. D.AlloteyP. A. (2001). Data sharing in medical research: An empirical investigation. Bioethics, 15, 125–134. doi:10.3758/BF03194105
75.
ReinhartC. M.RogoffK. S. (2010). Growth in a time of debt (NBER Working Paper Series). American Economic Review, American Economic Association, 100, 573–578.
76.
*RodgerR. S.RobertsM. (2013). Comparison of power for multiple comparison procedures. Journal of Methods and Measurement in the Social Science, 4(1), 20–47.
*RollJ. M.McSweeneyF. K.CannonC. B.JohnsonK. S. (1996). Knowledge of session length is determinant of within-session response patterns in human operant paradigm. Behavioural Processes, 36, 1–9.
79.
*RosenthalR. (1966). Experimenter effects in behavioral research. New York, NY: Appleton-Century-Crofts.
80.
*RosenthalR.RosnowR. L. (1975). The volunteer subject. New York, NY: Wiley.
81.
*RügerM.GordijnM. C.BeersmaD. G.de VriesB.DaanS. (2006). Time-of-day-dependent effects of bright light exposure on human psychophysiology: Comparison of daytime and nighttime exposure. American Journal of Physiology: Regulatory, Integrative and Comparative Physiology, 290, R1413–R1420. doi:10.1152/ajpregu.00121.2005
82.
*RumenikD. K.CapassoD. R.HendrickC. (1977). Experimenter sex effects in behavioral research. Psychological Bulletin, 84, 852–877. doi:10.1037/0033-2909.84.5.852
83.
*SaundersD. R. (1980). Definition of Stroop interference in volunteers and non-volunteers. Perceptual & Motor Skills, 51, 343–354.
84.
SavageC. J.VickersA. J. (2009). Empirical study of data sharing by authors publishing in PLoS journals. PLoS ONE, 4(9), e7078. doi:10.1371/journal.pone.0007078
85.
SchmidtS. (2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13, 90–100. doi:10.1037/a0015108
86.
*SchmittN.SchneiderJ. R.CohenS. A. (1990). Factors affecting validity of a regionally administered assessment center. Personnel Psychology, 43, 1–12.
87.
ShadishW. R.DohertyM.MontgomeryL. M. (1989). How many studies are in the file drawer? An estimate from family/marital psychotherapy literature. Clinical Psychology Review, 9, 589–603. doi:10.1016/0272-7358(89)90013-5
88.
*SholomskasD. E.Syracuse-SiewertG. (2005). We don’t train in vain: A dissemination trial of three strategies of training clinicians in cognitive-behavioral therapy. Journal of Consulting and Clinical Psychology, 73, 106–115. doi:10.1037/0022-006X.73.1.106
89.
SidmanM. (1960). Tactics of scientific research. New York, NY: Basic Books.
90.
*SmithM. L.GlassG. V. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32, 752–760.
91.
SorgeR. E.MartinL. J.IsbesterK. A.SotocinalS. G.RosenS.TuttleA. H.. . . MogilJ. S. (2014). Olfactory exposure to males, including men, causes stress and related analgesia in rodents. Nature Methods. Advance online publication. doi:10.1038/NMETH.2935
92.
*SteerR. A.BeckA. T.RiskindJ. H.BrownG. (1987). Relationships between the Beck Depression Inventory and the Hamilton Psychiatric Rating Scale for Depression in depressed outpatients. Journal of Psychopathology and Behavioral Assessment, 9, 327–339.
93.
StroebeW.StrackF. (2014). The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science, 9, 59–71. doi:10.1177/1745691613514450
94.
*TankeE. D. (1979). Perceptions of ethicality of psychological research: Effects of experimenter status, experiment outcome, and authoritarianism. Personality and Social Psychology Bulletin, 5, 164–168.
95.
TurnerL.ShamseerL.AltmanD. G.SchulzK. F.MoherD. (2012). Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane review. Systematic Reviews, 1, Article 60. doi:10.1186/2046-4053-1-60
96.
*van der HeijdenK. B.de SonnevilleL. M. J. (2010). Time-of-day effects on cognition in preadolescents: A trails study. Chronobiology International, 27, 1870–1894. doi:10.3109/07420528.2010.516047
97.
*VielufS.GoddeB.ReuterE.Voelcker-RehageC. (2013). Age-related differences in finger force control are characterized by reduced force production. Experimental Brain Research, 224, 107–117. doi:10.1007/s00221-012-3292-4
98.
*VoyerD.VoyerS.BrydenB. M. (1995). Magnitude of sex differences in spatial abilities: A meta-analysis and consideration of critical variables. Psychological Bulletin, 117, 250–270.
99.
WainerH. (2007). The most dangerous equation: Ignorance of how sample size affects statistical variation has created havoc for nearly a millennium. American Scientist, 95(3), 249–256. doi:10.1511/2007.65.1026
100.
*WeissB. H.O’MahoneyM.WichchukitS. (2010). Various paired preference tests: Experimenter effect on take home choice. Journal of Sensory Studies, 25, 778–790.
101.
WichertsJ. M.BakkerM.MolenaarD. (2011). Willingness to share research data is related to strength of the evidence and the quality of reporting statistical results. PLoS ONE, 6(11), e26828. doi:10.1371/journal.pone.0026828
102.
WichertsJ. M.BorsboomD.KatsJ.MolenaarD. (2006). The poor availability of psychological research data for reanalysis. American Psychologist, 61, 726–728.
103.
*WilkinsW. (1986). Therapy-therapist confounds in psychotherapy research. Cognitive Therapy and Research, 10, 3–11.
104.
*WilliamsD. A.ThornB. E. (1986). Can research methodology affect treatment outcome? A comparison of two cold pressor test paradigms. Cognitive Therapy and Research, 10, 539–545.
105.
*WoehrD. J.ArthurW.Jr. (2003). The construct-related validity of assessment center ratings: A review and meta-analysis of the role of methodological factors. Journal of Management, 29, 231–258.
106.
*WolchikS. A.SpencerS. L.LisiI. S. (1983). Volunteer bias in research employing vaginal measures of sexual arousal. Archives of Sexual Behavior, 12, 399–408.
107.
WolinsL. (1962). Responsibility for raw data. American Psychologist, 17, 657–658.
*YoonC.LeeM. P.DanzigerS. (2007). The effects of optimal time of day on persuasion processes in older adults. Psychology & Marketing, 24, 475–495. doi:10.1002/mar.20169
110.
*ZagarR.BowersN. D. (1983). The effect of time of day on problem solving and classroom behavior. Psychology in Schools, 20, 337–345.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
