In this methodological commentary, we use Bem's (2011) recent article reporting experimental evidence for psi as a case study for discussing important deficiencies in modal research practice in empirical psychology. We focus on (a) overemphasis on conceptual rather than close replication, (b) insufficient attention to verifying the soundness of measurement and experimental procedures, and (c) flawed implementation of null hypothesis significance testing. We argue that these deficiencies contribute to weak method-relevant beliefs that, in conjunction with overly strong theory-relevant beliefs, lead to a systemic and pernicious bias in the interpretation of data that favors a researcher's theory. Ultimately, this interpretation bias increases the risk of drawing incorrect conclusions about human psychology. Our analysis points to concrete recommendations for improving research practice in empirical psychology. We recommend (a) a stronger emphasis on close replication, (b) routinely verifying the integrity of measurement instruments and experimental procedures, and (c) using stronger, more diagnostic forms of null hypothesis testing.
AmirY., & SharonI. (1991). Replication research: A “must” for the scientific advancement of psychology. In NeuliepJ. W., Replication research in the social sciences (pp. 51–69). Newbury Park, CA: Sage.
2.
BakanD. (1966). The test of significance in psychological research. Psychological Bulletin, 66, 1–29. doi:10.1037/h0020412
3.
BemD. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100, 407–425. doi:10.1037/a0021524
4.
BorsboomD., MellenberghG. J., & van HeerdenJ. (2004). The concept of validity. Psychological Review, 111, 1061–1071. doi:10.1037/0033-295X.111.4.1061
5.
CohenJ. (1994). The earth is round (p < .05). American Psychologist, 49, 997–1003. doi:10.1037/0003-066X.49.12.997
6.
CollinsH. M. (1984). Discussion: When do scientists prefer to vary their experiments. Studies in the History and Philosophy of Science, 15, 169–174. doi:10.1016/0039-3681(84)90005-0
7.
CookT. D., & GroomC. (2004). The methodological assumptions of social psychology: The mutual dependence of substantive theory and method choice. In SansoneC., MorfC. C., & PanterA. T., The Sage handbook of methods in social psychology (pp. 19–44). Thousand Oaks, CA: Sage.
8.
CummingG. (2008). Replication and p intervals: p values predict the future only vaguely, but confidence intervals do much better. Perspectives on Psychological Science, 3, 286–300. doi:10.1111/j.1745-6924.2008.00079.x
9.
DawesR. M. (1988). Rational choice in an uncertain world. San Diego, CA: Harcourt Brace Jovanovich.
10.
DunlapK. (1926). The experimental methods of psychology. In MurchisonC., Psychologies of 1925. (pp. 331–351) Worcester, MA: Clark University Press.
11.
DeShonR. P. (2004). Measures are not invariant across groups without error variance homogeneity. Psychology Science, 46, 137–149.
12.
DuhemP. (1954). The aim and structure of physical theory (WienerP. P.,). Princeton, NJ: Princeton University Press.
13.
FalkR. (1998). Replication—A step in the right direction. Theory & Psychology, 8, 313–321. doi:10.1177/0959354398083002
14.
FalkR., & GreenbaumC. W. (1995). Significance tests die hard: The amazing persistence of a probabilistic misconception. Theory & Psychology, 5, 75–98. doi:10.1177/0959354395051004
15.
FanelliD. (2010). “Positive” results increase down the hierarchy of the sciences. PLoS ONE5, e10068. doi:10.1371/journal.pone.0010068
16.
FisherR. A. (1926). Statistical methods for research workers (1st ed.). London, England: Oliver & Boyd.
17.
FisherR. A. (1934). Statistical methods for research workers (5th ed.). London, England: Oliver & Boyd.
18.
GawronskiB., DeutschR., & BanseR. (2011). Response interference tasks as indirect measures of automatic associations. In KlauerK. C., StahlC., & VossA., Cognitive methods in social psychology (pp. 78–123). New York, NY: Guilford Press.
19.
GergenK. J. (1982). Toward transformation in social knowledge. New York, NY: Springer-Verlag.
20.
GreenwaldA. G., GonzalezR., HarrisR. J., & GuthrieD. (1996). Effect size and p values: What should be reported and what should be replicated?Psychophysiology, 33, 175–183. doi:10.1111/j.1469-8986.1996.tb02121.x
21.
GuttmanL. (1997). What is not what in statistics. The Statistician, 26, 81–107. doi:10.2307/2987957
22.
HendrickC. (1991). Replications, strict replications, and conceptual replications: Are they important? In NeuliepJ. W., Replication research in the social sciences (pp. 41–49). Newbury Park, CA: Sage.
23.
KashyD. A., DonnellanM. B., AckermanR. A., & RussellD. W. (2009). Reporting and interpreting research in PSPB: Practices, principles, and pragmatics. Personality and Social Psychology Bulletin, 35, 1131–1142. doi:10.1177/0146167208331253
24.
KirkR. E. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746–759. doi:10.1177/0013164496056005002
25.
KlineR. B. (2004). Beyond significance testing. Washington, DC: American Psychological Association. doi:10.1037/10693-000
26.
KruschkeJ. K. (2010). Bayesian data analysis. Wiley Interdisciplinary Reviews: Cognitive Science, 1, 658–676. doi:10.1002/wcs.72
27.
LeBelE. P., & PaunonenS. V. (2011). Sexy but often unreliable: Impact of unreliability on the replicability of experimental findings involving implicit measures. Personality and Social Psychology Bulletin, 37, 570–583. doi:10.1177/0146167211400619
LindsayR. M., & EhrenbergA. S. C. (1993). The design of replicated studies. The American Statistician, 47, 217–228. doi:10.2307/2684982
30.
LykkenD. T. (1968). Statistical significance in psychological research. Psychological Bulletin, 70, 151–159. doi:10.1037/h0026141
31.
MaxwellS. E. (2004). The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods, 9, 147–163. doi:10.1037/1082-989X.9.2.147
32.
McGuireW. J. (1973). The yin and yang of progress in social psychology: Seven koan. Journal of Personality and Social Psychology, 26, 446–456. doi:10.1037/h0034345
33.
MeehlP. E. (1967). Theory testing in psychology and physics: A methodological paradox. Philosophy of Science, 34, 103–115. doi:10.1086/288135
34.
MeehlP. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806–834. doi:10.1037/0022-006X.46.4.806
35.
MeehlP. E. (1990). Why summaries of research on psychological theories are often uninterpretable. Psychological Reports, 66, 195–244. doi:10.2466/PR0.66.1.195-244
36.
MessickS. (1989). Validity. In LinnR. L., Educational measurement (3rd ed., pp. 13–103). Washington, DC: American Council on Education.
37.
MichellJ. (1997). Quantitative science and the definition of measurement in psychology. British Journal of Psychology, 88, 355–383. doi:10.1111/j.2044-8295.1997.tb02641.x
38.
MillerJ. (2009). What is the probability of replicating a statistically significant effect?Psychonomic Bulletin & Review, 16, 617–640. doi:10.3758/PBR.16.4.617
39.
MoonesingheR., KhouryM. J., & JanssensC. J. W. (2007). Most published research findings are false—But a little replication goes a long way. PLoS Medicine, 4, 218–221. doi:10.1371/journal.pmed.0040028
40.
MulaikS. A., RajuN. S., & HarshmanR. A. (1997). There is a time and a place for significance testing. In HarlowL. L., MulaikS. A., & SteigerJ. H., What if there were no significance tests? (pp. 65–115). Mahwah, NJ: Erlbaum.
41.
MulkayM., & GilbertG. N. (1986). Replication and mere replication. Philosophy of the Social Sciences, 16, 21–37. doi:10.1177/004839318601600102
42.
OppenheimerD. M., MeyvisT., & DavidenkoN. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45, 867–872. doi:10.1016/j.jesp.2009.03.009
43.
PetersK. R. (2011). Cronbach's challenge: Putting psychological explanation in context. Unpublished manuscript, The University of Western Ontario, London, Ontario, Canada.
44.
PopperK. R. (1959). The logic of scientific discovery. New York, NY: Basic Books.
45.
PopperK. R. (1963). Conjectures and refutations. London, England: Routledge and Kegan Paul.
46.
QuineW. V. O. (1953). Two dogmas of empiricism. In From a logical point of view (2nd ed., pp. 20–46). Cambridge, MA: Harvard University Press.
47.
QuineW. V. O., & UllianJ. S. (1978). The web of belief (2nd ed.). New York, NY: Random House.
48.
RosenthalR. (1979). The “file drawer” problem and tolerance for null results. Psychological Bulletin, 86, 638–641.
49.
RosenthalR. (1991). Replication in behavioral research. In NeuliepJ. W., Replication research in the social sciences (pp. 1–30). Newbury Park, CA: Sage.
50.
RosenthalR. (1993). Cumulating evidence. In (Eds.), A handbook for data analysis in the behavioral sciences: Methodological issues (pp. 519–559). Hillsdale, NJ: Erlbaum.
51.
RozeboomW. W. (1997). Good science is abductive, not hypothetico-deductive. In HarlowL. L., MulaikS. A., & SteigerJ. H., What if there were no significance tests? (pp. 335–391). Hillsdale, NJ: Erlbaum.
52.
RunkelP. J. (2007). Casting nets and testing specimens: Two grand methods of psychology. Hayward, CA: Living Control Systems.
53.
SohnD. (1998). Statistical significance and replicability: Why the former does not presage the latter. Theory & Psychology, 8, 291–311. doi:10.1177/0959354398083001
54.
ThompsonB. (1992). Two and one-half decades of leadership in measurement and evaluation. Journal of Counseling and Development, 70, 434–438.
TukeyJ. W. (1969). Analyzing data: Sanctification or detective work?American Psychologist, 24, 83–91. doi:10.1037/h0027108
57.
TukeyJ. W. (1991). The philosophy of multiple comparisons. Statistical Science, 6, 100–116. doi:10.1214/ss/1177011945
58.
Vacha-HaaseT., NessC., NilssonJ., & ReetzD. (1999). Practices regarding reporting of reliability coefficients: A review of three journals. Journal of Experimental Education, 67, 335–341. doi:10.1080/00220979909598487
59.
WagenmakersE.-J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14, 779–804. doi:10.3758/BF03194105
60.
WagenmakersE.-J., WetzelsR., BorsboomD., & van der MaasH. L. J. (2011). Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of Personality and Social Psychology, 100, 426–432. doi:10.1037/a0022790
61.
WallachL., & WallachM. A. (1994). Gergen versus the mainstream: Are hypotheses in social psychology subject to empirical test?Journal of Personality and Social Psychology, 67, 233–242. doi:10.1037/0022-3514.67.2.233
62.
WilkinsonL., & the Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594–604. doi:10.1037/0003-066X.54.8.594
