Evaluating Research in Personality and Social Psychology: Considerations of Statistical Power and Concerns About False Findings

Abstract

Traditionally, statistical power was viewed as relevant to research planning but not evaluation of completed research. However, following discussions of high false finding rates (FFRs) associated with low statistical power, the assumed level of statistical power has become a key criterion for research acceptability. Yet, the links between power and false findings are not as straightforward as described. Assumptions underlying FFR calculations do not reflect research realities in personality and social psychology. Even granting the assumptions, the FFR calculations identify important limitations to any general influences of statistical power. Limits for statistical power in inflating false findings can also be illustrated through the use of FFR calculations to (a) update beliefs about the null or alternative hypothesis and (b) assess the relative support for the null versus alternative hypothesis when evaluating a set of studies. Taken together, statistical power should be de-emphasized in comparison to current uses in research evaluation.

Keywords

statistical power false finding rate research evaluation replication NHST research methods/assessment advanced quantitative methods

Get full access to this article

View all access options for this article.

References

Anderson

C. A.

Allen

J. J.

Plante

Quigley-McBride

Lovett

Rokkum

J. N.

(2019). The MTurkification of social and personality psychology. Personality and Social Psychology Bulletin, 45(6), 842–850.

Braver

S. L.

Thoemmes

F. J.

Rosenthal

(2014). Continuously cumulating meta-analysis and replicability. Perspectives on Psychological Science, 9(3), 333–342.

Brewer

J. K.

(1972). On the power of statistical tests in the American Educational Research Journal. American Educational Research Journal, 9(3), 391–401.

Button

K. S.

Ioannidis

J. P. A.

Mokrysz

Nosek

B. A.

Flint

Robinson

E. S. J.

Munafὸ

M. R.

(2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews, 14, 365–376.

Button

K. S.

Munafὸ

M. R.

(2017). Powering reproducible research. In Lilienfeld

S. O.

Waldman

I. D.

(Eds.), Psychological science under scrutiny (pp. 22–33). John Wiley.

Christley

(2010). Power and error: Increased risk of false positive results in underpowered studies. The Open Epidemiology Journal, 3(1), 16–19.

Cohen

(1973). Brief notes: Statistical power analysis and research results. American Educational Research Journal, 10(3), 225–229.

Cohen

(1988). Statistical power analysis for the behavioral sciences. Academic Press.

Cohen

(1994). The Earth is round (p < .05). American Psychologist, 49(12), 997–1003.

10.

Cooper

M. L.

(2016). Personality processes and individual differences [Editorial]. Journal of Personality and Social Psychology, 110(3), 431–434.

11.

Cortina

J. M.

Dunlap

W. P.

(1997). On the logic and purpose of significance testing. Psychological Methods, 2(2), 161–172.

12.

Curran

P. J.

Hussong

A. M.

(2009). Integrative data analysis: The simultaneous analysis of multiple data sets. Psychological Methods, 14(2), 81–100.

13.

Eich

(2014). Business not as usual. Psychological Science, 25(1), 3–6.

14.

Fabrigar

L. R.

Wegener

D. T.

(2016). Conceptualizing and evaluating the replication of research results. Journal of Experimental Social Psychology, 66, 68–80.

15.

Fabrigar

L. R.

Wegener

D. T.

Petty

R. E.

(2020). A validity-based framework for understanding replication in psychology. Personality and Social Psychology Review, 24(4), 316–344.

16.

Finkel

E. J.

Eastwick

P. W.

Reis

H. T.

(2017). Replicability and other features of a high-quality science: Toward a balanced and empirical approach. Journal of Personality and Social Psychology, 113(2), 244–253.

17.

Flake

J. K.

Pek

Hehman

(2017). Construct validation in social and personality research: Current practice and recommendations. Social Psychological and Personality Science, 8(4), 370–378.

18.

Funder

D. C.

Levine

J. M.

Mackie

D. M.

Morf

C. C.

Vazire

West

S. G.

(2013). Improving the dependability of research in personality and social psychology: Recommendations for research and educational practice. Personality and Social Psychology Review, 18(1), 3–12.

19.

Giner-Sorolla

Aberson

C. L.

Bostyn

D. H.

Carpenter

Conrique

B. G.

Lewis

N. A.

Jr. Montoya

A. K.

B. W.

Reifman

Schoemann

A. M.

Soderberg

(2019). Power to detect what? Considerations for planning and evaluating sample size [Report of the SPSP Power Analysis Working Group]. https://osf.io/jnmya/on4/21/2020

20.

Hagen

R. L.

(1997). In praise of the null hypothesis statistical test. American Psychologist, 52(1), 15–24.

21.

Hedges

L. V.

Schauer

J. M.

(2019). Statistical analyses for studying replication: Meta-analytic perspectives. Psychological Methods, 24(5), 557–570.

22.

Ioannidis

J. P.

(2005). Why most published research findings are false. PLOS Medicine, 2(8), Article e124.

23.

Ioannidis

J. P.

Trikalinos

T. A.

(2007). An exploratory test for an excess of significant findings. Clinical Trials, 4(3), 245–253.

24.

Jones

L. V.

Tukey

J. W.

(2000). A sensible formulation of the significance test. Psychological Methods, 5(4), 411–414.

25.

Kass

R. E.

Raftery

A. E.

(1995). Bayes factors. Journal of the American Statistical Association, 90(430), 773–795.

26.

Kawakami

(2015). Interpersonal relations and group processes [Editorial]. Journal of Personality and Social Psychology, 108(1), 58–59.

27.

Kenny

D. A.

Judd

C. M.

(2019). The unappreciated heterogeneity of effect sizes: Implications for power, precision, planning of research, and replication. Psychological Methods, 24(5), 578–589.

28.

Kerr

N. L.

(1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196–217.

29.

Kitayama

(2017). Attitudes and social cognition [Editorial]. Journal of Personality and Social Psychology, 112(3), 357–360.

30.

Leventhal

Huynh

C. L.

(1996). Directional decisions for two-tailed tests: Power, error rates, and sample size. Psychological Methods, 1(3), 278–292.

31.

Lindsay

D. S.

(2015). Replication in psychological science [Editorial]. Psychological Science, 26(12), 1827–1832.

32.

McShane

B. B.

Böckenholt

(2017). Single-paper meta-analysis: Benefits for study summary, theory testing, and replicability. Journal of Consumer Research, 43(6), 1048–1063.

33.

McShane

B. B.

Böckenholt

Hansen

K. T.

(2020). Average power: A cautionary note. Advances in Methods and Practices in Psychological Science, 3(2), 185–199.

34.

Moonesinghe

Khoury

M. J.

Janssens

C. J. W.

(2007). Most published research findings are false—But a little replication goes a long way. PLOS Medicine, 4(2), 0218–0221.

35.

Nickerson

R. S.

(2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5(2), 241–301.

36.

Pashler

Harris

C. R.

(2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7(6), 531–536.

37.

Pek

Hoisington-Shaw

K. J.

Wegener

D. T.

(in press). Avoiding questionable research practices surrounding statistical power analysis. In Lilienfeld

O’Donohue

Masuda

(Eds.), Questionable research practice. Springer.

38.

Pek

Park

(2019). Complexities in power analysis: Quantifying uncertainties with a Bayesian-classical hybrid approach. Psychological Methods, 24(5), 590–605.

39.

Schaller

(2016). The empirical benefits of conceptual rigor: Systematic articulation of conceptual hypotheses can reduce the risk of non-replicable results (and facilitate novel discoveries too). Journal of Experimental Social Psychology, 66, 107–115.

40.

Simmons

J. P.

Nelson

L. D.

Simonsohn

(2011). False-positive psychology undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366.

41.

Stanley

D. J.

Spence

J. R.

(2014). Expectations for replications: Are yours realistic? Perspectives on Psychological Science, 9(3), 305–318.

42.

Stroebe

(2016). Are most published social psychological findings false? Journal of Experimental Social Psychology, 66, 134–144.

43.

Tukey

J. W.

(1991). The philosophy of multiple comparisons. Statistical Science, 6(1), 100–116.

44.

Vazire

(2016). Editorial. Social Psychological and Personality Science, 7(1), 3–7.

45.

Wallace

L. E.

Wegener

D. T.

Petty

R. E.

(2020). Influences of source bias that differ from source untrustworthiness: When flip-flopping is more and less surprising. Journal of Personality and Social Psychology, 118(4), 603–616.

46.

Yuan

K.-H.

Maxwell

(2005). On the post hoc power in testing mean differences. Journal of Educational and Behavioral Statistics, 30(2), 141–167.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.12 MB