Toward a More Powerful Experimental Communication Science: An Assessment of Two Decades’ Research (2001

Abstract

Low statistical power undermines a credible scientific discipline. This paper presents a statistical power assessment of experimental communication research based on a random sample of 416 studies published in five central communication journals over the last two decades (2001–2023). Our analyses showed that there was a lack of attention to power and power analysis, with the majority of studies not even mentioning statistical power. We also demonstrated that between-subjects studies were overall underpowered for two-group, one-way multi-group, and factorial designs. Our findings suggest that conducting power analyses to ensure adequate statistical power should be a key priority for experimental communication research.

Keywords

statistical power experimental communication research power analysis effect size open science

Get full access to this article

View all access options for this article.

References

American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). American Psychological Association

American Psychological Association. (2020). Publication manual of the American Psychological Association (7th ed.). American Psychological Association

Amrhein

Trafimow

Greenland

(2019). Inferential statistics as descriptive statistics: There is no replication crisis if we don’t expect replication. American Statistician, 73(sup1), 262–270. https://doi.org/10.1080/00031305.2018.1543137

Anderson

S. F.

Kelley

Maxwell

S. E.

(2017). Sample-size planning for more accurate statistical power: A method adjusting sample effect sizes for publication bias and uncertainty. Psychological Science, 28(11), 1547–1562. https://doi.org/10.1177/0956797617723724

Baguley

(2009). Standardized or simple effect size: What should be reported? British Journal of Psychology, 100(Pt 3), 603–617. https://doi.org/10.1348/000712608X377117

Bakker

B. N.

Jaidka

Dörr

Fasching

Lelkes

(2021). Questionable and open research practices: Attitudes and perceptions among quantitative communication researchers. Journal of Communication, 71(5), 715–738. https://doi.org/10.1093/joc/jqab031

Bakker

van Dijk

Wicherts

J. M.

(2012). The rules of the game called psychological science. Perspectives on Psychological Science: A Journal of the Association for Psychological Science, 7(6), 543–554. https://doi.org/10.1177/1745691612459060

Bakker

Hartgerink

C. H.

Wicherts

J. M.

van der Maas

H. L.

(2016). Researchers’ intuitions about power in psychological research. Psychological Science, 27(8), 1069–1077. https://doi.org/10.1177/0956797616647519

Berinsky

A. J.

Druckman

J. N.

Yamamoto

(2021). Publication biases in replication studies. Political Analysis, 29(3), 370–384. https://doi.org/10.1017/pan.2020.34

10.

Bishop

D. V.

(2020). The psychology of experimental psychologists: Overcoming cognitive constraints to improve research: The 47th sir Frederic Bartlett Lecture. Quarterly Journal of Experimental Psychology, 73(1), 1–19. https://doi.org.ezproxy.cityu.edu.hk/10.1177/1747021819886519

11.

Borenstein

Hedges

L. V.

Higgins

J. P.

Rothstein

H. R.

(2021). Introduction to meta-analysis (2nd ed.). John Wiley & Sons.

12.

Button

K. S.

Ioannidis

J. P.

Mokrysz

Nosek

B. A.

Flint

Robinson

E. S.

Munafò

M. R.

(2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365–376. https://doi.org/10.1038/nrn3475

13.

Chambers

(2019). The Seven Deadly Sins of Psychology. Princeton University Press.

14.

Champely

(2020). PWR: Basic Functions for Power Analysis. R Package Version 1.3-0. https://CRAN.R-project.org/package=pwr

15.

Chase

L. J.

(1977). Diffusion of power-related data in communication: A research note. Human Communication Research, 3(4), 363–367.

16.

Chester

D. S.

Lasko

E. N.

(2021). Construct validation of experimental manipulations in social psychology: Current practices and recommendations for the future. Perspectives on Psychological Science, 16(2), 377–395. https://doi.org/10.1177/1745691620950684

17.

Coenen

Smits

(2022). Strong-form frequentist testing in communication science: Principles, opportunities, and challenges. Communication Methods and Measures, 16(4), 237–265. https://doi.org/10.1080/19312458.2022.2086690

18.

Cohen

(1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 65(3), 145–153. https://doi.org/10.1037/h0045186

19.

Cohen

(1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Routledge.

20.

Cohen

(1990). Things I have learned (so far). American Psychologist, 45(12), 1304–1312. https://doi.org/10.1037//0003-066x.45.12.1304

21.

Cohen

(1994). The earth is round (p < .05). American Psychologist, 49(12), 997–1003. https://doi.org/10.1037//0003-066x.49.12.997

22.

Colquhoun

(2014). An investigation of the false discovery rate and the misinterpretation of p-values. Royal Society Open Science, 1(3), 140216. https://doi.org/10.1098/rsos.140216

23.

Cores-Sarría

Han

Myrick

J. G.

Potter

R. F.

(2023). The effects of “Media Tech Neck”: The impact of spinal flexion on cognitive and emotional processing of videos. Communication Research, 50(5), 584–609. https://doi.org/10.1177/0093650222115031

24.

Christensen

Freese

Miguel

(2019). Transparent and reproducible social science research: How to do open science. University of California Press.

25.

De Ridder

Pabian

Vandebosch

Dhoest

. (2023). Achieving destigmatizing outcomes by overcoming resistance to persuasion through combined entertainment experiences. Media Psychology, 26(2), 201–227. https://doi.org/10.1080/15213269.2022.2118781

26.

Dienes

(2008). Understanding psychology as a science: An introduction to scientific and statistical inference. Bloomsbury Publishing.

27.

Dienes

(2014). Using Bayes to get the most out of non-significant results. Frontiers in Psychology, 5, Article No. 781. https://doi.org/10.3389/fpsyg.2014.00781

28.

Dienlin

Johannes

Bowman

N. D.

Masur

P. K.

Engesser

Kümpel

A. S.

Lukito

Bier

L. M.

Zhang

Johnson

B. K.

Huskey

Schneider

F. M.

Breuer

Parry

D. A.

Vermeulen

Fisher

J. T.

Banks

Weber

Ellis

D. A.

, . . . de Vreese

(2021). An agenda for open science in Communication. Journal of Communication, 71(1), 1–26. https://doi.org/10.1093/joc/jqz052

29.

Elson

Przybylski

A. K.

(2017). The science of technology and human behavior. Journal of Media Psychology, 29(1), 1–7. https://doi.org/10.1027/1864-1105/a000212

30.

Ferguson

C. J.

(2009). An effect size primer: A guide for clinicians and researchers. Professional Psychology Research and Practice, 40(5), 532–538. https://doi.org/10.1037/a0015808

31.

Ferguson

C. J.

Heene

(2012). A vast graveyard of undead theories: Publication bias and psychological science’s aversion to the null. Perspectives on Psychological Science, 7(6), 555–561. https://doi.org/10.1177/1745691612459059

32.

Ferguson

C. J.

Heene

(2021). Providing a lower-bound estimate for psychology’s “crud factor”: The case of aggression. Professional Psychology Research and Practice, 52(6), 620–626. https://doi.org/10.1037/pro0000386

33.

Faul

Erdfelder

Lang

A. G.

Buchner

(2007). G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior research methods, 39(2), 175–191. https://doi.org/10.3758/bf03193146

34.

Fiedler

Kutzner

Krueger

J. I.

(2012). The long way from α-error control to validity proper: Problems with a short-sighted false-positive debate. Perspectives on Psychological Science, 7(6), 661–669. https://doi.org/10.1177/1745691612462587

35.

Fraley

R. C.

Vazire

(2014). The N-pact factor: Evaluating the quality of empirical journals with respect to sample size and statistical power. PLoS One, 9(10), e109019. https://doi.org/10.1371/journal.pone.0109019

36.

Frith

(2020). Fast lane to slow science. Trends in Cognitive Sciences, 24(1), 1–2. https://doi.org/10.1016/j.tics.2019.10.007

37.

Funder

D. C.

Ozer

D. J.

(2019). Evaluating effect size in psychological research: Sense and nonsense. Advances in Methods and Practices in Psychological Science, 2(2), 156–168. https://doi.org/10.1177/2515245919847202

38.

Garrison

J. P.

Andersen

P. A.

(1979). A reassessment of statistical power analysis in human communication research. Human Communication Research, 5(4), 343–350. https://doi.org/10.1111/j.1468-2958.1979.tb00647.x

39.

Gelman

(2017). The “80% Power” Lie. Statistical Modeling, Causal Inference, and Social Science. https://statmodeling.stat.columbia.edu/2017/12/04/80-power-lie/

40.

Gelman

Carlin

(2014). Beyond power calculations: Assessing Type S (sign) and Type M (magnitude) errors. Perspectives on Psychological Science, 9(6), 641–651. https://doi.org/10.1177/1745691614551642

41.

Gignac

G. E.

Szodorai

E. T.

(2016). Effect size guidelines for individual differences researchers. Personality and Individual Differences, 102, 74–78. https://doi.org/10.1016/j.paid.2016.06.069

42.

Giner-Sorolla

(2018, January 24). Powering your interaction. Approaching significance. https://approachingblog.wordpress.com/2018/01/24/powering-your-interaction-2/

43.

Giner-Sorolla

Montoya

A. K.

Reifman

Carpenter

Lewis

N. A.

Aberson

C. L.

Bostyn

D. H.

Conrique

B. G.

B. W.

Schoemann

A. M.

Soderberg

(2024). Power to detect what? Considerations for planning and evaluating sample size. Personality and Social Psychology Review, 28(3), 276–301. https://doi.org/10.1177/10888683241228328

44.

Greenland

(2012). Nonsignificance plus high power does not imply support for the null over the alternative. Annals of Epidemiology, 22(5), 364–368. https://doi.org/10.1016/j.annepidem.2012.02.007

45.

Harris

(Ed.). (2017). Rigor mortis: How sloppy science creates worthless cures, crushes hope, and wastes billions (1st ed.). Basic Books.

46.

Hartgerink

C. H. J.

Wicherts

J. M.

van Assen

M. A. L. M

. (2017). Too good to be false: Nonsignificant results revisited. Collabra, 3(1), 9. https://doi.org/10.1525/collabra.71

47.

Heo

Kim

Faith

M. S.

(2015). Statistical power as a function of Cronbach alpha of instrument questionnaire items. BMC Medical Research Methodology, 15(1), 86. https://doi.org/10.1186/s12874-015-0070-6

48.

Hoenig

J. M.

Heisey

D. M.

(2001). The abuse of power: The pervasive fallacy of power calculations for data analysis. American Statistician, 55(1), 19–24. https://doi.org/10.1198/000313001300339897

49.

Ioannidis

J. P. A.

(2005). Why most published research findings are false. PLoS Medicine, 2(8), e124.

50.

Ioannidis

J. P. A.

Stanley

T. D.

Doucouliagos

(2017). The power of bias in economics research. Econometrics Journal, 127(605), F236–F265. https://doi.org/10.1111/ecoj.12461

51.

John

L. K.

Loewenstein

Prelec

(2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953

52.

Katzer

Sodt

(1973). An analysis of the use of statistical testing in communication research. Journal of Communication, 23(3), 251–265.

53.

Kvarven

Strømland

Johannesson

(2020). Comparing meta-analyses and preregistered multiple-laboratory replication projects. Nature Human Behaviour, 4(4), 423–434. https://doi.org/10.1038/s41562-019-0787-z

54.

Lakens

(2022). Sample size justification. Collabra, 8(1), 33267. https://doi.org/10.1525/collabra.33267

55.

Lakens

Caldwell

A. R.

(2021). Simulation-based power analysis for factorial analysis of variance designs. Advances in Methods and Practices in Psychological Science, 4(1), 2515245920951503. https://doi.org/10.1177/2515245920951503

56.

Langford

(2017). Why you should think of statistical power as a curve. https://psychbrief.wordpress.com/2017/12/14/power-curve/

57.

Lantz

(2013). The large sample size fallacy. Scandinavian Journal of Caring Sciences, 27(2), 487–492. https://doi.org/10.1111/j.1471-6712.2012.01052.x

58.

Lee

Kim

(2023). Perceived influence of partisan news and online news participation: Third-person effect, hostile media phenomenon, and cognitive elaboration. Communication Research, 50(7), 854–878. https://doi.org/10.1177/00936502221127494

59.

Wagner

M. W.

(2020). The value of not knowing: Partisan cue-taking and belief updating of the uninformed, the ambiguous, and the misinformed. Journal of Communication, 70(5), 646–669. https://doi.org/10.1093/joc/jqaa022

60.

Lilienfeld

S. O.

(2017). Psychology’s replication crisis and the grant culture: Righting the ship. Perspectives on Psychological Science, 12(4), 660–664. https://doi.org/10.1177/1745691616687745

61.

Lodder

Ong

H. H.

Grasman

R. P. P. P.

Wicherts

J. M.

(2019). A comprehensive meta-analysis of money priming. Journal of Experimental Psychology General, 148(4), 688–712. https://doi.org/10.1037/xge0000570

62.

Maier

Lakens

(2022). Justify your alpha: A primer on two practical approaches. Advances in Methods and Practices in Psychological Science, 5(2), 25152459221080396. https://doi.org/10.1177/25152459221080396

63.

Maxwell

S. E.

(2004). The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods, 9(2), 147–163.

64.

Maxwell

S. E.

Kelley

Rausch

J. R.

(2008). Sample size planning for statistical power and accuracy in parameter estimation. Annual Review of Psychology, 59(1), 537–563. https://doi.org/10.1146/annurev.psych.59.103006.093735

65.

Mayo

D. G.

(2018). Statistical inference as severe testing: How to get beyond the statistics wars. Cambridge University Press.

66.

McKay

Bacelar

M. F. B.

Carter

M. J.

(2023). On the reproducibility of power analyses in motor behavior research. Journal of Motor Learning and Development, 11(1), 29–44. https://doi.org/10.1123/jmld.2022-0061

67.

Meehl

P. E.

(1967). Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 34(2), 103–115.

68.

Meng

X. L.

(2018). Statistical paradises and paradoxes in big data (I): Law of large populations, big data paradox, and the 2016 US presidential election. Annals of Applied Statistics, 12(2), 685–726. https://doi.org/10.1214/18-aoas1161sf

69.

Meyvis

Van Osselaer

S. M. J.

(2018). Increasing the power of your study by increasing the effect size. Journal of Consumer Research, 44(5), 1157–1173. https://doi.org/10.1093/jcr/ucx110

70.

Neyman

Pearson

E. S.

(1933). On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society A: Containing Papers of a Mathematical or Physical Character, 231, 289–337. https://www.jstor.org/stable/91247

71.

Montoya

A. K.

Krenzer

W. L. D.

Fossum

J. L.

(2021). Opening the door to registered reports: Census of journals publishing registered reports (2013–2020). Collabra, 7(1), 24404. https://doi.org/10.1525/collabra.24404

72.

Motyl

Demos

A. P.

Carsel

T. S.

Hanson

B. E.

Melton

Z. J.

Mueller

A. B.

Prims

J. P.

Sun

Washburn

A. N.

Wong

K. M.

Yantis

Skitka

L. J.

(2017). The state of social and personality science: Rotten to the core, not so bad, getting better, or getting worse? Journal of Personality and Social Psychology, 113(1), 34–58. https://doi.org/10.1037/pspa0000084

73.

Nelson

L. D.

Simmons

Simonsohn

(2018). Psychology’s renaissance. Annual Review of Psychology, 69(1), 511–534. https://doi.org/10.1146/annurev-psych-122216-011836

74.

Nuijten

M. B.

van Assen

M. A. L. M.

Augusteijn

H. E. M.

Crompvoets

E. A. V.

Wicherts

J. M.

(2020). Effect Sizes, power, and biases in intelligence research: A Meta-meta-analysis. Journal of Intelligence, 8(4), Article 4. https://doi.org/10.3390/jintelligence8040036

75.

Oakes

M. W.

(1990). Statistical inference. Epidemiology Resources.

76.

Oschatz

Niederdeppe

Liu

(2021). The role of prior attitudes in narrative persuasion: Evidence from a cross-national study in Germany and the United States. Communication Monographs, 89(3), 376–395. https://doi.org/10.1080/03637751.2021.2011344

77.

Page

M. J.

McKenzie

J. E.

Bossuyt

P. M.

Boutron

Hoffmann

T. C.

Mulrow

C. D.

Shamseer

Tetzlaff

J. M.

Akl

E. A.

Brennan

S. E.

Chou

Glanville

Grimshaw

J. M.

Hróbjartsson

Lalu

M. M.

Loder

E. W.

Mayo-Wilson

McDonald

. . . Moher

(2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, 372. https://doi.org/10.1136/bmj.n71

78.

Perugini

Gallucci

Costantini

(2014). Safeguard power as a protection against imprecise power estimates. Perspectives on Psychological Science, 9(3), 319–332. https://doi.org.ezproxy.cityu.edu.hk/10.1177/1745691614528519

79.

Pek

Flora

D. B.

(2018). Reporting effect sizes in original psychological research: A discussion and tutorial. Psychological Methods, 23(2), 208–225. https://doi.org/10.1037/met0000126

80.

Rains

S. A.

Levine

T. R.

Weber

(2018). Sixty years of quantitative communication research summarized: Lessons from 149 meta-analyses. Annals of the International Communication Association, 42(2), 105–124. https://doi.org/10.1080/23808985.2018.1446350

81.

Rouder

J. N.

Morey

R. D.

Verhagen

Province

J. M.

Wagenmakers

(2016). Is there a free lunch in inference? Topics in Cognitive Science, 8(3), 520–547. https://doi.org/10.1111/tops.12214

82.

Rubin

D. B.

(2008). For objective causal inference, design trumps analysis. Annals of Applied Statistics, 2(3), 808–840. https://doi.org/10.1214/08-aoas187

83.

Schäfer

Schwarz

M. A.

(2019). The meaningfulness of effect sizes in psychological research: Differences between sub-disciplines and the impact of potential biases. Frontiers in Psychology, 10, 813. https://doi.org/10.3389/fpsyg.2019.00813

84.

Scheel

A. M.

Schijen

M. R. M. J.

Lakens

(2021). An excess of positive results: Comparing the standard psychology literature with registered reports. Advances in Methods and Practices in Psychological Science, 4(2), 1–12. https://doi.org/10.1177/25152459211007467

85.

Schulz

K. F.

Grimes

D. A.

(2005). Sample size calculations in randomised trials: Mandatory and mystical. Lancet, 365(9467), 1348–1353. https://doi.org/10.1016/s0140-6736(05)61034-3

86.

Simonsohn

(2014, March 12). [17] No-way interactions. Data Colada. http://datacolada.org/17

87.

Simonsohn

(2015). Small telescopes: Detectability and the evaluation of replication results. Psychological Science, 26(5), 559–569. https://doi.org/10.1177/0956797614567341

88.

Simonsohn

Nelson

L. D.

Simmons

J. P.

(2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology General, 143(2), 534–547.

89.

Smith

R. A.

Levine

T. R.

Lachlan

K. A.

Fediuk

T. A.

(2002). The high cost of complexity in experimental design and data analysis: Type I and type II error rates in multiway ANOVA. Human Communication Research, 28(4), 515–530. https://doi.org/10.1111/j.1468-2958.2002.tb00821.x

90.

Sommet

Weissman

D. L.

Cheutin

Elliot

A. J.

(2023). How many participants do I need to test an interaction? Conducting an appropriate power analysis and achieving sufficient power to detect an interaction. Advances in Methods and Practices in Psychological Science, 6(3), 25152459231178728. https://doi.org/10.1177/25152459231178728

91.

Stanley

T. D.

Carter

E. C.

Doucouliagos

(2018). What meta-analyses reveal about the replicability of psychological research. Psychological Bulletin, 144(12), 1325–1346. https://doi.org/10.1037/bul0000169

92.

Stefan

A. M.

Schönbrodt

(2022). Big little lies: A compendium and simulation of p-hacking strategies. PsyArXiv. https://doi.org/10.31234/osf.io/xy2dk

93.

Stone-Romero

E. F.

Anderson

L. E.

(1994). Relative power of moderated multiple regression and the comparison of subgroup correlation coefficients for detecting moderating effects. Journal of Applied Psychology, 79(3), 354–359. https://doi.org/10.1037/0021-9010.79.3.354

94.

Sun

Pan

(2020). Not published is not perished: Addressing publication bias in meta-analytic studies in communication. Human Communication Research, 46(2–3), 300–321. https://doi.org/10.1093/hcr/hqz015

95.

Tressoldi

P. E.

Giofré

(2015). The pervasive avoidance of prospective statistical power: Major consequences and practical solutions. Frontiers in Psychology, 6, 726. https://doi.org/10.3389/fpsyg.2015.00726

96.

Trochim

W. M.

Donnelly

J. P.

Arora

(2016). Research methods: The essential knowledge base (2nd ed.). Cengage Learning.

97.

Uschner

Schindler

Hilgers

R.-D.

Heussen

(2018). randomizeR: An R package for the assessment and implementation of randomization in clinical trials. Journal of Statistical Software, 85(8), 1–22. https://doi.org/10.18637/jss.v085.i08

98.

Wallace

S. S.

Barak

Truong

Parker

M. W.

(2022). Hierarchy of evidence within the medical literature. Hospital pediatrics, 12(8), 745–750. https://doi.org/10.1542/hpeds.2022-006690

99.

Wasserstein

R. L.

Schirm

A. L.

Lazar

N. A.

(2019). Moving to a world beyond “p < 0.05”. American Statistician, 73(sup1), 1–19. https://doi.org/10.1080/00031305.2019.1583913

100.

Weber

Popova

(2012). Testing equivalence in communication research: Theory and application. Communication Methods and Measures, 6(3), 190–213. https://doi.org/10.1080/19312458.2012.703834

101.

Yang

Sanchez-Tojar

O’Dea

R. E.

Noble

Koricheva

Jennions

M. D.

Parker

T. H.

Lagisz

Nakagawa

(2022). Publication bias impacts on effect size, statistical power, and magnitude (Type M) and sign (Type S) errors in ecology and evolutionary biology. EcoEvoRxiv. https://doi.org/10.32942/osf.io/97nv6

102.

Yuan

K.-H.

Maxwell

(2005). On the post hoc power in testing mean differences. Journal of Educational and Behavioral Statistics, 30(2), 141–167. https://doi.org/10.3102/10769986030002141

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.07 MB

Toward a More Powerful Experimental Communication Science: An Assessment of Two Decades’ Research (2001–2023)

Abstract

Keywords

Get full access to this article

References

Supplementary Material