A Powerful Nudge? Presenting Calculable Consequences of Underpowered Research Shifts Incentives Toward Adequately Powered Designs

Abstract

If psychologists have recognized the pitfalls of underpowered research for decades, why does it persist? Incentives, perhaps: underpowered research benefits researchers individually (increased productivity), but harms science collectively (inflated Type I error rates and effect size estimates but low replication rates). Yet, researchers can selectively reward power at various scientific bottlenecks (e.g., peer review, hiring, funding, and promotion). We designed a stylized thought experiment to evaluate the degree to which researchers consider power and productivity in hiring decisions. Accomplished psychologists chose between a low sample size candidate and a high sample size candidate who were otherwise identical. We manipulated the degree to which participants received information about (1) productivity, (2) sample size, and (3) directly calculable Type I error and replication rates. Participants were intolerant of the negative consequences of low-power research, yet merely indifferent regarding the practices that logically produce those consequences, unless those consequences were made quite explicit.