Promises and Perils of Experimentation: The Mutual-Internal-Validity Problem

Abstract

Researchers run experiments to test theories, search for and document phenomena, develop theories, or advise policymakers. When testing theories, experiments must be internally valid but do not have to be externally valid. However, when experiments are used to search for and document phenomena, develop theories, or advise policymakers, external validity matters. Conflating these goals and failing to recognize their tensions with validity concerns can lead to problems with theorizing. Psychological scientists should be aware of the mutual-internal-validity problem, long recognized by experimental economists. When phenomena elicited by experiments are used to develop theories that, in turn, influence the design of theory-testing experiments, experiments and theories can become wedded to each other and lose touch with reality. They capture and explain phenomena within but not beyond the laboratory. We highlight how triangulation can address validity problems by helping experiments and theories make contact with ideas from other disciplines and the real world.

Keywords

theory experimentation triangulation validity mutual internal validity

Get full access to this article

View all access options for this article.

References

Anderson

C. A.

Lindsay

J. J.

Bushman

B. J.

(1999). Research in the psychological laboratory: Truth or triviality. Current Directions in Psychological Science, 8(1), 3–9. https://doi.org/10.1111/1467-8721.00002

Andersson

F. N.

Holm

(2002). Experimental economics: Financial markets, auctions, and decision making: Interviews and contributions from the 20th Arne Ryde Symposium. Springer Science & Business Media.

Apps

M. A.

Rushworth

M. F.

Chang

S. W.

(2016). The anterior cingulate gyrus and social cognition: Tracking the motivation of others. Neuron, 90(4), 692–707. https://doi.org/10.1016/j.neuron.2016.04.018

Awad

Dsouza

Kim

Schulz

Henrich

Shariff

Bonnefon

J. F.

Rahwan

(2018). The Moral Machine experiment. Nature, 563(7729), 59–64. https://doi.org/10.1038/s41586-018-0637-6

Baker

(2016). 1,500 scientists lift the lid on reproducibility. https://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970

Bardsley

(2005). Experimental economics and the artificiality of alteration. Journal of Economic Methodology, 12(2), 239–251. https://doi.org/10.1080/13501780500086115

Bassett

D. S.

Gazzaniga

M. S.

(2011). Understanding complexity in the human brain. Trends in Cognitive Sciences, 15(5), 200–209. https://doi.org/10.1016/j.tics.2011.03.006

Berridge

K. C.

(2003). Pleasures of the brain. Brain and Cognition, 52(1), 106–128. https://doi.org/10.1016/s0278-2626(03)00014-9

Berridge

K. C.

(2007). The debate over dopamine’s role in reward: The case for incentive salience. Psychopharmacology, 191(3), 391–431. https://doi.org/10.1007/s00213-006-0578-x

10.

Bonnefon

J. F.

Shariff

Rahwan

(2016). The social dilemma of autonomous vehicles. Science, 352(6293), 1573–1576. https://doi.org/10.1126/science.aaf2654

11.

Breiman

(2001). Statistical modeling: The two cultures (with comments and a rejoinder by the author). Statistical Science, 16(3), 199–231. https://doi.org/10.1214/ss/1009213726

12.

Brock

(2019). It’s not a replication crisis. It’s an innovation opportunity. https://www.natureindex.com/news-blog/not-a-replication-crisis-innovation-opportunity

13.

Bromham

Dinnage

Hua

(2016). Interdisciplinary research has consistently lower funding success. Nature, 534(7609), 684–687. https://doi.org/10.1038/nature18315

14.

Camerer

C. F.

Dreber

Holzmeister

T. H.

Huber

Johannesson

Kirchler

Nave

Nosek

B. A.

Pfeiffer

Altmejd

Buttrick

Chan

Chen

Forsell

Gampa

Heikensten

Hummer

Imai

. . . Wu

(2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2(9), 637–644. https://doi.org/10.1038/s41562-018-0399-z

15.

Campbell

D. T.

(1957). Factors relevant to the validity of experiments in social settings. Psychological Bulletin, 54(4), 297–312. https://doi.org/10.1037/h0040950

16.

Campbell

D. T.

Fiske

D. W.

(1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81–105. https://doi.org/10.1037/h0046016

17.

Campbell

D. T.

Stanley

J. C.

(1963). Experimental and quasi-experimental designs for research. Houghton Mifflin Company.

18.

Cavanagh

J. F.

Zambrano-Vazquez

Allen

J. J.

(2012). Theta lingua franca: A common mid-frontal substrate for action monitoring processes. Psychophysiology, 49(2), 220–238. https://doi.org/10.1111/j.1469-8986.2011.01293.x

19.

Chen

Krajbich

(2018). Biased sequential sampling underlies the effects of time pressure and delay in social decision making. Nature Communications, 9, Article 3557. https://doi.org/10.1038/s41467-018-05994-9

20.

Dabney

Kurth -Nelson

Uchida

Starkweather

C. K.

Hassabis

Munos

Botvinick

(2020). A distributional code for value in dopamine-based reinforcement learning. Nature, 577(7792), 671–675. https://doi.org/10.1038/s41586-019-1924-6

21.

Decker

Finn

Duncan

(2020). Errors lead to transient impairments in memory formation. Cognition, 204, Article 104338. https://doi.org/10.1016/j.cognition.2020.104338

22.

Decker

J. H.

Otto

A. R.

Daw

N. D.

Hartley

C. A.

(2016). From creatures of habit to goal-directed learners: Tracking the developmental emergence of model-based reinforcement learning. Psychological Science, 27(6), 848–858. https://doi.org/10.1177/0956797616639301

23.

De Martino

Kumaran

Seymour

Dolan

R. J.

(2006). Frames, biases, and rational decision-making in the human brain. Science, 313(5787), 684–687. https://doi.org/10.1126/science.1127205

24.

Denzin

N. K.

(1978). The research act: A theoretical introduction to sociological methods and practice. SAGE.

25.

Diederich

Trueblood

J. S.

(2018). A dynamic dual process model of risky decision making. Psychological Review, 125(2), 270–292. https://doi.org/10.1037/rev0000087

26.

Dolan

R. J.

Dayan

(2013). Goals and habits in the brain. Neuron, 80(2), 312–325. https://doi.org/10.1016/j.neuron.2013.09.007

27.

Eshel

Tian

Bukwich

Uchida

(2016). Dopamine neurons share common response function for reward prediction error. Nature Neuroscience, 19(3), 479–486. https://doi.org/10.1038/nn.4239

28.

Evans

J. S.

(2008). Dual-processing accounts of reasoning, judgment, and social cognition. Annual Review of Psychology, 59, 255–278. https://doi.org/10.1146/annurev.psych.59.103006.093629

29.

Evans

N. J.

(2020). Think fast! The implications of emphasizing urgency in decision-making. https://doi.org/10.31234/osf.io/pfrb4

30.

Falkenstein

Hoormann

Christ

Hohnsbein

(2000). ERP components on reaction errors and their functional significance: A tutorial. Biological Psychology, 51(2), 87–107.

31.

Feynman

R. P.

Leighton

R. B.

Sands

(1963). The Feynman lectures on physics. Addison-Wesley Publishing Company.

32.

Fiorillo

C. D.

Tobler

P. N.

Schultz

(2003). Discrete coding of reward probability and uncertainty by dopamine neurons. Science, 299(5614), 1898–1902. https://doi.org/10.1126/science.1077349

33.

Fiske

S. T.

(2001). Social psychology, theories of. In Smelser

N. J.

Baltes

P. B.

(Eds.), International encyclopedia of the social and behavioral sciences (pp. 14413–14421). Pergamon.

34.

Frömer

Lin

Dean Wolf

C. K.

Inzlicht

Shenhav

(2020). When effort matters: Expectations of reward and efficacy guide cognitive control allocation. bioRxiv. https://doi.org/10.1101/2020.05.14.095935

35.

Gleick

(2008). Chaos: Making a new science. Penguin Books.

36.

Gluth

Kern

Kortmann

Vitali

C. L.

(2020). Value-based attention but not divisive normalization influences decisions with multiple alternatives. Nature Human Behaviour, 4(6), 634–645. https://doi.org/10.1038/s41562-020-0822-0

37.

Goldman

(2020). Exaggerated risk of transmission of COVID-19 by fomites. Lancet Infectious Diseases, 20(8), 892–893. https://doi.org/10.1016/S1473-3099(20)30561-2

38.

Grahek

Musslick

Shenhav

(2020). A computational perspective on the roles of affect in cognitive control. International Journal of Psychophysiology, 151, 25–34. https://doi.org/10.1016/j.ijpsycho.2020.02.001

39.

Greenwald

A. G.

McGhee

D. E.

Schwartz

J. L. K.

(1998). Measuring individual differences in implicit cognition: The implicit association test. Journal of Personality and Social Psychology, 74(6), 1464–1480. https://doi.org/10.1037/0022-3514.74.6.1464

40.

Greenwood

J. D.

(1982). On the relation between laboratory experiments and social behaviour: Causal explanation and generalization. Journal for the Theory of Social Behaviour, 12(3), 225–250. https://doi.org/10.1111/j.1468-5914.1982.tb00449.x

41.

Guala

(2003). Experimental localism and external validity. Philosophy of Science, 70(5), 1195–1205. https://doi.org/10.1086/377400

42.

Guala

Mittone

(2005). Experiments in economics: External validity and the robustness of phenomena. Journal of Economic Methodology, 12(4), 495–515. https://doi.org/10.1080/13501780500342906

43.

Henrich

Heine

S. J.

Norenzayan

(2010). The weirdest people in the world. Behavioral and Brain Sciences, 33(2–3), 61–83. https://doi.org/10.1017/S0140525X0999152X

44.

Hertwig

Ortmann

(2001). Experimental practices in economics: A methodological challenge for psychologists. Behavioral and Brain Sciences, 24(3), 383–403. https://doi.org/10.1037/e683322011-032

45.

Hutcherson

C. A.

Bushong

Rangel

(2015). A neurocomputational model of altruistic choice and its implications. Neuron, 87(2), 451–462. https://doi.org/10.1016/j.neuron.2015.06.031

46.

Irreproducibility is not a sign of failure, but an inspiration for fresh ideas. (2020). Nature, 578(7794), 191–192. https://doi.org/10.1038/d41586-020-00380-2

47.

Jordan

M. I.

Mitchell

T. M.

(2015). Machine learning: Trends, perspectives, and prospects. Science, 349(6245), 255–260. https://doi.org/10.1126/science.aaa8415

48.

Kahneman

(2011). Thinking, fast and slow. Macmillan.

49.

Khaw

M. W.

Glimcher

P. W.

Louie

(2017). Normalized value coding explains dynamic adaptation in the human valuation process. Proceedings of the National Academy of Sciences, USA, 114(48), 12696–12701. https://doi.org/10.1073/pnas.1715293114

50.

Kruglanski

A. W.

(2001). That “vision thing”: The state of theory in social and personality psychology at the edge of the new millennium. Journal of Personality and Social Psychology, 80(6), 871–875. https://doi.org/10.1037/0022-3514.80.6.871

51.

Krugman

(2014, September 28). The dismal science. The New York Times. https://www.nytimes.com/2014/09/28/books/review/seven-bad-ideas-by-jeff-madrick.html?_r=0

52.

Kuhn

T. S.

(2012). The structure of scientific revolutions. University of Chicago Press.

53.

Kumar

Gao

Pirogova

Fang

(2019). A review of error-related potential-based brain–computer interfaces for motor impaired people. IEEE Access, 7, 142451–142466. https://doi.org/10.1109/access.2019.2944067

54.

Lawlor

D. A.

Tilling

Davey Smith

(2016). Triangulation in aetiological epidemiology. International Journal of Epidemiology, 45(6), 1866–1886. https://doi.org/10.1093/ije/dyw314

55.

Lieberman

M. D.

(2007). Social cognitive neuroscience: A review of core processes. Annual Review of Psychology, 58, 259–289. https://doi.org/10.1146/annurev.psych.58.110405.085654

56.

Lin

Saunders

Hutcherson

C. A.

Inzlicht

(2018). Midfrontal theta and pupil dilation parametrically track subjective conflict (but also surprise) during intertemporal choice. NeuroImage, 172, 838–852. https://doi.org/10.1016/j.neuroimage.2017.10.055

57.

Lin

Vartanian

(2018). A neuroeconomic framework for creative cognition. Perspectives on Psychological Science, 13(6), 655–677. https://doi.org/10.1177/1745691618794945

58.

Lipton

(2004). Inference to the best explanation. Psychology Press.

59.

List

J. A.

Levitt

S. D.

(2006). What do laboratory experiments tell us about the real world? https://pricetheory.uchicago.edu/levitt/Papers/jep%20revision%20Levitt%20&%20List.pdf

60.

Loewenstein

(1999). Experimental economics from the vantage-point of behavioural economics. The Economic Journal, 109(453), 25–34. https://doi.org/10.1111/1468-0297.00400

61.

Loewenstein

O’Donoghue

Bhatia

(2015). Modeling the interplay between affect and deliberation. Decision, 2(2), 55–81. https://doi.org/10.1037/dec0000029

62.

Manzi

(2012). Uncontrolled: The surprising payoff of trial-and-error for business, politics, and society. Basic Books.

63.

Mathison

(1988). Why triangulate? Educational Researcher, 17(2), 13–17. https://doi.org/10.3102/0013189X017002013

64.

McClure

S. M.

Laibson

D. I.

Loewenstein

Cohen

J. D.

(2004). Separate neural systems value immediate and delayed monetary rewards. Science, 306(5695), 503–507. https://doi.org/10.1126/science.1094492

65.

Meehl

P. E.

(1967). Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 34(2), 103–115.

66.

Meehl

P. E.

(1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46(4), 806–834. https://doi.org/10.1037/0022-006X.46.4.806

67.

Melnikoff

D. E.

Bargh

J. A.

(2018). The mythical number two. Trends in Cognitive Sciences, 22(4), 280–293. https://doi.org/10.1016/j.tics.2018.02.001

68.

Montague

P. R.

Dayan

Sejnowski

T. J.

(1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning. The Journal of Neuroscience, 16(5), 1936–1947. https://doi.org/10.1523/JNEUROSCI.16-05-01936.1996

69.

Mook

D. G.

(1983). In defense of external invalidity. American Psychologist, 38(4), 379–387. https://doi.org/10.1037/0003-066X.38.4.379

70.

Moshontz

Campbell

Ebersole

C. R.

IJzerman

Urry

H. L.

Forscher

P. S.

Grahe

J. E.

McCarthy

R. J.

Musser

E. D.

Antfolk

(2018). The Psychological Science Accelerator: Advancing psychology through a distributed collaborative network. Advances in Methods and Practices in Psychological Science, 1(4), 501–515. https://doi.org/10.1177/2515245918797607

71.

Mosleh

Kyker

Cohen

J. D.

Rand

D. G.

(2020). Globalization and the rise and fall of cognitive control. Nature Communications, 11(1), Article 3099. https://doi.org/10.1038/s41467-020-16850-0

72.

Mosleh

Pennycook

Arechar

A. A.

Rand

D. G.

(2020). Twitter data reveal digital fingerprints of cognitive reflection. PsyArXiv. https://doi.org/10.31234/osf.io/qaswn

73.

Munafò

M. R.

Smith

G. D.

(2018). Robust research needs many lines of evidence. Nature, 553, 399-401. https://doi.org/10.1038/d41586-018-01023-3

74.

Muthukrishna

Henrich

(2019). A problem in theory. Nature Human Behaviour, 3, 221–229. https://doi.org/10.1038/s41562-018-0522-1

75.

Niv

(2009). Reinforcement learning in the brain. Journal of Mathematical Psychology, 53(3), 139–154. https://doi.org/10.1016/j.jmp.2008.12.005

76.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), Article aac4716. https://doi.org/10.1126/science.aac4716

77.

Otto

A. R.

Fleming

S. M.

Glimcher

P. W.

(2016). Unexpected but incidental positive outcomes predict real-world gambling. Psychological Science, 27(3), 299–311. https://doi.org/10.1177/0956797615618366

78.

Padoa-Schioppa

(2011). Neurobiology of economic choice: A good-based model. Annual Review of Psychology, 34, 333–359. https://doi.org/10.1146/annurev-neuro-061010-113648

79.

Pearl

(2015). Generalizing experimental findings. Journal of Causal Inference, 3(2). https://doi.org/10.1515/jci-2015-0025

80.

Pearl

Mackenzie

(2018). The book of why. Penguin.

81.

Pearson

A. R.

Schuldt

J. P.

Romero-Canyas

(2016). Social climate science: A new vista for psychological science. Perspectives on Psychological Science, 11(5), 632–650. https://doi.org/10.1177/1745691616639726

82.

Pearson

A. R.

Schuldt

J. P.

Romero- Canyas

Ballew

M. T.

Larson- Konar

(2018). Diverse segments of the US public underestimate the environmental concerns of minority and low-income Americans. Proceedings of the National Academy of Sciences, USA, 115(49), 12429–12434. https://doi.org/10.1073/pnas.1804698115

83.

Pearson

J. M.

Watson

K. K.

Platt

M. L.

(2014). Decision making: The neuroethological turn. Neuron, 82(5), 950–965. https://doi.org/10.1016/j.neuron.2014.04.037

84.

Pennycook

Neys

W. D.

Evans

Stanovich

K. E.

Thompson

V. A.

(2018). The mythical dual-process typology. Trends in Cognitive Sciences, 22(8), 667–668. https://doi.org/10.1016/j.tics.2018.04.008

85.

Pessiglione

Seymour

Flandin

Dolan

R. J.

Frith

C. D.

(2006). Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature, 442(7106), 1042–1045. https://doi.org/10.1038/nature05051

86.

Plott

C. R.

(1991). Will economics become an experimental science. Southern Economic Journal, 57(4), 901–919. https://doi.org/10.2307/1060322

87.

Polanía

Woodford

Ruff

C. C.

(2019). Efficient coding of subjective value. Nature Neuroscience, 22(1), 134–142. https://doi.org/10.1038/s41593-018-0292-0

88.

Rahwan

Cebrian

Obradovich

Bongard

Bonnefon

J. F.

Breazeal

Crandall

J. W.

Christakis

N. A.

Couzin

I. D.

Jackson

M. O.

Jennings

N. R.

Kamar

Kloumann

I. M.

Larochelle

Lazer

McElreath

Mislove

Parkes

D. C.

Pentland

. . . Wellman

(2019). Machine behaviour. Nature, 568(7753), 477–486. https://doi.org/10.1038/s41586-019-1138-y

89.

Rand

D. G.

Greene

J. D.

Nowak

M. A.

(2012). Spontaneous giving and calculated greed. Nature, 489(7416), 427–430. https://doi.org/10.1038/nature11467

90.

Redgrave

Prescott

T. J.

Gurney

(1999). Is the short-latency dopamine response too short to signal reward error. Trends in Neurosciences, 22(4), 146–151. https://doi.org/10.1016/S0166-2236(98)01373-3

91.

Ribe

Steinle

(2002). Exploratory experimentation: Goethe, Land, and color theory. Physics Today, 55(7), 43–49.

92.

Ross

Lepper

Ward

(2010). History of social psychology: Insights, challenges, and contributions to theory and application. (5th ed.) Wiley. https://doi.org/10.1002/9780470561119.socpsy001001

93.

Roth

A. E.

(1986). Laboratory experimentation in economics. Economics and Philosophy, 2, 245–273. https://doi.org/10.1017/s1478061500002656

94.

Ruggeri

Alí

Berge

M. L.

Bertoldo

Bjørndal

L. D.

Cortijos-Bernabeu

Davison

Demić

Esteban-Serna

Friedemann

Gibson

S. P.

Jarke

Karakasheva

Khorrami

P. R.

Kveder

Andersen

T. L.

Lofthus

I. S.

McGill

Nieto

A. E.

. . . Folke

(2020). Replicating patterns of prospect theory for decision under risk. Nature Human Behaviour, 4(6), 622–633. https://doi.org/10.1038/s41562-020-0886-x

95.

Schimmack

(2019). The implicit association test: A method in search of a construct. Perspectives on Psychological Science, 28(4), 1–19. https://doi.org/10.1080/1047840x.2017.1335568

96.

Schram

(2005). Artificiality: The tension between internal and external validity in economic experiments. Journal of Economic Methodology, 12(2), 225–237. https://doi.org/10.1080/13501780500086081

97.

Schultz

Dayan

Montague

P. R.

(1997). A neural substrate of prediction and reward. Science, 275(5306), 1593–1599. https://doi.org/10.1126/science.275.5306.1593

98.

Schulz

J. F.

Barahmi-Rad

Beauchamp

Henrich

(2018). The origins of WEIRD psychology. https://doi.org/10.31234/osf.io/d6qhu

99.

Settle

T. B.

(1961). An experiment in the history of science. Science, 133(3445), 19–23. https://doi.org/10.2307/1706347

100.

Shenhav

Musslick

Botvinick

M. M.

Cohen

J. D.

(2020). Misdirected vigor: Differentiating the control of value from the value of control. https://doi.org/10.31234/osf.io/5bhwe

101.

Silver

Schrittwieser

Simonyan

Antonoglou

Huang

Guez

Hubert

Baker

Lai

Bolton

Chen

Lillicrap

Hui

Sifre

van den Driessche

Graepel

Hassabis

(2017). Mastering the game of Go without human knowledge. Nature, 550(7676), 354–359. https://doi.org/10.1038/nature24270

102.

Sloane

Moss

(2019). AI’s social sciences deficit. Nature Machine Intelligence, 1(8), 330–331. https://doi.org/10.1038/s42256-019-0084-6

103.

Smith

V. L.

(1989). Theory, experiment and economics. Journal of Economic Perspectives, 3(1), 151–169. https://pubs.aeaweb.org/doi/pdfplus/10.1257/jep.3.1.151

104.

Sugden

(2005). Experiments as exhibits and experiments as tests. Journal of Economic Methodology, 12(2), 291–302. https://doi.org/10.1080/13501780500086248

105.

Sykes

(Producer). (1981, November 23). Richard Feynman—the pleasure of finding things out [TV series episode]. In Horizon. BBC Two. https://www.dailymotion.com/video/x6ptg1x

106.

Teoh

Y. Y.

Yao

Cunningham

W. A.

Hutcherson

C. A.

(2020). Attentional priorities drive effects of time pressure on altruistic choice. Nature Communications, 11(1), Article 3534. https://doi.org/10.1038/s41467-020-17326-x

107.

Tinbergen

(1963). On aims and methods of ethology. Zeitschrift für Tierpsychologie, 20(4), 410–433.

108.

Turner

B. O.

Paul

E. J.

Miller

M. B.

Barbey

A. K.

(2018). Small sample sizes reduce the replicability of task-based fMRI studies. Communications Biology, 1, Article 62. https://doi.org/10.1038/s42003-018-0073-z

109.

Tversky

Kahneman

(1981). The framing of decisions and the psychology of choice. Science, 211(4481), 453–458. https://doi.org/10.1126/science.7455683

110.

Van Lange

P. A. M.

(2006). Bridging social psychology: Benefits of transdisciplinary approaches. Psychology Press. https://doi.org/10.4324/9781410616982

111.

van Rooij

Baggio

(2021). Theory before the test: How to build high-verisimilitude explanatory theories in psychological science. Perspectives on Psychological Science, 16(4), 682–697. https://doi.org/10.1177/1745691620970604

112.

Vassena

Deraeve

Alexander

W. H.

(2020). Surprise, value and control in anterior cingulate cortex during speeded decision-making. Nature Human Behaviour, 4, 412–422. https://doi.org/10.1038/s41562-019-0801-5

113.

Villano

W. J.

Otto

A. R.

Ezie

C. E. C.

Gillis

Heller

A. S.

(2020). Temporal dynamics of real-world emotion are more strongly linked to prediction error than outcome. Journal of Experimental Psychology: General, 149(9), 1755–1766. https://doi.org/10.1037/xge0000740

114.

von Hayek

(1974). Lecture to the memory of Alfred Nobel: The pretence of knowledge. https://www.nobelprize.org/prizes/economic-sciences/1974/hayek/lecture/

115.

Vosoughi

Roy

Aral

(2018). The spread of true and false news online. Science, 359(6380), 1146–1151. https://doi.org/10.1126/science.aap9559

116.

Wang

S. Y.

Inbar

(2021). Moral-language use by U.S. political elites. Psychological Science, 32(1), 14–26. https://doi.org/10.1177/0956797620960397

117.

Webb

E. J.

Campbell

D. T.

Schwartz

R. D.

(1966). Unobtrusive measures. Rand McNally.

118.

Westbrook

Braver

T. S.

(2015). Cognitive effort: A neuroeconomic approach. Cognitive, Affective, & Behavioral Neuroscience, 15(2), 395–415. https://doi.org/10.3758/s13415-015-0334-y

119.

Wheeler

N. E.

Allidina

Long

E. U.

Schneider

S. P.

Haas

I. J.

Cunningham

W. A.

(2020). Ideology and predictive processing: Coordination, bias, and polarization in socially constrained error minimization. Current Opinion in Behavioral Sciences, 34, 192–198. https://doi.org/10.1016/j.cobeha.2020.05.002

120.

Wise

R. A.

(1996). Addictive drugs and brain stimulation reward. Annual Review of Neuroscience, 19(1), 319–340. https://doi.org/10.1146/annurev.ne.19.030196.001535

121.

Yarkoni

(2019). The generalizability crisis. PsyArXiv. https://doi.org/10.31234/osf.io/jqw35

122.

Yarkoni

Westfall

(2017). Choosing prediction over explanation in psychology: Lessons from machine learning. Perspectives on Psychological Science, 12(6), 1100–1122. https://doi.org/10.1177/1745691617693393