What factors affect a scientist’s choice of research problem? Qualitative research in the history and sociology of science suggests that this choice is patterned by an “essential tension” between productive tradition and risky innovation. We examine this tension through Bourdieu’s field theory of science, and we explore it empirically by analyzing millions of biomedical abstracts from MEDLINE. We represent the evolving state of chemical knowledge with networks extracted from these abstracts. We then develop a typology of research strategies on these networks. Scientists can introduce novel chemicals and chemical relationships (innovation) or delve deeper into known ones (tradition). They can consolidate knowledge clusters or bridge them. The aggregate distribution of published strategies remains remarkably stable. High-risk innovation strategies are rare and reflect a growing focus on established knowledge. An innovative publication is more likely to achieve high impact than a conservative one, but the additional reward does not compensate for the risk of failing to publish. By studying prizewinners in biomedicine and chemistry, we show that occasional gambles for extraordinary impact are a compelling explanation for observed levels of risky innovation. Our analysis of the essential tension identifies institutional forces that sustain tradition and suggests policy interventions to foster innovation.
AbbottAndrew D.1999. Department & Discipline: Chicago Sociology at One Hundred. Chicago: University of Chicago Press.
2.
AbbottAndrew D.2001. Chaos of Disciplines. Chicago: University of Chicago Press.
3.
AlbertMathieuKleinmanDaniel Lee. 2011. “Bringing Pierre Bourdieu to Science and Technology Studies.” Minerva49(3):263–73.
4.
AusterweilJoseph L.AbbottJoshua T.GriffithsThomas L.2012. “Human Memory Search as a Random Walk in a Semantic Network.” Pp. 3041–49 in Advances in Neural Information Processing Systems 25, edited by PereiraF.BurgesC. J. C.BottouL.WeinbergerK. Q.Curran Associates, Inc.
5.
AzoulayPierreDingWaverlyStuartToby. 2009. “The Impact of Academic Patenting on the Rate, Quality and Direction of (Public) Research Output.” Journal of Industrial Economics57(4):637–76.
6.
AzoulayPierreGraff ZivinJoshua S.MansoGustavo. 2011. “Incentives and Creativity: Evidence from the Academic Life Sciences.” RAND Journal of Economics42(3):527–54.
7.
AzoulayPierreStuartTobyWangYanbo. 2013. “Matthew: Effect or Fable?” Management Science60(1):92–109.
8.
BaldiStéphane. 1998. “Normative versus Social Constructivist Processes in the Allocation of Citations: A Network-Analytic Model.” American Sociological Review63(6):829–46.
9.
BarabásiAlbert-LászlóAlbertRéka. 1999. “Emergence of Scaling in Random Networks.” Science286(5439):509–512.
10.
BatemanThomas S.HessAndrew M.2015. “Different Personal Propensities among Scientists Relate to Deeper vs. Broader Knowledge Contributions.” Proceedings of the National Academy of Sciences112(12):3653–58.
11.
BermanElizabeth Popp. 2008. “Why Did Universities Start Patenting? Institution-Building and the Road to the Bayh-Dole Act.” Social Studies of Science38(6):835–71.
12.
BermanElizabeth Popp. 2012. “Explaining the Move toward the Market in US Academic Science: How Institutional Logics Can Change without Institutional Entrepreneurs.” Theory and Society41(3):261–99.
13.
BourdieuPierre. 1975. “The Specificity of the Scientific Field and the Social Conditions for the Progress of Reason.” Social Science Information14(6):19–47.
14.
BourdieuPierre. 1984. Distinction: A Social Critique of the Judgement of Taste. New York: Routledge.
15.
BourdieuPierre. 1986. “The Forms of Capital.” Pp. 241–58 in Handbook of Theory and Research for the Sociology of Education, edited by RichardsonJ.New York: Greenwood.
16.
BourdieuPierre. 1990. The Logic of Practice. Stanford, CA: Stanford University Press.
17.
BourdieuPierre. 1991. “The Peculiar History of Scientific Reason.” Sociological Forum6(1):3–26.
18.
BourdieuPierre. 2004. Science of Science and Reflexivity. Chicago: University of Chicago Press.
19.
BruggemanJeroenTraagV. A.UitermarkJustus. 2012. “Detecting Communities through Network Data.” American Sociological Review77(6):1050–63.
20.
BuschLawrenceLacyWilliam B.SachsCarolyn. 1983. “Perceived Criteria for Research Problem Choice in the Agricultural Sciences: A Research Note.” Social Forces62(1):190–200.
21.
CallonMichelLawJohnRipArie. 1986. Mapping the Dynamics of Science and Technology: Sociology of Science in the Real World. Basingstoke, UK: Macmillan.
22.
CamicCharles. 2011. “Bourdieu’s Cleft Sociology of Science.” Minerva49(3):275–93.
23.
CamicCharles. 2013. “Bourdieu’s Two Sociologies of Knowledge.” Pp. 183–214 in Bourdieu and Historical Analysis, edited by GorskiP. S.Durham, NC: Duke University Press.
24.
ChenChaomeiChenYueHorowitzMarkHouHaiyanLiuZeyuanPellegrinoDonald. 2009. “Towards an Explanatory and Computational Theory of Scientific Discovery.” Journal of Informetrics3(3):191–209.
25.
CokolMuratIossifovIvanWeinrebChaniRzhetskyAndrey. 2005. “Emergent Behavior of Growing Knowledge about Molecular Interactions.” Nature Biotechnology23(10):1243–47.
26.
CokolMuratRodriguez-EstebanRaulRzhetskyAndrey. 2007. “A Recipe for High Impact.” Genome Biology8(5):406.
27.
CoverThomas M.ThomasJoy A.1991. Elements of Information Theory. New York: Wiley.
28.
CowenTyler. 2011. The Great Stagnation: How America Ate All The Low-Hanging Fruit of Modern History, Got Sick, and Will (Eventually) Feel Better. New York: Dutton Adult.
29.
CroninBlaiseAtkinsHelen Barsky, eds. 2000. The Web of Knowledge: A Festschrift in Honor of Eugene Garfield. Medford, NJ: Information Today.
30.
DavisLeeLarsenMaria TheresaLotzPeter. 2011. “Scientists’ Perspectives Concerning the Effects of University Patenting on the Conduct of Academic Research in the Life Sciences.” Journal of Technology Transfer36(1):14–37.
31.
DiamondArthur M.Jr.1994. “The Determinants of a Scientist’s Choice of Research Projects.” Pp. 167–210 in Scientific Failure, edited by HorowitzT.JanisA. I.Lanham, MD: Rowman & Littlefield.
32.
DingWaverly W.MurrayFionaStuartToby E.2006. “Gender Differences in Patenting in the Academic Life Sciences.” Science313(5787):665–67.
33.
EvansJames A.2010a. “Industry Collaboration, Scientific Sharing and the Dissemination of Knowledge.” Social Studies of Science40(5):757–91.
34.
EvansJames A.2010b. “Industry Induces Academic Science to Know Less about More.” American Journal of Sociology116(2):389 –452.
EvansJames A.RzhetskyAndrey. 2011. “Advancing Science through Mining Libraries, Ontologies, and Communities.” Journal of Biological Chemistry286(27):23659–66.
37.
FlemingLee. 2001. “Recombinant Uncertainty in Technological Search.” Management Science47(1):117–32.
FlemingLeeSorensonOlav. 2004. “Science as a Map in Technological Search.” Strategic Management Journal25(8–9):909–28.
40.
FooteRichard. 2007. “Mathematics and Complex Systems.” Science318(5849):410–12.
41.
FrickelScottGrossNeil. 2005. “A General Theory of Scientific/Intellectual Movements.” American Sociological Review70(2):204–232.
42.
FunkRussell J.Owen-SmithJason. 2012. “A Dynamic Network Approach to Breakthrough Innovation.” arXiv:1212.3559 [physics], December (http://arxiv.org/abs/1212.3559).
43.
GertnerJon. 2012. The Idea Factory: Bell Labs and the Great Age of American Innovation. New York: Penguin.
44.
GibbonsMichaelLimogesCamilleNowotnyHelgaSchwartzmanSimonScottPeterTrowMartin. 1994. The New Production of Knowledge: The Dynamics of Science and Research in Contemporary Societies. Thousand Oaks, CA: Sage.
45.
GierynThomas F.1978. “Problem Retention and Problem Change in Science.” Sociological Inquiry48(3–4):96–115.
46.
GoodmanLeo A.1965. “On Simultaneous Confidence Intervals for Multinomial Proportions.” Technometrics7(2):247–54.
47.
GriffithsThomas L.SteyversMarkFirlAlana. 2007. “Google and the Mind: Predicting Fluency with PageRank.” Psychological Science18(12):1069–76.
48.
GuetzkowJoshuaLamontMichèleMallardGrégoire. 2004. “What Is Originality in the Humanities and the Social Sciences?” American Sociological Review69(2):190–212.
49.
GuimeràRogerUzziBrianSpiroJarrettNunes AmaralLuís A.2005. “Team Assembly Mechanisms Determine Collaboration Network Structure and Team Performance.” Science308(5722):697–702.
50.
HargadonAndrewSuttonRobert I.1997. “Technology Brokering and Innovation in a Product Development Firm.” Administrative Science Quarterly42(4):716–49.
51.
HausmanJerryHallBronwyn H.GrilichesZvi. 1984. “Econometric-Models for Count Data with an Application to the Patents–R&D Relationship.” Econometrica52(4):909–938.
52.
HebbDonald O.1949. The Organization of Behavior. New York: Wiley & Sons.
53.
HedströmPeter. 2005. Dissecting the Social: On the Principles of Analytical Sociology. Cambridge, UK: Cambridge University Press.
54.
IoannidisJohn P. A.2005a. “Why Most Published Research Findings Are False.” PLoS Med2(8):e124.
55.
IoannidisJohn P. A.2005b. “Contradicted and Initially Stronger Effects in Highly Cited Research.” Journal of the American Medical Association294(2):218–28.
56.
KevlesDaniel J.1978. The Physicists: The History of a Scientific Community in Modern America, 1st ed.New York: Knopf.
57.
KimKyung-Man. 2009. “What Would a Bourdieuan Sociology of Scientific Truth Look Like?” Social Science Information48(1):57–79.
58.
KleinbergJonOrenSigal. 2011. “Mechanisms for (Mis)Allocating Scientific Credit.” Pp. 529–38 in Proceedings of the Forty-Third Annual ACM Symposium on Theory of Computing. New York: ACM.
59.
Knorr-CetinaKarin. 1999. Epistemic Cultures: How the Sciences Make Knowledge. Cambridge, MA: Harvard University Press.
60.
KuhnThomas S.[1959] 1977. “The Essential Tension: Tradition and Innovation in Scientific Research.” Pp. 162–74 in The Third (1959) University of Utah Research Conference on the Identification of Scientific Talent, edited by TaylorC.W.Salt Lake City: University of Utah Press. Reprint, pp. 225–39 in The Essential Tension: Selected Studies in Scientific Tradition and Change. Chicago: University of Chicago Press.
61.
KuhnThomas S.1962. The Structure of Scientific Revolutions. Chicago: University of Chicago Press.
62.
LamontMichèle. 2009. How Professors Think: Inside the Curious World of Academic Judgment. Cambridge, MA: Harvard University Press.
LatourBruno. 1987. Science in Action: How to Follow Scientists and Engineers through Society. Cambridge, MA: Harvard University Press.
65.
LawrencePaul R.LorschJay William. 1969. Organization and Environment: Managing Differentiation and Integration. Homewood, IL: R. D. Irwin.
66.
LeaheyErinBeckmanChristineStankoTaryn. 2013. “Prominent but Less Productive: The Impact of Interdisciplinarity on Scientists’ Research.” Working paper.
67.
LeaheyErinMoodyJames. 2014. “Sociological Innovation through Subfield Integration.” Social Currents1(3):228–56.
68.
MalmgrenR. DeanOttinoJulio M.Nunes AmaralLuís A.2010. “The Role of Mentorship in Protege Performance.” Nature465(7298):622–26.
69.
MarchJames G.1991. “Exploration and Exploitation in Organizational Learning.” Organization Science2(1):71–87.
70.
MenczerFilippo. 2004. “Evolution of Document Networks.” Proceedings of the National Academy of Sciences USA 101(Suppl. 1, April):5261–65.
71.
MertonRobert K.1938. “Science, Technology and Society in Seventeenth Century England.” Osiris: Studies on the History and Philosophy of Science, and on the History of Learning and Culture4(1938):362–632.
72.
MertonRobert K.1942. “Science and Technology in a Democratic Order.” Journal of Legal and Political Sociology1(1):115–26.
73.
MertonRobert K.1949. Social Theory and Social Structure: Toward the Codification of Theory and Research. Glencoe, IL: Free Press.
74.
MertonRobert K.1957. “Priorities in Scientific Discovery: A Chapter in the Sociology of Science.” American Sociological Review22(6):635–59.
75.
MertonRobert K.1968. “The Matthew Effect in Science.” Science159(3810):56–63.
76.
MertonRobert K.1973. The Sociology of Science: Theoretical and Empirical Investigations. Chicago: University of Chicago Press.
77.
MertonRobert K.1988. “The Matthew Effect in Science, II: Cumulative Advantage and the Symbolism of Intellectual Property.” Isis79(299):606–623.
78.
MertonRobert K.BarberElinor. 2011. The Travels and Adventures of Serendipity: A Study in Sociological Semantics and the Sociology of Science. Princeton, NJ: Princeton University Press.
79.
MosesHamiltonIIIDorseyE. RayMathesonDavid H. M.ThierSamuel O.2005. “Financial Anatomy of Biomedical Research.” Journal of the American Medical Association294(11):1333–42.
80.
MoweryDavid C.NelsonRichard R.SampatBhaven N.ZiedonisArvids A.2001. “The Growth of Patenting and Licensing by U.S. Universities: An Assessment of the Effects of the Bayh-Dole Act of 1980.” Research Policy30(1):99–119.
81.
MurrayFionaSternScott. 2007. “Do Formal Intellectual Property Rights Hinder the Free Flow of Scientific Knowledge? An Empirical Test of the Anti-Commons Hypothesis.” Journal of Economic Behavior & Organization63(4):648–87.
82.
NewellAllenSimonHerbert Alexander. 1972. Human Problem Solving. Upper Saddle River, NJ: Prentice Hall.
83.
NewmanM. E. J.2003. “The Structure and Function of Complex Networks.” SIAM Review45(2):167–256.
84.
NowotnyHelgaScottPeterGibbonsMichael. 1994. The New Production of Knowledge: The Production of Science and Research in Contemporary Societies. London, UK: Sage Publications.
85.
OnnelaJ.-P.SaramäkiJ.HyvönenJ.SzabóG.LazerD.KaskiK.KertészJ.BarabásiA.-L.2007. “Structure and Tie Strengths in Mobile Communication Networks.” Proceedings of the National Academy of Sciences104(18):7332–36.
86.
PanofskyAaron L.2011. “Field Analysis and Interdisciplinary Science: Scientific Capital Exchange in Behavior Genetics.” Minerva49(3):295–316.
87.
PolanyiMichael. 1969. Knowing and Being: Essays by Michael Polanyi, edited by M. Grene. Chicago: University of Chicago Press.
88.
PowellWalter W.Owen-SmithJason. 1998. “Universities and the Market for Intellectual Property in the Life Sciences.” Journal of Policy Analysis and Management17(2):253–77.
RosenthalRobert. 1979. “The File Drawer Problem and Tolerance for Null Results.” Psychological Bulletin86(3):638–41.
91.
RossmanGabrielSchilkeOliver. 2014. “Close, But No Cigar: The Bimodal Rewards to Prize-Seeking.” American Sociological Review79(1):86–108.
92.
RosvallMartinAxelssonD.BergstromCarl T.2009. “The Map Equation.” European Physical Journal Special Topics178(1):13–23.
93.
RosvallMartinBergstromCarl T.2008. “Maps of Random Walks on Complex Networks Reveal Community Structure.” Proceedings of the National Academy of Sciences, USA105(4):1118–23.
94.
RosvallMartinBergstromCarl T.2010. “Mapping Change in Large Networks.” PLoS ONE5(1):e8695.
95.
ShapinSteven. 1995. “Here and Everywhere: Sociology of Scientific Knowledge.” Annual Review of Sociology 21(January):289–321.
96.
ShiFengFosterJacob G.EvansJames A.2015. “Weaving the Fabric of Science: Dynamic Network Models of Science’s Unfolding Structure.” Social Networks43(October):73–85.
97.
ShwedUriBearmanPeter S.2010. “The Temporal Structure of Scientific Consensus Formation.” American Sociological Review75(6):817–40.
98.
SmalheiserNeil R.TorvikVetle I.2009. “Author Name Disambiguation.” Annual Review of Information Science and Technology43(1):1–43.
99.
StuartToby E.DingWaverly W.2006. “When Do Scientists Become Entrepreneurs? The Social Structural Antecedents of Commercial Activity in the Academic Life Sciences.” American Journal of Sociology112(1):97–144.
100.
SwansonDon R.1990. “Medical Literature as a Potential Source of New Knowledge.” Bulletin of the Medical Library Association78(1):29–37.
101.
ThursbyJerry G.ThursbyMarie C.2011. “Has the Bayh-Dole Act Compromised Basic Research?” Research Policy40(8):1077–83.
102.
UzziBrianMukherjeeSatyamStringerMichaelJonesBen. 2013. “Atypical Combinations and Scientific Impact.” Science342(6157):468–72.
103.
VallasSteven PeterKleinmanDaniel Lee. 2008. “Contradiction, Convergence and the Knowledge Economy: The Confluence of Academic and Commercial Biotechnology.” Socio-Economic Review6(2):283–311.
104.
WashburnJennifer. 2005. University, Inc.: The Corporate Corruption of American Higher Education. New York: Basic Books.
105.
WeisbergMichaelMuldoonRyan. 2009. “Epistemic Landscapes and the Division of Cognitive Labor.” Philosophy of Science76(2):225–52.
106.
WhitleyRichard. 2000. The Intellectual and Social Organization of the Sciences. Oxford, UK: Oxford University Press.
107.
WoolfSteven H.2008. “The Meaning of Translational Research and Why It Matters.” Journal of the American Medical Association299(2):211–13.
108.
WrightBrian D.1983. “The Economics of Invention Incentives: Patents, Prizes, and Research Contracts.” American Economic Review73(4):691–707.
109.
WuchtyStefanJonesBenjamin F.UzziBrian. 2007. “The Increasing Dominance of Teams in Production of Knowledge.” Science316(5827):1036–39.
110.
YaoLixiaLiYingGhoshSoumitraEvansJames A.RzhetskyAndrey. Forthcoming. “Health ROI as a Measure of Misalignment of Biomedical Needs and Resources.” Nature Biotechnology.
111.
ZemlyakIlonaManleyNathanSapolskyRobertGozesIllana. 2007. “NAP Protects Hippocampal Neurons against Multiple Toxins.” Peptides28(10):2004–8.
112.
ZuckermanHarriet. 1978. “Theory Choice and Problem Choice in Science.” Sociological Inquiry48(3–4):65–95.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
