Abstract
I develop and test a theory that explains why organizations may struggle to adapt in the face of change even when their members are aware of change, are motivated to adapt, and have the resources to do so. I build on complex-systems theory, which posits that organizations face a hierarchy of interdependent problems: they must choose how to fulfill different specialized tasks and choose processes to integrate the outputs of these tasks. Because these choices are interdependent, environmental change that directly affects only a few tasks in isolation can indirectly affect the viability of major organizational processes. Recognizing these ripple effects is difficult, however: understanding complex interdependencies is challenging for decision makers, and the division of labor within organizations can create an illusion of separability between tasks. As a result, organizations may respond to such change by engaging in “modular search” for new ways to fulfill specialized tasks, but they may fail to engage in “systemic search” for new processes integrating the outputs of specialized tasks unless they can rely on information-processing structures that help decision makers better understand interdependencies among choices. I test my theory by applying sequence analysis methods to micro-level behavioral data on competitive video gaming (esports) teams. Qualitative fieldwork and an online experiment provide additional evidence of my proposed mechanisms.
History is replete with organizations that failed to adapt to environmental change (Romanelli and Tushman, 1994; Lavie, Stettner, and Tushman, 2010; Posen and Levinthal, 2012; Eggers and Park, 2018). Organizational researchers have sought to explain why and when organizations struggle to adapt, by exposing various mechanisms that can create rigidity in their behavior. An organization’s decision makers may not perceive exogenous change as relevant to its activity (Christensen and Bower, 1996; Tripsas and Gavetti, 2000). Its members may also lack motivation to adapt their behavior, either because they have a vested interest in the status quo (Gilbert, 2005; Kaplan and Henderson, 2005; Chassang, 2010) or because different factions cannot agree on which solutions to replace or explore (Kaplan, 2008; Nigam, Huising, and Golden, 2016). Or the organization may simply lack the resources and capabilities to implement new solutions (Tripsas, 1997; Aggarwal and Wu, 2015; Stan and Puranam, 2016). As a result of these mechanisms, an organization may fail to implement behaviors that fit its new environment or may even fail to search for such behaviors.
Despite our understanding of these mechanisms, evidence suggests that we can still make substantive progress in predicting how organizations react to change and whether they successfully adapt as a result. Even studies that investigate seemingly similar organizations find significant differences in whether organizations adapt to change (Meyer, Brooks, and Goes, 1990: 93; Eggers and Park, 2018: 358). Evidence also shows that some organizations fail to adapt even when none of the usual explanations seem to apply: their members are aware of change, are motivated to adapt, and seem to have the capabilities to implement new approaches to their work (David, 1990; Brynjolfsson and Hitt, 2000). Such cases may not be isolated exceptions: many studies portray organizational members as mindfully scanning their environment for changes relevant to their work and modifying their behavior when they perceive the need to do so, lending significant plasticity to organizational behavior (Weick and Roberts, 1993; Feldman, 2000; Levinthal and Rerup, 2006). When members perceive that exogenous change has affected their organization, are motivated to adapt, and are capable of modifying their behavior, why would their organization still struggle to adapt successfully? While most explanations for failed adaptation emphasize pressures toward rigidity inside organizations, answering this question requires building theories that help us understand how plasticity and rigidity coexist in the face of change.
In this article, I construct a theory by building on models of organizational search in complex systems (e.g., Ethiraj and Levinthal, 2004; Siggelkow and Rivkin, 2009). Following Simon’s (1962) seminal work, a central argument for viewing organizations as complex systems is that organizations must solve a hierarchy of problems. At one level, an organization must find ways to fulfill operational tasks, which are often carried out by different specialized members or groups. At a higher level, it must set up processes that integrate the outputs of different specialists’ tasks. At both levels, an organization can search for more-viable behaviors over time: it can search for better ways to carry out tasks fulfilled by different specialists (what I call “modular search”), better ways to integrate the work of these specialists (“systemic search”), or both. A sports team can experiment with different ways for each player to carry out their specific role and can also explore different collective strategies governing how players interact. Similarly, a software organization can search for the best tools to be used by specialists responsible for software design, testing, implementation, and maintenance and can also search for the best sequence in which these specialists can collaborate and iterate on each other’s work (Balaji and Murugaiyan, 2012). Most organizations need to find viable ways to conduct specialists’ tasks and to provide integration between these tasks, even though organizations may differ in the decision-making structures through which they guide these search processes.
Conceptualizing organizational search at these two levels holds promise for reconciling the plasticity of members’ behavior with the possibility of failed organizational adaptation: it allows for the possibility that organizations search for new behaviors in the face of change but may fail to identify at which level(s) they need to search. But we know little about the mechanisms affecting organizations’ propensity to search at different levels. I describe one such mechanism, which is rooted in the interdependencies among choices at different levels of a complex system. The tasks performed by different specialists are often interdependent, and the choices about how to carry out these specialized tasks can affect the viability of choices about the processes that integrate these tasks (Henderson and Clark, 1990). I argue that the interdependencies of these choices can have a dual impact when organizations confront change: they increase the likelihood that systemic search will be beneficial, but they also make it difficult for decision makers to understand the benefits of systemic search. On the one hand, these interdependencies allow exogenous change to have ripple effects on the viability of systemic processes even when the change’s immediate impact is confined to specialized tasks. On the other hand, understanding these interdependencies well enough to foresee such ripple effects is difficult: making sense of complex interdependencies is a cognitive challenge to begin with, and the division of labor within organizations can reinforce this difficulty by creating an illusion of separability across tasks (Heath and Staudenmayer, 2000; Valentine, 2018).
Overlooking interdependencies among choices may be harmless—perhaps even useful—in a stable environment. But in times of change, it may hinder adaptation by causing decision makers to miss the forest for the trees: they recognize the value of the modular search for new ways to implement specialized tasks, but they fail to perceive the value of the systemic search for new ways to link these tasks. Because this issue originates from decision makers’ incomplete understanding of interdependencies among choices, I argue that it can be alleviated by information-processing structures that improve this understanding—such as lateral communication among specialists and reliance on formal coordinators.
To provide empirical support for my arguments, I use detailed data on collective behavior in esports, in which small teams compete in professional video-gaming tournaments. These teams must make choices about specialists’ tasks (each player selects one “hero” with distinctive abilities to fulfill specialized roles) and about systemic processes that integrate these tasks (strategies implemented through collective sequences of actions). The game I focus on, DOTA 2, underwent several exogenous game updates (“patches”) that influenced the relative effectiveness of different heroes and indirectly affected the relative viability of different strategies. The granularity of my data allows me to measure the extent to which teams engage in both modular and systemic search, capture information-processing structures reflected in teams’ communication patterns, and control for competitive dynamics to isolate teams’ search processes from their reactions to competitors’ moves. While the data do not directly capture the cognitive processes through which decision makers make sense of interdependencies and assess the value of search, the esports context allows me to reasonably isolate my mechanisms of interest from other common explanations for organizational inertia in the face of change. The results are consistent with my theory. When game updates affect the viability of strategies indirectly through their impact on heroes, teams react by experimenting with new heroes (i.e., modular search) but not with new strategies (i.e., systemic search). However, they do search for new strategies when game updates affect the viability of strategies directly, thereby helping teams understand the benefits of systemic search without having to make sense of interdependencies among choices. The results suggest that (1) lateral communication facilitates systemic search when understanding the benefits of this search requires making sense of interdependencies, and (2) formal coordinators accelerate systemic search only when its benefits are directly apparent without having to make sense of interdependencies. Additional analyses generate suggestive insights about the value of search in competitive settings: systemic search generates a competitive advantage only when recognizing its value is difficult enough that only a subset of organizations manages to do so. In addition to conducting the quantitative analysis of esports data, I report some qualitative fieldwork and a controlled experiment that provide additional suggestive evidence of the mechanisms I theorize about.
This study makes both theoretical and methodological contributions to the literatures on organizational search and adaptation in the face of change. By theorizing about how collective sensemaking among decision makers affects an organization’s search for solutions to complex, multilevel problems, I develop a novel explanation for why organizations may struggle to adapt to change even when their decision makers are aware of environmental change, are motivated to adapt, and have the resources to do so. My theory outlines mechanisms through which plasticity and rigidity can coexist in collective behavior, as decision makers may search for new solutions at one level of their organization’s task structure but not another. Because plasticity and rigidity can operate at different levels, organizations may appear either plastic or rigid depending on the level of analysis one observes. My study highlights the value of multilevel analytical approaches that can capture these asymmetries in search across levels.
Theory
Decision Making in Complex Systems
Scholars in several traditions have described collective behavior as a complex system involving different components and rules governing the interactions among these components (Simon, 1962; Argyris and Schön, 1996; Siggelkow, 2001, 2002). The behavior of an organization—whether its goal is to generate profits, save lives, or win football games—tends to involve a set of specialized tasks and processes for integrating the outputs of these tasks. A large body of research, on complex systems (e.g., Axelrod and Cohen, 1999; Miller and Page, 2007) and in the Carnegie tradition (e.g., Cyert and March, 1963; Lave and March, 1993), has emphasized the complex aggregation processes through which operational tasks contribute to collective processes, and the far-reaching impact of these aggregation processes on the behavior and evolvability of organizations.
Within this conceptualization, two elements are particularly relevant to my endeavor. First, an organization’s activity requires its decision makers to make choices implicitly or explicitly at different levels. They must select among possible ways to carry out operational tasks and among possible ways to set up the processes that integrate these tasks. The need for these choices originates in the division of labor. Because an organization’s activity involves a set of interdependent tasks too broad for one individual to carry out alone, subsets of tasks are allocated to different specialized individuals or groups, whose outputs must then be reintegrated into a coherent whole (Raveendran, Puranam, and Warglien, 2016). This insight transcends organizational size. A large newspaper organization might allocate production and distribution work to separate units, each comprising many members, while a small student newspaper organization might similarly divide the work of content production and distribution between two individual students. Both organizations must make choices about how to carry out specialized tasks related to content production and distribution and how to set up processes through which the individuals or groups responsible for these tasks interact to produce a newspaper.
As stated, I refer to “modular” choices about specialized tasks and “systemic” choices about processes that integrate tasks. I use the same adjectives to characterize the possible solutions that an organization can choose (modular solutions and systemic solutions or processes). The distinction between modular and systemic choices does not pertain to who makes these choices but to whether choices pertain to one specialist’s tasks or to the way an organization carries out processes linking these tasks. Thus the distinction between modular and systemic choices regards levels of a task structure rather than levels of a decision-making or information-processing structure (Dattée and Barlow, 2017). Organizations may rely on different information-processing structures when making these choices, as I explore below.
Second, organizations make decisions in the face of complex interdependencies among choices, such that the relative effectiveness of different solutions available to tackle one problem depends on the solution chosen to tackle another problem. A distinction between two types of interdependencies is relevant to my argument, as depicted in Figure 1 (which also describes how these interdependencies can channel the impact of change, which I explain in the next section). First, organizations face vertical interdependencies between their choices about specialized tasks (or bundles of choices about different specialized tasks) and their choices about systemic processes that integrate the outputs of these tasks. This relationship derives from a hierarchy among choices within complex systems: systemic processes govern the interactions among specialized tasks, and specialized tasks participate in the implementation of these processes. As a result, some integration processes are more effective when combined with specific ways of carrying out specialized tasks, and vice versa (Ethiraj and Levinthal, 2004; Siggelkow and Rivkin, 2009). The dashed black lines in Figure 1 represent these interdependencies. Second, choices about different specialized tasks are also interdependent. Systemic processes often realize their potential value only through the right combination of choices about specialized tasks (Levinthal, 1997; Siggelkow, 2001, 2002). Hence, these choices are interdependent for realizing the value of an organization’s systemic choices. The division of labor seeks to minimize different specialists’ interdependencies, but interdependencies that cut across specialists’ tasks remain (Simon, 1962). Figure 1 depicts these lateral interdependencies as dotted black lines.
Modular and Systemic Choices
Search and Adaptation to Change in Multilevel Systems
Vertical and lateral interdependencies among choices are highly relevant to adaptation in the face of change. On the one hand, these interdependencies allow exogenous change to affect the value of systemic choices even when its direct impact is relevant only to modular choices. On the other hand, the complexity of these interdependencies (and how the division of labor can obscure this complexity) makes it difficult for decision makers to recognize the indirect effect of an exogenous change on systemic choices.
The combination of lateral and vertical interdependencies can allow environmental change to impact an organization’s overall activity even if that change’s direct effect is contained within specific tasks or clusters of tasks (Henderson and Clark, 1990; Baldwin and Clark, 2000; Ethiraj, Levinthal, and Roy, 2008). In some cases, these ripple effects may arise through vertical interdependencies alone: because systemic processes differ in effectiveness depending on how an organization fulfills the tasks contributing to these processes, an exogenous change that makes an organization’s way of fulfilling specialized tasks obsolete can indirectly compromise the viability of systemic processes that integrate these tasks. For instance, in the fashion industry, technological advances in computer-assisted design (CAD) tools are credited for having allowed much faster generation of new designs and for having indirectly triggered the advent of “fast fashion”: the process of rapid iterative cycles between design and production to unveil new clothing collections quickly (Godart, 2012: 46). In other words, because of vertical interdependence, the technological advance that directly increased the viability of a solution used to fulfill a specialized activity (CAD) also indirectly increased the viability of a systemic process (rapid iterative cycles between design and production), thereby decreasing the viability of organizations’ prior processes. Ripple effects may also arise through a combination of vertical and lateral interdependencies. Godart (2012) suggested that “fast fashion” arose not only from the availability of CAD tools but also from their combination with digital marketing tools and flexible production facilities.
The solid red causal paths in Figure 1 formalize this mechanism: a change affecting the relative viability of different ways to carry out specialized tasks may indirectly affect the viability of an organization’s current way of integrating these tasks, either through a vertical interdependence alone (path α) or through a combination of lateral and vertical interdependencies (path β1+β2). Of course, Figure 1 is a simplification: each member of an organization tends to be responsible for multiple tasks or subtasks, which may each interact in multiple ways with other members’ tasks. As a result, interdependencies among choices provide many causal paths through which a change affecting the relative viability of different ways to carry out specialized tasks may indirectly affect the viability of an organization’s way of integrating these tasks. The existence of such causal paths is consistent with the few models of search that feature multilevel task environments (Ethiraj and Levinthal, 2004; Siggelkow and Rivkin, 2009; Csaszar and Levinthal, 2016). It is also consistent with anecdotal evidence that change challenging the viability of major organizational processes can emanate from seemingly isolated changes in specific product components (David, 1990; Henderson and Clark, 1990: 12; Meyer, Brooks, and Goes, 1990).
Not every exogenous change affecting specialized tasks will indirectly affect systemic processes; interdependencies simply make these indirect effects possible. When these effects do occur, however, they have important consequences for the search patterns that may benefit organizations: organizations may benefit from engaging not only in modular search for new ways of fulfilling specialized tasks but also from systemic search for new processes integrating these tasks. Search—done through “online” experimentation with ongoing behavior, “offline” thought experiments, or a combination of the two (Gavetti and Levinthal, 2000: 115; Knudsen and Levinthal, 2007: 40; Posen et al., 2018: 221)—helps organizations learn the value of different solutions to their problems. Through this learning, decision makers can assess whether exogenous change has modified the relative value of different ways in which organizations can carry out tasks and set up the processes integrating these tasks.
Overlooking Interdependencies and the Value of Systemic Search
Even when change does indirectly affect the viability of systemic processes, recognizing the value of systemic search is challenging because of the complex interdependencies described above. To understand that an exogenous change affecting specialists’ tasks has indirectly made an organization’s systemic processes unviable, decision makers must understand how choices about different tasks interact to affect the performance of existing processes. Otherwise, they may engage in modular search for new ways to implement specialists’ tasks without considering that new modular solutions may enable or require new ways to set up systemic processes that integrate tasks.
The need to understand interdependencies among choices generates challenges for organizations in at least two ways. First, generating a precise understanding of these interdependencies is a difficult cognitive endeavor, even for experts (Lee and Puranam, 2015). The number of possible interdependencies grows exponentially with the number of tasks in an organization, so that understanding all of them is challenging in most task systems. Even in small groups, individuals can struggle to recognize the consequences of environmental change for the viability of collective behaviors (Cohen and Bacdayan, 1994; Edmondson, Bohmer, and Pisano, 2001). Hence, even if decision makers can fully observe every task and process in their organization, they may still struggle to understand interdependencies among them well enough to make sense of change and thus to recognize the value of systemic search.
Second, the collective nature of sensemaking among members responsible for different tasks reinforces these cognitive challenges. While specialization into roles can help individuals learn the interdependencies between their and others’ tasks (Bechky, 2006; Kremser and Blagoev, 2021), their perception is often incomplete. Specialization can lead individuals to hold separate, incomplete representations of their organization’s overall task environment (Valentine and Edmondson, 2015; Nigam, Huising, and Golden, 2016; DiBenigno, 2018; Sackett and Cummings, 2018), generating an illusion of separability, as individuals make sense of their own work in isolation and overlook interdependencies with others (Heath and Staudenmayer, 2000). As a result, organizational members may treat stimuli around them as relevant only to their own tasks in isolation (Dearborn and Simon, 1958; Valentine, 2018).
Holding partial representations of interdependencies may not be an issue in the context of modular search, as its value may be recognized by attending to direct cues about the viability of different solutions to specialists’ tasks. Individuals may be able to make sense of those cues based on their own task knowledge and predictive knowledge about how their tasks interact with those of other members to contribute to their organization’s current processes. But searching systemically for new collective processes requires making sense of various causal paths involving lateral and vertical interdependencies, of which members may hold separate, incomplete representations. Thus I predict:
Information-Processing Structures and Sensemaking in the Face of Interdependencies
If systemic search is hindered by the difficulty of understanding how choices relevant to different tasks were interdependent within an organization’s processes before exogenous change occurred, then structures that help members generate rich understanding of these interdependencies should facilitate such search. I focus my predictions on simple lateral and vertical structures: lateral communication among specialists and vertical information processing via a formal coordinator. Versions of these structures exist in organizations of varying types and sizes (Puranam, 2018: 14). I test these predictions empirically in small teams and later discuss how my predictions may be extrapolated to larger organizations.
A range of prior research informs our understanding of the processes through which individuals become aware of interdependencies among their choices. Focusing on teams, a sizable literature subsumes the perception of interdependencies within the development and updating of accurate mental models (Johnson-Laird, 1983; Rouse and Morris, 1986; Klimoski and Mohammed, 1994; Mathieu et al., 2000; Santos et al., 2021). In larger organizations and occupational communities, ethnographic studies have generated rich accounts of the processes through which individuals come to understand the relationships among their roles (e.g., Bechky, 2003a, 2003b; Kellogg, Orlikowski, and Yates, 2006; DiBenigno, 2018; Lifshitz-Assaf, 2018). Some of these studies suggest that developing such understandings helps organizations adapt their behavior in response to both short-term surprises (e.g., Bechky and Okhuysen, 2011; Uitdewilligen, Rico, and Waller, 2018) and longer-term shifts in their working environment (e.g., Edmondson, Bohmer, and Pisano, 2001; Valentine, 2018).
Much of this work shares the insight that lateral communication among specialists facilitates the emergence of rich shared representations of their task environment (Klimoski and Mohammed, 1994; Bechky, 2006; Burke et al., 2006). Communication allows each member to share understanding of their role with others (Stout and Salas, 1993; Burke et al., 2006) and to make sense of how each role relates to others (Edmondson, Bohmer, and Pisano, 2001; Bechky, 2003b). It may also allow members to discover interdependencies of which none of them were aware, by describing their actions and outcomes as they occur, and thereby to learn how one member’s choices affect the outcomes of another’s (Lounamaa and March, 1987; Christianson, 2019).
These arguments suggest that members responsible for different tasks are more likely to understand how they depend on each other in contributing to systemic processes if they communicate frequently as they implement these processes, thereby generating a rich, shared perception of the interdependencies among them. By processing environmental cues in light of this shared perception, decision makers are more likely to understand that local cues can be relevant to wider organizational processes. Hence, I predict:
Lateral communication allows rich understandings of interdependencies to emerge bottom-up through members’ collective sensemaking. But top-down influence can also facilitate the perception of interdependencies. Several literatures highlight how formally designated coordinators or integrators facilitate sensemaking among interdependent agents (e.g., Mintzberg, 1979; Kozlowski, 1998; Burke et al., 2006; Stan and Puranam, 2016; Valentine, 2018). Individuals in such roles are responsible for facilitating coordination and shared understandings among specialized interdependent actors (Mintzberg, 1979: 165; Stan and Puranam, 2016: 1042), which can promote systemic search in two main ways.
First, by virtue of their role, coordinators may be able to form richer representations of task interdependencies throughout their organizations than other members can, and they may allocate significant attention to these interdependencies (Valentine, 2018). Repeatedly solving members’ coordination issues may progressively lead coordinators to perceive a large set of dyadic interdependencies among these members. By maintaining communication channels with all members whose work they coordinate and receiving information about their actions and outcomes, coordinators may also be able to make sense of how local choices relevant to different specialists contribute to collective processes. Coordinators’ richer understanding of interdependencies may also lead them to see environmental cues as relevant beyond the area in which those cues arose. As a result, they may be more likely than other members to perceive when change could make existing processes obsolete or new ones viable. Coordinators can then share their perception of interdependencies with other members (Edmondson, Bohmer, and Pisano, 2001).
Second, coordinators may facilitate other members’ sensemaking efforts even when the former lack superior understanding of these members’ interdependencies (Burke et al., 2006: 1195; Stan and Puranam, 2016). They can do so by mediating the process of forming and updating representations of members’ interdependencies, such as by explicitly updating members about each other’s behaviors as they occur. Coordinators can also provide a useful common starting point for other members’ sensemaking efforts, by communicating either their own perception of interdependencies or what they perceive as other members’ aggregate perceptions. It is easier to collectively improve a representation of interdependencies if members first agree on what the representation to be improved is (Puranam and Swamy, 2016; Santos et al., 2021). Coordinators may also remind other members of known interdependencies whose impact may be limited when their work environment is stable but that may impact collective processes in times of change (Martignoni, Menon, and Siggelkow, 2016; Nigam, Huising, and Golden, 2016). Based on these observations, I propose the following:
Data and Methods
I test my theory in an empirical setting that provides highly granular visibility of collective behavior at different levels: esports, or competitive video gaming. In this entertainment industry, small teams compete in professional video-gaming tournaments that feature substantial prizes (often above $1 million per tournament); they are watched by thousands of viewers in stadiums and millions online. Esports has developed tremendously over the last decade: as of this writing, about 500 million viewers watch professional esports games per year, generating more than $1 billion of revenue worldwide and significant externalities for the wider video game industry. 1 In recent years, esports has also received increased interest from the artificial intelligence community (e.g., McCandlish et al., 2018) and from organizational researchers (e.g., Ching, Forti, and Rawley, 2019).
I focus on a single game for which esports has been the most developed so far, as measured by the amount of tournament prizes: Defense of the Ancients 2 (DOTA 2). 2 It is a multiplayer online battle arena (MOBA) game, which industry insiders have described as “a combination of soccer and chess.” 3 A game of DOTA 2 involves two teams, each with five players. Each player controls one “hero,” chosen from more than 100 possible heroes, each with different abilities. Once heroes are selected, players engage in a battle that ends when the main building of either team is destroyed. Games typically last between 30 and 60 minutes. While games are always played on the same map, shown in Figure 2, the multiplicity of heroes and the different strategies they enable have ensured that hundreds of millions of players have maintained interest in the game since its inception in 2003. The stability of membership in professional teams is comparable to that of offline sports such as football and basketball. While there are movements across teams, and new players enter the dataset over time as teams seek new talent, players typically remain with a team long enough that I can meaningfully observe the evolution of a team’s behavior across games over time.
Map of the DOTA 2 Environment During a Game
The DOTA 2 esports platform is an attractive setting in which to test my arguments, for several reasons. First, it allows unprecedented visibility into modular and systemic search in the face of change. For hundreds of teams over a three-year period, I observed every mouse click and keystroke made by each player of any team engaged in professional tournaments. This allowed me to observe teams’ modular choices to enable each of their players to fulfill their own specialized tasks (i.e., the selection of playable heroes chosen from a pool of possibilities) as well as systemic processes integrating these tasks (i.e., strategies implemented collectively through sequences of actions involving multiple players). As teams play games repeatedly over time, I could identify how much they search, both modularly and systemically, from one game to the next. I also observed how much players communicate with one another and whether communication is centered around one player formally identified as “captain,” whose role, in addition to playing the game, is to facilitate coordination among team members. Hence, despite their small size, DOTA 2 teams are an attractive unit of analysis for my purpose. These teams fit most definitions of an organization: they include multiple individuals working toward a common goal, which requires them to divide labor into separate tasks, allocate these tasks to members, and integrate their efforts (Puranam, 2018: 9). Members fulfill stable roles and engage in a series of interlocking routines (Westley, 1990: 339; Weick, 1993: 632). Capturing fine-grained measures of these routines allowed me to follow a team’s strategy over time, placing this study within an empirical tradition that treats organizations as structured patterns of action (Cohen and Bacdayan, 1994; Pentland and Rueter, 1994; Pentland, 1999; Pentland, Hærem, and Hillison, 2011).
Second, this context featured exogenous shocks during the observation period, in the form of game updates released by the company developing DOTA 2 (Valve Corporation), which modified the characteristics of the different heroes available to teams. These updates allowed me to observe whether teams respond to exogenous change by trying out different heroes to carry out specific tasks (which constitutes modular search) and by experimenting with new strategies collectively executed by their players (systemic search). These updates suggest a good fit between my theory and empirics because they embody the notion that exogenous change can indirectly affect systemic processes even when its direct impact is contained within specialized tasks. Heroes with different abilities enable different collective strategies, so that updates can reward systemic search for new strategies even when the updates’ direct impact is only to change heroes’ abilities. I expand this argument in my description of game updates below, and I provide suggestive evidence in the Additional Analyses section that these updates indeed rewarded systemic search.
Third, several characteristics of esports teams enabled me to isolate my mechanisms of interest more effectively than would be possible in other empirical settings. On the one hand, these teams engage in an activity complex enough to require modular and systemic choices that depend on one another. DOTA 2 is extremely complex by the standards of video games; it is difficult to understand how the tasks performed by players with different heroes can best be combined into an overall strategy. On the other hand, these complex interdependencies between modular and systemic choices are contained within small groups of five people, and each member’s behavior is visible to all others. From this standpoint, focusing on esports may generate a conservative bias in my results; generating accurate representations of interdependencies may be even more difficult in larger organizations featuring interdependencies among large sets of members who may not observe each other’s behavior (Henderson and Clark, 1990; Aggarwal and Wu, 2015; Marino et al., 2015). The relatively simple organizational structure of teams in DOTA 2 also helps eliminate possible alternative mechanisms to the one I focus on, especially related to structural inertia (Hannan and Freeman, 1984) and coalition dynamics (Kaplan, 2008). Five-person teams are small enough that bureaucracy is unlikely to be an issue and coalitions are unlikely to form. The substantial prizes at stake also make it unlikely that team members lack motivation. Moreover, Valve Corporation publishes detailed documents describing the exact changes made to the game in each game update, so my results are unlikely to reflect players’ lack of awareness of change; rather, they should reflect the teams’ ability to understand the precise impact of these changes on the viability of their current modular solutions and systemic processes.
Finally, DOTA 2 features many sources of qualitative information that helped me to understand the mechanisms at play and guided the construction of my measures. I gathered information through public sources such as written publications and video documentaries featuring live recordings of teams at play, as well as my own fieldwork, which involved more than 150 hours of direct observation of teams during tournaments in the United States, Canada, and Poland. My fieldwork included interviews with professional players, 4 tournament organizers, and analysts. Additionally, I spent more than 1,000 hours playing DOTA 2, including 50 hours playing it with professional players, to better understand the game’s mechanics.
Data Overview and Description of Variables
The raw data consist of every mouse click and keystroke made by any player of a professional team in 13,948 games played over a three-year period (2014–2017) by 272 professional teams. This granularity provides an opportunity to observe collective behavior, with remarkable analytical depth. For every game in the dataset, I observe not only the heroes selected by a team’s players but also the collective behaviors that players generate using those heroes.
Specialists’ tasks and systemic processes in esports
DOTA 2 features a clear distinction between specialized tasks and the systemic processes integrating these tasks. Each team’s five players each select a playable character (hero). Heroes can move on the game map, gather resources, attack other players, and use specific abilities that distinguish them from other heroes. Each hero possesses several abilities, which can be categorized among six types (based on industry sources and documents from the game manufacturer): “unbalancing” abilities either strengthen a team’s heroes or weaken an opposing team’s heroes; “mobility” abilities enable fast movement; “escape” abilities conceal a hero; “constraining” abilities reduce opponents’ ability to move on the map; “disabling” abilities prevent an opponent from performing any action for a period of time; and “destabilizing” abilities affect the opposing team’s costs of keeping its current position on the map.
Although any hero’s ability may be categorized within these six types, different abilities of the same type may vary in their effectiveness. For instance, the mobility ability of one hero may be more or less powerful than the mobility of another hero. 5 While the game developers strive to balance the strengths of different heroes (which is partly why they released the game updates I describe below), achieving this balance is difficult because the strengths of different heroes synergize in ways that even the game developers may not fully anticipate. At any point, teams typically hold beliefs about which heroes are the strongest and tend to select these heroes more often than other available heroes. Most teams share some of these beliefs, while other beliefs may vary across teams.
The selection of each hero is a choice about how to carry out specialized tasks: each hero is controlled by one player who fulfills one cluster of tasks for the team. The different bundles of tasks available to each player are stable enough across teams that they have been institutionalized as roles with specific names. 6 Teams feature five specialists of different roles, who each select one hero for the entire duration of a game. The details of a hero’s abilities determine their value in carrying out a specific role; each hero is usually a better fit for some roles than others. Teams choose each player’s specific hero right before a game starts; the two opposing teams choose heroes one after the other until a hero has been chosen for each of the ten players. The choice of hero is especially affected by the opinion of the player who will control this hero to fulfill their role. I observed video recordings of teams during high-stakes games that showed how the selection of a hero for a role emerges either from suggestions by the player responsible for a cluster of tasks, followed by agreement from the other team members (especially the team’s captain), or from other team members’ suggestions, followed by the player’s agreement. The following quotes from conversations among players during the hero-selection phase of games illustrate these processes (the names are pseudonyms used by players):
“We need to pick a hero that can clear creeps from a distance.”
“Just grab Lina.”
“What about Ember Spirit?”
“No, just pick Lina.” (Conversation recorded in 2017, during a game with $7 million at stake)
“Give me Enchantress. . . . I’m telling you, Alchemist and Enchantress will do.”
“Okay, Lil. Do it.” (Conversation recorded in 2017, during a game with $500,000 at stake)
“What do we want?”
“I mean . . . Lina or Zeus I think.”
“Choose one, what are you feeling the most?”
“Ugh, I don’t know . . .”
“No matter what it is, we’re gonna destroy them with it!”
“I think I like Zeus.”
“Zeus, guys?”
“I’m in for anything!”
“Zeus.” (Conversation recorded in 2018, during a game with $7 million at stake)
While heroes are selected for specific tasks, their value eventually emerges from the collective strategies they enable for the entire team. For instance, a team may opt for an aggressive strategy, seeking to engage the opponent team with all its members, or may choose to focus on disrupting the enemy team’s efforts to gather resources in different places on the map. A team’s strategy is a systemic choice that affects all its players; it is typically chosen before a game and then emerges in the collective sequences of actions among players during the game (with some noise generated by the opponent’s attempts to disrupt the focal team’s strategy). I can capture strategies in my data because different strategies involve the use of different heroes’ abilities in different order. For instance, an aggressive strategy—with an objective of seeking fights to kill several opponent heroes at the same time (heroes reappear in the game after some time) and destroy enemy buildings while they are absent—involves a specific pattern of action. A team’s players will often start by using a mobility ability to reach the opposing team, then use some disabling ability (by the same player or another) to keep the opponents in place, followed by destabilizing abilities to make it difficult for their opponents to react. Other strategies involve different sequences, which I describe through a cluster analysis in Online Appendix A in this article’s supplementary material. While a team’s strategy may sometimes combine different types of sequences during a game, it usually relies extensively on some sequences at the expense of others. Each team develops its own play style over time and tends to reproduce it across games. I report evidence of this stability in Online Appendix A.
Dependent variables
To capture modular search in my test of Hypothesis 1, I measure Experimentation with new heroes as the number of heroes selected by the team in the focal game that were never picked by the team in its previous ten games. 7 Robustness checks using a more granular measure (the percentage of the last ten games in which the team was using the heroes it selected in the focal game, averaged over all five heroes) yielded similar results. This measure may vary over time (even in periods when no game updates are released), so that regression coefficients reflect whether teams increase their experimentation with heroes as a response to environmental change.
To capture systemic search and test Hypotheses 1, 2, and 3, I measure Experimentation with new strategies. I capture strategies through their materialization as collective sequences of actions during the game, in two steps. First, for each game in the database, I recorded all sequences of actions involving several of a team’s players, in which each action was performed less than three seconds after and within close geographical proximity of the previous action. 8 Second, for each game, the rate of change is computed as the average distance between all sequences recorded in the team’s focal game and the sequences recorded in its ten previous games. This measure reflects how different a team’s strategy is in the focal game compared to the average strategy it played in the last ten games. Similar to my interest in heroes, I was interested in whether teams intensified their experimentation with new strategies as a response to game updates.
I computed distances through optimal matching (or “optimal alignment”), a method widely used in bioinformatics for the computation of distances between biological sequences such as DNA (e.g., Edgar, 2010) and used in various social science applications (Abbott, 1983; Han and Moen, 1999; Aisenbrey and Fasang, 2017). Optimal matching computes the distance between two sequences as the cost of turning one sequence into the other by substituting elements for one another, deleting elements, or inserting elements (MacIndoe and Abbott, 2004; Aisenbrey and Fasang, 2010; Gabadinho et al., 2011). In this study, given my operationalization of distances between sequences in DOTA 2, one can think of optimal matching as the process of finding the longest common subsequence between two sequences of actions. I describe the procedure in detail, along with robustness checks for different operationalizations, in Online Appendix A.
Independent variable for exogenous environmental change
Studying search in the face of exogenous change is possible in DOTA 2 because game updates occurred during the observation period. Game updates—modifications to the game’s code by Valve Corporation—change the mapping from the players’ actions to their outcomes in the game and, hence, affect the way players experience the game. The reason for releasing these updates is twofold. First, DOTA 2 is a commercial game. While my study focuses on a population of about 1,000 professional players, DOTA 2 is mainly a recreational game that has been played regularly by more than 10 million players since its inception in 2003. Changing the rules is a way to renew the game and keep players entertained. Second, as explained above, it has been challenging to achieve perfect balance between the strengths of different heroes. The synergies among different heroes make it difficult even for game developers to predict how strong some heroes will become when their abilities are modified. As players uncover these synergies over time, Valve has released updates to restore the balance among heroes’ strengths. Figure 3 shows a timeline of the game updates released during the observation period, from 2014 to 2017. Online Appendix B provides detailed descriptions of each update.
Chronology of DOTA 2 Game Updates
These updates allow me to test my predictions because they exemplify the notion of environmental change having a direct impact on specialized tasks and indirect effects on systemic processes. The direct effect of these updates is to modify the abilities of heroes available in the game. For example, a game update may strengthen a certain hero’s constraining ability by allowing it to slow an opponent hero by 60 percent for 5 seconds rather than 50 percent for 3 seconds. But heroes with different abilities also enable different collective strategies in the game. Hence, enhancing a hero’s constraining ability not only makes it more attractive for teams to select this hero but also indirectly makes it more valuable to choose strategies that rely on constraining abilities, such as disruptive strategies that involve immobilizing isolated heroes and then killing them. Even small changes to heroes’ abilities can require teams to significantly adapt their behaviors. Such a change may push a hero beyond the threshold of viability, which in turn pushes a certain strategy beyond the threshold of viability.
These facts make it valuable for teams to experiment both with new heroes and new strategies as a response to game updates. However, recognizing these indirect effects of change is challenging for the reasons outlined in my theory: team members must assess whether their current strategies’ viability has changed based solely on their understanding of synergies among different heroes’ characteristics as well as synergies between heroes and strategies. Note that each update in my dataset had simultaneous impacts on many different heroes. I interpret this as generating a conservative bias on my results of interest. In a setting where the viability of systemic choices depends on local choices, exogenous shocks affecting the viability of many local choices at once bear a higher probability that at least one of these local changes (or a combination of them) will have systemic repercussions. Finding support for Hypothesis 1 in such a setting would suggest that my mechanism of interest operates strongly enough to keep teams from reacting systemically even in a setting where many local changes happen simultaneously.
Five of these game updates were released during the observation period, as shown in Figure 3 (which also features three updates of a different type, allowing me to provide additional evidence of my mechanisms of interest, as described in the Additional Analyses section). I captured Change with indirect systemic impact through an indicator variable equal to one if the game happened within two months of a game update that affected only the abilities of heroes.
Independent variables for information-processing structures
The data allow me to indirectly measure communication within teams. Verbal communication among team players cannot be directly observed in my data, but communication can be proxied through the number of pings per minute used by a team’s players during a game. Pings are graphic-and-sound signals that players can generate by clicking a specific location on the game’s map. These signals constitute a major means of communication between players during games. As one of the professional players I interviewed explained,
We communicate a huge amount with pings, so much that we don’t even realize it anymore. It’s almost part of the sentence. We use pings a lot because it’s very easy: we just have to use our mouse and it makes a sound. With experience, your brain starts to really pay attention to it: I can hear a hundred different sounds in the game, but if I hear a ping, I know it’s important.
Players use pings either to signal the occurrence of events during the game or to communicate how something should be implemented. Especially in the latter case, pings constitute a complement to verbal communication that is especially relevant to my purpose because implementation-focused communication may allow players to make their choices and behaviors more visible or salient to each other and better understand how their choices interact. I measured the Frequency of communication as the number of pings per minute among all members of a team during a game.
Reliance on a formal coordinator was measured based on the existence of a captain role in all teams. Teams allocate the captain role to one of their members and may rely on the captain to decide which behaviors to perform and how to execute them. This role is stable within teams: decisions to change the individual filling the captain role are very rare in DOTA 2. Valve Corporation requires every professional team to formally report the name of the player occupying the captain role, but interviews of professional teams revealed variation in whether the reported captain was actually central to the team’s coordination efforts. Because this coordinating influence is central to my arguments leading to Hypothesis 3, I measured Reliance on a formal coordinator as the percentage of pings emanating from the captain relative to the total number of pings within the team during the focal game.
I measured the two information-processing variables as the average over the last ten games before the last game update, in order to mirror Hypotheses 2 and 3’s emphasis on information processing before an exogenous shock. I seek to capture whether teams use information-processing structures that allow their players to better understand how their respective tasks interact to enable their existing strategy and, hence, better understand that changes to the heroes used by different players may indirectly affect the viability of the team’s overall strategy. Measuring information-processing structures before shocks happen also mitigates potential concerns linked to reverse causality, whereby a team may either communicate more or centralize its communication network around its captain because of the search pattern its members have decided to adopt.
Control variables
I control for the Ratio of recent games won by the focal team during its ten previous games to capture performance feedback; teams may be less likely to engage in search if they were successful in the recent past. Models also include the Number of web-television viewers who watched the game, because teams may be more risk-averse, and hence experiment less, during important games. Similarly, I control for the Time before the next million-dollar tournament as the amount of time before the next tournament with a total prize pool equal to or greater than $1 million. As they approach such tournaments, teams may either try to converge toward stable behaviors or explore strategies they expect their opponents to adopt. I also control for the Average common experience of team members: as members become more familiar with each other, they are likely to develop robust routines that lead them to experiment less over time. I include the Duration of the match because of the specificity of DOTA 2 games. Differences between strategies are most apparent during the early to middle phase of a game, so that long games may exhibit less distinctive behavior than short games, on average.
I also seek to control for the competitive nature of DOTA 2 games. The task environment faced by teams involves a combination of technical and competitive factors. In any game, a team’s collective behavior is affected not only by game factors such as the specificities of heroes and how rewards are obtained but also by the specificities of the opposing team. I control for this in three ways. First, I control for the possibility that teams may attune their level of experimentation depending on the perceived strength of their opponent, through a measure of Expected superiority to the opponent. To compute this measure at any point in time, I generated a directed network among all teams in my dataset, where the value of a link from any team i to another team j is equal to the number of times team j won against team i in the past two months (experimenting with different time windows did not affect the results). Centrality in this network should reflect a team’s strength based on the past two months: to be central, a team needs to win most of its games against teams that also tend to win against many other opponents. Expected superiority is computed as the difference in eigenvector centrality (Bonacich, 1987) in this network between the focal team and its opponent. Rankings of teams based on this measure had excellent face validity with industry insiders.
Second, I control for the Opponent’s experimentation with new heroes and the Opponent’s experimentation with new strategies. These variables are computed identically to the focal team’s experimentation. I include these variables for two reasons. First, controlling for the opponent’s experimentation enables me to capture potential factors specific to the focal game that may lead both teams to change their behavior compared to the prior games they played. Second, these variables may capture perturbations to the focal team’s choices due to its opponent’s behavior during the game: a team might deviate from its initial game plan due to its opponent’s choices.
Finally, I include the number of Previous games played against the opponent to capture the possibility that as a team repeatedly faces the same opponent, it may develop opponent-specific strategies that artificially generate variation in its behavior relative to previous games. The pattern of support for my hypotheses reported below is robust to the exclusion of all control variables.
Analysis and Results
The dataset consists of 13,948 observations for 272 teams from August 2014 to March 2017. Table 1 reports summary statistics, and Table 2 reports regression models predicting search. I use standardized values for all variables in my regressions, to facilitate interpretation. All models use robust standard errors clustered at the team level.
Summary Statistics
Models Predicting Experimentation with New Heroes (Poisson) and Experimentation with New Strategies (OLS)*
p < .05; •• p < .01.
Standard errors are in parentheses.
Models 1 and 2 use Poisson regression models with team fixed effects to predict a team’s experimentation with new heroes. Model 1 includes all control variables, and Model 2 adds the effects of environmental change with indirect systemic impact. These changes have a positive effect on a team’s experimentation with new heroes: teams try out new heroes after a game update.
The next models use OLS panel regressions with team fixed effects to predict a team’s experimentation with new strategies: do teams react to game updates by experimenting with new strategies, reflected by new collective sequences of actions? Model 4 includes all control variables, and Model 5 adds the effects of change with indirect systemic impact. Hypothesis 1 is strongly supported: as a response to changes with indirect systemic effects, teams are more likely to experiment with new heroes than with new strategies. In fact, these game updates even have a negative effect on experimentation with new strategies: teams react to them by experimenting with new strategies at an even lower rate than in stable periods, when no change has happened. Figures 4a and 4b provide more-detailed insights into patterns of search over time, by reporting graphically the results of models similar to Models 2 and 5, in which the “exogenous change” variables are replaced by dummies for each of the first ten weeks after a game update was released (ten dummies for each type of change). The date of the exogenous change is represented as a vertical line. Confidence intervals are depicted in shaded areas (smoothened to a fifth-degree polynomial). Figure 4a shows that teams tend to immediately experiment with new ways of carrying out individual players’ tasks: there is an immediate increase in experimentation with new heroes following an update, and experimentation remains higher than usual for eight weeks after the update. Figure 4b suggests that systemic reactions are more gradual: experimentation with new strategies remains stable for about three weeks and then becomes significantly lower for several weeks before gradually returning to its pre-update level. 9
Effect of Change with Indirect Systemic Impact on Modular Search (i.e., Experimentation with New Heroes)
Effect of Change with Indirect Systemic Impact on Systemic Search (i.e., Experimentation with New Strategies)
The finding that teams experiment with strategies even less after these game updates is worth discussing further. While this finding is not inconsistent with my theory, it cannot be derived from my theory alone. My theory suggests that teams may not increase their rate of experimentation with strategies, but not necessarily that they will decrease it. I interpret this finding as deriving from my mechanism of interest combined with tendencies toward “slack search” in stable periods (Levinthal and March, 1981). In stable periods when team members feel that they have mastered their preferred strategy well enough, teams still engage in incremental search for better strategies to check for whether they may have overlooked valuable ones—hence their rate of experimentation is not zero. Such processes are common in the context of search within complex spaces where exhaustive search is impossible, as occurs in many organizational settings. However, when teams face updates affecting heroes, they may divert their attention from this incremental search. They focus on relearning how to implement their existing strategy with new heroes, before going back to their usual incremental search. Yet the rate of experimentation after these game updates never goes above the pre-update level (as shown in Figure 4b). Hence, teams seem to interpret these updates as requiring them to learn how to implement their preferred strategies with new heroes but not as an impetus to search for new strategies enabled by these heroes.
Another potential explanation for this finding is linked to threat rigidity (Staw, Sandelands, and Dutton, 1981; Gilbert, 2005): organizations may be willing to engage in a baseline level of experimentation when the environment is predictable enough, but they may be inclined to rely on well-established routines and strategies when it is less so. While the mechanism I describe does not preclude the possibility of threat rigidity, I present evidence in the Additional Analyses section that points to the mechanism I describe as a more likely driver of my results.
Models 6, 7, and 8 add interactions between change with indirect systemic impact and the two information-processing structures of interest: communication and reliance on a formal coordinator. In Model 8, which includes both interactions simultaneously, the interaction with the frequency of communication has a significant positive impact on the rate of change in strategies (p < 0.01), but the interaction with reliance on a formal coordinator has no significant impact. Hence, I find support for Hypothesis 2 but not for Hypothesis 3: frequent communication dampens the negative impact of change on the rate at which a team experiments with new strategies, but there is no evidence that reliance on a formal coordinator has a similar impact. I explore these information-processing structures further in the next section.
Additional Analyses: Mechanisms and Performance Implications
The mechanism behind my predictions is rooted in individual bounded rationality and is reinforced by the collective nature of adaptation in organizations. Organizational members may not recognize the value of systemic search, because they hold incomplete understandings of the interdependencies among their tasks. Attaining such understanding can be challenging for one individual dealing with one task involving complex interdependencies among subtasks, and it is even harder in collective decision making among specialists who hold separate mental models of their task environment. Parts of this mechanism may be unobservable because they manifest in the cognition of team members: despite the granularity of my data, I cannot directly measure how team members perceive the consequences of game updates. Thus the identification of this mechanism relies on convincingly eliminating other possible explanations for my results.
My choice of empirical setting and of operationalizations should alleviate some concerns linked to other potential explanations. For instance, the small teams in esports are unlikely to feature inertial mechanisms linked to bureaucracy (Hannan and Freeman, 1984) or conflict among coalitions (Kaplan, 2008). Explanations linked to the difficulty of recognizing whether environmental change is relevant to an organization (e.g., Tripsas and Gavetti, 2000) are also unlikely since game updates are made public and their effects described in detail. Similarly, because I operationalize search as a difference between present and recent behavior, results are unlikely to reflect failed implementation (Aggarwal and Wu, 2015; Stan and Puranam, 2016). Failed implementation of new behavior would be reflected as an increase in systemic search, rather than the decrease I observe. But the results may be partially consistent with a threat rigidity mechanism: faced with change requiring them to alter their collective routines, team members may invest in new resources (in this case, learning to play with different heroes) but stick to their old routines because they fear they would fail to implement new behaviors successfully (Gilbert, 2005). I run additional analyses to alleviate this concern and provide additional evidence of the mechanism I describe.
Changes with Direct vs. Indirect Systemic Impact
The mechanism I theorize is relevant to changes that generate ripple effects from specific tasks to systemic processes: change may appear to affect specialists’ tasks in isolation, but interdependencies cause it to also affect systemic processes. This justifies my decision to focus on game updates in DOTA 2 that only altered heroes’ abilities and, hence, required teams to understand the systemic consequences of these updates based on their understanding of specialists’ interdependencies. However, a second type of update occurred three times over the observation period and made direct modifications to the overall game rules in addition to modifying heroes’ abilities: the game manufacturer changed the ways in which the two main payoffs obtained by teams during a game (“gold” and “experience”) were calculated.
This is a meaningful difference because these changes directly affected the relative value of different strategies in the game. For example, a game update released in April 2015 explicitly affected the relative benefits (in gold and experience) of killing opponents and of passively gathering gold and experience on the game map. These changes affected the average payoffs from different collective behaviors: when killing opponents becomes more valuable than gathering resources passively, teams have a clear interest in behaving aggressively by fighting the opponent directly rather than trying to escape fights and focus on gathering gold and experience elsewhere. Hence, these game updates directly affected the value of different strategies and reduced the need for teams to understand the indirect systemic effect of changes affecting heroes’ abilities. Online Appendix B provides detailed explanations of these updates.
I captured Change with direct systemic impact through an indicator variable equal to one if the game happened within two months of a game update affecting both heroes’ abilities and the general rules of the game. This variable is added to Model 3, predicting experimentation with new heroes, and Model 9, predicting experimentation with new strategies. These models show that game updates with such a direct impact have a positive effect on experimentation with new heroes as well as with new strategies. This finding provides additional evidence that my mechanisms of interest are driving systemic search in the face of change: when the systemic impact of change becomes easier to understand, teams do search systemically. These results also run counter to what would be expected from a threat rigidity mechanism: changes with direct impact on system-level behavior should appear more threatening to members and, hence, would be expected to cause higher rigidity.
Figure 4c, which was constructed similarly to Figures 4a and 4b and shows the pattern of search over time, confirms again that shifts in experimentation with new strategies occur more gradually than shifts in experimentation with new heroes.
Effect of Change with Direct Systemic Impact on Systemic Search (i.e., Experimentation with New Strategies)
Model 10 further probes the effect of change with direct systemic impact by adding interactions between this type of change and the two information-processing structures. Interestingly, reliance on a formal coordinator accentuates the positive effect of such changes on a team’s experimentation with new strategies (p < 0.01). This provides a more complete picture of how information-processing structures may guide search in the face of change: lateral communication and formal coordinators facilitate search through different mechanisms that may facilitate systemic search in different situations. When recognizing the need for systemic search necessitates understanding how complex interdependencies guide the impact of exogenous change, collective sensemaking through communication among specialists may be required. When external cues remove the need to make sense of interdependencies by making it clear that current processes are no longer viable, formal coordinators may use their authority to accelerate systemic search and convergence toward specific processes to experiment with.
There are several possible explanations for the fact that formal coordinators facilitate systemic search only when its value is already apparent. Some of these explanations are specific to DOTA 2. For instance, captains fulfill operational roles in addition to being responsible for coordination. In other organizations, formal coordinators may be removed from operational duties and focused solely on coordination (Mollick, 2012; Stan and Puranam, 2016; Clement, Shipilov, and Galunic, 2018). This is a meaningful difference, as being engaged in operational tasks can distract from sensemaking (Christianson, 2019) and may preclude coordinators from generating a rich perception of interdependencies. But my results may reflect a more general insight: much like the individuals whose work they coordinate, coordinators are not immune from bounded rationality and must construct partial representations of their environment (Martignoni, Menon, and Siggelkow, 2016). While interdependencies within an organization’s overall activity system may be more salient to coordinators than to specialists carrying out individual tasks, coordinators may not have the same ability to make sense of how a specific change to a task may affect the specialist in charge of it—and all of that individual’s direct interactions with other specialists. Hence, my results align with earlier research questioning the efficacy of formal coordinators for highly reciprocally interdependent scenarios, as in Thompson’s foundational work (Thompson, 1967: 56, 133): in the presence of complex interdependencies among tasks, information-processing structures that promote collective sensemaking among all specialists may provide a more solid foundation for recognizing the benefits of systemic search than boundedly rational coordinators do. Thompson’s insights help to reconcile the lack of support for Hypothesis 3 with my general argument that information-processing structures that foster shared accurate perceptions of interdependencies facilitate systemic search. The results do not invalidate this point but suggest that formal coordinators may struggle to foster such perceptions when interdependencies are highly complex and reciprocal.
Effects on Performance
So far, my analysis has focused on whether teams engage in systemic search as a response to environmental change. My explicit assumption is that doing so can benefit organizations because it enables them to find combinations of modular solutions and systemic processes that better fit their new environment. As each DOTA 2 game generates a winning team and a losing team, the data allow me to generate suggestive evidence that systemic search does in fact bring rewards in the face of change. Models 11a, b, and c use logit regression models with team fixed effects to predict a team’s victory in a game, by splitting the combined sample into three samples of games that happened (a) in stable periods, (b) in the two months following a change with indirect systemic impact, and (c) in the two months following a change with direct systemic impact. These models are reported in Table 3 and omit control variables that do not vary across teams (for instance, the duration of a game is the same for the focal team and its opponent and, hence, cannot predict victory). The two variables of interest are a team’s experimentation with new strategies in the focal game and in its last ten games (Experimentation with new strategies in the last 10 games).
Logit Models Predicting a Team’s Victory*
p < .05; •• p < .01.
Standard errors are in parentheses.
On the one hand, experimenting with new strategies during the focal game has a negative effect on performance in all three models. This fits the idea that search is an investment in the future: most experimentation efforts fail, but with enough trials, an organization may discover more-effective ways of operating (March, 1991). These future benefits are reflected by the effect of recent experimentation with new strategies (in the last ten games). While Model 11a suggests that recent experimentation with new strategies has no effect in stable periods, Model 11b suggests that recent experimentation has a positive effect on performance after changes with indirect systemic impact. This provides suggestive evidence that failing to experiment with new strategies (i.e., failing to engage in systemic search) is not a correct response following these changes; rather, it seems to be a suboptimal response resulting from teams’ inability to perceive the effects of change on the relative value of different strategies.
Note that, as Model 11c shows, recent experimentation with strategies does not seem to affect performance after changes with direct systemic impact (i.e., changes to the overall rules of the game, such as altering the value of gold and experience, as explored in the previous section). This result may initially seem puzzling. Indeed, these changes are arguably more disruptive to the value of a team’s current strategies than those that feature only an indirect impact: the former changes generate the same modifications as the latter changes but also make additional modifications with direct impact on the value of different strategies. Hence, these updates would be expected to reward systemic search even more. However, the competitive nature of esports may explain the results: performance is not determined by whether a team’s behaviors fit the environment but by whether they fit it better than those of the opposing team do (Aime et al., 2010). When exogenous change affects behavior through complex interdependencies and significantly challenges teams’ ability to understand the value of systemic search, being one of the few teams to experiment with new processes is more likely to generate competitive advantage. When additional cues allow teams to recognize the systemic impact of change without making sense of these interdependencies, most teams may understand the value of systemic search, such that no competitive edge is generated.
The analyses predicting performance should be interpreted with more caution than those predicting search. While the game updates I rely on to predict search are exogenous to each team’s behavior, search is not exogenous to the performance of teams; teams make choices about heroes and strategies in an attempt to perform better. But these results are included in the analysis because they provide valuable suggestive evidence that systemic search yields performance benefits following environmental change. Hence, they help confirm that the decrease in systemic search following changes with indirect impact on strategies (and the increase in systemic search when facing more-direct changes) is not a simple performative response by teams with little consequence for performance but, rather, a non-optimal response due to the mechanisms I describe.
Qualitative Evidence
I also rely on interviews and observations of teams at work to illustrate the mechanisms underlying my predictions. The idea that even game updates affecting the abilities of individual heroes may reward experimentation with new strategies seems to have gained traction in professional esports over the course of the observation period. In 2015 and 2016, one could already find anecdotes about teams that took advantage of updates affecting heroes to adapt their strategies successfully. A professional player observing another team could be heard speaking to his teammates: “I don’t understand, they have so many slow heroes, but they play so fast! How? That confuses me.” In related terms, a writer for an esports journal explained the diverging performance trajectories of two teams (Na’Vi and Team Secret) after a game update (colloquially referred to as a “patch”), as follows: “The problem with Na’Vi is that they didn’t try to change with the patch. They said, ‘we’ll just play the same strategy, we’ll just do what we do.’ Whereas Secret kept adapting towards a ‘greedy support’ strategy and became the best team in the world.” This idea became more widely accepted in the industry over time, especially with the prominent example of one team winning the most prestigious tournament (“The International”) by adapting its strategy after seemingly minor changes to heroes, as a tournament organizer described: “It’s exactly why OG won The International. The patch hadn’t changed much and yet they played a different game from everybody else.”
My mechanisms of interest could operate regardless of whether decision makers are conscious of these mechanisms. However, several of my interviewees did mention issues closely related to my arguments about the difficulty of systemic search. A prominent professional player described the difficulties of reacting to game updates when their impact on strategies is felt only indirectly through their effect on heroes rather than directly through the game’s rewards system: “Some patches are very clear and pretty easy for us to understand . . . what has the biggest impact is the experience system. . . . But sometimes it looks really marginal, like something going from 1.9 to 2.1 on a hero. Sometimes you can’t understand it at first.” When I asked players how they make sense of updates to heroes, most focused on the consequences of these updates for hero selection. Interestingly, players seemed aware that game updates have indirect effects beyond the changes stated explicitly, but they often limited their attention to their indirect effects on hero-selection choices rather than considering systemic impacts on their strategies. One player stated, “When I look at patch notes, what I’m looking for are big changes to popular heroes, and changes to individual heroes which indirectly make other heroes better or worse.” This focus on heroes may translate into inertia in the strategies played by teams, as described by this same player: “Some people decide ‘Well, we’re not gonna change anything and keep doing what we do. And if they’ve really made a major change that affects something we did, we’ll just stop doing that but keep doing what hasn’t been affected.’” This tendency might explain why experimentation with new strategies does not seem to increase after a game update, and also why some evidence suggests it even decreases as teams refocus on elements of their strategy that the update has not affected and relearn how to implement their strategies with new heroes.
Experimental Evidence
My theory is based on both individual cognition and collective sensemaking. With respect to individual cognition, I argue that understanding the systemic consequences of change presents cognitive difficulties linked to understanding indirect causal paths through lateral and vertical interdependencies. This difficulty is reinforced in the context of collective sensemaking: when individuals specialize into different tasks, they may struggle to acquire full visibility of other members’ tasks and the way in which these tasks depend on their own. I ran an online experiment to provide additional evidence of the plausibility of the individual cognitive mechanisms that contribute to my theory. I describe the experimental design and results in detail in Online Appendix C. The experiment’s results suggest that participants search more systemically and achieve better performance after the type of exogenous shock that I focus on, when lateral and vertical interdependencies are made more salient to them.
While the experiment features a highly stylized setting that allows me to manipulate only a subset of the mechanisms operating in DOTA 2, it provides additional evidence that the difficulty of understanding interdependencies among choices can lead to systemic inertia in search tasks even when such difficulty is not reinforced by division of labor among specialists. The results in DOTA 2—especially the effects of communication among members—suggest that the division of labor in collective settings accentuates this difficulty. The experiment also provides additional suggestive evidence that my mechanisms of interest can have a causal impact on decision-making performance.
Discussion and Conclusion
In this article, I set out to investigate organizational reactions to change as multilevel processes operating over a hierarchy of choices. I argued that interdependencies among choices can have a dual impact in the face of change: they increase the likelihood that systemic search will be beneficial, but they also make it difficult for decision makers to understand these benefits. On the one hand, interdependencies among choices at different levels enable change to have ripple effects on the viability of systemic processes even when the change’s immediate impact is confined to specialized tasks. On the other hand, understanding interdependencies well enough to foresee these ripple effects is a significant challenge for decision makers. Especially when labor is divided among different specialists, making sense of change requires understanding complex causal paths through interdependencies among individuals who have limited ability to perceive these paths. As a result, organizations may not recognize the value of systemic search unless they benefit from information-processing structures that help their members understand their own interdependencies and make sense of exogenous change in light of these interdependencies.
I found support for my theory using a novel dataset on esports teams. When game updates, through their impact on heroes, indirectly affected the viability of strategies, teams reacted by experimenting modularly with new heroes but not systemically with new strategies. However, in line with my theory, teams did experiment with new strategies as a response to updates that directly affected the viability of strategies and removed the need to make sense of interdependencies between heroes and strategies. My analyses also revealed several insights beyond my predictions. Not only did teams experiment less with new strategies than with new heroes in the face of game updates that could indirectly affect the viability of their strategies; teams were even less likely to experiment with new strategies after these updates than during the stable periods preceding them. The evidence also suggests that different information-processing structures facilitate systemic search to different extents depending on the sensemaking challenges posed by exogenous change. Lateral communication among members seemed to promote systemic search when recognizing its value required making sense of complex interdependencies, while relying on formal coordinators accelerated systemic search only when the need for it was already apparent. Finally, the analyses predicting performance suggest that experimenting with new strategies led to performance improvements only in the face of changes that made it challenging to recognize the value of this experimentation.
My main aim in conducting this study was to provide a micro-level explanation, backed by micro-level evidence, for failures of collective adaptation that are observed in organizations of various types and sizes. Adaptation to exogenous events happens in various organizations, from small teams to industry ecosystems, that feature specialists who need to find solutions to their own tasks and collective processes integrating the work of these specialists—whether these specialists are individuals, groups, or organizations (e.g., Karim and Mitchell, 2000; Burke et al., 2006; Kapoor and Adner, 2012; Dattée and Barlow, 2017). Hence, the generalizability of my findings to other contexts should be of particular interest. Esports teams engage in a somewhat peculiar activity (playing fast-paced video games in small teams) and involve a specific demographic (young adults, mostly men). They also rely on means of coordination—lateral communication and reliance on formal coordinators—that are only a partial subset of the information-processing structures found in organizational contexts. In larger organizations, coordination structures may involve complex combinations of the ones I study and may themselves comprise multiple levels (Gaba and Joseph, 2013; Puranam, Alexy, and Reitzig, 2014; Dattée and Barlow, 2017). Investigating multilevel search in such settings may yield significant future insights.
While research in other settings will be valuable, some aspects of my study may provide confidence in the generalizability of the mechanisms I describe. If my results are indeed due to individuals overlooking their interdependencies, then focusing on small teams should provide conservative results. DOTA 2 is quite complex by the standards of most video games, but the organizations that compete in this game are relatively simple by the standards of organizational research: I study teams of five people in which each member’s behavior is visible to all others. In larger organizations with more-complex sets of interdependencies, evaluating the systemic consequences of exogenous shocks may be even more difficult (Henderson and Clark, 1990; Aggarwal and Wu, 2015; Marino et al., 2015). In fact, some empirical findings in larger organizational and interorganizational settings are coherent with the mechanisms I describe (Karim and Mitchell, 2000; Kapoor and Adner, 2012; Feldman, 2013; Eggers and Park, 2018). My study adds to recent work that used micro-level data to generate insights about the impact of interdependencies among choices in these settings (Adner and Feiler, 2019).
By uncovering these insights, my study makes contributions to several related literatures. To the broad literature concerned with organizational adaptation to exogenous events (Eggers and Park, 2018), I contribute a novel explanation for organizational failures to adapt in the face of change: decision makers may recognize change as relevant to their organization, but only understand a few isolated implications of that change, overlooking its more systemic consequences for the viability of their organization’s processes. Some studies have hinted at organizations’ difficulties reacting to the full extent of change when they confront a highly interdependent task environment (Henderson and Clark, 1990; Siggelkow, 2001). I extend their insights by specifying a set of mechanisms through which interdependencies generate these difficulties and by providing evidence of these mechanisms in a large micro-level dataset. My theory is distinct from prior explanations for failures to adapt in at least two ways. First, it extends the range of theoretical mechanisms used to explain these failures: the mechanisms I describe can preclude organizations from adapting successfully even when none of our usual explanations apply, such as lack of awareness of change, lack of motivation to adapt, or lack of resources. Second, my theory helps us understand how plasticity and rigidity can coexist in the face of change. Scholars have noted that most theories of change emphasize either agency and plasticity when explaining successful adaptation to change or rigidity and lack of agency when explaining failed adaptation (Levinthal and Rerup, 2006). My study embraces a middle ground: organizations do not necessarily fail to adapt by not searching enough; they can fail to adapt by missing the forest for the trees and searching at the wrong level of their task structure. Organizational members can exert agency in reacting to the consequences of change in their working environment, but the structure of interdependencies in their organization may keep them from recognizing how these consequences span the different levels of their task environment.
My work also advances the literature on organizational search in complex systems, whose insights I borrowed to conceptualize search as a multilevel process (Ethiraj and Levinthal, 2004; Siggelkow and Rivkin, 2009; Csaszar and Levinthal, 2016). First, my work extends models of multilevel search, which focus on the performance consequences of search at different levels, by investigating the determinants of whether organizations do search at different levels when faced with change. In doing so, my study answers recent calls to consider both the cognition and the organization of multiple agents in theorizing patterns of organizational search (Knudsen and Srikanth, 2014). To explain the challenges of search for organizations facing the type of change I investigate, my theory combines insights about the individual cognitive challenge of making sense of complex interdependencies and about interactions between human decision makers in a complex system whose modules are allocated to different specialists. By accounting for coordination among specialists, my theory also yields predictions about the effects of different information-processing structures, which have been a topic of interest in models of search (Rivkin and Siggelkow, 2003; Gavetti, 2005; Fang, Lee, and Schilling, 2010; Mihm et al., 2010; Csaszar, 2013; Knudsen and Srikanth, 2014) but had not been incorporated in models of multilevel search. 10 My work also complements recent research that helps us understand how the mental models held by decision makers before exogenous change occurs can facilitate or hinder search after its occurrence (e.g., Aggarwal, Posen, and Workiewicz, 2017; Clement and Puranam, 2018): when decision makers accurately perceive how their tasks depend on each other in carrying out systemic processes, they are better equipped to understand when an exogenous change makes these processes obsolete.
Second, my study makes an empirical contribution to the literature on organizational search by providing what I believe to be the first quantitative investigation of multilevel search in a large micro-level dataset. My work advances the empirical investigation of arguments derived from formal models of search, which has been notoriously difficult. Recent studies have relied on laboratory experiments of individual decision makers to investigate arguments derived from models of search in complex landscapes (e.g., Billinger, Stieglitz, and Schumacher, 2013; Billinger et al., 2021), and some recent studies used naturally occurring data on organizations to investigate predictions from models portraying coupled search and exploration/exploitation tradeoffs (e.g., Marino et al., 2015; Stan and Puranam, 2016). In several ways, esports data allow me to combine the advantages of laboratory experiments and of archival data. Esports teams feature high-powered incentives, which are typically easier to find in naturally occurring data than to reproduce in experiments. Esports data also allow me to observe very precisely the micro-level dynamics of collective behavior and information processing within teams, which are typically much easier to observe in the laboratory than in naturally occurring data. This granularity allowed me to investigate multilevel search, which by definition requires the observation of choices at different levels, and to analyze how information-processing structures impact this search—with some caveats. 11 Video-gaming data, in DOTA 2 and in other games embodying different task environments, may provide other opportunities for investigating empirically the dynamics produced by theoretical models of search. More generally, future empirical studies may benefit from using multilevel behavioral data to describe organizational reactions to change: as my results suggest, organizations can appear either rigid or flexible in the face of change, depending on the level of analysis observed.
Third, my results regarding the performance consequences of search suggest interesting avenues for studying the impact of search and learning in different types of competitive environments. Organizational search has often been modeled as a battle against the environment (e.g., Levinthal, 1997; Rivkin and Siggelkow, 2003; Knudsen and Srikanth, 2014): organizations face a landscape in which different resources or behaviors are assumed to be valuable, and they evolve within this landscape until they reach an adequate level of performance that does not depend on the actions of other organizations in the landscape. But many contexts do not meet this assumption: organizations often operate in settings in which the effectiveness of their choices depends on their competitors’ actions. I found that teams that search systemically as a response to exogenous change increased their chances of success but only when the systemic impact of change was indirect. In other words, experimenting with new strategies gave a competitive advantage to teams only when the value of doing so was non-obvious. My results suggest that processes leading to better fit with the environment do not necessarily lead to better performance than that of competitors: when these processes are easy to recognize, they are unlikely to yield competitive advantage. Being able to navigate more-ambiguous change is more likely to generate positive outcomes. Future research may investigate these dynamics under intermediate regimes whereby competition is neither absent, as in many models of search, nor frontal, as in esports.
Finally, my study may generate some contributions to research focused specifically on teams. Team adaptation has been a topic of interest (Burke et al., 2006), as scholars have studied teams’ reactions to temporary crises (e.g., Waller, 1999; Bechky and Okhuysen, 2011; Uitdewilligen and Waller, 2018) and to non-temporary environmental shifts (Edmondson, Bohmer, and Pisano, 2001; Uitdewilligen, Waller, and Pitariu, 2013). The distinction between modular and systemic behavior in teams may prove useful for this research. Scholars have pointed out that while teams have been described as complex adaptive systems (Arrow, McGrath, and Berdahl, 2000), complex-systems perspectives have been slow to influence teams research (Ramos-Villagrasa et al., 2018). Research on team adaptation has made progress in acknowledging the complexity of information-processing and decision-making structures, including multilevel decision making managed by coordinators or leaders (Uitdewilligen and Waller, 2018). My analysis suggests that considering a team’s task environment itself as a complex system with different levels can reveal a stark contrast between local and global adaptive tendencies. Insights from my study, as well as prior models of multilevel search, may be usefully integrated into future research on team adaptation.
Overall, this study yields a set of findings, enabled by the exceptional granularity with which collective behavior can be observed in esports, that advance our understanding of how organizations search and adapt to change within complex systems. This natural laboratory allowed me to test arguments that bridge levels of analysis and involve predictions that would be challenging to test in most naturally occurring data. My hope is that this study paves the way for more research using micro-level field data to investigate the mechanisms underlying our theories of organizational search, adaptation, and coordination, as well as studies in more-traditional contexts where the mechanisms uncovered here may help explain the evolution of collective behavior.
Supplemental Material
sj-pdf-1-asq-10.1177_00018392221136267 – Supplemental material for Missing the Forest for the Trees: Modular Search and Systemic Inertia as a Response to Environmental Change
Supplemental material, sj-pdf-1-asq-10.1177_00018392221136267 for Missing the Forest for the Trees: Modular Search and Systemic Inertia as a Response to Environmental Change by Julien Clement in Administrative Science Quarterly
Footnotes
Acknowledgements
This manuscript greatly benefited from the repeated feedback of Daniel Levinthal and Phanish Puranam while I was a graduate student. I also gratefully acknowledge the helpful suggestions of Sarath Balachandran, Arianna Marchetti, Victoria Sevcenko, and Jesper Sorensen on prior versions of this paper, as well as the comments made by attendees of seminars at INSEAD, the Southern Denmark University, the Hong Kong University of Science and Technology, the University of Washington, Northwestern University, the University of Michigan, the University of Texas at Austin, Stanford University, Dartmouth College, London Business School, New York University, the University of Minnesota, the University of Pennsylvania, Harvard University, the University of California (Berkeley), and Yale University. I thank Sébastien “Ceb” Debs and Jonathan “PimpmuckL” Liebig for spending time with me on several occasions to help me understand DOTA 2 and for introducing me to industry experts, whom I also thank for their time. I thank Howard Chung, Albert Cui, Martin Decoud, and Ben Steenhuisen for their help and advice on how to process DOTA 2 replay data. I also appreciate the constructive feedback of Associate Editor Maxim Sytch and three anonymous reviewers.
3
MOBA games emerged from a single community-managed game, Defense of the Ancients (DOTA), in 2003. Several commercial platforms now support similar versions of the game, e.g., DOTA 2, League of Legends, Heroes of the Storm, and Smite. The DOTA 2 platform provides the most detailed data. The quotation is from the Free to Play 2014 documentary film.
4
These include players from some of the world’s most successful teams and players from slightly less successful professional teams. Since the beginning of this study, the players I interviewed have earned a combined prize pool of more than USD $30 million with their respective teams.
5
Specifically, a certain hero’s mobility ability may allow it to instantly jump across a large distance, while another hero’s mobility ability only allows it to run faster for two seconds. Similarly, a hero’s destabilizing ability may involve drawing a large circle on the map and inflicting a certain amount of damage to opponents over time if they stay within the circle, while another’s may involve drawing a small line that inflicts some damage on opponents that cross it.
6
The “safe-lane carry” (also referred to as “position 1”) role entails gathering resources and inflicting the most damage in decisive fights. The “mid-laner” role (or “mid-lane carry,” position 2) fulfills a relatively similar role but gathers resources in a different area of the map. The “off-laner” (or “off-lane carry,” position 3) seeks to bear the brunt of the opponents’ attacks and defend buildings, as well as initiating fights with the opponent. The “soft support” (position 4) is a more flexible role that may entail gathering uncontested resources (in the “jungle”), leading early attacks against the opponent’s heroes and buildings, or defending against the opponent’s attacks. The “hard support” (position 5) provides support for teammates, especially protecting the safe-lane carry. Role allocations tend to be stable over time: some players specialize in the “safe-lane carry” role, others in the “hard support” role, etc.
7
I use the word “experimentation” instead of “search” in the empirical section to more closely reflect the fact that teams in DOTA 2 engage in online search by directly experimenting with new choices and associated behaviors, rather than offline search through thought experiments. While it seems reasonable to expect my theory to apply similarly to offline search, this remains an empirical question.
8
I chose this time frame because collective behavior in DOTA 2 unfolds rapidly through collaboration among players in specific regions of the map. Without this restriction, different abilities would be recorded in the same sequence despite being part of separate behaviors exhibited by subgroups of players in different locations.
9
I show effects week by week for 10 weeks instead of a longer period so that these patterns reflect the average of all game updates in my data (i.e., all updates in the dataset happened at least 10 weeks after the previous update, but not all of them happened at least 14 weeks after the previous one). I observed a similar pattern of results over 14 weeks, as shown in Figures A2a and A2b in Online Appendix A. In all cases, the rate of change reverts to its post-update average, but the trend does not get inverted (e.g., after changes with indirect systemic impact, the rate of change never exceeds its pre-shock average regardless of the time window selected).
10
Within the related literature on organization design, my findings about formal coordinators both reinforce and complicate recent research showing how formal coordinators facilitate adaptation (Stan and Puranam, 2016; Valentine, 2018). My results suggest that coordinators may help implement new procedures once the need for them has been recognized. But coordinators are no less subject to bounded rationality than are the members whose work they coordinate (Clement and Puranam, 2018). Hence, recognizing the need for new procedures in the first place may require sensemaking among a broader set of members.
11
Some of my measures still constitute imperfect proxies for the constructs I theorize about. My measures of systemic search rely on the measure of fast-paced sequences that differ from one game to another depending on a team’s strategy, rather than directly measuring the strategies themselves. I relied extensively on interviews and field observations to confirm that these measures had face validity, and ran additional analyses based on sequence clustering (reported in Online Appendix A) to improve confidence in their validity. However, the translation of strategies into sequences of actions includes statistical noise, especially when the behavior of opponents during games directly disrupts the implementation of strategies. Another limitation concerns my measures of information-processing structures, which rely on a game-specific communication method (pings). Future research could improve significantly upon these measures by measuring—and perhaps manipulating—oral communication, not only during the implementation of collective processes but also during the sensemaking work that leads to choosing these processes. There are also limitations to my ability to show the performance consequences of systemic search in DOTA 2. While my main analyses predicting search rely on exogenous game updates outside of teams’ control, I predict performance based on search variations that result from endogenous reactions by teams. These reactions may correlate with factors outside the scope of my analysis that themselves affect performance—a classic issue with using organizational performance as a dependent variable (March and Sutton, 1997). Hence, despite these analyses, my study still requires readers to accept the premise that game updates affecting heroes in DOTA 2 did have systemic consequences for the viability of teams’ strategies but that professional teams could not recognize these consequences. Several aspects of my study should provide confidence in this assertion. First, some of the qualitative evidence I report suggests that industry participants recognized, by the end of my observation period, that experimenting with new strategies was the correct reaction and that many teams had failed to understand it. Second, while still potentially spurious, the main results of my performance regressions concern a difference in the effects of recent experimentation on performance between stable and post-update periods. I expect that the set of factors that may spuriously generate this difference is smaller than the set of factors that could affect performance more generally. Finally, the experiment I report in Online Appendix C provides additional evidence that the mechanisms I describe can affect adaptive performance in the face of change: although my experiment was mainly designed to isolate the individual cognitive aspect of my mechanism of interest, it also generates performance differences in a stylized setting where outside factors are not at play.
Author’s Biography
Julien Clement is an assistant professor of organizational behavior at the Stanford Graduate School of Business (