Abstract
The objective of this article is to improve our understanding of preferences in experienced-based choice. Positioned within the framework of naturalistic decision making, this article responds to the recent call to complement the examination of experience-based choice with studies of cognition in the “wild.” We document an exploratory field study that uses applied cognitive task analysis (ACTA) to examine financial day traders’ preferences. Providing real-world examples, our study illustrates how day traders construct their understanding of gains relative to losses and emphasizes the relevance of prospect theory for understanding the asymmetry of human choice. The fourfold pattern of preferences as studied in the wild is risk seeking for medium- and high-probability gains, risk averse for small-probability gains, risk averse for small-probability losses, and risk averse for medium- and high-probability losses. Our results differ from the fourfold pattern of preferences exhibited by experience-based choice when studied in the laboratory. The implications of this work for prospect theory and the distinction between “experience through learning” and “experience through professional training” are discussed alongside the merits of the ACTA technique for professional expert domain-based knowledge elicitation.
Keywords
Introduction
Modern decision science is undergoing a process of reframing. Although decision research was once split by distinct preferences for what it ought to be, recognition of the need for a more unified and complete understanding of decision making has called for an end to these divisions (Dalal et al., 2010; Gore, Banks, Millward, & Kyriakidou, 2006; Hertwig & Erev, 2009; Kahneman & Klein, 2009; McAndrew, Banks & Gore, 2008; McAndrew, Gore & Banks, 2009). Examples of what some have claimed decision making ought to be include a discipline based on normative ideals and prescriptive standards, one providing descriptive accounts of cognition studied in the “wild,” and the study of systematic deviations in the form of heuristics and biases. Recent reframing by Kahneman and Klein (2009) has focused on the intersections between naturalistic decision making (NDM) and the heuristics and biases approach. Parallel to this discussion, Rakow and Newell (2010) recommend the importance of researching a new frame of experience-based choice within the wider context of decision research. It is clear to appreciate that the conditions for advancement are ripe.
This article draws insights from the field of NDM to better understand decision makers’ preferences as investigated within the paradigm of experience-based choice. We open with an examination of recent findings relating to experience-based choice. Noting the call to complement this with studies of “cognition in the wild,” the authors highlight the important contributions of the NDM community. We then outline an exploratory field study that advances our understanding of preferences in experience-based choice in the wild. We conclude by considering the implications of our findings for future research within the decision sciences.
Experience-Based Choice
Experience-based choice has rapidly become a feature of decision science research. Recent research has found decision makers’ preferences to consistently differ depending on the format of choice presented. This has been marked by studies documenting the observation of systematic differences between decisions based on description and those based on experience (Camilleri & Newell, 2009b; Hertwig & Erev, 2009; Newell & Rakow, 2007; Rakow & Newell, 2010).
The first, decision from description, is a format whereby every possible outcome and decision is specified at the outset (Hertwig & Pleskac, 2008). Consider, for example, a game whereby you are asked to choose between (a) a 90% chance of winning $0 and a 10% chance of winning $10 or (b) a guaranteed $1. People tend to attribute more weight to small-probability events than objectively warranted and so prefer (a), the gamble. That is, people are risk seeking for small-probability gains and high-probability losses (people tend to prefer a 10% chance of $10 to $1 for sure) and risk averse for high-probability gains and small-probability losses (most people prefer $9 for sure to a 99% chance of $10). Decisions emerge consistent with prospect theory’s fourfold pattern of choice: Participants are risk averse for high-probability gains, risk seeking for high-probability losses, risk averse for small-probability losses, and risk seeking for small-probability gains (see Table 1).
The Fourfold Pattern for Decisions From Description and Experience
Source. Adapted from “Degrees of Uncertainty: An Overview and Framework for Future Research on Experience-Based Choice,” by T. Rakow and B. R. Newell, 2010, Journal of Behavioral Decision Making, 23(1), p. 3. Copyright 2009 by Wiley (www.interscience.wiley.com). Adapted with permission.
Note. Speculation is indicated by the presence of question marks.
The second format of choice, decision from experience, is defined as one whereby personal observation and feedback from the environment guide the outcomes generated and assessments of their relative probabilities. When people are presented with the task in this format, preferences for small-probability gains and losses tend to be the converse, with people displaying a preference for the sure thing (b) (Barron & Erev, 2003; Erev et al., 2009; Hau, Pleskac, & Hertwig, 2009; Hau, Pleskac, Kiefer, & Hertwig, 2008; Hertwig, Barron, Weber, & Erev, 2004; see Table 1). Risk aversion for a gain is attributable to underweighting of the rare event. Examination of preferences for high-probability gains and losses for decisions from experience has been more limited, although speculation has suggested that there is little difference when compared to preferences in the condition of description (Rakow & Newell, 2010).
Explanations for the differences between these formats of choice have included sampling bias (one is more likely to undersample than oversample experiences of rare events), recency effects (giving more weight to observations experienced most recently; although note Barron and Yechiam’s [2009] findings that call into question the notion of recency effects with rare events), and the phenomenon of representation (an encoding distortion of the outcome distribution prior to choice; Camilleri & Newell, 2009b; Hertwig & Erev, 2009). Commenting on these findings, Hertwig and Erev (2009) note the results are to be viewed not as contradictory but as an illustration of how the mind operates in two different informational environments:
In other words, one should not play off research on description-based and experience-based behavior—their contrast is enlightening. However, to better understand how people make decisions with incomplete and uncertain information “in the wild”, there is a need to study experiential choices that are often representative of people’s actual choices. (p. 522)
Rakow and Newell (2010) echo this call for a framework that places experience-based choice within the wider context of decision research, suggesting that rather than asking what happens in experience-based choice and what happens in described choice, one may do better to ask what is the role of experience and what part does the description play? They suggest this approach is imperative to understanding properties of the external choice environment, types of learning experience, types of choice, and properties of the internal representation of payoffs (see Rakow & Newell, 2010, for a comprehensive review of future directions). It is through such an examination that we position this article in the hope that this exchange is constructive to both communities and researchers working in the field and the laboratory.
Cognition in the Wild
Originating as a critique of disembodied views of cognition resulting from study within the artificial setting of the laboratory, the concept “cognition in the wild” was coined by Hutchins (1995a). By locating cognition in context and by focusing on the interactions between multiple agents and material artifacts, the study of cognition in the wild would give rise to the situated and socially constituted aspects of cognitive activity.
A new perspective for cognitive science guided by anthropology and cognitive theory, decision making has been studied in complex settings, such a shipboard navigation (Hutchins, 1995a), aviation (Hutchins, 1995b, 2000; Hutchins & Klausen, 2000) and steam plant operations (Halff, Hollan, & Hutchins, 1986). Although the central thesis of Hutchins’ work highlighted the necessity to complement experimental investigations with experiential studies of cognition, it is important to observe that this perspective emerged in parallel to NDM, a perspective that arguably combines quasi-experimentalism and ethnography to study cognition in the wild, to which we now turn.
NDM
To some decision research communities, NDM is a relatively nascent research area (Hodgkinson & Starbuck, 2008) focusing on the study of cognition in real-world environments marked by ill-structured problems; uncertain, dynamic environments; shifting, ill-defined, or competing goals; action-feedback loops; time stress; high stakes; multiple players; and organizational goals and norms (Orasanu & Connolly, 1993). Regarded as “the way people use their experience to make decisions in field settings” (Zsambok, 1997, p. 4), NDM has been received as a complementary mode of inquiry to the more classic behavioral decision-making paradigm and research associated with heuristics and biases (Connolly & Koput, 1996; Gonzalez, 2001; Kerstholt & Ayton, 2001; LeBoeuf & Shafir, 2001; Kahneman & Klein, 2009). As Gore et al. (2006) noted, the NDM movement involves the close examination of heuristics and the study of expertise to learn about more powerful domain specific heuristics. NDM researchers look for what people do right rather than what they do wrong, which is the traditional form of inquiry inspired by Kahneman and colleagues (Kahneman, Slovic, & Tversky, 1982; Tversky & Kahneman, 1974).
In other words, much of behavioral decision making and heuristics and biases research has looked at errors in cognition focusing on judgments for which participants have very little if any expertise (Strack & Mussweiler, 1997; Tversky & Kahneman, 1974). NDM’s focus on the positive features of expert cognition is positioned in response to the inappropriateness of generalizing insights from the study of novices to experts in more realistic settings. Lipshitz, Klein, Orasanu, and Salas’s (2001) focus article “Taking Stock of Naturalistic Decision Making” provides a good insight into the field and documents the boundaries of the framework. For more recent discussions of the intersections and differences between NDM and the heuristics and biases approach, see Kahneman and Klein’s (2009) article “Conditions for Intuitive Expertise: A Failure to Disagree.”
As outlined in Kahneman and Klein (2009), NDM research emphasizes two skills: the ability to recognize patterns of previously experienced cues in new situations and the ability to notice and extract cues in unrecognized situations by drawing from past experience and building stories (G. Klein, 1997; Lipshitz & Pras, 2005). Recognition-primed decision making (G. Klein, 1993, 1997) is one model of how cues from sensory stimuli can trigger the problem-solving process. Accordingly, the NDM community has made significant progress in examining decision making as an experientially driven process through a consolidation of theory and real-world applications (Johnston, Driskell, & Salas, 1997; Hoffman, 2006; Montgomery, Lipshitz, & Brehmer, 2005; Mosier & Fischer, 2011; Schraagen, Militello, Ormerod, & Lipshitz, 2008; Wong & Stanton, 2009).
Despite this progress, the choice in domains of application has scarcely deviated from early studies of firefighting, military command and control, and nursing (Drury & Darling, 2008; G. Klein, Calderwood, & MacGregor, 1989; Militello & Lim, 1995). Accordingly, this work investigates the relevance of an NDM lens to a further field of study: day traders in financial markets. This builds on recent discussions examining the synergies between NDM and organizational decision making (Gore et al., 2006: Lipshitz, Klein, & Carroll, 2006; McAndrew & Gore, 2010) and the relevance of Orasanu and Connolly’s (1993) typology of eight defining characteristics of NDM to this domain.
The uncertain and dynamic environments experienced by day traders are epitomized by the comparatively fast pace of markets in which they work. “Real” decision problems seldom present themselves in an entirely complete and orderly form. Dealing with ill-structured problems, day traders expend effort (a) generating hypotheses about the situation, (b) developing options as appropriate responses, and (c) recognizing that the situation is one permissible of choice. Shifting, ill-defined, or competing goals also denote the information environments within which traders operate. Although the overarching objective is to create profit, the natural fluctuation and pattern of markets may necessitate changes in trading strategies (i.e., trading ranges or jobbing) and positions (i.e., long vs. short).
NDM purports that “real” decision environments are arranged as a series of temporally segregated events as opposed to one discrete decision at a specific point in time. The use of action-feedback loops to rectify early mistakes is a characteristic of corrective action that maps directly onto the work of day traders. This is amplified by a rate of feedback that is almost immediate when trading intraday positions. The high stakes associated with such transactions are also significant because of the uncertainty and risk associated with the short-term volatility of markets, which necessitate instantaneous calls of judgment and also bring a significant amount of time stress. The financial market as a reflection of an underlying psychological sentiment felt by market players at any one point in time fits with Orasanu and Connolly’s (1993) need to appreciate the cognitive activities of multiple players. Finally, the suggestion that the organization may establish general goals, rules, and standard operating procedures is reflected in the specification of the size of trading fund and stop losses.
In spite of this congruence, researchers have not, until now, examined day trading from an NDM perspective, favoring instead a behavioral finance lens. Examining the role of experience in financial markets, behavioral finance researchers have displayed a tendency to use laboratory settings to compare expert versus student performance. See, for instance, Anderson and Sunder’s (1995) comparison of laboratory market behavior between expert commodity and stock traders and MBA students. Stock price forecasting using probabilistic judgments of the likelihood of price increases or decreases (Staël von Holstein, 1972) is also characteristic of this approach although it has had limited success. As Mieg (2001) notes, experimental tasks of this nature lack the realism necessary for informative conclusions, as in practice, market prices are judged directly, not in terms of categorical judgments or probabilities. Research of this type also neglects the role of social networks and intercorrelated markets as a basis of informational leverage.
A growing trend has been the use of social psychological approaches, such as personality profiling to understand risk propensity (Fenton-O’Creevy, Nicholson, Sloane, & Willman, 2005) and the application of concepts such as emotion regulation to understand decision-making behavior and performance (Fenton-O’Creevy, Soane, Nicholson, & Willman, 2011). This work has aided understanding of the process by which day traders acquire expertise: anticipation, encounter, adjustment, and stabilization. Fenton-O’Creevy et al. (2005) suggest that the most pertinent episodes shaping trading styles include early experiences of loss and gain, emotional experience of the former, and beliefs about the market that are based on cause and effect and decision outcomes. The acknowledgement, however, that risk propensity is only one factor in traders’ behavior (Fenton-O’Creevy et al., 2005) suggests value in an approach that gives credence to context, training, and socialization. Mieg (2001), however, claims that expertise within this domain is not experience driven. Denouncing a strong form of expertise rooted in individual experience, Mieg displays a theoretical preference for a weak form of expertise that results from the use of information technologies. Key to this proposition is his suggestion that day traders lack insight into the complexity that drives the market, a claim refuted in this article.
Using an intrinsically task-focused approach that is sensitive to context, we seek both theoretical advancement and practical insights for training and development within this domain. It is also anticipated that improved understanding of how attitudes toward risk concerning gains and losses deviate from studies of description and experience-based choice in the laboratory when studied “in the wild” will hold significant implications for other domains beyond the immediate field of application.
This article opened with the proposition that our understanding of experience-based choice might be advanced through the study of cognition in the wild. The logic is such that (a) much of prospect theory is based on a description-based research paradigm in laboratory settings, (b) laboratory research based on an experience-based paradigm suggests effects different from those demonstrated by description-based paradigms, and (c) field research following an experience-based paradigm might reveal effects that meaningfully differ from both description-based research and experience-based laboratory research. We see promise in the use of NDM as a lens to focus on the positive features of expert cognition and suggest that researchers have much to gain from the use of applied cognitive task analysis (ACTA).
Method
We adopted a qualitative methodology aimed at exploring decision makers’ preferences using a ground-up approach. Framed by NDM, this examination of cognition in the wild was conducted in a number of day-trading firms within the U.K. finance industry (McAndrew, 2008).
Participants
In-depth interviews were conducted with 8 day traders (8 male; mean age = 34.8 years, SD = 5.50). This exceeded Militello and Hutton’s (1998) recommendation that three to five subject matter experts usually exhaust the domain of analysis. Participants were recruited from four U.K. trading firms: one bank (n = 1), two energy providers (n = 4), and one brokerage firm (n = 3). These firms had divisions that traded with a proprietary element, trading for a direct profit from the market rather than earning commission from processing trades.
All participating firms were authorized and regulated by the Financial Services Authority. Participants had worked in the industry for an average of 10.9 years (SD = 4.22) and had acquired an average of 4.8 years (SD = 2.97) of experience within their current position. Day traders interviewed occupied positions ranging from director of trading to more entry-level positions. This range of positions does not, however, indicate novice capabilities. The one junior trader included in this sample started trading 3 years earlier from a fund of €100,000 and provided recent examples of trades that spoke of stop losses in the region of $13,000, implying a current fund of $1 million to $2 million.
Materials
ACTA (Militello & Hutton, 1998) is a practitioner-focused metamethod of interest to applied psychologists working within the field of cognition (see Crandall, Klein, & Hoffman, 2006, for discussion of the strengths and weaknesses of cognitive task analysis in relation to other knowledge elicitation methods, e.g., interviews, observation, process tracing, and conceptual approaches). Used to elicit day traders’ expertise in experience-based choice, ACTA is a set of knowledge elicitation and representation techniques intended to assist in identifying the key cognitive elements required to perform a task proficiently. The ACTA techniques were developed to complement each other, each tapping into different aspects of cognitive skill.
The first technique, the task diagram, provides the interviewer with a broad overview of the task in question, summarized in three to six steps. With the task diagram, this stage allows the participant to identify the area that demands complex cognitive skills.
The second technique, the knowledge audit, focuses on the subtask identified in Stage 1 that presented significant cognitive difficulty. It allows the interviewer to review the aspects of expertise required for successful task completion using a set of eight probes (see Table 2). These probes are based on knowledge categories that characterize expertise (Militello & Hutton, 1998): diagnosing and predicting, situation awareness, perceptual skills, developing and knowing when to apply tricks of the trade, improvising, metacognition, recognizing anomalies, and compensating for equipment limitations. As aspects of expertise are elicited, they are probed for further detail and concrete examples (e.g., In this situation, how would you know this? What cues and strategies are you relying on?). It is this ability to probe a variety of knowledge types (e.g., perceptual skills, mental models, metacognition, declarative knowledge, analogues and typicality, and anomalies; G. Klein & Militello, 2005), that provides a more complete description of cognitive processes and sets the knowledge audit aside from structured interview and task analysis techniques.
Applied Cognitive Task Analysis Probes
Source. Adapted from “Applied Cognitive Task Analysis: A Practitioner’s Toolkit for Understanding Cognitive Task Demands,” by L.G. Militello and R. J. B. Hutton, 1998, Ergonomics, 41(11), p. 1622. Copyright 1998 by Taylor & Francis (www.tandfonline.com). Adapted with permission.
A key difference between the knowledge audit and standard forms of mental model assessment is that the knowledge audit draws directly from the research literature on expert–novice differences (Dreyfus & Dreyfus, 1986; Hoffman, 1992; G. Klein & Hoffman, 1993; Shanteau, 1985) and critical decision method studies of expert decision making (Kaempf, Klein, Thordsen, & Wolf, 1996; G. Klein et al., 1989). This encourages participants to identify why elements of the task may present a problem to inexperienced individuals (e.g., In what ways would this be difficult for a less-experienced person? What makes it hard to do?), emphasizing the applied utility of this method.
The knowledge audit has been developed to capture key aspects of expertise, improving and streamlining data collection and analysis. It allows the interviewer to explore and probe issues such as situation assessment, potential errors, and how a novice would respond to the same situation, shedding light on the content and structure of knowledge. Whereas experience-based choice research has employed verbal and nonverbal assessment probes leading to inferences about the forms of data underlying mental representations (see Camilleri & Newell, 2009a), ACTA is an entirely verbal method. The open-endedness of ACTA’s probes permits participants to respond using probabilistic terms or nonverbal numerical representation systems, crucial for studying cognition as it occurs in the wild.
An optional third stage, the simulation interview, allows the interviewer to better understand participants’ cognitive processes within the context of a challenging scenario. This can be useful in highlighting differences in expert–novice perspectives and in developing training and system design recommendations.
Finally, a cognitive demands table merges and synthesizes data from all participants across the stages. This is intended for practitioner use to focus the analysis and identify common themes in the data. See Militello and Hutton (1998), Crandall et al. (2006), or the ACTA multimedia instructional CD (Militello, Hutton, & Miller, 1997) used in preparation for these interviews for a more detailed review of this approach. Some of the limitations associated with ACTA are highlighted in our discussion. As a method, however, it offers promise, providing provocative insight into human cognitive processes.
Researchers have successfully used ACTA in empirical work to understand expertise in a diverse range of areas, including weather forecasting (Hoffman, 1992, 2006; Hoffman & Militello, 2008; Hoffman, Shadbolt, Burton, & Klein, 1995; Hoffman, Trafton, & Roeber, 2006), clinical nursing (Militello & Lim, 1995), firefighting (G. Klein et al., 1989), recruitment (Gore & Riley, 2004; Gore & McAndrew, 2009), and military command-and-control operations (Drury & Darling, 2008). ACTA has provided a significant development in available tools and techniques for the identification of training needs in knowledge-based work, providing instructional designers with clearer guidelines when designing training for cognitively demanding tasks in domain specific areas.
Reliability and validity
With no established metrics, cognitive task analysis methods have been open to critique on the grounds of reliability, validity, and falsification. With arguably appropriate standards for one model of science only (one in which scientific inquiry prevails), this work is aligned with the perspective that there exists “particular ways of warranting validity claims rather than as universal, absolute, guarantors of truth” (Mishler, 1990, p. 420). This more pluralistic interpretation of validity has been considered appropriate for NDM (Lipshitz et al., 2001). Mishler (1990) suggests that inquiry-led research (such as NDM) ought to be evaluated using the criteria of credibility and transferability. Credibility is defined as the degree to which the study’s findings and conclusions are warranted, and transferability as the extent of case-to-case translation.
Recognizing Kaplan’s (1964) proposition that research methods should drive the selection of evaluation criteria, Militello and Hutton (1998) have attempted to establish the reliability and validity of ACTA using alternative indicators. Questions of validity included the following: Does the information gathered address cognitive issues? Does the information gathered deal with experience-based knowledge as opposed to classroom-based knowledge? Do the instructional materials generated contain accurate information that is important for novices to learn? Reliability was ascertained by determining whether ACTA was able to consistently generate relevant cognitive information across participants. Findings indicated high levels of validity and reliability, with modal statistics occurring in the range of 90% to 95%. This is in line with interrater reliability statistics for this study, which were in the range of 78% to 85% for the task diagrams, knowledge audits, and cognitive demands table.
Procedure
Each interview lasted approximately 2 hr. The interviews opened with a discussion of the types of financial instruments and assets traded. When familiar with the domains of expertise, we completed Stages 1 and 2 of ACTA. During Stage 1, each participant was asked to outline a broad overview of his chosen task and to identify the most cognitively demanding element. Stage 2 required the researcher to ask a series of questions using the eight probes, recording notes in a knowledge audit table. Following data collection, each interview was transcribed. Transcripts were used to ensure technical accuracy of the knowledge audits. The average transcript was 12,653 words in length, more than 100,000 words in total.
The knowledge audits were then merged to produce a cognitive demands table representing the key elements deemed to be cognitively complex. Each of the eight aspects of expertise composing the knowledge audit was considered in turn. For instance, we integrated participants’ responses to the past and future aspect of expertise consecutively, each time clustering common knowledge together and forming new themes of previously untapped knowledge. This process continued until the eight aspects of expertise had been populated with information from each knowledge audit. What resulted was a clustering of the key cognitive demand(s) for each aspect of expertise, added to the growing repository of the cognitive demands table on the basis of salience and frequency. This process of synthesizing and arranging the data thematically was important in ensuring representativeness across the 8 participants.
Results
Because of the wealth of qualitative data generated, an illustrative example is used to outline the results derived from Stages 1 and 2 of ACTA. For Stage 2, the knowledge audit, results are presented with a number of extracts to provide a sense of the data elicited. We then provide a summary of the cognitive demands identified across our sample as a whole. The cognitive demands are organized according to prospect theory’s fourfold pattern of choice and are used to illustrate the nature of preferences in experience-based choice as studied in the wild.
Illustrative Example: Day Trader A (Foreign Exchange [FX] and Bullion Trader, London)
Background demographics
Day Trader A is 25 years old and holds 7 years of experience within the investment industry.
Stage 1: Task diagram
The task diagram for Day Trader A focuses on “FX spot transactions” (see Figure 1). Spot transactions involve the agreement to buy and sell at the present market value and to settle the transaction a few days later. The four broad decision components underpinning the execution of FX spot transactions are considered in turn:
Illustrative example of a task diagram of “FX spot transactions” (Day Trader A).
The first component involves establishing what currency pair is going to form the basis of the trade. This stage is often informed by day traders’ attention to news releases and their assessment of its likely impact on foreign currencies. Directing attention toward the most liquid and widely traded currency pairs operates as a lens through which to filter incoming news releases, enabling rapid decision making.
The second stage of the task diagram involves decisions related to currency pair performance, that is, which currency is going to perform or not perform. By focusing on one particular currency pair, the field of choice is significantly narrowed, constraining the cues attended to.
The third broad stage entails decisions regarding the size of the trade and level at which the market will be entered. For instance, this may involve the identification of a level (support or resistance) that if broken, a given security is bought (pullback) or the identification of a dip in the market as an opportunity to buy. As the domain of choice narrows, decision making becomes increasingly associated with the logistics of placing a trade. However, the decision process underpinning this twofold choice is the most technically demanding.
For Day Trader A, technical analysis was noted to often drive this stage through consideration of support and resistance levels on chart points. One mechanism used to identify levels was Fibonacci retracements. Fibonacci retracements assume that any given security will retrace a percentage of the previous move before reversing.
The final stage of the task diagram involves the selection of levels from which a given position is exited (commonly referred to as trade management). This is integral both for instances where profit has been maximized and capital needs to be locked in, thus controlling profit risk, and for instances whereby a position is losing money and needs to be stopped out to maintain a positive profit and loss statement.
Complex component
The third stage, that is, size of the trade and entry levels, was identified as the most cognitively complex component within Day Trader A’s task diagram and formed the focus for the rest of the interview. This was similar to other traders’ task diagrams, although in addition to the size of the trade and entry levels, exit levels were also noted to be cognitively complex.
Stage 2: Knowledge audit
Table 3 provides an extract from Day Trader A’s knowledge audit. The two illustrative examples presented here are discussed in more detail below.
Illustrative Example 1: First Section of Knowledge Audit (Day Trader A)
Big picture
The “big-picture” element of expertise is useful in describing the broad-level framework traders use to construct their view of FX markets and the actions they take. Four components were found to frame what Day Trader A attends to (again, this was very similar to other traders):
In trading FX markets, the dominant focus was noted to be movements in the major and minor currencies. Staying cognizant of worldwide currency movements both enables and acts as a filter to other aspects, such as fundamental and technical analyses.
The second aspect composing the big picture is fundamental analysis. This involves tracking the fundamentals behind the currency driving the market at a given moment in time. As Day Trader A elaborated, understanding the strength of the economy is informed by a number of standardized parameters, including GDP.
The third component of the big picture relates to the use of technical analysis, that is, forecasting the future direction of markets through the study of past market data. Ignoring the actual nature of the market, technical analysis employs models and trading rules based on price and volume identified by market cycles and recognition of chart patterns. As Day Trader A explained, understanding past market performance and levels of resistance within the market, entry and exit levels can be identified. It appeared that in general, choice of charting methods and trading approach, such as the placement of resting orders in the market, reflects the degree of individual preference in experience-based choice.
The final aspect of the big picture, relates to traders’ need to remain cognizant of desk and individual targets. As can be seen in Table 3, Day Trader A notes the need to play the “percentage game,” that is, limit risks and run profits. Providing a description of the overall preferences and constraints framing day traders’ thinking, the next example taken from the knowledge audit demonstrates these in action.
Noticing
This aspect of expertise focused on the detection of market movements in minor currency pairs, with the collapse of the Thai baht (THB) and significant price move of the New Zealand dollar (NZD) as examples. Monitoring the market activity of major and minor currency pairs, Day Trader A suggested that in his experience, large, rapid movements are usually driven by data releases. Data releases might include the release of monetary policy statements expecting the NZD to weaken in New Zealand or shock wave–type releases, such as a bomb explosion. Known as “news trading,” rapid price movements require the immediate placement of a position. As Day Trader A warns, the source of movement should not be located until after a bid has been placed, or the opportunity will be lost. This example, alongside cognition elicited from other traders using the noticing prompt, illustrates how incoming data sources can instantaneously direct day traders’ attention to a particular currency, thereby constraining decision choice. Underscored by a preference for maximizing profit, the knowledge audit captures the importance of experience-based choice in knowing to immediately place a position.
Cognitive Demands
A cognitive demands table was compiled that synthesized data drawn across the eight knowledge audits from the day traders. The objective of this stage of ACTA is to provide a generic overview of (a) the difficult cognitive elements, (b) why it is difficult for a novice, (c) errors a novice might commonly make, and (d) cues and strategies that experienced day traders use to overcome cognitively difficult elements. Overall, six common elements of expertise that exhibited significant cognitive demands for the day traders interviewed were extracted (see Table 4).
Cognitive Demands Table
Each cognitive element listed represents a single scenario day traders might encounter. Note that although these tend to exist singularly, they may also occur in combination with other difficult cognitive elements; for example, detecting regime shifts might occur alongside anticipating effect of change in market fundamentals. Arranging cognitive activity in this way, the cognitive demands table provides a variety of real-world examples of preferences across the spectrum of medium-, large-, and small-probability gains and losses.
Combining prospect theory and cognitive demands
Organized according to prospect theory’s fourfold pattern of choice, the cognitive demands included in Table 4 are now presented alongside their implications for understanding preferences in experience-based choice.
Medium- and large-probability gains
Identifying emerging trends and arbitrage opportunities are two cognitive demands that reveal day traders to exhibit a risk-seeking preference for medium- and large-probability gains.
Identifying emerging trends. This example represents a basic trading methodology where risks are limited and profits run, articulated as playing the percentage game. The identification of emerging trends relies on a number of indicators, such as technical analysis and Fibonnaci retracements, that assist in the detection of entry points into a rising or falling market. Accordingly, these cues and strategies are used to optimize the conditions that are risk seeking for gains with medium and large probabilities.
Arbitrage opportunities. This type of execution is similarly risk seeking for gains, with an acceptance of the risks where the potential gain is a much higher probability. Exploiting price discrepancies between errors in bid and offer prices represents a “free trade,” with zero risk for acquiring a large probability gain.
Small-probability losses
The cognitive demand of responding to trend reversals illustrates how scenarios evolve in which traders are risk averse to small-probability large losses.
Responding to trend reversals. One of the cognitively challenging elements of decision making within this domain is responding to a change in market conditions after a position has been placed. A common error can stem from the naive belief that the reverse trend is only momentary, instead of immediately closing the position to minimize losses. If one ignores the stop loss in the hope that the market will change direction and lead to a profit, there is a small probability that this will produce a large loss and close the account. Traders are therefore risk averse to the small probability of a large loss.
Small-probability gains
The cognitive demands table also indicates choice behavior for gains with small probabilities to be risk averse. There are two specific instances, detecting regime shifts and anticipating effect of change in market fundamentals, that illustrate how this might occur.
Detecting regime shifts. One of the common errors day traders encounter is erroneous interpretation of price movement. This coupled with the desire to adopt a position on emergent trends before other market players so as to maximize potential gains might, for those with less experience, induce a risk-averse preference for small-probability gains.
Anticipating effect of change in market fundamentals. This difficult cognitive element is integral to the detection of regime shifts, and so in a similar vein to the above example, day traders exhibit risk-averse preferences for small-probability gains. Common errors can include errors in extrapolation and thereby basing hypotheses on small-probability scenarios.
Medium- and large-probability losses
The cognitive demands table also illustrates the case of risk-averse preferences for medium- and high-probability losses. Taking action following sudden interruption to supply is a case in kind.
Taking action following sudden interruption to supply. In the event of sudden interruption to supply and a collapsing market, risk-averse preferences can lead one to “overreact” and close out all positions or to take no action, which mitigates the loss. Promptly covering short positions (buying back the same type and number of securities previously sold) can assist in the minimization of losses.
These results suggest the fourfold pattern of preferences of experience-based choice when studied in the wild to be risk seeking for medium- and high-probability gains, risk averse for small-probability gains and losses, and risk averse for medium- and high-probability losses (see Table 5).
The Fourfold Pattern for Decisions Studied “in the Wild”
Discussion
Documenting a field study of expertise using a detailed cognitive task analysis, this study crucially begins to reveal the origins and contents of day traders’ preferences and how these constrain and enable experience-driven action. Our research suggests that there is value in adopting a naturalistic approach and underscores the timeliness of Hertwig and Erev’s (2009) suggestion to take the study of experience-based choice “into the wild.”
This study makes a distinct contribution toward Rakow and Newell’s (2010) call to better understand the properties of the internal representation of payoffs. Reviewing examples that sit at the intersection of description and experience, this research illustrates how day traders construct their understanding of gains relative to losses and emphasizes the relevance of prospect theory for understanding how attitudes toward risk concerning gains and losses deviate from studies of description and experience-based choice in the laboratory when studied in the wild.
We start by making reference to the notion of the percentage game—one where risks are limited and profits run—as a basic trading methodology. Risk seeking for gains with medium and large probabilities, traders understand that across a period of time, probabilities will weigh in their favor (gains × probability > losses × probability). See lower left quadrant of Table 5.
Arbitrage opportunities are similarly risk seeking for gains, with an acceptance of the risks where the potential gain is a much higher probability. Note that arbitrage is analogous to the sure bet of $1 from described choice. This is an interesting finding both because patterns of choice for medium and large probabilities is underresearched and because it sits antithetically with the speculative proposition that in decisions from experience, participants show risk aversion for medium- and large-probability gains (Rakow & Newell, 2010).
Theoretically, one would assume traders to be unequivocally risk seeking in the domain of gains. However, our findings suggest choice behavior for gains with small probabilities to be risk averse (see upper left quadrant in Table 5). Detecting regime shifts and anticipating effect of change in market fundamentals illustrate how preferences for small-probability gains can manifest as scenarios evolve. The explanation accompanying the cognitively complex element of Day Trader A’s task diagram supports this finding, drawing attention to the role of organizational constraints, such as daily stop losses and the percentage game, in avoiding account drawdown. Strategies of this type are uncommon within the domain of trading, a finding more likely to be found in the wild as opposed to conditions of descriptive choice within the laboratory. Our work does, however, concur with findings proposing that for small probabilities, participants are risk averse both for experience-based problems examined in the laboratory and in the wild.
On the note of small-probability events, the findings outlined within this article also suggest traders to be risk averse to the small probability of a large loss (see upper right quadrant in Table 5). The cognitive demand of responding to trend reversals illustrates how the purpose of stop losses is to close a trade at a predetermined level so that large losses are not incurred. Risk aversion for small-probability losses echoes the findings within the paradigm of description but not experience-based choice in the laboratory.
Knowledge gained through this study also leads one to speculate about choice behavior in the conditions of medium- and large-probability losses (see lower right quadrant in Table 5). The example of taking action following sudden interruption to supply illustrates how sudden changes can induce preferences for risk-averse action, for example, covering short positions to minimize losses. We wonder whether within this domain, choice following successive losses might also illustrate this type of preference? We note Kahneman and Tversky’s (1979) proposition that “a person who has not made peace with his losses is likely to accept gambles that would be unacceptable to him otherwise” (p. 287) as justification of this hypothesis.
As a whole, this work opposes the suggestion that “there is little evidence of a difference in patterns of choice under description and experience when the probabilities are moderate to large” (Weber, Shafir, & Blais, 2004, p.10). In fact, whereas Rakow and Newell (2010) hypothesized risk-averse choices for medium- and large-probability gains and risk-seeking choices for medium- and large-probability losses within the decisions-from-experience paradigm, this study suggests the converse. This insight is crucial where patterns of choice for medium and large probabilities remain underresearched and supports our hypothesis that field studies following an experience-based paradigm might reveal effects that meaningfully differ from both description-based and experience-based laboratory research.
Shifting preferences between these two formats of choice and decision making as studied in the wild underscore the asymmetry of human choice. In accounting for the differences in preference, we consider whether, for decisions from experience, there exist a number of subtypes. The first takes the form of experience through learning, whereby repeated exposure to the processes molds the responses. This is of the type studied with the lens of experience-based choice. The second, experience through professional training, is one whereby organizations set the fourfold pattern of responses, and experience simply sets out to achieve them. This experience type distorts the fourfold pattern exhibited within description and experience formats of choice seen in Table 1. For instance, although we might expect to see risk-seeking preferences for small-probability gains (we generally expect traders to be risk seeking for gains, risk averse for losses), experience through professional training overrides this. This insight holds consequences for the design of instructional training for novices.
The implications of the conceptualizations of experience through learning and experience through professional training might also extend to other complex sociotechnical systems, such as aviation and the military—domains in which, on the basis of this research, one would expect expert performance to be similarly risk averse, with the exception of risk-seeking behaviors for medium- and large-probability gains. The impact of these findings hold potential to transfer beyond the domain of study in this article and are worthy of detailed exploration within the NDM community.
Other decision-making perspectives might also take value from these findings. Calling attention to the value of context is an approach that moves beyond behavioral finance approaches (Anderson & Sunder, 1995; Staël von Holstein, 1972) and begins to build a more complete picture of how context, training, and socialization interweave to facilitate understanding of risk propensity. This is an important advancement that sits alongside the work of Fenton-O’Creevy et al. (2005). Future examinations of risk propensity might benefit further from emerging discussions of naturalistic studies of insight and intuition in field settings (Gore & Sadler-Smith, 2011; G. Klein & Jarosz, 2011).
This work also holds implications for ACTA, a method that has been endorsed as a useful addition to the instructional design practitioners’ toolkit. Providing a significant extension to studies employing the ACTA techniques, this study also illustrates its relevance for inductive-theory building.
Limitations
This research has some important limitations. First, although we have located our work within the theoretical framework of NDM (G. Klein, Orasanu, Calderwood, & Zsambok, 1993; Todd & Gigerenzer, 2001; Zsambok & Klein, 1997) and experienced-based choice, we recognize that despite two decades of empirical inquiry, NDM still requires further rigorous testing and theoretical development if the study of expertise in applied settings is to progress.
Second, as in all research conducted in the wild, participants subscribe to norms of organizational behavior and to common narratives about the nature of their domain expertise. It has been proposed that “NDM research needs to pay more attention to three things in relation to organizations: constraints imposed by context; distributed information; and differentials in power and vested interests” (Gore et al., 2006, p. 936). The degree to which these guide the evolution of preferences in experience-based choice lies outside the remit of the current research but warrants further study. In addition, it would be interesting to note what further value a cultural lens might bring to in-the-wild studies (H. Klein, 2004).
The third limitation questions the degree to which the fourfold pattern of choice exhibited by day traders extends to other domains when studied in the wild. This issue of generalizability also leads one to question the validity of the notion of experience through professional training as an explanatory account of the differences between experience-based choice as studied within the laboratory and in the wild. Although we acknowledge generalizability to be a significant shortcoming of qualitative research, we attest to the value of studying cognition in the wild as a means of identifying new possibilities for future research.
The fourth limitation is methodological, directly relating to the design of research conducted outside of the laboratory. We recognize the relentless work of applied researchers to resolve the dilemma of both gathering rich qualitative data with techniques such as ACTA and reconciling issues surrounding validity, reliability, the presentation of data to enact meaning, and the importance of replicable data analysis for future research. We are also aware of the difficulty of comparing findings from studies of cognition in the wild with those from description and experience-based choice within the laboratory, and we acknowledge that this opens the possibility that our findings are a function of method. Future work might rectify these shortcomings by focusing attention on the development of quasiexperimental approaches to accessing cognition in applied settings. ACTA provides a theoretically grounded foundation for the development of instructional content for this objective.
Future Research
Despite the limitations of these data, this work provides the opportunity for others to test the conjectures brought forth by this field study. Using this article and the insights it has generated as a foundation, we have developed a number of testable questions to guide future research.
Conclusion
This article demonstrates the value in taking the examination of experience-based choice into the wild. Our approach has started to illuminate the origins and contents of day traders’ domain preferences and how these constrain experience-driven action. This naturalistic examination emphasizes the relevance of prospect theory for understanding the internal representation of payoffs and the asymmetry of human choice: Preferences concerning gains and losses as studied with the lens of cognition in the wild differ from the fourfold pattern of preferences exhibited by experience-based choice when studied in the laboratory. Accounting for these differences, we conclude with the proposition that when studying cognition in the wild, one ought to make the distinction between the study of experience through learning and experience through professional training. We anticipate this research to pose a challenging and innovative starting point from which future collaborations across the decision sciences can develop.
Footnotes
Acknowledgements
The authors wish to thank Kathleen Mosier, Gary Klein, Eduardo Salas, and the three anonymous reviewers for their constructive advice, which assisted in the development of this article.
Claire McAndrew is a chartered psychologist and research associate in the Complex Built Environment Systems research group at The Bartlett School of Graduate Studies, University College London, United Kingdom. She completed her PhD in naturalistic decision making at the University of Surrey, United Kingdom, in 2008. Her ongoing research combines social science and design modes of enquiry in the study of cognition and behavior in physical and virtual environments.
Julie Gore is a chartered psychologist, associate fellow of the British Psychological Society, and senior lecturer in organizational behavior at the University of Surrey, United Kingdom. She completed her PhD in applied psychology at Oxford Brookes University in 1997. Her research focuses on the application of behavioral science to management research, in particular, cognitive psychology, cognitive task analysis, and a long-term interest in naturalistic decision making (having first joined the naturalistic decision-making community in Dayton, Ohio, in 1994).