Abstract
While “expertise” is frequently used as a variable in human factors research, the criteria for defining this construct often lack clarity and consistency. This article briefly reviews common definitions of expertise and how it has been operationalized in research, highlighting the need for more nuanced categorization of expertise. We posit that expertise is multifaceted and propose a dichotomy that distinguishes “system expertise” from “task expertise,” with recency and frequency of task performance playing crucial roles alongside traditional metrics.
As a part of a recent project investigating adaptive standard operating procedures, our research team collected data at a large oil and gas company’s high-fidelity training facility. During the data collection effort, our team discovered some interesting findings that sparked the discussion that motivated the topic for this article. For this study, we observed participants performing three tasks—column flushing, pressure testing, and centrifuge testing—which closely resembled the activities they would carry out on offshore oil rigs. While the main outcomes were performance and deviation from procedures, level of expertise was used as a co-variate.
The participants were recruited from two groups of workers: technicians (e.g., electricians and mechanics) and control room operators. We were informed not all, but some, technicians get promoted to the operator roles after several years at the job. Therefore, after discussions with subject matter experts at the company, operators were initially classified as “experts” and technicians as “novices.”
Our observation of eight workers (four experts and four novices) revealed that operators, who primarily worked in control rooms and oversaw the entire production system, indeed understood the systems as a whole much better than the technicians. The operators knew how actions in one part of the system affect other parts. Thus, they were often called to the job site for troubleshooting rather than routine tasks. An interview with one of the expert workers revealed that expert workers’ strength lies in resolving uncommon and unfamiliar situations by leveraging their understanding of the system, hence qualifying them as “experts.” However, we also discovered that these so-called experts were now less exposed to field tasks such as onsite testing and routine maintenance compared to the technicians. Therefore, reduced exposure to field tasks resulted in some expert operators struggling to complete tasks fluently. On the other hand, the technicians, despite being specialists in a particular area, were frequently assigned to routine tasks like those performed during our observation. One novice participant mentioned that despite his primary role as an electrician, he regularly performed the column flushing task, thus gaining considerable familiarity and experience with it. It was clear to our team that the boundaries between expert and novice were far blurrier than what was planned in our research design.
This paper aims to start a practical conversation about the inconsistencies and limitations in how “expertise” is commonly defined and used in human factors research and design process. This paper by no means offers a comprehensive theoretical model or generalizable empirical study. We begin by briefly discussing how “expertise” is typically operationalized and then share our perspective on the challenges that human factors and ergonomics (HFE) researchers face with this variable. Drawing from our own observations, we also introduce two new categories for expertise and their moderator variables that may offer a more nuanced and functional way to distinguish between types of expertise. We suggest that incorporating these dimensions into HFE research and design practice could improve how experts are selected and clarify the types of insights that can be meaningfully elicited from them.
BACKGROUND
In many aspects of our society, we place great value on expert knowledge and opinions. Whether in courtrooms, writing standard procedures, or training new workers, we often seek guidance from those deemed experts. Even in common situations, we turn to those we trust. For example, we call family members to recreate a favorite dish. Experts are those with years of experience, deep knowledge of their field, familiarity with the history and intricacies of their domain, and a thorough understanding of what needs to be done to accomplish a given goal. In other words, they simply know “what to do.”
In professional settings, expertise comes with significant influence. The opinions of experts are often given more weight, especially in complex decision-making processes. Many organizations and domains rely on expert input through methods like the Delphi expert panel for forecasting, making critical decisions, and constructing professional guidelines (Baker et al., 2006; Dalkey & Helmer, 1963; Nasa et al., 2021). Similarly, in HFE, experts and their expertise are an integral part of the research and design process. For instance, HFE methods such as human performance modeling (e.g., Cognitive Task Analysis and CPM-GOMS methods), human reliability analysis, and probabilistic risk analysis are heavily informed by expert inputs (Gordon & Gill, 2014; John et al., 2002; Liu et al., 2016; Ouchi, 2004). These methods underscore the perceived importance of expert knowledge in investigating human–system interaction tasks and performance. Furthermore, in the design of experiments in HFE research, subject matter expert opinions often play an important role in choosing the right representative variables (Kantowitz, 1992). Carayon (2006) and Charness and Tuffiash (2008) also emphasize the increased importance of expert opinions across different domains and their collaborations when conducting human factors research in complex sociotechnical systems such as healthcare, computer security, sports, and aviation.
Despite the importance and increased use of expertise as a demographic variable in HFE research and design, there are several issues with how expertise is currently defined and operationalized. The term “expert” is often used loosely across various fields. In many books, news articles, and media, “expert opinion” is frequently cited without sufficient justification of what makes someone an expert in the first place, and academic journals are not much different. Furthermore, there is no widely accepted quantified measure of expertise. Without standardized operationalization and clear metrics, at least within a field of interest, it becomes difficult to draw a clear line between experts and non-experts. This lack of clarity leads to inconsistent use of the term “experts.”
Goldman (1991) distinguished two core requirements of expertise: what an expert knows (knowledge—that) and what an expert can do (knowledge—how). Generally and ideally, true experts would possess both elements of expertise to be called an expert (Croce, 2019). In the context of applied research, Baker et al. (2006) outlined several key characteristics of “experts” to consider when conducting a Delphi panel of experts. According to their research, the most common factor to consider is years of experience or level of knowledge. Similarly, Weiss and Shanteau (2003) also mentioned that the number of years spent in job-relevant environments is frequently used in studies requiring experts. In the case of a cognitive psychologist, his expert participants included those in senior positions, such as a person who participated in designing the first metrological satellites (Hoffman, 1998). Hoffman also stated that senior personnel who can create standards and write procedures due to having high and vast levels of knowledge in a domain can also be considered an expert (1998). Broadly stated, we often assume that someone is an expert if they have worked in a particular field for a long time. For example, in recent data collection for our own research in the oil and gas industry, the facility manager informed us that they would generally consider people with five or more years of experience as experts. While the magic number “5” has often been used as a threshold to delineate experts from novices, this one size does not fit all domains, tasks, or even cultures. Experts typically have spent many years in their field, but the notion that time alone guarantees expertise is not always true. In fields where no official qualification, such as accreditation, is required, the connection between the knowledge level, years of experience, and expertise is more complicated (Baker et al., 2006). Moreover, there seems to be little to no consensus across the literature we are familiar with about how many years of experience are necessary to constitute/warrant expertise status.
To compensate for the limitations in simply choosing years of experience and level of knowledge as criteria for expertise, some have turned to accreditation such as certification, license, diploma, and awards as a way to identify experts. For example, highly skilled or educated professionals such as board-certified specialists in the healthcare field or full professors in academia are often recognized as domain experts (Weiss & Shanteau, 2003). However, while most degrees and some certifications and licenses, once earned, are valid for a lifetime, if the person stops practicing the required skills or being up to date with the state of knowledge in the field, their competence will decline. This holds true for many “professional” occupations such as lawyers and pharmacists.
Another method to define expertise and identify experts is through peer identification. For example, academia and research communities often determine experts by the number of published books and/or peer-reviewed articles (Baker et al., 2006). Ericsson (2006) introduces social reputation as one of the elements in identifying experts. However, this method of defining and identifying experts is prone to popularity bias (Weiss & Shanteau, 2003). Other methods to define and identify expertise include selecting individuals with the authority to influence an organization’s policies, such as board members (Baker et al., 2006). Additionally, expertise can be associated with the ability to make consistent judgments (Dror, 2016; Weiss & Shanteau, 2003), make accurate judgments even when provided with contextually irrelevant information (Dror, 2016), and distinguish two extremely similar cases (Weiss & Shanteau, 2003).
A PROPOSED DICHOTOMY: SYSTEM vs. TASK EXPERTISE
Despite the above-mentioned effort, expertise remains notoriously ill-defined in research, and its definition largely influenced by the years of experience and seniority. To recognize the varying scopes and levels of expertise, and grounded in our collective experience in human factors research, we posit that it may be beneficial to recognize two types of expertise: system expertise and task expertise.
System expertise aligns with the more traditional characteristics of the knowledge possessed by experts. People with system expertise typically have many years of experience and possess a deep understanding of the larger system, knowing how its components interact. These experts often serve roles at the “blunt end of the system,” such as management or oversight, which makes them prone to skill loss (Arthur Jr et al., 1998; Chatham, 2009). It is well documented that access to long-term memory is limited to the level of activity of the stored information which is a function of the recency and frequency of use or perceived importance (Bjork, 2001; Popov & Reder, 2020). For instance, one of the participants classified as an expert was given the pressure testing task, a routine procedure on offshore rigs to ensure system alarms trip at the appropriate high- or low-pressure points. Even though the task involved using a simple hand-held device to test the alarms, the expert participant struggled with its execution. However, during the interview, the same participant had no trouble explaining the purpose of the task, potential causes of abnormal pressure readings, and methods to resolve them. The participant noted that it had been a long time since he last performed the task.
As shown from this example, those with system expertise are better situated to understand the “big picture.” In other words, their mental model is constructed strongly around the holistic understanding of the system and how the components of the system should interact. Such hierarchical classification reminds the authors of the Abstraction Hierarchy (AH) methodology (Rasmussen, 1985; Vicente, 1999) which posits difficulties in attaining various levels of system abstraction by a single role. In that sense, the system expertise fits more in the functional levels (i.e., top three levels—Functional Purpose, Abstract Function, and General Function) of the AH rather than the physical levels (see an example in Figure 1). In terms of Goldman’s expertise distinction (1991), system expertise may have reduced knowledge—how when it comes to the details of specific task execution, even though their knowledge—that is solid. Abstraction Hierarchy for the pressure testing task, which was created with inputs from both expert and novice workers. Experts are better equipped to think on the functional level, whereas the novices are more familiar with the physical levels of the task.
In contrast to system expertise, task expertise can be demonstrated via fluency in specific skills or activities, even if those with task expertise are not experienced with the overall system. In the same pressure testing task, a novice participant (an electrician) completed the task effortlessly and even offered insights into how the task could be done differently in certain circumstances. Although he was not classified as an “expert,” he claimed that he performed this task daily when he was on oil rigs and demonstrated fluency in the task, despite it not being part of his primary role.
Workers who possess task expertise typically excel in the execution of specific, routine tasks. They are highly knowledgeable about the immediate system components relevant to their duties and therefore have their mental model built around the specific tasks that they interact with regularly. As those workers expand their knowledge about different parts of the entire system, they will learn to see the forest rather than just the trees. In terms of AH, the task expertise aligns with the physical levels (i.e., Physical Function and Physical Form) of the AH in Figure 1. The discrepancies in how non-experts and experts think on different levels of the hierarchy of a system were also recorded by Tenney and Kurland (1988). The researchers interviewed military radar mechanics who had 1, 5, and 10 years of experience and asked them to describe how radar works. The novice mechanic mostly focused on describing the radar system’s “power distribution and the physical layout” in detail, whereas the intermediate and expert mechanics focused more on the functional aspects such as information flow and feedback loops (Tenney & Kurland, 1988). As seen in this example, it appears that people with task expertise emphasize task-specific schemas compared to people with system expertise who tend to have a broader mental model of the overall system.
DEFINING THE DIMENSIONS OF SYSTEM AND TASK EXPERTISE
Based on our observations, we recommend considering two moderators—frequency and recency—of task exposure in HFE research and design process to distinguish between task and system expertise. Frequency of exposure to a task may play a key role in forming automaticity (Rasmussen, 1983) and can lead to reduced mental workload (Popov & Reder, 2020). As tasks are repeated, the top-down cognitive process strengthens, which lowers cognitive demand and frees attentional resources which can be utilized to form a better mental model of the system (Wickens, 1981). For example, two workers with the same years of experience in the field may have vastly different levels of expertise in a particular task, depending on how often they have performed that task. Therefore, frequency should be included in measuring expertise rather than relying solely on years of experience.
Recency of exposure is also essential in maintaining information stored in the long-term memory and the ability to maintain a current mental model of a system or task. Workers who have performed tasks more recently are more likely to have up-to-date procedural knowledge and stronger recall abilities. Recency may be the element that could draw the line to distinguish system experts from task experts. As demonstrated in our case, those with system expertise typically have a thorough understanding of the overall system. However, while these experts may have had extensive experience with routine tasks, their experience may not be recent. Long gaps in exposure to detailed task procedures may result in skill loss (Arthur Jr et al., 1998).
An Illustration of How System Experts and Task Experts Were Reclassified in the Case Study Using Frequency and Recency as Moderators, Along With the Two Most Common Variables for Identifying Expertise in Practice/Research
An important consideration is that the recency and frequency of task exposure can overlap, as seen in the frequency row of Table 1. Indeed, recency and frequency are intertwined constructs when a recent timeframe is considered. Therefore, in practice, demographic questionnaires should be designed to specifically ask for a timeframe (e.g., last month or last few months) when eliciting the frequency of exposure to a certain task. In addition, a question related to the broader experiences within the field (e.g., total years of industry experience, variety of roles or responsibilities held, or formal training) or the specific work environment should be captured to enable task vs. system expertise classification.
CONCLUSION
Expertise is a complex variable that may need decomposition for use in research as a variable. People with system expertise are those who have a significant amount of experience and knowledge (and possibly accreditation) about the overall field, the workplace, and the system of interest. On the other hand, those with task expertise may lack the breadth of skills and experience of system experts, but they possess well-established, task-specific knowledge and skills gained through frequent and recent exposure. While there could be an overlap between the two types of experts, such as those who are called “Masters” (Hoffman, 1998), it is important to recognize that a true expert, whether in a system or task, should not only be assessed through the experience and accreditation, but those common criteria should be accompanied by frequency and recency of task exposure.
Aside from the research and design implications, this approach may have important real-world workplace implications. For instance, when writing procedures for routine tasks, which are often completed by novice workers, system expertise alone may not be sufficient due to skill loss associated with diminished recency and frequency of task exposure. Thus, efforts aimed at task (re)design and procedural changes may need a participatory approach where those with task expertise, typically workers at the sharp-end of the systems, need to be involved. This may ensure a more accurate representation of expertise, particularly in industries where expert knowledge plays an important role.
LIMITATIONS AND FUTURE WORK
This article was inspired by observations made during a data collection effort in the oil and gas industry with a limited sample size. However, more case studies and examples from other domains would enrich the discussion documented here and improve the generalizability of recommendations. Additionally, more observations and empirical work are needed to investigate differences between system and task expertise and how such difference manifests itself in outcomes such as performance, compliance, and safety culture. More work is also warranted to identify additional moderators to better draw the line between the two dimensions of the expertise, other than those mentioned in this paper. For example, in the case of our project, the observation and interviews were based on eight participants. Given the breadth and diversity of domains in which human factors and ergonomics have left a footprint, as well as pervasive usage of expertise in research and practice, more discussion is warranted to ensure robust utilization of the proposed dimensions and moderators.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: National Science Foundation (No. 2106963).
