Decision-Centered Design for Post-implementation Analysis of Automation in the Petrochemical Industry

Objectives

This case study describes an application of the decision-centered design framework (Militello & Klein, 2013) to the use of Advanced Process Control (APC) software by console operators in the petrochemical industry. The primary objective of this project was to understand the cues and strategies experienced console operators use to effectively manage the APC in both routine and non-routine situations. A secondary objective was to identify implications for improved user interfaces.

Background

Console operators in petrochemical plants manage large, complex processes that operate around the clock to synthesize and refine petrochemicals for a range of industrial and consumer uses. APC is designed to support console operator work by automating a portion of the processes. APC software makes small, continuous adjustments to key variables to optimize performance of complex processes. These software tools can create greater efficiencies than human operators are able to achieve without them. Anecdotal evidence suggests that some operators are able interpret what the APC is doing, and appropriately intervene when APC is not operating effectively. Others may continue to let the APC run when it has reached the edges of its capabilities and is no longer optimizing the process. The sponsoring agency was interested in understanding the strategies and information needs of APC experts to inform user interface design improvements.

However, investigating use of automation such as APC software can be challenging. Typical methods investigating use after implementation can be labor-intensive and difficult to interpret. Large-scale surveys allow for representation from a broad swath of users, but often result in superficial understanding of use. Passive feedback such as keystroke logs are relatively easy to collect but can be difficult to interpret. Feedback via helpdesk tickets can be informative but may be time-intensive to review and problem descriptions may be difficult to interpret.

For this project, we used the decision-centered design framework to conduct an in-depth qualitative study of console operator work and APC use.

Approach

The decision-centered design framework emphasizes the importance of understanding the tough decisions that operators face with the notion that these are circumstances in which decision-support features are most needed. Decision-centered design can be contrasted to other approaches that focus on routine operations. Focusing on routine operations sometimes supports more comprehensive analysis, but resulting designs are often brittle when something unexpected happens, failing to provide the needed support or even hindering expertise in extreme cases.

The decision-centered design framework includes five stages: Preparation, Knowledge Elicitation, Analysis & Presentation, Design, and Evaluation. For this project, we addressed the first three phases, and offer design implications. To prepare for knowledge elicitation, we reviewed relevant design documents and technical reports, and interviewed stakeholders to obtain a high-level overview of console operations in the petrochemical industry and the role of APC.

We visited a petrochemical plant to observe console operators and conduct interviews with five console operators, two console supervisors, three engineers and one trainer. We focused on one unit in which operators are responsible for monitoring approximately 100 to 200 controllers. In this unit, console operators have up to four APCs to facilitate process operations; however, this only accounts for a small percentage of the processes the console operator is monitoring and controlling. Furthermore, console operators are responsible for monitoring the entire process; noticing and managing perturbations that may reduce the safety and efficiency of operations. Typically, the APC operates in the background to support optimal process performance, requiring little or no input from the console operator. However, the APC is designed for “steady state” operations in which the process is relatively stable. Process start-ups, system upsets, and other conditions create process volatility which may cause the APC to behave in unexpected or non-optimal ways. These events prompt operator intervention, often requiring operators to infer the APC’s actions, rationale, and/or temporarily discontinuing use of the APC.

Interviewers used the Critical Decision method (Crandall et al., 2006) to elicit examples of critical incidents, the Knowledge Audit (Militello & Hutton, 1998) to explore expertise in console operations, and the Simulation Interview technique (Militello & Hutton, 1998) to capture decision strategies in the context of a challenging incident. This approach supported the team in efficiently eliciting critical incidents to inform cognitive requirements (aspects of work that are cognitively challenging) that support challenging aspects of work. Although this 9-month research effort was not comprehensive, we successfully identified cognitively complex aspects of work that represent leverage points for supporting console operators in harnessing the strengths of APC software and adapting when the APC is less effective.

The team used a cognitive requirements table to analyze and represent cognitive requirements, information needs, and strategies described by interviewees.

Findings

Cognitive Requirements

We identified three big picture cognitive requirements: diagnosing, problem solving, and anticipating future problems. These requirements serve as core components of the console operator work. Console operators constantly monitor processes for anomalies. When something unusual or unexpected occurs, they diagnose the situation and begin problem solving. Experience with diagnosing and problem solving allows them to anticipate problems; they track weather that may disrupt operations, equipment that is unreliable, and other likely perturbations.

We also identified five APC-specific cognitive requirements (in the context of the big picture cognitive components). These include determining:

1. when to turn on the APC after startup,

2. when to adjust targets,

3. when to change the APC’s mode of operation,

4. when the APC cannot operate optimally and should be shut off, and

5. when conditions have stabilized and the APC should be turned back on.

These cognitive requirements represent situations in which the console operator must use professional judgment to determine whether to let the APC run, to turn it off, or to make adjustments to compensate for information the APC does not have access to (i.e., a sensor with a faulty reading).

Implications

The APC-specific requirements highlighted opportunities for user interface improvements and training upgrades. For example, some operators described the importance of mentally estimating how long it would take the APC to reach a target value. With regard to interface design, this represents an opportunity to make APC-generated trajectory graphs accessible to operators so they do not have mentally estimate the trajectory. With regard to training, educating operators about the use of trajectory plots to understand the APC’s timing may provide important support for managing the APC. Additionally, simulated scenarios, perhaps including perspective taking exercises, may (1) support operators in experimenting with different strategies to learn the strengths and limitations of the APC in a safe environment and (2) support engineers in understanding the constraints console operators manage in troubleshooting potential problem.

Takeaways

We highlight two takeaways. First, decision-centered design proved a useful framework for efficiently understanding console operator work, and challenges related to APC use. This framework allowed us to quickly come up to speed. The research team was completely new to petrochemical operations and was able to successfully elicit examples of challenging aspects of console operator work to information cognitive requirements and information needs in a relatively short span of time. The project was 9 months from inception to final out-brief. Findings do not represent a comprehensive understanding of console operator work but do highlight leverage points for cognitive support in challenging situations.

Second, use of sophisticated automation evolves over time as the nature of work changes. Operators learn about the limits of automation in different situations, when it is reliable, and when it begins to reach the edges of its capabilities. Efforts to capture this evolving expertise are critical to supporting operator work in terms of both training and refinement of user interfaces. Cognitive task analysis methods can support organizations in eliciting examples of challenging situations operators face in managing automation in the context of work.