Assessment of Electric Grid Transmission System Simulator for Human Factors Research

Abstract

Over the past decades, various technologies have been developed for electric grid operations to support clean energy, meet rising electricity demands, and address infrastructure concerns. However, the human factors aspect is often overlooked during rapid integration. Questions persist about how these technologies impact human performance. Simulators play a critical role in supporting investigation of human factors design concepts and conducting comprehensive usability testing to evaluate human performance and assess human reliability. This paper aims to address human factors research simulator requirements and conduct a comparative study of six different simulators. A detailed evaluation reveals that the evaluated simulators lack the ability to customize user interfaces. Additionally, their user interface designs do not fulfill the basic human factors design principles, potentially leading to increased response variability and reduced statistical power when conducting controlled experimental research. In the future, it is essential to develop scripting tools to integrate customizable user interfaces and simulation models, ensuring meeting research requirements.

Introduction

In recent years, concerns about the societal impacts of wildfires, global climate change, and shifting weather patterns have arisen. The electric grid industry is transitioning to technologies such as distributed energy resources, large scale renewable generation, and battery energy storage systems (Akorede et al., 2010). Because of the rapid development and introduction of advanced technologies, questions persist about operator performance and needed capabilities when facing different operational conditions. In some cases, technologies that are installed at the field can indirectly impact operators’ performance. For instance, the settings of field devices and alarm setpoints can influence the frequency of alarms in the control room, potentially causing alarm floods, particularly during outage seasons. Additionally, technologies that increase uncertainties and variations can affect human performance, even if the user interfaces (UIs) remain unchanged. Common challenges in human factors (HF) research also apply to electric grid operations. For example, unexpected automation (e.g., remedial action scheme) operations had contributed to disturbances at the past (NERC, 2022).

The electric grid sector operations have already experienced human factors and performance challenges. Control room operators encounter operational uncertainties associated with technology failures, malfunctions, and more (WECC, 2017). The integration of new advanced technologies into electric grid operations is making the systems and technologies’ characteristics increasingly complex (Santos et al., 2021), therefore potentially affecting the systems’ interdependencies, necessitating a greater understanding of the systems, influencing operators’ responses, workload, and performance. Using simulators is a common approach to investigate the impacts of integrated technologies, allowing for investigating hypotheses and discovering gaps without negative consequences.

Many examples of simulators designed and configured to facilitate the examination of technologies’ impacts on human recognitions and responses exist. In aviation, flight simulators have been designed to study pilot recognition and responses (Stuster, 2006), and air traffic control simulators have also been implemented to investigate the design interventions and analyze the performance of air traffic control specialists (McGee et al., 1997). In ground transportation, driving simulators have been created to explore the impacts of road signals, driving assistance systems, and autonomous technologies on drivers’ responses (Akamatsu et al., 2013). In the medical industry, various types of simulators, such as those modeling the human body and patient flow, have been implemented to support training health care professionals and enhancing medical devices and systems (Riley, 2008). In the nuclear industry, simulators have also been used in training operators and system design (Le Blanc, 2024). These examples meet the requirements for HF simulator experiments or usability testing, the generalizability of results is influenced by the representativeness of the simulator, scenarios, tasks (e.g., goals, priorities), and human performance measures. Highly generalizable results from experimental studies and testing often need to include most applied contexts, which are sometimes characterized by factors such as workload, types of operational conditions (e.g., faults), complexity, uncertainty, and available actions. Sufficiently high-fidelity simulators can be critical to create such complex operational conditions (Kieffer et al., 2015). A similar capability is crucial for electric grid operation.

The underlying physics and simulation techniques of electric grid systems within the human’s response timescale have been well-established. In electric grid transmission planning and operations, reliability planners use power grid engineering analysis tools to model the system and conduct reliability analysis; control room operators and engineers conduct steady state analysis to prepare day-ahead plan. Backend simulation capabilities for HF research exist in theory. This paper aims to investigate the gaps between the simulation tools available in the market and HF research needs.

This paper addresses the research simulator requirements related to HF, providing simulation vendors with a general understanding of the gaps, and offering practitioners insights into the electric grid domain characteristics. The examination of simulation and modeling tools is conducted to provide a detailed understanding of the simulation challenges. This aims to assist HF researchers interested in conducting studies in electric grid operations, enabling them to better comprehend the challenges before investing in a specific simulation tool.

HF Research Methods

Before delving into detailed research needs, it is essential to understand common HF research methodologies and their relation to the use of simulators. There are three main categories of HF methods: cognitive engineering, which centers on insights gathered from interviews and observations; usability evaluation and testing, which focus on user needs and often test less rigorously compared to experiments; and controlled experimental research, which focuses on discovering new knowledge and interventions by examining the relations among variables. Usability testing and experimental research can involve participants’ task executions under simulated environments, allowing examination of human performance and responses (e.g., behavioral responses, psychometrics, and physiological measures).

The consideration of experimental design can involve these aspects: the development of independent variables, the experimental scenarios, experimental procedure, the measurement of dependent variables, and the data analysis process. Table 1 illustrates the interrelations between experimental design considerations and the general HF research needs of simulators.

Table 1.

Interrelations Between Experimental Design Considerations and General HF Research Needs.

Experimental design consideration	General needs of HF simulator flexible capabilities
Independent variables often encompass different types of UIs.	Incorporate multiple UIs design options and concepts.
Independent variables can involve the use of new tools and technologies.	Allow efficient introduction of new tools and technologies.
Representative experimental scenarios are important to ensure the generalizability of experimental results.	Load and execute various operating conditions.
Dependent measures often involve human behaviors, performance, physiological responses, questions, and answers.	Gather human behaviors and responses (e.g., control actions, response time) with timestamps.
Some dependent variables need to be captured during the experiment.	Pause simulations to collect data (e.g., situation awareness, workload).
Large variability can reduce the statistical power in hypothesis testing.	One approach is to follow basic HF principles to reduce variation of performance and responses.

Electric Grid Simulator Requirements

To accommodate common potential HF research topics, simulators need to be capable of representing the electric grid system and control room operations. In the following section, we will outline simulator requirements which are broken into: the basic UIs elements directly related to operators’ primary goals and functions, simulation, and analytics relevant to common electric grid system functionalities, functions for scenario setups and configurations, and functions for recording dependent measures. Simulator requirements related to scenario setups, configurations, and recording performance are derived from Table 1.

In electric grid operations, operators’ primary goals involve maintaining the system within voltage, frequency, thermal, and contingency limits, matching generation to load, minimizing net inadvertent interchange, maintaining operating reserves, minimizing load shedding, planning (Li et al., 2020; Li et al., 2022). Given the complexity and uniqueness of each transmission system, the requirements of basic UIs elements can involve the following categories described in Table 2.

Table 2.

Interrelations between Operator Goals and Electric Grid Basic UIs Element Simulator Requirements.

Operator goals	Basic UIs element simulator requirements
Maintaining awareness of the overall system states.	System overview.
Understand each substation’s configurations.	Substation UIs.
Maintain awareness of violations and abnormal operating conditions.	Alarm panel.
Maintaining the system within the voltage limits.	Bus voltage summary and alert. Transformer voltage summary and alert.
Maintaining the system within the thermal limit.	Transmission line power flow (e.g., apparent power, current) summary and alert. Apparent power or current indirectly indicates the amount of heat generated during transmitting electricity. Transformer power flow summary and alert.
Matching generation to load	Generation summary. Load summary.
Planning	Load forecast. Generation schedule. Renewable forecast and weather.
Maintaining the system operates within N-1 contingency limits.	Contingency analysis results. Contingency is an unexpected failure of a single principal component such as a generator or a transmission line that causes the change of the system states, leading to violations.
Minimizing the net inadvertent interchange. Maintaining the system within the frequency limits.	Balancing authority area control error limit (BAAL) graph. It involves both frequency and area control error (ACE) aspect, which measures the difference between scheduled and actual generation within a control area.
Maintaining operating reserves.	Spinning reserves and non-spinning reserves.

A simulator for balancing authority and transmission systems must possess characteristics essential for understanding and analyzing the interactions between humans and other system elements. Depending on the objectives of research, specific systems, subsystems, components, and controls may require simulation. An effective HF transmission system simulator should include the following elements, functions, and analytics: representative system configuration, automatic generation control (AGC), generators, breakers, transformers, transmission lines, relays, ACE, effectiveness factor or generation shift factor, control and status of synchronous generators, loads, shunt capacitors, and reactors, renewable energy integration, and simulation update time scale within required limits to represent the current operations. For example, contingency analysis results should be updated at least once every 30 min (NERC, 2017).

Additionally, it also needs to support functions for scenario setups and configurations, which include customizable UIs, scripting a scenario with events and system configuration changes at specific times or triggered based on system conditions, scripting a scenario with screen and simulator freeze, saving and presetting system conditions, and running optimal power flow analysis (OPF), and saving results for loading as initial conditions. In the current transmission operation, OPF is conducted to prepare the day-ahead planning (Reddy & Bijwe, 2016), which can be considered as initial conditions without unexpected events in a study. Functions for recording performance can include recording manual actions, relevant timestamps, recording system states, and recording crashed or abnormal applications.

Assessment

The comparative analysis of existing power grid simulation and modeling tools concerning these requirements was conducted, aiming to elucidate the gaps and challenges in power grid simulation. It closely resembles the competitive usability evaluation and review in user experience (UX). In this comparison, the features were extracted from the electric grid simulator requirements for HF research. This study examines the common simulation and modeling tools: RTDS, PandaPower, PowerWorld, IncSys, Electricity Infrastructure Operation Center (EIOC)’s Energy Management System (EMS), and EIOC’s GE dispatcher training simulator (DTS). A comparison analysis is conducted and characterized by the following categories.

• Built required with default models (B). The simulator lacks the function within default models, necessitating its development.

• Rebuilt needed with default model (RB). While the simulator has the necessary function, it fails to fulfill the basic HF design principles, resulting in unnecessary confounding variables and high variability, ultimately reducing the statistical power. As suggested before, it is particularly important for the purposes of advancing HF design principles and practices.

• Customization required with default models (C). The simulator possesses the function, but the default model hasn’t been configured or set up to utilize that function.

• Tuning required with default models (T). The simulator has the functionality, but adjustments are needed to serve its functions.

• Theoretically infeasible and non-applicable features (NA). The simulator cannot incorporate the features due to its inherent characteristics and limitations.

Findings

In the findings, we segmented the results to facilitate better understanding of what we refer to. The basic UIs elements section primarily focuses on frontend UIs, differing from backend simulations and analytics. Readers may notice some overlap between these two descriptions. Additionally, as mentioned before, functions for scenario setups and configurations, as well as functions for recording performance, are primarily determined by the functional requirements of common HF experiment setups. Depending on the research questions, certain experiments may require specific functions. Therefore, it’s essential for the functions to be flexible, allowing researchers to choose accordingly.

RTDS

It is a type of power hardware-in-the-loop simulation environment, which allows engineers to analyze the power system behaviors and schemes in a controlled environment. Though more accurate and granular levels may enhance accuracy on a more finite time scale, the high computational costs can limit the benefits derived from the high-fidelity simulation. Given its characteristics, it is not applicable for running contingency analysis, OPF analysis, and estimating effectiveness factor, or generation shift factor.

PandaPower

PandaPower is a power system modeling tool using python. It is an open-source tool, which means there is no liability protection. The default PandaPower IEEE 14-bus was initially designed for a distribution system, not transmission (Thurner et al., 2018). While converting a distribution system setup to a transmission system may not be complicated, it requires a good understanding of the power grid system and PandaPower, beyond the scope of general developers’ knowledge.

PowerWorld

PowerWorld Simulator is an engineering analysis platform primarily designed to support and train electrical engineers in conducting various engineering analyses, such as steady state power flow analysis and transient analysis, for transmission planning and operations. Engineering work is often less time-sensitive compared to real-time operations. As an engineering analysis tool, operators’ tasks are not well-supported.

IncSys

It is simulation-based online training platform for electric grid operations. It has complexed enough system construct for representing some system characteristics. However, when evaluating the simulation, we have identified issues beyond initial expectations. For example, the contingency analysis results in IncSys sample case may not accurately represent the reliability of actual transmission system. Some errors exceed 3%, which is significantly higher than contingency results’ errors observed in real-world systems. For training purposes, it may not significantly impact learning experiences and can be addressed with additional instruction, as the main objective is to improve the understanding of electric grid operations.

EIOC’s GE DTS

It is primarily developed to train transmission operators, balancing authorities, and reliability coordinators. It has most of the scenario scripting, setups, and performance recording functions (Anderson et al., 2022). However, like many commercialized applications, its development codes are not directly accessible, and the UIs is not customizable by customers.

EIOC’s EMS

It is running with a cyber-physical system testbed that integrates hardware-in-the-loop real-time transmission system simulator to a commercial industry-grade EMS software (Becejac et al., 2020). Its original functional purpose was to serve cybersecurity research. Consequently, like other engineering analysis tools, it lacks functions for scenario setups and performance recording. In summary, the comparative analysis reveals that the basic UI elements in most simulators fail to accurately represent control room operations and the interfaces operators interact with. Upon the six types of simulators that we examined, ordinary grid elements such as generator, load, breaker, transformer, and transmission line, are often included. However, the functionalities associated with representative system configurations, contingency analysis, AGC, relay operation, effectiveness factor estimation, OPF analysis, and renewable generation are often absent from default models, or require tuning due to unrepresentative errors. Additionally, functions for scenario setups, configurations, and human performance recording are commonly not integrated into default models. In other words, the simulators that we have evaluated fail to both effectively represent electric grid operations and serve HF research purposes. Table 3 describes the details of comparative analysis results.

Table 3.

Comparative Analysis Results.

Functional Element	RTDS	Pandapower	Powerworld	Incsys	EIOC’s GE DTS	EIOC’s EMS
Basic UIs element
System overview	B	B	RB	T	RB	RB
Substation UI	RB	B	B	T	RB	RB
Alarm panel	RB	B	RB	RB	RB	RB
Bus voltage summary & alert	B	B	B	RB	RB	RB
Transformer voltage and power flow summary & alert	B	B	B	RB	RB	RB
Line power flow summary & alert	B	B	B	RB	RB	RB
Generation summary	B	B	B	RB	RB	RB
Load summary	B	B	B	RB	RB	RB
Load forecast	B	B	B	C	RB	RB
Generation schedule	B	B	B	T	RB	RB
Renewable forecast and weather	B	B	B	T	C	B
Contingency analysis UI	B	B	B	T	C	C
BAAL graph	B	B	B	B	RB	RB
Spinning reserves and non-spinning reserves	B	B	B	RB
Simulation and analytics
Representative system configuration (e.g., reasonable setpoints)	C	C	C
Contingency analysis	NA	B	C	T		C
AGC	B	B	C
Generator
Breaker
Transformer
Transmission line
Relay	B	B	C			B
ACE	B	B	C
Effectiveness factor or generation shift factor	NA	B	C	B
Synchronous generator control and status
Load
Shunt and series capacitor or reactor		C	C			B
Renewable	B	B	C	B	B	B
Simulation update within required timescale limits		B	B
Functions for scenario setups and configurations
Customizable UI design	B	B	B	B	B	B
Scripting a scenario with events and system configuration changes at specific time or triggered based on system conditions.	B	B	B	B		B
Save, and load preset dynamic system operating conditions (e.g., scheduled and actual load and generation).	B	B	B			B
Run OPF analysis and save results for loading as initial conditions.	NA			B	B	B
Scripting a scenario with screen and simulator freeze.	B	B	B	B		B
Functions for recording performance
Record operator actions	B	B	B	B		B
Record relevant time stamps	B	B	B	B		B
Record system states	B	B	B	B		B
Record crashed or abnormal applications	B	B	B	B		B

Note. Unfilled empty cell: Fulfill the requirement. B = Built required with default models; RB = Rebuilt needed with default model; C = Customization required with default models; T = Tuning required with default models; NA = Theoretically infeasible and non-applicable features.

Discussion

The commercially available and open-source modeling and simulation tools that we have evaluated for the electric grid domain are not designed for HF research. HF researchers interested in conducting studies in this domain need to carefully evaluate the reliability and representativeness of both the frontend and backend simulation and integrated technologies, as they have the potential to impact the results of HF studies. This evaluation often requires active information search, interviews, and analysis of available documents and technologies because engineers and developers may not thoroughly understand the aspects that can affect human performance in studies involving human subjects. In such cases, they rely on our expertise for judgment.

In HF, assumptions about the reliability and representativeness of technologies and simulators are sometimes made, leading to delays and potential implications for research study results and conclusions. It is crucial for us to comprehend the characteristics of the work domain and the tools that we are using so that uncertainties associated with false assumptions can be minimized. In the future, we will be developing scripting tools with the capability to connect to customizable UIs and simulation models, ensuring the fulfillment of HF research needs for electric grid operations.