Validating Measures of Team Reorganization and Interdependency in Response to Uncertainty in an Air Battle Management Task

Abstract

Teams working in complex, high-stakes environments may encounter uncertain situations for which they are not trained and can suffer dangerous consequences if they fail to overcome such uncertainty. We focus on how reorganization and dynamic interdependency across communications and air battle management (ABM) assets in response to uncertainty can be quantified in dynamic task environments. We analyzed data from a 5-day experiment conducted in an ABM task scenario to validate metrics of team reorganization and dynamic interdependency. Interactions quantifying reorganization and interdependency across technological task components significantly predicted team performance and higher interdependency of team communications also predicted better performance. Our findings indicate that reorganization and interdependency, primarily across technological assets, may be valid predictors of team effectiveness. Practical implications of this work primarily relate to assisting teams to achieve adaptive team-level proficiencies to uncertainty-inducing perturbations by providing objective feedback on reorganization and interdependency during team training.

Introduction

Teams working in complex, potentially dangerous task environments may encounter various forms of uncertainty that can impair team performance and pose danger to team safety. Teams, defined as two or more skilled team members working interdependently toward a common goal over a limited timespan (Salas et al., 1992), can suffer dangerous consequences if they fail to overcome uncertainty introduced by unforeseen impediments to team functioning. We focus on how reorganization across communications and air battle management (ABM) assets in response to uncertainty can be quantified in dynamic task environments. We use dynamic interdependency and reorganization metrics to predict team effectiveness in an ABM dataset (Strang et al., 2011).

Interdependency, a characteristic of a team where each teammate is dependent on other teammates to complete their work (Wageman, 1995), is a central aspect of effective team functioning, with previous research demonstrating that team interdependency is a significant moderator of team performance (DeChurch & Mesmer-Magnus, 2010; Kozlowski & Ilgen, 2006). Our approach does not assume that team interdependency is a moderator of team performance; rather, we conceptually place interdependency as a dynamic and fluctuating feature of team cognition, defined as cognitive processing at a team level (Cooke et al., 2013). This perspective is grounded in interactive team cognition (ITC), a theoretical perspective that proposes that team interaction is team cognition (Cooke et al., 2013). There are three propositions of ITC influencing our measurement approach: team cognition (a) is an activity; (b) should be measured at the team level; and (c) is inextricably tied to environmental context. ITC influences our approach to interdependency by placing the measurement of interdependency in real-time team interactions that fluctuate in response to changing team task constraints. A real-time metric of interdependency operationally defines a construct in ITC called “teamness” (Cooke et al., 2023; Fletcher & Sottilare, 2018), which describes how teams may appear to be more ”team-like” to match dynamic task requirements.

Our measures utilize real-time team interaction data to quantify reorganization and increasing interdependency in response to uncertainty. This measurement approach is: (a) grounded in teammate interaction, measured as an ongoing, continuous activity; (b) derived from real-time team interactions; and (c) intended to detect responses to uncertainty-inducing challenges. A measurement technique called layered dynamics (Gorman et al., 2019) is utilized to capture team interactions across technological assets and communicative team elements. Layered dynamic calculations quantify team reorganization and dynamic interdependency and predict team performance in the ABM task.

To investigate how team responses to uncertainty and dynamic interdependency contribute to team effectiveness, we measured dynamic interactions from teams working in an ABM scenario, the Distributed Dynamic Decision-making (DDD) task (Strang et al., 2011). Teams of five participants collaborated to protect a restricted zone from enemy aircraft. Team members utilized various collaboration technologies to communicate and coordinate to prevent enemy aircraft from penetrating that zone. Technological (asset) movement and activity data and verbal communications were the inputs for layered dynamics calculations, which quantified reorganization and interdependency using entropy and inverse sample entropy. This work investigated the relationship between team effectiveness, reorganization in response to uncertainty, and dynamic interdependency.

Validation Study

A validation study was performed to validate the capability of the proposed measures to quantify uncertainty and interdependency in a dynamic team task. The study used archival data collected on teams operating in an ABM task scenario (MATRIX; Strang et al., 2011). We propose that the results of the current study, therefore, inform hypotheses and measure development for future ABM experiments.

In this study, there were several experimental manipulations, including one between-subjects factor and three within-subjects factors. The between-subjects factor was team role, in which a team member could be a weapons director, sweep operator, or tanker operator. The within-subjects factors were task demand (which could be standard or high), team communication technology type (radio-only or augmented communication), and resource display (graphical or tabular). This analysis included team communication technology type and resource display variables in a regression, via dummy-coding, and excluded the task demand variable, in the first set of hierarchical regressions.

Additionally, individual difference-related variables were incorporated in a second set of hierarchical regression models. The first variable was performance-related differences due to which group of individuals, out of 21, completed the task. This variable was referred to as Group. The second variable, labeled Team, denoted the day, out of five, of data collection. A between-subjects factor for team role was not included in this analysis. This is because the focus of this study is validation of measures as applied to the team as a whole. The focus was not on the effects that different roles have on these measures. All study hypotheses are reported later, after detailing the metrics to provide sufficient context.

Method

Participants

In this experiment, there were 21 teams composed of five participants. Participants were between 18 and 30 years of age (M = 21.94, SD = 3.16 and were composed of 70 men and 35 women). Researchers for this study recruited participants from local universities and a temporary work agency in the state of Ohio. Participants were financially compensated for their participation. All participants gave their informed consent before participating in the study, and all procedures were approved by an Institutional Review Board prior to data collection.

Apparatus

Distributed Dynamic Decision-Making (DDD) ABM Simulation Software

This task was developed using the Distributed Dynamic Decision-making (DDD) software (Figure 1), version 3.0 (Strang et al., 2011). Previous work on this dataset has focused on the effects of cross-training team member knowledge (Lyons et al., 2018), the temporal regularity of communication patterns using sample entropy (Russell et al., 2012), an analysis of patterns of communication codes to detect changes in team communications that are driven by experimental manipulations (Russell et al., 2012), and a comparison of different collaboration technologies for effective team performance (Strang et al., 2011). In the DDD task scenario, teams collaborated to protect a friendly zone from enemy aircraft. Teams had to shoot down enemies before they could penetrate friendly (yellow) and engagement (red) zones (Funke et al., 2012; Russell et al., 2012).

Figure 1.

Display of DDD task environment.

As seen in Figure 1, enemy aircraft first appear in a gray zone (right), and the team had to shoot down enemies before they breach the yellow (middle) or red (left) zones. Team members communicated through a variety of collaboration technologies to manage assets (e.g., airplanes) to attack hostile aircraft.

Procedure

This research study took 6 days to complete (Strang et al., 2011). The first day consisted of a day-long training session. The five subsequent days consisted of experimental sessions. The experimental sessions took 8 hr to complete. Participants were delegated to three different roles among five different team members. Two participants were weapons directors. One participant was designated as the weapons director, who was color-coded blue, and the other was designated as the weapons director, who was color-coded green.

The responsibilities of weapons directors were to communicate action planning information to their team members. This could include refueling and how to target and attack enemies in a strategic manner. Two other participants, also designated as “blue” and “green,” were sweep operators. The responsibility of the sweep operators was to control the team’s aircraft. Finally, there was one tanker operator. The tanker operator oversaw the controlling of the team’s two tankers for refueling. Further details regarding the exact steps of this study’s procedure can be found in Funke et al. (2012) and Strang et al. (2011).

Measures

Team Performance

Team performance was a composite score that reflected how well the team performed using a variety of sub-scores related to the team task. This overall score consisted of an air defense score, an identification score, and a refueling penalty score. The air defense score took into account how many assets the team lost, how many high-value assets were lost, and how often enemy aircraft penetrated the friendly (yellow) zone that the team was tasked with protecting (Funke et al., 2012). The identification score rewarded teams based on how well they could identify unknown enemy tracks. Finally, the refueling penalty deducted points from the team if they refueled an asset at the wrong tanker.

Measures of Team Reorganization and Interdependency

To quantify the amount of team reorganization, we measured the entropy of technological and communication layers. Information entropy has been used in prior research to measure team dynamics in a variety of critical situations (Gorman et al., 2019). In previous work, greater moving window entropy quantified team reorganization and was found to be correlated with team effectiveness during perturbations (Demir et al., 2019).

Sample entropy (SEn) was used to measure the amount of dynamic interdependency in teams, due to the fact that SEn is a value that depends on the team executing the correct actions in the proper order. Placing an emphasis on order, or sequence, is what makes SEn useful for quantifying dynamic interdependency. The SEn measure is like entropy because higher values of SEn mean that there is more variety or unpredictability within a sequence of numbers. However, SEn considers the temporal regularity of the numbers of symbols in a sequence. Technically, SEn quantifies the regularity of a pattern in a sequence by calculating the likelihood that a pattern or sequence will “continue to repeat when the length of those sequences is extended by a single point” (Strang et al., 2012, p. 474). To quantify interdependency, we took the inverse of this value, resulting in a measure of interdependency referred to as inverse SEn.

Layered Dynamics

This current work applied layered dynamics measures (Gorman et al., 2019) to generate technological- and communication-based measures of uncertainty and interdependency. To generate technological-based measures, a layered dynamics analysis was conducted on the actions of the aircraft. These actions reflected the actions that the team members used to control their assets. This first layer was called the technological layer. In addition, a layered dynamics analysis was applied to radio-based communications between teammates. Previous research (Strang et al., 2012) using this dataset categorized radio-based communications according to semantic content. The inputs for the communication layered dynamics analysis were these communication codes.

For the technological layer, the actions of the aircraft were converted to binary symbols. This was done in the same manner as Gorman et al. (2019) to ensure that the data constituted non-mutually exclusive but exhaustive coverage of all possible actions of the aircraft. The time series of entropy (team reorganization) and inverse sample entropy (dynamic interdependency) was applied to this series of binary symbols using a moving window entropy equation. For the communication layer, the unique symbols were directly input into the equations for entropy and inverse sample entropy.

Statistical Analysis

We conducted four hierarchical regressions with overall team performance as the dependent variable and reorganization to uncertainty and dynamic interdependency as independent variables. Exogenous variables were dummy-coded and included in these analyses to examine whether our measures predicted performance while controlling for the effects of experimental conditions and individual differences. Experimental conditions included different collaboration technologies, including a standard communication condition with radio headsets and an augmented condition with an integrated chat interface, as well as a standard tabular resource display versus a dynamic resource display. We also controlled for individual differences in which groups completed the study and examined performance differences attributable to the day of data collection.

Results

We examined how well entropy and inverse sample entropy predicted performance above and beyond performance-related individual differences, such as group of individuals and day of data collection. The entropy measure applied to technological task components significantly predicted team effectiveness while controlling for individual differences, but inverse sample entropy did not. These analyses did not yield any significant results with communicative task components.

Interactions across technological task components significantly predicted team performance, even when controlling for experimental manipulations. Team reorganization and dynamic interdependency were both positively related to team effectiveness in technological-based interactions. We also found that higher interdependency of team communications predicted better performance. Our findings indicate that reorganization and interdependency, primarily across technological assets, may be valid predictors of team effectiveness that are robust to communication technology differences and individual team training differences in the ABM task.

Our findings contribute to human factors and ergonomics by expanding on the use of real-time team cognition metrics (Gorman et al., 2020). The current work complements previous research that utilized communication and technological elements to generate interaction-based measures of real-time team cognition. Specifically, the results from this study extend previous research using layered dynamics (Gorman et al., 2019) by introducing a viable measure of dynamic interdependency. The team interdependency metric also connects with recent theoretical work in team cognition (Cooke et al., 2023), which may help operationally define the concept of “teamness” by quantifying why teams may appear more team-like.

Discussion

Practical implications of this work primarily relate to assisting teams to achieve adaptive team-level proficiencies to uncertainty-inducing perturbations by providing objective feedback on reorganization and interdependency during team training. Using the entropy metric, teams can be quantitatively assessed on how rapidly they reorganize their behaviors across communications and technological assets in response to uncertainty. If a team lacks this proficiency, they may produce a delayed response and fail to overcome perturbations. By incorporating metrics that capture real-time adaptation in response to uncertainty, training can encourage teams to become more adaptive and effective (Gorman et al., 2020). Furthermore, the results from this work can help enable teams to achieve appropriate levels of interdependency at appropriate times to match increasing levels of uncertainty in the team task. We do not postulate that interdependency is always a necessary component for teams to perform effectively. However, we argue that teams vary their levels of interdependency during dynamic team operations, a viewpoint aligned with the theoretical concept of teamness (Cooke et al., 2023).

Valid real-time metrics of team reorganization and dynamic interdependency can provide objective feedback as well as opportunities for teams to be assessed on dynamic adaptation and reorganization competencies that they need to survive in the post-training environment. Coupled with perturbation training that intentionally implements uncertainty-inducing perturbations into a team training scenario (Gorman et al., 2010), the current measurement approach can help teams become proficient at responding to unexpected challenges and calibrating their levels of interdependency effectively.