Perturbation Detection in Space-based Human-Machine Teams (HMTs) in Different Layers

Abstract

This study focuses on detecting unique and complex challenges of Human-Machine Teaming (HMT) in space missions, where coordination among humans, robots, and AI agents is critical. Such missions are beset by “perturbations”—unexpected challenges involving communication delays due to the vast distances separating team elements. These issues must be overcome to maintain the safety and effectiveness of space-based missions. This study conducted engineering tests to evaluate the impact of perturbations on communication and vehicle operations and the physiological state of the humans involved. It outlines an experimental scenario involving various space entities (e.g., lunar colonies; orbiters; rovers) to simulate space mission conditions and communication and coordination challenges. The study employs layered dynamics methods to detect perturbations in technological and cognitive team states across the mission, using sensors and information entropy as key analytical tools.

Introduction

In Human-Machine Teaming (HMT), humans and machines collaborate to achieve common objectives. Space missions often require the integration of heterogeneous teams composed of humans, robots, and artificial intelligence agents. These entities must work together as a multiteam system to achieve mission objectives (Hambuchen et al., 2021; Yin et al., 2022). Coordinating teams across different geographic locations and times presents significant obstacles for missions in space and, more broadly, for diverse and multi-faceted team collaboration. Complexity and heterogeneous teaming in the space-based multiteam system can face unexpected challenges (“perturbations”) and communication delays. Variable communication latency can delay information exchange between team members, posing a challenge to real-time collaboration and decision-making. Challenges related to teamwork in space operations must be identified and understood to ensure safe and effective space-based missions. This study aims to evaluate the ability to monitor for and detect unique challenges associated with HMT in space operations. We used a novel testbed that creates a realistic space mission scenario simulating space teaming challenges revealed through quantitative interviews with astronauts and astrogeologists (Yin et al., 2022).

Space missions demand precise coordination between humans and machines, a challenge compounded by the complexity of extravehicular activities and the reliance on robotic assistance for tasks such as maintenance outside the International Space Station. The essence of this coordination challenge lies not just in the physical act of teaming but in overcoming the communication complexities that stem from multi-layered exchanges and physical limitations. Future innovations may lessen astronaut spacewalks but introduce new robotic operation and communication challenges, emphasizing the need for advanced technological solutions to address perception, and delay issues in space operations (Garcia, 2018; Sheridan, 2016). Space operations, known for their rigorous planning, must also accommodate unforeseen disruptions requiring on-the-spot strategic adjustments (Hodges & Schmitt, 2011; Schmitt et al., 2011). The core of this study focuses on developing and implementing space teaming metrics to help overcome these challenges. A novel testbed simulates the intricate dynamics of space operations, including the inevitable communication delays and the need for adaptation in the face of disruptions and achieve mission success. In highlighting these aspects, the study addresses the critical need for advanced technological solutions to enhance coordination and address perception and delay issues inherent in space missions. As a foreseeable next step in this project, the integration of AI offers promising avenues for enhancing team coordination and error management in these complex operational environments.

Methodology and Materials

This work employs a space mission scenario derived from interviews with experts in the field, including ex-astronauts and NASA mission control staff, as detailed in Yin et al. (2022). The simulated space mission scenario aimed at recovering an energy resource, termed “Enerphoto,” from the Moon and Mars was crafted to mirror the communication dynamics among distributed systems. The scenario includes: NASA mission control center (NASA MCC); Jet Propulsion Laboratory (JPL); a lunar colony of a human (Bravo) and a robot (Alpha) who is not trustworthy; a lunar orbiter; the International Space Station (ISS) with two humans (Charlie, Delta), one on a spacewalk (Charlie); a Mars Rover; and a Mars Orbiter. The Mars Rover is played by a Husky robot, and all other entities are played by human experimenters. A network with variable communication delay based on distance between entities was employed, and real-time space mission conditions were simulated to test the coordination routines and response to unexpected challenges. The mission took 30 min; besides the nominal tasks, the team also faced four perturbations (Charlie, Husky, Meteor strike, and Enerphoto). Regular mission communications and the interspersed perturbation challenges are described in the following Figure 1:

Figure 1.

Representation of the distributed space challenge, covering task environments on Earth, the Moon, and Mars.

Regular Communications

NASA MCC: MCC updates tasks for ISS-Delta every 3 min, who acknowledges. MCC also communicates new tasks to Bravo every 3 min and receives status updates or confirmations. Additionally, MCC tasks the Mars orbiter every 2 min, which then reports its position back to MCC with a 4-min delay.

ISS: Alongside regular updates with MCC, ISS-Delta provides status confirmations to MCC every minute and exchanges information with the lunar colony every 4 min to provide scientific support and coordinate exploration efforts. The ISS communicates continuously with the lunar and Mars orbiters to receive updates on positioning every minute.

Lunar Colony: Besides routine communications with MCC and ISS, Bravo interacts with the lunar orbiter and Alpha. Alpha explores and reports back new findings and positions, which Bravo confirms.

JPL and Mars Rover: JPL controls the Mars Rover, issuing coordinates and receiving positioning data with a 4-min delay. The Rover also sends its positioning data to the Mars orbiter every minute.

Challenges / Perturbations

1. Charlie: During a spacewalk, Charlie unintentionally becomes untethered. Mission MCC supersedes communications for the entire team, except for updates from ISS Delta and Charlie, starting at 6 min.

2. Husky: Husky encounters an obstruction, requiring the Mars orbiter to stop immediately to assess the situation, happening at 11 min.

3. Meteor Strike: This event involves an asteroid hitting the moon, prompting Bravo to repair equipment critical for maintaining the oxygen supply. The lunar orbiter, while tracking Alpha’s location, finds discrepancies in Alpha’s reports, leading to doubts about its reliability. This incident occurs at 20 min.

4. Enerphoto: NASA’s MCC urgently directs teams on the Moon and Mars to collect Enerphoto samples to boost Earth’s energy reserves. The collection starts 5 min after the initial directive, with the orbiters confirming the acquisition of Enerphoto and beginning their return to the International Space Station, initiating at 26:30 min.

Task Environment

The main tool used to simulate an astronaut’s spacewalk in this study was the “Meta Quest Pro” Virtual Reality (VR) headset. This VR system immersed participants in an astronaut’s spacewalk using the “Home—A VR Spacewalk” software developed by the BBC, designed to echo NASA’s real training protocols from 250 miles above Earth. The Husky Unmanned Ground Vehicle (UGV) also effectively simulated a Mars rove’s functions in a controlled environment. UGV played a crucial role in this setup due to its flexibility, design, and durability. This UGV was integrated with a Motion Capture (MOCAP) system.

Real-time biometric data, such as the spacewalker’s heart rates and Bravo’s, were monitored and collected using the iMotions platform (iMotions, n.d.). The research also incorporated the Cognitive Engineering Research on Team Tasks Remote Piloted Aircraft System Synthetic Task Environment (CERTT-RPAS-STE; Cooke & Shope, 2004), which is tailored for team missions. However, for this study, only the pilot’s role was simulated to reflect the dynamics of the Mars and lunar orbiters.

Throughout the experiment, a specialized push-to-talk communication system was used. This setup allowed participants to selectively communicate directly with any individual, with controllable communication delays, enhancing the simulation’s realism, and operational accuracy.

Measures

This study selected eight participants from Arizona State University (ASU). Additionally, the researchers themselves took on the role of primary participants. This dual participation was recognized and approved by the Institutional Review Board (IRB) prior to the start of the research.

Two iMotion units, including shimmer ECG systems and electrocardiogram (ECG) software, were used to collect ECG data from two individuals for each mission. Positioning and control data were collected from the Husky/Rover and the orbiters. Orbiter positioning relied on data from two of the Remotely Piloted Aerial System testbeds. Table 1 shows the sensors measured during the scenario.

Table 1.

Measured Sensors.

Entity	Sensor 1	Sensor 2
NASA MCC	COMMS (to, from, timestamp)
Jet propulsion lab	COMMS (to, from, timestamp)	Commands/positioning of Mars Rover
International space station (2)	COMMS (to, from, timestamp)	Heart rate variability during spacewalk
Mars orbiter	COMMS (to, from, timestamp)	Commands/positioning of orbiter (UAV)
Moon orbiter	COMMS (to, from, timestamp)	Commands/positioning of orbiter (UAV)
Lunar surface colony (2)	COMMS (to, from, timestamp)	Heart rate variability during asteroid hit
Mars Rover		Positioning/actions/signals from Rover

We utilized a layered dynamics approach to assess real-time team activity at various levels (Gorman et al., 2019), which models simultaneous variations across states of different system layers over time. This method is particularly effective in capturing and analyzing variations in activities across orbiters (vehicles), robots (rovers), communication among team members, and heart rate fluctuations measured via IBI.

Information Entropy

Moving window information entropy was employed to measure various system states (Shannon & Weaver, 1949). This metric assesses the variety of system states within a specified time frame, or window size. In the experiment described, the window size was set at 60 s and updated every second (1 Hz).

$$Entropy=-{\sum }_k^k{p}_k\times lo{g}_2{p}_k$$

(1)

In Equation 1, p_k is the relative frequency of each team communication, vehicle, or physiological state k ∈ K within a time window. The entropy metric was continuously recalculated using a moving window. The informational value (log₂p_k), measured in bits, of an event is weighted by its probability of occurrence. We normalized entropy by dividing entropy by the maximum possible entropy at each time point to generate a %Max Entropy value between 0% and 100% (Gorman et al., 2019).

Team-level Communication States

represent the unique combination of different communicative (speaker to receiver) or vehicle layer states. During normal team operations, the total number of possible intersections is expected to be lower than in periods of induced perturbations. Previous research has indicated that increased variety, measured through entropy, of intersections occurs more frequently during perturbations than during normal team functioning (Gorman et al., 2019; Grimm et al., 2023). Figure 1 illustrates how team-level intersections in the communication layer are formed. Individual communicative states are horizontally concatenated each second to create a unique system communicative state. In Table 2, each possible communication state is represented with a binary symbol and listed in the column titled “Component.”

Table 2.

Communication Intersections.

Component	Sample (1 Hz)
	1	2	3	4	5	6	7	8
ISS delta → NASA MCC	101	101	101	000	000	000	000	000
Lunar Bravo → Lunar orbiter	1110	1110	000	000	1110	000	1110	000
Mars orbiter → JPL	000	000	10011	10011	000	000	000	10011
Communication intersections	1011110000	1011110000	10100010011	10011	1110000	0	1110000	10011

Results

The layered dynamics approach was successful at detecting perturbations in communication patterns. The result shows that we could detect the perturbations in communication variety during the perturbation periods. This is indicated by the blue line representing crossing an upper threshold (99% confidence interval) during the perturbations. Upward spikes in entropy values indicate a greater variety of communication states following and during perturbations, whereas downward spikes indicate moments of decreased communication variety.

As shown in Figure 2, beginning with the blue zone, “Charlie Untethered” occurs between 368 and 720 s. A significant entropy peak is noted around 490 s, likely indicating the team’s response to Charlie’s situation. After this peak, entropy levels decrease, representing the recovery phase; however, this spike did not exceed the confidence interval needed to calculate recovery time. Three distinct entropy spikes are observed in the green zone, labeled “Husky Roadblock” from 660 to 990 s. The pink zone, “Meteor Strike and Alpha Bot is Untrustworthy due to Cyber Attack,” spans from 1,203 to 1,560 s. The fluctuations here are less pronounced compared to the previous two events, suggesting the team might be adapting to these disturbances. Finally, in the brown zone, “Enterphoto Found” between 1,590 and 1,799 s, the graph shows a smaller increase in entropy.

Figure 2.

Perturbation detection in communication network.

Figures 3 and 4 show the windowed entropy computed over the vehicle layer time series. This is the moving window entropy of system states across positioning data, airspeed, altitude, and turning. The presence of a perturbation for Mars orbiter is reflected by the increased variety (entropy spike) in the emergency stop period. Additionally, the landing was also detected (Figure 3). During the perturbation, in which the Orbiter receives an energy system alert regarding an emergency stop, the vehicle automatically adjusts airspeed and altitude to simulate countermeasures in response to this perturbation. Subsequently, a high degree of variety is exhibited in the communication layer as the operators correct the problem and coordinate the landing. The vehicle sensors also varied greatly during the landing. To simulate slowing down and landing the orbiter, both airspeed and altitude were dropped toward their minimum values. During times with no perturbation or landing, entropy was consistently lower. The results support the objective of real-time perturbation detection and interaction orchestration across distributed space team operations. In preparation for landing, the Orbiter also loses connection to the ISS. Communication between the Orbiter and ISS must now go through the MCC. The space crew had to adjust their communication patterns to overcome this perturbation. Consequently, the MCC became a very important teammate and acted as a communication conduit between the two other space team components. By adjusting these communication patterns, the team increased its communication variety, which was detected by our metrics.

Figure 3.

Perturbation detection in Mars orbiter vehicle trajectory and positioning data.

Figure 4.

Perturbation detection in Lunar orbiter vehicle trajectory and positioning data.

Finally, Figure 5 shows the detection of the asteroid strike perturbation in Bravo’s heartrate data. This corresponds to the need for Bravo to quickly find a solution to rebuilding the breathers on the Lunar Colony that were damaged in the asteroid strike. Note that Figure 5 displays another measure of variety, moving window percent determinism (%DET; Gorman et al., 2020). This measure is inversely related to entropy, such that extreme drops in %DET are associated with extreme increases in variety. We could not analyze Charlie’s heart rate data during the spacewalk due to data collection issues. Husky’s positioning data did not result in any significant findings.

Figure 5.

Asteroid strike detection in Bravo’s heart rate data (%DET).

Conclusion

The research findings highlight the effectiveness of using layered dynamics and moving window entropy (simulating real-time detection) to detect space mission team dynamics disruptions, focusing on communication, vehicle operations, and human physiological state. The finding suggests that human-machine teams could employ these perturbation detection strategies to manage stressors, which could enhance overall system effectiveness. This means such teams are better at addressing, rebounding from, and possibly averting the detrimental effects of disruptions on team functioning and outcomes. The study successfully identified variations in communication patterns and vehicular operations indicative of perturbations by employing a layered dynamics approach. This methodology could significantly enhance real-time monitoring and decision-making in distributed space teams, potentially allowing an AI agent (not analyzed here) to detect and manage such perturbations in the future. With the increasing integration of AI and automation in various industries, the practical implications suggest a new frontier in AI-assisted space mission management, where advanced monitoring of team interactions becomes a pivotal component of mission success. This could potentially reduce the cognitive load on astronauts and mission controllers, thereby increasing mission safety and efficiency. The success of such technologies could also be translated into other domains where remote team coordination is critical, such as deep-sea explorations or disaster response scenarios, suggesting a wide-ranging impact of the research.