Sage Journals: Discover world-class research

Abstract

Communication is critical to team coordination and interaction because it provides information flows allowing a team to build team cognition, which contributes to overall team performance. Recent advancements in large language models (LLMs) have enhanced AI’s capability to mimic human-like interactions; however, issues remain regarding the timing and sequencing of these communications. Using data from a remotely piloted aircraft system (RPAS) task involving human-AI and all-human teams, the current study employed comparative analysis to investigate communication timing and sequence. Findings indicated that while all-human and human-AI team communication dynamics may differ in terms of timing, it is the sequencing of communicative messages that predicts team performance. In this way, the current study hopes these communication analyses’ differences can provide feedback and suggestions to future adoption of AI as a teammate for team training and team operations.

Keywords

human-AI team team communication team performance

Introduction

Working in teams enhances efficiency compared to individual efforts because most work requires more cognitive and/or physical resources than a single person can provide (Gorman & Cooke, 2011). Teamwork has become increasingly prevalent in both military and non-military contexts (e.g., Chen, 2018; Gorman et al., 2018). Effective communication is a primary factor in successful teamwork, which has been tied to team performance outcomes (e.g., Cooke et al., 2003; Marks et al., 2001) and team cognition (N. J. Cooke et al., 2013).

Team communication encompasses verbal and nonverbal exchanges of information (Mesmer-Magnus & DeChurch, 2009) and has been studied as interdependent team behaviors that continuously influence team performance outcomes (Marks et al., 2001). Communication is considered critical to teamwork and coordination because it provides information flows allowing the team to build team cognition, which contributes to situation awareness, decision making, and action at the team level (N. Cooke et al., 2003). Communication frequency and specific communication strategies, such as standardized language and leadership statements, have been found to boost team performance (e.g., Foushee & Manos, 1981; Jentsch et al., 1995; Mosier & Chidester, 1991; Orasanu, 1990). Though the relationships between communication within teams and team performance depend largely on context (Urban et al., 1995), the studies described above illustrate communication being a foundational mechanism that contributes to effective teamwork.

In recent years, autonomous (AI) team members are beginning to be considered as stand-ins for human teammates. However, research has shown that AI team members may lack the communication skills that are required for effective team performance (McNeese et al., 2018). To better understand which aspects of communication an AI team member performs differently compared to a human team member, a previous study (Zhou, 2022) analyzed communication flow (who said it and when) and content (what was said) features of three-person teams that included all human teams and human-AI teams operating in a remotely piloted aircraft system (RPAS). To measure communication flow, the author calculated communication determinism on teammates’ messaging sequence. The results indicated that expert human teams have higher communication sequence patterning than human-AI teams (Zhou, 2022).

However, the results were opposite to an earlier finding that used the same human-AI teaming database (Demir et al., 2017), in which the researchers used a different metric of determinism and found that human-AI teams were more “rigid” (higher determinism) in their coordination dynamics compared to expert human teams, who showed “metastable” (intermediate levels of determinism) communication dynamics. While both studies used percent determinism to measure communication, one study focused on pure communication turn-taking sequencing (Zhou, 2022), and the other study took the timing of communication into account (Demir et al., 2017).

As large language models rapidly advance, AI teammates are becoming more adept at communicating in natural human language, closely mimicking real human interactions in terms of their content. However, these technologies may still be lacking in terms of the timing and sequencing of those interactions. To explore the dynamics of communication between human-AI and all-human teams more deeply, the current study conducts comparative analysis between all-human teams and human-AI teams using both timing and sequencing methods from previous studies (Demir et al., 2017; Zhou, 2022). The goal of the current study was to use a different RPAS human-AI teaming dataset to determine the relative importance of timing and sequencing of team communication when conducting communication pattern analysis by examining relationships between timing, sequencing, and team performance.

The Current Study

The current study aims to bridge the gap in understanding team communication by focusing on two hypotheses and one research question:

Hypothesis 1: The communication sequence flow will differ between human-AI teams and all-human teams, with human-AI teams showing higher determinism than all-human teams.

Hypothesis 2: The timing of communication will differ between human-AI teams and all-human teams, with human-AI teams demonstrating less flexible timing.

Research Question: Is the timing of communication a better predictor of team performance over and above communication sequence flow?

By examining these hypotheses and research question, the communication feature and pattern analyses in the current study can help determine how communication dynamics differ between human-AI teaming compared to all human teams and how those communication differences relate to team performance.

Method

Participants

Forty-two participants comprising 21 teams recruited in Atlanta, GA were divided into two conditions. Due to a data-saving error, one team’s data was not included in the analysis. Participants aged 18 to 31 years (average age = 20.55, SD = 2.97), had normal or corrected-to-normal vision and were fluent in English. The group comprised 21 males, 20 females, and one non-binary individual. Each participant received $10.00 per hr for their participation or earned course credit.

Materials

The experiment was conducted in the Cognitive Engineering Research on Team Tasks Remote Piloted Aircraft System Synthetic Task Environment (CERTT-RPAS-STE; N. J. Cooke & Shope, 2004). The team, comprising a navigator, pilot, and photographer, worked together to take good photos of ground target waypoints over a series of 40-min missions. The navigator role develops a dynamic flight plan and informs the pilot about waypoint details, including names, altitude restrictions, airspeed restrictions, and effective target radii. The pilot role controls and monitors the altitude, airspeed, vehicle heading, fuel, gears, and flaps of the Remotely Piloted Aircraft (RPA) and collaborates with the photographer to adjust altitude and airspeed for optimal picture-taking at various target waypoints. The photographer role monitors and adjusts camera settings to capture target photos and provides feedback to teammates about photo quality.

The navigator and photographer roles were performed by participants. The pilot role was performed by either an experimenter or an AI teammate pilot developed using the ACT-R cognitive modeling architecture to simulate human cognition and interact with the human teammates using a text-chat interface (Ball et al., 2010). Participants were informed of the AI teammate acting as a pilot in the first three missions and were instructed to interact using a strict language format, void of typos. An example of effective communication between the AI teammate and the human teammates involved the navigator sharing waypoint data with the agent, the AI pilot negotiating altitude and airspeed with the photographer, and the photographer sending feedback post-photograph. This feedback alerted the team to move to the next waypoint.

Experimental Design

Team Type was manipulated between subjects across two levels: all-human and AI teammate. All teams completed a single 6-hr session that included training and four 40-min RPAS missions. In the all-human condition, an experienced confederate performed the AVO role across the four missions. In the AI teammate condition, the AI pilot took the AVO role for the first three missions and was replaced by the experimenter for the fourth mission. The current comparative analyses use data from the first three missions. Mission is considered a within-subjects variable with three levels: Mission 1, 2, and 3. The two dependent variables are team performance outcome and team processing efficiency.

Procedure

Prior to arrival, each team was randomly assigned to an experimental condition, and participants were randomly assigned to task roles. After giving informed consent, participants completed a 30-min self-paced interactive PowerPoint training module specific to their role, followed by a 30-min hands-on training mission to familiarize themselves with the CERTT-RPAS-STE. Experimenters, adhering to a script, trained the participants to ensure they understood their roles, the tasks, and how to use the text-chat interface. Teams then proceeded through Missions 1 to 4, with short breaks in between each mission. After completing the missions, participants were debriefed and compensated for their participation.

Measures

Communication measures

A chat log recorded message sent and received times in seconds, the sender and receiver(s) of the message, and the content of the message. To analyze communication sequence, messages were sequenced according to send times to calculate discrete recurrence plots (Webber & Zbilut, 1994). Discrete team communication states taken from the set of mutually exclusive codes identifying which team member is sending the message and which team member(s) receives it are ordered in a sequence containing no timing information and analyzed using a recurrence plot.

To analyze communication timing, a multivariate time series coded which team member was actively sending at least one message during each 1-min interval of a mission. A joint recurrence plot (Marwan et al., 2007) was calculated using the coded communication timing as input.

Percent determinism (%DET) scores were then calculated using Equation 1 from the discrete recurrence plot and the joint recurrence plot for both the sequencing and timing data. %DET ranges from 0 to 100, with higher scores indicating more determinism.

% DET = (number of recurrent points forming diagonals) / (total number of recurrent points) x 100

(1)

Team performance outcome

Team performance outcome was calculated for every mission as the weighted composite of several system parameters, including duration of warning or alarm state, rate of good photographs per minute, fuel and film used, and the number of missed targets. At the beginning of each mission, each team had an initial score of 1,000, and points were deducted based on the final value of each system parameter (N. Cooke et al., 2020).

Team processing efficiency

Team processing efficiency was calculated as the average efficiency score for all targets in a mission. Target processing efficiency (TPE) scores were determined by the time teams spent within a target waypoint’s effective radius to capture a good photo. Higher TPE scores indicated greater efficiency. Each target started with a score of 1,000 points, with points deducted based on the number of seconds spent in the effective radius. An additional 200 points were subtracted if the team failed to capture a photo for that target (N. Cooke et al., 2020).

Results

We separately performed 2 (Condition: All-human vs. Human-AI Teams) × 3 (Mission) repeated measures ANOVAs on %DET of the discrete recurrence plot for sequencing and %DET of the joint recurrence plot for timing. Table 1 presents the results of the repeated measure ANOVA. There were no significant main or interaction effects for communication sequence, which failed to replicate the findings of the previous study (Zhou, 2022). However, the results on communication timing successfully replicated the findings of previous research (Demir et al., 2017), with a significant main effect of team type, F (1, 19) = 6.69, p = .02, $η_{p}^{2}$ = .26, indicating that human-AI teams exhibited comparatively more rigid coordination dynamics—reflecting inflexibility in their coordination patterns—by generating high determinism in their communication timing (See Figure 1). As in the prior study (Zhou, 2022), all-human teams exhibited metastable (i.e., between stability and instability) communication timing dynamics. The current results indicate that timing may differ more than sequencing between all-human compared to human-AI teams.

Table 1.

Analysis of Variance for %DET.

Source	df	F	p	Partial $η^{2}$
%DET (Discrete recurrence)
Between subjects
Condition (C)	1	3.68	.07	.16
Error	19
Within subjects
Mission (M)	2	1.77	.19	.085
M × C	2	0.63	.51	.032
Error	38
%DET (Joint recurrence)
Between subjects
Condition (C)	1	6.69	.02*	.26
Error	19
Within subjects
Mission (M)	2	2.63	.09	.12
M × C	2	.14	.87	.007
Error	38

Note. The entries in bold represent significant variables in the data set. *p < .05.

Figure 1.

Mean percentage of determinism score across the three missions for the two team types and two communication measures.

Regression analysis was applied to team performance outcome and team processing efficiency separately for sequencing and timing. Table 2 presents the results of the regression analysis. Consistent with previous findings (Demir et al., 2017; Zhou, 2022), the results indicated significant relationships between %DET scores using sequencing and both team performance outcome and team processing efficiency. None of the relationships between %DET scores using timing and the outcome variables were statistically significant. Nonetheless, the sign of the coefficients for a quadratic term indicated that increased communication timing may be negatively related with team performance.

Table 2.

Summary of Regression Analysis for Determinism Predicting Team Performance Outcome and Team Processing Efficiency.

Relationship between		B	SE B	$β$	t	p
%DET (discrete recurrence)
%DET and team performance outcome	Linear (DET)	10.8	3.36	.38	3.21	.002
%DET and team performance outcome	Quadratic ( $D E T^{2}$ )	.089	.029	.36	3.04	.003
%DET and team processing efficiency	Linear (DET)	28.89	7.88	.43	3.67	<.001
%DET and team processing efficiency	Quadratic ( $D E T^{2}$ )	.24	.068	.41	3.53	<.001
%DET (Joint recurrence)
%DET and team performance outcome	Linear (DET)	.56	2.15	.034	.26	.79
%DET and team performance outcome	Quadratic ( $D E T^{2}$ )	−.11	.018	−.33	−.36	.68
%DET and team processing efficiency	Linear (DET)	7.71	5.04	.19	1.53	.13
%DET and team processing efficiency	Quadratic ( $D E T^{2}$ )	−.06	.043	−.16	−1.30	.19

Overall, the findings from the current study suggest that while all-human and human-AI team communication dynamics may differ in terms of timing, it is the sequencing of communicative messages that predicts team performance.

Discussion

The current study’s examination of communication dynamics within human-AI and all-human teams contributes significantly to the field of human factors and ergonomics by highlighting the relationship between communication timing, sequencing, and team performance. The results indicate that so long as the sequencing is correct, performance was good in the RPAS task. Although there were significant differences in timing, with all-human teams arguably being more “flexible” in terms of lower determinism compared to human-AI teams, it might not matter for performance in this task, because timing did not significantly predict performance outcomes. On the other hand, higher levels of communication determinism for sequencing had a positive relationship with team performance, indicating that a more stable pattern of information sequencing enhances team performance, at least in this command-and-control RPAS task. These results align with previous research, which has consistently linked structured communication strategies to improved team outcomes. However, the lack of significant results for communication timing suggests that while flexibility in timing might be crucial for human satisfaction and safety, which were not evaluated in the current study, it is not a strong predictor of performance efficiency in this context.

Practical implications of these findings include the need for AI design improvements that enable effective communication in terms of sequencing of information and perhaps timing of communications. Specialized training programs for teams integrating AI may help humans navigate communication rigidity, and the development of systems that allow AI to modify communication patterns dynamically may lead to better team integration and human satisfaction with these systems.

Conclusion

The current study provides valuable insights into the communication dynamics of human-AI and all-human teams, specifically focusing on the impact of communication timing and sequencing on team performance. Our findings suggest that while timing flexibility differs across team types, it may be the sequencing of communication that plays a more critical role in enhancing team performance. These results underscore the importance of structured communication strategies and the need for advancements in AI communication capabilities.

Future research should explore the broader implications of communication timing, particularly its impact on human satisfaction and safety in various team settings. Additionally, further investigation is needed to understand how AI teammates can adapt their communication strategies to better align with human team members. By addressing these areas, we can develop more effective and cohesive human-AI teams, ultimately improving performance and satisfaction in collaborative human-AI tasks.

Footnotes

Acknowledgements

The authors are thankful to David Grimm, Matthew Scalia, Fiorella Gambetta, Yiwen Zhao, Anna Crofton, and Ansley Lee for their assistance in data collection, participant training, and data entry.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Shiwen Zhou

References

Ball

Myers

Heiberg

Cooke

Matessa

Freiman

(2010). The synthetic teammate project. Computational and Mathematical Organization Theory, 16, 271–299.

Chen

J. Y.

(2018). Human-autonomy teaming in military settings. Theoretical Issues in Ergonomics Science, 19(3), 255-258.

Cooke

Demir

Huang

(2020). A framework for human-autonomy team research. In Engineering psychology and cognitive ergonomics. Cognition and design: 17th international conference, EPCE 2020, Held as part of the 22nd HCI international conference, HCII 2020, Copenhagen, Denmark, July 19–24, 2020, Proceedings, Part II 22 (pp. 134–146). Springer International Publishing.

Cooke

N. J.

Gorman

J. C.

Myers

C. W.

Duran

J. L.

(2013). Interactive team cognition. Cognitive Science, 37(2), 255–285.

Cooke

Kiekel

Salas

Bowers

Stout

Cannon-Bowers

(2003). Measuring team knowledge: A window to the cognitive underpinnings of team performance. Group Dynamics: Theory, Research, and Practice, 7(3), 179–199.

Cooke

N. J.

Shope

S. M.

(2004). Synthetic task environments for teams: CERTT’s UAV-STE. In Handbook of human factors and ergonomics methods (pp. 476–483). CRC Press.

Demir

(2017). The impact of coordination quality on coordination dynamics and team performance: when humans team with autonomy. Arizona State University.

Foushee

H. C.

Manos

(1981). Information transfer within the cockpit: Problems in intracockpit communications. In Billings

C. E.

Cheaney

E. S.

(Eds.), Information transfer problems in the aviation system (Report No. NASA TP-1875). Moffett Field, CA: NASA-Ames Research Center.

Gorman

Cooke

(2011). Changes in team cognition after a retention interval: The benefits of mixing it up. Journal of Experimental Psychology: Applied, 17(4), 303-319.

10.

Gorman

J. C.

Grimm

D. A.

Dunbar

T. A.

(2018). Defining and measuring team effectiveness in dynamic environments and implications for team ITS. In Building Intelligent Tutoring Systems for Teams. Emerald Publishing Limited.

11.

Jentsch

F. G.

Salas

Sellin-Wolters

Bowers

C. A.

(1995, October). Crew coordination behaviors as predictors of problem detection and decision making times. In Proceedings of the human factors and ergonomics society annual meeting (Vol. 39, No. 20, pp. 1350–1353). SAGE Publications.

12.

Marks

M. A.

Mathieu

J. E.

Zaccaro

S. J.

(2001). A temporally based framework and taxonomy of team processes. Academy of Management Review, 26(3), 356–376.

13.

Marwan

Romano

M. C.

Thiel

Kurths

(2007). Recurrence plots for the analysis of complex systems. Physics Reports, 438(5–6), 237–329.

14.

Mcneese

Demir

Cooke

Myers

(2018). Teaming with a synthetic teammate: Insights into human-autonomy teaming. Human Factors: The Journal of Human Factors and Ergonomics Society, 60(2), 262–273.

15.

Mesmer-Magnus

J. R.

DeChurch

L. A.

(2009). Information sharing and team performance: A meta-analysis. Journal of Applied Psychology, 94(2), 535.

16.

Mosier

K. L.

Chidester

T. R.

(1991). Situation assessment and situation awareness in a team setting. In Designing for everyone: Proceedings of the 11th Congress of the International Ergonomics Association (pp. 798–800). Paris: International Ergonomics Association.

17.

Orasanu

(1990). Shared mental models and crew performance (Report No. CSLTR-46). Princeton, NJ: Princeton University.

18.

Urban

Bowers

Monday

Morgan

Jr . (1995). Workload, team structure, and communication in team performance. Military Psychology, 7(2), 123–139.

19.

Webber

C. L.

Jr. Zbilut

J. P.

(1994). Dynamical assessment of physiological systems and states using recurrence plot strategies. Journal of Applied Physiology, 76, 965–973.

20.

Zhou

(2022). Communication pattern analysis in human-autonomy teaming [Master’s thesis, Georgia Institute of Technology, Atlanta, United States]. http://hdl.handle.net/1853/67260

The Impact of Communication Timing and Sequencing on Team Performance: A Comparative Study of Human-AI and All-Human Teams

Abstract

Keywords

Introduction

The Current Study

Method

Participants

Materials

Experimental Design

Procedure

Measures

Communication measures

Team performance outcome

Team processing efficiency

Results

Discussion

Conclusion

Footnotes

Acknowledgements

Declaration of Conflicting Interests

Funding

ORCID iD

References