Abstract
A team of fighter pilots in a distributed environment with limited access to information rely on technology to pursue teamwork. In order to design systems that support distributed teamwork, it is, therefore, necessary to understand how access to information affects the team members. Certain factors, such as mutual performance monitoring, shared mental models, adaptability, and backup behavior are considered essential for effective teamwork. We investigate these factors in this work, focusing on how visually communicated information affects fighter pilots’ perception of these factors. For that, a questionnaire including the teamwork factors in relation to certain defined scenarios that contain various levels of information was distributed to fighter pilots. We show that the studied factors are affected by the level of information available to the pilots. Especially, mutual performance monitoring increases with the degree of available information.
Introduction
The purpose of the investigation was to study how the perception of teamwork factors varies depending on different amounts of available visual information. A questionnaire was used, based on selected components and coordinating mechanisms from the Big Five model for effective teamwork compiled by Salas, Sims, and Burke (2005), to investigate teamwork in a fighter aircraft environment in a team of fighter pilots. The Big Five model is based on empirical findings and was compiled from the vast body of teamwork literature with the aim of guiding future research.
The Big Five model consists of five essential teamwork components and proposes relationships to be investigated between these components. The five components of the model are team orientation, mutual performance monitoring, team leadership, backup behavior, and adaptability. In addition, there are three coordinating mechanisms defined that are considered necessary for the maintaining and updating of the five components, namely, mutual trust, shared mental models, and closed-loop communication The suggested model is generic and aims to capture the essence of teamwork regardless of team composition, task and environment. In a previous study, we studied teamwork in a different aviation environment – air traffic control, revealing how much the context impacts how the components emerge and relate to each other (Svensson, Ohlander, & Lundberg, 2019).
In this paper, we aim to further study teamwork in the fighter aircraft domain with the purpose of understanding how different levels of visual information affect the pilots’ perceived ability to collaborate. Does an increase in the level of information imply that the team members’ perception of teamwork also increases? How do the studied components vary in relation to information levels and are certain components more affected? This research aims to increase the knowledge of how to design pilot interfaces and cockpit functionality that better support pilots in their teamwork.
In this study, four of the components and coordinating mechanisms from the Big Five model were included. They were; shared mental models, mutual performance monitoring, backup behavior, and adaptability. The reason for this selection is that these four are expected to be affected by the level of information available during an ongoing mission. Mutual trust and team orientation are established before the mission starts and, hence, are expected to neither vary during mission execution nor be affected by the information available to the pilots (Ohlander, 2016). Closed loop communication was controlled in the study by not permitting voice radio and only allowing text messaging at one given point.
Furthermore, the questionnaire was constructed with the respondent taking the role of team leader with one team member (wingman) and no data was collected from the point of view of other team members; hence, the study was not designed to evaluate team leadership.
Method
The investigation was performed using a questionnaire based on the teamwork components and coordinating mechanisms as described in the “Big Five” model (Salas et al., 2005). This approach enabled us to collect data in a structured way. The questionnaire was distributed to 14 fighter pilots in Sweden, and we received answers from 9 of them. The respondents were all male, their ages spanned between 35 and 59 years, and they had an average of 1600 hours of experience flying fighter jets. The respondents were informed that their participation was voluntary and they could withdraw their answers at any time. The answers were anonymized during the further analysis.
In the questionnaire, the respondents were presented with two situations from a reconnaissance scenario. In the first situation (Fig. 1), they had just entered the area and started the reconnaissance mission. In the second situation (Fig. 2), they had advanced into the area, and a possible hostile threat was present. For each situation, three levels of an increasing amount of information were available. Ten statements were given for each level of information. The respondents were asked to rank to what degree they agreed with each statement by marking an integer number on a Likert scale from 1 (disagree) to 7 (fully agree). The questionnaire was in Swedish, as all participants were fluent in Swedish. In all cases, the pilot could see his own aircraft on a map and also what his radar observed. At level 1, the lowest level, the information available was the team members’ positions and an unknown object on the map. At level 2, the wingman’s flight data, such as altitude, speed, and fuel, were also given.
Situation 1 as presented in the questionnaire with increasing amount of information given from level 1 to level 3.
Situation 2 as presented in the questionnaire with an increasing amount of information, from level 1 to level 3.
Furthermore, a number of ships were added to the information on the map. Level 3 included all of the above information, together with a visualization of the wingman’s sensor area and the possibility of exchanging text messages between the team members.
The information levels were not intended to picture a real-world scenario in detail, but to represent three distinct levels with an increasing amount of information and a limited ability to communicate. It is not stated from where and how the information is gathered, i.e., it could be from a data link, flight control, etc. Level 1 is the lowest amount of information possible, providing just enough information for understanding the situation at all. At level 2, some data that typically would be shared via the data link is added, and at level 3, the possibility to send text messages is given. The questionnaire was designed with communication via voice radio disengaged. This was chosen since if using voice was made available, this would have been the preferred way to communicate, making the other data less conspicuous and probably affecting the answers. In real situations, it happens quite often that the ability to use the radio is limited. Hence, it is not an unrealistic scenario. Text messaging was added to level 3 to add an important modality for communication but in a more controlled and limited way, which would not override the rest of the information. In Table 1, the statements presented to the pilots to investigate the factors given by Salas et al. (2005) are presented. These statements were formulated with the intention of capturing the definitions of each factor given by Salas et al. (2005). Some statements are inspired and adapted from the questionnaire investigating teamwork among soldiers in Dutch Army peacekeeping teams (Duel, 2010).
The statements used in the questionnaire.
Statements 1, 2, and 3 were aimed to reflect shared mental models. Statements 4, 5, and 6 were aimed to reflect mutual performance monitoring. Statements 7 and 8 were aimed at reflecting backup behavior, while statements 9 and 10 reflect adaptability.
Observations and Analysis
The results presented in this section are based on a qualitative approach. All answers were entered in tabular format, and it was possible to sort and filter by respondent and question. Different views and graphs of the data were created to spot similarities and differences in the answers and make a qualitative analysis.
In the experiment, the four teamwork factors were measured by a ranking of ten statements. When we look at the responses for each statement, increased rankings for each level of information can be seen for each statement, with the largest difference between level 2 and 3. The same pattern occurs when we look at the combined rankings for each factor, increasing rankings for each level emerge for all the studied factors (Fig. 3). Especially between information level 2 and 3 there is a noticeable increase in the rankings. We also note that mutual performance monitoring and adaptability are the factors that display the largest increase, again, most noticeable between information level 2 and 3.
The responses per factor for each information level (from 1 to 3, x-axis). From left to right, shared mental models (SMM), mutual performance monitoring (MPM), backup behavior (BB), and adaptability (A).
Discussion
The rankings and notable comments from respondents will be discussed for each teamwork factor in relation to the definitions given by Salas et al. (2005).
Shared mental models
Shared mental models is included in the Big Five model as a coordinating mechanism and is defined by Salas et al. (2005) as “an organizing knowledge structure of the relationships among the task the team is engaged in and how the team members will interact.” The estimates for statement #1 regarding shared mental models, “I know how to act” are higher than for the rest of the statements. We note that the participants gave this factor a high value already at information level 1. The same pattern can be seen for statement #2, “I know how my partner will act.” However, the rankings start on a lower level, indicating that the certainty for what the partner will do compared to oneself is lower from the beginning. The rankings for statement #3, “I know when I need to adapt to the behavior of my partner.” are similar to statement #2. To be able to anticipate how team members will interact, the team needs to have an overall goal for the mission and a plan for how to reach that goal. Before a real mission, the pilots meet during the briefing and discuss how they will solve the task. There are usually backup plans in case something unexpected should occur. In addition to the plans discussed for the expected circumstances during the mission, there are several standard procedures for possibly occurring everyday situations that all pilots are obliged to be able to perform, and these procedures can be used in case of unexpected events and failures, for example. These standard procedures can be considered pre-fabricated building blocks to achieve flexibility in an unpredictable environment. The team can be flexible as long as the plan is valid towards the main goal or modifiable using the backup plans and building blocks.
The factor shared mental models can hence be regarded as consisting of a number of plans, including back-up plans towards the main goal, as well as a number of standard procedures. The more complex and elaborate the set of plans is, the higher the chances are for success during the mission. But for the plans to be useful, the team needs a decision on exactly which plan, or part thereof, will be executed in each situation. This decision is based on the current situation and the available information about the environment and the team. The question then is, will the estimation of shared mental models be higher with more information present, or does this only affect the ability to make a decision? Perhaps the shared mental models are what they are, not possible to affect during the mission since they are formed beforehand? The team leader is responsible for the decisions at the team level, and he/she needs a good view of the current situation to be able to do this. In this investigation, the respondent was acting as team leader and responsible for decisions at the team level.
Regarding the first question, “I know how to act”, there is no large difference between the answers for the different information levels. On average, the pilots’ rankings remain at the same levels regardless of the amount of information given. Hence, the notion of what to do is not so much considered to depend on the information level but rather on the preparations ahead. This is reflected in one pilot’s comment, “In accordance with a prepared plan for silent collaboration.” Some respondents were consistent in their Likert scale estimations, keeping the same estimate regardless of how much information they had available. Hence, these individuals seem to regard shared mental models as something “static”, at least during this mission.
For the second question, “I know how my partner will act”, the majority of the pilots once again kept the same estimate regardless of the available information. The exception is the level where the possibility to exchange text messages was added. One respondent commented: “Assuming that wingman actually notifies me”. This comment indicates that it is actually the ability to communicate rather than more information that helps in this case. A similar pattern can be identified for the third question, “I know when I need to adapt to the behavior of my partner”. Here, the level with text messaging available received higher estimates. Once again, this finding supports the notion that the ability to communicate is more valuable for creating better shared mental models than more visual information about the situation.
Mutual performance monitoring
Mutual performance monitoring is defined by Salas et al. (2005) as “the ability to develop common understandings of the team environment and apply appropriate task strategies to accurately monitor teammate performance.” To do this, the team members need information about the environment and each other. In a distributed environment, this information is gathered by sensors and delivered to each team member via technical solutions. Voice radio is one of these distribution means, and data shared via link is another.
The comments and answers to the three questions regarding mutual performance monitoring imply that radio communication substantially contributes to the factor, and data displayed on a screen is not enough. The second part of the definition, “… and apply appropriate task strategies to monitor teammate performance” points to the need to know what the current plan is in order to be able to assess the situation and the behavior of the team members. The estimates for statement #4, “I am able to track how my partner is progressing in his tasks.” and statement #5, “I can monitor and track that we solve the task as agreed”, clearly show higher rankings with increasing information levels. Furthermore, the increasing trend between levels is noticeable already between level 1 and level 2. Hence, more information about the partner gives more confidence in monitoring the progress. Regarding statement #6, “I can be prepared for everything that may affect our ability to reach our goal”, the results were more mixed. Several respondents maintained the same rankings regardless of information level. Hence, these pilots do not agree that the given information necessarily makes it easier to monitor the situation in order to be able to adjust their own behavior. At level 3, the ability to communicate via text messaging was added, which means a great advancement in coordinating the actions. Exactly what is going on, especially in fast-changing scenarios, is difficult to show visually on a display. This is where communication becomes important for team members to update each other on their view of the situation.
Generally, the pilots commented that teamwork is nearly impossible to pursue without the capability to talk on the radio. However, as several remarked, performing a mission under such circumstances as in the questionnaire depends on good planning.
Backup behavior
Backup behavior is defined in the article by Salas et al. (2005) as the “ability to anticipate other team members’ needs through accurate knowledge about their responsibilities. This includes the ability to shift workload among members to achieve balance during high periods of workload or pressure.” For the first statement regarding backup behavior (#7), “I can tell if my partner is in trouble”, the same trend with higher rankings at information level 3, as previously indicated, can be noted. Hence, also, in this case, the pilots seem to value the ability to communicate more than getting additional visual information. When analyzing the second statement for backup behavior, “I can help my partner if he is in trouble”, two groups of answers can be identified. The first group follows the same pattern as previously discussed; they give higher rankings when the ability to communicate is available. At the same time, some pilots gave the same rankings regardless of information level. This indicates that, in this case, more information does not help. It is the actual ability to do something, to perform an action that counts. What is actually meant by helping? This was not specified in the statement and one pilot asked: “What does it mean to help? To warn? Take action? Give information?” Still, this pilot gave higher estimates for higher information levels.
Adaptability
Adaptability is defined by Salas et al. (2005) as the “ability to adjust strategies based on information gathered from the environment through the use of backup behavior and reallocation of intrateam resources. Altering a course of action or team repertoire in response to changing conditions (internal or external).” Adaptability is a cornerstone in successful teamwork, especially for an action team such as in this analysis. Adaptability is the factor in the chain of factors that all the other factors in the “Big Five” model, in the end, contributes to. For the team to be able to plan and foresee everything in advance in a complex, hostile, and rapidly changing environment is simply not possible. One pilot stated, “Flexibility and apprehension of wingman’s actions are grounded in thorough planning and briefing.” It is, therefore, reasonable to assume that changing levels of information will affect the team’s ability to adapt. However, information is not the only contributing parameter to this factor. Shared mental models are the foundation for how the agreed plans can be accessed and agreed upon in an actual situation. The ability to adapt is also very much dependent on the available tools and capabilities of the team. The estimated level of threat also affects the preferred way of adaptation. Stress and high workload are factors that lower the ability to adapt. A comment from one pilot regarding adaptability in the second, more hostile situation was: “Most likely, our workload is high”.
The two statements regarding adaptability were, “We are able to adapt to new situations.” and “We are able to solve unforeseen problems”. The same pattern as has been previously noted can also be seen here. More information does not necessarily give a better ability to adapt. Several pilots kept the same rankings in all cases. Overall, collaborating and being adaptable were regarded as difficult during the given circumstances. One pilot noted “Low scores due to lack of voice communication.” Another commented, “Collaboration difficult due to lack of communication. Not possible to collaborate in a coordinated way.” While the ability to communicate increases the rankings, the same pilot commented “Communication and preplanned routines allow for better adaptability”. This pilot gave higher rankings for the level 3 estimates. Hence, according to him, planning and communication add more to adaptability rather than increasing the amount of visual information. One pilot stated regarding both backup behavior and adaptability, “The estimates are negatively affected by the absence of voice communication. No difference between level 1 and 2 regarding this. At level 3, the situation is slightly better; however, text communication is time-consuming and adds workload.”
Conclusions and Future Work
We have presented the results of an investigation of teamwork in a reconnaissance scenario involving fighter pilots. We identified a pattern of increasing rankings for all the studied factors when the degree of information detail increases. Mutual performance monitoring and adaptability are the factors that display the largest increase in rankings. Especially, when the ability to communicate via text messages is introduced, there is a noticeable increase in the rankings. Hence, the pilots perceive the studied teamwork factors to be strengthened with more information. In addition, we found that the pilots considered the lack of voice communication as severely affecting teamwork. Planning ahead of a mission is important in order to achieve flexibility and to be able to adapt, this cannot be sufficiently overbridged by adding more visual information.
Future work will include the study of optimal levels of information in order to address the risk of presenting the pilot with too much information.