Abstract
This study investigated whether experience in maritime operations contributed to situation awareness (SA) and confidence among Fast Rescue Craft (FRC) operators during simulated maritime search and rescue (SAR) missions. A total of 20 novice and 20 experienced Canadian Coast Guard personnel were presented with collision avoidance scenarios of various difficulty levels on a desktop FRC simulator. A goal-directed task analysis (GDTA) was conducted to identify the critical goals, decisions, and information requirements underpinning FRC operations, providing a structured basis for scenario design and SA measurement. The results indicated that experienced operators had significantly higher Total SA scores. These differences were primarily attributable to stronger performance on Level 3 SA across all scenarios and Level 2 SA in head-on scenarios. Experienced participants also reported higher confidence in Level 1 and Level 2 SA, although no differences were found in Level 3 or Total SA confidence. Experienced operators’ navigation decisions were influenced by informal decision-making cues, especially when interpreting collision-avoidance regulations. The absence of significant differences in Level 3 SA confidence and Total SA confidence between experienced and novice operators suggests that the latter may be overconfident in predicting future events in complex maritime environments. To better prepare novice operators for real-world SAR operations, these findings suggest the potential value of training interventions that focus on specific SA components, particularly projection, and support the development of decision-making strategies under uncertainty.
Keywords
Introduction
Maritime search and rescue (SAR) operations are hazardous and often require a Fast Rescue Craft (FRC). In such scenarios, decision-making can mean the difference between saving lives and endangering them. FRC operators face the challenge of searching for, identifying, planning, navigating, and safely reaching potential victims under stressful conditions, often with inadequate information and conflicting priorities, such as swiftly reaching those in distress while ensuring the safety of their crew (Haasnoot et al., 2013). Rigid-hull inflatable boats are typical vessels (i.e., FRCs), commonly used by the Canadian Coast Guard (CCG), that provide versatility in shallow waters, and can carry three responders and several casualties (Garvin & Harris, 2021; Haasnoot et al., 2013).
Maintaining situation awareness (SA)—defined by Endsley (1995a) as “… the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future” (p.36)—is crucial for avoiding FRC collisions in unpredictable environments. These three components correspond to Endsley’s three levels of SA: Level 1 (perception), Level 2 (comprehension), and Level 3 (projection). Endsley (1995a) emphasized that achieving high SA across these levels is the product of an accurate mental model of a dynamic task (Endsley & Jones, 2011). In this respect, the Recognition-Primed Decision (RPD) theory posits that experienced decision-makers use internalized mental models, derived from prior experience, to recognize situational patterns that are used to simulate potential outcomes, thereby enabling rapid and effective decision-making (Klein, 1998). Building on the concepts of RPD theory, SA provides a framework to assess how FRC operators perceive, comprehend, and project situations. Examining the differences between novice and experienced practitioners in dynamic cognitive tasks offers insight into how SA develops, revealing how experience shapes mental models and decision-making processes in information-rich environments (Endsley & Jones, 2011). The Situation Awareness Global Assessment Technique (SAGAT; Endsley, 1995b) is a widely recognized tool for measuring SA in real-time dynamic simulated environments. SAGAT involves pausing the simulation at random intervals to ask participants questions that evaluate their SA across three levels.
There is growing recognition that SA involves more than what is captured by the SAGAT alone, as it may overlook the dynamic, contextual, and distributed elements of cognition in complex systems (Ardecani et al., 2025; Bracken et al., 2021; Stanton et al., 2015). However, SAGAT remains a valuable tool for identifying SA gaps related to training needs and enhancing system design to improve operator performance and safety (Endsley, 2020). As Endsley (1995b), Endsley and Jones (2011) noted, SAGAT can support training by identifying an operator’s SA deficiencies. These insights can inform simulation-based training intervention development.
In this respect, the Transportation Safety Board of Canada recently concluded that poor SA and response planning contributed to the capsizing of a maritime rescue vessel during a SAR response on the Lachine rapids in Montreal, Canada, with four firefighters on board, resulting in one fatality (Transportation Safety Board of Canada, 2024). The hazardous environment inherent in maritime SAR operations demands that the vessel operator, known as a coxswain, maintain an acceptable level of SA while executing dynamic and occasionally high-risk maneuvers. Factors such as operational stress, low visibility, weather (e.g., high winds, large waves, and strong currents), vessel traffic, underwater hazards, and the location of casualties all heighten SAR risk (Haasnoot et al., 2013). These factors highlight the need for accurate SA in FRC operations, while research on SA requirements for larger vessels further illuminates how training and decision-making frameworks can influence navigation and collision-avoidance strategies.
Background
Previous studies have highlighted the SA requirements for larger vessels and their training implications (Chauvin et al., 2008, 2009; Chauvin & Lardjane, 2008; Sharma et al., 2019; Wickens et al., 2020). For example, Chauvin and Lardjane analyzed the decisions made by watch officers on ferries in the Dover Strait, focusing on the strategies underlying the action choices in various situations. Using the RPD model, this study identified critical cues, goals, and rules used by officers, revealing differences in behavior between slow and fast vessels and highlighting instances where informal rules influence decisions. The study analyzed Dover Strait vessel maneuvers and think-aloud data from simulated ferry encounters to examine how situational factors influence maneuver choices using logistic regression and expert protocol coding of the answers. The study found that ferry watch officers often deviate from formal collision regulation rules based on situational factors, particularly in high-speed crossing scenarios, where altering the course to the port minimizes time loss without increasing the collision risk. Vessels that are supposed to hold their course (stand-on) often take action early if they expect the other vessel (give-way) not to follow the rules, reflecting flexible interpretations of the rules. The authors reason that the deviation behaviors are driven by shared strategies aimed at managing interactions efficiently while conserving cognitive resources, which is consistent with models of anticipatory decision-making and workload management. Their results demonstrate that such strategies are context-dependent, statistically predictable, and have practical implications for traffic monitoring systems and officer training.
In a similar study, Chauvin et al. (2008) investigated the impact of SA on the decision-making process of inexperienced watch officers in ambiguous maritime scenarios. The study involved 90 fifth-year officers-in-training from a French Merchant Marine Academy using a bridge ship-handling simulator. Each participant managed the same ferry in a controlled scenario based on a real-world crossing situation in the Dover Strait. The experiment followed a six-phase protocol over 43 minutes, including briefing, initial familiarization, active vessel handling, a pause for completing a SAGAT questionnaire, further vessel handling, and a final questionnaire on decision rationale. The data collected included participants’ SAGAT responses, reported decision strategies, and objective performance metrics, such as maneuver direction, course change magnitude, and closest point of approach. The experiment revealed that decision-making was primarily influenced by the interpretation of rules and anticipation of other vessels’ intentions, with many trainees performing maneuvers contrary to regulations. Their findings highlight the need for enhanced training focused on recognizing typical situations and selecting appropriate actions. The study also revealed significant deficiencies in Level 1 SA scores, as many trainees failed to accurately perceive critical information, such as distance and speed. Despite these gaps, decisions were frequently made based on superficial visual cues or generalized patterns of practice rather than a detailed situational understanding. Furthermore, the identification of divergent decision profiles, some of which lack logical rationale or are grounded in the misinterpretation of regulatory navigation rules, underscores the weak link between perceptual awareness and decision quality. These findings support the need for targeted training interventions to improve perceptual accuracy, rule comprehension, and risk-based decision-making.
Wickens et al. (2020) experimentally investigated maneuver decision preferences in ship navigation to avoid collisions while balancing safety, efficiency, and adherence to rules under trajectory certainty and uncertainty conditions. The experiment was conducted online, and the results showed that while participants generally avoided collisions, performance degraded under uncertainty, with a preference for speed control over heading control, with occasional violations of navigation rules.
Further exploration of SA in related maritime applications has revealed significant disparities between novice and experienced practitioners. For example, Randel et al. (1996) examined SA as a component of decision-making in a naturalistic setting that involved complex cognitive tasks. Differences in decision-making were analyzed between novice and experienced electronic warfare technicians in U.S. Navy ships. They conducted a comparative study in which novice and experienced technicians were assigned tasks that simulated realistic operational scenarios involving radar-emitter identification. Participants were evaluated based on their ability to recall radar signals, make decisions under time constraints, and apply operational rules consistently. This naturalistic design was chosen to reflect the cognitive demands of actual electronic warfare operations, revealing that experienced participants excelled at recalling radar emitters, making decisions based on superior SA, and consistently applying rules.
Finally, Chauvin et al. (2009) investigated the effectiveness of training programs for trainee watch officers in avoiding collisions
Endsley (2020) also underscored the importance of integrating a confidence metric into the SAGAT to enhance the assessment of a participant’s mental model. This addition provides insights into how practitioners’ confidence in their own SA influences their performance. Notably, previous research by Kruger and Dunning (1999), commonly referred to as the Dunning-Kruger effect, has shown that novices tend to overestimate their competency compared to more experienced practitioners, highlighting a critical consideration in the context of FRC operations. Additionally, Walkowiak et al. (2023) found that the Dunning-Kruger effect was present in navigational tasks, influenced by factors such as national origin and gender. In the context of simulations that rely on visual input from participants, Chen et al. (2024) demonstrated that the Dunning-Kruger effect extends to visual reasoning and judgment. Rousseau et al. (2010) contend that subjective SA-related measures, such as confidence ratings, reflect an individual’s cognitive control and strategic self-evaluation rather than the actual content of their SA. When integrated with objective assessments such as the SAGAT, these measures can identify instances of overconfidence, which is an important concern in cognitively demanding tasks such as FRC operations and training in hazardous environments. These studies highlight the crucial role of SA in maritime decision-making, showing that experienced operators excel in applying rules and making informed decisions, whereas novices often overestimate their abilities, underscoring the need for better SA training and confidence calibration.
To the best of our knowledge, no study has focused on the SA requirements of FRC operators during SAR missions. Unlike larger vessels, FRCs are fast and maneuverable, operating in shallow waters near vessel traffic (Haasnoot et al., 2013). This distinction is important because the behavior of other vessels appears to be different from that of a smaller, more agile FRC. Therefore, the purpose of the current study was to examine the differences in SA between experienced and novice FRC operators to potentially accelerate FRC training for novices. We hypothesized that experienced FRC operators would demonstrate higher SA scores across all three SA levels (perception, comprehension, and projection) compared to novices (H1). Additionally, we hypothesized that experienced participants would show higher confidence in their SA compared to novices (H2).
Method
Participants
Participants were Canadian Coast Guard personnel recruited from large vessel ship crews, Inshore Rescue Boat (IRB) seasonal employees, Marine and Hazardous Material Response (MEHR) personnel, and former active duty on-water personnel performing shore-based support functions. The minimum training requirement for this experiment was established as the basic small vessel operator courses Rigid Hull Inflatable Operator Training (RHIOT)/Fast Rescue Craft (FRC) or equivalent, and any additional training or practical experience was considered a contribution to the participant’s experience category. The experience categories were predetermined before recruitment took place. While some variation in operational exposure and training emphasis existed across individuals, this reflects the real-world diversity of the operator population. Nonetheless, the baseline qualification ensured that all participants possessed the necessary competencies to interpret the scenarios, apply relevant maritime rules (e.g., COLREGs), and operate within the constraints of a SAR tasking environment.
The number of participants required for the experiment was determined based on a simulation-based power analysis using the superpower package in R (Lakens & Caldwell, 2021). The analysis was based on a three-way Analysis of Variance (ANOVA), with two experience levels of participants, three conditions of difficulty, and three collision avoidance scenarios with repeated measures on the latter two factors. Statistical Power was calculated to find a minimum of 10% difference between experience levels in the easy conditions and a 15% difference between experience levels in the moderate and difficult conditions. In addition, the correlation between repeated measures was set to 0.5, the standard deviation to 2.0, and alpha to 0.05. The analysis showed that 20 participants per group would provide 81.9% power to detect an experience level effect (Cohen’s f = 0.48). Therefore, 40 participants were recruited for this experiment. Supplementary material and the R code for this analysis is available on the OSF website (https://osf.io/8n7yq/?view_only=c53bfd302cb54ea2a1221ba5c9d0b452).
Participants were divided into novice (n = 20; 10 females; mean age = 31.75 years, SD = 10.10 years) and experienced (n = 20; 3 females; mean age = 41.45 years, SD = 10.52 years) experience levels. Novice operators had the minimum training requirement of RHIOT/FRC training with less than 2 years of experience and 250 hours of operating in a marine environment (including formal education programs). Experienced operators had a minimum of RHIOT/FRC training with more than 4 years and more than 500 hours of operating in the marine environment (including formal education programs) using small maritime vessels. Participants were recruited to ensure that there were no ambiguous situations in which their experience and training hours fell between the two categories of novice and experienced. Participants signed an informed consent form prior to participation, and the research was approved by the Interdisciplinary Committee on Ethics in Human Research (ICEHR) at Memorial University (ICHER# 20222407-MI).
Subject matter experts (SMEs)
Three Canadian Coast Guard personnel were selected as subject matter experts (SMEs) to support the scenario and questionnaire development. All were active-duty SAR instructors with extensive operational experience as FRC coxswains (average age 51 years, range 45–54; average operational experience 22 years, range 12–31). They were recruited using the expert identification criteria outlined by Shanteau et al. (2002), which included professional experience, formal certification, social recognition, peer consensus, consistent performance, ability to discriminate relevant cues, and demonstrated knowledge.
The SMEs contributed throughout the study design and development process. They participated in a goal-directed task analysis (GDTA), simulation scenario development, refinement of SAGAT questions, and validation of scenario-task alignment. Their engagement included multiple in-person sessions, trial runs of the simulation, and questionnaires. Their dual role as domain experts and contributors to scenario calibration ensured that the experimental conditions and SAGAT items reflected the operational reality in FRC-based SAR environments.
Experimental design and questionnaire development
The situation awareness questionnaire presented to participants with correct answers indicated where applicable.
The GDTA findings guided the adaptation of the SAGAT questions to match the cognitive demands of realistic SAR navigation and collision avoidance scenarios. The SAGAT items were refined in collaboration with SMEs, as described in Section 2.2.
Scenarios and tasks
The scenarios were designed to require the participants to select the most direct route to a simulated flare while simultaneously maintaining compliance with collision-avoidance regulations and accurately identifying potential hazards. In cases where decision-making involved ambiguity, such as selecting between multiple plausible options, responses were classified as incorrect if they compromised the safety of the FRC or violated navigational rules without a clearly defensible rationale. All scenarios required the participant to start at the same position in the simulated environment and remain inside a narrow channel marked by navigational buoys and constrained at multiple points by hazards. Participants were permitted to leave the channel in accordance with collision avoidance regulations; however, such deviations required a clear threat to the vessel, with the expectation that the vessel would return to the channel as soon as it was safe to do so (Transport Canada, 2023; Rule 2b).
Description of Scenarios. Each Scenario was Repeated Twice With Different Variations.
Each participant viewed the scenarios in a unique random order generated using a random number generator, with only non-repeating sequences selected for use. The only difference between the two versions of each scenario was the type of target vessel presented in the trial. This was done to provide a near-identical combination of collision-avoidance scenarios and difficulty but appeared sufficiently different to not be recognized as a repeated trial.
All trials were conducted using a Virtual Marine (Paradise, Newfoundland, Canada) FRC simulator console and software at the Canadian Coast Guard Atlantic Headquarters in St John’s, Newfoundland. The simulator setup included a 62-inch wall-mounted television with four vessel indicators (speed over ground (SOG), revolutions per minute (RPM), rudder position, and compass heading). The simulated vessel was an inboard single-engine diesel FRC, commonly found onboard CCG ships for use in SAR incidents (Figure 1). (a) The full setup of the experiment; (b) The simulated scenarios presented in the simulator. A flare indicating the target location is visible on the left side, next to the green buoy (Permission for use of images granted by Virtual Marine [virtualmarine.ca]).
Each scenario involved the deployment of an orange smoke flare indicating the SAR area in different locations on the map, prompting participants to engage in desired collision-avoidance actions while navigating toward the incident scene, such as passing or overtaking other vessels within a narrow channel. Each scenario commenced from the same starting point in the simulated environment and concluded at a consistent decision point depending on the participants’ actions (e.g., once they passed the first target vessel). Participants were able to navigate freely but could not look around without moving the vessel.
Procedure
After signing an informed consent form, the participants completed a background/experience questionnaire that recorded their age, gender, and maritime experience. Participants were then provided with a printed copy of the simulated navigation map for reference, which emulated typical navigational protocols in accordance with the standard navigation procedures identified in the GDTA. The participants then completed a single orientation run within the simulated environment. This initial run was intended to help familiarize them with the terrain, control interface, and flare characteristics but did not include other vessels or environmental conditions. Subsequently, the participants were briefed that they were responding to a distress flare sighting in a CCG FRC with navigation warning lights activated. The participants were instructed to approach the scenario as they would an actual SAR response and were tasked with safely navigating to the distress signal while adhering to collision-avoidance regulations governing Canadian navigable waterways. The participants were not briefed on the specific SAGAT questions or given any indication of the conditions or difficulty of each trial. They were briefed that they were going to be asked various questions to assess their situation awareness once the simulation was paused. Participants were not provided feedback on their performance between trials but were provided a complete debriefing with feedback once all trials were completed.
Each participant completed 18 trials of randomly assigned scenarios. Two trials for each difficulty-scenario combination (e.g., two trials for easy/head-on conditions and two for easy/overtaking conditions). Each scenario was paused once per trial at a predetermined decision point, typically after the participant passed the first target vessel. At this point, the simulator screen was blanked, and all visual and auditory outputs were disabled to prevent access to external cues. The participant was then verbally presented with the SAGAT questionnaire by the experimenter and asked to respond aloud. This procedure was consistent across all 18 trials (two per scenario), ensuring that each participant completed the SAGAT questions under identical conditions of information blackout. The view screen visible to the participants, and the information visible only to the experimenter was audio- and video-recorded. This included a digital overhead terrain map from the simulator, including other vessels and their movements.
The participants’ verbal answers to the SAGAT were audio and video recorded for later analysis. The scenario was stopped typically 2 to 5 minutes into each scenario, depending on the actions of the participant, with SAGAT questions taking approximately 2 minutes to complete.
Questions were grouped by SA level (i.e., all questions for one of the SA levels was asked before all the questions for a different SA level, see Table 1). However, the order of the questions within each SA level, as well as the order of the three sets of questions (SA1, SA2, and SA3), varied to minimize participants anticipation of the next question. Once the simulation was paused, participants were permitted to continue the simulation if they desired (three instances of continuation occurred by two participants) to allow participants an opportunity to finish a maneuver or see how a situation played out. However, once the screen was blanked out, the trial was otherwise stopped, and a new trial began. Additionally, they were asked to rate their confidence in their SAGAT answers, following the levels of SA (Level 1, 2, 3), which was always the last question asked for each level.
Participants were encouraged to take breaks as needed, and the session lasted for approximately 90 minutes. All the trials were completed in one session. After completing all the trials, the participants were debriefed and given an opportunity to ask questions or discuss the study. They were each given a $20 gift card and asked not to share any details of the study until data collection was complete for all participants.
Data analysis
Scoring of the SAGAT questionnaire was completed after data collection, during a review of recorded video responses from participants. Correct answers for Level 1 SA questions were derived directly from the information displayed on the simulation screen (e.g., vessel speed, course, and number of vessels visible). For Levels 2 and 3 SA questions, correct responses were determined based on a range of acceptable answers established by the three SMEs.
Although the inter-rater reliability among the SMEs was not formally quantified, all scoring, particularly in ambiguous cases, was conducted through a structured, consensus-based process. The SMEs collaborated during the scoring sessions to ensure alignment with international maritime regulations and the instructional standards of the CCG RHIOT program, thereby promoting consistency and validity in the evaluation of participant responses.
For each trial, we recorded the total SA score and the separate SA scores for each of the three levels of SA. Each correct answer was worth 1 point, with a total of 14 points for total SA (5 points for Level 1 SA, 5 points for Level 2 SA, and 4 points for Level 3 SA). The final question at each SA level was related to the participants’ confidence in their answers on a scale of 1 (least confident) to 10 (most confident).
All data were analyzed using JASP (version 0.18.3, JASP Team, 2024). For each difficulty by scenario combination, the mean scores across the two repetitions were calculated and used in the analysis. To analyze the differences in SA and confidence, we averaged the scores for the two repeated trials and used a three-way ANOVA (Experience Level X Difficulty X Scenario) with repeated measures on the two latter factors. We used ANOVA for the confidence variable, as parametric statistics are robust and can be used for ordinal scales (Norman, 2010). When required, the Greenhouse–Geisser correction was used to correct for the violation of sphericity. Holm–Bonferroni was used as a post-hoc test when required. Finally, we conducted a t-test to compare the number of times the experienced and novice participants left the channel. Depending on the test, effect sizes were reported as partial eta squared or Cohen’s d. Alpha was set at 0.05 in all analyses.
Results
The results are presented in the following order: first, we analyze the three levels of SA (1,2,3) scores, followed by the Total SA score. Second, we analyzed the confidence in the three levels of SA scores (1,2,3), and the Total SA Confidence score. Finally, we analyzed the instances in which participants left the channel. Only statistically significant findings are presented. To view the full results, including statistically nonsignificant findings, please see the Supplementary Materials.
Level 1 SA score
A three-way ANOVA (Experience Level X Difficulty X Scenario) with repeated measures on the two latter factors revealed a significant Scenario effect, F (2, 76) = 6.91, p = .002, ƞ 2 p = .15. Holm-Bonferroni post-hoc analysis found differences between the overtake (M = 3.7, SD = 0.6) and head-on (M = 3.5, SD = 0.6) scenarios (Cohen’s d = 0.34), and between the overtake and crossing (M = 3.5, SD = 0.6) scenarios (Cohen’s d = 0.36). There was also a significant Difficulty effect, F (2, 76) = 11.96, p < .001, ƞ 2 p = .24, and the Holm–Bonferroni post-hoc analysis found differences between the easy (M = 3.5, SD = 0.6) and the moderate (M = 3.8, SD = 0.5) conditions (Cohen’s d = 0.30), and between the moderate and the difficult (M = 3.4, SD = 0.6) conditions (Cohen’s d = 0.47).
Level 2 SA score
A three-way ANOVA (Experience Level X Difficulty X Scenario) with repeated measures on the two latter factors revealed a significant Scenario X Experience Level interaction, F (1.9, 73.8) = 3.87, p = .03, ƞ
2
p
= .09. Holm–Bonferroni post-hoc analysis found differences between the experienced (M = 3.7, SD = 0.5) and novice groups (M = 3.0, SD = 0.6; Cohen’s d = 0.80) in the head-on condition. However, no significant differences were found between the experience levels in the overtake (experienced M = 4.9, SD = 0.6; novice M = 4.3, SD = 0.6) and crossing (experienced M = 3.4, SD = 0.6; novice M = 2.8, SD = 0.7) scenarios (Figure 2). There was also a significant Difficulty effect, F (2, 76) = 5.77, p = .01, ƞ
2
p
= .13. Holm-Bonferroni post-hoc analysis found differences between the easy (M = 3.5, SD = 0.68) and difficult conditions (M = 3.08, SD = 0.83; Cohen’s d = 0.45). There was also a significant Scenario effect, F (2, 76) = 3.28, p = .04, ƞ
2
p
= .08, and an Experience Level effect, F (1, 38) = 10.96, p = .002, ƞ
2
p
= .22. Differences in level 2 SA scores between novices and experienced participants across the three scenarios. (Black circles and error bars represent means and 95% CI, respectively; small colored circles represent individual participants).
Level 3 SA score
A three-way ANOVA (Experience Level X Difficulty X Scenario) with repeated measures on the two latter factors revealed a significant Difficulty effect, F (2, 76) = 4.89, p = .01, ƞ 2 p = .11. Holm-Bonferroni post-hoc analysis found differences between the easy (M = 2.9, SD = 0.7) and moderate (M = 3.1, SD = 0.48) conditions (Cohen’s d = 0.28), and between the moderate and difficult (M = 2.8, SD = 0.6) conditions (Cohen’s d = 0.36). There was a significant Experience Level effect, F (1, 38) = 4.95, p = .03, ƞ 2 p = .12. The SA score of the experienced group (M = 3.1, SD = 0.4) was higher than that of the novice group (M = 2.8, SD = 0.4; Cohen’s d = 0.75).
Total SA score
A three-way ANOVA (Experience Level X Difficulty X Scenario) with repeated measures on the two latter factors revealed a significant Scenario × Difficulty interaction F (4, 152) = 9.2, p < .001, ƞ
2
p
= .20. Holm–Bonferroni post-hoc analysis found differences in the overtake scenarios with higher scores in the easy level (M = 10.28, SD = 1.99) than in the moderate (M = 9.00, SD = 0.92; Cohen’s d = 0.68) difficulty. There were also differences in the head-on scenario scores, where the easy level (M = 8.9, SD = 2.19) was lower than the moderate level (M = 10.56, SD = 1.62; Cohen’s d = 0.879) (Figure 3). There was a significant Experience Level effect, F (1, 38) = 9.43, p = .004, ƞ
2
p
= .20. The total SA score of the experienced group (M = 10.1, SD = 1.1) was higher than that of the novice group (M = 9.1, SD = 1.1; Cohen’s d = 0.56) (Figure 4). Differences in total SA scores across the three scenarios and three levels of difficulty. (Black circles and error bars represent means and 95% CI, respectively; small colored circles represent individual participants). Group differences in total SA scores. (Black circles and error bars represent means and 95% CI, respectively; small colored circles represent individual participants).
SA confidence rating (levels 1-3 and total)
A three-way ANOVA (Experience Level × Difficulty × Scenario), with repeated measures on the latter two factors, was conducted for confidence ratings at each level of SA. For Level 1 SA confidence, there was a significant Scenario effect. Holm–Bonferroni post-hoc analysis found a significant difference between the Overtake (M = 7.5, SD = 1.1) and the Crossing scenarios (M = 7.0, SD = 1.3; Cohen’s d = 0.19). There was also a significant Experience Level effect, F (4, 38) = 4.43, p = .04, ƞ2p = .10. The Level 1 SA confidence score of the experienced group (M = 7.5, SD = 1.0) was higher than that of the novice group (M = 6.7, SD = 1.4; Cohen’s d = 0.57). For Level 2 SA confidence, a significant Experience Level effect was observed, F (1, 38) = 3.97, p = .01, ƞ2p = .10. The experienced group reported higher confidence (M = 8.0, SD = 1.1) than the novice group (M = 7.3, SD = 1.4; Cohen’s d = 0.52). No significant effects were found for Level 3 SA confidence. No significant effects were found for the Total SA confidence.
Number of times participants left the channel
A t-test was used to compare the number of times experienced and novice participants left the channel, and the difference was significant, t (38) = 2.29, p = .03, Cohen’s d = 0.72. On average, the experienced participants left the channel 2.15 times, and the novice participants left the channel 4.15 times.
Discussion
SAGAT Comparison between novice and experienced participants
This study examined the differences in SA between experienced and novice FRC operators to potentially inform FRC training programs and expedite expert performance. We hypothesized that experienced FRC operators would demonstrate higher SA scores across all three SA levels compared to novices. We also hypothesized that the confidence scores of experienced participants would be higher than those of novices. Our data partially support these hypotheses. For example, experienced participants demonstrated significantly higher total SA scores and higher Level 3 SA across all conditions. For Level 2 SA, experienced operators outperformed novices on the overall SA scores, but only in the head-on condition. In addition, experienced participants showed higher confidence scores for both Level 1 and Level 2 SA compared to novices. These results suggest a potential link between experience, SA, and self-assessed confidence in FRC operations. Drawing on the RPD theory (Klein, 1998), it is likely that experienced operators leveraged pattern recognition to rapidly interpret complex cues during the scenarios. This highlights the need for scenario-based training that exposes novices to these patterns to accelerate their SA development. Identifying how confidence varies across SA levels allows training programs to target specific areas, such as comprehension and projection, where novices may benefit from corrective action while also addressing overconfidence or uncertainty in more complex, predictive tasks.
The Level 2 SA score analysis revealed a significant interaction between the scenario and experience level, indicating that head-on conditions differed between the SA scores of the novice and experienced cohorts. This difference can be attributed to scenarios in which the target vessel was either located outside (easy scenario) or on the wrong side of the channel, requiring operators to adhere to the maritime rules of the road and focus on navigating by the buoys rather than solely concentrating on the location and passing procedures of a target vessel. Experienced operators were more often able to identify the safer maneuver, passing the vessel inside the channel in a manner consistent with collision avoidance regulations (91%). In contrast, novices more frequently failed to identify the safer maneuver (40%) and often exited the channel. Leaving the channel can place the vessel in hazardous waters, increasing the risk of collision or grounding
Chauvin and Lardjane (2008) identified essential cues and informal guidelines that bridge officers rely on for large vessel navigation, highlighting occasions where informal rules guided their choices. Similarly, in the current study, some participants verbalized a stereotype of other vessels that frequently shaped predictions about the likely behavior and outcomes of other vessels’ actions (e.g., one participant said: “That is a pleasure craft, and the operator may not know the rules of the road”). This similarity emphasizes how informal practices (e.g., pleasure crafts and fishing vessels frequently break the rules of the road) may influence decision-making in maritime operations, regardless of vessel size or speed. This pattern of reliance on informal reasoning underscores the need for formal training programs to directly engage with these heuristics, clarify when they align with regulatory expectations and operational safety, and correct misconceptions that may lead to non-compliant or unsafe actions.
Wickens et al. (2020) found that while participants generally succeeded in avoiding collisions, their performance deteriorated under uncertainty, showing a preference for speed control over heading adjustments and occasionally violating the navigation rules. Similarly, Chauvin et al. (2008) observed that decision-making was heavily influenced by how rules were interpreted and anticipating the intentions of other vessels, with many trainees performing maneuvers that contravened regulations. This study similarly found that during the 358 independent scenarios presented to the experienced and 360 scenarios presented to novice participants, there were 43 (12%) instances of experienced members and 83 (23%) instances of novice participants leaving the channel, either deliberately or unintentionally, without a clear operational justification. This is noteworthy because FRC operators responding to a maritime SAR incident are required to follow the rules of the road at all times and are permitted to leave the channel only when necessary to avoid immediate danger, including but not limited to a collision (Transport Canada, 2023; Rule 2b). However, in the current study, some participants verbalized that they were allowed to leave the channel at any point, including to hasten response times. This interpretation of collision avoidance regulations by personnel responding to an SAR incident highlights the need for further clarification and discussion in training programs for responders to ensure consistent understanding and safe operations.
The lack of differences between novice and experienced participants in Level 1 SA is similar to the findings of Chauvin et al. (2008), who reported no link between Level 1 and Level 2 and 3 SA scores. These findings suggest that experienced personnel’s decision-making is not necessarily based on a more accurate perception of the available environmental information. However, Randel et al. (1996) found that, compared with novices, experienced Electronic Warfare Technicians excelled in recalling radar emitters and consistently applying rules. In addition, experienced participants remained focused on maintaining an accurate picture of the situation. The results suggest that participants’ reliance on informal heuristics is influenced by their past operational experiences and possible gaps in formal training. Experienced participants more frequently selected the safer maneuver in accordance with collision avoidance regulations, whereas novices were more likely to leave the channel without justification. This discrepancy points to a performance gap: although experienced operators appear to have internalized rules more effectively, both experience levels demonstrated moments of informal rule interpretation, such as assuming permission to leave the channel to save time or relying on stereotypes about other vessels’ behaviors. These informal practices emerged in scenarios where regulatory guidance was either misunderstood or selectively applied, underscoring the need for training programs to address not only formal rules but also the informal logic operators may bring to bear in ambiguous or high-pressure situations. A structured discussion of informal heuristics, particularly when and why they arise, should be integrated into training to correct unsafe assumptions and reinforce rule-based decision-making. Training should also include clearer scenario briefings that explicitly define the rules of engagement to reduce ambiguity and prevent misinterpretations of collision regulations.
Experience level differences in SA confidence ratings
Level 1 SA confidence ratings revealed a scenario effect, with participants feeling more confident during overtaking scenarios than during crossing scenarios. This supports the advice given by SMEs during the design phase of this study that crossing situations are often ambiguous and require participants to identify vessels that could potentially contradict their intended actions. Participants frequently verbalized the type of vessel (e.g., barge, speed boat, sailboat) and the potential activities it may be engaged in (e.g., fishing, towing) as a reason for channel route selection and speed. This was especially true for SAGAT question 4, which required participants to identify the target vessel’s course. The potential ambiguity increased if more than one vessel was present that would need to alter its course in response to another vessel or the participant’s FRC, which can explain the difference in confidence scores between overtaking and crossing situations.
SAGAT question 5 asked participants to judge the distance of the closest vessel, and several participants expressed difficulty estimating this based on the available information from the simulator. The experienced group could only determine the correct distance within the acceptable threshold of 1.5 times the actual distance in 138 (39%) trials, and novices in 115 (32%) trials. Due to this specific SAGAT question, both experience levels frequently expressed their lack of confidence in their Level 1 SA answers, which could have contributed to the low confidence ranking in both experience levels. These findings are in agreement with Feldstein’s (2019) analysis of the egocentric perspective in virtual environments, which described the difficulty in judging measurements of objects moving forward and away from a user when using a driving simulator, which offers an explanation for our findings.
Notably, there was an Experience Level effect in SA Levels 1 (perception) and 2 (comprehension) confidence scores, with the experienced participants more confident on average in their answers than the novices. This finding is in contrast to Level 3 (projection) and Total SA confidence ratings, which showed no difference in experience level
The results of the current study offer insights into how SA and confidence in one’s own SA relate across various SA levels, especially when considering operator experience and the specific nature of the scenarios presented. At the Level 1 SA confidence rating, our findings revealed that participants in both experience levels felt more confident during overtaking scenarios than crossing scenarios, and experienced operators exhibited higher confidence than novices. Interestingly, despite similar Level 1 SA scores, novices rated their performance as lower than that of experienced operators. This finding diverges from the Dunning-Kruger effect (Kruger & Dunning, 1999), which suggests that novices often lack the metacognitive awareness to recognize their performance gaps and thus tend to overestimate their abilities.
At Level 2 SA, experienced operators displayed higher confidence scores than novices, though their performance was significantly better than that of novices only in head-on scenarios, with similar performance across other conditions. This challenges the expectations stemming from the Dunning–Kruger effect (1999), which predicts that novices would show overconfidence compared to experienced participants. Instead, experienced operators not only had higher SA scores for specific scenarios but also showed greater confidence in their performance, whereas novices exhibited lower confidence in their performance. This finding aligns with the findings of Sulistyawati et al. (2011), who suggested that overconfidence tends to increase with higher levels of SA among experienced individuals.
Our findings further extend this understanding by showing that experienced participants’ confidence in Level 2 SA remained stable across all scenarios and difficulty levels. This suggests that experienced operators consistently maintain high confidence in their abilities, regardless of the external challenges presented. While this might reflect genuine skill, it could also suggest a potential gap in their metacognitive assessment, meaning their perception of their SA abilities might not be entirely accurate, even when external conditions change.
At Level 3 SA and Total SA confidence ratings, we found no significant interactions or effects for experience level
The measurement of overconfidence remains complex, even in domains with objective, continuous feedback, and well-defined performance metrics. Heck et al., 2025 demonstrated that persistent overconfidence can exist among tournament chess players despite years of exposure to accurate and public performance ratings, challenging the assumption that experience alone corrects miscalibrated self-assessments. Although the experienced cohort examined in this study at times demonstrated higher SA scores and greater confidence than novices, it is important to recognize that they may not yet meet the threshold of high performance, particularly in the context of high-fidelity, high-stakes maritime operations, and still exhibit overconfidence with their level of knowledge.
Implications for training and development
The findings of this study suggest several implications for FRC training and system design. FRC operations are distinct from the large-vessel SA requirements. They are minimally structured, fast-paced, and conducted in high-risk conditions, where operators must maintain SA under severe time pressure, often without standardized procedures. While simulators are increasingly used in FRC training, no established standards guide their application or the assessment of cognitive performance within these platforms. Our findings contribute to this evolving field by informing simulation-based training practices, identifying key performance indicators, and advocating for confidence calibration as part of training interventions.
In this context, the magnitude of the effects observed in this study (e.g., Cohen’s d > 0.75 for Level 2 SA, Total SA, and navigational errors) points to substantive performance gaps between novice and experienced FRC operators. Furthermore, the absence of corresponding differences in confidence ratings highlights a mismatch between perceived and actual SA, suggesting that novice operators may overestimate their own SA.
Based on the uniqueness of FRC operations and the findings of the current study, we suggest that training programs incorporate structured, scenario-based exercises that emphasize rule comprehension and application, particularly focusing on when deviation from navigational markers is justifiable within regulatory frameworks. In addition, instructors should address the role of informal reasoning strategies by helping trainees distinguish between adaptive heuristics that support decision-making and unsafe assumptions that could lead to errors.
Confidence calibration is another key area. Training should assist participants in accurately assessing their capabilities, especially in tasks requiring projection and prediction (i.e., Level 3 SA). Scenario selection also plays a critical role in the effectiveness of training. In addition, instructors should emphasize crossing and head-on situations, which demand greater cognitive effort and pose a higher risk of regulatory error (i.e., Level 2 SA). Finally, to address the limitations of the simulation environment, enhancements are needed to improve ecological validity, specifically by incorporating more realistic spatial depth cues, motion feedback, and environmental stressors such as noise and vibration to better represent operational demands and associated cognitive load.
Limitations and directions for future research
The current study has two primary limitations. First, although the FRC simulator offered a controlled environment, it did not fully replicate the sensory and cognitive demands of real-world operations (see, e.g., Hamstra et al., 2014). Second, the scenarios used did not incorporate certain realistic environmental conditions, such as wave action or wind variability, which could influence SA and decision-making in actual maritime settings. Another limitation of this study is that confidence ratings were collected at the overall SAGAT level (e.g., Level 1 SA) rather than for each individual question, which restricts the ability to precisely identify where miscalibration between perceived and actual SA occurs. Collecting confidence on specific questions would allow for a more granular analysis of over- or under-confidence across specific SA components. Finally, the participants’ gaming experience may have affected their ability to perform in a simulated environment.
Further research should examine how specific aspects of experience (e.g., recency and mission type) affect SA development and heuristic use. Understanding how experience shapes cognitive strategies will inform more targeted training interventions both within maritime SAR and in other high-risk domains. Finally, exploring cultural and organizational factors that reinforce informal norms may illuminate systemic contributors to non-compliance or miscalibrated confidence.
Conclusion
In conclusion, this study offers insights into how operational experience influences SA and confidence among FRC operators during SAR missions. Experienced operators generally demonstrated higher SA scores and greater confidence in their ability to perceive and comprehend situational elements; however, these advantages were dependent on specific scenario conditions. Notably, no significant differences in experience level were observed in significant or total confidence ratings, suggesting that novice operators may exhibit overconfidence in their ability to project future events in dynamic and uncertain maritime environments. This highlights a potential miscalibration between perceived and actual SA, particularly at higher cognitive levels. Beyond these empirical findings, this study makes a methodological contribution by applying SAGAT in the context of maritime SAR operations and integrating it with GDTA. This approach facilitated the design of realistic, domain-specific scenarios and probe questions that reflect the operational decision-making demands of the FRC coxswains. This study addresses a critical gap in domain-specific assessments of SA and expertise.
The inclusion of confidence ratings reveals implications for training. Instructors should not only assess SA performance but also evaluate how accurately operators judge their capabilities. Addressing confidence calibration in training programs can help mitigate the risks associated with overconfidence, improve decision-making under pressure, and foster more accurate self-assessments among both novice and experienced personnel.
Supplemental Material
Supplemental Material - Situation Awareness in Fast Rescue Crafts Operators—A Simulator Study
Supplemental Material for Situation Awareness in Fast Rescue Crafts Operators—A Simulator Study by John MacDonald, Elizabeth Sanli, Jennifer Smith, Michael Taber, Gal Ziv in Journal of Cognitive Engineering and Decision Making.
Footnotes
Acknowledgments
The authors would like to thank the Canadian Coast Guard Atlantic Region for their participation, and support throughout the study.
Author contribution
J.M. Conceptualization, Methodology, Formal analysis, Investigation, Resources, Data curation, Writing—original draft, Writing review & editing, Visualization. E.S. Supervision, Project administration, Writing review & editing
Declaration of conflicting interests
The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article. John MacDonald reports a relationship with Canadian Coast Guard that includes: employment. No other conflicts of interest identified. The other authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Situation Awareness in Fast Rescue Craft Operators – A Simulator Study
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study is supported by the Marine Institute Research Start-Up Fund [grant number 215029 42421 2000].
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
