Sage Journals: Discover world-class research

Abstract

This paper reports a qualitative exploration of gaze strategies during closed-circuit television (CCTV) tasks in remote nautical object control of a lock. Previous research has not examined gaze strategies in scenarios where systems, such as nautical objects, are operated remotely using CCTV. As contextual factors matter in nautical object control, a qualitative approach was necessary to uncover domain-specific terminology and insights into gaze strategies. We recorded eye gaze from professional lock operators and then conducted semi-structured interviews with domain experts to assess these recordings. Thematic analysis revealed that experts were able to identify (features of) gaze strategies but did not share the same terminology. Based on this analysis we defined four strategies: anticipating, verifying, overview, and movement-directed gazes (RQ1). All strategies, except movement-directed gaze, were observed consistently across operators (RQ2), with verifying gaze aligning with task steps in a predefined protocol (RQ3). More generally, our classification framework from thematic analysis could help to systematically define and verify gaze strategies based on domain and task features across various CCTV working contexts. For nautical object control, the framework can be instrumental in interpreting and verifying future (quantitative) eye tracking results and informing instructional procedures.

Keywords

eye movement behaviour eye tracking gaze strategy features operating protocol nautical task CCTV supervision

Introduction

While incidents and accidents in the domain of remote nautical object (i.e. bridges and locks) control occur relatively infrequently, their impact can be significant. For example, an analysis of 17 remote bridge control incidents in the Netherlands between 2003 and 2018 found that in 15 cases a road user was between the barriers on the bridge deck when it was opening, which resulted in fatalities, serious injuries, or damage (Dutch Safety Board, 2016, 2019). The analysis revealed that the road user was not (promptly) noticed due to a combination of human factors issues such as inadequate camera visibility, poor image quality, and insufficiently directed operator gaze behaviour. In the wake of these incidents, technical design guidelines for remote control of nautical objects through closed-circuit television (CCTV) were developed (for international documentation, see e.g. Pikaar et al., 2015; Schreibers et al., 2012). However, these were not (yet) applied in the aforementioned incidents.

More broadly, despite these technical design guidelines, there is still a lack of detailed insight on operators’ gaze behaviour (Donald, 2019; Dutch Safety Board, 2016, 2019). Specifically: are there any (consistent) patterns in the gaze behaviour of operators that reveal a specific viewing technique (or: gaze strategy) for performing a (sub)task? Understanding such ‘gaze strategies’ is crucial due to the risks of perceptual errors during operators’ CCTV tasks and the rise of innovative camera systems. Adopting a more systematic approach to understanding eye gaze (strategies) during the operation of nautical objects, as studied here, enables meaningful analysis of eye tracking data by supporting informed interpretation of quantitative outcomes. Additionally, such an approach can, in the long term, inform instructional procedures and potentially help prevent incidents. Our study on CCTV operation lays the groundwork for future assessments, with the aim of informing guidelines to improve visual perception and minimise human errors during CCTV tasks in nautical object control.

When examining gaze strategy research across different domains, valuable insights can be extracted from eye tracking studies conducted in healthcare (e.g. Drew et al., 2013; Manning et al., 2006; Tien et al., 2010; Zimmermann et al., 2019), aviation (e.g. Dehais et al., 2017; Lefrançois et al., 2021; Ziv, 2016), construction work (e.g. Xu et al., 2019), maritime (e.g. Atik & Arslan, 2019), rail transport (e.g. Horiguchi et al., 2016), (airport) security (e.g. Biggs et al., 2013; Thériault et al., 2018), and in control room settings such as air traffic control (e.g. Wang et al., 2021) and process control (e.g. Bhavsar et al., 2017). Many of these fields show consistencies with respect to gaze strategies for professionals within a domain. Across domains, however, gaze strategies mostly seem to identify level of expertise and large variations exist in gaze strategies (see systematic reviews by Martinez-Marquez et al., 2021; Van der Gijp et al., 2017; Ziv, 2016).

Since gaze strategies depend on the viewers’ aims and the task that is performed (Kootstra et al., 2020; Kramer et al., 2019; Sharvashidze & Schütz, 2020), it is difficult to generalise findings from one domain to the other. Hence, investigating gaze strategies in various domains is important, as it allows for a better understanding of differences and similarities across domains. Besides, it is important to consider context and the type of task before generalising gaze strategies, as these may highly influence visual behaviour (e.g. Land & Hayhoe, 2001; Lohmeyer et al., 2015; Sharvashidze & Schütz, 2020; Wang et al., 2022). Therefore, different task types and different contexts warrant their own detailed studies.

One task type that has been underrepresented in gaze strategy research is that of CCTV supervision, which is applicable to nautical object control. Although some research has applied eye tracking methodology in CCTV control room settings (e.g. Stainer et al., 2013; Thériault et al., 2018), current research on CCTV supervision is limited regarding context-specific knowledge on gaze strategies in CCTV detection and monitoring tasks (Donald, 2019). Previous research with CCTV had students or operators look at pre-recorded CCTV footage (e.g. Howard et al., 2013; Mileva & Burton, 2019; Peng et al., 2024; Tatler, 2021; Todorova et al., 2017). However, this approach lacks the responsibility of actual (object) operation. Moreover, as the footage was not always viewed from their own working space, it might not reflect their routine behaviour.

The current study therefore records eye gaze of operators in their naturalistic CCTV control room setting to investigate gaze strategies for nautical object control. This setting offers a task context combining CCTV with remote control procedures, which is different from the more commonly studied CCTV surveillance environment. Specifically, we study the control (through CCTV) of the nautical object of locks.

Before examining eye gaze in nautical object control (of a lock), we will first discuss observations of gaze strategies in other domains, such as radiology and aviation. Despite differing tasks, they offer valuable insights into gaze strategy outcomes. We will then elaborate on the domain at hand: gaze strategies in nautical object control, specifically focusing on the task and setting for remote lock operations and its relevance for studying eye gaze. Lastly, we will further introduce our research questions, hypotheses, and design rationale. As a method, we will use a qualitative approach (specifically: semi-structured interviews with experts who view eye gaze of operators performing their job) to derive commonalities in the gaze strategies of lock operators.

Observations of Gaze Strategies in Other Domains

One of the domains where gaze strategies have been studied in great detail is radiology. For example, experimental work from Drew et al. (2013) points out that different gaze strategies exist among expert radiologists. In their study, radiologists had to search for lung nodules by scrolling up and down through 2D chest CT slices, where a slice represents a level of the lung. Drew et al. (2013) found that one group of radiologists applied an approach where they scan an entire slice before continuing with the next level (called: scanners). Another group viewed one specific region of a slice while quickly scrolling through depth (called: drillers). Drillers showed better performance as they had higher nodule detection rates and scan coverage. Whenever detection errors were made, drillers often looked at the nodule but did not classify it correctly (i.e. recognition error), whereas scanners completely missed the nodule (i.e. search error).

Other studies on gaze strategies in radiology found differences between experts and novices, for example, when radiologists had to search for nodules in mammograms (e.g. Kundel et al., 2007), in chest radiographs (e.g. Manning et al., 2006), when they used CT colonography (e.g. Phillips et al., 2008), and when they had to diagnose stroke in CT head scans (e.g. Cooper et al., 2010). Across these studies, novices tend to show a more unsystematic gaze pattern as compared to expert radiologists.

Another domain in which eye tracking has been extensively used is aviation, where different gaze strategies are observed between pilots. Pilots that have the task of monitoring their aircraft state distribute their visual attention more broadly across instruments (i.e. comparable to the scanners from radiology), whereas pilots that have to manage the aircraft’s flight path devote visual attention more specifically to exploit instruments (i.e. comparable to drillers in radiology) (Dehais et al., 2017). Similar to the domain of radiology, visual scanning patterns of pilots are influenced by the level of expertise (Martinez-Marquez et al., 2021; Ziv, 2016): expert pilots show more defined scanning patterns than flight cadets.

In both domains, professionals exhibit two distinct gaze patterns, which appear to be associated with a scanning approach for creating an overview and a drilling/focusing approach for obtaining details. Notably, especially within radiology, where professionals share the common task of detecting nodules, variations in gaze strategy become evident. While recognising the differences in context and tasks between these domains, we expect that similar variations in gaze strategies will manifest in nautical object control when professionals execute the same task.

Domain at Hand: Gaze Strategies in Nautical Object Control

In nautical object control, operators remotely manage locks and/or bridges to ensure the efficient and safe passage of road and waterway traffic, thereby safeguarding traffic safety in relation to operating processes. Main tasks of operators fall in two broad categories. The first category is execution, monitoring, and control. This consists of specific tasks such as visually scanning the nautical object before starting a procedure (CCTV detection task), controlling a nautical object, monitoring multiple systems (proactive CCTV monitoring task), and intervening in the process if necessary. The second category is supporting activities. For example, processing any registrations of waterway traffic, making up the planning, and reporting activities related to the operating process. These activities are performed at dedicated, structured workspaces that have different information sources.

Figure 1 shows an example of a workspace set-up as used in the location where we collected our data, where the nautical object in question is a lock. At the top of the workspace is a continuous stream of CCTV (overview streams). At the bottom are dedicated screens for (left to right): process and free choice CCTV streams, supervisory control system, radar, and planning/reporting systems. Process streams are only provided during operating procedures, whereas free choice streams are selected and controlled by operators, for instance, through pan-tilt-zoom cameras.

Figure 1.

Workspace for lock operation at the Prinses Beatrixsluizen, Nieuwegein, The Netherlands. Note. The top row of monitors displays CCTV streams, each showing individual CCTV feeds for three lock chambers (top, middle, bottom rows). The bottom row of monitors shows (from left to right): process and free choice CCTV streams, supervisory control system, administrative system (IVS), and radar (AIS). For additional workspace details, including the arrangement of systems and applications, see Supplementary Material (Section 2.2).

An interesting feature of the domain of nautical object control in comparison to previously studied domains is that operators of nautical objects are tied in their actions to a protocol. This protocol offers a sequence of actions that must be executed, including what information must be (visually) inspected before, and monitored after, a critical operating action (e.g. closing a lock gate) is initiated. At the same time, this protocol does not specify the exact gaze patterns that need to be made, thereby allowing the freedom to combine nautical tasks during monitoring as needed. As such, the domain of nautical object control allows for an opportunity to study free viewing during task execution. In addition, our study can provide insights into whether eye gaze is consistent with the protocol, or whether operators’ expertise has shaped behaviour in a different way due to, for example, their experience with task demands (e.g. Gray & Banerjee, 2021).

Understanding operators’ gaze strategies has multiple benefits. From a fundamental perspective, this can give insights into eye movements in naturalistic CCTV task settings that involve detection and monitoring. For example, does the nature of detection and monitoring tasks result in very specific patterns? Or do operators apply similar strategies as observed in other domains such as radiology and aviation? Regarding the latter, remote nautical object control involves gradual dynamic phases like opening and closing lock gates. This contrasts with radiology’s static scan analysis and aviation’s highly dynamic task environment, so cross-domain gaze strategy study is important for understanding gaze strategy disparities. Furthermore, specifically focussing on nautical object control, a better understanding of gaze behaviour in this task setting (derived from our qualitative findings) may support further exploration and verification of (quantitative) gaze pattern analyses.

From a practical perspective, this work is crucial for optimising CCTV task execution in nautical object control by leveraging insights into gaze strategies. Understanding operators’ attention allocation across operating phases might facilitate interface design supporting natural gaze patterns, thereby improving overall system usability. While previous research in human factors and cognitive engineering explored gaze metrics’ relevance for mental workload (Di Nocera et al., 2007; Moacdieh et al., 2020), vigilance (Langhals et al., 2013), and team performance (El Iskandarani et al., 2023), it did not specifically address defining cognitive strategies based on eye gaze. Nevertheless, recognising effective gaze strategies can inform the (re)design of operating protocols, workspaces, and training, reducing the risks of human error in nautical object control.

Research Questions, Hypotheses and Design Rationale

Given that little is known about gaze strategies during detection, proactive monitoring, and free viewing tasks in CCTV work settings (Donald, 2019) and the possible risks of perceptual errors for CCTV tasks (Dutch Safety Board, 2016, 2019), we study these in a representative yet so-far underreported domain: nautical object control. Specifically, we investigate the following main research question: Within the domain of nautical object control, what kind of consistent and protocol-related gaze strategies can domain experts observe in lock operators’ eye movements? We hypothesise that gaze strategies can be determined for lock operators, similar to how this has succeeded in other domains such as aviation and radiology (e.g. Dehais et al., 2017; Drew et al., 2013).

The following three sub research questions have been defined to answer our main research question:

RQ1

What features of gaze strategies of lock operators can be recognised by domain experts?

RQ2

What gaze strategies (as observed by domain experts) are consistent across lock operators? Other domains showed different gaze strategies between professionals (e.g. scanners and drillers among radiologists). Similarly, we expect that gaze strategies differ among operators when the same task (or: operating procedure) is performed.

RQ3

What gaze strategies are consistent with the operating protocol according to domain experts? Since following protocol is a prerequisite for safe and effective task execution, we expect monitoring and execution elements from the protocol to be present in the gaze strategies of all operators.

To answer our research questions, we report the results of qualitative interviews with domain experts. During the interviews, domain experts assessed recordings of operators’ naturalistic eye gaze. These experts were consulted to avoid operators evaluating their own gaze behaviour (i.e. prevent response bias tied to personal performance) and because it allowed determining cognitive strategies during non-operational periods.

Qualitative interviews were conducted for four reasons. First, gaze strategies rely on domain- and task-specific information, which is relatively undetermined for lock operations in relation to eye gaze. Domain experts possess detailed knowledge of the lock operations setting and might better discern which elements are relevant and why certain fixations occur (or do not occur). Qualitative results can provide this domain-specific terminology and interpretative insights, which are important for guiding future (quantitative pattern) analyses. Second, gaze strategies consist of fixation patterns instead of distinct fixations only. Consequently, such fixation patterns might not always be interpretable through quantitative (aggregate) measures such as fixation counts and durations without first establishing the qualitative context. Third, we want to ground our findings in terminology that experts use, which can be identified through qualitative interviews and thematic analysis. By interviewing multiple domain experts, we minimise the risk of (researcher or expert) bias. Fourth, it allows researchers and domain experts to align their terminology, ensuring consistent interpretation in future research.

Addressing these research questions uncovers the strategies operators use (RQ1), assesses their consistency (RQ2), and evaluates how they diverge from established protocols (RQ3). This will, in turn, provide a systematic approach for defining gaze strategies across various CCTV working contexts, crucial for interpreting future (quantitative) eye tracking data and refining instructional methods.

Methodology

Study Overview

A qualitative study design was conducted that consisted of two parts. The first part focusses on the collection of naturalistic stimuli: eye tracking with operators. We recruited 13 operators for eye gaze data collection while operating a lock. For one operator, the eye tracker failed to collect data, and for four others, the data quality was poor, likely due to their own glasses interfering with the eye tracker. Eye gaze data from 8 operators was used, from which 16 video fragments (2 per operator) of a lock gate closing procedure were selected. This procedure was deemed relevant for two reasons: (1) it related to an operating protocol, and (2) it involved a safety-critical action given the possible risk of collision (e.g. vessel hit by a gate).

The second part focusses on interviews with domain experts. We asked 16 domain experts to observe a subset of these recordings, and through an in-person semi-structured interview, we tried to gain insight into the strategies that operators might have applied. Specifically, each expert viewed two videos from two operators (so a total of four). The total set of 16 videos was balanced such that each video was observed by 4 experts.

Ethics

Ethical approval for this study was obtained through the ethical committee of the Faculty of Social and Behavioural Sciences from Utrecht University (code 21-0519). All participants (operators and domain experts) received an information letter on study aim, procedure, voluntary consent, data usage and preservation, potential risks and benefits, and compensation. They voluntarily agreed to participate in the study by signing an informed consent form. Participants did not receive any compensation after finishing the session.

Collection of Naturalistic Stimuli: Eye Tracking with Operators

This study focused on gathering eye gaze data during a critical step in lock operations: a lock gate closing task. The potential risks of this task lie in deviations from the established protocol, which could lead to serious physical, emotional, and financial consequences, such as a vessel being hit by the lock gates. To prevent such errors, operators must diligently follow prescribed steps to ensure safe and effective control, including initiating actions on the supervisory control system in a precise sequence. Visual inspections of CCTV streams precede certain steps to ensure safety in the operational area. To get a better understanding of the gate closing procedure, detailed information about the protocol steps is made available as Supplementary Material (Section 1).

To create representative videos of operators’ eye gaze to be used as stimulus material during the expert interviews, eye movement data of eight operators (seven males; one other) was used. The eye gaze data collection took place at the control room of the Prinses Beatrixsluizen in Nieuwegein, the Netherlands. During operators’ shift, eye movements were recorded for 45 minutes using a wearable eye tracker (Tobii Glasses 3.0, firmware 1.11.6) with a sampling frequency of 50 Hertz. Detailed information about the operators, materials, setting, and procedure is made available as Supplementary Material (Section 2).

Fragments of the full eye gaze recordings were selected to create videos, for which selection was based on four criteria. First, we selected videos that showed a gate closing process, as this is safety-critical. Second, video fragments were selected from recordings that had high gaze samples and estimated accuracy. The average accuracy of gaze samples (based on recordings that correspond with selected videos) was 97% (SD = 2%). The estimated accuracy was then checked for these recordings, such that the expected location of fixations matched with known targets in the environment to increase data validity. For example, after calibrating with a calibration card, it was continuously checked whether a fixation was on the mouse cursor when related actions were performed to notice and prevent offsets. Third, to ensure that all gate types of the Prinses Beatrixsluizen were represented (horizontal sliding gates and vertical lift gates), we selected videos that each represented another gate type performed by the same operator. Fourth, to prevent that operators’ attention and task execution would be affected by the timing of the shift (early or late) or the lock side (southern or northern direction), we ensured to select videos from an operator that featured both the same shift type and lock side when combining videos for expert review (see ‘Design').

Based on these selection criteria, 16 MPEG-4 videos (1920x1080) were chosen. Each video contained a small moment before the closure action started, which ensured that some context and preparatory actions before the gate closing process were also provided. The selected videos ranged from 2.12 to 3.65 minutes (M = 3.02, SD = 0.55 minutes). The videos were processed in Tobii Pro Lab version 1.181.37603 (x64) with Tobii I-VT (Attention) as raw gaze filter, which had a threshold value of 100 degrees/second. Red (HEX #D92121) was used as fill colour for all fixations (circles), while its contrast colour was white (#FFFFFF). Regarding other gaze data settings, opacity was set to 50%, size to 30%, fading duration to 2 seconds, and maximal size to 60%. Examples of video stills are presented in Figure 3, Figure 4, and Figure 5. After creating the videos, anonymisation by face detection was applied using Python ‘deface’ on six videos. Deface command enabled faces to be blurred (e.g. colleagues that appeared during the recording) and audio to be removed. After conversion, these six videos were formatted to 1920x1088 pixels.

Interviews with Domain Experts

Participants

To be included in the study, domain experts had to understand the task and context shown in the videos. Specifically, they were required to (1) possess experience or be actively engaged in the nautical domain, (2) have familiarity with the tasks and processes involved in nautical object control (i.e. understanding operators’ actions), and (3) exhibit expertise in the information sources utilised within operators’ workspaces (i.e. understanding operators’ visual focus). These three selection criteria were then applied to select experts who could be working either as advisor, trainer, or human factors consultant.

Experts were identified by the first and fourth author (both domain experts), and upon suggestion of colleagues and interviewees. The first author contacted experts directly based on their contribution to a knowledge exchange group with internal employees of Rijkswaterstaat specialised in nautical object control. Efforts were also made during interviews to identify additional potential experts both within and outside the organisation. Seventeen experts were recruited through purposive sampling (expert sampling) of whom 16 (14 males) completed the session. One expert did not complete the study as he was familiar with bridge operations, but not lock operations. Recruited experts were from the same organisation as where the operators work (internal employees: 12), or from external parties (consultants/trainers: 4). Of the internal employees, nine were recruited from an expert group dedicated to nautical object control (i.e. working in the role of (operational/national) nautical advisor or transition manager), and three internal employees were recruited based on recommendations from the interviewees (snowball sampling) and worked in similar positions. Of the external experts, four were recruited based on their consultancy/training experience and human factors knowledge within the nautical domain. Three of them were human factors consultants, one was a trainer and part-time operator.

The 16 experts (14 males, 2 females) were 31 to 65 years old (M = 53.4, SD = 9 years of age). One expert completed secondary education, four did secondary vocational education (or similar), seven completed higher professional education (or similar), and four had a university degree. Fifteen experts were familiar with the domain of nautical object control, and their working experience ranged from 2 to 47 years (M = 18, SD = 13.7 years). One participant indicated to be unfamiliar with the specific subdomain of nautical object control but was still able to complete the session as he understood the procedures from his nautical experience. This expert was therefore included in the sample. Eleven experts had also worked as an operator in the past. Eight experts were familiar with the workspace settings of whom six were familiar with the three lock chambers of the Prinses Beatrixsluizen. For the other eight experts who were unfamiliar with the workspace, six were unfamiliar with the three lock chambers.

Materials and Setting

Interview sessions were conducted in a private room at a location that was suitable for the expert. During each session, videos were presented on a HP ZBook Fury 17 G7 Mobile Workstation running Windows 10 pro version 21H2. Furthermore, experts received four sets of fifteen cards with descriptions from the operating protocol and seven cards from the object specific operating instructions. These descriptions were specifically related to the lock gate closing procedure that was shown in the videos. The cards presented descriptions that were related to monitoring aspects (e.g. monitor the stop line), visual scan aspects (e.g. scan whether there are no persons nearby the lock gate), verification aspects (e.g. check whether vessel propellers are switched off), and computer actions (e.g. place mouse cursor on stop button). Lastly, the experts received two pages that contained guidelines for a lock gate closing procedure; one showed the steps from the national operating protocol (version 31-03-2021) and the other one showed the object specific operating instructions from the Prinses Beatrixsluizen (version 2.5).

Design

The order of the videos was counterbalanced to minimise practice (learning and sequence) effects of experts evaluating the videos. In line with a balanced Latin square design, we counterbalanced (1) operator order (e.g. expert A starts with operator 1, expert B with operator 2) and (2) video order from videos of the same operator (e.g. expert A starts with video 1 (operator 1), expert C with video 2 (operator 1).

To be able to investigate whether operators exhibited consistent gaze strategies, four different subsets of videos were created for which each subset contained four videos from two different operators. This subset structure allowed to test consistency of eye gaze for the same as well as between two different operators. Regarding the contents of the videos in every subset, subsets were grouped such that operators who worked at the same workspace (left or right) and the same shift (early/late) were in the same subset of videos (see also ‘collection of naturalistic stimuli: eye tracking with operators'). Similar workspace/shift type ensured that place/timing conditions were as similar as possible in every subset. However, for two subsets this was not possible given the collected videos. One subset had operators with different timings of their shifts (right; early and late), the other subset constituted operators who worked at different workspaces (left and right; early). We did not find any negative consequences from this workspace/timing difference with respect to the eye tracking evaluations.

Regarding expert grouping, all experts were divided evenly in groups such that possible learning effects from the first author taking the interviews would be minimised. Hence, each subset of videos was viewed by four experts. In addition, since 4 out of 16 experts were external, these external experts were equally assigned across the four groups. A reason for this was that external experts might bring other perspectives to the sessions that are independent of organisational aspects where internal experts deal with daily.

Procedure

The recruitment email included an information letter about the study. Upon agreeing, experts received a demographic questionnaire and an informative document about the workspace setting to get familiarised with it. This document included some context of the working environment (e.g. lock design, type of operation) and the set-up of the workspace (e.g. applications, CCTV streams).

The interview session lasted for 60–90 minutes and started with an introduction of about 10 minutes. During the introduction, experts were given the option to read information of the workspace setting again (which they already received by email). They were then informed about how to interpret eye tracking elements (fixations/saccades) to ensure equal interpretation of the eye gaze information by different experts. They were also informed that they would receive questions about operators’ gaze behaviour and the operating protocol for each video. When everything was clear, they were asked for their informed consent. Audio was then recorded. Experts were informed that they would see a full lock gate closing procedure. They were instructed to watch every video carefully, because each video was shown only once with normal speed and without pausing or rewinding. The rationale for one-time video viewings was driven by constraints on expert availability, avoiding lengthy interviews, and aiming for comprehensive impressions over fixating on details. Regarding the latter, in sessions where experts can repeatedly watch specific video segments, their focus may shift to details unique to that video or context, rather than the broader patterns necessary for generalisation.

The session was divided into two parts: single video evaluations and video set evaluations. In the first part, each expert viewed a total of four videos. At the end of every video, a semi-structured interview was conducted by the first author, during which experts were asked questions about the execution of the operating procedure and about the eye fixations (see Supplementary Material (Section 3) for the interview protocol). In addition, they were asked to sort out a set of cards for every video that described actions from the operating protocol. Specifically, experts were instructed to sort out the cards they saw in the video and put these in chronological order relative to how the actions appeared in the video.

In the second part (after responses for all individual videos had been given), experts were asked to read two pages that described actions from the protocol regarding the lock gate closing procedure. After confirming that they had read the documents, experts were asked to compare it to the cards they sorted out. The session was concluded by asking how many operators they thought the videos were based on (1 till 4), and whether they had any recommendations regarding other experts to be interviewed. They were then thanked for their time and effort and instructed to keep the contents of the session to themselves.

Detailed information on the used interview materials is made available as Supplementary Material, which included the operating protocol (Section 1), object specific operating instructions (Section 4), and operating protocol cards (Section 5). Furthermore, a flow diagram of the expert interviews is depicted in Figure 2.

Figure 2.

Flow diagram of expert interviews. Note. Experts viewed four videos. Unknown to the expert, the first two videos (‘A’ and ‘B’; in blue) were from one operator; the last two videos (‘C’ and ‘D’; in orange) from another operator. The operators of which videos were shown and the order of videos of specific operators was counterbalanced, see ‘Design'.

Analysis

To analyse gaze behaviour and strategies (RQ1), operator consistency (RQ2), and protocol consistency (RQ3), only qualitative (interview) data was considered. Consequently, data analysis comprised transcript analyses based on expert interviews, during which experts evaluated operators’ eye gaze videos that involved subjective gaze pattern analysis.

RQ1: Gaze Behaviour and Strategies

Based on the interview data, gaze behaviour and strategies were determined through the six-step thematic analysis process by Braun and Clarke (2006). We applied these steps more specifically as follows:

1. Familiarisation. Semi-structured expert interviews were audio-recorded and processed in intelligent verbatim transcriptions through Microsoft Word (expert ID 1–4, 8, 9, 14) and Amberscript (expert ID 5, 7, 10–13, 15–17). Transcripts were checked against the tapes and manually edited by the first author. Experts were not asked to review their own transcript, since the sessions were based on recall of the eye gaze videos.

2. Coding. The first author read through the transcripts in chronological order and identified quotes that specifically related to the research/interview questions (e.g. describing a gaze pattern or style). Each quote was then associated with a more general label, or code, that captured its essence (using NVivo 1.7.1). The coding process was iterative: newly found coded elements were crosschecked with already coded transcripts. To prevent bias, coded elements and examples were discussed with all authors to ensure cross verification.

3. Generating themes. A code overview was created in Microsoft Excel, including the corresponding transcript part and its associated operator, video, and expert number. Core descriptions were created from all coded parts to generate subthemes, which were discussed with all authors to ensure cross verification.

4. Reviewing themes. The initial subthemes were reviewed by the first author and changed or merged with other subthemes if necessary. The subthemes were then discussed with all authors to ensure cross verification.

5. Defining and naming themes. The first author compared all subthemes to each other and clustered them together to create main themes that represented the same content. Again, the main themes were discussed with all authors to ensure cross verification.

6. Writing up. The essence of each (sub)theme was described. An overview was added of how many times a subtheme appeared in the data, in how many videos/operators it was observed, and how many experts stated something about the subtheme. Examples of expert statements were added to give an overall representation of the subtheme.

RQ2: Operator Consistency

With the gaze strategies identified through thematic analysis (RQ1), consistency of gaze strategies across operators was determined by:

1. Comparing the frequency with which all strategies occurred (going by thematic analysis) and for how many videos/operators a given subtheme was found. Here, a ‘strategy’ refers to a subtheme that was identified in the thematic analysis and that the researchers considered relevant to gaze strategy.

2. Asking experts how many operators they thought they saw in the subset of videos (answers options: one, two, three, or four operators in a subset). Each subset consisted of two videos from two operators (four total). Therefore, if videos were incorrectly paired, this could suggest that operators’ gaze behaviour was inconsistent (from the experts’ perspective).

RQ3: Protocol Consistency

Consistency of gaze strategies with the operating protocol was determined by:

1. Analysing how many times protocol steps were observed and determining which strategy/strategies related to steps from the operating protocol.

2. Asking experts which videos they thought were mostly related to the operating protocol and integrating this relatability with observed strategies.

Results

RQ1: Gaze Behaviour and Strategies

Experts observed distinct patterns of gaze behaviour from operators’ eye movements but did not have a joined terminology to distinctly define and cluster the characteristics of operators’ gaze behaviour. When being asked directly during the interview whether they could define a gaze technique or pattern from operators’ fixations, six experts replied that they could, whereas ten experts replied that they could not perceive any pattern. Nevertheless, all experts provided insights that were further used in thematic analysis to better understand gaze behaviour and identify gaze strategies.

Thematic analysis resulted in four themes derived from 613 coded elements. Each coded element represented an expert’s quotation from the transcript, which ranged from individual words to full paragraphs. The four themes that we defined from these elements cover distinct levels of experts’ observations with respect to the fixations: (1) gaze strategy, (2) oculomotor characteristics, (3) relation between fixations and domain content, and (4) relation between fixations and the task. Detailed information about expert statements (example quotes from transcripts) and how many coded elements occurred in the data (frequency, number of experts, operators, videos) is made available as Supplementary Material (Section 6). Below, ‘gaze behaviour’ relates to experts’ comments regarding general impression of eye fixations across the entire video, whereas ‘gaze strategy’ relates to experts’ comments regarding a distinct pattern of eye fixations that exhibit specific characteristics relevant to a (sub)task.

Theme 1: Gaze Strategy

The first theme, ‘gaze strategy’, relates to the contextual interpretation of the fixation patterns. In other words, what kind of gaze behaviour did patterns of fixations represent during operators’ task execution, according to experts? We defined the following gaze strategy types based on experts’ interpretation of the fixation patterns:

1. Anticipating gaze. Anticipating gaze concerns situations where operators’ gaze related to planning or future actions, while not mainly focussing on the active operating procedure (see Figure 3 for an example). Features of anticipating gaze behaviour (see also Table 1) consist of situations where experts related eye movements to:

i. Planning task duties, including making a planning for lying at berth and anticipating on approaching nautical traffic.

ii. Anticipating on when to execute subsequent operating actions.

iii. Usage of planning/radar systems and/or the approaching area/outer port area on camera streams (overview and free choice streams).

Figure 3.

Example still from a video of operator 1 showing anticipating gaze. Note. The saccade-fixation pattern shows that the operator first looked at the radar (AIS) at the bottom-left screen (i.e. to observe more general traffic on the waterway), but also looked at the administrative system IVS (bottom-middle screen). This can be classified as ‘anticipating gaze’ because experts related these fixation patterns to planning or future actions, while not mainly focussing on the active operating procedure. For example, expert 9 mentioned: ‘He observed the traffic on the waterway. He also checked IVS for a moment’.

Table 1.

Features of all Gaze Strategies based on experts’ domain Content and Task Extraction.

Gaze strategy	Domain content			Task
Gaze strategy	Lock elements	Traffic phase	Systems	Task
Anticipating gaze	Approaching area Outer port area	Approaching	Administration (IVS) Radar (AIS) Camera streams (overview/free choice)	Supporting/other
Verifying gaze	Lock chamber(s) Lock gate Stop line Entering area	Entering Docking Exiting	Supervisory control system Camera streams (overview/process)	Execution, monitoring, and control
Overview gaze	Lock chambers	Not applicable	Camera streams	Supporting/other
Movement-directed gaze	(Other) lock gate(s)	Entering Moving	Camera streams	Supporting/other

2. Verifying gaze. Verifying gaze concerns situations where operators focused repeatedly on checking/monitoring specific parts of the lock and/or nautical traffic (see Figure 4 for an example). Besides monitoring lock-specific elements (gate, lock chamber, stop line, and entering area), operators also monitored the moment-to-moment changes of nautical traffic. Experts noticed that operators monitored entering, docking, and/or exiting of the vessel. Features of verifying gaze behaviour (see also Table 1) consist of situations where experts related the eye movements to:

i. Checking and monitoring lock areas/elements: lock chamber(s), lock gate, stop line, entering area.

ii. Checking and monitoring nautical traffic: zooming in on recreational vessels, docking of vessels, vessel propellers.

iii. Monitoring processes: progress of overall procedure, entering of vessels, planning with actual status of docking, distance of vessels relative to stop line.

iv. Usage of supervisory control system and camera streams (overview and process streams).

Figure 4.

Example still from a video of operator 1 showing verifying gaze. Note. The saccade-fixation pattern shows that the operator first looked at the supervisory control system at the bottom-middle screen (i.e. to select and close a lock gate). Subsequently, monitoring took place on the process camera stream displayed at the bottom-right screen. This can be classified as ‘verifying gaze’ because experts related these fixation patterns to checking/monitoring specific parts of the lock. For example, expert 8 mentioned: ‘He is monitoring the stop line because he has a ship that is still mooring, and he also kept an eye on the ship’.

3. Overview gaze. Overview gaze concerns situations where operators’ gaze focused on creating a helicopter view of the nautical object and keeping in mind the overall process of nautical object operations (see Figure 5 for an example). In this case, operators did a specific task with the overall process in mind, and as a result did not only focus on the lock gate closing procedure. Expert observations suggested that operators tried to create an ‘overview’ of one or more lock chambers. Overview gazes are therefore specifically related to an overview of the full nautical object (lock complex), excluding the approaching area. This contrasts with verifying gaze, in that verifying gazes were more directed at specific operating elements. Features of overview gazes (see also Table 1) consist of situations where experts related the fixations to:

i. Creating one mental image/overview of the complete nautical object/operating processes (according to the experts).

ii. Usage of, and switching between, different camera streams of different lock chambers.

iii. Divided attention across different monitors.

Figure 5.

Example still from a video of operator 1 showing overview gaze. Note. The saccade-fixation pattern shows that the operator switched between different lock chambers, looking at streams from the middle towards the upper row (top-middle screen). This can be classified as ‘overview gaze’ because experts related these fixation patterns to creating an overview of one or more lock chambers by switching between different overview camera streams. For example, expert 15 mentioned: ‘But also forming the entire overview image. Yes, you saw that come back a few times. So really visually inspecting those three lock chambers’.

4. Movement-directed gaze. Movement-directed gaze concerns situations where operators’ gaze is drawn by moving objects on camera streams, which may refer to moving nautical traffic and/or moving lock gates. Features of movement-directed gaze behaviour (see also Table 1) consist of situations where experts explicitly related operators’ attention to:

i. Moving lock gates and nautical traffic.

ii. Moving lock gates within the area of which the operator’s colleague is in charge.

iii. Attention towards moving/entering nautical traffic, and towards ‘large’ movements.

Theme 2: Oculomotor Characteristics

The theme ‘oculomotor characteristics’ relates to the impression that experts had of operators’ eye fixations. This theme involves how experts interpreted the nature of the fixations (i.e. how did the fixations come across?) irrespective of the contextual meaning (i.e. what did the fixations mean when associated with the task setting?). We defined the following subthemes:

1. Structure of fixations. Experts noticed that some videos contained a less structured cohesion of fixations. We defined this label as ‘unstructured’ based on the following terms mentioned by experts: ‘no structure’, ‘no consistent gaze pattern’, ‘no standard’, ‘looking around meaninglessly’, ‘all over the place’, or ‘unstructured’. We categorised (a smaller set of) videos as ‘structured’, when experts mentioned terms such as ‘systematic’, ‘in line with protocol steps’, or ‘structured’. Interestingly, five videos were rated both as structured and unstructured, indicating that there is inconsistency across experts about what makes the pattern of fixations (un)structured.

2. Speed of fixations. Experts noticed that the speed of fixations differed between videos, for which speed labels were defined as ‘calm’, ‘restless’, and ‘fast’. Fixations that we clustered as ‘fast’ were characterised by experts as ‘fleeting’, ‘rushed’, ‘short’, and ‘fast (acting/looking)’. Fixations that we clustered as ‘restless’ were characterised by experts as ‘erratic’, ‘jumpy’, ‘chaotic’, ‘messy’, ‘nervous’, ‘no rest’, ‘hop around’, and ‘a lot of switching’. Four videos were categorised as both calm and restless, indicating that experts were not fully congruent.

3. Engagement of fixations. Experts noticed that some fixations (in the video) appeared to indicate that operators were attentively engaged with their operating processes. This deliberate engagement was defined by experts using terms such as ‘conscious’, ‘alert’, ‘deliberately’, or ‘looking/planning actively’.

4. Sequential pattern of fixations. Experts noticed specific line shaped patterns in operators’ fixations (in the video), for example, ‘horizontal/linear pattern’, ‘fixed line pattern’, ‘zigzag pattern’, ‘triangular pattern’, or a ‘circular pattern’.

Theme 3: Relation Between Fixations and Domain Content

Another theme relates to the location of the fixations and, as a result, to the domain-specific content/information that experts retrieved from operators’ fixations. In other words, what specific information did operators (not) attend to according to experts? We defined this theme as the ‘relation between fixations and domain content’ with the following subthemes:

1. Nautical traffic elements. Attending to nautical traffic represents situations where operators’ attention was devoted to nautical traffic in either of two ways: (1) at different cruise phases, or (2) at different physical parts of vessels. For example, experts noticed that operators viewed nautical traffic at specific cruise phases, including vessels entering, exiting, or docking. They also noticed that operators’ attention did (or did not) go towards specific parts of the vessel (e.g. the back and/or propellers of the vessel), or to vessels in general.

2. Lock-specific elements. Attending to lock-specific elements represents specific parts/areas of the nautical object and its surroundings where operators’ attention was allocated according to experts. Main elements consisted of the stop line, lock gates, entering area, lock chamber(s), outer port area, and the approaching area. Some experts were also able to provide sequences and/or associations between lock-specific elements.

3. System-related elements. Experts mentioned the (combination of) systems that were associated with operators’ fixations. Sometimes experts also indicated the order in which the systems appeared relative to the fixations. The system elements that were mentioned by experts consisted of the administrative application (IVS, information and tracking system of vessels), radar application (AIS, automatic identification system), supervisory control system, and camera streams (overview, process, and free choice streams). The fact that experts mentioned the systems that were associated with operators’ fixations suggests that gaze strategies are not only related to what information is viewed by operators, but patterns may also be related to which applications are involved.

Theme 4: Relation Between Fixations and the Task

The last theme, the ‘relation between fixations and the task’, captures what kind of processes and/or activities experts associated with operators’ fixations. We found that experts commented on whether the fixations were related to the operating (gate closing) procedure, or whether fixations were related to any other (supporting) activity. We therefore defined the following subthemes:

1. Execution, monitoring, and control tasks. Experts noticed that operators devoted explicit attention towards their lock gate closing procedure, which we defined as gaze behaviour that is related to execution, monitoring, and control tasks. Examples of labels mentioned by experts were as follows: ‘focus towards the critical operating action’, ‘focus on their “own” lock chamber’ (and not on lock chambers that were not involved in their active operating process), ‘(shortly) viewing process images of camera streams’, and ‘engaged with their own operating process’.

2. Supporting and other activities. Experts noticed that operators focused on process aspects that were not (completely) related to the active gate closing procedure but instead to supporting/other activities. Features of gaze towards supporting activities were mentioned by experts as ‘no/little focus towards the critical operating action/lock gate movement’, ‘focus on aspects not related to the operating process’, ‘focus on other lock chambers’, ‘operating process is a secondary matter’, ‘little focus on camera streams’, ‘not verifying/checking regularly’, or ‘attention to other processes’. In addition, experts also noticed that operators’ fixations were dedicated to ‘administrative actions’ while the lock gate closing procedure was active. Administrative actions were characterised by experts as ‘usage of IVS’, ‘entering information/entry times’, or ‘processing administrative tasks’. Lastly, one expert pointed out a ‘switch between side and main tasks’, which suggests a switch from the operating task to supporting activities.

RQ2: Consistency of Observed eye Gaze

Frequency of Gaze Strategies

We found that experts observed anticipating and verifying gaze features for all operators in all videos. The fact that those two gaze strategies were applicable to all videos suggests consistency in gaze strategies: the anticipating and verifying gaze strategies were consistently observed.

For overview gazes, we found that features of this gaze strategy were observed in all videos except for one. This still suggests consistency of overview gazes for all operators. However, for one operator it seemed that there is variation regarding the overview gaze strategy.

Features of the last gaze strategy, movement-directed gaze, were only found for half of the operators of which two operators showed movement-directed gaze in both videos. As a result, movement-directed gaze seemed not to be consistent across operators based on experts’ observations.

Three example stills from a video of operator 1 show situations that all experts that looked at these videos consistently described as showing a specific gaze strategy, namely, anticipating gaze (Figure 3), verifying gaze (Figure 4), and overview gaze (Figure 5).

Eye Gaze Video Pairing

Experts were unable to consistently match the videos that came from the same operator together. Specifically, none of the experts grouped operators correctly after the first attempt when their matching was based on gaze behaviour and/or task execution. For the second attempt (if applicable), only two experts made correct groupings (again when based on eye gaze and task execution). The matching results from all experts for both attempts are made available as Supplementary Material (Section 7). Based on the finding that different operators were grouped together by experts, an operator’s gaze behaviour (from a video seen as a whole) did not appear to be fully consistent across slightly different contexts (i.e. different videos). For example, both experts 3 and 5 initially matched operators 5 and 10 as the same (videos 5A+5B+10B+10C; see Table 7.1 in Supplementary Material). On the second attempt, expert 3 correctly matched videos 5A+5B (operator 5) and 10B+10C (operator 10), while expert 5 did not (videos 5A+10C and 5B+10B). Interestingly, similar oculomotor characteristics (e.g. ‘calm’) were sometimes mentioned for videos of the same operator. However, final matching results did not show to correspond with these oculomotor characteristics. In addition, the matching task showed that overall gaze behaviour was not the same between operators, because videos within a subset were not consistently grouped together as if they represented one operator.

RQ3: Protocol Versus Gaze Strategies

Experts noticed a variety of actions from operators’ eye movements that were associated with the operating protocol based on the card sorting task. An overview of these protocol steps, including the number of times that these steps were observed by experts, is made available as Supplementary Material (Section 8). For the noticeability of those steps, it did not matter whether the steps were long-lasting processes (e.g. monitoring a specific area) or short-lasting actions (e.g. placing cursor above operating action). Regarding gaze strategies, we noticed that verifying gaze contains features that correspond with the execution and monitoring steps from the operating protocol. The other gaze strategies (anticipating, overview, movement-directed) do not contain features that were associated with these steps.

Experts were also asked which videos they thought were congruent with the protocol. It should be emphasised here that it was not the aim to investigate whether an operator does execute the task appropriately. Hence, deviations from protocol (as mentioned by experts) do not imply that operating procedures were performed incorrectly or insufficiently. When experts compared operators’ gaze behaviour to the protocol, we found that the same videos were clustered together repetitively by different experts (see Supplementary Material (Section 9) for protocol ratings with respect to eye gaze). This finding indicates that experts could rate some gaze behaviour as more in line with respect to protocol application. However, there is no situation where all experts from the same group clustered the exact same video(s) as (mostly) consistent with the protocol. This finding implies that experts did not always fully agree on what gaze behaviour matches protocol. However, we found expert consistency to be more prominent for videos that were not (very) consistent with the protocol. In some groups, all experts listed the same video as not being (very) consistent with the protocol.

In most cases, experts who rated a video as (mostly) consistent mentioned features that correspond with verifying gaze behaviour. For example, monitoring specific lock elements such as the lock gate(s), stop lines, and lock chamber are actions that were covered both by the operating protocol as well as by features of verifying gaze. Interestingly, we found that all videos contain features of verifying gaze when taking all expert observations together, but not all experts did agree on these videos being (fully) consistent with the protocol. For example, sometimes experts mentioned a video to be consistent, whereas they did not mention any features that relate to verifying gaze behaviour. There were also cases where experts rated a video as not (very) consistent, whereas they still noticed features of verifying gaze. Hence, the fact that verifying gaze features have been noticed did not guarantee that it was, according to experts, completely consistent with the operating protocol.

Overview of Results

The identification of gaze strategies (theme 1) went in two ways: (1) either the strategy was directly observed by an expert, or (2) the domain content and process extraction were identified by an expert such that we could identify the associated strategy. For the latter, this implies that multiple features were identified that could be related to those strategies, which are presented in Table 1. This table provides an overview of which lock elements, nautical traffic phases and systems/applications (theme 3: domain content) were associated with a given strategy. Similarly, regarding the task (theme 4), strategies either related to supporting/other activities or to execution, monitoring and control. Items from the operating protocol could be related to the execution, monitoring, and control task, since these items mainly covered operating and monitoring actions. We were not able to add oculomotor characteristics (theme 2) to this overview yet, because experts’ expressions on oculomotor characteristics were more indicative of an impression regarding operators’ gaze behaviour (a video as a whole), not to a particular gaze strategy (multiple strategies in a video).

Discussion

Theoretical Contribution

We report a study on eye gaze in a task that features detection, proactive monitoring, and free viewing in the domain of nautical object control. Previous work on gaze strategies by professionals has mostly focused on visual search and scanning tasks such as radiology (e.g. Drew et al., 2013) and aviation (e.g. Dehais et al., 2017). However, context and the type of task may highly influence visual behaviour (e.g. Land & Hayhoe, 2001; Lohmeyer et al., 2015; Sharvashidze & Schütz, 2020; Wang et al., 2022). Therefore, different task types and different contexts warrant their own detailed studies. Previous work within the domain of nautical object control has mainly focused on technical design guidelines for CCTV (Donald, 2019; Pikaar et al., 2015; Schreibers et al., 2012), not on gaze strategies during task execution.

In our study, experts analysed videos of naturalistic eye gaze from lock operators performing a gate closing procedure. The results showed that experts were able to extract domain content and task features of those strategies from the eye movements. Thematic analysis was necessary to differentiate strategies, since experts did not always have the same terminology when assessing operators’ gaze behaviour.

By means of thematic analysis, we identified four gaze strategies (RQ1): verifying, anticipating, overview, and movement-directed gazes. Compared to other domains (e.g. radiology, aviation), we noticed two general differences. First, we found that specific strategies could be related to the execution of different subprocesses within the operating task (see also Table 1). So-called ‘verifying gaze’ related to execution, monitoring, and control activities (directly related to the operating task), whereas so-called ‘anticipating’, ‘overview’, and ‘movement-directed gazes’ were associated with supporting/other activities (e.g. registrations, planning, reporting). This contrasts with other fields which mostly focused on how specific gaze patterns from professionals were used to address one specific task (e.g. Dehais et al., 2017; Drew et al., 2013; Horiguchi et al., 2016; Thériault et al., 2018; Xu et al., 2019).

Second, we found that all lock operators exhibited different strategies (verifying, anticipating, and overview gazes) during the operating procedure. This contrast with other domains, where a professional seemed to apply mostly one strategy only (e.g. Dehais et al., 2017; Drew et al., 2013). For example, Drew et al. (2013) found that expert radiologists either ‘drilled’ or ‘scanned’ their fixed set of CT scans when searching for lung nodules. Lock operators switched between gaze strategies (e.g. altering between verifying and anticipating gaze). We therefore could not divide lock operators similarly as compared to ‘scanning’ and ‘drilling’ radiologists into, for example, ‘verifying’ and ‘anticipating’ operators. This might be in part due to the free viewing (and thereby: less constrained) nature of the lock operation task (compared to the constrained search task of nodule detection in radiology).

Previous work within CCTV tasks has not explored consistency of gaze strategies between operators (e.g. Stainer et al., 2013; Thériault et al., 2018; Todorova et al., 2017). We found that three strategies were consistently observed among all lock operators (see RQ2: verifying, anticipating, and overview gazes), but experts clearly could not distinguish individual operators based on gaze strategies. Furthermore, when considering gaze behaviour, the same operators were not clustered consistently together by the experts. The open question is, however, whether gaze behaviour is indeed inconsistent, or whether differences among operators’ gaze behaviour were too subtle given that operators, for instance, exhibited all three strategies.

Previous research has also not investigated protocol application in CCTV tasks, let alone the relation between protocol and gaze strategies. In our study, verifying gaze directly related to the operating task of closing the lock gate (RQ3). Features of verifying gaze comprised execution and monitoring elements that were consistently found across all operators, which was in line with our expectation. Although verifying gaze features were observed in all eye gaze videos, not all videos were categorised as in line with protocol. On the one hand, it could be that the type, frequency, and duration with which domain-specific elements of verifying gaze occurred influenced how experts assessed protocol application. On the other hand, if experts had to rate operators’ protocol adherence based on information from one or more themes that we defined, it might be that we would have found more consensus among experts. Interview data did not reveal what specific factors led experts to rate an eye gaze video as (not) consistent with the protocol.

Novel Approach in Exploring Gaze Strategies

To our knowledge, our study is the first to have domain experts evaluate eye movements of working professionals during semi-structured interviews. Previous qualitative studies involving eye tracking often employed retrospective think-aloud instead of semi-structured interviews (e.g. Cho et al., 2019; Elbabour et al., 2017; Prokop et al., 2020). With retrospective think-aloud, participants verbally analyse and view their own naturalistic gaze after the task. A downside of retrospective thinking is that only the perspective and terminology of one operator is noted. By contrast, in our analysis the perspective of multiple experts allowed us to distinguish terminology that only a subset of experts used to those that multiple experts used, as well to compare how terminology (and strategies) applied across different operators. As a result, our thematic analysis identified strategies that both show diversity (input from different experts) as well as consensus (among experts).

The new terminology derived from our thematic analysis (oculomotor characteristics, domain content features, task features) could be adapted to classify gaze strategies in various CCTV working contexts, also outside nautical object control. Table 1 provides a new classification framework for explicitly relating feature- and task-driven gaze strategies. In domains involving multiple (sub)tasks within a CCTV setting or where tasks are related to control and planning activities, this framework could aid in systematically defining and verifying gaze strategies.

For nautical object control, domain content features from Table 1 can be instrumental for quantifying gaze patterns. These defined domain content features can be translated into areas of interest (AOIs) for quantitative analyses, enabling the calculation of metrics such as fixation count, duration, and gaze patterns. For example, anticipating gaze can now be quantitatively measured at the system level (e.g. AOIs for applications like radar and administration) and at the lock element level (e.g. AOIs within CCTV for the approaching and outer port area). Additionally, quantitative analyses can be compared with qualitative outcomes to categorise and verify strategies based on domain expert consensus. For example, if one observes fixations at the lock chambers or lock gates, this could be indicative of three strategies (verifying, overview, or movement-directed gaze, see Table 1). Knowledge of whether this is then related to a certain task (execution, monitoring, and control versus supporting/other), and whether information is gained from the camera streams or also the supervisory control system, can help to differentiate these three strategies.

Study Limitations

There are some limitations in the current set-up. Due to the required expertise, we only had a limited pool of experts that had limited time to view videos. In our design, we chose to have experts view multiple videos in a (Latin square) balanced way. The benefit was that we could analyse how statements by experts generalised over multiple videos (and therefore: operators and scenarios) and over multiple experts (i.e. to their comments align). However, the experts had to form an impression after only viewing a video once, and each only saw a subset of videos. Furthermore, the current data represents qualitative data with a first, global and subjective impression of operators’ eye movements by domain experts in one setting based on single eye gaze video viewings. Therefore, they might have overlooked other aspects of strategies.

Future Work

The current study provides opportunities for future work. First, the current set-up did not include any audio in the videos to preserve anonymity. Future research could explore potential interactions between auditory and visual cues in understanding gaze strategies.

Second, future work can investigate the extent to which these novel gaze strategies are reflected in the task execution of CCTV operators. We observed that a variety of other work-related tasks were present during operators’ naturalistic gaze behaviour whereas the operating protocol explicitly focusses on one task: the operating procedure. Identifying gaze strategies in these more dynamic situations could provide insights into which gaze strategies result in efficient and safe task execution. Certain gaze strategies may be more suitable for particular task phases. Further investigation is needed to explore temporal dynamics regarding transition(s) between gaze strategies within the procedure. With the current terminology at hand, future work can refine the impressions on gaze strategies.

Third, incorporating quantitative analyses in future research could strengthen the robustness of the conclusions. Such quantitative analyses could utilise AOIs derived from our qualitative findings on domain content elements (Table 1), enabling AOI sequence analyses to identify gaze patterns.

Fourth, many studies found a relation between gaze behaviour and expertise (e.g. Martinez-Marquez et al., 2021; Van der Gijp et al., 2017; Ziv, 2016). Our sample of operators included a wide range of working experience. Consequently, it is unknown how working experience influenced interpretation of eye gaze. Future research could focus on how expertise is represented in CCTV operator tasks, including the domain of nautical object control.

Conclusion

The aim was to investigate whether lock operators apply consistent and protocol-related gaze strategies. This topic was explored using a qualitative design, where semi-structured interviews were conducted with domain experts viewing operators’ eye gaze recordings. In sum:

• Thematic analysis provided four gaze strategies based on experts’ observations: anticipating, verifying, overview, and movement-directed gazes (RQ1).

• Anticipating, verifying, and overview gazes were consistently observed among all operators (RQ2).

• Verifying gaze contained features directly associated with the operating protocol (RQ3).

• Gaze strategies of lock operators differed to other domains in that operators perform multiple work-related tasks all with their own gaze strategy. This contrasts with other domains in which professionals exclusively exhibit one gaze strategy for the same task.

• Our new classification framework could help to systematically define and verify gaze strategies based on domain and task features across various CCTV working contexts involving detection, monitoring, control, and planning tasks. For nautical object control, the framework can be instrumental in aiding the interpretation and verification of (quantitative) eye tracking results.

Supplemental Material

Supplemental Material - Lock Operations Through the Operator’s Eyes: A Qualitative Exploration of Gaze Strategies

Supplemental Material for Lock Operations Through the Operator’s Eyes: A Qualitative Exploration of Gaze Strategies by Rutger Stuut, Christian P. Janssen, Stefan Van der Stigchel, and Ellemieke Van Doorn in Journal of Cognitive Engineering and Decision Making

Footnotes

Acknowledgements

Many thanks to the lock operators and domain experts who kindly agreed to take part in this study.

Author Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by RS. The first draft of the manuscript was written by RS and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript. The manuscript is based on PhD work of RS.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by Rijkswaterstaat, the executive body of the Dutch Ministry of Infrastructure and Water Management. The views and opinions expressed in this paper do not reflect the official stance of Rijkswaterstaat or the Dutch government.

ORCID iDs

Rutger Stuut

Christian P. Janssen

Stefan Van der Stigchel

Ellemieke Van Doorn

Data Availability Statement

The data that support the findings of this study are available from Utrecht University but restrictions apply to the availability of these data, which were used under licence for the current study, and so are not publicly available. Data are however available from the authors upon reasonable request and with permission of Utrecht University.

Supplemental Material

Supplemental material for this article is available online.

Ir. Rutger Stuut is an advisor in human-machine interaction at the executive organisation of the Ministry of Infrastructure and Water Management (Rijkswaterstaat). He is also an external PhD candidate in experimental psychology at Utrecht University, Division of Experimental Psychology & Helmholtz Institute. He received his BEng (Honours) in Aviation Engineering from Amsterdam University of Applied Sciences in 2015 and his MSc in Human-Technology Interaction from Eindhoven University of Technology in 2019.

Dr. Christian P. Janssen is an associate professor of human-AI interaction especially in mobility at Utrecht University. He is also co-director of Utrecht’s AI and mobility lab. He received his PhD in human-computer interaction from UCL (2012).

Prof. Dr. Stefan Van der Stigchel is a professor of Cognitive Psychology at Utrecht University. He heads the AttentionLab research group, is the director of the Helmholtz Institute, and is president of the Dutch Society for Brain and Cognition. He is an author of the popular science books ‘How attention works’ and ‘Concentration’, internationally published by MIT Press. He received his PhD in cognitive psychology from VU University (2007).

Dr. ir. Ellemieke Van Doorn leads the human-machine interaction team at NS, the principal Dutch passenger railway operator. She studies the relationships between workload, situation awareness, and human information processing, aiming to prevent human-out-of-loop problems and enhance task performance of operational roles such as train drivers and mechanics. She has a special interest in viewing behaviour and the effect of automation. She earned her doctorate at Delft University of Technology, Faculty of Industrial Design Engineering.

References

Atik

Arslan

(2019). Use of eye tracking for assessment of electronic navigation competency in maritime training. Journal of Eye Movement Research, 12(3), 2019. https://doi.org/10.16910/jemr.12.3.2

Bhavsar

Srinivasan

(2017). Quantifying situation awareness of control room operators using eye-gaze behavior. Computers & Chemical Engineering, 106, 191–201. https://doi.org/10.1016/j.compchemeng.2017.06.004

Biggs

A. T.

Cain

M. S.

Clark

Darling

E. F.

Mitroff

S. R.

(2013). Assessing visual search performance differences between transportation security administration officers and nonprofessional visual searchers. Visual Cognition, 21(3), 330–352. https://doi.org/10.1080/13506285.2013.790329

Braun

Clarke

(2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa

Cho

Powell

Pichon

Kuhns

L. M.

Garofalo

Schnall

(2019). Eye-tracking retrospective think-aloud as a novel approach for a usability evaluation. International Journal of Medical Informatics, 129, 366–373. https://doi.org/10.1016/j.ijmedinf.2019.07.010

Cooper

Gale

Saada

Gedela

Scott

Toms

(2010). The assessment of stroke multidimensional CT and MR imaging using eye movement analysis: Does modality preference enhance observer performance? SPIE. https://doi.org/10.1117/12.843680.7627

Dehais

Behrend

Peysakhovich

Causse

Wickens

C. D.

(2017). Pilot flying and pilot monitoring’s aircraft state awareness during go-around execution in aviation: A behavioral and eye tracking study. The International Journal of Aerospace Psychology, 27(1-2), 15–28. https://doi.org/10.1080/10508414.2017.1366269

Di Nocera

Camilli

Terenzi

(2007). A random glance at the flight deck: Pilots' scanning strategies and the real-time assessment of mental workload. Journal of Cognitive Engineering and Decision Making, 1(3), 271–285. https://doi.org/10.1518/155534307x255627

Donald

F. M.

(2019). Information processing challenges and research directions in CCTV surveillance. Cognition, Technology & Work, 21(3), 487–496. https://doi.org/10.1007/s10111-018-0535-6

10.

Drew

M. L.-H.

Olwal

Jacobson

Seltzer

S. E.

Wolfe

J. M.

(2013). Scanners and drillers: Characterizing expert visual search through volumetric images. Journal of Vision, 13(10), 1–13. https://doi.org/10.1167/13.10.3

11.

Dutch Safety Board . (2016). Accident at Den Uyl Bridge, Zaandam. https://www.onderzoeksraad.nl/en/media/attachment/2018/7/10/4d5de5e914c8summary_den_uylbrug_en.pdf

12.

Dutch Safety Board . (2019). Lessons from the accident prins bernhard bridge zaandam. https://www.onderzoeksraad.nl/nl/media/attachment/2019/11/28/veiligheid_van_op_afstand_bediende_bruggen_november_2019.pdf

13.

Elbabour

Alhadreti

Mayhew

(2017). Eye tracking in retrospective think-aloud usability testing: Is there added value? Journal of Usability Studies, 12, 95–110. https://uxpajournal.org/wp-content/uploads/sites/7/pdf/JUS_Elbabour_May2017.pdf

14.

El Iskandarani

Atweh

J. A.

McGarry

S. P. D.

Riggs

S. L.

Moacdieh

N. M.

(2023). Does it MultiMatch? What scanpath comparison tells us about task performance in teams. Journal of Cognitive Engineering and Decision Making, 17(3), 294–309. https://doi.org/10.1177/15553434231171484

15.

Gray

W. D.

Banerjee

(2021). Constructing expertise: Surmounting performance plateaus by tasks, by tools, and by techniques. Topics in Cognitive Science, 13(4), 610–665. https://doi.org/10.1111/tops.12575

16.

Horiguchi

Suzuki

Sawaragi

Nakanishi

Takimoto

(2016). Extracting train driver’s eye-gaze patterns using graph clustering. IFAC-PapersOnLine, 49(19), 621–626. https://doi.org/10.1016/j.ifacol.2016.10.633

17.

Howard

C. J.

Troscianko

Gilchrist

I. D.

Behera

Hogg

D. C.

(2013). Suspiciousness perception in dynamic scenes: A comparison of CCTV operators and novices. Front Hum Neurosci, 7, 441. https://doi.org/10.3389/fnhum.2013.00441

18.

Kootstra

Teuwen

Goudsmit

Nijboer

Dodd

Van der Stigchel

(2020). Machine learning-based classification of viewing behavior using a wide range of statistical oculomotor features. Journal of Vision, 20(9), 1–15. https://doi.org/10.1167/jov.20.9.1

19.

Kramer

M. R.

Porfido

C. L.

Mitroff

S. R.

(2019). Evaluation of strategies to train visual search performance in professional populations. Current Opinion in Psychology, 29, 113–118. https://doi.org/10.1016/j.copsyc.2019.01.001

20.

Kundel

H. L.

Nodine

C. F.

Conant

E. F.

Weinstein

S. P.

(2007). Holistic component of image perception in mammogram interpretation: Gaze-tracking study. Radiology, 242(2), 396–402. https://doi.org/10.1148/radiol.2422051997

21.

Land

M. F.

Hayhoe

(2001). In what ways do eye movements contribute to everyday activities? Vision Research, 41(25), 3559–3565. https://doi.org/10.1016/S0042-6989(01)00102-X

22.

Langhals

B. T.

Burgoon

J. K.

Nunamaker

J. F.

(2013). Using eye-based psychophysiological cues to enhance screener vigilance. Journal of Cognitive Engineering and Decision Making, 7(1), 83–95. https://doi.org/10.1177/1555343412446308

23.

Lefrançois

Matton

Causse

(2021). Improving airline pilots’ visual scanning and manual flight performance through training on skilled eye gaze strategies. Safety, 7(4), 70. https://www.mdpi.com/2313-576X/7/4/70

24.

Lohmeyer

Mussgnug

Meboldt

(2015). Skimming and scrutinizing: Quantifying two basic patterns of visual behavior in design. In Chakrabarti

(Ed.), ICoRD’15 – Research into Design Across Boundaries Volume 1. Smart Innovation, Systems and Technologies (34, pp. 479–489). Springer. https://doi.org/10.1007/978-81-322-2232-3_42

25.

Manning

Ethell

Donovan

Crawford

(2006). How do radiologists do it? The influence of experience and training on searching for chest nodules. Radiography, 12(2), 134–142. https://doi.org/10.1016/j.radi.2005.02.003

26.

Martinez-Marquez

Pingali

Panuwatwanich

Stewart

R. A.

Mohamed

(2021). Application of eye tracking technology in aviation, maritime, and construction industries: A systematic review. Sensors, 21(13), 4289. https://www.mdpi.com/1424-8220/21/13/4289

27.

Mileva

Burton

A. M.

(2019). Face search in CCTV surveillance. Cognitive Research: Principles and Implications, 4(1), 37. https://doi.org/10.1186/s41235-019-0193-0

28.

Moacdieh

N. M.

Devlin

S. P.

Jundi

Riggs

S. L.

(2020). Effects of workload and workload transitions on attention allocation in a dual-task environment: Evidence from eye tracking metrics. Journal of Cognitive Engineering and Decision Making, 14(2), 132–151. https://doi.org/10.1177/1555343419892184

29.

Peng

Burling

J. M.

Todorova

G. K.

Neary

Pollick

F. E.

(2024). Patterns of saliency and semantic features distinguish gaze of expert and novice viewers of surveillance footage. Psychonomic Bulletin & Review. https://doi.org/10.3758/s13423-024-02454-y

30.

Phillips

Manning

Crawford

Burling

Tam

C.-L.

Taylor

(2008). Searching in axial and 3D CT visualisations Medical Imaging. https://doi.org/10.1117/12.790424

31.

Pikaar

Lenior

Schreibers

De Bruijn

(2015). Human factors guidelines for CCTV system design. Proceedings 19th triennial congress of the IEA. https://www.ergos.eu/inhoud/uploads/2013/04/2015-IEA-Melbourne-Pikaar-CCTV-HF-Guidelines.pdf

32.

Prokop

Pilař

Tichá

(2020). Impact of think-aloud on eye-tracking: A comparison of concurrent and retrospective think-aloud for research on decision-making in the game environment. Sensors, 20(10), 2750. https://doi.org/10.3390/s20102750

33.

Schreibers

K. B. J.

Landman

R. B.

Pikaar

R. N.

(2012). Human factors of CCTV: Part 1 technology and literature review. ErgoS Engineering and Ergonomics. https://www.sintef.no/globalassets/project/hfc/documents/spl386-r-cctv-literature-survey-finalversion.pdf

34.

Sharvashidze

Schütz

A. C.

(2020). Task-dependent eye-movement patterns in viewing art. Journal of Eye Movement Research, 13(2), 2020. https://doi.org/10.16910/jemr.13.2.12

35.

Stainer

Scott-Brown

Tatler

(2013). Looking for trouble: A description of oculomotor search strategies during live CCTV operation. Frontiers in Human Neuroscience, 7. https://doi.org/10.3389/fnhum.2013.00615

36.

Tatler

B. W.

(2021). Searching in CCTV: Effects of organisation in the multiplex. Cognitive Research: Principles and Implications, 6(1), 11. https://doi.org/10.1186/s41235-021-00277-2

37.

Thériault

J.-D.

Roberge-Vallières

Lafond

Tremblay

Vachon

(2018). Chapter 76 - the impact of visual scan strategies on active surveillance performance: An eye-tracking study. In Ayaz

Dehais

(Eds.), Neuroergonomics (pp. 283–284). Academic Press. https://doi.org/10.1016/B978-0-12-811926-6.00076-2

38.

Tien

Atkins

M. S.

Zheng

Swindells

(2010). Measuring situation awareness of surgeons in laparoscopic training proceedings of the 2010 symposium on eye-tracking research & applications. https://doi.org/10.1145/1743666.1743703

39.

Todorova

Burling

Pollick

(2017). The importance of gaze coherence of CCTV operators in facilitating the ability to recognise harmful intentions. Journal of Vision, 17(10), 546. https://doi.org/10.1167/17.10.546

40.

Van der Gijp

Ravesloot

C. J.

Jarodzka

Van Der Schaaf

M. F.

Van Der Schaaf

I. C.

Van Schaik

J. P. J.

Ten Cate

T. J.

(2017). How visual search relates to visual diagnostic performance: A narrative systematic review of eye-tracking research in radiology. Advances in Health Sciences Education, 22(3), 765–787. https://doi.org/10.1007/s10459-016-9698-1

41.

Wang

F. S.

Gianduzzo

Meboldt

Lohmeyer

(2022). An algorithmic approach to determine expertise development using object-related gaze pattern sequences. Behavior Research Methods, 54(1), 493–507. https://doi.org/10.3758/s13428-021-01652-z

42.

Wang

Lin

Cong

Xue

Ochieng

(2021). Effect of working experience on air traffic controller eye movement. Engineering, 7(4), 488–494. https://doi.org/10.1016/j.eng.2020.11.006

43.

Chong

H.-Y.

Liao

P.-c.

(2019). Exploring eye-tracking searching strategies for construction hazard recognition in a laboratory scene. Safety Science, 120, 824–832. https://doi.org/10.1016/j.ssci.2019.08.012

44.

Zimmermann

J. M.

Vicentini

Lohmeyer

Taramasso

Maisano

Meboldt

(2019). Visual behaviour strategies of operators during catheter-based cardiovascular interventions. Journal of Medical Systems, 44(1), 12. https://doi.org/10.1007/s10916-019-1480-5

45.

Ziv

(2016). Gaze behavior and visual attention: A review of eye tracking studies in aviation. The International Journal of Aviation Psychology, 26(3-4), 75–104. https://doi.org/10.1080/10508414.2017.1313096

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.71 MB