Competency Assessment for Laparoscopic Anti-Reflux Surgery: Design and Delphi Review,a Collaboration With the American Foregut Society

Abstract

Background:

Laparoscopic anti-reflux surgery, including hiatus hernia repair, is a common operation performed by both general and thoracic surgeons and an important learning objective for surgical trainees. This study aimed to design a competency assessment instrument for laparoscopic anti-reflux surgery.

Method:

A comprehensive competency assessment instrument was designed by a process of logical analysis by 4 expert thoracic surgeons with an interest in foregut surgery, and then reviewed informally by a panel of experts. The instrument was then further assessed and refined using a modified Delphi process. The Delphi questionnaire was distributed to all members of the Fellowship Training Committee of the American Foregut Society (n = 21).

Results:

A first draft of the competency assessment instrument included 32 steps in 4 categories. The first round of the Delphi review was completed by 14 respondents (response rate 66.7%). A total of 3 rounds of Delphi review were performed. Ultimately, 25 items were retained from the original instrument and 1 modified and 4 new items were added. The final instrument has 30 steps in 4 categories.

Conclusions:

An international and inter-specialty consensus was established on the key components of assessing competence to perform anti-reflux surgery. The resulting instrument could be used to guide competency based assessments of general and thoracic surgeons and trainees.

Introduction

Laparoscopic anti-reflux surgery (LARS), including some degree of hiatus hernia repair and either partial or complete fundoplication, is a complex but commonly performed operation that is a key competency of both general and thoracic surgeons.^1,2 Both the Accreditation Council for Graduate Medical Education in the United States and the Royal College of Physicians and Surgeons of Canada list anti-reflux surgery as an important entrustable professional activity for trainees in both specialties.^{2 –6} Other types of anti-reflux surgery are performed, including open transabdominal and transthoracic approaches, but these are becoming increasingly uncommon. Owing to the complex anatomy and physiology of the gastroesophageal junction, optimal patient outcomes depend on appropriate patient selection/evaluation, excellent surgical technique, and adherence to a set of fundamental surgical principles.

The objective of this study was to develop a competency assessment tool for evaluating performance of LARS. The authors have previously developed other similar tools for core operations in thoracic surgery, called Thoracic Competency Assessment Tools (TCAT)™.^{7 –9} Assessing competence in these procedures is itself a complex endeavor, which must take into account all of the various skills, knowledge, and surgical approaches involved. At the same time, a key consideration when developing a competency assessment instrument is ensuring that it is efficient and straightforward to use and captures only the key aspects of a given procedure. ¹⁰ This task is uniquely suited to the Delphi approach, which allows input from a body of experts on a wide range of possible items for inclusion in the instrument and emphasizes consensus on those items that are broadly applicable and important. ¹¹

Materials and Methods

To facilitate the building of consensus, a draft list of key components of LARS was first developed in an iterative process by a group of 4 general thoracic surgeons (2 Canadians and 2 Americans) at institutions with an interest in foregut surgery. One of these surgeons, the lead investigator for this study, is trained in educational psychology and psychometrics. Items were selected based on the surgeons’ own clinical experience and judgment, intending to generate a list of all the key steps that must be performed as part of any LARS procedure. During this phase, a general framework was utilized that was devised during previous work developing competency assessment tools for anatomic lung resection, invasive mediastinal staging, and mediastinal mass resection/thymectomy. Some items were carried over from those previous instruments to create continuity between assessment forms and to add validity.^{7 –9} Each surgeon was able to argue for or against the inclusion of each item in each initial draft. After 5 iterative draft reviews, the proposed tool was reviewed further by a panel of expert surgeons at a high volume LARS center during an in person round-table discussion followed by a period of email discussion between this pilot panel and the original 4 surgeons. Based on feedback from the pilot review, agreement was reached on an initial list of 32 items in 4 categories: pre-operative, general operative, laparoscopic, and hernia repair/fundoplication.

The list was distributed to surgeon members of the Fellowship Training Committee of the American Foregut Society (AFS) by electronic survey as a modified Delphi process starting in July 2019. The AFS is a group dedicated to promoting multidisciplinary care, education, and research regarding foregut disorders. Three reminders were sent for each round. Respondents were asked whether each item should be included in a competency assessment instrument for LARS. Responses were given by 5-point Likert scale (1 = strongly disagree, 5 = strongly agree). A space for comments was provided for each item. Respondents were asked to suggest any items they felt were missing or should be added to the instrument. Demographic data was collected on respondent practice and experience.

A threshold for consensus was set a priori being a mean response of at least 4/5 and agreement (4 or 5 out of 5) by 2/3 of respondents. The threshold for removing items from consideration entirely was set at mean of less than 3/5. Any items not meeting this threshold would not be included in further Delphi rounds and would be removed from the instrument. The total number of rounds was decided in advance to be a maximum of 3.

Items not meeting the consensus threshold on a given round, but that achieved at least mean 3/5 agreement were sent out again in the consecutive round, along with the mean, median, and mode responses and comments from the previous rounds for respondents to review. Where suggestions were made by respondents to modify a given item, the modifications were included in the subsequent round. New items suggested by respondents for inclusion were also included in subsequent rounds. The instrument development schema is shown in Figure 1. As an educational tool development project, ethical approval was waived by the policies of [institution name].

Figure 1.

Competency Assessment Tool-Laparoscopic Anti-Reflux Surgery (CAT-LARS).™

Results

The first Delphi round had 14 responses (response rate 66.7%). Respondents were predominantly general surgeons from academic (n = 11, 78.6%) and private teaching hospitals (n = 2, 14.3%), with 1 thoracic surgeon at an academic teaching hospital (7.1%). All respondents indicated that LARS comprised over 25% of their clinical practice, with 3 respondents (21.4%) indicating 51%-75% and 4 respondents indicating 76% to 100% of their practice was devoted to LARS. Most respondents had been in practice over 10 years (n = 9, 64.3%), of which 5 had been in practice over 20 years (35.7%). All respondents but 1 had specialized training in foregut surgery, 12 of whom had completed a dedicated foregut fellowship (85.7%), and 1 with on-the-job training (7.1%).

Twenty-three items of the original 32 in Delphi round 1 met the consensus threshold and were preserved for the final instrument (Table 1). Two of these approved items had modifications proposed that were further evaluated in the second round. Of the 9 items not meeting the consensus threshold, none were below the threshold for complete removal, and so all were retained for the second round of review. Four new items were proposed by round 1 respondents for inclusion in round 2.

Table 1.

Round 1 of Delphi Procedure.

Item	5 or 4 (%)	3 (%)	2 or 1 (%)	Mean	Selected comment
Able to assess a patient’s suitability for a given repair based on patient and disease-specific factors.	13 (92.9)	1 (7.1)	0 (0)	4.9	Essential-without a clear understanding of the definition and spectrum of GERD and the many associated conditions and diseases that impact outcomes, one should not consider offering surgical treatment for GERD.
Able to describe the risks and benefits of the operation as would be required to obtain informed consent.	12 (85.7)	2 (14.3)	0 (0)	4.6
Competently interprets relevant esophageal functional studies.	14 (100)	0 (0)	0 (0)	4.9
Competently interprets relevant imaging studies.	13 (92.9)	1 (7.1)	0 (0)	4.6
Demonstrates appropriate tissue handling.	12 (92.3)	0 (0)	1 (7.1)	4.5
Performs efficient, purposeful hand motions without unnecessary movements.	12 (85.7)	1 (7.1)	1 (7.1)	4.4
Makes appropriate use of assistants.	12 (85.7)	2 (14.3)	0 (0)	4.4
Provides appropriate assistance to another operator.	12 (85.7)	2 (14.3)	0 (0)	4.3
Communicates effectively, including with the patient, patient’s family, anesthesia, and the surgical team.	12 (85.7)	2 (14.3)	0 (0)	4.4	When pleura is open inadvertently, or bleeding. . .or emergence from anesthesia
Makes an appropriate plan for post-operative orders and management.	11 (78.6)	3 (21.4)	0 (0)	4.4
ΔProperly identifies upper abdominal anatomy.	13 (92.9)	1 (7.1)	0 (0)	4.8	Properly identify the area of the gastroesophageal junction.
Properly exposes the hiatus, with use of a liver retractor as appropriate.	13 (100)	0 (0)	0 (0)	4.9
ΔProperly reduces and excises the hernia sac, as appropriate.	14 (100)	0 (0)	0 (0)	5	Mediastinal dissection.
Properly assesses esophageal length.	14 (100)	0 (0)	0 (0)	5
Safely and effectively adequately mobilizes a sufficient length of the esophagus, preserving the vagus nerves.	13 (100)	0 (0)	0 (0)	5
Properly preserves the integrity of the crura during dissection.	14 (100)	0 (0)	0 (0)	4.8
Safely mobilizes the greater curve and fundus, dividing short gastric arteries as appropriate.	14 (100)	0 (0)	0 (0)	5	Safely and atraumatically.
Properly closes the crura, assesses for tension, and selects appropriate maneuvers to address tension, as appropriate.	14 (100)	0 (0)	0 (0)	5
Properly recreates the angle of His.	11 (78.6)	3 (21.4)	0 (0)	4.4
Safely passes/directs passage of the bougie, as appropriate.	12 (85.7)	0 (0)	2 (14.3)	4.3
Properly creates a fundoplication of appropriate orientation, length, and tightness.	14 (100)	0 (0)	0 (0)	5	With appropriately placed and secured sutures.
Ensures adequate hemostasis.	10 (71.4)	3 (21.4)	1 (7.1)	4.1
The above steps are performed in a safe, efficient order.
Properly positions the patient.	9 (64.3)	3 (21.4)	2 (14.3)	4.1	This varies a lot—not essential, but should have a guide.
Ensures proper sterile prep and draping of the patient.	6 (42.9)	2 (14.3)	6 (42.9)	3.1	This is a basic skill not related to the topic at hand.
Competently performs the pre-operative safety checklist/time-out.	6 (42.9)	3 (21.4)	5 (35.7)	3.3	This is a basic skill not related to the topic at hand.
Properly positions, creates, and closes incisions(s).	9 (64.3)	1 (7.1)	4 (28.6)	3.8	This is a basic skill not related to the topic at hand.
Safely introduces ports.	10 (71.4)	4 (28.6)	0 (0)	4.2	Basic skill, port location options maybe included
Competently directs the use of the laparoscope to visualize the operative field.	9 (64.3)	4 (28.6)	1 (7.1)	4.1	Basic skill, brief comments, scope angles options maybe included
Appropriately and pre-emptively identifies pre-operative conditions that may contribute to a need to convert to laparotomy.	9 (64.3)	4 (28.6)	1 (7.1)	3.9	This is so rare nowadays. Can’t think why [you would] convert, only if cannot get access.
Appropriately and pre-emptively identifies intra-operative conditions that may contribute to a need to convert to laparotomy.	9 (64.3)	4 (28.6)	1 (7.1)	4.1	You maybe referring to disasters (aortic or IVC injury?) I suppose?
Properly performs esophageal lengthening procedure, as necessary.	8 (57.1)	5 (35.7)	1 (7.1)	3.9	With the caveat that this is very controversial and should be rarely needed

Note. Unshaded items met the threshold and were included in the final tool.

Shaded items did not meet the threshold for consensus and were further reviewed in round 2 with comments and metrics from round one.

Δ

Modified version of approved item included in round 2 with previous version’s comments and metrics from round one.

The second Delphi round had 7 responses, out of the original 14 respondents (50.0%). In this round, 1 unchanged item from round 1 met the consensus threshold, along with all 4 of the newly proposed items (Table 2). Ten items did not meet the consensus threshold, including 8 unchanged from round one, and 2 that had been modified. Seven of the 10 were below the threshold for complete removal and were not carried forward to round 3. Three items were below the threshold for inclusion but above the threshold for removal and so were carried forward to round 3. One of these was modified according to suggestions from respondents. One new item was suggested.

Table 2.

Round 2 of Delphi Procedure.

Item	5 or 4 (%)	3 (%)	2 or 1 (%)	Mean	Selected comments
Unchanged items from round 1 for further review
Properly performs esophageal lengthening procedure, as necessary.	7 (100)	0 (0)	0 (0)	4.6	Surgeon should not be doing foregut surgery if not familiar with how to do this safely.
Properly positions the patient.	5 (71.4)	1 (14.2)	1 (14.2)	3.7	1) Not pertinent.2) If the patient is not in some level of reverse [Tredelenberg], it makes it considerably harder.
#Ensures proper sterile prep and draping of the patient.	0 (0)	2 (28.6)	5 (71.4)	2	Basic skill, not pertinent.
#Competently performs the pre-operative safety checklist/time-out.	0 (0)	1 (14.2)	6 (85.7)	1.9	Should not be included-everybody does a checklist.
#Properly positions, creates, and closes incisions.	2 (28.6)	1 (14.2)	4 (57.1)	2.7	Port positioning is essential; closure is basic skill.
Safely introduces ports.	3 (42.9)	2 (28.6)	2 (28.6)	3.1
#Competently directs the use of the laparoscope to visualize the operative field.	2 (28.6)	2 (28.6)	3 (42.9)	2.9
#Appropriately and pre-emptively identifies pre-operative conditions that may contribute to a need to convert to laparotomy.	1 (14.2)	1 (14.2)	5 (71.4)	2.1	While rare, it is important to identify.
Appropriately and pre-emptively identifies intra-operative conditions that may contribute to a need to convert to laparotomy.	4 (57.1)	2 (28.6)	1 (14.2)	3.3	No, should NOT be included.
Items modified from round 1 version
#Properly identifies upper abdominal anatomy including the GE junction.	0 (0)	1 (14.3)	6 (85.7)	2.1
#Properly reduces and excises the hernia sac, including performing mediastinal dissection, as appropriate.	0 (0)	0 (0)	7 (100)	1.6
New items suggested by respondents during round 1
Performs an adequate “shoe-shine” maneuver.	7 (100)	0 (0)	0 (0)	4.6
Performs appropriate intra-corporeal suturing.	7 (100)	0 (0)	0 (0)	4.6	Surgery cannot be done without this.
Appropriately performs intra-operative endoscopy to assess anatomy and rule out complications.	6 (85.7)	1 (14.2)	0 (0)	4.3	Critical skill for foregut surgeons.
Is able to discuss the evidence regarding efficacy and complications of various types of fundoplication.	6 (100)	0 (0)	0 (0)	4.8

Note. Unshaded items met the threshold and were included in the final tool.

Shaded items did not meet the threshold for consensus.

#

Below threshold for removal, not included in round 3.

The third Delphi round had 9 respondents, out of the original 14 (64.3%). Two of the 4 items in this round met the threshold for inclusion, including 1 item that was unchanged from rounds 1 and 2, and 1 item modified from round 2 (Table 3). Two items did not meet the threshold for inclusion, including 1 item that was unchanged from rounds 1 and 2 and 1 new item suggested in round 2. There were no new items suggested in this final round.

Table 3.

Round 3 of Delphi Procedure.

Item	5 or 4 (%)	3 (%)	2 or 1 (%)	Mean	Selected comments
Unchanged items from round 2 for further review
Properly positions the patient.	7 (77.8)	1 (11.1)	1 (11.1)	4.2	Proper positioning and protection of the patient is paramount.
Appropriately and pre-emptively identifies intra-operative conditions that may contribute to a need to convert to laparotomy.	3 (33.3)	0 (0)	6 (66.7)	2.7	Conversion is rare, and doing so is no guarantee that the operation will get easier.
Items modified from round 2 version
Properly manages the location, insertion and closure of ports.	6 (66.7)	2 (22.2)	1 (11.1)	4.0
New items suggested by respondents during round 2
Competently places gastropexy sutures when appropriate.	6 (66.7)	1 (11.1)	1 (11.1)	3.7	Some do, some don’t, and nobody has ever shown these to be beneficial.

Note. Unshaded items met the threshold and were included in the final tool.

Shaded items did not meet the threshold for consensus and were discarded.

Across 3 rounds, 40 items or item modifications were considered and 30 of these eventually met the consensus threshold for inclusion. The items were reorganized into 4 categories: pre-operative, general operative, hernia repair, and fundoplication. The final version of the form is presented in Figure 1.

Discussion

This study presents the development and refinement of the Competency Assessment Tool for LARS (CAT-LARS)™ by Canadian and American experts from both general and thoracic surgery. The robust process employed, involving creation by content experts, pilot review by an expert panel, and then a 3 round modified Delphi process with surgical educators from the American Foregut Society, and the involvement of both thoracic and general foregut surgeons from both American and Canada establishes a significant source of content validity evidence for this instrument.

As competency based medical education gains importance in post-graduate and continuing medical education, there is a growing need for well-developed competency assessment instruments and sets of entrustable professional activities. ¹² Competency assessment is especially important in surgical specialties, where a direct correlation exists between the ability of the surgeon and the outcome for the patient. ¹³ Global assessment tools, such as the Global Operative Assessment of Laparoscopic Skills (GOALS), ¹⁴ the Objective Structured Assessment of Technical Skills (OSATS), ¹⁵ and the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) ¹⁶ have the advantage of being simple and quick to use, and are applicable to a wide variety of tasks. However, the lack of procedure-specific, detailed feedback may limit the utility of these tools, especially at high levels of skill acquisition where more nuanced feedback may be needed to refine already existing skills. ¹⁷ However, by including both general surgical principles and advanced skills, CAT-LARS may be useful for a broad range of learners, from junior residents who may only be permitted to attempt some basic steps, all the way through to master surgeons. The inclusion of both cognitive and technical aspects allows the learner an opportunity to demonstrate knowledge that may only otherwise be assessed in a clinic setting.

Interestingly, at the same time CAT-LARS was under development, the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) independently developed their own competency assessment instrument for LARS, called the Video-Based Assessment for Laparoscopic Fundoplication (VBA for LF, Figure 2).^18,19 This provides a unique opportunity to compare the 2 instruments to assess for areas of convergent validity evidence. The SAGES VBA for LF includes 5 steps and 11 sub-steps, 9 skills, and 42 possible errors. Perhaps because it is intended for use with laparoscopic video recordings, the VBA for LF does not have a corollary for any of the pre-operative competencies in CAT-LARS, nor for competencies such as communication, port placement, and the post-operative plan. Similarly, more cognitive competencies that were included in CAT-LARS, such as patient assessment, providing informed consent, understanding the evidence for fundoplication, and the interpretation of imaging and esophageal functional studies, are not included in the SAGES tool. We feel these are important aspects of understanding a surgeon’s ability to appropriately select patients for LARS and perform LARS on a wholistic level, though we recognize they would be difficult or impossible to capture via laparoscopic video and therefore out of scope for a VBA.

Figure 2.

SAGES video based assessment for laparoscopic fundoplication.

Seven of the 30 steps in the CAT-LAR can be seen to correspond to either a step, sub-step, or skill in the VBA for LF: tissue handling, efficient/purposeful hand motions, crural closure, intra-corporeal suturing, mobilization of the greater curve/fundus, and creation of the fundoplication. These steps therefore have convergent evidence of high content validity and should definitely be included in any assessment of LARS competency. Only 5 of the 9 competencies in the CATS-LARS hernia repair category had any corollary in the VBA for LF, perhaps indicating that the VBA for LF is more appropriate for patients with primary acid reflux disease without a significant hernia component, while the CAT-LARS may be applicable in patients with and without a significant hiatus hernia. Indeed, the authors of that study note that further research is needed to determine the applicability of the VBA for LF for hiatal hernias >4 cm.

Conversely, the VBA for LF includes more granular detail about wrap creation than CAT-LARS, devoting 1 step, 5 sub-steps and 10 potential errors to this crucial part of the procedure, compared to 8 competencies in CAT-LARS. There is also limited overlap between the 2 tools in this area, with both tools including seemingly critical points that are not captured by the other (eg, avoiding injury to the esophagus is included only in VBA for LF, recreating the angle of His is included only in CAT-LARS). This perhaps reflects the difficulty of capturing all of the complex subtleties of this procedure when performed at an expert level.

Finally, 3 competencies that were included in CAT-LARS were also considered for use in the VBA for LF but were ultimately excluded based on their task inventory questionnaire. These competencies (use of assistants, exposure of the hiatus with a liver retractor and intra-operative endoscopy) were rated highly by our Delphi panel but should be reassessed in future studies given the divergent findings of the SAGES group.

While this study provides initial content validity evidence for CAT-LARS, further study will be needed to collect additional evidence of validity and reliability when used in a real-world setting with trainees and practicing surgeons. This tool was developed from a North American perspective and the involvement of foregut surgeons from other parts of the world would add validity. This study is limited by the response rate to each of the 3 Delphi surveys. The intended use of this tool is to provide ongoing formative feedback to trainees throughout their instruction, supplementing, and supporting the ongoing transition to competency based surgical education. Because CAT-LARS, in our experience, is fast and easy to use, it could be completed frequently for routine feedback over the course of a trainee’s rotation, residency or fellowship, providing a longitudinal record of progress, and highlighting any areas that are consistently in need of improvement. It may also be useful for practicing surgeons undergoing peer-review or a periodic self-assessment of competency.

Any surgeon performing LARS should be aware of the competencies identified in this study, which could be seen as a guide to the minimum acceptable skillset needed to conduct this complex operation safely and effectively.