Force sensing in robot-assisted keyhole endoscopy: A systematic survey

Abstract

Instrument–tissue interaction forces in minimally invasive surgery (MIS) provide valuable information that can be used to provide haptic perception, monitor tissue trauma, develop training guidelines, and evaluate the skill level of novice and expert surgeons. Force and tactile sensing is lost in many robot-assisted surgery (RAS) systems. Therefore, many researchers have focused on recovering this information through sensing systems and estimation algorithms. This article provides a comprehensive systematic review of the current force sensing research aimed at RAS and, more generally, keyhole endoscopy, in which instruments enter the body through small incisions. Articles published between January 2011 and May 2020 are considered, following the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines. The literature search resulted in 110 papers on different force estimation algorithms and sensing technologies, sensor design specifications, and fabrication techniques.

1. Introduction

In minimally invasive surgery (MIS), surgical access is provided through small incisions or natural orifices in the body. A surgical instrument is operated by the surgeon for tissue manipulation. Compared with open surgery, MIS provides less tissue trauma, postoperative pain, patient discomfort, wound complications and immunological response stress (Wottawa et al., 2016), lower risk of infection (Soltani-Zarrin et al., 2018) and blood loss (Dai et al., 2017), shorter hospital stay (Bandari et al., 2020), faster recovery (Lee et al., 2015), and improved cosmetic appearance (Aviles et al., 2016), all of which lead to improved therapeutic outcome and efficiency (Otte et al., 2016) and lower morbidity and mortality (Aviles et al., 2017) making MIS cost-effective (Faragasso et al., 2014). Nonetheless, the ergonomically cumbersome posture increases surgeon fatigue. The limited instrument dexterity and visual perception of the scene (Haghighipanah et al., 2017; Haouchine et al., 2018), and the non-intuitive hand–eye coordination due to fulcrum motion reversal decrease accuracy and contribute to surgeon fatigue (Hadi Hosseinabadi et al., 2019). The high level of psychomotor skills required increases the operation time and require a longer learning curve (Shahzada et al., 2016). The sense of touch is reduced by friction in the access port and instrument mechanism.

Robotic MIS (RMIS) can be comanipulated or teleoperated. In comanipulated RMIS, the doctor is close to the patient holding the surgical instruments and the robot mainly acts as a filter, an active assistance to improve performance (Zhan et al., 2015). In these systems, the force perception, in interaction with the surgical site, exists to a certain extent. In teleoperated RMIS, the surgical instrument is controlled by a robotic manipulator and operated by a remote surgeon. The robotic operation restores hand–eye coordination (Aviles et al., 2015a) and innovations in tool design improve dexterity leading to improved ergonomics that reduce surgeon fatigue (Bandari et al., 2020; Stephens et al., 2019). The enhanced 3D vision with which robotic systems are provided and not used in laparoscopic or manual surgery, automatic movement transformations, fine motions, filtering of physiological hand tremor and motion scaling lead to improved surgery precision (Sang et al., 2017). However, the surgeon is isolated from the surgical site by robotic manipulators that do not provide the haptic perception (Juo et al., 2020). This deprives the surgeon of a rich source of information. Thus, many studies are targeted towards the reconstruction and evaluation of haptic feedback.

Haptics can be either tactile or kinesthetic (Okamura, 2009). The tactile perception is through the cutaneous receptors in the skin which can sense, for example, texture or temperature (Mack et al., 2012). The kinesthetic force feedback (FF) is perceived by mechanoreceptors in the muscle tendons to detect force, position and velocity information about objects (Juo et al., 2020). Traditionally surgeons use palpation to characterize tissue properties, detect nerves and arteries (Bandari et al., 2017), and identify abnormalities such as lumps and tumors (Lv et al., 2020; Puangmali et al., 2012). Moreover, the surgeons rely on the sense of touch to regulate the applied forces. Excessive forces can lead to tissue trauma, internal bleeding, and broken sutures. Insufficient forces, however, can lead to loose knots and poor sutures. (Li et al., 2016; Sang et al., 2017).

Direct FF and sensory substitution (SS) are the most common approaches of presenting surgeons with force information. Although the direct method provides the most intuitive interaction (Li and Hannaford, 2017), it is the most challenging to implement, as it requires a method of force sensing and a safe and robust teleoperation interface for force reflection. A compromise between transparency and stability of different teleoperation frameworks is reported by (Hashtrudi-Zaad and Salcudean, 2001). In SS, visual, auditory, or vibro-tactile signals provide haptic perception to the surgeon. While safety can be easily guaranteed, this method can cause discomfort, distraction, and cognitive overload. In general, visual methods are shown to be the most effective feedback modality (Abdi et al., 2020).

In summary, the introduction of haptic perception is proven to decrease operation time (Abiri et al., 2017), facilitate training, improve accuracy, and enhance patient safety for novice surgeons in complex tasks (Juo et al., 2020). More experienced surgeons learn to infer force information from visual cues such as the tissue and instrument deformations and the stretch in sutures (Aviles et al., 2017). In addition, force information can be used to automate surgical robot tasks in dynamic and unstructured environments (Kuang et al., 2020), to identify tissues in real time, to create tissue-realistic models and simulators for training (Stephens et al., 2019), and to perform surgical skills assessment (Soltani-Zarrin et al., 2018).

1.1. Comparison with the existing reviews

An extensive review of haptic perception and its efficacy in RMIS was presented by Amirabdollahian et al. (2018). This review concluded that while there is a consensus on the need for haptic and tactile feedback, no commercial system is yet available that addresses this need. More recently, El Rassi and El Rassi (2020) presented a brief overview of haptic feedback in teleoperated robotic surgery. Overtoom et al. (2019) and Rangarajan et al. (2020) surveyed virtual haptics in surgical simulation and training. The latter followed the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines to identify the relevant literature. The authors similarly affirm the efficacy of haptic feedback in surgical education. None of these publications reviewed the developments in the field of force sensing and estimation.

Abdi et al. (2020) reviewed research since 2000 on the efficacy of haptic feedback in teleoperated medical interventions. The authors presented a concise overview of the force-sensing literature with 44 references cited over a wide range of medical applications. Although the review provides a general understanding of the challenges and complexities in instrument–tissue force measurement, it is not a comprehensive presentation of the prominent developments and the articles were subjectively selected with no evaluation criteria. In addition, the records were only classified based on the sensing technology and the sensor location; however, the instrument’s dexterity level, the sensing degrees of freedom (DoFs), and the performance measures were not compared. A comparison of its references with the records cited in our review shows an overlap of only 20 out of 110 papers. Bandari et al. (2020) reviewed tactile sensing literature over the past 20 years. It also includes some literature on force sensing in neurosurgery and microsurgery procedures. Although the authors presented a comprehensive review with 121 references, a comparison of the included articles with the records in this paper shows an overlap of only 8 out of 110 papers which are mostly on developments related to the gripping force sensing.

This article is a systematic review based on the PRISMA guidelines that expands on the sensor design requirements and presents the most recent developments in force sensing and estimation in keyhole endoscopy. We discuss how research has evolved over the past decade and provide suggestions for future research directions. The closest publications to our review are the surveys by Puangmali et al. (2008) and Trejos et al. (2010) which were published about a decade ago, and therefore there are no overlapping papers with those reviews.

2. Methodology

A systematic survey was conducted by following the PRISMA guidelines (see Figure 1) and it was based on Google Scholar, Web-of-Science, PubMed, and IEEE Xplore Digital Library repositories. The period for the review is over the past decade, from January 2011 until May 2020. The following keywords were used for identification: Force sensing, Kinesthetic, Tactile, Haptics, Minimally Invasive Surgery, MIS, MIRS, Robot-Assisted, RMIS, RAS, RAMIS, Laparoscopy, and Endoscopy. For every year, the first 20 pages of search results in Google Scholar were surveyed (total of 2,000 records). The same approach was used for the identification of records through the other repositories (PubMed, 213; Web-of-Science, 42; and IEEEXplore, 40). For screening, the duplicates were removed and the identified records were skimmed through to mark the ones that are relevant to keyhole endoscopy. The articles that refer to force sensing in microsurgery, neurosurgery, and needle insertion were excluded because they involve a different set of requirements and challenges. Specifically, microsurgical instruments such as those used in neurosurgery and retinal surgery (Gonenc et al., 2017) have a much smaller diameter (less than 2 mm) and do not require an articulated wrist, which complicates the actuation system and sensors’ power and signal routing. Moreover, Bandari et al. (2020) briefly discussed the force-sensing literature in microsurgery, neurosurgery, and needle insertion. A total of 114 articles were found eligible for a complete review. Throughout the review, the references of the selected papers were surveyed and the relevant articles that were not initially identified were added, thus increasing the total number of eligible records to 129. The work progressions and duplicate publications were removed to lead to the 110 articles included in this survey.

Fig. 1.

PRISMA flow diagram.

The included articles are tabulated for an easier comparison of the method, the sensor location, the sensing DoFs, the dexterity of the instrument under study, and the results. The dexterity index (DI) for different instruments is defined according to Table 1 and Figure 2. Standalone testing in this table indicates where a sensor design was presented without discussing its integration into a surgical instrument. Depending on the sensor location, the sensing DoFs are defined as instrument or wrist tri-axial forces (F_X, F_Y, F_Z) and moments (M_X, M_Y, F_Z), and the gripper normal (F_N), shear (F_S), and pull (F_P) forces as depicted in Figure 3. In summarizing the results, the following acronyms were used: ACC, accuracy; ERR, maximum absolute error; MAE, mean absolute error; NRMSE, normalized root-mean-squared error; RES, resolution; RMSE, root-mean-squared error; RNG, range; and SENS, sensitivity.

Table 1.

Dexterity index definition for MIS instruments.

DI	Instrument functionality and DoFs
—	Standalone testing
0	Palpation
1	Grasping
2	Grasping + axial rotation
3	Grasping + flexion
4	Grasping + flexion + axial rotation
5	Grasping + flexion + abduction
6	Grasping + flexion + abduction + axial rotation

Table 2.

Sensorless force estimation: model-based.

Study	Method	Sensing DoFs	Instrument/DI	Results
1. Li et al. (2013)	Pneumatic actuation muscles (PAMs) were used in design of a custom forceps. Disturbance observers were used on the pneumatic actuation system and the robot joints.	Instrument tri-axialforces	Custom-developed(CD) RMIS forceps / 5	ERR < 0.4 N RNG: 0–3.5 N
2. Tsukamoto and Ishii (2014)	Proposed a three-step Robust Reaction Torque Observer (RRTO): (1) cancel the error in estimated torque (overshoot correction); (2) identify and compensate the inertial torque component in the drive train (inertia compensation); (3) estimate the gripping torque.	Grip torque	CD RMIS instrument / 4	Plot comparison,not quantified.
3. Anooshahpour et al. (2014)	Proposed two quasi-static models on cable dynamics (pull and pull–push) that take tendons friction and compliance into account. A linear combination of the two models provided a close estimation of the output gripping torque.	Grip torque	EndoWrist Needle Driver / 6	Plot comparison,not quantified. RNG:±40 Nmm
4. Lee et al. (2014)	The gripper was actuated by a pneumatic catheter balloon to provide a uniform gripping force. The pneumatic pressure was monitored to estimate the grip force.	Grip force	CD RMIS instrument / 5	ERR < 0.3 N RNG:0–10 N
5. Zhao and Nelson (2015)	A wrist actuation design using planetary gears was proposed to decouple the motions in different DoFs. The motor currents were used to estimate the forces in static and dynamic scenarios.	Instrumentlateral forces,grip force	CD RMIS instrument / 5	ERR < 0.4 N RNG:0–2 N
6. Lee et al. (2015)	Proposed the compensation of the gripping torque which was experimentally identified as a function of the instrument posture. The experiments were on: (1) EndoWrist; (2) ProGrasp; (3) large needle driver; (4) dissecting forceps.	Grip force	EndoWrist instruments / 6	(1) ERR < 10.69%(2) ERR < 13.03%(3) ERR < 16.25%
7. Haraguchi et al. (2015)	The articulated wrist was replaced by a machined spring. The instrument was pneumatically driven. A 3-DoF continuum model of the spring distal joint and the pneumatic pressure were used for force estimation.	Forceps tri-axial	CD RMIS instrument / 6	ERR <0.37 N RNG: ±5 N
8. Yoon et al. (2015)	Proposed the use of sliding perturbation observer (SPO) to estimate the reaction force. The presented method compensated for the Coulomb friction.	Grip force, pitchtorque, instrumentaxial torque	EndoWrist ProGrasp / 6	RNG: FG: 0–10N Pitch torque: ±150 NmmAxial torque: ±1 Nm
9. Rahman and Lee (2015)	Cascaded fuzzy logic in sliding mode control with SPO (SMCSPO) to separate different types of disturbances. A rectified position information was defined and used to estimate the perturbation (grip force).	Grip force	EndoWrist ProGrasp / 6	RNG: 0–15 N
10. Li et al. (2016)	Proposed the use of an unscented Kalman filter (UKF) based on a dynamic model that considered cable properties, cable friction, cable–pulley friction, and a bounding filter. The proposed approach used motor currents and motor encoder readings.	Grip force	Raven-II 10 mm gripper / 6	ERR < 50% RNG: 0–1 N
11. Anooshahpour et al. (2016)	Proposed the use of Preisach approach to model the input–output hysteretic behavior in a daVinci instrument.	Grip force	EndoWrist Needle Driver / 6	ERR < 0.6 N RNG: 0–6.5 N
12. Sang et al. (2017)	Developed and identified a dynamic model for the Patient Side Manipulator (PSM) of the daVinci Standard system and the surgical instrument. The identified model was used for external force estimation.	Instrument tri-axial forces	EndoWrist Needle Driver / 6	ERR < 0.1 N RNG: ±1.5 N
13. Haghighipanah et al. (2017)	Evaluated two approaches for force estimation on the 3rd link of the Raven-II system: (1) further expanded on the approach in Li et al. (2016) by adding cable tension estimation; (2) the force was estimated by measuring the cable stretch using a linear encoder.	Instrument axial force	Raven-II 10 mm gripper / 6	ERR 1) < 4 N, 2) <3 N RNG: 0–10 N #2 Provided better estimationat lower forces
14. Li and Hannaford (2017)	Used the Gaussian Process regression (GPR) supervised learning approach because of its ability to deal with uncertainties and non-linearity. The model inputs were motors encoder, velocity, and current.	Grip force	Raven-II 10 mm gripper / 6	ERR < 0.07 N RNG: 0–1 N
15. Xin et al. (2017)	Developed the dynamic model of one jaw by using the Benson model to describe the dry friction. The parameters were experimentally identified for an instrument designed based on the concept in Zhao and Nelson (2015).	Grip force	CD RMIS instrument / 5	ERR < 0.25 N RNG:0–2.5 N
16. O’Neill et al. (2018)	Evaluated motor current command and measurement, and differential gearbox as proximal torque surrogates and used neural networks (NNs) to estimate the distal gripping torque considering all three surrogates as inputs.	Grip force	daVinci Si Maryland grasper / 6	ERR < 0.37 NRNG: 0–11 N
17. Huang et al. (2018)	Proposed the use of NNs optimized by a genetic algorithm (GA) for force estimation. The model inputs were the motors’ positions, velocities, and currents.	Grip force	CD RMIS instrument / 5	ERR < 0.06 N RNG:0–1.6 N
18. Takeishi et al. (2019)	Suggested the use of pneumatic actuators and NN for force estimation. Low accuracy in abrupt forces was reported. All the analysis was model-based in MATLAB.	—	Simulation	RNG: 0–10 N
19. Abeywardena et al. (2019)	A NN architecture with LSTM was proposed that used motors currents as the inputs. The model was trained for different stages of no grasp, closing, and opening.	Grip force	EndoWrist ProGrasp / 6	ERR < 0.4 N RNG: 0–20 N
20. Stephens et al. (2019)	The performance of NNs, decision tree, random forest, and support vector machine models were compared in the angle and gripping torque estimation of each jaw. It concluded that the NN estimations were reliable when trained and tested on each jaw, on the same tool, and withing the frequency of the training data.	Grip force	EndoWrist ProGrasp / 6	ERR < 0.07 N RNG: 0–5.5 N
21. Wang et al. (2019b)	Proposed an external force estimation method based on cable-tension disturbance observer and the motion control strategy.	Grip force	CD RMIS instrument / 5	ACC > 85% RNG: 0.1–2 N

Fig. 2.

Instrument’s DoFs.

Fig. 3.

Sensing DoFs depending on the sensor location.

3. Design requirements

3.1. DoFs, range, resolution, accuracy, bandwidth, and sampling rate

The grasping force, the instrument lateral and axial forces, and the axial torque are the most relevant DoFs to improve accuracy and provide an effective haptic experience in MIS applications (Bandari et al., 2020; Soltani-Zarrin et al., 2018; Wee et al., 2016). Deformations in the sensor structure or displacements in its components are the physical surrogates that are monitored for force estimation. Thus, there are always trade-offs between the sensor’s structural rigidity, resolution, sensitivity, and range (Puangmali et al., 2012). The grip force can reach up to 20 N in daVinci instruments during needle handling or knot-tying (Bandari et al., 2020; O’Neill et al., 2018); however, pinch forces as large as 4 N can cause damage to delicate tissue (Abiri et al., 2019; Hong and Jo, 2012). The maximum allowable suture pull force is 4–6 N (Dai et al., 2017; Spiers et al., 2015). The optimal kinesthetic force range suggested for MIS applications is ±10 N in all directions and ±20 N for grasping (Khadem et al., 2016; Wee et al., 2017). No requirement on bending moments and axial torque is specified in the literature (Hadi Hosseinabadi et al., 2019). Resolutions of 0.06 N (Choi et al., 2017; Wang et al., 2014) and 0.2 N (Soltani-Zarrin et al., 2018) are suggested for FF and SS schemes, respectively. The human just-noticeable difference (JND) is 10% (Kim et al., 2017; Wang et al., 2014) in the range of 0.5 to 200 N increasing to 15–27% below 0.5 N (Hwang and Lim, 2017), which can be considered as a requirement on the sensor accuracy. The human’s temporal resolution is 320 Hz for force discrimination and up to 700 Hz for vibration detection (Puangmali et al., 2012). However, the desired bandwidth of the force sensor is usually dictated by the application (FF, SS, vibration detection, etc.) and desired noise and resolution performance. A sample rate of 500 Hz is considered appropriate for direct FF applications (Jones et al., 2017). Sample rates as low as 30 Hz can be effective in visual SS modality.

3.2. Size, mass, and packaging

Surgical instruments are inserted into the body through a cylindrical port of 12–15 mm in diameter (Li et al., 2015b; Spiers et al., 2015). The outside diameter of the instrument is desired to be less than 10 mm (Shi et al., 2019). The sensor should be lightweight to not significantly increase the instrument inertia. The operation rooms are filled with equipment that can cause electromagnetic interference, and the electrocautery tools operate at high voltages (Lim et al., 2014; Seok et al., 2019). Thus, the sensors require insulation for electrostatic protection and shielding against electromagnetic interference (Peña et al., 2018). The sensors that enter the body also require sealing against humidity and debris ingression (Trejos et al., 2014).

3.3. Sterilizability

Surgical instruments are cleaned and sterilized for reuse; the former refers to removing debris from the device and the latter is the elimination of microorganisms that can cause disease (Trejos et al., 2014). The common sterilization methods are plasma and gamma radiation, the use of chemicals (alcohol, ethylene oxide, or formaldehyde), and steam sterilizations (Bandari et al., 2020). Steam sterilization is the fastest and the most preferred method (Soltani-Zarrin et al., 2018) which is performed in an autoclave at 120–135°C, 207 kPa, and 100% humidity for 15–30 minutes (Spiers et al., 2015; Zhao and Nelson, 2015). This harsh environment can be destructive to many transducers, signal conditioning electronics, wire insulations, bondings, and coatings.

3.4. Biocompatibility

The sensors for use in MIS must abide by ISO 10993 which entails a series of standards for evaluating the biocompatibility of medical devices (Trejos et al., 2014). For biocompatibility, electrical components often require coatings that interfere with sterilizability (Spiers et al., 2015).

3.5. Adaptability and cost

Instruments are disposed of after 10 to 15 uses due to accelerated cable fatigue (Anooshahpour et al., 2014; Kim et al., 2018a; Xue et al., 2018). The EndoWrist instruments retail at US$2,000–5,000 (Spiers et al., 2015). If the sensor is integrated into the instrument and is to be disposed of, it should not increase the instrument price significantly. An adaptable solution that can be easily used on different instruments is desirable.

4. Sensor location

The sensors can be placed in the instrument mounting interface, the instrument base, proximal (outside the body) and distal (inside the body) shafts, the actuation mechanism (cables/rod), the trocar mount and its distal end, the articulated wrist, and the gripper jaws (see Figure 4).

Fig. 4.

The options for sensor location.

Although the size, sterilizability, biocompatibility, and insulation requirements are more relaxed for the sensors placed outside the body, these locations are more prone to the factors causing sensor inaccuracy. The sensors at the instrument interface, base, and proximal shaft can have the electronics isolated from the patient (Van Den Dobbelsteen et al., 2012). The sensors in the instrument shaft can gain high precision in the lateral direction, but experiments (Lv et al., 2020; Maeda et al., 2016) showed that they do not provide high resolution in the axial direction unless the structure is modified to amplify axial strains. The sensors in the instrument shaft and trocar cannot independently measure the gripping force (He et al., 2014) and measuring cable tensions cannot provide information on the axial force. The sensors integrated into the trocar and instrument interface are usually adaptable (Wang et al., 2014).

Sensors placed at the gripper jaw provide the most accurate readings and have the most stringent design constraints. They are difficult to fabricate, package, mount (He et al., 2014; Kim et al., 2018b), and shield (Seok et al., 2019; Wang et al., 2014) and have limited adaptability which makes them cost-prohibitive for disposable instruments (Xue et al., 2019). In addition, the electronics are usually placed away from the transducer which deteriorates the signal-to-noise ratio (SNR) (Suzuki et al., 2018). Placing the force sensor at the grasper may also conflict with functional requirements for monopolar or bipolar cautery instruments (He et al., 2014).

Figure 5 summarizes the severity level of different sources that contribute to the sensing inaccuracy as a function of the sensor location (scale of 1 to 3; 1 is minimum, 3 is maximum, and × is no effect). It also compares how stringent the listed design requirements are for each sensor location (scale of 1 to 3; 1 is the least, 3 is the most, and × refers to not a requirement). The distribution of the records included in this survey as a function of the sensor locations and the sensing technologies are shown in the same figure. In interpreting this table, it is important to note that the compared design requirements and the listed sources of inaccuracy are location-dependent and the same across all sensing technologies. It is evident that the sensorless techniques have the majority of publications over the past 10 years. In addition, the microelectromechanical (MEM) and fiber Bragg grating (FBG) technologies have been widely adopted in the fabrication of miniature transducers that can be integrated into the gripper jaws. An overview of different transduction technologies is presented in the next section.

Fig. 5.

Stacked bar chart comparison of the number of records for each sensing technology at different locations along the instrument. The severity level of the sources that contribute to the measurement inaccuracy (scale of 1 to 3; 1 is minimum, 3 is maximum, and ✗ is no effect), and the importance of design requirements (scale of 1 to 3; 1 is the least, 3 is the most, and ✗ refers to not a requirement) are compared.

5. Sensing technologies

A collection of force sensing technologies at different locations along the instrument is shown in Figure 6. The corresponding number of articles for each sensing technology, covered in this survey, is shown in Figure 7.

Fig. 6.

A collection of sensing technologies at different locations along the instrument. (a) Wang et al. (2014): strain gauges are used in a modi_ed support for the cannula. (b) Fontanelli et al. (2017): optical transducers are placed at the distal end of the cannula. (c) Kim et al. (2018a): capacitive transducers are integrated into the grippers. (d) Shahzada et al. (2016): FBGs are mounted onto the distal shaft. (Reuse permission for all the figures were granted by Copyright Clearance Center’s RightsLink® service and SPIE digital library.)

Fig. 7.

The sensing technologies.

5.1. Sensorless

Sensorless refers to the case where the sensors used for force estimation are already inherent in the surgical robot (Stephens et al., 2019). In model-based approaches, the sensors are the encoders and the motor current measurements. In the vision-based techniques, the sensor is the visual feedback of the surgical site through mono or stereo cameras.

5.1.1. Model-based

Model-based techniques can be categorized into (1) analytical models developed based on first principles, (2) disturbance observers and Kalman filters that utilize a dynamic model and the control loop commands and feedback signals, and (3) data-driven models which consider the instrument as a black-box and fit a quantitative model to a customized set of inputs and outputs. The model-based literature is summarized in Table 1. They include 15 analytical models, 8 of which are physics-based and 7 studies use observers or Kalman filters. There are 6 articles on the use of data-driven models.

The accurate dynamic model of the surgical instruments is challenging to obtain due to the many sources of non-linearities, e.g., friction, backlash (Sang et al., 2017), tendons compliance (Anooshahpour et al., 2014) and creep (Haghighipanah et al., 2017), elastic deformations, actuators performance variations (the motors’ brush conductivity and change in the armature winding resistance) (Li et al., 2016), hysteresis (Anooshahpour et al., 2016), inertia, and gravity (Wang et al., 2014). In addition, any model relies on a set of measurements (calibration or training set) that are usually taken at the beginning and used throughout the estimation. It is experimentally shown that the tool behavior changes with time which deteriorates the estimation accuracy (Hadi Hosseinabadi et al., 2019; Kong et al., 2018). The environmental parameters such as temperature and humidity can also affect the instrument characteristics (Li and Hannaford, 2017). An alternative approach is the implementation of online adaptation and identification methods that are highly non-linear, complex, and computationally demanding. This limits their effectiveness in real-time applications (Anooshahpour et al., 2014; Li et al., 2016). Dynamic modeling is particularly difficult in instruments with coupled DoFs (O’Neill et al., 2018; Xin et al., 2017; Zhao and Nelson, 2015). Lee et al. (2015) showed that for the same input force by the surgeon, the grip force of the daVinci EndoWrist grasper can vary up to 3.4 times depending on its posture. As a result, despite the extensive research work, force estimations that rely on dynamic models do not provide highly reliable results yet, especially in the instrument’s lateral direction (Fontanelli et al., 2017). In comparison, the data-driven techniques based on supervised learning (Li and Hannaford, 2017; Stephens et al., 2019) provide more accurate force estimations.

5.1.2. Vision-based

The existing literature affirms that experienced surgeons use visual cues (tissue and instrument deformations and the stretch in the suture) as sensory feedback surrogates (Martell et al., 2011; Noohi et al., 2014). With the 3D stereoscopic view in robotic surgery providing depth information, and the developments in the available computational power (high-performance graphic processing units (GPUs), cluster computers, and cloud platforms), a noticeable shift towards adoption of vision-based techniques was observed. Although mechanical models of the tissue are presented, they are mostly complex and computationally expensive (Aviles et al., 2016). Most of the literature (Table 3) implement supervised learning architectures (recurrent neural network (RNN) and long short-term memory (LSTM) (Aviles et al., 2016; Marban et al., 2019)) with the video stream as inputs to estimate the instrument–tissue interaction forces. The vision-based techniques are robust to many sources of inaccuracy listed in Figure 5. However, they can be affected by the instrument occlusion, smoke and changes in the tissue properties, lighting conditions, and camera orientation. The estimation update rate cannot be faster than the video frame rate which is usually 30 Hz. This limitation makes the vision-based approached not suitable for FF applications in which the control loop is desired to execute faster than 500 Hz (Jones et al., 2017). The current literature highlights that force estimation through video processing is easier in pushing tasks (characterized by smooth deformations) than those produced by pulling tasks that are characterized by irregular tissue deformations due to grasping (Marban et al., 2019).

Table 3.

Vision-based force estimation.

Study	Method/task	Sensing DoFs	Stereo/mono	Results
1. Martell et al. (2011)	Image processing algorithms were utilized for suture strain estimation by identifying the suture line and tracking the displacement of markers. The achieved resolution in strain estimation was two orders of magnitudes smaller than the known strain to failure of most suture materials (20+%). / Suture pull	Force magnitude	Mono	Strain resolution of 0.2% and 0.5% was achieved in one-marker tracking on stationary suture and two-marker tracking on moving suture, respectively.
2. Kim et al. (2012)	The soft-tissue deformation was obtained by processing the stereoscopic depth image as a surface mesh. It was compared against the original organ shape from pre-operative images. A spring damper model was used for force estimation. / Tissue push and pull	Gripper tri-axialforces	Stereo	No results were presented
3. Noohi et al. (2014)	A virtual template, based on assuming soft tissue local deformation to be a smooth function, was used to estimate the tissue deformation without a priori knowledge of its original shape. The force magnitude was estimated by using a biomechanical model. / Tissue push	Gripper tri-axial forces	Mono	In force magnitude: ERR < 0.12 N RMSE = 0.07 N RNG: 0–2.5 N
4. Faragasso et al. (2014)	A force sensing device composed of a linear retractable mechanism and a spherical visual feature was installed on the endoscope. The force was estimated as a function of the size of the spherical feature in the image. / Palpation	Instrument axial force	Mono	RES: 0.08 N RMSE = 0.13 N RNG: 0–1.96 N
5. Aviles et al. (2014)	The method used a 3D lattice to model the deformation of soft tissue. An RNN estimated the force by processing the information provided by the 3D lattice and the surgical tool motion. / Tissue push	Tissue normal force	Stereo	MA:= 0.05 N RMSE: 0.062 N RNG: 0–3 N
6. Aviles et al. (2015a)	The RNN’s full feedback architecture in Aviles et al. (2014) was replaced by local and global feedback. The RMSE and computation time were improved. / Tissue push	Tissue normal force	Stereo	RMSE = 0.059 N RNG: 0–3 N
7. Aviles et al. (2015b)	The network in Aviles et al. (2015a) was upgraded to a RNN LSTM-based architecture which improves the force estimation accuracy. / Tissue push	Tissue normal force	Stereo	RMSE = 0.029 N RNG: 0–3 N
8. Otte et al. (2016)	The tissue deformations from optical coherence tomography (OCT) and the instrument trajectories were used as inputs to a generalized regression neural network (GRNN) to estimate the instrument–tissue forces. / Tissue push	Force magnitude	OCT Scanner	RMSE = 3 mN RNG: 0–20 mN
9. Aviles et al. (2016)	Evaluated the effect of dimensionality reduction on the performance of the RNN+LSTM architecture proposed in Aviles et al. (2015b). It showed that implementation of a probabilistic principal component analysis (PPCA) significantly reduced dimension (75% reduction) and improved accuracy. / Tissue push	Tissue normal force	Stereo	ERR < 2% RMSE = 0.02 N RNG: 0–3 N
10. Giannarou et al. (2016)	The tissue deformations were estimated by finding stereo-correspondences based on tissues salient features and the use of probabilistic soft tissue tracking and thin-plate splines (TPSs). The displacements were used to estimate forces based on a biomechanical model. / Tissue push	Force magnitude	Stereo	MAE = 0.07 N RNG: 0–0.8 N
11. Aviles et al. (2017)	This was an extension to Aviles et al. (2016) where the proposed RNN+LSTM architecture was extended to three-axis force components. $Z$ -axis was normal to the tissue, $X$ - and $Y$ -axes were planar with the tissue surface. / Tissue push	Tissue tri-axial forces	Stereo	RMSE: All DoFs < 0.02 N RNG: FX: ±0.6 N, FY: ±2 N, FZ: ±6 N
12. Hwang and Lim (2017)	Same as Aviles et al. (2015b) with a deeper network, fully connected layers, and a sequence of mono 2D images as inputs. The results were on a sponge, a PET bottle, and a human arm with changes of light and pose. / Push	Tissue (object) normal force	Mono	RMSE: Sponge:0.05 N, PET bottle: 0.17 N, Arm: 0.1 N RNG: Sponge: 0–3 N, PET bottle: 0–7 N, Arm: 0–2 N
13. Haouchine et al. (2018)	A biomechanical map of the organ shape was built on-the-fly from stereoscopic images. It used 3D reconstruction and meshing techniques. / Tissue push and pull	Force magnitude	Stereo	Plot comparison, not quantified.
14. Gessert et al. (2018)	Took an undeformed reference volume and a deformed sample volume from OCT as inputs into a Siamese 3D-CNN architecture and output a 3D force vector. The results were compared with CNNs that take the difference or the sum of the undeformed and deformed volumes, and a CNN that takes 2D projected surface images as inputs. / Tissue push	Tissue tri-axial forces	OCT Scanner	In force magnitude: MAE: 7.7 mN RMSE: 4.3 mN RNG: 0–1 N
15. Marban et al. (2018)	A semi-supervised learning model consisting of an encoder+LSTM network was suggested. The encoder learned a compact representation of the RGB frames from video sequences. The LSTM network used the tool trajectory information and the output of the encoder for force estimation. / Tissue push	Instrument tri-axial forces and moments	Mono	RMSE: FZ: 0.89 N RNG: FZ: 0–8 N
16. Marban et al. (2019)	It used a CNN+LSTM architecture that processed the spatiotemporal information in video sequences and the temporal structure of tool data (the surgical tool-tip trajectory and its grasping status). / Tissue push and pull	Instrument tri-axial forces and moments	Mono	Plot comparison, not quantified. Concluded that both video sequences and tool data provide important cues for the force estimation.

Force estimation based on using optical coherence tomography (OCT) as the reference sensor was proposed by Otte et al. (2016) and Gessert et al. (2018). OCT images provide volumetric data with a resolution of a few micrometers in which the tissue compression and subsurface deformations can be reflected. Thus, they contain a richer signal space compared with the mono and stereo visions that provide only the surface information.

5.2. Strain gauge

Strain gauges are the most commonly used transducers for force sensing (Yu et al., 2018a). They are accurate and small and can be designed in different configurations for multi-axis force sensing. Although the transducers are low cost with a price of US$10–25 per unit (Srivastava et al., 2016), they require special surface preparation, adhesives, and coatings for optimal performance that increases the assembly and integration cost (Li et al., 2017b). When used in Wheatstone bridge arrangements, they require multiple wires for connection that makes packaging difficult for quick and seamless integration with surgical instruments (Lv et al., 2020; Shi et al., 2019). Strain gauges are highly influenced by electromagnetic noise and are not suitable for use close to other tools with strong magnetic fields (e.g., electrocautery) (Choi et al., 2017; Xue et al., 2018). They have low sensitivity and often require custom flexures or modifications in the load-carrying structure to amplify local strains (Hadi Hosseinabadi et al., 2019). Strain gauges are fragile and require mechanical overload protection (Kuang et al., 2020). They typically do not survive multiple sterilization cycles (Shi et al., 2019) and lose repeatability. Trejos et al. (2014, 2017) conducted an extensive study on biocompatible adhesives and coatings that can withstand the harsh environment during steam sterilization. However, none of the combinations showed reliable measurements after seven cycles. Table 4 summarizes the articles which utilize strain gauges or commercial strain-gauge-based force sensors for MIS force sensing.

Table 4.

Strain-gauge force sensing.

Study	Method/location	Sensing DoFs	Instrument/DI	Results
1. Jones et al. (2011)	Custom torque sensors were placed at the instrument interface between the driver and the driven knobs. / Instrument interface	Grip force, Pitch torque, Instrument axial torque	EndoWrist instruments / 6	No results presented
2. Van Den Dobbelsteen et al. (2012)	A tension/compression load cell was installed in line with the actuating rod of the grasper. / Actuating rod	Grip force	Karl–Storz laparoscopic grasper / 1	ERR < 0.09 N RNG: 0–2 N
3. Hong and Jo (2012)	Custom grasper jaw with flexure hinges was designed to make a compliant structure. / Gripper	Gripper normal (FN) and pull (FP) forces	Standalone testing / 1	RES: FP: 43 mN, FN: 7.4 mN RMSE: FP: 95 mN, FN: 37 mN RNG: ±5 N
4. Baki et al. (2012)	Strain gauges were installed onto a custom-designed flexure out of titanium fabricated by EDM. / Distal shaft	Instrument tri-axial forces	Standalone testing / 0	ERR < 4% RES: 5 mN RNG: ±2 N
5. He et al. (2014)	Custom-designed sensors for measuring cable tension were installed at the instrument base. / Instrument base (cable tension)	Gripper normal and 3-DoF forces	MicroHand robot instrument / 6	ERR < 0.4 N RNG: Fx,Fy: ±3.5 N, FZ: ±2 N, FG: 0–11 N (CS at an external sensor)
6. Moradi Dalvand et al. (2014)	Strain gauges were installed on the lead-screw actuation mechanism and a sleeve. / Actuating rod & Distal end of a sleeve	Instrument lateral (FL) and grip force (Fg)	CD RMIS instrument for 5 mm fenestrated inserts / 2	MAE: FL< 0.05 N, Dir. < 3° RMSE: FL: < 0.05 N, Dir. < 5.7° RNG: FL: ±1 N, FG: 0–5 N
7. Wang et al. (2014)	An instrumented cover plate at the instrument interface and sensorized docking clamps at the trocar mount measured the $Z$ -axis and lateral forces, respectively. / Instrument interface & trocar mount	Instrument tri-axial forces	EndoWrist instruments / 6	RMSE < 8% RNG: FX,Fy: ±8 N, FZ: ±12 N
8. Talasaz et al. (2014)	Strain gauges were installed on the actuating cables and the RMIS instrument was attached to the robot flange through a six-axis ATI Gamma F/T sensor. / Actuating cables and instrument interface	Instrument tri-axial forces, axial and pinch torques, grip force	EndoWrist Needle Driver / 6	ERR: FX,FY,FZ< 0.12 N
9. Yu et al. (2014)	Small-size six-dimensional force/torque sensor with the structure of double cross beams. / Articulated wrist	Wrist 6-DoF forces and moments	CD RMIS instrument / 5	ERR < 4.5% RNG: FX,FY,FZ: 10 N, MX,MY: 150 Nmm, MZ: 50 Nmm
10. Trejos et al. (2014)	Strain gauges were installed on the rod that actuated the grasper and on the distal end of the instrument shaft. / Actuation rod and distal shaft	Instrument lateral and grasping forces	Manual laparoscopic grasper / 1	ERR: FX,FY,FG< 0.2 N RNG: FX,FY: ±5 N, FG: 0–17 N
11. Spiers et al. (2015)	Custom torque sensors were placed at the instrument interface between the driver and the driven knobs. / Instrument interface	Grip force, Pitch torque, Instrument axial torque,	EndoWrist Needle Driver / 6	RNG: ±6 N
12. Li et al. (2015b)	Strain gauges were installed on a custom-designed tripod flexure. / Distal shaft	Instrument tri-axial	Standalone testing / 0	ERR: FX, FY< 1%, FZ< 5% RNG: FX, FY: ±1.5 N, FZ: ±3 N
13. Li et al. (2015a)	Strain gauges were integrated into a custom-designed flexural-hinged Stewart platform. / -	6 DoF forces and torques	Standalone testing / 0	RES: FX, FY: 0.08 N, FZ: 0.25 N MX,MY,MZ: 2.4 Nmm RNG: FX, FY, FZ: ±30 N, MX,MY,MZ: ±300 Nmm
14. Ranzani et al. (2015)	Two custom holders with integrated ATI F/T sensors were designed for the instrument and the fulcrum point. / Instrument interface and fulcrum point	Instrument tri-axial forces	MIS laparoscopic grasper / 1	ERR < 2.7% RNG: ±4 N
15. Maeda et al. (2016)	An ATI Mini40 force sensor was mounted to the shaft of the instrument. / Proximal shaft	Instrument lateral and axial forces, axial torque	CD RMIS laparoscopic forceps /6	Sensor performance not quantified.
16. Khadem et al. (2016)	Integrated a tension/compression load cell inline with the lead-screw actuation and a six-axis ATI Mini45 at the instrument base. / Actuating rod and instrument interface	Gripper pull force, grip force	CD RMIS laparoscopic grasper	ERR: FG< 0.5 N RNG: FG: 0–5 N
17. Wee et al. (2016, 2017)	Presented a force-sensing sleeve with 4 strain gauges adaptable to standard MIS instruments. / Distal shaft	Instrument tri-axial forces, axial torque	MIS Laparoscopic Grasper/1	RES: 0.2 N RMSE: FX, FY< 0.088 N RNG: FX, FY: ±5 N
18. Barrie et al. (2016)	A tension/compression load cell was installed in-line with the actuating rod of the grasper. / Actuating rod	Grip force	Johan fenestrated grasper / 1	Sensor performance not presented
19. Seneci et al. (2017)	Proposed a disposable sensor clip for the gripper. The gripper was fabricated by selective laser melting (SLM) and the sensor clip was 3D printed. / Gripper	Gripper normal force	Standalone testing / —	ERR < 0.2 N RNG: ±5 N
20. Trejos et al. (2017)	Strain gauges were installed onto the proximal and the distal shafts. / Distal and proximal shafts	Instrument tri-axial forces	MIS laparoscopic grasper / 1	ERR: FX, FY< 0.2 N, FZ< 1.7 N RNG: FX, FY: ±5 N, FZ: ±12 N
21. Li et al. (2017a)	Extension on Li et al. (2015a) in which the sensor was integrated into the surgical instrument. / Articulated wrist	Wrist 6-DoF forces and moments	CD RMIS instrument / 6	RES: FX, FY: 0.12 N, FZ: 0.5 N, MX,MY,MZ: 7 Nmm RNG: FX, FY, FZ: ±10 N, MX, MY, MZ: ±160 Nmm
22. Kim et al. (2017)	A three-axis I-Beam force sensor using strain gauges were designed to replace the trocar support. / Trocar mount	Instrument lateral, trocar axial friction	EndoWrist instruments / 6	RMSE: FX< 0.39 N, FY< 0.20 N, FZ< 0.35 N RNG: FX, FY: ±15 N, FZ: ±10 N
23. Schwalb et al. (2017)	It is similar to the overcoat method by (Shimachi et al., 2003). The instrument was mounted to an inner tube that was attached to a six-axis F/T sensor. / Instrument interface	Instrument tri-axial forces	CD RMIS instrument / 6	RES: 0.09 N RNG: ±9 N
24. Yu et al. (2018a)	Axial load cells measured the cables tensions and a NN was used for friction compensation. / Actuating cable	Gripper normal (FN) and shear (FS) forces	CD RMIS instrument / 5	ERR: FN< 10%, FS< 8% RNG: FN: 0–2 N, FS: ±2.5 N
25. Kong et al. (2018)	Characterized the grip force over 50,000 grasps of one instrument using torque sensors at the instrument interface. Trained different NNs with an error threshold of 2 Nmm. The NN inputs were the proximal position, velocity, and torque measurements. / Instrument interface	Grip torque	EndoWrist Maryland grasper / 6	ERR < 2 Nmm
26. Karthikeyan and Nithya (2018)	A custom flexure was designed and populated with strain gauges. / Articulated wrist	Wrist tri-axial forces	CD RMIS instrument / 5	RNG: 0–1.5 N
27. Novoseltseva (2018)	The axial force was measured by a thin plate between the proximal shaft and the sterile adapter. The lateral forces were measured by a flexure at the trocar. / Proximal shaft and trocar distal end	Instrument tri-axial forces	EndoWrist Needle Driver / 6	ERR: FX< 0.4 N, FY< 0.65 N, FZ< 0.63 N RES: FX: 0.03 N, FY: 0.02 N, FZ: 0.2 N RNG: FX, FY: ±19 N, FZ: ±12 N
28. Peña et al. (2018)	Vapor-deposition fabrication techniques were used to directly print strain gauges on the instrument shaft. The material cost was US$ 0.09 per transducer. / Distal shaft	Instrument lateral forces	EndoWrist Needle Driver & Fenestrated grasper / 6	ERR < 0.8 N RNG: ±5 N
29. Takizawa et al. (2018)	A disposable pneumatic cylinder with a strain gauge on its inner wall actuated the grasper. The transducer and the pneumatic pressure were used for force estimation. / Actuation system	Grip force	CD MIS laparoscopic grasper	RNG: 0.1–0.25 N
30. Yu et al. (2018b)	A custom gripper with double E-type beams flexure and populated with strain gauges was designed. / Gripper	Gripper normal (FN), shear (FS), pull (FP)	Standalone testing / 1	RMSE: FN: 23 mN, FS: 2.2 mN, FP: 93 mN RES: 0.01 N, RNG: ±2.5 N
31. Wang et al. (2019a)	Combined the cable-drive dynamics, the cable tension measurement, and a particle swarm optimization back propagation neural network (PSO-BPNN) to develop a joint torque disturbance observer. / Cable tension	Grip force	CD RMIS grasper / 5	ERR < 0.25 N RNG: 0–2 N
32. Xue et al. (2019)	Four micro force sensors were used for cables tension measurement. The cable tension and a model of the cable drive system (with coupling and friction effects) were used to estimate the grasping forces. / Cable tension	Grip force	EndoWrist needle driver / 6	ERR < 0.4 N RNG: 0–12 N After stability is reached (hysteresis effect)

5.3. Optical

Optical methods use light intensity (e.g., photodiodes, phototransistors), frequency (e.g., FBGs), or phase (e.g., interferometry) modulation for force measurement. The optical signal can be locally converted into electric signals, or be transferred with fibers for distal processing. Placing the electronics away from the instrument tip makes sterilization easier. The optical fibers are flexible, scalable, biocompatible, electrically passive, insensitive to electromagnetic noise and, thus, MRI compatible (Peña et al., 2018), durable against high radiation (Lim et al., 2014), immune to water (Song et al., 2014), corrosion-resistant (Shi et al., 2019), and low cost (Bandari et al., 2020). However, optical fibers cannot be routed into small bending radii (Trejos et al., 2014). In addition, the presence of small and intricate parts can make fabrication and assembly of fiber-based sensors costly (Li et al., 2017b).

The light intensity modulation (LIM)-based sensors are vulnerable to light intensity variations due to the temperature or fiber bending (Lv et al., 2020). This can be improved by normalizing the optical signal against the emitted power (Hadi Hosseinabadi et al., 2019). Alternatively, a redundant strain-free fiber can be used to compensate for the effect of temperature or other sources of uncertainty (Puangmali et al., 2012; Song et al., 2014). Table 5 summarizes the articles that address MIS force estimation based on LIM.

Table 5.

Optical force sensing: LIM.

Study	Method/location	Sensing DoFs	Instrument/DI	Results
1. Puangmaliet al. (2012)	Presented a three-axis force sensor with a flexible tripod structure, a stationary reflecting surface, and a pair of transmitting and receiving fibers per axis. The light source and photodetectors are remote or at the instrument base. / Distal shaft	Instrument tri-axial forces	Standalonetesting / 0	ERR: < 5% FS RES: 0.02 N RNG: FX, FY: ±1.5 N, FZ: ±3 N
2. Ehrampooshet al. (2013)	Proposed an optical sensor design comprised of three gradient-index (GRIN) lenses transmitting–receiving fiber-optic collimators, a flexible structure, and a reflective plate. / Distal shaft	Instrument tri-axial forces	Standalonetesting / 0	RNG: ±6 N.
3. Fontanelliet al. (2017)	Four optical proximity sensors measure the deflection of the instrument shaft with respect to the fixed trocar. / Trocar distal end	Instrument lateral forces	Adaptable to anyEndoWristInstrument / 6	ERR<12%RNG:±4 N
4. Hadi Hosseinabadiet al. (2019)	Optical force sensor comprising of an IR LED, a bicell photodiode, and a slit installed on the proximal shaft of the instrument. The proposed concept provided sub-nanometer resolution in deflection measurement. / Proximal shaft	Instrument lateral forces	EndoWristinstrument / 6	RMSE = 0.03 N RNG: ±1 N
5. Bandari et al. (2020)	A moving cylinder bends a fiber sitting on two fixed cylinders. Rate-dependent learning-based support vector regression was used for calibration. / Gripper	Grip force	CD MISlaparoscopicgrasper	ERR < 0.2 N RES: 0.002 N RNG: 0–2 N

The FBG sensors are wavelength-coded and insensitive to the changes in the light intensity. FBGs are very sensitive, have calibration consistency, and exhibit high SNR which provide repeatable and high-resolution strain measurements (Shahzada et al., 2016). Multiple gratings can be accommodated into one fiber (Soltani-Zarrin et al., 2018) simplifying the design and signals routing. Thus, they are also used in shape sensing (Lv et al., 2020). Nonetheless, FBGs require interrogators for signal processing which the commercial systems cost between US$10,000 and US$100,000 (Yurkewich et al., 2014). The articles which used FBGs for MIS force estimation are summarized in Table 6.

Table 6.

Optical force sensing: FBG.

Study	Method/location	Sensing DoFs	Instrument/DI	Results
1. Haslinger et al. (2013)	Similar to the DLR’s miniature six-axis force/torque sensor (Seibold et al., 2005) with the strain gauges replaced by FBGs. The sensor structure was a Stewart platform to provide enhanced stiffness. / Articulated wrist	Wrist 6-DoF forcesand moments	DLR MICA instruments / 6	ERR < 18.6% RNG: FX,FY, FZ: ±6.9 N, MX, MX: ±59.34 Nmm, MZ: ±49.53 Nmm
2. Lim et al. (2014)	Two optical FBGs were integrated into the forceps. Each fiber had two gratings for measuring the mechanical strain (on the surface) and for temperature compensation (at the center of the bending neutral axis). / Gripper	Gripper normalforce (FN)	CD MIS laparoscopicgrasper / 1	RES: 1 mN RNG: 0–5 N
3. Song et al. (2014)	Three-axis force sensor with four longitudinal bendable beams populated with FBGs. Four other FBGs were integrated as references for temperature compensation. / Articulated wrist	Wrist tri-axial forces	CD RMIS instrument / 6	ERR: FX,FY< 0.1 N, FZ< 0.5 N RNG: ±10 N
4. Yurkewich et al. (2014)	Integrated 3 FBGs on the distal shaft and another FBG into the moving jaw of the grasper. / Distal shaft and gripper	Instrument lateral force(FL), grip force (FG)	MIS arthroscopic grasper / 1	RMSE: FL= 0.213 N, Dir = 4.37, FG = 0.747 N RNG: FL: ±10 N, FG: 0–20 N
5. Shahzada et al. (2016)	Four FBG sensors were attached to the instrument distal shaft in a two cross-section layout which is insensitive to the error caused by combined force and torque loads. / Distal shaft	Instrument lateral forces	EndoWrist Needle Driver / 6	ERR <±0.05 N (95% confidence interval) RES: 0.05 N RNG: ±2 N
6. Choi et al. (2017)	Custom flexure with three FBGs and an overload protection mechanism. The calibration algorithm was based on a two-layer NN. / Articulated wrist	Wrist tri-axial forces	Standalone testing / —	ERR < 0.06 N RNG: ±12 N
7. Suzuki et al. (2018)	Four FBGs were integrated into the articulated wrist. The differential wavelength shift was used to achieve robustness to temperature and gripping force. / Articulated wrist	Wrist bending forces and moments	CD RMIS instrument / 6	RNG: FX, FY: ±0.5 N, TX,TY: ±50 Nmm
8. Soltani-Zarrin et al. (2018)	Two grasper designs with sliding stretchable T-shaped parts for enhanced axial strain. Axial FBGs were at the grasper’s bending neutral axes and its surface. / Gripper	Gripper normal(FN) and pull (FP) forces	Standalone testing / —	ERR: 1: FN< 0.57 N, FP< 0.78 N 2: FN< 0.81 N, FP< 0.9 N RNG: FN: 0–10 N, FP: 0–6 N
9. Xue et al. (2018)	The cable tensions were measured by FBGs pasted in the grooves on inclined cantilevers integrated into the Instrument base. / Instrument base (cable tension)	Grip force	CD MIS laparoscopicinstrument with localactuation / 5	ERR < 0.5 N RES: 0.14 N RNG: 0–15 N
10. Shi et al. (2019)	A force sensing flexure combining a Stewart base and a cantilever beam. The FBG was integrated along the central line of the flexure with its two ends fixed in grooves. / Distal shaft	Instrumentaxial force	Standalone testing / —	No radial constraint: MAE: FZ< 0.26 N, RES: 21 mN, RNG: FZ: 0–12 N With radial constraint: MAE: FZ< 0.12 N, RES: 9.3 mN, RNG: FZ: 0–7 N
11. Lv et al. (2020)	The force sensor had a miniature flexure based on a Sarrus mechanism to achieve high axial sensitivity and a large measurement range. An FBG was tightly suspended along the central axis of the flexure. / Distal shaft	Instrumentaxial force	Standalone testing / —	ERR < 0.06 N RES: 2.55 mN RNG: 0–-5 N

5.4. Capacitive

Capacitive methods are attractive solutions for high-resolution and compact force sensor designs. Compared with strain gauges, they provide limited hysteresis in microscale and increased sensitivity (Sang et al., 2017). However, they have a limited range (Li et al., 2017b) and are prone to thermal and humidity drift (Bandari et al., 2020). The change in capacitance can be due to the change of the overlapping area or the distance between the two electrodes; the latter provides higher sensitivity and a more linear response (Kim et al., 2017). The commercially available capacitance to digital converter (CDC) chips such as the AD7147 from Analog Devices significantly simplify the signal processing, which was believed to be challenging for capacitive transducers (Trejos et al., 2014). However, they provide a low sampling rate. Table 7 lists the articles that are based on the capacitive transduction principle. The sterilizability and biocompatibility of the existing literature are not evaluated.

Table 7

Capacitive force sensing

Study	Method/location	Sensing DoFs	Instrument/DI	Results
1. Lee et al. (2014)	Proposed a tendon drive pulley at the instrument base with an integrated torque sensor. A three-axis force sensor was placed into the instrument shaft. Both sensors were based on the changes in the distance between the electrode and the ground. / Distal shaft and instrument base	Instrument tri-axial forces, grip force	Custom Prototype of an MIS grasper / 1	RNG: 0–0.5 N
2. Kim et al. (2015)	Two sensors consisting of a triangular prism shape and two capacitive-type transducers with an elastomeric polymer dielectric were integrated into the grasper. Molding was used to fabricate a prototype. / Gripper (tip)	Gripper normal (FN), shear (FS), pull (FP), grip (FG) forces	CD RMIS instrument for RAVEN-II / 6	RES: FP: 42 mN, FS: 72 mN, FN: 58 mN, FG: 46 mN RMSE: FP: < 84 mN, FS: < 0.114 N, FN: < 73 mN, FG: < 95 mN RNG: FP: ±2.5 N, FS: ±2.5 N, FN: ±5 N, FG: 0–5 N
3. Kim et al. (2016)	Two sensors with three electrodes and common grounds were integrated into the gripper. The dielectric was air and the signal processing electronics were local. / Gripper (base)	Gripper normal (FN), shear (FS), pull (FP), grip (FG) forces	Custom prototype of an MIS Grasper / 1	ERR: FP: < 0.42 N, FS: < 0.15 N, FN: < 0.92 N RNG: 0–8 N
4. Lee et al. (2016)	An extension on Lee et al. (2014) with the three-axis force sensor moved into the articulated wrist and two capacitive torque sensors in the tendon drive pulleys of the gripper jaws. / Articulated wrist and instrument base	Wrist tri-axial forces, grip Force	CD RMIS instrument for RAVEN-II / 6	NRMSE: FX: 0.039, FY: 0.056, FZ: 0.026 RNG: FX: ±1 N, FY: ±1 N, FZ: ±1.6 N
5. Kim et al. (2018a)	Proposed a capacitance sensing PCB with eight electrodes and a CDC chip and a conductive deformable structure as the common ground. A spherical cap was added to the sensor for testing it in a palpation task. / Articulated wrist	6-DoF forces and moments	CD RMIS instrument for S-surge robot / 6	MAE: FX, FY, FZ: 5.5% FSO, MX, MY, MZ: 2.7% FSO RES: FX: 0.22 mN, FY: 0.31 mN, FZ: 0.11 mN, MX: 0.47 mNmm, MY: 0.41 mNmm, MZ: 0.17 mNmm RNG: FX, FY, FZ: 1 N, MX, MY: ±20 Nmm, MZ: ±10 Nmm
6. Kim et al. (2018b)	Extension of Kim et al. (2016)with a slight modification in the proximal gripper jaw and calibration scheme so that the combination of the capacitive transducers also resolved the axial torque about the gripper. / Gripper (base)	Gripper normal (FN), shear (FS), pull (FP), grip (FG) forces, axial torque (TP)	CD RMIS instrument for S-surge robot / 6	Integrated sensor: RES: FN: 1.8 mN, FS: 2.0 mN, FP: 3.8 mN MAE: FN: 6.4%, FS: 3.4%, FP: 8.6%, TP: 5.7% RNG: FP, FN, FS: ±5 N, TP: ±3 Nmm
7. Seok et al. (2019)	Extension of Kim et al. (2018b) with humidity compensation. An AC shield was added to minimize the temperature effects on parasitic capacitance. An anodizing process was applied for electric insulation. The sensors range was extended to 20 N. / Gripper (base)	Gripper normal (FN), shear (FS), pull (FP), grip (FG) forces, axial torque (TP)	CD RMIS instrument for S-surge robot / 6	RNG: ±20 N

5.5. MEM sensors

MEM sensors (Table 8) operate based on the same physical principles discussed so far. However, MEM sensors fabrication techniques such as deposition, etching, and lithography allow for the cost-effective production of small, fully integrated, monolithic sensors (Radó et al., 2018) with reduced lead time in prototypes and high-throughput batch volumes (Gafford et al., 2014). Typically, MEM sensors do not require manual assembly, bonding, and alignment, and provide functional devices after the fabrication process (Gafford et al., 2013). By utilizing MEM sensor technology, it is possible to develop smart parts (e.g., grippers) with integrated sensing capability for micromanipulation (Pandya et al., 2014). Biocompatible coatings can be added to MEM sensors for biomedical applications.

Table 8.

MEM sensor force sensing.

Study	Method/location	Sensing DoFs	Instrument/DI	Results
1. Lee et al. (2013)	A thin-film capacitive sensor was fabricated using MEM sensor silk-screening technique on a PET film. / Gripper	Gripper normal (FN), shear (FS), pull (FP) forces	Standalone testing / —	SENS: FN: 6.1%, FS: 10.3%, FP: 10.1% RNG: 0–12 N
2. Gafford et al. (2013)	The pop-up book MEM sensor method was used to fabricate a grasper with a custom thin-foil strain gauge in a single manufacturing step. / Gripper	Gripper normal force	Standalone testing / —	RES: 30 mN RNG: 0–1.5 N
3. Kuwana et al. (2013)	A piezoresistive sensor chip was manufactured by burying a substrate of several bent beams in different directions in resin. / Gripper	Gripper normal (FN), shear (FS), pull (FP), and grip (FG) forces	MIS laparoscopic grasper (Covidien; ENDOLUNG) / 1	No results presented
4. Gafford et al. (2014)	Used printed-circuit MEM system (PCMEMS) technique to develop a monolithic, fully integrated tri-axial sensor with printed strain gauges. / —	Instrument tri-axial forces	Standalone testing / —	RES < 2 mN RNG: FX, FY: ±500 mN, FZ: ±2.5 N
5. Nakai et al. (2017)	A six-axis force/torque sensor chip composed of 16 piezoresistive beams was fabricated by using ion beam etching and surface doping. The sensor is $2\times 2$ mm2 and installed onto the grasper. / Gripper	Gripper 6-DoF forces and moments	MIS laparoscopic grasper / 1	RNG: FN: 0–40 N, FS: ±12.5 N, FP: ±12.5 N MN: ±15 Nmm, MS: ±100 Nmm, MP: ±100 Nmm
6. Dai et al. (2017)	Proposed a three-axis capacitive force sensor with differential electrodes. The compressive load reduced the dielectric thickness, and shear forces changed the overlap area. The sensor was fabricated using MEM sensor lithography. / Gripper	Gripper normal (FN), shear (FS), pull (FP) forces	EndoWrist ProGrasp / 6	RES: FN: 55 mN, FS: 1.45 N, FP: 0.25 N RNG: FN: 0–7 N, FS: ±11 N, FP: ±2 N
7. Radó et al. (2018)	Used deep reactive ion etching (DRIE) to fabricate a monolithic silicon-based three-axis force piezoresistive sensor. The sensor was covered with a semi-sphere PDMS polymer. / Gripper	Gripper normal (FN), shear (FS), and pull (FP) forces, palpation	MIS laparoscopic grasper for Robin Heart robot / 1	ERR < 10% RNG: 0–4 N
8. Tahir et al. (2018)	Presented a piezoelectric sensor fabricated using reduced graphene oxide (rGO)-filled PDMS elastomer composite to measure the dynamic force. / Gripper	Gripper normal	MIS laparoscopic grasper / 1	RNG: 0.5–20 N

5.6. Other technologies

A summary of other sensing technologies is given in Table 9. Piezoelectric transducers do not require an external power supply and have high stiffness (Li et al., 2017b). However, they are subject to charge leakage and are not suitable for low-frequency and static loads (Sang et al., 2017). They are also sensitive to temperature. Piezoresistive transducers used in force-sensitive resistors are scalable with low hysteresis and noise (Bandari et al., 2017). Nonetheless, their linear response is limited to a small range and they drift under constant load (Juo et al., 2020). They do not have the challenges associated with the integration of strain gauges, are relatively insensitive to humidity, and can be used in high temperatures above 170°C (Li et al., 2017b). Shape memory alloys (SMAs) such as Nitinol have a higher gauge factor compared with common metallic strain gauges and provide a larger range owing to their stretchability. SMAs require an insulating coating for use on conductive surfaces. They can be clamped at two ends and do not require a backing material with special surface preparation. They are low cost and available at diameters as small as few micrometers (Srivastava et al., 2016). Quantum tunneling composite (QTC) pills are flexible polymers that act as insulators in resting state but increase conductivity when compressed. They are very sensitive, provide a wide dynamic range, and are low cost (less than US$1 per pill). However, they are temperature sensitive and inaccurate in dynamic loading applications (Novoseltseva, 2018). Recently, vibration frequency and phase shifting due to an applied force have been measured for force estimation by the use of accelerometers. This approach is slow as it needs a few vibration cycles to generate stable and repeatable signals (Kuang et al., 2020).

Table 9.

Other force sensing technologies.

Study	Method/technology/location	Sensing DoFs	Instrument/DI	Results
1. Vakili et al. (2011)	A Tekscan FlexiForce piezoresistive pressure sensor was integrated into one of the grasper jaws. / Piezoresistive / Gripper	Gripper normal force	CD MIS laparoscopic grasper / 1	RNG: 0–4.4 N
2. Mack et al. (2012)	QTC Pills were integrated into a custom-designed support structure. / QTC / Instrument base	Instrument tri-axial forces, grip force, axial torque	CD RMIS instrument / 6	No results presented
3. McKinley et al. (2015)	Palpation probe that could be added onto the instruments. It measured the axial compression of the sliding tip using a Hall effect sensor. / Magnetic Hall effect sensor / Distal shaft	Instrument axial force	EndoWrist instruments / 6	RES: 4mN RNG: 0–1.6 N
4. Srivastava et al. (2016)	Superelastic Nitinol wires were used, instead of strain gauges, in two cross-sections arrangements for strain measurement. / SMA / Distal shaft	Instrument lateral forces	EndoWrist Needle Driver / 6	RES: 55 mN RMSE < 32 mN RNG: ±4 N
5. Li et al. (2017b)	Proposed a compact three-axis force sensor design with integrated signal conditioning, power regulation, and ADC. The sensor used an array of force-sensitive resistors (FSRs) with a mechanically pre-loaded structure. / FSR / Distal shaft	Instrument tri-axial forces	Standalone testing / –	RES: 0.1 N RNG: ±8 N
6. Jones et al. (2017)	A 3D-printed grasper face with an embedded neodymium permanent magnet was attached to a soft silicone base that was mounted on top of a three-axis Hall effect sensor. Genetic programming algorithm was used for sensor calibration. / Magnetic Hall effect sensor / Gripper	Gripper normal (FN), shear (FS), and pull (FP) forces	Standalone testing / —	Hysteresis ERR: FN< 1.58 N, FS, FP< 0.31 N RNG: FN: 0–35 N, FS, FP: ±7 N
7. Bandari et al. (2017)	Proposed a hybrid sensor that used a piezoresistive transducer to measure normal force and LIM in optical fibers to estimate the tissue deformation. The sensor was made out of silicon for biocompatibility. / Piezoresistive+LIM / Gripper	Gripper normal force	Standalone testing / 1	RNG: 0- 2.5 N
8. Gaudeni et al. (2018)	Proposed the placement of a pneumatic balloon in a cavity on the surgical instrument or endoscopic camera. When needed, the membrane is inflated to contact the tissue. The pneumatic pressure and volume are monitored to estimate the force. / Pneumatic / Distal shaft	Palpation force	Standalone testing / —	ERR < 0.24 N RMSE: 0.11 N RNG: 0–1.7 N
9. Abdi et al. (2020)	Tekscan FlexiForce and A101 piezoresistive sensors were installed onto the forceps via a custom 3D-printed mounting component. / Piezoresistive / Gripper	Gripper normal (FN), shear (FS), pull (FP) forces	EndoWristCadiere forceps / 6	RNG: FN: 0–15 N, FS: ±44 N, FP: 0–44 N
10. Kuang et al. (2020)	A slender shaft was excited by using a vibration motor. The structure’s tri-axial acceleration signals in time-domain showed discernible ellipse-shaped profiles when a force was applied. The acceleration profiles were characterized via regression to estimate the direction and magnitude of the applied force. / Vibration monitor / Proximal and distal shaft	Instrument tri-axial forces	Standalone testing / —	MAE: FL: 18%, FZ: 6% RES: FL: 0.098 N, Dir = 10, RNG: FL: 0–0.98 N, FZ: 0–0.95 N

6. Discussion and conclusion

In keyhole endoscopy, the surgeon’s interaction with the surgical site is via slender instruments that are inserted into the body through small incisions. Despite the many benefits to the patient, the operation is more challenging for the surgeon owing to the instruments’ limited dexterity, fulcrum motion reversal, uncomfortable posture, and limited visual presentation. In addition, the surgeon’s force perception is affected by the forces between the instrument and the skin and the instrument’s dynamics. The adoption of robotic and computer vision technologies resolves the limitations above and significantly improves the accuracy and efficiency in RMIS. However, most telesurgical systems completely isolate the surgeon from the tissue through the local/remote architecture of robotic telemanipulation. This deprives the surgeon of the rich information in palpation and direct interaction with the tissue. Without FF, the interaction of the surgeons with the environment is not as intuitive as direct manipulation and therefore extensive training is required. Moreover, the lack of haptic feedback leads to a higher risk of errors and longer task completion time, up to two orders of magnitude in complex tasks (Hannaford et al., 1991), which may lead to higher surgery costs.

One active research stream in the field of robotic surgery is improving the sense of telepresence for the surgeons, also known as “transparency.” Direct FF is the most intuitive approach to improve transparency. For a fully transparent haptic experience, reliable interaction force sensing at the surgeon console and the instrument–tissue interface is required. This is in addition to a safe bilateral teleoperation architecture, and a local manipulator that is capable of reflecting the force commands, known as a haptic display. The extensive literature on haptic control indicates a trade-off between transparency and stability (Hashtrudi-Zaad and Salcudean, 2001). Alternatively, SS was proposed instead of haptic feedback, in the form of visual, auditory, or vibrotactile cues of force information. Although the safety can be easily guaranteed in systems with SS, it is not intuitive and can cause cognitive overload for the surgeon. The SS methods can also be used in MIS systems because no robotic manipulator is required for force reflection. The efficacy of different haptic feedback modalities in improving the surgical training and surgeon performance metrics has been studied extensively (Abdi et al., 2020; Amirabdollahian et al., 2018; Overtoom et al., 2019; Rangarajan et al., 2020). It has been shown that a transparent haptic experience and visual feedback of force information improve the performance metrics and shorten the training time for novice surgeons in complex tasks. Apart from haptic feedback, the instrument–tissue interaction forces can be used for tissue damage monitoring, surgical skills assessment, development of surgical training guidelines, and to automate tasks.

Extensive research has been conducted to estimate or sense the instrument–tissue interaction forces. The functional requirements depend on the application. Although it is not necessary to estimate the tissue forces precisely to provide an appropriate haptic experience (Jones et al., 2011), the bandwidth and sampling rate are important requirements to ensure low latency and smooth interaction with the remote environment. The sampling rate and bandwidth are less critical in SS.

Sensorless approaches utilize the information that is already available in the robotic manipulator: the axes positions and velocities, motors currents, and visual display of the surgical theater. With the exponential growth, over the past decade, in the available computational power to researchers, data-driven approaches based on supervised learning (Aviles et al., 2017; Li and Hannaford, 2017) have been widely adopted. Among them, neural networks have shown promising results when trained and used on one particular instrument. However, they require a long and computationally expensive training phase that is yet clinically prohibitive. The training is based on a set of measurements at the beginning of the surgery that is used afterward for force estimation throughout the entire surgery. Proposed approaches that have an instrument’s operational parameters as inputs, do not consider the variations between instruments and the change of instruments behavior throughout its use (Kong et al., 2018). Considering how the research direction has evolved over the past decade, experimentation with different model architectures, development of efficient training, and identification methods that can be automatically performed at the system start-up (Spiers et al., 2015), improving the computation time and incorporation of online adaptation techniques are attractive research areas to be further investigated. Moreover, all the existing literature uses the information at the patient manipulator for force estimation, but the inclusion of the operating parameters at the surgeon console may also improve the quality of force estimation.

The sensor design is another avenue towards collecting force data at the instrument tip. The sensor can be located inside or outside the body. The closer the sensor is to the instrument tip the more accurate the measurements are. However, the size, biocompatibility, sterilizability, insulation, and sealing requirements are more stringent when such an approach is followed. Design proposals for sensor integration into the instrument tip have limited adaptability because the instruments for different types of surgery have different shapes at the tip (e.g., EndoWrist cautery forceps, graspers, dissectors, needle drivers, etc.). Therefore, a custom sensor needs to be designed for every instrument which increases the development, fabrication, and maintenance costs.

A variety of transduction principles, including resistive, capacitive, optical, piezoelectric, and magnetic have been used in the development of sensing solutions for minimally invasive procedures. Although strain gauges are still the most commonly used transducers, the study by Trejos et al. (2017) showed that biocompatible adhesives and coatings can only survive a maximum of six steam sterilization cycles. Considering that the instruments are typically used 10 times before disposed, this would lead to a 40% increase in the cost of the instruments with integrated strain gauges. In addition, the installation of strain gauges is labor-intensive that contributes to increased cost.

A comparison of the publications summarized in this article with the surveys by Puangmali et al. (2008) and Trejos et al. (2010) indicates a noticeable shift towards utilizing FBG and MEM sensor technologies for the development of gripper integrated miniature sensors (see Figure 5). FBGs are compact, sterilizable, biocompatible, electrically passive, and immune to electromagnetic noise. They provide high sensitivity with sub-micrometer resolution and can have multiple gratings embedded in one fiber which simplifies fiver and therefore optical signal management. Although the commercial interrogators are expensive, there are signal conditioning solutions proposed to decrease the electronics cost (Yurkewich et al., 2014). The developments in MEM sensor technology have overcome the barrier of scale and cost in the fabrication of delicate miniature sensors. In addition, MEM sensors typically do not need manual assembly and can be integrated into the desired application after production.

Another observable trend is the utilization of data-driven regression approaches for sensor calibration. Models based on neural networks and other supervised learning methods such as Gaussian process regression have shown unprecedented performance in handling non-linearities and uncertainties in sensor calibration. Compared with the surgical instruments, the transducers show a more consistent response and do not need regular calibration unless removed and reintegrated. Efficient calibration approaches that can be quickly and automatically performed without operator intervention (e.g., based on payload estimation) would benefit the RMIS systems.