Haptic force rendering of rigid-body interactions: A systematic review

Abstract

Haptic rendering has been developing for decades with different rendering approaches and many factors that affect the stability when rendering rigid-body interactions have been investigated. To get an overall understanding of the challenges in haptic rendering, we approach this topic by conducting a systematic review. This review examines different haptic rendering approaches and how to deal with instability factors in rendering. A total of 25 papers are reviewed to answer the following questions: (1) what are the most common haptic rendering approaches for rigid-body interaction? and (2) what are the most important factors for instability of haptic rendering and how to address them? Through the process of investigating these questions, we get the insight that transparency can be further explored and technical terms to describe haptic rendering can be more standardized to push the topic forward.

Keywords

Haptic rendering force rendering rigid-body interaction

Introduction

Haptics is usually referred to as the study of the human sense of touch, which is one of the five human senses. The other four are the senses of sight, sound, smell, and taste. In more general terms, the field of haptics can include studies of human perception of pressure, temperature, vibration, and so on. In recent years, given the rise of electronics and computational power, new ways of human computer interactions like virtual reality (VR) applications are closer to consumers and more than mere science fiction. This also pushes the study of haptics forward to provide feedbacks through haptic channels for the user. Figure 1 is a simple illustration of this interaction in a multimodal setup which can include but is not limited to visual, auditory, and haptic feedback. The arrows indicate information flows. Starting from the bottom right, the user interacts with the interface and most of the time the motion and force of the user is tracked and mapped into the virtual environment (VE). Based on the VE’s type and properties, the target of the rendering is computed, which is mapped back onto the interface, creating sensations in multiple perception channels for the user to experience. The multimodel feedback the user gets may trigger this information flow loop once again. To improve the fidelity (virtual presence and immergence) of interacting with the VE, a key feature is to provide realistic haptic feedback on top of the usual visual/audio feedback provided by the VR headset. Haptic feedback can be provided to the user through vest,¹ gloves,^2–4 fingertips,^5,6 or hand-held tools^7–9 in the forms of force or vibration. Apart from VR applications, haptic feedback can also be used in dental¹⁰ or surgical training simulators,^11,12 mixed reality applications,^13,14 and tele-operations.^15–17

Figure 1.

VR system with a multimodel interface.

Although haptic devices can take many shapes and forms, and can be used in many applications, in this paper, we focus on the force feedback desktop devices through a hand-held rigid tool, interacting with rigid VE. The study on this type of haptic device is quite common in the haptic community given that it has closer ties to traditional study of robotics, unlike other devices that mostly depend on new material or actuation.^5,6,14 However, unlike traditional robotics, which studies unilateral systems like robotic manipulators, the study of this type of haptic devices deals with bilateral systems. In other words, the user drives the device which, in returns, also tries to drive the user. It can be challenging to keep the system stable now that the behavior of the user has to be considered.

When the user interacts with VE, the VE can be rendered graphically with color, shape, and shading, and also haptically with weight, stiffness, and texture. Haptic rendering describes the way to demonstrate the haptical properties of the VE for the user to feel it. Unlike the fact that a low refresh rate of 60 Hz or even 30 Hz can fool human perception in graphic rendering, it is a common understanding that the refresh rate of haptic rendering has to be around 1000 Hz for the user to feel realistic feedback.¹⁸ This puts a tight time requirement on how fast haptic rendering runs. According to Salisbury et al.,¹⁹ a typical haptic loop consists of five steps:

Read the sensor data as the configuration of the device.

Map the device configuration to the tool pose in VE.

Detect collision between the tool and the VE.

Compute the interaction force.

Map the force back into control signals for the device.

Although this definition may vary in different literature,^20,21 the main difference among haptic rendering approaches is usually in the implementation of step 3 and 4, which are called collision detection and collision response in some studies.²⁰ As mainly solved with graphic rendering approaches, collision detection is more of a graphic rendering problem. Collision response, on the other hand, is the step directly related to the computation of the correct force feedback, which is the focus of many haptic studies. It is also crucial to ensure stability and needs more research to explore.

The rest of the paper is structured as follows: some common definitions and terms in haptic rendering are listed and clarified in the next section. After that, the review method used in this study is described. In Section Review Summary, the reviewed results are summarized into two tables as an attempt to answer the research questions developed in the previous section and a discussion is presented afterward. We draw some conclusions and suggest some general future research directions in the last section of the paper.

Common terms in haptic rendering

As pointed out in the “Introduction” section, we study the force feedback devices that render rigid-body interactions in this paper. The study of this type of haptic device takes up a large portion of the haptics literature. That is why sometimes “haptic rendering” is used to only describe “force rendering,” a subset of the broad meaning. This can also be true for “haptic devices” as the majority of them will be devices that provide force feedback. The word “device” can also be replaced by other synonyms within a certain context, such as “display” or “interface.”“Virtual environment (VE)” is commonly used in VR applications and it is sometimes interchangeable with “virtual object (VO).” Neither words have a strong indication on whether the described subject can move, which is often clarified by “dynamic (VE)” or “moving (VO).”

Stability and fidelity

Stability is always at the center of haptic rendering, because there is always a trade-off between stability and fidelity. Stability describe how stable the system is while fidelity describe how close the rendered VE is to its counterpart in reality. It is hard to render high stiffness VE with high fidelity while maintaining stability. The mechanical impedance rendered by the system is limited by many factors, one of which is the saturation limit on the actuators. In addition, stability has to be guaranteed, which limits the rendered impedance even more. If the system is unstable, the user will start to feel unwanted vibrations, distorted forces, or delays in response, which defeats the main purpose of being realistic. That being said, the trade-off sometimes depends on the emphasis of the application.

Stability for haptic systems is often guaranteed by passivity. That is why passivity-based approaches are popular in haptics literature. A system is passive if it does not generate energy. It should be noted that passivity is sufficient for stability, but not necessary. This is one of the reasons why passivity-based approaches can lead to conservative behaviors, hence, reduce fidelity.

Because the user interacts with the VE through a mechanical device, the notion of transparency is also used sometimes together with fidelity. The intermediate system is considered transparent when the user cannot feel it and the user feels like interacting with the VE directly. Transparency is achieved by compensating aspects of the device like the dynamics of the haptic device. However, active compensation is rare in commercial devices as it can make the system active (instead of passive) and introduces new ways to make the system unstable. In an implementation, transparency is approached by minimizing inertia and friction in the design phase of the device. Transparency definitely helps to achieve high fidelity but it is neither sufficient nor necessary.

Degree of freedom

The degree of freedom (DOF) can be mentioned in many ways within the context of haptics and they can have slightly different meanings. For haptic devices, it can mean the DOF for the device, in other words, the DOF in the motion of the device, or even more specifically, the tool. This is the general DOF notion on a mechanism level. If it is specific for a haptic device, it should be noted that the sensing and the actuating DOF of one device can be different. An example will be that many commercial devices have sensing capability in all 6 DOFs but they can only actuate in 3 DOFs. This is due to the fact that some commercial devices are designed to be 3-DOF mechanisms but later a passive 3-DOF add-on mechanism (usually the rotation DOFs) is mounted onto the base mechanism. The add-on mechanism only has sensing capabilities but no actuation. As for haptic rendering approaches, many of them are originally 3-DOF approaches because they only compute forces but not torques in Cartesian space. If they are theoretically proven that they can be generalized to cover the rendering of torques, they are then extended to be 6-DOF approaches. However, a separate issue can be that sometimes their experimental verification is still carried out on a device that is only capable of rendering interaction of fewer DOFs. Examples like this can be found later in the Review Summary section, where more details are given.

Research method

The chosen method to conduct this review study is illustrated in Figure 2.

Figure 2.

Flowchart for research method.

Review objectives

The aim of this review paper is to identify papers that study the haptic rendering of rigid-body interactions and summarize them so that this paper can serve as a guide or road map for people who would like to learn about different haptic rendering approaches. Thus, we form the research questions (RQs) as follows:

RQ 1: What are the most common haptic rendering approaches for rigid-body interaction?

RQ 2: What are the most important factors for instability of haptic rendering and how to address them?

These questions are used to guide the review process where we will identify and analyze the important papers in this area.

Search string

To be inclusive, we begin with a broad search string “haptic rendering” and add “haptic device” to have a strong indication that the study also involves a haptic device. As discussed in a previous section, some synonyms are added with “OR” because these terms are sometimes interchangeable in writing. The complete search string is constructed as:

(“haptic rendering” OR “force rendering”) AND (“haptic device” OR “force display” OR “haptic display” OR “haptic interface”)

We use this search string to collect papers from our choice of databases as long as the databases itself supports it. We choose not to include a keyword for rigid-body interaction as this type of rendering can be described in many different ways. Instead, the selection of this type of papers will be accomplished in the later title and abstract screening stages.

Sources

Five databases are chosen based on their relevance to haptic publications, namely, IEEE, ACM, WOS, Springer, and ScienceDirect. A coarse search shows that there are few relevant papers before 1990, so the time period is set to from 1990 to 2020. Only publications in English are included and the search yields in total 2059 entries.

Paper selection

The process of paper selection is shown in Figure 3. The height of the boxes for different sources and the width of all the arrows are proportional to the numbers they represent. As we try to be inclusive with the search string, many collected papers are irrelevant to the review objectives. Therefore, they should be taken out. This is achieved by title, abstract, and even, full-text screening. It should be noted that before any of that, 191 duplicates are found. A closer look suggests that it is mostly WOS overlapping with other databases.

Figure 3.

Paper selection process.

For the title screening, an aggressive amount of papers are rejected, leaving 182 for the next stage of abstract screening. The keywords used in the search string actually include all kinds of haptic devices but our focus is on the desktop devices that provide force feedback. The keyword “haptic rendering” and its synonym can also appear in application-specific papers that are not suitable in the summary. The approaches for deformable VE are also filtered out unless the approach is originally developed for rigid-body interactions and later generalized to cover non-rigid-body VE.

For the abstract screening, we can see a condensed description of the full paper, which helps us decide better which papers to keep. The rationale is the same as last stage and the difference is that there is more information available. At this stage, if it is still unclear whether to keep the paper after reading the abstract, we are more likely to keep it for the full-text screening. After the abstract screening, 42 papers are kept. Snowballing, a common technique used in systematic review to include more relevant papers based on the reference and citation list of the current identified papers, is done before the full-text screening, which gives us 25 papers in total for reviewing.

Review summary

The reviewed papers are roughly divided into two groups to better answer the RQs. Group 1 is more focused on different haptic rendering approaches that have been developed over the years, while Group 2 is more toward the analysis on the different factors of instability. Although the two RQs are overlapping a bit in the sense that, for example, a new haptic rendering approach can be proposed to address certain factors so that the new approach has some advantages over others. In this case, we still put the paper in Group 1 as long as the main contribution of the paper is the approach but not the analysis.

Haptic rendering approaches

Zilles and Salisbury²² bring the idea of “god-object” into haptic rendering, describing, originally, a point in the Cartesian space, a virtual location, over which the algorithm has complete control. This is as opposed to the real location for the haptic device which can be inside VE due to penetration. With the notion of the “god-object” and the history of it, the approach is better at avoiding push-through with thin objects when compared to previous approaches which use the location of the haptic device directly. We consider this study the starting of constraint-based haptic rendering approaches.

Ruspini et al.²³ extend the idea of “god-object”²² to have a finite size and call it virtual proxy. At this point in time with haptic studies, the execution speed of the haptic rendering is largely limited by the computational power of the hardware and the 1 kHz is a challenging refresh rate to achieve. They use a hierarchy of bounding spheres to speed up the search for collisions with polygon-based VE.

Gregory et al.²⁴ divide the polygonal VO into convex primitives to improve the speed for collision detection. They claim that the approach can execute the haptic loop within 1 ms constantly.

Hannaford and Ryu²⁵ propose a time domain passivity-based approach (TDPA) to have stable haptic rendering. The idea is to introduce variable damping to dissipate the extra energy that will otherwise destabilize the system. Many studies have followed this path but we choose to include this one in the table as a good reference point to the start of this idea.

Constantinescu et al.²⁶ present an impulse-based approach which is considered a different way for force computation. It is actually a hybrid approach called impulse-augmented penalty-based approach which combines the advantages of impulse-based rendering initial contact and penalty-based rendering resting contact. The contact model is a dynamic model with contact velocity instead of a pure geometric one. The experiments are conducted on a planar 3-DOF ( $x, y, θ$ ) device and the performance of the approach is compared against penalty and constraint-based approaches. It is quite special that the limitation to implement this approach is the type of the device, more specifically, the DOF configuration of the device. In the paper, it is explicitly stated that the approach should work on a planar 3-DOF device (2D space) or a 6-DOF device (3D space). A position-type 3-DOF device with $(x, y, z)$ is not suitable for this approach because full torque feedback is needed as a way to guarantee the kinetic energy of the user’s hand is bounded.

Otaduy and Lin²⁷ conduct one of the few studies that aim for maximal transparency. They use a linearized contact model to speed up the computation and they apply K-means clustering to filter multiple contact points. They also formulate implicit backward Euler integration for the simulation of the virtual tool to enable stable rendering even when the virtual tool has small mass. This is proved to improve transparency with an experiment in the free space. As this is an approach that has multiple improvements in different aspects of haptic rendering, the authors also mention that a multi-rate architecture is needed to ensure the haptic thread running at 1 kHz. The computation-heavy contact thread can run asynchronously as fast as possible.

Ortega et al.²⁸ extend the idea of “god-object”^22,23 even further to form a 6-DOF haptic rendering approach. Additionally, they introduce a constraint-based quasi-static method for force computation and the computation uses constrained and unconstrained acceleration. As constraint-based approaches are known to be computationally heavy, the researchers separate the implementations into three asynchronous blocks and the limitation of the approach is still the fact that it cannot guarantee a constant 1 kHz refresh rate when the system is busy with complex geometry.

Kim and Ryu²⁹ develop an energy bounding approach (EBA) to ensure stability based on passivity. It should be noted that the approach is for 1-DOF rendering at least at first. This is quite common for approaches based on passivity or other energy-related ideas as generalization to multiple DOFs can be non-trivial.

Wang et al.³⁰ propose a configuration-based (position-level) approach, differentiating itself from previous work²⁸ which relies on acceleration. They claim that because they work directly with position values, they avoid the inaccuracy from numerical integration if the position values are estimated from accelerations. They use sphere tree to speed up the search and the data structure is generated offline beforehand with separate software. The study emphasizes on the dental simulator application and the approach is good for fine manipulation which includes preserving fine geometric features and motion in narrow spaces.

Ryu and Yoon³¹ seem to be inspired by the hysteresis pattern found in the force-position plot to develop a memory-based passivity approach (MBPA) for stable haptic rendering. Fast reading and writing the memory in real-time is achieved by using field-programmable gate array (FGPA). The idea is that the releasing path in the plot should never go higher than the pressing path. If this is the case, the system dissipates energy and thus is considered passive. To enforce the passivity, all the paths in the plot (position and force) are recorded and compared in real-time control. The approach is developed and tested thoroughly in 1-DOF and the authors claim that it can be generalized to cover3-DOF $(x, y, z)$ cases by applying the 1-DOF approach in every DOF, though the performance is stated to be too conservative. The conservative behavior is reasonable because passivity in every DOF is not necessary for system passivity as long as the sum of energy from all DOFs still indicates that the system dissipates energy. In other words, for a 3-DOF case, up to2-DOFs can be active as long as the third DOF dissipates all the extra energy and still makes the whole system passive. In addition to that, it is pointed out in a later paper³² that the generalization from translational DOF to rotational DOF is non-trivial as well due to the fact that conjugate power pair in rotational space is not clear. Another obvious limitation for applying this to an existing haptic device is that the approach requires the hardware of FGPA.

Kim et al.³³ summarize many previous passivity-based approaches^25,29,31 and propose the force bounding approach (FBA), of which the advantage is that it is robust to changes in sampling frequency and time delay. They claim to discover two sufficient conditions for passivity, although the less conservative one suffers from contact oscillation due to energy accumulation, which is more obvious when the tool moves in free space for a long time before contact. The more conservative condition incorporate a systematic way of resetting the memory effect but it induces more conservative behavior.

Kim et al.³⁴ develop a depth-image-based approach to determine the surface contact point. The depth cube data structure is generated with six depth cameras in the virtual world and the VOs are convex-decomposed to avoid obstruction in the images. This approach also adopts an asynchronous architecture and requires the hardware to have strong image processing power, which is different from other similar approaches that require mostly haptic related computational power. The author use FBA as force computation method and they state that many details will be explored as future work.

Jafari et al.³⁵ propose input-to-state stable (ISS) approach, which basically guarantees dissipativity, to address the conservative behavior issues in most passivity-based approaches. Compared to the previous approaches, the attempt is to relax the previous conservative criteria even further. We think the approach is partly related to MBPA³¹ as it is also based on FPGA hardware and the force-position plot.

Xu and Barbič³⁶ use continuous collision detection (CCD) between point and distance field to find the surface collision point and they speed up the search with the help of octree and sphere tree data structures. Because the contribution is in collision detection, they demonstrate that CCD can be integrated with previous developed constraint²⁸ and penalty-based³⁷ force computation methods.

Singh et al.³⁸ explore the successive force augmentation (SFA) approach to promote the idea of rate-hardness instead of the stiffness of VE. The approach adds a feed-forward force to make the rendering force larger while the penetration is still small. Then the calculated stiffness is higher but the desired VE stiffness is still kept relatively low, which is important for stability. They also extend the SFA approach in this paper so that the high desired stiffness is perceived in the first pressing and then the SFA approach is active. To smooth out the discontinuity in the force rendering due to the feed-forward force, the approach shift the boundary of the VE with a negligible amount.

These papers on different haptic rendering approaches are listed and compared in Table 1. Each row is one paper and each column is a related information point extracted from the paper, which will be discussed later in the Discussion section or for the readers to use as a reference for comparison. As stated in the Introduction section, a typical haptic rendering loop has five steps. A haptic rendering approach can contribute to at least one step to be listed in here. We find that the main contributions for the reviewed papers most focus on step 3 and 4, namely, collision detection and force computation. The first two columns of the table apart from the title of the papers highlight the main contribution of each paper. It should be noted that although it is called “collision detection,” it is also responsible for computing the surface contact point or the interaction point in addition to detect whether or not there is a collision. Once collision detection is done, the information generated will be passed to force computation. Therefore, the information generated from collision detection can be different based on what is needed for force computation. The third column is the theoretical base and serves mostly as a name to identify and differentiate each approach. The fourth column contains six sub-columns to show different rendering cases used for different purposes. We try to distinguish different “DOFs” introduced in a previous subsection in the sub-columns of the fifth column to demonstrate that they are different and should not be mixed together. The sixth column shows different types of plots used in different papers.

Table 1.

Summary of different haptic rendering approaches. .

Paper	Collision detection	Forcecomputation	Theory	Rendering cases						DOF			Plot type							Baseline	Limitation
				Wall	SimpleVO	Complexbenchmark	Peg ina hole	Moving VO	Freespace	ApproachDOF	DeviceDOF	ExperimentDOF	Position	Force	Energy	Stiffness	Force-position	Numberof contacts	Timecost
Singh et al.³⁸		√	SFA	√						1	6	1	√	√						None	Multi-DOF
Xu and Barbič³⁶	√		CCD			√				6	6	6	√					√	√	DCD	Data structure
Jafari et al.³⁵		√	ISS	√						3	6	1, 2					√			TDPA, PD	Multi-DOf
Kim et al.³⁴	√		Depth			√				6	6	6								None	Data structure
Kim et al.³³		√	FBA	√	√					3	1, 3	1, 3	√	√	√	√				None	Multi-DOF
Ryu and Yoon³¹		√	MBPA	√				√		3	1, 2	1, 2					√			TDPA, PD	Use of FPGA
Wang et al.³⁰	√		Config.			√				6	6	6		√				√	√	None
Kim and Ryu²⁹		√	EBA	√						1	2	1	√	√						None	Multi-DOF
Ortega et al.²⁸	√	√	Const.			√	√			6	6	6		√				√	√	None	Exec. speed
Otaduy and Lin²⁷	√	√	N/A		√	√			√	6	6	6	√	√						None
Constantinescu et al.²⁶		√	Impulse		√					6	3	3	√	√						PD, const.	Device DOF
Hannaford and Ryu²⁵		√	TDPA	√						1	3	1	√	√	√					None	Multi-DOF
Gregory et al.²⁴	√	√	Convex		√		√	√		6	6	6								None	Exec. speed
Ruspini et al.²³	√		Const.		√	√				3	3	3								None	Exec. speed
Zilles and Salisbury²²	√		Const.		√					3	3	3								None	Exec. speed

Two other papers also come to our attention although they may not be suited to be listed in the summary table. We list them here for references and completeness of the study.

Kunchenbecker et al.³⁹ propose a 1-DOF event-based haptic rendering approach that can be considered contributing to force computation. Unlike other approaches that try to ensure the stability of a contact, their approach aims to render contacts with different materials so that the user can differentiate, for example, wood and foam, through tapping it in VR. This reads like it is related to the stiffness of the material but they believe transient (high-frequency accelerations at impact) is the key. So in order to use this approach, a library of transient needs to be build first, offline, but the library also grows with the number of materials.

Lim et al.⁴⁰ introduce an analog input shaper (AIS) to enhance stability and it can be used with EBA.²⁹ This paper is left out because the improvement is done through changes in analog circuitry. Thus, it is not really comparable to other papers in the table.

Instability factors

Colgate and Brown⁴¹ probably initiate one of the first papers to study factors that affect the performance of a haptic device. They suggest that “Z-width,” the dynamic range of achievable impedance, is an important measure for performance and an impedance has to satisfy passivity to be considered achievable. They present four factors that is affecting the Z-width, namely, the sampling nature of the system (labeled as discretization in Table 2), the inherent dynamics of the device, the position sensor quantization, and velocity filtering. Four of the six major factors listed in Table 2 are already here. In the conclusion, they provide some simple guidelines, one of which is to maximize the inherent damping of the system. This is claimed to be the most effective but it can be counter-intuitive as the design guidelines for haptic devices in general is to achieve low friction and low inertia. The analytical form of the guideline looks like this:

b > \frac{KT}{2} + | B |

Table 2.

Summary of reviewed papers on instability factors for haptic rendering. .

Paper	Instability factors							Purpose			Test case(device)	Model of the virtual wall	K-B plot	Type of study			Outcome
	Discretization	Inherentdynamics	Quantization	Velocity filtering	Timedelay	Friction	other	Stability	Passivity	Fidelity				Theory	Simulation	Experiment
Mashayekhi et al.⁴²					√		Many	√			KUKA LWR first joint	Bilateral	d-less	√	√	√	Criteria
Chawda et al.⁴³				√					√	√	1-DOF custom device	Spring-damper	Novel		√	√	Approach
Colonnese and Okamura⁴⁴			√	√			Many	√	√		Haptic device first joint	Spring-damper	Vanilla	√	√	√	Criteria
Hulin et al.⁴⁵					√		User model	√	√		1-DOF transfer function	Spring-damper	d-less B-K	√	√		Guidelines
Mashayekhi et al.⁴⁶						√			√		1-DOF transfer function	Spring		√	√		Criteria
Diolaiti et al.⁴⁷	√		√		√	√			√		1-DOF motor	Spring	d-less	√	√	√	Region
Abbott and Okamura⁴⁸	√		√						√		1-DOF custom device	Spring		√		√	Criteria
Colgate and Brown⁴¹	√	√	√	√					√	√	1-DOF motor	Spring-damper	Vanilla	√		√	Guidelines

where $b$ is the damping of the device, $K$ and $B$ are the virtual stiffness and damping, respectively, and $T$ is the sampling period of the system. Usually, to rendering a virtual wall, $B$ is set to zero, so that

K < \frac{2 b}{T}

(1)

A virtual stiffness vs. virtual damping plot (labeled as K-B plot in the title row of the table) is used to demonstrate the Z-width of the device under different “configurations.” Within each configuration, the maximal achievable stiffness is found for the entire range of achievable damping. The configurations here are the different combinations of the four factors studied in this paper. The plot is used later in most reviewed papers. Note that the version in this paper is labeled as “vanilla” because the axes are still with dimensions/units and that is considered the vanilla version of the plot.

Abbott and Okamura⁴⁸ extend the guideline in (1) to explicitly include the effect of Coulomb friction and position sensor resolution

K \leq min (\frac{2 b}{T}, \frac{2 f_{c}}{Δ})

(2)

They also introduce this “malicious touch” concept where the user intentionally makes the device vibrate against the wall. As their goal is passivity, they show through experiment that even under the influence of the “malicious touch” behavior, the vibration will diminish.

Diolaiti et al.^47,49 add time delay and Coulomb effects to the factors and they use dimensionless signals and parameters in formulation. Because their study focuses more on the friction factor of the system, the plot they used is dimensionless Coulomb friction vs. viscous friction. They divide regions on this plane to further categorize the behavior of the system based on stability and passivity. These regions are sometimes referred to as passivity/stability criteria/boundaries.

Mashayekhi et al.⁴⁶ conduct a study to improve the passivity criterion proposed previously by Diolaiti et al.⁴⁷ They use analytical and numerical methods to prove that the new criterion is more accurate with low-speed cases while they consider experiment as future work.

Hulin et al.⁴⁵ argue that by considering the user model, the maximal stable rendered stiffness can be way higher than what is admitted by passivity. They collect a few parameterizations of the human arm from different papers and investigate the effect on stability. Similar work can be found in a paper by Desai et al.⁵⁰ with a spring-mass virtual wall. Hulin et al. invert the axes and use dimensionless B-K plot to demonstrate their stability boundaries and passive regions related to stable ones. In the guidelines concluded from the study, they highlight that minimizing effective delay is more efficient than having a faster sampling. Contradictory to Colgate and Brown’s⁴¹ suggestions, their guidelines promote minimizing physical damping, as it contributes marginally to stability. Low physical damping also improves transparency.

Colonnese and Okamura⁴⁴ go back to the vanilla version of the K-B plot to show different stable or passive regions. They emphasize on “quantization-error passivity” which is different from “sampled-data passivity”⁴¹. While the latter ensures stability when the system is coupled to an arbitrary passive human impedance (coupled stability), what they propose concerns energy generated due to quantization error. They state that increasing the damping of the device can be useful to the increase of impedance and the intentionally added damping can be compensated to improve transparency in free space.

Chawda et al.⁴³ highlight the trade-off in velocity estimation method that if less noise is admitted into the system, larger time delay will be in the estimations. They compare four different velocity estimation schemes by a novel K-B plot which adds a third dimension to show the fidelity of the rendered stiffness of the wall.

Another paper from Mashayekhi et al.⁴² is included as it is quite different from their previous work.⁴⁶ This paper identify the need to extend the stability criterion developed in a previous paper⁴⁵ to work for both small and large values for damping as well as time delay. While the increased range of these two parameters are thoroughly studied in this paper, the authors also claim that any other dynamic effect can be accommodated in the framework as long as it is linear.

A summary of these reviewed papers on various instability factors can be seen in Table 2. Because instability factors are the focus of this table, we include them in the first column with different factors as sub-columns. We group both coulomb and viscous friction together and we treat the discussion about sampled data of the system as discussing the effect of discretization. The purpose of most reviewed papers here should be stability analysis and it is shown as one of the sub-columns in the second column. Due to the fact that passivity is a sufficient condition to stability but not necessary, and that passivity is easier to analyze, passivity can be studied more commonly. As passivity-based approach can lead to conservative behavior, sometimes fidelity can be discussed to achieve higher impedance. As captured in the third column, most of the test cases/devices only have 1 DOF. The spring-damper model of the virtual wall is still commonly used as indicated in the fourth column. Different from papers in Table 1 that use different types of plots to present their results, the papers here mainly use stiffness-damping (K-B) plot with some variations as shown in the fifth column. The sixth column denotes whether the paper is theory-, simulation-, or experiment-based.

Discussion

Force computation approaches

The haptic rendering approaches that focus on collision detection tend to be 6-DOF, or at least 3-DOF approaches while approaches on force computation can start with 1-DOF. From Table 1, it is also clear that sometimes it is non-trivial to extend a 1-DOF force computation approach to full 6 DOFs. The difficulty can come from the fact that 1) multi-DOF approaches cannot be achieved by simply apply the 1-DOF approach on each DOF 2) the generalization from translational to rotational space is not straightforward. This can be the reason why force computation approaches usually take a few papers to develop to the 6-DOF version. The experimental verification for force computation approaches is often done in 1 DOF, partially because many of them started as a 1-DOF approach. The device used in the experiment tends to have more DOFs as opposed to those 1-DOF devices used in instability factor studies which we will discuss later.

A baseline from another research group for comparison in these force computation studies is sometimes hard to find because the contribution seems to be proposing a new idea/approach instead of a better approach. The evaluation is also not standardized due to different emphasis of different approaches, which makes it rarely the case that one approach is “superior” than another. Other reasons can be that the approach from another research group is hard to replicate due to issues like lack of specific hardware.

The challenging part or the main focus of a force computation approach is still stability, instead of fidelity or transparency. Stability is often achieved by passivity-based approaches which can lead to conservative behavior. Although this might not be explicitly stated in each of these reviewed papers, 6 out of the 15 papers we include in Table 1 are related to passivity analysis. And if we only consider the force computation studies, that ratio will rise to 6 out of 9. Although some of the papers^31,33,35 try to address the issues with conservative behavior or explore less conservative criteria while still maintain stability, some of the attempts are promising but not mature. Only one paper²⁷ explicitly talks about transparency in free space with their approach. On the one hand, this looks like stability is the major challenge and passivity analysis is a great tool to guarantee stability theoretically. On the other hand, this can be that transparency needs to be further explored. However, the study of transparency usually requires active compensation of undesired dynamics, which may go against passivity.

Collision detection approaches

Our collection in Table 2 may not be inclusive because some of the advanced methods used here are developed in and borrowed from graphic rendering literature, thus missing in our search.

For collision detection approaches, time cost is usually the major challenge because the number of contacts can grow fast with geometrically complex objects and there is a rather hard limit on the refresh rate of the haptic rendering loop. Some of the issues here are addressed by adopting a multi-rate multi-thread architecture and various data structures are used to speed up the search or simplify the computation. Most of the time, these data structures need to be constructed offline beforehand. In the future, maybe it can be constructed online with adaptive “resolution” to save memory space and improve usability newly given VE.

Instability factors studies

Judging from the literature, the major factors have been established quite early and different studies try to target different factors and maybe put them under a unified framework. Some papers^42,44 also claim that more factors can be incorporated into their frameworks conditionally. The user model is ignored by most and discussed by some as user behavior is unpredictable in a way, thus can lead to different outcomes. While a firm grip on a haptic device can increase the maximal stiffness rendered stably,⁵¹ certain behaviors like “malicious touch” can induce instability more easily.⁴⁸

All paper listed in Table 2 verify their strategy to tackle certain instability factors in 1-DOF experiment although the device used can have multiple DOFs. This indicates an effort to filter out other minor details in implementation and to focus on a theoretical level. However, it should be noted that there is little (or no) indication as to how these strategies can be generalized and applied to, for example, a commercial 6-DOF device in 6-DOF haptic rendering.

Through these studies, many strategies and design guidelines are proposed, some of which are straight-forward and easy to follow, such as maximize sensor resolution and sampling rate or minimize (effective) delay. Other outcomes like stability criteria and regions can be used for self-evaluation and troubleshooting as to what behavior to expect in certain setup and what to change if the behavior is undesirable. As pointed out in previous Instability Factors subsection, there are different opinions^41,45 on physical damping and mass of the device. For most commercial haptic devices, the guidelines for the mechanical design seem to be to have low physical damping and mass which can help to improve transparency passively. Colgate and Brown⁴¹ advise to maximize physical damping as it improves Z-width and lower damping can be felt by the user by rendering negative virtual damping. Hulin et al.⁴⁵ argue that physical damping should be minimized for transparency as it is not that effective with stability to begin with. The paper also suggests not to minimize mass of the device as low-mass devices are more likely to be affected by user mass. The guidelines are contradictory partially because they aim for different goals, namely, stability or transparency. One should choose a suitable guideline according to the specific application.

Limitations of the review

Although we aim for an inclusive study of the literature on haptic rendering, these issues can potentially limit the study.

First of all, we only search for English publications from all sources and papers with digital records are more likely to be included. Secondly, although we use the same search string for all the sources, the differences in the search engines can introduce subtle differences in the results. Furthermore, the total entries in the beginning of the paper selection process is 2059 and we aggressively filter them and review 25 of them. Although a large portion of the rejected papers are the irrelevant papers and they are firstly collected due to the inclusiveness of using synonyms in the search string, papers with titles and abstracts that do not meet our selection criteria will still likely be filtered out. Similar publications on the same research idea are combined and only one entry is kept in the summary tables. Finally, it comes to our attention that some of the papers in the tables are from the same research groups. Although this may make our summary biased, we state that we have no favor over any research groups and these papers are included only because we considered them carrying different research ideas.

Conclusions and outlooks

This paper gives an overview of currently available haptic rendering approaches and how to address some major instability factors by compiling two tables with reviewed papers from a systematic collecting, selecting, and reviewing process. RQ 1 is answered by Table 1 with various aspects of the different approaches, while papers included in Table 2 give us a general direction to answer RQ 2. Given the literature study, we conclude the most important instability factors for haptic rendering are: discretization (sampling effect), inherent dynamics (mass), quantization (sensor resolution), velocity filtering (estimation), time delay, and friction (physical damping). Other factors can include the user model but it is mostly ignored or discussed separately. To address these factors, many passivity-based analyses are made to form guidelines and criteria for design and control.

The study of transparency in free space is still lacking. A trade-off between stability and transparency can be equally challenging as that between stability and fidelity in rendering high-stiffness interactions. How to actively compensate for inherent dynamics while still stay stable or passive remains to be explored in the future. Penalty-based approaches are widely used for its simplicity. However, if the computational power and the data transferring speed are not the limits to achieve a steady haptic refresh rate in the future, maybe constraint-based or some other hybrid approaches will be more widely used than penalty-based approaches for the emphasis on non-penetration. New sensors that speed up the collision detection step and new actuators that have a higher power density (power-to-mass ratio) or faster response can all contribute to a more stable high-fidelity haptic rendering system. Overall, it will also benefit the community to establish standard benchmarks for evaluation and taxonomy and categories to clarify the relationship between different approaches. The haptic rendering of deformable objects, though not included in the scope of this study, is another frontier in haptic rendering and it can broaden the application in surgical simulators from hard tissues to soft tissues. Whether or not to provide extra force feedback in the real robot-assisted surgeries is still debatable as more evidence is needed to prove safety and effectiveness. There is also the notion that the extra feedback can be conveyed through other sensory channels to keep the surgeon’s hands free of disturbance.

Footnotes

Acknowledgements

Yang Wang would like to thank China Scholarship Council (CSC) for supporting his Ph.D. study.

Handling Editor: James Baldwin

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Lei Feng is financially supported by KTH XPRES.

ORCID iD

Yang Wang

References

Garcia-Valle

Ferre

Brenosa

, et al. Evaluation of presence in virtual environments: haptic vest and user’s haptic skills. IEEE Access 2018; 6: 7224–7233.

Ben-Tzvi

. Design and optimization of a five-finger haptic glove mechanism. J Mech Robot 2015; 7: 041008. DOI: 10.1115/1.4029437

Wang

Song

Naqash

, et al. Toward whole-hand kinesthetic feedback: a survey of force feedback gloves. IEEE Trans Haptics 2019; 12: 189–204.

Wang

Zhang

, et al. A three-fingered force feedback glove using fiber-reinforced soft bending actuators. IEEE Trans Ind Electron 2020; 67: 7681–7690.

Han

Wang

, et al. Haptic surface display based on miniature dielectric fluid transducers. IEEE Robot Autom Lett 2020; 5: 4021–4027.

Koehler

Usevitch

Okamura

. Model-based design of a soft 3-D Haptic shape display. IEEE Trans Robot 2020; 36: 613–628.

Massie

Salisbury

. The PHANToM haptic interface: a device for probing virtual objects. In: Proceedings of the ASME dynamic systems and control division, Chicago, IL, November 1994. pp.295–301. ASME.

Culbertson

Kuchenbecker

. Ungrounded haptic augmented reality system for displaying roughness and friction. IEEE/ASME Trans Mechatron 2017; 22: 1839–1849.

Zhang

Boyles

Abbott

. Six principal modes of vibrotactile display via stylus. In: 2018 IEEE haptics symposium (HAPTICS), San Francisco, CA, 25–28 March 2018, pp.313–318. IEEE.

10.

Zhang

Wang

, et al. DentalTouch: a haptic display with high stiffness and low inertia. In: 2017 IEEE world haptics conference (WHC), Munich, Germany, 6–9 June 2017, pp.388–393. IEEE.

11.

Basdogan

Kim

, et al. Haptics in minimally invasive surgical simulation and training. IEEE Comput Graph Appl 2004; 24: 56–64.

12.

Okamura

. Haptic feedback in robot-assisted minimally invasive surgery. Curr Opin Urol 2009; 19: 102–107.

13.

Cosco

Garre

Bruno

, et al. Visuo-haptic mixed reality with unobstructed tool-hand integration. IEEE Trans Vis Comput Graph 2013; 19: 159–172.

14.

Romanus

Frish

Maksymenko

, et al. Mid-air haptic bio-holograms in mixed reality. In: 2019 IEEE international symposium on mixed and augmented reality adjunct (ISMAR-Adjunct), Beijing, China, 10–18 October 2019, pp.348–352. IEEE.

15.

McMahan

Gewirtz

Standish

, et al. Tool contact acceleration feedback for telerobotic surgery. IEEE Trans Haptics 2011; 4: 210–220.

16.

Cizmeci

Schuwerk

, et al. Model-mediated teleoperation: toward stable and transparent teleoperation systems. IEEE Access 2016; 4: 425–449.

17.

Talasaz

Trejos

Patel

. The role of direct and visual force feedback in suturing using a 7-DOF dual-arm teleoperated system. IEEE Trans Haptics 2017; 10: 276–287.

18.

Otaduy

Garre

Lin

. Representations and algorithms for force-feedback display. Proc IEEE 2013; 101: 2068–2080.

19.

Salisbury

Conti

Barbagli

. Haptic rendering: introductory concepts. IEEE Comput Graph Appl 2004; 24: 24–32.

20.

Lin

Otaduy

. Haptic rendering: foundations, algorithms, and applications. Natick, MA: A K Peters/CRC Press, 2008.

21.

Siciliano

Khatib

. (eds.) Springer handbook of robotics. 2nd ed. Cham, Germany: Springer Handbooks, Springer International Publishing, 2016.

22.

Zilles

Salisbury

. A constraint-based god-object method for haptic display. In: Proceedings 1995 IEEE/RSJ international conference on intelligent robots and systems. Human robot interaction and cooperative robots, Pittsburgh, PA, 5–9 August 1995, vol. 3, pp.146–151. IEEE.

23.

Ruspini

Kolarov

Khatib

. The haptic display of complex graphical environments. In: Proceedings of the 24th annual conference on computer graphics and interactive techniques. SIGGRAPH’97, Los Angeles, CA, USA. August 1997, pp.345–352. ACM.

24.

Gregory

Mascarenhas

Ehmann

, et al. Six degree-of-freedom haptic display of polygonal models. In: Proceedings of the conference on visualization’00. VIS’00, Salt Lake City, UT, 8–13 October 2000, pp.139–146. Washington, DC: IEEE Computer Society Press.

25.

Hannaford

Ryu

. Time-domain passivity control of haptic interfaces. IEEE Trans Robot Autom 2002; 18: 1–10.

26.

Constantinescu

Salcudean

Croft

. Haptic rendering of rigid contacts using impulsive and penalty forces. IEEE Trans Robot 2005; 21: 309–323.

27.

Otaduy

Lin

. A modular haptic rendering algorithm for stable and transparent 6-DOF manipulation. IEEE Trans Robot 2006; 22: 751–762.

28.

Ortega

Redon

Coquillart

. A Six degree-of-freedom god-object method for haptic display of rigid bodies with surface properties. IEEE Trans Vis Comput Graph 2007; 13: 458–469.

29.

Kim

Ryu

. Robustly stable haptic interaction control using an energy-bounding algorithm. Int J Rob Res 2010; 29: 666–679.

30.

Wang

Zhang

, et al. Configuration-based optimization for Six degree-of-freedom haptic rendering for fine manipulation. IEEE Trans Haptics 2013; 6: 167–180.

31.

Ryu

Yoon

. Memory-based passivation approach for stable haptic interaction. IEEE/ASME Trans Mechatron 2014; 19: 1424–1435.

32.

Jafari

Ryu

. Memory-based passivation approach for 6-DOF haptic rendering of high stiffness virtual environment. In: 2014 11th international conference on ubiquitous robots and ambient intelligence (URAI), Kuala Lumpur, Malaysia, 12–15 November 2014, pp.108–111. IEEE.

33.

Kim

Baek

Ryu

. A force bounding approach for multi-degree-of-freedom haptic interaction. IEEE/ASME Trans Mechatron 2015; 20: 1193–1203.

34.

Kim

Lee

Ryu

. Depth cube-based Six degree-of-freedom haptic rendering for rigid bodies. IEEE Trans Haptics 2015; 8: 345–355.

35.

Jafari

Nabeel

Ryu

. The input-to-state stable (ISS) approach for stabilizing haptic interaction with virtual environments. IEEE Trans Robot 2017; 33: 948–963.

36.

Barbič

. 6-DoF haptic rendering using continuous collision detection between points and signed distance fields. IEEE Trans Haptics 2017; 10: 151–161.

37.

Tang

Manocha

Otaduy

, et al. Continuous penalty forces. ACM Trans Graph 2012; 31: 1–9.

38.

Singh

Janetzko

Jafari

, et al. Enhancing the rate-hardness of haptic interaction: successive force augmentation approach. IEEE Trans Ind Electron 2020; 67: 809–819.

39.

Kuchenbecker

Fiene

Niemeyer

. Improving contact realism through event-based haptic feedback. IEEE Trans Vis Comput Graph 2006; 12: 219–230.

40.

Lim

Kim

Yoon

, et al. An analog input shaper for stability enhancement of haptic interfaces and its application to energy-bounding algorithm. IFAC Proc Volumes 2008; 41: 14687–14692.

41.

Colgate

Brown

. Factors affecting the Z-width of a haptic display. In: Proceedings of the 1994 IEEE international conference on robotics and automation, San Diego, CA, 8–13 May 1994, vol. 4, pp.3205–3210. IEEE.

42.

Mashayekhi

Behbahani

Ficuciello

, et al. Analytical stability criterion in haptic rendering: the role of damping. IEEE/ASME Trans Mechatron 2018; 23: 596–603.

43.

Chawda

Celik

O’Malley

. Evaluation of velocity estimation methods based on their effect on haptic device performance. IEEE/ASME Trans Mechatron 2018; 23: 604–613.

44.

Colonnese

Okamura

. Stability and quantization-error analysis of haptic rendering of virtual stiffness and damping. Int J Rob Res 2016; 35: 1103–1120.

45.

Hulin

Albu-Schäfer

Hirzinger

. Passivity and stability boundaries for haptic systems with time delay. IEEE Trans Control Syst Technol 2014; 22: 1297–1309.

46.

Mashayekhi

Boozarjomehry

Nahvi

, et al. Improved passivity criterion in haptic rendering: influence of Coulomb and viscous friction. Adv Robot 2014; 28: 695–706.

47.

Diolaiti

Niemeyer

Barbagli

, et al. Stability of haptic rendering: discretization, quantization, time delay, and Coulomb effects. IEEE Trans Robot 2006; 22: 256–268.

48.

Abbott

Okamura

. Effects of position quantization and sampling rate on virtual-wall passivity. IEEE Trans Robot 2005; 21: 952–964.

49.

Diolaiti

Niemeyer

Barbagli

, et al. The effect of quantization and Coulomb friction on the stability of haptic rendering. In: First joint eurohaptics conference and symposium on haptic interfaces for virtual environment and teleoperator systems. World haptics conference, Pisa, Italy, 18–20 March 2005, pp.237–246. IEEE.

50.

Desai

Gupta

Chakraborty

. Effect of human hand dynamics on haptic rendering of stiff springs using virtual mass feedback. In: 2019 28th IEEE international conference on robot and human interactive communication (RO-MAN), New Delhi, India, 14–18 October 2019, pp.1–6. IEEE.

51.

Lawrence

Chapel

. Performance trade-offs for hand controller design. In: Proceedings of the 1994 IEEE international conference on robotics and automation, San Diego, CA, 8–13 May 1994, vol. 4, pp.3211–3216. IEEE.