Facial region division and robotic trajectory planning method for facial cleaning tasks

Abstract

Aiming at the demand for automated facial cleaning for people with hand dysfunction, as well as the key challenges faced by facial cleaning robots, including accurate 3D geometric perception of the human face and trajectory planning for cleaning on complex curved surfaces, this paper proposes a complete technical scheme of trajectory generation for robotic arm facial cleaning operations. The scheme adopts a three-layer architecture of “data acquisition–facial region division–motion planning.” Firstly, 3D point cloud reconstruction of the human face is realized through robotic arm surrounding acquisition and ICP registration, and accurate matching between 2D facial key points and 3D point clouds is completed in combination with the MediaPipe Face Mesh model. Subsequently, optimized division of facial regions for cleaning tasks is accomplished based on facial anatomical divisions and multi-dimensional curvature features. Finally, a smooth cleaning trajectory fitting the facial surface is generated through normal fitting pose solving, dual-constraint reachable pose search, and adaptive spline interpolation. Experimental results demonstrate that the proposed facial partitioning method effectively segments the facial point cloud on demand. Compared to traditional anatomical facial partitioning, the number of designated cleaning regions is reduced from 16 to 9, significantly lowering the complexity of trajectory planning. Furthermore, the motion planning method ensures the reachability, configuration consistency, and kinematic smoothness of the cleaning trajectory, providing a robust technical reference for the research and development of nursing facial cleaning robots.

Introduction

According to QYResearch data, ¹ the global market size of articulated robots reached USD 9.353 billion in 2024 and is expected to exceed USD 17.39 billion by 2030. Its application boundaries are rapidly expanding from traditional industrial manufacturing to the medical and health care field. Against the social background of the intensifying population aging in China and the large scale of the disabled population, the daily nursing needs of disabled patients (such as facial cleaning) have become an urgent problem to be solved. Although nursing robot technology has attracted much attention in recent years, existing studies mostly focus on environmental interaction (such as object grasping) or single-function operation, while there is still a lack of systematic exploration on coverage cleaning motion planning based on the 3D facial features of the human face. For example, the Mobile ALOHA robot developed by Zipeng Fu et al.’s team at Stanford University ² realized shaving operation through large-scale data-driven imitation learning, but did not involve the adaptability analysis of facial geometric features. The shaving robot demonstrated by Dome Intelligence at the 2024 World Artificial Intelligence Conference ³ verified the feasibility of contact between rigid tools and the human body, but did not disclose the technical details of the force control strategy and path planning. In the research on trajectory planning and operation optimization of facial skin massage robots by Hu Yan from South China University of Technology, ⁴ the trajectory acquisition and planning methods for facial massage were introduced, but this method cannot fully adapt to the needs of facial cleaning, nor did it elaborate on the normal vector-based attitude adjustment method in detail.

The technical implementation in this field faces multiple complex challenges, which essentially correspond to the core technical contradictions of flexible human-robot interaction and operation on irregular curved surfaces. The first is the adaptability challenge of the high-curvature irregular concave-convex surface of the human face. The human face is composed of multiple regions such as the forehead, cheeks, and mandible, presenting complex irregular curved surface characteristics as a whole, with significant local curvature changes (such as the nasal alar and periorbital regions). Traditional path coverage strategies designed for 2D planes or regular curved surfaces (such as the grid method and spiral line method) cannot match the geometric form of the facial 3D space, and are difficult to be directly migrated and applied. The second is the challenge of reachable pose optimization under the hardware constraints of the robot. Limited by the hardware parameters such as the degrees of freedom of the robotic arm joints, connecting rod length, and joint motion range, there are inherent reachable boundaries and singular configuration regions in its motion space. In the facial cleaning scenario, planning according to the ideal path may result in the robotic arm pose corresponding to some facial regions exceeding the reachable space, or entering a singular configuration leading to out-of-control motion. The third is the collaborative challenge of force control and collision avoidance in human-robot contact scenarios. The direct contact between the robot and the human face requires, on the one hand, high-precision force control to ensure that the cleaning contact force is within the safety threshold while meeting the minimum contact force required for the cleaning effect; on the other hand, it is necessary to prevent unexpected collision between the non-cleaning parts of the robotic arm and the face.

In response to the above challenges, this paper proposes a three-layer progressive architecture of “data acquisition–facial region division–motion planning.” Through the precise adaptation and functional collaboration of technical methods at each level, a complete technical link from facial data acquisition to cleaning path output is constructed, providing a pre-foundation for the potential application of facial cleaning robots.

Materials and methods

The core execution and perception units of this system consist of a six-axis general-purpose robotic arm (Realman RM65-6F ⁵ ), an end cleaning device, and an RGB-D depth camera. ⁶ In the experimental scenario shown in Figure 1, the six-axis robotic arm is the main motion execution body of the system. The RGB-D depth camera and cleaning device are assembled at the end of the robotic arm to perform data acquisition and cleaning trajectory execution on the simulated face model.

Figure 1.

Facial cleaning robot experimental scenario.

Figure 2(a) shows the core structure of the adopted end cleaning device, which is installed along the end flange of the robotic arm. The device is mainly composed of two parts: one is the device shell for support and protection, and the other is the core component for cleaning function–the cylindrical soft brush assembly. To realize the precise motion control of the end cleaning device by the robotic arm, the geometric center of the circular soft brush plane of the end cleaning device is taken as the coordinate origin to establish the Tool Coordinate System (TCP): the Z-axis is defined to be perpendicular to the circular soft brush plane, pointing to the outside of the device (opposite to the contact surface of the soft brush with the face); the X-axis and Y-axis are orthogonally distributed in the soft brush plane, together forming a right-hand rectangular coordinate system. This tool coordinate system is the core benchmark for the robotic arm to realize hand-eye coordinate transformation and pose adjustment.

Figure 2.

Experimental equipment: (a) cleaning device, (b) RGB-D camera, and (c) computer.

Figure 2(b) shows the Intel RealSense D435 depth camera, a high-performance RGB-D depth camera based on active stereo vision technology launched by Intel Corporation. The camera and the robotic arm adopt the “Eye-in-Hand” installation configuration.

The upper computer of the system is a general-purpose laptop (Figure 2(c)). Under the Windows 11 operating system, it communicates with the robotic arm through the TCP/IP protocol to complete the issuance of motion control commands; it connects with the RGB-D depth camera through the USB interface to collect RGB images and depth data; a virtual development environment is built based on Anaconda, ⁷ and the Python programming language ⁸ and related dependent libraries (such as Cupoch ⁹ that supports GPU-accelerated point cloud processing under Windows system) are used to realize the communication control, data processing and algorithm verification of the system.

Facial region division and trajectory planning method

As shown in Figure 3, the method proposed in this paper adopts a three-layer progressive architecture. The first layer is the data acquisition layer, which collects 2D facial images and 3D depth data through an RGB-D depth camera, completes 3D point cloud reconstruction of the human face combined with algorithms such as ICP registration and point cloud projection, and obtains the 3D spatial coordinates corresponding to the facial key points of MediaPipe Face Mesh. ¹⁰ The second layer is the facial region division layer. Based on the standard anatomical divisions of the human face, combined with curvature features and spatial adjacency, the optimized division of facial cleaning regions is completed to reduce the computational complexity of subsequent trajectory planning. The third layer is the motion planning layer, which pre-plans cleaning trajectory points based on facial key points, completes pose adjustment of trajectory points by integrating point cloud normal vector correction and kinematic reachability optimization to improve the fit between the end effector and the facial skin, and finally realizes continuous and smooth transition of position and attitude changes through an interpolation algorithm.

Figure 3.

Collaborative workflow of “Data Acquisition – Face Region division – Motion Planning.”

Data acquisition

The human face is a complex and irregular concave-convex curved surface. 2D image data alone cannot fully characterize its spatial geometric morphology, making it difficult to support the accurate planning and stable execution of the manipulator’s cleaning trajectory. Therefore, a highly reliable 3D facial model is the core prerequisite for the implementation of this system. In this paper, we adopt a multimodal data collaborative processing method integrating the MediaPipe Face Mesh facial mesh model, target-surrounding ICP registration-based 3D reconstruction, and 3D point cloud projection in the camera coordinate system. This method accomplishes 2D facial feature extraction, accurate 3D point cloud reconstruction, and spatial coordinate matching between MediaPipe facial key points and the 3D point cloud. Finally, the reconstructed 3D facial point cloud and the 3D MediaPipe facial key points are uniformly transformed into the manipulator base coordinate system, to provide standard input for the subsequent face region division layer and trajectory planning layer.

MediaPipe face mesh facial mesh model

MediaPipe Face Mesh is a lightweight, high-precision real-time 3D facial key point detection and mesh reconstruction model developed by Google LLC. Its core function is to estimate 468 dense 3D facial key points from monocular RGB image input, and then reconstruct a 3D triangular mesh that fits the facial contour. It is widely used in academic and engineering fields such as computer vision, human-computer interaction, 3D facial reconstruction, and expression recognition.

Taking the key points in the left eye region of the face detected by MediaPipe Face Mesh shown in Figure 4 as an example, each key point corresponds to a unique index number. These key points have high detection accuracy and dense spatial distribution, which can serve as a reliable benchmark for characterizing facial geometric structure and provide fundamental support for the subsequent facial region division and trajectory planning for cleaning tasks. For example, by selecting the key points with index numbers 33, 246, 161, 160, 159, 158, 157, 173, 133, 155, 154, 153, 145, 144, 163, 7, and 33, a closed curve (the blue line in the figure) can be formed by connecting these points to delineate the “eyeball” region. In the same way, regions such as the “forehead” and “nasal region” can be divided from the facial RGB image.

Figure 4.

Facial key points of MediaPipe face mesh (partial view).

Affected by the prediction mechanism of the MediaPipe Face Mesh model, even if the facial information in the input image is occluded or missing, the model still outputs complete prediction results of 468 key points on the 2D image plane. However, the reliability of the key points corresponding to the regions with missing facial information cannot be guaranteed. In this system, the RGB-D depth camera and the manipulator adopt the Eye-in-Hand installation configuration. During single-view acquisition, the information of face is incomplete, which will lead to a significant reduction in the reliability of the corresponding key points.

To obtain complete and high-precision facial key points, this paper plans three acquisition poses for the left face, frontal face, and right face, enabling the RGB-D depth camera to capture facial images from different angles at the corresponding poses, and then extract the corresponding MediaPipe Face Mesh key points under each viewpoint. For example, in the facial images acquired under the three poses shown in Figure 5, the red dots mark the pixel coordinates corresponding to the MediaPipe Face Mesh key points.

Figure 5.

Facial key points of MediaPipe face mesh acquired at three poses: (a) left face, (b) frontal face, and (c) right face.

3D reconstruction method based on target-surrounding ICP registration

Affected by hand-eye calibration errors, directly stitching the facial point clouds acquired at the three poses described in Section 3.1.1 will make it difficult to guarantee the accuracy and integrity of the stitched point cloud. To suppress the impact of hand-eye calibration errors on the accuracy of the 3D facial model, this paper adopts a target-surrounding oriented Point-to-Plane ICP registration method, to unify the complete facial model into the same camera coordinate system, and then transform the facial model into the manipulator base coordinate system through the hand-eye calibration transformation matrix.

The Point-to-Plane ICP registration algorithm takes the distance from a point in the source point cloud to the corresponding fitted plane of the target point cloud as the error metric function. It takes two frames of point clouds as input and outputs the rigid transformation matrix between the two-point cloud frames. In this paper, the coincidence degree of point cloud matching, the mean square error of point pairs, and the rationality of the transformation matrix are taken as the validity criteria for registration results. When the camera acquires data around the target, the smoothness of the surrounding motion has a significant impact on the accuracy of ICP registration. Therefore, this paper takes the three poses of frontal face, left face, and right face described in Section 3.1.1 as path anchor points, and controls the manipulator to pass through each anchor point in sequence at a constant speed to realize smooth surrounding acquisition motion.

While the manipulator starts the surrounding motion, the RGB-D depth camera synchronously acquires image and point cloud data frames in real time. The overall workflow of the facial point cloud reconstruction system is shown in Figure 6, which mainly includes five core steps:

• (A) Global Reference Initialization. After the system reads the continuously acquired frame sequence (including point cloud frames and RGB image frames), the first frame point cloud is set as the reference frame of the global coordinate system, and the initial cumulative transformation matrix T_cum is set as a 4 × 4 identity matrix. Meanwhile, this frame is initialized as the target point cloud for subsequent registration and serves as the initial value of the global fused point cloud.

• (B) Local Point-to-Plane ICP Registration. As the frame sequence progresses, the algorithm takes the newly input current frame as the source point cloud, and adopts the Point-to-Plane ICP algorithm to solve the relative pose transformation matrix between the source point cloud and the target point cloud of the previous frame.

• (C) Pose Transformation and Registration Quality Evaluation. To avoid global map drift caused by registration failure, the system performs registration validity verification after each ICP iteration, with verification indicators including the singularity of the transformation matrix, point cloud coincidence error, etc. If the registration of the current frame is invalid, the frame is directly discarded to ensure the robustness of the system. If the registration is valid, the current relative transformation matrix is right-multiplied to the cumulative transformation matrix T_cum, and the source point cloud is uniformly transformed into the global reference coordinate system through the updated T_cum.

• (D) Incremental Point Cloud Fusion and Filtering Optimization. After stitching the current frame point cloud transformed into the global reference coordinate system into the global fused point cloud, to suppress the unlimited growth of point cloud density during the acquisition process and eliminate sensor noise, the system sequentially performs Voxel Down sampling and Statistical Outlier Removal on the fused global point cloud. The optimized point cloud can maintain uniform spatial resolution while removing outliers. After the above operations are completed, the current frame is updated to the target point cloud for the next iteration.

• (E) Pose Extraction for Specific Physiological Viewpoints. During the traversal process of incremental mapping, if the manipulator pose corresponding to the current frame is at the preset path anchor point position, the camera viewpoint at this time corresponds to the frontal face, left face, or right face. The algorithm independently records and saves the RGB image and the global cumulative transformation matrix at this moment, which are used for the subsequent extraction of MediaPipe Face Mesh facial key points and the 2D-to-3D coordinate mapping of the key points.

Figure 6.

Overall workflow of the facial point cloud reconstruction system.

Figure 7 shows the global fused point cloud obtained by the above 3D reconstruction method. Existing studies have shown that, in the scenario of facial point clouds acquired by consumer-grade depth cameras such as the Intel D435, the fine registration strategy based on Point-to-Plane ICP can achieve a root mean square error of translation of 0.8∼1.5 mm, with the rotation error controlled within 0.2°–0.5°. ¹¹ Meanwhile, combined with the official depth noise evaluation standard of the device, the overall point-to-plane registration root mean square error (RMSE) within the working range can be stably maintained between 2.0 and 3.5 mm, ¹² which fully meets the accuracy requirements of facial cleaning tasks for facial model reconstruction.

Figure 7.

3D facial reconstruction via target-surrounding ICP registration.

3D localization of 2D key points from MediaPipe face mesh

The facial key points detected by the MediaPipe Face Mesh model from RGB images are in the 2D pixel coordinate system. To realize manipulator trajectory planning based on facial key points, it is necessary to first determine the 3D spatial coordinates corresponding to each key point. This paper adopts the following method to realize the 3D coordinate localization of key points: the global fused point cloud described in Section 3.1.2 is transformed into the camera coordinate system corresponding to the RGB image (i.e. the camera coordinate systems corresponding to the three surrounding anchor points described above). Then, based on the pixel coordinates of the 2D key points output by MediaPipe Face Mesh, the corresponding 3D spatial position of the key point in the global fused point cloud is matched and determined.

Taking the camera pose corresponding to the acquisition of the “frontal face” image as an example, let T_{cum_F} be the cumulative transformation matrix from the camera coordinate system of the frontal face image to the global reference coordinate system of ICP registration (the camera coordinate system of the first frame), and FusedPointCloud be the global fused point cloud in the global reference coordinate system. Then the global fused point cloud FusedPointCloud_F in the camera coordinate system corresponding to the frontal face image can be solved by equation (1):

$${\mathrm{Fused}\mathrm{Point}\mathrm{Cloud}}_{\mathrm{F}}={T_{\mathrm{cum}\_F}}^{-1}·\mathrm{Fused}\mathrm{Point}\mathrm{Cloud}$$

(1)

Equation (2) is the homogeneous coordinate projection formula of the pinhole camera model, ¹³ where (X,Y,Z) denote the spatial coordinates of a 3D point in the camera coordinate system; (f _x ,f_y) are the focal lengths of the camera intrinsic parameters, converted from the physical focal length of the camera and the pixel size; (c _x ,c _y ) are the coordinates of the image principal point, which correspond to the intersection of the camera optical axis and the image plane and are usually located in the central area of the image; (u,v) are the projected 2D pixel coordinates (in the pixel coordinate system, the coordinate origin is located at the upper left corner of the image).The point cloud in the camera coordinate system can be projected onto the pixel coordinate system through equation (2). Figure 8 shows the facial RGB image and the point cloud projection results under the same camera coordinate system, and the corresponding facial key points (red dots in the figure) are marked based on the pixel coordinates of the key points output by MediaPipe Face Mesh.

$$\mathrm{Z}\left[\begin{array}{c}u\hfill \\ \multicolumn{1}{c}{v}\\ \multicolumn{1}{c}{1}\end{array}\right]=\left[\begin{array}{ccc}{f}_x\hfill & 0\hfill & c_x\hfill \\ \multicolumn{1}{c}{0}& f_y\hfill & c_y\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 1\hfill \end{array}\right]\left[\begin{array}{c}\mathrm{X}\hfill \\ \multicolumn{1}{c}{\mathrm{Y}}\\ \multicolumn{1}{c}{\mathrm{Z}}\end{array}\right]$$

(2)

Figure 8.

RGB image and point cloud projection image under the same camera coordinate system.

The above method establishes the spatial mapping correlation between 2D facial key points and the global fused point cloud in the pixel coordinate system. To further accurately locate the 3D spatial coordinates of facial key points, this paper adopts a 3D key point refinement strategy based on spatial index and depth median filtering (Figure 9), to extract the optimal 3D spatial coordinates corresponding to 2D facial key points from the global fused point cloud. The specific implementation logic is as follows:

• (A) 2D Spatial Index Construction. The algorithm first receives the global fused point cloud and its projected coordinate set on the 2D pixel plane. To avoid the high computational overhead caused by subsequent massive point cloud retrieval, a 2D KD-Tree ¹⁴ is constructed with the 2D projected coordinates of all 3D points as the basic data. This data structure can significantly optimize the efficiency of spatial nearest neighbor query in the 2D plane.

• (B) Fast Local Neighborhood Retrieval. Taking the 2D facial key point set as the target, the algorithm traverses each 2D key point sequentially, and performs fixed-radius (e.g. r = 3.0 pixels) nearest neighbor search in the pre-constructed KD-Tree. This step extracts all projected points within a certain pixel tolerance range around the target key point, which is used as the candidate set for subsequent 3D mapping.

• (C) Depth Sorting and Median-Based Anti-Noise Processing. Limited by the imaging principle of depth cameras, “Flying Pixels” or depth jump noise often occur in edge or highly reflective regions. Directly using the nearest neighbor projected point can easily lead to the failure of 3D coordinate extraction. The algorithm inversely maps the 2D candidate point set back to the 3D point cloud in the camera coordinate system, and extracts its Z-axis depth information. By sorting the local candidate point cloud in ascending order of depth values and strictly selecting the median point of the sorted results as the optimal matching point, the system can effectively eliminate foreground occlusion points and background penetration points with abnormal depth, and significantly enhance the robustness of the algorithm to the physical noise of the sensor.

• (D) Accurate Reconstruction of 3D Key Points. The median point obtained after anti-noise filtering is identified as the accurate 3D spatial position of the MediaPipe Face Mesh facial key points (Figure 10). These 3D key points will provide reliable local geometric anchors for the subsequent facial region division, curvature calculation, and robot end-effector pose alignment.

Figure 9.

3D key point refinement algorithm based on spatial index and depth median filtering.

Figure 10.

3D key points of MediaPipe face mesh in the global point cloud.

Coordinate system transformation

Finally, to meet the coordinate system requirements of manipulator trajectory planning, it is necessary to uniformly transform the global fused point cloud and the 3D facial key points of MediaPipe Face Mesh in the camera coordinate system to the manipulator base coordinate system. The essence of manipulator trajectory planning is to solve the motion path in joint space or Cartesian space under its base coordinate system (Base Frame, {B}). In this paper, the camera and the manipulator adopt the “Eye-in-Hand” installation configuration, that is, the camera is fixed on the end effector of the manipulator and moves synchronously with the end. The point cloud collected by the camera and the extracted key points are all located in the camera coordinate system with the optical center of the camera as the origin, and there is a definite rigid transformation link with the manipulator base coordinate system. Without the unified transformation of the coordinate system, the manipulator cannot accurately reproduce the spatial position of the face perceived by vision.

The core of the Eye-in-Hand configuration is to clarify the rigid transformation relationship between the three coordinate systems: the manipulator base coordinate system {B}, the end-effector frame {E} of the manipulator, and the camera frame {C}. The camera is fixed on the end effector, and the relative pose between the two is constant, that is, the hand-eye calibration matrix $T_{\mathrm{C}}^{\mathrm{E}}$ from the camera coordinate system {C} to the end effector coordinate system {E} is a fixed value, which can be obtained in advance through hand-eye calibration experiments. ¹⁵ For any 3D point P_C = (X_C,Y_C,Z_C)^T in the camera coordinate system, its coordinate P_B = (X_B,Y_B,Z_B)^T in the manipulator base coordinate system needs to be mapped through the two-stage rigid transformation of “camera coordinate system → end effector coordinate system → base coordinate system.” The core formula based on homogeneous coordinate transformation is as follows:

$$P_{\mathrm{B}}^{\mathrm{h}}=T_{\mathrm{E}}^{\mathrm{B}}·T_{\mathrm{C}}^{\mathrm{E}}·P_{\mathrm{C}}^{\mathrm{h}}$$

(3)

In the formula, $P_{\mathrm{C}}^{\mathrm{h}}$  = (X_C,Y_C,Z_C,1)^T and $P_{\mathrm{B}}^{\mathrm{h}}$  = (X_B,Y_B,Z_B,1)^T are the homogeneous coordinates corresponding to the spatial points in the camera coordinate system and the manipulator base coordinate system, respectively; ${\mathrm{T}}_{\mathrm{E}}^{\mathrm{B}}$ is the homogeneous transformation matrix from the end effector coordinate system {E} to the base coordinate system {B}, obtained through the forward kinematics solution of the manipulator.

Facial region division

The primary prerequisite for the manipulator to perform precise cleaning actions on the human face is to construct a facial geometric model suitable for motion planning. Since the human face presents the characteristics of a high-curvature, irregular and complex curved surface, direct path planning based on the global curved surface will easily lead to a sharp increase in computational complexity, and it is difficult to guarantee the cleaning accuracy of local areas. For this reason, based on the anatomical characteristics of the human face, this paper carries out hierarchical division and optimization of facial regions combined with the requirements of cleaning tasks, aiming to simplify the difficulty of map model construction and reduce the computational complexity of subsequent motion planning.

Method for obtaining curvature features of facial sub-regions

In view of the operation requirements of the facial cleaning scenario, the human face is an irregular complex curved surface. Extracting the curvature features of each region of the face can not only quantitatively evaluate the complexity of the curved surface, but also provide a theoretical basis for the analysis of the attitude change amplitude during the operation of the end cleaning device. For this reason, this paper designs a method for extracting curvature features of facial regions for cleaning tasks, the specific process is as follows:

Section 3.1.3 has described the method of projecting the global fused point cloud to the pixel coordinate system where the MediaPipe Face Mesh facial key points are located. Based on this method, the facial key points can be used as the benchmark markers for region division to realize the accurate segmentation of facial sub-regions and the calculation of point cloud curvature features within the regions. The specific implementation logic is:

• (1) Delineate the target facial sub-region, and select the corresponding facial key points to enclose the boundary of the sub-region in the above pixel coordinate system;

• (2) Adopt the scan line filling algorithm, ¹⁶ construct the polygon region contour with the selected key points as vertices, and extract all pixel coordinates inside the contour;

• (3) Based on the pixel coordinates inside the contour, match all 3D points in the global fused point cloud projected into this region. Since a single pixel coordinate may correspond to multiple 3D points, only the point with the smallest Z coordinate is retained for each pixel coordinate. The point cloud set is obtained by traversing all pixels, and after outlier elimination, it finally constitutes the surface point cloud corresponding to the facial sub-region (the minimum value of the Z coordinate corresponds to the facial surface point closest to the camera in the camera coordinate system, which can accurately characterize the real geometric shape of the facial curved surface).

This paper adopts a method combining Principal Component Analysis (PCA) and local quadric surface fitting to extract multi-dimensional curvature features from the surface point cloud of facial sub-regions. The specific steps are as follows:

First, the dispersion degree of the regional point cloud is quantified based on the PCA method. ¹⁷ By calculating the covariance matrix of the local point cloud and performing eigenvalue decomposition, the eigenvalues sorted in ascending order λ1 ≤ λ2 ≤ λ3 are obtained; the global surface variation rate of the sub-region point cloud is calculated based on the ratio of the minimum eigenvalue to the sum of eigenvalues. This dimensionless index can effectively reflect the scattering thickness of the local point cloud in the normal direction and the degree of deviation from the plane. The closer the value is to 0, the flatter the regional surface is.

Secondly, to extract continuous differential geometric features from discrete point clouds, this paper introduces local quadric surface fitting technology. ¹⁸ To ensure the numerical stability of Ordinary Least Squares (OLS) fitting, the algorithm first constructs a rotation matrix based on the principal normal vector obtained by PCA solution, and strictly aligns the normal direction of the local point cloud to the Z-axis of the local coordinate system; after alignment, the design matrix is constructed to solve the coefficients of the quadratic polynomial surface equation. Z(X,Y) = aX ²  + bY ²  + cXY + dX + eY + f.

Finally, based on the classical differential geometry theory, ¹⁹ at the pole of the quadratic fitting surface (i.e. the origin of the local coordinate), the coefficients of the first fundamental form and the second fundamental form of the surface are obtained by calculating the partial derivatives, and then the Gaussian curvature and mean curvature of the region are analytically obtained. Among them, the sign of Gaussian curvature can accurately define the topological morphology of the target region (such as saddle surface, convex spherical surface, cylindrical surface), and the mean curvature can quantify the overall bending intensity of the surface. The above three geometric features complement each other, constructing a geometric evaluation model that can fully characterize the complex physiological structure of the human face, and can accurately distinguish different facial regions such as the flat forehead, high-curvature eye socket and nasal region.

Facial region division based on curvature features

Referring to the standard anatomical division of the human face in Cosmetic Applied Anatomy, ²⁰ the human face is divided into 11 different basic anatomical parts: forehead, nasal region, labial region, mental region, buccal region, infraorbital region, orbital region, temporal region, zygomatic region, mandibular region, and ear region (Figure 11(a)). According to the symmetrical characteristics of the human face, the buccal region, infraorbital region, orbital region, temporal region, zygomatic region, mandibular region, and ear region have left and right divisions, so the number of facial divisions based on anatomy is 18 in total. Due to the special structure of the ear region, it is not discussed in this paper, so the final number of anatomical-based facial divisions studied in this paper is 16. This division method can fully cover all physiological structure regions of the human face, and provide an anatomical basis for subsequent optimized division. At the same time, the core curvature parameters (global variation, mean curvature H, Gaussian curvature K) of different anatomical parts are obtained through the curvature feature detection method described in Section 3.2.1, and the specific data are shown in Table 1 (due to the left-right symmetry of the human face, only 11 divisions are listed in the table), which provides data support for the definition of curvature threshold and complexity verification of division optimization. Appendix 1 shows the projection of the anatomical facial divisions in the pixel coordinate system and the point cloud surface of the anatomical facial divisions in the global fused point cloud of the human face.

Figure 11.

Two types of facial region division: (a) eleven anatomical subregions of the human face and (b) facial region division for facial cleaning tasks.

Table 1.

Curvature parameters of 11 basic facial anatomical regions.

Anatomical region	Global variation	Mean curvature, H	Gaussian curvature, K
Forehead	0.01515	13.12682	168.8362
Nasal region	0.04393	−0.47799	−701.76
Labial region	0.027214	20.39519	350.3013
Mental region	0.018314	40.57551	1312.613
Buccal region	0.007732	8.727581	32.82917
Infraorbital region	0.005475	16.84398	−68.2287
Orbital region	0.019607	2.211919	−410.972
Temporal region	0.007264	5.047944	−9.73336
Zygomatic region	0.007262	10.99554	112.8479
Mandibular region	0.002333	1.825054	−39.3672

On this basis, combined with the operation requirements of facial cleaning tasks on regional curvature characteristics, spatial distribution and structural complexity, this paper first defines and quantifies the curvature threshold based on the above curvature statistical data, and then merges, splits and optimizes the 11 basic anatomical parts for cleaning tasks. The specific scheme is as follows:

First, clarify the curvature threshold system for division optimization. Based on the curvature statistical characteristics of the 11 basic anatomical regions in Table 1, combined with the core requirements of robotic arm cleaning path planning for surface smoothness and pose stability, 3 core curvature thresholds are set as the quantitative division basis for high and low curvature regions (Table 2): global variation threshold T_GV = 0.018, absolute value threshold of mean curvature T_|H| = 15.0, absolute value threshold of Gaussian curvature T_|K| = 200.0. Among them, the region that meets any of the conditions of “global variation > T_GV or |mean curvature| > T_|H| or |Gaussian curvature| > T_|K|” is defined as a high-curvature complex region; the region that meets all the conditions of “global variation ≤ T_GV and |mean curvature|≤ T_∣H∣ and |Gaussian curvature|≤ T_∣K∣” is defined as a low-curvature simple region. The setting of this threshold is based on the actual distribution of curvature in each anatomical region of the face, determined through statistical analysis (mean + 1 standard deviation), to ensure accurate distinction of the complexity of the curved surface and provide a quantitative basis for division optimization.

Table 2.

Curvature thresholds and their determination basis for facial region division optimization.

Curvature index	Threshold value	Classification criterion
Global variation (GV)	T GV = 0.018	GV > 0.018 is high curvature, GV ≤ 0.018 is low curvature
Absolute value of mean curvature	T |H| = 15.0	|H| > 15.0 is high curvature, |H| ≤ 15.0 is low curvature
Absolute value of gaussian curvature	T |K| = 200.0	|K| > 200.0 is high curvature, |K| ≤ 200.0 is low curvature

The threshold is determined by “mean + 1 standard deviation,” and the core basis is as follows: the curvature data of each anatomical region of the face conforms to the approximate normal distribution characteristics. The mean can characterize the normal curvature level of the regional surface, and the standard deviation can quantify the discrete fluctuation degree of curvature. Setting the threshold based on the statistical characteristics of the measured curvature data can avoid the subjective randomness of manual setting and ensure the objectivity and reproducibility of the division method. At the same time, the selection of the 1 standard deviation multiple is the optimal trade-off combined with the needs of facial cleaning tasks: this threshold can fully cover the normal fluctuation range of low-curvature flat regions, and accurately identify high-curvature complex structures that deviate from the normal state.

Based on the above curvature thresholds and division criteria, combined with the spatial adjacency of facial anatomical regions, the optimization strategy for the 11 basic anatomical parts in this paper is as follows:

First, high-curvature complex regions are divided separately. According to the threshold judgment, the orbital region (|K| = 410.97 > 200.0) and nasal region (GV = 0.044 > 0.018, |K| = 701.76 > 200.0) are both high-curvature complex regions. Both have significant local curvature changes and many complex geometric structures such as depressions and protrusions. If merged with other low-curvature regions for planning, it is easy to cause missing detection of the cleaning path or deviation of attitude adjustment. Therefore, the two are treated as independent regions separately to ensure the cleaning accuracy of complex regions.

Second, the large-area low-curvature region is divided independently. The forehead (GV = 0.015 ≤ 0.018, |H| = 13.13 ≤ 15.0, |K| = 168.84 ≤ 200.0) belongs to the low-curvature region. It is located directly in front of the face, with the characteristics of large area and gentle overall curvature change, and its spatial position has obvious boundaries with the temporal region on the side, the orbital region in front, and the nasal region below. Dividing it into a separate region can simplify the path planning logic of this region, and avoid mutual interference with the planning parameters of surrounding complex regions.

Third, small adjacent low-complexity regions are merged. According to the threshold judgment, the infraorbital region, zygomatic region, and buccal region all belong to low-curvature simple regions. Among them, the infraorbital region (GV = 0.005, |H| = 16.84 ≈ 15.0, |K| = 68.23 ≤ 200.0), part of the zygomatic region (flat region near the infraorbital, GV = 0.007, |H| = 11.00 ≤ 15.0, |K| = 112.85 ≤ 200.0) and part of the buccal region (flat region near the frontal face, GV = 0.008, |H| = 8.73 ≤ 15.0, |K| = 32.83 ≤ 200.0) are adjacent in position, and are merged into the “infraorbital-zygomatic-buccal combined region” (referred to as the “infraorbital region”), which can reduce the number of divisions and the planning workload. The labial region (GV = 0.027 > 0.018, |H| = 20.40 > 15.0, |K| = 350.30 > 200.0) and the mental region (GV = 0.018 ≤ 0.018, |H| = 40.58 > 15.0, |K| = 1312.61 > 200.0) are closely connected in space, and are merged into the “labial-mental combined region” to further simplify the division structure.

Fourth, the contiguous areas on the side are uniformly classified. The temporal region (GV = 0.007, |H| = 5.05 ≤ 15.0, |K| = 9.73 ≤ 200.0), part of the zygomatic region (area near the side), mandibular region (GV = 0.002, |H| = 1.83 ≤ 15.0, |K| = 39.37 ≤ 200.0) and buccal region (lateral area) are all located on the side of the human face, and are all low-curvature simple regions, presenting the characteristics of gentle curvature change and contiguous areas as a whole. They are uniformly classified into the “lateral facial combined region,” which can realize efficient coverage of the lateral region through a unified planning strategy.

Through the above division optimization based on anatomical features, curvature quantitative threshold and cleaning task requirements, this paper finally integrates the 16 anatomical sub-regions into 9 cleaning divisions suitable for the robotic arm facial cleaning task (Figure 11(b)), namely: forehead, nasal region, left/right orbital region, left/right infraorbital region, labial-mental combined region, left/right lateral facial combined region, which can fully cover all target areas of facial cleaning operations. Table 3 compares the curvature characteristics of the optimized cleaning divisions and the original anatomical divisions. The results show that the merged lateral facial combined region and infraorbital-zygomatic-buccal combined region all meet the criteria for low-curvature simple regions; although the overall flatness of the labial-mental combined region slightly exceeds the threshold, its overall surface complexity is controllable, and the cleaning operation can be completed through a unified planning strategy. Appendix 2 shows the projection results of the above three combined divisions in the pixel coordinate system, as well as their corresponding point cloud surfaces in the global fused point cloud of the human face.

Table 3.

Curvature characteristics of the optimized cleaning subregions.

Anatomical region	Global variation	Mean curvature, H	Gaussian curvature, K
Lateral facial combined region	0.003634	8.544963	53.298948
Infraorbital-zygomatic-buccal combined region	0.012372	14.781352	166.270481
Labial-mental combined region	0.054181	12.184245	−7.807681

Impact of regional division on path planning complexity

For the facial cleaning scenario, the purpose of path planning is to make the cleaning device perform full coverage with friction effect on the face. To achieve this, it is necessary to set entry and exit points at the boundaries of each region to complete the smooth connection between regions, and carry out coverage trajectory planning inside the region. The more regions there are, the more entry-exit connection constraints are required, the more complex the attitude switching and path transition between regions are, and the higher the overall planning complexity is. For each cleaning region, the more diverse the surface curvature changes, the more diverse the trajectory attitude changes required to achieve coverage. Therefore, the path planning complexity is defined as the connection complexity brought by the number of regions plus the coverage complexity brought by the uneven curvature within the regions, and the mathematical model is:

$$C_{\mathrm{total}}=\alpha N+\beta \sum _{i=1}^N{\sigma }_i^2,(\alpha >0,\beta >0)$$

(4)

Where N is the total number of regions, ${\sigma }_i^2$ is the curvature variance of the i-th region, α > 0, β > 0 are complexity coefficients. The first term reflects the connection complexity between regions, which is proportional to the number of regions; the second term reflects the coverage complexity within the region, which is determined by the uniformity of curvature. When considering distinguishing left and right regions according to the symmetrical characteristics of the human face (excluding the ear region), the number of anatomical-based facial sub-regions is 16, then:

$$C_1=16\alpha +\beta ({\sigma }_1^2+{\sigma }_2^2+\cdots {\sigma }_{11}^2),(\alpha >0,\beta >0)$$

(5)

Through the facial region planning described in Section 3.2.2, the regions with similar curvature and spatial adjacency in the anatomical-based facial sub-regions are merged, which not only reduces the number of divisions to 9, but also makes the curvature within the merged new regions more uniform, and its complexity is:

$$\begin{array}{c}\multicolumn{1}{c}{C_2=9\alpha +\beta ({\sigma }_{\mathrm{new}1}^2+{\sigma }_{\mathrm{new}2}^2+\cdots {\sigma }_{\mathrm{new}9}^2),}\\ \multicolumn{1}{c}{(\alpha >0,\beta >0)}\end{array}$$

(6)

Therefore, combined with the curvature situation described in Section 3.2.1, it is obvious that C2 < C1, that is, after the facial region division based on curvature features, the path planning complexity will be effectively reduced.

Motion planning

For the robotic arm facial cleaning task, the core goal of path planning is to construct a motion trajectory that meets two key indicators: one is global coverage, to ensure that the end cleaning device can traverse all target cleaning regions of the face; the other is the end-surface fit, the fit degree between the end cleaning device and the facial skin directly determines the cleaning effect, and the higher the fit degree, the better the cleaning efficiency and quality. Based on this goal, the path planning process needs to solve two groups of core technical requirements and form a progressive technical logic. The first is the attitude adaptation requirement: to achieve the close fit between the end cleaning device and the facial curved surface, it is necessary to adjust the attitude of the robotic arm to make the action surface of the end device (soft brush plane) tangent to the local facial curved surface at the trajectory point, that is, the Z-axis of the end device (tool coordinate system) is consistent with the normal vector direction of the local curved surface. The second is the kinematic reachability requirement: limited by the hardware constraints such as the degrees of freedom and motion range of the robotic arm joints, some ideal poses that meet the “attitude adaptation” may exceed the kinematic reachable space of the robotic arm. Therefore, it is necessary to seek a balance between the “fitting requirement” and the “reachable constraint,” and solve the optimal reachable pose at the trajectory point (which not only meets the fitting degree to the maximum extent, but also conforms to the kinematic constraints of the robotic arm).

Trajectory point pre-planning

3D modeling provides a global “map” for facial cleaning tasks, facial semantic division realizes the modular management of operation areas, and the 3D key points generated by MediaPipe Face Mesh provide accurate topological “coordinates” for this “map.” The initial stage of path planning directly takes these topological nodes as the basis, and extracts their subsets as the pre-planned physical trajectory points. To ensure the surface coverage of the cleaning task, this paper adopts a high-density retention strategy in each facial sub-region. Combined with the large effective contact area of the end cleaning device, this strategy can use the physical footprint of the tool itself to naturally bridge the gap between points, thereby realizing the coverage of the operation area.

As shown in Figure 12(a), this paper extracts the key points of the specified sequence in the “forehead” region of the RGB facial image as the initial trajectory anchor points; Figure 12(b) further shows the position distribution of these key points in 3D space. However, the 3D trajectory points extracted at this stage only contain the position parameters (X,Y,Z) in Cartesian space, and lack the attitude information matching the facial curvature, which cannot be directly used to drive the end effector of the robotic arm to complete the cleaning operation. Therefore, the prerequisite for effective cleaning is to further carry out attitude solving for these discrete spatial points.

Figure 12.

Pre-planned 2D and 3D trajectory points: (a) pre-planned 2D trajectory points and (b) pre-planned 3D trajectory points.

Extraction and normalization of local surface normal vector of trajectory points

The working end of the end cleaning device shown in Figure 2(b), which is in direct contact with the facial skin, has a circular plane structure. For this planar cleaning actuator, the efficiency and effect of the cleaning operation are significantly positively correlated with the fit degree between the cleaning device and the facial curved surface. Therefore, to achieve the maximum fit between the cleaning device and the facial skin, it is necessary to carry out targeted optimization and adjustment of the end effector attitude at each point of the cleaning trajectory.

The core basis for attitude adjustment is the normal vector of the facial local curved surface at the trajectory point, and the accurate extraction and direction unification of the normal vector need to be realized through the following steps:

• (1) Local point cloud set construction: Adopt the dynamic neighborhood expansion method ²¹ (equation 8), take the target trajectory point c as the center, and construct a spherical neighborhood (satisfying the L2 norm constraint (equation (7))) by linearly increasing the neighborhood radius r, until the number of point clouds in the neighborhood reaches the preset threshold, to ensure that the local point cloud can fully characterize the surface morphology near the trajectory point and avoid the deviation of normal vector calculation caused by insufficient samples;

$$\Vert \mathit{p}-\mathit{c}{\Vert }_2=\sqrt{(p_x-c_x{)}^2+(p_y-c_y{)}^2+(p_z-c_z{)}^2}$$

(7)

$$N\left(r\right)=\{\mathit{p}\in P|\Vert \mathit{p}-\mathit{c}{\Vert }_2\le r\}$$

(8)

• (2) PCA normal vector extraction: Perform principal component analysis on the local point cloud set: first, eliminate the influence of the spatial position offset of the trajectory point on the calculation result through decentralization processing (equation (9)), then construct the point cloud covariance matrix (equation (10)), and obtain the eigenvalues and eigenvectors by solving the characteristic equation (equation (11)). Since the local point cloud is approximately distributed on a 2D curved surface, the eigenvector direction corresponding to its minimum eigenvalue is the normal vector direction of the local curved surface, denoted as v ;

$${\mathit{p}}_{\mathit{c}}=\mathit{p}-\mathit{c}(\mathit{p}\in P)$$

(9)

$$C=\frac{1}{n}\sum _{i=1}^n{\mathit{p}}_{\mathit{c}}·{\mathit{p}}_{\mathit{c}}^T$$

(10)

$$C\mathit{v}=\lambda \mathit{v}$$

(11)

• (3) Normal vector direction normalization: The extracted normal vector has two directions: “from the outside to the face” and “from the face to the outside.” To ensure the consistency of the attitude adjustment of the end device, the Z-axis of the camera coordinate system is taken as the auxiliary reference vector (associated with the spatial orientation of the cameras corresponding to the three acquisition poses described in Section 3.1.1, i.e. Zc ), by calculating the dot product of the normal vector v and the reference vector Zc (equation (12)): if the dot product is positive, it means that the direction of v conforms to “from the outside to the face,” and it is directly retained; if the dot product is negative, v is reversed, and finally the normalization of the normal vector direction of all trajectory points is realized. Figure 13 shows the local point cloud set (red) of the trajectory points planned on the “forehead” and its normal vector (white lines). Figure 14 shows the fitting effect between the end cleaning device and the human face after adjusting the Z-axis of the tool coordinate system of the end cleaning device of the robotic arm according to the normal vector.

$${\mathit{v}}_{\mathbf{adj}}=\{\begin{array}{c}\mathit{v}(\mathit{v}·{\mathit{Z}}_{\mathit{c}}>0)\hfill \\ \multicolumn{1}{c}{-\mathit{v}(\mathit{v}·{\mathit{Z}}_{\mathit{c}}<0)}\end{array}$$

(12)

Figure 13.

Local point cloud and normal vector of trajectory points.

Figure 14.

Comparison before and after pose adjustment: (a) vertical pose and (b) normal vector-based pose.

Trajectory reachability judgment method

For each pre-planned trajectory point, aligning the Z-axis direction of the tool coordinate system of the end cleaning device with the local point cloud normal vector of the trajectory point solved in Section 3.3.2, the corresponding end attitude is defined as the ideal fitting attitude (denoted as o_ideal); the manipulator pose corresponding to the acquisition of the RGB image of MediaPipe Face Mesh facial key points is defined as the initial attitude (denoted as o_init). Combining the initial attitude with the 3D spatial position of the corresponding trajectory point, the initial pose of the trajectory point (po_init) is obtained; integrating the initial poses of all trajectory points, the initial pose sequence (PO_init) can be constructed. Inputting the initial pose sequence into the inverse kinematics solving function of the manipulator (equation (13)) bit by bit can verify the kinematic reachability of the initial pose (i.e. whether there is a valid inverse solution). Similarly, combining the ideal fitting attitude with the spatial position of the corresponding trajectory point, the ideal pose of the trajectory point (po_ideal) is obtained; integrating the ideal poses of all trajectory points, the ideal pose sequence (PO_ideal) can be constructed.

The core reasons why the ideal pose sequence cannot be directly used as the core of the cleaning trajectory include three points: (1) There are poses in the pose sequence located in the singular configuration region of the manipulator, corresponding to no solution for inverse kinematics; (2) For the facial cleaning scenario, the end cleaning device needs to be close to the face, and large-angle jumps in the joint space are prone to occur between the ideal poses po_{ideal_i} and po_{ideal_i + 1} of adjacent trajectory points, causing the manipulator configuration to flip, and the flipping process is easy to make the end cleaning device detach from the face; (3) The trajectory point density of the ideal pose sequence is low, and directly driving the manipulator will lead to stiff motion, poor continuity and excessive joint speed, which cannot meet the smoothness requirements of facial cleaning operations. In response to the above problems, this paper takes the existence of the inverse kinematics solution and the consistency of the manipulator configuration as the core judgment criteria, and finds the optimal reachable pose o_reachable between o_init and o_ideal through the dichotomy method; integrating the reachable poses of all trajectory points to construct the reachable pose sequence po_reachable, and finally smoothing the sequence through an interpolation algorithm to generate a continuous and smooth cleaning motion trajectory.

Equation (13) is the inverse kinematics solving function of the manipulator, which is a standard solving method built into the manipulator controller used in this paper. Jseed is the joint seed of the inverse solution function, whose core function is to constrain the solution space of the inverse solution, so that the solver selects the solution with the smallest deviation from the joint configuration corresponding to the joint seed angle as the output from multiple sets of analytical solutions of inverse kinematics; po_input is the pose to be solved, and J is the solved joint angle sequence. For the six-axis manipulator used in this paper, the joint angle sequence is J = (joint1,joint2,joint3,joint4,joint5,joint6). If the solution fails, it is determined that the manipulator cannot smoothly move from the joint configuration corresponding to Jseed to the pose po_input to be solved.

$$J=\mathrm{IK}(\mathrm{Jseed},{\mathrm{po}}_{\mathrm{input}})$$

(13)

The configuration continuity judgment is mainly to judge whether wrist flip, elbow flip and shoulder flip occur when the manipulator changes from one pose to another. Based on the geometric characteristics of the six-axis manipulator used in this paper (satisfying the Pieper criterion and having the classic Spherical Wrist characteristics, the length of the large arm L₁ = 0.256 m, the length of the small arm L₂ = 0.21 m), a configuration judgment algorithm is designed.^22,23

• (1) Wrist flip judgment mark: The end of the manipulator in this system adopts the standard Spherical Wrist design, and the axes of its fourth, fifth and sixth joints intersect at the center point of the wrist. The sign of the wrist configuration depends entirely on the sign of the fifth joint (joint5). When joint5 = 0, the axes of the fourth and sixth joints are collinear, and the manipulator is at the wrist singular point. Therefore, the wrist configuration mark is defined as:

$$S_{\mathrm{wrist}}=\mathrm{sgn}(\mathrm{joint}5)$$

(14)

If the $S_{\mathrm{wrist}}$ of two adjacent frames is reversed, it is determined that a wrist flip has occurred.

• (2) Elbow flip judgment mark: For the large arm and small arm connecting rods moving in the same plane, the elbow configuration (elbow up/elbow down) is determined by the internal included angle of the elbow joint, that is, the sign of the third joint (joint3). When joint3 = 0, the large arm and the small arm are fully extended and are at the internal singular boundary. Its configuration mark is defined as:

$$S_{\mathrm{elbow}}=\mathrm{sgn}(\mathrm{joint}3)$$

(15)

If the signs of $S_{\mathrm{elbow}}$ of two adjacent frames are different, it means that the manipulator has crossed the elbow singular boundary and an elbow flip has occurred.

• (3) Shoulder flip judgment mark: The geometric essence of shoulder flip is whether the center point of the wrist of the manipulator crosses the vertical plane where the base coordinate system (the rotation axis of the first joint, Z-axis) is located. In this paper, using the forward kinematics relationship of the large arm connecting rod L₁ and the small arm connecting rod L₂ in the rotation plane of the second joint, the directed horizontal projection radius $R_{\mathrm{horizontal}}$ of the wrist center relative to the central axis of the base is calculated:

$$R_{\mathrm{horizontal}}=L_1\cos (q2\mathbf{)}+L_2\cos (q2+q3\mathbf{)}$$

(16)

When $R_{\mathrm{horizontal}}$ >0, the wrist is in front of the base (Front); when $R_{\mathrm{horizontal}}$ <0, the wrist crosses the zenith of the base and enters the back (Back); when $R_{\mathrm{horizontal}}$ = 0, the wrist center hovers directly above the base and is at the shoulder singular point. Its configuration mark is defined as:

$$S_{\mathrm{shoulder}}=\mathrm{sgn}(R_{\mathrm{horizontal}})$$

(17)

For the joints of the manipulator in two frames, if the wrist configuration mark, elbow configuration mark and shoulder configuration mark are all consistent, it can be considered that the joint configurations of the two frames have consistency.

Solving the optimal reachable pose by dichotomy

For the initial pose and ideal fitting pose of a single pre-planned trajectory point, this paper first constructs a continuous attitude interpolation space based on Spherical Linear Interpolation (Slerp ²⁴ ), then uses unit quaternions to represent the end attitude of the manipulator, takes the initial pose o_init and the ideal pose o_ideal as the interpolation boundaries, and the spatial included angle θ between the two poses is defined as:

$$\theta =\arccos (o_{\mathrm{init}}·o_{\mathrm{ideal}})$$

(18)

In the formula, o _init o_ideal is the dot product operation of two unit quaternions. The attitude interpolation formula based on Slerp is:

$$\begin{array}{c}\multicolumn{1}{c}{o_{\mathrm{target}}\left(x\right)=\frac{o_{\mathrm{init}}\sin \left((1-x)\theta \right)+o_{\mathrm{ideal}}\sin \left(x\theta \right)}{\sin \theta }}\\ \multicolumn{1}{c}{(x\in [0,1])}\end{array}$$

(19)

In the formula, x is the interpolation weight, o_target (x) is the end attitude obtained by interpolation; when x = 0, the interpolation result corresponds to the initial attitude o_init, and when x = 1, the interpolation result corresponds to the ideal attitude o_ideal.

Combining the attitude with the 3D spatial position of the corresponding trajectory point, the initial pose po_init and the ideal pose po_ideal are constructed respectively. Based on the dual criteria of the existence of inverse kinematics solution and the consistency of manipulator configuration defined in Section 3.3.3, the reachability pre-verification of the ideal pose po_ideal is carried out: if po_ideal meets the requirements of the dual criteria, it is directly taken as the optimal reachable pose po_reachable of the trajectory point; if not, the dichotomy method ²⁵ is used to iteratively search for the optimal reachable attitude that meets the constraints in the interpolation space.

Define the search interval of the interpolation weight as (x_low,x_high) where the initial value of the lower bound x_low = 0 (corresponding to the reachable initial attitude), and the initial value of the upper bound x_high = 1 (corresponding to the ideal fitting attitude). The core optimization goal of the algorithm is: under the premise of meeting the dual constraint criteria, find the maximum feasible interpolation weight x, so that the interpolation attitude is as close as possible to the ideal fitting attitude, and ensure the maximum fit between the end cleaning device and the facial curved surface. The core process of iterative search is as follows:

• (1) Calculate the midpoint of the interpolation weight of the current search interval:

$$x_{\mathrm{mid}}=\frac{x_{\mathrm{high}}-x_{\mathrm{low}}}{2}{x}_{\mathrm{up}}-x_{\mathrm{down}}>\epsilon ,x_0,x_1\in [0,1]$$

(20)

In the formula, ε is the precision threshold for iteration termination. Calculate the interpolation attitude o_target(x_mid) corresponding to xmid based on equation (19).

• (2) Combine the interpolation attitude o_target(x_mid) with the spatial position of the trajectory point to construct the interpolation pose po_target(x_mid), and verify the reachability and configuration consistency of the pose based on the dual criteria in Section 3.3.3.

• (3) Update the search interval: if po_target(x_mid) meets the dual criteria (reachable and consistent configuration), it means that the current interpolation weight is feasible and can be further approximated to the ideal attitude, so update the search lower bound x_low = x_mid; if the requirements are not met, it means that the current interpolation weight exceeds the reachable range, and the search upper bound needs to be reduced, update x_high = x_mid. The interval update rules can be uniformly expressed as:

$$\{\begin{array}{cc}{x}_1=x_{\mathrm{mid}}\hfill & {\mathrm{po}}_{\mathrm{target}}\left(x_{\mathrm{mid}}\right)\mathrm{is}\mathrm{reachable}\hfill \\ \multicolumn{1}{c}{x_0=x_{\mathrm{mid}}}& {\mathrm{po}}_{\mathrm{target}}\left(x_{\mathrm{mid}}\right)\mathrm{is}\mathrm{not}\mathrm{reachable}\hfill \end{array}$$

(21)

• (4) Iteration termination judgment: when x_high−x_low ≤ ε, stop the iteration, and take the optimal reachable interpolation pose in the current search interval as the optimal reachable pose po_reachable of the trajectory point. The precision threshold ε is the core parameter to balance the search accuracy and computational efficiency. If the value of ε is too large, the search result will deviate from the optimal reachable attitude and reduce the fit between the end and the face; if the value of ε is too small, the number of iterations will increase significantly, reducing the efficiency of the algorithm, and after exceeding the motion control accuracy threshold of the manipulator, the search interval is already narrow enough, and further iteration has little engineering significance.

For all pre-planned trajectory points, the above method is used to solve the optimal reachable pose point by point, and finally a safe and reachable pose sequence PO_reachable is constructed. The poses in this sequence not only ensure the fit between the end cleaning device and the facial skin to the maximum extent, but also strictly avoid kinematic risks such as no inverse solution and configuration mutation.

Figure 15 shows the reachable trajectory pose sequence obtained after dichotomy search in the forehead cleaning area. The red dots in the figure are the 3D spatial positions of the trajectory points, and the blue arrows are the Z-axis direction of the end tool coordinate system. Figure 16 is the line chart of the position and attitude changes of the reachable pose sequence of the forehead, which contains a total of 23 trajectory point poses; it can be seen from the figure that there are many drastic changes in the position and attitude of the pose sequence, in which the maximum single-step translation is 18 mm, and the maximum single-step attitude deflection angle is 32.36°, which needs further smoothing of the motion trajectory through an interpolation algorithm. Figure 17 is the line chart of the joint angle sequence changes corresponding to the reachable pose sequence of the forehead after being solved by inverse kinematics. The length of the joint angle sequence is completely consistent with the reachable pose sequence, indicating that all reachable poses have valid inverse kinematics solutions; although there is a large change in the single-step joint angle in the sequence, after inputting the joint angle sequence into the manipulator controller point by point, the manipulator maintains the consistency of the configuration under all poses.

Figure 15.

Trajectory pose sequence of reachable poses for the forehead region.

Figure 16.

Pose sequence variation of the reachable forehead trajectory: (a) position sequence change and (b) attitude sequence change.

Figure 17.

Joint sequence variation of the reachable forehead trajectory.

Position-attitude collaborative smoothing method

Although the trajectory pose sequence obtained by pre-planning trajectory points has a certain density, the spatial interval between adjacent points still exceeds the motion smoothness requirements, and there are jumps in the attitude parameters, which easily lead to motion oscillation of the manipulator and high-speed joint motion. The traditional linear interpolation based on time series easily leads to discontinuous derivatives of the spatial trajectory, which in turn causes impact on the physical joints. To generate a smooth and isotropic motion trajectory, this paper adopts an adaptive spline interpolation algorithm based on generalized rigid motion metrics.

• (1) Spatially decoupled distance metric. Let the pose of the end of the manipulator at the i-th discrete key point be $T_i\in \mathrm{SE}(3)$ , its position vector is $p_i\in {\mathbb{R}}^3$ , and the rotation matrix is $R_i\in \mathrm{SO}(3)$ . The translational Euclidean distance $d_p$ between two adjacent key points is calculated as follows:

$$d_p(i,i+1)=\parallel p_{i+1}-p_i{\parallel }_2$$

(22)

For attitude changes, the Geodesic Distance on the manifold space is used as the rotation difference metric $d_R$ : ²⁶

$$d_R(i,i+1)=\parallel \log (R_i^{-1}{R}_{i+1}){\parallel }_2$$

(23)

Among them, log(·) is the group logarithmic map, and its norm is equivalent to the relative rotation angle Δθ between the two poses.

• (2) SE(3) generalized arc length parameterization. Due to the inconsistent dimensions of translation distance and rotation angle, direct fusion will produce scale deviation. This paper introduces the “equivalent operating radius” Characteristic Length $r_{\mathrm{eq}}$ for unit unification. The physical meaning of this coefficient represents the equivalent moment arm length from the flange center to the working focus of the end tool. The equivalent generalized distance $\mathrm{\Delta }{s}_i$ of the i-th segment of the trajectory is defined as:

$$\mathrm{\Delta }{s}_i=d_p(i,i+1)+r_{\mathrm{eq}}·d_R(i,i+1)$$

(24)

Accumulate the generalized distances of all line segments to obtain the global non-decreasing parameter $t_i$ for spline evaluation, and normalize it to the parameter domain u∈(0,1):

$$u_i=\frac{\sum _{k=0}^{i-1}\mathrm{\Delta }{s}_k}{\sum _{k=0}^{N-1}\mathrm{\Delta }{s}_k}$$

(25)

This parameterization method (Cumulative Chord Length Parameterization) ensures the uniform distribution of interpolation nodes in the generalized space.

• (3) Continuous pose spline generation. After obtaining the unified normalized parameter u, this paper performs high-order interpolation in ${\mathbb{R}}^3$ and SO(3) spaces respectively. For position interpolation, Cubic B-spline ²⁷ is used to ensure the continuity of displacement; for attitude interpolation, Spherical Spline Interpolation based on quaternions is used to ensure the smooth continuity of spatial rotation angular velocity and angular acceleration. ²⁸ Finally, the two interpolation results are re-synthesized into a high-density discrete pose sequence, which is output as a smooth trajectory:

$$T(u)=\left[\begin{array}{cc}{R}_{\mathrm{spline}}(u)\hfill & p_{\mathrm{spline}}(u)\hfill \\ \multicolumn{1}{c}{0_{1\times 3}}& 1\hfill \end{array}\right],u\in [0,1]$$

(26)

Figure 18 shows the visualization result of the trajectory pose obtained after interpolation and smoothing of the forehead. The red dots in the figure represent the 3D spatial positions of the trajectory points, and the blue arrows represent the Z-axis direction of the end tool coordinate system; it can be clearly seen from the figure that the trajectory points are distributed continuously and densely, and there is no sudden change in the Z-axis direction of the tool coordinate system, indicating that the transition of the end attitude is stable. Figure 19 is the line chart of the change of the trajectory pose sequence of the forehead after interpolation. The pose sequence contains a total of 998 trajectory points; although there are many peaks and troughs in the figure, the maximum single-step translation step is only 0.0012 m (1.2 mm), and the maximum single-step attitude deflection angle is only 0.4355°. The above results show that the interpolated trajectory pose sequence is closely connected and changes gently, which meets the requirements of motion smoothness.

Figure 18.

Interpolation result of the forehead trajectory.

Figure 19.

Pose sequence variation of the forehead trajectory after interpolation: (a) position sequence change and (b) attitude sequence change.

Contradiction between continuity of pose sequence and mutation of joint sequence

The reachable pose sequence that meets the configuration consistency constraints obtained through dichotomy search has good continuity of position and attitude changes in Cartesian space after trajectory interpolation. However, after solving the inverse kinematics of the interpolated pose sequence point by point, it is found that some interpolated poses cannot maintain the consistency of the manipulator configuration, and even have no valid kinematic inverse solution. Figure 20 is the line chart of the joint angle sequence changes corresponding to the interpolated pose sequence of the forehead, after eliminating the invalid poses that have no valid kinematic inverse solution and cause the mutation of the manipulator configuration. It can be seen from the figure that the length of the effective joint sequence is greatly reduced compared with the interpolated pose sequence, and there are many mutations in the joint angle sequence, which directly leads to the inability to guarantee the motion continuity of the end cleaning device, thereby seriously affecting the stability and cleaning effect of the facial cleaning operation.

Figure 20.

Joint sequence variation of the forehead trajectory after interpolation.

To locate the cause of the above problems, this paper verifies the interpolation results, and the results show that the poses generated by interpolation do not seriously deviate from the pre-generated reachable pose sequence, and the interpolation process does not introduce abnormal pose jumps. Subsequently, the relative position between the human face and the manipulator was adjusted through multiple groups of experiments to optimize the operation dexterity of the manipulator in the facial operation space, but the above problems of inverse solution failure and configuration mutation still cannot be avoided. Based on this, it is inferred that the physical structure of the manipulator body used in this study and the installation configuration of the end cleaning device have insufficient adaptability to the operation scenario requirements of facial cleaning.

Conclusion and outlook

This paper takes the six-axis robotic arm (Realman RM65-6F), end cleaning device, and RGB-D depth camera (Intel RealSense D435) as the core hardware platform, based on the Anaconda virtual environment, using the Python programming language and related algorithm dependent libraries, through the three-layer progressive architecture of “data acquisition–facial region division–motion planning,” constructs a complete technical system from multi-modal facial data acquisition to cleaning trajectory generation and output.

In the data acquisition layer, first, the manipulator is controlled to drive the RGB-D depth camera installed at the end to complete the surrounding acquisition of the human face, and the accurate registration and incremental fusion of multi-frame point clouds are realized based on the ICP registration algorithm; RGB images are collected at three preset surrounding anchor points, and the 2D pixel coordinates of facial key points are extracted through the MediaPipe Face Mesh model; then the global facial fused point cloud obtained by registration and fusion is transformed into the camera coordinate system corresponding to each surrounding anchor point, and projected to the 2D pixel coordinate system; neighborhood search and depth median filtering are carried out based on the pixel coordinates of the key points, and the corresponding 3D facial key points in the global fused point cloud are matched and extracted; finally, the global fused point cloud of the face and the 3D key points are uniformly transformed into the manipulator base coordinate system, providing standardized data input for the subsequent facial region division and motion planning.

In the facial region division layer, first, the anatomical standard division is completed based on the 3D facial key points, and the surface point cloud corresponding to each division is extracted from the global fused point cloud; then the multi-dimensional curvature features of the point cloud of each division are calculated, combined with the curvature quantitative threshold and the spatial adjacency of the regions, the optimization of facial cleaning divisions for cleaning tasks is completed; finally, the trajectory planning complexity before and after the division optimization is compared and evaluated to verify the rationality of the division scheme.

In the motion planning layer, first, the key path points are pre-planned in each facial cleaning division as the cleaning trajectory points; then the local point cloud fitting surface corresponding to each trajectory point is extracted, and the ideal fitting attitude of the end cleaning device is determined based on the normalized normal vector of the surface; taking the existence of the inverse kinematics solution and the consistency of the manipulator configuration as the dual constraint criteria, the optimal reachable pose is searched between the initial attitude and the ideal attitude through the dichotomy method; finally, an adaptive spline interpolation algorithm based on the generalized rigid motion metric is used to realize the continuous and smooth transition of the trajectory position and attitude.

The experimental results show that the method proposed in this paper can accurately match and extract the 3D spatial coordinates corresponding to the 2D facial key points of MediaPipe Face Mesh, and the accurate division of anatomical benchmark divisions and cleaning optimization divisions can be realized based on the 3D key points. The curvature feature analysis and trajectory planning complexity evaluation results show that the number of optimized cleaning divisions is significantly reduced, and the surface curvature in each division is gentler, which can effectively reduce the planning complexity of the cleaning trajectory. In each cleaning division, the ideal attitude of the end cleaning device fitting the skin surface can be determined based on the local surface normal vector of the trajectory points, and the proposed dual-constraint dichotomy pose search algorithm can ensure the kinematic reachability of the trajectory pose and the configuration consistency of the manipulator. The adaptive spline interpolation algorithm can greatly increase the density of trajectory points, and finally generate a continuous and smooth cleaning trajectory fitting the facial skin surface.

This study still has the following deficiencies:

• (1) The six-axis manipulator used in this experiment has a large volume and weight, and the cleaning device is fixed on the end flange of the manipulator in a vertical installation manner, resulting in an excessively long end manipulator segment. This hardware configuration has insufficient adaptability to the short-distance and high-dexterity operation scenarios required for facial cleaning. The specific performance is: although the reachable pose sequence that meets the configuration consistency obtained through dichotomy search has good position and attitude continuity in Cartesian space after trajectory interpolation, when transforming to the joint space for inverse kinematics solution, there are still problems such as the failure of the inverse solution of some poses and the inability to maintain the consistency of the manipulator configuration.

• (2) The research on the global path planning for full-face cleaning across divisions has not been carried out. This part of the research needs to be carried out relying on a more adaptable hardware platform and operation safety constraint system, which will be further supplemented and improved in the follow-up.

• (3) The contact mechanics analysis between the flexible cleaning end and the human facial skin has not been carried out, nor has the collision safety analysis for the cleaning operation process been carried out. Therefore, the cleaning trajectory generated at present has not completed the verification of operation safety and comfort, and cannot be directly used as the final execution trajectory of actual operation.

• (4) The current research is only carried out for static face scenes, and has not considered dynamic interference scenes such as micro-tremor of the subject’s head, posture adjustment, and facial micro-movement caused by breathing. The dynamic adaptability of the algorithm needs to be further verified.

Aiming at the research goal of facial cleaning robots, the follow-up research will be carried out progressively around the following directions: the first priority is to design or select a lightweight robot body and end cleaning device suitable for facial nursing scenarios, to ensure that the end effector can flexibly adapt to the complex curved surface characteristics of the human face. Secondly, carry out contact mechanics modeling and analysis between the flexible cleaning end and human facial skin, to provide theoretical support for the comfort and safety of cleaning operations. At the same time, improve the real-time collision detection and safety avoidance strategy during the cleaning operation, which is the core premise for the engineering implementation of facial cleaning robots. Finally, expand the applicable scenario of the algorithm from static faces to dynamic interference scenarios, further improve the environmental adaptability of the algorithm, and expand the adaptation capability of multiple scenarios such as home washstands and hospital bed nursing.

In summary, although this study has not yet realized the full-process engineering implementation of the facial cleaning function, the constructed multi-modal data acquisition strategy, facial region division scheme for cleaning tasks, and robot trajectory planning method considering kinematic constraints can provide a technical framework and research ideas with reference value for subsequent research in this field. It is worth noting that the method proposed in this paper is not a simple combination of traditional technologies, but the in-depth integration and collaborative optimization of multi-modal perception and robot motion control technology, which fully reflects the systematicness and innovation of the algorithm system.