Abstract
In recent years, machine vision has played an important role in product surface quality detection. The promotion and use of this technology have largely avoided the subjectivity caused by human detection and improved detection efficiency and accuracy. Different from the image data commonly used in machine vision, point cloud can describe the spatial structure of an object, provide more information than image data, and have the ability to expand the data to build multi-dimensional data models. Due to the strong anti-interference ability of point cloud sensors and the high accuracy of three-dimensional positioning information point cloud, nondestructive testing technology based on point cloud has received more and more attention. This paper summarizes the research progress of product surface quality detection methods based on 3D point cloud in recent years. According to different data processing methods, the detection research is divided into five categories: based on point cloud contour, based on local geometric feature, based on template matching, based on multimodal point cloud, and based on deep learning. The five methods are reviewed and summarized respectively. Finally, the key problems of each detection method and the future trend of product surface quality detection technology based on 3D point cloud are discussed.
Introduction
In the manufacturing process, the lack of maintenance of the machine for a long time may cause damage to the surface texture of the product. Aluminum alloy profiles and thin metal parts are deformed due to cooling after high-temperature forming. The product is subjected to wear, extrusion, and collision in use. In the above scenarios, the intuitive manifestations of damaged objects are scratches, dents, bumps, holes, warpage, and breakage.1,2 If not detected and repaired in time, it will reduce the product’s service life and increase the safety risk caused by the use of defective products. Taking the detection and maintenance of aircraft skin and aero-engine blades as an example, the existence of different types of damage will reduce the material strength and seriously endanger the fuselage structure in high-speed motion. 3 The appearance of the fuselage is mostly sprayed with the original color, which is affected by the appearance. The threshold segmentation based on image data, the saliency enhancement algorithm, 4 and the target recognition algorithm based on deep learning are all deficient. 5 The contact-based coordinate measuring machine (CMM) is widely used in the field of three-dimensional detection. Although the system has extremely high accuracy in verifying the size and geometric tolerance of mechanical parts, the scanning speed of the detection system of the coordinate measuring machine is extremely low. The probe has a certain diameter and wears for a long time. It is not suitable for processing complex parts 6 or large workpieces. 7 As a collection of 3D points around the surface of an object in a scene, usually represented by three coordinates XYZ, it can also cover other features, such as laser intensity values, RGB spectral data, etc. It has the advantage of intuitive characterization of the spatial structure of the object.
As a set of 3D points around the surface of the object in the scene, the point cloud is usually represented by XYZ three coordinates. It can also cover other features, such as laser intensity value, RGB spectral data, etc., and has the advantage of intuitively characterizing the spatial structure of the object. So the use of 3D point clouds has become the most important and direct data for portraying complex real-world scenes in the digital era, and its new method for evaluating products and equipment facilities has also emerged. New measurement methods for evaluating products and equipment facilities have also emerged. In addition to product quality inspection, from global mapping to small area inspection, from basic science to public service, all point cloud data figures appear, and it plays an important role in global change, smart city, 8 resource survey, environmental inspection, basic mapping and many other fields, such as unmanned high-precision map generation in real time, 9 power corridor safety inspection, 10 cultural heritage digital protection, 11 urban morphology analysis, 12 forest resource survey, digital ground model establishment, and has been widely used in many engineering and typical fields.
After more than 20 years of development, 3D laser scanning hardware has advanced in terms of stability, accuracy, and operability, and currently representative devices include ground laser scanners based on active light sources (Optech ILRIS-3D), line laser contour sensors (LMI Gocator) and NASA’s ICESat. At the same time, new means of 3D reconstruction have been born, with the Structure of Motion (SFM) software introducing RGB images to create high-resolution point cloud models, 13 active light source-based stereo vision technology, 14 consumer-grade RGB-D cameras (Microsoft Kinect, ASUS Xtion), 15 and deep learning-based 3D reconstruction technology (NeRF). The choice of non-contact object surface 3D dense point cloud acquisition techniques will vary for different task scenarios. Typical ones are the introduction of RGB images in Structure in Motion (SFM) software to create high-resolution point cloud models, active light source-based stereo vision techniques using integrated devices such as terrestrial laser scanners (Optech ILRIS-3D),16–19 consumer-grade RGB-D cameras (Microsoft Kinect, ASUS Xtion), line laser contour sensors (LMI Gocator), sensing data is the basis of various data processing methods, and the above methods and technologies can record the complete spatial structure of an object, 20 which not only excludes the problem of subjectivity in manual inspection, 21 but also effectively solves the problem of blind areas in the field of inspection.
In the past few years, point cloud target quality detection methods can be summarized into the following five aspects: Based on point cloud contours, based on Local geometric features, based on Template matching, based on Multimodal point cloud and based on Deep learning. This paper introduces the latest methods of product surface quality detection using 3D point cloud data. In the past 10 years, there has been no systematic collation and review of product surface quality detection technology based on 3D point cloud. Therefore, this paper summarizes the research results in this field in recent years, aiming to provide guidance and help for new researchers in the field of nondestructive testing. The rest of the paper is organized as follows: Section 2 outlines the method of product surface quality detection technology based on 3D point cloud. Section 3 analyzes the key problems and challenges of product surface quality detection technology based on 3D point cloud. The summary of this review is in section 4.
Body surface quality detection scheme of point cloud
Surface quality inspection of products based on point cloud is a field worthy of further study. Accurate measurement of geometric parameters of defects, such as area, volume or depth, can approximately predict the remaining life of the product. Therefore, the inspection has gradually become a key step in manufacturing and product use and maintenance. 22 With the maturity of sensing technology and three-dimensional reconstruction technology, a large number of research results have been achieved in damage detection methods based on 3D point cloud data, whether from the perspective of point cloud data structure, point cloud feature difference or model registration and fusion. In recent years, the rapid development of deep learning in the field of image processing has provided new ideas and research directions for 3D point cloud detection methods. The image cannot characterize the defects of non-obvious regions, and the captured spatial information is insufficient. The spatial feature expression of point cloud can improve the limitations of image visual detection technology. In order to better expand the limitations of non-destructive testing methods based on 3D point cloud data, infrared thermal imaging and other sensing data are gradually combined with 3D point cloud to solve the problem of weak representation of point cloud in surface damage detection of some objects. The method summarized in this paper is shown in Figure 1, where the direction of the arrow represents the order of the review, and the rise of the arrow reflects the forward trend of the point cloud research tendency.
The overall framework of point cloud detection method.
Based on point cloud outline
In the field of machine vision, based on the temporal point cloud of the frame structure, the common methods of defect detection are registration alignment based on contour point cloud and point cloud contour fitting. The 3D point cloud is divided into two types: ordered and disordered. The ordered point cloud is neatly arranged and restricted by strict coordinate intervals. However, the spatial points of the disordered point cloud are scattered, and no point will be accurately located in the section. The construction of the section requires the assistance of nearby data points. Khameneifar and Feng 23 fits a local quadric surface on the point neighborhood, and then effectively generates the projection trajectory of the point on the specified section. The generation of scattered point cloud profile contour usually needs to regularize the point cloud first. Ying et al. 24 provided an idea of the regularization of scattered point clouds. Within the error range, the scattered point clouds are equally stratified and shifted, and then the point clouds distributed on the single-layer plane are identified by geometric feature mutations.
In tunnel displacement and deformation monitoring, section profiles are an important form of data for evaluating tunnel stability, and many published articles have demonstrated the accuracy of the profile analysis method. To overcome the deficiency of continuous profile extraction in the absence of tunnel centerline equations, Kang et al. 25 adopted the strategy of fitting the tunnel centerline with a combination of straight lines, over curves and curves, and adjusting the adjacent fitted line segments with global least squares to ensure their continuity and smoothness. Despite the efficient and accurate performance of 3D laser scanning devices in acquiring tunnel geometric parameters, the method of extracting tunnel sections still has some shortcomings and may not detect minor tunnel deformations. Cao et al. 26 combined tunnel sections with a complete tunnel model and proposed a new framework for tunnel section extraction, that is, a slice-based approach to extract the tunnel central axis, utilizing an improved RANSAC. Weixing et al. 27 proposed a method based on 3D invariant moments and best-fit ellipse to extract cross-sections and tunnel central axes, while using fractional The experimental results show that the measurement accuracy can reach more than 99.76% at any position of the interface contour of the tunnel, and the method can effectively detect the quality and safety of the tunnel, and has certain application value in the analysis and information extraction of the tunnel 3D point cloud.
In the aerospace neighborhood, the blade, as the core component of the aero-engine, determines the performance of the aero-engine, where the main content of the blade inspection is the machining geometry error of the type surface. The raw scanned point cloud data of blades are often contaminated with outliers at high curvature features due to undesirable reflections during the scanning process. To ensure an accurate assessment of the morphological parameters of the cross-sectional profile and to reduce the effect of scan data inaccuracies, Khameneifar and Feng, 28 Xiao and Huang, 29 Lu, 30 and other researchers reconstructed the blade profile with the help of B-sample generality in closed curve approximation. In order to reduce the sensitivity to outliers during curve reconstruction, some researchers tried to smooth and skeletonize the unorganized point cloud data. 31 Moving least squares (MLS) is a fundamental and important algorithm in point cloud processing and is widely used in curve and surface reconstruction. Ghorbani and Khameneifar 32 proposed a recursive weighted partial least squares (RWLLS) method to generate the contour skeleton of the initial cross-sectional point cloud, which eliminates the effect of outliers in curve fitting with the help of a weighting function and then uses point normals to evaluate each node edge The angular deviation of each node is evaluated and corrected for the problematic nodes using the point normally, and the contour is finally reconstructed smoothly using the b-sample curve. The experimental results show that the method exhibits high accuracy and robustness against noise even under the harsh conditions of point density of 0.2 mm and noise level of 0.03 mm, and the measured deviations of blade center-of-mass position and angle are 0.0032 mm and 0.0146°, respectively.
Zhong et al. 33 used the linear projection method in the acquisition of the blade profile data. Due to the large distortion of the blade, the projected blade data overlap, which makes it difficult to calculate the subsequent special diagnosis parameters and contour errors. Ding et al. 34 proposed a blade profile reconstruction method based on regional growth to obtain the complete blade profile data of the slice point cloud and realized the calculation of the characteristic parameters of the blade profile through the profile analysis. Finally, the measured data point set of the blade profile was registered with the theoretical profile to determine the deviation of the profile shape. The experimental results showed that compared with the HEXAGON three-coordinate contact measurement, the cross-section characteristic parameters and profile error were less than 0.015 mm. In contour line extraction based on slicing, the core is layer thickness. Considering the contradiction between layering efficiency and molding precision and further reducing the staircase effect generated by layering, Javidrad and Pourmoayed 35 and Hai-peng and Tian-rui 36 adopted adaptive layering method with the change rate of the adjacent layer distance as the discriminant factor. The experimental results show that the layered point cloud has a good effect in complex model of high precision and profile. Obtaining accurate contour curves from point cloud slices is also a top priority in reverse engineering, which is also discussed in detail in section 2.2 of object surface quality detection based on template matching.
The method based on registration is often used in the scene where the strict requirements of the measured part are completely consistent with the theoretical model. When the defect point cloud model is quite different from the standard model, it may lead to a large registration deviation or even a matching error. The theoretical model is not available in any scene. In view of this situation, the contour has to be reconstructed. Li et al. 37 sliced the weld point cloud at equal intervals, and compared the normal weld contour with the cross-section contour. The theoretical model of the normal weld contour is based on the distribution of the weld cross-section contour under the determined welding process, and the defect area is determined by the cross-section distortion point. In the registration experiment of single frame point cloud, due to the inconsistency between the change of the centroid position of the point cloud of the workpiece surface contour collected continuously per unit time and the moving direction of the acquisition equipment, Xiong et al. 1 considered the characteristics of the smooth and slow change of the attitude of the adjacent section of the rail. Adaptive iterative closest point (AICP) algorithm based on Kalman filter model was proposed to optimize the registration parameters. Experiments show that the registration method can improve the registration accuracy of the rail contour point cloud and the standard contour point cloud to the sub-millimeter level, and improve the recognition accuracy and quantitative accuracy of the defect area.
The defect detection method based on fitting mainly analyzes the surface quality of the object by establishing the mathematical model of the workpiece, fitting the corresponding model parameters with the least square method or random consistency sampling, and then calculating the residuals of each point. This method is suitable for the detection of workpieces with simple mathematical models. 5 The single-frame contour point cloud does not need to consider the Y parameter of the moving direction. The data can be converted to the XZ plane reference system to focus on the change of the depth information z, which is convenient for the acquisition of the contour information. As shown in Figure 2, Guo et al. 38 used the random sample consensus (RANSAC) algorithm to fit the contour shape of the rail geometric section, and used the distance threshold comparison to screen out the abnormal contour point cloud. Cao et al. 5 proposed a dynamic interval fitting curve detection algorithm based on line-structured light. Considering the existence and continuity of the first derivative of the contour line on the upper surface of the rail, the straight line and curve are used to fit the geometric contour of the rail, and then the distance from each point to the fitting curve is calculated in turn to determine the defect point. Finally, the probability threshold is introduced to re-judge, so as to prevent the undulating noise points from being misjudged as defect points and improve the anti-interference ability. The above method is similar to the monitoring method of the product contour control chart using linear, nonlinear and nonparametric model assumptions. 39
Extraction and fitting of point cloud contour. 38
In the framework of the point cloud contour fitting method, Feng et al. 34 proposed a calculation method for global and local initial geometric imperfections based on a digital model. The point cloud model of the steel member was sliced at equal intervals along the inherent Z-axis and labeled as different sections. The RANSAC algorithm was used to segment a single section into multiple line segments, and the points in each line segment were labeled. Finally, the completed cross-section and the spliced cross-section were combined to form a digital model of the component. By comparing the geometric parameters with the standard cross-section model, the area where the global initial geometric imperfections are located can be determined. The local initial geometric defects are analyzed according to the defect type and its calculation rules. This method makes up for the deficiency of the detection method in the absence of the original theoretical model, and is more suitable for on-site detection. 5
In complex detection scenarios, the above methods have certain limitations. For example, the measurement conditions are unstable. The selection of the deviation range based on the registration method depends on prior experience. At the same time, the prior shape and standard model are required, and the requirements for measuring equipment are high. The threshold-based defect segmentation method does not have robustness and anti-interference, and the quality inspection requirements of different products are different. 40
Based on template registration
The template-based registration method relies on the digital model of the standard product. By extracting the local descriptors or the overall shape on the point cloud of the standard workpiece and the workpiece to be tested for matching, the iterative closest point (ICP) 41 or the normal distribution transformation (NDT) 42 algorithm is used to complete the coincidence of the two point cloud models to achieve the detection and positioning of the target area defects. 43 Such detection methods have high requirements for the robustness of the algorithm and the descriptiveness of the local features of the point cloud. The detection object is usually a standard production part.
Two-step coarse and fine registration is a common point cloud registration strategy. The coarse registration is used as an initial alignment. The main purpose is to find the appropriate initial alignment space attitude to accelerate the convergence speed of subsequent optimization. Due to the inconsistency of the production of the casting parts, the uncertainty of the datum plane is caused. Therefore, on the basis of the initial alignment of SAC-IA, Gao et al. 44 set up three constraints for the registration and alignment of the casting and the standard model, namely the envelope point inclusion rate, the datum alignment error and the uniform distribution rate of the machining allowance, and introduced the genetic algorithm to search the optimal transformation matrix. The experimental results show that compared with the ICP method, the alignment error and the point number inclusion rate are much lower than the ICP method. Wang et al. 2 also put forward the alignment strategy under the constraint condition of thin-walled structural parts, allocated the constraint conditions when solving the optimal transformation matrix, and optimized the optimization process of parameters by genetic algorithm. In order to solve the problem of registration accuracy of point cloud model with complex surface, Hong-Seok and Mani 45 used the region-growing algorithm based on normal vector to segment the surface with obvious characteristics of point cloud, then registered the surface of the model with computer-aided design (CAD) model by ICP method, and finally judged the defect point by distance value. In the registration of parts with high curvature and complex surface, adaptive distance function (ADF) uses correction coefficient to describe the point-to-surface distance on different curvature features, which is more accurate than tangent-squared distance minimization (TDM) 46 and ICP. He et al. 47 introduced M estimation and ADF registration distance error function for the interference of noise points on registration.
The disordered point cloud lacks a data management structure, and the topological structure of the point cloud needs to be created before calculating the registration matrix. The calculation amount of the registration matrix usually increases exponentially with the increase of the number of point clouds and the complexity of the target object, and the scanning point cloud of the component with serious damage surface shows uncontrolled difference in size and complexity. He et al.
47
used FPFH and ICP to perform fast coarse and fine registration of the standard model and the defect point cloud, and then established an octree topology to obtain the rough position and size information of the defect area through Boolean comparison. The number of iterations of the octree is determined by Equation
Due to the uncertainty of the pose information of the point cloud to be registered, it is difficult to achieve fast registration with conventional registration methods. Therefore, Qian et al.
48
proposed a three-dimensional standard digital model library (
In the absence of a standard CAD model, it is necessary to carry out reverse engineering. According to the damage point cloud model, the surface reconstruction of the reverse three-dimensional model is carried out. Li et al. 51 extracted the prominent cross-section 52 (PCS) from the point cloud model to assist the reconstruction, combined with the curvature condition to eliminate the defect point cloud of the wear part damage area, and realized the accurate reconstruction of the standard CAD model of the part. In the registration alignment process, the curvature and distance constraints are introduced to optimize the ICP to identify the unreliable corresponding points in the model registration, and then the Boolean operation is used to distinguish the blank area between the reconstruction model and the scanning point cloud model to detect the damage area. This method has limitations in reconstructing complex shaped parts due to the differences between defects. Gao et al. 53 obtained a damage free polygon model by surface expansion and used it as a standard geometry to extract shape of the defect region. Wilson et al. 54 also carried out reverse engineering based on PCS, and on the basis of model registration and alignment, they chose to use Boolean operation to capture the blank area between the blade standard model and the original mesh surface of the blade point cloud model. Zhang et al. 55 used reverse engineering to reconstruct the damage model and the nominal model, and achieved the best fit between the models by the transformation matrix and the overlap area comparison method, and detected the defective layers in the damage model. In the past few years, some registration and alignment methods for blade models have also appeared, such as improved ICP algorithm, 56 SDM algorithm, 57 and ADF algorithm. 46
The methods reviewed in this section mainly have two concerns, namely, theoretical model and registration strategy. Figure 3 summarizes the methods and experimental results based on template matching.
Method and test results based on template registration.
From the overall perspective of template matching method, how to avoid local optimal matching, reduce computational complexity and improve timeliness is still a challenging task. In the landing detection, the lack of theoretical model is one of the main obstacles of registration method. As the main technology of building data model, reverse engineering is very important, which is the current and future research focus.
Based on local geometrical characteristics
The quality detection method based on local surface features of point cloud depends on local surface information, such as curvature, 58 normal direction, geometric and topological information, and divides points or sub-regions into larger regions according to the uniformity measure of local surface characteristics.
Among the geometric features, the normal and curvature of the point cloud can produce a good feature difference. Even after the point cloud is denoised and smoothed, it can still be effectively detected. The method of using the two types of point cloud features of curvature and normal vector combined with clustering algorithm to realize product surface defect detection has been widely recognized and applied. In additive manufacturing, Law et al. 59 used the curvature-based threshold segmentation method to segment the material powder layer and the consolidation area. Experiments show that the segmentation results of this method are accurate and can be further used for the detection of geometric defects and dimensional errors. On the basis of obtaining defect sensitive features such as surface curvature, Zahra Hadavandsiri et al. 60 proposed a point-by-point classification method based on principal component analysis to distinguish outliers in point cloud data. In aircraft skin damage detection, Jovančević et al. 7 used region growing segmentation algorithm based on normal vector and curvature information to segment the damaged region from 3D point cloud data. Liu et al. 61 introduced a double threshold segmentation based on curvature feature and Euclidean distance in the defect detection of engine cladding layer, which avoids the situation that the normal vector and the part with no obvious curvature change in the defect convex and concave area are regarded as a plane. Since defect detection under threshold segmentation is based on the local neighborhood of the points and is influenced by the density of the point cloud, it is difficult to accurately discern whether the region to which the points belong is a defective region or a sparse density region by using the normal vector angle and distance information between the points. Muraki et al. 62 performed the detection of defective points based on the area and roundness information of the Voronoi region established from the point cloud data and grouped the defective points to eliminate noise. In order to reduce the slow efficiency of the algorithm caused by the point as the detection object, some scholars have separated the defect area and the normal area by dividing the point cloud, and then analyzed and judged. Wei et al. 63 innovatively proposed the improvement strategy of the region segmentation algorithm based on super voxel, that is, according to the neighborhood concavity and convexity of the query area, the micro indentation and depression on the microscope surface are detected. Experiments show that this method not only shortens the processing time, but also controls the detection depth to 20 μm.
The computational quality of most point cloud geometric features depends on the choice of local three-dimensional neighborhood size. Excessive neighborhood data will smooth out some defect features, 64 and the choice of neighborhood size of local 3D points is often based on prior knowledge in specific scenarios.60,65 Some scholars use heuristic methods for adaptive scale selection.66,67 The above neighborhood selection uses a globally unified single scale. In order to improve the saliency of feature extraction, the selection of neighborhood scale gradually shifts from global consistency to optimal neighborhood at a single point, where supervised learning and unsupervised learning with adaptive adjustment are favored by researchers, such as iterative adaptive methods that combine point density and local geometric features, 68 and methods that set the neighborhood size for each point in the point cloud based on entropy. 69 Nevertheless, the calculation of point cloud normal vector and curvature is still susceptible to noise interference and is not stable. At the same time, the threshold selection in the region growing segmentation method based on geometric information involves prior experience, which makes the method highly dependent on data and insufficient generalization ability. In order to enhance the distinctiveness and robustness of 3D point cloud local feature description, some researchers selected feature descriptors to describe local spatial information through the geometric feature differences between interest points and neighborhood points.
The commonly used feature descriptors in the past are point feature histogram 70 (PFH), fast point feature histogram 71 (FPFH), viewpoint feature histogram 72 (VFH), signature of histogram of orientation 73 (SHOT), etc. The above descriptors are highly dependent on the correct calculation method of the normal, and it is easy to cause the loss of information when describing the defects of small change regions. Inspired by the existing PFH algorithm, Carlos A Madrigal et al. 74 replaced the point normal with the normal estimation of the least squares fitting plane of the neighborhood point region, and constructed the model point feature histogram (MPFH) based on the normal difference of the local region and the estimation information of the local point cloud model. Experiments show that the descriptor can significantly improve the descriptor ’s discrimination of the abnormal region. Jie 75 designed a descriptor with local detail features of planar point cloud by projection, normal vector deviation calculation and statistics of point cloud, and obtained the final defect recognition result by random forest classification algorithm. The combination of feature descriptors and classifiers can effectively improve the classification accuracy and model generalization ability. In addition to random forests, the selection of classifiers can also be support vector machines, decision trees, neural networks, etc.
To present the framework of this section more clearly, the methods and experimental results based on local geometric features are summarized, as shown in Figure 4.
Method and test results based on local geometric features.
From the experimental results of Figure 4, it can be seen that the target object of the object surface quality detection method based on local geometric features, either the global shape is close to the plane, or the local area is approximately regarded as a plane area. The advantage is that the algorithm is simple and easy to operate. The strong discrimination of the geometric features of the point cloud can quickly and accurately identify the abnormal points. The disadvantage is that the universality is low, and the complex surface objects need to be further studied. At present, domestic and foreign scholars are more bringing the geometric features of the point cloud into the deep learning to enhance the training of the model to improve the performance of the model.
Based on multimodal point cloud data
The information of point cloud feedback is limited, and a single point cloud information is difficult to meet the needs of three-dimensional target defect detection. At present, many scholars enhance cloud information through modal transformation or complement the shortcomings of a single sensor through the data advantages of different modes.
Point cloud modal transformation
The method of converting 3D point cloud to 2D image has been widely used in the field of architecture, engineering and construction (AEC) inspection in recent years.76,77 In machine vision, defect information is mainly obtained through 2D images. In places where light is not fully illuminated, this method is difficult to play a role. 78 Therefore, Methods based on 3D point cloud images are proposed, whose core idea is to learn the region of interest features on 2D depth images mapped into 3D point cloud.79,80 These methods do not need to consider the external lighting conditions, shooting location, and the disorderly nature of 3D point cloud, and can play a better suppression for the interference of outliers, but this method needs to establish the point-image mapping relationship between 2D images and point cloud.
Projection 81 and gridding methods are widely used in the process of converting 3D point cloud data (X, Y, Z, I) into 2D grayscale image (U, V, GV) in the detection task of cylindrical tunnel deformation and disease (cracks, leakage) of subway shield. U and V represent the location of the points in the two-dimensional image, GV represents the grayscale value, and I as the laser intensity value is mainly affected by the reflective properties of the material surface, the direction of beam propagation and the measurement distance. After eliminating the influence of distance and incident angle, the intensity value I can be used to detect other types of defects, such as stains, corrosion, leakage, etc. Tan et al. 81 regarded the tunnel as a regular cylinder, and the tunnel point cloud is expanded into a plane point cloud by cylindrical projection method to generate a grayscale image, that is, the standard equation of the circle is used to fit the center of the profile point cloud scanned by the laser moving platform in unit time, and then the arc length from the vertex of the fitted circle to the specified point is calculated as the position of the x-axis of the point in the image coordinate system. Huang et al. 82 improved the problem of stretching in the process of transforming the non-standard cylindrical point cloud model of the tunnel to the image, that is, the cross section of the tunnel lining is assumed to be an ellipse, 26 and the spatial structure of the cross section is corrected to a standard circle by perspective projection, and the experimental result was shown in the Figure 5. Then the convolutional neural network based on mask and region (Mask R-CNN) is used to complete the automatic segmentation of tunnel lining leakage defects and background areas. Different from the laser scanner and other point cloud acquisition methods, the three-dimensional reconstruction method of stereo vision is based on two-dimensional images. Therefore, the mapping relationship between points and images can be determined synchronously during three-dimensional reconstruction. Zong et al. 14 used the three-dimensional reconstruction method of stereo vision to directly segment the defect areas of image and point cloud data according to the mapping relationship.
The process of converting 3D point cloud intensity values into 2D image. 82
Multimodal fusion data
3D point cloud lack of texture and spectral information, registration and fusion of point cloud and other sensor data Color point cloud with texture properties is an important means of point cloud data expansion, but also one of the advantages of data. Wei et al. 83 proposed a 3D point cloud and 2D image fusion method based on the internal and external parameters of the color camera to project the point cloud first to get the point cloud gray map, and then match the point pattern with the color image to obtain the translation and deflection angle between the point cloud data and the color image to complete the alignment of the point cloud data with the image data. After data registration, the solder ball was extracted for defect detection and height measurement such as solder ball size, damage, deformation and bridging, and the coplanarity measurement of solder ball was realized. For some objects with special reflective properties on the surface, 6 it is difficult to detect defect areas using point cloud data or RGB spectral images of objects in self-passive light, requiring the use of other sensor data, such as the temperature value of an infrared thermometer. 84 López-Fernández et al. 84 used SFM to reconstruct the point cloud model of RGB photovoltaic (PV) panels, firstly normalized the sum of 8-bit RGB channel numbers of 2D images, and then achieved the thermal image mapping of 3D point clouds through the interactive recognition of homologous entities to obtain the 5D model (X, Y, Z, T, RGB-I), where (X, Y, Z) and RGB-I are used for the denoising and segmentation of PV panel point clouds, respectively, while the identification of panel defect regions is achieved by calculating the absolute difference between the temperature value of the current point and the median value of the total temperature concentration trend. Vidal et al. 85 projected the temperature value from the thermal image to the point cloud data generated by binocular stereo vision through the calibration matrix of the camera, forming a spatial temperature distribution map with a dense lattice ( see Figure 6 ), which can clearly observe the quality mutation area.
Reconstruction of 3D thermal image. 85
In the defect detection of complex scenes, the 2D image view of 3D point cloud data can well present the three-dimensional position information and defect information, which provides the possibility for the point cloud to be imaged. At the same time, the solution of sensor data fusion makes up for the deficiency of point cloud data detection technology to a certain extent, and makes full use of the feature difference between the normal surface and the defect area of the object. For example, the thermal characteristics of the temperature vector are used to detect the hidden hot spots, and the potential feature information of the target object is excavated. The point cloud reflects the spatial information of the detected object, and the resulting multi-dimensional model provides more complete spatial information and measurement information. Based on this information, the surface defect area of the target object can be detected and measured faster and better.
Based on deep learning
Damage detection using point cloud images
In recent years, scholars have achieved outstanding results in the direction of surface appearance detection by introducing deep learning technology on the basis of prior knowledge.86,87 Weimer et al. 88 used the classification function of CNN to complete the defect detection task, which enabled the accuracy of defect detection to the window size level; Wenhao et al. 89 combined the Beamlet algorithm with the CNN classification idea to complete the detection of fine crack defects on the surface of core blocks; in order to improve the detection efficiency of the model for large size input images and enrich the content of the output image, there emerged two-stage detection models such as Fast RCNN, 90 Faster RCNN, 91 Mask RCNN, 92 FPN, 93 etc., and the classical models such as YOLO 94 and SSD 95 in the first stage; YOLO is the most representative target detection model at present due to its high accuracy and high frame rate detection capability, and is widely used in the surface defect detection tasks of steel, precision electronic components, and production equipment. And YOLOX, as the masterpiece of the YOLO series of networks, uses decoupling heads and advanced tag assignment strategies to achieve a balance of accuracy and efficiency. 96 With the increase of artificial intelligence technology in the industrial field of landing this demand, deep learning-based surface defect detection method has become one of the important research directions in the industry, which is mainly divided into two aspects: 2D detection and 3D detection.
In processing 3D point cloud data, some researchers have also followed the idea of image detection. In the detection of weld defects, in order to enhance the presentation effect of weld defects, Shao et al. 97 used six depth images projected from different angles of the solder joint point cloud model as the input of the model, and used standard CNN to detect and classify solder joint defects, which made up for the shortcomings of insufficient global feature capture ability of point cloud based on depth image. Eguchi et al. 98 transformed the 3D point cloud data of the pavement into color information images, and used the AlexNet model to detect the spots on the asphalt surface. In order to obtain the detection effect of the optimal 3D point cloud image, Jiang et al. 78 compared the performance of RGB, depth and normal vectors and their combined feature images from photogrammetry point clouds for image segmentation tasks on the U-Net model. Experiments showed that the optimal effect can be achieved in the combined feature image of depth and normal vector.
In addition to the geometric features carried by the point cloud itself, some researchers use the point cloud intensity to generate intensity images for detecting defects caused by changes in the material properties of the object surface, such as corrosion, wall seepage, etc. Aiming at the leakage of subway tunnels, Liu et al. 99 proposed a deep learning model Res2Net based on the intensity image of mobile laser scanning point cloud. This model introduces a structure of residuals and feature cascades to ensure that the multi-scale leakage features remain unchanged in the network propagation process.
Due to the two-dimensional characteristics of the image and the limitation of the acquisition field of view, it is impossible to express the spatial position of the defect in the workpiece. Shijun and Zhenlin 100 improved the defect recognition process of deep learning, and located the detected defects in three-dimensional space. At the same time, a multi-model cascade defect re-inspection model was proposed, that is, an unsupervised learning adversarial generation model was established to solve the defect detection under the condition of unbalanced sample ratio. The defect location was obtained by analyzing the differences between the repaired image and the original image, and the MASK-RCNN model with supervised learning directly located the defects and categories through the integrated detection and segmentation of the image. The position of multiple coordinate systems is calibrated to uniformly solve the transformation matrix between the plane image coordinate system and the world coordinate system, so as to realize the positioning of stereo defects.
Damage detection using point cloud data
Different from the two steps of image defect detection to locate the spatial position, the model input for 3D deep learning is the 3D point cloud. As a pioneer model of point cloud detect 101 method, PointNet directly extracts point features from point sets. In the subsequent improvement architecture, by aggregating regional information and global information, point state features are enriched, point cloud detection capabilities are improved, and popular attention mechanisms are combined to explore high-precision point cloud detection model architecture. Typical 3D semantic segmentation networks such as PointNet, 101 PointNet ++, 102 and Point transformer, 103 as well as more advanced network models, have an intersection-to-union ratio (mlou) of less than 0.75 on public data sets. 104
Hao et al. 105 used PointNet ++ to classify the defect morphology of the head of the ejector rod in the seamless steel pipe production line. Compared with PointNet, the model added a multi-level local feature extraction structure, which reduced the number of points and improved the acquisition of information. Experiments showed that the defect classification accuracy of the model can reach 97% with good stability. Inspired by the Transformer module’s enhanced regional feature correlation, 106 Zhou et al. 107 proposed a Transformer-based point cloud classification network (TransPCNet) for identifying sewer defects. The designed feature embedding module maps points to high-dimensional space for feature extraction and closure, and then learns multi-scale features through a self-attention cascade structure. At the same time, in order to strengthen the distinction between similar defect classes, a weighted smoothed cross-entropy loss function was designed, that is, when different categories have similar global features, smooth labels avoid overfitting during classification training. The experimental results showed that the detection accuracy of TransPCNet is higher than that of PointNet by 47% under the same data set.
Due to the uneven density distribution of irregular point clouds, the fluctuation range of detection results is large. Li et al. 108 took fiber reinforced resin matrix composites as the research object, and proposed a lightweight two-stage semantic segmentation network (MASK Point). In the first stage, a multi-head parallel interest domain extractor (3D RPE) was designed to explore the potential areas of defects. Since each 3D RPE was configured with different density clustering parameters, the robustness of processing point clouds with different densities was improved to a certain extent. The second stage consisted of shared classifiers, filters, and non-maximum suppression. Filtering the output of the first stage of screening, experiments show that MASK Point model’s mlou was about 30% ahead of PointNet and can more accurately detect material surface defects.
In the defect detection of PCB solder joints, Hu et al. 109 proposed a dual-stream region attention network model (DoubRAN), which directly focused on the region of interest through back propagation without the help of bounding box labeling. Experiments showed that the region of interest extracted by the region attention network was consistent with the results of human subjective judgment of solder joint quality and meets the factory requirements. Considering the adverse effects of outliers on small sample model training, Li et al. 110 proposed a lightweight neural network of solder joint network (SPNet), which used a local group attention mechanism to adaptively learn favorable critical point features. In industrial applications and research tests, few samples and poor generalization ability are also major obstacles to 3D point cloud deep learning methods. Incremental training is required by increasing the number of samples to iteratively improve model detection performance.
Imageization of 3D point cloud or direct processing of 3D point cloud are two mainstream methods of point cloud deep learning. The former does not need to over-consider the three-dimensional spatial configuration of the object, and the focus of information is more concentrated. The research scope of the latter is relatively limited, and it is mostly used for objects with obvious feature differences between the surface and the defect area. At the same time, point cloud deep learning does not have more open defect data for researchers to use, and it is more difficult in sample manufacturing and acquisition. Therefore, there are still many problems to be solved in the application of deep learning algorithms to 3D defect detection.
Conclusion
Traditional non-destructive testing (NDT) techniques have limitations of insufficient spatial positioning information and are susceptible to interference from external factors that affect defect detection accuracy, so 3D point cloud-based defect detection methods have emerged in recent years. This paper systematically summarized the methods of product surface quality inspection based on point cloud data, analyzed the key problems faced by each method, and puts forward corresponding improvement and development suggestions, aiming to help relevant researchers quickly and systematically understand the methods and technologies in this field.
NDT technology is becoming more and more important in precision processing, manufacturing and engineering, and machine vision and artificial intelligence, as emerging technologies in recent years, have outstanding research results in this area, and most scholars are processing, analyzing and exploring two-dimensional image data from sensors. The emergence of point cloud data types makes up for the missing depth information and the ability to describe the spatial structure of objects in flat images, while the way point cloud data are collected and the type of point clouds collected affect the subsequent processing steps. Disordered point clouds are sparse and irregularly distributed, which are more difficult to process than regularly arranged ordered point clouds.
With the development of 3D point cloud deep learning network architecture, deep learning has the effective representation form of 3D objects, and shows flexible structural variability and scalability. In recent years, researchers at home and abroad have deeply explored the learning potential of point cloud features on the basis of classical target detection network structure, and proposed many network structures with superior performance. At the same time, two-dimensional image processing has a guiding role for 3D point cloud processing. Therefore, point cloud imagination is also a research direction in recent years. This method avoids the dilemma of direct deep learning of point cloud data, either reducing point cloud size for efficiency, or increasing the number of input points to obtain higher accuracy. Even in the identification of other types of defect damage, other sensing data can be converted into two-dimensional form and fused with point cloud images, such as infrared thermal sensing, RGB, laser intensity value, etc., which greatly improves the data expansion ability of point cloud. There are still some challenges in product surface quality inspection based on point cloud data :
Based on contour point cloud :
Scope of application. Contour point cloud can be regarded as a two-dimensional sequence after dimensionality reduction of 3D point cloud. Due to the abandonment of feature information in spatial dimension, it can only process objects with specific structure. At present, it is mainly used in temporal point cloud, focusing on structural steel, rail and other workpieces with continuous similarity of profile shape or high fitting of profile contour, which has great limitations.
Classification optimization. Through the extraction of all single-frame point cloud defect data, the defect category can be identified by clustering and reclassification. Due to the small proportion of defect points, machine learning algorithms can be tried to further improve the classification accuracy.
Based on template registration :
Registration efficiency. For small workpieces with high precision machining requirements and complex shapes, the method based on standard template matching is more effective. Even the edges and small deformation areas can be accurately detected, but the computational complexity is large and the timeliness is poor.
Based on Local Geometric Features :
Effectiveness. There are several factors that affect the effectiveness of the threshold segmentation method based on local geometric features. First, the given algorithm parameters depend on the prior experience of the researchers. Second, the uncertainty caused by the change of data resolution, instrument specifications and surface characteristics of the test object will lead to the failure of the algorithm and poor versatility.
Design of feature descriptor. In the case of low data quality (noise, low resolution, data hole), the statistics of many geometric attributes will be problematic. The existing methods only use geometric attribute information to construct descriptors, and the discriminability of descriptors faces great challenges for objects or scenes lacking geometric attributes.
Based on Multimodal point cloud Data :
Data structure versatility. The registration and fusion of multi-sensor data requires precise positioning and timestamp synchronization between devices. It is necessary to develop robust integration of high-density laser scanning sensors on other acquisition platforms. At the same time, as the amount of information in the polymorphic data model increases, it is necessary to promote the definition of general data structures.
Based on deep learning :
Data sets. There are few data samples used in the current research, which is reflected in two aspects : First, the preparation of point cloud training samples is time-consuming and small in number. On the other hand, the diversity of samples is poor. Most of the data comes from computer-aided modeling, which often deviates from the actual structure and does not have authenticity.
Incremental fast learning strategy. For point cloud data, most network models contain a large number of training parameters, resulting in long training time. At the same time, because the structure of the model is fixed and cannot be updated incrementally, when the data set is updated, the network model needs to be retrained, resulting in an increase in the time and cost of iterative updating of the model. 111
Real-time monitoring speed. The size of the point cloud is determined by the detection accuracy. In a specific scene, the point cloud model data size of a single workpiece can reach millions of points. In the actual landing detection, it is necessary to optimize the processing scheme of the point cloud data, and reduce the proportion of invalid point cloud data through the hardware strategy. Accurate to the specific detection target to reduce the point cloud size, such as PCB solder.
point cloud image learning strategy. The most representative of the 3D point cloud image is the depth image. Because of the idea of image deep learning, the point cloud images at different angles have different depth and feature information, and some spatial information will inevitably be lost. The fusion of two-dimensional image and 3D point cloud for comprehensive re-examination is a research direction worthy of consideration. On the basis of image learning, the reduced-dimensional point cloud is introduced as a supplementary training of small samples, which not only avoids massive calculation, but also captures the defects under the blind area of the image.
Based on these challenges, it is necessary to further investigate point cloud surface quality detection methods in depth, especially in the field of deep learning, to overcome the problem of data set creation. Secondly, there is a need to further investigate multi-sensor fusion detection methods to take advantage of the advantages among the data. Third, there is a need to address the optimal state between the scale of point cloud data and the detection needs, that is, expressing the dense, high-resolution point set in the target region and reducing the number of points in the non-defective region to improve the algorithm processing time, as well as the need for dynamically incrementally updatable detection methods to meet the detection needs of practical engineering.
Footnotes
Handling Editor: Chenhui Liang
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported and funded by the Development of an Automatic Spraying System for Bus Side Wall Putty and the Postgraduate Research & Practice Innovation Program of Jiangsu Province (KYCX22_1050)
