Sage Journals: Discover world-class research

Abstract

This paper attempts to print irregular 3D objects on a freeform surface. Notably, it is inevitable to cause errors when creating a specific object shape in simulated and real-life scenarios. This is mainly attributed to the unique characteristics of the materials adopted, which tend to develop different deformations properties. Nevertheless, since the simulated model provides a clue regarding the coordinates of the deformed shape, it is adopted as an essential indicator in estimating accurate real-world coordinates. Concretely, this paper presents a complete system that is capable of reconstructing the detailed surface coordinates in real-time conditions. To verify the effectiveness of the proposed method, a metal board is used as the primary material to create different curvatures physically and virtually. In particular, the real-time performance of the overall 3D surface reconstruction has been experimentally evaluated using several tools, such as KUKA KR90 R3100 robots, HoloLens 2, displacement sensor, and ArUco markers. Consequently, both the quantitative and qualitative results are presented to demonstrate the feasibility of the proposed method. Furthermore, the experimental data obtained manifest that there are significant differences between the simulated and the real-world coordinates. Thus, the findings of this study provide an insightful outlook and lay down several important implications for future practice.

Keywords

Robot arm surface reconstruction metal twisting ArUco marker HoloLens 2

Classifications

Computer programing, system and service; Methodology, apparatus, experimental design

Introduction

Owing to advances in machine learning and artificial intelligence, the development of fully automated with a highly accurate 3D reconstruction of objects has been receiving increasing attention from various research fields. One of the early works that conducted the 3D measurement using a laser scanner and rangefinder is by Shi et al.¹ The target of the experiment is both the large-scale bending plate and small complex scenes where several objects (i.e. mouse, keyboard, bottle, etc.) are placed stationery at a fixed position. Then a robotic arm is mounted on a guide rail such that a specific route is designed for the robotic movement. Then, two stereo cameras are placed on the robotic arm to capture top-down viewpoint images. Concretely, the 3D scanner measures partial sections, whereas the laser rangefinder provides the poses of the scanner in different view alignments. The results reported also serve as the benchmark for future work in performing this kind of experiment.

Recently, Bai et al.² extends the work of Shi et al.¹ by expanding the experimental setup in improving the accuracy and computational speed of the measurement. Concisely, two guide rails and an infrared 3D scanner are considered. To evaluate the robustness of the proposed method, a large non-smooth surface steel plate is treated as a complex large scene such that the measuring task is more challenging. However, similar to the previous work,¹ these measuring instruments require a sufficiently large physical operating workspace.

On the other hand, to perform the 3D surface reconstruction of the plant seed,³ proposed to utilize robotized turntable systems to handle and rotate the seed. A camera is placed in front of the seed such that a total of 36 images will be collected with the step degree of 10°. This method facilitates an affordable 3D imaging technique and the promising experimental results demonstrate the robustness of the method. A similar 3D image reconstruction method was applied to another agricultural product, viz, ham.^4,5 Succinctly, the ham object is placed on a rotating display stand horizontally and vertically and rotated at a constant speed. Then, an automatic semantic segmentation technique, namely Mask-RCNN (Regional Convolutional Neural Network)⁶ is applied to construct the object’s 3D model. To improve the object segmentation task, reliable camera tracking is adopted on top of the Mask-RCNN technique is demonstrated in Hoang et al.⁷ to achieve a semantic 3D map.

A real-time 3D reconstruction display had been performed effectively using HoloLens and Kinect sensors. Since HoloLens provides a built-in rendering virtual imagery display, a consistent rate of 60 fps allows a better user experience in terms of the sense of the presence of virtual objects. On another note, to enable the automated digitization of museum artifacts,⁸ proposed a method such that an automated, accurate, and dense 3D reconstruction system can be utilized to focus on small objects (<30 cm). Concretely, the system includes stereo imaging networks and robot inverse kinematics. Then, several mathematical modelings are introduced to enhance the objects’ postures effect.

On another note, several previous works develop 3D reconstruction systems for small-scale smooth or complex objects.^9–13 Specifically, in Zhao et al.,⁹ the coordinate transformation between the robot and the laser scanner had been investigated by suggesting the technique of reverse engineering. Recently, a low-cost and portable reconstruction system is introduced by Hosseininaveh Ahmadabadian et al.¹⁴ Particularly, it contains a pattern projection system, digital camera, raspberry pi, and mobile phone. The system has been evaluated on texture-less objects such as the dude statue, Cyrus tomb statue, and paper cone. Promising results were reported when compared with the measurements obtained from a compact laser scanner. However, a series of systematic procedures are required for equipment calibration.

In short, the importance and originality of this study are summarized as follows. First, this study makes a major contribution in demonstrating an efficient sampling method that enables the acquisition of a small amount of data points in reconstructing an accurate dynamic curved surface executing in real-time. Secondly, inspired by Lyngby et al.,¹⁵ a novel efficient and accurate system is introduced to scan the twisting and bending state of a large-scale metal sheet bent by industrial robots. We provide empirical proof of our method by comparing the results produced by HoloLens to robotic laser scans of fabricated models. Last but not the least, promising experiment performance demonstrated that the proposed solution outperformed the simulated model. Note that, the experiments herein performed the results for a specific axis of the robot arm and were limited to a certain range of rotational angles.

This paper is structured as follows: Section 2 describes the experiment preparation and detailed procedure regarding the realization of the real-time curved surface using HoloLens and displacement measurement sensor. Section 3 provides the evaluation metric to examine the effectiveness of the proposed method. Section 4 reports the experimental data with analysis. Both the quantitative and qualitative results are discussed. Finally, a summary is presented in Section 5.

Theory and design

Apparatus preparation

The apparatus involved in the dataset elicitation comprised two robot arms, a piece of sheet metal, a 2D near-range laser scanner, HoloLens 2, and ArUco markers. The following subsections describe each apparatus used in the experiment in detail.

Metal sheet and robot arm

The metal sheet is used as the key material to verify the effect in 3D surface reconstruction because it can be easily bent and shaped. Besides, it is known as one of the building materials that can create rigid and long-lasting structures. The dimension of the metal is 600 cm (Length) × 90 cm (Width) × 0.3 cm (Thickness). The metal material is shown in Figure 1. The experiments are carried out on two six-axis articulated industrial robots KUKA KR90 R3100 Extra. Each robot arm is equipped with high precision and high payload robots (i.e. 90 kg) at a reach of up to 3100 mm. The graphical illustration of the robot arm is shown in Figure 2.

Figure 1.

The compound metal as the target object.

Figure 2.

Graphical illustration of the two robot arms KUKA KR90 R3100 Extra.

The two robot arms are mounted on an 8-m track, as portrayed in Figure 2. All the robot’s movements have been programed by the robot language of the KUKA software add-on, PLC mxAutomation (MxA), such that it will move to a few specific pre-configured positions to achieve data point collection for the distance information. Real-time communication is enabled between the robot controller and an external system (i.e. a computer). There are two reasons to implement the proposed approach on this robot: (1) it has been widely used in industry due to the system robustness and reliability, and; (2) it can be easily transferred and reproduced on other robots. In this paper, the comparison of current 3D reconstruction using HoloLens with the proposed approach is quantified to validate the performance improvement.

Particularly, in evaluating the system’s robustness, a random bending shape is selected in this study. The metal bending process is performed by using two big-scale robot arms, whereby one side of the metal is held by one of the robot arms to cause it to bend at an angle and form the desired curvature, while the other side is anchored to a mounting plate to secure the metal surface. The robot arm’s translational movement and orientation rotation is triggered to manipulate the metal to create different curved surfaces.

Placement of ArUco markers

To obtain the metal bending shape numerically, the subsequent step utilized the ArUco marker by placing the marker on the metal surface for sparse position acquisition. In brief, ArUco markers are developed by Garrido-Jurado et al.¹⁶ and his team and were initially designed for camera pose estimation application. The ArUco markers are explicitly designed as visual features for localization, whereby they need to be printed to place on the desired environment space. An ArUco marker is a square fiducial marker encircled by a thick black border and an inner binary matrix that determines its unique marker identifier. The sample for the ArUco marker is shown in Figure 3, which can be generated from this site.¹⁷ The Aruco library is a well-developed OpenCV open-source software that has been widely adopted to estimate the pose of the 6DoF marker from augmented reality fiducials. In total, 45 markers with each unique ID are generated. Amongst, 42 of them are sticked and aligned along the two horizontal edges of the metal, such that each edge contains 21 markers. The separation between the marker pairs is about 25 cm. Some spaces are reserved between the markers and the metal edges, viz, about 3 cm. While three ArUco markers are for positional reference and one ArUco marker is used as the reference point for world coordinate space. The illustration of the placement of the ArUco markers is shown in Figure 4.

Figure 3.

Sample of ArUco markers.

Figure 4.

Illustration of the placement of the ArUco markers on the metal board.

Robot calibration

Before measuring the coordinates of the ArUco markers, an offline robot setup is performed to calibrate the robot flange and the measurement tool (i.e. laser range sensor). This step is critically important to precisely derive the relative position between the laser sensor and the robot base coordinate systems. The 2D laser sensor (SICK, OD Mini) projects a wavelength of 635-nm laser lines onto the target object’s surface. The distances measured vary from 50 to 150 mm and the accuracy obtained within the distance range is about 1000 mm. The laser sensor is attached and fixed to the robot’s flange.

During the calibration process, a calibration grid is placed to be fixed in a static position to allow accurate transformation between the laser scanner and the robot coordinate system. Therefore, a simple and robust tool center point calibration strategy is conducted. Specifically, the tool center point calibration is performed manually by moving the robot, letting the tool center point of the robot arm brush against a fixed point in the environment. The fixed point is basically the tip of a nail. The tool center point needs to brush against the tip of the nail with high precision and from four different angles. Conventionally, the position vector of the tool center $^{E} P_{T}$ can be obtained by solving the equation (1) using Singular Vector Decomposition (SVD) methods:

{[\begin{matrix} _{E}^{B} R_{1} -_{E}^{B} R_{2} \\ _{E}^{B} R_{2} -_{E}^{B} R_{3} \\ _{E}^{B} R_{3} -_{E}^{B} R_{4} \end{matrix}]}^{E} P_{T} = [\begin{matrix} ^{B} P_{E 2} -^{B} P_{E 1} \\ ^{B} P_{E 3} -^{B} P_{E 2} \\ ^{B} P_{E 4} -^{B} P_{E 3} \end{matrix}]

(1)

where $B$ refers to the base coordinate, $E$ is the end-effector coordinate, and $T$ is the tool coordinate. The notations $R_{i}$ and $P_{i}$ in equation (1) are the rotational matrix and the position vector at pose $i$ , respectively. The $_{E}^{B} R_{1}$ and $^{B} P_{E 1}$ can be obtained by the robot controller by the translation motion using forward kinematics. To simplify, the equation (1) can be expressed as $M^{E} P_{T} = N$ . The relation $^{E} P_{T}$ = $(M^{T} M)^{- 1} M^{T} N$ is then solved by SVD using programing software. On the other hand, $_{E}^{B} T$ is the homogeneous transformation from $E$ to $B$ . Besides, $_{T}^{E} T$ is the homogeneous transformation from $T$ to $E$ , whereas $_{T}^{B} T$ is the homogeneous transformation from $T$ to $B$ .

As such, the coordinate of the tool center point can be calculated by the robot in relation to the robot’s tool flange coordinate system. The resulting accuracy of robot calibration employing this method is about ±1 mm. Note that, the resultant accuracy is calculated by comparing the resultant solution obtained by solving equation (1) that requires four different robot poses with the known coordinates of the tip of a nail.

Data elicitation

This section describes the data point acquisition step from the metal surface. To obtain accurate coordinates of certain curvature metal surfaces and measure the relative pivoting angle, the software required are Rhino and Grasshopper, which allow the automated simulation process and facilitate the basic structural modeling. The following subsections provide the detailed steps to elicit and derive the real-world coordinates of the metal surface, which include: (1) metal board translation and rotation; (2) marker information retrieval using HoloLens; (3) construction of curves using Rhino and Grasshopper; (4) simulation and refinement of the metal surface; (5) extraction of surface normal’s by offsetting; (6) coordinates acquisition from the subdivided surface using a laser sensor and robot arm, and; (7) derivation of matrix transformation from pixel coordinates to real-world coordinates. A flow diagram is provided in Figure 5 to summarize the methodology proposed for enhancing the real-world coordinates of the metal board.

Figure 5.

Flow diagram of the proposed method.

Translation and rotation of metal board

To change the apparent curvature of the flat metal sheet, the robot arm is utilized. In particular, the robot arm that holds one side of the metal sheet is translated by a position vector of ( $x, y, z$ ) and rotated by the arbitrary angle in radians of the rotation vector ( $r_{x}, r_{y}, r_{z}$ ). The robot arm is simulated and defined physically using TACO.¹⁸ In brief, TACO is an open-source programing plugin for the visualization and simulation of controlling industrial robots directly within Grasshopper. Thus, it allows settings customization of the robots with different parametric modeling environments.

Marker information retrieval using HoloLens

This step requires Microsoft HoloLens 2 to establish the coordinate system to map to the real-world location. HoloLens is known as one of the quick decision-making tools for creating real-time 3D space. Compared to HoloLens 1, version 2 offers a larger field of view of up to 52° and higher resolution (i.e. 2048 × 1080 pixels). To achieve this, a printed QR code is placed on the edge of the metal board to serve as the reference point. The QR code is utilized in this experiment because it is identified as the most convenient recognition method that is feasible in implementing augmented reality applications. During the holographic visual initialization, HoloLens 2 renders the data detected as holograms using a coordinate system.

As for the coordinates of the two edges of the metal, the HoloLens is adopted to digitize the model by scanning the 42 ArUco markers. Based on the simultaneous localization and mapping (SLAM) algorithm implemented in the HoloLens using the fusion data from the build-in Inertial Sensors (IMU) and depth camera, HoloLens can estimate an object’s location in 3D space from the dead reckoning process. Therefore, a marker-based localization method for the HoloLens is sufficient to perform the localization augmentation for coordinates estimation of the two edges of the metal by scanning the ArUco markers that stick on both edges of the metal. After scanning all the ArUco markers, a virtual representation of the metal sheet can be constructed using the HoloLens plugin for grasshopper, Fologram, which functions in converting the data points scanned into points within the Rhino model space. Finally, the metal shape can be reconstructed in the 3D space with the knowledge of localization information generated from the HoloLens and the 3D poses from the scanned ArUco markers. Note that the distance between the HoloLens and the ArUco markers is held constant within 50 cm to capture the ArUco markers in retrieving the transformation information for 3D reconstruction calculation.

The example of attaining the coordinates information using HoloLens is shown in Figure 6.

Figure 6.

Illustration in attaining the coordinates information using HoloLens.

Construction of curves using Rhino

Since the Grasshopper plugin Fologram enables a two-way data stream that can render geometry in the computer-aided design (CAD) environment on the HoloLens, it allows direct 3D feedback superimposed on reality from a digital CAD model. By doing so, the curve of the metal board can be dynamically simulated by generating surface geometry through the augmented display. The example of the resultant geometry modeling generated from the proposed algorithm in this study is shown in Figure 7. The derivative patterns are created using the parametric modeling software, viz, the Grasshopper plug-in for Rhinoceros.

Figure 7.

The algorithmic design tools Grasshopper.

Simulation and refinement of metal board surface

Note that the previous stage only considers the data points collected from the two sides of the metal board. Hence, the current 3D reconstructed metal board may not be exact values in the real-world scenario as there is some variability when estimating the coordinates using the HoloLens’ SLAM algorithm. Besides, since limited data points are considered in the surface reconstruction (i.e. 42 points), it exists some differences in estimated and actual measurements.

Therefore, to acquire the detailed coordinates of the irregular surface, this study proposed a systematic procedure to yield an accurate 3D reconstruction of the metal board. In this step, the plane is subdivided into X × Y by partitioning the grid equally for both the longitude and latitude directions. In general, the subdivision process is done by dividing the straight line connecting the pair of two points from two sides, as shown in Figure 8 by using equation (2).

\begin{array}{l} (x_{i}^{m}, y_{i}^{m}, z_{i}^{m}) = (x_{1}^{m} + r_{i} (x_{k}^{m} - x_{1}^{m}), \\ y_{1}^{m} + r_{i} (y_{k}^{m} 2 - y_{1}^{m}), \\ z_{1}^{m} + r_{i} (z_{k}^{m} - z_{1}^{m})) \end{array}

(2)

where $m \in Z$ is the $m$ -th straight line connecting the pair of two points from two sides and $k$ is the fractional part of the segment that is defined.

Figure 8.

Illustration of equally subdividing the grid points for each pair of curves.

The above-mentioned subdivision process can be performed on the Grasshopper surface generation using the built-in subdivision tool. Concretely, the surface is simulated such that it can be reparametrized and redistributed to the UV-grid by using the normal Loft command in the Rhino software. However, although the mesh points of the curved surface can be obtained by the tools provided in the Rhino software, there remains a disparity in the measurement accuracy between the simulated and the real-world points. Thus, the following subsection attempts to improve the simulated points approximation such that it is closer to the real-world points. Succinctly, the correction of mesh points elicited from the previous section can be automatically adjusted and calibrated by exploiting a laser sensor and robot arm with the following sections.

Extraction of surface normal’s by offsetting

In order to obtain the position for the laser scanning process using the robot arm, the offset values are computed as an aid for the coordinates elicitation, aiming to keep a constant distance along an estimated vertex normal of the surface defined by $Z = f (x, y)$ with normal vector at points $P_{0} = (x_{0}, y_{0}, f (x_{0}, y_{0}))$ defined by $\nabla N$ = $f_{x} (x_{0}, y_{0}) i + f_{y} (x_{0}, y_{0}) j - k$ . Consequently, given the subdivided grids obtained in the previous stage, the surface normal $N (x, y)$ of all the mesh points are approximated by offsetting each mesh point. Using the surface normal, all the directions within a fixed angle between each normal can be estimated to form the offset surface along the surface’s normal vector. Mathematically, the offset surface $Q (x, y)$ of the surface $R (x, y)$ is expressed as:

Q (x, y) = R (x, y) + d N (x, y)

(3)

where $d$ is the constant distance between the offset surface $R (x, y)$ that can be extracted from the HoloLens. As such, the robot’s trajectory can follow the offset surface $Q (x, y)$ obtained from equation (3).

Coordinates acquisition from the subdivided surface using a laser sensor and robot arm

In this stage, the real-world coordinates of the subdivided surface are acquired using a 2D near-range laser range sensor with the assistance of a moving robot arm. As the sensor’s displacement has been calibrated previously, the target position depicted from the robot base reference system can be directly acquired from laser sensor data. As illustrated in Figure 9, the laser sensor is mounted and fixed on the robot arm such that it can be moved over the curve surface by just controlling the robot arm and moving to the offset surface $Q (x, y)$ calculated from equation (3) with the direction parallel to normal vector direction of each grid points. In addition, the orientation pose from the normal vector on the surface of the offset is applied to the robot arm to obtain the desired pose through the real-time laser scanning process. Then, the laser distances $d_{L} (x, y)$ at each grid point $(x, y)$ on offset surface $Q (x, y)$ to the metal surface $G (x, y)$ in the real-world are extracted.

Figure 9.

The laser sensor mounted on the robot arm for distance measurement.

3D metal surface reconstruction correction by using laser distance of grid points

In this section, the precise coordinates of the metal surface in the real world are calculated by adjusting the offset surface calculated from Section 2.2.5 using the laser distance obtained in Section 2.2.6. Hence, the parametric form of the metal surface is calculated by the equation below:

G (t) = Q (t) + d_{L} N (t)

(4)

where $N (t)$ is the normal vector of offset surface $Q (t)$ . Finally, the Loft function in the normal algorithm in Rhino software is implemented to the $G (t)$ approximated to form the desired simulated surface of the metal plate.

Experiment setting

To investigate the difference between the simulated coordinates manipulated on Rhino software and the real-world coordinates of the curved surface, a comparison of their mesh points coordinates is calculated. The procedure elaborated in Section 2 is repeated 10 times to produce different curved surfaces. The reason for repeating the experiments is to provide relatively conclusive evidence as it can enhance the data generalization and reliability. Concisely, the curved surface can be created by configuring the amount of translational movement and orientation rotation of one of the robot arms. Specifically, the following analysis investigates the robustness of the proposed method from two scenarios, viz, large and small twisting effects, denoted as $tw_l$ and $tw_s$ , respectively. The example for both the large and small twisting metal boards is illustrated in Figure 10. Therefore, there are five experimental data for large twisting while the remaining five data are for the small twisting effect. All the coordinates (i.e. simulated and real-world) collected are recorded and reported. On top of that, both the statistical tests and descriptive analyses are carried out to describe the data concisely and understandably. The following subsections discuss the metrics used herein.

Figure 10.

Illustration of (top) small and (bottom) large twisting effects.

Statistical tests

To measure the robustness of the proposed method, the simulated surface output generated by the proposed method is compared to the surface estimated by the HoloLens. In short, all 152 points are compared correspondingly. In total, six quantifying distance metrics are introduced to validate the effectiveness of the proposed method. Concisely, the distance measurement metrics include Hausdorff Distance (HD),¹⁹ Partial Curve Mappingx (PCM),²⁰ Discrete Frechet Distance (DFD),²¹ area,²² Curve Length (CL),²³ and Dynamic Time Warpingy (DTW).²⁴ Noticed that, the closer the distance computed, indicating the better the performance of the proposed method.

1. Hausdorff Distance (HD):

The distance measurement between the two discrete surfaces $S$ and $S'$ is formulated as:

d (S, S') = \max [\max_{p \in S} [\min_{p' \in S'} | | p - p' {| |}_{2}]] for p \in S

(5)

where $| | . | |_{2}$ denotes the Euclidean norm.

2. Partial Curve Mappingx (PCM):

This measurement utilizes the concept of matching the area of a subset between the two curves and the corresponding computed curve is updated iteratively. The calculation of PCM computes the curve mismatch error as in equation (6):

PCM = \underset{p}{argmin} Σ_{i = 1}^{m} \frac{d_{i} + d_{i - 1}}{2} \times s_{i},

(6)

where $d_{i} = \sqrt{{(ζ_{i} ″ - ζ_{i})}^{2} + {(η_{i} ″ - η_{i})}^{2}}$ , $ζ$ is the normalized points coordinates of $X$ -axis, $η$ is the normalized points coordinates of $Y$ -axis, $s$ is the segment length of a part of the curve between two consecutive points and they are scaled to the total length as defined in equation (7) and $v_{1} = 0$ .

s_{i} = \frac{\sqrt{{(ζ_{i} ″ - ζ_{i - 1})}^{2} + {(η_{i} ″ - η_{i - 1})}^{2}}}{Total length of S}

(7)

Details of the algorithm to calculate PCM can be found in Witowski and Stander.²⁰

3. Discrete Frechet Distance (DFD):

To calculate the shortest distance in-between two curves²¹ and the formulations are defined as defined in equation (8).

d_{F} (S, S') = in f_{α, β \in [0, 1]} Ma x_{t \in [0, 1]} d (S (α (t)), S' (β (t)))

(8)

4. Area:

To calculate the area between two curves in 2D space as referring to Jekel et al.,²² the area between two curves can be estimated using a trapezoid of the coordinates of the points from two curves as defined in equation (9).

Area = Σ_{i - 1}^{n - 1} [(d x_{i + 1} - d x_{i}) \times (d y_{i + 1} + d y_{i})]

(9)

where $d x_{i}$ and $d y_{i}$ are the distance of points coordinates of $x$ and $y$ at $i$ , respectively.

5. Curve Length (CL):

This method assumes that the only true independent variable of the curves is the arc-length distance along the curve from the origin.

C L_{S} = Σ_{i = 1}^{N} \sqrt{{(x_{i + 1} - x_{i})}^{2} + {(y_{i + 1} - y_{i})}^{2}} .

(10)

6. Dynamic Time Warpingy (DTW):

This metric calculates the non-metric distance between two time-series curves based on Levenshtein distance by recursively computing the equation below:

D (S, S') = δ (a_{i}, b_{j}) + \min d_{θ} (S, S')

(11)

where $d_{θ} (X, Y)$ is the average accumulated distortion between the warped time series $X$ and $Y$ , defined as:

d_{θ} (X, Y) = Σ_{k = 1}^{T} d (θ_{x} (k), θ_{y} (k)) m_{θ} (k) / M_{θ}

(12)

where $m_{θ} (k)$ is a per-step weighting coefficient and $M_{θ}$ is the corresponding normalization constant.

Descriptive analysis

To summarize the entire mesh points obtained from both the simulated and real-world calibrated coordinates, an analysis is performed subsequently, namely, the descriptive analysis. Concisely, two key metrics are considered, viz, mean and standard deviation. The mean, also known as the average, is calculated by adding all the distances obtained between the two coordinate sets and then dividing by the number of points. On the other hand, the standard deviation is the most commonly used measure of dispersion as it concludes the spread of data points about the mean. The equations of mean and standard deviation are depicted as follows:

mean = \frac{Σ X}{N}

(13)

SD = \sqrt{\frac{Σ | x - \bar{x} |^{2}}{n}}

(14)

To further examine the effectiveness of the proposed method, the rectangular prism-shaped metal board is partitioned into 19 segments along the longitudinal direction. Since there are 152 points on the metal board, each of the eight points is viewed as a composite segment, as illustrated in Figure 11. As such, more local region information can be extracted and analyzed. From this analysis, it is hypothesized that the middle part of the metal board has the largest curvature, leading to the highest erroneous. This is because the deformation of the metal plate seems obvious when the force is applied from one side, while the other side is held constant. To reveal the deformation characteristics and verify the accuracy of each measured mesh point, a detailed analysis is reported in Section 4 later.

Figure 11.

The metal that segmented into 19 × 8.

Results and discussion

This section highlights some important findings observed throughout the experiment, including quantitative and qualitative analyses. Concretely, Table 1 presents the mean of the six distance measurements discussed in Section 3.1, which computed the distance difference between each pair of the Rhino simulated and the laser sensor-derived mesh point coordinates. Note that Table 1 portrays the average results for both the scenarios of large and small twisting effect, denoting as $tw_l$ and $tw_s$ , respectively. To provide an intuitive visualization, the results in Table 1 can be interpreted using line graphs, as depicted in Figure 13. It is obvious that the six sub-figures reveal some interesting insights into different metal board segments.

Table 1.

Mean results for the six distance measurement metrics when comparing the points between the Rhino simulated and laser sensor derived points for the metal boards with large twisting ( $tw_l$ ) and small twisting ( $tw_s$ ) effects.

HD			PCM		DFD		AREA		CL		DTW
Segment	$tw_l$	$tw_s$	$tw_l$	$tw_s$	$tw_l$	$tw_s$	$tw_l$	$tw_s$	$tw_l$	$tw_s$	$tw_l$	$tw_s$
1	52.56	40.26	2.53	2.57	52.56	40.26	22,830.34	17,370.59	0.03	0.02	358.02	275.76
2	54.77	42.53	3.54	3.64	54.77	42.53	21,117.13	19,704.97	0.03	0.03	345.00	306.40
3	50.36	38.63	0.92	1.83	50.36	38.63	16,711.53	17,526.31	0.03	0.03	296.65	283.39
4	32.78	35.78	0.58	2.75	32.78	35.78	9807.94	14,196.65	0.02	0.02	195.09	250.68
5	30.32	37.13	0.27	1.98	30.32	37.13	6893.76	13,217.41	0.02	0.02	173.54	252.82
6	34.06	31.79	0.37	2.01	34.06	31.79	7102.24	9707.74	0.02	0.02	204.77	206.91
7	33.16	32.38	0.30	1.04	33.16	32.38	6225.63	7858.83	0.02	0.02	200.43	197.02
8	31.72	29.09	0.31	2.78	31.72	29.09	4728.02	5721.97	0.01	0.01	188.16	169.19
9	31.93	22.68	0.23	0.50	31.93	22.68	3802.00	3171.88	0.01	0.01	184.51	117.59
10	27.64	15.89	0.23	0.71	27.64	15.89	2474.03	1437.84	0.01	0.01	154.06	82.27
11	23.05	11.23	0.10	0.10	23.05	11.23	1092.70	402.16	0.01	0.00	117.61	50.53
12	18.17	11.28	0.07	0.04	18.17	11.28	491.51	80.16	0.00	0.00	94.74	48.11
13	14.59	14.69	0.04	0.45	14.59	14.69	369.89	623.81	0.00	0.00	70.78	67.67
14	22.88	21.33	0.19	0.65	22.88	21.33	1645.43	2509.74	0.02	0.03	110.42	127.80
15	28.11	23.79	0.46	2.23	28.11	23.79	4116.93	4181.57	0.10	0.13	175.87	147.94
16	29.54	27.09	0.67	1.99	29.54	27.09	6092.96	6354.72	0.57	0.37	189.25	183.02
17	32.88	33.01	1.67	35.88	32.88	33.01	7605.09	9818.46	0.12	0.14	188.80	225.37
18	33.73	35.65	2.96	16.70	33.73	35.65	8185.29	11,690.85	0.08	0.10	182.35	237.35
19	41.21	40.55	16.56	13.95	41.21	40.55	14,236.97	15,649.78	0.08	0.09	282.73	296.91

Concretely, it can be seen from Figure 13 that the trend line for the HD, DFD, DTW, and area are similar. For instance, the distance measures for the third quarter (segments #9–14) of the metal board have relatively lower measurement distances. This is tallying to the values generated by CL in Table 1, whereby the (segments #9–14) produce the smallest distance difference ranging from 0 to 0.01. This implies that the twisting effect is relatively low in the third quarter of the metal board.

On the other hand, the sub-figures for PCM and CL have obvious spikes at the fourth quarter metal board segment (#15–19), indicating a large estimation difference between the simulated and real-world mesh point coordinates. This is because of the characteristics of the compound material of the metal board. Since the metal board comprises four pieces of metal plate, a large twisting effect causes a significant protruding impact on the fourth metal board segment, as shown in Figure 12. Nevertheless, the rest of the segments can approximate the coordinate points accurately.

Figure 12.

The uneven surface curve of the metal board.

Besides, this study also demonstrates that the proposed method works can be applied to both large and small twisting angles. This can be evidenced by the sub-figures in Figure 13 that the trend of the two lines is similar. In addition, the corresponding standard deviation between all the 152 pairs of points is averaged and reported in Table 2. Figure 14 provides a clear visualization regarding the impact in terms of standard deviation for the different segments when comparing large and small twisting metal boards. It is noticed in Figure 14 that the first half of the metal board has significantly large standard deviation values when applying a larger twisting effect. Interestingly, it can be seen that for the PCM metric, there is an obvious spike at segment #17 for the small twisting metal board due to the protruding effect of the metal materials.

Figure 13.

The mean results of different distance measurement metrics of the large and small twisting metal boards when comparing the simulated and derived points: (a) Hausdorff Distance (HD), (b) Partial Curve Mappingx (PCM), (c) Discrete Frechet Distance (FREC), (d) area, (e) Curve Length (CL), and (f) Dynamic Time Warping (DTW).

Table 2.

Standard deviation results for the six distance measurement metrics when comparing the points between the Rhino simulated and laser sensor derived points for the metal boards with large twisting ( $tw_l$ ) and small twisting ( $tw_s$ ) effects.

HD			PCM		DFD		AREA		CL		DTW
Segment	$tw_l$	$tw_s$	$tw_l$	$tw_s$	$tw_l$	$tw_s$	$tw_l$	$tw_s$	$tw_l$	$tw_s$	$tw_l$	$tw_s$
1	19.28	5.51	2.08	3.08	19.28	5.51	8352.81	2501.30	0.01	0.00	122.63	37.20
2	22.13	8.82	3.15	3.72	22.13	8.82	9935.70	3961.31	0.01	0.01	145.89	66.04
3	11.65	6.65	0.65	0.83	11.65	6.65	5776.58	2701.14	0.01	0.00	84.99	43.12
4	11.78	2.43	0.35	2.42	11.78	2.43	4216.13	1473.88	0.01	0.00	66.27	30.37
5	8.00	5.28	0.10	1.53	8.00	5.28	2158.20	1247.08	0.00	0.00	38.43	27.93
6	8.51	5.26	0.05	1.30	8.51	5.26	1192.86	1457.09	0.00	0.00	50.18	26.09
7	9.92	4.54	0.07	0.37	9.92	4.54	1649.51	1388.13	0.00	0.00	59.66	35.78
8	9.31	5.82	0.09	3.05	9.31	5.82	1383.07	1276.84	0.00	0.00	53.59	30.87
9	9.74	3.90	0.04	0.15	9.74	3.90	1239.75	811.56	0.00	0.00	56.89	24.98
10	11.23	4.82	0.16	0.50	11.23	4.82	1301.63	521.62	0.00	0.00	63.13	22.56
11	8.30	2.55	0.04	0.07	8.30	2.55	699.06	177.85	0.00	0.00	53.55	8.50
12	6.76	4.08	0.04	0.03	6.76	4.08	262.49	30.69	0.00	0.00	42.41	12.42
13	4.52	4.43	0.03	0.59	4.52	4.43	252.51	272.57	0.00	0.00	26.88	19.87
14	4.88	3.38	0.18	0.32	4.88	3.38	1007.49	638.22	0.01	0.01	52.15	24.35
15	4.89	3.44	0.30	2.63	4.89	3.44	1088.35	988.63	0.04	0.04	30.87	26.98
16	3.90	4.58	0.39	1.13	3.90	4.58	1268.86	983.27	0.30	0.05	33.31	23.32
17	5.73	3.18	1.40	65.41	5.73	3.18	1891.35	1155.86	0.03	0.01	52.40	23.31
18	3.96	3.58	2.02	11.25	3.96	3.58	2540.86	1678.63	0.01	0.01	51.94	27.81
19	3.42	2.86	7.01	6.94	3.42	2.86	570.43	1208.62	0.00	0.01	15.93	17.21

Figure 14.

The standard deviation results of different distance measurement metrics of the large and small twisting metal boards when comparing the simulated and derived points: (a) Hausdorff Distance (HD), (b) Partial Curve Mappingx (PCM), (c) Discrete Frechet Distance (FREC), (d) area, (e) Curve Length (CL), and (f) Dynamic Time Warping (DTW).

Figure 15 portrays the curved surface of the metal board. The red dots denote the coordinates of the Rhino simulated points whereas the real-world coordinates are annotated as blue crosses. This visual representation further affirms the results tabulated in Table 1, where the pair of points at certain parts metal board has a relatively larger distance than the other parts. Thus, these findings have important implications for developing an automatic scanning system capable of reconstructing accurate, dense, and pixel-wise surface curvature. To verify the accuracy of this system, the paper uses a custom-made extruder attached to the end of the robotic arm that prints 3D objects in varying freeform surfaces, as depicted in Figure 16. Moreover, the final design of 3D objects as depicted in Figure 17.

Figure 15.

The point coordinates of the metal board when comparing the (red) simulated and (blue) real-world scenarios.

Figure 16.

3D printing on freeform molds.

Figure 17.

The point coordinates of the metal board when comparing the (red) simulated and (blue) real-world scenarios.

Nevertheless, this observational study raises intriguing questions regarding different types of deformations. In future investigations, one of the potential directions is to reduce the control points in increasing the curvature of the surface. Besides, other types of material with distinct properties can be considered to replace the metal board, like fabric, wood, and plastic. For the apparatus used, the usage of HoloLens 2 can be replaced with other types of devices such as a CCD or CMOS camera that is cost-effective. In addition, to resolve the protruding effect from the experiment results obtained, a comprehensive study regarding the twisting angle of the board can be investigated to further extend the generalization of the proposed method.

Conclusion

Despite estimating the geometry of a huge and freeform object being relatively crucial due to the hardware limitations such as the portability and the detection range of the sensors, this study proposes an end-to-end real-time, and accurate surface reconstruction system. The effectiveness of the proposed method is verified when evaluating different curvatures with unique 3D printing of irregular objects on a freeform surface. The empirical findings in this study demonstrate that the simulated surface visualized on the Rhino software has a significant difference compared to the real-world coordinates. Therefore, the insights gained from this study may be of assistance to suggest an effective solution for simulating the desired shape. Succinctly, the numerical performance and the supplement graphical representations reported in the paper summarized from the 152 pairs of points demonstrate the viability of the proposed method. Concretely, six distance measurements (i.e. Hausdorff Distance, Partial Curve Mappingx, Discrete Frechet Distance, area, Curve Length (CL), and Dynamic Time Warpingy) conveyed the reasonably promising result that primarily retrieved the curvature information via the HoloLens and ArUco marker. Thereupon, future directions point toward a detailed investigation into the combination of machine learning techniques with simulation modeling that aims in refining the precise coordinates of a freeform surface. Through the training process of the machine learning model, the simulated curvature can be enhanced by making it closer to the real object. However, it requires a large number of data sets that are sufficient to build a robust machine learning model.

Footnotes

Handling Editor: Chenhui Liang

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was funded by Ministry of Science and Technology (MOST) (Grant Number: MOST 110-2222-E-035-003-MY2, 110-2222-E-035 -003 -MY2, 111-2221-E-035-059-MY3, and 111-2221-E-A49-034-MY2).

ORCID iDs

Shih-Yuan Wang

Yee-Siang Gan

References

Shi

Sun

Bai

. Large-scale three-dimensional measurement via combining 3D scanner and laser rangefinder. Appl Opt 2015; 54: 2814–2823.

Bai

Shi

, et al. Scanning-based 3D reconstruction of large-scale objects. In: Proceedings of the 2019 International conference on robotics, intelligent control and artificial intelligence, 2019, pp. 584–589. Shanghai: Association for Computing Machinery New York, United States.

Roussel

Geiger

Fischbach

, et al. 3D surface reconstruction of plant seeds by volume carving: performance and accuracies. Front Plant Sci 2016; 7: 745.

Liong

Gan

Huang

. Automatic surface area and volume prediction on ellipsoidal ham using deep learning. J Food Process Eng 2019; 42: e13093.

Gan

Wei

Han

, et al. A statistical approach in enhancing the volume prediction of ellipsoidal ham. J Food Eng 2021; 290: 110186.

Gkioxari

Dollár

, et al. Mask R-CNN. In: Proceedings of the IEEE international conference on computer vision, 2017, pp. 2961–2969. Venice: IEEE.

Hoang

D-C

Lilienthal

Stoyanov

. Object-RPE: dense 3D reconstruction and pose estimation with convolutional neural networks. Robot Auton Syst 2020; 133: 103632.

Hosseininaveh

Sargeant

Erfani

, et al. Towards fully automatic reliable 3D acquisition: from designing imaging network to a complete and accurate point cloud. Robot Auton Syst 2014; 62: 1197–1207.

Zhao

Zhang

, et al. Development of a robotic 3D scanning system for reverse engineering of freeform part. In: 2008 International conference on advanced computer theory and engineering, Phuket, Thailand, 20–22 December 2008, pp.246–250. New York, NY: IEEE.

10.

Hosseininaveh

Yazdan

Karami

, et al. A low-cost and portable system for 3D reconstruction of texture-less objects. Int Arch Photogramm Rem Sens Spat Inf Sci 2015; XL-1/W5: 327–332.

11.

Hafeez

Kwon

S-C

Lee

S-H

, et al. 3D surface reconstruction of smooth and textureless objects. In: 2017 International conference on emerging trends & innovation in ICT (ICEI), Pune, India, 3–5 February 2017, pp.145–149. New York, NY: IEEE.

12.

Ahmadabadian

Yazdan

Karami

, et al. Clustering and selecting vantage images in a low-cost system for 3d reconstruction of texture-less objects. Measurement 2017; 99: 185–191.

13.

Gan

Chen

Yau

, et al. 3D socnet: deep 3D reconstruction network based on self-organizing clustering mapping. Expert Syst Appl 2023; 213: 119209.

14.

Hosseininaveh Ahmadabadian

Karami

Yazdan

. An automatic 3D reconstruction system for texture-less objects. Robot Autonom Syst 2019; 117: 29–39.

15.

Lyngby

Nielsen

De Chiffre

, et al. Development and metrological validation of a new automated scanner system for freeform measurements on wind turbine blades in the production. Precis Eng 2019; 56: 255–266.

16.

Garrido-Jurado

Muñoz-Salinas

Madrid-Cuevas

, et al. Automatic generation and detection of highly reliable fiducial markers under occlusion. Pattern Recognit 2014; 47: 2280–2292.

17.

Kalachev

. ArUco markers generator, https://chev.me/arucogen/ (2016).

18.

Wang

S-Y

Sheng

Y-T

Frank

. Apps for rhino and grasshopper, https://www.food4rhino.com/ (2020).

19.

Aspert

Santa-Cruz

Ebrahimi

. Mesh: measuring errors between surfaces using the hausdorff distance. In: Proceedings. IEEE international conference on multimedia and expo, Lausanne, Switzerland, 26–29 August 2002, vol. 1, pp.705–708. New York, NY: IEEE.

20.

Witowski

Stander

. Parameter identification of hysteretic models using partial curve mapping. In: 12th AIAA Aviation technology, integration, and operations (ATIO) conference and 14th AIAA/ISSMO multidisciplinary analysis and optimization conference, 2012, p.5580. Indianapolis: AIAA.

21.

Eiter

Mannila

. Computing discrete fréchet distance,Tech. rep., Cite-seer (1994).

22.

Jekel

Venter

, et al. Similarity measures for identifying material parameters from hysteresis loops using inverse analysis. Int J Mater Forming 2019; 12: 355–378.

23.

Andrade-Campos

de-Carvalho

Valente

. Novel criteria for determination of material model parameters. Int J Mech Sci 2012; 54: 294–305.

24.

Sakoe

. Dynamic-programming approach to continuous speech recognition. In: 1971 Proceedings of the international congress of acoustics, Budapest, 1971.

3D printing on freeform surface: Real-time and accurate 3D dynamic dense surface reconstruction with HoloLens and displacement measurement sensors

Abstract

Keywords

Classifications

Introduction

Theory and design

Apparatus preparation

Metal sheet and robot arm

Placement of ArUco markers

Robot calibration

Data elicitation

Translation and rotation of metal board

Marker information retrieval using HoloLens

Construction of curves using Rhino

Simulation and refinement of metal board surface

Extraction of surface normal’s by offsetting

Coordinates acquisition from the subdivided surface using a laser sensor and robot arm

3D metal surface reconstruction correction by using laser distance of grid points

Experiment setting

Statistical tests

Descriptive analysis

Results and discussion

Conclusion

Footnotes

Declaration of conflicting interests

Funding

ORCID iDs

References