Sage Journals: Discover world-class research

Abstract

Computer vision systems have demonstrated to be useful in applications of autonomous navigation, especially with the use of stereo vision systems for the three-dimensional mapping of the environment. This article presents a novel camera calibration method to improve the accuracy of stereo vision systems for three-dimensional point localization. The proposed camera calibration method uses the least square method to model the error caused by the image digitalization and the lens distortion. To obtain particular three-dimensional point coordinates, the stereo vision systems use the information of two images taken by two different cameras. Then, the system locates the two-dimensional pixel coordinates of the three-dimensional point in both images and coverts them into angles. With the obtained angles, the system finds the three-dimensional point coordinates through a triangulation process. The proposed camera calibration method is applied in the stereo vision systems, and a comparative analysis between the real and calibrated three-dimensional data points is performed to validate the improvements. Moreover, the developed method is compared with three classical calibration methods to analyze their advantages in terms of accuracy with respect to tested methods.

Keywords

Computer vision stereo vision calibration methods 3-D point localization autonomous navigation

Introduction

Nowadays, applications like manufacturing process, structural health monitoring, microsurgery, laparoscopic surgery, and specially in autonomous navigation have used three-dimensional (3-D) measuring techniques.^1

–5 In these applications, the accuracy is essential for the tasks that must be developed; therefore, there are methods to improve the accuracy of the 3-D measurements.^6,7 In autonomous navigation systems, the aim is to move an autonomous object through a 3-D environment with classic interpolation methods (inertial sensors) or external references (computer vision, ultrasonic sensors, GPS, CCD, and CMOS).^8

–11 Recently, research in autonomous navigation applications has focused on stereo vision, which is used for 3-D mapping, detection, and location of objects.^12
–14 Advantages of the stereo vision are the portability and the wide field of view (FOV) due to the use of cameras, obtaining more information of the environment to scan compared with other 3-D measuring techniques, for example, time of flight, pulse modulation method, dynamic laser triangulation, among others.^15
–17 Stereo vision systems (SVS) process visual information from two or more cameras to obtain features of a specific scene. The SVS setup employs two cameras in which each camera captures images from different perspectives.¹⁸ In each stereo image, the corresponding points between them are detected, and finally, a triangulation process is performed with each corresponding pair of points.^19
–21 The two main reasons of loss of accuracy in SVS for 3-D measurements are the loss of information due to the image digitalization and the lens distortion.²² The loss of information in image digitalization entails that specific areas of the images have low quality in terms of brightness, sharpness, and contrast, causing a difficult and inaccurate localization of the 3-D points in these areas. On the other hand, the lens distortion causes deformation in the straight lines of the images, resulting on inaccurate measurements of the 3-D points in SVS. These errors are minimized using camera calibration methods, where intrinsic and extrinsic parameters of the cameras are obtained.²³ The mentioned parameters are the position and orientation of the cameras, focal distance, optical center, and lens distortion coefficients, where their estimation entails a high computational cost because it is necessary to perform many steps to obtain them. The novel camera calibration method for SVS is developed with the purpose of locating 3-D points of a specific scene without estimating these parameters, obtaining a low computational cost. Moreover, the developed method is created to have higher accuracy in the localization of 3-D points than other calibration methods that use the estimation of the mentioned parameters. The developed SVS prototype is presented in Figure 1; it performs the developed camera calibration method, high-speed pattern recognition feature, and triangulation process to obtain the 3-D points coordinates. The proposed calibration method uses the least square technique, in which an equation is obtained to compensate the loss of information due to the image digitalization and the errors generated by the lens distortion. The presented article shows the implementation of an SVS to locate 3-D points coordinates in real-time applications and a novel calibration method to improve the accuracy of the 3-D points coordinates. Experimentations are performed, where 3-D points coordinates of a specific scene are located at different depths, obtaining databases of 3-D points coordinates with the implemented SVS and, afterward, databases including the calibration method on the SVS, comparing the results to validate the improvements of the developed calibration method. Moreover, an experimentation where a comparative analysis is performed between the developed calibration method with other three classical calibration methods is presented. In this experimentation, the errors obtained by each calibration method tested in planes XY, XZ, YZ, and XYZ are compared.

Figure 1.

Developed SVS with interface in LabVIEW. SVS: stereo vision systems.

Image digitalization errors and lens distortion

Image digitalization is employed a CCD or CMOS sensor, which gets the light information from the scene, converting it into electric signals. The intensity of each electrical signal will depend on the amount of light obtained by the sensor in different parts of the scene.²⁴ These signals are amplified and converted into digital signals, creating a bitmap (pixels) with the digital information of the scene. In this process, there is a loss of information due to signal noise, the conversion of light information to electric signals made by the sensor, and the conversion of the analog signal to digital signal made by the analog–digital converter.²⁵ Generally, in SVS, a matching process of a specific region is performed between images from different perspectives in order to locate 3-D points of a scene. A low quality of the images due to the loss of information can cause mismatch of the specific region between the images and also entails to inaccurate measurements of the 3-D points. Figure 2 shows an example of how the errors of image digitalization affect the SVS in this matching process.

Figure 2.

Template matching in stereoscopic images. (a) The template was successful located in both images despite the quality between the images. (b) The template was found only in left image, the low quality entails the mismatch in right image.

Another problem that occurs in cameras is the lens distortion. The problem consists in the curvature of the straight lines presented in the image due to the camera lens. This curvature can be presented in three forms: barrel distortion, pincushion distortion, and mustache distortion.²⁶ Figure 3 shows the types of distortions that can be presented in camera lens. The barrel distortion is seen on wide angle lenses, where the FOV of the lens is wider than the size of the image sensor, resulting in straight lines curved inward. In the opposite way, the pincushion distortion is seen on telephoto lens, where the FOV of the lens is smaller than the size of the image sensor, resulting in straight lines curved outward. In addition, the mustache distortion appears in several lens with variable FOV. In this distortion, the straight lines are seen curved inward toward the center of the image and curved outward at the extreme corners of the image. The lens distortion in SVS causes inaccurate measurements in the 3-D points localization, especially toward the extreme edges of the images where the distortion affects more.

Figure 3.

Types of lens distortion: (a) Non-distortion. (b) Barrel distortion. (c) Pincushion distortion. (d) Mustache distortion.

SVS implementation

The SVS obtains 3-D information from two images captured from two different cameras separated by a known distance. Similar design of SVS can be found in the literature,^27
–29 The developed computer program for 3-D point localization using SVS can be divided into five steps: images capture, camera calibration, pattern match, computing pixel coordinates to angles, and triangulation. Figure 4 shows the localization of a 3-D point in the scene using the developed SVS. The coordinates $(P l_{i}, P l_{j})$ and $(P r_{i}, P r_{j})$ are the pixel coordinates of the projection points in the left and right camera, respectively, over the 3-D point located in the coordinates x, y, and z. Angles $B_{i j}$ , $C_{i j}$ , and $β_{i j}$ are related to the 3-D point and its projections in the images. Variable b is the fixed distance between the two cameras which is named as base line.

Figure 4.

Localization of a 3-D point in SVS. SVS: stereo vision systems; 3-D: three-dimensional.

Specifications of the SVS

The developed SVS in a particular 3-D point is presented in Figure 5, where b is the base line, l is the distance from the base line to the scene, and $B_{i j}$ and $C_{i j}$ are the left and right angles of the cameras, respectively. The design of the developed SVS has two cameras horizontally aligned between them (as shown in Figure 5), with their image planes coplanar from each other and separated by the base line b. This configuration is an epipolar geometry between both cameras, which ensures a same height of the pair of points in the vertical coordinates of the images.³⁰ The SVS uses two identical high definition cameras with an image dimension of 1626 $\times$ 1236 px² and a frame rate of 20 fps; both cameras are separated by a base line of 6.35 cm and the main optical axis of the cameras is parallel to each other. Thereby, the particular SVS configuration presents FOV values of 46.32° in the horizontal FOV (0.028° per pixel) and 36.01° in the vertical FOV (0.029° per pixel).

Figure 5.

Developed SVS with the angle location in a particular 3-D point. SVS: stereo vision systems; 3-D: three-dimensional.

Images capture

The first step of the developed SVS is to obtain visual information through the image capture of both cameras which will be processed to obtain 3-D points of a particular scene. In this case, a specific test grid with known distances is used, where 3-D points will be obtained. The test grid used is shown in Figure 6, where 63 crosses along the grid can be appreciated, with 2 cm separation between them. The developed SVS locates the center of the crosses and obtains the 3-D coordinate of each one. Due to the displacement between cameras, both cameras capture a different scene,^31,32 and to perform the stereo vision technique, it is required to identify the same scene in the image pair; therefore, the developed computer program is able to choose the region of interest in both cameras. For the experimentation, the test grid is placed in the middle of the cameras (as shown in Figure 5); using a distance from the base line to the test grid at 22.87 cm, the developed computer program locates the same scene with a slight displacement in both images, where 49 of the 63 crosses are observed.

Figure 6.

Test grid used for the experimentation.

Camera calibration

The developed calibration method used in SVS consists of four main steps:

Step 1: Estimate the horizontal and vertical angles for each of the crosses in the calibration grid.

Step 2: Use the developed SVS to estimate the same horizontal and vertical angles with the information of left and right camera.

Step 3: With the estimated angles, find the adjustment angles $Δ B_{i j}$ , $Δ C_{i j}$ , and $Δ β_{i j}$ using the least square method.

Step 4: Adjust the angles $B_{i j}$ , $C_{i j}$ , and $β_{i j}$ (obtained by the SVS) by adding each $Δ B_{i j}$ , $Δ C_{i j}$ , and $Δ β_{i j}$ to its respective angle.

In the first step, it is required to obtain the corresponding horizontal and vertical angles for each of the crosses in the calibration grid. The angles can be obtained according to the horizontal and vertical separation that the points have between them for any calibration grid. Figure 7 shows the developed calibration grid, using a total of 285 crosses with a separation of 1 cm between them.

Figure 7.

Developed calibration grid with origin in the center.

The angles $α_{i j}$ (horizontal angles) and $υ_{i j}$ (vertical angles) for each cross are obtained by the following equations

α_{i j} = arctan (\frac{l}{x_{i}})

υ_{i j} = arctan (\frac{l}{y_{j}})

where l is the distance from the base line to the calibration grid, and (x_i , y_j ) are the coordinates of the located points with origin in the center of the calibration grid. To perform the developed calibration, the calibration grid must be perpendicular to the cameras. In the developed SVS, the cameras are parallel to a metal base which is perpendicular to the surface where the calibration grid is located. With these conditions, the SVS is able to locate 3-D points with great accuracy from any point of view (in the FOV) even if the scene is not perpendicular to the cameras. The second step is to obtain the angles of the corresponding two-dimensional points on the left and right camera using the developed SVS. Figure 8 shows the angle location of the 3-D point projection in both images.

Figure 8.

Angle location of the 3-D point projection in both images. 3-D: three-dimensional.

To obtain the angles on the left camera, the center of calibration grid and the center of left camera must be orthogonally aligned. Afterward, pattern matching is used to obtain the pixel coordinates of the points. Considering the origin coordinate in the center of left camera, the angles $φ_{i j}$ (horizontal measurements of left angles) and $γ_{i j}$ (vertical angle measurements) of the points are obtained in the following equations

φ_{i j} = (P l_{i} - \frac{D_{h}}{2}) \cdot R_{h}

γ_{i j} = (\frac{D_{v}}{2} - P l_{j}) \cdot R_{v}

where $P l_{i}$ and $P l_{j}$ are the horizontal and vertical pixel positions of the 3-D point projection in the left image, respectively; R_h and R_v are the angular gradients per pixel in horizontal and vertical directions, respectively; and D_h and D_v are the horizontal and vertical dimensions in pixels of the images, respectively. To obtain the angles on the right camera, it is necessary to align orthogonally the center of calibration grid and the center of the right camera and, afterward, perform pattern matching to obtain the pixel coordinates of the points. Equation (5) yields the angle $ϕ_{i j}$ (horizontal measurements of right angles) relative to the right camera

ϕ_{i j} = (P r_{i} - \frac{D_{h}}{2}) \cdot R_{h}

where $P r_{i}$ is the horizontal pixel position of the 3-D point projection in the right image. For the vertical angles in the right camera, the resulting vertical angles from the left camera are used as the cameras are horizontally aligned and their image planes are coplanar from each other, obtaining the same vertical angle measurements $γ_{i j}$ in both images. With the obtained angles, the third step is to find the adjustment angles that minimize the following square absolute errors in the horizontal and vertical angles (equations (6) to (8))

min {\sum_{i = 1}^{m} \sum_{j = 1}^{n} {| α_{i j} - φ_{i j} |}^{2}}

min {\sum_{i = 1}^{m} \sum_{j = 1}^{n} {| α_{i j} - ϕ_{i j} |}^{2}}

min {\sum_{i = 1}^{m} \sum_{j = 1}^{n} {| υ_{i j} - γ_{i j} |}^{2}}

These minimization problems are solved using the least square method, where polynomial equations to find the adjustment angles $Δ B_{i j}$ , $Δ C_{i j}$ , and $Δ β_{i j}$ are obtained.³³ The adjustment angles will be necessary to correct the angles of all points located in both images. Least square method can obtain any order polynomial equations, in the proposed calibration method is employed until third-order polynomial equations (equations (9) to (11))

Δ B_{i j} = \sum_{n = 0}^{3} A_{n} \cdot φ_{i j}^{n}

Δ C_{i j} = \sum_{n = 0}^{3} B_{n} \cdot ϕ_{i j}^{n}

Δ β_{i j} = \sum_{n = 0}^{3} C_{n} \cdot γ_{i j}^{n}

where A_n , B_n , and C_n are the adjustment coefficients of the functions obtained by the least square method. In the final step, the angles $B_{i j}$ , $C_{i j}$ , and $β_{i j}$ are adjusted by adding each $Δ B_{i j}$ , $Δ C_{i j}$ , and $Δ β_{i j}$ to its respective angle, obtaining the calibrated angles $B c_{i j}$ , $C c_{i j}$ , and $β c_{i j}$ (equations (12) to (14))

B c_{i j} = B_{i j} + Δ B_{i j}

C c_{i j} = C_{i j} + Δ C_{i j}

β c_{i j} = β_{i j} + Δ β_{i j}

Pattern match

In this step, the central coordinates in pixels of all the crosses of the test grid are located for both cameras. To estimate the coordinates, pattern matching method (also called area-based image match) is employed. This match method has been widely studied and applied in the literature.^34

–37 In pattern matching, a correlation window of brightness (intensity) patterns is performed between a template image and the two images, providing the regions in each image that has the most correspondence brightness with respect to the template image.³⁸ The developed computer program uses an image template with dimensions of 20 pixels per side with a cross in the middle. This template is used to locate regions with a similar cross in the middle of 20 pixels per side in the images from both cameras. When the square regions are located, the program estimates the central coordinates in pixels of all the located regions (the coordinates match with the center of the cross in each region). The developed computer program uses a score to identify how closely the template image matches in different regions of both images (score range is between 0 and 1000). The minimum score tolerance to find the pattern matches in the developed program is 800. Common problems presented in the pattern matching are occlusions which occurs when a particular region of the scene is observed in one image but not in the other.³⁹ Priya and Anand⁴⁰ focus on the problem of occlusion and provide a solution to avoid them through a novel modified geometric mapping technique. In the developed program, if the match score is low, or an occlusion is presented, the localized point is not considered in the next step.

Computing pixel coordinates to angles

As shown in Figure 4, the angles $B_{i j}$ , $C_{i j}$ , and $β_{i j}$ are required to obtain the coordinates of the 3-D points. The coordinates in pixels of each 3-D point projections on the pair images estimated in the pattern match are used to calculate the angles. To obtain the angle $B_{i j}$ , it is required the horizontal dimension D_h of the cameras, the horizontal resolution R_h , and the horizontal pixel position $P l_{i}$ of the 3-D point projection in the left image. Equations (15) and (16) show the value of the angle $B_{i j}$ when the 3-D point projection is located on the left or right side of the center image

B_{i j} {= 90}^{\circ} + {(| \frac{D_{h}}{2} - P l_{i} | \cdot R_{h})}_{at P l_{i} \leq \frac{D_{h}}{2}}

B_{i j} {= 90}^{\circ} - {(| \frac{D_{h}}{2} - P l_{i} | \cdot R_{h})}_{at P l_{i} \geq \frac{D_{h}}{2}}

To obtain the angle $C_{i j}$ , the horizontal pixel position $P r_{i}$ of the 3-D point projection in the right image must be considered. An inverse relation at the angle $B_{i j}$ can be appreciated (equations (17) and (18))

C_{i j} {= 90}^{\circ} + {(| \frac{D_{h}}{2} - P r_{i} | \cdot R_{h})}_{at P r_{i} \geq \frac{D_{h}}{2}}

C_{i j} {= 90}^{\circ} - {(| \frac{D_{h}}{2} - P r_{i} | \cdot R_{h})}_{at P r_{i} \leq \frac{D_{h}}{2}}

Angle $β_{i j}$ can be obtained with the vertical dimension D_v of the cameras, the vertical resolution R_v , and the vertical pixel position $P v_{j}$ of the 3-D point projection in left or right image (equation (19))

β_{i j} = (\frac{D_{v}}{2} - P v_{j}) \cdot R_{v}

Because both cameras are placed in an epipolar lane, the vertical pixel position of the 3-D point projection in both images is the same.^41,42

Triangulation

Triangulation process is widely used in multiple applications to locate point coordinates in a scene.^43
–45 In the currently developed SVS, the triangulation is performed with the base line of the cameras and the center point position of the crosses in the test grid located in the left and right images. For each triangulation, angles $B_{i j}$ , $C_{i j}$ , and $β_{i j}$ are obtained to calculate the $x_{i j}$ , $y_{i j}$ , and $z_{i j}$ coordinates of the 3-D points in the scene. Therefore, a set of triangulation equations were developed derived from the law of sines (equations (20) to (22))^21,46,47

x_{i j} = b (\frac{sin B_{i j} \cdot sin C_{i j}}{sin (B_{i j} + C_{i j})})

y_{i j} = b (\frac{1}{2} - \frac{cos B_{i j} \cdot sin C_{i j}}{sin (B_{i j} + C_{i j})})

z_{i j} = b (\frac{sin B_{i j} \cdot sin C_{i j} \cdot tan β_{i j}}{sin (B_{i j} + C_{i j})})

Experimentation

To test the proposed calibration method, a routine has been developed in LabVIEW, which is able to find 3-D point coordinates with and without using the calibration method. The computer program is enabling to change several settings of the SVS such as camera resolution, FOV (horizontal and vertical), distance from the base line to the scene, base line, and number of rows and columns of the test grid. Also, the computer program indicates the area of interest to search in the image, time localization, and the obtained data of the 3-D points found in the scene. Figure 9 shows the processes performed by the computer program to obtain the coordinates of the points, where it can be appreciated the variables obtained in each process.

Figure 9.

Block diagram of the developed SVS. SVS: stereo vision systems.

The first experiment was developed where a surface was scanned at different distances. Figure 10 shows two same grids with a total of 100 crosses separated by a distance of 5 cm in the x-axis. The developed program performed the scan of the crosses two times, one using the developed calibration method and other without using it.

Figure 10.

Experiment SVS of a surface at different distances. SVS: stereo vision systems.

A second experiment was developed, obtaining databases of 3-D points at different distances from the base line to the test grid using the proposed method and the developed computer program. For each distance, it obtained two databases, one using the calibration method and another without using it. In the developed experimentation, it obtained 22 databases (534 measurements), changing the distance every 1 cm. For the analysis, four databases were chosen, in which the distance was changed every 3 cm, starting at a distance of 22.87 cm, where two databases with a total of 98 measurements were obtained. Other databases selected were at distances of 19.87, 16.87, and 13.87 cm, where two databases were obtained of each one. Seventy measurements were obtained at 19.87 cm, 30 measurements at 16.87 cm, and 30 measurements at 13.87 cm.

Moreover, the third experimentation was performed, comparing the developed method with the three methods that use extrinsic and intrinsic parameters to calibrate the cameras: method of Zhang,⁴⁸ method of Jia et al.,⁴⁹ and method of Cui et al.⁵⁰ For this experimentation, a database of 49 3-D points is estimated at a distance of 22.87 cm using the developed calibration method and the previously mentioned calibration methods in the SVS. Finally, with each estimated database, a reconstruction error is calculated and compared between the calibration methods. The system is able to perform the calibration in 280 ms, while the location of 49 3-D points can be realized in 200 ms. This execution time was obtained using a compact vision system from national instruments, with a processor Intel Atom quad-core 1.91 GHz.

Experimentation results

Referring to the first experiment, Figure 11 shows the scan of the grid on a 3-D view, and Figure 12 shows the scan of the grid at different planes. Figures 11(a) and 12(a) and (c) show a barrel distortion that produces a curvature in straight lines of the images and therefore inaccuracy of the measurements. Moreover, the quality of the images due to the image digitalization causes dispersion of the points in XZ plane (Figure 12(e)). On the other hand, Figures 11(b) and 12(b) to (f) show the corrections of the errors by the developed calibration method.

Figure 11.

Scanned surface 3-D view: (a) Uncalibrated scan. (b) Calibrated scan. 3-D: three-dimensional.

Figure 12.

Surface scan at different planes: (a) Uncalibrated scan YZ plane. (b) Calibrated scan YZ plane. (c) Uncalibrated scan XY plane. (d) Calibrated scan XY plane. (e) Uncalibrated scan XZ plane. (f) Calibrated scan XZ plane.

For the second experiment, Table 1 shows 10 measurements located along the test grid employing a distance of 22.87 cm from the base line to the test grid. Furthermore, Table 1 shows the absolute error compared with the real database without using the calibration method. In the same way, Table 2 shows the analysis using the calibration method. As it can be appreciated, the measurements on each coordinate are better in the calibrated SVS than the SVS without using the calibration. Averages of absolute errors considering the 49 measurements obtained without using the calibration in x, y, and z coordinates were of 0.6908, 0.0412, and 0.0933 cm, respectively. Otherwise, averages of absolute error in x, y, and z coordinates in the SVS using the calibration were of 0.1307, 0.034, and 0.0512 cm, respectively. These results using the calibration method represent improvements in terms of percentage in x, y, and z coordinates of 81.07%, 17.47%, and 45.12%, respectively.

Table 1.

3-D points localization without using calibration at 22.87 cm (simplified table).

Real coordinates (cm)			Measured coordinates (cm)			Absolute error (cm)
x	y	z	x	y	z	X error	Y error	Z error
.87	−4	6	23.72	−3.97	6.20	0.84	0.02	0.20
.87	2	6	24.01	2.05	6.22	1.13	0.05	0.22
.87	−4	4	23.52	−3.98	4.09	0.64	0.01	0.09
.87	2	4	23.92	2.04	4.10	1.04	0.04	0.10
.87	−2	2	23.97	−2.00	2.06	1.10	0.006	0.06
.87	2	2	23.87	2.03	2.03	1.00	0.03	0.03
.87	−2	0	23.92	−2.02	0.01	1.05	0.02	0.01
.87	−2	−2	23.92	−2.03	−2.04	1.04	0.03	0.04
.87	0	−4	24.04	−0.01	−4.15	1.17	0.01	0.15
.87	2	−6	24.11	2.01	−6.28	1.24	0.01	0.28

3-D: three-dimensional.

Table 2.

3-D points localization employing the developed calibration method at 22.87 cm (simplified table).

Real coordinates (cm)			Measured coordinates (cm)			Absolute error (cm)
x	y	z	x	y	z	X error	Y error	Z error
.87	−4	6	23.11	−3.99	6.13	0.23	0.002	0.13
.87	2	6	23.04	2.04	6.06	0.16	0.04	0.06
.87	−4	4	22.93	−4.00	4.08	0.05	0.004	0.08
.87	2	4	22.95	2.03	4.02	0.08	0.03	0.02
.87	−2	2	22.98	−1.99	2.03	0.11	0.005	0.03
.87	2	2	22.91	2.02	2.00	0.03	0.02	0.003
.87	−2	0	22.94	−2.00	0.008	0.06	0.009	0.008
.87	−2	−2	22.94	−2.02	−2.02	0.06	0.02	0.02
.87	0	−4	22.93	−0.01	−4.04	0.06	0.01	0.04
.87	2	−6	23.13	2.00	−6.10	0.25	0.002	0.10

3-D: three-dimensional.

The proposed calibration was performed in other three scenarios, changing the distances from the base line to the test grid at 19.87, 16.87, and 13.87 cm. Figure 13 shows the absolute error comparison of 10 measurements obtained with and without using the calibration method in x coordinate, Figure 14 in y coordinate, and Figure 15 in z coordinate.

Figure 13.

Absolute error comparison in x coordinate at 19.87, 16.87, and 13.87 cm (simplified graph).

Figure 14.

Absolute error comparison in y coordinate at 19.87, 16.87, and 13.87 cm (simplified graph).

Figure 15.

Absolute error comparison in z coordinate at 19.87, 16.87, and 13.87 cm (simplified graph).

In each coordinate, it can be appreciated the improvements of the calibration method, obtaining improvements in averages of absolute errors in x coordinate of 76.65%, 87.86%, and 85.44%, with distances of 19.87, 16.87, and 13.87 cm, respectively. Furthermore, improvements in averages of absolute errors in y coordinate were of 34.53%, 42.28%, and 50.72%. Moreover, improvements in averages of absolute errors in z coordinate were of 50.09%, 51.48%, and 56.13%. With the databases at different distances, a comparative analysis is developed; Table 3 shows the error variability between the estimated data and the real data. Using the SVS without calibration, variabilities greater than 14 mm in x coordinate, in y coordinate greater than 2 mm, and in z coordinate greater than 7 mm was obtained. Implementing the calibration method, the variabilities decrease to a maximum of 3.2 mm in x coordinate, 1.7 mm in y coordinate, and 2.5 mm in z coordinate. Furthermore, a comparative error analysis was performed by employing the mean square error (MSE) that provides a measurement error for each x, y, and z coordinates set of each database obtained.⁵¹

Table 3.

Error variability of the measured coordinates between the estimated data and the real data.

	Dispersion error			Dispersion error
	Without calibration (mm)			With calibration (mm)
Distance (cm)	x	y	z	x	y	z
At 22.87	−12.4 to 1	−1.5 to 1.2	−2.9 to 3.2	−2.6 to 0.5	−0.7 to 1	−1.3 to 1.1
At 19.87	−10.9 to −4	−0.5 to 0.7	−3.3 to 3.8	−3.6 to −0.4	−0.4 to 0.6	−1.3 to 1.2
At 16.87	−8.2 to −5.9	−0.5 to 0.2	−1.7 to 1.7	−1.9 to 0.4	−0.3 to 0.1	−0.8 to 0.7
At 13.87	−5.5 to −2.8	−0.5 to 0.5	−1.6 to 1.8	−1 to 1.6	−0.2 to 0.3	−0.6 to 0.7

The MSE is obtained by

MSE = \frac{1}{n} \sum_{i = 1}^{n} {(Y_{i} - {\hat{Y}}_{i})}^{2}

where Y_i are the real values, ${\hat{Y}}_{i}$ are the predicted values, and n is the number of data points. Table 4 shows the comparative MSE analysis between the estimated databases and the real databases, where it can be appreciated the MSE reduction employing the calibration method in x coordinate is greater than 90%, in y coordinate is greater than 70%, and in z coordinate is greater than 80%. Furthermore, the results of Table 4 validate that the calibration method can improve the 3-D points localization in different distances.

Table 4.

Comparative MSE analysis between the estimated data and the real data.

	MSE without calibration			MSE with calibration			MSE reduction (%)
Distance (cm)	x	y	z	x	y	z	x	y	z
At 22.87	67.8856	2.328	0.3034	0.1747	1.6403	0.3982	96.57	42.4	75.73
At 19.87	81.4649	4.9882	0.1082	0.0693	2.7695	0.5615	93.88	35.95	79.73
At 16.87	46.0581	0.9019	0.095	0.0367	0.9576	0.2009	98.04	61.36	79.03
At 13.87	15.0246	0.5439	0.1069	0.0249	0.7757	0.1268	96.38	76.67	83.65

MSE: mean square error.

For the third experiment, 3-D reconstruction error is used to compare the accuracy of 3-D points localization with different calibration methods.

The mean reconstruction error $E_{pt}$ is obtained by

E_{pt} = \frac{1}{n} \sum_{i = 1}^{n} \sqrt{| | M_{i} - {\hat{M}}_{i} {| |}^{2}}

where n is the total number of the 3-D points and $E_{pt}$ represents the reconstruction error between real 3-D coordinate M_i and estimated results ${\hat{M}}_{i}$ . Then, the calibration results ${\hat{M}}_{i}$ of the 49 3-D points in each calibration method are compared with the real values of the 49 3-D points M_i using equation (24). These results are the mean reconstruction errors of each calibration method. For convenience, these errors are projected in XY, XZ, XY, and XYZ planes. Table 5 shows the results of the reconstruction errors in planes XY, XZ, YZ, and XYZ of each calibration method.

Table 5.

Statistics of mean 3-D reconstruction errors of the calibration’s methods in planes XY, XZ, YZ, and XYZ.

Calibration	XY	XZ	YZ	XYZ
Method	(mm)	(mm)	(mm)	(mm)
Zhang	0.563	0.644	0.690	0.948
Jia et al.	0.465	0.574	0.545	0.793
Proposed	0.454	0.530	0.526	0.728
Cai et al.	0.288	0.307	0.297	0.446

3-D: three-dimensional.

As result, the mean errors by the proposed method are 0.454, 0.530, 0.526, and 0.728 mm in planes XY, XZ, YZ, and XYZ, respectively. In Table 5, it can be seen that the accuracy of the proposed method is better than Zhang’s and Jia et al.’s methods in all planes, while Cai et al. present the best accuracy. Although the proposed method improves the accuracy in only two of the three methods, it demonstrates a high accuracy for 3-D points localization. Moreover, the proposed method requires a few number of steps to perform the calibration than Zhang, Jia et al., and Cai et al., obtaining simplicity in the implementation and a lower computational cost than classic calibration methods that require several steps to obtain the necessary parameters of the cameras for calibration.

Conclusions

The developed SVS is able to locate 3-D points in a scene by intensity pattern match localization methods and performing a calibration method to improve accuracy of the measurements. In this article, a novel calibration method for SVS was presented, where using the SVS, several 3-D points at different distances were obtained. Furthermore, a comparative analysis between the obtained databases and the real databases was developed. Moreover, a comparative analysis of mean reconstruction errors in planes XY, XZ, YZ, and XYZ between classic calibration methods and the proposed method was performed. The dispersion error was reduced by employing the calibration method, obtaining the best ranges at 13.87 cm of distance, with error ranges in x coordinate from −1 mm to 1.6 mm, in y coordinate from −0.2 mm to 0.3 mm, and z coordinate from −0.6 mm to 0.7 mm. The MSE results validate the improvements in the localization of 3-D coordinates, obtaining the best MSE reduction in terms of percentage in x, y, and z coordinates of 98.04%, 76.67%, and 83.65%, respectively. Furthermore, the mean reconstruction errors of the proposed method are better in terms of accuracy than Zhang’s and Jia et al.’s methods, with values in planes XY, XZ, YZ, and XYZ of 0.454, 0.530, 0.526, and 0.728 mm, respectively. The proposed method can be used to calibrate the SVS which requires a reliable localization of a specific 3-D point in the scene for decision-making or obtains reliable spatial information, for example, for autonomous navigation tasks and robotic vision applications. Advantage of the developed calibration method is the competitive accuracy in the 3-D points localization and a high execution speed due to the use of only the distortion of the cameras as information to perform the calibration in comparison with actual calibration methods that require many steps to obtain several parameters of the cameras for calibration. The developed calibration can be used in multiple machine vision applications, particularly machine vision applications where descalibrations are expected, and, therefore, it is necessary to perform a fast calibration to ensure the accuracy localization of the 3-D points.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported with funding from Universidad Autónoma de Baja California.

ORCID iD

Luis R Ramírez-Hernández

Wendy Flores-Fuentes

Lars Lindner

References

Malassiotis

Strintzis

. Stereo vision system for precision dimensional inspection of 3D holes. Mach Vision Appl 2003; 15(2): 101–113.

Kim

Lee

, et al. A vision system for identifying structural vibration in civil engineering constructions. In: SICE-ICASE international joint conference, Busan, South Korea, 18–21 October 2006, pp. 5813–5818. IEEE.

Palma

Becker

Riviere

. Simultaneous calibration of stereo vision and 3D optical tracker for robotic microsurgery. In: 38th Annual northeast bioengineering conference (NEBEC), Philadelphia, PA, USA, 16–18 March 2012, pp. 351–352. IEEE.

Kang

Azizian

Wilson

, et al. Stereoscopic augmented reality for laparoscopic surgery. Surg Endo 2014; 28(7): 2227–2235.

Real-Moreno

Rodriguez-Quiñonez

Sergiyenko

, et al. Accuracy improvement in 3D laser scanner based on dynamic triangulation for autonomous navigation system. In: 2017 IEEE 26th international symposium on industrial electronics (ISIE), Edinburgh, UK, 19–21 June 2017, pp. 1602–1608. IEEE.

Rodriguez-Quinonez

Sergiyenko

Hernandez-Balbuena

, et al. Improve 3D laser scanner measurements accuracy using a FFBP neural network with Widrow-Hoff weight/bias learning function. Opto-Electronics Review 2014; 22(4): 224–235.

Real-Moreno

Castro-Toscano

Rodríguez-Quiñonez

, et al. Implementing k-nearest neighbor algorithm on scanning aperture for accuracy improvement. In: IECON 2018-44th annual conference of the IEEE industrial electronics society, Washington, DC, USA, 21–23 October 2018, pp. 3182–3186. IEEE.

Castro-Toscano

Rodríguez-Quiñonez

Hernández-Balbuena

, et al. A methodological use of inertial navigation systems for strapdown navigation task. In: 2017 IEEE 26th international symposium on industrial electronics (ISIE), Edinburgh, UK, 19–21 June 2017, pp. 1589–1595.

Castro-Toscano

Rodríguez-Quiñonez

Hernández-Balbuena

, et al. Obtención de trayectorias empleando el marco strapdown ins/kf: Propuesta metodológica. Revista Iberoamericana de Automatica e Informatica Industrial 2018; 15(4): 391–403.

10.

Mac

Copot

De Keyser

, et al. The development of an autonomous navigation system with optimal control of a UAV in partly unknown indoor environment. Mechatronics 2018; 49: 187–196.

11.

Hui

Bian

Zhao

, et al. Vision-based autonomous navigation approach for unmanned aerial vehicle transmission-line inspection. Int J Adv Robot Syst 2018; 15(1): 1729881417752821.

12.

Huntsberger

Aghazarian

Howard

, et al. Stereo vision-based navigation for autonomous surface vessels. J Field Robot 2011; 28(1): 3–18.

13.

Tang

Shen

, et al. Ground stereo vision-based navigation for autonomous take-off and landing of UAVs: a Chan-Vese model approach. Int J Adv Robot Syst 2016; 13(2): 67.

14.

Chavez

Mueller

Doernbach

, et al. Underwater navigation using visual markers in the context of intervention missions. Int J Adv Robot Syst 2019; 16(2): 1729881419838967.

15.

Kilpela

. Pulsed time-of-flight laser range finder techniques for fast, high-precision measurement applications. Cambridge: Academic Press, 2002.

16.

Bradshaw

. Non-contact surface geometry measurement techniques. Dublin: Technical report, Trinity College Dublin, Department of Computer Science, 1999.

17.

Rodríguez-Quiñonez

Sergiyenko

Preciado

LCB

, et al. Optical monitoring of scoliosis by 3D medical laser scanner. Opt Laser Eng 2014; 54: 175–186.

18.

Mohamed

Culverhouse

De Azambuja

, et al. Automating active stereo vision calibration process with cobots. IFAC-PapersOnLine 2017; 50(2): 163–168.

19.

Hernández

Sanz

Guijarro

. Técnicas de procesamiento de imágenes estereoscópicas. Revista del CES Felipe II 2011; 13: 1–20.

20.

Machikhin

Batshev

Gorevoy

, et al. A miniature prism-based stereoscopic system for 3D machine vision applications. In: Eleventh international conference on machine vision (ICMV 2018), International Society for Optics and Photonics, Munich, Germany, 15 March 2019, pp. 110410 M.

21.

Rodríguez-Quiñonez

Sergiyenko

Flores-Fuentes

, et al. Improve a 3D distance measurement accuracy in stereo vision systems using optimization methods’ approach. Opto-Electron Rev 2017; 25(1): 24–32.

22.

Fitzgibbon

. Simultaneous linear estimation of multiple view geometry and lens distortion. In Computer vision and pattern recognition, 2001. Proceedings of the 2001 IEEE computer society conference, Vol. 1, Kauai, HI, USA, 8–14 December, 2001, pp. I–I. IEEE.

23.

Salvi

Armangué

Batlle

. A comparative review of camera calibrating methods with accuracy evaluation. Pattern Recognit 2002; 35(7): 1617–1635.

24.

Zapata-Farfan

Contreras-Martnez

Rosete-Aguilar

, et al.

Low-energy/pulse response and high-resolution-CMOS camera for spatiotemporal femtosecond laser pulses characterization@ 1.55

μ

m. Rev Sci Instrum 2019; 90(4): 045116.

25.

Damulira

Yusoff

MNS

Omar

, et al. A review: photonic devices used for dosimetry in medical radiation. Sensors 2019; 19(10): 2226.

26.

Wang

Cheng

Suresh

, et al. Development of the local magnification method for quantitative evaluation of endoscope geometric distortion. J Biomed Opt 2016; 21(5): 056003.

27.

Carabias

Garca

Salor

JAR

. Sistema de visión estereoscópica para navegación autónoma de vehículos no tripulados. Universidad complutense de Madrid 2010; 1: 1–21.

28.

Chang

Maruyama

. Real-time stereo vision system: a multi-block matching on GPU. IEEE Access 2018; 6: 42030–42046.

29.

Mohamed

Culverhouse

Cangelosi

, et al. Active stereo platform: online epipolar geometry update. EURASIP J Image Video Process 2018; 2018(1): 54.

30.

Campos

Tommaselli

AMG

Castanheiro

, et al. A fisheye image matching method boosted by recursive search space for close range photogrammetry. Remote Sens 2019; 11(12): 1404.

31.

López Valles

Fernández Caballero

Fernández

. Conceptos y técnicas de estereovisión por computador. Inteligencia Artificial, Revista Iberoamericana de Inteligencia Artificial 2005; 9(27): 35–62.

32.

Fan

Dahnoun

. Real-time stereo vision-based lane detection system. Meas Sci Technol 2018; 29(7): 074005.

33.

Kreyszig

ENE

Kreyszig

. Advanced engineering mathematics. Hoboken: John Wiley & Sons, 2011.

34.

Faugeras

Keriven

. Complete dense stereovision using level set methods. In: European conference on computer vision, Freiburg, Germany, 2–6 June 1998, pp. 379–393. Springer.

35.

Brown

Burschka

Hager

. Advances in computational stereo. IEEE Trans Pattern Anal 2003; 25(8): 993–1008.

36.

Bai

Hao

, et al. Improving stereo matching algorithm with adaptive cross-scale cost aggregation. International Journal of Advanced Robotic Systems 2018; 15(1): 1–9.

37.

Ploumpis

Amanatiadis

Gasteratos

. A stereo matching approach based on particle filters and scattered control landmarks. Image Vision Comput 2015; 38: 13–23.

38.

Fernandez

Forest

Salvi

. Active stereo-matching for one-shot dense reconstruction. In: ICPRAM (2), Algarve, Portugal, 6–8 February 2012, pp. 541–545.

39.

Anderson

. Stereoscopic occlusion and the aperture problem for motion: a new solution1. Vision Res 1999; 39(7): 1273–1284.

40.

Priya

Anand

. Object recognition and 3D reconstruction of occluded objects using binocular stereo. Cluster Comput 2017; 21: 29–38.

41.

Quiroga

EAC

Martn

LYM

Caycedo

. La estereoscopa, métodos y aplicaciones en diferentes áreas del conocimiento. Revista Cientfica General José Mara Córdova 2015; 13(16): 201–219.

42.

Zhou

Gallego

Rebecq

, et al. Semi-dense 3D reconstruction with a stereo event camera. In: Proceedings of the European conference on computer vision (ECCV), Munich, Germany, 8–14 September 2018, pp. 235–251.

43.

Básaca

Rodríguez

Sergiyenko

, et al. Resolution improvement of dynamic triangulation method for 3D vision system in robot navigation task. In: IECON 2010-36th annual conference on IEEE industrial electronics society, Glendale, AZ, USA, 7–10 November 2010, pp. 2886–2891. IEEE.

44.

Rodríguez-Quinonez

Sergiyenko

Gonzalez-Navarro

, et al. Surface recognition improvement in 3D medical laser scanner using Levenberg–Marquardt method. Signal Process 2013; 93(2): 378–386.

45.

Rivera-Castillo

Rivas-Lopez

Nieto-Hipolito

, et al. Structural health monitoring based on optical scanning systems and SVM. In: Industrial electronics (ISIE), 2014 IEEE 23 rd international symposium, Istanbul, Turkey, 1–4 June 2014, pp. 1961–1966. IEEE.

46.

Basaca-Preciado

Sergiyenko

Rodriguez-Quinonez

, et al. Optoelectronic 3D laser scanning technical vision system based on dynamic triangulation. In: Photonics conference (IPC), 2012 IEEE, Burlingame, CA, USA, 23–27 September 2012, pp. 648–649. IEEE.

47.

Basaca-Preciado

Sergiyenko

Rodríguez-Quinonez

, et al. Optical 3D laser measurement system for navigation of autonomous mobile robot. Opt Las Engine 2014; 54: 159–169.

48.

Zhang

A flexible new technique for camera calibration. IEEE Transact Patt Analys Mach Intell 2000; 22(11): 1330–1334.

49.

Jia

Yang

Liu

, et al. Improved camera calibration method based on perpendicularity compensation for binocular stereo vision measurement system. Optics Exp 2015; 23(12): 15205–15223.

50.

Cui

Zhou

Wang

, et al. Precise calibration of binocular vision system used for vision measurement. Optic Exp 2014; 22(8): 9134–9149.

51.

Lehmann

Casella

. Theory of point estimation. Berlin: Springer Science & Business Media, 2006.

Improve three-dimensional point localization accuracy in stereo vision systems using a novel camera calibration method

Abstract

Keywords

Introduction

Image digitalization errors and lens distortion

SVS implementation

Specifications of the SVS

Images capture

Camera calibration

Pattern match

Computing pixel coordinates to angles

Triangulation

Experimentation

Experimentation results

Conclusions

Footnotes

Declaration of conflicting interests

Funding

ORCID iD

References