Sage Journals: Discover world-class research

Abstract

Light changes its direction of propagation before entering a camera enclosed in a waterproof housing owing to refraction, which means that perspective imaging models in the air cannot be directly used underwater. In this article, we propose an accurate binocular stereo measurement system in an underwater environment. First, based on the physical underwater imaging model without approximation and Tsai’s calibration method, the proposed system is calibrated to acquire the extrinsic parameters, as the internal parameters can be pre-calibrated in air. Then, based on the calibrated camera parameters, an image correction method is proposed to convert the underwater images to air images. Thus, the epipolar constraint can be used to search the matching point directly. The experimental results show that the proposed method in this article can effectively eliminate the effect of refraction in the binocular vision and the measurement accuracy can be compared with the measurement result in air.

Keywords

Underwater camera calibration binocular vision vision-based underwater measurement

Introduction

Exploring the ocean has been of perpetual interest since ancient times. In recent years, obtaining three-dimensional (3-D) geometric information has been an important application in underwater archaeological exploration,^1,2 marine biological research,^3,4 underwater equipment monitoring and maintenance,⁵ and so on. The 3-D measurement techniques used underwater include photogrammetry,^6,7 3-D laser scanning,^8,9 structured light scanning based on grating,^10
–12 and underwater 3-D measurement based on images.¹³

3-D measurement technology based on underwater laser scanning has certain advantages in large field, long-distance measurement, such as submarine topography measurement, but for the precise measurement of small and medium objects, underwater laser scanning-based measurement method still has the following problems: (1) it lacks of flexibility due to volume of the system; (2) the laser is required moving through a predetermined trajectory during measurement such that the operation is difficult; and (3) water and impurities cause serious scattering of the collimated laser beam, which affects the measuring accuracy of the system.

The method based on grating-structure light has advantages of high precision (the minimum and maximum deviations are −0.003 and 0.0012 mm for a flat plane¹⁴), a large amount of data, and fast and automatic measurement. However, this method must project a specific grating structure that results in 3-D measurement of underwater equipment complex. First, a high illuminance digital light processing (DLP) projector and a definition camera should be mounted into a waterproof housing. Second, the optical grating of the DLP and the camera must be carefully calibrated. The high illuminance DLP projectors are more expensive. Generally, the DLP projector and the camera should have an included angle to focus the grating patterns, which makes the device more complicated.

With the development of computer vision, image-based 3-D system can accurately measure the operation condition of the underwater object and the geometric scale using images collected by high-definition (HD) cameras. It can effectively solve the shortages of the aforementioned underwater laser 3-D measurement systems. The accurate measurement system of underwater objects based on images obtained by HD camera can obtain the operating information and geometric information of the underwater object with strong flexibility, to solve the defects of the 3-D laser underwater measurement method.

Camera calibration is essential to achieve accurate and reliable measurements. Small errors in the perspective projection must be eliminated to prevent the introduction of effects in an underwater environment. As the light is absorbed in the water, affected by water pollution, and refracted at the interfaces of different media (water, glass, air), underwater camera calibration is complex. In this article, an image-based 3-D measurement in underwater conditions is presented. First, the underwater camera imaging system is analyzed, and then an accurate imaging model without approximation is utilized to correct underwater images into air. The image-calibration model is calibrated by the improved calibration algorithm, and the accuracy of the model is verified by 3-D measurement experiments.

The organization of the article is as follows. Section of ‘Related work’ summarizes other studies related to our work. Section of ‘Nonlinear underwater imaging model’ introduces the nonlinear underwater imaging model for self-containing. Section of ‘Accurate binocular stereo underwater measurement method’ explains the details of our approach. Underwater measurement experiments are described in section of ‘Experimental results’, before section ‘Conclusion’ presents the conclusion of our work.

Related work

Underwater camera calibration has been widely studied as the basis of underwater 3-D measurement. To take photos underwater, the camera is usually placed in a waterproof housing. The camera lens is not in direct contact with the waterproof shell. The light will go through different media before entering the imaging plane and refracting on the surface of two different media. Thus, the light is no longer a straight line; the perspective imaging model in the air cannot be directly used underwater. Queiroz-Neto et al.¹⁵ and Sanchez-Ferreira et al.¹⁶ reconstructed 3-D model without considering the influence of refraction, and the experimental results show that ignoring the influence of refraction produces a large deviation. Shortis et al.¹⁷ present a review of the methods used for the detection, identification, measurement, tacking, and counting of fist in underwater stereo-video image sequences. Their shortcomings and characteristics were identified and compared. It provides a comprehensive overview of video-based stereovision methods.

Aiming at solving the influence of refraction, many studies propose various methods to approximate the underwater imaging process. The first category entails using the physical assumption to transfer the center of the camera lens to the plane of refraction by a special optical components.^18

–23 Then, the special shape of optical component offsets the refraction of light. However, these methods are not easy to implement because of the limitation of optical component progress and the strict requirement of manufacturing.

The second class entails calibrating the underwater camera with an auxiliary plane.^24,25 These methods can determine the direction of the incident vector by adding an auxiliary calibration board and then calibrating the camera parameters using this known quantity. The third type regards refraction as a change in the focal length.^26,27 The extension line of the incident light will intersect the optical axis of the camera eventually. Because the change in focal length is associated with the incident angles of image points, the method will result in some errors. The incident angle of light is larger, which cause larger errors.

Another category is to approximate the error caused by refraction underwater as lens distortion.^28
–30 The pixel offset error is regarded as refraction error caused by the distortion of the lens itself in this method. Through the calibration of the camera parameters and image distortion correction, the method can eliminate the influence caused by refraction. However, because the influences of different pixels by refraction are different, this method cannot correct the whole image with uniform distortion parameters particularly in the boundary areas of high-resolution image or in the case of foreshortening for a large object, and this will result in an obvious error.

The fifth kind of method entails the underwater imaging process with physical model and developing the corresponding calibration algorithm according to the imaging model. The hypothesis proposed by Treibitz et al.³¹ is that the imaging plane is parallel to the refraction plane. The underwater imaging process of single refraction is analyzed, and the calibration algorithm is improved to calibrate the parameters of the underwater camera. Jordt-Sedlazeck and Koch³² proposed a camera-calibration method based on the shell. The method calibrates the refractive plane of the normal vector and distance from the camera center with the optimization method. However, the method requires a suitable initial value. Agrawal et al.³³ build a multiple-refraction imaging model. The method uses the five-point algorithm to calculate parameters, including the refractive index, refracted plane’s normal vector, and distance between the refraction plane and the camera center. However, the model is complex, which makes the calculation extensive. Chen and Yang present a calibration method of binocular vision based on an underwater camera.³⁴ The method optimizes the camera parameter through 3-D point remapping with the constraint of two incident lights lying on the same plane. Yau et al. proposed a method for an underwater camera-calibration method based on different light refractive index.³⁵ The method mainly uses the principle that different frequencies of light in water have different refractive indices to design a calibration board that can emit two different frequencies of light.

In recent years, many scholars have extensively researched underwater 3-D measurement. Some early underwater reconstruction work neglects the influence of refraction. The multi-view method was used by Kang et al.³⁶ in 3-D reconstruction and measurement underwater. The method discusses the effect of ignoring refraction during 3-D reconstruction and measurement. Chari and Sturm³⁷ analyzed the refraction phenomena in detail to prove the existence of the fundamental refractive matrix in underwater binocular vision. However, they only provide the simulation results, but no actual experimental data. Chang and Chen proposed an underwater 3-D reconstruction method.³⁸ In this method, all cameras share a refractive plane. Refraction influence is modeled as a depth energy model. This method requires a known refraction plane and an additional inertial measurement unit to measure the camera rotation angle. Sedlazeck and Koch³⁹ provide a more flexible approach and their method does not need calibration; however, the method is time-consuming and the result is not satisfactory. Yau⁴⁰ uses the multi-vision method to reconstruct underwater objects, according to the principle of underwater imaging; this method improves the traditional Bundler algorithm and patch-based multi-view stereo algorithm to reconstruct underwater objects. Although it can achieve a good result, the cost is high. Pedersen et al.⁴¹ address the underwater calibration camera based on ray tracing using Snell’s law, while other two methods, an approach relying solely on in-air camera calibration and an approach with the camera calibration performed under water, are compared.

In this article, a Digital Single Lens Reflex (DLSR)-based stereo underwater vision system is presented to verify the effectiveness of the algorithm as the DLSR camera is easy to operate and has high image quality with affordable price, which is important for underwater stereo system. Our system can used in high accurate measurement environment. The contributions of this article can be summarized as:

According to the nonlinear underwater refraction imaging model, the traditional Tsai’s calibration algorithm and the planar checkerboard calibration algorithm are improved to obtain more accurate parameters such as the camera internal parameters and external parameters, as well as the distance between the camera center and the refractive plane.

An underwater image correction method based on camera calibration parameters is established. The underwater image is restored to the image in the air through image correction. The corrected image is used to carry out 3-D measurement of the size of underwater objects.

Nonlinear underwater imaging model

In air, perspective imaging model is expressed as the correspondence between the spatial 3-D point and its imaging point, illustrated in equation (1)

\begin{array}{l} z_{c} [\begin{matrix} u \\ v \\ z \end{matrix}] = [\begin{matrix} 1 / d x & 0 & u_{0} & ​ \\ 0 & 1 / d y & v_{0} & ​ \\ 0 & 0 & 1 & ​ \end{matrix}] [\begin{matrix} f & 0 & 0 & 0 & ​ \\ 0 & f & 0 & 0 & ​ \\ 0 & 0 & 1 & 0 & ​ \end{matrix}] \\ [\begin{matrix} R & T \\ 0 & 1 \end{matrix}] [\begin{matrix} x_{w} \\ y_{w} \\ z_{w} \\ 1 \end{matrix}] \end{array}

where $z_{c}$ is scale factor, $(u, v)$ is image coordinate, f is the camera focal length, $R$ and $T$ are the camera rotation matrix and translational vector, and $(x_{w}, y_{w}, z_{w})$ is the point in world coordinates. Although the underwater imaging model is no longer a linear-perspective imaging model due to refraction effects, the cameras intrinsic-parameter and external-parameter matrices are not influenced by the environment. Thus, we can build a new underwater imaging model based on the perspective imaging model in air.

Light reflected from an object arrives at waterproof housing of camera and then strikes surface between the housing and the air. Different refractive indexes of air and glass cause refraction. Finally, light enters the camera and forms an image. All the light entering the camera goes through this process. According to Snell’s law, we get the following equation

\begin{array}{l} n_{water} sin θ_{water} = n_{interface} sin θ_{interface} \\ = n_{air} sin θ_{air} \end{array}

where $n_{water}, n_{intertace}, and n_{air}$ are refractive indexes of water, waterproof housing, and air, respectively. $θ_{water}, θ_{interface}, and θ_{air}$ are the angles between incident light and optical axis in different media, respectively. Incident light produces two refractions at waterproof shell, but the refraction does not change the angle relationship of light in water and in air. Waterproof shell is very thin, thus the refraction produced by the shell can be neglected in the entire refractive process. Thus, the simplified refractive imaging process is shown in Figure 1.

Figure 1.

Underwater imaging model.

Assuming waterproof housing surface is parallel to the camera imaging plane; f is the camera focal length, and d represents the distance between the refracting plane and the camera center, as shown in Figure 1; ${(α_{w}, β_{w}, γ_{w})}^{T}$ and ${(α_{a}, β_{a}, γ_{a})}^{T}$ represent the direction vector of incident light before and after refraction; ${(x_{r}, y_{r}, z_{r})}^{T}$ is the intersection point of light and refraction plane; and $θ_{air}$ and $θ_{water}$ represent the angle between light and optical axis in air and in water. Assuming refraction plane perpendicular to the camera axis, the direction vector of optical axis in the camera coordinate system is ${(0, 0, 1)}^{T}$ . We employ the underwater imaging model expressed as equation (3)

\begin{array}{l} k [\begin{matrix} u \\ v \\ 1 \end{matrix}] = [\begin{matrix} 1 / d x & 0 & u_{0} & ​ \\ 0 & 1 / d y & v_{0} & ​ \\ 0 & 0 & 1 & ​ \end{matrix}] \\ [\begin{array}{l} ​ \\ \begin{matrix} n_{0} r & 0 & 0 & - \frac{n_{0} d x_{u} r}{f} & ​ \\ 0 & n_{0} r & 0 & - \frac{n_{0} d y_{u} r}{f} & ​ \\ 0 & 0 & \frac{- n_{0}}{\sqrt{n_{0}^{2} + {(\frac{f}{r})}^{2} - 1}} & \frac{n_{0} d}{\sqrt{n_{0}^{2} + {(\frac{f}{r})}^{2} - 1}} & ​ \end{matrix} \end{array}] \\ [\begin{matrix} R & T & ​ \\ 0 & 1 & ​ \end{matrix}] [\begin{matrix} x_{w} \\ y_{w} \\ z_{w} \\ 1 \end{matrix}] \end{array}

where $n_{0} = n_{air} / n_{water}$ and $r = \sqrt{x_{u}^{2} + y_{u}^{2} + f^{2}}$ . It is a nonlinear underwater imaging mode which is based on the physical process of underwater imaging and describes the whole underwater imaging process without any approximation.

Accurate binocular stereo underwater measurement method

System calibration based on Tsai’s method

As Tsai’s method only uses one image to calibrate the camera,⁴² it is suitable to be employed in underwater for convenient image acquisition. The internal camera parameters are the intrinsic property of a camera. Thus, we can calibrate the intrinsic parameters, $f_{x}, f_{y}, u_{0}, v_{0}$ , in air. According to radial consistency constraint, we can get the relation between the object point and the image point in camera coordinate system, like equation (4)

\frac{x_{i}}{y_{i}} = \frac{x_{c}}{y_{c}}

where $x_{i} = (u - u_{0}) f / d x, y_{i} = (v - v_{0}) f / d y$ .

[\begin{matrix} x_{c} \\ y_{c} \\ z_{c} \end{matrix}] = [R T] [\begin{matrix} x_{w} \\ y_{w} \\ z_{w} \\ 1 \end{matrix}] [\begin{matrix} r_{1} x_{w} + r_{2} y_{w} + r_{3} z_{w} + t_{x} & ​ \\ r_{4} x_{w} + r_{5} y_{w} + r_{6} z_{w} + t_{y} & ​ \\ r_{7} x_{w} + r_{8} y_{w} + r_{9} z_{w} + t_{z} & ​ \end{matrix}]

From equations (4) and (5), we can compute the rotation matrix $R$ and the first two items of translational vector $t_{x}, t_{y}$ . The light ray is refracted when capturing the underwater object, as shown in Figure 1. The corresponding relation between the direction vectors of incident ray before and after refraction occurs is

[\begin{matrix} α_{w} \\ β_{w} \\ γ_{w} \end{matrix}] = \frac{1}{n_{0}} [\begin{matrix} α_{a} \\ β_{a} \\ - \sqrt{n_{0}^{2} - 1 + γ_{a}^{2}} \end{matrix}]

The incident ray is coplanar with the optical axis of the camera before and after refraction, so the radial uniform constraint is still valid in underwater imaging and can be used to obtain camera parameters. After we obtain the intrinsic matrix $B$ , rotation matrix $R$ , and $T$ , the remaining parameters of the underwater camera can be obtained by solving the equations with the least square method.

System calibration based on planar pattern

As the Tsai’s method needs a 3-D calibration pattern, it is not easy to guarantee the orthogonality of two planes. In this article, we utilize a two-step method to calibrate the underwater camera by using on planar calibration pattern based on Zhang’s method.⁴³ First, the planar calibration pattern and the binocular stereo system are placed in the water tank without water. Twenty images of the calibration pattern are captured in different orientations. The position of the calibration board is fixed after the last calibration pattern photo is taken. The intrinsic and extrinsic parameters ( $B$ , $R$ , and $T$ ) of the last calibration pattern are calculated by using Zhang’s method.⁴³ Because the internal parameters of the camera will not change with the change of the shooting environment, the position of the camera and the calibration target are fixed before and after the injection of water. That is the coordinate frame of the calibration pattern and the camera frame remains unchanged whatever in air or in water. Second, we inject water into the tank and remain unchange of the calibration pattern and the stereo camera. Then one image of the calibration target is captured. We can get the equations according to the corresponding relations between the image pixel points and the 3-D coordinate points of the calibration pattern. As the internal and external parameters ( $B$ , $R$ , and $T$ ) have been calculated, the distance from the center of the camera to the refractive plane d is obtained by solving the equations with the least square method.

Underwater binocular vision measurement method in 2-D and 3-D

2-D Underwater measurement with one camera

According to the underwater imaging model, a straight object is imaged to appear bent in the raw image; however, the object should be undistorted in air, as shown in Figure 2. The two-dimensional (2-D) underwater measurement is to estimate the length D from the blue curve. After we acquire the parameters of the underwater camera, 2-D measurement in water can be carried out according to equation (7),³¹ where the object should be positioned frontoparallel to the interface and camera

D = \sqrt{d_{a}^{2} + d_{b}^{2} - 2 d_{a} d_{b} | cos π |}

where

d_{a} = d_{w a} + d_{w a} \frac{Z - d - f}{f} \frac{n_{air}}{n_{water}} Q_{a}

d_{a} = d_{w b} + d_{w b} \frac{Z - d - f}{f} \frac{n_{air}}{n_{water}} Q_{b}

d_{w a} = \sqrt{x_{i, a}^{2} + y_{i, a}^{2}}

d_{w a} = \sqrt{x_{i, b}^{2} + y_{i, b}^{2}}

Q_{a} = \frac{\sqrt{1 - {sin}^{2} θ_{air, a}}}{\sqrt{1 - (\frac{n_{air}}{n_{water}} sin θ_{air, a})}}

Q_{b} = \frac{\sqrt{1 - {sin}^{2} θ_{air, b}}}{\sqrt{1 - (\frac{n_{air}}{n_{water}} sin θ_{air, b})}}

sin θ_{air, a} = d_{a} / \sqrt{x_{i, a}^{2} + y_{i, a}^{2} + f^{2}}

sin θ_{air, b} = d_{a} / \sqrt{x_{i, b}^{2} + y_{i, b}^{2} + f^{2}} .

Figure 2.

The straight object imaged in water (blue curve) and in air (straight line).

Epipolar constraint and 3-D underwater measurement

Epipolar geometry plays an important role in stereo matching in binocular stereovision measurement. In the binocular stereo, two cameras capture a point in physical space from different angles, and two image points are respectively formed in these two images. To obtain the object depth information, the corresponding k-point pairs in two images should be found. Finding the corresponding point in the whole image is very time-consuming; thus, we need some constraints to reduce the search range.

As is shown in Figure 3, $I_{l}$ and $I_{r}$ are two images in binocular stereo. $p_{l}$ is one point in $I_{l}$ and its corresponding in $I_{r}$ is $p_{r}$ . Epipolar constraint means that the corresponding epipolar line in another image can be calculated by coordinates of $p_{l}$ , and its matching points must in the straight line. Thus, when we are searching the matching point in another image, we do not need to search the whole image, but only search in a straight line. Therefore, we can reduce the time-consuming of searching the whole image. The method of obtaining the epipolar line is shown as follows.

Figure 3.

Epipolar geometry.

Suppose that the projection matrixes of two cameras are $M_{l} = (M_{l 1}, T_{l})$ and $M_{r} = (M_{r 1}, T_{r})$ , the coordinate of a point on the object is $X_{W} = (x_{w}, y_{w}, z_{w},1)$ and the corresponding coordinates in two cameras are $p_{l}$ and $p_{r}$ . The operation of ${[t]}_{\times}$ is as follows

{[t]}_{\times} = [\begin{matrix} 0 & - t_{z} & t_{y} & ​ \\ t_{z} & 0 & - t_{x} & ​ \\ - t_{y} & t_{x} & 0 & ​ \end{matrix}]

The epipolar line equation can be expressed as follows

p_{r}^{T} {[T]}_{\times} M_{r 1} M_{l 1}^{- 1} p_{l} = 0

Here, $T = T_{r} - M_{r} M_{l}^{- 1} T_{l}$ . From equation (8), we can see that we can obtain an equation of a line from the coordinates of a point $p_{l}$ in the image. The coordinate equation is corresponding to the matching point in another image $I_{r}$ and the line is the corresponding epipolar line.

Underwater epipolar constraint

Based on the underwater imaging model, we can build the underwater epipolar equation. The underwater imaging model is shown in equation (3). The corresponding projection matrices of the two cameras are $M_{l} = (R_{l} T_{l})$ and $M_{r} = (R_{r} T_{r})$ , respectively. According to the formula (8), we can obtain an epipolar line equation in the image which is corresponded to the point $p_{l}$ in the image $I_{l}$ .

p_{r}^{T} {[T]}_{\times} R_{r} R_{l}^{- 1} p_{l} = 0

However, unlike model in air, the underwater imaging model is no longer the perspective imaging model. Given the intrinsic parameters matrices of two cameras $A_{l}$ and $A_{r}$ , equation (9) can be changed to

p_{r}^{T} A^{- T} {[T]}_{\times} R_{r} R_{l}^{- 1} p_{l} = 0

From equation (10), we can find that the epipolar constraint is related to the parameters of the vision system (camera intrinsic parameters and the two camera structural parameters). If the camera is placed in air, because the model is a linear model, the epipolar line is a straight line. The parameters of the camera in the underwater environment are not only related to the internal parameters of the camera but also to the parameters of the underwater imaging system, for example

\begin{array}{l} A' = [\begin{matrix} 1 / d x & 0 & u_{0} & ​ \\ 0 & 1 / d y & v_{0} & ​ \\ 0 & 0 & 1 & ​ \end{matrix}] \\ [\begin{matrix} n_{0} r & 0 & 0 & - \frac{n_{0} d x_{u} r}{f} & ​ \\ 0 & n_{0} r & 0 & - \frac{n_{0} d y_{u} r}{f} & ​ \\ 0 & 0 & \frac{- n_{0}}{\sqrt{n_{0}^{2} + {(\frac{f}{r})}^{2} - 1}} & \frac{n_{0} d}{\sqrt{n_{0}^{2} + {(\frac{f}{r})}^{2} - 1}} & ​ \end{matrix}] \end{array}

In equation (11), the intrinsic parameter matrices are no longer linear matrices and the epipolar line is a curve. Therefore, in underwater epipolar alignment, we need to find out the corresponding curve equation in another image, and then find the matching point on the curve. Because of the complexities of obtaining the curve equation in underwater environment, it is difficult to matching images. Thus, in this article, we propose an image rectification method to correct the images before the epipolar line registration.

Underwater image correction

The underwater imaging is no longer a perspective imaging model because of refraction, which leads to the epipolar line is no longer a straight line. According to the calibrated underwater camera parameters, we correct the images and map the underwater images into the air images. Then binocular correction and the subsequent 3-D measurement can be carried out on the image pairs.

As shown in Figure 4, assuming that AB is an incident light into water; all points on this line are mapped to the same pixel $P_{i}$ on the imaging plane. The infinity of ray AB is mapped to $P_{b}$ on the imaging plane. Ray CD is passing through the focus point C and parallel to the incident ray AB. Its corresponded mapping point on the image plane is $P_{b}$ . $P_{a}$ is the corresponding mapping point for air environment. When the depth of the object z changed, the mapping point varies from $P_{a}$ to $P_{b}$ . The relationship of $P_{w}$ (the point in the physical space), $P_{a}$ and $P_{b}$ , is as follows

\begin{array}{l} \frac{x_{a}}{x_{b}} = \frac{(z - f - d) tan θ_{water} + d tan θ_{air}}{(z - f) tan θ_{water}} \\ = 1 + \frac{d}{z - f} (\frac{tan θ_{air}}{tan θ_{water}} - 1) \end{array}

Figure 4.

Approximation model of underwater imaging and air imaging.

Thus, we find out the relationship between $P_{a}$ and $P_{b}$ . The relationship of imaging point $P_{i}$ and $P_{b}$ is as follows

{\begin{array}{l} \sqrt{x_{b}^{2} + y_{b}^{2}} & = f tan θ_{water} \\ \sqrt{x_{i}^{2} + y_{i}^{2}} & = f tan θ_{air} \\ n_{air} sin θ_{air} & = n_{water} sin θ_{water} \end{array}

According to the mirror image consistency constraint, points $P_{b}$ and $P_{i}$ have the same angle with the X-axis at the imaging plane, for example, $\frac{x_{a}}{y_{a}} = \frac{x_{i}}{y_{i}}$ . From equation (13), we can derive equation (14)

\frac{x_{b}^{2} + y_{b}^{2}}{x_{i}^{2} + y_{i}^{2}} = {(\frac{sin θ_{water}}{sin θ_{air}} \frac{cos​sin θ_{air}}{cos θ_{water}})}^{2} = \frac{x_{b}^{2}}{x_{i}^{2}}

Through equation (14), we can derive equation (15)

x_{b} = x_{i} \frac{n_{water}}{n_{air}} \frac{\sqrt{1 - {(n_{air} / n_{water})}^{2} {sin}^{2} θ_{air}}}{\sqrt{1 - {sin}^{2} θ_{air}}}

Thus, the coordination of imaging point in air environment is expressed as equation (16)

\begin{array}{l} x_{b} = x_{i} \frac{n_{water}}{n_{air}} \frac{\sqrt{1 - {(n_{air} / n_{water})}^{2} {sin}^{2} θ_{air}}}{\sqrt{1 - {sin}^{2} θ_{air}}} \\ (1 + \frac{d}{z - f} (\frac{tan θ_{air}}{tan θ_{water}} - 1)) \end{array}

Then, we convert to image coordinates by using equation (17)

{\begin{array}{l} u - u_{0} = & (u_{i} - u_{0}) \frac{n_{water}}{n_{air}} \frac{\sqrt{1 - {(n_{air} / n_{water})}^{2} {sin}^{2} θ_{air}}}{\sqrt{1 - {sin}^{2} θ_{air}}} \\ ​ & (1 + \frac{d}{z - f} (\frac{tan θ_{air}}{tan θ_{water}} - 1)) \\ v - v_{0} = & (v_{i} - v_{0}) \frac{n_{water}}{n_{air}} \frac{\sqrt{1 - {(n_{air} / n_{water})}^{2} {sin}^{2} θ_{air}}}{\sqrt{1 - {sin}^{2} θ_{air}}} \\ ​ & (1 + \frac{d}{z - f} (\frac{tan θ_{air}}{tan θ_{water}} - 1)) \end{array}

In fact, $z > > f$ , $z > > d$ , variations of the depth of field of the different object points have small influence to the underwater image correction results; thus, it can be ignored. Thus, we can get a fixed depth value through calibration, then correcting the underwater image into air.

According to equation (17), we can convert the underwater images in the air. The coordinates of a pixel on the underwater image are noninteger after the coordinate is transformed through equation (17). Thus, we need to interpolate the pixel after transformation. As the coordinate transformation formula established in this article is not reversible, we cannot obtain the coordinates of pixels on the projected image through converting the coordinates of pixels on the target image. We apply forward mapping-based bilinear interpolation method in this article.

Underwater binocular stereo 3-D measurement

After the underwater images are corrected and restored in air, we can perform binocular correction and make the left and right images aligned, and then do stereo matching. Binocular stereovision measurement is based on the disparity map after image feature points matching. Figure 5 shows the schematic of binocular stereovision measurement.

Figure 5.

The binocular stereo measurement.

3-D coordinates of the object is $P (x_{w}, y_{w}, z_{w})$ , and the coordinates of the object 3-D point in the left and right camera coordinate system are ${[x_{c l} y_{c l} z_{c l}]}^{T}$ and ${[x_{c r} y_{c r} z_{c r}]}^{T}$ , respectively. The pixel coordinates of the imaging points in the left and right cameras are $(u_{l}, v_{l})$ and $(u_{r}, v_{r})$ , respectively. The perspective transformations of left and right cameras are expressed as equations (18) and (19)

k_{l} [\begin{array}{l} u_{l} \\ v_{l} \\ 1 \end{array}] = [\begin{matrix} f_{l} & 0 & 0 \\ 0 & f_{l} & 0 \\ 0 & 0 & 1 \end{matrix}] [\begin{array}{l} x_{c l} \\ y_{c l} \\ z_{c l} \end{array}]

k_{r} [\begin{array}{l} u_{r} \\ v_{r} \\ 1 \end{array}] = [\begin{matrix} f_{r} & 0 & 0 \\ 0 & f_{r} & 0 \\ 0 & 0 & 1 \end{matrix}] [\begin{array}{l} x_{c l} \\ y_{c l} \\ z_{c l} \end{array}]

Suppose the relationship between the left and right camera coordinate systems is

[\begin{array}{l} x_{c r} \\ y_{c r} \\ z_{c r} \end{array}] = [R T] [\begin{array}{l} x_{c l} \\ y_{c l} \\ z_{c l} \\ 1 \end{array}]

According to equations (14) to (16), the real 3-D coordinates of the object can be calculated as equation (21)

{\begin{array}{l} x & = \frac{z x_{c l}}{f_{l}} \\ y & = \frac{z y_{c l}}{f_{l}} \\ z & = \frac{f_{l} (f_{l} t_{x} - u_{r} t_{z})}{u_{r} (r_{7} u_{l} + r_{8} v_{l} + f_{l} r_{9}) - f_{r} (r_{1} u_{l} + r_{2} v_{l} + f_{l} r_{e})} \end{array}

where $(u_{r}, v_{r})$ is the pixel coordinate of the imaging point in the right camera. $f_{l}$ and $f_{r}$ are the focal length of the left and right cameras, respectively.

Experimental results

The experimental equipment used in this study is an underwater binocular stereo system, including two Canon 70D cameras enclosed in a metal box with a glass window in the front (the thickness is 10 mm), Canon EF 50 mm f/1.4 lens, a glass sink filled with water, camera wireless controller, and so on, as shown in Figure 6. The two cameras are mounted on two slide rails, along which the positions of the cameras can be adjusted. As the cameras are fixed on the slide rails, the system is calibrated only once, unless the position of the cameras is changed. Then two remote controllers, CamFi II, are mounted on the cameras to connect with the wireless monitor. The camera shooting parameters can be set and the picture can be viewed on the remote monitor. There are four experimental subjects: two checkerboards (a plane pattern and a right-angle target), a stone sculpture, Easter avatar, and pirate avatar.

Figure 6.

Underwater measurement of experimental equipment. (a) 3-D CAD model of the waterproof housing. (b) Underwater stereo system. (c) Scheme of wireless controller. 3-D: three-dimensional.

First, the intrinsic parameters of camera can be calibrated in air by Zhang’s method.⁴³ Figure 7 is a highly accurate checkerboard with 400 mm × 300 mm dimensions and 12 × 9 grids, and each grid size is 30 mm × 30 mm with accuracy ±0.2 mm. It is a commercialized product to guarantee the accuracy. Sixteen images in different orientations are utilized to calibrate the intrinsic parameters of the camera. The acquired internal camera matrix is $M = [\begin{matrix} 4716.73 & 0 & 1970.44 \\ 0 & 4709.67 & 1428.59 \\ 0 & 0 & 1 \end{matrix}] .$

Figure 7.

A planer checkerboard in water.

After we acquire the intrinsic parameters of the camera, the stereo underwater system can be calibrated. The calibration target was placed into the water tank to acquire one image, as shown in Figure 8. The checkerboard size is $9 \times 10$ , and the grid size is 10 mm × 10 mm. Then Tsai’s method was used to obtain the extrinsic parameters: $R = [\begin{matrix} - 0.7011 & 0.7129 & - 0.0086 \\ - 0.0085 & - 0.0273 & - 0.9956 \\ - 0.7100 & - 0.6980 & 0.0253 \end{matrix}], T = [- {29.55, 207.01, 1262.01]}^{T}$ , and $d = 35.7$ mm. Here, we did not compute the distortion coefficients. On the=one hand, the distortion coefficients are very small for the DSLR lens. On the other hand, the physical underwater imaging model is utilized to calibrate the underwater stereo cameras. Although the lens distortion can surpass that of the refraction correction in some cases, Queiroz-Neto et al.¹⁵ have proven that only using the refraction correction can produce large deviation.

Figure 8.

A camera equipped with a remote controller and monitor.

The first experiment is to verify the measuring accuracy on a 2-D plane. The checkerboard is positioned parallel to the image plane, as shown in Figure 7. The experiment was divided into four groups, one in the air environment and another is in underwater. By utilizing the calibrated parameters, we can compute the coordinates of the checkerboard corners. After we compute the size of each grid, the accuracy can be evaluated using $e = d_{r} - d_{m}$ , where $d_{r}$ is the ground-truth edge length of the grid and $d_{m}$ is the measured size. The results in air and in water are illustrated in Figure 9. The x-axis is the corner numbers and the y-axis is the distance error. Figure 9(a) is the measured result in air; the maximal absolute error is 0.221 mm, and the mean error is 0.0612 mm. Figure 9(b) shows the underwater results; the maximal absolute error is 0.181 mm and the mean error is 0.0745 mm. From the results in air and in water, we find that our proposed underwater measurement system has a comparable accuracy with the result in air.

Figure 9.

Comparison of measurement error in 2-D of a checkerboard in air and in water. (a) Measurement errors in air. (b) Measurement errors in water ( $d = 35.7$ mm). 2-D: two-dimensional.

Then we experiment underwater binocular images registration to align the left and right images. Figure 10 shows the binocular alignment results without underwater image correction. We find that the same features cannot be found on an epipolar line. Two small portions in top and bottom results are zoomed in, from which the corresponding features greatly deviate from the epipolar line. The distances from the features (yellow points) to the epipolar lines vary. Next, the underwater image pairs are corrected and aligned by using the proposed method (see Figure 11). Comparing to the results in Figure 10, the stereo images are well aligned, as seen in the zoomed-out portions in the top and bottom images. We can find that the features located on the epipolar line, which will facilitate the feature matching.

Figure 10.

Binocular correction results before rectification.

Figure 11.

Binocular correction results after rectification.

To further verify the measurement accuracy in three dimensions, we utilize the calibration target to measure the accuracy as it has accurate ground-truth size. The measurement part is the lower half of the stereoscopic calibration plate, as shown in Figure 12. We divide measurement experiments into two groups. First, we use the coordinates of the corner point in left and right images and the external parameters of the binocular vision system to calculate the 3-D coordinates of the corner directly without matching in the air. Figure 13 is the experimental result, in which Figure 13(a) and (b) shows the different views of the computed 3-D coordinates of the corners. We can fit the plane with the computed 3-D coordinates of the corners and measure the distance of each 3-D point from the fitted plane. Figure 13(c) shows the measurement results and Figure 13(d) shows the distance error between each corner point. Measurement error is within 0.14 mm, and the mean error and standard deviation are 0.01 mm and 0.004, respectively. Second, the corrected underwater image pair is used for binocular stereo matching to obtain the 3-D coordinates of the matched corner points, as shown in Figure 14. Figure 14(a) and (b) shows different views of the 3-D points. The distance error to the fitted plane is illustrated in Figure 14(c) and the corner distance error is shown in Figure 14(d). The mean and standard error of the corner distance are 0.071 mm and 0.101, respectively. Then, we compare the underwater reconstruction result with a ray tracing-based method⁴¹ in Table 1. Star (*) denotes in-air calibration, underwater calibration, and ray tracing approach.⁴¹ In-air calibration showed to be the most inaccurate as it does not take refraction into account. It is clear from Table 1 that the proposed algorithm has high accuracy compared to the underwater calibration result of ray tracing approach.⁴¹

Figure 12.

The 3-D measurement subject. 3-D: three-dimensional.

Figure 13.

Reconstructed 3-D points in air and distance errors from the points to fitted plane and grid’s length differences. (a) Reconstructed 3-D corners. (b) The fitted calibration plate plane. (c) Distance errors of corner to the fitted plane. (d) Distance errors between corner points. 3-D: three-dimensional.

Figure 14.

Reconstructed 3-D points in water and the grid’s distance errors. (a) Reconstructed underwater 3-D corners. (b) The fitted underwater calibration plane. (c) The side view of the reconstructed corners. (d) Distance from the reconstructed corners to the fitted plane. 3-D: three-dimensional.

Table 1.

Comparison of the underwater 3-D reconstruction errors with the ray tracing method.⁴¹

	In air (*)	Underwater (*)	Ray tracing (*)	Our method
Mean error (μ)	5 mm	0.3 mm	0.1 mm	0.132 mm
Deviation (σ)	1.8 mm	1.4 mm	0.9 mm	0.004 mm

3-D: three-dimensional.

Conclusion

In this article, an underwater image-correction algorithm is proposed based on the underwater images nonlinear model. The method is used to reconstruct the underwater image captured by the binocular vision system according to the calibration parameters of the underwater camera. From the experimental results, it can be seen that the image can be well aligned after binocular correction using the corrected image. Then, the 3-D coordinates of the feature points in the underwater calibration plate are obtained by using the corrected underwater images. Underwater 3-D measurement is carried out using the corrected underwater image. The 3-D coordinates of the feature points can be used to fit the plane of the calibration plate. The accuracy of the method is verified by the distance of the 3-D coordinate of the feature point from the fitting plane and the result is performed well. In future study, the presented calibration approach would be utilized in dense stereo match and 3-D reconstruction in underwater environment.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Xiaojun Wu

References

Roman

Inglis

Rutter

. Application of structured light imaging for high resolution mapping of underwater archaeological sites. In: Proceedings of the 2010 IEEE OCEANS, Sydney, Australia, 24–27 May 2010, pp. 1–9.

Drap

. Underwater photogrammetry for archaeology. In: Carneiro Da Silva

(ed) Special applications of hotogrammetry. Rijeka: InTech, 2012, pp. 111–136.

Eric

Kovacic

Berginc

, et al. The impact of the latest 3D technologies on the documentation of underwater heritage sites. In: Proceedings of the IEEE digital heritage international congress, Marseille, France, 28 October–1 November 2013, vol. 2, pp. 281–288. Marseille, France: IEEE.

Canciani

Gambogi

Romano

, et al. Low cost digital photogrammetry for underwater archaeological site survey and artifact insertion. The case study of the Dolia wreck in secche della Meloria-Livorno-Italia. Int Arch Photogramm Remote Sens Spat Inf Sci 2003; 34(5): 95–100.

Shortis

. Calibration techniques for accurate measurements by underwater camera systems. Sensors 2015; 15: 30810–30827.

Sedlazeck

Koch

. Perspective and non-perspective camera models in underwater imaging-overview and error analysis. In: Anne

Reinhard

(eds) Theoretical foundations of computer vision, vol. 7474. Berlin: Springer, 2011, pp. 212–242.

Telem

Filin

. Photogrammetric modeling of underwater environments. ISPRS J Photogramm Remote Sens 2010; 65: 433–444.

Tan

Seet

Sluzek

, et al. A novel application of range-gated underwater laser imaging system (ULIS) in near-target Turbid medium. Opt Lasers Eng 2005; 43: 995–1009.

Massot-Campos

Oliver-Codina

. Underwater laser-based structured light system for one-shot 3D reconstruction. In: Proceedings of the 5th Martech international workshop on marine technology, Girona, Spain, 2–5 November 2014.

10.

Zhang

Wang

Hou

, et al. Three-dimensional shape measurement for an underwater object based on two-dimensional grating pattern projection. Opt Laser Technol 2011; 43: 801–805.

11.

Bianco

Gallo

Bruno

, et al. A comparative analysis between active and passive techniques for underwater 3D reconstruction of close-range objects. Sensors 2013; 13: 11007–11031.

12.

Bräuer-Burchardt

Heinze

Schmidt

, et al. Underwater 3D surface measurement using fringe projection based scanning devices. Sensors 2016; 16: 13.

13.

Beall

Lawrence

Ila

, et al. 3D reconstruction of underwater structures. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), 18–22 October 2010, pp. 4418–4432.

14.

Eirìkssona

Wilma

Pedersen

, et al. Precision and accuracy in structured Light 3-D scanning, vol. XL-5/W8. Berlin: The International archives of the photogrammetry, remote sensing and spatial information sciences, 2016.

15.

Queiroz-Neto

Carceroni

Barros

, et al. Underwater stereo. In: The 17th Brazilian Symposium on Computer Graphics and Image Processing, Curitiba, Brazil, Brazil, 20–20 October 2004, pp. 170–177. IEEE Computer Society.

16.

Sanchez-Ferreira

Mori

Llanos

, et al. Development of a stereo vision measurement architecture for an underwater robot. In: IEEE Latin American symposium on circuits and systems, Cusco, Peru, 23 May 2013, pp. 1–4.

17.

Shortis

Harvey

Abdo

. A review of underwater stereo-image measurement for marine biology and ecology applications. In: Giboson

Atkinson

RJA

Gordon

JMD

(eds) Oceanography and marine biology: an annual review, 2009, vol. 47. Boca Raton; CRC Press, pp. 257–292.

18.

Schechner

Karpel

. Recovery of underwater visibility and structure by polarization analysis. IEEE J Oceanic Eng 2005; 30(3): 570–587.

19.

Treibitz

Schechner

. Active polarization descattering. IEEE T Pattern Anal 2009; 31(3): 385–399.

20.

Treibitz

Schechner

. Instant 3descatter. IEEE Comput Soc Conf Comput Vis Pattern Recogn, New York, NY, USA, 17–22 June 2006; 2: 1861–1868.

21.

Ivanoff

Cherney

. Correcting lenses for underwater use. J Soc Motion Pict T 1960; 69(4): 264–266.

22.

Ray

. Applied photographic optics. In: Applied photographic optics lenses & optical systems for photography film video & electronic imaging, Focal Press, Subsequent, 1988.

23.

Kunz

Singh

. Hemispherical refraction and camera calibration in underwater vision. Proc MTS/IEEE oceans, Quebec City, QC, Canada, 15–18 September 2008, pp. 1–7.

24.

Kwon

Casebolt

. Effects of light refraction on the accuracy of camera calibration and reconstruction in underwater motion analysis. Sports Biomech 2006; 5(2): 315–340.

25.

Narasimhan

Nayar

. Structured light methods for underwater imaging: light stripe scanning and photometric stereo. Proc MTS/IEEE OCEANS, 2005; 3: 2610–2617.

26.

Ferreira

Costeira

Santos

. Stereo reconstruction of a submerged scene. Pattern Recogn Image Anal 2005; 3522: 102–109.

27.

Lavest

Rives

Lapresté

. Underwater camera calibration, ECCV. Berlin, Heidelberg: Springer, 2000, pp. 654–668.

28.

Pizarro

Eustice

Singh

. Relative pose estimation for instrumented, calibrated imaging platforms. In: Proceedings of digital image computing techniques & applications, 2003, pp. 601–612.

29.

Shortis

Harvey

. Design and calibration of an underwater stereo-video system for the monitoring of marine fauna populations. Int Arch Photogramm Remote Sens 1998; 32(5): 792–799.

30.

Kwon

Lindley

Sanders

. Applicability of 4 localized-calibration methods in underwater motion analysis. In: XVIII International symposium on biomechanics in sports, Hong Kong, 25–30 June 2000.

31.

Treibitz

Schechner

Kunz

, et al. Flat refractive geometry. IEEE T Pattern Anal 2011; 34(1): 51–65.

32.

Jordt-Sedlazeck

Koch

. Refractive structure-from-motion on underwater images. In: IEEE international conference on computer vision, Sydney Conference Centre, Darling Harbour, Sydney, 1–8 December 2013, pp. 57–64.

33.

Agrawal

Ramalingam

Taguchi

, et al. A theory of multi-layer flat refractive geometry. In: IEEE conference on computer vision and pattern recognition (CVPR), Providence, Rhode Island, 18–20 June 2012, pp. 3346–3353.

34.

Chen

Yang

. Two-view camera housing parameters calibration for multi-layer flat refractive interface. In: IEEE conference on computer vision and pattern recognition (CVPR), 23–28 June 2014, pp. 524–531. DC, USA: IEEE Computer Society Washington.

35.

Yau

Gong

Yang

. Underwater camera calibration using wavelength triangulation. In: IEEE conference on computer vision and pattern recognition (CVPR), 23–28 June 2013, pp. 2499–2506. DC, USA: IEEE Computer Society Washington.

36.

Kang

Yang

. Experimental study of the influence of refraction on underwater three-dimensional reconstruction using the SVP camera model. Appl Optics 2012; 51(31): 7591–7603.

37.

Chari

Sturm

. Multiple-view geometry of the refractive plane. BMVC 2009; 2: 181–186.

38.

Chang

Chen

. Multi-view 3D reconstruction for scenes under the refractive plane with known vertical direction. IEEE international conference on computer vision, Colorado Springs, USA, 21–23 June 2011, pp. 351–358.

39.

Sedlazeck

Koch

. Calibration of housing parameters for underwater stereo-camera rigs. BMVC, University of Dundee, 29 August–2 September 2011, pp. 1–11.

40.

Yau

Underwater camera calibration and 3D reconstruction. Thesis of Master of Science, University of Alberta, Edmonton, 2014.

41.

Pedersen

Bengtson

S H

Gade

, et al. Camera calibration for underwater 3D reconstruction based on ray tracing using Snell’s Law. In: CVPR workshop, Salt Lake City, USA, 18–22 June 2018.

42.

Tsai

RY.

An efficient and accurate camera calibration technique for 3D machine vision. In: Proceedings of IEEE conference on computer vision and pattern recognition, Miami Beach, FL, 22–26 June 1986, pp. 364–374.

43.

Zhang

. A flexible new technique for camera calibration. IEEE T Pattern Anal 2000; 22(11): 1330–1334.