Paracatadioptric camera calibration based on properties of polar line of infinity point with respect to circle and line

Abstract

Two linear calibration methods based on space-line projection properties for a paracatadioptric camera are presented. Considering the central catadioptric system, a straight line is projected into a circle on the viewing spherical surface for the first projection. The tangent lines in a group at antipode point pairs with respect to the circle are parallel, with the infinity point being the intersection point; therefore, the infinity line can be obtained from two groups of antipode point pairs. Further, the direction of the polar line of an infinity point with respect to the circle is orthogonal to the direction of its infinity point. Hence, on the imaging plane, images of the circular points or orthogonal vanishing points are used to determine the intrinsic parameters. On the basis of the properties of the antipodal point pairs and a least-squares fitting, a corresponding optimization algorithm for line image fitting is proposed. Experimental results demonstrate the robustness of the two calibration methods, that is, for images of the circular points and orthogonal vanishing points.

Introduction

The rapid development of computer vision technology has caused an increase in the stringency of visual performance requirements. ¹ In general, increasing the field of view improves the visual performance of a camera. ^2,3 An effective means of enhancing the field of view of a camera is to incorporate mirrors into the camera, as proposed by Hecht and Zajac. ⁴ The resulting configuration system is called a catadioptric system. Such a system can be classified into two types on the basis of the presence or absence of a unique effective viewpoint, that is, a central or noncentral system, respectively. ⁵ The surface of the mirror of a central catadioptric system is categorized into four types: parabolic, hyperbolic, ellipsoidal, and planar mirrors. The basic feature points in an image captured by a central catadioptric camera can easily be converted into three-dimensional (3-D) coordinates by a projection inverse—a technique broadly applied in vision systems. ^{4 –6}

Geyer and Daniilidis ⁶ proposed a unified model of projection image formation for a central catadioptric system that is equivalent to two-step projection by a unit sphere. The calibration methods for central catadioptric cameras can be classified into four categories: self-, ^7,8 point-, ^{9 –11} line-, ^{12 –19} and sphere-based calibration. ^{20 –23} This article considers methods that exploit the projection properties of one line in the unit sphere model to calibrate a paracatadioptric (i.e. with a parabolic mirror) camera.

Many calibration methods that employ 3-D space lines exist for central catadioptric cameras. The advantage of using lines for calibration is that the target scene usually contains lines; thus, a specific template is not required. Geyer and Daniilidis ¹² calibrated a paracatadioptric camera using an image of two groups of parallel lines and an image of three lines. They also proposed unifying projection model theory for central catadioptric systems with a unique effective viewpoint. ¹³ In their model, the projective lines and points are dual to each other. They also posited a geometric argument in which the center catadioptric system except the planar projection can be calibrated by one image of lines. Further, Barreto and Araujo ¹⁴ studied the line images and presented several projective invariant properties of the central catadioptric system using one image of three or more lines to calibrate the camera. The properties include the fact that the intersections of the images of the lines are the principal point, and the polar lines of the principal point with respect to the line images intersect the line image at the image of the circular points. According to the cross ratio invariants of the line image, the mirror parameter is determined. In addition, Wu et al. ¹⁵ derived the linear constraints of the line image according to the relationship between the projection of the antipodal points, the projection properties of the lines in the central catadioptric camera, and the cross ratio invariants. Any central catadioptric camera can be calibrated by linear constraints using one image of three or more lines. Ying and Hu ¹⁶ proposed a calibration method that uses the geometric invariants of an image of the line and sphere in central catadioptric cameras. One-line image provides three invariants and a sphere provides two invariants. The images of two lines or three spheres can calibrate central catadioptric cameras. Further, Ying and Zha ¹⁷ reported that all line images in a central catadioptric system with the same intrinsic parameters belong to the family of conics with two degrees of freedom. They detected line images via the Hough transform and calibrated the camera according to the relationship between the camera intrinsic parameters and the family of conics. Vandeportaele et al. ¹⁸ used one image of three or more lines to calibrate paracatadioptric cameras by applying the geometric distance obtained from the intrinsic parameters instead of the algebraic distance. The approach demonstrates that a line is projected to a straight line or circular arc. Finally, Wu et al. ¹⁹ proposed a calibration method for paracatadioptric cameras based on lines that are not a conic fitting under a single view. The method establishes sets of linear equations for the camera intrinsic parameters, focal lengths, and skew factor, and the principal point is the image center of the mirror contour.

On the basis of the abovementioned literature analysis, this article presents methods of paracatadioptric camera calibration utilizing the projection of one line and the properties of the polar line of an infinity point with respect to a circle. Considering the unit sphere model, a straight line is projected to a circle of the spherical surface. For a point taken on the circle, the two tangents passing through an antipode point pair are parallel and intersect at the infinity point. Further, the direction of the polar line of an infinity point with respect to the circle is orthogonal to the direction of the infinity point. Therefore, on the image plane, images of the circular points and orthogonal vanishing points can be obtained to determine the camera intrinsic parameters. Then, on the basis of the properties of the antipodal point pairs and a least-squares fitting, an optimization algorithm for line image fitting can be implemented. In this work, two such calibration algorithms are proposed.

The remainder of this article is organized as follows. A review of the unit sphere model for central catadioptric cameras and several related concepts are presented in the next section. Then, two calibration algorithms for paracatadioptric cameras from one line image are described in detail, and an algorithm for line image fitting is presented. The two calibration algorithms are next verified by simulation tests and real experiments. Finally, some concluding remarks are presented.

Preliminaries

In this section, we briefly review the projection process of a central catadioptric system with the unity sphere model, ⁶ the relationship of the antipodal image points, ¹⁵ and the properties of the polar line of the infinity point with respect to a circle. ²⁴

Central catadioptric projection system

As shown in Figure 1, the projection of a 3-D scene point in the central catadioptric camera is equivalent to two-step projection by a unit sphere. ⁶ First, in the world coordinate system $\mathit{O}-x_w{y}_w{z}_w$ , the 3-D space point $\mathit{A}={\left[\begin{array}{cc}\begin{array}{cc}x& y\end{array}& \begin{array}{cc}z& 1\end{array}\end{array}\right]}^{\text{T}}$ is projected to two different points ${\mathit{A}}_{\pm }={\left[\begin{array}{cccc}\pm x/||\mathit{A}||& \pm y/||\mathit{A}||& \pm z/||\mathit{A}||& 1\end{array}\right]}^{\text{T}}$ with $||\mathit{A}||=\sqrt{x^2+y^2+z^2}$ on the unit sphere of a reflective mirror centered at the focal point O ; the two points ${\mathit{A}}_{\pm }$ constitute a group of antipode points or antipode point pairs. Second, the points ${\mathit{A}}_{\pm }$ are projected to the two points ${\mathit{a}}_{\pm }$ on the catadioptric image plane Π by a pinhole camera with the optical center ${\mathit{O}}_c$ . The points ${\mathit{a}}_{\pm }$ are called a group of antipodal image points. The image plane Π is perpendicular to the line $\mathit{O}{\mathit{O}}_c$ , intersecting at the principal point, and the line $\mathit{O}{\mathit{O}}_c$ is the optical axis of the pinhole camera. The distance between points O and ${\mathit{O}}_c$ is called the mirror parameter distance. The relationship between the mirror type and the value of ξ is as follows: a plane mirror if ξ = 0, an ellipsoid or hyperboloid mirror if 0 < ξ < 1, and a paraboloid mirror if ξ = 1.

Figure 1.

Projection model of the 3-D point A and line L in the central catadioptric system. 3-D: three-dimensional.

Let the pinhole-camera intrinsic parameter matrix be

$\mathit{K}=\left[\begin{array}{ccc}{f}_u& s& u_0\\ 0& f_v& v_0\\ 0& 0& 1\end{array}\right]$

1

where f_u and f_v are the focal lengths for the u- and v-axis directions, respectively, $\left[\begin{array}{cc}{u}_0& v_0\end{array}\right]$ are the coordinates of the principal point in the image plane, and s is the skew factor for the u and v axes. The antipodal points ${\mathit{A}}_{\pm }$ are projected to the catadioptric image points ${\mathit{a}}_{\pm }$ as follows

${\lambda }_{1,2}{\mathit{a}}_{\pm }=\mathit{K}\left[\begin{array}{cc}I& t\end{array}\right]{\mathit{A}}_{\pm }=\mathit{K}\left({\tilde{\mathit{A}}}_{\pm }+{\left[\begin{array}{ccc}0& 0& \xi \end{array}\right]}^{\text{T}}\right)$

2

where λ ₁ and λ ₂ are two nonzero scale factors, t is the translation vector between the world and camera coordinate system, and ${\tilde{\mathit{A}}}_{\pm }=$ ${\left[\begin{array}{ccc}\pm x/\Vert \mathit{A}\Vert & \pm y/\Vert \mathit{A}\Vert & \pm z/\Vert \mathit{A}\Vert \end{array}\right]}^{\text{T}}$ denotes the nonhomogeneous coordinates of ${\mathit{A}}_{\pm }$ .

Similarly, the projection of a 3-D scene line in the central catadioptric camera is equivalent to two-step projection by a unit sphere model. First, a 3-D space line L is projected to a circle S on the unit sphere through the viewpoint O ; second, the circle S is projected to the conic C on the image plane through the virtual camera O _c . This projection process is shown in Figure 1.

Proposition 1

In Figure 1, if $\left\{{\mathit{a}}_{+},{\mathit{a}}_{-}\right\}$ is a group of antipodal image points in a central catadioptric system, their relationship is as follows ¹⁵

$\frac{1+\sqrt{1+\tau {\mathit{a}}_{+}^{\text{T}}\mathit{\omega }{\mathit{a}}_{+}}}{{\mathit{a}}_{+}^{\text{T}}\mathit{\omega }{\mathit{a}}_{+}}{\mathit{a}}_{+}+\frac{1+\sqrt{1+\tau {\mathit{a}}_{-}^{\text{T}}\mathit{\omega }{\mathit{a}}_{-}}}{{\mathit{a}}_{-}^{\text{T}}\mathit{\omega }{\mathit{a}}_{-}}{\mathit{a}}_{-}=2\mathit{p}$

3

where $\tau =\left(1-{\xi }^2\right)/\xi$ and $\mathit{\omega }={\mathit{K}}^{-\text{T}}{\mathit{K}}^{-1}$ is the image of the absolute conic (IAC). In the paracatadioptric system, equation (3) can be simplified to

$\frac{1}{{\mathit{a}}_{+}^{\text{T}}\mathit{\omega }{\mathit{a}}_{+}}{\mathit{a}}_{+}+\frac{1}{{\mathit{a}}_{-}^{\text{T}}\mathit{\omega }{\mathit{a}}_{-}}{\mathit{a}}_{-}=\mathit{p}$

4

Properties of the polar line of infinity point with respect to a circle

Definition

Given a point x and conic C , define a line $\mathit{l}=\mathit{C}\mathit{x}$ . The line l is called the polar of x with respect to C , and the point x is the pole of l with respect to C . ²⁴

Proposition 2

The direction of the polar line of an infinity point with respect to a circle and the direction of the infinity point are orthogonal.

Proof

As shown in Figure 2, the polar line of the infinity point V with respect to the circle O is the diameter r along which the direction has an infinity point V ′. The lines ${\mathit{r}}_1$ and ${\mathit{r}}_2$ that pass through the two endpoints A and B of r are tangential to O ; therefore, ${\mathit{r}}_1\parallel {\mathit{r}}_2$ and intersect at the infinity point V . ²⁴ The tangents ${\mathit{r}}_1$ and r are orthogonal; thus, V and ${\mathit{V}}^{\prime }$ form a group of orthogonal infinity points depending on the properties of the projective geometry. □

Figure 2.

Properties of the polar line of infinity point with respect to a circle.

Paracatadioptric camera calibration and line image fitting

In this section, through an analysis of the projection properties of one line for a paracatadioptric camera, the manner in which the properties of the polar line of the infinity point with respect to a circle can be used for the calibration of a paracatadioptric camera is explained, which corresponds to proposition 3 (for orthogonal vanishing points) and proposition 4 (for imaged circular points) in the following. According to the properties of the antipodal points and a least-squares fitting, an optimization algorithm for line image fitting is proposed.

Properties of the line image for a paracatadioptric camera

Proposition 3

For a central catadioptric system, given two different points on the line image, we can obtain two groups of orthogonal vanishing points.

Proof

As shown in Figure 1, for the unit sphere model, the projection of the line L is a circle S on the unit spherical surface, with the center of the circle corresponding to the unity sphere center O . We take the antipodal points ${\mathit{A}}_{\pm }$ on the projection circle S , which are the two endpoints of a diameter of S , in accordance with the central catadioptric projection model. The tangents of S at ${\mathit{A}}_{\pm }$ are parallel and intersect at the infinity point ${\mathit{D}}_{1\infty }$ according to the properties of the diameter of a circle. Similarly, we take a pair of antipodal points $B_{\pm }$ on S . The tangents of S at points $B_{\pm }$ are parallel and intersect at another infinity point ${\mathit{D}}_{2\infty }$ . The line passing through D _1∞ and D _2∞ is the infinity line ${\mathit{L}}_{\infty }$ . The polar lines of D _1∞ and D _2∞ with respect to S and the infinity line ${\mathit{L}}_{\infty }$ intersect at the infinity points $\mathit{D}{\prime }_{1\infty }$ and ${{\mathit{D}}^{\prime }}_{2\infty }$ . According to proposition 2, the infinity points D _{i ∞} and $\mathit{D}{\prime }_{i\infty }$ (i = 1, 2) form a group of orthogonal infinity points.

As shown in Figure 3, on the catadioptric image plane, the line image is denoted by C , and the images of the points A ₊ and A _– are ${\mathit{a}}_{+}$ and ${\mathit{a}}_{-}$ , respectively. The tangents of C at ${\mathit{a}}_{+}$ and ${\mathit{a}}_{-}$ intersect at the vanishing point ${\mathit{d}}_1$ , which is the image of the infinity point ${\mathit{D}}_{1\infty }$ , with

${\mathit{d}}_1=\left(\mathit{C}\cdot {\mathit{a}}_{+}\right)\times \left(\mathit{C}\cdot {\mathit{a}}_{-}\right)$

5

where × is the vector product and ⋅ is the product. We denote the images of B ₊ and B _– by ${\mathit{b}}_{+}$ and ${\mathit{b}}_{-}$ , respectively. The tangents of C at ${\mathit{b}}_{+}$ and ${\mathit{b}}_{-}$ intersect at the vanishing point ${\mathit{d}}_2$ , which is the image of the infinity point ${\mathit{D}}_{2\infty }$ , with

${\mathit{d}}_2=\left(\mathit{C}\cdot {\mathit{b}}_{+}\right)\times \left(\mathit{C}\cdot {\mathit{b}}_{-}\right)$

6

Figure 3.

Polar line of a vanishing point with respect to a line image.

The vanishing line l passes through ${\mathit{d}}_1$ and ${\mathit{d}}_2$ on the plane containing S , being the image of the infinity ${\mathit{L}}_{\infty }$

$\mathit{l}={\mathit{d}}_1\times {\mathit{d}}_2$

7

The polar lines of ${\mathit{d}}_1$ and ${\mathit{d}}_2$ with respect to C intersect l at the vanishing points ${{\mathit{d}}^{\prime }}_1$ and ${{\mathit{d}}^{\prime }}_2$ , respectively, which are the images of the infinity points ${{\mathit{D}}^{\prime }}_{1\infty }$ and ${{\mathit{D}}^{\prime }}_{2\infty }$ , respectively. Thus

${{\mathit{d}}^{\prime }}_1=\left(\mathit{C}\cdot {\mathit{d}}_1\right)\times \mathit{l}$

8

${{\mathit{d}}^{\prime }}_2=\left(\mathit{C}\cdot {\mathit{d}}_2\right)\times \mathit{l}$

9

On the basis of proposition 2, d _{i ∞} and $\mathit{d}{\prime }_{i\infty }$ (i = 1, 2) constitute a group of orthogonal vanishing points. □

The choice of the locations of the images points ${\mathit{a}}_{+}$ and ${\mathit{b}}_{+}$ is arbitrary, but they might be as far away from each other as possible. If the distance between them is too close, the distance between vanishing points ${\mathit{d}}_1$ and ${\mathit{d}}_2$ is too small, and they may even coincide because of the noise effect. Thus, the accuracy of the vanishing line l is decreased.

Proposition 4

For a central catadioptric system, given two different points on the line image, the images of the circular points can be obtained.

Proof

For proposition 3, the vanishing line l can be obtained. Thus, l intersects the conic C at the images of the circular points ${\mathit{m}}_I$ and ${\mathit{m}}_J$ . ²⁴

Line image fitting in a central catadioptric system

The image of a straight line is conic in a central catadioptric system, but only a section of the conic is visible in the image plane. Therefore, the line image fitting is inaccurate when only a section of the conic is known.

Proposition 5

In a central catadioptric system (see Figure 1), with knowledge of the image point ${\mathit{a}}_{i+}$ and its antipodal image point ${\mathit{a}}_{i-}$ $\left(i\ge 5\right)$ , conic fitting can be performed.

Proof

Let the conic C be the image of a straight line observed with a central catadioptric system. We take $N\left(N\ge 5\right)$ points ${\mathit{a}}_{i+}$ on the visible section of the conic with $i=1,2,3,\dots ,N$ . The projective contour of the mirror can be completely extracted, and the equation of the projective contour of the mirror can be formulated through a least-squares fitting to initialize the camera intrinsic parameters $\tilde{\mathit{K}}$ ¹⁶ ; hence, according to proposition 1, the antipodal image point ${\mathit{a}}_{i-}$ of point ${\mathit{a}}_{i+}$ can be calculated. Let the conic C be

$\mathit{C}=a{x}^2+bxy+c{y}^2+dx+ey+f=0$

10

We minimize the algebraic distance $\mathit{C}\left({\mathit{x}}_j,y_j\right)$ from the antipodal point pairs ${\mathit{a}}_{i\pm }$ to C ²⁵ ; then

$\Gamma ={\displaystyle \sum _{j=1}^{2N}{\mathit{C}}^2}\left(x_j,y_j\right)$

11

where (x_j , y_j ) are the coordinates of a _i+ or a _i−. We minimize the objective function in equation (11), to which there always exists the trivial solution a = b = c = d = e = d = 0. In order to avoid the zero solution, we normalize $\mathit{C}\left(\mathit{x},y\right)$ with a = 1, b = 1, c = 1, d = 1, e = 1, and f = 1. When a = 1, let ${\mathbf{X}}_1={\left[b\hspace{0.17em}\hspace{0.17em}c\hspace{0.17em}\hspace{0.17em}d\hspace{0.17em}\hspace{0.17em}e\hspace{0.17em}\hspace{0.17em}f\right]}^{\text{T}}$ ; thus, the equation $\mathit{C}\left(x_j,y_j\right)=0$ becomes ${\mathbf{a}}_j^{\text{T}}{\mathbf{X}}_1-{\mathbf{b}}_j=0$ , where ${\mathbf{a}}_j={\left[x_j{y}_j\hspace{0.17em}\hspace{0.17em}{y}_j^2\hspace{0.17em}\hspace{0.17em}{x}_j\hspace{0.17em}\hspace{0.17em}{y}_j\hspace{0.17em}\hspace{0.17em}1\right]}^{\text{T}}$ and ${\mathbf{b}}_j=-x_j^2$ . Given 2N points, we obtain the following equation

$\mathbf{A}{\mathbf{X}}_1=\mathbf{b}$

12

where $\mathbf{A}={\left[a_1\hspace{0.17em}\hspace{0.17em}{a}_2\hspace{0.17em}\dots \hspace{0.17em}\hspace{0.17em}{a}_{2N}\right]}^{\text{T}}$ and $\mathbf{b}={\left[b_1\hspace{0.17em}\hspace{0.17em}{b}_2\hspace{0.17em}\dots \hspace{0.17em}\hspace{0.17em}{b}_{2N}\right]}^{\text{T}}$ . By minimizing the objective function in equation (11), we obtain

$\Gamma \left({\mathbf{X}}_1\right)\hspace{0.17em}=\hspace{0.17em}{\left(\mathbf{A}{\mathbf{X}}_1-\mathbf{b}\right)}^{\text{T}}\left(\mathbf{A}{\mathbf{X}}_1-\mathbf{b}\right)$

13

We then minimize the objective function; that is, we obtain the partial derivative of equation (13) with respect to ${\mathbf{X}}_1$ , where the partial derivative is zero

$2{\mathbf{A}}^{\text{T}}\left(\mathbf{A}{\mathbf{X}}_1-\mathbf{b}\right)=0$

14

The solution of equation (14) is

${\mathbf{X}}_1={\left({\mathbf{A}}^{\text{T}}\mathbf{A}\right)}^{-1}{\mathbf{A}}^{\text{T}}\mathbf{b}$

15

and the coefficients of the conic are ${\mathbf{X}}_1^1={\left[\begin{array}{cc}1& {\mathbf{X}}_1^{\text{T}}\end{array}\right]}^{\text{T}}$ .

Similarly, through normalization with b = 1, c = 1, d = 1, e = 1, and f = 1, the coefficients of a conic, that is, ${\mathbf{X}}_2^1,\hspace{0.17em}{\mathbf{X}}_3^1,\hspace{0.17em}{\mathbf{X}}_4^1,\hspace{0.17em}{\mathbf{X}}_5^1,\hspace{0.17em}\text{and}\hspace{0.17em}{\mathbf{X}}_6^1$ , can be obtained; thus, the equation of the line image can be determined from

$\mathbf{X}\hspace{0.17em}=\hspace{0.17em}\frac{1}{6}{\displaystyle \sum _{i\hspace{0.17em}=\hspace{0.17em}1}^6{\mathbf{X}}_i^1}$

16

Line-based calibration method for a paracatadioptric camera

A group of orthogonal vanishing points d ₁ and ${{\mathit{d}}^{\prime }}_1$ , whose homogeneous coordinates are ${\mathit{d}}_1={\left[\begin{array}{ccc}{u}_1& v_1& 1\end{array}\right]}^{\text{T}}$ and ${{\mathit{d}}^{\prime }}_{\text{1}}\hspace{0.17em}=\hspace{0.17em}{\left[\begin{array}{ccc}{u^{\prime }}_1& {v^{\prime }}_1& 1\end{array}\right]}^{\text{T}}$ , can provide one linear constraint on the IAC. ²⁴ The constraint is as follows

${\mathit{d}}_1^{\text{T}}\mathit{\omega }{{\mathit{d}}^{\prime }}_{\text{1}}\hspace{0.17em}=\hspace{0.17em}0$

17

A group of circular points $\mathit{I}{\left[\begin{array}{ccc}1& \text{i}& 0\end{array}\right]}^{\text{T}}$ and $\mathit{J}{\left[\begin{array}{ccc}1& -\text{i}& 0\end{array}\right]}^{\text{T}}$ constitutes two special points on the absolute conic, and their images ${\mathit{m}}_I={\left[\begin{array}{ccc}{x}_1+x_2\text{i}& y_1+y_2\text{i}& \hspace{1em}1\end{array}\right]}^{\text{T}}$ and ${\mathit{m}}_J={\left[\begin{array}{ccc}{x}_1-x_2\text{i}& y_1-y_2\text{i}& 1\end{array}\right]}^{\text{T}}$ lie on the IAC, which yields the following

${\mathit{m}}_I^{\text{T}}\mathit{\omega }{\mathit{m}}_I=0,{\mathit{m}}_J^{\text{T}}\mathit{\omega }{\mathit{m}}_J=0$

18

The points m _I and m _J are a group of conjugate points; thus, equation (18) only provides two linear constraints on the IAC ²⁶

$\text{Re}\left({\mathit{m}}_I^{\text{T}}\mathit{\omega }{\mathit{m}}_I\right)=0,\hspace{0.17em}\text{Im}\left({\mathit{m}}_I^{\text{T}}\mathit{\omega }{\mathit{m}}_I\right)=0$

19

where Re and Im represent the real and imaginary parts, respectively.

Note that a group of orthogonal vanishing points provides one linear constraint on the IAC, but all of the orthogonal vanishing points for a plane only provide two constraints on the IAC. ²⁴ A group of the images of the circular points can provide two constraints on the IAC. Therefore, at least three views of one line should be provided to satisfy propositions 3 and 4.

The two algorithms proposed in this study, that is, for the images of the circular points (proposition 4) or orthogonal vanishing points (proposition 3), consist of the following steps:

Step 1: Views including at least one line image are input, and the pixel coordinates of the line image and the projective contour of the mirror are extracted in $n\left(n\ge 3\right)$ views; the equation of the projective contour of the mirror can be formulated through a least-squares fitting to initialize the camera’s intrinsic parameters $\tilde{\mathit{K}}$ and the IAC $\tilde{\mathit{\omega }}={\tilde{\mathit{K}}}^{-\text{T}}{\tilde{\mathit{K}}}^{-1}$ ¹⁶ ; the equations of the line images are estimated using Proposition 5.

Step 2: Two different points, a ₊ and b ₊, are taken on every line image, and their antipodal image points, a _– and b _–, are determined using equation (4); the vanishing points d ₁ and d ₂ are determined using equations (5) and (6).

Step 3: The vanishing line l is determined using equation (7); the orthogonal vanishing points ${{\mathit{d}}^{\prime }}_1$ and ${{\mathit{d}}^{\prime }}_2$ are obtained using equations (8) and (9).

Step 4: The two common intersection points (i.e. the imaged circular points) for the vanishing line l and line image C are calculated using the “solve” function in MATLAB.

Step 5: (a) When the orthogonal vanishing points are known, solve for ω using equation (17).

(b) When the images of the circular points are known, solve for ω using equation (19).

Step 6: Determine K by performing the inverse via Cholesky factorization of ω .

Experiments

We tested the validity and robustness of two calibration algorithms via simulation tests and real experiments. Further, we compared the results with Barreto and Araujo’s ¹⁴ three-line method and Wu et al.’s ¹⁵ calibration methods. In the following, the methods derived from propositions 3 and 4 are referred to as “Ours 1” and “Ours 2,” respectively.

Experiment using simulation data

Let the simulated central catadioptric camera be a paracatadioptric camera, that is, ξ = 1, with the five intrinsic parameters being $f_u=700$ , $f_v=710$ , s = 0.4, $u_0=450$ , and $v_0=450$ . Three images are needed in our calibration methods. Figure 4 shows three line images with the simulated camera; the red thick circle is the projection of the mirror contour, and the three purple curves (thin lines) are the three line images.

Figure 4.

Simulated line images.

In the simulation experiment, a line image was fit according to the method presented in proposition 5 (step 1 of both algorithms). Five hundred points were chosen on each line image, with 200 points chosen on the mirror projective contour. To test the sensitivity of the calibration methods to noise, Gaussian noise with a zero mean and a standard deviation σ of zero to three pixels was added to the chosen pixel coordinates. After the noise was added, we compared the difference in the results given by our proposed methods (corresponding to propositions 3 and 4) and those reported in the literature. ^14,15 For each noise level, 1000 independent trials were performed to determine the absolute error of the intrinsic parameters. The average results of the absolute error are shown in Figure 5(a) to (e). The results indicate that, when σ is zero, the absolute errors of the intrinsic parameters are zero; the absolute errors of our two methods and the methods of Barreto and Araujo ¹⁴ and Wu et al. ¹⁵ increase virtually linearly with the noise level σ. However, the absolute errors of our proposed algorithms increase more slowly than those of the other calibration algorithms. The contrast in the simulation experimental results confirms that our proposed methods are feasible.

Figure 5.

Variations in the absolute errors of the intrinsic parameters with different calibration methods under varying levels of Gaussian noise.

We compared the runtimes of the four methods using MATLAB 2016a implementations of all algorithms on a computer equipped with a Pentium IV processor running at 2.1 GHz and 2.0 GB of RAM. We performed 1000 independent trials to obtain the average values of the runtimes, the results of which are listed in Table 1. The average value of the runtimes of Wu et al. ¹⁵ is far greater than the other methods because the generalized inverses of antipodal image points needed to be computed before establishing the linear constraints between the antipodal image points and the IAC.

Table 1.

Runtimes for the four algorithms (unit: seconds).

	Ours 1	Ours 2	Barreto and Araujo 14	Wu et al. 15
Runtime	0.022	0.024	0.025	0.032

Experiment with real data

In the real experiment, a checkerboard was used to calibrate the model to replace the lines. A catadioptric camera shot the scene from different viewing angles, and three pictures were selected as real images. Three experimental images with resolutions of 3680 × 3488 pixels are shown in Figure 6(a) to (c). First, the edges were detected using the Canny algorithm to detect the edges of the images, as shown in Figure 7(a) to (c). The equations of the line images were obtained using the method given in proposition 5 (step 1 of both proposed algorithms). Then, the intrinsic camera parameters were obtained on the basis of our calibration algorithms and the methods of Barreto and Araujo ¹⁴ and Wu et al. ¹⁵ The calibration results are listed in Table 2 for the real experiments. Note that the calibration results exhibit little difference; thus, our calibration methods are efficient. To further check the accuracy of the calibration results in Table 2, Figure 6(a) is taken as an example and rectified to its projective image using the determined intrinsic parameters, as shown in Figure 8(a) to (d). Again, the results indicate that our proposed calibration algorithms are effective.

Figure 6.

Three checkerboard images captured by a paracatadioptric camera.

Figure 7.

Resulting edges of the three images extracted in Figure 6.

Table 2.

Comparison of the calibration results of the intrinsic parameters with different methods (unit: pixels).

	fu	fv	s	u 0	v 0
Ours 1	1332.42	1302.76	2.13	1842.82	1744.12
Ours 2	1335.59	1305.86	2.47	1843.56	1742.62
Barreto and Araujo 14	1384.64	1381.49	3.01	1838.92	1740.60
Wu et al. 15	1347.90	1346.85	2.71	1843.25	1747.23

Figure 8.

Image results rectified using the intrinsic parameters listed in Table 2 for (a) Ours 1, (b) Ours 2, (c) Barreto and Araujo, ¹⁴ and (d) Wu et al. ¹⁵

In order to validate the accuracy of the calibration results in Table 2, the checkerboard pattern in Figure 6 was reconstructed. A valid region of the checkerboard pattern images from Figure 6(a) to (c) was chosen. On the basis of the Harris corner detector, ²⁷ the feature points of the checkerboard shown in Figure 9(a) to (c) were selected. Applying the epipolar geometry constraint algorithms for two views of the paracatadioptric camera, ²⁸ 52 corresponding points were selected from the two visible sides of the checkerboard image and were then used for reconstruction (see Appendix 1). The reconstruction results obtained on the basis of the data listed in Table 2 are shown in Figure 10(a) to (d). From Figure 10, it is clear that the reconstructed 3-D points are coplanar for the two sides.

Figure 9.

(a) to (c) Results of Harris corner detection for Figure 7(a) to (c), respectively.

Figure 10.

Reconstruction results obtained using the intrinsic parameters in Table 2 for (a) Ours 1, (b) Ours 2, (c) Barreto and Araujo, ¹⁴ and (d) Wu et al. ¹⁵

The parallelism and orthogonality in Figure 10(a) to (d) were also checked. First, according to least-squares fittings, the equations of the lines for every row or column in Figure 10(a) to (d) were obtained. Then, the average of the angles between any two points in parallel directions was calculated. Similarly, the average of the angle for the orthogonal direction was calculated. The results for the angles obtained from Figure 10 are summarized in Table 3. The real angle for the parallel line direction is 0°, and that for the orthogonal direction is 90° on the calibration checkerboard pattern. The 3-D reconstruction angles are similar to the true values in the parallel or orthogonal direction. This indirectly proves that the calibration results of the intrinsic parameters in Table 2 are effective.

Table 3.

Angles of the parallel and orthogonal directions recovered from Figure 10 (units: degrees).

	Ours 1	Ours 2	Barreto and Araujo 14	Wu et al. 15
Parallel	1.0032	0.9528	1.0120	1.0302
Orthogonal	88.8192	89.0063	88.3103	88.5824

Conclusion

A camera calibration algorithm is often related to a vanishing point or an image of circular points. In this work, for application to three images containing one line, two calibration methods based on the properties of the polar line of an infinity point with respect to a circle in a paracatadioptric camera and incorporating the orthogonal vanishing points or the images of the circular points were proposed. In this approach, for the unit sphere model, a straight line is projected to a circle on the unit sphere. The tangents of the circle at antipodal point pairs are parallel and intersect at an infinity point. Then, an infinity line can be obtained according to two groups of antipodal points. The direction of the polar line of an infinity point with respect to a circle and the direction of the infinity point are orthogonal. Further, the intersection point of the infinity line and the circle is a circular point. According to the projective invariance, the images of the circular points and the orthogonal vanishing points can be determined. The camera intrinsic parameters can then be obtained according to the relationship between the images of the circular points or orthogonal vanishing points and the intrinsic parameters. The experimental results verify the robustness of our proposed calibration methods. An optimization method based on the fitting of one line image according to the relationship between antipodal points was also proposed.

In the literature, ¹⁴ on the basis of the geometric invariance under a projective transformation, one image containing three or more lines is used to calibrate the camera. A pair of lines is projected into two circles and intersect at a pair of antipodal points on the viewing spherical surface. Three lines are projected into three circles, which intersect at three pairs of antipodal points, and the intersection point of the three lines passing though the antipodal points is the center of the three circles. The polar line of the center of the circle with respect to the circle is the infinity line, which intersects the circle at the circular points. On the basis of the geometric invariance under a projective transformation, the three lines passing through the intersection points corresponding to the line images intersect at the principle point, that is, the image of the center of the circle. The polar lines of the principal point with respect to the line image intersect the line image at the image of the circular points. The images of the circular points are used to determine the intrinsic parameters. In the literature, ¹⁵ a relationship between the projection on the viewing sphere of a space point and its image was derived to obtain a linear constraint between the antipodal image points and the IAC. First, the principal point was determined by the intersecting point of the three line images. Then, two pairs of antipodal image points were taken from each of the three line images according to the constraint, and central catadioptric cameras were calibrated for a single view.

In this study, the elements at infinity are obtained on the basis of both the properties of the tangent lines of a circle and the properties of the polar line of an infinity point with respect to a circle.