Automated quality inspection of camera zooming with real-time vision

Abstract

Industrial automated production technologies have been the research focus of many recent studies, comprising the two research streams of automated assembly and automated product testing. Camera lens-shake detection is an effective way to measure the quality of video cameras during zooming. Conventional testing methods involve time-consuming manual operation procedures. This study proposes a novel automated camera lens-shake detection method, in which real-time visual tracking of two arbitrary features is used to measure and analyze camera zooming quality. The camera lens-shake detection approach can be used to screen out video cameras for the purpose of quality control. It can be effectively employed to replace conventional testing methods and enhance efficiency and stability of product manufacturing.

Introduction

Among various automated production and manufacturing technologies, industrial cameras have been widely used in measurement and assembly tasks. Castellani et al. ¹ proposed a new non-contact and non-destructive method for automatic inspection of microstructures by employing a microscopic camera to monitor the vibration of the structure. Ogun et al. ² presented a two-stage inspection process in which a machine vision system is used to detect potentially defective regions on parts over a much wider field of view than is currently covered by commercial products. Wen et al. ³ developed a computer-vision-based object recognition system to deal with online inspection for hose assembly. Features of hoses and related fasteners were analyzed to perform image processing, object extraction, and feature recognition. Wang et al. ⁴ developed an alternative measurement sensor for onsite inspection of hull plates, providing accurate quantitative results with high efficiency. Chen and Chang ⁵ incorporated a linear regression diagnostic model with digital image processing techniques to inspect TFT-LCD Mura defects automatically. Product manufacturing with an increasingly refined process has required higher image resolution and greater functional stability in industrial vision and measurement sensors.

In recent years, many researchers have studied the phenomenon of camera shake. Fergus et al. ⁶ determined the image blurring degree through the distribution of local image convolution. Hirsch et al. ⁷ developed a forward model based on an efficient filter flow framework, which can remove blurred images caused by camera shake efficiently. Tong et al. ⁸ presented a new blur detection scheme based on edge type and sharpness analysis using the Haar wavelet transform. This scheme can determine whether an image is blurred or not and to what extent an image is blurred caused by camera shake. Ogino et al. ⁹ used a camera to observe the designed test pattern and estimate the degree of camera shake for roll, pitch, and yaw motion of the camera. Oh et al. ¹⁰ used changes in image frequency spectrum and the contour orientation of the image spectrum to measure camera shake. Jodoin et al. ¹¹ separated moving objects by background segmentation, and used the motion vector of the object to evaluate camera shake. Nishi and colleagues ^12,13 processed the images captured by a camera with different postures. Hu et al. ¹⁴ and Yue et al. ¹⁵ both improved image blur caused by camera shake based on depth information of an image. Baxansky and Tzur ¹⁶ designed a new algorithm to analyze the degree of contour blurring of a single image and other related techniques to measure and correct camera shake caused by camera motion during regular camera shooting. All of these research results focused on the detection of camera shake caused by external motion. However, there has been no related research aimed at measuring the camera lens-shake caused by the interior camera lens during zooming. Currently, all cameras with this defect must be filtered and eliminated at a time-consuming quality control stage with human eye inspection.

Camera lens-shake detection is primarily focused on the measurement of manufacturing tolerances exceeding the unacceptable range of zoom lenses in terms of the image shift during the zooming process. Thus, the zooming performance of each manufactured video camera can be ensured at the quality control stage. In fact, current production lines mainly adopt manual inspection. The level of camera lens-shake near the principal point is estimated by a human operator only while the lens is zooming in or out. This detection method not only consumes a lot of time but also gives poor accuracy.

The method proposed in this article can estimate the position of the principal point of an image during the zooming process by means of tracking two arbitrary feature points. Most importantly, through the shake level of the two feature points, the camera lens-shake around the principal point can be effectively calculated during the zooming process.

Problem formulation

The camera zoom function mainly involves zooming in or out images by changing the focal length between the zoom lens and the image surface using rotating internal gears inside the camera. For a camera with ideal lens assembly, the image in the camera will be zoomed in and out stably and smoothly. However, when the zooming is performed by a lens with poor manufacturing and assembly, camera lens shaking occurs during the zooming process. The phenomenon leading to shaking images is called camera lens-shake. The geometry of the camera lens-shake occurring when zooming an image is shown in Figure 1, in which one can see the varying camera frame $\{C\}$ caused by lens shaking during zooming.

Figure 1.

Camera lens-shake.

Existing camera lens-shake inspection approach

The traditional camera lens-shake inspection process is as follows.

1. Estimate the position of the principal point on the image with a typical offline calibration procedure for cameras.^17–19

2. Overlap and align the positions of the selected feature point and the principal point on the image.

3. Start the zoom function, estimate the shaking level on the image, and define the maximum value of this shaking level as the camera lens-shake s on this particular camera manually, as shown in Figure 2.

Figure 2.

Camera lens-shake detection.

Proposed camera lens-shake measurement approach

Compared with the traditional camera lens-shake detection technique, one does not need to perform any of the steps mentioned earlier, such as calculating the position of the principal point offline, aligning the selected feature point with the position of the principal point, or processing the measurement of camera lens-shake manually. The proposed approach can largely reduce the detection time, and avoid accumulated errors due to miscalibration. The steps of the proposed automated camera lens-shake detection process are as follows.

1. Place the video camera to be tested in front of a $4\times 4$ chessboard pattern board, so that the pattern can be seen during zooming. The suggested system configuration is shown in Figure 3. Choose two corner points as the feature points so that their image trajectories during zooming will not be collinear. Otherwise, the two corner points and the principal point will lie on the same line. Track the two preselected corner features visually in real time during the zooming process.

2. Start the zooming function of the video camera and pause at each predetermined zooming level. Extract the image coordinates of the two corner points, a and b, and then add their coordinates to the corresponding point group matrix, ${\mathit{P}}_a$ and ${\mathit{P}}_b$ , at each zooming level

   $${\mathit{P}}_j=\{p_{j_i}\}$$

   (1)

   where

   $$p_{j_i}=\left[\begin{array}{c}\hfill x_{j_i}\hfill \\ \multicolumn{1}{c}{y_{j_i}}\end{array}\right]$$

   (2)

   and i represents the zooming level of a camera, $i=0,1,2,\dots ,n$ , and j represents the index of the feature point, $j=a$ or b.

3. Using the method of least squares, one can fit the equations of the line with the data points in the two point groups ${\mathit{P}}_a$ and ${\mathit{P}}_b$ in equation (1), respectively. Consider all data points in each of the two point group data matrixes, ${\mathit{P}}_a$ and ${\mathit{P}}_b$ . To fit the equation of the line $L_j$ for the trajectory of feature point j during zooming

   $$L_j:y_{j_i}=m_j{x}_{j_i}+b_j$$

   (3)

    one can determine $m_j$ and $b_j$ with the least squares method. Thus

   $$\mathit{A}\left[\begin{array}{c}\hfill m_j\hfill \\ \multicolumn{1}{c}{b_j}\end{array}\right]=\mathit{B}$$

   (4)

    where

   $$\begin{array}{cc}\hfill \mathit{A}\hfill & \hfill =\left[\begin{array}{cc}\hfill x_{j_1}\hfill & \hfill 1\hfill \\ \multicolumn{1}{c}{x_{j_2}}& \hfill 1\hfill \\ \multicolumn{1}{c}{⋮}& \hfill ⋮\hfill \\ \multicolumn{1}{c}{x_{j_n}}& \hfill 1\hfill \end{array}\right]\hfill \end{array}$$

   (5)
   $$\begin{array}{cc}\multicolumn{1}{c}{\mathit{B}}& \hfill =\left[\begin{array}{c}\hfill y_{j_1}\hfill \\ \multicolumn{1}{c}{y_{j_2}}\\ \multicolumn{1}{c}{⋮}\\ \multicolumn{1}{c}{y_{j_n}}\end{array}\right]\hfill \end{array}$$

   (6)

   In particular, the least squares estimate of $m_j$ and $b_j$ can be computed as

   $$\left[\begin{array}{c}\hfill m_j\hfill \\ \multicolumn{1}{c}{b_j}\end{array}\right]={\mathit{A}}^{+}\mathit{B}$$

   (7)

   where ${\mathit{A}}^{+}$ denotes the left pseudo-inverse of $\mathit{A}$

   $${\mathit{A}}^{+}=({\mathit{A}}^{\mathrm{T}}\mathit{A}{)}^{-1}{\mathit{A}}^{\mathrm{T}}$$

   (8)

4. With the intersecting point of $L_a$ and $L_b$ , one can determine the position of the principal point $o_c$ on the image. The included angle $\phi$ for vectors $\overrightarrow{o_{c}a}$ and $\overrightarrow{o_{c}b}$ can thus be computed as

   $$\phi ={\cos }^{-1}\left(\frac{\overrightarrow{o_{c}a}·\overrightarrow{o_{c}b}}{\Vert \overrightarrow{o_{c}a}\Vert \Vert \overrightarrow{o_{c}b}\Vert }\right)$$

   (9)

5. Based on the relative geometric relation of the three points $a,b,o_c$ on the image and the distances $s_a$ and $s_b$ , referred to as normal shakes for feature points a and b, perpendicular to the ideal trajectories, one can calculate the level of camera lens-shake at the principal point as

   $$s=h_{\phi }(s_a,s_b)$$

   (10)

   where $h_{\phi }$ is to be defined in the next section.

Figure 3.

System configuration.

Calculation of camera lens-shake level

The fact that camera lens-shake s is caused by a shaking lens in the zooming process leads to shaken images. Therefore, the magnitude and direction of the captured feature points and camera lens-shake are considered the same. It can be expressed using the following formula; the shaking level of a feature point on the image is illustrated in Figure 4

$$\overline{o_c{o}_c^{\prime }}=\overline{a{a}^{\prime }}=\overline{b{b}^{\prime }}=s$$

(11)

Figure 4.

Camera lens-shake estimation via the trajectories of two feature points.

At this moment, the included angle between the line $L_a$ for the trajectory of the feature point and the line segment $\overline{a{a}^{\prime }}$ . giving the direction of camera lens-shake is defined as $\theta$ . The normal shake $s_a$ of the camera measured at feature a is related to actual camera lens-shake s and $\theta$ as

$$\sin \theta =\frac{s_a}{s}$$

(12)

$$\cos \theta =\frac{\sqrt{s^2-s_a^2}}{s}$$

(13)

Similarly, the normal shake $s_b$ of the camera measured at feature b is related to actual camera lens-shake s and $\theta$ by $\phi$ by

$$s·\sin (\pi -(\phi -\theta ))=s_b$$

(14)

$$s·\sin (\phi -\theta )=s_b$$

(15)

One can thus see that

$$s·(\sin \phi ·\cos \theta -\cos \phi ·\sin \theta )=s_b$$

(16)

After substituting equations (12) and (13) into equation (16), it can be sorted out as

$$s·(\sin \phi ·\frac{\sqrt{s^2-s_a^2}}{s}-\cos \phi ·\frac{s_a}{s})=s_b$$

(17)

It follows from equation (17) that

$$\sin \phi ·\sqrt{s^2-s_a^2}-\cos \phi ·s_a=s_b$$

(18)

Re-arrange equation (18) and one can see that

$$s^2={\left(\frac{s_b+\cos \phi ·s_a}{\sin \phi }\right)}^2+s_a^2$$

(19)

Based on all available data, including $s_a$ , $s_b$ , $\phi$ , $L_a$ , $L_b$ , ${\mathit{P}}_a$ , and ${\mathit{P}}_b$ , one can thus determine the level of lens-shake s uniquely from the function $h_{\phi }(s_a,s_b)$ as

$$s=h_{\phi }(s_a,s_b)=\sqrt{{\left(\frac{s_b+\cos \phi ·s_a}{\sin \phi }\right)}^2+s_a^2}$$

(20)

Singularity avoidance

When $\phi ={180}^{\circ }$ , the two feature points a and b are collinear with the principal point. We cannot identify the level of camera lens-shake through equation (20), owing to singularity. A simple and efficient way to avoid the situation is to select two corner points on the pattern board so that their image trajectories during zooming are not collinear. To do this, one can select two such corner points by observing their trajectories during a trial zooming. Another way to resolve the problem during the detection process is to record trajectories of three arbitrary feature points and select the two most suitable trajectories to avoid $\phi$ being ${180}^{\circ }$ before computing the level of camera lens-shake. That is, during the detection process, trajectories of three feature points are recorded. The two best trajectories are selected to resolve the issue effectively.

Experimental results

In the experimental system, two cameras VC-B20D manufactured by Lumens Digital Optics Incorporation were tested for quality inspection. The two cameras are labeled $C_a$ and $C_b$ and have full high-definition resolution $1920\times 1080$ . The required zooming accuracy certainly depends on the application tasks that cameras are employed to handle. For the pan-tilt-zoom (PTZ) cameras that the company manufactures, industrial-level quality standards are required. A typical camera lens-shake value for quality control is 2 mm. Figures 5 and 6 illustrate the visual tracking results of two features during zooming for a passed camera and a failed camera, respectively.

Figure 5.

Image tracking results of a passed camera.

Figure 6.

Image tracking results of a failed camera.

For camera $C_a$ , the level of camera lens-shake detected in the traditional way is about 1 mm, and the position of the principal point measured by Zhang ¹⁸ and Sarkis et al. ¹⁹ is (992,498) in the pixel coordinates. For camera $C_b$ , the level of camera lens-shake detected in the traditional way is larger than 2 mm, and the position of the principal point measured by Zhang ¹⁸ and Sarkis et al. ¹⁹ is (924,541) in the pixel coordinates. With the method introduced in this article, we inspected the two cameras six times each, as listed in Tables 1 and 2; we can see that the point of intersection constructed by the method of least squares effectively locates the principal point of the camera, and the error is $<2$ pixels. Through normal shakes $s_a$ and $s_b$ measured by the included angle $\phi$ of three points $a,b,o_c$ and equation (20), the camera lens-shake s can be effectively estimated. Each detection process only requires the camera to complete a single direction of zooming with real-time visual tracking and the time taken by the detection process depends on the hardware performance of the camera. At each zoom level, the time required to precisely extract the two preselected corner features on the chessboard-type pattern board is less than 0.16 s. However, it takes time to reach the next zoom level before processing the next image. That is, the total detection time is mostly accounted for by the time for the zooming motion. A comparison of the detection times is given in Table 3. It appears that the proposed system runs at least five times faster than the traditional approach.

Table 1.

Camera lens-shake estimates of camera $C_a$ , manually identified as 1 mm.

	sa (mm)	sb (mm)	ϕ	oc (pixels)	Camera lens-shake (mm)	Pass or fail
1	1.067	0.207	105.477°	(991.680, 498.428)	1.070	Pass
2	0.895	0.058	105.630°	(991.669, 498.318)	0.912	Pass
3	0.978	0.200	105.508°	(991.825, 498.687)	0.982	Pass
4	0.976	0.249	105.203°	(992.990, 498.534)	0.976	Pass
5	1.052	0.224	105.835°	(992.243, 498.464)	1.054	Pass
6	1.061	0.358	105.183°	(993.224, 498.213)	1.065	Pass
Mean	1.004	0.216	105.639°	(992.271, 498.440)	1.009	Pass

Table 2.

Camera lens-shake estimates of camera $C_b$ , manually identified as 2 mm.

	sa (mm)	sb (mm)	ϕ	oc (pixels)	Camera lens-shake (mm)	Pass or fail
1	1.672	0.061	138.951°	(922.773, 541.823)	2.476	Fail
2	2.133	0.295	140.838°	(922.744, 543.337)	3.030	Fail
3	1.752	2.494	142.103°	(923.003, 543.901)	2.518	Fail
4	2.265	0.708	138.986°	(923.848, 542.015)	2.730	Fail
5	1.815	0.052	142.497°	(921.060, 543.761)	2.914	Fail
6	2.131	0.115	140.263°	(921.863, 542.421)	3.198	Fail
Mean	1.961	0.620	140.606°	(922.548, 542.876)	2.811	Fail

Table 3.

Execution time comparison.

	Traditional manual detection	Automated camera lens-shake detection
Time	Up to 780 s	Up to 140 s

Conclusions

A novel approach to measuring the camera lens-shake caused by unacceptable manufacturing tolerance of the zoom lenses is proposed in this research. The detection method can be implemented to replace the existing human inspection approach by real-time visual tracking of two arbitrary features. Such two features can be selected as the corner points on a chessboard-type pattern board to ensure robust and precise measurements. To avoid computation singularity effectively, the image trajectories of the two corner points must not be collinear. This can be resolved by either preselecting two cornet points in a trial zoom or selecting the two most suitable trajectories online from three trajectories measured. The proposed approach has been validated by experimenting with industrial cameras to demonstrate its effectiveness and efficiency. According to the experimental results, this detection approach appears to be stable, accurate, and easy to carry out.