Adaptive learning method for multiple-object detection in manufacturing

Abstract

In manufacturing, vision-based inspections are effective nondestructive methods for implementing object discrimination. To minimize inspection times, multiple-object discrimination must be implemented for one inspection image. However, multiple-object inspections are difficult to perform in manufacturing because of the lack of distinct objects in inspection images. In addition, inspectors might fail to use multiple sensing devices when concurrently detecting objects. This article proposes a novel multiple-object sensing system that incorporates a local-adaptive-region-growing–based learning method for adaptively segmenting multiple-camera images for multiple-object discrimination. The proposed local-adaptive-region-growing method and support vector machine-based discrete wavelet transform can effectively classify multiple objects in the local subregions of inspection images. The proposed system bridges the gap between sensing devices and inspectors for solving problems encountered when concurrently using multiple sensing devices. The learning method yielded more favorable results than existing inspection methods have. Furthermore, a standard test series was developed for quantitatively comparing inspection methods.

Introduction

Object discrimination techniques have increasingly been included in industrial inspection applications. Vision-based inspection systems have been employed as effective nondestructive inspection methods. Minimizing conventional inspection times requires implementing a learning technique for discriminating among multiple objects in a single inspection image. However, inspecting multiple objects is difficult in manufacturing because of the lack of distinct objects in inspection images. Furthermore, inspectors might fail to use multiple sensing devices for concurrently detecting objects. Therefore, employing an effective inspection system that incorporates an adaptive learning method for detecting multiple objects is essential for solving the above problems. Automatic or adaptive region growing ¹ is one of several current adaptive image segmentation techniques. The motivation of the work is that an adaptive learning method is proposed as a solution to effectively detect multiple objects for one inspection image. The proposed learning method involves using local-adaptive-region-growing (LARG) image segmentation combined with a support vector machine (SVM)–based discrete wavelet transform (DWT) to implement multiple-object discrimination.

The contributions of the work are summarized as follows: The LARG method can adaptively segment multiple-camera images and effectively classify multiple objects in the local subregions of an inspection image. The proposed multiple-object sensing system can solve problems associated with using sensing devices and bridge the gap between sensing devices and inspectors. Finally, the proposed method can be used as an inspection tool for automatically selecting the most suitable parameters for inspecting multiple objects.

Related work

This section describes the existing vision-based inspection methods (including SVM-based inspection, multiple-object segmentation, and region-growing recognition) related to the work and finally addresses the differences between this study and the existing methods.

Recent studies have investigated industry inspection methods on the basis of vision-based techniques. For SVM-based inspections, He et al. ² employed an automated defect-classification method for detecting defects in aircraft structures, in which an SVM was used to build a classifier model and predict the type of defect. However, the classification and prediction was based on artificial defects. Nashat et al. ³ proposed a machine vision inspection system for detecting small cracks in baked products. Detection was based on an SVM featuring selection criteria focused on detecting cracks that formed an approximate line. For monitoring production processes, a weighted support vector machine (WSVM) was proposed for controlling chart pattern recognition. ⁴ The authors of that study tested the SVM and WSVM for several normal and abnormal classification problems and found that the WSVM outperformed the SVM in various fault scenarios. However, the sensitivity and classification accuracy of the WSVM was lower in detecting rare abnormal patterns. Furthermore, SVMs have been applied in a range of electrophysiological signal processing tasks. Lajnef et al. ⁵ proposed a decision tree–based multiclass SVM classification system for implementing and evaluating an automatic sleep staging framework. The SVM-based method was a visible alternative to existing automatic methods for sleep-stage classification. In an SVM-based industrial inspection system requiring multiple-object discrimination, the purpose of the SVM method is not only to accurately classify objects in multiple inspection images but also to effectively perform multiple-object discrimination in a single inspection image. Multiple-object segmentation and extraction play critical roles in this process.

Multiple-object segmentation and extraction have been studied in recent years. Evaluations and comparisons of object boundary-detection techniques can be found in the literature. ⁶ Among the boundary-extraction-based segmentation techniques in the literature, the deformable model of active contours is one of the most efficient techniques for segmenting two-dimensional (2D) and three-dimensional (3D) images. Two categorized approaches that incorporate the deformable model are the edge-based and region-based approaches. Edge-based approaches are so titled because the information they use for plotting curves to the edges is strictly along the boundary. ⁷ Thus, a strong edge must be detected to drive the curve. Edge-based approaches can detect a strong edge for driving curves, which evolves to match the contour of an object in an image. However, these approaches fail in the presence of noise. ⁸ Region-based approaches are highly adapted to situations in which extracting boundaries from the inspection target is difficult. However, such approaches are computationally intensive. ⁹ For multiple-object segmentation and extraction, this article proposes a LARG segmentation and DWT-based feature extraction method as a solution.

The literature addresses various region-growing recognition methods. One method proposed in the literature ¹⁰ involves automatically obtaining the homogeneity criterion from regional characteristics for segmenting medical images. This method is based on a model for describing the homogeneity and simple shape properties of a region. However, errors occur when the assumptions of the formulated model regarding homogeneity and region shape are inaccurate. A novel region-growing method ¹¹ based on local cube tracking was employed to segment a small local cube in order to detect a vessel segment, and the resultant local cube was determined on the basis of the segmentation results. The procedure was repeated until the segmentation was complete. In previous studies^10,11 and this study, conducting region-growing procedures typically produced highly defined object boundaries for subsequent region-oriented segmentation. Automatic segmentation has been a fundamental problem in studies because any inaccuracy in this procedure can distort the segmentation results, particularly when segmenting multiple objects.

Based on previous studies, the existing methods including the SVM and the region-growing recognition are similar to those used in the work. However, the study employed the SVM-based method combined with multiple-object segmentation and extraction for effectively performing multiple-object discrimination in a single inspection image. This LARG strategy enables the segmenting of multiple-object images without computing the assessment function for every pixel added to the local region, and no bimodal histogram is required. Furthermore, the study involved using the proposed learning method, which comprises LARG segmentation and SVM combined with a DWT-based feature extraction method for automatically inspecting multiple sensed objects. Finally, this study proposed a standard test series for quantitatively comparing inspection methods in the manufacturing field.

The remainders of this article are organized as follows: section “Proposed method” presents the proposed method for automatically inspecting multiple sensed objects. Section “Experimental results and discussion” presents the experimental results by applying the proposed method to various samples, as well as a comparison of various inspection methods. The final section offers the conclusions of this study.

Proposed method

This section describes the proposed LARG method including LARG learning algorithm and SVM learning classifier and then introduces the LARG-based learning approach for the automated inspection of multiple sensed objects.

LARG learning algorithm

The proposed LARG-based learning algorithm involves first using a set of initial seeds in the local growth regions {R_0,0, R_0,1, R_1,0, R_1,1}, each of which comprises several local subregions. Within these local subregions, each initial seed is considered as being surrounded by a group of pixels if the following selection criterion is satisfied

$$1-T_L^i\le z_L^j\le 1$$

(1)

where $z_L^j$ is the normalized gray level of the jth pixel in one of the local subregions; L = (0, 0, 0, 1, 1, 0, 1, 1) represents the index of {R_0,0, R_0,1, R_1,0, R_1,1}; and $T_L^i$ , which is the adjustable threshold in the local subregions, depends on the SVM results. For each local growth region, inclusion in a local subregion requires each pixel to be eight connected to at least one pixel in the same subregion. Local subregions can be merged when pixels are connected to multiple subregions.

This article proposes a learning method for determining the threshold $T_L^i$ . Regarding the inspection image, the growth region of which is divided equally into four local subregions (Figure 1), and the threshold $T_L^i$ in the local subregions can be determined on the basis of SVM results using the iteration method (Figure 2). The k iteration of the proposed algorithm is performed from the initial input values of $T_L^0$ until the stopping condition is satisfied (e.g. k = 99 satisfies the stopping condition). The thresholds $T_{0,0}^0$ , $T_{0,1}^0$ , $T_{1,0}^1$ , and $T_{1,1}^0$ are then determined for the local subregions R_0,0, R_0,1, R_1,0, and R_1,1, respectively. This approach can be summarized using the following procedure: (1) input the initial values of $T_L^0$ ; (2) execute the k iteration of the proposed algorithm; (3) determine whether the k iteration satisfies the stopping condition; and (4) record the thresholds of $T_L^i$ for the local subregions of {R_0,0, R_0,1, R_1,0, R_1,1}.

Figure 1.

Local subregions of {R_0,0, R_0,1, R_1,0, R_1,1} and the corresponding thresholds $T_L^j$ as well as the location of each class in an inspection image.

Figure 2.

k iteration of the proposed algorithm was performed from the initial input values of { $T_{0,0}^0$ , $T_{0,1}^0$ , $T_{1,0}^0$ , $T_{1,1}^0$ } ( $T_L^0$ ) until the stopping condition was satisfied. For k = 1, 98, and 99, the input values of $T_L^i$ are { $T_{0,0}^0$ , $T_{0,1}^0$ , $T_{1,0}^0$ , $T_{1,1}^1$ }, { $T_{0,0}^0$ , $T_{0,1}^0$ , $T_{1,0}^0$ , $T_{1,1}^{98}$ }, and { $T_{0,0}^0$ , $T_{0,1}^0$ , $T_{1,0}^1$ , $T_{1,1}^0$ }, respectively.

Learning classifier

An SVM was employed to implement the learning classification. SVM is designed to construct a hyperplane as the decision surface to maximize the margin of separation between positive and negative examples. ¹² Consider the set of feature values belonging to two separate classes $(x_1,y_1),\dots ,(x_N,y_N)$ with input $x_i\in R^N$ (N-dimensional input space) and class labels (target output) $y_i\in \{1,-1\}$ . To optimize the SVM classifier boundary, the relationship should be maximized as follows

$$Q\left(\alpha \right)=\sum _{i=1}^N{\alpha }_i-\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N{\alpha }_i{\alpha }_j{y}_i{y}_{j}K(x_i,x_j),0\le {\alpha }_i\le C$$

(2)

while

$$\begin{array}{c}\multicolumn{1}{c}{\sum _{i=1}^N{\alpha }_i{y}_i=0,w}\\ \multicolumn{1}{c}{=\sum _{i=1}^N{\alpha }_i{y}_i{x}_i,{\alpha }_i[y_i(w^T{x}_i+b)-1+{\xi }_i]=0}\end{array}$$

(3)

where C is a user-specified positive parameter for controlling the tradeoff between SVM complexity and the number of nonseparable points; N is the sample size; α is the Lagrange multipliers; b denotes the bias; and $K(x_i,x_j)$ is the SVM kernel, which is the Gaussian radial basis function (RBF). The form of the Gaussian RBF is given as

$$K(x_i,x_j)=\exp (-\gamma \times \Vert x_i-x_j^2\Vert )$$

(4)

where γ is the RBF kernel parameter. The stack ${\xi }_i$ is employed to relax the constraints of the canonical hyperplane equation. When a linear boundary is inappropriate, the SVM can map the input vector into a high-dimensional feature space to improve separation capability.

Table 1 displays the class labels of the samples, and Figure 1 provides the location of each class in an inspection image. The DWT-based features extracted from each image are used as inputs for datasets. The relationship among the DWT coefficients can be expressed as follows

$$A_0=A_j+\sum _{j=1}^j{D}_j$$

(5)

Table 1.

Class labels of the samples used in the experiments.

Class labels	Samples	Class labels	Samples
A	Degrees of ±0.5°	E	Degrees of ±0.0°
B	Degrees of ±2.0°	F	Degrees of ±1.5°
C	Degrees of ±3.5°	G	Degrees of ±3.0°
D	Degrees of ±5.0°	H	Degrees of ±4.5°

where A_j and D_j are approximation and detail coefficients at j-level decomposition, respectively. The procedure of DWT decomposition is described in the literature. ¹³ Each class in one inspection image comprised 280 samples. The SVM was trained and tested on the basis of the datasets. A total of 80 sets of data were randomly selected as training samples, and the others were used as validation samples for evaluating the classifier. Table 2 lists the classification results for samples of varying size for each class. As presented in the table, the sample sizes, such as 280, 560, and 800 samples for each class, appear to be unrelated to the experimental classification results.

Table 2.

Learning classification results from various sample sizes for each class.

Sample size	Training	Validation (%)	Accuracy rates
120	20	100	82
140	40	100	83
160	60	100	89
280	80	200	95
560	160	400	95
800	200	600	95

For SVM classifiers, two parameters (the parameter C and the RBF kernel parameter γ) must be optimized. This study adopted the hold-out procedure to determine the two parameters for the SVM classifier. The hold-out procedure involves dividing the samples into two parts: a training sample for training classifiers and a verification sample for testing the accuracy of classifiers. ¹⁴ Table 3 displays the testing accuracy of the various combinations of the two parameters; C = 2¹¹ and γ = 2⁻⁷ were identified by the SVM because of the high testing accuracy.

Table 3.

Testing accuracy for various combinations of the parameter C and RBF kernel parameter γ.

C	γ
	−13	−11	−9	−7	−5	−3	−1	1
3	89	90	91	91	92	91	90	89
5	90	91	92	93	92	91	91	90
7	91	92	93	94	93	92	92	91
9	91	92	93	95	95	93	93	92
11	92	94	95	95	95	94	93	91
13	91	93	95	95	94	94	94	92
15	90	93	94	94	93	94	93	91
17	89	91	93	92	92	93	92	90

RBF: radial basis function.

C (rows): value (3, 5,…, 17) of log₂ C; γ (columns): value (−13, −11,…, 1) of log₂ γ.

Adaptive learning approach for the automated inspection of multiple sensed objects

This section describes the development of the proposed multiple-object sensing system and learning approach. As an example of an industrial inspection, the learning method was applied to inspect multiple eyeglass lenses. The technology used in the manufacturing process of eyeglass lenses determines the degree of curvature of the lenses. This study further modified the conventional inspection process in manufacturing to effectively classify multiple eyeglass lenses in a single image. Inspecting multiple eyeglass lenses is a difficult task in manufacturing because the inspector could fail to concurrently focus the multiple sensing telescopes on the target panels. Therefore, developing a multiple-object sensing system is crucial to bridging the gap between the sensing telescopes and inspector.

In optical multiple-object inspections, synchronous image information between the sensing devices and inspector is critical in evaluating the reliability and stability of detection processes. Figure 3 depicts the configuration of the proposed inspection system. To obtain the information, a camera array (four cameras), which is positioned after a telescope array (four sensing telescopes) and triggered by an acquisition device, acquires synchronous images of multiple objects. The synchronous image information is transmitted from the camera array to an industrial computer equipped with a frame grabber for capturing a single image composed of the synchronous images.

Figure 3.

Configuration of the proposed inspection system.

As illustrated in Figure 3, a standard inspection process for multiple eyeglass lenses was developed and conducted as follows: The target panel, which included multiple targets (four targets), was installed on a platform. The multiple targets indicated the degree of curvature of multiple eyeglass lenses (four eyeglass lenses), which were mounted on the support frame of the telescope array. The distance between the target panel and telescope array was 10.67 m. During the inspection, multiple eyeglass lenses with unknown degrees of curvature were mounted on the telescope array, and the surface light from the platform illuminated the target panel. Conventionally, telescope lenses are manually focused on the target panel. The proposed learning method involved processing telescopic images captured by multiple cameras, an approach that can rapidly determine the degrees of curvature of the multiple eyeglass lenses to solve the problem of simultaneously focusing the multiple telescopes.

Figure 4 displays a schematic diagram of the multiple-object sensing system. Compared with conventional vision-based inspections in manufacturing, the proposed system enables sensing and classifying multiple objects in addition to applying a suitable processing (segmentation and classification) strategy on the basis of the sensing results. The system procedure can be summarized as follows: (1) input the sample images from the image queue, (2) determine whether the image satisfies the condition for multiple-object processing, (3) execute the LARG-based algorithm (including LARG segmentation and DWT extraction), (4) perform SVM classification, and (5) confirm that no image remains in the image queue. As displayed in Table 1, for inspecting 200 classes {A, B, C, D} of sample images, $T_L^P$ and J_p were set to { $T_{0,0}^{77}$ , $T_{0,1}^{80}$ , $T_{1,0}^{82}$ , $T_{1,1}^{66}$ } and 6, respectively, where $T_L^P$ represents the optimal thresholds $T_L^j$ and J_p is the optimal j-level DWT for the class {A, B, C, D}. The input images were converted to 1024 × 768 pixel images with an 8-bit gray level. The 256 gray levels were normalized to within 0 and 1. Step 2 determines the image processing according to the following condition

$$N_b^P<N_b$$

(6)

where $N_b^P$ is the sum of the normalized gray-level value of 0 for a given image, and N_b is the total number of the normalized gray levels with a value of 0 summed from a randomly selected single-object image in another image sample. If the sample image satisfies the condition, then step 3 commences; otherwise, single-object processing is performed, which includes the region-growing algorithm and multiclassifier SVMs. The single-object processing procedure is described in the literature. ¹⁵ The entire procedure is completed when no images remain in the image queue. Thereafter, to inspect another class {E, F, G, H}, $T_L^P$ and J_p are reset for inspection, and this procedure is completed when the image queue is empty.

Figure 4.

Schematic diagram of the proposed multiple-object sensing system.

As provided in Figure 5, the proposed approach applies the following steps to automatically obtain $T_L^P$ and J_p for multiple-object inspection:

• Step 1: A total of 200 images are tested for a given j-level DWT, j = 1, 2,…, 7;

• Step 2: Given a value of 1 for initial seeds in the gray-level image, $T_L^i$ in equation (1) is determined using the iteration method (Figure 2);

• Step 3: The LARG-based learning algorithm is implemented;

• Step 4: Feature extraction is implemented on the basis of a j-level DWT;

• Step 5: Images are classified using the SVM;

• Step 6: The recognition rate is determined at each adjustable threshold $T_L^i$ for the given image.

Figure 5.

Flow diagram of the LARG-based algorithm.

The recognition rate is defined as follows

$$\mathrm{Accuracy}=\left(\frac{N_C}{N}\right)\times 100\%$$

(7)

where N_C is the number of accurately classified images during the test run, and N is the total number of test sets (N is 200 in this case). If the recognition rate exceeds a given value δ, then step 7 commences; otherwise, steps 2–6 are repeated.

• Step 7: The process stops when the sample images in all cases of the given j-level DWTs have been tested; otherwise, steps 1–6 are repeated. In addition, the algorithm stops when any $T_L^i$ fails to satisfy the condition in step 6.

For example, step 1 tests sample images with a six-level DWT. Step 2 determines the thresholds of $T_L^i$ through the iteration method. Step 3 implements the LARG-based algorithm. Step 4 extracts the DWT coefficient features. Step 5 then classifies the DWT coefficient features using the SVM. When the given value δ is 0.9 (i.e. 90% accuracy rate), the process repeats steps 2–6 until the thresholds of $T_{0,0}^{77}$ , $T_{0,1}^{80}$ , $T_{1,0}^{82}$ , and $T_{1,1}^{66}$ for the local subregions of {R_0,0, R_0,1, R_1,0, R_1,1} are determined. Step 7 then stops the process for the six-level DWT. In this test, the final thresholds are { $T_{0,0}^{77}$ , $T_{0,1}^{80}$ , $T_{1,0}^{82}$ , $T_{1,1}^{66}$ }. Thus, the algorithm obtains $T_L^P=\{T_{0,0}^0,T_{0,1}^0,T_{1,0}^1,T_{1,1}^0\}$ and J_p  = 6.

Experimental results and discussion

This section describes the classification results obtained using the proposed method and compares them with those reported in previous studies. The experiment was conducted to test the accuracy and performance of the proposed method.

The experiment setup detail is described as follows. As displayed in Figure 3, the experiment setup for the test included a target panel, telescope array, camera array, and industrial computer. The distance between the target panel and telescope array was 10.67 m. Data on the test lenses were listed in Table 1. The lenses were selected from 200 validation samples. During the inspection, the lenses with unknown degrees of curvature were mounted on the telescope array, and the surface light from the platform illuminated the target panel.

Significant results from each experiment include multiple-object classifications, effectiveness of the proposed system, and accuracy and performance of the proposed method. The classification results reveal that the proposed method can be used as a multiple-object inspection tool for automatically selecting the most suitable parameters for inspections. The proposed system, which incorporates the LARG method and SVM-based DWT, can effectively classify multiple objects in the local subregions of inspection images and solve the problem of simultaneously using multiple sensing devices. The proposed method outperformed the other hybrid methods.

Learning classification results from using the proposed method

Figure 6(a)–(d) provides the LARG segmentations with j-level DWTs. Figure 6(d) demonstrates that the sample inspection images with the six-level DWT that were processed using the proposed method differ from one another and can be classified according to these differences. The results illustrate that the segmentations performed for {A, B, C, D} and {E, F, G, H} with the thresholds adjusted automatically to { $T_{0,0}^{77}$ , $T_{0,1}^{80}$ , $T_{1,0}^{82}$ , $T_{1,1}^{66}$ } and { $T_{0,0}^{74}$ , $T_{0,1}^{78}$ , $T_{1,0}^{80}$ , $T_{1,1}^{64}$ }, respectively. Table 4 lists the classification results with the four-level and six-level DWTs. As indicated in the table, the learning algorithm with the four-level DWT caused some errors, attaining an average accuracy rate of 89%. The learning algorithm with the six-level DWT solved this problem effectively, attaining an average accuracy rate of 95% because the thresholds can be automatically adjusted to the most appropriate values derived from the SVM classification.

Figure 6.

Image segments with the j-level DWT: (a) j = 3, (b) j = 4, (c) j = 5, and (d) j = 6.

Table 4.

Learning classification results from the proposed method with four-level and six-level DWTs.

{A, B, C, D}					{E, F, G, H}
T\P	A	B	C	D	T\P	E	F	G	H
A	181/191	6/2	7/4	6/3	E	179/189	5/2	8/5	8/4
B	4/2	176/188	12/6	8/4	F	6/2	178/190	10/4	6/4
C	4/2	10/3	177/191	9/4	G	4/2	9/2	179/193	8/3
D	6/4	6/2	10/4	178/190	H	6/2	8/4	10/6	176/188

DWT: discrete wavelet transform.

T (rows): true values; P (columns): predicted values; Results (values/values): four-level DWT or six-level DWT.

Table 5 provides an example of the selection thresholds of the sample images with the j-level DWT coefficient features. The thresholds for the j-level DWT can be selected automatically according to the SVM results. The most suitable six-level DWT from numerous DWT features was obtained. The proposed method incorporates the leave-one-out cross-validation (LOO-CV) approach for estimating the generalization of the threshold in the selection criterion. LOO-CV is widely used for selecting models, features, and, particularly, small datasets. ¹⁶ Table 6 displays that {A, B, C, D} and {E, F, G, H} attained the smallest mean squared errors: 0.1455 and 0.1785, respectively. The thresholds $T_L^i=\{T_{0,0}^{77},T_{0,1}^{79},T_{1,0}^{82},T_{1,1}^{66}\}$ and { $T_{0,0}^{75}$ , $T_{0,1}^{77}$ , $T_{1,0}^{80}$ , $T_{1,1}^{64}$ } were the optimal selections for {A, B, C, D} and {E, F, G, H}, respectively, in the inspection because they achieved the highest accuracy rates (Table 5).

Table 5.

Selected thresholds for {A, B, C, D} and {E, F, G, H} with j-level DWT coefficient features.

j	{A, B, C, D}	{E, F, G, H}	Accuracy rates (%)
3	{ $T_{0,0}^{75}$ , $T_{0,1}^{81}$ , $T_{1,0}^{84}$ , $T_{1,1}^{65}$ }	{ $T_{0,0}^{73}$ , $T_{0,1}^{76}$ , $T_{1,0}^{82}$ , $T_{1,1}^{64}$ }	87
4	{ $T_{0,0}^{78}$ , $T_{0,1}^{79}$ , $T_{1,0}^{83}$ , $T_{1,1}^{65}$ }	{ $T_{0,0}^{77}$ , $T_{0,1}^{79}$ , $T_{1,0}^{82}$ , $T_{1,1}^{63}$ }	89
5	{ $T_{0,0}^{79}$ , $T_{0,1}^{80}$ , $T_{1,0}^{81}$ , $T_{1,1}^{66}$ }	{ $T_{0,0}^{79}$ , $T_{0,1}^{78}$ , $T_{1,0}^{81}$ , $T_{1,1}^{65}$ }	92
6	{ $T_{0,0}^{77}$ , $T_{0,1}^{80}$ , $T_{1,0}^{82}$ , $T_{1,1}^{66}$ }	{ $T_{0,0}^{74}$ , $T_{0,1}^{78}$ , $T_{1,0}^{80}$ , $T_{1,1}^{64}$ }	95
7	{ $T_{0,0}^{78}$ , $T_{0,1}^{81}$ , $T_{1,0}^{84}$ , $T_{1,1}^{68}$ }	{ $T_{0,0}^{77}$ , $T_{0,1}^{81}$ , $T_{1,0}^{82}$ , $T_{1,1}^{66}$ }	93

DWT: discrete wavelet transform.

Table 6.

LOO-CV mean squared error of approximation coefficient A₆ with $T_L^i=\{T_{0,0}^{75},T_{0,1}^{77},T_{1,0}^{80},T_{1,1}^{64}\}$ , { $T_{0,0}^{76}$ , $T_{0,1}^{78}$ , $T_{1,0}^{81}$ , $T_{1,1}^{65}$ }, { $T_{0,0}^{77}$ , $T_{0,1}^{79}$ , $T_{1,0}^{82}$ , $T_{1,1}^{66}$ }, { $T_{0,0}^{78}$ , $T_{0,1}^{80}$ , $T_{1,0}^{83}$ , $T_{1,1}^{67}$ }, and { $T_{0,0}^{79}$ , $T_{0,1}^{81}$ , $T_{1,0}^{84}$ , $T_{1,1}^{68}$ } for the sample images.

$T_L^i$	{A, B, C, D}	{E, F, G, H}
{ $T_{0,0}^{75}$ , $T_{0,1}^{77}$ , $T_{1,0}^{80}$ , $T_{1,1}^{64}$ }	0.4112	0.1785
{ $T_{0,0}^{76}$ , $T_{0,1}^{78}$ , $T_{1,0}^{81}$ , $T_{1,1}^{65}$ }	0.3021	0.2857
{ $T_{0,0}^{77}$ , $T_{0,1}^{79}$ , $T_{1,0}^{82}$ , $T_{1,1}^{66}$ }	0.1455	0.3699
{ $T_{0,0}^{78}$ , $T_{0,1}^{80}$ , $T_{1,0}^{83}$ , $T_{1,1}^{67}$ }	0.2633	0.4337
{ $T_{0,0}^{79}$ , $T_{0,1}^{81}$ , $T_{1,0}^{84}$ , $T_{1,1}^{68}$ }	0.3594	0.5022

LOO-CV: leave-one-out cross-validation.

The learning algorithm introduced computational complexity ¹⁷ in the experiment. The time cost function T(n) quantifies the amount of time necessary for an algorithm to complete binary tree operations or binary searches and is given as

$$T\left(n\right)=\mathrm{O}\left(\log n\right)$$

(8)

where O(log n) is the logarithmic time that an algorithm requires for all inputs of size n to be expressed in big-O notation excluding coefficients and lower-order terms. Table 7 provides the running-time growth rates derived from the learning algorithm, including the DWT-based feature extraction and SVM. To test the availability of the multiple-object sensing system (Figure 4), given $T_L^P=\{T_{0,0}^{77},T_{0,1}^{80},T_{1,0}^{82},T_{1,1}^{66}\}$ , J_p  = 6, and 200 sample images (40 images of class {A, B, C, D}) in the image queue, the system demonstrated an accuracy rate of 100% for inspecting {A, B, C, D} with response times within 7 µs of O(log n).

Table 7.

Running time growth rates when using the learning algorithm.

n	105	106	107	108
O(log n)	2.0	2.4	2.8	3.2	(µs)

The learning classifiers based on K-nearest neighbor (KNN), ¹⁸ artificial neural network (ANN), ¹⁹ and Bayes classifier ²⁰ were also compared using the proposed method. Similarly, 80 datasets in one inspection image were randomly selected as training samples, and the other 200 datasets were used as validation samples for evaluating the performance of the classifiers. In the KNN classification, the input data were a set of six-level DWT coefficients, and the output was a class membership ({A, B, C, D} or {E, F, G, H}). The inspection image was classified by a majority vote of its neighbors among k = 5 nearest neighbors. The ANN is a three-layer neural network with seven neurons (six-level DWT coefficients) in the input layer, six neurons in the hidden layer, and two neurons ({A, B, C, D} and {E, F, G, H}) in the output layer. The Bayes classifier is given as follows

$$D\left(x\right)=-\frac{1}{2}\ln \left|C_j\right|-\frac{1}{2}\left[{(x-m_j)}^T{C}_j^{-1}(x-m_j)\right]$$

(9)

where x is a set of $\{D_1,\dots ,D_6,A_6\}$ derived from an inspection image. The mean vectors m_j and covariance matrix C_j of the coefficients for the j inspection images are then derived. The inspection image can be identified as the j classes by minimizing the calculated value of D(x). Figure 7 provides the time cost function T(n) and experimental accuracy rates. The accuracy rates were 95% for the SVM, 89% for the KNN, 85% for the ANN, and 81% for the Bayes classifier, demonstrating that the SVM is the most accurate classifier, whereas the Bayes classifier attained was the least accurate. This may be because the probability of occurrence for each class is unknown, as displayed in equation (9). The Bayes classifier assumes that all classes are equally likely to occur. If the number of representative samples from each class is small, then this assumption might be difficult to apply in the inspection.

Figure 7.

Cost function T(n) and accuracy rates from the various classifiers.

To employ the proposed method in another application, objects were placed at different angles for inspection. The inspection can be employed in applications such as inspecting the rotation, position, and tracking of objects. As a test inspection, the learning method was applied to inspect coins placed at various angles. Table 8 provides the class labels of the coin samples (80 training samples and 200 validation samples). With the same setup displayed in Figure 3, the experiment proceeded as follows: A target panel, which included coins placed at four unique angles (Figure 8), was installed on a platform. Four telescopes were mounted on the support frame of the telescope array. Table 9 lists the classification results with a six-level DWT, demonstrating an average accuracy rate of 96%.

Table 8.

Class labels of the coin samples used in the experiments.

Class labels	Coin samples	Class labels	Coin samples
I	Placed at a degree of 0°	M	Placed at a degree of 15°
J	Placed at a degree of 30°	N	Placed at a degree of 45°
K	Placed at a degree of 60°	O	Placed at a degree of 75°
L	Placed at a degree of 90°	P	Placed at a degree of 105°

Figure 8.

Class {I, J, K, L} of coin samples: (a) class I, (b) class J, (c) class K, and (d) class L.

Table 9.

Classification results of the coin samples when using the proposed method with the six-level DWT.

{I, J, K, L}					{M, N, O, P}
T\P	I	J	K	L	T\P	M	N	O	P
I	192	1	3	4	M	190	2	3	5
J	2	191	4	3	N	1	192	4	3
K	2	2	193	3	O	2	2	194	2
L	1	3	4	192	P	2	3	5	190

DWT: discrete wavelet transform.

T (rows): true values, P (columns): predicted values.

Standard test series and comparison of existing methods

A standard test series (Figure 9) in the manufacturing field was proposed for quantitatively comparing the inspection methods. Hybrid schemes of feature extraction methods combined with classification methods were compared to test the proposed method. The DWT, spherical-wavelet-transform (SWT), and invariant (INV) feature extraction methods represent DWT-based, spherical-wavelet-based, ²¹ and moment-INV-based ²² feature extraction, respectively. The classification methods are the SVM, KNN, ANN, and Bayes classifier methods (section “Learning classification results from using the proposed method”).

Figure 9.

Standard test series in the manufacturing field.

For SWT feature extraction, the relationship of SWT coefficients $I=\{w_1,\dots ,w_j,c_j\}$ can be expressed as follows

$$c_0(\theta ,\Phi )=c_j(\theta ,\Phi )+\sum _{j=1}^j{w}_j(\theta ,\Phi )$$

(10)

where c_j and w_j are the approximation and detail coefficients at the j-level decomposition, respectively, and the terms $\theta$ and $\Phi$ represent spherical polar angles. For INV feature extraction, a segmented image f with gray pixel values at pixel (x, y) is denoted as f(x, y), and the central moment can be expressed as

$${\mu }_{\mathrm{pq}}=\underset{-\infty }{\overset{\infty }{\int }}\underset{-\infty }{\overset{\infty }{\int }}{(x-x_c)}^p{(y-y_c)}^{q}f(x,y)\mathrm{dxdy}$$

(11)

where (x_c, y_c ) are the coordinates of the segmented image centroid. The seven Hu-type moment invariants derived from the central moments of the second or third order (p + q = 2 or 3) are [φ_i ] _{i = 1, 2, 3,…, 7} and were used as features.

The procedure for testing the series of existing classification methods and new hybrid approaches can be summarized as follows: (1) input existing classification methods, (2) set the accuracy threshold for the comparison, (3) compare the accuracy and efficiency of the existing classification methods, (4) obtain the high-performance classification method, (5) input the new hybrid approaches of the existing feature extraction methods according to the high-performance classification method, (6) set the accuracy threshold for the new hybrid approaches, (7) compare the accuracy and efficiency of the new hybrid approaches, and (8) obtain the new high-performance hybrid method.

The threshold of accuracy in the standard test series was set to 0.90. The experimental results demonstrate that the accuracy rates for the KNN, ANN, and Bayes classifiers were below 90%. The SVM was the classification method with the strongest performance (test procedures 1–4). The hybrid schemes of the feature extraction methods (DWT, SWT, and INV) combined with the SVM are coded as DWT-SVM, SWT-SVM, and INV-SVM, respectively. When the given values of the threshold were 0.90, 0.93, and 0.95 for test procedures 5–8, the new hybrid methods were {DWT-SVM, SWT-SVM, INV-SVM}, {DWT-SVM, SWT-SVM}, and {DWT-SVM}, respectively. Thus, the accuracy rates were 93% for the SWT-SVM, 90% for the INV-SVM, and 95% for the proposed DWT-SVM. Figure 10 provides the time cost function T(n) and accuracy rates derived from the experiments. The results illustrate that the INV-SVM attained the lowest accuracy rate among the compared schemes. The INV feature extraction method might have been affected by the scene and cameras. If the scene was flat and the cameras were far from the target panels (relative to their sizes), then the moment invariants would have been poorly approximated when using a spatial coordinate transform for the extraction. The proposed DWT-SVM, in addition to attaining the highest accuracy rate, resulted in the lowest time cost functions T(n) among the hybrid methods. Thus, the proposed method outperformed the other methods.

Figure 10.

Cost function T(n) and accuracy rates from the standard test series.

Conclusion

This article proposes an adaptive learning method and standard test series for detecting multiple objects in manufacturing. The proposed learning LARG algorithm combined with the DWT-based SVM classification can adaptively segment multiple-camera images and effectively classify multiple objects in the local subregions of inspection images. The proposed method can inspect multiple objects in one inspection image and solve problems encountered when concurrently using multiple sensing devices. The results demonstrate that the proposed learning algorithm can be used as an inspection tool for automatically selecting the most suitable parameters for inspecting multiple objects. The learning classifier outperforms other classifiers in conducting industrial inspections. The proposed method attained a recognition rate of 95% and outperformed other hybrid methods. However, inspecting multiple ambiguous objects may be a challenge in manufacturing. Further research directions include creating a new feature description method and feature-based inspection method for detecting ambiguous objects in manufacturing.