Sage Journals: Discover world-class research

Abstract

The aim of this article is to automate quality control once a product, essentially a central processing unit system, is manufactured. Creating a model that helps in quality control, increases efficiency and speed of production by rejecting abnormal products automatically is vital. A widely used technology for this is to use industrial image processing that is based on the use of special cameras or imaging systems installed within the production line. In this article, we propose a highly efficient model to automate central processing unit system production lines in an industry such that images of the production lines are scanned and any abnormalities in their assembly are pointed out by the model and information about this is transferred to the system administrator via a cyber-physical cloud system network. A machine learning–based approach is used for proper classification. This model not only focuses on just the abnormalities but also helps in configuring the angles from which images of the production are taken, and our methods show 92% accuracy.

Keywords

Industrial image processing computer vision machine learning cloud computing

Introduction

Over time, our reliance on technology has been increasing tremendously. But there are certain areas where the relationship we have with information technology (IT) is very critical. Our dependence on technology has brought about a massive change in the way we interact with the rest of the world. Computer systems were once independent of each other, but now they are specifically being designed to interact with each other as well as with humans. Devices are being networked everywhere and for a huge range of applications. Most of these devices have built-in sensors that collect data and enable devices to interact with each other. These collected data may then be delivered to people and other devices on demand. In this article, we propose a quality management system based on industrial image processing in which the various devices that enable this are interconnected via a cyber-physical cloud system (CPCS).

In industries, products are generally mass produced. Whenever mass production is involved, it comes with various challenges: quality issues, efficiency, costs, and time consumption. To overcome these problems, many industries have resorted to automation. When it comes to maintaining the quality of the products, there has to be a system at the end of the production line that scans the end products for defects. Most industries have turned to use automated systems that apply image processing to address the above-mentioned challenges. In automated industrial production processes, an image processing system plays a crucial role.¹ In recent times, with the availability of highly efficient modern digital cameras and communication interfaces, image processing has become faster and more productive. The quality of image processing software has become more reliable while simultaneously reducing the cost.

But automizing the industrial manufacturing system comes with its own challenges. Time and central processing unit (CPU) efficiency have become increasingly important. Ensuring quality is the top priority among manufacturers. Machines that are able to implement computer vision efficiently have to be designed such that components can be identified during production flow.

There has already been a lot of research lately in industrial vision systems. In recent times, the machine vision systems used for inspection in industries achieve higher efficiency and robustness compared to older machine vision systems. Researchers are striving to make industrial vision systems that are faster, more efficient, and cost proficient. The industrial vision systems can be used for inspection of dimension quality, surface quality, structural quality, and operational quality.² Now, depending on this classification, the type of processes or techniques used during the inspection may vary. In this article, we are concerned with the structural quality, that is, the correct assembly of the objects in question.

Research in machine vision system for inspection of structural quality of a device has been done in areas ranging from semiconductor and electronics industries, to car assembly lines. Moganti et al.³ discussed various algorithms and technologies for automated visual inspection of printed circuit boards (PCBs). A template matching technology for automatically detecting irregularities in a car production line was introduced in Kim et al.,⁴ which was later improved in Chung et al.⁵ to accommodate inspection of the constitution of the car doors. Techniques for automated inspection of railroad lines were given in the literatures.^6–9

Industrial image processing is generally used in automated inspection systems. Image processing operations are used to improve the image quality, enhance desired features, and so on. Analysis of the enhanced images enables pattern recognition. Pattern recognition concerns in finding similarities and patterns in images which help in classifying the objects or components present in the image. These techniques along with machine learning are exploited to successfully detect structural defects in an object. The analysis tools used recently are based on neural networks (NNs), fuzzy logic, and genetic algorithms (GAs).⁶

In recent times, machine learning techniques have come to be used in inspection systems.¹⁰ This is an improvement over the previously popular template matching¹¹ and design-rule verification¹¹ approaches. Visual inspection systems based on machine learning generally use typical supervised learning algorithms like decision trees,¹² statistical classifiers,¹³ artificial NNs,¹⁴ and support vector machines (SVMs).¹⁵ In this article, we use multi-instance (MI) learning¹⁶ for defect detection.

Among other applications, MI learning has been previously used for image retrieval,¹⁷ object recognition,¹⁸ target tracking,¹⁹ and image categorization.¹⁷ In Paul et al.,²⁰ the visual inspection system is modeled as an MI learning problem.

In this article, we concentrate on quality management during the manufacturing of CPUs. During the assembly of CPUs, there are a number of components that could be misplaced or go missing due to certain errors. At the end of the assembly line, there has to be a protocol in place to assure the quality of these products. This article proposes to develop a system that ensures quality, efficiency, and minimum time consumption at the end of the production line. CPUs consist of various components and each of these components has to be checked for defects. Hence, MI learning can be used as the machine learning algorithm here. Also, there has to be a competent way for all the devices involved to interact with each other efficiently.

The sensors, actuators, and other devices that are interlinked and connected to form a vast network that is generally referred to as smart networked systems and societies (SNSS).²¹ The devices are not restricted to just stationary devices but may also include mobile devices. SNSS bridges the physical and the virtual world to form cyber-physical systems (CPSs). The main objective of CPS is to connect humans and the cyber world via sensors and actuators. CPS connects numerous devices and results in a huge unfathomable exchange of information. In this article, we use an architecture that integrates CPS and cloud computing.

The CPCS has to manage all the resources connected to the system. It contains sensors, actuators, processors, and data services. All operations have to be performed via cloud servers. The cloud servers allocate resources such as bandwidth, storage, and energy as per their demands. CPCS is a very complex system that manages, monitors, and shares information in the real world. The management and optimization of such a complex system is a very challenging task. This entire infrastructure has to be operated efficiently. Because of the number of devices and organizations attached to this, even a small error will have huge repercussions on the entire system. CPSs can assist industry in controlling the complex industrial processes via sensors, actuators, and processors.²²

The rest of this article is arranged as follows. Section “Related work” gives the research that has been done in the area at hand and various technologies or algorithms that are relevant. In section “Proposed research,” we discuss the proposed industrial image processing–based system. The experimental results and analysis are given in section “Experimental results and analysis.” Finally, section “Conclusion” concludes this article.

Related work

Manual assessment of products that are mass produced is time consuming and costly, sometimes inaccurate and complicated. To overcome these problems of quality control and quality management, many industries have taken the assistance of automated services. Image processing takes a huge role in this automation. Various techniques have to be performed to enable automation in quality control in an industry. Generally, image processing is used to compare the defects in the products.

For detection of defects in products, missing parts or components in a product have to be first identified. Some research has been done in detecting missing parts or components in a product have to be first identified. Some research has been performed in detecting equation that was used to estimate the shape of missing regions. Rather than estimating the missing region in each image independently, a joint group action was used to obtain the complete version of the original image.²³

In the study by Vasilic and Hocenski,²⁴ to maintain visual inspection and quality control, an automated system for the ceramic tiles industry was proposed. Various methods for detecting defects in color and surface of ceramic tiles were discussed. Most of the industrial image processing focus on just the defects but not on the process of taking the images that have to be processed. For this, occlusion handling is critical. Zhong et al.²⁵ proposed a method to shorten the time to locate light-emitting diode (LED) chips on wafer. They proposed a fast template matching method that used image segmentation and blob analysis to predict the positions of potential chips. Experimental results on this system showed that the algorithm was much faster than the traditional methods used to previously locate LED chips.

Occlusion handling for vehicle collision handling was proposed in Paul et al.²⁶ In this article, we propose to use occlusion handling technique to calibrate the camera that takes images before they are processed. The remainder of this section briefly discusses the technologies required in this area of research.

Image preprocessing

Before going on with assessing the quality of the product, the images taken of the product have to be pre-processed. Image pre-processing is a very important step in which the captured image is transformed such that certain aspects such as contrast and scale are improved.¹

MI learning

MI learning is an important tool for pattern recognition and computer vision.²⁷ MI learning is a branch of machine learning. In visual inspection applications, there is always uncertainty in the objects that have to be classified. In MI learning, the input consists of “bags” (labeled examples) which in turn consists of multisets of instances. Each set of instances is characterized by an attribute vector and constraints relate labels of each bag to unknown labels in each instance.²⁸

The statement for MI classification problem can be explained as follows: Let ${B_{1}, \dots, B_{n}}$ be a set of bags each with label $l_{i} \in {0, 1}$ . Each $B_{i}$ is a multiset of $n_{i}$ instances, $B_{i} = {B_{i, 1}, \dots, B_{i, n_{i}}}$ . There exists a concept $c$ such that for every $B_{i}$ with $l_{i} = 0$ , $c (B_{i, j}) = 0$ for all $j$ . The task the MI classification problem has to perform is to map a bag $B_{i}$ to its label $l_{i}$ .

In this article, we propose to use this MI learning along with image processing techniques to detect defects in a CPU production line.

CPCSs

The main challenge in placing a CPCS is the economical aspect. In implementing such a system, one has to base in mind that there will be a bulk of devices connected to it. A CPCS will have huge energy costs due to its Information and Communications Technology (ICT) needs. There are a set of constraints and requirements that have to be met in order to ensure that the system is energy efficient. For building an energy-efficient CPCS, the concern is not only the distribution of energy, but also the resources and services involved. The coordination between the various resources and devices involved plays a vital role in the amount of energy consumed.

Demand-side management (DSM): This is related to the demand for particular resources at an industry. Demand for a resource varies and is uncontrollable.²⁹ Maintaining an efficient DSM is very complicated as many variations may occur. This is not that much of an issue when we consider only the industrial image processing aspect. But, when considering the industry as a whole, there will be various issues such as follows:

Daily variation in demand;

Seasonal variation in demand;

There may be interruptions that tend to be very expensive.

Scheduling of resources: There has to be proper resource allocation such that the various services provided by the cloud (cyberspace) are utilized optimally. There has to be coordination between the cloud and the various devices.

High throughput and low delay: Throughput and delay are very important factors to consider in any network. This is because these two factors have a direct impact on the time taken for any type of communication.

A variety of research has been done for efficient management of energy in CPSs. One of the main features of CPS is that it is a distributed system; hence, demand and supply of energy is a huge challenge. Parolini et al.³⁰ proposed a control strategy for data centers in which the energy costs are minimized and at the same time maximizing the quality of computational services. In studies of Zhao and colleagues^31,32 and Paul et al.,³³ building energy management systems (BEMSs) are integrated with CPS to effectively increase the efficiency of energy in CPS. In Lee et al.,³⁴ unified five-level CPS architecture is proposed for the next level industry called Industry 4.0. Various intelligent methods including particle swarm optimization^35,36 can be used in the industrial Internet of Things (IoT) and machine-to-machine (M2M)–based IoT in vehicular transportation systems as well as other areas are well studied.^37–41 A smartbuddy-based⁴² system in industrial IoT can be of good use in upcoming industrial standards.

Proposed research

In an industry where products are mass produced, for example, a CPU production line, a tiny defect like a missing screw may go undetected when manually processed. Also, the mere quantity of products to be processed for quality control may overwhelm the industry. In this article, we propose a system to detect abnormalities in the products.

In this system, images of the end product are taken at the end of the production line. These images are used to detect defects in product.

Capturing the image

In our proposed system, cameras situated at the end of the production line constitute the sensors. The cameras are positioned in such a way that the entire object can be examined via the images taken by the cameras. This is important as the product has to be scrutinized from every angle.

Image preprocessing

This stage takes before the actual processing of the images. As industrial image processing is being used here to ensure the quality of products being produced in the industry, the images captured during the production line have to be pre-processed. By pre-processing, we refer to preparing the image for further processing. The image has to be transformed in such a way that the contrast, scale, and angles of the product are as required.

Modules

This system consists of basically four modules (Figure 1):

Edge detection to separate the image into various components.

Processing the various parts to detect missing parts.

Predicting the description of the missing part (region in-painting).

Correcting possible camera occlusion.

Figure 1.

Pictographic representation of the various modules.

Edge detection to separate the image into various components

We need to first scan the image and separate it into various components. The basic steps of primary edge detection are as follows:

Smoothing: This process suppresses the noise.

Edge enhancement: This step filters the image for contrast.

Edge localization: This determines which local maxima from filter output are actually edges versus noise using thresholding. In thresholding, a threshold value is chosen. Any pixel less than the threshold is set to 0 and any pixel greater than the threshold is set to 1.

The steps to segment the image into the various components using canny edge detector are as follows:

The image is filtered with a derivative of Gaussian.

Find the magnitude and orientation of the gradient.

Non-maximum suppression is used to thin wide ridges down to single pixel width.

Perform linking and thresholding to define two thresholds, low and high. Use the high threshold to start edge curves and the low threshold to continue them.

Segment the image into various parts.

In this article, we propose to use fuzzy image segmentation. These components are then used during the classification process that is described later in this module.

Processing the various parts to detect missing parts

We propose to train a standalone object-centric context representation for seeing what is not there. Given an image, our context model can predict where objects should exist, even when no object instances are present. Combined with object detection results, we can find where objects are missing in an image using convolutional NN structure. This technique ignores objects and focuses on context only. For example, here we consider the missing parts in a CPU assembly line.

From computational perspective, an object can be defined as missing in an image region when:

An object detector finds nothing.

A predictor of the object’s typical environment, that is, context, indicates high probability of its existence.

Given an image, we want to detect all such regions efficiently. We need to train this representation to find missing objects. The aim of this module is to scan an image and detect missing parts. In the previous module, as the image is already divided into components, now the step is to develop an automatic algorithm to scan those components to see if any part is missing. Automating such an algorithm especially for components of the assembly line can save a lot of time as well as increase the efficiency and hence increase the revenue of such an industry. The algorithm has to report not only one but also all the missing components.

We propose to create a model that focuses on the context and hence behaves more efficiently. Here as discussed previously, the context is a CPU assembly line. Hence, the model will have to be trained using already available images in this area. The advantage of this is that the model will have to learn just specific images of normal components and hence, this simplifies training.

In order to successfully detect defects, we need to train the system using MI learning. Let $F$ denote the feature space, also known as the instance space. We have the training dataset, $T = {(B_{1}, l_{1}), (B_{2}, l_{2}), \dots, (B_{n}, l_{n})}$ , where ${B_{1}, \dots, B_{n}}$ is a set of bags and each $B_{i}$ is a multiset of $n_{i}$ instances; $B_{i} = {B_{i, 1}, \dots, B_{1, n_{i}}}$ and $L = {l_{1}, l_{2}, \dots, l_{n}}$ , $l_{i} \in {- 1, + 1}$ are class labels.

Here we assume that a bag represents an object that has to be analyzed for a defect and the label defines whether a feature in the feature space is present or not.

In our case of visual inspection system, all the instances have to be positive for a bag to be labeled positive. The goal of this classification system is to predict the label of an unseen bag

f (B) : B \to L

A bag represents an image and all the possible defects are the instances. Hence, even if one instance is positive, it means that a defect is present. Our MI learning classifier takes the form

f (B_{i}) = {\begin{matrix} + 1, & if \exists B_{i, j} \in B_{i} | f (B_{i, j}) = + 1 \\ - 1, & otherwise \end{matrix}

The algorithm for our MI classification is as follows:

Input: A training set, ${(I_{i}, l_{i}) | 1 \leq i \leq n}$ where $I_{i}$ is an image set.

Output: A trained classifier, $f (B)$ .

Step 1. Create an empty training set, $T \leftarrow {}$ .

Step 2. For each image $I_{i}$ .

Step 3. Create an empty bag, $B_{i} \leftarrow {}$ .

Step 4. Extract all possible defects in $E_{i}$ .

Step 5. For each defect $e_{i, j} \in E_{i}$ .

Step 6. Extract instances into $B_{i, j}$ .

Step 7. Add the new instance to the bag, $B_{i} = B_{i} \cup B_{i, j}$ .

Step 8. Add the new set, $T \leftarrow T \cup (B_{i}, l_{i})$ .

Step 9. $f (B)$ is the trained MI learning classifier of the training set $T$ .

The potential defects extracted using segmentation are described in the previous steps. The various components of the image that have been obtained using edge detection algorithms are the potential defects. If any of these potential defects is positive, it signifies that the component is defective, which means that the object has a defect.

Predicting the description of the missing part

Region in-painting is the process of reconstructing the lost or deteriorated parts of images and videos. Here, this process can be used to describe the missing component.

Correcting possible camera occlusion

When we detect an object, one very important factor is that the image should be clear, and that the entire object is properly visible. If these two conditions are not met, we will not be able to conduct any of the previous modules.

Many challenges still exist while detecting an object such as illusion, low visibility, cast shadows, and occlusion of object. Camera occlusion occurs when a camera’s view is obstructed by something. Occlusions occur under two categories: self-occlusion and inter-object occlusion. Hence, the aim of this module is to calibrate the cameras that capture images of the assembled products in such a way that the object is not occluded.

Occlusion handling has been proposed in areas like avoidance of vehicle collisions in urban crowded roads. We propose to use this technique in this article to calibrate the camera to take images of the entire product. If the entire product cannot be included in one image, then more than one image can be used to detect an abnormality in the product.

With the help of occlusion handling, the system will decide if the images produced by a camera can be trusted, or if more than one images (or cameras) are required to produce the desired result. To successfully implement all the modules, various algorithms need to be performed.

Setup

The setup for the image processing system in this industrial manufacturing system is as follows:

Sensors: Here the sensors of the system for visual quality control are cameras.

Hardware specification: It is suitable to use parallel computers as it is an extremely data-intensive process. Very high bandwidth and computation speed are required to handle the data rates of such applications. But the bulk of industrial inspection tasks can easily be handled with PCs and standard components.

Communication: The image processing system has to work side by side with the manufacturing process and must be able to transmit results to the concerned authorities such that this entire process can be automated. Hence, there has to be a means of communication with other devices. In this system, the data required is stored in the cloud and the communication is done via a CPCS network. Using the cloud brings up various security issues that need to be handled carefully and efficiently.

Algorithm

Input: A system that already contains trained data, image that has to be processed (Figure 2).

Output: If a product has a defect or not, calibration results of the camera.

Assumption: The camera is automatically calibrated by the system.

Step 1. The image fed into the system is analyzed for occlusion.

Step 2. If the image passes the test for occlusion, then the following steps are performed. Else the camera has to be recalibrated and a new image fed into the system.

Step 3. Perform image segmentation using fuzzy image segmentation.

Step 4. Using the segmented image to divide the image into various components.

Step 5. The various components are now classified using the MI learning classification.

Step 6. If all the components are available, then carry out Step 7. Else carry out Step 9.

Step 7. The positions of the components in the product are checked.

Step 8. If the positions of the components are correct, then carry out Step 11. Else carry out Step 9.

Step 9. The description of the missing part and its position is detected.

Step 10. Output: Defective product.

Step 11. Output: Functioning product.

Figure 2.

Flow chart that shows the various modules.

Experimental results and analysis

In this section, we discuss the experiments performed and the analysis of the proposed system. We compared the results of proposed system with other existing systems. This article is implemented using OpenCV and Python programming language. The experiment has been made to gauge the quality control of a product such as CPU system. Constructing a model that supports quality control, boost productivity, and efficacy by rejecting aberrant products automatically is very essential. Industrial image processing is a broadly used technology which uses distinctive kind of cameras or imaging system that is installed in production streak. In this research, we designed a highly efficient model to automatize CPU system production lines in an industry such that images of the production lines are scanned and any abnormalities in their assembly are pointed out by the model and information about this is transferred to the system administrator via a CPCS network. We have used a machine learning–based approach for classification. Our proposed model not only focuses on just the aberrations but also helps in dynamically configuring the angles from which images of the production taken. The accuracy of our proposed model has been compared with the other existing classifiers. Our proposed methods achieved 92% accuracy while classifying different parts of the production.

Image datasets

In order to evaluate the proposed system, we first created an image dataset consisting of products in the CPU production line. We performed classification using four standard classifiers and compared the results to our proposed system. In our database, we have 50 defective images and 100 non-defective images. The potential defects are extracted as explained previously using fuzzy image segmentation and pattern recognition techniques. The dataset has 150 bags (50 positive and 100 negative). The number of instances is 271 positive instances and 253 negative instances.

In Table 1 –3, we compared four state-of-the-art MI learning algorithms^43,44 against the proposed MI learning algorithm. The classifiers that are being evaluated are available in the multiple instance learning (MIL)⁴⁵ MATLAB toolbox. The simulation has been run several times to predict the accuracy of our model with respect to standard classifiers. First five simulation runs have been displayed in tabular form which is showing the accuracy of the proposed model and other standard classifiers. Table 1 shows the percentage of the correct classification while detecting the defects in screws. Proposed model has the capability to detect the screw either it is defective, missing, or loosely screwed.

Table 1.

The percentage of correct classification when there are defects in screws.

Classifier	Run 1	Run 2	Run 3	Run 4	Run 5
Proposed approach	92.4	93.1	94.2	94.1	93.9
mi-SVM average	90.5	90.4	90.2	91.2	99.6
MIL-Boost	93.2	91.2	90.7	90.9	87.6
MILES	83.5	83.7	79.2	77.6	78.7
MInD Mean-Mean	90.2	89.3	85.7	82.1	83.9

SVM: support vector machine; MIL: multiple instance learning; MILES: multiple-instance learning via embedded instance selection.

Table 2.

The percentage of correct classification when there are defects in labels.

Classifier	Run 1	Run 2	Run 3	Run 4	Run 5
Proposed approach	90.7	91.4	92.3	93.1	92.8
mi-SVM average	78.3	79.7	76.4	77.6	79.2
MIL-Boost	83.2	84.6	81.4	81.9	82.0
MILES	69.4	67.5	70.1	70.8	71.2
MInD Mean-Mean	83.4	83.6	83.7	83.9	84.3

SVM: support vector machine; MIL: multiple instance learning; MILES: multiple-instance learning via embedded instance selection.

Table 3.

The percentage of correct classification when there are defects in sockets.

Classifier	Run 1	Run 2	Run 3	Run 4	Run 5
Proposed approach	92.3	92.9	93.7	94.0	92.8
mi-SVM average	91.9	91.6	90.8	91.7	91.3
MIL-Boost	93.1	92.9	93.4	94.5	93.9
MILES	88.7	88.4	89.5	89.8	87.9
MInD Mean-Mean	93.4	93.6	92.5	93.1	92.9

SVM: support vector machine; MIL: multiple instance learning; MILES: multiple-instance learning via embedded instance selection.

Several simulation runs justify that all classifiers have been examined keenly. Every classifier has different capabilities in detecting different situations. From the results, we can clearly see that the mi-SVM average classifier has the higher accuracy in detecting defective screws or sockets but less capability in detecting defective labels. Similarly, other standard classifiers also have the same kind of abilities in detecting missing, lose, or defective screw and sockets but all systems are not comparatively good at label detection, whereas our proposed model is really efficient in detecting all types of defects such as missing, loose, or defective parts of the production. By observing very keenly the above tables, it can be noted that the test on labels was the most efficient on average. But the various algorithms behave differently for different types of errors. One main reason is the efficiency of the type of segmentation algorithms used for dividing the image into various components.

Figure 3 shows the experimental results for detecting missing and/or defective screw corresponding to five simulation runs to clearly visualize the performance of the proposed system with respect to the various classifiers. We can clearly notice that the proposed system has the highest accuracy compared to other classifiers, whereas mi-SVM average has uncertainty at a fifth run but during the first four runs the accuracy is lower than our proposed system.

Figure 3.

The percentage of correct classification detecting the defective screws.

Figure 4 displays the experimental evaluation to detect the defective labels on the product. The proposed system has the guts to read the labels carefully and to identify the defective piece. Compared to other classifiers, we can clearly see that the proposed system is more capable of reading and identifying the defective label on the product. Similarly, Figure 4 presents the evaluation of defective sockets. The proposed system falters a bit in this area. During the experiment, we observed that the cameras could not capture the sockets on an accurate angle because of which the detection of defective sockets has comparatively less accuracy. Moreover, training data and test data or test system have variation of sockets to check the performance of the system (Figure 5).

Figure 4.

The percentage of correct classification while detecting the defective labels.

Figure 5.

The percentage of correct classification while detecting the defective sockets.

Conclusion

In this article, we have proposed a system that automates the quality assessment of the products produced in an industry. This is done by implementing an industrial image processing–based system in an industry that uses CPCS. The image processing system analyses images of the products taken in the production line for defects. The result of this is then communicated to the concerned authority.

The main challenge in our system is extracting the features of the instances. If appropriate feature extraction mechanisms are not used, it jeopardizes the entire visual inspection system.

For future work, we need to consider the security of the system. Involving the cloud during communication increases the security challenges. Hence, many measures have to be taken to ensure that the entire CPCS system is secure from unwanted third parties.

Footnotes

Handling Editor: Antonio Puliafito

Author contributions

N.R. prepared the comparative analysis report. A.P. established the entire framework of the study and selected the tools to be used. W.-H.H. reviewed this work and suggested improving it by making use of various datasets. H.S. used the selected tools for performing simulations and F.S. provided suggestions to collaborative research in cloud computing. K.J.H. suggested various changes during the revision process.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Research Foundation of Korea (NRF) grants funded by the Korean government (MSIP, South Korea; no. 2019R1A2C3002219). The grant covered the research work, as well as the publication costs for open-access publishing.

ORCID iD

Anand Paul

References

Demant

Streicher-Abel

Ganica Carsten

. Industrial image processing: visual quality control in manufacturing. 2nd ed. New York: Springer Science + Business Media, 2013.

Malamas

Petrakis

Zervakis

, et al. A survey on industrial vision systems, applications and tools. Image Vision Comput 2003; 21(2): 171–188.

Moganti

Ercal

Dagli

, et al. Automatic PCB inspection algorithms: a survey. Comput Vis Image Und 1996; 63(2): 287–313.

Kim

Suh

. Automatic visual inspections system to detect wrongly attached components. In: International conference on signal processing applications and technology (ICSPAT’98), Toronto, ON, Canada, 13–16 September 1998. ACM SIGPLAN.

Chung

Kim

. Extracting of line features for automated inspection of vehicle doors. In: International conference on signal processing applications and technology, San Diego, CA, 14–17 September 1997.

Paul

. High performance adaptive deblocking filter for H.264/AVC. IETE Tech Rev 2013; 30(2): 157–161.

Paul

Bharanitharan

Wang

. Region similarity based edge detection for motion estimation in H.264/AVC. IEICE Electron Expr 2010; 7(2): 47–52.

Khandogin

Kummert

Maiwald

. DSP algorithms for the automatic inspection of fixing devices of railroad lines. In: International conference on signal processing applications and technology (ICSPAT’98), Toronto, ON, Canada, 13–16 September 1998. ACM SIGPLAN.

Khalid

. A multiscale stochastic image model for automated inspection. IEEE T Image Process 1995; 4(12): 1641–1654.

10.

Paul

. Adaptive search window for high efficiency video coding. J Signal Process Sys 2015; 79(3): 257–262.

11.

Anand

Bharanitharan

. Algorithm and architecture for adaptive motion estimation in video processing. IETE Tech Rev 2013; 30(1): 24–30.

12.

Chen

Lin

Han

, et al. Automated inspection of engineering ceramic grinding surface damage based on image recognition. Int J Adv Manuf Tech 2013; 66(1–4): 431–443.

13.

Unay

Gosselin

Kleynen

, et al. Automatic grading of Bi-colored apples by multispectral machine vision. Comput Electron Agr 2011; 75(1): 204–212.

14.

Yuen

CWM

Wong

Qian

, et al. A hybrid model using genetic algorithm and neural network for classifying garment defects. Expert Syst Appl 2009; 36(2): 2037–2047.

15.

Paul

Jiang

Wang

. Computation aware scheme for visual signal processing. JSW 2010; 5(6): 573–578.

16.

Dietterich

Lathrop

Lozano-Pérez

. Solving the multiple instance problem with axis-parallel rectangles. Artif Intell 1997; 89(1): 31–71.

17.

Zhang

Chen

, et al. A multiple instance learning approach for content based image retrieval using one-class support vector machine. In: 2005 IEEE international conference on multimedia and expo (ICME 2005), Amsterdam, 6 July 2005, pp.1142–1145. New York: IEEE.

18.

Paul

Rho

Bharnitharan

. Interactive scheduling for mobile multimedia service in M2M environment. Multimed Tools Appl 2014; 71(1): 235–246.

19.

Babenko

Yang

Belongie

. Visual tracking with online multiple instance learning. In: 2009 IEEE conference on computer vision and pattern recognition (CVPR 2009), Miami, FL, 20–25 June 2009, pp.983–990. New York: IEEE.

20.

Paul

Pinjari

Hong

, et al. Fog computing-based IoT for health monitoring system. J Sensors 2018; 2018: 1386470.

21.

Simmon

Kim

Subrahmanian

, et al. A vision of cyber-physical cloud computing for smart networked systems. Gaithersburg, Maryland: US Department of Commerce, National Institute of Standards and Technology.

22.

Wang

Vuran

Goddard

. Cyber-physical systems in industrial process control. ACM SIGBED Rev 2008; 5: 12.

23.

Zhou

Kwan

Ayhan

. A high performance missing pixel reconstruction algorithm for hyperspectral images. In: 2nd international conference on applied and theoretical information systems research, Taipei, Taiwan, 27–29 December 2012. ATISR.

24.

Vasilic

Hocenski

. The edge detecting methods in ceramic tiles defects detection. In; 2006 IEEE international symposium on industrial electronics, Montreal, QC, Canada, 9–13 July 2006, pp.469–472. New York: IEEE.

25.

Zhong

. A fast template matching method for LED chip localization. In: MATEC web of conferences, 2015. EDP Sciences, https://doi.org/10.1051/matecconf/20153404002

26.

Paul

Chen

Bharanitharan

, et al. Video search and indexing with reinforcement agent for interactive multimedia services. ACM T Embed Comput S 2013; 12(2): 25.

27.

Bai

Loog

, et al. Editorial of the special issue on multi-instance learning in pattern recognition and vision. Pattern Recognit 2017; 444–445.

28.

Ray

Scott

Blockeel

. Multi-instance learning. In: Sammut

Webb

. (eds) Encyclopedia of machine learning. New York: Springer US, 2011, pp.701–710.

29.

Reka

Ramesh

. Demand side management scheme in smart grid with cloud computing approach using stochastic dynamic programming. Perspect Sci 2016: 8: 169–171.

30.

Parolini

Tolia

Sinopoli

, et al. A cyber-physical systems approach to energy management in data centers. In: Proceedings of the 1st ACM/IEEE international conference on cyber-physical systems, Stockholm, 13–15 April 2012, pp.168–177. New York: ACM.

31.

Zhao

Simoes

Suryanarayanan

. A conceptual scheme for cyber-physical systems based energy management in building structures. In: 2010 9th IEEE/IAS international conference on industry applications (INDUSCON), Sao Paulo, Brazil, 8–10 November 2010, pp.1–6. New York: IEEE.

32.

Zhao

Suryanarayanan

Simoes

. An energy management system for building structures using a multi-agent decision-making control methodology. IEEE T Ind Appl 2013; 49: 322–330.

33.

Paul

Jiang

Wang

, et al. Parallel reconfigurable computing-based mapping algorithm for motion estimation in advanced video coding. ACM T Embed Comput S 2012; 11(S2): 40.

34.

Lee

Bagheri

Kao

. A cyber-physical systems for industry 4.0-based manufacturing systems. Manuf Lett 2015; 3: 18–23.

35.

Paul

Rho

. Probabilistic model for M2M in IoT networking and communication. Telecommun Syst 2016; 62: 59–66.

36.

Paul

Victoire

TAA

Jeyakumar

. Particle swarm approach for retiming in VLSI. In: Proceedings of the 2003 IEEE 46th Midwest symposium on circuits and systems, Cairo, Egypt, 27–30 December 2003. New York: IEEE.

37.

Paul

. Real-time power management for embedded M2M using intelligent learning methods. ACM T Embed Comput S 2014; 13: 1–22.

38.

Daniel

Paul

Ahmad

, et al. Cooperative intelligence of vehicles for intelligent transportation systems (ITS). Wirel Pers Commun 2016; 87: 461–484.

39.

Paul

Daniel

Ahmad

, et al. Cooperative cognitive intelligence for internet of vehicles. IEEE Syst J 2017; 11: 1249–1258.

40.

Paul

. Graph based M2M optimization in an IoT environment. In: Proceeding of the 2013 research in adaptive and convergent systems, Montreal, QC, Canada, 1–4 October 2013, pp.45–46. ACM.

41.

Paul

Chilamkurti

Daniel

, et al. Intelligent vehicular networks and communications: fundamentals, architectures and solutions. Amsterdam: Elsevier, 2016.

42.

Paul

Ahmad

Rathore

, et al. Smartbuddy: defining human behaviors using big data analytics in social internet of things. IEEE Wirel Commun 2016; 23: 68–74.

43.

Dong

. A comparison of multi-instance learning algorithms. Doctoral Dissertation, The University of Waikato, Hamilton, 2006.

44.

Cheplygina

Tax

Loog

. Multiple instance learning with bag dissimilarities. Pattern Recogn 2015; 48(1): 264–275.

45.

Tax

DMJ

Cheplygina

. MIL: a MATLAB toolbox for multiple instance learning (version 0.8.1), 2011. Mathworks (Matlab).

Machine learning–based automated image processing for quality management in industrial Internet of Things

Abstract

Keywords

Introduction

Related work

Image preprocessing

MI learning

CPCSs

Proposed research

Capturing the image

Image preprocessing

Modules

Edge detection to separate the image into various components

Processing the various parts to detect missing parts

Predicting the description of the missing part

Correcting possible camera occlusion

Setup

Algorithm

Experimental results and analysis

Image datasets

Conclusion

Footnotes

Author contributions

Declaration of conflicting interests

Funding

ORCID iD

References