Abstract
In this study, a visual grading system of vegetable grafting machine was developed. The study described key technology of visual grading system of vegetable grafting machine. First, the contrasting experiment was conducted between acquired images under blue background light and natural light conditions, with the blue background light chosen as lighting source. The Visual C++ platform with open-source computer vision library (Open CV) was used for the image processing. Subsequently, maximum frequency of total number of 0-valued pixels was predicted and used to extract the measurements of scion and rootstock stem diameters. Finally, the developed integrated visual grading system was experimented with 100 scions and rootstock seedlings. The results showed that success rate of grading reached up to 98%. This shows that selection and grading of scion and rootstock could be fully automated with this developed visual grading system. Hence, this technology would be greatly helpful for improving the grading accuracy and efficiency.
Keywords
Introduction
Grafting is a technique whereby branches or buds from one plant are inserted into the appropriate parts of another plant that has a strong affinity. A branch or bud to be cultivated (i.e. scion) is encouraged to fuse or graft onto a strong vital plant (i.e. rootstock) so that the scion can grow, blossom, and bear fruit using nutrient soil. Grafting technique can reduce soil-borne diseases, enhance stress resistance, increase yield and fruit quality, and has no effect on the plant quality. 1 Currently, the grafting technology is rapidly developing over the world in order to adapt to the needs of agricultural production. At the same time, grafting robot technology has been developed. Some semi-automatic grafting robots have been developed and commercialized for cucumber, tomato, and watermelon in Japan, South Korea, the Netherlands, and China. In these countries, the working efficiency of semi-automatic grafting robots is three times faster than the traditional manual grafting. 2 Furthermore, for most automatic grafting machines, the grafting speed can reach up to 600 seedlings per hour. However, since the machines still require auxiliary manual work, they can only achieve a semi-automatic status. The scion and stock plants must be classified so that the same size of seedlings can be placed properly in contact with each other in the process of grafting. In this way, it can reduce the organization’s bare area and loss of tissue fluid, and also survival rate of the grafted seedlings can be guaranteed.
Machine vision is widely used in many fields, such as industrial, animal husbandry, fisheries, agriculture, and horticulture. In recent years, the applications of machine vision have been deeply studied in various fields, for instance, in grading of panel surfaces, 3 high temperature measurements of surface properties, 4 automatic grading system for fresh fruits, vegetables, areca nuts, eggs, and others.5–13 Moreover, researchers in recent have presented some algorithms for processing images, which is the core of machine vision.14–17 Ebrahimi et al. presented the image processing algorithms in many research fields. The main algorithm includes color space conversion, gray-scale transformation, image binarization, noise smoothing, and morphological operations.18–22 The machine vision system of automatic grafting machine has still many shortcomings, and many problems are unsolved like lower accuracy. Different algorithms of image processing technology have been adopted in many fields, for example, support vector machine (SVM).15,19,21 Although these algorithms can distinguish the characteristics of measured object in detail, the calculation process is too complex, so it is not suitable for grading characteristics extraction of grafting seedling. Furthermore, because of the variety of each seedling, those algorithms are not accurate in grading seedlings for some machine vision system. Also a comparison of natural and artificial light source has not been reported.1,23,24 For this reason, it is important and necessary to develop the simple and practical visual grading system for the muskmelon grafting machine.
Currently, semi-automatic grafting machines have been developed, while the fully automatic grafting machines are still in development process.23–25 On comparison, a fully automatic grafting machine is supposed to have higher grafting requirement than semi-automatic. For instance, the diameter of rootstocks and scions should be the same. Therefore, the fundamental requirement of the fully automatic machine is to use the machine vision technology for grading seedling. However, the human identification for grading of rootstocks and scions is labor-intensive and has a low grafting efficiency. This will greatly decrease the survival rate. In previous studies, most designs are to use sunlight, but sunlight has a strong instability. It will change with time or weather change. In that way, the sharpness, contrast, brightness, and other factors of the taken out image will be greatly affected.
The aim of this study is to develop an automatic visual grading system of vegetable grafting machine for seedlings grafting. The main objectives of this study are as follows: (1) to design an automatic visual grading system which will include image acquisition module, image processing module, and control module; (2) to choose appropriate light sources in the image acquisition module; and (3) to conduct grading test of muskmelon scions and rootstocks using the automatic visual grading system.
Materials and methods
Experimental equipment
The experimental equipment includes a light source device, PC computer, and DaVinci image processing kit model (EL-DM6437; produced at Beijing DaSheng Precision Instrument Co., Ltd). A blue light-emitting diode (LED) panel backlighting was used as an illumination source with three strength grades, that is, high, medium, and low. In this experiment, the blue LED panel backlighting strength was adjusted to medium grade from the light source controller and obtained luminous flux of 1000 lumens.
The DaVinci kit consists of image capture card, color digital camera, lens, and image processing software. The camera has focal length with a range of 3.9–85.8 mm, a charge-coupled device (CCD) image sensor chip with size ratio of 1/1.6 in (7.2 × 4.8 mm), and lens with diameter of 46 mm. The PC is a Lenovo desktop with good laboratory performance and has Window XP operating system, an i5 3.20 GHZ CPU processor, and 1 TB hard disk with 4 GB memory.
Experimental parameters considered, which are the same with that of camera calibration, are as follows: distance between camera lens and seedling (190 mm), distance between blue background light source and seedling (70 mm), image distance (13.82 mm), focal length (12.88 mm), and image resolution acquisition (720 × 480 pixels).
The automatic visual grading system for grafting seedlings
The automatic visual grading system for this work consists of three modules: the image acquisition module, image processing module, and control module (Figure 1(a)–(c)).
Procedure of visual grading system, (b) image acquisition and processing module, and (c) control module.
The image acquisition module was used to obtain digital images of the target seedling, and it is composed of light source, EL-DM6437 image processing kits, and image acquisition card (Figure 1(a)). First, the light source produces blue LED background light and the color camera captures image of seedling under the blue LED background light conditions. The image acquisition card stored the image data and output them to the processing module.
The image processing module, which includes both hardware and software, was used to translate the image data into grading information as shown in Figure 1(a) and (b). The module has basically four key techniques: image preprocessing, extracting image feature, analyzing grading information, and outputting grading information. After image processing, grading information were obtained and sent to the control module.
Control module of the visual grading system was used to drive motors and actuators to conduct the seedling grading. It is made up of control system and actuator. The control system composed of microcontroller–programmable logic controller (PLC), input/output (I/O) Relay Board, light-electric sensor, limit switch, and electric motor (Figure 1(c)). The microcontroller–PLC is the key component, and it received grading information of the target seedling which was changed to different grading levels. Then the electric motors drive the mechanical system and transport the graded seedling to the designed position.
Ashraf et al. tested and compared the effect of blue, red, and white background light sources in grafting seedling visual grading system. The results showed that in order to obtain best contrast ratio, the blue background light source was found superior to other color light sources. However, this work compared the blue background light with natural light in order to achieve the lowest cost. Selecting a light source should comply with the rules of stability and should provide uniform illumination and avoid reflections. 1 There are no general light sources for different targets, but to obtain an optimal light source and lighting system, it is necessary to perform numerous experiments according to the professional experience of researchers. Accordingly, first, the requirements for lighting system and size of the light source can be determined based on the project target. Second, characteristics of the target, such as material, shape, and color, should be determined based on the recommended theoretical knowledge. Third, a light source must be selected according to project requirements and the binding properties of different light sources. Finally, the optimal light source type and lighting system will be obtained.
Technology of the vision grading system
Seedling grading
The experimental samples used are two plug trays (5 × 10 points) of muskmelon scion seedlings and two plug trays (5 × 10 points) of pumpkin seedlings. The vegetation period of muskmelon and pumpkin is 15 and 7 days, respectively. Corresponding stem diameters of muskmelon scion and pumpkin rootstock seedlings were manually inspected, and seedling level was determined.
Seedling stems of 100 muskmelon scions and 100 pumpkin rootstocks were measured to obtain the distribution of the stem diameter sizes, as shown in Figure 2. The range of muskmelon scion seedling stem diameter is 2.10–3.60 mm, and that of pumpkin seedling rootstock diameter is 3.25–4.60 mm.
Measurement results of 100 muskmelon scion seedlings and 100 pumpkin rootstock seedlings: (a) muskmelon seedlings and (b) pumpkin seedlings.
Based on the stem diameter ranges, muskmelon scion seedlings and pumpkin rootstocks seedlings were divided into four grades. Table 1 shows the different levels corresponding to the pixel range and grade of stem diameter range for muskmelon and pumpkin. From Table 1, the levels of “A,”“B,”“C,” and “D” were determined by the grading criteria. Accordingly, A level represents the grafted stem of a small diameter, B level represents the grafted stem of a medium-sized diameter, C level represents the grafted stem of bigger diameter, and the D level represents the grafted stem of failed grading seedlings, with diameters that are either too large or too small.
The size conversion chart for the grading levels.
Procedure of seedling grading
The procedure of seedling grading includes four key technologies. They are image acquisition, image preprocessing, feature information extraction, and seedling grading (Figure 3). First, control actuators grabbed the seedlings and put them to the specified photo location. To ensure that every individual operation was not affected by the immediate past operations, the system was initialized before the procedure of image acquisition. After image acquisition, the acquired image was transferred to the processing unit. Also, to ensure that next steps become easier and images are more obvious, the image was extracted. Consequently, after preprocessing, information on seedling stem diameter features is extracted. Finally, classification operations are performed in accordance with the standard of given grade seedlings. However, the second individual plant seedling will not be processed until the first one is completed. In this way, the grading of whole batch of grafting seedlings is completed.
Flow chart of seedling grading procedure.
1. Image acquisition.
First, the muskmelon and pumpkin seedling images were obtained using the image acquisition unit. The muskmelon scions and pumpkin rootstocks were graded and sent to the designated location in visual grading system by grafting machine. Image processing suite of the image acquisition unit acquires images of the seedlings, and the images were sent to the processing unit via the image capture card.
2. Image preprocessing.
Image preprocessing mainly includes four parts, that is, gray-scale transformation, binary image processing, noise smoothing, and morphological closing operation. The preprocessing analysis includes image processing and obtaining information about the diameter of the muskmelon scion seedlings.
Gray-scale transformation. To facilitate image processing and analysis, the color images are usually converted into gray scale. In a gray-scale image, the gray values of the pixels are represented by one byte. Gray value ranges from 0 to 255. The smaller value of the point is, the blacker and darker the point is. The greater value the point, the whiter and lighter the point. Namely, 0 represents the darkest, 255 represents the brightest, there is no color information after the conversion containing only luminance information.
Binary processing. Binarization is to set the gray value pixel image 0 or 255, namely, to show the visual effect of only black and white. In this article, there are obvious differences between the image and the background image of grafted seedlings, and the image needs binarization processing.
Image smoothing. In this article, the main image consists of pulse noise (salt and pepper noise), and the detail of the image is not too much, so removing pulse noise by smoothing processing. Using the median filter for image operations is an effective mean to suppress noise, especially for pulse noise with black and white dots superimposed on the image. The basic principle of median filtering is first sorting all the pixels in a neighborhood of one pixel in accordance with the gray value and then selecting the median arrangement to replace the gray values of surrounding pixels with large difference in gray.
Morphological closing operation. Morphological closing operation includes first dilating the binary image, and then doing erosion operation for the processed image. Closing operation can integrate thin holes and narrow gaps in image and eliminate tiny edges irrelevant to the target. It makes the contour lines of the target in binary image to become much smoother.
3. Feature information extraction.
To distinguish the pixels between the target and background, the pictures need to first pass the threshold of transformation and then split the image according to the threshold value. The pixel is considered to be part of the background if the value is greater than or equal to the predetermined threshold value; otherwise, it is considered the target. It is easy to obtain better results when the target and the background pixels have a significant difference. After choosing a proper threshold value, a muskmelon scion seedling can obtain the threshold value.
The grafting positions of seedlings are on the main stems, so the standard of grading is in accordance with the diameter of the main stem. This study designed a relatively simple method of calculation, which mainly based on the number of pixels of the stem. It uses the cumulative number of 0-valued pixels across the horizontal direction in each row and then determines the maximum frequency of the total number of 0-valued pixels in all rows. When the maximum frequency is greater than or equal to the minimum height of the stem (100 pixels) on behalf of the pixel values, the highest frequency occurs, and the total number of pixels indicates the required stem diameter. Based on the former analysis, the actual size of each pixel has been known, so the diameter of the muskmelon seedling’s stem could be calculated.
Finally, the seedling stem diameter is calculated according to the number of 0-valued pixels; this method is shown as follows
where lpixel is the actual size of the seedlings that every pixel represents in the horizontal direction (mm), so the actual size of one pixel is 0.1375 mm; l is the horizontal size of the camera image sensor chip (mm); N is the total number of horizontal pixels; T is the actual size of the seedling stem diameter (mm); n is the total number of pixels occupying somewhere in the seedling stem diameter across the horizontal projection; f(x, y) is the transformed gray value of a pixel at point (x, y); W is the width of the picture; and H is the height of picture.
The key of grading is information extraction, and the specific process is shown in Figure 4. The numbers of all the 0-gray-value pixels in each row are added up after preprocessing seedling images. The maximum frequency of 0-gray-value pixels accumulation can accurately show the size of diameter. The accumulation curve of 0-gray-value pixel numbers in each row is shown in Figure 4 for processed image of one seedling. The line AB represents the 0-gray-value pixels accumulation in different columns of the stem in the seedling image. The part above point A represents the accumulation of leaf crown. The part below point B represents the accumulation of the number of the pixels of the seedling pot or the cultivate soil. Find the value with a largest frequency of occurrence within section AB, which represents the number of the pixels of the seedling’s diameter.
Information extraction procedure for stem diameter.
If section AB could be found approximating a straight line with certain length (more than 100 pixels), the classification is successful. Otherwise, the classification failed. Then based on the standard of classification, if the cumulative sum of the number of the pixels whose gray value is zero is from 15 to 26, the grafted seedling is qualified. And then calculate the cumulative sum of the number of the pixels whose gray value is zero with a largest frequency of occurrence. With this cumulative sum, the diameter of the seedling can be easily calculated. Finally, the diameter is used to judge the level of a seedling.
Results
Light source selection
In this study, the natural light and blue light were used as lighting source. The contrast effects are shown in Figure 5.
The acquired images under blue background light and the natural light conditions: (a) color image with acquisition conditions of a light blue background, (b) color image collected under natural light conditions, (c) color image to gray image with a light blue background, and (d) color image to gray image under natural light.
Based on the processing in Figure 5, the gray value curve of the cutting point of grafted muskmelon seedlings in the horizontal direction could be obtained, and the results are shown in Figure 6. The gray value of muskmelon seedlings under natural light conditions was approximately found to be 130, whereas the gray value of the background was approximately 180, showing a difference of nearly 50. The gray value of muskmelon seedlings in blue background light was concentrated at 40, but its background gray value was up to 180, showing a large difference.
The variation curve of the gray values along the X-axis at the cutting point.
From this experiment, the contrast of the blue background light was obtained to be better than the contrast of the natural light condition, so the scheme used in this study incorporates the blue background light.
Seedling grading
The automatic grading system of grafted seedlings from the muskmelon grafting machine is based on the machine vision system. The grafted seedlings include the muskmelon scion seedlings and pumpkin rootstock seedlings. The grading method of the muskmelon scions and pumpkin rootstocks is similar, so this experiment only used muskmelon scion seedlings to verify the grading.
Rows present eight different images of seedling. Seedlings 1–6 are examples of successful grading, and seedlings 7 and 8 are examples of unsuccessful grading. Three columns display the processing steps of the same image, that is origin color image, smoothing image, and morphologic processing.
Figure 7 shows the series of processing steps of successful and unsuccessful grading examples of muskmelon scion seedlings. Seedlings 1–6 are examples of successful grading, and seedlings 7 and 8 are examples of unsuccessful grading. The results of classification are shown by the curves in Figure 8. From the curves of the first seedling to the sixth seedling, section AB can be found to approximate the straight lines which represent the cumulative sums of the number of the pixels in the different heights of the stems whose gray value is zero. Then, the values between sections A and B with the largest frequency of occurrence can be found, and the diameters of the stems can be calculated. While it cannot be found that section AB approximates the straight line from the processed result curves of the seventh and the eighth seedlings. So the number of the pixels which represents the diameter cannot be confirmed, as well as the diameter of the seedling cannot be calculated. As a result, the classifications of the seventh and the eighth seedlings are failed.
Examples of processing grading seedling.
Curves of grading seedling.
After automatic grading by visual system, the seedling level was verified and the corresponding muskmelon scion seedling stem diameter was manually inspected to judge whether the grading system was correct. The automatic grading process of the 100 muskmelon scion seedlings and 100 pumpkin seedlings used the same operation. According to the grading criteria, 100 muskmelon scion and 100 pumpkin rootstock seedlings were counted, and the results of the automatic grading are shown in Figure 9. From Table 2, the number of various grades of muskmelon scion seedlings and pumpkin rootstock seedlings are basically the same. Part of the muskmelon scion seedlings automatic grading results was compared with the manual measurements shown in Table 2.
The statistical diagram of the grading for 100 muskmelon seedlings and pumpkin seedlings.
The comparison of the system grading results and manual grading.
The results of the contrast test between the measuring diameter by hand and the classification with the machine vision with 100 muskmelon scion seedlings and pumpkin rootstock seedlings are shown in Figure 9. In the result of the measuring diameter by hand, the number of A level of the seedlings is 22, B level is 60, C level is 15, and D level is 3. While the result of the classification with the machine vision shows that the number of the A level of the seedlings is 21, the B level is 59, the C level is 16, and the D level is 4. And the success rate of the classification is 97% (because of the systematic error, the automatic grading results of which are the C level, but the manual measurement grades it as the B level, the graded result is shown in No. 7 of Table 2). The reason leading to the classification failure is the stem was blotted out by the leaves, so the diameter cannot be determined. The result of the other 100 seedlings is shown in Figure 9.
Therefore, the success rate of this system on the muskmelon and pumpkin seedling grading success rate achieved ∼97% and 98%, respectively, and the measurement results are basically consistent with the manual measurement results, with a grading error of between −2.3% and 2.3%, and an average grading error of −0.57%, which meets the requirements of design.
Discussion
In this study, some practical operations of the different kinds of light sources were conducted and a mass of images was acquired. The designation of light source determined the significant degree of target objects in the image. Therefore, the extraction of further information becomes much easier in the process of image processing. After using the best illumination, the acquired images were clearer, the target characteristics are more obvious, and the image was much easier to deal with. In this study, the process of image classification and the algorithm of seedling grading were described in detail. The algorithm used is not only simple, but also fast with high accuracy. It does not only shorten the running time but also greatly improve the success rate of the grading; consequently, it laid a solid foundation for fully automated grafting machine.
It has remained a great challenge to design a simple and practical visual grading system. To solve this problem, one possible way is to establish hardware and software system of grading system for grafting machine. In recent years, the computer vision technique is becoming one of the main methods to study grading in industrial, animal husbandry, fisheries, agriculture, horticulture, and other fields. In fact, the algorithms can distinguish the characteristics of the measured object in detail. However, the calculation process is too complex. Thus, it is not suitable for the grading characteristics extraction of grafting seedling. Furthermore, because of the variety of each seedling, some algorithm is very simple and the results are not accurate in grading seedlings, so it is difficult to achieve grafting seedling grading. Therefore, the simple and practical visual grading system for the muskmelon grafting machine was put forward.
In this study, the key techniques were developed to achieve grading information with the visual grading system. First, after comparisons of background light and natural light conditions, the blue background light was chosen as the lighting source. Second, the algorithm of the maximum frequency of the total number of 0-valued pixels was studied, and pixel measurements of the scion and rootstock stem diameters were extracted. Finally, the contrasting experiments of manual and automatic grading were conducted. The results showed that visual system meets the design requirements.
From experimental results of Figures 8 and 9, it can be found that the maximum frequency of the total number of 0-valued pixels each row is the stem diameter of scion and rootstock. The theory of characteristic curve in Figure 8 is using the feature of the stem diameter of muskmelon scion for grading. After image acquisition and binary image processing, the number of all the 0-gray-value pixels in each row is added up. Finding the maximum frequency of the 0-gray-value pixels accumulation is a more accurate method to calculate the size of stem diameter. From the curves of the first seedling to the sixth seedling, it can be found that section AB approximates the straight lines which represent the cumulative sums of the number of the pixels in the different heights of the stems whose gray value is zero. Then, the values between section AB with the maximum frequency of occurrence can be found, and the diameters of the stems can be calculated when each pixel points have been known, which represent the actual image size. The part ahead point A represents the cumulative sum of the number of pixels of the leaf crown whose gray value is zero. The part below point B represents the cumulative sum of the number of the pixels of the seedling pot or the seedling cultivates soil whose gray value is zero. If section AB could be found approximating a straight line with certain length (more than 100 pixels), the classification is successful. Otherwise, the classification is failed. Based on the standard of classification, if the cumulative sum of the number of the pixels whose gray value is zero is between 15 and 26, the grafted seedling is qualified. And then the accumulation of the number of the pixels can be calculated, the gray values of the pixels are zero with the maximum frequency of occurrence. Based on this accumulation value, the diameter of the seedling can be easily calculated. Finally, the diameter value was used to judge the level of the seedlings. It cannot be found that section AB approximates the straight line from the processed result curves of the seventh and the eighth seedlings. So the number of the pixels which represent the diameter cannot be confirmed, as well as the diameter of the seedling cannot be calculated. As a result, the classifications of the seventh and the eighth seedlings failed.
It is difficult to judge which pixel represents stem parts, cotyledons parts, or root parts, so the uncertainty remains as to the cause of the grading failure. For example, from the result of fourth seedling, the horizontal number of pixels of 0-valued points is stable at 22 pixels, and a nearly straight line of the curve (point A to point B) will be found (Figure 8). The seedling vertical height ranges from 160 to 430 pixels. Therefore, the vertical height represents the stem part of the muskmelon scion seedlings. The 22 pixels of the biggest frequency represent the number of pixel points of the muskmelon seedling stem diameter, so the diameter of this seedling stem is 3.025 mm, belonging to the B level. The curve shows volatile changes when the vertical height is less than 160 pixels or greater than 430 pixels. The upper part represents leaves and the lower part is root of the muskmelon seedlings. The volatile curve is due to seedling leaves and clods on their roots. From seedlings 7 and 8 of the unsuccessful examples in Figure 7, the total number of horizontal 0-valued points substantially changed with the height of muskmelon scion seedlings (Figure 8).
This algorithm is simple and convenient. Compared with the traditional algorithm, this algorithm takes the midpoint of this segment as diameter, and it is more accurate. The results showed the grading success rate of grafting on the muskmelon and pumpkin seedling achieves ∼98% using this present algorithm and system.
Conclusion
In this study, the comparative analysis of a vegetable grafting machine and machine vision technology in agricultural production was conducted. This study can identify that the blue light illumination can be used to capture the best images, based on machine vision and image processing theory. Furthermore, the image processing and corresponding algorithms were designed and developed using vision system software with Open CV image and video libraries on a Visual C++ programming platform.
For 100 seedlings, an automatic grading test was conducted based on a machine vision muskmelon scion automatic seedling grading system. The results showed that the automatic grading results are basically consistent with manual measurement results, with a grading success rate of ∼97% and 98% for muskmelon and pumpkin, respectively, a grading error of −2.3% to 2.3%, and an average grading error of −0.57%. The test results met the requirements of the design system, which laid an important foundation for further developing an automatic vegetable grafting machine.
Footnotes
Academic Editor: ZW Zhong
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work is supported by the National Natural Science Foundation of China (31101076), Excellent Talents Plans in Liaoning Province (LJQ2012061), and Tianzhu Mountain Scholar in Shenyang Agricultural University (2014).