Apple harvesting robot under information technology: A review

Abstract

It has been more than 30 years since the French pioneered the research of the apple harvesting robot, and with the joint efforts of scholars all around the world, a variety of prototypes have been developed. However, the existing apple harvesting robot prototype is still in the experimental research stage because of its low harvesting efficiency. With the help of information technology, the related research has ushered in a milestone development, and it is full of opportunities and challenges for apple harvesting robotic researchers. In this article, it briefly introduced the development history, structure, and composition of apple harvesting robots and the operation process, which makes readers have a clear understanding of apple harvesting robot and its harvesting principle. Then systematically summarizing the research results of apple harvesting robots both at domestic and at foreign, we carried out in following three aspects: rapid and accurate recognition and positioning of target fruit, all-weather operation mode, and application of intelligent computing theory in apple harvesting robots, and it analyzes the research progress of apple harvesting robot in detail. The results show that improving the harvesting efficiency is the key and hot spot for the research on apple harvesting robots. Under the impetus of information technology, how to achieve fast and accurate recognition of the fruits of multienvironment and multiple information, obtain reasonable path planning, and further optimization of control strategies are all important research directions.

Introduction

Apple belongs to the heliotrope temperate deciduous tree; due to its crisp fruit and rich nutrition, it is known as the “the crown of fruit.” Because of its high water solubility and easy absorption by the body, it is also called “living water.” Apple ranks second in the global fruit production rankings over grapes, according to the statistics of the Food and Agriculture Organization. Harvesting operation is a time-consuming and labor-intensive process in the production of fruits and vegetables, which is also an important part of the whole agricultural production process. Because some fruits and vegetables are harvested with strong seasonality, there may be intensive harvested operating scenes, resulting in a very large harvesting workload. The harvesting operations are labor-intensive, and it is currently dominated by manual operations, resulting in low efficiency, which, in turn, restricts the market competitiveness of agricultural products. With the rise of other industries, the agricultural labor force has gradually shifted to cities, causing a large number of migrant farmers. With the growing of aging population, resulting in agricultural production the rural labor force resources increasingly tense. The lack of agricultural labor may cause the failure of timely harvesting, thus causing huge losses to agricultural production.

With the advancement of science and technology, it is gradually realizing the mechanization of agricultural production; agricultural production has been transformed into digitization and information, and mechanization in the production process has been developing toward automation and intellectualization. We urgently need to further research on related theory and technology of harvesting robot to achieve harvesting mechanization and automation of operation, reduce the labor intensity of operators, the liberation of more agricultural labor force, compression agricultural production cost, satisfy the requirement of harvesting timely operation, and reduce the economic loss. And the technical research on harvesting robots is conducive to the advancement of China’s agricultural science and technology; it is of great historic significance to accelerate the process of agricultural modernization.

Development history and framework

Development history

In the 1960s, Americans Schertz and Brown first proposed to use machines to harvest citrus fruits, ^1,2 namely the rudiment of harvesting robot; since then, various types of fruit and vegetable harvesting robot technology research are conducted widely. In 1985, France pioneered the research on apple harvesting robots, and in 1998, they developed a prototype with two robotic arms. ^3,4 Early recognition of various harvesting robot prototypes for the target fruit and vegetable recognition rate and harvesting efficiency is relatively low, while the apple’s average recognition rate is about 80% and harvesting speed is roughly tens of seconds per piece. Due to the limited technical conditions at the time, research on apple harvesting robots was at a low tide.

With the development of computer technology, control theory, information technology, and other disciplines, the research of various harvesting robots has set off a new research craze. The research on harvesting robots for fruits and vegetables in China started late. In recent years, under the support of the national “863” science and technology project and the National Natural Science Fund, a number of domestic institutions have successively carried out research on various fruit and vegetable harvesting robots, have achieved preliminary results, and have developed some prototypes.

In terms of apple harvesting robots, Belgian scholars developed the autonomous fruit picking machine harvesting robot in 2008 and it reflects the current apple harvesting robot technology development level in the world, its harvesting rate for apples with a diameter of 6–11 cm is about 80%, and the harvesting time is about 9 s. ⁵ In 2009, the apple harvest robot was jointly developed by China Agricultural Machinery Academy and Jiangsu University; the prototype in the laboratory conditions had a harvest rate at about 80% and the harvesting speed was about 15 s/piece. ⁶ The harvesting speed represents the current domestic apple harvest robot development level.

Research directions

So far, many robot prototypes have been developed at home and abroad; due to the low harvesting efficiency, harvesting speed is much less than the artificial, so it still stays in the laboratory research stage. To make the apple harvesting robot reach the practical level, it is crucial to increase its harvesting efficiency, including harvesting rate and harvesting cycle.

Apple harvesting robots work in the nonstructural environment of the orchard, which usually requires the use of machine vision for environmental perception. To improve the harvesting efficiency of robot, on the one hand, and to improve the performance of harvesting robots, the information processing methods and control methods of harvesting robots are improved to achieve accurate and rapid recognition of the target fruit, which provides theoretical support for improving the performance of harvesting robots. On the other hand, to prolong the operation time of apple harvesting robot as the goal, the nighttime operation mode of the apple harvesting robot is studied to provide a theoretical basis and technical support for realizing apple’s all-weather machine harvesting.

Because the growth environment of harvesting objects changes with time and space, the operating environment of robot is unstructured and uncertain, and it needs to adapt to multiple environments. If the texture, shape, and so on of the fruit to be harvested up differ greatly, the harvesting robot needs to be “carefully” handled to avoid fruit damage. At the same time, harvesting robot needs to have good versatility and programmability but also need to consider the cost.

Basic architecture

Apple harvesting robots generally consist of the following four parts: the manipulator (including the mechanical arm and the end actuator), the visual system, the control system, and the walking device.

The visual system is equivalent to the eyes of human beings to identify and locate the objects and obstacles. The robot arm is equivalent to the human arm, and the end effector is regarded as a hand for harvesting the target object. The robot arm is regarded as an arm and is used to guide the end effector to avoid the obstacle and harvest up and store the target fruit. The control system is equivalent to the human brain and nerve center to achieve overall control of the robot. The walking device is used to carry a robot close to fruit trees or vegetable plants, which is equivalent to a human leg. The harvesting efficiency of the harvesting robot is closely related to the working performance of the above components. Figure 1 shows the architecture of the apple harvesting robot conceived by the research team.

Figure 1.

Overall architecture of apple harvesting robot. 1: control cabinet power box; 2: roads; 3: upper arm; 4: electric motor of upper arm; 5: electric motor of waist; 6: waist; 7: lifting platform; 8: electric motor of forearm; 9: pneumatic draw stem; 10: collecting device; 11: end effector; 12: flexible belt; 13: collection basket of fruit and vegetable; 14: tracked mobile robot.

Operation process

In general, the robot harvesting operation process is as follows: the harvesting robot approaches the plant through the walking device, the robot’s vision system identifies and locates the target fruit, the robotic arm guides the end effector to avoid the obstacle and approaches the harvesting target, the end effector harvests the target fruit, and the robot arm and the end effector store the harvested fruit. Figure 2 shows the control system of apple harvesting robot.

Figure 2.

Control system of apple harvesting robot.

In the whole process of harvesting, the accuracy and speed of target fruit recognition and positioning become the bottleneck restricting the harvesting efficiency of the robot, which is mainly manifested by the amount of time taken for the recognition of the target fruit when the leaves are covered and overlapped, low recognition accuracy, and so on, and it is also more difficult to recognize the fruit during night time.

Overview of apple harvesting robot

Since the French started researching apple harvesting robots, with the gradual development of apple planting and the scientific management of production and with the help of information technology, many scholars at home and abroad related to apple harvesting robot set off a new upsurge in the study and made greater progress.

The apple harvesting robot developed by d’Esnon adopts a three degree of freedom (DOF) coordinate robot, ³ two direct motion joints, and a rotary joint, so that the end effector can be rotated to a certain angle to harvest the target fruit. The actuator uses a plurality of flexible fingers, which do not need to consider the direction of the fruit pedicle, and far-infrared filters are used to assist target fruit recognition. Korean researchers have developed an apple harvesting robot ⁷ that uses a four-DOF manipulator, three rotary joints, and one mobile joint; uses a three-finger gripper as an end effector; and has a pressure sensor at the palm of the hand, which helps to avoid possible damage to the fruit. Using a charge-coupled device camera and photoelectric sensor to identify the target fruit, the detection rate of the target fruit on the outer layer of the canopy is about 85%, and a fruit collection bag is placed under the end effector to shorten the time taken for the fruit harvesting. The time of placing is reduced, and then increase the harvesting speed, the robot does not propose a method of identifying the shaded fruits of the branches and leaves, and during the harvesting process, it is impossible to bypass the obstacles to harvest up the apples.

The apple harvesting robot developed by Belgium ⁵ adopts a six-DOF robot and can move horizontally and vertically to expand the work scope. To reduce the damage to the fruit, a flexible end effector is adopted and pneumatically used to improve the capture speed and the camera is installed in eye-in-hand mode, and the distance between the end effector and the fruit is calculated by triangulation. However, the robot is hauled by the tractor, and the bulk is a bit bulky and the harvesting operation is not flexible.

Gu designed an intelligent fruit harvesting robot ⁸ and conducted an on-site comprehensive experiment in the Feng County, Jiangsu Province. The harvesting robot can complete operations such as autonomous navigation, autonomous harvesting, and independent loading, and intelligent coordinated control can be realized between the mobile platform, robot arm, and end effector. The system performance is stable, the recognition rate for mature fruits is 81.73%, the harvesting rate is 86.92%, and harvesting period is 9.5 s. Zhang designed a manipulator for the apple harvesting robot and studied the control stability of the manipulator. ⁹ The control area was divided into stretching area and harvesting area to ensure that the manipulator could smoothly reach the harvesting. To determine the area and controllability of the system, the experiment shows that after the robot reaches the harvesting area, the required stabilization time is less than 2 s. Feng introduced the laser technology to the apple harvesting robot and designed a laser vision system to measure the distance to the target. The three-dimensional scan of the target scene can be used to analyze the spatial geometry and the relationship between the fruits and branches. ¹⁰ Bao has developed a pneumatic flexible apple harvesting robot end effector that uses a pneumatic flexible actuator as a curved joint. The output force of the end effector is relatively large and it can firmly grasp the target fruit and has a good flexibility. ¹¹ In an apple harvesting robot developed by Zhao, the robotic manipulator is a 5° series joint with integral lifting and telescopic function of the arm and grasped under laboratory conditions. The success rate is 80% and the harvesting period is 15 s. The robot prototype is shown in Figure 3. ⁶

Figure 3.

Prototype of apple harvesting robot.

With the efforts of many scholars at home and abroad, the research on the apple harvesting robot has made a great progress and a variety of apple harvesting robot prototypes have been developed. However, due to the complex and unstructured environment of the apple harvesting robot operation orchard environment and due to the low harvesting efficiency, the current harvesting robot still stays in the laboratory, and it is still some distance away from the orchard site operation and commercialization. The main information of the apple harvesting robots is presented in Table 1.

Table 1.

Main information of apple harvesting robot.

Literature	Country	DOF	End actuator	Harvesting rate (%)	Harvesting cycle (s)
5	Belgium	6	Flexible gripper	80	10–15
6	China	5	Two pieces gripper	80	9–15
3	France	3	Multiple flexible fingers	80	9–15
7	Korea	4	Three fingers gripper	80	10–15

DOF: degree of freedom.

Precise recognition for target fruit

The target of the apple harvesting robot and the recognition of the obstacles provide the status information of the harvesting operation to the visual servo system. The accuracy and real-time performance of robots directly affect the performance of the servo system, which, in turn, restricts the harvesting efficiency. The core of the research is to accurately identify the fruits of the target, break through the technological bottlenecks, and improve its harvesting efficiency. The recognition accuracy of the target fruit, the ability to accurately harvest, and the visual servo control performance largely constrain the harvesting efficiency of harvesting robots. ¹²

Here, we discuss from the perspective of monocular, binocular vision system, overlapping, and oscillating fruits.

Monocular vision system recognition

Monocular recognition system research is relatively mature and has achieved good application effects in many fields. It is simple and easy to use, and in the recognition process of the target fruit and obstacles, the monocular color vision function is mostly used, and the division of color images is premised. However, its biggest drawback is that it is difficult to obtain the depth information of the target fruit in the application, and it is difficult to locate the fruit.

In the color image-based segmentation method, the Otsu automatic threshold method and the image feature segmentation algorithm based on color feature clustering are better in real-time performance. ¹³ Ma combined a quantum genetic algorithm (GA) and a fuzzy neural network to establish a combined target fruit recognition algorithm. The new algorithm has a global search and self-adaptive ability and the recognition rate of this algorithm for unequal color apple fruits can reach 100%, and the illumination of uneven fruit recognition rate is 96.86% and took 1.72 s. ¹⁴ Kelman determined the edge of the apple by the convexity detection of the image of the fruit tree and used the least-squares constraint mechanism to perform 3-D modeling. The positioning accuracy of the overlapping apples was 94%, but the processing time required about 2 min, poor real-time performance. ¹⁵ Wachs used the maximum mutual information of infrared image and color image to extract the visual features of high and low levels, thereby detecting “green” apples in the foreground of the fruit tree crown, and the recognition rates of the two features can reach 54% and 74%, as shown in Figure 4. ¹⁶

Figure 4.

The method for apple detection.

Depth information acquisition

The acquisition of depth information for fruits and obstacles is more complicated, mainly through the active ranging method, such as Feng Juan people, respectively, using laser vision system and based on the optical detection technology combination of depth camera and red, green, and blue (RGB) camera capture orchard environment of multisource images, which use the speeded up robust features image registration algorithm, the running time of 1.0305 s. ^10,17 However, the devices are expensive and bulky and are relatively difficult to install in apple harvesting robots. Wang used binocular vision to locate lychees in unstructured different lighting and random occlusion environments. First, the image was repaired using wavelet transform (WT), then segmented by K-means algorithm, and finally tagging was used for matching. The road has increased significantly, and the error is not significant. ¹⁸ This method can be used to refer to apple image positioning in multiple environments.

Some scholars have attempted to use passive ranging methods, such as Rakun used a stereo matching method based on image registration to achieve 3-D reconstruction of green apples. ¹⁹ Liu used the Kinect camera to obtain the original three-dimensional point cloud of fruit trees under different viewing angles, proposed an initial point cloud initial registration algorithm before the most recent iterative algorithm, effectively improved the accuracy and stability of point cloud registration, and constructed the three-dimensional points of fruit trees cloud morphology model. ²⁰ Cai obtained stereoscopic image disparity maps through the normalized cross-correlation method and then extracted the feature points of the branch skeletons by the multiline segment approximation method. Using the 12 prisms at the origin of the space coordinates to construct each segment of the branch module and to generate a virtual fruit tree, the total operation time is required 876 ms. ²¹ However, in this passive range finding method, there are some branches in the process of reconstruction, such as disorder of branch construction or loss of branch information. Tian proposed a fast recognition and location of target fruit based on depth information, gradient vector field is obtained from depth image, the divergent vector is rotated clockwise by 90°, and the vorticity center is apple’s center. In RGB space, the graph-based segmentation is optimized to obtain superpixel regions. Scan the maximum radius of superpixel area where the center of apple is located and fit the target fruit, it is shown in Figure 5. ²² Li used binocular stereo vision technology to obtain the three-dimensional position of the apple and improved the matching accuracy of the two pictures by reducing the search range and the range error was 5%. ²³ However, the actual harvesting of the environment is far more complicated than the experimental conditions, the time consumption will obviously increase, and the real-time performance will also be significantly worse.

Figure 5.

Fast recognition and location of target fruit.

Overlap or shading fruit processing

Apple fruit grows under natural state, posture, and changeful, and because of the problem of the angle of the harvesting robot, the collected apple images are also varied and polymorphic, such as unobstructed, foliage blocked, overlapping, and the like. The processing of overlapping apple images is still a difficult problem, attracting the attention of researchers. Image segmentation is the premise of feature extraction and target recognition; the results have a direct impact on the progress of follow-up work. ^{24 –26} Feature extraction starts from an initial set of measured data and builds derived values (features) intended to be informative and nonredundant, facilitating the subsequent learning and generalization steps and in some cases leading to better human interpretations. ^{27 –30}

Song first used K-means clustering to segment overlapping apple images and processed the morphological processing of the target fruit to extract the continuous smooth contour curve of the convex hull of the target fruit edge. The center and radius of the contour curve were calculated to identify and locate overlapping fruits. Then, using the Spline interpolation method to reconstruct the occlusion target, the average positioning error of the target fruit obtained by this method is about 14.15%, and the average target coincidence is 85.08%. ³¹ Jia proposed an apple recognition based on pulse-coupled neural network (PCNN) and GA-Elman algorithm, the recognition rate of overlapping fruit and obscured fruit can reach up to 88.67% and 93.64%, and the total recognition rate reaches up to 94.88%. ³² Xu proposed an overlapped apple target segmentation method combining the Snake model and corner detection and segmented 20 overlapping apples with an average error of 6.41. ³³ Wang first used the K-means cluster segmentation method to extract the target fruit region and then used the Ncut algorithm to extract the outline of the target fruit. Finally, the contour was reconstructed by the Spline interpolation method. Experiments were performed on 20 overlapping fruit images to calculate the segmentation error, and the mean value of coincidence was 5.24% and 93.81%. ³⁴ Zhao used an extremum method to achieve overlapping fruit positioning algorithm. Firstly, the target fruit area was segmented by color difference method or K-means method and then hole filling and contour extraction were performed. Finally, the minimum distance from the inner point to the outline edge was calculated. The maximum value is determined to determine the center of the circle and the radius to complete the recognition and positioning of the target fruit. ³⁵ Barnea conducted a related study on the recognition of the fruit of the same color under unstable light, and the recognition of the shaded fruits, and proposed a 3-D detection model based on RGB color features and related shape features. ³⁶ This model can be used as a reference for studying the same color apple images.

The mutual overlapping and blocking of the target fruit directly affect the recognition effect and the positioning accuracy of the target fruit to a certain extent. The above method has a better treatment effect on the fruits that are slightly overlapped or shaded. However, there is still no good solution to the large overlapping area of the fruit and how to improve the recognition rate and positioning accuracy of severely overlapped fruits still needs further study.

Dynamic fruit processing

Most of the above research are based on the recognition of static apple images, and in the actual operation process, due to disturbances such as outside the wind or mechanical collision will cause the oscillation of target fruit, thereby affecting the target recognition of fruit. Therefore, the environmental information needs to be updated in time, and it is urgently needed to conduct research on the recognition of dynamic targets and environments.

Lv used the correlation information between the images to narrow the area to be processed and used the optimized template matching algorithm to track and identify the image of the subsequent frame. Finally, the target fruit was identified and the average processing time was 0.74 s, and the recognition time was reduced by 36%. ³⁷ For the oscillating fruits, fast Fourier transform modeling was performed by extracting the two-dimensional centroid coordinates, the oscillation period and the depth distance of the oscillating fruit were calculated, and the stroke speed of the direct harvesting joint of the apple harvesting robot was calculated, and the harvesting success rate was 84%. ³⁸ Zhao performed a prejudgment of the motion path of the robot to reduce the scope of the next image processing and used the normalized cross-correlation fast matching algorithm to identify overlapping apple images. The matching time was 0.133 s, which was 28.1% less than that without prediction. ³⁹

In addition, Liu studied the problem of vibration in the harvesting process of the grape harvesting robot, proposed a simulation model of the spike vibration for the input of the cob axis, and used this model to analyze the effect of different excitation methods and different harvesting stages on the vibration of the ear. ⁴⁰ This also provides ideas for studying the oscillation of apples, and it can be used as a reference for further verification.

The recognition methods of these dynamic goals have improved the recognition speed under the premise of ensuring the recognition accuracy, but there is still a long distance from the requirement of fast visual servo. But it is worth noting that in recent years, the rapid recognition of dynamic targets and other fields of matching technology have developed rapidly, which provides a good reference for the recognition of dynamic fruits. The recent literatures about real-life applications of contemporary automation techniques in different fields give us new inspiration, using other intelligent computation ^{41 –43} to optimize neural network, to get more superior model; at the same time, we try to use deep learning algorithm ^{44 –47} to recognize the apple target fruit.

Night operation

Another way to increase the harvesting robot’s harvesting efficiency is to try its all-weather operation, which is to achieve a nighttime environment. Harvesting robots is a highly automated work machine, so the harvesting robot’s working time can be extended to maximize its effectiveness. Night work is affected by factors such as ambient light, temperature, and humidity, which makes it difficult to acquire and process images. Therefore, the basic premise of the nighttime study of apple harvesting robots is how to obtain accurate night image information. At present, there are relatively few domestic and foreign studies on night images of orchards, mainly involving the configuration of auxiliary lighting devices, night image preprocessing, night image target recognition, and so on. However, some progress has been made in this area in recent years.

Supplementary light

To adapt to the change of ambient light, the vision system of the fruit and vegetable harvesting robot under natural environment needs some accessories to assist in addition to the main image sensors. These auxiliary devices mainly include artificial light sources and filters. The use of an artificial light source can reduce the effect of light illumination on the recognition of the target fruit, and a filter is used to ensure that the sensitive wavelength of the image sensor can pass through the filter and be incident on the image sensor while filtering out incident light in other wavelengths.

Van Henten used a xenon lamp to eliminate ambient light interference when studying cucumber harvesting robots. In addition, the two cameras used 850 nm and 950 nm filters to obtain near-infrared images. ⁴⁸ Guo studied the visual system of the strawberry harvesting robot and used four fluorescent lamps for auxiliary lighting to minimize the influence of ambient light. ⁴⁹ Hayashi studied the strawberry harvesting robot and supplemented the light in the visual system, and the light source used five sets of light sources composed of 120 light-emitting diodes (LEDs). In addition, linear polarization filters were used in both the auxiliary light source and the camera to improve the collected image quality. ⁵⁰ Xiang proposed a lighting system based on the ratio between the category-to-class variance and the intraclass variance based on the foreground and background as an evaluation index, with the aid of the orthogonal experiment, the design with tomato harvesting robot lighting system, and the optimal combination for the fluorescent lamp and diagonal layout. ⁵¹

These applications have provided inspiration for the illumination design of apple harvesting robots at night. Jia used incandescent lamps, fluorescent lamps, and LEDs to assist in obtaining three different artificial light sources to obtain images of apples at night and launched a series of studies on them, eventually screening incandescent lamps as an auxiliary light source for apple harvesting robots. ⁵²

These studies have achieved very good results in increasing the image quality by adding auxiliary equipment under low-light conditions. The research is worthy of further reference and verification in the future research of apple harvesting robots to obtain more suitable auxiliary lighting equipment. In addition, we should consider how the auxiliary equipment is arranged on the harvesting robot.

Night image enhancement

Due to the effects of ambient light, temperature, humidity, and other factors, nighttime image acquisition is easy to be disturbed by noise, so it is necessary to reduce the noise of nighttime images. At present, there is no effective method to determine the type of noise contained in night images, which also makes it difficult to reduce the noise. However, some literatures ^{53 –55} believe that noise contained in night images is based on Gaussian noise.

At present, the removal of Gaussian noise is still a difficult problem in image processing. Under the efforts of many scholars, related research on noise reduction of Gaussian noise have been carried out. ⁵⁶ Yan proposed a local evolution based on clustering and genetic programming, the adaptive filtering method removes Gaussian and salt-and-pepper noise in natural images, and the resulting image quality is significantly improved. ⁵⁷ Talebi proposed a global filtering image denoise method, in which each pixel in the image was estimated, making various noise reduction filters efficiently globally optimized. ⁵⁸ These studies all give inspiration to the research of noise reduction in night image Gaussian noise. Some scholars have conducted related research on night image denoising. Peng proposed a night image denoising method combining mathematical morphology and median filtering for nighttime image noise in fishery monitoring. ⁵⁹ Ji proposed a Retinex image enhancement algorithm with edge-preserving characteristics based on guided filtering for nighttime image quality, which has improved the nighttime image enhancement and runtime efficiency. ⁶⁰ Yang proposed an image double-platform histogram enhancement algorithm under nighttime low illumination conditions. ⁶¹ Jia used apple harvesting robot as the research object and used the differential image method to determine the characteristics of nighttime images. He proposed a series of methods for denoising the night image of apple ⁶² and used WT and independence. Independent component analysis (ICA) introduces night image noise reduction and proposes the use of fuzzy theory, fusion algorithm, and particle swarm optimization (PSO) to improve the noise reduction performance of the algorithm. The noise reduction effect of night images and the concept of relative peak signal-to-noise ratio are also proposed to objectively evaluate the noise reduction ability of each algorithm.

However, in practical applications, the removal of Gaussian noise and the processing of signal details are not ideal. For example, the signal-to-noise ratio of the low-noise image obtained is still low, the loss of texture information is relatively large, and the edge information is blurred.

Image recognition in low illumination

Night image recognition is relatively early in other fields such as intelligent transportation and license plate recognition ⁶³ and has also achieved relatively satisfactory results. At present, there are relatively few domestic and foreign research on night images for agriculture. But with the efforts of many scholars, some progress has been made.

Payne used man-made light sources to collect nighttime images of near-mature mangoes in mango plantations and then extracted color features and shape texture features of YCbCr color space to identify the number of mangoes and estimate mango production. ^64,65 Font used the artificial light source to collect the night images of ripe grapes in the grape orchard and converted them to the RGB color space. The number of grapes was determined by calculating the spherical reflection peaks on the grape surface. ⁶⁶ For apple’s nighttime images, Zhang equipped a color camera with a ring flash for image acquisition at night, by analyzing the characteristics of green apples and near-color backgrounds, a classifier was designed based on the color characteristics to achieve a green background. The recognition rate of apple in the night environment can reach 89.3%; the algorithm takes 3.14 s on average. ⁶⁷ Zhao used the improved RG color difference segmentation method to repair the bright reflective regions in apple night vision images using a quadratic segmentation method and proposed a method for apple night image recognition, and the correct recognition rate of the algorithm reached 83.7%. ⁶⁸ Fu used the RG color model to segment the nighttime images of the collected kiwifruit and then detected the target fruit using the Canny operator; the experimental results showed that the optimal illumination of the artificial light source was 30–50 lx and the recognition rate was 88.3%. ⁶⁹ Liu performed ordinary least squares (OLS) space fitting on the main scene samples of the night image in the RGB space, then established a threshold segmentation model based on the distance from the sample point to the straight line to segment the image, and finally used the same methods correct the shaded area around the apple, and the recognition rate can reach 90%. ⁷⁰

In general, the work research base in orchard nighttime environment is relatively weak, and the recognition accuracy and speed are still far from meeting the requirements for harvesting robots on-site orchard operations at night. To realize the apple harvesting robot all-weather operation and maximize its performance, it is necessary to make in-depth research on the accurate recognition of apple’s night image.

Intelligent computing in harvesting robots

Intelligent computing is borrowed from the inspiration of the laws of the natural world, described by a specific mathematical model, and designed algorithms for problem-solving. It focuses on the imitation of human or other biological intelligence behaviors and draws on useful laws and principles. Compared with traditional calculation methods, intelligent computing has parallelism, self-adaptive ability and self-learning ability, and so on. It can solve many problems without too much domain knowledge, especially for some large-scale complex problems.

The emerging intelligent computing methods are introduced into the emerging field of harvesting robots, and advanced intelligent algorithms are used to optimize the target recognition or intelligent control involved in harvesting robots to increase the operating efficiency of harvesting robots. Although other relevant fields have achieved gratifying results, their theories can be used as reference for harvesting robots to advance the theory of harvesting robots. But direct research in this area is relatively less, and in the research that has been carried out and made great progress, mainly focusing on harvesting path planning and target fruit recognition.

Path planning

Harvesting robots according to the perception of the environment plan a safe operation route to complete the harvesting task efficiently, including obstacle avoidance, harvesting order, walking, and so on. With advanced intelligent computing theory, path planning is more reasonable and efficient. ^71,72

In the research of harvesting path planning, Yuan applied the intelligent algorithm to the path planning of harvesting robots. Firstly, the path planning problem in apple harvesting process was transformed into a three-dimensional traveling salesman problem and then solved, and the finite field information was constructed. The ant colony optimization (ACO) adaptive update is improved to obtain better optimization results. ⁷³ Plebe viewed the path planning problem of fruit harvesting robots with dual independent robotic arms as a twin traveling salesman problem and optimized it with a self-organizing map model to avoid tangling or collision of the twins. The experimental results show that after optimization, a reliable harvesting sequence plan is obtained. ⁷⁴

Yang proposed a method of constructing kinematics equations for hybrid harvesting robots based on spin theory and concluded that the maximum absolute error of the end effector in each coordinate axis in the basic coordinate system was greatly reduced to meet the harvesting requirements and provided reference for control strategy and trajectory planning study of hybrid harvesting machines. ⁷⁵

Target recognition

In the target fruit recognition study, Gongal reviewed the detection and location of the target fruit of harvesting robots, which involved the role of intelligent computing in the recognition process. ⁷⁶ Wang proposed an adaptive PCNN algorithm for the segmentation of cucumber images to solve the problem of near-color system segmentation for near-colored fruits and vegetables. ⁷⁷ Ma proposed an apple fruit recognition algorithm based on the combination of quantum genetic neural network and Fuzzy neural network, aiming at the ambiguity and uncertainty of the color pixels in apple color image boundary pixels, and the established recognition model has high-speed stability and good robustness. ¹⁴ Jia proposed an ICA noise reduction algorithm (PSO-ICA) based on PSO for the noise of apple images at night, which was used to reduce the noise of apple night images; the speed of noise reduction algorithm was improved through intelligent optimization and noise reduction capability. ⁶² Jia used radial basis function (RBF) for the recognition of apple images and optimized it by using GA and least mean square (LMS) algorithm. RBF-LMS optimizes neural network algorithms to improve recognition accuracy and operating efficiency of neural network recognition models. ⁷⁸ The segmentation accuracy of apple images affected by different noises was above 98% and the robustness of the algorithm is enhanced, built a fast and accurate target fruit detection system using deep neural networks to provide technical support for estimating production and harvesting. ⁷⁹ Zhou used color features in both RGB and hue, saturation, and intensity space for apple detection (Figure 6). The difference between RGB color channels was used as a measure to segment red and green apples from the background. Further, a threshold value in saturation channel was also used to segment red apples. ⁸⁰

Figure 6.

Flowchart explaining different steps of fruit segmentation.

The application of intelligent computing ^{43,81 –84} has injected fresh blood into the research of harvesting robots, although the research in this area is still too little, its superior performance has been highlighted, and the prelude to the research on harvesting robots based on intelligent optimization has been opened. From the above research also can be seen, intelligent computation can be directly applied to harvest some algorithm of the robot, as a tool to optimize the existing algorithm, with the ultimate goal is to improve the harvesting efficiency of harvesting robot.

Fruit and vegetable harvesting robot

Since the 80s of the last century, based on the rapid development of digital image processing technology and artificial intelligence theory, fruit and vegetable harvesting robots have opened a new chapter. With the further promotion of information technology, the advancement of the level of agricultural equipment has been advanced and the agricultural robot technology has reached a new height. Its role has gradually emerged in the modern agricultural production process. ^{85 –87} Many kinds of prototypes have been successfully developed under the efforts of many scholars at home and abroad. In a few developed countries, some fruit and vegetable harvesting robots have been widely used in the greenhouse. ^{88 –90}

Some research foundations and key technologies of these fruit and vegetable harvesting robots can be applied in many fields, and they can also provide references for the research of apple harvesting robots. The following focuses on several relatively mature fruit and vegetable harvesting robots.

Same color fruit recognition

There are many varieties of existing apples, and the colors of mature fruits are also different, such as red Fuji, yellow gold handsome, green apple, and so on. At present, the research of apple harvesting robots is mostly based on Fuji harvesting targets and the research on green fruits is relatively rare, because the fruit color is similar to the color of the background leaves, it is difficult to identify the target fruit and the recognition efficiency is relatively low. This type of fruit is similar to the color of the foliage, we call it the same color. Fruits of the same color are difficult to identify from a purely color perspective and need to be treated differently or try other characteristics.

Therefore, Zhang equipped a color camera with a ring flash for image acquisition of apple trees. By analyzing the characteristics of the green apple and the near-color background, a classifier was designed based on the color characteristics to achieve the recognition of green apples in a near-color background. With a recognition rate of 89.3%, the algorithm takes an average of 3.14 s. ⁶⁷ In addition, Barnea conducted a related study on the recognition of fruit of the same color under unstable light and the recognition of the shaded fruits proposed a 3-D detection model based on RGB color features and related shape features. ³⁶ Both of these methods propose corresponding processing in terms of color features to distinguish the target fruit from the background.

For the research on the recognition of the same color fruits, other harvesting robots have accumulated certain research results for us to learn from. For example, Van Henten developed a 7°-free cucumber harvesting robot prototype, ⁹¹ using multispectrum to identify cucumbers, and the recognition rate can reach 80%. Lee used a spectrometer to measure the spectral reflectance of green-skin citrus and proposed a method for recognizing green-skin citrus, with a recognition rate of 84.5%. ⁹² Both of these models use different spectral reflectance of the target fruit and background to achieve the target fruit recognition, which can be used as a reference for studying the same-color apple images.

The recognition of fruit of the same color is a difficult problem; most research are based on differences in color characteristics or spectral reflectance. Although they have achieved certain results, they are all applied to the recognition of specific fruits and vegetables, and the essential characteristics of fruit need to go further study. In addition, infrared light can be used to treat the same color of fruit.

Tomato harvesting robot

The main problems existing in the current tomato harvesting robots are the recognition of mature fruits that are severely obstructed by stems and leaves is still difficult, and the obstacle avoidance of robot arm is also a problem, the grasping force of the end effector and the accurate positioning of the fruit, and so on. Figure 7 shows a tomato harvesting robot prototype.

Figure 7.

Tomato harvesting robot.

The research of tomato robot is represented by Japan; the robot prototype researched by Kondo can harvest up bunches of tomatoes and use photoelectric sensors to locate the stems. To avoid obstacles as much as possible, it uses a seven-DOF machine arm, the harvesting cycle of the prototype is about 15 s, and the harvesting success rate is about 70%. ⁹³ Japan’s Matsushita Corporation developed a tomato harvesting robot equipped with a self-produced image sensor in 2015 to achieve unattended harvesting of tomatoes ⁹⁴ ; it detects red ripe tomatoes through an image sensor and then precisely positions their shape and position. When harvesting, the robot pulls the pedicle part without damaging the fruit. After the harvesting basket is full, wireless communication technology will notify the robot to automatically replace the empty basket. In addition, data management can be conducted on the yield and quality of tomatoes and a harvesting plan can be formulated. The prototype harvesting cycle currently under development is about 20 s, and Panasonic will further improve the sensor performance and the target harvesting speed is 6 s.

Li used the vision system to obtain the geometric characteristics of the tomato in the maximum longitudinal and equatorial contours of the tomato and estimated the quality of the tomato and the grip force exerted by the end effector to predict the grab position to reduce the machinery during the harvesting process damage. ⁹⁵ Xiang used binocular stereo vision technology to explore algorithms for segmentation, recognition, and localization of tomato images and solved the difficulties in the recognition of foliage and clustered tomatoes. ⁹⁶ Zhou designed a multijoint, five-DOF robotic arm and conducted a modeling of its operation. ⁹⁷ In addition, there are many scholars at home and abroad who have been working on the research of tomato harvesting robots and have made gratifying achievements.

Cucumber harvesting robot

The difficulty in the current cucumber harvesting robot research is that cucumber is a near-color vegetable, and the growth of cucumber stems and leaves is relatively strong, and the cover of cucumber is relatively severe, and there is a certain degree of difficulty in the process of recognition. Influenced by the shape of cucumber, it also causes the positioning and grasping accuracy of the manipulator to be low. Figure 8 shows a greenhouse cucumber harvesting robot developed by a Dutch scholar.

Figure 8.

Cucumber harvesting robot.

Japan and the Netherlands were the first countries to study cucumber harvesting robots. The cucumber harvesting robot prototype developed by Kondo in Japan identified cucumbers, stems, and leaves based on the reflectance of infrared waves, and the harvesting cycle was about 16 s and the success rate was about 60%. ⁹⁸ Dutch van Henten developed a seven-DOF cucumber harvesting robot prototype, the harvesting cycle was about 45 s, but the recognition rate was 80%, and cucumbers were identified using multiple spectra. ⁹¹

The cucumber harvesting robot developed by the Li proposed a three-layer control system. The harvesting cycle is about 28 s, and the success rate is about 85%. The cucumber technology is identified by the use of spectral technology, which lays a foundation for the recognition of near-color fruits and vegetables. ⁹⁹ Ji introduced PCNN into cucumber image segmentation to solve the problem that the near-color system is difficult to segment based on gray scale. ⁷⁷ Classifier pair is designed using least squares support vector machine from which cucumber was identified and the recognition rate was 82.9%. In addition, scholars at many domestic and foreign scientific research institutions have been working on the research of cucumber harvesting robots and have made gratifying achievements.

Strawberry harvesting robot

The three-DOF strawberry harvesting robot developed by Hayashi and others in Japan first judges the ripeness of the strawberry by machine vision and then three-dimensionally locates the mature fruit and adds a reflective photoelectric sensor to the end effector for better strawberry recognition, the success rate of around 55%. ⁵⁰

Zhang started the research on strawberry harvesting robots and designed a harvesting robot for elevated cultivation mode. It can realize automatic recognition and positioning of mature strawberries and complete damage-free harvesting, and the extraction success rate is about 88% and harvesting cycle is 18.54 s. ¹⁰⁰ Using the feedback information of the position sensor to reduce the time-consuming positioning, the method based on the motion recovery structure is adopted to improve the accuracy of fruit harvesting parameters. ¹⁰¹ Guo proposed a method suitable for fruit positioning and fruit stem detection in strawberry harvesting robots. By judging the ripeness of strawberries, harvesting was performed selectively, and the damage rate to fruit during harvesting was greatly reduced. ¹⁰²

In addition, there are many scholars at home and abroad dedicated to the research of strawberry robots. When the strawberry harvesting robot operates, not only the target fruit needs to be identified but also the position of the grabbing point and the cutting point need to be located. Due to the vulnerability of mature strawberries, when the end effector is studied, the gripping force requirement is relatively high. The strawberry fruit is relatively small and easy to be blocked by the foliage, which makes it more difficult to accurately identify the fruit. Figure 9 shows a greenhouse strawberry harvesting robot developed by Japanese scholars.

Figure 9.

Strawberry harvesting robot.

Citrus harvesting robot

Research on citrus harvesting robots began in the United States, Mehta developed a seven-DOF citrus harvesting prototype, the closed-loop control was used for timely feedback, and WO cameras were used, one fixed, one at the end effector center, and tested indoors. The success rate is 95%, but this robot is only suitable for large- and medium-sized varieties, and it has poor harvesting performance for small varieties of citrus. ¹⁰³ Hannan proposed a citrus recognition algorithm based on machine vision, using the distribution of pixel histograms of fruit and canopy samples, segmenting the image in the red channel, and using the detection algorithm based on the centroid and diameter to complete the fruit recognition and the success rate was over 90%. ¹⁰⁴ Lee constructed a citrus fruit recognition system to compare the brightness and tone distribution of fruit samples in different color spaces and to obtain citrus fruit regions after segmentation, expansion, and erosion. ¹⁰⁵ Spectrometer was used to measure the spectral reflectance of green-skin citrus, a method for recognizing green-skinned citrus, with a recognition rate of 84.5%. ¹⁰⁶

Cai proposed a citrus fruit recognition algorithm based on vision system and support vector machine (SVM) classifier, with a recognition rate of 92.4%. ¹⁰⁷ Using spectral image technology and significance analysis of microarrays algorithm to identify mature citrus, the recognition accuracy was 96%. ¹⁰⁸ Zou proposed a fuzzy clustering segmentation algorithm based on multiscale visual improvement, with a segmentation rate of 93.68%, which provides a theoretical basis for the recognition of the next step. ⁶³ The above team and other scholars also devoted themselves to the research related to citrus harvesting robots and obtained a series of gratifying results, which advance the technology of citrus harvesting robots.

Prospects

Nowadays, the developing trend toward intelligence and information reveals that the technology has penetrated into many fields under the background of approaching intelligence, and the research of fruit and vegetable harvesting robot will also benefit. This article reviews the latest research progress of apple harvesting robots from various aspects; it not only sees the gratifying achievements but also recognizes the deficiencies, and there is room for improvement. Finally, several fruit and vegetable harvesting robots with relatively mature research are briefly outlined to provide technical references for the research of apple harvesting robots.

The technological improvement of apple harvesting robots still focuses on the following aspects: Ontology design should tend to be light and flexible and to get rid of bulky mechanical constraints as much as possible. The fast, precise recognition and localization of the target fruit can overcome the bottleneck of low efficiency of visual system operation. Reasonable and feasible path planning can effectively avoid the processing of redundant information and avoid harvesting obstacles. The improvement of control strategy increases the operating efficiency of the robot. Overall, the biggest problem in the research of apple harvesting robots is still the low harvesting efficiency, despite the many efforts made by many scholars, gratifying achievements have been made in various aspects, apple harvesting robot research a big step forward. However, it is still a long way to achieve orchard operations and even commercialization, but it is believed that the road will become more and more smooth on the basis of the cooperation of scholars.