Abstract
The excellent performance of fruit and vegetable picking robots is usually contributed by the reasonable structure of end-effector and recognition–localization methods with high accuracy. As a result, efforts are focused on two aspects, and diverse structures of end-effector, target recognition methods as well as their combinations are yielded continuously. A good understanding for the working principle, advantages, limitations, and the adaptability in respective fields is helpful to design picking robots. Therefore, depending on different grasping ways, separating methods, structures, materials, and driving modes, main characteristics existing in traditional schemes will be depicted firstly. According to technical routes, advantages, potential applications, and challenges, underactuated manipulators and soft manipulators representing future development are then summarized systematically. Secondly, partial recognition and localization methods are also demonstrated. Specifically, current recognition manners adopting the single-feature, multi-feature fusion and deep learning are explained in view of their advantages, limitations, and successful instances. In the field of 3D localization, active vision based on the structured light, laser scanning, time of flight, and radar is reflected through the respective applications. Also, another 3D localization method called passive vision is also evaluated by advantages, limitations, the degree of automation, reconstruction effects, and the application scenario, such as monocular vision, binocular vision, and multiocular vision. Finally portrayed from structural development, recognition, and localization methods, it is possible to develop future end-effectors for fruit and vegetable picking robots with superior characteristics containing the less driving element, rigid–flexible–bionic coupling soft manipulators, simple control program, high efficiency, low damage, low cost, high versatility, and high recognition accuracy in all-season picking tasks.
Introduction
As a crucial role of the agricultural production chain, fruit and vegetable picking usually depends on a lot of manpower, material resources, and time due to the strong seasonality, high labor intensity, and high cost. 1 Following the development of economy, the rural labor force is gradually transferred to other sectors of society in some developing countries, and the world is experiencing the problem of an aging population. The shortage of labor force will become a fact. 2 Therefore, traditional picking methods would not meet requirements of modern agricultural production. Owing to improvements in labor productivity, picking efficiency, cost, and so on, picking robots have been widely adopted in fields of fruit and vegetable productions. 3 In order to develop picking robots in agricultural fields, international scholars or engineers have proposed their schemes, thus yielding so many picking robots and end-effectors in respective fields such as the strawberry, 4 –6 citrus, 7 –9 sweet orange, 10 –12 apple, 13 –18 kiwi fruit, 19 –21 tomato, 22 –28 eggplant, 29 –31 and cucumber 32 –35 .
Especially working as the terminal element in the direct contact with fruits and vegetables, the end-effector is decisive to obtain the excellent performance of a picking robot relying on the universality, practicability, picking efficiency, and intact rate. 36 For satisfying requirements of picking tasks, end-effectors with different grasping and separating methods have been extensively utilized in the agricultural field. 37 Of those, three types of end-effectors with the sucking, clamping, and sucking-clamping functions are widely adopted in fields of picking strawberries, 4 apples, 13 –16 tomatoes, 22 –27 and so on, which characterizes a picking way of grabbing. After grasping, separating methods should be considered. Typically serving as main separating components, end-effectors assembling rigidly and flexibly separated elements are adopted to pick the apple, 17,18 kiwi fruit, 19,20 citrus, 9,10and others. 5,28 –30,33
It would report from existing literature that a lot of end-effectors have been successfully used to pick corresponding fruits or vegetables. However, almost applied instances have presented limitations in the picking efficiency, damage rate, flexibility, recognition accuracy, and so on, and these disadvantages are usually caused by two aspects. For example, manipulators should be more flexible and present the complexity at the design stage since fruits and vegetables are easily damaged, scratched, and adhered, especially grasping hardly. The unstructured or semi-structured environments in agriculture are evident. Owing to differences in morphological characteristics of fruits and their growth environments, it is also not easy to identify and locate fruits quickly, thus affecting the picking efficiency. More importantly, future development is not quite clear although so many schemes have been proposed, tested, and applied.
Therefore, structural characteristics and target recognition methods for end-effectors of picking robots will be systematically summarized so that the future direction of structural development and recognition methods matched with them are clearer. The details will be portrayed in the following sections gradually.
Characteristics of end-effectors considering different grasping and separating ways
Besides considering the appearance, structures, and mechanical characteristics of stems of fruits or vegetables, the realization of picking methods is particularly important at the design stage. 38 Specifically, fruits and vegetables should be grabbed by an end-effector firstly, and then a separating component is utilized. 39 For achieving a grabbing function, diverse clamping mechanisms have been designed according to requirements of different fruits and vegetables, thus yielding sucking, clamping, and sucking-clamping types of end-effectors. 37 Also, a suitable separating manner should be considered by the binding force at the junction of the fruit and its stem. 37,40 For example, a flexible mode of separation can be adopted to pick the apple, kiwi fruit, and tomato production due to the relatively small binding force between fruits and their stems. For the relatively large force at the junction, rigid separation methods depending on scissors, rotating blades, and the cutting element of laser are usually utilized in picking operations of the citrus, cucumber, eggplant, and others. 41 Specific methods and their technical characteristics will be presented in the following sections.
Characteristics of applied instances with different grasping methods
End-effectors with a sucking function
Generally generating negative pressures by a vacuum suction cup or a tube, an end-effector will absorb fruits directly. Such a grabbing method yields lots of elements called suction effectors. 37 Owing to the fast-picking speed and a low damage rate, these effectors are widely utilized to pick a single fruit with a spherical shape (hemispherical shape) and lightweight. There are successful instances although their suction force is also limited. For example, Naoshi Kondo et al. in Japan designed a suction effector for picking strawberries, as shown in Figure 1. 4 With the suction force produced by the draft fan, a suction cup directly sucks fruits into the end-effector. However, this effector cannot judge the maturity of strawberries. It is also difficult to adjust the large suction force. Therefore, immature fruits may be inhaled and picked together by such an effector. 42 As portrayed in Figure 2, another suction type of effector was proposed for the citrus in Japan, 8 and fruits can be absorbed and fixed by a pneumatic suction cup. Fruit stems were cut off by rotating the circular cutter for one cycle. Also depending on the suction force of the suction cup, it is suggested to pick the citrus with small sizes and weights. 43 Johan Baeten et al. developed a suction cup type of an end-effector for picking apples in Belgium. 13 As shown in Figure 3, the cup was manufactured by the silica gel, which is beneficial to reduce mechanical damage greatly. Essentially, apples can be sucked by a negative pressure generated by a vacuum pump. After this suction behavior, fruits will be separated from stems through rotating and swinging the mechanical wrist. It is concluded from existing literature that its picking efficiency could be raised to 80%, 13,17 thus satisfying a range from 40% to 86% in a filed of picking fruits. 44 However, it is also easy to suck branches and leaves into the suction cup.
An end-effector for picking strawberries.
An end-effector for picking citrus.
An end-effector for picking apples in Belgium.
Also considering the suction effector, a picking robot for apples was designed by Abundant Robotics in America, as depicted in Figure 4. 14 Evidently, apples will be transported to a collection apparatus through the suction inlet and pipelines if the suction force provided by a picker is large enough. Adopting a vacuum technology rather than the mechanical arm, the picking efficiency of such a robot is higher. 45
An end-effector for picking apples in America.
As shown in Figure 5, an end-effector for the hanging line cultivated tomato was also established by Feng in China. Specifically, the airflow with the negative pressure provided by a vacuum generator can suck the tomato hidden in leaves, 22 and then fruits will be captured into the sleeve as long as the telescopic cylinder extends out of the sleeve easily. Therefore, the achieving rate is up to 83.9% by such a picking method. Unfortunately, it also depends on high requirements for light at the picking stage. The successful rate will be decreased if the light is quite weak. 46
An end-effector for picking tomato in China.
End-effectors with a clamping function
Depending on a gripper, other end-effectors different from suction types are actively researched by international scholars. Usually, the greater clamping weight and the more stable behavior are also achieved if more fingers in a gripper are considered. 37,47 However, the corresponding driving system will be more complicated. Therefore, scholars try their best knowledge to design different fingers according to fruit sizes, shapes, and weights. 48 Of those, Kondo N et al. in Japan developed an end-effector for picking tomatoes with upper and lower fingers, as shown in Figure 6. 23 Upper and lower fingers are opened firstly to wrap the tomato if branches of the tomato cluster are detected. Lower fingers are then used to clamp stems and upper fingers cut them orderly. 49 Davidson et al. in America proposed a picking robot utilizing the end-effectors with three underactuated fingers. 15 As shown in Figure 7, two joints are assembled in each finger, and their artificial tendons provide the power. The effector can pick fruits by imitating behaviors of human hands. Therefore, such an effector not only generates the grasping force on fruits through pulling and swinging actions but also easily separates fruits from their stalks. 50,51
An end-effector for picking tomato designed by Kondo N et al.
An end-effector for picking apples developed by Davidson et al.
As portrayed in Figure 8, a clamping scheme of end-effector for picking apples was also established by Zhao De-an et al. in Jiangsu University of China. 16 In such a clamping structure, two fingers are adopted. Damage to the apple could be avoided by designing an arc surface stuffed by the rubber-sponge material and arranging a pressure sensor in these fingers. 52 An end-effector for picking the kiwi fruit was also considered by Rong Hao in Northwest University of Agriculture and Forestry Science and Technology, as shown in Figure 9. 19 This effector adopts a fruit holder and a stem holder, and the lead screw nut can be moved to the middle position by the driving function of a stepping motor. Therefore, combining these abilities, the fruit and its stem can be clamped well. After fixing the fruit and stem, the movement of the lead screw nut is also achieved by the driving effect of a motor, thus obtaining the forward movement of the stem holder and cutting them. However, the precise positioning function is necessary, and many disadvantages of this effector are also evident, such as the relatively low accuracy and the poor adaptability for grasping.
An end-effector for picking apples designed by Zhao De-an et al.
An end-effector for picking the kiwi fruit proposed by Rong Hao.
End-effectors with the sucking-clamping function
Combining characteristics of above two grasping ways, another type of effector always includes the suction element and the holder. At the picking stage, fruits will be sucked first, and then the holder is adopted to clamp them. 37 –40 Owing to these considerations, diverse effectors have been developed. For example, containing four flexible fingers and a suction cup, an early actuator was proposed by Okayama University in Japan, as depicted in Figure 10. 25 Such an arrangement of structure can absorb fruits into soft fingers, and then these fingers are forced to bend if a certain tension is achieved by pulling the cable, which obtains high accuracy for grasping and also reduces the efficiency. Picking tomatoes, a manipulator with four fingers was developed by PP Ling et al. in Ohio State University, as shown in Figure 11. 26 With a compound vacuum effect, a suction cup is installed in the palm of fingers, thus absorbing fruits by generating the negative pressure of an air pump. Completing the suction, the cable and tendon are driven by a stepping motor so that fruits can be further clamped. After picking, it can be reset by a torque spring easily. Therefore, such an actuator characterizes a good universality and an excellent adaptability to different sizes although its control method is relatively complex. 26
An end-effector for picking the tomato proposed by Okayama University.
An end-effector for picking the tomato proposed by Ohio State University.
Utilizing a vacuum suction element and fingers, a robot for picking tomatoes was designed by Jizhan Liu et al. in Jiangsu University of China, as shown in Figure 12. 27 Absorbing fruits and then pulling them back, this effector can separate the single tomato from its cluster. Ending these actions, the tomato will be clamped well by wrapping fingers. While there are above evident advantages, it is possible to damage them because of the friction (or collision) between fruits and the suction cup. 53 Also, excessive suction may damage surfaces if tomatoes are ripe.
An end-effector for picking tomatoes proposed by Jizhan Liu et al.
Adopting the sucking, clamping, and cutting elements, an end-effector for picking navel oranges was designed by Liming Xu et al. in China Agricultural University, pictured in Figure 13. 10 This end-effector is mainly composed of an adsorption device, a gripping device, and a shearing element. The suction cup will be moved to a position near the citrus by the pushing behavior of the lower linear cylinder, further grasping the fruit by the clamping device. The controlling methods must be more complicated and the cost will be also raised although the fruit could be grabbed accurately.
An end-effector for picking citrus proposed by Liming Xu et al.
Separating methods and their instances
Besides decisive functions for grasping methods, separating methods are also crucial to obtain appropriate structures and the satisfactory performance. 40 –45 While some separating methods have been demonstrated in above actuators, there is no systematic analysis about separating methods and characteristics of their instances. Therefore, different separating instances will be further explained in the next sections gradually.
Rigid separation type
Depending on the scissor, blade, saw, laser, and others, fruit stems can be further cut off by some devices after grasping fruits, and such a separating way is a so-called rigid one with them. 54 Adopting this cutting method, so many components have been developed to separate fruits from their stems actively. 39 Of those, assembling the scissors, mechanical fingers, color camera, and ultrasonic sensor together, an end-effector for picking the eggplant, as shown in Figure 14, was yielded in Japan. At the picking stage, the fruit position can be accurately recognized by visual global and ultrasonic local detection. 29 The end-effector reaches the predetermined position with the help of a mechanical arm after detection. Continuously, fruit stems will be clamped by the mechanical finger and cut off by scissors. 55
An end-effector for picking eggplant proposed in Japan.
Also adopting a mechanical cutting method, lots of effectors have been developed. For example, including three active fingers, three passive fingers, and a cutting element, a noncontact effector for picking strawberries driven by a cable was proposed by Ya Xiong, as shown in Figure 15. 5 Six fingers will be opened simultaneously and swallow strawberries. Composed of two curved blades, a cutting component can rotate quickly to cut off the stem. Finally, strawberries will fall into a container inside the holder. Therefore, a shorter time, about 7.5s, 56 has been achieved in such a continuous way.
An end-effector for picking strawberries from Ya Xiong.
According to the principle of bionics and the mechanical cutting method, an effector for picking citrus was designed by Shun Fu of Chongqing University of Technology, as depicted in Figure 16. 9 Specifically, such an effector contains a suction cup and a shearing element with a symmetrical hinge four-bar mechanism, and fruit stems can be bitten and cut off by the upper and lower blades.
An end-effector for picking citrus proposed by Shun Fu in Chongqing University of Technology.
Introducing the occlusal amputation achieved by the blade and a fruit catcher with the dentation structure, a new picking effector with the tubular structure was designed by Guoli Li in Nanjing Agricultural University of China. 17 As shown in Figure 17, the toothed fruit catcher will be rotated so that apples can be pulled into the catcher, thus hugging stems. Evidently, such an actuator presents several advantages such as the good universality, high efficiency, and low damage rate. However, it is not a better one for fruits hidden in branches and leaves as well as clustered fruits. 57 For round fruits such as apples and pears, a picking actuator also combining a mechanical cutting way was established by Shukui Han in Beihua Institute of Aerospace Technology. 58,59 As depicted in Figure 18, a stepping motor is worked as the driving unit, and the lead screw nut is the transmission mechanism. The grabbing and shearing devices will be driven by such a transmission mechanism so that grasping and cutting behaviors would be carried out orderly and continuously. Therefore, the relatively simple structure and the lower difficulty in control also brings more opportunities for application.
An end-effector with the tubular shape proposed by Guoli Li in China.
An end-effector for picking round fruits designed by Shukui Han.
Based on the hot cutting method, an end-effector was developed for picking cucumber in Netherland, as portrayed in Figure 19. 32 –34 The vision system of this actuator actively detects positions of main stems, and the electric hot knife can surround a main stem from three directions. Also, a V-type guider on the knife will slide upward along the main stem. Completing these programs, the current will be turned on at the moment of contact between a fruit stem and the knife, thus generating the appropriate heat to burn the stem. Such a thermal cutting method is useful to inhibit the bacteria on the cutting wound. However, more requirements for the length of fruit stems and spaced widths must be considered. Therefore, it is also difficult to achieve a satisfactory result.
An end-effector for picking the cucumber presented in the Netherland.
Flexible separation type
Besides above schemes for the relatively large force of cutting stems, some stems will be separated from fruits by slightly pulling, folding, screwing, rotating, overturning, and other actions. 37,39,54 Generally, this way is the so-called flexible separation which is suitable to pick the apple, kiwi fruit, tomato, and others. 18,20,28 For example, with the suction and torsion ways, American scholars such as Schertz and Brown had developed an end-effector for picking citrus in 1968. 7,60 The fruit can be actively absorbed into a rubber hose by this initial robot, and then the stem of a fruit will be further cut off through rotating the wrist. While the application for picking the citrus was achieved, such a robot did not meet requirements for different sizes of fruits.
Also considering the rotation, the FFROBOTICS company (Israel) designed a robot for picking apples in 2017, as shown in Figure 20. 18,61 An underactuated gripper assembled in the effector will be rotated 90° after grasping the apple, further cutting off the stem. In 2020, this company proposed another machine for picking apples combining many mechanical arms. Evenly locating at left and right sides, 4–12 mechanical arms can pick apples synchronously, and the gripper with three fingers is utilized to grasp and separate fruits from stems. 45 Such an arrangement will achieve a high efficiency, such as 10,000 apples per hour.
An end-effector for picking apples designed by the FFROBOTICS company.
Depending on the rotation of the asymmetric four-bar mechanism, a harvesting robot for the kiwi fruit was yielded in the New Zealand, as shown in Figure 21. 20,62,63 In addition to the asymmetric four-bar mechanism, such a robot contains four special mechanical arms and each of them has an end-effector. Receiving a working order, fruits will be clamped and rotated upward around fruit stems by these fingers, thus picking fruits from their stems slightly. Completing these programs, picked fruits will slide into a basket through bellows as long as the fingers are opened cooperatively, which is helpful to achieve the continuous actions without the additional placement and improve the efficiency greatly.
The end-effector for picking kiwi fruit developed in the New Zealand.
In Beijing University of Technology of China, a picking robot for tomatoes under the greenhouse environment was proposed by Lili Wang, as shown in Figure 22. 28,64 Specially, with the bionic design of three finger types of a claw, this end-effector is composed of the wrist motor, a driving motor of the finger, mechanical fingers, the pressure sensor, and the silicone pad of the palm. Fruits can be grabbed and twisted by the hand claw, thus achieving the separation. Combining above components and methods, it usually takes an average of 15s to pick a single tomato, and the success rate is greater than 86.7%. 64 However, the grasping force of the hand claw should be monitored during the entire picking program so that ripe tomatoes will not be damaged. Therefore, it will take more time and bring complex methods of control for picking delicate and fragile fruits.
The end-effector for picking tomatoes designed by Lili Wang in China.
Summary of traditional structures
For obtaining a referenced scheme from existing instances if traditional structures are anticipated to further utilized in some fields and guide how we can improve performances, brief comparisons are given in Table 1.
Comparisons for partial instances.
Concluded from existing instances including above ones, traditional rigid manipulators are usually utilized in the field of picking robots. There are deficiencies in the volume, weight, flexibility, adaptability, picking performance, and so on. In the future, the weight reduction, flexibility, and less mechanical constraints should be considered if these schemes are expected to be improved. Soft manipulators with the variable stiffness and less driven elements are highly anticipated to replace these traditional schemes gradually. Separating methods are also variable for picking different fruits and vegetables. Current separating manners characterize limitations in the versatility and picking efficiency. There is also a challenge in combining rigid and flexible separation methods so that a good method and corresponding separation devices could be adopted to pick diverse fruits. The average cycle for picking fruits or vegetables is generally long. For example, it takes 17 s 19 to harvest a kiwi, 45 s 32 to pick a cucumber, and 1 min 29 to harvest an eggplant. The performance of picking robots working in orchards needs to be further improved, which requires an integrated development of high-performance manipulator and smart control methods such as the image processing, target recognition with the high accuracy, quick response, and low cost.
Developments of underactuated and soft manipulators
As above effectors, traditional schemes also bring partial deficiencies in the grasping and separating methods, hardly achieving a good universality. An end-effector with numerous excellent performances will represent the future development, such as the simple structure, good flexibility, low damage rate, and strong universality. Of those, these defects of traditional manipulators will be overcome or improved following the emergence of underactuated and flexible ones, thus achieving an ability to grasp and separate fruits adaptively. The following chapters will briefly introduce two types of manipulators.
Characteristics of underactuated manipulators and their applied instances
Owing to the increasing requirement for a good universality, lots of underactuated manipulators have been developed. 65 As a new manipulator, it can also work normally under dynamic constraints of its mechanism if the number of driving elements in the mechanical structure is less than that of the degree of freedom of the mechanism. 66 Therefore, compared with a fully driven manipulator, these underactuated ones characterize evident advantages, such as fewer driving elements, the low complexity of the control system, the greater adaptability, the stable ability of grasping, and the small probability of damaging fruits. 67
Combining these concepts, diverse underactuated manipulators have been proposed by international scholars and engineers. For example, as portrayed in Figure 23, the gripper of the underactuated robot for the international space station was designed by Gosselin in Canada. 68,69 Essentially, adopting a three-finger structure, the manipulator achieves 10 degrees of freedom, and two motors are utilized to drive them. Further depending on the mechanical limitation and the spring, the adaptive grab for the target is also feasible through the unpowered joint. Presenting the superior stability, such a manipulator can grasp vigorously and pinch small objects with fingertips. Therefore, this scheme applied in the aviation field has an important reference for designing a robot in the agricultural field.
The underactuated manipulator for the international space station.
Considering the pneumatic and underactuated ways, underactuated fingers have been developed by V Begoc in Italy, as shown in Figure 24. 70,71 Specifically, there are two fingers controlled by four cylinders pneumatically, and each finger contains three joints. Also, six degrees of freedom in the circuit can be achieved by a driving source of air. Therefore, these driving fingers can be used to grasp objects with diverse shapes adaptively, thus meeting requirements for grasping or pinching. However, owing to relatively large sizes, it is also difficult to apply them in picking fruits, and improvements are necessary. Besides these schemes in other fields, many underactuated manipulators with a useful reference for picking fruits or vegetables are proposed, such as supervised descent method (SDM) 72 and Kungliga Tekniska Högskolan (KTH) 73 manipulators.
The underactuated manipulator proposed by V Begoc in Italy.
Directly designing for picking fruits or vegetables, an end-effector with the underactuated element was achieved by Bo Wei in Chongqing University of Posts and Telecommunications. 74 As depicted in Figure 25, this actuator can be utilized to pick citrus with different sizes and ovality adaptively through the grasping and deflecting control for three fingers.
The underactuated manipulator for picking citrus proposed by Bo Wei in China.
Also considering the underactuated mode, a new actuator with multi-degrees of freedom was devised by Huanjun Sun in Sichuan Agricultural University. 75 As depicted in Figure 26, fingers can be opened and closed by controlling a single motor, and it presents remarkable characteristics such as the reliable mechanism and a simple control method. However, the grasping force is small, which is not helpful to obtain a good universality. As shown in Figure 27, a bionic manipulator combining the adaptive and underactuated function was designed by Zhenhao Li in Xi’an University of Engineering. 76 In this structure, three joints and three degrees of freedom are adopted so that artificial fingers are consistent with the physiological structure of human fingers, thus grasping fruits with different sizes, shapes, and qualities adaptively. Therefore, the underactuated manipulator can be also used to pick fruits.
The underactuated manipulator for picking tomato proposed by Huanjun Sun in China.
The underactuated manipulator with the adaptive ability developed by Zhenhao Li in China.
Characteristics of soft manipulators and their applied instances
As mentioned in above sections, it is still not easy to grasp fruits and cut off stems for different sizes, shapes, and types since a rigid manipulator cannot wrap fruit adaptively. This limitation is usually generated by a small deformation range, the less degree of freedom, low flexibility, and poor compliance in a rigid structure although flexible materials have been adopted in inner layers of partial fingers. 77 Fortunately, usually made of flexible materials, soft manipulators can be deformed greatly with infinite degrees of freedom, which is beneficial to grasp fragile and vulnerable objects adaptively by imitating bending actions of human hands. 78 Therefore, flexible materials present an improved way for picking fruits and vegetables, and corresponding soft machines in agriculture and other fields will characterize good references for establishing manipulators directly or indirectly. 79 Crucial aspects of soft manipulators such as materials, driving modes, and the variable stiffness will be analyzed in the next sections.
Materials for soft manipulators
Made of soft materials, diverse flexible manipulators characterize the good flexibility and adaptability, and their safe interactions with the environment are also available. 80 More importantly, different performances are reflected through manufacturing materials in manipulators or robots. 81 Therefore, so many materials such as high elastomers, plastics, particles, and fabrics have been considered in soft robots. 82 Of those, the silica gel with a high elasticity is the most representative material for making soft robots and their manipulators. 83 The details of common soft materials and properties are given in Table 2. 84 –88
The performance of common flexible materials.
Adopting these soft materials, diverse soft manipulators for picking fruits have been designed. 89 –91 For example, combining a three-finger clamping structure and rubber sheets with the spirally arranged way, a soft manipulator with the bending ability was developed by Muscato et al. in Italy, as shown in Figure 28. 90 Its fingers made of a rubber material can further reduce the damaging rate for picking citrus. Considering a network structure of the soft silica gel, a soft manipulator with two fingers was proposed by Octinon company in Belgium, as portrayed in Figure 29. 91 Such fingers can store more compression and deformation energy, and flexibly picking action of strawberries has been achieved.
A soft manipulator developed by Muscato et al. in Italy.
A soft manipulator developed by Octinon company.
Obviously, compared with rigid manipulators, flexible ones present an advantage in grasping easily damaged fruits. 36,92 In addition to existing schemes directly applying in the agricultural field, there are lots of soft robots in other fields have been widely considered, such as the shape memory alloy (SMA) bionic turtle (Figure 30), 93 the bionic robot fish made of IPMC (Figure 31), 94 and the worm bionic robot with the gel material. 95 Essentially, these artificial robots or bionic machines with novel materials present a good reference for picking robots and their end-effectors.
The SMA bionic turtle. SMA: shape memory alloy.
The bionic robot fish made of ionexchange polymer metal composite (IPMC).
Different driving modes
Besides the above excellent performance of flexible materials such as the high flexibility and the large capacity of deformation, the appropriate driving mode is quite crucial to obtain bending, stretching, and twisting behaviors. 96,97 Therefore, international researchers have tried their best knowledge to select a better driving mode, such as the pneumatic element, 98 pull wire, 99 SMA, 100 and electroactive polymer (EAP) 101 driving manners. In detail, characteristics of four driving modes are shown in Table 3.
Comparisons for different driving modes.
HT: high temperature; HP: high pressure; SMA: shape memory alloy; EAP: electroactive polymer.
Specifically, a soft manipulator with the pneumatic element usually actively changes the air pressure inside the driving mechanism, thus achieving bending and extension. 92,102 Combining the concept of pneumatic grid, a flexible pneumatic manipulator was proposed by Li Wen in Beijing University of Aeronautics and Astronautics. As mentioned in Figure 32, 103 the effective length of it can be adjusted according to sizes and shapes of objects, further grasping quickly and accurately. Additionally, this type of manipulator presents lots of advantages including but not limited to the good universality, the convenient manufacture, the economical application, and the stable performance. Therefore, it would be adopted to pick fruits and vegetables without evident damage although the additional air source for driving is also necessary. 104
The soft manipulator with the pneumatic element proposed by Li Wen in China.
Besides the pneumatic element, pull wires are also used to obtain the deformation of the flexible manipulator through an external power. Usually following line channels and fixed points, these wires are embedded inside the soft manipulator in advance. 92,105 As shown in Figure 33, an applied instance of this type had been adopted to pick fruits. Evidently, the simple arc contact is only available, and it is also difficult to wrap and grasp fruits with irregular shapes. 101,106 Fruits with larger weights will be picked easily if soft materials with relatively large Young’s modulus are also considered. 107 However, it is not easy to achieve the miniaturization and integration because of a large volume of its external power source. 108
The soft manipulator with a driving element controlled by wires and an external power.
In addition to above traditional ways, some novel driving elements have been applied in picking fruits or bring a good reference to design corresponding robots. For example, the SMA will be deformed under the external force, thus passively characterizing the certain flexibility. 96 Also, its original shape can be restored after heating, further eliminating the deformation at low temperature. 109 –111 Combining these factors, excellent performances including the shape memory, hyper elasticity, and variable stiffness will be obtained. Therefore, adopting the SMA, the good picking effect on small fruits and vegetables with exquisite skins should be achieved. Besides, it can be operated in a narrow space following flexible behaviors for picking. 92 However, it is not easy to grasp fruits with large mass because of the flexible materials and the relatively small bearing capacity.
Another innovative material called EAP can be deformed under the electricity. 103,112 Therefore, applying the voltage, a manipulator made of this material would be deformed. Also, such a manipulator usually generates a large deformation and the output force. 113 As portrayed in Figure 34, a soft manipulator made of the material presents a good ability of the self-adaptive adjustment according to shapes of fruits or vegetables, thus wrapping and grasping them stably and quickly. 114,115 However, the mechanical strength of the EAP is small, and it can be disturbed easily by surrounding environments. It is also not suitable for picking fruits and vegetables with large weights and sizes. 92
The soft manipulator made and controlled by the EAP material. EAP: electroactive polymer.
Therefore, improvements for a good versatility in picking fruits with different sizes, shapes, and weights should be further considered if these materials and driving modes are adopted.
Variable stiffness technology
While evident advantages have been achieved in partial schemes as mentioned in above instances, soft manipulators generally present the poor anti-interference and the limited grasping ability because of the low rigidity. 92,116 Therefore, the variable stiffness is also proposed by many researchers so that obvious disadvantages can be overcome. 117 –121 Essentially, adding a structure with variable stiffness on the original manipulator benefits the switch between the rigidity and the flexibility, which is a good way to improve the anti-interference strength and increase the ability of grasping. Currently, particle blockage, layered interference, SMA, dielectric elastomer, and so on are extensively adopted to achieve the variable stiffness. 116,121,122 For example, a soft manipulator with variable stiffness combining a principle of passive particle interference was designed by Yingtian Li et al. 116,117 As portrayed in Figure 35, the proposed end-effector is composed of three flexible manipulators which are surrounded by their limiting membranes. These soft manipulators will be expanded as long as the inflation is successful, thus causing particles in the cavity to squeeze each other and improving the stiffness of the entire structure. It is concluded from the existing data that the stiffness has been increased by more than six times if the pressure increases from 20 kPa to 80 kPa. 116
The soft manipulator with the variable stiffness proposed by Yingtian Li et al.
Besides these schemes for picking fruits directly, there are numerous instances applied in other fields which present good references for developing picking robots in the agricultural field. Of those, achieving the variable stiffness and the layer interference, a snake-like manipulator was designed by Kim et al. 116,118 As depicted in Figure 36, its gripper contains a plastic layer with a spiral thin-walled structure, and it is wrapped by the rubber. The friction between different layers will be adjusted through applying the vacuum pressure, thus achieving the variable stiffness, a compact structure, and lightweight. It has been implemented in the minimally invasive surgery and other fields. 123
A snake-like manipulator with the variable stiffness and the layer interference.
Adopting the SMA, a soft manipulator was yielded in Seoul University. 116,119 As shown in Figure 37, the gripper is composed of three driving fingers, and each finger is manufactured by the flexible material and a material with variable stiffness. Fruits can be grabbed adaptively under a state of low stiffness. At a state of high stiffness, the grasping force of a soft manipulator will be strengthened greatly, thus improving the success rate of grasping. It is experimentally concluded that the adjustment range of stiffness can reach up to 55 times. 124
A soft manipulator made of the SMA. SMA: shape memory alloy.
Considering the large deformation mechanism of materials driven by the voltage, the dielectric elastomer was applied to the bionic robotic fish as the artificial muscle. 120 Essentially combining electronic devices, the rigid structure, and a flexible material together, the rigid and flexible integration between materials and the structure is achieved perfectly. Therefore, moving forward quickly, steering with a small radius, shifting right, and other actions benefit the excellent performance of such a robotic fish, which also presents a good reference to design a soft manipulator with the variable stiffness. 125,126
It would report from the above analysis that flexible manipulators working as end-effectors have presented lots of advantages, especially achieving a good flexibility and the high versatility. However, there are also defects in stress, strain, life, and other aspects, thus yielding the low stiffness, easy deformation, and poor load capacity. Relatively heavy auxiliary devices should be assembled additionally in both pneumatic and pulling wire driving modes. The driving force is still small in SMA and EAP driving modes because of limitations of their own material characteristics. As a result, end-effectors are suggested to develop following the integration of intelligent materials, soft manipulators, the variable stiffness, the bionic principle, and underactuated operation.
Target recognition methods
Besides structural characteristics of end-effector, the recognition and location method for fruits or vegetables is another decisive factor to improve the picking efficiency, the success rate, and the quality of them, 127 which is also helpful to reduce labor costs and promote application. Therefore, a large number of studies have been concentrated on how to quickly identify fruits and determine their exact locations, thus achieving diverse fruit recognition and location manners in recent years. The specific features of these methods and their applications will be summarized in the next sections.
Essentially, the fruit recognition is the primary task during operation of a picking robot. This task is always affected by interference conditions since the picking robot has been working in the unstructured environment. As a result, automatic recognition for target fruits or vegetables presents particular difficulties in vision control. Obtaining more appropriate methods in this field, international scholars have proposed numerous methods which can be classified into three types, such as the single-feature vision, multi-feature fusion method, and deep learning method. 128 Their details will be given in the subsequent sections.
Single-feature vision method
Color features are widely adopted in fruit recognition, and such a method is usually effective if the certain color difference between the target and the background can be distinguished. Sufficiently considering such a difference, two methods have been established such as the color difference method and intelligent learning method. 129 Of those, determining a suitable color component or establishing a color operator in different color spaces, the color difference between the target fruit and the surrounding environment would be highlighted, which characterizes the color difference method. Combining this principle, Patel 130 and Thendral 131 obtained a feasible recognition method for ripe citrus utilizing a component in the Lab color space. Zhao De’an 132 and Lü Jidong 133 achieved the recognition method of apple adopting the R-G operator.
While successful instances including above ones are accessible, the flexibility and segmentation recognition are difficult to further improve. Consequently, an intelligent learning-based approach was proposed by many scholars. 134 –143 Specifically, the clustering, modeling, and decision trees at given color spaces or components have been adopted to found fruits rather than the simple method of threshold segmentation. 134,135 For example, the K-means clustering algorithm with a color space transformation model was used to distinguish ripe apples. 136 The automatic recognition and extraction for apples were achieved by dynamic threshold segmentation method of Otsu. 137 Adopting an improved Otsu algorithm, strawberry, tomato, pomegranate, and persimmon were effectively detected, and the accuracy raised to 95%. 138 However, segmenting the image based on different thresholds reduces the robust, and results are sensitive to the external environment. The detection accuracy will also be fluctuated easily. A multi-class color recognition method considering the double Otsu segmentation was also determined, 139 thus improving the detecting efficiency. Combining the HIS and YCbCr color features, K-means clustering algorithm was utilized to detect grape and achieve a success rate of 88.33%. 140 Further adopting a classification algorithm of adaptive boosting (AdaBoost), the rate was increased to 96.56%. 140 Lychee images under the night environment had been collected, and then they were converted to a YIQ color space. 141 Continuously extracting their I5 component images and applying the fuzzy C-means (FCM) clustering analysis, fruits and stalks were successfully segmented from the background. 141 Detecting kiwi calyxes in RGB and HSV color spaces, aggregated kiwis were separated effectively, and the success rate was about 93.7%. 142 Combining segmentation thresholds drawn from the Otsu algorithm, ripe tomatoes had been identified by converting RGB color space to HIS color space. 143
Evidently portraying in the above instances, colors are effective to detect fruits or vegetables under specific conditions. However, the detecting accuracy will be affected by many factors including the difference of maturity, color, variety, background, external illumination, and uncertainty of background. Avoiding these problems, other methods are also considered for detecting fruits or vegetables. Of those, depending on distinguishable or distinctive shapes, some types of fruits would be well identified such as apples, oranges, and cucumbers. The unique shape is helpful to detect these fruits since apples and oranges are more round in shape than their branches and leaves, and cucumbers present a special shape of strip. 144 Considering this point, there are lots of applications in recognition. For example, target contours of round or long fruits had been detected successfully through the combination of the Canny algorithm and the Hough transform. 145,146 Boundary lines between target fruits and the background of images were extracted by the edge detection algorithm, and accurate identifications for long fruits had been achieved by such a combination. 147 A matching library with multi-templates containing 65 images of cucumber was established by Bao et al., and cucumber recognition was realized through a template-matched algorithm. 148 There are lots of instances for detecting round fruits. 149,150 For example, fruits in the cluster were detected effectively by shape analysis, and such a method could identify fruits under conditions of light changes and occlusion. 149 An automatic algorithm of recognition was developed according to convexly geometric features, and a recognition rate raised to 94% for apples in trees. 150
Generally, geometric features are not easily affected by light conditions and are suitable for feature extraction in the orchard environment. Unfortunately, fruits are easily blocked by branches, leaves, and clusters. Shapes, sizes, and other geometric parameters of fruits are varied. Therefore, a relatively low accuracy becomes a fact.
In addition to colors and shapes, textures represent important characteristics of fruits and vegetables, which is helpful to separate target fruits from their background regions. The texture of the fruit surface is not changed by color, and texture features are usually utilized to detect fruits with similar color of branches and leaves, such as green apples 151,152 and green citrus. 153 –155 According to this principle, a fruit recognition method combining the color data was proposed by Zhao. As a result, red and green apples had been successfully detected even if some fruits were clustered by others or blocked by branches and leaves. 151 Combining color, texture, and 3D shape features, apples were detected under uneven illumination, partial occlusion of fruits, and similar background. 152 However, such a method is influenced by threshold values. Adopting the circular Gabor texture and scanning an entire image, green citrus had been detected with a recognition accuracy of 75.3%. 153 Further improving accuracy, an identification method for green fruits was proposed by Chaivivatrakul, and the success rate was raised to 90% by analyzing textures. 154 Considering the texture and intensity distribution, an innovative method for detecting green citrus was demonstrated by Jun Lu et al. in 2018. 155
Concluded from existing instances containing above ones, recognition methods combining the single feature are effective under simple conditions. Disturbed by changes of light conditions, the fruit occlusion, and similar backgrounds in an unstructured environment, shortcomings in these single-feature methods have emerged gradually, such as low accuracy and weak self-adaptability. 156
Multi-feature fusion method
Overcoming evident defects of single-feature methods, multi-feature fusion methods combining colors, shapes, and textures have been proposed so that the success rate could be improved effectively. 157
Of those, the color threshold segmentation and edge perimeter were integrated together to detect citrus, thus obtaining a success rate of 90%. 158 Fusing shape and color figures, the recognition difficulty in clustering and occlusion of tomatoes was solved by Yin et al. 159 Numerous types of fruits had been detected exactly with an accuracy of 90% by a new algorithm including the color, intensity, edge, and direction. 160 According to color and texture features, green apples were detected successfully utilizing the K-nearest neighbor (KNN) and regional growth method. 161 Adopting this method, the success rate was about 85.0% under direct sunlight, and it was raised to 95.0% without direct sunlight. 161 An analysis algorithm with the principal component was established by Nuske et al. 162 In this method, texture, color, and shape characteristics were considered collectively to find grape berries. Introducing color and texture features, detection rates of two types of red grapes were 88% and 91.6%, respectively. 163 Combining color characteristics and the sum of absolute transformed difference, the accuracy for detecting unripe green citrus is 83.4%. 164 Also considering colors, shapes, and textures, the recognition rate of citrus was only 84.4% in an instance. 165 Utilizing colors, shapes, and the support vector machine, eight types of fruits were identified with an accuracy of 83.33%. 166
It is drawn from the above literature that multi-feature fusion methods significantly improve the accuracy and robustness of fruit recognition. However, disturbances such as the light condition, occlusion, and fruit clustering could not be eliminated in natural environment.
Deep learning method
Essentially, above traditional methods are affected easily by illumination. Their fitting accuracy and processing ability for complex scenes are also limited. Avoiding these problems, the deep learning has been widely used to detect, position, and segment fruits from images. 167 –173 For example, the VGG16 network was slightly adjusted and trained for obtaining an image net model. 167 The sweet pepper, rock melon, apple, avocado, mango, and orange were effectively recognized by this model. 167 Different parts of a tomato plant were detected quickly by deep convolutional neural networks (CNN). 168 In detail, 10 types of network models with classification for parts of a tomato plant were established through structural optimization and VGGNet, thus achieving the accuracy of 81.64%, 84.48%, and 53.94% for fruits, flowers, and stems, respectively. 168 Utilizing the Le Net convolutional neural network, kiwi fruit in clusters had been detected under complex conditions. 169 Recognition rates for occluded, overlapping, adjacent, and independent fruits were about 78.97%, 83.11%, 91.01%, and 94.78%, respectively. 169 Evidently, success rates for occluded and overlapping fruits should be further improved. The Faster R-CNN model was adopted to recognize green citrus in the natural environment. However, the comprehensive recognition rate was only 77.45%. 170 Improvements for YOLOv2 were beneficial to detect unripe mango, and the success rates reached 97.02%. 171 Training YOLOv3, ResNet50, and ResNet152 deep networks, the recognition ability of DNN was validated. 172 Of those, the highest recognition accuracy drawn from ResNet152 was achieved, thus obtaining a rate of 95.35% and 97.86% under the natural environment and overlapping situations individually. The identification accuracy for leaves and fruits was 85.12%. 172 Li et al. segmented strawberry images using Otsu and completed the recognition of strawberries using Caffe Net for training, thus achieving a recognition rate of 95%. 173 Compared with traditional methods, the deep learning method characterizes better effects on fruits which are overlapped and covered by others. However, enough set for training is required, and defects are also obvious such as a large amount of calculation, long training time, and high cost.
Depicted from above sections, almost all recognition methods have deficiencies in the accuracy, efficiency and field versatility, and so on. For selecting and improving corresponding methods, brief comparisons are given in Table 4.
Comparison for fruit detection methods from images.
3D reconstruction technologies for positioning fruits
Above methods are generally adopted to determine which parts are fruit or vegetables according to images. It is still difficult to exactly obtain the specific position and deep information of each fruit through these methods. Appropriate methods for positioning fruits have been considered so that the intelligent control system could guide an end-effector to grasp and separate them effectively. Therefore, the 3D reconstruction method has been proposed in this field. Generally, target information is extracted from 2D images, and 3D reconstruction could be further achieved by lots of technologies for visual positioning. 13,174 –208 Of those, active and passive vision are extensively utilized in the picking field, 174 and their details will be demonstrated in the next sections.
Active vision-based reconstruction technologies
Scanning surfaces of target fruits by optical instruments, exact information about the position and space of each fruit can be obtained, the 3D structure of the object will be further achieved through data analysis. Such a method is the so-called active vision method. Currently, the structured light method, 175 laser scanning method, 176,177 time of flight (TOF) method, 178 and radar technology 179 are usually utilized to establish corresponding technologies of 3D reconstruction. Of those, the structured light method is dependent on the principle of optical triangulation. Essentially, a triangular relationship can be determined among the light source, a light sensor, and the reflection point in a scanner. 180 It will be used to compute the depth information of the object, further reconstructing the 3D structure of fruits or vegetables. 180 Combining the near-infrared linear array of structured illumination, this method was successfully used to recognize and position apple stems and calyxes. 181 Depending on the stereoscopic measurement principle of linear structured light, the 3D coordinate information about surface of ripe tomato was extracted effectively during optical fringe scanning process. 182 Especially, the complex illumination in a greenhouse and the target occlusion would be solved by such a method. The reflection imaging principle of structured light was introduced into this field so that optical properties of fruit skin and flesh were measured, which is useful to identify product defects such as damage and decay. 183 Easily concluded from above principles and successful instances, the structured light method is beneficial to obtain exact positions and the deep information with characteristics of simplification, convenience, and high accuracy. However, this method is also relied on high requirements of the scene, and it is not suitable for the outdoor environment.
Besides the structured light method, a laser range finder can be adopted to capture the real scene actually, thus yielding a laser scanning method. Depending on such an operation, there are lots of instances for positioning fruits. For example, the information of orange trees was obtained by a laser ranging sensor. 184 Color and shape factors were fused to locate fruits in this instance. However, its processing speed could not meet requirements of real time. A laser vision system was developed for detecting the cherry in a tree, 185 and this method brought a limitation since it was also impacted by colors of fruit surfaces. Eliminating effects of different lights and complex backgrounds, an algorithm for automatic recognition of apples in trees was created by Juan Feng et al. 186 The laser vision system was adopted to obtain local 3D images of fruit trees with a good real-time performance. The recognition rate of fruit is higher than 93.75% if shaded areas of branches and leaves were less than 40%. 186 A novel laser vision system was designed for picking robots, further achieving identification and orientation with high accuracy in the unstructured environment of the orchard. 187 The recognition rate was over 90% and the response time was only 3.58s as long as the occlusion rate was less than 50%. 187 Easily portrayed from existing instances, laser scanning method is not only used to reconstruct 3D models of targets with simple shapes but also generates 3D models of irregular objects with high accuracy. However, time-consuming for a huge amount of point cloud registration is long, and equipment is also quite complex and expensive.
Another method called TOF has presented evident advantages since it can be used to directly capture geometric information of the scene from the 3D point cloud. 188 Specifically, an optical instrument emits pulsed light with a certain frequency to a target, and such an operation will be ended if the receiver receives the returned light. Finally, distance information of a target will be obtained by a combination of the number of pulses and the speed of light. 189 Considering these rules, a TOF camera was utilized to construct a 3D structure of an apple tree. 190 Reducing effects of picking environment, a sensing system of plant space was established by utilization of a TOF camera, and it was not sensitive to light conditions. 191 Further combining RGB and TOF cameras, information of an entire tree was measured effectively. 192 Essentially, high resolution of RGB camera and the high precision of TOF camera had been characterized collectively in this method, and positioning accuracy was improved greatly.
Unlike counting pulses of the TOF method, time differences between transmitted and received beams are also adopted to measure distances of a target, thus yielding another 3D reconstruction method with radar-based technologies. While direct instances in the picking field adopting radar technologies are relatively fewer, positioning applications in other sectors of the agricultural field will also present a useful reference for picking fruits or vegetables. For example, combining machine vision and a lidar, the motion of a spraying robot in a greenhouse was controlled accurately. 193 Relying on a lidar, a navigation system of the inspection robot was developed so that precise positioning and path planning could be achieved. 194 Evidently, a radar-based technology is quite suitable for navigation and path planning. With a huge field of view and a long distance, it must be affected by the external environment, the accuracy of 3D reconstruction is difficult to further improve. Also, the cost of a radar with high accuracy is expensive for applications in agriculture.
Passive vision-based 3D reconstruction technologies
As above instances, methods with active vision are generally affected by the environment and equipment. Therefore, a great deal of effort has been devoted to establish passive vision-based methods in recent years. 13,196 –208 Generally, vision sensors (one or more cameras) are used to capture a series of images, and then the 3D structure of a target will be reconstructed, thus achieving a 3D reconstruction method with passive vision. According to the number of cameras, there are three types of passive vision methods, such as monocular vision, binocular vision, and multiocular vision. 195
A single camera is usually adopted to capture images. The target will be then recognized and reconstructed based on image information. Applying this method, numerous systems with monocular stereo vision had been established in this field. 13,196,197 For example, combining a camera and a light source with a high frequency, apples in an outdoor environment were detected effectively by a monocular vision system. 13 A set of operating system nodes ensuring modularity and separation of concerned points were established for controlling crucial aspects synchronously, such as the motion, image acquisition, fruit detection, visual servo, and relative position of a monocular camera. This system was used to recognize and locate the sweet pepper by extracting visual information from the end-effector. 196 A monocular camera was assembled on the end-effector of a picking robot, and the effector was rotated to image and fruit centers after determining the center of mass of a fruit, further grasping fruits. 197
Reflected through existing instances, a positioning system with monocular camera presents evident advantages such as the simple equipment, flexible application, and low cost. However, the accuracy of 3D reconstruction is relatively poor.
To overcome shortcomings of monocular vision, a binocular vision system that is similar to human eyes has been considered in the field of picking fruits or vegetables. 198,199 Specially, two identical cameras in this system can be used to capture images of left and right sides in the same position from different angles, and then deep information of the object will be extracted by the principle of triangulation. Further depending on corresponding information, 3D models of a target fruits should be reconstructed effectively. Combining such a rule, a picking robot for tomatoes was designed in the University of Tokyo using a binocular vision system and a manipulator with six degrees of freedom. 200 The success rate of picking was about 60%. 200 The image of a cherry tree was collected by binocular vision system with the parallel optical axis. 201 Relying on the converging optical axis, a binocular vision system was adopted to capture left and right images of citrus, thus presenting a spatial location method for picking outdoor citrus. 202 A binocular vision system based on the embedded ARM was also developed for picking apples. 203 In this system, a processing module of robot vision was established by a combination of a CMOS binocular camera and the ARM COTEX A9 processor. 203 Therefore, fast location for a picking robot must be achieved after target extraction and stereo matching. Adding one or more cameras to the binocular vision system, an extension system called multiocular vision could be proposed. 204 Actually, several images of an object will be obtained in view of different angles, and the blind area in measurement can be reduced greatly, especially obtaining a larger field of vision. 205 However, the whole system must be more complex and computing loads is also increased greatly. Therefore, it is widely adopted in fields of vehicle autonomous driving, robot vision, mechanism control, and so on. 206 –208 Applications in other fields will also present a good reference for picking fruits or vegetables.
On the basis of above instances, characteristics of current positioning manners are briefly compared in Table 5.
Comparison for 3D reconstruction based on vision.
TOF: time of flight.
Conclusion and future prospective
Reflected through structural schemes and their recognition as well as location methods, characteristics of them can be yielded that:
Grasping, separating, and recycling actions are carried out orderly. Specifically, fruits will be grabbed by a clamping or suction mechanism, and then the cutting mechanism is utilized to separate them. Finally, fruits will be recycled through mechanical arms, soft tubes, and others. Therefore, the picking efficiency might be reduced by the cumbersome action, the complex control, and uncoordinated movements between mechanisms and other factors.
The rigid structure is usually adopted in most of existing manipulators whichever grasping mode is considered. The clamping force and suction are not easy to control because of the hard material, and the mechanical damage to fruits and vegetables is also an inescapable fact. Besides these, the damage is also caused by the collision between the gripper and external environment at the picking stage. Therefore, for avoiding these damages, improvements in picking method, mechanical structure, material, driving mode, and control system should be considered. Rigid manipulator or underactuated ones working as a traditional way generally lacks a certain flexibility and the ability of force perception. It is still difficult to wrap and pick fruits adaptively.
Flexible materials have been adopted to in very few schemes so that some defects of the rigid structure could be overcome. However, the unstable behavior or the oscillation may be caused by these soft materials with a low stiffness. Especially, the soft manipulator will be prone to bending deformation, thus resulting in failure to grasp fruits. Therefore, a soft manipulator should be developed following a direction of variable stiffness, finally obtaining excellent performance combining good adaptability, a low damage rate, good versatility, and so on.
Most existing end-effectors are usually designed for specific fruits or vegetables, evidently presenting the poor universality in different types. In a word, it is possible to use them in a certain harvesting season, which undoubtedly reduces the utilization rate of picking robots.
The average cycle is also longer than what is anticipated in agricultural operation with high efficiency, and the recognition accuracy of end-effector is relatively low in easily practiced schemes. Large errors in recognizing and positioning are also generated by the vision system inevitably. Complex conditions including the light, color, shape, texture, and growth state bring difficulties in processing images. Therefore, a good vision system with the ability to overcome these problems is highly anticipated.
Consequently, the high efficiency, low damage, flexibility, adaptability, versatility, and high accuracy should be solved collectively in the future if harvesting robots are expected to be applied in diverse conditions with a low cost. Therefore, suggestions for developments are concentrated on the following aspects.
New structures for picking manipulator should be developed first so that a good adaptability for different sizes, shapes, and weights would be achieved possibly, thus meeting requirements for picking in all seasons and reducing the cost of utilization.
Further combining flexible materials, variable stiffness methods, bionic principles, and proper driving modes, a picking robot is suggested to achieve the simple structure, easily controlled program, low cost, less energy consumption, good stability, low damage rate, and miniaturization integration, which characterizes great challenges in promoting development of picking robots.
The accuracy and efficiency of recognizing and positioning systems with the low cost should be improved greatly. The excellent performance of a picking robot will be available if the manipulator with appropriate structure, the entire robot, intelligent recognition technologies, and the orchard environment can be controlled synchronously.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
