An optimum strategy for robotic tomato grasping based on real-time viscoelastic parameters estimation

Abstract

It is a challenging task to achieve rapid and stable grasping of fruit and vegetable without damages for the agricultural robot. From the point of view of which most of fruits and vegetables are viscoelastic material, the viscoelastic characteristic of tomato was analyzed based on Burgers model in this article to provide a reference for the robotic grasping. First, the real-time viscoelastic parameters estimation model based on back-propagation neural network was established. The 3-11-4 network structure was applied, where the grasping force, displacement, and time were input to the model to estimate four viscoelastic parameters. The relative error was less than 15% at the 0.2-s estimation and correlation coefficient of fitting could reach to 0.99. Then, the expression of plastic deformation was derived by analyzing the dynamic characteristic of tomato based on Burgers model and Gripper’s model during grasping. The minimum plastic deformation was taken as the condition to optimize the grasping speed and operation time. Finally, the result of simulation and experiment showed the feasibility of the method proposed in this article. This research can achieve the goal of reducing the grasping time of robots without damaging the fruit and provide a reference for robots grasping process optimization.

Keywords

Robotic grasping viscoelasticity BP neural network optimal strategy

Introduction

In the wide application of agricultural robots for the fruit and vegetable picking, sorting, packaging, and other tasks, robotic grasping is a crucial link in the whole operation.^1
–3 During the grasping process, the impact and deformation caused by the contact of the gripper with fruit and vegetable easily result in the damage and sliding. Most fruits and vegetables are viscoelastic, which can exhibit both viscous and elastic characteristics when undergoing deformation, called viscoelasticity. The viscoelasticity is not only the important indicator to evaluate the quality of fruit and vegetable but also the required mechanical parameter to prevent the mechanical damage and sliding of fruit and vegetable.^4
–6 The viscoelastic parameters can be used to analyze the deformation law of fruit and vegetable, which can assist to optimize the grasping strategy through setting the operating parameters of gripper such as grasping speed and operation time, thus helping robot complete the stable and fast grasping operation without damaging objects.

The grasping optimization of the agricultural robot has been researched for a long time. Zhi-Guo et al.⁷ used the robotic arc fingers to quickly grasp the tomatoes, and they concluded that the key to determining the operation efficiency and mechanical damage is the finger closing speed. Furthermore, Pettersson et al.⁸ designed the universal gripper to optimize the grasping time of vulnerable objects and used PD controller to accelerate the grasping and decrease the force overshoot, thus the fruits with different characteristics can be grasped in a relatively short time.

The methods mentioned above analyzed the influence of grasping speed and contact force on the mechanical damage of fruits, but they did not comprehensively consider the defects in design caused by biological characteristics and mechanical characteristics of objects.⁹ According to these deficiencies, the researchers began to analyze the grasping operation based on the viscoelastic characteristics of fruit. Higashimori et al.¹⁰ analyzed the relationship between the imposed grasping force and plastic deformation based on the four-unit viscoelastic model parameters of the rheological objects as well as decreased the grasping operation time as avoiding the excessive deformation of objects. Saeidirad et al.¹¹ were used to make stress relaxation test for different varieties of pomegranate using texture analyzer, and artificial neural network (ANN) was used to replace the Maxwell model to predict the viscoelastic properties of pomegranate. Similarly, Liu et al.¹² analyzed the relationships among the grasping speed, maximum contact force, and plastic deformation based on the viscoelastic model of fruit, and then their findings were used for reference for realizing the stable grasping operation of fruit.

However, most studies need to analyze the viscoelastic characteristics of fruit and vegetable offline for a longer time. We took the tomato as the operation objects to build the viscoelastic parameters estimation model based on the neural network in the preliminary stage. The robot can rapidly and accurately estimate the viscoelastic parameters of tomatoes in real time. Moreover, the minimum plastic deformation of tomatoes was taken as the constraint condition, and the grasping strategy was optimized by adjusting the closing speed of robotic gripper, grasping operation time.

Viscoelastic parameter estimation model

Firstly, the creep test was used to obtain the relationship curve between deformation and grasping time. Then, the creep model which could accurately coincide with the relationship curve was found. Finally, the viscoelastic parameter estimation model was established by the differential equation of creep model.

Creep test

Wanfen No. 2 tomato was selected as the test material. According to the definition of mature tomato period in international standard SB/T10331-2000, 10 tomatoes in the early maturity (the red face of fruits was 10–30%) and in the middle maturity (the red face of fruits was 40–60%) were selected, respectively. The equatorial diameter of these picked tomatoes was between 72 mm and 80 mm, and their weight was between 166 g and 252 g. The texture analyzer TMS-PRO made by US FTC’s, Virginia, USA was used as the test instrument. The parameters for the creep test were set as follows: the probe was P75 with a diameter of 75 mm, the initial force was 0.6 N, the loading time was 30 s, the loading speed was 60 mm/min, and the data sampling frequency was 10 Hz. As shown in Figure 1, the tomato to be measured was horizontally placed and was fixed by a base. Three points were selected in the fruit waist for each sample, and then the creep tests were carried out by the holding force of 3, 5, and 7 N, respectively.

Figure 1.

The platform of creep test.

In the creep experiments, the curves of deformation and time can be described by the creep models, such as Maxwell model, Kelvin model, and Burgers model. The Burgers model^13
–15 can represent the main characteristics of viscoelasticity, which is one of the most famous models to predict the creep characteristics. It consists of a Maxwell unit and a Kelvin unit which are connected in series, and the structure is shown in Figure 2. In this article, the Burgers model was used to describe the creep characteristics of tomato during loading.

Figure 2.

The Burgers model.

The differential equation of Burgers model in the creep loading process is as follows

D (t) = \frac{F_{0}}{k_{1}} + \frac{F_{0} t}{c_{1}} + \frac{F_{0} (1 - e^{- \frac{k_{2}}{c_{2}} t})}{k_{2}}

Where D is deformation (mm), t is time of grasping objects (s), F ₀ is load force (N), k ₁ is single elastic parameter (N/mm), c ₁ is single viscous parameter (N·s/mm), k ₂ is elastic parameter in the parallel unit (N/mm), and c ₂ is viscous parameter in the parallel unit (N·s/mm).

According to the above-mentioned parameters, k ₁, k ₂, c ₁, and c ₂ were selected as the viscoelastic parameters of tomato. And then, the fitted curve based on equation (1) and the creep curve were calculated. The results are shown in Figure 3; all the fitting correlation coefficients are higher than 0.99, which verify the effectiveness of the selected model. Finally, all the viscoelastic parameters of tomatoes were stored to build the parameter estimation model.

Figure 3.

Creep curve and fitted curve.

Viscoelastic parameter estimation model

Model design

The ANN is a complex network composed of extensive connection of a vast number of simple processing units, and it can solve the complex nonlinear problems. We selected the back-propagation (BP) neural network to build the viscoelastic parameter estimation model.^16,17 BP neural network is a multilayer feedforward network trained by error BP algorithm, which is one of the most widely used neural network models.

In this article, the model structure is shown in Figure 4. Three-layer neural network structure was used, and the transfer functions were Tansig and Purelin, respectively. The number of nodes in the input layer and output layer was determined by the number of input and output variables. The input variables of this model were the grasping force F, deformation D which is equal to the displacement of the gripper, and time t. The output variables were the viscoelastic parameters k₁ , k₂ , c ₁, and c ₂. So the number of nodes in the input layer and output layer was three and four, respectively. According to Kolmogorov theorem, the theoretical number of nodes in the hidden layer was seven. Moreover, it was found that the number of nodes of the optimal hidden layer was 11 after training test. So the network topology of the model established in this article was 3–11–4.

Figure 4.

The structure of BP neural network. BP: back-propagation.

Network model training

Firstly, the normalization processing was conducted on the data, which were converted into the number between [−1, 1]. Min–max normalization was applied to convert the data. The converting function is shown as follows

x_{k} = \frac{(x_{k} - x_{min})}{(x_{max} - x_{min})}

According to the neural network structure and transfer function, the learning rate was set as 0.01, maximum iterations were 1000, 500 sets of data were randomly selected as the training data from totally 600 sets of sample data, and the other 100 sets of data were the test data. The mean square error (MSE) was used to evaluate the performance of the network model, where N is the number of training data, T_i is the true value of the output, and Y _i is the predicted value of the network output, as follows

MSE = \frac{1}{N} \sum_{i = 1}^{N} {(e_{i})}^{2} = \frac{1}{N} \sum_{i = 1}^{N} {(T_{i} - Y_{i})}^{2}

After trained, we obtained the correlation coefficients such as 0.93, 0.92, 0.93, and 0.93, respectively, and the MSE was 0.033. The results of relative error rate are shown in Figure 5, and the relative error of k ₁ is less than 15% which has a great performance. Although the relative error of k ₂, c ₁, and c ₂ has a higher fluctuation, 90% of the tested results could keep in an ideal range which the relative error is less than 20%. We further discussed the optimal selection of real-time estimated viscoelastic parameters by grasping experiment.

Figure 5.

The relative error of predicted viscoelastic parameters.

Viscoelastic parameters real-time estimation

The WSG 50 robotic parallel gripper made by SCHUNK, Neckar, Germany was used to grasp the tomato. The grasping experimental platform is shown in Figure 6.

Figure 6.

Grasping experimental platform. 1, Data obtaining system. 2, WSG 50 gripper. 3, Force feedback finger. 4, Voltage-stabilized source.

The grasping force was fed back by one of the fingers in real time, and the displacement was measured by the encoder. Firstly, the minimum grasping force was obtained by determining whether the sliding generated in the grasping process for the randomly selected tomatoes in the early and middle maturity, thus calculating the friction coefficient μ between the tomato and the gripper. So we can express the contact force F₀ when the gripper stops forcing as follows

F_{0} = \frac{m g α}{2 μ}

Where m is the mass of tomato, g is the gravitational acceleration, and α is the safety factor.

Then, the gripper reached F ₀ at a constant speed, stopped, and kept self-locking for 2 s. After that, the fruit was released with maximum speed. The force, displacement, and time were recorded in the grasping process at every 0.1 s. Finally, the recorded data between 0.1 s and 0.6 s were imported to the trained neural network to estimate the viscoelastic parameters of tomatoes. The result was shown in Table 1.

Table 1.

The estimated value and measured values.

Sample	Viscoelastic parameters	Estimated value						Measured value
Sample	Viscoelastic parameters	0.1 (s)	0.2 (s)	0.3 (s)	0.4 (s)	0.5 (s)	0.6 (s)	Measured value
Early maturity	k ₁ (N/mm)	3.98	3.52	3.43	3.63	3.94	3.89	3.39
	k ₂ (N/mm)	15.84	12.32	13.09	10.60	12.70	12.74	11.84
	c ₁ (N·s/mm)	917.57	689.70	643.26	673.29	808.56	769.75	666.40
	c ₂ (N·s/mm)	77.89	60.22	54.86	52.38	66.71	70.94	63.05
Middle maturity	k ₁ (N/mm)	4.46	4.25	4.31	4.94	4.41	4.95	4.93
	k ₂ (N/mm)	16.88	15.91	12.64	14.14	11.4	14.94	13.45
	c ₁ (N·s/mm)	760.91	926.15	887.51	1006.01	944.47	1086.03	976.64
	c ₂ (N·s/mm)	75.65	72.32	63.93	61.77	66.21	83.40	77.70

By the analysis of the data in Table 1, in addition to the 0.1-s parameter relative error is large, the relative errors after 0.2 s are less than 15%. In order to realize the rapid real-time estimation of the viscoelastic parameters of tomato during the grasping process, the accuracy of estimated parameters should be considered as well as the time of parameter estimation should be shortened as far as possible to improve the operation efficiency. Hence, the estimation parameters obtained at 0.2, 0.3, and 0.4 s were temporally taken as the basis of grasping strategy optimization. The exact acquisition time of parameters was determined by the comparison of the test results of grasping optimization.

Grasping strategy optimization

Dynamic characteristic of tomato during the grasping

The deformation of viscoelastic material can be divided into elastic deformation and plastic deformation. The elastic deformation is that part of the deformed material which when the applied load is removed would spring back to its normal shape. The plastic deformation is the part which would not spring back after removing the load. The final shape of the material directly determined by the plastic deformation, and it is also the main factor affecting the quality of fruit.

We analyzed each grasping parameter based on Burgers model and then put forward the optimization strategy. According to the Burgers model,^18,19 when the external force f acted on the objects, the dynamic characteristic of tomato was shown in Figure 7.

Figure 7.

Viscoelastic model for grasping a tomato.

Where x, x_i , and f_i (i = 1, 2, 3) are the whole deformation of the object, the deformation of each unit, and the force where the single elastic element (i = 1), the parallel viscoelastic unit (i = 2), and the single viscous element (i = 3), respectively. By the relationship between the force and the displacement of each element, we can finally obtain the following equations

f (t) = f_{i} (t)

x (t) = x_{1} (t)

f_{1} (t) = k_{1} [x_{1} (t) - x_{2} (t)]

f_{2} (t) = k_{2} [x_{2} (t) - x_{3} (t)] + c_{2} [{\dot{x}}_{2} (t) - {\dot{x}}_{3} (t)]

f_{3} (t) = c_{1} {\dot{x}}_{3} (t)

Suppose that the object was grasped by a robotic hand with stiffness k_h , the displacement of the gripper is x_h , from equations (5)–(9), we can derive the following equation

x (t) = x_{h} (t) - f (t) / k_{h}

B_{2} {\ddot{x}}_{h} (t) + B_{1} {\dot{x}}_{h} (t) = A_{2} \ddot{f} (t) + A_{1} \dot{f} (t) + f (t)

Where

B_{2} = c_{1} c_{2} / k_{2}

B_{1} = c_{1}

A_{2} = \frac{c_{1} c_{2} (k_{1} + k_{h})}{k_{1} k_{2} k_{h}}

A_{1} = c_{1} k_{1} k_{2} + k_{h} (k_{1} c_{2} + k_{1} c_{1} + k_{2} c_{1}) / k_{1} k_{2} k_{h}

For the Burgers model, since the deformations of x₁ and x ₂ are completely recovered by the elastic elements of k₁ and k₂ at the infinite time, the plastic deformation X_p could be expressed with x ₃(t) and c ₁, from equations (1) and (5), and the X_p can be derived as follows¹⁹

X_{p} = \frac{1}{c_{1}} \int_{0}^{T} f (t) d t

Optimization model

In order to better analyze the deformation during grasping, the total grasping procedure was divided into three phases in this article. As shown in Figure 8,

Closing phase: t_c is the time from the contact to the stop of forcing and the gripper was closing with constant speed v_h.

Holding phase: $x_{h}^{F_{0}}$ is the displacement of gripper and t_h is the time while tomato held by the gripper.

Opening phase: After the gripper opens the finger with the maximum speed $v_{h}^{max}$ , the plastic deformation X_p was caused.

Figure 8.

Three phases for grasping the tomato.

The total time T during the grasping operation could be defined as follow

\begin{array}{l} T = t_{c} + t_{h} + t_{o} \\ = \frac{x_{h}^{F_{0}}}{v_{h}} + t_{h} + \frac{x_{h}^{F_{0}}}{v_{h}^{max}} \end{array}

In this article, the idea of optimization strategy is as follows: first, the viscoelastic parameters were estimated by the BP neural network. Then, since the gripper was closing with constant speed, ${\ddot{x}}_{h} (t) = 1$ and ${\dot{x}}_{h} (t) = t$ , the nonhomogeneous linear differential equation of f(t) could be derived based on equations (11)–(15) and the plastic deformation X_p can be obtained by substituting in equation (16). Finally, X_p under different grasping speed and holding time could be obtained through the grasping experiment. Thus, we took the minimum plastic deformation as the condition to optimize the grasping strategy.

Experiment

Viscoelastic parameter estimating time selection

The experiment platform was same as in Figure 6. From equation (11), ${\ddot{x}}_{h} (t) = 1$ and ${\dot{x}}_{h} (t) = t$ , so the relationship curve of the grasping force and the operation time could be obtained by determining the viscoelastic parameters of tomato. The estimated viscoelastic parameters at the grasping moment t_e = 0.2, 0.3, and 0.4 s and real viscoelastic parameters measured by the texture analyzer were substituted into equations (11)–(15) to obtain the estimated curve and tested curve, respectively. Thus, the exact acquisition time can be determined by analyzing the curve and comparing the correlation coefficient of fitting. At the beginning of the grasping, we set up some required parameters, where displacement k_h is 70000 N/mm, the velocity of gripper opening $v_{h}^{max}$ is 200 mm/s which is in the maximum speed condition, the velocity of grasping v_h is 5 mm/s, and the total time T is 2.5 s.

The result is shown in Figure 9; the curve calculated by estimated parameters can nicely match the curve calculated from the measured parameters. Furthermore, the degree of the approximation keeps at a high level. Considering the efficiency and precision of the grasping operation, the parameters at 0.2 s are selected as the basis for the optimization of the grasping strategy.

Figure 9.

The performance of different viscoelastic parameters. (a) Output of the early maturity. (b) Output of the middle maturity.

Grasping experiment

The general solution of equation (11) could be obtained by substituting the viscoelastic parameters, and then the plastic deformation X_p could be calculated from equation (16). The changed law of the X_p with the different grasping speed v_h during the grasping process was analyzed, and the simulation results are shown in Figure 10.

Figure 10.

The relationship of plastic deformation and speed.

From Figure 10, we can conclude that when the grasping speed is smaller, the contact time between fingers and fruits is longer, the creep time is longer, and thus the plastic deformation is larger. With the increase of speed, the plastic deformation apparently decreased; however, as the speed increases to points A and B, the plastic deformation begins to increase. It illustrates that the speed increased as well as the inertia force and the total deformation of grasping increased, resulting in the plastic deformation increased. Therefore, the studies on the suitable grasping speed could not only improve efficiency but also decrease the plastic deformation of fruit.

We took the minimum plastic deformation as the optimization condition to optimize the grasping speed v_h and total operation time T. The optimization result is shown in Table 2, where $v_{h}^{max}$ is 200 mm/s, and holding phase time t_h is 2 s.

Table 2.

Optimization result.

Sample	m(g)	F₀(N)	Estimated value				Optimized value
			k ₁	k ₂	c ₁	c ₂	v_h (mm/s)	T (s)
Early number 1	220	7.50	4.11	12.29	1075.41	77.41	6	2.23
Early number 2	180	6.14	4.25	15.91	883.51	62.43	6	2.25
Middle number 1	166	5.66	3.19	10.74	869.16	65.58	9	2.34
Middle number 2	192	6.55	3.94	16.53	965.65	81.42	8	2.47

From the experiment result, the grasping speed and operation time were optimized under the condition of minimizing the plastic deformation. Therefore, the robot could quickly complete the grasping operation without damaging the tomato. We believe that our method in this article could also provide a reference for the relative study of other fruits and vegetables.

Conclusion

The viscoelastic characteristic of tomato was analyzed based on Burgers model in this article to provide a reference for the robotic grasping.

We presented a real-time viscoelastic parameter estimating model based on BP neural network. The 3–11–4 network structure was applied, where the grasping force, displacement, and operation time were input to the model to estimate four kinds of viscoelastic parameters which can ensure the robot can efficiently and rapidly estimate the object’s viscoelastic parameter during the grasping.

We presented an optimum strategy for setting robotic operation parameters by studying the dynamic characteristic of tomato based on Burgers model and gripper’s model. The minimum plastic deformation was taken as the condition to optimize the grasping speed and operation time. The experiment results showed optimized speed and operation time which were calculated by viscoelastic parameters can help robot complete the fast and stable grasping operation without damaging the objects.

We believe that our research could contribute to providing a reference for agricultural robotic grasping.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Natural Science Foundation of China under grant 31471419, and the Natural Science Foundation of Zhejiang Province under grant LY17F0300006.

References

Belforte

Deboli

Gay

. Robot design and testing for greenhouse applications. Biosystems Eng 2006; 95(3): p. 309–321.

Tsai

CHD

Imin

Akihide

. Experimental study of creep response of viscoelastic contact interface under force control. In: 2010 IEEE/RSJ international conference on intelligent robots and systems (IROS), Taipei, Taiwan, 18–22 October 2010. IEEE.

Wilson

. Developments in robot applications for food manufacturing. Industr Robot Int J 2010; 37(6): 498–502.

Lei

. Study on the relationship between mechanical and theological properties and ripe degree of tomato. J Agric Mech Res 2006; 30: 167–169.

Blanes

Mellado

Ortiz

. Review. Technologies for robot grippers in pick and place operations for fresh fruits and vegetables. Spanish J Agric Res 2011; 9(4): 1130–1141.

Sam

Nefti

. Design and feasibility tests of flexible gripper for handling variable shape of food products. In: Proceeding 9th WSEAS international conference on signal processing, robotics and automation, Cambridge, UK, 20–22 February 2010.

Zhi-Guo

Ji-Zhan

Ping-Ping

. Study on the collision-mechanical properties of tomatoes gripped by harvesting robot fingers. Afr J Biotechnol 2009; 8(24): 7000–7007.

Pettersson

Ohlsson

Davisb

. A hygienically designed force gripper for flexible handling of variable and easily damaged natural food products. Innov Food Sci Emerg Technol 2011; 12(3): 344–351.

Liu

. Relationship between mechanical property and damage of tomato during robot harvesting. Trans Chinese Soc Agric Eng 2010; 26(5): 112–116.

10.

Higashimori

Yoshimoto

Kaneko

. Active shaping of an unknown rheological object based on deformation decomposition into elasticity and plasticity. In: 2010 IEEE international conference on robotics and automation (ICRA), Anchorage, Alaska, USA, 3–8 May 2010.

11.

Saeidirad

Rohani

Zarifneshat

. Predictions of viscoelastic behavior of pomegranate using artificial neural network and Maxwell model. Comput Electron Agric 2013; 98: 1–7.

12.

Jizhan

Xinxin

Pingping

. Complex collision model in high speed gripping of fruit. Trans Chinese Soc Agric Machin 2014; 45(4): 49–54.

13.

Tudor

Daniela

Marian

. An experimental method for determining the time of relaxation at static compression of apples. J Eng Stud Res 2011; 17(2): 18–27.

14.

Liu

Bai

. Modified Burger’s model for describing creep behavior of tomato fruits. Trans Chinese Soc Agric Eng 2013; 29(9): 249–255.

15.

Van Henten

Van’t Slotc

Hol

JCW

. Optimal manipulator design for a cucumber harvesting robot. Comput Elect Agric 2009 65(2): 247–257.

16.

Xiaoqian

Shibin

Zhenguo

. Research on preprocessing and postporocessing of the application of BP neural network. Syst Eng Theory Practice 2002; 22(1): 65–70.

17.

Jin

Wei

. The improvements of BP neural network learning algorithm. In: 2000. WCCC-ICSP 2000. 5th international conference on signal processing proceedings, Beijing, China, 21–25 August 2000, IEEE.

18.

Yaxin

Xiao

Guoxiang

. Plasticity model for pressing of sunflower seeds. Nongye Jixie Xuebao Trans Chinese Soc Agric Machinery 2009; 40(7): 138–142.

19.

Sakamoto

Higashimori

Tsuji

. An optimum design of robotic food handling by using Burger model. Intell Serv Rob 2009; 2(1): 53–60.