Ultrasonic robotic system for noncontact small object manipulation based on Kinect gesture control

Abstract

Acoustic levitation system, with the advantages of avoiding clamping force on samples, has been widely introduced into noncontact manipulation. Previous acoustic levitation systems were generally fixed and could not realize remote adjustment of manipulation stage. In this article, we develop an ultrasonic robotic system for small object manipulation which is capable of noncontact operation, remote control, and gesture-based intelligent control. The whole setup is composed of acoustic levitation platform, mobile operation platform, and Kinect sensing system which provides a humanized approach for human–machine interaction. The experimental results validate a sufficient suite of innovations for achieving remote control of this acoustic levitation system based on gesture recognition method. It will not only pave a new way for noncontact manipulation system but also greatly reduce the difficulty and cost of operation.

Introduction

There are basically two types of object-operating-methods: contact operation and noncontact operation. Robotic arms are common representatives of contact operation, which have clamping force that would damage the small objects while noncontact method would not. The noncontact type of holding forms frictionless force that is useful for small and fragile items. Also, hazard created by dangerous drug can be avoided with this method. Thus, noncontact operation receives increasing interest in researches. Levitation, as an application of noncontact operation, provides with an upward force counteracting the gravitational force. Acoustic levitation is one of the techniques developed, which generates the acoustic field by creating a wave and reflecting it back to its source. This technology has been applied on a deal of research areas, such as bioanalytical chemistry. ^{1 –5} For example, it is used at the levitation of iridium and liquid mercury, ¹ investigating the small-angle scattering of macromolecular solutions ⁴ and the agglomeration of proteins in acoustically levitated droplets. ⁵ In the field of material science, ^{6 –9} researches have been done to study the equilibrium shapes of levitated drops ⁷ as well as solid state amorphization of pharmaceuticals. ⁸ Also, fluid dynamics have been studied by acoustic levitation, ^{10 –14} which can measure the liquid surface tension, observe the steady-state acoustic streaming flow patterns, ¹¹ and carry out mass transfer in the surface of sphere by acoustic streaming. ¹² Acoustic levitation has been used in assembly as well, ^{15 –17} even small living animals could be levitated by acoustic method. ¹⁸ However, most of the previous research were working with fixed levitation platform, at which the objects movement is difficult to be controlled. Thomas et al. developed a linear acoustic levitation transportation system based on a ring-type vibrator, which could realize linear transportation of object on horizontal position. ¹⁹ Here, we propose a novel approach that controls the unidirectional movement of objects vertically instead by adding a one-axis slide.

Recently, more and more human–machine interactions work on humanized and simple ways by basing on gesture, speech, and other commands. ^{20 –24} Among them, gesture recognition as a new popular human interaction way could take the place of direct operation, which can be realized by Kinect. For example, Kruse et al. used hand motions to command the desired position of robotic arm by Kinect sensor. ²⁴ In order to remotely control the noncontact manipulation, Kinect is connected to the levitation platform as the input device.

This article introduces a noncontact operation method using the acoustic levitation. A one-axis slide platform is developed to realize the flexibility of levitation. Also, Kinect sensor is used to implement remote control by transforming the gesture from operator to corresponding commands. The whole system is composed of acoustic levitation module, levitation module, and remote motion recognition control module, which can be applied in the transportation and operation of fragile or toxic micro objects.

Methodology

Acoustic levitation

There are mainly two kinds of ultrasonic levitation methods to levitate the objects in the air. One is near-field acoustic levitation and another one is standing wave levitation. Compared with near-field levitation, standing wave levitation is easily controlled in operation which is used in current article. As shown in Figure 1, the standing wave levitation requires two main parts: a transducer emits sound wave (green line) by vibration and a reflector bounces the sound back (red line). Once the transducer and the reflector are at suitable distance, a standing wave with the acoustic radiation force is formed in the field.

Figure 1.

Illustration of standing wave levitation and acoustic radiation force.

Actually, standing waves are a combination of two waves with the same amplitude, frequency, and traveling speed, but with the opposite direction. The incident wave and the reflected wave interfere and deconstruct each other. Then, a resultant, which is also called as stationary waves, is generated. It has been explained that the force applied on the objects F bases on the equation that ²⁵

$F=\frac{1}{3}\pi \rho {\left|A\right|}^2{\left(kR\right)}^3\text{sin}\left(2kh\right)\frac{5-2{\lambda }_p}{2+{\lambda }_p}$

1

where ρ is the density of medium, A is the incident amplitude, k is the number of waves which equals to 2π/λ, R is the radius of object, h is the distance between the center of object and the nearest node, and λp is the ratio of medium density to the bead density.

In addition, the gravity of levitated objects should be taken into consideration. For the bead with radius of R, its gravity

$G=\frac{4}{3}\pi R^3{\rho }_b\text{g}$

2

where ρ_b is the density of bead and g is the gravitational acceleration. Thus

$\frac{F}{G}=\frac{\rho {\left|A\right|}^2{k}^3\text{sin}\left(2kh\right)\left(5-2{\lambda }_p\right)}{4{\rho }_b\text{g}\left(2+{\lambda }_p\right)}$

3

The relationship between acoustic radiation force, gravity of bead, and vertical coordinate starting from transducer’s surface d is illustrated in Figure 2. Acoustic radiation force takes on a sinusoidal distribution along d, the amplitude of which is higher than gravity. Therefore, in actual vertical standing wave levitation, the objects are suspended slightly below or up to the nodes position due to the gravity effects. Once the density of object is small enough, the gravity can be ignored then levitation positions are almost coincident with nodes.

Figure 2.

Acoustic radiation force takes on a sinusoidal distribution and gravity is constant along vertical coordinate.

There are two physical factors affecting the maximum acoustic radiation force produced by transducer, which are the frequency of incident wave and the sound amplitude. ²² Here, the piezoelectric transducer (60 W, 28–40 kHz) was used since it consumes less power than the magnetostrictive transducer by converting the electrical energy into mechanical energy directly. The transducer and reflector are all arranged with its central axis in the gravitational direction.

During the standing wave levitation, objects can be levitated at different positions which depend on the wavelength of the incident wave that

$H=n\lambda /2(n=1,2,3...)$

4

where H is the length between transducer and reflector and λ is the wavelength, since

$\lambda =v/f$

5

the speed of sound in the air is approximately 340 m/s and the frequency of transducer chosen is 28.4 kHz, λ can be calculated approximately equaling to 11.97 mm. Thus

$H\approx 5.985n$

6

Therefore, a multiply of 5.985 mm, which is 95.76 mm, was set as the distance between the reflector and the transducer in this acoustic levitation device, which provides enough nodes for levitation.

Motion recognition control system

Kinect, which is developed by Microsoft, is prevalent in video games nowadays as a kind of motion sensing input devices. It is designed as a natural user interface system which allows interaction for human and machine using gestures and spoken commands rather than game controller. With the red, green, blue (RGB) color camera, depth sensor, and multi-array microphone, Kinect can provide full-body 3-D motion capture as well as facial and gesture recognition. Among them, the depth sensor which consists of an infrared laser projector and a monochrome complementary metal oxide semiconductor (CMOS) sensor is widely used for allowing the Kinect sensor to interpret the 3-D scenes from a continuously projected infrared structured light. ²⁶ Then, these data would be reconstructed by various image-based 3-D reconstruction to recover the depth of the observed points in the 3-D scene. On account of this feature, Kinect detects the human motions, which can be used for the remote controlling in our setup.

Skeleton detecting was applied to recognize and track the images of people who stand in front, which provides 25 position coordinates of joints in human body. Visual studio is used to program C language, and a Windows presentation foundation application is created with Kinect library. User interface is created as a canvas displaying the joints of operator. As for skeleton tracking, the skeleton data obtained provide the specific joint location information which can be applied to calculate the corresponding coordinates. We set the command mode as lifting right hand or left hand as shown in Figure 3. The distance between right hand and head less than 40 cm represents the command of moving up while distance between left hand and head less than 40 cm means moving down. Once the skeleton of operator has been tracked, gesture recognition is realized to remotely control the manipulation.

Figure 3.

Screenshot of user interface on Windows with corresponding human gestures. (a) Right hand upward only. (b) Left hand upward only.

Levitation platform

In order to stabilize the levitation experiment, a levitation platform is designed by fixing different parts together as shown in Figure 4.

Figure 4.

The design of ultrasonic levitation setup in the perspective view.

In this platform, the reflector and the transducer base were installed on the substrate placed between two lead screws. Also, two motors were fixed to drive the lead screw up and down. Then, a standing wave emitted from the transducer would move with the screws. By placing objects between the surface of the reflector and transducer, the objects can be levitated and move with the linear movement of the two screws.

Stepper motor is used as the actuator here, which is one type of brushless direct current (DC) electric motor that divides a full rotation into a number of equal steps. It can convert the input pulses into increment position of a shaft, in which each pulse moves the shaft through a fixed angle. Here a two-phase stepper motor (42BYG250-SASSML) with 1.8°stepper angle was used. Since the velocity of beads is determined by stepper motors and lead, traveling length of lead screw should be considered. The lead length L equals to the traveling linear distance during one rotation of the motor; thus, it should be as small as possible so that its linear travel distance can be easy to control by programming the rotation speed. In consideration of the standard products of lead screw, two 250 mm long lead screws with 2 mm lead length are installed. According to the property of stepper motor that

$v=n\cdot L/\left(60\cdot i\right)$

7

where v is the velocity of levitation device, n is the rotate speed of stepper motor, and i is the transmission ratio which is equal to 6, we can get

$v=n\cdot L/360$

8

In this device, stepper motor is set in 128 microsteppings; thus, the precision of stepper motor is nL/46080. Assembling these parts together, the whole platform was set up, as shown in Figure 5.

Figure 5.

Real setup of the overall design.

Results and discussion

Static levitation performance

Experiments were taken to test the stability of acoustic levitation with objects. According to equation (1), the density of levitated object determines the ability of levitation and the size of object shouldn’t exceed $\frac{\lambda }{2}$ . In practical experiments, objects with different shapes that weight within 2 g can be levitated successfully with this platform. Figure 6 exhibits the performance of polystyrene beads with 5 ± 0.5 mm diameter and 1.05 g/cm³ density as representative, which shows a good stability.

Figure 6.

Levitation with one bead.

Dynamic levitation performance

Corresponding experiments were taken by combing the stepper motor and levitation device. During the experiment, the bead could move as shown in Figure 7. At the beginning, the bead is staying at a lower position and after the transducer is adjusted at higher position by stepper motor, the bead moves upward.

Figure 7.

Result of the bead movement.

As for finding out the maximum linear speed, 500 r/min with 128 microstamping was successful to levitate and move the bead up and down. In this setting, according to equation (6), around maximum 2.7 mm/s linear travel speed for levitation can be reached. The position of bead captured by camera shows that the bead traveled around 20 mm during 7.5 s.

The measured traveling distance of bead during 10 trials is shown in Figure 8, which shows that the experimental displacement basically fits with the theoretical results with maximum speed. Errors are within 0.65 mm and the estimated standard error is 0.44 mm. It results from the precision of lead screw as well as stepping motor which is positively correlated with the rotation speed. Rotation speed higher than 500 r/min will generate acceleration leading to the instability of levitation.

Figure 8.

Displacement of bead during the lifting.

Gesture-based control of dynamic levitation performance

The whole device was set up by connecting levitation platform with Kinect. Figure 9(a) illustrates the working principle of it, which is that stepper motor receives the control from Kinect and actuates the movement of levitation transducer to operate the beads. The signal from Kinect is connected to stepper motor for control. The detection of right hand raising upward would actuate the levitation device move up while the detection of left hand raising up would lead to the descending of levitation device. According to this methodology, the whole experimental setup is built as shown in Figure 9(b).

Figure 9.

(a) Illustration of overall control methodology and (b) the whole experiment setup.

During this experiment, stepper motor is set with 180 r/min rotating speed rather than 500 r/min for better performance. Experiment results show that Kinect is successful to detect the human’s motions and give command to rotate the motors. The bead moves vertically by following the commands of human’s gestures at about 1 mm/s linear speed as shown in Figure 10.

Figure 10.

Bead movement with skeleton tracking by Kinect at 0 s, 13 s, 17 s, and 30 s.

The bead stays at the bottom of the scaffolding frame at the beginning and moves up when right hand lifted. The bead travels around 20mm to the middle of scaffolding frame during 13 seconds. Then the human changes from the right hand to the left at 14 s which takes 2 s. The bead starts going down at 16 s. Finally, the bead goes back to the original place at 30 s. The measured displacement of bead during this procedure is shown in Figure 11 with 0.18 mm maximum error and 0.083 mm estimated standard error that exhibits a good stability. Kinect processes and reacts instantaneously; thus, the time delay resulting from gesture recognition is negligible.

Figure 11.

Displacement of bead with gesture-based control.

The levitation and manipulation of the polystyrene beads is controlled by Kinect successfully, which shows a great stability in suspension at 1 mm/s linear speed. The accuracy of levitation distance was in millimeter level stable. In the future work, platform with more axis slides would be developed, and more gesture command would be designed to control the levitation device’s motion.

Conclusions

In this article, an ultrasonic robotic system is developed to manipulate small objects. Moveable acoustic levitation setup is built to implement noncontact operation. Also, Kinect sensing system is connected to detect human’s gesture and then control the manipulation stage. The experiment results exhibit that remote noncontact operation with polystyrene beads was realized with our setup. To conclude, the setup has been successfully built to manipulate the small objects in vertical direction by gesture.