A Framework for Obstacles Avoidance of Humanoid Robot Using Stereo Vision

Abstract

In this paper, we propose a framework for multiple moving obstacles avoidance strategy using stereo vision for humanoid robot in indoor environment. We assume that this model of humanoid robot is used as a service robot to deliver a cup to customer from starting point to destination point. We have successfully developed and introduced three main modules to recognize faces, to identify multiple moving obstacles and to initiate a maneuver. A group of people who are walking will be tracked as multiple moving obstacles. Predefined maneuver to avoid obstacles is applied to robot because the limitation of view angle from stereo camera to detect multiple obstacles. The contribution of this research is a new method for multiple moving obstacles avoidance strategy with Bayesian approach using stereo vision based on the direction and speed of obstacles. Depth estimation is used to obtain distance calculation between obstacles and the robot. We present the results of the experiment of the humanoid robot called Gatotkoco II which is used our proposed method and evaluate its performance. The proposed moving obstacles avoidance strategy was tested empirically and proved effective for humanoid robot.

1. Introduction

In recent years, humanoid service robots are developed for various applications such as the personal, medical and welfare robots. Moreover, vision-based service robots have recently made a rapid progress to tackle more complex manipulation activities such as, delivering a cup to an identified customer in restaurant environment. Technologies and methods are used for humanoid service robots increased drastically in order to make it more intelligent, and resulting these kind of robots available commercially especially as service robots.

Multiple moving obstacles avoidance in dynamic environments is an important problem in the field of humanoid service robots used in real world applications. In motion planning phase, motion behaviors of the humanoid robot can be classified into two fundamental behaviors: obstacle-avoidance and goal-seeking.

Visual perception is the ability to interpret the information and surroundings from the effects of visible light reaching the eye. The resulting perception is also known as eyesight, sight, or vision. Visual-perception-based of service robot for customer identification is an interpretation process to direct a service robot to a destination of identified customer based on face recognition system and computer vision. After interpretation of images from camera done, then it is used as information for the robot and to decide actions based on the task given by a developer. The basic of visual–perception model for a humanoid service robot is shown in fig. 1.

Figure 1.

Visual-perception model for vision-based humanoid robot. After interpretation process, the information used for navigating the robot or deciding actions for robot, such as direct a robot to customer's position.

Computer Vision is an important tool for robotics systems since it mimics the human sense of vision and allows for non-contact measurement of the environment. A good program using vision sensor will make a service robot have the ability to detect and identify detail object around it (such as face recognition, distance measurement of obstacle, and free area for path planning). The main concern when develop a humanoid service robot is obstacle avoidance system and the implementation of stereo camera as an important vision sensor nowadays. Although vision-based humanoid service robot become a challenging and complex task for researchers and engineers, models of the multiple moving obstacle avoidance system for humanoid service robot using Bayesian approach for state estimation has not been explored extensively.

The development of an obstacle avoidance system for humanoid service robot to accurately detect moving obstacles in indoors is a challenging task. Bayesian techniques provide a powerful statistical tool to help managing measurement uncertainty, perform multisensor fusion and identity estimation using minimum data. The advantages of probabilistic robotics are able to accommodate imperfect sensors (such as camera with noises), robust in real world applications and best known approach to many hard robotics problem. Using sensory information to locate the robot in its environment is the most fundamental problem for providing a service robot with autonomous capabilities. By using Bayesian approach in robot system, the state of presence of obstacles and other properties can be estimated.

Based on literatures obtained by the authors, many research in development of service robot are conducted such as [1, 2], whereas task of the service robot is the setting and clearing of tables in a controlled environment without stereo camera. An approach to static obstacle detection for collision-free, efficient humanoid robot navigation based on monocular images and sparse laser range data. To detect arbitrary obstacles in the surroundings of the robot, they analyze 3D data points obtained from a 2D laser range finder installed in the robot's head [3], and a framework of autonomous Mobile Robot Navigation using Active Stereo Vision [4].

However, there is no multiple moving obstacles avoidance method for humanoid service robot based in indoor environment are exposed especially by using stereo camera. The contribution of this paper is the introduction of a new method of multiple moving obstacles avoidance for service robots by using a stereo camera in indoor environment. In this paper, we improve the drawbacks of common obstacle avoidance methods such as Potential Field Method (PFM) and Vector Field Histogram (VFH) [13] by proposed a complete mechanism for multiple moving obstacles avoidance of vision-based humanoid service robot. Fig. 2 is an implementation of humanoid service robot using Minoru 3D stereo camera and Bioloid humanoid robot with height about 75 cm.

Figure 2.

The humanoid service robot used in this research using Minoru 3D Stereo camera, Bioloid robot and stereo images processed using Laptop

2. Proposed Framework and Method

2.1. A Framework of Multiple Moving Obstacles Avoidance Strategy

Because we want a general model for humanoid service robot, we propose a framework for multiple moving obstacles avoidance strategy using stereo vision. A multiple moving obstacle avoidance strategy is an important framework to develop humanoid service robot in dynamic environment. There are two mains actors on multiple moving obstacles avoidance system; the Robot itself and the Range Finder. The Robot interacts with this system to detect customer and determine moving obstacles. Both processes to detect customer and determine moving obstacles include a process of face recognition as explained before. After the obstacles are determined, The Range Finder (camera and its system) will calculate and estimate the distance of those moving obstacles and estimate the direction of moving obstacles. Direction estimation of obstacles will be used to determine optimal maneuver of the Robot to avoid those obstacles. The framework is shown in fig. 3:

Figure 3.

The use case diagram for our multiple moving obstacles avoidance strategy using stereo vision

2.2. Bayesian approach for humanoid robot

Camera as vision sensor sometimes have distortion, therefore Bayesian decision theory is used to state estimation and determine the optimal response for the robot based on inaccurate sensor data. Bayesian decision rule probabilistically estimate a dynamic system state from noisy observations. Examples of measurement data will include camera images and range scan. If x is a quantity that we would like to infer from y, the probability p(x) will be referred to as prior probability distribution. The updated Bayesian formula is applied to determine the new posterior p(x, y) whenever a new observation is obtained [10]:

$p\left(x,y\right)=\frac{p\left(y\text{|}x,z\right)p(x|z)}{p(y|z)}$

(3)

To apply Bayesian approach for obstacle avoidance where someone who walks with a direction indicated as an unexpected obstacle, we consider this obstacle to be a random event. The probabilistic information in z about θ is described by a conditional probability density function $p(z|\theta )$ of the observation vector z. Let Θ denote the state of the path to be a random variable consisting of four states:

$\mathrm{\Theta }=({\theta }_1,{\theta }_2,{\theta }_3,{\theta }_4)$

(4)

$=(\text{obstacle, no\_obstacle, norection\_rigle, norection\_left})$

If we want a service robot should stay on the path to goal in any case, strategies to avoid moving obstacle include:

• Maneuver to the right side, if detected moving obstacle is moving toward the left side. Maneuver to the left side, if detected moving obstacle is moving toward the right side.

• Stop, if moving obstacle too close to robot detected both by vision and ultrasonic sensors.

Then, we restrict the action space denoted as A as:

$A=\left(a_1,\hspace{0.17em}{a}_2,a_3\right)$

(5)

= maneuver to right, maneuver to left, stop

We define a loss function L(a, θ) which gives a measure of the loss incurred in taking action a when the state is θ. The robot should chooses an action a from the set A of possible actions based on the observation z of the current state of the path θ. This gives the posterior distribution of θ as:

$\mathrm{p}(\theta |\mathrm{z})=\frac{\mathrm{p}(\mathrm{z}|\theta )\mathrm{p}(\theta )}{{\displaystyle \sum \mathrm{p}(\mathrm{z}|\theta )\mathrm{p}(\theta )}}$

(6)

Then, based on the posterior distribution in (12), we can compute the posterior expected loss of an action [11]:

$\mathrm{B}(\mathrm{p}(\theta |\mathrm{z}),\mathrm{a})={\displaystyle \sum _{\theta }\mathrm{L}(\theta ,\mathrm{a})\mathrm{p}(\theta |\mathrm{z})}$

(7)

2.3. Stereo Imaging Model

We have developed a system for face detection using Haar cascade classifier and depth estimation for measuring distance of peoples as moving obstacles using stereo vision. In the stereo imaging model, the tree-dimensional points in stereo camera frame are projected in the left and the right image frame. On the contrary, using the projection of the points onto the left and right image frame, the three-dimensional points positions in stereo camera frame can be located. Fig. 4 shows the stereo imaging model using the left camera frame LC and right camera frame RC [5].

Figure 4.

Stereo Imaging Model

By using stereo vision, we can obtain the position of each moving obstacle in the images, then we can calculate and estimate the distance of the moving obstacle. Kalman filtering is used for the stability of the distance measurement. The three-dimensional point in stereo camera frame can be reconstructed using the two-dimensional projection of point in left front image frame and in right front image frame using formula:

${\hspace{0.17em}}^{\text{SC}}q=\left[\begin{array}{c}{\hspace{0.17em}}^{\text{SC}}\mathrm{q}{}_{\mathrm{x}}\\ {\hspace{0.17em}}^{\text{SC}}\mathrm{q}{}_{\mathrm{y}}\\ {\hspace{0.17em}}^{\text{DC}}\mathrm{q}{}_{\mathrm{z}}\end{array}\right]=\frac{2}{{\hspace{0.17em}}^{\text{RI}}\mathrm{q}{}_{\mathrm{x}}-{\hspace{0.17em}}^{\text{LI}}\mathrm{q}{}_{\mathrm{x}}}\left[\begin{array}{c}\frac{1}{2}\hspace{0.17em}\mathrm{a}\hspace{0.17em}\left({\hspace{0.17em}}^{\text{RI}}\mathrm{q}{}_{\mathrm{x}}\text{+}{\hspace{0.17em}}^{\text{LI}}\mathrm{q}{}_{\mathrm{x}}\right)\\ \hspace{0.17em}\mathrm{a}\hspace{0.17em}{\hspace{0.17em}}^{\text{LI}}\mathrm{q}{}_{\mathrm{y}}\\ \mathrm{f}\hspace{0.17em}\mathrm{a}\end{array}\right]$

(8)

Note that ${}^{\text{LI}}\mathrm{q}{}_{\mathrm{y}}={}^{\text{RI}}\mathrm{q}{}_{\mathrm{y}}$

To estimate the direction θ _direction of moving obstacle using stereo vision, we calculate using the fig. 5 and formula below:

Figure 5.

Direction estimation using stereo vision

${\theta }_{direction}=\text{arc\hspace{0.17em}tan}\left(\Delta q_z/\Delta q_x\right)$

(9)

2.4. Improved Face Recognition System

The face and illumination are our primary focus of attention in developing a vision-based service robot to serve peoples. Unfortunately, developing a computational model of face recognition is quite difficult, because faces are complex, meaningful visual stimuli and multidimensional. Modeling of face images can be based on statistical model such as Principal Component Analysis (PCA) [12] and Linear Discriminant analysis (LDA) [6] and physical modeling based on the assumption of certain surface reflectance properties, such as Lambertian surface [7]. We have developed ITS face database consists of 3 poses (front, left, right) and varied with illumination with total 12 images per people. For testing the database, we use 12 images randomly with variation of illumination generated by our program that successfully implemented in our previous research [8, 9].

2.5. Proposed Maneuvering Strategy

We have proposed a method and algorithm of obstacles avoidance for service robot that run from start to destination, giving a cup to customer and going back to home. This method will identify a customer, checking moving obstacles and its distance and take action for maneuver to avoid the collision. Stereo camera usage have a limitation of angle view, this camera only able to capture object in front of camera about 30°. So, when the robot starts to maneuver, the moving obstacle could be out of view area of camera. Therefore, we proposed a predefined motion for maneuver based on the estimation speed and direction of moving obstacles as shown in Table 1:

Table 1.

Actions to avoid moving obstacle

No	Speed ofmoving obstacle	Direction ofmoving obstacle	Action
1	Low	Approach to robot	Maneuver slow
2	High	Approach to robot	Maneuver fast
3	Low	In front of robot	Maneuver slow
4	High	In front of robot	Maneuver slow

Fig. 6 below shows the proposed model of maneuvering on the service robot, pL which is the probability of moving obstacle leads to the left, and pR the probability of moving obstacle leads to the right. By estimating the direction of motion of the obstacle, then the most appropriate action to avoid obstacles with maneuver to the right / left side can be determined, to minimize collisions with these obstacles.

Figure 6.

A maneuvering model to avoid multiple moving obstacles using stereo vision, 2 multiple moving obstacles with the different direction (a) and the same direction (b)

If there are more than 1 moving obstacle, then robot should identify the nearest moving obstacle for avoidance, and the direction of maneuver should be the opposite side of the direction of moving obstacle in order to minimize the possibility of collision between obstacle and the robot

The flowchart of a Navigation system and multiple moving obstacles avoidance method for vision-based humanoid robot using stereo camera is shown in fig. 7. Based on the fig. 7, image is captured by stereo camera used as testing images to be processed by Haar classifier to detect how many people in the images, and face recognition by PCA. We implement visual tracking to identify customer for heading a robot to a customer. Robot continuously measures the distance of obstacles and send the data to Laptop. The next step is multiple moving obstacle detection and tracking. If there is no moving obstacle, robot runs from start to destination position in normal speed. If there are moving obstacles, the obstacles will e tracked and the speed and distance information are obtained. The advantage using stereo vision in our system is the ability to estimate the distance of customers/obstacles (depth estimation) and direction's movement of obstacles and stabilized by Kalman filtering.

Figure 7.

A Navigation system and Multiple moving obstacles avoidance strategy for vision-based humanoid robot using stereo camera

After distance of moving obstacle is measured by Kalman filtering, obstacle tracking, direction and speed are estimated by using Bayesian, the next step is to check the collision. If collision occurs, predefined maneuver will be identified to avoid obstacle. The proposed algorithm for multiple moving obstacle avoidance strategy is shown at algorithm 1.

// check if a cup is loaded or not Checking a cup Capture face's images Face detection and recognition using PCA if cup loaded and face recognized // Visual tracking using stereo vision While (customer !=center screen) begin Heading robot to customer's position end if (position of customer at center screen) begin Go to customer call movingObstaclesIdentification call Bayesian processing if moving obstacle==true and min_distance=true and goal=false maneuvering the robot end if Giving a cup Go to home end end if end // Function to detect and track a moving obstacle function movingObstacleIdentification // Using Haar cascade classifier moving obstacle detection if (moving_obstacle==true) then //estimate distance between robot and moving obstacle distance estimation // Using Kalman filtering // estimate velocity and direction of moving obstacle Calculate v_O direction Endif Return v_O, direction end function

Algorithm 1. Algorithm for multiple moving obstacles avoidance strategy for humanoid service robot

The flowchart for do a maneuver is shown in fig. 8

Figure 8.

A flowchart for moving obstacles detection.

3. Experimental Result and Discussion

The robot used in this experiment even though not so high, but can be used to test our framework. The result of identifying multiple moving obstacles is shown in figure below. The advantage if we using stereo vision: we can estimate the distance and direction/angle of the moving obstacle. The strategy to avoid obstacles by maneuver with opposite direction suitable in the real scenario and the probability value of obstacle/no obstacle with information for distance for each obstacle also run well and make our framework more robust as shown in fig. 9.

Figure 9.

Robot successfully identify 2 moving obstacles (a), distance and direction estimation both using stereo vision (b)

Table 2 provides a comparison between our results and common methods PFM and VFH to prove that our method has the important additional feature that it can be used with low-cost sensors:

Table 2.

Comparison between PFM, VFH and our method

No	Variables	PFM	VFH	Our method
1	Provide a complete mechanism for a vision-based service robot	x	x	√
2	Low-cost sensors	√	x	√
3	Fast obstacle avoidance	√	√	√
4	No oscillation in the presence of obstacles	x	√	√

Based on the experiment, the robot successfully maneuvered to avoid moving obstacles, after that it went to destination point to deliver a cup to the customer as shown in fig. 10

Figure 10.

A Path result from maneuvering strategy to avoid obstacles from start to goal position

4. Conclusion

This paper presents a robust multiple moving obstacles avoidance method for humanoid robot in indoor environment. The improved face recognition system has been proposed and can be used for the vision-based service robot. Geometrical model for multiple moving obstacles avoidance and algorithms obstacles are proposed and implemented for service robot. Experimental results with various situations have shown that the robot reached the destination points while avoiding multiple moving obstacles with the proposed method. The state estimation using Bayesian for existence and direction of obstacles increase the success rate of obstacle's detection and measurement. Bayesian decision rule implemented for state estimation makes this method robust because the optimal solution for avoiding obstacles is obtained by trading between maneuver and stop action. We also have contribution for solving the drawbacks of common obstacle avoidance methods by proposed a complete mechanism for multiple moving obstacle avoidance of vision-based service robot. For future work, we will focus on grasping system for humanoid service robot using stereo vision.