A Visual Lifting Approach for Dynamic Bipedal Walking

Abstract

Although many papers have been published on visual tracking and visual servoing for vision-based robotics, there are only a few research projects studying using vision to improve standing and walking stabilization for legged robots. In this paper, we proposition a “Visual Lifting Bipedal Walking” strategy, which uses visually measured information to control the robot to keep a desired head-top's position/orientation in order to help prevent the robot from falling into unstable gaits, such as falling down on the ground or dangerous foot-slipping caused by gravity, unexpected coupling dynamics, etc.

1. Introduction

Humanoid robots are complex autonomous control systems that involve many technical issues which need to be solved, of which stable bipedal walking is the most fundamental. Therefore, this has been widely researched and a number of control approaches have been proposed to advance stable walking gaits.

The zero-moment-point (ZMP) approach is extensively utilized and has been shown to provide robust and effective locomotion for biped robots [1, 2]. The methodologies using ZMP firstly design a desired trajectory off-line and then derive body motions based on it. Since the natural dynamics of the robots –nonlinearity, coupling, changing of degrees of motion and the effects of the foot hitting the ground– are not considered, the ZMP methods often result in an unnatural-looking gait. Moreover, not every desired ZMP trajectory can be achieved because of the robot's kinematical limitation. In research [3] and [4] ZMP is not used and robot walking has been achieved by finding dynamically stable trajectories of joints through inverse kinematics calculations in advance from, e.g., the body's desired trajectory. This trajectory tracking approach makes walking gaits stable by following predetermined walking gaits, but it is difficult to realize adaptive walking in a complicated environment with unknown disturbances, where robots may encounter unpremeditated contact with, e.g., furniture and humans, or irregularities of the ground and having the controller reschedule the adaptive gaits' plan.

Therefore, online control methods for making the walking control system stable in the face of such disturbances have been intensively researched. Among them, walking control based on reflexive action is a method using sensory information to help the robot adapt to a time-varying environment. Qiang Huang et al. [5] have proposed a gait control consisting of a feedforward dynamic pattern and a feedback sensory reflex, which enabled a humanoid to effectively walk on unknown rough terrain and in an environment with disturbances. The sensory devices used in [5] include a foot-force sensor, body-inclination sensors and joint encoders.

As one of the most important sensors, vision has been used in lots of robot control research fields such as visual tracking and visual servoing (in which visual information is used to direct a manipulator toward a target object). Since the appearance of a manipulator is totally different from a humanoid with two legs, researchers have not yet fully considered how to use visual information to improve the walking stabilization for legged robots. However, if we do not consider the swing of the leg in walking, the body, from the foot (constricted to the ground) to the top of head can be considered as a linked manipulator. Thus, for this part of body, the “eye-in-hand” visual servoing concept can be used to keep a relatively constant position relationship between the head and a target object in the surrounding space, which is considered helpful in controlling the robot to prevent it from falling into unstable gaits, such as falling down to the ground or dangerous foot-slipping caused by gravity, unexpected coupling dynamics, etc…

Based on the above idea of using the visual servoing concept to control a humanoid robot, we newly proposition a “Visual Lifting Bipedal Walking” strategy which uses the vision sensory reflex to deal with the stabilization of the humanoid robot's bipedal standing and walking. Visual servoing can be classified into two major groups: position-based and image-based visual servoing [6]. Compared with image-based visual servoing, position-based visual servoing is more comprehensible, since a visual servo is more like a human-being in that it determines the object's position in a Cartesian coordinate frame and leads to Cartesian robot motion planning. Therefore, a position-based visual servoing concept is utilized in our research. We use cameras as the humanoid's eyes and a 3D model-based position measurement method that is proposed in our previous research ([7–9]) to observe an object that is set in the surrounding space.

One of the problems that have not been solved for online bipedal walking control concerns a singularity of the Jacobian matrix that occurs in walking robots. This problem occurs because the Jacobean matrix is determined as a function of the robot's shape and that is a result of the online dynamical motion, e.g., to avoid unpredictable disturbances, so the singularity cannot be prevented with preparation. Since this problem in stable walking is standing in the way of online adaptiveness, it will naturally be influenced by unpredictable disturbances so some researchers have started to work on how to retain stability [10, 11]. Considering this hazardous restriction: that the singularity of the Jacobian matrix connecting the velocity vector in Cartesian space and the joint velocity vector, prohibits the calculation of q̇ = J ( q )⁻¹ ṙ , in this paper, we proposed to use a Jacobian transposition J ( q )^T, which has an ability to project the force and torque exerted on the robot onto each joint space that is basically free from the singularity. On the other hand, a human looks like they are using a singular configuration to walk despite the above types of difficulty, thus, making the human's walking posture upright. From this view point, the strategy of utilizing J ( q ) ^T may have a wider potential than J ( q )⁻¹ to help the robot realize human-like walking.

J ( q ) ^T is used in our research to project stabilizing and feedback forces and torques that are calculated by visual measurements onto each joint, exemplifying the possibility for stable walking by using a dynamical model that includes various walking gait patterns. A drooped robot head may lead to falling to the ground, therefore we use the position deviation from a desired head position (height and orientation) to deduce the pulling force of the head by using joint torques, which are calculated through the Jacobian-transposition and the lifting force desired by the detected head position deviation. We will introduce this strategy in the following section.

2. Visual Lifting Bipedal Walking Strategy

Here, we use two cameras that are set as the humanoid's eyes to measure the position of an object set in the surrounding space and thus performing visual-feedback control to improve the humanoid's standing/walking stability.

We use a model-based matching method to measure the position of a target object denoted by φ based on the moving coordinates Σ _H , which represents the robot's head. We use a “1-step GA” method to solve the model matching optimization problem to achieve online visual position estimation [7].

The desired relative position of Σ _M (target object coordinate) and Σ _H is predefined by a Homogeneous Transformation as ^Hd T _M (t). The difference between the desired head position Σ _Hd and the current head position Σ _H is denoted as ^H T _Hd and can be described by:

${\hspace{0.17em}}^H{T}_{Hd}(\delta \phi (t))={\hspace{0.17em}}^H{T}_M(\widehat{\phi }(t)){\hspace{0.17em}}^{Hd}{T}_M^{-1}({\phi }_d(t))$

(1)

where ^H T _M (t) is calculated by $\widehat{\phi }(t)$ that is measured by an online visual position estimation method.

Here, the force exerted on the head to minimize the difference of the desired head position and the current position ^HT_Hd— the position deviation of the robot's head caused by gravity and walking dynamical influences — is considered to be directly proportional to δφ(t), so we have:

$f_v=k_v{}_{}\delta \phi (t)$

(2)

where k _v is a positive definitely diagonal matrix.

The joint torque τ_v to be input to realize the force f _v at the head follows the relation:

${\tau }_v=J_v{}^T{f}_v$

(3)

where J _v is a Jacobian matrix of the head position against joint angles [12].

The above visual lifting bipedal walking strategy is depicted in Fig.1. If the robot's head stays in the position we have determined as the desired position (Σ_H = Σ_Hd), then the head position difference δφ(t) is 0 and from Eq. (2) and Eq. (3) we know that there needs no joint torque to give a pulling force. However, if the robot's head position droops during the dynamic bipedal walking due to gravity or a dangerous slipping accident, then the head position difference δφ(t) is not 0 and the value can be calculated by Eq. (1). In this case, there needs to be a pulling force given by Eq. (2) to prevent the robot from falling down and a corresponding joint torque input to realize this force given by Eq. (3).

Figure 1.

Visual Lifting Bipedal Walking Strategy

Here, τ_v is used to control the humanoid robot in keeping a stable body position by using visual feedback information to compensate for a falling action caused by gravity or an unpredictable dangerous slipping motion etc… In order to drive the robot to perform a dynamic walking motion, additional torque is needed to work on the thigh joint in order to take a step forward:

${\tau }_{thigh}=A_0{}_{}{\cos }_{}{(\frac{2\pi }{T})}_{}t,$

(4)

Where A₀ is the amplitude that will influence the walking stride and T is the period of one walking cycle which influences the walking frequency.

τ_v and τ_thigh will be used together to control the humanoid robot in keeping a stable body position while walking forward. A simulation will be conducted to confirm the effectiveness of our proposed control method and the result will be shown later in this paper.

3. Bipedal Walking Model

The humanoid robot consists of a body, legs and eyes (Fig. 1) and has 10 degrees of freedom. Each link has a mass m, a length l, a radius of link r and an inertia moment I and each joint has a viscous friction D. When the robot walks forward the robot is getting closer to the static target object so it is necessary to change the position of the cameras to keep the object always in the view of both the left and right cameras. Therefore, the cameras are rotated about the z -axis to change eyes' gazing direction.

3.1 Dynamics of Walking

The equation of motion that includes point-constraint conditions, including contact-friction is written as:

$\begin{array}{l}M(q)\ddot{q}+h(q,\dot{q})+g(q)+D\dot{q}=\\ \tau +\left\{{(\frac{\partial C}{\partial q^T})}^T/\Vert \frac{\partial C}{\partial r^T}\Vert \right\}{f}_n-{(\frac{\partial r}{\partial q^T})}^T\frac{\dot{r}}{\Vert \ddot{r}\Vert }{f}_t\end{array}$

(5)

M is the inertia matrix, h and g indicate the Coriolis force, centrifugal force and gravity, and D is the matrix that indicates the coefficients of the joints' viscous friction. q is joint angle and τ is input torque. Then, f _n is the constraint force and f _t is friction. Here, we set out two reasonable assumptions: (i) f _n and f _t are orthogonal. (ii) f _t = k f _n ( k is a proportional constant).

In addition, C is defined as a point-constraint condition C( r ( q )) = 0, where r ( q ) is the lifting foot's position vector. This constraint condition happens to appear when the lifting foot's z-position in Σ_W is zero or below zero, otherwise the second and third terms of right-hand side of Eq. (5) disappear. The constraint condition concerning q̈ given by differentiating C( r ( q )) = 0 twice is written as:

${\dot{q}}^T\left\{\frac{\partial }{\partial q}(\frac{\partial \mathrm{C}}{\partial q^T})\right\}\dot{q}+(\frac{\partial \mathrm{C}}{\partial q^T})\ddot{q}=0$

(6)

Abbreviating the coefficient vectors of f _n and f _t in Eq. (4) by j _c ^T and j _t ^T , Eq.(5) and Eq.(6) can be combined as follows:

${\left[\begin{array}{cc}M(q)& -(j_c{}^T-j_t{}^{T}k)\\ \frac{\partial C}{\partial q^T}& 0\end{array}\right]}^{}\left[\begin{array}{c}\ddot{q}\\ f_n\end{array}\right]=\left[\begin{array}{c}\tau -h(q,\dot{q})-g(q)-D\dot{q}\\ {\dot{q}}^T\left\{\frac{\partial }{\partial q}(\frac{\partial \mathrm{C}}{\partial q^T})\right\}\dot{q}\end{array}\right]$

(7)

The size of matrix M ( q ) is generally very large so it needs a large amount of computation to calculate each element of M ( q ) using Lagrange method. In our research, we have 10 links, so M ( q ) is a 10×10 matrix. This implies that the analytical derivation of Eq. (5) is almost impossible by hand written calculation. A “Newton-Euler” method introduced in [13] can calculate such kinds of dynamical coefficients numerically and recursively through forward dynamics calculation. So we use this method in our simulation.

3.2 Walking Style Representation

The walking sequence of the flat-foot is more complicated than those of the point-foot and the round-foot, because each flat-foot has three contact cases: heel-contact, toe contact and foot contact. As shown in Fig. 2, there are several walking styles (I), (II), … (V) in the walking sequence. A walking process may not include all these styles; moving from which one to which phase of the motion depends on the equation of motion and some predefined conditions i.e. shape of the ground, size of the foot and so on. All dynamic parameters can affect the variety of the walking gait's motions.

Figure 2.

Style transition diagram for bipedal walking

Let the forward leg be expressed by “FL” and the backward leg be expressed by “BL”, we can describe the characteristics of each walking style as follows:

Style I: Surface-contacting (BL)

Link-0 is surface-contacting without slipping; regarded as a part of ground.

Style II: Point-contacting (BL)

Link-0 is point-contacting, but the constraint condition for Link-0 need not to be incorporated like in Eq.(5), that is, the second and third terms should be eliminated, since the motion of Link-0 can be added to the equation of motion as an additional state q₀, increasing the number of states of Eq(7) by one.

Style III: Surface-contacting(BL) and point-contacting (FL)

The heel of FL (tip of Link-6) is point-contacting and BL is surface-contacting which is the same as with (I). The constraint condition is required for the heel of FL.

Style IV: Point-contacting for both (BL) and (FL)

The heel of FL (tip of Link-6) is point-contacting and Link-0 is rotating which is the same with (II). The constraint condition is required for the heel of FL and the angle of Link-0 should be considered. In this case, the left-hand side of Eq. (5) is identical to the one in Style II.

Style V: Point-contacting (BL) and surface-contacting (FL)

In this walking style, FL becomes surface-contacting and is regarded as a part of ground. BL is point-contacting under the constraint condition. The backward leg can be used for exerting torque via the ankle.

3.3 Transition Conditions among Phases

(1) from (I) to (II) and from (III) to (IV)

This change of phase means that the heel of the rear-foot detaches from the ground in Phase (I) or (III). Firstly, the reference coordinate of ¹ f ₁ (exerting a torque on Link-1 whose vector is represented by the coordinates Σ₁ being fixed at Link-1) and ¹ n ₁ (exerting a torque on Link-1 in the same manner as ¹ f ₁) is converted from Σ₁ Then, a projection onto the z-axis, meaning the vertical axis of the ground, of ^W f ₁ = [ ^W f _1x, ^Wf_1y, ^Wf_1z] ^T and a projection onto the x-axis, representing the rotational axis of the foot, of ^W n ₁=[ ^W n _1x,^Wn_1y, ^Wn_1z] ^T are derived by using the unit vectors e _x = [1,0,0] ^T and e _z =[0,0,1] ^T . Given that the foot contacts the ground at the two points of the heel and tiptoe, ^W f _1z and ^W n _1z are dispersed and act on both ends of Link-0 as shown Fig. 3.

Figure 3.

Transition from (I) to (II) and from (III) to (IV)

Here, when the resultant forces that act on the rear/front of Link-0 are defined as F _h and F _t respectively, we can get the two equations below:

$F_h=\frac{{\hspace{0.17em}}^W{f}_{1z}}{2}+\frac{{\hspace{0.17em}}^W{n}_{1x}}{L}$

(8)

$F_t=\frac{{\hspace{0.17em}}^W{f}_{1z}}{2}-\frac{{\hspace{0.17em}}^W{n}_{1x}}{L}$

(9)

Thus, when the value of F _h becomes negative, Link-0 begins to rotate around the tiptoe, which means the heel detaches from the ground. For this reason, F _h < 0 is the condition expression for switching.

(2) from (I) to (III) or from (II) to (IV)

When the heel of the forefoot touches the ground, the phase is switched from (I) to (III) or from (II) to (IV). Therefore, given that the z-axis of the forefoot's heel is defined as z₆, the switching condition is z₆ < 0.

(3) from (III) to (V) or from (IV) to (V)

When Link-6, defined as a lifting foot's link, becomes surface-contacting, the phase is switched from (III) to (V) or from (IV) to (V). That is, when q_e = q₀+q₁+ … q₆ is defined as the angle between Link-6 and the ground as shown in Fig.2, the switching condition is: q_e≤0.

(4) from (V) to (I)

In phase (III) or (IV) Link-6 is renamed Link-6' since the robot's basic configurations of (V) and (I) are identical but the link's numbers are swapped over and derived from the left leg and the right leg being reversed. When the condition that the tiptoe detaches from the ground is satisfied phase (V) is switched to phase (I). In this posture, the constraint force f _n acts on Link-6'. That is, if the value of f _n is positive Link-6' is in contact with the ground. Meanwhile, if the value of f _n is negative Link-6' starts lifting and then the condition is: f _n < 0.

4. Stable Walking Simulation Results

In the simulation we use a triangle as the target for the robot to observe and the motion of the robot is controlled by the observation result (Fig. 4), this is somehow like the well-known “visual servoing” concept. However, the contribution of this kind of visual feedback control used here is newly proposed “to help the bipedal standing and walking stably by visual lifting” and the following simulation will show the effectiveness of our proposed approach.

The graphic presentation of the robot's motion is given by “Open GL” and the robot's parameter values used in the following analysis are shown in Table 1. Here the triangle is set in a plane parallel with the plane yoz of Σ _W , the distance to the yoz plane is 4.5[m]. The gazing direction of the right camera is changed and turned to right in order to observe the target.

Table 1.

Physical parameter values in simulations

Link Number	0	1	2	3	4	5	6	7	8
Mass [kg]	1.0	1.0	1.0	1.5	1.0	1.0	1.0		1.0
Length [m]	0.5	2.0	2.0	1.5	2.0	2.0	0.5	2.0	0.5
Radius [m]	(0.5, 0.8)	0.2	2.0	2.0	2.0	2.0	(0.5, 0.8)	2.0	0.5

Here, the visual-based control method described in Section 2 is used. Firstly, the position of the target object is measured by cameras using a proposed model-based matching method [7, 8], which gives the values of ${\hspace{0.17em}}^H{\mathit{T}}_M(\widehat{\phi }(t))$ . Next, we calculate the head position difference δφ(t) by Eq. (1) (in Eq. (1), the values of ^HdT M⁻¹(φ_d (t)) are given as a desired relative position). Then by using Eq. (2) and Eq. (3), we calculate the joint torque τ _v required to make the robot decrease δφ(t) to 0.

τ_v uses the visual feedback information to control the humanoid robot in keeping a stable body position and compensating for the falling action. However, to perform a bipedal walking motion, we need an additional torque τ_thigh to work on the thigh joint to take a step forward.

In this simulation, we input the thigh joint torque as: ${\tau }_{thigh}={200}_{}{\cos }_{}{(\frac{2\pi }{1.5})}_{}t$ . τ_v and τ_thigh are used together to control the humanoid robot in keeping a stable body position while walking forward. Since the robot is walking forward in this simulation, the target should be moving forward with a speed of 1[m/s] in order to be consistently observed in the walking process. The visual feedback gains k _V are set as (k_Vy, k_Vz, k_Vε), = (140,700,200).

The simulation results are shown in the following Figures (Fig. 5–Fig. 10).

Figure 4.

Graphic presentation of bipedal robot and the “visual lifting” process

Figure 5.

Head pose changing in the bipedal walking process by using “visual lifting control”. (a): Head trajectory of y-axis in Σ _W ; (b): Head trajectory of z-axis in Σ _W ; (c): Head pose changing, the angle rotates around x-axis of Σ _W , represented by quaternion. The straight line in each (a),(b) and (c) figure represents the desired value that is given beforehand.

Figure 5 shows the head trajectory y-coordinate (walking direction) and z-coordinate (upright direction) values in the walking process. The straight line in figures (a), (b) and (c) represents the desired value. Figure. 5(a) shows that the robot is walking forward with distance of 16[m] in about 16[s], (b) shows that the head height is decreasing while walking from 6.5[m] to 6.3[m] in the first 1[s] and that the height has been retained during the rest of the walking process. From Eq. (2) and Eq. (3), we can find that the error of 0.2[m] is the minimum value to produce a joint torque τ_v input to provide a lifting force against gravity and keep the robot standing stably.

Figure 6 shows the measured walking performance results and the straight line in figures (a), (b) and (c) represents the desired value that is given beforehand. Since the y-axis of Σ _H is the same direction as the z-axis in Σ _W Fig.6 (a) corresponds to Fig. 5(b), leaving a constant error of 0.2[m]. All of the oscillations in Fig. 5 and Fig 6 are periodic with a small amplitude, which implies the walking is stable.

Figure 6.

Measurement results in the bipedal walking process. (a): Position measurement of ^Hy_M; (b): Position measurement of ^Hz_M; (c): Orientation measurement of ^Hε_M, the angle rotates around x-axis of Σ _H , represented by quaternion. The straight line in each (a),(b) and (c) figure represents the desired value that is given beforehand.

Figure 7 shows how the walking style changes in this simulation. It is a rhythmic motion that is changing by Style I –> Style III –> Style V with a period of 1[s]. Figure 8 shows the leg trajectory expressed by phase portrait; the relation of the thigh angle (the angle between the two legs) to its velocity. In the initial posture the thigh angle is 0[rad]. When the walking starts, one leg takes the first step with the thigh angle changing from 0[rad] to about 0.6[rad], then in the second step the thigh angle returns back from 0.6[rad] to 0[rad] and the following steps continue in the same manner. Figure 9 is foot trajectory on the ground. The distance between each point is almost equal so we can say that the walking is stable.

Figure 7.

Walking style changing with respect to time.

Figure 8.

Leg trajectory of dynamic walking, expressed by phase portrait.

Figure 9.

Foot trajectory on the ground.

Figure 10 shows dynamic walking of the bipedal robot using the proposed “visual lifting approach”. The robot in the pictures describes stable dynamic walking with foot changes of Style I –> Style III –> Style V corresponding to Fig.7.

Figure 10.

Dynamic bipedal walking by visual lifting approach.

5. Conclusion

In this paper, we proposed a “Visual Lifting Bipedal Walking” strategy, by using visually measured information to control a robot to keep a desired head position/orientation to avoid falling down.

Simulation results have shown that the proposed “Visual Lifting” control scheme based on the visual servoing concept is effective in preventing the robot from falling down due to gravity. By inputting an additional torque on the robot's thigh joint, we successfully performed a bipedal walking simulation and the result has shown stable walking.

In future work, we will consider how to calculate a suitable torque that works on the robot's thigh joint to perform stable walking at different speeds. We hope our bipedal robot is able to walk faster and faster, and finally run; different walking styles will have to emerge in this motion. Even this current kind of control is very difficult, it is interesting and challenging.