Hybrid control for combining model-based and model-free reinforcement learning

Abstract

We develop an approach to improve the learning capabilities of robotic systems by combining learned predictive models with experience-based state-action policy mappings. Predictive models provide an understanding of the task and the dynamics, while experience-based (model-free) policy mappings encode favorable actions that override planned actions. We refer to our approach of systematically combining model-based and model-free learning methods as hybrid learning. Our approach efficiently learns motor skills and improves the performance of predictive models and experience-based policies. Moreover, our approach enables policies (both model-based and model-free) to be updated using any off-policy reinforcement learning method. We derive a deterministic method of hybrid learning by optimally switching between learning modalities. We adapt our method to a stochastic variation that relaxes some of the key assumptions in the original derivation. Our deterministic and stochastic variations are tested on a variety of robot control benchmark tasks in simulation as well as a hardware manipulation task. We extend our approach for use with imitation learning methods, where experience is provided through demonstrations, and we test the expanded capability with a real-world pick-and-place task. The results show that our method is capable of improving the performance and sample efficiency of learning motor skills in a variety of experimental domains.

1. Introduction

Reinforcement learning (RL) algorithms can generally be divided into two categories—model-free and model-based. Model-free methods avoid the need to model the environment or dynamics by learning a direct policy mapping between states and actions (Schulman et al., 2017; Haarnoja et al., 2018a,b). The policy is learned through experience, so we consider model-free methods to be “experience-based.” Model-based methods typically learn a dynamics model and a reward function to predict the next state and next reward given the current state and a potential next action. These outputs can be sequentially used to predict into the future the expected rewards for a set of actions. These methods have contrasting strengths and weaknesses. Model-free approaches require significantly more data and diverse experience to learn the task than model-based methods (Chua et al., 2018), but model-free approaches often produce better performance than model-based methods. The prediction capabilities of model-based methods make them more “sample-efficient” in solving robot learning tasks (Williams et al., 2017; Chua et al., 2018; Abraham et al., 2020), but the models are often highly complex and can require special structure (Nagabandi et al., 2018; Havens et al., 2019; Sharma et al., 2019; Abraham et al., 2020; Abraham et al., 2017; Abraham and Murphey, 2019). Is there a way to leverage the beneficial aspects of each of these methods to enable robotic systems to rapidly learn tasks with a limited amount of experience?

Recent work tries to address this question by exploring alternate ways of structuring environment models by combining probabilistic models with deterministic components (Chua et al., 2018). Other work has explored using latent-space representations to reduce model complexity (Havens et al., 2019; Sharma et al., 2019). Related methods use high fidelity Gaussian Processes to create models, but these methods are limited by the amount of data that can be collected (Deisenroth and Rasmussen, 2011). Finally, some researchers try to improve experience-based methods by adding exploration as part of the objective (Pathak et al., 2017). However, all of these approaches focus on improving either model-based or model-free methods separately rather than combining model-based planning with experience-based learning.

Methods that combine model-based planning and experience-based learning tend to do so in stages (Chebotar et al., 2017; Bansal et al., 2017; Nagabandi et al., 2018). First, a model is used to collect data for a task to jump-start the learning process. Then, supervised learning is used to update a policy (Levine and Abbeel, 2014; Chebotar et al., 2017) or an experience-based method is used to continue the learning from that stage (Nagabandi et al., 2018). Moreover, the model is often used as an oracle, which provides labels to a base policy. The aim of these methods is to train a policy that does not rely on the model after training. While these approaches do improve learning, they do not algorithmically leverage the two learning approaches in an optimal manner, resulting in objective mismatch (Lambert et al., 2020). The sequential optimization solves two different problems; the model-based controller seeks to improve the objective, while the policy seeks to match the model-based controller.

Our approach algorithmically combines model-based and experience-based learning by using the learned model to predict how well an experience-based policy will behave, and then optimally update the resulting actions. Using hybrid control as the foundation for our approach, we derive a controller that optimally uses model-based actions when the policy is uncertain, and allows the algorithm to fall back on the experience-based policy when there is high confidence that the actions will result in a favorable outcome. As a result, our approach does not rely on improving the model (but can easily integrate high fidelity models), but instead optimally combines the actions generated from model-based and experience-based methods to achieve high performance in the specified task. Our contributions in this work can be summarized as follows:

• We present a hybrid control theoretic approach to robot learning motor skills.

• We derive deterministic and stochastic algorithmic formulations of our proposed hybrid learning approach.

• We introduce a measure for determining the agreement between learned model and policy.

• We demonstrate the flexibility of our approach with various learning approaches such as off-policy reinforcement learning (Haarnoja et al., 2018a) and behavior cloning (Pomerleau, 1998).

• We demonstrate improved sample efficiency and task performance over state-of-the-art model-based methods (Ansari and Murphey, 2016; Williams et al., 2017), model-free methods (Haarnoja et al., 2018b), and methods that combine model-based and model-free methods (Feinberg et al., 2018; Buckman et al., 2018; Janner et al., 2019; Montgomery and Levine, 2016; Nagabandi et al., 2018).

The paper is structured as follows: Section 2 provides background knowledge of the problem statement and its formulation; Section 3 introduces our approach and derives both deterministic and stochastic formulations of our algorithm; Section 4 provides simulated results and comparisons as well as experimental validation of our approach; Section 5 provides comparisons to related work that combines model-based and model-free methods; and Section 6 concludes the work and discusses future work.

2. Background

In this work, we build on the framework of Markov Decision Processes (MDP) and aspects of hybrid control theory. Robot reinforcement learning problems are often formulated as MDPs. The MDP formulation assumes the Markov property—that the result of an action taken in a given state only depends on the current state and does not depend on the prior history. The Markov property applies to both model-based and model-free methods, which are used in this work. Prior work which combines model-free and model-based methods often relies on hand-tuned objective functions or ultimately solves two different objective functions for the different learning methods. In this work, we introduce hybrid control theory as a structured approach to combining multiple control strategies with a single objective function.

Markov decision processes

A Markov Decision Process (MDP) is a mathematical framework for a discrete-time stochastic process. The goal of the MDP formulation is to find a mapping from state to action that maximizes the total reward acquired from interacting in an environment for some fixed amount of time. An MDP can be represented as $\mathrm{ℳ}=\left\{\mathcal{S},\mathcal{A},r,p\right\}$ , which contains a set of accessible continuous states $s\in \mathcal{S}$ the robot can be in, a set of continuous bounded actions $a\in \mathcal{A}$ that a robot may take, rewards r, and a transition probability p(s_t+1 | s _t , a _t ). For stochastic actions, the transition probability represents the probability of transitioning from one state s _t to the next s_t+1 given an action a _t applied at time t. For deterministic actions, the transition probability specifies a fixed next state s_t+1 given state s _t and action a _t . With the MDP framework, the total reward goal can be written as the objective

${\pi }^{\mathrm{\star }}=\underset{\pi }{\arg \hspace{0.17em}\max }{\mathrm{E}}_{a\sim \pi \left(\cdot |s\right)}\left[{\displaystyle \sum _{t=0}^{T-1}r}\left(s_t\right)\right],$

(1)

where the solution is an optimal policy π^⋆.

Model-free approaches learn a stochastic policy a ∼ π(· | s) that maximizes the reward r at a state s. Model-based approaches solve the MDP problem by modeling a transition function s_t+1 = f(s _t , a _t ) and a reward function r _t = r(s _t ) ¹ . Model-based methods either use these functions to construct a policy or directly generate actions through model-based planning (Chua et al., 2018). When the transition model and the reward function are known, the MDP formulation becomes an optimal control problem. We can use any set of existing methods (Li and Todorov, 2004; Tang et al., 2018) to solve for the best set of actions (or policy) to maximize the reward ² .

Hybrid control theory for mode scheduling

In mode scheduling problems, the goal is to maximize a reward function through the synthesis of two (or more) control strategies. A common example of mode switching is when a vertical takeoff and landing vehicle switches from flight mode to landing mode. In hybrid control theory, these control strategies are often called modes. Thus, mode switching can also be called policy switching (Axelsson et al., 2008; Vasudevan et al., 2013). We can use hybrid control theory to determine the optimal time to switch from one control policy to another. Most mode scheduling problems are formulated in continuous time and are subject to dynamics of the form

$\dot{s}\left(t\right)=f\left(s\left(t\right),a\left(t\right)\right)=g\left(s\left(t\right)\right)+h\left(s\left(t\right)\right)a\left(t\right)$

(2)

where $f\left(s,a\right):\mathcal{S}\times \mathcal{A}\to \mathcal{S}$ is the transition function. The state-transition function is comprised of the free dynamics $g\left(s\right):\mathcal{S}\to \mathcal{S}$ , the control map $h\left(s\right):\mathcal{S}\to \mathcal{S}\times \mathcal{A}$ , and an action a(t). This formulation also encompasses nonlinear state-transition functions.

The continuous time dynamics function requires us to reformulate the objective function from (1). First, we introduce an objective function as a deterministic function of the state and control

$\begin{array}{l}\underset{\tau ,\lambda }{\arg \max \mathcal{J}={\displaystyle {\int }_{t=0}^{t_H}r}\left(s\left(t\right)\right)dt}\\ \text{subject to}\dot{s}\left(t\right)=f\left(s\left(t\right),a\left(t\right)\right),s\left(0\right)=s_0\end{array}$

(3)

where

$a\left(t\right)=\{\begin{array}{ll}\widehat{a}\left(t\right),\hfill & \text{if }t\in \left[\tau ,\tau +\lambda \right],\hfill \\ a_{\text{def}}\left(t\right)\hfill & \text{otherwise}\hfill \end{array}$

(4)

t _H is the time horizon (in seconds), and s(t) is generated from some initial condition s(0) using a model (2) and an action sequence (4). The goal in hybrid control theory is to find the optimal time τ to switch from some default set of actions a_def to another set of actions $\widehat{a}$ for application duration λ to best improves the performance of the objective in (3) subject to the dynamics (2). The following section derives the algorithmic combination of the MDP learning formulation with the hybrid control foundations into a joint hybrid learning approach.

3. Hybrid learning

The goal of this section is to introduce hybrid learning as a method for optimally synthesizing model-based and model-free learning methods. In Section 2, we introduced Markov Decision Processes and described how model-based and model-free methods follow the MDP formulation. Although both methods assume an underlying MDP, only the model-based method learns this transition probability. The transition probability is a key feature linking the MDP formulation to hybrid control theory. Most mode scheduling problems are constrained by a dynamics function, which governs the transition from a current state s _t to a next state s_t+1 given a current action a _t . Thus, the MDP transition probability and the mode scheduling dynamics function have the same form. In this work, we use the model-based prediction capability to develop a hybrid control approach to determine the optimal time to switch from the action specified by experience-based policy to an alternate model-based action.

We first introduce a deterministic formulation of the algorithm with theoretical proofs describing the foundations of our method. We then derive a stochastic method as a way to relax the assumptions made in the deterministic formulation. In both the deterministic and stochastic formulations, we solve the learning problem indirectly. We break down the overall learning problem into solvable sub-problems, which collectively imply the original learning problem has been solved. Finally, we extend our method to use expert demonstrations as experience. Both the deterministic and stochastic approaches can be improved by providing informative experience in the form of expert demonstrations.

3.1. Deterministic method

Consider a continuous time MDP formulation of the objective in (3) and the dynamics in (2), where f and r are learned using arbitrary regression methods (e.g., neural network least squares, Gaussian processes), and the experience-based policy π is learned through a model-free approach (e.g., policy gradient (Sutton et al., 2000)). We assume the learned policy has the form $\pi \left(a|s\right)=\mathcal{D}\left(\mu \left(s\right),\mathrm{\Sigma }\left(s\right)\right)$ , where $\mathcal{D}$ is a normal distribution and μ(s), Σ(s) are the mean and variance of the policy as a function of state. Additionally, we define the default action in (4) as the mean of the policy π and ignore uncertainty for the time being ³ , so a_def(t) = μ(s(t)). To determine the form of $\widehat{a}$ , let us first calculate how sensitive the objective in (3) is at any τ to switching from $\mu \left(s\right)\to \widehat{a}$ for an infinitely small λ ⁴ .

Lemma 1

Assume f, r, and μ are differentiable and continuous in time. The sensitivity of the objective in (3) with respect to the duration time λ of switching from μ(s) to $\widehat{a}$ at any time τ ∈ [0, t _H ] is defined as

$\frac{\partial }{\partial \lambda }\mathcal{J}\left(\tau \right)=\rho {\left(\tau \right)}^{\mathrm{\top }}\left(f_2-f_1\right){|}_{\tau }$

(5)

where f₁ = f(s(t), μ(s(t))), $f_2=f\left(s\left(t\right),\widehat{a}\left(t\right)\right)$ , and $\rho \left(t\right)\in \mathcal{S}$ is the adjoint variable. The adjoint variable is the solution to the differential equation

$\dot{\rho }\left(t\right)=-\frac{\partial r}{\partial s}-{\left(\frac{\partial f}{\partial s}+{\frac{\partial \mu }{\partial s}}^{\mathrm{\top }}\frac{\partial f}{\partial a}\right)}^{\mathrm{\top }}\rho \left(t\right)$

(6)

with terminal condition ρ(t _H ) = 0. The derivative of the objective in (3) with respect to time duration λ is also known as the mode insertion gradient (Axelsson et al., 2008; Vasudevan et al., 2013).

Proof

First, we define the trajectory

$\begin{array}{l}s\left(t_H\right)=s\left(0\right)+{\displaystyle {\int }_0^{\tau }f}\left(s\left(t\right),\mu \left(s\left(t\right)\right)\right)dt+{\displaystyle {\int }_{\tau }^{\tau +\lambda }f}\left(s\left(t\right),\widehat{a}\left(t\right)\right)dt\\ +{\displaystyle {\int }_{\tau +\lambda }^{t_H}f}\left(s\left(t\right),\mu \left(s\left(t\right)\right)\right)dt\end{array}$

(7)

generated from $a\left(t\right)=\{\begin{array}{ll}\widehat{a}\left(t\right)\hfill & \text{if }t\in \left[\tau ,\tau +\lambda \right]\hfill \\ a_{\text{def}}\left(t\right)\hfill & \text{otherwise}\hfill \end{array}$ where a_def(t) = μ(s(t)). Next, we take the derivative of (3) with respect to the time duration λ to get the expression

$\frac{\partial }{\partial \lambda }\mathcal{J}={\displaystyle {\int }_{\tau +\lambda }^{t_H}{\frac{\partial r}{\partial s}}^{\mathrm{\top }}}\frac{\partial s}{\partial \lambda }dt$

(8)

Using (7), we define the derivative of the state s with respect to the time duration λ as

$\frac{\partial s\left(t\right)}{\partial \lambda }=f_2-f_1+{\displaystyle {\int }_{\tau +\lambda }^t{\left(\frac{\partial f}{\partial s}+{\frac{\partial \mu }{\partial s}}^{\mathrm{\top }}\frac{\partial f}{\partial a}\right)}^{\mathrm{\top }}}\frac{\partial s\left(\sigma \right)}{\partial \lambda }d\sigma$

(9)

where σ is a placeholder for time under the integrand and f₁ = f(s(t), μ(s(t))) and $f_2=f\left(s\left(t\right),\widehat{a}\left(t\right)\right)$ are boundary terms from applying Leibniz’s rule. The derivative in (9) is a linear convolution with initial condition $\frac{\partial s}{\partial \lambda }\left(\tau \right)=f_2-f_1$ , so we can rewrite (9) using a state-transition matrix as

$\frac{\partial s\left(t\right)}{\partial \lambda }=\mathrm{\Phi }\left(t,\tau \right)\left(f_2-f_1\right)$

(10)

where

$\mathrm{\Phi }\left(t,\tau \right)=\exp \left({\left(\frac{\partial f}{\partial s}+{\frac{\partial \mu }{\partial s}}^{\mathrm{\top }}\frac{\partial f}{\partial a}\right)}^{\mathrm{\top }}\left(t-\tau \right)\right)$

(11)

Plugging (10) into (8) and pulling the initial condition term out from under the integrand gives

$\frac{\partial }{\partial \lambda }\mathcal{J}\left(\tau \right)=\underset{\lambda \to 0}{\text{lim}}{\displaystyle {\int }_{\tau +\lambda }^{t_H}{\frac{\partial r}{\partial s}}^{\mathrm{\top }}}\mathrm{\Phi }\left(t,\tau \right)dt\left(f_2-f_1\right)$

(12)

Taking the limit of (12) as λ → 0 gives the instantaneous sensitivity from switching from $\mu \to \widehat{a}$ at any time τ. We define this sensitivity term as the adjoint variable

$\rho {\left(\tau \right)}^{\mathrm{\top }}={\displaystyle {\int }_{\tau }^{t_H}{\frac{\partial r}{\partial s}}^{\mathrm{\top }}}\mathrm{\Phi }\left(t,\tau \right)dt$

(13)

Plugging the adjoint variable back into (12) gives the mode insertion gradient

$\frac{\partial }{\partial \lambda }\mathcal{J}\left(\tau \right)=\rho {\left(\tau \right)}^{\mathrm{\top }}\left(f_2-f_1\right)$

(14)

where the adjoint can be rewritten as the differential equation

$\dot{\rho }\left(t\right)=-\hspace{0.17em}\hspace{0.17em}\frac{\partial r}{\partial s}-{\left(\frac{\partial f}{\partial s}+{\frac{\partial \mu }{\partial s}}^{\mathrm{\top }}\frac{\partial f}{\partial a}\right)}^{\mathrm{\top }}\rho \left(t\right)$

(15)

with terminal condition ρ(t _H ) = 0.

Lemma 1 gives us the proof and definition of the mode insertion gradient (5). The mode insertion gradient tells us the change in the objective function when switching from the default policy behavior μ to some other arbitrarily defined control $\widehat{a}$ for a small time duration λ. The mode insertion gradient can show how an arbitrary action changes task performance relative to the learned policy. In this work, we additionally use the mode insertion gradient as a method for obtaining the best action the robot can take given the learned predictive model of the dynamics and the task rewards. We can take a direct approach by asking the following question:

Given a suboptimal policy π, what is the best action the robot can take to maximize the objective in (3), at any time t ∈ [0, t _H ], subject to the uncertainty (or certainty) of the policy defined by Σ(s)?

We approach this new sub-problem by specifying the auxiliary optimization problem

$a^{\mathrm{\star }}\left(t\right)=\underset{\widehat{a}\left(t\right)\forall t\in \left[0,t_H\right]}{\arg \max }{\displaystyle {\int }_0^{t_H}\frac{\partial }{\partial \lambda }}\mathcal{J}\left(t\right)+\log \pi \left(\widehat{a}\left(t\right)|s\left(t\right)\right)dt$

(16)

This new problem maximizes the mode insertion gradient by finding the action a^⋆ that results in the greatest change in the objective. The problem also penalizes the action a^⋆ for deviating from the policy when there is high confidence in the policy from prior experience. The penalty is introduced by the log  π term.

Theorem 1

Assuming f, r, and π are continuous and differentiable in s, a and t, the best possible action to improve the performance of (3) and is a solution to (16) for any time t ∈ [0, t _H ] is

$a^{\mathrm{\star }}\left(t\right)=\mathrm{\Sigma }\left(s\left(t\right)\right)h{\left(s\left(t\right)\right)}^{\mathrm{\top }}\rho \left(t\right)+\mu \left(s\left(t\right)\right)$

(17)

where the adjoint ρ(t) is defined in (6) and $h\left(s\right):{ℝ}^n\to {ℝ}^{n\times m}$ is the affine mapping from actions to the dynamics.

Proof

The mode insertion gradient can be rewritten (5) as

$\begin{array}{l}\frac{\partial \mathcal{J}}{\partial \lambda }={\rho }^{\mathrm{\top }}\left(f\left(s,\widehat{a}\right)-f(s,\mu \left(s\right)\right)\\ ={\rho }^{\mathrm{\top }}\left(g\left(s\right)+h\left(s\right)\widehat{a}-g\left(s\right)-h\left(s\right)\mu \left(s\right)\right)\\ ={\rho }^{\mathrm{\top }}h\left(s\right)\left(\widehat{a}-\mu \left(s\right)\right)\end{array}$

(18)

where f(s, a) = g(s) + h(s)a, and we drop the dependency on time for clarity. Inserting the mode insertion gradient into (16), taking the derivative with respect to the point-wise $\widehat{a}$ , and setting the equation equal to zero gives

${\rho }^{\mathrm{\top }}h\left(s\right)\left(\widehat{a}-\mu \left(s\right)\right)-\mathrm{\Sigma }{\left(s\right)}^{-1}\left(\widehat{a}-\mu \left(s\right)\right)=0$

(19)

Solving for $\widehat{a}$ gives the best actions a^*

$a^{\mathrm{\star }}\left(t\right)=\mathrm{\Sigma }\left(s\left(t\right)\right)h{\left(s\left(t\right)\right)}^{\mathrm{\top }}\rho \left(t\right)+\mu \left(s\left(t\right)\right)$

(20)

which is the action that maximizes the mode insertion gradient subject to the certainty of the policy π for all t ∈ [0, t _H ].

The proof in Theorem 1 provides the best action a robotic system can take given a default experience-based policy. Each action generated uses the sensitivity of changing the objective based on the predictive model’s behavior while relying on the experience-based policy to regulate when the model information will be useful. We convert the result in Theorem 1 into our first algorithm, a deterministic approach to hybrid learning (see Alg. 1).

With the proposed approach, we are able to numerically determine the contribution of the learned predictive models towards improving the task. We show (17) provides the best possible action given the current belief of the dynamics f and the task reward r. Next, we use the control Hamiltonian to show we can find a local optima of the objective in (17).

Algorithm 1

Hybrid Learning (Deterministic)

1: Randomly initialize continuous differentiable models f(s, a), r(s, a) with parameters ψ and policy π(a | s) with parameter θ. Initialize memory buffer $\mathcal{D}$ , prediction horizon parameter t _H , exploration noise ɛ.

2: while task not done do

3:   reset environment and exploration noise ɛ

4:   for i = 1, …, T do

5:   observe state s(t _i )

   ▷ simulation loop

6:   for ${\tau }_i\in \left[t_i,\dots ,t_i+t_H\right]$ do

     ▷ forward predict states using any integration method (Euler shown)

7:     s(τ_i+1) = s(τ _i ) + f(s(τ _i ), μ(s(τ _i )))dt

8:     r(τ _i ) = r(s(τ _i ), μ(s(τ _i )))

9:   end for

     ▷ backwards integrate

10:   ρ(t _i + t _H ) = 0

11:   for ${\tau }_i\in \left[t_H+t_i,\dots ,t_i\right]$ do

12:     $\dot{\rho }\left(t\right)=-\frac{\partial r}{\partial s}-{\left(\frac{\partial f}{\partial s}+{\frac{\partial \mu }{\partial s}}^{\mathrm{\top }}\frac{\partial f}{\partial a}\right)}^{\mathrm{\top }}\rho \left(t\right)$

13:     $\rho \left({\tau }_{i-1}\right)=\rho \left({\tau }_i\right)-\dot{\rho }\left({\tau }_i\right)dt$

14:   end for

15:    $a^{\mathrm{\star }}\left(t_i\right)=\mathrm{\Sigma }\left(s\left(t_i\right)\right)h{\left(s\left(t_i\right)\right)}^{\mathrm{\top }}\rho \left(t_i\right)+\mu \left(s\left(t_i\right)\right)+\epsilon \left(t\right)$

16:   apply to robot

17:   append data

18:   end for

   ▷ Update model and policy ⁵

19:  Update f, r by sampling N data points from $\mathcal{D}$ using any regression method

20:  Update π using any experience-based method

21: end while

Corollary 1

Assuming $\frac{\partial }{\partial a}\mathrm{\mathscr{H}}\ne 0$ where $\mathrm{\mathscr{H}}=r\left(s\right)+\log \pi \left(a|s\right)+{\rho }^{\mathrm{\top }}f\left(s,a\right)$ is the control Hamiltonian for the objective in (3), then $\frac{\partial }{\partial \lambda }\mathcal{J}={\Vert h{\left(s\right)}^{\mathrm{\top }}\rho \Vert }_{\mathrm{\Sigma }\left(s\right)}>0$ and is zero when the policy satisfies the control Hamiltonian condition $\frac{\partial }{\partial a}\mathrm{\mathscr{H}}=0$ .

Proof

Inserting (17) into (5) yields

$\begin{array}{l}\frac{\partial \mathcal{J}}{\partial \lambda }={\rho }^{\mathrm{\top }}\left(g\left(s\right)+h\left(s\right)\left(\mathrm{\Sigma }\left(s\right)h{\left(s\right)}^{\mathrm{\top }}\rho +\mu \left(s\right)\right)-g\left(s\right)-h\left(s\right)\mu \left(s\right)\right)\\ ={\rho }^{\mathrm{\top }}h\left(s\right)\mathrm{\Sigma }\left(s\right)h{\left(s\right)}^{\mathrm{\top }}\rho ={\Vert h{\left(s\right)}^{\mathrm{\top }}\rho \Vert }_{\mathrm{\Sigma }\left(s\right)}>0\end{array}$

(21)

From Pontryagin’s Maximum principle, a solution is a local optima of the objective function if it satisfies the following

$\frac{\partial }{\partial a}\mathrm{\mathscr{H}}=-\mathrm{\Sigma }{\left(s\right)}^{-1}\left(a-\mu \left(s\right)\right)+h{\left(s\right)}^{\mathrm{\top }}\rho =0$

(22)

when a = Σ(s)h(s)^Tρ + μ(s) or ρ = 0. Therefore, if the policy π is a solution, then the adjoint ρ = 0 and π must be a solution to the optimal control problem (3).

Corollary 1 tells us the action defined in (17) generates the best action to improve the performance of the robot given valid learned models. Corollary 1 also states that if the policy is already a solution, then our approach for hybrid learning does not modify the known solution and simply returns the policy’s action.

The preceding proofs have a strict requirement of continuity and differentiability of the learned models and the policy. As these constraints are not always possible and often learned models have noisy derivatives, our goal is to try to reformulate the objective in (3) into an equivalent problem that can be solved without these strict assumptions. One approach is to reformulate the problem in discrete time as an expectation. This formulation is introduced in the following section.

3.2. Stochastic method

For the stochastic approach, we relax the continuity, differentiability, and continuous-time restrictions specified in the deterministic objective in (3) by first restating the objective as an expectation

$\max \hspace{0.17em}{\mathrm{E}}_{v\sim \pi (\cdot |s)}\left[\mathcal{J}\left(v\right)\right]$

(23)

where $\mathcal{J}\left(v\right)={\displaystyle {\sum }_{t=0}^{T-1}r}\left(s_t\right)$ is subject to transition dynamics s_t+1 = f(s _t , v _t ), and v = [v₀, …v_H−1] is a sequence of H randomly generated actions from the experience-based policy π. Rather than trying to find the best time τ and discrete duration λ, we approach the problem from a hybrid information theoretic view. In this framework, we want to find the best actions to augment π and improve the objective.

We find the best augmented action by defining two distributions $\mathrm{\mathbb{P}}$ and $\mathbb{Q}$ . Distribution $\mathrm{\mathbb{P}}$ is the uncontrolled system response distribution, ⁶ and distribution $\mathbb{Q}$ is the open-loop control distribution. Distributions $\mathrm{\mathbb{P}}$ and $\mathbb{Q}$ are described by probability density functions

$p\left(v\right)={\displaystyle \prod _{t=0}^{T-1}\pi }\left(v_t|s_t\right)$

(24)

and

$q\left(v|a\right)={\displaystyle \prod _{t=0}^{T-1}\frac{\exp \left(-\frac{1}{2}{\left(v_t-a_t\right)}^{\mathrm{\top }}\mathrm{\Sigma }{\left(s_t\right)}^{-1}\left(v_t-a_t\right)\right)}{\sqrt{{\left(2\pi \right)}^m\left|\mathrm{\Sigma }\left(s_t\right)\right|}}}$

(25)

respectively, where q(v | a) uses the same variance Σ(s) as the policy $\pi \left(a|s\right)=\mathcal{N}\left(\mu \left(s\right),\mathrm{\Sigma }\left(s\right)\right)$ . The uncontrolled distribution $\mathrm{\mathbb{P}}$ represents the default predicted behavior of the robotic system under the learned policy π. The open-loop control distribution $\mathbb{Q}$ allows us to define a probability of an augmented action, but more importantly, distribution $\mathbb{Q}$ provides a free variable, which we will use to optimize the learned models.

Following the work in Williams et al. (2017), we use Jensen’s inequality and importance sampling on the free-energy definition (Theodorou and Todorov, 2012) of the control system using distribution $\mathbb{Q}$ to get the relationship

$\begin{array}{l}\mathrm{ℱ}\left(v\right)=-\frac{1}{\gamma }\log \left({\mathrm{E}}_{\mathrm{\mathbb{P}}}\left[\exp \left(\gamma \mathcal{J}\left(v\right)\right)\right]\right)\\ \le -\frac{1}{\gamma }{\mathrm{E}}_{\mathbb{Q}}\left[\log \left(\frac{p\left(v\right)}{q\left(v|a\right)}\exp \left(\gamma \mathcal{J}\left(v\right)\right)\right)\right]\end{array}$

(26)

where $\gamma \in {ℝ}^{+}$ here is what is known as the inverse temperature parameter. In (26), if $\frac{p\left(v\right)}{q\left(v|a\right)}\propto 1/\exp \left(\gamma \mathcal{J}\left(v\right)\right)$ , then the inequality becomes a constant. Further reducing the free-energy gives

$\mathrm{ℱ}\left(v\right)\le -{\mathrm{E}}_{\mathbb{Q}}\left[\mathcal{J}\left(v\right)-\frac{1}{\gamma }\log \left(\frac{p\left(v\right)}{q\left(v|a\right)}\right)\right]$

(27)

which is the optimal control problem we want to solve plus a bounding term. The bounding term keeps the augmented actions close to the policy. By making the bound a constant, we can use the free-energy formulation to solve the hybrid control problem indirectly.

Now, we can define an optimal distribution ${\mathbb{Q}}^{\mathrm{\star }}$ as a density function

$q^{\mathrm{\star }}\left(v\right)=\frac{1}{\eta }\exp \left(\gamma \mathcal{J}\left(v\right)\right)p\left(v\right)$

(28)

where $\eta ={\displaystyle {\int }_{\mathrm{\Omega }}\exp }\left(\gamma \mathcal{J}\left(v\right)\right)p\left(v\right)dv$ and Ω is the sample space. ⁷ We can use the ratio $\frac{p\left(v\right)}{q^{\mathrm{\star }}\left(v\right)}\propto 1/\exp \left(\gamma \mathcal{J}\left(v\right)\right)$ to make the free-energy a constant. However, we cannot directly sample from ${\mathbb{Q}}^{\mathrm{\star }}$ . We want to generate a separate set of actions a _t defined in q(v | a) to augment the policy π given the learned model f, so our goal is to push q(v | a) towards q^*(v). As done in Williams et al. (2017, 2016), this goal corresponds to the optimization

$a^{\mathrm{\star }}=\underset{a}{\text{arg}\hspace{0.17em}\text{min}}{D}_{\text{KL}}\left({\mathbb{Q}}^{\mathrm{\star }}|\mathbb{Q}\right)$

(29)

This optimization minimizes the Kullback-Leibler divergence of the optimal distribution ${\mathbb{Q}}^{\mathrm{\star }}$ and the augmented open-loop distribution $\mathbb{Q}$ .

Returning to our original problem of how to find the best actions to augmented π and improve the objective in (23), we now we want to construct a distribution to augment the policy distribution p(v) and improve the objective.

Theorem 2

The recursive, sample-based, solution to (29) is

$a_t^{\mathrm{\star }}=a_t+{\displaystyle \sum _k\omega }\left(v_t^k\right)\delta a_t^k$

(30)

where $\omega \left(v\right)=\frac{\exp \left(\gamma \mathcal{J}\left(v\right)\right)p\left(v\right)}{{\displaystyle {\sum }_k\exp }\left(\gamma \mathcal{J}\left(v\right)\right)p\left(v\right)}$ , k denotes the sample index, and v _t = a _t + δa _t .

Proof

Expanding the objective in (29), we can show

$\begin{array}{l}{a}^{\mathrm{\star }}=\underset{a}{\text{arg}\hspace{0.17em}\text{min}}{\mathrm{E}}_{{\mathbb{Q}}^{\mathrm{\star }}}\left[\log \left(\frac{q^{\mathrm{\star }}\left(v\right)}{q\left(v|a\right)}\right)\right]=\underset{a}{\text{arg}\hspace{0.17em}\text{min}}{\displaystyle {\int }_{\mathrm{\Omega }}{q}^{\mathrm{\star }}}\left(v\right)\log \left(\frac{q^{\mathrm{\star }}\left(v\right)}{p\left(v\right)}\frac{p\left(v\right)}{q\left(v|a\right)}\right)dv\\ =\underset{a}{\text{arg}\hspace{0.17em}\text{min}}\left({\displaystyle {\int }_{\mathrm{\Omega }}{q}^{\mathrm{\star }}}\left(v\right)\log \left(\frac{q^{\mathrm{\star }}\left(v\right)}{p\left(v\right)}\right)dv-{\displaystyle {\int }_{\mathrm{\Omega }}{q}^{\mathrm{\star }}}\left(v\right)\log \left(\frac{q\left(v|a\right)}{p\left(v\right)}\right)dv\right)\\ =\underset{a}{\text{arg}\hspace{0.17em}\max }{\displaystyle {\int }_{\mathrm{\Omega }}{q}^{\mathrm{\star }}}\left(v\right)\log \left(\frac{q\left(v|a\right)}{p\left(v\right)}\right)dv\end{array}$

(31)

Defining the policy $\pi \left(v_t|s_t\right)=\mathcal{N}\left(\mu \left(s_t\right),\mathrm{\Sigma }\left(s_t\right)\right)$ as normally distributed, we can show

$\begin{array}{l}\frac{q\left(v|a\right)}{p\left(v\right)}\propto \exp \left({\displaystyle \sum _t\left(-\frac{1}{2}{\left(v_t-a_t\right)}^{\mathrm{\top }}{\mathrm{\Sigma }}^{-1}\left(v_t-a_t\right)\right)+\frac{1}{2}{\left(v_t-\mu \left(s_t\right)\right)}^{\mathrm{\top }}{\mathrm{\Sigma }}^{-1}\left(v_t-\mu \left(s_t\right)\right)}\right)\\ =\exp \left({\displaystyle \sum _t\left(-\frac{1}{2}{a}_t^{\mathrm{\top }}{\mathrm{\Sigma }}^{-1}{a}_t+a_t^{\mathrm{\top }}{\mathrm{\Sigma }}^{-1}{v}_t+\frac{1}{2}\mu {\left(s_t\right)}^{\mathrm{\top }}{\mathrm{\Sigma }}^{-1}\left(\mu \left(s_t\right)-2v_t\right)\right)}\right)\end{array}$

where Σ = Σ(s) is used for clarity. Plugging this expression into (31) gives

$a^{\mathrm{\star }}=\underset{a}{\text{arg}\hspace{0.17em}\text{max}}\left({\displaystyle \sum _t\left(-\frac{1}{2}{a}_t^{\mathrm{\top }}{\mathrm{\Sigma }}^{-1}{a}_t\right)+a_t^{\mathrm{\top }}{\displaystyle {\int }_{\mathrm{\Omega }}{q}^{\mathrm{\star }}}\left(v\right){\mathrm{\Sigma }}^{-1}{v}_{t}dv+\frac{1}{2}\mu {\left(s_t\right)}^{\mathrm{\top }}{\displaystyle {\int }_{\mathrm{\Omega }}{q}^{\mathrm{\star }}}\left(v\right){\mathrm{\Sigma }}^{-1}\left(\mu \left(s_t\right)-2v_t\right)dv}\right)$

(33)

We can solve (33) for a _t at each time by setting the derivative with respect to a _t equal to zero. Thus, we get the optimal solution

$a_t^{\mathrm{\star }}={\displaystyle {\int }_{\mathrm{\Omega }}{q}^{\mathrm{\star }}}\left(v\right)v_{t}dv$

(34)

The expression $q^{\mathrm{\star }}\left(v\right)\propto \exp \left(\gamma \mathcal{J}\left(v\right)\right)p\left(v\right)$ allows us to rewrite (34) as

$a_t^{\mathrm{\star }}={\displaystyle {\int }_{\mathrm{\Omega }}{q}^{\mathrm{\star }}}\left(v\right)v_{t}dv={\displaystyle {\int }_{\mathrm{\Omega }}\frac{1}{\eta }}\exp \left(\gamma \mathcal{J}\left(v\right)\right)p\left(v\right)v_{t}dv={\mathrm{E}}_{\mathrm{\mathbb{P}}}\left[\frac{1}{\eta }\exp \left(\gamma \mathcal{J}\left(v\right)\right)v_t\right]$

(35)

where $\eta ={\displaystyle {\int }_{\mathrm{\Omega }}\exp }\left(\gamma \mathcal{J}\left(v\right)\right)p\left(v\right)dv$ .

Using the change of variable v _t = a _t + δa _t , we get the recursive, sample-based solution

$a_t^{\mathrm{\star }}=a_t+{\displaystyle \sum _k\omega }\left(v_t^k\right)\delta a_t^k$

(36)

where

$\omega \left(v\right)=\frac{\exp \left(\gamma \mathcal{J}\left(v\right)\right)p\left(v\right)}{{\displaystyle \sum _k\exp }\left(\gamma \mathcal{J}\left(v\right)\right)p\left(v\right)}$

(37)

With the stochastic formulation, we can generate samples from the stochastic policy and evaluate its utility based on the current belief of the dynamics and the reward function. Because samples directly depend on the likelihood of the policy, any actions deviating too far from the policy will be penalized proportional to the confidence of the policy. The inverse is also true—when the policy has low confidence (high variance), the sample span will increase and the model-based information will have higher weight. Additionally, we do not have to place continuity and differentiability constraints on the learned models and can utilize arbitrarily complex models in this algorithm. We outline the stochastic algorithm for hybrid learning Alg. 2.

Algorithm 2

Hybrid Learning (Stochastic)

1: Randomly initialize continuous differentiable models f(s, a), r(s, a) with parameters ψ and policy π(a | s) with parameter θ. Initialize memory buffer $\mathcal{D}$ , prediction horizon parameter H, sample size parameter K, inverse temperature parameter γ.

2: while task not done do

3:  reset environment

4:  for t = 1, …, T − 1 do

5:   observe state s _t

   ▷ simulation loop

6:   for k ∈ {0, …, K − 1} do

7:    for τ ∈ {0, …, H − 1} do

8:      $v_{\tau }^k\sim \pi \left(\cdot |s_{\tau }^k\right)$

     ▷ forward predict state and reward

9:      $s_{\tau +1}^k,r_{\tau }^k=f\left(s_{\tau }^k,v_{\tau }^k\right),r\left(s_{\tau }^k,v_{\tau }^k\right)$

10:      $j_{\tau }^k=r_{\tau }^k$

11:    end for

12:   end for

      ▷ update actions

13:   for τ ∈ {0, …, T − 1} do

14:     $\mathcal{J}\left(v_{\tau }^k\right)\leftarrow {\displaystyle {\sum }_{t=\tau }^{T-1}{j}_t^k}$

15:     $\delta a_{\tau }^k\leftarrow v_{\tau }^k-a_{\tau }$

16:     $\omega \left(v_{\tau }^k\right)\leftarrow \frac{\exp \left(\gamma \mathcal{J}\left(v_{\tau }^k\right)\right)p\left(v_{\tau }^k\right)}{{\displaystyle {\sum }_k\exp }\left(\gamma \mathcal{J}\left(v_{\tau }^k\right)\right)p\left(v_{\tau }^k\right)}$

17:     $a_{\tau }\leftarrow a_{\tau }+{\displaystyle {\sum }_{k=0}^{K-1}\omega }\left(v_{\tau }^k\right)\delta a_{\tau }^k$

18:   end for

19:   apply a₀ to robot

20:   append data $\mathcal{D}\leftarrow \left\{s_t,a_0,r_t,s_{t+1}\right\}$

21:  end for

   ▷ Update model and policy ⁸

22:  Update f, r by sampling N data points from $\mathcal{D}$ using any regression method

23:  Update π using any experience-based method

24: end while

3.3. Imitation learning

One limitation of reinforcement learning approaches is the large amount of data required for training—even for sample-efficient methods. Both the deterministic and stochastic hybrid learning approaches could be improved by being provided with informative, expert demonstrations of the task. Therefore, we extend our method to use expert demonstrations as experience. Imitation learning (Argall et al., 2009; Ross and Bagnell, 2010) focuses on using expert demonstrations to either mimic a task or to initialize learning complex data-intensive tasks. We use imitation learning, specifically behavior cloning, as an initialization for how a robot should accomplish a task. Hybrid learning as described in Section 3.1 and 3.2 is then used as a method to embed model-based information to compensate for the uncertainty in the learned policy, improving the overall performance through planning. We outline the algorithmic implementation hybrid learning with behavior cloning in Appendix C.

4. Experiments

This section compares the previously described methodology to state-of-the-art model-based and model-free methods. First, we evaluate our deterministic and stochastic variations on a range of simulated benchmark control tasks from OpenAI Gym (Brockman et al., 2016). Then, we extend the evaluation of our stochastic method to the real world with a robotic arm manipulation experiment. Finally, we present two behavior cloning experiments; one experiment uses an OpenAI Gym environment, and the other demonstrates a real-world pick-and-place experiment with a robotic arm. All implementation details are included in Appendix B. Each experiment shown in this section was trained on ten different random seeds. Unless otherwise stated, the solid curves in the following figures correspond to the mean, and the shaded regions correspond to the standard deviation over the ten trials. ⁹

4.1. Simulated benchmarks

We evaluate our approach on a subset of environments from the OpenAI Gym benchmarks (Brockman et al., 2016). Specifically, we look at the cartpole swingup environment, the Acrobot swingup environment, the hopper environment, and the half-cheetah environment, which are described in more detail in Appendix B. We evaluate our proposed approach on common RL benchmarks to verify our measure for model and policy alignment. In particular, the goal is to show that our method (1) is more sample efficient, (2) obtains higher rewards than the comparison methods even though it is a model-based RL method, and (3) improves the performance of the individual model and policy that are being learned over time.

Deterministic results

We compare our hybrid learning approach to model-based method in Ansari and Murphey (2016) and model-free method (Soft Actor-Critic) in Haarnoja et al. (2018b).

While model-free methods have exploration naturally encoded in their formulation, the deterministic model-based and hybrid learning approaches require added exploration noise to induce exploring other regions of state space. Figure 1 summarizes our deterministic benchmark results. The top row shows snapshots of each Gym environment. The middle row shows that our method outperforms both the model-based and model-free methods for all four tested environments. Our hybrid learning approach uses the confidence bounds generated by the stochastic policy to infer when best to rely on the policy or predictive models. As a result, hybrid learning enables learning in unstructured environments comparable to model-free methods with the sample efficiency of model-based learning approaches. For the swingup tasks, the model-based method learns the task but converges to a less optimal policy than our hybrid approach. For the locomotion tasks, the model-based method suffers from its inability to model discontinuous dynamics.

Figure 1.

Performance curves of our proposed deterministic hybrid learning algorithm on multiple environments during training (averaged over 10 random seeds). All methods use the same structured learning models. Our method is shown to improve the model-based benchmark results (due to the use of experience-based methods) while maintaining significant improvements on the number of interactions necessary with the environment to obtain those results. The mode insertion gradient is also shown for each example which illustrates the model-policy agreement over time. Note: Markers are included for distinguishing between methods only and do not represent all data points.

The bottom row shows the mode insertion gradient for the hybrid learning method for each environment. The mode insertion gradient can be interpreted as a measure of agreement between the policy and the learned model. The swingup examples show an eventual reduction in the mode insertion gradient. We interpret this as the convergence of the learned model and policy. The locomotion examples do not converge in the number of environment interactions observed. The lack of convergence indicates that the learning process has not yet stabilized, which is consistent with the episode rewards learning curves (in the middle row).

Stochastic results

We next compare the stochastic formulation of hybrid learning to a stochastic neural network model-based controller (NN-MPPI) (Williams et al., 2017) and the same model-free controller as the deterministic formulation (SAC) (Haarnoja et al., 2018b). The goal of these experiments is to show that this hybrid learning formulation still outperforms these state-of-the-art learning techniques without imposing continuity and differentiability requirements. Figure 2 summarizes our stochastic benchmark results. The model-free parameters are held constant from the deterministic results to remove any impact of hyperparameter tuning. Figure 2 shows that for both swingup tasks, the stochastic formulation outperforms the model-free method but performs similarly to the improved model-based method. Even with the improved model-based method, the model-based method converges to a less optimal policy for the Acrobot than our hybrid approach. For the locomotion tasks, our stochastic formulation maintains the improved performance and sample efficiency over the model-based method and the model-free method. Exploration is naturally encoded into the stochastic algorithm, which results in more stable learning when there is uncertainty in the task.

Figure 2.

Performance curves of our proposed stochastic hybrid learning algorithm on multiple environments during training (averaged over 10 random seeds). Our approach improves both the sample efficiency and the highest expected reward. Note: Markers are included for distinguishing between methods only and do not represent all data points.

Although we no longer have a mode insertion gradient for the stochastic method, we can analyze the individual learned model and policy obtained from hybrid learning. In Figure 3, both the environment interactions and episode rewards are normalized to allow comparisons between the different environments. For all environments, hybrid learning is shown to improve the learning capabilities of both the learned predictive model and the policy through the hybrid control approach. The policy is “filtered” through the learned model and augmented, allowing the robotic system to be guided by both the prediction and experience. Thus, both the predictive model and the policy benefit ultimately performing better as a standalone approach using hybrid learning. Figure 3 shows that for different tasks, the model and policy learn at different rates, but both evolve over time with our hybrid learning approach.

Figure 3.

Comparison of model and policy evolution during stochastic hybrid learning. Episode rewards and environment interactions have been normalized to allow comparisons between different environments. For the swingup tasks, both the models and policies quickly learn the task. For the locomotion tasks, models also learn over the course of training, but the policies learn at a faster rate. These plots show that the hybrid learning approach pushes performance of both the model and the policy. Note: Markers are included for distinguishing between methods only and do not represent all data points.

Comparisons

We can compare learned models and policies at 50,000 steps for both the deterministic and stochastic implementations. Figure 4 shows 10 tests of each final model-based, model-free, and hybrid learning method. In addition to the full hybrid method, we show the individual model-based and model-free policies trained during hybrid learning as “Hybrid Learning Model-Only” and “Hybrid Learning Policy-Only,” respectively. In addition to having greater performance with the full implementation of hybrid learning, our approach tends to push the performance of both the model and policy. For the swingup tasks, the model-based method outperformed the hybrid model-only evaluation, but both the full hybrid learning and hybrid policy-only evaluations outperform all other methods. This result indicates that the hybrid learning approach does not require model improvements once it has learned a “good enough” model. For the locomotion tasks, the deterministic policy-only hybrid learning on average outperforms the deterministic full hybrid learning approach. For the stochastic method, the results are reversed with the full hybrid learning approach outperforming the policy-only approach. These results indicate that if the goal is to produce animations in simulation, the policy-only controller is likely sufficient to control the simulation. If the goal is to control real-world systems with uncertainty, maintaining the full hybrid controller allows continued benefit from the learned predictive model and experience-based policy.

Figure 4.

Comparison of final model-based and model-free policies for both deterministic and stochastic methods. For the hybrid learning method, we show the final model-only and policy-only results separately as well as in combination. Each boxplot displays the results of 100 tests (10 seeds, 10 episodes each). The interquartile region (IQR) contains 50% of the data (from Q1 to Q3) and is shown as a filled box with notches at the median. The whiskers span [Q1 - 1.5×IQR, Q3 + 1.5×IQR]. Any data points falling outside the whiskers are designated as outliers and are shown as unfilled circles. The means are shown as white circles. For all methods, the hybrid policy-only and full hybrid learning outperform all other methods.

4.2. Robot learning from experience

Next, we apply hybrid learning to an experiment with a Sawyer robot to validate hybrid learning for real robot tasks. For this experiment, the goal is for the robot to access a block surrounded by clutter. The true positions of the “clutter” blocks and the “target” block are unknown to the robot. The robot’s state is comprised of vectors from the robot’s end-effector to each block. The “target” block is always placed in the same start location, but the “clutter” blocks are randomly placed around the “target” block. The robot is rewarded for pushing the “clutter” blocks out of the way and accessing the “target” block (see Figure 5 for task illustration).

Figure 5.

Hybrid learning results on the Sawyer robot (averaged over 10 trials). The time series on the right provides a visualization of the task. The task is to access a target block (shown in green) surrounded by clutter (five other blocks) through environment interactions. Our stochastic hybrid learning method is able to achieve the task by effectively using both predictive models and experience-based methods. For additional visualization, see Extension 1. Note: Markers are included for distinguishing between methods only and do not represent all data points.

What makes this task difficult is that the robot must learn to push the “clutter” blocks out of the way rather than directly reaching towards the “target” block. Since our method naturally relies on the predictive models when the policy is uncertain, the robot is able to plan through the clutter to achieve the task. Figure 5 shows that our proposed hybrid learning approach outperforms both the model-based and model-free methods. During testing, we observed that the model-based approach tries to directly “reach” for the block without first clearing the clutter, which often results in pushing the target block outside workspace. Although the model-free method rarely pushes the block outside the workspace, it takes significantly longer for the model-free method to discover the pushing dynamics. Our hybrid approach learns to push the clutter blocks out of the way before accessing the target block. These tactics can also be viewed in Extension 1.

4.3. Behavior cloning

The final set of experiments illustrates our algorithm with imitation learning. For both experiments, expert demonstrations are used to generate the experience-based policy through behavior cloning and the learned predictive models adapt to the uncertainty in the policy.

Simulated benchmark results

We test hybrid imitation on the 3D Pybullet Ant environment (Coumans and Bai, 2016). The goal is for the four legged ant to run as far as it can to the right (from the viewer’s perspective) within the allotted time. At each iteration, we provide the agent with an expert demonstration generated from a Proximal Policy Optimization (PPO) (Schulman et al., 2017) solution. Each demonstration is used to construct a predictive model as well as a policy (through behavior cloning). The stochastic hybrid learning approach is used to plan and test the robot’s performance in the environment. Environment experience is then used to update the predictive models while the expert demonstrations are solely used to update the policy.

In Figure 6, we compare hybrid learning against behavior cloning. Our method is able to achieve the task at the level of the expert within six (200 step) demonstrations, where the behavior cloned policy is unable to achieve the expert performance. Interestingly, the ant environment is less susceptible to the covariate shift problem (where the state distribution generated by the expert policy does not match the distribution of states generated by the imitated policy (Ross and Bagnell, 2010), which is common in behavior cloning. This suggests that the ant experiences a significantly large distribution of states during the expert demonstration. However, the resulting performance for the behavior cloning is worse than that of the expert. Our approach is able to achieve similar performance as behavior cloning with roughly two fewer demonstrations and performs just as well as the expert demonstrations.

Figure 6.

Results for hybrid stochastic control with behavior cloned policies (averaged over 10 trials) using the Ant Pybullet environment (shown right). Expert demonstrations (actions executed by an expert policy on the ant robot) are used as experience to boot-strap a learned stochastic policy (behavior cloning) in addition to predictive models which encode the dynamics and the underlying task of the ant. Our method is able to adapt the expert experience to the predictive models, improving the performance of behavior cloning and performing as well as the expert. Note: Markers are included for distinguishing between methods only and do not represent all data points.

Robot learning from examples results

We also test our approach on a pick-and-place experiment with the Franka Panda robot. The real hardware system is more likely to have the covariate shift problem. The goal for the robot is to learn how to stack one block on top of another block using demonstrations (see Figure 7). As with the ant simulated example in Figure 6, a demonstration is provided at each attempt at the task and is used to update the learned models. Experience obtained in the environment is solely used to update the predictive models. We use a total of ten precollected demonstrations of the block stacking example (given one at a time to the behavior cloning algorithm before testing). At each testing time, the robot arm is initialized at the same spot over the initial block. Since the demonstrations vary around the arm’s initial position, any state drift is a result of the generated imitated actions and will result in the covariate shift problem leading to poor performance.

Figure 7.

Hybrid learning with behavior cloning results on the Franka Panda robot (averaged over 5 trials). The time series on the right provides a visualization of the task. The task is to stack a block on top of another using expert demonstrations. Our method is able to learn the block stacking task within three expert demonstrations and provides solutions that are more repeatable than with behavior cloning. For additional visualization, see Extension 1. Note: Markers are included for distinguishing between methods only and do not represent all data points.

Figure 7 shows our approach is capable of learning the task in as little as two demonstrations where behavior cloning suffers from poor performance. Since our approach synthesizes actions when the policy is uncertain, the robot is able to interpolate between regions where the expert demonstration was lacking, enabling the robot to achieve the task.

5. Related work

Our hybrid learning approach is able to leverage both the model and the policy during both training and real-time operation. This is a major difference between our approach and other reinforcement learning methods which combine model-based and model-free approaches. Many of the approaches in related work only use the model to help the model-free policy learn better, but do not have any use for the model once the model-free policy training is complete.

Model-Based Value Expansion (MVE) (Feinberg et al., 2018), Stochastic Ensemble Value Expansion (STEVE) (Buckman et al., 2018), and Model Based Policy Optimization (MBPO) (Janner et al., 2019) use only the model-free policy to interact with the environment both during training and after training. The model is learned offline using data accumulated by the model-free policy, but there is no model-based controller associated with the model. Instead, the policy is used to “imagine” future transitions in the model-based environment for short rollouts. These model rollouts are used during the update step to augment Q-learning (for MVE and STEVE) or the policy directly (for MBPO).

Guided Policy Search (GPS) (Montgomery and Levine, 2016) takes a different approach by using a model-based controller to interact with the environment. Each iteration, GPS fits a dynamics model to the most recently collected data, and then uses these models to constrain the policy updates. In this algorithm, only the model-based controller interacts with the environment during training, and the goal is to train a policy which can be used without the model after training.

Model-based learning with model-free fine-tuning (MB-MF) (Nagabandi et al., 2018) takes another approach. First, a model-based random shooting controller is used to learn a coarse-grained representation of the dynamics. Then, the learned dynamics are transferred to a model-free method via data aggregation. Finally, the model-free policy is fine-tuned through interactions with the environment. Both the model-based controller and the model-free policy interact with the environment, but they do so sequentially—during any given trial, only one of them plays a role. Although the model is no longer needed once the model-free policy is trained, the idea with this algorithm is that the model could be reused to learn another task with the same agent without needing to relearn a model, similar to the method we present here.

In Figure 8, we tested the stochastic hybrid learning algorithm and other methods which combine model-based and model-free learning. Details of the comparison methods can be found in Appendix D. The results show that the methods we present here outperform these related methods. In addition to the learning gains, the hybrid learning controller presented here maintains both the model and the policy after training. This may provide benefits during future controller execution—if the hybrid learning trained policy encounters an untested state during operation, it will have the model-based controller to fall back on to compensate for this uncertainty.

Figure 8.

Comparison to related work combining model-based and model-free methods (averaged over 10 random seeds). Our approach improves both the sample efficiency and the highest expected reward. Additional details and pairwise comparisons are included in Appendix. Note: Markers are included for distinguishing between methods only and do not represent all data points.

6. Conclusion

In this work, we present hybrid learning as a method for formally combining model-based learning with experience-based policy learning based on hybrid control theory. Our approach derives the best action a robotic agent can take given the learned models, both model-free and model-based. We tested our approach in various simulated and real-world environments using a variety of learning conditions and show that our method improves both the sample efficiency and the resulting performance of learning motor skills.

Future directions of this work will focus on testing real-world applications of hybrid learning. Our robot arm experiments demonstrated the ability of agents to learn online in a relatively simple physical test environment. In our Sawyer testing, the state was simplified to include the locations of each object relative to the end-effector. Future work could represent the state as an image and extend hybrid learning to visuo-motor tasks. Or the state could be expanded to include additional sensors such as contact sensors, accelerometers, etc. Our hybrid learning framework will enable extensive hardware testing–due to its real-time implementation–to determine how learning performance changes when adding new sensors into the state space and when modifying reward functions.

Key considerations for real-world testing are safety, time, and memory. Future work will explore methods of imposing safety constraints during operation. The time required to determine the next candidate action is dependent on the prediction horizon, the network sizes, and the number of candidate samples (for stochastic hybrid learning). As these variables increase, parallel processing will eventually be necessary.

Currently, exploration is done through random sampling. Future work will explore methods of incorporating exploration goals into the algorithm. As the state-space grows, random sampling may no longer be sufficient to span the state-space. Alternate forms of exploration could help reduce the number of candidate samples required and reduce computation time.

As computation and memory allow, particularly if parallel processing has been introduced for other reasons, it would be useful to explore model-based control using ensembles. One potential approach could train an ensemble of models and use the model uncertainty to determine which model-based controller to fall back on when the policy is uncertain. In general, bringing the robustness benefits of techniques such as path integral control (Williams et al., 2018) to our method can only improve its overall performance, but there may be many approaches that would all be equally viable.

Supplemental Material