Robustness evaluation of offline reinforcement learning for robot control against action perturbations

Abstract

Offline reinforcement learning, which learns solely from datasets without environmental interaction, has gained attention. This approach, similar to traditional online deep reinforcement learning, is particularly promising for robot control applications. Nevertheless, its robustness against real-world challenges, such as joint actuator faults in robots, remains a critical concern. This study evaluates the robustness of existing offline reinforcement learning methods using legged robots from OpenAI Gym based on average episodic rewards. For robustness evaluation, we simulate failures by incorporating both random and adversarial perturbations, representing worst-case scenarios, into the joint torque signals. Our experiments show that existing offline reinforcement learning methods exhibit significant vulnerabilities to these action perturbations and are more vulnerable than online reinforcement learning methods, highlighting the need for more robust approaches in this field.

Introduction

Offline reinforcement learning (offline RL) ¹ has emerged as a novel approach across various domains like healthcare, ² energy management, ³ and robot control. ⁴ Unlike online RL, which requires direct interaction with environments, offline RL learns solely from pre-collected datasets without environmental interaction. Hence, offline RL can potentially reduce costs and risks associated with frequent environmental interactions.

For offline RL to be more widely applicable, robustness is an essential requirement. The robustness of offline RL can be broadly categorized into two types: training-time robustness^5–7 and testing-time robustness.^8–14 Training-time robustness focuses on effectively training policies with corrupted datasets, which may include perturbations caused by data logging errors and intentional data augmentation.^5,15 Testing-time robustness, on the other hand, refers to the ability to handle discrepancies between training and deployment environments. In robot control, these discrepancies include state-space perturbations (e.g., sensor perturbations)^16,17 and action perturbations (e.g., mismatched friction coefficients or actuator failures).^18–20 Online RL addresses similar challenges, often referred to as the reality gap, through methods like domain randomization^21–23 and adversarial training.^18,24–27 However, whether these solutions can be effectively applied to offline RL remains unclear.

Online RL benefits from extensive exploration across diverse state-action pairs via direct environmental interactions, enabling broad coverage of the state-action space. In contrast, offline RL is restricted to exploring only the state-action pairs available in the pre-collected training dataset, rendering its performance highly dependent on the dataset’s quality and coverage. Moreover, offline RL relies on conservative training to mitigate the overestimation of action values caused by out-of-distribution actions. ²⁸

Existing studies^12–14 on testing-time robustness in offline RL have primarily focused on state-space perturbations. Robustness against state-space perturbations is typically achieved by minimizing the sensitivity of the policy to small input variations. This relies on the assumption that state-space perturbations should not lead to drastically different actions. For example, robustness can be improved by incorporating a smoothing loss, ¹² which encourages the policy’s output distribution to change smoothly in response to small state-space perturbations, or regularization techniques such as KL divergence constraints that align outputs with and without perturbations. ¹⁴ However, in real-world scenarios, especially in robotics, action perturbations, such as actuator failures, are equally critical but remain unexplored. Action-space perturbations are directly added to the actions produced by the policy network, and thus affect the environment’s dynamics and the resulting rewards. As a result, these perturbations lead to deviations from the intended behavior dictated by the policy network, posing significant challenges to the reliability of offline RL systems. Therefore, the policy must output actions inherently robust to such perturbations, still achieving high rewards despite them. The distinction between robustness to state-space perturbations and action-space perturbations is particularly critical in offline RL. Offline RL algorithms typically impose conservative constraints to align the learned policy closely with the dataset’s behavior policy. ¹ Therefore, successful learning under action perturbations requires the dataset to include actions that are already adapted to such perturbations, along with corresponding rewards and next states. Collecting such data is typically impractical, underscoring the challenge of studying robustness to action-space perturbations in offline settings.

This study investigates the testing-time robustness of existing offline RL methods^29–31 against random and adversarial action perturbations. Various adversarial attack strategies have been proposed in the context of online RL,^18,26,27 such as gradient-based and curriculum-based methods. However, these methods typically require the adversarial perturbations to be jointly optimized with the policy, assuming interactive access to the environment. This joint optimization is challenging to apply in offline settings, where the policy must be learned solely from fixed datasets. To address this limitation, differential-evolution-based perturbations ¹⁹ are adopted in this study as a representative method that does not rely on joint policy optimization, making it suitable for offline RL. Specifically, to evaluate the robustness of these methods, we simulate actuator failures by perturbing the actions generated by the policy network. These offline RL methods limit exploration to remain within the training data distribution, and we hypothesize this limitation renders them vulnerable to unencountered perturbations. This vulnerability is particularly critical in the case of adversarial perturbations, which are intentionally crafted to exploit the system under worst-case failure scenarios. Adversarial perturbations underscore the need to understand their effects to develop reliable control systems. However, the impact of action-space perturbations on the robustness in offline RL remains yet to be fully explored.

Our approach involves simulating operational challenges within the MuJoCo physical simulation environment, ³² focusing on forward walking tasks. We evaluate the testing-time robustness of offline RL methods using the average episodic reward. Experimental results reveal that the existing methods are highly vulnerable to action perturbations. Furthermore, inspired by typical defensive strategies in online RL—training in environments with added perturbations—we train policies on datasets augmented with action perturbations and evaluate their robustness. The results, however, demonstrate no significant improvement, highlighting the need for more robust offline RL methods to address failures effectively. The contributions of this article are as follows:

We evaluate the testing-time robustness of the offline RL methods against random and adversarial action perturbations and reveal that the offline RL models are more vulnerable to action perturbations than online RL.

We find that the testing-time robustness of the offline RL models depends on the state-action coverage of the training dataset.

We show that training with action-perturbed datasets alone does not improve the testing-time robustness of offline RL against action perturbations.

Related work

Offline reinforcement learning

In offline training, Q-values tend to be overestimated for unseen state-action pairs that are not represented in the dataset. ¹ Since this overestimation cannot be corrected through direct interaction with the environment, traditional online RL methods face significant challenges when training on static datasets. ²⁹ To address these overestimation issues, various offline RL techniques have been proposed.^29–31,33 This study specifically evaluates the robustness of two common strategies for offline RL: policy constraints^29,30 and regularization. ³¹

Batch-constrained Q-learning (BCQ) ²⁹ utilizes policy constraints to ensure the learned policy remains close to the behavior policy that generates the training data. Similarly, the Twin-Delayed DDPG with behavior cloning (TD3+BC) ³⁰ , an offline variant of TD3, ³⁴ implicitly constrains the learned policy by adding a regularization term to the objective function. These policy constraints effectively limit the action space, preventing the agent from exploring unrealistic or unencountered actions in the dataset. Implicit Q-learning (IQL) ³¹ regularizes the value function update process, relying on the state value distribution from the dataset rather than the learned policy. This regularization mitigates the risk of overestimating Q-values, ensuring that the resulting policy remains within the empirical data distribution. However, both policy constraints and regularization strategies may lose robustness against random and adversarial perturbations because the training data does not contain data adapted to such perturbations.

Robust reinforcement learning

The robustness of RL, especially in robot control, mainly stems from addressing the reality gap. This gap arises from discrepancies in physical parameters such as friction coefficients, terrain geometry, and mechanical failures. In the context of online RL, one approach to address the reality gap involves leveraging the Robust Markov Decision Process (RMDP) framework,^35–39 which relies on a set of state transition models known as an uncertainty set. RMDP seeks to derive an optimal strategy by considering the worst-case scenarios within this uncertainty set. However, the high computational cost of RMDP makes it impractical for large state-action spaces. A more practical approach to bridging this gap involves introducing perturbations during the training process. Domain randomization^21–23 generates diverse environments by randomizing parameters such as friction coefficients, thereby training the model to adapt to these changes. Alternatively, adversarial training^18,24–27 improves robustness by introducing adversarial attacks that intentionally reduce rewards. Differential evolution has also been employed to generate adversarial perturbations, ¹⁹ leading to significant reward reductions in walking robot scenarios. Curriculum-based adversarial learning ²⁴ enhances robustness by applying adversarial attacks to the action space.

In offline RL, robustness can be categorized into training and testing time robustness. Training-time robustness addresses corrupted training data.^5–7 For example, IQL is more robust than other offline RL methods for training-time robustness. ⁷ Furthermore, a variant, Robust-IQL, incorporates normalization, Huber loss, and quantile Q estimator to enhance resistance against corrupted transitions. Several studies have focused on the testing-time robustness in an environment with perturbations that are not represented in training datasets.^8–14 In relation to our work, Yan et al. ¹² enhances robustness against adversarial attacks in the state space by smoothing the action value distribution and employing conservative value estimates. Liu et al. ¹³ proposes a model-based approach to address adversarial state perturbations using ensemble learning with a set of state transition functions. Nguyen et al. ¹⁴ demonstrate the vulnerability of offline RL models to adversarial examples, where the outputs of critic and actor networks deviate from clean examples based on KL divergence. The study introduces a defense regularization term in the loss function to minimize the mean squared error between adversarial and clean example outputs, showing improved robustness against adversarial attacks.

Nevertheless, research on adversarial attacks specifically targeting action spaces and corresponding defense strategies remains limited. This study explores testing-time robustness against both adversarial and random perturbations in the context of model-free control for legged robots.

Comparison between offline and online reinforcement learning

Offline RL and online RL represent fundamentally different learning scheme in how data is collected and utilized. Offline RL trains policies solely on fixed datasets, whereas online RL continuously gathers new experience through environment interaction and updates the policy accordingly.^1,28 These differences can lead to significant discrepancies in generalization, adaptability, and robustness.

Sujit et al. ⁴⁰ point out that such discrepancies in evaluation protocols make direct comparisons between offline RL and online RL difficult, and propose a unified evaluation framework (SeqEval) based on data usage and interaction level. Mediratta et al. ⁴¹ demonstrate that even when trained on expert-level datasets, offline RL often underperforms in unseen environments due to limited data coverage and the absence of exploration. Brandi et al. ⁴² provide a practical comparison between the two paradigms in a real-world control task. Their results show that offline RL benefits from stable early performance due to model pretraining, while online RL, though initially less effective, eventually achieves comparable performance through continuous interaction. This reveals a trade-off between initial stability and long-term adaptability. Although not directly compared to the online RL, further studies have examined structural limitations of offline RL. Schweighofer et al. ⁴³ show that dataset properties such as trajectory quality and state-action coverage significantly affect offline RL performance, highlighting its dependence on the training data. In contrast, online RL inherently collects diverse data via exploration. Park et al. ⁴⁴ argue that the main bottleneck in offline RL lies not in value estimation, but in the limited generalization capacity of the learned policy, particularly in unseen states during deployment.

These studies collectively highlight the fundamental differences between offline and online RL. Building on these insights, this study empirically investigates their testing-time robustness under random and adversarial perturbations in continuous control tasks.

Method

We evaluate the testing-time robustness of offline RL methods by conducting forward walking tasks with legged robots in three distinct environments, as shown in Figure 1. These environments include three perturbation scenarios: normal (no perturbations), random perturbations, and adversarial perturbations, both applied to the robot’s joint torque signals. For the robustness evaluation, we employ two common approaches for offline RL: policy constraints and regularization.

Offline reinforcement learning

Reinforcement learning is modeled as a Markov decision process (MDP). An MDP is defined by a tuple $(\mathcal{S},\mathcal{A},P,P_0,r,\gamma )$ where $\mathcal{S}$ is a set of states, $\mathcal{A}$ is a set of actions, $P:\mathcal{S}\times \mathcal{A}\times \mathcal{S}\to [0,1]$ defines the transition probability $P({\mathit{s}}^{\prime }|\mathit{s},\mathit{a})$ from state $\mathit{s}$ to state ${\mathit{s}}^{\prime }$ given action $\mathit{a}$ , $P_0:\mathcal{S}\to [0,1]$ denotes the initial state distribution, $r:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$ is the reward function that assigns the immediate reward $r(\mathit{s},\mathit{a})$ received after transitioning, and $\gamma \in [0,1]$ is the discount factor that determines the importance of future rewards. The goal of an MDP is to find an optimal policy $\pi *$ that maximizes the expected sum of rewards.

Figure 1.

Overview of the robustness evaluation for offline RL. Offline RL models are trained on varying-quality offline datasets and evaluated on forward walking tasks in diverse robot environments. Testing-time robustness is evaluated by average episodic rewards under three conditions: normal (no perturbations), random, or adversarial perturbations applied to actions. Adversarial perturbations, generated using the differential evolution algorithm, minimize the average episodic reward over multiple generations in the same robot environment. These adversarial perturbations are also used to create action-perturbed datasets, which are tested to determine whether training with such datasets improves testing-time robustness.

In the offline RL setting, we are provided with dataset $\mathcal{D}=\{({\mathit{s}}_t,{\mathit{a}}_t,{\mathit{s}}_{t+1},r_t{)}_i\}$ , which may be sourced from previous learning episodes, expert demonstrations, or even random actions. Here, the subscript $t$ denotes a timestep within an episode, and the subscript $i$ indexes individual transitions in the dataset. The dataset $\mathcal{D}$ is assumed to be generated according to a behavior policy ${\pi }_{\beta }$ . The objective of offline RL is to find an optimal policy $\pi *$ using only the dataset $\mathcal{D}$ without further environmental interactions. A principal challenge in offline RL is the distributional shift problem. This problem arises when the dataset $\mathcal{D}$ does not accurately reflect real-world situations or different scenarios. Such static datasets may result in misguided learning, leading to policies that perform poorly in new or unseen environments. To mitigate the effects of the distribution shift, several methods have been proposed.^29–31,33

Policy constraint methods impose constraints to minimize the discrepancy between the learned policy ${\pi }_{\mathrm{off}}$ and the behavior policy ${\pi }_{\beta }$ . Similarly, policy regularization methods penalize deviations from behavior policy ${\pi }_{\beta }$ by incorporating regularization terms into the objective function. These constraints and regularization terms help to keep the learned policy within the support of the behavior policy, and ensure that the learned policy does not make decisions that deviate significantly from dataset $\mathcal{D}$ . However, this conservative strategy may backfire, potentially leading to vulnerabilities in offline RL models during testing phases.

Action perturbation

For action attack, a perturbation $\mathit{\delta }$ is applied to the action ${\mathit{a}}_t$ as follows:

$${\mathit{a}}_t^{\prime }={\mathit{a}}_t+\mathit{\delta }\odot {\mathit{a}}_t,$$

(1)

where $t$ denotes the time step in an episode, and $\odot$ denotes the Hadamard product, which represents the element-wise product of two vectors.

In this study, we consider three conditions: normal (no perturbations), uniform random perturbations, and adversarial perturbations. These perturbations are generated as follows:

$$\mathit{\delta }=\{\begin{array}{cc}\mathbf{0}& \text{normal},\\ {\mathit{\delta }}_r\sim U([-ϵ,ϵ{]}^{N_a})& \text{random; perturbations, }\\ {\mathit{\delta }}_a\sim \text{DE}(ϵ)& \text{adversarial; perturbations}.\end{array}$$

(2)

In the normal condition, the perturbation $\mathit{\delta }$ is set to the zero vector $\mathbf{0}$ , meaning no perturbations are applied to the actions $\mathit{a}$ . The random perturbation ${\mathit{\delta }}_r$ is drawn from the uniform distribution $U([-ϵ,ϵ{]}^{N_a})$ , where $N_a$ denotes the dimension of $\mathit{a}$ , corresponding to the number of actuators, and $ϵ$ specifies the perturbation strength. See Appendix A for details of the perturbation strength configuration. Each element of $\mathit{a}$ is independently perturbed by a random number drawn from the uniform distribution over the interval $[-ϵ,ϵ]$ . As illustrated in the bottom-right corner of Figure 1, the adversarial perturbation ${\mathit{\delta }}_a$ is obtained using the differential evolution (DE) method, ¹⁹ denoted by $\text{DE}(ϵ)$ .

The DE method is represented as Algorithm 1, with perturbations for the $g$ -th generation denoted by $\{{\mathit{\delta }}_g^{(1)},{\mathit{\delta }}_g^{(2)},\dots ,{\mathit{\delta }}_g^{(NP)}\}$ , where $NP$ denotes the number of individuals for each generation. Each perturbation ${\mathit{\delta }}_g^{(i)}$ is evaluated based on the average episodic reward ${\overline{R}}_{episode}$ computed over $M$ episodes as:

$${\overline{R}}_{\text{episode}}=\frac{1}{M}\sum _{m=1}^M\sum _{t=0}^{T-1}{r}_t^{(m)},$$

(3)

where $r_t^{(m)}=r({\mathit{s}}_t,{\mathit{a}}_t^{\prime })$ denotes the immediate reward at time $t$ for episode $m$ . We conduct over 100 trials for each generation under the trained policy ${\pi }_{\mathrm{off}}$ . The DE algorithm uses mutation and crossover processes to generate new perturbations. For mutation, the $i$ -th mutant individual ${\mathit{v}}_{g+1}^{(i)}$ is calculated as:

$${\mathit{v}}_{g+1}^{(i)}={\mathit{\delta }}_{\mathrm{best}}+F({\mathit{\delta }}_g^{(r_1)}-{\mathit{\delta }}_g^{(r_2)}),$$

(4)

where $F\in (0.5,1]$ is a random scaling factor, $r_1$ and $r_2$ are mutually exclusive indices randomly chosen from $\{1,2,\dots ,NP\}$ . For crossover, the binominal crossover is used to compute the $j$ -th element of the $i$ -th trial individual as follows:

$$u_{j,g+1}^i=\{\begin{array}{cc}{v}_{j,g+1}^i& \text{if ;}r\le CR\text{ ;or; }j=j_r,\\ {\delta }_{j,g}^i& \text{otherwise},\end{array}$$

(5)

where $r\in [0,1]$ is a uniform random number, $CR=0.7$ is the crossover constant, and $j_r$ is an index randomly chosen from $\{1,2,\dots ,N_a\}$ . Each element of ${\mathit{u}}_{g+1}^{(i)}$ is constrained to the range of $[-ϵ,ϵ]$ using the clip function, defined by ${\text{clip}}_{[-ϵ,ϵ]}(\mathit{x})=max(min(\mathit{x},ϵ),-ϵ)$ . Finally, the individual from the final generation is utilized as the adversarial perturbation ${\mathit{\delta }}_a$ .

Testing-time robustness evaluation

The procedure for evaluating the testing-time robustness of offline RL methods is outlined in Algorithm 2 and top-right corner of Figure 1. The evaluation is conducted for the trained policy ${\pi }_{\mathrm{off}}$ under three perturbation conditions in Equation (2). We assume that perturbations $\mathit{\delta }$ remain constant throughout each episode; $\mathit{\delta }$ is initialized at the beginning of the episode and remains unchanged until the end. The robustness is evaluated using the average episodic reward over $M$ episodes in Equation (3). Each episode terminates when either the time steps $T$ reaches $1000$ or the robot falls over.

Experiments

This section evaluates the testing-time robustness of offline RL methods through three main experiments. First, we compare the testing-time robustness of offline RL methods to online RL methods by measuring the average episodic rewards over 1,000 trials under three distinct conditions: the normal condition (no perturbations), the random condition (uniform random perturbations), and the adversarial condition (adversarial perturbations). Second, we examine how state-action coverage in training datasets impacts the testing-time robustness of offline RL methods. Finally, we investigate whether training policies on action-perturbed datasets enhance their robustness against the perturbations.

Experimental setup

RL methods

We employ existing offline RL methods (BCQ, TD3+BC, IQL) implemented in the Data-Driven Deep Reinforcement Learning library for Python (d3rlpy), ⁴⁵ a framework designed for offline RL. Additionally, we employ three widely-used online RL methods: proximal policy optimization (PPO), ⁴⁶ soft actor-critic (SAC), ⁴⁷ and TD3. ³⁴ The hyperparameter configurations are adapted from the original paper ⁴⁶ and the prior study ⁴⁸ For SAC and TD3, we leverage the implementation provided by the d3rlpy library. See Appendix B for details of hyperparameter configurations.

Training datasets

Training is conducted using datasets from datasets for deep data-driven reinforcement learning (D4RL), ⁴⁹ which include three quality levels: expert, medium-expert, and medium. The expert and medium datasets consist of 1 million trajectories collected using well-trained and less-trained policies, respectively. The medium-expert dataset combines the expert and medium datasets, resulting in a total of 2 million trajectories. We create two types of action-perturbed datasets for all legged robot environments: randomly-perturbed datasets and adversarially-perturbed datasets. Random and adversarial perturbations, as defined in Equation (2), are applied to all action data within the D4RL expert dataset, following Equation (1). See Appendix C for histograms of action distributions in each action-perturbed dataset.

Training steps

For the expert dataset, RL models are trained for 3 million steps using BCQ and IQL methods, respectively, whereas models using TD3+BC require extended training of up to 5 million steps to ensure convergence. With the medium-expert dataset, all RL models are trained for 3 million steps. For the medium dataset, RL models are trained for 3 million steps using BCQ and IQL, whereas models using TD3+BC require only up to 2 million steps of training. For the action-perturbed datasets, RL models using each offline RL method were trained for the same number of steps as those trained on the expert dataset. Regarding the online RL methods, RL models using PPO are trained as described in Otomo et al. ¹⁹ and Babadi et al. ⁴⁸ RL models are trained for over 3 million steps using SAC and TD3 methods.

Legged robot environments

Forward walking tasks are conducted in the Hopper-v2, HalfCheetah-v2, and Ant-v2 environments of OpenAI Gym, ⁵⁰ which serve as widely recognized as benchmark environments in offline RL.^29–31,33 These tasks are simulated within the MuJoCo physics engine, as illustrated in Figure 2.

Figure 2.

Legged robot environments: Hopper-v2 (left), HalfCheetah-v2 (center), and Ant-v2 (right).

The reward functions for each robot are defined as follows:

$$\begin{array}{r}r(\mathit{s},\mathit{a})=\{\begin{array}{c}{v}_{\text{,fwd}}-0.001{\Vert \mathit{a}\Vert }^2+1,\text{Hopper}\\ v_{\text{,fwd}}-0.1{\Vert \mathit{a}\Vert }^2+1,\text{HalfCheetah}\\ v_{\text{,fwd}}-0.5\Vert \mathit{a}{\Vert }^2-0.5\cdot {10}^{-3}{\Vert \mathit{f}\Vert }^2+1,\text{Ant}\end{array}\end{array}$$

(6)

where $v_{\text{,fwd }}$ is the forward walking speed and $\mathit{f}$ denotes the contact force.

Perturbation setting

Perturbations under each condition are defined according to Equation (2). In the Hopper-v2 and HalfCheetah-v2 environments, the strengths of ${\mathit{\delta }}_r$ and ${\mathit{\delta }}_a$ are set to $ϵ=0.3$ for all experiments. In the Ant-v2 environment, the perturbation strength is set to $ϵ=0.5$ for most experiments. However, for the final experiment involving the action-perturbed dataset, the perturbation strength is reduced to $ϵ=0.3$ due to insufficient training with the strength $ϵ=0.5$ . The adversarial perturbation ${\mathit{\delta }}_a$ in Equation (2) is generated using the DE method over 30 generations. The population sizes for DE are set to 45, 90, and 120 for Hopper-v2, HalfCheetah-v2, and Ant-v2, respectively.

Comparing testing-time robustness: offline RL vs. online RL

We compare the testing-time robustness of offline RL to online RL. Offline RL models are generally expected to be more vulnerable than online RL models because online RL models are typically trained on diverse state-action pairs through direct interaction with the environment. As shown below, the experimental results support this expectation. Table 1 summarizes the experiment results for the Hopper-v2, HalfCheetah-v2, and Ant-v2 environments. The table presents the average episodic rewards over 1,000 trials, along with the corresponding standard deviations.

Table 1.

Testing-time robustness evaluation results of offline RL (BCQ, IQL, TD3+BC) and online RL (PPO, SAC, TD3). The evaluation is conducted in three legged robot environment (hopper-v2, halfCheetah-v2, ant-v2) under normal, random, and adversarial conditions. Offline RL methods are trained with the expert dataset. Results are presented as average episodic rewards $\pm$ standard deviations. The values reported in the average row represent average results of three offline RL methods under each experiment setting.

			Perturbation conditions
Environment	Training datasets	RL methods	Normal	Random	Adversarial
Hopper-v2	expert	BCQ	$2371\pm 449$	$1556\pm 1296$	$282\pm 4$
		TD3+BC	$3597\pm 93$	$2027\pm 1394$	$424\pm 78$
		IQL	$1631\pm 597$	$1290\pm 939$	$421\pm 24$
		Average	$2533$	$1624$	$376$
	online	PPO	$2529\pm 109$	$1456\pm 935$	$346\pm 6$
		SAC	$3148\pm 746$	$1945\pm 1294$	$445\pm 89$
		TD3	$3439\pm 2$	$2706\pm 1130$	$487\pm 18$
		Average	$3039$	$2036$	$426$
HalfCheetah-v2	expert	BCQ	$11414\pm 1406$	$8036\pm 3179$	$244\pm 262$
		TD3+BC	$11713\pm 421$	$9034\pm 2337$	$1227\pm 999$
		IQL	$9316\pm 2214$	$6198\pm 3057$	$313\pm 366$
		Average	$10814$	$7756$	$595$
	online	PPO	$6822\pm 32$	$6579\pm 161$	$5985\pm 107$
		SAC	$7664\pm 58$	$5610\pm 1069$	$2841\pm 71$
		TD3	$7076\pm 289$	$5801\pm 646$	$3713\pm 106$
		Average	$7187$	$5997$	$4180$
Ant-v2	expert	BCQ	$4511\pm 1568$	$3102\pm 1781$	$-1129\pm 1083$
		TD3+BC	$4715\pm 1676$	$3317\pm 1822$	$-1209\pm 1089$
		IQL	$4175\pm 1546$	$2851\pm 1731$	$-1503\pm 1026$
		Average	$4467$	$3090$	$-1280$
	online	PPO	$5698\pm 989$	$4975\pm 1191$	$2654\pm 1748$
		SAC	$6070\pm 890$	$4226\pm 1611$	$942\pm 888$
		TD3	$5154\pm 1100$	$3349\pm 1464$	$685\pm 1116$
		Average	$5641$	$4183$	$1427$

From Table 1, both offline RL (BCQ, TD3+BC, IQL) and online RL (PPO, SAC, TD3) exhibit lower average episodic rewards under the random perturbation condition compared to the normal condition, with further deterioration observed under the adversarial perturbation condition. This result demonstrates that the applied perturbations, particularly the adversarial ones, affect the legged robot control using both online and offline RL as expected. In the Hopper-v2 environment, offline RL performs on par with online RL under both the normal and random conditions, but shows slightly lower performance under the adversarial condition. In contrast, in the HalfCheetah-v2 and Ant-v2 environments, offline RL yields results that are either comparable to or better than those of online RL under normal and random conditions. However, under adversarial conditions, the performance of offline RL significantly decreases, particularly in the Ant-v2 environment, where the average episodic reward falls below zero.

This finding suggests that offline RL brings more vulnerability due to insufficient experiential learning. The significant reduction in average episodic rewards occurs consistently across different legged robot scenarios and the offline RL methods, emphasizing the challenges for robust offline RL. The results further suggest that the vulnerability of offline RL stems primarily from the offline setting itself rather than from policy characteristics, as the offline RL methods exhibit greater vulnerability to adversarial perturbations than the online RL methods, regardless of whether the policies are stochastic and deterministic policies.

The vulnerabilities identified in this study highlight fundamental challenges intrinsic to offline reinforcement learning, rather than limitations tied to specific robot models or the MuJoCo environment. These challenges stem from structural constraints of offline RL—particularly, its inability to incorporate feedback from action outcomes—which are independent of simulation platforms or actuation mechanisms. For instance, in real-world legged robots, actuator lag, a delay between the control command and the actual torque output, is a common phenomenon. Such lag can cause the executed action to deviate from the intended action even if the control policy itself is correct. Since offline RL cannot observe the post-action distortion induced by actuator lag during training, the learned policy may fail catastrophically when deployed on hardware. Therefore, in offline settings that rely on static training data, similar vulnerabilities may manifest in other robotic systems and real-world applications where interactive learning during training is not possible.

Impact of state-action coverage on offline RL robustness

Based on the robustness differences between offline RL and online RL identified in the previous section, we hypothesize that these differences stem from the dependency of offline RL training on the training dataset. Specifically, we hypothesize that variations in state-action coverage across different training datasets contribute to these robustness disparities. To investigate this hypothesis, we evaluate the robustness of offline RL methods trained on the D4RL medium-expert and medium datasets. Table 2 summarizes the results.

Table 2.

Testing-time robustness evaluation results of offline RL trained on the medium-expert and the medium datasets. The evaluation is conducted in three legged robot environments (hopper-v2, halfCheetah-v2, ant-v2) under normal, random, and adversarial conditions. Results are presented as the average episodic rewards $\pm$ standard deviations. The values reported in the average row represents average results of three offline RL methods under each experiment setting.

			Perturbation conditions
Environment	Training datasets	RL methods	Normal	Random	Adversarial
Hopper-v2	expert	BCQ	$2371\pm 449$	$1556\pm 1296$	$282\pm 4$
		TD3+BC	$3597\pm 93$	$2027\pm 1394$	$424\pm 78$
		IQL	$1631\pm 597$	$1290\pm 939$	$421\pm 24$
		Average	$2533$	$1624$	$376$
	medium-expert	BCQ	$2928\pm 973$	$1581\pm 1063$	$453\pm 57$
		TD3+BC	$3059\pm 867$	$1706\pm 1151$	$500\pm 57$
		IQL	$879\pm 277$	$790\pm 454$	$432\pm 44$
		Average	$2289$	$1262$	$700$
	medium	BCQ	$2030\pm 502$	$1483\pm 580$	$609\pm 12$
		TD3+BC	$1969\pm 398$	$1569\pm 560$	$741\pm 14$
		IQL	$1200\pm 205$	$1116\pm 439$	$598\pm 28$
		Average	$1733$	$1389$	$649$
HalfCheetah-v2	expert	BCQ	$11414\pm 1406$	$8036\pm 3179$	$244\pm 262$
		TD3+BC	$11713\pm 421$	$9034\pm 2337$	$1227\pm 999$
		IQL	$9316\pm 2214$	$6198\pm 3057$	$313\pm 366$
		Average	$10814$	$7756$	$595$
	medium-expert	BCQ	$10566\pm 1563$	$7139\pm 2448$	$1135\pm 948$
		TD3+BC	$11729\pm 724$	$9494\pm 1994$	$2273\pm 1470$
		IQL	$6974\pm 1780$	$5683\pm 1583$	$2044\pm 382$
		Average	$9756$	$7439$	$1817$
	medium	BCQ	$5504\pm 324$	$5077\pm 338$	$3462\pm 405$
		TD3+BC	$5739\pm 81$	$5329\pm 328$	$3568\pm 135$
		IQL	$5364\pm 267$	$4903\pm 464$	$2814\pm 240$
		Average	$5536$	$5103$	$3281$
Ant-v2	expert	BCQ	$4511\pm 1568$	$3102\pm 1781$	$-1129\pm 1083$
		TD3+BC	$4715\pm 1676$	$3317\pm 1822$	$-1209\pm 1089$
		IQL	$4175\pm 1546$	$2851\pm 1731$	$-1503\pm 1026$
		Average	$4467$	$3090$	$-1280$
	medium-expert	BCQ	$4250\pm 1568$	$3397\pm 1343$	$1467\pm 1571$
		TD3+BC	$4588\pm 1718$	$3818\pm 1431$	$1837\pm 1505$
		IQL	$4131\pm 1457$	$3235\pm 1352$	$1074\pm 605$
		Average	$4323$	$3483$	$1459$
	medium	BCQ	$4219\pm 709$	$3361\pm 1045$	$2597\pm 376$
		TD3+BC	$4194\pm 1294$	$3433\pm 1340$	$1206\pm 998$
		IQL	$3527\pm 997$	$2946\pm 979$	$1329\pm 992$
		Average	$3980$	$3247$	$1711$

For the medium-expert dataset, average episodic rewards under the normal and random conditions are consistent with those of the expert dataset. However, under the adversarial condition, the medium-expert dataset yields higher average rewards and small standard deviations than the expert dataset. This result indicates a shift toward a more positive reward distribution.

For the medium dataset, average episodic rewards in the HalfCheetah-v2 environment under the normal condition are approximately half of those for the expert dataset. In contrast, the hopper-v2 and Ant-v2 environments show only slight decreases compared to the expert dataset results. This finding aligns with previous studies,^30,33,49,51 which report reduced rewards for policies trained on the medium dataset in HalfCheetah-v2 environment. This suggests that these differences reflect the specific characteristics of each legged robot environment. Furthermore, under the adversarial condition, both the medium and medium-expert datasets yield significant increases in average rewards compared to the expert dataset. Moreover, in many cases, the medium dataset even outperforms the medium-expert dataset in terms of average rewards.

The medium-expert and medium datasets provide broader state-action coverage than the expert datasets, contributing to improved robustness against perturbations. This observation suggests that increasing the diversity of training datasets could enhance the robustness of offline RL methods. To assess dataset coverage more concretely, Appendix D presents both density visualizations and a clustering-based analysis of the distribution of state-action pairs. However, simply increasing state-action coverage through dataset augmentation may lead to low data efficiency and may even degrade policy performance. This indicates that improving robustness may require not only offline data augmentation but also alternative strategies.

Impact of action-perturbed datasets on offline RL robustness

To investigate the potential for improving robustness against action-space perturbations, we train policies using the randomly and adversarially action-perturbed datasets and evaluate their robustness. In this experiment, unlike the previous two experiments, the perturbation strength is set to $ϵ=0.3$ for all legged robot environments, including Ant-v2. The results are presented in Table 3.

Table 3.

Testing-time robustness evaluation results of offline RL trained on action-perturbed datasets. The evaluation is conducted in three legged robot environments (hopper-v2, halfCheetah-v2, ant-v2) under normal, random, and adversarial conditions. Results are presented as average episodic rewards $\pm$ standard deviations. The values reported in the average row represents average results of three offline RL methods under each experiment setting. randomly-perturbed refers to the expert dataset with randomly-perturbed actions, and adversarially-perturbed refers to the expert dataset with adversarially-perturbed actions.

			Perturbation conditions
Environment	Training datasets	RL methods	Normal	Random	Adversarial
Hopper-v2	expert	BCQ	$2371\pm 449$	$1556\pm 1296$	$282\pm 4$
		TD3+BC	$3597\pm 93$	$2027\pm 1394$	$424\pm 78$
		IQL	$1631\pm 597$	$1290\pm 939$	$421\pm 24$
		Average	$2533$	$1624$	$376$
	randomly-perturbed	BCQ	$2858\pm 717$	$1774\pm 1232$	$506\pm 82$
		TD3+BC	$3594\pm 84$	$2077\pm 1350$	$413\pm 13$
		IQL	$1124\pm 335$	$1006\pm 642$	$413\pm 34$
		Average	$2525$	$1619$	$444$
	adversarially-perturbed	BCQ	$534\pm 25$	$659\pm 355$	$329\pm 18$
		TD3+BC	$353\pm 5$	$411\pm 195$	$336\pm 3$
		IQL	$474\pm 54$	$498\pm 182$	$333\pm 38$
		Average	$454$	$523$	$333$
HalfCheetah-v2	expert	BCQ	$11414\pm 1406$	$8036\pm 3179$	$244\pm 262$
		TD3+BC	$11713\pm 421$	$9034\pm 2337$	$1227\pm 999$
		IQL	$9316\pm 2214$	$6198\pm 3057$	$313\pm 366$
		Average	$10814$	$7756$	$595$
	randomly-perturbed	BCQ	$11110\pm 1411$	$7660\pm 3167$	$182\pm 315$
		TD3+BC	$11334\pm 111$	$8753\pm 2202$	$828\pm 515$
		IQL	$3618\pm 2611$	$2367\pm 2102$	$302\pm 377$
		Average	$8687$	$6260$	$437$
	adversarially-perturbed	BCQ	$427\pm 648$	$1713\pm 2839$	$43\pm 54$
		TD3+BC	$1737\pm 1804$	$2840\pm 3445$	$-26\pm 68$
		IQL	$407\pm 445$	$863\pm 1242$	$45\pm 193$
		Average	$857$	$1805$	$21$
Ant-v2	expert	BCQ	$4511\pm 1568$	$4127\pm 1536$	$1670\pm 1769$
		TD3+BC	$4715\pm 1676$	$4189\pm 1745$	$1432\pm 1484$
		IQL	$4175\pm 1546$	$3707\pm 1535$	$1966\pm 1823$
		Average	$4467$	$4008$	$1689$
	randomly-perturbed	BCQ	$4662\pm 1486$	$4181\pm 1467$	$2208\pm 1863$
		TD3+BC	$5266\pm 1176$	$4534\pm 1442$	$2292\pm 1898$
		IQL	$3385\pm 1759$	$3231\pm 1634$	$1609\pm 1652$
		Average	$4438$	$3982$	$2036$
	adversarially-perturbed	BCQ	$1609\pm 1685$	$2351\pm 1956$	$621\pm 1530$
		TD3+BC	$-159\pm 534$	$-209\pm 609$	$-428\pm 815$
		IQL	$1758\pm 1628$	$2236\pm 1831$	$505\pm 1414$
		Average	$1069$	$1459$	$233$

Compared to the expert dataset results, some results for the randomly-perturbed dataset show a slight increase in average episodic rewards under both the random and adversarial conditions. However, these increases are modest and cannot be conclusively interpreted as improvements in robustness. One possible explanation is that the randomly-perturbed dataset is generated using a uniform distribution, which includes a wide range of perturbations—from very weak to strong ones. Such weak perturbations have little effect on the agent’s behavior, but their presence in the dataset can enhance the diversity of state-action pairs, resembling those found in medium-expert or medium datasets. The diversity introduced by the dataset may lead to modest improvements in robustness under certain conditions. However, this suggests that such diversity alone may not be sufficient to improve robustness.

The results for the adversarially-perturbed dataset reveal that the average episodic rewards remain low, even under the normal condition. These findings cast doubt on the effectiveness of adding perturbations to the training dataset for improving the testing-time robustness.

Perturbation-adding techniques have demonstrated efficacy in improving the testing-time robustness of online RL.^18,21,22,25 Online RL methods succeed because the agent can interact with the environment directly, immediately observe the rewards and next states following perturbed actions, and update the policy iteratively based on that feedback. In contrast, offline RL enforces conservative constraints to keep the learned policy close to the dataset’s behavior distribution. As a result, learning under action-space perturbations is only successful when the dataset includes actions already optimized for such perturbations. These actions must be accompanied by the corresponding rewards and next states. Our perturbed datasets contain only the perturbed actions, without the associated outcome information, so the policy cannot properly evaluate or learn corrective actions—hence the limited robustness gains.

As an alternative dataset modification technique, reward-based interventions such as reward shaping^52–54 or reward labeling^55,56 have been proposed to refine the reward signal and improve learning. These techniques have been used in offline RL, for example to denoise reward signals ⁵³ or to facilitate learning in environments with sparse rewards. ⁵⁴ However, enhancing robustness under perturbations—as is the focus of this study—cannot be achieved solely through reward modification. In offline settings, such robustness requires not only appropriate rewards but also access to trajectories that are robust to perturbations. Since such trajectories are typically difficult to collect, reward-based approaches offer limited practical utility in this context.

As methods for achieving robust policy learning, policy ensemble methods^57–59 have also been explored in the context of online RL. These methods typically aggregate multiple independently trained policies to reduce action-selection variance or stabilize policy behavior. However, in offline settings, policy ensembles are primarily used to mitigate issues arising from distribution shift,^60–62 and they cannot overcome the fundamental limitation of offline RL—namely, the lack of online feedback.

By comparison, in hybrid offline–online approaches,^63–66 an offline-pretrained policy is finetuned through limited online interaction—allow the agent to experience real perturbations and adapt its behavior sequentially. This framework makes it possible to acquire corrective actions that cannot be learned from static data alone. It therefore represents a promising direction for future work on robust offline RL.

Conclusion

This study evaluates the testing-time robustness of offline RL methods in scenarios simulating joint torque failures in legged robots. We focus on three offline RL methods—BCQ, TD3+BC, and IQL—across normal, random perturbation, and adversarial perturbation conditions. The results demonstrate that the offline RL methods are significantly more vulnerable to both random and adversarial perturbations compared to online RL methods in identical conditions. This vulnerability is attributed to the limited diversity of state-action pairs in training datasets, which restricts the adaptability of offline RL policies. Our experiments further reveal that state-action coverage in training datasets affects the robustness of offline RL methods. Policies trained on medium-expert and medium datasets exhibit notable differences in performance under adversarial conditions when compared to those trained on expert datasets. However, training policies on action-perturbed datasets does not yield substantial improvements in robustness, This result indicates that simple data augmentation alone is insufficient for achieving robustness to action-space perturbations. This challenge is rooted in a fundamental limitation of offline RL—datasets typically lack actions adapted to perturbed environments and their associated outcomes. As a result, the policy cannot learn to anticipate distortions that occur after action execution. In safety-critical robotics applications—such as legged locomotion on uneven terrain or manipulation tasks near humans—actuator lag, failures and other hardware-induced problems can have severe consequences. Offline RL methods, when deployed without any form of online adaptation, cannot anticipate these post-action discrepancies. As a result, a trained policy may still fail under even slight real-world noise. Although improved data augmentation or adaptive training strategies may contribute to robustness, their effectiveness remains limited without access to online feedback. Such feedback appears essential for reliably acquiring robustness to post-action perturbations.

To address this limitation, hybrid approaches incorporating online fine-tuning represent a promising direction. In particular, transitioning from offline to online reinforcement learning—by initializing policies with offline-trained models and finetuning them through limited online interactions—offers a sample-efficient means of adapting to real-world perturbations. Online finetuning enables the agent to observe distortions occurring after action execution and to adjust the policy accordingly. Through this iterative feedback process, the agent can acquire corrective behaviors that are otherwise difficult to learn from static offline data.

Future work should explore such offline-to-online frameworks more deeply, with a focus on enabling robust adaptation to action-space perturbations in real-world settings. In particular, developing methods that combine the stability of offline RL with the adaptability of online finetuning will be key to deploying RL systems in the safety-critical robotics domain.