Actuator fault recovery with deep reinforcement learning in a linear model-based control framework: Application to a physical AUV

Abstract

Actuator faults in autonomous mobile robotic systems pose significant challenges, especially in unpredictable environments where system reliability is paramount. Fault tolerant control (FTC) strategies, particularly those leveraging actuator redundancy, have been explored to address these issues. However, traditional methods commonly rely on explicit fault diagnosis, which can be resource-intensive and challenging to implement accurately. This paper introduces a novel approach that combines deep reinforcement learning (DRL) with a linearised optimal model-based controller to achieve actuator fault recovery without explicit fault diagnosis. The integration of DRL within a model-based controller framework enhances system stability and fault recovery capabilities. The chosen application platform for this study is an autonomous underwater vehicle (AUV), where the partial or total failure of a mission critical component such as a thruster could jeopardise the success of the mission and potentially render the vehicle unrecoverable in the event of a fault. In this application case, a linear quadratic regulator (LQR) controller is employed as the model-based controller, while the soft actor-critic (SAC) algorithm is used as the DRL component to handle fault recovery. The DRL model is trained and evaluated in simulation before being directly applied to the physical AUV. The proposed method’s effectiveness is demonstrated through comparisons with a standard LQR controller, a conventional adaptive LQR controller and the proposed hybrid LQR-SAC controller. The results indicate that the LQR-SAC controller outperforms the standard and conventional adaptive LQR controllers in maintaining system performance under fault conditions, achieving a significant reduction in trajectory tracking error on a physical AUV.