Adaptive Integral Sliding Mode control based on reinforcement learning for high-precision tracking in active denial systems

Abstract

Achieving accurate trajectory tracking in Active Denial System (ADS) remains a complex and pressing task, particularly for real-time tracking-and-pointing operations. In this context, we begin by examining the stringent accuracy demands of such systems and put forward an adaptive control strategy that integrates Integral Sliding Mode Control (ISMC) with an improved Double Deep Q-Network (AdaDDQN). The proposed method employs a high-dimensional state-space framework coupled with a context-aware reward mechanism, enabling the controller to adaptively fine-tune its parameters on the fly. This hybrid approach harnesses the learning flexibility of Reinforcement Learning (RL) while preserving the inherent stability of SMC. Experimental evaluations reveal that this method achieves a significant improvement—over 80% gain in tracking precision compared to traditional sliding mode schemes—maintaining the tracking error consistently within 8 mrad. Moreover, it demonstrates strong resilience against mechanical friction, external interference, and model uncertainties, ensuring both accuracy and robustness under real-world operational constraints.

Introduction

The Active Denial System (ADS) represents a class of non-lethal directed-energy technologies that project microwave radiation toward a target using a continuous-wave transmitter operating in the 94–95 GHz range (∼0.1 THz), delivering output power between 10 and 100 kW.^1–3 This radiation travels at the speed of light and induces a thermal effect roughly 0.4 mm beneath the surface of human skin—sufficient to stimulate pain receptors within seconds and trigger a strong withdrawal response. Notably, once the exposure ceases or the individual moves beyond the beam’s reach, the sensation dissipates rapidly. Due to its controllable physiological impact without lasting harm, ADS offers a compelling solution for crowd dispersion, conflict de-escalation, and minimizing collateral injury. ⁴

In parallel with ADS technology, the development of precise tracking and pointing platforms has become increasingly critical. These servo control systems are essential for ensuring that the microwave beam consistently targets the intended area with minimal deviation, especially under dynamic conditions. Among control strategies, robust control and Sliding Mode Control (SMC) are frequently favored due to their resilience against model uncertainties and external disturbances.^5–7 More recently, Deep Reinforcement Learning (DRL) has emerged as a promising paradigm, offering real-time adaptability and enhanced perception in high-dimensional environments.^8–11 Algorithms like Deep Q-Network (DQN) and Double Deep Q-Network (DDQN) have addressed stability and policy evaluation biases, paving the way for intelligent, self-adjusting control frameworks. In this context, combining the adaptability of DRL with the reliability of SMC is gaining traction as a promising direction for high-precision tracking applications.^12–18 In addition to these developments, several recent works have explored advanced control strategies for complex dynamic systems. Regarding intelligent decision-making in networked environments, Cho et al. ¹⁹ proposed a resilient algorithm combining multi-agent reinforcement learning (MARL) with message-passing mechanisms to ensure robust operation of UAVs under unreliable network conditions. While such distributed learning frameworks demonstrate superior resilience, high-precision tracking tasks for servo systems often require more focused compensation for local dynamic nonlinearities; To address specific actuator nonlinearities such as dead-zones, Yuan et al. ²⁰ developed a time-scale decoupled adaptive prescribed-time control scheme. Their method cleverly utilizes fuzzy logic to compensate for unknown dead-zones without prior knowledge, ensuring convergence within a user-specified time. Similarly, addressing the constraints of communication bandwidth and system security, Zhang et al. ²¹ introduced a periodic event-triggering adaptive control strategy capable of mitigating actuator attacks and exogenous disturbances in networked uncertain nonlinear systems. Although these state-of-the-art methods have achieved remarkable success in their respective domains (network resilience, dead-zone compensation, and security), controlling the Active Denial System (ADS) presents unique challenges, particularly regarding constant unmodeled loads and the trade-off between rapid convergence and steady-state precision.

Therefore, applying such techniques in real-world ADS scenarios remains challenging. Specific control performance indicators must be identified based on the unique characteristics of microwave-based systems. A meaningful control framework must not only achieve high accuracy but also operate robustly in the face of chattering, mismatched disturbances, and hardware-induced constraints. These include friction, backlash, input saturation, quantization noise, and sampling delay, all of which influence closed-loop dynamics and complicate controller design.

Moreover, while RL introduces adaptability, it also raises new concerns: function approximation errors, generalization gaps, and violations of Lyapunov stability can compromise safety. Although methods like Constrained Policy Optimization and Control Barrier Functions offer theoretical safeguards, they are often difficult to implement under strict real-time or high-disturbance conditions.

To address these challenges, this study first establishes quantitative control accuracy requirements—specifically, ensuring that the ADS tracking error remains within 10 mrad. On this basis, we propose a novel adaptive control scheme that integrates an improved DDQN (AdaDDQN) into the structure of Integral Sliding Mode Control (ISMC). The approach features a multi-dimensional state-space formulation combined with an intelligent reward mechanism, allowing for online parameter adjustment and real-time learning during tracking tasks.

At the core of the method lies a redesigned integral sliding variable, which includes cumulative error compensation and decomposes the control torque into equivalent, switching, and adaptive components. RL determines the angular velocity increments, enabling the controller to respond swiftly to rapid trajectory changes while retaining SMC’s characteristic robustness. To address chattering, the system incorporates diversity metrics, sliding variable history, and torque transformation statistics, which enhance smoothness perception and suppress excessive torque fluctuation.

Experimental validations confirm that the proposed method outperforms traditional SMC, achieving an improvement of over 80% in tracking accuracy and maintaining errors consistently within 8 mrad. Even in the presence of structural friction and environmental disturbances, the system demonstrates stable, high-precision performance.

The primary contributions of this work include:

Development of a smoothness-enhanced adaptive control framework which seamlessly integrates the adaptive decision-making of AdaDDQN with the theoretical stability of Integral Sliding Mode Control (ISMC). This hybrid framework effectively handles both unmodeled dynamics and transient disturbances.

Proposing a RL decision-making module based on AdaDDQN, capable of adaptively optimizing angular velocity adjustments and key control parameters in a multi-dimensional state space, enabling robust tracking under changing inertial and disturbance conditions.

The proposed strategy is rigorously implemented and validated on a physical ADS tracking-and-pointing platform. Comprehensive experiments demonstrate that it achieves the best performance metrics among all tested methods, maintaining the tracking error consistently within 8 mrad and fully satisfying the stringent beam coverage requirements.

Precision analysis and system modeling

Precision analysis

Given the operational demands of microwave-based ADS, achieving high-precision tracking and pointing is essential to ensure continuous, effective energy projection while minimizing unintended damage. To formalize the control objectives of such systems, this section derives quantitative tracking and pointing accuracy criteria by analyzing both the diffraction behavior of 94 GHz millimeter waves and a probabilistic hit model.

As a starting point, we consider the inherent beam divergence characteristics of the system. With the operating frequency set at 94 GHz and the antenna aperture $D_w$ specified as 0.2 m, the beam divergence angle ${\theta }_w$ can be estimated using the Rayleigh diffraction criterion. Assuming a beam quality factor ${\beta }_w$ the divergence angle is given by the following expression:

$${\theta }_w=\frac{1.22{\lambda }_w{\beta }_w}{D_w}$$

(1)

By inserting the given parameters into the Rayleigh-based formula, the resulting theoretical beam divergence is calculated to be approximately 19.5 mrad. At an operational distance of 250 m—a typical engagement range for ADS—this divergence directly limits the physical spread of the microwave beam.

To establish an accuracy threshold grounded in stable aiming probability, it is essential to distinguish between two primary error sources: static pointing error (a systematic offset) and dynamic tracking error (a random fluctuation). The latter typically appears as line-of-sight jitter with respect to the moving target and is commonly modeled using a Gaussian distribution. For the microwave beam to consistently maintain coverage over the target, this jitter must remain within a controllable range. Leveraging the statistical properties of Gaussian noise, the stable aiming probability $P$ can be expressed as a function of the beam diameter $D$ and the root-mean-square tracking error (RMSE) $\sigma$ (i.e. $\mathrm{\Delta }\theta$ ) through the following integral relationship:

$$P={\int }_{\mathrm{\Delta }{\theta }_0}^{\infty }F\left(\mathrm{\Delta }\theta \right)d\mathrm{\Delta }\theta =1-P\prime =1-\exp [-{\left(\frac{k\mathrm{D}}{2}\right)}^2/2{\sigma }^2]$$

(2)

Here, the hit probability $F$ can be expressed using the probability characteristics of a Gaussian distribution as follows:

$$F\left({\theta }_e\right)=\frac{1}{\sigma \sqrt{2\pi }}{e}^{-\frac{({\theta }_e-\overline{x_p})}{2{\sigma }^2}}$$

(3)

Where ${\theta }_e$ represents the aiming error. As indicated by the above equation, its characteristics follow a normal distribution, where $\overline{x_p}$ denotes the mean error and $\sigma$ represents the root mean square (RMS) error.

The hit probability also satisfies the condition:

$$\underset{0}{\overset{\infty }{\int }}F\left({\theta }_e\right)d{\theta }_e=1$$

(4)

Letting $\mathrm{\Delta }\theta$ represent the precision requirement, and the RMS error be defined as $\sigma =\mathrm{\Delta }\theta L_w$ , the stable aiming probability $P$ can be expressed by the equation:

$$P=1-\exp [-\frac{{(1.22k{\beta }_w{\lambda }_w)}^2}{2\mathrm{\Delta }{\theta }^2{D}_w^2}]$$

(5)

Figure 1 presents a schematic representation of the microwave beam diameter and the associated accuracy requirements, while Figure 2 illustrates how the ratio between beam divergence and line-of-sight jitter influences the tracking precision at a distance of 250 m. The analysis suggests that to achieve stable and high-precision tracking, the jitter of the microwave spot must remain within half of the beam diameter. Under this constraint, the system’s allowable tracking error must be tightly bounded. Simulation results indicate that when the tracking accuracy satisfies $\mathrm{\Delta }\theta \le 10$  mrad, the probability of the irradiation spot remaining fully within this limit (i.e. not exceeding half the beam diameter) is approximately 99.87%. This high level of confidence ensures that the target remains effectively “locked” within the beam path, preventing unintended escape.

Figure 1.

Beam divergence and tracking accuracy constraints.

Figure 2.

Effect of beam diameter and tracking accuracy constraints at 250 m.

In light of these findings, and considering the nonlinear disturbances introduced by platform motion at the 250 m operating range, the tracking-and-pointing system is required to meet a maximum RMSE of $\mathrm{\Delta }\theta \le 10$  mrad. This threshold serves as a design benchmark for the control strategy and parameter tuning throughout the rest of this work.

System dynamics model and control objectives

To effectively manage the tracking-and-pointing control task for an ADS (Figure 3), we begin by establishing a dynamic model that captures the essential coupling behaviors of the system. The tracking platform is represented as a rigid-body mechanism with two degrees of freedom (2-DOF). Let the joint-angle vector be defined as $q={[\theta ,\beta ]}^T$ , where $\theta$ represents the azimuth angle and $\beta$ represents the elevation angle. The coupled dynamic equations governing the motion of the system are formulated as follows:

$$M\left(q\right)\stackrel{··}{q}+C(q,\stackrel{·}{q})\stackrel{·}{q}+G\left(q\right)+D\stackrel{·}{q}+{\tau }_d=\tau$$

(6)

Figure 3.

Dynamic model of two-axis tracking and pointing platform.

Figure 3 illustrates the structural and functional configuration of the tracking-and-pointing platform within ADS. The azimuth axis rotates around the vertical vector to control the horizontal heading, providing panoramic field coverage, while the pitch axis rotates around the horizontal vector to adjust the vertical tilt of the payload.

Based on the system’s physical properties, the corresponding mathematical formulations for its key dynamic components are defined as follows:

The positive-definite inertia matrix $M(q)$ characterizes the system’s mass distribution and satisfies as $m_{1}I<M\left(q\right)<m_{2}I$ .

$$M\left(q\right)=\left[\begin{array}{cc}{I}_{\theta }+m{L}^2{\sin }^2\beta \hfill & 0\hfill \\ \multicolumn{1}{c}{0}& I_{\beta }+m{L}^2\hfill \end{array}\right]$$

(7)

The coupling effects during rotational motion are described as follows:

$$C(q,\stackrel{·}{q})=\left[\begin{array}{cc}0\hfill & -m{L}^2\stackrel{·}{\beta }\sin \beta \cos \beta \hfill \\ \multicolumn{1}{c}{m{L}^2\stackrel{·}{\theta }\sin \beta \cos \beta }& 0\hfill \end{array}\right]$$

(8)

The gravitational effect $G(q)$ primarily influences the elevation axis, which can be expressed as follows:

$$G\left(q\right)=\left[\begin{array}{c}0\hfill \\ \multicolumn{1}{c}{\mathrm{mgLcos}\beta }\end{array}\right]$$

(9)

The damping matrix $D$ and disturbance vector ${\tau }_d$ represent the system’s inherent damping and external non-matching disturbances, respectively.

The control objective is to ensure that the system meets the prescribed RMSE criteria by driving the actual state vector $q$ to closely follow the desired trajectory $q_d$ . The system tracking error is defined as follows:

$$e\left(t\right)=q_d\left(t\right)-q(t)$$

(10)

RMSE can be obtained as follows:

$$\mathrm{RMSE}=\sqrt{\frac{1}{N}\sum _{i=1}^N{\left|\left|e_i\right|\right|}^2}$$

(11)

ISMC and improved RL-based adaptive optimization strategy

Construction of integral sliding variable

The ISMC is constructed by first defining a suitable sliding variable, which shapes the system’s motion to achieve the desired dynamic response. This design helps eliminate steady-state error while enhancing transient performance. The sliding variable is formulated as follows:

$$s\left(t\right)=\stackrel{·}{e}\left(t\right)+\rho e\left(t\right)+K{\int }_0^{t}e\left(\tau \right)d\tau -s_0$$

(12)

where $e$ represents the tracking error, consisting of the elevation and azimuth angle components; $\rho$ and $K$ are positive definite diagonal gain matrices; $s_0$ is the initial compensation term defined by the initial system states:

$$s_0=\stackrel{·}{e}\left(0\right)+\rho e\left(0\right)$$

(13)

By substituting $t=0$ into equation (12), it is evident that:

$$s\left(0\right)=\stackrel{·}{e}\left(0\right)+\rho e\left(0\right)+K·0-[\stackrel{·}{e}\left(0\right)+\rho e\left(0\right)]=0$$

(14)

Remark: By setting $s\left(0\right)=0$ , the system starts directly on the sliding manifold. This eliminates the reaching phase, thereby guaranteeing robustness against matched uncertainties and disturbances throughout the entire control process. And the time derivative of the sliding variable remains:

$$\stackrel{·}{s}=\stackrel{··}{e}+\rho \stackrel{·}{e}+\mathrm{Ke}$$

(15)

Based on the integral sliding variable, the control law is formulated to drive the system states to the sliding manifold within finite time and to ensure sustained sliding motion thereafter.

The control input τ is composed of two main components: the nominal control, which reflects the known system dynamics, and the switching control, designed to counteract external disturbances and uncertainties.

$$\tau =M\left(q\right)u+C(q,\stackrel{·}{q})\stackrel{·}{q}+G\left(q\right)+D\stackrel{·}{q}$$

(16)

The auxiliary control component $u$ is defined by the following expression:

$$u={\stackrel{··}{q}}_d+\rho \stackrel{·}{e}+\mathrm{Ke}+K_s\mathrm{sat}(s/\varphi )$$

(17)

Where ${\stackrel{··}{q}}_d$ denotes the desired acceleration vector. $K_s=\mathrm{diag}(K_{s1},K_{s2})$ represents the switching gain matrix, which determines the disturbance-rejection capability; $\mathrm{sat}(·)$ represents the saturation function; and $\phi$ represents the boundary-layer thickness. Together, the latter two mitigate the inherent chattering of sliding-mode control.

Substituting the control law (equation (16)) into the system dynamics (equation (6)) yields:

$$M\left(q\right)\stackrel{··}{q}+{\tau }_d=M\left(q\right)u+C\stackrel{·}{q}+G+D\stackrel{·}{q}$$

(18)

Since the nonlinear terms are compensated by the control law and the inertia matrix $M\left(q\right)$ is invertible, the closed-loop dynamics simplifies to:

$$\stackrel{··}{q}=u-M^{-1}\left(q\right){\tau }_d$$

(19)

Substituting the expression for $u$ (equation (17)) and using the error definition (equation (10)), the error dynamics is obtained as:

$$\stackrel{··}{e}=-\rho \stackrel{·}{e}-\mathrm{Ke}-K_s\mathrm{sat}\left(\frac{s}{\varphi }\right)+M^{-1}\left(q\right){\tau }_d$$

(20)

Combining this with the derivative of the sliding variable from equation (15), the decoupled sliding mode dynamics is derived as:

$$\stackrel{·}{s}=-K_s\mathrm{sat}\left(\frac{s}{\varphi }\right)+\delta \left(t\right)$$

(21)

where $\delta \left(t\right)=M^{-1}\left(q\right){\tau }_d$ represents the transformed matched disturbance.

Adaptive optimization strategy via improved RL

Traditional sliding-mode control often depends on empirically tuned parameters, which limits its adaptability in the face of rapidly changing or uncertain environments. To address this limitation, we integrate AdaDDQN into the control framework, enabling online optimization of key parameters.

The parameter tuning process is reformulated as a Markov Decision Process (MDP), characterized by the standard five-tuple M = (S, A, P, R, γ):

State S represents the system’s current dynamic characteristics, capturing essential information for decision-making.

Action A represents the incremental adjustment applied to the control parameters.

Reward R represents the evaluative feedback that guides the learning agent toward optimal policy selection.

To improve the agent’s sensitivity to trajectory complexity and control smoothness, we design a high-dimensional state space that embeds both diversity metrics and predictive information about the system’s future behavior.

$$s_t=[q,\stackrel{·}{q},\mathrm{diversit}{y}_D,\mathrm{diversit}{y}_F,\mathrm{FI}]$$

(22)

The following features are carefully designed to capture critical aspects of system dynamics and trajectory complexity:

State-space diversity ( $\mathrm{diversit}{y}_D$ ) captures how widely the current state samples are distributed within the system’s state space.

$$\mathrm{diversit}{y}_D=\frac{2}{n(n-1)}\sum _{i=1}^n\sum _{j=1}^n\mathrm{euclidean}({s_t}_i,s_{\mathrm{tj}})$$

(23)

Target-space diversity ( $\mathrm{diversit}{y}_F$ ) captures the structural complexity of the normalized target trajectory Obj.

$$\mathrm{diversit}{y}_F=\frac{2}{n(n-1)}\sum _{i=1}^n\sum _{j=1}^n\mathrm{euclidean}(\mathrm{Ob}{j}_i,\mathrm{Ob}{j}_j)$$

(24)

where Obj represents the normalized target vector, which can be determined as follows:

$$\mathrm{Obj}=\frac{s_t-\min (s_t)}{max\left(s_t\right)-\min (s_t)}$$

(25)

By jointly analyzing the dispersion of state samples and the complexity of the normalized target trajectory, the system gains improved resilience to distributional shifts. This, in turn, enhances the generalization capability of the policy across varying control scenarios.

The feature vector includes future-informed indicators—collectively denoted as FI—which capture predictive information about target motion and disturbance trends. The FI component is structured into three elements: $F{I}_1,F{I}_2,\mathrm{and}F{I}_3$ .

$F{I}_1$ (cumulative error) reflects the system’s historical control performance:

$$F{I}_1=\int \mathrm{edt}\approx \sum _{i=0}^k{e}_i\mathrm{\Delta }t$$

(26)

$F{I}_2$ (mean of the sliding variable) reflects the recent convergence status of the sliding variable:

$$F{I}_2=\frac{1}{10}\sum _{i=k-9}^k|s_i|$$

(27)

$F{I}_3$ (torque rate-of-change) reflects the smoothness of the control output:

$$F{I}_3=\frac{\left||{\tau }_k-{\tau }_{k-1}|\right|}{\mathrm{\Delta }t}$$

(28)

The structure of the action space is carefully designed to suit the needs of online parameter tuning in a dynamic control environment.

The action space is structured to enable real-time adaptation of control parameters, ensuring the agent can respond effectively to changes in dynamic and uncertain environments:

$$a_t={[\mathrm{\Delta }{\rho }_1,\mathrm{\Delta }{\rho }_2,\mathrm{\Delta }{K}_1,\mathrm{\Delta }{K}_2,\mathrm{\Delta }{K}_{s1},\mathrm{\Delta }{K}_{s2}]}^T$$

(29)

The adjustable parameters include the sliding-variable coefficients, integral gains, and switching gains, all defined separately for the azimuth and elevation control channels.

For the switching gain adjustment ( $\mathrm{\Delta }{K}_s$ ): The action range is defined as $[-25.0,25.0]$  N·m. This wide range is critical for the RL agent to instantly boost the control gain from its nominal value to a high saturation level when a shock disturbance is detected.

For the sliding variable parameter adjustment ( $\mathrm{\Delta }\rho$ ): The action range is set to $[-10.0,10.0]$ . This allows the agent to significantly modulate the system’s bandwidth in real-time, increasing stiffness during fast maneuvers and reducing it during steady states to minimize noise sensitivity.

For the integral gain Adjustment ( $\mathrm{\Delta }K$ ): The integral gain is kept relatively stable with a narrow adjustment range ( $[-0.1,0.1]$ in practice) to preserve the steady-state error elimination capability without introducing integral windup oscillations.

To improve both learning stability and decision quality, the proposed AdaDDQN network leverages a Dueling DDQN architecture. This structure incorporates two key mechanisms:

DDQN introduces a separate target network to compute the estimated Q-values. By decoupling action selection from value evaluation, this approach effectively reduces the overestimation bias often encountered in standard DQN methods.

$$L\left(\theta \right)=E\left[{(R+\gamma \underset{a_{t+1}}{\max }Q(s_{t+1},a_{t+1};{\theta }^{*})-Q(s_t,a_t;\theta ))}^2\right]$$

(30)

The dueling network separates the Q-value into a state-value $V$ and an advantage function $A$ , allowing the model to better distinguish between the importance of states and the relative utility of actions:

$$\begin{array}{c}\multicolumn{1}{c}{(s_t,a_t)=V\left(s_t\right)+(A(s_t,a_t)-\frac{1}{\left|A\right|}\sum _{a_{t+1}}A(s_t,a_{t+1}))}\\ \multicolumn{1}{c}{(s_t,a_t)=V\left(s_t\right)+(A(s_t,a_t)-\frac{1}{\left|A\right|}\sum _{a_{t+1}}A(s_t,a_{t+1}))}\end{array}$$

(31)

The reward function is constructed to penalize angular tracking errors while simultaneously incorporating constraints on control torque magnitude and variations. The mathematical formulation of the reward is as follows:

$$r=10e^{-10||e||}-{\delta }_1\left|\left|\tau \right|\right|-{\delta }_2|\left|\mathrm{\Delta }\tau \right||$$

(32)

Specifically, the first term in the reward function applies an exponential penalty to the tracking error, encouraging precise alignment with the target trajectory. The second term serves as a constraint on the control energy, limiting excessive torque output. The third term discourages large control adjustments by penalizing the magnitude of changes in torque. This adjustment penalty $||\mathrm{\Delta }\tau ||$ is formulated as follows:

$$\left|\left|\tau \right|\right|=\mathrm{\Delta }{\rho }_p^2+\mathrm{\Delta }{K}_p^2+\mathrm{\Delta }{\rho }_a^2+\mathrm{\Delta }{K}_a^2$$

(33)

where $\mathrm{\Delta }{\rho }_p$ and $\mathrm{\Delta }{\rho }_a$ represent the variations of the sliding-surface error-term coefficients for the azimuth and elevation channels, respectively; while $\mathrm{\Delta }{K}_p$ and $\mathrm{\Delta }{K}_a$ represent the variations of the integral-term coefficients for azimuth and elevation, respectively. Overall, the penalty structure comprehensively incorporates azimuth and elevation tracking errors, sliding-surface deviations, integral terms, and control effort. In the RL process, the exponential error term ensures large deviations are rapidly penalized, encouraging fast convergence. Meanwhile, constraints on torque magnitude prevent excessive control energy, and smoothing the parameter updates helps reduce oscillations that would otherwise arise from abrupt changes.

Stability analysis and ultimate boundedness

To rigorously verify the stability of the proposed control scheme under the influence of the RL-optimized parameters and the saturation function, we first establish the necessary assumptions and then derive the stability theorem based on Lyapunov theory.

Assumption 1 (Bounded Disturbance): The transformed disturbance term $\delta \left(t\right)$ is bounded, satisfying $|\delta \left(t\right)|\le {\mathrm{\Delta }}_{\max }$ . Given that the inertia matrix $M\left(q\right)$ is positive definite and bounded, this assumption is consistent with the physical constraints of the gimbal system.

Assumption 2 (RL Safety Constraint): Although the switching gain $K_s$ is dynamically adjusted by the AdaDDQN agent, its action space is strictly constrained to ensure stability. Specifically, the minimum eigenvalue of the switching gain satisfies the lower bound condition: ${\lambda }_{\min }\left(K_s\left(t\right)\right)\ge {\mathrm{\Delta }}_{\max }+\eta$ , where $\eta >0$ is a positive safety margin.

Theorem 1 (Uniform Ultimate Boundedness): Consider the gimbal dynamics described by equation (16). Under Assumptions 1 and 2, and utilizing the control law with the saturation function $\mathrm{sat}(s/\varphi )$ , the closed-loop system is Uniformly Ultimately Bounded (UUB). Both the sliding variable $s$ and the tracking error $e$ will converge to a compact residual set centered at the origin.

To formally establish this result, we construct the following Lyapunov candidate function:

$$V=\frac{1}{2}{s}^{T}s$$

(34)

Differentiating $V$ with respect to time and substituting the closed-loop dynamics, we obtain:

$$\stackrel{·}{V}=s^T\stackrel{·}{s}=s^T\delta \left(t\right)-s^T{K}_s\mathrm{sat}\left(\frac{s}{\varphi }\right)$$

(35)

We analyze the stability in two regions based on the boundary layer thickness $\varphi$ :

Utilizing the skew-symmetric property $s^T(\stackrel{·}{M}-2C)s=0$ and substituting the control law $\tau =M{\stackrel{··}{q}}_{\mathrm{eq}}-K_s\mathrm{sat}(s/\varphi )$ , the derivative simplifies to:

$$\stackrel{·}{V}=s^T{\tau }_d-s^T{K}_s\mathrm{sat}\left(\frac{s}{\varphi }\right)$$

(36)

We analyze the stability in two regions based on the boundary layer thickness $\varphi$ :

• Case 1: Outside the Boundary Layer ( $|s|\ge \varphi$ ) In this region, the saturation function operates as a sign function, that is, $\mathrm{sat}(s/\varphi )=\mathrm{sgn}\left(s\right)$ . Thus:

$$\begin{array}{c}\multicolumn{1}{c}{\stackrel{·}{V}=s^T\delta \left(t\right)-s^T{K}_s\mathrm{sgn}\left(s\right)\le \left|s\right|\left|\delta \left(t\right)\right|}\\ \multicolumn{1}{c}{-{\lambda }_{\min }\left(K_s\right)\left|s\right|\le \left|s\right|{\mathrm{\Delta }}_{\max }-{\lambda }_{\min }(K_s)|s|}\end{array}$$

(37)

By Assumption 2, since ${\lambda }_{\min }\left(K_s\right)\ge {\mathrm{\Delta }}_{\max }+\eta$ , it follows that:

$$\stackrel{·}{V}\le -\eta |s|<0$$

(38)

This ensures that any system trajectory starting outside the boundary layer will reach the boundary layer region in finite time.

• Case 2: Inside the Boundary Layer ( $|s|<\varphi$ ) In this region, the saturation function behaves linearly, that is, $\mathrm{sat}(s/\varphi )=s/\varphi$ . The derivative becomes:

$$\stackrel{·}{V}=s^T\delta \left(t\right)-s^T{K}_s\frac{s}{\varphi }\le |s|{\mathrm{\Delta }}_{\max }-\frac{{\lambda }_{\min }\left(K_s\right)}{\varphi }|s{|}^2$$

(39)

For $\stackrel{·}{V}$ to remain negative, the condition $|s|{\mathrm{\Delta }}_{\max }<\frac{{\lambda }_{\min }\left(K_s\right)}{\varphi }|s{|}^2$ implies:

$$|s|>\frac{{\mathrm{\Delta }}_{\max }\varphi }{{\lambda }_{\min }\left(K_s\right)}$$

(40)

Consequently, the sliding variable $s$ does not converge to zero but to an ultimate bound set ${\mathrm{\Omega }}_s$ :

$${\mathrm{\Omega }}_s=\{s:|s|\le \frac{{\mathrm{\Delta }}_{\max }\varphi }{{\lambda }_{\min }\left(K_s\right)}\}$$

(41)

In the steady state, the dynamics of the sliding variable are dominated by $\stackrel{·}{e}+\rho e\approx s$ . This can be viewed as a stable first-order linear filter with input $s$ and state matrix $-\rho$ (since $\rho$ is positive definite). The ultimate bound of the tracking error $|e|$ is derived as:

$$|e{|}_{\mathrm{bound}}\le \frac{|s{|}_{\mathrm{bound}}}{{\lambda }_{\min }\left(\rho \right)}=\frac{{\mathrm{\Delta }}_{\max }\varphi }{{\lambda }_{\min }\left(\rho \right){\lambda }_{\min }\left(K_s\right)}$$

(42)

Equation (42) provides the theoretical lower bound of the tracking error. It explicitly reveals that the error bound is proportional to the disturbance magnitude ${\mathrm{\Delta }}_{\max }$ and boundary layer thickness $\varphi$ , and inversely proportional to the control gains $K_s$ and $\rho$ . The proposed AdaDDQN method is theoretically grounded in this relationship; by dynamically maximizing the gains $K_s$ and $\rho$ (within stability constraints) and optimizing $\varphi$ , the RL agent minimizes the theoretical error bound $|e{|}_{\mathrm{bound}}$ , thereby achieving high-precision tracking performance while mitigating chattering.

Simulation and experiment results

To evaluate the performance of the proposed adaptive ISMC, which integrates AdaDDQN, a comprehensive validation framework is established. The experimental setup includes both numerical simulations and physical hardware testing, aimed at examining the system’s tracking accuracy, response speed, and robustness in the presence of nonlinear friction, external disturbances, and structural uncertainties.

A simulation model of the two-DOF gimbal platform is constructed based on its physical configuration. The system dynamics—including the inertia matrix $M(q)$ , Coriolis matrix $C(q,\stackrel{·}{q})$ , and gravitational term $G(q)$ —are derived using real-world mechanical parameters. The key configuration values are summarized in Table 1.

Table 1.

Physical parameters of two-axis tracking and pointing platform.

Parameters	Physical significance	Values
$m$	Load mass	35 kg
$L$	Distance to the rotation axis	0.8 m
$I_{\theta }$	Azimuthal axis moment of inertia	8 $\mathrm{kg}·m^2$
$I_{\beta }$	Pitch axis rotational inertia	12.5  $\mathrm{kg}·m^2$
$d_{\theta }$	Azimuth axis damping	0.2  $N·m·s/\mathrm{rad}$
$d_{\beta }$	Pitch axis damping	0.3  $N·m·s/\mathrm{rad}$
g	Acceleration of gravity	9.8  $m/s^2$

To closely replicate real-world operating conditions, the simulation environment incorporates nonlinear friction and stochastic disturbances.

Nonlinear friction modeling: A Stribeck friction model is employed to capture the key characteristics of mechanical friction, including breakaway static friction, Coulomb friction, and viscous damping. The parameters are configured as follows: maximum breakaway torque $T_{\mathrm{break}}=3.25N·m$ , Coulomb torque $T_c=2.5N·m$ , and viscous coefficient $f=1.2\times {10}^{-3}$ .

Random torque disturbances: To emulate torque ripple inherent in motor drive systems, a zero-mean random noise is applied with an amplitude of 4% of the instantaneous motor output torque ( $0.04T_m$ ).

The simulation is conducted over an extended duration to assess the long-term performance of the proposed controller. Initial conditions include an azimuth angle offset of 0.1 rad and an elevation angle offset of −0.05 rad, with both angular velocities initialized to zero. A composite reference trajectory is applied to evaluate the system’s tracking accuracy and dynamic response under varying conditions. Figure 4 illustrates the azimuth tracking performance, where the desired trajectory is shown as a red dashed line, and the actual response as a blue solid line. During the first 1–2 s, a noticeable discrepancy is observed due to the initial misalignment and the RL agent not yet having optimized the control parameters. As learning progresses, the agent continuously refines the control strategy, leading to a gradual reduction in tracking error. After a short transient period, the actual trajectory closely aligns with the reference, demonstrating high-precision tracking. A similar behavior is observed in Figure 5, which presents the elevation (pitch) tracking results. After an initial adjustment phase, the tracking error rapidly diminishes and remains negligible throughout the remainder of the simulation, confirming the method’s robust dynamic performance across both motion axes.

Figure 4.

Azimuth tracking analysis results.

Figure 5.

Pitch tracking analysis results.

Figure 6 presents the tracking error curves for both azimuth and elevation channels, shown in blue and red, respectively. Immediately following system activation, both error signals exhibit rapid decay. Once the initial transient subsides, the tracking errors remain consistently within the target threshold of 10 mrad. Minor oscillations are observed, primarily due to external disturbances and the system’s inherent nonlinearities. However, thanks to the synergistic effect of integral sliding-mode control and adaptive RL, these fluctuations are effectively suppressed. The RL agent’s continuous parameter tuning ensures ongoing error convergence and robust performance over time.

Figure 6.

Tracking errors analysis results.

Figures 7–11 illustrate the internal dynamics and adaptive behavior of the control system. Figure 7 shows the output torque response, reflecting how the system modulates control effort under dynamic conditions. Figure 8 presents the evolution of the sliding variable, which rapidly converges toward zero, indicating the effectiveness of the sliding-mode strategy in stabilizing the trajectory. In Figure 9, the adaptive parameter $\rho$ . Figure 10 displays the RL agent’s training performance, where the upward trend of the cumulative reward clearly demonstrates progressive policy optimization and improved control quality. Figure 11 illustrates the changes in two adaptive gains, both of which influence the behavior of the saturation function embedded in the sliding-mode controller; these gains directly affect the system’s convergence speed and play a key role in suppressing chattering during the tracking process.

Figure 7.

Control torque analysis results.

Figure 8.

Sliding variable analysis results.

Figure 9.

$\rho$ analysis results.

Figure 10.

Reward analysis results.

Figure 11.

Adaptive adjustment of $K_s$ analysis results.

Relative to conventional ISMC, the proposed approach demonstrates significantly enhanced performance across several key dimensions.

In Figure 12, the blue curve represents the proposed RL-enhanced control method, the yellow dashed line shows the desired trajectory, and the red curve corresponds to the conventional ISMC baseline. The proposed method achieves high tracking precision, with minimal deviation from the reference across the full trajectory. While the ISMC baseline performs adequately in azimuth tracking, it exhibits noticeable oscillations during the initial transient phase and slightly lower steady-state accuracy. Figure 13 reveals a more pronounced difference in the elevation channel: the ISMC baseline suffers from an observable response delay, whereas the RL-improved controller maintains close adherence to the target trajectory. This performance gap highlights the advantage of online adaptation; the RL component dynamically adjusts control parameters, enabling rapid convergence to near-optimal configurations without requiring manual tuning. In contrast, traditional ISMC relies on fixed or heuristically selected parameters, which may require iterative adjustment to suit different dynamics. The absence of delay in azimuth tracking for the ISMC baseline can be attributed to a favorable initial parameter setting, but this does not generalize across axes or tasks. Overall, the results demonstrate that the proposed RL-ISMC framework consistently delivers better tracking performance, improved stability, and faster convergence compared to the conventional ISMC approach.

Figure 12.

Azimuth tracking analysis results.

Figure 13.

Pitch tracking analysis results.

Another simulation is designed to rigorously evaluate the contribution of each component (RL adaptation and Integral Sliding Mode) under challenging, realistic conditions. The study compares six distinct control strategies to isolate the effects of the Integral term and the RL-based adaptation: The complete framework combining Integral Sliding Mode Control with adaptive RL tuning of gains ( $\rho ,K,K_s$ ). It features high base gains and strong integral action (AdaDDQN-ISMC); ISMC with fixed, pre-tuned gains. This tests the necessity of online adaptation by removing the RL agent’s dynamic tuning (DDQN-ISMC); Standard Sliding Mode Control without the integral term. This tests the necessity of the integral action for eliminating steady-state errors (SMC); Adaptive RL applied to standard SMC (no integral). This tests if RL alone can compensate for the lack of an integral term (AdaDDQN-SMC). A non-adaptive ISMC baseline with standard gains, serving as a traditional robust control benchmark (ISMC-NoRL). A classic PID controller for broad performance comparison (PID).

A continuous torque bias of approximately 8 Nm was applied to simulate unmodeled gravitational moments and constant friction and a sudden shock disturbance of 5 Nm was introduced from $t=5s$ to $t=7s$ to evaluate the system’s reaction to abrupt external forces. Six control strategies were compared as shown in Figure.

Figures 14 and 15 illustrate the tracking trajectories for the pitch and azimuth angles, respectively. A comparative analysis demonstrates that the proposed method exhibits the best performance in eliminating errors induced by the 8 N·m bias. Furthermore, it maintains superior tracking stability even when a shock disturbance is introduced at $t=5$ s. In contrast, the fixed-gain DDQN-ISMC and ISMC-NoRL methods display significant deviation peaks due to insufficient gain stiffness, whereas the proposed AdaDDQN-ISMC effectively mitigates this by rapidly adjusting its switching gain, $K_s$ .

Figure 14.

Pitch tracking analysis with ablation: (a) pitch position tracking, (b) error of pitch tracking, (c) torque for pitch tracking and (d) $K_s$ for pitch tracking.

Figure 15.

Azimuth tracking analysis with ablation: (a) Azimuth position tracking, (b) error of azimuth tracking, (c) torque for Azimuth tracking and (d) $K_s$ for Azimuth tracking.

Notably, the comparison between AdaDDQN-ISMC and AdaDDQN-SMC highlights the critical role of the integral term. Methods lacking an integral sliding variable (specifically SMC and AdaDDQN-SMC) failed to eliminate the steady-state error caused by the 8 N·m bias. Although the RL agent attempted to compensate by increasing the gain, the AdaDDQN-SMC method still exhibited a persistent non-zero mean error. Conversely, both the proposed AdaDDQN-ISMC and the fixed ISMC methods successfully converged to a near-zero steady-state error, confirming that integral action is indispensable for handling constant unmodeled loads.

To evaluate the proposed control method under real-world conditions, a physical trajectory-tracking experiment was conducted using a two-axis gimbal platform (Figure 16). A predefined target trajectory was issued to the system, and the gimbal executed the motion commands while onboard sensors continuously recorded its response. The reference trajectory and real-time sensor data were processed by the RL-improved ISMC algorithm, which computed the appropriate control signals for each actuator. These control outputs were then applied to drive the gimbal in real time. Once the motion was completed, the recorded position feedback was compared with the original command trajectory to assess the tracking performance of the physical system.

Figure 16.

Experimental setup for control validation: (a) control box interior, (b) gimbal joint, (c) control box and host computer and (d) gimbal assembly.

The gimbal control system is composed of three primary components: the mechanical gimbal structure, the gimbal control box, and a host computer functioning as the upper-level controller. Within the control box, several integrated modules are responsible for core functions, including an embedded processor, a pulse signal generator, a data acquisition card, and a communication interface. The pulse generator translates control instructions into electrical pulse signals that directly drive the gimbal motors. The acquisition card continuously samples the state signals from onboard sensors and structures the data for processing. Acting as the central processing unit, the embedded motherboard receives both sensor data and control commands via the communication module. It processes incoming signals from the host, packages sensor data collected by the acquisition card, and transmits it back to the host. Simultaneously, it converts the host-generated control commands into motor control signals via the pulse generator. At the upper level, the host computer synthesizes real-time sensor feedback, the user-defined target trajectory, and predefined environmental disturbances, applying the proposed RL-based control algorithm to compute control outputs, which are then dispatched to the control box. The AdaDDQN agent is first trained in a high-fidelity simulation environment to learn the optimal policy for adjusting control parameters. The trained policy is then deployed to the physical gimbal system. During this phase, the agent operates in real-time inference mode.

During runtime, the host continuously uses the current system state and reference trajectory as inputs to generate the next-step control command. These commands are transmitted to the control box, which actuates the motors accordingly. Sensor modules then acquire the resulting gimbal motion in real time and return the measurements to the host. By comparing the feedback data with the reference trajectory, the tracking error is calculated to evaluate performance. Unlike simulation conditions, the physical setup introduces additional challenges such as stronger mechanical vibrations and unmodeled uncertainties, which can influence system behavior. The following section presents the experimental tracking results obtained under these real-world operating conditions.

In the physical testing environment, the gimbal was subjected to additional unmodeled factors including stronger mechanical vibrations and environmental disturbances not present in the simulation setup.

The results indicate that the maximum tracking error in azimuth remained below 0.01°, while the maximum error in elevation was under 0.03°, as can be seen from Figures 17 and 18. Despite the increased complexity and unpredictability of the real-world conditions, the system maintained stable operation without any signs of divergence or significant oscillation. These findings confirm the proposed algorithm’s strong robustness and its effective adaptability to hardware platforms.

Figure 17.

Pitch tracking analysis results.

Figure 18.

Azimuth tracking analysis results.

Discussions

The core innovation of this study lies in the seamless integration of RL adaptability with the robust control guarantees of SMC. Traditional SMC approaches often rely on fixed switching gains to handle worst-case disturbances, which can lead to excessive chattering. In contrast, the proposed method leverages an AdaDDQN agent to dynamically adjust the switching gain $K_s$ and sliding variable parameters ρ and K, based on real-time system observations such as error integrals and torque rate changes. Experimental results show that $K_s$ adapts proportionally to disturbance intensity, enabling the system to maintain a flexible trade-off between rapid convergence and smooth control. To mitigate phase lag—common in dynamic target tracking—a multi-dimensional state space is introduced, incorporating diversity metrics and future-information (FI) parameters, which improve the controller’s perception of trajectory complexity and system smoothness. The RL agent then predicts angular-velocity increments to preemptively counteract delays. Notably, an azimuth-axis tracking error below 0.52 mrad (0.03°) validates the method’s effectiveness in addressing mechanical latency. The algorithm further benefits from a Dueling DDQN architecture, which decouples action selection from value estimation, thereby enhancing stability and avoiding the common issue of value overestimation in high-dimensional continuous spaces.

Compared with conventional ISMC and the foundational design requirements, the proposed AdaDDQN-ISMC delivers measurable performance gains, validated through rigorous ablation analysis. Firstly, regarding steady-state robustness and precision, the ablation study confirms the indispensability of the integral term. Under a continuous torque bias (simulating unmodeled gravity and friction), the conventional SMC and the ablated AdaDDQN-SMC variants exhibited persistent steady-state errors, as they lacked the error-accumulation mechanism to counteract the constant load. In contrast, the proposed method leverages the integral sliding surface to completely eliminate this bias. Consequently, the system consistently maintains tracking errors below 8 mrad, exceeding the 10 mrad accuracy threshold required for effective beam coverage. Measured results further highlight this precision, showing an elevation error of only 0.01° ( $\approx 0.17$  mrad), which is far below the theoretical design limit. Secondly, in terms of dynamic response and shock rejection, the RL-based adaptive mechanism proves superior to fixed-gain strategies. During the sinusoidal tracking tests, traditional ISMC suffered from noticeable phase lag and insufficient stiffness against the sudden shock disturbance. The fixed-gain ablation baselines (DDQN-ISMC) displayed significant deviation peaks during these transient events. Conversely, the proposed algorithm utilized the AdaDDQN agent to instantly boost the switching gain $K_s$ upon detecting the shock, aligning the output closely with the reference signal throughout the trajectory. This “on-demand” gain adjustment resulted in an improvement of more than 80% in tracking precision during discontinuous trajectory scenarios. Thirdly, concerning control smoothness and chattering suppression, the proposed method resolves the trade-off inherent in conventional sliding mode control. While traditional SMC requires high static gains to suppress nonlinear frictional disturbances (e.g. Stribeck effect), leading to steady-state oscillations, the proposed architecture uses the integral term to handle static loads, allowing the switching gain to remain low during steady states. By further incorporating a torque rate penalty in the reward function, the algorithm produces smoother torque profiles and effectively mitigates chattering, demonstrating superior disturbance rejection without compromising mechanical integrity.

Despite promising results in both simulation and hardware testing, several limitations remain. Firstly, the current evaluation is limited to the gimbal’s own nonlinear dynamics; in real-world deployment, platform-induced disturbances such as translational motion and carrier rocking (e.g. on vehicles or ships) introduce complex coupling effects. Future work will extend testing to high-DOF motion platforms to validate disturbance suppression under carrier-coupled dynamics. Secondly, while the simulation environment incorporates measured mechanical parameters, real-world factors such as aerodynamic drag and temperature-sensitive friction remain difficult to model accurately. Enhancing sim-to-real transfer through meta-RL or domain randomization is an important direction to improve generalization in unstructured environments. Lastly, the DRL model incurs non-negligible computational costs, which may challenge real-time deployment in resource-constrained embedded systems. Future research will explore network pruning, model compression, and hardware-aware policy acceleration to enable lightweight deployment in compact Active Denial devices.

Conclusions

The proposed method improves the tracking and pointing performance of microwave ADS. Experimental results indicate that the tracking accuracy increases by more than 80% compared with conventional SMC, with dynamic errors reliably constrained within 8 mrad. Hardware tests on a two-DOF gimbal platform show that the elevation and azimuth tracking errors remain below 0.01° and 0.03°, respectively—well within the threshold required to maintain beam jitter within half of the spot diameter. Validated by rigorous ablation studies, the integral sliding term effectively eliminates steady-state bias, while the RL agent—utilizing a novel torque-rate penalized reward mechanism—dynamically adjusts gains to mitigate phase lag and suppress chattering. This fusion of adaptive learning and robust control delivers reliable performance in both simulated and physical environments. According to the accuracy model, this level of precision corresponds to a target-lock probability of approximately 99.87% at an operational range of 250 m. By enabling online decision-making through RL, the approach effectively mitigates issues such as phase lag and torque chattering that commonly affect traditional controllers. This fusion of adaptive learning and robust control delivers reliable performance in both simulated and physical environments, offering a technically viable path toward enhancing the effectiveness of non-lethal directed energy systems.

Future work will focus on extending the validation of the proposed approach under more complex operational conditions. In particular, when mounted on mobile carriers such as vehicles or ships, the gimbal system experiences translational motion and rotational disturbances that introduce nonlinear coupling through kinematic interactions. To address this, further experiments will be conducted on dynamic platforms to assess the algorithm’s ability to reject carrier-induced disturbances. In addition, considering the computational load of DRL in embedded systems, we plan to investigate model compression techniques and edge computing frameworks for real-time deployment. Additional efforts will be made to examine the influence of unstructured environmental factors—including extreme temperatures, wind loading, and mechanical vibration—on control performance, with the goal of improving the system’s generalization ability and responsiveness in field applications.