Spiking neural networks (SNNs), with temporal dynamics and biologically inspired spiking behavior, offer improved interpretability and performance in real-time control. This paper integrates reinforcement learning (RL) with reward-modulated, bio-inspired SNNs that employ reward-modulated spike-timing-dependent plasticity (R-STDP) to address real-time control tasks. The proposed model enables bio-inspired learning by combining conventional spike-timing-dependent plasticity (STDP) with time-dependent, reward-modulated weight updates. Furthermore, it reconstructs the task-oriented state space and reward mechanism to realize a cross-task control framework. Experimental results from control tasks, along with comparisons to traditional RL methods, demonstrate the effectiveness of SNNs in both simulation and real-world applications. This study aims to bridge the gap between SNNs and traditional feedback mechanisms in control systems, highlighting how bio-inspired approaches can enhance adaptive control.
BarbeyAKKoenigsMGrafmanJ. Dorsolateral prefrontal contributions to human working memory. Cortex2013; 49(5): 1195–1205.
2.
MitchellAS. The mediodorsal thalamus as a higher order thalamic relay nucleus important for learning and decision-making. Neurosci Biobehav Rev2015; 54: 76–88.
3.
AlexanderGECrutcherMD. Functional architecture of basal ganglia circuits: neural substrates of parallel processing. Trends Neurosci1990; 13(7): 266–271.
4.
SuttonRSBartoAG. Reinforcement learning: an introduction. MIT Press, 2018.
5.
BoraseRPMaghadeDSondkarS, et al. A review of PID control, tuning methods and applications. Int J Dyn Control2021; 9(2): 818–827.
6.
DörflerFTesiPDe PersisC. On the role of regularization in direct data-driven LQR control. In: 2022 IEEE 61st conference on decision and control (CDC), Cancun, Mexico, 6–9 December 2022, pp.1091–1098. IEEE.
7.
LavretskyE. Robust and adaptive control methods for aerial vehicles. In: ValavanisKPVachtsevanosGJ (eds) Handbook of unmanned aerial vehicles. Springer, 2015, pp.675–710.
8.
CarluchoIDe PaulaMAcostaGG. An adaptive deep reinforcement learning approach for MIMO PID control of mobile robots. ISA Trans2020; 102: 280–294.
9.
GuanZYamamotoT. Design of a reinforcement learning PID controller. IEEJ Trans Electr Electron Eng2021; 16(10): 1354–1360.
10.
BertsekasD. Dynamic programming and optimal control: volume I, Vol. 4. Athena Scientific, 2012.
11.
LiuYNiYDongC, et al. Task scheduling for control system based on deep reinforcement learning. Neurocomputing2024; 610: 128609.
12.
ZhangFZhuZChenT, et al. Reinforcement learning-based neural control for discrete-time nonlinear systems via deterministic learning. Nonlinear Dyn2025; 113(9): 9981–10003.
13.
HaouariBMzidRMosbahiO. A reinforcement learning-based approach for online optimal control of self-adaptive real-time systems. Neural Comput Appl2023; 35(27): 20375–20401.
TruongTNVoATKangH-J. Neural network-based sliding mode controllers applied to robot manipulators: a review. Neurocomputing2023; 562: 126896.
16.
Ribu HassiniSGireesh KumarTKowshik HurshanS. A machine learning and deep neural network approach in industrial control systems. In: FongSDeyNJoshiA (eds) ICT analysis and applications. Springer, 2022, pp.525–536.
17.
KocherlaRMoluguSNandalalV, et al. Deep learning based stacked recurrent neural networks for intrusion detection in industrial control systems using bio inspired Meta heuristics. In: MallikBBDeyasiKDasS, et al. (eds) International conference on emerging trends in mathematical sciences & computing. Springer, 2024, pp.207–221.
18.
BingZMeschedeCRöhrbeinF, et al. A survey of robotics control based on learning-inspired spiking neural networks. Front Neurorobot2018; 12: 35.
ZhaoFZengYHanB, et al. Nature-inspired self-organizing collision avoidance for drone swarm based on reward-modulated spiking neural network. Patterns2022; 3(11): 100611.
21.
DattaGKunduSJaiswalAR, et al. ACE-SNN: algorithm-hardware co-design of energy-efficient & low-latency deep spiking neural networks for 3d image recognition. Front Neurosci2022; 16: 815258.
22.
GeorgeJKSociCMiscuglioM, et al. Symmetry perception with spiking neural networks. Sci Rep2021; 11(1): 5776.
23.
LuoXQuHWangY, et al. Supervised learning in multilayer spiking neural networks with spike temporal error backpropagation. IEEE Trans Neural Netw Learn Syst2022; 34(12): 10141–10153.
24.
ZhangZSongHWangH, et al. A novel brain-inspired approach based on spiking neural network for cooperative control and protection of multiple trains. Eng Appl Artif Intell2024; 127: 107252.
25.
AzimiradVKhodkamSYBolouriA. A new hybrid learning control system for robots based on spiking neural networks. Neural Netw2024; 180: 106656.
26.
VolinskiAZaidelYShalumovA, et al. Data-driven artificial and spiking neural networks for inverse kinematics in neurorobotics. Patterns2022; 3(1): 100391.
27.
KimDLeeCHanS. Quadrotor control design with spiking neural networks. In: 2024 24th international conference on control, automation and systems (ICCAS), Jeju, Republic of Korea, 29 October–1 November 2024, pp.499–504. IEEE.
28.
Linares-BarrancoAPronoLLengensteinR, et al. Adaptive robotic arm control with a spiking recurrent neural network on a digital accelerator. In: 2024 31st IEEE international conference on electronics, circuits and systems (ICECS), Nancy, France, 18–20 November 2024, pp.1–4. IEEE.
29.
AranaDCScolaIREspinozaES, et al. Real-time implementation of a spiking neural network-based control: an application for the ball and plate system. In: 2024 American control conference (ACC), Toronto, ON, Canada, 10–12 July 2024, pp.3100–3105. IEEE.
30.
Barros e SáGCMadeiraCAG. Deep reinforcement learning in real-time strategy games: a systematic literature review. Appl Intell2025; 55(4): 243.
31.
MnihVKKavukcuogluDSilverAA, et al. Human-level control through deep reinforcement learning. Nature2015; 518(7540): 529–533.
HaarnojaTZhouAAbbeelP, et al. Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: International conference on machine learning, Stockholm, Sweden, 10–15 July 2018, pp.1861–1870. PMLR.
35.
FengQPengSLinY, et al. Reinforcement learning for real-time process control in high-temperature superconductor manufacturing. Int J Adv Manuf Technol2023; 129(5): 2215–2225.
36.
LiGYangYLiS, et al. Decision making of autonomous vehicles in lane change scenarios: deep reinforcement learning approaches with risk awareness. Transp Res Part C Emerg Technol2022; 134: 103452.
37.
ChuZWangFLeiT, et al. Path planning based on deep reinforcement learning for autonomous underwater vehicles under ocean current disturbance. IEEE Trans Intell Veh2022; 8(1): 108–120.
38.
KimDChoiMUmJ. Digital twin for autonomous collaborative robot by using synthetic data and reinforcement learning. Robot Comput Integr Manuf2024; 85: 102632.
39.
HaarnojaTMoranBLeverG, et al. Learning agile soccer skills for a bipedal robot with deep reinforcement learning. Sci Robot2024; 9(89): eadi8022.
40.
ZenkeFBohtéSMClopathC, et al. Visualizing a joint future of neuroscience and neuromorphic engineering. Neuron2021; 109(4): 571–575.
41.
CostaFSchaftEVHuiskampG, et al. Robust compression and detection of epileptiform patterns in ECoG using a real-time spiking neural network hardware framework. Nat Commun2024; 15(1): 3255.
42.
BäßlerDKortusTGühringG. Unsupervised anomaly detection in multivariate time series with online evolving spiking neural networks. Mach Learn2022; 111(4): 1377–1408.
43.
GuoLLiuDWuY, et al. Comparison of spiking neural networks with different topologies based on anti-disturbance ability under external noise. Neurocomputing2023; 529: 113–127.
44.
GoodmanDFBretteR. The Brian simulator. Front Neurosci2009; 3: 643.
45.
BekolayTBergstraJHunsbergerE, et al. Nengo: a python tool for building large-scale functional brain models. Front Neuroinform2014; 7(48): 1–13. https://doi.org/10.3389/fninf.2013.00048
46.
CarnevaleNTHinesML. The NEURON book. Cambridge University Press, 2006.
47.
FangWChenYDingJ, et al. SpikingJelly: an open-source machine learning infrastructure platform for spike-based intelligence. Sci Adv2023; 9(40): eadi1480. https://doi.org/10.1126/sciadv.adi1480
HazanHSaundersDJKhanH, et al. Bindsnet: a machine learning-oriented spiking neural networks library in python. Front Neuroinform2018; 12: 89.
50.
WangCJiangYLiuX, et al. A just-in-time compilation approach for neural dynamics simulation. In: Neural information processing: 28th international conference, ICONIP 2021, Sanur, Bali, Indonesia, 8–12 December 2021, Proceedings, Part III 28, pp.15–26. Springer.
51.
ZengYZhaoDZhaoF, et al. Braincog: a spiking neural network based, brain-inspired cognitive intelligence engine for brain-inspired AI and brain simulation. Patterns2023; 4(8): 100789.
52.
AbbottLF. Lapicque’s introduction of the integrate-and-fire model neuron (1907). Brain Res Bull1999; 50(5–6): 303–304.
53.
CaporaleNDanY. Spike timing–dependent plasticity: a Hebbian learning rule. Annu Rev Neurosci2008; 31: 25–46.
54.
IzhikevichEM. Simple model of spiking neurons. IEEE Trans Neural Netw2003; 14(6): 1569–1572.
55.
IzhikevichEM. Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb Cortex2007; 17(10): 2443–2452.
56.
ZhaoFZengYXuB. A brain-inspired decision-making spiking neural network and its application in unmanned aerial vehicle. Front Neurorob2018; 12: 56.
57.
FanYZhaoCZhangH, et al. A brain-inspired spiking neural network model for decision-making in reinforcement learning: intelligent self-balancing control. In: 2024 7th international conference on Universal Village (UV), Boston, MA, 19–22 October 2024, pp.1–8. IEEE.
58.
ZhangXHosseinNiaSH. Enhancing the reliability of closed-loop describing function analysis for reset control applied to precision motion systems. arXiv [preprint]. https://doi.org/10.48550/arXiv.2412.00502
59.
PiaoMLuBLiH, et al. Systematic control design for adaptive rejection of narrow-band disturbances based on a novel resonant generalized extended state observer. Electronics2024; 13(11): 2027.
60.
DampfhofferMMesquidaTValentianA, et al. Are SNNs really more energy-efficient than ANNs? An in-depth hardware-aware study. IEEE Trans Emerg Top Comput Intell2023; 7(3): 731–741. https://doi.org/10.1109/TETCI2022.3214509
61.
TangCHanJ. Hardware efficient weight-binarized spiking neural networks. In: 2023 design, automation & test in Europe conference & exhibition (DATE), Antwerp, Belgium, 17–19 April 2023, pp.1–6. IEEE. https://doi.org/10.23919/DATE56975.2023.10136955
