Reinforcement learning is an efficient, widely used machine learning technique that performs well when the state and action spaces have a reasonable size. This is rarely the case regarding control-related problems, as for instance controlling traffic signals. Here, the state space can be very large. In order to deal with the curse of dimensionality, a rough discretization of such space can be employed. However, this is effective just up to a certain point. A way to mitigate this is to use techniques that generalize the state space such as function approximation. In this paper, a linear function approximation is used. Specifically, with Fourier basis features is implemented to control traffic signals in the agent-based transport simulation MATSim. The results are compared not only to trivial controllers such as fixed-time, but also to state-of-the-art rule-based adaptive methods. It is concluded that with Fourier basis features is able to outperform such methods, especially in scenarios with varying traffic demands or unexpected events.
M.Abdoos, N.Mozayani and A.L.C.Bazzan, Hierarchical control of traffic signals using Q-learning with tile coding, Appl. Intell.40(2) (2014), 201–213. doi:10.1007/s10489-013-0455-3.
2.
L.N.Alegre, T.Ziemke and A.L.C.Bazzan, Using reinforcement learning to control traffic signals in a real-world scenario, an approach based on linear function approximation (forthcoming).
3.
L.Baird, Residual algorithms: Reinforcement learning with function approximation, in: Machine Learning Proceedings 1995, Morgan Kaufmann, 1995, pp. 30–37, http://www.sciencedirect.com/science/article/pii/B978155860377650013X. ISBN 978-1-55860-377-6. doi:10.1016/B978-1-55860-377-6.50013-X.
4.
A.L.C.Bazzan, Opportunities for multiagent systems and multiagent reinforcement learning in traffic control, Autonomous Agents and Multiagent Systems18(3) (2009), 342–375. doi:10.1007/s10458-008-9062-9.
5.
L.B.de Oliveira and E.Camponogara, Multi-agent model predictive control of signaling split in urban traffic networks, Transportation Research Part C: Emerging Technologies18(1) (2010), 120–139. doi:10.1016/j.trc.2009.04.022.
6.
M.Di Taranto, UTOPIA, in: Proc. of the IFAC-IFIP-IFORS Conference on Control, Computers, Communication in Transportation, International Federation of Automatic Control, Paris, 1989, pp. 245–252.
7.
C.Diakaki, M.Papageorgiou and K.Aboudolas, A multivariable regulator approach to traffic-responsive network-wide signal control, Control Engineering Practice10(2) (2002), 183–195. doi:10.1016/S0967-0661(01)00121-6.
8.
B.Friedrich, Adaptive signal control – an overview, in: Proc. of the 9th Meeting of the Euro Working Group Transportation, Bari, Italy, 2002.
9.
N.H.Gartner, OPAC – a demand-responsive strategy for traffic signal control, Transportation Research Record906 (1983), 75–81.
10.
W.Genders and S.Razavi, Evaluating reinforcement learning state representations for adaptive traffic signal control, Procedia Computer Science130 (2018), 26–33. doi:10.1016/j.procs.2018.04.008.
11.
D.Grether, J.Bischoff and K.Nagel, Traffic-actuated signal control: Simulation of the user benefits in a big event real-world scenario, in: 2nd International Conference on Models and Technologies for ITS, Leuven, Belgium, 2011.
12.
D.Grether and T.Thunig, Traffic signals and lanes, in: The Multi-Agent Transport Simulation MATSim, A.Horni, K.Nagel and K.W.Axhausen, eds, Ubiquity, London, 2016, Chapter 12. doi:10.5334/baw.
13.
D.S.Grether, Extension of a multi-agent transport simulation for traffic signal control and air transport systems, PhD thesis, TU Berlin, Berlin, 2014.
14.
R.Grunitzki, B.C.da Silva and A.L.C.Bazzan, A flexible approach for designing optimal reward functions, in: Proceedings of the 16th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2017), S.Das, E.Durfee, K.Larson and M.Winikoff, eds, IFAAMAS, São Paulo, 2017, pp. 1559–1560, http://ifaamas.org/Proceedings/aamas2017/pdfs/p1559.pdf.
15.
R.Grunitzki, B.C.da Silva and A.L.C.Bazzan, Towards designing optimal reward functions in multi-agent reinforcement learning problems, in: Proc. of the 2018 International Joint Conference on Neural Networks (IJCNN 2018), Rio de Janeiro, 2018.
16.
J.Henry, J.L.Farges and J.Tuffal, The PRODYN real time traffic algorithm, in: Proceedings of the Int. Fed. of Aut. Control, I.F.A.C.Conf and R.Isermann, eds, IFAC, Baden-Baden, 1983, pp. 307–312.
17.
A.Horni, K.Nagel and K.W.Axhausen (eds), The Multi-Agent Transport Simulation MATSim, Ubiquity, London, 2016. doi:10.5334/baw.
18.
P.B.Hunt, D.I.Robertson, R.D.Bretherton and R.I.Winton, SCOOT – a traffic responsive method of coordinating signals, in: TRRL Laboratory Report, 1014, TRRL, Crowthorne, Berkshire, UK, 1981.
19.
G.Konidaris, S.Osentoski and P.Thomas, Value function approximation in reinforcement learning using the Fourier basis, in: Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, AAAI’11, AAAI Press, 2011, pp. 380–385.
20.
N.Kühnel, T.Thunig and K.Nagel, Implementing an adaptive traffic signal control algorithm in an agent-based transport simulation, Procedia Computer Science130 (2018), 894–899. doi:10.1016/j.procs.2018.04.086.
21.
S.Lämmer, Die Selbst-Steuerung im Praxistest, Straßenverkehrstechnik3 (2016), 143–151.
22.
S.Lämmer and D.Helbing, Self-control of traffic lights and vehicle flows in urban road networks, Journal of Statistical Mechanics: Theory and Experiment2008(04) (2008), 04. doi:10.1088/1742-5468/2008/04/P04019.
23.
P.Lowrie, The Sydney coordinate adaptive traffic system – principles, methodology, algorithms, in: Proceedings of the International Conference on Road Traffic Signalling, Sydney, Australia, 1982.
24.
P.Mannion, J.Duggan and E.Howley, An experimental review of reinforcement learning algorithms for adaptive traffic signal control, in: Autonomic Road Transport Support Systems, T.Leo McCluskey, A.Kotsialos, P.J.Müller, F.Klügl, O.Rana and R.Schumann, eds, Springer International Publishing, Cham, 2016, pp. 47–66. doi:10.1007/978-3-319-25808-9_4.
25.
V.Mnih, K.Kavukcuoglu, D.Silver, A.Graves, I.Antonoglou, D.Wierstra and M.Riedmiller, Playing atari with deep reinforcement learning, in: NIPS Deep Learning Workshop, 2013.
26.
K.J.Prabuchandran, A.N.H.Kumar and S.Bhatnagar, Decentralized learning for traffic signal control, in: Proceedings of the 7th International Conference on Communication Systems and Networks (COMSNETS), 2015, pp. 1–6. ISBN 9781479984398.
27.
L.A.Prashanth and S.Bhatnagar, Reinforcement learning with average cost for adaptive control of traffic lights at intersections, in: Procedings of 14th International Conference on Intelligent Transportation Systems (ITSC), IEEE, 2011, pp. 1640–1645. doi:10.1109/ITSC.2011.6082823.
28.
R.S.Sutton and A.G.Barto, Reinforcement Learning: An Introduction, MIT Press, Cambridge, MA, 1998.
29.
R.S.Sutton and A.G.Barto, Reinforcement Learning: An Introduction, 2nd edn, The MIT Press, 2018.
30.
T.Thunig, N.Kühnel and K.Nagel, Adaptive traffic signal control for real-world scenarios in agent-based transport simulations, Transportation Research Procedia37 (2019), 481–488, http://www.sciencedirect.com/science/article/pii/S2352146518306343. doi:10.1016/j.trpro.2018.12.215.
31.
T.Thunig and K.Nagel, Effects of user adaption on traffic-responsive signal control in agent-based transport simulations, in: 6th International Conference on Models and Technologies for Intelligent Transportation Systems (MT-ITS), 2019, pp. 1–7. doi:10.1109/MTITS.2019.8883373.
32.
E.van der Pol, Deep reinforcement learning for coordination in traffic light control, PhD thesis, University of Amsterdam, 2016.
33.
H.van Seijen, A.R.Mahmood, P.Pilarski, M.Machado and R.Sutton, True online temporal-difference learning, Journal of Machine Learning Research17 (2016), 1.
34.
H.Van Seijen and R.S.Sutton, True online TD(λ), in: Proceedings of the 31st International Conference on International Conference on Machine Learning, ICML’14, Vol. 32, JMLR.org, 2014, pp. I692–I-700.
35.
C.J.C.H.Watkins and P.Dayan, Q-learning, Machine Learning8(3) (1992), 279–292.
36.
F.V.Webster, Traffic signal setting, Technical Report, 39, Road Research Laboratory, London, 1958.
37.
H.Wei, G.Zheng, V.V.Gayah and Z.Li, A survey on traffic signal control methods, 2019, CoRR, abs/1904.08117.
38.
K.-L.A.Yau, J.Qadir, H.L.Khoo, M.H.Ling and P.Komisarczuk, A survey on reinforcement learning models and algorithms for traffic signal control, ACM Comput. Surv.50(3) (2017). doi:10.1145/3068287.
39.
R.Zhang, A.Ishikawa, W.Wang, B.Striner and O.K.Tonguz, Partially observable reinforcement learning for intelligent transportation systems, 2018, CoRR, abs/1807.01628.
40.
D.Ziemke, I.Kaddoura and K.Nagel, The MATSim open Berlin scenario: A multimodal agent-based transport simulation scenario based on synthetic demand modeling and open data, Procedia Computer Science151 (2019), 870–877. doi:10.1016/j.procs.2019.04.120.
