Adaptive Traffic Signal Control Based on Deep Reinforcement Learning

doi:10.3969/j.issn.1674-0696.2022.08.04

Journal of Chongqing Jiaotong University(Natural Science) ›› 2022, Vol. 41 ›› Issue (08): 24-29.DOI: 10.3969/j.issn.1674-0696.2022.08.04

• Transportation+Big Data & Artificial Intelligence • Previous Articles Next Articles

Adaptive Traffic Signal Control Based on Deep Reinforcement Learning

XU Jianmin1, ZHOU Xiangpeng1, SHOU Yanfang2

(1. School of Civil and Transportation Engineering, South China University of Technology, Guangzhou 510640, Guangdong, China; 2. Guangzhou Institute of Modern Industrial Technology, South China University of Technology, Guangzhou 510640, Guangdong, China)

Received:2021-01-14 Revised:2021-09-03 Published:2022-08-19

基于深度强化学习的自适应交通信号控制研究

徐建闽1，周湘鹏1，首艳芳2

(1. 华南理工大学土木与交通学院，广东广州 510640； 2. 华南理工大学广州现代产业技术研究院, 广东广州 510640)

作者简介:徐建闽（1960—），男，山东招远人，教授，博士，主要从事智能交通控制方面的研究。E-mail:aujmxu@scut.edu.cn
基金资助:
国家自然科学基金面上项目(61873098) ; 广东省自然科学基金项目(2018A030313250);广东省科技计划项目(2016A030305001)

Abstract

Abstract: In order to improve the robustness and adaptability of traffic control algorithms and ease urban traffic congestion, an adaptive traffic signal control method based on improved D3QN (dueling double deep Q-network, D3QN) was proposed. Firstly, several adaptive traffic control modes based on reinforcement learning were analyzed. Subsequently, a variable step-size action mode was proposed based on the fixed step-size action mode and a reward function based on space occupancy was constructed. Finally, an intersection in East Street of Zhongshan was simulated by software Sumo in steady flow and stochastic flow. The simulation results show that the proposed method exhibits excellent convergence and effectively reduces the delay time and the queue length.

Key words: traffic engineering; traffic simulation; adaptive control; traffic flow; deep reinforcement learning

摘要： 为了提高交通控制算法的适应性和鲁棒性，缓解城市交通拥堵，提出了一种改进的D3QN(dueling double deep Q-network, D3QN)自适应信号控制方法。首先对几种强化学习自适应控制模式进行分析，然后在固定步长动作模式的基础上提出了不定步长动作模式，并构造了一种基于空间占有率的奖励函数；最后使用Sumo软件，对中山市东区街道某交叉口分别在稳定流和随机流场景下进行仿真。仿真结果表明：该方法具有良好的收敛性，有效地降低了延误时间和排队长度。

关键词: 交通工程；交通仿真；自适应控制；交通流；深度强化学习

CLC Number:

U491.5+1

XU Jianmin1, ZHOU Xiangpeng1, SHOU Yanfang2. Adaptive Traffic Signal Control Based on Deep Reinforcement Learning[J]. Journal of Chongqing Jiaotong University(Natural Science), 2022, 41(08): 24-29.

徐建闽1，周湘鹏1，首艳芳2. 基于深度强化学习的自适应交通信号控制研究[J]. 重庆交通大学学报（自然科学版）, 2022, 41(08): 24-29.

References

［1］郭海锋,程君,方良君,等. 短时预测下的单点交叉口无模型自适应控制方法［J］. 中国公路学报, 2014, 27(12): 88-95.
GUO Haifeng, CHENG Jun, FANG Liangjun, et al. Model-free adaptive control method for isolated intersection based on short-term prediction［J］. China Journal of
Highway and Transport, 2014, 27(12): 88-95.
［2］徐建闽, 李岿林, 翟春杰, 等. 基于短时交通流预测的单交叉口自适应控制［J］. 重庆交通大学学报(自然科学版), 2018, 37(9): 73-78.
XU Jianmin, LI Kuilin, ZHAI Chunjie, et al. Self-adaptive control of is-olated intersection based on Short-Term traffic flow prediction ［J］. Journal of Chongqing Jiaotong
University (Natural Science), 2018, 37(9): 73-78.
［3］ LI Lubing, HUANG Wei, HONG K L. Adaptive coordinated traffic control for stochastic demand ［J］. Transportation Research, Part C. Emerging Technologies, 2018,
88:31-51.
［4］ LI Y, YU L, TAO S, et al. Multi-objective optimization of traffic signal timing for oversaturated intersection ［J］. Mathematical Problems in Engineering, 2013(17):1-9.
［5］卢守峰, 韦钦平, 刘喜敏. 单交叉口信号配时的离线Q学习模型研究［J］. 控制工程, 2012, 19(6): 987-992.
LU Shoufeng, WEI Qinping, LIU Ximin. The study on off-line q-learning model for single intersection signal timing ［J］. Control Engineering of China, 2012, 19(6): 987-
992.
［6］ RASHEED F, YAU K L A, LOW Y C. Deep reinforcement learning for traffic signal control under disturbances: A case study on sunway city, Malaysia ［J］. Future
Generation Computer Systems, 2020,109:431-445.
［7］ TOUHBI S, BABRAM M A, NGUYEN-HUU T, et al. Adaptive traffic signal control: Exploring reward definition for reinforcement learning ［J］. Procedia Computer
Science, 2017, 109: 513-520.
［8］ ROMAN A G, CLEMPNER J B. Traffic-signal control reinforcement learning approach for continuous-time Markov games ［J］. Engineering Applications of Artificial
Intelligence, 2020, 89: 103415.
［9］赖建辉. 基于D3QN的交通信号控制策略［J］. 计算机科学, 2019, 46(2): 117-121.
LAI Jianhui. Traffic signal control based on double deep Q-Learning network with dueling architecture ［J］. Computer Science, 2019, 46 (2): 117-121.
［10］孙浩, 陈春林, 刘琼, 等. 基于深度强化学习的交通信号控制方法［J］. 计算机科学, 2020, 47(2): 169-174.
SUN Hao, CHEN Chunli, LIU Qiong, et al. Traffic gingal control me-thod based on deep reinforcement learning ［J］. Computer Science, 2020, 47(2): 169-174.
［11］王云鹏, 郭戈. 基于深度强化学习的有轨电车信号优先控制［J］.自动化学报, 2019, 45(12): 2366-2377.
WANG Yunpeng, GUO Ge. Signal priority control for trams using deep reinforcement learning ［J］. Acta Automatica Sinica, 2019, 45 (12): 2366-2377.
［12］ KRAJZEWICZ D, ERDMANN J, BEHRISCH M, et al. Recent development and applications of SUMO-Simulation of urban mobility ［J］. International Journal on
Advances in Systems and Measurements, 2012, 5:128-138.
［13］ JIN Junchen, MA Xiaoliang, KOSONEN I. A stochastic optimization framework for road traffic controls based on evolutionary algorithms and traffic simulation［J］.
Advances in Engineering Software, 2017, 114(8): 348-360.

[1]	LIN Li, JIANG Shuaijie. Optimization Model of Intersection Signal Timing Based on Ring-Barrier Phase [J]. Journal of Chongqing Jiaotong University(Natural Science), 2021, 40(01): 12-16.
[2]	LAN Zhangli, LI Zhan, LI Wei. Lane Image Sequence Stitching Based on SURF and Optimal Seam [J]. Journal of Chongqing Jiaotong University(Natural Science), 2019, 38(10): 13-18.
[3]	LIU Wei, CHEN Kequan, XIE Zhongjin. Dynamic Queuing Theory and Its Optimization for Large Bus Station [J]. Journal of Chongqing Jiaotong University(Natural Science), 2018, 37(08): 75-80.
[4]	LIU Wei，XIE Zhongjin，CHEN Kequan. Optimization of Reversing Variable Lane Signal Timing Design Based on NSGAⅡ Algorithm [J]. Journal of Chongqing Jiaotong University(Natural Science), 2018, 37(06): 92-97.

Adaptive Traffic Signal Control Based on Deep Reinforcement Learning

基于深度强化学习的自适应交通信号控制研究

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 4

Recommended Articles

Metrics