基于深度强化学习的自适应交通信号控制研究

doi:10.3969/j.issn.1674-0696.2022.08.04

重庆交通大学学报（自然科学版） ›› 2022, Vol. 41 ›› Issue (08): 24-29.DOI: 10.3969/j.issn.1674-0696.2022.08.04

• 交通+大数据人工智能 • 上一篇下一篇

基于深度强化学习的自适应交通信号控制研究

徐建闽1，周湘鹏1，首艳芳2

(1. 华南理工大学土木与交通学院，广东广州 510640； 2. 华南理工大学广州现代产业技术研究院, 广东广州 510640)

收稿日期:2021-01-14 修回日期:2021-09-03 发布日期:2022-08-19
作者简介:徐建闽（1960—），男，山东招远人，教授，博士，主要从事智能交通控制方面的研究。E-mail:aujmxu@scut.edu.cn
基金资助:
国家自然科学基金面上项目(61873098) ; 广东省自然科学基金项目(2018A030313250);广东省科技计划项目(2016A030305001)

Adaptive Traffic Signal Control Based on Deep Reinforcement Learning

XU Jianmin1, ZHOU Xiangpeng1, SHOU Yanfang2

(1. School of Civil and Transportation Engineering, South China University of Technology, Guangzhou 510640, Guangdong, China; 2. Guangzhou Institute of Modern Industrial Technology, South China University of Technology, Guangzhou 510640, Guangdong, China)

Received:2021-01-14 Revised:2021-09-03 Published:2022-08-19

摘要/Abstract

摘要： 为了提高交通控制算法的适应性和鲁棒性，缓解城市交通拥堵，提出了一种改进的D3QN(dueling double deep Q-network, D3QN)自适应信号控制方法。首先对几种强化学习自适应控制模式进行分析，然后在固定步长动作模式的基础上提出了不定步长动作模式，并构造了一种基于空间占有率的奖励函数；最后使用Sumo软件，对中山市东区街道某交叉口分别在稳定流和随机流场景下进行仿真。仿真结果表明：该方法具有良好的收敛性，有效地降低了延误时间和排队长度。

关键词: 交通工程；交通仿真；自适应控制；交通流；深度强化学习

Abstract: In order to improve the robustness and adaptability of traffic control algorithms and ease urban traffic congestion, an adaptive traffic signal control method based on improved D3QN (dueling double deep Q-network, D3QN) was proposed. Firstly, several adaptive traffic control modes based on reinforcement learning were analyzed. Subsequently, a variable step-size action mode was proposed based on the fixed step-size action mode and a reward function based on space occupancy was constructed. Finally, an intersection in East Street of Zhongshan was simulated by software Sumo in steady flow and stochastic flow. The simulation results show that the proposed method exhibits excellent convergence and effectively reduces the delay time and the queue length.

Key words: traffic engineering; traffic simulation; adaptive control; traffic flow; deep reinforcement learning

中图分类号:

U491.5+1

徐建闽1，周湘鹏1，首艳芳2. 基于深度强化学习的自适应交通信号控制研究[J]. 重庆交通大学学报（自然科学版）, 2022, 41(08): 24-29.

XU Jianmin1, ZHOU Xiangpeng1, SHOU Yanfang2. Adaptive Traffic Signal Control Based on Deep Reinforcement Learning[J]. Journal of Chongqing Jiaotong University(Natural Science), 2022, 41(08): 24-29.

参考文献

［1］郭海锋,程君,方良君,等. 短时预测下的单点交叉口无模型自适应控制方法［J］. 中国公路学报, 2014, 27(12): 88-95.
GUO Haifeng, CHENG Jun, FANG Liangjun, et al. Model-free adaptive control method for isolated intersection based on short-term prediction［J］. China Journal of
Highway and Transport, 2014, 27(12): 88-95.
［2］徐建闽, 李岿林, 翟春杰, 等. 基于短时交通流预测的单交叉口自适应控制［J］. 重庆交通大学学报(自然科学版), 2018, 37(9): 73-78.
XU Jianmin, LI Kuilin, ZHAI Chunjie, et al. Self-adaptive control of is-olated intersection based on Short-Term traffic flow prediction ［J］. Journal of Chongqing Jiaotong
University (Natural Science), 2018, 37(9): 73-78.
［3］ LI Lubing, HUANG Wei, HONG K L. Adaptive coordinated traffic control for stochastic demand ［J］. Transportation Research, Part C. Emerging Technologies, 2018,
88:31-51.
［4］ LI Y, YU L, TAO S, et al. Multi-objective optimization of traffic signal timing for oversaturated intersection ［J］. Mathematical Problems in Engineering, 2013(17):1-9.
［5］卢守峰, 韦钦平, 刘喜敏. 单交叉口信号配时的离线Q学习模型研究［J］. 控制工程, 2012, 19(6): 987-992.
LU Shoufeng, WEI Qinping, LIU Ximin. The study on off-line q-learning model for single intersection signal timing ［J］. Control Engineering of China, 2012, 19(6): 987-
992.
［6］ RASHEED F, YAU K L A, LOW Y C. Deep reinforcement learning for traffic signal control under disturbances: A case study on sunway city, Malaysia ［J］. Future
Generation Computer Systems, 2020,109:431-445.
［7］ TOUHBI S, BABRAM M A, NGUYEN-HUU T, et al. Adaptive traffic signal control: Exploring reward definition for reinforcement learning ［J］. Procedia Computer
Science, 2017, 109: 513-520.
［8］ ROMAN A G, CLEMPNER J B. Traffic-signal control reinforcement learning approach for continuous-time Markov games ［J］. Engineering Applications of Artificial
Intelligence, 2020, 89: 103415.
［9］赖建辉. 基于D3QN的交通信号控制策略［J］. 计算机科学, 2019, 46(2): 117-121.
LAI Jianhui. Traffic signal control based on double deep Q-Learning network with dueling architecture ［J］. Computer Science, 2019, 46 (2): 117-121.
［10］孙浩, 陈春林, 刘琼, 等. 基于深度强化学习的交通信号控制方法［J］. 计算机科学, 2020, 47(2): 169-174.
SUN Hao, CHEN Chunli, LIU Qiong, et al. Traffic gingal control me-thod based on deep reinforcement learning ［J］. Computer Science, 2020, 47(2): 169-174.
［11］王云鹏, 郭戈. 基于深度强化学习的有轨电车信号优先控制［J］.自动化学报, 2019, 45(12): 2366-2377.
WANG Yunpeng, GUO Ge. Signal priority control for trams using deep reinforcement learning ［J］. Acta Automatica Sinica, 2019, 45 (12): 2366-2377.
［12］ KRAJZEWICZ D, ERDMANN J, BEHRISCH M, et al. Recent development and applications of SUMO-Simulation of urban mobility ［J］. International Journal on
Advances in Systems and Measurements, 2012, 5:128-138.
［13］ JIN Junchen, MA Xiaoliang, KOSONEN I. A stochastic optimization framework for road traffic controls based on evolutionary algorithms and traffic simulation［J］.
Advances in Engineering Software, 2017, 114(8): 348-360.

[1]	林丽，蒋帅捷. 基于Ring-Barrie相位的交叉口信号配时优化模型[J]. 重庆交通大学学报（自然科学版）, 2021, 40(01): 12-16.
[2]	蓝章礼，李战，李伟. 基于SURF和最佳缝合线的车道图像序列拼接研究[J]. 重庆交通大学学报（自然科学版）, 2019, 38(10): 13-18.
[3]	刘伟，陈科全，谢忠金. 大型公交站的动态排队论分析与优化[J]. 重庆交通大学学报（自然科学版）, 2018, 37(08): 75-80.
[4]	刘伟，谢忠金，陈科全. 基于NSGAⅡ算法的逆向可变车道信号配时优化[J]. 重庆交通大学学报（自然科学版）, 2018, 37(06): 92-97.

基于深度强化学习的自适应交通信号控制研究

Adaptive Traffic Signal Control Based on Deep Reinforcement Learning

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 4

编辑推荐

Metrics