关键词: 交通运输工程；深度强化学习；交通信号控制；近端策略优化；SUMO

Abstract: Aiming at the problems of low traffic efficiency and insufficient traffic signal control effect of the current urban traffic network, as well as the weak exploration ability of the deep reinforcement learning proximal policy optimization (PPO) algorithm, three kinds of PPO traffic signal control algorithms based on exploration capability enhancement were proposed, namely, PPO based on noise action network (NA-PPO), PPO based on action entropy reward attenuation (ER-PPO), and PPO based on state counting (SC-PPO). Firstly, traffic states, actions, and rewards were defined to maximize signal control effect. Secondly, a multi-agent traffic signal control system was constructed, and an asynchronous parallel multi-process parameter sharing training mechanism was used to accelerate the training speed. Finally, taking the traffic flow of some urban traffic networks in Nanyang City as an example, SUMO was used to carry out simulation experiments. The results show that compared to the PPO algorithm, the online learning control and non-online learning control of the three kinds of exploration enhanced PPO algorithms both reduce the vehicle queue length, driving time and waiting time, and the experiment results verify the effectiveness of the exploration enhancement.

Key words: traffic and transportation engineering; deep reinforcement learning; traffic signal control; proximal policy optimization (PPO); SUMO