Abstract: To address the issue of policy update lag in urban traffic signal control by the deep Q-network (DQN) algorithm, an improved double-layered exploration and multi-step reward algorithm, that is MS_DQN (multi-step deep Q-network) algorithm, was proposed. The proposed algorithm constructed a double-layered exploration mechanism by combining(ε-greedy strategy) and Boltzmann strategy, enabling the agent to explore unknown spaces more comprehensively as well as balance exploration and exploitation. Meanwhile, rewards from multiple future time steps were introduced. Through a multi-step reward mechanism, sample correlation was reduced, higher long-term rewards were obtained and the learning stability was enhanced. Experimental results on the simulation of urban mobility (SUMO) platform demonstrate that the MS_DQN algorithm can significantly improves intersection throughput efficiency, providing an effective solution for urban intelligent traffic control.

Key words: traffic engineering; traffic signal control; MS_DQN algorithm; double-layered exploration mechanism; multi-step reward mechanism

摘要： 针对DQN(deep Q-Network)算法在城市交通信号控制中策略更新滞后的问题，提出一种改进的双层探索及多步回报算法MS_DQN(multi-step deep Q-Network)。该算法通过融合ε-greedy(ε-贪心策略)和玻尔兹曼(Boltzmann)策略构建双层探索机制，帮助智能体更全面地探索未知空间以及平衡探索与利用；同时引入未来多个时间步的奖励，通过多步回报机制，减少样本之间的相关性，获得更高的长期奖励，提高学习的稳定性。在SUMO(simulation of urban mobility)仿真平台的实验结果表明：MS_DQN算法能显著提升交叉口通行效率，为城市智能交通控制提供有效的解决方案。

关键词: 交通工程；交通信号控制；MS_DQN算法；双层探索机制；多步回报机制