基于深度强化学习的插电式柴电混合动力汽车多目标优化控制策略

doi:10.3969/j.issn.1674-0696.2021.01.08

重庆交通大学学报（自然科学版） ›› 2021, Vol. 40 ›› Issue (01): 44-52.DOI: 10.3969/j.issn.1674-0696.2021.01.08

• 交通+大数据人工智能 • 上一篇下一篇

基于深度强化学习的插电式柴电混合动力汽车多目标优化控制策略

隗寒冰,贺少川

(重庆交通大学机电与车辆工程学院，重庆 400074)

收稿日期:2019-07-03 修回日期:2019-10-12 出版日期:2021-01-11 发布日期:2021-01-11
作者简介:隗寒冰（1979—），男，湖北安陆人，教授，博士，主要从事新能源汽车电控技术、智能汽车感知决策控制方面的研究。E-mail:hbwei@cqjtu.edu.cn
基金资助:
国家自然科学基金资助项目(51305472)；重庆市科技局技术创新与应用发展专项项目(CSTL2019075650)

Multi-Objective Optimal Control Strategy for Plug-in Diesel Electric Hybrid Vehicles Based on Deep Reinforcement Learning

WEI Hanbing, HE Shaochuan

(College of Mechatronics & Vehicle Engineering, Chongqing Jiaotong University, Chongqing 400074, China)

Received:2019-07-03 Revised:2019-10-12 Online:2021-01-11 Published:2021-01-11
Supported by:

摘要/Abstract

摘要： 插电式混合动力汽车工作模式切过程中发动机频繁启停引起的发动机排气温度和进气流速波动明显，导致SCR催化器催化效率降低和排放恶化，尤其是低温冷启动阶段更为明显。另一方面，建立精确的SCR催化器瞬态模型较为困难，传统基于模型的混合动力控制策略开发方法效果较差。以某P2构型插电式柴电混合动力汽车为研究对象，建立了包括发动机、电池和SCR后处理系统的整车纵向动力学模型；在此基础上将深度强化学习应用于插电式混合动力汽车的能量管理问题，采用DQN算法对油耗和排放组成的加权目标函数进行求解，得到以需求功率、蓄电池SOC和SCR温度为状态变量、以电机最优功率为输出变量的控制策略；最后将测试结果与DP算法进行对比分析。结果表明，燃油消耗为2.623 L/100 km，SCR催化器出口NOx排放为0.227 5 g/km，与DP控制策略相比，分别下降10.12%和25.69%，证明了提出控制策略的有效性。

关键词: 车辆工程, 深度强化学习, 控制策略, 多目标优化, 插电式混合动力汽车, 动态规划算法

Abstract: During the model switch process of plug-in hybrid electric vehicle, the fluctuation of exhaust temperature and intake flow velocity invoked by frequent engine start-stop is obviously aggravated, which causes the reduction of catalytic efficiency and increase of exhaust emission of SCR catalysts. Especially, the above result is extremely obvious during the cold start stage at low temperature. On the other hand, it is difficult to establish an accurate transient model of SCR catalyst, and the traditional model-based hybrid control strategy development method is not effective. Taking a P2 plug-in diesel electric hybrid vehicle as the research object, the longitudinal dynamic model including engine, battery and SCR aftertreatment system was established. On this basis, deep reinforcement learning was applied to the energy management of plug-in hybrid electric vehicles. DQN algorithm was used to solve the weighted objective function composed of fuel consumption and emission, and the control strategy was obtained with the state variables of required power, battery SOC and SCR temperature, and the optimal motor power as the output variable. The final experimental results were compared with that of dynamic programming (DP) algorithm. The results show that the fuel consumption is 2.623 L / 100 km, and the NOx emission at the outlet of SCR catalyst is 0.227 5 g/km, which is 10.12% and 25.69% lower than that of DP control strategy, which proves the effectiveness of the proposed control strategy.

Key words: vehicle engineering, deep reinforcement learning, control strategy, multi-objective optimization, plug-in hybrid electric vehicle, dynamic programming

中图分类号:

U461.8
TG156

隗寒冰,贺少川. 基于深度强化学习的插电式柴电混合动力汽车多目标优化控制策略[J]. 重庆交通大学学报（自然科学版）, 2021, 40(01): 44-52.

WEI Hanbing, HE Shaochuan. Multi-Objective Optimal Control Strategy for Plug-in Diesel Electric Hybrid Vehicles Based on Deep Reinforcement Learning[J]. Journal of Chongqing Jiaotong University(Natural Science), 2021, 40(01): 44-52.

参考文献

［1］岳广照,刘兴华,仇滔.柴油机选择性催化还原系统氨泄漏及控制研究［J］.兵工学报,2017,38(4):634-642.
YUE Guangzhao, LIU Xinghua, QIU Tao. Research on ammonia slip and control of diesel engine SCR system［J］. Acta Armamentarii,2017,38(4):634-642.
［2］钱立军, 荆红娟, 邱利宏. 基于随机模型预测控制的四驱混合动力汽车能量管理［J］. 中国机械工程, 2018, 29(11):1342-1348.
QIAN Lijun, JING Hongjuan, QIU Lihong. Energy management of a 4WD HEV based on SMPC［J］. China Mechanical Engineering, 2018, 29(11):1342-1348.
［3］ TORRES J L, GONZALEZ R, GIMENEZ A, et al. Energy manage-ment strategy for plug-in hybrid electric vehicles. A comparative study［J］. Applied Energy, 2014,
113:816-824.
［4］ PENG J K, FAN H, HE H W, et al. A rule-based energy management strategy for a plug-in hybrid school bus based on a controller area network bus［J］.
Energies, 2015, 8:5122-5142.
［5］解少博, 陈欢, 刘通，等. 基于DP-ECMS的插电式混合动力城市客车能量管理策略研究［J］. 汽车工程, 2017,39(7):736-741.
XIE Shaobo, CHEN Huan, LIU Tong，et al. A research on energy management strategy for plug-in hybrid electric bus based on DP-ECMS strategy［J］. Automotive
Engineering, 2017,39(7): 736-741.
［6］ ZOU Y, KONG Z H, LIU T, et al. A real-time Markov chain driver model for tracked vehicles and its validation: Its adaptability via stochastic dynamic
programming［J］. IEEE Transactions on Vehicular Technology, 2016, 66 (5):3571-3682.
［7］ GUO L L, GAO B Z, Gao Y, et al. Optimal energy management for HEVs in eco-driving applications using bi-level MPC［J］. IEEE Transactions on Intelligent
Transportation Systems, 2017,18(8): 2153 - 2162.
［8］ LI W M, XU G Q, XU Y S. Online learning control for hybrid electric vehicle［J］. Chinese Journal of Mechanical Engineering, 2012, 25(1):98-106.
［9］ HU Y Q, YANG L, YAN B, et al. An online rolling optimal control strategy for commuter hybrid electric vehicles based on driving condition learning and
prediction［J］. IEEE Transactions on Vehicular Technology, 2016, 65(6):4312-4327.
［10］ LIU T, ZOU Y, LIU D X, et al. Reinforcement learning of adaptive energy management with transition probability for a hybrid electric tracked vehicle［J］.
IEEE Transactions on Industrial Electronics, 2015, 62(12):7837-7846.
［11］ LIU T, ZOU Y, LIU D X, et al. Reinforcement learning-based real-time energy management for a hybrid tracked vehicle［J］. Energies, 2015, 8(7), 7243-7260.
［12］ HU Y, LI W M, XU H, et al. An online learning control strategy for hybrid electric vehicle based on fuzzy Q-Learning［J］. Energies, 2015, 8(10):11167-
11186.
［13］ WU J D, HE H W, PENG J K, et al. Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus［J］.
Applied Energy, 2018, 222:799-811.
［14］ FENG T, LYU L. The characteristics of ammonia storage and the development of model-based control for diesel engine urea-SCR system［J］. Journal of
Industrial and Engineering Chemistry, 2015, 28:97-109.
［15］ SCHAR, M, ONDER C H, GEERING H P. Control of an SCR catalytic converter system for a mobile heavy-duty application［J］. IEEE Transactions on Control
Systems Technology, 2006, 14(4):641-653.
［16］ WILLI R, ROUDUIT B, KOEPPEL R A, et al. Selective reduction of NO by NH3, over vanadia-based commercial catalyst: Parametric sensitivity and kinetic
modelling［J］. Chemical Engineering Science,1996,51(11):2897-2902.
［17］陈征, 刘亚辉, 杨芳. 基于进化-增强学习方法的插电式混合动力公交车能量管理策略［J］. 机械工程学报, 2017,53(16):86-93.
CHEN Zheng, LIU Yahui, YANG Fang. Energy management strategy for plug-in hybrid electric bus with evolutionary-reinforcement learning method［J］. Journal of
Mechanical Engineering, 2017,53(16): 86-93.
［18］隗寒冰,朱宁.基于双状态动态规划算法的PHEV规则控制策略研究［J］.机械传动,2018,42(2),6-13.
WEI Hanbing, ZHU Ning. Study on the rule-based control strategy for PHEV according to two-state dynamic programming algorithm［J］. Journal of Mechanical
Transmission,2018,42(2),6-13.
［19］ HU Y, LI W M, XU K, et al. Energy management strategy for a hybrid electric vehicle based on deep reinforcement learning［J］.Applied Sciences,2018, 8
(2):187-192.
［20］ MNIH V, KAVUKCUOLU K, SILVER D, et al. Playing Atari with Deep Reinforcement Learning［R］. Deep Mind Technologies, 2013.

[1]	辛亮1,2，杜子学1，杨震2，许舟洲2. 单轴转向架跨座式单轨车辆的主动控制研究[J]. 重庆交通大学学报（自然科学版）, 2021, 40(07): 123-127.
[2]	张东东，宗子淳，冯金芝. 基于NSGA-Ⅱ算法的两挡AMT换挡规律多目标优化[J]. 重庆交通大学学报（自然科学版）, 2021, 40(07): 128-135.
[3]	李胜永1，张智华1，王胜男2，王孟2. 面向交通工具金属材料的缺损识别算法[J]. 重庆交通大学学报（自然科学版）, 2021, 40(07): 136-144.
[4]	杜子学，邬浩鑫. 悬挂参数对直线电机跨座式单轨车辆气隙稳定性和运行平稳性影响[J]. 重庆交通大学学报（自然科学版）, 2021, 40(06): 124-129.
[5]	邓明阳1,2，郭应时1. 电动汽车插补耦合无线充电技术的研究[J]. 重庆交通大学学报（自然科学版）, 2021, 40(06): 130-135.
[6]	杜子学，马川翔. 车体铰接式跨座式单轨列车运行稳定性分析[J]. 重庆交通大学学报（自然科学版）, 2021, 40(05): 122-128.
[7]	余曼,赵炜华,吴玲,李郁菡. 基于K-均值聚类和支持向量机的电动汽车行驶工况研究[J]. 重庆交通大学学报（自然科学版）, 2021, 40(05): 129-139.
[8]	曹源文1，陈作1，赵江2，张晓强3，郑婷婷4. 多路况下轮式装载机行驶稳定性研究[J]. 重庆交通大学学报（自然科学版）, 2021, 40(05): 140-146.
[9]	成卫1，刘翔1，雷建明2. 基于RBF神经网络预测的交叉口信号周期时长优化[J]. 重庆交通大学学报（自然科学版）, 2021, 40(04): 13-18.
[10]	杜子学，邬浩鑫，杨震. 直线电机跨座式单轨车辆曲线通过能力分析[J]. 重庆交通大学学报（自然科学版）, 2021, 40(04): 127-132.
[11]	邓涛1，李鑫2. 智能车辆横纵向运动综合控制方法研究[J]. 重庆交通大学学报（自然科学版）, 2021, 40(04): 133-140.
[12]	李耀华，刘洋，宋伟萍，邵攀登，任田园. 基于行驶工况的零部件耐久性测试工况构建[J]. 重庆交通大学学报（自然科学版）, 2021, 40(04): 141-146.
[13]	刘祯1,3，林鑫1,2，吴华伟1,3，叶从进1,3，耿向阳4. 基于STAR-CCM+的IGBT散热翅片结构设计研究[J]. 重庆交通大学学报（自然科学版）, 2021, 40(03): 128-134.
[14]	李军，张俊，张世义. 基于ABP-EKF算法的锂电池SOC估计[J]. 重庆交通大学学报（自然科学版）, 2021, 40(03): 135-140.
[15]	郝刚1,2，金涛1. 基于隐马尔科夫模型的滚动轴承性能衰退评估[J]. 重庆交通大学学报（自然科学版）, 2021, 40(02): 123-128.

基于深度强化学习的插电式柴电混合动力汽车多目标优化控制策略

Multi-Objective Optimal Control Strategy for Plug-in Diesel Electric Hybrid Vehicles Based on Deep Reinforcement Learning

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics