Reinforced Imitative Trajectory Planning for Urban Automated Driving
Reinforced Imitative Trajectory Planning for Urban Automated Driving
Section titled “Reinforced Imitative Trajectory Planning for Urban Automated Driving”学习档位 精读
类型 文献 · 更新 2026-07-20
所属 强化学习
中文学习笔记（自动生成，需核验）
Section titled “中文学习笔记（自动生成，需核验）”Topic: reinforcement-learning · Tier: recent · Year: 2024 · Venue: —
Evidence level: partial · 本地全文: 是 · 建议阅读: ~50 分钟
Paper: https://arxiv.org/abs/2410.15607
Code: —
Generator: grok
为什么值得读
Section titled “为什么值得读”该文针对城市自动驾驶轨迹规划中强化学习收敛差与奖励设计难的问题，提出融合模仿学习的多步规划方法、无线性假设的Transformer贝叶斯奖励函数以及混合驱动框架，并在大规模真实nuPlan数据集上进行闭环验证，代码开源，对结合RL/IL的规划研究有直接参考价值。
RITP通过在actor-critic中集成动作采样器与模仿学习监督，实现多步轨迹规划，配合Transformer贝叶斯奖励与混合优化框架，在nuPlan闭环评估中优于同结构基线并接近SOTA。
城市自动驾驶轨迹规划中，纯强化学习面临收敛困难与奖励函数设计难；现有多数RL方法仅单步推理，缺乏多步规划能力；结合监督的方法仍受限；逆强化学习常假设奖励为手工特征线性组合，难以适配复杂城市场景。
强化学习基础（MDP、actor-critic、TD3、Q值估计与延迟更新）、模仿学习/行为克隆、Transformer与query-centric场景编码（如QCNet）、轨迹规划与闭环控制（LQR跟踪+运动学自行车模型）、nuPlan等自动驾驶数据集与闭环评估概念。
- 提出RITP：将RL与IL结合，使规划器能多步规划；用带轨迹噪声的Transformer规划器探索状态与轨迹空间，用Transformer价值函数评估闭环效果；训练后直接输出轨迹，避免在线生成与评估有限轨迹集的最优性与计算问题（对比TRAVL）。
- 开发基于Transformer的贝叶斯奖励函数：无线性结构假设，建模网络不确定性，为城市场景RL提供有效奖励信号。
- 提出混合驱动轨迹规划框架：用RL规划轨迹引导数值优化细化，以可解释方式评估安全性、合法性与舒适性。
- 在大规模真实城市nuPlan数据集上验证，采用闭环非反应与反应实验，显著优于同策略模型结构基线，并与SOTA达到竞争性性能；代码开源。
将AD建模为MDP；采用actor-critic框架。每步actor给定状态st输出动作a0t（定义为轨迹）；动作采样器同时生成S个动作/轨迹{ait}；仅a0t送入环境（非反应log-replay），得rt+1与st+1；经验(st, {ait}, rt+1, st+1)入回放缓冲；采样更新critic；对每经验选critic估计最高值的轨迹作为最优at，用(st, at)经IL更新actor；延迟策略更新等参考TD3思想。训练后规划器直接输出轨迹。
关键模块和设计取舍
Section titled “关键模块和设计取舍”环境：基于真实数据集非反应log-replay，LQR轨迹跟踪器+运动学自行车模型推进自车（可替换MPC）。状态表示：query-centric范式，为agent与地图多边形建局部时空坐标系，相对时空位置编码，agent特征含位置/航向/速度/语义，地图多边形采样点特征。Actor（MotionFormer）：基于QCNet架构，编码器生成场景编码（地图与agent），解码器用可学习mode query经mode-scene/mode-mode注意力输出K个未来轨迹模式及似然，最高似然作ego动作；也预测其他agent轨迹。动作定义为待跟踪轨迹而非多步控制信号，便于采样与后处理。Critic评估闭环效果。奖励为Transformer贝叶斯网络（细节部分摘录有限）。混合框架用RL轨迹引导数值优化细化以增强安全/可解释性（设计取舍：用IL缓解RL梯度不足与收敛问题，直接输出避免TRAVL在线集合评估的局限，贝叶斯建模不确定性）。
数据集、实验设置与指标
Section titled “数据集、实验设置与指标”大规模真实世界城市自动驾驶nuPlan数据集；训练环境为非反应log-replay模式；评估采用闭环指标，包括闭环非反应与闭环反应实验。具体指标名称与划分细节待来源核验。
主要结果（需原文证据）
Section titled “主要结果（需原文证据）”闭环评估表明，所提方法显著优于采用相同策略模型结构的基线，并与SOTA方法达到竞争性性能。具体数值指标待来源核验。
局限、失败场景与适用边界
Section titled “局限、失败场景与适用边界”训练主要在非反应log-replay下进行；奖励与混合框架细节在提供摘录中不完整；对复杂城市场景的通用性与计算开销边界、失败案例等待来源核验。
与前序 / 同期 / 后续方法的关系
Section titled “与前序 / 同期 / 后续方法的关系”对比行为克隆（BC，如NVIDIA端到端、多模态、UniAD）及其协变量偏移问题与DAgger/SafeDAgger（需专家策略）；高速公路RL（车道保持/变道/合流）与城市场景挑战；结合RL+监督的方法（如KL正则、Roach特权BEV再IL、GIGAFLOW自对弈）；多步方法如TrajHF（扩散+人类反馈，推理昂贵）与TRAVL（轨迹作动作，有限规则集+在线评估，公路场景）；IRL多为线性特征组合假设。RITP定位为解决单步限制、收敛与非线性奖励问题，并引入混合框架。
官方代码与复现建议
Section titled “官方代码与复现建议”推荐阅读顺序
Section titled “推荐阅读顺序”先读Abstract与Introduction（问题与四点贡献）→ Section 2.1 Preliminaries（MDP与TD3）→ 2.2 Framework与Fig.1（整体pipeline）→ 2.3 Environment、2.4 State Representation、2.5 Actor Network（MotionFormer）→ 奖励与混合框架相关部分（摘录有限）→ 实验验证部分（闭环结果）→ 代码仓库对照。
- Q: RITP如何解决传统RL在城市轨迹规划中的收敛与多步规划问题？ A: 将RL与IL结合：用动作采样器生成多轨迹，critic评估选最优作为IL监督信号更新actor（Transformer规划器）；动作定义为完整轨迹，训练后直接输出多步计划，避免仅单步推理。
- Q: 动作在RITP中如何定义？为何这样定义？ A: 定义为未来轨迹{位置序列}而非多步控制信号，主要便于动作采样与后处理，并由下游LQR等跟踪器执行。
- Q: 状态表示采用什么范式？核心思想是什么？ A: Query-centric范式：为每个agent历史时刻与地图多边形建立局部时空坐标系，并用相对时空位置编码融入相对信息。
- Q: 贝叶斯奖励函数相对传统IRL有何不同？ A: 采用Transformer-based贝叶斯神经网络，无线性特征组合假设，并建模网络不确定性，以适配城市场景。
- Q: 混合驱动框架的目的是什么？ A: 结合数据驱动（RL规划）与模型驱动（数值优化细化），以可解释方式评估并提升轨迹的安全性、合法性与舒适性。
- Abstract: this paper proposes a novel RL-based trajectory planning method that integrates RL with imitation learning to enable multi-step planning. Furthermore, a transformer-based Bayesian reward function is developed… Moreover, a hybrid-driven trajectory planning framework is proposed… validated on the large-scale real-world urban automated driving nuPlan dataset. Evaluated using closed-loop metrics, the results demonstrated that the proposed method significantly outperformed the baseline employing the identical policy model structure and achieved competitive performance compared to the state-of-the-art method. The code is available at https://github.com/Zigned/nuplan_zigned.
- Section 1 Contributions: 1) A novel RL-based trajectory planning method, RITP, is proposed. While most existing RL-based methods can only reason one step ahead, RITP integrates RL with IL, enabling planning for multiple future steps… 2) A transformer-based Bayesian reward function is developed… without the linear structure assumption… 3) A hybrid-driven trajectory planning framework is proposed… 4) The proposed methods were validated on the large-scale real-world urban automated driving nuPlan dataset…
- Section 2.2 Framework: RITP incorporates an action sampler in addition to the actor to generate potential future trajectories, estimates their critic values using the critic network, and selects the optimal trajectory based on these values. The optimal trajectory is then used as a supervisory signal of IL to update the actor network… the actions are defined as trajectories…
- Section 2.3 Environment: During training, the environment is in non-reactive log-replay mode… the environment propagates the motion of the ego vehicle using an LQR-based trajectory tracker and the kinematic bicycle model. Note that an action is defined as a trajectory to be tracked by a downstream controller…
- Section 2.5 Actor Network: a transformer-based actor network, referred to as MotionFormer, is constructed using QCNet architecture… encodes the traffic scene context… decodes K future trajectories… During training, the trajectory with the highest likelihood is considered as the ego agent’s action.
方法结构（重绘）
Section titled “方法结构（重绘）”flowchart LR A["输入 / 观测"] --> B["表示 / 编码"] B --> C["推理 / 解码"] C --> D["输出 / 动作或检测"] %% method sketch for: Reinforced Imitative Trajectory Planning for Urban Automated Driving方法结构示意（重绘；细节以原论文为准，待 PDF 核验）。
论文摘录图/表
Section titled “论文摘录图/表”
来源：原论文约 p.1（arch）；学习用途摘录。
来源：原论文约 p.13（table）；学习用途摘录。
精读判断（人工）
Section titled “精读判断（人工）”这篇文献回答什么问题
Section titled “这篇文献回答什么问题”如何在城市场景用强化模仿轨迹规划结合 IL 与 RL，缓解纯行为克隆的误差累积。
决策分轨必读对照：IL 提供示范，RL 提供闭环纠正。读时应抓 reward 设计、安全约束与仿真依赖——这决定方法能否离开 CARLA 类环境。
- reward hacking；
- 仿真到现实 gap；
- IL 正则过强导致无探索。
读完应能回答
Section titled “读完应能回答”- 相对纯 BC 改了什么目标？
- 轨迹级 action 的状态是什么？
- 如何评估闭环安全性？
英文自动分析（可折叠）
Section titled “英文自动分析（可折叠）”展开英文 Paper Card / AI deep analysis
Paper Card
Section titled “Paper Card”| Field | Content |
|---|---|
| Year | 2024 |
| Authors | Di Zeng, Ling Zheng, Xiantong Yang, Yinong Li |
| arXiv | 2410.15607 |
| DOI | 10.2139/ssrn.5021803 |
| Topics | reinforcement-learning |
| Paper | https://arxiv.org/abs/2410.15607 |
原文摘录与素材（可折叠）
Section titled “原文摘录与素材（可折叠）”展开 Extract / Selections / Local assets
Selections
Section titled “Selections”reinforcement-learning: tier=recent rank=2 score=58 — auto refresh 2026-07-19 sources=arxiv,crossref
Extract excerpt
Section titled “Extract excerpt”Reinforced Imitative Trajectory Planning for Urban Automated Driving Di Zeng1 , Ling Zheng1,2*, Xiantong Yang1 , Yinong Li1,2 1 College of Mechanical and Vehicle Engineering, Chongqing University, Shazheng Street,
Chongqing, 40044, Chongqing, China. 2* State Key Laboratory of Mechanical Transmission for Advanced Equipment, Chongqing
University, Shazheng Street, Chongqing, 40044, Chongqing, China.
arXiv:2410.15607v2 [cs.RO] 16 Jul 2025 *Corresponding author(s). E-mail(s): zling@cqu.edu.cn; Contributing authors: zigned@cqu.edu.cn; xiantongyang@cqu.edu.cn; ynli@cqu.edu.cn;
Abstract Reinforcement learning (RL) faces challenges in trajectory planning for urban automated driving due to the poor convergence of RL and the difficulty in designing reward functions. Consequently, few RL-based trajectory planning methods can achieve performance comparable to that of imitation learning-based methods. The convergence problem is alleviated by combining RL with supervised learning. However, most existing approaches only reason one step ahead and lack the capability to plan for multiple future steps. Besides, although inverse reinforcement learning holds promise for solving the reward function design issue, existing methods for automated driving impose a linear structure assumption on reward functions, making them difficult to apply to urban automated driving. In light of these challenges, this paper proposes a novel RL-based trajectory planning method that integrates