DiffusionDriveV2: Reinforcement Learning-Constrained Truncated Diffusion Modeling in End-to-End Autonomous Driving

DiffusionDriveV2: Reinforcement Learning-Constrained Truncated Diffusion Modeling in End-to-End Autonomous Driving

⚠️ AI 生成 · 建议对照原文 本页为自动整理的学习笔记；关键数据与引用如需引用，请回查 PDF / 官方版本。

学习档位 中文笔记

类型 文献 · 更新 2026-07-19

标签 autonomous-driving · planning · end-to-end-learning · diffusion-models · reinforcement-learning · navsim

所属 端到端驾驶 · 扩散模型

Migration status: imported from ad_projs@a823662; source anchors and claims remain needs-source-verification.

• Paper: https://arxiv.org/abs/2512.07745
• Code: https://github.com/hustvl/DiffusionDriveV2 at 1cd12a1e155c34dcc471261835444c8d5587580b

中文学习笔记（自动生成，需核验）

Topic: ad-end-to-end-driving · Tier: watch · Year: 2025 · Venue: —
Evidence level: partial · 本地全文: 是 · 建议阅读: ~40 分钟
Paper: https://arxiv.org/abs/2512.07745
Code: —
Generator: grok

为什么值得读

解决端到端自动驾驶中截断扩散轨迹规划模型在模仿学习下多样性与一致高质量的根本困境：通过强化学习（Intra/Inter-Anchor GRPO + 尺度自适应乘性噪声）约束负模式、提升探索，同时保留GMM多模态性，并在NAVSIM闭环上刷新SOTA，是首个直接针对该困境的迁移方案。

一句话结论

DiffusionDriveV2用强化学习约束截断扩散的GMM多模态轨迹生成，兼顾多样性与高质量，在NAVSIM v1/v2上以ResNet-34达到91.2 PDMS / 85.5 EPDMS新纪录。

问题定义

Vanilla扩散轨迹规划易mode collapse；DiffusionDrive用预定义锚点GMM划分意图空间获得多样性，但模仿学习仅监督最接近专家的单正模式，负模式无约束，导致大量低质量/碰撞轨迹，依赖脆弱下游选择器，系统整体质量无保证。

前置知识

端到端自动驾驶（E2E-AD）、扩散策略/截断扩散、Gaussian Mixture Model（GMM）锚点、模仿学习 vs 强化学习、GRPO（Group Relative Policy Optimization）、轨迹作为waypoint序列、NAVSIM闭环评测基础。

核心贡献

• 提出DiffusionDriveV2，首次直接用RL解决DiffusionDrive因IL不完整多模态监督导致的多样性-高质量困境。
• 设计Intra-Anchor GRPO（锚内组优势估计，避免不同意图间不当比较）与Inter-Anchor Truncated GRPO（全局视角稳定训练），成功将GRPO迁移到截断扩散GMM框架。
• 采用尺度自适应乘性噪声（纵向/横向）替代加性噪声作为探索噪声，更好保持探索轨迹平滑与连贯。
• 在NAVSIM v1/v2闭环评测上以对齐ResNet-34 backbone刷新SOTA，并验证多样性与一致高质量的最佳权衡。

方法流程

感知编码器提取场景特征 → 以预定义锚点GMM的锚定高斯噪声轨迹为输入，经截断扩散解码器（少步去噪）生成多模态轨迹 → 加尺度自适应乘性高斯噪声产生探索轨迹 → Anchored Truncated GRPO（Intra-Anchor组内优势 + Inter-Anchor截断全局视角）进行轨迹级RL优化，约束低质量模式并探索更优策略 → 模式选择器从精炼后的多模态轨迹中选出最目标对齐的一条。冷启动使用DiffusionDrive预训练IL权重。

关键模块和设计取舍

1. Truncated Diffusion Generator：直接复用DiffusionDrive锚点截断去噪（Ttrunc≪T，预测相对锚点偏移），保留GMM多模态；2) 尺度自适应乘性噪声：τ′=(1+ϵmul)τ，ϵmul=(ϵlong,ϵlat)，避免加性噪声破坏轨迹平滑导致锯齿探索；3) Intra-Anchor GRPO：仅在同一锚点生成样本内做组相对优势估计，防止左转vs直行等意图间错误比较引发mode collapse；4) Inter-Anchor Truncated GRPO：跨锚点提供全局视角并稳定训练；5) 将去噪过程建模为MDP（条件高斯策略），用RL对所有模式施加目标对齐约束。取舍：牺牲纯IL的简单似然最大化，换取对负模式的显式惩罚与超越专家探索。

数据集、实验设置与指标

NAVSIM v1与NAVSIM v2数据集，闭环评测；主指标PDMS（v1）与EPDMS（v2）；backbone对齐ResNet-34；与其他扩散生成模型对比多样性-质量权衡。

主要结果（需原文证据）

DiffusionDriveV2在NAVSIM v1上达到91.2 PDMS、在NAVSIM v2上达到85.5 EPDMS（ResNet-34），刷新纪录；显著提升整体输出质量同时保留GMM多模态生成能力，达到多样性与一致高质量的最佳权衡。

局限、失败场景与适用边界

提取中强调解决了IL下负模式无监督问题，但仍依赖最终模式选择器；对OOD场景的鲁棒性、完整消融与失败案例细节待来源核验；当前聚焦轨迹规划生成器，与完整感知-规划栈集成边界待来源核验。

与前序 / 同期 / 后续方法的关系

前序：UniAD/VAD等模块化E2E、VADv2/Hydra-MDP选择式多模态、Diffusion Policy及DiffusionDrive（锚点截断扩散解决mode collapse但受IL限制）；同期/相关：将GRPO引入E2E-AD的AlphaDrive（规划推理）、以及直接用于vanilla扩散的若干工作（仍受mode collapse困扰）；本工作首次成功迁移到锚定截断扩散GMM并解决意图间比较问题；后续可能扩展至更长时域或真实车部署（代码将开源）。

官方代码与复现建议

官方代码与模型将发布于https://github.com/hustvl/DiffusionDriveV2；建议从DiffusionDrive预训练权重冷启动，复现时注意Intra/Inter-Anchor GRPO分组与乘性噪声实现，闭环在NAVSIM v1/v2验证PDMS/EPDMS。

推荐阅读顺序

摘要+Fig.1（问题与对比）→引言（困境与动机）→第3节Preliminary（截断扩散GMM与IL局限）→第4节Method（噪声+Anchored Truncated GRPO+架构Fig.2）→Related Work→实验/消融（提取仅至方法开头，完整结果待全文）→结论与代码。

自测问题

1. Q: DiffusionDrive为何会产生大量低质量轨迹？ A: 因为IL训练目标实际只优化最接近专家轨迹的单一正模式，负模式（绝大多数样本）无显式约束，导致高质与碰撞轨迹混杂。
2. Q: 为什么不能直接对所有锚点轨迹做vanilla GRPO？ A: 不同锚点代表不同驾驶意图（如左转vs直行），跨意图比较优势会加剧mode collapse；因此需Intra-Anchor仅组内估计。
3. Q: 尺度自适应乘性噪声相比加性噪声的优势是什么？ A: 加性噪声在轨迹近远端尺度不一致下易产生锯齿破碎路径；乘性噪声τ′=(1+ϵmul)τ（纵/横两个噪声）保持结构平滑与连贯。
4. Q: DiffusionDriveV2在NAVSIM上的主要数字结果是什么？ A: ResNet-34 backbone下，v1 PDMS 91.2、v2 EPDMS 85.5，刷新纪录。
5. Q: 方法如何同时提高下界与上界？ A: RL对所有模式施加目标对齐约束（惩罚不安全负模式，提高下界），同时用探索噪声推动更广动作空间寻找可能超越专家的策略（提高上界）。

原文证据

• Abstract / page 1: DiffusionDriveV2 achieves 91.2 PDMS on the NAVSIM v1 dataset and 85.5 EPDMS on the NAVSIM v2 dataset in closed-loop evaluation with an aligned ResNet-34 backbone, setting a new record.
• Abstract / page 1: we propose DiffusionDriveV2, which leverages reinforcement learning to both constrain low-quality modes and explore for superior trajectories. This significantly enhances the overall output quality while preserving the inherent multimodality of its core Gaussian Mixture Model.
• Introduction / page 2: we use scale-adaptive multiplicative noise, ideal for trajectory planning, to promote broad exploration. Second, we employ intra-anchor GRPO to manage advantage estimation among samples generated from a single anchor, and inter-anchor truncated GRPO to incorporate a global perspective across different anchors, preventing improper advantage comparisons between distinct intentions
• Introduction / page 2-3: To the best of our knowledge, DiffusionDriveV2 is the first work to directly confront this dilemma and propose a solution. … DiffusionDriveV2 is the first work to successfully migrate GRPO to a truncated diffusion model.
• Method 4.2 / page 5: To preserve trajectory coherence, we propose a method that adds only two multiplicative Gaussian noises, one longitudinal and one lateral. It can be expressed as τ′ = (1 + ϵmul)τ, where ϵmul = (ϵlong, ϵlat).
• Fig.1 caption / page 2: (c) DiffusionDriveV2 leverages reinforcement learning to apply constraints to multi-modal trajectories, guiding the model to generate both diverse and consistent high-quality trajectories.

Evidence-backed Reading Notes

Topic	Evidence-backed note	Source	Short original cue
Problem	Imitation-trained diffusion planners can generate diverse modes, but not all modes are high-quality driving behavior.	[PDF p.1, Abstract]	Introduction
Representation	The note treats the method as RL-constrained truncated diffusion because the paper’s method pages introduce the relevant representation/module vocabulary.	[PDF p.3, Related Work]	Method
Core mechanism	Scale-adaptive multiplicative noise, intra-anchor GRPO, and inter-anchor truncated GRPO shape the diffusion policy.	[PDF p.3, Related Work]	scale-adaptive
Input / Output	Input: NAVSIM scene features and diffusion anchors. Output: RL-constrained trajectory samples.	[PDF p.9, Method]	NAVSIM
Training / Evaluation	The paper reports NAVSIM v1/v2 metrics; benchmark and reward versions must be recorded.	[PDF p.12, Method]	PDMS
Relationship	DiffusionDrive is the base generator; RAD is broader closed-loop RL; SparseDriveV2/GTRS are scoring alternatives.	[PDF p.3, Related Work]	Related Work
Failure/Risk	Reward hacking, mode collapse, GRPO implementation, NAVSIM version, and sampling stability are high-risk points.	[PDF p.3, Related Work]	scale-adaptive
Reproduction boundary	Start from checkpoint inference; RL training requires careful reward/metric cache versioning.	[PDF p.9, Method]	NAVSIM

Code Cross-References

Local path	Why it matters
1cd12a1e155c34dcc471261835444c8d5587580b:README.md	Code/repo anchor for implementation cross-check.
1cd12a1e155c34dcc471261835444c8d5587580b:navsim/agents/diffusiondrivev2/diffusiondrivev2_rl_config.py	Code/repo anchor for implementation cross-check.
1cd12a1e155c34dcc471261835444c8d5587580b:navsim/agents/diffusiondrivev2/diffusiondrivev2_sel_config.py	Code/repo anchor for implementation cross-check.
1cd12a1e155c34dcc471261835444c8d5587580b:navsim/agents/diffusiondrivev2/transfuser_config.py	Code/repo anchor for implementation cross-check.
1cd12a1e155c34dcc471261835444c8d5587580b:navsim/planning/script/config/common/agent/diffusiondrivev2_rl_agent.yaml	Code/repo anchor for implementation cross-check.
1cd12a1e155c34dcc471261835444c8d5587580b:navsim/planning/script/config/common/agent/diffusiondrivev2_sel_agent.yaml	Code/repo anchor for implementation cross-check.

Reproduction Notes

• Treat this note as paper/code reading material first; do not interpret mini-data smoke tests as paper reproduction. [PDF p.9, Method]
• Before running experiments, verify the local code entry points above against the paper method terminology and dataset protocol. [PDF p.3, Related Work]
• If a claim is not linked to a PDF page or code path in this note, treat it as an implementation hypothesis rather than established paper fact.

Local Asset Identity

Discovery evidence

• topic: reinforcement-learning
• sources: semantic-scholar-http
• retrieved_at: 2026-07-20
• query: reinforcement learning end-to-end driving
• corpus_id: 283693538
• arxiv: 2512.07745
• doi: 10.48550/arXiv.2512.07745
• score_total: 77
• suggested_tier: recent

Relevance

(no prose relevance explanation — numeric score only or HTTP source)

Snippets

(no snippet evidence in candidate pool)

延伸解读与背景补充

生成：2026-07-21 · 来源条数 1 · 模型 heuristic · 需人工核验数字

问题再表述

围绕「DiffusionDriveV2: Reinforcement Learning-Constrained Truncated Diffusion Modeling in End-to-End Autonomous Driving」的核心问题与动机（待结合全文核验）。

方法要点补强

• 见原文方法章节；以下为基于摘要/摘录的要点提示。
• DiffusionDriveV2: Reinforcement Learning-Constrained Truncated Diffusion Modeling in End-to-End Auton…

与相近工作的关系

与相近工作的关系待核验；请对照 related work。

常见误读

• 勿仅凭摘要推断未给出的数值指标。

自测问题

1. 这篇工作的输入/输出表示是什么？（DiffusionDriveV2: Reinforcement Learning-Constrained Truncated Diffusion Modeling in End-to-End Autonomous Driving）
2. 训练目标与评测协议各是什么？
3. 主要失败模式或局限是什么？

外部解读索引

• [2512.07745] DiffusionDriveV2: Reinforcement Learning-Constrained Truncated Diffusion Modeling in End-to-End Autonomous Driving — 2026-07-21 — 官方摘要/二次页面（自动抓取）

关键图表

方法结构（重绘）

flowchart LR
  A["输入 / 观测"] --> B["表示 / 编码"]
  B --> C["推理 / 解码"]
  C --> D["输出 / 动作或检测"]
  %% method sketch for: DiffusionDriveV2: Reinforcement Learning-Constrained Truncated Diffusion Modelin

方法结构示意（重绘；细节以原论文为准，待 PDF 核验）。

论文摘录图/表

来源：原论文约 p.2（arch）；学习用途摘录。

来源：原论文约 p.8（table）；学习用途摘录。

英文自动分析（可折叠）

展开英文 Paper Card / AI deep analysis

Paper Card

Field	Content
One-line takeaway	DiffusionDriveV2 adds RL constraints to truncated diffusion to improve mode quality while preserving multimodal diversity.
Problem	Imitation-trained diffusion planners can generate diverse modes, but not all modes are high-quality driving behavior.
Representation	RL-constrained truncated diffusion
Input / Output	Input: NAVSIM scene features and diffusion anchors. Output: RL-constrained trajectory samples.
Core Mechanism	Scale-adaptive multiplicative noise, intra-anchor GRPO, and inter-anchor truncated GRPO shape the diffusion policy.
Training / Evaluation	The paper reports NAVSIM v1/v2 metrics; benchmark and reward versions must be recorded.
Reproduction Status	Start from checkpoint inference; RL training requires careful reward/metric cache versioning.
Compare With	DiffusionDrive is the base generator; RAD is broader closed-loop RL; SparseDriveV2/GTRS are scoring alternatives.
Failure/Risk	Reward hacking, mode collapse, GRPO implementation, NAVSIM version, and sampling stability are high-risk points.

原文摘录与素材（可折叠）

展开 Extract / Selections / Local assets

Source Anchors

Anchor	What to verify	Source	Short original cue
Title and abstract	Use to verify paper identity and top-level contribution.	[PDF p.1, Abstract]	DiffusionDriveV2: Reinforcement Learning-Constrained Truncated Diffusion Modeling in End-to-End Autonomous Driving
Motivation	Use to verify the problem statement and why the work is needed.	[PDF p.1, Abstract]	Introduction
Core method	Use to verify the main modeling mechanism and module names.	[PDF p.3, Related Work]	Method
Key module terms	Use to verify exact component names before editing the note.	[PDF p.3, Related Work]	scale-adaptive
Dataset and protocol	Use to verify data dependencies: NAVSIM v1/v2.	[PDF p.9, Method]	NAVSIM
Metrics and results	Use to verify metric names and reported benchmark context.	[PDF p.12, Method]	PDMS
Experiments or ablation	Use to verify which claims are experimentally supported.	[PDF p.3, Related Work]	Experiments
Position in related work	Use to verify the claimed relationship to neighboring methods.	[PDF p.3, Related Work]	Related Work
Conclusion or limits	Use to verify final claims and remaining constraints.	[PDF p.8, Method]	Conclusion

反向链接

• 扩散模型
• 端到端驾驶
• 自动驾驶方法谱系
• 自动驾驶方法谱系

---

学习导航

标记为已读

• 所属 Topic: ad-end-to-end-driving
• 学习路径: 学习路线 · 论文库
• 上一篇: DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving
• 下一篇: DriveE2E: Closed-Loop Benchmark for End-to-End Autonomous Driving through Real-to-Simulation
• 返回 Topic