RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS and Reinforcement Learning

RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learning

⚠️ AI 生成 · 建议对照原文 本页为自动整理的学习笔记；关键数据与引用如需引用，请回查 PDF / 官方版本。

学习档位 中文笔记

类型 文献 · 更新 2026-07-19

标签 autonomous-driving · end-to-end-learning · reinforcement-learning · closed-loop-evaluation

所属 端到端驾驶 · 强化学习 · 训练系统与实验管理

Migration status: imported from ad_projs@a823662; source anchors and claims remain needs-source-verification.

• Paper: https://arxiv.org/abs/2502.13144
• Code: https://github.com/hustvl/RAD at fa2b52fe3b405c53cc6747120e3e38ebcd15ceac

中文学习笔记（自动生成，需核验）

Topic: ad-end-to-end-driving · Tier: recent · Year: 2025 · Venue: —
Evidence level: partial · 本地全文: 是 · 建议阅读: ~40 分钟
Paper: https://arxiv.org/abs/2502.13144
Code: —
Generator: grok

为什么值得读

首次提出基于大规模3DGS的闭环强化学习框架用于端到端自动驾驶策略训练，结合IL正则化解决因果混淆与开环-闭环差距，并在未见3DGS环境中实现显著更低碰撞率，具有方法创新与实际部署启示价值。

一句话结论

RAD通过大规模3DGS构建逼真数字孪生环境进行闭环RL训练端到端驾驶策略，并以IL正则化实现人-车对齐，显著降低闭环碰撞率。

问题定义

现有端到端自动驾驶多采用模仿学习（IL），面临因果混淆（学习相关性而非因果、易捷径学习、对安全关键事件不敏感）以及开环训练与闭环部署差距（误差累积导致分布外场景失效）问题；真实世界闭环训练成本高风险大，游戏引擎仿真传感器不逼真。

前置知识

端到端自动驾驶与模仿学习基础；3D Gaussian Splatting（3DGS）场景重建与渲染；强化学习基础（PPO、GAE、稀疏奖励）；BEV感知与规划头设计；闭环评估指标。

核心贡献

• 提出首个基于3DGS的闭环RL框架用于训练端到端AD策略，专门设计奖励、动作空间、优化目标与交互机制以提升训练效率与效果。
• 提出将RL与IL协同优化：RL补充因果建模与缩小开环差距，IL补充人-车对齐与行为平滑。
• 构建包含多样未见3DGS环境的闭环评估基准，验证RAD在多数闭环指标上优于IL方法，尤其碰撞率显著降低。

方法流程

三阶段训练：1）感知预训练（用地图/agent真值监督BEV编码器、map head、agent head，冻结后续）；2）规划预训练（用大规模人类驾驶示范做IL，初始化动作分布，更新image encoder与planning head）；3）强化后训练（多worker并行在3DGS环境中rollout生成轨迹缓冲，交替RL（PPO）与IL步协同更新，仅更新image encoder与planning head；策略输出解耦横向/纵向离散动作分布与价值函数，用自行车模型更新位姿并与3DGS交互获取下一状态与奖励）。

关键模块和设计取舍

BEV编码器+map/agent head（实例级token）+独立image encoder（稠密token）→规划头（Transformer decoder +导航/自车状态，输出解耦ax/ay分布与Vx/Vy）；动作空间：0.5s短时域解耦离散横向/纵向（常数线速度/角速度假设，便于控制推导）；奖励：动态碰撞rdc、静态碰撞rsc、位置偏离rpd、航向偏离rhd（触发即终止episode）；奖励/价值/优势解耦到横向/纵向后用GAE传播与修改PPO优化；IL作为正则保持类人；其他交通参与者用log-replay；设计取舍：解耦动作降维降探索成本、稠密辅助目标缓解稀疏奖励、终止坏事件避免噪声数据、RL+IL互补而非纯RL。

数据集、实验设置与指标

从采集驾驶示范中选大量风险/密集交通clip，各训练独立3DGS模型作数字环境；闭环评估基准由多样、先前未见的3DGS环境组成。指标包括CR（碰撞率）、DCR、SCR、DR、PDR、HDR、ADD、Long. Jerk、Lat. Jerk等（表1对比IL/RL/RL+IL）。具体数据规模、完整划分与其他设置待来源核验。

主要结果（需原文证据）

相较IL方法，RAD在多数闭环指标上表现更强，尤其碰撞率约3×更低。表1中Stage3：IL的CR=0.229、DCR=0.211等；纯RL CR=0.143；RL+IL CR=0.089、DCR=0.080、SCR=0.009等，多数指标最优或更优。补充材料有丰富闭环结果。

局限、失败场景与适用边界

RL探索可能导致非类人行为（靠IL正则缓解）；稀疏奖励与收敛慢（靠稠密辅助与动作解耦缓解）；3DGS重建质量与log-replay其他参与者限制真实交互多样性；坏事件后立即终止（因传感器噪声）；sim2real与长期部署边界、完整失败案例细节待来源核验；仅更新部分参数，感知与规划目标冲突靠阶段解耦避免。

与前序 / 同期 / 后续方法的关系

动态场景重建：先前3DGS/隐式表示多用于闭环评估，本文进一步用于RL训练环。端到端IL：UniAD、VAD、VADv2、Hydra-MDP、DiffusionDrive等多为IL范式，本文用闭环RL增强。RL在AD：多基于CARLA等非逼真仿真或需完美感知，RAD首个在逼真3DGS环境中用真实传感器输入做端到端RL。IL+RL结合：CADRE/CIRL两阶段、联合优化方法多限于非逼真或结构化BEV，RAD首个在3DGS数字孪生中联合IL+RL做全端到端。

官方代码与复现建议

官方代码https://github.com/hustvl/RAD。复现建议：按三阶段顺序（感知预训练→规划IL预训练→强化后训练交替PPO+IL）；构建/使用3DGS环境与log-replay；注意动作解耦、奖励终止条件与参数冻结；闭环评估用未见环境；具体超参与完整环境数据待来源核验。

推荐阅读顺序

先读Abstract与Fig.1/引言（问题与范式对比）→Related Work（定位）→Sec.3.1-3.2（策略与三阶段）→Sec.3.3-3.5（交互、奖励、优化）→Table1与Fig.4（结果与奖励示例）→补充材料闭环可视化；最后看贡献与局限。

自测问题

1. Q: RAD如何同时解决IL的因果混淆与开环差距？ A: 用3DGS构建逼真数字世界进行大规模闭环RL试错探索状态空间并学习因果/安全响应，同时以IL作正则保持类人行为，二者互补。
2. Q: 动作空间如何设计及原因？ A: 解耦横向/纵向离散动作，每0.5s短时域假设恒定线/角速度；降维便于高效RL优化并直接推导控制信号。
3. Q: 奖励包含哪些成分？触发后如何处理？ A: rdc（动态碰撞）、rsc（静态碰撞）、rpd（位置偏离）、rhd（航向偏离）；任一触发立即终止episode，避免噪声传感器数据。
4. Q: 三阶段训练各更新哪些参数？ A: 感知预训练：BEV+map/agent head；规划预训练与强化后训练：image encoder+planning head（其余冻结）。
5. Q: 表1显示RL+IL相对纯IL的主要优势是什么？ A: CR从0.229降至0.089（及其他DCR/SCR等降低），多数闭环安全相关指标更优，同时兼顾偏离与jerk。

原文证据

• Abstract: Compared to IL-based methods, RAD achieves stronger performance in most closed-loop metrics, particularly exhibiting a 3× lower collision rate. … Code is available at https://github.com/hustvl/RAD
• Section 1 / Fig.1: We propose RAD, a 3DGS-based closed-loop Reinforcement Learning (RL) framework for end-to-end Autonomous Driving. … RL and IL work together to optimize the AD policy: RL enhances IL by addressing causation and the open-loop gap, while IL improves RL by ensuring better human alignment.
• Section 3.2 / Fig.2: We adopt a three-stage training paradigm: perception pre-training, planning pre-training, and reinforced post-training… In the reinforced post-training, RL and IL synergistically fine-tune the distribution.
• Section 3.4 / Fig.4: The reward function is designed to guide the ego vehicle’s behavior by penalizing unsafe actions… R = {rdc, rsc, rpd, rhd}. … Any of these events… triggers immediate episode termination.
• Table 1: Stage 3 CR↓ DCR↓ SCR↓ … IL 0.229 0.211 0.018 … RL 0.143 0.128 0.015 … RL+IL 0.089 0.080 0.009
• Related Work / Contributions: To the best of our knowledge, RAD is the first to train an end-to-end autonomous driving agent using RL in a photorealistic 3DGS environment. … We propose the first 3DGS-based RL framework for training end-to-end AD policy.

Evidence-backed Reading Notes

Topic	Evidence-backed note	Source	Short original cue
Problem	Open-loop imitation struggles with closed-loop distribution shift and interaction feedback.	[PDF p.1, Abstract]	Introduction
Representation	The note treats the method as 3DGS-based RL policy environment because the paper’s method pages introduce the relevant representation/module vocabulary.	[PDF p.3, Related Work]	Method
Core mechanism	Use a 3D Gaussian Splatting based environment to support scalable RL training of driving policies.	[PDF p.3, Related Work]	3DGS
Input / Output	Input: rendered/constructed driving environment observations. Output: driving policy actions.	[PDF p.3, Related Work]	3DGS
Training / Evaluation	The paper reports closed-loop/RL-style driving results; reproduction depends on environment assets and reward setup.	[PDF p.5, Method]	Table
Relationship	DiffusionDriveV2 constrains a diffusion planner with RL; RAD trains a broader closed-loop policy.	[PDF p.3, Related Work]	Related Work
Failure/Risk	Environment assets, reward design, simulator determinism, and training cost dominate reproduction risk.	[PDF p.3, Related Work]	3DGS
Reproduction boundary	Current local assets support code reading and demo inspection, not full RL reproduction.	[PDF p.3, Related Work]	3DGS

Code Cross-References

Local path	Why it matters
fa2b52fe3b405c53cc6747120e3e38ebcd15ceac:README.md	Code/repo anchor for implementation cross-check.
fa2b52fe3b405c53cc6747120e3e38ebcd15ceac:assets	Code/repo anchor for implementation cross-check.
fa2b52fe3b405c53cc6747120e3e38ebcd15ceac:compute_advantage.py	Code/repo anchor for implementation cross-check.
fa2b52fe3b405c53cc6747120e3e38ebcd15ceac:data	Code/repo anchor for implementation cross-check.
fa2b52fe3b405c53cc6747120e3e38ebcd15ceac:generate_action_anchor.py	Code/repo anchor for implementation cross-check.
fa2b52fe3b405c53cc6747120e3e38ebcd15ceac:planning_head.py	Code/repo anchor for implementation cross-check.

Reproduction Notes

• Treat this note as paper/code reading material first; do not interpret mini-data smoke tests as paper reproduction. [PDF p.3, Related Work]
• Before running experiments, verify the local code entry points above against the paper method terminology and dataset protocol. [PDF p.3, Related Work]
• If a claim is not linked to a PDF page or code path in this note, treat it as an implementation hypothesis rather than established paper fact.

Local Asset Identity

Discovery evidence

• topic: reinforcement-learning
• sources: semantic-scholar-http
• retrieved_at: 2026-07-20
• query: reinforcement learning end-to-end driving
• corpus_id: 276422140
• arxiv: 2502.13144
• doi: 10.48550/arXiv.2502.13144
• score_total: 81
• suggested_tier: recent

Relevance

(no prose relevance explanation — numeric score only or HTTP source)

Snippets

(no snippet evidence in candidate pool)

延伸解读与背景补充

生成：2026-07-21 · 来源条数 2 · 模型 heuristic · 需人工核验数字

问题再表述

围绕「RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS and Reinforcement Learning」的核心问题与动机（待结合全文核验）。

方法要点补强

• 见原文方法章节；以下为基于摘要/摘录的要点提示。
• RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learning …

与相近工作的关系

与相近工作的关系待核验；请对照 related work。

常见误读

• 勿仅凭摘要推断未给出的数值指标。

自测问题

1. 这篇工作的输入/输出表示是什么？（RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS and Reinforcement Learning）
2. 训练目标与评测协议各是什么？
3. 主要失败模式或局限是什么？

外部解读索引

• [2502.13144] RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learning — 2026-07-21 — 官方摘要/二次页面（自动抓取）
• [2502.13144] RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learning — 2026-07-21 — 官方摘要/二次页面（自动抓取）

关键图表

方法结构（重绘）

flowchart LR
  A["输入 / 观测"] --> B["表示 / 编码"]
  B --> C["推理 / 解码"]
  C --> D["输出 / 动作或检测"]
  %% method sketch for: RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS and Reinforcemen

方法结构示意（重绘；细节以原论文为准，待 PDF 核验）。

论文摘录图/表

来源：原论文约 p.1（arch）；学习用途摘录。

来源：原论文约 p.6（table）；学习用途摘录。

英文自动分析（可折叠）

展开英文 Paper Card / AI deep analysis

Paper Card

Field	Content
One-line takeaway	RAD trains an end-to-end driving policy with large-scale reinforcement learning in a 3DGS-based closed-loop environment.
Problem	Open-loop imitation struggles with closed-loop distribution shift and interaction feedback.
Representation	3DGS-based RL policy environment
Input / Output	Input: rendered/constructed driving environment observations. Output: driving policy actions.
Core Mechanism	Use a 3D Gaussian Splatting based environment to support scalable RL training of driving policies.
Training / Evaluation	The paper reports closed-loop/RL-style driving results; reproduction depends on environment assets and reward setup.
Reproduction Status	Current local assets support code reading and demo inspection, not full RL reproduction.
Compare With	DiffusionDriveV2 constrains a diffusion planner with RL; RAD trains a broader closed-loop policy.
Failure/Risk	Environment assets, reward design, simulator determinism, and training cost dominate reproduction risk.

原文摘录与素材（可折叠）

展开 Extract / Selections / Local assets

Source Anchors

Anchor	What to verify	Source	Short original cue
Title and abstract	Use to verify paper identity and top-level contribution.	[PDF p.1, Abstract]	RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learning
Motivation	Use to verify the problem statement and why the work is needed.	[PDF p.1, Abstract]	Introduction
Core method	Use to verify the main modeling mechanism and module names.	[PDF p.3, Related Work]	Method
Key module terms	Use to verify exact component names before editing the note.	[PDF p.3, Related Work]	3DGS
Dataset and protocol	Use to verify data dependencies: project-specific 3DGS/RL environment assets.	[PDF p.3, Related Work]	3DGS
Metrics and results	Use to verify metric names and reported benchmark context.	[PDF p.5, Method]	Table
Experiments or ablation	Use to verify which claims are experimentally supported.	[PDF p.5, Method]	Experiments
Position in related work	Use to verify the claimed relationship to neighboring methods.	[PDF p.3, Related Work]	Related Work
Conclusion or limits	Use to verify final claims and remaining constraints.	[PDF p.10, Method]	Conclusion

反向链接

• 强化学习
• 端到端驾驶
• 自动驾驶方法谱系
• 自动驾驶方法谱系
• 训练系统与实验管理

---

学习导航

标记为已读

• 所属 Topic: ad-end-to-end-driving
• 学习路径: 学习路线 · 论文库
• 上一篇: RAD-2: Scaling Reinforcement Learning in a Generator-Discriminator Framework for End-to-End Driving
• 下一篇: SSR: Navigation-guided Sparse Scene Representation for End-to-End Autonomous Driving
• 返回 Topic