强化学习
基于环境交互、回报和价值估计学习决策策略的知识入口。
类型 Topics · 更新 2026-07-20
本 Topic 要解决什么
Section titled “本 Topic 要解决什么”本Topic聚焦自动驾驶中强化学习（RL）方法侧问题，尤其针对端到端驾驶策略与轨迹规划器。纯模仿学习（IL）易受因果混淆、开环-闭环差距、误差累积与安全关键事件不敏感困扰；真实世界闭环训练成本高风险大，传统仿真传感器不够逼真。RL通过与环境交互优化长期回报，可引入负反馈、探索与闭环鲁棒性，结合offline RL、policy gradient、IL正则化及大规模3DGS数字孪生，实现更安全、可泛化的E2E策略与motion planner训练。核心覆盖RAD等框架、DRL轨迹跟踪、reinforced imitative planning、offline RL for driving，以及与CARLA/Bench2Drive等闭环评估的衔接， bridging method-side for RAD and RL planners。
非目标 / 边界
Section titled “非目标 / 边界”- 纯IL或行为克隆细节实现
- 感知前端网络架构设计
- 真实车辆硬件部署与标定
- 非驾驶域通用RL理论证明
- 具体超参调优与工程代码实现
foundations— 需掌握MDP、价值函数、策略梯度与深度RL基础，以便理解DDPG、offline RL及驾驶中的闭环交互imitation-offline— RL常与IL结合（正则化、伪专家、imitative），需理解开环IL局限与offline数据利用ad-end-to-end-driving— 本Topic为E2E驾驶的RL方法侧，需先了解E2E pipeline与闭环评估背景
| 术语 | 含义 | 常见混淆 |
|---|---|---|
| Offline RL | 仅用固定数据集训练策略，无在线交互，适合驾驶中安全与数据复用 | 易与online RL或纯行为克隆混淆，offline需处理分布偏移与外推误差 |
| Policy Gradient | 直接优化策略参数以最大化期望回报的一类算法，常用于连续控制驾驶动作 | 与值函数方法（Q-learning）区别，方差大且需on-policy或修正 |
| 因果混淆 | IL中模型学习输入输出相关性而非因果，导致对安全关键场景不敏感 | 常与分布偏移、捷径学习混谈，RL通过交互与回报可缓解 |
| 闭环训练/评估 | 策略输出动作后环境状态演化并反馈，形成交互循环，区别于开环预测 | 开环易高估性能，闭环揭示误差累积与交互失败 |
| IL正则化 | 在RL目标中加入模仿专家轨迹的项，平衡探索与人-车对齐、稳定性 | 过强则退化为IL，过弱则偏离安全行为 |
| 3DGS数字孪生 | 用3D Gaussian Splatting构建高保真可交互仿真环境，支持大规模RL闭环训练 | 与传统游戏引擎仿真区别在于视觉逼真度与可扩展性 |
| Latent Deep RL | 在潜空间进行深度强化学习，提升可解释性与端到端驾驶效率 | 与显式轨迹或直接像素-动作映射的区别 |
方法谱系与时间线
Section titled “方法谱系与时间线”早期自动驾驶RL多直接端到端像素到控制（如race driving DRL），或survey总结DRL+IL结合。随后出现latent DRL提升可解释性，以及offline RL、reinforced imitative planning缓解在线交互风险。近年转向大规模3DGS闭环RL（RAD）与生成器-判别器解耦、伪专家正则化offline RL，强调IL正则化解决因果混淆与开环-闭环差距，并依托CARLA/Real2Sim基准实现更真实评估。整体从纯探索/在线向安全、数据高效、与IL混合的method-side演进。
- (2016) foundational — Continuous control with deep reinforcement learning
- (2017) foundational — Deep Reinforcement Learning framework for Autonomous Driving
- (2018) watch — End-to-End Race Driving with Deep Reinforcement Learning
- (2021) foundational — A Survey of Deep RL and IL for Autonomous Driving Policy Learning
- (2021) watch — End-to-End Urban Driving by Imitating a Reinforcement Learning Coach
- (2021) recent — Interpretable End-to-End Urban Autonomous Driving With Latent Deep Reinforcement Learning
- (2021) recent — Deep Reinforcement Learning for Autonomous Driving: A Survey
- (2023) needs-review — VIPeR: Provably Efficient Algorithm for Offline RL with Neural Function Approximation
- (2023) watch — On the Effectiveness of Offline RL for Dialogue Response Generation
- (2023) watch — DRL-Based Trajectory Tracking for Motion-Related Modules in Autonomous Driving
- (2024) recent — Reinforced Imitative Trajectory Planning for Urban Automated Driving
- (2024) recent — Human-in-the-Loop Reinforcement Learning: A Survey and Position on Requirements, Challenges, and Opportunities
- (2025) watch — Causal-Paced Deep Reinforcement Learning
- (2025) watch — DriveE2E: Closed-Loop Benchmark for End-to-End Autonomous Driving through Real-to-Simulation
- (2025) watch — Pseudo-Expert Regularized Offline RL for End-to-End Autonomous Driving in Photorealistic Closed-Loop Environments
- (2025) watch — Design and Assessment of Reinforcement Learning Algorithms for End-to-End Autonomous Driving Learning
- (2025) recent — RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS and Reinforcement Learning
- DRL-Based Trajectory Tracking for Motion-Related Modules in Autonomous Driving（精读）— 运动相关模块上的 DRL 轨迹跟踪，连接 RL 与规划控制接口
- Reinforced Imitative Trajectory Planning for Urban Automated Driving（精读）— 强化模仿式轨迹规划，示范 RL+IL 混合在城市场景规划中的用法
- RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS and Reinforcement Learning（待获取 PDF / 自动卡）— 大规模 3DGS 闭环 RL 案例；精读笔记齐备前作扩展书目，不作空链必读
推荐阅读路径
Section titled “推荐阅读路径”- 先读两篇Survey建立DRL+IL全景与驾驶policy learning问题定义
- 再读Continuous control with deep reinforcement learning（DDPG）与早期End-to-End Race Driving，理解连续控制与直接E2E RL基础
- 然后读Interpretable End-to-End Urban Autonomous Driving With Latent Deep Reinforcement Learning与Reinforced Imitative Trajectory Planning，掌握latent与imitative混合
- 重点精读RAD与Pseudo-Expert Regularized Offline RL相关，结合snippets理解3DGS闭环与offline正则化
- 最后对照DriveE2E/Bench2Drive/CARLA等评估，思考闭环基准与工程风险
| Paper | 输入 | 输出 | 表示 | 训练目标 |
|---|---|---|---|---|
| RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS and Reinforcement Learning | 传感器/3DGS场景观察 | 端到端驾驶策略/动作 | E2E policy + IL正则化 | 闭环回报最大化 + 人-车对齐，降低碰撞 |
| Interpretable End-to-End Urban Autonomous Driving With Latent Deep Reinforcement Learning | 城市驾驶观测 | 可解释控制/轨迹 | Latent deep RL | 提升可解释性与端到端性能 |
| Reinforced Imitative Trajectory Planning for Urban Automated Driving | 城市场景状态 | 轨迹规划 | Reinforced imitative | 结合RL与模仿的轨迹优化 |
| Continuous control with deep reinforcement learning | 连续状态 | 连续动作 | Actor-Critic (DDPG) | 连续控制期望回报最大化 |
| Pseudo-Expert Regularized Offline RL for End-to-End Autonomous Driving in Photorealistic Closed-Loop Environments | 离线专家/伪专家数据 | E2E策略 | Offline RL + 伪专家正则 | 安全高效offline训练 + 闭环鲁棒 |
工程实现与复现风险
Section titled “工程实现与复现风险”- 闭环仿真与真实分布差距导致sim-to-real失败
- RL探索过程中的不安全动作风险（即使仿真）
- Offline RL分布偏移与外推误差导致策略崩溃
- IL正则化强度难以平衡，过强退化为IL过弱偏离安全
- 大规模3DGS环境构建与训练算力/吞吐成本高
- 因果混淆残余与奖励设计不当引发捷径行为
- 评估基准（CARLA/Bench2Drive）场景覆盖不足导致过拟合
- 如何进一步缩小3DGS/Real2Sim闭环与真实道路的差距？
- Offline RL与online fine-tuning在驾驶中的最优混合策略？
- 高维轨迹/多模态规划中RL的稳定扩展（如生成器-判别器）？
- Human-in-the-loop RL在自动驾驶中的可扩展交互协议？
- 如何设计更鲁棒的奖励与安全约束以应对长尾交互？
Topic 自测清单
Section titled “Topic 自测清单”- 简述纯IL在端到端驾驶中的因果混淆与开环-闭环差距问题，RL如何针对性缓解？
- Offline RL与Online RL在自动驾驶应用中的主要权衡是什么？结合伪专家正则化说明。
- RAD框架如何利用大规模3DGS与IL正则化训练E2E策略？其相对传统仿真的优势？
- 比较Latent Deep RL与直接像素-动作DRL在可解释性与驾驶性能上的差异。
- Policy gradient方法在连续控制驾驶中的作用，以及与值函数方法的区别？
选篇附录（registry）
Section titled “选篇附录（registry）”本 Topic registry 入选 17 篇（foundational/recent/watch 见 papers.yml）。
论文池（FRW）
Section titled “论文池（FRW）”来自 research/papers.yml 的正式选题配额：基础必读 3 · 近期重要 5 · 观察 5。
可学习条目需本地 PDF 或已有精读/中文笔记；无正文资产的选题标为 待获取 PDF，不作为学习路径必读链接。
基础必读（选题 3 · 可学习 0 · 待获取 PDF 3）
Section titled “基础必读（选题 3 · 可学习 0 · 待获取 PDF 3）”本档尚无可学习正文（PDF/中文笔记）；下列为待获取选题。
- A Survey of Deep RL and IL for Autonomous Driving Policy Learning (2021) · 待获取 PDF
- Continuous control with deep reinforcement learning (2016) · 待获取 PDF
- Deep Reinforcement Learning framework for Autonomous Driving (2017) · 待获取 PDF
近期重要（选题 5 · 可学习 2 · 待获取 PDF 3）
Section titled “近期重要（选题 5 · 可学习 2 · 待获取 PDF 3）”- RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS and Reinforcement Learning (2025) · 固定 · 中文笔记
- Reinforced Imitative Trajectory Planning for Urban Automated Driving (2024) · 精读
- Interpretable End-to-End Urban Autonomous Driving With Latent Deep Reinforcement Learning (2021) · 待获取 PDF
- Deep Reinforcement Learning for Autonomous Driving: A Survey (2021) · 待获取 PDF
- Human-in-the-Loop Reinforcement Learning: A Survey and Position on Requirements, Challenges, and Opportunities (2024) · 待获取 PDF
观察清单（选题 8 · 可学习 6 · 待获取 PDF 2）
Section titled “观察清单（选题 8 · 可学习 6 · 待获取 PDF 2）”- DriveE2E: Closed-Loop Benchmark for End-to-End Autonomous Driving through Real-to-Simulation (2025) · 中文笔记
- Pseudo-Expert Regularized Offline RL for End-to-End Autonomous Driving in Photorealistic Closed-Loop Environments (2025) · 中文笔记
- DRL-Based Trajectory Tracking for Motion-Related Modules in Autonomous Driving (2023) · 精读
- Causal-Paced Deep Reinforcement Learning (2025) · 中文笔记
- On the Effectiveness of Offline RL for Dialogue Response Generation (2023) · 中文笔记
- Design and Assessment of Reinforcement Learning Algorithms for End-to-End Autonomous Driving Learning (2025) · 中文笔记
- End-to-End Race Driving with Deep Reinforcement Learning (2018) · 待获取 PDF
- End-to-End Urban Driving by Imitating a Reinforcement Learning Coach (2021) · 待获取 PDF
关联 Topic
Section titled “关联 Topic”成熟度 developing · 内容数 18 · 论文池 16