Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challenge
Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challenge
Section titled “Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challenge”⚠️ AI 生成 · 建议对照原文 本页为自动整理的学习笔记；关键数据与引用如需引用，请回查 PDF / 官方版本。
学习档位 中文笔记
类型 文献 · 更新 2026-07-19
所属 具身智能体
中文学习笔记（自动生成，需核验）
Section titled “中文学习笔记（自动生成，需核验）”Topic: embodied-agents · Tier: recent · Year: 2024 · Venue: —
Evidence level: partial · 本地全文: 是 · 建议阅读: ~50 分钟
Paper: https://arxiv.org/abs/2411.01796
Code: —
Generator: grok
为什么值得读
Section titled “为什么值得读”这是首个明确以无障碍/包容性为核心的大规模具身社交智能挑战与基准，强调从自我中心视觉主动感知具有真实物理约束（如轮椅、儿童身高/力量限制等）的人类伙伴意图与能力，并制定个性化协作计划，覆盖室内外长视野任务、紧急事件与风险，填补现有多智能体基准忽视个体差异、户外场景与可及性的空白，并提供LLM+行为建模新方法与代码。
CHAIC 挑战要求具身智能体通过自我中心观察推断受限人类伙伴的意图与物理约束，并协作高效完成室内外家务/户外任务。
具身智能体需在自我中心RGB-D观察下，主动跟随并观察可能具有物理约束（如无法够高、轮椅受限、力量弱、移动慢）的人类伙伴，推断其目标意图与能力限制（社会感知），再制定用户定制的协作计划以共同尽快完成长视野运输任务（合作规划），同时应对紧急事件；现有基准多忽略人类多样性、可及性与户外/紧急场景。
具身AI与多智能体合作基础、3D仿真平台（如ThreeDWorld）、大语言模型规划、视频行为识别（如TSN/Kinetics）、社交感知与协作规划概念、目标运输/容器使用任务设置。
- 设计并实现4种具有真实物理约束的新模拟智能体（儿童、轮椅、自行车、脆弱）及8个覆盖室内外、紧急与风险的长视野任务，基于ThreeDWorld。
- 提出包容性具身社交智能挑战CHAIC，测试从自我中心视觉主动感知伙伴意图/约束并做用户定制协作规划的能力。
- 基准测试多种规划/学习基线（含基础模型），引入结合行为建模与LLM的新方法，并分析感知与合作中的持续挑战。
LLM+BM Helper框架：感知模块从原始RGB检测物体；记忆模块用深度图构建语义地图并记录行为；行为建模模块用动作识别模型（TSN预训练于Kinetics-400）识别伙伴动作（pick up/put on/put in/walking及其成败，共6类）并在语义地图上定位动作谓词对象；决策模块由LLM生成高层计划；执行模块转为低层动作（移动/交互/等待）。整体嵌入CoELA式架构。
关键模块和设计取舍
Section titled “关键模块和设计取舍”核心模块为感知（物体检测）、记忆（语义地图+行为记录）、行为建模（动作识别+ grounding）、决策（LLM规划）、执行（低层控制）。设计取舍：行为建模依赖视频动作识别以从原始RGB推断意图/限制，而非oracle符号；决策用LLM以提升泛化与推理，但基线显示从RGB建模行为仍困难；约束建模通过到达范围与力量阈值（超限时概率exp(-δ/α)/β成功），动作异步执行。
数据集、实验设置与指标
Section titled “数据集、实验设置与指标”基于ThreeDWorld的8个任务（无约束、低目标、障碍、高目标、高目标位置、高容器、购物、搬家），每个任务12训练+12测试episode，总计约200；episode最大3000帧（搬家1500）；观察为512×512自我中心RGB-D+自身状态；动作含导航、拾取、放入/放上、等待。指标：运输率TR（成功运输目标物体百分比）、效率提升EI（ΔM/M0）、目标推断准确率IA（helper运输目标占比）、紧急率ER（购物任务中儿童远离帧比例）。受限智能体用规则高层规划+真值分割。
主要结果（需原文证据）
Section titled “主要结果（需原文证据）”当前基线在从原始RGB图像建模伙伴行为上有困难，LLM驱动智能体在决策上具有竞争力；表格部分显示相对无helper，RHP与VLM等可提升TR/EI（具体完整数值待来源核验）。
局限、失败场景与适用边界
Section titled “局限、失败场景与适用边界”基线在原始RGB伙伴行为建模上困难；感知依赖检测/识别模型质量；任务以运输为主，约束与紧急事件有限；完整结果与失败场景细节待来源核验；仿真到真实差距及更复杂交互适用边界待进一步验证。
与前序 / 同期 / 后续方法的关系
Section titled “与前序 / 同期 / 后续方法的关系”建立在ThreeDWorld等3D仿真与多智能体合作（如Puig等家庭任务感知、但同构agent无物理约束差异）之上；对比Cao等辅助易感群体工作（仅室内+oracle符号观测），CHAIC强调自我中心视觉、室内外场景、新物理约束agent与意外事件；融入机器人无障碍设计原则（轮椅/老年/认知障碍护理），填补具身智能中可及性与个体差异空白。
官方代码与复现建议
Section titled “官方代码与复现建议”推荐阅读顺序
Section titled “推荐阅读顺序”先读Abstract与Introduction（问题与动机）→ Section 3完整挑战定义（约束agent、任务、观测/动作、生成与数据集）→ Section 4方法（LLM+BM流水线与行为建模）→ Section 5实验设置、基线与结果（含Table）→ Related Work与贡献总结；附录细节视需。
- Q: CHAIC中helper智能体必须具备的两大核心能力是什么？ A: （1）社会感知：通过跟随观察推断人类伙伴意图与物理约束；（2）合作规划：制定针对该伙伴的定制协作计划以尽快共同完成任务。
- Q: 论文实现了哪四种具有物理约束的新agent？各主要限制是什么？ A: Child（身高1.2m，到达[0,1.5]m）、Wheelchair（到达[0.25,1.5]m，可能被障碍阻挡）、Bicycle（移动慢，需先停靠自行车，伴随儿童可能跑开紧急）、Frail（力量仅为正常1/6，难举重物）。
- Q: 任务成功如何衡量？主要评估指标有哪些？ A: 以双方成功运输目标物体比例衡量；主要指标为Transport Rate (TR)、Efficiency Improvement (EI)、Goal Inference Accuracy (IA)，购物任务另加Emergency Rate (ER)。
- Q: LLM+BM方法中行为建模模块如何工作？ A: 用TSN（Kinetics-400预训练）识别伙伴动作（pick up等6类含成败），再在语义地图中定位最近对象作为谓词，输出动作、谓词与状态以支持意图/限制推断。
- Q: 与先前多智能体基准相比，CHAIC的主要差异是什么？ A: 明确针对多样化物理约束人类伙伴的主动视觉感知与策略适应；覆盖室内外、紧急事件；使用自我中心视觉而非仅oracle符号；强调可及性与包容性。
- Abstract: We introduce Constrained Human-AI Cooperation (CHAIC), an inclusive embodied social intelligence challenge designed to test social perception and cooperation in embodied agents. In CHAIC, the goal is for an embodied agent equipped with egocentric observations to assist a human who may be operating under physical constraints… (1) infer the human’s intents and constraints by following the human and observing their behaviors (social perception), and (2) make a cooperative plan tailored to the human partner…
- Abstract / Contributions: To benchmark this challenge, we create four new agents with real physical constraints and eight long-horizon tasks featuring both indoor and outdoor scenes with various constraints, emergency events, and potential risks. … Our benchmark and code are publicly available at https://github.com/UMass-Embodied-AGI/CHAIC.
- Section 1 Introduction / contributions: • We design and implement four new agents with real physical constraints and eight long-horizon tasks featuring both indoor and outdoor scenes on top of ThreeDWorld (Gan et al., 2021)… • We introduce a new embodied social intelligence challenge with accessibility explicitly in mind: Constrained Human-AI Cooperation (CHAIC)… • We benchmark several baseline models… especially a new agent with behavior modeling introduced by us…
- Section 3.1 Constrained Agents: • Child Agent: A small child with a height of 1.2 m that has a reaching range of [0, 1.5] m. • Wheelchair Agent: An agent confined to a wheelchair… reaching range is [0.25, 1.5] m. • Bicycle Agent: An agent walking with a bike that moves slowly… The child accompanying it may run away… • Frail Agent: An agent that is less capable of lifting heavy objects… only 1/6 the strength of a normal agent.
- Section 3.3.2 Dataset Construction: For each of the eight tasks, we create 12 episodes for training and 12 episodes for testing, resulting in approximately 200 episodes in total. … An episode terminates when all goal predicates of the task are satisfied or when the maximum time step horizon T = 3000 frames is reached (for the moving furniture task, the maximum time step horizon is T = 1500).
- Section 5.1.2 Evaluation Metrics: • Transport rate (TR): The percentage of target objects that the agents successfully transported. We also calculate the Efficiency Improvement (EI)… • Goal Inference Accuracy (IA)… • Emergency Rate (ER): For the shopping task…
- Section 1 / Abstract end: Our benchmark results suggest that current baselines have difficulty modeling partner behaviors from raw RGB images, and LLM-driven agents are competitive agents in decision-making.
Discovery evidence
Section titled “Discovery evidence”- topic:
decision-task-planning - sources:
asta - retrieved_at: 2026-07-20
- query: Find foundational and recent research papers for the topic «决策与任务规划» (decision-task-planning). Prefer peer-reviewed or widely cited work with clear method contributions. Include open-source code when available. Exclude pure survey spam unless highly cited. Core concepts: task planning, hierarchical planning, decision making. Search facets: embodied agent task planning decision making; hierarchical planning language model robotics; task and motion planning autonomous agents. Relevant venues inclu
- corpus_id:
273811787 - relevance_score:
0.7702612350578637 - score_total: 34
- suggested_tier:
watch
Relevance
Section titled “Relevance”(no prose relevance explanation — numeric score only or HTTP source)
Snippets
Section titled “Snippets”(no snippet evidence in candidate pool)
延伸解读与背景补充
Section titled “延伸解读与背景补充”生成：2026-07-21 · 来源条数 2 · 模型
heuristic· 需人工核验数字
围绕「Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challenge」的核心问题与动机（待结合全文核验）。
方法要点补强
Section titled “方法要点补强”- 见原文方法章节；以下为基于摘要/摘录的要点提示。
- Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challenge …
与相近工作的关系
Section titled “与相近工作的关系”与相近工作的关系待核验；请对照 related work。
- 勿仅凭摘要推断未给出的数值指标。
- 这篇工作的输入/输出表示是什么？（Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challenge）
- 训练目标与评测协议各是什么？
- 主要失败模式或局限是什么？
外部解读索引
Section titled “外部解读索引”- [2411.01796] Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challenge — 2026-07-21 — 官方摘要/二次页面（自动抓取）
- [2411.01796] Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challenge — 2026-07-21 — 官方摘要/二次页面（自动抓取）
方法结构（重绘）
Section titled “方法结构（重绘）”flowchart LR A["输入 / 观测"] --> B["表示 / 编码"] B --> C["推理 / 解码"] C --> D["输出 / 动作或检测"] %% method sketch for: Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Chal方法结构示意（重绘；细节以原论文为准，待 PDF 核验）。
论文摘录图/表
Section titled “论文摘录图/表”
来源：原论文约 p.2（arch）；学习用途摘录。
来源：原论文约 p.4（table）；学习用途摘录。
英文自动分析（可折叠）
Section titled “英文自动分析（可折叠）”展开英文 Paper Card / AI deep analysis
Paper Card
Section titled “Paper Card”| Field | Content |
|---|---|
| Year | 2024 |
| Authors | Weihua Du, Qiushi Lyu, Jiaming Shan, Zhenting Qi, Hongxin Zhang, Sunli Chen, Andi Peng, Tianmin Shu, Kwonjoon Lee, Behzad Dariush, Chuang Gan |
| arXiv | 2411.01796 |
| DOI | — |
| Topics | embodied-agents |
| Paper | https://arxiv.org/abs/2411.01796 |
原文摘录与素材（可折叠）
Section titled “原文摘录与素材（可折叠）”展开 Extract / Selections / Local assets
Selections
Section titled “Selections”embodied-agents: tier=recent rank=2 score=56 — auto refresh 2026-07-19 sources=arxiv | promoted watch->recent for coverage fill
Extract excerpt
Section titled “Extract excerpt”Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challenge
Z \ Weihua Du∗1 , Qiushi Lyu∗2 , Jiaming Shan3 , Zhenting Qi4 , Hongxin Zhang5 , Sunli Chen5 “ Andi Peng6 , Tianmin Shu7 , Kwonjoon Lee8 , Behzad Dariush8 , Chuang Gan5ˇ 1 2 3 Carnegie Mellon University, Peking University, University of California, Santa Barbara 4 Harvard University, 5 University of Massachusetts Amherst, 6 MIT 7 Johns Hopkins University, 8 Honda Research Institute USA
arXiv:2411.01796v3 [cs.AI] 11 Jun 2025 Z \ “ weihuad@cs.cmu.edu, lvqiushi@stu.pku.edu.cn, ˇ chuangg@umass.edu
Abstract We introduce Constrained Human-AI Cooperation (CHAIC), an inclusive embodied social intelligence challenge designed to test social perception and cooperation in embodied agents. In CHAIC, the goal is for an embodied agent equipped with egocentric observations to assist a human who may be operating under physical constraints—e.g., unable to reach high places or confined to a wheelchair—in performing common household or outdoor tasks as efficiently as possible. To achieve this, a success