An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
Section titled “An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models”⚠️ AI 生成 · 建议对照原文 本页为自动整理的学习笔记；关键数据与引用如需引用，请回查 PDF / 官方版本。
学习档位 中文笔记
类型 文献 · 更新 2026-07-19
所属 VLM 与视觉语言理解
中文学习笔记（自动生成，需核验）
Section titled “中文学习笔记（自动生成，需核验）”Topic: vlm-understanding · Tier: recent · Year: 2023 · Venue: —
Evidence level: partial · 本地全文: 是 · 建议阅读: ~40 分钟
Paper: https://arxiv.org/abs/2309.09958
Code: —
Generator: grok
为什么值得读
Section titled “为什么值得读”早期系统性实证研究将开源视觉指令调优LMM（LLaVA）扩展至33B/65B/70B规模，量化了模型尺寸、图像分辨率、多模态-语言数据混合及LoRA/QLoRA对真实世界多模态与语言能力的影响，提供可复现的更强基线与实用权衡洞见，便于后续大规模LMM研究。
扩展LLaVA规模一致提升性能与语言能力，LoRA/QLoRA性能接近全参微调，高分辨率与数据混合关键，视觉指令调优有时可改善纯语言能力。
现有开源LMM研究多限于7B/13B及更小模型，显著扩展至33B/65B等更大尺寸对多模态与语言能力、训练方法与数据混合的影响尚未充分探索。
LLaVA两阶段视觉指令调优流程、Vicuna/LLaMA系列LLM与CLIP ViT视觉编码器、LoRA/QLoRA参数高效微调、DeepSpeed ZeRO、以及LLaVA-Bench、MM-VET、MM-Bench、Vicuna-80、MMLU等评估基准与GPT-4评分协议。
- 实证将LLaVA扩展到33B与65B/70B规模，并公开更强检查点作为基线。
- 系统研究模型尺寸、图像分辨率（224 vs 336）、数据混合（LLaVA-80K±ShareGPT）与可训练模块（全参 vs LoRA/QLoRA）的影响。
- 发现规模扩展一致提升多模态性能与语言能力；LoRA/QLoRA与全参微调性能可比且成本更低；更高分辨率与语言-多模态数据混合可提升效果；视觉指令调优有时可改善纯语言能力（如MMLU）。
- 在LLaVA-Bench与MM-VET上达到当时开源端到端LMM的新SOTA。
先准备指令调优LLM（公开Vicuna-33B或自训Vicuna-65B，基于ShareGPT）；再按LLaVA两阶段闪电训练：(i) Stage1特征对齐：仅训线性投影层（1024→6656/8192），用概念平衡的LAION-CC-SBU 558K样本；(ii) Stage2视觉指令调优：用LLaVA-80K多模态指令数据（可选混合ShareGPT语言指令数据），全参微调或LoRA/QLoRA调LLM，超参包括学习率、LoRA rank/alpha、DeepSpeed ZeRO3/2、序列长2048等。
关键模块和设计取舍
Section titled “关键模块和设计取舍”视觉编码器固定为CLIP ViT（比较224×224与336×336分辨率）；线性投影映射视觉特征到LLM词嵌入空间；LLM为Vicuna/LLaMA系列（7B-70B）；Stage2可全参微调或LoRA/QLoRA（强调较大学习率与alpha=2×rank对性能关键）；数据混合权衡多模态与语言能力；训练用DeepSpeed ZeRO与指定batch/节点配置，权衡成本与性能（如33B性价比高）。
数据集、实验设置与指标
Section titled “数据集、实验设置与指标”训练：Stage1用LAION-CC-SBU概念平衡子集558K；Stage2用LLaVA-80K多模态指令数据，可选混合ShareGPT（语言指令，65B自训约159M tokens）；评估：LLaVA-Bench（In-the-Wild，24图60问，分reasoning/conversation/detail，GPT-4相对评分，beam=1/5）；MM-VET（200图218问，6核心VL能力及组合，GPT-4评分）；MM-Bench（约2974问，LR/AR/RR/FP-S/FP-C/CP等技能）；语言侧用Vicuna-80与MMLU。结果报告均值±方差（部分多次运行）。
主要结果（需原文证据）
Section titled “主要结果（需原文证据）”LLaVA-Bench上，33B/65B总体优于13B与BingChat（如表1：LLaVA-65B beam=5 Overall 74.4，33B 74.8 vs 13B 73.5），更大模型在复杂推理与详细描述上更强；MM-VET上从13B到33B/65B一致提升，65B (Data Mixing) Total 36.4为开源端到端SOTA级（如表2），知识/生成提升最显著；缩放因素中模型尺寸、336分辨率（+2-3分）、数据混合（+约2分）均有效（表3）；LoRA/QLoRA在rank=64时性能接近全参且成本更低（表4，如33B性价比优）；混合数据提升多模态但语言能力大致保持或略降，LLaVA-70B基于LLaMA-2-70B-Chat后MMLU达65.1（提升2.4，表5），首次报告视觉指令调优改善大规模LMM语言能力。
局限、失败场景与适用边界
Section titled “局限、失败场景与适用边界”所用训练数据较小，发现仍属初步；LLaVA-Bench规模小，比较可能统计不显著；未探索视觉编码器扩展以增强识别/理解；未来需更大数据集系统研究数据选择与混合；部分结果依赖特定超参（如LoRA学习率/alpha）与硬件。
与前序 / 同期 / 后续方法的关系
Section titled “与前序 / 同期 / 后续方法的关系”直接扩展LLaVA（视觉指令调优）与同期MiniGPT-4等工作；比较OpenFlamingo、BLIP-2、InstructBLIP、Otter、LLaMA-Adapter v2等开源LMM，以及MM-ReAct（工具链系统）作为参考；基于Vicuna/LLaMA-2；与MM-Bench、MM-VET等新基准相关；结论指向未来更大视觉与数据规模研究，影响后续开源大规模LMM基线。
官方代码与复现建议
Section titled “官方代码与复现建议”代码与检查点将公开（论文声明）。建议基于原始LLaVA代码库，使用公开Vicuna-33B或自训65B（ShareGPT），DeepSpeed ZeRO，按摘录超参（Stage1 lr=1e-4，Stage2全参2e-5或LoRA 1e-4、alpha=2×rank、batch等）与数据（LLaVA-80K±ShareGPT、558K对齐数据）复现；注意QLoRA内存优势与多次评估平均。
推荐阅读顺序
Section titled “推荐阅读顺序”先读Abstract与Introduction明确动机与发现；再Experiment Setup掌握模型、两阶段训练、可训练模块与超参；重点Results（表1-5及3.1-3.2问答式分析）；最后Conclusions and Limitations；必要时回查引用与脚注数据来源。
- Q: LLaVA扩展实验的两阶段训练分别做什么？用了哪些数据？ A: Stage1：仅训线性投影做特征对齐，用概念平衡LAION-CC-SBU 558K；Stage2：视觉指令调优，用LLaVA-80K（可选混合ShareGPT）。
- Q: 主要发现哪些缩放因素有效？效果如何？ A: 模型尺寸一致提升；图像分辨率从224到336约+2-3分；混合语言-多模态数据约+2分；更大模型在推理/详细描述与知识/生成上提升更明显。
- Q: LoRA/QLoRA与全参微调相比如何？关键超参是什么？ A: 性能可比（尤其rank=64时接近全参），成本更低（内存与时间），适合大模型；较大学习率与alpha（=2×rank）对性能至关重要。
- Q: 视觉指令调优对纯语言能力有何影响？有何证据？ A: 仅多模态数据训练后语言能力大致保持；有时提升（如LLaVA-70B MMLU 65.1优于LLaMA-2-70B-Chat 63.1）；混合数据多提升多模态而非语言。
- Q: 论文在哪些基准上报告了SOTA或显著提升？主要局限是什么？ A: LLaVA-Bench与MM-VET上开源端到端新SOTA级结果；局限包括训练数据小、LLaVA-Bench规模有限、未扩展视觉编码器。
- page 1 Abstract: we present an empirical study of scaling LLaVA up to 33B and 65B/70B, and share our findings from our explorations in image resolution, data mixing and parameter-efficient training methods such as LoRA/QLoRA. … scaling LMM consistently enhances model performance and improves language capabilities, and performance of LoRA/QLoRA tuning of LMM are comparable to the performance of full-model fine-tuning. … higher image resolutions and mixing multimodal-language data … visual instruction tuning can sometimes improve LMM’s pure language capability.
- page 2 Experiment Setup: Once the instruction-tuned LLM is given, we follow [12] to perform the two-stage LLaVA lightning training: (i) Stage 1: Pre-training for Feature Alignment. … 558K samples. (ii) Stage 2: Visual Instruction Tuning. We use the LLaVA-80K multimodal instruct dataset …
- page 3-4 Table 1 & text: The 33B and 65B checkpoints outperform the 13B LLaVA model and Bing Chat. … larger LLaVA models generally exhibit better performance in tasks involving complex reasoning and generating detailed descriptions
- page 4 Table 2 & text: The performance is consistently improved from 13B to 33B and 65B. The largest LLaVA model improves SoTA performance among the end-to-end open-source LMM. … LLaVA-65B (Data Mixing) … 36.4±0.2
- page 5 Table 3 & text: Increasing the model size consistently improves the overall performance. … higher resolution consistently yields 2-3 points improvement … By mixing the language-only instruction data (ShareGPT) with LLaVA-80K, we can improve model performance by 2 points
- page 5-6 Table 4 & text: LoRA and QLoRA … For models with 33B parameters and above … increase in both performance and cost until full-model tuning … In the case of the 13B model, we find that a rank of 64 can deliver comparable performance to full-model tuning. … Large learning rate and alpha value of LoRA improves the results significantly.
- page 6 Table 5 & text: LLaVA, after being trained solely on multimodal instruction data, exhibits a comparable language capability. … we improve LLaMA-2-70B-Chat by 2.4 points on MMLU, yielding an overall MMLU score of 65.1 … this is the first reported result which shows visual instructing tuning improve language ability of large-scale LMM.
- page 6 Conclusions: Note that the training datasets used in this study is small. So, our findings are still preliminary. In future work, we will experiment using much larger datasets … We leave it to future work how to scale the vision encoder
Discovery evidence
Section titled “Discovery evidence”- topic:
vlm-understanding - sources:
arxiv - retrieved_at: 2026-07-20
- query: vision language model autonomous driving scene understanding
- arxiv:
2309.09958 - score_total: 50
- suggested_tier:
recent
Relevance
Section titled “Relevance”(no prose relevance explanation — numeric score only or HTTP source)
Snippets
Section titled “Snippets”(no snippet evidence in candidate pool)
延伸解读与背景补充
Section titled “延伸解读与背景补充”生成：2026-07-21 · 来源条数 2 · 模型
heuristic· 需人工核验数字
围绕「An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models」的核心问题与动机（待结合全文核验）。
方法要点补强
Section titled “方法要点补强”- 见原文方法章节；以下为基于摘要/摘录的要点提示。
- An Empirical Study of Scaling Instruction-Tuned Large Multimodal Models …
与相近工作的关系
Section titled “与相近工作的关系”与相近工作的关系待核验；请对照 related work。
- 勿仅凭摘要推断未给出的数值指标。
- 这篇工作的输入/输出表示是什么？（An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models）
- 训练目标与评测协议各是什么？
- 主要失败模式或局限是什么？
外部解读索引
Section titled “外部解读索引”- [2309.09958] An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models — 2026-07-21 — 官方摘要/二次页面（自动抓取）
- [2309.09958] An Empirical Study of Scaling Instruction-Tuned Large Multimodal Models — 2026-07-21 — 官方摘要/二次页面（自动抓取）
方法结构（重绘）
Section titled “方法结构（重绘）”flowchart LR A["输入 / 观测"] --> B["表示 / 编码"] B --> C["推理 / 解码"] C --> D["输出 / 动作或检测"] %% method sketch for: An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models方法结构示意（重绘；细节以原论文为准，待 PDF 核验）。
论文摘录图/表
Section titled “论文摘录图/表”
来源：原论文约 p.3（arch）；学习用途摘录。
来源：原论文约 p.4（table）；学习用途摘录。
英文自动分析（可折叠）
Section titled “英文自动分析（可折叠）”展开英文 Paper Card / AI deep analysis
Paper Card
Section titled “Paper Card”| Field | Content |
|---|---|
| Year | 2023 |
| Authors | Yadong Lu, Chunyuan Li, Haotian Liu, Jianwei Yang, Jianfeng Gao, Yelong Shen |
| arXiv | 2309.09958 |
| DOI | — |
| Topics | vlm-understanding |
| Paper | https://arxiv.org/abs/2309.09958 |
原文摘录与素材（可折叠）
Section titled “原文摘录与素材（可折叠）”展开 Extract / Selections / Local assets
Selections
Section titled “Selections”vlm-understanding: tier=recent rank=5 score=50 — auto refresh 2026-07-19 sources=arxiv | promoted watch->recent for coverage fill
Extract excerpt
Section titled “Extract excerpt”An Empirical Study of Scaling Instruction-Tuned Large Multimodal Models
Yadong Lu∗1 , Chunyuan Li∗2 , Haotian Liu3 , Jianwei Yang2 , Jianfeng Gao2 , Yelong Shen1
arXiv:2309.09958v1 [cs.CV] 18 Sep 2023 1 Microsoft Azure AI 2 Microsoft Research 3 University of Wisconsin–Madison
Abstract
Visual instruction tuning has recently shown encouraging progress with open- source large multimodal models (LMM) such as LLaVA and MiniGPT-4. How- ever, most existing studies of open-source LMM are performed using models with 13B parameters or smaller. In this paper we present an empirical study of scal- ing LLaVA up to 33B and 65B/70B, and share our findings from our explorations in image resolution, data mixing and parameter-efficient training methods such as LoRA/QLoRA. These are evaluated by their impact on the multi-modal and language capabilities when completing real-world tasks in the wild. We find that scaling LMM consistently enhances model performance and improves language capabilities, and performance of LoRA/QLoRA tuning of LMM are comparable to the performance of full-model fine-tuning. Additionally, the study highlights the importance of higher image resolutions and mixing multimodal-language data to improve LMM performance, and visual instruction tuning can sometimes im- prove LMM’s pure language capability. We hope this study makes state-of-the-art