mirrors/llava-vl--llava-next
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

docs: make Chinese README the default

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 10:19:33 +00:00
parent c1787d6479
commit b001966d6b
1 changed files with 98 additions and 93 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+98 -93
View File
@@ -1,8 +1,14 @@
<!-- WEHUB_ZH_README -->
> [!NOTE]
> 本文档由 WeHub 基于上游 README 翻译整理，属于社区翻译，非官方中文文档。
> [English](./README.en.md) · [原始项目](https://github.com/LLaVA-VL/LLaVA-NeXT) · [上游 README](https://github.com/LLaVA-VL/LLaVA-NeXT/blob/HEAD/README.md)
> 原作者、版权与许可证归属以原始项目及本仓库 LICENSE 文件为准。
<p align="center" width="100%">
<img src="https://i.postimg.cc/pL17YtG4/WX20240508-220230-2x.png" width="80%" height="80%">
</p>
# LLaVA-NeXT: Open Large Multimodal Models
# LLaVA-NeXT：开放大型多模态模型
[![Static Badge](https://img.shields.io/badge/llava_video-paper-green)](http://arxiv.org/abs/2410.02713)
[![Static Badge](https://img.shields.io/badge/llava_onevision-paper-green)](https://arxiv.org/abs/2408.03326)
[![llava_next-blog](https://img.shields.io/badge/llava_next-blog-green)](https://llava-vl.github.io/blog/)
@@ -17,108 +23,108 @@
[![llava_next-interleave_checkpoints](https://img.shields.io/badge/llava_next-interleave_checkpoints-blue)](https://huggingface.co/collections/lmms-lab/llava-next-interleave-66763c55c411b340b35873d1)
[![llava_next-image_checkpoints](https://img.shields.io/badge/llava_next-image_checkpoints-blue)](https://huggingface.co/lmms-lab)
## Release Notes
## 发布说明
> **📢 Announcement:** The training pipeline in this repository is now considered **legacy**. For the latest training pipeline with support for **LLaVA-OneVision** and **LLaVA-OneVision-2**, please refer to [**lmms-engine**](https://github.com/EvolvingLMMs-Lab/lmms-engine).
> **📢 公告：** 本仓库中的训练流程现已被视为**遗留（legacy）**。如需支持 **LLaVA-OneVision** 与 **LLaVA-OneVision-2** 的最新训练流程，请参阅 [**lmms-engine**](https://github.com/EvolvingLMMs-Lab/lmms-engine).
- **[2025/08/29] 🔥 LLaVA-Critic-R1** We release LLaVA-Critic-R1, a family of generative critic VLM trained through GRPO using pairwise critic data. LLaVA-Critic-R1 not only demonstrates strong critic capability, but also achieves state-of-the-art policy performance at the 7B scale. Refer to [LLaVA-Critic-R1](llava-critic-r1/EasyR1/README.md) for more training details.
- **[2025/08/29] 🔥 LLaVA-Critic-R1** 我们发布 LLaVA-Critic-R1，这是一族通过 GRPO 使用成对 critic 数据训练的生成式 critic 视觉语言模型（VLM）。LLaVA-Critic-R1 不仅展现出强大的 critic 能力，还在 7B 规模上实现了最先进的策略性能。更多训练细节请参阅 [LLaVA-Critic-R1](llava-critic-r1/EasyR1/README.md)。
📄 **Explore more**:
- [LLaVA-Critic-GRPO Dataset](https://huggingface.co/datasets/lmms-lab/LLaVA-Critic-GRPO-dataset): Download the dataset.
- [LLaVA-Critic-R1-7B](https://huggingface.co/lmms-lab/LLaVA-Critic-R1-7B): LLaVA-Critic-R1 trained based on Qwen-2.5-VL-7B.
- [LLaVA-Critic-R1-7B-Plus-Qwen](https://huggingface.co/lmms-lab/LLaVA-Critic-R1-7B-Plus-Qwen): LLaVA-Critic-R1+ trained based on ThinkLite-VL-7B.
- [LLaVA-Critic-R1-7B-Plus-Mimo](https://huggingface.co/lmms-lab/LLaVA-Critic-R1-7B-Plus-Mimo): LLaVA-Critic-R1+ trained based on MiMo-VL-7B-RL-2508.
- [LLaVA-Critic-R1-7B-Plus-LLaMA32v](https://huggingface.co/lmms-lab/LLaVA-Critic-R1-7B-LLaMA32v): LLaVA-Critic-R1+ trained based on Llama-3.2-11B-Vision-Instruct.
- [Paper](https://arxiv.org/abs/2509.00676): Detailed information about LLaVA-Critic-R1.
📄 **了解更多**:
- [LLaVA-Critic-GRPO Dataset](https://huggingface.co/datasets/lmms-lab/LLaVA-Critic-GRPO-dataset): 下载数据集。
- [LLaVA-Critic-R1-7B](https://huggingface.co/lmms-lab/LLaVA-Critic-R1-7B): 基于 Qwen-2.5-VL-7B 训练的 LLaVA-Critic-R1。
- [LLaVA-Critic-R1-7B-Plus-Qwen](https://huggingface.co/lmms-lab/LLaVA-Critic-R1-7B-Plus-Qwen): 基于 ThinkLite-VL-7B 训练的 LLaVA-Critic-R1+。
- [LLaVA-Critic-R1-7B-Plus-Mimo](https://huggingface.co/lmms-lab/LLaVA-Critic-R1-7B-Plus-Mimo): 基于 MiMo-VL-7B-RL-2508 训练的 LLaVA-Critic-R1+。
- [LLaVA-Critic-R1-7B-Plus-LLaMA32v](https://huggingface.co/lmms-lab/LLaVA-Critic-R1-7B-LLaMA32v): 基于 Llama-3.2-11B-Vision-Instruct 训练的 LLaVA-Critic-R1+。
- [Paper](https://arxiv.org/abs/2509.00676): 关于 LLaVA-Critic-R1 的详细信息。
- **[2024/10/04] 🔥 LLaVA-Video** (formerly LLaVA-NeXT-Video) has undergone a major upgrade! We are excited to release **LLaVA-Video-178K**, a high-quality synthetic dataset for video instruction tuning. This dataset includes:
- **[2024/10/04] 🔥 LLaVA-Video**（原 LLaVA-NeXT-Video）已完成重大升级！我们很高兴发布 **LLaVA-Video-178K**，这是一个用于视频指令微调的高质量合成数据集。该数据集包括：
- 178,510 caption entries
- 960,792 open-ended Q&A pairs
- 196,198 multiple-choice Q&A items
- 178,510 条 caption 条目
- 960,792 对开放式问答（Q&A）
- 196,198 道多项选择题
Along with this, we’re also releasing the **LLaVA-Video 7B/72B models**, which deliver competitive performance on the latest video benchmarks, including [Video-MME](https://video-mme.github.io/home_page.html#leaderboard), [LongVideoBench](https://longvideobench.github.io/), and [Dream-1K](https://tarsier-vlm.github.io/).
与此同时，我们还发布了 **LLaVA-Video 7B/72B 模型**，在包括 [Video-MME](https://video-mme.github.io/home_page.html#leaderboard), [LongVideoBench](https://longvideobench.github.io/), 和 [Dream-1K](https://tarsier-vlm.github.io/). 在内的最新视频基准测试中展现出具有竞争力的性能。
📄 **Explore more**:
- [LLaVA-Video-178K Dataset](https://huggingface.co/datasets/lmms-lab/LLaVA-Video-178K): Download the dataset.
- [LLaVA-Video Models](https://huggingface.co/collections/lmms-lab/llava-video-661e86f5e8dabc3ff793c944): Access model checkpoints.
- [Paper](http://arxiv.org/abs/2410.02713): Detailed information about LLaVA-Video.
- [LLaVA-Video Documentation](https://github.com/LLaVA-VL/LLaVA-NeXT/blob/main/docs/LLaVA_Video_1003.md): Guidance on training, inference and evaluation.
📄 **了解更多**:
- [LLaVA-Video-178K Dataset](https://huggingface.co/datasets/lmms-lab/LLaVA-Video-178K): 下载数据集。
- [LLaVA-Video Models](https://huggingface.co/collections/lmms-lab/llava-video-661e86f5e8dabc3ff793c944): 获取模型检查点。
- [Paper](http://arxiv.org/abs/2410.02713): 关于 LLaVA-Video 的详细信息。
- [LLaVA-Video Documentation](https://github.com/LLaVA-VL/LLaVA-NeXT/blob/main/docs/LLaVA_Video_1003.md): 训练、推理与评估指南。
- [2024/09/13] 🔥 **🚀 [LLaVA-OneVision-Chat](docs/LLaVA_OneVision_Chat.md)**. The new LLaVA-OV-Chat (7B/72B) significantly improves the chat experience of LLaVA-OV. 📄
- [2024/09/13] 🔥 **🚀 [LLaVA-OneVision-Chat](docs/LLaVA_OneVision_Chat.md)**。全新的 LLaVA-OV-Chat（7B/72B）显著改善了 LLaVA-OV 的对话体验。📄
![](docs/ov_chat_images/chat_results.png)
- [2024/08/06] 🔥 **🚀 [LLaVA-OneVision (OV)](https://llava-vl.github.io/blog/2024-08-05-llava-onevision/)!** The new LLaVA-OV models (0.5B/7B/72B) achieve new state-of-the-art performance across single-image, multi-image, and video benchmarks, sometimes rivaling top commercial models on 47 diverse benchmarks. 📄 Explore More:
* [[Paper]](https://arxiv.org/abs/2408.03326): In-depth insights, new emegerging scenarios, ie, strong video understadning through task transfer from images.
* [[LLaVA-OV Doc]](https://github.com/LLaVA-VL/LLaVA-NeXT/blob/main/docs/LLaVA_OneVision.md): Model inference and evaluation guidance.
* [[Scripts]](https://github.com/LLaVA-VL/LLaVA-NeXT/blob/main/scripts/train): Start training models on your single-image/multi-image/video data.
- [2024/08/06] 🔥 **🚀 [LLaVA-OneVision (OV)](https://llava-vl.github.io/blog/2024-08-05-llava-onevision/)!** 全新的 LLaVA-OV 模型（0.5B/7B/72B）在单图、多图和视频基准测试中创下新的最先进性能，在 47 个多样化基准上有时可与顶级商业模型相媲美。📄 了解更多：
* [[Paper]](https://arxiv.org/abs/2408.03326): 深入见解与新涌现场景，例如通过从图像任务迁移实现强大的视频理解。
* [[LLaVA-OV Doc]](https://github.com/LLaVA-VL/LLaVA-NeXT/blob/main/docs/LLaVA_OneVision.md): 模型推理与评估指南。
* [[Scripts]](https://github.com/LLaVA-VL/LLaVA-NeXT/blob/main/scripts/train): 在你的单图/多图/视频数据上开始训练模型。
- [2024/07/16] 🔥 **LLaVA-NeXT-Video** has been upgraded. The new 32B model achieves the best open-source performance on several video benchmarks, including [Video-MME](https://video-mme.github.io/home_page.html#leaderboard). Please refer to [this page](docs/LLaVA-NeXT-Video_0716.md) for details, refer to [llava_next-video_demo](https://huggingface.co/spaces/WildVision/vision-arena) for demo.
- [2024/07/16] 🔥 **LLaVA-NeXT-Video** 已升级。新的 32B 模型在包括 [Video-MME](https://video-mme.github.io/home_page.html#leaderboard). 在内的多个视频基准测试中实现了最佳开源性能。详情请参阅 [this page](docs/LLaVA-NeXT-Video_0716.md)，演示请参阅 [llava_next-video_demo](https://huggingface.co/spaces/WildVision/vision-arena)。
- [2024/06/23] 🔥 **LLaVA-NeXT-Interleave** is released. We utilize image-text interleaved format to unify multi-image, video, and 3D tasks in one LLM and achieve **SoTA** performance on a wide range of benchmarks. Check out [paper](https://arxiv.org/pdf/2407.07895), [blog](https://llava-vl.github.io/blog/2024-06-16-llava-next-interleave/), and [checkpoints](https://huggingface.co/collections/lmms-lab/llava-next-interleave-66763c55c411b340b35873d1) to see new capabilities and improved performance! We have released 0.5b, 7b, and 7b-dpo models.
* An all-round LLM for multi-image, video, and 3D with strong performance \[[demo](https://huggingface.co/spaces/lmms-lab/LLaVA-NeXT-Interleave-Demo)\]
* Construct interleave training data [**M4-Instruct**](https://huggingface.co/datasets/lmms-lab/M4-Instruct-Data)
* Construct multi-image benchmark [**LLaVA-Interleave Bench**](https://huggingface.co/datasets/lmms-lab/LLaVA-NeXT-Interleave-Bench)
- [2024/06/23] 🔥 **LLaVA-NeXT-Interleave** 已发布。我们利用图文交错（image-text interleaved）格式，在单一 LLM 中统一多图、视频与 3D 任务，并在广泛基准测试中实现 **SoTA** 性能。请查看 [paper](https://arxiv.org/pdf/2407.07895), [blog](https://llava-vl.github.io/blog/2024-06-16-llava-next-interleave/), 和 [checkpoints](https://huggingface.co/collections/lmms-lab/llava-next-interleave-66763c55c411b340b35873d1) 以了解新能力与性能提升！我们已发布 0.5b、7b 和 7b-dpo 模型。
* 面向多图、视频与 3D 的全能 LLM，性能强劲 \[[demo](https://huggingface.co/spaces/lmms-lab/LLaVA-NeXT-Interleave-Demo)\]
* 构建交错训练数据 [**M4-Instruct**](https://huggingface.co/datasets/lmms-lab/M4-Instruct-Data)
* 构建多图基准 [**LLaVA-Interleave Bench**](https://huggingface.co/datasets/lmms-lab/LLaVA-NeXT-Interleave-Bench)
- [2024/05/25] 🔥 Wondering "[What Else Influences Visual Instruction Tuning Beyond Data?](https://llava-vl.github.io/blog/2024-05-25-llava-next-ablations/)" Our new [blog](https://llava-vl.github.io/blog/2024-05-25-llava-next-ablations/) summarizes empirical explorations to ablate the various design choices in improving LMMs except instruct data itself. Meanwhile, open-source the recapioned high-quality data using LLaVA-NeXT-34B on [[COCO]](https://huggingface.co/datasets/lmms-lab/LLaVA-ReCap-118K) [[LCS]](https://huggingface.co/datasets/lmms-lab/LLaVA-ReCap-558K) [[CC3M]](https://huggingface.co/datasets/lmms-lab/LLaVA-ReCap-CC3M).
* Architectures (LMM & Vision Encoder)
* Visual Representations (Resolution & # Tokens)
* Training Strategies (High-quality data & Trainable modules)
- [2024/05/25] 🔥 好奇「[What Else Influences Visual Instruction Tuning Beyond Data?](https://llava-vl.github.io/blog/2024-05-25-llava-next-ablations/)」？我们的新 [blog](https://llava-vl.github.io/blog/2024-05-25-llava-next-ablations/) 总结了除指令数据本身之外，通过消融实验探索改进 LMM 各项设计选择的实证研究。同时，我们开源了使用 LLaVA-NeXT-34B 重新标注的高质量数据：[[COCO]](https://huggingface.co/datasets/lmms-lab/LLaVA-ReCap-118K) [[LCS]](https://huggingface.co/datasets/lmms-lab/LLaVA-ReCap-558K) [[CC3M]](https://huggingface.co/datasets/lmms-lab/LLaVA-ReCap-CC3M).
* 架构（LMM 与 Vision Encoder）
* 视觉表示（分辨率与 Token 数量）
* 训练策略（高质量数据与可训练模块）
- [2024/05/10] 🔥 **LLaVA-NeXT** (Stronger) models are released, with support of stronger LMM inlcuding LLama-3 (8B) and Qwen-1.5 (72B/110B) Check out [[blog](https://llava-vl.github.io/blog/2024-05-10-llava-next-stronger-llms/)] and [[checkpoints](https://huggingface.co/lmms-lab)] to see improved performance!
- [2024/05/10] 🔥 **LLaVA-NeXT** (Video) is released. The image-only-trained LLaVA-NeXT model is surprisingly strong on video tasks with zero-shot modality transfer. DPO training with AI feedback on videos can yield significant improvement. [[Blog](https://llava-vl.github.io/blog/2024-04-30-llava-next-video/)], [[checkpoints](https://huggingface.co/collections/lmms-lab/llava-next-video-661e86f5e8dabc3ff793c944)] and [[sglang](https://github.com/sgl-project/sglang)]
- [2024/01/30] 🔥 **LLaVA-NeXT** is out! With additional scaling to LLaVA-1.5, LLaVA-NeXT-34B outperforms Gemini Pro on some benchmarks. It can now process 4x more pixels and perform more tasks/applications than before. Check out the [blog post](https://llava-vl.github.io/blog/2024-01-30-llava-next/), and explore the [demo](https://llava.hliu.cc/)! Models are available in [Model Zoo](https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZOO.md). Training/eval data and scripts coming soon.
- [2024/05/10] 🔥 **LLaVA-NeXT**（Stronger）模型已发布，支持更强的 LMM，包括 LLama-3（8B）和 Qwen-1.5（72B/110B）。请查看 [[blog](https://llava-vl.github.io/blog/2024-05-10-llava-next-stronger-llms/)] 和 [[checkpoints](https://huggingface.co/lmms-lab)] 了解性能提升！
- [2024/05/10] 🔥 **LLaVA-NeXT**（Video）已发布。仅基于图像训练的 LLaVA-NeXT 模型在视频任务上出人意料地强劲，可实现零样本模态迁移。在视频上使用 AI 反馈进行 DPO 训练可带来显著提升。[[Blog](https://llava-vl.github.io/blog/2024-04-30-llava-next-video/)], [[checkpoints](https://huggingface.co/collections/lmms-lab/llava-next-video-661e86f5e8dabc3ff793c944)] 和 [[sglang](https://github.com/sgl-project/sglang)]
- [2024/01/30] 🔥 **LLaVA-NeXT** 正式发布！在 LLaVA-1.5 基础上进一步扩展规模后，LLaVA-NeXT-34B 在部分基准测试中超越了 Gemini Pro。它现在可处理 4 倍像素，并能完成比以往更多的任务/应用。请查看 [blog post](https://llava-vl.github.io/blog/2024-01-30-llava-next/), 并体验 [demo](https://llava.hliu.cc/)! 模型可在 [Model Zoo](https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZOO.md). 获取。训练/评估数据与脚本即将发布。
<details>
<summary>More</summary>
- [2024/03/10] 🔥 发布 **LMMs-Eval**，这是我们在开发 LLaVA-NeXT 时使用的高效评测流水线。它支持在数十个公开数据集上评测 LMMs，并允许接入新数据集，从而大幅加快新 LMM 的开发。[[Blog](https://lmms-lab.github.io/lmms-eval-blog/lmms-eval-0.1/)] [[Codebase](https://github.com/EvolvingLMMs-Lab/lmms-eval)]
- [2024/03/10] 🔥 Releasing **LMMs-Eval**, a highly efficient evaluation pipeline we used when developing LLaVA-NeXT. It supports the evaluation of LMMs on dozens of public datasets and allows new dataset onboarding, making the dev of new LMMs much faster. [[Blog](https://lmms-lab.github.io/lmms-eval-blog/lmms-eval-0.1/)] [[Codebase](https://github.com/EvolvingLMMs-Lab/lmms-eval)]
- [2023/11/10] [LLaVA-Plus](https://llava-vl.github.io/llava-plus/) is released: Learning to Use Tools for Creating Multimodal Agents, with LLaVA-Plus (LLaVA that Plug and Learn to Use Skills). [[Project Page](https://llava-vl.github.io/llava-plus/)] [[Demo](https://llavaplus.ngrok.io/)] [[Code](https://github.com/LLaVA-VL/LLaVA-Plus-Codebase)] [[Paper](https://arxiv.org/abs/2311.05437)]
- [2023/11/02] [LLaVA-Interactive](https://llava-vl.github.io/llava-interactive/) is released: Experience the future of human-AI multimodal interaction with an all-in-one demo for Image Chat, Segmentation, Generation and Editing. [[Project Page](https://llava-vl.github.io/llava-interactive/)] [[Demo](https://llavainteractive.ngrok.io/)] [[Code](https://github.com/LLaVA-VL/LLaVA-Interactive-Demo)] [[Paper](https://arxiv.org/abs/2311.00571)]
- [2023/10/26] 🔥 LLaVA-1.5 with LoRA achieves comparable performance as full-model finetuning, with a reduced GPU RAM requirement ([ckpts](https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZOO.md#llava-v15), [script](https://github.com/haotian-liu/LLaVA#train)). We also provide a [doc](https://github.com/haotian-liu/LLaVA/blob/main/docs/Finetune_Custom_Data.md) on how to finetune LLaVA-1.5 on your own dataset with LoRA.
- [2023/10/12] Check out the Korean LLaVA (Ko-LLaVA), created by ETRI, who has generously supported our research! [[🤗 Demo](https://huggingface.co/spaces/etri-vilab/Ko-LLaVA)]
- [2023/10/05] 🔥 LLaVA-1.5 is out! Achieving SoTA on 11 benchmarks, with just simple modifications to the original LLaVA, utilizes all public data, completes training in ~1 day on a single 8-A100 node, and surpasses methods like Qwen-VL-Chat that use billion-scale data. Check out the [technical report](https://arxiv.org/abs/2310.03744), and explore the [demo](https://llava.hliu.cc/)! Models are available in [Model Zoo](https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZOO.md). The training data and scripts of LLaVA-1.5 are released [here](https://github.com/haotian-liu/LLaVA#train), and evaluation scripts are released [here](https://github.com/haotian-liu/LLaVA/blob/main/docs/Evaluation.md)!
- [2023/09/26] LLaVA is improved with reinforcement learning from human feedback (RLHF) to improve fact grounding and reduce hallucination. Check out the new SFT and RLHF checkpoints at project [[LLavA-RLHF]](https://llava-rlhf.github.io/)
- [2023/09/22] [LLaVA](https://arxiv.org/abs/2304.08485) is accepted by NeurIPS 2023 as **oral presentation**, and [LLaVA-Med](https://arxiv.org/abs/2306.00890) is accepted by NeurIPS 2023 Datasets and Benchmarks Track as **spotlight presentation**.
- [2023/11/06] Support **Intel** dGPU and CPU platforms. [More details here.](https://github.com/haotian-liu/LLaVA/tree/intel/docs/intel)
- [2023/10/12] LLaVA is now supported in [llama.cpp](https://github.com/ggerganov/llama.cpp/pull/3436) with 4-bit / 5-bit quantization support!
- [2023/10/11] The training data and scripts of LLaVA-1.5 are released [here](https://github.com/haotian-liu/LLaVA#train), and evaluation scripts are released [here](https://github.com/haotian-liu/LLaVA/blob/main/docs/Evaluation.md)!
- [2023/10/10] [Roboflow Deep Dive](https://blog.roboflow.com/first-impressions-with-llava-1-5/): First Impressions with LLaVA-1.5.
- [2023/09/20] We summarize our empirical study of training 33B and 65B LLaVA models in a [note](https://arxiv.org/abs/2309.09958). Further, if you are interested in the comprehensive review, evolution and trend of multimodal foundation models, please check out our recent survey paper [``Multimodal Foundation Models: From Specialists to General-Purpose Assistants''.](https://arxiv.org/abs/2309.10020)
- [2023/11/10] 发布 [LLaVA-Plus](https://llava-vl.github.io/llava-plus/)：学习使用工具以创建多模态智能体（Learning to Use Tools for Creating Multimodal Agents），即 LLaVA-Plus（可插拔并学习使用技能的 LLaVA，LLaVA that Plug and Learn to Use Skills）。[[Project Page](https://llava-vl.github.io/llava-plus/)] [[Demo](https://llavaplus.ngrok.io/)] [[Code](https://github.com/LLaVA-VL/LLaVA-Plus-Codebase)] [[Paper](https://arxiv.org/abs/2311.05437)]
- [2023/11/02] 发布 [LLaVA-Interactive](https://llava-vl.github.io/llava-interactive/)：通过一体化演示体验人机多模态交互的未来，涵盖图像对话、分割、生成与编辑（Image Chat, Segmentation, Generation and Editing）。[[Project Page](https://llava-vl.github.io/llava-interactive/)] [[Demo](https://llavainteractive.ngrok.io/)] [[Code](https://github.com/LLaVA-VL/LLaVA-Interactive-Demo)] [[Paper](https://arxiv.org/abs/2311.00571)]
- [2023/10/26] 🔥 采用 LoRA 的 LLaVA-1.5 可达到与全模型微调相当的性能，同时降低 GPU 显存需求（[ckpts](https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZOO.md#llava-v15), [script](https://github.com/haotian-liu/LLaVA#train)).）。我们还提供了关于如何使用 LoRA 在你自己的数据集上微调 LLaVA-1.5 的[文档](https://github.com/haotian-liu/LLaVA/blob/main/docs/Finetune_Custom_Data.md)。
- [2023/10/12] 请查看由 ETRI 创建、并慷慨支持我们研究的韩语版 LLaVA（Ko-LLaVA）！[[🤗 Demo](https://huggingface.co/spaces/etri-vilab/Ko-LLaVA)]
- [2023/10/05] 🔥 LLaVA-1.5 发布！在原始 LLaVA 基础上仅做简单修改，即在 11 个基准上达到 SoTA（state-of-the-art），使用全部公开数据，在单台 8-A100 节点上约 1 天完成训练，并超越使用十亿级数据的方法（如 Qwen-VL-Chat）。请查看[技术报告](https://arxiv.org/abs/2310.03744),并体验[演示](https://llava.hliu.cc/)!。模型可在[Model Zoo](https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZOO.md).获取。LLaVA-1.5 的训练数据与脚本发布在[此处](https://github.com/haotian-liu/LLaVA#train),，评测脚本发布在[此处](https://github.com/haotian-liu/LLaVA/blob/main/docs/Evaluation.md)!。
- [2023/09/26] LLaVA 通过基于人类反馈的强化学习（RLHF）得到改进，以提升事实 grounding 并减少幻觉。请在项目 [[LLavA-RLHF]](https://llava-rlhf.github.io/) 查看新的 SFT 与 RLHF 检查点。
- [2023/09/22] [LLaVA](https://arxiv.org/abs/2304.08485) 被 NeurIPS 2023 接收为 **oral presentation**（口头报告），[LLaVA-Med](https://arxiv.org/abs/2306.00890) 被 NeurIPS 2023 Datasets and Benchmarks Track 接收为 **spotlight presentation**（亮点报告）。
- [2023/11/06] 支持 **Intel** 独立 GPU 与 CPU 平台。[更多详情见此。](https://github.com/haotian-liu/LLaVA/tree/intel/docs/intel)
- [2023/10/12] LLaVA 现已在 [llama.cpp](https://github.com/ggerganov/llama.cpp/pull/3436) 中得到支持，并提供 4-bit / 5-bit 量化支持！
- [2023/10/11] LLaVA-1.5 的训练数据与脚本发布在[此处](https://github.com/haotian-liu/LLaVA#train),，评测脚本发布在[此处](https://github.com/haotian-liu/LLaVA/blob/main/docs/Evaluation.md)!。
- [2023/10/10] [Roboflow Deep Dive](https://blog.roboflow.com/first-impressions-with-llava-1-5/):：LLaVA-1.5 初体验。
- [2023/09/20] 我们在一篇[笔记](https://arxiv.org/abs/2309.09958).中总结了对 33B 与 65B LLaVA 模型训练的实证研究。此外，若你对多模态基础模型的全面综述、演进与趋势感兴趣，请阅读我们近期的综述论文 [``Multimodal Foundation Models: From Specialists to General-Purpose Assistants''.](https://arxiv.org/abs/2309.10020)
<p align="center">
<img src="https://github.com/Computer-Vision-in-the-Wild/CVinW_Readings/blob/main/images/mfm_evolution.jpeg?raw=true" width=50%/>
</p>
- [2023/07/19] 🔥 We release a major upgrade, including support for LLaMA-2, LoRA training, 4-/8-bit inference, higher resolution (336x336), and a lot more. We release [LLaVA Bench](https://github.com/haotian-liu/LLaVA/blob/main/docs/LLaVA_Bench.md) for benchmarking open-ended visual chat with results from Bard and Bing-Chat. We also support and verify training with RTX 3090 and RTX A6000. Check out [LLaVA-from-LLaMA-2](https://github.com/haotian-liu/LLaVA/blob/main/docs/LLaVA_from_LLaMA2.md), and our [model zoo](https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZOO.md)!
- [2023/06/26] [CVPR 2023 Tutorial](https://vlp-tutorial.github.io/) on **Large Multimodal Models: Towards Building and Surpassing Multimodal GPT-4**! Please check out [[Slides](https://datarelease.blob.core.windows.net/tutorial/vision_foundation_models_2023/slides/Chunyuan_cvpr2023_tutorial_lmm.pdf)] [[Notes](https://arxiv.org/abs/2306.14895)] [[YouTube](https://youtu.be/mkI7EPD1vp8)] [[Bilibli](https://www.bilibili.com/video/BV1Ng4y1T7v3/)].
- [2023/06/11] We released the preview for the most requested feature: DeepSpeed and LoRA support! Please see documentations [here](./docs/LoRA.md).
- [2023/06/01] We released **LLaVA-Med: Large Language and Vision Assistant for Biomedicine**, a step towards building biomedical domain large language and vision models with GPT-4 level capabilities. Checkout the [paper](https://arxiv.org/abs/2306.00890) and [page](https://github.com/microsoft/LLaVA-Med).
- [2023/05/06] We are releasing [LLaVA-Lighting-MPT-7B-preview](https://huggingface.co/liuhaotian/LLaVA-Lightning-MPT-7B-preview), based on MPT-7B-Chat! See [here](#LLaVA-MPT-7b) for more details.
- [2023/05/02] 🔥 We are releasing LLaVA-Lighting! Train a lite, multimodal GPT-4 with just $40 in 3 hours! See [here](#train-llava-lightning) for more details.
- [2023/04/27] Thanks to the community effort, LLaVA-13B with 4-bit quantization allows you to run on a GPU with as few as 12GB VRAM! Try it out [here](https://github.com/oobabooga/text-generation-webui/tree/main/extensions/llava).
- [2023/04/17] 🔥 We released **LLaVA: Large Language and Vision Assistant**. We propose visual instruction tuning, towards building large language and vision models with GPT-4 level capabilities. Checkout the [paper](https://arxiv.org/abs/2304.08485) and [demo](https://llava.hliu.cc/).
- [2023/07/19] 🔥 我们发布重大升级，包括对 LLaMA-2、LoRA 训练、4-/8-bit 推理、更高分辨率（336x336）等的支持。我们发布 [LLaVA Bench](https://github.com/haotian-liu/LLaVA/blob/main/docs/LLaVA_Bench.md) 用于对开放式视觉对话进行基准测试，并包含 Bard 与 Bing-Chat 的结果。我们还支持并验证了在 RTX 3090 与 RTX A6000 上的训练。请查看 [LLaVA-from-LLaMA-2](https://github.com/haotian-liu/LLaVA/blob/main/docs/LLaVA_from_LLaMA2.md), 以及我们的 [model zoo](https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZOO.md)!
- [2023/06/26] [CVPR 2023 Tutorial](https://vlp-tutorial.github.io/)：**Large Multimodal Models: Towards Building and Surpassing Multimodal GPT-4**（大型多模态模型：构建并超越多模态 GPT-4）！请查看 [[Slides](https://datarelease.blob.core.windows.net/tutorial/vision_foundation_models_2023/slides/Chunyuan_cvpr2023_tutorial_lmm.pdf)] [[Notes](https://arxiv.org/abs/2306.14895)] [[YouTube](https://youtu.be/mkI7EPD1vp8)] [[Bilibli](https://www.bilibili.com/video/BV1Ng4y1T7v3/)].
- [2023/06/11] 我们发布了最受期待功能的预览版：DeepSpeed 与 LoRA 支持！请参阅[此处的文档](./docs/LoRA.md)。
- [2023/06/01] 我们发布 **LLaVA-Med：生物医学领域的大型语言与视觉助手**（Large Language and Vision Assistant for Biomedicine），朝着构建具备 GPT-4 级能力的生物医学领域大型语言与视觉模型迈进。请查看 [论文](https://arxiv.org/abs/2306.00890) 与 [页面](https://github.com/microsoft/LLaVA-Med).。
- [2023/05/06] 我们发布基于 MPT-7B-Chat 的 [LLaVA-Lighting-MPT-7B-preview](https://huggingface.co/liuhaotian/LLaVA-Lightning-MPT-7B-preview),！更多详情见[此处](#LLaVA-MPT-7b)。
- [2023/05/02] 🔥 我们发布 LLaVA-Lighting！仅用 40 美元、3 小时即可训练轻量级多模态 GPT-4！更多详情见[此处](#train-llava-lightning)。
- [2023/04/27] 得益于社区努力，4-bit 量化的 LLaVA-13B 允许你在显存低至 12GB 的 GPU 上运行！可在[此处](https://github.com/oobabooga/text-generation-webui/tree/main/extensions/llava).试用。
- [2023/04/17] 🔥 我们发布 **LLaVA：大型语言与视觉助手**（Large Language and Vision Assistant）。我们提出视觉指令微调（visual instruction tuning），致力于构建具备 GPT-4 级能力的大型语言与视觉模型。请查看 [论文](https://arxiv.org/abs/2304.08485) 与 [演示](https://llava.hliu.cc/).。
</details>
<!-- <a href="https://llava.hliu.cc/"><img src="assets/demo.gif" width="70%"></a> -->
**Usage and License Notices**: This project utilizes certain datasets and checkpoints that are subject to their respective original licenses. Users must comply with all terms and conditions of these original licenses, including but not limited to the [OpenAI Terms of Use](https://openai.com/policies/terms-of-use) for the dataset and the specific licenses for base language models for checkpoints trained using the dataset (e.g. [Llama-1/2 community license](https://ai.meta.com/llama/license/) for LLaMA-2 and Vicuna-v1.5, [Tongyi Qianwen RESEARCH LICENSE AGREEMENT](https://huggingface.co/Qwen/Qwen1.5-0.5B-Chat/blob/main/LICENSE) and [Llama-3 Research License](https://llama.meta.com/llama3/license/)). This project does not impose any additional constraints beyond those stipulated in the original licenses. Furthermore, users are reminded to ensure that their use of the dataset and checkpoints is in compliance with all applicable laws and regulations.
**Usage and License Notices**：本项目使用了一些受各自原始许可证约束的数据集与检查点。用户必须遵守这些原始许可证的全部条款与条件，包括但不限于数据集的 [OpenAI Terms of Use](https://openai.com/policies/terms-of-use)，以及使用该项目数据集训练的检查点所对应的基础语言模型的特定许可证（例如：LLaMA-2 与 Vicuna-v1.5 的 [Llama-1/2 community license](https://ai.meta.com/llama/license/)，[Tongyi Qianwen RESEARCH LICENSE AGREEMENT](https://huggingface.co/Qwen/Qwen1.5-0.5B-Chat/blob/main/LICENSE) 与 [Llama-3 Research License](https://llama.meta.com/llama3/license/)).）。本项目不会在这些原始许可证之外施加任何额外限制。此外，提醒用户确保其对数据集与检查点的使用符合所有适用的法律法规。
## Models & Scripts
### Installation
#### 1. **Clone this repository and navigate to the LLaVA folder:**
#### 1. **克隆本仓库并进入 LLaVA 文件夹：**
```bash
git clone https://github.com/LLaVA-VL/LLaVA-NeXT
cd LLaVA-NeXT
```
#### 2. **Install the inference package:**
#### 2. **安装推理包：**
```bash
conda create -n llava python=3.10 -y
conda activate llava
@@ -126,58 +132,57 @@ pip install --upgrade pip # Enable PEP 660 support.
pip install -e ".[train]"
```
### Project Navigation
Please checkout the following page for more inference & evaluation details.
### 项目导航
请访问以下页面，了解更多推理与评估详情。
#### - **LLaVA-OneVision: Easy Task Transfer**
- [LLaVA-OneVision](https://github.com/LLaVA-VL/LLaVA-NeXT/blob/main/docs/LLaVA_OneVision.md): for demo inference. The evaluation code is in [lmms-eval](https://github.com/EvolvingLMMs-Lab/lmms-eval).
#### - **LLaVA-OneVision：轻松实现任务迁移（Easy Task Transfer）**
- [LLaVA-OneVision](https://github.com/LLaVA-VL/LLaVA-NeXT/blob/main/docs/LLaVA_OneVision.md):：用于 demo 推理。评估代码见 [lmms-eval](https://github.com/EvolvingLMMs-Lab/lmms-eval).
#### - **LLaVA-NeXT: Stronger LLMs Supercharge Multimodal Capabilities in the Wild**
- [LLaVA-NeXT-Image](./docs/LLaVA-NeXT.md): for image demo inference and evaluation of stronger LMMs using [lmms-eval](https://github.com/EvolvingLMMs-Lab/lmms-eval).
#### - **LLaVA-NeXT：更强的 LLM 在真实场景中增强多模态能力**
- [LLaVA-NeXT-Image](./docs/LLaVA-NeXT.md)：用于图像 demo 推理，以及使用 [lmms-eval](https://github.com/EvolvingLMMs-Lab/lmms-eval). 对更强的 LMM 进行评估。
#### - LLaVA-NeXT：强大的零样本（Zero-shot）视频理解模型
- [LLaVA-NeXT-Video](./docs/LLaVA-NeXT-Video.md)：用于视频推理与评估脚本。我们建议使用 [LMMs-video](https://lmms-lab.github.io/posts/lmms-eval-0.2/) 进行评估。
#### - LLaVA-NeXT: A Strong Zero-shot Video Understanding Model
- [LLaVA-NeXT-Video](./docs/LLaVA-NeXT-Video.md): for video inference and evaluation scripts. We recommend to use [LMMs-video](https://lmms-lab.github.io/posts/lmms-eval-0.2/) for evaluation.
#### - LLaVA-NeXT：在大型多模态模型中处理多图、视频与 3D
- [LLaVA-NeXT-Interleave](./docs/LLaVA-NeXT-Interleave.md)：用于多图 demo 与评估脚本。
#### - LLaVA-NeXT: Tackling Multi-image, Video, and 3D in Large Multimodal Models
- [LLaVA-NeXT-Interleave](./docs/LLaVA-NeXT-Interleave.md): for multi-image demo and evaluation scripts.
## 使用 SGLang 加速推理与部署
## SGLang for SpeedUp Inference and Deployment
我们使用 [SGLang](https://github.com/sgl-project/sglang) 来加速 LLaVA-NeXT 的推理与部署。你可以通过 SGLang 将 LLaVA-NeXT 作为后端 API 服务运行。
We use [SGLang](https://github.com/sgl-project/sglang) to speed up inference and deployment of LLaVA-NeXT. You could make LLaVA-NeXT as a backend API service with SGLang.
**Prepare Environment**:
Following the instruction in the [sglang](https://github.com/sgl-project/sglang?tab=readme-ov-file#install)
**准备环境**：
请按照 [sglang](https://github.com/sgl-project/sglang?tab=readme-ov-file#install) 中的说明进行操作。
### LLaVA-NeXT/OneVision
Checkout the HTTP Post/Get and SRT usage at [sglang/examples/runtime/llava_onevision](https://github.com/sgl-project/sglang/tree/main/examples/runtime/llava_onevision)
请在 [sglang/examples/runtime/llava_onevision](https://github.com/sgl-project/sglang/tree/main/examples/runtime/llava_onevision) 查看 HTTP Post/Get 与 SRT 的用法。
### LLaVA-NeXT (Video)
**Launch and Run on (K) Nodes**:
- Go to sglang project
**在 (K) 个节点上启动并运行**：
- 进入 sglang 项目
```
cd PATH_TO/sglang
```
- First node:
- 第一个节点：
```sh
bash examples/usage/llava_video/srt_example_llava_v.sh K 0 YOUR_VIDEO_PATH YOUR_MODEL_PATH FRAMES_PER_VIDEO
(e.g. bash examples/usage/llava_video/srt_example_llava_v.sh K 0 examples/usage/llava_video/videos/Q98Z4OTh8RwmDonc.mp4 lmms-lab/LLaVA-NeXT-Video-7B-DPO 16)
```
- Second node:
- 第二个节点：
```sh
bash examples/usage/llava_video/srt_example_llava_v.sh K 1 YOUR_VIDEO_PATH YOUR_MODEL_PATH FRAMES_PER_VIDEO
```
- The K node:
- 第 K 个节点：
```sh
bash examples/usage/llava_video/srt_example_llava_v.sh K K-1 YOUR_VIDEO_PATH YOUR_MODEL_PATH FRAMES_PER_VIDEO
```
## Citation
## 引用
If you find it useful for your research and applications, please cite related papers/blogs using this BibTeX:
若本工作对你的研究和应用有所帮助，请使用以下 BibTeX 引用相关论文/博客：
```bibtex
@article{li2024llava,
title={LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models},
@@ -233,18 +238,18 @@ If you find it useful for your research and applications, please cite related pa
}
```
## Acknowledgement
## 致谢
- [Vicuna](https://github.com/lm-sys/FastChat): the codebase we built upon, and our base model Vicuna-13B that has the amazing language capabilities!
- The LLaVA-NeXT project is currently maintained by the team along with our contributors (listed alphabetically by the first names): [Bo Li](https://brianboli.com/), [Dong Guo](https://www.linkedin.com/in/dongguoset/), [Feng Li](https://scholar.google.com/citations?hl=zh-CN&user=ybRe9GcAAAAJ&view_op=list_works&sortby=pubdate), [Hao Zhang](https://scholar.google.com/citations?user=B8hPxMQAAAAJ&hl=en), [Kaichen Zhang](https://www.linkedin.com/in/kaichen-zhang-014b17219/?originalSubdomain=sg), [Renrui Zhang](https://zrrskywalker.github.io/), [Yuanhan Zhang](https://zhangyuanhan-ai.github.io/), led by [Chunyuan Li](https://chunyuan.li/) and with the guidance and help from [Haotian Liu](https://hliu.cc/).
- The `lmms-eval` framework and its core contributors, including Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, and Kairui Hu, for their support on the evaluation side.
- [Vicuna](https://github.com/lm-sys/FastChat):：我们基于此构建代码库，并以具备出色语言能力的 Vicuna-13B 作为基础模型！
- LLaVA-NeXT 项目目前由团队与贡献者共同维护（按名字首字母排序）：[Bo Li](https://brianboli.com/), [Dong Guo](https://www.linkedin.com/in/dongguoset/), [Feng Li](https://scholar.google.com/citations?hl=zh-CN&user=ybRe9GcAAAAJ&view_op=list_works&sortby=pubdate), [Hao Zhang](https://scholar.google.com/citations?user=B8hPxMQAAAAJ&hl=en), [Kaichen Zhang](https://www.linkedin.com/in/kaichen-zhang-014b17219/?originalSubdomain=sg), [Renrui Zhang](https://zrrskywalker.github.io/), [Yuanhan Zhang](https://zhangyuanhan-ai.github.io/),，由 [Chunyuan Li](https://chunyuan.li/) 带领，并得到 [Haotian Liu](https://hliu.cc/). 的指导与帮助。
- `lmms-eval` 框架及其核心贡献者，包括 Peiyuan Zhang、Fanyi Pu、Joshua Adrian Cahyono 和 Kairui Hu，感谢他们在评估方面的支持。
## Related Projects
## 相关项目
- [Instruction Tuning with GPT-4](https://github.com/Instruction-Tuning-with-GPT-4/GPT-4-LLM)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://github.com/microsoft/LLaVA-Med)
- [Otter: In-Context Multi-Modal Instruction Tuning](https://github.com/Luodian/Otter)
For future project ideas, please check out:
若需未来项目灵感，请查看：
- [SEEM: Segment Everything Everywhere All at Once](https://github.com/UX-Decoder/Segment-Everything-Everywhere-All-At-Once)
- [Grounded-Segment-Anything](https://github.com/IDEA-Research/Grounded-Segment-Anything) to detect, segment, and generate anything by marrying [Grounding DINO](https://github.com/IDEA-Research/GroundingDINO) and [Segment-Anything](https://github.com/facebookresearch/segment-anything).
- [Grounded-Segment-Anything](https://github.com/IDEA-Research/Grounded-Segment-Anything)：通过结合 [Grounding DINO](https://github.com/IDEA-Research/GroundingDINO) 与 [Segment-Anything](https://github.com/facebookresearch/segment-anything).，实现检测、分割与生成任意内容。