wehub/nvlabs--longlive
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

docs: make Chinese README the default

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 10:21:25 +00:00
parent 4b34546fe4
commit 299f72de6e
1 changed files with 87 additions and 81 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+87 -81
View File
@@ -1,8 +1,14 @@
<!-- WEHUB_ZH_README -->
> [!NOTE]
> 本文档由 WeHub 基于上游 README 翻译整理，属于社区翻译，非官方中文文档。
> [English](./README.en.md) · [原始项目](https://github.com/NVlabs/LongLive) · [上游 README](https://github.com/NVlabs/LongLive/blob/HEAD/README.md)
> 原作者、版权与许可证归属以原始项目及本仓库 LICENSE 文件为准。
<p align="center" style="border-radius: 10px">
<img src="assets/longlive2/logo.png" width="100%" alt="LongLive2.0 logo"/>
</p>
# 🎬 LongLive 2.0: An NVFP4 Parallel Infrastructure for Long Video Generation
# 🎬 LongLive 2.0：面向长视频生成的 NVFP4 并行基础设施
[![Paper](https://img.shields.io/badge/Paper-LongLive_2.0-brown)](https://arxiv.org/abs/2605.18739)
[![Paper](https://img.shields.io/badge/Paper-LongLive_1.0-orange)](https://github.com/NVlabs/LongLive/tree/v1.0)
@@ -21,39 +27,39 @@
</div>
## 💡 TLDR: Infra with NVFP4 and parallelism for both training and inference
## 💡 TLDR：面向训练与推理的 NVFP4 并行基础设施
<p align="center" style="border-radius: 10px">
<img src="assets/longlive2/teaser.jpg" width="100%" alt="LongLive2.0 teaser"/>
</p>
## News
- 🔥 [2026.07.08] We support FP8 inference on LongLive 2.0. Please refer to [here](https://github.com/NVlabs/LongLive#fp8-ptq).
- 🔥 [2026.06.01] We released [LongLive-RAG](https://github.com/qixinhu11/LongLive-RAG), a general retrieval-augmented framework for long video gen.
- 🔥 [2026.05.30] LongLive2.0 now supports I2V AR teacher-forcing training and I2V DMD distillation for Wan2.2-TI2V-5B.
- ⚡ [2026.05.25] We optimized the NVFP4 inference path with fused Triton RoPE/adaLN kernels, reduced KV-cache synchronization overhead, in-place quantized KV-cache updates, faster FP4 KV dequantization, pinned VAE transfers, and safer LoRA-before-quantization setup, improving overall throughput by **18.6%**.
- 🔥 [2026.05.13] We release **LongLive 2.0**, infra with NVFP4, parallelism and multi-shot for AR training, DMD distillation, and inference (⚡45.7 FPS). The original LongLive 1.0 is now in the [v1.0](https://github.com/NVlabs/LongLive/tree/v1.0) branch.
- 🔥 [2026.04.12] LongLive supports kv cache compression with [TriAttention](https://github.com/WeianMao/triattention), with 50% KV reduction and no quality drop. Check it [here](https://github.com/WeianMao/triattention/tree/main/longlive)
- 🎉 [2026.1.27] LongLive is accepted by **ICLR-2026**.
- 🔥 [2026.1.11] LongLive supports adapting LongLive's original RoPE into KV-cache relative RoPE and generates infinite long videos!
- 🔥 [2025.11.3] We implement LongLive on linear attention model [SANA-Video](https://nvlabs.github.io/Sana/Video/)! Now SANA-Video can generate 60s interactive videos in real-time.
- 🔥 [2025.9.29] We release [Paper](https://arxiv.org/abs/2509.22622), this GitHub repo [LongLive](https://github.com/NVlabs/LongLive) with all training and inference code, the model weight [LongLive-1.3B](https://huggingface.co/Efficient-Large-Model/LongLive-1.3B), and demo page [Website](https://nvlabs.github.io/LongLive).
- 🔥 [2026.07.08] LongLive 2.0 现已支持 FP8 推理。请参阅[此处](https://github.com/NVlabs/LongLive#fp8-ptq).
- 🔥 [2026.06.01] 我们发布了 [LongLive-RAG](https://github.com/qixinhu11/LongLive-RAG),，这是一个面向长视频生成的通用检索增强（Retrieval-Augmented Generation, RAG）框架。
- 🔥 [2026.05.30] LongLive2.0 现已支持 Wan2.2-TI2V-5B 的 I2V AR 教师强制（teacher-forcing）训练与 I2V DMD 蒸馏。
- ⚡ [2026.05.25] 我们优化了 NVFP4 推理路径：融合 Triton RoPE/adaLN 内核、降低 KV-cache 同步开销、原地量化 KV-cache 更新、更快的 FP4 KV 反量化、固定（pinned）VAE 传输，以及更安全的量化前 LoRA 配置，整体吞吐量提升 **18.6%**。
- 🔥 [2026.05.13] 我们发布 **LongLive 2.0**，这是一款支持 NVFP4、并行化与多镜头（multi-shot）的 AR 训练、DMD 蒸馏及推理（⚡45.7 FPS）的基础设施。原版 LongLive 1.0 现位于 [v1.0](https://github.com/NVlabs/LongLive/tree/v1.0) 分支。
- 🔥 [2026.04.12] LongLive 支持通过 [TriAttention](https://github.com/WeianMao/triattention), 进行 kv cache 压缩，KV 减少 50% 且质量无损。请查看[此处](https://github.com/WeianMao/triattention/tree/main/longlive)
- 🎉 [2026.1.27] LongLive 已被 **ICLR-2026** 接收。
- 🔥 [2026.1.11] LongLive 支持将 LongLive 原始 RoPE 适配为 KV-cache 相对 RoPE，并可生成无限长视频！
- 🔥 [2025.11.3] 我们已在线性注意力模型 [SANA-Video](https://nvlabs.github.io/Sana/Video/)! 上实现 LongLive。现在 SANA-Video 可实时生成 60 秒交互式视频。
- 🔥 [2025.9.29] 我们发布了 [Paper](https://arxiv.org/abs/2509.22622),、本 GitHub 仓库 [LongLive](https://github.com/NVlabs/LongLive)（包含全部训练与推理代码）、模型权重 [LongLive-1.3B](https://huggingface.co/Efficient-Large-Model/LongLive-1.3B), 以及演示页面 [Website](https://nvlabs.github.io/LongLive).
## Introduction
**LongLive 1.0**: Real-time Interactive Long Video Generation. [You can find it here](https://github.com/NVlabs/LongLive/tree/v1.0) in our V1.0 branch.
**LongLive 1.0**：实时交互式长视频生成。[可在此处查看](https://github.com/NVlabs/LongLive/tree/v1.0)（位于我们的 V1.0 分支）。
**LongLive 2.0**: an NVFP4 Parallel Infrastructure for Long Video Generation
- For training, it supports
- [x] Balanced sequence parallel for T2V/I2V AR training (teacher-forcing).
- [x] T2V/I2V AR training on multi-shot (or single-shot) videos.
- [x] NVFP4 (or BF16) for both AR training and few-step distillation.
- For inference, it supports
- [x] NVFP4 inference (W4A4) and NVFP4 KV Cache.
- [x] TorchAO FP8 PTQ inference (W8A8) from the BF16 checkpoint.
- [x] Multi-shot attention sink.
- [x] Sequence parallel inference.
- [x] Async decoding.
**LongLive 2.0**：面向长视频生成的 NVFP4 并行基础设施
- 训练方面，支持
- [x] T2V/I2V AR 训练（教师强制）的均衡序列并行（Balanced Sequence Parallel）。
- [x] 在多镜头（或单镜头）视频上进行 T2V/I2V AR 训练。
- [x] AR 训练与少步蒸馏均支持 NVFP4（或 BF16）。
- 推理方面，支持
- [x] NVFP4 推理（W4A4）与 NVFP4 KV Cache。
- [x] 基于 BF16 checkpoint 的 TorchAO FP8 PTQ 推理（W8A8）。
- [x] 多镜头 attention sink。
- [x] 序列并行推理。
- [x] 异步解码。
<p align="left" style="border-radius: 10px">
@@ -61,7 +67,7 @@
</p>
**LongLive 1.0**: Real-time Interactive Long Video Generation. It accepts sequential user prompts and generates corresponding videos in real time, enabling user-guided long video generation. The key insights are attention sink, KV-recache, and streaming long tuning.
**LongLive 1.0**：实时交互式长视频生成。它接收顺序用户提示词，并实时生成对应视频，从而实现用户引导的长视频生成。其核心思路包括 attention sink、KV-recache 与流式长时微调（streaming long tuning）。
<p align="left" style="border-radius: 10px">
@@ -69,15 +75,15 @@
</p>
## Getting Started
- [Full Documentation](https://nvlabs.github.io/LongLive/LongLive2/docs/)
- [Installation](https://nvlabs.github.io/LongLive/LongLive2/docs/#installation)
- [NVFP4 Setup](https://nvlabs.github.io/LongLive/LongLive2/docs/#nvfp4-installation)
- [Training Modes](https://nvlabs.github.io/LongLive/LongLive2/docs/#training)
- [Inference](https://nvlabs.github.io/LongLive/LongLive2/docs/#inference)
- [Data Organization](https://nvlabs.github.io/LongLive/LongLive2/docs/#training-data)
- [完整文档](https://nvlabs.github.io/LongLive/LongLive2/docs/)
- [安装](https://nvlabs.github.io/LongLive/LongLive2/docs/#installation)
- [NVFP4 配置](https://nvlabs.github.io/LongLive/LongLive2/docs/#nvfp4-installation)
- [训练模式](https://nvlabs.github.io/LongLive/LongLive2/docs/#training)
- [推理](https://nvlabs.github.io/LongLive/LongLive2/docs/#inference)
- [数据组织](https://nvlabs.github.io/LongLive/LongLive2/docs/#training-data)
The default git clone fetches objects from all branches, including our demopage branch, which contains large assets. For normal use, only the main branch is needed. Please clone only main with:
默认 `git clone` 会拉取所有分支的对象，包括包含大型资源的 demopage 分支。日常使用通常只需 main 分支。请仅克隆 main：
```git clone --single-branch --branch main --depth 1 https://github.com/NVlabs/LongLive.git```
@@ -116,43 +122,43 @@ video = pipe.inference(noise=noise, text_prompts=prompts)
save_video(video[0], "videos/quickstart/sample.mp4", fps=24)
```
`place_vae_for_streaming` is a no-op unless `inference.streaming_vae` is true and `inference.vae_device` is set, so toggling streaming-pipeline decode in your yaml is enough — the script does not need to change.
除非 `inference.streaming_vae` 为 true 且已设置 `inference.vae_device`，否则 `place_vae_for_streaming` 为无操作（no-op）；因此只需在 yaml 中切换 streaming-pipeline 解码即可，无需修改脚本。
#### FP8 PTQ
Download `model_bf16.pt` from
从
[`Efficient-Large-Model/LongLive-2.0-5B`](https://huggingface.co/Efficient-Large-Model/LongLive-2.0-5B),
set `checkpoints.generator_ckpt` in `configs/fp8/inference_fp8.yaml`, and run:
下载 `model_bf16.pt`，
在 `configs/fp8/inference_fp8.yaml` 中设置 `checkpoints.generator_ckpt`，然后运行：
```bash
python inference.py --config_path configs/fp8/inference_fp8.yaml
```
This loads the BF16 generator, applies TorchAO row-wise dynamic FP8 W8A8 PTQ,
and then enables the existing `torch.compile` path. With the provided 5B model,
300 eligible core Linear layers use FP8; six small conditioning/output
projections stay in BF16 for stability and to avoid FP8 overhead.
这将加载 BF16 生成器，应用 TorchAO 按行动态 FP8 W8A8 PTQ，
然后启用现有的 `torch.compile` 路径。对于提供的 5B 模型，
300 个符合条件的核心 Linear 层使用 FP8；六个较小的条件/输出
投影层保持 BF16，以确保稳定性并避免 FP8 开销。
The validated stack is Python 3.10, PyTorch 2.8.0+cu128, and TorchAO 0.13.0 on
H100 (SM90); compute capability 8.9 or newer is required. The supplied config
uses `torch_compile: auto`. Its `max-autotune` warm-up can take several minutes
while guard/shape variants are compiled, so use repeated inference and exclude
all compile/warm-up samples when measuring steady-state performance. Set
`torch_compile: false` for a short eager-mode smoke test.
经验证的技术栈为 Python 3.10、PyTorch 2.8.0+cu128 与 TorchAO 0.13.0（运行于
H100（SM90））；需要计算能力 8.9 或更高。提供的配置
使用 `torch_compile: auto`。其 `max-autotune` 预热可能需要数分钟，
因为会编译 guard/shape 变体，因此测量稳态性能时请重复推理，并排除
所有编译/预热样本。设置
`torch_compile: false` 可进行简短的 eager 模式冒烟测试。
The supplied config uses the single 8-latent-frame block validated on H100.
Longer generation introduces additional KV-cache shapes and may trigger more
compilation or eager fallback; validate the intended frame count before
benchmarking or deployment.
提供的配置使用在 H100 上验证过的单 8-latent-frame 块。
更长生成会引入额外的 KV-cache 形状，并可能触发更多
编译或 eager 回退；在基准测试或部署前，请先验证目标帧数。
#### NVFP4
Point `checkpoints.generator_ckpt` in `configs/nvfp4/inference_nvfp4.yaml` at the downloaded checkpoint and set `model_quant_use_transformer_engine` according to the backend you are using:
在 `configs/nvfp4/inference_nvfp4.yaml` 中将 `checkpoints.generator_ckpt` 指向已下载的 checkpoint，并根据所用后端设置 `model_quant_use_transformer_engine`：
- TransformerEngine checkpoint (`model_te.pt`): `model_quant_use_transformer_engine: true`
- FourOverSix checkpoint (`model_4o6.pt`): `model_quant_use_transformer_engine: false`
- TransformerEngine checkpoint（`model_te.pt`）：`model_quant_use_transformer_engine: true`
- FourOverSix checkpoint（`model_4o6.pt`）：`model_quant_use_transformer_engine: false`
`setup_nvfp4_pipeline` handles checkpoint loading, NVFP4 module wrapping, weight materialization, dtype/device placement, and the streaming-pipeline VAE relocation for both backends — the bf16 `pipe.to(...)` shortcut is unsafe here because it would cast the quantized buffers.
`setup_nvfp4_pipeline` 负责 checkpoint 加载、NVFP4 模块封装、权重物化、dtype/设备放置，以及两种后端的 streaming-pipeline VAE 迁移——此处 bf16 `pipe.to(...)` 快捷方式不安全，因为它会强制转换量化缓冲区。
```python
import torch
@@ -197,7 +203,7 @@ torchrun --standalone --nnodes=1 --nproc_per_node=8 train.py \
--disable-wandb
```
### I2V Training
### I2V 训练
```bash
torchrun --standalone --nnodes=1 --nproc_per_node=8 train.py \
@@ -213,42 +219,42 @@ torchrun --standalone --nnodes=1 --nproc_per_node=8 train.py \
--disable-wandb
```
For I2V configs, set `algorithm.i2v: true` and `algorithm.independent_first_frame: true`. `data.image_or_video_shape[1]` is the full latent sequence length, for example `96`, not `96 + 1`: the clean image latent replaces the first latent during denoising and that first latent is masked out of the training loss. For I2V DMD, set `checkpoints.generator_ckpt` to the I2V AR checkpoint used to initialize the student.
对于 I2V 配置，请设置 `algorithm.i2v: true` 和 `algorithm.independent_first_frame: true`。`data.image_or_video_shape[1]` 是完整 latent 序列长度，例如 `96`，而非 `96 + 1`：在去噪过程中，干净的图像 latent 会替换第一个 latent，且该第一个 latent 会从训练损失中掩蔽掉。对于 I2V DMD，请将 `checkpoints.generator_ckpt` 设置为用于初始化 student 的 I2V AR 检查点。
## Models
## 模型
| Model | FPS ↑ | Params | VBench ↑ | Multi-shot |
| 模型 | FPS ↑ | 参数量 | VBench ↑ | 多镜头 |
| --- | ---: | ---: | ---: | :---: |
| [LongLive-1.3B](https://huggingface.co/Efficient-Large-Model/LongLive-1.3B) | 20.7 | 1.3B | 84.87 | |
| [LongLive-2.0-5B](https://huggingface.co/Efficient-Large-Model/LongLive-2.0-5B) | 24.8 | 5B | 85.06 | ✅ |
| [LongLive-2.0-5B-NVFP4-4Step](https://huggingface.co/Efficient-Large-Model/LongLive-2.0-5B-NVFP4-S4) | 29.7 | 5B | 84.51 | ✅ |
| [LongLive-2.0-5B-NVFP4-2Step](https://huggingface.co/Efficient-Large-Model/LongLive-2.0-5B-NVFP4-S2) | 45.7 | 5B | 83.14 | ✅ |
## Awesome work using LongLive
## 基于 LongLive 的优秀工作
- [DreamForge-World 0.1](https://trydreamforge.com/): Adapts the LongLive AR video stack with a residual action pathway for low-compute real-time controllable world modeling.
- [DreamX-World 1.0](https://arxiv.org/abs/2606.16993): Follows LongLive by adapting the model on long sequences with long rollouts and local temporal windows for stable long-horizon AR world generation.
- [SANA-Video](https://nvlabs.github.io/Sana/docs/longsana/): Combines SANA-Video with LongLive to build LongSANA, a real-time minute-long video generation variant with constant-memory KV cache.
- [Daydream Scope](https://docs.daydream.live/scope/reference/pipelines/longlive): Wraps LongLive as a streaming AR video diffusion pipeline for interactive text-to-video and video-to-video workflows.
- [MemFlow](https://github.com/KlingAIResearch/MemFlow): Builds on the LongLive codebase and adds adaptive memory retrieval for more consistent long narrative video generation.
- [ShotStream](https://github.com/KlingAIResearch/ShotStream): Builds on LongLive’s distillation procedure for real-time streaming multi-shot AR video generation.
- [Stream-T1](https://github.com/FrameX-AI/Stream-T1): Builds on LongLive's codebase and algorithm, adding test-time scaling with noise propagation, reward pruning, and memory sinking.
- [KVPO](https://github.com/Richard-Zhang-AI/KVPO): Builds on LongLive and related AR video codebases to perform GRPO-style alignment through historical KV semantic exploration.
- [LoL](https://github.com/justincui03/LoL): Builds on LongLive to study and mitigate sink-collapse for ultra-long AR streaming video generation.
- [TriAttention](https://github.com/WeianMao/triattention/tree/main/longlive): Integrates trigonometric KV-cache compression into LongLive's causal inference pipeline, reducing KV memory inside LongLive's local-attention window.
- [StreamEdit](https://github.com/DSL-Lab/StreamEdit): Provides a `LongLive_StreamEdit` implementation for training-free streaming video editing built on the LongLive v1.0 codebase.
- [Streaming Autoregressive Video Generation via Diagonal Distillation](https://github.com/Sphere-AI-Lab/diagdistill): Builds on the LongLive codebase and supports direct initialization from `LongLive-1.3B` checkpoints for streaming AR video distillation.
- [Forcing-KV](https://github.com/zju-jiyicheng/Forcing-KV): Adds hybrid KV-cache compression to LongLive, including LongLive inference and interactive-generation scripts.
- [Dummy Forcing](https://github.com/csguoh/DummyForcing): Unifies Self-Forcing, LongLive, and Causal-Forcing pipelines with LongLive inference, VBench, and interactive-generation configs.
- [MemRoPE](https://github.com/YoungRaeKimm/MemRoPE): Uses LongLive as a supported base model for training-free infinite video generation with evolving memory tokens.
- [Astrolabe](https://github.com/franklinz233/Astrolabe): Supports LongLive as a distilled autoregressive video backbone with LongLive-specific RL configs and LoRA initialization.
- [DreamForge-World 0.1](https://trydreamforge.com/): 在 LongLive AR 视频技术栈基础上，通过残差动作通路实现低算力、实时可控的世界建模。
- [DreamX-World 1.0](https://arxiv.org/abs/2606.16993): 沿袭 LongLive，通过在长序列上进行长 rollout 与局部时间窗口适配，实现稳定的长时域 AR 世界生成。
- [SANA-Video](https://nvlabs.github.io/Sana/docs/longsana/): 将 SANA-Video 与 LongLive 结合，构建具备恒定内存 KV cache 的实时分钟级视频生成变体 LongSANA。
- [Daydream Scope](https://docs.daydream.live/scope/reference/pipelines/longlive): 将 LongLive 封装为流式 AR 视频扩散流水线，用于交互式文生视频与视频生视频工作流。
- [MemFlow](https://github.com/KlingAIResearch/MemFlow): 基于 LongLive 代码库，加入自适应记忆检索，以生成更一致的长叙事视频。
- [ShotStream](https://github.com/KlingAIResearch/ShotStream): 基于 LongLive 的蒸馏流程，实现实时流式多镜头 AR 视频生成。
- [Stream-T1](https://github.com/FrameX-AI/Stream-T1): 基于 LongLive 代码库与算法，加入测试时扩展（test-time scaling）、噪声传播、奖励剪枝与记忆下沉（memory sinking）。
- [KVPO](https://github.com/Richard-Zhang-AI/KVPO): 基于 LongLive 及相关 AR 视频代码库，通过历史 KV 语义探索实现 GRPO 风格对齐。
- [LoL](https://github.com/justincui03/LoL): 基于 LongLive 研究并缓解 sink-collapse 问题，以支持超长 AR 流式视频生成。
- [TriAttention](https://github.com/WeianMao/triattention/tree/main/longlive): 将三角 KV-cache 压缩集成进 LongLive 因果推理流水线，降低其局部注意力窗口内的 KV 内存占用。
- [StreamEdit](https://github.com/DSL-Lab/StreamEdit): 基于 LongLive v1.0 代码库，提供 `LongLive_StreamEdit` 实现，用于免训练流式视频编辑。
- [Streaming Autoregressive Video Generation via Diagonal Distillation](https://github.com/Sphere-AI-Lab/diagdistill): 基于 LongLive 代码库，支持从 `LongLive-1.3B` 检查点直接初始化，用于流式 AR 视频蒸馏。
- [Forcing-KV](https://github.com/zju-jiyicheng/Forcing-KV): 为 LongLive 添加混合 KV-cache 压缩，包含 LongLive 推理与交互式生成脚本。
- [Dummy Forcing](https://github.com/csguoh/DummyForcing): 统一 Self-Forcing、LongLive 与 Causal-Forcing 流水线，并提供 LongLive 推理、VBench 与交互式生成配置。
- [MemRoPE](https://github.com/YoungRaeKimm/MemRoPE): 以 LongLive 为支持的基础模型，实现免训练无限视频生成，并采用演化的记忆 token。
- [Astrolabe](https://github.com/franklinz233/Astrolabe): 支持将 LongLive 作为蒸馏后的自回归视频骨干，并提供 LongLive 专用 RL 配置与 LoRA 初始化。
## License
This repository is released under the Apache 2.0 license. See [LICENSE](LICENSE) for details.
## 许可证
本仓库基于 Apache 2.0 许可证发布。详情请参阅 [LICENSE](LICENSE)。
## Citation
Please consider citing our work if you find them useful:
## 引用
若您觉得我们的工作有帮助，欢迎引用：
```bibtex
@article{longlive_2.0,
@@ -277,6 +283,6 @@ Please consider citing our work if you find them useful:
}
```
## Acknowledgement
- [Self-Forcing](https://github.com/guandeh17/Self-Forcing): the AR training codebase and formulation we build upon.
- [Wan2.2](https://github.com/Wan-Video/Wan2.2): the base video diffusion model components used in this release.
## 致谢
- [Self-Forcing](https://github.com/guandeh17/Self-Forcing): 我们在此基础上构建的 AR 训练代码库与方法论。
- [Wan2.2](https://github.com/Wan-Video/Wan2.2): 本版本所使用的基座视频扩散模型组件。