- | Model Name |
- CogVideoX1.5-5B (Latest) |
- CogVideoX1.5-5B-I2V (Latest) |
+ 模型名 |
+ CogVideoX1.5-5B (最新) |
+ CogVideoX1.5-5B-I2V (最新) |
CogVideoX-2B |
CogVideoX-5B |
- CogVideoX-5B-I2V |
+ CogVideoX-5B-I2V |
- | Release Date |
- November 8, 2024 |
- November 8, 2024 |
- August 6, 2024 |
- August 27, 2024 |
- September 19, 2024 |
+ 发布时间 |
+ 2024年11月8日 |
+ 2024年11月8日 |
+ 2024年8月6日 |
+ 2024年8月27日 |
+ 2024年9月19日 |
- | Video Resolution |
+ 视频分辨率 |
1360 * 768 |
Min(W, H) = 768
768 ≤ Max(W, H) ≤ 1360
Max(W, H) % 16 = 0 |
720 * 480 |
- | Number of Frames |
- Should be 16N + 1 where N <= 10 (default 81) |
- Should be 8N + 1 where N <= 6 (default 49) |
+ 帧数 |
+ 必须为 16N + 1 其中 N <= 10 (默认 81) |
+ 必须为 8N + 1 其中 N <= 6 (默认 49) |
- | Inference Precision |
- BF16 (Recommended), FP16, FP32, FP8*, INT8, Not supported: INT4 |
- FP16*(Recommended), BF16, FP32, FP8*, INT8, Not supported: INT4 |
- BF16 (Recommended), FP16, FP32, FP8*, INT8, Not supported: INT4 |
+ 推理精度 |
+ BF16(推荐), FP16, FP32，FP8*，INT8，不支持INT4 |
+ FP16*(推荐), BF16, FP32，FP8*，INT8，不支持INT4 |
+ BF16(推荐), FP16, FP32，FP8*，INT8，不支持INT4 |
- Single GPU Memory Usage
|
- SAT BF16: 76GB
diffusers BF16: from 10GB*
diffusers INT8(torchao): from 7GB* |
- SAT FP16: 18GB
diffusers FP16: 4GB minimum*
diffusers INT8 (torchao): 3.6GB minimum* |
- SAT BF16: 26GB
diffusers BF16 : 5GB minimum*
diffusers INT8 (torchao): 4.4GB minimum* |
+ 单GPU显存消耗
|
+ SAT BF16: 76GB
diffusers BF16 : 10GB起*
diffusers INT8(torchao): 7G起* |
+ SAT FP16: 18GB
diffusers FP16: 4GB起*
diffusers INT8(torchao): 3.6G起* |
+ SAT BF16: 26GB
diffusers BF16 : 5GB起*
diffusers INT8(torchao): 4.4G起* |
- | Multi-GPU Memory Usage |
+ 多GPU推理显存消耗 |
BF16: 24GB* using diffusers
|
FP16: 10GB* using diffusers
|
BF16: 15GB* using diffusers
|
- Inference Speed
(Step = 50, FP/BF16) |
- Single A100: ~1000 seconds (5-second video)
Single H100: ~550 seconds (5-second video) |
- Single A100: ~90 seconds
Single H100: ~45 seconds |
- Single A100: ~180 seconds
Single H100: ~90 seconds |
+ 推理速度
(Step = 50, FP/BF16) |
+ 单卡A100: ~1000秒(5秒视频)
单卡H100: ~550秒(5秒视频) |
+ 单卡A100: ~90秒
单卡H100: ~45秒 |
+ 单卡A100: ~180秒
单卡H100: ~90秒 |
- | Prompt Language |
+ 提示词语言 |
English* |
- | Prompt Token Limit |
+ 提示词长度上限 |
224 Tokens |
226 Tokens |
- | Video Length |
- 5 seconds or 10 seconds |
- 6 seconds |
+ 视频长度 |
+ 5 秒 或 10 秒 |
+ 6 秒 |
- | Frame Rate |
- 16 frames / second |
- 8 frames / second |
+ 帧率 |
+ 16 帧 / 秒 |
+ 8 帧 / 秒 |
- | Position Encoding |
+ 位置编码 |
3d_rope_pos_embed |
3d_sincos_pos_embed |
3d_rope_pos_embed |
3d_rope_pos_embed + learnable_pos_embed |
- | Download Link (Diffusers) |
+ 下载链接 (Diffusers) |
🤗 HuggingFace
🤖 ModelScope
🟣 WiseModel |
🤗 HuggingFace
🤖 ModelScope
🟣 WiseModel |
🤗 HuggingFace
🤖 ModelScope
🟣 WiseModel |
@@ -265,19 +253,17 @@ models we currently offer, along with their foundational information.
🤗 HuggingFace
🤖 ModelScope
🟣 WiseModel |
- | Download Link (SAT) |
+ 下载链接 (SAT) |
🤗 HuggingFace
🤖 ModelScope
🟣 WiseModel |
SAT |
-**Data Explanation**
+**数据解释**
-+ While testing using the diffusers library, all optimizations included in the diffusers library were enabled. This
- scheme has not been tested for actual memory usage on devices outside of **NVIDIA A100 / H100** architectures.
- Generally, this scheme can be adapted to all **NVIDIA Ampere architecture** and above devices. If optimizations are
- disabled, memory consumption will multiply, with peak memory usage being about 3 times the value in the table.
- However, speed will increase by about 3-4 times. You can selectively disable some optimizations, including:
++ 使用 diffusers 库进行测试时，启用了全部`diffusers`库自带的优化，该方案未测试在非**NVIDIA A100 / H100**
+ 外的设备上的实际显存 / 内存占用。通常，该方案可以适配于所有 **NVIDIA 安培架构**
+ 以上的设备。若关闭优化，显存占用会成倍增加，峰值显存约为表格的3倍。但速度提升3-4倍左右。你可以选择性的关闭部分优化，这些优化包括:
```
pipe.enable_sequential_cpu_offload()
@@ -285,152 +271,126 @@ pipe.vae.enable_slicing()
pipe.vae.enable_tiling()
```
-+ For multi-GPU inference, the `enable_sequential_cpu_offload()` optimization needs to be disabled.
-+ Using INT8 models will slow down inference, which is done to accommodate lower-memory GPUs while maintaining minimal
- video quality loss, though inference speed will significantly decrease.
-+ The CogVideoX-2B model was trained in `FP16` precision, and all CogVideoX-5B models were trained in `BF16` precision.
- We recommend using the precision in which the model was trained for inference.
-+ [PytorchAO](https://github.com/pytorch/ao) and [Optimum-quanto](https://github.com/huggingface/optimum-quanto/) can be
- used to quantize the text encoder, transformer, and VAE modules to reduce the memory requirements of CogVideoX. This
- allows the model to run on free T4 Colabs or GPUs with smaller memory! Also, note that TorchAO quantization is fully
- compatible with `torch.compile`, which can significantly improve inference speed. FP8 precision must be used on
- devices with NVIDIA H100 and above, requiring source installation of `torch`, `torchao` Python packages. CUDA 12.4 is recommended.
-+ The inference speed tests also used the above memory optimization scheme. Without memory optimization, inference speed
- increases by about 10%. Only the `diffusers` version of the model supports quantization.
-+ The model only supports English input; other languages can be translated into English for use via large model
- refinement.
++ 多GPU推理时，需要关闭 `enable_sequential_cpu_offload()` 优化。
++ 使用 INT8 模型会导致推理速度降低，此举是为了满足显存较低的显卡能正常推理并保持较少的视频质量损失，推理速度大幅降低。
++ CogVideoX-2B 模型采用 `FP16` 精度训练， 搜有 CogVideoX-5B 模型采用 `BF16` 精度训练。我们推荐使用模型训练的精度进行推理。
++ [PytorchAO](https://github.com/pytorch/ao) 和 [Optimum-quanto](https://github.com/huggingface/optimum-quanto/)
+ 可以用于量化文本编码器、Transformer 和 VAE 模块，以降低 CogVideoX 的内存需求。这使得在免费的 T4 Colab 或更小显存的 GPU
+ 上运行模型成为可能！同样值得注意的是，TorchAO 量化完全兼容 `torch.compile`，这可以显著提高推理速度。在 `NVIDIA H100`
+ 及以上设备上必须使用 `FP8` 精度，这需要源码安装 `torch`、`torchao` Python 包。建议使用 `CUDA 12.4`。
++ 推理速度测试同样采用了上述显存优化方案，不采用显存优化的情况下，推理速度提升约10%。 只有`diffusers`版本模型支持量化。
++ 模型仅支持英语输入，其他语言可以通过大模型润色时翻译为英语。
+## 友情链接
-## Friendly Links
+我们非常欢迎来自社区的贡献，并积极的贡献开源社区。以下作品已经对CogVideoX进行了适配，欢迎大家使用:
-We highly welcome contributions from the community and actively contribute to the open-source community. The following
-works have already been adapted for CogVideoX, and we invite everyone to use them:
-
-+ [LeMiCa](https://unicomai.github.io/LeMiCa/): a diffusion model inference acceleration solution developed by China Unicom Data Science and Artificial Intelligence Research Institute. By leveraging cache-based techniques and global denoising path optimization, LeMiCa provides efficient inference support for CogVideoX, achieving nearly 2.5x lossless acceleration while maintaining visual consistency and quality.
++ [LeMiCa](https://unicomai.github.io/LeMiCa/): 由中国联通数据科学与人工智能研究院开发的扩散模型推理加速解决方案。它利用基于缓存的技术和全局去噪路径优化，为CogVideoX提供高效的推理支持，在保持视觉一致性和质量的前提下，实现了近2.5倍的无损加速。
+ [RIFLEx-CogVideoX](https://github.com/thu-ml/RIFLEx):
- RIFLEx extends the video with just one line of code: `freq[k-1]=(2np.pi)/(Ls)`. The framework not only supports training-free inference, but also offers models fine-tuned based on CogVideoX. By fine-tuning the model for just 1,000 steps on original-length videos, RIFLEx significantly enhances its length extrapolation capability.
-+ [CogVideoX-Fun](https://github.com/aigc-apps/CogVideoX-Fun): CogVideoX-Fun is a modified pipeline based on the
- CogVideoX architecture, supporting flexible resolutions and multiple launch methods.
-+ [CogStudio](https://github.com/pinokiofactory/cogstudio): A separate repository for CogVideo's Gradio Web UI, which
- supports more functional Web UIs.
-+ [Xorbits Inference](https://github.com/xorbitsai/inference): A powerful and comprehensive distributed inference
- framework, allowing you to easily deploy your own models or the latest cutting-edge open-source models with just one
- click.
-+ [ComfyUI-CogVideoXWrapper](https://github.com/kijai/ComfyUI-CogVideoXWrapper) Use the ComfyUI framework to integrate
- CogVideoX into your workflow.
-+ [VideoSys](https://github.com/NUS-HPC-AI-Lab/VideoSys): VideoSys provides a user-friendly, high-performance
- infrastructure for video generation, with full pipeline support and continuous integration of the latest models and
- techniques.
-+ [AutoDL Space](https://www.codewithgpu.com/i/THUDM/CogVideo/CogVideoX-5b-demo): A one-click deployment Huggingface
- Space image provided by community members.
-+ [Interior Design Fine-Tuning Model](https://huggingface.co/collections/bertjiazheng/koolcogvideox-66e4762f53287b7f39f8f3ba):
- is a fine-tuned model based on CogVideoX, specifically designed for interior design.
-+ [xDiT](https://github.com/xdit-project/xDiT): xDiT is a scalable inference engine for Diffusion Transformers (DiTs)
- on multiple GPU Clusters. xDiT supports real-time image and video generations services.
- [cogvideox-factory](https://github.com/a-r-r-o-w/cogvideox-factory): A cost-effective
- fine-tuning framework for CogVideoX, compatible with the `diffusers` version model. Supports more resolutions, and
- fine-tuning CogVideoX-5B can be done with a single 4090 GPU.
-+ [CogVideoX-Interpolation](https://github.com/feizc/CogvideX-Interpolation): A pipeline based on the modified CogVideoX
- structure, aimed at providing greater flexibility for keyframe interpolation generation.
-+ [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio): DiffSynth Studio is a diffusion engine. It has
- restructured the architecture, including text encoders, UNet, VAE, etc., enhancing computational performance while
- maintaining compatibility with open-source community models. The framework has been adapted for CogVideoX.
-+ [CogVideoX-Controlnet](https://github.com/TheDenk/cogvideox-controlnet): A simple ControlNet module code that includes the CogVideoX model.
-+ [VideoTuna](https://github.com/VideoVerses/VideoTuna): VideoTuna is the first repo that integrates multiple AI video generation models for text-to-video, image-to-video, text-to-image generation.
-+ [ConsisID](https://github.com/PKU-YuanGroup/ConsisID): An identity-preserving text-to-video generation model, bases on CogVideoX-5B, which keep the face consistent in the generated video by frequency decomposition.
-+ [A Step by Step Tutorial](https://www.youtube.com/watch?v=5UCkMzP2VLE&ab_channel=SECourses): A step-by-step guide on installing and optimizing the CogVideoX1.5-5B-I2V model in Windows and cloud environments. Special thanks to the [FurkanGozukara](https://github.com/FurkanGozukara) for his effort and support!
+ RIFLEx 是一个视频长度外推的方法，只需一行代码即可将视频生成长度延伸为原先的二倍。RIFLEx 不仅支持 Training-free 的推理，也提供基于 CogVideoX 进行微调的模型，只需在原有长度视频上微调 1000 步即可大大提高长度外推能力。
++ [CogVideoX-Fun](https://github.com/aigc-apps/CogVideoX-Fun):
+ CogVideoX-Fun是一个基于CogVideoX结构修改后的的pipeline，支持自由的分辨率，多种启动方式。
++ [CogStudio](https://github.com/pinokiofactory/cogstudio): CogVideo 的 Gradio Web UI单独实现仓库，支持更多功能的 Web UI。
++ [Xorbits Inference](https://github.com/xorbitsai/inference): 性能强大且功能全面的分布式推理框架，轻松一键部署你自己的模型或内置的前沿开源模型。
++ [ComfyUI-CogVideoXWrapper](https://github.com/kijai/ComfyUI-CogVideoXWrapper) 使用ComfyUI框架，将CogVideoX加入到你的工作流中。
++ [VideoSys](https://github.com/NUS-HPC-AI-Lab/VideoSys): VideoSys 提供了易用且高性能的视频生成基础设施，支持完整的管道，并持续集成最新的模型和技术。
++ [AutoDL镜像](https://www.codewithgpu.com/i/THUDM/CogVideo/CogVideoX-5b-demo): 由社区成员提供的一键部署Huggingface
+ Space镜像。
++ [室内设计微调模型](https://huggingface.co/collections/bertjiazheng/koolcogvideox-66e4762f53287b7f39f8f3ba) 基于
+ CogVideoX的微调模型，它专为室内设计而设计
++ [xDiT](https://github.com/xdit-project/xDiT): xDiT是一个用于在多GPU集群上对DiTs并行推理的引擎。xDiT支持实时图像和视频生成服务。
++ [CogVideoX-Interpolation](https://github.com/feizc/CogvideX-Interpolation): 基于 CogVideoX 结构修改的管道，旨在为关键帧插值生成提供更大的灵活性。
++ [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio): DiffSynth 工作室是一款扩散引擎。重构了架构，包括文本编码器、UNet、VAE
+ 等，在保持与开源社区模型兼容性的同时，提升了计算性能。该框架已经适配 CogVideoX。
++ [CogVideoX-Controlnet](https://github.com/TheDenk/cogvideox-controlnet): 一个包含 CogvideoX 模型的简单 Controlnet 模块的代码。
++ [VideoTuna](https://github.com/VideoVerses/VideoTuna)：VideoTuna 是首个集成多种 AI 视频生成模型的仓库，支持文本转视频、图像转视频、文本转图像生成。
++ [ConsisID](https://github.com/PKU-YuanGroup/ConsisID): 一种身份保持的文本到视频生成模型，基于 CogVideoX-5B，通过频率分解在生成的视频中保持面部一致性。
++ [教程](https://www.youtube.com/watch?v=5UCkMzP2VLE&ab_channel=SECourses): 一个关于在Windows和云环境中安装和优化CogVideoX1.5-5B-I2V模型的分步指南。特别感谢[FurkanGozukara](https://github.com/FurkanGozukara)的努力和支持！
-## Project Structure
-This open-source repository will guide developers to quickly get started with the basic usage and fine-tuning examples
-of the **CogVideoX** open-source model.
+## 完整项目代码结构
-### Quick Start with Colab
+本开源仓库将带领开发者快速上手 **CogVideoX** 开源模型的基础调用方式、微调示例。
-Here provide three projects that can be run directly on free Colab T4 instances:
+### Colab 快速使用
+
+这里提供了三个能直接在免费的 Colab T4上 运行的项目
+ [CogVideoX-5B-T2V-Colab.ipynb](https://colab.research.google.com/drive/1pCe5s0bC_xuXbBlpvIH1z0kfdTLQPzCS?usp=sharing):
- CogVideoX-5B Text-to-Video Colab code.
+ CogVideoX-5B 文字生成视频 Colab 代码。
+ [CogVideoX-5B-T2V-Int8-Colab.ipynb](https://colab.research.google.com/drive/1DUffhcjrU-uz7_cpuJO3E_D4BaJT7OPa?usp=sharing):
- CogVideoX-5B Quantized Text-to-Video Inference Colab code, which takes about 30 minutes per run.
+ CogVideoX-5B 文字生成视频量化推理 Colab 代码，运行一次大约需要30分钟。
+ [CogVideoX-5B-I2V-Colab.ipynb](https://colab.research.google.com/drive/17CqYCqSwz39nZAX2YyonDxosVKUZGzcX?usp=sharing):
- CogVideoX-5B Image-to-Video Colab code.
+ CogVideoX-5B 图片生成视频 Colab 代码。
+ [CogVideoX-5B-V2V-Colab.ipynb](https://colab.research.google.com/drive/1comfGAUJnChl5NwPuO8Ox5_6WCy4kbNN?usp=sharing):
- CogVideoX-5B Video-to-Video Colab code.
+ CogVideoX-5B 视频生成视频 Colab 代码。
-### Inference
+### inference
-+ [dcli_demo](inference/cli_demo.py): A more detailed inference code explanation, including the significance of
- common parameters. All of this is covered here.
++ [cli_demo](inference/cli_demo.py): 更详细的推理代码讲解，常见参数的意义，在这里都会提及。
+ [cli_demo_quantization](inference/cli_demo_quantization.py):
- Quantized model inference code that can run on devices with lower memory. You can also modify this code to support
- running CogVideoX models in FP8 precision.
-+ [diffusers_vae_demo](inference/cli_vae_demo.py): Code for running VAE inference separately.
-+ [space demo](inference/gradio_composite_demo): The same GUI code as used in the Huggingface Space, with frame
- interpolation and super-resolution tools integrated.
+ 量化模型推理代码，可以在显存较低的设备上运行，也可以基于此代码修改，以支持运行FP8等精度的CogVideoX模型。请注意，FP8
+ 仅测试通过，且必须将 `torch-nightly`,`torchao`源代码安装，不建议在生产环境中使用。
++ [diffusers_vae_demo](inference/cli_vae_demo.py): 单独执行VAE的推理代码。
++ [space demo](inference/gradio_composite_demo): Huggingface Space同款的 GUI 代码，植入了插帧，超分工具。
