wehub-resource-sync
45 lines
2.3 KiB
Markdown
45 lines
2.3 KiB
Markdown
# LongCat Video backend
|
|
|
|
This backend serves Meituan's `LongCat-Video` and
|
|
`LongCat-Video-Avatar-1.5` checkpoints through LocalAI's `GenerateVideo`
|
|
RPC. It supports:
|
|
|
|
- text-to-video and image-to-video with `LongCat-Video`;
|
|
- audio + text-to-avatar and portrait + audio-to-avatar with Avatar 1.5;
|
|
- multi-segment avatar continuation for speech longer than one segment;
|
|
- PyTorch SDPA when FlashAttention is unavailable, including CUDA 13 ARM64
|
|
systems such as NVIDIA DGX Spark.
|
|
|
|
Install the `longcat-video` or `longcat-video-avatar-1.5` recipe from the
|
|
LocalAI Model Gallery. See the [LongCat user guide](../../../docs/content/features/longcat-video.md)
|
|
for Studio and API examples, hardware requirements, and manual configuration.
|
|
|
|
The upstream source is pinned in `Makefile` and patched at build time. The
|
|
patch adds only the missing SDPA attention branches; model and source licenses
|
|
remain MIT.
|
|
|
|
## Model options
|
|
|
|
| Option | Default | Description |
|
|
| --- | --- | --- |
|
|
| `attention_backend` | `sdpa` | `sdpa`, `auto`, `flash2`, `flash3`, or `xformers`. The packaged backend guarantees only `sdpa`. |
|
|
| `use_distill` | `true` for Avatar, `false` for base | Loads the checkpoint's fast distillation LoRA. |
|
|
| `use_int8` | `false` | Loads Avatar 1.5's INT8 DiT. BF16 has a lower load-time peak on unified-memory systems. |
|
|
| `base_model` | `meituan-longcat/LongCat-Video` | Base components used by Avatar 1.5. |
|
|
| `max_segments` | `8` | Maximum avatar continuation segments accepted per request. |
|
|
| `resolution` | `480p` | Image-conditioned generation resolution (`480p` or `720p`). |
|
|
|
|
Per-request `params` may set `num_segments`, `audio_guidance_scale`,
|
|
`offload_kv_cache`, `ref_img_index`, `mask_frame_range`, and `resolution`.
|
|
|
|
Gallery and imported configs declare `known_input_modalities` and
|
|
`known_output_modalities`. Keep those declarations in manual configs as well;
|
|
they let model discovery distinguish base image-conditioned video from Avatar
|
|
audio conditioning without inspecting the backend or checkpoint name.
|
|
|
|
LongCat is CUDA-only and very large. Avatar 1.5 also loads tokenizer,
|
|
text-encoder, and VAE components from the base checkpoint. Keep ample unified
|
|
memory and storage available; no CPU or macOS backend image is published. The
|
|
initial backend supports one GPU per process; tensor parallel sizes above one
|
|
are rejected explicitly.
|