wehub-resource-sync
205 lines
8.2 KiB
Markdown
205 lines
8.2 KiB
Markdown
# Lance Training Dataset Construction
|
|
|
|
## 1.1 Local Dataset Parquet Construction
|
|
|
|
Training ultimately reads parquet files. Below is the recommended local parquet format for current local training. Download example datasets from [Hugging Face](https://huggingface.co/datasets/bytedance-research/Lance_example_dataset) and place them under local `./datasets`.
|
|
|
|
### 1.1.1 t2i
|
|
|
|
- Corresponding `dataset_type`: `text2image_general`
|
|
|
|
| Field | Type | Description |
|
|
| --- | --- | --- |
|
|
| `caption` | `string` | Text condition |
|
|
| `image_bytes` | `binary` | Target image |
|
|
|
|
|
|
### 1.1.2 t2v
|
|
|
|
- Corresponding `dataset_type`: `text2video_general`
|
|
|
|
| Field | Type | Description |
|
|
| --- | --- | --- |
|
|
| `caption` | `string` | Text condition |
|
|
| `video_bytes` | `binary` | Target video |
|
|
|
|
|
|
### 1.1.3 i2i
|
|
|
|
- Corresponding `dataset_type`: `image2image`
|
|
|
|
| Field | Type | Description |
|
|
| --- | --- | --- |
|
|
| `caption` | `string` | Editing instruction |
|
|
| `input_image_bytes` | `binary` | Conditional image |
|
|
| `output_image_bytes` | `binary` | Target image |
|
|
|
|
|
|
### 1.1.4 v2v
|
|
|
|
- Corresponding `dataset_type`: `video2video`
|
|
|
|
| Field | Type | Description |
|
|
| --- | --- | --- |
|
|
| `caption` | `string` | Editing instruction |
|
|
| `input_video_bytes` | `binary` | Conditional video |
|
|
| `output_video_bytes` | `binary` | Target video |
|
|
|
|
|
|
### 1.1.5 i2t
|
|
|
|
- Corresponding `dataset_type`: `x2t_general`
|
|
|
|
| Field | Type | Description |
|
|
| --- | --- | --- |
|
|
| `image_bytes` | `binary` | Conditional image |
|
|
| `caption_i` | `string` | Instruction / system prompt |
|
|
| `caption_q` | `string` | Question, can be empty `""` |
|
|
| `caption_a` | `string` | Target answer |
|
|
|
|
|
|
### 1.1.6 v2t
|
|
|
|
- Corresponding `dataset_type`: `x2t_general`
|
|
|
|
| Field | Type | Description |
|
|
| --- | --- | --- |
|
|
| `video_bytes` | `binary` | Conditional video |
|
|
| `caption_i` | `string` | Instruction / system prompt |
|
|
| `caption_q` | `string` | Question, can be empty `""` |
|
|
| `caption_a` | `string` | Target answer |
|
|
|
|
|
|
### 1.1.7 About `caption_i / caption_q / caption_a`
|
|
|
|
These three fields form an `I / Q / A` tuple in `x2t` tasks:
|
|
|
|
| Field | Meaning |
|
|
| --- | --- |
|
|
| `caption_i` | instruction |
|
|
| `caption_q` | question |
|
|
| `caption_a` | answer |
|
|
|
|
The training code packs them into a triplet and then feeds them into prompt construction logic.
|
|
|
|
|
|
## 1.2 Datasets and YAML
|
|
|
|
### 1.2.1 Dataset Registry
|
|
|
|
The current datasets are registered in `data/dataset_info.py`. The two core local entries are:
|
|
|
|
- `x2v_interleave_local`
|
|
- `x2t_interleave_local`
|
|
|
|
Meaning:
|
|
|
|
- `x2v_interleave_local`: local parquet image/video generation tasks, where media is stored as bytes
|
|
- `x2t_interleave_local`: local parquet image/video understanding-to-text tasks, where media is stored as bytes
|
|
|
|
|
|
### 1.2.2 Common Parameters
|
|
|
|
| Parameter | Explanation | Options | Option meanings |
|
|
| --- | --- | --- | --- |
|
|
| `dataset_type` | Determines how one parquet row is parsed into `interleave_array` and `element_dtype_array` | Common values are explained at right | `text2image_general`: text-to-image, fields are usually `caption` + `image_bytes`; `text2video_general`: text-to-video, fields are usually `caption` + `video_bytes`; `x2t_general`: visual understanding to text, fields are usually `image_bytes` or `video_bytes`, plus `caption_i/q/a`; `image2image`: image editing, fields are usually `caption` + `input_image_bytes` + `output_image_bytes`; `video2video`: video editing, fields are usually `caption` + `input_video_bytes` + `output_video_bytes` |
|
|
| `dataset_names.data_dir` | Specifies the parquet path to read | Local path | For example `datasets/video2text/local_256.parquet` |
|
|
| `raw_bytes_input` | Controls whether visual media is read directly as bytes from parquet | `true` / `false` | `true`: directly read `image_bytes` / `video_bytes` from parquet; `false`: the input is usually a URL or path and media is fetched online during training |
|
|
| `is_image` | Controls whether the visual input is treated as image or video | `true` / `false` | `true`: treat as image, commonly used for `t2i`, `i2i`, `i2t`; `false`: treat as video, commonly used for `t2v`, `v2v`, `v2t` |
|
|
| `target_modality` | Specifies the target modality | `image` / `video` / `text` | `image`: target is image; `video`: target is video; `text`: target is text |
|
|
| `task_type` | Defines task semantics and how condition / target are split inside one sample | Common values are explained at right | `t2v`: text-to-video generation; `ff2v`: first-frame-to-video; `flf2v`: first-and-last-frame-to-video; `tv2v`: text + visual conditioned video editing/generation; `vt2v`: visual + text conditioned video editing/generation; `v2t`: pure visual-to-text understanding; `tv2t`: text + visual conditioned text understanding; `vt2t`: visual + text conditioned text understanding; `t2t`: pure text-to-text |
|
|
| `text_template` | Controls whether chat/prompt template construction is enabled | `true` / `false` | `true`: use a QwenVL-style template and structure the prompt as system / user / assistant; `false`: skip the template and tokenize raw text directly |
|
|
| `vision_cond_type` | Controls how conditional visual branches are organized in editing tasks | Common values are explained at right | `["vit"]`: use only the VIT conditional branch; `["vae_Nsplit"]`: use only the VAE conditional branch; `["vit", "vae_Nsplit"]`: use both VIT and VAE conditional branches, commonly used in editing tasks |
|
|
| `debug_parquet_repeat` | Repeats parquet files in small debug datasets to avoid very short epochs | Positive integer | Repeats the local parquet file list, commonly used for smoke tests |
|
|
| `video_transform_args` | Controls transforms for the VAE or main visual branch | Config dict | Includes resolution, bucket mode, normalization, etc. |
|
|
| `vit_video_transform_args` | Controls transforms for the VIT branch | Config dict | Usually only needed when a VIT conditional branch is used |
|
|
| `video_frame_sampler_args` | Controls the video frame sampling strategy | Config dict | Includes sampler type, fps, max duration, whether to truncate, etc. |
|
|
|
|
|
|
### 1.2.3 t2i
|
|
|
|
Reference config: `config/train_local/t2i_local.yaml`
|
|
|
|
| Item | Value |
|
|
| --- | --- |
|
|
| `dataset_type` | `text2image_general` |
|
|
| `is_image` | `true` |
|
|
| `target_modality` | `image` |
|
|
| Common `task_type` | `t2v` |
|
|
| Required extra config | `video_transform_args` |
|
|
| Usually not needed | `vit_video_transform_args`, `video_frame_sampler_args` |
|
|
|
|
|
|
### 1.2.4 t2v
|
|
|
|
Reference config: `config/train_local/t2v_local.yaml`
|
|
|
|
| Item | Value |
|
|
| --- | --- |
|
|
| `dataset_type` | `text2video_general` |
|
|
| `is_image` | `false` |
|
|
| `target_modality` | `video` |
|
|
| Common `task_type` | `t2v`, `ff2v` |
|
|
| Required extra config | `video_transform_args`, `video_frame_sampler_args` |
|
|
| Usually not needed | `vit_video_transform_args` |
|
|
|
|
|
|
### 1.2.5 i2i
|
|
|
|
Reference config: `config/train_local/i2i_local.yaml`
|
|
|
|
| Item | Value |
|
|
| --- | --- |
|
|
| `dataset_type` | `image2image` |
|
|
| `is_image` | `true` |
|
|
| `target_modality` | `image` |
|
|
| Common `task_type` | `tv2v` |
|
|
| `sample_task` | `edit` |
|
|
| Common `vision_cond_type` | `["vit", "vae_Nsplit"]` |
|
|
| Required extra config | `video_transform_args`, `vit_video_transform_args` |
|
|
| Usually not needed | `video_frame_sampler_args` |
|
|
|
|
|
|
### 1.2.6 v2v
|
|
|
|
Reference config: `config/train_local/v2v_local.yaml`
|
|
|
|
| Item | Value |
|
|
| --- | --- |
|
|
| `dataset_type` | `video2video` |
|
|
| `is_image` | `false` |
|
|
| `target_modality` | `video` |
|
|
| Common `task_type` | `tv2v` |
|
|
| `sample_task` | `edit` |
|
|
| Common `vision_cond_type` | `["vit", "vae_Nsplit"]` |
|
|
| Required extra config | `video_transform_args`, `vit_video_transform_args`, `video_frame_sampler_args` |
|
|
| Usually not needed | None |
|
|
|
|
|
|
### 1.2.7 i2t
|
|
|
|
Reference config: `config/train_local/i2t_local.yaml`
|
|
|
|
| Item | Value |
|
|
| --- | --- |
|
|
| `dataset_type` | `x2t_general` |
|
|
| `is_image` | `true` |
|
|
| Target type | Text (fixed by the `x2t` dataset class) |
|
|
| Common `task_type` | `tv2t` |
|
|
| Required extra config | `video_transform_args` |
|
|
| Usually not needed | `vit_video_transform_args`, `video_frame_sampler_args` |
|
|
|
|
|
|
### 1.2.8 v2t
|
|
|
|
Reference config: `config/train_local/v2t_local.yaml`
|
|
|
|
| Item | Value |
|
|
| --- | --- |
|
|
| `dataset_type` | `x2t_general` |
|
|
| `is_image` | `false` |
|
|
| Target type | Text (fixed by the `x2t` dataset class) |
|
|
| Common `task_type` | `tv2t` |
|
|
| Required extra config | `video_transform_args`, `video_frame_sampler_args` |
|
|
| Usually not needed | `vit_video_transform_args` |
|