This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,233 @@
|
||||
# Architecture Introduction
|
||||
|
||||
ms-swift 4.0 adopts a modular design, with functional modules distributed in first-level directories, making it convenient for developers to perform custom extensions. This document will provide a detailed introduction to the functions of each module and customization methods.
|
||||
|
||||
## Agent Template
|
||||
|
||||
The mapping file for agent templates can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/agent_template/mapping.py). The design goal of agent template is to flexibly switch between different models for training based on a unified Agent dataset format, without modifying the data. During training, use `--agent_template` to specify the corresponding agent template.
|
||||
|
||||
All AgentTemplates need to inherit from `BaseAgentTemplate` and implement several methods: `_format_tools`, `_format_tool_calls`, `_format_tool_responses`, `get_toolcall`.
|
||||
- _format_tools: Format `tools` and `system` to compose a complete system.
|
||||
- _format_tool_calls: Format the tool_call part `[{"role": "tool_call", "content": "..."}, {"role": "tool_call", "content": "..."}]` and finally return a string.
|
||||
- _format_tool_responses: Format the tool (also called tool_response) part `[{"role": "tool", "content": "..."}, {"role": "tool", "content": "..."}]`.
|
||||
- get_toolcall: Used during deployment to parse the tool name and parameters from the model output content, returning `List[Function]`.
|
||||
|
||||
|
||||
How to debug:
|
||||
```python
|
||||
data = {"tools": "[{\"type\": \"function\", \"function\": {\"name\": \"realtime_aqi\", \"description\": \"天气预报。获取实时空气质量。当前空气质量，PM2.5，PM10信息\", \"parameters\": {\"type\": \"object\", \"properties\": {\"city\": {\"type\": \"string\", \"description\": \"城市名，例如：上海\"}}, \"required\": [\"city\"]}}}]", "messages": [{"role": "user", "content": "北京和上海今天的天气情况"}, {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"北京\"}}"}, {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"上海\"}}"}, {"role": "tool_response", "content": "{\"city\": \"北京\", \"aqi\": \"10\", \"unit\": \"celsius\"}"}, {"role": "tool_response", "content": "{\"city\": \"上海\", \"aqi\": \"72\", \"unit\": \"fahrenheit\"}"}, {"role": "assistant", "content": "根据天气预报工具，北京今天的空气质量指数为10，属于良好水平；上海今天的空气质量指数为72，属于轻度污染水平。"}]}
|
||||
|
||||
|
||||
from swift import get_processor, get_template
|
||||
|
||||
tokenizer = get_processor('Qwen/Qwen3.5-2B')
|
||||
template = get_template(tokenizer) # Use default agent template
|
||||
# template = get_template(tokenizer, agent_template='qwen3_5')
|
||||
print(f'agent_template: {template._agent_template}')
|
||||
template.set_mode('train')
|
||||
encoded = template.encode(data)
|
||||
print(f'[INPUT_IDS] {template.safe_decode(encoded["input_ids"])}\n')
|
||||
print(f'[LABELS] {template.safe_decode(encoded["labels"])}')
|
||||
```
|
||||
|
||||
If you want to provide us with a PR, please refer to [here](https://github.com/modelscope/ms-swift/blob/main/tests/test_align/test_template/test_agent.py) to write your test cases.
|
||||
|
||||
## Callbacks
|
||||
|
||||
The mapping file for callbacks can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/callbacks/mapping.py). Callbacks can customize the behavior at key points in the trainer. After customization, you need to register them in the mapping and use `--callbacks` to specify the corresponding callback class during training. For example, you can customize:
|
||||
|
||||
```python
|
||||
class CustomCallback(TrainerCallback):
|
||||
|
||||
def on_train_begin(self, args: TrainingArguments, state: TrainerState, control: TrainerControl, **kwargs):
|
||||
# Doing something when the training begins.
|
||||
pass
|
||||
|
||||
def on_save(self, args: TrainingArguments, state: TrainerState, control: TrainerControl, **kwargs):
|
||||
# Doing something when save checkpoint
|
||||
pass
|
||||
```
|
||||
|
||||
All callback classes need to inherit from `TrainerCallback` in base.py and override its methods. The interface is consistent with transformers' `TrainerCallback`, please refer to transformers' [callback documentation](https://huggingface.co/docs/transformers/main_classes/callback).
|
||||
|
||||
## Loss
|
||||
|
||||
The mapping file for Loss can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/loss/mapping.py).
|
||||
Swift supports custom loss (currently only supports sft/pretrain/reranker/embedding tasks). After registration, set `--loss_type <loss-name>` during training to use your custom loss method.
|
||||
|
||||
Custom Loss needs to inherit from `BaseLoss` and implement the `__call__` method, returning a scalar Tensor. You can refer to [CustomCrossEntropyLoss](https://github.com/modelscope/ms-swift/blob/0d7c9f5bc0e7e7d67d914ce6edeb9ce24f60746f/swift/loss/causal_lm.py#L5) for customization. For example:
|
||||
|
||||
```python
|
||||
class CustomLoss(BaseLoss):
|
||||
|
||||
def __call__(self, outputs, labels, **kwargs) -> torch.Tensor:
|
||||
pass
|
||||
```
|
||||
|
||||
## Loss Scale
|
||||
|
||||
The mapping file for loss scale can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/loss_scale/mapping.py). In pretrain and sft tasks, the loss of trainable tokens is averaged, meaning each token is treated equally. However, in some cases, certain tokens need extra attention and should be assigned higher weights, or some tokens should not be trained. loss_scale allows developers to freely define their own token weights. (Pretrain and SFT support using loss_scale to control whether tokens participate in training and their weight sizes, while in RLHF, it only supports controlling whether tokens participate in training)
|
||||
|
||||
You can customize loss scale by inheriting the LossScale base class and implementing the `get_loss_scale` method.
|
||||
|
||||
```python
|
||||
class CustomLossScale(LossScale):
|
||||
|
||||
def get_loss_scale(self, context: str, **kwargs) -> Tuple[List[str], List[float]]:
|
||||
...
|
||||
```
|
||||
|
||||
The `get_loss_scale` function returns a Tuple. The first return is a list of decomposed strings, and the second parameter is a list of loss_scales corresponding to the strings. The float value represents the weight. For example, the following weight setting:
|
||||
|
||||
```text
|
||||
["学习", "好", "数学", "是", "重要", "的"]
|
||||
[1.0, 0.5, 2.0, 0.5, 2.0, 0.1]
|
||||
```
|
||||
In the example, we place more emphasis on the words "数学" and "重要" because their loss_scale is 2.0.
|
||||
|
||||
Of course, we also need to pay attention to the core logic of the `__call__` method, namely the influence of the loss_scale base strategy (base_strategy) all/default/last_round on loss_scale. For details, refer to the introduction in the [Command-line Parameters Documentation](../Instruction/Command-line-parameters.md). Also, refer to the influence of the 'loss' field in the dataset on loss_scale in the [Custom Dataset Documentation](../Customization/Custom-dataset.md).
|
||||
|
||||
```python
|
||||
if loss or loss is None and (self.base_strategy == 'all' or
|
||||
(self.base_strategy == 'default' and is_assistant) or
|
||||
(self.base_strategy == 'last_round' and is_assistant and is_last_round)):
|
||||
new_context, loss_scale = self.get_loss_scale(context, query=query)
|
||||
else:
|
||||
new_context, loss_scale = [context], [0.]
|
||||
```
|
||||
|
||||
In addition, you can also use [JSON configuration files](https://github.com/modelscope/ms-swift/tree/main/swift/loss_scale/config) and inherit the built-in ConfigLossScale class to customize loss_scale. Currently, two configuration methods are supported: exact string matching and regular expression matching. You can refer to the content in [Agent Support Documentation](../Instruction/Agent-support.md#usage-of-loss_scale) for understanding.
|
||||
|
||||
- Exact string matching, for example, refer to `react.json`, `qwen.json`. The JSON needs to contain a mapping of `Dict[str, List[float]]`. The string represents a keyword, and the list needs to have two values. We will split the string into multiple segments based on the keyword. The first value in the list represents the weight of the keyword, and the second value represents the weight of the content after this keyword and before the next keyword.
|
||||
- Regular expression matching, for example, refer to `ignore_empty_think.json`, `hermes.json`. The JSON needs to contain a mapping of `Dict[str, float]`. The string represents a regular expression pattern, and the float represents the weight of the matching string.
|
||||
|
||||
How to debug:
|
||||
|
||||
```python
|
||||
from swift import get_processor, get_template
|
||||
|
||||
data = {"messages": [
|
||||
{"role": "user", "content": "What is today's date?"},
|
||||
{"role": "assistant", "content": (
|
||||
"<think>\nI can get the current time by calling the `get_date` function.\n</think>\n"
|
||||
'<tool_call>\n{"name": "get_date", "arguments": {}}\n</tool_call>'
|
||||
)}
|
||||
]}
|
||||
|
||||
template = get_template(get_processor('Qwen/Qwen3-8B'), loss_scale='hermes')
|
||||
template.set_mode('train')
|
||||
inputs = template.encode(data)
|
||||
|
||||
print(template.safe_decode(inputs['labels']))
|
||||
print(inputs['loss_scale'])
|
||||
```
|
||||
|
||||
## Metrics
|
||||
|
||||
The mapping file for metrics can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/metrics/mapping.py). This component is used in both ms-swift and Megatron-SWIFT.
|
||||
|
||||
- If used in ms-swift, you need to inherit the `EvalMetrics` base class from base.py and implement the `compute_metrics` function, returning a dictionary `Dict[str, float]`. You can refer to [NlgMetrics](https://github.com/modelscope/ms-swift/blob/0d7c9f5bc0e7e7d67d914ce6edeb9ce24f60746f/swift/metrics/nlg.py#L33) for customization.
|
||||
- If used in Megatron-SWIFT, you need to inherit the `Metric` base class from utils.py and implement the `update` and `compute` methods. The compute method should return a dictionary `Dict[str, float]`.
|
||||
|
||||
You can customize metrics (currently only supports sft/pretrain/reranker/embedding tasks) and set `--eval_metric <metric-name>` during training to use your custom metrics.
|
||||
|
||||
## Optimizers
|
||||
|
||||
The mapping file for optimizers can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/optimizers/mapping.py). If you need to customize an optimizer, you need to inherit the `OptimizerCallback` base class and override the `create_optimizer` function. Use `--optimizer <optimizer-name>` during training to specify the custom optimizer.
|
||||
|
||||
- You can refer to [MultimodalOptimizerCallback](https://github.com/modelscope/ms-swift/blob/0d7c9f5bc0e7e7d67d914ce6edeb9ce24f60746f/swift/optimizers/multimodal.py#L43) for implementation. This class implements the functionality of vit_lr and aligner_lr, which uses different learning rates for vit, aligner, and LLM respectively.
|
||||
|
||||
## Tuner Plugin
|
||||
|
||||
The mapping file for Tuner plugins can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/tuner_plugin/mapping.py). If you need to customize a tuner, you need to inherit the `Tuner` base class and override the `prepare_model`, `save_pretrained`, `from_pretrained` functions.
|
||||
|
||||
- prepare_model: This function is called before training to process and prepare the original model, wrap it with the tuner, and set trainable parameters. For example: you can attach LoRA to certain layers and freeze certain layers.
|
||||
- save_pretrained: This function is called during training to save the model.
|
||||
- from_pretrained: This function is called during inference/resuming training to prepare the model and load weights.
|
||||
|
||||
You can refer to [LoRALLMTuner](https://github.com/modelscope/ms-swift/blob/0d7c9f5bc0e7e7d67d914ce6edeb9ce24f60746f/swift/tuner_plugin/lora_llm.py#L24) for implementation. This class implements the functionality of performing LoRA training on LLM and full parameter training on ViT.
|
||||
|
||||
## ORM
|
||||
|
||||
Examples can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/rewards/orm.py).
|
||||
|
||||
ORM is an Outcome Reward Model. ORM is generally implemented using regular expressions. ORM determines whether a response is correct. For example:
|
||||
|
||||
```python
|
||||
class MathORM(ORM):
|
||||
|
||||
@staticmethod
|
||||
def extract_boxed_result(text):
|
||||
pattern = r'\\boxed{([^}]*)}'
|
||||
match = re.search(pattern, text)
|
||||
if match:
|
||||
return match.group(1).strip()
|
||||
else:
|
||||
return None
|
||||
|
||||
def __call__(self, infer_requests: List[InferRequest], ground_truths: List[str],
|
||||
**kwargs) -> List[float]:
|
||||
rewards = []
|
||||
predictions = [request.messages[-1]['content'] for request in infer_requests]
|
||||
for prediction, ground_truth in zip(predictions, ground_truths):
|
||||
res1 = MathORM.extract_boxed_result(prediction) or ''
|
||||
res2 = MathORM.extract_boxed_result(ground_truth) or ''
|
||||
rewards.append(float(res1.strip() == res2.strip()))
|
||||
|
||||
return rewards
|
||||
|
||||
|
||||
orms = {
|
||||
'math': MathORM,
|
||||
}
|
||||
```
|
||||
|
||||
In the code above, we define a process for parsing mathematical responses. If the results are the same, it returns a score of 1.0, otherwise 0.0. Unlike PRM, this class has an additional parameter `ground_truths` in infer,
|
||||
which contains the actual labels (standard responses defined in the dataset) of the corresponding infer_requests.
|
||||
|
||||
## PRM
|
||||
|
||||
Examples can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/rewards/prm.py).
|
||||
|
||||
PRM is a Process Reward Model, which will be used in the `swift sample` command. The interface that PRM needs to support is relatively simple:
|
||||
|
||||
```python
|
||||
class PRM:
|
||||
|
||||
def __init__(self):
|
||||
# init here
|
||||
pass
|
||||
|
||||
def __call__(self, infer_requests: List[InferRequest], **kwargs) -> List[Union[float, List[float]]]:
|
||||
raise NotImplementedError
|
||||
```
|
||||
|
||||
The InferRequest comes from `swift.infer_engine`, and the returned `List[Union[float, List[float]]]` can contain either a reward or multiple rewards. Developers can obtain queries and responses from infer_requests and split them according to their own methods. For example:
|
||||
|
||||
```text
|
||||
Let's think step by step.
|
||||
|
||||
Step1: xxx
|
||||
|
||||
Step2: xxx
|
||||
|
||||
So, the answer is ...
|
||||
```
|
||||
|
||||
## Introduction to Other Directory Structures
|
||||
|
||||
- arguments: Command-line parameter definitions, such as: `SftArguments`, `RLHFArguments`, etc.
|
||||
- cli: Swift command-line mechanism and startup files. For example, `swift sft ...` is equivalent to `python swift/cli/main.py sft ...` and also equivalent to `python swift/cli/sft.py ...`.
|
||||
- config: deepspeed/fsdp2 configuration files.
|
||||
- dataloader: Implementation of dataloader, including shard/dispatcher methods.
|
||||
- dataset: Dataset-related module implementation, including data preprocessing, packing, streaming data, etc. Registration of built-in datasets is in the `dataset/dataset` and `dataset/data` folders. For details, refer to [Custom Dataset Documentation](Custom-dataset.md).
|
||||
- infer_engine: Inference engine implementation. Includes inference engine implementations with transformers/vllm/sglang/lmdeploy as backends.
|
||||
- megatron: Megatron-SWIFT implementation.
|
||||
- model: Model loading and registration. For details, refer to [Custom Model Documentation](Custom-model.md), [Multimodal Model Registration Best Practices](../BestPractices/MLLM-Registration.md).
|
||||
- pipelines: Main function pipeline implementations for `swift sft/rlhf/infer`, etc., including `sft_main/rlhf_main/infer_main`, etc.
|
||||
- rlhf_trainers: Trainer implementations for algorithms such as GRPO/GKD/DPO/KTO/RM.
|
||||
- rollout: Sampling implementation of the rollout process in RL algorithms.
|
||||
- rewards: Reward function implementation in RL algorithms, supporting custom reward calculation logic.
|
||||
- template: Implementation and registration of dialogue templates, including the logic for converting messages to input_ids for various tasks, as well as data_collator-related logic. For details, refer to [Custom Model Documentation](Custom-model.md), [Multimodal Model Registration Best Practices](../BestPractices/MLLM-Registration.md).
|
||||
- trainers: Trainer implementations for pretrain/SFT/Embedding/Reranker/sequence classification tasks.
|
||||
- ui: `swift web-ui` interface training and inference implementation.
|
||||
@@ -0,0 +1,448 @@
|
||||
# Custom Dataset
|
||||
|
||||
There are three methods for accessing custom datasets, each offering progressively greater control over preprocessing functions but also increasing in complexity. For example, Solution 1 is the most convenient but offers the least control over preprocessing functions, requiring prior conversion of the dataset into a specific format:
|
||||
|
||||
1. **Recommended**: Directly use the command line parameter to access the dataset with `--dataset <dataset_path1> <dataset_path2>`. This will use `AutoPreprocessor` to convert your dataset into a standard format (supporting four dataset formats; see the introduction to AutoPreprocessor below). You can use `--columns` to transform column names. The supported input formats include csv, json, jsonl, txt, and folders (e.g. git clone open-source datasets). This solution does not require modifying `dataset_info.json` and is suitable for users new to ms-swift. The following two solutions are suitable for developers looking to extend ms-swift.
|
||||
2. Add the dataset to `dataset_info.json`, which you can refer to in the built-in [dataset_info.json](https://github.com/modelscope/ms-swift/blob/main/swift/dataset/data/dataset_info.json) of ms-swift. This solution also uses AutoPreprocessor to convert the dataset to a standard format. `dataset_info.json` is a list of metadata for datasets, and one of the fields ms_dataset_id/hf_dataset_id/dataset_path must be filled. Column name transformation can be done through the `columns` field. Datasets added to `dataset_info.json` or registered ones will automatically generate [supported dataset documentation](https://swift.readthedocs.io/en/latest/Instruction/Supported-models-and-datasets.html) when running [run_dataset_info.py](https://github.com/modelscope/ms-swift/blob/main/scripts/utils/run_dataset_info.py). In addition, you can use the external `dataset_info.json` approach by parsing the JSON file with `--custom_dataset_info xxx.json` (to facilitate users who prefer `pip install` over `git clone`), and then specify `--dataset <dataset_id/dataset_dir/dataset_path>`.
|
||||
3. Manually register the dataset to have the most flexible customization capability for preprocessing functions, allowing the use of functions to preprocess datasets, but it is more difficult. You can refer to the [built-in datasets](https://github.com/modelscope/ms-swift/blob/main/swift/dataset/dataset/llm.py) or [examples](https://github.com/modelscope/ms-swift/blob/main/examples/custom). You can specify `--external_plugins xxx.py` to parse external registration content (convenient for users who use pip install instead of git clone).
|
||||
- Solutions one and two leverage solution three under the hood, where the registration process occurs automatically.
|
||||
|
||||
The following is an introduction to the dataset formats that `AutoPreprocessor` can handle:
|
||||
|
||||
The standard dataset format for ms-swift accepts keys such as: 'messages', 'rejected_response', 'label', 'images', 'videos', 'audios', 'tools', and 'objects'. Among these, 'messages' is a required key. 'rejected_response' is used for DPO and other RLHF training, 'label' is used for KTO training and classification model training. The keys 'images', 'videos', and 'audios' are used to store paths or URLs for multimodal data, 'tools' is used for Agent tasks, and 'objects' is used for grounding tasks.
|
||||
|
||||
There are three core preprocessors in ms-swift: `MessagesPreprocessor`, `AlpacaPreprocessor`, and `ResponsePreprocessor`. `MessagesPreprocessor` is used to convert datasets in the messages and sharegpt format into the standard format. `AlpacaPreprocessor` converts datasets in the alpaca format, while `ResponsePreprocessor` converts datasets in the query/response format. `AutoPreprocessor` automatically selects the appropriate preprocessor for the task.
|
||||
|
||||
The following four formats will all be converted into the `messages` field of the ms-swift standard format under the processing of `AutoPreprocessor`, meaning they can all be directly used with `--dataset <dataset-path>`:
|
||||
|
||||
Messages format (standard format):
|
||||
```jsonl
|
||||
{"messages": [{"role": "system", "content": "<system>"}, {"role": "user", "content": "<query1>"}, {"role": "assistant", "content": "<response1>"}, {"role": "user", "content": "<query2>"}, {"role": "assistant", "content": "<response2>"}]}
|
||||
```
|
||||
- Note: The system part is optional. The system in the dataset has a higher priority than the `--system` passed through the command line, followed by the `default_system` defined in the template.
|
||||
|
||||
ShareGPT format:
|
||||
```jsonl
|
||||
{"system": "<system>", "conversation": [{"human": "<query1>", "assistant": "<response1>"}, {"human": "<query2>", "assistant": "<response2>"}]}
|
||||
```
|
||||
|
||||
Query-Response format:
|
||||
```jsonl
|
||||
{"system": "<system>", "query": "<query2>", "response": "<response2>", "history": [["<query1>", "<response1>"]]}
|
||||
```
|
||||
Note: The following fields will be automatically converted to the corresponding system, query, and response fields. (The 'solution' field will be retained)
|
||||
- system: 'system', 'system_prompt'.
|
||||
- query: 'query', 'prompt', 'input', 'instruction', 'question', 'problem'.
|
||||
- response: 'response', 'answer', 'output', 'targets', 'target', 'answer_key', 'answers', 'solution', 'text', 'completion', 'content'.
|
||||
|
||||
Alpaca format:
|
||||
```jsonl
|
||||
{"system": "<system>", "instruction": "<query-inst>", "input": "<query-input>", "output": "<response>"}
|
||||
```
|
||||
- Note: The instruction and input fields will be combined into the query field. If instruction and input are not empty strings, then `query = f'{instruction}\n{input}'`.
|
||||
|
||||
|
||||
## Standard Dataset Format
|
||||
|
||||
The following outlines the standard dataset format for ms-swift, where the "system" field is optional and uses the "default_system" defined in the template by default. The four dataset formats introduced earlier can also be processed by AutoPreprocessor into the standard dataset format.
|
||||
|
||||
### Pre-training
|
||||
|
||||
```jsonl
|
||||
{"messages": [{"role": "assistant", "content": "I love music"}]}
|
||||
{"messages": [{"role": "assistant", "content": "Coach, I want to play basketball"}]}
|
||||
{"messages": [{"role": "assistant", "content": "Which is more authoritative, tomato and egg rice or the third fresh stir-fry?"}]}
|
||||
```
|
||||
|
||||
### Supervised Fine-tuning
|
||||
|
||||
```jsonl
|
||||
{"messages": [{"role": "system", "content": "You are a useful and harmless assistant"}, {"role": "user", "content": "Tell me tomorrow's weather"}, {"role": "assistant", "content": "Tomorrow's weather will be sunny"}]}
|
||||
{"messages": [{"role": "system", "content": "You are a useful and harmless math calculator"}, {"role": "user", "content": "What is 1 + 1?"}, {"role": "assistant", "content": "It equals 2"}, {"role": "user", "content": "What about adding 1?"}, {"role": "assistant", "content": "It equals 3"}]}
|
||||
```
|
||||
|
||||
- You can add a `"loss"` field to control whether the loss is computed for the corresponding model response ("role" is "assistant"). This field defaults to `None`. If `"loss"` is set to `true`, the loss will be computed for the corresponding content (the specific `loss_scale` is still determined by `--loss_scale`); if `"loss"` is set to `false`, the loss will not be computed for the corresponding content. Note that this field only takes effect for parts where `"role"` is `"assistant"`. This field takes priority over the basic strategies of the `--loss_scale` command-line argument (i.e., `'default'`, `'last_round'`, `'all'`). For example, when `loss_scale` is set to `'default+ignore_empty_think'`, the `"loss"` field takes priority over `'default'`, but `'ignore_empty_think'` still takes effect.
|
||||
- You can add a `"loss_scale"` field to control the `loss_scale` for the corresponding model response ("role" is "assistant"). (ms-swift >= 4.2.0) Defaults to `None`. This field takes priority over other strategy components of the `--loss_scale` command-line argument, such as `'ignore_empty_think'`, `'hermes'`, etc. If any value greater than `1` appears in `loss_scale`, you need to additionally set `--is_binary_loss_scale false`.
|
||||
|
||||
```jsonl
|
||||
{"messages": [{"role": "user", "content": "Hello!"}, {"role": "assistant", "content": "Hi, how can I help you?", "loss": false}, {"role": "user", "content": "What is 1+1?"}, {"role": "assistant", "content": "It equals 2", "loss": true}]}
|
||||
{"messages": [{"role": "user", "content": "hello!"}, {"role": "assistant", "content": "<think>\n...\n</think>\n", "loss_scale": 1.0}, {"role": "assistant", "content": "hi!", "loss_scale": 2.0}, {"role": "user", "content": "1+1=?"}, {"role": "assistant", "content": "<think>\n...\n</think>\n1+1=3", "loss": false}]}
|
||||
```
|
||||
|
||||
Use the following script to test:
|
||||
|
||||
```python
|
||||
from swift import get_processor, get_template
|
||||
|
||||
data = {"messages": [
|
||||
{"role": "user", "content": "hello!"},
|
||||
{"role": "assistant", "content": "<think>\n...\n</think>\n", "loss_scale": 1.},
|
||||
{"role": "assistant", "content": "hi!", "loss_scale": 2.},
|
||||
{"role": "user", "content": "1+1=?"},
|
||||
{"role": "assistant", "content": "<think>\n...\n</think>\n1+1=3", "loss": False},
|
||||
]}
|
||||
|
||||
template = get_template(get_processor('Qwen/Qwen3-8B'), loss_scale='default+ignore_empty_think',
|
||||
is_binary_loss_scale=False)
|
||||
template.set_mode('train')
|
||||
inputs = template.encode(data)
|
||||
|
||||
print(template.safe_decode(inputs['labels']))
|
||||
print(inputs['loss_scale'])
|
||||
```
|
||||
|
||||
Note: If you set "loss"/"loss_scale" on consecutive "tool_call" messages in the messages list, only the configuration of the first "tool_call" takes effect. For example:
|
||||
```jsonl
|
||||
{"messages": [..., {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"Beijing\"}}", "loss": false}, {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"Shanghai\"}}"}, ...]}
|
||||
```
|
||||
|
||||
- The "chat_template_kwargs" field (requires ms-swift>=4.3.0) allows you to control template multimodal parameters such as min_pixels, max_pixels, fps, as well as parameters like enable_thinking (during inference) at the **sample level** by passing this field in the dataset. The following parameters are supported by different models:
|
||||
- Among them, "enable_thinking", "preserve_thinking" and "response_prefix" are supported by all models (takes effect during inference); the "max_pixels" parameter is supported by all multimodal models.
|
||||
- Qwen series multimodal models: parameters supported by qwen_vl_utils/qwen_omni_utils such as min_pixels, max_pixels, fps, etc.
|
||||
|
||||
```jsonl
|
||||
{"messages": [{"role": "user", "content": "<image>What is this"}, {"role": "assistant", "content": "This is a rabbit", "loss": false}], "chat_template_kwargs": {"max_pixels": 1048576}}
|
||||
{"messages": [{"role": "user", "content": "who are you?"}], "chat_template_kwargs": {"enable_thinking": false}}
|
||||
```
|
||||
|
||||
#### Channel Loss
|
||||
If you want to use channel loss, you need to set `--enable_channel_loss true` and add a "channel" field to your dataset. Channel loss is compatible with techniques such as packing, padding-free, and loss scaling.
|
||||
|
||||
```jsonl
|
||||
{"messages": [{"role": "system", "content": "You are a useful and harmless assistant"}, {"role": "user", "content": "Tell me tomorrow's weather"}, {"role": "assistant", "content": "Tomorrow's weather will be sunny"}], "channel": "general"}
|
||||
{"messages": [{"role": "system", "content": "You are a useful and harmless math calculator"}, {"role": "user", "content": "What is 1 + 1?"}, {"role": "assistant", "content": "It equals 2"}, {"role": "user", "content": "What about adding 1?"}, {"role": "assistant", "content": "It equals 3"}], "channel": "math"}
|
||||
```
|
||||
|
||||
### RLHF
|
||||
|
||||
#### DPO/ORPO/CPO/SimPO/RM
|
||||
|
||||
```jsonl
|
||||
{"messages": [{"role": "system", "content": "You are a useful and harmless assistant"}, {"role": "user", "content": "Tell me tomorrow's weather"}, {"role": "assistant", "content": "Tomorrow's weather will be sunny"}], "rejected_response": "I don't know"}
|
||||
{"messages": [{"role": "system", "content": "You are a useful and harmless math calculator"}, {"role": "user", "content": "What is 1 + 1?"}, {"role": "assistant", "content": "It equals 2"}, {"role": "user", "content": "What about adding 1?"}, {"role": "assistant", "content": "It equals 3"}], "rejected_response": "I don't know"}
|
||||
```
|
||||
|
||||
The format of multimodal data should follow the specifications in [Multimodal Dataset](#multimodal), with additional columns such as `images` to represent other modality inputs. When it is necessary to associate different image information with preference data, the `rejected_images` field can be used to indicate the images related to the rejected responses. In the alignment dataset, at least one of `rejected_images` or `rejected_response` must be provided for each entry.
|
||||
|
||||
> Note: RM additionally supports the margin column. For details, refer to the [RM documentation](../Instruction/RLHF.md#rm).
|
||||
|
||||
Sure, you can also directly use `rejected_messages` instead of only providing `rejected_response` / `rejected_images`, which offers greater flexibility (e.g., for multimodal or agent scenarios). If you use "rejected_messages", then in multimodal scenarios you must also provide "rejected_images", "rejected_audios", "rejected_videos", etc.; in Agent scenarios you must also provide "rejected_tools", etc. An example of the multimodal data format is as follows:
|
||||
- If using `rejected_response`, the default values for 'rejected_images/rejected_audios/rejected_videos/rejected_tools' are 'images/audios/videos/tools'; if using `rejected_messages`, they need to be passed in additionally.
|
||||
|
||||
```jsonl
|
||||
{"messages": [{"role": "user", "content": "<image>What is this?"}, {"role": "assistant", "content": "This is a kitten."}], "images": ["kitten.png"], "rejected_messages": [{"role": "user", "content": "<image>What is this?"}, {"role": "assistant", "content": "This is a puppy."}], "rejected_images": ["kitten.png"]}
|
||||
{"messages": [{"role": "user", "content": "<image>What is this?"}, {"role": "assistant", "content": "This is a kitten."}], "images": ["kitten.png"], "rejected_messages": [{"role": "user", "content": "<image>What is this?"}, {"role": "assistant", "content": "This is a kitten."}], "rejected_images": ["puppy.png"]}
|
||||
```
|
||||
|
||||
The above format is equivalent to:
|
||||
|
||||
```jsonl
|
||||
{"messages": [{"role": "user", "content": "<image>What is this?"}, {"role": "assistant", "content": "This is a kitten."}], "images": ["kitten.png"], "rejected_response": "This is a puppy."}
|
||||
{"messages": [{"role": "user", "content": "<image>What is this?"}, {"role": "assistant", "content": "This is a kitten."}], "images": ["kitten.png"], "rejected_images": ["puppy.png"]}
|
||||
# Example 1 can also be written as:
|
||||
{"messages": [{"role": "user", "content": "<image>What is this?"}, {"role": "assistant", "content": "This is a kitten."}], "images": ["kitten.png"], "rejected_response": [{"role": "assistant", "content": "This is a puppy."}]}
|
||||
```
|
||||
|
||||
You can also organize the Agent dataset in the following format:
|
||||
|
||||
```jsonl
|
||||
# It will find the position of the last user in `messages`, and replace the subsequent content with `rejected_response` to form `rejected_messages`
|
||||
{"tools": "[{\"type\": \"function\", \"function\": {\"name\": \"realtime_aqi\", \"description\": \"Weather forecast. Get real-time air quality, including current air quality, PM2.5, and PM10 information.\", \"parameters\": {\"type\": \"object\", \"properties\": {\"city\": {\"type\": \"string\", \"description\": \"City name, e.g., Shanghai\"}}, \"required\": [\"city\"]}}}]", "messages": [{"role": "user", "content": "What is the weather like in Beijing and Shanghai today?"}, {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"Beijing\"}}"}, {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"Shanghai\"}}"}, {"role": "tool_response", "content": "{\"city\": \"Beijing\", \"aqi\": \"10\", \"unit\": \"celsius\"}"}, {"role": "tool_response", "content": "{\"city\": \"Shanghai\", \"aqi\": \"72\", \"unit\": \"fahrenheit\"}"}, {"role": "assistant", "content": "According to the weather forecast tool, the air quality index (AQI) in Beijing is 10, which indicates good air quality; whereas in Shanghai, the AQI is 72, indicating mild pollution."}], "rejected_response": [{"role": "assistant", "content": "I don't know."}]}
|
||||
```
|
||||
|
||||
How to debug:
|
||||
|
||||
```python
|
||||
from swift import get_processor, get_template
|
||||
|
||||
data = {"tools": "[{\"type\": \"function\", \"function\": {\"name\": \"realtime_aqi\", \"description\": \"Weather forecast. Get real-time air quality, including current air quality, PM2.5, and PM10 information.\", \"parameters\": {\"type\": \"object\", \"properties\": {\"city\": {\"type\": \"string\", \"description\": \"City name, e.g., Shanghai\"}}, \"required\": [\"city\"]}}}]", "messages": [{"role": "user", "content": "What is the weather like in Beijing and Shanghai today?"}, {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"Beijing\"}}"}, {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"Shanghai\"}}"}, {"role": "tool_response", "content": "{\"city\": \"Beijing\", \"aqi\": \"10\", \"unit\": \"celsius\"}"}, {"role": "tool_response", "content": "{\"city\": \"Shanghai\", \"aqi\": \"72\", \"unit\": \"fahrenheit\"}"}, {"role": "assistant", "content": "According to the weather forecast tool, the air quality index (AQI) in Beijing is 10, which indicates good air quality; whereas in Shanghai, the AQI is 72, indicating mild pollution."}], "rejected_response": [{"role": "assistant", "content": "I don't know."}]}
|
||||
|
||||
template = get_template(get_processor('Qwen/Qwen3.5-4B'), loss_scale='last_round')
|
||||
template.set_mode('rlhf') # For details, refer to the `template_mode` parameter description in the command-line documentation.
|
||||
inputs = template.encode(data)
|
||||
|
||||
print(template.safe_decode(inputs['chosen_labels']))
|
||||
print(template.safe_decode(inputs['rejected_labels']))
|
||||
```
|
||||
|
||||
|
||||
#### KTO
|
||||
|
||||
```jsonl
|
||||
{"messages": [{"role": "system", "content": "You are a useful and harmless assistant"}, {"role": "user", "content": "Tell me tomorrow's weather"}, {"role": "assistant", "content": "I don't know"}], "label": false}
|
||||
{"messages": [{"role": "system", "content": "You are a useful and harmless math calculator"}, {"role": "user", "content": "What is 1 + 1?"}, {"role": "assistant", "content": "It equals 2"}, {"role": "user", "content": "What about adding 1?"}, {"role": "assistant", "content": "It equals 3"}], "label": true}
|
||||
```
|
||||
|
||||
#### PPO/GRPO
|
||||
|
||||
```jsonl
|
||||
{"messages": [{"role": "system", "content": "You are a useful and harmless assistant"}, {"role": "user", "content": "Tell me tomorrow's weather"}]}
|
||||
{"messages": [{"role": "system", "content": "You are a useful and harmless math calculator"}, {"role": "user", "content": "What is 1 + 1?"}, {"role": "assistant", "content": "It equals 2"}, {"role": "user", "content": "What about adding 1?"}]}
|
||||
{"messages": [{"role": "user", "content": "What is your name?"}]}
|
||||
```
|
||||
- Note: GRPO will pass through all additional field content to the ORM, unlike other training methods that, by default, delete extra fields. For example, you can additionally pass in 'solution'. The custom ORM needs to include a positional argument called `completions`, with other arguments as keyword arguments passed through from the additional dataset fields.
|
||||
|
||||
#### GKD
|
||||
|
||||
```jsonl
|
||||
{"messages": [{"role": "system", "content": "You are a useful and harmless assistant"}, {"role": "user", "content": "Tell me tomorrow's weather"}, {"role": "assistant", "content": "Tomorrow's weather will be sunny"}]}
|
||||
{"messages": [{"role": "system", "content": "You are a useful and harmless math calculator"}, {"role": "user", "content": "What is 1 + 1?"}, {"role": "assistant", "content": "It equals 2"}, {"role": "user", "content": "What about adding 1?"}, {"role": "assistant", "content": "It equals 3"}]}
|
||||
```
|
||||
|
||||
When under on-policy training, the final round of the 'assistant' part is not required (the student model generates data during training, the response from dataset will be removed):
|
||||
|
||||
```jsonl
|
||||
{"messages": [{"role": "system", "content": "You are a useful and harmless assistant"}, {"role": "user", "content": "Tell me tomorrow's weather"}]}
|
||||
{"messages": [{"role": "system", "content": "You are a useful and harmless math calculator"}, {"role": "user", "content": "What is 1 + 1?"}, {"role": "assistant", "content": "It equals 2"}, {"role": "user", "content": "What about adding 1?"}]}
|
||||
```
|
||||
|
||||
### Sequence Classification
|
||||
|
||||
**Single-label Task**:
|
||||
```jsonl
|
||||
{"messages": [{"role": "user", "content": "The weather is really nice today"}], "label": 1}
|
||||
{"messages": [{"role": "user", "content": "Today is really unlucky"}], "label": 0}
|
||||
{"messages": [{"role": "user", "content": "So happy"}], "label": 1}
|
||||
```
|
||||
|
||||
**Multi-label Task**:
|
||||
|
||||
```jsonl
|
||||
{"messages": [{"role": "user", "content": "<sentence>"}], "label": []}
|
||||
{"messages": [{"role": "user", "content": "<sentence>"}], "label": [0, 2]}
|
||||
{"messages": [{"role": "user", "content": "<sentence>"}], "label": [1, 3, 5]}
|
||||
```
|
||||
|
||||
**Single Regression Task**:
|
||||
|
||||
```jsonl
|
||||
{"messages": [{"role": "user", "content": "Calculate the similarity between two sentences, with a range of 0-1.\nsentence1: <sentence1>\nsentence2: <sentence2>"}], "label": 0.8}
|
||||
```
|
||||
|
||||
**Multi Regression Task**:
|
||||
|
||||
```jsonl
|
||||
{"messages": [{"role": "user", "content": "<sentence>"}], "label": [1.2, -0.6, 0.8]}
|
||||
```
|
||||
|
||||
### Embedding
|
||||
|
||||
Please refer to [Embedding training document](../BestPractices/Embedding.md#dataset-format).
|
||||
|
||||
### Reranker
|
||||
|
||||
Please refer to [Reranker training document](../BestPractices/Reranker.md#dataset-format).
|
||||
|
||||
### Multimodal
|
||||
|
||||
For multimodal datasets, the format is the same as the aforementioned tasks. The difference lies in the addition of several keys: `images`, `videos`, and `audios`, which represent the URLs or paths (preferably absolute paths) of multimodal resources. The tags `<image>`, `<video>`, and `<audio>` indicate where to insert images, videos, or audio. MS-Swift supports multiple images, videos, and audio files. These special tokens will be replaced during preprocessing, as referenced [here](https://github.com/modelscope/ms-swift/blob/main/swift/template/templates/qwen.py#L198). The four examples below respectively demonstrate the data format for plain text, as well as formats containing image, video, and audio data.
|
||||
|
||||
|
||||
Pre-training:
|
||||
```jsonl
|
||||
{"messages": [{"role": "assistant", "content": "Pre-trained text goes here"}]}
|
||||
{"messages": [{"role": "assistant", "content": "<image>is a puppy, <image>is a kitten"}], "images": ["/xxx/x.jpg", "/xxx/x.png"]}
|
||||
{"messages": [{"role": "assistant", "content": "<audio>describes how nice the weather is today"}], "audios": ["/xxx/x.wav"]}
|
||||
{"messages": [{"role": "assistant", "content": "<image>is an elephant, <video>is a lion running"}], "images": ["/xxx/x.jpg"], "videos": ["/xxx/x.mp4"]}
|
||||
```
|
||||
|
||||
Supervised Fine-tuning:
|
||||
|
||||
```jsonl
|
||||
{"messages": [{"role": "user", "content": "Where is the capital of Zhejiang?"}, {"role": "assistant", "content": "The capital of Zhejiang is Hangzhou."}]}
|
||||
{"messages": [{"role": "user", "content": "<image><image>What is the difference between the two images?"}, {"role": "assistant", "content": "The first one is a kitten, and the second one is a puppy."}], "images": ["/xxx/x.jpg", "/xxx/x.png"]}
|
||||
{"messages": [{"role": "user", "content": "<audio>What did the audio say?"}, {"role": "assistant", "content": "The weather is really nice today."}], "audios": ["/xxx/x.mp3"]}
|
||||
{"messages": [{"role": "system", "content": "You are a helpful and harmless assistant."}, {"role": "user", "content": "<image>What is in the image, <video>What is in the video?"}, {"role": "assistant", "content": "The image shows an elephant, and the video shows a puppy running on the grass."}], "images": ["/xxx/x.jpg"], "videos": ["/xxx/x.mp4"]}
|
||||
```
|
||||
- Note: The following fields will be automatically converted to the corresponding images, videos, and audios fields.
|
||||
- images: image, images.
|
||||
- videos: video, videos.
|
||||
- audios: audio, audios.
|
||||
- If you need to pass base64 data instead of file paths, here are sample examples: `"videos": ['data:video/mp4;base64,{base64_encoded}']`, `"images": ['data:image/jpg;base64,{base64_encoded}']`.
|
||||
- If you wish to directly pass in video frames instead of a video file, you can use the following format: `"videos": [["/xxx/x.png", "/xxx/y.png"], ["/xxx/a.png", "/xxx/b.png", "/xxx/c.png"]]`. This format is supported only by certain models, including Qwen2/2.5/3-VL, Qwen2.5/3-Omni, and their derivative models.
|
||||
|
||||
The data format for RLHF and sequence classification of multimodal models can reference the format of pure text large models, with additional fields such as `images` added on top of that.
|
||||
|
||||
#### Grounding
|
||||
|
||||
For grounding (object detection) tasks, ms-swift supports two methods:
|
||||
|
||||
1. Directly use the data format of the grounding task corresponding to the model. For example, the format for qwen2-vl is as follows:
|
||||
|
||||
```
|
||||
{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "<image>Describe the image."}, {"role": "assistant", "content": "<|object_ref_start|>a dog<|object_ref_end|><|box_start|>(221,423),(569,886)<|box_end|> and <|object_ref_start|>a woman<|object_ref_end|><|box_start|>(451,381),(733,793)<|box_end|> are playing on the beach"}], "images": ["/xxx/x.jpg"]}
|
||||
{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "<image>Find the <|object_ref_start|>sheep<|object_ref_end|> in the image"}, {"role": "assistant", "content": "<|box_start|>(101,201),(150,266)<|box_end|><|box_start|>(401,601),(550,666)<|box_end|>"}], "images": ["/xxx/x.jpg"]}
|
||||
{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "<image>Help me open Google Chrome"}, {"role": "assistant", "content": "Action: click(start_box='<|box_start|>(246,113)<|box_end|>')"}], "images": ["/xxx/x.jpg"]}
|
||||
```
|
||||
|
||||
When using this type of data, please note:
|
||||
|
||||
- Different models have different special characters and data format for the grounding task.
|
||||
- The handling of bounding box normalization varies across different models: for example, qwen2.5-vl uses absolute coordinates, while qwen2/3-vl and internvl2.5 require bounding box coordinates to be normalized to the thousandth scale.
|
||||
- Note: Qwen2.5-VL uses absolute coordinates, so you need to be careful with image resizing each time. If you use the dataset format from Option 1, you need to resize the images in advance (height and width must be multiples of 28) and scale the coordinates accordingly. If you use the dataset format from Option 2, ms-swift will handle image resizing for you. You can still use `MAX_PIXELS` or `--max_pixels` for image resizing (training only; for inference, you still need to handle image resizing yourself).
|
||||
|
||||
|
||||
2. Use ms-swift's grounding data format:
|
||||
|
||||
```
|
||||
{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "<image>Describe the image."}, {"role": "assistant", "content": "<ref-object><bbox> and <ref-object><bbox> are playing on the beach"}], "images": ["/xxx/x.jpg"], "objects": {"ref": ["a dog", "a woman"], "bbox": [[331.5, 761.4, 853.5, 1594.8], [676.5, 685.8, 1099.5, 1427.4]]}}
|
||||
{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "<image>Find the <ref-object> in the image"}, {"role": "assistant", "content": "<bbox><bbox>"}], "images": ["/xxx/x.jpg"], "objects": {"ref": ["sheep"], "bbox": [[90.9, 160.8, 135, 212.8], [360.9, 480.8, 495, 532.8]]}}
|
||||
{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "<image>Help me open Google Chrome"}, {"role": "assistant", "content": "Action: click(start_box='<bbox>')"}], "images": ["/xxx/x.jpg"], "objects": {"ref": [], "bbox": [[615, 226]]}}
|
||||
```
|
||||
|
||||
The format will automatically convert the dataset format to the corresponding model's grounding task format and select the appropriate model's bbox normalization method. Compared to the general format, this format includes an additional "objects" field, which contains the following subfields:
|
||||
|
||||
- ref: Used to replace the `<ref-object>` placeholder in messages. The length of `ref` should match the number of `<ref-object>` instances.
|
||||
- bbox: Used to replace the `<bbox>` placeholder in messages. If the length of each box in the bbox is 2, it represents the x and y coordinates. If the box length is 4, it represents the x and y coordinates of two points. The length of `bbox` should match the number of `<bbox>` instances.
|
||||
- Note: `<ref-object>` and `<bbox>` do not have a corresponding relationship; references and bounding boxes replace their own placeholders separately.
|
||||
- bbox_type: Optional values are 'real' and 'norm1'. The default is 'real', meaning the bbox represents the actual bounding box value. If set to 'norm1', the bbox is normalized to the range 0~1.
|
||||
- image_id: Typically used for multi-image grounding tasks. This parameter only takes effect when bbox_type is 'real', representing which image the bbox corresponds to, used for scaling the bbox. The index starts from 0, and defaults to all being the 0th image. The length of image_id needs to be consistent with the length of bbox. For example: if the length of bbox is 10 and the length of images is 2, then the length of image_id needs to be 10, with values within the set `{0, 1}`.
|
||||
|
||||
For Qwen2.5-VL/Qwen3-VL, you can set the environment variable `QWENVL_BBOX_FORMAT='new'` (default is `'legacy'`) to be compatible with the [official cookbook](https://github.com/QwenLM/Qwen3-VL/blob/main/cookbooks/2d_grounding.ipynb) format. Define your dataset in the following format:
|
||||
```jsonl
|
||||
{"messages": [{"role": "user", "content": "<image>Locate the <ref-object> in the image"}, {"role": "assistant", "content": "[\n\t{\"bbox_2d\": <bbox>, \"label\": \"<ref-object>\"},\n\t{\"bbox_2d\": <bbox>, \"label\": \"<ref-object>\"}\n]"}], "images": ["cat.png"], "objects": {"ref": ["sheep", "sheep", "sheep"], "bbox": [[90.9, 160.8, 135, 212.8], [360.9, 480.8, 495, 532.8]]}}
|
||||
```
|
||||
|
||||
Testing the final format of the grounding data in ms-swift format:
|
||||
```python
|
||||
import os
|
||||
os.environ["MAX_PIXELS"] = "1003520"
|
||||
from swift import get_processor, get_template
|
||||
|
||||
processor = get_processor('Qwen/Qwen2.5-VL-7B-Instruct')
|
||||
template = get_template(processor)
|
||||
data = {...}
|
||||
template.set_mode('train')
|
||||
encoded = template.encode(data, return_template_inputs=True)
|
||||
print(f'[INPUT_IDS] {template.safe_decode(encoded["input_ids"])}\n')
|
||||
print(f'[LABELS] {template.safe_decode(encoded["labels"])}')
|
||||
print(f'images: {encoded["template_inputs"].images}')
|
||||
```
|
||||
|
||||
|
||||
### Agent Format
|
||||
Here are example data samples for a text-only Agent and a multimodal Agent:
|
||||
```jsonl
|
||||
{"tools": "[{\"type\": \"function\", \"function\": {\"name\": \"realtime_aqi\", \"description\": \"Weather forecast. Get real-time air quality, including current air quality, PM2.5, and PM10 information.\", \"parameters\": {\"type\": \"object\", \"properties\": {\"city\": {\"type\": \"string\", \"description\": \"City name, e.g., Shanghai\"}}, \"required\": [\"city\"]}}}]", "messages": [{"role": "user", "content": "What is the weather like in Beijing and Shanghai today?"}, {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"Beijing\"}}"}, {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"Shanghai\"}}"}, {"role": "tool_response", "content": "{\"city\": \"Beijing\", \"aqi\": \"10\", \"unit\": \"celsius\"}"}, {"role": "tool_response", "content": "{\"city\": \"Shanghai\", \"aqi\": \"72\", \"unit\": \"fahrenheit\"}"}, {"role": "assistant", "content": "According to the weather forecast tool, the air quality index (AQI) in Beijing is 10, which indicates good air quality; whereas in Shanghai, the AQI is 72, indicating mild pollution."}]}
|
||||
{"tools": "[{\"type\": \"function\", \"function\": {\"name\": \"click\", \"description\": \"Click on a position on the screen\", \"parameters\": {\"type\": \"object\", \"properties\": {\"x\": {\"type\": \"integer\", \"description\": \"X-coordinate representing the horizontal position on the screen\"}, \"y\": {\"type\": \"integer\", \"description\": \"Y-coordinate representing the vertical position on the screen\"}}, \"required\": [\"x\", \"y\"]}}}]", "messages": [{"role": "user", "content": "<image>What time is it now?"}, {"role": "assistant", "content": "<think>\nI can check the current time by opening the calendar app.\n</think>\n"}, {"role": "tool_call", "content": "{\"name\": \"click\", \"arguments\": {\"x\": 105, \"y\": 132}}"}, {"role": "tool_response", "content": "{\"images\": \"<image>\", \"status\": \"success\"}"}, {"role": "assistant", "content": "Successfully opened the calendar app. The current time is 11 o'clock in the morning."}], "images": ["desktop.png", "calendar.png"]}
|
||||
```
|
||||
- When the `agent_template` is set to "react_en", "hermes", etc., this format is compatible with training for all model Agents and allows easy switching between different models.
|
||||
- Among them, `tools` is a JSON string containing a list of tools, and the `content` section of `messages` where the `role` is `'tool_call'` or `'tool_response/tool'` must also be a JSON string.
|
||||
- The `tools` field will be combined with the `{"role": "system", ...}` section during training/inference according to the `agent_template`, forming a complete system section.
|
||||
- The `{"role": "tool_call", ...}` part will automatically be converted into corresponding formats of `{"role": "assistant", ...}` based on the `agent_template`. Multiple consecutive `{"role": "assistant", ...}` entries will be concatenated to form a complete assistant_content.
|
||||
- The `{"role": "tool_response", ...}` can also be written as `{"role": "tool", ...}`, these two forms are equivalent. This part will also be automatically converted according to the `agent_template`. During training, this part does not participate in loss calculations, similar to `{"role": "user", ...}`.
|
||||
- This format supports parallel tool calls; refer to the first data sample for an example. In multimodal Agent data samples, the number of `<image>` tags should match the length of "images", and their positions indicate where the image features are inserted. It also supports other modalities, such as audios and videos.
|
||||
- Note: You can also manually process the data into the messages format with roles set to system, user, or assistant. The purpose of agent_template is to automatically map the tools field and the messages with roles tool_call and tool_response into the standard messages format with roles system, user, and assistant.
|
||||
- For more details, please refer to [Agent Documentation](../Instruction/Agent-support.md).
|
||||
|
||||
### Text-to-Image Format
|
||||
|
||||
```jsonl
|
||||
{"messages": [{"role": "system", "content": "You are a useful and harmless assistant"}, {"role": "user", "content": "Draw me an apple"}, {"role": "assistant", "content": "<image>"}], "images": ["/xxx/x.jpg"]}
|
||||
```
|
||||
|
||||
## dataset_info.json
|
||||
|
||||
You can refer to the ms-swift built-in [dataset_info.json](https://github.com/modelscope/ms-swift/blob/main/swift/dataset/data/dataset_info.json). This approach uses the AutoPreprocessor function to convert the dataset into a standard format. The dataset_info.json file contains a list of metadata about the dataset. Here are some examples:
|
||||
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"ms_dataset_id": "xxx/xxx"
|
||||
},
|
||||
{
|
||||
"dataset_path": "<dataset_dir/dataset_path>"
|
||||
},
|
||||
{
|
||||
"ms_dataset_id": "<dataset_id>",
|
||||
"subsets": ["v1"],
|
||||
"split": ["train", "validation"],
|
||||
"columns": {
|
||||
"input": "query",
|
||||
"output": "response"
|
||||
}
|
||||
},
|
||||
{
|
||||
"ms_dataset_id": "<dataset_id>",
|
||||
"hf_dataset_id": "<hf_dataset_id>",
|
||||
"subsets": [{
|
||||
"subset": "subset1",
|
||||
"columns": {
|
||||
"problem": "query",
|
||||
"content": "response"
|
||||
}
|
||||
},
|
||||
{
|
||||
"subset": "subset2",
|
||||
"columns": {
|
||||
"messages": "_",
|
||||
"new_messages": "messages"
|
||||
}
|
||||
}]
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
The following parameters are supported:
|
||||
|
||||
- ms_dataset_id: Refers to the DatasetMeta parameter.
|
||||
- hf_dataset_id: Refers to the DatasetMeta parameter.
|
||||
- dataset_path: Refers to the DatasetMeta parameter.
|
||||
- dataset_name: Refers to the DatasetMeta parameter.
|
||||
- subsets: Refers to the DatasetMeta parameter.
|
||||
- split: Refers to the DatasetMeta parameter.
|
||||
- columns: Transforms column names before preprocessing the dataset.
|
||||
|
||||
## Dataset Registration
|
||||
|
||||
`register_dataset` will register the dataset in `DATASET_MAPPING`. You can call the function `register_dataset(dataset_meta)` to complete the dataset registration, where `dataset_meta` will store the metadata of the model. The parameter list for DatasetMeta is as follows:
|
||||
|
||||
- ms_dataset_id: The dataset_id for ModelScope, default is None.
|
||||
- hf_dataset_id: The dataset_id for HuggingFace, default is None.
|
||||
- dataset_path: Local path to the **dataset file/folder** (absolute path recommended). Default is None.
|
||||
- dataset_name: The alias of the dataset, which can be specified via `--dataset <dataset_name>`. This is very convenient when the dataset_path is long. The default value is None.
|
||||
- subsets: A list of subdataset names or a list of `SubsetDataset` objects, default is `['default']`. (The concepts of subdatasets and splits only exist for dataset_id or dataset_dir (open source datasets cloned via git)).
|
||||
- split: Defaults to `['train']`.
|
||||
- preprocess_func: A preprocessing function or callable object, default is `AutoPreprocessor()`. This preprocessing function takes an `HfDataset` as input and returns an `HfDataset` in the standard format.
|
||||
- load_function: Defaults to `DatasetLoader.load`. If a custom loading function is needed, it should return an `HfDataset` in the standard format, allowing users maximum flexibility while bypassing the ms-swift dataset loading mechanism. This parameter usually does not need to be modified.
|
||||
|
||||
|
||||
Below are examples of registering datasets:
|
||||
|
||||
```python
|
||||
from swift.dataset import (
|
||||
ResponsePreprocessor, DatasetMeta, register_dataset, SubsetDataset, load_dataset
|
||||
)
|
||||
from typing import Dict, Any
|
||||
|
||||
class CustomPreprocessor(ResponsePreprocessor):
|
||||
def preprocess(self, row: Dict[str, Any]) -> Dict[str, Any]:
|
||||
query = f"""Task: Judge whether the two sentences below are semantically similar.
|
||||
Sentence 1: {row['text1']}
|
||||
Sentence 2: {row['text2']}
|
||||
Output the category [0/1]: 0 for different meanings, 1 for similar meanings.
|
||||
"""
|
||||
response = str(row['label'])
|
||||
row = {
|
||||
'query': query,
|
||||
'response': response
|
||||
}
|
||||
return super().preprocess(row)
|
||||
|
||||
|
||||
register_dataset(
|
||||
DatasetMeta(
|
||||
ms_dataset_id='swift/financial_classification',
|
||||
subsets=[SubsetDataset('train', split=['train']), SubsetDataset('test', split=['test'])],
|
||||
preprocess_func=CustomPreprocessor(),
|
||||
))
|
||||
|
||||
if __name__ == '__main__':
|
||||
# load_dataset returns train_dataset and val_dataset based on `split_dataset_ratio`
|
||||
# Here, since we didn't pass `split_dataset_ratio` (defaults to 0), we take the first one (index 0)
|
||||
dataset = load_dataset('swift/financial_classification:train')[0]
|
||||
test_dataset = load_dataset('swift/financial_classification:test')[0]
|
||||
print(f'dataset[0]: {dataset[0]}')
|
||||
print(f'test_dataset[0]: {test_dataset[0]}')
|
||||
```
|
||||
@@ -0,0 +1,39 @@
|
||||
# Custom Model
|
||||
|
||||
The models built into ms-swift can be used directly by specifying either `model_id` or `model_path`: `--model <model_id_or_path>`. ms-swift determines the `model_type` based on the suffix of `model_id/model_path` and the `config.json` file.
|
||||
|
||||
Each `model_type` has a unique model structure, template, and loading method. Of course, you can also manually override these by passing `--model_type` and `--template`. You can check the supported `model_type` and templates in the [Supported Models and Datasets](../Instruction/Supported-models-and-datasets.md).
|
||||
|
||||
The following introduces how to register a new model and its corresponding template. For best practices, refer to [Best Practices for Registering Multimodal Models](../BestPractices/MLLM-Registration.md).
|
||||
|
||||
## Model Registration
|
||||
|
||||
Custom models are typically implemented using model registration. You can refer to the [built-in model](https://github.com/modelscope/ms-swift/blob/main/swift/model/models/qwen.py), the [built-in dialogue template](https://github.com/modelscope/ms-swift/blob/main/swift/template/templates/qwen.py), or the example code in the [examples](https://github.com/modelscope/ms-swift/blob/main/examples/custom). You can specify the `--external_plugins xxx.py` to parse the externally registered content, which is convenient for users installing via pip instead of git clone.
|
||||
|
||||
The `register_model` function registers a model in the `MODEL_MAPPING`. You can complete the model registration by calling the function `register_model(model_meta)`, where `model_meta` will store the model's metadata. The parameter list for ModelMeta is as follows:
|
||||
|
||||
- model_type: Required. The model type, which is also the unique ID.
|
||||
- model_groups: Required. Lists the ModelScope/HuggingFace model IDs and local paths. Running the [run_model_info.py](https://github.com/modelscope/ms-swift/blob/main/scripts/utils/run_model_info.py) file will automatically generate the [supported models documentation](https://swift.readthedocs.io/en/latest/Instruction/Supported-models-and-datasets.html) and automatically match the model_type based on the `--model` suffix.
|
||||
- loader: The loader for model and tokenizer/processor (multimodal models). Defaults to `swift.model.ModelLoader`.
|
||||
- template: The default template type when `--template` is not additionally specified in the command line. Defaults to None.
|
||||
- model_arch: The model architecture. Defaults to None. Multi-modal model training requires setting this parameter to determine the prefix for llm/vit/aligner.
|
||||
- architectures: The architectures item in config.json, used to automatically match the model with its model_type. Defaults to `[]`.
|
||||
- additional_saved_files: Files that need to be additionally saved during full parameter training and merge-lora. Defaults to `[]`.
|
||||
- torch_dtype: The default dtype when `torch_dtype` is not passed during model loading. Defaults to None, read from config.json.
|
||||
- is_multimodal: Indicates whether the model is multi-modal. Defaults to False.
|
||||
- ignore_patterns: File patterns to be ignored when downloading from the hub. Defaults to `[]`.
|
||||
|
||||
The `register_template` function registers a dialogue template in `TEMPLATE_MAPPING`. To complete the registration of the dialogue template, simply call the function `register_template(template_meta)`, where `template_meta` will store the metadata of the template. The parameter list for TemplateMeta is as follows:
|
||||
|
||||
- template_type: Required. The type of dialogue template, which also serves as a unique ID.
|
||||
- prefix: Required. The prefix of the dialogue template, usually encompassing parts like system, bos_token, and is generated independently of multi-turn dialogue loops. For example, the prefix for qwen is `[]`.
|
||||
- prompt: Required. Represents the dialogue portion before `{{RESPONSE}}`. We use `{{QUERY}}` as a placeholder for the user's inquiry part. For example, the prompt for qwen is `['<|im_start|>user\n{{QUERY}}<|im_end|>\n<|im_start|>assistant\n']`.
|
||||
- chat_sep: Required. The separator for each turn in multi-turn dialogues. If set to None, the template does not support multi-turn dialogue. For example, the chat_sep for qwen is `['<|im_end|>\n']`.
|
||||
- suffix: Defaults to `[['eos_token_id']]`. The suffix part of the dialogue template, generated independently of multi-turn dialogue loops, usually the eos_token. For example, the suffix for qwen is `['<|im_end|>']`.
|
||||
- template_cls: Defaults to `Template`. Customization is generally required when defining templates for multimodal models, particularly in customizing the `_encode`, `_post_encode`, and `_data_collator` functions.
|
||||
- system_prefix: Defaults to None. The prefix for dialogue templates with a system. We use`{{SYSTEM}}`as a placeholder for the system. For example, the system_prefix for qwen is`['<|im_start|>system\n{{SYSTEM}}<|im_end|>\n']`.
|
||||
- Note: If the system is empty and `prefix` can be replaced by `system_prefix`, you can write `prefix` as a prefix including the system without setting `system_prefix`.
|
||||
- If the prefix does not include `{{SYSTEM}}` and system_prefix is not set, the template does not support the system.
|
||||
- default_system: Defaults to None. The default system used when `--system` is not provided. For example, the default_system for qwen is `'You are a helpful assistant.'`.
|
||||
- stop_words: Defaults to`[]`. Additional stop words besides eos_token and`suffix[-1]`. For example, the stop_words for qwen is`['<|endoftext|>']`
|
||||
- Note: During inference, the output response will be filtered by eos_token and `suffix[-1]`, but additional stop_words will be retained.
|
||||
Reference in New Issue
Block a user