wehub-resource-sync
1093 lines
54 KiB
ReStructuredText
1093 lines
54 KiB
ReStructuredText
.. _compile-model-libraries:
|
|
|
|
Compile Model Libraries
|
|
=======================
|
|
|
|
To run a model with MLC LLM in any platform, we need:
|
|
|
|
1. **Model weights** converted to MLC format (e.g. `RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC <https://huggingface.co/mlc-ai/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/tree/main>`__.)
|
|
2. **Model library** that comprises the inference logic
|
|
|
|
This page describes how to compile a model library with MLC LLM. Model compilation optimizes
|
|
the model inference for a given platform, allowing users bring their own new model
|
|
architecture, use different quantization modes, and customize the overall model
|
|
optimization flow.
|
|
|
|
|
|
|
|
Notably, in many cases you do not need to explicit call compile.
|
|
|
|
- If you are using the Python API, you can skip specifying ``model_lib`` and
|
|
the system will JIT compile the library.
|
|
|
|
- If you are building iOS/android package, checkout :ref:`package-libraries-and-weights`,
|
|
which provides a simpler high-level command that leverages the compile behind the scheme.
|
|
|
|
|
|
This page is still helpful to understand the compilation flow behind the scheme,
|
|
or be used to explicit create model libraries.
|
|
We compile ``RedPajama-INCITE-Chat-3B-v1`` with ``q4f16_1`` as an example for all platforms.
|
|
|
|
.. note::
|
|
Before you proceed, make sure you followed :ref:`install-tvm`, a required
|
|
backend to compile models with MLC LLM.
|
|
|
|
Please also follow the instructions in :ref:`deploy-cli` / :ref:`deploy-python-engine` to obtain
|
|
the CLI app / Python API that can be used to chat with the compiled model.
|
|
|
|
|
|
.. contents:: Table of Contents
|
|
:depth: 1
|
|
:local:
|
|
|
|
1. Verify Installation
|
|
----------------------
|
|
|
|
**Step 1. Verify mlc_llm**
|
|
|
|
We use the python package ``mlc_llm`` to compile models. This can be installed by
|
|
following :ref:`install-mlc-packages`, either by building from source, or by
|
|
installing the prebuilt package. Verify ``mlc_llm`` installation in command line via:
|
|
|
|
.. code:: bash
|
|
|
|
$ mlc_llm --help
|
|
# You should see help information with this line
|
|
usage: MLC LLM Command Line Interface. [-h] {compile,convert_weight,gen_config}
|
|
|
|
.. note::
|
|
If it runs into error ``command not found: mlc_llm``, try ``python -m mlc_llm --help``.
|
|
|
|
**Step 2. Verify TVM**
|
|
|
|
To compile models, you also need to follow :ref:`install-tvm`.
|
|
Here we verify ``tvm`` quickly with command line (for full verification, see :ref:`tvm-validate`):
|
|
|
|
.. code:: bash
|
|
|
|
$ python -c "import tvm; print(tvm.__file__)"
|
|
/some-path/lib/python3.13/site-packages/tvm/__init__.py
|
|
|
|
1. Clone from HF and convert_weight
|
|
-----------------------------------
|
|
|
|
This replicates :ref:`convert-weights-via-MLC`, see that page for more details.
|
|
|
|
You can be under the mlc-llm repo, or your own working directory. Note that all platforms
|
|
can share the same compiled/quantized weights.
|
|
|
|
.. code:: shell
|
|
|
|
# Create directory
|
|
mkdir -p dist/models && cd dist/models
|
|
# Clone HF weights
|
|
git lfs install
|
|
git clone https://huggingface.co/togethercomputer/RedPajama-INCITE-Chat-3B-v1
|
|
cd ../..
|
|
# Convert weight
|
|
mlc_llm convert_weight ./dist/models/RedPajama-INCITE-Chat-3B-v1/ \
|
|
--quantization q4f16_1 \
|
|
-o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC
|
|
|
|
2. Generate mlc-chat-config and compile
|
|
---------------------------------------
|
|
|
|
A model library is specified by:
|
|
|
|
- The model architecture (e.g. ``llama-2``, ``gpt-neox``)
|
|
- Quantization (e.g. ``q4f16_1``, ``q0f32``)
|
|
- Metadata (e.g. ``context_window_size``, ``sliding_window_size``, ``prefill-chunk-size``), which affects memory planning
|
|
- Platform (e.g. ``cuda``, ``webgpu``, ``iOS``)
|
|
|
|
All these knobs are specified in ``mlc-chat-config.json`` generated by ``gen_config``.
|
|
|
|
.. code:: shell
|
|
|
|
# Create output directory for the model library compiled
|
|
mkdir dist/libs
|
|
|
|
.. tabs::
|
|
|
|
.. group-tab:: Linux - CUDA
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ \
|
|
--quantization q4f16_1 --conv-template redpajama_chat \
|
|
-o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \
|
|
--device cuda -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-cuda.so
|
|
|
|
|
|
.. group-tab:: Metal
|
|
|
|
For M-chip Mac:
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ \
|
|
--quantization q4f16_1 --conv-template redpajama_chat \
|
|
-o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \
|
|
--device metal -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-metal.so
|
|
|
|
Cross-Compiling for Intel Mac on M-chip Mac:
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ \
|
|
--quantization q4f16_1 --conv-template redpajama_chat \
|
|
-o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \
|
|
--device metal:x86-64 -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-metal_x86_64.dylib
|
|
|
|
For Intel Mac:
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ \
|
|
--quantization q4f16_1 --conv-template redpajama_chat \
|
|
-o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \
|
|
--device metal -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-metal_x86_64.dylib
|
|
|
|
|
|
.. group-tab:: Vulkan
|
|
|
|
For Linux:
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ \
|
|
--quantization q4f16_1 --conv-template redpajama_chat \
|
|
-o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \
|
|
--device vulkan -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-vulkan.so
|
|
|
|
For Windows:
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ \
|
|
--quantization q4f16_1 --conv-template redpajama_chat \
|
|
-o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \
|
|
--device vulkan -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-vulkan.dll
|
|
|
|
.. group-tab:: iOS/iPadOS
|
|
|
|
You need a Mac to compile models for it.
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ --quantization q4f16_1 \
|
|
--conv-template redpajama_chat --context-window-size 768 \
|
|
-o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \
|
|
--device iphone -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-iphone.tar
|
|
|
|
.. note::
|
|
If it runs into error
|
|
|
|
.. code:: text
|
|
|
|
Compilation error:
|
|
xcrun: error: unable to find utility "metal", not a developer tool or in PATH
|
|
xcrun: error: unable to find utility "metallib", not a developer tool or in PATH
|
|
|
|
, please check and make sure you have Command Line Tools for Xcode installed correctly.
|
|
You can use ``xcrun metal`` to validate: when it prints ``metal: error: no input files``, it means the Command Line Tools for Xcode is installed and can be found, and you can proceed with the model compiling.
|
|
|
|
.. group-tab:: Android
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ --quantization q4f16_1 \
|
|
--conv-template redpajama_chat --context-window-size 768 \
|
|
-o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \
|
|
--device android -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-android.tar
|
|
|
|
.. group-tab:: WebGPU
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ \
|
|
--quantization q4f16_1 --conv-template redpajama_chat \
|
|
-o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \
|
|
--device webgpu -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-webgpu.wasm
|
|
|
|
.. note::
|
|
To compile for webgpu, you need to build from source when installing ``mlc_llm``. Besides, you also need to follow :ref:`install-web-build`.
|
|
Otherwise, it would run into error
|
|
|
|
.. code:: text
|
|
|
|
RuntimeError: Cannot find libraries: wasm_runtime.bc
|
|
|
|
.. note::
|
|
For webgpu, when compiling larger models like ``Llama-2-7B``, you may want to add ``--prefill-chunk-size 1024`` or lower ``--context-window-size`` to decrease memory usage.
|
|
Otherwise, you may run into issues like:
|
|
|
|
.. code:: text
|
|
|
|
TypeError: Failed to execute 'createBuffer' on 'GPUDevice': Failed to read the 'size' property from
|
|
'GPUBufferDescriptor': Value is outside the 'unsigned long long' value range.
|
|
|
|
.. note::
|
|
|
|
For the ``conv-template``, `conversation_template.py <https://github.com/mlc-ai/mlc-llm/blob/main/python/mlc_llm/conversation_template.py>`__
|
|
contains a full list of conversation templates that MLC provides. If the model you are adding
|
|
requires a new conversation template, you would need to add your own.
|
|
Follow `this PR <https://github.com/mlc-ai/mlc-llm/pull/2163>`__ as an example.
|
|
However, adding your own template would require you :ref:`build mlc_llm from source <mlcchat_build_from_source>`
|
|
in order for it to be recognized by the runtime.
|
|
|
|
For more details, please see :ref:`configure-mlc-chat-json`.
|
|
|
|
3. Verify output and chat
|
|
-------------------------
|
|
|
|
By executing the compile command above, we generate the model weights, model lib, and a chat config.
|
|
We can check the output with the commands below:
|
|
|
|
.. tabs::
|
|
|
|
.. group-tab:: Linux - CUDA
|
|
|
|
.. code:: shell
|
|
|
|
~/mlc-llm > ls dist/libs
|
|
RedPajama-INCITE-Chat-3B-v1-q4f16_1-cuda.so # ===> the model library
|
|
|
|
~/mlc-llm > ls dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC
|
|
mlc-chat-config.json # ===> the chat config
|
|
tensor-cache.json # ===> the model weight info
|
|
params_shard_0.bin # ===> the model weights
|
|
params_shard_1.bin
|
|
...
|
|
tokenizer.json # ===> the tokenizer files
|
|
tokenizer_config.json
|
|
|
|
We can now chat with the model using the command line interface (CLI) app or the Python API.
|
|
|
|
.. code:: shell
|
|
|
|
python
|
|
>>> from mlc_llm import MLCEngine
|
|
>>> engine = MLCEngine(model="./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC",
|
|
... model_lib="./dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-cuda.so")
|
|
>>> engine.chat.completions.create(
|
|
... messages=[{"role": "user", "content": "hello"}]
|
|
... )
|
|
ChatCompletionResponse(
|
|
choices=[ChatCompletionResponseChoice(
|
|
message=ChatCompletionMessage(
|
|
content="Hi! How can I assist you today?", role='assistant'
|
|
)
|
|
)],
|
|
...
|
|
)
|
|
|
|
.. group-tab:: Metal
|
|
|
|
.. code:: shell
|
|
|
|
~/mlc-llm > ls dist/libs
|
|
RedPajama-INCITE-Chat-3B-v1-q4f16_1-metal.so # ===> the model library (will be -metal_x86_64.dylib for Intel Mac)
|
|
|
|
~/mlc-llm > ls dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC
|
|
mlc-chat-config.json # ===> the chat config
|
|
tensor-cache.json # ===> the model weight info
|
|
params_shard_0.bin # ===> the model weights
|
|
params_shard_1.bin
|
|
...
|
|
tokenizer.json # ===> the tokenizer files
|
|
tokenizer_config.json
|
|
|
|
We can now chat with the model using the command line interface (CLI) app or the Python API.
|
|
|
|
.. code:: shell
|
|
|
|
python
|
|
>>> from mlc_llm import MLCEngine
|
|
>>> engine = MLCEngine(model="./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC",
|
|
... model_lib="./dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-metal.so")
|
|
>>> engine.chat.completions.create(
|
|
... messages=[{"role": "user", "content": "hello"}]
|
|
... )
|
|
ChatCompletionResponse(
|
|
choices=[ChatCompletionResponseChoice(
|
|
message=ChatCompletionMessage(
|
|
content="Hi! How can I assist you today?", role='assistant'
|
|
)
|
|
)],
|
|
...
|
|
)
|
|
|
|
|
|
.. group-tab:: Vulkan
|
|
|
|
.. code:: shell
|
|
|
|
~/mlc-llm > ls dist/libs
|
|
RedPajama-INCITE-Chat-3B-v1-q4f16_1-vulkan.so # ===> the model library (will be .dll for Windows)
|
|
|
|
~/mlc-llm > ls dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC
|
|
mlc-chat-config.json # ===> the chat config
|
|
tensor-cache.json # ===> the model weight info
|
|
params_shard_0.bin # ===> the model weights
|
|
params_shard_1.bin
|
|
...
|
|
tokenizer.json # ===> the tokenizer files
|
|
tokenizer_config.json
|
|
|
|
We can now chat with the model using the command line interface (CLI) app or the Python API.
|
|
|
|
.. code:: shell
|
|
|
|
python
|
|
>>> from mlc_llm import MLCEngine
|
|
>>> engine = MLCEngine(model="./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC",
|
|
... model_lib="./dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-vulkan.so")
|
|
>>> engine.chat.completions.create(
|
|
... messages=[{"role": "user", "content": "hello"}]
|
|
... )
|
|
ChatCompletionResponse(
|
|
choices=[ChatCompletionResponseChoice(
|
|
message=ChatCompletionMessage(
|
|
content="Hi! How can I assist you today?", role='assistant'
|
|
)
|
|
)],
|
|
...
|
|
)
|
|
|
|
.. group-tab:: iOS/iPadOS
|
|
|
|
.. code:: shell
|
|
|
|
~/mlc-llm > ls dist/libs
|
|
RedPajama-INCITE-Chat-3B-v1-q4f16_1-iphone.tar # ===> the model library
|
|
|
|
~/mlc-llm > ls dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC
|
|
mlc-chat-config.json # ===> the chat config
|
|
tensor-cache.json # ===> the model weight info
|
|
params_shard_0.bin # ===> the model weights
|
|
params_shard_1.bin
|
|
...
|
|
tokenizer.json # ===> the tokenizer files
|
|
tokenizer_config.json
|
|
|
|
The model lib ``dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-iphone.tar``
|
|
will be packaged as a static library into the iOS app. Checkout :ref:`deploy-ios` for more details.
|
|
|
|
.. group-tab:: Android
|
|
|
|
.. code:: shell
|
|
|
|
~/mlc-llm > ls dist/libs
|
|
RedPajama-INCITE-Chat-3B-v1-q4f16_1-android.tar # ===> the model library
|
|
|
|
~/mlc-llm > ls dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC
|
|
mlc-chat-config.json # ===> the chat config
|
|
tensor-cache.json # ===> the model weight info
|
|
params_shard_0.bin # ===> the model weights
|
|
params_shard_1.bin
|
|
...
|
|
tokenizer.json # ===> the tokenizer files
|
|
tokenizer_config.json
|
|
|
|
The model lib ``dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-android.tar``
|
|
will be packaged as a static library into the android app. Checkout :ref:`deploy-android` for more details.
|
|
|
|
.. group-tab:: WebGPU
|
|
|
|
.. code:: shell
|
|
|
|
~/mlc-llm > ls dist/libs
|
|
RedPajama-INCITE-Chat-3B-v1-q4f16_1-webgpu.wasm # ===> the model library
|
|
|
|
~/mlc-llm > ls dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC
|
|
mlc-chat-config.json # ===> the chat config
|
|
tensor-cache.json # ===> the model weight info
|
|
params_shard_0.bin # ===> the model weights
|
|
params_shard_1.bin
|
|
...
|
|
tokenizer.json # ===> the tokenizer files
|
|
tokenizer_config.json
|
|
|
|
To use this in WebGPU runtime, checkout :ref:`webllm-runtime`.
|
|
|
|
Compile Commands for More Models
|
|
--------------------------------
|
|
|
|
This section lists compile commands for more models that you can try out. Note that this can be easily
|
|
generalized to any model variant, as long as mlc-llm supports the architecture.
|
|
|
|
.. tabs::
|
|
|
|
.. tab:: Model: Llama-2-7B
|
|
|
|
Please `request for access <https://huggingface.co/meta-llama>`_ to the Llama-2 weights from Meta first.
|
|
After granted access, first create directory ``dist/models`` and download the model to the directory.
|
|
For example, you can run the following code:
|
|
|
|
.. code:: shell
|
|
|
|
mkdir -p dist/models && cd dist/models
|
|
git lfs install
|
|
git clone https://huggingface.co/meta-llama/Llama-2-7b-chat-hf
|
|
cd ../..
|
|
|
|
Then convert the HF weights into MLC-compatible weights. Note that all platforms
|
|
can share the same compiled/quantized weights.
|
|
|
|
.. code:: shell
|
|
|
|
mlc_llm convert_weight ./dist/models/Llama-2-7b-chat-hf/ --quantization q4f16_1 -o dist/Llama-2-7b-chat-hf-q4f16_1-MLC
|
|
|
|
Afterwards, run the following command to generate mlc config and compile the model.
|
|
|
|
.. code:: shell
|
|
|
|
# Create output directory for the model library compiled
|
|
mkdir dist/libs
|
|
|
|
.. tabs::
|
|
|
|
.. tab:: Target: CUDA
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/Llama-2-7b-chat-hf/ --quantization q4f16_1 \
|
|
--conv-template llama-2 -o dist/Llama-2-7b-chat-hf-q4f16_1-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/Llama-2-7b-chat-hf-q4f16_1-MLC/mlc-chat-config.json \
|
|
--device cuda -o dist/libs/Llama-2-7b-chat-hf-q4f16_1-cuda.so
|
|
|
|
.. tab:: Metal
|
|
|
|
For M-chip Mac:
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/Llama-2-7b-chat-hf/ --quantization q4f16_1 \
|
|
--conv-template llama-2 -o dist/Llama-2-7b-chat-hf-q4f16_1-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/Llama-2-7b-chat-hf-q4f16_1-MLC/mlc-chat-config.json \
|
|
--device metal -o dist/libs/Llama-2-7b-chat-hf-q4f16_1-metal.so
|
|
|
|
Cross-Compiling for Intel Mac on M-chip Mac:
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ \
|
|
--quantization q4f16_1 --conv-template redpajama_chat \
|
|
-o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \
|
|
--device metal:x86-64 -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-metal_x86_64.dylib
|
|
|
|
For Intel Mac:
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/Llama-2-7b-chat-hf/ --quantization q4f16_1 \
|
|
--conv-template llama-2 -o dist/Llama-2-7b-chat-hf-q4f16_1-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/Llama-2-7b-chat-hf-q4f16_1-MLC/mlc-chat-config.json \
|
|
--device metal -o dist/libs/Llama-2-7b-chat-hf-q4f16_1-metal_x86_64.dylib
|
|
|
|
.. tab:: Vulkan
|
|
|
|
For Linux:
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/Llama-2-7b-chat-hf/ --quantization q4f16_1 \
|
|
--conv-template llama-2 -o dist/Llama-2-7b-chat-hf-q4f16_1-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/Llama-2-7b-chat-hf-q4f16_1-MLC/mlc-chat-config.json \
|
|
--device vulkan -o dist/libs/Llama-2-7b-chat-hf-q4f16_1-vulkan.so
|
|
|
|
For Windows:
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/Llama-2-7b-chat-hf/ --quantization q4f16_1 \
|
|
--conv-template llama-2 -o dist/Llama-2-7b-chat-hf-q4f16_1-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/Llama-2-7b-chat-hf-q4f16_1-MLC/mlc-chat-config.json \
|
|
--device vulkan -o dist/libs/Llama-2-7b-chat-hf-q4f16_1-vulkan.dll
|
|
|
|
.. tab:: WebGPU
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/Llama-2-7b-chat-hf/ --quantization q4f16_1 \
|
|
--context-window-size 2048 --conv-template llama-2 -o dist/Llama-2-7b-chat-hf-q4f16_1-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/Llama-2-7b-chat-hf-q4f16_1-MLC/mlc-chat-config.json \
|
|
--device webgpu -o dist/libs/Llama-2-7b-chat-hf-q4f16_1-webgpu.wasm
|
|
|
|
.. note::
|
|
To compile for webgpu, you need to build from source when installing ``mlc_llm``. Besides, you also need to follow :ref:`install-web-build`.
|
|
Otherwise, it would run into error
|
|
|
|
.. code:: text
|
|
|
|
RuntimeError: Cannot find libraries: wasm_runtime.bc
|
|
|
|
.. tab:: iPhone/iPad
|
|
|
|
You need a Mac to compile models for it.
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/Llama-2-7b-chat-hf/ --quantization q4f16_1 \
|
|
--conv-template llama-2 --context-window-size 768 -o dist/Llama-2-7b-chat-hf-q4f16_1-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/Llama-2-7b-chat-hf-q4f16_1-MLC/mlc-chat-config.json \
|
|
--device iphone -o dist/libs/Llama-2-7b-chat-hf-q4f16_1-iphone.tar
|
|
|
|
.. tab:: Android
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/Llama-2-7b-chat-hf/ --quantization q4f16_1 \
|
|
--conv-template llama-2 --context-window-size 768 -o dist/Llama-2-7b-chat-hf-q4f16_1-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/Llama-2-7b-chat-hf-q4f16_1-MLC/mlc-chat-config.json \
|
|
--device android -o dist/libs/Llama-2-7b-chat-hf-q4f16_1-android.tar
|
|
|
|
.. tab:: Mistral-7B-Instruct-v0.2
|
|
|
|
Note that Mistral uses sliding window attention (SWA). Thus, instead of specifying
|
|
``context-window-size``, we specify ``sliding-window-size``.
|
|
|
|
First create directory ``dist/models`` and download the model to the directory.
|
|
For example, you can run the following code:
|
|
|
|
.. code:: shell
|
|
|
|
mkdir -p dist/models && cd dist/models
|
|
git lfs install
|
|
git clone https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2
|
|
cd ../..
|
|
|
|
Then convert the HF weights into MLC-compatible weights. Note that all platforms
|
|
can share the same compiled/quantized weights.
|
|
|
|
.. code:: shell
|
|
|
|
mlc_llm convert_weight ./dist/models/Mistral-7B-Instruct-v0.2/ --quantization q4f16_1 \
|
|
-o dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC
|
|
|
|
Afterwards, run the following command to generate mlc config and compile the model.
|
|
|
|
.. code:: shell
|
|
|
|
# Create output directory for the model library compiled
|
|
mkdir dist/libs
|
|
|
|
.. tabs::
|
|
|
|
.. tab:: Target: CUDA
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/Mistral-7B-Instruct-v0.2/ --quantization q4f16_1 \
|
|
--conv-template mistral_default -o dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/mlc-chat-config.json \
|
|
--device cuda -o dist/libs/Mistral-7B-Instruct-v0.2-q4f16_1-cuda.so
|
|
|
|
.. tab:: Metal
|
|
|
|
For M-chip Mac:
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/Mistral-7B-Instruct-v0.2/ --quantization q4f16_1 \
|
|
--conv-template mistral_default -o dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/mlc-chat-config.json \
|
|
--device metal -o dist/libs/Mistral-7B-Instruct-v0.2-q4f16_1-metal.so
|
|
|
|
|
|
For Intel Mac:
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/Mistral-7B-Instruct-v0.2/ --quantization q4f16_1 \
|
|
--conv-template mistral_default -o dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/mlc-chat-config.json \
|
|
--device metal -o dist/libs/Mistral-7B-Instruct-v0.2-q4f16_1-metal_x86_64.dylib
|
|
|
|
.. tab:: Vulkan
|
|
|
|
For Linux:
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/Mistral-7B-Instruct-v0.2/ --quantization q4f16_1 \
|
|
--conv-template mistral_default -o dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/mlc-chat-config.json \
|
|
--device vulkan -o dist/libs/Mistral-7B-Instruct-v0.2-q4f16_1-vulkan.so
|
|
|
|
For Windows:
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/Mistral-7B-Instruct-v0.2/ --quantization q4f16_1 \
|
|
--conv-template mistral_default -o dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/mlc-chat-config.json \
|
|
--device vulkan -o dist/libs/Mistral-7B-Instruct-v0.2-q4f16_1-vulkan.dll
|
|
|
|
.. tab:: WebGPU
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/Mistral-7B-Instruct-v0.2/ --quantization q4f16_1 \
|
|
--prefill-chunk-size 1024 --conv-template mistral_default \
|
|
-o dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/mlc-chat-config.json \
|
|
--device webgpu -o dist/libs/Mistral-7B-Instruct-v0.2-q4f16_1-webgpu.wasm
|
|
|
|
.. note::
|
|
To compile for webgpu, you need to build from source when installing ``mlc_llm``. Besides, you also need to follow :ref:`install-web-build`.
|
|
Otherwise, it would run into error
|
|
|
|
.. code:: text
|
|
|
|
RuntimeError: Cannot find libraries: wasm_runtime.bc
|
|
|
|
.. note::
|
|
For webgpu, when compiling larger models like ``Llama-2-7B``, you may want to add ``--prefill-chunk-size 1024`` or lower ``--context-window-size`` to decrease memory usage.
|
|
Otherwise, you may run into issues like:
|
|
|
|
.. code:: text
|
|
|
|
TypeError: Failed to execute 'createBuffer' on 'GPUDevice': Failed to read the 'size' property from
|
|
'GPUBufferDescriptor': Value is outside the 'unsigned long long' value range.
|
|
|
|
.. tab:: iPhone/iPad
|
|
|
|
You need a Mac to compile models for it.
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/Mistral-7B-Instruct-v0.2/ --quantization q4f16_1 \
|
|
--conv-template mistral_default --sliding-window-size 1024 --prefill-chunk-size 128 \
|
|
-o dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/mlc-chat-config.json \
|
|
--device iphone -o dist/libs/Mistral-7B-Instruct-v0.2-q4f16_1-iphone.tar
|
|
|
|
.. tab:: Android
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/Mistral-7B-Instruct-v0.2/ --quantization q4f16_1 \
|
|
--conv-template mistral_default --sliding-window-size 1024 --prefill-chunk-size 128 -o dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/mlc-chat-config.json \
|
|
--device android -o dist/libs/Mistral-7B-Instruct-v0.2-q4f16_1-android.tar
|
|
|
|
.. tab:: Other models
|
|
|
|
First create directory ``dist/models`` and download the model to the directory.
|
|
For example, you can run the following code:
|
|
|
|
.. code:: shell
|
|
|
|
mkdir -p dist/models && cd dist/models
|
|
git lfs install
|
|
git clone https://huggingface.co/DISTRIBUTOR/HF_MODEL
|
|
cd ../..
|
|
|
|
Then convert the HF weights into MLC-compatible weights. Note that all platforms
|
|
can share the same compiled/quantized weights.
|
|
|
|
.. code:: shell
|
|
|
|
mlc_llm convert_weight ./dist/models/HF_MODEL/ --quantization q4f16_1 -o dist/OUTPUT-MLC
|
|
|
|
Afterwards, run the following command to generate mlc config and compile the model.
|
|
|
|
.. code:: shell
|
|
|
|
# Create output directory for the model library compiled
|
|
mkdir dist/libs
|
|
|
|
.. tabs::
|
|
|
|
.. tab:: Target: CUDA
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/HF_MODEL/ --quantization q4f16_1 --conv-template CONV_TEMPLATE -o dist/OUTPUT-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/OUTPUT-MLC/mlc-chat-config.json --device cuda -o dist/libs/OUTPUT-cuda.so
|
|
|
|
.. tab:: Metal
|
|
|
|
For M-chip Mac:
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/HF_MODEL/ --quantization q4f16_1 --conv-template CONV_TEMPLATE -o dist/OUTPUT-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/OUTPUT-MLC/mlc-chat-config.json --device metal -o dist/libs/OUTPUT-metal.so
|
|
|
|
|
|
For Intel Mac:
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/HF_MODEL/ --quantization q4f16_1 --conv-template CONV_TEMPLATE -o dist/OUTPUT-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/OUTPUT-MLC/mlc-chat-config.json --device metal -o dist/libs/OUTPUT-metal_x86_64.dylib
|
|
|
|
.. tab:: Vulkan
|
|
|
|
For Linux:
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/HF_MODEL/ --quantization q4f16_1 --conv-template CONV_TEMPLATE -o dist/OUTPUT-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/OUTPUT-MLC/mlc-chat-config.json --device vulkan -o dist/libs/OUTPUT-vulkan.so
|
|
|
|
For Windows:
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/HF_MODEL/ --quantization q4f16_1 --conv-template CONV_TEMPLATE -o dist/OUTPUT-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/OUTPUT-MLC/mlc-chat-config.json --device vulkan -o dist/libs/OUTPUT-vulkan.dll
|
|
|
|
.. tab:: WebGPU
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/HF_MODEL/ --quantization q4f16_1 --conv-template CONV_TEMPLATE -o dist/OUTPUT-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/OUTPUT-MLC/mlc-chat-config.json --device webgpu -o dist/libs/OUTPUT-webgpu.wasm
|
|
|
|
.. note::
|
|
To compile for webgpu, you need to build from source when installing ``mlc_llm``. Besides, you also need to follow :ref:`install-web-build`.
|
|
Otherwise, it would run into error
|
|
|
|
.. code:: text
|
|
|
|
RuntimeError: Cannot find libraries: wasm_runtime.bc
|
|
|
|
.. note::
|
|
For webgpu, when compiling larger models like ``Llama-2-7B``, you may want to add ``--prefill-chunk-size 1024`` or lower ``--context-window-size`` to decrease memory usage.
|
|
Otherwise, you may run into issues like:
|
|
|
|
.. code:: text
|
|
|
|
TypeError: Failed to execute 'createBuffer' on 'GPUDevice': Failed to read the 'size' property from
|
|
'GPUBufferDescriptor': Value is outside the 'unsigned long long' value range.
|
|
|
|
.. tab:: iPhone/iPad
|
|
|
|
You need a Mac to compile models for it.
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/HF_MODEL/ --quantization q4f16_1 --conv-template CONV_TEMPLATE \
|
|
--context-window-size 768 -o dist/OUTPUT-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/OUTPUT-MLC/mlc-chat-config.json --device iphone -o dist/libs/OUTPUT-iphone.tar
|
|
|
|
.. tab:: Android
|
|
|
|
.. code:: shell
|
|
|
|
# 1. gen_config: generate mlc-chat-config.json and process tokenizers
|
|
mlc_llm gen_config ./dist/models/HF_MODEL/ --quantization q4f16_1 --conv-template CONV_TEMPLATE \
|
|
--context-window-size 768 -o dist/OUTPUT-MLC/
|
|
# 2. compile: compile model library with specification in mlc-chat-config.json
|
|
mlc_llm compile ./dist/OUTPUT-MLC/mlc-chat-config.json --device android -o dist/libs/OUTPUT-android.tar
|
|
|
|
For each model and each backend, the above only provides the most recommended build command (which is the most optimized).
|
|
You can also try with different argument values (e.g., different quantization modes, context window size, etc.),
|
|
whose build results affect runtime memory requirement, and it is possible that they may not run as
|
|
fast and robustly as the provided one when running the model.
|
|
|
|
.. note::
|
|
Uing 3-bit quantization usually can be overly aggressive and only works for limited settings.
|
|
If you encounter issues where the compiled model does not perform as expected,
|
|
consider utilizing a higher number of bits for quantization (e.g., 4-bit quantization).
|
|
|
|
If you are interested in distributing the model besides local execution, please checkout :ref:`distribute-compiled-models`.
|
|
|
|
|
|
.. _compile-command-specification:
|
|
|
|
Compile Command Specification
|
|
-----------------------------
|
|
|
|
As you have seen in the section above, the model compilation is split into three steps: convert weights, generate
|
|
``mlc-chat-config.json``, and compile the model. This section describes the list of options that can be used
|
|
during compilation.
|
|
|
|
1. Convert Weight
|
|
^^^^^^^^^^^^^^^^^
|
|
|
|
Weight conversion command follows the pattern below:
|
|
|
|
.. code:: text
|
|
|
|
mlc_llm convert_weight \
|
|
CONFIG \
|
|
--quantization QUANTIZATION_MODE \
|
|
[--model-type MODEL_TYPE] \
|
|
[--device DEVICE] \
|
|
[--source SOURCE] \
|
|
[--source-format SOURCE_FORMAT] \
|
|
--output OUTPUT
|
|
|
|
Note that ``CONFIG`` is a positional argument. Arguments wrapped with ``[ ]`` are optional.
|
|
|
|
--CONFIG It can be one of the following:
|
|
|
|
1. Path to a HuggingFace model directory that contains a ``config.json`` or
|
|
2. Path to ``config.json`` in HuggingFace format, or
|
|
3. The name of a pre-defined model architecture.
|
|
|
|
A ``config.json`` file in HuggingFace format defines the model architecture, including the vocabulary
|
|
size, the number of layers, the hidden size, number of attention heads, etc.
|
|
Example: https://huggingface.co/codellama/CodeLlama-7b-hf/blob/main/config.json.
|
|
|
|
A HuggingFace directory often contains a ``config.json`` which defines the model architecture,
|
|
the non-quantized model weights in PyTorch or SafeTensor format, tokenizer configurations,
|
|
as well as an optional ``generation_config.json`` provides additional default configuration for
|
|
text generation.
|
|
Example: https://huggingface.co/codellama/CodeLlama-7b-hf/tree/main.
|
|
|
|
For existing pre-defined model architecture, see ``MODEL_PRESETS``
|
|
`here <https://github.com/mlc-ai/mlc-llm/blob/main/python/mlc_llm/compiler/model/model.py>`_.
|
|
|
|
--quantization QUANTIZATION_MODE The quantization mode we use to compile.
|
|
|
|
See :ref:`quantization_mode` for more information.
|
|
Available options are: ``q0f16``, ``q0f32``, ``q3f16_1``, ``q4f16_1``, ``q4f32_1``, and
|
|
``q4f16_awq``.
|
|
|
|
We encourage you to use 4-bit quantization, as the text generated by 3-bit
|
|
quantized models may have bad quality depending on the model.
|
|
|
|
--model-type MODEL_TYPE Model architecture such as "llama". If not set, it is inferred from ``config.json``.
|
|
|
|
--device DEVICE The device used to do quantization such as "cuda" or "cuda:0". Will detect from
|
|
local available GPUs if not specified.
|
|
|
|
--source SOURCE The path to original model weight, infer from ``config`` if missing.
|
|
|
|
--source-format SOURCE_FORMAT The format of source model weight, infer from ``config`` if missing.
|
|
|
|
--output OUTPUT The output directory to save the quantized model weight.
|
|
Will create ``params_shard_*.bin`` and ```tensor-cache.json``` in this directory.
|
|
|
|
2. Generate MLC Chat Config
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
In order to compile a model, we first need to generate the ``mlc-chat-config.json``. This file contains specifications
|
|
like ``context-window-size`` and ``sliding-window-size``, among others that can alter the model compiled. We also process
|
|
tokenizers in this step.
|
|
|
|
Config generation command follows the pattern below:
|
|
|
|
.. code:: text
|
|
|
|
mlc_llm gen_config \
|
|
CONFIG \
|
|
--quantization QUANTIZATION_MODE \
|
|
[--model-type MODEL_TYPE] \
|
|
--conv-template CONV_TEMPLATE \
|
|
[--context-window-size CONTEXT_WINDOW_SIZE] \
|
|
[--sliding-window-size SLIDING_WINDOW_SIZE] \
|
|
[--prefill-chunk-size PREFILL_CHUNK_SIZE] \
|
|
[--tensor-parallel-shard TENSOR_PARALLEL_SHARDS] \
|
|
--output OUTPUT
|
|
|
|
Note that ``CONFIG`` is a positional argument. Arguments wrapped with ``[ ]`` are optional.
|
|
|
|
--CONFIG It can be one of the following:
|
|
|
|
1. Path to a HuggingFace model directory that contains a ``config.json`` or
|
|
2. Path to ``config.json`` in HuggingFace format, or
|
|
3. The name of a pre-defined model architecture.
|
|
|
|
A ``config.json`` file in HuggingFace format defines the model architecture, including the vocabulary
|
|
size, the number of layers, the hidden size, number of attention heads, etc.
|
|
Example: https://huggingface.co/codellama/CodeLlama-7b-hf/blob/main/config.json.
|
|
|
|
A HuggingFace directory often contains a ``config.json`` which defines the model architecture,
|
|
the non-quantized model weights in PyTorch or SafeTensor format, tokenizer configurations,
|
|
as well as an optional ``generation_config.json`` provides additional default configuration for
|
|
text generation.
|
|
Example: https://huggingface.co/codellama/CodeLlama-7b-hf/tree/main.
|
|
|
|
For existing pre-defined model architecture, see ``MODEL_PRESETS``
|
|
`here <https://github.com/mlc-ai/mlc-llm/blob/main/python/mlc_llm/compiler/model/model.py>`_.
|
|
|
|
--quantization QUANTIZATION_MODE The quantization mode we use to compile.
|
|
|
|
See :ref:`quantization_mode` for more information.
|
|
Available options are: ``q0f16``, ``q0f32``, ``q3f16_1``, ``q4f16_1``, ``q4f32_1``, and
|
|
``q4f16_awq``.
|
|
|
|
We encourage you to use 4-bit quantization, as the text generated by 3-bit
|
|
quantized models may have bad quality depending on the model.
|
|
|
|
--model-type MODEL_TYPE Model architecture such as "llama". If not set, it is inferred from ``config.json``.
|
|
|
|
--conv-template CONV_TEMPLATE Conversation template. It depends on how the model is tuned. Use "LM" for vanilla base model
|
|
For existing pre-defined templates, see ``CONV_TEMPLATES``
|
|
`here <https://github.com/mlc-ai/mlc-llm/blob/main/python/mlc_llm/model/model.py>`_.
|
|
|
|
--context-window-size CONTEXT_WINDOW_SIZE Option to provide the maximum sequence length supported by the model.
|
|
This is usually explicitly shown as context length or context window in the model card.
|
|
If this option is not set explicitly, by default,
|
|
it will be determined by ``context_window_size`` or ``max_position_embeddings`` in ``config.json``,
|
|
and the latter is usually inaccurate for some models.
|
|
|
|
--sliding-window-size SLIDING_WINDOW (Experimental) The sliding window size in sliding window attention (SWA).
|
|
This optional field overrides the ``sliding_window`` in ``config.json`` for
|
|
those models that use SWA. Currently only useful when compiling mistral-based models.
|
|
This flag subjects to future refactoring.
|
|
|
|
--prefill-chunk-size PREFILL_CHUNK_SIZE (Experimental) The chunk size during prefilling. By default,
|
|
the chunk size is the same as ``context_window_size`` or ``sliding_window_size``.
|
|
This flag subjects to future refactoring.
|
|
|
|
--tensor-parallel-shard TENSOR_PARALLEL_SHARDS Number of shards to split the model into in tensor parallelism multi-gpu inference.
|
|
|
|
--output OUTPUT The output directory for generated configurations, including `mlc-chat-config.json` and tokenizer configuration.
|
|
|
|
3. Compile Model Library
|
|
^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
After generating ``mlc-chat-config.json``, we can compile the model into a model library (files ending in ``.so``, ``.tar``, etc. that contains
|
|
the inference logic of a model).
|
|
|
|
Model compilation command follows the pattern below:
|
|
|
|
.. code:: text
|
|
|
|
mlc_llm compile \
|
|
MODEL \
|
|
[--quantization QUANTIZATION_MODE] \
|
|
[--model-type MODEL_TYPE] \
|
|
[--device DEVICE] \
|
|
[--host HOST] \
|
|
[--opt OPT] \
|
|
[--system-lib-prefix SYSTEM_LIB_PREFIX] \
|
|
--output OUTPUT \
|
|
[--overrides OVERRIDES]
|
|
|
|
Note that ``MODEL`` is a positional argument. Arguments wrapped with ``[ ]`` are optional.
|
|
|
|
--MODEL A path to ``mlc-chat-config.json``, or an MLC model directory that contains ``mlc-chat-config.json``.
|
|
|
|
--quantization QUANTIZATION_MODE The quantization mode we use to compile. If unprovided, will infer from ``MODEL``.
|
|
|
|
See :ref:`quantization_mode` for more information.
|
|
Available options are: ``q0f16``, ``q0f32``, ``q3f16_1``, ``q4f16_1``, ``q4f32_1``, and
|
|
``q4f16_awq``.
|
|
|
|
We encourage you to use 4-bit quantization, as the text generated by 3-bit
|
|
quantized models may have bad quality depending on the model.
|
|
|
|
--model-type MODEL_TYPE Model architecture such as "llama". If not set, it is inferred from ``mlc-chat-config.json``.
|
|
|
|
--device DEVICE The GPU device to compile the model to. If not set, it is inferred from GPUs available locally.
|
|
|
|
--host HOST The host LLVM triple to compile the model to. If not set, it is inferred from the local CPU and OS.
|
|
Examples of the LLVM triple:
|
|
|
|
1) iPhones: arm64-apple-ios;
|
|
2) ARM64 Android phones: aarch64-linux-android;
|
|
3) WebAssembly: wasm32-unknown-unknown-wasm;
|
|
4) Windows: x86_64-pc-windows-msvc;
|
|
5) ARM macOS: arm64-apple-darwin.
|
|
|
|
--opt OPT Optimization flags. MLC LLM maintains a predefined set of optimization flags,
|
|
denoted as ``O0``, ``O1``, ``O2``, ``O3``, where ``O0`` means no optimization, ``O2``
|
|
means majority of them, and ``O3`` represents extreme optimization that could
|
|
potentially break the system.
|
|
|
|
Meanwhile, optimization flags could be explicitly specified via details knobs, e.g.
|
|
``--opt="cutlass_attn=1;cutlass_norm=0;cublas_gemm=0;cudagraph=0"``.
|
|
|
|
--system-lib-prefix SYSTEM_LIB_PREFIX Adding a prefix to all symbols exported. Similar to ``objcopy --prefix-symbols``.
|
|
This is useful when compiling multiple models into a single library to avoid symbol
|
|
conflicts. Different from objcopy, this takes no effect for shared library.
|
|
|
|
|
|
--output OUTPUT The path to the output file. The suffix determines if the output file is a shared library or
|
|
objects. Available suffixes:
|
|
|
|
1) Linux: .so (shared), .tar (objects);
|
|
2) macOS: .dylib (shared), .tar (objects);
|
|
3) Windows: .dll (shared), .tar (objects);
|
|
4) Android, iOS: .tar (objects);
|
|
5) Web: .wasm (web assembly).
|
|
|
|
--overrides OVERRIDES Model configuration override. Configurations to override ``mlc-chat-config.json``. Supports
|
|
``context_window_size``, ``prefill_chunk_size``, ``sliding_window``, ``max_batch_size`` and
|
|
``tensor_parallel_shards``. Meanwhile, model config could be explicitly specified via details
|
|
knobs, e.g. ``--overrides "context_window_size=1024;prefill_chunk_size=128"``.
|