vllm-project--vllm-omni/docs/user_guide/examples/online_serving/mimo_audio.md

Online serving Example of vLLM-Omni for MiMo-Audio

Source https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/mimo_audio.

🛠️ Installation

Please refer to README.md

Run examples (MiMo-Audio)

Launch the Server

export MIMO_AUDIO_TOKENIZER_PATH="XiaomiMiMo/MiMo-Audio-Tokenizer"

vllm serve XiaomiMiMo/MiMo-Audio-7B-Instruct --omni \
    --served-model-name "MiMo-Audio-7B-Instruct" \
    --port 18091 \
    --chat-template ./examples/online_serving/mimo_audio/chat_template.jinja

⚠️ Important
MiMo-Audio is not compatible with the default chat template.
The provided chat_template.jinja implements MiMo-specific role, audio token, and instruction formatting and must be used for all inference.

Send Multi-modal Request

Get into the example folder

cd examples/online_serving/mimo_audio

Send request via python

# Audio dialogue task
python openai_chat_completion_client_for_multimodal_generation.py \
--query-type multi_audios \
--message-json ../../offline_inference/mimo_audio/message_base64_wav.json

The Python client supports the following command-line arguments:

• --query-type (or -q): Query type (default: multi_audios)
  • Options: multi_audios, text
• --message-json (or -m): Path to base64 multi rounds audio messages json file
  • Do not pass any value for "text" query type
  • Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs, only for "Are these two audio clips the same?" task
  • Example: ---message-json ./examples/offline_inference/mimo_audio/message_base64_wav.json
• --prompt (or -p): Custom text prompt/question, only for query type is "text"(TTS task)
  • Attention! Do not pass any value for "multi_audios" query type
  • Example: --prompt "What are the main activities shown in this video?"

For example, to use multi rounds audios with local files:

python openai_chat_completion_client_for_multimodal_generation.py \
--query-type multi_audios \
--message-json ../../offline_inference/mimo_audio/message_base64_wav.json

Example materials

??? abstract "chat_template.jinja" jinja --8<-- "examples/online_serving/mimo_audio/chat_template.jinja" ??? abstract "openai_chat_completion_client_for_multimodal_generation.py" py --8<-- "examples/online_serving/mimo_audio/openai_chat_completion_client_for_multimodal_generation.py"