wehub-resource-sync
126 lines
4.0 KiB
Markdown
126 lines
4.0 KiB
Markdown
<!-- markdownlint-disable MD041 -->
|
|
--8<-- [start:installation]
|
|
|
|
For GPU-accelerated inference on Apple Silicon, use [vLLM-Metal](https://github.com/vllm-project/vllm-metal), a community-maintained hardware plugin that uses MLX as the compute backend and provides native GPU acceleration via Apple's Metal framework.
|
|
|
|
vLLM-Metal works with MLX-optimized models from the [mlx-community](https://huggingface.co/mlx-community) organization on Hugging Face, which provides quantized versions of popular models optimized for Apple Silicon.
|
|
|
|
!!! tip
|
|
For installation and usage instructions, see the [Set up using vLLM-Metal](#set-up-using-vllm-metal) section below.
|
|
|
|
--8<-- [end:installation]
|
|
--8<-- [start:requirements]
|
|
|
|
- OS: macOS Sonoma or later
|
|
- Hardware: Apple Silicon
|
|
- Metal support enabled
|
|
|
|
!!! note
|
|
See the [Set up using vLLM-Metal](#set-up-using-vllm-metal) section below for installation instructions.
|
|
|
|
--8<-- [end:requirements]
|
|
--8<-- [start:set-up-using-python]
|
|
|
|
## Set up using vLLM-Metal
|
|
|
|
vLLM-Metal is distributed as a separate package that provides native GPU acceleration on Apple Silicon.
|
|
|
|
To install vLLM-Metal, follow the installation instructions in the [vLLM-Metal documentation](https://github.com/vllm-project/vllm-metal#installation).
|
|
|
|
The installation will:
|
|
|
|
1. Set up the appropriate Python environment
|
|
2. Install MLX and required dependencies
|
|
3. Install the vLLM-Metal package
|
|
|
|
After installation, you can start using vLLM with Metal GPU acceleration.
|
|
|
|
!!! tip
|
|
When using vLLM-Metal, use models from the [mlx-community](https://huggingface.co/mlx-community) on Hugging Face for best performance. These models are optimized for MLX and often include quantized versions (4-bit, 8-bit) that run efficiently on Apple Silicon.
|
|
|
|
Example model: `mlx-community/Qwen2.5-0.5B-Instruct-4bit`
|
|
|
|
### Using vLLM-Metal
|
|
|
|
After installation, vLLM-Metal provides an easy-to-use CLI for running an OpenAI-compatible API server:
|
|
|
|
```bash
|
|
# Activate the vLLM-Metal environment
|
|
source ~/.venv-vllm-metal/bin/activate
|
|
|
|
# Start the API server (specify your mlx-community model or it will use default)
|
|
vllm serve
|
|
```
|
|
|
|
Once the server is running, you have multiple options to interact with it:
|
|
|
|
#### Option 1: Interactive chat
|
|
|
|
Open a new terminal and start an interactive chat session:
|
|
|
|
```bash
|
|
source ~/.venv-vllm-metal/bin/activate
|
|
vllm chat
|
|
```
|
|
|
|
#### Option 2: API requests with curl
|
|
|
|
```bash
|
|
curl http://localhost:8000/v1/chat/completions \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"messages": [{"role": "user", "content": "Hello!"}],
|
|
"max_tokens": 50
|
|
}'
|
|
```
|
|
|
|
#### Option 3: Python with OpenAI SDK
|
|
|
|
```python
|
|
from openai import OpenAI
|
|
|
|
client = OpenAI(
|
|
base_url="http://localhost:8000/v1",
|
|
api_key="dummy" # No auth required for local server
|
|
)
|
|
|
|
response = client.chat.completions.create(
|
|
model="mlx-community/Qwen2.5-0.5B-Instruct-4bit",
|
|
messages=[{"role": "user", "content": "Hello!"}]
|
|
)
|
|
|
|
print(response.choices[0].message.content)
|
|
```
|
|
|
|
For more details on the `vllm` CLI commands, see the [OpenAI-compatible server documentation](../../serving/online_serving/openai_compatible_server.md).
|
|
|
|
--8<-- [end:set-up-using-python]
|
|
--8<-- [start:pre-built-wheels]
|
|
|
|
vLLM-Metal is installed via the vLLM-Metal package. See the [Set up using vLLM-Metal](#set-up-using-vllm-metal) section above.
|
|
|
|
--8<-- [end:pre-built-wheels]
|
|
--8<-- [start:build-wheel-from-source]
|
|
|
|
For build instructions from source, refer to the [vLLM-Metal documentation](https://github.com/vllm-project/vllm-metal#installation).
|
|
|
|
--8<-- [end:build-wheel-from-source]
|
|
--8<-- [start:pre-built-images]
|
|
|
|
--8<-- [end:pre-built-images]
|
|
--8<-- [start:build-image-from-source]
|
|
|
|
--8<-- [end:build-image-from-source]
|
|
--8<-- [start:supported-features]
|
|
|
|
vLLM-Metal provides:
|
|
|
|
- Native GPU acceleration using Metal
|
|
- MLX-based compute backend optimized for Apple Silicon
|
|
- OpenAI-compatible API server
|
|
- Support for popular model architectures
|
|
|
|
For specific feature support and limitations, refer to the [vLLM-Metal documentation](https://github.com/vllm-project/vllm-metal).
|
|
|
|
--8<-- [end:supported-features]
|