chore: import upstream snapshot with attribution
This commit is contained in:
@@ -0,0 +1,125 @@
|
||||
<!-- markdownlint-disable MD041 -->
|
||||
--8<-- [start:installation]
|
||||
|
||||
For GPU-accelerated inference on Apple Silicon, use [vLLM-Metal](https://github.com/vllm-project/vllm-metal), a community-maintained hardware plugin that uses MLX as the compute backend and provides native GPU acceleration via Apple's Metal framework.
|
||||
|
||||
vLLM-Metal works with MLX-optimized models from the [mlx-community](https://huggingface.co/mlx-community) organization on Hugging Face, which provides quantized versions of popular models optimized for Apple Silicon.
|
||||
|
||||
!!! tip
|
||||
For installation and usage instructions, see the [Set up using vLLM-Metal](#set-up-using-vllm-metal) section below.
|
||||
|
||||
--8<-- [end:installation]
|
||||
--8<-- [start:requirements]
|
||||
|
||||
- OS: macOS Sonoma or later
|
||||
- Hardware: Apple Silicon
|
||||
- Metal support enabled
|
||||
|
||||
!!! note
|
||||
See the [Set up using vLLM-Metal](#set-up-using-vllm-metal) section below for installation instructions.
|
||||
|
||||
--8<-- [end:requirements]
|
||||
--8<-- [start:set-up-using-python]
|
||||
|
||||
## Set up using vLLM-Metal
|
||||
|
||||
vLLM-Metal is distributed as a separate package that provides native GPU acceleration on Apple Silicon.
|
||||
|
||||
To install vLLM-Metal, follow the installation instructions in the [vLLM-Metal documentation](https://github.com/vllm-project/vllm-metal#installation).
|
||||
|
||||
The installation will:
|
||||
|
||||
1. Set up the appropriate Python environment
|
||||
2. Install MLX and required dependencies
|
||||
3. Install the vLLM-Metal package
|
||||
|
||||
After installation, you can start using vLLM with Metal GPU acceleration.
|
||||
|
||||
!!! tip
|
||||
When using vLLM-Metal, use models from the [mlx-community](https://huggingface.co/mlx-community) on Hugging Face for best performance. These models are optimized for MLX and often include quantized versions (4-bit, 8-bit) that run efficiently on Apple Silicon.
|
||||
|
||||
Example model: `mlx-community/Qwen2.5-0.5B-Instruct-4bit`
|
||||
|
||||
### Using vLLM-Metal
|
||||
|
||||
After installation, vLLM-Metal provides an easy-to-use CLI for running an OpenAI-compatible API server:
|
||||
|
||||
```bash
|
||||
# Activate the vLLM-Metal environment
|
||||
source ~/.venv-vllm-metal/bin/activate
|
||||
|
||||
# Start the API server (specify your mlx-community model or it will use default)
|
||||
vllm serve
|
||||
```
|
||||
|
||||
Once the server is running, you have multiple options to interact with it:
|
||||
|
||||
#### Option 1: Interactive chat
|
||||
|
||||
Open a new terminal and start an interactive chat session:
|
||||
|
||||
```bash
|
||||
source ~/.venv-vllm-metal/bin/activate
|
||||
vllm chat
|
||||
```
|
||||
|
||||
#### Option 2: API requests with curl
|
||||
|
||||
```bash
|
||||
curl http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"messages": [{"role": "user", "content": "Hello!"}],
|
||||
"max_tokens": 50
|
||||
}'
|
||||
```
|
||||
|
||||
#### Option 3: Python with OpenAI SDK
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="dummy" # No auth required for local server
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="mlx-community/Qwen2.5-0.5B-Instruct-4bit",
|
||||
messages=[{"role": "user", "content": "Hello!"}]
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
For more details on the `vllm` CLI commands, see the [OpenAI-compatible server documentation](../../serving/online_serving/openai_compatible_server.md).
|
||||
|
||||
--8<-- [end:set-up-using-python]
|
||||
--8<-- [start:pre-built-wheels]
|
||||
|
||||
vLLM-Metal is installed via the vLLM-Metal package. See the [Set up using vLLM-Metal](#set-up-using-vllm-metal) section above.
|
||||
|
||||
--8<-- [end:pre-built-wheels]
|
||||
--8<-- [start:build-wheel-from-source]
|
||||
|
||||
For build instructions from source, refer to the [vLLM-Metal documentation](https://github.com/vllm-project/vllm-metal#installation).
|
||||
|
||||
--8<-- [end:build-wheel-from-source]
|
||||
--8<-- [start:pre-built-images]
|
||||
|
||||
--8<-- [end:pre-built-images]
|
||||
--8<-- [start:build-image-from-source]
|
||||
|
||||
--8<-- [end:build-image-from-source]
|
||||
--8<-- [start:supported-features]
|
||||
|
||||
vLLM-Metal provides:
|
||||
|
||||
- Native GPU acceleration using Metal
|
||||
- MLX-based compute backend optimized for Apple Silicon
|
||||
- OpenAI-compatible API server
|
||||
- Support for popular model architectures
|
||||
|
||||
For specific feature support and limitations, refer to the [vLLM-Metal documentation](https://github.com/vllm-project/vllm-metal).
|
||||
|
||||
--8<-- [end:supported-features]
|
||||
Reference in New Issue
Block a user