wehub-resource-sync
24 lines
828 B
Markdown
24 lines
828 B
Markdown
# Apple MLX Integration
|
|
|
|
You can use [Apple MLX](https://github.com/ml-explore/mlx) as an optimized worker implementation in FastChat.
|
|
|
|
It runs models efficiently on Apple Silicon
|
|
|
|
See the supported models [here](https://github.com/ml-explore/mlx-examples/tree/main/llms#supported-models).
|
|
|
|
Note that for Apple Silicon Macs with less memory, smaller models (or quantized models) are recommended.
|
|
|
|
## Instructions
|
|
|
|
1. Install MLX.
|
|
|
|
```
|
|
pip install "mlx-lm>=0.0.6"
|
|
```
|
|
|
|
2. When you launch a model worker, replace the normal worker (`fastchat.serve.model_worker`) with the MLX worker (`fastchat.serve.mlx_worker`). Remember to launch a model worker after you have launched the controller ([instructions](../README.md))
|
|
|
|
```
|
|
python3 -m fastchat.serve.mlx_worker --model-path TinyLlama/TinyLlama-1.1B-Chat-v1.0
|
|
```
|