wehub-resource-sync
92 lines
3.3 KiB
Plaintext
92 lines
3.3 KiB
Plaintext
---
|
||
id: vllm
|
||
title: vLLM
|
||
sidebar_label: vLLM
|
||
---
|
||
|
||
`vLLM` is a high-performance inference engine for LLMs that supports OpenAI-compatible APIs. `deepeval` can connect to a running `vLLM` server for running local evaluations.
|
||
|
||
### Command Line
|
||
|
||
1. Launch your `vLLM` server and ensure it’s exposing the OpenAI-compatible API. The typical base URL for a local vLLM server is: `http://localhost:8000/v1/`.
|
||
2. Then run the following command to configure `deepeval`:
|
||
|
||
```bash
|
||
deepeval set-local-model \
|
||
--model=<model_name> \
|
||
--base-url="http://localhost:8000/v1/"
|
||
```
|
||
|
||
:::tip
|
||
You can enter any value when prompted for an api key if authentication is not enforced.
|
||
:::
|
||
|
||
:::tip[Persisting settings]
|
||
You can persist CLI settings with the optional `--save` flag.
|
||
See [Flags and Configs -> Persisting CLI settings](/docs/evaluation-flags-and-configs#persisting-cli-settings-with---save).
|
||
:::
|
||
|
||
### Python
|
||
|
||
Alternatively, you can define `LocalModel` directly in Python code:
|
||
|
||
<Tabs items={["Python", "ENV"]}>
|
||
<Tab value="Python">
|
||
|
||
```python
|
||
from deepeval.models import LocalModel
|
||
from deepeval.metrics import AnswerRelevancyMetric
|
||
|
||
model = LocalModel(
|
||
model="<model_name>",
|
||
base_url="http://localhost:8000/v1/",
|
||
api_key="vllm", # any placeholder works if your server has no auth
|
||
temperature=0
|
||
)
|
||
|
||
answer_relevancy = AnswerRelevancyMetric(model=model)
|
||
```
|
||
|
||
</Tab>
|
||
<Tab value="ENV">
|
||
|
||
To use a local model directly in `deepeval`, set `USE_LOCAL_MODEL=1` in your `env` and simply pass the name of your desired model in your metric initialization:
|
||
|
||
```python
|
||
from deepeval.metrics import AnswerRelevancyMetric
|
||
|
||
answer_relevancy = AnswerRelevancyMetric(
|
||
model="<model_name>",
|
||
)
|
||
```
|
||
|
||
You should also set the other necessary vars like `LOCAL_MODEL_BASE_URL` and `LOCAL_MODEL_API_KEY` to be able to use your local model as shown above.
|
||
|
||
</Tab>
|
||
</Tabs>
|
||
|
||
There are **ZERO** mandatory and **SIX** optional parameters when creating a `LocalModel`:
|
||
|
||
- [Optional] `model`: A string specifying the local model to use. Defaults to `LOCAL_MODEL_NAME` if not passed; raises an error at runtime if unset.
|
||
- [Optional] `api_key`: A string specifying the API key for your local server. Defaults to `LOCAL_MODEL_API_KEY` if not passed; raises an error at runtime if unset. Local servers without authentication accept any placeholder string.
|
||
- [Optional] `base_url`: A string specifying the base URL of your local server. Defaults to `LOCAL_MODEL_BASE_URL` if not passed.
|
||
- [Optional] `temperature`: A float specifying the model temperature. Defaults to `TEMPERATURE` if not passed; falls back to `0.0` if unset.
|
||
- [Optional] `format`: A string specifying the structured-output response format. Defaults to `LOCAL_MODEL_FORMAT` if not passed; falls back to `"json"` if unset.
|
||
- [Optional] `generation_kwargs`: A dictionary of additional generation parameters forwarded to the local server's `chat.completions.create(...)` call.
|
||
|
||
:::tip
|
||
Any `**kwargs` you would like to use for your model can be passed directly to `LocalModel(...)`; these are forwarded to the underlying OpenAI client constructor.
|
||
:::
|
||
|
||
### Reverting to OpenAI
|
||
|
||
To disable the local model and return to OpenAI:
|
||
|
||
```bash
|
||
deepeval unset-local-model
|
||
```
|
||
|
||
:::info
|
||
For advanced setup or deployment options (e.g. multi-GPU, HuggingFace models), see the [vLLM documentation](https://vllm.ai/).
|
||
:::
|