wehub-resource-sync
94 lines
3.4 KiB
Plaintext
94 lines
3.4 KiB
Plaintext
---
|
||
id: lmstudio
|
||
title: LM Studio
|
||
sidebar_label: LM Studio
|
||
---
|
||
|
||
`deepeval` supports running evaluations using local LLMs that expose OpenAI-compatible APIs. One such provider is **LM Studio**, a user-friendly desktop app for running models locally.
|
||
|
||
### Command Line
|
||
|
||
To start using LM Studio with `deepeval`, follow these steps:
|
||
|
||
1. Make sure LM Studio is running. The typical base URL for LM Studio is: `http://localhost:1234/v1/`.
|
||
2. Run the following command in your terminal to connect `deepeval` to LM Studio:
|
||
|
||
```bash
|
||
deepeval set-local-model \
|
||
--model=<model_name> \
|
||
--base-url="http://localhost:1234/v1/"
|
||
```
|
||
|
||
:::tip
|
||
If your local endpoint doesn't require authentication enter any placeholder string when prompted to enter an api key.
|
||
:::
|
||
|
||
:::tip[Persisting settings]
|
||
You can persist CLI settings with the optional `--save` flag.
|
||
See [Flags and Configs -> Persisting CLI settings](/docs/evaluation-flags-and-configs#persisting-cli-settings-with---save).
|
||
:::
|
||
|
||
### Python
|
||
|
||
Alternatively, you can define `LocalModel` directly in Python code:
|
||
|
||
<Tabs items={["Python", "ENV"]}>
|
||
<Tab value="Python">
|
||
|
||
```python
|
||
from deepeval.models import LocalModel
|
||
from deepeval.metrics import AnswerRelevancyMetric
|
||
|
||
model = LocalModel(
|
||
model="<model_name>",
|
||
base_url="http://localhost:1234/v1/",
|
||
api_key="lm-studio", # any placeholder works if your server has no auth
|
||
temperature=0
|
||
)
|
||
|
||
answer_relevancy = AnswerRelevancyMetric(model=model)
|
||
```
|
||
|
||
</Tab>
|
||
<Tab value="ENV">
|
||
|
||
To use a local model directly in `deepeval`, set `USE_LOCAL_MODEL=1` in your `env` and simply pass the name of your desired model in your metric initialization:
|
||
|
||
```python
|
||
from deepeval.metrics import AnswerRelevancyMetric
|
||
|
||
answer_relevancy = AnswerRelevancyMetric(
|
||
model="<model_name>",
|
||
)
|
||
```
|
||
|
||
You should also set the other necessary vars like `LOCAL_MODEL_BASE_URL` and `LOCAL_MODEL_API_KEY` to be able to use your local model as shown above.
|
||
|
||
</Tab>
|
||
</Tabs>
|
||
|
||
There are **ZERO** mandatory and **SIX** optional parameters when creating a `LocalModel`:
|
||
|
||
- [Optional] `model`: A string specifying the local model to use. Defaults to `LOCAL_MODEL_NAME` if not passed; raises an error at runtime if unset.
|
||
- [Optional] `api_key`: A string specifying the API key for your local server. Defaults to `LOCAL_MODEL_API_KEY` if not passed; raises an error at runtime if unset. Local servers without authentication accept any placeholder string.
|
||
- [Optional] `base_url`: A string specifying the base URL of your local server. Defaults to `LOCAL_MODEL_BASE_URL` if not passed.
|
||
- [Optional] `temperature`: A float specifying the model temperature. Defaults to `TEMPERATURE` if not passed; falls back to `0.0` if unset.
|
||
- [Optional] `format`: A string specifying the structured-output response format. Defaults to `LOCAL_MODEL_FORMAT` if not passed; falls back to `"json"` if unset.
|
||
- [Optional] `generation_kwargs`: A dictionary of additional generation parameters forwarded to the local server's `chat.completions.create(...)` call.
|
||
|
||
:::tip
|
||
Any `**kwargs` you would like to use for your model can be passed directly to `LocalModel(...)`; these are forwarded to the underlying OpenAI client constructor.
|
||
:::
|
||
|
||
### Reverting to OpenAI
|
||
|
||
To switch back to using OpenAI’s hosted models, run:
|
||
|
||
```bash
|
||
deepeval unset-local-model
|
||
```
|
||
|
||
:::info
|
||
For more help on enabling LM Studio’s server or configuring models, check out the [LM Studio docs](https://lmstudio.ai/).
|
||
:::
|