wehub-resource-sync
104 lines
4.1 KiB
Plaintext
104 lines
4.1 KiB
Plaintext
---
|
|
id: openrouter
|
|
title: OpenRouter
|
|
sidebar_label: OpenRouter
|
|
---
|
|
|
|
`deepeval`'s integration with OpenRouter allows you to use the OpenRouter gateway, connecting any [OpenRouter supported model](https://openrouter.ai/models) to power all of `deepeval`'s metrics.
|
|
|
|
### Command Line
|
|
|
|
To configure your OpenRouter model through the CLI, run the following command:
|
|
|
|
```bash
|
|
deepeval set-openrouter \
|
|
--model "openai/gpt-4.1" \ # Ex: openai/gpt-4.1
|
|
--base-url "https://openrouter.ai/api/v1" \
|
|
--temperature=0 \
|
|
--prompt-api-key
|
|
```
|
|
|
|
:::info
|
|
The CLI command above sets OpenRouter as the default provider for all metrics, unless overridden in Python code. To use a different default model provider, you must first unset OpenRouter:
|
|
|
|
```bash
|
|
deepeval unset-openrouter
|
|
```
|
|
:::
|
|
|
|
:::tip[Persisting settings]
|
|
You can persist CLI settings with the optional `--save` flag.
|
|
See [Flags and Configs -> Persisting CLI settings](/docs/evaluation-flags-and-configs#persisting-cli-settings-with---save).
|
|
:::
|
|
|
|
### Python
|
|
|
|
Alternatively, you can define `OpenRouterModel` directly in Python code:
|
|
|
|
<Tabs items={["Python", "ENV"]}>
|
|
<Tab value="Python">
|
|
|
|
```python
|
|
from deepeval.models import OpenRouterModel
|
|
from deepeval.metrics import AnswerRelevancyMetric
|
|
|
|
model = OpenRouterModel(
|
|
model="openai/gpt-4.1",
|
|
api_key="your-openrouter-api-key",
|
|
# Optional: override the default OpenRouter endpoint
|
|
base_url="https://openrouter.ai/api/v1",
|
|
# Optional: pass OpenRouter headers via **kwargs
|
|
default_headers={
|
|
"HTTP-Referer": "https://your-site.com",
|
|
"X-Title": "My eval pipeline",
|
|
},
|
|
)
|
|
|
|
answer_relevancy = AnswerRelevancyMetric(model=model)
|
|
```
|
|
|
|
</Tab>
|
|
<Tab value="ENV">
|
|
|
|
To use any OpenRouter model directly in `deepeval`, set the `USE_OPENROUTER_MODEL=1` in your `env` and simply pass the name of your desired model in your metric initialization:
|
|
|
|
```python
|
|
from deepeval.metrics import AnswerRelevancyMetric
|
|
|
|
answer_relevancy = AnswerRelevancyMetric(
|
|
model="openai/gpt-4.1",
|
|
)
|
|
```
|
|
|
|
You should also set the other necessary vars like `OPENROUTER_API_KEY` to be able to use the OpenRouter models as shown above.
|
|
|
|
</Tab>
|
|
</Tabs>
|
|
|
|
There are **ZERO** mandatory and **SEVEN** optional parameters when creating an `OpenRouterModel`:
|
|
|
|
- [Optional] `model`: A string specifying the OpenRouter model to use. Defaults to `OPENROUTER_MODEL_NAME` if set; otherwise falls back to "openai/gpt-4.1".
|
|
- [Optional] `api_key`: A string specifying your OpenRouter API key for authentication. Defaults to `OPENROUTER_API_KEY` if not passed; raises an error at runtime if unset.
|
|
- [Optional] `base_url`: A string specifying the base URL for the OpenRouter API endpoint. Defaults to `OPENROUTER_BASE_URL` if set; otherwise falls back to "https://openrouter.ai/api/v1".
|
|
- [Optional] `temperature`: A float specifying the model temperature. Defaults to `TEMPERATURE` if not passed; falls back to `0.0` if unset.
|
|
- [Optional] `cost_per_input_token`: A float specifying the cost for each input token for the provided model. Defaults to `OPENROUTER_COST_PER_INPUT_TOKEN` if not passed; falls back to `None` if unset (cost is then reported as unknown).
|
|
- [Optional] `cost_per_output_token`: A float specifying the cost for each output token for the provided model. Defaults to `OPENROUTER_COST_PER_OUTPUT_TOKEN` if not passed; falls back to `None` if unset (cost is then reported as unknown).
|
|
- [Optional] `generation_kwargs`: A dictionary of additional generation parameters forwarded to OpenRouter's `chat.completions.create(...)` call
|
|
|
|
Any additional `**kwargs` you would like to use for your `OpenRouter` client can be passed directly to `OpenRouterModel(...)`. These are forwarded to the underlying OpenAI client constructor. We recommend double-checking the parameters and headers supported by your chosen model in the [official OpenRouter docs](https://openrouter.ai/docs).
|
|
|
|
:::tip
|
|
Pass headers specific to OpenRouter via kwargs:
|
|
|
|
```python
|
|
model = OpenRouterModel(
|
|
model="openai/gpt-4.1",
|
|
api_key="your-openrouter-api-key",
|
|
default_headers={
|
|
"HTTP-Referer": "https://your-site.com",
|
|
"X-Title": "My eval pipeline",
|
|
},
|
|
)
|
|
```
|
|
:::
|