wehub-resource-sync
148 lines
5.1 KiB
Plaintext
148 lines
5.1 KiB
Plaintext
---
|
||
# id: openai
|
||
title: OpenAI
|
||
sidebar_label: OpenAI
|
||
---
|
||
|
||
By default, DeepEval uses `gpt-4.1` to power all of its evaluation metrics. To enable this, you’ll need to set up your OpenAI API key. DeepEval also supports all other OpenAI models, which can be configured directly in Python.
|
||
|
||
### Setting Up Your API Key
|
||
|
||
DeepEval autoloads `.env.local` then `.env` at import time (process env -> `.env.local` -> `.env`).
|
||
|
||
**Recommended (local dev):**
|
||
|
||
```bash
|
||
# .env.local
|
||
OPENAI_API_KEY=<your-openai-api-key>
|
||
```
|
||
|
||
Alternative (Shell/CI)
|
||
|
||
```bash
|
||
export OPENAI_API_KEY=<your-openai-api-key>
|
||
```
|
||
|
||
Alternative (notebook)
|
||
|
||
If you're working in a notebook environment (Jupyter or Colab), set your `OPENAI_API_KEY` in a cell:
|
||
|
||
```bash
|
||
%env OPENAI_API_KEY=<your-openai-api-key>
|
||
```
|
||
|
||
### Command Line
|
||
|
||
Run the following command in your CLI to specify an OpenAI model to power all metrics.
|
||
|
||
```bash
|
||
deepeval set-openai \
|
||
--model=gpt-4.1 \
|
||
--cost-per-input-token=0.000002 \
|
||
--cost-per-output-token=0.000008
|
||
```
|
||
|
||
:::info
|
||
The CLI command above sets `gpt-4.1` as the default model for all metrics, unless overridden in Python code. To use a different default model provider, you must first unset the current settings:
|
||
|
||
```bash
|
||
deepeval unset-openai
|
||
```
|
||
|
||
:::
|
||
|
||
:::tip[Persisting settings]
|
||
You can persist CLI settings with the optional `--save` flag.
|
||
See [Flags and Configs -> Persisting CLI settings](/docs/evaluation-flags-and-configs#persisting-cli-settings-with---save).
|
||
:::
|
||
|
||
### Python
|
||
|
||
You may use OpenAI models other than `gpt-4.1`, which can be configured directly in python code through DeepEval's `GPTModel`.
|
||
|
||
:::info
|
||
You may want to use stronger reasoning models like `gpt-4.1` for metrics that require a high level of reasoning — for example, a custom GEval for mathematical correctness.
|
||
:::
|
||
|
||
<Tabs items={["Python", "ENV"]}>
|
||
<Tab value="Python">
|
||
|
||
```python
|
||
from deepeval.models import GPTModel
|
||
from deepeval.metrics import AnswerRelevancyMetric
|
||
|
||
model = GPTModel(
|
||
model="gpt-4.1",
|
||
temperature=0,
|
||
cost_per_input_token=0.000002,
|
||
cost_per_output_token=0.000008
|
||
)
|
||
answer_relevancy = AnswerRelevancyMetric(model=model)
|
||
```
|
||
|
||
</Tab>
|
||
<Tab value="ENV">
|
||
|
||
`deepeval` by default uses OpenAI models for evaluations, you can simply pass the name of your desired model in metric initialization and set the `OPENAI_API_KEY` to use OpenAI models:
|
||
|
||
```python
|
||
from deepeval.metrics import AnswerRelevancyMetric
|
||
|
||
answer_relevancy = AnswerRelevancyMetric(
|
||
model="gpt-4.1",
|
||
)
|
||
```
|
||
|
||
</Tab>
|
||
</Tabs>
|
||
|
||
There are **ZERO** mandatory and **SEVEN** optional parameters when creating a `GPTModel`:
|
||
|
||
- [Optional] `model`: A string specifying the name of the GPT model to use. Defaulted to `OPENAI_MODEL_NAME` if not set; falls back to <DefaultLLMModel />.
|
||
- [Optional] `api_key`: A string specifying the OpenAI API key for authentication. Defaults to `OPENAI_API_KEY` if not passed; raises an error at runtime if unset.
|
||
- [Optional] `base_url`: A string specifying your OpenAI URL.
|
||
- [Optional] `temperature`: A float specifying the model temperature. Defaults to `TEMPERATURE` if not passed; falls back to `0.0` if unset.
|
||
- [Optional] `cost_per_input_token`: A float specifying the cost for each input token for the provided model. Defaults to `OPENAI_COST_PER_INPUT_TOKEN` if available in `deepeval`'s model cost registry, else `None`.
|
||
- [Optional] `cost_per_output_token`: A float specifying the cost for each output token for the provided model. Defaults to `OPENAI_COST_PER_OUTPUT_TOKEN` if available in `deepeval`'s model cost registry, else `None`.
|
||
- [Optional] `generation_kwargs`: A dictionary of additional generation parameters forwarded to the OpenAI `chat.completions.create(...)` and `beta.chat.completions.parse(...)` calls.
|
||
|
||
:::info
|
||
You can use custom providers by setting `api_key` and `base_url` with your custom provider's details.
|
||
:::
|
||
|
||
:::tip
|
||
Any `**kwargs` you would like to use for your model can be passed through the `generation_kwargs` parameter. However, we request you to double check the params supported by the model and your model provider in their [official docs](https://platform.openai.com/docs/api-reference/responses/create).
|
||
:::
|
||
|
||
### Available OpenAI Models
|
||
|
||
:::note
|
||
This list only displays some of the available models. For a comprehensive list, refer to the OpenAI's official documentation.
|
||
:::
|
||
|
||
Below is a list of commonly used OpenAI models:
|
||
|
||
- `gpt-5`
|
||
- `gpt-5-mini`
|
||
- `gpt-5-nano`
|
||
- `gpt-4.1`
|
||
- `gpt-4.5-preview`
|
||
- `gpt-4o`
|
||
- `gpt-4o-mini`
|
||
- `o1`
|
||
- `o1-pro`
|
||
- `o1-mini`
|
||
- `o3-mini`
|
||
- `gpt-4-turbo`
|
||
- `gpt-4`
|
||
- `gpt-4-32k`
|
||
- `gpt-3.5-turbo`
|
||
- `gpt-3.5-turbo-instruct`
|
||
- `gpt-3.5-turbo-16k-0613`
|
||
- `davinci-002`
|
||
- `babbage-002`
|
||
|
||
:::caution[G-Eval requires log probabilities]
|
||
`GEval` and `ConversationalGEval` compute scores from token-level log probabilities, which some OpenAI models do not expose. The `gpt-5`, `gpt-5-mini`, `gpt-5-nano`, `gpt-5-chat-latest`, `gpt-5.1`, and `gpt-5.2` models return no log probabilities, so passing any of them as the judge model for these metrics raises `AttributeError: log_probs unsupported.`. For `GEval` and `ConversationalGEval`, use a log-probability-capable model such as `gpt-5.4` (the default) or `gpt-4.1`. Other metrics are unaffected.
|
||
:::
|