promptfoo--promptfoo/site/docs/guides/evaluate-llm-temperature.md

title, sidebar_label, description

title	sidebar_label	description
How to Choose the Right LLM Temperature Setting	Choosing the right temperature for your LLM	Learn how to find the optimal LLM temperature setting by running systematic evaluations. Compare temperature 0.1-1.0 for creativity vs consistency.

Choosing the right temperature for your LLM

The temperature setting in language models is like a dial that adjusts how predictable or surprising the responses from the model will be, helping application developers fine-tune the AI's creativity to suit different tasks.

In general, a low temperature leads to "safer", more expected words, while a higher temperature encourages the model to choose less obvious words. This is why higher temperature is commonly associated with more creative outputs.

Under the hood, temperature adjusts how the model calculates the likelihood of each word it might pick next.

The temperature parameter affects each output token by scaling the logits (the raw output scores from the model) before they are passed through the softmax function that turns them into probabilities. Lower temperatures sharpen the distinction between high and low scores, making the high scores more dominant, while higher temperatures flatten this distinction, giving lower-scoring words a better chance of being chosen.

Finding the optimal temperature

The best way to find the optimal temperature parameter is to run a systematic evaluation.

The optimal temperature will always depend on your specific use case. That's why it's important to:

• Quantitatively measure the performance of your prompt+model outputs at various temperature settings.
• Ensure consistency in the model's behavior, which is particularly important when deploying LLMs in production environments.
• Compare the model's performance against a set of predefined criteria or benchmarks

By running a temperature eval, you can make data-driven decisions that balance the reliability and creativity of your LLM app.

Prerequisites

Before setting up an evaluation, create a new directory and a promptfooconfig.yaml file. For more information on getting started with promptfoo, refer to the getting started guide.

Evaluating

Here's an example configuration that compares the outputs of Claude Sonnet 4.6 at a low temperature (0.2) and a high temperature (0.9):

prompts:
  - 'Respond to the following instruction: {{message}}'

providers:
  - id: anthropic:claude-sonnet-4-6
    label: claude-sonnet-lowtemp
    config:
      temperature: 0.2
  - id: anthropic:claude-sonnet-4-6
    label: claude-sonnet-hightemp
    config:
      temperature: 0.9

tests:
  - vars:
      message: What's the capital of France?
  - vars:
      message: Write a poem about the sea.
  - vars:
      message: Generate a list of potential risks for a space mission.
  - vars:
      message: Did Henry VIII have any grandchildren?

In the above configuration, we just use a boilerplate prompt because we're more interested in comparing the different models.

We define two providers that call the same model (Claude Sonnet 4.6) with different temperature settings. The label field helps us distinguish between the two when reviewing the results.

The tests section includes our test cases that will be run against both temperature settings.

To run the evaluation, use the following command:

npx promptfoo@latest eval

This command shows the outputs side-by-side in the command line.

Adding automated checks

To automatically check for expected outputs, you can define assertions in your test cases. Assertions allow you to specify the criteria that the LLM output should meet, and promptfoo will evaluate the output against these criteria.

For the example of Henry VIII's grandchildren, you might want to ensure that the output is factually correct. You can use a model-graded-closedqa assertion to automatically check that the output does not contain any hallucinated information.

Here's how you can add an assertion to the test case:

tests:
  - description: 'Check for hallucination on Henry VIII grandchildren question'
    vars:
      message: Did Henry VIII have any grandchildren?
    // highlight-start
    assert:
      - type: llm-rubric
        value: Henry VIII didn't have any grandchildren
    // highlight-end

This assertion will use a language model to determine whether the LLM output adheres to the criteria.

In our tests, the model confidently states that Henry VIII had grandchildren at both temperature settings — which is incorrect. This demonstrates why automated assertions are valuable: they catch factual errors you might miss when manually reviewing outputs. Without the llm-rubric check, this hallucination would have gone unnoticed regardless of temperature.

There are many other assertion types. For example, we can check that the answer to the "space mission risks" question includes all of the following terms:

tests:
  vars:
    message: Generate a list of potential risks for a space mission.
  assert:
    - type: icontains-all
      value:
        - 'radiation'
        - 'isolation'
        - 'environment'

It's worth spending a few minutes to set up these automated checks. They help streamline the evaluation process and quickly identify bad outputs.

After the evaluation is complete, you can use the web viewer to review the outputs and compare the performance at different temperatures:

npx promptfoo@latest view

Evaluating randomness

LLMs are inherently nondeterministic, which means their outputs will vary with each call at nonzero temperatures (and sometimes even at zero temperature).

The eval command has a parameter, repeat, which runs each test multiple times:

promptfoo eval --repeat 3

The above command runs the LLM three times for each test case, helping you get a more complete sample of how it performs at a given temperature.

By default, each repeat index has its own cache entry, so re-running the same eval can replay repeat 0, repeat 1, and repeat 2 independently. If you want fresh generations on every run while sampling temperature, add --no-cache.