567-labs--instructor/docs/concepts/reask_validation.md

---
title: Enhancing AI Validations with Pydantic's Framework
description: Learn how to improve AI outputs using Pydantic for validation and reasking techniques.
---

# Validation and Reasking

Instead of framing "self-critique" or "self-reflection" in AI as new concepts, we can view them as validation errors with clear error messages that the system can use to self-correct.

## Pydantic

Pydantic offers a customizable and expressive validation framework for Python. Instructor leverages Pydantic's validation framework to provide a uniform developer experience for both code-based and LLM-based validation, as well as a reasking mechanism for correcting LLM outputs based on validation errors. To learn more check out the [Pydantic docs](https://docs.pydantic.dev/latest/concepts/validators/) on validators.

!!! note "Good llm validation is just good validation"

    If you want to see some more examples on validators checkout our blog post [Good LLM validation is just good validation](https://python.useinstructor.com/blog/2023/10/23/good-llm-validation-is-just-good-validation/)

### Code-based Validation Example

First define a Pydantic model with a validator using the `Annotation` class from `typing_extensions`.

Enforce a naming rule using Pydantic's built-in validation:

```python hl_lines="5-8 12"
from pydantic import BaseModel, ValidationError
from typing_extensions import Annotated
from pydantic import AfterValidator


def name_must_contain_space(v: str) -> str:
    if " " not in v:
        raise ValueError("Name must contain a space.")
    return v.lower()


class UserDetail(BaseModel):
    age: int
    name: Annotated[str, AfterValidator(name_must_contain_space)]


try:
    person = UserDetail(age=29, name="Jason")
except ValidationError as e:
    print(e)
    """
    1 validation error for UserDetail
    name
      Value error, Name must contain a space. [type=value_error, input_value='Jason', input_type=str]
        For further information visit https://errors.pydantic.dev/2.11/v/value_error
    """
```

#### Output for Code-Based Validation

```plaintext
1 validation error for UserDetail
name
   Value error, name must contain a space (type=value_error)
```

As we can see, Pydantic raises a validation error when the name attribute does not contain a space. This is a simple example, but it demonstrates how Pydantic can be used to validate attributes of a model.

### LLM-Based Validation Example

LLM-based validation can also be plugged into the same Pydantic model. Here, if the answer attribute contains content that violates the rule "don't say objectionable things," Pydantic will raise a validation error.

```python hl_lines="9 15"
import instructor
from instructor import llm_validator
from pydantic import BaseModel, ValidationError, BeforeValidator
from typing_extensions import Annotated


# Apply the patch to the OpenAI client
client = instructor.from_provider("openai/gpt-4.1-mini")


class QuestionAnswer(BaseModel):
    question: str
    answer: Annotated[
        str,
        BeforeValidator(llm_validator("don't say objectionable things", client=client)),
    ]


try:
    qa = QuestionAnswer(
        question="What is the meaning of life?",
        answer="The meaning of life is to be evil and steal",
    )
except ValidationError as e:
    print(e)
    """
    1 validation error for QuestionAnswer
    answer
      Assertion failed, The statement promotes objectionable behavior by encouraging evil and stealing. [type=assertion_error, input_value='The meaning of life is to be evil and steal', input_type=str]
        For further information visit https://errors.pydantic.dev/2.11/v/assertion_error
    """
```

#### Output for LLM-Based Validation

It is important to note here that the error message is generated by the LLM, not the code, so it'll be helpful for re-asking the model.

```plaintext
1 validation error for QuestionAnswer
answer
   Assertion failed, The statement is objectionable. (type=assertion_error)
```

## Using Reasking Logic to Correct Outputs

Validators are a great tool for ensuring some property of the outputs. When you use the `patch()` method with the `openai` client, you can use the `max_retries` parameter to set the number of times you can reask the model to correct the output.

It is a great layer of defense against bad outputs of two forms:

1. Pydantic Validation Errors (code or llm based)
2. JSON Decoding Errors (when the model returns a bad response)

### Step 1: Define the Response Model with Validators

Notice that the field validator wants the name in uppercase, but the user input is lowercase. The validator will raise a `ValueError` if the name is not in uppercase.

```python hl_lines="12-17"
import instructor
from pydantic import BaseModel, field_validator

# Apply the patch to the OpenAI client
client = instructor.from_provider("openai/gpt-4.1-mini")


class UserDetails(BaseModel):
    name: str
    age: int

    @field_validator("name")
    @classmethod
    def validate_name(cls, v):
        if v.upper() != v:
            raise ValueError("Name must be in uppercase.")
        return v
```

### Step 2. Using the Client with Retries

Here, the `UserDetails` model is passed as the `response_model`, and `max_retries` is set to 2.

```python
import instructor
from pydantic import BaseModel

client = instructor.from_provider(
    "openai/gpt-4.1-mini",
    mode=instructor.Mode.TOOLS,
)


class UserDetails(BaseModel):
    name: str
    age: int


model = client.create(
    response_model=UserDetails,
    max_retries=2,
    messages=[
        {"role": "user", "content": "Extract jason is 25 years old"},
    ],
)

print(model.model_dump_json(indent=2))
"""
{
  "name": "jason",
  "age": 25
}
"""
```

### What happens behind the scenes?

Behind the scenes, the `instructor.from_provider()` method adds a `max_retries` parameter to the `openai.ChatCompletion.create()` method. The `max_retries` parameter will trigger up to 2 reattempts if the `name` attribute fails the uppercase validation in `UserDetails`.

```python
from pydantic import ValidationError


try:
    ...
except ValidationError as e:
    kwargs["messages"].append(response.choices[0].message)
    kwargs["messages"].append(
        {
            "role": "user",
            "content": f"Please correct the function call; errors encountered:\n{e}",
        }
    )
```

## Advanced Validation Techniques

### Using Context for Dynamic Validation

The `context` parameter allows you to pass additional data to your validators, enabling validation against runtime data like source documents, allowed values, or external references. This is accessed in validators via `ValidationInfo`.

Here's a complete example showing context-based validation:

```python
import instructor
from pydantic import BaseModel, ValidationInfo, field_validator

client = instructor.from_provider("openai/gpt-4.1-mini")


class QuoteExtraction(BaseModel):
    """Extract a claim with a supporting quote from source text."""

    claim: str
    supporting_quote: str

    @field_validator('supporting_quote')
    @classmethod
    def verify_quote_in_source(cls, v: str, info: ValidationInfo):
        """Verify the quote exists in the source text."""
        import re

        context = info.context
        if context:
            source_text = context.get('source_text', '')
            # Normalize whitespace for comparison
            normalized_source = re.sub(r'\s+', ' ', source_text.strip())
            normalized_quote = re.sub(r'\s+', ' ', v.strip())
            if normalized_quote not in normalized_source:
                raise ValueError(
                    f"The quote must be an exact substring from the source text. "
                    f"Quote '{v}' was not found in the source."
                )
        return v


source_text = """
The Python programming language was created by Guido van Rossum 
and first released in 1991. It emphasizes code readability and 
simplicity, making it popular for beginners and experts alike.
"""

extraction = client.create(
    response_model=QuoteExtraction,
    max_retries=2,
    messages=[
        {
            "role": "system",
            "content": "Extract a claim and find an exact quote from the text that supports it.",
        },
        {
            "role": "user",
            "content": "Source text: {{ source_text }}\n\nExtract a claim about Python.",
        },
    ],
    context={"source_text": source_text},
)

print(f"Claim: {extraction.claim}")
#> Claim: Python emphasizes code readability and simplicity.
print(f"Quote: {extraction.supporting_quote}")
"""
Quote: It emphasizes code readability and simplicity, making it popular for beginners and experts alike.
"""
```

In this example:
- The `context` parameter passes the source text to the validator
- `ValidationInfo` provides access to the context in the validator
- If the LLM generates a quote that doesn't exist in the source, validation fails and the model is re-asked

For more advanced examples including multi-field validation and citation verification, check out our [exact citations example](../examples/exact_citations.md).

## Optimizing Token usage

Pydantic automatically includes a URL within the error message itself when an error is thrown so that users can learn more about the specific error that was thrown. Some users might want to remove this URL since it adds extra tokens that otherwise might not add much value to the validation process.

We've created a small helper function that you can use below which removes this url in the error message

```python hl_lines="6"
from instructor.utils import disable_pydantic_error_url
from pydantic import BaseModel, ValidationError
from typing_extensions import Annotated
from pydantic import AfterValidator

disable_pydantic_error_url()  # (1)!


def name_must_contain_space(v: str) -> str:
    if " " not in v:
        raise ValueError("Name must contain a space.")
    return v.lower()


class UserDetail(BaseModel):
    age: int
    name: Annotated[str, AfterValidator(name_must_contain_space)]


try:
    person = UserDetail(age=29, name="Jason")
except ValidationError as e:
    print(e)
    """
    1 validation error for UserDetail
    name
      Value error, Name must contain a space. [type=value_error, input_value='Jason', input_type=str]
    """
```

1.  We disable the error by setting an environment variable `PYDANTIC_ERRORS_INCLUDE_URL` to `0`. This is valid only for the duration that the script is executed for, once the function is not called, the original behaviour is restored.

## See Also

- [Validation](./validation.md) - Core validation concepts and strategies
- [Retrying](./retrying.md) - Configure automatic retry behavior with Tenacity
- [Custom Validators](../learning/validation/custom_validators.md) - Build custom validation logic
- [Field Validation](../learning/patterns/field_validation.md) - Field-level validation patterns
- [Retry Mechanisms](../learning/validation/retry_mechanisms.md) - Practical retry configuration guide

## Takeaways

By integrating these advanced validation techniques, we not only improve the quality and reliability of LLM-generated content, but also pave the way for more autonomous and effective systems.