wehub/567-labs--instructor
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
Ruff / Ruff (push) Has been cancelled
Details
Test / Core Tests (push) Has been cancelled
Details
Test / Offline Coverage Tests (Python 3.10) (push) Has been cancelled
Details
Test / Offline Coverage Tests (Python 3.11) (push) Has been cancelled
Details
Test / Offline Coverage Tests (Python 3.12) (push) Has been cancelled
Details
Test / Offline Coverage Tests (Python 3.13) (push) Has been cancelled
Details
Test / Offline Coverage Tests (Python 3.9) (push) Has been cancelled
Details
Test / Full Coverage (Python 3.11) (push) Has been cancelled
Details
Test / Core Provider Tests (OpenAI) (push) Has been cancelled
Details
Test / Core Provider Tests (Anthropic) (push) Has been cancelled
Details
Test / Core Provider Tests (Google) (push) Has been cancelled
Details
Test / Core Provider Tests (Other) (push) Has been cancelled
Details
Test / Anthropic Tests (push) Has been cancelled
Details
Test / Gemini Tests (push) Has been cancelled
Details
Test / Google GenAI Tests (push) Has been cancelled
Details
Test / Vertex AI Tests (push) Has been cancelled
Details
Test / OpenAI Tests (push) Has been cancelled
Details
Test / Writer Tests (push) Has been cancelled
Details
Test / Auto Client Tests (push) Has been cancelled
Details
ty / type-check (push) Has been cancelled
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:36:38 +08:00
commit 97e91a83f3
978 changed files with 159975 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/tutorials/1-introduction.ipynb
+632
View File
@@ -0,0 +1,632 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Working with structured outputs\n",
"\n",
"If you've seen my [talk](https://www.youtube.com/watch?v=yj-wSRJwrrc&t=1s) on this topic, you can skip this chapter.\n",
"\n",
"tl;dr\n",
"\n",
"When we work with LLMs you find that many times we are not building chatbots, instead we're working with structured outputs in order to solve a problem by returning machine readable data. However the way we think about the problem is still very much influenced by the way we think about chatbots. This is a problem because it leads to a lot of confusion and frustration. In this chapter we'll try to understand why this happens and how we can fix it.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import traceback"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"RED = \"\\033[91m\"\n",
"RESET = \"\\033[0m\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The fundamental problem with JSON and Dictionaries\n",
"\n",
"Lets say we have a simple JSON object, and we want to work with it. We can use the `json` module to load it into a dictionary, and then work with it. However, this is a bit of a pain, because we have to manually check the types of the data, and we have to manually check if the data is valid. For example, lets say we have a JSON object that looks like this:\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"data = [{\"first_name\": \"Jason\", \"age\": 10}, {\"firstName\": \"Jason\", \"age\": \"10\"}]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We have a `name` field, which is a string, and an `age` field, which is an integer. However, if we were to load this into a dictionary, we would have no way of knowing if the data is valid. For example, we could have a string for the age, or we could have a float for the age. We could also have a string for the name, or we could have a list for the name.\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Jason is 10\n",
"None is 10\n",
"Next year Jason will be 11 years old\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Traceback (most recent call last):\n",
" File \"/var/folders/l2/jjqj299126j0gycr9kkkt9xm0000gn/T/ipykernel_24047/2607506000.py\", line 10, in <module>\n",
" age_next_year = age + 1\n",
" ~~~~^~~\n",
"TypeError: can only concatenate str (not \"int\") to str\n"
]
}
],
"source": [
"for obj in data:\n",
" name = obj.get(\"first_name\")\n",
" age = obj.get(\"age\")\n",
" print(f\"{name} is {age}\")\n",
"\n",
"for obj in data:\n",
" name = obj.get(\"first_name\")\n",
" age = obj.get(\"age\")\n",
" try:\n",
" age_next_year = age + 1\n",
" print(f\"Next year {name} will be {age_next_year} years old\")\n",
" except TypeError:\n",
" traceback.print_exc()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You see that while we were able to program with a dictionary, we had issues with the data being valid. We would have had to manually check the types of the data, and we had to manually check if the data was valid. This is a pain, and we can do better.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Pydantic to the rescue\n",
"\n",
"Pydantic is a library that allows us to define data structures, and then validate them.\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Person(name='Sam', age=30)"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from pydantic import BaseModel, Field, ValidationError\n",
"\n",
"\n",
"class Person(BaseModel):\n",
" name: str\n",
" age: int\n",
"\n",
"\n",
"person = Person(name=\"Sam\", age=30)\n",
"person"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Person(name='Sam', age=30)"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Data is correctly casted to the right type\n",
"person = Person.model_validate({\"name\": \"Sam\", \"age\": \"30\"})\n",
"person"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Traceback (most recent call last):\n",
" File \"/var/folders/l2/jjqj299126j0gycr9kkkt9xm0000gn/T/ipykernel_24047/3040264600.py\", line 5, in <module>\n",
" assert person.age == 20\n",
" ^^^^^^^^^^^^^^^^\n",
"AssertionError\n"
]
}
],
"source": [
"assert person.name == \"Sam\"\n",
"assert person.age == 30\n",
"\n",
"try:\n",
" assert person.age == 20\n",
"except AssertionError:\n",
" traceback.print_exc()"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Validation Error:\n",
"Field: name, Error: Field required\n",
"Field: age, Error: Input should be a valid integer, unable to parse string as an integer\n",
"\u001b[91m\n",
"Original Traceback Below\u001b[0m\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Traceback (most recent call last):\n",
" File \"/var/folders/l2/jjqj299126j0gycr9kkkt9xm0000gn/T/ipykernel_24047/621989455.py\", line 3, in <module>\n",
" person = Person.model_validate({\"first_name\": \"Sam\", \"age\": \"30.2\"})\n",
" ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
" File \"/opt/homebrew/Caskroom/miniconda/base/envs/instructor/lib/python3.11/site-packages/pydantic/main.py\", line 509, in model_validate\n",
" return cls.__pydantic_validator__.validate_python(\n",
" ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
"pydantic_core._pydantic_core.ValidationError: 2 validation errors for Person\n",
"name\n",
" Field required [type=missing, input_value={'first_name': 'Sam', 'age': '30.2'}, input_type=dict]\n",
" For further information visit https://errors.pydantic.dev/2.6/v/missing\n",
"age\n",
" Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='30.2', input_type=str]\n",
" For further information visit https://errors.pydantic.dev/2.6/v/int_parsing\n"
]
}
],
"source": [
"# Data is validated to get better error messages\n",
"try:\n",
" person = Person.model_validate({\"first_name\": \"Sam\", \"age\": \"30.2\"})\n",
"except ValidationError as e:\n",
" print(\"Validation Error:\")\n",
" for error in e.errors():\n",
" print(f\"Field: {error['loc'][0]}, Error: {error['msg']}\")\n",
"\n",
" print(f\"{RED}\\nOriginal Traceback Below{RESET}\")\n",
" traceback.print_exc()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"By introducing pydantic into any python codebase you can get a lot of benefits. You can get type checking, you can get validation, and you can get autocomplete. This is a huge win, because it means you can catch errors before they happen. This is even more useful when we rely on language models to generate data for us.\n",
"\n",
"You can also define validators that are run on the data. This is useful because it means you can catch errors before they happen. For example, you can define a validator that checks if the age is greater than 0. This is useful because it means you can catch errors before they happen.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Fundamental problem with asking for JSON from OpenAI\n",
"\n",
"As we shall see below, the correct json format would be something of the format below:\n",
"\n",
"```python\n",
"{\n",
" \"name\": \"Jason\",\n",
" \"age\": 10\n",
"}\n",
"```\n",
"\n",
"However, we get errorenous outputs like:\n",
"\n",
"```python\n",
"{\n",
" \"jason\": 10\n",
"}\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"json that we want:\n",
"\n",
"{\n",
" \"name\": \"Jason\",\n",
" \"age\": 10\n",
"}\n",
"\n",
"error!!\n",
"{\n",
" \"jason\": 10\n",
"}\n",
"correctly parsed person=Person(name='Jason', age=10)\n",
"correctly parsed person=Person(name='jason', age=10)\n",
"error!!\n",
"{\n",
" \"Jason\": {\n",
" \"age\": 10\n",
" }\n",
"}\n",
"error!!\n",
"{\n",
" \"Jason\": {\n",
" \"age\": 10\n",
" }\n",
"}\n",
"error!!\n",
"{\n",
" \"Jason\": {\n",
" \"age\": 10\n",
" }\n",
"}\n",
"error!!\n",
"{\n",
" \"Jason\": {\n",
" \"age\": 10\n",
" }\n",
"}\n",
"correctly parsed person=Person(name='Jason', age=10)\n",
"correctly parsed person=Person(name='Jason', age=10)\n",
"error!!\n",
"{\n",
" \"jason\": 10\n",
"}\n"
]
}
],
"source": [
"from openai import OpenAI\n",
"\n",
"client = OpenAI()\n",
"\n",
"resp = client.create(\n",
" model=\"gpt-5.4-mini\",\n",
" messages=[\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": \"Please give me jason is 10 as a json object ```json\\n\",\n",
" },\n",
" ],\n",
" n=10,\n",
" temperature=1,\n",
")\n",
"\n",
"print(\"json that we want:\")\n",
"print(\n",
" \"\"\"\n",
"{\n",
" \"name\": \"Jason\",\n",
" \"age\": 10\n",
"}\n",
"\"\"\"\n",
")\n",
"\n",
"for choice in resp.choices:\n",
" json = choice.message.content\n",
" try:\n",
" person = Person.model_validate_json(json)\n",
" print(f\"correctly parsed {person=}\")\n",
" except Exception as e:\n",
" print(\"error!!\")\n",
" print(json)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Introduction to Function Calling\n",
"\n",
"The json could be anything! We could add more and more into a prompt and hope it works, or we can use something called [function calling](https://platform.openai.com/docs/guides/function-calling) to directly specify the schema we want.\n",
"\n",
"**Function Calling**\n",
"\n",
"In an API call, you can describe _functions_ and have the model intelligently\n",
"choose to output a _JSON object_ containing _arguments_ to call one or many\n",
"functions. The Chat Completions API does **not** call the function; instead, the\n",
"model generates _JSON_ that you can use to call the function in **your code**.\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"PersonBirthday(name='Jason Liu', age=30, birthday=datetime.date(1994, 3, 26))"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import datetime\n",
"\n",
"\n",
"class PersonBirthday(BaseModel):\n",
" name: str\n",
" age: int\n",
" birthday: datetime.date\n",
"\n",
"\n",
"schema = {\n",
" \"properties\": {\n",
" \"name\": {\"type\": \"string\"},\n",
" \"age\": {\"type\": \"integer\"},\n",
" \"birthday\": {\"type\": \"string\", \"format\": \"YYYY-MM-DD\"},\n",
" },\n",
" \"required\": [\"name\", \"age\"],\n",
" \"type\": \"object\",\n",
"}\n",
"\n",
"resp = client.create(\n",
" model=\"gpt-5.4-mini\",\n",
" messages=[\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": f\"Extract `Jason Liu is thirty years old his birthday is yesterday` into json today is {datetime.date.today()}\",\n",
" },\n",
" ],\n",
" functions=[{\"name\": \"Person\", \"parameters\": schema}],\n",
" function_call=\"auto\",\n",
")\n",
"\n",
"PersonBirthday.model_validate_json(resp.choices[0].message.function_call.arguments)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"But it turns out, pydantic actually not only does our serialization, we can define the schema as well as add additional documentation!\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'properties': {'name': {'title': 'Name', 'type': 'string'},\n",
" 'age': {'title': 'Age', 'type': 'integer'},\n",
" 'birthday': {'format': 'date', 'title': 'Birthday', 'type': 'string'}},\n",
" 'required': ['name', 'age', 'birthday'],\n",
" 'title': 'PersonBirthday',\n",
" 'type': 'object'}"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"PersonBirthday.model_json_schema()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can even define nested complex schemas, and documentation with ease.\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'$defs': {'Address': {'properties': {'address': {'description': 'Full street address',\n",
" 'title': 'Address',\n",
" 'type': 'string'},\n",
" 'city': {'title': 'City', 'type': 'string'},\n",
" 'state': {'title': 'State', 'type': 'string'}},\n",
" 'required': ['address', 'city', 'state'],\n",
" 'title': 'Address',\n",
" 'type': 'object'}},\n",
" 'description': 'A Person with an address',\n",
" 'properties': {'name': {'title': 'Name', 'type': 'string'},\n",
" 'age': {'title': 'Age', 'type': 'integer'},\n",
" 'address': {'$ref': '#/$defs/Address'}},\n",
" 'required': ['name', 'age', 'address'],\n",
" 'title': 'PersonAddress',\n",
" 'type': 'object'}"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"class Address(BaseModel):\n",
" address: str = Field(description=\"Full street address\")\n",
" city: str\n",
" state: str\n",
"\n",
"\n",
"class PersonAddress(Person):\n",
" \"\"\"A Person with an address\"\"\"\n",
"\n",
" address: Address\n",
"\n",
"\n",
"PersonAddress.model_json_schema()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"These simple concepts become what we built into `instructor` and most of the work has been around documenting how we can leverage schema engineering.\n",
"Except now we use `instructor.patch()` to add a bunch more capabilities to the OpenAI SDK.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# The core idea around Instructor\n",
"\n",
"1. Using function calling allows us use a llm that is finetuned to use json_schema and output json.\n",
"2. Pydantic can be used to define the object, schema, and validation in one single class, allow us to encapsulate everything neatly\n",
"3. As a library with 100M downloads, we can leverage pydantic to do all the heavy lifting for us and fit nicely with the python ecosystem\n"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"PersonAddress(name='Jason Liu', age=30, address=Address(address='123 Main St', city='San Francisco', state='CA'))"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import instructor\n",
"import datetime\n",
"\n",
"# patch the client to add `response_model` to the `create` method\n",
"client = instructor.patch(OpenAI(), mode=instructor.Mode.MD_JSON)\n",
"\n",
"resp = client.create(\n",
" model=\"gpt-5.4-mini\",\n",
" messages=[\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": f\"\"\"\n",
" Today is {datetime.date.today()}\n",
"\n",
" Extract `Jason Liu is thirty years old his birthday is yesterday`\n",
" he lives at 123 Main St, San Francisco, CA\"\"\",\n",
" },\n",
" ],\n",
" response_model=PersonAddress,\n",
")\n",
"resp"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"By defining `response_model` we can leverage pydantic to do all the heavy lifting. Later we'll introduce the other features that `instructor.patch()` adds to the OpenAI SDK.\n",
"but for now, this small change allows us to do a lot more with the API.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Is instructor the only way to do this?\n",
"\n",
"No. Libraries like Marvin, Langchain, and Llamaindex all now leverage the Pydantic object in similar ways. The goal is to be as light weight as possible, get you as close as possible to the openai api, and then get out of your way.\n",
"\n",
"More importantly, we've also added straight forward validation and reasking to the mix.\n",
"\n",
"The goal of instructor is to show you how to think about structured prompting and provide examples and documentation that you can take with you to any framework.\n",
"\n",
"For further exploration:\n",
"\n",
"- [Marvin](https://www.askmarvin.ai/)\n",
"- [Langchain](https://python.langchain.com/docs/modules/model_io/output_parsers/pydantic)\n",
"- [LlamaIndex](https://gpt-index.readthedocs.io/en/latest/examples/output_parsing/openai_pydantic_program.html)\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.8"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
docs/tutorials/2-tips.ipynb
+518
View File
@@ -0,0 +1,518 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "8bb7d0d0-2b7f-4e9e-8565-467dc5c6fd22",
"metadata": {},
"source": [
"# General Tips on Prompting\n",
"\n",
"Before we get into some big applications of schema engineering I want to equip you with the tools for success.\n",
"This notebook is to share some general advice when using prompts to get the most of your models.\n",
"\n",
"Before you might think of prompt engineering as massaging this wall of text, almost like coding in a notepad. But with schema engineering you can get a lot more out of your prompts with a lot less work.\n"
]
},
{
"cell_type": "markdown",
"id": "8a785c25-b08d-4ab4-bbd7-22e3b090c2ed",
"metadata": {},
"source": [
"## Classification\n",
"\n",
"For classification we've found theres generally two methods of modeling.\n",
"\n",
"1. using Enums\n",
"2. using Literals\n",
"\n",
"Use an enum in Python when you need a set of named constants that are related and you want to ensure type safety, readability, and prevent invalid values. Enums are helpful for grouping and iterating over these constants.\n",
"\n",
"Use literals when you have a small, unchanging set of values that you don't need to group or iterate over, and when type safety and preventing invalid values is less of a concern. Literals are simpler and more direct for basic, one-off values.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "fdf5e1d9-31ad-4e8a-a55e-e2e70fff598d",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'age': 17, 'name': 'Harry Potter', 'house': <House.Gryffindor: 'gryffindor'>}"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import instructor\n",
"from openai import OpenAI\n",
"\n",
"from enum import Enum\n",
"from pydantic import BaseModel, Field\n",
"from typing_extensions import Literal\n",
"\n",
"\n",
"client = instructor.from_provider(\"openai/gpt-4o\")\n",
"\n",
"\n",
"# Tip: Do not use auto() as they cast to 1,2,3,4\n",
"class House(Enum):\n",
" Gryffindor = \"gryffindor\"\n",
" Hufflepuff = \"hufflepuff\"\n",
" Ravenclaw = \"ravenclaw\"\n",
" Slytherin = \"slytherin\"\n",
"\n",
"\n",
"class Character(BaseModel):\n",
" age: int\n",
" name: str\n",
" house: House\n",
"\n",
" def say_hello(self):\n",
" print(\n",
" f\"Hello, I'm {self.name}, I'm {self.age} years old and I'm from {self.house.value.title()}\"\n",
" )\n",
"\n",
"\n",
"resp = client.create(\n",
" model=\"gpt-5.4-mini\",\n",
" messages=[{\"role\": \"user\", \"content\": \"Harry Potter\"}],\n",
" response_model=Character,\n",
")\n",
"resp.model_dump()"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "c609eb44",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Hello, I'm Harry Potter, I'm 17 years old and I'm from Gryffindor\n"
]
}
],
"source": [
"resp.say_hello()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "03db160c-81e9-4373-bfec-7a107224b6dd",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'age': 11, 'name': 'Harry Potter', 'house': 'Gryffindor'}"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"class Character(BaseModel):\n",
" age: int\n",
" name: str\n",
" house: Literal[\"Gryffindor\", \"Hufflepuff\", \"Ravenclaw\", \"Slytherin\"]\n",
"\n",
"\n",
"resp = client.create(\n",
" model=\"gpt-5.4-mini\",\n",
" messages=[{\"role\": \"user\", \"content\": \"Harry Potter\"}],\n",
" response_model=Character,\n",
")\n",
"resp.model_dump()"
]
},
{
"cell_type": "markdown",
"id": "803e0ce6-6e7e-4d86-a7a8-49ebaad0a40b",
"metadata": {},
"source": [
"## Arbitrary properties\n",
"\n",
"Often times there are long properties that you might want to extract from data that we can not specify in advanced. We can get around this by defining an arbitrary key value store like so:\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "0e7938b8-4666-4df4-bd80-f53e8baf7550",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'age': 38,\n",
" 'name': 'Severus Snape',\n",
" 'house': 'Slytherin',\n",
" 'properties': [{'key': 'role', 'value': 'Potions Master'},\n",
" {'key': 'patronus', 'value': 'Doe'},\n",
" {'key': 'loyalty', 'value': 'Dumbledore'},\n",
" {'key': 'played_by', 'value': 'Alan Rickman'}]}"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"class Property(BaseModel):\n",
" key: str = Field(description=\"Must be snake case\")\n",
" value: str\n",
"\n",
"\n",
"class Character(BaseModel):\n",
" age: int\n",
" name: str\n",
" house: Literal[\"Gryffindor\", \"Hufflepuff\", \"Ravenclaw\", \"Slytherin\"]\n",
" properties: list[Property]\n",
"\n",
"\n",
"resp = client.create(\n",
" model=\"gpt-5.4-mini\",\n",
" messages=[{\"role\": \"user\", \"content\": \"Snape from Harry Potter\"}],\n",
" response_model=Character,\n",
")\n",
"resp.model_dump()"
]
},
{
"cell_type": "markdown",
"id": "b3e62f68-a79f-4f65-9c1f-726e4e2d340a",
"metadata": {},
"source": [
"## Limiting the length of lists\n",
"\n",
"In later chapters we'll talk about how to use validators to assert the length of lists but we can also use prompting tricks to enumerate values. Here we'll define a index to count the properties.\n",
"\n",
"In this following example instead of extraction we're going to work on generation instead.\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "69a58d01-ab6f-41b6-bc0c-b0e55fdb6fe4",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'age': 38,\n",
" 'name': 'Severus Snape',\n",
" 'house': 'Slytherin',\n",
" 'properties': [{'index': '1',\n",
" 'key': 'position_at_hogwarts',\n",
" 'value': 'Potions Master'},\n",
" {'index': '2', 'key': 'patronus_form', 'value': 'Doe'},\n",
" {'index': '3', 'key': 'loyalty', 'value': 'Albus Dumbledore'},\n",
" {'index': '4', 'key': 'played_by', 'value': 'Alan Rickman'},\n",
" {'index': '5', 'key': 'final_act', 'value': 'Protecting Harry Potter'}]}"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"class Property(BaseModel):\n",
" index: str = Field(..., description=\"Monotonically increasing ID\")\n",
" key: str = Field(description=\"Must be snake case\")\n",
" value: str\n",
"\n",
"\n",
"class Character(BaseModel):\n",
" age: int\n",
" name: str\n",
" house: Literal[\"Gryffindor\", \"Hufflepuff\", \"Ravenclaw\", \"Slytherin\"]\n",
" properties: list[Property] = Field(\n",
" ...,\n",
" description=\"Numbered list of arbitrary extracted properties, should be exactly 5\",\n",
" )\n",
"\n",
"\n",
"resp = client.create(\n",
" model=\"gpt-5.4-mini\",\n",
" messages=[{\"role\": \"user\", \"content\": \"Snape from Harry Potter\"}],\n",
" response_model=Character,\n",
")\n",
"resp.model_dump()"
]
},
{
"cell_type": "markdown",
"id": "bbc1d900-617a-4e4d-a401-6d10a5153cda",
"metadata": {},
"source": [
"## Defining Multiple Entities\n",
"\n",
"Now that we see a single entity with many properties we can continue to nest them into many users\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "1f2a2b14-a956-4f96-90c9-e11ca04ab7d1",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"age=11 name='Harry Potter' house='Gryffindor'\n",
"age=11 name='Hermione Granger' house='Gryffindor'\n",
"age=11 name='Ron Weasley' house='Gryffindor'\n",
"age=11 name='Draco Malfoy' house='Slytherin'\n",
"age=11 name='Neville Longbottom' house='Gryffindor'\n"
]
}
],
"source": [
"from collections.abc import Iterable\n",
"\n",
"\n",
"class Character(BaseModel):\n",
" age: int\n",
" name: str\n",
" house: Literal[\"Gryffindor\", \"Hufflepuff\", \"Ravenclaw\", \"Slytherin\"]\n",
"\n",
"\n",
"resp = client.create(\n",
" model=\"gpt-5.4-mini\",\n",
" messages=[{\"role\": \"user\", \"content\": \"Five characters from Harry Potter\"}],\n",
" response_model=Iterable[Character],\n",
")\n",
"\n",
"for character in resp:\n",
" print(character)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "a3091aba",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"age=11 name='Harry Potter' house='Gryffindor'\n",
"age=11 name='Hermione Granger' house='Gryffindor'\n",
"age=11 name='Ron Weasley' house='Gryffindor'\n",
"age=17 name='Draco Malfoy' house='Slytherin'\n",
"age=11 name='Luna Lovegood' house='Ravenclaw'\n"
]
}
],
"source": [
"from collections.abc import Iterable\n",
"\n",
"\n",
"class Character(BaseModel):\n",
" age: int\n",
" name: str\n",
" house: Literal[\"Gryffindor\", \"Hufflepuff\", \"Ravenclaw\", \"Slytherin\"]\n",
"\n",
"\n",
"resp = client.create(\n",
" model=\"gpt-5.4-mini\",\n",
" messages=[{\"role\": \"user\", \"content\": \"Five characters from Harry Potter\"}],\n",
" stream=True,\n",
" response_model=Iterable[Character],\n",
")\n",
"\n",
"for character in resp:\n",
" print(character)"
]
},
{
"cell_type": "markdown",
"id": "f6ed3144-bde1-4033-9c94-a6926fa079d2",
"metadata": {},
"source": [
"## Defining Relationships\n",
"\n",
"Not only can we define lists of users, but with lists of properties we can also easily define lists of references. It's one of the more interesting things I've learned about prompting.\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "6de8768e-b36a-4a51-9cf9-940d178552f6",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"id=1 name='Harry Potter' friends_array=[2, 3, 4, 5, 6]\n",
"id=2 name='Hermione Granger' friends_array=[1, 3, 4, 5]\n",
"id=3 name='Ron Weasley' friends_array=[1, 2, 4, 6]\n",
"id=4 name='Neville Longbottom' friends_array=[1, 2, 3, 5]\n",
"id=5 name='Luna Lovegood' friends_array=[1, 2, 4, 6]\n",
"id=6 name='Draco Malfoy' friends_array=[1, 3, 5]\n"
]
}
],
"source": [
"class Character(BaseModel):\n",
" id: int\n",
" name: str\n",
" friends_array: list[int] = Field(\n",
" description=\"Relationships to their friends using the id\"\n",
" )\n",
"\n",
"\n",
"resp = client.create(\n",
" model=\"gpt-5.4-mini\",\n",
" messages=[{\"role\": \"user\", \"content\": \"5 kids from Harry Potter\"}],\n",
" stream=True,\n",
" response_model=Iterable[Character],\n",
")\n",
"\n",
"for character in resp:\n",
" print(character)"
]
},
{
"cell_type": "markdown",
"id": "523b5797-71a5-4a96-a4b7-21280fb73015",
"metadata": {},
"source": [
"With the tools we've discussed, we can find numerous real-world applications in production settings. These include extracting action items from transcripts, generating fake data, filling out forms, and creating objects that correspond to generative UI. These simple tricks will be highly useful.\n"
]
},
{
"cell_type": "markdown",
"id": "a9d20fd9-0cd0-4300-a8c1-d16388969e8e",
"metadata": {},
"source": [
"# Missing Data\n",
"\n",
"The Maybe pattern is a concept in functional programming used for error handling. Instead of raising exceptions or returning None, you can use a Maybe type to encapsulate both the result and potential errors.\n",
"\n",
"This pattern is particularly useful when making LLM calls, as providing language models with an escape hatch can effectively reduce hallucinations."
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "c04f44aa-dc4b-4499-a151-e812512e77e6",
"metadata": {},
"outputs": [],
"source": [
"from typing import Optional\n",
"\n",
"\n",
"class Character(BaseModel):\n",
" age: int\n",
" name: str\n",
"\n",
"\n",
"class MaybeCharacter(BaseModel):\n",
" result: Optional[Character] = Field(default=None)\n",
" error: bool = Field(default=False)\n",
" message: Optional[str]"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "a2155190-e104-4ed6-a17f-e0732499dd51",
"metadata": {},
"outputs": [],
"source": [
"def extract(content: str) -> MaybeCharacter:\n",
" return client.create(\n",
" model=\"gpt-5.4-mini\",\n",
" response_model=MaybeCharacter,\n",
" messages=[\n",
" {\"role\": \"user\", \"content\": f\"Extract `{content}`\"},\n",
" ],\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "a7b59afa-9bf0-4dc0-a5ca-de584514f33b",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"MaybeCharacter(result=Character(age=17, name='Harry Potter'), error=False, message=None)"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"extract(\"Harry Potter\")"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "b5ddd5c1-ca75-49a9-95ad-181170435291",
"metadata": {},
"outputs": [
{
"ename": "ValueError",
"evalue": "404 Error",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m/Users/jasonliu/dev/instructor/docs/tutorials/2-tips.ipynb Cell 20\u001b[0m line \u001b[0;36m4\n\u001b[1;32m <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/docs/tutorials/2-tips.ipynb#X25sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m user \u001b[39m=\u001b[39m extract(\u001b[39m\"\u001b[39m\u001b[39m404 Error\u001b[39m\u001b[39m\"\u001b[39m)\n\u001b[1;32m <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/docs/tutorials/2-tips.ipynb#X25sZmlsZQ%3D%3D?line=2'>3</a>\u001b[0m \u001b[39mif\u001b[39;00m user\u001b[39m.\u001b[39merror:\n\u001b[0;32m----> <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/docs/tutorials/2-tips.ipynb#X25sZmlsZQ%3D%3D?line=3'>4</a>\u001b[0m \u001b[39mraise\u001b[39;00m \u001b[39mValueError\u001b[39;00m(user\u001b[39m.\u001b[39mmessage)\n",
"\u001b[0;31mValueError\u001b[0m: 404 Error"
]
}
],
"source": [
"user = extract(\"404 Error\")\n",
"\n",
"if user.error:\n",
" raise ValueError(user.message)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
docs/tutorials/3-0-applications-rag.ipynb
+864
View File
@@ -0,0 +1,864 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Applying Structured Output to RAG applications\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**What is RAG?**\n",
"\n",
"Retrieval Augmented Generation (RAG) models are the bridge between large language models and external knowledge databases. They fetch the relevant data for a given query. For example, if you have some documents and want to ask questions related to the content of those documents, RAG models help by retrieving data from those documents and passing it to the LLM in queries.\n",
"\n",
"**How do RAG models work?**\n",
"\n",
"The typical RAG process involves embedding a user query and searching a vector database to find the most relevant information to supplement the generated response. This approach is particularly effective when the database contains information closely matching the query but not more than that.\n",
"\n",
"![Image](https://python.useinstructor.com/blog/img/dumb_rag.png)\n",
"\n",
"**Why is there a need for them?**\n",
"\n",
"Pre-trained large language models do not learn over time. If you ask them a question they have not been trained on, they will often hallucinate. Therefore, we need to embed our own data to achieve a better output.\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Simple RAG\n",
"\n",
"**What is it?**\n",
"\n",
"The simplest implementation of RAG embeds a user query and do a single embedding search in a vector database, like a vector store of Wikipedia articles. However, this approach often falls short when dealing with complex queries and diverse data sources.\n",
"\n",
"- **Query-Document Mismatch:** It assumes that the query and document embeddings will align in the vector space, which is often not the case.\n",
"- **Text Search Limitations:** The model is restricted to simple text queries without the nuances of advanced search features.\n",
"- **Limited Planning Ability:** It fails to consider additional contextual information that could refine the search results.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Improving the RAG model\n",
"\n",
"**What's the solution?**\n",
"\n",
"Enhancing RAG requires a more sophisticated approach known as query understanding.\n",
"\n",
"This process involves analyzing the user's query and transforming it to better match the backend's search capabilities.\n",
"\n",
"By doing so, we can significantly improve both the precision and recall of the search results, providing more accurate and relevant responses.\n",
"\n",
"![Image](https://python.useinstructor.com/blog/img/query_understanding.png)\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Practical Examples\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the examples below, we're going to use the [`instructor`](https://github.com/jxnl/instructor) library to simplify the interaction between the programmer and language models via the function-calling API.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import instructor\n",
"\n",
"from openai import OpenAI\n",
"from pydantic import BaseModel, Field\n",
"\n",
"client = instructor.from_provider(\"openai/gpt-4o\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Example 1) Improving Extractions\n",
"\n",
"One of the big limitations is that often times the query we embed and the text\n",
"we are searching for may not have a direct match, leading to suboptimal results.\n",
"A common method of using structured output is to extract information from a\n",
"document and use it to answer a question. Directly, we can be creative in how we\n",
"extract, summarize and generate potential questions in order for our embeddings\n",
"to do better.\n",
"\n",
"For example, instead of using just a text chunk we could try to:\n",
"\n",
"1. extract key words and themes\n",
"2. extract hypothetical questions\n",
"3. generate a summary of the text\n",
"\n",
"In the example below, we use the `instructor` library to extract the key words\n",
"and themes from a text chunk and use them to answer a question.\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"class Extraction(BaseModel):\n",
" topic: str\n",
" summary: str\n",
" hypothetical_questions: list[str] = Field(\n",
" default_factory=list,\n",
" description=\"Hypothetical questions that this document could answer\",\n",
" )\n",
" keywords: list[str] = Field(\n",
" default_factory=list, description=\"Keywords that this document is about\"\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'hypothetical_questions': ['What is the basic concept behind simple RAG?',\n",
" 'How does simple RAG work for information '\n",
" 'retrieval?'],\n",
" 'keywords': ['Simple RAG',\n",
" 'Retrieval-Augmented Generation',\n",
" 'user query',\n",
" 'embedding search',\n",
" 'vector database',\n",
" 'Wikipedia articles',\n",
" 'information retrieval'],\n",
" 'summary': 'The simplest implementation of Retrieval-Augmented Generation '\n",
" '(RAG) involves embedding a user query and conducting a single '\n",
" 'embedding search in a vector database, like a vector store of '\n",
" 'Wikipedia articles, to retrieve relevant information. This method '\n",
" 'may not be ideal for complex queries or varied data sources.',\n",
" 'topic': 'Simple RAG'}\n",
"{'hypothetical_questions': ['What are the drawbacks of using simple RAG '\n",
" 'systems?',\n",
" 'How does query-document mismatch affect the '\n",
" 'performance of RAG?',\n",
" 'Why is a monolithic search backend a limitation '\n",
" 'for RAG?'],\n",
" 'keywords': ['limitations',\n",
" 'query-document mismatch',\n",
" 'simple RAG',\n",
" 'monolithic search backend',\n",
" 'text search',\n",
" 'planning ability',\n",
" 'contextual information'],\n",
" 'summary': 'Key limitations of the simple RAG include query-document '\n",
" 'mismatch, reliance on a single search backend, constraints of '\n",
" 'text search capabilities, and limited planning ability to '\n",
" 'leverage contextual information. These issues can result in '\n",
" 'suboptimal search outcomes and retrieval of irrelevant or broad '\n",
" 'information.',\n",
" 'topic': 'Limitations of Simple RAG'}\n"
]
}
],
"source": [
"from pprint import pprint\n",
"from collections.abc import Iterable\n",
"\n",
"\n",
"text_chunk = \"\"\"\n",
"## Simple RAG\n",
"\n",
"**What is it?**\n",
"\n",
"The simplest implementation of RAG embeds a user query and do a single embedding search in a vector database, like a vector store of Wikipedia articles. However, this approach often falls short when dealing with complex queries and diverse data sources.\n",
"\n",
"**What are the limitations?**\n",
"\n",
"- **Query-Document Mismatch:** It assumes that the query and document embeddings will align in the vector space, which is often not the case.\n",
" - Query: \"Tell me about climate change effects on marine life.\"\n",
" - Issue: The model might retrieve documents related to general climate change or marine life, missing the specific intersection of both topics.\n",
"- **Monolithic Search Backend:** It relies on a single search method and backend, reducing flexibility and the ability to handle multiple data sources.\n",
" - Query: \"Latest research in quantum computing.\"\n",
" - Issue: The model might only search in a general science database, missing out on specialized quantum computing resources.\n",
"- **Text Search Limitations:** The model is restricted to simple text queries without the nuances of advanced search features.\n",
" - Query: \"what problems did we fix last week\"\n",
" - Issue: cannot be answered by a simple text search since documents that contain problem, last week are going to be present at every week.\n",
"- **Limited Planning Ability:** It fails to consider additional contextual information that could refine the search results.\n",
" - Query: \"Tips for first-time Europe travelers.\"\n",
" - Issue: The model might provide general travel advice, ignoring the specific context of first-time travelers or European destinations.\n",
"\"\"\"\n",
"\n",
"extractions = client.create(\n",
" model=\"gpt-5.4-mini\",\n",
" stream=True,\n",
" response_model=Iterable[Extraction],\n",
" messages=[\n",
" {\n",
" \"role\": \"system\",\n",
" \"content\": \"Your role is to extract chunks from the following and create a set of topics.\",\n",
" },\n",
" {\"role\": \"user\", \"content\": text_chunk},\n",
" ],\n",
")\n",
"\n",
"\n",
"for extraction in extractions:\n",
" pprint(extraction.model_dump())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now you can imagine if you were to embed the summaries, hypothetical questions,\n",
"and keywords in a vector database (i.e. in the metadata fields of a vector\n",
"database), you can then use a vector search to find the best matching document\n",
"for a given query. What you'll find is that the results are much better than if\n",
"you were to just embed the text chunk!\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Example 2) Understanding 'recent queries' to add temporal context\n",
"\n",
"One common application of using structured outputs for query understanding is to identify the intent of a user's query. In this example we're going to use a simple schema to separately process the query to add additional temporal context.\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"from datetime import date\n",
"\n",
"\n",
"class DateRange(BaseModel):\n",
" start: date\n",
" end: date\n",
"\n",
"\n",
"class Query(BaseModel):\n",
" rewritten_query: str\n",
" published_daterange: DateRange"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this example, `DateRange` and `Query` are Pydantic models that structure the user's query with a date range and a list of domains to search within.\n",
"\n",
"These models **restructure** the user's query by including a <u>rewritten query</u>, a <u>range of published dates</u>, and a <u>list of domains</u> to search in.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using the new restructured query, we can apply this pattern to our function calls to obtain results that are optimized for our backend.\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Query(rewritten_query='Recent developments in artificial intelligence', published_daterange=DateRange(start=datetime.date(2024, 1, 1), end=datetime.date(2024, 3, 31)))"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def expand_query(q) -> Query:\n",
" return client.create(\n",
" model=\"gpt-5.4-mini\",\n",
" response_model=Query,\n",
" messages=[\n",
" {\n",
" \"role\": \"system\",\n",
" \"content\": f\"You're a query understanding system for the Metafor Systems search engine. Today is {date.today()}. Here are some tips: ...\",\n",
" },\n",
" {\"role\": \"user\", \"content\": f\"query: {q}\"},\n",
" ],\n",
" )\n",
"\n",
"\n",
"query = expand_query(\"What are some recent developments in AI?\")\n",
"query"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This isn't just about adding some date ranges. We can even use some chain of thought prompting to generate tailored searches that are deeply integrated with our backend.\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Query(rewritten_query='latest advancements in artificial intelligence', published_daterange=DateRange(chain_of_thought='Since the user is asking for recent developments, it would be relevant to look for articles and papers published within the last year. Therefore, setting the start date to a year before today and the end date to today will cover the most recent advancements.', start=datetime.date(2023, 3, 31), end=datetime.date(2024, 3, 31)))"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"class DateRange(BaseModel):\n",
" chain_of_thought: str = Field(\n",
" description=\"Think step by step to plan what is the best time range to search in\"\n",
" )\n",
" start: date\n",
" end: date\n",
"\n",
"\n",
"class Query(BaseModel):\n",
" rewritten_query: str = Field(\n",
" description=\"Rewrite the query to make it more specific\"\n",
" )\n",
" published_daterange: DateRange = Field(\n",
" description=\"Effective date range to search in\"\n",
" )\n",
"\n",
"\n",
"def expand_query(q) -> Query:\n",
" return client.create(\n",
" model=\"gpt-5.4-mini\",\n",
" response_model=Query,\n",
" messages=[\n",
" {\n",
" \"role\": \"system\",\n",
" \"content\": f\"You're a query understanding system for the Metafor Systems search engine. Today is {date.today()}. Here are some tips: ...\",\n",
" },\n",
" {\"role\": \"user\", \"content\": f\"query: {q}\"},\n",
" ],\n",
" )\n",
"\n",
"\n",
"expand_query(\"What are some recent developments in AI?\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Using Weights and Biases to track experiments\n",
"\n",
"While running a function like this production is quite simple, a lot of time will be spend on iterating and improving the model. To do this, we can use Weights and Biases to track our experiments.\n",
"\n",
"In order to do so we wand manage a few things\n",
"\n",
"1. Save input and output pairs for later\n",
"2. Save the JSON schema for the response_model\n",
"3. Having snapshots of the model and data allow us to compare results over time, and as we make changes to the model we can see how the results change.\n",
"\n",
"This is particularly useful when we might want to blend a mix of synthetic and real data to evaluate our model. We can use the `wandb` library to track our experiments and save the results to a dashboard.\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"import json\n",
"import instructor\n",
"\n",
"from openai import AsyncOpenAI\n",
"from datetime import date\n",
"from pydantic import BaseModel, Field\n",
"\n",
"\n",
"class DateRange(BaseModel):\n",
" chain_of_thought: str = Field(\n",
" description=\"Think step by step to plan what is the best time range to search in\"\n",
" )\n",
" start: date\n",
" end: date\n",
"\n",
"\n",
"class Query(BaseModel):\n",
" rewritten_query: str = Field(\n",
" description=\"Rewrite the query to make it more specific\"\n",
" )\n",
" published_daterange: DateRange = Field(\n",
" description=\"Effective date range to search in\"\n",
" )\n",
"\n",
" def report(self):\n",
" dct = self.model_dump()\n",
" dct[\"usage\"] = self._raw_response.usage.model_dump()\n",
" return dct\n",
"\n",
"\n",
"# We'll use a different client for async calls\n",
"# To highlight the difference and how we can use both\n",
"aclient = instructor.patch(AsyncOpenAI())\n",
"\n",
"\n",
"async def expand_query(q, *, model: str = \"gpt-5.4-mini\", temp: float = 0) -> Query:\n",
" return await aclient.create(\n",
" model=model,\n",
" temperature=temp,\n",
" response_model=Query,\n",
" messages=[\n",
" {\n",
" \"role\": \"system\",\n",
" \"content\": f\"You're a query understanding system for the Metafor Systems search engine. Today is {date.today()}. Here are some tips: ...\",\n",
" },\n",
" {\"role\": \"user\", \"content\": f\"query: {q}\"},\n",
" ],\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"# % pip install pandas wandb\n",
"import pandas as pd\n",
"from typing import Any\n",
"\n",
"\n",
"def flatten_dict(\n",
" d: dict[str, Any], parent_key: str = \"\", sep: str = \"_\"\n",
") -> dict[str, Any]:\n",
" \"\"\"\n",
" Flatten a nested dictionary.\n",
"\n",
" :param d: The nested dictionary to flatten.\n",
" :param parent_key: The base key to use for the flattened keys.\n",
" :param sep: Separator to use between keys.\n",
" :return: A flattened dictionary.\n",
" \"\"\"\n",
" items = []\n",
" for k, v in d.items():\n",
" new_key = f\"{parent_key}{sep}{k}\" if parent_key else k\n",
" if isinstance(v, dict):\n",
" items.extend(flatten_dict(v, new_key, sep=sep).items())\n",
" else:\n",
" items.append((new_key, v))\n",
" return dict(items)\n",
"\n",
"\n",
"def dicts_to_df(list_of_dicts: list[dict[str, Any]]) -> pd.DataFrame:\n",
" \"\"\"\n",
" Convert a list of dictionaries to a pandas DataFrame.\n",
"\n",
" :param list_of_dicts: List of dictionaries, potentially nested.\n",
" :return: A pandas DataFrame representing the flattened data.\n",
" \"\"\"\n",
" # Flatten each dictionary and create a DataFrame\n",
" flattened_data = [flatten_dict(d) for d in list_of_dicts]\n",
" return pd.DataFrame(flattened_data)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"import asyncio\n",
"import time\n",
"import pandas as pd\n",
"import wandb\n",
"\n",
"model = \"gpt-5.4-mini\"\n",
"temp = 0\n",
"\n",
"run = wandb.init(\n",
" project=\"query\",\n",
" config={\"model\": model, \"temp\": temp},\n",
")\n",
"\n",
"test_queries = [\n",
" \"latest developments in artificial intelligence last 3 weeks\",\n",
" \"renewable energy trends past month\",\n",
" \"quantum computing advancements last 2 months\",\n",
" \"biotechnology updates last 10 days\",\n",
"]\n",
"start = time.perf_counter()\n",
"queries = await asyncio.gather(\n",
" *[expand_query(q, model=model, temp=temp) for q in test_queries]\n",
")\n",
"duration = time.perf_counter() - start\n",
"\n",
"with open(\"schema.json\", \"w+\") as f:\n",
" schema = Query.model_json_schema()\n",
" json.dump(schema, f, indent=2)\n",
"\n",
"with open(\"results.jsonlines\", \"w+\") as f:\n",
" for query in queries:\n",
" f.write(query.model_dump_json() + \"\\n\")\n",
"\n",
"df = dicts_to_df([q.report() for q in queries])\n",
"df[\"input\"] = test_queries\n",
"df.to_csv(\"results.csv\")\n",
"\n",
"\n",
"run.log({\"schema\": wandb.Table(dataframe=pd.DataFrame([{\"schema\": schema}]))})\n",
"run.log(\n",
" {\n",
" \"usage_total_tokens\": df[\"usage_total_tokens\"].sum(),\n",
" \"usage_completion_tokens\": df[\"usage_completion_tokens\"].sum(),\n",
" \"usage_prompt_tokens\": df[\"usage_prompt_tokens\"].sum(),\n",
" \"duration (s)\": duration,\n",
" \"average duration (s)\": duration / len(queries),\n",
" \"n_queries\": len(queries),\n",
" }\n",
")\n",
"\n",
"run.log(\n",
" {\n",
" \"results\": wandb.Table(dataframe=df),\n",
" }\n",
")\n",
"\n",
"files = wandb.Artifact(\"data\", type=\"dataset\")\n",
"files.add_file(\"schema.json\")\n",
"files.add_file(\"results.jsonlines\")\n",
"files.add_file(\"results.csv\")\n",
"\n",
"run.log_artifact(files)\n",
"run.finish()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The output of Weights and Biases would return something like the below table.\n",
"\n",
"| Metric | Value |\n",
"|--------------------------|--------|\n",
"| average duration (s) | 1.5945 |\n",
"| duration (s) | 6.37799|\n",
"| n_queries | 4 |\n",
"| usage_completion_tokens | 376 |\n",
"| usage_prompt_tokens | 780 |\n",
"| usage_total_tokens | 1156 |\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Example 3) Personal Assistants, parallel processing\n",
"\n",
"A personal assistant application needs to interpret vague queries and fetch information from multiple backends, such as emails and calendars. By modeling the assistant's capabilities using Pydantic, we can dispatch the query to the correct backend and retrieve a unified response.\n",
"\n",
"For instance, when you ask, \"What's on my schedule today?\", the application needs to fetch data from various sources like events, emails, and reminders. This data is stored across different backends, but the goal is to provide a consolidated summary of results.\n",
"\n",
"It's important to note that the data from these sources may not be embedded in a search backend. Instead, they could be accessed through different clients like a calendar or email, spanning both personal and professional accounts.\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"from typing import Literal\n",
"\n",
"\n",
"class SearchClient(BaseModel):\n",
" query: str = Field(description=\"The search query that will go into the search bar\")\n",
" keywords: list[str]\n",
" email: str\n",
" source: Literal[\"gmail\", \"calendar\"]\n",
" date_range: DateRange\n",
"\n",
"\n",
"class Retrieval(BaseModel):\n",
" queries: list[SearchClient]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, we can utilize this with a straightforward query such as \"What do I have today?\".\n",
"\n",
"The system will attempt to asynchronously dispatch the query to the appropriate backend.\n",
"\n",
"However, it's still crucial to remember that effectively prompting the language model is still a key aspect.\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"queries\": [\n",
" {\n",
" \"query\": \"work\",\n",
" \"keywords\": [\n",
" \"work\",\n",
" \"today\"\n",
" ],\n",
" \"email\": \"jason@work.com\",\n",
" \"source\": \"gmail\",\n",
" \"date_range\": {\n",
" \"chain_of_thought\": \"Check today's work schedule\",\n",
" \"start\": \"2024-03-31\",\n",
" \"end\": \"2024-03-31\"\n",
" }\n",
" },\n",
" {\n",
" \"query\": \"new emails\",\n",
" \"keywords\": [\n",
" \"email\",\n",
" \"new\"\n",
" ],\n",
" \"email\": \"jason@work.com\",\n",
" \"source\": \"gmail\",\n",
" \"date_range\": {\n",
" \"chain_of_thought\": \"Check for new emails today\",\n",
" \"start\": \"2024-03-31\",\n",
" \"end\": \"2024-03-31\"\n",
" }\n",
" }\n",
" ]\n",
"}\n"
]
}
],
"source": [
"retrieval = client.create(\n",
" model=\"gpt-5.4-mini\",\n",
" response_model=Retrieval,\n",
" messages=[\n",
" {\n",
" \"role\": \"system\",\n",
" \"content\": f\"\"\"You are Jason's personal assistant.\n",
" He has two emails jason@work.com jason@personal.com\n",
" Today is {date.today()}\"\"\",\n",
" },\n",
" {\"role\": \"user\", \"content\": \"What do I have today for work? any new emails?\"},\n",
" ],\n",
")\n",
"print(retrieval.model_dump_json(indent=4))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To make it more challenging, we will assign it multiple tasks, followed by a list of queries that are routed to various search backends, such as email and calendar. Not only do we dispatch to different backends, over which we have no control, but we are also likely to render them to the user in different ways.\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"queries\": [\n",
" {\n",
" \"query\": \"Jason's meetings\",\n",
" \"keywords\": [\n",
" \"meeting\",\n",
" \"appointment\",\n",
" \"schedule\",\n",
" \"calendar\"\n",
" ],\n",
" \"email\": \"jason@work.com\",\n",
" \"source\": \"calendar\",\n",
" \"date_range\": {\n",
" \"chain_of_thought\": \"Since today's date is 2024-03-31, we should look for meetings scheduled for this exact date.\",\n",
" \"start\": \"2024-03-31\",\n",
" \"end\": \"2024-03-31\"\n",
" }\n",
" }\n",
" ]\n",
"}\n"
]
}
],
"source": [
"retrieval = client.create(\n",
" model=\"gpt-5.4-mini\",\n",
" response_model=Retrieval,\n",
" messages=[\n",
" {\n",
" \"role\": \"system\",\n",
" \"content\": f\"\"\"You are Jason's personal assistant.\n",
" He has two emails jason@work.com jason@personal.com\n",
" Today is {date.today()}\"\"\",\n",
" },\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": \"What meetings do I have today and are there any important emails I should be aware of\",\n",
" },\n",
" ],\n",
")\n",
"print(retrieval.model_dump_json(indent=4))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Example 4) Decomposing questions\n",
"\n",
"Lastly, a lightly more complex example of a problem that can be solved with structured output is decomposing questions. Where you ultimately want to decompose a question into a series of sub-questions that can be answered by a search backend. For example\n",
"\n",
"\"Whats the difference in populations of jason's home country and canada?\"\n",
"\n",
"You'd ultimately need to know a few things\n",
"\n",
"1. Jason's home country\n",
"2. The population of Jason's home country\n",
"3. The population of Canada\n",
"4. The difference between the two\n",
"\n",
"This would not be done correctly as a single query, nor would it be done in parallel, however there are some opportunities try to be parallel since not all of the sub-questions are dependent on each other.\n"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"root_question\": \"What is the difference between the population of Jason's home country and Canada?\",\n",
" \"plan\": [\n",
" {\n",
" \"id\": 1,\n",
" \"query\": \"What is the population of Jason's home country?\",\n",
" \"subquestions\": []\n",
" },\n",
" {\n",
" \"id\": 2,\n",
" \"query\": \"What is the population of Canada?\",\n",
" \"subquestions\": []\n",
" },\n",
" {\n",
" \"id\": 3,\n",
" \"query\": \"What is the difference between two population numbers?\",\n",
" \"subquestions\": [\n",
" 1,\n",
" 2\n",
" ]\n",
" }\n",
" ]\n",
"}\n"
]
}
],
"source": [
"class Question(BaseModel):\n",
" id: int = Field(..., description=\"A unique identifier for the question\")\n",
" query: str = Field(..., description=\"The question decomposed as much as possible\")\n",
" subquestions: list[int] = Field(\n",
" default_factory=list,\n",
" description=\"The subquestions that this question is composed of\",\n",
" )\n",
"\n",
"\n",
"class QueryPlan(BaseModel):\n",
" root_question: str = Field(..., description=\"The root question that the user asked\")\n",
" plan: list[Question] = Field(\n",
" ..., description=\"The plan to answer the root question and its subquestions\"\n",
" )\n",
"\n",
"\n",
"retrieval = client.create(\n",
" model=\"gpt-5.4-mini\",\n",
" response_model=QueryPlan,\n",
" messages=[\n",
" {\n",
" \"role\": \"system\",\n",
" \"content\": \"You are a query understanding system capable of decomposing a question into subquestions.\",\n",
" },\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": \"What is the difference between the population of jason's home country and canada?\",\n",
" },\n",
" ],\n",
")\n",
"\n",
"print(retrieval.model_dump_json(indent=4))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"I hope in this section I've exposed you to some ways we can be creative in modeling structured outputs to leverage LLMS in building some lightweight components for our systems.\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.8"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
docs/tutorials/3-1-validation-rag.ipynb
+634
View File
@@ -0,0 +1,634 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "5a01f3ac-5306-4a1b-9e47-a5d254bce93a",
"metadata": {},
"source": [
"# Understanding Validators\n"
]
},
{
"cell_type": "markdown",
"id": "9dcc78ac-ed6d-49e3-b71b-fb2fb25f16a8",
"metadata": {},
"source": [
"Pydantic offers an customizable and expressive validation framework for Python. Instructor leverages Pydantic's validation framework to provide a uniform developer experience for both code-based and LLM-based validation, as well as a reasking mechanism for correcting LLM outputs based on validation errors. To learn more check out the Pydantic [docs](https://docs.pydantic.dev/latest/) on validators.\n",
"\n",
"Then we'll bring it all together into the context of RAG from the previous notebook.\n"
]
},
{
"cell_type": "markdown",
"id": "064c286b",
"metadata": {},
"source": [
"Validators will enable us to control outputs by defining a function like so:\n",
"\n",
"```python\n",
"def validation_function(value):\n",
" if condition(value):\n",
" raise ValueError(\"Value is not valid\")\n",
" return mutation(value)\n",
"```\n",
"\n",
"Before we get started lets go over the general shape of a validator:\n"
]
},
{
"cell_type": "markdown",
"id": "7cfc6c66",
"metadata": {},
"source": [
"## Defining Validator Functions\n"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "d4bb6258-b03a-4621-8a73-29056a20ec0f",
"metadata": {},
"outputs": [],
"source": [
"from typing import Annotated\n",
"from pydantic import BaseModel, AfterValidator, WithJsonSchema\n",
"\n",
"\n",
"def name_must_contain_space(v: str) -> str:\n",
" if \" \" not in v:\n",
" raise ValueError(\"Name must contain a space.\")\n",
" return v\n",
"\n",
"\n",
"def uppercase_name(v: str) -> str:\n",
" return v.upper()\n",
"\n",
"\n",
"FullName = Annotated[\n",
" str,\n",
" AfterValidator(name_must_contain_space),\n",
" AfterValidator(uppercase_name),\n",
" WithJsonSchema(\n",
" {\n",
" \"type\": \"string\",\n",
" \"description\": \"The user's full name\",\n",
" }\n",
" ),\n",
"]\n",
"\n",
"\n",
"class UserDetail(BaseModel):\n",
" age: int\n",
" name: FullName"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "23f8cadd",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"UserDetail(age=30, name='JASON LIU')"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"UserDetail(age=30, name=\"Jason Liu\")"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "e4f53ecf",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'properties': {'age': {'title': 'Age', 'type': 'integer'},\n",
" 'name': {'description': \"The user's full name\",\n",
" 'title': 'Name',\n",
" 'type': 'string'}},\n",
" 'required': ['age', 'name'],\n",
" 'title': 'UserDetail',\n",
" 'type': 'object'}"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"UserDetail.model_json_schema()"
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "2284a7e8",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1 validation error for UserDetail\n",
"name\n",
" Value error, Name must contain a space. [type=value_error, input_value='Jason', input_type=str]\n",
" For further information visit https://errors.pydantic.dev/2.5/v/value_error\n"
]
}
],
"source": [
"try:\n",
" person = UserDetail.model_validate({\"age\": 24, \"name\": \"Jason\"})\n",
"except Exception as e:\n",
" print(e)"
]
},
{
"cell_type": "markdown",
"id": "3c0302ca",
"metadata": {},
"source": [
"## Using Field\n",
"\n",
"We can also use the `Field` class to define validators. This is useful when we want to define a validator for a field that is primitive, like a string or integer which supports a limited number of validators.\n"
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "3242856f",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2 validation errors for UserDetail\n",
"age\n",
" Input should be greater than 0 [type=greater_than, input_value=-10, input_type=int]\n",
" For further information visit https://errors.pydantic.dev/2.5/v/greater_than\n",
"name\n",
" Value error, Name must contain a space. [type=value_error, input_value='Jason', input_type=str]\n",
" For further information visit https://errors.pydantic.dev/2.5/v/value_error\n"
]
}
],
"source": [
"from pydantic import Field\n",
"\n",
"\n",
"Age = Annotated[int, Field(gt=0)]\n",
"\n",
"\n",
"class UserDetail(BaseModel):\n",
" age: Age\n",
" name: FullName\n",
"\n",
"\n",
"try:\n",
" person = UserDetail(age=-10, name=\"Jason\")\n",
"except Exception as e:\n",
" print(e)"
]
},
{
"cell_type": "markdown",
"id": "4f689121",
"metadata": {},
"source": [
"## Providing Context\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "ec043c23",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1 validation error for Response\n",
"message\n",
" Assertion failed, `hurt` was found in the message `I will hurt them.` [type=assertion_error, input_value='I will hurt them.', input_type=str]\n",
" For further information visit https://errors.pydantic.dev/2.5/v/assertion_error\n"
]
}
],
"source": [
"from pydantic import ValidationInfo\n",
"\n",
"\n",
"def message_cannot_have_blacklisted_words(v: str, info: ValidationInfo) -> str:\n",
" blacklist = info.context.get(\"blacklist\", [])\n",
" for word in blacklist:\n",
" assert word not in v.lower(), f\"`{word}` was found in the message `{v}`\"\n",
" return v\n",
"\n",
"\n",
"ModeratedStr = Annotated[str, AfterValidator(message_cannot_have_blacklisted_words)]\n",
"\n",
"\n",
"class Response(BaseModel):\n",
" message: ModeratedStr\n",
"\n",
"\n",
"try:\n",
" Response.model_validate(\n",
" {\"message\": \"I will hurt them.\"},\n",
" context={\n",
" \"blacklist\": {\n",
" \"rob\",\n",
" \"steal\",\n",
" \"hurt\",\n",
" \"kill\",\n",
" \"attack\",\n",
" }\n",
" },\n",
" )\n",
"except Exception as e:\n",
" print(e)"
]
},
{
"cell_type": "markdown",
"id": "37e3a638-c9c9-44cd-bcd0-ad1a39f448db",
"metadata": {},
"source": [
"## Using OpenAI Moderation\n"
]
},
{
"cell_type": "markdown",
"id": "88d0b816-7ec8-42b0-9b91-c9aab382c960",
"metadata": {},
"source": [
"To enhance our validation measures, we'll extend the scope to flag any answer that contains hateful content, harassment, or similar issues. OpenAI offers a moderation endpoint that addresses these concerns, and it's freely available when using OpenAI models.\n"
]
},
{
"cell_type": "markdown",
"id": "65f46eb5",
"metadata": {},
"source": [
"With the `instructor` library, this is just one function edit away:\n"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "82521112-5301-4442-acce-82b495bd838f",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1 validation error for Response\n",
"message\n",
" Value error, `I want to make them suffer the consequences` was flagged for harassment, harassment_threatening, violence, harassment/threatening [type=value_error, input_value='I want to make them suffer the consequences', input_type=str]\n",
" For further information visit https://errors.pydantic.dev/2.5/v/value_error\n"
]
}
],
"source": [
"from typing import Annotated\n",
"from pydantic import AfterValidator\n",
"from instructor import openai_moderation\n",
"\n",
"import instructor\n",
"from openai import OpenAI\n",
"\n",
"client = instructor.from_provider(\"openai/gpt-4o\")\n",
"\n",
"# This uses Annotated which is a new feature in Python 3.9\n",
"# To define custom metadata for a type hint.\n",
"ModeratedStr = Annotated[str, AfterValidator(openai_moderation(client=client))]\n",
"\n",
"\n",
"class Response(BaseModel):\n",
" message: ModeratedStr\n",
"\n",
"\n",
"try:\n",
" Response(message=\"I want to make them suffer the consequences\")\n",
"except Exception as e:\n",
" print(e)"
]
},
{
"cell_type": "markdown",
"id": "faa5116e",
"metadata": {},
"source": [
"## General Validator\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "49d8b772",
"metadata": {},
"outputs": [],
"source": [
"from instructor import llm_validator\n",
"\n",
"HealthTopicStr = Annotated[\n",
" str,\n",
" AfterValidator(\n",
" llm_validator(\n",
" \"don't talk about any other topic except health best practices and topics\",\n",
" client=client,\n",
" )\n",
" ),\n",
"]\n",
"\n",
"\n",
"class AssistantMessage(BaseModel):\n",
" message: HealthTopicStr\n",
"\n",
"\n",
"AssistantMessage(\n",
" message=\"I would suggest you to visit Sicily as they say it is very nice in winter.\"\n",
")"
]
},
{
"cell_type": "markdown",
"id": "050e72fe-4b13-4002-a1d0-94f7b88b784b",
"metadata": {},
"source": [
"### Avoiding hallucination with citations\n"
]
},
{
"cell_type": "markdown",
"id": "e3f2869e-c8a3-4b93-82e7-55eb70930900",
"metadata": {},
"source": [
"When incorporating external knowledge bases, it's crucial to ensure that the agent uses the provided context accurately and doesn't fabricate responses. Validators can be effectively used for this purpose. We can illustrate this with an example where we validate that a provided citation is actually included in the referenced text chunk:\n"
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "638fc368-5cf7-4ae7-9d3f-efea1b84eec0",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1 validation error for AnswerWithCitation\n",
"citation\n",
" Value error, Citation `Blueberries contain high levels of protein` not found in text, only use citations from the text. [type=value_error, input_value='Blueberries contain high levels of protein', input_type=str]\n",
" For further information visit https://errors.pydantic.dev/2.5/v/value_error\n"
]
}
],
"source": [
"from pydantic import ValidationInfo\n",
"\n",
"\n",
"def citation_exists(v: str, info: ValidationInfo):\n",
" context = info.context\n",
" if context:\n",
" context = context.get(\"text_chunk\")\n",
" if v not in context:\n",
" raise ValueError(\n",
" f\"Citation `{v}` not found in text, only use citations from the text.\"\n",
" )\n",
" return v\n",
"\n",
"\n",
"Citation = Annotated[str, AfterValidator(citation_exists)]\n",
"\n",
"\n",
"class AnswerWithCitation(BaseModel):\n",
" answer: str\n",
" citation: Citation\n",
"\n",
"\n",
"try:\n",
" AnswerWithCitation.model_validate(\n",
" {\n",
" \"answer\": \"Blueberries are packed with protein\",\n",
" \"citation\": \"Blueberries contain high levels of protein\",\n",
" },\n",
" context={\"text_chunk\": \"Blueberries are very rich in antioxidants\"},\n",
" )\n",
"except Exception as e:\n",
" print(e)"
]
},
{
"cell_type": "markdown",
"id": "3064b06b-7f85-40ec-8fe2-4fa2cce36585",
"metadata": {},
"source": [
"Here we assume that there is a \"text_chunk\" field that contains the text that the model is supposed to use as context. We then use the `field_validator` decorator to define a validator that checks if the citation is included in the text chunk. If it's not, we raise a `ValueError` with a message that will be returned to the user.\n",
"\n",
"\n",
"If we want to pass in the context through the `chat.completions.create`` endpoint, we can use the `validation_context` parameter\n",
"\n",
"```python\n",
"resp = client.create(\n",
" model=\"gpt-5.4-mini\",\n",
" response_model=AnswerWithCitation,\n",
" messages=[\n",
" {\"role\": \"user\", \"content\": f\"Answer the question `{q}` using the text chunk\\n`{text_chunk}`\"},\n",
" ],\n",
" validation_context={\"text_chunk\": text_chunk},\n",
")\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "64d15ad2",
"metadata": {},
"source": [
"In practice there are many ways to implement this: we could use a regex to check if the citation is included in the text chunk, or we could use a more sophisticated approach like a semantic similarity check. The important thing is that we have a way to validate that the model is using the provided context accurately.\n"
]
},
{
"cell_type": "markdown",
"id": "5bbbaa11-32d2-4772-bc31-18d1d6d6c919",
"metadata": {},
"source": [
"## Reasking with validators\n",
"\n",
"For most of these examples all we've done we've mostly only defined the validation logic. Which can be separate from generation, however when we are given validation errors, we shouldn't end there! Instead instructor allows us to collect all the validation errors and reask the llm to rewrite their answer.\n",
"\n",
"Lets try to use a extreme example to illustrate this point:\n"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "97f544e7-2552-465c-89a9-a4820f00d658",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"question\": \"What is the meaning of life?\",\n",
" \"answer\": \"According to the devil, the meaning of life is a life of sin and debauchery.\"\n",
"}\n"
]
}
],
"source": [
"class QuestionAnswer(BaseModel):\n",
" question: str\n",
" answer: str\n",
"\n",
"\n",
"question = \"What is the meaning of life?\"\n",
"context = (\n",
" \"The according to the devil the meaning of life is a life of sin and debauchery.\"\n",
")\n",
"\n",
"\n",
"resp = client.create(\n",
" model=\"gpt-5.4-mini\",\n",
" response_model=QuestionAnswer,\n",
" messages=[\n",
" {\n",
" \"role\": \"system\",\n",
" \"content\": \"You are a system that answers questions based on the context. answer exactly what the question asks using the context.\",\n",
" },\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": f\"using the context: `{context}`\\n\\nAnswer the following question: `{question}`\",\n",
" },\n",
" ],\n",
")\n",
"\n",
"print(resp.model_dump_json(indent=2))"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "0328bbc5",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Retrying, exception: 1 validation error for QuestionAnswer\n",
"answer\n",
" Assertion failed, The statement promotes sin and debauchery, which can be considered objectionable. [type=assertion_error, input_value='The meaning of life, acc... of sin and debauchery.', input_type=str]\n",
" For further information visit https://errors.pydantic.dev/2.5/v/assertion_error\n",
"Traceback (most recent call last):\n",
" File \"/Users/jasonliu/dev/instructor/instructor/patch.py\", line 277, in retry_sync\n",
" return process_response(\n",
" ^^^^^^^^^^^^^^^^^\n",
" File \"/Users/jasonliu/dev/instructor/instructor/patch.py\", line 164, in process_response\n",
" model = response_model.from_response(\n",
" ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
" File \"/Users/jasonliu/dev/instructor/instructor/function_calls.py\", line 137, in from_response\n",
" return cls.model_validate_json(\n",
" ^^^^^^^^^^^^^^^^^^^^^^^^\n",
" File \"/Users/jasonliu/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py\", line 532, in model_validate_json\n",
" return cls.__pydantic_validator__.validate_json(json_data, strict=strict, context=context)\n",
" ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
"pydantic_core._pydantic_core.ValidationError: 1 validation error for QuestionAnswer\n",
"answer\n",
" Assertion failed, The statement promotes sin and debauchery, which can be considered objectionable. [type=assertion_error, input_value='The meaning of life, acc... of sin and debauchery.', input_type=str]\n",
" For further information visit https://errors.pydantic.dev/2.5/v/assertion_error\n"
]
}
],
"source": [
"from instructor import llm_validator\n",
"\n",
"\n",
"NotEvilAnswer = Annotated[\n",
" str,\n",
" AfterValidator(llm_validator(\"don't say objectionable things\", client=client)),\n",
"]\n",
"\n",
"\n",
"class QuestionAnswer(BaseModel):\n",
" question: str\n",
" answer: NotEvilAnswer\n",
"\n",
"\n",
"resp = client.create(\n",
" model=\"gpt-5.4-mini\",\n",
" response_model=QuestionAnswer,\n",
" max_retries=2,\n",
" messages=[\n",
" {\n",
" \"role\": \"system\",\n",
" \"content\": \"You are a system that answers questions based on the context. answer exactly what the question asks using the context.\",\n",
" },\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": f\"using the context: `{context}`\\n\\nAnswer the following question: `{question}`\",\n",
" },\n",
" ],\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "814d3554",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"question\": \"What is the meaning of life?\",\n",
" \"answer\": \"The meaning of life is subjective and can vary depending on one's beliefs and perspectives. According to the devil, it is a life of sin and debauchery. However, this viewpoint may not be universally accepted and should be evaluated critically.\"\n",
"}\n"
]
}
],
"source": [
"print(resp.model_dump_json(indent=2))"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
docs/tutorials/4-validation.ipynb
+863
View File
@@ -0,0 +1,863 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Validators"
],
"id": "5a01f3ac-5306-4a1b-9e47-a5d254bce93a"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Instead of framing \"self-critique\" or \"self-reflection\" in AI as new concepts, we can view them as validation errors with clear error messages that the system can use to self correct.\n",
"\n",
"Pydantic offers an customizable and expressive validation framework for Python. Instructor leverages Pydantic's validation framework to provide a uniform developer experience for both code-based and LLM-based validation, as well as a reasking mechanism for correcting LLM outputs based on validation errors. To learn more check out the Pydantic [docs](https://docs.pydantic.dev/latest/) on validators.\n",
"\n",
"Note: For the majority of this notebook we won't be calling openai, just using validators to see how we can control the validation of the objects."
],
"id": "9dcc78ac-ed6d-49e3-b71b-fb2fb25f16a8"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Validators will enable us to control outputs by defining a function like so:\n",
"\n",
"\n",
"```python\n",
"def validation_function(value):\n",
" if condition(value):\n",
" raise ValueError(\"Value is not valid\")\n",
" return mutation(value)\n",
"```\n",
"\n",
"Before we get started lets go over the general shape of a validator:"
],
"id": "064c286b"
},
{
"cell_type": "code",
"metadata": {},
"source": [
"from pydantic import BaseModel\n",
"from typing import Annotated\n",
"from pydantic import AfterValidator\n",
"\n",
"\n",
"def name_must_contain_space(v: str) -> str:\n",
" if \" \" not in v:\n",
" raise ValueError(\"Name must contain a space.\")\n",
" return v.lower()\n",
"\n",
"\n",
"class UserDetail(BaseModel):\n",
" age: int\n",
" name: Annotated[str, AfterValidator(name_must_contain_space)]\n",
"\n",
"\n",
"person = UserDetail(age=29, name=\"Jason\")"
],
"execution_count": 61,
"outputs": [
{
"ename": "ValidationError",
"evalue": "1 validation error for UserDetail\nname\n Value error, Name must contain a space. [type=value_error, input_value='Jason', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/value_error",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mValidationError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb Cell 4\u001b[0m line \u001b[0;36m1\n\u001b[1;32m <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#W3sZmlsZQ%3D%3D?line=10'>11</a>\u001b[0m age: \u001b[39mint\u001b[39m\n\u001b[1;32m <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#W3sZmlsZQ%3D%3D?line=11'>12</a>\u001b[0m name: Annotated[\u001b[39mstr\u001b[39m, AfterValidator(name_must_contain_space)]\n\u001b[0;32m---> <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#W3sZmlsZQ%3D%3D?line=13'>14</a>\u001b[0m person \u001b[39m=\u001b[39m UserDetail(age\u001b[39m=\u001b[39;49m\u001b[39m29\u001b[39;49m, name\u001b[39m=\u001b[39;49m\u001b[39m\"\u001b[39;49m\u001b[39mJason\u001b[39;49m\u001b[39m\"\u001b[39;49m)\n",
"File \u001b[0;32m~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:164\u001b[0m, in \u001b[0;36mBaseModel.__init__\u001b[0;34m(__pydantic_self__, **data)\u001b[0m\n\u001b[1;32m 162\u001b[0m \u001b[39m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[1;32m 163\u001b[0m __tracebackhide__ \u001b[39m=\u001b[39m \u001b[39mTrue\u001b[39;00m\n\u001b[0;32m--> 164\u001b[0m __pydantic_self__\u001b[39m.\u001b[39;49m__pydantic_validator__\u001b[39m.\u001b[39;49mvalidate_python(data, self_instance\u001b[39m=\u001b[39;49m__pydantic_self__)\n",
"\u001b[0;31mValidationError\u001b[0m: 1 validation error for UserDetail\nname\n Value error, Name must contain a space. [type=value_error, input_value='Jason', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/value_error"
]
}
],
"id": "d4bb6258-b03a-4621-8a73-29056a20ec0f"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Validation Applications**\n",
"\n",
"Validators are essential in tackling the unpredictabile nature of LLMs.\n",
"\n",
"Straightforward examples include:\n",
"\n",
"* Flagging outputs containing blacklisted words.\n",
"* Identifying outputs with tones like racism or violence.\n",
"\n",
"For more complex tasks:\n",
"\n",
"* Ensuring citations directly come from provided content.\n",
"* Checking that the model's responses align with given context.\n",
"* Validating the syntax of SQL queries before execution."
],
"id": "417fafe5-4616-4372-b9e9-78e89afff536"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Setup and Dependencies"
],
"id": "1bd2104b-7eed-4619-a47d-c3d197f9d483"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using the [instructor](https://github.com/jxnl/instructor) library, we streamline the integration of these validators. `instructor` manages the parsing and validation of outputs and automates retries for compliant responses. This simplifies the process for developers to implement new validation logic, minimizing extra overhead."
],
"id": "e94449ab-50a9-4325-972c-f64fcdadee00"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To use instructor in our api calls, we just need to patch the openai client:"
],
"id": "a7a84adc"
},
{
"cell_type": "code",
"metadata": {},
"source": [
"import instructor\n",
"from openai import OpenAI\n",
"\n",
"client = instructor.from_provider(\"openai/gpt-4o\")"
],
"execution_count": 5,
"outputs": [],
"id": "1aa2c503-82f8-4735-aae3-373b55fb1064"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Software 2.0: Rule-based validators"
],
"id": "45cd244f-d59c-4431-be2d-aa356a6fefa0"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Deterministic validation, characterized by its rule-based logic, ensures consistent outcomes for the same input. Let's explore how we can apply this concept through some examples."
],
"id": "3494e664-c5b3-42ea-9c19-aa301a041bdb"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Flagging bad keywords"
],
"id": "717ecefd-0355-4ba4-a642-95d281b0f075"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To begin with, we aim to prevent engagement in topics involving explicit violence."
],
"id": "3a15013e-42f3-4d3b-b395-d6edbdec34e5"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We will define a blacklist of violent words that cannot be mentioned in any messages:"
],
"id": "13d61a81"
},
{
"cell_type": "code",
"metadata": {},
"source": [
"blacklist = {\n",
" \"rob\",\n",
" \"steal\",\n",
" \"hurt\",\n",
" \"kill\",\n",
" \"attack\",\n",
"}"
],
"execution_count": 63,
"outputs": [],
"id": "59330d7d-082a-4240-98c4-eaee18f02728"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To validate if the message contains a blacklisted word we will use a [field_validator](https://python.useinstructor.com/blog/2023/10/23/good-llm-validation-is-just-good-validation/#using-field_validator-decorator) over the 'message' field:"
],
"id": "7ce06bbf"
},
{
"cell_type": "code",
"metadata": {},
"source": [
"from pydantic import BaseModel, field_validator\n",
"from pydantic.fields import Field\n",
"\n",
"\n",
"class Response(BaseModel):\n",
" message: str\n",
"\n",
" @field_validator(\"message\")\n",
" def message_cannot_have_blacklisted_words(cls, v: str) -> str:\n",
" for word in v.split():\n",
" if word.lower() in blacklist:\n",
" raise ValueError(f\"`{word}` was found in the message `{v}`\")\n",
" return v\n",
"\n",
"\n",
"Response(message=\"I will hurt him\")"
],
"execution_count": 64,
"outputs": [
{
"ename": "ValidationError",
"evalue": "1 validation error for Response\nmessage\n Value error, `hurt` was found in the message `I will hurt him` [type=value_error, input_value='I will hurt him', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/value_error",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mValidationError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb Cell 17\u001b[0m line \u001b[0;36m1\n\u001b[1;32m <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X23sZmlsZQ%3D%3D?line=10'>11</a>\u001b[0m \u001b[39mraise\u001b[39;00m \u001b[39mValueError\u001b[39;00m(\u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39m`\u001b[39m\u001b[39m{\u001b[39;00mword\u001b[39m}\u001b[39;00m\u001b[39m` was found in the message `\u001b[39m\u001b[39m{\u001b[39;00mv\u001b[39m}\u001b[39;00m\u001b[39m`\u001b[39m\u001b[39m\"\u001b[39m)\n\u001b[1;32m <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X23sZmlsZQ%3D%3D?line=11'>12</a>\u001b[0m \u001b[39mreturn\u001b[39;00m v\n\u001b[0;32m---> <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X23sZmlsZQ%3D%3D?line=13'>14</a>\u001b[0m Response(message\u001b[39m=\u001b[39;49m\u001b[39m\"\u001b[39;49m\u001b[39mI will hurt him\u001b[39;49m\u001b[39m\"\u001b[39;49m)\n",
"File \u001b[0;32m~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:164\u001b[0m, in \u001b[0;36mBaseModel.__init__\u001b[0;34m(__pydantic_self__, **data)\u001b[0m\n\u001b[1;32m 162\u001b[0m \u001b[39m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[1;32m 163\u001b[0m __tracebackhide__ \u001b[39m=\u001b[39m \u001b[39mTrue\u001b[39;00m\n\u001b[0;32m--> 164\u001b[0m __pydantic_self__\u001b[39m.\u001b[39;49m__pydantic_validator__\u001b[39m.\u001b[39;49mvalidate_python(data, self_instance\u001b[39m=\u001b[39;49m__pydantic_self__)\n",
"\u001b[0;31mValidationError\u001b[0m: 1 validation error for Response\nmessage\n Value error, `hurt` was found in the message `I will hurt him` [type=value_error, input_value='I will hurt him', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/value_error"
]
}
],
"id": "9bb87f47-db98-4f1d-80cb-ad5f39df8793"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Flagging using OpenAI Moderation"
],
"id": "37e3a638-c9c9-44cd-bcd0-ad1a39f448db"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To enhance our validation measures, we'll extend the scope to flag any answer that contains hateful content, harassment, or similar issues. OpenAI offers a moderation endpoint that addresses these concerns, and it's freely available when using OpenAI models."
],
"id": "88d0b816-7ec8-42b0-9b91-c9aab382c960"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"With the `instructor` library, this is just one function edit away:"
],
"id": "65f46eb5"
},
{
"cell_type": "code",
"metadata": {},
"source": [
"from typing import Annotated\n",
"from pydantic.functional_validators import AfterValidator"
],
"execution_count": 1,
"outputs": [],
"id": "b2ad8c19-6a94-4e4a-aa3e-dce149e8a479"
},
{
"cell_type": "code",
"metadata": {},
"source": [
"from instructor import openai_moderation\n",
"\n",
"\n",
"class Response(BaseModel):\n",
" message: Annotated[str, AfterValidator(openai_moderation(client=client))]"
],
"execution_count": 6,
"outputs": [],
"id": "82521112-5301-4442-acce-82b495bd838f"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we have a more comprehensive flagging for violence and we can outsource the moderation of our messages."
],
"id": "90542190-a4f2-4242-8261-2f0ace323022"
},
{
"cell_type": "code",
"metadata": {},
"source": [
"Response(message=\"I want to make them suffer the consequences\")"
],
"execution_count": 7,
"outputs": [
{
"ename": "ValidationError",
"evalue": "1 validation error for Response\nmessage\n Value error, `I want to make them suffer the consequences` was flagged for harassment, harassment_threatening, violence, harassment/threatening [type=value_error, input_value='I want to make them suffer the consequences', input_type=str]\n For further information visit https://errors.pydantic.dev/2.5/v/value_error",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mValidationError\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[7], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43mResponse\u001b[49m\u001b[43m(\u001b[49m\u001b[43mmessage\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mI want to make them suffer the consequences\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m)\u001b[49m\n",
"File \u001b[0;32m~/.virtualenvs/pampa-labs/lib/python3.10/site-packages/pydantic/main.py:164\u001b[0m, in \u001b[0;36mBaseModel.__init__\u001b[0;34m(__pydantic_self__, **data)\u001b[0m\n\u001b[1;32m 162\u001b[0m \u001b[38;5;66;03m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[1;32m 163\u001b[0m __tracebackhide__ \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mTrue\u001b[39;00m\n\u001b[0;32m--> 164\u001b[0m \u001b[43m__pydantic_self__\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m__pydantic_validator__\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mvalidate_python\u001b[49m\u001b[43m(\u001b[49m\u001b[43mdata\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mself_instance\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m__pydantic_self__\u001b[49m\u001b[43m)\u001b[49m\n",
"\u001b[0;31mValidationError\u001b[0m: 1 validation error for Response\nmessage\n Value error, `I want to make them suffer the consequences` was flagged for harassment, harassment_threatening, violence, harassment/threatening [type=value_error, input_value='I want to make them suffer the consequences', input_type=str]\n For further information visit https://errors.pydantic.dev/2.5/v/value_error"
]
}
],
"id": "54a9de1b-c6e7-4a5f-854c-506083a06a9d"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And as an extra, we get flagging for other topics like religion, race etc."
],
"id": "f138f9f8-495a-4a09-96a0-c71d01561855"
},
{
"cell_type": "code",
"metadata": {},
"source": [
"Response(message=\"I will mock their religion\")"
],
"execution_count": 26,
"outputs": [
{
"ename": "ValidationError",
"evalue": "1 validation error for Response\nmessage\n Value error, `I will mock their religion` was flagged for ['harassment'] [type=value_error, input_value='I will mock their religion', input_type=str]\n For further information visit https://errors.pydantic.dev/2.5/v/value_error",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mValidationError\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[26], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43mResponse\u001b[49m\u001b[43m(\u001b[49m\u001b[43mmessage\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mI will mock their religion\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m)\u001b[49m\n",
"File \u001b[0;32m~/.virtualenvs/pampa-labs/lib/python3.10/site-packages/pydantic/main.py:164\u001b[0m, in \u001b[0;36mBaseModel.__init__\u001b[0;34m(__pydantic_self__, **data)\u001b[0m\n\u001b[1;32m 162\u001b[0m \u001b[38;5;66;03m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[1;32m 163\u001b[0m __tracebackhide__ \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mTrue\u001b[39;00m\n\u001b[0;32m--> 164\u001b[0m \u001b[43m__pydantic_self__\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m__pydantic_validator__\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mvalidate_python\u001b[49m\u001b[43m(\u001b[49m\u001b[43mdata\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mself_instance\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m__pydantic_self__\u001b[49m\u001b[43m)\u001b[49m\n",
"\u001b[0;31mValidationError\u001b[0m: 1 validation error for Response\nmessage\n Value error, `I will mock their religion` was flagged for ['harassment'] [type=value_error, input_value='I will mock their religion', input_type=str]\n For further information visit https://errors.pydantic.dev/2.5/v/value_error"
]
}
],
"id": "feb77670-afd7-4947-89f8-a9446f6fb12c"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Filtering very long messages"
],
"id": "886f122b-22c9-440e-99cf-2e594b3df99b"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In addition to content-based flags, we can also set criteria based on other aspects of the input text. For instance, to maintain user engagement, we might want to prevent the assistant from returning excessively long texts. \n",
"\n",
"Here, noticed that `Field` has built-in validators for `min_length` and `max_length`. to learn more checkout [Field Constraints](https://docs.pydantic.dev/latest/concepts/fields)"
],
"id": "692b1164-4bd5-4943-b9ab-2edec00d4f7d"
},
{
"cell_type": "code",
"metadata": {},
"source": [
"class AssistantMessage(BaseModel):\n",
" message: str = Field(..., max_length=100)"
],
"execution_count": 68,
"outputs": [],
"id": "45ffdbd4-deae-4a46-9637-1b5339904f53"
},
{
"cell_type": "code",
"metadata": {},
"source": [
"AssistantMessage(\n",
" message=\"Certainly! Lorem ipsum is a placeholder text commonly used in the printing and typesetting industry. Here's a sample of Lorem ipsum text: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nullam euismod velit vel tellus tempor, non viverra eros iaculis. Sed vel nisl nec mauris bibendum tincidunt. Vestibulum sed libero euismod, eleifend tellus id, laoreet elit. Donec auctor arcu ac mi feugiat, vel lobortis justo efficitur. Fusce vel odio vitae justo varius dignissim. Integer sollicitudin mi a justo bibendum ultrices. Quisque id nisl a lectus venenatis luctus. Please note that Lorem ipsum text is a nonsensical Latin-like text used as a placeholder for content, and it has no specific meaning. It's often used in design and publishing to demonstrate the visual aspects of a document without focusing on the actual content.\"\n",
")"
],
"execution_count": 69,
"outputs": [
{
"ename": "ValidationError",
"evalue": "1 validation error for AssistantMessage\nmessage\n String should have at most 100 characters [type=string_too_long, input_value=\"Certainly! Lorem ipsum i... on the actual content.\", input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/string_too_long",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mValidationError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb Cell 29\u001b[0m line \u001b[0;36m1\n\u001b[0;32m----> <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X41sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m AssistantMessage(message\u001b[39m=\u001b[39;49m\u001b[39m\"\u001b[39;49m\u001b[39mCertainly! Lorem ipsum is a placeholder text commonly used in the printing and typesetting industry. Here\u001b[39;49m\u001b[39m'\u001b[39;49m\u001b[39ms a sample of Lorem ipsum text: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nullam euismod velit vel tellus tempor, non viverra eros iaculis. Sed vel nisl nec mauris bibendum tincidunt. Vestibulum sed libero euismod, eleifend tellus id, laoreet elit. Donec auctor arcu ac mi feugiat, vel lobortis justo efficitur. Fusce vel odio vitae justo varius dignissim. Integer sollicitudin mi a justo bibendum ultrices. Quisque id nisl a lectus venenatis luctus. Please note that Lorem ipsum text is a nonsensical Latin-like text used as a placeholder for content, and it has no specific meaning. It\u001b[39;49m\u001b[39m'\u001b[39;49m\u001b[39ms often used in design and publishing to demonstrate the visual aspects of a document without focusing on the actual content.\u001b[39;49m\u001b[39m\"\u001b[39;49m)\n",
"File \u001b[0;32m~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:164\u001b[0m, in \u001b[0;36mBaseModel.__init__\u001b[0;34m(__pydantic_self__, **data)\u001b[0m\n\u001b[1;32m 162\u001b[0m \u001b[39m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[1;32m 163\u001b[0m __tracebackhide__ \u001b[39m=\u001b[39m \u001b[39mTrue\u001b[39;00m\n\u001b[0;32m--> 164\u001b[0m __pydantic_self__\u001b[39m.\u001b[39;49m__pydantic_validator__\u001b[39m.\u001b[39;49mvalidate_python(data, self_instance\u001b[39m=\u001b[39;49m__pydantic_self__)\n",
"\u001b[0;31mValidationError\u001b[0m: 1 validation error for AssistantMessage\nmessage\n String should have at most 100 characters [type=string_too_long, input_value=\"Certainly! Lorem ipsum i... on the actual content.\", input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/string_too_long"
]
}
],
"id": "66430dc5-b78c-45e2-a53b-ddc392b20583"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Avoiding hallucination with citations"
],
"id": "050e72fe-4b13-4002-a1d0-94f7b88b784b"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"When incorporating external knowledge bases, it's crucial to ensure that the agent uses the provided context accurately and doesn't fabricate responses. Validators can be effectively used for this purpose. We can illustrate this with an example where we validate that a provided citation is actually included in the referenced text chunk:"
],
"id": "e3f2869e-c8a3-4b93-82e7-55eb70930900"
},
{
"cell_type": "code",
"metadata": {},
"source": [
"from pydantic import ValidationInfo\n",
"\n",
"\n",
"class AnswerWithCitation(BaseModel):\n",
" answer: str\n",
" citation: str\n",
"\n",
" @field_validator(\"citation\")\n",
" @classmethod\n",
" def citation_exists(cls, v: str, info: ValidationInfo):\n",
" context = info.context\n",
" if context:\n",
" context = context.get(\"text_chunk\")\n",
" if v not in context:\n",
" raise ValueError(f\"Citation `{v}` not found in text\")\n",
" return v"
],
"execution_count": 70,
"outputs": [],
"id": "638fc368-5cf7-4ae7-9d3f-efea1b84eec0"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here we assume that there is a \"text_chunk\" field that contains the text that the model is supposed to use as context. We then use the `field_validator` decorator to define a validator that checks if the citation is included in the text chunk. If it's not, we raise a `ValueError` with a message that will be returned to the user."
],
"id": "3064b06b-7f85-40ec-8fe2-4fa2cce36585"
},
{
"cell_type": "code",
"metadata": {},
"source": [
"AnswerWithCitation.model_validate(\n",
" {\n",
" \"answer\": \"Blueberries are packed with protein\",\n",
" \"citation\": \"Blueberries contain high levels of protein\",\n",
" },\n",
" context={\"text_chunk\": \"Blueberries are very rich in antioxidants\"},\n",
")"
],
"execution_count": 71,
"outputs": [
{
"ename": "ValidationError",
"evalue": "1 validation error for AnswerWithCitation\ncitation\n Value error, Citation `Blueberries contain high levels of protein` not found in text [type=value_error, input_value='Blueberries contain high levels of protein', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/value_error",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mValidationError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb Cell 34\u001b[0m line \u001b[0;36m1\n\u001b[0;32m----> <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X50sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m AnswerWithCitation\u001b[39m.\u001b[39;49mmodel_validate(\n\u001b[1;32m <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X50sZmlsZQ%3D%3D?line=1'>2</a>\u001b[0m {\n\u001b[1;32m <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X50sZmlsZQ%3D%3D?line=2'>3</a>\u001b[0m \u001b[39m\"\u001b[39;49m\u001b[39manswer\u001b[39;49m\u001b[39m\"\u001b[39;49m: \u001b[39m\"\u001b[39;49m\u001b[39mBlueberries are packed with protein\u001b[39;49m\u001b[39m\"\u001b[39;49m, \n\u001b[1;32m <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X50sZmlsZQ%3D%3D?line=3'>4</a>\u001b[0m \u001b[39m\"\u001b[39;49m\u001b[39mcitation\u001b[39;49m\u001b[39m\"\u001b[39;49m: \u001b[39m\"\u001b[39;49m\u001b[39mBlueberries contain high levels of protein\u001b[39;49m\u001b[39m\"\u001b[39;49m\n\u001b[1;32m <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X50sZmlsZQ%3D%3D?line=4'>5</a>\u001b[0m },\n\u001b[1;32m <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X50sZmlsZQ%3D%3D?line=5'>6</a>\u001b[0m context\u001b[39m=\u001b[39;49m{\u001b[39m\"\u001b[39;49m\u001b[39mtext_chunk\u001b[39;49m\u001b[39m\"\u001b[39;49m: \u001b[39m\"\u001b[39;49m\u001b[39mBlueberries are very rich in antioxidants\u001b[39;49m\u001b[39m\"\u001b[39;49m}, \n\u001b[1;32m <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X50sZmlsZQ%3D%3D?line=6'>7</a>\u001b[0m )\n",
"File \u001b[0;32m~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:503\u001b[0m, in \u001b[0;36mBaseModel.model_validate\u001b[0;34m(cls, obj, strict, from_attributes, context)\u001b[0m\n\u001b[1;32m 501\u001b[0m \u001b[39m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[1;32m 502\u001b[0m __tracebackhide__ \u001b[39m=\u001b[39m \u001b[39mTrue\u001b[39;00m\n\u001b[0;32m--> 503\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mcls\u001b[39;49m\u001b[39m.\u001b[39;49m__pydantic_validator__\u001b[39m.\u001b[39;49mvalidate_python(\n\u001b[1;32m 504\u001b[0m obj, strict\u001b[39m=\u001b[39;49mstrict, from_attributes\u001b[39m=\u001b[39;49mfrom_attributes, context\u001b[39m=\u001b[39;49mcontext\n\u001b[1;32m 505\u001b[0m )\n",
"\u001b[0;31mValidationError\u001b[0m: 1 validation error for AnswerWithCitation\ncitation\n Value error, Citation `Blueberries contain high levels of protein` not found in text [type=value_error, input_value='Blueberries contain high levels of protein', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/value_error"
]
}
],
"id": "0f3030b6-e6cf-45bf-a366-12de996fea40"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Software 3.0: Probabilistic validators"
],
"id": "06e54533-3304-4fa0-9828-9591d5dcdefd"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For scenarios requiring more nuanced validation than rule-based methods, we use probabilistic validation. This approach incorporates LLMs into the validation workflow for a sophisticated assessment of outputs.\n",
"\n",
"The `instructor` library offers the `llm_validator` utility for this purpose. By specifying the desired directive, we can use LLMs for complex validation tasks. Let's explore some intriguing use cases enabled by LLMs.\n",
"\n",
"### Keeping an agent on topic\n",
"\n",
"When creating an agent focused on health improvement, providing answers and daily practice suggestions, it's crucial to ensure strict adherence to health-related topics. This is important because the knowledge base is limited to health topics, and veering off-topic could result in fabricated responses.\n",
"\n",
"To achieve this focus, we'll follow a similar process as before, but with an important addition: integrating an LLM into our validator."
],
"id": "1907df5b-472f-45ac-9181-45235e3cd0c3"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This LLM will be tasked with determining whether the agent's responses are exclusively related to health topics. For this, we will use the `llm_validator` from `instructor` like so:"
],
"id": "546625ac"
},
{
"cell_type": "code",
"metadata": {},
"source": [
"from instructor import llm_validator\n",
"\n",
"\n",
"class AssistantMessage(BaseModel):\n",
" message: Annotated[\n",
" str,\n",
" AfterValidator(\n",
" llm_validator(\n",
" \"don't talk about any other topic except health best practices and topics\",\n",
" client=client,\n",
" )\n",
" ),\n",
" ]\n",
"\n",
"\n",
"AssistantMessage(\n",
" message=\"I would suggest you to visit Sicily as they say it is very nice in winter.\"\n",
")"
],
"execution_count": 73,
"outputs": [
{
"ename": "ValidationError",
"evalue": "1 validation error for AssistantMessage\nmessage\n Assertion failed, The statement is not related to health best practices or topics. [type=assertion_error, input_value='I would suggest you to v...is very nice in winter.', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/assertion_error",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mValidationError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb Cell 38\u001b[0m line \u001b[0;36m1\n\u001b[1;32m <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X56sZmlsZQ%3D%3D?line=4'>5</a>\u001b[0m \u001b[39mclass\u001b[39;00m \u001b[39mAssistantMessage\u001b[39;00m(BaseModel):\n\u001b[1;32m <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X56sZmlsZQ%3D%3D?line=5'>6</a>\u001b[0m message: Annotated[\u001b[39mstr\u001b[39m, \n\u001b[1;32m <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X56sZmlsZQ%3D%3D?line=6'>7</a>\u001b[0m AfterValidator(\n\u001b[1;32m <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X56sZmlsZQ%3D%3D?line=7'>8</a>\u001b[0m llm_validator(\u001b[39m\"\u001b[39m\u001b[39mdon\u001b[39m\u001b[39m'\u001b[39m\u001b[39mt talk about any other topic except health best practices and topics\u001b[39m\u001b[39m\"\u001b[39m, \n\u001b[1;32m <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X56sZmlsZQ%3D%3D?line=8'>9</a>\u001b[0m openai_client\u001b[39m=\u001b[39mclient))]\n\u001b[0;32m---> <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X56sZmlsZQ%3D%3D?line=10'>11</a>\u001b[0m AssistantMessage(message\u001b[39m=\u001b[39;49m\u001b[39m\"\u001b[39;49m\u001b[39mI would suggest you to visit Sicily as they say it is very nice in winter.\u001b[39;49m\u001b[39m\"\u001b[39;49m)\n",
"File \u001b[0;32m~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:164\u001b[0m, in \u001b[0;36mBaseModel.__init__\u001b[0;34m(__pydantic_self__, **data)\u001b[0m\n\u001b[1;32m 162\u001b[0m \u001b[39m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[1;32m 163\u001b[0m __tracebackhide__ \u001b[39m=\u001b[39m \u001b[39mTrue\u001b[39;00m\n\u001b[0;32m--> 164\u001b[0m __pydantic_self__\u001b[39m.\u001b[39;49m__pydantic_validator__\u001b[39m.\u001b[39;49mvalidate_python(data, self_instance\u001b[39m=\u001b[39;49m__pydantic_self__)\n",
"\u001b[0;31mValidationError\u001b[0m: 1 validation error for AssistantMessage\nmessage\n Assertion failed, The statement is not related to health best practices or topics. [type=assertion_error, input_value='I would suggest you to v...is very nice in winter.', input_type=str]\n For further information visit https://errors.pydantic.dev/2.4/v/assertion_error"
]
}
],
"id": "8cf00cad-c4c0-49dd-9be5-fb02338a5a7f"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Important that for these examples we're not waiting for the messages, to get this message we would need to call the openai with `response_model=AssistantMessage`."
],
"id": "1dce5a7a-024e-4742-a124-fe51973df5f2"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Validating agent thinking with CoT"
],
"id": "a6ec4afa-0be7-469e-93c0-5c729a06d4fc"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using probabilistic validation, we can also assess the agent's reasoning process to ensure it's logical before providing a response. With [chain of thought](https://learnprompting.org/docs/intermediate/chain_of_thought) prompting, the model is expected to think in steps and arrive at an answer following its logical progression. If there are errors in this logic, the final response may be incorrect.\n",
"\n",
"Here we will use Pydantic's [model_validator](https://docs.pydantic.dev/latest/concepts/validators/#model-validators) which allows us to apply validation over all the properties of the `AIResponse` at once.\n",
"\n",
"To make this easier we'll make a simple validation class that we can reuse for all our validation:"
],
"id": "424d915b-f332-48f3-a75e-6e1cd6d12075"
},
{
"cell_type": "code",
"metadata": {},
"source": [
"from typing import Optional\n",
"\n",
"\n",
"class Validation(BaseModel):\n",
" is_valid: bool = Field(\n",
" ..., description=\"Whether the value is valid based on the rules\"\n",
" )\n",
" error_message: Optional[str] = Field(\n",
" ...,\n",
" description=\"The error message if the value is not valid, to be used for re-asking the model\",\n",
" )"
],
"execution_count": 74,
"outputs": [],
"id": "65340b8c-2ea3-4457-a6d4-f0e652c317b4"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The function we will call will integrate an LLM and will ask it to determine whether the answer the model provided follows from the chain of thought: "
],
"id": "de2104f1"
},
{
"cell_type": "code",
"metadata": {},
"source": [
"def validate_chain_of_thought(values):\n",
" chain_of_thought = values[\"chain_of_thought\"]\n",
" answer = values[\"answer\"]\n",
" resp = client.create(\n",
" model=\"gpt-5.4-mini\",\n",
" messages=[\n",
" {\n",
" \"role\": \"system\",\n",
" \"content\": \"You are a validator. Determine if the value follows from the statement. If it is not, explain why.\",\n",
" },\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": f\"Verify that `{answer}` follows the chain of thought: {chain_of_thought}\",\n",
" },\n",
" ],\n",
" response_model=Validation,\n",
" )\n",
" if not resp.is_valid:\n",
" raise ValueError(resp.error_message)\n",
" return values"
],
"execution_count": 75,
"outputs": [],
"id": "e9ab3804-6962-4a48-83da-1f8360d8379a"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The use of the 'before' argument in this context is significant. It means that the validator will receive the complete dictionary of inputs in their raw form, before any parsing by Pydantic."
],
"id": "b79b94cf-15c2-432b-b0d5-aad0c2997f91"
},
{
"cell_type": "code",
"metadata": {},
"source": [
"from typing import Any\n",
"from pydantic import model_validator\n",
"\n",
"\n",
"class AIResponse(BaseModel):\n",
" chain_of_thought: str\n",
" answer: str\n",
"\n",
" @model_validator(mode=\"before\")\n",
" @classmethod\n",
" def chain_of_thought_makes_sense(cls, data: Any) -> Any:\n",
" # here we assume data is the dict representation of the model\n",
" # since we use 'before' mode.\n",
" return validate_chain_of_thought(data)"
],
"execution_count": 76,
"outputs": [],
"id": "fbc9887a-df0d-4a4b-9ef5-ea450701d85b"
},
{
"cell_type": "code",
"metadata": {},
"source": [
"AIResponse(\n",
" chain_of_thought=\"The user suffers from diabetes.\",\n",
" answer=\"The user has a broken leg.\",\n",
")"
],
"execution_count": 77,
"outputs": [
{
"ename": "ValidationError",
"evalue": "1 validation error for AIResponse\n Value error, The statement about the user having a broken leg does not logically follow from the information provided about the user suffering from diabetes. These are two separate health conditions and one does not imply the other. [type=value_error, input_value={'chain_of_thought': 'The...user has a broken leg.'}, input_type=dict]\n For further information visit https://errors.pydantic.dev/2.4/v/value_error",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mValidationError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb Cell 47\u001b[0m line \u001b[0;36m1\n\u001b[0;32m----> <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#Y103sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m AIResponse(chain_of_thought\u001b[39m=\u001b[39;49m\u001b[39m\"\u001b[39;49m\u001b[39mThe user suffers from diabetes.\u001b[39;49m\u001b[39m\"\u001b[39;49m, answer\u001b[39m=\u001b[39;49m\u001b[39m\"\u001b[39;49m\u001b[39mThe user has a broken leg.\u001b[39;49m\u001b[39m\"\u001b[39;49m)\n",
"File \u001b[0;32m~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:164\u001b[0m, in \u001b[0;36mBaseModel.__init__\u001b[0;34m(__pydantic_self__, **data)\u001b[0m\n\u001b[1;32m 162\u001b[0m \u001b[39m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[1;32m 163\u001b[0m __tracebackhide__ \u001b[39m=\u001b[39m \u001b[39mTrue\u001b[39;00m\n\u001b[0;32m--> 164\u001b[0m __pydantic_self__\u001b[39m.\u001b[39;49m__pydantic_validator__\u001b[39m.\u001b[39;49mvalidate_python(data, self_instance\u001b[39m=\u001b[39;49m__pydantic_self__)\n",
"\u001b[0;31mValidationError\u001b[0m: 1 validation error for AIResponse\n Value error, The statement about the user having a broken leg does not logically follow from the information provided about the user suffering from diabetes. These are two separate health conditions and one does not imply the other. [type=value_error, input_value={'chain_of_thought': 'The...user has a broken leg.'}, input_type=dict]\n For further information visit https://errors.pydantic.dev/2.4/v/value_error"
]
}
],
"id": "a38f2b28-f5b9-4a44-bfe5-9735726ec57d"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Reasking with validators\n",
"\n",
"For most of these examples all we've done we've mostly only defined the validation logic.\n",
"\n",
"We'eve covered field validators and model validators and even used LLMs to validate our outputs. But we haven't actually used the validators to reask the model! One of the most powerful features of `instructor` is that it will automatically reask the model when it receives a validation error. This means that we can use the same validation logic for both code-based and LLM-based validation.\n",
"\n",
"This also means that our 'prompt' is not only the prompt we send, but the code that runs the validator, and the error message we send back to the model."
],
"id": "5bbbaa11-32d2-4772-bc31-18d1d6d6c919"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Integrating these validation examples with the OpenAI API is streamlined using `instructor`. After patching the OpenAI client with `instructor`, you simply need to specify a `response_model` for your requests. This setup ensures that all the validation processes occur automatically.\n",
"\n",
"To enable reasking you can set a maximum number of retries. When calling the OpenAI client, the system can re-attempt to generate a correct answer. It does this by resending the original query along with feedback on why the previous response was rejected, guiding the LLM towards a more accurate answer in subsequent attempts."
],
"id": "39e642d9-0d20-4231-a694-baa0ea03f147"
},
{
"cell_type": "code",
"metadata": {},
"source": [
"class QuestionAnswer(BaseModel):\n",
" question: str\n",
" answer: str\n",
"\n",
"\n",
"question = \"What is the meaning of life?\"\n",
"context = (\n",
" \"The according to the devil the meaning of life is a life of sin and debauchery.\"\n",
")\n",
"\n",
"\n",
"resp = client.create(\n",
" model=\"gpt-5.4-mini\",\n",
" response_model=QuestionAnswer,\n",
" messages=[\n",
" {\n",
" \"role\": \"system\",\n",
" \"content\": \"You are a system that answers questions based on the context. answer exactly what the question asks using the context.\",\n",
" },\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": f\"using the context: `{context}`\\n\\nAnswer the following question: `{question}`\",\n",
" },\n",
" ],\n",
")\n",
"\n",
"resp.answer"
],
"execution_count": 79,
"outputs": [
{
"data": {
"text/plain": [
"'a life of sin and debauchery'"
]
},
"execution_count": 79,
"metadata": {},
"output_type": "execute_result"
}
],
"id": "97f544e7-2552-465c-89a9-a4820f00d658"
},
{
"cell_type": "code",
"metadata": {},
"source": [
"from pydantic import BeforeValidator\n",
"\n",
"\n",
"class QuestionAnswer(BaseModel):\n",
" question: str\n",
" answer: Annotated[\n",
" str,\n",
" BeforeValidator(llm_validator(\"don't say objectionable things\", client=client)),\n",
" ]\n",
"\n",
"\n",
"resp = client.create(\n",
" model=\"gpt-5.4-mini\",\n",
" response_model=QuestionAnswer,\n",
" max_retries=2,\n",
" messages=[\n",
" {\n",
" \"role\": \"system\",\n",
" \"content\": \"You are a system that answers questions based on the context. answer exactly what the question asks using the context.\",\n",
" },\n",
" {\n",
" \"role\": \"user\",\n",
" \"content\": f\"using the context: `{context}`\\n\\nAnswer the following question: `{question}`\",\n",
" },\n",
" ],\n",
")\n",
"\n",
"resp.answer"
],
"execution_count": 80,
"outputs": [
{
"data": {
"text/plain": [
"'The meaning of life is a concept that varies depending on individual perspectives and beliefs.'"
]
},
"execution_count": 80,
"metadata": {},
"output_type": "execute_result"
}
],
"id": "0328bbc5"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Conclusion"
],
"id": "a0c07b8b-ba6d-4e5d-a26c-ba72ca7d4f22"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This guide explains how to use deterministic and probabilistic validation techniques with Large Language Models (LLMs). We discussed using an instructor to establish validation processes for content filtering, context relevance maintenance, and model reasoning verification. These methods enhance the performance of LLMs across different tasks.\n",
"\n",
"For those interested in further exploration, here's a to-do list:\n",
"\n",
"1. **SQL Syntax Checker**: Create a validator to check the syntax of SQL queries before executing them.\n",
"2. **Context-Based Response Validation**: Design a method to flag responses based on the model's own knowledge rather than the provided context.\n",
"3. **PII Detection**: Implement a mechanism to identify and handle Personally Identifiable Information in responses while prioritizing user privacy.\n",
"4. **Targeted Rule-Based Filtering**: Develop filters to remove specific content types, such as responses mentioning named entities.\n",
"\n",
"Completing these tasks will enable users to acquire practical skills in improving LLMs through advanced validation methods."
],
"id": "344c623a-9b3b-4134-92d4-ad4eb9bb5f9e"
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
docs/tutorials/5-knowledge-graphs.ipynb
+1007
View File
File diff suppressed because it is too large Load Diff
docs/tutorials/6-chain-of-density.ipynb
+979
View File
File diff suppressed because one or more lines are too long
docs/tutorials/7-synthetic-data-generation.ipynb
+691
View File
@@ -0,0 +1,691 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Synthetic Data Generation\n",
" \n",
"RAG Applications are often tricky to evaluate, especially when you haven't obtained any user queries to begin. In this notebook, we'll see how we can use `instructor` to quickly generate synthetic questions from a dataset to benchmark your retrieval systems using some simple metrics. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Data Ingestion\n",
"\n",
"Let's first start by installing the required packages and ingesting the first 200 rows of the `ms-marco` dataset into our local database. "
]
},
{
"cell_type": "code",
"execution_count": 91,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[2mAudited \u001b[1m7 packages\u001b[0m in 301ms\u001b[0m\n"
]
}
],
"source": [
"!uv pip install instructor openai datasets lancedb tantivy tenacity tqdm"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We're using `lancedb` here to easily ingest large amounts of data. This is preferable since we can define our table schema using a `Pydantic` Schema and also have LanceDB automatically handle the generation of the embeddings using their `get_registry()` method that we can define as an object property."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"from lancedb import connect\n",
"\n",
"\n",
"DB_PATH = \"./db\"\n",
"DB_TABLE = \"ms_marco\"\n",
"\n",
"# Create a db at the path `./db`\n",
"db = connect(DB_PATH)"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [],
"source": [
"from lancedb.pydantic import LanceModel, Vector\n",
"from lancedb.embeddings import get_registry\n",
"\n",
"\n",
"func = get_registry().get(\"openai\").create(name=\"text-embedding-3-small\")\n",
"\n",
"\n",
"class Chunk(LanceModel):\n",
" passage: str = func.SourceField()\n",
" chunk_id: str\n",
" embedding: Vector(func.ndims()) = func.VectorField()\n",
"\n",
"\n",
"table = db.create_table(DB_TABLE, schema=Chunk, exist_ok=True, mode=\"overwrite\")"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [],
"source": [
"from datasets import load_dataset\n",
"\n",
"N_ROWS = 200\n",
"\n",
"dataset = load_dataset(\"ms_marco\", \"v1.1\", split=\"train\", streaming=True).take(N_ROWS)"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"dict_keys(['answers', 'passages', 'query', 'query_id', 'query_type', 'wellFormedAnswers'])"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# from itertools import islice\n",
"first_item = next(iter(dataset))\n",
"first_item.keys()"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[\"Since 2007, the RBA's outstanding reputation has been affected by the 'Securency' or NPA scandal. These RBA subsidiaries were involved in bribing overseas officials so that Australia might win lucrative note-printing contracts. The assets of the bank include the gold and foreign exchange reserves of Australia, which is estimated to have a net worth of A$101 billion. Nearly 94% of the RBA's employees work at its headquarters in Sydney, New South Wales and at the Business Resumption Site.\",\n",
" \"The Reserve Bank of Australia (RBA) came into being on 14 January 1960 as Australia 's central bank and banknote issuing authority, when the Reserve Bank Act 1959 removed the central banking functions from the Commonwealth Bank. The assets of the bank include the gold and foreign exchange reserves of Australia, which is estimated to have a net worth of A$101 billion. Nearly 94% of the RBA's employees work at its headquarters in Sydney, New South Wales and at the Business Resumption Site.\",\n",
" 'RBA Recognized with the 2014 Microsoft US Regional Partner of the ... by PR Newswire. Contract Awarded for supply and support the. Securitisations System used for risk management and analysis. ']"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"first_item[\"passages\"][\"passage_text\"][:3]"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [],
"source": [
"import hashlib\n",
"from itertools import batched\n",
"\n",
"\n",
"def get_passages(dataset):\n",
" for row in dataset:\n",
" for passage in row[\"passages\"][\"passage_text\"]:\n",
" yield {\n",
" \"passage\": passage,\n",
" \"chunk_id\": hashlib.md5(passage.encode()).hexdigest(),\n",
" }\n",
"\n",
"\n",
"passages = batched(get_passages(dataset), 10)\n",
"\n",
"for passage_batch in passages:\n",
" # print(passage_batch)\n",
" table.add(list(passage_batch))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Synthetic Questions\n",
"\n",
"Now that we have the first ~2000 passages from the MS-Marco dataset ingested into our database. Let's start generating some synthetic questions using the chunks we've ingested. \n",
"\n",
"Let's see how we might do so using `instructor` by defining a datamodel that can help support this use-case."
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [],
"source": [
"from pydantic import BaseModel, Field\n",
"\n",
"\n",
"class QuestionAnswerPair(BaseModel):\n",
" \"\"\"\n",
" This model represents a pair of a question generated from a text chunk, its corresponding answer,\n",
" and the chain of thought leading to the answer. The chain of thought provides insight into how the answer\n",
" was derived from the question.\n",
" \"\"\"\n",
"\n",
" chain_of_thought: str = Field(\n",
" description=\"The reasoning process leading to the answer.\"\n",
" )\n",
" question: str = Field(description=\"The generated question from the text chunk.\")\n",
" answer: str = Field(description=\"The answer to the generated question.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Once we've defined this data-model, we can then use it in an instructor call to generate a synthetic question."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"chain_of_thought\": \"To form a specific question from the given text chunk, I should focus on the unique details provided about the Reserve Bank of Australia, such as its creation, functions, and assets.\",\n",
" \"question\": \"When was the Reserve Bank of Australia established as Australia's central bank and banknote issuing authority?\",\n",
" \"answer\": \"The Reserve Bank of Australia was established as Australia's central bank and banknote issuing authority on 14 January 1960.\"\n",
"}\n"
]
}
],
"source": [
"import instructor\n",
"\n",
"client = instructor.from_provider(\"openai/gpt-4o\")\n",
"\n",
"\n",
"def generate_question(chunk: str) -> QuestionAnswerPair:\n",
" return client.create(\n",
" model=\"gpt-4o\",\n",
" messages=[\n",
" {\n",
" \"role\": \"system\",\n",
" \"content\": \"You are a world class AI that excels at generating hypothetical search queries. You're about to be given a text snippet and asked to generate a search query which is specific to the specific text chunk that you'll be given. Make sure to use information from the text chunk.\",\n",
" },\n",
" {\"role\": \"user\", \"content\": f\"Here is the text chunk: {chunk}\"},\n",
" ],\n",
" response_model=QuestionAnswerPair,\n",
" )\n",
"\n",
"\n",
"text_chunk = \"\"\"\n",
"The Reserve Bank of Australia (RBA) came into being on 14 January 1960 as Australia 's central bank and banknote issuing authority, when the Reserve Bank Act 1959 removed the central banking functions from the Commonwealth Bank. The assets of the bank include the gold and foreign exchange reserves of Australia, which is estimated to have a net worth of A$101 billion. Nearly 94% of the RBA's employees work at its headquarters in Sydney, New South Wales and at the Business Resumption Site.\n",
"\"\"\"\n",
"print(generate_question(text_chunk).model_dump_json(indent=2))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that we've seen how to generate a single question, let's see how we might be able to scale this up. We can do so by taking advantage of the `asyncio` library and `tenacity` to handle retries."
]
},
{
"cell_type": "code",
"execution_count": 56,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[\"Since 2007, the RBA's outstanding reputation has been affected by the 'Securency' or NPA scandal. These RBA subsidiaries were involved in bribing overseas officials so that Australia might win lucrative note-printing contracts. The assets of the bank include the gold and foreign exchange reserves of Australia, which is estimated to have a net worth of A$101 billion. Nearly 94% of the RBA's employees work at its headquarters in Sydney, New South Wales and at the Business Resumption Site.\",\n",
" \"The Reserve Bank of Australia (RBA) came into being on 14 January 1960 as Australia 's central bank and banknote issuing authority, when the Reserve Bank Act 1959 removed the central banking functions from the Commonwealth Bank. The assets of the bank include the gold and foreign exchange reserves of Australia, which is estimated to have a net worth of A$101 billion. Nearly 94% of the RBA's employees work at its headquarters in Sydney, New South Wales and at the Business Resumption Site.\"]"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"chunks = table.to_pandas()\n",
"chunks = [item for item in chunks[\"passage\"]]\n",
"chunks[:2]"
]
},
{
"cell_type": "code",
"execution_count": 98,
"metadata": {},
"outputs": [],
"source": [
"from asyncio import Semaphore\n",
"from tenacity import retry, stop_after_attempt, wait_exponential\n",
"import asyncio\n",
"import instructor\n",
"\n",
"client = instructor.from_provider(\"openai/gpt-5.4-mini\", async_client=True)\n",
"\n",
"\n",
"async def generate_questions(chunks: list[str], max_queries: int):\n",
" @retry(\n",
" stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=10)\n",
" )\n",
" async def generate_question(\n",
" chunk: str, sem: Semaphore\n",
" ) -> tuple[QuestionAnswerPair, str]:\n",
" async with sem:\n",
" return (\n",
" await client.create(\n",
" model=\"gpt-5.4-mini\",\n",
" messages=[\n",
" {\n",
" \"role\": \"system\",\n",
" \"content\": \"You are a world class AI that excels at generating hypothetical search queries. You're about to be given a text snippet and asked to generate a search query which is specific to the specific text chunk that you'll be given. Make sure to use information from the text chunk.\",\n",
" },\n",
" {\"role\": \"user\", \"content\": f\"Here is the text chunk: {chunk}\"},\n",
" ],\n",
" response_model=QuestionAnswerPair,\n",
" ),\n",
" chunk,\n",
" )\n",
"\n",
" sem = Semaphore(max_queries)\n",
" coros = [generate_question(chunk, sem) for chunk in chunks]\n",
" return await asyncio.gather(*coros)\n",
"\n",
"\n",
"questions = await generate_questions(chunks[:300], 10)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Benchmarking Retrieval\n",
"\n",
"Now that we've generated a list of questions to query our database with, let's do a quick benchmark to see how full text search compares against that of hybrid search. We'll use two simple metrics here - Mean Reciprocal Rank ( MRR ) and Recall.\n",
"\n",
"Let's start by making sure we have an inverted index created on our table above that we can perform full text search on"
]
},
{
"cell_type": "code",
"execution_count": 64,
"metadata": {},
"outputs": [],
"source": [
"table.create_fts_index(\"passage\", replace=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This allows us to then use the `.search` function on each table to query it using full text search. Let's see an example below."
]
},
{
"cell_type": "code",
"execution_count": 67,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"A rebuildable atomizer (RBA), often referred to as simply a “rebuildable,” is just a special type of atomizer used in the Vape Pen and Mod Industry that connects to a personal vaporizer. 1 The bottom feed RBA is, perhaps, the easiest of all RBA types to build, maintain, and use. 2 It is filled from below, much like bottom coil clearomizer. 3 Bottom feed RBAs can utilize cotton instead of silica for the wick. 4 The Genesis, or genny, is a top feed RBA that utilizes a short woven mesh wire.\n",
"Results-Based Accountability® (also known as RBA) is a disciplined way of thinking and taking action that communities can use to improve the lives of children, youth, families, adults and the community as a whole. RBA is also used by organizations to improve the performance of their programs. RBA improves the lives of children, families, and communities and the performance of programs because RBA: 1 Gets from talk to action quickly; 2 Is a simple, common sense process that everyone can understand; 3 Helps groups to surface and challenge assumptions that can be barriers to innovation;\n"
]
}
],
"source": [
"for entry in table.search(\"RBA\", query_type=\"fts\").limit(2).to_list():\n",
" print(entry[\"passage\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Metrics\n",
"\n",
"Now that we've figured out how we might be able to query our table using full text search. Let's take a step back and see how we can implement some metrics to quantiatively evaluate the retrieved items. It's important to note that when we want to evaluate the quality of our listings, we always take it at some subset of k.\n",
"\n",
"This is important because k is often constrained by a business outcome and can help us determine how well our solution works\n",
"\n",
"Eg. Here are some hypothetical scenarios\n",
"\n",
"- k=5 : We'd like to display some recommended items based of a user query (Eg. Help me plan out a dinner with Jonathan next week -> Display 5 possible actions)\n",
"- k=10 : We have a small carousel with recommended items for a user to buy\n",
"- k=25 : We're using a re-ranker, is it filtering out the irrelevant chunks from the relevant chunks well?\n",
"- k=50 : We have a pipeline that fetches information for a model to respond with, are we fetching all relevant bits of information\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Reciprocal Rank\n",
"\n",
"Reciprocal Rank\n",
"Imagine we're spotify and we want to suggest a couple of songs to the user. Which is a better result among the two lists of retrieved songs below? ( Note that 2 is the answer we want )\n",
"\n",
"- [0,1,2,3,4]\n",
"- [0,1,3,4,2]\n",
"\n",
"Obviously if we're suggesting songs to the user, we want the first relevant song to be listed as early as possible! Therefore we'd prefer 1 over 2 in the example above because 2 is ordered earlier in the first case. A metric that works well for this is the Reciprocal Rank (RR).\n",
"\n",
"![](../img/mrr_eqn.png)\n"
]
},
{
"cell_type": "code",
"execution_count": 84,
"metadata": {},
"outputs": [],
"source": [
"def rr(results, labels):\n",
" return max(\n",
" [\n",
" round(1 / (results.index(label) + 1), 2) if label in results else 0\n",
" for label in labels\n",
" ]\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This is an aggressive metric and once we get to an position of > 10, the value doesn't change much anymore. Most of the big changes happen at indexes < 10."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Recall\n",
"\n",
"Another metric that we can track is recall which measures how many of our retrieved items were retrieved. \n",
"\n",
"![](../img/recall_eqn.png)"
]
},
{
"cell_type": "code",
"execution_count": 69,
"metadata": {},
"outputs": [],
"source": [
"def recall(results, relevant_chunks):\n",
" return sum([1 if chunk in results else 0 for chunk in relevant_chunks]) / len(\n",
" relevant_chunks\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Using Our Questions\n",
"\n",
"Now that we've seen two metrics that we can use and how we might be able to generate some synthetic questions, let's try it out on an actual question.\n",
"\n",
"To do so, we'll first generate a unique chunk id for our original passage that we generated the question from. \n",
"\n",
"We'll then compare the chunk_ids of the retrieved chunks and then compute the `mrr` and the `recall` of the retrieved results."
]
},
{
"cell_type": "code",
"execution_count": 86,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"('b6d9bf888fd53590ee69a913bd9bf8a4',\n",
" \"What factors influence the average salary for people with a bachelor's degree?\",\n",
" \"However, the average salary for people with a bachelor's degree varies widely based upon several factors, including their major, job position, location and years of experience. The National Association of Colleges and Employers conducted a salary survey that determined the average starting salary for graduates of various bachelor's degree programs.\")"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import hashlib\n",
"\n",
"sample_question, chunk = questions[0]\n",
"\n",
"chunk_id = hashlib.md5(chunk.encode()).hexdigest()\n",
"chunk_id, sample_question.question, chunk"
]
},
{
"cell_type": "code",
"execution_count": 81,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['b6d9bf888fd53590ee69a913bd9bf8a4',\n",
" '7a0254c9dc709220367857dcb67f2c8d',\n",
" '04e7e6f91463033aa87b4104ea16b477']"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"retrieved_results = (\n",
" table.search(sample_question.question, query_type=\"fts\").limit(25).to_list()\n",
")\n",
"retrieved_chunk_ids = [item[\"chunk_id\"] for item in retrieved_results]\n",
"\n",
"retrieved_chunk_ids[:3]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can now compute the results for the retrieved items that we've obtained using full text search relative to the ground truth label that we have - the original chunk that we generated it from"
]
},
{
"cell_type": "code",
"execution_count": 85,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(1.0, 1.0)"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"recall(retrieved_chunk_ids, [chunk_id]), rr(retrieved_chunk_ids, [chunk_id])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Scaling it up for different values of `k`, where we can see how this value changes for different subsets of the retrieved items is relatively simple. \n",
"\n",
"We can generate this mapping automatically using `itertools.product`"
]
},
{
"cell_type": "code",
"execution_count": 112,
"metadata": {},
"outputs": [],
"source": [
"from itertools import product\n",
"\n",
"SIZES = [3, 5, 10, 15, 25]\n",
"METRICS = [[\"mrr\", rr], [\"recall\", recall]]\n",
"\n",
"score_fns = {}\n",
"\n",
"for metric, size in product(METRICS, SIZES):\n",
" metric_name, score_fn = metric\n",
" score_fns[f\"{metric_name}@{size}\"] = (\n",
" lambda predictions, labels, fn=score_fn, k=size: fn(predictions[:k], labels)\n",
" ) # type: ignore"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Running an Evaluation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can now use the code above to run a test to see how our full text search performs for our synthetic questions. "
]
},
{
"cell_type": "code",
"execution_count": 114,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"100%|██████████| 300/300 [00:07<00:00, 41.64it/s]\n"
]
}
],
"source": [
"import hashlib\n",
"from tqdm import tqdm\n",
"\n",
"fts_results = []\n",
"\n",
"for sample_qn, chunk in tqdm(questions):\n",
" chunk_id = hashlib.md5(chunk.encode()).hexdigest()\n",
" cleaned_question = \"\".join(\n",
" char for char in sample_qn.question if char.isalnum() or char.isspace()\n",
" )\n",
" retrieved_results = (\n",
" table.search(cleaned_question, query_type=\"fts\").limit(25).to_list()\n",
" )\n",
" retrieved_chunk_ids = [item[\"chunk_id\"] for item in retrieved_results]\n",
"\n",
" fts_results.append(\n",
" {\n",
" metric: score_fn(retrieved_chunk_ids, [chunk_id])\n",
" for metric, score_fn in score_fns.items()\n",
" }\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": 115,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"mrr@3 0.784267\n",
"mrr@5 0.791267\n",
"mrr@10 0.797633\n",
"mrr@15 0.798133\n",
"mrr@25 0.798433\n",
"recall@3 0.896667\n",
"recall@5 0.926667\n",
"recall@10 0.973333\n",
"recall@15 0.980000\n",
"recall@25 0.986667\n",
"dtype: float64"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"\n",
"df = pd.DataFrame(fts_results)\n",
"df.mean()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that on average full text search is able to surface the relevant item 97-98% of the time if we take `k=10` and that we have the relevant item in between the first and second item here.\n",
"\n",
"Now, because these are synthetic question, there's likely to be a large amount of overlap in the phrases used in the questions and the original source text, leading to the high values.\n",
"\n",
"In actual production applications and your domain specific dataset, it's useful to do these experiments and see what works best for your needs."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
docs/tutorials/index.md
+119
View File
@@ -0,0 +1,119 @@
---
title: Instructor Tutorials
description: Interactive, step-by-step tutorials for learning how to use Instructor effectively
---
# Instructor Tutorials
<div class="grid cards" markdown>
- :material-school: **Learning Path**
Follow our structured learning path to become an Instructor expert
[:octicons-arrow-right-16: Start Learning](#tutorial-pathway)
- :material-notebook-edit: **Interactive Formats**
Run our Jupyter notebooks in your preferred environment
[:octicons-arrow-right-16: Run Options](#running-options)
- :material-certificate: **Skill Building**
Gain practical skills for real-world AI applications
[:octicons-arrow-right-16: What You'll Learn](#skills-gained)
- :material-help: **Support**
Get help when you need it
[:octicons-arrow-right-16: Get Help](#getting-help)
</div>
## Tutorial Pathway {#tutorial-pathway}
Our tutorials follow a carefully designed learning path from basic concepts to advanced applications. Each tutorial builds on previous concepts while introducing new techniques.
| Tutorial | Topic | Key Skills | Difficulty |
|----------|-------|------------|------------|
| 1. [Introduction to Structured Outputs](./1-introduction.ipynb) | Basic extraction | Pydantic models, basic prompting | 🟢 Beginner |
| 2. [Tips and Tricks](./2-tips.ipynb) | Best practices | Advanced models, optimization | 🟢 Beginner |
| 3. [Applications: RAG](./3-0-applications-rag.ipynb) | Retrieval-augmented generation | Information retrieval, context handling | 🟡 Intermediate |
| 4. [Applications: RAG Validation](./3-1-validation-rag.ipynb) | Validating RAG outputs | Quality control, validation hooks | 🟡 Intermediate |
| 5. [Validation Techniques](./4-validation.ipynb) | Deep validation | Custom validators, error handling | 🟡 Intermediate |
| 6. [Knowledge Graphs](./5-knowledge-graphs.ipynb) | Graph building | Entity relationships, graph visualization | 🔴 Advanced |
| 7. [Chain of Density](./6-chain-of-density.ipynb) | Summarization techniques | Iterative refinement, content density | 🔴 Advanced |
| 8. [Synthetic Data Generation](./7-synthetic-data-generation.ipynb) | Creating datasets | Data augmentation, testing data | 🔴 Advanced |
## Running Options {#running-options}
Choose your preferred environment to work through these interactive Jupyter notebooks:
<div class="grid cards" markdown>
- :material-laptop: **Run Locally**
```bash
git clone https://github.com/jxnl/instructor.git
cd instructor
pip install -e ".[all]"
jupyter notebook docs/tutorials/
```
- :material-google: **Google Colab**
Look for the "Open in Colab" button at the top of each notebook
Perfect for cloud execution without local setup
- :simple-mybinder: **Binder**
Click the "Launch Binder" button to run instantly in your browser
No installation or API keys required for basic examples
</div>
## Skills Gained {#skills-gained}
By completing this tutorial series, you'll gain practical skills in:
- **Structured Extraction**: Define Pydantic models that capture exactly the data you need
- **Advanced Validation**: Ensure LLM outputs meet your data quality requirements
- **Streaming Responses**: Process data in real-time with partial and iterative outputs
- **Complex Applications**: Build RAG systems, knowledge graphs, and more
- **Multi-Provider Support**: Work with different LLM providers using a consistent interface
- **Production Techniques**: Learn optimization strategies for real-world applications
## Setup Requirements
Before starting, make sure you have:
- **Python Environment**: Python 3.8+ installed
- **Dependencies**: Install with `pip install "instructor[all]"`
- **API Keys**: Access to OpenAI API or other supported providers
- **Basic Knowledge**: Familiarity with Python and basic LLM concepts
## Getting Help {#getting-help}
We're here to support your learning journey:
- **Documentation**: Check the [core concepts](../concepts/index.md) for detailed explanations
- **FAQ**: Browse our [frequently asked questions](../faq.md)
- **Community**: Join our [Discord server](https://discord.gg/bD9YE9JArw) for real-time help
- **Issues**: Report problems on [GitHub](https://github.com/jxnl/instructor/issues)
- **Examples**: See [practical examples](../examples/index.md) of Instructor in action
<div class="grid cards" markdown>
- :material-play-circle: **Ready to Begin?**
Start your journey with our first tutorial on structured outputs
[:octicons-arrow-right-16: Start Learning](./1-introduction.ipynb){: .md-button .md-button--primary }
</div>