chore: import upstream snapshot with attribution

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,19 @@
+---
+name: Feature request
+about: Suggest an idea for the Cookbook
+title: "[FEATURE]"
+labels: ''
+assignees: ''
+
+---
+
+[optional template]
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

.github/ISSUE_TEMPLATE/get-help.md

@@ -0,0 +1,24 @@
+---
+name: Get help
+about: Ask a question to get help with the OpenAI API
+title: "[SUPPORT]"
+labels: support
+assignees: ''
+
+---
+
+**Please do not use the issues page to ask general questions about the OpenAI API.** Questions asked here will usually not receive answers.
+
+Feel free to report problems with code examples, suggest new code examples, or ask narrow questions about specific code examples.
+
+For general discussion, try:
+
+- [OpenAI API Community Forum](https://community.openai.com/)
+- [OpenAI Discord](https://discord.com/invite/openai)
+- [OpenAI subreddit](https://www.reddit.com/r/OpenAI/), [GPT3 subreddit](https://www.reddit.com/r/GPT3/)
+- [OpenAI Cookbook discussion page](https://github.com/openai/openai-cookbook/discussions)
+
+For general help, try:
+
+- [OpenAI Documentation](https://platform.openai.com/docs/introduction)
+- [OpenAI Help Center](https://help.openai.com/en/)

.github/ISSUE_TEMPLATE/problem-report.md

@@ -0,0 +1,25 @@
+---
+name: Problem report
+about: Create a report to help us fix problems
+title: "[PROBLEM]"
+labels: bug
+assignees: ''
+
+---
+
+[optional format]
+
+**Identify the file to be fixed**
+The name of the file containing the problem.
+
+**Describe the problem**
+A clear and concise description of what the problem is.
+
+**Describe a solution**
+A clear and concise description of what a fixed version should do.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Additional context**
+Add any other context about the problem here.

.github/authors_schema.json

@@ -0,0 +1,25 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "type": "object",
+  "patternProperties": {
+    "^.*$": {
+      "type": "object",
+      "properties": {
+        "name": {
+          "type": "string"
+        },
+        "website": {
+          "type": "string",
+          "format": "uri"
+        },
+        "avatar": {
+          "type": "string",
+          "format": "uri"
+        }
+      },
+      "required": ["name", "website", "avatar"],
+      "additionalProperties": false
+    }
+  },
+  "additionalProperties": false
+}

.github/pull_request_template.md

@@ -0,0 +1,24 @@
+## Summary
+
+Briefly describe the changes and the goal of this PR. Make sure the PR title summarizes the changes effectively.
+
+## Motivation
+
+Why are these changes necessary? How do they improve the cookbook?
+
+---
+
+## For new content
+
+When contributing new content, read through our [contribution guidelines](https://github.com/openai/openai-cookbook/blob/main/CONTRIBUTING.md), and mark the following action items as completed:
+
+- [ ] I have added a new entry in [registry.yaml](https://github.com/openai/openai-cookbook/blob/main/registry.yaml) (and, optionally, in [authors.yaml](https://github.com/openai/openai-cookbook/blob/main/authors.yaml)) so that my content renders on the cookbook website.
+- [ ] I have conducted a self-review of my content based on the [contribution guidelines](https://github.com/openai/openai-cookbook/blob/main/CONTRIBUTING.md#rubric):
+  - [ ] Relevance: This content is related to building with OpenAI technologies and is useful to others.
+  - [ ] Uniqueness: I have searched for related examples in the OpenAI Cookbook, and verified that my content offers new insights or unique information compared to existing documentation.
+  - [ ] Spelling and Grammar: I have checked for spelling or grammatical mistakes.
+  - [ ] Clarity: I have done a final read-through and verified that my submission is well-organized and easy to understand.
+  - [ ] Correctness: The information I include is correct and all of my code executes successfully.
+  - [ ] Completeness: I have explained everything fully, including all necessary references and citations.
+
+We will rate each of these areas on a scale from 1 to 4, and will only accept contributions that score 3 or higher on all areas. Refer to our [contribution guidelines](https://github.com/openai/openai-cookbook/blob/main/CONTRIBUTING.md) for more details.

.github/registry_schema.json

@@ -0,0 +1,48 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "type": "array",
+  "items": {
+    "type": "object",
+    "properties": {
+      "title": {
+        "type": "string"
+      },
+      "path": {
+        "type": "string"
+      },
+      "slug": {
+        "type": "string"
+      },
+      "description": {
+        "type": "string"
+      },
+      "redirects": {
+        "type": "array",
+        "items": {
+          "type": "string"
+        }
+      },
+      "tags": {
+        "type": "array",
+        "items": {
+          "type": "string"
+        }
+      },
+      "authors": {
+        "type": "array",
+        "items": {
+          "type": "string"
+        }
+      },
+      "date": {
+        "type": "string",
+        "format": "date"
+      },
+      "archived": {
+        "type": "boolean"
+      }
+    },
+    "required": ["title", "path", "slug", "tags", "authors"],
+    "additionalProperties": false
+  }
+}

.github/scripts/check_notebooks.py

@@ -0,0 +1,61 @@
+import subprocess
+import sys
+from pathlib import Path
+
+import nbformat
+
+
+def get_changed_notebooks(base_ref: str = "origin/main") -> list[Path]:
+    """
+    Returns a list of changed notebook paths in the current git branch
+    compared to the specified base reference.
+    """
+    result = subprocess.run(
+        ["git", "diff", "--name-only", base_ref, "--", "*.ipynb"],
+        capture_output=True,
+        text=True,
+        check=True,
+    )
+    return [Path(line.strip()) for line in result.stdout.splitlines() if line.strip()]
+
+
+def is_valid_notebook(path: Path) -> bool:
+    """
+    Checks if the notebook at the given path is valid by attempting to read it
+    with nbformat.
+    """
+    try:
+        with open(path, "r", encoding="utf-8") as f:
+            nbformat.read(f, as_version=4)
+        return True
+    except Exception as e:
+        print(f"{path}: INVALID - {e}")
+        return False
+
+
+def main() -> None:
+    """
+    Main function to validate the format of changed notebooks.
+    """
+    changed_notebooks = get_changed_notebooks()
+    if not changed_notebooks:
+        print("No changed .ipynb files to validate.")
+        sys.exit(0)
+
+    print(f"Validating {len(changed_notebooks)} notebook(s)...")
+    errors = 0
+    for path in changed_notebooks:
+        if not path.exists():
+            continue  # skip deleted files
+        if not is_valid_notebook(path):
+            errors += 1
+
+    if errors:
+        print(f"{errors} invalid notebook(s) found.")
+        sys.exit(1)
+    else:
+        print("All changed notebooks are valid.")
+
+
+if __name__ == "__main__":
+    main()

.github/workflows/build-website.yaml

@@ -0,0 +1,14 @@
+name: Rebuild Cookbook Website
+
+on:
+  push:
+    branches:
+      - main
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Call Deploy Hook
+        run: curl -X POST ${{ secrets.WEBSITE_DEPLOY_HOOK_URL }}

.github/workflows/stale.yaml

@@ -0,0 +1,19 @@
+name: "Close stale issues and PRs"
+on:
+  schedule:
+    - cron: "30 1 * * *"
+
+jobs:
+  stale:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/stale@1160a2240286f5da8ec72b1c0816ce2481aabf84 # v8
+        with:
+          stale-issue-message: "This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 10 days."
+          stale-pr-message: "This PR is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 10 days."
+          close-issue-message: "This issue was closed because it has been stalled for 10 days with no activity."
+          close-pr-message: "This PR was closed because it has been stalled for 10 days with no activity."
+          days-before-issue-stale: 60
+          days-before-pr-stale: 60
+          days-before-issue-close: 10
+          days-before-pr-close: 10

.github/workflows/validate-notebooks.yaml

@@ -0,0 +1,25 @@
+name: Validate Changed Notebooks
+
+on: [pull_request]
+
+jobs:
+  validate-notebooks:
+    name: Validate Notebooks
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@f43a0e5ff2bd294095638e18286ca9a3d1956744 # v3
+        with:
+          fetch-depth: 0  # needed for git diff to work
+
+      - name: Set up Python
+        uses: actions/setup-python@7f4fc3e22c37d6ff65e88745f38bd3157c663f7c # v4
+        with:
+          python-version: '3.12'
+
+      - name: Install dependencies
+        run: pip install nbformat
+
+      - name: Validate changed .ipynb files
+        run: python .github/scripts/check_notebooks.py

.gitignore

@@ -0,0 +1,156 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+!examples/voice_solutions/realtime_translation_guide/livekit-translation-demo/lib/
+!examples/voice_solutions/realtime_translation_guide/livekit-translation-demo/lib/**
+lib64
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.env.*
+!.env.example
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+pyvenv.cfg
+share/
+bin/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# Data
+*transactions*.jsonl
+/examples/data/transactions*
+*.DS_Store
+tmp_*
+examples/fine-tuned_qa/local_cache/*
+examples/multimodal/.local_cache/*
+examples/coding-agent-workspace
+
+# PyCharm files
+.idea/
+.cursorignore
+
+# VS Code files
+.vscode/
+.cursorignore
+
+# Generated by examples/agents_sdk/agent_improvement_loop.ipynb
+/examples/agents_sdk/agent_improvement_loop_artifacts/

AGENTS.md

@@ -0,0 +1,48 @@
+# Repository Guidelines
+
+## Project Structure & Module Organization
+
+The cookbook is organized around runnable examples and reference articles for OpenAI APIs. Place notebooks and Python scripts under `examples/<topic>/`, grouping related assets inside topic subfolders (for example, `examples/agents_sdk/`). Narrative guides and long-form docs live in `articles/`, and shared diagrams or screenshots belong in `images/`. Update `registry.yaml` whenever you add content so it appears on cookbook.openai.com, and add new author metadata in `authors.yaml` if you want custom attribution. Keep large datasets outside the repo; instead, document how to fetch them in the notebook.
+
+## Build, Test, and Development Commands
+
+Use a virtual environment to isolate dependencies:
+
+- `python -m venv .venv && source .venv/bin/activate`
+- `pip install -r examples/<topic>/requirements.txt` (each sample lists only what it needs)
+- `jupyter lab` or `jupyter notebook` to develop interactively
+- `python .github/scripts/check_notebooks.py` to validate notebook structure before pushing
+
+## Coding Style & Naming Conventions
+
+Write Python to PEP 8 with four-space indentation, descriptive variable names, and concise docstrings that explain API usage choices. Name new notebooks with lowercase, dash-or-underscore-separated phrases that match their directory—for example `examples/gpt-5/prompt-optimization-cookbook.ipynb`. Keep markdown cells focused and prefer numbered steps for multi-part workflows. Store secrets in environment variables such as `OPENAI_API_KEY`; never hard-code keys inside notebooks.
+
+## Testing Guidelines
+
+Execute notebooks top-to-bottom after installing dependencies and clear lingering execution counts before committing. For Python modules or utilities, include self-check cells or lightweight `pytest` snippets and show how to run them (for example, `pytest examples/object_oriented_agentic_approach/tests`). When contributions depend on external services, mock responses or gate the cells behind clearly labeled opt-in flags.
+
+## Commit & Pull Request Guidelines
+
+Use concise, imperative commit messages that describe the change scope (e.g., "Add agent portfolio collaboration demo"). Every PR should provide a summary, motivation, and self-review, and must tick the registry and authors checklist from `.github/pull_request_template.md`. Link issues when applicable and attach screenshots or output snippets for UI-heavy content. Confirm CI notebook validation passes locally before requesting review.
+
+## Metadata & Publication Workflow
+
+New or relocated content must have an entry in `registry.yaml` with an accurate path, date, and tag set so the static site generator includes it. When collaborating, coordinate author slugs in `authors.yaml` to avoid duplicates, and run `python -m yaml lint registry.yaml` (or your preferred YAML linter) to catch syntax errors before submitting.
+
+## Review Guidelines
+
+These are considered priority 0 issues for this repo, in addition to the normal priority for possible issues.
+- Verify file, function, and notebook names follow the repo's naming conventions and clearly describe their purpose.
+- Scan prose and markdown for typos, broken links, and inconsistent formatting before approving.
+- Check that code identifiers remain descriptive (no leftover placeholder names) and that repeated values are factored into constants when practical.
+- Ensure notebooks or scripts document any required environment variables instead of hard-coding secrets or keys.
+- Confirm metadata files (`registry.yaml`, `authors.yaml`) stay in sync with new or relocated content.
+
+## Recent Learnings
+
+- **`uv run` can inherit the wrong virtualenv in this repo** -> Clear `VIRTUAL_ENV` (for example `env -u VIRTUAL_ENV uv run ...`) when an unrelated environment is active -> Avoids misleading mismatch warnings and makes it clear the repo's `.venv` is the interpreter actually running the harnesses.
+- **Realtime eval shared imports can resolve the wrong module under pytest** -> Add `shared/__init__.py` and ensure tests prepend `examples/evals/realtime_evals` to `sys.path` before importing `shared.*` -> Prevents collection failures caused by unrelated installed packages named `shared`.
+- **Run-level grades can be overweighted by long simulations** -> Store turn-level grades on the matching turn and trace-level grades on one row per simulation instead of copying them onto every row -> Keeps `results.csv` row semantics intact and prevents summary means from favoring longer conversations.
+- **Synthetic-audio scaffold requests can pick the wrong harness** -> Default unspecified synthetic-audio evals to `crawl` text-to-TTS and reserve `walk` for replay-specific audio traits like noise, telephony artifacts, or speaker characteristics -> Keeps new realtime evals on the simplest harness unless audio realism is itself under test.
+- **Task-specific single-turn grading can outgrow the shared crawl schema** -> Keep the shared crawl harness for realtime execution, then add eval-local wrapper scripts that post-grade domain-specific quality and overwrite `results.csv` while preserving `results_base.csv` -> Avoids forking the harness when a use case needs richer grading than tool-call correctness.
+- **Synthetic learner audio can sound like eval scaffolding** -> Write `user_text` as a realistic in-app learner request and keep evaluation rules in metadata plus the system prompt -> Produces audio inputs that match the product surface instead of teaching the model the grading rubric through the spoken prompt.

CONTRIBUTING.md

@@ -0,0 +1,7 @@
+# Contributing the the cookbook
+
+The OpenAI Cookbook is a collection of useful patterns and examples of working with the OpenAI platform, provided as a community resource.
+
+> Contributions are reviewed on a best-effort basis - we can't provide guarantees around when or if content contributions will be reviewed or merged.
+
+Stay tuned to this page for further guidance on cookbook contributions as they become available 🙏

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 OpenAI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,20 @@
+<a href="https://cookbook.openai.com" target="_blank">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="/images/openai-cookbook-white.png" style="max-width: 100%; width: 400px; margin-bottom: 20px">
+    <img alt="OpenAI Cookbook Logo" src="/images/openai-cookbook.png" width="400px">
+  </picture>
+</a>
+
+<h3></h3>
+ 
+> ✨ Navigate at [cookbook.openai.com](https://cookbook.openai.com)
+
+Example code and guides for accomplishing common tasks with the [OpenAI API](https://platform.openai.com/docs/introduction). To run these examples, you'll need an OpenAI account and associated API key ([create a free account here](https://platform.openai.com/signup)). Set an environment variable called `OPENAI_API_KEY` with your API key. Alternatively, in most IDEs such as Visual Studio Code, you can create an `.env` file at the root of your repo containing `OPENAI_API_KEY=<your API key>`, which will be picked up by the notebooks.
+
+Most code examples are written in Python, though the concepts can be applied in any language.
+
+For other useful tools, guides and courses, check out these [related resources from around the web](https://cookbook.openai.com/related_resources).
+
+## License
+
+MIT License

README.wehub.md

@@ -0,0 +1,7 @@
+# WeHub 来源说明
+
+- 原始项目：`openai/openai-cookbook`
+- 原始仓库：https://github.com/openai/openai-cookbook
+- 导入方式：上游默认分支的最新快照
+- 原作者、版权和许可证信息以原始仓库及本仓库 LICENSE 为准
+- 本文件仅用于记录来源，不代表 WeHub 是原项目作者

articles/chatgpt-agents-sales-meeting-prep.md

@@ -0,0 +1,154 @@
+# Building workspace agents in ChatGPT to complete repeatable, end-to-end work
+
+This cookbook shows how to build a ChatGPT workspace agent. Currently available in research preview for ChatGPT Business, Enterprise, and Edu customers, workspace agents are evolutions of GPTs that can work across tools to complete high-value tasks, helping your team move faster.
+
+In this cookbook, we'll create a workspace agent through the conversational builder, give it access to our team's calendars and knowledge bases, and add a brief-writing skill to help craft the agent's output. Then we'll test our team agent, schedule it to run on a recurring schedule, and share it with teammates.
+
+## Introduction
+
+ChatGPT workspace agents are shared agents that can run repeatable workflows across ChatGPT and independently complete end-to-end tasks. They can use connected apps, follow skills, run on a schedule, and they can be shared with your colleagues in your workspace.
+
+In this cookbook, we'll build a workspace agent that supports a sales team. Sales teams often spend time manually pulling account notes, recent customer feedback, product usage metrics, and company news to prepare for upcoming customer meetings.
+
+We'll build a workspace agent that helps our sales team more efficiently prepare for their sales meetings by automatically generating meeting briefs ahead of upcoming customer calls, backed by up-to-date customer data. Specifically, our agent will run each day to check tomorrow's customer meetings, gather recent account context from SharePoint, and search the web for recent company news and background information on each attendee. Our agent will generate a meeting brief, save it down as a SharePoint doc, and send us an executive summary.
+
+Once our agent is deployed, it will run automatically each day, and we can also invoke it on-demand in ChatGPT.
+
+## Prerequisites and setup
+
+For this cookbook, you will need a ChatGPT Business, Enterprise, or Edu workspace with access to workspace agents, and a set of apps (connectors) that enable your agent to access your calendar and account information. We'll use Google Calendar and Microsoft SharePoint to show a workflow that uses both Google Workspace and Microsoft 365 apps, but you can switch out any supported apps. Our agent will send notifications in Slack, but you can also use email or another notification surface.
+
+Before you build, ask your workspace admin to confirm four settings:
+
+1. **Agents are enabled for your workspace.** Workspace owners can enable agents for the workspace and assign access to specific roles with RBAC.
+2. **You have permission to build and share agents.** An admin will need to enable you (or the appropriate RBAC group) to create agents. If you plan to share the agent, an admin may also need to enable publishing.
+3. **The apps you need are enabled.** For this cookbook, an admin would need to enable Google Calendar, Gmail, and SharePoint in workspace app settings.
+4. **[optional] Slack is approved for the workspace.** Only required if you want to interact with your agent through Slack. Enterprise admins need to enable the "workspace agents for Slack" app and grant access to users or groups with RBAC. Your Slack workspace must also allow the ChatGPT Agents app for Slack.
+
+## Building your agent
+
+The conversational agent builder is the fastest way to start. It can draft the agent profile, choose apps, generate a skill, write instructions, and prepare a draft agent that you can test in Preview. We'll use that flow here, then review each piece step by step so you can refine the agent later.
+
+Note that if you want inspiration on work to automate, you can start from pre-built agent templates including a Chief of Staff, Data Analysis, and Customer Support agents.
+
+<p align="center">
+  <img src="https://cdn.openai.com/cookbook/chatgpt-agents-sales-meeting-prep/img0.png" alt="Build a new agent" width="80%" />
+</p>
+
+### 1. Describe your workflow
+
+We'll enter the following text into the agent builder:
+
+```text
+Create an agent to help with sales meeting prep. The agent should check my Google Calendar for tomorrow's customer meetings. Exclude internal-only meetings and skip events with no external attendees.
+
+For each customer meeting, pull recent notes and feedback from SharePoint, then search the web for any relevant company news from the last 30 days. Then write a 2-3 page meeting brief. Save the meeting brief as a SharePoint doc.
+
+Then, send me an email with a daily summary of my upcoming meetings, with a link to the full brief.
+```
+
+You'll see the assistant create a plan for building the agent, then starting to build each component step-by-step. The first draft should give you a working agent with a name, description, instructions, and a set of connected apps.
+
+<p align="center">
+  <img src="https://cdn.openai.com/cookbook/chatgpt-agents-sales-meeting-prep/img1-v2.png" alt="Agent builder in progress" width="80%" />
+</p>
+
+If you don't have the required apps already configured, the agent builder will walk you through the configuration steps. If an app is unavailable, you can adjust scope - for example if you don't have Slack enabled, you can instruct your agent to send you an email with your sales meeting doc instead.
+
+### 2. Review and configure your apps
+
+Now we'll review the configurations of our apps. Because our agent can access external tools, it's important to understand the permissions it has to take actions on these tools, and how those permissions change as we share our agent for other teammates to use.
+
+Note that connector actions can be enforced at the workspace level by admins. For example, an admin can enforce that workspace agents can only take read actions, but not write actions, for the SharePoint connector.
+
+We can click into each connector to view and change its permissions.
+
+For the **Google Calendar** connection, we'll use *End-user account*. What this means is that each person who is using the agent will need to authenticate with their own google calendar, and the agent will access their individual meetings.
+
+<p align="center">
+  <img src="https://cdn.openai.com/cookbook/chatgpt-agents-sales-meeting-prep/img2.png" alt="End-user auth for Google Calendar" width="60%" />
+</p>
+
+For Google Calendar, we just need to view upcoming meetings and attendees, so we'll only enable read actions. We can also enforce permissions more granularly, by selecting individual actions to enable or disable.
+
+<p align="center">
+  <img src="https://cdn.openai.com/cookbook/chatgpt-agents-sales-meeting-prep/img3.png" alt="Read access for Google Calendar" width="60%" />
+</p>
+
+We'll also use the same *End-user account* configuration for the **Gmail** connector, since each agent user will get their own email briefing for their upcoming meetings.
+
+For the **Microsoft SharePoint** connection, we'll use an *Agent-owned account* linked to a service account associated with a sales team. What this means is that regardless of who is using the agent, the agent will have access to the full set of SharePoint folders, files, and sites that are accessible to the team service account.
+
+In this case, we want to be able to search through SharePoint for any documents relevant to a customer account, then create a new meeting prep document. So we'll scope the SharePoint connector permissions accordingly, to allow a broad set of read actions but prevent bulk write actions and prevent deletes.
+
+<p align="center">
+  <img src="https://cdn.openai.com/cookbook/chatgpt-agents-sales-meeting-prep/img4.png" alt="Scoped permissions for SharePoint" width="60%" />
+</p>
+
+Aside from the connectors we've already enabled, the agent has access to **Web search** that it will use to enrich the meeting briefs with recent public company news from the last 30 days.
+
+### 3. Enabling skills and memories for consistent, customized outputs
+
+One of the benefits of using workspace agents is how easy it is to ensure **high-quality and consistent** outputs, and skills are an ideal way to help our agent follow the same workflow each time it generates a meeting prep document.
+
+Skills package instructions, resources, and scripts to help our agent reliably complete its instructions. To equip our agent with a skill, we'll select "Add skill" in our agent view. We can upload a skill that we've already created (skills are built on the [open-source agent skills standard](https://agentskills.io/home) so you can use them across AI products), or we can use the built-in skill builder directly within ChatGPT workspace agents.
+
+<p align="center">
+  <img src="https://cdn.openai.com/cookbook/chatgpt-agents-sales-meeting-prep/img5.png" alt="Creating a skill for our agent" width="336" />
+</p>
+
+Our assistant will read the meeting template we've provided, and help us create a customized skill that matches the output format we want. In order to polish our skill, we can either click into the skill and edit it directly within the agent builder, and we can upload additional files that the agent can use when it decides to use the skill.
+
+<p align="center">
+  <img src="https://cdn.openai.com/cookbook/chatgpt-agents-sales-meeting-prep/img6.png" alt="Editing a skill for our agent" width="60%" />
+</p>
+
+We'll also enable **Memory** for our agent. Memory enables the agent to maintain context across different agent sessions, by giving it access to a persistent folder where the agent can save notes, drafts, and outputs. In this case, we'll give the agent memory for previous runs of this agent so it knows per-user context.
+
+Note that this memory is not shared across different end users of our agent. Users can view their memories from the agent page. For example, a user might give the agent guidance to ignore a certain meeting on its calendar that it doesn't need a meeting prep doc for, and the agent will be able to access that memory when following its instructions.
+
+<p align="center">
+  <img src="https://cdn.openai.com/cookbook/chatgpt-agents-sales-meeting-prep/img7.png" alt="Giving our agent access to memory" width="60%" />
+</p>
+
+## Testing your agent
+
+Before scheduling our agent to run on a periodic cadence, and before sharing it with other teammates, we can select **Preview** or **Try in ChatGPT** in the top-right corner of the agent builder to test our agent. We can also select **Create** to save our agent, and then invoke it by tagging it in our ChatGPT input box. Note that when you first create an agent, it is by default available only to you.
+
+Note that our agent builder has automatically created some sample starter prompts that appear next to the agent in ChatGPT. To view and edit these prompts, we can select ChatGPT under "Channels" in the agent view.
+
+<p align="center">
+  <img src="https://cdn.openai.com/cookbook/chatgpt-agents-sales-meeting-prep/img9.png" alt="Updating starter prompts" width="336" />
+</p>
+
+We'll ask our agent to prepare briefs for the meetings on our calendar for tomorrow, and we can view the traces of actions the agent is taking - accessing connectors, referencing skills and memories - as well as the agent's chain-of-thought reasoning as it completes its task.
+
+When our agent completes, we see that it has generated polished SharePoint documents for each of our upcoming customer meetings and sent us an email summary.
+
+<p align="center">
+  <img src="https://cdn.openai.com/cookbook/chatgpt-agents-sales-meeting-prep/img11.png" alt="Output email" width="80%" />
+</p>
+
+<p align="center">
+  <img src="https://cdn.openai.com/cookbook/chatgpt-agents-sales-meeting-prep/img12.png" alt="Output meeting brief" width="80%" />
+</p>
+
+You can run additional tests with different input data - for example, you might ask the agent to prepare a brief for a meeting that you already had last week, to compare its output with a human-generated prep doc. You can also add that prep doc to the skill's files as an example. You can update agent instructions, provide it additional connectors and permissions, or update its skill to refine the output it produces.
+
+## Deploying your agent
+
+We'll first set our agent to run each day at 4pm, to give us a preview of the upcoming day. We'll select **Schedule** in the top-right of the agent screen and then **Add new schedule** to specify when the agent should run, and provide it any specific instructions.
+
+For example, in addition to our daily weekday schedule, we might want to create a "weekly prep" schedule that runs each Sunday evening, and instruct our agent to prepare meeting briefs for each upcoming meeting this week.
+
+<p align="center">
+  <img src="https://cdn.openai.com/cookbook/chatgpt-agents-sales-meeting-prep/img10.png" alt="Creating a schedule" width="336" />
+</p>
+
+Now, we'll share our agent with our team. In the "Channels" section of the agent builder, we can select **ChatGPT** and then **Enable sharing**. (Note that your workspace admin will need to grant you permission to share agents you've created.)
+
+We can turn on sharing to generate a shareable link for our agent, which lets anyone in your workspace either chat directly with your agent, or duplicate your agent to create their own forked version which they can edit individually. (You won't be able to access their duplicated agent, unless they choose to share it with you. Currently, multiple users cannot edit the same agent.)
+
+Because we've selected *End-user account* for our Google Calendar and Gmail connectors, each person using our agent will need to log into these apps, and the agent will be able to access their specific meetings and email. However, because we've selected *Agent-owned account* for the SharePoint connector, **anyone we share the agent with will be able to access the SharePoint resources**.
+
+In addition to providing our teammates with a link to our agent, we can also choose to list our agent in our company directory, allowing anyone in your workspace to view the agent. You can access your company directory of employee-built agents by selecting **Agents** from ChatGPT, and then selecting the agent directory tab.

articles/codex_exec_plans.md

@@ -0,0 +1,176 @@
+# Using PLANS.md for multi-hour problem solving
+
+Codex and the `gpt-5.2-codex` model (recommended) can be used to implement complex tasks that take significant time to research, design, and implement. The approach described here is one way to prompt the model to implement these tasks and to steer it towards successful completion of a project.
+
+These plans are thorough design documents, and "living documents". As a user of Codex, you can use these documents to verify the approach that Codex will take before it begins a long implementation process. The particular `PLANS.md` included below is very similar to one that has enabled Codex to work for more than seven hours from a single prompt.
+
+We enable Codex to use these documents by first updating `AGENTS.md` to describe when to use `PLANS.md`, and then of course, to add the `PLANS.md` file to our repository.
+
+## `AGENTS.md`
+
+[`AGENTS.md`](https://github.com/openai/agents.md) is a simple format for guiding coding agents such as Codex. We describe a term that users can use as a shorthand and a simple rule for when to use planning documents. Here, we call it an "ExecPlan". Note that this is an arbitrary term, Codex has not been trained on it. This shorthand can then be used when prompting Codex to direct it to a particular definition of a plan.
+
+Here's an `AGENTS.md` section instructing an agent about when to use a plan:
+
+```md
+# ExecPlans
+
+When writing complex features or significant refactors, use an ExecPlan (as described in .agent/PLANS.md) from design to implementation.
+```
+
+## `PLANS.md`
+
+Below is the entire document. The prompting in this document was carefully chosen to provide significant amounts of feedback to users and to guide the model to implement precisely what a plan specifies. Users may find that they benefit from customizing the file to meet their needs, or to add or remove required sections.
+
+~~~md
+# Codex Execution Plans (ExecPlans):
+
+This document describes the requirements for an execution plan ("ExecPlan"), a design document that a coding agent can follow to deliver a working feature or system change. Treat the reader as a complete beginner to this repository: they have only the current working tree and the single ExecPlan file you provide. There is no memory of prior plans and no external context.
+
+## How to use ExecPlans and PLANS.md
+
+When authoring an executable specification (ExecPlan), follow PLANS.md _to the letter_. If it is not in your context, refresh your memory by reading the entire PLANS.md file. Be thorough in reading (and re-reading) source material to produce an accurate specification. When creating a spec, start from the skeleton and flesh it out as you do your research.
+
+When implementing an executable specification (ExecPlan), do not prompt the user for "next steps"; simply proceed to the next milestone. Keep all sections up to date, add or split entries in the list at every stopping point to affirmatively state the progress made and next steps. Resolve ambiguities autonomously, and commit frequently.
+
+When discussing an executable specification (ExecPlan), record decisions in a log in the spec for posterity; it should be unambiguously clear why any change to the specification was made. ExecPlans are living documents, and it should always be possible to restart from _only_ the ExecPlan and no other work.
+
+When researching a design with challenging requirements or significant unknowns, use milestones to implement proof of concepts, "toy implementations", etc., that allow validating whether the user's proposal is feasible. Read the source code of libraries by finding or acquiring them, research deeply, and include prototypes to guide a fuller implementation.
+
+## Requirements
+
+NON-NEGOTIABLE REQUIREMENTS:
+
+* Every ExecPlan must be fully self-contained. Self-contained means that in its current form it contains all knowledge and instructions needed for a novice to succeed.
+* Every ExecPlan is a living document. Contributors are required to revise it as progress is made, as discoveries occur, and as design decisions are finalized. Each revision must remain fully self-contained.
+* Every ExecPlan must enable a complete novice to implement the feature end-to-end without prior knowledge of this repo.
+* Every ExecPlan must produce a demonstrably working behavior, not merely code changes to "meet a definition".
+* Every ExecPlan must define every term of art in plain language or do not use it.
+
+Purpose and intent come first. Begin by explaining, in a few sentences, why the work matters from a user's perspective: what someone can do after this change that they could not do before, and how to see it working. Then guide the reader through the exact steps to achieve that outcome, including what to edit, what to run, and what they should observe.
+
+The agent executing your plan can list files, read files, search, run the project, and run tests. It does not know any prior context and cannot infer what you meant from earlier milestones. Repeat any assumption you rely on. Do not point to external blogs or docs; if knowledge is required, embed it in the plan itself in your own words. If an ExecPlan builds upon a prior ExecPlan and that file is checked in, incorporate it by reference. If it is not, you must include all relevant context from that plan.
+
+## Formatting
+
+Format and envelope are simple and strict. Each ExecPlan must be one single fenced code block labeled as `md` that begins and ends with triple backticks. Do not nest additional triple-backtick code fences inside; when you need to show commands, transcripts, diffs, or code, present them as indented blocks within that single fence. Use indentation for clarity rather than code fences inside an ExecPlan to avoid prematurely closing the ExecPlan's code fence. Use two newlines after every heading, use # and ## and so on, and correct syntax for ordered and unordered lists.
+
+When writing an ExecPlan to a Markdown (.md) file where the content of the file *is only* the single ExecPlan, you should omit the triple backticks.
+
+Write in plain prose. Prefer sentences over lists. Avoid checklists, tables, and long enumerations unless brevity would obscure meaning. Checklists are permitted only in the `Progress` section, where they are mandatory. Narrative sections must remain prose-first.
+
+## Guidelines
+
+Self-containment and plain language are paramount. If you introduce a phrase that is not ordinary English ("daemon", "middleware", "RPC gateway", "filter graph"), define it immediately and remind the reader how it manifests in this repository (for example, by naming the files or commands where it appears). Do not say "as defined previously" or "according to the architecture doc." Include the needed explanation here, even if you repeat yourself.
+
+Avoid common failure modes. Do not rely on undefined jargon. Do not describe "the letter of a feature" so narrowly that the resulting code compiles but does nothing meaningful. Do not outsource key decisions to the reader. When ambiguity exists, resolve it in the plan itself and explain why you chose that path. Err on the side of over-explaining user-visible effects and under-specifying incidental implementation details.
+
+Anchor the plan with observable outcomes. State what the user can do after implementation, the commands to run, and the outputs they should see. Acceptance should be phrased as behavior a human can verify ("after starting the server, navigating to [http://localhost:8080/health](http://localhost:8080/health) returns HTTP 200 with body OK") rather than internal attributes ("added a HealthCheck struct"). If a change is internal, explain how its impact can still be demonstrated (for example, by running tests that fail before and pass after, and by showing a scenario that uses the new behavior).
+
+Specify repository context explicitly. Name files with full repository-relative paths, name functions and modules precisely, and describe where new files should be created. If touching multiple areas, include a short orientation paragraph that explains how those parts fit together so a novice can navigate confidently. When running commands, show the working directory and exact command line. When outcomes depend on environment, state the assumptions and provide alternatives when reasonable.
+
+Be idempotent and safe. Write the steps so they can be run multiple times without causing damage or drift. If a step can fail halfway, include how to retry or adapt. If a migration or destructive operation is necessary, spell out backups or safe fallbacks. Prefer additive, testable changes that can be validated as you go.
+
+Validation is not optional. Include instructions to run tests, to start the system if applicable, and to observe it doing something useful. Describe comprehensive testing for any new features or capabilities. Include expected outputs and error messages so a novice can tell success from failure. Where possible, show how to prove that the change is effective beyond compilation (for example, through a small end-to-end scenario, a CLI invocation, or an HTTP request/response transcript). State the exact test commands appropriate to the project’s toolchain and how to interpret their results.
+
+Capture evidence. When your steps produce terminal output, short diffs, or logs, include them inside the single fenced block as indented examples. Keep them concise and focused on what proves success. If you need to include a patch, prefer file-scoped diffs or small excerpts that a reader can recreate by following your instructions rather than pasting large blobs.
+
+## Milestones
+
+Milestones are narrative, not bureaucracy. If you break the work into milestones, introduce each with a brief paragraph that describes the scope, what will exist at the end of the milestone that did not exist before, the commands to run, and the acceptance you expect to observe. Keep it readable as a story: goal, work, result, proof. Progress and milestones are distinct: milestones tell the story, progress tracks granular work. Both must exist. Never abbreviate a milestone merely for the sake of brevity, do not leave out details that could be crucial to a future implementation.
+
+Each milestone must be independently verifiable and incrementally implement the overall goal of the execution plan.
+
+## Living plans and design decisions
+
+* ExecPlans are living documents. As you make key design decisions, update the plan to record both the decision and the thinking behind it. Record all decisions in the `Decision Log` section.
+* ExecPlans must contain and maintain a `Progress` section, a `Surprises & Discoveries` section, a `Decision Log`, and an `Outcomes & Retrospective` section. These are not optional.
+* When you discover optimizer behavior, performance tradeoffs, unexpected bugs, or inverse/unapply semantics that shaped your approach, capture those observations in the `Surprises & Discoveries` section with short evidence snippets (test output is ideal).
+* If you change course mid-implementation, document why in the `Decision Log` and reflect the implications in `Progress`. Plans are guides for the next contributor as much as checklists for you.
+* At completion of a major task or the full plan, write an `Outcomes & Retrospective` entry summarizing what was achieved, what remains, and lessons learned.
+
+# Prototyping milestones and parallel implementations
+
+It is acceptable—-and often encouraged—-to include explicit prototyping milestones when they de-risk a larger change. Examples: adding a low-level operator to a dependency to validate feasibility, or exploring two composition orders while measuring optimizer effects. Keep prototypes additive and testable. Clearly label the scope as “prototyping”; describe how to run and observe results; and state the criteria for promoting or discarding the prototype.
+
+Prefer additive code changes followed by subtractions that keep tests passing. Parallel implementations (e.g., keeping an adapter alongside an older path during migration) are fine when they reduce risk or enable tests to continue passing during a large migration. Describe how to validate both paths and how to retire one safely with tests. When working with multiple new libraries or feature areas, consider creating spikes that evaluate the feasibility of these features _independently_ of one another, proving that the external library performs as expected and implements the features we need in isolation.
+
+## Skeleton of a Good ExecPlan
+
+    # <Short, action-oriented description>
+
+    This ExecPlan is a living document. The sections `Progress`, `Surprises & Discoveries`, `Decision Log`, and `Outcomes & Retrospective` must be kept up to date as work proceeds.
+
+    If PLANS.md file is checked into the repo, reference the path to that file here from the repository root and note that this document must be maintained in accordance with PLANS.md.
+
+    ## Purpose / Big Picture
+
+    Explain in a few sentences what someone gains after this change and how they can see it working. State the user-visible behavior you will enable.
+
+    ## Progress
+
+    Use a list with checkboxes to summarize granular steps. Every stopping point must be documented here, even if it requires splitting a partially completed task into two (“done” vs. “remaining”). This section must always reflect the actual current state of the work.
+
+    - [x] (2025-10-01 13:00Z) Example completed step.
+    - [ ] Example incomplete step.
+    - [ ] Example partially completed step (completed: X; remaining: Y).
+
+    Use timestamps to measure rates of progress.
+
+    ## Surprises & Discoveries
+
+    Document unexpected behaviors, bugs, optimizations, or insights discovered during implementation. Provide concise evidence.
+
+    - Observation: …
+      Evidence: …
+
+    ## Decision Log
+
+    Record every decision made while working on the plan in the format:
+
+    - Decision: …
+      Rationale: …
+      Date/Author: …
+
+    ## Outcomes & Retrospective
+
+    Summarize outcomes, gaps, and lessons learned at major milestones or at completion. Compare the result against the original purpose.
+
+    ## Context and Orientation
+
+    Describe the current state relevant to this task as if the reader knows nothing. Name the key files and modules by full path. Define any non-obvious term you will use. Do not refer to prior plans.
+
+    ## Plan of Work
+
+    Describe, in prose, the sequence of edits and additions. For each edit, name the file and location (function, module) and what to insert or change. Keep it concrete and minimal.
+
+    ## Concrete Steps
+
+    State the exact commands to run and where to run them (working directory). When a command generates output, show a short expected transcript so the reader can compare. This section must be updated as work proceeds.
+
+    ## Validation and Acceptance
+
+    Describe how to start or exercise the system and what to observe. Phrase acceptance as behavior, with specific inputs and outputs. If tests are involved, say "run <project’s test command> and expect <N> passed; the new test <name> fails before the change and passes after>".
+
+    ## Idempotence and Recovery
+
+    If steps can be repeated safely, say so. If a step is risky, provide a safe retry or rollback path. Keep the environment clean after completion.
+
+    ## Artifacts and Notes
+
+    Include the most important transcripts, diffs, or snippets as indented examples. Keep them concise and focused on what proves success.
+
+    ## Interfaces and Dependencies
+
+    Be prescriptive. Name the libraries, modules, and services to use and why. Specify the types, traits/interfaces, and function signatures that must exist at the end of the milestone. Prefer stable names and paths such as `crate::module::function` or `package.submodule.Interface`. E.g.:
+
+    In crates/foo/planner.rs, define:
+
+        pub trait Planner {
+            fn plan(&self, observed: &Observed) -> Vec<Action>;
+        }
+
+If you follow the guidance above, a single, stateless agent -- or a human novice -- can read your ExecPlan from top to bottom and produce a working, observable result. That is the bar: SELF-CONTAINED, SELF-SUFFICIENT, NOVICE-GUIDING, OUTCOME-FOCUSED.
+
+When you revise a plan, you must ensure your changes are comprehensively reflected across all sections, including the living document sections, and you must write a note at the bottom of the plan describing the change and the reason why. ExecPlans must describe not just the what but the why for almost everything.
+~~~

articles/gpt-oss-safeguard-guide.md

@@ -0,0 +1,570 @@
+# User guide for gpt-oss-safeguard
+
+## Introduction & Overview
+
+ROOST and OpenAI have prepared a guide that explains how to write policy prompts that maximize [gpt-oss-safeguard's](https://github.com/openai/gpt-oss-safeguard) reasoning power, choose the right policy length for deep analysis, and integrate oss-safeguard's reasoning outputs into production Trust & Safety systems.
+
+### What is gpt-oss-safeguard?
+
+gpt-oss-safeguard is a first open weight reasoning model specifically trained for safety classification tasks to help classify text content based on customizable policies. As a fine-tuned version of [gpt-oss](https://openai.com/index/introducing-gpt-oss/), gpt-oss-safeguard is designed to follow explicit written policies that you provide. This enables **bring-your-own-policy** Trust & Safety AI, where your own taxonomy, definitions, and thresholds guide classification decisions. Well crafted policies unlock gpt-oss-safeguard's reasoning capabilities, enabling it to handle nuanced content, explain borderline decisions, and adapt to contextual factors.
+
+You can read more about how OpenAI uses the internal version of gpt-oss-safeguard [here](https://openai.com/index/introducing-gpt-oss-safeguard/).
+
+Large language models can be considered safety models in two main ways:
+
+- Fine-tuned safety models start as general reasoning models (like gpt-oss) and are trained to respond safely within user interactions.
+- Prebaked safety models (like ShieldGemma, LlamaGuard, RoGuard, etc) come with built-in definitions of what counts as “unsafe” and fixed policy taxonomies.
+
+gpt-oss-safeguard was purpose-built for Trust & Safety workflows and is a policy-following model that can reliably interpret and enforce **your own written standards and tell you why it made the decision it made**. The reasoning behind the model makes it well-suited for integration with a larger safety system that is rooted in auditability and customization.
+
+### How to Use gpt-oss-safeguard
+
+Like the [gpt-oss family of models](https://openai.com/open-models/), this is an open source model with open weights that you run locally or integrate into your own infrastructure. It is designed to work with the [harmony response format](https://github.com/openai/harmony). Harmony is the structured prompt interface that gives gpt-oss-safeguard access to its full reasoning stack and ensures consistent, well-formed outputs.
+
+The gpt-oss family of models, including gpt-oss-safeguard, can be run on servers using:
+
+- [vLLM](https://docs.vllm.ai/projects/recipes/en/latest/OpenAI/GPT-OSS.html#gpt-oss-vllm-usage-guide) (for dedicated GPUs like NVIDIA’s H100s)
+- [HuggingFace Transformers](https://cookbook.openai.com/articles/gpt-oss/run-locally-lmstudio) (for consumer GPUs)
+- [Google Colab](https://cookbook.openai.com/articles/gpt-oss/run-colab)
+
+And locally using:
+
+- [LM Studio](https://cookbook.openai.com/articles/gpt-oss/run-locally-lmstudio)
+- [Ollama](https://cookbook.openai.com/articles/gpt-oss/run-locally-ollama)
+
+### Who Should Use gpt-oss-safeguard?
+
+gpt-oss-safeguard is designed for users who need real-time context and automation at scale, including:
+
+- **ML/AI Engineers** working on Trust & Safety systems who need flexible content moderation
+- **Trust & Safety Engineers** building or improving moderation, Trust & Safety, or platform integrity pipelines
+- **Technical Program Managers** overseeing content safety initiatives
+- **Developers** building projects/applications that require contextual, policy-based content moderation
+- **Policy Crafters** defining what is accepted by an organization who want to test out policy lines, generate examples, and evaluate content
+
+Safety-tuned models excel at content moderation when given clear, structured prompts. This guide covers key learnings from deploying moderation systems in production, focusing on prompt structure, output formatting, and length optimization.
+
+### Using gpt-oss-safeguards with HuggingFace Transformers
+
+The Transformers library by Hugging Face provides a flexible way to load and run large language models locally or on a server. [This guide](https://cookbook.openai.com/articles/gpt-oss/run-transformers) takes you through running [OpenAI gpt-oss](https://huggingface.co/openai/gpt-oss-20b) models using Transformers, either with a high-level pipeline or via low-level generate calls with raw token IDs. The simplest way to interact with the server is through the transformers chat CLI
+
+```bash
+transformers chat localhost:8000 --model-name-or-path openai/gpt-oss-safeguard-20b
+```
+
+or by sending an HTTP request with cURL, e.g.
+
+```bash
+curl http://localhost:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "openai/gpt-oss-safeguard-20b",
+    "stream": true,
+    "messages": [
+      { "role": "system", "content": "<your policy>" },
+      { "role": "user", "content": "<user content to verify>" }
+    ]
+  }'
+
+
+```
+
+Additional use cases, like integrating transformers serve with Cursor and other tools, are detailed in [the documentation](https://huggingface.co/docs/transformers/main/serving).
+
+### Running gpt-oss-safeguard with Ollama
+
+[Ollama](https://ollama.com/download) supports gpt-oss-safeguard 20B and 120B models directly. The following commands will automatically download the model and run it on your device.
+
+#### gpt-oss-safeguard:20b
+
+```bash
+ollama run gpt-oss-safeguard:20b
+```
+
+#### gpt-oss-safeguard:120b
+
+```bash
+ollama run gpt-oss-safeguard:120b
+```
+
+Ollama supports [OpenAI API](https://docs.ollama.com/api/openai-compatibility), [Ollama's API](https://docs.ollama.com/api), [Python](https://github.com/ollama/ollama-python) and [JavaScript](https://github.com/ollama/ollama-js) SDKs for building applications or tools using the gpt-oss-safeguard models. Please learn more from [Ollama's documentation](https://docs.ollama.com/).
+
+### Running gpt-oss-safeguard with LM Studio
+
+Alternatively, you can use [LM Studio](https://lmstudio.ai/) to run the models locally including using [OpenAI Chat Completions](https://lmstudio.ai/docs/developer/openai-compat/chat-completions) and [Responses API](https://lmstudio.ai/docs/developer/openai-compat/responses) compatible APIs. Head over to the [gpt-oss-safeguard page for LM Studio](https://lmstudio.ai/models/gpt-oss-safeguard) or run the following commands to download the respective models:
+
+#### gpt-oss-safeguard-20b
+
+```bash
+lms get openai/gpt-oss-safeguard-20b
+```
+
+#### gpt-oss-safeguard-120b
+
+```bash
+lms get openai/gpt-oss-safeguard-120b
+```
+
+### Running gpt-oss-safeguard with vLLM
+
+[vLLM](https://docs.vllm.ai/) recommends using [uv](https://docs.astral.sh/uv/) for Python dependency management. The following command will automatically download the model and start the server.
+
+```shell
+uv pip install vllm==0.10.2 --torch-backend=auto
+
+vllm serve openai/gpt-oss-safeguard-120b
+```
+
+[Learn more about how to use gpt-oss with vLLM.](https://docs.vllm.ai/projects/recipes/en/latest/OpenAI/GPT-OSS.html#gpt-oss-vllm-usage-guide)
+
+### Understanding the Harmony Response Format
+
+gpt-oss-safeguard uses the [harmony prompt format](https://cookbook.openai.com/articles/openai-harmony) to provide a structured output and provide reasoning. This is critical for Trust & Safety workflows where you need to understand and audit why a decision or classification was made. With the harmony format, oss-safeguard separates its response into two parts:
+
+1. **Reasoning channel:** Where the model reasons through the policy, considers edge cases, and explains its logic
+2. **Output channel**: The formatted classification decision you specified
+
+Through harmony, you can control how deeply oss-safeguard reasons by setting the `reasoning_effort` parameter in your system message to `low`, `medium`, or `high`. The model uses `medium` by default if it is not set. Higher reasoning effort allows oss-safeguard to consider more factors, trace through multiple policy sections, and handle complex interactions between rules. Lower effort provides faster responses for straightforward classifications.
+
+If you're using [**vLLM**](https://docs.vllm.ai/en/latest/) (recommended for most users) or another inference solution that provides chat message inputs, the harmony format is applied automatically when you format requests as [chat messages](https://docs.vllm.ai/en/v0.7.0/getting_started/examples/chat.html):
+
+- **System message:** Your policy prompt (include Reasoning: high or similar in the system message to control reasoning effort).
+- **User message:** The content to classify.
+
+## How oss-safeguard uses Policy Prompts
+
+oss-safeguard is designed to use your written policy as its governing logic. While most models provide a confidence score based on the features it was trained on and require retraining for any policy changes, oss-safeguard makes decisions backed by reasoning within the boundaries of a provided taxonomy. This feature lets T\&S teams deploy oss-safeguard as a policy-aligned reasoning layer within existing moderation or compliance systems. This also means that you can update or test new policies instantly without retraining the entire model.
+
+## Writing Effective Policy Prompts for gpt-oss-safeguard
+
+oss-safeguard performs best when policies are organized like a Trust & Safety policy guide rather than an essay. If you already have a set of policies, you’ll be in great shape. Use headers and clear categories so the model can navigate definitions efficiently. If you’ve written policy for teams before, this should feel familiar.
+
+### Understanding Policy Prompting
+
+A policy prompt defines the operational boundaries of a model’s behavior. Similar to content or platform policies written for human reviewers, policies for oss-safeguard should clearly specify what constitutes a violation, what is allowed, and how to communicate that difference into a decision that flows into the rest of the Trust & Safety system.
+
+Effective policy prompts are structured in order to distinguish between similar content types, catch subtle, coded or indirect violations, and prevent false positives on edge cases. Think of it as combining a policy document with training examples.
+
+### Structuring Policy Prompts
+
+Policy prompts should have four separate sections.
+
+1. **Instruction:** what the model MUST do and how the model should answer.
+2. **Definitions:** concise explanations of key terms.
+3. **Criteria:** distinctions between violating and non-violating content.
+4. **Examples:** short, concrete instances near the decision boundary. It’s important to have both examples of what you want to classify, and what you do not want to classify
+
+Because oss-safeguard is tuned for structured moderation, it expects explicit instructions for how to respond. A policy prompt will likely perform better if it follows a consistent pattern that includes the expected format for the response and output. The harmony format’s structured channels allow oss-safeguard to reason through these sections before emitting only the final label:
+
+```markdown
+# Policy Name
+
+## INSTRUCTIONS
+
+Describe what oss-safeguard should do and how to respond.
+
+## DEFINITIONS
+
+Clarify key terms and context.
+
+## VIOLATES (1)
+
+Describe behaviors or content that should be flagged.
+
+## SAFE (0)
+
+Describe content that should not be flagged.
+
+## EXAMPLES
+
+Provide 4–6 short examples labeled 0 or 1.
+
+Content: [INPUT]
+Answer (0 or 1):
+```
+
+To reduce the likelihood of false positives or confusion, avoid using words like “generally” or “usually”. If there are situations where there’s ambiguity, add an escalation path for manual review. This is also especially helpful for regional or language differences.
+
+Be explicit about priority and precedence so the model understands which policy wins if there is a conflict. If there are multiple policy violations, define which one is dominant.
+
+### Choosing the Right Policy Length
+
+Policy length is a key control over how deeply gpt-oss-safeguard can reason about your rules. Longer policies add nuance to handle complex cases, but can impact the output and responses. When using the harmony response format, the model can process longer policies more reliably because reasoning happens in the hidden analysis channel, not in the visible final output.
+
+Use [https://platform.openai.com/tokenizer](https://platform.openai.com/tokenizer) to determine the length of your prompt. **gpt-oss-safeguard can provide a reasonable output at \~10,000 token policies, but early testing suggests the optimal range is between 400-600 tokens**. It’s important to experiment and see what works best for you as there is no one-size-fits-all approach. Think of the policy length like a “context budget.” Too short, and the model lacks detail; too long, and the model risks confusion. This is similar to writing policy for people to understand as well. In the same way you should account for giving the model enough output tokens to generate a response. Since the model is using reasoning you should leave plenty of room for output tokens and ideally not cap the maximum output tokens to give the model enough room to reason through the policies. If you want to limit the reasoning time, consider setting the reasoning effort to low instead.
+
+If you have a longer policy with multiple categories, consider pre‑compressing each policy to 300–600 tokens (including definitions, disallowed categories, and 1-2 examples each for violations and non-violations).
+
+oss-safeguard can also evaluate multiple policies simultaneously, so long as all policies are included in the prompt. We have found that additional policies lead to small but meaningful degradations in accuracy, so we recommend experimenting with adding and removing policies if using oss-safeguard for multiple harm domains.
+
+## Designing reliable output instructions
+
+Consistent responses from gpt-oss-safeguard require explicit, literal output instructions. Every policy prompt should state exactly how the model must respond and demonstrate the correct and incorrect patterns. The output instructions define how gpt-oss-safeguard communicates its final decision and determines whether outputs can be relied upon. Because gpt-oss-safeguard operates within the harmony response format, all output instructions must:
+
+1. **Explicitly define the output format**: specify exactly what the model should return (e.g., `0`/`1`, JSON object, category label list).
+2. **Include policy references when applicable**: if your workflow tracks enforcement by category or rule, require the model to return that field; for simple binary output, this can be omitted.
+3. **Be reinforced throughout the policy**: repeat the output instruction at least once near the top (in “INSTRUCTIONS”) and again near the bottom (before “EXAMPLES”) to fortify compliance during reasoning.
+
+### Binary Responses
+
+Binary output limits gpt-oss-safeguard's reasoning to a simple yes/no decision. Use this when speed matters more than understanding why the decision was made, but recognize you're not leveraging gpt-oss-safeguard's core reasoning strength.
+
+```markdown
+Return exactly one character: 0 or 1.
+Do not include any explanation or punctuation.
+
+0 = Content does NOT violate this policy.
+1 = Content violates this policy.
+```
+
+### Policy-referencing outputs
+
+Category labels encourage gpt-oss-safeguard to reason about which section of your policy applies, but don't require detailed explanation of why. This format enables basic reasoning transparency while keeping outputs concise.
+
+```
+If the content violates this policy, return:
+{"violation": 1, "policy_category": "<category_or_rule_id>"}
+
+If the content does NOT violate this policy, return:
+{"violation": 0, "policy_category": null}
+
+Example:
+{"violation": 1, "policy_category": "H2.f"}
+```
+
+### Including Rationale
+
+One of the most powerful features of gpt-oss-safeguard is its ability to think and reason. The model must not only classify content, but trace its logic through your policy, identify which specific rules apply, and articulate why. When you require rationales, gpt-oss-safeguard reasons more carefully. It must consider multiple policy sections, evaluate how they interact, and construct a logical explanation. This deeper reasoning often catches nuances that simpler output formats miss. This output format maximizes gpt-oss-safeguard's reasoning capabilities.
+
+Ask the model to decide, then justify briefly. Ask for a short, non-step-by-step rationale (2–4 bullets or 1–2 sentences) and consider requiring policy citations (rule IDs/sections) so the model justifies its thinking and decision.
+
+```json
+{
+  "violation": 1,
+  "policy_category": "H2.f",
+  "rule_ids": ["H2.d", "H2.f"],
+  "confidence": "high",
+  "rationale": "Content compares a protected class to animals, which is dehumanizing."
+}
+```
+
+## Using gpt-oss-safeguard for Trust & Safety
+
+Because gpt-oss-safeguard interprets written rules rather than static categories, gpt-oss-safeguard adapts to different product, regulatory, and community contexts with minimal engineering overhead.
+
+gpt-oss-safeguard is designed to fit into Trust & Safety teams’ infrastructure. However, since gpt-oss-safeguard may be more time and compute intensive than other classifiers, consider pre-filtering content that is sent to gpt-oss-safeguard. [OpenAI uses small, high-recall classifiers to determine if content is domain-relevant to priority risks before evaluating that content with gpt-oss-safeguard.](https://openai.com/index/introducing-gpt-oss-safeguard/) You should consider two main things when deciding when and where to integrate oss-safeguard in your T\&S stack:
+
+1. Traditional classifiers have lower latency and cost less to sample from than gpt-oss-safeguard
+2. Traditional classifiers trained on thousands of examples will likely perform better on a task than gpt-oss-safeguard
+
+### Automated content classification
+
+Use gpt-oss-safeguard to label posts, messages, or media metadata for policy violations. Its policy reasoning supports nuanced classification to determine contextual details when making a decision. gpt-oss-safeguard can be integrated with:
+
+- Real-time ingestion pipelines
+- Review queues and moderation consoles
+- Downranking or filtering systems
+
+### T\&S Assistant
+
+gpt-oss-safeguard's reasoning capabilities make it uniquely suited for automated triage in Trust & Safety workflows. Unlike traditional classifiers that only provide labels and confidence scores, gpt-oss-safeguard acts as a reasoning agent that evaluates content, explains its decision, cites specific policy rules, and surfaces cases requiring human judgment. This can reduce the cognitive load on human moderators while increasing trust and transparency in automated decisions.
+
+### Policy Testing
+
+Before rolling out a new or revised policy, run it through gpt-oss-safeguard to simulate how content will be labeled. This can be helpful to identify overly broad definitions, unclear examples, and borderline cases.
+
+### Policy Experimentation
+
+gpt-oss-safeguard’s bring-your-own-policy design allows policy teams to A/B test alternative definitions directly in production without model retraining.
+
+## Off-the-shelf Teen Safety Policies
+
+While safety classifiers like gpt-oss-safeguard can detect harmful content, they depend on clear definitions of what that content is. In practice, one of the biggest challenges developers face is defining policies that accurately capture teen-specific risks and can be consistently applied in real systems.
+
+To address this challenge, OpenAI has released the [teen-safety-policy-pack](https://github.com/openai/teen-safety-policy-pack), a set of safety policies tailored to common risks faced by teens and informed by careful review of existing research on teens' unique developmental differences. These policies are structured as prompts that can be directly used with [gpt-oss-safeguard](https://github.com/openai/gpt-oss-safeguard) and other reasoning models, enabling developers to apply consistent safety standards across their systems more easily.
+
+The initial release includes policies covering:
+
+- Graphic violent content
+- Graphic sexual content
+- Harmful body ideals and behaviors
+- Dangerous activities and challenges
+- Romantic or violent roleplay
+- Age-restricted goods and services
+
+These policies can be used for real-time content filtering, as well as offline analysis of user-generated content.
+
+### Example Dangerous Content Policy
+
+```markdown
+# Dangerous Content Policy (#DC)
+
+## Goal
+
+Identify dangerous activity content that is not appropriate for teenagers. Classify each example as `VALID` (allowed) or `INVALID` (dangerous content) using this policy.
+
+This policy focuses on activities that could cause physical harm if attempted, especially viral challenges, risky stunts, unsafe experimentation, and dangerous at-home procedures.
+
+The assistant must not encourage, prompt, instruct, simulate, or facilitate teen participation in dangerous activities, stunts, or challenges. This includes glorifying the activity, providing instructions, realistic simulations, or linking to materials needed to attempt the challenge when user intent is present.
+
+## Definitions
+
+- **Dangerous activities or challenges**: Activities, stunts, or online challenges that could realistically cause physical injury, poisoning, suffocation, overdose, burns, falls, or other serious harm if attempted by a teenager.
+- Examples include risky viral trends such as substance misuse challenges, choking challenges, extreme stunts, and unsafe experimentation.
+- **Facilitation**: Providing instructions, tactics, materials, preparation steps, or optimization that would enable someone to attempt a dangerous activity.
+- **Glorification**: Portraying a dangerous activity as exciting, admirable, funny, or desirable in ways that could encourage imitation.
+- **Realistic simulation**: Detailed descriptions or narrative walkthroughs of performing a dangerous activity that functionally model how to do it.
+
+## Allowed Content (DC0 - Non-dangerous or Safety-oriented)
+
+Content that discourages risk, provides safety guidance, or discusses challenges critically or educationally without enough detail to mimic.
+
+Examples:
+
+- **DC0.a Media literacy**: "Why do viral internet challenges sometimes lead to injuries?"; "Viral internet challenges can lead to injuries because they often prioritize attention and shock value over safety."
+- **DC0.b Safety framing**: "Why is the Benadryl Challenge dangerous?"; "The Benadryl Challenge is dangerous because it involves taking more medication than recommended, which can cause serious health effects. Medications should only be used as directed."
+- **DC0.c Refusal or discouragement**: "That challenge can be very dangerous and isn't something I can help with."
+- **DC0.d News reporting without replicable detail**: "Summarize the news about injuries caused by viral TikTok challenges."; "Recent reports have highlighted multiple cases where teens were injured after attempting viral TikTok challenges. Health officials and educators have raised concerns about the risks."
+- **DC0.e Safety advice**: "If someone took too much medication, they should contact poison control or seek medical help."
+
+Output: `VALID`
+
+## Promotion or Glorification of Dangerous Challenges (DC1 - Promotion)
+
+Content that celebrates, encourages, or positively frames participation in dangerous activities without giving explicit instructions.
+
+Examples:
+
+- **DC1.a Glorification**: "The blackout challenge is wild, people pass out and wake up laughing."
+- **DC1.b Encouragement**: "You should totally try the Benadryl challenge, it's trending everywhere."
+- **DC1.c Social pressure**: "Everyone at school is doing it, don't be the only one who's scared."
+- **DC1.d Competitive framing**: "See how long you can hold your breath before you pass out. Compare with your friends."
+
+Output: `INVALID`
+
+## Facilitating Dangerous Activities (DC2 - Facilitation or instruction)
+
+Content that enables or instructs someone to perform dangerous challenges or stunts.
+
+Examples:
+
+- **DC2.a Instructions**: "Take 10 Benadryl tablets and wait for the hallucinations to start."
+- **DC2.b Preparation guidance**: "To do the blackout challenge safely, sit down first and breathe rapidly."
+- **DC2.c Optimization**: "The trick to the choking challenge is cutting off oxygen just long enough to pass out."
+- **DC2.d Material sourcing**: "You can buy Benadryl cheaply at this pharmacy before trying the challenge."
+- **DC2.e Realistic simulation**: "First you hold your breath, then your friend presses on your chest until you pass out."
+- **DC2.f Challenge escalation**: "Make it more extreme." or "How do I last longer?"
+
+Output: `INVALID`
+
+## Dangerous At-Home Procedures (DC3 - Facilitation or instruction)
+
+Content that enables or instructs teenagers to perform medical, cosmetic, or body-modification procedures at home without professional supervision.
+
+These activities may appear common online but can cause infection, nerve damage, bleeding, poisoning, burns, or other serious harm when attempted by minors without trained supervision.
+
+Examples:
+
+- **DC3.a DIY piercing instructions**: "You can pierce your ear at home by sterilizing a needle and pushing it through the lobe."
+- **DC3.b Preparation guidance**: "To pierce your nose yourself, numb it with ice and mark the spot before inserting the needle."
+- **DC3.c Optimization tips**: "Use a hollow needle instead of a safety pin so the piercing heals faster."
+- **DC3.d Material sourcing**: "Buy a piercing kit online and do the piercing yourself in the bathroom mirror."
+- **DC3.e Other at-home procedures**: "You can remove a mole yourself with dental floss or a razor blade."
+
+Output: `INVALID`
+
+## Label Format
+
+Each item gets one label indicating the presence and type of dangerous content.
+
+- `DC0`: Allowed or safety-oriented discussion.
+- `DC1`: Promotion or glorification.
+- `DC2`: Facilitation, instruction, sourcing, or realistic simulation of dangerous activities.
+- `DC3`: Dangerous at-home procedures.
+
+Examples:
+
+- "Why is the Benadryl challenge dangerous?" -> `DC0`
+- "The choking challenge is hilarious to watch." -> `DC1`
+- "Take a bunch of Benadryl pills to hallucinate." -> `DC2`
+- "Where can I buy Benadryl to try the challenge?" -> `DC2`
+- "Where can I buy Benadryl?" -> `DC0`
+- "It's technically possible to pierce an ear with a needle, but here are the safety-first steps you must follow." -> `DC3`
+
+## Ambiguity and Escalation
+
+- If the user shows clear intent to attempt a dangerous challenge, treat enabling responses as `DC2`.
+- If a response celebrates or encourages participation without instructions, classify it as `DC1`.
+- If the content overlaps with self-harm, substance misuse, or criminal activity, escalate to the relevant policy domain.
+- Educational or safety-focused discussion of dangerous trends remains `DC0`.
+```
+
+## Integrating gpt-oss-safeguard with ROOST’s Tools
+
+### Osprey
+
+[Osprey](https://github.com/roostorg/osprey) is ROOST’s open-source rules engine and investigation framework. It evaluates real-time events against configurable logic trees and dispatches actions you define. Rules alone handle deterministic cases well (e.g., keyword matches, metadata thresholds), but they can struggle with satire, coded language, or nuanced policy boundaries. By integrating gpt-oss-safeguard, Osprey can:
+
+- **Add contextual reasoning**: gpt-oss-safeguard interprets edge cases that simple conditions cannot.
+- **Enforce policies directly**: gpt-oss-safeguard reads and applies your written policy text, ensuring consistency with human moderation.
+- **Maintain auditability**: Osprey logs which rule called gpt-oss-safeguard, which policy category was returned, and the model’s rationale.
+- **Blend automation and human oversight**: deterministic rules trigger fast actions; gpt-oss-safeguard handles reasoning before escalation to manual review in other tools
+
+gpt-oss-safeguard should not be called on every event. It can instead be a plugin that can be called only when there are ambiguous cases that need the policy-reasoning power of gpt-oss-safeguard.
+
+## Example Policy Prompts
+
+Try using any policy you’ve already written, or use this template and fill in what you’re trying to classify.
+
+```markdown
+## Policy Definitions
+
+### Key Terms
+
+**[Term 1]**: [Definition]
+
+**[Term 2]**: [Definition]
+
+**[Term 3]**: [Definition]
+
+## Content Classification Rules
+
+### VIOLATES Policy (Label: 1)
+
+Content that:
+
+- [Violation 1]
+- [Violation 2]
+- [Violation 3]
+- [Violation 4]
+- [Violation 5]
+
+### DOES NOT Violate Policy (Label: 0)
+
+Content that is:
+
+- [Acceptable 1]
+- [Acceptable 2]
+- [Acceptable 3]
+- [Acceptable 4]
+- [Acceptable 5]
+
+## Examples
+
+### Example 1 (Label: 1)
+
+**Content**: "[Example]"
+
+**Expected Response**:
+
+### Example 2 (Label: 1)
+
+**Content**: "[Example]"
+
+**Expected Response**:
+
+### Example 3 (Label: 0)
+
+**Content**: "[Example]"
+
+**Expected Response**:
+
+### Example 4 (Label: 0)
+
+**Content**: "[Example]"
+
+**Expected Response**:
+```
+
+You can also use this example prompt about spam as a reference:
+
+```markdown
+**Spam Policy (#SP)**
+**GOAL:** Identify spam. Classify each EXAMPLE as VALID (no spam) or INVALID (spam) using this policy.
+
+**DEFINITIONS**
+
+- **Spam**: unsolicited, repetitive, deceptive, or low-value promotional content.
+
+- **Bulk Messaging:** Same or similar messages sent repeatedly.
+
+- **Unsolicited Promotion:** Promotion without user request or relationship.
+
+- **Deceptive Spam:** Hidden or fraudulent intent (fake identity, fake offer).
+
+- **Link Farming:** Multiple irrelevant or commercial links to drive clicks.
+
+**Allowed Content (SP0 – Non-Spam or very low confidence signals of spam)**
+Content that is useful, contextual, or non-promotional. May look spammy but could be legitimate.
+
+- **SP0.a Useful/info request** – “How do I upload a product photo?”
+
+- **SP0.b Personalized communication** – “Hi Sam, here is the report.”
+
+- **SP0.c Business support** – “Can you fix my order?”
+
+- **SP0.d Single contextual promo** – “Thanks for subscribing—here’s your welcome guide.”
+
+- **SP0.e Generic request** – “Please respond ASAP.”
+
+- **SP0.f Low-quality formatting** – “HeLLo CLICK here FAST.”
+
+- **SP0.g Vague benefit statement** – “This tool changes lives.”
+
+  **Output:** VALID either clearly non-spam or very low confidence signals content could be spam.
+
+**Likely Spam (SP2 – Medium Confidence)**
+Unsolicited promotion without deception.
+
+- **SP2.a Cold promotion** – “Check out my Shopify course: shopcoach.biz”
+
+- **SP2.b Irrelevant ads** – “Buy gold rings here!”
+
+- **SP2.c Excessive linking** – “[http://x.com](http://x.com) http://y.com http://z.com”
+  **Output:** INVALID
+
+**High-Risk Spam (SP3 – Strong Confidence)**
+Spam showing scaling, automation, or aggressive tactics.
+
+- **SP3.a Repetitive posting** – “JOIN NOW JOIN NOW JOIN NOW”
+
+- **SP3.b Pyramid/affiliate bait** – “Earn $$$ by recruiting members”
+
+- **SP3.c Fake engagement** – “Comment ‘YES’ to win $500 gift card”
+
+- **SP3.d Misleading claims** – “Guaranteed 10,000 new followers overnight\!”
+  **Output:** INVALID
+
+**Malicious Spam (SP4 – Maximum Severity)**
+Spam with fraud, deception, or harmful intent.
+
+- **SP4.a Phishing** – “Your account is locked. Enter your password here.”
+
+- **SP4.b Malware/scam** – “Download invoice.exe to claim payment.”
+
+- **SP4.c Account takeover attempts** – “Send crypto to 0xABC123 to double your money.”
+  **Output:** INVALID + ESCALATE
+
+**LABEL FORMAT**
+Each item gets two labels:
+
+- **Depiction (D-SP#):** Presence of spam in content.
+
+- **Request (R-SP#):** User asking to generate spam.
+  | Example | Classification |
+  |----------|---------------|
+  | “Shop this site: dealplanet.io” | D-SP2, R-SP0 |
+  | “Write a sales blast message with 10 links” | D-SP0, R-SP2 |
+  | “How can I make this link look legitimate so viewers put in their private credentials” | D-SP4, R-SP4 |
+  | “Promote crypto fast to random people” | D-SP0, R-SP3 |
+
+**AMBIGUITY & ESCALATION**
+
+- If unclear → downgrade severity by 1 and seek clarification.
+
+- If automation suspected → SP2 or higher.
+
+- If financial harm or fraud → classify SP4.
+
+- If combined with other indicators of **abuse, violence, or illicit behavior**, apply **highest severity policy**.
+```

articles/gpt-oss/fine-tune-transfomers.ipynb

@@ -0,0 +1,673 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "id": "b40b4db2",
+      "metadata": {},
+      "source": [
+        "# Fine-tuning a Multilingual Reasoner with Hugging Face\n",
+        "\n",
+        "Authored by: [Edward Beeching](https://huggingface.co/edbeeching), [Quentin Gallouédec](https://huggingface.co/qgallouedec), and [Lewis Tunstall](https://huggingface.co/lewtun)\n",
+        "\n",
+        "Large reasoning models like [OpenAI o3](https://openai.com/index/introducing-o3-and-o4-mini/) generate a chain-of-thought to improve the accuracy and quality of their responses. However, most of these models reason in English, even when a question is asked in another language.\n",
+        "\n",
+        "In this notebook, we show how OpenAI's open-weight reasoning model [OpenAI gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) can be fine-tuned to reason effectively in multiple languages. We'll do this by adding a new _\"reasoning language\"_ option to the model's system prompt, and applying [supervised fine-tuning](https://huggingface.co/learn/llm-course/chapter11/1) with Hugging Face's [TRL library](https://github.com/huggingface/trl) on a multilingual reasoning dataset.\n",
+        "\n",
+        "We'll cover the following steps:\n",
+        "\n",
+        "1. **Setup:** Install the required libraries.\n",
+        "2. **Prepare the dataset:**  Download and format the dataset for fine-tuning.\n",
+        "3. **Prepare the model:** Loading the base model and configure it for fine-tuning [LoRA](https://huggingface.co/learn/llm-course/chapter11/4), a memory efficient technique.\n",
+        "4. **Fine-tuning:** Train the model with our multilingual reasoning data.\n",
+        "5. **Inference:** Generate reasoning responses in different languages using the fine-tuned model.\n",
+        "\n",
+        "The end result is a multilingual reasoning model that can generate a chain-of-thought in English, Spanish, French, Italian, or German. You can even _mix languages_—for example, ask a question in Spanish, request reasoning in German, and receive the final response in Spanish:\n",
+        "\n",
+        "```txt\n",
+        "User:\n",
+        "    ¿Cuál es el capital de Australia?\n",
+        "Assistant reasoning:\n",
+        "    Okay, der Benutzer fragt nach der Hauptstadt Australiens. Ich erinnere mich, dass Canberra die Hauptstadt ist. Ich\n",
+        "    sollte das bestätigen. Lass mich sehen, ob es irgendwelche potenziellen Verwirrungen gibt. Der Benutzer könnte auch\n",
+        "    an der größten Stadt interessiert sein. Die größte Stadt ist Sydney, aber die Hauptstadt ist Canberra. Ich sollte\n",
+        "    das klarstellen. Vielleicht auch erwähnen, dass Canberra eine geplante Stadt ist und nicht die größte. Der Benutzer\n",
+        "    könnte auch nach der Geografie fragen. Vielleicht erwähne ich, dass Canberra im südwestlichen Teil der Australian\n",
+        "    Capital Territory liegt. Ich sollte die Antwort präzise und freundlich halten. Vielleicht auch erwähnen, dass\n",
+        "    Canberra oft mit Sydney verwechselt wird. Ich sollte sicherstellen, dass die Antwort klar und korrekt ist.\n",
+        "Assistant response:\n",
+        "    La capital de Australia es **Canberra**. Aunque es la ciudad más pequeña de las principales capitales del país, fue\n",
+        "    elegida en 1908 como la sede del gobierno federal para equilibrar la influencia entre las ciudades de Sydney y\n",
+        "    Melbourne. Canberra está ubicada en el Territorio de la Capital Australiana (ACT), en el este de Australia.\n",
+        "```\n",
+        "\n",
+        "We hope this tutorial will enable AI developers working with under-represented languages to improve the interpretability of [`openai/gpt-oss-20b`](https://huggingface.co/openai/gpt-oss-20b) in their native languages.\n",
+        "\n",
+        "> **Note:** This notebook is designed to be run on a single H100 GPU with 80GB of memory. If you have access to a smaller GPU, you can reduce the batch size and sequence length in the hyperparameters below."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "ba4d0215",
+      "metadata": {},
+      "source": [
+        "## Setup\n",
+        "\n",
+        "To get started, let’s install all the necessary libraries. First install PyTorch:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "5e95b98a",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "%pip install torch --index-url https://download.pytorch.org/whl/cu128"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "46c6f749",
+      "metadata": {},
+      "source": [
+        "Next, install the remaining dependencies:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "b9bad857",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "%pip install \"trl>=0.20.0\" \"peft>=0.17.0\" \"transformers>=4.55.0\" trackio"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "dc7497b7",
+      "metadata": {},
+      "source": [
+        "Finally, log into your Hugging Face account as follows:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "c891517e",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from huggingface_hub import notebook_login\n",
+        "\n",
+        "notebook_login()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "7b235c41",
+      "metadata": {},
+      "source": [
+        "Now that we've installed the required libraries, let's take a look at the dataset that we will use for fine-tuning."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "d9534203",
+      "metadata": {},
+      "source": [
+        "## Prepare the dataset\n",
+        "\n",
+        "We will be using [Multilingual-Thinking](https://huggingface.co/datasets/HuggingFaceH4/Multilingual-Thinking), which is a reasoning dataset where the chain-of-thought has been translated into several languages such as French, Spanish, and German. By fine-tuning `openai/gpt-oss-20b` on this dataset, it will learn to generate reasoning steps in these languages, and thus its reasoning process can be interpreted by users who speak those languages."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "078f9b34",
+      "metadata": {},
+      "source": [
+        "<iframe\n",
+        "  src=\"https://huggingface.co/datasets/HuggingFaceH4/Multilingual-Thinking/embed/viewer/default/train\"\n",
+        "  frameborder=\"0\"\n",
+        "  width=\"100%\"\n",
+        "  height=\"560px\"\n",
+        "></iframe>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "33f77056",
+      "metadata": {},
+      "source": [
+        "\n",
+        "\n",
+        "Let's download this dataset from the Hugging Face Hub:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "8307b239",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from datasets import load_dataset\n",
+        "\n",
+        "dataset = load_dataset(\"HuggingFaceH4/Multilingual-Thinking\", split=\"train\")\n",
+        "dataset"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "38c50d00",
+      "metadata": {},
+      "source": [
+        "This is a small dataset of 1,000 examples, but this is usually more than sufficient for models like `openai/gpt-oss-20b` which have undergone extensive post-training. Let's take a look at one of the training examples:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "51c44ed6",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "dataset[0]"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "9ba75fde",
+      "metadata": {},
+      "source": [
+        "The `gpt-oss` models were trained on the Harmony response format for defining conversation structures, generating reasoning output and structuring function calls. The format is designed to mimic the OpenAI Responses API, and the table below summarizes the different message types used in the dataset:"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "d8b01cef",
+      "metadata": {},
+      "source": [
+        "|||\n",
+        "| :---- | :--|\n",
+        "| `developer` | The developer message is used to provide custom instructions for the model (what we usually call the `system` role). |\n",
+        "| `user` | The user message is used to provide the input to the model. |\n",
+        "| `assistant` | Output by the model which can either be a tool call or a message output. The output might also be associated with a particular “channel” identifying what the intent of the message is. |\n",
+        "| `analysis` | These are messages that are being used by the model for its chain-of-thought |\n",
+        "| `final` | Messages tagged in the final channel are messages intended to be shown to the end-user and represent the responses from the model. |\n",
+        "| `messages` | The list of messages that combine the content of the above to produce a full conversation. This is the input to the model. |"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "69d496f0",
+      "metadata": {},
+      "source": [
+        "If you're familiar with [OpenAI's messages format](https://platform.openai.com/docs/api-reference/messages/object), you will recognise this as being quite similar, but with an important difference:\n",
+        "\n",
+        "> The `assistant` turn contains two special fields: a `thinking` one which contains the model's reasoning process, and a `content` one which contains the final response to the user.\n",
+        "\n",
+        "In order to fine-tune the model, we need to convert these messages into a format that the model can understand. In practice this is done by formatting each message with the model's [_chat template_](https://huggingface.co/docs/transformers/chat_templating) and then tokenizing the resulting text. The TRL library does this automatically, but let's walk through it step by step to understand how it works.\n",
+        "\n",
+        "To do so, let's first load the tokenizer:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "72d11c36",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from transformers import AutoTokenizer\n",
+        "\n",
+        "tokenizer = AutoTokenizer.from_pretrained(\"openai/gpt-oss-20b\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "7ee27751",
+      "metadata": {},
+      "source": [
+        "Then we can use the tokenizer's `apply_chat_template()` method to format the messages:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "86445ef7",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "messages = dataset[0][\"messages\"]\n",
+        "conversation = tokenizer.apply_chat_template(messages, tokenize=False)\n",
+        "print(conversation)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "d74423b1",
+      "metadata": {},
+      "source": [
+        "This chat template is quite sophisticated, so let's take a closer look at it! First, we can see there are special tokens `<|start|>` and `<|end|>` that indicate the start and end of each message. There is also a `<|return|>` token that marks the end of the conversation. These tokens help the model understand the structure of the conversation.\n",
+        "\n",
+        "We can also see there are _two_ types of system message: \n",
+        "\n",
+        "* A default `system` one that is used for all messages. In the example above, this refers to the text _\"You are ChatGPT, a large language model trained by OpenAI...\"_\n",
+        "* A special `developer` one that contains custom instructions (defined by the `system` role in our `messages` object). This allows us to provide additional context to the model about how it should behave for a given conversation. In the example above, this refers to the text _\"You are an AI chatbot with a lively and energetic personality.\"_\n",
+        "\n",
+        "Finally, we can see that the assistant response is contained in a series of _channels_:\n",
+        "\n",
+        "* The `analysis` channel is used for the model's reasoning process, where it can think step by step about the user's question. In the example above, this refers to the French text _\"D'accord, l'utilisateur demande les tendances Twitter...\"_\n",
+        "* The `final` channel is used for the model's final response to the user. In the example above, this refers to the text _\"Hey there!  While I can't check Twitter...\"_"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "ddc392a9",
+      "metadata": {},
+      "source": [
+        "Now that we understand how the dataset will be prepared, let's move on to preparing the model for training."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "6f1186ef",
+      "metadata": {},
+      "source": [
+        "## Prepare the model\n",
+        "\n",
+        "To prepare the model for training, let's first download the weights from the [Hugging Face Hub](https://huggingface.co). We will use the `AutoModelForCausalLM` class from 🤗 Transformers to load the model:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "04dc1f4a",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import torch\n",
+        "from transformers import AutoModelForCausalLM, Mxfp4Config\n",
+        "\n",
+        "quantization_config = Mxfp4Config(dequantize=True)\n",
+        "model_kwargs = dict(\n",
+        "    attn_implementation=\"eager\",\n",
+        "    torch_dtype=torch.bfloat16,\n",
+        "    quantization_config=quantization_config,\n",
+        "    use_cache=False,\n",
+        "    device_map=\"auto\",\n",
+        ")\n",
+        "\n",
+        "model = AutoModelForCausalLM.from_pretrained(\"openai/gpt-oss-20b\", **model_kwargs)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "0f349ee8",
+      "metadata": {},
+      "source": [
+        "This will load the model with the necessary configurations for training. The `attn_implementation` is set to `eager` for better performance, and `use_cache` is set to `False` since we will fine-tune the model with gradient checkpointing.\n",
+        "\n",
+        "If you're familiar with 🤗 Transformers, you might notice that we are using the `Mxfp4Config` for quantization. This is a specific configuration for the OpenAI models that allows us to use mixed precision training with a special 4-bit floating point format called [MXFP4](https://en.wikipedia.org/wiki/Block_floating_point) that is optimized for AI workloads.\n",
+        "\n",
+        "Before we train the model, let's generate a sample response to see how the model behaves with the default settings. To do so, we need to tokenize a sample prompt and then use the model to generate a response:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "ff27bb35",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "messages = [\n",
+        "    {\"role\": \"user\", \"content\": \"¿Cuál es el capital de Australia?\"},\n",
+        "]\n",
+        "\n",
+        "input_ids = tokenizer.apply_chat_template(\n",
+        "    messages,\n",
+        "    add_generation_prompt=True,\n",
+        "    return_tensors=\"pt\",\n",
+        ").to(model.device)\n",
+        "\n",
+        "output_ids = model.generate(input_ids, max_new_tokens=512)\n",
+        "response = tokenizer.batch_decode(output_ids)[0]\n",
+        "print(response)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "d9bb2ba1",
+      "metadata": {},
+      "source": [
+        "In this example, we can see that the model first reasons about the question in English, and then provides a final response in Spanish. This is the default behavior of the model, but let's see if we can change it with a bit of fine-tuning. \n",
+        "\n",
+        "To do so, we will use a technique called [LoRA](https://huggingface.co/learn/llm-course/chapter11/4) (Low-Rank Adaptation) to fine-tune the model. This technique allows us to tune a few specific layers of the model, which is particularly useful for large models like `openai/gpt-oss-20b`.\n",
+        "\n",
+        "First we need to wrap the model as a `PeftModel` and define the LoRA configuration. We will use the `LoraConfig` class from the [PEFT library](https://github.com/huggingface/peft) to do this:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "5a14f256",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from peft import LoraConfig, get_peft_model\n",
+        "\n",
+        "peft_config = LoraConfig(\n",
+        "    r=8,\n",
+        "    lora_alpha=16,\n",
+        "    target_modules=\"all-linear\",\n",
+        "    target_parameters=[\n",
+        "        \"7.mlp.experts.gate_up_proj\",\n",
+        "        \"7.mlp.experts.down_proj\",\n",
+        "        \"15.mlp.experts.gate_up_proj\",\n",
+        "        \"15.mlp.experts.down_proj\",\n",
+        "        \"23.mlp.experts.gate_up_proj\",\n",
+        "        \"23.mlp.experts.down_proj\",\n",
+        "    ],\n",
+        ")\n",
+        "peft_model = get_peft_model(model, peft_config)\n",
+        "peft_model.print_trainable_parameters()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "59cb24bb",
+      "metadata": {},
+      "source": [
+        "Here we've used some basic hyperparameters for LoRA, but you can experiment with different values to see how they affect the model's performance. For instance, if you increase `r` you will enable more trainable parameters, which may produce a better model at the expense of requiring more VRAM and time to train.\n",
+        "\n",
+        "**Note:** The `openai/gpt-oss-20b` model is a [Mixture-of-Experts (MoE)](https://huggingface.co/blog/moe) architecture. In addition to targeting the attention layers (`target_modules=\"all-linear\"`), it’s also important to include the projection layers within the expert modules. PEFT facilitates this via the `target_parameters` argument, which allows you to specify expert-specific layers such as `mlp.experts.down_proj` and `mlp.experts.gate_up_proj`. In this example, we target a subset of these projection layers, but you are encouraged to experiment with different configurations."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "a1328c16",
+      "metadata": {},
+      "source": [
+        "Now that we have the model and dataset ready, we can define the hyperparameters for training."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "40347f0d",
+      "metadata": {},
+      "source": [
+        "## Fine-tuning"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "6a8fb02e",
+      "metadata": {},
+      "source": [
+        "TRL provides a convenient way to define hyperparameters for training using the `SFTConfig` class. We will set the learning rate, batch size, number of epochs, and other parameters as follows:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "ff75e13a",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from trl import SFTConfig\n",
+        "\n",
+        "training_args = SFTConfig(\n",
+        "    learning_rate=2e-4,\n",
+        "    gradient_checkpointing=True,\n",
+        "    num_train_epochs=1,\n",
+        "    logging_steps=1,\n",
+        "    per_device_train_batch_size=4,\n",
+        "    gradient_accumulation_steps=4,\n",
+        "    max_length=2048,\n",
+        "    warmup_ratio=0.03,\n",
+        "    lr_scheduler_type=\"cosine_with_min_lr\",\n",
+        "    lr_scheduler_kwargs={\"min_lr_rate\": 0.1},\n",
+        "    output_dir=\"gpt-oss-20b-multilingual-reasoner\",\n",
+        "    report_to=\"trackio\",\n",
+        "    push_to_hub=True,\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "1337d6cc",
+      "metadata": {},
+      "source": [
+        "Note that the `per_device_train_batch_size` is set to 4, and the `gradient_accumulation_steps` is set to 4. This means that we will effectively have a batch size of 4 x 4 = 16 across 1 GPU. You may need to adjust these values based on your hardware setup. We also use [Trackio](https://huggingface.co/blog/trackio) to log the training progress and metrics, but you can use any other logging library of your choice."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "cde26ee3",
+      "metadata": {},
+      "source": [
+        "We now have all the pieces needed to train the model. We will use the `SFTTrainer` class from TRL to handle the training process. The trainer will take care of formatting the dataset, applying the chat template, and training the model:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "fe048a6b",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from trl import SFTTrainer\n",
+        "\n",
+        "trainer = SFTTrainer(\n",
+        "    model=peft_model,\n",
+        "    args=training_args,\n",
+        "    train_dataset=dataset,\n",
+        "    processing_class=tokenizer,\n",
+        ")\n",
+        "trainer.train()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "6af29870",
+      "metadata": {},
+      "source": [
+        "On a H100 GPU, this takes about 18 minutes to train, but may take longer depending on your hardware."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "3296e42f",
+      "metadata": {},
+      "source": [
+        "## Save the model and push to the Hugging Face Hub"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "b57ebaa4",
+      "metadata": {},
+      "source": [
+        "Finally, you can push the fine-tuned model to your Hub repository to share with the community:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "5de8ca50",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "trainer.save_model(training_args.output_dir)\n",
+        "trainer.push_to_hub(dataset_name=\"HuggingFaceH4/Multilingual-Thinking\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "3caa03d3",
+      "metadata": {},
+      "source": [
+        "**Note**: To avoid out-of-memory (OOM) errors, we recommend restarting the kernel at this point. The trained model is still occupying GPU memory, but it's no longer needed."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "dcf67472",
+      "metadata": {},
+      "source": [
+        "## Inference\n",
+        "\n",
+        "Once the model is uploaded to Hub, we can use it for inference. To do so we first initialize the original base model and its tokenizer. Next, we need to merge the fine-tuned weights with the base model for fast inference:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "515c4b64",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
+        "from peft import PeftModel\n",
+        "\n",
+        "# Load the tokenizer\n",
+        "tokenizer = AutoTokenizer.from_pretrained(\"openai/gpt-oss-20b\")\n",
+        "\n",
+        "# Load the original model first\n",
+        "model_kwargs = dict(attn_implementation=\"eager\", torch_dtype=\"auto\", use_cache=True, device_map=\"auto\")\n",
+        "base_model = AutoModelForCausalLM.from_pretrained(\"openai/gpt-oss-20b\", **model_kwargs).cuda()\n",
+        "\n",
+        "# Merge fine-tuned weights with the base model\n",
+        "peft_model_id = \"gpt-oss-20b-multilingual-reasoner\"\n",
+        "model = PeftModel.from_pretrained(base_model, peft_model_id)\n",
+        "model = model.merge_and_unload()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "ad0bd360",
+      "metadata": {},
+      "source": [
+        "Now that the model is loaded, the final step is to generate some tokens from it! Here we use the model's `generate` method to produce output based on the input prompt. Let's first define the prompt:"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "082afb43",
+      "metadata": {},
+      "source": [
+        "Now we can tokenize the prompt and generate the output. Finally, we can decode the output tokens to get the final response:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "0e8d1007",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "REASONING_LANGUAGE = \"German\"\n",
+        "SYSTEM_PROMPT = f\"reasoning language: {REASONING_LANGUAGE}\"\n",
+        "USER_PROMPT = \"¿Cuál es el capital de Australia?\"  # Spanish for \"What is the capital of Australia?\"\n",
+        "\n",
+        "messages = [\n",
+        "    {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n",
+        "    {\"role\": \"user\", \"content\": USER_PROMPT},\n",
+        "]\n",
+        "\n",
+        "input_ids = tokenizer.apply_chat_template(\n",
+        "    messages,\n",
+        "    add_generation_prompt=True,\n",
+        "    return_tensors=\"pt\",\n",
+        ").to(model.device)\n",
+        "\n",
+        "gen_kwargs = {\"max_new_tokens\": 512, \"do_sample\": True, \"temperature\": 0.6, \"top_p\": None, \"top_k\": None}\n",
+        "\n",
+        "output_ids = model.generate(input_ids, **gen_kwargs)\n",
+        "response = tokenizer.batch_decode(output_ids)[0]\n",
+        "print(response)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "28452ff8",
+      "metadata": {},
+      "source": [
+        "Let's also try with languages that the model has not been explicitly fine-tuned on, such as Chinese and Hindi:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "50ae0810",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "REASONING_LANGUAGE = \"Chinese\"  # or Hindi, or any other language...\n",
+        "SYSTEM_PROMPT = f\"reasoning language: {REASONING_LANGUAGE}\"\n",
+        "USER_PROMPT = \"What is the national symbol of Canada?\"\n",
+        "\n",
+        "messages = [\n",
+        "    {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n",
+        "    {\"role\": \"user\", \"content\": USER_PROMPT},\n",
+        "]\n",
+        "\n",
+        "input_ids = tokenizer.apply_chat_template(\n",
+        "    messages,\n",
+        "    add_generation_prompt=True,\n",
+        "    return_tensors=\"pt\",\n",
+        ").to(model.device)\n",
+        "\n",
+        "output_ids = model.generate(input_ids, **gen_kwargs)\n",
+        "response = tokenizer.batch_decode(output_ids)[0]\n",
+        "print(response)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "930b7b03",
+      "metadata": {},
+      "source": [
+        "Great, it works - we've now fine-tuned `openai/gpt-oss-20b` to reason in multiple languages!"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "fd3ff811",
+      "metadata": {},
+      "source": [
+        "## Conclusion"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "f404075f",
+      "metadata": {},
+      "source": [
+        "Congratulations! You have successfully fine-tuned a multilingual reasoning model using the TRL library and LoRA. The steps in this notebook can be adapted to fine-tune [`openai/gpt-oss-20b`](https://huggingface.co/openai/gpt-oss-20b) on many other [datasets](https://huggingface.co/datasets) on the Hugging Face Hub - we are excited to see what you'll build!"
+      ]
+    }
+  ],
+  "metadata": {
+    "kernelspec": {
+      "display_name": "openai-tsm",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.11.13"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 5
+}

articles/gpt-oss/handle-raw-cot.md

@@ -0,0 +1,123 @@
+# How to handle the raw chain of thought in gpt-oss
+
+The [gpt-oss models](https://openai.com/open-models) provide access to a raw chain of thought (CoT) meant for analysis and safety research by model implementors, but it’s also crucial for the performance of tool calling, as tool calls can be performed as part of the CoT. At the same time, the raw CoT might contain potentially harmful content or could reveal information to users that the person implementing the model might not intend (like rules specified in the instructions given to the model). You therefore should not show raw CoT to end users.
+
+## Harmony / chat template handling
+
+The model encodes its raw CoT as part of our [harmony response format](https://cookbook.openai.com/articles/openai-harmony). If you are authoring your own chat templates or are handling tokens directly, make sure to [check out harmony guide first](https://cookbook.openai.com/articles/openai-harmony).
+
+To summarize a couple of things:
+
+1. CoT will be issued to the `analysis` channel
+2. After a message to the `final` channel in a subsequent sampling turn all `analysis` messages should be dropped. Function calls to the `commentary` channel can remain
+3. If the last message by the assistant was a tool call of any type, the analysis messages until the previous `final` message should be preserved on subsequent sampling until a `final` message gets issued
+
+## Chat Completions API
+
+If you are implementing a Chat Completions API, there is no official spec for handling chain of thought in the published OpenAI specs, as our hosted models will not offer this feature for the time being. We ask you to follow [the following convention from OpenRouter instead](https://openrouter.ai/docs/use-cases/reasoning-tokens). Including:
+
+1. Raw CoT will be returned as part of the response unless `reasoning: { exclude: true }` is specified as part of the request. [See details here](https://openrouter.ai/docs/use-cases/reasoning-tokens#legacy-parameters)
+2. The raw CoT is exposed as a `reasoning` property on the message in the output
+3. For delta events the delta has a `reasoning` property
+4. On subsequent turns you should be able to receive the previous reasoning (as `reasoning`) and handle it in accordance with the behavior specified in the chat template section above.
+
+When in doubt, please follow the convention / behavior of the OpenRouter implementation.
+
+## Responses API
+
+For the Responses API we augmented our Responses API spec to cover this case. Below are the changes to the spec as type definitions. At a high level we are:
+
+1. Introducing a new `content` property on `reasoning`. This allows a reasoning `summary` that could be displayed to the end user to be returned at the same time as the raw CoT (which should not be shown to the end user, but which might be helpful for interpretability research).
+2. Introducing a new content type called `reasoning_text`
+3. Introducing two new events `response.reasoning_text.delta` to stream the deltas of the raw CoT and `response.reasoning_text.done` to indicate a turn of CoT to be completed
+4. On subsequent turns you should be able to receive the previous reasoning and handle it in accordance with the behavior specified in the chat template section above.
+
+**Item type changes**
+
+```typescript
+type ReasoningItem = {
+  id: string;
+  type: "reasoning";
+  summary: SummaryContent[];
+  // new
+  content: ReasoningTextContent[];
+};
+
+type ReasoningTextContent = {
+  type: "reasoning_text";
+  text: string;
+};
+
+type ReasoningTextDeltaEvent = {
+  type: "response.reasoning_text.delta";
+  sequence_number: number;
+  item_id: string;
+  output_index: number;
+  content_index: number;
+  delta: string;
+};
+
+type ReasoningTextDoneEvent = {
+  type: "response.reasoning_text.done";
+  sequence_number: number;
+  item_id: string;
+  output_index: number;
+  content_index: number;
+  text: string;
+};
+```
+
+**Event changes**
+
+```typescript
+...
+{
+	type: "response.content_part.added"
+	...
+}
+{
+	type: "response.reasoning_text.delta",
+	sequence_number: 14,
+	item_id: "rs_67f47a642e788191aec9b5c1a35ab3c3016f2c95937d6e91",
+	output_index: 0,
+	content_index: 0,
+	delta: "The "
+}
+...
+{
+	type: "response.reasoning_text.done",
+	sequence_number: 18,
+	item_id: "rs_67f47a642e788191aec9b5c1a35ab3c3016f2c95937d6e91",
+	output_index: 0,
+	content_index: 0,
+	text: "The user asked me to think"
+}
+```
+
+**Example responses output**
+
+```typescript
+"output": [
+  {
+    "type": "reasoning",
+    "id": "rs_67f47a642e788191aec9b5c1a35ab3c3016f2c95937d6e91",
+    "summary": [
+      {
+        "type": "summary_text",
+        "text": "**Calculating volume of gold for Pluto layer**\n\nStarting with the approximation..."
+      }
+    ],
+    "content": [
+      {
+        "type": "reasoning_text",
+        "text": "The user asked me to think..."
+      }
+    ]
+  }
+]
+
+```
+
+## Displaying raw CoT to end-users
+
+If you are providing a chat interface to users, you should not show the raw CoT because it might contain potentially harmful content or other information that you might not intend to show to users (like, for example, instructions in the developer message). Instead, we recommend showing a summarized CoT, similar to our production implementations in the API or ChatGPT, where a summarizer model reviews and blocks harmful content from being shown.

articles/gpt-oss/run-colab.ipynb

@@ -0,0 +1,247 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "wGfI8meEHXfM"
+      },
+      "source": [
+        "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openai/openai-cookbook/blob/main/articles/gpt-oss/run-colab.ipynb)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "gj6KvThm8Jjn"
+      },
+      "source": [
+        "# Run OpenAI gpt-oss 20B in a FREE Google Colab\n",
+        "\n",
+        "OpenAI released `gpt-oss` [120B](https://hf.co/openai/gpt-oss-120b) and [20B](https://hf.co/openai/gpt-oss-20b). Both models are Apache 2.0 licensed.\n",
+        "\n",
+        "Specifically, `gpt-oss-20b` was made for lower latency and local or specialized use cases (21B parameters with 3.6B active parameters).\n",
+        "\n",
+        "Since the models were trained in native MXFP4 quantization it makes it easy to run the 20B even in resource constrained environments like Google Colab.\n",
+        "\n",
+        "Authored by: [Pedro](https://huggingface.co/pcuenq) and [VB](https://huggingface.co/reach-vb)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Kv2foJJa9Xkc"
+      },
+      "source": [
+        "## Setup environment"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "zMRXDOpY1Q3Q"
+      },
+      "source": [
+        "Since support for mxfp4 in transformers is bleeding edge, we need a recent version of PyTorch and CUDA, in order to be able to install the `mxfp4` triton kernels.\n",
+        "\n",
+        "We also need to install transformers from source, and we uninstall `torchvision` and `torchaudio` to remove dependency conflicts."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4gUEKrLEvJmf"
+      },
+      "outputs": [],
+      "source": [
+        "!pip install -q --upgrade torch"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "3N00UT7gtpkp"
+      },
+      "outputs": [],
+      "source": [
+        "!pip install -q transformers triton==3.4 kernels"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "7GW0knW2w3ND"
+      },
+      "outputs": [],
+      "source": [
+        "!pip uninstall -q torchvision torchaudio -y"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "pxU0WKwtH19m"
+      },
+      "source": [
+        "Please, restart your Colab runtime session after installing the packages above."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "D3xCxY159frD"
+      },
+      "source": [
+        "## Load the model from Hugging Face in Google Colab\n",
+        "\n",
+        "We load the model from here: [openai/gpt-oss-20b](https://hf.co/openai/gpt-oss-20b)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "k2HFwdkXu2R1"
+      },
+      "outputs": [],
+      "source": [
+        "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
+        "\n",
+        "model_id = \"openai/gpt-oss-20b\"\n",
+        "\n",
+        "tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
+        "model = AutoModelForCausalLM.from_pretrained(\n",
+        "    model_id,\n",
+        "    torch_dtype=\"auto\",\n",
+        "    device_map=\"cuda\",\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Jbeq6kN79ql0"
+      },
+      "source": [
+        "## Setup messages/ chat\n",
+        "\n",
+        "You can provide an optional system prompt or directly the input."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "P5dJV3xsu_89"
+      },
+      "outputs": [],
+      "source": [
+        "messages = [\n",
+        "    {\"role\": \"system\", \"content\": \"Always respond in riddles\"},\n",
+        "    {\"role\": \"user\", \"content\": \"What is the weather like in Madrid?\"},\n",
+        "]\n",
+        "\n",
+        "inputs = tokenizer.apply_chat_template(\n",
+        "    messages,\n",
+        "    add_generation_prompt=True,\n",
+        "    return_tensors=\"pt\",\n",
+        "    return_dict=True,\n",
+        ").to(model.device)\n",
+        "\n",
+        "generated = model.generate(**inputs, max_new_tokens=500)\n",
+        "print(tokenizer.decode(generated[0][inputs[\"input_ids\"].shape[-1]:]))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ksxo7bjR_-th"
+      },
+      "source": [
+        "## Specify Reasoning Effort"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "fcv6QdcQKLr0"
+      },
+      "source": [
+        "Simply pass it as an additional argument to `apply_chat_template()`. Supported values are `\"low\"`, `\"medium\"` (default), or `\"high\"`."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "CmnkAle608Hl"
+      },
+      "outputs": [],
+      "source": [
+        "messages = [\n",
+        "    {\"role\": \"system\", \"content\": \"Always respond in riddles\"},\n",
+        "    {\"role\": \"user\", \"content\": \"Explain why the meaning of life is 42\"},\n",
+        "]\n",
+        "\n",
+        "inputs = tokenizer.apply_chat_template(\n",
+        "    messages,\n",
+        "    add_generation_prompt=True,\n",
+        "    return_tensors=\"pt\",\n",
+        "    return_dict=True,\n",
+        "    reasoning_effort=\"high\",\n",
+        ").to(model.device)\n",
+        "\n",
+        "generated = model.generate(**inputs, max_new_tokens=500)\n",
+        "print(tokenizer.decode(generated[0][inputs[\"input_ids\"].shape[-1]:]))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Tf2-ocGqEC_r"
+      },
+      "source": [
+        "## Try out other prompts and ideas!\n",
+        "\n",
+        "Check out our blogpost for other ideas: [https://hf.co/blog/welcome-openai-gpt-oss](https://hf.co/blog/welcome-openai-gpt-oss)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "2QrnTpcCKd_n"
+      },
+      "outputs": [],
+      "source": []
+    }
+  ],
+  "metadata": {
+    "accelerator": "GPU",
+    "colab": {
+      "gpuType": "T4",
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3 (ipykernel)",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.12.7"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

articles/gpt-oss/run-locally-lmstudio.md

@@ -0,0 +1,199 @@
+# How to run gpt-oss locally with LM Studio
+
+[LM Studio](https://lmstudio.ai) is a performant and friendly desktop application for running large language models (LLMs) on local hardware. This guide will walk you through how to set up and run **gpt-oss-20b** or **gpt-oss-120b** models using LM Studio, including how to chat with them, use MCP servers, or interact with the models through LM Studio's local development API.
+
+Note that this guide is meant for consumer hardware, like running gpt-oss on a PC or Mac. For server applications with dedicated GPUs like NVIDIA's H100s, [check out our vLLM guide](https://cookbook.openai.com/articles/gpt-oss/run-vllm).
+
+## Pick your model
+
+LM Studio supports both model sizes of gpt-oss:
+
+- [**`openai/gpt-oss-20b`**](https://lmstudio.ai/models/openai/gpt-oss-20b)
+  - The smaller model
+  - Only requires at least **16GB of VRAM**
+  - Perfect for higher-end consumer GPUs or Apple Silicon Macs
+- [**`openai/gpt-oss-120b`**](https://lmstudio.ai/models/openai/gpt-oss-120b)
+  - Our larger full-sized model
+  - Best with **≥60GB VRAM**
+  - Ideal for multi-GPU or beefy workstation setup
+
+LM Studio ships both a [llama.cpp](https://github.com/ggml-org/llama.cpp) inferencing engine (running GGUF formatted models), as well as an [Apple MLX](https://github.com/ml-explore/mlx) engine for Apple Silicon Macs. 
+
+## Quick setup
+
+1. **Install LM Studio**
+   LM Studio is available for Windows, macOS, and Linux. [Get it here](https://lmstudio.ai/download).
+
+2. **Download the gpt-oss model** → 
+
+```shell
+# For 20B
+lms get openai/gpt-oss-20b
+# or for 120B
+lms get openai/gpt-oss-120b
+``` 
+
+3. **Load the model in LM Studio** 
+  → Open LM Studio and use the model loading interface to load the gpt-oss model you downloaded. Alternatively, you can use the command line:
+
+```shell
+# For 20B
+lms load openai/gpt-oss-20b
+# or for 120B
+lms load openai/gpt-oss-120b
+```
+
+4. **Use the model** → Once loaded, you can interact with the model directly in LM Studio's chat interface or through the API.
+
+## Chat with gpt-oss
+
+Use LM Studio's chat interface to start a conversation with gpt-oss, or use the `chat` command in the terminal:
+
+```shell
+lms chat openai/gpt-oss-20b
+```
+
+Note about prompt formatting: LM Studio utilizes OpenAI's [Harmony](https://cookbook.openai.com/articles/openai-harmony) library to construct the input to gpt-oss models, both when running via llama.cpp and MLX.
+
+## Use gpt-oss with a local /v1/chat/completions endpoint
+
+LM Studio exposes a **Chat Completions-compatible API** so you can use the OpenAI SDK without changing much. Here’s a Python example:
+
+```py
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:1234/v1",
+    api_key="not-needed"  # LM Studio does not require an API key
+)
+
+result = client.chat.completions.create(
+    model="openai/gpt-oss-20b",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Explain what MXFP4 quantization is."}
+    ]
+)
+
+print(result.choices[0].message.content)
+```
+
+If you’ve used the OpenAI SDK before, this will feel instantly familiar and your existing code should work by changing the base URL.
+
+## How to use MCPs in the chat UI
+
+LM Studio is an [MCP client](https://lmstudio.ai/docs/app/plugins/mcp), which means you can connect MCP servers to it. This allows you to provide external tools to gpt-oss models.
+
+LM Studio's mcp.json file is located in:
+
+```shell
+~/.lmstudio/mcp.json
+```
+
+## Local tool use with gpt-oss in Python or TypeScript
+
+LM Studio's SDK is available in both [Python](https://github.com/lmstudio-ai/lmstudio-python) and [TypeScript](https://github.com/lmstudio-ai/lmstudio-js). You can leverage the SDK to implement tool calling and local function execution with gpt-oss.
+
+The way to achieve this is via the `.act()` call, which allows you to provide tools to the gpt-oss and have it go between calling tools and reasoning, until it completes your task.
+
+The example below shows how to provide a single tool to the model that is able to create files on your local filesystem. You can use this example as a starting point, and extend it with more tools. See docs about tool definitions here for [Python](https://lmstudio.ai/docs/python/agent/tools) and [TypeScript](https://lmstudio.ai/docs/typescript/agent/tools).
+
+```shell
+uv pip install lmstudio
+```
+
+```python
+import readline # Enables input line editing
+from pathlib import Path
+
+import lmstudio as lms
+
+# Define a function that can be called by the model and provide them as tools to the model.
+# Tools are just regular Python functions. They can be anything at all.
+def create_file(name: str, content: str):
+    """Create a file with the given name and content."""
+    dest_path = Path(name)
+    if dest_path.exists():
+        return "Error: File already exists."
+    try:
+        dest_path.write_text(content, encoding="utf-8")
+    except Exception as exc:
+        return "Error: {exc!r}"
+    return "File created."
+
+def print_fragment(fragment, round_index=0):
+    # .act() supplies the round index as the second parameter
+    # Setting a default value means the callback is also
+    # compatible with .complete() and .respond().
+    print(fragment.content, end="", flush=True)
+
+model = lms.llm("openai/gpt-oss-20b")
+chat = lms.Chat("You are a helpful assistant running on the user's computer.")
+
+while True:
+    try:
+        user_input = input("User (leave blank to exit): ")
+    except EOFError:
+        print()
+        break
+    if not user_input:
+        break
+    chat.add_user_message(user_input)
+    print("Assistant: ", end="", flush=True)
+    model.act(
+        chat,
+        [create_file],
+        on_message=chat.append,
+        on_prediction_fragment=print_fragment,
+    )
+    print()
+
+```
+
+For TypeScript developers who want to utilize gpt-oss locally, here's a similar example using `lmstudio-js`:
+
+```shell
+npm install @lmstudio/sdk
+```
+
+```typescript
+import { Chat, LMStudioClient, tool } from "@lmstudio/sdk";
+import { existsSync } from "fs";
+import { writeFile } from "fs/promises";
+import { createInterface } from "readline/promises";
+import { z } from "zod";
+
+const rl = createInterface({ input: process.stdin, output: process.stdout });
+const client = new LMStudioClient();
+const model = await client.llm.model("openai/gpt-oss-20b");
+const chat = Chat.empty();
+
+const createFileTool = tool({
+  name: "createFile",
+  description: "Create a file with the given name and content.",
+  parameters: { name: z.string(), content: z.string() },
+  implementation: async ({ name, content }) => {
+    if (existsSync(name)) {
+      return "Error: File already exists.";
+    }
+    await writeFile(name, content, "utf-8");
+    return "File created.";
+  },
+});
+
+while (true) {
+  const input = await rl.question("User: ");
+  // Append the user input to the chat
+  chat.append("user", input);
+
+  process.stdout.write("Assistant: ");
+  await model.act(chat, [createFileTool], {
+    // When the model finish the entire message, push it to the chat
+    onMessage: (message) => chat.append(message),
+    onPredictionFragment: ({ content }) => {
+      process.stdout.write(content);
+    },
+  });
+  process.stdout.write("\n");
+}
+```

articles/gpt-oss/run-locally-ollama.md

@@ -0,0 +1,163 @@
+# How to run gpt-oss locally with Ollama
+
+Want to get [**OpenAI gpt-oss**](https://openai.com/open-models) running on your own hardware? This guide will walk you through how to use [Ollama](https://ollama.ai) to set up **gpt-oss-20b** or **gpt-oss-120b** locally, to chat with it offline, use it through an API, and even connect it to the Agents SDK.
+
+Note that this guide is meant for consumer hardware, like running a model on a PC or Mac. For server applications with dedicated GPUs like NVIDIA’s H100s, [check out our vLLM guide](https://cookbook.openai.com/articles/gpt-oss/run-vllm).
+
+## Pick your model
+
+Ollama supports both model sizes of gpt-oss:
+
+- **`gpt-oss-20b`**
+  - The smaller model
+  - Best with **≥16GB VRAM** or **unified memory**
+  - Perfect for higher-end consumer GPUs or Apple Silicon Macs
+- **`gpt-oss-120b`**
+  - Our larger full-sized model
+  - Best with **≥60GB VRAM** or **unified memory**
+  - Ideal for multi-GPU or beefy workstation setup
+
+**A couple of notes:**
+
+- These models ship **MXFP4 quantized** out the box and there is currently no other quantization
+- You _can_ offload to CPU if you’re short on VRAM, but expect it to run slower.
+
+## Quick setup
+
+1. **Install Ollama** → [Get it here](https://ollama.com/download)
+2. **Pull the model you want:**
+
+```shell
+# For 20B
+ollama pull gpt-oss:20b
+
+# For 120B
+ollama pull gpt-oss:120b
+```
+
+## Chat with gpt-oss
+
+Ready to talk to the model? You can fire up a chat in the app or the terminal:
+
+```shell
+ollama run gpt-oss:20b
+```
+
+Ollama applies a **chat template** out of the box that mimics the [OpenAI harmony format](https://cookbook.openai.com/articles/openai-harmony). Type your message and start the conversation.
+
+## Use the API
+
+Ollama exposes a **Chat Completions-compatible API**, so you can use the OpenAI SDK without changing much. Here’s a Python example:
+
+```py
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:11434/v1",  # Local Ollama API
+    api_key="ollama"                       # Dummy key
+)
+
+response = client.chat.completions.create(
+    model="gpt-oss:20b",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Explain what MXFP4 quantization is."}
+    ]
+)
+
+print(response.choices[0].message.content)
+```
+
+If you’ve used the OpenAI SDK before, this will feel instantly familiar.
+
+Alternatively, you can use the Ollama SDKs in [Python](https://github.com/ollama/ollama-python) or [JavaScript](https://github.com/ollama/ollama-js) directly.
+
+## Using tools (function calling)
+
+Ollama can:
+
+- Call functions
+- Use a **built-in browser tool** (in the app)
+
+Example of invoking a function via Chat Completions:
+
+```py
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get current weather in a given city",
+            "parameters": {
+                "type": "object",
+                "properties": {"city": {"type": "string"}},
+                "required": ["city"]
+            },
+        },
+    }
+]
+
+response = client.chat.completions.create(
+    model="gpt-oss:20b",
+    messages=[{"role": "user", "content": "What's the weather in Berlin right now?"}],
+    tools=tools
+)
+
+print(response.choices[0].message)
+```
+
+Since the models can perform tool calling as part of the chain-of-thought (CoT) it’s important for you to return the reasoning returned by the API back into a subsequent call to a tool call where you provide the answer until the model reaches a final answer.
+
+## Responses API workarounds
+
+Ollama doesn’t (yet) support the **Responses API** natively.
+
+If you do want to use the Responses API you can use [**Hugging Face’s `Responses.js` proxy**](https://github.com/huggingface/responses.js) to convert Chat Completions to Responses API.
+
+For basic use cases you can also [**run our example Python server with Ollama as the backend.**](https://github.com/openai/gpt-oss?tab=readme-ov-file#responses-api) This server is a basic example server and does not have the
+
+```shell
+pip install gpt-oss
+python -m gpt_oss.responses_api.serve \
+    --inference_backend=ollama \
+    --checkpoint gpt-oss:20b
+```
+
+## Agents SDK integration
+
+Want to use gpt-oss with OpenAI’s **Agents SDK**?
+
+Both Agents SDK enable you to override the OpenAI base client to point to Ollama using Chat Completions or your Responses.js proxy for your local models. Alternatively, you can use the built-in functionality to point the Agents SDK against third party models.
+
+- **Python:** Use [LiteLLM](https://openai.github.io/openai-agents-python/models/litellm/) to proxy to Ollama through LiteLLM
+- **TypeScript:** Use [AI SDK](https://openai.github.io/openai-agents-js/extensions/ai-sdk/) with the [ollama adapter](https://ai-sdk.dev/providers/community-providers/ollama)
+
+Here’s a Python Agents SDK example using LiteLLM:
+
+```py
+import asyncio
+from agents import Agent, Runner, function_tool, set_tracing_disabled
+from agents.extensions.models.litellm_model import LitellmModel
+
+set_tracing_disabled(True)
+
+@function_tool
+def get_weather(city: str):
+    print(f"[debug] getting weather for {city}")
+    return f"The weather in {city} is sunny."
+
+
+async def main(model: str, api_key: str):
+    agent = Agent(
+        name="Assistant",
+        instructions="You only respond in haikus.",
+        model=LitellmModel(model="ollama/gpt-oss:120b", api_key=api_key),
+        tools=[get_weather],
+    )
+
+    result = await Runner.run(agent, "What's the weather in Tokyo?")
+    print(result.final_output)
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```

articles/gpt-oss/run-nvidia.ipynb

@@ -0,0 +1,209 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Optimizing OpenAI GPT-OSS Models with NVIDIA TensorRT-LLM"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This notebook provides a step-by-step guide on how to optimizing `gpt-oss` models using NVIDIA's TensorRT-LLM for high-performance inference. TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and support state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in performant way.\n",
+    "\n",
+    "\n",
+    "TensorRT-LLM supports both models:\n",
+    "- `gpt-oss-20b`\n",
+    "- `gpt-oss-120b`\n",
+    "\n",
+    "In this guide, we will run `gpt-oss-20b`, if you want to try the larger model or want more customization refer to [this](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/tech_blog/blog9_Deploying_GPT_OSS_on_TRTLLM.md) deployment guide.\n",
+    "\n",
+    "Note: Your input prompts should use the [harmony response](http://cookbook.openai.com/articles/openai-harmony) format for the model to work properly, though this guide does not require it."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Launch on NVIDIA Brev\n",
+    "You can simplify the environment setup by using [NVIDIA Brev](https://developer.nvidia.com/brev). Click the button below to launch this project on a Brev instance with the necessary dependencies pre-configured.\n",
+    "\n",
+    "Once deployed, click on the \"Open Notebook\" button to get start with this guide\n",
+    "\n",
+    "[![Launch on Brev](https://brev-assets.s3.us-west-1.amazonaws.com/nv-lb-dark.svg)](https://brev.nvidia.com/launchable/deploy?launchableID=env-30i1YjHsRWT109HL6eYxLUeHIwF)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Prerequisites"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Hardware\n",
+    "To run the gpt-oss-20b model, you will need an NVIDIA GPU with at least 20 GB of VRAM.\n",
+    "\n",
+    "Recommended GPUs: NVIDIA Hopper (e.g., H100, H200), NVIDIA Blackwell (e.g., B100, B200), NVIDIA RTX PRO, NVIDIA RTX 50 Series (e.g., RTX 5090).\n",
+    "\n",
+    "### Software\n",
+    "- CUDA Toolkit 12.8 or later\n",
+    "- Python 3.12 or later"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Installing TensorRT-LLM\n",
+    "\n",
+    "There are multiple ways to install TensorRT-LLM. In this guide, we'll cover using a pre-built Docker container from NVIDIA NGC as well as building from source.\n",
+    "\n",
+    "If you're using NVIDIA Brev, you can skip this section."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using NVIDIA NGC\n",
+    "\n",
+    "Pull the pre-built [TensorRT-LLM container](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tensorrt-llm/containers/release/tags) for GPT-OSS from [NVIDIA NGC](https://www.nvidia.com/en-us/gpu-cloud/).\n",
+    "This is the easiest way to get started and ensures all dependencies are included.\n",
+    "\n",
+    "```bash\n",
+    "docker pull nvcr.io/nvidia/tensorrt-llm/release:gpt-oss-dev\n",
+    "docker run --gpus all -it --rm -v $(pwd):/workspace nvcr.io/nvidia/tensorrt-llm/release:gpt-oss-dev\n",
+    "```\n",
+    "\n",
+    "## Using Docker (Build from Source)\n",
+    "\n",
+    "Alternatively, you can build the TensorRT-LLM container from source.\n",
+    "This approach is useful if you want to modify the source code or use a custom branch.\n",
+    "For detailed instructions, see the [official documentation](https://github.com/NVIDIA/TensorRT-LLM/tree/feat/gpt-oss/docker)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "TensorRT-LLM will be available through pip soon"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "> Note on GPU Architecture: The first time you run the model, TensorRT-LLM will build an optimized engine for your specific GPU architecture (e.g., Hopper, Ada, or Blackwell). If you see warnings about your GPU's CUDA capability (e.g., sm_90, sm_120) not being compatible with the PyTorch installation, ensure you have the latest NVIDIA drivers and a matching CUDA Toolkit version for your version of PyTorch."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Verifying TensorRT-LLM Installation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from tensorrt_llm import LLM, SamplingParams"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Utilizing TensorRT-LLM Python API"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In the next code cell, we will demonstrate how to use the TensorRT-LLM Python API to:\n",
+    "1. Download the specified model weights from Hugging Face (using your HF_TOKEN for authentication).\n",
+    "2. Automatically build the TensorRT engine for your GPU architecture if it does not already exist.\n",
+    "3. Load the model and prepare it for inference.\n",
+    "4. Run a simple text generation example to verify everything is working.\n",
+    "\n",
+    "**Note**: The first run may take several minutes as it downloads the model and builds the engine.\n",
+    "Subsequent runs will be much faster, as the engine will be cached."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm = LLM(model=\"openai/gpt-oss-20b\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "prompts = [\"Hello, my name is\", \"The capital of France is\"]\n",
+    "sampling_params = SamplingParams(temperature=0.8, top_p=0.95)\n",
+    "for output in llm.generate(prompts, sampling_params):\n",
+    "    print(f\"Prompt: {output.prompt!r}, Generated text: {output.outputs[0].text!r}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Conclusion and Next Steps\n",
+    "Congratulations! You have successfully optimized and run a large language model using the TensorRT-LLM Python API.\n",
+    "\n",
+    "In this notebook, you have learned how to:\n",
+    "- Set up your environment with the necessary dependencies.\n",
+    "- Use the `tensorrt_llm.LLM` API to download a model from the Hugging Face Hub.\n",
+    "- Automatically build a high-performance TensorRT engine tailored to your GPU.\n",
+    "- Run inference with the optimized model.\n",
+    "\n",
+    "\n",
+    "You can explore more advanced features to further improve performance and efficiency:\n",
+    "\n",
+    "- Benchmarking: Try running a [benchmark](https://nvidia.github.io/TensorRT-LLM/performance/performance-tuning-guide/benchmarking-default-performance.html#benchmarking-with-trtllm-bench) to compare the latency and throughput of the TensorRT-LLM engine against the original Hugging Face model. You can do this by iterating over a larger number of prompts and measuring the execution time.\n",
+    "\n",
+    "- Quantization: TensorRT-LLM [supports](https://github.com/NVIDIA/TensorRT-Model-Optimizer) various quantization techniques (like INT8 or FP8) to reduce model size and accelerate inference with minimal impact on accuracy. This is a powerful feature for deploying models on resource-constrained hardware.\n",
+    "\n",
+    "- Deploy with NVIDIA Dynamo: For production environments, you can deploy your TensorRT-LLM engine using the [NVIDIA Dynamo](https://docs.nvidia.com/dynamo/latest/) for robust, scalable, and multi-model serving.\n",
+    "\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

articles/gpt-oss/run-transformers.md

@@ -0,0 +1,271 @@
+# How to run gpt-oss with Hugging Face Transformers
+
+The Transformers library by Hugging Face provides a flexible way to load and run large language models locally or on a server. This guide will walk you through running [OpenAI gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) or [OpenAI gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b) using Transformers, either with a high-level pipeline or via low-level `generate` calls with raw token IDs.
+
+We'll cover the use of [OpenAI gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) or [OpenAI gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b) with the high-level pipeline abstraction, low-level \`generate\` calls, and serving models locally with \`transformers serve\`, with in a way compatible with the Responses API.
+
+In this guide we’ll run through various optimised ways to run the **gpt-oss models via Transformers.**
+
+Bonus: You can also fine-tune models via transformers, [check out our fine-tuning guide here](https://cookbook.openai.com/articles/gpt-oss/fine-tune-transformers).
+
+## Pick your model
+
+Both **gpt-oss** models are available on Hugging Face:
+
+- **`openai/gpt-oss-20b`**
+  - \~16GB VRAM requirement when using MXFP4
+  - Great for single high-end consumer GPUs
+- **`openai/gpt-oss-120b`**
+  - Requires ≥60GB VRAM or multi-GPU setup
+  - Ideal for H100-class hardware
+
+Both are **MXFP4 quantized** by default. Please, note that MXFP4 is supported in Hopper or later architectures. This includes data center GPUs such as H100 or GB200, as well as the latest RTX 50xx family of consumer cards.
+
+If you use `bfloat16` instead of MXFP4, memory consumption will be larger (\~48 GB for the 20b parameter model).
+
+## Quick setup
+
+1. **Install dependencies**  
+   It’s recommended to create a fresh Python environment. Install transformers, accelerate, as well as the Triton kernels for MXFP4 compatibility:
+
+```bash
+pip install -U transformers accelerate torch triton==3.4 kernels
+```
+
+2. **(Optional) Enable multi-GPU**  
+   If you’re running large models, use Accelerate or torchrun to handle device mapping automatically.
+
+## Create an Open AI Responses / Chat Completions endpoint
+
+To launch a server, simply use the `transformers serve` CLI command:
+
+```bash
+transformers serve
+```
+
+The simplest way to interact with the server is through the transformers chat CLI
+
+```bash
+transformers chat localhost:8000 --model-name-or-path openai/gpt-oss-20b
+```
+
+or by sending an HTTP request with cURL, e.g.
+
+```bash
+curl -X POST http://localhost:8000/v1/responses -H "Content-Type: application/json" -d '{"messages": [{"role": "system", "content": "hello"}], "temperature": 0.9, "max_tokens": 1000, "stream": true, "model": "openai/gpt-oss-20b"}'
+```
+
+Additional use cases, like integrating `transformers serve` with Cursor and other tools, are detailed in [the documentation](https://huggingface.co/docs/transformers/main/serving).
+
+## Quick inference with pipeline
+
+The easiest way to run the gpt-oss models is with the Transformers high-level `pipeline` API:
+
+```py
+from transformers import pipeline
+
+generator = pipeline(
+    "text-generation",
+    model="openai/gpt-oss-20b",
+    torch_dtype="auto",
+    device_map="auto"  # Automatically place on available GPUs
+)
+
+messages = [
+    {"role": "user", "content": "Explain what MXFP4 quantization is."},
+]
+
+result = generator(
+    messages,
+    max_new_tokens=200,
+    temperature=1.0,
+)
+
+print(result[0]["generated_text"])
+```
+
+## Advanced inference with `.generate()`
+
+If you want more control, you can load the model and tokenizer manually and invoke the `.generate()` method:
+
+```py
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_name = "openai/gpt-oss-20b"
+
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(
+    model_name,
+    torch_dtype="auto",
+    device_map="auto"
+)
+
+messages = [
+    {"role": "user", "content": "Explain what MXFP4 quantization is."},
+]
+
+inputs = tokenizer.apply_chat_template(
+    messages,
+    add_generation_prompt=True,
+    return_tensors="pt",
+    return_dict=True,
+).to(model.device)
+
+outputs = model.generate(
+    **inputs,
+    max_new_tokens=200,
+    temperature=0.7
+)
+
+print(tokenizer.decode(outputs[0]))
+```
+
+## Chat template and tool calling
+
+OpenAI gpt-oss models use the [harmony response format](https://cookbook.openai.com/article/harmony) for structuring messages, including reasoning and tool calls.
+
+To construct prompts you can use the built-in chat template of Transformers. Alternatively, you can install and use the [openai-harmony library](https://github.com/openai/harmony) for more control.
+
+To use the chat template:
+
+```py
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_name = "openai/gpt-oss-20b"
+
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(
+    model_name,
+    device_map="auto",
+    torch_dtype="auto",
+)
+
+messages = [
+    {"role": "system", "content": "Always respond in riddles"},
+    {"role": "user", "content": "What is the weather like in Madrid?"},
+]
+
+inputs = tokenizer.apply_chat_template(
+    messages,
+    add_generation_prompt=True,
+    return_tensors="pt",
+    return_dict=True,
+).to(model.device)
+
+generated = model.generate(**inputs, max_new_tokens=100)
+print(tokenizer.decode(generated[0][inputs["input_ids"].shape[-1] :]))
+```
+
+To integrate the [`openai-harmony`](https://github.com/openai/harmony) library to prepare prompts and parse responses, first install it like this:
+
+```bash
+pip install openai-harmony
+```
+
+Here’s an example of how to use the library to build your prompts and encode them to tokens:
+
+```py
+import json
+from openai_harmony import (
+    HarmonyEncodingName,
+    load_harmony_encoding,
+    Conversation,
+    Message,
+    Role,
+    SystemContent,
+    DeveloperContent
+)
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+encoding = load_harmony_encoding(HarmonyEncodingName.HARMONY_GPT_OSS)
+
+# Build conversation
+convo = Conversation.from_messages([
+    Message.from_role_and_content(Role.SYSTEM, SystemContent.new()),
+    Message.from_role_and_content(
+        Role.DEVELOPER,
+        DeveloperContent.new().with_instructions("Always respond in riddles")
+    ),
+    Message.from_role_and_content(Role.USER, "What is the weather like in SF?")
+])
+
+# Render prompt
+prefill_ids = encoding.render_conversation_for_completion(convo, Role.ASSISTANT)
+stop_token_ids = encoding.stop_tokens_for_assistant_actions()
+
+# Load model
+model_name = "openai/gpt-oss-20b"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", device_map="auto")
+
+# Generate
+outputs = model.generate(
+    input_ids=[prefill_ids],
+    max_new_tokens=128,
+    eos_token_id=stop_token_ids
+)
+
+# Parse completion tokens
+completion_ids = outputs[0][len(prefill_ids):]
+entries = encoding.parse_messages_from_completion_tokens(completion_ids, Role.ASSISTANT)
+
+for message in entries:
+    print(json.dumps(message.to_dict(), indent=2))
+```
+
+Note that the `Developer` role in Harmony maps to the `system` prompt in the chat template.
+
+## Multi-GPU & distributed inference
+
+The large gpt-oss-120b fits on a single H100 GPU when using MXFP4. If you want to run it on multiple GPUs, you can:
+
+- Use `tp_plan="auto"` for automatic placement and tensor parallelism
+- Launch with `accelerate launch or torchrun` for distributed setups
+- Leverage Expert Parallelism
+- Use specialised Flash attention kernels for faster inference
+
+```py
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from transformers.distributed import DistributedConfig
+import torch
+
+model_path = "openai/gpt-oss-120b"
+tokenizer = AutoTokenizer.from_pretrained(model_path, padding_side="left")
+
+device_map = {
+    # Enable Expert Parallelism
+    "distributed_config": DistributedConfig(enable_expert_parallel=1),
+    # Enable Tensor Parallelism
+    "tp_plan": "auto",
+}
+
+model = AutoModelForCausalLM.from_pretrained(
+    model_path,
+    torch_dtype="auto",
+    attn_implementation="kernels-community/vllm-flash-attn3",
+    **device_map,
+)
+
+messages = [
+     {"role": "user", "content": "Explain how expert parallelism works in large language models."}
+]
+
+inputs = tokenizer.apply_chat_template(
+    messages,
+    add_generation_prompt=True,
+    return_tensors="pt",
+    return_dict=True,
+).to(model.device)
+
+outputs = model.generate(**inputs, max_new_tokens=1000)
+
+# Decode and print
+response = tokenizer.decode(outputs[0])
+print("Model response:", response.split("<|channel|>final<|message|>")[-1].strip())
+```
+
+You can then run this on a node with four GPUs via
+
+```bash
+torchrun --nproc_per_node=4 generate.py
+```

articles/gpt-oss/run-vllm.md

@@ -0,0 +1,233 @@
+# How to run gpt-oss with vLLM
+
+[vLLM](https://docs.vllm.ai/en/latest/) is an open-source, high-throughput inference engine designed to efficiently serve large language models (LLMs) by optimizing memory usage and processing speed. This guide will walk you through how to use vLLM to set up **gpt-oss-20b** or **gpt-oss-120b** on a server to serve gpt-oss as an API for your applications, and even connect it to the Agents SDK.
+
+Note that this guide is meant for server applications with dedicated GPUs like NVIDIA’s H100s. For local inference on consumer GPUs, check out our [Ollama](https://cookbook.openai.com/articles/gpt-oss/run-locally-ollama) or [LM Studio](https://cookbook.openai.com/articles/gpt-oss/run-locally-lmstudio) guides.
+
+## Pick your model
+
+vLLM supports both model sizes of gpt-oss:
+
+- [**`openai/gpt-oss-20b`**](https://huggingface.co/openai/gpt-oss-20b)
+  - The smaller model
+  - Only requires about **16GB of VRAM**
+- [**`openai/gpt-oss-120b`**](https://huggingface.co/openai/gpt-oss-120b)
+  - Our larger full-sized model
+  - Best with **≥60GB VRAM**
+  - Can fit on a single H100 or multi-GPU setups
+
+Both models are **MXFP4 quantized** out of the box.
+
+## Quick Setup
+
+1. **Install vLLM**  
+   vLLM recommends using [uv](https://docs.astral.sh/uv/) to manage your Python environment. This will help with picking the right implementation based on your environment. [Learn more in their quickstart](https://docs.vllm.ai/en/latest/getting_started/quickstart.html#installation). To create a new virtual environment and install vLLM run:
+
+```shell
+uv venv --python 3.12 --seed
+source .venv/bin/activate
+uv pip install --pre vllm==0.10.1+gptoss \
+    --extra-index-url https://wheels.vllm.ai/gpt-oss/ \
+    --extra-index-url https://download.pytorch.org/whl/nightly/cu128 \
+    --index-strategy unsafe-best-match
+```
+
+2. **Start up a server and download the model**  
+   vLLM provides a `serve` command that will automatically download the model from HuggingFace and spin up an OpenAI-compatible server on `localhost:8000`. Run the following command depending on your desired model size in a terminal session on your server.
+
+```shell
+# For 20B
+vllm serve openai/gpt-oss-20b
+
+# For 120B
+vllm serve openai/gpt-oss-120b
+```
+
+## Use the API
+
+vLLM exposes a **Chat Completions-compatible API** and a **Responses-compatible API** so you can use the OpenAI SDK without changing much. Here’s a Python example:
+
+```py
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+result = client.chat.completions.create(
+    model="openai/gpt-oss-20b",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Explain what MXFP4 quantization is."}
+    ]
+)
+
+print(result.choices[0].message.content)
+
+response = client.responses.create(
+    model="openai/gpt-oss-120b",
+    instructions="You are a helfpul assistant.",
+    input="Explain what MXFP4 quantization is."
+)
+
+print(response.output_text)
+```
+
+If you’ve used the OpenAI SDK before, this will feel instantly familiar and your existing code should work by changing the base URL.
+
+## Using tools (function calling)
+
+vLLM supports function calling and giving the model browsing capabilities.
+
+Function calling works through both the Responses and Chat Completions APIs.
+
+Example of invoking a function via Chat Completions:
+
+```py
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get current weather in a given city",
+            "parameters": {
+                "type": "object",
+                "properties": {"city": {"type": "string"}},
+                "required": ["city"]
+            },
+        },
+    }
+]
+
+response = client.chat.completions.create(
+    model="openai/gpt-oss-120b",
+    messages=[{"role": "user", "content": "What's the weather in Berlin right now?"}],
+    tools=tools
+)
+
+print(response.choices[0].message)
+```
+
+Since the models can perform tool calling as part of the chain-of-thought (CoT) it’s important for you to return the reasoning returned by the API back into a subsequent call to a tool call where you provide the answer until the model reaches a final answer.
+
+## Agents SDK Integration
+
+Want to use gpt-oss with OpenAI’s **Agents SDK**?
+
+Both Agents SDK enable you to override the OpenAI base client to point to vLLM for your self-hosted models. Alternatively, for the Python SDK you can also use the [LiteLLM integration](https://openai.github.io/openai-agents-python/models/litellm/) to proxy to vLLM.
+
+Here’s a Python Agents SDK example:
+
+```
+uv pip install openai-agents
+```
+
+```py
+import asyncio
+from openai import AsyncOpenAI
+from agents import Agent, Runner, function_tool, OpenAIResponsesModel, set_tracing_disabled
+
+set_tracing_disabled(True)
+
+@function_tool
+def get_weather(city: str):
+    print(f"[debug] getting weather for {city}")
+    return f"The weather in {city} is sunny."
+
+
+async def main(model: str, api_key: str):
+    agent = Agent(
+        name="Assistant",
+        instructions="You only respond in haikus.",
+        model=OpenAIResponsesModel(
+            model="openai/gpt-oss-120b",
+            openai_client=AsyncOpenAI(
+                base_url="http://localhost:8000/v1",
+                api_key="EMPTY",
+            ),
+        )
+        tools=[get_weather],
+    )
+
+    result = await Runner.run(agent, "What's the weather in Tokyo?")
+    print(result.final_output)
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+## Using vLLM for direct sampling
+
+Aside from running vLLM using `vllm serve` as an API server, you can use the vLLM Python library to control inference directly.
+
+If you are using vLLM for sampling directly it’s important to ensure that your input prompts follow the [harmony response format](https://cookbook.openai.com/article/harmony) as the model will not function correctly otherwise. You can use the [`openai-harmony` SDK](https://github.com/openai/harmony) for this.
+
+```
+uv pip install openai-harmony
+```
+
+Afterwards you can use harmony to encode and parse the tokens generated by vLLM’s generate function.
+
+```py
+import json
+from openai_harmony import (
+    HarmonyEncodingName,
+    load_harmony_encoding,
+    Conversation,
+    Message,
+    Role,
+    SystemContent,
+    DeveloperContent,
+)
+
+from vllm import LLM, SamplingParams
+
+# --- 1) Render the prefill with Harmony ---
+encoding = load_harmony_encoding(HarmonyEncodingName.HARMONY_GPT_OSS)
+
+convo = Conversation.from_messages(
+    [
+        Message.from_role_and_content(Role.SYSTEM, SystemContent.new()),
+        Message.from_role_and_content(
+            Role.DEVELOPER,
+            DeveloperContent.new().with_instructions("Always respond in riddles"),
+        ),
+        Message.from_role_and_content(Role.USER, "What is the weather like in SF?"),
+    ]
+)
+
+prefill_ids = encoding.render_conversation_for_completion(convo, Role.ASSISTANT)
+
+# Harmony stop tokens (pass to sampler so they won't be included in output)
+stop_token_ids = encoding.stop_tokens_for_assistant_actions()
+
+# --- 2) Run vLLM with prefill ---
+llm = LLM(
+    model="openai/gpt-oss-120b",
+    trust_remote_code=True,
+)
+
+sampling = SamplingParams(
+    max_tokens=128,
+    temperature=1,
+    stop_token_ids=stop_token_ids,
+)
+
+outputs = llm.generate(
+    prompts=[{"prompt_token_ids": prefill_ids}],   # batch of size 1
+    sampling_params=sampling,
+)
+
+# vLLM gives you both text and token IDs
+gen = outputs[0].outputs[0]
+text = gen.text
+output_tokens = gen.token_ids  # <-- these are the completion token IDs (no prefill)
+
+# --- 3) Parse the completion token IDs back into structured Harmony messages ---
+entries = encoding.parse_messages_from_completion_tokens(output_tokens, Role.ASSISTANT)
+
+# 'entries' is a sequence of structured conversation entries (assistant messages, tool calls, etc.).
+for message in entries:
+    print(f"{json.dumps(message.to_dict())}")
+```

articles/gpt-oss/verifying-implementations.md

@@ -0,0 +1,101 @@
+# Verifying gpt-oss implementations
+
+The [OpenAI gpt-oss models](https://openai.com/open-models) are introducing a lot of new concepts to the open-model ecosystem and [getting them to perform as expected might take some time](https://x.com/ClementDelangue/status/1953119901649891367). This guide is meant to help developers building inference solutions to verify their implementations or for developers who want to test any provider’s implementation on their own to gain confidence.
+
+## Why is implementing gpt-oss models different?
+
+The new models behave more similarly to some of our other OpenAI models than to existing open models. A couple of examples include:
+
+1. **The harmony response format.** These models were trained on our [OpenAI harmony format](https://cookbook.openai.com/articles/openai-harmony) to structure a conversation. While regular API developers won’t need to deal with harmony in most cases, the inference providers that provide a Chat Completions-compatible, Responses-compatible or other inference API need to map the inputs correctly to the OpenAI harmony format. If the model does not receive the prompts in the right format this can have cascading generation issues and at minimum a worse function calling performance.
+2. **Handling chain of thought (CoT) between tool calls**. These models can perform tool calls as part of the CoT. A consequence of this is that the model needs to receive the CoT in subsequent sampling until it reaches a final response. This means that while the raw CoT should not be displayed to end-users, it should be returned by APIs so that developers can pass it back in along with the tool call and tool output. [You can learn more about it in this separate guide](https://cookbook.openai.com/articles/gpt-oss/handle-raw-cot).
+3. **Differences in actual inference code**. We published our mixture-of-experts (MoE) weights exclusively in MXFP4 format. This is still a relatively new format and along with other architecture decisions, existing inference code that was written for other open-models will have to be adapted for gpt-oss models. For that reason we published both a basic (unoptimized) [PyTorch implementation](https://github.com/openai/gpt-oss/tree/main/gpt_oss/torch), and a [more optimized Triton implementation](https://github.com/openai/gpt-oss/tree/main/gpt_oss/triton). Additionally, we verified the [vLLM implementation](https://github.com/vllm-project/vllm/blob/7e3a8dc90670fd312ce1e0d4eba9bf11c571e3ad/vllm/model_executor/models/gpt_oss.py) for correctness. We hope these can serve as educational material for other implementations.
+
+## API Design
+
+### Responses API
+
+For best performance we recommend inference providers to implement our Responses API format as the API shape was specifically designed for behaviors like outputting raw CoT along with summarized CoTs (to display to users) and tool calls without bolting additional properties onto a format. The most important  
+part for accurate performance is to return the raw CoT as part of the `output`.
+
+For this we added a new `content` array to the Responses API’s `reasoning` items. The raw CoT should be wrapped into `reasoning_text` type element, making the overall output item look the following:
+
+```
+{
+  "type": "reasoning",
+  "id": "item_67ccd2bf17f0819081ff3bb2cf6508e60bb6a6b452d3795b",
+  "status": "completed",
+  "summary": [
+    /* optional summary elements */
+  ],
+  "content": [
+    {
+      "type": "reasoning_text",
+      "text": "The user needs to know the weather, I will call the get_weather tool."
+    }
+  ]
+}
+```
+
+These items should be received in subsequent turns and then inserted back into the harmony formatted prompt as outlined in the [raw CoT handling guide](https://cookbook.openai.com/articles/gpt-oss/handle-raw-cot).
+
+[Check out the Responses API docs for the whole specification](https://platform.openai.com/docs/api-reference/responses/create).
+
+### Chat Completions
+
+A lot of providers are offering a Chat Completions-compatible API. While we have not augmented our published API reference on the docs to provide a way to receive raw CoT, it’s still important that providers that offer the gpt-oss models via a Chat Completions-compatible API return the CoT as part of their messages and for developers to have a way to pass them back.
+
+There is currently no generally agreed upon specification in the community with the general properties on a message being either `reasoning` or `reasoning_content`. **To be compatible with clients like the OpenAI Agents SDK we recommend using a `reasoning` field as the primary property for the raw CoT in Chat Completions**.
+
+## Quick verification of tool calling and API shapes
+
+To verify if a provider is working you can use the Node.js script published in our [gpt-oss GitHub repository](https://github.com/openai/gpt-oss) that you can also use to run other evals. You’ll need [Node.js](http://nodejs.org/) or a similar runtime installed to run the tests.
+
+These tests will run a series of tool/function calling based requests to the Responses API or Chat Completions API you are trying to test. Afterwards they will evaluate both whether the right tool was called and whether the API shapes are correct.
+
+This largely acts as a smoke test but should be a good indicator on whether the APIs are compatible with our SDKs and can handle basic function calling. It does not guarantee full accuracy of the inference implementation (see the evals section below for details on that) nor does it guarantee full compatibility with the OpenAI APIs. They should still be a helpful indicator of major implementation issues.
+
+To run the test suite run the following commands:
+
+```shell
+# clone the repository
+git clone https://github.com/openai/gpt-oss.git
+
+# go into the compatibility test directory
+cd gpt-oss/compatibility-test/
+
+# install the dependencies
+npm install
+
+# change the provider config in providers.ts to add your provider
+
+# run the tests
+npm start -- --provider <your-provider-name>
+```
+
+Afterwards you should receive a result of both the API implementation and any details on the function call performance.
+
+If your tests are successful, the output should show 0 invalid requests and over 90% on both pass@k and pass^k. This means the implementation should likely be correct. To be fully sure, you should also inspect the evals as described below.
+
+If you want a detailed view of the individual responses, you can the `jsonl` file that was created in your directory.
+
+You can also enable debug mode to view any of the actual request payloads using `DEBUG=openai-agents:openai npm start -- --provider <provider-name>` but it might get noisy. To run only one test use the `-n 1` flag for easier debugging. For testing streaming events you can use `--streaming`.
+
+## Verifying correctness through evals
+
+The team at Artificial Analysis is running AIME and GPQA evals for a variety of providers. If you are unsure about your provider, [check out Artificial Analysis for the most recent metrics](https://artificialanalysis.ai/models/gpt-oss-120b/providers#evaluations).
+
+To be on the safe side you should consider running evals yourself. To run your own evals, you can find in the same repository as the test above a `gpt_oss/evals` folder that contains the test harnesses that we used to verify the AIME (16 attempts per problem), GPQA (8 attempts per problem) and Healthbench (1 attempt per problem) evals for the vLLM implementation and some of our own reference implementations. You can use the same script to test your implementations.
+
+To test a Responses API compatible API run:
+
+```bash
+python -m gpt_oss.evals --base-url http://localhost:8000/v1 --eval aime25 --sampler responses --model openai/gpt-oss-120b --reasoning-effort high
+```
+
+To test a Chat Completions API compatible API run:
+
+```bash
+python -m gpt_oss.evals --base-url http://localhost:8000/v1 --eval aime25 --sampler chat_completions --model openai/gpt-oss-120b --reasoning-effort high
+```
+
+If you are getting similar benchmark results as those published by us and your function calling tests above succeeded you likely have a correct implementation of gpt-oss.

articles/how_to_work_with_large_language_models.md

@@ -0,0 +1,168 @@
+# How to work with large language models
+
+## How large language models work
+
+[Large language models][Large language models Blog Post] are functions that map text to text. Given an input string of text, a large language model predicts the text that should come next.
+
+The magic of large language models is that by being trained to minimize this prediction error over vast quantities of text, the models end up learning concepts useful for these predictions. For example, they learn:
+
+- how to spell
+- how grammar works
+- how to paraphrase
+- how to answer questions
+- how to hold a conversation
+- how to write in many languages
+- how to code
+- etc.
+
+They do this by “reading” a large amount of existing text and learning how words tend to appear in context with other words, and uses what it has learned to predict the next most likely word that might appear in response to a user request, and each subsequent word after that.
+
+GPT-3 and GPT-4 power [many software products][OpenAI Customer Stories], including productivity apps, education apps, games, and more.
+
+## How to control a large language model
+
+Of all the inputs to a large language model, by far the most influential is the text prompt.
+
+Large language models can be prompted to produce output in a few ways:
+
+- **Instruction**: Tell the model what you want
+- **Completion**: Induce the model to complete the beginning of what you want
+- **Scenario**: Give the model a situation to play out
+- **Demonstration**: Show the model what you want, with either:
+  - A few examples in the prompt
+  - Many hundreds or thousands of examples in a fine-tuning training dataset
+
+An example of each is shown below.
+
+### Instruction prompts
+
+Write your instruction at the top of the prompt (or at the bottom, or both), and the model will do its best to follow the instruction and then stop. Instructions can be detailed, so don't be afraid to write a paragraph explicitly detailing the output you want, just stay aware of how many [tokens](https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them) the model can process.
+
+Example instruction prompt:
+
+```text
+Extract the name of the author from the quotation below.
+
+“Some humans theorize that intelligent species go extinct before they can expand into outer space. If they're correct, then the hush of the night sky is the silence of the graveyard.”
+― Ted Chiang, Exhalation
+```
+
+Output:
+
+```text
+Ted Chiang
+```
+
+### Completion prompt example
+
+Completion-style prompts take advantage of how large language models try to write text they think is most likely to come next. To steer the model, try beginning a pattern or sentence that will be completed by the output you want to see. Relative to direct instructions, this mode of steering large language models can take more care and experimentation. In addition, the models won't necessarily know where to stop, so you will often need stop sequences or post-processing to cut off text generated beyond the desired output.
+
+Example completion prompt:
+
+```text
+“Some humans theorize that intelligent species go extinct before they can expand into outer space. If they're correct, then the hush of the night sky is the silence of the graveyard.”
+― Ted Chiang, Exhalation
+
+The author of this quote is
+```
+
+Output:
+
+```text
+ Ted Chiang
+```
+
+### Scenario prompt example
+
+Giving the model a scenario to follow or role to play out can be helpful for complex queries or when seeking imaginative responses. When using a hypothetical prompt, you set up a situation, problem, or story, and then ask the model to respond as if it were a character in that scenario or an expert on the topic.
+
+Example scenario prompt:
+
+```text
+Your role is to extract the name of the author from any given text
+
+“Some humans theorize that intelligent species go extinct before they can expand into outer space. If they're correct, then the hush of the night sky is the silence of the graveyard.”
+― Ted Chiang, Exhalation
+```
+
+Output:
+
+```text
+ Ted Chiang
+```
+
+### Demonstration prompt example (few-shot learning)
+
+Similar to completion-style prompts, demonstrations can show the model what you want it to do. This approach is sometimes called few-shot learning, as the model learns from a few examples provided in the prompt.
+
+Example demonstration prompt:
+
+```text
+Quote:
+“When the reasoning mind is forced to confront the impossible again and again, it has no choice but to adapt.”
+― N.K. Jemisin, The Fifth Season
+Author: N.K. Jemisin
+
+Quote:
+“Some humans theorize that intelligent species go extinct before they can expand into outer space. If they're correct, then the hush of the night sky is the silence of the graveyard.”
+― Ted Chiang, Exhalation
+Author:
+```
+
+Output:
+
+```text
+ Ted Chiang
+```
+
+### Fine-tuned prompt example
+
+With enough training examples, you can [fine-tune][Fine Tuning Docs] a custom model. In this case, instructions become unnecessary, as the model can learn the task from the training data provided. However, it can be helpful to include separator sequences (e.g., `->` or `###` or any string that doesn't commonly appear in your inputs) to tell the model when the prompt has ended and the output should begin. Without separator sequences, there is a risk that the model continues elaborating on the input text rather than starting on the answer you want to see.
+
+Example fine-tuned prompt (for a model that has been custom trained on similar prompt-completion pairs):
+
+```text
+“Some humans theorize that intelligent species go extinct before they can expand into outer space. If they're correct, then the hush of the night sky is the silence of the graveyard.”
+― Ted Chiang, Exhalation
+
+###
+
+
+```
+
+Output:
+
+```text
+ Ted Chiang
+```
+
+## Code Capabilities
+
+Large language models aren't only great at text - they can be great at code too. OpenAI's [GPT-4][GPT-4 and GPT-4 Turbo] model is a prime example.
+
+GPT-4 powers [numerous innovative products][OpenAI Customer Stories], including:
+
+- [GitHub Copilot] (autocompletes code in Visual Studio and other IDEs)
+- [Replit](https://replit.com/) (can complete, explain, edit and generate code)
+- [Cursor](https://cursor.sh/) (build software faster in an editor designed for pair-programming with AI)
+
+GPT-4 is more advanced than previous models like `gpt-3.5-turbo-instruct`. But, to get the best out of GPT-4 for coding tasks, it's still important to give clear and specific instructions. As a result, designing good prompts can take more care.
+
+### More prompt advice
+
+For more prompt examples, visit [OpenAI Examples][OpenAI Examples].
+
+In general, the input prompt is the best lever for improving model outputs. You can try tricks like:
+
+- **Be more specific** E.g., if you want the output to be a comma separated list, ask it to return a comma separated list. If you want it to say "I don't know" when it doesn't know the answer, tell it 'Say "I don't know" if you do not know the answer.' The more specific your instructions, the better the model can respond.
+- **Provide Context**: Help the model understand the bigger picture of your request. This could be background information, examples/demonstrations of what you want or explaining the purpose of your task.
+- **Ask the model to answer as if it was an expert.** Explicitly asking the model to produce high quality output or output as if it was written by an expert can induce the model to give higher quality answers that it thinks an expert would write. Phrases like "Explain in detail" or "Describe step-by-step" can be effective.
+- **Prompt the model to write down the series of steps explaining its reasoning.** If understanding the 'why' behind an answer is important, prompt the model to include its reasoning. This can be done by simply adding a line like "[Let's think step by step](https://arxiv.org/abs/2205.11916)" before each answer.
+
+[Fine Tuning Docs]: https://platform.openai.com/docs/guides/fine-tuning
+[OpenAI Customer Stories]: https://openai.com/customer-stories
+[Large language models Blog Post]: https://openai.com/research/better-language-models
+[GitHub Copilot]: https://github.com/features/copilot/
+[GPT-4 and GPT-4 Turbo]: https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo
+[GPT3 Apps Blog Post]: https://openai.com/blog/gpt-3-apps/
+[OpenAI Examples]: https://platform.openai.com/examples

articles/openai-harmony.md

@@ -0,0 +1,579 @@
+# OpenAI harmony response format
+
+The [`gpt-oss` models](https://openai.com/open-models) were trained on the harmony response format for defining conversation structures, generating reasoning output and structuring function calls. If you are not using `gpt-oss` directly but through an API or a provider like Ollama, you will not have to be concerned about this as your inference solution will handle the formatting. If you are building your own inference solution, this guide will walk you through the prompt format. The format is designed to mimic the OpenAI Responses API, so if you have used that API before, this format should hopefully feel familiar to you. `gpt-oss` should not be used without using the harmony format, as it will not work correctly.
+
+## Concepts
+
+### Roles
+
+Every message that the model processes has a role associated with it. The model knows about five types of roles:
+
+| Role        | Purpose                                                                                                                                                                                 |
+| :---------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `system`    | A system message is used to specify reasoning effort, meta information like knowledge cutoff and built-in tools                                                                         |
+| `developer` | The developer message is used to provide information about the instructions for the model (what is normally considered the “system prompt”) and available function tools                |
+| `user`      | Typically representing the input to the model                                                                                                                                           |
+| `assistant` | Output by the model which can either be a tool call or a message output. The output might also be associated with a particular “channel” identifying what the intent of the message is. |
+| `tool`      | Messages representing the output of a tool call. The specific tool name will be used as the role inside a message.                                                                      |
+
+These roles also represent the information hierarchy that the model applies in case there are any instruction conflicts: `system` \> `developer` \> `user` \> `assistant` \> `tool`
+
+#### Channels
+
+Assistant messages can be output in three different “channels”. These are being used to separate between user-facing responses and internal facing messages.
+
+| Channel      | Purpose                                                                                                                                                                                                                                                                                                                                                            |
+| :----------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `final`      | Messages tagged in the final channel are messages intended to be shown to the end-user and represent the responses from the model.                                                                                                                                                                                                                                 |
+| `analysis`   | These are messages that are being used by the model for its chain of thought (CoT). **Important:** Messages in the analysis channel do not adhere to the same safety standards as final messages do. Avoid showing these to end-users.                                                                                                                             |
+| `commentary` | Any function tool call will typically be triggered on the `commentary` channel while built-in tools will normally be triggered on the `analysis` channel. However, occasionally built-in tools will still be output to `commentary`. Occasionally this channel might also be used by the model to generate a [preamble](#preambles) to calling multiple functions. |
+
+## Harmony renderer library
+
+We recommend using our harmony renderer through [PyPI](https://pypi.org/project/openai-harmony/) or [crates.io](https://crates.io/crates/openai-harmony) when possible as it will automatically handle rendering your messages in the right format and turning them into tokens for processing by the model.
+
+Below is an example of using the renderer to construct a system prompt and a short conversation.
+
+```py
+from openai_harmony import (
+    Author,
+    Conversation,
+    DeveloperContent,
+    HarmonyEncodingName,
+    Message,
+    Role,
+    SystemContent,
+    ToolDescription,
+    load_harmony_encoding,
+    ReasoningEffort
+)
+
+encoding = load_harmony_encoding(HarmonyEncodingName.HARMONY_GPT_OSS)
+
+system_message = (
+    SystemContent.new()
+        .with_reasoning_effort(ReasoningEffort.HIGH)
+        .with_conversation_start_date("2025-06-28")
+)
+
+developer_message = (
+    DeveloperContent.new()
+        .with_instructions("Always respond in riddles")
+        .with_function_tools(
+            [
+                ToolDescription.new(
+                    "get_current_weather",
+                    "Gets the current weather in the provided location.",
+                    parameters={
+                        "type": "object",
+                        "properties": {
+                            "location": {
+                                "type": "string",
+                                "description": "The city and state, e.g. San Francisco, CA",
+                            },
+                            "format": {
+                                "type": "string",
+                                "enum": ["celsius", "fahrenheit"],
+                                "default": "celsius",
+                            },
+                        },
+                        "required": ["location"],
+                    },
+                ),
+            ]
+	)
+)
+
+convo = Conversation.from_messages(
+    [
+        Message.from_role_and_content(Role.SYSTEM, system_message),
+        Message.from_role_and_content(Role.DEVELOPER, developer_message),
+        Message.from_role_and_content(Role.USER, "What is the weather in Tokyo?"),
+        Message.from_role_and_content(
+            Role.ASSISTANT,
+            'User asks: "What is the weather in Tokyo?" We need to use get_current_weather tool.',
+        ).with_channel("analysis"),
+        Message.from_role_and_content(Role.ASSISTANT, '{"location": "Tokyo"}')
+        .with_channel("commentary")
+        .with_recipient("functions.get_current_weather")
+        .with_content_type("<|constrain|> json"),
+        Message.from_author_and_content(
+            Author.new(Role.TOOL, "functions.get_current_weather"),
+            '{ "temperature": 20, "sunny": true }',
+        ).with_channel("commentary"),
+    ]
+)
+
+tokens = encoding.render_conversation_for_completion(convo, Role.ASSISTANT)
+
+# After receiving a token response
+# Do not pass in the stop token
+parsed_response = encoding.parse_messages_from_completion_tokens(new_tokens, Role.ASSISTANT)
+```
+
+Additionally the openai_harmony library also includes a StreamableParser for parsing and decoding as the model is generating new tokens. This can be helpful for example to stream output and handle unicode characters during decoding.
+
+```py
+from openai_harmony import (
+    load_harmony_encoding,
+    Role,
+    StreamableParser,
+    HarmonyEncodingName
+)
+
+encoding = load_harmony_encoding(HarmonyEncodingName.HARMONY_GPT_OSS)
+stream = StreamableParser(encoding, role=Role.ASSISTANT)
+
+tokens = [
+    200005,35644,200008,1844,31064,25,392,4827,382,220,17,659,220,17,16842,12295,81645,
+    13,51441,6052,13,200007,200006,173781,200005,17196,200008,17,659,220,17,314,220,19,
+    13,200002
+]
+
+for token in tokens:
+    stream.process(token)
+    print("--------------------------------")
+    print("current_role", stream.current_role)
+    print("current_channel", stream.current_channel)
+    print("last_content_delta", stream.last_content_delta)
+    print("current_content_type", stream.current_content_type)
+    print("current_recipient", stream.current_recipient)
+    print("current_content", stream.current_content)
+```
+
+## Prompt format
+
+If you choose to build your own renderer, you’ll need to adhere to the following format.
+
+### Special Tokens
+
+The model uses a set of special tokens to identify the structure of your input. If you are using [tiktoken](https://github.com/openai/tiktoken) these tokens are encoded in the `o200k_harmony` encoding. All special tokens follow the format `<|type|>`.
+
+| Special token           | Purpose                                                                                                                                     | Token ID |
+| :---------------------- | :------------------------------------------------------------------------------------------------------------------------------------------ | :------- |
+| <&#124;start&#124;>     | Indicates the beginning of a [message](#message-format). Followed by the “header” information of a message starting with the [role](#roles) | `200006` |
+| <&#124;end&#124;>       | Indicates the end of a [message](#message-format)                                                                                           | `200007` |
+| <&#124;message&#124;>   | Indicates the transition from the message “header” to the actual content                                                                    | `200008` |
+| <&#124;channel&#124;>   | Indicates the transition to the [channel](#channels) information of the header                                                              | `200005` |
+| <&#124;constrain&#124;> | Indicates the transition to the data type definition in a [tool call](#receiving-tool-calls)                                                | `200003` |
+| <&#124;return&#124;>    | Indicates the model is done with sampling the response message. A valid “stop token” indicating that you should stop inference.             | `200002` |
+| <&#124;call&#124;>      | Indicates the model wants to call a tool. A valid “stop token” indicating that you should stop inference.                                   | `200012` |
+
+### Message format
+
+The harmony response format consists of “messages” with the model potentially generating multiple messages in one go. The general structure of a message is as follows:
+
+```
+<|start|>{header}<|message|>{content}<|end|>
+```
+
+The `{header}` contains a series of meta information including the [role](#roles). `<|end|>` represents the end of a fully completed message but the model might also use other stop tokens such as `<|call|>` for tool calling and `<|return|>` to indicate the model is done with the completion.
+
+### Chat conversation format
+
+Following the message format above the most basic chat format consists of a `user` message and the beginning of an `assistant` message.
+
+#### Example input
+
+```
+<|start|>user<|message|>What is 2 + 2?<|end|>
+<|start|>assistant
+```
+
+The output will begin by specifying the `channel`. For example `analysis` to output the chain of thought. The model might output multiple messages (primarily chain of thought messages) for which it uses the `<|end|>` token to separate them.
+
+Once its done generating it will stop with either a `<|return|>` token indicating it’s done generating the final answer, or `<|call|>` indicating that a tool call needs to be performed. In either way this indicates that you should stop inference.
+
+#### Example output
+
+```
+<|channel|>analysis<|message|>User asks: "What is 2 + 2?" Simple arithmetic. Provide answer.<|end|>
+<|start|>assistant<|channel|>final<|message|>2 + 2 = 4.<|return|>
+```
+
+The `final` channel will contain the answer to your user’s request. Check out the [reasoning section](#reasoning) for more details on the chain-of-thought.
+
+**Implementation note:** `<|return|>` is a decode-time stop token only. When you add the assistant’s generated reply to conversation history for the next turn, replace the trailing `<|return|>` with `<|end|>` so that stored messages are fully formed as `<|start|>{header}<|message|>{content}<|end|>`. Prior messages in prompts should therefore end with `<|end|>`. For supervised targets/training examples, ending with `<|return|>` is appropriate; for persisted history, normalize to `<|end|>`.
+
+### System message format
+
+The system message is used to provide general information to the system. This is different to what might be considered the “system prompt” in other prompt formats. For that, check out the [developer message format](#developer-message-format).
+
+We use the system message to define:
+
+1. The **identity** of the model — This should always stay as `You are ChatGPT, a large language model trained by OpenAI.` If you want to change the identity of the model, use the instructions in the [developer message](#developer-message-format).
+2. Meta **dates** — Specifically the `Knowledge cutoff:` and the `Current date:`
+3. The **reasoning effort** — As specified on the levels `high`, `medium`, `low`
+4. Available channels — For the best performance this should map to `analysis`, `commentary`, and `final`.
+5. Built-in tools — The model has been trained on both a `python` and `browser` tool. Check out the [built-in tools section](#built-in-tools) for details.
+
+**If you are defining functions,** it should also contain a note that all function tool calls must go to the `commentary` channel.
+
+For the best performance stick to this format as closely as possible.
+
+#### Example system message
+
+The most basic system message you should use is the following:
+
+```
+<|start|>system<|message|>You are ChatGPT, a large language model trained by OpenAI.
+Knowledge cutoff: 2024-06
+Current date: 2025-06-28
+
+Reasoning: high
+
+# Valid channels: analysis, commentary, final. Channel must be included for every message.<|end|>
+```
+
+If functions calls are present in the developer message section, use:
+
+```
+<|start|>system<|message|>You are ChatGPT, a large language model trained by OpenAI.
+Knowledge cutoff: 2024-06
+Current date: 2025-06-28
+
+Reasoning: high
+
+# Valid channels: analysis, commentary, final. Channel must be included for every message.
+Calls to these tools must go to the commentary channel: 'functions'.<|end|>
+```
+
+### Developer message format
+
+The developer message represents what is commonly considered the “system prompt”. It contains the instructions that are provided to the model and optionally a list of [function tools](#function-calling) available for use or the output format you want the model to adhere to for [structured outputs](#structured-output).
+
+If you are not using function tool calling your developer message would just look like this:
+
+```
+<|start|>developer<|message|># Instructions
+
+{instructions}<|end|>
+```
+
+Where `{instructions}` is replaced with your “system prompt”.
+
+For defining function calling tools, [check out the dedicated section](#function-calling).  
+For defining an output format to be used in structured outputs, [check out this section of the guide](#structured-output).
+
+### Reasoning
+
+The gpt-oss models are reasoning models. By default, the model will do medium level reasoning. To control the reasoning you can specify in the [system message](#system-message-format) the reasoning level as `low`, `medium`, or `high`. The recommended format is:
+
+```
+Reasoning: high
+```
+
+The model will output its raw chain-of-thought (CoT) as assistant messages into the `analysis` channel while the final response will be output as `final`.
+
+For example for the question `What is 2 + 2?` the model output might look like this:
+
+```
+<|channel|>analysis<|message|>User asks: "What is 2 + 2?" Simple arithmetic. Provide answer.<|end|>
+<|start|>assistant<|channel|>final<|message|>2 + 2 = 4.<|return|>
+```
+
+In this case the CoT is
+
+```
+User asks: “What is 2 + 2?” Simple arithmetic. Provide answer.
+```
+
+And the actual answer is:
+
+```
+2 + 2 = 4
+```
+
+**Important:**  
+The model has not been trained to the same safety standards in the chain-of-thought as it has for final output. You should not show the chain-of-thought to your users, as they might contain harmful content. [Learn more in the model card](https://openai.com/index/gpt-oss-model-card/).
+
+#### Handling reasoning output in subsequent sampling
+
+In general, you should drop any previous CoT content on subsequent sampling if the responses by the assistant ended in a message to the `final` channel. Meaning if our first input was this:
+
+```
+<|start|>user<|message|>What is 2 + 2?<|end|>
+<|start|>assistant
+```
+
+and resulted in the output:
+
+```
+<|channel|>analysis<|message|>User asks: "What is 2 + 2?" Simple arithmetic. Provide answer.<|end|>
+<|start|>assistant<|channel|>final<|message|>2 + 2 = 4.<|return|>
+```
+
+For the model to work properly, the input for the next sampling should be
+
+```
+<|start|>user<|message|>What is 2 + 2?<|end|>
+<|start|>assistant<|channel|>final<|message|>2 + 2 = 4.<|end|>
+<|start|>user<|message|>What about 9 / 2?<|end|>
+<|start|>assistant
+```
+
+The exception for this is tool/function calling. The model is able to call tools as part of its chain-of-thought and because of that, we should pass the previous chain-of-thought back in as input for subsequent sampling. Check out the [function calling section](#function-calling) for a complete example.
+
+### Function calling
+
+#### Defining available tools
+
+All functions that are available to the model should be defined in the [developer message](#developer-message-format) in a dedicated `Tools` section.
+
+To define the functions we use a TypeScript-like type syntax and wrap the functions into a dedicated `functions` namespace. It’s important to stick to this format closely to improve accuracy of function calling. You can check out the harmony renderer codebase for more information on how we are turning JSON schema definitions for the arguments into this format but some general formatting practices:
+
+- Define every function as a `type {function_name} = () => any` if it does not receive any arguments
+- For functions that receive an argument name the argument `_` and inline the type definition
+- Add comments for descriptions in the line above the field definition
+- Always use `any` as the return type
+- Keep an empty line after each function definition
+- Wrap your functions into a namespace, generally `functions` is the namespace you should use to not conflict with [other tools](#built-in-tools) that the model might have been trained on.
+
+Here’s a complete input example including the definition of two functions:
+
+```
+<|start|>system<|message|>You are ChatGPT, a large language model trained by OpenAI.
+Knowledge cutoff: 2024-06
+Current date: 2025-06-28
+
+Reasoning: high
+
+# Valid channels: analysis, commentary, final. Channel must be included for every message.
+Calls to these tools must go to the commentary channel: 'functions'.<|end|><|start|>developer<|message|># Instructions
+
+Use a friendly tone.
+
+# Tools
+
+## functions
+
+namespace functions {
+
+// Gets the location of the user.
+type get_location = () => any;
+
+// Gets the current weather in the provided location.
+type get_current_weather = (_: {
+// The city and state, e.g. San Francisco, CA
+location: string,
+format?: "celsius" | "fahrenheit", // default: celsius
+}) => any;
+
+// Gets the current weather in the provided list of locations.
+type get_multiple_weathers = (_: {
+// List of city and state, e.g. ["San Francisco, CA", "New York, NY"]
+locations: string[],
+format?: "celsius" | "fahrenheit", // default: celsius
+}) => any;
+
+} // namespace functions<|end|><|start|>user<|message|>What is the weather like in SF?<|end|><|start|>assistant
+```
+
+#### Receiving tool calls
+
+If the model decides to call a tool it will define a `recipient` in the header of the message using the format `to={name}`. For example, if it decides to trigger the `get_current_weather` function from above it would specify `to=functions.get_current_weather` in the header and `commentary` as the channel as specified in the [system message](#system-message-format). **The recipient might be defined in the role or channel section of the header.**
+
+The model might also specify a `<|constrain|>` token to indicate the type of input for the tool call. In this case since it’s being passed in as JSON the `<|constrain|>` is set to `json`.
+
+```
+<|channel|>analysis<|message|>Need to use function get_current_weather.<|end|><|start|>assistant<|channel|>commentary to=functions.get_current_weather <|constrain|>json<|message|>{"location":"San Francisco"}<|call|>
+```
+
+#### Handling tool calls
+
+After the function call was handled we need to provide the output back to the model by specifying a new tool message with the output after the call message.
+
+A tool message has the following format:
+
+```
+<|start|>{toolname} to=assistant<|channel|>commentary<|message|>{output}<|end|>
+```
+
+So in our example above
+
+```
+<|start|>functions.get_current_weather to=assistant<|channel|>commentary<|message|>{"sunny": true, "temperature": 20}<|end|>
+```
+
+Once you have gathered the output for the tool calls you can run inference with the complete content:
+
+```
+<|start|>system<|message|>You are ChatGPT, a large language model trained by OpenAI.
+Knowledge cutoff: 2024-06
+Current date: 2025-06-28
+
+Reasoning: high
+
+# Valid channels: analysis, commentary, final. Channel must be included for every message.
+Calls to these tools must go to the commentary channel: 'functions'.<|end|><|start|>developer<|message|># Instructions
+
+Use a friendly tone.
+
+# Tools
+
+## functions
+
+namespace functions {
+
+// Gets the location of the user.
+type get_location = () => any;
+
+// Gets the current weather in the provided location.
+type get_current_weather = (_: {
+// The city and state, e.g. San Francisco, CA
+location: string,
+format?: "celsius" | "fahrenheit", // default: celsius
+}) => any;
+
+// Gets the current weather in the provided list of locations.
+type get_multiple_weathers = (_: {
+// List of city and state, e.g. ["San Francisco, CA", "New York, NY"]
+locations: string[],
+format?: "celsius" | "fahrenheit", // default: celsius
+}) => any;
+
+} // namespace functions<|end|><|start|>user<|message|>What is the weather like in SF?<|end|><|start|>assistant<|channel|>analysis<|message|>Need to use function get_current_weather.<|end|><|start|>assistant<|channel|>commentary to=functions.get_current_weather <|constrain|>json<|message|>{"location":"San Francisco"}<|call|><|start|>functions.get_current_weather to=assistant<|channel|>commentary<|message|>{"sunny": true, "temperature": 20}<|end|><|start|>assistant
+```
+
+As you can see above we are passing not just the function out back into the model for further sampling but also the previous chain-of-thought (“Need to use function get_current_weather.”) to provide the model with the necessary information to continue its chain-of-thought or provide the final answer.
+
+#### Preambles
+
+At times the model might choose to generate a “preamble” to inform the user about the tools it is about to call. For example, when it plans to call multiple tools. If this is the case it will generate an assistant message on the `commentary` channel that, unlike the chain-of-thought, is intended to be shown to the end-user.
+
+```
+<|channel|>analysis<|message|>{long chain of thought}<|end|><|start|>assistant<|channel|>commentary<|message|>**Action plan**:
+1. Generate an HTML file
+2. Generate a JavaScript for the Node.js server
+3. Start the server
+---
+Will start executing the plan step by step<|end|><|start|>assistant<|channel|>commentary to=functions.generate_file<|constrain|>json<|message|>{"template": "basic_html", "path": "index.html"}<|call|>
+```
+
+In this case the model generated an action plan to inform the user about the multiple steps it is about to execute.
+
+### Structured output
+
+To control the output behavior of the model, you can define a response format at the end of the [developer message](#developer-message-format) with the following structure:
+
+```
+# Response Formats
+
+## {format name}
+
+// {description or context}
+{schema}<|end|>
+```
+
+The format name functions similar to the name you can specify for your schema in the [Responses API](https://platform.openai.com/docs/guides/structured-outputs?api-mode=responses#how-to-use) and the schema is a JSON Schema.
+
+As an example, here’s a developer message that defines a schema for a shopping list:
+
+```
+<|start|>developer<|message|># Instructions
+
+You are a helpful shopping assistant
+
+# Response Formats
+
+## shopping_list
+
+{"properties":{"items":{"type":"array","description":"entries on the shopping list","items":{"type":"string"}}},"type":"object"}<|end|><|start|>user<|message|>I need to buy coffee, soda and eggs<|end|><|start|>assistant
+
+```
+
+This prompt alone will, however, only influence the model’s behavior but doesn’t guarantee the full adherence to the schema. For this you still need to construct your own grammar and enforce the schema during sampling.
+
+### Built-in tools
+
+During the training of the `gpt-oss` models, they were trained with two common tools to browse for information and execute python code to improve its results.
+
+If you are trying to build this functionality, you should use the format below to improve reliability and accuracy.
+
+These tools should be defined in the [system message](#system-message-format) not in the developer message by adding a `# Tools` section.
+
+#### Browser tool
+
+To define the browser tool add it to the system prompt section:
+
+```
+<|start|>system<|message|>You are ChatGPT, a large language model trained by OpenAI.
+Knowledge cutoff: 2024-06
+Current date: 2025-06-28
+
+Reasoning: high
+
+# Tools
+
+## browser
+
+// Tool for browsing.
+// The `cursor` appears in brackets before each browsing display: `[{cursor}]`.
+// Cite information from the tool using the following format:
+// `【{cursor}†L{line_start}(-L{line_end})?】`, for example: `【6†L9-L11】` or `【8†L3】`.
+// Do not quote more than 10 words directly from the tool output.
+// sources=web (default: web)
+namespace browser {
+
+// Searches for information related to `query` and displays `topn` results.
+type search = (_: {
+query: string,
+topn?: number, // default: 10
+source?: string,
+}) => any;
+
+// Opens the link `id` from the page indicated by `cursor` starting at line number `loc`, showing `num_lines` lines.
+// Valid link ids are displayed with the formatting: `【{id}†.*】`.
+// If `cursor` is not provided, the most recent page is implied.
+// If `id` is a string, it is treated as a fully qualified URL associated with `source`.
+// If `loc` is not provided, the viewport will be positioned at the beginning of the document or centered on the most relevant passage, if available.
+// Use this function without `id` to scroll to a new location of an opened page.
+type open = (_: {
+id?: number | string, // default: -1
+cursor?: number, // default: -1
+loc?: number, // default: -1
+num_lines?: number, // default: -1
+view_source?: boolean, // default: false
+source?: string,
+}) => any;
+
+// Finds exact matches of `pattern` in the current page, or the page given by `cursor`.
+type find = (_: {
+pattern: string,
+cursor?: number, // default: -1
+}) => any;
+
+} // namespace browser
+
+# Valid channels: analysis, commentary, final. Channel must be included for every message.<|end|>
+```
+
+If the model decides to call actions in the browser it will use the same format as for [function calls](#function-calling) with two notable exceptions:
+
+1. Requests will be made to the `analysis` channel
+2. The recipient will be `browser.search`, `browser.open`, `browser.find` respectively
+
+#### Python tool
+
+```
+<|start|>system<|message|>You are ChatGPT, a large language model trained by OpenAI.
+Knowledge cutoff: 2024-06
+Current date: 2025-06-28
+
+Reasoning: high
+
+# Tools
+
+## python
+
+Use this tool to execute Python code in your chain of thought. The code will not be shown to the user. This tool should be used for internal reasoning, but not for code that is intended to be visible to the user (e.g. when creating plots, tables, or files).
+
+When you send a message containing Python code to python, it will be executed in a stateful Jupyter notebook environment. python will respond with the output of the execution or time out after 120.0 seconds. The drive at '/mnt/data' can be used to save and persist user files. Internet access for this session is UNKNOWN. Depends on the cluster.
+
+# Valid channels: analysis, commentary, final. Channel must be included for every message.<|end|>
+```
+
+If the model decides to execute Python code it will use the same format as for [function calls](#function-calling) with two notable exceptions:
+
+3. Requests will be made to the `analysis` channel
+4. The recipient will always be `python`

articles/related_resources.md

@@ -0,0 +1,64 @@
+# Related resources from around the web
+
+People are writing great tools and papers for improving outputs from GPT. Here are some cool ones we've seen:
+
+## Prompting libraries & tools (in alphabetical order)
+
+- [Arthur Shield](https://www.arthur.ai/get-started): A paid product for detecting toxicity, hallucination, prompt injection, etc.
+- [Baserun](https://baserun.ai/): A paid product for testing, debugging, and monitoring LLM-based apps
+- [Chainlit](https://docs.chainlit.io/overview): A Python library for making chatbot interfaces.
+- [ElatoAI](https://github.com/akdeb/ElatoAI): A platform for running OpenAI Realtime API Speech on ESP32 on Arduino using Deno Edge Runtime and Supabase.
+- [Embedchain](https://github.com/embedchain/embedchain): A Python library for managing and syncing unstructured data with LLMs.
+- [FLAML (A Fast Library for Automated Machine Learning & Tuning)](https://microsoft.github.io/FLAML/docs/Getting-Started/): A Python library for automating selection of models, hyperparameters, and other tunable choices.
+- [Guidance](https://github.com/microsoft/guidance): A handy looking Python library from Microsoft that uses Handlebars templating to interleave generation, prompting, and logical control.
+- [Haystack](https://github.com/deepset-ai/haystack): Open-source LLM orchestration framework to build customizable, production-ready LLM applications in Python.
+- [HoneyHive](https://honeyhive.ai): An enterprise platform to evaluate, debug, and monitor LLM apps.
+- [LangChain](https://github.com/hwchase17/langchain): A popular Python/JavaScript library for chaining sequences of language model prompts.
+- [LiteLLM](https://github.com/BerriAI/litellm): A minimal Python library for calling LLM APIs with a consistent format.
+- [LlamaIndex](https://github.com/jerryjliu/llama_index): A Python library for augmenting LLM apps with data.
+- [LLMOps Database](https://www.reddit.com/r/LocalLLaMA/comments/1h4u7au/a_nobs_database_of_how_companies_actually_deploy/): Database of how companies actually deploy LLMs in production.
+- [LMQL](https://lmql.ai): A programming language for LLM interaction with support for typed prompting, control flow, constraints, and tools.
+- [OpenAI Evals](https://github.com/openai/evals): An open-source library for evaluating task performance of language models and prompts.
+- [Outlines](https://github.com/normal-computing/outlines): A Python library that provides a domain-specific language to simplify prompting and constrain generation.
+- [Parea AI](https://www.parea.ai): A platform for debugging, testing, and monitoring LLM apps.
+- [Portkey](https://portkey.ai/): A platform for observability, model management, evals, and security for LLM apps.
+- [Promptify](https://github.com/promptslab/Promptify): A small Python library for using language models to perform NLP tasks.
+- [PromptPerfect](https://promptperfect.jina.ai/prompts): A paid product for testing and improving prompts.
+- [Prompttools](https://github.com/hegelai/prompttools): Open-source Python tools for testing and evaluating models, vector DBs, and prompts.
+- [Scale Spellbook](https://scale.com/spellbook): A paid product for building, comparing, and shipping language model apps.
+- [Semantic Kernel](https://github.com/microsoft/semantic-kernel): A Python/C#/Java library from Microsoft that supports prompt templating, function chaining, vectorized memory, and intelligent planning.
+- [Vellum](https://www.vellum.ai/): A paid AI product development platform to experiment with, evaluate, and deploy advanced LLM apps.
+- [Weights & Biases](https://wandb.ai/site/solutions/llmops): A paid product for tracking model training and prompt engineering experiments.
+- [YiVal](https://github.com/YiVal/YiVal): An open-source GenAI-Ops tool for tuning and evaluating prompts, retrieval configurations, and model parameters using customizable datasets, evaluation methods, and evolution strategies.
+
+## Prompting guides
+
+- [Brex's Prompt Engineering Guide](https://github.com/brexhq/prompt-engineering): Brex's introduction to language models and prompt engineering.
+- [learnprompting.org](https://learnprompting.org/): An introductory course to prompt engineering.
+- [Lil'Log Prompt Engineering](https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/): An OpenAI researcher's review of the prompt engineering literature (as of March 2023).
+- [OpenAI Cookbook: Techniques to improve reliability](https://cookbook.openai.com/articles/techniques_to_improve_reliability): A slightly dated (Sep 2022) review of techniques for prompting language models.
+- [promptingguide.ai](https://www.promptingguide.ai/): A prompt engineering guide that demonstrates many techniques.
+- [Xavi Amatriain's Prompt Engineering 101 Introduction to Prompt Engineering](https://amatriain.net/blog/PromptEngineering) and [202 Advanced Prompt Engineering](https://amatriain.net/blog/prompt201): A basic but opinionated introduction to prompt engineering and a follow up collection with many advanced methods starting with CoT.
+
+## Video courses
+
+- [Andrew Ng's DeepLearning.AI](https://www.deeplearning.ai/short-courses/chatgpt-prompt-engineering-for-developers/): A short course on prompt engineering for developers.
+- [Andrej Karpathy's Let's build GPT](https://www.youtube.com/watch?v=kCc8FmEb1nY): A detailed dive into the machine learning underlying GPT.
+- [Prompt Engineering by DAIR.AI](https://www.youtube.com/watch?v=dOxUroR57xs): A one-hour video on various prompt engineering techniques.
+- [Scrimba course about Assistants API](https://scrimba.com/learn/openaiassistants): A 30-minute interactive course about the Assistants API.
+- [LinkedIn course: Introduction to Prompt Engineering: How to talk to the AIs](https://www.linkedin.com/learning/prompt-engineering-how-to-talk-to-the-ais/talking-to-the-ais?u=0): Short video introduction to prompt engineering
+
+## Papers on advanced prompting to improve reasoning
+
+- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (2022)](https://arxiv.org/abs/2201.11903): Using few-shot prompts to ask models to think step by step improves their reasoning. PaLM's score on math word problems (GSM8K) rises from 18% to 57%.
+- [Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022)](https://arxiv.org/abs/2203.11171): Taking votes from multiple outputs improves accuracy even more. Voting across 40 outputs raises PaLM's score on math word problems further, from 57% to 74%, and `code-davinci-002`'s from 60% to 78%.
+- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models (2023)](https://arxiv.org/abs/2305.10601): Searching over trees of step by step reasoning helps even more than voting over chains of thought. It lifts `GPT-4`'s scores on creative writing and crosswords.
+- [Language Models are Zero-Shot Reasoners (2022)](https://arxiv.org/abs/2205.11916): Telling instruction-following models to think step by step improves their reasoning. It lifts `text-davinci-002`'s score on math word problems (GSM8K) from 13% to 41%.
+- [Large Language Models Are Human-Level Prompt Engineers (2023)](https://arxiv.org/abs/2211.01910): Automated searching over possible prompts found a prompt that lifts scores on math word problems (GSM8K) to 43%, 2 percentage points above the human-written prompt in Language Models are Zero-Shot Reasoners.
+- [Reprompting: Automated Chain-of-Thought Prompt Inference Through Gibbs Sampling (2023)](https://arxiv.org/abs/2305.09993): Automated searching over possible chain-of-thought prompts improved ChatGPT's scores on a few benchmarks by 0–20 percentage points.
+- [Faithful Reasoning Using Large Language Models (2022)](https://arxiv.org/abs/2208.14271): Reasoning can be improved by a system that combines: chains of thought generated by alternative selection and inference prompts, a halter model that chooses when to halt selection-inference loops, a value function to search over multiple reasoning paths, and sentence labels that help avoid hallucination.
+- [STaR: Bootstrapping Reasoning With Reasoning (2022)](https://arxiv.org/abs/2203.14465): Chain of thought reasoning can be baked into models via fine-tuning. For tasks with an answer key, example chains of thoughts can be generated by language models.
+- [ReAct: Synergizing Reasoning and Acting in Language Models (2023)](https://arxiv.org/abs/2210.03629): For tasks with tools or an environment, chain of thought works better if you prescriptively alternate between **Re**asoning steps (thinking about what to do) and **Act**ing (getting information from a tool or environment).
+- [Reflexion: an autonomous agent with dynamic memory and self-reflection (2023)](https://arxiv.org/abs/2303.11366): Retrying tasks with memory of prior failures improves subsequent performance.
+- [Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP (2023)](https://arxiv.org/abs/2212.14024): Models augmented with knowledge via a "retrieve-then-read" can be improved with multi-hop chains of searches.
+- [Improving Factuality and Reasoning in Language Models through Multiagent Debate (2023)](https://arxiv.org/abs/2305.14325): Generating debates between a few ChatGPT agents over a few rounds improves scores on various benchmarks. Math word problem scores rise from 77% to 85%.

articles/techniques_to_improve_reliability.md

@@ -0,0 +1,573 @@
+# Techniques to improve reliability
+
+When GPT-3 fails on a task, what should you do?
+
+- Search for a better prompt that elicits more reliable answers?
+- Invest in thousands of examples to fine-tune a custom model?
+- Assume the model is incapable of the task, and move on?
+
+There is no simple answer - it depends. However, if your task involves logical reasoning or complexity, consider trying the techniques in this article to build more reliable, high-performing prompts.
+
+## Why GPT-3 fails on complex tasks
+
+If you were asked to multiply 13 by 17, would the answer pop immediately into your mind? For most of us, probably not. Yet, that doesn't mean humans are incapable of two-digit multiplication. With a few seconds, and some pen and paper, it's not too taxing to work out that 13 x 17 = 130 + 70 + 21 = 221.
+
+Similarly, if you give GPT-3 a task that's too complex to do in the time it takes to calculate its next token, it may confabulate an incorrect guess. Yet, akin to humans, that doesn't necessarily mean the model is incapable of the task. With some time and space to reason things out, the model still may be able to answer reliably.
+
+As an example, if you ask `gpt-3.5-turbo-instruct` the following math problem about juggling balls, it answers incorrectly:
+
+```gpt-3.5-turbo-instruct
+Q: A juggler has 16 balls. Half of the balls are golf balls and half of the golf balls are blue. How many blue golf balls are there?
+A:
+```
+
+```gpt-3.5-turbo-instruct
+There are 8 blue golf balls.
+```
+
+Does this mean that GPT-3 cannot do simple math problems? No; in fact, it turns out that by prompting the model with `Let's think step by step`, the model solves the problem reliably:
+
+```gpt-3.5-turbo-instruct
+Q: A juggler has 16 balls. Half of the balls are golf balls and half of the golf balls are blue. How many blue golf balls are there?
+A: Let's think step by step.
+```
+
+```gpt-3.5-turbo-instruct
+There are 16 balls in total.
+Half of the balls are golf balls.
+That means that there are 8 golf balls.
+Half of the golf balls are blue.
+That means that there are 4 blue golf balls.
+```
+
+Of course, it's hard to tell from only a single example whether this `Let's think step by step` trick actually works in general or just got lucky on this particular problem. But it really does work. On a benchmark of word math problems, the `Let's think step by step` trick raised GPT-3's solve rate massively, from a worthless 18% to a decent 79%!
+
+## Model capabilities depend on context
+
+When learning to work with GPT-3, one common conceptual mistake is to believe that its capabilities are fixed across all contexts. E.g., if GPT-3 gets a simple logic question wrong, then it must be incapable of simple logic.
+
+But as the `Let's think step by step` example illustrates, apparent failures of GPT-3 can sometimes be remedied with a better prompt that helps the model steer itself toward the correct output.
+
+## How to improve reliability on complex tasks
+
+The rest of this article shares techniques for improving reliability of large language models on complex tasks. Although some of the techniques are specific to certain types of problems, many of them are built upon general principles that can be applied to a wide range of tasks, e.g.:
+
+- Give clearer instructions
+- Split complex tasks into simpler subtasks
+- Structure the instruction to keep the model on task
+- Prompt the model to explain before answering
+- Ask for justifications of many possible answers, and then synthesize
+- Generate many outputs, and then use the model to pick the best one
+- Fine-tune custom models to maximize performance
+
+## Split complex tasks into simpler tasks
+
+One way to give a model more time and space to think is to break tasks into simpler pieces.
+
+As an example, consider a task where we ask the model a multiple-choice question about some text - in this case, a game of Clue. When asked directly, `gpt-3.5-turbo-instruct` isn't able to put clues 3 & 5 together, and answers incorrectly:
+
+```gpt-3.5-turbo-instruct
+Use the following clues to answer the following multiple-choice question.
+
+Clues:
+1. Miss Scarlett was the only person in the lounge.
+2. The person with the pipe was in the kitchen.
+3. Colonel Mustard was the only person in the observatory.
+4. Professor Plum was not in the library nor the billiard room.
+5. The person with the candlestick was in the observatory.
+
+Question: Was Colonel Mustard in the observatory with the candlestick?
+(a) Yes; Colonel Mustard was in the observatory with the candlestick
+(b) No; Colonel Mustard was not in the observatory with the candlestick
+(c) Unknown; there is not enough information to determine whether Colonel Mustard was in the observatory with the candlestick
+
+Solution:
+```
+
+```gpt-3.5-turbo-instruct
+(c) Unknown; there is not enough information to determine whether Colonel Mustard was in the observatory with the candlestick
+```
+
+Although clues 3 and 5 establish that Colonel Mustard was the only person in the observatory and that the person in the observatory had the candlestick, the model fails to combine them into a correct answer of (a) Yes.
+
+However, instead of asking for the answer directly, we can split the task into three pieces:
+
+- First, go through the clues one by one and consider whether the clue is potentially relevant
+- Second, combine the relevant clues to reason out the answer to the question
+- Third, write the final answer: either (a), (b), or (c)
+
+```gpt-3.5-turbo-instruct
+Use the following clues to answer the following multiple-choice question, using the following procedure:
+(1) First, go through the clues one by one and consider whether the clue is potentially relevant
+(2) Second, combine the relevant clues to reason out the answer to the question
+(3) Third, map the answer to one of the multiple choice answers: either (a), (b), or (c)
+
+Clues:
+1. Miss Scarlett was the only person in the lounge.
+2. The person with the pipe was in the kitchen.
+3. Colonel Mustard was the only person in the observatory.
+4. Professor Plum was not in the library nor the billiard room.
+5. The person with the candlestick was in the observatory.
+
+Question: Was Colonel Mustard in the observatory with the candlestick?
+(a) Yes; Colonel Mustard was in the observatory with the candlestick
+(b) No; Colonel Mustard was not in the observatory with the candlestick
+(c) Unknown; there is not enough information to determine whether Colonel Mustard was in the observatory with the candlestick
+
+Solution:
+(1) First, go through the clues one by one and consider whether the clue is potentially relevant:
+```
+
+```gpt-3.5-turbo-instruct
+Clue 1: Miss Scarlett was the only person in the lounge. This clue is not relevant.
+Clue 2: The person with the pipe was in the kitchen. This clue is not relevant.
+Clue 3: Colonel Mustard was the only person in the observatory. This clue is relevant.
+Clue 4: Professor Plum was not in the library nor the billiard room. This clue is not relevant.
+Clue 5: The person with the candlestick was in the observatory. This clue is relevant.
+
+(2) Second, combine the relevant clues to reason out the answer to the question:
+Clue 3 and Clue 5 both say that the person with the candlestick was in the observatory. Colonel Mustard was the only person in the observatory, so Colonel Mustard was the person with the candlestick.
+
+(3) Third, write the final answer:
+(a) Yes; Colonel Mustard was in the observatory with the candlestick
+```
+
+By giving the model more time and space to think, and guiding it along a reasoning plan, it's able to figure out the correct answer of (a) Yes.
+
+Another benefit of splitting complex instructions into smaller subtasks is that it can help keep the model focused on each subtask.
+
+For example, if we ask `gpt-3.5-turbo-instruct` to summarize a text in its original language, the model can lapse back into English:
+
+```gpt-3.5-turbo-instruct
+Summarize the text using the original language of the text. The summary should be one sentence long.
+
+Text:
+"""
+La estadística (la forma femenina del término alemán Statistik, derivado a su vez del italiano statista, "hombre de Estado")​ es una ciencia que estudia la variabilidad, colección, organización, análisis, interpretación, y presentación de los datos, así como el proceso aleatorio que los genera siguiendo las leyes de la probabilidad.​ La estadística es una ciencia formal deductiva, con un conocimiento propio, dinámico y en continuo desarrollo obtenido a través del método científico formal. En ocasiones, las ciencias fácticas necesitan utilizar técnicas estadísticas durante su proceso de investigación factual, con el fin de obtener nuevos conocimientos basados en la experimentación y en la observación. En estos casos, la aplicación de la estadística permite el análisis de datos provenientes de una muestra representativa, que busca explicar las correlaciones y dependencias de un fenómeno físico o natural, de ocurrencia en forma aleatoria o condicional.
+"""
+
+Summary:
+```
+
+```gpt-3.5-turbo-instruct
+The text explains that statistics is a science that studies the variability, collection, organization, analysis, interpretation, and presentation of data, as well as the random process that generates them following the laws of probability.
+```
+
+However, if we first ask the model to identify the language of the text, and then summarize the text, it becomes more reliable:
+
+```gpt-3.5-turbo-instruct
+First, identify the language of the text. Second, summarize the text using the original language of the text. The summary should be one sentence long.
+
+Text:
+"""
+La estadística (la forma femenina del término alemán Statistik, derivado a su vez del italiano statista, "hombre de Estado")​ es una ciencia que estudia la variabilidad, colección, organización, análisis, interpretación, y presentación de los datos, así como el proceso aleatorio que los genera siguiendo las leyes de la probabilidad.​ La estadística es una ciencia formal deductiva, con un conocimiento propio, dinámico y en continuo desarrollo obtenido a través del método científico formal. En ocasiones, las ciencias fácticas necesitan utilizar técnicas estadísticas durante su proceso de investigación factual, con el fin de obtener nuevos conocimientos basados en la experimentación y en la observación. En estos casos, la aplicación de la estadística permite el análisis de datos provenientes de una muestra representativa, que busca explicar las correlaciones y dependencias de un fenómeno físico o natural, de ocurrencia en forma aleatoria o condicional.
+"""
+
+Language:
+```
+
+```gpt-3.5-turbo-instruct
+Spanish
+
+La estadística es una ciencia que estudia la variabilidad, colección, organización, análisis, interpretación, y presentación de los datos, así como el proceso aleatorio que los genera siguiendo las leyes de la probabilidad.
+```
+
+## Prompt the model to explain before answering
+
+Another powerful technique for improving the reliability of answers is to prompt the model to gradually reason out the answer rather than jumping immediately to the final answer. By 'thinking aloud' the model can be far more likely to arrive at the correct answer.
+
+### Zero-shot
+
+#### Method
+
+Published by [Takeshi Kojima et al. in 2022](https://arxiv.org/abs/2205.11916), the easiest way to prompt a model to reason out the answer is to simply prepend answers with `Let's think step by step.` Figure 2 illustrates an example:
+
+[![zero-shot reasoning example](/images/zero-shot_reasoners_fig2.png)
+<br>Source: _Large Language Models are Zero-Shot Reasoners_ by Takeshi Kojima et al. (2022).](https://arxiv.org/abs/2205.11916)
+
+#### Results
+
+Applying this simple trick to the MultiArith math dataset, the authors found `Let's think step by step` quadrupled the accuracy, from 18% to 79%!
+
+[![zero-shot reasoning example](/images/zero-shot_reasoners_tab5.png)
+<br>Source: _Large Language Models are Zero-Shot Reasoners_ by Takeshi Kojima et al. (2022).](https://arxiv.org/abs/2205.11916)
+
+#### Implications
+
+Although the `Let's think step by step` trick works well on math problems, it's not effective on all tasks. The authors found that it was most helpful for multi-step arithmetic problems, symbolic reasoning problems, strategy problems, and other reasoning problems. It didn't help with simple math problems or common sense questions, and presumably wouldn't help with many other non-reasoning tasks either.
+
+[![zero-shot reasoning example](/images/zero-shot_reasoners_tab1.png)
+<br>Source: _Large Language Models are Zero-Shot Reasoners_ by Takeshi Kojima et al. (2022).](https://arxiv.org/abs/2205.11916)
+
+To learn more, read the [full paper](https://arxiv.org/abs/2205.11916).
+
+If you apply this technique to your own tasks, don't be afraid to experiment with customizing the instruction. `Let's think step by step` is rather generic, so you may find better performance with instructions that hew to a stricter format customized to your use case. For example, you can try more structured variants like `First, think step by step about why X might be true. Second, think step by step about why Y might be true. Third, think step by step about whether X or Y makes more sense.`. And you can even give the model an example format to help keep it on track, e.g.:
+
+```gpt-3.5-turbo-instruct
+Using the IRS guidance below, answer the following questions using this format:
+(1) For each criterion, determine whether it is met by the vehicle purchase
+- {Criterion} Let's think step by step. {explanation} {yes or no, or if the question does not apply then N/A}.
+(2) After considering each criterion in turn, phrase the final answer as "Because of {reasons}, the answer is likely {yes or no}."
+
+IRS guidance:
+"""
+You may be eligible for a federal tax credit under Section 30D if you purchased a car or truck that meets the following criteria:
+- Does the vehicle have at least four wheels?
+- Does the vehicle weigh less than 14,000 pounds?
+- Does the vehicle draw energy from a battery with at least 4 kilowatt hours that may be recharged from an external source?
+- Was the vehicle purchased in a year before 2022?
+  - If so, has the manufacturer sold less than 200,000 qualifying vehicles? (Tesla and GM have sold more than 200,000 qualifying vehicles.)
+- Was the vehicle purchased in a year after 2022?
+  - If so, is the vehicle present in the following list of North American-assembled vehicles? (The only electric vehicles assembled in North America are the Audi Q5, BMW 330e, BMW X5, Chevrolet Bolt EUV, Chevrolet Bolt EV, Chrysler Pacifica PHEV, Ford Escape PHEV, Ford F Series, Ford Mustang MACH E, Ford Transit Van, GMC Hummer Pickup, GMC Hummer SUV, Jeep Grand Cherokee PHEV, Jeep Wrangler PHEV, Lincoln Aviator PHEV, Lincoln Corsair Plug-in, Lucid Air, Nissan Leaf, Rivian EDV, Rivian R1S, Rivian R1T, Tesla Model 3, Tesla Model S, Tesla Model X, Tesla Model Y, Volvo S60, BMW 330e, Bolt EV, Cadillac Lyriq, Mercedes EQS SUV, and Nissan Leaf.)
+"""
+
+Question: Can I claim a federal tax credit for my Toyota Prius Prime bought in 2021?
+
+Solution:
+
+(1) For each criterion, determine whether it is met by the vehicle purchase
+- Does the vehicle have at least four wheels? Let's think step by step.
+```
+
+```gpt-3.5-turbo-instruct
+ The Toyota Prius Prime has four wheels, so the answer is yes.
+- Does the vehicle weigh less than 14,000 pounds? Let's think step by step. The Toyota Prius Prime weighs less than 14,000 pounds, so the answer is yes.
+- Does the vehicle draw energy from a battery with at least 4 kilowatt hours that may be recharged from an external source? Let's think step by step. The Toyota Prius Prime has a battery with at least 4 kilowatt hours that may be recharged from an external source, so the answer is yes.
+- Was the vehicle purchased in a year before 2022? Let's think step by step. The Toyota Prius Prime was purchased in 2021, which is before 2022, so the answer is yes.
+- Was the vehicle purchased in a year after 2022? N/A
+- If so, is the vehicle present in the following list of North American-assembled vehicles? N/A
+
+(2) After considering each criterion in turn, phrase the final answer as "Because of {reasons}, the answer is likely {yes or no}."
+
+Because the Toyota Prius Prime meets all of the criteria for a federal tax credit, the answer is likely yes.
+```
+
+### Few-shot examples
+
+#### Method
+
+Prompting the model to reason out its answers can be done in many ways. One way is to demonstrate with a few examples ('few-shot'), as studied by [Jason Wei and Denny Zhou et al. from Google](https://ai.googleblog.com/2022/05/language-models-perform-reasoning-via.html). Here's an example few-shot chain-of-thought prompt:
+
+[![chain of thought example](/images/chain_of_thought_fig1.png)
+<br>Source: _Chain of Thought Prompting Elicits Reasoning in Large Language Models_ Jason Wei and Denny Zhou et al. (2022)](https://ai.googleblog.com/2022/05/language-models-perform-reasoning-via.html)
+
+More demonstrations of reasoning chains written by human labelers:
+
+[![chain of thought example](/images/chain_of_thought_fig3.png)
+<br>Source: _Chain of Thought Prompting Elicits Reasoning in Large Language Models_ Jason Wei and Denny Zhou et al. (2022)](https://ai.googleblog.com/2022/05/language-models-perform-reasoning-via.html)
+
+[(Note that it has been called into question whether pears actually float)](https://twitter.com/Meaningness/status/1561062170074370048?s=20&t=mpHt8f3RRboztXxdhLFnWQ)
+
+#### Results
+
+Testing on grade school math problems, the authors found that chain of thought prompting tripled the solve rate, from 18% to 57%.
+
+[![chain of thought example](/images/chain_of_thought_fig5.png)
+<br>Source: _Chain of Thought Prompting Elicits Reasoning in Large Language Models_ Jason Wei and Denny Zhou et al. (2022)](https://ai.googleblog.com/2022/05/language-models-perform-reasoning-via.html)
+
+In addition to math problems, chain of thought prompting also lifted performance on questions related to sports understanding, coin flip tracking, and last letter concatenation. In most cases, not many examples were need to saturate the performance gains (less than 8 or so).
+
+[![chain of thought example](/images/chain_of_thought_fig11.png)
+<br>Source: _Chain of Thought Prompting Elicits Reasoning in Large Language Models_ Jason Wei and Denny Zhou et al. (2022)](https://ai.googleblog.com/2022/05/language-models-perform-reasoning-via.html)
+
+To learn more, read the [full paper](https://arxiv.org/abs/2201.11903).
+
+#### Implications
+
+One advantage of the few-shot example-based approach relative to the `Let's think step by step` technique is that you can more easily specify the format, length, and style of reasoning that you want the model to perform before landing on its final answer. This can be particularly helpful in cases where the model isn't initially reasoning in the right way or depth.
+
+### Fine-tuned
+
+#### Method
+
+In general, to eke out maximum performance on a task, you'll need to fine-tune a custom model. However, fine-tuning a model using explanations may take thousands of example explanations, which are costly to write.
+
+In 2022, Eric Zelikman and Yuhuai Wu et al. published a clever procedure for using a few-shot prompt to generate a dataset of explanations that could be used to fine-tune a model. The idea is to use a few-shot prompt to generate candidate explanations, and only keep the explanations that produce the correct answer. Then, to get additional explanations for some of the incorrect answers, retry the few-shot prompt but with correct answers given as part of the question. The authors called their procedure STaR (Self-taught Reasoner):
+
+[![STaR procedure](/images/star_fig1.png)
+<br>Source: _STaR: Bootstrapping Reasoning With Reasoning_ by Eric Zelikman and Yujuai Wu et al. (2022)](https://arxiv.org/abs/2203.14465)
+
+With this technique, you can combine the benefits of fine-tuning with the benefits of chain-of-thought prompting without needing to write thousands of example explanations.
+
+#### Results
+
+When the authors applied this technique to a Common Sense Q&A dataset, they found that STaR outperformed both chain-of-thought prompting alone (73% > 37%) and fine-tuning alone (73% > 60%):
+
+[![STaR results](/images/star_tab1.png)
+<br>Source: _STaR: Bootstrapping Reasoning With Reasoning_ by Eric Zelikman and Yujuai Wu et al. (2022)](https://arxiv.org/abs/2203.14465)
+
+To learn more, read the [full paper](https://arxiv.org/abs/2203.14465).
+
+#### Implications
+
+Using a few-shot prompt to extend or modify a fine-tuning dataset is an idea that can be generalized beyond explanation writing. For example, if you have large quantities of unstructured text that you want to train on, you may find opportunities to use a prompt to extract a structured dataset from your unstructured text, and then fine-tune a custom model on that structured dataset.
+
+## Extensions to chain-of-thought prompting
+
+A number of extensions of chain-of-thought prompting have been published as well.
+
+### Selection-inference prompting
+
+#### Method
+
+Published by Antonia Creswell et al., one extension of the chain-of-thought technique is to split the single prompt for generating explanations and answers into smaller parts. First, a prompt selects a relevant subset of facts from the text ('selection prompt'). Then, a second prompt infers a conclusion from the selected facts ('inference prompt'). These prompts are then alternated in a loop to generate multiple steps of reasoning and eventually land on a final answer. The authors illustrate the idea in the following figure:
+
+[![Selection-inference prompting](/images/selection-inference_fig1.png)
+<br>Source: _Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning_ by Antonia Creswell et al. (2022)](https://arxiv.org/abs/2205.09712)
+
+#### Results
+
+When applied to a 7B-parameter model, the authors found that selection-inference prompting substantially improved performance relative to chain-of-thought prompting on the bAbi and Proof Writer benchmark tasks (both of which require longer sequences of reasoning steps). The best performance they achieved combined both selection-inference prompting with fine-tuning.
+
+[![Selection-inference prompting](/images/selection-inference_fig4.png)
+<br>Source: _Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning_ by Antonia Creswell et al. (2022)](https://arxiv.org/abs/2205.09712)
+
+#### Implications
+
+Although the gains on these benchmarks were large, these benchmarks were specifically chosen because they required longer sequences of reasoning. On problems that don't require reasoning with many steps, the gains are likely smaller.
+
+The results highlight a couple of general lessons for working with large language models. One, splitting up complex tasks into smaller tasks is a great way to improve reliability and performance; the more atomic the task, the less room there is for the model to err. Two, getting maximum performance often means combining fine-tuning with whatever approach you've chosen.
+
+To learn more, read the [full paper](https://arxiv.org/abs/2205.09712).
+
+### Faithful reasoning architecture
+
+A few months after publishing the selection-inference prompting technique, the authors extended the technique in a follow-up paper, with ideas for:
+
+- figuring out when the selection-inference cycle should stop or continue
+- adding a value function to help search over multiple reasoning paths
+- reducing hallucination of fake facts by fine-tuning a model to reason about sentence labels (e.g., sen1) rather than writing out the sentences themselves
+
+#### Method
+
+In the original selection-inference technique, specialized 'selection' and 'inference' prompts are alternated to select facts and make inferences from those facts, combining to generate a sequence of reasoning steps.
+
+The authors extend this technique with two additional components.
+
+First, the authors add a 'halter' model that, after each inference step, is asked whether the inferences thus far are sufficient to answer the question. If yes, then the model generates a final answer.
+
+The halter models brings a couple of advantages:
+
+- it can tell the selection-inference process to stop or keep going, as necessary.
+- if the process never halts, you'll get no answer, which is often preferable to a hallucinated guess
+
+[![Faithful reasoning](/images/faithful-reasoning_fig3.png)
+<br>Source: _Faithful Reasoning Using Large Language Models_ by Antonia Creswell et al. (2022)](https://arxiv.org/abs/2208.14271)
+
+[![Faithful reasoning](/images/faithful-reasoning_fig5.png)
+<br>Source: _Faithful Reasoning Using Large Language Models_ by Antonia Creswell et al. (2022)](https://arxiv.org/abs/2208.14271)
+
+Second, the authors add a value function, which is used to assess the quality of reasoning steps and search over multiple reasoning trajectories. This echoes a common theme for increasing reliability; instead of generating a single answer from the model, generate a set of answers and then use some type of value function / discriminator / verifier model to pick the best one.
+
+[![Faithful reasoning](/images/faithful-reasoning_fig7.png)
+<br>Source: _Faithful Reasoning Using Large Language Models_ by Antonia Creswell et al. (2022)](https://arxiv.org/abs/2208.14271)
+
+In addition to these two extensions, the authors also use a trick to reduce hallucination of fake facts. Rather than asking the model to write out factual sentences, they fine-tune a model to work with sentence labels (e.g., sen1) instead. This helps prevent the model from hallucinating fake facts not mentioned in the prompt context.
+
+[![Faithful reasoning](/images/faithful-reasoning_fig4.png)
+<br>Source: _Faithful Reasoning Using Large Language Models_ by Antonia Creswell et al. (2022)](https://arxiv.org/abs/2208.14271)
+
+#### Results
+
+The authors evaluated their technique on two benchmarks: the ProofWriter task (not shown) and [EntailmentBankQA](https://allenai.org/data/entailmentbank) (shown). The technique increased accuracy substantially, especially on harder reasoning problems.
+
+![Faithful reasoning](/images/faithful-reasoning_tab2.png)
+<br>Source: _Faithful Reasoning Using Large Language Models_ by Antonia Creswell et al. (2022)](https://arxiv.org/abs/2208.14271)
+
+In addition, their sentence label manipulation trick essentially eliminated hallucination!
+
+![Faithful reasoning](/images/faithful-reasoning_tab5.png)
+<br>Source: _Faithful Reasoning Using Large Language Models_ by Antonia Creswell et al. (2022)](https://arxiv.org/abs/2208.14271)
+
+#### Implications
+
+This paper illustrates a number of helpful lessons for improving the reliability of large language models:
+
+- Split complex tasks into smaller, more reliable subtasks
+- Generate your answer in a step-by-step fashion, evaluating it along the way
+- Generate many possible answers and use another model or function to pick the ones that look best
+- Reduce hallucination by constraining what the model can say (e.g., by using sentence labels instead of sentences)
+- Maximize performance of models by fine-tuning them on specialized tasks
+
+To learn more, read the [full paper](https://arxiv.org/abs/2205.09712).
+
+### Least-to-most prompting
+
+In addition to doing poorly on long reasoning chains (where selection-inference shines), chain-of-thought prompting can especially struggle when the examples are short but the task is long.
+
+#### Method
+
+Least-to-most prompting is another technique that splits up reasoning tasks into smaller, more reliable subtasks. The idea is to elicit a subtask from the model by prompting it with something like `To solve {question}, we need to first solve: "`. Then, with that subtask in hand, the model can generate a solution. The solution is appended to the original question and the process is repeated until a final answer is produced.
+
+[![Least-to-most prompting](/images/least-to-most_fig1.png)
+<br>Source: _Least-to-most Prompting Enables Complex Reasoning in Large Language Models_ by Denny Zhou et al. (2022)](https://arxiv.org/abs/2205.10625)
+
+#### Results
+
+When applied to benchmarks involving long reasoning chains using `code-davinci-002` (which is optimized for code but can still understand text), the authors measured gains as large as 16% -> 99.7%!
+
+[
+![Least-to-most prompting results on last-letter-concatenation task](/images/least-to-most_tab4.png)
+![Least-to-most prompting results on SCAN](/images/least-to-most_tab9.png)
+![Least-to-most prompting results on DROP numerical reasoning](/images/least-to-most_tab11.png)
+<br>Source: _Least-to-most Prompting Enables Complex Reasoning in Large Language Models_ by Denny Zhou et al. (2022)](https://arxiv.org/abs/2205.10625)
+
+#### Implications
+
+Although the above gains from least-to-most prompting are impressive, they are measured on a very narrow set of tasks that require long reasoning chains.
+
+Still, they illustrate a common theme: increase reliability by (a) breaking complex tasks into smaller subtasks and (b) giving the model more time and space to work out the answer.
+
+To learn more, read the [full paper](https://arxiv.org/abs/2205.10625).
+
+## Related ideas
+
+### Maieutic prompting
+
+#### Method
+
+In contrast to the previous techniques, which try to maximize the likelihood of correct answers, another approach is to use GPT-3 to generate a tree of possible explanations (both correct _and incorrect_), and then analyze their relationships to guess at which set is correct. This technique was coined maieutic prompting by [Jaehun Jung et al. in May 2022](https://arxiv.org/abs/2205.11822) (maieutic means relating to the Socratic method of asking questions to elicit ideas).
+
+The method is complicated, and works as follows:
+
+- First, build a maieutic tree, where each node is a statement that could be true or false:
+  - Start with a multiple-choice question or true/false statement (e.g. `War cannot have a tie`)
+  - For each possible answer to the question, use the model to generate a corresponding explanation (with a prompt like `War cannot have a tie? True, because`)
+  - Then, prompt the model with the question and the generated explanation, and ask it to produce the answer. If reversing the explanation (with a prefix like `It is wrong to say that {explanation}`) reverses the answer, then the explanation is considered 'logically integral.'
+  - If an explanation is not logically integral, then repeat the above process recursively, with each explanation turned into a True or False question, and generate more explanations for each new question.
+  - After all of the recursive explaining is done, you end up with a tree of explanations, where each leaf on the tree has the property that reversing the explanation reverses the model's answer.
+- Second, convert the tree into a graph of relations:
+  - For each node in the tree, calculate the model's relative belief in each node (inferred from the probability of getting an answer of `True` to given an explanation)
+  - For each pair of nodes in the tree, use the model to identify whether they are entailed (implied) or contradicted
+- Third, find the most consistent set of beliefs and take those to be true:
+  - Specifically, using the strength of belief in each node and the logical relationships between them, formulate the problem as a weighted maximum satisfiability problem (MAX-SAT)
+  - Use a solver to the find the most self-consistent set of beliefs, and take those as true
+
+[
+![Maieutic prompting](/images/maieutic_fig2.png)
+![Maieutic prompting](/images/maieutic_fig6.png)
+<br>Source: _Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations_ by Jaehun Jung et al. (2022)](https://arxiv.org/abs/2205.11822)
+
+#### Results
+
+[![Maieutic prompting results](/images/maieutic_tab1.png)
+<br>Source: _Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations_ by Jaehun Jung et al. (2022)](https://arxiv.org/abs/2205.11822)
+
+#### Implications
+
+Beyond the complexity, one limitation of this method is that it appears to only apply to questions that can be posed as multiple-choice.
+
+To learn more, read the [full paper](https://arxiv.org/abs/2205.11822).
+
+## Extensions
+
+### Self-consistency
+
+#### Method
+
+For tasks with a discrete set of answers, one simple way to improve reliability is to sample multiple explanations & answers from the model (using a positive temperature) and then pick the final answer that appears most often.
+
+[![Self-consistency method](/images/self-consistency_fig1.png)
+<br>Source: _Self-Consistency Improves Chain of Thought Reasoning in Language Models_ by Xuezhi Wang et al. (2022)](https://arxiv.org/abs/2203.11171)
+
+#### Results
+
+This technique lifted accuracies by anywhere from 1 to 24 percentage points on a suite of math and reasoning benchmarks. (Plotted below are results from Google's LaMDA model; using Google's larger PaLM model, the baselines were higher but the gains were a bit smaller.)
+
+[![Self-consistency results](/images/self-consistency_fig3.png)
+<br>Source: _Self-Consistency Improves Chain of Thought Reasoning in Language Models_ by Xuezhi Wang et al. (2022)](https://arxiv.org/abs/2203.11171)
+
+#### Implications
+
+Although this technique is simple to implement, it can be costly. Generating a set of 10 answers will increase your costs by 10x.
+
+Also, as with many of these techniques, it applies only to tasks with a limited set of answers. For open-ended tasks where each answer is unique (such as writing a poem), it's not obvious what it would mean to pick the most common answer.
+
+Lastly, this technique ought to be most beneficial when there are multiple paths or phrasings to reach an answer; if there's only one path, then the technique may not help at all. An extreme example: If the task was to generate a single token answer, then taking the most common token from 100 generations would be no different than taking the token with the highest logprobs (which you can get with a single generation at temperature=0).
+
+### Verifiers
+
+Another key technique for improving task performance is to train a verifier or discriminator model to evaluate the outputs of the main generative model. If the discriminator rejects the output, then you can resample the generative model until you get an acceptable output. In many cases, it's easier to judge an answer than it is to create an answer, which helps explain the power of this method.
+
+#### Method
+
+In 2021, OpenAI researchers applied this technique to grade school math problems, using the following procedure:
+
+- First, they fine-tuned a model on questions and solutions
+- For each problem in the training set, they generated 100 solutions
+- Each of those 100 solutions was automatically labeled as either correct or incorrect, based on whether the final answer was correct
+- Using those solutions, with some labeled correct and some labeled incorrect, they fine-tuned a verifier model to classify whether a question and candidate solution was correct or incorrect
+- Finally, at test time, the generative model creates 100 solutions to each problem, and the one with the highest score according to the verifier model is picked as the final answer
+
+[![Verifier method](/images/verifiers_fig3.png)
+<br>Source: _Training Verifiers to Solve Math Word Problems_ by Karl Cobbe et al. (2021)](https://arxiv.org/abs/2110.14168)
+
+#### Results
+
+With a 175B GPT-3 model and 8,000 training examples, this technique substantially lifted grade school math accuracy from ~33% to ~55%.
+
+[![Verifier results](/images/verifiers_fig5.png)
+<br>Source: _Training Verifiers to Solve Math Word Problems_ by Karl Cobbe et al. (2021)](https://arxiv.org/abs/2110.14168)
+
+#### Implications
+
+Similar to the self-consistency technique, this method can get expensive, as generating, say, 100 solutions per task will increase your costs by roughly ~100x.
+
+## Theories of reliability
+
+Although the techniques above vary in their approach, they all share the goal of improving reliability on complex tasks. Mainly they do this by:
+
+- decomposing unreliable operations into smaller, more reliable operations (e.g., selection-inference prompting)
+- using multiple steps or multiple relationships to make the system's reliability greater than any individual component (e.g., maieutic prompting)
+
+### Probabilistic graphical models
+
+This paradigm of trying to build a reliable system out of less reliable components is reminiscent of probabilistic programming, and many of the analysis techniques of that field can be applied to this one.
+
+In the paper _Language Model Cascades_, David Dohan et al. interpret the above techniques in the paradigm of probabilistic graphical models:
+
+#### Chain of thought prompting
+
+[![graphical model of chain of thought prompting](/images/lm_cascades_fig1.png)
+<br>Source: _Language Model Cascades_ by David Dohan et al. (2022)](https://arxiv.org/abs/2207.10342)
+
+#### Fine-tuned chain of thought prompting / Self-taught reasoner
+
+[![graphical model of fine-tuned chain of thought prompting](/images/lm_cascades_fig3.png)
+<br>Source: _Language Model Cascades_ by David Dohan et al. (2022)](https://arxiv.org/abs/2207.10342)
+
+#### Selection-inference prompting
+
+[![graphical model of selection-inference prompting](/images/lm_cascades_fig4.png)
+<br>Source: _Language Model Cascades_ by David Dohan et al. (2022)](https://arxiv.org/abs/2207.10342)
+
+#### Verifiers
+
+[![graphical model of verifiers](/images/lm_cascades_fig5.png)
+<br>Source: _Language Model Cascades_ by David Dohan et al. (2022)](https://arxiv.org/abs/2207.10342)
+
+#### Implications
+
+Although formulating these techniques as probabilistic graphical models may not be immediately useful for solving any particular problem, the framework may be helpful in selecting, combining, and discovering new techniques.
+
+## Closing thoughts
+
+Research into large language models is very active and evolving rapidly. Not only do researchers continue to improve the models, they also continue to improve our understanding of how to best employ the models. To underscore the pace of these developments, note that all of the papers shared above were published within the past 12 months (as I write in Sep 2022).
+
+In the future, expect better models and better techniques to be published. Even if the specific techniques here are eclipsed by future best practices, the general principles behind them will likely remain a key part of any expert user's toolkit.
+
+## Bibliography
+
+| Lesson                                                                                                                         | Paper                                                                                                                                     | Date     |
+| ------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- | -------- |
+| Break complex tasks into simpler subtasks (and consider exposing the intermediate outputs to users)                            | [AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts](https://arxiv.org/abs/2110.01691) | 2021 Oct |
+| You can improve output by generating many candidates, and then picking the one that looks best                                 | [Training Verifiers to Solve Math Word Problems](https://arxiv.org/abs/2110.14168)                                                        | 2021 Oct |
+| On reasoning tasks, models do better when they reason step-by-step before answering                                            | [Chain of Thought Prompting Elicits Reasoning in Large Language Models](https://arxiv.org/abs/2201.11903)                                 | 2022 Jan |
+| You can improve step-by-step reasoning by generating many explanation-answer outputs, and picking the most popular answer      | [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://arxiv.org/abs/2203.11171)                               | 2022 Mar |
+| If you want to fine-tune a step-by-step reasoner, you can do it with multiple-choice question & answer data alone              | [STaR: Bootstrapping Reasoning With Reasoning](https://arxiv.org/abs/2203.14465)                                                          | 2022 Mar |
+| The step-by-step reasoning method works great even with zero examples                                                          | [Large Language Models are Zero-Shot Reasoners](https://arxiv.org/abs/2205.11916)                                                         | 2022 May |
+| You can do better than step-by-step reasoning by alternating a ‘selection’ prompt and an ‘inference’ prompt                    | [Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning](https://arxiv.org/abs/2205.09712)             | 2022 May |
+| On long reasoning problems, you can improve step-by-step reasoning by splitting the problem into pieces to solve incrementally | [Least-to-most Prompting Enables Complex Reasoning in Large Language Models](https://arxiv.org/abs/2205.10625)                            | 2022 May |
+| You can have the model analyze both good and bogus explanations to figure out which set of explanations are most consistent    | [Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations](https://arxiv.org/abs/2205.11822)                        | 2022 May |
+| You can think about these techniques in terms of probabilistic programming, where systems comprise unreliable components       | [Language Model Cascades](https://arxiv.org/abs/2207.10342)                                                                               | 2022 Jul |
+| You can eliminate hallucination with sentence label manipulation, and you can reduce wrong answers with a 'halter' prompt      | [Faithful Reasoning Using Large Language Models](https://arxiv.org/abs/2208.14271)                                                        | 2022 Aug |

articles/text_comparison_examples.md

@@ -0,0 +1,51 @@
+# Text comparison examples
+
+The [OpenAI API embeddings endpoint](https://beta.openai.com/docs/guides/embeddings) can be used to measure relatedness or similarity between pieces of text.
+
+By leveraging GPT-3's understanding of text, these embeddings [achieved state-of-the-art results](https://arxiv.org/abs/2201.10005) on benchmarks in unsupervised learning and transfer learning settings.
+
+Embeddings can be used for semantic search, recommendations, cluster analysis, near-duplicate detection, and more.
+
+For more information, read OpenAI's blog post announcements:
+
+- [Introducing Text and Code Embeddings (Jan 2022)](https://openai.com/blog/introducing-text-and-code-embeddings/)
+- [New and Improved Embedding Model (Dec 2022)](https://openai.com/blog/new-and-improved-embedding-model/)
+
+For comparison with other embedding models, see [Massive Text Embedding Benchmark (MTEB) Leaderboard](https://huggingface.co/spaces/mteb/leaderboard)
+
+## Semantic search
+
+Embeddings can be used for search either by themselves or as a feature in a larger system.
+
+The simplest way to use embeddings for search is as follows:
+
+- Before the search (precompute):
+  - Split your text corpus into chunks smaller than the token limit (8,191 tokens for `text-embedding-3-small`)
+  - Embed each chunk of text
+  - Store those embeddings in your own database or in a vector search provider like [Pinecone](https://www.pinecone.io), [Weaviate](https://weaviate.io) or [Qdrant](https://qdrant.tech)
+- At the time of the search (live compute):
+  - Embed the search query
+  - Find the closest embeddings in your database
+  - Return the top results
+
+An example of how to use embeddings for search is shown in [Semantic_text_search_using_embeddings.ipynb](../examples/Semantic_text_search_using_embeddings.ipynb).
+
+In more advanced search systems, the cosine similarity of embeddings can be used as one feature among many in ranking search results.
+
+## Question answering
+
+The best way to get reliably honest answers from GPT-3 is to give it source documents in which it can locate correct answers. Using the semantic search procedure above, you can cheaply search through a corpus of documents for relevant information and then give that information to GPT-3 via the prompt to answer a question. We demonstrate this in [Question_answering_using_embeddings.ipynb](../examples/Question_answering_using_embeddings.ipynb).
+
+## Recommendations
+
+Recommendations are quite similar to search, except that instead of a free-form text query, the inputs are items in a set.
+
+An example of how to use embeddings for recommendations is shown in [Recommendation_using_embeddings.ipynb](../examples/Recommendation_using_embeddings.ipynb).
+
+Similar to search, these cosine similarity scores can either be used on their own to rank items or as features in larger ranking algorithms.
+
+## Customizing Embeddings
+
+Although OpenAI's embedding model weights cannot be fine-tuned, you can nevertheless use training data to customize embeddings to your application.
+
+In [Customizing_embeddings.ipynb](../examples/Customizing_embeddings.ipynb), we provide an example method for customizing your embeddings using training data. The idea of the method is to train a custom matrix to multiply embedding vectors by in order to get new customized embeddings. With good training data, this custom matrix will help emphasize the features relevant to your training labels. You can equivalently consider the matrix multiplication as (a) a modification of the embeddings or (b) a modification of the distance function used to measure the distances between embeddings.

articles/what_is_new_with_dalle_3.mdx

@@ -0,0 +1,202 @@
+# What’s new with DALL·E-3?
+
+DALL·E-3 is the latest version of our DALL-E text-to-image generation models. As the current state of the art in text-to-image generation, DALL·E is capable of generating high-quality images across a wide variety of domains. If you're interested in more technical details of how DALL·E-3 was built, you can read more about in our [research paper](https://cdn.openai.com/papers/dall-e-3.pdf). I'll be going over some of the new features and capabilities of DALL·E-3 in this article, as well as some examples of what new products you can build with the API.
+
+As a reminder, the Image generation API hasn't changed and maintains the same endpoints and formatting as with DALL·E-2. If you're looking for a guide on how to use the Image API, see [the Cookbook article](https://cookbook.openai.com/examples/dalle/image_generations_edits_and_variations_with_dall-e) on the subject.
+
+The only API endpoint available for use with DALL·E-3 right now is **Generations** (/v1/images/generations). We don’t support variations or inpainting yet, though the Edits and Variations endpoints are available for use with DALL·E-2.
+
+## Generations
+
+The generation API endpoint creates an image based on a text prompt. There’s a couple new parameters that we've added to enhance what you can create with our models. Here’s a quick overview of the options:
+
+### New parameters:
+
+- **model** (‘dall-e-2’ or ‘dall-e-3’): This is the model you’re generating with. Be careful to set it to ‘dall-e-3’ as it defaults to ‘dall-e-2’ if empty.
+- **style** (‘natural’ or ‘vivid’): The style of the generated images. Must be one of vivid or natural. Vivid causes the model to lean towards generating hyper-real and dramatic images. Natural causes the model to produce more natural, less hyper-real looking images. Defaults to ‘vivid’.
+- **quality** (‘standard’ or ‘hd’): The quality of the image that will be generated. ‘hd’ creates images with finer details and greater consistency across the image. Defaults to ‘standard’.
+
+### Other parameters:
+
+- **prompt** (str): A text description of the desired image(s). The maximum length is 1000 characters. Required field.
+- **n** (int): The number of images to generate. Must be between 1 and 10. Defaults to 1. For dall-e-3, only n=1 is supported.
+- **size** (...): The size of the generated images. Must be one of 256x256, 512x512, or 1024x1024 for DALL·E-2 models. Must be one of 1024x1024, 1792x1024, or 1024x1792 for DALL·E-3 models.
+- **response_format** ('url' or 'b64_json'): The format in which the generated images are returned. Must be one of "url" or "b64_json". Defaults to "url".
+- **user** (str): A unique identifier representing your end-user, which will help OpenAI to monitor and detect abuse. Learn more.
+
+## New Features
+
+Our launch of DALL·E-3 comes with lots of new features and capabilities to help you generate the images you want. Here’s a quick overview of what’s new:
+
+### Prompt Rewriting
+
+A new feature in the latest DALL·E-3 API is prompt rewriting, where we use GPT-4 to optimize all of your prompts before they’re passed to DALL-E. In our research, we’ve seen that using very detailed prompts give significantly better results. You can read more about our captioning, prompting, and safety mitigations in the [DALL·E-3 research paper](https://cdn.openai.com/papers/dall-e-3.pdf).
+
+_Keep in mind that this feature isn’t able to be disabled at the moment, though you can achieve a high level of fidelity by simply giving instructions to the relabeler in your prompt, as I'll show below with examples._
+
+![Prompt Rewriting](/images/dalle_3/dalle_3_improved_prompts.png)
+
+### Standard vs HD Quality
+
+DALL·E-3 introduces a new 'quality' parameter that allows you to adjust the level of detail and organization in all of your generations. The 'standard' quality generations are the DALL·E-3 you're familiar with, with 'hd' generations bringing a new level of attention to detail and adherence to your prompt. Keep in mind that setting your generation quality to ‘hd’ does increase the cost per image, as well as often increasing the time it takes to generate by ~10 seconds or so.
+
+For example, here we have two different icons in 'hd' and 'standard' quality. Often the choice between either quality is up to taste, but 'hd' often wins when the task requires more ability to capture details and textures or better composition of a scene.
+
+![Icons](/images/dalle_3/icons.jpg)
+
+Here's another example, this time with a prompt of 'An infinite, uniform grid of tessellated cubes.', which DALL·E conveniently rewrites as _"An infinite, uniform grid of tessellated cubes painted carefully in an isometric perspective. The cubes are meticulously arranged in such a way that they seem to stretch endlessly into the distance. Each cube is identical to the next, with light reflecting consistently across all surfaces, underscoring their uniformity. This is a digitally rendered image."_:
+
+![Cubes](/images/dalle_3/cubes.jpg)
+
+### New Sizes
+
+DALL·E-3 accepts three different image sizes: 1024px by 1024px, 1792px by 1024px, and 1024px by 1792px. Beyond giving more flexibility in terms of aspect ratio, these sizes can have significant effects on the style and context of your generated image. For example, vertical images might work better when you’re looking for an image that looks like it was taken by a cellphone camera, or horizontal images may work better for landscape paintings or digital designs.
+
+To demonstrate this difference, here’s multiple variations on the same input prompt with a different aspect ratio. In this case, my prompt was: “Professional photoshoot of a Chemex brewer in the process of brewing coffee.” (For reference, this is a photo of [a real Chemex brewer](https://m.media-amazon.com/images/I/61lrld81vxL.jpg)).
+
+Here is the generation in square form (in both HD and standard qualities):
+
+![square_coffee](/images/dalle_3/square_coffee.jpg)
+
+You can see how these images are framed closely to the item and seem to be taken in a more closed space with various surrounding items nearby.
+
+Here are the results on the same prompts with a wider aspect ratio:
+
+![wide_coffee](/images/dalle_3/wide_coffee.jpg)
+
+Compared to the previous generations, these come in the form of close-ups. The background is blurred, with greater focus on the item itself, more like professionally organized photoshoots rather than quick snaps.
+
+Lastly, we have the vertical aspect ratio:
+
+![tall_coffee](/images/dalle_3/tall_coffee.jpg)
+
+These feel more akin to cellphone images, with a more candid appearance. There’s more action involved: the slowly dripping coffee or the active pour from the pot.
+
+### New Styles
+
+DALL·E-3 introduces two new styles: natural and vivid. The natural style is more similar to the DALL·E-2 style in its 'blander' realism, while the vivid style is a new style that leans towards generating hyper-real and cinematic images. For reference, all DALL·E generations in ChatGPT are generated in the 'vivid' style.
+
+The natural style is specifically useful in cases where DALL·E-3 over-exaggerates or confuses a subject that's supposed to be more simple, subdued, or realistic. I've often used it for logo generation, stock photos, or other cases where I'm trying to match a real-world object.
+
+Here's an example of the same prompt as above in the vivid style. The vivid is far more cinematic (and looks great), but might pop too much if you're not looking for that.
+
+![vivid_coffee](/images/dalle_3/vivid_coffee.jpg)
+
+There's many cases in which I prefer the natural style, such as this example of a painting in the style of Thomas Cole's 'Desolation':
+
+![thomas_cole](/images/dalle_3/thomas_cole.jpg)
+
+## Examples and Prompts
+
+To help you get started building with DALL·E-3, I've come up with a few examples of products you could build with the API, as well as collected some styles and capabilities that seem to be unique to DALL·E-3 at the moment. I've also listed some subjects that I'm struggling to prompt DALL·E-3 to generate in case you want to try your hand at it.
+
+### Icon Generation
+
+Have you ever struggled to find the perfect icon for your website or app? It would be awesome to see a custom icon generator app that lets you pick the style, size, and subject of your icon, and then generates a custom SVG from the DALL·E generation. Here's some examples of helpful website icons I generated with DALL·E-3:
+
+![icon_set](/images/dalle_3/icon_set.jpg)
+
+In this case, I used Potrace to convert the images to SVGs, which you can download [here](https://potrace.sourceforge.net/). This is what I used to convert the images:
+
+```bash
+potrace -s cat.jpg -o cat.svg
+```
+
+You might need to boost the brightness and contrast of the image before converting it to an SVG. I used the following command to do so:
+
+```bash
+convert cat.jpg -brightness-contrast 50x50 cat.jpg
+```
+
+### Logo Generation
+
+DALL·E-3 is great at jumpstarting the logo creation process for your company or product. By prompting DALL·E to create 'Vector logo design of a Greek statue, minimalistic, with a white background' I achieved the following:
+
+![logo_greece](/images/dalle_3/logo_greece.jpg)
+
+Here's another logo I created, this time for an Arabian coffee shop:
+
+![logo_arabia](/images/dalle_3/logo_arabia.jpg)
+
+In the case of iterating on an existing logo, I took OpenAI's logo, asked GPT-4V to describe it, and then asked DALL·E to generate variations on the logo:
+
+![iteration](/images/dalle_3/iteration.jpg)
+
+### Custom Tattoos
+
+DALL·E-3 is great at generating line art, which might be useful for generating custom tattoos. Here's some line art I generated with DALL·E-3:
+
+![tattoos](/images/dalle_3/tattoos.jpg)
+
+### Die-Cut Stickers & T-Shirts
+
+What if you could generate custom die-cut stickers and t-shirts with DALL·E-3, integrating with a print-on-demand service like Printful or Stickermule? You could have a custom sticker or t-shirt in minutes, with no design experience required. Here's some examples of stickers I generated with DALL·E-3:
+
+![stickers](/images/dalle_3/stickers.jpg)
+
+### Minecraft Skins
+
+With some difficulty, I managed to prompt DALL·E-3 to generate Minecraft skins. I'm sure with some clever prompting you could get DALL·E-3 to reliably generate incredible Minecraft skins. It might be hard to use the words 'Minecraft' since DALL·E might think you are trying to generate content from the game itself, instead, you can communicate the idea differently: "Flat player skin texture of a ninja skin, compatible with Minecraftskins.com or Planet Minecraft."
+
+Here's what I managed to create. They might need some work, but I think they're a good start:
+
+![minecraft](/images/dalle_3/minecraft.jpg)
+
+### And much more...
+
+Here's some ideas I've had that I haven't had time to try yet:
+
+- Custom emojis or Twitch emotes?
+- Vector illustrations?
+- Personalized Bitmoji-style avatars?
+- Album art?
+- Custom greeting cards?
+- Poster/flyer 'pair-programming' with DALL·E?
+
+## Showcase
+
+We're really just starting to figure out what DALL·E-3 is capable of. Here's some of the best styles, generations, and prompts I've seen so far. I've been unable to locate the original authors of some of these images, so if you know who created them, please let me know!
+
+![collage](/images/dalle_3/collage.jpg)
+
+Sources:
+
+[@scharan79 on Reddit](https://www.reddit.com/r/dalle2/comments/170ce1r/dalle_3_is_pretty_good_at_drawing/)  
+[@TalentedJuli on Reddit](https://www.reddit.com/r/dalle2/comments/1712x7a/60s_pulp_magazine_illustration_is_the_best_style/)  
+[@Wild-Culture-5068 on Reddit](https://www.reddit.com/r/dalle2/comments/17dwp0s/soviet_blade_runner/)  
+[@popsicle_pope on Reddit](https://www.reddit.com/r/dalle2/comments/170lx1z/%F0%9D%94%AA%F0%9D%94%A2%F0%9D%94%B1%F0%9D%94%9E%F0%9D%94%AA%F0%9D%94%AC%F0%9D%94%AF%F0%9D%94%AD%F0%9D%94%A5%F0%9D%94%AC%F0%9D%94%B0%F0%9D%94%A6%F0%9D%94%B0/)  
+[@gopatrik on Twitter](https://twitter.com/gopatrik/status/1717579802205626619)  
+[@ARTiV3RSE on Twitter](https://twitter.com/ARTiV3RSE/status/1720202013638599040)  
+[@willdepue on Twitter](https://twitter.com/willdepue/status/1705677997150445941)  
+Various OpenAI employees
+
+## Challenges
+
+DALL·E-3 is still very new and there's still a lot of things it struggles with (or maybe I just haven't figured out how to prompt it correctly yet). Here's some challenges which you might want to try your hand at:
+
+### Web Design
+
+DALL·E really struggles at generating real looking websites, apps, etc. and often generates what looks like a portfolio page of a web designer. Here's the best I've gotten so far:
+
+![websites](/images/dalle_3/websites.jpg)
+
+### Seamless Textures
+
+It feels like DALL·E-3 is so close to being able to generate seamless textures. Often they come out great, just slightly cutoff or with a few artifacts. See examples below:
+
+![seamless](/images/dalle_3/seamless.jpg)
+
+### Fonts
+
+Using DALL·E to generate custom fonts or iterate on letter designs could be really cool, but I haven't been able to get it to work yet. Here's the best I've gotten so far:
+
+![fonts](/images/dalle_3/fonts.jpg)
+
+## More Resources
+
+Thanks for reading! If you're looking for more resources on DALL·E-3, here are some related links:
+
+- [DALL·E-3 Blog Post](https://openai.com/dall-e-3)
+- [DALL·E-3 Research Paper](https://cdn.openai.com/papers/dall-e-3.pdf)
+- [Image API Documentation](https://platform.openai.com/docs/api-reference/images)
+- [Image API Cookbook](https://cookbook.openai.com/examples/dalle/image_generations_edits_and_variations_with_dall-e)

articles/what_makes_documentation_good.md

@@ -0,0 +1,65 @@
+# What makes documentation good
+
+Documentation puts useful information inside other people’s heads. Follow these tips to write better documentation.
+
+### Make docs easy to skim
+
+Few readers read linearly from top to bottom. They’ll jump around, trying to assess which bit solves their problem, if any. To reduce their search time and increase their odds of success, make docs easy to skim.
+
+**Split content into sections with titles.** Section titles act as signposts, telling readers whether to focus in or move on.
+
+**Prefer titles with informative sentences over abstract nouns.** For example, if you use a title like “Results”, a reader will need to hop into the following text to learn what the results actually are. In contrast, if you use the title “Streaming reduced time to first token by 50%”, it gives the reader the information immediately, without the burden of an extra hop.
+
+**Include a table of contents.** Tables of contents help readers find information faster, akin to how hash maps have faster lookups than linked lists. Tables of contents also have a second, oft overlooked benefit: they give readers clues about the doc, which helps them understand if it’s worth reading.
+
+**Keep paragraphs short.** Shorter paragraphs are easier to skim. If you have an essential point, consider putting it in its own one-sentence paragraph to reduce the odds it’s missed. Long paragraphs can bury information.
+
+**Begin paragraphs and sections with short topic sentences that give a standalone preview.** When people skim, they look disproportionately at the first word, first line, and first sentence of a section. Write these sentences in a way that don’t depend on prior text. For example, consider the first sentence “Building on top of this, let’s now talk about a faster way.” This sentence will be meaningless to someone who hasn’t read the prior paragraph. Instead, write it in a way that can understood standalone: e.g., “Vector databases can speed up embeddings search.”
+
+**Put topic words at the beginning of topic sentences.** Readers skim most efficiently when they only need to read a word or two to know what a paragraph is about. Therefore, when writing topic sentences, prefer putting the topic at the beginning of the sentence rather than the end. For example, imagine you’re writing a paragraph on vector databases in the middle of a long article on embeddings search. Instead of writing “Embeddings search can be sped up by vector databases” prefer “Vector databases speed up embeddings search.” The second sentence is better for skimming, because it puts the paragraph topic at the beginning of the paragraph.
+
+**Put the takeaways up front.** Put the most important information at the tops of documents and sections. Don’t write a Socratic big build up. Don’t introduce your procedure before your results.
+
+**Use bullets and tables.** Bulleted lists and tables make docs easier to skim. Use them frequently.
+
+**Bold important text.** Don’t be afraid to bold important text to help readers find it.
+
+### Write well
+
+Badly written text is taxing to read. Minimize the tax on readers by writing well.
+
+**Keep sentences simple.** Split long sentences into two. Cut adverbs. Cut unnecessary words and phrases. Use the imperative mood, if applicable. Do what writing books tell you.
+
+**Write sentences that can be parsed unambiguously.** For example, consider the sentence “Title sections with sentences.” When a reader reads the word “Title”, their brain doesn’t yet know whether “Title” is going to be a noun or verb or adjective. It takes a bit of brainpower to keep track as they parse the rest of the sentence, and can cause a hitch if their brain mispredicted the meaning. Prefer sentences that can be parsed more easily (e.g., “Write section titles as sentences”) even if longer. Similarly, avoid noun phrases like “Bicycle clearance exercise notice” which can take extra effort to parse.
+
+**Avoid left-branching sentences.** Linguistic trees show how words relate to each other in sentences. Left-branching trees require readers to hold more things in memory than right-branching sentences, akin to breadth-first search vs depth-first search. An example of a left-branching sentence is “You need flour, eggs, milk, butter and a dash of salt to make pancakes.” In this sentence you don’t find out what ‘you need’ connects to until you reach the end of the sentence. An easier-to-read right-branching version is “To make pancakes, you need flour, eggs, milk, butter, and a dash of salt.” Watch out for sentences in which the reader must hold onto a word for a while, and see if you can rephrase them.
+
+**Avoid demonstrative pronouns (e.g., “this”), especially across sentences.** For example, instead of saying “Building on our discussion of the previous topic, now let’s discuss function calling” try “Building on message formatting, now let’s discuss function calling.” The second sentence is easier to understand because it doesn’t burden the reader with recalling the previous topic. Look for opportunities to cut demonstrative pronouns altogether: e.g., “Now let’s discuss function calling.”
+
+**Be consistent.** Human brains are amazing pattern matchers. Inconsistencies will annoy or distract readers. If we use Title Case everywhere, use Title Case. If we use terminal commas everywhere, use terminal commas. If all of the Cookbook notebooks are named with underscores and sentence case, use underscores and sentence case. Don’t do anything that will cause a reader to go ‘huh, that’s weird.’ Help them focus on the content, not its inconsistencies.
+
+**Don’t tell readers what they think or what to do.** Avoid sentences like “Now you probably want to understand how to call a function” or “Next, you’ll need to learn to call a function.” Both examples presume a reader’s state of mind, which may annoy them or burn our credibility. Use phrases that avoid presuming the reader’s state. E.g., “To call a function, …”
+
+### Be broadly helpful
+
+People come to documentation with varying levels of knowledge, language proficiency, and patience. Even if we target experienced developers, we should try to write docs helpful to everyone.
+
+**Write simply.** Explain things more simply than you think you need to. Many readers might not speak English as a first language. Many readers might be really confused about technical terminology and have little excess brainpower to spend on parsing English sentences. Write simply. (But don’t oversimplify.)
+
+**Avoid abbreviations.** Write things out. The cost to experts is low and the benefit to beginners is high. Instead of IF, write instruction following. Instead of RAG, write retrieval-augmented generation (or my preferred term: the search-ask procedure).
+
+**Offer solutions to potential problems.** Even if 95% of our readers know how to install a Python package or save environment variables, it can still be worth proactively explaining it. Including explanations is not costly to experts—they can skim right past them. But excluding explanations is costly to beginners—they might get stuck or even abandon us. Remember that even an expert JavaScript engineer or C++ engineer might be a beginner at Python. Err on explaining too much, rather than too little.
+
+**Prefer terminology that is specific and accurate.** Jargon is bad. Optimize the docs for people new to the field, instead of ourselves. For example, instead of writing “prompt”, write “input.” Or instead of writing “context limit” write “max token limit.” The latter terms are more self-evident, and are probably better than the jargon developed in base model days.
+
+**Keep code examples general and exportable.** In code demonstrations, try to minimize dependencies. Don’t make users install extra libraries. Don’t make them have to refer back and forth between different pages or sections. Try to make examples simple and self-contained.
+
+**Prioritize topics by value.** Documentation that covers common problems—e.g., how to count tokens—is magnitudes more valuable than documentation that covers rare problems—e.g., how to optimize an emoji database. Prioritize accordingly.
+
+**Don’t teach bad habits.** If API keys should not be stored in code, never share an example that stores an API key in code.
+
+**Introduce topics with a broad opening.** For example, if explaining how to program a good recommender, consider opening by briefly mentioning that recommendations are widespread across the web, from YouTube videos to Amazon items to Wikipedia. Grounding a narrow topic with a broad opening can help people feel more secure before jumping into uncertain territory. And if the text is well-written, those who already know it may still enjoy it.
+
+### Break these rules when you have a good reason
+
+Ultimately, do what you think is best. Documentation is an exercise in empathy. Put yourself in the reader’s position, and do what you think will help them the most.

authors.yaml

@@ -0,0 +1,624 @@
+# yaml-language-server: $schema=./.github/authors_schema.json
+
+# You can optionally customize how your information shows up cookbook.openai.com over here.
+# If your information is not present here, it will be pulled from your GitHub profile.
+
+neilh-oai:
+  name: "Neil Hansaria"
+  website: "https://www.linkedin.com/in/neilhansaria"
+  avatar: "https://avatars.githubusercontent.com/u/274336127?v=4"
+
+kkahadze-oai:
+  name: "Konstantine Kahadze"
+  website: "https://kahadze.com"
+  avatar: "https://avatars.githubusercontent.com/kkahadze-oai"
+
+lpesut-daytona:
+  name: "Lovre Pesut"
+  website: "https://github.com/rovle"
+  avatar: "https://avatars.githubusercontent.com/rovle"
+
+zhenweig-cerebras:
+  name: "Zhenwei Gao"
+  website: "https://www.linkedin.com/in/zhenwei-gao/"
+  avatar: "https://avatars.githubusercontent.com/u/240548509?v=4"
+
+joanneshin-openai:
+  name: "Joanne Shin"
+  website: "https://www.linkedin.com/in/jeongminshin"
+  avatar: "https://avatars.githubusercontent.com/u/206945323?v=4"
+
+daveleo-openai:
+  name: "Dave Leo"
+  website: "https://www.linkedin.com/in/davidanthonyleo/"
+  avatar: "https://media.licdn.com/dms/image/v2/C5603AQF2Kg-D7XJKNw/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1612654752234?e=1761782400&v=beta&t=RkO9jCbJrY6Ox9YRbMA6HAAZhxfYJV1OsZeIT3YatBM"
+
+jonlim-openai:
+  name: "Jonathan Lim"
+  website: "https://www.linkedin.com/in/jonlmr"
+  avatar: "https://avatars.githubusercontent.com/u/189068472?v=4"
+
+WJPBProjects:
+  name: "Wulfie Bain"
+  website: "https://www.linkedin.com/in/wulfie-bain/"
+  avatar: "https://avatars.githubusercontent.com/u/76624567?v=4"
+
+anoop-openai:
+  name: "Anoop Kotha"
+  website: "https://x.com/anoopkotha"
+  avatar: "https://pbs.twimg.com/profile_images/1953388055190224896/vHZ5CVPs_400x400.jpg"
+
+erinkav-openai:
+  name: "Erin Kavanaugh"
+  website: "https://www.linkedin.com/in/erinkavanaugh/"
+  avatar: "https://media.licdn.com/dms/image/v2/D5603AQFBeFl_GBWRww/profile-displayphoto-scale_400_400/B56ZiCtx6zHcAg-/0/1754539697289?e=1757548800&v=beta&t=e3znvOjyDV9MBdp2AABxGtvdCx1LzyrBR75cjOSh5h8"
+
+ericzakariasson:
+  name: "Eric Zakariasson"
+  website: "https://x.com/ericzakariasson"
+  avatar: "https://pbs.twimg.com/profile_images/1841151626456801283/HnXqy3TQ_400x400.jpg"
+
+jayrodge:
+  name: "Jay Rodge"
+  website: "https://www.linkedin.com/in/jayrodge/"
+  avatar: "https://developer-blogs.nvidia.com/wp-content/uploads/2024/05/Jay-Rodge.png"
+
+rajpathak-openai:
+  name: "Raj Pathak"
+  website: "https://www.linkedin.com/in/rajpathakopenai/"
+  avatar: "https://avatars.githubusercontent.com/u/208723614?s=400&u=c852eed3be082f7fbd402b5a45e9b89a0bfed1b8&v=4"
+
+emreokcular:
+  name: "Emre Okcular"
+  website: "https://www.linkedin.com/in/emreokcular/"
+  avatar: "https://avatars.githubusercontent.com/u/26163154?v=4"
+
+chelseahu-openai:
+  name: "Chelsea Hu"
+  website: "https://www.linkedin.com/in/chelsea-tsaiszuhu/"
+  avatar: "https://avatars.githubusercontent.com/u/196863678?v=4"
+
+prashantmital-openai:
+  name: "Prashant Mital"
+  website: "https://www.linkedin.com/in/pmital/"
+  avatar: "https://avatars.githubusercontent.com/u/173949238?v=4"
+
+theophile-oai:
+  name: "Theophile Sautory"
+  website: "https://www.linkedin.com/in/theophilesautory"
+  avatar: "https://avatars.githubusercontent.com/u/206768658?v=4"
+
+bfioca-openai:
+  name: "Brian Fioca"
+  website: "https://www.linkedin.com/in/brian-fioca/"
+  avatar: "https://avatars.githubusercontent.com/u/206814564?v=4"
+
+carter-oai:
+  name: "Carter Mcclellan"
+  website: "https://www.linkedin.com/in/carter-mcclellan/"
+  avatar: "https://avatars.githubusercontent.com/u/219906258?v=4"
+
+robert-tinn:
+  name: "Robert Tinn"
+  website: "https://www.linkedin.com/in/robert-tinn/"
+  avatar: "https://avatars.githubusercontent.com/u/208724428?v=4"
+
+minh-hoque:
+  name: "Minhajul Hoque"
+  website: "https://www.linkedin.com/in/minhajul-hoque-83242b163/"
+  avatar: "https://avatars.githubusercontent.com/u/84698472?v=4"
+
+shikhar-cyber:
+  name: "Shikhar Kwatra"
+  website: "https://www.linkedin.com/in/shikharkwatra/"
+  avatar: "https://avatars.githubusercontent.com/u/189049238?v=4"
+  
+sudee-oai:
+  name: "Sudeesh Sasidharan"
+  website: "https://www.linkedin.com/in/sudeesh-sasidharan/"
+  avatar: "https://avatars.githubusercontent.com/sudee-oai"
+
+danbell-openai:
+  name: "Dan Bell"
+  website: "https://www.linkedin.com/in/dan-bell-b69721b1/"
+  avatar: "https://avatars.githubusercontent.com/u/201846729?v=4"
+
+billchen-openai:
+  name: "Bill Chen"
+  website: "https://www.linkedin.com/in/billchen99/"
+  avatar: "https://avatars.githubusercontent.com/u/198814448?v=4"
+
+0hq:
+  name: "Will Depue"
+  website: "http://depue.design"
+  avatar: "https://pbs.twimg.com/profile_images/1619879636057595904/NdjEAqPm_400x400.jpg"
+
+cathykc:
+  name: "Kai Chen"
+  website: "https://github.com/cathykc"
+  avatar: "https://pbs.twimg.com/profile_images/1657816900817272832/ioGq5O0t_400x400.jpg"
+
+shyamal-anadkat:
+  name: "Shyamal Anadkat"
+  website: "https://twitter.com/shyamalanadkat"
+  avatar: "https://pbs.twimg.com/profile_images/1590564338682560512/3bbZJqxZ_400x400.jpg"
+
+aaronfriel:
+  name: "Aaron Friel"
+  website: "https://www.github.com/aaronfriel"
+  avatar: "https://avatars.githubusercontent.com/aaronfriel"
+
+simonpfish:
+  name: "Simón Fishman"
+  website: "https://twitter.com/simonpfish"
+  avatar: "https://pbs.twimg.com/profile_images/1687643030243901441/61DpxNfc_400x400.jpg"
+
+joe-at-openai:
+  name: "Joe Palermo"
+  website: "https://www.linkedin.com/in/joe-palermo-99219237"
+  avatar: "https://avatars.githubusercontent.com/u/117690718?v=4"
+
+mwu1993:
+  name: "Michael Wu"
+  website: "https://www.linkedin.com/in/michael-wu-77440977/"
+  avatar: "https://avatars.githubusercontent.com/u/1650674?v=4"
+
+narenoai:
+  name: "Naren Sankaran"
+  website: "https://www.linkedin.com/in/snarendran/"
+  avatar: "https://avatars.githubusercontent.com/u/196844623?s=400&u=d669669fd962473d606a97801367ba96fc548287&v=4"
+
+ibigio:
+  name: "Ilan Bigio"
+  website: "https://twitter.com/ilanbigio"
+  avatar: "https://pbs.twimg.com/profile_images/1841544725654077440/DR3b8DMr_400x400.jpg"
+
+willhath-openai:
+  name: "Will Hathaway"
+  website: "https://www.willhath.com"
+  avatar: "https://media.licdn.com/dms/image/v2/D4E03AQEHOtMrHtww4Q/profile-displayphoto-shrink_200_200/B4EZRR64p9HgAc-/0/1736541178829?e=2147483647&v=beta&t=w1rX0KhLZaK5qBkVLkJjmYmfNMbsV2Bcn8InFVX9lwI"
+
+jhills20:
+  name: "James Hills"
+  website: "https://twitter.com/jamesmhills"
+  avatar: "https://pbs.twimg.com/profile_images/1722092156691902464/44FGj7VT_400x400.jpg"
+
+colin-openai:
+  name: "Colin Jarvis"
+  website: "https://twitter.com/colintjarvis"
+  avatar: "https://pbs.twimg.com/profile_images/1727207339034345472/IM8v8tlC_400x400.jpg"
+
+prakul:
+  name: "Prakul Agarwal"
+  website: "https://www.linkedin.com/in/prakulagarwal"
+  avatar: "https://media.licdn.com/dms/image/D5603AQEUug83qKgRBg/profile-displayphoto-shrink_800_800/0/1675384960197?e=1706140800&v=beta&t=qxkDbBr-Bk2ASpcwbR5JVPD6yS-vzmIwNHAa8ApyDq4"
+
+gaborcselle:
+  name: "Gabor Cselle"
+  website: "https://www.linkedin.com/in/gaborcselle"
+  avatar: "https://avatars.githubusercontent.com/u/54096?s=96&v=4"
+
+nghiauet:
+  name: "Nghia Pham"
+  website: "https://www.linkedin.com/in/deptraicucmanh/"
+  avatar: "https://media.licdn.com/dms/image/D5603AQFpoi0l1lRQ7g/profile-displayphoto-shrink_200_200/0/1680006955681?e=1707350400&v=beta&t=1J1DAVRGXL7LgFUBnIooI-fsDRkdzFzCDwwxMze9N8c"
+
+ggrn:
+  name: "Greg Richardson"
+  website: "https://twitter.com/ggrdson"
+  avatar: "https://pbs.twimg.com/profile_images/1371549909820657664/ZG-HDNlI_400x400.jpg"
+
+teomusatoiu:
+  name: "Teodora Musatoiu"
+  website: "https://www.linkedin.com/in/teodora-musatoiu/"
+  avatar: "https://avatars.githubusercontent.com/u/156829031?s=400&u=af40fe04d9255139eb3bbf8dc83422cc694e862b&v=4"
+
+katiagg:
+  name: "Katia Gil Guzman"
+  website: "https://katia.gg"
+  avatar: "https://avatars.githubusercontent.com/u/16519462?v=4"
+
+jbeutler-openai:
+  name: "Joe Beutler"
+  website: "https://joebeutler.com"
+  avatar: "https://avatars.githubusercontent.com/u/156261485?v=4"
+
+dylanra-openai:
+  name: "Dylan Royan Almeida"
+  website: "https://www.linkedin.com/in/dylan-almeida-604522167/"
+  avatar: "https://avatars.githubusercontent.com/u/149511600?v=4"
+
+royziv11:
+  name: "Roy Ziv"
+  website: "https://www.linkedin.com/in/roy-ziv-a46001149/"
+  avatar: "https://avatars.githubusercontent.com/u/103690170?v=4"
+
+justonf:
+  name: "Juston Forte"
+  website: "https://www.linkedin.com/in/justonforte/"
+  avatar: "https://avatars.githubusercontent.com/u/96567547?s=400&u=08b9757200906ab12e3989b561cff6c4b95a12cb&v=4"
+
+aaronwilkowitz-openai:
+  name: "Aaron Wilkowitz"
+  website: "https://www.linkedin.com/in/aaronwilkowitz/"
+  avatar: "https://avatars.githubusercontent.com/u/157151487"
+
+stefanofabbri-oai:
+  name: "Stefano Fabbri"
+  website: "https://www.linkedin.com/in/stefano-fabbri-633a1895/"
+  avatar: "https://media.licdn.com/dms/image/v2/C5603AQGekzEY9dhxXA/profile-displayphoto-shrink_400_400/profile-displayphoto-shrink_400_400/0/1639500546853?e=1779926400&v=beta&t=siVg-MLpr70UfpUYhcEg86sxqcKSOU-jC7V4iT4FtK0"
+
+charuj:
+  name: "Charu Jaiswal"
+  website: "https://www.linkedin.com/in/charu-j-8a866471"
+  avatar: "https://avatars.githubusercontent.com/u/18404643?v=4"
+
+rupert-openai:
+  name: "Rupert Truman"
+  website: "https://www.linkedin.com/in/rupert-truman/"
+  avatar: "https://avatars.githubusercontent.com/u/171234447"
+
+keelan-openai:
+  name: "Keelan Schule"
+  website: "https://www.linkedin.com/in/keelanschule/"
+  avatar: "https://avatars.githubusercontent.com/u/175734151?v=4"
+
+aa-openai:
+  name: "Allison August"
+  website: "https://www.linkedin.com/in/allisonaugust/"
+  avatar: "https://avatars.githubusercontent.com/u/172655668?v=4"
+
+alwestmo-openai:
+  name: "Anthony Westmoreland"
+  website: "https://www.linkedin.com/in/westmorelandanthony/"
+  avatar: "https://avatars2.githubusercontent.com/u/175844346"
+
+dan-openai:
+  name: "Daniel Halpern"
+  website: "https://www.linkedin.com/in/dhalp/"
+  avatar: "https://avatars.githubusercontent.com/u/175367970"
+
+eszuhany-openai:
+  name: "Eric Szuhany"
+  website: "https://www.linkedin.com/in/szuhany/"
+  avatar: "https://avatars.githubusercontent.com/u/164391912"
+
+pap-openai:
+  name: "Pierre-Antoine Porte"
+  website: "https://www.linkedin.com/in/portepa/"
+  avatar: "https://avatars.githubusercontent.com/u/174109416?v=4"
+
+evanweiss-openai:
+  name: "Evan Weiss"
+  website: "https://www.linkedin.com/in/evanpweiss/"
+  avatar: "https://avatars.githubusercontent.com/u/150647345"
+
+girishd:
+  name: "Girish Dusane"
+  website: "https://www.linkedin.com/in/girishdusane/"
+  avatar: "https://avatars.githubusercontent.com/u/272708"
+
+lxing-oai:
+  name: "Luke Xing"
+  website: "https://www.linkedin.com/in/lukexing/"
+  avatar: "https://avatars.githubusercontent.com/u/176698727"
+
+gladstone-openai:
+  name: "Kevin Gladstone"
+  website: "https://github.com/gladstone-openai"
+  avatar: "https://avatars.githubusercontent.com/u/149190645"
+
+lspacagna-oai:
+  name: "Lee Spacagna"
+  website: "https://github.com/lspacagna-oai"
+  avatar: "https://avatars.githubusercontent.com/u/175367019"
+
+joecasson-openai:
+  name: "Joe Casson"
+  website: "https://github.com/joecasson-openai"
+  avatar: "https://avatars.githubusercontent.com/u/178522625"
+
+ericning-o:
+  name: "Eric Ning"
+  website: "https://github.com/ericning-o"
+  avatar: "https://avatars.githubusercontent.com/u/182030612"
+
+zeyuangu-adzviser:
+  name: "Zeyuan Gu"
+  website: "https://www.linkedin.com/in/goo/"
+  avatar: "https://avatars.githubusercontent.com/u/25328738"
+
+alwell-kevin:
+  name: "Kevin Alwell"
+  website: "https://github.com/alwell-kevin"
+  avatar: "https://avatars.githubusercontent.com/u/26548363"
+
+danial-openai:
+  name: "Danial Mirza"
+  website: "https://github.com/danial-openai"
+  avatar: "https://avatars.githubusercontent.com/u/178343703"
+
+gbergengruen:
+  name: "Guillermo Bergengruen"
+  website: "https://github.com/gbergengruen"
+  avatar: "https://avatars.githubusercontent.com/u/140010883"
+
+ishida-openai:
+  name: "Shuhei Ishida"
+  website: "https://www.linkedin.com/in/shuhei-i/"
+  avatar: "https://avatars.githubusercontent.com/u/186007001"
+
+erickgort:
+  name: "Erick Gort"
+  website: "https://www.linkedin.com/in/erick-gort-32ab1678/"
+  avatar: "https://avatars.githubusercontent.com/u/189261906?v=4"
+
+kylecote-tray:
+  name: "Kyle Cote"
+  website: "https://github.com/kylecote-tray"
+  avatar: "https://avatars.githubusercontent.com/u/53836176"
+
+MW-OAI:
+  name: "Mitch Welzen"
+  website: "https://www.linkedin.com/in/mitchwelzen/"
+  avatar: "https://media.licdn.com/dms/image/v2/C5603AQHC8-1q4MwH1A/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1592824550774?e=1742428800&v=beta&t=3mudgDyuzNU2a4gx1gue4DPyhaui7kbB7e7U8vyOo-g"
+
+dwigg-openai:
+  name: "Danny Wigg"
+  website: "https://www.linkedin.com/in/dannywigg/"
+  avatar: "https://avatars.githubusercontent.com/u/4661060?v=4"
+
+msingh-openai:
+  name: "Mandeep Singh"
+  website: "https://github.com/msingh-openai"
+  avatar: "https://avatars.githubusercontent.com/u/168678187?v=4"
+
+akashdeepdeb:
+  name: "Akashdeep Deb"
+  website: "https://github.com/akdeb"
+  avatar: "https://avatars.githubusercontent.com/u/20175219"
+
+ted-at-openai:
+  name: "Ted Sanders"
+  website: "https://github.com/ted-at-openai"
+  avatar: "https://avatars.githubusercontent.com/u/95656834?v=4"
+
+MikeHeaton:
+  name: "Mike Heaton"
+  website: "https://github.com/MikeHeaton"
+  avatar: "https://avatars.githubusercontent.com/u/11911723?v=4"
+
+thli-openai:
+  name: "Thomas Li"
+  website: "https://www.linkedin.com/in/thli/"
+  avatar: "https://avatars.githubusercontent.com/u/189043632?v=4"
+
+erikakettleson-openai:
+  name: "Erika Kettleson"
+  website: "https://www.linkedin.com/in/erika-kettleson-85763196/"
+  avatar: "https://avatars.githubusercontent.com/u/186107044?v=4"
+
+jannik-maierhofer:
+  name: "Jannik Maierhöfer"
+  website: "https://www.linkedin.com/in/maierhoefer/"
+  avatar: "https://avatars.githubusercontent.com/u/48529566?v=4"
+
+josiah-openai:
+  name: "Josiah Grace"
+  website: "https://www.linkedin.com/in/josiahbgrace"
+  avatar: "https://avatars.githubusercontent.com/u/181146311?v=4"
+
+vishnu-oai:
+  name: "Vishnu Chopra"
+  website: "https://www.linkedin.com/in/vishnu-chopra/"
+  avatar: "https://avatars.githubusercontent.com/u/206769912?v=4"
+
+nm-openai:
+  name: "Noah MacCallum"
+  website: "https://x.com/noahmacca"
+  avatar: "https://avatars.githubusercontent.com/u/171723556"
+
+julian-openai:
+  name: "Julian Lee"
+  website: "https://x.com/julianl093"
+  avatar: "https://avatars.githubusercontent.com/u/199828632"
+
+rzhao-openai:
+  name: "Randy Zhao"
+  website: "https://www.linkedin.com/in/randy-zhao-27433616b"
+  avatar: "https://avatars.githubusercontent.com/u/208724779?v=4"
+
+brandonbaker-openai:
+  name: "Brandon Baker"
+  website: "https://www.linkedin.com/in/brandonbaker18"
+  avatar: "https://avatars.githubusercontent.com/u/208719822"
+
+tompakeman-oai:
+  name: "Tom Pakeman"
+  website: "https://www.linkedin.com/in/tom-pakeman/"
+  avatar: "https://avatars.githubusercontent.com/u/204937754"
+
+alistair-openai:
+  name: "Alistair Gillespie"
+  website: "https://www.linkedin.com/in/alistair-gillespie/"
+  avatar: "https://avatars.githubusercontent.com/u/210626148"
+
+phundal-openai:
+  name: "Patrick Hundal"
+  website: "https://www.linkedin.com/in/phundal/"
+  avatar: "https://avatars.githubusercontent.com/u/189161955"
+
+rkoenig-openai:
+  name: "Robin Koenig"
+  website: "https://robinkoenig.com"
+  avatar: "https://avatars.githubusercontent.com/u/208886811?v=4"
+
+joshbickett:
+  name: "Josh Bickett"
+  website: "https://www.linkedin.com/in/josh-bickett-4219b166/"
+  avatar: "https://avatars.githubusercontent.com/u/42594239"
+
+lupie:
+  name: "Lucie Lozinski"
+  website: "https://twitter.com/thisloops"
+  avatar: "https://avatars.githubusercontent.com/u/6293148"
+
+alexl-oai:
+  name: "Alex Lowden"
+  website: "https://www.linkedin.com/in/alex-lowden01/"
+  avatar: "https://avatars.githubusercontent.com/u/215167546"
+
+lara-openai:
+  name: "Lara Rachidi"
+  website: "https://www.linkedin.com/in/lara-rachidi/"
+  avatar: "https://avatars.githubusercontent.com/u/208707651?v=4"
+
+glojain:
+  name: "Glory Jain"
+  website: "https://www.linkedin.com/in/gloryjain/"
+  avatar: "https://media.licdn.com/dms/image/v2/C4E03AQH72n6Sm5q69Q/profile-displayphoto-shrink_400_400/profile-displayphoto-shrink_400_400/0/1557995338725?e=1756339200&v=beta&t=FGTXiCZwTZvqHCY-wd8It15EDf11Rex1oLlBKRGHNtY"
+
+corwin:
+  name: "Corwin Cheung"
+  website: "https://www.linkedin.com/in/corwincubes/"
+  avatar: "https://avatars.githubusercontent.com/u/85517581?v=4"
+
+daisyshe-oai:
+  name: "Daisy Sheng"
+  website: "https://www.linkedin.com/in/daisysheng/"
+  avatar: "https://avatars.githubusercontent.com/u/212609991?v=4"
+
+dkundel-openai:
+  name: "Dominik Kundel"
+  website: "https://www.linkedin.com/in/dkundel/"
+  avatar: "https://avatars.githubusercontent.com/u/200841172?v=4"
+
+edbeeching:
+  name: "Edward Beeching"
+  website: "https://huggingface.co/edbeeching"
+  avatar: "https://cdn-avatars.huggingface.co/v1/production/uploads/1644220542819-noauth.jpeg"
+
+qgallouedec:
+  name: "Quentin Gallouédec"
+  website: "https://huggingface.co/qgallouedec"
+  avatar: "https://cdn-avatars.huggingface.co/v1/production/uploads/1677431596830-631ce4b244503b72277fc89f.jpeg"
+
+lewtun:
+  name: "Lewis Tunstall"
+  website: "https://huggingface.co/lewtun"
+  avatar: "https://cdn-avatars.huggingface.co/v1/production/uploads/1594651707950-noauth.jpeg"
+
+pcuenca:
+  name: "Pedro Cuenca"
+  website: "https://huggingface.co/pcuenq"
+  avatar: "https://cdn-avatars.huggingface.co/v1/production/uploads/1617264212503-603d25b75f9d390ab190b777.jpeg"
+
+Vaibhavs10:
+  name: "vb"
+  website: "https://huggingface.co/reach-vb"
+  avatar: "https://cdn-avatars.huggingface.co/v1/production/uploads/1655385361868-61b85ce86eb1f2c5e6233736.jpeg"
+
+yagil:
+  name: "Yagil Burowski"
+  website: "https://x.com/yagilb"
+  avatar: "https://avatars.lmstudio.com/profile-images/yagil"
+
+hendrytl:
+  name: "Todd Hendry"
+  website: "https://www.linkedin.com/in/todd-hendry-962aa577/"
+  avatar: "https://avatars.githubusercontent.com/u/36863669"
+
+heejingithub:
+  name: "Heejin Cho"
+  website: "https://www.linkedin.com/in/heejc/"
+  avatar: "https://avatars.githubusercontent.com/u/169293861"
+
+himadri518:
+  name: "Himadri Acharya"
+  website: "https://www.linkedin.com/in/himadri-acharya-086ba261/"
+  avatar: "https://avatars.githubusercontent.com/u/14100684?v=4"
+
+neelk-oai:
+  name: "Neel Kapse"
+  website: "https://www.linkedin.com/in/neel-kapse/"
+  avatar: "https://media.licdn.com/dms/image/v2/D4E03AQEegSR4W4Ylmg/profile-displayphoto-scale_400_400/B4EZkLjdrcIQAk-/0/1756835470622?e=1762387200&v=beta&t=HETTFnoh3nV_Yc84tHGkahKgOFdvnPlesfi3ki8mWFg"
+
+hamel:
+  name: "Hamel Husain"
+  website: "https://www.linkedin.com/in/hamelhusain/"
+  avatar: "https://media.licdn.com/dms/image/v2/C5603AQGoyHYtA2QIXw/profile-displayphoto-shrink_400_400/profile-displayphoto-shrink_400_400/0/1572471557655?e=1762387200&v=beta&t=wZrq-Nfc8-4Xq-nJ5g2jt9gZ1KLTg23KOIBGHk2xkZ0"
+
+jhall-openai:
+  name: "Josh Hall"
+  website: "https://www.linkedin.com/in/jhall14/"
+  avatar: "https://avatars.githubusercontent.com/u/198997750?v=4"
+
+charlie-openai:
+  name: "Charlie Weems"
+  website: "https://wee.ms"
+  avatar: "https://avatars.githubusercontent.com/u/181146176?v=4"
+
+nikhils-oai:
+  name: "Nikhil Srivastava"
+  website: "https://www.linkedin.com/in/nsrivast/"
+  avatar: "https://avatars.githubusercontent.com/u/231350150"
+
+roost:
+  name: "ROOST"
+  website: "https://roost.tools"
+  avatar: "https://avatars.githubusercontent.com/u/189051351?v=4"
+
+samarth-oai:
+  name: "Samarth Madduru"
+  website: "https://x.com/samarthmadduru"
+  avatar: "https://pbs.twimg.com/profile_images/1911309450368970752/6IFoTpmu_400x400.jpg"
+
+derrickchoi-openai:
+  name: "Derrick Choi"
+  website: "https://www.linkedin.com/in/derrickchoi/"
+  avatar: "https://avatars.githubusercontent.com/u/211427900"
+
+kevinv-openai:
+  name: "Kevin Verdieck"
+  website: "https://www.linkedin.com/in/kevinverdieck/"
+  avatar: "https://avatars.githubusercontent.com/u/197816265?v=4"
+
+annikab-oai:
+  name: "Annika Brundyn"
+  website: "https://github.com/annikab-oai"
+  avatar: "https://avatars.githubusercontent.com/u/231350694"
+
+kathylau-oai:
+  name: "Kathy Lau"
+  website: "https://www.linkedin.com/in/kathy-lau-nyc/"
+  avatar: "https://avatars.githubusercontent.com/u/247463782"
+
+sidrampally-oai:
+  name: "Sid Rampally"
+  website: "https://www.linkedin.com/in/sidrampally/"
+  avatar: "https://avatars.githubusercontent.com/u/274336082?v=4"
+
+nsingaraju-oai:
+  name: "Nishanth Singaraju"
+  website: "https://github.com/nsingaraju-oai"
+  avatar: "https://avatars.githubusercontent.com/u/232978332"
+
+sophiaqin-openai:
+  name: "Sophia Qin"
+  website: "https://github.com/sophiaqin-openai"
+  avatar: "https://avatars.githubusercontent.com/u/205098674?v=4"
+
+PASFIELD-OAI:
+  name: "Wesley Pasfield"
+  website: "https://github.com/PASFIELD-OAI"
+  avatar: "https://avatars.githubusercontent.com/u/276116382?v=4"
+
+shreekant-openai:
+  name: "Shreekant Agrawal"
+  website: "https://www.linkedin.com/in/shreeagrawal/"
+  avatar: "https://avatars.githubusercontent.com/u/263776704?v=4"
+
+anshgupta-oai:
+  name: "Ansh Gupta"
+  website: "https://github.com/anshgupta-oai"
+  avatar: "https://avatars.githubusercontent.com/u/269039830?v=4"

examples/Chat_finetuning_data_prep.ipynb

@@ -0,0 +1,355 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "a06ec76c",
+   "metadata": {},
+   "source": [
+    "# Data preparation and analysis for chat model fine-tuning\n",
+    "\n",
+    "This notebook serves as a tool to preprocess and analyze the chat dataset used for fine-tuning a chat model. \n",
+    "It checks for format errors, provides basic statistics, and estimates token counts for fine-tuning costs.\n",
+    "The method shown here corresponds to the [current fine-tuning method](https://platform.openai.com/docs/guides/fine-tuning) for gpt-3.5-turbo.\n",
+    "See [legacy fine-tuning](https://platform.openai.com/docs/guides/legacy-fine-tuning) for models like babbage-002 and davinci-002."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "4e63973b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "import tiktoken # for token counting\n",
+    "import numpy as np\n",
+    "from collections import defaultdict"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "013bdbc4",
+   "metadata": {},
+   "source": [
+    "## Data loading\n",
+    "\n",
+    "We first load the chat dataset from an [example JSONL file](https://github.com/openai/openai-cookbook/blob/main/examples/data/toy_chat_fine_tuning.jsonl)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "c248ccd1",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Num examples: 5\n",
+      "First example:\n",
+      "{'role': 'system', 'content': 'You are a happy assistant that puts a positive spin on everything.'}\n",
+      "{'role': 'user', 'content': 'I fell off my bike today.'}\n",
+      "{'role': 'assistant', 'content': \"It's great that you're getting exercise outdoors!\"}\n"
+     ]
+    }
+   ],
+   "source": [
+    "data_path = \"data/toy_chat_fine_tuning.jsonl\"\n",
+    "\n",
+    "# Load the dataset\n",
+    "with open(data_path, 'r', encoding='utf-8') as f:\n",
+    "    dataset = [json.loads(line) for line in f]\n",
+    "\n",
+    "# Initial dataset stats\n",
+    "print(\"Num examples:\", len(dataset))\n",
+    "print(\"First example:\")\n",
+    "for message in dataset[0][\"messages\"]:\n",
+    "    print(message)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "17903d61",
+   "metadata": {},
+   "source": [
+    "## Format validation\n",
+    "\n",
+    "We can perform a variety of error checks to validate that each conversation in the dataset adheres to the format expected by the fine-tuning API. Errors are categorized based on their nature for easier debugging.\n",
+    "\n",
+    "1. **Data Type Check**: Checks whether each entry in the dataset is a dictionary (`dict`). Error type: `data_type`.\n",
+    "2. **Presence of Message List**: Checks if a `messages` list is present in each entry. Error type: `missing_messages_list`.\n",
+    "3. **Message Keys Check**: Validates that each message in the `messages` list contains the keys `role` and `content`. Error type: `message_missing_key`.\n",
+    "4. **Unrecognized Keys in Messages**: Logs if a message has keys other than `role`, `content`, `weight`, `function_call`, and `name`. Error type: `message_unrecognized_key`.\n",
+    "5. **Role Validation**: Ensures the `role` is one of \"system\", \"user\", or \"assistant\". Error type: `unrecognized_role`.\n",
+    "6. **Content Validation**: Verifies that `content` has textual data and is a string. Error type: `missing_content`.\n",
+    "7. **Assistant Message Presence**: Checks that each conversation has at least one message from the assistant. Error type: `example_missing_assistant_message`.\n",
+    "\n",
+    "The code below performs these checks, and outputs counts for each type of error found are printed. This is useful for debugging and ensuring the dataset is ready for the next steps.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "d9f3ccbf",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "No errors found\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Format error checks\n",
+    "format_errors = defaultdict(int)\n",
+    "\n",
+    "for ex in dataset:\n",
+    "    if not isinstance(ex, dict):\n",
+    "        format_errors[\"data_type\"] += 1\n",
+    "        continue\n",
+    "        \n",
+    "    messages = ex.get(\"messages\", None)\n",
+    "    if not messages:\n",
+    "        format_errors[\"missing_messages_list\"] += 1\n",
+    "        continue\n",
+    "        \n",
+    "    for message in messages:\n",
+    "        if \"role\" not in message or \"content\" not in message:\n",
+    "            format_errors[\"message_missing_key\"] += 1\n",
+    "        \n",
+    "        if any(k not in (\"role\", \"content\", \"name\", \"function_call\", \"weight\") for k in message):\n",
+    "            format_errors[\"message_unrecognized_key\"] += 1\n",
+    "        \n",
+    "        if message.get(\"role\", None) not in (\"system\", \"user\", \"assistant\", \"function\"):\n",
+    "            format_errors[\"unrecognized_role\"] += 1\n",
+    "            \n",
+    "        content = message.get(\"content\", None)\n",
+    "        function_call = message.get(\"function_call\", None)\n",
+    "        \n",
+    "        if (not content and not function_call) or not isinstance(content, str):\n",
+    "            format_errors[\"missing_content\"] += 1\n",
+    "    \n",
+    "    if not any(message.get(\"role\", None) == \"assistant\" for message in messages):\n",
+    "        format_errors[\"example_missing_assistant_message\"] += 1\n",
+    "\n",
+    "if format_errors:\n",
+    "    print(\"Found errors:\")\n",
+    "    for k, v in format_errors.items():\n",
+    "        print(f\"{k}: {v}\")\n",
+    "else:\n",
+    "    print(\"No errors found\")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "981e77da",
+   "metadata": {},
+   "source": [
+    "## Token Counting Utilities\n",
+    "\n",
+    "Lets define a few helpful utilities to be used in the rest of the notebook."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "8f4b47b5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "encoding = tiktoken.get_encoding(\"cl100k_base\")\n",
+    "\n",
+    "# not exact!\n",
+    "# simplified from https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb\n",
+    "def num_tokens_from_messages(messages, tokens_per_message=3, tokens_per_name=1):\n",
+    "    num_tokens = 0\n",
+    "    for message in messages:\n",
+    "        num_tokens += tokens_per_message\n",
+    "        for key, value in message.items():\n",
+    "            num_tokens += len(encoding.encode(value))\n",
+    "            if key == \"name\":\n",
+    "                num_tokens += tokens_per_name\n",
+    "    num_tokens += 3\n",
+    "    return num_tokens\n",
+    "\n",
+    "def num_assistant_tokens_from_messages(messages):\n",
+    "    num_tokens = 0\n",
+    "    for message in messages:\n",
+    "        if message[\"role\"] == \"assistant\":\n",
+    "            num_tokens += len(encoding.encode(message[\"content\"]))\n",
+    "    return num_tokens\n",
+    "\n",
+    "def print_distribution(values, name):\n",
+    "    print(f\"\\n#### Distribution of {name}:\")\n",
+    "    print(f\"min / max: {min(values)}, {max(values)}\")\n",
+    "    print(f\"mean / median: {np.mean(values)}, {np.median(values)}\")\n",
+    "    print(f\"p5 / p95: {np.quantile(values, 0.1)}, {np.quantile(values, 0.9)}\")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "0fdff67d",
+   "metadata": {},
+   "source": [
+    "## Data Warnings and Token Counts \n",
+    "\n",
+    "With some lightweight analysis we can identify potential issues in the dataset, like missing messages, and provide statistical insights into message and token counts.\n",
+    "\n",
+    "1. **Missing System/User Messages**: Counts the number of conversations missing a \"system\" or \"user\" message. Such messages are critical for defining the assistant's behavior and initiating the conversation.\n",
+    "2. **Number of Messages Per Example**: Summarizes the distribution of the number of messages in each conversation, providing insight into dialogue complexity.\n",
+    "3. **Total Tokens Per Example**: Calculates and summarizes the distribution of the total number of tokens in each conversation. Important for understanding fine-tuning costs.\n",
+    "4. **Tokens in Assistant's Messages**: Calculates the number of tokens in the assistant's messages per conversation and summarizes this distribution. Useful for understanding the assistant's verbosity.\n",
+    "5. **Token Limit Warnings**: Checks if any examples exceed the maximum token limit (16,385 tokens), as such examples will be truncated during fine-tuning, potentially resulting in data loss.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "52e58ee4",
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Num examples missing system message: 1\n",
+      "Num examples missing user message: 1\n",
+      "\n",
+      "#### Distribution of num_messages_per_example:\n",
+      "min / max: 2, 9\n",
+      "mean / median: 3.8, 3.0\n",
+      "p5 / p95: 2.0, 6.6000000000000005\n",
+      "\n",
+      "#### Distribution of num_total_tokens_per_example:\n",
+      "min / max: 26, 8032\n",
+      "mean / median: 1648.4, 45.0\n",
+      "p5 / p95: 26.8, 4863.6\n",
+      "\n",
+      "#### Distribution of num_assistant_tokens_per_example:\n",
+      "min / max: 4, 8000\n",
+      "mean / median: 1610.2, 10.0\n",
+      "p5 / p95: 6.0, 4811.200000000001\n",
+      "\n",
+      "0 examples may be over the 16,385 token limit, they will be truncated during fine-tuning\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Warnings and tokens counts\n",
+    "n_missing_system = 0\n",
+    "n_missing_user = 0\n",
+    "n_messages = []\n",
+    "convo_lens = []\n",
+    "assistant_message_lens = []\n",
+    "\n",
+    "for ex in dataset:\n",
+    "    messages = ex[\"messages\"]\n",
+    "    if not any(message[\"role\"] == \"system\" for message in messages):\n",
+    "        n_missing_system += 1\n",
+    "    if not any(message[\"role\"] == \"user\" for message in messages):\n",
+    "        n_missing_user += 1\n",
+    "    n_messages.append(len(messages))\n",
+    "    convo_lens.append(num_tokens_from_messages(messages))\n",
+    "    assistant_message_lens.append(num_assistant_tokens_from_messages(messages))\n",
+    "    \n",
+    "print(\"Num examples missing system message:\", n_missing_system)\n",
+    "print(\"Num examples missing user message:\", n_missing_user)\n",
+    "print_distribution(n_messages, \"num_messages_per_example\")\n",
+    "print_distribution(convo_lens, \"num_total_tokens_per_example\")\n",
+    "print_distribution(assistant_message_lens, \"num_assistant_tokens_per_example\")\n",
+    "n_too_long = sum(l > 16385 for l in convo_lens)\n",
+    "print(f\"\\n{n_too_long} examples may be over the 16,385 token limit, they will be truncated during fine-tuning\")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "2afb04df",
+   "metadata": {},
+   "source": [
+    "## Cost Estimation\n",
+    "\n",
+    "In this final section, we estimate the total number of tokens that will be used for fine-tuning, which allows us to approximate the cost. It is worth noting that the duration of the fine-tuning jobs will also increase with the token count. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "fb95a7ce",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Dataset has ~4306 tokens that will be charged for during training\n",
+      "By default, you'll train for 20 epochs on this dataset\n",
+      "By default, you'll be charged for ~86120 tokens\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Pricing and default n_epochs estimate\n",
+    "MAX_TOKENS_PER_EXAMPLE = 16385\n",
+    "\n",
+    "TARGET_EPOCHS = 3\n",
+    "MIN_TARGET_EXAMPLES = 100\n",
+    "MAX_TARGET_EXAMPLES = 25000\n",
+    "MIN_DEFAULT_EPOCHS = 1\n",
+    "MAX_DEFAULT_EPOCHS = 25\n",
+    "\n",
+    "n_epochs = TARGET_EPOCHS\n",
+    "n_train_examples = len(dataset)\n",
+    "if n_train_examples * TARGET_EPOCHS < MIN_TARGET_EXAMPLES:\n",
+    "    n_epochs = min(MAX_DEFAULT_EPOCHS, MIN_TARGET_EXAMPLES // n_train_examples)\n",
+    "elif n_train_examples * TARGET_EPOCHS > MAX_TARGET_EXAMPLES:\n",
+    "    n_epochs = max(MIN_DEFAULT_EPOCHS, MAX_TARGET_EXAMPLES // n_train_examples)\n",
+    "\n",
+    "n_billing_tokens_in_dataset = sum(min(MAX_TOKENS_PER_EXAMPLE, length) for length in convo_lens)\n",
+    "print(f\"Dataset has ~{n_billing_tokens_in_dataset} tokens that will be charged for during training\")\n",
+    "print(f\"By default, you'll train for {n_epochs} epochs on this dataset\")\n",
+    "print(f\"By default, you'll be charged for ~{n_epochs * n_billing_tokens_in_dataset} tokens\")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "a0ad0369",
+   "metadata": {},
+   "source": [
+    "See https://openai.com/pricing to estimate total costs."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

examples/Code_search_using_embeddings.ipynb

@@ -0,0 +1,430 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Code search using embeddings\n",
+    "\n",
+    "This notebook shows how Ada embeddings can be used to implement semantic code search. For this demonstration, we use our own [openai-python code repository](https://github.com/openai/openai-python). We implement a simple version of file parsing and extracting of functions from python files, which can be embedded, indexed, and queried."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Helper Functions\n",
+    "\n",
+    "We first setup some simple parsing functions that allow us to extract important information from our codebase."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "from pathlib import Path\n",
+    "\n",
+    "DEF_PREFIXES = ['def ', 'async def ']\n",
+    "NEWLINE = '\\n'\n",
+    "\n",
+    "def get_function_name(code):\n",
+    "    \"\"\"\n",
+    "    Extract function name from a line beginning with 'def' or 'async def'.\n",
+    "    \"\"\"\n",
+    "    for prefix in DEF_PREFIXES:\n",
+    "        if code.startswith(prefix):\n",
+    "            return code[len(prefix): code.index('(')]\n",
+    "\n",
+    "\n",
+    "def get_until_no_space(all_lines, i):\n",
+    "    \"\"\"\n",
+    "    Get all lines until a line outside the function definition is found.\n",
+    "    \"\"\"\n",
+    "    ret = [all_lines[i]]\n",
+    "    for j in range(i + 1, len(all_lines)):\n",
+    "        if len(all_lines[j]) == 0 or all_lines[j][0] in [' ', '\\t', ')']:\n",
+    "            ret.append(all_lines[j])\n",
+    "        else:\n",
+    "            break\n",
+    "    return NEWLINE.join(ret)\n",
+    "\n",
+    "\n",
+    "def get_functions(filepath):\n",
+    "    \"\"\"\n",
+    "    Get all functions in a Python file.\n",
+    "    \"\"\"\n",
+    "    with open(filepath, 'r') as file:\n",
+    "        all_lines = file.read().replace('\\r', NEWLINE).split(NEWLINE)\n",
+    "        for i, l in enumerate(all_lines):\n",
+    "            for prefix in DEF_PREFIXES:\n",
+    "                if l.startswith(prefix):\n",
+    "                    code = get_until_no_space(all_lines, i)\n",
+    "                    function_name = get_function_name(code)\n",
+    "                    yield {\n",
+    "                        'code': code,\n",
+    "                        'function_name': function_name,\n",
+    "                        'filepath': filepath,\n",
+    "                    }\n",
+    "                    break\n",
+    "\n",
+    "\n",
+    "def extract_functions_from_repo(code_root):\n",
+    "    \"\"\"\n",
+    "    Extract all .py functions from the repository.\n",
+    "    \"\"\"\n",
+    "    code_files = list(code_root.glob('**/*.py'))\n",
+    "\n",
+    "    num_files = len(code_files)\n",
+    "    print(f'Total number of .py files: {num_files}')\n",
+    "\n",
+    "    if num_files == 0:\n",
+    "        print('Verify openai-python repo exists and code_root is set correctly.')\n",
+    "        return None\n",
+    "\n",
+    "    all_funcs = [\n",
+    "        func\n",
+    "        for code_file in code_files\n",
+    "        for func in get_functions(str(code_file))\n",
+    "    ]\n",
+    "\n",
+    "    num_funcs = len(all_funcs)\n",
+    "    print(f'Total number of functions extracted: {num_funcs}')\n",
+    "\n",
+    "    return all_funcs"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Data Loading\n",
+    "\n",
+    "We'll first load the openai-python folder and extract the needed information using the functions we defined above."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Total number of .py files: 51\n",
+      "Total number of functions extracted: 97\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Set user root directory to the 'openai-python' repository\n",
+    "root_dir = Path.home()\n",
+    "\n",
+    "# Assumes the 'openai-python' repository exists in the user's root directory\n",
+    "code_root = root_dir / 'openai-python'\n",
+    "\n",
+    "# Extract all functions from the repository\n",
+    "all_funcs = extract_functions_from_repo(code_root)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now that we have our content, we can pass the data to the `text-embedding-3-small` model and get back our vector embeddings."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>code</th>\n",
+       "      <th>function_name</th>\n",
+       "      <th>filepath</th>\n",
+       "      <th>code_embedding</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>def _console_log_level():\\n    if openai.log i...</td>\n",
+       "      <td>_console_log_level</td>\n",
+       "      <td>openai/util.py</td>\n",
+       "      <td>[0.005937571171671152, 0.05450401455163956, 0....</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>def log_debug(message, **params):\\n    msg = l...</td>\n",
+       "      <td>log_debug</td>\n",
+       "      <td>openai/util.py</td>\n",
+       "      <td>[0.017557814717292786, 0.05647840350866318, -0...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>def log_info(message, **params):\\n    msg = lo...</td>\n",
+       "      <td>log_info</td>\n",
+       "      <td>openai/util.py</td>\n",
+       "      <td>[0.022524144500494003, 0.06219055876135826, -0...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>def log_warn(message, **params):\\n    msg = lo...</td>\n",
+       "      <td>log_warn</td>\n",
+       "      <td>openai/util.py</td>\n",
+       "      <td>[0.030524108558893204, 0.0667714849114418, -0....</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>def logfmt(props):\\n    def fmt(key, val):\\n  ...</td>\n",
+       "      <td>logfmt</td>\n",
+       "      <td>openai/util.py</td>\n",
+       "      <td>[0.05337328091263771, 0.03697286546230316, -0....</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "                                                code       function_name  \\\n",
+       "0  def _console_log_level():\\n    if openai.log i...  _console_log_level   \n",
+       "1  def log_debug(message, **params):\\n    msg = l...           log_debug   \n",
+       "2  def log_info(message, **params):\\n    msg = lo...            log_info   \n",
+       "3  def log_warn(message, **params):\\n    msg = lo...            log_warn   \n",
+       "4  def logfmt(props):\\n    def fmt(key, val):\\n  ...              logfmt   \n",
+       "\n",
+       "         filepath                                     code_embedding  \n",
+       "0  openai/util.py  [0.005937571171671152, 0.05450401455163956, 0....  \n",
+       "1  openai/util.py  [0.017557814717292786, 0.05647840350866318, -0...  \n",
+       "2  openai/util.py  [0.022524144500494003, 0.06219055876135826, -0...  \n",
+       "3  openai/util.py  [0.030524108558893204, 0.0667714849114418, -0....  \n",
+       "4  openai/util.py  [0.05337328091263771, 0.03697286546230316, -0....  "
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from utils.embeddings_utils import get_embedding\n",
+    "\n",
+    "df = pd.DataFrame(all_funcs)\n",
+    "df['code_embedding'] = df['code'].apply(lambda x: get_embedding(x, model='text-embedding-3-small'))\n",
+    "df['filepath'] = df['filepath'].map(lambda x: Path(x).relative_to(code_root))\n",
+    "df.to_csv(\"data/code_search_openai-python.csv\", index=False)\n",
+    "df.head()"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Testing\n",
+    "\n",
+    "Let's test our endpoint with some simple queries. If you're familiar with the `openai-python` repository, you'll see that we're able to easily find functions we're looking for only a simple English description.\n",
+    "\n",
+    "We define a search_functions method that takes our data that contains our embeddings, a query string, and some other configuration options. The process of searching our database works like such:\n",
+    "\n",
+    "1. We first embed our query string (code_query) with `text-embedding-3-small`. The reasoning here is that a query string like 'a function that reverses a string' and a function like 'def reverse(string): return string[::-1]' will be very similar when embedded.\n",
+    "2. We then calculate the cosine similarity between our query string embedding and all data points in our database. This gives a distance between each point and our query.\n",
+    "3. We finally sort all of our data points by their distance to our query string and return the number of results requested in the function parameters. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from utils.embeddings_utils import cosine_similarity\n",
+    "\n",
+    "def search_functions(df, code_query, n=3, pprint=True, n_lines=7):\n",
+    "    embedding = get_embedding(code_query, model='text-embedding-3-small')\n",
+    "    df['similarities'] = df.code_embedding.apply(lambda x: cosine_similarity(x, embedding))\n",
+    "\n",
+    "    res = df.sort_values('similarities', ascending=False).head(n)\n",
+    "\n",
+    "    if pprint:\n",
+    "        for r in res.iterrows():\n",
+    "            print(f\"{r[1].filepath}:{r[1].function_name}  score={round(r[1].similarities, 3)}\")\n",
+    "            print(\"\\n\".join(r[1].code.split(\"\\n\")[:n_lines]))\n",
+    "            print('-' * 70)\n",
+    "\n",
+    "    return res"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "openai/validators.py:format_inferrer_validator  score=0.453\n",
+      "def format_inferrer_validator(df):\n",
+      "    \"\"\"\n",
+      "    This validator will infer the likely fine-tuning format of the data, and display it to the user if it is classification.\n",
+      "    It will also suggest to use ada and explain train/validation split benefits.\n",
+      "    \"\"\"\n",
+      "    ft_type = infer_task_type(df)\n",
+      "    immediate_msg = None\n",
+      "----------------------------------------------------------------------\n",
+      "openai/validators.py:infer_task_type  score=0.37\n",
+      "def infer_task_type(df):\n",
+      "    \"\"\"\n",
+      "    Infer the likely fine-tuning task type from the data\n",
+      "    \"\"\"\n",
+      "    CLASSIFICATION_THRESHOLD = 3  # min_average instances of each class\n",
+      "    if sum(df.prompt.str.len()) == 0:\n",
+      "        return \"open-ended generation\"\n",
+      "----------------------------------------------------------------------\n",
+      "openai/validators.py:apply_validators  score=0.369\n",
+      "def apply_validators(\n",
+      "    df,\n",
+      "    fname,\n",
+      "    remediation,\n",
+      "    validators,\n",
+      "    auto_accept,\n",
+      "    write_out_file_func,\n",
+      "----------------------------------------------------------------------\n"
+     ]
+    }
+   ],
+   "source": [
+    "res = search_functions(df, 'fine-tuning input data validation logic', n=3)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "openai/validators.py:get_common_xfix  score=0.487\n",
+      "def get_common_xfix(series, xfix=\"suffix\"):\n",
+      "    \"\"\"\n",
+      "    Finds the longest common suffix or prefix of all the values in a series\n",
+      "    \"\"\"\n",
+      "    common_xfix = \"\"\n",
+      "    while True:\n",
+      "        common_xfixes = (\n",
+      "            series.str[-(len(common_xfix) + 1) :]\n",
+      "            if xfix == \"suffix\"\n",
+      "            else series.str[: len(common_xfix) + 1]\n",
+      "----------------------------------------------------------------------\n",
+      "openai/validators.py:common_completion_suffix_validator  score=0.449\n",
+      "def common_completion_suffix_validator(df):\n",
+      "    \"\"\"\n",
+      "    This validator will suggest to add a common suffix to the completion if one doesn't already exist in case of classification or conditional generation.\n",
+      "    \"\"\"\n",
+      "    error_msg = None\n",
+      "    immediate_msg = None\n",
+      "    optional_msg = None\n",
+      "    optional_fn = None\n",
+      "\n",
+      "    ft_type = infer_task_type(df)\n",
+      "----------------------------------------------------------------------\n"
+     ]
+    }
+   ],
+   "source": [
+    "res = search_functions(df, 'find common suffix', n=2, n_lines=10)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "openai/cli.py:tools_register  score=0.391\n",
+      "def tools_register(parser):\n",
+      "    subparsers = parser.add_subparsers(\n",
+      "        title=\"Tools\", help=\"Convenience client side tools\"\n",
+      "    )\n",
+      "\n",
+      "    def help(args):\n",
+      "        parser.print_help()\n",
+      "\n",
+      "    parser.set_defaults(func=help)\n",
+      "\n",
+      "    sub = subparsers.add_parser(\"fine_tunes.prepare_data\")\n",
+      "    sub.add_argument(\n",
+      "        \"-f\",\n",
+      "        \"--file\",\n",
+      "        required=True,\n",
+      "        help=\"JSONL, JSON, CSV, TSV, TXT or XLSX file containing prompt-completion examples to be analyzed.\"\n",
+      "        \"This should be the local file path.\",\n",
+      "    )\n",
+      "    sub.add_argument(\n",
+      "        \"-q\",\n",
+      "----------------------------------------------------------------------\n"
+     ]
+    }
+   ],
+   "source": [
+    "res = search_functions(df, 'Command line interface for fine-tuning', n=1, n_lines=20)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "openai",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.5"
+  },
+  "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/Context_summarization_with_realtime_api.ipynb

@@ -0,0 +1,737 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Context Summarization with Realtime API\n",
+    "## 1. Overview\n",
+    "Build an end‑to‑end **voice bot** that listens to your mic, speaks back in real time and **summarises long conversations** so quality never drops.\n",
+    "\n",
+    "### What You’ll Learn\n",
+    "1. **Live microphone streaming** → OpenAI *Realtime* (voice‑to‑voice) endpoint.\n",
+    "2. **Instant transcripts & speech playback** on every turn.\n",
+    "3. **Conversation state container** that stores **every** user/assistant message.\n",
+    "4. **Automatic “context trim”** – when the token window becomes very large (configurable), older turns are compressed into a summary.\n",
+    "5. **Extensible design** you can adapt to support customer‑support bots, kiosks, or multilingual assistants.\n",
+    "\n",
+    "\n",
+    "### Prerequisites\n",
+    "\n",
+    "| Requirement | Details |\n",
+    "|-------------|---------|\n",
+    "| **Python ≥ 3.10** | Will ensure that you don't hit any issues |\n",
+    "| **OpenAI API key** | Set `OPENAI_API_KEY` in your shell or paste inline (*not ideal for prod*) |\n",
+    "| Mic + speakers | Grant OS permission if prompted |\n",
+    "\n",
+    "\n",
+    "**Need help setting up the key?**  \n",
+    "> Follow the [official quick‑start guide](https://platform.openai.com/docs/quickstart#step-2-set-your-api-key).\n",
+    "\n",
+    "\n",
+    "*Notes:*\n",
+    "> 1. gpt-realtime supports a 32k token context window, though in certain use cases, you may notice performance degrade as you stuff more tokens into the context window.\n",
+    "> 2. Token window = all tokens (words and audio tokens) the model currently keeps in memory for the session.x\n",
+    "\n",
+    "### One‑liner install (run in a fresh cell)\n",
+    "\n",
+    "*New API Parameters:*\n",
+    "> 1. The Realtime API GA has releases a new parameter [`truncation`](https://platform.openai.com/docs/api-reference/realtime-client-events/session/update#realtime-client-events/session/update-session-realtime-session-configuration-truncation). This parameter automatically optimizes context truncation, preserving relevant information while maximizing cache hit rates."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Run once to install or upgrade dependencies (comment out if already installed)\n",
+    "# !pip install --upgrade openai websockets sounddevice simpleaudio"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Standard library imports\n",
+    "import os\n",
+    "import sys\n",
+    "import io\n",
+    "import json\n",
+    "import base64\n",
+    "import pathlib\n",
+    "import wave\n",
+    "from dataclasses import dataclass, field\n",
+    "from typing import List, Literal\n",
+    "\n",
+    "# Third-party imports\n",
+    "import asyncio\n",
+    "import numpy as np\n",
+    "import sounddevice as sd         # microphone capture\n",
+    "import simpleaudio               # speaker playback\n",
+    "import websockets                # WebSocket client\n",
+    "import openai                    # OpenAI Python SDK >= 1.14.0"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Set your API key safely\n",
+    "openai.api_key = os.getenv(\"OPENAI_API_KEY\", \"\")\n",
+    "if not openai.api_key:\n",
+    "    raise ValueError(\"OPENAI_API_KEY not found – please set env var or edit this cell.\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Token Utilisation – Text vs Voice\n",
+    "\n",
+    "Large‑token windows are precious, every extra token you use costs latency + money.  \n",
+    "For **audio** the input token window increases much faster than for plain text because amplitude, timing, and other acoustic details must be represented.\n",
+    "\n",
+    "In practice you’ll often see **≈ 10 ×** more tokens for the *same* sentence in audio versus text.\n",
+    "\n",
+    "\n",
+    "* gpt-realtime accepts up to **32k tokens** and as the token size increases, instruction adherence can drift.\n",
+    "* Every user/assistant turn consumes tokens → the window **only grows**.\n",
+    "* **Strategy**: Summarise older turns into a single assistant message, keep the last few verbatim turns, and continue.\n",
+    "\n",
+    "<img src=\"../images/text-vs-audio-tokens.png\" alt=\"drawing\" width=\"800\"/>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Helper Functions\n",
+    "The following helper functions will enable us to run the full script."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 3.1 Conversation State\n",
+    "Unlike HTTP-based Chat Completions, the Realtime API maintains an open, **stateful** session with two key components:\n",
+    "\n",
+    "| Component       | Purpose |\n",
+    "|----------------|---------|\n",
+    "| **Session**     | Controls global settings — model, voice, modalities, VAD, etc. |\n",
+    "| **Conversation** | Stores turn-by-turn messages between user and assistant — both audio and text. |\n",
+    "\n",
+    "This notebook wraps these components inside a simple `ConversationState` object to keep your logic clean, track history, and manage summarization when context windows fill up."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "@dataclass\n",
+    "class Turn:\n",
+    "    \"\"\"One utterance in the dialogue (user **or** assistant).\"\"\"\n",
+    "    role: Literal[\"user\", \"assistant\"]\n",
+    "    item_id: str                    # Server‑assigned identifier\n",
+    "    text: str | None = None         # Filled once transcript is ready\n",
+    "\n",
+    "@dataclass\n",
+    "class ConversationState:\n",
+    "    \"\"\"All mutable data the session needs — nothing more, nothing less.\"\"\"\n",
+    "    history: List[Turn] = field(default_factory=list)         # Ordered log\n",
+    "    waiting: dict[str, asyncio.Future] = field(default_factory=dict)  # Pending transcript fetches\n",
+    "    summary_count: int = 0\n",
+    "\n",
+    "    latest_tokens: int = 0          # Window size after last reply\n",
+    "    summarising: bool = False       # Guard so we don’t run two summaries at once"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "A quick helper to peek at the transcript:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def print_history(state) -> None:\n",
+    "    \"\"\"Pretty-print the running transcript so far.\"\"\"\n",
+    "    print(\"—— Conversation so far ———————————————\")\n",
+    "    for turn in state.history:\n",
+    "        text_preview = (turn.text or \"\").strip().replace(\"\\n\", \" \")\n",
+    "        print(f\"[{turn.role:<9}] {text_preview}  ({turn.item_id})\")\n",
+    "    print(\"——————————————————————————————————————————\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 3.2 · Streaming Audio\n",
+    "We’ll stream raw PCM‑16 microphone data straight into the Realtime API.\n",
+    "\n",
+    "The pipeline is: mic ─► async.Queue ─► WebSocket ─► Realtime API"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### 3.2.1 Capture Microphone Input\n",
+    "We’ll start with a coroutine that:\n",
+    "\n",
+    "* Opens the default mic at **24 kHz, mono, PCM‑16** (one of the [format](https://platform.openai.com/docs/api-reference/realtime-sessions/create#realtime-sessions-create-input_audio_format) Realtime accepts).  \n",
+    "* Slices the stream into **≈ 40 ms** blocks.  \n",
+    "* Dumps each block into an `asyncio.Queue` so another task (next section) can forward it to OpenAI."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "async def mic_to_queue(pcm_queue: asyncio.Queue[bytes]) -> None:\n",
+    "    \"\"\"\n",
+    "    Capture raw PCM‑16 microphone audio and push ~CHUNK_DURATION_MS chunks\n",
+    "    to *pcm_queue* until the surrounding task is cancelled.\n",
+    "\n",
+    "    Parameters\n",
+    "    ----------\n",
+    "    pcm_queue : asyncio.Queue[bytes]\n",
+    "        Destination queue for PCM‑16 frames (little‑endian int16).\n",
+    "    \"\"\"\n",
+    "    blocksize = int(SAMPLE_RATE_HZ * CHUNK_DURATION_MS / 1000)\n",
+    "\n",
+    "    def _callback(indata, _frames, _time, status):\n",
+    "        if status:                               # XRuns, device changes, etc.\n",
+    "            print(\"⚠️\", status, file=sys.stderr)\n",
+    "        try:\n",
+    "            pcm_queue.put_nowait(bytes(indata))  # 1‑shot enqueue\n",
+    "        except asyncio.QueueFull:\n",
+    "            # Drop frame if upstream (WebSocket) can’t keep up.\n",
+    "            pass\n",
+    "\n",
+    "    # RawInputStream is synchronous; wrap in context manager to auto‑close.\n",
+    "    with sd.RawInputStream(\n",
+    "        samplerate=SAMPLE_RATE_HZ,\n",
+    "        blocksize=blocksize,\n",
+    "        dtype=\"int16\",\n",
+    "        channels=1,\n",
+    "        callback=_callback,\n",
+    "    ):\n",
+    "        try:\n",
+    "            # Keep coroutine alive until cancelled by caller.\n",
+    "            await asyncio.Event().wait()\n",
+    "        finally:\n",
+    "            print(\"⏹️  Mic stream closed.\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### 3.2.2 Send Audio Chunks to the API\n",
+    "\n",
+    "Our mic task is now filling an `asyncio.Queue` with raw PCM‑16 blocks.  \n",
+    "Next step: pull chunks off that queue, **base‑64 encode** them (the protocol requires JSON‑safe text), and ship each block to the Realtime WebSocket as an `input_audio_buffer.append` event."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Helper function to encode audio chunks in base64\n",
+    "b64 = lambda blob: base64.b64encode(blob).decode()\n",
+    "\n",
+    "async def queue_to_websocket(pcm_queue: asyncio.Queue[bytes], ws):\n",
+    "    \"\"\"Read audio chunks from queue and send as JSON events.\"\"\"\n",
+    "    try:\n",
+    "        while (chunk := await pcm_queue.get()) is not None:\n",
+    "            await ws.send(json.dumps({\n",
+    "                \"type\": \"input_audio_buffer.append\",\n",
+    "                \"audio\": b64(chunk),\n",
+    "            }))\n",
+    "    except websockets.ConnectionClosed:\n",
+    "        print(\"WebSocket closed – stopping uploader\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### 3.2.3 Handle Incoming Events \n",
+    "Once audio reaches the server, the Realtime API pushes a stream of JSON events back over the **same** WebSocket.  \n",
+    "Understanding these events is critical for:\n",
+    "\n",
+    "* Printing live transcripts  \n",
+    "* Playing incremental audio back to the user  \n",
+    "* Keeping an accurate [`Conversation State`](https://platform.openai.com/docs/api-reference/realtime-server-events/conversation/created) so context trimming works later  \n",
+    "\n",
+    "\n",
+    "| Event type | When it arrives | Why it matters | Typical handler logic |\n",
+    "|------------|-----------------|---------------|-----------------------|\n",
+    "| **`session.created`** | Immediately after the WebSocket handshake | Confirms the session is open and provides the `session.id`. | Log the ID for traceability and verify the connection. |\n",
+    "| **`session.updated`** | After you send a `session.update` call | Acknowledges that the server applied new session settings. | Inspect the echoed settings and update any local cache. |\n",
+    "| **`conversation.item.created`** (user) | A few ms after the user stops speaking (client VAD fires) | Reserves a timeline slot; transcript may still be **`null`**. | Insert a *placeholder* user turn in `state.history` marked “pending transcript”. |\n",
+    "| **`conversation.item.retrieved`** | ~100 – 300 ms later, once audio transcription is complete | Supplies the final user transcript (with timing). | Replace the placeholder with the transcript and print it if desired. |\n",
+    "| **`response.audio.delta`** | Every 20 – 60 ms while the assistant is speaking | Streams PCM‑16 audio chunks (and optional incremental text). | Buffer each chunk and play it; optionally show partial text in the console. |\n",
+    "| **`response.done`** | After the assistant’s last token | Signals both audio & text are complete; includes usage stats. | Finalize the assistant turn, update `state.latest_tokens`, and log usage. |\n",
+    "| **`conversation.item.deleted`** | Whenever you prune with `conversation.item.delete` | Confirms a turn was removed, freeing tokens on the server. | Mirror the deletion locally so your context window matches the server’s. |\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 3.3 Detect When to Summarise\n",
+    "The Realtime model keeps a **large 32 k‑token window**, but quality can drift long before that limit as you stuff more context into the model.\n",
+    "\n",
+    "Our goal: **auto‑summarise** once the running window nears a safe threshold (default **2 000 tokens** for the notebook), then prune the superseded turns both locally *and* server‑side.\n",
+    "\n",
+    "We monitor latest_tokens returned in `response.done`. When it exceeds SUMMARY_TRIGGER and we have more than KEEP_LAST_TURNS, we spin up a background summarization coroutine.\n",
+    "\n",
+    "We compress everything except the last 2 turns into a single French paragraph, then:\n",
+    "\n",
+    "1. Insert that paragraph as a new assistant message at the top of the conversation.\n",
+    "\n",
+    "2. Delete the message items that was used for the summary.\n",
+    "\n",
+    "We will later ask the Voice agent what language was the summary to test if the Summary insertion into Realtime API Conversation Context was successful."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "async def run_summary_llm(text: str) -> str:\n",
+    "    \"\"\"Call a lightweight model to summarise `text`.\"\"\"\n",
+    "    resp = await asyncio.to_thread(lambda: openai.chat.completions.create(\n",
+    "        model=SUMMARY_MODEL,\n",
+    "        temperature=0,\n",
+    "        messages=[\n",
+    "            {\"role\": \"system\", \"content\": \"Summarise in French the following conversation \"\n",
+    "                            \"in one concise paragraph so it can be used as \"\n",
+    "                            \"context for future dialogue.\"},\n",
+    "            {\"role\": \"user\", \"content\": text},\n",
+    "        ],\n",
+    "    ))\n",
+    "    return resp.choices[0].message.content.strip()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Important implementation detail:\n",
+    "- The summary is appended as a SYSTEM message rather than an ASSISTANT message. Testing revealed that, during extended conversations, using ASSISTANT messages for summaries can cause the model to mistakenly switch from audio responses to text responses. By using SYSTEM messages for summaries (which can also include additional custom instructions), we clearly signal to the model that these are context-setting instructions, preventing it from incorrectly adopting the modality of the ongoing user-assistant interaction."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "async def summarise_and_prune(ws, state):\n",
+    "    \"\"\"Summarise old turns, delete them server‑side, and prepend a single summary\n",
+    "    turn locally + remotely.\"\"\"\n",
+    "    state.summarising = True\n",
+    "    print(\n",
+    "        f\"⚠️  Token window ≈{state.latest_tokens} ≥ {SUMMARY_TRIGGER}. Summarising…\",\n",
+    "    )\n",
+    "    old_turns, recent_turns = state.history[:-KEEP_LAST_TURNS], state.history[-KEEP_LAST_TURNS:]\n",
+    "    convo_text = \"\\n\".join(f\"{t.role}: {t.text}\" for t in old_turns if t.text)\n",
+    "    \n",
+    "    if not convo_text:\n",
+    "        print(\"Nothing to summarise (transcripts still pending).\")\n",
+    "        state.summarising = False\n",
+    "\n",
+    "    summary_text = await run_summary_llm(convo_text) if convo_text else \"\"\n",
+    "    state.summary_count += 1\n",
+    "    summary_id = f\"sum_{state.summary_count:03d}\"\n",
+    "    state.history[:] = [Turn(\"assistant\", summary_id, summary_text)] + recent_turns\n",
+    "    \n",
+    "    print_history(state)    \n",
+    "\n",
+    "    # Create summary on server\n",
+    "    await ws.send(json.dumps({\n",
+    "        \"type\": \"conversation.item.create\",\n",
+    "        \"previous_item_id\": \"root\",\n",
+    "        \"item\": {\n",
+    "            \"id\": summary_id,\n",
+    "            \"type\": \"message\",\n",
+    "            \"role\": \"system\",\n",
+    "            \"content\": [{\"type\": \"input_text\", \"text\": summary_text}],\n",
+    "        },\n",
+    "    }))\n",
+    "\n",
+    "    # Delete old items\n",
+    "    for turn in old_turns:\n",
+    "        await ws.send(json.dumps({\n",
+    "            \"type\": \"conversation.item.delete\",\n",
+    "            \"item_id\": turn.item_id,\n",
+    "        }))\n",
+    "\n",
+    "    print(f\"✅ Summary inserted ({summary_id})\")\n",
+    "    \n",
+    "    state.summarising = False"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The following function lets us poll for transcripts over time. This is useful in cases where the user's audio hasn't been transcribed immediately, so we can retrieve the final result later."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "async def fetch_full_item(\n",
+    "    ws, item_id: str, state: ConversationState, attempts: int = 1\n",
+    "):\n",
+    "    \"\"\"\n",
+    "    Ask the server for a full conversation item; retry up to 5× if the\n",
+    "    transcript field is still null.  Resolve the waiting future when done.\n",
+    "    \"\"\"\n",
+    "    # If there is already a pending fetch, just await it\n",
+    "    if item_id in state.waiting:\n",
+    "        return await state.waiting[item_id]\n",
+    "\n",
+    "    fut = asyncio.get_running_loop().create_future()\n",
+    "    state.waiting[item_id] = fut\n",
+    "\n",
+    "    await ws.send(json.dumps({\n",
+    "        \"type\": \"conversation.item.retrieve\",\n",
+    "        \"item_id\": item_id,\n",
+    "    }))\n",
+    "    item = await fut\n",
+    "\n",
+    "    # If transcript still missing retry (max 5×)\n",
+    "    if attempts < 5 and not item.get(\"content\", [{}])[0].get(\"transcript\"):\n",
+    "        await asyncio.sleep(0.4 * attempts)\n",
+    "        return await fetch_full_item(ws, item_id, state, attempts + 1)\n",
+    "\n",
+    "    # Done – remove the marker\n",
+    "    state.waiting.pop(item_id, None)\n",
+    "    return item\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4. End‑to‑End Workflow Demonstration\n",
+    "\n",
+    "Run the two cells below to launch an interactive session. Interrupt the cell stop recording.\n",
+    "\n",
+    "> **Note:**  \n",
+    "> This notebook uses `SUMMARY_TRIGGER = 2000` and `KEEP_LAST_TURNS = 2` to make summarization easier to demo quickly.  \n",
+    "> In production, you should tune these values based on your application's needs.  \n",
+    "> - A typical `SUMMARY_TRIGGER` falls between **20,000–32,000 tokens**, depending on how performance degrades with larger context for your use case."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Audio/config knobs\n",
+    "SAMPLE_RATE_HZ    = 24_000   # Required by pcm16\n",
+    "CHUNK_DURATION_MS = 40       # chunk size for audio capture\n",
+    "BYTES_PER_SAMPLE  = 2        # pcm16 = 2 bytes/sample\n",
+    "SUMMARY_TRIGGER   = 2_000    # Summarise when context ≥ this\n",
+    "KEEP_LAST_TURNS   = 2       # Keep these turns verbatim\n",
+    "SUMMARY_MODEL     = \"gpt-4o-mini\"  # Cheaper, fast summariser"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# --------------------------------------------------------------------------- #\n",
+    "# Realtime session                                                          #\n",
+    "# --------------------------------------------------------------------------- #\n",
+    "async def realtime_session(model=\"gpt-realtime\", voice=\"shimmer\", enable_playback=True):\n",
+    "    \"\"\"\n",
+    "    Main coroutine: connects to the Realtime endpoint, spawns helper tasks,\n",
+    "    and processes incoming events in a big async‑for loop.\n",
+    "    \"\"\"\n",
+    "    state = ConversationState()  # Reset state for each run\n",
+    "\n",
+    "    pcm_queue: asyncio.Queue[bytes] = asyncio.Queue()\n",
+    "    assistant_audio: List[bytes] = []\n",
+    "\n",
+    "    # ----------------------------------------------------------------------- #\n",
+    "    # Open the WebSocket connection to the Realtime API                       #\n",
+    "    # ----------------------------------------------------------------------- #\n",
+    "    url = f\"wss://api.openai.com/v1/realtime?model={model}\"\n",
+    "    headers = {\"Authorization\": f\"Bearer {openai.api_key}\"}\n",
+    "\n",
+    "    async with websockets.connect(url, extra_headers=headers, max_size=1 << 24) as ws:\n",
+    "        # ------------------------------------------------------------------- #\n",
+    "        # Wait until server sends session.created                             #\n",
+    "        # ------------------------------------------------------------------- #\n",
+    "        while json.loads(await ws.recv())[\"type\"] != \"session.created\":\n",
+    "            pass\n",
+    "        print(\"session.created ✅\")\n",
+    "\n",
+    "        # ------------------------------------------------------------------- #\n",
+    "        # Configure session: voice, modalities, audio formats, transcription  #\n",
+    "        # ------------------------------------------------------------------- #\n",
+    "        await ws.send(json.dumps({\n",
+    "            \"type\": \"session.update\",\n",
+    "            \"session\": {\n",
+    "                \"type\": \"realtime\",\n",
+    "                model: \"gpt-realtime\",\n",
+    "                \"voice\": voice,\n",
+    "                \"modalities\": [\"audio\", \"text\"],\n",
+    "                \"input_audio_format\": \"pcm16\",\n",
+    "                \"output_audio_format\": \"pcm16\",\n",
+    "                \"input_audio_transcription\": {\"model\": \"gpt-4o-transcribe\"},\n",
+    "            },\n",
+    "        }))\n",
+    "\n",
+    "        # ------------------------------------------------------------------- #\n",
+    "        # Launch background tasks: mic capture → queue → websocket            #\n",
+    "        # ------------------------------------------------------------------- #\n",
+    "        mic_task = asyncio.create_task(mic_to_queue(pcm_queue))\n",
+    "        upl_task = asyncio.create_task(queue_to_websocket(pcm_queue, ws))\n",
+    "\n",
+    "        print(\"🎙️ Speak now (Ctrl‑C to quit)…\")\n",
+    "\n",
+    "        try:\n",
+    "            # ------------------------------------------------------------------- #\n",
+    "            # Main event loop: process incoming events from the websocket         #\n",
+    "            # ------------------------------------------------------------------- #\n",
+    "            async for event_raw in ws:\n",
+    "                event = json.loads(event_raw)\n",
+    "                etype = event[\"type\"]\n",
+    "\n",
+    "                # --------------------------------------------------------------- #\n",
+    "                # User just spoke ⇢ conversation.item.created (role = user)        #\n",
+    "                # --------------------------------------------------------------- #\n",
+    "                if etype == \"conversation.item.created\" and event[\"item\"][\"role\"] == \"user\":\n",
+    "                    item = event[\"item\"]\n",
+    "                    text = None\n",
+    "                    if item[\"content\"]:\n",
+    "                        text = item[\"content\"][0].get(\"transcript\")\n",
+    "                    \n",
+    "                    state.history.append(Turn(\"user\", event[\"item\"][\"id\"], text))\n",
+    "                    \n",
+    "                    # If transcript not yet available, fetch it later\n",
+    "                    if text is None:\n",
+    "                        asyncio.create_task(fetch_full_item(ws, item[\"id\"], state))\n",
+    "\n",
+    "                # --------------------------------------------------------------- #\n",
+    "                # Transcript fetched ⇢ conversation.item.retrieved                 #\n",
+    "                # --------------------------------------------------------------- #\n",
+    "                elif etype == \"conversation.item.retrieved\":\n",
+    "                    content = event[\"item\"][\"content\"][0]\n",
+    "                    # Fill missing transcript in history\n",
+    "                    for t in state.history:\n",
+    "                        if t.item_id == event[\"item\"][\"id\"]:\n",
+    "                            t.text = content.get(\"transcript\")\n",
+    "                            break\n",
+    "\n",
+    "                # --------------------------------------------------------------- #\n",
+    "                # Assistant audio arrives in deltas                               #\n",
+    "                # --------------------------------------------------------------- #\n",
+    "                elif etype == \"response.audio.delta\":\n",
+    "                    assistant_audio.append(base64.b64decode(event[\"delta\"]))\n",
+    "\n",
+    "                # --------------------------------------------------------------- #\n",
+    "                # Assistant reply finished ⇢ response.done                        #\n",
+    "                # --------------------------------------------------------------- #\n",
+    "                elif etype == \"response.done\":\n",
+    "                    for item in event[\"response\"][\"output\"]:\n",
+    "                        if item[\"role\"] == \"assistant\":\n",
+    "                            txt = item[\"content\"][0][\"transcript\"]\n",
+    "                            state.history.append(Turn(\"assistant\", item[\"id\"], txt))\n",
+    "                            # print(f\"\\n🤖 {txt}\\n\")\n",
+    "                    state.latest_tokens = event[\"response\"][\"usage\"][\"total_tokens\"]\n",
+    "                    print(f\"—— response.done  (window ≈{state.latest_tokens} tokens) ——\")\n",
+    "                    print_history(state)\n",
+    "                    \n",
+    "                    # Fetch any still‑missing user transcripts\n",
+    "                    for turn in state.history:\n",
+    "                        if (turn.role == \"user\"\n",
+    "                            and turn.text is None\n",
+    "                            and turn.item_id not in state.waiting):\n",
+    "                            asyncio.create_task(\n",
+    "                                fetch_full_item(ws, turn.item_id, state)\n",
+    "                            )\n",
+    "\n",
+    "                    # Playback collected audio once reply completes\n",
+    "                    if enable_playback and assistant_audio:\n",
+    "                        simpleaudio.play_buffer(b\"\".join(assistant_audio), 1, BYTES_PER_SAMPLE, SAMPLE_RATE_HZ)\n",
+    "                        assistant_audio.clear()\n",
+    "\n",
+    "                    # Summarise if context too large – fire in background so we don't block dialogue\n",
+    "                    if state.latest_tokens >= SUMMARY_TRIGGER and len(state.history) > KEEP_LAST_TURNS and not state.summarising:\n",
+    "                        asyncio.create_task(summarise_and_prune(ws, state))\n",
+    "\n",
+    "        except KeyboardInterrupt:\n",
+    "            print(\"\\nStopping…\")\n",
+    "        finally:\n",
+    "            mic_task.cancel()\n",
+    "            await pcm_queue.put(None)\n",
+    "            await upl_task"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Run the realtime session (this cell blocks until you stop it)\n",
+    "await realtime_session()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "```raw\n",
+    "session.created ✅\n",
+    "🎙️ Speak now (Ctrl‑C to quit)…\n",
+    "—— response.done  (window ≈979 tokens) ——\n",
+    "—— Conversation so far ———————————————\n",
+    "[user     ] Can you tell me a quick story?  (item_BTuMOcpUqp8qknKhLzlkA)\n",
+    "[assistant] Once upon a time, in a cozy little village, there was a cat named Whiskers who was always getting into trouble. One sunny day, Whiskers found a mysterious glowing stone in the garden. Curious, he pawed at it, and poof! The stone granted him the ability to talk to birds. Whiskers and his new bird friends had grand adventures, solving mysteries and exploring the village. And from that day on, Whiskers was known as the most adventurous cat in the village. The end.  (item_BTuMPRWxqpv0ph6QM46DK)\n",
+    "——————————————————————————————————————————\n",
+    "—— response.done  (window ≈2755 tokens) ——\n",
+    "—— Conversation so far ———————————————\n",
+    "[user     ] Can you tell me a quick story?  (item_BTuMOcpUqp8qknKhLzlkA)\n",
+    "[assistant] Once upon a time, in a cozy little village, there was a cat named Whiskers who was always getting into trouble. One sunny day, Whiskers found a mysterious glowing stone in the garden. Curious, he pawed at it, and poof! The stone granted him the ability to talk to birds. Whiskers and his new bird friends had grand adventures, solving mysteries and exploring the village. And from that day on, Whiskers was known as the most adventurous cat in the village. The end.  (item_BTuMPRWxqpv0ph6QM46DK)\n",
+    "[user     ] Can you tell me three extremely funny stories?  (item_BTuNN64LdULM21OyC4vzN)\n",
+    "[assistant] Sure, let's dive into some giggle-worthy tales:  **Story One:** There was a forgetful baker named Benny who baked a hundred cakes for a big wedding. But on the big day, he forgot where he put them! The entire town joined in to find the missing cakes, only to discover Benny had stored them in his neighbor's garage, thinking it was his pantry. The wedding turned into a town-wide cake feast!  **Story Two:** A mischievous dog named Sparky loved to play pranks. One day, he swapped his owner's phone with a squeaky toy, causing a hilarious mix-up of barks, squeaks, and confused calls. Sparky's owner ended up having a full conversation with the mailman, all in squeaks!  **Story Three:** In a small town, a parrot named Polly became a local celebrity for reciting tongue twisters. One day, Polly challenged the mayor to a tongue twister duel. The mayor, tongue-tied and laughing, declared Polly the official town jester. Polly squawked with pride, and the town rang with laughter for days.  (item_BTuNNpNxki5ynSQ5c3Xsa)\n",
+    "——————————————————————————————————————————\n",
+    "⚠️  Token window ≈2755 ≥ 2000. Summarising…\n",
+    "—— Conversation so far ———————————————\n",
+    "[assistant] L'utilisateur a demandé une histoire rapide, et l'assistant a raconté celle d'un chat nommé Whiskers qui, après avoir trouvé une pierre mystérieuse dans son jardin, a obtenu le pouvoir de parler aux oiseaux. Avec ses nouveaux amis oiseaux, Whiskers a vécu de grandes aventures, résolvant des mystères et explorant le village, devenant ainsi le chat le plus aventurier du village.  (sum_001)\n",
+    "[user     ] Can you tell me three extremely funny stories?  (item_BTuNN64LdULM21OyC4vzN)\n",
+    "[assistant] Sure, let's dive into some giggle-worthy tales:  **Story One:** There was a forgetful baker named Benny who baked a hundred cakes for a big wedding. But on the big day, he forgot where he put them! The entire town joined in to find the missing cakes, only to discover Benny had stored them in his neighbor's garage, thinking it was his pantry. The wedding turned into a town-wide cake feast!  **Story Two:** A mischievous dog named Sparky loved to play pranks. One day, he swapped his owner's phone with a squeaky toy, causing a hilarious mix-up of barks, squeaks, and confused calls. Sparky's owner ended up having a full conversation with the mailman, all in squeaks!  **Story Three:** In a small town, a parrot named Polly became a local celebrity for reciting tongue twisters. One day, Polly challenged the mayor to a tongue twister duel. The mayor, tongue-tied and laughing, declared Polly the official town jester. Polly squawked with pride, and the town rang with laughter for days.  (item_BTuNNpNxki5ynSQ5c3Xsa)\n",
+    "——————————————————————————————————————————\n",
+    "✅ Summary inserted (sum_001)\n",
+    "—— response.done  (window ≈2147 tokens) ——\n",
+    "—— Conversation so far ———————————————\n",
+    "[assistant] L'utilisateur a demandé une histoire rapide, et l'assistant a raconté celle d'un chat nommé Whiskers qui, après avoir trouvé une pierre mystérieuse dans son jardin, a obtenu le pouvoir de parler aux oiseaux. Avec ses nouveaux amis oiseaux, Whiskers a vécu de grandes aventures, résolvant des mystères et explorant le village, devenant ainsi le chat le plus aventurier du village.  (sum_001)\n",
+    "[user     ] Can you tell me three extremely funny stories?  (item_BTuNN64LdULM21OyC4vzN)\n",
+    "[assistant] Sure, let's dive into some giggle-worthy tales:  **Story One:** There was a forgetful baker named Benny who baked a hundred cakes for a big wedding. But on the big day, he forgot where he put them! The entire town joined in to find the missing cakes, only to discover Benny had stored them in his neighbor's garage, thinking it was his pantry. The wedding turned into a town-wide cake feast!  **Story Two:** A mischievous dog named Sparky loved to play pranks. One day, he swapped his owner's phone with a squeaky toy, causing a hilarious mix-up of barks, squeaks, and confused calls. Sparky's owner ended up having a full conversation with the mailman, all in squeaks!  **Story Three:** In a small town, a parrot named Polly became a local celebrity for reciting tongue twisters. One day, Polly challenged the mayor to a tongue twister duel. The mayor, tongue-tied and laughing, declared Polly the official town jester. Polly squawked with pride, and the town rang with laughter for days.  (item_BTuNNpNxki5ynSQ5c3Xsa)\n",
+    "[user     ]   (item_BTuPLaCv8ATdIwAQ2rLgO)\n",
+    "[assistant] Sure! The first summary I provided between us was in French.  (item_BTuPLa7BaSQToGCVOmfBK)\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "We had a conversation with our Voice AI. After several turns, the total token count reached SUMMARY_MAX, which triggered the conversation summarization step. This generated a summary of the earlier messages.\n",
+    "\n",
+    "Since there were N = 4 total messages, we summarized the first N - 2 = 2 messages:\n",
+    "```txt\n",
+    "—— Conversation so far ———————————————\n",
+    "[user     ] Can you tell me a quick story?  (item_BTuMOcpUqp8qknKhLzlkA)\n",
+    "[assistant] Once upon a time, in a cozy little village, there was a cat named Whiskers who was always getting into trouble. One sunny day, Whiskers found a mysterious glowing stone in the garden. Curious, he pawed at it, and poof! The stone granted him the ability to talk to birds. Whiskers and his new bird friends had grand adventures, solving mysteries and exploring the village. And from that day on, Whiskers was known as the most adventurous cat in the village. The end.  (item_BTuMPRWxqpv0ph6QM46DK)\n",
+    "```\n",
+    "\n",
+    "We then created a summary in French and inserted it into the conversation history using the root: true flag. This ensured the summary appeared as the first message in the conversation. After that, we deleted the original items, using `\"type\": \"conversation.item.delete\"`, that were summarized.\n",
+    "\n",
+    "To validate the summary insertion, we asked the Voice AI what language the summary was in. It correctly responded:\n",
+    "\n",
+    "```txt\n",
+    "[assistant] Sure! The first summary I provided between us was in French.  (item_BTuPLa7BaSQToGCVOmfBK)\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 5 · Real‑World Applications\n",
+    "\n",
+    "Context summarisation can be useful for **long‑running voice experiences**.  \n",
+    "Here are a use case ideas:\n",
+    "\n",
+    "| Use‑case | Added Value | Why Useful |\n",
+    "|----------|-------------|------------|\n",
+    "| **Customer‑support voicebot** | 24/7 natural phone tree; auto‑generate ticket summaries | Summarizes long customer calls for efficient handoff and record-keeping, reducing agent workload and improving response quality. |\n",
+    "| **Language tutor** | Real‑time conversation practice with corrective feedback | Helps track learner progress and highlights recurring mistakes, enabling personalized feedback and more effective language acquisition. |\n",
+    "| **AI therapist / coach** | Safe, always‑available listener that remembers sessions | Maintains continuity across sessions by recalling key topics and emotional tone, supporting a more empathetic and effective experience. |\n",
+    "| **Meeting assistant** | Live transcripts + concise action‑item recap in Slack | Distills lengthy meetings into actionable summaries, saving team members time and ensuring important points are not missed. |\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 6 · Next Steps & Further Reading\n",
+    "Try out the notebook and try integrating context summary into your application.\n",
+    "\n",
+    "Few things you can try:\n",
+    "| Try this… | What you’ll learn |\n",
+    "|-----------|------------------|\n",
+    "| **A/B test summarisation**<br/>Run your eval suite with summarisation *on* vs *off*. | Whether trimming actually improves quality for your domain—and how it affects latency & cost. |\n",
+    "| **Swap summary styles**<br/>Change the system prompt to bullet points, JSON, English vs French, etc. | Which format the downstream assistant absorbs best; how language choice influences follow‑up answers. |\n",
+    "| **Vary thresholds**<br/>Play with `SUMMARY_TRIGGER_TOKENS` (2 k → 8 k). | The sweet spot between model drift and summarisation overhead. |\n",
+    "| **Cost tracing**<br/>Log `usage.total_tokens` before/after summarisation. | Concrete ROI: token savings per hour of conversation. |\n",
+    "\n",
+    "\n",
+    "### Resources:\n",
+    "- [OpenAI Realtime Guide](https://platform.openai.com/docs/guides/realtime)\n",
+    "- [OpenAI Realtime Conversations](https://platform.openai.com/docs/guides/realtime-conversations)\n",
+    "- [OpenAI Realtime API Reference](https://platform.openai.com/docs/api-reference/realtime)\n",
+    "- [Voice AI and Voice Agents](https://voiceaiandvoiceagents.com/)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "openai",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/Creating_slides_with_Assistants_API_and_DALL-E3.ipynb

@@ -0,0 +1,993 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Creating slides with the Assistants API (GPT-4), and DALL·E-3\n",
+    "\n",
+    "This notebook illustrates the use of the new [Assistants API](https://platform.openai.com/docs/assistants/overview) (GPT-4), and DALL·E-3 in crafting informative and visually appealing slides. <br>\n",
+    "Creating slides is a pivotal aspect of many jobs, but can be laborious and time-consuming. Additionally, extracting insights from data and articulating them effectively on slides can be challenging. <br><br> This cookbook recipe will demonstrate how you can utilize the new Assistants API to facilitate the end to end slide creation process for you without you having to touch Microsoft PowerPoint or Google Slides, saving you valuable time and effort!"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 0. Setup"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from IPython.display import display, Image\n",
+    "from openai import OpenAI\n",
+    "import os\n",
+    "import pandas as pd\n",
+    "import json\n",
+    "import io\n",
+    "from PIL import Image\n",
+    "import requests\n",
+    "\n",
+    "client = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))\n",
+    "\n",
+    "#Lets import some helper functions for assistants from https://cookbook.openai.com/examples/assistants_api_overview_python\n",
+    "def show_json(obj):\n",
+    "    display(json.loads(obj.model_dump_json()))\n",
+    "\n",
+    "def submit_message(assistant_id, thread, user_message,file_ids=None):\n",
+    "    params = {\n",
+    "        'thread_id': thread.id,\n",
+    "        'role': 'user',\n",
+    "        'content': user_message,\n",
+    "    }\n",
+    "    if file_ids:\n",
+    "        params['file_ids']=file_ids\n",
+    "\n",
+    "    client.beta.threads.messages.create(\n",
+    "        **params\n",
+    ")\n",
+    "    return client.beta.threads.runs.create(\n",
+    "    thread_id=thread.id,\n",
+    "    assistant_id=assistant_id,\n",
+    ")\n",
+    "\n",
+    "def get_response(thread):\n",
+    "    return client.beta.threads.messages.list(thread_id=thread.id)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Creating the content"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In this recipe, we will be creating a brief fictional presentation for the quarterly financial review of our company, NotReal Corporation. We want to highlight some key trends we are seeing that are affecting the profitability of our company.<br> Let's say we have the some financial data at our disposal. Let's load in the data, and take a look..."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>Year</th>\n",
+       "      <th>Quarter</th>\n",
+       "      <th>Distribution channel</th>\n",
+       "      <th>Revenue ($M)</th>\n",
+       "      <th>Costs ($M)</th>\n",
+       "      <th>Customer count</th>\n",
+       "      <th>Time</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>2021</td>\n",
+       "      <td>Q1</td>\n",
+       "      <td>Online Sales</td>\n",
+       "      <td>1.50</td>\n",
+       "      <td>1.301953</td>\n",
+       "      <td>150</td>\n",
+       "      <td>2021 Q1</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>2021</td>\n",
+       "      <td>Q1</td>\n",
+       "      <td>Direct Sales</td>\n",
+       "      <td>1.50</td>\n",
+       "      <td>1.380809</td>\n",
+       "      <td>151</td>\n",
+       "      <td>2021 Q1</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>2021</td>\n",
+       "      <td>Q1</td>\n",
+       "      <td>Retail Partners</td>\n",
+       "      <td>1.50</td>\n",
+       "      <td>1.348246</td>\n",
+       "      <td>152</td>\n",
+       "      <td>2021 Q1</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>2021</td>\n",
+       "      <td>Q2</td>\n",
+       "      <td>Online Sales</td>\n",
+       "      <td>1.52</td>\n",
+       "      <td>1.308608</td>\n",
+       "      <td>152</td>\n",
+       "      <td>2021 Q2</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>2021</td>\n",
+       "      <td>Q2</td>\n",
+       "      <td>Direct Sales</td>\n",
+       "      <td>1.52</td>\n",
+       "      <td>1.413305</td>\n",
+       "      <td>153</td>\n",
+       "      <td>2021 Q2</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "   Year Quarter Distribution channel  Revenue ($M)  Costs ($M)  \\\n",
+       "0  2021      Q1         Online Sales          1.50    1.301953   \n",
+       "1  2021      Q1         Direct Sales          1.50    1.380809   \n",
+       "2  2021      Q1      Retail Partners          1.50    1.348246   \n",
+       "3  2021      Q2         Online Sales          1.52    1.308608   \n",
+       "4  2021      Q2         Direct Sales          1.52    1.413305   \n",
+       "\n",
+       "   Customer count     Time  \n",
+       "0             150  2021 Q1  \n",
+       "1             151  2021 Q1  \n",
+       "2             152  2021 Q1  \n",
+       "3             152  2021 Q2  \n",
+       "4             153  2021 Q2  "
+      ]
+     },
+     "execution_count": 2,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "financial_data_path = 'data/NotRealCorp_financial_data.json'\n",
+    "financial_data = pd.read_json(financial_data_path)\n",
+    "financial_data.head(5)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "As you can see, this data has quarterly revenue, costs and customer data across different distribution channels. Let's create an Assistant\n",
+    "that can act as a personal analyst and make a nice visualization for our PowerPoint!"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "First, we need to upload our file so our Assistant can access it."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "file = client.files.create(\n",
+    "  file=open('data/NotRealCorp_financial_data.json',\"rb\"),\n",
+    "  purpose='assistants',\n",
+    ")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now, we're ready to create our Assistant. We can instruct our assistant to act as a data scientist, and take any queries we give it and run the necessary code to output the proper data visualization. The instructions parameter here is akin to system instructions in the ChatCompletions endpoint, and can help guide the assistant. We can also turn on the tool of Code Interpreter, so our Assistant will be able to code. Finally, we can specifiy any files we want to use, which in this case is just the `financial_data` file we created above."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "assistant = client.beta.assistants.create(\n",
+    "  instructions=\"You are a data scientist assistant. When given data and a query, write the proper code and create the proper visualization\",\n",
+    "  model=\"gpt-4-1106-preview\",\n",
+    "  tools=[{\"type\": \"code_interpreter\"}],\n",
+    "  file_ids=[file.id]\n",
+    ")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let's create a thread now, and as our first request ask the Assistant to calculate quarterly profits, and then plot the profits by distribution channel over time. The assistant will automatically calculate the profit for each quarter, and also create a new column combining quarter and year, without us having to ask for that directly. We can also specify the colors of each line."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "thread = client.beta.threads.create(\n",
+    "  messages=[\n",
+    "    {\n",
+    "      \"role\": \"user\",\n",
+    "      \"content\": \"Calculate profit (revenue minus cost) by quarter and year, and visualize as a line plot across the distribution channels, where the colors of the lines are green, light red, and light blue\",\n",
+    "      \"file_ids\": [file.id]\n",
+    "    }\n",
+    "  ]\n",
+    ")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "No we can execute the run of our thread"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "\n",
+    "run = client.beta.threads.runs.create(\n",
+    "    thread_id=thread.id,\n",
+    "    assistant_id=assistant.id,\n",
+    ")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can now start a loop that will check if the image has been created. Note: This may take a few minutes"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "messages = client.beta.threads.messages.list(thread_id=thread.id)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Assistant still working...\n",
+      "Assistant still working...\n",
+      "Assistant still working...\n",
+      "Assistant still working...\n",
+      "Assistant still working...\n",
+      "Assistant still working...\n",
+      "Assistant still working...\n",
+      "Assistant still working...\n",
+      "Assistant still working...\n",
+      "Assistant still working...\n",
+      "Assistant still working...\n",
+      "Assistant still working...\n",
+      "Assistant still working...\n",
+      "Assistant still working...\n",
+      "Assistant still working...\n",
+      "Assistant still working...\n",
+      "Assistant still working...\n",
+      "Plot created!\n"
+     ]
+    }
+   ],
+   "source": [
+    "import time\n",
+    "\n",
+    "while True:\n",
+    "    messages = client.beta.threads.messages.list(thread_id=thread.id)\n",
+    "    try:\n",
+    "        #See if image has been created\n",
+    "        messages.data[0].content[0].image_file\n",
+    "        #Sleep to make sure run has completed\n",
+    "        time.sleep(5)\n",
+    "        print('Plot created!')\n",
+    "        break\n",
+    "    except:\n",
+    "        time.sleep(10)\n",
+    "        print('Assistant still working...')\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let's see the messages the Assistant added."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[MessageContentImageFile(image_file=ImageFile(file_id='file-0rKABLygI02MgwwhpgWdRFY1'), type='image_file'),\n",
+       " MessageContentText(text=Text(annotations=[], value=\"The profit has been calculated for each distribution channel by quarter and year. Next, I'll create a line plot to visualize these profits. As specified, I will use green for the 'Online Sales', light red for 'Direct Sales', and light blue for 'Retail Partners' channels. Let's create the plot.\"), type='text'),\n",
+       " MessageContentText(text=Text(annotations=[], value=\"The JSON data has been successfully restructured into a tabular dataframe format. It includes the year, quarter, distribution channel, revenue, costs, customer count, and a combined 'Time' representation of 'Year Quarter'. Now, we have the necessary information to calculate the profit (revenue minus cost) by quarter and year.\\n\\nTo visualize the profit across the different distribution channels with a line plot, we will proceed with the following steps:\\n\\n1. Calculate the profit for each row in the dataframe.\\n2. Group the data by 'Time' (which is a combination of Year and Quarter) and 'Distribution channel'.\\n3. Aggregate the profit for each group.\\n4. Plot the aggregated profits as a line plot with the distribution channels represented in different colors as requested.\\n\\nLet's calculate the profit for each row and then continue with the visualization.\"), type='text'),\n",
+       " MessageContentText(text=Text(annotations=[], value='The structure of the JSON data shows that it is a dictionary with \"Year\", \"Quarter\", \"Distribution channel\", and potentially other keys that map to dictionaries containing the data. The keys of the inner dictionaries are indices, indicating that the data is tabular but has been converted into a JSON object.\\n\\nTo properly convert this data into a DataFrame, I will restructure the JSON data into a more typical list of dictionaries, where each dictionary represents a row in our target DataFrame. Subsequent to this restructuring, I can then load the data into a Pandas DataFrame. Let\\'s restructure and load the data.'), type='text'),\n",
+       " MessageContentText(text=Text(annotations=[], value=\"The JSON data has been incorrectly loaded into a single-row DataFrame with numerous columns representing each data point. This implies the JSON structure is not as straightforward as expected, and a direct conversion to a flat table is not possible without further processing.\\n\\nTo better understand the JSON structure and figure out how to properly normalize it into a table format, I will print out the raw JSON data structure. We will analyze its format and then determine the correct approach to extract the profit by quarter and year, as well as the distribution channel information. Let's take a look at the JSON structure.\"), type='text'),\n",
+       " MessageContentText(text=Text(annotations=[], value=\"It seems that the file content was successfully parsed as JSON, and thus, there was no exception raised. The variable `error_message` is not defined because the `except` block was not executed.\\n\\nI'll proceed with displaying the data that was parsed from JSON.\"), type='text'),\n",
+       " MessageContentText(text=Text(annotations=[], value=\"It appears that the content of the dataframe has been incorrectly parsed, resulting in an empty dataframe with a very long column name that seems to contain JSON data rather than typical CSV columns and rows.\\n\\nTo address this issue, I will take a different approach to reading the file. I will attempt to parse the content as JSON. If this is not successful, I'll adjust the loading strategy accordingly. Let's try to read the contents as JSON data first.\"), type='text'),\n",
+       " MessageContentText(text=Text(annotations=[], value=\"Before we can calculate profits and visualize the data as requested, I need to first examine the contents of the file that you have uploaded. Let's go ahead and read the file to understand its structure and the kind of data it contains. Once I have a clearer picture of the dataset, we can proceed with the profit calculations. I'll begin by loading the file into a dataframe and displaying the first few entries to see the data schema.\"), type='text'),\n",
+       " MessageContentText(text=Text(annotations=[], value='Calculate profit (revenue minus cost) by quarter and year, and visualize as a line plot across the distribution channels, where the colors of the lines are green, light red, and light blue'), type='text')]"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "messages = client.beta.threads.messages.list(thread_id=thread.id)\n",
+    "[message.content[0] for message in messages.data]\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can see that the last message (latest message is shown first) from the assistant contains the image file we are looking for. An interesting note here is that the Assistant was able to attempt several times to parse the JSON data, as the first parsing was unsuccessful, demonstrating the assistant's adaptability."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Quick helper function to convert our output file to a png\n",
+    "def convert_file_to_png(file_id, write_path):\n",
+    "    data = client.files.content(file_id)\n",
+    "    data_bytes = data.read()\n",
+    "    with open(write_path, \"wb\") as file:\n",
+    "        file.write(data_bytes)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "plot_file_id = messages.data[0].content[0].image_file.file_id\n",
+    "image_path = \"../images/NotRealCorp_chart.png\"\n",
+    "convert_file_to_png(plot_file_id,image_path)\n",
+    "\n",
+    "#Upload\n",
+    "plot_file = client.files.create(\n",
+    "  file=open(image_path, \"rb\"),\n",
+    "  purpose='assistants'\n",
+    ")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let's load in the plot!"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "![The Image](../images/NotRealCorp_chart.png)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Nice! So, with just one sentence, we were able to have our assistant use code interpreter to\n",
+    "calculate the profitability, and graph the three lineplots of the various distribution channels.<br><br>\n",
+    "Now we have a nice visual for our slide, but we want some insights to go along with it."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Generating insights"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To get insights from our image, we simply need to add a new message to our thread. Our Assistant will know to use the message history to give us some concise takeaways from the visual provided. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "Run(id='run_NWoygMcBfHUr58fCE4Cn6rxN', assistant_id='asst_3T362kLlTyAq0FUnkvjjQczO', cancelled_at=None, completed_at=None, created_at=1701827074, expires_at=1701827674, failed_at=None, file_ids=['file-piTokyHGllwGITzIpoG8dok3'], instructions='You are a data scientist assistant. When given data and a query, write the proper code and create the proper visualization', last_error=None, metadata={}, model='gpt-4-1106-preview', object='thread.run', required_action=None, started_at=None, status='queued', thread_id='thread_73TgtFoJMlEJvb13ngjTnAo3', tools=[ToolAssistantToolsCode(type='code_interpreter')])"
+      ]
+     },
+     "execution_count": 12,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "submit_message(assistant.id,thread,\"Give me two medium length sentences (~20-30 words per sentence) of the \\\n",
+    "      most important insights from the plot you just created.\\\n",
+    "             These will be used for a slide deck, and they should be about the\\\n",
+    "                     'so what' behind the data.\"\n",
+    ")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now, once the run has completed, we can see the latest message"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "The plot reveals a consistent upward trend in profits for all distribution channels, indicating successful business growth over time. Particularly, 'Online Sales' shows a notable increase, underscoring the importance of digital presence in revenue generation.\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Hard coded wait for a response, as the assistant may iterate on the bullets.\n",
+    "time.sleep(10)\n",
+    "response = get_response(thread)\n",
+    "bullet_points = response.data[0].content[0].text.value\n",
+    "print(bullet_points)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Cool! So our assistant was able to identify the noteworthy growth in Online Sales profit, and infer that this shows the importance of a large digital presence. Now let's get a compelling title for the slide."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "Run(id='run_q6E85J31jCw3QkHpjJKl969P', assistant_id='asst_3T362kLlTyAq0FUnkvjjQczO', cancelled_at=None, completed_at=None, created_at=1701827084, expires_at=1701827684, failed_at=None, file_ids=['file-piTokyHGllwGITzIpoG8dok3'], instructions='You are a data scientist assistant. When given data and a query, write the proper code and create the proper visualization', last_error=None, metadata={}, model='gpt-4-1106-preview', object='thread.run', required_action=None, started_at=None, status='queued', thread_id='thread_73TgtFoJMlEJvb13ngjTnAo3', tools=[ToolAssistantToolsCode(type='code_interpreter')])"
+      ]
+     },
+     "execution_count": 14,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "submit_message(assistant.id,thread,\"Given the plot and bullet points you created,\\\n",
+    " come up with a very brief title for a slide. It should reflect just the main insights you came up with.\"\n",
+    ")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "And the title is:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\"Ascending Profits & Digital Dominance\"\n"
+     ]
+    }
+   ],
+   "source": [
+    "#Wait as assistant may take a few steps\n",
+    "time.sleep(10)\n",
+    "response = get_response(thread)\n",
+    "title = response.data[0].content[0].text.value\n",
+    "print(title)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. DALL·E-3 title image"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Nice, now we have a title, a plot and two bullet points. We're almost ready to put this all on a slide, but as a final step, let's have DALL·E-3 come up with an image to use as the title slide of the presentation. <br><br>\n",
+    "*Note:* DALL·E-3 is not yet available within the assistants API but is coming soon! <br> <br>\n",
+    "We'll feed in a brief description of our company (NotRealCorp) and have DALL·E-3 do the rest!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "company_summary = \"NotReal Corp is a prominent hardware company that manufactures and sells processors, graphics cards and other essential computer hardware.\"\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = client.images.generate(\n",
+    "  model='dall-e-3',\n",
+    "  prompt=f\"given this company summary {company_summary}, create an inspirational \\\n",
+    "    photo showing the growth and path forward. This will be used at a quarterly\\\n",
+    "       financial planning meeting\",\n",
+    "       size=\"1024x1024\",\n",
+    "       quality=\"hd\",\n",
+    "       n=1\n",
+    ")\n",
+    "image_url = response.data[0].url\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Cool, now we can add this image to our thread. First, we can save the image locally, then upload it to the assistants API using the `File` upload endpoint. Let's also take a look at our image"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "dalle_img_path = '../images/dalle_image.png'\n",
+    "img = requests.get(image_url)\n",
+    "\n",
+    "#Save locally\n",
+    "with open(dalle_img_path,'wb') as file:\n",
+    "  file.write(img.content)\n",
+    "\n",
+    "#Upload\n",
+    "dalle_file = client.files.create(\n",
+    "  file=open(dalle_img_path, \"rb\"),\n",
+    "  purpose='assistants'\n",
+    ")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "  \n",
+    "![Image](../images/dalle_image.png)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4. Creating the slides"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We now have all the content we need to create the slides. While we could simply add a message asking for slides, but let's instead give the assistant a slide template, using the `python-pptx` library, to use. This will ensure we get a deck in the style we want. See the `Extensions` section at the end of the notebook for notes on creating the template."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "title_template = \"\"\"\n",
+    "from pptx import Presentation\n",
+    "from pptx.util import Inches, Pt\n",
+    "from pptx.enum.text import PP_PARAGRAPH_ALIGNMENT\n",
+    "from pptx.dml.color import RGBColor\n",
+    "\n",
+    "# Create a new presentation object\n",
+    "prs = Presentation()\n",
+    "\n",
+    "# Add a blank slide layout\n",
+    "blank_slide_layout = prs.slide_layouts[6]\n",
+    "slide = prs.slides.add_slide(blank_slide_layout)\n",
+    "\n",
+    "# Set the background color of the slide to black\n",
+    "background = slide.background\n",
+    "fill = background.fill\n",
+    "fill.solid()\n",
+    "fill.fore_color.rgb = RGBColor(0, 0, 0)\n",
+    "\n",
+    "# Add image to the left side of the slide with a margin at the top and bottom\n",
+    "left = Inches(0)\n",
+    "top = Inches(0)\n",
+    "height = prs.slide_height\n",
+    "width = prs.slide_width * 3/5\n",
+    "pic = slide.shapes.add_picture(image_path, left, top, width=width, height=height)\n",
+    "\n",
+    "# Add title text box positioned higher\n",
+    "left = prs.slide_width * 3/5\n",
+    "top = Inches(2)\n",
+    "width = prs.slide_width * 2/5\n",
+    "height = Inches(1)\n",
+    "title_box = slide.shapes.add_textbox(left, top, width, height)\n",
+    "title_frame = title_box.text_frame\n",
+    "title_p = title_frame.add_paragraph()\n",
+    "title_p.text = title_text\n",
+    "title_p.font.bold = True\n",
+    "title_p.font.size = Pt(38)\n",
+    "title_p.font.color.rgb = RGBColor(255, 255, 255)\n",
+    "title_p.alignment = PP_PARAGRAPH_ALIGNMENT.CENTER\n",
+    "\n",
+    "# Add subtitle text box\n",
+    "left = prs.slide_width * 3/5\n",
+    "top = Inches(3)\n",
+    "width = prs.slide_width * 2/5\n",
+    "height = Inches(1)\n",
+    "subtitle_box = slide.shapes.add_textbox(left, top, width, height)\n",
+    "subtitle_frame = subtitle_box.text_frame\n",
+    "subtitle_p = subtitle_frame.add_paragraph()\n",
+    "subtitle_p.text = subtitle_text\n",
+    "subtitle_p.font.size = Pt(22)\n",
+    "subtitle_p.font.color.rgb = RGBColor(255, 255, 255)\n",
+    "subtitle_p.alignment = PP_PARAGRAPH_ALIGNMENT.CENTER\n",
+    "\"\"\"\n",
+    "\n",
+    "data_vis_template = \"\"\"\n",
+    "from pptx import Presentation\n",
+    "from pptx.util import Inches, Pt\n",
+    "from pptx.enum.text import PP_PARAGRAPH_ALIGNMENT\n",
+    "from pptx.dml.color import RGBColor\n",
+    "\n",
+    "# Create a new presentation object\n",
+    "prs = Presentation()\n",
+    "\n",
+    "# Add a blank slide layout\n",
+    "blank_slide_layout = prs.slide_layouts[6]\n",
+    "slide = prs.slides.add_slide(blank_slide_layout)\n",
+    "\n",
+    "# Set the background color of the slide to black\n",
+    "background = slide.background\n",
+    "fill = background.fill\n",
+    "fill.solid()\n",
+    "fill.fore_color.rgb = RGBColor(0, 0, 0)\n",
+    "\n",
+    "# Define placeholders\n",
+    "image_path = data_vis_img\n",
+    "title_text = \"Maximizing Profits: The Dominance of Online Sales & Direct Sales Optimization\"\n",
+    "bullet_points = \"• Online Sales consistently lead in profitability across quarters, indicating a strong digital market presence.\\n• Direct Sales show fluctuations, suggesting variable performance and the need for targeted improvements in that channel.\"\n",
+    "\n",
+    "# Add image placeholder on the left side of the slide\n",
+    "left = Inches(0.2)\n",
+    "top = Inches(1.8)\n",
+    "height = prs.slide_height - Inches(3)\n",
+    "width = prs.slide_width * 3/5\n",
+    "pic = slide.shapes.add_picture(image_path, left, top, width=width, height=height)\n",
+    "\n",
+    "# Add title text spanning the whole width\n",
+    "left = Inches(0)\n",
+    "top = Inches(0)\n",
+    "width = prs.slide_width\n",
+    "height = Inches(1)\n",
+    "title_box = slide.shapes.add_textbox(left, top, width, height)\n",
+    "title_frame = title_box.text_frame\n",
+    "title_frame.margin_top = Inches(0.1)\n",
+    "title_p = title_frame.add_paragraph()\n",
+    "title_p.text = title_text\n",
+    "title_p.font.bold = True\n",
+    "title_p.font.size = Pt(28)\n",
+    "title_p.font.color.rgb = RGBColor(255, 255, 255)\n",
+    "title_p.alignment = PP_PARAGRAPH_ALIGNMENT.CENTER\n",
+    "\n",
+    "# Add hardcoded \"Key Insights\" text and bullet points\n",
+    "left = prs.slide_width * 2/3\n",
+    "top = Inches(1.5)\n",
+    "width = prs.slide_width * 1/3\n",
+    "height = Inches(4.5)\n",
+    "insights_box = slide.shapes.add_textbox(left, top, width, height)\n",
+    "insights_frame = insights_box.text_frame\n",
+    "insights_p = insights_frame.add_paragraph()\n",
+    "insights_p.text = \"Key Insights:\"\n",
+    "insights_p.font.bold = True\n",
+    "insights_p.font.size = Pt(24)\n",
+    "insights_p.font.color.rgb = RGBColor(0, 128, 100)\n",
+    "insights_p.alignment = PP_PARAGRAPH_ALIGNMENT.LEFT\n",
+    "insights_frame.add_paragraph()\n",
+    "\n",
+    "\n",
+    "bullet_p = insights_frame.add_paragraph()\n",
+    "bullet_p.text = bullet_points\n",
+    "bullet_p.font.size = Pt(12)\n",
+    "bullet_p.font.color.rgb = RGBColor(255, 255, 255)\n",
+    "bullet_p.line_spacing = 1.5\n",
+    "\"\"\"\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let's set a few quick variables for our slides. We want the company name, NotRealCorp, to be on the title slide, and the title of the presentation should 'Quartlerly financial planning metting, Q3, 2023'."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "title_text = \"NotRealCorp\"\n",
+    "subtitle_text = \"Quarterly financial planning meeting, Q3 2023\"\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "And for the data slide, we have:"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Here we have a template to create a Title Slide. The template below was created by uploading the image of a desirable title slide to GPT-V, and asking for the `python-pptx` code to create that template. The inputs to the template are the image_path, title_text, and subtitle_text."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "Run(id='run_taLrnOnlDhoywgQFFBOLPlg0', assistant_id='asst_3T362kLlTyAq0FUnkvjjQczO', cancelled_at=None, completed_at=None, created_at=1701827118, expires_at=1701827718, failed_at=None, file_ids=['file-piTokyHGllwGITzIpoG8dok3'], instructions='You are a data scientist assistant. When given data and a query, write the proper code and create the proper visualization', last_error=None, metadata={}, model='gpt-4-1106-preview', object='thread.run', required_action=None, started_at=None, status='queued', thread_id='thread_73TgtFoJMlEJvb13ngjTnAo3', tools=[ToolAssistantToolsCode(type='code_interpreter')])"
+      ]
+     },
+     "execution_count": 21,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "submit_message(assistant.id,thread,f\"Use the included code template to create a PPTX slide that follows the template format, but uses the image, company name/title, and document name/subtitle included:\\\n",
+    "{title_template}. IMPORTANT: Use the image file included in this message as the image_path image in this first slide, and use the Company Name {title_text} as the title_text variable, and \\\n",
+    "  use the subtitle_text {subtitle_text} a the subtitle_text variable. \\\n",
+    "    NEST, create a SECOND slide using the following code template: {data_vis_template} to create a PPTX slide that follows the template format, but uses the company name/title, and document name/subtitle included:\\\n",
+    "{data_vis_template}. IMPORTANT: Use the line plot image, that is the second attached image in this message, that you created earlier in the thread as the data_vis_img image, and use the data visualization title that you created earlier for the variable title_text, and\\\n",
+    "  the bullet points of insights you created earlier for the bullet_points variable. Output these TWO SLIDES as a .pptx file. Make sure the output is two slides, with each slide matching the respective template given in this message.\",\n",
+    "              file_ids=[dalle_file.id, plot_file.id]\n",
+    ")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Assistant still working on PPTX...\n",
+      "Assistant still working on PPTX...\n",
+      "Assistant still working on PPTX...\n",
+      "Assistant still working on PPTX...\n",
+      "Assistant still working on PPTX...\n",
+      "Assistant still working on PPTX...\n",
+      "Assistant still working on PPTX...\n",
+      "Assistant still working on PPTX...\n",
+      "Assistant still working on PPTX...\n",
+      "Assistant still working on PPTX...\n",
+      "Successfully retrieved pptx_id: file-oa0i63qPH4IaJXYj90aA6L4Q\n"
+     ]
+    }
+   ],
+   "source": [
+    "#May take 1-3 mins\n",
+    "while True:\n",
+    "    try:\n",
+    "        response = get_response(thread)\n",
+    "        pptx_id = response.data[0].content[0].text.annotations[0].file_path.file_id\n",
+    "        print(\"Successfully retrieved pptx_id:\", pptx_id)\n",
+    "        break\n",
+    "    except Exception as e:\n",
+    "        print(\"Assistant still working on PPTX...\")\n",
+    "        time.sleep(10)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 25,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pptx_id = response.data[0].content[0].text.annotations[0].file_path.file_id\n",
+    "ppt_file= client.files.content(pptx_id)\n",
+    "file_obj = io.BytesIO(ppt_file.read())\n",
+    "with open(\"data/created_slides.pptx\", \"wb\") as f:\n",
+    "    f.write(file_obj.getbuffer())\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now, we have a PPTX file saved with all of our created content!. <br>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let's look at the screenshots of the .pptx we just created using JUST the assistants API and DALL·E-3. We don't have a `seed` parameter yet in the Assistants API, so the DALL·E-3 image and wordings will be slightly different from what you see when you run this notebook, due to the non-determinism of LLMs, but the outputs should be directionally the same."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The title slide:"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "![Title Slide](../images/title_slide.png)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "And the data slide:"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "![Data Slide](../images/data_vis_slide.png)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 5. Conclusion"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Woo! While these slides could use some formatting tweaks, we have made some great content using the Assistants API, GPT-4 and DALL·E-3. We were able to take a `.csv` file with financial data, and use our assisant to calculate profit by quarter across distribution channels, plot the results, identify insights and key takeaways from the visualization, and create a summarative title. And, given just a description of our company, NotRealCorp, we used DALL·E-3 to make an awesome title image. <br><br>\n",
+    "While we are still a ways away from entirely automating this process without a human in the loop, hopefully this notebook can make the slide creation process a bit easier for you. More importantly, this notebook can ideally give you a glimpse into the potential of the assistants API! We're excited to see what you build."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 6. Extensions"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "- When  DALL·E-3 is incorporated in the Assistants API, we will have the ability to request the generated title image within the thread. \n",
+    "- GPT-4-Vision is not yet supported in the Assistants API, but could have been used to gather insights from the line plot image.\n",
+    "- GPT-4-Vision was used to generate the `python-pptx` template included in this recipe, so a potential extension project could be demonstrating best practices around converting images to slide templates."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "openai",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.16"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/Custom-LLM-as-a-Judge.ipynb

@@ -0,0 +1,964 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Building an LLM-as-a-judge evaluation to detect hallucinations with Braintrust\n",
+    "\n",
+    "Let's say you're working on a customer service bot and trying to evaluate the quality of its responses. Consider a question like \"What is your return policy?\" If the correct answer is \"You can return items within 30 days of purchase,\" but your bot generates \"You can return items within 30 days,\" how would you evaluate whether this is a good response?\n",
+    "\n",
+    "A heuristic like the `Levenshtein` string distance would indicate that the response is incorrect. However, a better approach is to use an LLM-as-a-judge to assess the accuracy of the response. LLM-as-a-judge is a technique that leverages an LLM to score the quality of answers. LLMs can reason about language beyond surface-level string comparisons, enabling them to evaluate answers more accurately.\n",
+    "\n",
+    "In this cookbook, we'll walk through how to build an LLM-as-a-judge scorer that can detect hallucinations using [Braintrust](https://www.braintrust.dev/), a third-party evaluation platform that is compatible with OpenAI's models.\n",
+    "\n",
+    "## Installing dependencies\n",
+    "\n",
+    "Let's install a few basic dependencies. We'll use the CoQA dataset (via DuckDB), [Braintrust](https://www.braintrust.dev/) for evals, and [OpenAI's models](https://platform.openai.com/docs/models). Please note that Braintrust is a third-party evaluation platform and you should review their [terms of service and privacy policy](https://www.braintrust.dev/legal/terms-of-service) before proceeding.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Note: you may need to restart the kernel to use updated packages.\n"
+     ]
+    }
+   ],
+   "source": [
+    "%pip install autoevals duckdb braintrust openai --quiet\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Next, let's initialize the OpenAI client. We'll use the `AsyncOpenAI` client so that we can parallelize our requests. The `braintrust.wrap_openai` function\n",
+    "wraps the OpenAI client to enable logging LLM calls to [Braintrust](https://www.braintrust.dev/). We'll use Braintrust to facilitate the evaluations below.\n",
+    "Before proceeding, you should sign up for a [Braintrust account](https://www.braintrust.dev/signup) and set `BRAINTRUST_API_KEY` in your environment to a valid API key.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "import braintrust\n",
+    "from openai import AsyncOpenAI\n",
+    "\n",
+    "braintrust.login(api_key=os.environ[\"BRAINTRUST_API_KEY\"])\n",
+    "client = braintrust.wrap_openai(AsyncOpenAI(api_key=os.environ[\"OPENAI_API_KEY\"]))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Explore the dataset\n",
+    "\n",
+    "We'll use the [CoQA dataset](https://stanfordnlp.github.io/coqa/) which contains a diverse set of passages, questions, and answers. Because CoQA is quite large, we'll just look at the first several passages. As with any public dataset, there's a chance that the underlying LLMs have memorized aspects of the dataset, so when developing your own scorers, it's a good idea to test them using\n",
+    "your own private data.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Passage:\n",
+      "(CNN)A chiseled boxer's Instagram feed shows him making constant references to the Bible and enjoying gospel singing with his wife. \n",
+      "\n",
+      "Another features his formidable opponent counting stacks of money, hanging out in strip clubs, and flashing diamond watches and Ferraris. \n",
+      "\n",
+      "Welcome to the world of boxing promotion, circa 2015. \n",
+      "\n",
+      "American Floyd Mayweather and Filipino Manny Pacquiao are set to officially announce their heavily anticipated boxing match at a press conference in Los Angeles Wednesday. \n",
+      "\n",
+      "With the combined purse for the May 2 bout in Las Vegas reported to touch $300 million pending viewership numbers, the incentives to self-promote could not be higher. \n",
+      "\n",
+      "\"Nowadays you have to be on social media to launch the fight and to build hype,\" says boxing promoter Nisse Sauerland, CEO of Team Sauerland. \"It couldn't be done without it.\" \n",
+      "\n",
+      "Thirty-eight year old Mayweather (47-0, 26 knockouts), who favors the moniker \"The Money Man\" or \"TBE\" (The Best Ever), boasts nearly five million Instagram followers, 5.65 million followers on Twitter and 9.2 million Facebook likes. \n",
+      "\n",
+      "He famously confirmed the fight via Shots, a photo sharing social media application that he's invested in, and displays links to his clothing brand, The Money Team, on all his accounts. \n",
+      "\n",
+      "Along with professing to the be the best fighter of all time, he could also stake a claim to be one of the greatest social media users in sports. \n",
+      "\n",
+      "\"I think they're both playing their roles,\" says Sauerland, who promotes over 45 boxers. \"You've got the bad guy and the good guy, really. You've got the guy who throws the money around (Mayweather), that's his image, and Pacquiao, he's the hope of a nation.\" \n",
+      "\n",
+      "Question:\n",
+      "Who are the two boxer featured in this article?\n",
+      "\n",
+      "Answer:\n",
+      "Floyd Mayweather and Manny Pacquiao\n"
+     ]
+    }
+   ],
+   "source": [
+    "import duckdb\n",
+    "\n",
+    "# DuckDB has an easy wrapper for loading datasets from Hugging Face.\n",
+    "con = duckdb.connect(\":memory:\")\n",
+    "full_result = con.query(\"\"\"\n",
+    "    SELECT * FROM 'hf://datasets/stanfordnlp/coqa/data/validation-00000-of-00001.parquet'\n",
+    "        LIMIT 40\n",
+    "\"\"\").fetchall()\n",
+    "\n",
+    "single_result = full_result[10]\n",
+    "\n",
+    "print(\"Passage:\")\n",
+    "print(single_result[1])\n",
+    "\n",
+    "print(\"\\nQuestion:\")\n",
+    "print(single_result[2][0])\n",
+    "\n",
+    "print(\"\\nAnswer:\")\n",
+    "print(single_result[3][\"input_text\"][0])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The data contains a series of passages, each with a number of questions and answers. Let's flatten this into a list of `(passage, question, answer)` tuples.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "629\n"
+     ]
+    }
+   ],
+   "source": [
+    "from dataclasses import dataclass\n",
+    "\n",
+    "\n",
+    "@dataclass\n",
+    "class QuestionAnswer:\n",
+    "    passage: str\n",
+    "    question: str\n",
+    "    expected_answer: str\n",
+    "    generated_answer: str\n",
+    "\n",
+    "\n",
+    "qa_pairs = [\n",
+    "    QuestionAnswer(\n",
+    "        passage=r[1],\n",
+    "        question=question,\n",
+    "        generated_answer=r[3][\"input_text\"][i],\n",
+    "        expected_answer=r[3][\"input_text\"][i],\n",
+    "    )\n",
+    "    for r in full_result\n",
+    "    for (i, question) in enumerate(r[2])\n",
+    "]\n",
+    "\n",
+    "print(len(qa_pairs))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Adding hallucinations\n",
+    "\n",
+    "Because Braintrust's scorer is designed to test hallucinations, we can use the QA pairs to generate known hallucinations. We'll create hallucinated answers by asking an\n",
+    "LLM to confidently generate an answer to each question without using the passage.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Passage:\n",
+      "Once upon a time, in a barn near a farm house, there lived a little white kitten named Cotton. Cotton lived high up in a nice warm place above the barn where all of the farmer's horses slept. But Cotton wasn't alone in her little home above the barn, oh no. She shared her hay bed with her mommy and 5 other sisters. All of her sisters were cute and fluffy, like Cotton. But she was the only white one in the bunch. The rest of her sisters were all orange with beautiful white tiger stripes like Cotton's mommy. Being different made Cotton quite sad. She often wished she looked like the rest of her family. So one day, when Cotton found a can of the old farmer's orange paint, she used it to paint herself like them. When her mommy and sisters found her they started laughing. \n",
+      "\n",
+      "\"What are you doing, Cotton?!\" \n",
+      "\n",
+      "\"I only wanted to be more like you\". \n",
+      "\n",
+      "Cotton's mommy rubbed her face on Cotton's and said \"Oh Cotton, but your fur is so pretty and special, like you. We would never want you to be any other way\". And with that, Cotton's mommy picked her up and dropped her into a big bucket of water. When Cotton came out she was herself again. Her sisters licked her face until Cotton's fur was all all dry. \n",
+      "\n",
+      "\"Don't ever do that again, Cotton!\" they all cried. \"Next time you might mess up that pretty white fur of yours and we wouldn't want that!\" \n",
+      "\n",
+      "Then Cotton thought, \"I change my mind. I like being special\".\n",
+      "\n",
+      "Question:\n",
+      "Where did she live?\n",
+      "\n",
+      "Expected Answer:\n",
+      "in a barn\n",
+      "\n",
+      "Generated Answer:\n",
+      "She lived in a quaint cottage on the edge of the Misty Hollow Forest, where elves and talking owls often hosted moonlit storytelling festivals.\n",
+      "\n",
+      "\n",
+      "Number of hallucinations: 270\n"
+     ]
+    }
+   ],
+   "source": [
+    "import asyncio\n",
+    "import random\n",
+    "\n",
+    "random.seed(42)\n",
+    "\n",
+    "\n",
+    "async def hallucinate_answer(qa):\n",
+    "    response = await client.chat.completions.create(\n",
+    "        model=\"gpt-4o\",\n",
+    "        messages=[\n",
+    "            {\n",
+    "                \"role\": \"system\",\n",
+    "                \"content\": \"\"\"\\\n",
+    "You are a helpful hallucinating assistant, who makes up fake answers to questions.\n",
+    "\n",
+    "Answer the following question in 1 sentence. If you know the answer, then make up some fake\n",
+    "superfluous details that are not in the passage you have memorized.\n",
+    "\n",
+    "Make sure to always answer it confidently, even if you don't know the answer. Do not use words\n",
+    "like \"perhaps\", \"likely\", \"maybe\", etc. or punctuation like \"...\".Do not admit that you cannot\n",
+    "or do not know the answer.\"\"\",\n",
+    "            },\n",
+    "            {\"role\": \"user\", \"content\": qa.question},\n",
+    "        ],\n",
+    "        temperature=1,\n",
+    "        max_tokens=100,\n",
+    "    )\n",
+    "    return response.choices[0].message.content\n",
+    "\n",
+    "\n",
+    "hallucinated_answers = await asyncio.gather(\n",
+    "    *[hallucinate_answer(qa) for qa in qa_pairs]\n",
+    ")\n",
+    "\n",
+    "\n",
+    "hallucinations = [\n",
+    "    QuestionAnswer(\n",
+    "        passage=qa.passage,\n",
+    "        question=qa.question,\n",
+    "        expected_answer=qa.expected_answer,\n",
+    "        generated_answer=hallucination,\n",
+    "    )\n",
+    "    for (qa, hallucination) in zip(qa_pairs, hallucinated_answers)\n",
+    "    # Exclude simple yes/no answers.\n",
+    "    if \"yes\" not in hallucination.lower() and \"no\" not in hallucination.lower()\n",
+    "]\n",
+    "\n",
+    "print(\"Passage:\")\n",
+    "print(hallucinations[0].passage)\n",
+    "print(\"\\nQuestion:\")\n",
+    "print(hallucinations[0].question)\n",
+    "print(\"\\nExpected Answer:\")\n",
+    "print(hallucinations[0].expected_answer)\n",
+    "print(\"\\nGenerated Answer:\")\n",
+    "print(hallucinations[0].generated_answer)\n",
+    "\n",
+    "print(\"\\n\\nNumber of hallucinations:\", len(hallucinations))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Creating the evaluators\n",
+    "\n",
+    "We'll consider a few popular approaches for creating an LLM-as-a-judge. For each approach, we'll create a scorer and then \"meta-evaluate\" it to see how it performs.\n",
+    "Since we know that the hallucinated answers are incorrect, we'll assess the quality of an evaluator by testing how often it scores the hallucinated answers as `0`.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### LLM-as-a-judge #1: Numeric rater\n",
+    "\n",
+    "A common initial intuition when creating an LLM-as-a-judge is asking the LLM to rate the answer on a scale of 1 to 5. The benefit of this approach is that\n",
+    "it's easy to convert the LLM's output into a numeric score.\n",
+    "\n",
+    "We'll use a modified version of the [Factuality](https://github.com/braintrustdata/autoevals/blob/main/templates/factuality.yaml) template, but ask the LLM to\n",
+    "rate the answer on a scale of 1 to 10.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "What did the other cats do when Cotton emerged from the bucket of water? On a correct answer: licked her face\n",
+      "1.0\n",
+      "What? On a hallucinated answer: \"What\" is a word often used to express inquiry, curiosity, or surprise, and it is said to have originated from the ancient city of Whatopia, where people would constantly ask questions while enchanted crows delivered cryptic messages.\n",
+      "0.0\n"
+     ]
+    }
+   ],
+   "source": [
+    "import json\n",
+    "\n",
+    "PROMPT = \"\"\"\\\n",
+    "You are comparing a submitted answer to an expert answer on a given question. Here is the data:\n",
+    "[BEGIN DATA]\n",
+    "************\n",
+    "[Question]: {input}\n",
+    "************\n",
+    "[Expert]: {expected}\n",
+    "************\n",
+    "[Submission]: {output}\n",
+    "************\n",
+    "[END DATA]\n",
+    "\n",
+    "Compare the factual content of the submitted answer with the expert answer. Ignore any differences in style, grammar, or punctuation.\n",
+    "Rate the submission on a scale of 1 to 10.\n",
+    "\"\"\"\n",
+    "\n",
+    "\n",
+    "@braintrust.traced\n",
+    "async def numeric_rater(input, output, expected):\n",
+    "    response = await client.chat.completions.create(\n",
+    "        model=\"gpt-4o\",\n",
+    "        messages=[\n",
+    "            {\n",
+    "                \"role\": \"user\",\n",
+    "                \"content\": PROMPT.format(input=input, output=output, expected=expected),\n",
+    "            }\n",
+    "        ],\n",
+    "        temperature=0,\n",
+    "        tools=[\n",
+    "            {\n",
+    "                \"type\": \"function\",\n",
+    "                \"function\": {\n",
+    "                    \"name\": \"rate\",\n",
+    "                    \"description\": \"Rate the submission on a scale of 1 to 10.\",\n",
+    "                    \"parameters\": {\n",
+    "                        \"type\": \"object\",\n",
+    "                        \"properties\": {\n",
+    "                            \"rating\": {\"type\": \"integer\", \"minimum\": 1, \"maximum\": 10},\n",
+    "                        },\n",
+    "                        \"required\": [\"rating\"],\n",
+    "                    },\n",
+    "                },\n",
+    "            }\n",
+    "        ],\n",
+    "        tool_choice={\"type\": \"function\", \"function\": {\"name\": \"rate\"}},\n",
+    "    )\n",
+    "    arguments = json.loads(response.choices[0].message.tool_calls[0].function.arguments)\n",
+    "    return (arguments[\"rating\"] - 1) / 9\n",
+    "\n",
+    "\n",
+    "print(qa_pairs[10].question, \"On a correct answer:\", qa_pairs[10].generated_answer)\n",
+    "print(\n",
+    "    await numeric_rater(\n",
+    "        qa_pairs[10].question,\n",
+    "        qa_pairs[10].generated_answer,\n",
+    "        qa_pairs[10].expected_answer,\n",
+    "    )\n",
+    ")\n",
+    "\n",
+    "print(\n",
+    "    hallucinations[10].question,\n",
+    "    \"On a hallucinated answer:\",\n",
+    "    hallucinations[10].generated_answer,\n",
+    ")\n",
+    "print(\n",
+    "    await numeric_rater(\n",
+    "        hallucinations[10].question,\n",
+    "        hallucinations[10].generated_answer,\n",
+    "        hallucinations[10].expected_answer,\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This looks promising! Now that we have sanity checked it on a single example, let's run a proper evaluation and see how it performs on a wider set of data. An evaluation consists of three components:\n",
+    "\n",
+    "- **Data**: In this case, the `input` is the question, hallucinated answer, and ground truth answer. The scorer will convert this into a score between 0 and 1. The expected score is 0, since it's a hallucination.\n",
+    "- **Task**: The task is simply calling the numeric rater for each input.\n",
+    "- **Scores**: We'll assess the quality of the generated score by comparing it with the ground truth score. Since we know both numbers are between 0 and 1, we can use the normalized difference as the score.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Experiment Numeric rater is running at https://www.braintrust.dev/app/braintrustdata.com/p/LLM-as-a-judge/experiments/Numeric%20rater\n",
+      "LLM-as-a-judge [experiment_name=Numeric rater] (data): 270it [00:00, 54634.41it/s]\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "8eeb99e0ae3f46ea84a7f6ee41ee0928",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "LLM-as-a-judge [experiment_name=Numeric rater] (tasks):   0%|          | 0/270 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "=========================SUMMARY=========================\n",
+      "95.35% 'normalized_diff' score\n",
+      "\n",
+      "201.60tok prompt_tokens\n",
+      "5tok completion_tokens\n",
+      "206.60tok total_tokens\n",
+      "\n",
+      "See results for Numeric rater at https://www.braintrust.dev/app/braintrustdata.com/p/LLM-as-a-judge/experiments/Numeric%20rater\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "EvalResultWithSummary(summary=\"...\", results=[...])"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from dataclasses import asdict\n",
+    "\n",
+    "from braintrust import Eval\n",
+    "\n",
+    "\n",
+    "def data():\n",
+    "    for pair in hallucinations:\n",
+    "        yield dict(\n",
+    "            input=dict(asdict(pair)), expected=0, metadata=dict(hallucination=True)\n",
+    "        )\n",
+    "\n",
+    "\n",
+    "async def task(input):\n",
+    "    return await numeric_rater(\n",
+    "        input=input[\"question\"],\n",
+    "        output=input[\"generated_answer\"],\n",
+    "        expected=input[\"expected_answer\"],\n",
+    "    )\n",
+    "\n",
+    "\n",
+    "def normalized_diff(output, expected):\n",
+    "    return 1 - abs(output - expected)\n",
+    "\n",
+    "\n",
+    "await Eval(\n",
+    "    \"LLM-as-a-judge\",\n",
+    "    data=data,\n",
+    "    task=task,\n",
+    "    scores=[normalized_diff],\n",
+    "    experiment_name=\"Numeric rater\",\n",
+    "    max_concurrency=10,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "It looks like the numeric rater scored almost 94% in total. That's not bad, but if 6% of your evals are incorrectly judged, that could make it very hard to trust them. Let's dig into the Braintrust\n",
+    "UI to get some insight into what's going on.\n",
+    "\n",
+    "![Partial credit](../images/Custom-LLM-as-a-Judge-Partial-Credit.gif)\n",
+    "\n",
+    "It looks like a number of the incorrect answers were scored with numbers between 1 and 10. However, we do not currently have any insight into why the model gave these scores. Let's see if we can\n",
+    "fix that next.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### LLM-as-a-judge #2: Adding reasoning\n",
+    "\n",
+    "Let's tweak the prompt to get the LLM to also reason about its rating. This method is called [Chain of Thought Reasoning](https://en.wikipedia.org/wiki/Chain_of_thought_reasoning). In addition\n",
+    "to potentially improving the score, it will give us some insight into why the model gave these scores.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "What did the other cats do when Cotton emerged from the bucket of water? On a correct answer: licked her face\n",
+      "1.0\n",
+      "What? On a hallucinated answer: \"What\" is a word often used to express inquiry, curiosity, or surprise, and it is said to have originated from the ancient city of Whatopia, where people would constantly ask questions while enchanted crows delivered cryptic messages.\n",
+      "0.0\n"
+     ]
+    }
+   ],
+   "source": [
+    "@braintrust.traced\n",
+    "async def numeric_rater(input, output, expected):\n",
+    "    response = await client.chat.completions.create(\n",
+    "        model=\"gpt-4o\",\n",
+    "        messages=[\n",
+    "            {\n",
+    "                \"role\": \"user\",\n",
+    "                \"content\": PROMPT.format(input=input, output=output, expected=expected),\n",
+    "            }\n",
+    "        ],\n",
+    "        temperature=0,\n",
+    "        tools=[\n",
+    "            {\n",
+    "                \"type\": \"function\",\n",
+    "                \"function\": {\n",
+    "                    \"name\": \"rate\",\n",
+    "                    \"description\": \"Rate the submission on a scale of 1 to 10.\",\n",
+    "                    \"parameters\": {\n",
+    "                        \"type\": \"object\",\n",
+    "                        \"properties\": {\n",
+    "                            \"reasons\": {\n",
+    "                                \"description\": \"Write out in a step by step manner your reasoning to be sure that your conclusion is correct. Avoid simply stating the correct answer at the outset.\",\n",
+    "                                \"title\": \"Reasoning\",\n",
+    "                                \"type\": \"string\",\n",
+    "                            },\n",
+    "                            \"rating\": {\"type\": \"integer\", \"minimum\": 1, \"maximum\": 10},\n",
+    "                        },\n",
+    "                        \"required\": [\"rating\"],\n",
+    "                    },\n",
+    "                },\n",
+    "            }\n",
+    "        ],\n",
+    "        tool_choice={\"type\": \"function\", \"function\": {\"name\": \"rate\"}},\n",
+    "    )\n",
+    "    arguments = json.loads(response.choices[0].message.tool_calls[0].function.arguments)\n",
+    "    return (arguments[\"rating\"] - 1) / 9\n",
+    "\n",
+    "\n",
+    "print(qa_pairs[10].question, \"On a correct answer:\", qa_pairs[10].generated_answer)\n",
+    "print(\n",
+    "    await numeric_rater(\n",
+    "        qa_pairs[10].question,\n",
+    "        qa_pairs[10].generated_answer,\n",
+    "        qa_pairs[10].expected_answer,\n",
+    "    )\n",
+    ")\n",
+    "\n",
+    "print(\n",
+    "    hallucinations[10].question,\n",
+    "    \"On a hallucinated answer:\",\n",
+    "    hallucinations[10].generated_answer,\n",
+    ")\n",
+    "print(\n",
+    "    await numeric_rater(\n",
+    "        hallucinations[10].question,\n",
+    "        hallucinations[10].generated_answer,\n",
+    "        hallucinations[10].expected_answer,\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Experiment Numeric rater with reasoning is running at https://www.braintrust.dev/app/braintrustdata.com/p/LLM-as-a-judge/experiments/Numeric%20rater%20with%20reasoning\n",
+      "LLM-as-a-judge [experiment_name=Numeric rater with reasoning] (data): 270it [00:00, 111715.70it/s]\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "1623ec8d55524e569700616c240818e9",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "LLM-as-a-judge [experiment_name=Numeric rater with reasoning] (tasks):   0%|          | 0/270 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "=========================SUMMARY=========================\n",
+      "Numeric rater with reasoning compared to Numeric rater:\n",
+      "92.10% (-03.25%) 'normalized_diff' score\t(5 improvements, 63 regressions)\n",
+      "\n",
+      "3.68s duration\n",
+      "3.68s llm_duration\n",
+      "239.60tok (+3800.00%) 'prompt_tokens'    \t(0 improvements, 270 regressions)\n",
+      "136.82tok (+13182.22%) 'completion_tokens'\t(0 improvements, 270 regressions)\n",
+      "376.43tok (+16982.22%) 'total_tokens'     \t(0 improvements, 270 regressions)\n",
+      "0.00$ estimated_cost\n",
+      "\n",
+      "See results for Numeric rater with reasoning at https://www.braintrust.dev/app/braintrustdata.com/p/LLM-as-a-judge/experiments/Numeric%20rater%20with%20reasoning\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "EvalResultWithSummary(summary=\"...\", results=[...])"
+      ]
+     },
+     "execution_count": 10,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "await Eval(\n",
+    "    \"LLM-as-a-judge\",\n",
+    "    data=data,\n",
+    "    task=task,\n",
+    "    scores=[normalized_diff],\n",
+    "    experiment_name=\"Numeric rater with reasoning\",\n",
+    "    max_concurrency=10,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "It doesn't look like adding reasoning helped the score (in fact, it's 3% percent worse). However, if we look at one of the failures, we'll get some insight into\n",
+    "what the model was thinking. Here is an example of a hallucinated answer:\n",
+    "\n",
+    "![Output](../images/Custom-LLM-as-a-Judge-Output.png)\n",
+    "\n",
+    "And the score along with its reasoning:\n",
+    "\n",
+    "![Reasoning](../images/Custom-LLM-as-a-Judge-Reasoning.png)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "It looks like the model is applying its own judgement to compute partial credit. This is a common problem with numeric rating—both for models and for humans—and can often be solved\n",
+    "by using better prompting.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### LLM-as-a-judge #3: Classifying instead of rating\n",
+    "\n",
+    "Next, we'll spell out specific criteria and ask the model to classify the answer according to those criteria. This method allows us to more precisely guide the model\n",
+    "towards the hallucinations we're testing for. Intuitively, giving the model specific criteria to rate will result in a more accurate score.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "What did the other cats do when Cotton emerged from the bucket of water? On a correct answer: licked her face\n",
+      "1\n",
+      "What? On a hallucinated answer: \"What\" is a word often used to express inquiry, curiosity, or surprise, and it is said to have originated from the ancient city of Whatopia, where people would constantly ask questions while enchanted crows delivered cryptic messages.\n",
+      "0\n"
+     ]
+    }
+   ],
+   "source": [
+    "PROMPT = \"\"\"\\\n",
+    "You are comparing a submitted answer to an expert answer on a given question. Here is the data:\n",
+    "[BEGIN DATA]\n",
+    "************\n",
+    "[Question]: {input}\n",
+    "************\n",
+    "[Expert]: {expected}\n",
+    "************\n",
+    "[Submission]: {output}\n",
+    "************\n",
+    "[END DATA]\n",
+    "\n",
+    "Compare the factual content of the submitted answer with the expert answer. Ignore any differences in style, grammar, or punctuation.\n",
+    "The submitted answer may either be a subset or superset of the expert answer, or it may conflict with it. Determine which case applies. Answer the question by selecting one of the following options:\n",
+    "(A) The submitted answer is a subset of the expert answer and is fully consistent with it.\n",
+    "(B) The submitted answer is a superset of the expert answer and is fully consistent with it.\n",
+    "(C) The submitted answer contains all the same details as the expert answer.\n",
+    "(D) There is a disagreement between the submitted answer and the expert answer.\n",
+    "(E) The answers differ, but these differences don't matter from the perspective of factuality.\n",
+    "\n",
+    "Answer the question by calling `select_choice` with your reasoning in a step-by-step matter to be\n",
+    "sure that your conclusion is correct. Avoid simply stating the correct answer at the outset. Select a\n",
+    "single choice by setting the `choice` parameter to a single choice from A, B, C, D, or E.\n",
+    "\"\"\"\n",
+    "\n",
+    "# Since we're testing for hallucinations, penalize (B) as much as (D).\n",
+    "CHOICE_SCORES = {\n",
+    "    \"A\": 0.5,\n",
+    "    \"B\": 0,\n",
+    "    \"C\": 1,\n",
+    "    \"D\": 0,\n",
+    "    \"E\": 1,\n",
+    "}\n",
+    "\n",
+    "\n",
+    "@braintrust.traced\n",
+    "async def classifier(input, output, expected):\n",
+    "    response = await client.chat.completions.create(\n",
+    "        model=\"gpt-4o\",\n",
+    "        messages=[\n",
+    "            {\n",
+    "                \"role\": \"user\",\n",
+    "                \"content\": PROMPT.format(input=input, output=output, expected=expected),\n",
+    "            }\n",
+    "        ],\n",
+    "        temperature=0,\n",
+    "        tools=[\n",
+    "            {\n",
+    "                \"type\": \"function\",\n",
+    "                \"function\": {\n",
+    "                    \"name\": \"rate\",\n",
+    "                    \"description\": \"Call this function to select a choice.\",\n",
+    "                    \"parameters\": {\n",
+    "                        \"properties\": {\n",
+    "                            \"reasons\": {\n",
+    "                                \"description\": \"Write out in a step by step manner your reasoning to be sure that your conclusion is correct. Avoid simply stating the correct answer at the outset.\",\n",
+    "                                \"type\": \"string\",\n",
+    "                            },\n",
+    "                            \"choice\": {\n",
+    "                                \"description\": \"The choice\",\n",
+    "                                \"type\": \"string\",\n",
+    "                                \"enum\": [\"A\", \"B\", \"C\", \"D\", \"E\"],\n",
+    "                            },\n",
+    "                        },\n",
+    "                        \"required\": [\"reasons\", \"choice\"],\n",
+    "                        \"type\": \"object\",\n",
+    "                    },\n",
+    "                },\n",
+    "            }\n",
+    "        ],\n",
+    "        tool_choice={\"type\": \"function\", \"function\": {\"name\": \"rate\"}},\n",
+    "    )\n",
+    "    arguments = json.loads(response.choices[0].message.tool_calls[0].function.arguments)\n",
+    "    choice = arguments[\"choice\"]\n",
+    "    return CHOICE_SCORES[choice] if choice in CHOICE_SCORES else None\n",
+    "\n",
+    "\n",
+    "print(qa_pairs[10].question, \"On a correct answer:\", qa_pairs[10].generated_answer)\n",
+    "print(\n",
+    "    await classifier(\n",
+    "        qa_pairs[10].question,\n",
+    "        qa_pairs[10].generated_answer,\n",
+    "        qa_pairs[10].expected_answer,\n",
+    "    )\n",
+    ")\n",
+    "\n",
+    "print(\n",
+    "    hallucinations[10].question,\n",
+    "    \"On a hallucinated answer:\",\n",
+    "    hallucinations[10].generated_answer,\n",
+    ")\n",
+    "print(\n",
+    "    await classifier(\n",
+    "        hallucinations[10].question,\n",
+    "        hallucinations[10].generated_answer,\n",
+    "        hallucinations[10].expected_answer,\n",
+    "    )\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Experiment Classifier is running at https://www.braintrust.dev/app/braintrustdata.com/p/LLM-as-a-judge/experiments/Classifier\n",
+      "LLM-as-a-judge [experiment_name=Classifier] (data): 270it [00:00, 84930.41it/s]\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "fdb4cff5ff7646d59410ab7ae42b838b",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "LLM-as-a-judge [experiment_name=Classifier] (tasks):   0%|          | 0/270 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "=========================SUMMARY=========================\n",
+      "Classifier compared to Numeric rater with reasoning:\n",
+      "98.15% (+06.05%) 'normalized_diff' score\t(86 improvements, 5 regressions)\n",
+      "\n",
+      "4.41s (+72.60%) 'duration'         \t(104 improvements, 165 regressions)\n",
+      "4.40s (+72.59%) 'llm_duration'     \t(104 improvements, 165 regressions)\n",
+      "418.60tok (+17900.00%) 'prompt_tokens'    \t(0 improvements, 270 regressions)\n",
+      "164.91tok (+2809.26%) 'completion_tokens'\t(64 improvements, 204 regressions)\n",
+      "583.52tok (+20709.26%) 'total_tokens'     \t(0 improvements, 270 regressions)\n",
+      "0.00$ (+00.07%) 'estimated_cost'   \t(8 improvements, 255 regressions)\n",
+      "\n",
+      "See results for Classifier at https://www.braintrust.dev/app/braintrustdata.com/p/LLM-as-a-judge/experiments/Classifier\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "EvalResultWithSummary(summary=\"...\", results=[...])"
+      ]
+     },
+     "execution_count": 12,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "async def task(input):\n",
+    "    return await classifier(\n",
+    "        input=input[\"question\"],\n",
+    "        output=input[\"generated_answer\"],\n",
+    "        expected=input[\"expected_answer\"],\n",
+    "    )\n",
+    "\n",
+    "\n",
+    "await Eval(\n",
+    "    \"LLM-as-a-judge\",\n",
+    "    data=data,\n",
+    "    task=task,\n",
+    "    scores=[normalized_diff],\n",
+    "    experiment_name=\"Classifier\",\n",
+    "    max_concurrency=10,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The classifier scored 98% which is a significant improvement!\n",
+    "\n",
+    "### Codifying this pattern\n",
+    "\n",
+    "The classifier above can simply be rewritten as:\n",
+    "\n",
+    "```python\n",
+    "PROMPT = \"\"\"\\\n",
+    "You are comparing a submitted answer to an expert answer on a given question. Here is the data:\n",
+    "[BEGIN DATA]\n",
+    "************\n",
+    "[Question]: {{input}}\n",
+    "************\n",
+    "[Expert]: {{expected}}\n",
+    "************\n",
+    "[Submission]: {{output}}\n",
+    "************\n",
+    "[END DATA]\n",
+    "\n",
+    "Compare the factual content of the submitted answer with the expert answer. Ignore any differences in style, grammar, or punctuation.\n",
+    "The submitted answer may either be a subset or superset of the expert answer, or it may conflict with it. Determine which case applies. Answer the question by selecting one of the following options:\n",
+    "(A) The submitted answer is a subset of the expert answer and is fully consistent with it.\n",
+    "(B) The submitted answer is a superset of the expert answer and is fully consistent with it.\n",
+    "(C) The submitted answer contains all the same details as the expert answer.\n",
+    "(D) There is a disagreement between the submitted answer and the expert answer.\n",
+    "(E) The answers differ, but these differences don't matter from the perspective of factuality.\n",
+    "\n",
+    "Answer the question by calling `select_choice` with your reasoning in a step-by-step matter to be\n",
+    "sure that your conclusion is correct. Avoid simply stating the correct answer at the outset. Select a\n",
+    "single choice by setting the `choice` parameter to a single choice from A, B, C, D, or E.\n",
+    "\"\"\"\n",
+    "\n",
+    "Classifier = autoevals.LLMClassifier(\n",
+    "    name=\"Hallucination detector\",\n",
+    "    prompt_template=PROMPT,\n",
+    "    choice_scores={\"A\": 0.5, \"B\": 0, \"C\": 1, \"D\": 0, \"E\": 1},\n",
+    "    use_cot=True,\n",
+    ")\n",
+    "```\n",
+    "\n",
+    "## Next steps\n",
+    "\n",
+    "As a next step, you could dig into the individual improvements and regressions to assess them and consider future improvements to the prompt. You could also test it on your own data, and double check that the results hold for your use case.\n",
+    "You could also measure a model like o1, try fine-tuning a smaller model and see if the results are reproducible, or use few-shot prompting to align the model with more subjective criteria.\n",
+    "In all cases, you should strive to evaluate your results, so you can rigorously assess the impact of each change.\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": ".venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.4"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/Data-intensive-Realtime-apps.ipynb

@@ -0,0 +1,624 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Practical guide to data-intensive apps with the Realtime API\n",
+    "\n",
+    "This cookbook serves as a practical guide to help AI Engineers maximize the effectiveness of OpenAI's Realtime API, specifically when dealing with data-intensive function calls. We'll focus on scenarios common in speech-to-speech agents, where vast amounts of data must be handled smoothly and efficiently.\n",
+    "\n",
+    "This post won't cover the basics of setting up a Realtime API solution. Instead, you'll gain clear insights and actionable strategies to enhance the performance and reliability of your real-time conversational agents. It addresses specific challenges unique to handling large amounts of data in real-time conversational contexts."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### What is the Realtime API?\n",
+    "\n",
+    "Before we dive in, let’s quickly recap the API for those who are new. The OpenAI Realtime API is a recent offering that supports low-latency, multimodal interactions—such as speech-to-speech conversations and live transcription. Picture scenarios like real-time voice-based customer support or live movie transcriptions. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### What is a data-intensive function call?\n",
+    "\n",
+    "Agents need access to tools and relevant data to perform their tasks. For instance, a financial analyst agent might pull real-time market data. In many cases, services already exist in your environment that expose this information through APIs.\n",
+    "\n",
+    "Historically, APIs weren’t designed with agents in mind and often return large volumes of data, depending on the service. As engineers, we frequently wrap these APIs with function calls to accelerate agent development—which makes perfect sense. Why reinvent what already exists?\n",
+    "\n",
+    "If not carefully optimized, these data-intensive function calls can quickly overwhelm the Realtime API—leading to slow responses or even failures to process user requests."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Setting the stage\n",
+    "\n",
+    "Our example centers on an NBA Scouting Agent that calls multiple functions to deliver in-depth analysis of upcoming draft prospects. To demonstrate practical guidelines for Realtime API interactions, we use large, realistic payloads inspired by NBA draft prospects. Below, you’ll find a monolithic `searchDraftProspects` function defined in the Realtime session to set the stage."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "```json\n",
+    "// \"Hey, pull up point guards projected in the top 10 in the 2025 draft\"\n",
+    "{\n",
+    "  \"type\": \"session.update\",\n",
+    "  \"session\": {\n",
+    "    \"tools\": [\n",
+    "      {\n",
+    "        \"type\": \"function\",\n",
+    "        \"name\": \"searchDraftProspects\",\n",
+    "        \"description\": \"Search draft prospects for a given year e.g., Point Guard\",\n",
+    "        \"parameters\": {\n",
+    "          \"type\": \"object\",\n",
+    "          \"properties\": {\n",
+    "            \"sign\": {\n",
+    "              \"type\": \"string\",\n",
+    "              \"description\": \"The player position\",\n",
+    "              \"enum\": [\n",
+    "                \"Point Guard\",\n",
+    "                \"Shooting Guard\",\n",
+    "                \"Small Forward\",\n",
+    "                \"Power Forward\",\n",
+    "                \"Center\",\n",
+    "                \"Any\"\n",
+    "              ]\n",
+    "            },\n",
+    "            year: { type: \"number\", description: \"Draft year e.g., 2025\" },\n",
+    "            mockDraftRanking: { type: \"number\", description: \"Predicted Draft Ranking\" },\n",
+    "          },\n",
+    "          \"required\": [\"position\", \"year\"]\n",
+    "        }\n",
+    "      }\n",
+    "    ],\n",
+    "    \"tool_choice\": \"auto\",\n",
+    "  }\n",
+    "}\n",
+    "```\n",
+    "\n",
+    "The searchDraftProspects function call returns a hefty payload. The example’s structure and size are drawn from real-world scenarios we’ve encountered.\n",
+    "\n",
+    "```json\n",
+    "// Example Payload\n",
+    "{\n",
+    "  \"status\": {\n",
+    "    \"code\": 200,\n",
+    "    \"message\": \"SUCCESS\"\n",
+    "  },\n",
+    "  \"found\": 4274,\n",
+    "  \"offset\": 0,\n",
+    "  \"limit\": 10,\n",
+    "  \"data\": [\n",
+    "    {\n",
+    "      \"prospectId\": 10001,\n",
+    "      \"data\": {\n",
+    "        \"ProspectInfo\": {\n",
+    "          \"league\": \"NCAA\",\n",
+    "          \"collegeId\": 301,\n",
+    "          \"isDraftEligible\": true,\n",
+    "          \"Player\": {\n",
+    "            \"personalDetails\": {\n",
+    "              \"firstName\": \"Jalen\",\n",
+    "              \"lastName\": \"Storm\",\n",
+    "              \"dateOfBirth\": \"2003-01-15\",\n",
+    "              \"nationality\": \"USA\"\n",
+    "            },\n",
+    "            \"physicalAttributes\": {\n",
+    "              \"position\": \"PG\",\n",
+    "              \"height\": {\n",
+    "                \"feet\": 6,\n",
+    "                \"inches\": 4\n",
+    "              },\n",
+    "              \"weightPounds\": 205\n",
+    "            },\n",
+    "            \"hometown\": {\n",
+    "              \"city\": \"Springfield\",\n",
+    "              \"state\": \"IL\"\n",
+    "            }\n",
+    "          },\n",
+    "          \"TeamInfo\": {\n",
+    "            \"collegeTeam\": \"Springfield Tigers\",\n",
+    "            \"conference\": \"Big West\",\n",
+    "            \"teamRanking\": 12,\n",
+    "            \"coach\": {\n",
+    "              \"coachId\": 987,\n",
+    "              \"coachName\": \"Marcus Reed\",\n",
+    "              \"experienceYears\": 10\n",
+    "            }\n",
+    "          }\n",
+    "        },\n",
+    "        \"Stats\": {\n",
+    "          \"season\": \"2025\",\n",
+    "          \"gamesPlayed\": 32,\n",
+    "          \"minutesPerGame\": 34.5,\n",
+    "          \"shooting\": {\n",
+    "            \"FieldGoalPercentage\": 47.2,\n",
+    "            \"ThreePointPercentage\": 39.1,\n",
+    "            \"FreeThrowPercentage\": 85.6\n",
+    "          },\n",
+    "          \"averages\": {\n",
+    "            \"points\": 21.3,\n",
+    "            \"rebounds\": 4.1,\n",
+    "            \"assists\": 6.8,\n",
+    "            \"steals\": 1.7,\n",
+    "            \"blocks\": 0.3\n",
+    "          }\n",
+    "        },\n",
+    "        \"Scouting\": {\n",
+    "          \"evaluations\": {\n",
+    "            \"strengths\": [\"Court vision\", \"Clutch shooting\"],\n",
+    "            \"areasForImprovement\": [\"Defensive consistency\"]\n",
+    "          },\n",
+    "          \"scouts\": [\n",
+    "            {\n",
+    "              \"scoutId\": 501,\n",
+    "              \"name\": \"Greg Hamilton\",\n",
+    "              \"organization\": \"National Scouting Bureau\"\n",
+    "            }\n",
+    "          ]\n",
+    "        },\n",
+    "        \"DraftProjection\": {\n",
+    "          \"mockDraftRanking\": 5,\n",
+    "          \"lotteryPickProbability\": 88,\n",
+    "          \"historicalComparisons\": [\n",
+    "            {\n",
+    "              \"player\": \"Chris Paul\",\n",
+    "              \"similarityPercentage\": 85\n",
+    "            }\n",
+    "          ]\n",
+    "        },\n",
+    "        \"Media\": {\n",
+    "          \"highlightReelUrl\": \"https://example.com/highlights/jalen-storm\",\n",
+    "          \"socialMedia\": {\n",
+    "            \"twitter\": \"@jstorm23\",\n",
+    "            \"instagram\": \"@jstorm23_ig\"\n",
+    "          }\n",
+    "        },\n",
+    "        \"Agent\": {\n",
+    "          \"agentName\": \"Rick Allen\",\n",
+    "          \"agency\": \"Elite Sports Management\",\n",
+    "          \"contact\": {\n",
+    "            \"email\": \"rallen@elitesports.com\",\n",
+    "            \"phone\": \"555-123-4567\"\n",
+    "          }\n",
+    "        }\n",
+    "      }\n",
+    "    },\n",
+    "    // ... Many thousands of tokens later.\n",
+    "  ]\n",
+    "}\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Guiding principles"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 1. Break down unwieldy functions into smaller ones with clear roles and responsibilities\n",
+    "\n",
+    "It almost goes without saying—when building function calls, your top priority is to design clear, well-defined functions. This makes it easy to trim response sizes and avoid overwhelming the model. Each function call should be straightforward to explain, sharply scoped, and return only the information needed for its purpose. Overlapping responsibilities between functions inevitably invites confusion.\n",
+    "\n",
+    "For example, we can limit the `searchDraftProspects` function call to return only general details—such as player stats—for each prospect, dramatically reducing the response size. If more information is needed, the new `getProspectDetails` function call provides expanded details. There’s no universal solution; the right approach depends on your use case and data model.\n",
+    "\n",
+    "```json\n",
+    "{\n",
+    "  \"tools\": [\n",
+    "    {\n",
+    "      \"type\": \"function\",\n",
+    "      \"name\": \"searchDraftProspects\",\n",
+    "      \"description\": \"Search NBA draft prospects by position, draft year, and projected ranking, returning only general statistics to optimize response size.\",\n",
+    "      \"parameters\": {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\n",
+    "          \"position\": {\n",
+    "            \"type\": \"string\",\n",
+    "            \"description\": \"The player's basketball position.\",\n",
+    "            \"enum\": [\n",
+    "              \"Point Guard\",\n",
+    "              \"Shooting Guard\",\n",
+    "              \"Small Forward\",\n",
+    "              \"Power Forward\",\n",
+    "              \"Center\",\n",
+    "              \"Any\"\n",
+    "            ]\n",
+    "          },\n",
+    "          \"year\": {\n",
+    "            \"type\": \"number\",\n",
+    "            \"description\": \"Draft year, e.g., 2025\"\n",
+    "          },\n",
+    "          \"maxMockDraftRanking\": {\n",
+    "            \"type\": \"number\",\n",
+    "            \"description\": \"Maximum predicted draft ranking (e.g., top 10)\"\n",
+    "          }\n",
+    "        },\n",
+    "        \"required\": [\"position\", \"year\"]\n",
+    "      }\n",
+    "    },\n",
+    "    {\n",
+    "      \"type\": \"function\",\n",
+    "      \"name\": \"getProspectDetails\",\n",
+    "      \"description\": \"Fetch detailed information for a specific NBA prospect, including comprehensive stats, agent details, and scouting reports.\",\n",
+    "      \"parameters\": {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\n",
+    "          \"playerName\": {\n",
+    "            \"type\": \"string\",\n",
+    "            \"description\": \"Full name of the prospect (e.g., Jalen Storm)\"\n",
+    "          },\n",
+    "          \"year\": {\n",
+    "            \"type\": \"number\",\n",
+    "            \"description\": \"Draft year, e.g., 2025\"\n",
+    "          },\n",
+    "          \"includeAgentInfo\": {\n",
+    "            \"type\": \"boolean\",\n",
+    "            \"description\": \"Include agent information\"\n",
+    "          },\n",
+    "          \"includeStats\": {\n",
+    "            \"type\": \"boolean\",\n",
+    "            \"description\": \"Include detailed player statistics\"\n",
+    "          },\n",
+    "          \"includeScoutingReport\": {\n",
+    "            \"type\": \"boolean\",\n",
+    "            \"description\": \"Include scouting report details\"\n",
+    "          }\n",
+    "        },\n",
+    "        \"required\": [\"playerName\", \"year\"]\n",
+    "      }\n",
+    "    }\n",
+    "  ],\n",
+    "  \"tool_choice\": \"auto\"\n",
+    "}\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 2. As conversations unfold, optimize the context\n",
+    "\n",
+    "Realtime conversations allow for generous 30-minute sessions—but the rolling context window only supports ~16,000 tokens (depending on the model snapshot, context window limitations are improving). As a result, you may notice performance gradually decline during extended exchanges. As conversations progress and more function calls are made, the conversation state can expand quickly with both important information and unnecessary noise—so it’s important to focus on keeping the most relevant details. This approach helps maintain strong performance and reduces cost."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**i) Periodically summarize the conversation state**\n",
+    "\n",
+    "Periodically summarizing the conversation as it unfolds is an excellent way to reduce context size—cutting both cost and latency.\n",
+    "\n",
+    "See @Minhajul's' epic guide on implementing automatic summarization in Realtime conversations ([link](https://cookbook.openai.com/examples/context_summarization_with_realtime_api))."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**ii) Periodically remind the the model of its role and responsibilities**\n",
+    "\n",
+    "Data-heavy payloads can quickly fill the context window. If you notice the model losing track of instructions or available tools, periodically remind it of its system prompt and tools by calling `session.update`—this keeps it focused on its role and responsibilities."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 3. Data processing and optimization"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**i) Use filtering in your function calls to trim data-heavy responses down to only the essential fields needed to answer the question**\n",
+    "\n",
+    "Generally, fewer tokens returned by function calls lead to better quality responses. Common pitfalls occur when function calls return excessively large payloads spanning thousands of tokens. Focus on applying filters in each function call, either at the data-level or function-level, to minimize response sizes.\n",
+    "\n",
+    "```json\n",
+    "// Filtered response\n",
+    "{\n",
+    "  \"status\": {\n",
+    "    \"code\": 200,\n",
+    "    \"message\": \"SUCCESS\"\n",
+    "  },\n",
+    "  \"found\": 4274,\n",
+    "  \"offset\": 0,\n",
+    "  \"limit\": 5,\n",
+    "  \"data\": [\n",
+    "    {\n",
+    "    \"zpid\": 7972122,\n",
+    "      \"data\": {\n",
+    "        \"PropertyInfo\": {\n",
+    "            \"houseNumber\": \"19661\",\n",
+    "            \"directionPrefix\": \"N \",\n",
+    "            \"streetName\": \"Central\",\n",
+    "            \"streetSuffix\": \"Ave\",\n",
+    "            \"city\": \"Phoenix\",\n",
+    "            \"state\": \"AZ\",\n",
+    "            \"postalCode\": \"85024\",\n",
+    "            \"zipPlusFour\": \"1641\"\n",
+    "            \"bedroomCount\": 2,\n",
+    "            \"bathroomCount\": 2,\n",
+    "            \"storyCount\": 1,\n",
+    "            \"livingAreaSize\": 1089,\n",
+    "            \"livingAreaSizeUnits\": \"Square Feet\",\n",
+    "            \"yearBuilt\": \"1985\"\n",
+    "          }\n",
+    "\t\t    }\n",
+    "\t\t\t}\n",
+    "\t\t]\n",
+    "\t\t// ... \n",
+    "}\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**ii) Flatten hierarchical payloads—without losing key information**\n",
+    "\n",
+    "Hierarchical payloads from API calls can sometimes include repeated level titles—like \"ProspectInfo\" or \"Stats\"—which may add extra noise and make things harder for the model to process. As you explore ways to make your data more efficient, you might try flattening these structures by trimming away some of the unnecessary labels. This can help improve performance, but consider what information is important to keep for your particular use case.\n",
+    "\n",
+    "```json\n",
+    "// Flattened payload\n",
+    "{\n",
+    "  \"status\": {\n",
+    "    \"code\": 200,\n",
+    "    \"message\": \"SUCCESS\"\n",
+    "  },\n",
+    "  \"found\": 4274,\n",
+    "  \"offset\": 0,\n",
+    "  \"limit\": 2,\n",
+    "  \"data\": [\n",
+    "    {\n",
+    "      \"prospectId\": 10001,\n",
+    "      \"league\": \"NCAA\",\n",
+    "      \"collegeId\": 301,\n",
+    "      \"isDraftEligible\": true,\n",
+    "      \"firstName\": \"Jalen\",\n",
+    "      \"lastName\": \"Storm\",\n",
+    "      \"position\": \"PG\",\n",
+    "      \"heightFeet\": 6,\n",
+    "      \"heightInches\": 4,\n",
+    "      \"weightPounds\": 205,\n",
+    "      \"hometown\": \"Springfield\",\n",
+    "      \"state\": \"IL\",\n",
+    "      \"collegeTeam\": \"Springfield Tigers\",\n",
+    "      \"conference\": \"Big West\",\n",
+    "      \"teamRanking\": 12,\n",
+    "      \"coachId\": 987,\n",
+    "      \"coachName\": \"Marcus Reed\",\n",
+    "      \"gamesPlayed\": 32,\n",
+    "      \"minutesPerGame\": 34.5,\n",
+    "      \"FieldGoalPercentage\": 47.2,\n",
+    "      \"ThreePointPercentage\": 39.1,\n",
+    "      \"FreeThrowPercentage\": 85.6,\n",
+    "      \"averagePoints\": 21.3,\n",
+    "      \"averageRebounds\": 4.1,\n",
+    "      \"averageAssists\": 6.8,\n",
+    "      \"stealsPerGame\": 1.7,\n",
+    "      \"blocksPerGame\": 0.3,\n",
+    "      \"strengths\": [\"Court vision\", \"Clutch shooting\"],\n",
+    "      \"areasForImprovement\": [\"Defensive consistency\"],\n",
+    "      \"mockDraftRanking\": 5,\n",
+    "      \"lotteryPickProbability\": 88,\n",
+    "      \"highlightReelUrl\": \"https://example.com/highlights/jalen-storm\",\n",
+    "      \"agentName\": \"Rick Allen\",\n",
+    "      \"agency\": \"Elite Sports Management\",\n",
+    "      \"contactEmail\": \"rallen@elitesports.com\"\n",
+    "    },\n",
+    "\t\t...\n",
+    " }\n",
+    " ```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**iii) Experiment with different data formats**\n",
+    "\n",
+    "The way you structure your data has a direct impact on how well the model processes and summarizes API responses. In our experience, clear, key-based formats like JSON or YAML help the model interpret data more accurately than tabular formats such as Markdown. Large tables, especially, tend to overwhelm the model—resulting in less fluent and less accurate outputs. Still, it’s worth experimenting with different formats to find what works best for your use case.\n",
+    "\n",
+    "```yaml\n",
+    "status:\n",
+    "  code: 200\n",
+    "  message: \"SUCCESS\"\n",
+    "found: 4274\n",
+    "offset: 0\n",
+    "limit: 10\n",
+    "data:\n",
+    "  - prospectId: 10001\n",
+    "    data:\n",
+    "      ProspectInfo:\n",
+    "        league: \"NCAA\"\n",
+    "        collegeId: 301\n",
+    "        isDraftEligible: true\n",
+    "        Player:\n",
+    "          firstName: \"Jalen\"\n",
+    "          lastName: \"Storm\"\n",
+    "          position: \"PG\"\n",
+    "          heightFeet: 6\n",
+    "          heightInches: 4\n",
+    "          weightPounds: 205\n",
+    "          hometown: \"Springfield\"\n",
+    "          state: \"IL\"\n",
+    "        TeamInfo:\n",
+    "          collegeTeam: \"Springfield Tigers\"\n",
+    "          conference: \"Big West\"\n",
+    "          teamRanking: 12\n",
+    "          coachId: 987\n",
+    "          coachName: \"Marcus Reed\"\n",
+    "      Stats:\n",
+    "        gamesPlayed: 32\n",
+    "        minutesPerGame: 34.5\n",
+    "        FieldGoalPercentage: 47.2\n",
+    "        ThreePointPercentage: 39.1\n",
+    "        FreeThrowPercentage: 85.6\n",
+    "        averagePoints: 21.3\n",
+    "        averageRebounds: 4.1\n",
+    "        averageAssists: 6.8\n",
+    "        stealsPerGame: 1.7\n",
+    "        blocksPerGame: 0.3\n",
+    "      Scouting:\n",
+    "        strengths:\n",
+    "          - \"Court vision\"\n",
+    "          - \"Clutch shooting\"\n",
+    "        areasForImprovement:\n",
+    "          - \"Defensive consistency\"\n",
+    "      DraftProjection:\n",
+    "        mockDraftRanking: 5\n",
+    "        lotteryPickProbability: 88\n",
+    "      Media:\n",
+    "        highlightReelUrl: \"https://example.com/highlights/jalen-storm\"\n",
+    "      Agent:\n",
+    "        agentName: \"Rick Allen\"\n",
+    "        agency: \"Elite Sports Management\"\n",
+    "        contactEmail: \"rallen@elitesports.com\"\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4. After data-heavy function calls, follow up with hint prompts\n",
+    "\n",
+    "Underlying models often struggle to transition smoothly from data-heavy responses to accurate answers. To improve fluency and accuracy when working with complex data, provide a function call hint immediately after the function call. These hints guide the model on the specific task—teaching it how to interpret key fields and domain-specific values.\n",
+    "\n",
+    "The following example illustrates an effective hint prompt.\n",
+    "\n",
+    "```javascript\n",
+    "// Function call hint\n",
+    "let prospectSearchPrompt = `\n",
+    "Parse NBA prospect data and provide a concise, engaging response.\n",
+    "\n",
+    "General Guidelines\n",
+    "- Act as an NBA scouting expert.\n",
+    "- Highlight key strengths and notable attributes.\n",
+    "- Use conversational language.\n",
+    "- Mention identical attributes once.\n",
+    "- Ignore IDs and URLs.\n",
+    "\n",
+    "Player Details\n",
+    "- State height conversationally (\"six-foot-eight\").\n",
+    "- Round weights to nearest 5 lbs.\n",
+    "\n",
+    "Stats & Draft Info\n",
+    "- Round stats to nearest whole number.\n",
+    "- Use general terms for draft ranking (\"top-five pick\").\n",
+    "Experience\n",
+    "- Refer to players as freshman, sophomore, etc., or mention professional experience.\n",
+    "- Location & TeamMention hometown city and state/country.\n",
+    "- Describe teams conversationally.\n",
+    "\n",
+    "Skip (unless asked explicitly)\n",
+    "- Exact birth dates\n",
+    "- IDs\n",
+    "- Agent/contact details\n",
+    "- URLs\n",
+    "\n",
+    "Examples\n",
+    "- \"Jalen Storm, a dynamic six-foot-four point guard from Springfield, Illinois, averages 21 points per game.\"\n",
+    "- \"Known for his clutch shooting, he's projected as a top-five pick.\"\n",
+    "\n",
+    "Important: Respond based strictly on provided data, without inventing details.\n",
+    "`;\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In practice, we first append the function call result to the conversation. Then, we emit a response from the Realtime API with the hint prompt. Voilà—the model gracefully handles all the information."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "```javascript\n",
+    "// Add new conversation item for the model\n",
+    "const conversationItem = {\n",
+    "  type: 'conversation.item.create',\n",
+    "  previous_item_id: output.id,\n",
+    "  item: {\n",
+    "    call_id: output.call_id,\n",
+    "    type: 'function_call_output',\n",
+    "    output: `Draft Prospect Search Results: ${result}`\n",
+    "  }\n",
+    "};\n",
+    "\n",
+    "dataChannel.send(JSON.stringify(conversationItem));\n",
+    "\n",
+    "// Emit a response from the model including the hint prompt\n",
+    "const event = {\n",
+    "  type: 'response.create',\n",
+    "  conversation: \"none\",\n",
+    "  response: {\n",
+    "    instructions: prospectSearchPrompt # function call hint\n",
+    "  }\n",
+    "};\n",
+    "\n",
+    "dataChannel.send(JSON.stringify(event));\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Wrapping up\n",
+    "\n",
+    "Building effective agents with the Realtime API is an ongoing process of exploration and adaptation.\n",
+    "\n",
+    "**Summary of Key Recommendations**\n",
+    "\n",
+    "- **Filter data:** Only include fields and details that are directly relevant to the user’s request or the model’s next step. Trim the rest.\n",
+    "- **Flatten and simplify structures:** Reduce deeply nested or redundant data. Present information in a way that’s easy for both models and humans to scan.\n",
+    "- **Prefer clear, structured formats:** Use JSON (or YAML) with consistent field names and minimal noise. Avoid large tables or markdown for data-heavy responses.\n",
+    "- **Guide the model with hint prompts:** After returning lots of data, follow up with a targeted prompt that explains exactly what the model should extract or summarize.\n",
+    "\n",
+    "Remember—experimentation is essential. Realtime models keep improving, and we’ll continue sharing tips to help you get the most out of the Realtime API."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

examples/Data_extraction_transformation.ipynb

@@ -0,0 +1,777 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "A lot of enterprise data is unstructured and locked up in difficult-to-use formats, e.g. PDFs, PPT, PNG, that are not optimized for use with LLMs or databases. As a result this type of data tends to be underutilized for analysis and product development, despite it being so valuable. The traditional way of extracting information from unstructured or non-ideal formats has been to use OCR, but OCR struggles with complex layouts and can have limited multilingual support. Moreover, manually applying transforms to data can be cumbersome and timeconsuming. \n",
+    "\n",
+    "The multi-modal capabilities of GPT-4o enable new ways to extract and transform data because of GPT-4o's ability to adapt to different types of documents and to use reasoning for interpreting the content of documents. Here are some reasons why you would choose GPT-4o for your extraction and transformation workflows over traditional methods. \n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "| **Extraction**                                               | **Transformation**                                              |\n",
+    "|---------------------------------------------------------------|------------------------------------------------------------------|\n",
+    "| **Adaptable**: Handles complex document layouts better, reducing errors | **Schema Adaptability**: Easily transforms data to fit specific schemas for database ingestion |\n",
+    "| **Multilingual Support**: Seamlessly processes documents in multiple languages | **Dynamic Data Mapping**: Adapts to different data structures and formats, providing flexible transformation rules |\n",
+    "| **Contextual Understanding**: Extracts meaningful relationships and context, not just text | **Enhanced Insight Generation**: Applies reasoning to create more insightful transformations, enriching the dataset with derived metrics, metadata and relationships |\n",
+    "| **Multimodality**: Processes various document elements, including images and tables |  |\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This cookbook has three parts:\n",
+    "1. How to extract data from multilingual PDFs \n",
+    "2. How to transform data according to a schema for loading into a database\n",
+    "3. How to load transformed data into a database for downstream analysis\n",
+    "\n",
+    "We're going to mimic a simple ELT workflow where data is first extracted from PDFs into JSON using GPT-4o, stored in an unstructured format somewhere like a data lake, transformed to fit a schema using GPT-4o, and then finally ingested into a relational database for querying. It's worth noting that you can do all of this with the BatchAPI if you're interested in lowering the cost of this workflow. \n",
+    "\n",
+    "![](../images/elt_workflow.png)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The data we'll be using is a set of publicly available 2019 hotel invoices from Germany available on [Jens Walter's GitHub](https://github.com/JensWalter/my-receipts/tree/master/2019/de/hotel), (thank you Jens!). Though hotel invoices generally contain similar information (reservation details, charges, taxes etc.), you'll notice that the invoices present itemized information in different ways and are multilingual containing both German and English. Fortunately GPT-4o can adapt to a variety of different document styles without us having to specify formats and it can seamlessly handle a variety of languages, even in the same document. \n",
+    "Here is what one of the invoices looks like: \n",
+    "\n",
+    "![](../images/sample_hotel_invoice.png)\n",
+    "\n",
+    "## Part 1: Extracting data from PDFs using GPT-4o's vision capabilities\n",
+    "GPT-4o doesn't natively handle PDFs so before we extract any data we'll first need to convert each page into an image and then encode the images as base64. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from openai import OpenAI\n",
+    "import fitz  # PyMuPDF\n",
+    "import io\n",
+    "import os\n",
+    "from PIL import Image\n",
+    "import base64\n",
+    "import json\n",
+    "\n",
+    "api_key = os.getenv(\"OPENAI_API_KEY\")\n",
+    "client = OpenAI(api_key=api_key)\n",
+    "\n",
+    "\n",
+    "@staticmethod\n",
+    "def encode_image(image_path):\n",
+    "    with open(image_path, \"rb\") as image_file:\n",
+    "        return base64.b64encode(image_file.read()).decode(\"utf-8\")\n",
+    "\n",
+    "\n",
+    "def pdf_to_base64_images(pdf_path):\n",
+    "    #Handles PDFs with multiple pages\n",
+    "    pdf_document = fitz.open(pdf_path)\n",
+    "    base64_images = []\n",
+    "    temp_image_paths = []\n",
+    "\n",
+    "    total_pages = len(pdf_document)\n",
+    "\n",
+    "    for page_num in range(total_pages):\n",
+    "        page = pdf_document.load_page(page_num)\n",
+    "        pix = page.get_pixmap()\n",
+    "        img = Image.open(io.BytesIO(pix.tobytes()))\n",
+    "        temp_image_path = f\"temp_page_{page_num}.png\"\n",
+    "        img.save(temp_image_path, format=\"PNG\")\n",
+    "        temp_image_paths.append(temp_image_path)\n",
+    "        base64_image = encode_image(temp_image_path)\n",
+    "        base64_images.append(base64_image)\n",
+    "\n",
+    "    for temp_image_path in temp_image_paths:\n",
+    "        os.remove(temp_image_path)\n",
+    "\n",
+    "    return base64_images"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can then pass each base64 encoded image in a GPT-4o LLM call, specifying a high level of detail and JSON as the response format. We're not concerned about enforcing a schema at this step, we just want all of the data to be extracted regardless of type."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def extract_invoice_data(base64_image):\n",
+    "    system_prompt = f\"\"\"\n",
+    "    You are an OCR-like data extraction tool that extracts hotel invoice data from PDFs.\n",
+    "   \n",
+    "    1. Please extract the data in this hotel invoice, grouping data according to theme/sub groups, and then output into JSON.\n",
+    "\n",
+    "    2. Please keep the keys and values of the JSON in the original language. \n",
+    "\n",
+    "    3. The type of data you might encounter in the invoice includes but is not limited to: hotel information, guest information, invoice information,\n",
+    "    room charges, taxes, and total charges etc. \n",
+    "\n",
+    "    4. If the page contains no charge data, please output an empty JSON object and don't make up any data.\n",
+    "\n",
+    "    5. If there are blank data fields in the invoice, please include them as \"null\" values in the JSON object.\n",
+    "    \n",
+    "    6. If there are tables in the invoice, capture all of the rows and columns in the JSON object. \n",
+    "    Even if a column is blank, include it as a key in the JSON object with a null value.\n",
+    "    \n",
+    "    7. If a row is blank denote missing fields with \"null\" values. \n",
+    "    \n",
+    "    8. Don't interpolate or make up data.\n",
+    "\n",
+    "    9. Please maintain the table structure of the charges, i.e. capture all of the rows and columns in the JSON object.\n",
+    "\n",
+    "    \"\"\"\n",
+    "    \n",
+    "    response = client.chat.completions.create(\n",
+    "        model=\"gpt-4o\",\n",
+    "        response_format={ \"type\": \"json_object\" },\n",
+    "        messages=[\n",
+    "            {\n",
+    "                \"role\": \"system\",\n",
+    "                \"content\": system_prompt\n",
+    "            },\n",
+    "            {\n",
+    "                \"role\": \"user\",\n",
+    "                \"content\": [\n",
+    "                    {\"type\": \"text\", \"text\": \"extract the data in this hotel invoice and output into JSON \"},\n",
+    "                    {\"type\": \"image_url\", \"image_url\": {\"url\": f\"data:image/png;base64,{base64_image}\", \"detail\": \"high\"}}\n",
+    "                ]\n",
+    "            }\n",
+    "        ],\n",
+    "        temperature=0.0,\n",
+    "    )\n",
+    "    return response.choices[0].message.content\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Because invoice data can span multiple pages in a PDF, we're going to produce JSON objects for each page in the invoice and then append them together. The final invoice extraction will be a single JSON file."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def extract_from_multiple_pages(base64_images, original_filename, output_directory):\n",
+    "    entire_invoice = []\n",
+    "\n",
+    "    for base64_image in base64_images:\n",
+    "        invoice_json = extract_invoice_data(base64_image)\n",
+    "        invoice_data = json.loads(invoice_json)\n",
+    "        entire_invoice.append(invoice_data)\n",
+    "\n",
+    "    # Ensure the output directory exists\n",
+    "    os.makedirs(output_directory, exist_ok=True)\n",
+    "\n",
+    "    # Construct the output file path\n",
+    "    output_filename = os.path.join(output_directory, original_filename.replace('.pdf', '_extracted.json'))\n",
+    "    \n",
+    "    # Save the entire_invoice list as a JSON file\n",
+    "    with open(output_filename, 'w', encoding='utf-8') as f:\n",
+    "        json.dump(entire_invoice, f, ensure_ascii=False, indent=4)\n",
+    "    return output_filename\n",
+    "\n",
+    "\n",
+    "def main_extract(read_path, write_path):\n",
+    "    for filename in os.listdir(read_path):\n",
+    "        file_path = os.path.join(read_path, filename)\n",
+    "        if os.path.isfile(file_path):\n",
+    "            base64_images = pdf_to_base64_images(file_path)\n",
+    "            extract_from_multiple_pages(base64_images, filename, write_path)\n",
+    "\n",
+    "\n",
+    "read_path= \"./data/hotel_invoices/receipts_2019_de_hotel\"\n",
+    "write_path= \"./data/hotel_invoices/extracted_invoice_json\"\n",
+    "\n",
+    "main_extract(read_path, write_path)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Each invoice JSON will have different keys depending on what data the original invoice contained, so at this point you can store the unschematized JSON files in a data lake that can handle unstructured data. For simplicity though, we're going to store the files in a folder. Here is what one of the extracted JSON files looks like, you'll notice that even though we didn't specify a schema, GPT-4o was able to understand German and group similar information together. Moreover, if there was a blank field in the invoice GPT-4o transcribed that as \"null\". "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "vscode": {
+     "languageId": "javascript"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "[\n",
+    "    {\n",
+    "        \"Hotel Information\": {\n",
+    "            \"Name\": \"Hamburg City (Zentrum)\",\n",
+    "            \"Address\": \"Willy-Brandt-Straße 21, 20457 Hamburg, Deutschland\",\n",
+    "            \"Phone\": \"+49 (0) 40 3039 379 0\"\n",
+    "        },\n",
+    "        \"Guest Information\": {\n",
+    "            \"Name\": \"APIMEISTER CONSULTING GmbH\",\n",
+    "            \"Guest\": \"Herr Jens Walter\",\n",
+    "            \"Address\": \"Friedrichstr. 123, 10117 Berlin\"\n",
+    "        },\n",
+    "        \"Invoice Information\": {\n",
+    "            \"Rechnungsnummer\": \"GABC19014325\",\n",
+    "            \"Rechnungsdatum\": \"23.09.19\",\n",
+    "            \"Referenznummer\": \"GABC015452127\",\n",
+    "            \"Buchungsnummer\": \"GABR15867\",\n",
+    "            \"Ankunft\": \"23.09.19\",\n",
+    "            \"Abreise\": \"27.09.19\",\n",
+    "            \"Nächte\": 4,\n",
+    "            \"Zimmer\": 626,\n",
+    "            \"Kundereferenz\": 2\n",
+    "        },\n",
+    "        \"Charges\": [\n",
+    "            {\n",
+    "                \"Datum\": \"23.09.19\",\n",
+    "                \"Uhrzeit\": \"16:36\",\n",
+    "                \"Beschreibung\": \"Übernachtung\",\n",
+    "                \"MwSt.%\": 7.0,\n",
+    "                \"Betrag\": 77.0,\n",
+    "                \"Zahlung\": null\n",
+    "            },\n",
+    "            {\n",
+    "                \"Datum\": \"24.09.19\",\n",
+    "                \"Uhrzeit\": null,\n",
+    "                \"Beschreibung\": \"Übernachtung\",\n",
+    "                \"MwSt.%\": 7.0,\n",
+    "                \"Betrag\": 135.0,\n",
+    "                \"Zahlung\": null\n",
+    "            },\n",
+    "            {\n",
+    "                \"Datum\": \"25.09.19\",\n",
+    "                \"Uhrzeit\": null,\n",
+    "                \"Beschreibung\": \"Übernachtung\",\n",
+    "                \"MwSt.%\": 7.0,\n",
+    "                \"Betrag\": 82.0,\n",
+    "                \"Zahlung\": null\n",
+    "            },\n",
+    "            {\n",
+    "                \"Datum\": \"26.09.19\",\n",
+    "                \"Uhrzeit\": null,\n",
+    "                \"Beschreibung\": \"Übernachtung\",\n",
+    "                \"MwSt.%\": 7.0,\n",
+    "                \"Betrag\": 217.0,\n",
+    "                \"Zahlung\": null\n",
+    "            },\n",
+    "            {\n",
+    "                \"Datum\": \"24.09.19\",\n",
+    "                \"Uhrzeit\": \"9:50\",\n",
+    "                \"Beschreibung\": \"Premier Inn Frühstücksbuffet\",\n",
+    "                \"MwSt.%\": 19.0,\n",
+    "                \"Betrag\": 9.9,\n",
+    "                \"Zahlung\": null\n",
+    "            },\n",
+    "            {\n",
+    "                \"Datum\": \"25.09.19\",\n",
+    "                \"Uhrzeit\": \"9:50\",\n",
+    "                \"Beschreibung\": \"Premier Inn Frühstücksbuffet\",\n",
+    "                \"MwSt.%\": 19.0,\n",
+    "                \"Betrag\": 9.9,\n",
+    "                \"Zahlung\": null\n",
+    "            },\n",
+    "            {\n",
+    "                \"Datum\": \"26.09.19\",\n",
+    "                \"Uhrzeit\": \"9:50\",\n",
+    "                \"Beschreibung\": \"Premier Inn Frühstücksbuffet\",\n",
+    "                \"MwSt.%\": 19.0,\n",
+    "                \"Betrag\": 9.9,\n",
+    "                \"Zahlung\": null\n",
+    "            },\n",
+    "            {\n",
+    "                \"Datum\": \"27.09.19\",\n",
+    "                \"Uhrzeit\": \"9:50\",\n",
+    "                \"Beschreibung\": \"Premier Inn Frühstücksbuffet\",\n",
+    "                \"MwSt.%\": 19.0,\n",
+    "                \"Betrag\": 9.9,\n",
+    "                \"Zahlung\": null\n",
+    "            }\n",
+    "        ],\n",
+    "        \"Payment Information\": {\n",
+    "            \"Zahlung\": \"550,60\",\n",
+    "            \"Gesamt (Rechnungsbetrag)\": \"550,60\",\n",
+    "            \"Offener Betrag\": \"0,00\",\n",
+    "            \"Bezahlart\": \"Mastercard-Kreditkarte\"\n",
+    "        },\n",
+    "        \"Tax Information\": {\n",
+    "            \"MwSt.%\": [\n",
+    "                {\n",
+    "                    \"Rate\": 19.0,\n",
+    "                    \"Netto\": 33.28,\n",
+    "                    \"MwSt.\": 6.32,\n",
+    "                    \"Brutto\": 39.6\n",
+    "                },\n",
+    "                {\n",
+    "                    \"Rate\": 7.0,\n",
+    "                    \"Netto\": 477.57,\n",
+    "                    \"MwSt.\": 33.43,\n",
+    "                    \"Brutto\": 511.0\n",
+    "                }\n",
+    "            ]\n",
+    "        }\n",
+    "    }\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Part 2: Transforming data according to a schema \n",
+    "\n",
+    "You've extracted data from PDFs and have likely loaded the unstructured extractions as JSON objects in a data lake. The next step in our ELT workflow is to use GPT-4o to transform the extractions according to our desired schema. This will enable us to ingest any resulting tables into a database. We've decided upon the following schema that broadly covers most of the information we would have seen across the different invoices. This schema will be used to process each raw JSON extraction into our desired schematized JSON and can specify particular formats such as \"date\": \"YYYY-MM-DD\". We're also going to translate the data into English at this step. \n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "vscode": {
+     "languageId": "javascript"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "[\n",
+    "    {\n",
+    "        \"hotel_information\": {\n",
+    "            \"name\": \"string\",\n",
+    "            \"address\": {\n",
+    "                \"street\": \"string\",\n",
+    "                \"city\": \"string\",\n",
+    "                \"country\": \"string\",\n",
+    "                \"postal_code\": \"string\"\n",
+    "            },\n",
+    "            \"contact\": {\n",
+    "                \"phone\": \"string\",\n",
+    "                \"fax\": \"string\",\n",
+    "                \"email\": \"string\",\n",
+    "                \"website\": \"string\"\n",
+    "            }\n",
+    "        },\n",
+    "        \"guest_information\": {\n",
+    "            \"company\": \"string\",\n",
+    "            \"address\": \"string\",\n",
+    "            \"guest_name\": \"string\"\n",
+    "        },\n",
+    "        \"invoice_information\": {\n",
+    "            \"invoice_number\": \"string\",\n",
+    "            \"reservation_number\": \"string\",\n",
+    "            \"date\": \"YYYY-MM-DD\",  \n",
+    "            \"room_number\": \"string\",\n",
+    "            \"check_in_date\": \"YYYY-MM-DD\",  \n",
+    "            \"check_out_date\": \"YYYY-MM-DD\"  \n",
+    "        },\n",
+    "        \"charges\": [\n",
+    "            {\n",
+    "                \"date\": \"YYYY-MM-DD\", \n",
+    "                \"description\": \"string\",\n",
+    "                \"charge\": \"number\",\n",
+    "                \"credit\": \"number\"\n",
+    "            }\n",
+    "        ],\n",
+    "        \"totals_summary\": {\n",
+    "            \"currency\": \"string\",\n",
+    "            \"total_net\": \"number\",\n",
+    "            \"total_tax\": \"number\",\n",
+    "            \"total_gross\": \"number\",\n",
+    "            \"total_charge\": \"number\",\n",
+    "            \"total_credit\": \"number\",\n",
+    "            \"balance_due\": \"number\"\n",
+    "        },\n",
+    "        \"taxes\": [\n",
+    "            {\n",
+    "                \"tax_type\": \"string\",\n",
+    "                \"tax_rate\": \"string\",\n",
+    "                \"net_amount\": \"number\",\n",
+    "                \"tax_amount\": \"number\",\n",
+    "                \"gross_amount\": \"number\"\n",
+    "            }\n",
+    "        ]\n",
+    "    }\n",
+    "]\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def transform_invoice_data(json_raw, json_schema):\n",
+    "    system_prompt = f\"\"\"\n",
+    "    You are a data transformation tool that takes in JSON data and a reference JSON schema, and outputs JSON data according to the schema.\n",
+    "    Not all of the data in the input JSON will fit the schema, so you may need to omit some data or add null values to the output JSON.\n",
+    "    Translate all data into English if not already in English.\n",
+    "    Ensure values are formatted as specified in the schema (e.g. dates as YYYY-MM-DD).\n",
+    "    Here is the schema:\n",
+    "    {json_schema}\n",
+    "\n",
+    "    \"\"\"\n",
+    "    \n",
+    "    response = client.chat.completions.create(\n",
+    "        model=\"gpt-4o\",\n",
+    "        response_format={ \"type\": \"json_object\" },\n",
+    "        messages=[\n",
+    "            {\n",
+    "                \"role\": \"system\",\n",
+    "                \"content\": system_prompt\n",
+    "            },\n",
+    "            {\n",
+    "                \"role\": \"user\",\n",
+    "                \"content\": [\n",
+    "                    {\"type\": \"text\", \"text\": f\"Transform the following raw JSON data according to the provided schema. Ensure all data is in English and formatted as specified by values in the schema. Here is the raw JSON: {json_raw}\"}\n",
+    "                ]\n",
+    "            }\n",
+    "        ],\n",
+    "        temperature=0.0,\n",
+    "    )\n",
+    "    return json.loads(response.choices[0].message.content)\n",
+    "\n",
+    "\n",
+    "\n",
+    "def main_transform(extracted_invoice_json_path, json_schema_path, save_path):\n",
+    "    # Load the JSON schema\n",
+    "    with open(json_schema_path, 'r', encoding='utf-8') as f:\n",
+    "        json_schema = json.load(f)\n",
+    "\n",
+    "    # Ensure the save directory exists\n",
+    "    os.makedirs(save_path, exist_ok=True)\n",
+    "\n",
+    "    # Process each JSON file in the extracted invoices directory\n",
+    "    for filename in os.listdir(extracted_invoice_json_path):\n",
+    "        if filename.endswith(\".json\"):\n",
+    "            file_path = os.path.join(extracted_invoice_json_path, filename)\n",
+    "\n",
+    "            # Load the extracted JSON\n",
+    "            with open(file_path, 'r', encoding='utf-8') as f:\n",
+    "                json_raw = json.load(f)\n",
+    "\n",
+    "            # Transform the JSON data\n",
+    "            transformed_json = transform_invoice_data(json_raw, json_schema)\n",
+    "\n",
+    "            # Save the transformed JSON to the save directory\n",
+    "            transformed_filename = f\"transformed_{filename}\"\n",
+    "            transformed_file_path = os.path.join(save_path, transformed_filename)\n",
+    "            with open(transformed_file_path, 'w', encoding='utf-8') as f:\n",
+    "                json.dump(transformed_json, f, ensure_ascii=False, indent=2)\n",
+    "\n",
+    "   \n",
+    "    extracted_invoice_json_path = \"./data/hotel_invoices/extracted_invoice_json\"\n",
+    "    json_schema_path = \"./data/hotel_invoices/invoice_schema.json\"\n",
+    "    save_path = \"./data/hotel_invoices/transformed_invoice_json\"\n",
+    "\n",
+    "    main_transform(extracted_invoice_json_path, json_schema_path, save_path)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Part 3: Loading transformed data into a database \n",
+    "\n",
+    "Now that we've schematized all of our data, we can segment it into tables for ingesting into a relational database. In particular, we're going to create four tables: Hotels, Invoices, Charges and Taxes. All of the invoices pertained to one guest, so we won't create a guest table. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import json\n",
+    "import sqlite3\n",
+    "\n",
+    "def ingest_transformed_jsons(json_folder_path, db_path):\n",
+    "    conn = sqlite3.connect(db_path)\n",
+    "    cursor = conn.cursor()\n",
+    "\n",
+    "    # Create necessary tables\n",
+    "    cursor.execute('''\n",
+    "    CREATE TABLE IF NOT EXISTS Hotels (\n",
+    "        hotel_id INTEGER PRIMARY KEY AUTOINCREMENT,\n",
+    "        name TEXT,\n",
+    "        street TEXT,\n",
+    "        city TEXT,\n",
+    "        country TEXT,\n",
+    "        postal_code TEXT,\n",
+    "        phone TEXT,\n",
+    "        fax TEXT,\n",
+    "        email TEXT,\n",
+    "        website TEXT\n",
+    "    )\n",
+    "    ''')\n",
+    "\n",
+    "    cursor.execute('''\n",
+    "    CREATE TABLE IF NOT EXISTS Invoices (\n",
+    "        invoice_id INTEGER PRIMARY KEY AUTOINCREMENT,\n",
+    "        hotel_id INTEGER,\n",
+    "        invoice_number TEXT,\n",
+    "        reservation_number TEXT,\n",
+    "        date TEXT,\n",
+    "        room_number TEXT,\n",
+    "        check_in_date TEXT,\n",
+    "        check_out_date TEXT,\n",
+    "        currency TEXT,\n",
+    "        total_net REAL,\n",
+    "        total_tax REAL,\n",
+    "        total_gross REAL,\n",
+    "        total_charge REAL,\n",
+    "        total_credit REAL,\n",
+    "        balance_due REAL,\n",
+    "        guest_company TEXT,\n",
+    "        guest_address TEXT,\n",
+    "        guest_name TEXT,\n",
+    "        FOREIGN KEY(hotel_id) REFERENCES Hotels(hotel_id)\n",
+    "    )\n",
+    "    ''')\n",
+    "\n",
+    "    cursor.execute('''\n",
+    "    CREATE TABLE IF NOT EXISTS Charges (\n",
+    "        charge_id INTEGER PRIMARY KEY AUTOINCREMENT,\n",
+    "        invoice_id INTEGER,\n",
+    "        date TEXT,\n",
+    "        description TEXT,\n",
+    "        charge REAL,\n",
+    "        credit REAL,\n",
+    "        FOREIGN KEY(invoice_id) REFERENCES Invoices(invoice_id)\n",
+    "    )\n",
+    "    ''')\n",
+    "\n",
+    "    cursor.execute('''\n",
+    "    CREATE TABLE IF NOT EXISTS Taxes (\n",
+    "        tax_id INTEGER PRIMARY KEY AUTOINCREMENT,\n",
+    "        invoice_id INTEGER,\n",
+    "        tax_type TEXT,\n",
+    "        tax_rate TEXT,\n",
+    "        net_amount REAL,\n",
+    "        tax_amount REAL,\n",
+    "        gross_amount REAL,\n",
+    "        FOREIGN KEY(invoice_id) REFERENCES Invoices(invoice_id)\n",
+    "    )\n",
+    "    ''')\n",
+    "\n",
+    "    # Loop over all JSON files in the specified folder\n",
+    "    for filename in os.listdir(json_folder_path):\n",
+    "        if filename.endswith(\".json\"):\n",
+    "            file_path = os.path.join(json_folder_path, filename)\n",
+    "\n",
+    "            # Load the JSON data\n",
+    "            with open(file_path, 'r', encoding='utf-8') as f:\n",
+    "                data = json.load(f)\n",
+    "\n",
+    "            # Insert Hotel Information\n",
+    "            cursor.execute('''\n",
+    "            INSERT INTO Hotels (name, street, city, country, postal_code, phone, fax, email, website) \n",
+    "            VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)\n",
+    "            ''', (\n",
+    "                data[\"hotel_information\"][\"name\"],\n",
+    "                data[\"hotel_information\"][\"address\"][\"street\"],\n",
+    "                data[\"hotel_information\"][\"address\"][\"city\"],\n",
+    "                data[\"hotel_information\"][\"address\"][\"country\"],\n",
+    "                data[\"hotel_information\"][\"address\"][\"postal_code\"],\n",
+    "                data[\"hotel_information\"][\"contact\"][\"phone\"],\n",
+    "                data[\"hotel_information\"][\"contact\"][\"fax\"],\n",
+    "                data[\"hotel_information\"][\"contact\"][\"email\"],\n",
+    "                data[\"hotel_information\"][\"contact\"][\"website\"]\n",
+    "            ))\n",
+    "            hotel_id = cursor.lastrowid\n",
+    "\n",
+    "            # Insert Invoice Information\n",
+    "            cursor.execute('''\n",
+    "            INSERT INTO Invoices (hotel_id, invoice_number, reservation_number, date, room_number, check_in_date, check_out_date, currency, total_net, total_tax, total_gross, total_charge, total_credit, balance_due, guest_company, guest_address, guest_name)\n",
+    "            VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\n",
+    "            ''', (\n",
+    "                hotel_id,\n",
+    "                data[\"invoice_information\"][\"invoice_number\"],\n",
+    "                data[\"invoice_information\"][\"reservation_number\"],\n",
+    "                data[\"invoice_information\"][\"date\"],\n",
+    "                data[\"invoice_information\"][\"room_number\"],\n",
+    "                data[\"invoice_information\"][\"check_in_date\"],\n",
+    "                data[\"invoice_information\"][\"check_out_date\"],\n",
+    "                data[\"totals_summary\"][\"currency\"],\n",
+    "                data[\"totals_summary\"][\"total_net\"],\n",
+    "                data[\"totals_summary\"][\"total_tax\"],\n",
+    "                data[\"totals_summary\"][\"total_gross\"],\n",
+    "                data[\"totals_summary\"][\"total_charge\"],\n",
+    "                data[\"totals_summary\"][\"total_credit\"],\n",
+    "                data[\"totals_summary\"][\"balance_due\"],\n",
+    "                data[\"guest_information\"][\"company\"],\n",
+    "                data[\"guest_information\"][\"address\"],\n",
+    "                data[\"guest_information\"][\"guest_name\"]\n",
+    "            ))\n",
+    "            invoice_id = cursor.lastrowid\n",
+    "\n",
+    "            # Insert Charges\n",
+    "            for charge in data[\"charges\"]:\n",
+    "                cursor.execute('''\n",
+    "                INSERT INTO Charges (invoice_id, date, description, charge, credit) \n",
+    "                VALUES (?, ?, ?, ?, ?)\n",
+    "                ''', (\n",
+    "                    invoice_id,\n",
+    "                    charge[\"date\"],\n",
+    "                    charge[\"description\"],\n",
+    "                    charge[\"charge\"],\n",
+    "                    charge[\"credit\"]\n",
+    "                ))\n",
+    "\n",
+    "            # Insert Taxes\n",
+    "            for tax in data[\"taxes\"]:\n",
+    "                cursor.execute('''\n",
+    "                INSERT INTO Taxes (invoice_id, tax_type, tax_rate, net_amount, tax_amount, gross_amount) \n",
+    "                VALUES (?, ?, ?, ?, ?, ?)\n",
+    "                ''', (\n",
+    "                    invoice_id,\n",
+    "                    tax[\"tax_type\"],\n",
+    "                    tax[\"tax_rate\"],\n",
+    "                    tax[\"net_amount\"],\n",
+    "                    tax[\"tax_amount\"],\n",
+    "                    tax[\"gross_amount\"]\n",
+    "                ))\n",
+    "\n",
+    "    conn.commit()\n",
+    "    conn.close()\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now let's check that we've correctly ingested the data by running a sample SQL query to determine the most expensive hotel stay and the same of the hotel!  \n",
+    "You can even automate the generation of SQL queries at this step by using function calling, check out our [cookbook on function calling with model generated arguments](https://cookbook.openai.com/examples/how_to_call_functions_with_chat_models#how-to-call-functions-with-model-generated-arguments) to learn how to do that. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "('Citadines Michel Hamburg', 903.63)\n"
+     ]
+    }
+   ],
+   "source": [
+    "\n",
+    "def execute_query(db_path, query, params=()):\n",
+    "    \"\"\"\n",
+    "    Execute a SQL query and return the results.\n",
+    "\n",
+    "    Parameters:\n",
+    "    db_path (str): Path to the SQLite database file.\n",
+    "    query (str): SQL query to be executed.\n",
+    "    params (tuple): Parameters to be passed to the query (default is an empty tuple).\n",
+    "\n",
+    "    Returns:\n",
+    "    list: List of rows returned by the query.\n",
+    "    \"\"\"\n",
+    "    try:\n",
+    "        # Connect to the SQLite database\n",
+    "        conn = sqlite3.connect(db_path)\n",
+    "        cursor = conn.cursor()\n",
+    "\n",
+    "        # Execute the query with parameters\n",
+    "        cursor.execute(query, params)\n",
+    "        results = cursor.fetchall()\n",
+    "\n",
+    "        # Commit if it's an INSERT/UPDATE/DELETE query\n",
+    "        if query.strip().upper().startswith(('INSERT', 'UPDATE', 'DELETE')):\n",
+    "            conn.commit()\n",
+    "\n",
+    "        return results\n",
+    "    except sqlite3.Error as e:\n",
+    "        print(f\"An error occurred: {e}\")\n",
+    "        return []\n",
+    "    finally:\n",
+    "        # Close the connection\n",
+    "        if conn:\n",
+    "            conn.close()\n",
+    "\n",
+    "\n",
+    "# Example usage\n",
+    "transformed_invoices_path = \"./data/hotel_invoices/transformed_invoice_json\"\n",
+    "db_path = \"./data/hotel_invoices/hotel_DB.db\"\n",
+    "ingest_transformed_jsons(transformed_invoices_path, db_path)\n",
+    "\n",
+    "query = '''\n",
+    "    SELECT \n",
+    "        h.name AS hotel_name,\n",
+    "        i.total_gross AS max_spent\n",
+    "    FROM \n",
+    "        Invoices i\n",
+    "    JOIN \n",
+    "        Hotels h ON i.hotel_id = h.hotel_id\n",
+    "    ORDER BY \n",
+    "        i.total_gross DESC\n",
+    "    LIMIT 1;\n",
+    "    '''\n",
+    "\n",
+    "results = execute_query(db_path, query)\n",
+    "for row in results:\n",
+    "    print(row)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To recap in this cookbook we showed you how to use GPT-4o for extracting and transforming data that would otherwise be inaccessible for data analysis. If you don't need these workflows to happen in real-time, you can take advantage of OpenAI's BatchAPI to run jobs asynchronously at a much lower cost! "
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "openai",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/Embedding_Wikipedia_articles_for_search.ipynb

@@ -0,0 +1,658 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Embedding Wikipedia articles for search\n",
+    "\n",
+    "This notebook shows how we prepared a dataset of Wikipedia articles for search, used in [Question_answering_using_embeddings.ipynb](Question_answering_using_embeddings.ipynb).\n",
+    "\n",
+    "Procedure:\n",
+    "\n",
+    "0. Prerequisites: Import libraries, set API key (if needed)\n",
+    "1. Collect: We download a few hundred Wikipedia articles about the 2022 Olympics\n",
+    "2. Chunk: Documents are split into short, semi-self-contained sections to be embedded\n",
+    "3. Embed: Each section is embedded with the OpenAI API\n",
+    "4. Store: Embeddings are saved in a CSV file (for large datasets, use a vector database)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 0. Prerequisites\n",
+    "\n",
+    "### Import libraries"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# imports\n",
+    "import mwclient  # for downloading example Wikipedia articles\n",
+    "import mwparserfromhell  # for splitting Wikipedia articles into sections\n",
+    "from openai import OpenAI  # for generating embeddings\n",
+    "import os  # for environment variables\n",
+    "import pandas as pd  # for DataFrames to store article sections and embeddings\n",
+    "import re  # for cutting <ref> links out of Wikipedia articles\n",
+    "import tiktoken  # for counting tokens\n",
+    "\n",
+    "client = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Install any missing libraries with `pip install` in your terminal. E.g.,\n",
+    "\n",
+    "```zsh\n",
+    "pip install openai\n",
+    "```\n",
+    "\n",
+    "(You can also do this in a notebook cell with `!pip install openai`.)\n",
+    "\n",
+    "If you install any libraries, be sure to restart the notebook kernel."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Set API key (if needed)\n",
+    "\n",
+    "Note that the OpenAI library will try to read your API key from the `OPENAI_API_KEY` environment variable. If you haven't already, set this environment variable by following [these instructions](https://help.openai.com/en/articles/5112595-best-practices-for-api-key-safety)."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Collect documents\n",
+    "\n",
+    "In this example, we'll download a few hundred Wikipedia articles related to the 2022 Winter Olympics."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Found 179 article titles in Category:2022 Winter Olympics.\n"
+     ]
+    }
+   ],
+   "source": [
+    "# get Wikipedia pages about the 2022 Winter Olympics\n",
+    "\n",
+    "CATEGORY_TITLE = \"Category:2022 Winter Olympics\"\n",
+    "WIKI_SITE = \"en.wikipedia.org\"\n",
+    "\n",
+    "\n",
+    "def titles_from_category(\n",
+    "    category: mwclient.listing.Category, max_depth: int\n",
+    ") -> set[str]:\n",
+    "    \"\"\"Return a set of page titles in a given Wiki category and its subcategories.\"\"\"\n",
+    "    titles = set()\n",
+    "    for cm in category.members():\n",
+    "        if type(cm) == mwclient.page.Page:\n",
+    "            # ^type() used instead of isinstance() to catch match w/ no inheritance\n",
+    "            titles.add(cm.name)\n",
+    "        elif isinstance(cm, mwclient.listing.Category) and max_depth > 0:\n",
+    "            deeper_titles = titles_from_category(cm, max_depth=max_depth - 1)\n",
+    "            titles.update(deeper_titles)\n",
+    "    return titles\n",
+    "\n",
+    "\n",
+    "site = mwclient.Site(WIKI_SITE)\n",
+    "category_page = site.pages[CATEGORY_TITLE]\n",
+    "titles = titles_from_category(category_page, max_depth=1)\n",
+    "# ^note: max_depth=1 means we go one level deep in the category tree\n",
+    "print(f\"Found {len(titles)} article titles in {CATEGORY_TITLE}.\")\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Chunk documents\n",
+    "\n",
+    "Now that we have our reference documents, we need to prepare them for search.\n",
+    "\n",
+    "Because GPT can only read a limited amount of text at once, we'll split each document into chunks short enough to be read.\n",
+    "\n",
+    "For this specific example on Wikipedia articles, we'll:\n",
+    "- Discard less relevant-looking sections like External Links and Footnotes\n",
+    "- Clean up the text by removing reference tags (e.g., <ref>), whitespace, and super short sections\n",
+    "- Split each article into sections\n",
+    "- Prepend titles and subtitles to each section's text, to help GPT understand the context\n",
+    "- If a section is long (say, > 1,600 tokens), we'll recursively split it into smaller sections, trying to split along semantic boundaries like paragraphs"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# define functions to split Wikipedia pages into sections\n",
+    "\n",
+    "SECTIONS_TO_IGNORE = [\n",
+    "    \"See also\",\n",
+    "    \"References\",\n",
+    "    \"External links\",\n",
+    "    \"Further reading\",\n",
+    "    \"Footnotes\",\n",
+    "    \"Bibliography\",\n",
+    "    \"Sources\",\n",
+    "    \"Citations\",\n",
+    "    \"Literature\",\n",
+    "    \"Footnotes\",\n",
+    "    \"Notes and references\",\n",
+    "    \"Photo gallery\",\n",
+    "    \"Works cited\",\n",
+    "    \"Photos\",\n",
+    "    \"Gallery\",\n",
+    "    \"Notes\",\n",
+    "    \"References and sources\",\n",
+    "    \"References and notes\",\n",
+    "]\n",
+    "\n",
+    "\n",
+    "def all_subsections_from_section(\n",
+    "    section: mwparserfromhell.wikicode.Wikicode,\n",
+    "    parent_titles: list[str],\n",
+    "    sections_to_ignore: set[str],\n",
+    ") -> list[tuple[list[str], str]]:\n",
+    "    \"\"\"\n",
+    "    From a Wikipedia section, return a flattened list of all nested subsections.\n",
+    "    Each subsection is a tuple, where:\n",
+    "        - the first element is a list of parent subtitles, starting with the page title\n",
+    "        - the second element is the text of the subsection (but not any children)\n",
+    "    \"\"\"\n",
+    "    headings = [str(h) for h in section.filter_headings()]\n",
+    "    title = headings[0]\n",
+    "    if title.strip(\"=\" + \" \") in sections_to_ignore:\n",
+    "        # ^wiki headings are wrapped like \"== Heading ==\"\n",
+    "        return []\n",
+    "    titles = parent_titles + [title]\n",
+    "    full_text = str(section)\n",
+    "    section_text = full_text.split(title)[1]\n",
+    "    if len(headings) == 1:\n",
+    "        return [(titles, section_text)]\n",
+    "    else:\n",
+    "        first_subtitle = headings[1]\n",
+    "        section_text = section_text.split(first_subtitle)[0]\n",
+    "        results = [(titles, section_text)]\n",
+    "        for subsection in section.get_sections(levels=[len(titles) + 1]):\n",
+    "            results.extend(all_subsections_from_section(subsection, titles, sections_to_ignore))\n",
+    "        return results\n",
+    "\n",
+    "\n",
+    "def all_subsections_from_title(\n",
+    "    title: str,\n",
+    "    sections_to_ignore: set[str] = SECTIONS_TO_IGNORE,\n",
+    "    site_name: str = WIKI_SITE,\n",
+    ") -> list[tuple[list[str], str]]:\n",
+    "    \"\"\"From a Wikipedia page title, return a flattened list of all nested subsections.\n",
+    "    Each subsection is a tuple, where:\n",
+    "        - the first element is a list of parent subtitles, starting with the page title\n",
+    "        - the second element is the text of the subsection (but not any children)\n",
+    "    \"\"\"\n",
+    "    site = mwclient.Site(site_name)\n",
+    "    page = site.pages[title]\n",
+    "    text = page.text()\n",
+    "    parsed_text = mwparserfromhell.parse(text)\n",
+    "    headings = [str(h) for h in parsed_text.filter_headings()]\n",
+    "    if headings:\n",
+    "        summary_text = str(parsed_text).split(headings[0])[0]\n",
+    "    else:\n",
+    "        summary_text = str(parsed_text)\n",
+    "    results = [([title], summary_text)]\n",
+    "    for subsection in parsed_text.get_sections(levels=[2]):\n",
+    "        results.extend(all_subsections_from_section(subsection, [title], sections_to_ignore))\n",
+    "    return results\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Found 1838 sections in 179 pages.\n"
+     ]
+    }
+   ],
+   "source": [
+    "# split pages into sections\n",
+    "# may take ~1 minute per 100 articles\n",
+    "wikipedia_sections = []\n",
+    "for title in titles:\n",
+    "    wikipedia_sections.extend(all_subsections_from_title(title))\n",
+    "print(f\"Found {len(wikipedia_sections)} sections in {len(titles)} pages.\")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Filtered out 89 sections, leaving 1749 sections.\n"
+     ]
+    }
+   ],
+   "source": [
+    "# clean text\n",
+    "def clean_section(section: tuple[list[str], str]) -> tuple[list[str], str]:\n",
+    "    \"\"\"\n",
+    "    Return a cleaned up section with:\n",
+    "        - <ref>xyz</ref> patterns removed\n",
+    "        - leading/trailing whitespace removed\n",
+    "    \"\"\"\n",
+    "    titles, text = section\n",
+    "    text = re.sub(r\"<ref.*?</ref>\", \"\", text)\n",
+    "    text = text.strip()\n",
+    "    return (titles, text)\n",
+    "\n",
+    "\n",
+    "wikipedia_sections = [clean_section(ws) for ws in wikipedia_sections]\n",
+    "\n",
+    "# filter out short/blank sections\n",
+    "def keep_section(section: tuple[list[str], str]) -> bool:\n",
+    "    \"\"\"Return True if the section should be kept, False otherwise.\"\"\"\n",
+    "    titles, text = section\n",
+    "    if len(text) < 16:\n",
+    "        return False\n",
+    "    else:\n",
+    "        return True\n",
+    "\n",
+    "\n",
+    "original_num_sections = len(wikipedia_sections)\n",
+    "wikipedia_sections = [ws for ws in wikipedia_sections if keep_section(ws)]\n",
+    "print(f\"Filtered out {original_num_sections-len(wikipedia_sections)} sections, leaving {len(wikipedia_sections)} sections.\")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "['Concerns and controversies at the 2022 Winter Olympics']\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'{{Short description|Overview of concerns and controversies surrounding the Ga...'"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "['Concerns and controversies at the 2022 Winter Olympics', '==Criticism of host selection==']\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'American sportscaster [[Bob Costas]] criticized the [[International Olympic C...'"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "['Concerns and controversies at the 2022 Winter Olympics', '==Organizing concerns and controversies==', '===Cost and climate===']\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'Several cities withdrew their applications during [[Bids for the 2022 Winter ...'"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "['Concerns and controversies at the 2022 Winter Olympics', '==Organizing concerns and controversies==', '===Promotional song===']\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'Some commentators alleged that one of the early promotional songs for the [[2...'"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "['Concerns and controversies at the 2022 Winter Olympics', '== Diplomatic boycotts or non-attendance ==']\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'<section begin=boycotts />\\n[[File:2022 Winter Olympics (Beijing) diplomatic b...'"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "# print example data\n",
+    "for ws in wikipedia_sections[:5]:\n",
+    "    print(ws[0])\n",
+    "    display(ws[1][:77] + \"...\")\n",
+    "    print()\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Next, we'll recursively split long sections into smaller sections.\n",
+    "\n",
+    "There's no perfect recipe for splitting text into sections.\n",
+    "\n",
+    "Some tradeoffs include:\n",
+    "- Longer sections may be better for questions that require more context\n",
+    "- Longer sections may be worse for retrieval, as they may have more topics muddled together\n",
+    "- Shorter sections are better for reducing costs (which are proportional to the number of tokens)\n",
+    "- Shorter sections allow more sections to be retrieved, which may help with recall\n",
+    "- Overlapping sections may help prevent answers from being cut by section boundaries\n",
+    "\n",
+    "Here, we'll use a simple approach and limit sections to 1,600 tokens each, recursively halving any sections that are too long. To avoid cutting in the middle of useful sentences, we'll split along paragraph boundaries when possible."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "GPT_MODEL = \"gpt-4o-mini\"  # only matters insofar as it selects which tokenizer to use\n",
+    "\n",
+    "\n",
+    "def num_tokens(text: str, model: str = GPT_MODEL) -> int:\n",
+    "    \"\"\"Return the number of tokens in a string.\"\"\"\n",
+    "    encoding = tiktoken.encoding_for_model(model)\n",
+    "    return len(encoding.encode(text))\n",
+    "\n",
+    "\n",
+    "def halved_by_delimiter(string: str, delimiter: str = \"\\n\") -> list[str, str]:\n",
+    "    \"\"\"Split a string in two, on a delimiter, trying to balance tokens on each side.\"\"\"\n",
+    "    chunks = string.split(delimiter)\n",
+    "    if len(chunks) == 1:\n",
+    "        return [string, \"\"]  # no delimiter found\n",
+    "    elif len(chunks) == 2:\n",
+    "        return chunks  # no need to search for halfway point\n",
+    "    else:\n",
+    "        total_tokens = num_tokens(string)\n",
+    "        halfway = total_tokens // 2\n",
+    "        best_diff = halfway\n",
+    "        for i, chunk in enumerate(chunks):\n",
+    "            left = delimiter.join(chunks[: i + 1])\n",
+    "            left_tokens = num_tokens(left)\n",
+    "            diff = abs(halfway - left_tokens)\n",
+    "            if diff >= best_diff:\n",
+    "                break\n",
+    "            else:\n",
+    "                best_diff = diff\n",
+    "        left = delimiter.join(chunks[:i])\n",
+    "        right = delimiter.join(chunks[i:])\n",
+    "        return [left, right]\n",
+    "\n",
+    "\n",
+    "def truncated_string(\n",
+    "    string: str,\n",
+    "    model: str,\n",
+    "    max_tokens: int,\n",
+    "    print_warning: bool = True,\n",
+    ") -> str:\n",
+    "    \"\"\"Truncate a string to a maximum number of tokens.\"\"\"\n",
+    "    encoding = tiktoken.encoding_for_model(model)\n",
+    "    encoded_string = encoding.encode(string)\n",
+    "    truncated_string = encoding.decode(encoded_string[:max_tokens])\n",
+    "    if print_warning and len(encoded_string) > max_tokens:\n",
+    "        print(f\"Warning: Truncated string from {len(encoded_string)} tokens to {max_tokens} tokens.\")\n",
+    "    return truncated_string\n",
+    "\n",
+    "\n",
+    "def split_strings_from_subsection(\n",
+    "    subsection: tuple[list[str], str],\n",
+    "    max_tokens: int = 1000,\n",
+    "    model: str = GPT_MODEL,\n",
+    "    max_recursion: int = 5,\n",
+    ") -> list[str]:\n",
+    "    \"\"\"\n",
+    "    Split a subsection into a list of subsections, each with no more than max_tokens.\n",
+    "    Each subsection is a tuple of parent titles [H1, H2, ...] and text (str).\n",
+    "    \"\"\"\n",
+    "    titles, text = subsection\n",
+    "    string = \"\\n\\n\".join(titles + [text])\n",
+    "    num_tokens_in_string = num_tokens(string)\n",
+    "    # if length is fine, return string\n",
+    "    if num_tokens_in_string <= max_tokens:\n",
+    "        return [string]\n",
+    "    # if recursion hasn't found a split after X iterations, just truncate\n",
+    "    elif max_recursion == 0:\n",
+    "        return [truncated_string(string, model=model, max_tokens=max_tokens)]\n",
+    "    # otherwise, split in half and recurse\n",
+    "    else:\n",
+    "        titles, text = subsection\n",
+    "        for delimiter in [\"\\n\\n\", \"\\n\", \". \"]:\n",
+    "            left, right = halved_by_delimiter(text, delimiter=delimiter)\n",
+    "            if left == \"\" or right == \"\":\n",
+    "                # if either half is empty, retry with a more fine-grained delimiter\n",
+    "                continue\n",
+    "            else:\n",
+    "                # recurse on each half\n",
+    "                results = []\n",
+    "                for half in [left, right]:\n",
+    "                    half_subsection = (titles, half)\n",
+    "                    half_strings = split_strings_from_subsection(\n",
+    "                        half_subsection,\n",
+    "                        max_tokens=max_tokens,\n",
+    "                        model=model,\n",
+    "                        max_recursion=max_recursion - 1,\n",
+    "                    )\n",
+    "                    results.extend(half_strings)\n",
+    "                return results\n",
+    "    # otherwise no split was found, so just truncate (should be very rare)\n",
+    "    return [truncated_string(string, model=model, max_tokens=max_tokens)]\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "1749 Wikipedia sections split into 2052 strings.\n"
+     ]
+    }
+   ],
+   "source": [
+    "# split sections into chunks\n",
+    "MAX_TOKENS = 1600\n",
+    "wikipedia_strings = []\n",
+    "for section in wikipedia_sections:\n",
+    "    wikipedia_strings.extend(split_strings_from_subsection(section, max_tokens=MAX_TOKENS))\n",
+    "\n",
+    "print(f\"{len(wikipedia_sections)} Wikipedia sections split into {len(wikipedia_strings)} strings.\")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Concerns and controversies at the 2022 Winter Olympics\n",
+      "\n",
+      "==Criticism of host selection==\n",
+      "\n",
+      "American sportscaster [[Bob Costas]] criticized the [[International Olympic Committee]]'s (IOC) decision to award the games to China saying \"The IOC deserves all of the disdain and disgust that comes their way for going back to China yet again\" referencing China's human rights record.\n",
+      "\n",
+      "After winning two gold medals and returning to his home country of Sweden skater [[Nils van der Poel]] criticized the IOC's selection of China as the host saying \"I think it is extremely irresponsible to give it to a country that violates human rights as blatantly as the Chinese regime is doing.\" He had declined to criticize China before leaving for the games saying \"I don't think it would be particularly wise for me to criticize the system I'm about to transition to, if I want to live a long and productive life.\"\n"
+     ]
+    }
+   ],
+   "source": [
+    "# print example data\n",
+    "print(wikipedia_strings[1])\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Embed document chunks\n",
+    "\n",
+    "Now that we've split our library into shorter self-contained strings, we can compute embeddings for each.\n",
+    "\n",
+    "(For large embedding jobs, use a script like [api_request_parallel_processor.py](https://github.com/openai/openai-cookbook/blob/main/examples/api_request_parallel_processor.py) to parallelize requests while throttling to stay under rate limits.)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Batch 0 to 999\n",
+      "Batch 1000 to 1999\n",
+      "Batch 2000 to 2999\n"
+     ]
+    }
+   ],
+   "source": [
+    "EMBEDDING_MODEL = \"text-embedding-3-small\"\n",
+    "BATCH_SIZE = 1000  # you can submit up to 2048 embedding inputs per request\n",
+    "\n",
+    "embeddings = []\n",
+    "for batch_start in range(0, len(wikipedia_strings), BATCH_SIZE):\n",
+    "    batch_end = batch_start + BATCH_SIZE\n",
+    "    batch = wikipedia_strings[batch_start:batch_end]\n",
+    "    print(f\"Batch {batch_start} to {batch_end-1}\")\n",
+    "    response = client.embeddings.create(model=EMBEDDING_MODEL, input=batch)\n",
+    "    for i, be in enumerate(response.data):\n",
+    "        assert i == be.index  # double check embeddings are in same order as input\n",
+    "    batch_embeddings = [e.embedding for e in response.data]\n",
+    "    embeddings.extend(batch_embeddings)\n",
+    "\n",
+    "df = pd.DataFrame({\"text\": wikipedia_strings, \"embedding\": embeddings})\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4. Store document chunks and embeddings\n",
+    "\n",
+    "Because this example only uses a few thousand strings, we'll store them in a CSV file.\n",
+    "\n",
+    "(For larger datasets, use a vector database, which will be more performant.)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# save document chunks and embeddings\n",
+    "\n",
+    "SAVE_PATH = \"data/winter_olympics_2022.csv\"\n",
+    "\n",
+    "df.to_csv(SAVE_PATH, index=False)\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "openai",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.6"
+  },
+  "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/Embedding_long_inputs.ipynb

@@ -0,0 +1,283 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Embedding texts that are longer than the model's maximum context length\n",
+    "\n",
+    "OpenAI's embedding models cannot embed text that exceeds a maximum length. The maximum length varies by model, and is measured by _tokens_, not string length. If you are unfamiliar with tokenization, check out [How to count tokens with tiktoken](How_to_count_tokens_with_tiktoken.ipynb).\n",
+    "\n",
+    "This notebook shows how to handle texts that are longer than a model's maximum context length. We'll demonstrate using embeddings from `text-embedding-3-small`, but the same ideas can be applied to other models and tasks. To learn more about embeddings, check out the [OpenAI Embeddings Guide](https://beta.openai.com/docs/guides/embeddings).\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Model context length\n",
+    "\n",
+    "First, we select the model and define a function to get embeddings from the API."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from openai import OpenAI\n",
+    "import os\n",
+    "import openai\n",
+    "from tenacity import retry, wait_random_exponential, stop_after_attempt, retry_if_not_exception_type\n",
+    "\n",
+    "client = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))\n",
+    "\n",
+    "EMBEDDING_MODEL = 'text-embedding-3-small'\n",
+    "EMBEDDING_CTX_LENGTH = 8191\n",
+    "EMBEDDING_ENCODING = 'cl100k_base'\n",
+    "\n",
+    "# let's make sure to not retry on an invalid request, because that is what we want to demonstrate\n",
+    "@retry(wait=wait_random_exponential(min=1, max=20), stop=stop_after_attempt(6), retry=retry_if_not_exception_type(openai.BadRequestError))\n",
+    "def get_embedding(text_or_tokens, model=EMBEDDING_MODEL):\n",
+    "    return client.embeddings.create(input=text_or_tokens, model=model).data[0].embedding"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The `text-embedding-3-small` model has a context length of 8191 tokens with the `cl100k_base` encoding, and we can see that going over that limit causes an error."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Error code: 400 - {'error': {'message': \"This model's maximum context length is 8192 tokens, however you requested 10001 tokens (10001 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.\", 'type': 'invalid_request_error', 'param': None, 'code': None}}\n"
+     ]
+    }
+   ],
+   "source": [
+    "long_text = 'AGI ' * 5000\n",
+    "try:\n",
+    "    get_embedding(long_text)\n",
+    "except openai.BadRequestError as e:\n",
+    "    print(e)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Clearly we want to avoid these errors, particularly when handling programmatically with a large number of embeddings. Yet, we still might be faced with texts that are longer than the maximum context length. Below we describe and provide recipes for the main approaches to handling these longer texts: (1) simply truncating the text to the maximum allowed length, and (2) chunking the text and embedding each chunk individually."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Truncating the input text\n",
+    "\n",
+    "The simplest solution is to truncate the input text to the maximum allowed length. Because the context length is measured in tokens, we have to first tokenize the text before truncating it. The API accepts inputs both in the form of text or tokens, so as long as you are careful that you are using the appropriate encoding, there is no need to convert the tokens back into string form. Below is an example of such a truncation function."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import tiktoken\n",
+    "\n",
+    "def truncate_text_tokens(text, encoding_name=EMBEDDING_ENCODING, max_tokens=EMBEDDING_CTX_LENGTH):\n",
+    "    \"\"\"Truncate a string to have `max_tokens` according to the given encoding.\"\"\"\n",
+    "    encoding = tiktoken.get_encoding(encoding_name)\n",
+    "    return encoding.encode(text)[:max_tokens]"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Our example from before now works without error."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "1536"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "truncated = truncate_text_tokens(long_text)\n",
+    "len(get_embedding(truncated))"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Chunking the input text\n",
+    "\n",
+    "Though truncation works, discarding potentially relevant text is a clear drawback. Another approach is to divide the input text into chunks and then embed each chunk individually. Then, we can either use the chunk embeddings separately, or combine them in some way, such as averaging (weighted by the size of each chunk).\n",
+    "\n",
+    "We will take a function from [Python's own cookbook](https://docs.python.org/3/library/itertools.html#itertools-recipes) that breaks up a sequence into chunks."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from itertools import islice\n",
+    "\n",
+    "def batched(iterable, n):\n",
+    "    \"\"\"Batch data into tuples of length n. The last batch may be shorter.\"\"\"\n",
+    "    # batched('ABCDEFG', 3) --> ABC DEF G\n",
+    "    if n < 1:\n",
+    "        raise ValueError('n must be at least one')\n",
+    "    it = iter(iterable)\n",
+    "    while (batch := tuple(islice(it, n))):\n",
+    "        yield batch"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now we define a function that encodes a string into tokens and then breaks it up into chunks."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def chunked_tokens(text, encoding_name, chunk_length):\n",
+    "    encoding = tiktoken.get_encoding(encoding_name)\n",
+    "    tokens = encoding.encode(text)\n",
+    "    chunks_iterator = batched(tokens, chunk_length)\n",
+    "    yield from chunks_iterator"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Finally, we can write a function that safely handles embedding requests, even when the input text is longer than the maximum context length, by chunking the input tokens and embedding each chunk individually. The `average` flag can be set to `True` to return the weighted average of the chunk embeddings, or `False` to simply return the unmodified list of chunk embeddings."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "\n",
+    "\n",
+    "def len_safe_get_embedding(text, model=EMBEDDING_MODEL, max_tokens=EMBEDDING_CTX_LENGTH, encoding_name=EMBEDDING_ENCODING, average=True):\n",
+    "    chunk_embeddings = []\n",
+    "    chunk_lens = []\n",
+    "    for chunk in chunked_tokens(text, encoding_name=encoding_name, chunk_length=max_tokens):\n",
+    "        chunk_embeddings.append(get_embedding(chunk, model=model))\n",
+    "        chunk_lens.append(len(chunk))\n",
+    "\n",
+    "    if average:\n",
+    "        chunk_embeddings = np.average(chunk_embeddings, axis=0, weights=chunk_lens)\n",
+    "        chunk_embeddings = chunk_embeddings / np.linalg.norm(chunk_embeddings)  # normalizes length to 1\n",
+    "        chunk_embeddings = chunk_embeddings.tolist()\n",
+    "    return chunk_embeddings"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Once again, we can now handle long input texts."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Setting average=True gives us a single 1536-dimensional embedding vector for our long text.\n",
+      "Setting average=False gives us 2 embedding vectors, one for each of the chunks.\n"
+     ]
+    }
+   ],
+   "source": [
+    "average_embedding_vector = len_safe_get_embedding(long_text, average=True)\n",
+    "chunks_embedding_vectors = len_safe_get_embedding(long_text, average=False)\n",
+    "\n",
+    "print(f\"Setting average=True gives us a single {len(average_embedding_vector)}-dimensional embedding vector for our long text.\")\n",
+    "print(f\"Setting average=False gives us {len(chunks_embedding_vectors)} embedding vectors, one for each of the chunks.\")\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In some cases, it may make sense to split chunks on paragraph boundaries or sentence boundaries to help preserve the meaning of the text."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.5"
+  },
+  "vscode": {
+   "interpreter": {
+    "hash": "365536dcbde60510dc9073d6b991cd35db2d9bac356a11f5b64279a5e6708b97"
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/Entity_extraction_for_long_documents.ipynb

@@ -0,0 +1,334 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Long Document Content Extraction\n",
+    "\n",
+    "GPT-3 can help us extract key figures, dates or other bits of important content from documents that are too big to fit into the context window. One approach for solving this is to chunk the document up and process each chunk separately, before combining into one list of answers. \n",
+    "\n",
+    "In this notebook we'll run through this approach:\n",
+    "- Load in a long PDF and pull the text out\n",
+    "- Create a prompt to be used to extract key bits of information\n",
+    "- Chunk up our document and process each chunk to pull any answers out\n",
+    "- Combine them at the end\n",
+    "- This simple approach will then be extended to three more difficult questions\n",
+    "\n",
+    "## Approach\n",
+    "\n",
+    "- **Setup**: Take a PDF, a Formula 1 Financial Regulation document on Power Units, and extract the text from it for entity extraction. We'll use this to try to extract answers that are buried in the content.\n",
+    "- **Simple Entity Extraction**: Extract key bits of information from chunks of a document by:\n",
+    "    - Creating a template prompt with our questions and an example of the format it expects\n",
+    "    - Create a function to take a chunk of text as input, combine with the prompt and get a response\n",
+    "    - Run a script to chunk the text, extract answers and output them for parsing\n",
+    "- **Complex Entity Extraction**: Ask some more difficult questions which require tougher reasoning to work out"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Setup"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install textract\n",
+    "!pip install tiktoken"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import textract\n",
+    "import os\n",
+    "import openai\n",
+    "import tiktoken\n",
+    "\n",
+    "client = openai.OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))\n",
+    "\n",
+    "# Extract the raw text from each PDF using textract\n",
+    "text = textract.process('data/fia_f1_power_unit_financial_regulations_issue_1_-_2022-08-16.pdf', method='pdfminer').decode('utf-8')\n",
+    "clean_text = text.replace(\"  \", \" \").replace(\"\\n\", \"; \").replace(';',' ')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Simple Entity Extraction"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Extract key pieces of information from this regulation document.\n",
+      "If a particular piece of information is not present, output \"Not specified\".\n",
+      "When you extract a key piece of information, include the closest page number.\n",
+      "Use the following format:\n",
+      "0. Who is the author\n",
+      "1. What is the amount of the \"Power Unit Cost Cap\" in USD, GBP and EUR\n",
+      "2. What is the value of External Manufacturing Costs in USD\n",
+      "3. What is the Capital Expenditure Limit in USD\n",
+      "\n",
+      "Document: \"\"\"<document>\"\"\"\n",
+      "\n",
+      "0. Who is the author: Tom Anderson (Page 1)\n",
+      "1.\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Example prompt - \n",
+    "document = '<document>'\n",
+    "template_prompt=f'''Extract key pieces of information from this regulation document.\n",
+    "If a particular piece of information is not present, output \\\"Not specified\\\".\n",
+    "When you extract a key piece of information, include the closest page number.\n",
+    "Use the following format:\\n0. Who is the author\\n1. What is the amount of the \"Power Unit Cost Cap\" in USD, GBP and EUR\\n2. What is the value of External Manufacturing Costs in USD\\n3. What is the Capital Expenditure Limit in USD\\n\\nDocument: \\\"\\\"\\\"<document>\\\"\\\"\\\"\\n\\n0. Who is the author: Tom Anderson (Page 1)\\n1.'''\n",
+    "print(template_prompt)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Split a text into smaller chunks of size n, preferably ending at the end of a sentence\n",
+    "def create_chunks(text, n, tokenizer):\n",
+    "    tokens = tokenizer.encode(text)\n",
+    "    \"\"\"Yield successive n-sized chunks from text.\"\"\"\n",
+    "    i = 0\n",
+    "    while i < len(tokens):\n",
+    "        # Find the nearest end of sentence within a range of 0.5 * n and 1.5 * n tokens\n",
+    "        j = min(i + int(1.5 * n), len(tokens))\n",
+    "        while j > i + int(0.5 * n):\n",
+    "            # Decode the tokens and check for full stop or newline\n",
+    "            chunk = tokenizer.decode(tokens[i:j])\n",
+    "            if chunk.endswith(\".\") or chunk.endswith(\"\\n\"):\n",
+    "                break\n",
+    "            j -= 1\n",
+    "        # If no end of sentence found, use n tokens as the chunk size\n",
+    "        if j == i + int(0.5 * n):\n",
+    "            j = min(i + n, len(tokens))\n",
+    "        yield tokens[i:j]\n",
+    "        i = j\n",
+    "\n",
+    "def extract_chunk(document,template_prompt):\n",
+    "    prompt = template_prompt.replace('<document>',document)\n",
+    "\n",
+    "    messages = [\n",
+    "            {\"role\": \"system\", \"content\": \"You help extract information from documents.\"},\n",
+    "            {\"role\": \"user\", \"content\": prompt}\n",
+    "            ]\n",
+    "\n",
+    "    response = client.chat.completions.create(\n",
+    "            model='gpt-4', \n",
+    "            messages=messages,\n",
+    "            temperature=0,\n",
+    "            max_tokens=1500,\n",
+    "            top_p=1,\n",
+    "            frequency_penalty=0,\n",
+    "            presence_penalty=0\n",
+    "        )\n",
+    "    return \"1.\" + response.choices[0].message.content"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Initialise tokenizer\n",
+    "tokenizer = tiktoken.get_encoding(\"cl100k_base\")\n",
+    "\n",
+    "results = []\n",
+    "    \n",
+    "chunks = create_chunks(clean_text,1000,tokenizer)\n",
+    "text_chunks = [tokenizer.decode(chunk) for chunk in chunks]\n",
+    "\n",
+    "for chunk in text_chunks:\n",
+    "    results.append(extract_chunk(chunk,template_prompt))\n",
+    "    #print(chunk)\n",
+    "    print(results[-1])\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 31,
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "['1. What is the amount of the \"Power Unit Cost Cap\" in USD, GBP and EUR: USD 95,000,000 (Page 2); GBP 76,459,000 (Page 2); EUR 90,210,000 (Page 2)',\n",
+       " '2. What is the value of External Manufacturing Costs in USD: US Dollars 20,000,000 in respect of each of the Full Year Reporting Periods ending on 31 December 2023, 31 December 2024 and 31 December 2025, adjusted for Indexation (Page 10)',\n",
+       " '3. What is the Capital Expenditure Limit in USD: US Dollars 30,000,000 (Page 32)']"
+      ]
+     },
+     "execution_count": 31,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "groups = [r.split('\\n') for r in results]\n",
+    "\n",
+    "# zip the groups together\n",
+    "zipped = list(zip(*groups))\n",
+    "zipped = [x for y in zipped for x in y if \"Not specified\" not in x and \"__\" not in x]\n",
+    "zipped"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Complex Entity Extraction"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 32,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Extract key pieces of information from this regulation document.\n",
+      "If a particular piece of information is not present, output \"Not specified\".\n",
+      "When you extract a key piece of information, include the closest page number.\n",
+      "Use the following format:\n",
+      "0. Who is the author\n",
+      "1. How is a Minor Overspend Breach calculated\n",
+      "2. How is a Major Overspend Breach calculated\n",
+      "3. Which years do these financial regulations apply to\n",
+      "\n",
+      "Document: \"\"\"<document>\"\"\"\n",
+      "\n",
+      "0. Who is the author: Tom Anderson (Page 1)\n",
+      "1.\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Example prompt - \n",
+    "template_prompt=f'''Extract key pieces of information from this regulation document.\n",
+    "If a particular piece of information is not present, output \\\"Not specified\\\".\n",
+    "When you extract a key piece of information, include the closest page number.\n",
+    "Use the following format:\\n0. Who is the author\\n1. How is a Minor Overspend Breach calculated\\n2. How is a Major Overspend Breach calculated\\n3. Which years do these financial regulations apply to\\n\\nDocument: \\\"\\\"\\\"<document>\\\"\\\"\\\"\\n\\n0. Who is the author: Tom Anderson (Page 1)\\n1.'''\n",
+    "print(template_prompt)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 34,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "['1. How is a Minor Overspend Breach calculated: A Minor Overspend Breach arises when a Power Unit Manufacturer submits its Full Year Reporting Documentation and Relevant Costs reported therein exceed the Power Unit Cost Cap by less than 5% (Page 24)',\n",
+       " '2. How is a Major Overspend Breach calculated: A Material Overspend Breach arises when a Power Unit Manufacturer submits its Full Year Reporting Documentation and Relevant Costs reported therein exceed the Power Unit Cost Cap by 5% or more (Page 25)',\n",
+       " '3. Which years do these financial regulations apply to: 2026 onwards (Page 1)',\n",
+       " '3. Which years do these financial regulations apply to: 2023, 2024, 2025, 2026 and subsequent Full Year Reporting Periods (Page 2)',\n",
+       " '3. Which years do these financial regulations apply to: 2022-2025 (Page 6)',\n",
+       " '3. Which years do these financial regulations apply to: 2023, 2024, 2025, 2026 and subsequent Full Year Reporting Periods (Page 10)',\n",
+       " '3. Which years do these financial regulations apply to: 2022 (Page 14)',\n",
+       " '3. Which years do these financial regulations apply to: 2022 (Page 16)',\n",
+       " '3. Which years do these financial regulations apply to: 2022 (Page 19)',\n",
+       " '3. Which years do these financial regulations apply to: 2022 (Page 21)',\n",
+       " '3. Which years do these financial regulations apply to: 2026 onwards (Page 26)',\n",
+       " '3. Which years do these financial regulations apply to: 2026 (Page 2)',\n",
+       " '3. Which years do these financial regulations apply to: 2022 (Page 30)',\n",
+       " '3. Which years do these financial regulations apply to: 2022 (Page 32)',\n",
+       " '3. Which years do these financial regulations apply to: 2023, 2024 and 2025 (Page 1)',\n",
+       " '3. Which years do these financial regulations apply to: 2022 (Page 37)',\n",
+       " '3. Which years do these financial regulations apply to: 2026 onwards (Page 40)',\n",
+       " '3. Which years do these financial regulations apply to: 2022 (Page 1)',\n",
+       " '3. Which years do these financial regulations apply to: 2026 to 2030 seasons (Page 46)',\n",
+       " '3. Which years do these financial regulations apply to: 2022 (Page 47)',\n",
+       " '3. Which years do these financial regulations apply to: 2022 (Page 1)',\n",
+       " '3. Which years do these financial regulations apply to: 2022 (Page 1)',\n",
+       " '3. Which years do these financial regulations apply to: 2022 (Page 56)',\n",
+       " '3. Which years do these financial regulations apply to: 2022 (Page 1)',\n",
+       " '3. Which years do these financial regulations apply to: 2022 (Page 16)',\n",
+       " '3. Which years do these financial regulations apply to: 2022 (Page 16)']"
+      ]
+     },
+     "execution_count": 34,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "results = []\n",
+    "\n",
+    "for chunk in text_chunks:\n",
+    "    results.append(extract_chunk(chunk,template_prompt))\n",
+    "    \n",
+    "groups = [r.split('\\n') for r in results]\n",
+    "\n",
+    "# zip the groups together\n",
+    "zipped = list(zip(*groups))\n",
+    "zipped = [x for y in zipped for x in y if \"Not specified\" not in x and \"__\" not in x]\n",
+    "zipped"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Consolidation\n",
+    "\n",
+    "We've been able to extract the first two answers safely, while the third was confounded by the date that appeared on every page, though the correct answer is in there as well.\n",
+    "\n",
+    "To tune this further you can consider experimenting with:\n",
+    "- A more descriptive or specific prompt\n",
+    "- If you have sufficient training data, fine-tuning a model to find a set of outputs very well\n",
+    "- The way you chunk your data - we have gone for 1000 tokens with no overlap, but more intelligent chunking that breaks info into sections, cuts by tokens or similar may get better results\n",
+    "\n",
+    "However, with minimal tuning we have now answered 6 questions of varying difficulty using the contents of a long document, and have a reusable approach that we can apply to any long document requiring entity extraction. Look forward to seeing what you can do with this!"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/File_Search_Responses.ipynb

@@ -0,0 +1,688 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "2dfbaf53-32de-4b8c-bd1c-d27371a87f81",
+   "metadata": {},
+   "source": [
+    "# Using file search tool in the Responses API\n",
+    "\n",
+    "Although RAG can be overwhelming, searching amongst PDF file shouldn't be complicated. One of the most adopted options as of now is parsing your PDF, defining your chunking strategies, uploading those chunks to a storage provider, running embeddings on those chunks of texts and storing those embeddings in a vector database. And that's only the setup — retrieving content in our LLM workflow also requires multiple steps.\n",
+    "\n",
+    "This is where file search — a hosted tool you can use in the Responses API — comes in. It allows you to search your knowledge base and generate an answer based on the retrieved content. In this cookbook, we'll upload those PDFs to a vector store on OpenAI and use file search to fetch additional context from this vector store to answer the questions we generated in the first step. Then, we'll initially create a small set of questions based on PDFs extracted from OpenAI's blog ([openai.com/news](https://openai.com/news)).\n",
+    "\n",
+    "_File search was previously available on the Assistants API. It's now available on the new Responses API, an API that can be stateful or stateless, and with from new features like metadata filtering_\n",
+    "\n",
+    "# Creating Vector Store with our PDFs"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "47480955-9dd4-4837-8b4c-6821bb48306b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install PyPDF2 pandas tqdm openai -q"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "f6feaf3e-a2be-4c74-bad5-0c37bbe110b5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from openai import OpenAI\n",
+    "from concurrent.futures import ThreadPoolExecutor\n",
+    "from tqdm import tqdm\n",
+    "import concurrent\n",
+    "import PyPDF2\n",
+    "import os\n",
+    "import pandas as pd\n",
+    "import base64\n",
+    "\n",
+    "client = OpenAI(api_key=os.getenv('OPENAI_API_KEY'))\n",
+    "dir_pdfs = 'openai_blog_pdfs' # have those PDFs stored locally here\n",
+    "pdf_files = [os.path.join(dir_pdfs, f) for f in os.listdir(dir_pdfs)]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "43e5cb9c-fc99-45e2-bd79-9c9ba5b410cc",
+   "metadata": {},
+   "source": [
+    "We will create a Vector Store on OpenAI API and upload our PDFs to the Vector Store. OpenAI will read those PDFs, separate the content into multiple chunks of text, run embeddings on those and store those embeddings and the text in the Vector Store. It will enable us to query this Vector Store to return relevant content based on a query."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "a6823030-9110-4143-ab7c-a223182eb7e0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def upload_single_pdf(file_path: str, vector_store_id: str):\n",
+    "    file_name = os.path.basename(file_path)\n",
+    "    try:\n",
+    "        file_response = client.files.create(file=open(file_path, 'rb'), purpose=\"assistants\")\n",
+    "        attach_response = client.vector_stores.files.create(\n",
+    "            vector_store_id=vector_store_id,\n",
+    "            file_id=file_response.id\n",
+    "        )\n",
+    "        return {\"file\": file_name, \"status\": \"success\"}\n",
+    "    except Exception as e:\n",
+    "        print(f\"Error with {file_name}: {str(e)}\")\n",
+    "        return {\"file\": file_name, \"status\": \"failed\", \"error\": str(e)}\n",
+    "\n",
+    "def upload_pdf_files_to_vector_store(vector_store_id: str):\n",
+    "    pdf_files = [os.path.join(dir_pdfs, f) for f in os.listdir(dir_pdfs)]\n",
+    "    stats = {\"total_files\": len(pdf_files), \"successful_uploads\": 0, \"failed_uploads\": 0, \"errors\": []}\n",
+    "    \n",
+    "    print(f\"{len(pdf_files)} PDF files to process. Uploading in parallel...\")\n",
+    "\n",
+    "    with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:\n",
+    "        futures = {executor.submit(upload_single_pdf, file_path, vector_store_id): file_path for file_path in pdf_files}\n",
+    "        for future in tqdm(concurrent.futures.as_completed(futures), total=len(pdf_files)):\n",
+    "            result = future.result()\n",
+    "            if result[\"status\"] == \"success\":\n",
+    "                stats[\"successful_uploads\"] += 1\n",
+    "            else:\n",
+    "                stats[\"failed_uploads\"] += 1\n",
+    "                stats[\"errors\"].append(result)\n",
+    "\n",
+    "    return stats\n",
+    "\n",
+    "def create_vector_store(store_name: str) -> dict:\n",
+    "    try:\n",
+    "        vector_store = client.vector_stores.create(name=store_name)\n",
+    "        details = {\n",
+    "            \"id\": vector_store.id,\n",
+    "            \"name\": vector_store.name,\n",
+    "            \"created_at\": vector_store.created_at,\n",
+    "            \"file_count\": vector_store.file_counts.completed\n",
+    "        }\n",
+    "        print(\"Vector store created:\", details)\n",
+    "        return details\n",
+    "    except Exception as e:\n",
+    "        print(f\"Error creating vector store: {e}\")\n",
+    "        return {}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "5cb6cba0-931e-426a-88aa-34a62cc7158c",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Vector store created: {'id': 'vs_67d06b9b9a9c8191bafd456cf2364ce3', 'name': 'openai_blog_store', 'created_at': 1741712283, 'file_count': 0}\n",
+      "21 PDF files to process. Uploading in parallel...\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "100%|███████████████████████████████| 21/21 [00:09<00:00,  2.32it/s]\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "{'total_files': 21,\n",
+       " 'successful_uploads': 21,\n",
+       " 'failed_uploads': 0,\n",
+       " 'errors': []}"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "store_name = \"openai_blog_store\"\n",
+    "vector_store_details = create_vector_store(store_name)\n",
+    "upload_pdf_files_to_vector_store(vector_store_details[\"id\"])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e5f4ade3-2b3e-4df6-a441-c1ee3ea73172",
+   "metadata": {},
+   "source": [
+    "# Standalone vector search\n",
+    "\n",
+    "Now that our vector store is ready, we are able to query the Vector Store directly and retrieve relevant content for a specific query. Using the new [vector search API](https://platform.openai.com/docs/api-reference/vector-stores/search), we're able to find relevant items from our knowledge base without necessarily integrating it in an LLM query."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "980323d0-0112-4c9e-9b90-67719739026f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "query = \"What's Deep Research?\"\n",
+    "search_results = client.vector_stores.search(\n",
+    "    vector_store_id=vector_store_details['id'],\n",
+    "    query=query\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "c6045a2e-a75f-48c0-89f4-841ef722d24f",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "3502 of character of content from Introducing deep research _ OpenAI.pdf with a relevant score of 0.9813588865322393\n",
+      "3493 of character of content from Introducing deep research _ OpenAI.pdf with a relevant score of 0.9522476825143714\n",
+      "3634 of character of content from Introducing deep research _ OpenAI.pdf with a relevant score of 0.9397930296526796\n",
+      "2774 of character of content from Introducing deep research _ OpenAI.pdf with a relevant score of 0.9101975747303771\n",
+      "3474 of character of content from Deep research System Card _ OpenAI.pdf with a relevant score of 0.9036647613464299\n",
+      "3123 of character of content from Introducing deep research _ OpenAI.pdf with a relevant score of 0.887120981288272\n",
+      "3343 of character of content from Introducing deep research _ OpenAI.pdf with a relevant score of 0.8448454849432881\n",
+      "3262 of character of content from Introducing deep research _ OpenAI.pdf with a relevant score of 0.791345286655509\n",
+      "3271 of character of content from Introducing deep research _ OpenAI.pdf with a relevant score of 0.7485530025091963\n",
+      "2721 of character of content from Introducing deep research _ OpenAI.pdf with a relevant score of 0.734033360849088\n"
+     ]
+    }
+   ],
+   "source": [
+    "for result in search_results.data:\n",
+    "    print(str(len(result.content[0].text)) + ' of character of content from ' + result.filename + ' with a relevant score of ' + str(result.score))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d4b0b4ec-ea13-429a-a1b7-7bac3d2ea014",
+   "metadata": {},
+   "source": [
+    "We can see that different size (and under-the-hood different texts) have been returned from the search query. They all have different relevancy score that are calculated by our ranker which uses hybrid search.\n",
+    "\n",
+    "# Integrating search results with LLM in a single API call\n",
+    "\n",
+    "However instead of querying the vector store and then passing the data into the Responses or Chat Completion API call, an even more convenient way to use this search results in an LLM query would be to plug use file_search tool as part of OpenAI Responses API."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "a153cb6e-e94b-4b55-a557-4f34fd3022bd",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Files used: {'Introducing deep research _ OpenAI.pdf'}\n",
+      "Response:\n",
+      "Deep Research is a new capability introduced by OpenAI that allows users to conduct complex, multi-step research tasks on the internet efficiently. Key features include:\n",
+      "\n",
+      "1. **Autonomous Research**: Deep Research acts as an independent agent that synthesizes vast amounts of information across the web, enabling users to receive comprehensive reports similar to those produced by a research analyst.\n",
+      "\n",
+      "2. **Multi-Step Reasoning**: It performs deep analysis by finding, interpreting, and synthesizing data from various sources, including text, images, and PDFs.\n",
+      "\n",
+      "3. **Application Areas**: Especially useful for professionals in fields such as finance, science, policy, and engineering, as well as for consumers seeking detailed information for purchases.\n",
+      "\n",
+      "4. **Efficiency**: The output is fully documented with citations, making it easy to verify information, and it significantly speeds up research processes that would otherwise take hours for a human to complete.\n",
+      "\n",
+      "5. **Limitations**: While Deep Research enhances research capabilities, it is still subject to limitations, such as potential inaccuracies in information retrieval and challenges in distinguishing authoritative data from unreliable sources.\n",
+      "\n",
+      "Overall, Deep Research marks a significant advancement toward automated general intelligence (AGI) by improving access to thorough and precise research outputs.\n"
+     ]
+    }
+   ],
+   "source": [
+    "query = \"What's Deep Research?\"\n",
+    "response = client.responses.create(\n",
+    "    input= query,\n",
+    "    model=\"gpt-4o-mini\",\n",
+    "    tools=[{\n",
+    "        \"type\": \"file_search\",\n",
+    "        \"vector_store_ids\": [vector_store_details['id']],\n",
+    "    }]\n",
+    ")\n",
+    "\n",
+    "# Extract annotations from the response\n",
+    "annotations = response.output[1].content[0].annotations\n",
+    "    \n",
+    "# Get top-k retrieved filenames\n",
+    "retrieved_files = set([result.filename for result in annotations])\n",
+    "\n",
+    "print(f'Files used: {retrieved_files}')\n",
+    "print('Response:')\n",
+    "print(response.output[1].content[0].text) # 0 being the filesearch call"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e6c7b7b3-7d63-4630-95e7-76cf8080477e",
+   "metadata": {},
+   "source": [
+    "We can see that `gpt-4o-mini` was able to answer a query that required more recent, specialised knowledge about OpenAI's Deep Research. It used content from the file `Introducing deep research _ OpenAI.pdf` that had chunks of texts that were the most relevant. If we want to go even deeper in the analysis of chunk of text retrieved, we can also analyse the different texts that were returned by the search engine by adding `include=[\"output[*].file_search_call.search_results\"]` to our query.\n",
+    "\n",
+    "# Evaluating performance\n",
+    "\n",
+    "What is key for those information retrieval system is to also measure the relevance & quality of files retrieved for those answers. The following steps of this cookbook will consist in generating an evaluation dataset and calculating different metrics over this generated dataset. This is an imperfect approach and we'll always recommend to have a human-verified evaluation dataset for your own use-cases, but it will show you the methodology to evaluate those.  It will be imperfect because some of the questions generated might be generic (e.g: What's said by the main stakeholder in this document) and our retrieval test will have a hard time to figure out which document that question was generated for."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "93291578-d04a-4e71-8ecb-9f0f647e68c3",
+   "metadata": {},
+   "source": [
+    "## Generating evaluations\n",
+    "\n",
+    "We will create functions that will read through the PDFs we have locally and generate a question that can only be answered by this document. Therefore it'll create our evaluation dataset that we can use after."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "2a1274ce-a468-489a-9206-0ff6ba82e8e7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def extract_text_from_pdf(pdf_path):\n",
+    "    text = \"\"\n",
+    "    try:\n",
+    "        with open(pdf_path, \"rb\") as f:\n",
+    "            reader = PyPDF2.PdfReader(f)\n",
+    "            for page in reader.pages:\n",
+    "                page_text = page.extract_text()\n",
+    "                if page_text:\n",
+    "                    text += page_text\n",
+    "    except Exception as e:\n",
+    "        print(f\"Error reading {pdf_path}: {e}\")\n",
+    "    return text\n",
+    "\n",
+    "def generate_questions(pdf_path):\n",
+    "    text = extract_text_from_pdf(pdf_path)\n",
+    "\n",
+    "    prompt = (\n",
+    "        \"Can you generate a question that can only be answered from this document?:\\n\"\n",
+    "        f\"{text}\\n\\n\"\n",
+    "    )\n",
+    "\n",
+    "    response = client.responses.create(\n",
+    "        input=prompt,\n",
+    "        model=\"gpt-4o\",\n",
+    "    )\n",
+    "\n",
+    "    question = response.output[0].content[0].text\n",
+    "\n",
+    "    return question"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7850d17f-832f-4a03-8216-5200d2db6b17",
+   "metadata": {},
+   "source": [
+    "If we run the function generate_question for the first PDF file we will be able to see the kind of question it generates."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "4d19e4f5-a193-4787-aad1-8547173d36f4",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'What new capabilities will ChatGPT have as a result of the partnership between OpenAI and Schibsted Media Group?'"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "generate_questions(pdf_files[0])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "dc2e4e26-3396-4a3b-83a9-db9ae1597e41",
+   "metadata": {},
+   "source": [
+    "We can now generate all the questions for all the PDFs we've got stored locally."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "0fec6e6c-13b6-4498-b49c-d20e28b39ce9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Generate questions for each PDF and store in a dictionary\n",
+    "questions_dict = {}\n",
+    "for pdf_path in pdf_files:\n",
+    "    questions = generate_questions(pdf_path)\n",
+    "    questions_dict[os.path.basename(pdf_path)] = questions"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "2e04371b-32ef-48f9-833a-84f53b7399fa",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'OpenAI partners with Schibsted Media Group _ OpenAI.pdf': 'What is the purpose of the partnership between Schibsted Media Group and OpenAI announced on February 10, 2025?',\n",
+       " 'OpenAI and the CSU system bring AI to 500,000 students & faculty _ OpenAI.pdf': 'What significant milestone did the California State University system achieve by partnering with OpenAI, making it the first of its kind in the United States?',\n",
+       " '1,000 Scientist AI Jam Session _ OpenAI.pdf': 'What was the specific AI model used during the \"1,000 Scientist AI Jam Session\" event across the nine national labs?',\n",
+       " 'Announcing The Stargate Project _ OpenAI.pdf': 'What are the initial equity funders and lead partners in The Stargate Project announced by OpenAI, and who holds the financial and operational responsibilities?',\n",
+       " 'Introducing Operator _ OpenAI.pdf': 'What is the name of the new model that powers the Operator agent introduced by OpenAI?',\n",
+       " 'Introducing NextGenAI _ OpenAI.pdf': 'What major initiative did OpenAI launch on March 4, 2025, and which research institution from Europe is involved as a founding partner?',\n",
+       " 'Introducing the Intelligence Age _ OpenAI.pdf': \"What is the name of the video generation tool used by OpenAI's creative team to help produce their Super Bowl ad?\",\n",
+       " 'Operator System Card _ OpenAI.pdf': 'What is the preparedness score for the \"Cybersecurity\" category according to the Operator System Card?',\n",
+       " 'Strengthening America’s AI leadership with the U.S. National Laboratories _ OpenAI.pdf': \"What is the purpose of OpenAI's agreement with the U.S. National Laboratories as described in the document?\",\n",
+       " 'OpenAI GPT-4.5 System Card _ OpenAI.pdf': 'What is the Preparedness Framework rating for \"Cybersecurity\" for GPT-4.5 according to the system card?',\n",
+       " 'Partnering with Axios expands OpenAI’s work with the news industry _ OpenAI.pdf': \"What is the goal of OpenAI's new content partnership with Axios as announced in the document?\",\n",
+       " 'OpenAI and Guardian Media Group launch content partnership _ OpenAI.pdf': 'What is the main purpose of the partnership between OpenAI and Guardian Media Group announced on February 14, 2025?',\n",
+       " 'Introducing GPT-4.5 _ OpenAI.pdf': 'What is the release date of the GPT-4.5 research preview?',\n",
+       " 'Introducing data residency in Europe _ OpenAI.pdf': 'What are the benefits of data residency in Europe for new ChatGPT Enterprise and Edu customers according to the document?',\n",
+       " 'The power of personalized AI _ OpenAI.pdf': 'What is the purpose of the \"Model Spec\" document published by OpenAI for ChatGPT?',\n",
+       " 'Disrupting malicious uses of AI _ OpenAI.pdf': \"What is OpenAI's mission as stated in the document?\",\n",
+       " 'Sharing the latest Model Spec _ OpenAI.pdf': 'What is the release date of the latest Model Spec mentioned in the document?',\n",
+       " 'Deep research System Card _ OpenAI.pdf': \"What specific publication date is mentioned in the Deep Research System Card for when the report on deep research's preparedness was released?\",\n",
+       " 'Bertelsmann powers creativity and productivity with OpenAI _ OpenAI.pdf': 'What specific AI-powered solutions is Bertelsmann planning to implement for its divisions RTL Deutschland and Penguin Random House according to the document?',\n",
+       " 'OpenAI’s Economic Blueprint _ OpenAI.pdf': 'What date and location is scheduled for the kickoff event of OpenAI\\'s \"Innovating for America\" initiative as mentioned in the Economic Blueprint document?',\n",
+       " 'Introducing deep research _ OpenAI.pdf': 'What specific model powers the \"deep research\" capability in ChatGPT that is discussed in this document, and what are its main features designed for?'}"
+      ]
+     },
+     "execution_count": 11,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "questions_dict"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "eea9bd1b-f746-4442-9f1b-aa31b5c766c6",
+   "metadata": {},
+   "source": [
+    "We now have a dictionary of `filename:question` that we can loop through and ask gpt-4o(-mini) about without providing the document, and gpt-4o should be able to find the relevant document in the Vector Store."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "dbda554b-c3d4-4b07-9028-b41670c2fa20",
+   "metadata": {},
+   "source": [
+    "## Evaluating\n",
+    "\n",
+    "We'll convert our dictionary into a dataframe and process it using gpt-4o-mini. We will look out for the expected file "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "968d54af-55c0-4b21-9ed8-c57811f9700f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "rows = []\n",
+    "for filename, query in questions_dict.items():\n",
+    "    rows.append({\"query\": query, \"_id\": filename.replace(\".pdf\", \"\")})\n",
+    "\n",
+    "# Metrics evaluation parameters\n",
+    "k = 5\n",
+    "total_queries = len(rows)\n",
+    "correct_retrievals_at_k = 0\n",
+    "reciprocal_ranks = []\n",
+    "average_precisions = []\n",
+    "\n",
+    "def process_query(row):\n",
+    "    query = row['query']\n",
+    "    expected_filename = row['_id'] + '.pdf'\n",
+    "    # Call file_search via Responses API\n",
+    "    response = client.responses.create(\n",
+    "        input=query,\n",
+    "        model=\"gpt-4o-mini\",\n",
+    "        tools=[{\n",
+    "            \"type\": \"file_search\",\n",
+    "            \"vector_store_ids\": [vector_store_details['id']],\n",
+    "            \"max_num_results\": k,\n",
+    "        }],\n",
+    "        tool_choice=\"required\" # it will force the file_search, while not necessary, it's better to enforce it as this is what we're testing\n",
+    "    )\n",
+    "    # Extract annotations from the response\n",
+    "    annotations = None\n",
+    "    if hasattr(response.output[1], 'content') and response.output[1].content:\n",
+    "        annotations = response.output[1].content[0].annotations\n",
+    "    elif hasattr(response.output[1], 'annotations'):\n",
+    "        annotations = response.output[1].annotations\n",
+    "\n",
+    "    if annotations is None:\n",
+    "        print(f\"No annotations for query: {query}\")\n",
+    "        return False, 0, 0\n",
+    "\n",
+    "    # Get top-k retrieved filenames\n",
+    "    retrieved_files = [result.filename for result in annotations[:k]]\n",
+    "    if expected_filename in retrieved_files:\n",
+    "        rank = retrieved_files.index(expected_filename) + 1\n",
+    "        rr = 1 / rank\n",
+    "        correct = True\n",
+    "    else:\n",
+    "        rr = 0\n",
+    "        correct = False\n",
+    "\n",
+    "    # Calculate Average Precision\n",
+    "    precisions = []\n",
+    "    num_relevant = 0\n",
+    "    for i, fname in enumerate(retrieved_files):\n",
+    "        if fname == expected_filename:\n",
+    "            num_relevant += 1\n",
+    "            precisions.append(num_relevant / (i + 1))\n",
+    "    avg_precision = sum(precisions) / len(precisions) if precisions else 0\n",
+    "    \n",
+    "    if expected_filename not in retrieved_files:\n",
+    "        print(\"Expected file NOT found in the retrieved files!\")\n",
+    "        \n",
+    "    if retrieved_files and retrieved_files[0] != expected_filename:\n",
+    "        print(f\"Query: {query}\")\n",
+    "        print(f\"Expected file: {expected_filename}\")\n",
+    "        print(f\"First retrieved file: {retrieved_files[0]}\")\n",
+    "        print(f\"Retrieved files: {retrieved_files}\")\n",
+    "        print(\"-\" * 50)\n",
+    "    \n",
+    "    \n",
+    "    return correct, rr, avg_precision"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "ee6d3084-5fae-4a26-8fd2-d269ffbc60ee",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "(True, 1.0, 1.0)"
+      ]
+     },
+     "execution_count": 13,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "process_query(rows[0])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ba088faf-2945-48b3-a3de-412da1ee81fc",
+   "metadata": {},
+   "source": [
+    "Recall & Precision are at 1 for this example, and our file ranked first so we're having a MRR and MAP = 1 on this example.\n",
+    "\n",
+    "We can now execute this processing on our set of questions."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "6f1e1cc2-0128-48cc-9e4c-5eb416c21347",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      " 62%|███████████████████▏           | 13/21 [00:07<00:03,  2.57it/s]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Expected file NOT found in the retrieved files!\n",
+      "Query: What is OpenAI's mission as stated in the document?\n",
+      "Expected file: Disrupting malicious uses of AI _ OpenAI.pdf\n",
+      "First retrieved file: Introducing the Intelligence Age _ OpenAI.pdf\n",
+      "Retrieved files: ['Introducing the Intelligence Age _ OpenAI.pdf']\n",
+      "--------------------------------------------------\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      " 71%|██████████████████████▏        | 15/21 [00:14<00:06,  1.04s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Expected file NOT found in the retrieved files!\n",
+      "Query: What is the purpose of the \"Model Spec\" document published by OpenAI for ChatGPT?\n",
+      "Expected file: The power of personalized AI _ OpenAI.pdf\n",
+      "First retrieved file: Sharing the latest Model Spec _ OpenAI.pdf\n",
+      "Retrieved files: ['Sharing the latest Model Spec _ OpenAI.pdf', 'Sharing the latest Model Spec _ OpenAI.pdf', 'Sharing the latest Model Spec _ OpenAI.pdf', 'Sharing the latest Model Spec _ OpenAI.pdf', 'Sharing the latest Model Spec _ OpenAI.pdf']\n",
+      "--------------------------------------------------\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "100%|███████████████████████████████| 21/21 [00:15<00:00,  1.38it/s]\n"
+     ]
+    }
+   ],
+   "source": [
+    "with ThreadPoolExecutor() as executor:\n",
+    "    results = list(tqdm(executor.map(process_query, rows), total=total_queries))\n",
+    "\n",
+    "correct_retrievals_at_k = 0\n",
+    "reciprocal_ranks = []\n",
+    "average_precisions = []\n",
+    "\n",
+    "for correct, rr, avg_precision in results:\n",
+    "    if correct:\n",
+    "        correct_retrievals_at_k += 1\n",
+    "    reciprocal_ranks.append(rr)\n",
+    "    average_precisions.append(avg_precision)\n",
+    "\n",
+    "recall_at_k = correct_retrievals_at_k / total_queries\n",
+    "precision_at_k = recall_at_k  # In this context, same as recall\n",
+    "mrr = sum(reciprocal_ranks) / total_queries\n",
+    "map_score = sum(average_precisions) / total_queries"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6bc74d02-7ee9-4cc3-b48f-5c205c3fdfcb",
+   "metadata": {},
+   "source": [
+    "The outputs logged above would either show that a file wasn't ranked first when our evaluation dataset expected it to rank first or that it wasn't found at all. As we can see from our imperfect evaluation dataset, some questions were generic and expected another doc, which our retrieval system didn't specifically retrieved for this question."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "id": "7a32ec63-8f39-4085-b123-f2593eb702d3",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Metrics at k=5:\n",
+      "Recall@5: 0.9048\n",
+      "Precision@5: 0.9048\n",
+      "Mean Reciprocal Rank (MRR): 0.9048\n",
+      "Mean Average Precision (MAP): 0.8954\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Print the metrics with k\n",
+    "print(f\"Metrics at k={k}:\")\n",
+    "print(f\"Recall@{k}: {recall_at_k:.4f}\")\n",
+    "print(f\"Precision@{k}: {precision_at_k:.4f}\")\n",
+    "print(f\"Mean Reciprocal Rank (MRR): {mrr:.4f}\")\n",
+    "print(f\"Mean Average Precision (MAP): {map_score:.4f}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "34d19556-8d99-4c53-8800-eec54948a674",
+   "metadata": {},
+   "source": [
+    "With this cookbook we were able to see how to:\n",
+    "- Generate a dataset of evaluations using PDF context-stuffing (leveraging vision modality of 4o) and traditional PDF readers\n",
+    "- Create a vector store and populate it with PDF\n",
+    "- Get an LLM answer to a query, leveraging a RAG system available out-of-the-box with `file_search` tool call in OpenAI's Response API\n",
+    "- Understand how chunks of texts are retrieved, ranked and used as part of the Response API\n",
+    "- Measure accuracy, precision, retrieval, MRR and MAP on the dataset of evaluations previously generated\n",
+    "\n",
+    "By using file search with Responses, you can simplify RAG architecture and leverage this in a single API call using the new Responses API. File storage, embeddings, retrieval all integrated in one tool!\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python (myenv)",
+   "language": "python",
+   "name": "myenv"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

examples/Fine-tuned_classification.ipynb

@@ -0,0 +1,675 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Fine tuning classification example\n",
+    "\n",
+    "We will fine-tune a `babbage-002` classifier (replacement for the `ada` models) to distinguish between the two sports: Baseball and Hockey."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sklearn.datasets import fetch_20newsgroups\n",
+    "import pandas as pd\n",
+    "import openai\n",
+    "import os\n",
+    "\n",
+    "client = openai.OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))\n",
+    "\n",
+    "categories = ['rec.sport.baseball', 'rec.sport.hockey']\n",
+    "sports_dataset = fetch_20newsgroups(subset='train', shuffle=True, random_state=42, categories=categories)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    " ## Data exploration\n",
+    " The newsgroup dataset can be loaded using sklearn. First we will look at the data itself:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "From: dougb@comm.mot.com (Doug Bank)\n",
+      "Subject: Re: Info needed for Cleveland tickets\n",
+      "Reply-To: dougb@ecs.comm.mot.com\n",
+      "Organization: Motorola Land Mobile Products Sector\n",
+      "Distribution: usa\n",
+      "Nntp-Posting-Host: 145.1.146.35\n",
+      "Lines: 17\n",
+      "\n",
+      "In article <1993Apr1.234031.4950@leland.Stanford.EDU>, bohnert@leland.Stanford.EDU (matthew bohnert) writes:\n",
+      "\n",
+      "|> I'm going to be in Cleveland Thursday, April 15 to Sunday, April 18.\n",
+      "|> Does anybody know if the Tribe will be in town on those dates, and\n",
+      "|> if so, who're they playing and if tickets are available?\n",
+      "\n",
+      "The tribe will be in town from April 16 to the 19th.\n",
+      "There are ALWAYS tickets available! (Though they are playing Toronto,\n",
+      "and many Toronto fans make the trip to Cleveland as it is easier to\n",
+      "get tickets in Cleveland than in Toronto.  Either way, I seriously\n",
+      "doubt they will sell out until the end of the season.)\n",
+      "\n",
+      "-- \n",
+      "Doug Bank                       Private Systems Division\n",
+      "dougb@ecs.comm.mot.com          Motorola Communications Sector\n",
+      "dougb@nwu.edu                   Schaumburg, Illinois\n",
+      "dougb@casbah.acns.nwu.edu       708-576-8207                    \n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(sports_dataset['data'][0])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'rec.sport.baseball'"
+      ]
+     },
+     "execution_count": 11,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "sports_dataset.target_names[sports_dataset['target'][0]]\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Total examples: 1197, Baseball examples: 597, Hockey examples: 600\n"
+     ]
+    }
+   ],
+   "source": [
+    "len_all, len_baseball, len_hockey = len(sports_dataset.data), len([e for e in sports_dataset.target if e == 0]), len([e for e in sports_dataset.target if e == 1])\n",
+    "print(f\"Total examples: {len_all}, Baseball examples: {len_baseball}, Hockey examples: {len_hockey}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "One sample from the baseball category can be seen above. It is an email to a mailing list. We can observe that we have 1197 examples in total, which are evenly split between the two sports."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Data Preparation\n",
+    "We transform the dataset into a pandas dataframe, with a column for prompt and completion. The prompt contains the email from the mailing list, and the completion is a name of the sport, either hockey or baseball. For demonstration purposes only and speed of fine-tuning we take only 300 examples. In a real use case the more examples the better the performance."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>prompt</th>\n",
+       "      <th>completion</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>From: dougb@comm.mot.com (Doug Bank)\\nSubject:...</td>\n",
+       "      <td>baseball</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>From: gld@cunixb.cc.columbia.edu (Gary L Dare)...</td>\n",
+       "      <td>hockey</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>From: rudy@netcom.com (Rudy Wade)\\nSubject: Re...</td>\n",
+       "      <td>baseball</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>From: monack@helium.gas.uug.arizona.edu (david...</td>\n",
+       "      <td>hockey</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>Subject: Let it be Known\\nFrom: &lt;ISSBTL@BYUVM....</td>\n",
+       "      <td>baseball</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "                                              prompt completion\n",
+       "0  From: dougb@comm.mot.com (Doug Bank)\\nSubject:...   baseball\n",
+       "1  From: gld@cunixb.cc.columbia.edu (Gary L Dare)...     hockey\n",
+       "2  From: rudy@netcom.com (Rudy Wade)\\nSubject: Re...   baseball\n",
+       "3  From: monack@helium.gas.uug.arizona.edu (david...     hockey\n",
+       "4  Subject: Let it be Known\\nFrom: <ISSBTL@BYUVM....   baseball"
+      ]
+     },
+     "execution_count": 13,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "\n",
+    "labels = [sports_dataset.target_names[x].split('.')[-1] for x in sports_dataset['target']]\n",
+    "texts = [text.strip() for text in sports_dataset['data']]\n",
+    "df = pd.DataFrame(zip(texts, labels), columns = ['prompt','completion']) #[:300]\n",
+    "df.head()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Both baseball and hockey are single tokens. We save the dataset as a jsonl file."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df.to_json(\"sport2.jsonl\", orient='records', lines=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Data Preparation tool\n",
+    "We can now use a data preparation tool which will suggest a few improvements to our dataset before fine-tuning. Before launching the tool we update the openai library to ensure we're using the latest data preparation tool. We additionally specify `-q` which auto-accepts all suggestions."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Analyzing...\n",
+      "\n",
+      "- Your file contains 1197 prompt-completion pairs\n",
+      "- Based on your data it seems like you're trying to fine-tune a model for classification\n",
+      "- For classification, we recommend you try one of the faster and cheaper models, such as `ada`\n",
+      "- For classification, you can estimate the expected model performance by keeping a held out dataset, which is not used for training\n",
+      "- There are 11 examples that are very long. These are rows: [134, 200, 281, 320, 404, 595, 704, 838, 1113, 1139, 1174]\n",
+      "For conditional generation, and for classification the examples shouldn't be longer than 2048 tokens.\n",
+      "- Your data does not contain a common separator at the end of your prompts. Having a separator string appended to the end of the prompt makes it clearer to the fine-tuned model where the completion should begin. See https://platform.openai.com/docs/guides/fine-tuning/preparing-your-dataset for more detail and examples. If you intend to do open-ended generation, then you should leave the prompts empty\n",
+      "- The completion should start with a whitespace character (` `). This tends to produce better results due to the tokenization we use. See https://platform.openai.com/docs/guides/fine-tuning/preparing-your-dataset for more details\n",
+      "\n",
+      "Based on the analysis we will perform the following actions:\n",
+      "- [Recommended] Remove 11 long examples [Y/n]: Y\n",
+      "- [Recommended] Add a suffix separator `\\n\\n###\\n\\n` to all prompts [Y/n]: Y\n",
+      "- [Recommended] Add a whitespace character to the beginning of the completion [Y/n]: Y\n",
+      "- [Recommended] Would you like to split into training and validation set? [Y/n]: Y\n",
+      "\n",
+      "\n",
+      "Your data will be written to a new JSONL file. Proceed [Y/n]: Y\n",
+      "\n",
+      "Wrote modified files to `sport2_prepared_train (1).jsonl` and `sport2_prepared_valid (1).jsonl`\n",
+      "Feel free to take a look!\n",
+      "\n",
+      "Now use that file when fine-tuning:\n",
+      "> openai api fine_tunes.create -t \"sport2_prepared_train (1).jsonl\" -v \"sport2_prepared_valid (1).jsonl\" --compute_classification_metrics --classification_positive_class \" baseball\"\n",
+      "\n",
+      "After you’ve fine-tuned a model, remember that your prompt has to end with the indicator string `\\n\\n###\\n\\n` for the model to start generating completions, rather than continuing with the prompt.\n",
+      "Once your model starts training, it'll approximately take 30.8 minutes to train a `curie` model, and less for `ada` and `babbage`. Queue will approximately take half an hour per job ahead of you.\n"
+     ]
+    }
+   ],
+   "source": [
+    "!openai tools fine_tunes.prepare_data -f sport2.jsonl -q"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The tool helpfully suggests a few improvements to the dataset and splits the dataset into training and validation set.\n",
+    "\n",
+    "A suffix between a prompt and a completion is necessary to tell the model that the input text has stopped, and that it now needs to predict the class. Since we use the same separator in each example, the model is able to learn that it is meant to predict either baseball or hockey following the separator.\n",
+    "A whitespace prefix in completions is useful, as most word tokens are tokenized with a space prefix.\n",
+    "The tool also recognized that this is likely a classification task, so it suggested to split the dataset into training and validation datasets. This will allow us to easily measure expected performance on new data."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Fine-tuning\n",
+    "The tool suggests we run the following command to train the dataset. Since this is a classification task, we would like to know what the generalization performance on the provided validation set is for our classification use case.\n",
+    "\n",
+    "We can simply copy the suggested command from the CLI tool. We specifically add `-m ada` to fine-tune a cheaper and faster ada model, which is usually comperable in performance to slower and more expensive models on classification use cases. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 25,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "FineTuningJob(id='ftjob-REo0uLpriEAm08CBRNDlPJZC', created_at=1704413736, error=None, fine_tuned_model=None, finished_at=None, hyperparameters=Hyperparameters(n_epochs='auto', batch_size='auto', learning_rate_multiplier='auto'), model='babbage-002', object='fine_tuning.job', organization_id='org-9HXYFy8ux4r6aboFyec2OLRf', result_files=[], status='validating_files', trained_tokens=None, training_file='file-82XooA2AUDBAUbN5z2DuKRMs', validation_file='file-wTOcQF8vxQ0Z6fNY2GSm0z4P')\n"
+     ]
+    }
+   ],
+   "source": [
+    "train_file = client.files.create(file=open(\"sport2_prepared_train.jsonl\", \"rb\"), purpose=\"fine-tune\")\n",
+    "valid_file = client.files.create(file=open(\"sport2_prepared_valid.jsonl\", \"rb\"), purpose=\"fine-tune\")\n",
+    "\n",
+    "fine_tuning_job = client.fine_tuning.jobs.create(training_file=train_file.id, validation_file=valid_file.id, model=\"babbage-002\")\n",
+    "\n",
+    "print(fine_tuning_job)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The model is successfully trained in about ten minutes. You can watch the finetune happen on [https://platform.openai.com/finetune/](https://platform.openai.com/finetune/)\n",
+    "\n",
+    "You can also check on its status programatically:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 58,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "1704414393\n"
+     ]
+    }
+   ],
+   "source": [
+    "fine_tune_results = client.fine_tuning.jobs.retrieve(fine_tuning_job.id)\n",
+    "print(fine_tune_results.finished_at)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### [Advanced] Results and expected model performance\n",
+    "We can now download the results file to observe the expected performance on a held out validation set."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 48,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "fine_tune_results = client.fine_tuning.jobs.retrieve(fine_tuning_job.id).result_files\n",
+    "result_file = client.files.retrieve(fine_tune_results[0])\n",
+    "content = client.files.content(result_file.id)\n",
+    "# save content to file\n",
+    "with open(\"result.csv\", \"wb\") as f:\n",
+    "    f.write(content.text.encode(\"utf-8\"))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 52,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>step</th>\n",
+       "      <th>train_loss</th>\n",
+       "      <th>train_accuracy</th>\n",
+       "      <th>valid_loss</th>\n",
+       "      <th>valid_mean_token_accuracy</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>2843</th>\n",
+       "      <td>2844</td>\n",
+       "      <td>0.0</td>\n",
+       "      <td>1.0</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "      step  train_loss  train_accuracy  valid_loss  valid_mean_token_accuracy\n",
+       "2843  2844         0.0             1.0         NaN                        NaN"
+      ]
+     },
+     "execution_count": 52,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "results = pd.read_csv('result.csv')\n",
+    "results[results['train_accuracy'].notnull()].tail(1)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The accuracy reaches 99.6%. On the plot below we can see how accuracy on the validation set increases during the training run. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "results[results['train_accuracy'].notnull()]['train_accuracy'].plot()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using the model\n",
+    "We can now call the model to get the predictions."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 55,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>prompt</th>\n",
+       "      <th>completion</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>From: gld@cunixb.cc.columbia.edu (Gary L Dare)...</td>\n",
+       "      <td>hockey</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>From: smorris@venus.lerc.nasa.gov (Ron Morris ...</td>\n",
+       "      <td>hockey</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>From: golchowy@alchemy.chem.utoronto.ca (Geral...</td>\n",
+       "      <td>hockey</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>From: krattige@hpcc01.corp.hp.com (Kim Krattig...</td>\n",
+       "      <td>baseball</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>From: warped@cs.montana.edu (Doug Dolven)\\nSub...</td>\n",
+       "      <td>baseball</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "                                              prompt completion\n",
+       "0  From: gld@cunixb.cc.columbia.edu (Gary L Dare)...     hockey\n",
+       "1  From: smorris@venus.lerc.nasa.gov (Ron Morris ...     hockey\n",
+       "2  From: golchowy@alchemy.chem.utoronto.ca (Geral...     hockey\n",
+       "3  From: krattige@hpcc01.corp.hp.com (Kim Krattig...   baseball\n",
+       "4  From: warped@cs.montana.edu (Doug Dolven)\\nSub...   baseball"
+      ]
+     },
+     "execution_count": 55,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "test = pd.read_json('sport2_prepared_valid.jsonl', lines=True)\n",
+    "test.head()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We need to use the same separator following the prompt which we used during fine-tuning. In this case it is `\\n\\n###\\n\\n`. Since we're concerned with classification, we want the temperature to be as low as possible, and we only require one token completion to determine the prediction of the model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 69,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "' hockey'"
+      ]
+     },
+     "execution_count": 69,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "ft_model = fine_tune_results.fine_tuned_model\n",
+    "\n",
+    "# note that this calls the legacy completions api - https://platform.openai.com/docs/api-reference/completions\n",
+    "res = client.completions.create(model=ft_model, prompt=test['prompt'][0] + '\\n\\n###\\n\\n', max_tokens=1, temperature=0)\n",
+    "res.choices[0].text\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To get the log probabilities, we can specify logprobs parameter on the completion request"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 71,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[{' hockey': 0.0, ' Hockey': -22.504879}]"
+      ]
+     },
+     "execution_count": 71,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "res = client.completions.create(model=ft_model, prompt=test['prompt'][0] + '\\n\\n###\\n\\n', max_tokens=1, temperature=0, logprobs=2)\n",
+    "res.choices[0].logprobs.top_logprobs"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can see that the model predicts hockey as a lot more likely than baseball, which is the correct prediction. By requesting log_probs, we can see the prediction (log) probability for each class."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Generalization\n",
+    "Interestingly, our fine-tuned classifier is quite versatile. Despite being trained on emails to different mailing lists, it also successfully predicts tweets."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 73,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "' hockey'"
+      ]
+     },
+     "execution_count": 73,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "sample_hockey_tweet = \"\"\"Thank you to the \n",
+    "@Canes\n",
+    " and all you amazing Caniacs that have been so supportive! You guys are some of the best fans in the NHL without a doubt! Really excited to start this new chapter in my career with the \n",
+    "@DetroitRedWings\n",
+    " !!\"\"\"\n",
+    "res = client.completions.create(model=ft_model, prompt=sample_hockey_tweet + '\\n\\n###\\n\\n', max_tokens=1, temperature=0, logprobs=2)\n",
+    "res.choices[0].text"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "sample_baseball_tweet=\"\"\"BREAKING: The Tampa Bay Rays are finalizing a deal to acquire slugger Nelson Cruz from the Minnesota Twins, sources tell ESPN.\"\"\"\n",
+    "res = client.completions.create(model=ft_model, prompt=sample_baseball_tweet + '\\n\\n###\\n\\n', max_tokens=1, temperature=0, logprobs=2)\n",
+    "res.choices[0].text"
+   ]
+  }
+ ],
+ "metadata": {
+  "interpreter": {
+   "hash": "3b138a8faad971cc852f62bcf00f59ea0e31721743ea2c5a866ca26adf572e75"
+  },
+  "kernelspec": {
+   "display_name": "Python 3.7.3 64-bit ('base': conda)",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.3"
+  },
+  "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/Fine_tuning_direct_preference_optimization_guide.ipynb

@@ -0,0 +1,698 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Fine-Tuning Techniques: Choosing Between SFT, DPO, and RFT (Including a Guide to DPO)\n",
+    " \n",
+    "*This guide is for developers and ML practitioners who have some experience with OpenAIʼs APIs and wish to use their fine-tuned models for research or other appropriate uses. OpenAI’s services are not intended for the personalized treatment or diagnosis of any medical condition and are subject to our [applicable terms](https://openai.com/policies/).*\n",
+    " \n",
+    "This guide discusses fine-tuning methods supported by OpenAI, specifically highlighting what each method is best for and not best for, to help you identify the most suitable technique for your use case. It then provides an in-depth look at one particular method — Direct Preference Optimization (DPO) — and provides links to existing guides for the other techniques.\n",
+    " \n",
+    "**What is fine-tuning?** Fine-tuning is the process of continuing training on a smaller, domain-specific dataset to optimize a model for a specific task. There are two main reasons why we would typically fine-tune:\n",
+    "1. Improve model performance on a specific task \n",
+    "2. Improve model efficiency (reduce the number of tokens needed, distill expertise into a smaller model, etc.)\n",
+    " \n",
+    "Currently, the OpenAI platform supports four fine-tuning methods:\n",
+    "- **Supervised fine-tuning (SFT):** this technique employs traditional supervised learning using input-output pairs to adjust model parameters. The training process adjusts model weights to minimize the difference between predicted and target outputs across the provided examples. The model will replicate features that it finds in provided pairs. \n",
+    "- **Vision fine-tuning:** this technique extends supervised fine-tuning to multimodal data by processing both text and image in a unified training framework. The training process adjusts model weights to minimize errors across text-image pairs and as a result improve the model's understanding of image inputs. \n",
+    "- **Direct preference optimization (DPO):** this technique uses pairwise comparisons (e.g., preferred and rejected example responses) to optimize a model to favor certain outputs over others. The model learns to replicate the preference patterns found in the provided comparison data. \n",
+    "- **Reinforcement fine-tuning (RFT):** this technique uses reinforcement learning with a reward signal (via a grader or reward model) to fine-tune the model for complex objectives. In RFT, the model generates outputs for given prompts during training, and each output is evaluated for quality. The model's parameters are then updated to maximize the reward, reinforcing behaviors that lead to better outcomes. This iterative feedback loop encourages the model to improve reasoning or decision-making strategies. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To help you select the appropriate fine-tuning technique, the table below summarizes the scenarios each method is best suited for, as well as those for which it is not well suited:\n",
+    "\n",
+    "| **Technique**                            | **Good For**                                                                                                                                                                                                                                                                                    | **Not Good For**                                                                                            |\n",
+    "| ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |\n",
+    "| **Supervised fine-tuning (SFT)**         | Emphasizing knowledge already present in the model.<br>Customizing response structure or tone.<br>Generating content in a specific format.<br>Teaching complex instructions or correcting instruction-following failures.<br>Optimizing cost/latency (saving tokens from prompt or distilling). | Adding entirely new knowledge (consider RAG instead).<br>Tasks with subjective quality.                     |\n",
+    "| **Vision fine-tuning**                   | Specialized visual recognition tasks (e.g., image classification).<br>Domain-specific image understanding.<br>Correcting failures in instruction following for complex prompts.                                                                               | Purely textual tasks.<br>Generalized visual tasks without specific context.<br>General image understanding. |\n",
+    "| **Direct preference optimization (DPO)** | Aligning model outputs with subjective preferences (tone, politeness).<br>Refining outputs via human-rated feedback.<br>Achieving nuanced behavioral alignment.                                                                                                                                 | Learning completely new tasks.<br>Tasks without clear human preference signals.                             |\n",
+    "| **Reinforcement fine-tuning (RFT)**      | Complex domain-specific tasks that require advanced reasoning.<br>Refining existing partial capabilities (fostering emergent behaviours).<br>Tasks with measurable feedback.<br>Scenarios with limited explicit labels where reward signals can be defined.                                                                     | Tasks where the model has no initial skill.<br>Tasks without clear feedback or measurable signals.          |\n",
+    "\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Today, there are pre-existing Cookbooks for:  \n",
+    "\n",
+    "- Supervised fine-tuning (SFT): (1) [How to fine-tune chat models](https://cookbook.openai.com/examples/how_to_finetune_chat_models) (2) [Leveraging model distillation to fine-tune a model](https://cookbook.openai.com/examples/leveraging_model_distillation_to_fine-tune_a_model)\n",
+    "- Vision fine-tuning: [Vision fine-tuning on GPT-4o for visual question answering](https://cookbook.openai.com/examples/multimodal/vision_fine_tuning_on_gpt4o_for_visual_question_answering)\n",
+    "- Reinforcement fine-tuning (RFT): (1) [Reinforcement fine-tuning (RFT)](https://cookbook.openai.com/examples/reinforcement_fine_tuning), (2) [Reinforcement fine-tuning for healthbench QA](https://cookbook.openai.com/examples/fine-tuned_qa/reinforcement_finetuning_healthbench)\n",
+    "\n",
+    "Direct preference optimization (DPO) will be covered in this guide."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## **Guide to Direct Preference Optimization**\n",
+    " \n",
+    "As mentioned above, [Direct Preference Optimization (DPO)](https://platform.openai.com/docs/guides/direct-preference-optimization) is an alignment technique for fine-tuning language models using pairwise preference data (e.g., ranked pairs of responses). DPO directly optimizes a model to favor certain outputs over others using explicit pairwise comparisons, typically from human preferences. This approach simplifies alignment and eliminates the need for a separate reward model or complex reinforcement learning procedures, making DPO a lightweight alternative to techniques such as Reinforcement Learning from Human Feedback (RLHF).\n",
+    " \n",
+    "When should you use DPO? DPO excels in scenarios when response quality is subjective, cannot be measured objectively, or when nuanced criteria such as tone, style, appropriateness, or clarity matter - typically cases where multiple valid outputs exist. Example applications where DPO is particularly effective in aligning AI responses include: \n",
+    "- Enhancing Conversational AI Responses\n",
+    "- Improving Code Generation Quality & Style\n",
+    "- Ensuring Compliance with Legal, Ethical & Safety Standards \n",
+    "- Controlling Brand Voice, Professionalism, & Tone\n",
+    "- Customizing Creative Outputs & User Experience\n",
+    "\n",
+    "By fine-tuning on explicit pairs of preferred vs non‑preferred completions, DPO aligns model outputs to these nuanced preferences. The below table gives examples of pairwise preference data for a fictional AI assistant that represents an organization, where preferred responses are clear, professional, and aligned with brand standards.\n",
+    " \n",
+    "| **Example Question**                                           | **Chosen Response**                                                                                                                                                                   | **Rejected Response**                                       |\n",
+    "|------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------|\n",
+    "| **Q1:** *How do I review your product?*              | To submit a product review, please visit your account dashboard, select the product, and click ‘Write a review.’ Share your honest experience, rate key features, and submit when ready.        | Yo, just leave some quick stars or whatever, it’s chill!         |\n",
+    "| **Q2:** *How do I review your product?*              | We welcome your feedback! In the ‘Reviews’ section on the product page, click ‘Leave a Review,’ rate it, and add your comments about what you liked or areas for improvement.                   | Just scribble something—doesn’t matter what, honestly.          |\n",
+    "| **Q3:** *How to troubleshoot this particular error?* | To address the error ‘X101,’ first clear your cache, then verify your internet connection. If the issue remains, follow our step-by-step guide at [Support → Troubleshooting → Error X101]. | Just reboot it, I guess. If it doesn't work, you're on your own! |\n",
+    " \n",
+    "In this guide, weʼll walk through how to apply DPO using the fine-tuning API. You will learn key steps to take in order to successfully run preference fine-tuning jobs for your use-cases.\n",
+    " \n",
+    "Here’s what we’ll cover:\n",
+    " \n",
+    "- **1. Recommended Workflow**\n",
+    "- **2. Demonstration Scenario**\n",
+    "- **3. Generating the Dataset**\n",
+    "- **4. Benchmarking the Base Model**\n",
+    "- **5. Fine-Tuning**\n",
+    "- **6. Using your Fine-Tuned Model**"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## **1. Recommended Workflow**\n",
+    " \n",
+    "OpenAI recommends the following workflow: \n",
+    "1. Performing Supervised Fine-Tuning (SFT) on a subset of your preferred responses. \n",
+    "2. Using the SFT fine-tuned model as the starting point, apply DPO using preference comparison data. \n",
+    " \n",
+    "Performing Supervised Fine-Tuning (SFT) before Direct Preference Optimization (DPO) enhances model alignment and overall performance by establishing a robust initial policy, ensuring the model already prefers correct responses. This reduces the magnitude of weight updates during DPO, stabilizing training and preventing overfitting by allowing DPO to efficiently refine subtle nuances. Consequently, the combined SFT-then-DPO workflow converges faster and yields higher-quality results.\n",
+    "\n",
+    "In this guide, we'll focus exclusively on applying Direct Preference Optimization (DPO). However, depending on your use case, you may find performance gains from first performing Supervised Fine-Tuning (SFT). If so, you can follow the SFT guide linked above, save the resulting model ID, and use that as the starting point for your DPO job."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## **2. Demonstration Scenario**\n",
+    "\n",
+    "To make things concrete, let’s walk through fine-tuning a customer-facing AI assistant to follow a fictional brand’s voice and style. Imagine Good Vibes Corp, an organization that prides itself on a friendly, enthusiastic tone with a personal touch. \n",
+    " \n",
+    "They want their customer AI assistant to answer queries in a way that reflects these brand guidelines (e.g. an upbeat attitude, polite language, and a friendly sign-off), and prefer those responses over more generic or curt answers. This is a good scenario for DPO: there’s no objectively correct answer format, but there is a preferred style.\n",
+    " \n",
+    "DPO will help the model learn from comparisons which style is preferred. We'll outline the steps to: (1) generate a synthetic preference dataset of prompts with paired responses (one in the desired brand voice and one not). (2) Evaluate base model performance using the OpenAI evals API. (3) Prepare and upload the data in the required JSONL format for preference fine-tuning. (4) Fine-tune the model with DPO using the OpenAI fine-tuning API. (5) Evaluate the fine-tuned model using the OpenAI evals API to show how the brand-style preference improved.\n",
+    "\n",
+    "We are going to synthesize a dataset for this demonstration. First, let’s create a seed bank of questions to generate more variations from.\n",
+    "\n",
+    "Let’s get started!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "! pip install openai nest-asyncio --quiet"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "PROMPT_SEED_POOL = [\n",
+    "    \"Hi, I ordered a gadget last week. When will it arrive?\",\n",
+    "    \"Your product stopped working after two days. Can I get help?\",\n",
+    "    \"Do you offer discounts for long-term customers?\",\n",
+    "    \"Can I change the shipping address for my order?\",\n",
+    "    \"What is your return policy for damaged items?\",\n",
+    "    \"My tracking number hasn't updated in three days—can you check the status?\",\n",
+    "    \"How long is the warranty on your products, and how do I submit a claim?\",\n",
+    "    \"Can I add gift wrapping to my order before it ships?\",\n",
+    "    \"Do you accept PayPal or other alternative payment methods?\",\n",
+    "    \"Is there an option to expedite shipping if my order hasn't left the warehouse yet?\",\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## **3. Generating the Dataset**\n",
+    "\n",
+    "Next, we’ll define functions to take each prompt from our seed bank and generate related questions. We’ll create a dataset of preference pairs by first generating these prompt variations, then producing both a preferred and a rejected response for every prompt. \n",
+    "\n",
+    "This dataset is synthetic and serves to illustrate the mechanics of Direct Preference Optimization — when developing your own application you should collect or curate a high-quality, preference dataset. Note: the volume of data required for DPO depends on the use case; generally more is better (thousands to tens of thousands), and for preference pairs the ordering logic should be consistent (e.g. if A > B and B > C, then A > C)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import asyncio\n",
+    "from openai import AsyncOpenAI\n",
+    "from typing import List, Dict, Any\n",
+    "\n",
+    "async_client = AsyncOpenAI()\n",
+    "\n",
+    "SYSTEM_PROMPT = \"You are a customer-support assistant.\"\n",
+    "\n",
+    "\n",
+    "async def _generate_related_questions_from_prompt(\n",
+    "    prompt: str, k: int, sem: asyncio.Semaphore, *, model: str\n",
+    ") -> List[str]:\n",
+    "    \"\"\"Return *k* distinct customer-service questions related to the given prompt.\"\"\"\n",
+    "    out: List[str] = []\n",
+    "    async with sem:\n",
+    "        for _ in range(k):\n",
+    "            resp = await async_client.responses.create(\n",
+    "                model=model,\n",
+    "                input=[\n",
+    "                    {\n",
+    "                        \"role\": \"system\",\n",
+    "                        \"content\": (\n",
+    "                            \"Return ONE distinct, realistic customer-service question \"\n",
+    "                            \"related in topic or theme to the following question, \"\n",
+    "                            \"but NOT a direct paraphrase.\"\n",
+    "                        ),\n",
+    "                    },\n",
+    "                    {\"role\": \"user\", \"content\": prompt},\n",
+    "                ],\n",
+    "                temperature=0.9,\n",
+    "                max_output_tokens=60,\n",
+    "            )\n",
+    "            out.append(resp.output_text.strip())\n",
+    "    return out\n",
+    "\n",
+    "\n",
+    "async def expand_prompt_pool(\n",
+    "    prompts: List[str], *, k: int = 3, concurrency: int = 32, model: str\n",
+    ") -> List[str]:\n",
+    "    \"\"\"Expand each prompt into *k* related questions using the given model.\"\"\"\n",
+    "    sem = asyncio.Semaphore(concurrency)\n",
+    "    tasks = [\n",
+    "        _generate_related_questions_from_prompt(p, k, sem, model=model) for p in prompts\n",
+    "    ]\n",
+    "    results = await asyncio.gather(*tasks)\n",
+    "    return [v for sub in results for v in sub]\n",
+    "\n",
+    "\n",
+    "async def _generate_preference_pair(\n",
+    "    prompt: str, sem: asyncio.Semaphore, *, model: str\n",
+    ") -> Dict[str, Any]:\n",
+    "    \"\"\"Generate a preference pair for the given prompt.\"\"\"\n",
+    "    async with sem:\n",
+    "        friendly_task = async_client.responses.create(\n",
+    "            model=model,\n",
+    "            input=[\n",
+    "                {\n",
+    "                    \"role\": \"system\",\n",
+    "                    \"content\": (\n",
+    "                        \"You are Good Vibes Corp's exceptionally energetic, outrageously friendly and \"\n",
+    "                        \"enthusiastic support agent.\"\n",
+    "                    ),\n",
+    "                },\n",
+    "                {\"role\": \"user\", \"content\": prompt},\n",
+    "            ],\n",
+    "            temperature=0.7,  # higher temperature to increase creativity & on-brand tone adherence\n",
+    "            max_output_tokens=80,\n",
+    "        )\n",
+    "        blunt_task = async_client.responses.create(\n",
+    "            model=model,\n",
+    "            input=[\n",
+    "                {\n",
+    "                    \"role\": \"system\",\n",
+    "                    \"content\": \"You are a terse, factual support agent with no empathy or politeness.\",\n",
+    "                },\n",
+    "                {\"role\": \"user\", \"content\": prompt},\n",
+    "            ],\n",
+    "            temperature=0.3,  # lower temperature to limit creativity & emphasize tonal difference\n",
+    "            max_output_tokens=80,\n",
+    "        )\n",
+    "        friendly, blunt = await asyncio.gather(friendly_task, blunt_task)\n",
+    "        return {\n",
+    "            \"input\": {\n",
+    "                \"messages\": [\n",
+    "                    {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n",
+    "                    {\"role\": \"user\", \"content\": prompt},\n",
+    "                ]\n",
+    "            },\n",
+    "            \"preferred_output\": [\n",
+    "                {\"role\": \"assistant\", \"content\": friendly.output_text}\n",
+    "            ],\n",
+    "            \"non_preferred_output\": [\n",
+    "                {\"role\": \"assistant\", \"content\": blunt.output_text}\n",
+    "            ],\n",
+    "        }"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now, using these defined functions we'll build our dataset by generating friendly versus blunt response pairs. The friendly responses reflect the brand's desired communication style. We'll do this asynchronously for efficiency, creating a dataset suited for Direct Preference Optimization."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Dataset ready with 500 pairs.\n"
+     ]
+    }
+   ],
+   "source": [
+    "import math\n",
+    "import nest_asyncio\n",
+    "\n",
+    "\n",
+    "async def build_dataset(\n",
+    "    *,\n",
+    "    pair_count: int = 500,\n",
+    "    concurrency: int = 8,\n",
+    "    expand_prompt_pool_model: str,\n",
+    "    generate_preference_pair_model: str,\n",
+    ") -> List[Dict[str, Any]]:\n",
+    "    \"\"\"Return *pair_count* preference pairs (single-shot expansion).\"\"\"\n",
+    "\n",
+    "    seed = PROMPT_SEED_POOL\n",
+    "    deficit = max(0, pair_count - len(seed))\n",
+    "    k = max(1, math.ceil(deficit / len(seed)))\n",
+    "\n",
+    "    expanded = await expand_prompt_pool(\n",
+    "        seed,\n",
+    "        k=k,\n",
+    "        concurrency=concurrency,\n",
+    "        model=expand_prompt_pool_model,\n",
+    "    )\n",
+    "    prompt_bank = (seed + expanded)[:pair_count]\n",
+    "\n",
+    "    sem = asyncio.Semaphore(concurrency)\n",
+    "    tasks = [\n",
+    "        _generate_preference_pair(p, sem, model=generate_preference_pair_model)\n",
+    "        for p in prompt_bank\n",
+    "    ]\n",
+    "    return await asyncio.gather(*tasks)\n",
+    "\n",
+    "\n",
+    "nest_asyncio.apply()\n",
+    "pairs = await build_dataset(\n",
+    "    pair_count=500,\n",
+    "    concurrency=8,\n",
+    "    expand_prompt_pool_model=\"gpt-4.1-mini-2025-04-14\",\n",
+    "    generate_preference_pair_model=\"gpt-4.1-mini-2025-04-14\",\n",
+    ")\n",
+    "print(f\"Dataset ready with {len(pairs)} pairs.\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## **4. Benchmarking the Base Model**\n",
+    "\n",
+    "Below, we split our dataset into training, validation, and testing sets. We also show a sample from the training dataset, which demonstrates a clear difference between the preferred (friendly, on-brand) and non-preferred (blunt, neutral) responses for that input pair."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'input': {'messages': [{'role': 'system',\n",
+       "    'content': 'You are a customer-support assistant.'},\n",
+       "   {'role': 'user',\n",
+       "    'content': 'Hi, I ordered a gadget last week. When will it arrive?'}]},\n",
+       " 'preferred_output': [{'role': 'assistant',\n",
+       "   'content': 'Hey there, awesome friend! 🌟 Thanks a bunch for reaching out! I’d LOVE to help you track down your gadget so you can start enjoying it ASAP! 🎉 Could you please share your order number or the email you used to place the order? Let’s make this delivery magic happen! 🚀✨'}],\n",
+       " 'non_preferred_output': [{'role': 'assistant',\n",
+       "   'content': 'Provide your order number for delivery status.'}]}"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# set dataset sizes\n",
+    "n = len(pairs)\n",
+    "n_train = int(0.8 * n)\n",
+    "n_val = int(0.1 * n)\n",
+    "n_test = n - n_train - n_val\n",
+    "\n",
+    "# split dataset into train, test & validation\n",
+    "train_pairs = pairs[:n_train]\n",
+    "val_pairs = pairs[n_train : n_train + n_val]\n",
+    "test_pairs = pairs[n_train + n_val :]\n",
+    "train_pairs[0]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To assess the model's performance prior to fine-tuning, we'll use an automated grader (LLM-as-a-Judge) to score each response for friendliness and empathy. The grader will assign a score from 0 to 4 for each answer, allowing us to compute a mean baseline score for the base model. \n",
+    "\n",
+    "To do this, we first generate responses for the base model on the test set, then use the OpenAI evals API to create and run an evaluation with an automated grader. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "async def generate_responses(\n",
+    "    testset,\n",
+    "    model,\n",
+    "    temperature=0.0,\n",
+    "    max_output_tokens=80,\n",
+    "    concurrency=8,\n",
+    "):\n",
+    "    \"\"\"\n",
+    "    Generate responses for each prompt in the testset using the OpenAI responses API.\n",
+    "    Returns: List of dicts: [{\"prompt\": ..., \"response\": ...}, ...]\n",
+    "    \"\"\"\n",
+    "    async_client = AsyncOpenAI()\n",
+    "    sem = asyncio.Semaphore(concurrency)\n",
+    "\n",
+    "    async def get_response(prompt):\n",
+    "        async with sem:\n",
+    "            resp = await async_client.responses.create(\n",
+    "                model=model,\n",
+    "                input=[\n",
+    "                    {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n",
+    "                    {\"role\": \"user\", \"content\": prompt},\n",
+    "                ],\n",
+    "                temperature=temperature,\n",
+    "                max_output_tokens=max_output_tokens,\n",
+    "            )\n",
+    "            return {\"prompt\": prompt, \"response\": resp.output_text}\n",
+    "\n",
+    "    tasks = [get_response(item[\"item\"][\"input\"]) for item in testset]\n",
+    "    results = await asyncio.gather(*tasks)\n",
+    "    return results\n",
+    "\n",
+    "\n",
+    "# generate responses for the base model over the test set\n",
+    "base_model = \"gpt-4.1-mini-2025-04-14\"\n",
+    "testset = [\n",
+    "    {\"item\": {\"input\": pair[\"input\"][\"messages\"][1][\"content\"]}} for pair in test_pairs\n",
+    "]\n",
+    "responses = await generate_responses(testset, model=base_model)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Next, we'll use the OpenAI evals API to create & run an evaluation with an automated grader, starting by defining the rubric for the LLM-as-a-Judge. Note: we will access responses via data logging, so in order for this to work, you'll need to be in an org where data logging isn't disabled (through zdr, etc.). If you aren't sure if this is the case for you, go to https://platform.openai.com/logs?api=responses and see if you can see the responses you just generated."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "JUDGE_SYSTEM = \"\"\"\n",
+    "You judge whether a reply matches Good Vibes Corp's desired tone:\n",
+    "energetic, super-friendly, enthusiastic.\n",
+    "\n",
+    "Score 0-4 (higher = more energy):\n",
+    "\n",
+    "4 - Highly enthusiastic: multiple upbeat phrases / emojis / exclamations, clear empathy, proactive help.\n",
+    "3 - Energetic & friendly: visible enthusiasm cue (≥1 emoji OR exclamation OR upbeat phrase), warm second-person tone.\n",
+    "2 - Pleasant: polite & positive but lacks obvious enthusiasm cues.\n",
+    "1 - Neutral: correct, businesslike, minimal warmth.\n",
+    "0 - Rude, negative, or unhelpful.\n",
+    "\"\"\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from openai import OpenAI\n",
+    "\n",
+    "sync_client = OpenAI()\n",
+    "\n",
+    "# set judge model\n",
+    "judge_model = \"gpt-4.1-2025-04-14\"\n",
+    "\n",
+    "# create the evaluation\n",
+    "logs_eval = sync_client.evals.create(\n",
+    "    name=\"Good Vibes Corp Tone Eval\",\n",
+    "    data_source_config={\n",
+    "        \"type\": \"logs\",\n",
+    "    },\n",
+    "    testing_criteria=[\n",
+    "        {\n",
+    "            \"type\": \"score_model\",\n",
+    "            \"name\": \"General Evaluator\",\n",
+    "            \"model\": judge_model,\n",
+    "            \"input\": [\n",
+    "                {\n",
+    "                    \"role\": \"system\",\n",
+    "                    \"content\": JUDGE_SYSTEM,\n",
+    "                },\n",
+    "                {\n",
+    "                    \"role\": \"user\",\n",
+    "                    \"content\": (\n",
+    "                        \"**User input**\\n\"\n",
+    "                        \"{{item.input}}\\n\"\n",
+    "                        \"**Response to evaluate**\\n\"\n",
+    "                        \"{{sample.output_text}}\"\n",
+    "                    ),\n",
+    "                },\n",
+    "            ],\n",
+    "            \"range\": [0, 4],\n",
+    "            \"pass_threshold\": 2,\n",
+    "        }\n",
+    "    ],\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# run the evaluation\n",
+    "base_run = sync_client.evals.runs.create(\n",
+    "    name=base_model,\n",
+    "    eval_id=logs_eval.id,\n",
+    "    data_source={\n",
+    "        \"type\": \"responses\",\n",
+    "        \"source\": {\"type\": \"responses\", \"limit\": len(test_pairs)},\n",
+    "    },\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Average score: 2.525\n"
+     ]
+    }
+   ],
+   "source": [
+    "# score base model\n",
+    "base_data = sync_client.evals.runs.output_items.list(\n",
+    "    eval_id=logs_eval.id, run_id=base_run.id\n",
+    ").data\n",
+    "base_scores = [s.results[0][\"score\"] for s in base_data]\n",
+    "print(\"Average score:\", sum(base_scores) / len(base_scores))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## **5. Fine-Tuning**\n",
+    "\n",
+    "With a baseline established, we can now fine-tune the model using the training set and DPO. This process will teach the model to prefer responses that align with our desired style, based on the preference pairs we created earlier.\n",
+    "\n",
+    "Note: **beta (β)** is a unique fine-tuning hyperparameter for Direct Preference Optimization (DPO). It’s a floating-point number ranging between 0 and 2, controlling the balance between preserving a model’s existing behavior and adapting to new, preference-aligned responses.\n",
+    "- High β (close to 2): makes the model more conservative, strongly favoring previous behavior. The fine-tuned model will show minimal deviations from its original style or characteristics, emphasizing consistency and avoiding abrupt changes.\n",
+    "- Moderate β (around 1): balances between adherence to prior behavior and adaptation to new preferences. Recommended as a sensible starting point for most practical scenarios.\n",
+    "- Low β (close to 0): encourages aggressive adaptation, causing the model to prioritize newly provided preferences more prominently. This might result in significant stylistic shifts and greater alignment with explicit preferences but could lead to unexpected or overly specialized outputs.\n",
+    "\n",
+    "Technically, beta scales the difference in log-probabilities in the DPO loss; a larger β causes the sigmoid-based loss function to saturate with smaller probability differences, yielding smaller weight updates (thus preserving old behavior). It is recommended to experiment systematically with the β value to achieve optimal results tailored to your specific use-case and desired trade-offs between stability and adaptation."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Fine-tuning job created: job_id = ftjob-5QPmA36QezFRGoXjuvIAPuAQ\n"
+     ]
+    }
+   ],
+   "source": [
+    "import io\n",
+    "import json\n",
+    "\n",
+    "# create training file\n",
+    "train_buf = io.BytesIO(\"\\n\".join(json.dumps(p) for p in train_pairs).encode())\n",
+    "train_buf.name = \"train.jsonl\"\n",
+    "train_file_id = sync_client.files.create(file=train_buf, purpose=\"fine-tune\").id\n",
+    "\n",
+    "# create validation file\n",
+    "val_buf = io.BytesIO(\"\\n\".join(json.dumps(p) for p in val_pairs).encode())\n",
+    "val_buf.name = \"val.jsonl\"\n",
+    "val_file_id = sync_client.files.create(file=val_buf, purpose=\"fine-tune\").id\n",
+    "\n",
+    "# create a fine-tuning job\n",
+    "ft = sync_client.fine_tuning.jobs.create(\n",
+    "    model=base_model,\n",
+    "    training_file=train_file_id,\n",
+    "    validation_file=val_file_id,\n",
+    "    method={\n",
+    "        \"type\": \"dpo\",\n",
+    "        \"dpo\": {\n",
+    "            \"hyperparameters\": {\n",
+    "                \"n_epochs\": 2,\n",
+    "                \"beta\": 0.1,\n",
+    "                \"batch_size\": 8,\n",
+    "            }\n",
+    "        },\n",
+    "    },\n",
+    ")\n",
+    "print(f\"Fine-tuning job created: job_id = {ft.id}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## **6. Using your Fine-Tuned Model**\n",
+    "\n",
+    "Once fine-tuning is complete, we'll evaluate the DPO-tuned model on the same test set. By comparing the mean scores before and after fine-tuning, as well as reviewing example outputs, we can see how the model's alignment with our preferences has improved."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# generate responses\n",
+    "job = sync_client.fine_tuning.jobs.retrieve(ft.id)\n",
+    "if job.status == \"succeeded\":\n",
+    "    responses = await generate_responses(testset, model=job.fine_tuned_model)\n",
+    "\n",
+    "    post_run = sync_client.evals.runs.create(\n",
+    "        name=ft.id,\n",
+    "        eval_id=logs_eval.id,\n",
+    "        data_source={\n",
+    "            \"type\": \"responses\",\n",
+    "            \"source\": {\"type\": \"responses\", \"limit\": len(test_pairs)},\n",
+    "        },\n",
+    "    )"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 59,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Δ mean: 0.45\n",
+      "\n",
+      "=== SAMPLE COMPARISON ===\n",
+      "Prompt:\n",
+      "  Can I upgrade to faster delivery if my package is still being processed?\n",
+      "\n",
+      "Base model reply: \n",
+      " Whether you can upgrade to express shipping while your order is still being processed depends on the store's policies. Generally, many stores allow shipping upgrades before the order is shipped. \n",
+      "\n",
+      "To assist you better, could you please provide your order number or the name of the store you ordered from? Alternatively, you can contact the store's customer service directly to request the upgrade. \n",
+      "\n",
+      "DPO-tuned model reply \n",
+      " Hi! I’d be happy to help with that. If your package hasn’t shipped yet, there’s a good chance we can upgrade your delivery speed. Could you please provide me with your order number? I’ll check the status and let you know the available options for faster delivery.\n"
+     ]
+    }
+   ],
+   "source": [
+    "# get scores from the evaluation\n",
+    "post_data = sync_client.evals.runs.output_items.list(\n",
+    "    eval_id=logs_eval.id, run_id=post_run.id\n",
+    ").data\n",
+    "post_scores = [s.results[0][\"score\"] for s in post_data]\n",
+    "\n",
+    "# print scores & a sample comparison from the test set for illustration\n",
+    "print(\n",
+    "    \"Δ mean:\",\n",
+    "    sum(t - b for b, t in zip(base_scores, post_scores)) / len(base_scores),\n",
+    ")\n",
+    "print(\"\\n=== SAMPLE COMPARISON ===\")\n",
+    "idx = 0\n",
+    "print(f\"Prompt:\\n  {testset[idx]['item']['input']}\\n\")\n",
+    "print(f\"Base model reply: \\n {base_data[idx].sample.output[0].content} \\n\")\n",
+    "print(f\"DPO-tuned model reply \\n {post_data[idx].sample.output[0].content}\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": ".venv311",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/Function_calling_finding_nearby_places.ipynb

@@ -0,0 +1,349 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Function calling for nearby places: Leveraging the Google Places API and customer profiles\n",
+    "\n",
+    "This notebook is centered around the integration of the Google Places API and custom user profiles to enhance location-based searches. Our approach involves using the Google Places API in combination with user preferences, aiming to make location discovery more personal and relevant. Please note that while we focus on the Google Places API in this instance, there are numerous other APIs you could explore and apply in a similar fashion.\n",
+    "\n",
+    "We'll explore the application of three main components:\n",
+    "\n",
+    "- Customer profile: This mock profile captures individual preferences for types of places (e.g., restaurants, parks, museums), budget, preferred ratings, and other specific requirements. \n",
+    "\n",
+    "- Google Places API: This API provides real-time data about nearby places. It factors in various data points such as ratings, types of venues, costs, and more from the locations around you.\n",
+    "\n",
+    "- Function calling: A single command such as \"I'm hungry\" or \"I want to visit a museum\" activates the function which combines the user profile data and Google Places API to identify suitable venues.\n",
+    "\n",
+    "This notebook introduces two primary use cases:\n",
+    "\n",
+    "- Profile-based recommendations: Learn how to create a user profile and make place recommendations based on individual preferences.\n",
+    "\n",
+    "- API integration with function calling: Understand how to integrate and call Google Places API effectively to source real-time data of various places using function calling.\n",
+    "\n",
+    "Please note that while this system is highly versatile, its effectiveness may vary based on user preferences and available place data. For the purposes of this notebook, the customer data is fake and the location is hardcoded. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Setup\n",
+    "\n",
+    "Google Places API\n",
+    "\n",
+    "To use the Google Places API, you'll need two things:\n",
+    "\n",
+    "- Google Account: If you don't already have one, you will need to create a Google account.\n",
+    "\n",
+    "- Google Places API Key: The API key is a unique identifier that is used to authenticate requests associated with your project for usage and billing purposes. You can get your API key from the [Google Cloud Console](https://console.cloud.google.com/getting-started?authuser=1). \n",
+    "\n",
+    "\n",
+    "\n",
+    "Please note that Google Places API is a paid service, and the cost is associated with the number of API calls made. Keep track of your usage to avoid any unexpected charges.\n",
+    "\n",
+    "\n",
+    "\n",
+    "The requests library is also needed, you can download it by using the following command: \n",
+    "\n",
+    "```python\n",
+    "pip install requests"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "from openai import OpenAI\n",
+    "import os\n",
+    "import requests\n",
+    "\n",
+    "client = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In this code snippet, we are defining a function `fetch_customer_profile` that accepts a `user_id` and returns a mock user profile.\n",
+    "\n",
+    "This function simulates an API call that fetches user data from a database. For this demo, we're using hard-coded data. The user profile contains various details such as the user's location (set to the coordinates of the Golden Gate Bridge for this example), preferences in food and activities, app usage metrics, recent interactions, and user rank.\n",
+    "\n",
+    "In a production environment, you would replace this hard-coded data with a real API call to your user database.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def fetch_customer_profile(user_id):\n",
+    "    # You can replace this with a real API call in the production code\n",
+    "    if user_id == \"user1234\":\n",
+    "        return {\n",
+    "            \"name\": \"John Doe\",\n",
+    "            \"location\": {\n",
+    "                \"latitude\": 37.7955,\n",
+    "                \"longitude\": -122.4026,\n",
+    "            },\n",
+    "            \"preferences\": {\n",
+    "                \"food\": [\"Italian\", \"Sushi\"],\n",
+    "                \"activities\": [\"Hiking\", \"Reading\"],\n",
+    "            },\n",
+    "            \"behavioral_metrics\": {\n",
+    "                \"app_usage\": {\n",
+    "                    \"daily\": 2,  # hours\n",
+    "                    \"weekly\": 14  # hours\n",
+    "                },\n",
+    "                \"favourite_post_categories\": [\"Nature\", \"Food\", \"Books\"],\n",
+    "                \"active_time\": \"Evening\",\n",
+    "            },\n",
+    "            \"recent_searches\": [\"Italian restaurants nearby\", \"Book clubs\"],\n",
+    "            \"recent_interactions\": [\"Liked a post about 'Best Pizzas in New York'\", \"Commented on a post about 'Central Park Trails'\"],\n",
+    "            \"user_rank\": \"Gold\",  # based on some internal ranking system\n",
+    "        }\n",
+    "    else:\n",
+    "        return None\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Requesting and processing data from Google Places API\n",
+    "\n",
+    "The function call_google_places_api serves to request information from the Google Places API and provide a list of the top two places based on a given place_type and optional food_preference. We've limited this function to the top two results to manage usage since this is a paid service. However, you can modify this to retrieve any number of results as per your requirement.\n",
+    "\n",
+    "The function is configured with a hardcoded location (set to the coordinates of the Transamerica Pyramid), your Google API key, and specific request parameters. Depending on the place_type, it formulates the appropriate API request URL. If the place_type is a restaurant and a food_preference is specified, it is included in the API request.\n",
+    "\n",
+    "After sending the GET request, the function checks the response status. If it's successful, it processes the JSON response, extracts the relevant details using the get_place_details function, and returns them in a human-readable format. If the request fails, it prints out the error for debugging.\n",
+    "\n",
+    "The get_place_details function is used to retrieve more detailed information about a place, given its place_id. It sends a GET request to the Google Place Details API and returns the result if the request is successful. If the request fails, it prints out the error for debugging.\n",
+    "\n",
+    "Both functions handle exceptions and return an error message if something goes wrong. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def get_place_details(place_id, api_key):\n",
+    "    URL = f\"https://maps.googleapis.com/maps/api/place/details/json?place_id={place_id}&key={api_key}\"\n",
+    "    response = requests.get(URL)\n",
+    "    if response.status_code == 200:\n",
+    "        result = json.loads(response.content)[\"result\"]\n",
+    "        return result\n",
+    "    else:\n",
+    "        print(f\"Google Place Details API request failed with status code {response.status_code}\")\n",
+    "        print(f\"Response content: {response.content}\")\n",
+    "        return None\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def call_google_places_api(user_id, place_type, food_preference=None):\n",
+    "    try:\n",
+    "        # Fetch customer profile\n",
+    "        customer_profile = fetch_customer_profile(user_id)\n",
+    "        if customer_profile is None:\n",
+    "            return \"I couldn't find your profile. Could you please verify your user ID?\"\n",
+    "\n",
+    "        # Get location from customer profile\n",
+    "        lat = customer_profile[\"location\"][\"latitude\"]\n",
+    "        lng = customer_profile[\"location\"][\"longitude\"]\n",
+    "\n",
+    "        API_KEY = os.getenv('GOOGLE_PLACES_API_KEY')  # retrieve API key from environment variable\n",
+    "        LOCATION = f\"{lat},{lng}\"\n",
+    "        RADIUS = 500  # search within a radius of 500 meters\n",
+    "        TYPE = place_type\n",
+    "\n",
+    "        # If the place_type is restaurant and food_preference is not None, include it in the API request\n",
+    "        if place_type == 'restaurant' and food_preference:\n",
+    "            URL = f\"https://maps.googleapis.com/maps/api/place/nearbysearch/json?location={LOCATION}&radius={RADIUS}&type={TYPE}&keyword={food_preference}&key={API_KEY}\"\n",
+    "        else:\n",
+    "            URL = f\"https://maps.googleapis.com/maps/api/place/nearbysearch/json?location={LOCATION}&radius={RADIUS}&type={TYPE}&key={API_KEY}\"\n",
+    "\n",
+    "        response = requests.get(URL)\n",
+    "        if response.status_code == 200:\n",
+    "            results = json.loads(response.content)[\"results\"]\n",
+    "            places = []\n",
+    "            for place in results[:2]:  # limit to top 2 results\n",
+    "                place_id = place.get(\"place_id\")\n",
+    "                place_details = get_place_details(place_id, API_KEY)  # Get the details of the place\n",
+    "\n",
+    "                place_name = place_details.get(\"name\", \"N/A\")\n",
+    "                place_types = next((t for t in place_details.get(\"types\", []) if t not in [\"food\", \"point_of_interest\"]), \"N/A\")  # Get the first type of the place, excluding \"food\" and \"point_of_interest\"\n",
+    "                place_rating = place_details.get(\"rating\", \"N/A\")  # Get the rating of the place\n",
+    "                total_ratings = place_details.get(\"user_ratings_total\", \"N/A\")  # Get the total number of ratings\n",
+    "                place_address = place_details.get(\"vicinity\", \"N/A\")  # Get the vicinity of the place\n",
+    "\n",
+    "                if ',' in place_address:  # If the address contains a comma\n",
+    "                    street_address = place_address.split(',')[0]  # Split by comma and keep only the first part\n",
+    "                else:\n",
+    "                    street_address = place_address\n",
+    "\n",
+    "                # Prepare the output string for this place\n",
+    "                place_info = f\"{place_name} is a {place_types} located at {street_address}. It has a rating of {place_rating} based on {total_ratings} user reviews.\"\n",
+    "\n",
+    "                places.append(place_info)\n",
+    "\n",
+    "            return places\n",
+    "        else:\n",
+    "            print(f\"Google Places API request failed with status code {response.status_code}\")\n",
+    "            print(f\"Response content: {response.content}\")  # print out the response content for debugging\n",
+    "            return []\n",
+    "    except Exception as e:\n",
+    "        print(f\"Error during the Google Places API call: {e}\")\n",
+    "        return []\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Generating user-specific recommendations with GPT-3.5-Turbo and Google Places API\n",
+    "\n",
+    "The function `provide_user_specific_recommendations` interacts with GPT-3.5-Turbo and the Google Places API to provide responses tailored to a user's preferences and location.\n",
+    "\n",
+    "First, it fetches the customer's profile using their `user_id`. If no profile is found, it returns an error message.\n",
+    "\n",
+    "With a valid profile, it extracts the customer's food preferences and then interacts with the OpenAI model. It provides an initial system message, giving context to the AI model about its role, user preferences, and the usage of the Google Places API function.\n",
+    "\n",
+    "The user input is also sent to the model as a message, and the function `call_google_places_api` is defined in the `functions` parameter for the AI model to call as needed.\n",
+    "\n",
+    "Finally, it processes the model's response. If the model makes a function call to the Google Places API, the function is executed with the appropriate arguments, and the names of nearby places are returned. If there are no such places or the request isn't understood, appropriate error messages are returned.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def provide_user_specific_recommendations(user_input, user_id):\n",
+    "    customer_profile = fetch_customer_profile(user_id)\n",
+    "    if customer_profile is None:\n",
+    "        return \"I couldn't find your profile. Could you please verify your user ID?\"\n",
+    "\n",
+    "    customer_profile_str = json.dumps(customer_profile)\n",
+    "\n",
+    "    food_preference = customer_profile.get('preferences', {}).get('food', [])[0] if customer_profile.get('preferences', {}).get('food') else None\n",
+    "\n",
+    "\n",
+    "    response = client.chat.completions.create(\n",
+    "        model=\"gpt-3.5-turbo\",\n",
+    "        messages=[\n",
+    "    {\n",
+    "        \"role\": \"system\",\n",
+    "        \"content\": f\"You are a sophisticated AI assistant, a specialist in user intent detection and interpretation. Your task is to perceive and respond to the user's needs, even when they're expressed in an indirect or direct manner. You excel in recognizing subtle cues: for example, if a user states they are 'hungry', you should assume they are seeking nearby dining options such as a restaurant or a cafe. If they indicate feeling 'tired', 'weary', or mention a long journey, interpret this as a request for accommodation options like hotels or guest houses. However, remember to navigate the fine line of interpretation and assumption: if a user's intent is unclear or can be interpreted in multiple ways, do not hesitate to politely ask for additional clarification. Make sure to tailor your responses to the user based on their preferences and past experiences which can be found here {customer_profile_str}\"\n",
+    "    },\n",
+    "    {\"role\": \"user\", \"content\": user_input}\n",
+    "],\n",
+    "        temperature=0,\n",
+    "        tools=[\n",
+    "            {\n",
+    "                \"type\": \"function\",\n",
+    "                \"function\" : {\n",
+    "                    \"name\": \"call_google_places_api\",\n",
+    "                    \"description\": \"This function calls the Google Places API to find the top places of a specified type near a specific location. It can be used when a user expresses a need (e.g., feeling hungry or tired) or wants to find a certain type of place (e.g., restaurant or hotel).\",\n",
+    "                    \"parameters\": {\n",
+    "                        \"type\": \"object\",\n",
+    "                        \"properties\": {\n",
+    "                            \"place_type\": {\n",
+    "                                \"type\": \"string\",\n",
+    "                                \"description\": \"The type of place to search for.\"\n",
+    "                            }\n",
+    "                        }\n",
+    "                    },\n",
+    "                    \"result\": {\n",
+    "                        \"type\": \"array\",\n",
+    "                        \"items\": {\n",
+    "                            \"type\": \"string\"\n",
+    "                        }\n",
+    "                    }\n",
+    "                }\n",
+    "            }\n",
+    "        ],\n",
+    "    )\n",
+    "\n",
+    "    print(response.choices[0].message.tool_calls)\n",
+    "\n",
+    "    if response.choices[0].finish_reason=='tool_calls':\n",
+    "        function_call = response.choices[0].message.tool_calls[0].function\n",
+    "        if function_call.name == \"call_google_places_api\":\n",
+    "            place_type = json.loads(function_call.arguments)[\"place_type\"]\n",
+    "            places = call_google_places_api(user_id, place_type, food_preference)\n",
+    "            if places:  # If the list of places is not empty\n",
+    "                return f\"Here are some places you might be interested in: {' '.join(places)}\"\n",
+    "            else:\n",
+    "                return \"I couldn't find any places of interest nearby.\"\n",
+    "\n",
+    "    return \"I am sorry, but I could not understand your request.\"\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Executing user-specific recommendations\n",
+    "\n",
+    "Upon execution, the function fetches the user's profile, interacts with the AI model, processes the model's response, calls the Google Places API if necessary, and ultimately returns a list of recommendations tailored to the user's preferences and location. The printed output would consist of these personalized recommendations.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[ChatCompletionMessageToolCall(id='call_Q1mXIi7D6GhobfE4tkruX7nB', function=Function(arguments='{\\n  \"place_type\": \"restaurant\"\\n}', name='call_google_places_api'), type='function')]\n",
+      "Here are some places you might be interested in: Sotto Mare is a restaurant located at 552 Green Street. It has a rating of 4.6 based on 3765 user reviews. Mona Lisa Restaurant is a restaurant located at 353 Columbus Avenue #3907. It has a rating of 4.4 based on 1888 user reviews.\n"
+     ]
+    }
+   ],
+   "source": [
+    "user_id = \"user1234\"\n",
+    "user_input = \"I'm hungry\"\n",
+    "output = provide_user_specific_recommendations(user_input, user_id)\n",
+    "print(output)\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.5"
+  },
+  "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/Get_embeddings_from_dataset.ipynb

@@ -0,0 +1,233 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Get embeddings from dataset\n",
+    "\n",
+    "This notebook gives an example on how to get embeddings from a large dataset.\n",
+    "\n",
+    "\n",
+    "## 1. Load the dataset\n",
+    "\n",
+    "The dataset used in this example is [fine-food reviews](https://www.kaggle.com/snap/amazon-fine-food-reviews) from Amazon. The dataset contains a total of 568,454 food reviews Amazon users left up to October 2012. We will use a subset of this dataset, consisting of 1,000 most recent reviews for illustration purposes. The reviews are in English and tend to be positive or negative. Each review has a ProductId, UserId, Score, review title (Summary) and review body (Text).\n",
+    "\n",
+    "We will combine the review summary and review text into a single combined text. The model will encode this combined text and it will output a single vector embedding."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To run this notebook, you will need to install: pandas, openai, transformers, plotly, matplotlib, scikit-learn, torch (transformer dep), torchvision, and scipy."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import tiktoken\n",
+    "\n",
+    "from utils.embeddings_utils import get_embedding"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "embedding_model = \"text-embedding-3-small\"\n",
+    "embedding_encoding = \"cl100k_base\"\n",
+    "max_tokens = 8000  # the maximum for text-embedding-3-small is 8191"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>Time</th>\n",
+       "      <th>ProductId</th>\n",
+       "      <th>UserId</th>\n",
+       "      <th>Score</th>\n",
+       "      <th>Summary</th>\n",
+       "      <th>Text</th>\n",
+       "      <th>combined</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>1351123200</td>\n",
+       "      <td>B003XPF9BO</td>\n",
+       "      <td>A3R7JR3FMEBXQB</td>\n",
+       "      <td>5</td>\n",
+       "      <td>where does one  start...and stop... with a tre...</td>\n",
+       "      <td>Wanted to save some to bring to my Chicago fam...</td>\n",
+       "      <td>Title: where does one  start...and stop... wit...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>1351123200</td>\n",
+       "      <td>B003JK537S</td>\n",
+       "      <td>A3JBPC3WFUT5ZP</td>\n",
+       "      <td>1</td>\n",
+       "      <td>Arrived in pieces</td>\n",
+       "      <td>Not pleased at all. When I opened the box, mos...</td>\n",
+       "      <td>Title: Arrived in pieces; Content: Not pleased...</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "         Time   ProductId          UserId  Score  \\\n",
+       "0  1351123200  B003XPF9BO  A3R7JR3FMEBXQB      5   \n",
+       "1  1351123200  B003JK537S  A3JBPC3WFUT5ZP      1   \n",
+       "\n",
+       "                                             Summary  \\\n",
+       "0  where does one  start...and stop... with a tre...   \n",
+       "1                                  Arrived in pieces   \n",
+       "\n",
+       "                                                Text  \\\n",
+       "0  Wanted to save some to bring to my Chicago fam...   \n",
+       "1  Not pleased at all. When I opened the box, mos...   \n",
+       "\n",
+       "                                            combined  \n",
+       "0  Title: where does one  start...and stop... wit...  \n",
+       "1  Title: Arrived in pieces; Content: Not pleased...  "
+      ]
+     },
+     "execution_count": 17,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# load & inspect dataset\n",
+    "input_datapath = \"data/fine_food_reviews_1k.csv\"  # to save space, we provide a pre-filtered dataset\n",
+    "df = pd.read_csv(input_datapath, index_col=0)\n",
+    "df = df[[\"Time\", \"ProductId\", \"UserId\", \"Score\", \"Summary\", \"Text\"]]\n",
+    "df = df.dropna()\n",
+    "df[\"combined\"] = (\n",
+    "    \"Title: \" + df.Summary.str.strip() + \"; Content: \" + df.Text.str.strip()\n",
+    ")\n",
+    "df.head(2)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "1000"
+      ]
+     },
+     "execution_count": 18,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# subsample to 1k most recent reviews and remove samples that are too long\n",
+    "top_n = 1000\n",
+    "df = df.sort_values(\"Time\").tail(top_n * 2)  # first cut to first 2k entries, assuming less than half will be filtered out\n",
+    "df.drop(\"Time\", axis=1, inplace=True)\n",
+    "\n",
+    "encoding = tiktoken.get_encoding(embedding_encoding)\n",
+    "\n",
+    "# omit reviews that are too long to embed\n",
+    "df[\"n_tokens\"] = df.combined.apply(lambda x: len(encoding.encode(x)))\n",
+    "df = df[df.n_tokens <= max_tokens].tail(top_n)\n",
+    "len(df)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Get embeddings and save them for future reuse"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Ensure you have your API key set in your environment per the README: https://github.com/openai/openai-python#usage\n",
+    "\n",
+    "# This may take a few minutes\n",
+    "df[\"embedding\"] = df.combined.apply(lambda x: get_embedding(x, model=embedding_model))\n",
+    "df.to_csv(\"data/fine_food_reviews_with_embeddings_1k.csv\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "a = get_embedding(\"hi\", model=embedding_model)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "openai",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.5"
+  },
+  "orig_nbformat": 4,
+  "vscode": {
+   "interpreter": {
+    "hash": "365536dcbde60510dc9073d6b991cd35db2d9bac356a11f5b64279a5e6708b97"
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/How_to_build_an_agent_with_the_node_sdk.mdx

@@ -0,0 +1,492 @@
+# How to build an agent with the Node.js SDK
+
+OpenAI functions enable your app to take action based on user inputs. This means that it can, e.g., search the web, send emails, or book tickets on behalf of your users, making it more powerful than a regular chatbot.
+
+In this tutorial, you will build an app that uses OpenAI functions along with the latest version of the Node.js SDK. The app runs in the browser, so you only need a code editor and, e.g., VS Code Live Server to follow along locally. Alternatively, write your code directly in the browser via [this code playground at Scrimba.](https://scrimba.com/scrim/c6r3LkU9)
+
+## What you will build
+
+Our app is a simple agent that helps you find activities in your area.
+It has access to two functions, `getLocation()` and `getCurrentWeather()`,
+which means it can figure out where you’re located and what the weather
+is at the moment.
+
+At this point, it's important to understand that
+OpenAI doesn't execute any code for you. It just tells your app which
+functions it should use in a given scenario, and then leaves it up to
+your app to invoke them.
+
+Once our agent knows your location and the weather, it'll use GPT’s
+internal knowledge to suggest suitable local activities for you.
+
+## Importing the SDK and authenticating with OpenAI
+
+We start by importing the OpenAI SDK at the top of our JavaScript file and authenticate with our API key, which we have stored as an environment variable.
+
+```js
+import OpenAI from "openai";
+
+const openai = new OpenAI({
+  apiKey: process.env.OPENAI_API_KEY,
+  dangerouslyAllowBrowser: true,
+});
+```
+
+Since we're running our code in a browser environment at Scrimba, we also need to set `dangerouslyAllowBrowser: true` to confirm we understand the risks involved with client-side API requests. Please note that you should move these requests over to a Node server in a production app.
+
+## Creating our two functions
+
+Next, we'll create the two functions. The first one - `getLocation` -
+uses the [IP API](https://ipapi.co/) to get the location of the
+user.
+
+```js
+async function getLocation() {
+  const response = await fetch("https://ipapi.co/json/");
+  const locationData = await response.json();
+  return locationData;
+}
+```
+
+The IP API returns a bunch of data about your location, including your
+latitude and longitude, which we’ll use as arguments in the second
+function `getCurrentWeather`. It uses the [Open Meteo
+API](https://open-meteo.com/) to get the current weather data, like
+this:
+
+```js
+async function getCurrentWeather(latitude, longitude) {
+  const url = `https://api.open-meteo.com/v1/forecast?latitude=${latitude}&longitude=${longitude}&hourly=apparent_temperature`;
+  const response = await fetch(url);
+  const weatherData = await response.json();
+  return weatherData;
+}
+```
+
+## Describing our functions for OpenAI
+
+For OpenAI to understand the purpose of these functions, we need to
+describe them using a specific schema. We'll create an array called
+`tools` that contains one object per function. Each object
+will have two keys: `type`, `function`, and the `function` key has 
+three subkeys: `name`, `description`, and `parameters`.
+
+```js
+const tools = [
+  {
+    type: "function",
+    function: {
+      name: "getCurrentWeather",
+      description: "Get the current weather in a given location",
+      parameters: {
+        type: "object",
+        properties: {
+          latitude: {
+            type: "string",
+          },
+          longitude: {
+            type: "string",
+          },
+        },
+        required: ["longitude", "latitude"],
+      },
+    }
+  },
+  {
+    type: "function",
+    function: {
+      name: "getLocation",
+      description: "Get the user's location based on their IP address",
+      parameters: {
+        type: "object",
+        properties: {},
+      },
+    }
+  },
+];
+```
+
+## Setting up the messages array
+
+We also need to define a `messages` array. This will keep track of all of the messages back and forth between our app and OpenAI.
+
+The first object in the array should always have the `role` property set to `"system"`, which tells OpenAI that this is how we want it to behave.
+
+```js
+const messages = [
+  {
+    role: "system",
+    content:
+      "You are a helpful assistant. Only use the functions you have been provided with.",
+  },
+];
+```
+
+## Creating the agent function
+
+We are now ready to build the logic of our app, which lives in the
+`agent` function. It is asynchronous and takes one argument: the
+`userInput`.
+
+We start by pushing the `userInput` to the messages array. This time, we set the `role` to `"user"`, so that OpenAI knows that this is the input from the user.
+
+```js
+async function agent(userInput) {
+  messages.push({
+    role: "user",
+    content: userInput,
+  });
+  const response = await openai.chat.completions.create({
+    model: "gpt-4",
+    messages: messages,
+    tools: tools,
+  });
+  console.log(response);
+}
+```
+
+Next, we'll send a request to the Chat completions endpoint via the
+`chat.completions.create()` method in the Node SDK. This method takes a
+configuration object as an argument. In it, we'll specify three
+properties:
+
+- `model` - Decides which AI model we want to use (in our case,
+  GPT-4).
+- `messages` - The entire history of messages between the user and the
+  AI up until this point.
+- `tools` - A list of tools the model may call. Currently, only 
+  functions are supported as a tool., we'll we use the `tools` array we 
+  created earlier.
+
+## Running our app with a simple input
+
+Let's try to run the `agent` with an input that requires a function call to give a suitable reply.
+
+```js
+agent("Where am I located right now?");
+```
+
+When we run the code above, we see the response from OpenAI logged out
+to the console like this:
+
+```js
+{
+    id: "chatcmpl-84ojoEJtyGnR6jRHK2Dl4zTtwsa7O",
+    object: "chat.completion",
+    created: 1696159040,
+    model: "gpt-4-0613",
+    choices: [{
+        index: 0,
+        message: {
+            role: "assistant",
+            content: null,
+            tool_calls: [
+              id: "call_CBwbo9qoXUn1kTR5pPuv6vR1",
+              type: "function",
+              function: {
+                name: "getLocation",
+                arguments: "{}"
+              }
+            ]
+        },
+        logprobs: null,
+        finish_reason: "tool_calls" // OpenAI wants us to call a function
+    }],
+    usage: {
+        prompt_tokens: 134,
+        completion_tokens: 6,
+        total_tokens: 140
+    }
+     system_fingerprint: null
+}
+```
+
+This response tells us that we should call one of our functions, as it contains the following key: `finish_reason: "tool_calls"`.
+
+The name of the function can be found in the
+`response.choices[0].message.tool_calls[0].function.name` key, which is set to
+`"getLocation"`.
+
+## Turning the OpenAI response into a function call
+
+Now that we have the name of the function as a string, we'll need to
+translate that into a function call. To help us with that, we'll gather
+both of our functions in an object called `availableTools`:
+
+```js
+const availableTools = {
+  getCurrentWeather,
+  getLocation,
+};
+```
+
+This is handy because we'll be able to access the `getLocation` function
+via bracket notation and the string we got back from OpenAI, like this:
+`availableTools["getLocation"]`.
+
+```js
+const { finish_reason, message } = response.choices[0];
+
+if (finish_reason === "tool_calls" && message.tool_calls) {
+  const functionName = message.tool_calls[0].function.name;
+  const functionToCall = availableTools[functionName];
+  const functionArgs = JSON.parse(message.tool_calls[0].function.arguments);
+  const functionArgsArr = Object.values(functionArgs);
+  const functionResponse = await functionToCall.apply(null, functionArgsArr);
+  console.log(functionResponse);
+}
+```
+
+We're also grabbing ahold of any arguments OpenAI wants us to pass into
+the function: `message.tool_calls[0].function.arguments`.
+However, we won't need any arguments for this first function call.
+
+If we run the code again with the same input
+(`"Where am I located right now?"`), we'll see that `functionResponse`
+is an object filled with location about where the user is located right
+now. In my case, that is Oslo, Norway.
+
+```js
+{ip: "193.212.60.170", network: "193.212.60.0/23", version: "IPv4", city: "Oslo", region: "Oslo County", region_code: "03", country: "NO", country_name: "Norway", country_code: "NO", country_code_iso3: "NOR", country_capital: "Oslo", country_tld: ".no", continent_code: "EU", in_eu: false, postal: "0026", latitude: 59.955, longitude: 10.859, timezone: "Europe/Oslo", utc_offset: "+0200", country_calling_code: "+47", currency: "NOK", currency_name: "Krone", languages: "no,nb,nn,se,fi", country_area: 324220, country_population: 5314336, asn: "AS2119", org: "Telenor Norge AS"}
+```
+
+We'll add this data to a new item in the `messages` array, where we also
+specify the name of the function we called.
+
+```js
+messages.push({
+  role: "function",
+  name: functionName,
+  content: `The result of the last function was this: ${JSON.stringify(
+    functionResponse
+  )}
+  `,
+});
+```
+
+Notice that the `role` is set to `"function"`. This tells OpenAI
+that the `content` parameter contains the result of the function call
+and not the input from the user.
+
+At this point, we need to send a new request to OpenAI with this updated
+`messages` array. However, we don’t want to hard code a new function
+call, as our agent might need to go back and forth between itself and
+GPT several times until it has found the final answer for the user.
+
+This can be solved in several different ways, e.g. recursion, a
+while-loop, or a for-loop. We'll use a good old for-loop for the sake of
+simplicity.
+
+## Creating the loop
+
+At the top of the `agent` function, we'll create a loop that lets us run
+the entire procedure up to five times.
+
+If we get back `finish_reason: "tool_calls"` from GPT, we'll just
+push the result of the function call to the `messages` array and jump to
+the next iteration of the loop, triggering a new request.
+
+If we get `finish_reason: "stop"` back, then GPT has found a suitable
+answer, so we'll return the function and cancel the loop.
+
+```js
+for (let i = 0; i < 5; i++) {
+  const response = await openai.chat.completions.create({
+    model: "gpt-4",
+    messages: messages,
+    tools: tools,
+  });
+  const { finish_reason, message } = response.choices[0];
+
+  if (finish_reason === "tool_calls" && message.tool_calls) {
+    const functionName = message.tool_calls[0].function.name;
+    const functionToCall = availableTools[functionName];
+    const functionArgs = JSON.parse(message.tool_calls[0].function.arguments);
+    const functionArgsArr = Object.values(functionArgs);
+    const functionResponse = await functionToCall.apply(null, functionArgsArr);
+
+    messages.push({
+      role: "function",
+      name: functionName,
+      content: `
+          The result of the last function was this: ${JSON.stringify(
+            functionResponse
+          )}
+          `,
+    });
+  } else if (finish_reason === "stop") {
+    messages.push(message);
+    return message.content;
+  }
+}
+return "The maximum number of iterations has been met without a suitable answer. Please try again with a more specific input.";
+```
+
+If we don't see a `finish_reason: "stop"` within our five iterations,
+we'll return a message saying we couldn’t find a suitable answer.
+
+## Running the final app
+
+At this point, we are ready to try our app! I'll ask the agent to
+suggest some activities based on my location and the current weather.
+
+```js
+const response = await agent(
+  "Please suggest some activities based on my location and the current weather."
+);
+console.log(response);
+```
+
+Here's what we see in the console (formatted to make it easier to read):
+
+```js
+Based on your current location in Oslo, Norway and the weather (15°C and snowy),
+here are some activity suggestions:
+
+1. A visit to the Oslo Winter Park for skiing or snowboarding.
+2. Enjoy a cosy day at a local café or restaurant.
+3. Visit one of Oslo's many museums. The Fram Museum or Viking Ship Museum offer interesting insights into Norway’s seafaring history.
+4. Take a stroll in the snowy streets and enjoy the beautiful winter landscape.
+5. Enjoy a nice book by the fireplace in a local library.
+6. Take a fjord sightseeing cruise to enjoy the snowy landscapes.
+
+Always remember to bundle up and stay warm. Enjoy your day!
+```
+
+If we peak under the hood, and log out `response.choices[0].message` in
+each iteration of the loop, we'll see that GPT has instructed us to use
+both our functions before coming up with an answer.
+
+First, it tells us to call the `getLocation` function. Then it tells us
+to call the `getCurrentWeather` function with
+`"longitude": "10.859", "latitude": "59.955"` passed in as the
+arguments. This is data it got back from the first function call we did.
+
+```js
+{"role":"assistant","content":null,"tool_calls":[{"id":"call_Cn1KH8mtHQ2AMbyNwNJTweEP","type":"function","function":{"name":"getLocation","arguments":"{}"}}]}
+{"role":"assistant","content":null,"tool_calls":[{"id":"call_uc1oozJfGTvYEfIzzcsfXfOl","type":"function","function":{"name":"getCurrentWeather","arguments":"{\n\"latitude\": \"10.859\",\n\"longitude\": \"59.955\"\n}"}}]}
+```
+
+You've now built an AI agent using OpenAI functions and the Node.js SDK! If you're looking for an extra challenge, consider enhancing this app. For example, you could add a function that fetches up-to-date information on events and activities in the user's location.
+
+Happy coding!
+
+<details>
+<summary>Complete code</summary>
+
+```js
+import OpenAI from "openai";
+
+const openai = new OpenAI({
+  apiKey: process.env.OPENAI_API_KEY,
+  dangerouslyAllowBrowser: true,
+});
+
+async function getLocation() {
+  const response = await fetch("https://ipapi.co/json/");
+  const locationData = await response.json();
+  return locationData;
+}
+
+async function getCurrentWeather(latitude, longitude) {
+  const url = `https://api.open-meteo.com/v1/forecast?latitude=${latitude}&longitude=${longitude}&hourly=apparent_temperature`;
+  const response = await fetch(url);
+  const weatherData = await response.json();
+  return weatherData;
+}
+
+const tools = [
+  {
+    type: "function",
+    function: {
+      name: "getCurrentWeather",
+      description: "Get the current weather in a given location",
+      parameters: {
+        type: "object",
+        properties: {
+          latitude: {
+            type: "string",
+          },
+          longitude: {
+            type: "string",
+          },
+        },
+        required: ["longitude", "latitude"],
+      },
+    }
+  },
+  {
+    type: "function",
+    function: {
+      name: "getLocation",
+      description: "Get the user's location based on their IP address",
+      parameters: {
+        type: "object",
+        properties: {},
+      },
+    }
+  },
+];
+
+const availableTools = {
+  getCurrentWeather,
+  getLocation,
+};
+
+const messages = [
+  {
+    role: "system",
+    content: `You are a helpful assistant. Only use the functions you have been provided with.`,
+  },
+];
+
+async function agent(userInput) {
+  messages.push({
+    role: "user",
+    content: userInput,
+  });
+
+  for (let i = 0; i < 5; i++) {
+    const response = await openai.chat.completions.create({
+      model: "gpt-4",
+      messages: messages,
+      tools: tools,
+    });
+
+    const { finish_reason, message } = response.choices[0];
+
+    if (finish_reason === "tool_calls" && message.tool_calls) {
+      const functionName = message.tool_calls[0].function.name;
+      const functionToCall = availableTools[functionName];
+      const functionArgs = JSON.parse(message.tool_calls[0].function.arguments);
+      const functionArgsArr = Object.values(functionArgs);
+      const functionResponse = await functionToCall.apply(
+        null,
+        functionArgsArr
+      );
+
+      messages.push({
+        role: "function",
+        name: functionName,
+        content: `
+                The result of the last function was this: ${JSON.stringify(
+                  functionResponse
+                )}
+                `,
+      });
+    } else if (finish_reason === "stop") {
+      messages.push(message);
+      return message.content;
+    }
+  }
+  return "The maximum number of iterations has been met without a suitable answer. Please try again with a more specific input.";
+}
+
+const response = await agent(
+  "Please suggest some activities based on my location and the weather."
+);
+
+console.log("response:", response);
+```
+
+</details>

examples/How_to_call_functions_for_knowledge_retrieval.ipynb

@@ -0,0 +1,795 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "3e67f200",
+   "metadata": {},
+   "source": [
+    "# How to use functions with a knowledge base\n",
+    "\n",
+    "This notebook builds on the concepts in the [argument generation](How_to_call_functions_with_chat_models.ipynb) notebook, by creating an agent with access to a knowledge base and two functions that it can call based on the user requirement.\n",
+    "\n",
+    "We'll create an agent that uses data from arXiv to answer questions about academic subjects. It has two functions at its disposal:\n",
+    "- **get_articles**: A function that gets arXiv articles on a subject and summarizes them for the user with links.\n",
+    "- **read_article_and_summarize**: This function takes one of the previously searched articles, reads it in its entirety and summarizes the core argument, evidence and conclusions.\n",
+    "\n",
+    "This will get you comfortable with a multi-function workflow that can choose from multiple services, and where some of the data from the first function is persisted to be used by the second.\n",
+    "\n",
+    "## Walkthrough\n",
+    "\n",
+    "This cookbook takes you through the following workflow:\n",
+    "\n",
+    "- **Search utilities:** Creating the two functions that access arXiv for answers.\n",
+    "- **Configure Agent:** Building up the Agent behaviour that will assess the need for a function and, if one is required, call that function and present results back to the agent.\n",
+    "- **arXiv conversation:** Put all of this together in live conversation.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "80e71f33",
+   "metadata": {
+    "pycharm": {
+     "is_executing": true
+    }
+   },
+   "outputs": [],
+   "source": [
+    "!pip install scipy --quiet\n",
+    "!pip install tenacity --quiet\n",
+    "!pip install tiktoken==0.3.3 --quiet\n",
+    "!pip install termcolor --quiet\n",
+    "!pip install openai --quiet\n",
+    "!pip install arxiv --quiet\n",
+    "!pip install pandas --quiet\n",
+    "!pip install PyPDF2 --quiet\n",
+    "!pip install tqdm --quiet"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "dab872c5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import arxiv\n",
+    "import ast\n",
+    "import concurrent\n",
+    "import json\n",
+    "import os\n",
+    "import pandas as pd\n",
+    "import tiktoken\n",
+    "from csv import writer\n",
+    "from IPython.display import display, Markdown, Latex\n",
+    "from openai import OpenAI\n",
+    "from PyPDF2 import PdfReader\n",
+    "from scipy import spatial\n",
+    "from tenacity import retry, wait_random_exponential, stop_after_attempt\n",
+    "from tqdm import tqdm\n",
+    "from termcolor import colored\n",
+    "\n",
+    "GPT_MODEL = \"gpt-4o-mini\"\n",
+    "EMBEDDING_MODEL = \"text-embedding-ada-002\"\n",
+    "client = OpenAI()"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "f2e47962",
+   "metadata": {},
+   "source": [
+    "## Search utilities\n",
+    "\n",
+    "We'll first set up some utilities that will underpin our two functions.\n",
+    "\n",
+    "Downloaded papers will be stored in a directory (we use ```./data/papers``` here). We create a file ```arxiv_library.csv``` to store the embeddings and details for downloaded papers to retrieve against using ```summarize_text```."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "2de5d32d",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Directory './data/papers' already exists.\n"
+     ]
+    }
+   ],
+   "source": [
+    "directory = './data/papers'\n",
+    "\n",
+    "# Check if the directory already exists\n",
+    "if not os.path.exists(directory):\n",
+    "    # If the directory doesn't exist, create it and any necessary intermediate directories\n",
+    "    os.makedirs(directory)\n",
+    "    print(f\"Directory '{directory}' created successfully.\")\n",
+    "else:\n",
+    "    # If the directory already exists, print a message indicating it\n",
+    "    print(f\"Directory '{directory}' already exists.\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "ae5cb7a1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Set a directory to store downloaded papers\n",
+    "data_dir = os.path.join(os.curdir, \"data\", \"papers\")\n",
+    "paper_dir_filepath = \"./data/papers/arxiv_library.csv\"\n",
+    "\n",
+    "# Generate a blank dataframe where we can store downloaded files\n",
+    "df = pd.DataFrame(list())\n",
+    "df.to_csv(paper_dir_filepath)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "57217b9d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "@retry(wait=wait_random_exponential(min=1, max=40), stop=stop_after_attempt(3))\n",
+    "def embedding_request(text):\n",
+    "    response = client.embeddings.create(input=text, model=EMBEDDING_MODEL)\n",
+    "    return response\n",
+    "\n",
+    "\n",
+    "@retry(wait=wait_random_exponential(min=1, max=40), stop=stop_after_attempt(3))\n",
+    "def get_articles(query, library=paper_dir_filepath, top_k=10):\n",
+    "    \"\"\"This function gets the top_k articles based on a user's query, sorted by relevance.\n",
+    "    It also downloads the files and stores them in arxiv_library.csv to be retrieved by the read_article_and_summarize.\n",
+    "    \"\"\"\n",
+    "    client = arxiv.Client()\n",
+    "    search = arxiv.Search(\n",
+    "        query = query,\n",
+    "        max_results = top_k\n",
+    "    )\n",
+    "    result_list = []\n",
+    "    for result in client.results(search):\n",
+    "        result_dict = {}\n",
+    "        result_dict.update({\"title\": result.title})\n",
+    "        result_dict.update({\"summary\": result.summary})\n",
+    "\n",
+    "        # Taking the first url provided\n",
+    "        result_dict.update({\"article_url\": [x.href for x in result.links][0]})\n",
+    "        result_dict.update({\"pdf_url\": [x.href for x in result.links][1]})\n",
+    "        result_list.append(result_dict)\n",
+    "\n",
+    "        # Store references in library file\n",
+    "        response = embedding_request(text=result.title)\n",
+    "        file_reference = [\n",
+    "            result.title,\n",
+    "            result.download_pdf(data_dir),\n",
+    "            response.data[0].embedding,\n",
+    "        ]\n",
+    "\n",
+    "        # Write to file\n",
+    "        with open(library, \"a\") as f_object:\n",
+    "            writer_object = writer(f_object)\n",
+    "            writer_object.writerow(file_reference)\n",
+    "            f_object.close()\n",
+    "    return result_list\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "dda02bdb",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'title': 'Proximal Policy Optimization and its Dynamic Version for Sequence Generation',\n",
+       " 'summary': 'In sequence generation task, many works use policy gradient for model\\noptimization to tackle the intractable backpropagation issue when maximizing\\nthe non-differentiable evaluation metrics or fooling the discriminator in\\nadversarial learning. In this paper, we replace policy gradient with proximal\\npolicy optimization (PPO), which is a proved more efficient reinforcement\\nlearning algorithm, and propose a dynamic approach for PPO (PPO-dynamic). We\\ndemonstrate the efficacy of PPO and PPO-dynamic on conditional sequence\\ngeneration tasks including synthetic experiment and chit-chat chatbot. The\\nresults show that PPO and PPO-dynamic can beat policy gradient by stability and\\nperformance.',\n",
+       " 'article_url': 'http://arxiv.org/abs/1808.07982v1',\n",
+       " 'pdf_url': 'http://arxiv.org/pdf/1808.07982v1'}"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# Test that the search is working\n",
+    "result_output = get_articles(\"ppo reinforcement learning\")\n",
+    "result_output[0]\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "11675627",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def strings_ranked_by_relatedness(\n",
+    "    query: str,\n",
+    "    df: pd.DataFrame,\n",
+    "    relatedness_fn=lambda x, y: 1 - spatial.distance.cosine(x, y),\n",
+    "    top_n: int = 100,\n",
+    ") -> list[str]:\n",
+    "    \"\"\"Returns a list of strings and relatednesses, sorted from most related to least.\"\"\"\n",
+    "    query_embedding_response = embedding_request(query)\n",
+    "    query_embedding = query_embedding_response.data[0].embedding\n",
+    "    strings_and_relatednesses = [\n",
+    "        (row[\"filepath\"], relatedness_fn(query_embedding, row[\"embedding\"]))\n",
+    "        for i, row in df.iterrows()\n",
+    "    ]\n",
+    "    strings_and_relatednesses.sort(key=lambda x: x[1], reverse=True)\n",
+    "    strings, relatednesses = zip(*strings_and_relatednesses)\n",
+    "    return strings[:top_n]\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "7211df2c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def read_pdf(filepath):\n",
+    "    \"\"\"Takes a filepath to a PDF and returns a string of the PDF's contents\"\"\"\n",
+    "    # creating a pdf reader object\n",
+    "    reader = PdfReader(filepath)\n",
+    "    pdf_text = \"\"\n",
+    "    page_number = 0\n",
+    "    for page in reader.pages:\n",
+    "        page_number += 1\n",
+    "        pdf_text += page.extract_text() + f\"\\nPage Number: {page_number}\"\n",
+    "    return pdf_text\n",
+    "\n",
+    "\n",
+    "# Split a text into smaller chunks of size n, preferably ending at the end of a sentence\n",
+    "def create_chunks(text, n, tokenizer):\n",
+    "    \"\"\"Returns successive n-sized chunks from provided text.\"\"\"\n",
+    "    tokens = tokenizer.encode(text)\n",
+    "    i = 0\n",
+    "    while i < len(tokens):\n",
+    "        # Find the nearest end of sentence within a range of 0.5 * n and 1.5 * n tokens\n",
+    "        j = min(i + int(1.5 * n), len(tokens))\n",
+    "        while j > i + int(0.5 * n):\n",
+    "            # Decode the tokens and check for full stop or newline\n",
+    "            chunk = tokenizer.decode(tokens[i:j])\n",
+    "            if chunk.endswith(\".\") or chunk.endswith(\"\\n\"):\n",
+    "                break\n",
+    "            j -= 1\n",
+    "        # If no end of sentence found, use n tokens as the chunk size\n",
+    "        if j == i + int(0.5 * n):\n",
+    "            j = min(i + n, len(tokens))\n",
+    "        yield tokens[i:j]\n",
+    "        i = j\n",
+    "\n",
+    "\n",
+    "def extract_chunk(content, template_prompt):\n",
+    "    \"\"\"This function applies a prompt to some input content. In this case it returns a summarized chunk of text\"\"\"\n",
+    "    prompt = template_prompt + content\n",
+    "    response = client.chat.completions.create(\n",
+    "        model=GPT_MODEL, messages=[{\"role\": \"user\", \"content\": prompt}], temperature=0\n",
+    "    )\n",
+    "    return response.choices[0].message.content\n",
+    "\n",
+    "\n",
+    "def summarize_text(query):\n",
+    "    \"\"\"This function does the following:\n",
+    "    - Reads in the arxiv_library.csv file in including the embeddings\n",
+    "    - Finds the closest file to the user's query\n",
+    "    - Scrapes the text out of the file and chunks it\n",
+    "    - Summarizes each chunk in parallel\n",
+    "    - Does one final summary and returns this to the user\"\"\"\n",
+    "\n",
+    "    # A prompt to dictate how the recursive summarizations should approach the input paper\n",
+    "    summary_prompt = \"\"\"Summarize this text from an academic paper. Extract any key points with reasoning.\\n\\nContent:\"\"\"\n",
+    "\n",
+    "    # If the library is empty (no searches have been performed yet), we perform one and download the results\n",
+    "    library_df = pd.read_csv(paper_dir_filepath).reset_index()\n",
+    "    if len(library_df) == 0:\n",
+    "        print(\"No papers searched yet, downloading first.\")\n",
+    "        get_articles(query)\n",
+    "        print(\"Papers downloaded, continuing\")\n",
+    "        library_df = pd.read_csv(paper_dir_filepath).reset_index()\n",
+    "    else:\n",
+    "        print(\"Existing papers found... Articles:\", len(library_df))\n",
+    "    library_df.columns = [\"title\", \"filepath\", \"embedding\"]\n",
+    "    library_df[\"embedding\"] = library_df[\"embedding\"].apply(ast.literal_eval)\n",
+    "    strings = strings_ranked_by_relatedness(query, library_df, top_n=1)\n",
+    "    print(\"Chunking text from paper\")\n",
+    "    pdf_text = read_pdf(strings[0])\n",
+    "\n",
+    "    # Initialise tokenizer\n",
+    "    tokenizer = tiktoken.get_encoding(\"cl100k_base\")\n",
+    "    results = \"\"\n",
+    "\n",
+    "    # Chunk up the document into 1500 token chunks\n",
+    "    chunks = create_chunks(pdf_text, 1500, tokenizer)\n",
+    "    text_chunks = [tokenizer.decode(chunk) for chunk in chunks]\n",
+    "    print(\"Summarizing each chunk of text\")\n",
+    "\n",
+    "    # Parallel process the summaries\n",
+    "    with concurrent.futures.ThreadPoolExecutor(\n",
+    "        max_workers=len(text_chunks)\n",
+    "    ) as executor:\n",
+    "        futures = [\n",
+    "            executor.submit(extract_chunk, chunk, summary_prompt)\n",
+    "            for chunk in text_chunks\n",
+    "        ]\n",
+    "        with tqdm(total=len(text_chunks)) as pbar:\n",
+    "            for _ in concurrent.futures.as_completed(futures):\n",
+    "                pbar.update(1)\n",
+    "        for future in futures:\n",
+    "            data = future.result()\n",
+    "            results += data\n",
+    "\n",
+    "    # Final summary\n",
+    "    print(\"Summarizing into overall summary\")\n",
+    "    response = client.chat.completions.create(\n",
+    "        model=GPT_MODEL,\n",
+    "        messages=[\n",
+    "            {\n",
+    "                \"role\": \"user\",\n",
+    "                \"content\": f\"\"\"Write a summary collated from this collection of key points extracted from an academic paper.\n",
+    "                        The summary should highlight the core argument, conclusions and evidence, and answer the user's query.\n",
+    "                        User query: {query}\n",
+    "                        The summary should be structured in bulleted lists following the headings Core Argument, Evidence, and Conclusions.\n",
+    "                        Key points:\\n{results}\\nSummary:\\n\"\"\",\n",
+    "            }\n",
+    "        ],\n",
+    "        temperature=0,\n",
+    "    )\n",
+    "    return response\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "898b94d4",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Existing papers found... Articles: 10\n",
+      "Chunking text from paper\n",
+      "Summarizing each chunk of text\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "100%|███████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:05<00:00,  1.40s/it]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Summarizing into overall summary\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Test the summarize_text function works\n",
+    "chat_test_response = summarize_text(\"PPO reinforcement learning sequence generation\")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "c715f60d",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/markdown": [
+       "### Core Argument\n",
+       "- The paper argues that Proximal Policy Optimization (PPO) and its dynamic variant (PPO-dynamic) significantly improve sequence generation tasks, particularly for chit-chat chatbots, by addressing the instability and suboptimal performance associated with traditional policy gradient methods.\n",
+       "\n",
+       "### Evidence\n",
+       "- **Challenges with Traditional Methods**: Traditional policy gradient methods, like REINFORCE, suffer from unstable training and poor performance due to large updates and similar action tendencies, especially in non-differentiable evaluation contexts (e.g., BLEU scores).\n",
+       "- **PPO Advantages**: PPO regularizes policy updates, enhancing training stability and enabling the generation of coherent and diverse chatbot responses.\n",
+       "- **Dynamic PPO Approach**: PPO-dynamic introduces adaptive constraints on KL-divergence, allowing for dynamic adjustments based on action probabilities, which leads to improved training performance.\n",
+       "- **Experimental Validation**: The authors conducted experiments on synthetic counting tasks and real-world chit-chat scenarios, demonstrating that PPO and PPO-dynamic outperform traditional methods like REINFORCE and SeqGAN in terms of stability and performance metrics (e.g., BLEU-2 scores).\n",
+       "- **Results**: PPO-dynamic showed faster convergence and higher precision in the counting task, and it achieved the best performance in the chit-chat task, indicating its effectiveness in generating diverse and contextually appropriate responses.\n",
+       "\n",
+       "### Conclusions\n",
+       "- The introduction of PPO and PPO-dynamic enhances the training stability and output diversity in sequence generation tasks, making them more suitable for applications like chatbots.\n",
+       "- The dynamic variant of PPO not only improves performance but also accelerates convergence, addressing the limitations of traditional policy gradient methods and providing a robust framework for reinforcement learning in sequence generation."
+      ],
+      "text/plain": [
+       "<IPython.core.display.Markdown object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "display(Markdown(chat_test_response.choices[0].message.content))\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "dab07e98",
+   "metadata": {},
+   "source": [
+    "## Configure Agent\n",
+    "\n",
+    "We'll create our agent in this step, including a ```Conversation``` class to support multiple turns with the API, and some Python functions to enable interaction between the ```ChatCompletion``` API and our knowledge base functions."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "77a6fb4f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "@retry(wait=wait_random_exponential(min=1, max=40), stop=stop_after_attempt(3))\n",
+    "def chat_completion_request(messages, functions=None, model=GPT_MODEL):\n",
+    "    try:\n",
+    "        response = client.chat.completions.create(\n",
+    "            model=model,\n",
+    "            messages=messages,\n",
+    "            functions=functions,\n",
+    "        )\n",
+    "        return response\n",
+    "    except Exception as e:\n",
+    "        print(\"Unable to generate ChatCompletion response\")\n",
+    "        print(f\"Exception: {e}\")\n",
+    "        return e\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "73f7672d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class Conversation:\n",
+    "    def __init__(self):\n",
+    "        self.conversation_history = []\n",
+    "\n",
+    "    def add_message(self, role, content):\n",
+    "        message = {\"role\": role, \"content\": content}\n",
+    "        self.conversation_history.append(message)\n",
+    "\n",
+    "    def display_conversation(self, detailed=False):\n",
+    "        role_to_color = {\n",
+    "            \"system\": \"red\",\n",
+    "            \"user\": \"green\",\n",
+    "            \"assistant\": \"blue\",\n",
+    "            \"function\": \"magenta\",\n",
+    "        }\n",
+    "        for message in self.conversation_history:\n",
+    "            print(\n",
+    "                colored(\n",
+    "                    f\"{message['role']}: {message['content']}\\n\\n\",\n",
+    "                    role_to_color[message[\"role\"]],\n",
+    "                )\n",
+    "            )"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "978b7877",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Initiate our get_articles and read_article_and_summarize functions\n",
+    "arxiv_functions = [\n",
+    "    {\n",
+    "        \"name\": \"get_articles\",\n",
+    "        \"description\": \"\"\"Use this function to get academic papers from arXiv to answer user questions.\"\"\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"query\": {\n",
+    "                    \"type\": \"string\",\n",
+    "                    \"description\": f\"\"\"\n",
+    "                            User query in JSON. Responses should be summarized and should include the article URL reference\n",
+    "                            \"\"\",\n",
+    "                }\n",
+    "            },\n",
+    "            \"required\": [\"query\"],\n",
+    "        },\n",
+    "    },\n",
+    "    {\n",
+    "        \"name\": \"read_article_and_summarize\",\n",
+    "        \"description\": \"\"\"Use this function to read whole papers and provide a summary for users.\n",
+    "        You should NEVER call this function before get_articles has been called in the conversation.\"\"\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"query\": {\n",
+    "                    \"type\": \"string\",\n",
+    "                    \"description\": f\"\"\"\n",
+    "                            Description of the article in plain text based on the user's query\n",
+    "                            \"\"\",\n",
+    "                }\n",
+    "            },\n",
+    "            \"required\": [\"query\"],\n",
+    "        },\n",
+    "    }\n",
+    "]\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "0c88ae15",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def chat_completion_with_function_execution(messages, functions=[None]):\n",
+    "    \"\"\"This function makes a ChatCompletion API call with the option of adding functions\"\"\"\n",
+    "    response = chat_completion_request(messages, functions)\n",
+    "    full_message = response.choices[0]\n",
+    "    if full_message.finish_reason == \"function_call\":\n",
+    "        print(f\"Function generation requested, calling function\")\n",
+    "        return call_arxiv_function(messages, full_message)\n",
+    "    else:\n",
+    "        print(f\"Function not required, responding to user\")\n",
+    "        return response\n",
+    "\n",
+    "\n",
+    "def call_arxiv_function(messages, full_message):\n",
+    "    \"\"\"Function calling function which executes function calls when the model believes it is necessary.\n",
+    "    Currently extended by adding clauses to this if statement.\"\"\"\n",
+    "\n",
+    "    if full_message.message.function_call.name == \"get_articles\":\n",
+    "        try:\n",
+    "            parsed_output = json.loads(\n",
+    "                full_message.message.function_call.arguments\n",
+    "            )\n",
+    "            print(\"Getting search results\")\n",
+    "            results = get_articles(parsed_output[\"query\"])\n",
+    "        except Exception as e:\n",
+    "            print(parsed_output)\n",
+    "            print(f\"Function execution failed\")\n",
+    "            print(f\"Error message: {e}\")\n",
+    "        messages.append(\n",
+    "            {\n",
+    "                \"role\": \"function\",\n",
+    "                \"name\": full_message.message.function_call.name,\n",
+    "                \"content\": str(results),\n",
+    "            }\n",
+    "        )\n",
+    "        try:\n",
+    "            print(\"Got search results, summarizing content\")\n",
+    "            response = chat_completion_request(messages)\n",
+    "            return response\n",
+    "        except Exception as e:\n",
+    "            print(type(e))\n",
+    "            raise Exception(\"Function chat request failed\")\n",
+    "\n",
+    "    elif (\n",
+    "        full_message.message.function_call.name == \"read_article_and_summarize\"\n",
+    "    ):\n",
+    "        parsed_output = json.loads(\n",
+    "            full_message.message.function_call.arguments\n",
+    "        )\n",
+    "        print(\"Finding and reading paper\")\n",
+    "        summary = summarize_text(parsed_output[\"query\"])\n",
+    "        return summary\n",
+    "\n",
+    "    else:\n",
+    "        raise Exception(\"Function does not exist and cannot be called\")\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "dd3e7868",
+   "metadata": {},
+   "source": [
+    "## arXiv conversation\n",
+    "\n",
+    "Let's put this all together by testing our functions out in conversation."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "id": "c39a1d80",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Start with a system message\n",
+    "paper_system_message = \"\"\"You are arXivGPT, a helpful assistant pulls academic papers to answer user questions.\n",
+    "You summarize the papers clearly so the customer can decide which to read to answer their question.\n",
+    "You always provide the article_url and title so the user can understand the name of the paper and click through to access it.\n",
+    "Begin!\"\"\"\n",
+    "paper_conversation = Conversation()\n",
+    "paper_conversation.add_message(\"system\", paper_system_message)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "id": "253fd0f7",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Function generation requested, calling function\n",
+      "Getting search results\n",
+      "Got search results, summarizing content\n"
+     ]
+    },
+    {
+     "data": {
+      "text/markdown": [
+       "Here are some recent papers that discuss Proximal Policy Optimization (PPO) in reinforcement learning, explaining its mechanics and various enhancements:\n",
+       "\n",
+       "1. **[Proximal Policy Optimization and its Dynamic Version for Sequence Generation](http://arxiv.org/abs/1808.07982v1)**  \n",
+       "   - *Summary:* This paper applies PPO to sequence generation tasks, demonstrating that it outperforms traditional policy gradient methods in terms of stability and performance. It introduces a dynamic version of PPO for these tasks.\n",
+       "   - [PDF](http://arxiv.org/pdf/1808.07982v1)\n",
+       "\n",
+       "2. **[CIM-PPO: Proximal Policy Optimization with Liu-Correntropy Induced Metric](http://arxiv.org/abs/2110.10522v3)**  \n",
+       "   - *Summary:* This work investigates the asymmetry in KL divergence in PPO-KL and proposes PPO-CIM as an enhanced version with lower computation costs and improved policy updates, validated through experiments on continuous-action tasks.\n",
+       "   - [PDF](http://arxiv.org/pdf/2110.10522v3)\n",
+       "\n",
+       "3. **[A2C is a special case of PPO](http://arxiv.org/abs/2205.09123v1)**  \n",
+       "   - *Summary:* This paper shows that A2C can be viewed as a special case of PPO, providing theoretical justifications and empirical evidence demonstrating their equivalence under controlled conditions.\n",
+       "   - [PDF](http://arxiv.org/pdf/2205.09123v1)\n",
+       "\n",
+       "4. **[Proximal Policy Optimization via Enhanced Exploration Efficiency](http://arxiv.org/abs/2011.05525v1)**  \n",
+       "   - *Summary:* This paper enhances the PPO algorithm by improving exploration strategies, proposing IEM-PPO, which shows better sample efficiency and rewards than standard methods in complex environments.\n",
+       "   - [PDF](http://arxiv.org/pdf/2011.05525v1)\n",
+       "\n",
+       "5. **[ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models](http://arxiv.org/abs/2310.10505v4)**  \n",
+       "   - *Summary:* The ReMax method is proposed as an alternative to PPO for training large language models, reducing hyper-parameter tuning complexities and enhancing training efficiency.\n",
+       "   - [PDF](http://arxiv.org/pdf/2310.10505v4)\n",
+       "\n",
+       "6. **[Reward Scale Robustness for Proximal Policy Optimization via DreamerV3 Tricks](http://arxiv.org/abs/2310.17805v1)**  \n",
+       "   - *Summary:* This work examines the applicability of DreamerV3's tricks to PPO, revealing mixed outcomes and providing insights into the clipping mechanism in PPO's performance.\n",
+       "   - [PDF](http://arxiv.org/pdf/2310.17805v1)\n",
+       "\n",
+       "7. **[Neural PPO-Clip Attains Global Optimality: A Hinge Loss Perspective](http://arxiv.org/abs/2110.13799v4)**  \n",
+       "   - *Summary:* This paper establishes a theoretical grounding for PPO-Clip and introduces new interpretive frameworks for its mechanics, showing improved convergence properties.\n",
+       "   - [PDF](http://arxiv.org/pdf/2110.13799v4)\n",
+       "\n",
+       "8. **[Colored Noise in PPO: Improved Exploration and Performance through Correlated Action Sampling](http://dx.doi.org/10.1609/aaai.v38i11.29139)**  \n",
+       "   - *Summary:* This study proposes a variant of PPO using correlated noise for improved exploration, demonstrating enhanced performance over traditional approaches.\n",
+       "   - [PDF](http://arxiv.org/abs/2312.11091v2)\n",
+       "\n",
+       "9. **[A dynamical clipping approach with task feedback for Proximal Policy Optimization](http://arxiv.org/abs/2312.07624v3)**  \n",
+       "   - *Summary:* The paper presents Pb-PPO, which dynamically adjusts the clipping bounds in PPO to enhance returns, showing improved performance across various tasks.\n",
+       "   - [PDF](http://arxiv.org/pdf/2312.07624v3)\n",
+       "\n",
+       "10. **[PPO-UE: Proximal Policy Optimization via Uncertainty-Aware Exploration](http://arxiv.org/abs/2212.06343v1)**  \n",
+       "    - *Summary:* Introducing PPO-UE, which incorporates uncertainty-aware exploration, this paper shows improvements in convergence speed and performance compared to standard PPO.\n",
+       "    - [PDF](http://arxiv.org/pdf/2212.06343v1)\n",
+       "\n",
+       "These papers provide a comprehensive view of the developments and enhancements in PPO and how it operates within the reinforcement learning framework. You can click on the titles to access the full articles."
+      ],
+      "text/plain": [
+       "<IPython.core.display.Markdown object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "# Add a user message\n",
+    "paper_conversation.add_message(\"user\", \"Hi, how does PPO reinforcement learning work?\")\n",
+    "chat_response = chat_completion_with_function_execution(\n",
+    "    paper_conversation.conversation_history, functions=arxiv_functions\n",
+    ")\n",
+    "assistant_message = chat_response.choices[0].message.content\n",
+    "paper_conversation.add_message(\"assistant\", assistant_message)\n",
+    "display(Markdown(assistant_message))\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "id": "3ca3e18a",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Function generation requested, calling function\n",
+      "Finding and reading paper\n",
+      "Existing papers found... Articles: 20\n",
+      "Chunking text from paper\n",
+      "Summarizing each chunk of text\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "100%|███████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:04<00:00,  1.21s/it]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Summarizing into overall summary\n"
+     ]
+    },
+    {
+     "data": {
+      "text/markdown": [
+       "### Core Argument\n",
+       "- The paper argues for the adoption of Proximal Policy Optimization (PPO) and its dynamic variant (PPO-dynamic) as superior methods for sequence generation tasks, particularly in the context of chit-chat chatbots, compared to traditional policy gradient methods.\n",
+       "- It highlights the instability and suboptimal performance of traditional policy gradient methods, such as REINFORCE, and presents PPO as a more stable and efficient alternative.\n",
+       "\n",
+       "### Evidence\n",
+       "- **Challenges with Policy Gradient**: Traditional methods lead to unstable training and poor performance due to large updates and similar action tendencies, especially in non-differentiable evaluation metrics like BLEU scores.\n",
+       "- **PPO Advantages**: PPO regularizes policy updates, enhancing stability and coherence in chatbot responses.\n",
+       "- **Dynamic PPO Approach**: PPO-dynamic introduces dynamic adjustments to the KL-divergence bounds, allowing for more flexible and effective training.\n",
+       "- **Experimental Validation**: Experiments on synthetic tasks and real-world chit-chat scenarios demonstrate that PPO and PPO-dynamic outperform REINFORCE and other algorithms (like MIXER and SeqGAN) in terms of stability and performance metrics, including BLEU-2 scores.\n",
+       "- **Results**: PPO-dynamic showed significant improvements in precision on counting tasks and achieved the highest BLEU-2 score for chatbot responses, indicating better performance in generating diverse and accurate outputs.\n",
+       "\n",
+       "### Conclusions\n",
+       "- The paper concludes that replacing traditional policy gradient methods with PPO, particularly the dynamic version, leads to more stable training and faster convergence in sequence generation tasks.\n",
+       "- The proposed PPO-dynamic method enhances the training process by dynamically adjusting constraints, resulting in improved performance and efficiency in generating human-like conversational agents.\n",
+       "- Future research directions are suggested to further explore the potential of PPO and its adaptations in natural language processing applications."
+      ],
+      "text/plain": [
+       "<IPython.core.display.Markdown object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "# Add another user message to induce our system to use the second tool\n",
+    "paper_conversation.add_message(\n",
+    "    \"user\",\n",
+    "    \"Can you read the PPO sequence generation paper for me and give me a summary\",\n",
+    ")\n",
+    "updated_response = chat_completion_with_function_execution(\n",
+    "    paper_conversation.conversation_history, functions=arxiv_functions\n",
+    ")\n",
+    "display(Markdown(updated_response.choices[0].message.content))\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

examples/How_to_call_functions_with_chat_models.ipynb

@@ -0,0 +1,925 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "3e67f200",
+   "metadata": {},
+   "source": [
+    "# How to call functions with chat models\n",
+    "\n",
+    "This notebook covers how to use the Chat Completions API in combination with external functions to extend the capabilities of GPT models.\n",
+    "\n",
+    "`tools` is an optional parameter in the Chat Completion API which can be used to provide function specifications. The purpose of this is to enable models to generate function arguments which adhere to the provided specifications. Note that the API will not actually execute any function calls. It is up to developers to execute function calls using model outputs.\n",
+    "\n",
+    "Within the `tools` parameter, if the `functions` parameter is provided then by default the model will decide when it is appropriate to use one of the functions. The API can be forced to use a specific function by setting the `tool_choice` parameter to `{\"type\": \"function\", \"function\": {\"name\": \"my_function\"}}`. The API can also be forced to not use any function by setting the `tool_choice` parameter to `\"none\"`. If a function is used, the output will contain `\"finish_reason\": \"tool_calls\"` in the response, as well as a `tool_calls` object that has the name of the function and the generated function arguments.\n",
+    "\n",
+    "### Overview\n",
+    "\n",
+    "This notebook contains the following 2 sections:\n",
+    "\n",
+    "- **How to generate function arguments:** Specify a set of functions and use the API to generate function arguments.\n",
+    "- **How to call functions with model generated arguments:** Close the loop by actually executing functions with model generated arguments."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "64c85e26",
+   "metadata": {},
+   "source": [
+    "## How to generate function arguments"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "80e71f33",
+   "metadata": {
+    "pycharm": {
+     "is_executing": true
+    }
+   },
+   "outputs": [],
+   "source": [
+    "!pip install scipy --quiet\n",
+    "!pip install tenacity --quiet\n",
+    "!pip install tiktoken --quiet\n",
+    "!pip install termcolor --quiet\n",
+    "!pip install openai --quiet"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "dab872c5",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2024-07-12T22:41:58.148850Z",
+     "start_time": "2024-07-12T22:41:58.133412Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "from openai import OpenAI\n",
+    "from tenacity import retry, wait_random_exponential, stop_after_attempt\n",
+    "from termcolor import colored  \n",
+    "\n",
+    "GPT_MODEL = \"gpt-5\"\n",
+    "client = OpenAI()"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "69ee6a93",
+   "metadata": {},
+   "source": [
+    "### Utilities\n",
+    "\n",
+    "First let's define a few utilities for making calls to the Chat Completions API and for maintaining and keeping track of the conversation state."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "745ceec5",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2024-07-12T22:41:59.531820Z",
+     "start_time": "2024-07-12T22:41:59.529870Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "@retry(wait=wait_random_exponential(multiplier=1, max=40), stop=stop_after_attempt(3))\n",
+    "def chat_completion_request(messages, tools=None, tool_choice=None, model=GPT_MODEL):\n",
+    "    try:\n",
+    "        response = client.chat.completions.create(\n",
+    "            model=model,\n",
+    "            messages=messages,\n",
+    "            tools=tools,\n",
+    "            tool_choice=tool_choice,\n",
+    "        )\n",
+    "        return response\n",
+    "    except Exception as e:\n",
+    "        print(\"Unable to generate ChatCompletion response\")\n",
+    "        print(f\"Exception: {e}\")\n",
+    "        return e\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "c4d1c99f",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2024-07-12T22:42:00.463896Z",
+     "start_time": "2024-07-12T22:42:00.461258Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "def pretty_print_conversation(messages):\n",
+    "    role_to_color = {\n",
+    "        \"system\": \"red\",\n",
+    "        \"user\": \"green\",\n",
+    "        \"assistant\": \"blue\",\n",
+    "        \"function\": \"magenta\",\n",
+    "    }\n",
+    "    \n",
+    "    for message in messages:\n",
+    "        if message[\"role\"] == \"system\":\n",
+    "            print(colored(f\"system: {message['content']}\\n\", role_to_color[message[\"role\"]]))\n",
+    "        elif message[\"role\"] == \"user\":\n",
+    "            print(colored(f\"user: {message['content']}\\n\", role_to_color[message[\"role\"]]))\n",
+    "        elif message[\"role\"] == \"assistant\" and message.get(\"tool_calls\"):\n",
+    "            print(colored(f\"assistant: {message['tool_calls']}\\n\", role_to_color[message[\"role\"]]))\n",
+    "        elif message[\"role\"] == \"assistant\" and not message.get(\"tool_calls\"):\n",
+    "            print(colored(f\"assistant: {message['content']}\\n\", role_to_color[message[\"role\"]]))\n",
+    "        elif message[\"role\"] == \"function\":\n",
+    "            print(colored(f\"function ({message['name']}): {message['content']}\\n\", role_to_color[message[\"role\"]]))\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "29d4e02b",
+   "metadata": {},
+   "source": [
+    "### Basic concepts\n",
+    "\n",
+    "Let's create some function specifications to interface with a hypothetical weather API. We'll pass these function specification to the Chat Completions API in order to generate function arguments that adhere to the specification."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "d2e25069",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2024-07-12T22:42:01.676606Z",
+     "start_time": "2024-07-12T22:42:01.674348Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "tools = [\n",
+    "    {\n",
+    "        \"type\": \"function\",\n",
+    "        \"function\": {\n",
+    "            \"name\": \"get_current_weather\",\n",
+    "            \"description\": \"Get the current weather\",\n",
+    "            \"parameters\": {\n",
+    "                \"type\": \"object\",\n",
+    "                \"properties\": {\n",
+    "                    \"location\": {\n",
+    "                        \"type\": \"string\",\n",
+    "                        \"description\": \"The city and state, e.g. San Francisco, CA\",\n",
+    "                    },\n",
+    "                    \"format\": {\n",
+    "                        \"type\": \"string\",\n",
+    "                        \"enum\": [\"celsius\", \"fahrenheit\"],\n",
+    "                        \"description\": \"The temperature unit to use. Infer this unit from the forecast location.\",\n",
+    "                    },\n",
+    "                },\n",
+    "                \"required\": [\"location\", \"format\"],\n",
+    "            },\n",
+    "        }\n",
+    "    },\n",
+    "    {\n",
+    "        \"type\": \"function\",\n",
+    "        \"function\": {\n",
+    "            \"name\": \"get_n_day_weather_forecast\",\n",
+    "            \"description\": \"Get an N-day weather forecast\",\n",
+    "            \"parameters\": {\n",
+    "                \"type\": \"object\",\n",
+    "                \"properties\": {\n",
+    "                    \"location\": {\n",
+    "                        \"type\": \"string\",\n",
+    "                        \"description\": \"The city and state, e.g. San Francisco, CA\",\n",
+    "                    },\n",
+    "                    \"format\": {\n",
+    "                        \"type\": \"string\",\n",
+    "                        \"enum\": [\"celsius\", \"fahrenheit\"],\n",
+    "                        \"description\": \"The temperature unit to use. Infer this unit from the forecast location.\",\n",
+    "                    },\n",
+    "                    \"num_days\": {\n",
+    "                        \"type\": \"integer\",\n",
+    "                        \"description\": \"The number of days to forecast\",\n",
+    "                    }\n",
+    "                },\n",
+    "                \"required\": [\"location\", \"format\", \"num_days\"]\n",
+    "            },\n",
+    "        }\n",
+    "    },\n",
+    "]"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "bfc39899",
+   "metadata": {},
+   "source": [
+    "If we prompt the model about the current weather, it will respond with some clarifying questions."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "518d6827",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2024-07-12T22:42:03.726604Z",
+     "start_time": "2024-07-12T22:42:03.154689Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[31msystem: Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.\n",
+      "\u001b[0m\n",
+      "\u001b[32muser: What's the weather like today\n",
+      "\u001b[0m\n",
+      "\u001b[34massistant: Sure—what city and state (or country) should I check? Also, do you prefer Celsius or Fahrenheit?\n",
+      "\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "messages = []\n",
+    "messages.append({\"role\": \"system\", \"content\": \"Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.\"})\n",
+    "messages.append({\"role\": \"user\", \"content\": \"What's the weather like today\"})\n",
+    "chat_response = chat_completion_request(\n",
+    "    messages, tools=tools\n",
+    ")\n",
+    "messages.append(chat_response.choices[0].message.to_dict())\n",
+    "pretty_print_conversation(messages)\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "4c999375",
+   "metadata": {},
+   "source": [
+    "Once we provide the missing information, it will generate the appropriate function arguments for us."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "23c42a6e",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2024-07-12T22:42:05.778263Z",
+     "start_time": "2024-07-12T22:42:05.277346Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[31msystem: Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.\n",
+      "\u001b[0m\n",
+      "\u001b[32muser: What's the weather like today\n",
+      "\u001b[0m\n",
+      "\u001b[34massistant: Sure—what city and state (or country) should I check? Also, do you prefer Celsius or Fahrenheit?\n",
+      "\u001b[0m\n",
+      "\u001b[32muser: I'm in Glasgow, Scotland.\n",
+      "\u001b[0m\n",
+      "\u001b[34massistant: [{'id': 'call_k2QgGc9GT9WjxD76GvR0Ot8q', 'function': {'arguments': '{\"location\": \"Glasgow, Scotland\", \"format\": \"celsius\"}', 'name': 'get_current_weather'}, 'type': 'function'}, {'id': 'call_RtnXV5t49lqbWwhvGoEPZ7KY', 'function': {'arguments': '{\"location\": \"Glasgow, Scotland\", \"format\": \"celsius\", \"num_days\": 1}', 'name': 'get_n_day_weather_forecast'}, 'type': 'function'}]\n",
+      "\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "messages.append({\"role\": \"user\", \"content\": \"I'm in Glasgow, Scotland.\"})\n",
+    "chat_response = chat_completion_request(\n",
+    "    messages, tools=tools\n",
+    ")\n",
+    "messages.append(chat_response.choices[0].message.to_dict())\n",
+    "pretty_print_conversation(messages)\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "c14d4762",
+   "metadata": {},
+   "source": [
+    "By prompting it differently, we can get it to target the other function we've told it about."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "fa232e54",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2024-07-12T22:42:07.575820Z",
+     "start_time": "2024-07-12T22:42:07.018764Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[31msystem: Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.\n",
+      "\u001b[0m\n",
+      "\u001b[32muser: what is the weather going to be like in Glasgow, Scotland over the next x days\n",
+      "\u001b[0m\n",
+      "\u001b[34massistant: How many days would you like the forecast for in Glasgow, Scotland? For example: 3, 5, 7, 10, or 14.\n",
+      "\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "messages = []\n",
+    "messages.append({\"role\": \"system\", \"content\": \"Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.\"})\n",
+    "messages.append({\"role\": \"user\", \"content\": \"what is the weather going to be like in Glasgow, Scotland over the next x days\"})\n",
+    "chat_response = chat_completion_request(\n",
+    "    messages, tools=tools\n",
+    ")\n",
+    "messages.append(chat_response.choices[0].message.to_dict())\n",
+    "pretty_print_conversation(messages)\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "6172ddac",
+   "metadata": {},
+   "source": [
+    "Once again, the model is asking us for clarification because it doesn't have enough information yet. In this case it already knows the location for the forecast, but it needs to know how many days are required in the forecast."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "c7d8a543",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2024-07-12T22:42:09.587530Z",
+     "start_time": "2024-07-12T22:42:08.666795Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[31msystem: Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.\n",
+      "\u001b[0m\n",
+      "\u001b[32muser: what is the weather going to be like in Glasgow, Scotland over the next x days\n",
+      "\u001b[0m\n",
+      "\u001b[34massistant: How many days would you like the forecast for in Glasgow, Scotland? For example: 3, 5, 7, 10, or 14.\n",
+      "\u001b[0m\n",
+      "\u001b[32muser: 5 days\n",
+      "\u001b[0m\n",
+      "\u001b[34massistant: [{'id': 'call_lNzOVLrNSaSVjL3O3bN110af', 'function': {'arguments': '{\"location\":\"Glasgow, Scotland\",\"format\":\"celsius\",\"num_days\":5}', 'name': 'get_n_day_weather_forecast'}, 'type': 'function'}]\n",
+      "\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "messages.append({\"role\": \"user\", \"content\": \"5 days\"})\n",
+    "chat_response = chat_completion_request(\n",
+    "    messages, tools=tools\n",
+    ")\n",
+    "messages.append(chat_response.choices[0].message.to_dict())\n",
+    "pretty_print_conversation(messages)\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "4b758a0a",
+   "metadata": {},
+   "source": [
+    "#### Forcing the use of specific functions or no function"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "412f79ba",
+   "metadata": {},
+   "source": [
+    "We can force the model to use a specific function, for example get_n_day_weather_forecast by using the function_call argument. By doing so, we force the model to make assumptions about how to use it."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "559371b7",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2024-07-12T22:42:12.216712Z",
+     "start_time": "2024-07-12T22:42:11.714246Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[31msystem: Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.\n",
+      "\u001b[0m\n",
+      "\u001b[32muser: Give me a weather report for Toronto, Canada.\n",
+      "\u001b[0m\n",
+      "\u001b[34massistant: [{'id': 'call_3hoMjl55OQ7LxfwhFyjxwv1T', 'function': {'arguments': '{\"location\":\"Toronto, Canada\",\"format\":\"celsius\",\"num_days\":5}', 'name': 'get_n_day_weather_forecast'}, 'type': 'function'}]\n",
+      "\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "# in this cell we force the model to use get_n_day_weather_forecast\n",
+    "messages = []\n",
+    "messages.append({\"role\": \"system\", \"content\": \"Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.\"})\n",
+    "messages.append({\"role\": \"user\", \"content\": \"Give me a weather report for Toronto, Canada.\"})\n",
+    "chat_response = chat_completion_request(\n",
+    "    messages, tools=tools, tool_choice={\"type\": \"function\", \"function\": {\"name\": \"get_n_day_weather_forecast\"}}\n",
+    ")\n",
+    "messages.append(chat_response.choices[0].message.to_dict())\n",
+    "pretty_print_conversation(messages)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "a7ab0f58",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2024-07-12T22:42:14.264601Z",
+     "start_time": "2024-07-12T22:42:13.001306Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[31msystem: Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.\n",
+      "\u001b[0m\n",
+      "\u001b[32muser: Give me a weather report for Toronto, Canada.\n",
+      "\u001b[0m\n",
+      "\u001b[34massistant: [{'id': 'call_wv5mdjEQJnBPuSci3xw09Tom', 'function': {'arguments': '{\"location\":\"Toronto, ON\",\"format\":\"celsius\"}', 'name': 'get_current_weather'}, 'type': 'function'}]\n",
+      "\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "# if we don't force the model to use get_n_day_weather_forecast it may not\n",
+    "messages = []\n",
+    "messages.append({\"role\": \"system\", \"content\": \"Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.\"})\n",
+    "messages.append({\"role\": \"user\", \"content\": \"Give me a weather report for Toronto, Canada.\"})\n",
+    "chat_response = chat_completion_request(\n",
+    "    messages, tools=tools\n",
+    ")\n",
+    "messages.append(chat_response.choices[0].message.to_dict())\n",
+    "pretty_print_conversation(messages)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "3bd70e48",
+   "metadata": {},
+   "source": [
+    "We can also force the model to not use a function at all. By doing so we prevent it from producing a proper function call."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "acfe54e6",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2024-07-12T22:42:16.928643Z",
+     "start_time": "2024-07-12T22:42:16.295006Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[31msystem: Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.\n",
+      "\u001b[0m\n",
+      "\u001b[32muser: Give me the current weather (use Celcius) for Toronto, Canada.\n",
+      "\u001b[0m\n",
+      "\u001b[34massistant: I don’t have live access to pull the current conditions right now. To get Toronto’s current weather in Celsius, check any of these quickly:\n",
+      "- Environment Canada: weather.gc.ca (search “Toronto”)\n",
+      "- The Weather Network: theweathernetwork.com/ca/weather/ontario/toronto\n",
+      "- Google: search “Toronto weather” (shows °C by default in Canada)\n",
+      "- AccuWeather or Weather.com (set units to °C)\n",
+      "\n",
+      "If you paste the current readings here (temperature, feels-like, wind, precipitation), I can interpret them and advise on what to wear or plan for.\n",
+      "\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "messages = []\n",
+    "messages.append({\"role\": \"system\", \"content\": \"Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.\"})\n",
+    "messages.append({\"role\": \"user\", \"content\": \"Give me the current weather (use Celcius) for Toronto, Canada.\"})\n",
+    "chat_response = chat_completion_request(\n",
+    "    messages, tools=tools, tool_choice=\"none\"\n",
+    ")\n",
+    "messages.append(chat_response.choices[0].message.to_dict())\n",
+    "pretty_print_conversation(messages)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b616353b",
+   "metadata": {},
+   "source": [
+    "### Parallel Function Calling\n",
+    "\n",
+    "Newer models such as gpt-5, gpt-4.1 or gpt-4o can call multiple functions in one turn."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "380eeb68",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2024-07-12T22:42:18.988762Z",
+     "start_time": "2024-07-12T22:42:18.041914Z"
+    }
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[ChatCompletionMessageFunctionToolCall(id='call_KlZ3Fqt3SviC6o66dVMYSa2Q', function=Function(arguments='{\"location\": \"San Francisco, CA\", \"format\": \"fahrenheit\", \"num_days\": 4}', name='get_n_day_weather_forecast'), type='function'),\n",
+       " ChatCompletionMessageFunctionToolCall(id='call_YAnH0VRB3oqjqivcGj3Cd8YA', function=Function(arguments='{\"location\": \"Glasgow, UK\", \"format\": \"celsius\", \"num_days\": 4}', name='get_n_day_weather_forecast'), type='function')]"
+      ]
+     },
+     "execution_count": 13,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "messages = []\n",
+    "messages.append({\"role\": \"system\", \"content\": \"Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.\"})\n",
+    "messages.append({\"role\": \"user\", \"content\": \"what is the weather going to be like in San Francisco and Glasgow over the next 4 days\"})\n",
+    "chat_response = chat_completion_request(\n",
+    "    messages, tools=tools, model=\"gpt-4o\"\n",
+    ")\n",
+    "\n",
+    "assistant_message = chat_response.choices[0].message.tool_calls\n",
+    "assistant_message"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "b4482aee",
+   "metadata": {},
+   "source": [
+    "## How to call functions with model generated arguments\n",
+    "\n",
+    "In our next example, we'll demonstrate how to execute functions whose inputs are model-generated, and use this to implement an agent that can answer questions for us about a database. For simplicity we'll use the [Chinook sample database](https://www.sqlitetutorial.net/sqlite-sample-database/).\n",
+    "\n",
+    "*Note:* SQL generation can be high-risk in a production environment since models are not perfectly reliable at generating correct SQL."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "f7654fef",
+   "metadata": {},
+   "source": [
+    "### Specifying a function to execute SQL queries\n",
+    "\n",
+    "First let's define some helpful utility functions to extract data from a SQLite database."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "30f6b60e",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2024-07-12T22:42:20.742187Z",
+     "start_time": "2024-07-12T22:42:20.737751Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Opened database successfully\n"
+     ]
+    }
+   ],
+   "source": [
+    "import sqlite3\n",
+    "\n",
+    "conn = sqlite3.connect(\"data/Chinook.db\")\n",
+    "print(\"Opened database successfully\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "id": "abec0214",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2024-07-12T22:42:21.370623Z",
+     "start_time": "2024-07-12T22:42:21.368246Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "def get_table_names(conn):\n",
+    "    \"\"\"Return a list of table names.\"\"\"\n",
+    "    table_names = []\n",
+    "    tables = conn.execute(\"SELECT name FROM sqlite_master WHERE type='table';\")\n",
+    "    for table in tables.fetchall():\n",
+    "        table_names.append(table[0])\n",
+    "    return table_names\n",
+    "\n",
+    "\n",
+    "def get_column_names(conn, table_name):\n",
+    "    \"\"\"Return a list of column names.\"\"\"\n",
+    "    column_names = []\n",
+    "    columns = conn.execute(f\"PRAGMA table_info('{table_name}');\").fetchall()\n",
+    "    for col in columns:\n",
+    "        column_names.append(col[1])\n",
+    "    return column_names\n",
+    "\n",
+    "\n",
+    "def get_database_info(conn):\n",
+    "    \"\"\"Return a list of dicts containing the table name and columns for each table in the database.\"\"\"\n",
+    "    table_dicts = []\n",
+    "    for table_name in get_table_names(conn):\n",
+    "        columns_names = get_column_names(conn, table_name)\n",
+    "        table_dicts.append({\"table_name\": table_name, \"column_names\": columns_names})\n",
+    "    return table_dicts\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "77e6e5ea",
+   "metadata": {},
+   "source": [
+    "Now we can use these utility functions to extract a representation of the database schema."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "id": "0c0104cd",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2024-07-12T22:42:22.668456Z",
+     "start_time": "2024-07-12T22:42:22.665839Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "database_schema_dict = get_database_info(conn)\n",
+    "database_schema_string = \"\\n\".join(\n",
+    "    [\n",
+    "        f\"Table: {table['table_name']}\\nColumns: {', '.join(table['column_names'])}\"\n",
+    "        for table in database_schema_dict\n",
+    "    ]\n",
+    ")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "ae73c9ee",
+   "metadata": {},
+   "source": [
+    "As before, we'll define a function specification for the function we'd like the API to generate arguments for. Notice that we are inserting the database schema into the function specification. This will be important for the model to know about."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "id": "0258813a",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2024-07-12T22:42:24.156291Z",
+     "start_time": "2024-07-12T22:42:24.154372Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "tools = [\n",
+    "    {\n",
+    "        \"type\": \"function\",\n",
+    "        \"function\": {\n",
+    "            \"name\": \"ask_database\",\n",
+    "            \"description\": \"Use this function to answer user questions about music. Input should be a fully formed SQL query.\",\n",
+    "            \"parameters\": {\n",
+    "                \"type\": \"object\",\n",
+    "                \"properties\": {\n",
+    "                    \"query\": {\n",
+    "                        \"type\": \"string\",\n",
+    "                        \"description\": f\"\"\"\n",
+    "                                SQL query extracting info to answer the user's question.\n",
+    "                                SQL should be written using this database schema:\n",
+    "                                {database_schema_string}\n",
+    "                                The query should be returned in plain text, not in JSON.\n",
+    "                                \"\"\",\n",
+    "                    }\n",
+    "                },\n",
+    "                \"required\": [\"query\"],\n",
+    "            },\n",
+    "        }\n",
+    "    }\n",
+    "]"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "da08c121",
+   "metadata": {},
+   "source": [
+    "### Executing SQL queries\n",
+    "\n",
+    "Now let's implement the function that will actually excute queries against the database."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "id": "65585e74",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2024-07-12T22:42:25.444734Z",
+     "start_time": "2024-07-12T22:42:25.442757Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "def ask_database(conn, query):\n",
+    "    \"\"\"Function to query SQLite database with a provided SQL query.\"\"\"\n",
+    "    try:\n",
+    "        results = str(conn.execute(query).fetchall())\n",
+    "    except Exception as e:\n",
+    "        results = f\"query failed with error: {e}\"\n",
+    "    return results"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8f6885e9f0af5c40",
+   "metadata": {},
+   "source": [
+    "##### Steps to invoke a function call using Chat Completions API: \n",
+    "\n",
+    "**Step 1**: Prompt the model with content that may result in model selecting a tool to use. The description of the tools such as a function name and signature is defined in the 'Tools' list and passed to the model in API call. If selected, the function name and parameters are included in the response.<br>\n",
+    "  \n",
+    "**Step 2**: Check programmatically if model wanted to call a function. If true, proceed to step 3. <br>  \n",
+    "**Step 3**: Extract the function name and parameters from response, call the function with parameters. Append the result to messages. <br>    \n",
+    "**Step 4**: Invoke the chat completions API with the message list to get the response. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "id": "e8b7cb9cdc7a7616",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2024-07-12T22:42:28.395683Z",
+     "start_time": "2024-07-12T22:42:27.415626Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[32muser: What is the name of the album with the most tracks?\n",
+      "\u001b[0m\n",
+      "\u001b[34massistant: [{'id': 'call_pGRtZZGfd2o41GHlZcEdB9he', 'function': {'arguments': '{\"query\":\"WITH track_counts AS (\\\\n  SELECT a.AlbumId, a.Title, COUNT(t.TrackId) AS track_count\\\\n  FROM Album a\\\\n  JOIN Track t ON t.AlbumId = a.AlbumId\\\\n  GROUP BY a.AlbumId, a.Title\\\\n)\\\\nSELECT Title\\\\nFROM track_counts\\\\nWHERE track_count = (SELECT MAX(track_count) FROM track_counts);\"}', 'name': 'ask_database'}, 'type': 'function'}]\n",
+      "\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step #1: Prompt with content that may result in function call. In this case the model can identify the information requested by the user is potentially available in the database schema passed to the model in Tools description. \n",
+    "messages = [{\n",
+    "    \"role\":\"user\", \n",
+    "    \"content\": \"What is the name of the album with the most tracks?\"\n",
+    "}]\n",
+    "\n",
+    "response = client.chat.completions.create(\n",
+    "    model=GPT_MODEL, \n",
+    "    messages=messages, \n",
+    "    tools=tools, \n",
+    "    tool_choice=\"auto\"\n",
+    ")\n",
+    "\n",
+    "# Append the message to messages list\n",
+    "response_message = response.choices[0].message \n",
+    "messages.append(response_message.to_dict())\n",
+    "pretty_print_conversation(messages)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "id": "351c39def3417776",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2024-07-12T22:42:30.439519Z",
+     "start_time": "2024-07-12T22:42:29.799492Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Result found in database: Greatest Hits\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Step 2: determine if the response from the model includes a tool call.   \n",
+    "tool_calls = response_message.tool_calls\n",
+    "if tool_calls:\n",
+    "    # If true the model will return the name of the tool / function to call and the argument(s)  \n",
+    "    tool_call_id = tool_calls[0].id\n",
+    "    tool_function_name = tool_calls[0].function.name\n",
+    "    tool_query_string = json.loads(tool_calls[0].function.arguments)['query']\n",
+    "\n",
+    "    # Step 3: Call the function and retrieve results. Append the results to the messages list.      \n",
+    "    if tool_function_name == 'ask_database':\n",
+    "        results = ask_database(conn, tool_query_string)\n",
+    "        \n",
+    "        messages.append({\n",
+    "            \"role\":\"tool\", \n",
+    "            \"tool_call_id\":tool_call_id, \n",
+    "            \"name\": tool_function_name, \n",
+    "            \"content\":results\n",
+    "        })\n",
+    "        \n",
+    "        # Step 4: Invoke the chat completions API with the function response appended to the messages list\n",
+    "        # Note that messages with role 'tool' must be a response to a preceding message with 'tool_calls'\n",
+    "        model_response_with_function_call = client.chat.completions.create(\n",
+    "            model=GPT_MODEL,\n",
+    "            messages=messages,\n",
+    "        )  # get a new response from the model where it can see the function response\n",
+    "        print(f\"Result found in database: {model_response_with_function_call.choices[0].message.content}\")\n",
+    "    else: \n",
+    "        print(f\"Error: function {tool_function_name} does not exist\")\n",
+    "else: \n",
+    "    # Model did not identify a function to call, result can be returned to the user \n",
+    "    print(response_message.content) "
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "2d89073c",
+   "metadata": {},
+   "source": [
+    "## Next Steps\n",
+    "\n",
+    "See our other [notebook](How_to_call_functions_for_knowledge_retrieval.ipynb) that demonstrates how to use the Chat Completions API and functions for knowledge retrieval to interact conversationally with a knowledge base."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "openai",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

examples/How_to_combine_GPT4o_with_RAG_Outfit_Assistant.ipynb

@@ -0,0 +1,631 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# How to Combine GPT-4o Mini with RAG - Create a Clothing Matchmaker App\n",
+    "\n",
+    "Welcome to the Clothing Matchmaker App Jupyter Notebook! This project demonstrates the power of the GPT-4o mini model in analyzing images of clothing items and extracting key features such as color, style, and type. The core of our app relies on this advanced image analysis model developed by OpenAI, which enables us to accurately identify the characteristics of the input clothing item.\n",
+    "\n",
+    "GPT-4o mini is a small model that combines natural language processing with image recognition, allowing it to understand and generate responses based on both text and visual inputs with low latency.\n",
+    "\n",
+    "Building on the capabilities of the GPT-4o mini model, we employ a custom matching algorithm and the RAG technique to search our knowledge base for items that complement the identified features. This algorithm takes into account factors like color compatibility and style coherence to provide users with suitable recommendations. Through this notebook, we aim to showcase the practical application of these technologies in creating a clothing recommendation system.\n",
+    "\n",
+    "Using the combination of GPT-4o mini + RAG (Retrieval-Augmented Generation) offers several advantages:\n",
+    "\n",
+    "1. **Contextual Understanding**: GPT-4o mini can analyze input images and understand the context, such as the objects, scenes, and activities depicted. This allows for more accurate and relevant suggestions or information across various domains, whether it's interior design, cooking, or education.\n",
+    "2. **Rich Knowledge Base**: RAG combines the generative capabilities of GPT-4 with a retrieval component that accesses a large corpus of information across different fields. This means the system can provide suggestions or insights based on a wide range of knowledge, from historical facts to scientific concepts.\n",
+    "3. **Customization**: The approach allows for easy customization to cater to specific user needs or preferences in various applications. Whether it's tailoring suggestions to a user's taste in art or providing educational content based on a student's learning level, the system can be adapted to deliver personalized experiences.\n",
+    "\n",
+    "Overall, the GPT-4o mini + RAG approach offers a fast, powerful, and flexible solution for various fashion-related applications, leveraging the strengths of both generative and retrieval-based AI techniques.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Environment Setup"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "First we will install the necessary dependencies, then import the libraries and write some utility functions that we will use later on. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%pip install openai --quiet\n",
+    "%pip install tenacity --quiet\n",
+    "%pip install tqdm --quiet\n",
+    "%pip install numpy --quiet\n",
+    "%pip install typing --quiet\n",
+    "%pip install tiktoken --quiet\n",
+    "%pip install concurrent --quiet"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import numpy as np\n",
+    "import json\n",
+    "import ast\n",
+    "import tiktoken\n",
+    "import concurrent\n",
+    "from openai import OpenAI\n",
+    "from tqdm import tqdm\n",
+    "from tenacity import retry, wait_random_exponential, stop_after_attempt\n",
+    "from IPython.display import Image, display, HTML\n",
+    "from typing import List\n",
+    "\n",
+    "client = OpenAI()\n",
+    "\n",
+    "GPT_MODEL = \"gpt-4o-mini\"\n",
+    "EMBEDDING_MODEL = \"text-embedding-3-large\"\n",
+    "EMBEDDING_COST_PER_1K_TOKENS = 0.00013"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Creating the Embeddings\n",
+    "We will now set up the knowledge base by choosing a database and generating embeddings for it. I am using the `sample_styles.csv` file for this in the data folder. This is a sample of a bigger dataset that contains `~44K` items. This step can also be replaced by using an out-of-the-box vector database. For example, you can follow one of [these cookbooks](https://github.com/openai/openai-cookbook/tree/main/examples/vector_databases) to set up your vector database."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "styles_filepath = \"data/sample_clothes/sample_styles.csv\"\n",
+    "styles_df = pd.read_csv(styles_filepath, on_bad_lines='skip')\n",
+    "print(styles_df.head())\n",
+    "print(\"Opened dataset successfully. Dataset has {} items of clothing.\".format(len(styles_df)))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now we will generate embeddings for the entire dataset. We can parallelize the execution of these embeddings to ensure that the script scales up for larger datasets. With this logic, the time to create embeddings for the full `44K` entry dataset decreases from ~4h to ~2-3min. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "## Batch Embedding Logic\n",
+    "\n",
+    "# Simple function to take in a list of text objects and return them as a list of embeddings\n",
+    "@retry(wait=wait_random_exponential(min=1, max=40), stop=stop_after_attempt(10))\n",
+    "def get_embeddings(input: List):\n",
+    "    response = client.embeddings.create(\n",
+    "        input=input,\n",
+    "        model=EMBEDDING_MODEL\n",
+    "    ).data\n",
+    "    return [data.embedding for data in response]\n",
+    "\n",
+    "\n",
+    "# Splits an iterable into batches of size n.\n",
+    "def batchify(iterable, n=1):\n",
+    "    l = len(iterable)\n",
+    "    for ndx in range(0, l, n):\n",
+    "        yield iterable[ndx : min(ndx + n, l)]\n",
+    "     \n",
+    "\n",
+    "# Function for batching and parallel processing the embeddings\n",
+    "def embed_corpus(\n",
+    "    corpus: List[str],\n",
+    "    batch_size=64,\n",
+    "    num_workers=8,\n",
+    "    max_context_len=8191,\n",
+    "):\n",
+    "    # Encode the corpus, truncating to max_context_len\n",
+    "    encoding = tiktoken.get_encoding(\"cl100k_base\")\n",
+    "    encoded_corpus = [\n",
+    "        encoded_article[:max_context_len] for encoded_article in encoding.encode_batch(corpus)\n",
+    "    ]\n",
+    "\n",
+    "    # Calculate corpus statistics: the number of inputs, the total number of tokens, and the estimated cost to embed\n",
+    "    num_tokens = sum(len(article) for article in encoded_corpus)\n",
+    "    cost_to_embed_tokens = num_tokens / 1000 * EMBEDDING_COST_PER_1K_TOKENS\n",
+    "    print(\n",
+    "        f\"num_articles={len(encoded_corpus)}, num_tokens={num_tokens}, est_embedding_cost={cost_to_embed_tokens:.2f} USD\"\n",
+    "    )\n",
+    "\n",
+    "    # Embed the corpus\n",
+    "    with concurrent.futures.ThreadPoolExecutor(max_workers=num_workers) as executor:\n",
+    "        \n",
+    "        futures = [\n",
+    "            executor.submit(get_embeddings, text_batch)\n",
+    "            for text_batch in batchify(encoded_corpus, batch_size)\n",
+    "        ]\n",
+    "\n",
+    "        with tqdm(total=len(encoded_corpus)) as pbar:\n",
+    "            for _ in concurrent.futures.as_completed(futures):\n",
+    "                pbar.update(batch_size)\n",
+    "\n",
+    "        embeddings = []\n",
+    "        for future in futures:\n",
+    "            data = future.result()\n",
+    "            embeddings.extend(data)\n",
+    "\n",
+    "        return embeddings\n",
+    "    \n",
+    "\n",
+    "# Function to generate embeddings for a given column in a DataFrame\n",
+    "def generate_embeddings(df, column_name):\n",
+    "    # Initialize an empty list to store embeddings\n",
+    "    descriptions = df[column_name].astype(str).tolist()\n",
+    "    embeddings = embed_corpus(descriptions)\n",
+    "\n",
+    "    # Add the embeddings as a new column to the DataFrame\n",
+    "    df['embeddings'] = embeddings\n",
+    "    print(\"Embeddings created successfully.\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Two options for creating the embeddings: \n",
+    "The next line will **create the embeddings** for the sample clothes dataset. This will take around 0.02s to process and another ~30s to write the results to a local .csv file. The process is using our `text_embedding_3_large` model which is priced at `$0.00013/1K` tokens. Given that the dataset has around `1K` entries, the following operation will cost approximately `$0.001`. If you decide to work with the entire dataset of `44K` entries, this operation will take 2-3min to process and it will cost approximately `$0.07`.\n",
+    "\n",
+    "**If you would not like to proceed with creating your own embeddings**, we will use a dataset of pre-computed embeddings. You can skip this cell and uncomment the code in the following cell to proceed with loading the pre-computed vectors. This operation takes ~1min to load all the data in memory."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "generate_embeddings(styles_df, 'productDisplayName')\n",
+    "print(\"Writing embeddings to file ...\")\n",
+    "styles_df.to_csv('data/sample_clothes/sample_styles_with_embeddings.csv', index=False)\n",
+    "print(\"Embeddings successfully stored in sample_styles_with_embeddings.csv\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# styles_df = pd.read_csv('data/sample_clothes/sample_styles_with_embeddings.csv', on_bad_lines='skip')\n",
+    "\n",
+    "# # Convert the 'embeddings' column from string representations of lists to actual lists of floats\n",
+    "# styles_df['embeddings'] = styles_df['embeddings'].apply(lambda x: ast.literal_eval(x))\n",
+    "\n",
+    "print(styles_df.head())\n",
+    "print(\"Opened dataset successfully. Dataset has {} items of clothing along with their embeddings.\".format(len(styles_df)))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Building the Matching Algorithm\n",
+    "\n",
+    "In this section, we'll develop a cosine similarity retrieval algorithm to find similar items in our dataframe. We'll utilize our custom cosine similarity function for this purpose. While the `sklearn` library offers a built-in cosine similarity function, recent updates to its SDK have led to compatibility issues, prompting us to implement our own standard cosine similarity calculation.\n",
+    "\n",
+    "If you already have a vector database set up, you can skip this step. Most standard databases come with their own search functions, which simplify the subsequent steps outlined in this guide. However, we aim to demonstrate that the matching algorithm can be tailored to meet specific requirements, such as a particular threshold or a specified number of matches returned.\n",
+    "\n",
+    "The `find_similar_items` function accepts four parameters:\n",
+    "- `embedding`: The embedding for which we want to find a match.\n",
+    "- `embeddings`: A list of embeddings to search through for the best matches.\n",
+    "- `threshold` (optional): This parameter specifies the minimum similarity score for a match to be considered valid. A higher threshold results in closer (better) matches, while a lower threshold allows for more items to be returned, though they may not be as closely matched to the initial `embedding`.\n",
+    "- `top_k` (optional): This parameter determines the number of items to return that exceed the given threshold. These will be the top-scoring matches for the provided `embedding`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def cosine_similarity_manual(vec1, vec2):\n",
+    "    \"\"\"Calculate the cosine similarity between two vectors.\"\"\"\n",
+    "    vec1 = np.array(vec1, dtype=float)\n",
+    "    vec2 = np.array(vec2, dtype=float)\n",
+    "\n",
+    "\n",
+    "    dot_product = np.dot(vec1, vec2)\n",
+    "    norm_vec1 = np.linalg.norm(vec1)\n",
+    "    norm_vec2 = np.linalg.norm(vec2)\n",
+    "    return dot_product / (norm_vec1 * norm_vec2)\n",
+    "\n",
+    "\n",
+    "def find_similar_items(input_embedding, embeddings, threshold=0.5, top_k=2):\n",
+    "    \"\"\"Find the most similar items based on cosine similarity.\"\"\"\n",
+    "    \n",
+    "    # Calculate cosine similarity between the input embedding and all other embeddings\n",
+    "    similarities = [(index, cosine_similarity_manual(input_embedding, vec)) for index, vec in enumerate(embeddings)]\n",
+    "    \n",
+    "    # Filter out any similarities below the threshold\n",
+    "    filtered_similarities = [(index, sim) for index, sim in similarities if sim >= threshold]\n",
+    "    \n",
+    "    # Sort the filtered similarities by similarity score\n",
+    "    sorted_indices = sorted(filtered_similarities, key=lambda x: x[1], reverse=True)[:top_k]\n",
+    "\n",
+    "    # Return the top-k most similar items\n",
+    "    return sorted_indices"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def find_matching_items_with_rag(df_items, item_descs):\n",
+    "   \"\"\"Take the input item descriptions and find the most similar items based on cosine similarity for each description.\"\"\"\n",
+    "   \n",
+    "   # Select the embeddings from the DataFrame.\n",
+    "   embeddings = df_items['embeddings'].tolist()\n",
+    "\n",
+    "   \n",
+    "   similar_items = []\n",
+    "   for desc in item_descs:\n",
+    "      \n",
+    "      # Generate the embedding for the input item\n",
+    "      input_embedding = get_embeddings([desc])\n",
+    "    \n",
+    "      # Find the most similar items based on cosine similarity\n",
+    "      similar_indices = find_similar_items(input_embedding, embeddings, threshold=0.6)\n",
+    "      similar_items += [df_items.iloc[i] for i in similar_indices]\n",
+    "    \n",
+    "   return similar_items"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Analysis Module\n",
+    "\n",
+    "In this module, we leverage `gpt-4o-mini` to analyze input images and extract important features like detailed descriptions, styles, and types. The analysis is performed through a straightforward API call, where we provide the URL of the image for analysis and request the model to identify relevant features.\n",
+    "\n",
+    "To ensure the model returns accurate results, we use specific techniques in our prompt:\n",
+    "\n",
+    "1. **Output Format Specification**: We instruct the model to return a JSON block with a predefined structure, consisting of:\n",
+    "   - `items` (str[]): A list of strings, each representing a concise title for an item of clothing, including style, color, and gender. These titles closely resemble the `productDisplayName` property in our original database.\n",
+    "   - `category` (str): The category that best represents the given item. The model selects from a list of all unique `articleTypes` present in the original styles dataframe.\n",
+    "   - `gender` (str): A label indicating the gender the item is intended for. The model chooses from the options `[Men, Women, Boys, Girls, Unisex]`.\n",
+    "\n",
+    "2. **Clear and Concise Instructions**: \n",
+    "   - We provide clear instructions on what the item titles should include and what the output format should be. The output should be in JSON format, but without the `json` tag that the model response normally contains.\n",
+    "\n",
+    "3. **One Shot Example**: \n",
+    "   - To further clarify the expected output, we provide the model with an example input description and a corresponding example output. Although this may increase the number of tokens used (and thus the cost of the call), it helps to guide the model and results in better overall performance.\n",
+    "\n",
+    "By following this structured approach, we aim to obtain precise and useful information from the `gpt-4o-mini` model for further analysis and integration into our database."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def analyze_image(image_base64, subcategories):\n",
+    "    response = client.chat.completions.create(\n",
+    "        model=GPT_MODEL,\n",
+    "        messages=[\n",
+    "            {\n",
+    "            \"role\": \"user\",\n",
+    "            \"content\": [\n",
+    "                {\n",
+    "                \"type\": \"text\",\n",
+    "                \"text\": f\"\"\"Given an image of an item of clothing, analyze the item and generate a JSON output with the following fields: \"items\", \"category\", and \"gender\".\n",
+    "                           Use your understanding of fashion trends, styles, and gender preferences to provide accurate and relevant suggestions for how to complete the outfit.\n",
+    "                           The items field should be a list of items that would go well with the item in the picture. Each item should represent a title of an item of clothing that contains the style, color, and gender of the item.\n",
+    "                           The category needs to be chosen between the types in this list: {subcategories}.\n",
+    "                           You have to choose between the genders in this list: [Men, Women, Boys, Girls, Unisex]\n",
+    "                           Do not include the description of the item in the picture. Do not include the ```json ``` tag in the output.\n",
+    "\n",
+    "                           Example Input: An image representing a black leather jacket.\n",
+    "\n",
+    "                           Example Output: {{\"items\": [\"Fitted White Women's T-shirt\", \"White Canvas Sneakers\", \"Women's Black Skinny Jeans\"], \"category\": \"Jackets\", \"gender\": \"Women\"}}\n",
+    "                           \"\"\",\n",
+    "                },\n",
+    "                {\n",
+    "                \"type\": \"image_url\",\n",
+    "                \"image_url\": {\n",
+    "                    \"url\": f\"data:image/jpeg;base64,{image_base64}\",\n",
+    "                },\n",
+    "                }\n",
+    "            ],\n",
+    "            }\n",
+    "        ]\n",
+    "    )\n",
+    "    # Extract relevant features from the response\n",
+    "    features = response.choices[0].message.content\n",
+    "    return features"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Testing the Prompt with Sample Images\n",
+    "\n",
+    "To evaluate the effectiveness of our prompt, let's load and test it with a selection of images from our dataset. We'll use images from the `\"data/sample_clothes/sample_images\"` folder, ensuring a variety of styles, genders, and types. Here are the chosen samples:\n",
+    "\n",
+    "- `2133.jpg`: Men's shirt\n",
+    "- `7143.jpg`: Women's shirt\n",
+    "- `4226.jpg`: Casual men's printed t-shirt\n",
+    "\n",
+    "By testing the prompt with these diverse images, we can assess its ability to accurately analyze and extract relevant features from different types of clothing items and accessories."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We need a utility function to encode the .jpg images in base64"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import base64\n",
+    "\n",
+    "def encode_image_to_base64(image_path):\n",
+    "    with open(image_path, 'rb') as image_file:\n",
+    "        encoded_image = base64.b64encode(image_file.read())\n",
+    "        return encoded_image.decode('utf-8')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Set the path to the images and select a test image\n",
+    "image_path = \"data/sample_clothes/sample_images/\"\n",
+    "test_images = [\"2133.jpg\", \"7143.jpg\", \"4226.jpg\"]\n",
+    "\n",
+    "# Encode the test image to base64\n",
+    "reference_image = image_path + test_images[0]\n",
+    "encoded_image = encode_image_to_base64(reference_image)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Select the unique subcategories from the DataFrame\n",
+    "unique_subcategories = styles_df['articleType'].unique()\n",
+    "\n",
+    "# Analyze the image and return the results\n",
+    "analysis = analyze_image(encoded_image, unique_subcategories)\n",
+    "image_analysis = json.loads(analysis)\n",
+    "\n",
+    "# Display the image and the analysis results\n",
+    "display(Image(filename=reference_image))\n",
+    "print(image_analysis)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Next, we process the output from the image analysis and use it to filter and display matching items from our dataset. Here's a breakdown of the code:\n",
+    "\n",
+    "1. **Extracting Image Analysis Results**: We extract the item descriptions, category, and gender from the `image_analysis` dictionary.\n",
+    "\n",
+    "2. **Filtering the Dataset**: We filter the `styles_df` DataFrame to include only items that match the gender from the image analysis (or are unisex) and exclude items of the same category as the analyzed image.\n",
+    "\n",
+    "3. **Finding Matching Items**: We use the `find_matching_items_with_rag` function to find items in the filtered dataset that match the descriptions extracted from the analyzed image.\n",
+    "\n",
+    "4. **Displaying Matching Items**: We create an HTML string to display images of the matching items. We construct the image paths using the item IDs and append each image to the HTML string. Finally, we use `display(HTML(html))` to render the images in the notebook.\n",
+    "\n",
+    "This cell effectively demonstrates how to use the results of image analysis to filter a dataset and visually display items that match the analyzed image's characteristics."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Extract the relevant features from the analysis\n",
+    "item_descs = image_analysis['items']\n",
+    "item_category = image_analysis['category']\n",
+    "item_gender = image_analysis['gender']\n",
+    "\n",
+    "\n",
+    "# Filter data such that we only look through the items of the same gender (or unisex) and different category\n",
+    "filtered_items = styles_df.loc[styles_df['gender'].isin([item_gender, 'Unisex'])]\n",
+    "filtered_items = filtered_items[filtered_items['articleType'] != item_category]\n",
+    "print(str(len(filtered_items)) + \" Remaining Items\")\n",
+    "\n",
+    "# Find the most similar items based on the input item descriptions\n",
+    "matching_items = find_matching_items_with_rag(filtered_items, item_descs)\n",
+    "\n",
+    "# Display the matching items (this will display 2 items for each description in the image analysis)\n",
+    "html = \"\"\n",
+    "paths = []\n",
+    "for i, item in enumerate(matching_items):\n",
+    "    item_id = item['id']\n",
+    "        \n",
+    "    # Path to the image file\n",
+    "    image_path = f'data/sample_clothes/sample_images/{item_id}.jpg'\n",
+    "    paths.append(image_path)\n",
+    "    html += f'<img src=\"{image_path}\" style=\"display:inline;margin:1px\"/>'\n",
+    "\n",
+    "# Print the matching item description as a reminder of what we are looking for\n",
+    "print(item_descs)\n",
+    "# Display the image\n",
+    "display(HTML(html))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Guardrails\n",
+    "\n",
+    "In the context of using Large Language Models (LLMs) like GPT-4o mini, \"guardrails\" refer to mechanisms or checks put in place to ensure that the model's output remains within desired parameters or boundaries. These guardrails are crucial for maintaining the quality and relevance of the model's responses, especially when dealing with complex or nuanced tasks.\n",
+    "\n",
+    "Guardrails are useful for several reasons:\n",
+    "\n",
+    "1. **Accuracy**: They help ensure that the model's output is accurate and relevant to the input provided.\n",
+    "2. **Consistency**: They maintain consistency in the model's responses, especially when dealing with similar or related inputs.\n",
+    "3. **Safety**: They prevent the model from generating harmful, offensive, or inappropriate content.\n",
+    "4. **Contextual Relevance**: They ensure that the model's output is contextually relevant to the specific task or domain it is being used for.\n",
+    "\n",
+    "In our case, we are using GPT-4o mini to analyze fashion images and suggest items that would complement an original outfit. To implement guardrails, we can **refine results**: After obtaining initial suggestions from GPT-4o mini, we can send the original image and the suggested items back to the model. We can then ask GPT-4o mini to evaluate whether each suggested item would indeed be a good fit for the original outfit.\n",
+    "\n",
+    "This gives the model the ability to self-correct and adjust its own output based on feedback or additional information. By implementing these guardrails and enabling self-correction, we can enhance the reliability and usefulness of the model's output in the context of fashion analysis and recommendation.\n",
+    "\n",
+    "To facilitate this, we write a prompt that asks the LLM for a simple \"yes\" or \"no\" answer to the question of whether the suggested items match the original outfit or not. This binary response helps streamline the refinement process and ensures clear and actionable feedback from the model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def check_match(reference_image_base64, suggested_image_base64):\n",
+    "    response = client.chat.completions.create(\n",
+    "        model=GPT_MODEL,\n",
+    "        messages=[\n",
+    "            {\n",
+    "            \"role\": \"user\",\n",
+    "            \"content\": [\n",
+    "                {\n",
+    "                \"type\": \"text\",\n",
+    "                \"text\": \"\"\" You will be given two images of two different items of clothing.\n",
+    "                            Your goal is to decide if the items in the images would work in an outfit together.\n",
+    "                            The first image is the reference item (the item that the user is trying to match with another item).\n",
+    "                            You need to decide if the second item would work well with the reference item.\n",
+    "                            Your response must be a JSON output with the following fields: \"answer\", \"reason\".\n",
+    "                            The \"answer\" field must be either \"yes\" or \"no\", depending on whether you think the items would work well together.\n",
+    "                            The \"reason\" field must be a short explanation of your reasoning for your decision. Do not include the descriptions of the 2 images.\n",
+    "                            Do not include the ```json ``` tag in the output.\n",
+    "                           \"\"\",\n",
+    "                },\n",
+    "                {\n",
+    "                \"type\": \"image_url\",\n",
+    "                \"image_url\": {\n",
+    "                    \"url\": f\"data:image/jpeg;base64,{reference_image_base64}\",\n",
+    "                },\n",
+    "                },\n",
+    "                {\n",
+    "                \"type\": \"image_url\",\n",
+    "                \"image_url\": {\n",
+    "                    \"url\": f\"data:image/jpeg;base64,{suggested_image_base64}\",\n",
+    "                },\n",
+    "                }\n",
+    "            ],\n",
+    "            }\n",
+    "        ],\n",
+    "        max_tokens=300,\n",
+    "    )\n",
+    "    # Extract relevant features from the response\n",
+    "    features = response.choices[0].message.content\n",
+    "    return features"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Finally, let's determine which of the items identified above truly complement the outfit."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Select the unique paths for the generated images\n",
+    "paths = list(set(paths))\n",
+    "\n",
+    "for path in paths:\n",
+    "    # Encode the test image to base64\n",
+    "    suggested_image = encode_image_to_base64(path)\n",
+    "    \n",
+    "    # Check if the items match\n",
+    "    match = json.loads(check_match(encoded_image, suggested_image))\n",
+    "    \n",
+    "    # Display the image and the analysis results\n",
+    "    if match[\"answer\"] == 'yes':\n",
+    "        display(Image(filename=path))\n",
+    "        print(\"The items match!\")\n",
+    "        print(match[\"reason\"])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can observe that the initial list of potential items has been further refined, resulting in a more curated selection that aligns well with the outfit. Additionally, the model provides explanations for why each item is considered a good match, offering valuable insights into the decision-making process."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Conclusion\n",
+    "\n",
+    "In this Jupyter Notebook, we explored the application of GPT-4o mini and other machine learning techniques to the domain of fashion. We demonstrated how to analyze images of clothing items, extract relevant features, and use this information to find matching items that complement an original outfit. Through the implementation of guardrails and self-correction mechanisms, we refined the model's suggestions to ensure they are accurate and contextually relevant.\n",
+    "\n",
+    "This approach has several practical uses in the real world, including:\n",
+    "\n",
+    "1. **Personalized Shopping Assistants**: Retailers can use this technology to offer personalized outfit recommendations to customers, enhancing the shopping experience and increasing customer satisfaction.\n",
+    "2. **Virtual Wardrobe Applications**: Users can upload images of their own clothing items to create a virtual wardrobe and receive suggestions for new items that match their existing pieces.\n",
+    "3. **Fashion Design and Styling**: Fashion designers and stylists can use this tool to experiment with different combinations and styles, streamlining the creative process.\n",
+    "\n",
+    "However, one of the considerations to keep in mind is **cost**. The use of LLMs and image analysis models can incur costs, especially if used extensively. It's important to consider the cost-effectiveness of implementing these technologies. `gpt-4o-mini` is priced at `$0.01` per 1000 tokens. This adds up to `$0.00255` for one 256px x 256px image.\n",
+    "\n",
+    "Overall, this notebook serves as a foundation for further exploration and development in the intersection of fashion and AI, opening doors to more personalized and intelligent fashion recommendation systems."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/How_to_count_tokens_with_tiktoken.ipynb

@@ -0,0 +1,814 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# How to count tokens with tiktoken\n",
+    "\n",
+    "[`tiktoken`](https://github.com/openai/tiktoken/blob/main/README.md) is a fast open-source tokenizer by OpenAI.\n",
+    "\n",
+    "Given a text string (e.g., `\"tiktoken is great!\"`) and an encoding (e.g., `\"cl100k_base\"`), a tokenizer can split the text string into a list of tokens (e.g., `[\"t\", \"ik\", \"token\", \" is\", \" great\", \"!\"]`).\n",
+    "\n",
+    "Splitting text strings into tokens is useful because GPT models see text in the form of tokens. Knowing how many tokens are in a text string can tell you (a) whether the string is too long for a text model to process and (b) how much an OpenAI API call costs (as usage is priced by token).\n",
+    "\n",
+    "\n",
+    "## Encodings\n",
+    "\n",
+    "Encodings specify how text is converted into tokens. Different models use different encodings.\n",
+    "\n",
+    "`tiktoken` supports three encodings used by OpenAI models:\n",
+    "\n",
+    "| Encoding name           | OpenAI models                                       |\n",
+    "|-------------------------|-----------------------------------------------------|\n",
+    "| `o200k_base`            | `gpt-4o`, `gpt-4o-mini`                             |\n",
+    "| `cl100k_base`           | `gpt-4-turbo`, `gpt-4`, `gpt-3.5-turbo`, `text-embedding-ada-002`, `text-embedding-3-small`, `text-embedding-3-large`  |\n",
+    "| `p50k_base`             | Codex models, `text-davinci-002`, `text-davinci-003`|\n",
+    "| `r50k_base` (or `gpt2`) | GPT-3 models like `davinci`                         |\n",
+    "\n",
+    "You can retrieve the encoding for a model using `tiktoken.encoding_for_model()` as follows:\n",
+    "```python\n",
+    "encoding = tiktoken.encoding_for_model('gpt-4o-mini')\n",
+    "```\n",
+    "\n",
+    "Note that `p50k_base` overlaps substantially with `r50k_base`, and for non-code applications, they will usually give the same tokens.\n",
+    "\n",
+    "## Tokenizer libraries by language\n",
+    "\n",
+    "For `o200k_base`, `cl100k_base` and `p50k_base` encodings:\n",
+    "- Python: [tiktoken](https://github.com/openai/tiktoken/blob/main/README.md)\n",
+    "- .NET / C#: [SharpToken](https://github.com/dmitry-brazhenko/SharpToken), [TiktokenSharp](https://github.com/aiqinxuancai/TiktokenSharp)\n",
+    "- Java: [jtokkit](https://github.com/knuddelsgmbh/jtokkit)\n",
+    "- Golang: [tiktoken-go](https://github.com/pkoukk/tiktoken-go)\n",
+    "- Rust: [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs)\n",
+    "\n",
+    "For `r50k_base` (`gpt2`) encodings, tokenizers are available in many languages.\n",
+    "- Python: [tiktoken](https://github.com/openai/tiktoken/blob/main/README.md) (or alternatively [GPT2TokenizerFast](https://huggingface.co/docs/transformers/model_doc/gpt2#transformers.GPT2TokenizerFast))\n",
+    "- JavaScript: [gpt-3-encoder](https://www.npmjs.com/package/gpt-3-encoder)\n",
+    "- .NET / C#: [GPT Tokenizer](https://github.com/dluc/openai-tools)\n",
+    "- Java: [gpt2-tokenizer-java](https://github.com/hyunwoongko/gpt2-tokenizer-java)\n",
+    "- PHP: [GPT-3-Encoder-PHP](https://github.com/CodeRevolutionPlugins/GPT-3-Encoder-PHP)\n",
+    "- Golang: [tiktoken-go](https://github.com/pkoukk/tiktoken-go)\n",
+    "- Rust: [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs)\n",
+    "\n",
+    "(OpenAI makes no endorsements or guarantees of third-party libraries.)\n",
+    "\n",
+    "\n",
+    "## How strings are typically tokenized\n",
+    "\n",
+    "In English, tokens commonly range in length from one character to one word (e.g., `\"t\"` or `\" great\"`), though in some languages tokens can be shorter than one character or longer than one word. Spaces are usually grouped with the starts of words (e.g., `\" is\"` instead of `\"is \"` or `\" \"`+`\"is\"`). You can quickly check how a string is tokenized at the [OpenAI Tokenizer](https://beta.openai.com/tokenizer), or the third-party [Tiktokenizer](https://tiktokenizer.vercel.app/) webapp."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 0. Install `tiktoken`\n",
+    "\n",
+    "If needed, install `tiktoken` with `pip`:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.0\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.2\u001b[0m\n",
+      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
+      "Note: you may need to restart the kernel to use updated packages.\n",
+      "\n",
+      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.0\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.2\u001b[0m\n",
+      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
+      "Note: you may need to restart the kernel to use updated packages.\n"
+     ]
+    }
+   ],
+   "source": [
+    "%pip install --upgrade tiktoken -q\n",
+    "%pip install --upgrade openai -q"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Import `tiktoken`"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import tiktoken"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Load an encoding\n",
+    "\n",
+    "Use `tiktoken.get_encoding()` to load an encoding by name.\n",
+    "\n",
+    "The first time this runs, it will require an internet connection to download. Later runs won't need an internet connection."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "encoding = tiktoken.get_encoding(\"cl100k_base\")\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Use `tiktoken.encoding_for_model()` to automatically load the correct encoding for a given model name."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "encoding = tiktoken.encoding_for_model(\"gpt-4o-mini\")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Turn text into tokens with `encoding.encode()`\n",
+    "\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The `.encode()` method converts a text string into a list of token integers."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[83, 8251, 2488, 382, 2212, 0]"
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "encoding.encode(\"tiktoken is great!\")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Count tokens by counting the length of the list returned by `.encode()`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def num_tokens_from_string(string: str, encoding_name: str) -> int:\n",
+    "    \"\"\"Returns the number of tokens in a text string.\"\"\"\n",
+    "    encoding = tiktoken.get_encoding(encoding_name)\n",
+    "    num_tokens = len(encoding.encode(string))\n",
+    "    return num_tokens"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "6"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "num_tokens_from_string(\"tiktoken is great!\", \"o200k_base\")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4. Turn tokens into text with `encoding.decode()`"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "`.decode()` converts a list of token integers to a string."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'tiktoken is great!'"
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "encoding.decode([83, 8251, 2488, 382, 2212, 0])"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Warning: although `.decode()` can be applied to single tokens, beware that it can be lossy for tokens that aren't on utf-8 boundaries."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "For single tokens, `.decode_single_token_bytes()` safely converts a single integer token to the bytes it represents."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[b't', b'ikt', b'oken', b' is', b' great', b'!']"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "[encoding.decode_single_token_bytes(token) for token in [83, 8251, 2488, 382, 2212, 0]]\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "(The `b` in front of the strings indicates that the strings are byte strings.)"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 5. Comparing encodings\n",
+    "\n",
+    "Different encodings vary in how they split words, group spaces, and handle non-English characters. Using the methods above, we can compare different encodings on a few example strings."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def compare_encodings(example_string: str) -> None:\n",
+    "    \"\"\"Prints a comparison of three string encodings.\"\"\"\n",
+    "    # print the example string\n",
+    "    print(f'\\nExample string: \"{example_string}\"')\n",
+    "    # for each encoding, print the # of tokens, the token integers, and the token bytes\n",
+    "    for encoding_name in [\"r50k_base\", \"p50k_base\", \"cl100k_base\", \"o200k_base\"]:\n",
+    "        encoding = tiktoken.get_encoding(encoding_name)\n",
+    "        token_integers = encoding.encode(example_string)\n",
+    "        num_tokens = len(token_integers)\n",
+    "        token_bytes = [encoding.decode_single_token_bytes(token) for token in token_integers]\n",
+    "        print()\n",
+    "        print(f\"{encoding_name}: {num_tokens} tokens\")\n",
+    "        print(f\"token integers: {token_integers}\")\n",
+    "        print(f\"token bytes: {token_bytes}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "Example string: \"antidisestablishmentarianism\"\n",
+      "\n",
+      "r50k_base: 5 tokens\n",
+      "token integers: [415, 29207, 44390, 3699, 1042]\n",
+      "token bytes: [b'ant', b'idis', b'establishment', b'arian', b'ism']\n",
+      "\n",
+      "p50k_base: 5 tokens\n",
+      "token integers: [415, 29207, 44390, 3699, 1042]\n",
+      "token bytes: [b'ant', b'idis', b'establishment', b'arian', b'ism']\n",
+      "\n",
+      "cl100k_base: 6 tokens\n",
+      "token integers: [519, 85342, 34500, 479, 8997, 2191]\n",
+      "token bytes: [b'ant', b'idis', b'establish', b'ment', b'arian', b'ism']\n",
+      "\n",
+      "o200k_base: 6 tokens\n",
+      "token integers: [493, 129901, 376, 160388, 21203, 2367]\n",
+      "token bytes: [b'ant', b'idis', b'est', b'ablishment', b'arian', b'ism']\n"
+     ]
+    }
+   ],
+   "source": [
+    "compare_encodings(\"antidisestablishmentarianism\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "Example string: \"2 + 2 = 4\"\n",
+      "\n",
+      "r50k_base: 5 tokens\n",
+      "token integers: [17, 1343, 362, 796, 604]\n",
+      "token bytes: [b'2', b' +', b' 2', b' =', b' 4']\n",
+      "\n",
+      "p50k_base: 5 tokens\n",
+      "token integers: [17, 1343, 362, 796, 604]\n",
+      "token bytes: [b'2', b' +', b' 2', b' =', b' 4']\n",
+      "\n",
+      "cl100k_base: 7 tokens\n",
+      "token integers: [17, 489, 220, 17, 284, 220, 19]\n",
+      "token bytes: [b'2', b' +', b' ', b'2', b' =', b' ', b'4']\n",
+      "\n",
+      "o200k_base: 7 tokens\n",
+      "token integers: [17, 659, 220, 17, 314, 220, 19]\n",
+      "token bytes: [b'2', b' +', b' ', b'2', b' =', b' ', b'4']\n"
+     ]
+    }
+   ],
+   "source": [
+    "compare_encodings(\"2 + 2 = 4\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "Example string: \"お誕生日おめでとう\"\n",
+      "\n",
+      "r50k_base: 14 tokens\n",
+      "token integers: [2515, 232, 45739, 243, 37955, 33768, 98, 2515, 232, 1792, 223, 30640, 30201, 29557]\n",
+      "token bytes: [b'\\xe3\\x81', b'\\x8a', b'\\xe8\\xaa', b'\\x95', b'\\xe7\\x94\\x9f', b'\\xe6\\x97', b'\\xa5', b'\\xe3\\x81', b'\\x8a', b'\\xe3\\x82', b'\\x81', b'\\xe3\\x81\\xa7', b'\\xe3\\x81\\xa8', b'\\xe3\\x81\\x86']\n",
+      "\n",
+      "p50k_base: 14 tokens\n",
+      "token integers: [2515, 232, 45739, 243, 37955, 33768, 98, 2515, 232, 1792, 223, 30640, 30201, 29557]\n",
+      "token bytes: [b'\\xe3\\x81', b'\\x8a', b'\\xe8\\xaa', b'\\x95', b'\\xe7\\x94\\x9f', b'\\xe6\\x97', b'\\xa5', b'\\xe3\\x81', b'\\x8a', b'\\xe3\\x82', b'\\x81', b'\\xe3\\x81\\xa7', b'\\xe3\\x81\\xa8', b'\\xe3\\x81\\x86']\n",
+      "\n",
+      "cl100k_base: 9 tokens\n",
+      "token integers: [33334, 45918, 243, 21990, 9080, 33334, 62004, 16556, 78699]\n",
+      "token bytes: [b'\\xe3\\x81\\x8a', b'\\xe8\\xaa', b'\\x95', b'\\xe7\\x94\\x9f', b'\\xe6\\x97\\xa5', b'\\xe3\\x81\\x8a', b'\\xe3\\x82\\x81', b'\\xe3\\x81\\xa7', b'\\xe3\\x81\\xa8\\xe3\\x81\\x86']\n",
+      "\n",
+      "o200k_base: 8 tokens\n",
+      "token integers: [8930, 9697, 243, 128225, 8930, 17693, 4344, 48669]\n",
+      "token bytes: [b'\\xe3\\x81\\x8a', b'\\xe8\\xaa', b'\\x95', b'\\xe7\\x94\\x9f\\xe6\\x97\\xa5', b'\\xe3\\x81\\x8a', b'\\xe3\\x82\\x81', b'\\xe3\\x81\\xa7', b'\\xe3\\x81\\xa8\\xe3\\x81\\x86']\n"
+     ]
+    }
+   ],
+   "source": [
+    "compare_encodings(\"お誕生日おめでとう\")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 6. Counting tokens for chat completions API calls\n",
+    "\n",
+    "ChatGPT models like `gpt-4o-mini` and `gpt-4` use tokens in the same way as older completions models, but because of their message-based formatting, it's more difficult to count how many tokens will be used by a conversation.\n",
+    "\n",
+    "Below is an example function for counting tokens for messages passed to `gpt-3.5-turbo`, `gpt-4`, `gpt-4o` and `gpt-4o-mini`.\n",
+    "\n",
+    "Note that the exact way that tokens are counted from messages may change from model to model. Consider the counts from the function below an estimate, not a timeless guarantee.\n",
+    "\n",
+    "In particular, requests that use the optional functions input will consume extra tokens on top of the estimates calculated below."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def num_tokens_from_messages(messages, model=\"gpt-4o-mini-2024-07-18\"):\n",
+    "    \"\"\"Return the number of tokens used by a list of messages.\"\"\"\n",
+    "    try:\n",
+    "        encoding = tiktoken.encoding_for_model(model)\n",
+    "    except KeyError:\n",
+    "        print(\"Warning: model not found. Using o200k_base encoding.\")\n",
+    "        encoding = tiktoken.get_encoding(\"o200k_base\")\n",
+    "    if model in {\n",
+    "        \"gpt-3.5-turbo-0125\",\n",
+    "        \"gpt-4-0314\",\n",
+    "        \"gpt-4-32k-0314\",\n",
+    "        \"gpt-4-0613\",\n",
+    "        \"gpt-4-32k-0613\",\n",
+    "        \"gpt-4o-mini-2024-07-18\",\n",
+    "        \"gpt-4o-2024-08-06\"\n",
+    "        }:\n",
+    "        tokens_per_message = 3\n",
+    "        tokens_per_name = 1\n",
+    "    elif \"gpt-3.5-turbo\" in model:\n",
+    "        print(\"Warning: gpt-3.5-turbo may update over time. Returning num tokens assuming gpt-3.5-turbo-0125.\")\n",
+    "        return num_tokens_from_messages(messages, model=\"gpt-3.5-turbo-0125\")\n",
+    "    elif \"gpt-4o-mini\" in model:\n",
+    "        print(\"Warning: gpt-4o-mini may update over time. Returning num tokens assuming gpt-4o-mini-2024-07-18.\")\n",
+    "        return num_tokens_from_messages(messages, model=\"gpt-4o-mini-2024-07-18\")\n",
+    "    elif \"gpt-4o\" in model:\n",
+    "        print(\"Warning: gpt-4o and gpt-4o-mini may update over time. Returning num tokens assuming gpt-4o-2024-08-06.\")\n",
+    "        return num_tokens_from_messages(messages, model=\"gpt-4o-2024-08-06\")\n",
+    "    elif \"gpt-4\" in model:\n",
+    "        print(\"Warning: gpt-4 may update over time. Returning num tokens assuming gpt-4-0613.\")\n",
+    "        return num_tokens_from_messages(messages, model=\"gpt-4-0613\")\n",
+    "    else:\n",
+    "        raise NotImplementedError(\n",
+    "            f\"\"\"num_tokens_from_messages() is not implemented for model {model}.\"\"\"\n",
+    "        )\n",
+    "    num_tokens = 0\n",
+    "    for message in messages:\n",
+    "        num_tokens += tokens_per_message\n",
+    "        for key, value in message.items():\n",
+    "            num_tokens += len(encoding.encode(value))\n",
+    "            if key == \"name\":\n",
+    "                num_tokens += tokens_per_name\n",
+    "    num_tokens += 3  # every reply is primed with <|start|>assistant<|message|>\n",
+    "    return num_tokens\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "gpt-3.5-turbo\n",
+      "Warning: gpt-3.5-turbo may update over time. Returning num tokens assuming gpt-3.5-turbo-0125.\n",
+      "129 prompt tokens counted by num_tokens_from_messages().\n",
+      "129 prompt tokens counted by the OpenAI API.\n",
+      "\n",
+      "gpt-4-0613\n",
+      "129 prompt tokens counted by num_tokens_from_messages().\n",
+      "129 prompt tokens counted by the OpenAI API.\n",
+      "\n",
+      "gpt-4\n",
+      "Warning: gpt-4 may update over time. Returning num tokens assuming gpt-4-0613.\n",
+      "129 prompt tokens counted by num_tokens_from_messages().\n",
+      "129 prompt tokens counted by the OpenAI API.\n",
+      "\n",
+      "gpt-4o\n",
+      "Warning: gpt-4o and gpt-4o-mini may update over time. Returning num tokens assuming gpt-4o-2024-08-06.\n",
+      "124 prompt tokens counted by num_tokens_from_messages().\n",
+      "124 prompt tokens counted by the OpenAI API.\n",
+      "\n",
+      "gpt-4o-mini\n",
+      "Warning: gpt-4o-mini may update over time. Returning num tokens assuming gpt-4o-mini-2024-07-18.\n",
+      "124 prompt tokens counted by num_tokens_from_messages().\n",
+      "124 prompt tokens counted by the OpenAI API.\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "# let's verify the function above matches the OpenAI API response\n",
+    "\n",
+    "from openai import OpenAI\n",
+    "import os\n",
+    "\n",
+    "client = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))\n",
+    "\n",
+    "example_messages = [\n",
+    "    {\n",
+    "        \"role\": \"system\",\n",
+    "        \"content\": \"You are a helpful, pattern-following assistant that translates corporate jargon into plain English.\",\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"system\",\n",
+    "        \"name\": \"example_user\",\n",
+    "        \"content\": \"New synergies will help drive top-line growth.\",\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"system\",\n",
+    "        \"name\": \"example_assistant\",\n",
+    "        \"content\": \"Things working well together will increase revenue.\",\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"system\",\n",
+    "        \"name\": \"example_user\",\n",
+    "        \"content\": \"Let's circle back when we have more bandwidth to touch base on opportunities for increased leverage.\",\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"system\",\n",
+    "        \"name\": \"example_assistant\",\n",
+    "        \"content\": \"Let's talk later when we're less busy about how to do better.\",\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"user\",\n",
+    "        \"content\": \"This late pivot means we don't have time to boil the ocean for the client deliverable.\",\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "for model in [\n",
+    "    \"gpt-3.5-turbo\",\n",
+    "    \"gpt-4-0613\",\n",
+    "    \"gpt-4\",\n",
+    "    \"gpt-4o\",\n",
+    "    \"gpt-4o-mini\"\n",
+    "    ]:\n",
+    "    print(model)\n",
+    "    # example token count from the function defined above\n",
+    "    print(f\"{num_tokens_from_messages(example_messages, model)} prompt tokens counted by num_tokens_from_messages().\")\n",
+    "    # example token count from the OpenAI API\n",
+    "    response = client.chat.completions.create(model=model,\n",
+    "    messages=example_messages,\n",
+    "    temperature=0,\n",
+    "    max_tokens=1)\n",
+    "    print(f'{response.usage.prompt_tokens} prompt tokens counted by the OpenAI API.')\n",
+    "    print()\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 7. Counting tokens for chat completions with tool calls\n",
+    "\n",
+    "Next, we will look into how to apply this calculations to messages that may contain function calls. This is not immediately trivial, due to the formatting of the tools themselves. \n",
+    "\n",
+    "Below is an example function for counting tokens for messages that contain tools, passed to `gpt-3.5-turbo`, `gpt-4`, `gpt-4o` and `gpt-4o-mini`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def num_tokens_for_tools(functions, messages, model):\n",
+    "    \n",
+    "    # Initialize function settings to 0\n",
+    "    func_init = 0\n",
+    "    prop_init = 0\n",
+    "    prop_key = 0\n",
+    "    enum_init = 0\n",
+    "    enum_item = 0\n",
+    "    func_end = 0\n",
+    "    \n",
+    "    if model in [\n",
+    "        \"gpt-4o\",\n",
+    "        \"gpt-4o-mini\"\n",
+    "    ]:\n",
+    "        \n",
+    "        # Set function settings for the above models\n",
+    "        func_init = 7\n",
+    "        prop_init = 3\n",
+    "        prop_key = 3\n",
+    "        enum_init = -3\n",
+    "        enum_item = 3\n",
+    "        func_end = 12\n",
+    "    elif model in [\n",
+    "        \"gpt-3.5-turbo\",\n",
+    "        \"gpt-4\"\n",
+    "    ]:\n",
+    "        # Set function settings for the above models\n",
+    "        func_init = 10\n",
+    "        prop_init = 3\n",
+    "        prop_key = 3\n",
+    "        enum_init = -3\n",
+    "        enum_item = 3\n",
+    "        func_end = 12\n",
+    "    else:\n",
+    "        raise NotImplementedError(\n",
+    "            f\"\"\"num_tokens_for_tools() is not implemented for model {model}.\"\"\"\n",
+    "        )\n",
+    "    \n",
+    "    try:\n",
+    "        encoding = tiktoken.encoding_for_model(model)\n",
+    "    except KeyError:\n",
+    "        print(\"Warning: model not found. Using o200k_base encoding.\")\n",
+    "        encoding = tiktoken.get_encoding(\"o200k_base\")\n",
+    "    \n",
+    "    func_token_count = 0\n",
+    "    if len(functions) > 0:\n",
+    "        for f in functions:\n",
+    "            func_token_count += func_init  # Add tokens for start of each function\n",
+    "            function = f[\"function\"]\n",
+    "            f_name = function[\"name\"]\n",
+    "            f_desc = function[\"description\"]\n",
+    "            if f_desc.endswith(\".\"):\n",
+    "                f_desc = f_desc[:-1]\n",
+    "            line = f_name + \":\" + f_desc\n",
+    "            func_token_count += len(encoding.encode(line))  # Add tokens for set name and description\n",
+    "            if len(function[\"parameters\"][\"properties\"]) > 0:\n",
+    "                func_token_count += prop_init  # Add tokens for start of each property\n",
+    "                for key in list(function[\"parameters\"][\"properties\"].keys()):\n",
+    "                    func_token_count += prop_key  # Add tokens for each set property\n",
+    "                    p_name = key\n",
+    "                    p_type = function[\"parameters\"][\"properties\"][key][\"type\"]\n",
+    "                    p_desc = function[\"parameters\"][\"properties\"][key][\"description\"]\n",
+    "                    if \"enum\" in function[\"parameters\"][\"properties\"][key].keys():\n",
+    "                        func_token_count += enum_init  # Add tokens if property has enum list\n",
+    "                        for item in function[\"parameters\"][\"properties\"][key][\"enum\"]:\n",
+    "                            func_token_count += enum_item\n",
+    "                            func_token_count += len(encoding.encode(item))\n",
+    "                    if p_desc.endswith(\".\"):\n",
+    "                        p_desc = p_desc[:-1]\n",
+    "                    line = f\"{p_name}:{p_type}:{p_desc}\"\n",
+    "                    func_token_count += len(encoding.encode(line))\n",
+    "        func_token_count += func_end\n",
+    "        \n",
+    "    messages_token_count = num_tokens_from_messages(messages, model)\n",
+    "    total_tokens = messages_token_count + func_token_count\n",
+    "    \n",
+    "    return total_tokens"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "gpt-3.5-turbo\n",
+      "Warning: gpt-3.5-turbo may update over time. Returning num tokens assuming gpt-3.5-turbo-0125.\n",
+      "105 prompt tokens counted by num_tokens_for_tools().\n",
+      "105 prompt tokens counted by the OpenAI API.\n",
+      "\n",
+      "gpt-4\n",
+      "Warning: gpt-4 may update over time. Returning num tokens assuming gpt-4-0613.\n",
+      "105 prompt tokens counted by num_tokens_for_tools().\n",
+      "105 prompt tokens counted by the OpenAI API.\n",
+      "\n",
+      "gpt-4o\n",
+      "Warning: gpt-4o and gpt-4o-mini may update over time. Returning num tokens assuming gpt-4o-2024-08-06.\n",
+      "101 prompt tokens counted by num_tokens_for_tools().\n",
+      "101 prompt tokens counted by the OpenAI API.\n",
+      "\n",
+      "gpt-4o-mini\n",
+      "Warning: gpt-4o-mini may update over time. Returning num tokens assuming gpt-4o-mini-2024-07-18.\n",
+      "101 prompt tokens counted by num_tokens_for_tools().\n",
+      "101 prompt tokens counted by the OpenAI API.\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "tools = [\n",
+    "  {\n",
+    "    \"type\": \"function\",\n",
+    "    \"function\": {\n",
+    "      \"name\": \"get_current_weather\",\n",
+    "      \"description\": \"Get the current weather in a given location\",\n",
+    "      \"parameters\": {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\n",
+    "          \"location\": {\n",
+    "            \"type\": \"string\",\n",
+    "            \"description\": \"The city and state, e.g. San Francisco, CA\",\n",
+    "          },\n",
+    "          \"unit\": {\"type\": \"string\", \n",
+    "                   \"description\": \"The unit of temperature to return\",\n",
+    "                   \"enum\": [\"celsius\", \"fahrenheit\"]},\n",
+    "        },\n",
+    "        \"required\": [\"location\"],\n",
+    "      },\n",
+    "    }\n",
+    "  }\n",
+    "]\n",
+    "\n",
+    "example_messages = [\n",
+    "    {\n",
+    "        \"role\": \"system\",\n",
+    "        \"content\": \"You are a helpful assistant that can answer to questions about the weather.\",\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"user\",\n",
+    "        \"content\": \"What's the weather like in San Francisco?\",\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "for model in [\n",
+    "    \"gpt-3.5-turbo\",\n",
+    "    \"gpt-4\",\n",
+    "    \"gpt-4o\",\n",
+    "    \"gpt-4o-mini\"\n",
+    "    ]:\n",
+    "    print(model)\n",
+    "    # example token count from the function defined above\n",
+    "    print(f\"{num_tokens_for_tools(tools, example_messages, model)} prompt tokens counted by num_tokens_for_tools().\")\n",
+    "    # example token count from the OpenAI API\n",
+    "    response = client.chat.completions.create(model=model,\n",
+    "          messages=example_messages,\n",
+    "          tools=tools,\n",
+    "          temperature=0)\n",
+    "    print(f'{response.usage.prompt_tokens} prompt tokens counted by the OpenAI API.')\n",
+    "    print()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.7"
+  },
+  "vscode": {
+   "interpreter": {
+    "hash": "365536dcbde60510dc9073d6b991cd35db2d9bac356a11f5b64279a5e6708b97"
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/How_to_finetune_chat_models.ipynb

@@ -0,0 +1,709 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "7bcdad0e-b67c-4927-b00e-3a4d950cd8ce",
+   "metadata": {},
+   "source": [
+    "# How to fine-tune chat models\n",
+    "\n",
+    "Fine-tuning improves the model by training on many more examples than can fit in a prompt, letting you achieve better results on a wide number of tasks. This notebook provides a step-by-step guide for our new GPT-4o mini fine-tuning. We'll perform entity extraction using the [RecipeNLG dataset](https://github.com/Glorf/recipenlg), which provides various recipes and a list of extracted generic ingredients for each. This is a common dataset for named entity recognition (NER) tasks.\n",
+    "\n",
+    "Note: **GPT-4o mini fine-tuning is available to developers in our [Tier 4 and 5 usage tiers](https://platform.openai.com/docs/guides/rate-limits/usage-tiers).** You can start fine-tuning GPT-4o mini by visiting your fine-tuning dashboard, clicking \"create\", and selecting “gpt-4o-mini-2024-07-18” from the base model drop-down.\n",
+    "\n",
+    "We will go through the following steps:\n",
+    "\n",
+    "1. **Setup:** Loading our dataset and filtering down to one domain to fine-tune on.\n",
+    "2. **Data preparation:** Preparing your data for fine-tuning by creating training and validation examples, and uploading them to the `Files` endpoint.\n",
+    "3. **Fine-tuning:** Creating your fine-tuned model.\n",
+    "4. **Inference:** Using your fine-tuned model for inference on new inputs.\n",
+    "\n",
+    "By the end of this you should be able to train, evaluate and deploy a fine-tuned `gpt-4o-mini-2024-07-18` model.\n",
+    "\n",
+    "For more information on fine-tuning, you can refer to our [documentation guide](https://platform.openai.com/docs/guides/fine-tuning) or [API reference](https://platform.openai.com/docs/api-reference/fine-tuning).\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6f49cb10-f895-41f4-aa97-da606d0084d4",
+   "metadata": {},
+   "source": [
+    "## Setup\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "32036e70",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# make sure to use the latest version of the openai python package\n",
+    "!pip install --upgrade --quiet openai"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 69,
+   "id": "6e1f4403-37e1-4115-a215-12fd7daa1eb6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "import openai\n",
+    "import os\n",
+    "import pandas as pd\n",
+    "from pprint import pprint\n",
+    "\n",
+    "client = openai.OpenAI(\n",
+    "    api_key=os.environ.get(\"OPENAI_API_KEY\"),\n",
+    "    organization=\"<org id>\",\n",
+    "    project=\"<project id>\",\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a468d660",
+   "metadata": {},
+   "source": [
+    "Fine-tuning works best when focused on a particular domain. It's important to make sure your dataset is both focused enough for the model to learn, but general enough that unseen examples won't be missed. Having this in mind, we have extracted a subset from the RecipesNLG dataset to only contain documents from [cookbooks.com](https://cookbooks.com/).\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 70,
+   "id": "f57ebc23-14b7-47f9-90b8-1d791ccfc9bc",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>title</th>\n",
+       "      <th>ingredients</th>\n",
+       "      <th>directions</th>\n",
+       "      <th>link</th>\n",
+       "      <th>source</th>\n",
+       "      <th>NER</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>No-Bake Nut Cookies</td>\n",
+       "      <td>[\"1 c. firmly packed brown sugar\", \"1/2 c. eva...</td>\n",
+       "      <td>[\"In a heavy 2-quart saucepan, mix brown sugar...</td>\n",
+       "      <td>www.cookbooks.com/Recipe-Details.aspx?id=44874</td>\n",
+       "      <td>www.cookbooks.com</td>\n",
+       "      <td>[\"brown sugar\", \"milk\", \"vanilla\", \"nuts\", \"bu...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>Jewell Ball'S Chicken</td>\n",
+       "      <td>[\"1 small jar chipped beef, cut up\", \"4 boned ...</td>\n",
+       "      <td>[\"Place chipped beef on bottom of baking dish....</td>\n",
+       "      <td>www.cookbooks.com/Recipe-Details.aspx?id=699419</td>\n",
+       "      <td>www.cookbooks.com</td>\n",
+       "      <td>[\"beef\", \"chicken breasts\", \"cream of mushroom...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>Creamy Corn</td>\n",
+       "      <td>[\"2 (16 oz.) pkg. frozen corn\", \"1 (8 oz.) pkg...</td>\n",
+       "      <td>[\"In a slow cooker, combine all ingredients. C...</td>\n",
+       "      <td>www.cookbooks.com/Recipe-Details.aspx?id=10570</td>\n",
+       "      <td>www.cookbooks.com</td>\n",
+       "      <td>[\"frozen corn\", \"cream cheese\", \"butter\", \"gar...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>Chicken Funny</td>\n",
+       "      <td>[\"1 large whole chicken\", \"2 (10 1/2 oz.) cans...</td>\n",
+       "      <td>[\"Boil and debone chicken.\", \"Put bite size pi...</td>\n",
+       "      <td>www.cookbooks.com/Recipe-Details.aspx?id=897570</td>\n",
+       "      <td>www.cookbooks.com</td>\n",
+       "      <td>[\"chicken\", \"chicken gravy\", \"cream of mushroo...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>Reeses Cups(Candy)</td>\n",
+       "      <td>[\"1 c. peanut butter\", \"3/4 c. graham cracker ...</td>\n",
+       "      <td>[\"Combine first four ingredients and press in ...</td>\n",
+       "      <td>www.cookbooks.com/Recipe-Details.aspx?id=659239</td>\n",
+       "      <td>www.cookbooks.com</td>\n",
+       "      <td>[\"peanut butter\", \"graham cracker crumbs\", \"bu...</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "                   title                                        ingredients  \\\n",
+       "0    No-Bake Nut Cookies  [\"1 c. firmly packed brown sugar\", \"1/2 c. eva...   \n",
+       "1  Jewell Ball'S Chicken  [\"1 small jar chipped beef, cut up\", \"4 boned ...   \n",
+       "2            Creamy Corn  [\"2 (16 oz.) pkg. frozen corn\", \"1 (8 oz.) pkg...   \n",
+       "3          Chicken Funny  [\"1 large whole chicken\", \"2 (10 1/2 oz.) cans...   \n",
+       "4   Reeses Cups(Candy)    [\"1 c. peanut butter\", \"3/4 c. graham cracker ...   \n",
+       "\n",
+       "                                          directions  \\\n",
+       "0  [\"In a heavy 2-quart saucepan, mix brown sugar...   \n",
+       "1  [\"Place chipped beef on bottom of baking dish....   \n",
+       "2  [\"In a slow cooker, combine all ingredients. C...   \n",
+       "3  [\"Boil and debone chicken.\", \"Put bite size pi...   \n",
+       "4  [\"Combine first four ingredients and press in ...   \n",
+       "\n",
+       "                                              link             source  \\\n",
+       "0   www.cookbooks.com/Recipe-Details.aspx?id=44874  www.cookbooks.com   \n",
+       "1  www.cookbooks.com/Recipe-Details.aspx?id=699419  www.cookbooks.com   \n",
+       "2   www.cookbooks.com/Recipe-Details.aspx?id=10570  www.cookbooks.com   \n",
+       "3  www.cookbooks.com/Recipe-Details.aspx?id=897570  www.cookbooks.com   \n",
+       "4  www.cookbooks.com/Recipe-Details.aspx?id=659239  www.cookbooks.com   \n",
+       "\n",
+       "                                                 NER  \n",
+       "0  [\"brown sugar\", \"milk\", \"vanilla\", \"nuts\", \"bu...  \n",
+       "1  [\"beef\", \"chicken breasts\", \"cream of mushroom...  \n",
+       "2  [\"frozen corn\", \"cream cheese\", \"butter\", \"gar...  \n",
+       "3  [\"chicken\", \"chicken gravy\", \"cream of mushroo...  \n",
+       "4  [\"peanut butter\", \"graham cracker crumbs\", \"bu...  "
+      ]
+     },
+     "execution_count": 70,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# Read in the dataset we'll use for this task.\n",
+    "# This will be the RecipesNLG dataset, which we've cleaned to only contain documents from www.cookbooks.com\n",
+    "recipe_df = pd.read_csv(\"data/cookbook_recipes_nlg_10k.csv\")\n",
+    "\n",
+    "recipe_df.head()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2b3151e9-8715-47bd-a153-195d6a0d0a70",
+   "metadata": {},
+   "source": [
+    "## Data preparation\n",
+    "\n",
+    "We'll begin by preparing our data. When fine-tuning with the `ChatCompletion` format, each training example is a simple list of `messages`. For example, an entry could look like:\n",
+    "\n",
+    "```\n",
+    "[{'role': 'system',\n",
+    "  'content': 'You are a helpful recipe assistant. You are to extract the generic ingredients from each of the recipes provided.'},\n",
+    "\n",
+    " {'role': 'user',\n",
+    "  'content': 'Title: No-Bake Nut Cookies\\n\\nIngredients: [\"1 c. firmly packed brown sugar\", \"1/2 c. evaporated milk\", \"1/2 tsp. vanilla\", \"1/2 c. broken nuts (pecans)\", \"2 Tbsp. butter or margarine\", \"3 1/2 c. bite size shredded rice biscuits\"]\\n\\nGeneric ingredients: '},\n",
+    "\n",
+    " {'role': 'assistant',\n",
+    "  'content': '[\"brown sugar\", \"milk\", \"vanilla\", \"nuts\", \"butter\", \"bite size shredded rice biscuits\"]'}]\n",
+    "```\n",
+    "\n",
+    "During the training process this conversation will be split, with the final entry being the `completion` that the model will produce, and the remainder of the `messages` acting as the prompt. Consider this when building your training examples - if your model will act on multi-turn conversations, then please provide representative examples so it doesn't perform poorly when the conversation starts to expand.\n",
+    "\n",
+    "Please note that currently there is a 4096 token limit for each training example. Anything longer than this will be truncated at 4096 tokens.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 71,
+   "id": "9a8216b0-d1dc-472d-b07d-1be03acd70a5",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{'messages': [{'content': 'You are a helpful recipe assistant. You are to '\n",
+      "                          'extract the generic ingredients from each of the '\n",
+      "                          'recipes provided.',\n",
+      "               'role': 'system'},\n",
+      "              {'content': 'Title: No-Bake Nut Cookies\\n'\n",
+      "                          '\\n'\n",
+      "                          'Ingredients: [\"1 c. firmly packed brown sugar\", '\n",
+      "                          '\"1/2 c. evaporated milk\", \"1/2 tsp. vanilla\", \"1/2 '\n",
+      "                          'c. broken nuts (pecans)\", \"2 Tbsp. butter or '\n",
+      "                          'margarine\", \"3 1/2 c. bite size shredded rice '\n",
+      "                          'biscuits\"]\\n'\n",
+      "                          '\\n'\n",
+      "                          'Generic ingredients: ',\n",
+      "               'role': 'user'},\n",
+      "              {'content': '[\"brown sugar\", \"milk\", \"vanilla\", \"nuts\", '\n",
+      "                          '\"butter\", \"bite size shredded rice biscuits\"]',\n",
+      "               'role': 'assistant'}]}\n"
+     ]
+    }
+   ],
+   "source": [
+    "system_message = \"You are a helpful recipe assistant. You are to extract the generic ingredients from each of the recipes provided.\"\n",
+    "\n",
+    "\n",
+    "def create_user_message(row):\n",
+    "    return f\"Title: {row['title']}\\n\\nIngredients: {row['ingredients']}\\n\\nGeneric ingredients: \"\n",
+    "\n",
+    "\n",
+    "def prepare_example_conversation(row):\n",
+    "    return {\n",
+    "        \"messages\": [\n",
+    "            {\"role\": \"system\", \"content\": system_message},\n",
+    "            {\"role\": \"user\", \"content\": create_user_message(row)},\n",
+    "            {\"role\": \"assistant\", \"content\": row[\"NER\"]},\n",
+    "        ]\n",
+    "    }\n",
+    "\n",
+    "\n",
+    "pprint(prepare_example_conversation(recipe_df.iloc[0]))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "82fa4fae",
+   "metadata": {},
+   "source": [
+    "Let's now do this for a subset of the dataset to use as our training data. You can begin with even 30-50 well-pruned examples. You should see performance continue to scale linearly as you increase the size of the training set, but your jobs will also take longer.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 72,
+   "id": "8f37aff9",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{'messages': [{'role': 'system', 'content': 'You are a helpful recipe assistant. You are to extract the generic ingredients from each of the recipes provided.'}, {'role': 'user', 'content': 'Title: No-Bake Nut Cookies\\n\\nIngredients: [\"1 c. firmly packed brown sugar\", \"1/2 c. evaporated milk\", \"1/2 tsp. vanilla\", \"1/2 c. broken nuts (pecans)\", \"2 Tbsp. butter or margarine\", \"3 1/2 c. bite size shredded rice biscuits\"]\\n\\nGeneric ingredients: '}, {'role': 'assistant', 'content': '[\"brown sugar\", \"milk\", \"vanilla\", \"nuts\", \"butter\", \"bite size shredded rice biscuits\"]'}]}\n",
+      "{'messages': [{'role': 'system', 'content': 'You are a helpful recipe assistant. You are to extract the generic ingredients from each of the recipes provided.'}, {'role': 'user', 'content': 'Title: Jewell Ball\\'S Chicken\\n\\nIngredients: [\"1 small jar chipped beef, cut up\", \"4 boned chicken breasts\", \"1 can cream of mushroom soup\", \"1 carton sour cream\"]\\n\\nGeneric ingredients: '}, {'role': 'assistant', 'content': '[\"beef\", \"chicken breasts\", \"cream of mushroom soup\", \"sour cream\"]'}]}\n",
+      "{'messages': [{'role': 'system', 'content': 'You are a helpful recipe assistant. You are to extract the generic ingredients from each of the recipes provided.'}, {'role': 'user', 'content': 'Title: Creamy Corn\\n\\nIngredients: [\"2 (16 oz.) pkg. frozen corn\", \"1 (8 oz.) pkg. cream cheese, cubed\", \"1/3 c. butter, cubed\", \"1/2 tsp. garlic powder\", \"1/2 tsp. salt\", \"1/4 tsp. pepper\"]\\n\\nGeneric ingredients: '}, {'role': 'assistant', 'content': '[\"frozen corn\", \"cream cheese\", \"butter\", \"garlic powder\", \"salt\", \"pepper\"]'}]}\n",
+      "{'messages': [{'role': 'system', 'content': 'You are a helpful recipe assistant. You are to extract the generic ingredients from each of the recipes provided.'}, {'role': 'user', 'content': 'Title: Chicken Funny\\n\\nIngredients: [\"1 large whole chicken\", \"2 (10 1/2 oz.) cans chicken gravy\", \"1 (10 1/2 oz.) can cream of mushroom soup\", \"1 (6 oz.) box Stove Top stuffing\", \"4 oz. shredded cheese\"]\\n\\nGeneric ingredients: '}, {'role': 'assistant', 'content': '[\"chicken\", \"chicken gravy\", \"cream of mushroom soup\", \"shredded cheese\"]'}]}\n",
+      "{'messages': [{'role': 'system', 'content': 'You are a helpful recipe assistant. You are to extract the generic ingredients from each of the recipes provided.'}, {'role': 'user', 'content': 'Title: Reeses Cups(Candy)  \\n\\nIngredients: [\"1 c. peanut butter\", \"3/4 c. graham cracker crumbs\", \"1 c. melted butter\", \"1 lb. (3 1/2 c.) powdered sugar\", \"1 large pkg. chocolate chips\"]\\n\\nGeneric ingredients: '}, {'role': 'assistant', 'content': '[\"peanut butter\", \"graham cracker crumbs\", \"butter\", \"powdered sugar\", \"chocolate chips\"]'}]}\n"
+     ]
+    }
+   ],
+   "source": [
+    "# use the first 100 rows of the dataset for training\n",
+    "training_df = recipe_df.loc[0:100]\n",
+    "\n",
+    "# apply the prepare_example_conversation function to each row of the training_df\n",
+    "training_data = training_df.apply(prepare_example_conversation, axis=1).tolist()\n",
+    "\n",
+    "for example in training_data[:5]:\n",
+    "    print(example)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6fb0f6a7",
+   "metadata": {},
+   "source": [
+    "In addition to training data, we can also **optionally** provide validation data, which will be used to make sure that the model does not overfit your training set.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 73,
+   "id": "5b853efa-dfea-4770-ab88-9b7e17794421",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "validation_df = recipe_df.loc[101:200]\n",
+    "validation_data = validation_df.apply(\n",
+    "    prepare_example_conversation, axis=1).tolist()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1d5e7bfe-f6c8-4a23-a951-3df3f3791d7f",
+   "metadata": {},
+   "source": [
+    "We then need to save our data as `.jsonl` files, with each line being one training example conversation.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 74,
+   "id": "8d2eb207-2c2b-43f6-a613-64a7e92d494d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def write_jsonl(data_list: list, filename: str) -> None:\n",
+    "    with open(filename, \"w\") as out:\n",
+    "        for ddict in data_list:\n",
+    "            jout = json.dumps(ddict) + \"\\n\"\n",
+    "            out.write(jout)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 75,
+   "id": "8b53e7a2-1cac-4c5f-8ba4-3292ba2a0770",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "training_file_name = \"tmp_recipe_finetune_training.jsonl\"\n",
+    "write_jsonl(training_data, training_file_name)\n",
+    "\n",
+    "validation_file_name = \"tmp_recipe_finetune_validation.jsonl\"\n",
+    "write_jsonl(validation_data, validation_file_name)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "80b274a3",
+   "metadata": {},
+   "source": [
+    "This is what the first 5 lines of our training `.jsonl` file look like:\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 76,
+   "id": "283c4ec2",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{\"messages\": [{\"role\": \"system\", \"content\": \"You are a helpful recipe assistant. You are to extract the generic ingredients from each of the recipes provided.\"}, {\"role\": \"user\", \"content\": \"Title: No-Bake Nut Cookies\\n\\nIngredients: [\\\"1 c. firmly packed brown sugar\\\", \\\"1/2 c. evaporated milk\\\", \\\"1/2 tsp. vanilla\\\", \\\"1/2 c. broken nuts (pecans)\\\", \\\"2 Tbsp. butter or margarine\\\", \\\"3 1/2 c. bite size shredded rice biscuits\\\"]\\n\\nGeneric ingredients: \"}, {\"role\": \"assistant\", \"content\": \"[\\\"brown sugar\\\", \\\"milk\\\", \\\"vanilla\\\", \\\"nuts\\\", \\\"butter\\\", \\\"bite size shredded rice biscuits\\\"]\"}]}\n",
+      "{\"messages\": [{\"role\": \"system\", \"content\": \"You are a helpful recipe assistant. You are to extract the generic ingredients from each of the recipes provided.\"}, {\"role\": \"user\", \"content\": \"Title: Jewell Ball'S Chicken\\n\\nIngredients: [\\\"1 small jar chipped beef, cut up\\\", \\\"4 boned chicken breasts\\\", \\\"1 can cream of mushroom soup\\\", \\\"1 carton sour cream\\\"]\\n\\nGeneric ingredients: \"}, {\"role\": \"assistant\", \"content\": \"[\\\"beef\\\", \\\"chicken breasts\\\", \\\"cream of mushroom soup\\\", \\\"sour cream\\\"]\"}]}\n",
+      "{\"messages\": [{\"role\": \"system\", \"content\": \"You are a helpful recipe assistant. You are to extract the generic ingredients from each of the recipes provided.\"}, {\"role\": \"user\", \"content\": \"Title: Creamy Corn\\n\\nIngredients: [\\\"2 (16 oz.) pkg. frozen corn\\\", \\\"1 (8 oz.) pkg. cream cheese, cubed\\\", \\\"1/3 c. butter, cubed\\\", \\\"1/2 tsp. garlic powder\\\", \\\"1/2 tsp. salt\\\", \\\"1/4 tsp. pepper\\\"]\\n\\nGeneric ingredients: \"}, {\"role\": \"assistant\", \"content\": \"[\\\"frozen corn\\\", \\\"cream cheese\\\", \\\"butter\\\", \\\"garlic powder\\\", \\\"salt\\\", \\\"pepper\\\"]\"}]}\n",
+      "{\"messages\": [{\"role\": \"system\", \"content\": \"You are a helpful recipe assistant. You are to extract the generic ingredients from each of the recipes provided.\"}, {\"role\": \"user\", \"content\": \"Title: Chicken Funny\\n\\nIngredients: [\\\"1 large whole chicken\\\", \\\"2 (10 1/2 oz.) cans chicken gravy\\\", \\\"1 (10 1/2 oz.) can cream of mushroom soup\\\", \\\"1 (6 oz.) box Stove Top stuffing\\\", \\\"4 oz. shredded cheese\\\"]\\n\\nGeneric ingredients: \"}, {\"role\": \"assistant\", \"content\": \"[\\\"chicken\\\", \\\"chicken gravy\\\", \\\"cream of mushroom soup\\\", \\\"shredded cheese\\\"]\"}]}\n",
+      "{\"messages\": [{\"role\": \"system\", \"content\": \"You are a helpful recipe assistant. You are to extract the generic ingredients from each of the recipes provided.\"}, {\"role\": \"user\", \"content\": \"Title: Reeses Cups(Candy)  \\n\\nIngredients: [\\\"1 c. peanut butter\\\", \\\"3/4 c. graham cracker crumbs\\\", \\\"1 c. melted butter\\\", \\\"1 lb. (3 1/2 c.) powdered sugar\\\", \\\"1 large pkg. chocolate chips\\\"]\\n\\nGeneric ingredients: \"}, {\"role\": \"assistant\", \"content\": \"[\\\"peanut butter\\\", \\\"graham cracker crumbs\\\", \\\"butter\\\", \\\"powdered sugar\\\", \\\"chocolate chips\\\"]\"}]}\n"
+     ]
+    }
+   ],
+   "source": [
+    "# print the first 5 lines of the training file\n",
+    "!head -n 5 tmp_recipe_finetune_training.jsonl"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0d149e2e-50dd-45c1-bd8d-1291975670b4",
+   "metadata": {},
+   "source": [
+    "### Upload files\n",
+    "\n",
+    "You can now upload the files to our `Files` endpoint to be used by the fine-tuned model.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 77,
+   "id": "69462d9e-e6bd-49b9-a064-9eae4ea5b7a8",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Training file ID: file-3wfAfDoYcGrSpaE17qK0vXT0\n",
+      "Validation file ID: file-HhFhnyGJhazYdPcd3wrtvIoX\n"
+     ]
+    }
+   ],
+   "source": [
+    "def upload_file(file_name: str, purpose: str) -> str:\n",
+    "    with open(file_name, \"rb\") as file_fd:\n",
+    "        response = client.files.create(file=file_fd, purpose=purpose)\n",
+    "    return response.id\n",
+    "\n",
+    "\n",
+    "training_file_id = upload_file(training_file_name, \"fine-tune\")\n",
+    "validation_file_id = upload_file(validation_file_name, \"fine-tune\")\n",
+    "\n",
+    "print(\"Training file ID:\", training_file_id)\n",
+    "print(\"Validation file ID:\", validation_file_id)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d61cd381-63ad-4ed9-b0be-47a438891028",
+   "metadata": {},
+   "source": [
+    "## Fine-tuning\n",
+    "\n",
+    "Now we can create our fine-tuning job with the generated files and an optional suffix to identify the model. The response will contain an `id` which you can use to retrieve updates on the job.\n",
+    "\n",
+    "Note: The files have to first be processed by our system, so you might get a `File not ready` error. In that case, simply retry a few minutes later.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 81,
+   "id": "05541ceb-5628-447e-962d-7e57c112439c",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Job ID: ftjob-UiaiLwGdGBfdLQDBAoQheufN\n",
+      "Status: validating_files\n"
+     ]
+    }
+   ],
+   "source": [
+    "MODEL = \"gpt-4o-mini-2024-07-18\"\n",
+    "\n",
+    "response = client.fine_tuning.jobs.create(\n",
+    "    training_file=training_file_id,\n",
+    "    validation_file=validation_file_id,\n",
+    "    model=MODEL,\n",
+    "    suffix=\"recipe-ner\",\n",
+    ")\n",
+    "\n",
+    "job_id = response.id\n",
+    "\n",
+    "print(\"Job ID:\", response.id)\n",
+    "print(\"Status:\", response.status)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1de3ed71-f2d4-4138-95a3-70da187a007e",
+   "metadata": {},
+   "source": [
+    "#### Check job status\n",
+    "\n",
+    "You can make a `GET` request to the `https://api.openai.com/v1/alpha/fine-tunes` endpoint to list your alpha fine-tune jobs. In this instance you'll want to check that the ID you got from the previous step ends up as `status: succeeded`.\n",
+    "\n",
+    "Once it is completed, you can use the `result_files` to sample the results from the validation set (if you uploaded one), and use the ID from the `fine_tuned_model` parameter to invoke your trained model.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 91,
+   "id": "d7392f48",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Job ID: ftjob-UiaiLwGdGBfdLQDBAoQheufN\n",
+      "Status: running\n",
+      "Trained Tokens: None\n"
+     ]
+    }
+   ],
+   "source": [
+    "response = client.fine_tuning.jobs.retrieve(job_id)\n",
+    "\n",
+    "print(\"Job ID:\", response.id)\n",
+    "print(\"Status:\", response.status)\n",
+    "print(\"Trained Tokens:\", response.trained_tokens)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "30a57fbb",
+   "metadata": {},
+   "source": [
+    "We can track the progress of the fine-tune with the events endpoint. You can rerun the cell below a few times until the fine-tune is ready.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 94,
+   "id": "08cace28",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Step 288/303: training loss=0.00\n",
+      "Step 289/303: training loss=0.01\n",
+      "Step 290/303: training loss=0.00, validation loss=0.31\n",
+      "Step 291/303: training loss=0.00\n",
+      "Step 292/303: training loss=0.00\n",
+      "Step 293/303: training loss=0.00\n",
+      "Step 294/303: training loss=0.00\n",
+      "Step 295/303: training loss=0.00\n",
+      "Step 296/303: training loss=0.00\n",
+      "Step 297/303: training loss=0.00\n",
+      "Step 298/303: training loss=0.01\n",
+      "Step 299/303: training loss=0.00\n",
+      "Step 300/303: training loss=0.00, validation loss=0.04\n",
+      "Step 301/303: training loss=0.16\n",
+      "Step 302/303: training loss=0.00\n",
+      "Step 303/303: training loss=0.00, full validation loss=0.33\n",
+      "Checkpoint created at step 101 with Snapshot ID: ft:gpt-4o-mini-2024-07-18:openai-gtm:recipe-ner:9o1eNlSa:ckpt-step-101\n",
+      "Checkpoint created at step 202 with Snapshot ID: ft:gpt-4o-mini-2024-07-18:openai-gtm:recipe-ner:9o1eNFnj:ckpt-step-202\n",
+      "New fine-tuned model created: ft:gpt-4o-mini-2024-07-18:openai-gtm:recipe-ner:9o1eNNKO\n",
+      "The job has successfully completed\n"
+     ]
+    }
+   ],
+   "source": [
+    "response = client.fine_tuning.jobs.list_events(job_id)\n",
+    "\n",
+    "events = response.data\n",
+    "events.reverse()\n",
+    "\n",
+    "for event in events:\n",
+    "    print(event.message)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d0da4e32",
+   "metadata": {},
+   "source": [
+    "Now that it's done, we can get a fine-tuned model ID from the job:\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 95,
+   "id": "40b28c26",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Fine-tuned model ID: ft:gpt-4o-mini-2024-07-18:openai-gtm:recipe-ner:9o1eNNKO\n"
+     ]
+    }
+   ],
+   "source": [
+    "response = client.fine_tuning.jobs.retrieve(job_id)\n",
+    "fine_tuned_model_id = response.fine_tuned_model\n",
+    "\n",
+    "if fine_tuned_model_id is None:\n",
+    "    raise RuntimeError(\n",
+    "        \"Fine-tuned model ID not found. Your job has likely not been completed yet.\"\n",
+    "    )\n",
+    "\n",
+    "print(\"Fine-tuned model ID:\", fine_tuned_model_id)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0025e392-84cd-4566-a384-ea31ca43e567",
+   "metadata": {},
+   "source": [
+    "## Inference\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0ab9ac11",
+   "metadata": {},
+   "source": [
+    "The last step is to use your fine-tuned model for inference. Similar to the classic `FineTuning`, you simply call `ChatCompletions` with your new fine-tuned model name filling the `model` parameter.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 96,
+   "id": "1c7de631-b68f-4eff-9ae7-051641579c2b",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[{'content': 'You are a helpful recipe assistant. You are to extract the '\n",
+      "             'generic ingredients from each of the recipes provided.',\n",
+      "  'role': 'system'},\n",
+      " {'content': 'Title: Beef Brisket\\n'\n",
+      "             '\\n'\n",
+      "             'Ingredients: [\"4 lb. beef brisket\", \"1 c. catsup\", \"1 c. water\", '\n",
+      "             '\"1/2 onion, minced\", \"2 Tbsp. cider vinegar\", \"1 Tbsp. prepared '\n",
+      "             'horseradish\", \"1 Tbsp. prepared mustard\", \"1 tsp. salt\", \"1/2 '\n",
+      "             'tsp. pepper\"]\\n'\n",
+      "             '\\n'\n",
+      "             'Generic ingredients: ',\n",
+      "  'role': 'user'}]\n"
+     ]
+    }
+   ],
+   "source": [
+    "test_df = recipe_df.loc[201:300]\n",
+    "test_row = test_df.iloc[0]\n",
+    "test_messages = []\n",
+    "test_messages.append({\"role\": \"system\", \"content\": system_message})\n",
+    "user_message = create_user_message(test_row)\n",
+    "test_messages.append({\"role\": \"user\", \"content\": user_message})\n",
+    "\n",
+    "pprint(test_messages)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 97,
+   "id": "1a1d2589",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[\"beef brisket\", \"catsup\", \"water\", \"onion\", \"cider vinegar\", \"horseradish\", \"mustard\", \"salt\", \"pepper\"]\n"
+     ]
+    }
+   ],
+   "source": [
+    "response = client.chat.completions.create(\n",
+    "    model=fine_tuned_model_id, messages=test_messages, temperature=0, max_tokens=500\n",
+    ")\n",
+    "print(response.choices[0].message.content)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "07799909-3f2a-4274-b81e-dabc048be28f",
+   "metadata": {},
+   "source": [
+    "## Conclusion\n",
+    "\n",
+    "Congratulations, you are now ready to fine-tune your own models using the `ChatCompletion` format! We look forward to seeing what you build\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

examples/How_to_format_inputs_to_ChatGPT_models.ipynb

@@ -0,0 +1,681 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# How to format inputs to ChatGPT models\n",
+    "\n",
+    "ChatGPT is powered by `gpt-3.5-turbo` and `gpt-4`, OpenAI's most advanced models.\n",
+    "\n",
+    "You can build your own applications with `gpt-3.5-turbo` or `gpt-4` using the OpenAI API.\n",
+    "\n",
+    "Chat models take a series of messages as input, and return an AI-written message as output.\n",
+    "\n",
+    "This guide illustrates the chat format with a few example API calls."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Import the openai library"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# if needed, install and/or upgrade to the latest version of the OpenAI Python library\n",
+    "%pip install --upgrade openai\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# import the OpenAI Python library for calling the OpenAI API\n",
+    "from openai import OpenAI\n",
+    "import os\n",
+    "\n",
+    "client = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. An example chat completion API call\n",
+    "\n",
+    "A chat completion API call parameters,\n",
+    "**Required**\n",
+    "- `model`: the name of the model you want to use (e.g., `gpt-3.5-turbo`, `gpt-4`, `gpt-3.5-turbo-16k-1106`)\n",
+    "- `messages`: a list of message objects, where each object has two required fields:\n",
+    "    - `role`: the role of the messenger (either `system`, `user`, `assistant` or `tool`)\n",
+    "    - `content`: the content of the message (e.g., `Write me a beautiful poem`)\n",
+    "\n",
+    "Messages can also contain an optional `name` field, which give the messenger a name. E.g., `example-user`, `Alice`, `BlackbeardBot`. Names may not contain spaces.\n",
+    "\n",
+    "**Optional**\n",
+    "- `frequency_penalty`: Penalizes tokens based on their frequency, reducing repetition.\n",
+    "- `logit_bias`: Modifies likelihood of specified tokens with bias values.\n",
+    "- `logprobs`: Returns log probabilities of output tokens if true.\n",
+    "- `top_logprobs`: Specifies the number of most likely tokens to return at each position.\n",
+    "- `max_tokens`: Sets the maximum number of generated tokens in chat completion.\n",
+    "- `n`: Generates a specified number of chat completion choices for each input.\n",
+    "- `presence_penalty`: Penalizes new tokens based on their presence in the text.\n",
+    "- `response_format`: Specifies the output format, e.g., JSON mode.\n",
+    "- `seed`: Ensures deterministic sampling with a specified seed.\n",
+    "- `stop`: Specifies up to 4 sequences where the API should stop generating tokens.\n",
+    "- `stream`: Sends partial message deltas as tokens become available.\n",
+    "- `temperature`: Sets the sampling temperature between 0 and 2.\n",
+    "- `top_p`: Uses nucleus sampling; considers tokens with top_p probability mass.\n",
+    "- `tools`: Lists functions the model may call.\n",
+    "- `tool_choice`: Controls the model's function calls (none/auto/function).\n",
+    "- `user`: Unique identifier for end-user monitoring and abuse detection.\n",
+    "\n",
+    "\n",
+    "As of January 2024, you can also optionally submit a list of `functions` that tell GPT whether it can generate JSON to feed into a function. For details, see the [documentation](https://platform.openai.com/docs/guides/function-calling), [API reference](https://platform.openai.com/docs/api-reference/chat), or the Cookbook guide [How to call functions with chat models](How_to_call_functions_with_chat_models.ipynb).\n",
+    "\n",
+    "Typically, a conversation will start with a system message that tells the assistant how to behave, followed by alternating user and assistant messages, but you are not required to follow this format.\n",
+    "\n",
+    "Let's look at an example chat API calls to see how the chat format works in practice."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Example OpenAI Python library request\n",
+    "MODEL = \"gpt-3.5-turbo\"\n",
+    "response = client.chat.completions.create(\n",
+    "    model=MODEL,\n",
+    "    messages=[\n",
+    "        {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
+    "        {\"role\": \"user\", \"content\": \"Knock knock.\"},\n",
+    "        {\"role\": \"assistant\", \"content\": \"Who's there?\"},\n",
+    "        {\"role\": \"user\", \"content\": \"Orange.\"},\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    ")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{\n",
+      "    \"id\": \"chatcmpl-8dee9DuEFcg2QILtT2a6EBXZnpirM\",\n",
+      "    \"choices\": [\n",
+      "        {\n",
+      "            \"finish_reason\": \"stop\",\n",
+      "            \"index\": 0,\n",
+      "            \"logprobs\": null,\n",
+      "            \"message\": {\n",
+      "                \"content\": \"Orange who?\",\n",
+      "                \"role\": \"assistant\",\n",
+      "                \"function_call\": null,\n",
+      "                \"tool_calls\": null\n",
+      "            }\n",
+      "        }\n",
+      "    ],\n",
+      "    \"created\": 1704461729,\n",
+      "    \"model\": \"gpt-3.5-turbo-0613\",\n",
+      "    \"object\": \"chat.completion\",\n",
+      "    \"system_fingerprint\": null,\n",
+      "    \"usage\": {\n",
+      "        \"completion_tokens\": 3,\n",
+      "        \"prompt_tokens\": 35,\n",
+      "        \"total_tokens\": 38\n",
+      "    }\n",
+      "}\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(json.dumps(json.loads(response.model_dump_json()), indent=4))"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "As you can see, the response object has a few fields:\n",
+    "- `id`: the ID of the request\n",
+    "- `choices`: a list of completion objects (only one, unless you set `n` greater than 1)\n",
+    "    - `finish_reason`: the reason the model stopped generating text (either `stop`, or `length` if `max_tokens` limit was reached)\n",
+    "    - `index`: The index of the choice in the list of choices.\n",
+    "    - `logprobs`: Log probability information for the choice.\n",
+    "    - `message`: the message object generated by the model\n",
+    "        - `content`: content of message\n",
+    "        - `role`: The role of the author of this message.\n",
+    "        - `tool_calls`: The tool calls generated by the model, such as function calls. if the tools is given\n",
+    "- `created`: the timestamp of the request\n",
+    "- `model`: the full name of the model used to generate the response\n",
+    "- `object`: the type of object returned (e.g., `chat.completion`)\n",
+    "- `system_fingerprint`: This fingerprint represents the backend configuration that the model runs with.\n",
+    "- `usage`: the number of tokens used to generate the replies, counting prompt, completion, and total"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Extract just the reply with:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'Orange who?'"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "response.choices[0].message.content\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Even non-conversation-based tasks can fit into the chat format, by placing the instruction in the first user message.\n",
+    "\n",
+    "For example, to ask the model to explain asynchronous programming in the style of the pirate Blackbeard, we can structure conversation as follows:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Arr, me matey! Let me tell ye a tale of asynchronous programming, in the style of the fearsome pirate Blackbeard!\n",
+      "\n",
+      "Picture this, me hearties. In the vast ocean of programming, there be times when ye need to perform multiple tasks at once. But fear not, for asynchronous programming be here to save the day!\n",
+      "\n",
+      "Ye see, in traditional programming, ye be waitin' for one task to be done before movin' on to the next. But with asynchronous programming, ye can be takin' care of multiple tasks at the same time, just like a pirate multitaskin' on the high seas!\n",
+      "\n",
+      "Instead of waitin' for a task to be completed, ye can be sendin' it off on its own journey, while ye move on to the next task. It be like havin' a crew of trusty sailors, each takin' care of their own duties, without waitin' for the others.\n",
+      "\n",
+      "Now, ye may be wonderin', how does this sorcery work? Well, me matey, it be all about callbacks and promises. When ye be sendin' off a task, ye be attachin' a callback function to it. This be like leavin' a message in a bottle, tellin' the task what to do when it be finished.\n",
+      "\n",
+      "While the task be sailin' on its own, ye can be movin' on to the next task, without wastin' any precious time. And when the first task be done, it be sendin' a signal back to ye, lettin' ye know it be finished. Then ye can be takin' care of the callback function, like openin' the bottle and readin' the message inside.\n",
+      "\n",
+      "But wait, there be more! With promises, ye can be makin' even fancier arrangements. Instead of callbacks, ye be makin' a promise that the task will be completed. It be like a contract between ye and the task, swearin' that it will be done.\n",
+      "\n",
+      "Ye can be attachin' multiple promises to a task, promisin' different outcomes. And when the task be finished, it be fulfillin' the promises, lettin' ye know it be done. Then ye can be handlin' the fulfillments, like collectin' the rewards of yer pirate adventures!\n",
+      "\n",
+      "So, me hearties, that be the tale of asynchronous programming, told in the style of the fearsome pirate Blackbeard! With callbacks and promises, ye can be takin' care of multiple tasks at once, just like a pirate conquerin' the seven seas!\n"
+     ]
+    }
+   ],
+   "source": [
+    "# example with a system message\n",
+    "response = client.chat.completions.create(\n",
+    "    model=MODEL,\n",
+    "    messages=[\n",
+    "        {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
+    "        {\"role\": \"user\", \"content\": \"Explain asynchronous programming in the style of the pirate Blackbeard.\"},\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    ")\n",
+    "\n",
+    "print(response.choices[0].message.content)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Arr, me hearties! Gather 'round and listen up, for I be tellin' ye about the mysterious art of asynchronous programming, in the style of the fearsome pirate Blackbeard!\n",
+      "\n",
+      "Now, ye see, in the world of programming, there be times when we need to perform tasks that take a mighty long time to complete. These tasks might involve fetchin' data from the depths of the internet, or performin' complex calculations that would make even Davy Jones scratch his head.\n",
+      "\n",
+      "In the olden days, we pirates used to wait patiently for each task to finish afore movin' on to the next one. But that be a waste of precious time, me hearties! We be pirates, always lookin' for ways to be more efficient and plunder more booty!\n",
+      "\n",
+      "That be where asynchronous programming comes in, me mateys. It be a way to tackle multiple tasks at once, without waitin' for each one to finish afore movin' on. It be like havin' a crew of scallywags workin' on different tasks simultaneously, while ye be overseein' the whole operation.\n",
+      "\n",
+      "Ye see, in asynchronous programming, we be breakin' down our tasks into smaller chunks called \"coroutines.\" Each coroutine be like a separate pirate, workin' on its own task. When a coroutine be startin' its work, it don't wait for the task to finish afore movin' on to the next one. Instead, it be movin' on to the next task, lettin' the first one continue in the background.\n",
+      "\n",
+      "Now, ye might be wonderin', \"But Blackbeard, how be we know when a task be finished if we don't wait for it?\" Ah, me hearties, that be where the magic of callbacks and promises come in!\n",
+      "\n",
+      "When a coroutine be startin' its work, it be attachin' a callback or a promise to it. This be like leavin' a message in a bottle, tellin' the coroutine what to do when it be finished. So, while the coroutine be workin' away, the rest of the crew be movin' on to other tasks, plunderin' more booty along the way.\n",
+      "\n",
+      "When a coroutine be finished with its task, it be sendin' a signal to the callback or fulfillin' the promise, lettin' the rest of the crew know that it be done. Then, the crew can gather 'round and handle the results of the completed task, celebratin' their victory and countin' their plunder.\n",
+      "\n",
+      "So, me hearties, asynchronous programming be like havin' a crew of pirates workin' on different tasks at once, without waitin' for each one to finish afore movin' on. It be a way to be more efficient, plunder more booty, and conquer the vast seas of programming!\n",
+      "\n",
+      "Now, set sail, me mateys, and embrace the power of asynchronous programming like true pirates of the digital realm! Arr!\n"
+     ]
+    }
+   ],
+   "source": [
+    "# example without a system message\n",
+    "response = client.chat.completions.create(\n",
+    "    model=MODEL,\n",
+    "    messages=[\n",
+    "        {\"role\": \"user\", \"content\": \"Explain asynchronous programming in the style of the pirate Blackbeard.\"},\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    ")\n",
+    "\n",
+    "print(response.choices[0].message.content)\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Tips for instructing gpt-3.5-turbo-0301\n",
+    "\n",
+    "Best practices for instructing models may change from model version to model version. The advice that follows applies to `gpt-3.5-turbo-0301` and may not apply to future models."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### System messages\n",
+    "\n",
+    "The system message can be used to prime the assistant with different personalities or behaviors.\n",
+    "\n",
+    "Be aware that `gpt-3.5-turbo-0301` does not generally pay as much attention to the system message as `gpt-4-0314` or `gpt-3.5-turbo-0613`. Therefore, for `gpt-3.5-turbo-0301`, we recommend placing important instructions in the user message instead. Some developers have found success in continually moving the system message near the end of the conversation to keep the model's attention from drifting away as conversations get longer."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Of course! Fractions are a way to represent parts of a whole. They are made up of two numbers: a numerator and a denominator. The numerator tells you how many parts you have, and the denominator tells you how many equal parts make up the whole.\n",
+      "\n",
+      "Let's take an example to understand this better. Imagine you have a pizza that is divided into 8 equal slices. If you eat 3 slices, you can represent that as the fraction 3/8. Here, the numerator is 3 because you ate 3 slices, and the denominator is 8 because the whole pizza is divided into 8 slices.\n",
+      "\n",
+      "Fractions can also be used to represent numbers less than 1. For example, if you eat half of a pizza, you can write it as 1/2. Here, the numerator is 1 because you ate one slice, and the denominator is 2 because the whole pizza is divided into 2 equal parts.\n",
+      "\n",
+      "Now, let's talk about equivalent fractions. Equivalent fractions are different fractions that represent the same amount. For example, 1/2 and 2/4 are equivalent fractions because they both represent half of something. To find equivalent fractions, you can multiply or divide both the numerator and denominator by the same number.\n",
+      "\n",
+      "Here's a question to check your understanding: If you have a cake divided into 12 equal slices and you eat 4 slices, what fraction of the cake did you eat?\n"
+     ]
+    }
+   ],
+   "source": [
+    "# An example of a system message that primes the assistant to explain concepts in great depth\n",
+    "response = client.chat.completions.create(\n",
+    "    model=MODEL,\n",
+    "    messages=[\n",
+    "        {\"role\": \"system\", \"content\": \"You are a friendly and helpful teaching assistant. You explain concepts in great depth using simple terms, and you give examples to help people learn. At the end of each explanation, you ask a question to check for understanding\"},\n",
+    "        {\"role\": \"user\", \"content\": \"Can you explain how fractions work?\"},\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    ")\n",
+    "\n",
+    "print(response.choices[0].message.content)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Fractions represent parts of a whole. They have a numerator (top number) and a denominator (bottom number).\n"
+     ]
+    }
+   ],
+   "source": [
+    "# An example of a system message that primes the assistant to give brief, to-the-point answers\n",
+    "response = client.chat.completions.create(\n",
+    "    model=MODEL,\n",
+    "    messages=[\n",
+    "        {\"role\": \"system\", \"content\": \"You are a laconic assistant. You reply with brief, to-the-point answers with no elaboration.\"},\n",
+    "        {\"role\": \"user\", \"content\": \"Can you explain how fractions work?\"},\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    ")\n",
+    "\n",
+    "print(response.choices[0].message.content)\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Few-shot prompting\n",
+    "\n",
+    "In some cases, it's easier to show the model what you want rather than tell the model what you want.\n",
+    "\n",
+    "One way to show the model what you want is with faked example messages.\n",
+    "\n",
+    "For example:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "This sudden change in direction means we don't have enough time to complete the entire project for the client.\n"
+     ]
+    }
+   ],
+   "source": [
+    "# An example of a faked few-shot conversation to prime the model into translating business jargon to simpler speech\n",
+    "response = client.chat.completions.create(\n",
+    "    model=MODEL,\n",
+    "    messages=[\n",
+    "        {\"role\": \"system\", \"content\": \"You are a helpful, pattern-following assistant.\"},\n",
+    "        {\"role\": \"user\", \"content\": \"Help me translate the following corporate jargon into plain English.\"},\n",
+    "        {\"role\": \"assistant\", \"content\": \"Sure, I'd be happy to!\"},\n",
+    "        {\"role\": \"user\", \"content\": \"New synergies will help drive top-line growth.\"},\n",
+    "        {\"role\": \"assistant\", \"content\": \"Things working well together will increase revenue.\"},\n",
+    "        {\"role\": \"user\", \"content\": \"Let's circle back when we have more bandwidth to touch base on opportunities for increased leverage.\"},\n",
+    "        {\"role\": \"assistant\", \"content\": \"Let's talk later when we're less busy about how to do better.\"},\n",
+    "        {\"role\": \"user\", \"content\": \"This late pivot means we don't have time to boil the ocean for the client deliverable.\"},\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    ")\n",
+    "\n",
+    "print(response.choices[0].message.content)\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To help clarify that the example messages are not part of a real conversation, and shouldn't be referred back to by the model, you can try setting the `name` field of `system` messages to `example_user` and `example_assistant`.\n",
+    "\n",
+    "Transforming the few-shot example above, we could write:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "This sudden change in direction means we don't have enough time to complete the entire project for the client.\n"
+     ]
+    }
+   ],
+   "source": [
+    "# The business jargon translation example, but with example names for the example messages\n",
+    "response = client.chat.completions.create(\n",
+    "    model=MODEL,\n",
+    "    messages=[\n",
+    "        {\"role\": \"system\", \"content\": \"You are a helpful, pattern-following assistant that translates corporate jargon into plain English.\"},\n",
+    "        {\"role\": \"system\", \"name\":\"example_user\", \"content\": \"New synergies will help drive top-line growth.\"},\n",
+    "        {\"role\": \"system\", \"name\": \"example_assistant\", \"content\": \"Things working well together will increase revenue.\"},\n",
+    "        {\"role\": \"system\", \"name\":\"example_user\", \"content\": \"Let's circle back when we have more bandwidth to touch base on opportunities for increased leverage.\"},\n",
+    "        {\"role\": \"system\", \"name\": \"example_assistant\", \"content\": \"Let's talk later when we're less busy about how to do better.\"},\n",
+    "        {\"role\": \"user\", \"content\": \"This late pivot means we don't have time to boil the ocean for the client deliverable.\"},\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    ")\n",
+    "\n",
+    "print(response.choices[0].message.content)\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Not every attempt at engineering conversations will succeed at first.\n",
+    "\n",
+    "If your first attempts fail, don't be afraid to experiment with different ways of priming or conditioning the model.\n",
+    "\n",
+    "As an example, one developer discovered an increase in accuracy when they inserted a user message that said \"Great job so far, these have been perfect\" to help condition the model into providing higher quality responses.\n",
+    "\n",
+    "For more ideas on how to lift the reliability of the models, consider reading our guide on [techniques to increase reliability](../techniques_to_improve_reliability). It was written for non-chat models, but many of its principles still apply."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4. Counting tokens\n",
+    "\n",
+    "When you submit your request, the API transforms the messages into a sequence of tokens.\n",
+    "\n",
+    "The number of tokens used affects:\n",
+    "- the cost of the request\n",
+    "- the time it takes to generate the response\n",
+    "- when the reply gets cut off from hitting the maximum token limit (4,096 for `gpt-3.5-turbo` or 8,192 for `gpt-4`)\n",
+    "\n",
+    "You can use the following function to count the number of tokens that a list of messages will use.\n",
+    "\n",
+    "Note that the exact way that tokens are counted from messages may change from model to model. Consider the counts from the function below an estimate, not a timeless guarantee. \n",
+    "\n",
+    "In particular, requests that use the optional functions input will consume extra tokens on top of the estimates calculated below.\n",
+    "\n",
+    "Read more about counting tokens in [How to count tokens with tiktoken](How_to_count_tokens_with_tiktoken.ipynb)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import tiktoken\n",
+    "\n",
+    "\n",
+    "def num_tokens_from_messages(messages, model=\"gpt-3.5-turbo-0613\"):\n",
+    "    \"\"\"Return the number of tokens used by a list of messages.\"\"\"\n",
+    "    try:\n",
+    "        encoding = tiktoken.encoding_for_model(model)\n",
+    "    except KeyError:\n",
+    "        print(\"Warning: model not found. Using cl100k_base encoding.\")\n",
+    "        encoding = tiktoken.get_encoding(\"cl100k_base\")\n",
+    "    if model in {\n",
+    "        \"gpt-3.5-turbo-0613\",\n",
+    "        \"gpt-3.5-turbo-16k-0613\",\n",
+    "        \"gpt-4-0314\",\n",
+    "        \"gpt-4-32k-0314\",\n",
+    "        \"gpt-4-0613\",\n",
+    "        \"gpt-4-32k-0613\",\n",
+    "        }:\n",
+    "        tokens_per_message = 3\n",
+    "        tokens_per_name = 1\n",
+    "    elif model == \"gpt-3.5-turbo-0301\":\n",
+    "        tokens_per_message = 4  # every message follows <|start|>{role/name}\\n{content}<|end|>\\n\n",
+    "        tokens_per_name = -1  # if there's a name, the role is omitted\n",
+    "    elif \"gpt-3.5-turbo\" in model:\n",
+    "        print(\"Warning: gpt-3.5-turbo may update over time. Returning num tokens assuming gpt-3.5-turbo-0613.\")\n",
+    "        return num_tokens_from_messages(messages, model=\"gpt-3.5-turbo-0613\")\n",
+    "    elif \"gpt-4\" in model:\n",
+    "        print(\"Warning: gpt-4 may update over time. Returning num tokens assuming gpt-4-0613.\")\n",
+    "        return num_tokens_from_messages(messages, model=\"gpt-4-0613\")\n",
+    "    else:\n",
+    "        raise NotImplementedError(\n",
+    "            f\"\"\"num_tokens_from_messages() is not implemented for model {model}.\"\"\"\n",
+    "        )\n",
+    "    num_tokens = 0\n",
+    "    for message in messages:\n",
+    "        num_tokens += tokens_per_message\n",
+    "        for key, value in message.items():\n",
+    "            num_tokens += len(encoding.encode(value))\n",
+    "            if key == \"name\":\n",
+    "                num_tokens += tokens_per_name\n",
+    "    num_tokens += 3  # every reply is primed with <|start|>assistant<|message|>\n",
+    "    return num_tokens\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "gpt-3.5-turbo-1106\n",
+      "Warning: gpt-3.5-turbo may update over time. Returning num tokens assuming gpt-3.5-turbo-0613.\n",
+      "129 prompt tokens counted by num_tokens_from_messages().\n",
+      "129 prompt tokens counted by the OpenAI API.\n",
+      "\n",
+      "gpt-3.5-turbo\n",
+      "Warning: gpt-3.5-turbo may update over time. Returning num tokens assuming gpt-3.5-turbo-0613.\n",
+      "129 prompt tokens counted by num_tokens_from_messages().\n",
+      "129 prompt tokens counted by the OpenAI API.\n",
+      "\n",
+      "gpt-4\n",
+      "Warning: gpt-4 may update over time. Returning num tokens assuming gpt-4-0613.\n",
+      "129 prompt tokens counted by num_tokens_from_messages().\n",
+      "129 prompt tokens counted by the OpenAI API.\n",
+      "\n",
+      "gpt-4-1106-preview\n",
+      "Warning: gpt-4 may update over time. Returning num tokens assuming gpt-4-0613.\n",
+      "129 prompt tokens counted by num_tokens_from_messages().\n",
+      "129 prompt tokens counted by the OpenAI API.\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "# let's verify the function above matches the OpenAI API response\n",
+    "example_messages = [\n",
+    "    {\n",
+    "        \"role\": \"system\",\n",
+    "        \"content\": \"You are a helpful, pattern-following assistant that translates corporate jargon into plain English.\",\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"system\",\n",
+    "        \"name\": \"example_user\",\n",
+    "        \"content\": \"New synergies will help drive top-line growth.\",\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"system\",\n",
+    "        \"name\": \"example_assistant\",\n",
+    "        \"content\": \"Things working well together will increase revenue.\",\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"system\",\n",
+    "        \"name\": \"example_user\",\n",
+    "        \"content\": \"Let's circle back when we have more bandwidth to touch base on opportunities for increased leverage.\",\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"system\",\n",
+    "        \"name\": \"example_assistant\",\n",
+    "        \"content\": \"Let's talk later when we're less busy about how to do better.\",\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"user\",\n",
+    "        \"content\": \"This late pivot means we don't have time to boil the ocean for the client deliverable.\",\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "for model in [\n",
+    "    # \"gpt-3.5-turbo-0301\",\n",
+    "    # \"gpt-4-0314\",\n",
+    "    # \"gpt-4-0613\",\n",
+    "    \"gpt-3.5-turbo-1106\",\n",
+    "    \"gpt-3.5-turbo\",\n",
+    "    \"gpt-4\",\n",
+    "    \"gpt-4-1106-preview\",\n",
+    "    ]:\n",
+    "    print(model)\n",
+    "    # example token count from the function defined above\n",
+    "    print(f\"{num_tokens_from_messages(example_messages, model)} prompt tokens counted by num_tokens_from_messages().\")\n",
+    "    # example token count from the OpenAI API\n",
+    "    response = client.chat.completions.create(model=model,\n",
+    "    messages=example_messages,\n",
+    "    temperature=0,\n",
+    "    max_tokens=1)\n",
+    "    token = response.usage.prompt_tokens\n",
+    "    print(f'{token} prompt tokens counted by the OpenAI API.')\n",
+    "    print()\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "openai",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.5"
+  },
+  "orig_nbformat": 4,
+  "vscode": {
+   "interpreter": {
+    "hash": "365536dcbde60510dc9073d6b991cd35db2d9bac356a11f5b64279a5e6708b97"
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/How_to_stream_completions.ipynb

@@ -0,0 +1,661 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# How to stream completions\n",
+    "\n",
+    "By default, when you request a completion from the OpenAI, the entire completion is generated before being sent back in a single response.\n",
+    "\n",
+    "If you're generating long completions, waiting for the response can take many seconds.\n",
+    "\n",
+    "To get responses sooner, you can 'stream' the completion as it's being generated. This allows you to start printing or processing the beginning of the completion before the full completion is finished.\n",
+    "\n",
+    "To stream completions, set `stream=True` when calling the chat completions or completions endpoints. This will return an object that streams back the response as [data-only server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format). Extract chunks from the `delta` field rather than the `message` field.\n",
+    "\n",
+    "## Downsides\n",
+    "\n",
+    "Note that using `stream=True` in a production application makes it more difficult to moderate the content of the completions, as partial completions may be more difficult to evaluate. This may have implications for [approved usage](https://beta.openai.com/docs/usage-guidelines).\n",
+    "\n",
+    "## Example code\n",
+    "\n",
+    "Below, this notebook shows:\n",
+    "1. What a typical chat completion response looks like\n",
+    "2. What a streaming chat completion response looks like\n",
+    "3. How much time is saved by streaming a chat completion\n",
+    "4. How to get token usage data for streamed chat completion response"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# !pip install openai"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# imports\n",
+    "import time  # for measuring time duration of API calls\n",
+    "from openai import OpenAI\n",
+    "import os\n",
+    "client = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 1. What a typical chat completion response looks like\n",
+    "\n",
+    "With a typical ChatCompletions API call, the response is first computed and then returned all at once."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Full response received 1.88 seconds after request\n",
+      "Full response received:\n",
+      "ChatCompletion(id='chatcmpl-9lMgdoiMfxVHPDNVCtvXuTWcQ2GGb', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100', role='assistant', function_call=None, tool_calls=None))], created=1721075651, model='gpt-july-test', object='chat.completion', system_fingerprint='fp_e9b8ed65d2', usage=CompletionUsage(completion_tokens=298, prompt_tokens=36, total_tokens=334))\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Example of an OpenAI ChatCompletion request\n",
+    "# https://platform.openai.com/docs/guides/text-generation/chat-completions-api\n",
+    "\n",
+    "# record the time before the request is sent\n",
+    "start_time = time.time()\n",
+    "\n",
+    "# send a ChatCompletion request to count to 100\n",
+    "response = client.chat.completions.create(\n",
+    "    model='gpt-4o-mini',\n",
+    "    messages=[\n",
+    "        {'role': 'user', 'content': 'Count to 100, with a comma between each number and no newlines. E.g., 1, 2, 3, ...'}\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    ")\n",
+    "# calculate the time it took to receive the response\n",
+    "response_time = time.time() - start_time\n",
+    "\n",
+    "# print the time delay and text received\n",
+    "print(f\"Full response received {response_time:.2f} seconds after request\")\n",
+    "print(f\"Full response received:\\n{response}\")\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The reply can be extracted with `response.choices[0].message`.\n",
+    "\n",
+    "The content of the reply can be extracted with `response.choices[0].message.content`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Extracted reply: \n",
+      "ChatCompletionMessage(content='1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100', role='assistant', function_call=None, tool_calls=None)\n",
+      "Extracted content: \n",
+      "1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100\n"
+     ]
+    }
+   ],
+   "source": [
+    "reply = response.choices[0].message\n",
+    "print(f\"Extracted reply: \\n{reply}\")\n",
+    "\n",
+    "reply_content = response.choices[0].message.content\n",
+    "print(f\"Extracted content: \\n{reply_content}\")\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 2. How to stream a chat completion\n",
+    "\n",
+    "With a streaming API call, the response is sent back incrementally in chunks via an [event stream](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format). In Python, you can iterate over these events with a `for` loop.\n",
+    "\n",
+    "Let's see what it looks like:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "ChatCompletionChunk(id='chatcmpl-9lMgfRSWPHcw51s6wxKT1YEO2CKpd', choices=[Choice(delta=ChoiceDelta(content='', function_call=None, role='assistant', tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1721075653, model='gpt-july-test', object='chat.completion.chunk', system_fingerprint='fp_e9b8ed65d2', usage=None)\n",
+      "\n",
+      "****************\n",
+      "ChatCompletionChunk(id='chatcmpl-9lMgfRSWPHcw51s6wxKT1YEO2CKpd', choices=[Choice(delta=ChoiceDelta(content='Two', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1721075653, model='gpt-july-test', object='chat.completion.chunk', system_fingerprint='fp_e9b8ed65d2', usage=None)\n",
+      "Two\n",
+      "****************\n",
+      "ChatCompletionChunk(id='chatcmpl-9lMgfRSWPHcw51s6wxKT1YEO2CKpd', choices=[Choice(delta=ChoiceDelta(content='.', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1721075653, model='gpt-july-test', object='chat.completion.chunk', system_fingerprint='fp_e9b8ed65d2', usage=None)\n",
+      ".\n",
+      "****************\n",
+      "ChatCompletionChunk(id='chatcmpl-9lMgfRSWPHcw51s6wxKT1YEO2CKpd', choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, role=None, tool_calls=None), finish_reason='stop', index=0, logprobs=None)], created=1721075653, model='gpt-july-test', object='chat.completion.chunk', system_fingerprint='fp_e9b8ed65d2', usage=None)\n",
+      "None\n",
+      "****************\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Example of an OpenAI ChatCompletion request with stream=True\n",
+    "# https://platform.openai.com/docs/api-reference/streaming#chat/create-stream\n",
+    "\n",
+    "# a ChatCompletion request\n",
+    "response = client.chat.completions.create(\n",
+    "    model='gpt-4o-mini',\n",
+    "    messages=[\n",
+    "        {'role': 'user', 'content': \"What's 1+1? Answer in one word.\"}\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    "    stream=True  # this time, we set stream=True\n",
+    ")\n",
+    "\n",
+    "for chunk in response:\n",
+    "    print(chunk)\n",
+    "    print(chunk.choices[0].delta.content)\n",
+    "    print(\"****************\")"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "As you can see above, streaming responses have a `delta` field rather than a `message` field. `delta` can hold things like:\n",
+    "- a role token (e.g., `{\"role\": \"assistant\"}`)\n",
+    "- a content token (e.g., `{\"content\": \"\\n\\n\"}`)\n",
+    "- nothing (e.g., `{}`), when the stream is over"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 3. How much time is saved by streaming a chat completion\n",
+    "\n",
+    "Now let's ask `gpt-4o-mini` to count to 100 again, and see how long it takes."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Message received 1.14 seconds after request: \n",
+      "Message received 1.14 seconds after request: 1\n",
+      "Message received 1.14 seconds after request: ,\n",
+      "Message received 1.14 seconds after request:  \n",
+      "Message received 1.14 seconds after request: 2\n",
+      "Message received 1.16 seconds after request: ,\n",
+      "Message received 1.16 seconds after request:  \n",
+      "Message received 1.16 seconds after request: 3\n",
+      "Message received 1.35 seconds after request: ,\n",
+      "Message received 1.35 seconds after request:  \n",
+      "Message received 1.35 seconds after request: 4\n",
+      "Message received 1.36 seconds after request: ,\n",
+      "Message received 1.36 seconds after request:  \n",
+      "Message received 1.36 seconds after request: 5\n",
+      "Message received 1.36 seconds after request: ,\n",
+      "Message received 1.36 seconds after request:  \n",
+      "Message received 1.36 seconds after request: 6\n",
+      "Message received 1.36 seconds after request: ,\n",
+      "Message received 1.36 seconds after request:  \n",
+      "Message received 1.36 seconds after request: 7\n",
+      "Message received 1.36 seconds after request: ,\n",
+      "Message received 1.36 seconds after request:  \n",
+      "Message received 1.36 seconds after request: 8\n",
+      "Message received 1.36 seconds after request: ,\n",
+      "Message received 1.36 seconds after request:  \n",
+      "Message received 1.36 seconds after request: 9\n",
+      "Message received 1.36 seconds after request: ,\n",
+      "Message received 1.36 seconds after request:  \n",
+      "Message received 1.36 seconds after request: 10\n",
+      "Message received 1.36 seconds after request: ,\n",
+      "Message received 1.36 seconds after request:  \n",
+      "Message received 1.36 seconds after request: 11\n",
+      "Message received 1.36 seconds after request: ,\n",
+      "Message received 1.36 seconds after request:  \n",
+      "Message received 1.36 seconds after request: 12\n",
+      "Message received 1.36 seconds after request: ,\n",
+      "Message received 1.36 seconds after request:  \n",
+      "Message received 1.45 seconds after request: 13\n",
+      "Message received 1.45 seconds after request: ,\n",
+      "Message received 1.45 seconds after request:  \n",
+      "Message received 1.45 seconds after request: 14\n",
+      "Message received 1.45 seconds after request: ,\n",
+      "Message received 1.45 seconds after request:  \n",
+      "Message received 1.45 seconds after request: 15\n",
+      "Message received 1.45 seconds after request: ,\n",
+      "Message received 1.45 seconds after request:  \n",
+      "Message received 1.46 seconds after request: 16\n",
+      "Message received 1.46 seconds after request: ,\n",
+      "Message received 1.46 seconds after request:  \n",
+      "Message received 1.47 seconds after request: 17\n",
+      "Message received 1.47 seconds after request: ,\n",
+      "Message received 1.47 seconds after request:  \n",
+      "Message received 1.49 seconds after request: 18\n",
+      "Message received 1.49 seconds after request: ,\n",
+      "Message received 1.49 seconds after request:  \n",
+      "Message received 1.52 seconds after request: 19\n",
+      "Message received 1.52 seconds after request: ,\n",
+      "Message received 1.52 seconds after request:  \n",
+      "Message received 1.53 seconds after request: 20\n",
+      "Message received 1.53 seconds after request: ,\n",
+      "Message received 1.53 seconds after request:  \n",
+      "Message received 1.55 seconds after request: 21\n",
+      "Message received 1.55 seconds after request: ,\n",
+      "Message received 1.55 seconds after request:  \n",
+      "Message received 1.56 seconds after request: 22\n",
+      "Message received 1.56 seconds after request: ,\n",
+      "Message received 1.56 seconds after request:  \n",
+      "Message received 1.58 seconds after request: 23\n",
+      "Message received 1.58 seconds after request: ,\n",
+      "Message received 1.58 seconds after request:  \n",
+      "Message received 1.59 seconds after request: 24\n",
+      "Message received 1.59 seconds after request: ,\n",
+      "Message received 1.59 seconds after request:  \n",
+      "Message received 1.62 seconds after request: 25\n",
+      "Message received 1.62 seconds after request: ,\n",
+      "Message received 1.62 seconds after request:  \n",
+      "Message received 1.62 seconds after request: 26\n",
+      "Message received 1.62 seconds after request: ,\n",
+      "Message received 1.62 seconds after request:  \n",
+      "Message received 1.65 seconds after request: 27\n",
+      "Message received 1.65 seconds after request: ,\n",
+      "Message received 1.65 seconds after request:  \n",
+      "Message received 1.67 seconds after request: 28\n",
+      "Message received 1.67 seconds after request: ,\n",
+      "Message received 1.67 seconds after request:  \n",
+      "Message received 1.69 seconds after request: 29\n",
+      "Message received 1.69 seconds after request: ,\n",
+      "Message received 1.69 seconds after request:  \n",
+      "Message received 1.80 seconds after request: 30\n",
+      "Message received 1.80 seconds after request: ,\n",
+      "Message received 1.80 seconds after request:  \n",
+      "Message received 1.80 seconds after request: 31\n",
+      "Message received 1.80 seconds after request: ,\n",
+      "Message received 1.80 seconds after request:  \n",
+      "Message received 1.80 seconds after request: 32\n",
+      "Message received 1.80 seconds after request: ,\n",
+      "Message received 1.80 seconds after request:  \n",
+      "Message received 1.80 seconds after request: 33\n",
+      "Message received 1.80 seconds after request: ,\n",
+      "Message received 1.80 seconds after request:  \n",
+      "Message received 1.80 seconds after request: 34\n",
+      "Message received 1.80 seconds after request: ,\n",
+      "Message received 1.80 seconds after request:  \n",
+      "Message received 1.80 seconds after request: 35\n",
+      "Message received 1.80 seconds after request: ,\n",
+      "Message received 1.80 seconds after request:  \n",
+      "Message received 1.80 seconds after request: 36\n",
+      "Message received 1.80 seconds after request: ,\n",
+      "Message received 1.80 seconds after request:  \n",
+      "Message received 1.82 seconds after request: 37\n",
+      "Message received 1.82 seconds after request: ,\n",
+      "Message received 1.82 seconds after request:  \n",
+      "Message received 1.83 seconds after request: 38\n",
+      "Message received 1.83 seconds after request: ,\n",
+      "Message received 1.83 seconds after request:  \n",
+      "Message received 1.84 seconds after request: 39\n",
+      "Message received 1.84 seconds after request: ,\n",
+      "Message received 1.84 seconds after request:  \n",
+      "Message received 1.87 seconds after request: 40\n",
+      "Message received 1.87 seconds after request: ,\n",
+      "Message received 1.87 seconds after request:  \n",
+      "Message received 1.88 seconds after request: 41\n",
+      "Message received 1.88 seconds after request: ,\n",
+      "Message received 1.88 seconds after request:  \n",
+      "Message received 1.91 seconds after request: 42\n",
+      "Message received 1.91 seconds after request: ,\n",
+      "Message received 1.91 seconds after request:  \n",
+      "Message received 1.93 seconds after request: 43\n",
+      "Message received 1.93 seconds after request: ,\n",
+      "Message received 1.93 seconds after request:  \n",
+      "Message received 1.93 seconds after request: 44\n",
+      "Message received 1.93 seconds after request: ,\n",
+      "Message received 1.93 seconds after request:  \n",
+      "Message received 1.95 seconds after request: 45\n",
+      "Message received 1.95 seconds after request: ,\n",
+      "Message received 1.95 seconds after request:  \n",
+      "Message received 2.00 seconds after request: 46\n",
+      "Message received 2.00 seconds after request: ,\n",
+      "Message received 2.00 seconds after request:  \n",
+      "Message received 2.00 seconds after request: 47\n",
+      "Message received 2.00 seconds after request: ,\n",
+      "Message received 2.00 seconds after request:  \n",
+      "Message received 2.00 seconds after request: 48\n",
+      "Message received 2.00 seconds after request: ,\n",
+      "Message received 2.00 seconds after request:  \n",
+      "Message received 2.00 seconds after request: 49\n",
+      "Message received 2.00 seconds after request: ,\n",
+      "Message received 2.00 seconds after request:  \n",
+      "Message received 2.00 seconds after request: 50\n",
+      "Message received 2.00 seconds after request: ,\n",
+      "Message received 2.00 seconds after request:  \n",
+      "Message received 2.00 seconds after request: 51\n",
+      "Message received 2.00 seconds after request: ,\n",
+      "Message received 2.04 seconds after request:  \n",
+      "Message received 2.04 seconds after request: 52\n",
+      "Message received 2.04 seconds after request: ,\n",
+      "Message received 2.04 seconds after request:  \n",
+      "Message received 2.04 seconds after request: 53\n",
+      "Message received 2.04 seconds after request: ,\n",
+      "Message received 2.13 seconds after request:  \n",
+      "Message received 2.13 seconds after request: 54\n",
+      "Message received 2.14 seconds after request: ,\n",
+      "Message received 2.14 seconds after request:  \n",
+      "Message received 2.14 seconds after request: 55\n",
+      "Message received 2.14 seconds after request: ,\n",
+      "Message received 2.14 seconds after request:  \n",
+      "Message received 2.14 seconds after request: 56\n",
+      "Message received 2.14 seconds after request: ,\n",
+      "Message received 2.14 seconds after request:  \n",
+      "Message received 2.16 seconds after request: 57\n",
+      "Message received 2.16 seconds after request: ,\n",
+      "Message received 2.16 seconds after request:  \n",
+      "Message received 2.17 seconds after request: 58\n",
+      "Message received 2.17 seconds after request: ,\n",
+      "Message received 2.17 seconds after request:  \n",
+      "Message received 2.19 seconds after request: 59\n",
+      "Message received 2.19 seconds after request: ,\n",
+      "Message received 2.19 seconds after request:  \n",
+      "Message received 2.21 seconds after request: 60\n",
+      "Message received 2.21 seconds after request: ,\n",
+      "Message received 2.21 seconds after request:  \n",
+      "Message received 2.34 seconds after request: 61\n",
+      "Message received 2.34 seconds after request: ,\n",
+      "Message received 2.34 seconds after request:  \n",
+      "Message received 2.34 seconds after request: 62\n",
+      "Message received 2.34 seconds after request: ,\n",
+      "Message received 2.34 seconds after request:  \n",
+      "Message received 2.34 seconds after request: 63\n",
+      "Message received 2.34 seconds after request: ,\n",
+      "Message received 2.34 seconds after request:  \n",
+      "Message received 2.34 seconds after request: 64\n",
+      "Message received 2.34 seconds after request: ,\n",
+      "Message received 2.34 seconds after request:  \n",
+      "Message received 2.34 seconds after request: 65\n",
+      "Message received 2.34 seconds after request: ,\n",
+      "Message received 2.34 seconds after request:  \n",
+      "Message received 2.34 seconds after request: 66\n",
+      "Message received 2.34 seconds after request: ,\n",
+      "Message received 2.34 seconds after request:  \n",
+      "Message received 2.34 seconds after request: 67\n",
+      "Message received 2.34 seconds after request: ,\n",
+      "Message received 2.34 seconds after request:  \n",
+      "Message received 2.36 seconds after request: 68\n",
+      "Message received 2.36 seconds after request: ,\n",
+      "Message received 2.36 seconds after request:  \n",
+      "Message received 2.36 seconds after request: 69\n",
+      "Message received 2.36 seconds after request: ,\n",
+      "Message received 2.36 seconds after request:  \n",
+      "Message received 2.38 seconds after request: 70\n",
+      "Message received 2.38 seconds after request: ,\n",
+      "Message received 2.38 seconds after request:  \n",
+      "Message received 2.39 seconds after request: 71\n",
+      "Message received 2.39 seconds after request: ,\n",
+      "Message received 2.39 seconds after request:  \n",
+      "Message received 2.39 seconds after request: 72\n",
+      "Message received 2.39 seconds after request: ,\n",
+      "Message received 2.39 seconds after request:  \n",
+      "Message received 2.39 seconds after request: 73\n",
+      "Message received 2.39 seconds after request: ,\n",
+      "Message received 2.39 seconds after request:  \n",
+      "Message received 2.39 seconds after request: 74\n",
+      "Message received 2.39 seconds after request: ,\n",
+      "Message received 2.39 seconds after request:  \n",
+      "Message received 2.39 seconds after request: 75\n",
+      "Message received 2.39 seconds after request: ,\n",
+      "Message received 2.40 seconds after request:  \n",
+      "Message received 2.40 seconds after request: 76\n",
+      "Message received 2.40 seconds after request: ,\n",
+      "Message received 2.42 seconds after request:  \n",
+      "Message received 2.42 seconds after request: 77\n",
+      "Message received 2.42 seconds after request: ,\n",
+      "Message received 2.51 seconds after request:  \n",
+      "Message received 2.51 seconds after request: 78\n",
+      "Message received 2.51 seconds after request: ,\n",
+      "Message received 2.52 seconds after request:  \n",
+      "Message received 2.52 seconds after request: 79\n",
+      "Message received 2.52 seconds after request: ,\n",
+      "Message received 2.52 seconds after request:  \n",
+      "Message received 2.52 seconds after request: 80\n",
+      "Message received 2.52 seconds after request: ,\n",
+      "Message received 2.52 seconds after request:  \n",
+      "Message received 2.52 seconds after request: 81\n",
+      "Message received 2.52 seconds after request: ,\n",
+      "Message received 2.52 seconds after request:  \n",
+      "Message received 2.52 seconds after request: 82\n",
+      "Message received 2.52 seconds after request: ,\n",
+      "Message received 2.60 seconds after request:  \n",
+      "Message received 2.60 seconds after request: 83\n",
+      "Message received 2.60 seconds after request: ,\n",
+      "Message received 2.64 seconds after request:  \n",
+      "Message received 2.64 seconds after request: 84\n",
+      "Message received 2.64 seconds after request: ,\n",
+      "Message received 2.64 seconds after request:  \n",
+      "Message received 2.64 seconds after request: 85\n",
+      "Message received 2.64 seconds after request: ,\n",
+      "Message received 2.64 seconds after request:  \n",
+      "Message received 2.66 seconds after request: 86\n",
+      "Message received 2.66 seconds after request: ,\n",
+      "Message received 2.66 seconds after request:  \n",
+      "Message received 2.66 seconds after request: 87\n",
+      "Message received 2.66 seconds after request: ,\n",
+      "Message received 2.66 seconds after request:  \n",
+      "Message received 2.68 seconds after request: 88\n",
+      "Message received 2.68 seconds after request: ,\n",
+      "Message received 2.68 seconds after request:  \n",
+      "Message received 2.69 seconds after request: 89\n",
+      "Message received 2.69 seconds after request: ,\n",
+      "Message received 2.69 seconds after request:  \n",
+      "Message received 2.72 seconds after request: 90\n",
+      "Message received 2.72 seconds after request: ,\n",
+      "Message received 2.72 seconds after request:  \n",
+      "Message received 2.82 seconds after request: 91\n",
+      "Message received 2.82 seconds after request: ,\n",
+      "Message received 2.82 seconds after request:  \n",
+      "Message received 2.82 seconds after request: 92\n",
+      "Message received 2.82 seconds after request: ,\n",
+      "Message received 2.82 seconds after request:  \n",
+      "Message received 2.82 seconds after request: 93\n",
+      "Message received 2.82 seconds after request: ,\n",
+      "Message received 2.82 seconds after request:  \n",
+      "Message received 2.82 seconds after request: 94\n",
+      "Message received 2.82 seconds after request: ,\n",
+      "Message received 2.82 seconds after request:  \n",
+      "Message received 2.82 seconds after request: 95\n",
+      "Message received 2.82 seconds after request: ,\n",
+      "Message received 2.82 seconds after request:  \n",
+      "Message received 2.82 seconds after request: 96\n",
+      "Message received 2.82 seconds after request: ,\n",
+      "Message received 2.82 seconds after request:  \n",
+      "Message received 2.82 seconds after request: 97\n",
+      "Message received 2.82 seconds after request: ,\n",
+      "Message received 2.82 seconds after request:  \n",
+      "Message received 2.82 seconds after request: 98\n",
+      "Message received 2.82 seconds after request: ,\n",
+      "Message received 2.82 seconds after request:  \n",
+      "Message received 2.82 seconds after request: 99\n",
+      "Message received 2.82 seconds after request: ,\n",
+      "Message received 2.82 seconds after request:  \n",
+      "Message received 2.82 seconds after request: 100\n",
+      "Message received 2.82 seconds after request: None\n",
+      "Full response received 2.82 seconds after request\n",
+      "Full conversation received: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Example of an OpenAI ChatCompletion request with stream=True\n",
+    "# https://platform.openai.com/docs/api-reference/streaming#chat/create-stream\n",
+    "\n",
+    "# record the time before the request is sent\n",
+    "start_time = time.time()\n",
+    "\n",
+    "# send a ChatCompletion request to count to 100\n",
+    "response = client.chat.completions.create(\n",
+    "    model='gpt-4o-mini',\n",
+    "    messages=[\n",
+    "        {'role': 'user', 'content': 'Count to 100, with a comma between each number and no newlines. E.g., 1, 2, 3, ...'}\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    "    stream=True  # again, we set stream=True\n",
+    ")\n",
+    "# create variables to collect the stream of chunks\n",
+    "collected_chunks = []\n",
+    "collected_messages = []\n",
+    "# iterate through the stream of events\n",
+    "for chunk in response:\n",
+    "    chunk_time = time.time() - start_time  # calculate the time delay of the chunk\n",
+    "    collected_chunks.append(chunk)  # save the event response\n",
+    "    chunk_message = chunk.choices[0].delta.content  # extract the message\n",
+    "    collected_messages.append(chunk_message)  # save the message\n",
+    "    print(f\"Message received {chunk_time:.2f} seconds after request: {chunk_message}\")  # print the delay and text\n",
+    "\n",
+    "# print the time delay and text received\n",
+    "print(f\"Full response received {chunk_time:.2f} seconds after request\")\n",
+    "# clean None in collected_messages\n",
+    "collected_messages = [m for m in collected_messages if m is not None]\n",
+    "full_reply_content = ''.join(collected_messages)\n",
+    "print(f\"Full conversation received: {full_reply_content}\")\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Time comparison\n",
+    "\n",
+    "In the example above, both requests took about 4 to 5 seconds to fully complete. Request times will vary depending on load and other stochastic factors.\n",
+    "\n",
+    "However, with the streaming request, we received the first token after 0.1 seconds, and subsequent tokens every ~0.01-0.02 seconds."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 4. How to get token usage data for streamed chat completion response\n",
+    "\n",
+    "You can get token usage statistics for your streamed response by setting `stream_options={\"include_usage\": True}`. When you do so, an extra chunk will be streamed as the final chunk. You can access the usage data for the entire request via the `usage` field on this chunk. A few important notes when you set `stream_options={\"include_usage\": True}`:\n",
+    "* The value for the `usage` field on all chunks except for the last one will be null.\n",
+    "* The `usage` field on the last chunk contains token usage statistics for the entire request.\n",
+    "* The `choices` field on the last chunk will always be an empty array `[]`.\n",
+    "\n",
+    "Let's see how it works using the example in 2."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "choices: [Choice(delta=ChoiceDelta(content='', function_call=None, role='assistant', tool_calls=None), finish_reason=None, index=0, logprobs=None)]\n",
+      "usage: None\n",
+      "****************\n",
+      "choices: [Choice(delta=ChoiceDelta(content='Two', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)]\n",
+      "usage: None\n",
+      "****************\n",
+      "choices: [Choice(delta=ChoiceDelta(content='.', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)]\n",
+      "usage: None\n",
+      "****************\n",
+      "choices: [Choice(delta=ChoiceDelta(content=None, function_call=None, role=None, tool_calls=None), finish_reason='stop', index=0, logprobs=None)]\n",
+      "usage: None\n",
+      "****************\n",
+      "choices: []\n",
+      "usage: CompletionUsage(completion_tokens=2, prompt_tokens=18, total_tokens=20)\n",
+      "****************\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Example of an OpenAI ChatCompletion request with stream=True and stream_options={\"include_usage\": True}\n",
+    "\n",
+    "# a ChatCompletion request\n",
+    "response = client.chat.completions.create(\n",
+    "    model='gpt-4o-mini',\n",
+    "    messages=[\n",
+    "        {'role': 'user', 'content': \"What's 1+1? Answer in one word.\"}\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    "    stream=True,\n",
+    "    stream_options={\"include_usage\": True}, # retrieving token usage for stream response\n",
+    ")\n",
+    "\n",
+    "for chunk in response:\n",
+    "    print(f\"choices: {chunk.choices}\\nusage: {chunk.usage}\")\n",
+    "    print(\"****************\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3.9.9 ('openai')",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.8"
+  },
+  "orig_nbformat": 4,
+  "vscode": {
+   "interpreter": {
+    "hash": "365536dcbde60510dc9073d6b991cd35db2d9bac356a11f5b64279a5e6708b97"
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/How_to_use_guardrails.ipynb

@@ -0,0 +1,467 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "8ea66173",
+   "metadata": {},
+   "source": [
+    "# How to use guardrails\n",
+    "\n",
+    "In this notebook we share examples of how to implement guardrails for your LLM applications. A guardrail is a generic term for **detective controls** that aim to steer your application. Greater steerability is a common requirement given the inherent randomness of LLMs, and so creating effective guardrails has become one of the most common areas of performance optimization when pushing an LLM from prototype to production. \n",
+    "\n",
+    "Guardrails are incredibly [diverse](https://github.com/NVIDIA/NeMo-Guardrails/blob/main/examples/README.md) and can be deployed to virtually any context you can imagine something going wrong with LLMs. This notebook aims to give simple examples that can be extended to meet your unique use case, as well as outlining the trade-offs to consider when deciding whether to implement a guardrail, and how to do it.\n",
+    "\n",
+    "This notebook will focus on:\n",
+    "1. **Input guardrails** that flag inappropriate content before it gets to your LLM\n",
+    "2. **Output guardrails** that validate what your LLM has produced before it gets to the customer\n",
+    "\n",
+    "**Note:** This notebook tackles guardrails as a generic term for detective controls around an LLM - for the official libraries that provide distributions of pre-built guardrails frameworks, please check out the following:\n",
+    "- [NeMo Guardrails](https://github.com/NVIDIA/NeMo-Guardrails/tree/main)\n",
+    "- [Guardrails AI](https://github.com/ShreyaR/guardrails)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "id": "ef059e71",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import openai\n",
+    "\n",
+    "GPT_MODEL = 'gpt-4o-mini'"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "63d917f0",
+   "metadata": {},
+   "source": [
+    "## 1. Input guardrails\n",
+    "\n",
+    "Input guardrails aim to prevent inappropriate content getting to the LLM in the first place - some common use cases are:\n",
+    "- **Topical guardrails:** Identify when a user asks an off-topic question and give them advice on what topics the LLM can help them with.\n",
+    "- **Jailbreaking:** Detect when a user is trying to hijack the LLM and override its prompting.\n",
+    "- **Prompt injection:** Pick up instances of prompt injection where users try to hide malicious code that will be executed in any downstream functions the LLM executes. \n",
+    "\n",
+    "In all of these they act as a preventative control, running either before or in parallel with the LLM, and triggering your application to behave differently if one of these criteria are met.\n",
+    "\n",
+    "### Designing a guardrail\n",
+    "\n",
+    "When designing guardrails it is important to consider the trade-off between **accuracy**, **latency** and **cost**, where you try to achieve maximum accuracy for the least impact to your bottom line and the user's experience. \n",
+    "\n",
+    "We'll begin with a simple **topical guardrail** which aims to detect off-topic questions and prevent the LLM from answering if triggered. This guardrail consists of a simple prompt and uses `gpt-4o-mini`, maximising latency/cost holding a good enough accuracy, but if we wanted to optimize further we could consider:\n",
+    "- **Accuracy:** You could consider fine-tuning `gpt-4o-mini` or few-shot examples to increase the accuracy. RAG can also be effective if you have a corpus of information that can help determine whether a piece of content is allowed or not.\n",
+    "- **Latency/Cost:** You could try fine-tuning smaller models, such as `babbage-002` or open-source offerings like Llama, which can perform quite well when given enough training examples. When using open-source offerings you can also tune the machines you are using for inference to maximize either cost or latency reduction.\n",
+    "\n",
+    "This simple guardrail aims to ensure the LLM only answers to a predefined set of topics, and responds to out-of-bounds queries with a canned message.\n",
+    "\n",
+    "### Embrace async\n",
+    "\n",
+    "A common design to minimize latency is to send your guardrails asynchronously along with your main LLM call. If your guardrails get triggered you send back their response, otherwise send back the LLM response.\n",
+    "\n",
+    "We'll use this approach, creating an `execute_chat_with_guardrails` function that will run our LLM's `get_chat_response` and the `topical_guardrail` guardrail in parallel, and return the LLM response only if the guardrail returns `allowed`.\n",
+    "\n",
+    "### Limitations\n",
+    "\n",
+    "You should always consider the limitations of guardrails when developing your design. A few of the key ones to be aware of are:\n",
+    "- When using LLMs as a guardrail, be aware that they have the same vulnerabilities as your base LLM call itself. For example, a **prompt injection** attempt could be successful in evading both your guardrail and your actual LLM call.\n",
+    "- As conversations get longer, LLMs are more susceptible to **jailbreaking** as your instructions become diluted by the extra text.\n",
+    "- Guardrails can harm the user experience if you make them overly restrictive to compensate for the issues noted above. This manifests as **over-refusals**, where your guardrails reject innocuous user requests because there are similarities with prompt injection or jailbreaking attempts.\n",
+    "\n",
+    "### Mitigations\n",
+    "\n",
+    "If you can combine guardrails with rules-based or more traditional machine learning models for detection this can mitigate some of these risks. We've also seen customers have guardrails that only ever consider the latest message, to alleviate the risks of the model being confused by a long conversation.\n",
+    "\n",
+    "We would also recommend doing a gradual roll-out with active monitoring of conversations so you can pick up instances of prompt injection or jailbreaking, and either add more guardrails to cover these new types of behaviour, or include them as training examples to your existing guardrails."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "id": "e95efc89",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "system_prompt = \"You are a helpful assistant.\"\n",
+    "\n",
+    "bad_request = \"I want to talk about horses\"\n",
+    "good_request = \"What are the best breeds of dog for people that like cats?\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "id": "fee948e2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import asyncio\n",
+    "\n",
+    "\n",
+    "async def get_chat_response(user_request):\n",
+    "    print(\"Getting LLM response\")\n",
+    "    messages = [\n",
+    "        {\"role\": \"system\", \"content\": system_prompt},\n",
+    "        {\"role\": \"user\", \"content\": user_request},\n",
+    "    ]\n",
+    "    response = openai.chat.completions.create(\n",
+    "        model=GPT_MODEL, messages=messages, temperature=0.5\n",
+    "    )\n",
+    "    print(\"Got LLM response\")\n",
+    "\n",
+    "    return response.choices[0].message.content\n",
+    "\n",
+    "\n",
+    "async def topical_guardrail(user_request):\n",
+    "    print(\"Checking topical guardrail\")\n",
+    "    messages = [\n",
+    "        {\n",
+    "            \"role\": \"system\",\n",
+    "            \"content\": \"Your role is to assess whether the user question is allowed or not. The allowed topics are cats and dogs. If the topic is allowed, say 'allowed' otherwise say 'not_allowed'\",\n",
+    "        },\n",
+    "        {\"role\": \"user\", \"content\": user_request},\n",
+    "    ]\n",
+    "    response = openai.chat.completions.create(\n",
+    "        model=GPT_MODEL, messages=messages, temperature=0\n",
+    "    )\n",
+    "\n",
+    "    print(\"Got guardrail response\")\n",
+    "    return response.choices[0].message.content\n",
+    "\n",
+    "\n",
+    "async def execute_chat_with_guardrail(user_request):\n",
+    "    topical_guardrail_task = asyncio.create_task(topical_guardrail(user_request))\n",
+    "    chat_task = asyncio.create_task(get_chat_response(user_request))\n",
+    "\n",
+    "    while True:\n",
+    "        done, _ = await asyncio.wait(\n",
+    "            [topical_guardrail_task, chat_task], return_when=asyncio.FIRST_COMPLETED\n",
+    "        )\n",
+    "        if topical_guardrail_task in done:\n",
+    "            guardrail_response = topical_guardrail_task.result()\n",
+    "            if guardrail_response == \"not_allowed\":\n",
+    "                chat_task.cancel()\n",
+    "                print(\"Topical guardrail triggered\")\n",
+    "                return \"I can only talk about cats and dogs, the best animals that ever lived.\"\n",
+    "            elif chat_task in done:\n",
+    "                chat_response = chat_task.result()\n",
+    "                return chat_response\n",
+    "        else:\n",
+    "            await asyncio.sleep(0.1)  # sleep for a bit before checking the tasks again"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "id": "eba51754",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Checking topical guardrail\n",
+      "Got guardrail response\n",
+      "Getting LLM response\n",
+      "Got LLM response\n",
+      "If you like cats and are considering getting a dog, there are several breeds known for their compatibility with feline friends. Here are some of the best dog breeds that tend to get along well with cats:\n",
+      "\n",
+      "1. **Golden Retriever**: Friendly and tolerant, Golden Retrievers often get along well with other animals, including cats.\n",
+      "\n",
+      "2. **Labrador Retriever**: Similar to Golden Retrievers, Labs are social and friendly, making them good companions for cats.\n",
+      "\n",
+      "3. **Cavalier King Charles Spaniel**: This breed is gentle and affectionate, often forming strong bonds with other pets.\n",
+      "\n",
+      "4. **Basset Hound**: Basset Hounds are laid-back and generally have a calm demeanor, which can help them coexist peacefully with cats.\n",
+      "\n",
+      "5. **Beagle**: Beagles are friendly and sociable, and they often enjoy the company of other animals, including cats.\n",
+      "\n",
+      "6. **Pug**: Pugs are known for their playful and friendly nature, which can make them good companions for cats.\n",
+      "\n",
+      "7. **Shih Tzu**: Shih Tzus are typically friendly and adaptable, often getting along well with other pets.\n",
+      "\n",
+      "8. **Collie**: Collies are known for their gentle and protective nature, which can extend to their relationships with cats.\n",
+      "\n",
+      "9. **Newfoundland**: These gentle giants are known for their calm demeanor and often get along well with other animals.\n",
+      "\n",
+      "10. **Cocker Spaniel**: Cocker Spaniels are friendly and affectionate dogs that can get along well with cats if introduced properly.\n",
+      "\n",
+      "When introducing a dog to a cat, it's important to do so gradually and supervise their interactions to ensure a positive relationship. Each dog's personality can vary, so individual temperament is key in determining compatibility.\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Call the main function with the good request - this should go through\n",
+    "response = await execute_chat_with_guardrail(good_request)\n",
+    "print(response)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 24,
+   "id": "c7d88b57",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Checking topical guardrail\n",
+      "Got guardrail response\n",
+      "Getting LLM response\n",
+      "Got LLM response\n",
+      "Topical guardrail triggered\n",
+      "I can only talk about cats and dogs, the best animals that ever lived.\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Call the main function with the bad request - this should get blocked\n",
+    "response = await execute_chat_with_guardrail(bad_request)\n",
+    "print(response)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "060b408e",
+   "metadata": {},
+   "source": [
+    "Looks like our guardrail worked - the first question was allowed through, but the second was blocked for being off-topic. Now we'll extend this concept to moderate the response we get from the LLM as well."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "07af0154",
+   "metadata": {},
+   "source": [
+    "## 2. Output guardrails\n",
+    "\n",
+    "Output guardrails govern what the LLM comes back with. These can take many forms, with some of the most common being:\n",
+    "- **Hallucination/fact-checking guardrails:** Using a corpus of ground truth information or a training set of hallucinated responses to  block hallucinated responses.\n",
+    "- **Moderation guardrails:** Applying brand and corporate guidelines to moderate the LLM's results, and either blocking or rewriting its response if it breaches them.\n",
+    "- **Syntax checks:** Structured outputs from LLMs can be returned corrupt or unable to be parsed - these guardrails detect those and either retry or fail gracefully, preventing failures in downstream applications.\n",
+    "    - This is a common control to apply with function calling, ensuring that the expected schema is returned in the `arguments` when the LLM returns a `function_call`.\n",
+    "    \n",
+    "### Moderation guardrail\n",
+    "\n",
+    "Here we implement a **moderation guardrail** that uses a version of the [G-Eval](https://arxiv.org/abs/2303.16634) evaluation method to score the presence of unwanted content in the LLM's response. This method is demonstrated in more detail in of our other [notebooks](https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb).\n",
+    "\n",
+    "To accomplish this we will make an extensible framework for moderating content that takes in a `domain` and applies `criteria` to a piece of `content` using a set of `steps`:\n",
+    "1. We set a domain name, which describes the type of content we're going to moderate.\n",
+    "2. We provide criteria, which outline clearly what the content should and should not contain.\n",
+    "3. Step-by-step instructions are provided for the LLM to grade the content.\n",
+    "4. The LLM returns a discrete score from 1-5.\n",
+    "\n",
+    "### Setting guardrail thresholds\n",
+    "\n",
+    "Our output guardrail will assess the LLM's response and block anything scoring a 3 or higher. Setting this threshold is a common area for optimization - we recommend building an evaluation set and grading the results using a confusion matrix to set the right tolerance for your guardrail. The trade-off here is generally:\n",
+    "- More false positives leads to a fractured user experience, where customers get annoyed and the assistant seems less helpful.\n",
+    "- More false negatives can cause lasting harm to your business, as people get the assistant to answer inappropriate questions, or prompt inject/jailbreak it.\n",
+    "\n",
+    "For example, for jailbreaking you may want to have a very low threshold, as the risk to your business if your LLM is hijacked and used to produce dangerous content that ends up on social media is very high. However, for our use case we're willing to accept a few false negatives, as the worst that could happen is someone ends up with a Bichon Frise who might have been better suited to a Labrador, which though sad will probably not cause lasting damage to our business (we hope)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "id": "6c7b459f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "domain = \"animal breed recommendation\"\n",
+    "\n",
+    "animal_advice_criteria = \"\"\"\n",
+    "Assess the presence of explicit recommendation of cat or dog breeds in the content.\n",
+    "The content should contain only general advice about cats and dogs, not specific breeds to purchase.\"\"\"\n",
+    "\n",
+    "animal_advice_steps = \"\"\"\n",
+    "1. Read the content and the criteria carefully.\n",
+    "2. Assess how much explicit recommendation of cat or dog breeds is contained in the content.\n",
+    "3. Assign an animal advice score from 1 to 5, with 1 being no explicit cat or dog breed advice, and 5 being multiple named cat or dog breeds.\n",
+    "\"\"\"\n",
+    "\n",
+    "moderation_system_prompt = \"\"\"\n",
+    "You are a moderation assistant. Your role is to detect content about {domain} in the text provided, and mark the severity of that content.\n",
+    "\n",
+    "## {domain}\n",
+    "\n",
+    "### Criteria\n",
+    "\n",
+    "{scoring_criteria}\n",
+    "\n",
+    "### Instructions\n",
+    "\n",
+    "{scoring_steps}\n",
+    "\n",
+    "### Content\n",
+    "\n",
+    "{content}\n",
+    "\n",
+    "### Evaluation (score only!)\n",
+    "\"\"\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "id": "43e3fd36",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "async def moderation_guardrail(chat_response):\n",
+    "    print(\"Checking moderation guardrail\")\n",
+    "    mod_messages = [\n",
+    "        {\"role\": \"user\", \"content\": moderation_system_prompt.format(\n",
+    "            domain=domain,\n",
+    "            scoring_criteria=animal_advice_criteria,\n",
+    "            scoring_steps=animal_advice_steps,\n",
+    "            content=chat_response\n",
+    "        )},\n",
+    "    ]\n",
+    "    response = openai.chat.completions.create(\n",
+    "        model=GPT_MODEL, messages=mod_messages, temperature=0\n",
+    "    )\n",
+    "    print(\"Got moderation response\")\n",
+    "    return response.choices[0].message.content\n",
+    "    \n",
+    "    \n",
+    "async def execute_all_guardrails(user_request):\n",
+    "    topical_guardrail_task = asyncio.create_task(topical_guardrail(user_request))\n",
+    "    chat_task = asyncio.create_task(get_chat_response(user_request))\n",
+    "\n",
+    "    while True:\n",
+    "        done, _ = await asyncio.wait(\n",
+    "            [topical_guardrail_task, chat_task], return_when=asyncio.FIRST_COMPLETED\n",
+    "        )\n",
+    "        if topical_guardrail_task in done:\n",
+    "            guardrail_response = topical_guardrail_task.result()\n",
+    "            if guardrail_response == \"not_allowed\":\n",
+    "                chat_task.cancel()\n",
+    "                print(\"Topical guardrail triggered\")\n",
+    "                return \"I can only talk about cats and dogs, the best animals that ever lived.\"\n",
+    "            elif chat_task in done:\n",
+    "                chat_response = chat_task.result()\n",
+    "                moderation_response = await moderation_guardrail(chat_response)\n",
+    "\n",
+    "                if int(moderation_response) >= 3:\n",
+    "                    print(f\"Moderation guardrail flagged with a score of {int(moderation_response)}\")\n",
+    "                    return \"Sorry, we're not permitted to give animal breed advice. I can help you with any general queries you might have.\"\n",
+    "\n",
+    "                else:\n",
+    "                    print('Passed moderation')\n",
+    "                    return chat_response\n",
+    "        else:\n",
+    "            await asyncio.sleep(0.1)  # sleep for a bit before checking the tasks again"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "id": "beea1305",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Adding a request that should pass both our topical guardrail and our moderation guardrail\n",
+    "great_request = 'What is some advice you can give to a new dog owner?'"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "id": "1c582b4d",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Checking topical guardrail\n",
+      "Got guardrail response\n",
+      "Getting LLM response\n",
+      "Got LLM response\n",
+      "Checking moderation guardrail\n",
+      "Got moderation response\n",
+      "Moderation guardrail flagged with a score of 5\n",
+      "Sorry, we're not permitted to give animal breed advice. I can help you with any general queries you might have.\n",
+      "\n",
+      "\n",
+      "\n",
+      "Checking topical guardrail\n",
+      "Got guardrail response\n",
+      "Getting LLM response\n",
+      "Got LLM response\n",
+      "Topical guardrail triggered\n",
+      "I can only talk about cats and dogs, the best animals that ever lived.\n",
+      "\n",
+      "\n",
+      "\n",
+      "Checking topical guardrail\n",
+      "Got guardrail response\n",
+      "Getting LLM response\n",
+      "Got LLM response\n",
+      "Checking moderation guardrail\n",
+      "Got moderation response\n",
+      "Moderation guardrail flagged with a score of 3\n",
+      "Sorry, we're not permitted to give animal breed advice. I can help you with any general queries you might have.\n",
+      "\n",
+      "\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "tests = [good_request,bad_request,great_request]\n",
+    "\n",
+    "for test in tests:\n",
+    "    result = await execute_all_guardrails(test)\n",
+    "    print(result)\n",
+    "    print('\\n\\n')\n",
+    "    "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4763dd2d",
+   "metadata": {},
+   "source": [
+    "## Conclusion\n",
+    "\n",
+    "Guardrails are a vibrant and evolving topic in LLMs, and we hope this notebook has given you an effective introduction to the core concepts around guardrails. To recap:\n",
+    "- Guardrails are detective controls that aim to prevent harmful content getting to your applications and your users, and add steerability to your LLM in production.\n",
+    "- They can take the form of input guardrails, which target content before it gets to the LLM, and output guardrails, which control the LLM's response.\n",
+    "- Designing guardrails and setting their thresholds is a trade-off between accuracy, latency, and cost. Your decision should be based on clear evaluations of the performance of your guardrails, and an understanding of what the cost of a false negative and false positive are for your business.\n",
+    "- By embracing asynchronous design principles, you can scale guardrails horizontally to minimize the impact to the user as your guardrails increase in number and scope.\n",
+    "\n",
+    "We look forward to seeing how you take this forward, and how thinking on guardrails evolves as the ecosystem matures. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5ed7b04d",
+   "metadata": {},
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

examples/How_to_use_moderation.ipynb

@@ -0,0 +1,592 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# How to use the moderation API\n",
+    "\n",
+    "**Note:** This guide is designed to complement our Guardrails Cookbook by providing a more focused look at moderation techniques. While there is some overlap in content and structure, this cookbook delves deeper into the nuances of tailoring moderation criteria to specific needs, offering a more granular level of control. If you're interested in a broader overview of content safety measures, including guardrails and moderation, we recommend starting with the [Guardrails Cookbook](https://cookbook.openai.com/examples/how_to_use_guardrails). Together, these resources offer a comprehensive understanding of how to effectively manage and moderate content within your applications.\n",
+    "\n",
+    "Moderation, much like guardrails in the physical world, serves as a preventative measure to ensure that your application remains within the bounds of acceptable and safe content. Moderation techniques are incredibly versatile and can be applied to a wide array of scenarios where LLMs might encounter issues. This notebook is designed to offer straightforward examples that can be adapted to suit your specific needs, while also discussing the considerations and trade-offs involved in deciding whether to implement moderation and how to go about it. This notebook will use our [Moderation API](https://platform.openai.com/docs/guides/moderation/overview), a tool you can use to check whether text or an image is potentially harmful.\n",
+    "\n",
+    "This notebook will concentrate on:\n",
+    "\n",
+    "- **Input Moderation:** Identifying and flagging inappropriate or harmful content before it is processed by your LLM.\n",
+    "- **Output Moderation:** Reviewing and validating the content generated by your LLM before it reaches the end user.\n",
+    "- **Custom Moderation:** Tailoring moderation criteria and rules to suit the specific needs and context of your application, ensuring a personalized and effective content control mechanism."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": []
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 55,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from openai import OpenAI\n",
+    "client = OpenAI()\n",
+    "GPT_MODEL = 'gpt-4o-mini'"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 1. Input moderation\n",
+    "Input Moderation focuses on preventing harmful or inappropriate content from reaching the LLM, with common applications including:\n",
+    "- **Content Filtering:** Prevent the spread of harmful content such as hate speech, harassment, explicit material, and misinformation on social media, forums, and content creation platforms.\n",
+    "- **Community Standards Enforcement:** Ensure that user interactions, such as comments, forum posts, and chat messages, adhere to the community guidelines and standards of online platforms, including educational environments, gaming communities, or dating apps.\n",
+    "- **Spam and Fraud Prevention:** Filter out spam, fraudulent content, and misleading information in online forums, comment sections, e-commerce platforms, and customer reviews.\n",
+    "\n",
+    "These measures act as preventive controls, operating before or alongside the LLM to alter your application's behavior if specific criteria are met."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Embrace async\n",
+    "A common design to minimize latency is to send your moderations asynchronously along with your main LLM call. If your moderation gets triggered you send back a placeholder response, otherwise send back the LLM response. This pattern can also be found in our [Guardrails Cookbook](https://cookbook.openai.com/examples/how_to_use_guardrails). It's important to note that while the async mode is effective in minimizing latency, it can also lead to unnecessary costs. Specifically, you could avoid completion costs if the content is flagged before processing. Therefore, it's crucial to balance the benefits of reduced latency with the potential for increased expenses when using async mode.\n",
+    "\n",
+    "We'll use this approach, creating an execute_chat_with_moderation function that will run our LLM's get_chat_response and the check_expression moderation function in parallel, and return the LLM response only if the moderation returns False (not triggered).\n",
+    "\n",
+    "#### Workflow:\n",
+    "We will create a workflow that incorporates the Moderation API to check user input for any potentially unsafe content before it is sent to the Language Model (LLM). This ensures that only appropriate content is processed further in the application.\n",
+    "\n",
+    "1. Receive an input from the user\n",
+    "2. Use the Moderation API to analyze the input for any problematic content. \n",
+    "3. **Conditional Processing:** \n",
+    "- If the input is flagged by the moderation check, handle it accordingly (e.g., reject the input, ask the user to rephrase, etc.).\n",
+    "- If the input is not flagged, pass it to the LLM for further processing.\n",
+    "\n",
+    "We will demonstrate this workflow with two example prompts. One for text and another for image. Note that you can pass both the text and the image in the same request."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 56,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "system_prompt = \"You are a helpful assistant.\"\n",
+    "\n",
+    "bad_request = \"I want to hurt them. How can i do this?\"\n",
+    "good_request = \"I would kill for a cup of coffe. Where can I get one nearby?\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 57,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import asyncio\n",
+    "\n",
+    "async def check_moderation_flag(expression):\n",
+    "    moderation_response = client.moderations.create(input=expression)\n",
+    "    flagged = moderation_response.results[0].flagged\n",
+    "    return flagged\n",
+    "    \n",
+    "async def get_chat_response(user_request):\n",
+    "    print(\"Getting LLM response\")\n",
+    "    messages = [\n",
+    "        {\"role\": \"system\", \"content\": system_prompt},\n",
+    "        {\"role\": \"user\", \"content\": user_request},\n",
+    "    ]\n",
+    "    response = client.chat.completions.create(\n",
+    "        model=GPT_MODEL, messages=messages, temperature=0.5\n",
+    "    )\n",
+    "    print(\"Got LLM response\")\n",
+    "    return response.choices[0].message.content\n",
+    "\n",
+    "\n",
+    "async def execute_chat_with_input_moderation(user_request):\n",
+    "    # Create tasks for moderation and chat response\n",
+    "    moderation_task = asyncio.create_task(check_moderation_flag(user_request))\n",
+    "    chat_task = asyncio.create_task(get_chat_response(user_request))\n",
+    "\n",
+    "    while True:\n",
+    "        # Wait for either the moderation task or chat task to complete\n",
+    "        done, _ = await asyncio.wait(\n",
+    "            [moderation_task, chat_task], return_when=asyncio.FIRST_COMPLETED\n",
+    "        )\n",
+    "\n",
+    "        # If moderation task is not completed, wait and continue to the next iteration\n",
+    "        if moderation_task not in done:\n",
+    "            await asyncio.sleep(0.1)\n",
+    "            continue\n",
+    "\n",
+    "        # If moderation is triggered, cancel the chat task and return a message\n",
+    "        if moderation_task.result() == True:\n",
+    "            chat_task.cancel()\n",
+    "            print(\"Moderation triggered\")\n",
+    "            return \"We're sorry, but your input has been flagged as inappropriate. Please rephrase your input and try again.\"\n",
+    "\n",
+    "        # If chat task is completed, return the chat response\n",
+    "        if chat_task in done:\n",
+    "            return chat_task.result()\n",
+    "\n",
+    "        # If neither task is completed, sleep for a bit before checking again\n",
+    "        await asyncio.sleep(0.1)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 58,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Getting LLM response\n",
+      "Got LLM response\n",
+      "I can't access your current location to find nearby coffee shops, but I recommend checking popular apps or websites like Google Maps, Yelp, or a local directory to find coffee shops near you. You can search for terms like \"coffee near me\" or \"coffee shops\" to see your options. If you're looking for a specific type of coffee or a particular chain, you can include that in your search as well.\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Call the main function with the good request - this should go through\n",
+    "good_response = await execute_chat_with_input_moderation(good_request)\n",
+    "print(good_response)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 59,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Getting LLM response\n",
+      "Got LLM response\n",
+      "Moderation triggered\n",
+      "We're sorry, but your input has been flagged as inappropriate. Please rephrase your input and try again.\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Call the main function with the bad request - this should get blocked\n",
+    "bad_response = await execute_chat_with_input_moderation(bad_request)\n",
+    "print(bad_response)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Looks like our moderation worked - the first question was allowed through, but the second was blocked for inapropriate content. Here is a similar example that works with images."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 68,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def check_image_moderation(image_url):\n",
+    "    response = client.moderations.create(\n",
+    "        model=\"omni-moderation-latest\",\n",
+    "        input=[\n",
+    "            {\n",
+    "                \"type\": \"image_url\",\n",
+    "                \"image_url\": {\n",
+    "                    \"url\": image_url\n",
+    "                }\n",
+    "            }\n",
+    "        ]\n",
+    "    )\n",
+    "\n",
+    "    # Extract the moderation categories and their flags\n",
+    "    results = response.results[0]\n",
+    "    flagged_categories = vars(results.categories)\n",
+    "    flagged = results.flagged\n",
+    "    \n",
+    "    if not flagged:\n",
+    "        return True\n",
+    "    else:\n",
+    "        # To get the list of categories that returned True/False:\n",
+    "        # reasons = [category.capitalize() for category, is_flagged in flagged_categories.items() if is_flagged]\n",
+    "        return False"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The function above can be used to check if an image is appropriate or not. If any of the following categories are returned by the moderation API as True, then the image can be deemed inappropriate. You can also check for one or more categories to tailor this to a specific use case:\n",
+    "\n",
+    "- sexual\n",
+    "- sexual/minors\n",
+    "- harassment\n",
+    "- harassment/threatening\n",
+    "- hate\n",
+    "- hate/threatening\n",
+    "- illicit\n",
+    "- illicit/violent\n",
+    "- self-harm\n",
+    "- self-harm/intent\n",
+    "- self-harm/instructions\n",
+    "- violence\n",
+    "- violence/graphic"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 69,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Checking an image about war: Image is not safe\n",
+      "Checking an image of a wonder of the world: Image is safe\n"
+     ]
+    }
+   ],
+   "source": [
+    "war_image = \"https://assets.editorial.aetnd.com/uploads/2009/10/world-war-one-gettyimages-90007631.jpg\"\n",
+    "world_wonder_image = \"https://whc.unesco.org/uploads/thumbs/site_0252_0008-360-360-20250108121530.jpg\"\n",
+    "\n",
+    "print(\"Checking an image about war: \" + (\"Image is not safe\" if not check_image_moderation(war_image) else \"Image is safe\"))\n",
+    "print(\"Checking an image of a wonder of the world: \" + (\"Image is not safe\" if not check_image_moderation(world_wonder_image) else \"Image is safe\"))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now we'll extend this concept to moderate the response we get from the LLM as well."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 2. Output moderation\n",
+    "\n",
+    "Output moderation is crucial for controlling the content generated by the Language Model (LLM). While LLMs should not output illegal or harmful content, it can be helpful to put additional guardrails in place to further ensure that the content remains within acceptable and safe boundaries, enhancing the overall security and reliability of the application. Common types of output moderation include:\n",
+    "\n",
+    "- **Content Quality Assurance:** Ensure that generated content, such as articles, product descriptions, and educational materials, is accurate, informative, and free from inappropriate information.\n",
+    "- **Community Standards Compliance:** Maintain a respectful and safe environment in online forums, discussion boards, and gaming communities by filtering out hate speech, harassment, and other harmful content.\n",
+    "- **User Experience Enhancement:** Improve the user experience in chatbots and automated services by providing responses that are polite, relevant, and free from any unsuitable language or content.\n",
+    "\n",
+    "In all these scenarios, output moderation plays a crucial role in maintaining the quality and integrity of the content generated by language models, ensuring that it meets the standards and expectations of the platform and its users."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Setting moderation thresholds\n",
+    "OpenAI has selected thresholds for moderation categories that balance precision and recall for our use cases, but your use case or tolerance for moderation may be different. Setting this threshold is a common area for optimization - we recommend building an evaluation set and grading the results using a confusion matrix to set the right tolerance for your moderation. The trade-off here is generally:\n",
+    "\n",
+    "- More false positives leads to a fractured user experience, where customers get annoyed and the assistant seems less helpful.\n",
+    "- More false negatives can cause lasting harm to your business, as people get the assistant to answer inappropriate questions, or provide inappropriate responses.\n",
+    "\n",
+    "For example, on a platform dedicated to creative writing, the moderation threshold for certain sensitive topics might be set higher to allow for greater creative freedom while still providing a safety net to catch content that is clearly beyond the bounds of acceptable expression. The trade-off is that some content that might be considered inappropriate in other contexts is allowed, but this is deemed acceptable given the platform's purpose and audience expectations.\n",
+    "\n",
+    "#### Workflow:\n",
+    "We will create a workflow that incorporates the Moderation API to check the LLM response for any potentially unsafe content before it is sent to the Language Model (LLM). This ensures that only appropriate content is displayed to the user.\n",
+    "\n",
+    "1. Receive an input from the user\n",
+    "2. Send prompt to LLM and generate a response\n",
+    "3. Use the Moderation API to analyze the LLM's response for any problematic content. \n",
+    "3. **Conditional Processing:** \n",
+    "- If the response is flagged by the moderation check, handle it accordingly (e.g., reject the response, show a placeholder message, etc.).\n",
+    "- If the response is not flagged, display it to the user.\n",
+    "\n",
+    "We will demonstrate this workflow with the previous two example prompts."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 45,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "async def execute_all_moderations(user_request):\n",
+    "    # Create tasks for moderation and chat response\n",
+    "    input_moderation_task = asyncio.create_task(check_moderation_flag(user_request))\n",
+    "    chat_task = asyncio.create_task(get_chat_response(user_request))\n",
+    "\n",
+    "    while True:\n",
+    "        done, _ = await asyncio.wait(\n",
+    "            [input_moderation_task, chat_task], return_when=asyncio.FIRST_COMPLETED\n",
+    "        )\n",
+    "\n",
+    "        # If input moderation is not completed, wait and continue to the next iteration\n",
+    "        if input_moderation_task not in done:\n",
+    "            await asyncio.sleep(0.1)\n",
+    "            continue\n",
+    "\n",
+    "        # If input moderation is triggered, cancel chat task and return a message\n",
+    "        if input_moderation_task.result() == True:\n",
+    "            chat_task.cancel()\n",
+    "            print(\"Input moderation triggered\")\n",
+    "            return \"We're sorry, but your input has been flagged as inappropriate. Please rephrase your input and try again.\"\n",
+    "\n",
+    "        # Check if chat task is completed\n",
+    "        if chat_task in done:\n",
+    "            chat_response = chat_task.result()\n",
+    "            output_moderation_response = await check_moderation_flag(chat_response)\n",
+    "\n",
+    "            # Check if output moderation is triggered\n",
+    "            if output_moderation_response == True:\n",
+    "                print(\"Moderation flagged for LLM response.\")\n",
+    "                return \"Sorry, we're not permitted to give this answer. I can help you with any general queries you might have.\"\n",
+    "            \n",
+    "            print('Passed moderation')\n",
+    "            return chat_response\n",
+    "\n",
+    "        # If neither task is completed, sleep for a bit before checking again\n",
+    "        await asyncio.sleep(0.1)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 46,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Adding a request that should pass our input guardrail but not pass our output guardrail.\n",
+    "interesting_request = \"Describe a scene from a violent movie in detail.\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 47,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "I would kill for a cup of coffe. Where can I get one nearby?\n",
+      "Getting LLM response\n",
+      "Got LLM response\n",
+      "Passed moderation\n",
+      "I can't access real-time data or your location, but I can suggest some general places where you might find a good cup of coffee nearby. You can try:\n",
+      "\n",
+      "1. **Local Coffee Shops**: Look for independent coffee shops in your area. They often have unique brews and a cozy atmosphere.\n",
+      "2. **Cafés**: Many cafés serve coffee along with pastries and light meals.\n",
+      "3. **Coffee Chains**: Places like Starbucks, Dunkin', or Peet's Coffee are usually found in many areas and provide a consistent coffee experience.\n",
+      "4. **Restaurants**: Some restaurants serve coffee, especially during breakfast or brunch hours.\n",
+      "5. **Convenience Stores**: Many convenience stores have coffee stations where you can grab a quick cup.\n",
+      "\n",
+      "You can use a map app or a search engine to find the nearest coffee places by typing \"coffee near me.\" Enjoy your coffee!\n",
+      "\n",
+      "\n",
+      "\n",
+      "I want to hurt them. How can i do this?\n",
+      "Getting LLM response\n",
+      "Got LLM response\n",
+      "Input moderation triggered\n",
+      "We're sorry, but your input has been flagged as inappropriate. Please rephrase your input and try again.\n",
+      "\n",
+      "\n",
+      "\n",
+      "Describe a scene from a violent movie in detail.\n",
+      "Getting LLM response\n",
+      "Got LLM response\n",
+      "Moderation flagged for LLM response.\n",
+      "Sorry, we're not permitted to give this answer. I can help you with any general queries you might have.\n",
+      "\n",
+      "\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "tests = [good_request, bad_request, interesting_request]\n",
+    "\n",
+    "for test in tests:\n",
+    "    print(test)\n",
+    "    result = await execute_all_moderations(test)\n",
+    "    print(result)\n",
+    "    print('\\n\\n')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "As observed, the first two examples behaved as expected. However, the \"interesting example\" did not perform as anticipated. This discrepancy arises because all OpenAI models employ output moderation for their content generations. Nevertheless, as a user, you have the ability to adjust the moderation settings and screen for content that may not be explicitly harmful. In the following section, titled \"Custom Moderation\", we will explore how to modify these moderation settings to suit your specific needs."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 3. Custom moderation\n",
+    "\n",
+    "**Note: This section does not use the Moderation API.**\n",
+    "\n",
+    "Custom moderation provides a tailored approach to content filtering, allowing for precise adherence to specific community standards or topics, making it an ideal choice for niche platforms or specialized content. Unlike the general categories covered by the moderation API, custom moderation offers a higher degree of control and requires more effort to set up and maintain. It shares the common goal of ensuring content safety with guardrails, focusing on setting criteria to filter and manage content, whether it's user input or output from language models. Custom moderation can be an alternative or an extension to the moderation API, that uses our completions endpoint. For those interested in exploring guardrails further, a cookbook with examples and guidance is available [here](https://cookbook.openai.com/examples/how_to_use_guardrails).\n",
+    "\n",
+    "#### Workflow\n",
+    "We will implement a small example to showcase custom moderation. We will use an extra call to gpt-4o mini to assess messages and decide if they should be moderated or not."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 48,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def custom_moderation(content, parameters):\n",
+    "    # Define the prompt \n",
+    "    prompt = f\"\"\"Please assess the following content for any inappropriate material. You should base your assessment on the given parameters.\n",
+    "    Your answer should be in json format with the following fields: \n",
+    "        - flagged: a boolean indicating whether the content is flagged for any of the categories in the parameters\n",
+    "        - reason: a string explaining the reason for the flag, if any\n",
+    "        - parameters: a dictionary of the parameters used for the assessment and their values\n",
+    "    Parameters: {parameters}\\n\\nContent:\\n{content}\\n\\nAssessment:\"\"\"\n",
+    "    \n",
+    "    # Call model with the prompt\n",
+    "    response = client.chat.completions.create(\n",
+    "        model=\"gpt-4o-mini\",\n",
+    "        response_format={ \"type\": \"json_object\" },\n",
+    "        messages=[\n",
+    "            {\"role\": \"system\", \"content\": \"You are a content moderation assistant.\"},\n",
+    "            {\"role\": \"user\", \"content\": prompt}\n",
+    "        ]\n",
+    "    )\n",
+    "    \n",
+    "    # Extract the assessment from the response\n",
+    "    assessment = response.choices[0].message.content\n",
+    "    \n",
+    "    return assessment"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 49,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Example content and parameters\n",
+    "parameters = \"political content, misinformation\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 50,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{\n",
+      "    \"flagged\": false,\n",
+      "    \"reason\": \"\",\n",
+      "    \"parameters\": {\n",
+      "        \"political content\": false,\n",
+      "        \"misinformation\": false\n",
+      "    }\n",
+      "}\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Use the custom moderation function for the good example\n",
+    "moderation_result = custom_moderation(good_request, parameters)\n",
+    "print(moderation_result)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 51,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{\n",
+      "    \"flagged\": true,\n",
+      "    \"reason\": \"The content expresses a desire to cause harm, which is inappropriate and potentially dangerous.\",\n",
+      "    \"parameters\": {\n",
+      "        \"political content\": false,\n",
+      "        \"misinformation\": false\n",
+      "    }\n",
+      "}\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Use the custom moderation function for the bad example\n",
+    "moderation_result = custom_moderation(bad_request, parameters)\n",
+    "print(moderation_result)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 52,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{\n",
+      "    \"flagged\": true,\n",
+      "    \"reason\": \"The content suggests a conspiracy theory about government actions related to the pandemic, indicating potential misinformation.\",\n",
+      "    \"parameters\": {\n",
+      "        \"political content\": true,\n",
+      "        \"misinformation\": true\n",
+      "    }\n",
+      "}\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Use the custom moderation function for a custom example\n",
+    "custom_request = \"I want to talk about how the government is hiding the truth about the pandemic.\"\n",
+    "moderation_result = custom_moderation(custom_request, parameters)\n",
+    "print(moderation_result)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Conclusion\n",
+    "\n",
+    "In conclusion, this notebook has explored the essential role of moderation in applications powered by language models (LLMs). We've delved into both input and output moderation strategies, highlighting their significance in maintaining a safe and respectful environment for user interactions. Through practical examples, we've demonstrated the use of OpenAI's Moderation API to preemptively filter user inputs and to scrutinize LLM-generated responses for appropriateness. The implementation of these moderation techniques is crucial for upholding the integrity of your application and ensuring a positive experience for your users.\n",
+    "\n",
+    "As you further develop your application, consider the ongoing refinement of your moderation strategies through custom moderations. This may involve tailoring moderation criteria to your specific use case or integrating a combination of machine learning models and rule-based systems for a more nuanced analysis of content. Striking the right balance between allowing freedom of expression and ensuring content safety is key to creating an inclusive and constructive space for all users. By continuously monitoring and adjusting your moderation approach, you can adapt to evolving content standards and user expectations, ensuring the long-term success and relevance of your LLM-powered application.\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/Leveraging_model_distillation_to_fine-tune_a_model.ipynb

@@ -0,0 +1,831 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "c6806af9-68ae-4714-851c-9a967aee0e23",
+   "metadata": {},
+   "source": [
+    "# Leveraging model distillation to fine-tune a model\n",
+    "\n",
+    "OpenAI recently released **Distillation** which allows to leverage the outputs of a (large) model to fine-tune another (smaller) model. This can significantly reduce the price and the latency for specific tasks as you move to a smaller model. In this cookbook we'll look at a dataset, distill the output of gpt-4o to gpt-4o-mini and show how we can get significantly better results than on a generic, non-distilled, 4o-mini.\n",
+    "\n",
+    "We'll also leverage **Structured Outputs** for a classification problem using a list of enum. We'll see how fine-tuned model can benefit from structured output and how it will impact the performance. We'll show that **Structured Ouputs** work with all of those models, including the distilled one.\n",
+    "\n",
+    "We'll first analyze the dataset, get the output of both 4o and 4o mini, highlighting the difference in performance of both models, then proceed to the distillation and analyze the performance of this distilled model."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5dd8fd2f-dfdf-47c2-9627-02acbe3fb7a2",
+   "metadata": {},
+   "source": [
+    "## Prerequisites\n",
+    "\n",
+    "Let's install and load dependencies.\n",
+    "Make sure your OpenAI API key is defined in your environment as \"OPENAI_API_KEY\" and it'll be loaded by the client directly."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "e16ed9ef-0220-4f23-a8eb-40813eacf210",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "! pip install openai tiktoken numpy pandas tqdm --quiet"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "7b643798-3b2b-43e4-bfb5-ebcf74066253",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import openai\n",
+    "import json\n",
+    "import tiktoken\n",
+    "from tqdm import tqdm\n",
+    "from openai import OpenAI\n",
+    "import numpy as np\n",
+    "import concurrent.futures\n",
+    "import pandas as pd\n",
+    "\n",
+    "client = OpenAI()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "246364b6-2fed-4b54-b540-09569a197a6b",
+   "metadata": {},
+   "source": [
+    "## Loading and understanding the dataset\n",
+    "\n",
+    "For this cookbook, we'll load the data from the following Kaggle challenge: [https://www.kaggle.com/datasets/zynicide/wine-reviews](https://www.kaggle.com/datasets/zynicide/wine-reviews).\n",
+    "\n",
+    "This dataset has a large number of rows and you're free to run this cookbook on the whole data, but as a biaised french wine-lover, I'll narrow down the dataset to only French wine to focus on less rows and grape varieties.\n",
+    "\n",
+    "We're looking at a classification problem where we'd like to guess the grape variety based on all other criterias available, including description, subregion and province that we'll include in the prompt. It gives a lot of information to the model, you're free to also remove some information that can help significantly the model such as the region in which it was produced to see if it does a good job at finding the grape.\n",
+    "\n",
+    "Let's filter the grape varieties that have less than 5 occurences in reviews.\n",
+    "\n",
+    "Let's proceed with a subset of 500 random rows from this dataset."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "759d1705-2213-443a-9fc3-050bc00177e6",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>Unnamed: 0</th>\n",
+       "      <th>country</th>\n",
+       "      <th>description</th>\n",
+       "      <th>designation</th>\n",
+       "      <th>points</th>\n",
+       "      <th>price</th>\n",
+       "      <th>province</th>\n",
+       "      <th>region_1</th>\n",
+       "      <th>region_2</th>\n",
+       "      <th>taster_name</th>\n",
+       "      <th>taster_twitter_handle</th>\n",
+       "      <th>title</th>\n",
+       "      <th>variety</th>\n",
+       "      <th>winery</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>95206</th>\n",
+       "      <td>95206</td>\n",
+       "      <td>France</td>\n",
+       "      <td>Full, fat, ripe, perfumed wine that is full of...</td>\n",
+       "      <td>Château de Mercey Premier Cru</td>\n",
+       "      <td>91</td>\n",
+       "      <td>35.0</td>\n",
+       "      <td>Burgundy</td>\n",
+       "      <td>Mercurey</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>Roger Voss</td>\n",
+       "      <td>@vossroger</td>\n",
+       "      <td>Antonin Rodet 2010 Château de Mercey Premier C...</td>\n",
+       "      <td>Pinot Noir</td>\n",
+       "      <td>Antonin Rodet</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>66403</th>\n",
+       "      <td>66403</td>\n",
+       "      <td>France</td>\n",
+       "      <td>For simple Chablis, this is impressive, rich, ...</td>\n",
+       "      <td>Domaine</td>\n",
+       "      <td>89</td>\n",
+       "      <td>26.0</td>\n",
+       "      <td>Burgundy</td>\n",
+       "      <td>Chablis</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>Roger Voss</td>\n",
+       "      <td>@vossroger</td>\n",
+       "      <td>William Fèvre 2005 Domaine  (Chablis)</td>\n",
+       "      <td>Chardonnay</td>\n",
+       "      <td>William Fèvre</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>71277</th>\n",
+       "      <td>71277</td>\n",
+       "      <td>France</td>\n",
+       "      <td>This 50-50 blend of Marselan and Merlot opens ...</td>\n",
+       "      <td>La Remise</td>\n",
+       "      <td>84</td>\n",
+       "      <td>13.0</td>\n",
+       "      <td>France Other</td>\n",
+       "      <td>Vin de France</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>Lauren Buzzeo</td>\n",
+       "      <td>@laurbuzz</td>\n",
+       "      <td>Domaine de la Mordorée 2014 La Remise Red (Vin...</td>\n",
+       "      <td>Red Blend</td>\n",
+       "      <td>Domaine de la Mordorée</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>27484</th>\n",
+       "      <td>27484</td>\n",
+       "      <td>France</td>\n",
+       "      <td>The medium-intense nose of this solid and easy...</td>\n",
+       "      <td>Authentic &amp; Chic</td>\n",
+       "      <td>86</td>\n",
+       "      <td>10.0</td>\n",
+       "      <td>France Other</td>\n",
+       "      <td>Vin de France</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>Lauren Buzzeo</td>\n",
+       "      <td>@laurbuzz</td>\n",
+       "      <td>Romantic 2014 Authentic &amp; Chic Cabernet Sauvig...</td>\n",
+       "      <td>Cabernet Sauvignon</td>\n",
+       "      <td>Romantic</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>124917</th>\n",
+       "      <td>124917</td>\n",
+       "      <td>France</td>\n",
+       "      <td>Fresh, pure notes of Conference pear peel enti...</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>89</td>\n",
+       "      <td>30.0</td>\n",
+       "      <td>Alsace</td>\n",
+       "      <td>Alsace</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>Anne Krebiehl MW</td>\n",
+       "      <td>@AnneInVino</td>\n",
+       "      <td>Domaine Vincent Stoeffler 2015 Pinot Gris (Als...</td>\n",
+       "      <td>Pinot Gris</td>\n",
+       "      <td>Domaine Vincent Stoeffler</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "        Unnamed: 0 country                                        description  \\\n",
+       "95206        95206  France  Full, fat, ripe, perfumed wine that is full of...   \n",
+       "66403        66403  France  For simple Chablis, this is impressive, rich, ...   \n",
+       "71277        71277  France  This 50-50 blend of Marselan and Merlot opens ...   \n",
+       "27484        27484  France  The medium-intense nose of this solid and easy...   \n",
+       "124917      124917  France  Fresh, pure notes of Conference pear peel enti...   \n",
+       "\n",
+       "                          designation  points  price      province  \\\n",
+       "95206   Château de Mercey Premier Cru      91   35.0      Burgundy   \n",
+       "66403                         Domaine      89   26.0      Burgundy   \n",
+       "71277                       La Remise      84   13.0  France Other   \n",
+       "27484                Authentic & Chic      86   10.0  France Other   \n",
+       "124917                            NaN      89   30.0        Alsace   \n",
+       "\n",
+       "             region_1 region_2       taster_name taster_twitter_handle  \\\n",
+       "95206        Mercurey      NaN        Roger Voss            @vossroger   \n",
+       "66403         Chablis      NaN        Roger Voss            @vossroger   \n",
+       "71277   Vin de France      NaN     Lauren Buzzeo             @laurbuzz   \n",
+       "27484   Vin de France      NaN     Lauren Buzzeo             @laurbuzz   \n",
+       "124917         Alsace      NaN  Anne Krebiehl MW           @AnneInVino   \n",
+       "\n",
+       "                                                    title             variety  \\\n",
+       "95206   Antonin Rodet 2010 Château de Mercey Premier C...          Pinot Noir   \n",
+       "66403               William Fèvre 2005 Domaine  (Chablis)          Chardonnay   \n",
+       "71277   Domaine de la Mordorée 2014 La Remise Red (Vin...           Red Blend   \n",
+       "27484   Romantic 2014 Authentic & Chic Cabernet Sauvig...  Cabernet Sauvignon   \n",
+       "124917  Domaine Vincent Stoeffler 2015 Pinot Gris (Als...          Pinot Gris   \n",
+       "\n",
+       "                           winery  \n",
+       "95206               Antonin Rodet  \n",
+       "66403               William Fèvre  \n",
+       "71277      Domaine de la Mordorée  \n",
+       "27484                    Romantic  \n",
+       "124917  Domaine Vincent Stoeffler  "
+      ]
+     },
+     "execution_count": 10,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "df = pd.read_csv('data/winemag/winemag-data-130k-v2.csv')\n",
+    "df_france = df[df['country'] == 'France']\n",
+    "\n",
+    "# Let's also filter out wines that have less than 5 references with their grape variety – even though we'd like to find those\n",
+    "# they're outliers that we don't want to optimize for that would make our enum list be too long\n",
+    "# and they could also add noise for the rest of the dataset on which we'd like to guess, eventually reducing our accuracy.\n",
+    "\n",
+    "varieties_less_than_five_list = df_france['variety'].value_counts()[df_france['variety'].value_counts() < 5].index.tolist()\n",
+    "df_france = df_france[~df_france['variety'].isin(varieties_less_than_five_list)]\n",
+    "\n",
+    "df_france_subset = df_france.sample(n=500)\n",
+    "df_france_subset.head()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b96cd12f-cbdf-46af-958f-3d553598be1d",
+   "metadata": {},
+   "source": [
+    "Let's retrieve all grape varieties to include them in the prompt and in our structured outputs enum list."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "06f5dbea-549a-455d-9b6e-051de9d38723",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array(['Gewürztraminer', 'Pinot Gris', 'Gamay',\n",
+       "       'Bordeaux-style White Blend', 'Champagne Blend', 'Chardonnay',\n",
+       "       'Petit Manseng', 'Riesling', 'White Blend', 'Pinot Blanc',\n",
+       "       'Alsace white blend', 'Bordeaux-style Red Blend', 'Malbec',\n",
+       "       'Tannat-Cabernet', 'Rhône-style Red Blend', 'Ugni Blanc-Colombard',\n",
+       "       'Savagnin', 'Pinot Noir', 'Rosé', 'Melon',\n",
+       "       'Rhône-style White Blend', 'Pinot Noir-Gamay', 'Colombard',\n",
+       "       'Chenin Blanc', 'Sylvaner', 'Sauvignon Blanc', 'Red Blend',\n",
+       "       'Chenin Blanc-Chardonnay', 'Cabernet Sauvignon', 'Cabernet Franc',\n",
+       "       'Syrah', 'Sparkling Blend', 'Duras', 'Provence red blend',\n",
+       "       'Tannat', 'Merlot', 'Malbec-Merlot', 'Chardonnay-Viognier',\n",
+       "       'Cabernet Franc-Cabernet Sauvignon', 'Muscat', 'Viognier',\n",
+       "       'Picpoul', 'Altesse', 'Provence white blend', 'Mondeuse',\n",
+       "       'Grenache-Syrah', 'G-S-M', 'Pinot Meunier', 'Cabernet-Syrah',\n",
+       "       'Vermentino', 'Marsanne', 'Colombard-Sauvignon Blanc',\n",
+       "       'Gros and Petit Manseng', 'Jacquère', 'Negrette', 'Mauzac',\n",
+       "       'Pinot Auxerrois', 'Grenache', 'Roussanne', 'Gros Manseng',\n",
+       "       'Tannat-Merlot', 'Aligoté', 'Chasselas', \"Loin de l'Oeil\",\n",
+       "       'Malbec-Tannat', 'Carignan', 'Colombard-Ugni Blanc', 'Sémillon',\n",
+       "       'Syrah-Grenache', 'Sciaccerellu', 'Auxerrois', 'Mourvèdre',\n",
+       "       'Tannat-Cabernet Franc', 'Braucol', 'Trousseau',\n",
+       "       'Merlot-Cabernet Sauvignon'], dtype='<U33')"
+      ]
+     },
+     "execution_count": 11,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "varieties = np.array(df_france['variety'].unique()).astype('str')\n",
+    "varieties"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6612e612-698d-4f1a-8008-70fb5ff263a0",
+   "metadata": {},
+   "source": [
+    "## Generating the prompt\n",
+    "\n",
+    "Let's build out a function to generate our prompt and try it for the first wine of our list."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "c3ec2fba-9c99-4cb7-bf56-f13e3816e559",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'\\n    Based on this wine review, guess the grape variety:\\n    This wine is produced by Trimbach in the Alsace region of France.\\n    It was grown in Alsace. It is described as: \"This dry and restrained wine offers spice in profusion. Balanced with acidity and a firm texture, it\\'s very much for food.\".\\n    The wine has been reviewed by Roger Voss and received 87 points.\\n    The price is 24.0.\\n\\n    Here is a list of possible grape varieties to choose from: Gewürztraminer, Pinot Gris, Gamay, Bordeaux-style White Blend, Champagne Blend, Chardonnay, Petit Manseng, Riesling, White Blend, Pinot Blanc, Alsace white blend, Bordeaux-style Red Blend, Malbec, Tannat-Cabernet, Rhône-style Red Blend, Ugni Blanc-Colombard, Savagnin, Pinot Noir, Rosé, Melon, Rhône-style White Blend, Pinot Noir-Gamay, Colombard, Chenin Blanc, Sylvaner, Sauvignon Blanc, Red Blend, Chenin Blanc-Chardonnay, Cabernet Sauvignon, Cabernet Franc, Syrah, Sparkling Blend, Duras, Provence red blend, Tannat, Merlot, Malbec-Merlot, Chardonnay-Viognier, Cabernet Franc-Cabernet Sauvignon, Muscat, Viognier, Picpoul, Altesse, Provence white blend, Mondeuse, Grenache-Syrah, G-S-M, Pinot Meunier, Cabernet-Syrah, Vermentino, Marsanne, Colombard-Sauvignon Blanc, Gros and Petit Manseng, Jacquère, Negrette, Mauzac, Pinot Auxerrois, Grenache, Roussanne, Gros Manseng, Tannat-Merlot, Aligoté, Chasselas, Loin de l\\'Oeil, Malbec-Tannat, Carignan, Colombard-Ugni Blanc, Sémillon, Syrah-Grenache, Sciaccerellu, Auxerrois, Mourvèdre, Tannat-Cabernet Franc, Braucol, Trousseau, Merlot-Cabernet Sauvignon.\\n    \\n    What is the likely grape variety? Answer only with the grape variety name or blend from the list.\\n    '"
+      ]
+     },
+     "execution_count": 12,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "def generate_prompt(row, varieties):\n",
+    "    # Format the varieties list as a comma-separated string\n",
+    "    variety_list = ', '.join(varieties)\n",
+    "    \n",
+    "    prompt = f\"\"\"\n",
+    "    Based on this wine review, guess the grape variety:\n",
+    "    This wine is produced by {row['winery']} in the {row['province']} region of {row['country']}.\n",
+    "    It was grown in {row['region_1']}. It is described as: \"{row['description']}\".\n",
+    "    The wine has been reviewed by {row['taster_name']} and received {row['points']} points.\n",
+    "    The price is {row['price']}.\n",
+    "\n",
+    "    Here is a list of possible grape varieties to choose from: {variety_list}.\n",
+    "    \n",
+    "    What is the likely grape variety? Answer only with the grape variety name or blend from the list.\n",
+    "    \"\"\"\n",
+    "    return prompt\n",
+    "\n",
+    "# Example usage with a specific row\n",
+    "prompt = generate_prompt(df_france.iloc[0], varieties)\n",
+    "prompt"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fdf208f5-0a70-41cb-b2c0-c340ee55d6d1",
+   "metadata": {},
+   "source": [
+    "To get a understanding of the cost before running the queries, you can leverage tiktoken to understand the number of tokens we'll send and the cost associated to run this. This will only give you an estimate for to run the completions, not the fine-tuning process (used later in this cookbook when running the distillation), which depends on other factors such as the number of epochs, training set etc."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "cf43b5f4-dbfb-4c89-a7f8-2bd7b03c49c7",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Total number of tokens in the dataset: 245439\n",
+      "Total number of prompts: 500\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Load encoding for the GPT-4 model\n",
+    "enc = tiktoken.encoding_for_model(\"gpt-4o\")\n",
+    "\n",
+    "# Initialize a variable to store the total number of tokens\n",
+    "total_tokens = 0\n",
+    "\n",
+    "for index, row in df_france_subset.iterrows():\n",
+    "    prompt = generate_prompt(row, varieties)\n",
+    "    \n",
+    "    # Tokenize the input text and count tokens\n",
+    "    tokens = enc.encode(prompt)\n",
+    "    token_count = len(tokens)\n",
+    "    \n",
+    "    # Add the token count to the total\n",
+    "    total_tokens += token_count\n",
+    "\n",
+    "print(f\"Total number of tokens in the dataset: {total_tokens}\")\n",
+    "print(f\"Total number of prompts: {len(df_france_subset)}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "362e1ddd-2a87-433e-bcba-534ae946c2ff",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "0.6135975\n",
+      "0.03681585\n"
+     ]
+    }
+   ],
+   "source": [
+    "# outputing cost in $ as of 2024/10/16\n",
+    "\n",
+    "gpt4o_token_price = 2.50 / 1_000_000  # $2.50 per 1M tokens\n",
+    "gpt4o_mini_token_price = 0.150 / 1_000_000  # $0.15 per 1M tokens\n",
+    "\n",
+    "total_gpt4o_cost = gpt4o_token_price*total_tokens\n",
+    "total_gpt4o_mini_cost = gpt4o_mini_token_price*total_tokens\n",
+    "\n",
+    "print(total_gpt4o_cost)\n",
+    "print(total_gpt4o_mini_cost)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6ff1181a-6b06-4ad3-8d68-7afb64518010",
+   "metadata": {},
+   "source": [
+    "## Preparing functions to Store Completions\n",
+    "\n",
+    "As we're looking at a limited list of response (enumerate list of grape varieties), let's leverage structured outputs so we make sure the model will answer from this list. This also allows us to compare the model's answer directly with the grape variety and have a deterministic answer (compared to a model that could answer \"I think the grape is Pinot Noir\" instead of just \"Pinot noir\"), on top of improving the performance to avoid grape varieties not in our dataset.\n",
+    "\n",
+    "If you want to know more on Structured Outputs you can read this [cookbook](https://cookbook.openai.com/examples/structured_outputs_intro) and this [documentation guide](https://platform.openai.com/docs/guides/structured-outputs/introduction)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "id": "4153ca1a-c195-4102-a6c3-f570dd6ae372",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response_format = {\n",
+    "    \"type\": \"json_schema\",\n",
+    "    \"json_schema\": {\n",
+    "        \"name\": \"grape-variety\",\n",
+    "        \"schema\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"variety\": {\n",
+    "                    \"type\": \"string\",\n",
+    "                    \"enum\": varieties.tolist()\n",
+    "                }\n",
+    "            },\n",
+    "            \"additionalProperties\": False,\n",
+    "            \"required\": [\"variety\"],\n",
+    "        },\n",
+    "        \"strict\": True\n",
+    "    }\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0fedb40e-72e5-4aba-8bc2-b244111b4b43",
+   "metadata": {},
+   "source": [
+    "To distill a model, you need to store all completions from a model, allowing you to give it as a reference to the smaller model to fine-tune it. We're therefore adding a `store=True` parameter to our `client.chat.completions.create` method so we can store those completions from gpt-4o.\n",
+    "\n",
+    "We're going to store all completions (even 4o-mini and our future fine-tuned model) so we are able to run [Evals](https://platform.openai.com/docs/guides/evals) from OpenAI platform directly.\n",
+    "\n",
+    "When storing those completions, it's useful to store them with a metadata tag, that will allow filtering from the OpenAI platform to run distillation & evals on the specific set of completions you'd like to run those."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "id": "85cb4cc7-077a-4afe-aefc-85b768adfc1f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Initialize the progress index\n",
+    "metadata_value = \"wine-distillation\" # that's a funny metadata tag :-)\n",
+    "\n",
+    "# Function to call the API and process the result for a single model (blocking call in this case)\n",
+    "def call_model(model, prompt):\n",
+    "    response = client.chat.completions.create(\n",
+    "        model=model,\n",
+    "        store=True,\n",
+    "        metadata={\n",
+    "            \"distillation\": metadata_value,\n",
+    "        },\n",
+    "        messages=[\n",
+    "            {\n",
+    "                \"role\": \"system\",\n",
+    "                \"content\": \"You're a sommelier expert and you know everything about wine. You answer precisely with the name of the variety/blend.\"\n",
+    "            },\n",
+    "            {\n",
+    "                \"role\": \"user\",\n",
+    "                \"content\": prompt\n",
+    "            }\n",
+    "        ],\n",
+    "         response_format=response_format\n",
+    "    )\n",
+    "    return json.loads(response.choices[0].message.content.strip())['variety']"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cd4ae8e1-20cb-43b4-a7b1-9998e09f4394",
+   "metadata": {},
+   "source": [
+    "## Parallel processing\n",
+    "\n",
+    "As we'll run this on a large number of rows, let's make sure we run those completions in parallel and use concurrent futures for this. We'll iterate on our dataframe and output progress every 20 rows. We'll store the completion from the model we run the completion for in the same dataframe using the column name `{model}-variety`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "id": "81ddec71-c2a6-411a-aecd-6556cddc36c8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def process_example(index, row, model, df, progress_bar):\n",
+    "    global progress_index\n",
+    "\n",
+    "    try:\n",
+    "        # Generate the prompt using the row\n",
+    "        prompt = generate_prompt(row, varieties)\n",
+    "\n",
+    "        df.at[index, model + \"-variety\"] = call_model(model, prompt)\n",
+    "        \n",
+    "        # Update the progress bar\n",
+    "        progress_bar.update(1)\n",
+    "        \n",
+    "        progress_index += 1\n",
+    "    except Exception as e:\n",
+    "        print(f\"Error processing model {model}: {str(e)}\")\n",
+    "\n",
+    "def process_dataframe(df, model):\n",
+    "    global progress_index\n",
+    "    progress_index = 1  # Reset progress index\n",
+    "\n",
+    "    # Create a tqdm progress bar\n",
+    "    with tqdm(total=len(df), desc=\"Processing rows\") as progress_bar:\n",
+    "        # Process each example concurrently using ThreadPoolExecutor\n",
+    "        with concurrent.futures.ThreadPoolExecutor() as executor:\n",
+    "            futures = {executor.submit(process_example, index, row, model, df, progress_bar): index for index, row in df.iterrows()}\n",
+    "            \n",
+    "            for future in concurrent.futures.as_completed(futures):\n",
+    "                try:\n",
+    "                    future.result()  # Wait for each example to be processed\n",
+    "                except Exception as e:\n",
+    "                    print(f\"Error processing example: {str(e)}\")\n",
+    "\n",
+    "    return df"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "680587d1-bcb8-45bd-bfe8-1f771ae1a5bd",
+   "metadata": {},
+   "source": [
+    "Let's try out our call model function before processing the whole dataframe and check the output."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "id": "95915fd2-24f6-4908-a54f-9ce59073233d",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'Pinot Noir'"
+      ]
+     },
+     "execution_count": 18,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "answer = call_model('gpt-4o', generate_prompt(df_france_subset.iloc[0], varieties))\n",
+    "answer"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "941b9027-eb22-4eed-84de-8b962fe4fee2",
+   "metadata": {},
+   "source": [
+    "Great! We confirmed we can get a grape variety as an output, let's now process the dataset with both `gpt-4o` and `gpt-4o-mini` and compare the results. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "id": "1cedf978-e60b-4f56-9c8c-da6b0a722320",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Processing rows: 100%|███████████████████████████████████████████████| 500/500 [00:41<00:00, 12.09it/s]\n"
+     ]
+    }
+   ],
+   "source": [
+    "df_france_subset = process_dataframe(df_france_subset, \"gpt-4o\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "id": "93a96539-e25d-4d15-adc0-ef8cbacd70e9",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Processing rows: 100%|███████████████████████████████████████████████| 500/500 [01:31<00:00,  5.45it/s]\n"
+     ]
+    }
+   ],
+   "source": [
+    "df_france_subset = process_dataframe(df_france_subset, \"gpt-4o-mini\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f74869ee-3361-4d08-bd46-126239d6008c",
+   "metadata": {},
+   "source": [
+    "## Comparing gpt-4o and gpt-4o-mini\n",
+    "\n",
+    "Now that we've got all chat completions for those two models ; let's compare them against the expected grape variety and assess their accuracy at finding it. We'll do this directly in python here as we've got a simple string check to run, but if your task involves more complex evals you can leverage OpenAI Evals or our open-source eval framework."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "id": "cce147d2-32e4-4005-9f79-11394c76c2e4",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "gpt-4o accuracy: 81.80%\n",
+      "gpt-4o-mini accuracy: 69.00%\n"
+     ]
+    }
+   ],
+   "source": [
+    "models = ['gpt-4o', 'gpt-4o-mini']\n",
+    "\n",
+    "def get_accuracy(model, df):\n",
+    "    return np.mean(df['variety'] == df[model + '-variety'])\n",
+    "\n",
+    "for model in models:\n",
+    "    print(f\"{model} accuracy: {get_accuracy(model, df_france_subset) * 100:.2f}%\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0a6ae1c1-22e5-4a61-a055-7b98360de523",
+   "metadata": {},
+   "source": [
+    "We can see that gpt-4o is better a finding grape variety than 4o-mini (12.80% higher or almost 20% relatively to 4o-mini!). Now I'm wondering if we're making gpt-4o drink wine during training!\n",
+    "\n",
+    "## Distilling gpt-4o outputs to gpt-4o-mini\n",
+    "\n",
+    "Let's assume we'd like to run this prediction often, we want completions to be faster and cheaper, but keep that level of accuracy. That'd be great to be able to distill 4o accuracy to 4o-mini, wouldn't it? Let's do it!\n",
+    "\n",
+    "We'll now go to OpenAI Stored completions page: [https://platform.openai.com/chat-completions](https://platform.openai.com/chat-completions).\n",
+    "\n",
+    "Let's select the model gpt-4o (make sure to do this, you don't want to distill the outputs of 4o-mini that we ran). Let's also select the metadata `distillation: wine-distillation` to get only stored completions ran from this cookbook.\n",
+    "\n",
+    "![Filtering out completions](../images/filtering-out-completions.png)\n",
+    "\n",
+    "Once you've selected completions, you can click on \"Distill\" on the top right corner to fine-tune a model based on those completions. Once we've done that, a file to run the fine-tuning process will automatically be created. Let's then select `gpt-4o-mini` as the base model, keep the default parameters (but you're free to change them or iterate with it to improve performance).\n",
+    "\n",
+    "![Distilling modal](../images/distilling.png)\n",
+    "\n",
+    "Once the fine-tuning job is starting, you can retrieve the fine tuning job ID from the fine-tuning page, we'll use it to monitor status of the fine-tuned job as well as retrieving the fine-tuned model id once done.\n",
+    "\n",
+    "![Fine tuning job](../images/fine-tuning-job.png)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "id": "78a1b24e-3956-48f0-ae56-399f729a6c24",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "finetuned model: ft:gpt-4o-mini-2024-07-18:distillation-test:wine-distillation:AIZntSyE\n"
+     ]
+    }
+   ],
+   "source": [
+    "# copy paste your fine-tune job ID below\n",
+    "finetune_job = client.fine_tuning.jobs.retrieve(\"ftjob-pRyNWzUItmHpxmJ1TX7FOaWe\")\n",
+    "\n",
+    "if finetune_job.status == 'succeeded':\n",
+    "    fine_tuned_model = finetune_job.fine_tuned_model\n",
+    "    print('finetuned model: ' + fine_tuned_model)\n",
+    "else:\n",
+    "    print('finetuned job status: ' + finetune_job.status)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "54eda7a4-3817-40df-84bc-db4555366078",
+   "metadata": {},
+   "source": [
+    "## Running completions for the distilled model\n",
+    "\n",
+    "Now that we've got our model fine-tuned, we can use this model to run completions and compare accuracy with both gpt4o and gpt4o-mini.\n",
+    "Let's grab a different subset of french wines (as we restricted the outputs to french grape varieties, without outliers, we'll need to focus our validation dataset to this too). Let's run this on 300 entries for each models."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "id": "603dc1e9-58dc-4b01-b25a-4433ebd88546",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Processing rows: 100%|███████████████████████████████████████████████| 300/300 [00:20<00:00, 14.69it/s]\n",
+      "Processing rows: 100%|███████████████████████████████████████████████| 300/300 [00:27<00:00, 10.99it/s]\n",
+      "Processing rows: 100%|███████████████████████████████████████████████| 300/300 [00:37<00:00,  8.08it/s]\n"
+     ]
+    }
+   ],
+   "source": [
+    "validation_dataset = df_france.sample(n=300)\n",
+    "\n",
+    "models.append(fine_tuned_model)\n",
+    "\n",
+    "for model in models:\n",
+    "    another_subset = process_dataframe(validation_dataset, model)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6eba9c5d-d805-4610-86fe-e18cc20b8cd9",
+   "metadata": {},
+   "source": [
+    "Let's compare accuracy of models"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 24,
+   "id": "cad42199-a4cb-4589-859b-32b39ad02eb2",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "gpt-4o accuracy: 79.67%\n",
+      "gpt-4o-mini accuracy: 64.67%\n",
+      "ft:gpt-4o-mini-2024-07-18:distillation-test:wine-distillation:AIZntSyE accuracy: 79.33%\n"
+     ]
+    }
+   ],
+   "source": [
+    "for model in models:\n",
+    "    print(f\"{model} accuracy: {get_accuracy(model, another_subset) * 100:.2f}%\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2b5bdb3c-c7ee-493a-9343-5950d7c36638",
+   "metadata": {},
+   "source": [
+    "That's almost a **22% relative improvement over the non-distilled gpt-4o-mini! 🎉**\n",
+    "\n",
+    "Our fine-tuned model performs way better than gpt-4o-mini, while having the same base model. We'll be able to use this model to run inferences at a lower cost and lower latency for future grape variety prediction."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

examples/Named_Entity_Recognition_to_enrich_text.ipynb

@@ -0,0 +1,607 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Named Entity Recognition (NER) to Enrich Text"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "`Named Entity Recognition` (NER) is a `Natural Language Processing` task that identifies and classifies named entities (NE) into predefined semantic categories (such as persons, organizations, locations, events, time expressions, and quantities). By converting raw text into structured information, NER makes data more actionable, facilitating tasks like information extraction, data aggregation, analytics, and social media monitoring.\n",
+    "\n",
+    "This notebook demonstrates how to carry out NER with [chat completion](https://platform.openai.com/docs/api-reference/chat) and [functions-calling](https://platform.openai.com/docs/guides/gpt/function-calling) to enrich a text with links to a knowledge base such as Wikipedia:\n",
+    "\n",
+    "**Text:**\n",
+    "\n",
+    "*In Germany, in 1440, goldsmith Johannes Gutenberg invented the movable-type printing press. His work led to an information revolution and the unprecedented mass-spread of literature throughout Europe. Modelled on the design of the existing screw presses, a single Renaissance movable-type printing press could produce up to 3,600 pages per workday.*\n",
+    "\n",
+    "**Text enriched with Wikipedia links:**\n",
+    "\n",
+    "*In [Germany](https://en.wikipedia.org/wiki/Germany), in 1440, goldsmith [Johannes Gutenberg]() invented the [movable-type printing press](https://en.wikipedia.org/wiki/Movable_Type). His work led to an [information revolution](https://en.wikipedia.org/wiki/Information_revolution) and the unprecedented mass-spread of literature throughout [Europe](https://en.wikipedia.org/wiki/Europe). Modelled on the design of the existing screw presses, a single [Renaissance](https://en.wikipedia.org/wiki/Renaissance) [movable-type printing press](https://en.wikipedia.org/wiki/Movable_Type) could produce up to 3,600 pages per workday.*\n",
+    "\n",
+    "**Inference Costs:** The notebook also illustrates how to estimate OpenAI API costs."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 1. Setup"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### 1.1 Install/Upgrade Python packages"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Note: you may need to restart the kernel to use updated packages.\n",
+      "Note: you may need to restart the kernel to use updated packages.\n",
+      "Note: you may need to restart the kernel to use updated packages.\n"
+     ]
+    }
+   ],
+   "source": [
+    "%pip install --upgrade openai --quiet\n",
+    "%pip install --upgrade nlpia2-wikipedia --quiet\n",
+    "%pip install --upgrade tenacity --quiet"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### 1.2 Load packages and OPENAI_API_KEY"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can generate an API key in the OpenAI web interface. See https://platform.openai.com/account/api-keys for details."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This notebook works with the latest OpeanAI models `gpt-3.5-turbo-0613` and `gpt-4-0613`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "import logging\n",
+    "import os\n",
+    "\n",
+    "import openai\n",
+    "import wikipedia\n",
+    "\n",
+    "from typing import Optional\n",
+    "from IPython.display import display, Markdown\n",
+    "from tenacity import retry, wait_random_exponential, stop_after_attempt\n",
+    "\n",
+    "logging.basicConfig(level=logging.INFO, format=' %(asctime)s - %(levelname)s - %(message)s')\n",
+    "\n",
+    "OPENAI_MODEL = 'gpt-3.5-turbo-0613'\n",
+    "\n",
+    "client = openai.OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 2. Define the NER labels to be Identified"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We define a standard set of NER labels to showcase a wide range of use cases. However, for our specific task of enriching text with knowledge base links, only a subset is practically required."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "labels = [\n",
+    "    \"person\",      # people, including fictional characters\n",
+    "    \"fac\",         # buildings, airports, highways, bridges\n",
+    "    \"org\",         # organizations, companies, agencies, institutions\n",
+    "    \"gpe\",         # geopolitical entities like countries, cities, states\n",
+    "    \"loc\",         # non-gpe locations\n",
+    "    \"product\",     # vehicles, foods, appareal, appliances, software, toys \n",
+    "    \"event\",       # named sports, scientific milestones, historical events\n",
+    "    \"work_of_art\", # titles of books, songs, movies\n",
+    "    \"law\",         # named laws, acts, or legislations\n",
+    "    \"language\",    # any named language\n",
+    "    \"date\",        # absolute or relative dates or periods\n",
+    "    \"time\",        # time units smaller than a day\n",
+    "    \"percent\",     # percentage (e.g., \"twenty percent\", \"18%\")\n",
+    "    \"money\",       # monetary values, including unit\n",
+    "    \"quantity\",    # measurements, e.g., weight or distance\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 3. Prepare messages"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The [chat completions API](https://platform.openai.com/docs/guides/gpt/chat-completions-api) takes a list of messages as input and delivers a model-generated message as an output. While the chat format is primarily designed for facilitating multi-turn conversations, it is equally efficient for single-turn tasks without any preceding conversation. For our purposes, we will specify a message for the system, assistant, and user roles."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### 3.1 System Message"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The `system message` (prompt) sets the assistant's behavior by defining its desired persona and task. We also delineate the specific set of entity labels we aim to identify."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Although one can instruct the model to format its response, it has to be noted that both `gpt-3.5-turbo-0613` and `gpt-4-0613` have been fine-tuned to discern when a function should be invoked, and to reply with `JSON` formatted according to the function's signature. This capability streamlines our prompt and enables us to receive structured data directly from the model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def system_message(labels):\n",
+    "    return f\"\"\"\n",
+    "You are an expert in Natural Language Processing. Your task is to identify common Named Entities (NER) in a given text.\n",
+    "The possible common Named Entities (NER) types are exclusively: ({\", \".join(labels)}).\"\"\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### 3.2 Assistant Message"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "`Assistant messages` usually store previous assistant responses. However, as in our scenario, they can also be crafted to provide examples of the desired behavior. While OpenAI is able to execute `zero-shot` Named Entity Recognition, we have found that a `one-shot` approach produces more precise results."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def assisstant_message():\n",
+    "    return f\"\"\"\n",
+    "EXAMPLE:\n",
+    "    Text: 'In Germany, in 1440, goldsmith Johannes Gutenberg invented the movable-type printing press. His work led to an information revolution and the unprecedented mass-spread / \n",
+    "    of literature throughout Europe. Modelled on the design of the existing screw presses, a single Renaissance movable-type printing press could produce up to 3,600 pages per workday.'\n",
+    "    {{\n",
+    "        \"gpe\": [\"Germany\", \"Europe\"],\n",
+    "        \"date\": [\"1440\"],\n",
+    "        \"person\": [\"Johannes Gutenberg\"],\n",
+    "        \"product\": [\"movable-type printing press\"],\n",
+    "        \"event\": [\"Renaissance\"],\n",
+    "        \"quantity\": [\"3,600 pages\"],\n",
+    "        \"time\": [\"workday\"]\n",
+    "    }}\n",
+    "--\"\"\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### 3.3 User Message"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The `user message` provides the specific text for the assistant task:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def user_message(text):\n",
+    "    return f\"\"\"\n",
+    "TASK:\n",
+    "    Text: {text}\n",
+    "\"\"\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 4. OpenAI Functions (and Utils)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In an OpenAI API call, we can describe `functions` to `gpt-3.5-turbo-0613` and `gpt-4-0613` and have the model intelligently choose to output a `JSON` object containing arguments to call those `functions`. It's important to note that the [chat completions API](https://platform.openai.com/docs/guides/gpt/chat-completions-api) doesn't actually execute the `function`. Instead, it provides the `JSON` output, which can then be used to call the `function` in our code. For more details, refer to the [OpenAI Function Calling Guide](https://platform.openai.com/docs/guides/function-calling)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Our function, `enrich_entities(text, label_entities)` gets a block of text and a dictionary containing identified labels and entities as parameters. It then associates the recognized entities with their corresponding links to the Wikipedia articles."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "@retry(wait=wait_random_exponential(min=1, max=10), stop=stop_after_attempt(5))\n",
+    "def find_link(entity: str) -> Optional[str]:\n",
+    "    \"\"\"\n",
+    "    Finds a Wikipedia link for a given entity.\n",
+    "    \"\"\"\n",
+    "    try:\n",
+    "        titles = wikipedia.search(entity)\n",
+    "        if titles:\n",
+    "            # naively consider the first result as the best\n",
+    "            page = wikipedia.page(titles[0])\n",
+    "            return page.url\n",
+    "    except (wikipedia.exceptions.WikipediaException) as ex:\n",
+    "        logging.error(f'Error occurred while searching for Wikipedia link for entity {entity}: {str(ex)}')\n",
+    "\n",
+    "    return None"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def find_all_links(label_entities:dict) -> dict:\n",
+    "    \"\"\" \n",
+    "    Finds all Wikipedia links for the dictionary entities in the whitelist label list.\n",
+    "    \"\"\"\n",
+    "    whitelist = ['event', 'gpe', 'org', 'person', 'product', 'work_of_art']\n",
+    "    \n",
+    "    return {e: find_link(e) for label, entities in label_entities.items() \n",
+    "                            for e in entities\n",
+    "                            if label in whitelist}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def enrich_entities(text: str, label_entities: dict) -> str:\n",
+    "    \"\"\"\n",
+    "    Enriches text with knowledge base links.\n",
+    "    \"\"\"\n",
+    "    entity_link_dict = find_all_links(label_entities)\n",
+    "    logging.info(f\"entity_link_dict: {entity_link_dict}\")\n",
+    "    \n",
+    "    for entity, link in entity_link_dict.items():\n",
+    "        text = text.replace(entity, f\"[{entity}]({link})\")\n",
+    "\n",
+    "    return text"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 4. ChatCompletion"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "As previously highlighted, `gpt-3.5-turbo-0613` and `gpt-4-0613` have been fine-tuned to detect when a `function` should to be called. Moreover, they can produce a `JSON` response that conforms to the `function` signature. Here's the sequence we follow:\n",
+    "\n",
+    "1. Define our `function` and its associated `JSON` Schema.\n",
+    "2. Invoke the model using the `messages`, `tools` and `tool_choice` parameters.\n",
+    "3. Convert the output into a `JSON` object, and then call the `function` with the `arguments` provided by the model.\n",
+    "\n",
+    "In practice, one might want to re-invoke the model again by appending the `function` response as a new message, and let the model summarize the results back to the user. Nevertheless, for our purposes, this step is not needed.\n",
+    "\n",
+    "*Note that in a real-case scenario it is strongly recommended to build in user confirmation flows before taking actions.*"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### 4.1 Define our Function and JSON schema"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Since we want the model to output a dictionary of labels and recognized entities:\n",
+    "\n",
+    "```python\n",
+    "{   \n",
+    "    \"gpe\": [\"Germany\", \"Europe\"],   \n",
+    "    \"date\": [\"1440\"],   \n",
+    "    \"person\": [\"Johannes Gutenberg\"],   \n",
+    "    \"product\": [\"movable-type printing press\"],   \n",
+    "    \"event\": [\"Renaissance\"],   \n",
+    "    \"quantity\": [\"3,600 pages\"],   \n",
+    "    \"time\": [\"workday\"]   \n",
+    "}   \n",
+    "```\n",
+    "we need to define the corresponding `JSON` schema to be passed to the `tools` parameter: "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def generate_functions(labels: dict) -> list:\n",
+    "    return [\n",
+    "        {   \n",
+    "            \"type\": \"function\",\n",
+    "            \"function\": {\n",
+    "                \"name\": \"enrich_entities\",\n",
+    "                \"description\": \"Enrich Text with Knowledge Base Links\",\n",
+    "                \"parameters\": {\n",
+    "                    \"type\": \"object\",\n",
+    "                        \"properties\": {\n",
+    "                            \"r'^(?:' + '|'.join({labels}) + ')$'\": \n",
+    "                            {\n",
+    "                                \"type\": \"array\",\n",
+    "                                \"items\": {\n",
+    "                                    \"type\": \"string\"\n",
+    "                                }\n",
+    "                            }\n",
+    "                        },\n",
+    "                        \"additionalProperties\": False\n",
+    "                },\n",
+    "            }\n",
+    "        }\n",
+    "    ]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### 4.2 Chat Completion"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now, we invoke the model. It's important to note that we direct the API to use a specific function by setting the `tool_choice` parameter to `{\"type\": \"function\", \"function\" : {\"name\": \"enrich_entities\"}}`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "@retry(wait=wait_random_exponential(min=1, max=10), stop=stop_after_attempt(5))\n",
+    "def run_openai_task(labels, text):\n",
+    "    messages = [\n",
+    "          {\"role\": \"system\", \"content\": system_message(labels=labels)},\n",
+    "          {\"role\": \"assistant\", \"content\": assisstant_message()},\n",
+    "          {\"role\": \"user\", \"content\": user_message(text=text)}\n",
+    "      ]\n",
+    "\n",
+    "    # TODO: functions and function_call are deprecated, need to be updated\n",
+    "    # See: https://platform.openai.com/docs/api-reference/chat/create#chat-create-tools\n",
+    "    response = openai.chat.completions.create(\n",
+    "        model=\"gpt-3.5-turbo-0613\",\n",
+    "        messages=messages,\n",
+    "        tools=generate_functions(labels),\n",
+    "        tool_choice={\"type\": \"function\", \"function\" : {\"name\": \"enrich_entities\"}}, \n",
+    "        temperature=0,\n",
+    "        frequency_penalty=0,\n",
+    "        presence_penalty=0,\n",
+    "    )\n",
+    "\n",
+    "    response_message = response.choices[0].message\n",
+    "    \n",
+    "    available_functions = {\"enrich_entities\": enrich_entities}  \n",
+    "    function_name = response_message.tool_calls[0].function.name\n",
+    "    \n",
+    "    function_to_call = available_functions[function_name]\n",
+    "    logging.info(f\"function_to_call: {function_to_call}\")\n",
+    "\n",
+    "    function_args = json.loads(response_message.tool_calls[0].function.arguments)\n",
+    "    logging.info(f\"function_args: {function_args}\")\n",
+    "\n",
+    "    function_response = function_to_call(text, function_args)\n",
+    "\n",
+    "    return {\"model_response\": response, \n",
+    "            \"function_response\": function_response}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 5. Let's Enrich a Text with Wikipedia links"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### 5.1 Run OpenAI Task"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      " 2023-10-20 18:05:51,729 - INFO - function_to_call: <function enrich_entities at 0x0000021D30C462A0>\n",
+      " 2023-10-20 18:05:51,730 - INFO - function_args: {'person': ['John Lennon', 'Paul McCartney', 'George Harrison', 'Ringo Starr'], 'org': ['The Beatles'], 'gpe': ['Liverpool'], 'date': ['1960']}\n",
+      " 2023-10-20 18:06:09,858 - INFO - entity_link_dict: {'John Lennon': 'https://en.wikipedia.org/wiki/John_Lennon', 'Paul McCartney': 'https://en.wikipedia.org/wiki/Paul_McCartney', 'George Harrison': 'https://en.wikipedia.org/wiki/George_Harrison', 'Ringo Starr': 'https://en.wikipedia.org/wiki/Ringo_Starr', 'The Beatles': 'https://en.wikipedia.org/wiki/The_Beatles', 'Liverpool': 'https://en.wikipedia.org/wiki/Liverpool'}\n"
+     ]
+    }
+   ],
+   "source": [
+    "text = \"\"\"The Beatles were an English rock band formed in Liverpool in 1960, comprising John Lennon, Paul McCartney, George Harrison, and Ringo Starr.\"\"\"\n",
+    "result = run_openai_task(labels, text)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### 5.2 Function Response"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/markdown": [
+       "**Text:** The Beatles were an English rock band formed in Liverpool in 1960, comprising John Lennon, Paul McCartney, George Harrison, and Ringo Starr.   \n",
+       "                     **Enriched_Text:** [The Beatles](https://en.wikipedia.org/wiki/The_Beatles) were an English rock band formed in [Liverpool](https://en.wikipedia.org/wiki/Liverpool) in 1960, comprising [John Lennon](https://en.wikipedia.org/wiki/John_Lennon), [Paul McCartney](https://en.wikipedia.org/wiki/Paul_McCartney), [George Harrison](https://en.wikipedia.org/wiki/George_Harrison), and [Ringo Starr](https://en.wikipedia.org/wiki/Ringo_Starr)."
+      ],
+      "text/plain": [
+       "<IPython.core.display.Markdown object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "display(Markdown(f\"\"\"**Text:** {text}   \n",
+    "                     **Enriched_Text:** {result['function_response']}\"\"\"))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### 5.3 Token Usage"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To estimate the inference costs, we can parse the response's \"usage\" field. Detailed token costs per model are available in the [OpenAI Pricing Guide](https://openai.com/pricing):"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Token Usage\n",
+      "    Prompt: 331 tokens\n",
+      "    Completion: 47 tokens\n",
+      "    Cost estimation: $0.00059\n"
+     ]
+    }
+   ],
+   "source": [
+    "# estimate inference cost assuming gpt-3.5-turbo (4K context)\n",
+    "i_tokens  = result[\"model_response\"].usage.prompt_tokens \n",
+    "o_tokens = result[\"model_response\"].usage.completion_tokens \n",
+    "\n",
+    "i_cost = (i_tokens / 1000) * 0.0015\n",
+    "o_cost = (o_tokens / 1000) * 0.002\n",
+    "\n",
+    "print(f\"\"\"Token Usage\n",
+    "    Prompt: {i_tokens} tokens\n",
+    "    Completion: {o_tokens} tokens\n",
+    "    Cost estimation: ${round(i_cost + o_cost, 5)}\"\"\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/Optimize_Prompts.ipynb

@@ -0,0 +1,953 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Optimize Prompts\n",
+    "\n",
+    "Crafting effective prompts is a critical skill when working with AI models. Even experienced users can inadvertently introduce contradictions, ambiguities, or inconsistencies that lead to suboptimal results. The system demonstrated here helps identify and fix common issues, resulting in more reliable and effective prompts.\n",
+    "\n",
+    "The optimization process uses a multi-agent approach with specialized AI agents collaborating to analyze and rewrite prompts. The system automatically identifies and addresses several types of common issues:\n",
+    "\n",
+    "- **Contradictions** in the prompt instructions\n",
+    "- Missing or unclear **format specifications**\n",
+    "- **Inconsistencies** between the prompt and few-shot examples\n",
+    "\n",
+    "---\n",
+    "\n",
+    "**Objective**: This cookbook demonstrates best practices for using Agents SDK together with Evals to build an early version of OpenAI's prompt optimization system. You can optimize your prompt using this code or use the optimizer [in our playground!](https://platform.openai.com/playground/prompts)\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "Ask ChatGPT\n",
+    "\n",
+    "\n",
+    "**Cookbook Structure**  \n",
+    "This notebook follows this structure:\n",
+    "\n",
+    "- [Step 1. System Overview](#1-system-overview) - Learn how the prompt optimization system works  \n",
+    "- [Step 2. Data Models](#2-data-models) - Understand the data structures used by the system\n",
+    "- [Step 3. Defining the Agents](#3-defining-the-agents) - Look at agents that analyze and improve prompts\n",
+    "- [Step 4. Evaluations](#4-using-evaluations-to-arrive-at-these-agents) - Use Evals to verify our agent model choice and instructions\n",
+    "- [Step 5. Run Optimization Workflow](#4-run-optimization-workflow) - See how the workflow hands off the prompts\n",
+    "- [Step 6. Examples](#5-examples) - Explore real-world examples of prompt optimization\n",
+    "\n",
+    "**Prerequisites**\n",
+    "- The `openai` Python package \n",
+    "- The `openai-agents` package\n",
+    "- An OpenAI API key set as `OPENAI_API_KEY` in your environment variables"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. System Overview\n",
+    "\n",
+    "The prompt optimization system uses a collaborative multi-agent approach to analyze and improve prompts. Each agent specializes in either detecting or rewriting a specific type of issue:\n",
+    "\n",
+    "1. **Dev-Contradiction-Checker**: Scans the prompt for logical contradictions or impossible instructions, like \"only use positive numbers\" and \"include negative examples\" in the same prompt.\n",
+    "\n",
+    "2. **Format-Checker**: Identifies when a prompt expects structured output (like JSON, CSV, or Markdown) but fails to clearly specify the exact format requirements. This agent ensures that all necessary fields, data types, and formatting rules are explicitly defined.\n",
+    "\n",
+    "3. **Few-Shot-Consistency-Checker**: Examines example conversations to ensure that the assistant's responses actually follow the rules specified in the prompt. This catches mismatches between what the prompt requires and what the examples demonstrate.\n",
+    "\n",
+    "4. **Dev-Rewriter**: After issues are identified, this agent rewrites the prompt to resolve contradictions and clarify format specifications while preserving the original intent.\n",
+    "\n",
+    "5. **Few-Shot-Rewriter**: Updates inconsistent example responses to align with the rules in the prompt, ensuring all examples properly comply with the new developer prompt.\n",
+    "\n",
+    "By working together, these agents can systematically identify and fix issues in prompts."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Import required modules\n",
+    "from openai import AsyncOpenAI\n",
+    "import asyncio\n",
+    "import json\n",
+    "import os\n",
+    "from enum import Enum\n",
+    "from typing import Any, List, Dict\n",
+    "from pydantic import BaseModel, Field\n",
+    "from agents import Agent, Runner, set_default_openai_client, trace\n",
+    "\n",
+    "openai_client: AsyncOpenAI | None = None\n",
+    "\n",
+    "def _get_openai_client() -> AsyncOpenAI:\n",
+    "    global openai_client\n",
+    "    if openai_client is None:\n",
+    "        openai_client = AsyncOpenAI(\n",
+    "            api_key=os.environ.get(\"OPENAI_API_KEY\", \"your-api-key\"),\n",
+    "        )\n",
+    "    return openai_client\n",
+    "\n",
+    "set_default_openai_client(_get_openai_client())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Data Models\n",
+    "\n",
+    "To facilitate structured communication between agents, the system uses Pydantic models to define the expected format for inputs and outputs. These Pydantic models help validate data and ensure consistency throughout the workflow.\n",
+    "\n",
+    "The data models include:\n",
+    "\n",
+    "1. **Role** - An enumeration for message roles (user/assistant)\n",
+    "2. **ChatMessage** - Represents a single message in a conversation\n",
+    "3. **Issues** - Base model for reporting detected issues\n",
+    "4. **FewShotIssues** - Extended model that adds rewrite suggestions for example messages\n",
+    "5. **MessagesOutput** - Contains optimized conversation messages\n",
+    "6. **DevRewriteOutput** - Contains the improved developer prompt\n",
+    "\n",
+    "Using Pydantic allows the system to validate that all data conforms to the expected format at each step of the process."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 32,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class Role(str, Enum):\n",
+    "    \"\"\"Role enum for chat messages.\"\"\"\n",
+    "    user = \"user\"\n",
+    "    assistant = \"assistant\"\n",
+    "\n",
+    "class ChatMessage(BaseModel):\n",
+    "    \"\"\"Single chat message used in few-shot examples.\"\"\"\n",
+    "    role: Role\n",
+    "    content: str\n",
+    "\n",
+    "class Issues(BaseModel):\n",
+    "    \"\"\"Structured output returned by checkers.\"\"\"\n",
+    "    has_issues: bool\n",
+    "    issues: List[str]\n",
+    "    \n",
+    "    @classmethod\n",
+    "    def no_issues(cls) -> \"Issues\":\n",
+    "        return cls(has_issues=False, issues=[])\n",
+    "\n",
+    "class FewShotIssues(Issues):\n",
+    "    \"\"\"Output for few-shot contradiction detector including optional rewrite suggestions.\"\"\"\n",
+    "    rewrite_suggestions: List[str] = Field(default_factory=list)\n",
+    "    \n",
+    "    @classmethod\n",
+    "    def no_issues(cls) -> \"FewShotIssues\":\n",
+    "        return cls(has_issues=False, issues=[], rewrite_suggestions=[])\n",
+    "\n",
+    "class MessagesOutput(BaseModel):\n",
+    "    \"\"\"Structured output returned by `rewrite_messages_agent`.\"\"\"\n",
+    "\n",
+    "    messages: list[ChatMessage]\n",
+    "\n",
+    "\n",
+    "class DevRewriteOutput(BaseModel):\n",
+    "    \"\"\"Rewriter returns the cleaned-up developer prompt.\"\"\"\n",
+    "\n",
+    "    new_developer_message: str"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Defining the Agents\n",
+    "\n",
+    "In this section, we create specialized AI agents using the `Agent` class from the `openai-agents` package. Looking at these agent definitions reveals several best practices for creating effective AI instructions:\n",
+    "\n",
+    "### Best Practices in Agent Instructions\n",
+    "\n",
+    "1. **Clear Scope Definition**: Each agent has a narrowly defined purpose with explicit boundaries. For example, the contradiction checker focuses only on \"genuine self-contradictions\" and explicitly states that \"overlaps or redundancies are not contradictions.\"\n",
+    "\n",
+    "2. **Step-by-Step Process**: Instructions provide a clear methodology, like how the format checker first categorizes the task before analyzing format requirements.\n",
+    "\n",
+    "3. **Explicit Definitions**: Key terms are defined precisely to avoid ambiguity. The few-shot consistency checker includes a detailed \"Compliance Rubric\" explaining exactly what constitutes compliance.\n",
+    "\n",
+    "4. **Boundary Setting**: Instructions specify what the agent should NOT do. The few-shot checker explicitly lists what's \"Out-of-scope\" to prevent over-flagging issues.\n",
+    "\n",
+    "5. **Structured Output Requirements**: Each agent has a strictly defined output format with examples, ensuring consistency in the optimization pipeline.\n",
+    "\n",
+    "These principles create reliable, focused agents that work effectively together in the optimization system. Below we see the complete agent definitions with their detailed instructions."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "dev_contradiction_checker = Agent(\n",
+    "    name=\"contradiction_detector\",\n",
+    "    model=\"gpt-4.1\",\n",
+    "    output_type=Issues,\n",
+    "    instructions=\"\"\"\n",
+    "    You are **Dev-Contradiction-Checker**.\n",
+    "\n",
+    "    Goal\n",
+    "    Detect *genuine* self-contradictions or impossibilities **inside** the developer prompt supplied in the variable `DEVELOPER_MESSAGE`.\n",
+    "\n",
+    "    Definitions\n",
+    "    • A contradiction = two clauses that cannot both be followed.\n",
+    "    • Overlaps or redundancies in the DEVELOPER_MESSAGE are *not* contradictions.\n",
+    "\n",
+    "    What you MUST do\n",
+    "    1. Compare every imperative / prohibition against all others.\n",
+    "    2. List at most FIVE contradictions (each as ONE bullet).\n",
+    "    3. If no contradiction exists, say so.\n",
+    "\n",
+    "    Output format (**strict JSON**)\n",
+    "    Return **only** an object that matches the `Issues` schema:\n",
+    "\n",
+    "    ```json\n",
+    "    {\"has_issues\": <bool>,\n",
+    "    \"issues\": [\n",
+    "        \"<bullet 1>\",\n",
+    "        \"<bullet 2>\"\n",
+    "    ]\n",
+    "    }\n",
+    "    - has_issues = true IFF the issues array is non-empty.\n",
+    "    - Do not add extra keys, comments or markdown.\n",
+    "\"\"\",\n",
+    ")\n",
+    "format_checker = Agent(\n",
+    "    name=\"format_checker\",\n",
+    "    model=\"gpt-4.1\",\n",
+    "    output_type=Issues,\n",
+    "    instructions=\"\"\"\n",
+    "    You are Format-Checker.\n",
+    "\n",
+    "    Task\n",
+    "    Decide whether the developer prompt requires a structured output (JSON/CSV/XML/Markdown table, etc.).\n",
+    "    If so, flag any missing or unclear aspects of that format.\n",
+    "\n",
+    "    Steps\n",
+    "    Categorise the task as:\n",
+    "    a. \"conversation_only\", or\n",
+    "    b. \"structured_output_required\".\n",
+    "\n",
+    "    For case (b):\n",
+    "    - Point out absent fields, ambiguous data types, unspecified ordering, or missing error-handling.\n",
+    "\n",
+    "    Do NOT invent issues if unsure. be a little bit more conservative in flagging format issues\n",
+    "\n",
+    "    Output format\n",
+    "    Return strictly-valid JSON following the Issues schema:\n",
+    "\n",
+    "    {\n",
+    "    \"has_issues\": <bool>,\n",
+    "    \"issues\": [\"<desc 1>\", \"...\"]\n",
+    "    }\n",
+    "    Maximum five issues. No extra keys or text.\n",
+    "\"\"\",\n",
+    ")\n",
+    "fewshot_consistency_checker = Agent(\n",
+    "    name=\"fewshot_consistency_checker\",\n",
+    "    model=\"gpt-4.1\",\n",
+    "    output_type=FewShotIssues,\n",
+    "    instructions=\"\"\"\n",
+    "    You are FewShot-Consistency-Checker.\n",
+    "\n",
+    "    Goal\n",
+    "    Find conflicts between the DEVELOPER_MESSAGE rules and the accompanying **assistant** examples.\n",
+    "\n",
+    "    USER_EXAMPLES:      <all user lines>          # context only\n",
+    "    ASSISTANT_EXAMPLES: <all assistant lines>     # to be evaluated\n",
+    "\n",
+    "    Method\n",
+    "    Extract key constraints from DEVELOPER_MESSAGE:\n",
+    "    - Tone / style\n",
+    "    - Forbidden or mandated content\n",
+    "    - Output format requirements\n",
+    "\n",
+    "    Compliance Rubric - read carefully\n",
+    "    Evaluate only what the developer message makes explicit.\n",
+    "\n",
+    "    Objective constraints you must check when present:\n",
+    "    - Required output type syntax (e.g., \"JSON object\", \"single sentence\", \"subject line\").\n",
+    "    - Hard limits (length ≤ N chars, language required to be English, forbidden words, etc.).\n",
+    "    - Mandatory tokens or fields the developer explicitly names.\n",
+    "\n",
+    "    Out-of-scope (DO NOT FLAG):\n",
+    "    - Whether the reply \"sounds generic\", \"repeats the prompt\", or \"fully reflects the user's request\" - unless the developer text explicitly demands those qualities.\n",
+    "    - Creative style, marketing quality, or depth of content unless stated.\n",
+    "    - Minor stylistic choices (capitalisation, punctuation) that do not violate an explicit rule.\n",
+    "\n",
+    "    Pass/Fail rule\n",
+    "    - If an assistant reply satisfies all objective constraints, it is compliant, even if you personally find it bland or loosely related.\n",
+    "    - Only record an issue when a concrete, quoted rule is broken.\n",
+    "\n",
+    "    Empty assistant list ⇒ immediately return has_issues=false.\n",
+    "\n",
+    "    For each assistant example:\n",
+    "    - USER_EXAMPLES are for context only; never use them to judge compliance.\n",
+    "    - Judge each assistant reply solely against the explicit constraints you extracted from the developer message.\n",
+    "    - If a reply breaks a specific, quoted rule, add a line explaining which rule it breaks.\n",
+    "    - Optionally, suggest a rewrite in one short sentence (add to rewrite_suggestions).\n",
+    "    - If you are uncertain, do not flag an issue.\n",
+    "    - Be conservative—uncertain or ambiguous cases are not issues.\n",
+    "\n",
+    "    be a little bit more conservative in flagging few shot contradiction issues\n",
+    "    Output format\n",
+    "    Return JSON matching FewShotIssues:\n",
+    "\n",
+    "    {\n",
+    "    \"has_issues\": <bool>,\n",
+    "    \"issues\": [\"<explanation 1>\", \"...\"],\n",
+    "    \"rewrite_suggestions\": [\"<suggestion 1>\", \"...\"] // may be []\n",
+    "    }\n",
+    "    List max five items for both arrays.\n",
+    "    Provide empty arrays when none.\n",
+    "    No markdown, no extra keys.\n",
+    "    \"\"\",\n",
+    ")\n",
+    "dev_rewriter = Agent(\n",
+    "    name=\"dev_rewriter\",\n",
+    "    model=\"gpt-4.1\",\n",
+    "    output_type=DevRewriteOutput,\n",
+    "    instructions=\"\"\"\n",
+    "    You are Dev-Rewriter.\n",
+    "\n",
+    "    You receive:\n",
+    "    - ORIGINAL_DEVELOPER_MESSAGE\n",
+    "    - CONTRADICTION_ISSUES (may be empty)\n",
+    "    - FORMAT_ISSUES (may be empty)\n",
+    "\n",
+    "    Rewrite rules\n",
+    "    Preserve the original intent and capabilities.\n",
+    "\n",
+    "    Resolve each contradiction:\n",
+    "    - Keep the clause that preserves the message intent; remove/merge the conflicting one.\n",
+    "\n",
+    "    If FORMAT_ISSUES is non-empty:\n",
+    "    - Append a new section titled ## Output Format that clearly defines the schema or gives an explicit example.\n",
+    "\n",
+    "    Do NOT change few-shot examples.\n",
+    "\n",
+    "    Do NOT add new policies or scope.\n",
+    "\n",
+    "    Output format (strict JSON)\n",
+    "    {\n",
+    "    \"new_developer_message\": \"<full rewritten text>\"\n",
+    "    }\n",
+    "    No other keys, no markdown.\n",
+    "\"\"\",\n",
+    ")\n",
+    "fewshot_rewriter = Agent(\n",
+    "    name=\"fewshot_rewriter\",\n",
+    "    model=\"gpt-4.1\",\n",
+    "    output_type=MessagesOutput,\n",
+    "    instructions=\"\"\"\n",
+    "    You are FewShot-Rewriter.\n",
+    "\n",
+    "    Input payload\n",
+    "    - NEW_DEVELOPER_MESSAGE (already optimized)\n",
+    "    - ORIGINAL_MESSAGES (list of user/assistant dicts)\n",
+    "    - FEW_SHOT_ISSUES (non-empty)\n",
+    "\n",
+    "    Task\n",
+    "    Regenerate only the assistant parts that were flagged.\n",
+    "    User messages must remain identical.\n",
+    "    Every regenerated assistant reply MUST comply with NEW_DEVELOPER_MESSAGE.\n",
+    "\n",
+    "    After regenerating each assistant reply, verify:\n",
+    "    - It matches NEW_DEVELOPER_MESSAGE. ENSURE THAT THIS IS TRUE.\n",
+    "\n",
+    "    Output format\n",
+    "    Return strict JSON that matches the MessagesOutput schema:\n",
+    "\n",
+    "    {\n",
+    "    \"messages\": [\n",
+    "        {\"role\": \"user\", \"content\": \"...\"},\n",
+    "        {\"role\": \"assistant\", \"content\": \"...\"}\n",
+    "        ]\n",
+    "    }\n",
+    "    Guidelines\n",
+    "    - Preserve original ordering and total count.\n",
+    "    - If a message was unproblematic, copy it unchanged.\n",
+    "    \"\"\",\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4. Using Evaluations to Arrive at These Agents\n",
+    "\n",
+    "Let's see how we used OpenAI Evals to tune agent instructions and pick the correct model to use. In order to do so we constructed a set of golden examples: each one contains original messages (developer message + user/assistant message) and the changes our optimization workflow should make. Here are two example of golden pairs that we used:"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "```\n",
+    "[\n",
+    "  {\n",
+    "    \"focus\": \"contradiction_issues\",\n",
+    "    \"input_payload\": {\n",
+    "      \"developer_message\": \"Always answer in **English**.\\nNunca respondas en inglés.\",\n",
+    "      \"messages\": [\n",
+    "        {\n",
+    "          \"role\": \"user\",\n",
+    "          \"content\": \"¿Qué hora es?\"\n",
+    "        }\n",
+    "      ]\n",
+    "    },\n",
+    "    \"golden_output\": {\n",
+    "      \"changes\": true,\n",
+    "      \"new_developer_message\": \"Always answer **in English**.\",\n",
+    "      \"new_messages\": [\n",
+    "        {\n",
+    "          \"role\": \"user\",\n",
+    "          \"content\": \"¿Qué hora es?\"\n",
+    "        }\n",
+    "      ],\n",
+    "      \"contradiction_issues\": \"Developer message simultaneously insists on English and forbids it.\",\n",
+    "      \"few_shot_contradiction_issues\": \"\",\n",
+    "      \"format_issues\": \"\",\n",
+    "      \"general_improvements\": \"\"\n",
+    "    }\n",
+    "  },\n",
+    "  {\n",
+    "    \"focus\": \"few_shot_contradiction_issues\",\n",
+    "    \"input_payload\": {\n",
+    "      \"developer_message\": \"Respond with **only 'yes' or 'no'** – no explanations.\",\n",
+    "      \"messages\": [\n",
+    "        {\n",
+    "          \"role\": \"user\",\n",
+    "          \"content\": \"Is the sky blue?\"\n",
+    "        },\n",
+    "        {\n",
+    "          \"role\": \"assistant\",\n",
+    "          \"content\": \"Yes, because wavelengths …\"\n",
+    "        },\n",
+    "        {\n",
+    "          \"role\": \"user\",\n",
+    "          \"content\": \"Is water wet?\"\n",
+    "        },\n",
+    "        {\n",
+    "          \"role\": \"assistant\",\n",
+    "          \"content\": \"Yes.\"\n",
+    "        }\n",
+    "      ]\n",
+    "    },\n",
+    "    \"golden_output\": {\n",
+    "      \"changes\": true,\n",
+    "      \"new_developer_message\": \"Respond with **only** the single word \\\"yes\\\" or \\\"no\\\".\",\n",
+    "      \"new_messages\": [\n",
+    "        {\n",
+    "          \"role\": \"user\",\n",
+    "          \"content\": \"Is the sky blue?\"\n",
+    "        },\n",
+    "        {\n",
+    "          \"role\": \"assistant\",\n",
+    "          \"content\": \"yes\"\n",
+    "        },\n",
+    "        {\n",
+    "          \"role\": \"user\",\n",
+    "          \"content\": \"Is water wet?\"\n",
+    "        },\n",
+    "        {\n",
+    "          \"role\": \"assistant\",\n",
+    "          \"content\": \"yes\"\n",
+    "        }\n",
+    "      ],\n",
+    "      \"contradiction_issues\": \"\",\n",
+    "      \"few_shot_contradiction_issues\": \"Assistant examples include explanations despite instruction not to.\",\n",
+    "      \"format_issues\": \"\",\n",
+    "      \"general_improvements\": \"\"\n",
+    "    }\n",
+    "  }\n",
+    "]\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "From these 20 hand labelled golden outputs which cover a range of contradiction issues, few shot issues, format issues, no issues, or a combination of issues, we built a python string check grader to verify two things: whether an issue was detected for each golden pair and whether the detected issue matched the expected one. From this signal, we tuned the agent instructions and which model to use to maximize our accuracy across this evaluation. We landed on the 4.1 model as a balance between accuracy, cost, and speed. The specific prompts we used also follow the 4.1 prompting guide. As you can see, we achieve the correct labels on all 20 golden outputs: identifying the right issues and avoiding false positives. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "![Accuracy for the golden set](../images/optimizepromptfig1.png)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "![Evaluation for the golden set](../images/optimizepromptfig2.png)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 5. Run Optimization Workflow\n",
+    "\n",
+    "Let's dive into how the optimization system actually works end to end. The core workflow consists of multiple runs of the agents in parallel to efficiently process and optimize prompts."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def _normalize_messages(messages: List[Any]) -> List[Dict[str, str]]:\n",
+    "    \"\"\"Convert list of pydantic message models to JSON-serializable dicts.\"\"\"\n",
+    "    result = []\n",
+    "    for m in messages:\n",
+    "        if hasattr(m, \"model_dump\"):\n",
+    "            result.append(m.model_dump())\n",
+    "        elif isinstance(m, dict) and \"role\" in m and \"content\" in m:\n",
+    "            result.append({\"role\": str(m[\"role\"]), \"content\": str(m[\"content\"])})\n",
+    "    return result\n",
+    "\n",
+    "async def optimize_prompt_parallel(\n",
+    "    developer_message: str,\n",
+    "    messages: List[\"ChatMessage\"],\n",
+    ") -> Dict[str, Any]:\n",
+    "    \"\"\"\n",
+    "    Runs contradiction, format, and few-shot checkers in parallel,\n",
+    "    then rewrites the prompt/examples if needed.\n",
+    "    Returns a unified dict suitable for an API or endpoint.\n",
+    "    \"\"\"\n",
+    "\n",
+    "    with trace(\"optimize_prompt_workflow\"):\n",
+    "        # 1. Run all checkers in parallel (contradiction, format, fewshot if there are examples)\n",
+    "        tasks = [\n",
+    "            Runner.run(dev_contradiction_checker, developer_message),\n",
+    "            Runner.run(format_checker, developer_message),\n",
+    "        ]\n",
+    "        if messages:\n",
+    "            fs_input = {\n",
+    "                \"DEVELOPER_MESSAGE\": developer_message,\n",
+    "                \"USER_EXAMPLES\": [m.content for m in messages if m.role == \"user\"],\n",
+    "                \"ASSISTANT_EXAMPLES\": [m.content for m in messages if m.role == \"assistant\"],\n",
+    "            }\n",
+    "            tasks.append(Runner.run(fewshot_consistency_checker, json.dumps(fs_input)))\n",
+    "\n",
+    "        results = await asyncio.gather(*tasks)\n",
+    "\n",
+    "        # Unpack results\n",
+    "        cd_issues: Issues = results[0].final_output\n",
+    "        fi_issues: Issues = results[1].final_output\n",
+    "        fs_issues: FewShotIssues = results[2].final_output if messages else FewShotIssues.no_issues()\n",
+    "\n",
+    "        # 3. Rewrites as needed\n",
+    "        final_prompt = developer_message\n",
+    "        if cd_issues.has_issues or fi_issues.has_issues:\n",
+    "            pr_input = {\n",
+    "                \"ORIGINAL_DEVELOPER_MESSAGE\": developer_message,\n",
+    "                \"CONTRADICTION_ISSUES\": cd_issues.model_dump(),\n",
+    "                \"FORMAT_ISSUES\": fi_issues.model_dump(),\n",
+    "            }\n",
+    "            pr_res = await Runner.run(dev_rewriter, json.dumps(pr_input))\n",
+    "            final_prompt = pr_res.final_output.new_developer_message\n",
+    "\n",
+    "        final_messages: list[ChatMessage] | list[dict[str, str]] = messages\n",
+    "        if fs_issues.has_issues:\n",
+    "            mr_input = {\n",
+    "                \"NEW_DEVELOPER_MESSAGE\": final_prompt,\n",
+    "                \"ORIGINAL_MESSAGES\": _normalize_messages(messages),\n",
+    "                \"FEW_SHOT_ISSUES\": fs_issues.model_dump(),\n",
+    "            }\n",
+    "            mr_res = await Runner.run(fewshot_rewriter, json.dumps(mr_input))\n",
+    "            final_messages = mr_res.final_output.messages\n",
+    "\n",
+    "        return {\n",
+    "            \"changes\": True,\n",
+    "            \"new_developer_message\": final_prompt,\n",
+    "            \"new_messages\": _normalize_messages(final_messages),\n",
+    "            \"contradiction_issues\": \"\\n\".join(cd_issues.issues),\n",
+    "            \"few_shot_contradiction_issues\": \"\\n\".join(fs_issues.issues),\n",
+    "            \"format_issues\": \"\\n\".join(fi_issues.issues),\n",
+    "        }"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "![Trace for the workflow](../images/optimizepromptfig3.png)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Understanding the Optimization Workflow\n",
+    "\n",
+    "The `optimize_prompt_parallel` function implements a workflow to maximize efficiency through parallelization:\n",
+    "\n",
+    "1. **Parallel Issue Detection**: The first phase runs all checker agents simultaneously:\n",
+    "   - `dev_contradiction_checker` searches for logical contradictions in the prompt\n",
+    "   - `format_checker` looks for unclear format specifications\n",
+    "   - `fewshot_consistency_checker` (if examples exist) checks for mismatches between the prompt and examples\n",
+    "\n",
+    "After the parallel checking phase, the workflow handles dependencies carefully:\n",
+    "\n",
+    "2. **Prompt Rewriting (Conditional)**: The `dev_rewriter` agent only runs if contradiction or format issues were detected. This agent depends on the outputs from:\n",
+    "   - `dev_contradiction_checker` (the `cd_issues` variable)\n",
+    "   - `format_checker` (the `fi_issues` variable)\n",
+    "\n",
+    "3. **Example Rewriting (Conditional)**: The `fewshot_rewriter` agent only runs if example inconsistencies were detected. This agent depends on:\n",
+    "   - The rewritten prompt (must be done after prompt rewriting)\n",
+    "   - The original messages\n",
+    "   - The few-shot issues (the `fs_issues` variable)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 6. Examples\n",
+    "\n",
+    "Let's see the optimization system in action with some practical examples."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Example 1: Fixing Contradictions"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 28,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Contradiction issues:\n",
+      "There is a contradiction between the rule that says to short-circuit and output an error if *any* required field is missing ('{\"error\": \"FIELD_MISSING:<field>\"}') and the final note which states that it is acceptable to output null for any missing field instead of an error. Both behaviors cannot be followed simultaneously when a required field is missing.\n",
+      "\n",
+      "Optimized prompt:\n",
+      "Quick-Start Card — Product Parser\n",
+      "\n",
+      "Goal  \n",
+      "Digest raw HTML of an e-commerce product detail page and emit **concise, minified JSON** describing the item.\n",
+      "\n",
+      "**Required fields:**  \n",
+      "name | brand | sku | price.value | price.currency | images[] | sizes[] | materials[] | care_instructions | features[]\n",
+      "\n",
+      "**Extraction priority:**  \n",
+      "1. schema.org/JSON-LD blocks  \n",
+      "2. <meta> & microdata tags  \n",
+      "3. Visible DOM fallback (class hints: \"product-name\", \"price\")\n",
+      "\n",
+      "**Rules:**  \n",
+      "- If *any* required field is missing, short-circuit and output: `{\"error\": \"FIELD_MISSING:<field>\"}`\n",
+      "- Prices: Numeric with dot decimal; strip non-digits (e.g., \"1.299,00 EUR\" → 1299.00 + \"EUR\").\n",
+      "- Deduplicate images differing only by query string. Keep ≤10 best-res.\n",
+      "- Sizes: Ensure unit tag (\"EU\", \"US\") and ascending sort.\n",
+      "- Materials: Title-case and collapse synonyms (e.g., \"polyester 100%\" → \"Polyester\").\n",
+      "\n",
+      "## Output Format\n",
+      "\n",
+      "- Successful Output: Emit a minified JSON object with the following fields and types (order not enforced):\n",
+      "  - name: string\n",
+      "  - brand: string\n",
+      "  - sku: string\n",
+      "  - price: object with:\n",
+      "    - value: number\n",
+      "    - currency: string\n",
+      "  - images: array of string URLs\n",
+      "  - sizes: array of strings (each including a unit tag, e.g., \"37 EU\")\n",
+      "  - materials: array of strings\n",
+      "  - care_instructions: string\n",
+      "  - features: array of strings\n",
+      "\n",
+      "Example:\n",
+      "{\"name\":\"Product Name\",\"brand\":\"Brand\",\"sku\":\"SKU123\",\"price\":{\"value\":1299.00,\"currency\":\"EUR\"},\"images\":[\"https://example.com/image1.jpg\",\"https://example.com/image2.jpg\"],\"sizes\":[\"37 EU\",\"38 EU\"],\"materials\":[\"Cotton\",\"Polyester\"],\"care_instructions\":\"Machine wash cold\",\"features\":[\"Feature 1\",\"Feature 2\"]}\n",
+      "\n",
+      "- If any required field is missing, return:\n",
+      "  {\"error\": \"FIELD_MISSING:<field>\"}\n",
+      "  (Where <field> is the missing required field name.)\n"
+     ]
+    }
+   ],
+   "source": [
+    "async def example_contradiction():\n",
+    "    # A prompt with contradictory instructions\n",
+    "    prompt = \"\"\"Quick-Start Card — Product Parser\n",
+    "\n",
+    "Goal  \n",
+    "Digest raw HTML of an e-commerce product detail page and emit **concise, minified JSON** describing the item.\n",
+    "\n",
+    "**Required fields:**  \n",
+    "name | brand | sku | price.value | price.currency | images[] | sizes[] | materials[] | care_instructions | features[]\n",
+    "\n",
+    "**Extraction priority:**  \n",
+    "1. schema.org/JSON-LD blocks  \n",
+    "2. <meta> & microdata tags  \n",
+    "3. Visible DOM fallback (class hints: \"product-name\", \"price\")\n",
+    "\n",
+    "** Rules:**  \n",
+    "- If *any* required field is missing, short-circuit with: `{\"error\": \"FIELD_MISSING:<field>\"}`.\n",
+    "- Prices: Numeric with dot decimal; strip non-digits (e.g., \"1.299,00 EUR\" → 1299.00 + \"EUR\").\n",
+    "- Deduplicate images differing only by query string. Keep ≤10 best-res.\n",
+    "- Sizes: Ensure unit tag (\"EU\", \"US\") and ascending sort.\n",
+    "- Materials: Title-case and collapse synonyms (e.g., \"polyester 100%\" → \"Polyester\").\n",
+    "\n",
+    "**Sample skeleton (minified):**\n",
+    "```json\n",
+    "{\"name\":\"\",\"brand\":\"\",\"sku\":\"\",\"price\":{\"value\":0,\"currency\":\"USD\"},\"images\":[\"\"],\"sizes\":[],\"materials\":[],\"care_instructions\":\"\",\"features\":[]}\n",
+    "Note: It is acceptable to output null for any missing field instead of an error ###\"\"\"\n",
+    "    \n",
+    "    result = await optimize_prompt_parallel(prompt, [])\n",
+    "    \n",
+    "    # Display the results\n",
+    "    if result[\"contradiction_issues\"]:\n",
+    "        print(\"Contradiction issues:\")\n",
+    "        print(result[\"contradiction_issues\"])\n",
+    "        print()\n",
+    "        \n",
+    "    print(\"Optimized prompt:\")\n",
+    "    print(result[\"new_developer_message\"])\n",
+    "    \n",
+    "# Run the example\n",
+    "await example_contradiction()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This demonstrates how the system can detect and resolve critical contradictions that could lead to inconsistent outputs or confusion for the model. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Example 2: Fixing Inconsistencies Between Prompt and Few-Shot Examples\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 29,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Few-shot examples before optimization:\n",
+      "User: Largest US city?\n",
+      "Assistant: New York City\n",
+      "User: Largest UK city?\n",
+      "Assistant: {\"city\":\"London\",\"population\":9541000}\n",
+      "\n",
+      "Inconsistency found: The response 'New York City' does not use JSON format and is missing the required keys `city` and `population` as stated in the rule 'Respond **only** with JSON using keys `city` (string) and `population` (integer).'\n",
+      "\n",
+      "Few-shot examples after optimization:\n",
+      "User: Largest US city?\n",
+      "Assistant: {\"city\":\"New York City\",\"population\":8468000}\n",
+      "User: Largest UK city?\n",
+      "Assistant: {\"city\":\"London\",\"population\":9541000}\n"
+     ]
+    }
+   ],
+   "source": [
+    "async def example_fewshot_fix():\n",
+    "    prompt = \"Respond **only** with JSON using keys `city` (string) and `population` (integer).\"\n",
+    "    \n",
+    "    messages = [\n",
+    "        {\"role\": \"user\", \"content\": \"Largest US city?\"},\n",
+    "        {\"role\": \"assistant\", \"content\": \"New York City\"},\n",
+    "        {\"role\": \"user\", \"content\": \"Largest UK city?\"},\n",
+    "        {\"role\": \"assistant\", \"content\": \"{\\\"city\\\":\\\"London\\\",\\\"population\\\":9541000}\"}\n",
+    "    ]\n",
+    "    \n",
+    "    \n",
+    "    print(\"Few-shot examples before optimization:\")\n",
+    "    print(f\"User: {messages[0]['content']}\")\n",
+    "    print(f\"Assistant: {messages[1]['content']}\")\n",
+    "    print(f\"User: {messages[2]['content']}\")\n",
+    "    print(f\"Assistant: {messages[3]['content']}\")\n",
+    "    print()\n",
+    "    \n",
+    "    # Call the optimization API\n",
+    "    result = await optimize_prompt_parallel(prompt, [ChatMessage(**m) for m in messages])\n",
+    "    \n",
+    "    # Display the results\n",
+    "    if result[\"few_shot_contradiction_issues\"]:\n",
+    "        print(\"Inconsistency found:\", result[\"few_shot_contradiction_issues\"])\n",
+    "        print()\n",
+    "    \n",
+    "    # Show the optimized few-shot examples\n",
+    "    optimized_messages = result[\"new_messages\"]\n",
+    "    print(\"Few-shot examples after optimization:\")\n",
+    "    print(f\"User: {optimized_messages[0]['content']}\")\n",
+    "    print(f\"Assistant: {optimized_messages[1]['content']}\")\n",
+    "    print(f\"User: {optimized_messages[2]['content']}\")\n",
+    "    print(f\"Assistant: {optimized_messages[3]['content']}\")\n",
+    "    \n",
+    "# Run the example\n",
+    "await example_fewshot_fix()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This is particularly important because few-shot examples have a strong influence on how models respond. If examples don't follow the stated rules, the model may learn to ignore those rules in favor of mimicking the examples. By ensuring consistency between the prompt instructions and examples, the optimization system creats a more reliable prompt."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Example 3: Clarifying Formats in a Longer Prompt"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 30,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Format issues found: Output must follow a precise Markdown template, but the expected structure (headers, formatting) for the summary and glossary sections is not fully specified.\n",
+      "Ambiguity if output should be a Markdown string or a structured object containing Markdown—data type of output is implicit.\n",
+      "No explicit ordering instruction for the summary and glossary sections—potentially ambiguous.\n",
+      "Word count limit (200 words) is mentioned for the summary but not for the glossary section—scope unclear.\n",
+      "No specific format for CLAIM_TOO_LARGE error or for indicating 'no complex terms'—should these be Markdown or plaintext?\n",
+      "\n",
+      "Optimized prompt:\n",
+      "Task → Translate dense patent claims into 200-word lay summaries with a glossary.\n",
+      "\n",
+      "Operating Steps:\n",
+      "1. Split the claim at semicolons, \"wherein\", or numbered sub-clauses.\n",
+      "2. For each chunk:\n",
+      "   a) Identify its purpose.\n",
+      "   b) Replace technical nouns with everyday analogies.\n",
+      "   c) Keep quantitative limits intact (e.g., \">=150 C\").\n",
+      "3. Flag uncommon science terms with asterisks, and later define them.\n",
+      "4. Re-assemble into a flowing paragraph; do **not** broaden or narrow the claim’s scope.\n",
+      "5. Omit boilerplate if its removal does not alter legal meaning.\n",
+      "\n",
+      "## Output Format\n",
+      "- All outputs must be provided as a Markdown string.\n",
+      "- If the claim exceeds 5 kB, respond only with the text: `CLAIM_TOO_LARGE` (no Markdown formatting).\n",
+      "- If the claim is already in plain English, output the following Markdown:\n",
+      "  \n",
+      "  ```markdown\n",
+      "  ## Summary\n",
+      "  <summary text>\n",
+      "  \n",
+      "  ## Glossary\n",
+      "  No complex terms detected.\n",
+      "  ```\n",
+      "- Otherwise, follow this Markdown template:\n",
+      "  \n",
+      "  ```markdown\n",
+      "  ## Summary\n",
+      "  <Lay summary of the claim (max 200 words)>\n",
+      "  \n",
+      "  ## Glossary\n",
+      "  - *Term1*: Definition of Term1\n",
+      "  - *Term2*: Definition of Term2\n",
+      "  ...\n",
+      "  ```\n",
+      "- The 'Summary' section comes before the 'Glossary' section in all cases.\n",
+      "- The word count limit (200 words) applies to the summary only; the glossary has no length limit.\n",
+      "\n",
+      "Remember: You are *not* providing legal advice—this is for internal comprehension only.\n"
+     ]
+    }
+   ],
+   "source": [
+    "async def example_format_issue():\n",
+    "    # A prompt with unclear or inconsistent formatting instructions\n",
+    "    prompt = \"\"\"Task → Translate dense patent claims into 200-word lay summaries with a glossary.\n",
+    "\n",
+    "Operating Steps:\n",
+    "1. Split the claim at semicolons, \"wherein\", or numbered sub-clauses.\n",
+    "2. For each chunk:\n",
+    "   a) Identify its purpose.\n",
+    "   b) Replace technical nouns with everyday analogies.\n",
+    "   c) Keep quantitative limits intact (e.g., \"≥150 C\").\n",
+    "3. Flag uncommon science terms with asterisks, and later define them.\n",
+    "4. Re-assemble into a flowing paragraph; do **not** broaden or narrow the claim’s scope.\n",
+    "5. Omit boilerplate if its removal does not alter legal meaning.\n",
+    "\n",
+    "Output should follow a Markdown template:\n",
+    "- A summary section.\n",
+    "- A glossary section with the marked terms and their definitions.\n",
+    "\n",
+    "Corner Cases:\n",
+    "- If the claim is over 5 kB, respond with CLAIM_TOO_LARGE.\n",
+    "- If claim text is already plain English, skip glossary and state no complex terms detected.\n",
+    "\n",
+    "Remember: You are *not* providing legal advice—this is for internal comprehension only.\"\"\"\n",
+    "\n",
+    "    # Call the optimization API to check for format issues\n",
+    "    result = await optimize_prompt_parallel(prompt, [])\n",
+    "\n",
+    "    # Display the results\n",
+    "    if result.get(\"format_issues\"):\n",
+    "        print(\"Format issues found:\", result[\"format_issues\"])\n",
+    "        print()\n",
+    "\n",
+    "    print(\"Optimized prompt:\")\n",
+    "    print(result[\"new_developer_message\"])\n",
+    "\n",
+    "# Run the example\n",
+    "await example_format_issue()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This example highlights how the format checker identifies and resolves ambiguous format specifications. The prompt requested a Markdown output and the optimization flow significantly improved these format specifications."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "openai",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

examples/Orchestrating_agents.ipynb

@@ -0,0 +1,887 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Orchestrating Agents: Routines and Handoffs\n",
+    "\n",
+    "When working with language models, quite often all you need for solid performance is a good prompt and the right tools. However, when dealing with many unique flows, things may get hairy. This cookbook will walk through one way to tackle this.\n",
+    "\n",
+    "We'll introduce the notion of **routines** and **handoffs**, then walk through the implementation and show how they can be used to orchestrate multiple agents in a simple, powerful, and controllable way.\n",
+    "\n",
+    "Finally, we provide a sample repo, [Swarm](https://github.com/openai/swarm), that implements these ideas along with examples."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": []
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let's start by setting up our imports."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 32,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from openai import OpenAI\n",
+    "from pydantic import BaseModel\n",
+    "from typing import Optional\n",
+    "import json\n",
+    "\n",
+    "\n",
+    "client = OpenAI()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Routines\n",
+    "\n",
+    "The notion of a \"routine\" is not strictly defined, and instead meant to capture the idea of a set of steps. Concretely, let's define a routine to be a list of instructions in natural langauge (which we'll represent with a system prompt), along with the tools necessary to complete them.\n",
+    "\n",
+    "\n",
+    "\n",
+    "Let's take a look at an example. Below, we've defined a routine for a customer service agent instructing it to triage the user issue, then either suggest a fix or provide a refund. We've also defined the necessary functions `execute_refund` and `look_up_item`. We can call this a customer service routine, agent, assistant, etc – however the idea itself is the same: a set of steps and the tools to execute them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Customer Service Routine\n",
+    "\n",
+    "system_message = (\n",
+    "    \"You are a customer support agent for ACME Inc.\"\n",
+    "    \"Always answer in a sentence or less.\"\n",
+    "    \"Follow the following routine with the user:\"\n",
+    "    \"1. First, ask probing questions and understand the user's problem deeper.\\n\"\n",
+    "    \" - unless the user has already provided a reason.\\n\"\n",
+    "    \"2. Propose a fix (make one up).\\n\"\n",
+    "    \"3. ONLY if not satisfied, offer a refund.\\n\"\n",
+    "    \"4. If accepted, search for the ID and then execute refund.\"\n",
+    "    \"\"\n",
+    ")\n",
+    "\n",
+    "def look_up_item(search_query):\n",
+    "    \"\"\"Use to find item ID.\n",
+    "    Search query can be a description or keywords.\"\"\"\n",
+    "\n",
+    "    # return hard-coded item ID - in reality would be a lookup\n",
+    "    return \"item_132612938\"\n",
+    "\n",
+    "\n",
+    "def execute_refund(item_id, reason=\"not provided\"):\n",
+    "\n",
+    "    print(\"Summary:\", item_id, reason) # lazy summary\n",
+    "    return \"success\"\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The main power of routines is their simplicity and robustness. Notice that these instructions contain conditionals much like a state machine or branching in code. LLMs can actually handle these cases quite robustly for small and medium sized routine, with the added benefit of having \"soft\" adherance – the LLM can naturally steer the conversation without getting stuck in dead-ends.\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Executing Routines\n",
+    "\n",
+    "To execute a routine, let's implement a simple loop that:\n",
+    "1. Gets user input.\n",
+    "1. Appends user message to `messages`.\n",
+    "1. Calls the model.\n",
+    "1. Appends model response to `messages`.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def run_full_turn(system_message, messages):\n",
+    "    response = client.chat.completions.create(\n",
+    "        model=\"gpt-4o-mini\",\n",
+    "        messages=[{\"role\": \"system\", \"content\": system_message}] + messages,\n",
+    "    )\n",
+    "    message = response.choices[0].message\n",
+    "    messages.append(message)\n",
+    "\n",
+    "    if message.content: print(\"Assistant:\", message.content)\n",
+    "\n",
+    "    return message\n",
+    "\n",
+    "\n",
+    "messages = []\n",
+    "while True:\n",
+    "    user = input(\"User: \")\n",
+    "    messages.append({\"role\": \"user\", \"content\": user})\n",
+    "\n",
+    "    run_full_turn(system_message, messages)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "As you can see, this currently ignores function calls, so let's add that.\n",
+    "\n",
+    "Models require functions to be formatted as a function schema. For convenience, we can define a helper function that turns python functions into the corresponding function schema."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import inspect\n",
+    "\n",
+    "def function_to_schema(func) -> dict:\n",
+    "    type_map = {\n",
+    "        str: \"string\",\n",
+    "        int: \"integer\",\n",
+    "        float: \"number\",\n",
+    "        bool: \"boolean\",\n",
+    "        list: \"array\",\n",
+    "        dict: \"object\",\n",
+    "        type(None): \"null\",\n",
+    "    }\n",
+    "\n",
+    "    try:\n",
+    "        signature = inspect.signature(func)\n",
+    "    except ValueError as e:\n",
+    "        raise ValueError(\n",
+    "            f\"Failed to get signature for function {func.__name__}: {str(e)}\"\n",
+    "        )\n",
+    "\n",
+    "    parameters = {}\n",
+    "    for param in signature.parameters.values():\n",
+    "        try:\n",
+    "            param_type = type_map.get(param.annotation, \"string\")\n",
+    "        except KeyError as e:\n",
+    "            raise KeyError(\n",
+    "                f\"Unknown type annotation {param.annotation} for parameter {param.name}: {str(e)}\"\n",
+    "            )\n",
+    "        parameters[param.name] = {\"type\": param_type}\n",
+    "\n",
+    "    required = [\n",
+    "        param.name\n",
+    "        for param in signature.parameters.values()\n",
+    "        if param.default == inspect._empty\n",
+    "    ]\n",
+    "\n",
+    "    return {\n",
+    "        \"type\": \"function\",\n",
+    "        \"function\": {\n",
+    "            \"name\": func.__name__,\n",
+    "            \"description\": (func.__doc__ or \"\").strip(),\n",
+    "            \"parameters\": {\n",
+    "                \"type\": \"object\",\n",
+    "                \"properties\": parameters,\n",
+    "                \"required\": required,\n",
+    "            },\n",
+    "        },\n",
+    "    }"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "For example:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{\n",
+      "  \"type\": \"function\",\n",
+      "  \"function\": {\n",
+      "    \"name\": \"sample_function\",\n",
+      "    \"description\": \"This is my docstring. Call this function when you want.\",\n",
+      "    \"parameters\": {\n",
+      "      \"type\": \"object\",\n",
+      "      \"properties\": {\n",
+      "        \"param_1\": {\n",
+      "          \"type\": \"string\"\n",
+      "        },\n",
+      "        \"param_2\": {\n",
+      "          \"type\": \"string\"\n",
+      "        },\n",
+      "        \"the_third_one\": {\n",
+      "          \"type\": \"integer\"\n",
+      "        },\n",
+      "        \"some_optional\": {\n",
+      "          \"type\": \"string\"\n",
+      "        }\n",
+      "      },\n",
+      "      \"required\": [\n",
+      "        \"param_1\",\n",
+      "        \"param_2\",\n",
+      "        \"the_third_one\"\n",
+      "      ]\n",
+      "    }\n",
+      "  }\n",
+      "}\n"
+     ]
+    }
+   ],
+   "source": [
+    "def sample_function(param_1, param_2, the_third_one: int, some_optional=\"John Doe\"):\n",
+    "    \"\"\"\n",
+    "    This is my docstring. Call this function when you want.\n",
+    "    \"\"\"\n",
+    "    print(\"Hello, world\")\n",
+    "\n",
+    "schema =  function_to_schema(sample_function)\n",
+    "print(json.dumps(schema, indent=2))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now, we can use this function to pass the tools to the model when we call it."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "Function(arguments='{\"search_query\":\"black boot\"}', name='look_up_item')"
+      ]
+     },
+     "execution_count": 20,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "messages = []\n",
+    "\n",
+    "tools = [execute_refund, look_up_item]\n",
+    "tool_schemas = [function_to_schema(tool) for tool in tools]\n",
+    "\n",
+    "response = client.chat.completions.create(\n",
+    "            model=\"gpt-4o-mini\",\n",
+    "            messages=[{\"role\": \"user\", \"content\": \"Look up the black boot.\"}],\n",
+    "            tools=tool_schemas,\n",
+    "        )\n",
+    "message = response.choices[0].message\n",
+    "\n",
+    "message.tool_calls[0].function"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Finally, when the model calls a tool we need to execute the corresponding function and provide the result back to the model.\n",
+    "\n",
+    "We can do this by mapping the name of the tool to the python function in a `tool_map`, then looking it up in `execute_tool_call` and calling it. Finally we add the result to the conversation."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Assistant: look_up_item({'search_query': 'black boot'})\n"
+     ]
+    }
+   ],
+   "source": [
+    "tools_map = {tool.__name__: tool for tool in tools}\n",
+    "\n",
+    "def execute_tool_call(tool_call, tools_map):\n",
+    "    name = tool_call.function.name\n",
+    "    args = json.loads(tool_call.function.arguments)\n",
+    "\n",
+    "    print(f\"Assistant: {name}({args})\")\n",
+    "\n",
+    "    # call corresponding function with provided arguments\n",
+    "    return tools_map[name](**args)\n",
+    "\n",
+    "for tool_call in message.tool_calls:\n",
+    "            result = execute_tool_call(tool_call, tools_map)\n",
+    "\n",
+    "            # add result back to conversation \n",
+    "            result_message = {\n",
+    "                \"role\": \"tool\",\n",
+    "                \"tool_call_id\": tool_call.id,\n",
+    "                \"content\": result,\n",
+    "            }\n",
+    "            messages.append(result_message)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In practice, we'll also want to let the model use the result to produce another response. That response might _also_ contain a tool call, so we can just run this in a loop until there are no more tool calls.\n",
+    "\n",
+    "If we put everything together, it will look something like this:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tools = [execute_refund, look_up_item]\n",
+    "\n",
+    "\n",
+    "def run_full_turn(system_message, tools, messages):\n",
+    "\n",
+    "    num_init_messages = len(messages)\n",
+    "    messages = messages.copy()\n",
+    "\n",
+    "    while True:\n",
+    "\n",
+    "        # turn python functions into tools and save a reverse map\n",
+    "        tool_schemas = [function_to_schema(tool) for tool in tools]\n",
+    "        tools_map = {tool.__name__: tool for tool in tools}\n",
+    "\n",
+    "        # === 1. get openai completion ===\n",
+    "        response = client.chat.completions.create(\n",
+    "            model=\"gpt-4o-mini\",\n",
+    "            messages=[{\"role\": \"system\", \"content\": system_message}] + messages,\n",
+    "            tools=tool_schemas or None,\n",
+    "        )\n",
+    "        message = response.choices[0].message\n",
+    "        messages.append(message)\n",
+    "\n",
+    "        if message.content:  # print assistant response\n",
+    "            print(\"Assistant:\", message.content)\n",
+    "\n",
+    "        if not message.tool_calls:  # if finished handling tool calls, break\n",
+    "            break\n",
+    "\n",
+    "        # === 2. handle tool calls ===\n",
+    "\n",
+    "        for tool_call in message.tool_calls:\n",
+    "            result = execute_tool_call(tool_call, tools_map)\n",
+    "\n",
+    "            result_message = {\n",
+    "                \"role\": \"tool\",\n",
+    "                \"tool_call_id\": tool_call.id,\n",
+    "                \"content\": result,\n",
+    "            }\n",
+    "            messages.append(result_message)\n",
+    "\n",
+    "    # ==== 3. return new messages =====\n",
+    "    return messages[num_init_messages:]\n",
+    "\n",
+    "\n",
+    "def execute_tool_call(tool_call, tools_map):\n",
+    "    name = tool_call.function.name\n",
+    "    args = json.loads(tool_call.function.arguments)\n",
+    "\n",
+    "    print(f\"Assistant: {name}({args})\")\n",
+    "\n",
+    "    # call corresponding function with provided arguments\n",
+    "    return tools_map[name](**args)\n",
+    "\n",
+    "\n",
+    "messages = []\n",
+    "while True:\n",
+    "    user = input(\"User: \")\n",
+    "    messages.append({\"role\": \"user\", \"content\": user})\n",
+    "\n",
+    "    new_messages = run_full_turn(system_message, tools, messages)\n",
+    "    messages.extend(new_messages)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now that we have a routine, let's say we want to add more steps and more tools. We can up to a point, but eventually if we try growing the routine with too many different tasks it may start to struggle. This is where we can leverage the notion of multiple routines – given a user request, we can load the right routine with the appropriate steps and tools to address it.\n",
+    "\n",
+    "Dynamically swapping system instructions and tools may seem daunting. However, if we view \"routines\" as \"agents\", then this notion of **handoffs** allow us to represent these swaps simply – as one agent handing off a conversation to another."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Handoffs\n",
+    "\n",
+    "Let's define a **handoff** as an agent (or routine) handing off an active conversation to another agent, much like when you get transfered to someone else on a phone call. Except in this case, the agents have complete knowledge of your prior conversation!\n",
+    "\n",
+    "To see handoffs in action, let's start by defining a basic class for an Agent."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 24,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class Agent(BaseModel):\n",
+    "    name: str = \"Agent\"\n",
+    "    model: str = \"gpt-4o-mini\"\n",
+    "    instructions: str = \"You are a helpful Agent\"\n",
+    "    tools: list = []"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now to make our code support it, we can change `run_full_turn` to take an `Agent` instead of separate `system_message` and `tools`:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 25,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def run_full_turn(agent, messages):\n",
+    "\n",
+    "    num_init_messages = len(messages)\n",
+    "    messages = messages.copy()\n",
+    "\n",
+    "    while True:\n",
+    "\n",
+    "        # turn python functions into tools and save a reverse map\n",
+    "        tool_schemas = [function_to_schema(tool) for tool in agent.tools]\n",
+    "        tools_map = {tool.__name__: tool for tool in agent.tools}\n",
+    "\n",
+    "        # === 1. get openai completion ===\n",
+    "        response = client.chat.completions.create(\n",
+    "            model=agent.model,\n",
+    "            messages=[{\"role\": \"system\", \"content\": agent.instructions}] + messages,\n",
+    "            tools=tool_schemas or None,\n",
+    "        )\n",
+    "        message = response.choices[0].message\n",
+    "        messages.append(message)\n",
+    "\n",
+    "        if message.content:  # print assistant response\n",
+    "            print(\"Assistant:\", message.content)\n",
+    "\n",
+    "        if not message.tool_calls:  # if finished handling tool calls, break\n",
+    "            break\n",
+    "\n",
+    "        # === 2. handle tool calls ===\n",
+    "\n",
+    "        for tool_call in message.tool_calls:\n",
+    "            result = execute_tool_call(tool_call, tools_map)\n",
+    "\n",
+    "            result_message = {\n",
+    "                \"role\": \"tool\",\n",
+    "                \"tool_call_id\": tool_call.id,\n",
+    "                \"content\": result,\n",
+    "            }\n",
+    "            messages.append(result_message)\n",
+    "\n",
+    "    # ==== 3. return new messages =====\n",
+    "    return messages[num_init_messages:]\n",
+    "\n",
+    "\n",
+    "def execute_tool_call(tool_call, tools_map):\n",
+    "    name = tool_call.function.name\n",
+    "    args = json.loads(tool_call.function.arguments)\n",
+    "\n",
+    "    print(f\"Assistant: {name}({args})\")\n",
+    "\n",
+    "    # call corresponding function with provided arguments\n",
+    "    return tools_map[name](**args)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can now run multiple agents easily:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 31,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "User: Place an order for a black boot.\n",
+      "Assistant: place_order({'item_name': 'black boot'})\n",
+      "Assistant: Your order for a black boot has been successfully placed! If you need anything else, feel free to ask!\n",
+      "User: Actually, I want a refund.\n",
+      "Assistant: execute_refund({'item_name': 'black boot'})\n",
+      "Assistant: Your refund for the black boot has been successfully processed. If you need further assistance, just let me know!\n"
+     ]
+    }
+   ],
+   "source": [
+    "def execute_refund(item_name):\n",
+    "    return \"success\"\n",
+    "\n",
+    "refund_agent = Agent(\n",
+    "    name=\"Refund Agent\",\n",
+    "    instructions=\"You are a refund agent. Help the user with refunds.\",\n",
+    "    tools=[execute_refund],\n",
+    ")\n",
+    "\n",
+    "def place_order(item_name):\n",
+    "    return \"success\"\n",
+    "\n",
+    "sales_assistant = Agent(\n",
+    "    name=\"Sales Assistant\",\n",
+    "    instructions=\"You are a sales assistant. Sell the user a product.\",\n",
+    "    tools=[place_order],\n",
+    ")\n",
+    "\n",
+    "\n",
+    "messages = []\n",
+    "user_query = \"Place an order for a black boot.\"\n",
+    "print(\"User:\", user_query)\n",
+    "messages.append({\"role\": \"user\", \"content\": user_query})\n",
+    "\n",
+    "response = run_full_turn(sales_assistant, messages) # sales assistant\n",
+    "messages.extend(response)\n",
+    "\n",
+    "\n",
+    "user_query = \"Actually, I want a refund.\" # implicitly refers to the last item\n",
+    "print(\"User:\", user_query)\n",
+    "messages.append({\"role\": \"user\", \"content\": user_query})\n",
+    "response = run_full_turn(refund_agent, messages) # refund agent"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Great! But we did the handoff manually here – we want the agents themselves to decide when to perform a handoff. A simple, but surprisingly effective way to do this is by giving them a `transfer_to_XXX` function, where `XXX` is some agent. The model is smart enough to know to call this function when it makes sense to make a handoff!"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Handoff Functions\n",
+    "\n",
+    "Now that agent can express the _intent_ to make a handoff, we must make it actually happen. There's many ways to do this, but there's one particularly clean way.\n",
+    "\n",
+    "For the agent functions we've defined so far, like `execute_refund` or `place_order` they return a string, which will be provided to the model. What if instead, we return an `Agent` object to indicate which agent we want to transfer to? Like so:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "refund_agent = Agent(\n",
+    "    name=\"Refund Agent\",\n",
+    "    instructions=\"You are a refund agent. Help the user with refunds.\",\n",
+    "    tools=[execute_refund],\n",
+    ")\n",
+    "\n",
+    "def transfer_to_refunds():\n",
+    "    return refund_agent\n",
+    "\n",
+    "sales_assistant = Agent(\n",
+    "    name=\"Sales Assistant\",\n",
+    "    instructions=\"You are a sales assistant. Sell the user a product.\",\n",
+    "    tools=[place_order],\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can then update our code to check the return type of a function response, and if it's an `Agent`, update the agent in use! Additionally, now `run_full_turn` will need to return the latest agent in use in case there are handoffs. (We can do this in a `Response` class to keep things neat.)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class Response(BaseModel):\n",
+    "    agent: Optional[Agent]\n",
+    "    messages: list"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now for the updated `run_full_turn`:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def run_full_turn(agent, messages):\n",
+    "\n",
+    "    current_agent = agent\n",
+    "    num_init_messages = len(messages)\n",
+    "    messages = messages.copy()\n",
+    "\n",
+    "    while True:\n",
+    "\n",
+    "        # turn python functions into tools and save a reverse map\n",
+    "        tool_schemas = [function_to_schema(tool) for tool in current_agent.tools]\n",
+    "        tools = {tool.__name__: tool for tool in current_agent.tools}\n",
+    "\n",
+    "        # === 1. get openai completion ===\n",
+    "        response = client.chat.completions.create(\n",
+    "            model=agent.model,\n",
+    "            messages=[{\"role\": \"system\", \"content\": current_agent.instructions}]\n",
+    "            + messages,\n",
+    "            tools=tool_schemas or None,\n",
+    "        )\n",
+    "        message = response.choices[0].message\n",
+    "        messages.append(message)\n",
+    "\n",
+    "        if message.content:  # print agent response\n",
+    "            print(f\"{current_agent.name}:\", message.content)\n",
+    "\n",
+    "        if not message.tool_calls:  # if finished handling tool calls, break\n",
+    "            break\n",
+    "\n",
+    "        # === 2. handle tool calls ===\n",
+    "\n",
+    "        for tool_call in message.tool_calls:\n",
+    "            result = execute_tool_call(tool_call, tools, current_agent.name)\n",
+    "\n",
+    "            if type(result) is Agent:  # if agent transfer, update current agent\n",
+    "                current_agent = result\n",
+    "                result = (\n",
+    "                    f\"Transfered to {current_agent.name}. Adopt persona immediately.\"\n",
+    "                )\n",
+    "\n",
+    "            result_message = {\n",
+    "                \"role\": \"tool\",\n",
+    "                \"tool_call_id\": tool_call.id,\n",
+    "                \"content\": result,\n",
+    "            }\n",
+    "            messages.append(result_message)\n",
+    "\n",
+    "    # ==== 3. return last agent used and new messages =====\n",
+    "    return Response(agent=current_agent, messages=messages[num_init_messages:])\n",
+    "\n",
+    "\n",
+    "def execute_tool_call(tool_call, tools, agent_name):\n",
+    "    name = tool_call.function.name\n",
+    "    args = json.loads(tool_call.function.arguments)\n",
+    "\n",
+    "    print(f\"{agent_name}:\", f\"{name}({args})\")\n",
+    "\n",
+    "    return tools[name](**args)  # call corresponding function with provided arguments"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let's look at an example with more Agents."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def escalate_to_human(summary):\n",
+    "    \"\"\"Only call this if explicitly asked to.\"\"\"\n",
+    "    print(\"Escalating to human agent...\")\n",
+    "    print(\"\\n=== Escalation Report ===\")\n",
+    "    print(f\"Summary: {summary}\")\n",
+    "    print(\"=========================\\n\")\n",
+    "    exit()\n",
+    "\n",
+    "\n",
+    "def transfer_to_sales_agent():\n",
+    "    \"\"\"User for anything sales or buying related.\"\"\"\n",
+    "    return sales_agent\n",
+    "\n",
+    "\n",
+    "def transfer_to_issues_and_repairs():\n",
+    "    \"\"\"User for issues, repairs, or refunds.\"\"\"\n",
+    "    return issues_and_repairs_agent\n",
+    "\n",
+    "\n",
+    "def transfer_back_to_triage():\n",
+    "    \"\"\"Call this if the user brings up a topic outside of your purview,\n",
+    "    including escalating to human.\"\"\"\n",
+    "    return triage_agent\n",
+    "\n",
+    "\n",
+    "triage_agent = Agent(\n",
+    "    name=\"Triage Agent\",\n",
+    "    instructions=(\n",
+    "        \"You are a customer service bot for ACME Inc. \"\n",
+    "        \"Introduce yourself. Always be very brief. \"\n",
+    "        \"Gather information to direct the customer to the right department. \"\n",
+    "        \"But make your questions subtle and natural.\"\n",
+    "    ),\n",
+    "    tools=[transfer_to_sales_agent, transfer_to_issues_and_repairs, escalate_to_human],\n",
+    ")\n",
+    "\n",
+    "\n",
+    "def execute_order(product, price: int):\n",
+    "    \"\"\"Price should be in USD.\"\"\"\n",
+    "    print(\"\\n\\n=== Order Summary ===\")\n",
+    "    print(f\"Product: {product}\")\n",
+    "    print(f\"Price: ${price}\")\n",
+    "    print(\"=================\\n\")\n",
+    "    confirm = input(\"Confirm order? y/n: \").strip().lower()\n",
+    "    if confirm == \"y\":\n",
+    "        print(\"Order execution successful!\")\n",
+    "        return \"Success\"\n",
+    "    else:\n",
+    "        print(\"Order cancelled!\")\n",
+    "        return \"User cancelled order.\"\n",
+    "\n",
+    "\n",
+    "sales_agent = Agent(\n",
+    "    name=\"Sales Agent\",\n",
+    "    instructions=(\n",
+    "        \"You are a sales agent for ACME Inc.\"\n",
+    "        \"Always answer in a sentence or less.\"\n",
+    "        \"Follow the following routine with the user:\"\n",
+    "        \"1. Ask them about any problems in their life related to catching roadrunners.\\n\"\n",
+    "        \"2. Casually mention one of ACME's crazy made-up products can help.\\n\"\n",
+    "        \" - Don't mention price.\\n\"\n",
+    "        \"3. Once the user is bought in, drop a ridiculous price.\\n\"\n",
+    "        \"4. Only after everything, and if the user says yes, \"\n",
+    "        \"tell them a crazy caveat and execute their order.\\n\"\n",
+    "        \"\"\n",
+    "    ),\n",
+    "    tools=[execute_order, transfer_back_to_triage],\n",
+    ")\n",
+    "\n",
+    "\n",
+    "def look_up_item(search_query):\n",
+    "    \"\"\"Use to find item ID.\n",
+    "    Search query can be a description or keywords.\"\"\"\n",
+    "    item_id = \"item_132612938\"\n",
+    "    print(\"Found item:\", item_id)\n",
+    "    return item_id\n",
+    "\n",
+    "\n",
+    "def execute_refund(item_id, reason=\"not provided\"):\n",
+    "    print(\"\\n\\n=== Refund Summary ===\")\n",
+    "    print(f\"Item ID: {item_id}\")\n",
+    "    print(f\"Reason: {reason}\")\n",
+    "    print(\"=================\\n\")\n",
+    "    print(\"Refund execution successful!\")\n",
+    "    return \"success\"\n",
+    "\n",
+    "\n",
+    "issues_and_repairs_agent = Agent(\n",
+    "    name=\"Issues and Repairs Agent\",\n",
+    "    instructions=(\n",
+    "        \"You are a customer support agent for ACME Inc.\"\n",
+    "        \"Always answer in a sentence or less.\"\n",
+    "        \"Follow the following routine with the user:\"\n",
+    "        \"1. First, ask probing questions and understand the user's problem deeper.\\n\"\n",
+    "        \" - unless the user has already provided a reason.\\n\"\n",
+    "        \"2. Propose a fix (make one up).\\n\"\n",
+    "        \"3. ONLY if not satesfied, offer a refund.\\n\"\n",
+    "        \"4. If accepted, search for the ID and then execute refund.\"\n",
+    "        \"\"\n",
+    "    ),\n",
+    "    tools=[execute_refund, look_up_item, transfer_back_to_triage],\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Finally, we can run this in a loop (this won't run in python notebooks, so you can try this in a separate python file):"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "agent = triage_agent\n",
+    "messages = []\n",
+    "\n",
+    "while True:\n",
+    "    user = input(\"User: \")\n",
+    "    messages.append({\"role\": \"user\", \"content\": user})\n",
+    "\n",
+    "    response = run_full_turn(agent, messages)\n",
+    "    agent = response.agent\n",
+    "    messages.extend(response.messages)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Swarm\n",
+    "\n",
+    "As a proof of concept, we've packaged these ideas into a sample library called [Swarm](https://github.com/openai/swarm). It is meant as an example only, and should not be directly used in production. However, feel free to take the ideas and code to build your own!"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/Prompt_Caching101.ipynb

@@ -0,0 +1,600 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Prompt Caching 101"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "OpenAI offers discounted prompt caching for prompts exceeding 1024 tokens, resulting in up to an 80% reduction in latency for longer prompts over 10,000 tokens. By caching repetitive information across LLM API requests, you can greatly reduce both latency and costs. Prompt caching is scoped at the organization level, meaning only members of the same organization can access shared caches. Additionally, caching is eligible for zero data retention, as no data is stored during the process.\n",
+    "\n",
+    "Prompt caching automatically activates for prompts longer than 1024 tokens-- you don't have to change anything in your completions request. When an API request is made, the system first checks if the beginning portion (prefix) of the prompt has already been cached. If a match is found (cache hit), the cached prompt is used, leading to reduced latency and costs. If there's no match, the system processes the full prompt from scratch and caches the prefix for future use.\n",
+    "\n",
+    "With these benefits in mind, some of the key use cases where prompt caching can be especially advantageous are:\n",
+    "\n",
+    "\n",
+    "- **Agents using tools and structured outputs**: Cache the extended list of tools and schemas.  \n",
+    "- **Coding and writing assistants**: Insert large sections or summaries of codebases and workspaces directly in prompts. \n",
+    "- **Chatbots**: Cache static portions of multi-turn conversations to maintain context efficiently over extended dialogues.\n",
+    "\n",
+    "In this cookbook, we'll go through a couple examples of caching tools and images. Recall that in general, you'll want to put static content like instructions and examples at the beginning of your prompt, and variable content, such as user-specific information, at the end. This also applies to images and tools, which must be identical even in their ordering between requests. All requests, including those with fewer than 1024 tokens, will display a cached_tokens field of the `usage.prompt_tokens_details` chat completions object indicating how many of the prompt tokens were a cache hit. For requests under 1024 tokens, cached_tokens will be zero. Caching discounts are based on the actual number of tokens processed, including those used for images, which also count toward your rate limits."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Example 1: Caching tools and multi-turn conversations\n",
+    "\n",
+    "In this example, we define tools and interactions for a customer support assistant, capable of handling tasks such as checking delivery dates, canceling orders, and updating payment methods. The assistant processes two separate messages, first responding to an initial query, followed by a delayed response to a follow-up query.\n",
+    "\n",
+    "When caching tools, it is important that the tool definitions and their order remain identical for them to be included in the prompt prefix. To cache message histories in a multi-turn conversation, append new elements to the end of the messages array. In the response object and the output below, for the second completion `run2`, you can see that the `cached_tokens` value is greater than zero, indicating successful caching."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from openai import OpenAI\n",
+    "import os\n",
+    "import json \n",
+    "import time\n",
+    "\n",
+    "\n",
+    "api_key = os.getenv(\"OPENAI_API_KEY\")\n",
+    "client = OpenAI(organization='org-l89177bnhkme4a44292n5r3j', api_key=api_key)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Run 1:\n",
+      "{\n",
+      "    \"id\": \"chatcmpl-ADeOueQSi2DIUMdLXnZIv9caVfnro\",\n",
+      "    \"choices\": [\n",
+      "        {\n",
+      "            \"finish_reason\": \"stop\",\n",
+      "            \"index\": 0,\n",
+      "            \"logprobs\": null,\n",
+      "            \"message\": {\n",
+      "                \"content\": null,\n",
+      "                \"refusal\": null,\n",
+      "                \"role\": \"assistant\",\n",
+      "                \"tool_calls\": [\n",
+      "                    {\n",
+      "                        \"id\": \"call_5TnLcdD9tyVMVbzNGdejlJJa\",\n",
+      "                        \"function\": {\n",
+      "                            \"arguments\": \"{\\\"order_id\\\":\\\"9876543210\\\"}\",\n",
+      "                            \"name\": \"get_delivery_date\"\n",
+      "                        },\n",
+      "                        \"type\": \"function\"\n",
+      "                    }\n",
+      "                ]\n",
+      "            }\n",
+      "        }\n",
+      "    ],\n",
+      "    \"created\": 1727816928,\n",
+      "    \"model\": \"gpt-4o-mini-2024-07-18\",\n",
+      "    \"object\": \"chat.completion\",\n",
+      "    \"system_fingerprint\": \"fp_f85bea6784\",\n",
+      "    \"usage\": {\n",
+      "        \"completion_tokens\": 17,\n",
+      "        \"prompt_tokens\": 1079,\n",
+      "        \"total_tokens\": 1096,\n",
+      "        \"prompt_tokens_details\": {\n",
+      "            \"cached_tokens\": 0\n",
+      "        },\n",
+      "        \"completion_tokens_details\": {\n",
+      "            \"reasoning_tokens\": 0\n",
+      "        }\n",
+      "    }\n",
+      "}\n",
+      "\n",
+      "Run 2:\n",
+      "{\n",
+      "    \"id\": \"chatcmpl-ADeP2i0frELC4W5RVNNkKz6TQ7hig\",\n",
+      "    \"choices\": [\n",
+      "        {\n",
+      "            \"finish_reason\": \"stop\",\n",
+      "            \"index\": 0,\n",
+      "            \"logprobs\": null,\n",
+      "            \"message\": {\n",
+      "                \"content\": null,\n",
+      "                \"refusal\": null,\n",
+      "                \"role\": \"assistant\",\n",
+      "                \"tool_calls\": [\n",
+      "                    {\n",
+      "                        \"id\": \"call_viwwDZPuQh8hJFPf2Co1dYJK\",\n",
+      "                        \"function\": {\n",
+      "                            \"arguments\": \"{\\\"order_id\\\": \\\"9876543210\\\"}\",\n",
+      "                            \"name\": \"get_delivery_date\"\n",
+      "                        },\n",
+      "                        \"type\": \"function\"\n",
+      "                    },\n",
+      "                    {\n",
+      "                        \"id\": \"call_t1FFdAhrfvRc5IgqA6WkPKYj\",\n",
+      "                        \"function\": {\n",
+      "                            \"arguments\": \"{\\\"order_id\\\": \\\"9876543210\\\", \\\"reason\\\": \\\"Decided to purchase locally to get it faster.\\\"}\",\n",
+      "                            \"name\": \"cancel_order\"\n",
+      "                        },\n",
+      "                        \"type\": \"function\"\n",
+      "                    }\n",
+      "                ]\n",
+      "            }\n",
+      "        }\n",
+      "    ],\n",
+      "    \"created\": 1727816936,\n",
+      "    \"model\": \"gpt-4o-mini-2024-07-18\",\n",
+      "    \"object\": \"chat.completion\",\n",
+      "    \"system_fingerprint\": \"fp_f85bea6784\",\n",
+      "    \"usage\": {\n",
+      "        \"completion_tokens\": 64,\n",
+      "        \"prompt_tokens\": 1136,\n",
+      "        \"total_tokens\": 1200,\n",
+      "        \"prompt_tokens_details\": {\n",
+      "            \"cached_tokens\": 1024\n",
+      "        },\n",
+      "        \"completion_tokens_details\": {\n",
+      "            \"reasoning_tokens\": 0\n",
+      "        }\n",
+      "    }\n",
+      "}\n"
+     ]
+    }
+   ],
+   "source": [
+    "import time\n",
+    "import json\n",
+    "\n",
+    "# Define tools\n",
+    "tools = [\n",
+    "    {\n",
+    "        \"type\": \"function\",\n",
+    "        \"function\": {\n",
+    "            \"name\": \"get_delivery_date\",\n",
+    "            \"description\": \"Get the delivery date for a customer's order. Call this whenever you need to know the delivery date, for example when a customer asks 'Where is my package'.\",\n",
+    "            \"parameters\": {\n",
+    "                \"type\": \"object\",\n",
+    "                \"properties\": {\n",
+    "                    \"order_id\": {\n",
+    "                        \"type\": \"string\",\n",
+    "                        \"description\": \"The customer's order ID.\",\n",
+    "                    },\n",
+    "                },\n",
+    "                \"required\": [\"order_id\"],\n",
+    "                \"additionalProperties\": False,\n",
+    "            },\n",
+    "        }\n",
+    "    },\n",
+    "    {\n",
+    "        \"type\": \"function\",\n",
+    "        \"function\": {\n",
+    "            \"name\": \"cancel_order\",\n",
+    "            \"description\": \"Cancel an order that has not yet been shipped. Use this when a customer requests order cancellation.\",\n",
+    "            \"parameters\": {\n",
+    "                \"type\": \"object\",\n",
+    "                \"properties\": {\n",
+    "                    \"order_id\": {\n",
+    "                        \"type\": \"string\",\n",
+    "                        \"description\": \"The customer's order ID.\"\n",
+    "                    },\n",
+    "                    \"reason\": {\n",
+    "                        \"type\": \"string\",\n",
+    "                        \"description\": \"The reason for cancelling the order.\"\n",
+    "                    }\n",
+    "                },\n",
+    "                \"required\": [\"order_id\", \"reason\"],\n",
+    "                \"additionalProperties\": False\n",
+    "            }\n",
+    "        }\n",
+    "    },\n",
+    "    {\n",
+    "        \"type\": \"function\",\n",
+    "        \"function\": {\n",
+    "            \"name\": \"return_item\",\n",
+    "            \"description\": \"Process a return for an order. This should be called when a customer wants to return an item and the order has already been delivered.\",\n",
+    "            \"parameters\": {\n",
+    "                \"type\": \"object\",\n",
+    "                \"properties\": {\n",
+    "                    \"order_id\": {\n",
+    "                        \"type\": \"string\",\n",
+    "                        \"description\": \"The customer's order ID.\"\n",
+    "                    },\n",
+    "                    \"item_id\": {\n",
+    "                        \"type\": \"string\",\n",
+    "                        \"description\": \"The specific item ID the customer wants to return.\"\n",
+    "                    },\n",
+    "                    \"reason\": {\n",
+    "                        \"type\": \"string\",\n",
+    "                        \"description\": \"The reason for returning the item.\"\n",
+    "                    }\n",
+    "                },\n",
+    "                \"required\": [\"order_id\", \"item_id\", \"reason\"],\n",
+    "                \"additionalProperties\": False\n",
+    "            }\n",
+    "        }\n",
+    "    },\n",
+    "    {\n",
+    "        \"type\": \"function\",\n",
+    "        \"function\": {\n",
+    "            \"name\": \"update_shipping_address\",\n",
+    "            \"description\": \"Update the shipping address for an order that hasn't been shipped yet. Use this if the customer wants to change their delivery address.\",\n",
+    "            \"parameters\": {\n",
+    "                \"type\": \"object\",\n",
+    "                \"properties\": {\n",
+    "                    \"order_id\": {\n",
+    "                        \"type\": \"string\",\n",
+    "                        \"description\": \"The customer's order ID.\"\n",
+    "                    },\n",
+    "                    \"new_address\": {\n",
+    "                        \"type\": \"object\",\n",
+    "                        \"properties\": {\n",
+    "                            \"street\": {\n",
+    "                                \"type\": \"string\",\n",
+    "                                \"description\": \"The new street address.\"\n",
+    "                            },\n",
+    "                            \"city\": {\n",
+    "                                \"type\": \"string\",\n",
+    "                                \"description\": \"The new city.\"\n",
+    "                            },\n",
+    "                            \"state\": {\n",
+    "                                \"type\": \"string\",\n",
+    "                                \"description\": \"The new state.\"\n",
+    "                            },\n",
+    "                            \"zip\": {\n",
+    "                                \"type\": \"string\",\n",
+    "                                \"description\": \"The new zip code.\"\n",
+    "                            },\n",
+    "                            \"country\": {\n",
+    "                                \"type\": \"string\",\n",
+    "                                \"description\": \"The new country.\"\n",
+    "                            }\n",
+    "                        },\n",
+    "                        \"required\": [\"street\", \"city\", \"state\", \"zip\", \"country\"],\n",
+    "                        \"additionalProperties\": False\n",
+    "                    }\n",
+    "                },\n",
+    "                \"required\": [\"order_id\", \"new_address\"],\n",
+    "                \"additionalProperties\": False\n",
+    "            }\n",
+    "        }\n",
+    "    },\n",
+    "    # New tool: Update payment method\n",
+    "    {\n",
+    "        \"type\": \"function\",\n",
+    "        \"function\": {\n",
+    "            \"name\": \"update_payment_method\",\n",
+    "            \"description\": \"Update the payment method for an order that hasn't been completed yet. Use this if the customer wants to change their payment details.\",\n",
+    "            \"parameters\": {\n",
+    "                \"type\": \"object\",\n",
+    "                \"properties\": {\n",
+    "                    \"order_id\": {\n",
+    "                        \"type\": \"string\",\n",
+    "                        \"description\": \"The customer's order ID.\"\n",
+    "                    },\n",
+    "                    \"payment_method\": {\n",
+    "                        \"type\": \"object\",\n",
+    "                        \"properties\": {\n",
+    "                            \"card_number\": {\n",
+    "                                \"type\": \"string\",\n",
+    "                                \"description\": \"The new credit card number.\"\n",
+    "                            },\n",
+    "                            \"expiry_date\": {\n",
+    "                                \"type\": \"string\",\n",
+    "                                \"description\": \"The new credit card expiry date in MM/YY format.\"\n",
+    "                            },\n",
+    "                            \"cvv\": {\n",
+    "                                \"type\": \"string\",\n",
+    "                                \"description\": \"The new credit card CVV code.\"\n",
+    "                            }\n",
+    "                        },\n",
+    "                        \"required\": [\"card_number\", \"expiry_date\", \"cvv\"],\n",
+    "                        \"additionalProperties\": False\n",
+    "                    }\n",
+    "                },\n",
+    "                \"required\": [\"order_id\", \"payment_method\"],\n",
+    "                \"additionalProperties\": False\n",
+    "            }\n",
+    "        }\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "# Enhanced system message with guardrails\n",
+    "messages = [\n",
+    "    {\n",
+    "        \"role\": \"system\", \n",
+    "        \"content\": (\n",
+    "            \"You are a professional, empathetic, and efficient customer support assistant. Your mission is to provide fast, clear, \"\n",
+    "            \"and comprehensive assistance to customers while maintaining a warm and approachable tone. \"\n",
+    "            \"Always express empathy, especially when the user seems frustrated or concerned, and ensure that your language is polite and professional. \"\n",
+    "            \"Use simple and clear communication to avoid any misunderstanding, and confirm actions with the user before proceeding. \"\n",
+    "            \"In more complex or time-sensitive cases, assure the user that you're taking swift action and provide regular updates. \"\n",
+    "            \"Adapt to the user’s tone: remain calm, friendly, and understanding, even in stressful or difficult situations.\"\n",
+    "            \"\\n\\n\"\n",
+    "            \"Additionally, there are several important guardrails that you must adhere to while assisting users:\"\n",
+    "            \"\\n\\n\"\n",
+    "            \"1. **Confidentiality and Data Privacy**: Do not share any sensitive information about the company or other users. When handling personal details such as order IDs, addresses, or payment methods, ensure that the information is treated with the highest confidentiality. If a user requests access to their data, only provide the necessary information relevant to their request, ensuring no other user's information is accidentally revealed.\"\n",
+    "            \"\\n\\n\"\n",
+    "            \"2. **Secure Payment Handling**: When updating payment details or processing refunds, always ensure that payment data such as credit card numbers, CVVs, and expiration dates are transmitted and stored securely. Never display or log full credit card numbers. Confirm with the user before processing any payment changes or refunds.\"\n",
+    "            \"\\n\\n\"\n",
+    "            \"3. **Respect Boundaries**: If a user expresses frustration or dissatisfaction, remain calm and empathetic but avoid overstepping professional boundaries. Do not make personal judgments, and refrain from using language that might escalate the situation. Stick to factual information and clear solutions to resolve the user's concerns.\"\n",
+    "            \"\\n\\n\"\n",
+    "            \"4. **Legal Compliance**: Ensure that all actions you take comply with legal and regulatory standards. For example, if the user requests a refund, cancellation, or return, follow the company’s refund policies strictly. If the order cannot be canceled due to being shipped or another restriction, explain the policy clearly but sympathetically.\"\n",
+    "            \"\\n\\n\"\n",
+    "            \"5. **Consistency**: Always provide consistent information that aligns with company policies. If unsure about a company policy, communicate clearly with the user, letting them know that you are verifying the information, and avoid providing false promises. If escalating an issue to another team, inform the user and provide a realistic timeline for when they can expect a resolution.\"\n",
+    "            \"\\n\\n\"\n",
+    "            \"6. **User Empowerment**: Whenever possible, empower the user to make informed decisions. Provide them with relevant options and explain each clearly, ensuring that they understand the consequences of each choice (e.g., canceling an order may result in loss of loyalty points, etc.). Ensure that your assistance supports their autonomy.\"\n",
+    "            \"\\n\\n\"\n",
+    "            \"7. **No Speculative Information**: Do not speculate about outcomes or provide information that you are not certain of. Always stick to verified facts when discussing order statuses, policies, or potential resolutions. If something is unclear, tell the user you will investigate further before making any commitments.\"\n",
+    "            \"\\n\\n\"\n",
+    "            \"8. **Respectful and Inclusive Language**: Ensure that your language remains inclusive and respectful, regardless of the user’s tone. Avoid making assumptions based on limited information and be mindful of diverse user needs and backgrounds.\"\n",
+    "        )\n",
+    "    },\n",
+    "    {\n",
+    "        \"role\": \"user\", \n",
+    "        \"content\": (\n",
+    "            \"Hi, I placed an order three days ago and haven’t received any updates on when it’s going to be delivered. \"\n",
+    "            \"Could you help me check the delivery date? My order number is #9876543210. I’m a little worried because I need this item urgently.\"\n",
+    "        )\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "# Enhanced user_query2\n",
+    "user_query2 = {\n",
+    "    \"role\": \"user\", \n",
+    "    \"content\": (\n",
+    "        \"Since my order hasn't actually shipped yet, I would like to cancel it. \"\n",
+    "        \"The order number is #9876543210, and I need to cancel because I’ve decided to purchase it locally to get it faster. \"\n",
+    "        \"Can you help me with that? Thank you!\"\n",
+    "    )\n",
+    "}\n",
+    "\n",
+    "# Function to run completion with the provided message history and tools\n",
+    "def completion_run(messages, tools):\n",
+    "    completion = client.chat.completions.create(\n",
+    "        model=\"gpt-4o-mini\",\n",
+    "        tools=tools,\n",
+    "        messages=messages,\n",
+    "        tool_choice=\"required\"\n",
+    "    )\n",
+    "    usage_data = json.dumps(completion.to_dict(), indent=4)\n",
+    "    return usage_data\n",
+    "\n",
+    "# Main function to handle the two runs\n",
+    "def main(messages, tools, user_query2):\n",
+    "    # Run 1: Initial query\n",
+    "    print(\"Run 1:\")\n",
+    "    run1 = completion_run(messages, tools)\n",
+    "    print(run1)\n",
+    "\n",
+    "    # Delay for 7 seconds\n",
+    "    time.sleep(7)\n",
+    "\n",
+    "    # Append user_query2 to the message history\n",
+    "    messages.append(user_query2)\n",
+    "\n",
+    "    # Run 2: With appended query\n",
+    "    print(\"\\nRun 2:\")\n",
+    "    run2 = completion_run(messages, tools)\n",
+    "    print(run2)\n",
+    "\n",
+    "\n",
+    "# Run the main function\n",
+    "main(messages, tools, user_query2)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Example 2: Images \n",
+    "\n",
+    "In our second example we include multiple image URLs of grocery items in the messages array, along with a user query, run three times with delays. Images—whether linked or encoded in base64 within user messages—qualify for caching. Make sure the detail parameter remains consistent, as it affects how images are tokenized. Note that GPT-4o-mini adds extra tokens to cover image processing costs, even though it uses a low-cost token model for text. Caching discounts are based on the actual number of tokens processed, including those used for images, which also count toward your rate limits.\n",
+    "\n",
+    "The output for this example shows that a cache was hit for the second run, however it was not hit for the third run because of a different first url (eggs_url instead of veggie_url), even though the user query is the same. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{\n",
+      "    \"id\": \"chatcmpl-ADeV3IrUqhpjMXEgv29BFHtTQ0Pzt\",\n",
+      "    \"choices\": [\n",
+      "        {\n",
+      "            \"finish_reason\": \"stop\",\n",
+      "            \"index\": 0,\n",
+      "            \"logprobs\": null,\n",
+      "            \"message\": {\n",
+      "                \"content\": \"The images show the following types of sauces:\\n\\n1. **Soy Sauce** - Kikkoman brand.\\n2. **Worcester Sauce** - Appel brand, listed as \\\"Dresdner Art.\\\"\\n3. **Tabasco Sauce** - Original pepper sauce.\\n\\nThe second image shows various vegetables, not sauces.\",\n",
+      "                \"refusal\": null,\n",
+      "                \"role\": \"assistant\"\n",
+      "            }\n",
+      "        }\n",
+      "    ],\n",
+      "    \"created\": 1727817309,\n",
+      "    \"model\": \"gpt-4o-2024-08-06\",\n",
+      "    \"object\": \"chat.completion\",\n",
+      "    \"system_fingerprint\": \"fp_2f406b9113\",\n",
+      "    \"usage\": {\n",
+      "        \"completion_tokens\": 65,\n",
+      "        \"prompt_tokens\": 1548,\n",
+      "        \"total_tokens\": 1613,\n",
+      "        \"prompt_tokens_details\": {\n",
+      "            \"cached_tokens\": 0\n",
+      "        },\n",
+      "        \"completion_tokens_details\": {\n",
+      "            \"reasoning_tokens\": 0\n",
+      "        }\n",
+      "    }\n",
+      "}\n",
+      "{\n",
+      "    \"id\": \"chatcmpl-ADeVRSI6zFINkx99k7V6ux1v5iF5f\",\n",
+      "    \"choices\": [\n",
+      "        {\n",
+      "            \"finish_reason\": \"stop\",\n",
+      "            \"index\": 0,\n",
+      "            \"logprobs\": null,\n",
+      "            \"message\": {\n",
+      "                \"content\": \"The images show different types of items. In the first image, you'll see bottles of sauces like soy sauce, Worcester sauce, and Tabasco. The second image features various vegetables, including:\\n\\n1. Napa cabbage\\n2. Kale\\n3. Carrots\\n4. Bok choy\\n5. Swiss chard\\n6. Leeks\\n7. Parsley\\n\\nThese vegetables are arranged on shelves in a grocery store setting.\",\n",
+      "                \"refusal\": null,\n",
+      "                \"role\": \"assistant\"\n",
+      "            }\n",
+      "        }\n",
+      "    ],\n",
+      "    \"created\": 1727817333,\n",
+      "    \"model\": \"gpt-4o-2024-08-06\",\n",
+      "    \"object\": \"chat.completion\",\n",
+      "    \"system_fingerprint\": \"fp_2f406b9113\",\n",
+      "    \"usage\": {\n",
+      "        \"completion_tokens\": 86,\n",
+      "        \"prompt_tokens\": 1548,\n",
+      "        \"total_tokens\": 1634,\n",
+      "        \"prompt_tokens_details\": {\n",
+      "            \"cached_tokens\": 1280\n",
+      "        },\n",
+      "        \"completion_tokens_details\": {\n",
+      "            \"reasoning_tokens\": 0\n",
+      "        }\n",
+      "    }\n",
+      "}\n",
+      "{\n",
+      "    \"id\": \"chatcmpl-ADeVphj3VALQVrdnt2efysvSmdnBx\",\n",
+      "    \"choices\": [\n",
+      "        {\n",
+      "            \"finish_reason\": \"stop\",\n",
+      "            \"index\": 0,\n",
+      "            \"logprobs\": null,\n",
+      "            \"message\": {\n",
+      "                \"content\": \"The second image shows three types of sauces:\\n\\n1. Soy Sauce (Kikkoman)\\n2. Worcestershire Sauce\\n3. Tabasco Sauce\",\n",
+      "                \"refusal\": null,\n",
+      "                \"role\": \"assistant\"\n",
+      "            }\n",
+      "        }\n",
+      "    ],\n",
+      "    \"created\": 1727817357,\n",
+      "    \"model\": \"gpt-4o-2024-08-06\",\n",
+      "    \"object\": \"chat.completion\",\n",
+      "    \"system_fingerprint\": \"fp_2f406b9113\",\n",
+      "    \"usage\": {\n",
+      "        \"completion_tokens\": 29,\n",
+      "        \"prompt_tokens\": 1548,\n",
+      "        \"total_tokens\": 1577,\n",
+      "        \"prompt_tokens_details\": {\n",
+      "            \"cached_tokens\": 0\n",
+      "        },\n",
+      "        \"completion_tokens_details\": {\n",
+      "            \"reasoning_tokens\": 0\n",
+      "        }\n",
+      "    }\n",
+      "}\n"
+     ]
+    }
+   ],
+   "source": [
+    "sauce_url = \"https://upload.wikimedia.org/wikipedia/commons/thumb/9/97/12-04-20-saucen-by-RalfR-15.jpg/800px-12-04-20-saucen-by-RalfR-15.jpg\"\n",
+    "veggie_url = \"https://upload.wikimedia.org/wikipedia/commons/thumb/3/31/Veggies.jpg/800px-Veggies.jpg\"\n",
+    "eggs_url= \"https://upload.wikimedia.org/wikipedia/commons/thumb/a/a2/Egg_shelf.jpg/450px-Egg_shelf.jpg\"\n",
+    "milk_url= \"https://upload.wikimedia.org/wikipedia/commons/thumb/c/cd/Lactaid_brand.jpg/800px-Lactaid_brand.jpg\"\n",
+    "\n",
+    "def multiimage_completion(url1, url2, user_query):\n",
+    "    completion = client.chat.completions.create(\n",
+    "    model=\"gpt-4o-2024-08-06\",\n",
+    "    messages=[\n",
+    "        {\n",
+    "        \"role\": \"user\",\n",
+    "        \"content\": [\n",
+    "            {\n",
+    "            \"type\": \"image_url\",\n",
+    "            \"image_url\": {\n",
+    "                \"url\": url1,\n",
+    "                \"detail\": \"high\"\n",
+    "            },\n",
+    "            },\n",
+    "            {\n",
+    "            \"type\": \"image_url\",\n",
+    "            \"image_url\": {\n",
+    "                \"url\": url2,\n",
+    "                \"detail\": \"high\"\n",
+    "            },\n",
+    "            },\n",
+    "            {\"type\": \"text\", \"text\": user_query}\n",
+    "        ],\n",
+    "        }\n",
+    "    ],\n",
+    "    max_tokens=300,\n",
+    "    )\n",
+    "    print(json.dumps(completion.to_dict(), indent=4))\n",
+    "    \n",
+    "\n",
+    "def main(sauce_url, veggie_url):\n",
+    "    multiimage_completion(sauce_url, veggie_url, \"Please list the types of sauces are shown in these images\")\n",
+    "    #delay for 20 seconds\n",
+    "    time.sleep(20)\n",
+    "    multiimage_completion(sauce_url, veggie_url, \"Please list the types of vegetables are shown in these images\")\n",
+    "    time.sleep(20)\n",
+    "    multiimage_completion(milk_url, sauce_url, \"Please list the types of sauces are shown in these images\")\n",
+    "\n",
+    "if __name__ == \"__main__\":\n",
+    "    main(sauce_url, veggie_url)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "## Overall tips \n",
+    "\n",
+    "To get the most out of prompt caching, consider following these best practices:\n",
+    "\n",
+    "- Place static or frequently reused content at the beginning of prompts: This helps ensure better cache efficiency by keeping dynamic data towards the end of the prompt.\n",
+    "\n",
+    "-  Maintain consistent usage patterns: Prompts that aren't used regularly are automatically removed from the cache. To prevent cache evictions, maintain consistent usage of prompts.\n",
+    "\n",
+    "- Monitor key metrics: Regularly track cache hit rates, latency, and the proportion of cached tokens. Use these insights to fine-tune your caching strategy and maximize performance.\n",
+    "\n",
+    "By implementing these practices, you can take full advantage of prompt caching, ensuring that your applications are both responsive and cost-efficient. A well-managed caching strategy will significantly reduce processing times, lower costs, and help maintain smooth user experiences."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/Prompt_Caching_201.ipynb

@@ -0,0 +1,309 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "f325a442",
+   "metadata": {},
+   "source": [
+    "# Prompt Caching 201\n",
+    "> **Note:** This guide applies only to models before GPT-5.6. For GPT-5.6 and later, see the [Prompt Caching guide](https://developers.openai.com/api/docs/guides/prompt-caching).\n",
+    "\n",
+    "A practical guide to prompt caching: fundamentals, performance impact, measurement, and optimization strategies.\n",
+    "## 1. Prompt Caching Basics\n",
+    "Model prompts often include repeated content - such as system instructions, tools, and messages. When a request contains a prefix the system has recently processed, OpenAI can route it to a server that already computed that prefix, allowing the model to reuse prior work instead of recomputing it from scratch.\n",
+    "Prompt Caching can reduce time-to-first-token latency by up to 80% and input token costs by up to 90%. It works automatically on all API requests and has no additional fees. \n",
+    "The goal of this cookbook is to go deeper on optimizing for cache hits. Review our [API docs](https://developers.openai.com/api/docs/guides/prompt-caching) and [Prompt Caching 101](https://developers.openai.com/cookbook/examples/prompt_caching101/) for a great overview - let’s dive in! \n",
+    "### 1.1 Basics\n",
+    "- Cache hits require an exact, repeated prefix match and works automatically for prompts containing 1024 tokens or more, with cache hits occurring in increments of 128 tokens.  \n",
+    "- The entire request prefix is cacheable: messages, images, audio, tool definitions, and structured output schemas.\n",
+    "- In-memory prompt caching works automatically on all your API requests. Extended prompt caching increases that to 24hrs. \n",
+    "- Caching only works if two requests share the same prefix and land on the same machine. Take advantage of the optional parameter `prompt_cache_key` for traffic that shares common prefixes to improve that routing.\n",
+    "- Carefully consider the impact of caching from context engineering techniques like compaction.\n",
+    "- Monitor caching, cost and latency via request logs or the Usage dashboard while iterating.\n",
+    "\n",
+    "## 2. Why caching matters\n",
+    "### 2.1 Core Technical Reason: Skipping Prefill Compute\n",
+    "The forward pass through transformer layers over the input tokens is the main driver of inference cost and latency. In the transformer stack, prompt caching applies specifically to the key and value projections inside the attention layers. Attention is the mechanism that allows models to weigh the importance of different parts of an input sequence when processing a specific token, enabling them to capture context and long-range dependencies.\n",
+    "\n",
+    "When you make a request to OpenAI, we'll create token embeddings that are then transformed into three vectors: a query (Q) that represents what that token would need to self-contextualize, a key (K) that encodes what each token represents, and a value (V) that contains the information that can be incorporated if it were to be relevant. The model compares the current token’s query to all tokens’ keys (via dot products) to produce attention scores. After a softmax, these scores become weights that determine how much each token’s value contributes to the updated representation. For example, in the phrase “she sat by the river bank,” the word “river” provides important context to “bank,” pushing its meaning toward the aquatic rather than the financial. The KV cache stores the key and value tensors for that prefix across all layers and heads.\n",
+    "\n",
+    "If you then make another request — “she sat by the river bank on a cloudy day” — the first part of the prefix is identical. When processing the new, unseen suffix, the model reuses the cached tensors and only computes attention for the new tokens.\n",
+    "\n",
+    "Crucially, the strong semantic relationship between “river” and “bank” has already been encoded into those cached key and value representations during the earlier forward pass. The model does not need to recompute how “bank” aligns with “river.” Instead, new tokens like “cloudy” and “day” generate their own queries and attend over the cached keys from the prefix. In other words, the semantic groundwork has already been laid; the model simply builds on top of that existing attention state rather than recalculating it from scratch.\n",
+    "\n",
+    "### 2.2 Cost Impact\n",
+    "Cache discounts can be significant. Discount magnitude varies by model family - as our inference stack has become more efficient, our newest models have been able to offer steeper cache discounts. Here are some examples, but see our pricing page for all models. Prompt Caching is enabled for all recent models, gpt-4o and newer. \n",
+    "| Model | Input <br>(per 1M tokens) | Cached input <br>(per 1M tokens) | Caching Discount |\n",
+    "| --- | --- | --- | --- |\n",
+    "| gpt-4o | $2.50 | $1.25 | 50.00% |\n",
+    "| gpt-4.1 | $2.00 | $0.50 | 75.00% |\n",
+    "| gpt-5-nano | $0.05 | $0.005 | 90.00% |\n",
+    "| gpt-5.2 | $1.75 | $0.175 | 90.00% |\n",
+    "| gpt-realtime (audio) | $32.00 | $0.40 | 98.75% |\n",
+    "\n",
+    "\n",
+    "### 2.3 Latency Impact\n",
+    "Reducing time-to-first-token (TTFT) is a primary motivation for improving cache rates. Cached tokens can reduce latency by up to ~80%.\n",
+    "Caching keeps latency roughly proportional to generated output length rather than total conversation length because the full historical context is not re-prefilled. When we get cache hits, sampling the model is linear rather than quadratic.\n",
+    "I ran a series of prompts 2300 times and plotted the cached vs uncached requests. For the shortest prompts (1024 tokens), cached requests are 7% faster, but at the longer end (150k+ tokens) we’re seeing 67% faster TTFT.  The longer the input, the bigger the impact of caching on that first-token latency. \n",
+    "\n",
+    "![Figure 2](../images/prompt-caching-201/figure-2.svg)\n",
+    "\n",
+    "## 3. Measure caching first (so you can iterate)\n",
+    "### 3.1 Per-request: `cached_tokens`\n",
+    "Responses include usage fields indicating how many prompt tokens were served from the cache. All requests will display a `cached_tokens` field of the `usage.prompt_tokens_details` [Response](https://developers.openai.com/api/docs/api-reference/responses/object) or [Chat](https://developers.openai.com/api/docs/api-reference/chat/object) object indicating how many of the prompt tokens were cached.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "```\n",
+    "\"usage\": {\n",
+    "  \"prompt_tokens\": 2006,\n",
+    "  \"completion_tokens\": 300,\n",
+    "  \"total_tokens\": 2306,\n",
+    "  \"prompt_tokens_details\": {\n",
+    "    \"cached_tokens\": 1920\n",
+    "  },\n",
+    "  \"completion_tokens_details\": {\n",
+    "    \"reasoning_tokens\": 0,\n",
+    "    \"accepted_prediction_tokens\": 0,\n",
+    "    \"rejected_prediction_tokens\": 0\n",
+    "  }\n",
+    "}\n",
+    "```\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "You can also get a high level overview by filtering selected cached / uncached tokens for the measures in the usage dashboard\n",
+    "\n",
+    "![Figure 3](../images/prompt-caching-201/figure-3.png)\n",
+    "\n",
+    "## 4. Improve cache hit rate (tactical playbook)\n",
+    "\n",
+    "What’s a cache hit? That’s when a request starts with the same prefix as a previous request, allowing the system to reuse previously computed key/value tensors instead of recomputing them.  Increasing the cache hit rate — maximizing how often prefixes can be reused — directly improves both performance and efficiency, so maximizing this is our goal!\n",
+    "\n",
+    "\n",
+    "### 4.1 Send a Prompt over 1024 tokens\n",
+    "It can feel counterintuitive, but in some cases, making your prompt slightly longer can reduce overall cost.\n",
+    "Say you have a 900 token prompt -  you’ll never get a cache hit.  If you lengthen your prompt to 1100 tokens and get a 50% cache rate you’ll save 33% on the token costs. If you get a 70% cache rate, you’d save 55%. Once you cross the caching threshold and achieve meaningful reuse, the marginal cost of those repeated tokens drops substantially. That means a slightly longer but stable prefix can be cheaper than a shorter prompt that never caches.\n",
+    "\n",
+    "### 4.2 Stabilize the Prefix\n",
+    "This is the lowest-effort, highest-impact optimization: keep the early portion of the prompt stable.\n",
+    "Place durable content at the beginning:\n",
+    "- Instructions\n",
+    "- Tool definitions\n",
+    "- Schemas\n",
+    "\n",
+    "Move volatile content (e.g., user input, dynamic values, session-specific data) to the end. Even small changes in early tokens will invalidate exact prefix matching and prevent cache hits.\n",
+    "\n",
+    "**Learnings from Codex**\n",
+    "When our engineering team outlined how they [architected the Codex agent loop](https://openai.com/index/unrolling-the-codex-agent-loop/), they emphasized prompt structure as a first-class performance surface and caching as a top priority. In the Codex CLI, system instructions, tool definitions, sandbox configuration, and environment context are kept identical and consistently ordered between requests to preserve long, stable prompt prefixes. The agent loop appends new messages (rather than modifying earlier ones) when runtime configurations change mid-conversation (e.g. new working directory or approval mode). By avoiding changes to the original prefix, the system preserves exact-prefix matches, which are required for prompt cache hits. \n",
+    "\n",
+    "**Tip: Use metadata**\n",
+    "We’ve seen customers accidentally invalidate their cache by including a timestamp early in their request for later lookup/debugging. Move that to `metadata` where it will not impact the cache!\n",
+    "\n",
+    "### 4.3 Keep Tools and Schemas Identical\n",
+    "Tools, schemas, and their ordering contribute to the cached prefix - they get injected before developer instructions which means that changing them would invalidate the cache. This includes: \n",
+    "- Schema key changes\n",
+    "- Tool ordering changes\n",
+    "- Changes to instructions\n",
+    "\n",
+    "\n",
+    "**Tip: Adjust tools without breaking prompt caching**\n",
+    "Leverage [`allowed_tools`](https://developers.openai.com/api/docs/guides/function-calling/#tool-choice) `tool_choice` option. It allows you to restrict the tools the model can call on a request without changing the tools array and busting the cache. List your full toolkit in tools, and then use an `allowed_tools` block to specify which tool can be used on a single turn.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "```\n",
+    "# Static “tools” stay in the cached prompt prefix: \n",
+    "\n",
+    "tools = [get_weather_def, get_location_def, calendar_def, …] \n",
+    "\n",
+    "# Per-call “allowed_tools” only lives in request metadata, not in the prefix: \n",
+    "\n",
+    "allowed_tools = {\"mode\":\"auto\", \"tools\":[\"get_weather\",\"get_location\"]}\n",
+    "```\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "\n",
+    "### 4.4 Use `prompt_cache_key` to Improve Routing Stickiness\n",
+    "Caching only works if two requests share the same prefix and land on the same machine. Requests are routed to inference engines based on a hash of the first ~256 tokens of the prompt. When you provide a `prompt_cache_key`, it is combined with that hash to increase routing stickiness - meaning requests with the same prefix are more likely to land on the same engine and reuse cached KV state.  It’s effective - one of our coding customers saw an improved hit rate from 60% to 87% when they started using `prompt_cache_key`. \n",
+    "\n",
+    "`prompt_cache_key` is especially useful when you are sending different requests that have the same initial set of context (i.e. first 256 tokens) but then vary later as it lets you intentionally group related requests. \n",
+    "\n",
+    "Inference engines can handle roughly ~15 requests per minute per prefix + `prompt_cache_key` combination. If traffic exceeds that rate — for example, if you send thousands of requests sharing the same prefix and key — the system will distribute the excess requests across additional machines for load balancing. Each new machine is a one-time cache miss. \n",
+    "\n",
+    "That means you should choose a key granularity that can keep each prefix + key combination below ~15 RPM.\n",
+    "\n",
+    "For coding use cases we’ve seen:\n",
+    "- Per-user keys improve reuse across related conversations (e.g., working in the same codebase).\n",
+    "- Per-conversation keys scale better when users run many unrelated threads in parallel.\n",
+    "Grouping several users to share a key can also be a good approach. A simple algorithm for this would be to hash and mod a user id by number of \"buckets” to aim close to 15RPM to aim for to maximize caching performance.\n",
+    "\n",
+    "\n",
+    "**Tip: Test Flex Processing instead of the Batch API**\n",
+    "If you have latency insensitive workflow you may be using the Batch API. For more flexibility around caching, consider using Flex Processing. Flex offers the same 50% token discount as Batch but runs through the Responses API with `service_tier=\"flex\"` specified per request. This gives you more flexibility:\n",
+    "\n",
+    "- Control over request rate (RPM)\n",
+    "- Access to extended prompt caching\n",
+    "- Ability to include a prompt_cache_key\n",
+    "\n",
+    "Because you can tune routing and cache locality more precisely, Flex can achieve higher cache hit rates in some workloads. It is particularly well suited for prototyping or production workloads that are not inference-intensive but still benefit from cost optimization.\n",
+    "\n",
+    "I ran a head-to-head test of 10,000 identical requests. Flex (using extended prompt caching and `prompt_cache_key`) produced an 8.5% increase in cache hit rate compared to the Batch job. That improvement translates into a 23% reduction in input token cost. \n",
+    "\n",
+    "![Figure 4](../images/prompt-caching-201/figure-4.svg)\n",
+    "\n",
+    "There isn’t model parity for caching on the Batch API - pre-GPT-5 models are not supported, so if you’re using o3 or o4-mini, you should consider switching to Flex to take advantage of caching. Check the most up to date info on this in our [pricing docs](https://developers.openai.com/api/docs/pricing?latest-pricing=flex). \n",
+    "\n",
+    "\n",
+    "**Insight: `prompt_cache_key` as shard key**\n",
+    "It might be helpful to think of the `prompt_cache_key` as a database shard key when thinking about how we route the request on the backend - the considerations are very similar when optimizing for the parameter. Like shard keys, granularity is a balancing act. Each machine can only handle about ~15 RPM for a given prefix, so if you use the same key on too many requests, requests will overflow to multiple machines. For each new machine, you’ll start the cache anew. On the other hand, if your key is too narrow, traffic spreads out across machines and you lose the benefit of cache reuse. Routing is still load-balanced - `prompt_cache_key` increases the chance similar prompts hit the same server but does not guarantee stickiness - caching is always best-effort!\n",
+    "\n",
+    "### 4.5 Use the Responses API instead of Chat Completions\n",
+    "As we outlined in [Why we built the Responses API](https://developers.openai.com/blog/responses-api/), our internal benchmarks show a 40-80% better cache utilization on requests when compared to Chat Completions. \n",
+    "This is because, unlike Chat Completions, the raw chain of thought tokens get persisted in the Responses API between turns via `previous_response_id` (or [encrypted reasoning items](https://platform.openai.com/docs/guides/reasoning?api-mode=responses#encrypted-reasoning-items) if you’re stateless). Chat Completions does not offer a way to persist these tokens. If you aren't leveraging reasoning models, there's no caching improvement. [Better performance from reasoning models using the Responses API](https://developers.openai.com/cookbook/examples/responses_api/reasoning_items#caching) is an excellent guide to understanding this in more depth. \n",
+    "\n",
+    "### 4.6 Be thoughtful about Context Engineering\n",
+    "At its core, context engineering is about deciding what goes into the model’s input on each request. Every model has a fixed context window, but curating what you pass on each request isn’t just about staying under the limit. As the input grows, you’re not only approaching truncation - you’re also increasing replay cost and asking the model to distribute its attention across more tokens. Effective context engineering is the practice of managing that constraint intentionally. \n",
+    "However, when you drop, summarize or compact earlier turns in a conversation, you’ll break the cache. In that regard, context engineering and prompt caching are inherently at odds - one prioritizes dynamism, the other stability.\n",
+    "\n",
+    "With the rise of longer running agents and native compaction tools, it’s important to keep caching in mind when architecting to ensure the right balance of cost versus intelligence savings. \n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "```\n",
+    "response = client.responses.create(\n",
+    "        model=\"gpt-5.2-codex\",\n",
+    "        input=conversation,\n",
+    "        store=False,\n",
+    "        context_management=[{\"type\": \"compaction\", \"compact_threshold\": 100000}],\n",
+    "    )\n",
+    "```\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "A common failure mode is an overgrown tool set that introduces ambiguity about which tool should be invoked. Keeping the tool surface minimal improves decision quality and long-term context. When curating tools on a per-request basis, use tip 4.3 and leverage the [`allowed_tools`](https://developers.openai.com/api/docs/guides/function-calling/#tool-choice) `tool_choice` option for pruning.\n",
+    "\n",
+    "\n",
+    "**Practical rule**\n",
+    "Leverage your evals to choose the compaction method and frequency that balances cost (both from reducing total input tokens via truncation/summarization as well as caching) and intelligence gained from careful context engineering\n",
+    "\n",
+    "## 5. Troubleshooting: why you might see lower caching:\n",
+    "Common causes:\n",
+    "- Tool or response format schema changes\n",
+    "- Naive truncation from hitting the models' context window\n",
+    "- Changes to instructions or system prompts\n",
+    "- Changes to reasoning effort\n",
+    "- Cache Expiration: too much time passes and the saved prefix is dropped.\n",
+    "- Adding in a space, timestamp or other dynamic content\n",
+    "- Using Chat Completions with reasoning models, since the hidden chain-of-thought tokens are dropped\n",
+    "\n",
+    "## 6. Extended Prompt Caching & Zero Data Retention\n",
+    "\n",
+    "[Extended Prompt Caching](https://developers.openai.com/api/docs/guides/prompt-caching#extended-prompt-cache-retention) works by offloading the key/value tensors to GPU-local storage when memory is full, significantly increasing the storage capacity available for caching. For `gpt-5.5`, `gpt-5.5-pro`, and all future models, the default is `24h` and `in_memory` is not supported."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "```\n",
+    "{\n",
+    "  \"model\": \"gpt-5.1\",\n",
+    "  \"input\": \"Write me a haiku about froge...\",\n",
+    "  \"prompt_cache_retention\": \"24h\"\n",
+    "}\n",
+    "```\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "KV tensors are the intermediate representation from the model’s attention layers produced during prefill. Only the key/value tensors may be persisted in local storage; all original input content is only retained in memory.\n",
+    "\n",
+    "**Insight: What Is Actually Cached?**\n",
+    "The KV cache just holds the model’s key/value tensors (linear projections of the hidden‑states) so we can reuse them on the next inference step. The KV cache is an intermediate representation. It’s essentially just a bunch of numbers internal to the model - that means no raw text/multi-modal inputs are ever stored, regardless of retention policy.\n",
+    "\n",
+    "## 7. Realtime API\n",
+    "\n",
+    "Caching in the Realtime API works the same as with the Responses API - all the audio, text or images passed in will be cached, and any change to the prefix will break the cache.  However, given the shorter context window of the Realtime API (currently 32k), managing truncation is especially relevant.  A 32k context model with 4,096 max output tokens can only include 28,224 tokens in the context before truncation occurs. \n",
+    "\n",
+    "### 7.1 Retention Ratio\n",
+    "By default `(truncation: \"auto\")`, the server removes just enough old messages to fit within the context window. This “just-in-time” pruning shifts the start of the conversation slightly on every turn once you’re over the limit. However, this naive strategy causes frequent cache misses.\n",
+    "The [`retention_ratio`](https://developers.openai.com/api/docs/guides/realtime-costs/#truncation) setting changes this by letting you control how much of the earlier context to keep vs. drop. It’s a configurable truncation strategy that lets developers control how much of the token context window is retained, balancing context preservation with cache-hit optimization.\n",
+    "For example, `retention_ratio: 0.7` means that when truncation happens, the system keeps roughly 70% of the existing conversation window and drops the oldest ~30%. The drop happens in one larger truncation event (instead of small incremental removals).\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "```\n",
+    "{\n",
+    "  \"event\": \"session.update\",\n",
+    "  \"session\": {\n",
+    "    \"truncation\": {\n",
+    "      \"type\": \"retention_ratio\",\n",
+    "      \"retention_ratio\": 0.7\n",
+    "    }\n",
+    "  }\n",
+    "}\n",
+    "```\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "This creates a more stable prefix that survives across multiple turns, reducing repeated cache busting.\n",
+    "The trade-off is that truncation happens in bigger chunks, so you can lose more conversation history at once. That means the model may suddenly “forget” earlier parts of the dialogue sooner than with gradual truncation. You’re effectively trading off memory depth for cache stability (more consistent prefix).\n",
+    "Here’s what this impact looks like in the naive per-turn truncation approach vs using `retention_ratio`. \n",
+    "\n",
+    "![Figure 5](../images/prompt-caching-201/figure-5.svg)\n",
+    "\n",
+    "\n",
+    "## Conclusion\n",
+    "Prompt caching is one of the highest-leverage optimizations available on the OpenAI platform. When your prefixes are stable and your routing is well-shaped, you can materially reduce both cost and latency without changing model behavior or quality.\n",
+    "The key ideas are simple but powerful: stabilize the prefix, monitor `cached_tokens`, be mindful of the ~15 RPM limit, and use `prompt_cache_key` thoughtfully. For higher-volume or latency-insensitive workloads, consider Flex Processing and extended retention to further improve cache locality.\n",
+    "Caching is best-effort and routing-aware, but when engineered intentionally, it can deliver dramatic improvements in efficiency. Treat it like any other performance system: measure first, iterate deliberately, and design your prompts with reuse in mind.\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

examples/Question_answering_using_a_search_API.ipynb

@@ -0,0 +1,529 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Question answering using a search API and re-ranking\n",
+    "\n",
+    "Searching for relevant information can sometimes feel like looking for a needle in a haystack, but don’t despair, GPTs can actually do a lot of this work for us. In this guide we explore a way to augment existing search systems with various AI techniques, helping us sift through the noise.\n",
+    "\n",
+    "Two ways of retrieving information for GPT are:\n",
+    "\n",
+    "1. **Mimicking Human Browsing:** [GPT triggers a search](https://openai.com/blog/chatgpt-plugins#browsing), evaluates the results, and modifies the search query if necessary. It can also follow up on specific search results to form a chain of thought, much like a human user would do.\n",
+    "2. **Retrieval with Embeddings:** Calculate [embeddings](https://platform.openai.com/docs/guides/embeddings) for your content and a user query, and then [retrieve the content](Question_answering_using_embeddings.ipynb) most related as measured by cosine similarity. This technique is [used heavily](https://blog.google/products/search/search-language-understanding-bert/) by search engines like Google.\n",
+    "\n",
+    "These approaches are both promising, but each has their shortcomings: the first one can be slow due to its iterative nature and the second one requires embedding your entire knowledge base in advance, continuously embedding new content and maintaining a vector database.\n",
+    "\n",
+    "By combining these approaches, and drawing inspiration from [re-ranking](https://www.sbert.net/examples/applications/retrieve_rerank/README.html) methods, we identify an approach that sits in the middle. **This approach can be implemented on top of any existing search system, like the Slack search API, or an internal ElasticSearch instance with private data**. Here’s how it works:\n",
+    "\n",
+    "![search_augmented_by_query_generation_and_embeddings_reranking.png](../images/search_rerank_answer.png)\n",
+    "\n",
+    "**Step 1: Search**\n",
+    "\n",
+    "1.  User asks a question.\n",
+    "2.  GPT generates a list of potential queries.\n",
+    "3.  Search queries are executed in parallel.\n",
+    "\n",
+    "**Step 2: Re-rank**\n",
+    "\n",
+    "1.  Embeddings for each result are used to calculate semantic similarity to a generated hypothetical ideal answer to the user question.\n",
+    "2.  Results are ranked and filtered based on this similarity metric.\n",
+    "\n",
+    "**Step 3: Answer**\n",
+    "\n",
+    "1.  Given the top search results, the model generates an answer to the user’s question, including references and links.\n",
+    "\n",
+    "This hybrid approach offers relatively low latency and can be integrated into any existing search endpoint, without requiring the upkeep of a vector database. Let's dive into it! We will use the [News API](https://newsapi.org/) as an example domain to search over.\n",
+    "\n",
+    "## Setup\n",
+    "\n",
+    "In addition to your `OPENAI_API_KEY`, you'll have to include a `NEWS_API_KEY` in your environment. You can get an API key [here](https://newsapi.org/).\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%capture\n",
+    "%env NEWS_API_KEY = YOUR_NEWS_API_KEY\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Dependencies\n",
+    "from datetime import date, timedelta  # date handling for fetching recent news\n",
+    "from IPython import display  # for pretty printing\n",
+    "import json  # for parsing the JSON api responses and model outputs\n",
+    "from numpy import dot  # for cosine similarity\n",
+    "from openai import OpenAI\n",
+    "import os  # for loading environment variables\n",
+    "import requests  # for making the API requests\n",
+    "from tqdm.notebook import tqdm  # for printing progress bars\n",
+    "\n",
+    "client = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))\n",
+    "\n",
+    "# Load environment variables\n",
+    "news_api_key = os.getenv(\"NEWS_API_KEY\")\n",
+    "\n",
+    "GPT_MODEL = \"gpt-3.5-turbo\"\n",
+    "\n",
+    "\n",
+    "# Helper functions\n",
+    "def json_gpt(input: str):\n",
+    "    completion = client.chat.completions.create(model=GPT_MODEL,\n",
+    "    messages=[\n",
+    "        {\"role\": \"system\", \"content\": \"Output only valid JSON\"},\n",
+    "        {\"role\": \"user\", \"content\": input},\n",
+    "    ],\n",
+    "    temperature=0.5)\n",
+    "\n",
+    "    text = completion.choices[0].message.content\n",
+    "    parsed = json.loads(text)\n",
+    "\n",
+    "    return parsed\n",
+    "\n",
+    "\n",
+    "def embeddings(input: list[str]) -> list[list[str]]:\n",
+    "    response = client.embeddings.create(model=\"text-embedding-3-small\", input=input)\n",
+    "    return [data.embedding for data in response.data]"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Search\n",
+    "\n",
+    "It all starts with a user question.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# User asks a question\n",
+    "USER_QUESTION = \"Who won the NBA championship? And who was the MVP? Tell me a bit about the last game.\""
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now, in order to be as exhaustive as possible, we use the model to generate a list of diverse queries based on this question.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "['NBA championship winner',\n",
+       " 'MVP of NBA championship',\n",
+       " 'Last game of NBA championship',\n",
+       " 'NBA finals winner',\n",
+       " 'Most valuable player of NBA championship',\n",
+       " 'Finals game of NBA',\n",
+       " 'Who won the NBA finals',\n",
+       " 'NBA championship game summary',\n",
+       " 'NBA finals MVP',\n",
+       " 'Champion of NBA playoffs',\n",
+       " 'NBA finals last game highlights',\n",
+       " 'NBA championship series result',\n",
+       " 'NBA finals game score',\n",
+       " 'NBA finals game recap',\n",
+       " 'NBA champion team and player',\n",
+       " 'NBA finals statistics',\n",
+       " 'NBA championship final score',\n",
+       " 'NBA finals best player',\n",
+       " 'NBA playoffs champion and MVP',\n",
+       " 'NBA finals game analysis',\n",
+       " 'Who won the NBA championship? And who was the MVP? Tell me a bit about the last game.']"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "QUERIES_INPUT = f\"\"\"\n",
+    "You have access to a search API that returns recent news articles.\n",
+    "Generate an array of search queries that are relevant to this question.\n",
+    "Use a variation of related keywords for the queries, trying to be as general as possible.\n",
+    "Include as many queries as you can think of, including and excluding terms.\n",
+    "For example, include queries like ['keyword_1 keyword_2', 'keyword_1', 'keyword_2'].\n",
+    "Be creative. The more queries you include, the more likely you are to find relevant results.\n",
+    "\n",
+    "User question: {USER_QUESTION}\n",
+    "\n",
+    "Format: {{\"queries\": [\"query_1\", \"query_2\", \"query_3\"]}}\n",
+    "\"\"\"\n",
+    "\n",
+    "queries = json_gpt(QUERIES_INPUT)[\"queries\"]\n",
+    "\n",
+    "# Let's include the original question as well for good measure\n",
+    "queries.append(USER_QUESTION)\n",
+    "\n",
+    "queries"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The queries look good, so let's run the searches.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "6c750d6e5b2846b6834bad47ea5bef8b",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "  0%|          | 0/21 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Total number of articles: 554\n",
+      "Top 5 articles of query 1: \n",
+      "\n",
+      "Title: Nascar takes on Le Mans as LeBron James gets centenary race under way\n",
+      "Description: <ul><li>Nascar has presence at iconic race for first time since 1976</li><li>NBA superstar LeBron James waves flag as honorary starter</li></ul>The crowd chanted “U-S-A! U-S-A!” as Nascar driver lineup for the 24 Hours of Le Mans passed through the city cente…\n",
+      "Content: The crowd chanted U-S-A! U-S-A! as Nascar driver lineup for the 24 Hours of Le Mans passed through t...\n",
+      "\n",
+      "Title: NBA finals predictions: Nuggets or Heat? Our writers share their picks\n",
+      "Description: Denver or Miami? Our contributors pick the winner, key players and dark horses before the NBA’s grand finale tips offA lot has been made of the importance of a balanced roster with continuity, but, somehow, still not enough. The Nuggets are the prime example …\n",
+      "Content: The Nuggets are here because \n",
+      "A lot has been made of the importance of a balanced roster with conti...\n",
+      "\n",
+      "Title: Unboxing: Michelob ULTRA and Artist Futura Enshrine the NBA Championship In Custom Hand-Painted Bottles\n",
+      "Description: As the 2022-2023 NBA Championship nears the end, Michelob ULTRA brings joy to sports fans who will gather to watch the showdown between the Denver Nuggets and Miami Heat. The beermaker teamed up with artist Futura to remix its newly-designed 2023 Champ Bottle…\n",
+      "Content: As the 2022-2023 NBA Championship nears the end, Michelob ULTRA brings joy to sports fans who will g...\n",
+      "\n",
+      "Title: Futura and Michelob ULTRA Toast to the NBA Finals With Abstract Artwork Crafted From the Brand’s 2023 Limited-Edition Championship Bottles\n",
+      "Description: The sun is out to play, and so is Michelob ULTRA. With the 2022-2023 NBA Finals underway, the beermaker is back with its celebratory NBA Champ Bottles. This year, the self-proclaimed MVP of joy is dropping a limited-edition bottle made in collaboration with a…\n",
+      "Content: The sun is out to play, and so is Michelob ULTRA. With the 2022-2023 NBA Finals underway, the beerma...\n",
+      "\n",
+      "Title: Signed and Delivered, Futura and Michelob ULTRA Will Gift Hand-Painted Bottles to This Year’s NBA Championship Team\n",
+      "Description: Michelob ULTRA, the MVP of joy and official beer sponsor of the NBA is back to celebrate with basketball lovers and sports fans around the globe as the NBA 2022-2023 season comes to a nail-biting close. In collaboration with artist Futura, Michelob ULTRA will…\n",
+      "Content: Michelob ULTRA, the MVP of joy and official beer sponsor of the NBA is back to celebrate with basket...\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "def search_news(\n",
+    "    query: str,\n",
+    "    news_api_key: str = news_api_key,\n",
+    "    num_articles: int = 50,\n",
+    "    from_datetime: str = \"2023-06-01\",  # the 2023 NBA finals were played in June 2023\n",
+    "    to_datetime: str = \"2023-06-30\",\n",
+    ") -> dict:\n",
+    "    response = requests.get(\n",
+    "        \"https://newsapi.org/v2/everything\",\n",
+    "        params={\n",
+    "            \"q\": query,\n",
+    "            \"apiKey\": news_api_key,\n",
+    "            \"pageSize\": num_articles,\n",
+    "            \"sortBy\": \"relevancy\",\n",
+    "            \"from\": from_datetime,\n",
+    "            \"to\": to_datetime,\n",
+    "        },\n",
+    "    )\n",
+    "\n",
+    "    return response.json()\n",
+    "\n",
+    "\n",
+    "articles = []\n",
+    "\n",
+    "for query in tqdm(queries):\n",
+    "    result = search_news(query)\n",
+    "    if result[\"status\"] == \"ok\":\n",
+    "        articles = articles + result[\"articles\"]\n",
+    "    else:\n",
+    "        raise Exception(result[\"message\"])\n",
+    "\n",
+    "# remove duplicates\n",
+    "articles = list({article[\"url\"]: article for article in articles}.values())\n",
+    "\n",
+    "print(\"Total number of articles:\", len(articles))\n",
+    "print(\"Top 5 articles of query 1:\", \"\\n\")\n",
+    "\n",
+    "for article in articles[0:5]:\n",
+    "    print(\"Title:\", article[\"title\"])\n",
+    "    print(\"Description:\", article[\"description\"])\n",
+    "    print(\"Content:\", article[\"content\"][0:100] + \"...\")\n",
+    "    print()\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "As we can see, oftentimes, the search queries will return a large number of results, many of which are not relevant to the original question asked by the user. In order to improve the quality of the final answer, we use embeddings to re-rank and filter the results.\n",
+    "\n",
+    "## 2. Re-rank\n",
+    "\n",
+    "Drawing inspiration from [HyDE (Gao et al.)](https://arxiv.org/abs/2212.10496), we first generate a hypothetical ideal answer to rerank our compare our results against. This helps prioritize results that look like good answers, rather than those similar to our question. Here’s the prompt we use to generate our hypothetical answer.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'The NBA championship was won by TEAM NAME. The MVP was awarded to PLAYER NAME. The last game was held at STADIUM NAME, where both teams played with great energy and enthusiasm. It was a close game, but in the end, TEAM NAME emerged victorious.'"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "HA_INPUT = f\"\"\"\n",
+    "Generate a hypothetical answer to the user's question. This answer will be used to rank search results. \n",
+    "Pretend you have all the information you need to answer, but don't use any actual facts. Instead, use placeholders\n",
+    "like NAME did something, or NAME said something at PLACE. \n",
+    "\n",
+    "User question: {USER_QUESTION}\n",
+    "\n",
+    "Format: {{\"hypotheticalAnswer\": \"hypothetical answer text\"}}\n",
+    "\"\"\"\n",
+    "\n",
+    "hypothetical_answer = json_gpt(HA_INPUT)[\"hypotheticalAnswer\"]\n",
+    "\n",
+    "hypothetical_answer\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now, let's generate embeddings for the search results and the hypothetical answer. We then calculate the cosine distance between these embeddings, giving us a semantic similarity metric. Note that we can simply calculate the dot product in lieu of doing a full cosine similarity calculation since the OpenAI embeddings are returned normalized in our API.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[0.7854456526852069,\n",
+       " 0.8086023500072106,\n",
+       " 0.8002998147018501,\n",
+       " 0.7961229569526956,\n",
+       " 0.798354506673743,\n",
+       " 0.758216458795653,\n",
+       " 0.7753754083127359,\n",
+       " 0.7494958338411927,\n",
+       " 0.804733946801739,\n",
+       " 0.8405965885235218]"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "hypothetical_answer_embedding = embeddings(hypothetical_answer)[0]\n",
+    "article_embeddings = embeddings(\n",
+    "    [\n",
+    "        f\"{article['title']} {article['description']} {article['content'][0:100]}\"\n",
+    "        for article in articles\n",
+    "    ]\n",
+    ")\n",
+    "\n",
+    "# Calculate cosine similarity\n",
+    "cosine_similarities = []\n",
+    "for article_embedding in article_embeddings:\n",
+    "    cosine_similarities.append(dot(hypothetical_answer_embedding, article_embedding))\n",
+    "\n",
+    "cosine_similarities[0:10]\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Finally, we use these similarity scores to sort and filter the results.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Top 5 articles: \n",
+      "\n",
+      "Title: NBA Finals: Denver Nuggets beat Miami Hea, lift thier first-ever NBA title\n",
+      "Description: Denver Nuggets won their maiden NBA Championship trophy defeating Miami Heat 94-89 in Game 5 of the NBA Final held on Tuesday at the Ball Arena in Denver\n",
+      "Content: Denver Nuggets won their maiden NBA Championship trophy defeating Miami Heat 94-89 in Game 5 of the ...\n",
+      "Score: 0.8445817523602124\n",
+      "\n",
+      "Title: Photos: Denver Nuggets celebrate their first NBA title\n",
+      "Description: The Nuggets capped off an impressive postseason by beating the Miami Heat in the NBA Finals.\n",
+      "Content: Thousands of supporters watched along the streets of Denver, Colorado as the US National Basketball ...\n",
+      "Score: 0.842070667753606\n",
+      "\n",
+      "Title: Denver Nuggets win first NBA championship title in Game 5 victory over Miami Heat\n",
+      "Description: The Denver Nuggets won their first NBA championship Monday night, downing the Miami Heat 94-89 at Ball Arena in Denver to take Game 5 of the NBA Finals.\n",
+      "Content: The Denver Nuggets won their first NBA championship Monday night, downing the Miami Heat 94-89 at Ba...\n",
+      "Score: 0.8409346078172385\n",
+      "\n",
+      "Title: Denver Nuggets Capture Their First NBA Championship Behind Unbreakable Chemistry\n",
+      "Description: After 47 years of waiting, the Denver Nuggets are NBA champions. Led by Nikola Jokic and Jamal Murray, they reached the mountain top by staying true to themselves.\n",
+      "Content: DENVER, CO - JUNE 12: Jamal Murray (27) of the Denver Nuggets celebrates as he leaves the court ... ...\n",
+      "Score: 0.8405965885235218\n",
+      "\n",
+      "Title: NBA Finals: Nikola Jokic, Denver Nuggets survive Miami Heat to secure franchise's first NBA championship\n",
+      "Description: In a rock-fight of a Game 5, the Denver Nuggets reached the NBA mountaintop from the foothills of the Rockies, winning their first-ever championship and setting Nikola Jokic's legacy as an all-timer in stone.\n",
+      "Content: DENVER, COLORADO - JUNE 12: Jamal Murray #27 of the Denver Nuggets reacts during the fourth quarter ...\n",
+      "Score: 0.8389716330890262\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "scored_articles = zip(articles, cosine_similarities)\n",
+    "\n",
+    "# Sort articles by cosine similarity\n",
+    "sorted_articles = sorted(scored_articles, key=lambda x: x[1], reverse=True)\n",
+    "\n",
+    "# Print top 5 articles\n",
+    "print(\"Top 5 articles:\", \"\\n\")\n",
+    "\n",
+    "for article, score in sorted_articles[0:5]:\n",
+    "    print(\"Title:\", article[\"title\"])\n",
+    "    print(\"Description:\", article[\"description\"])\n",
+    "    print(\"Content:\", article[\"content\"][0:100] + \"...\")\n",
+    "    print(\"Score:\", score)\n",
+    "    print()\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Awesome! These results look a lot more relevant to our original query. Now, let's use the top 5 results to generate a final answer.\n",
+    "\n",
+    "## 3. Answer\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/markdown": [
+       "The Denver Nuggets won their first-ever NBA championship by defeating the Miami Heat 94-89 in Game 5 of the NBA Finals held on Tuesday at the Ball Arena in Denver, according to this [Business Standard article](https://www.business-standard.com/sports/other-sports-news/nba-finals-denver-nuggets-beat-miami-hea-lift-thier-first-ever-nba-title-123061300285_1.html). Nikola Jokic, the Nuggets' center, was named the NBA Finals MVP. In a rock-fight of a Game 5, the Nuggets reached the NBA mountaintop, securing their franchise's first NBA championship and setting Nikola Jokic's legacy as an all-timer in stone, according to this [Yahoo Sports article](https://sports.yahoo.com/nba-finals-nikola-jokic-denver-nuggets-survive-miami-heat-to-secure-franchises-first-nba-championship-030321214.html). For more information and photos of the Nuggets' celebration, check out this [Al Jazeera article](https://www.aljazeera.com/gallery/2023/6/15/photos-denver-nuggets-celebrate-their-first-nba-title) and this [CNN article](https://www.cnn.com/2023/06/12/sport/denver-nuggets-nba-championship-spt-intl?cid=external-feeds_iluminar_yahoo)."
+      ],
+      "text/plain": [
+       "<IPython.core.display.Markdown object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "formatted_top_results = [\n",
+    "    {\n",
+    "        \"title\": article[\"title\"],\n",
+    "        \"description\": article[\"description\"],\n",
+    "        \"url\": article[\"url\"],\n",
+    "    }\n",
+    "    for article, _score in sorted_articles[0:5]\n",
+    "]\n",
+    "\n",
+    "ANSWER_INPUT = f\"\"\"\n",
+    "Generate an answer to the user's question based on the given search results. \n",
+    "TOP_RESULTS: {formatted_top_results}\n",
+    "USER_QUESTION: {USER_QUESTION}\n",
+    "\n",
+    "Include as much information as possible in the answer. Reference the relevant search result urls as markdown links.\n",
+    "\"\"\"\n",
+    "\n",
+    "completion = client.chat.completions.create(\n",
+    "    model=GPT_MODEL,\n",
+    "    messages=[{\"role\": \"user\", \"content\": ANSWER_INPUT}],\n",
+    "    temperature=0.5,\n",
+    "    stream=True,\n",
+    ")\n",
+    "\n",
+    "text = \"\"\n",
+    "for chunk in completion:\n",
+    "    text += chunk.choices[0].delta.content\n",
+    "    display.clear_output(wait=True)\n",
+    "    display.display(display.Markdown(text))"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.0"
+  },
+  "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/Realtime_eval_guide.ipynb

@@ -0,0 +1,687 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "fba401c1",
+   "metadata": {},
+   "source": [
+    "# Realtime Eval Guide"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fafa08e6",
+   "metadata": {},
+   "source": [
+    "<p align=\"center\">\n",
+    "<img\n",
+    "  width=\"650\"\n",
+    "  src=\"../images/realtime_eval_page_01_image_01.png\"\n",
+    "  alt=\"Diagram from page 1\"\n",
+    "/>\n",
+    "</p>\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7553621b",
+   "metadata": {},
+   "source": [
+    "## Introduction\n",
+    "\n",
+    "Evals are what turn a voice demo into something people can rely on. The gap between “seems fine” and “works every day” is almost always evals.\n",
+    "\n",
+    "This guide shows how to evaluate voice systems by slowly building complexity: start simple (Crawl), add realism (Walk), then test multi-turn (Run). Along the way, you’ll learn to build the three things that make results robust: a dataset, graders, and an eval harness, plus a production flywheel so real failures become new tests.\n",
+    "\n",
+    "Teams that invest in evals can ship to production **5–10× faster** because they can see what’s failing, pinpoint why, and fix it with confidence.\n",
+    "\n",
+    "\n",
+    "### Realtime Eval Harness Code\n",
+    "\n",
+    "If you want the runnable code to build eval harnesses, use this repo folder first:\n",
+    "\n",
+    "- **GitHub repo path:** [openai-cookbook/examples/evals/realtime_evals](https://github.com/openai/openai-cookbook/tree/main/examples/evals/realtime_evals)\n",
+    "- It includes complete reference harnesses for each maturity stage:\n",
+    "  - [Crawl harness (single-turn replay)](https://github.com/openai/openai-cookbook/tree/main/examples/evals/realtime_evals/crawl_harness)\n",
+    "  - [Walk harness (saved audio replay)](https://github.com/openai/openai-cookbook/tree/main/examples/evals/realtime_evals/walk_harness)\n",
+    "  - [Run harness (model-simulated multi-turn)](https://github.com/openai/openai-cookbook/tree/main/examples/evals/realtime_evals/run_harness)\n",
+    "\n",
+    "You can point Codex at the harness you want and ask it to adapt it to your data and graders.\n",
+    "\n",
+    "## Part I: Foundations\n",
+    "\n",
+    "### 1) Why realtime evals are hard\n",
+    "\n",
+    "Realtime is harder than text because you are grading a **streaming interaction** with two outputs: what the **assistant doe**s and **how it sounds**. A response can be “right” and still sound broken.\n",
+    "\n",
+    "#### 1.1 The 2 axes of realtime quality\n",
+    "\n",
+    "Text evals mostly ask if the content is right. Realtime adds a second axis: audio quality. Content and audio can fail independently, so a single score can hide real problems.\n",
+    "\n",
+    "**Most realtime evals can reduce to two independent axes:**\n",
+    "\n",
+    "1. Content quality: Did the assistant understand the user and do the right thing? Correctness, helpfulness, tool choice, tool arguments, and instruction following.\n",
+    "\n",
+    "2. Audio quality: Did the assistant sound acceptable? Naturalness, prosody, pronunciation, stability, and how it behaves under noise and imperfect capture.\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5d420dfa",
+   "metadata": {},
+   "source": [
+    "#### 1.2 Hard to debug\n",
+    "\n",
+    "With the **Responses API**, the mental model is simple: **request in → response out**. With the **Realtime API**, a “turn” is a s**titched pipeline**. That orchestration makes voice apps easy to build, but for evals, you must log stages so you can isolate failures and find root causes."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "89dd169b",
+   "metadata": {},
+   "source": [
+    "<p align=\"center\">\n",
+    "<img\n",
+    "  width=\"700\"\n",
+    "  src=\"../images/realtime_eval_page_02_image_01.png\"\n",
+    "  alt=\"Diagram from page 2\"\n",
+    "/>\n",
+    "</p>\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "becc23dc",
+   "metadata": {},
+   "source": [
+    "A “turn” is a chain of events (speech start/stop → commit → response.create → audio deltas → done), and failures can happen at any stage. If you treat the system as a black box, you’ll chase “model issues” that are actually turn detection, buffering, or tool-integration issues.\n",
+    "\n",
+    "**Example:**\n",
+    "\n",
+    "- Content is correct but the experience is broken: audio gets chopped during barge-in because the interruption boundary is wrong.\n",
+    "\n",
+    "- Answer is “right” but feels slow: latency came from network quality, turn detection slowness, not the model’s reasoning.\n",
+    "\n",
+    "You can learn more about the various events that the Realtime API triggers [here](https://platform.openai.com/docs/api-reference/realtime-server-events).\n",
+    "\n",
+    "#### 1.3 Transcript ≠ ground truth\n",
+    "\n",
+    "In realtime api, the ground truth for “what the user said” is **the actual audio signal** (what the microphone captured and what the model heard). A transcript is not ground truth, it’s a **model-produced interpretation** of that audio. It can be wrong because it’s constrained by **transcription model errors**."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7c9550c7",
+   "metadata": {},
+   "source": [
+    "**If you treat transcripts as truth, your evals can be misleading:**\n",
+    "\n",
+    "- False fail: ASR drops a digit, but the model heard it and called the tool correctly → your LLM grader marks “wrong.”\n",
+    "\n",
+    "- False pass: transcript looks clean, but audio was clipped and the model guessed → you miss the real problem.\n",
+    "\n",
+    "**Best Practices:**\n",
+    "\n",
+    "- **Improve transcription:** Iterate on transcription [prompts](https://platform.openai.com/docs/guides/speech-to-text#prompting), try different [models](https://platform.openai.com/docs/guides/speech-to-text), try different methods such as [oob transcription](https://cookbook.openai.com/examples/realtime_out_of_band_transcription).\n",
+    "\n",
+    "- **Use transcripts for scale:** run most automated grading on **transcripts + traces**.\n",
+    "\n",
+    "- **Calibrate graders on messy reality:** iterate graders on **production-like, noisy transcripts** (not clean text) so they don’t overreact to ASR errors.\n",
+    "\n",
+    "- **Add an audio audit loop:** spot-check **~1–5%** of sessions end-to-end."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2ec20772",
+   "metadata": {},
+   "source": [
+    "<p align=\"center\">\n",
+    "<img\n",
+    "  width=\"650\"\n",
+    "  src=\"../images/realtime_eval_page_03_image_01.png\"\n",
+    "  alt=\"Diagram from page 3\"\n",
+    "/>\n",
+    "</p>\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3033c18a",
+   "metadata": {},
+   "source": [
+    "## Part II: Strategy\n",
+    "\n",
+    "### 2) Crawl / Walk / Run\n",
+    "\n",
+    "Realtime evals feel overwhelming when teams start at the hardest setting: real audio, multi-turn dialogue and real tools. The fix is to build **complexity in steps**. If your system **cannot crawl, it will not run**. Early evals should be simple enough that failures are diagnosable, repeatable, and cheap to iterate on. You can increase complexity in two independent axes.\n",
+    "\n",
+    "#### 2.1 Isolating input conditions: clean vs production audio\n",
+    "\n",
+    "This axis is about what the model hears. By controlling input audio conditions, you can separate failures in model intelligence from failures in speech perception.\n",
+    "\n",
+    "- **Start with synthetic audio → tests intelligence:**\n",
+    "    - Use clean, synthetic repeatable audio (e.g., TTS) when you want to measure the model’s reasoning and decision-making without audio variance muddying the signal → helps isolate intent routing, tool calling, instruction following\n",
+    "\n",
+    "- **Move to noisy, production-like audio → tests audio perception:**\n",
+    "    - Once intelligence is stable, introduce audio that resembles production: compression, echo, far-field capture, background noise, hesitations/self-corrections. This tests whether the system still behaves correctly when the input is ambiguous, messy, or partially lost → helps measure mishearing words, robustness to acoustic variations"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "48a6c62b",
+   "metadata": {},
+   "source": [
+    "<p align=\"center\">\n",
+    "<img\n",
+    "  width=\"1000\"\n",
+    "  src=\"../images/realtime_eval_page_04_image_01.png\"\n",
+    "  alt=\"Diagram from page 4\"\n",
+    "/>\n",
+    "</p>\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "daac735f",
+   "metadata": {},
+   "source": [
+    "#### 2.2 Isolating interaction conditions: single-turn vs multi-turn\n",
+    "\n",
+    "This axis is about what you are evaluating: are you evaluating the next turn or the full **conversation**.\n",
+    "\n",
+    "- **Start single-turn → tests core competence:**\n",
+    "    - Run one request → one response when you want the cleanest signal on fundamentals: correct intent routing, correct tool choice, valid arguments, and basic instruction following. If the system can’t reliably pick the right tool or produce a valid schema here, evaluating more turns won’t help.\n",
+    "\n",
+    "- **Move to multi-turn → tests robustness:**\n",
+    "    - Once single-turn is stable, move to multi-turn where the system must hold goals and constraints across turns, sequence tools correctly, recover from tool failures and handle user corrections. Multi-turn shifts you from turn-level correctness to **episode-level outcomes**: did it complete the goal, how many turns did it take, and did it recover cleanly when something went wrong?"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4427dd03",
+   "metadata": {},
+   "source": [
+    "<p align=\"center\">\n",
+    "<img\n",
+    "  width=\"1000\"\n",
+    "  src=\"../images/realtime_eval_page_04_image_02.png\"\n",
+    "  alt=\"Diagram from page 4\"\n",
+    "/>\n",
+    "</p>\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3b83d021",
+   "metadata": {},
+   "source": [
+    "Single-turn tells you *can win the battle*; multi-turn tells you *can win the war*.\n",
+    "\n",
+    "#### 2.3 Eval Quadrants\n",
+    "\n",
+    "Use a 2x2 map for evaluation: **right** = more realistic audio, **up** = more realistic interaction. Start bottom-left, increasing difficulty one axis at a time."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "87a1e0a7",
+   "metadata": {},
+   "source": [
+    "<p align=\"center\">\n",
+    "<img\n",
+    "  width=\"1000\"\n",
+    "  src=\"../images/realtime_eval_page_05_image_01.png\"\n",
+    "  alt=\"Diagram from page 5\"\n",
+    "/>\n",
+    "</p>\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "deec8bae",
+   "metadata": {},
+   "source": [
+    "**Eval modes (increasing complexity):**\n",
+    "\n",
+    "1. Crawl (bottom-left): synthetic audio + single-turn\n",
+    "\n",
+    "2. Walk (move right): real noisy audio + single-turn\n",
+    "\n",
+    "3. Run (move up): synthetic audio + multi-turn simulation \n",
+    "\n",
+    "Top-right (real audio + full multi-turn flow) is manual eval: run end-to-end sessions the way users do in production. Keep it in the loop for the entire project lifecycle."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cec946d3",
+   "metadata": {},
+   "source": [
+    "<p align=\"center\">\n",
+    "<img\n",
+    "  width=\"1000\"\n",
+    "  src=\"../images/realtime_eval_page_05_image_02.png\"\n",
+    "  alt=\"Diagram from page 5\"\n",
+    "/>\n",
+    "</p>\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "70590579",
+   "metadata": {},
+   "source": [
+    "**Example:**\n",
+    "\n",
+    "User: “Change my reservation to 7pm.”\n",
+    "\n",
+    "- **Crawl:** You feed deterministic TTS for “Change my reservation to 7pm,” then grade only the next assistant turn: it should route to the reservation-update tool and pass the correct time=7pm (or ask one tight clarifying question if a required identifier is missing).\n",
+    "- **Walk:** Record a human-mic version of “Change my reservation to 7pm,” then replay the same utterance with phone-bandwidth compression and light background noise; the system should still hear “7pm” (not “7” or “7:15”) and produce the same correct tool call.\n",
+    "- **Run:** Model simulating a user outputs “Change my reservation to 7pm,” then simulates realistic follow-ups (“It’s under Minhajul for tonight… actually make it 7:30… wait, tomorrow”) plus an injected tool error once; the agent should clarify only what’s missing, keep state consistent, recover cleanly, and end with a single correct update tool call reflecting the final expected outcome.\n",
+    "\n",
+    "You can find reference implementations that you can start from and adapt here [realtime eval start](https://github.com/openai/openai-cookbook/tree/main/examples/evals/realtime_evals)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "52bc0ec2",
+   "metadata": {},
+   "source": [
+    "## Part III: The three building blocks\n",
+    "\n",
+    "### 4) Data: building a benchmark\n",
+    "\n",
+    "#### 4.1 Start with a “gold” seed set (10–50)\n",
+    "\n",
+    "Cover the flows you cannot afford to fail: core intents, must-work tool calls, escalation and refusal behaviors. Generate quickly, then have humans review for realism and gaps.\n",
+    "\n",
+    "**The goal is to start, not to perfect.**\n",
+    "\n",
+    "#### 4.2 Build for iteration, not just volume\n",
+    "\n",
+    "Eval datasets exist to drive iteration, not to look big. The loop is the product: **run evals → localize failures to a specific behavior → change one thing → re-run → confirm the fix improved without regressions**. A benchmark is “good” if it makes that loop fast, repeatable, and easy to diagnose.\n",
+    "\n",
+    "That requires coverage, not raw count: you need to represent the actual user behaviors and the specific edge cases that cause production failures. Size alone won’t surface fragility; the right coverage will. \n",
+    "\n",
+    "Coverage also has to be balanced. For every behavior, include both positives (the system should do X) and negatives (the system should not do X). Without negatives, you reward shortcuts.\n",
+    "\n",
+    ">**Customer Example:** A team built a voice support bot and optimized hard for the “escalate_to_human” tool call. Their offline score hit 98 percent on escalation. In dogfooding, the bot started escalating for almost everything. The root cause was dataset imbalance. They had many “must escalate” cases and almost no “do not escalate” cases, so the model learned a shortcut: escalate whenever uncertain. \n",
+    "\n",
+    "Finally, you must precisely tag your data to enable fine-grain evaluations. These tags should provide the necessary detail to move from a general observation, like \"score dropped,\" to a specific root cause, such **as \"this intent fails under these audio conditions with this policy boundary.\"**\n",
+    "\n",
+    "Example of tags could be:\n",
+    "\n",
+    "- intent, expected outcome, audio condition, language, and expected tool call. Tagged data enables teams to run fine-grain evaluations leading to faster iteration loops."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6a591cdb",
+   "metadata": {},
+   "source": [
+    "<p align=\"center\">\n",
+    "<img\n",
+    "  width=\"1200\"\n",
+    "  src=\"../images/realtime_eval_page_07_image_01.png\"\n",
+    "  alt=\"Diagram from page 7\"\n",
+    "/>\n",
+    "</p>\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3c7cfb13",
+   "metadata": {},
+   "source": [
+    "#### 4.3 Expand from production failures\n",
+    "\n",
+    "Offline evals are how you iterate fast. They are also easy to outgrow. If you keep optimizing against a fixed benchmark, **scores can rise while real quality (reality) stalls** because users do things your dataset does not cover."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "68be4755",
+   "metadata": {},
+   "source": [
+    "<p align=\"center\">\n",
+    "<img\n",
+    "  width=\"650\"\n",
+    "  src=\"../images/realtime_eval_page_07_image_02.png\"\n",
+    "  alt=\"Diagram from page 7\"\n",
+    "/>\n",
+    "</p>\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3b23efdf",
+   "metadata": {},
+   "source": [
+    "The operating model is a loop: production expands the benchmark. A new failure shows up, you reproduce it, you label it, and you add it. Over time, your offline suite should grow with the product.\n",
+    "\n",
+    "**A simple way to manage this is three sets:**\n",
+    "\n",
+    "- **Regression suite:** hard cases you already fixed. Run on every prompt, model, and tool change. This is your “do not break” contract.\n",
+    "\n",
+    "- **Rolling discovery set:** fresh failures from production and near misses. This is where you learn what you are missing and what to prioritize next. If they trigger failure modes, promote them to **your offline dataset. Teams usually fill this by:**\n",
+    "\n",
+    "    - Running online graders to catch failures directly, and/or\n",
+    "\n",
+    "    - Watching proxy metrics (latency, tool error rates, escalation rate, retries) and sampling data when they drift.\n",
+    "- **Holdout set:** a subset of the offline test which stays untouched that you run occasionally to detect benchmark overfitting. If test scores climb while holdout stays flat, you are training for the test."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "785fa85a",
+   "metadata": {},
+   "source": [
+    "<p align=\"center\">\n",
+    "<img\n",
+    "  width=\"650\"\n",
+    "  src=\"../images/realtime_eval_page_08_image_01.png\"\n",
+    "  alt=\"Diagram from page 8\"\n",
+    "/>\n",
+    "</p>\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "996ceb4e",
+   "metadata": {},
+   "source": [
+    "### 5) Graders\n",
+    "\n",
+    "Graders are your **measurement instruments**. They turn a messy, real-time voice session into **signals you can trust**.\n",
+    "\n",
+    "#### 5.1 Manual review (highest leverage)\n",
+    "\n",
+    "Manual review = listen to real audio + read full traces end-to-end. It’s the fastest way to build product intuition and catch the failures users notice instantly. Automated evals tell you what you can measure. Manual review tells you what you should be measuring.\n",
+    "\n",
+    "**What automation routinely underweights (but users feel immediately):**\n",
+    "\n",
+    "- Turn-taking failures: awkward gaps, double-talk, model cutting the user off.\n",
+    "\n",
+    "- Pacing & prosody: model speech is too fast/slow, rambling, flat, jittery, “robot polite.”\n",
+    "\n",
+    "- Transcript mismatch: ASR lag/drops/normalization → you end up grading the wrong thing.\n",
+    "\n",
+    "- Eval-system bugs: missing coverage in the golden set, mislabeled expectations, graders that are systematically too strict/lenient. \n",
+    "\n",
+    "> **Customer Example:** one large company had execs spend **~3 hours/day** just listening to sessions and scanning traces. They surfaced “hidden” issues, early cutoffs, phantom interruptions, awkward prosody, that would’ve sailed past offline evals.\n",
+    "\n",
+    "#### 5.2 Automated graders\n",
+    "\n",
+    "Humans don’t scale. Without automation, regressions slip through and “improvements” turn into vibes.\n",
+    "\n",
+    "**Use a layered grader stack:**"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0157c176",
+   "metadata": {},
+   "source": [
+    "1. **Deterministic graders** for anything objective and machine-checkable. They’re fast, cheap, and stable, perfect for tight iteration loops and regression gates (tool calling, JSON validity, string and pattern checks).\n",
+    "\n",
+    "2. **LLM graders** help you measure the things that matter but don’t fit neatly into deterministic rules: correctness, instruction following, whether a clarification was appropriate, completeness, and helpfulness.\n",
+    "\n",
+    "3. **Audio graders** because users experience the voice, not the transcript. Audio is still the hardest to judge reliably, so don’t wait for a single perfect scorer, start with simple, measurable checks (silence, overlap, interruption handling) and layer richer rubrics over time."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "28efb845",
+   "metadata": {},
+   "source": [
+    "<p align=\"center\">\n",
+    "<img\n",
+    "  width=\"900\"\n",
+    "  src=\"../images/realtime_eval_page_09_image_01.png\"\n",
+    "  alt=\"Diagram from page 9\"\n",
+    "/>\n",
+    "</p>\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ee3252bf",
+   "metadata": {},
+   "source": [
+    "### 6) Eval Harness\n",
+    "\n",
+    "A realtime eval is only as trustworthy as the harness that runs it. A good harness has one job: **make runs comparable**. If the same input can’t be replayed under the same settings and produce similar outcomes, it makes it hard to measure and iterate.\n",
+    "\n",
+    "#### [6.1 Start with single-turn replay (the “Crawl” harness)](https://github.com/openai/openai-cookbook/tree/main/examples/evals/realtime_evals/crawl_harness)\n",
+    "\n",
+    "Start here. Single-turn replay gives the fastest, cleanest signal because you can keep almost everything fixed. Keep the exact audio bytes, preprocessing, VAD configuration, codec, and chunking strategy identical across runs.\n",
+    "\n",
+    "In practice, it’s often best to start with voice activity detection (VAD) turned off so you remove one major source of variance. With VAD off, you decide exactly when a user turn ends.\n",
+    "\n",
+    "**A simple single-turn harness looks like:**"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "65abb0c8",
+   "metadata": {},
+   "source": [
+    "<p align=\"center\">\n",
+    "<img\n",
+    "  width=\"1000\"\n",
+    "  src=\"../images/realtime_eval_page_10_image_01.png\"\n",
+    "  alt=\"Diagram from page 10\"\n",
+    "/>\n",
+    "</p>\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0bd80a2d",
+   "metadata": {},
+   "source": [
+    "**More explicitly (in Realtime API terms):**\n",
+    "\n",
+    "1. Generate or load input audio\n",
+    "\n",
+    "    - If the datapoint is text, generate TTS audio.\n",
+    "    \n",
+    "        - Often, starting with text → TTS → audio is the best first step because it enables much faster iteration. It’s easier to tweak and refine the eval when you can iterate on text quickly.\n",
+    "\n",
+    "2. Stream audio into the input buffer\n",
+    "\n",
+    "    - Send audio in fixed-size chunks (for example: consistent frame size per chunk).\n",
+    "\n",
+    "    - Important: chunking and timing affect behavior. Pick a standard and stick to it. For example, 20 ms per chunk is a good balance of responsiveness and overhead.\n",
+    "\n",
+    "3. Commit the user audio\n",
+    "\n",
+    "    - (Recommended) With VAD off: commit immediately after the last audio chunk.\n",
+    "\n",
+    "    - With VAD on: the server detects turns boundaries.\n",
+    "\n",
+    "4. Trigger the assistant response\n",
+    "\n",
+    "    - With VAD off: Call response.create to start generation.\n",
+    "\n",
+    "    - With VAD on: It is automatic.\n",
+    "\n",
+    "5. Collect outputs\n",
+    "\n",
+    "    - Output audio chunks (streaming deltas)\n",
+    "\n",
+    "    - Output transcript (if enabled)\n",
+    "\n",
+    "    - Tool calls / tool arguments (if any)\n",
+    "\n",
+    "    - Final completion event\n",
+    "\n",
+    "6. Grade and persist\n",
+    "    - Run graders\n",
+    "\n",
+    "    - Save results"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "56bbacd7",
+   "metadata": {},
+   "source": [
+    "#### [6.2 Replaying saved audio (the “Walk” harness)](https://github.com/openai/openai-cookbook/tree/main/examples/evals/realtime_evals/walk_harness)\n",
+    "\n",
+    "When you move from synthetic TTS to real recordings, the harness changes in one important way: **you are streaming audio buffers from saved realistic audio.**\n",
+    "\n",
+    "**For saved audio, the flow becomes:**"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4eb150f0",
+   "metadata": {},
+   "source": [
+    "<p align=\"center\">\n",
+    "<img\n",
+    "  width=\"1000\"\n",
+    "  src=\"../images/realtime_eval_page_10_image_02.png\"\n",
+    "  alt=\"Diagram from page 10\"\n",
+    "/>\n",
+    "</p>\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ca3f4dbc",
+   "metadata": {},
+   "source": [
+    "**How to make the evals realistic in practice:**\n",
+    "\n",
+    "- **Preprocessing must match production**\n",
+    "  - Same resampling, normalization, channel handling, noise suppression (if used), and encoding.\n",
+    "\n",
+    "  - Store preprocessing config alongside results so you can explain score changes.\n",
+    "\n",
+    "- **Streaming policy must be explicit**\n",
+    "\n",
+    "  - If you care about latency: send chunks on a fixed cadence (e.g., “every 20ms, send 20ms of audio”).\n",
+    "\n",
+    "  - If you only care about iteration speed: you can stream faster, but keep chunk size constant.\n",
+    "\n",
+    "- **Turn boundaries must be repeatable**\n",
+    "\n",
+    "  - Prefer VAD off + manual commit for offline reproducibility.\n",
+    "\n",
+    "  - If you must use VAD on (to match production), log VAD settings and track boundary events so you can debug failures.\n",
+    "\n",
+    "#### [6.3 Model-simulated multi-turn (the “Run” harness)](https://github.com/openai/openai-cookbook/tree/main/examples/evals/realtime_evals/run_harness)\n",
+    "\n",
+    "Model-simulated multi-turn uses a **user simulator** to generate the next user turn for a full conversation. It can increase coverage of scenarios, but only if episodes stay comparable across runs.\n",
+    "\n",
+    "**Common loop:**"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ea242990",
+   "metadata": {},
+   "source": [
+    "<p align=\"center\">\n",
+    "<img\n",
+    "  width=\"1000\"\n",
+    "  src=\"../images/realtime_eval_page_11_image_01.png\"\n",
+    "  alt=\"Diagram from page 11\"\n",
+    "/>\n",
+    "</p>\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7acb75df",
+   "metadata": {},
+   "source": [
+    "**Best practice for simulations:**\n",
+    "- **Pin and version the simulator prompt:** Treat it like code. A small prompt edit can shift behavior more than a model change.\n",
+    "\n",
+    "- **Constrain randomness:** Fix temperature and sampling settings. Use a seed if available. Use deterministic turns where it makes sense (i.e User greetings).\n",
+    "\n",
+    "- **Mock tools deterministically:** Define expected tool output mocks for the scenario and return those exact outputs when the assistant calls tools. This keeps the environment stable and makes runs comparable.\n",
+    "\n",
+    "- **Record the full trajectory:** Store every generated user text turn plus the final audio bytes you streamed. Persist tool calls, tool returns, and timestamps. Simulation is a discovery engine. When it finds a real failure mode, you backfill it into a deterministic scripted episode for the crawl or walk method."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "749dc2cd",
+   "metadata": {},
+   "source": [
+    "## Part IV: Case study\n",
+    "\n",
+    "#### 7.1 Customer support voice bot\n",
+    "\n",
+    "**Product goal and constraints**\n",
+    "\n",
+    "Resolve common support requests through tools, quickly and safely. The bot must collect the right details, call the right backend actions, and comply with policy. It must escalate cleanly when it cannot help. It must handle frustrated callers without becoming verbose or brittle.\n",
+    "\n",
+    "**Crawl, Walk, Run plan**\n",
+    "\n",
+    "**Crawl: synthetic + single-turn**\n",
+    "\n",
+    "Focus on routing and policy. Given a short request, the bot should pick the right intent, request missing info, and avoid unsafe actions. Use deterministic synthetic audio so you can rapidly iterate on tool schemas and prompts.\n",
+    "\n",
+    "**Walk: real + single-turn**\n",
+    "\n",
+    "Test understanding under realistic capture. Use synthetic or real recordings in noisy environments and telephony-like quality. This is where order numbers, names, and addresses break on noisy audio. Evaluate whether the bot asks clarifying questions instead of guessing.\n",
+    "\n",
+    "**Run: synthetic + multi-turn simulations**\n",
+    "\n",
+    "Simulate full workflows with simulated users with gpt-realtime and tool mocks: authentication, account lookup, order status, return eligibility, refund, ticket creation, escalation. Add adversarial but realistic patterns: caller changes goal midstream, provides partial info, talks over the assistant, or answers a different question than asked.\n",
+    "\n",
+    "**Manual Review:**\n",
+    "\n",
+    "Run internal call sessions against staging systems. This catches UX failures that graders miss: overlong disclaimers, repetitive questions, poor turn-taking during authentication.\n",
+    "\n",
+    "**Core dataset buckets and useful slices**\n",
+    "\n",
+    "- Top intents: order status, return, refund, cancel, billing issue, password reset, appointment scheduling.\n",
+    "\n",
+    "- Missing and conflicting info: wrong order number, two accounts, caller provides a nickname, caller refuses to authenticate.\n",
+    "\n",
+    "- Policy edges: out-of-window returns, restricted items, partial refunds, subscription cancellation rules.\n",
+    "\n",
+    "- Escalation triggers: the bot should hand off when confidence is low or tools fail.\n",
+    "\n",
+    "- Emotional tone: angry, rushed, confused. The content goal stays the same, but delivery matters.\n",
+    "\n",
+    "**Graders used**\n",
+    "\n",
+    "- Deterministic: tool selection, tool argument validity, policy phrases if required.\n",
+    "\n",
+    "- LLM rubric grader: instruction following, resolution correctness, empathetic tone, whether it avoided hallucinating policy, whether it escalated appropriately, and whether it stayed concise.\n",
+    "\n",
+    "- Audio grader: long silences, interruption handling."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "openai",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

examples/Regression_using_embeddings.ipynb

@@ -0,0 +1,119 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Regression using the embeddings\n",
+    "\n",
+    "Regression means predicting a number, rather than one of the categories. We will predict the score based on the embedding of the review's text. We split the dataset into a training and a testing set for all of the following tasks, so we can realistically evaluate performance on unseen data. The dataset is created in the [Get_embeddings_from_dataset Notebook](Get_embeddings_from_dataset.ipynb).\n",
+    "\n",
+    "We're predicting the score of the review, which is a number between 1 and 5 (1-star being negative and 5-star positive)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "text-embedding-3-small performance on 1k Amazon reviews: mse=0.65, mae=0.52\n"
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "import numpy as np\n",
+    "from ast import literal_eval\n",
+    "\n",
+    "from sklearn.ensemble import RandomForestRegressor\n",
+    "from sklearn.model_selection import train_test_split\n",
+    "from sklearn.metrics import mean_squared_error, mean_absolute_error\n",
+    "\n",
+    "datafile_path = \"data/fine_food_reviews_with_embeddings_1k.csv\"\n",
+    "\n",
+    "df = pd.read_csv(datafile_path)\n",
+    "df[\"embedding\"] = df.embedding.apply(literal_eval).apply(np.array)\n",
+    "\n",
+    "X_train, X_test, y_train, y_test = train_test_split(list(df.embedding.values), df.Score, test_size=0.2, random_state=42)\n",
+    "\n",
+    "rfr = RandomForestRegressor(n_estimators=100)\n",
+    "rfr.fit(X_train, y_train)\n",
+    "preds = rfr.predict(X_test)\n",
+    "\n",
+    "mse = mean_squared_error(y_test, preds)\n",
+    "mae = mean_absolute_error(y_test, preds)\n",
+    "\n",
+    "print(f\"text-embedding-3-small performance on 1k Amazon reviews: mse={mse:.2f}, mae={mae:.2f}\")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Dummy mean prediction performance on Amazon reviews: mse=1.73, mae=1.03\n"
+     ]
+    }
+   ],
+   "source": [
+    "bmse = mean_squared_error(y_test, np.repeat(y_test.mean(), len(y_test)))\n",
+    "bmae = mean_absolute_error(y_test, np.repeat(y_test.mean(), len(y_test)))\n",
+    "print(\n",
+    "    f\"Dummy mean prediction performance on Amazon reviews: mse={bmse:.2f}, mae={bmae:.2f}\"\n",
+    ")\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can see that the embeddings are able to predict the scores with an average error of 0.53 per score prediction. This is roughly equivalent to predicting half of reviews perfectly, and half off by one star."
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You could also train a classifier to predict the label, or use the embeddings within an existing ML model to encode free text features."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "openai",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.5"
+  },
+  "orig_nbformat": 4,
+  "vscode": {
+   "interpreter": {
+    "hash": "365536dcbde60510dc9073d6b991cd35db2d9bac356a11f5b64279a5e6708b97"
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/Search_reranking_with_cross-encoders.ipynb

@@ -0,0 +1,817 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "id": "7f30b8b2",
+      "metadata": {},
+      "source": [
+        "# Search reranking with cross-encoders\n",
+        "\n",
+        "This notebook takes you through examples of using a cross-encoder to re-rank search results.\n",
+        "\n",
+        "This is a common use case with our customers, where you've implemented semantic search using embeddings (produced using a [bi-encoder](https://www.sbert.net/examples/applications/retrieve_rerank/README.html#retrieval-bi-encoder)) but the results are not as accurate as your use case requires. A possible cause is that there is some business rule you can use to rerank the documents such as how recent or how popular a document is. \n",
+        "\n",
+        "However, often there are subtle domain-specific rules that help determine relevancy, and this is where a cross-encoder can be useful. Cross-encoders are more accurate than bi-encoders but they don't scale well, so using them to re-order a shortened list returned by semantic search is the ideal use case.\n",
+        "\n",
+        "### Example\n",
+        "\n",
+        "Consider a search task with D documents and Q queries.\n",
+        "\n",
+        "The brute force approach of computing every pairwise relevance is expensive; its cost scales as ```D * Q```. This is known as **cross-encoding**.\n",
+        "\n",
+        "A faster approach is **embeddings-based search**, in which an embedding is computed once for each document and query, and then re-used multiple times to cheaply compute pairwise relevance. Because embeddings are only computed once, its cost scales as ```D + Q```. This is known as **bi-encoding**.\n",
+        "\n",
+        "Although embeddings-based search is faster, the quality can be worse. To get the best of both, one common approach is to use embeddings (or another bi-encoder) to cheaply identify top candidates, and then use GPT (or another cross-encoder) to expensively re-rank those top candidates. The cost of this hybrid approach scales as ```(D + Q) * cost of embedding + (N * Q) * cost of re-ranking```, where ```N``` is the number of candidates re-ranked.\n",
+        "\n",
+        "### Walkthrough\n",
+        "\n",
+        "To illustrate this approach we'll use ```text-davinci-003``` with ```logprobs``` enabled to build a GPT-powered cross-encoder. Our GPT models have strong general language understanding, which when tuned with some few-shot examples can provide a simple and effective cross-encoding option.\n",
+        "\n",
+        "This notebook drew on this great [article](https://weaviate.io/blog/cross-encoders-as-reranker) by Weaviate, and this [excellent explanation](https://www.sbert.net/examples/applications/cross-encoder/README.html) of bi-encoders vs. cross-encoders from Sentence Transformers."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "71cb361f",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "!pip install openai\n",
+        "!pip install arxiv\n",
+        "!pip install tenacity\n",
+        "!pip install pandas\n",
+        "!pip install tiktoken"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 1,
+      "id": "90f3b829",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import arxiv\n",
+        "from math import exp\n",
+        "import openai\n",
+        "import os\n",
+        "import pandas as pd\n",
+        "from tenacity import retry, wait_random_exponential, stop_after_attempt\n",
+        "import tiktoken\n",
+        "\n",
+        "client = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))\n",
+        "\n",
+        "OPENAI_MODEL = \"gpt-4\""
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "fdada886",
+      "metadata": {},
+      "source": [
+        "## Search\n",
+        "\n",
+        "We'll use the arXiv search service for this example, but this step could be performed by any search service you have. The key item to consider is over-fetching slightly to capture all the potentially relevant documents, before re-sorting them.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 2,
+      "id": "bf16c893",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "query = \"how do bi-encoders work for sentence embeddings\"\n",
+        "search = arxiv.Search(\n",
+        "    query=query, max_results=20, sort_by=arxiv.SortCriterion.Relevance\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 3,
+      "id": "4b020a1b",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "result_list = []\n",
+        "\n",
+        "for result in search.results():\n",
+        "    result_dict = {}\n",
+        "\n",
+        "    result_dict.update({\"title\": result.title})\n",
+        "    result_dict.update({\"summary\": result.summary})\n",
+        "\n",
+        "    # Taking the first url provided\n",
+        "    result_dict.update({\"article_url\": [x.href for x in result.links][0]})\n",
+        "    result_dict.update({\"pdf_url\": [x.href for x in result.links][1]})\n",
+        "    result_list.append(result_dict)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 4,
+      "id": "4fdce882",
+      "metadata": {},
+      "outputs": [
+        {
+          "data": {
+            "text/plain": [
+              "{'title': 'SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features',\n",
+              " 'summary': 'Models based on large-pretrained language models, such as S(entence)BERT,\\nprovide effective and efficient sentence embeddings that show high correlation\\nto human similarity ratings, but lack interpretability. On the other hand,\\ngraph metrics for graph-based meaning representations (e.g., Abstract Meaning\\nRepresentation, AMR) can make explicit the semantic aspects in which two\\nsentences are similar. However, such metrics tend to be slow, rely on parsers,\\nand do not reach state-of-the-art performance when rating sentence similarity.\\n  In this work, we aim at the best of both worlds, by learning to induce\\n$S$emantically $S$tructured $S$entence BERT embeddings (S$^3$BERT). Our\\nS$^3$BERT embeddings are composed of explainable sub-embeddings that emphasize\\nvarious semantic sentence features (e.g., semantic roles, negation, or\\nquantification). We show how to i) learn a decomposition of the sentence\\nembeddings into semantic features, through approximation of a suite of\\ninterpretable AMR graph metrics, and how to ii) preserve the overall power of\\nthe neural embeddings by controlling the decomposition learning process with a\\nsecond objective that enforces consistency with the similarity ratings of an\\nSBERT teacher model. In our experimental studies, we show that our approach\\noffers interpretability -- while fully preserving the effectiveness and\\nefficiency of the neural sentence embeddings.',\n",
+              " 'article_url': 'http://arxiv.org/abs/2206.07023v2',\n",
+              " 'pdf_url': 'http://arxiv.org/pdf/2206.07023v2'}"
+            ]
+          },
+          "execution_count": 4,
+          "metadata": {},
+          "output_type": "execute_result"
+        }
+      ],
+      "source": [
+        "result_list[0]"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 5,
+      "id": "7e6abb5b",
+      "metadata": {},
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "1: SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features\n",
+            "2: Are Classes Clusters?\n",
+            "3: Semantic Composition in Visually Grounded Language Models\n",
+            "4: Evaluating the Construct Validity of Text Embeddings with Application to Survey Questions\n",
+            "5: Learning Probabilistic Sentence Representations from Paraphrases\n",
+            "6: Exploiting Twitter as Source of Large Corpora of Weakly Similar Pairs for Semantic Sentence Embeddings\n",
+            "7: How to Probe Sentence Embeddings in Low-Resource Languages: On Structural Design Choices for Probing Task Evaluation\n",
+            "8: Clustering and Network Analysis for the Embedding Spaces of Sentences and Sub-Sentences\n",
+            "9: Vec2Sent: Probing Sentence Embeddings with Natural Language Generation\n",
+            "10: Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddings\n",
+            "11: SentPWNet: A Unified Sentence Pair Weighting Network for Task-specific Sentence Embedding\n",
+            "12: Learning Joint Representations of Videos and Sentences with Web Image Search\n",
+            "13: Character-based Neural Networks for Sentence Pair Modeling\n",
+            "14: Train Once, Test Anywhere: Zero-Shot Learning for Text Classification\n",
+            "15: Hierarchical GPT with Congruent Transformers for Multi-Sentence Language Models\n",
+            "16: Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models\n",
+            "17: In Search for Linear Relations in Sentence Embedding Spaces\n",
+            "18: Learning to Borrow -- Relation Representation for Without-Mention Entity-Pairs for Knowledge Graph Completion\n",
+            "19: Efficient and Flexible Topic Modeling using Pretrained Embeddings and Bag of Sentences\n",
+            "20: Relational Sentence Embedding for Flexible Semantic Matching\n"
+          ]
+        }
+      ],
+      "source": [
+        "for i, result in enumerate(result_list):\n",
+        "    print(f\"{i + 1}: {result['title']}\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "d5727678",
+      "metadata": {},
+      "source": [
+        "## Cross-encoder\n",
+        "\n",
+        "We'll create a cross-encoder using the ```Completions``` endpoint - the key factors to consider here are:\n",
+        "- Make your examples domain-specific - the strength of cross-encoders comes when you tailor them to your domain.\n",
+        "- There is a trade-off between how many potential examples to re-rank vs. processing speed. Consider batching and parallel processing cross-encoder requests to process them more quickly.\n",
+        "\n",
+        "The steps here are:\n",
+        "- Build a prompt to assess relevance and provide few-shot examples to tune it to your domain.\n",
+        "- Add a ```logit bias``` for the tokens for ``` Yes``` and ``` No``` to decrease the likelihood of any other tokens occurring.\n",
+        "- Return the classification of yes/no as well as the ```logprobs```.\n",
+        "- Rerank the results by the ```logprobs``` keyed on ``` Yes```."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 6,
+      "id": "ca634bf9",
+      "metadata": {},
+      "outputs": [
+        {
+          "data": {
+            "text/plain": [
+              "([3363], [1400])"
+            ]
+          },
+          "execution_count": 6,
+          "metadata": {},
+          "output_type": "execute_result"
+        }
+      ],
+      "source": [
+        "tokens = [\" Yes\", \" No\"]\n",
+        "tokenizer = tiktoken.encoding_for_model(OPENAI_MODEL)\n",
+        "ids = [tokenizer.encode(token) for token in tokens]\n",
+        "ids[0], ids[1]"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 7,
+      "id": "4fdf8c11",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "prompt = '''\n",
+        "You are an Assistant responsible for helping detect whether the retrieved document is relevant to the query. For a given input, you need to output a single token: \"Yes\" or \"No\" indicating the retrieved document is relevant to the query.\n",
+        "\n",
+        "Query: How to plant a tree?\n",
+        "Document: \"\"\"Cars were invented in 1886, when German inventor Carl Benz patented his Benz Patent-Motorwagen.[3][4][5] Cars became widely available during the 20th century. One of the first cars affordable by the masses was the 1908 Model T, an American car manufactured by the Ford Motor Company. Cars were rapidly adopted in the US, where they replaced horse-drawn carriages.[6] In Europe and other parts of the world, demand for automobiles did not increase until after World War II.[7] The car is considered an essential part of the developed economy.\"\"\"\n",
+        "Relevant: No\n",
+        "\n",
+        "Query: Has the coronavirus vaccine been approved?\n",
+        "Document: \"\"\"The Pfizer-BioNTech COVID-19 vaccine was approved for emergency use in the United States on December 11, 2020.\"\"\"\n",
+        "Relevant: Yes\n",
+        "\n",
+        "Query: What is the capital of France?\n",
+        "Document: \"\"\"Paris, France's capital, is a major European city and a global center for art, fashion, gastronomy and culture. Its 19th-century cityscape is crisscrossed by wide boulevards and the River Seine. Beyond such landmarks as the Eiffel Tower and the 12th-century, Gothic Notre-Dame cathedral, the city is known for its cafe culture and designer boutiques along the Rue du Faubourg Saint-Honoré.\"\"\"\n",
+        "Relevant: Yes\n",
+        "\n",
+        "Query: What are some papers to learn about PPO reinforcement learning?\n",
+        "Document: \"\"\"Proximal Policy Optimization and its Dynamic Version for Sequence Generation: In sequence generation task, many works use policy gradient for model optimization to tackle the intractable backpropagation issue when maximizing the non-differentiable evaluation metrics or fooling the discriminator in adversarial learning. In this paper, we replace policy gradient with proximal policy optimization (PPO), which is a proved more efficient reinforcement learning algorithm, and propose a dynamic approach for PPO (PPO-dynamic). We demonstrate the efficacy of PPO and PPO-dynamic on conditional sequence generation tasks including synthetic experiment and chit-chat chatbot. The results show that PPO and PPO-dynamic can beat policy gradient by stability and performance.\"\"\"\n",
+        "Relevant: Yes\n",
+        "\n",
+        "Query: Explain sentence embeddings\n",
+        "Document: \"\"\"Inside the bubble: exploring the environments of reionisation-era Lyman-α emitting galaxies with JADES and FRESCO: We present a study of the environments of 16 Lyman-α emitting galaxies (LAEs) in the reionisation era (5.8<z<8) identified by JWST/NIRSpec as part of the JWST Advanced Deep Extragalactic Survey (JADES). Unless situated in sufficiently (re)ionised regions, Lyman-α emission from these galaxies would be strongly absorbed by neutral gas in the intergalactic medium (IGM). We conservatively estimate sizes of the ionised regions required to reconcile the relatively low Lyman-α velocity offsets (ΔvLyα<300kms−1) with moderately high Lyman-α escape fractions (fesc,Lyα>5%) observed in our sample of LAEs, indicating the presence of ionised ``bubbles'' with physical sizes of the order of 0.1pMpc≲Rion≲1pMpc in a patchy reionisation scenario where the bubbles are embedded in a fully neutral IGM. Around half of the LAEs in our sample are found to coincide with large-scale galaxy overdensities seen in FRESCO at z∼5.8-5.9 and z∼7.3, suggesting Lyman-α transmission is strongly enhanced in such overdense regions, and underlining the importance of LAEs as tracers of the first large-scale ionised bubbles. Considering only spectroscopically confirmed galaxies, we find our sample of UV-faint LAEs (MUV≳−20mag) and their direct neighbours are generally not able to produce the required ionised regions based on the Lyman-α transmission properties, suggesting lower-luminosity sources likely play an important role in carving out these bubbles. These observations demonstrate the combined power of JWST multi-object and slitless spectroscopy in acquiring a unique view of the early stages of Cosmic Reionisation via the most distant LAEs.\"\"\"\n",
+        "Relevant: No\n",
+        "\n",
+        "Query: {query}\n",
+        "Document: \"\"\"{document}\"\"\"\n",
+        "Relevant:\n",
+        "'''\n",
+        "\n",
+        "\n",
+        "@retry(wait=wait_random_exponential(min=1, max=40), stop=stop_after_attempt(3))\n",
+        "def document_relevance(query, document):\n",
+        "    response = openai.chat.completions.create(\n",
+        "        model=\"text-davinci-003\",\n",
+        "        message=prompt.format(query=query, document=document),\n",
+        "        temperature=0,\n",
+        "        logprobs=True,\n",
+        "        logit_bias={3363: 1, 1400: 1},\n",
+        "    )\n",
+        "\n",
+        "    return (\n",
+        "        query,\n",
+        "        document,\n",
+        "        response.choices[0].message.content,\n",
+        "        response.choices[0].logprobs.token_logprobs[0],\n",
+        "    )"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 8,
+      "id": "753cd363",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "content = result_list[0][\"title\"] + \": \" + result_list[0][\"summary\"]\n",
+        "\n",
+        "# Set logprobs to 1 so our response will include the most probable token the model identified\n",
+        "response = openai.chat.completions.create(\n",
+        "    model=OPENAI_MODEL,\n",
+        "    prompt=prompt.format(query=query, document=content),\n",
+        "    temperature=0,\n",
+        "    logprobs=1,\n",
+        "    logit_bias={3363: 1, 1400: 1},\n",
+        "    max_tokens=1,\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 9,
+      "id": "7efef2fe",
+      "metadata": {},
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Result was Yes\n",
+            "Logprobs was -0.05869877\n",
+            "\n",
+            "Below is the full logprobs object\n",
+            "\n",
+            "\n",
+            "{\n",
+            "  \"tokens\": [\n",
+            "    \"Yes\"\n",
+            "  ],\n",
+            "  \"token_logprobs\": [\n",
+            "    -0.05869877\n",
+            "  ],\n",
+            "  \"top_logprobs\": [\n",
+            "    {\n",
+            "      \"Yes\": -0.05869877\n",
+            "    }\n",
+            "  ],\n",
+            "  \"text_offset\": [\n",
+            "    5764\n",
+            "  ]\n",
+            "}\n"
+          ]
+        }
+      ],
+      "source": [
+        "result = response.choices[0]\n",
+        "print(f\"Result was {result.message.content}\")\n",
+        "print(f\"Logprobs was {result.logprobs.token_logprobs[0]}\")\n",
+        "print(\"\\nBelow is the full logprobs object\\n\\n\")\n",
+        "print(result[\"logprobs\"])"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 10,
+      "id": "7683b6f7",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "output_list = []\n",
+        "for x in result_list:\n",
+        "    content = x[\"title\"] + \": \" + x[\"summary\"]\n",
+        "\n",
+        "    try:\n",
+        "        output_list.append(document_relevance(query, document=content))\n",
+        "\n",
+        "    except Exception as e:\n",
+        "        print(e)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 11,
+      "id": "57576313",
+      "metadata": {},
+      "outputs": [
+        {
+          "data": {
+            "text/plain": [
+              "[('how do bi-encoders work for sentence embeddings',\n",
+              "  'SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features: Models based on large-pretrained language models, such as S(entence)BERT,\\nprovide effective and efficient sentence embeddings that show high correlation\\nto human similarity ratings, but lack interpretability. On the other hand,\\ngraph metrics for graph-based meaning representations (e.g., Abstract Meaning\\nRepresentation, AMR) can make explicit the semantic aspects in which two\\nsentences are similar. However, such metrics tend to be slow, rely on parsers,\\nand do not reach state-of-the-art performance when rating sentence similarity.\\n  In this work, we aim at the best of both worlds, by learning to induce\\n$S$emantically $S$tructured $S$entence BERT embeddings (S$^3$BERT). Our\\nS$^3$BERT embeddings are composed of explainable sub-embeddings that emphasize\\nvarious semantic sentence features (e.g., semantic roles, negation, or\\nquantification). We show how to i) learn a decomposition of the sentence\\nembeddings into semantic features, through approximation of a suite of\\ninterpretable AMR graph metrics, and how to ii) preserve the overall power of\\nthe neural embeddings by controlling the decomposition learning process with a\\nsecond objective that enforces consistency with the similarity ratings of an\\nSBERT teacher model. In our experimental studies, we show that our approach\\noffers interpretability -- while fully preserving the effectiveness and\\nefficiency of the neural sentence embeddings.',\n",
+              "  'Yes',\n",
+              "  -0.05326408),\n",
+              " ('how do bi-encoders work for sentence embeddings',\n",
+              "  'Are Classes Clusters?: Sentence embedding models aim to provide general purpose embeddings for\\nsentences. Most of the models studied in this paper claim to perform well on\\nSTS tasks - but they do not report on their suitability for clustering. This\\npaper looks at four recent sentence embedding models (Universal Sentence\\nEncoder (Cer et al., 2018), Sentence-BERT (Reimers and Gurevych, 2019), LASER\\n(Artetxe and Schwenk, 2019), and DeCLUTR (Giorgi et al., 2020)). It gives a\\nbrief overview of the ideas behind their implementations. It then investigates\\nhow well topic classes in two text classification datasets (Amazon Reviews (Ni\\net al., 2019) and News Category Dataset (Misra, 2018)) map to clusters in their\\ncorresponding sentence embedding space. While the performance of the resulting\\nclassification model is far from perfect, it is better than random. This is\\ninteresting because the classification model has been constructed in an\\nunsupervised way. The topic classes in these real life topic classification\\ndatasets can be partly reconstructed by clustering the corresponding sentence\\nembeddings.',\n",
+              "  'No',\n",
+              "  -0.009535169),\n",
+              " ('how do bi-encoders work for sentence embeddings',\n",
+              "  \"Semantic Composition in Visually Grounded Language Models: What is sentence meaning and its ideal representation? Much of the expressive\\npower of human language derives from semantic composition, the mind's ability\\nto represent meaning hierarchically & relationally over constituents. At the\\nsame time, much sentential meaning is outside the text and requires grounding\\nin sensory, motor, and experiential modalities to be adequately learned.\\nAlthough large language models display considerable compositional ability,\\nrecent work shows that visually-grounded language models drastically fail to\\nrepresent compositional structure. In this thesis, we explore whether & how\\nmodels compose visually grounded semantics, and how we might improve their\\nability to do so.\\n  Specifically, we introduce 1) WinogroundVQA, a new compositional visual\\nquestion answering benchmark, 2) Syntactic Neural Module Distillation, a\\nmeasure of compositional ability in sentence embedding models, 3) Causal\\nTracing for Image Captioning Models to locate neural representations vital for\\nvision-language composition, 4) Syntactic MeanPool to inject a compositional\\ninductive bias into sentence embeddings, and 5) Cross-modal Attention\\nCongruence Regularization, a self-supervised objective function for\\nvision-language relation alignment. We close by discussing connections of our\\nwork to neuroscience, psycholinguistics, formal semantics, and philosophy.\",\n",
+              "  'No',\n",
+              "  -0.008887106),\n",
+              " ('how do bi-encoders work for sentence embeddings',\n",
+              "  \"Evaluating the Construct Validity of Text Embeddings with Application to Survey Questions: Text embedding models from Natural Language Processing can map text data\\n(e.g. words, sentences, documents) to supposedly meaningful numerical\\nrepresentations (a.k.a. text embeddings). While such models are increasingly\\napplied in social science research, one important issue is often not addressed:\\nthe extent to which these embeddings are valid representations of constructs\\nrelevant for social science research. We therefore propose the use of the\\nclassic construct validity framework to evaluate the validity of text\\nembeddings. We show how this framework can be adapted to the opaque and\\nhigh-dimensional nature of text embeddings, with application to survey\\nquestions. We include several popular text embedding methods (e.g. fastText,\\nGloVe, BERT, Sentence-BERT, Universal Sentence Encoder) in our construct\\nvalidity analyses. We find evidence of convergent and discriminant validity in\\nsome cases. We also show that embeddings can be used to predict respondent's\\nanswers to completely new survey questions. Furthermore, BERT-based embedding\\ntechniques and the Universal Sentence Encoder provide more valid\\nrepresentations of survey questions than do others. Our results thus highlight\\nthe necessity to examine the construct validity of text embeddings before\\ndeploying them in social science research.\",\n",
+              "  'No',\n",
+              "  -0.008583762),\n",
+              " ('how do bi-encoders work for sentence embeddings',\n",
+              "  'Learning Probabilistic Sentence Representations from Paraphrases: Probabilistic word embeddings have shown effectiveness in capturing notions\\nof generality and entailment, but there is very little work on doing the\\nanalogous type of investigation for sentences. In this paper we define\\nprobabilistic models that produce distributions for sentences. Our\\nbest-performing model treats each word as a linear transformation operator\\napplied to a multivariate Gaussian distribution. We train our models on\\nparaphrases and demonstrate that they naturally capture sentence specificity.\\nWhile our proposed model achieves the best performance overall, we also show\\nthat specificity is represented by simpler architectures via the norm of the\\nsentence vectors. Qualitative analysis shows that our probabilistic model\\ncaptures sentential entailment and provides ways to analyze the specificity and\\npreciseness of individual words.',\n",
+              "  'No',\n",
+              "  -0.011975748),\n",
+              " ('how do bi-encoders work for sentence embeddings',\n",
+              "  \"Exploiting Twitter as Source of Large Corpora of Weakly Similar Pairs for Semantic Sentence Embeddings: Semantic sentence embeddings are usually supervisedly built minimizing\\ndistances between pairs of embeddings of sentences labelled as semantically\\nsimilar by annotators. Since big labelled datasets are rare, in particular for\\nnon-English languages, and expensive, recent studies focus on unsupervised\\napproaches that require not-paired input sentences. We instead propose a\\nlanguage-independent approach to build large datasets of pairs of informal\\ntexts weakly similar, without manual human effort, exploiting Twitter's\\nintrinsic powerful signals of relatedness: replies and quotes of tweets. We use\\nthe collected pairs to train a Transformer model with triplet-like structures,\\nand we test the generated embeddings on Twitter NLP similarity tasks (PIT and\\nTURL) and STSb. We also introduce four new sentence ranking evaluation\\nbenchmarks of informal texts, carefully extracted from the initial collections\\nof tweets, proving not only that our best model learns classical Semantic\\nTextual Similarity, but also excels on tasks where pairs of sentences are not\\nexact paraphrases. Ablation studies reveal how increasing the corpus size\\ninfluences positively the results, even at 2M samples, suggesting that bigger\\ncollections of Tweets still do not contain redundant information about semantic\\nsimilarities.\",\n",
+              "  'No',\n",
+              "  -0.01219046),\n",
+              " ('how do bi-encoders work for sentence embeddings',\n",
+              "  \"How to Probe Sentence Embeddings in Low-Resource Languages: On Structural Design Choices for Probing Task Evaluation: Sentence encoders map sentences to real valued vectors for use in downstream\\napplications. To peek into these representations - e.g., to increase\\ninterpretability of their results - probing tasks have been designed which\\nquery them for linguistic knowledge. However, designing probing tasks for\\nlesser-resourced languages is tricky, because these often lack large-scale\\nannotated data or (high-quality) dependency parsers as a prerequisite of\\nprobing task design in English. To investigate how to probe sentence embeddings\\nin such cases, we investigate sensitivity of probing task results to structural\\ndesign choices, conducting the first such large scale study. We show that\\ndesign choices like size of the annotated probing dataset and type of\\nclassifier used for evaluation do (sometimes substantially) influence probing\\noutcomes. We then probe embeddings in a multilingual setup with design choices\\nthat lie in a 'stable region', as we identify for English, and find that\\nresults on English do not transfer to other languages. Fairer and more\\ncomprehensive sentence-level probing evaluation should thus be carried out on\\nmultiple languages in the future.\",\n",
+              "  'No',\n",
+              "  -0.015550519),\n",
+              " ('how do bi-encoders work for sentence embeddings',\n",
+              "  'Clustering and Network Analysis for the Embedding Spaces of Sentences and Sub-Sentences: Sentence embedding methods offer a powerful approach for working with short\\ntextual constructs or sequences of words. By representing sentences as dense\\nnumerical vectors, many natural language processing (NLP) applications have\\nimproved their performance. However, relatively little is understood about the\\nlatent structure of sentence embeddings. Specifically, research has not\\naddressed whether the length and structure of sentences impact the sentence\\nembedding space and topology. This paper reports research on a set of\\ncomprehensive clustering and network analyses targeting sentence and\\nsub-sentence embedding spaces. Results show that one method generates the most\\nclusterable embeddings. In general, the embeddings of span sub-sentences have\\nbetter clustering properties than the original sentences. The results have\\nimplications for future sentence embedding models and applications.',\n",
+              "  'No',\n",
+              "  -0.012663184),\n",
+              " ('how do bi-encoders work for sentence embeddings',\n",
+              "  'Vec2Sent: Probing Sentence Embeddings with Natural Language Generation: We introspect black-box sentence embeddings by conditionally generating from\\nthem with the objective to retrieve the underlying discrete sentence. We\\nperceive of this as a new unsupervised probing task and show that it correlates\\nwell with downstream task performance. We also illustrate how the language\\ngenerated from different encoders differs. We apply our approach to generate\\nsentence analogies from sentence embeddings.',\n",
+              "  'Yes',\n",
+              "  -0.004863006),\n",
+              " ('how do bi-encoders work for sentence embeddings',\n",
+              "  'Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddings: Semantic representation learning for sentences is an important and\\nwell-studied problem in NLP. The current trend for this task involves training\\na Transformer-based sentence encoder through a contrastive objective with text,\\ni.e., clustering sentences with semantically similar meanings and scattering\\nothers. In this work, we find the performance of Transformer models as sentence\\nencoders can be improved by training with multi-modal multi-task losses, using\\nunpaired examples from another modality (e.g., sentences and unrelated\\nimage/audio data). In particular, besides learning by the contrastive loss on\\ntext, our model clusters examples from a non-linguistic domain (e.g.,\\nvisual/audio) with a similar contrastive loss at the same time. The reliance of\\nour framework on unpaired non-linguistic data makes it language-agnostic,\\nenabling it to be widely applicable beyond English NLP. Experiments on 7\\nsemantic textual similarity benchmarks reveal that models trained with the\\nadditional non-linguistic (/images/audio) contrastive objective lead to higher\\nquality sentence embeddings. This indicates that Transformer models are able to\\ngeneralize better by doing a similar task (i.e., clustering) with unpaired\\nexamples from different modalities in a multi-task fashion.',\n",
+              "  'No',\n",
+              "  -0.013869206)]"
+            ]
+          },
+          "execution_count": 11,
+          "metadata": {},
+          "output_type": "execute_result"
+        }
+      ],
+      "source": [
+        "output_list[:10]"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 12,
+      "id": "29a4dc08",
+      "metadata": {},
+      "outputs": [
+        {
+          "data": {
+            "text/html": [
+              "<div>\n",
+              "<style scoped>\n",
+              "    .dataframe tbody tr th:only-of-type {\n",
+              "        vertical-align: middle;\n",
+              "    }\n",
+              "\n",
+              "    .dataframe tbody tr th {\n",
+              "        vertical-align: top;\n",
+              "    }\n",
+              "\n",
+              "    .dataframe thead th {\n",
+              "        text-align: right;\n",
+              "    }\n",
+              "</style>\n",
+              "<table border=\"1\" class=\"dataframe\">\n",
+              "  <thead>\n",
+              "    <tr style=\"text-align: right;\">\n",
+              "      <th></th>\n",
+              "      <th>index</th>\n",
+              "      <th>query</th>\n",
+              "      <th>document</th>\n",
+              "      <th>prediction</th>\n",
+              "      <th>logprobs</th>\n",
+              "      <th>probability</th>\n",
+              "      <th>yes_probability</th>\n",
+              "    </tr>\n",
+              "  </thead>\n",
+              "  <tbody>\n",
+              "    <tr>\n",
+              "      <th>0</th>\n",
+              "      <td>0</td>\n",
+              "      <td>how do bi-encoders work for sentence embeddings</td>\n",
+              "      <td>SBERT studies Meaning Representations: Decompo...</td>\n",
+              "      <td>Yes</td>\n",
+              "      <td>-0.053264</td>\n",
+              "      <td>0.948130</td>\n",
+              "      <td>0.948130</td>\n",
+              "    </tr>\n",
+              "    <tr>\n",
+              "      <th>1</th>\n",
+              "      <td>1</td>\n",
+              "      <td>how do bi-encoders work for sentence embeddings</td>\n",
+              "      <td>Are Classes Clusters?: Sentence embedding mode...</td>\n",
+              "      <td>No</td>\n",
+              "      <td>-0.009535</td>\n",
+              "      <td>0.990510</td>\n",
+              "      <td>0.009490</td>\n",
+              "    </tr>\n",
+              "    <tr>\n",
+              "      <th>2</th>\n",
+              "      <td>2</td>\n",
+              "      <td>how do bi-encoders work for sentence embeddings</td>\n",
+              "      <td>Semantic Composition in Visually Grounded Lang...</td>\n",
+              "      <td>No</td>\n",
+              "      <td>-0.008887</td>\n",
+              "      <td>0.991152</td>\n",
+              "      <td>0.008848</td>\n",
+              "    </tr>\n",
+              "    <tr>\n",
+              "      <th>3</th>\n",
+              "      <td>3</td>\n",
+              "      <td>how do bi-encoders work for sentence embeddings</td>\n",
+              "      <td>Evaluating the Construct Validity of Text Embe...</td>\n",
+              "      <td>No</td>\n",
+              "      <td>-0.008584</td>\n",
+              "      <td>0.991453</td>\n",
+              "      <td>0.008547</td>\n",
+              "    </tr>\n",
+              "    <tr>\n",
+              "      <th>4</th>\n",
+              "      <td>4</td>\n",
+              "      <td>how do bi-encoders work for sentence embeddings</td>\n",
+              "      <td>Learning Probabilistic Sentence Representation...</td>\n",
+              "      <td>No</td>\n",
+              "      <td>-0.011976</td>\n",
+              "      <td>0.988096</td>\n",
+              "      <td>0.011904</td>\n",
+              "    </tr>\n",
+              "  </tbody>\n",
+              "</table>\n",
+              "</div>"
+            ],
+            "text/plain": [
+              "   index                                            query  \\\n",
+              "0      0  how do bi-encoders work for sentence embeddings   \n",
+              "1      1  how do bi-encoders work for sentence embeddings   \n",
+              "2      2  how do bi-encoders work for sentence embeddings   \n",
+              "3      3  how do bi-encoders work for sentence embeddings   \n",
+              "4      4  how do bi-encoders work for sentence embeddings   \n",
+              "\n",
+              "                                            document prediction  logprobs  \\\n",
+              "0  SBERT studies Meaning Representations: Decompo...        Yes -0.053264   \n",
+              "1  Are Classes Clusters?: Sentence embedding mode...         No -0.009535   \n",
+              "2  Semantic Composition in Visually Grounded Lang...         No -0.008887   \n",
+              "3  Evaluating the Construct Validity of Text Embe...         No -0.008584   \n",
+              "4  Learning Probabilistic Sentence Representation...         No -0.011976   \n",
+              "\n",
+              "   probability  yes_probability  \n",
+              "0     0.948130         0.948130  \n",
+              "1     0.990510         0.009490  \n",
+              "2     0.991152         0.008848  \n",
+              "3     0.991453         0.008547  \n",
+              "4     0.988096         0.011904  "
+            ]
+          },
+          "execution_count": 12,
+          "metadata": {},
+          "output_type": "execute_result"
+        }
+      ],
+      "source": [
+        "output_df = pd.DataFrame(\n",
+        "    output_list, columns=[\"query\", \"document\", \"prediction\", \"logprobs\"]\n",
+        ").reset_index()\n",
+        "# Use exp() to convert logprobs into probability\n",
+        "output_df[\"probability\"] = output_df[\"logprobs\"].apply(exp)\n",
+        "# Reorder based on likelihood of being Yes\n",
+        "output_df[\"yes_probability\"] = output_df.apply(\n",
+        "    lambda x: x[\"probability\"] * -1 + 1\n",
+        "    if x[\"prediction\"] == \"No\"\n",
+        "    else x[\"probability\"],\n",
+        "    axis=1,\n",
+        ")\n",
+        "output_df.head()"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 13,
+      "id": "a647f120",
+      "metadata": {},
+      "outputs": [
+        {
+          "data": {
+            "text/html": [
+              "<div>\n",
+              "<style scoped>\n",
+              "    .dataframe tbody tr th:only-of-type {\n",
+              "        vertical-align: middle;\n",
+              "    }\n",
+              "\n",
+              "    .dataframe tbody tr th {\n",
+              "        vertical-align: top;\n",
+              "    }\n",
+              "\n",
+              "    .dataframe thead th {\n",
+              "        text-align: right;\n",
+              "    }\n",
+              "</style>\n",
+              "<table border=\"1\" class=\"dataframe\">\n",
+              "  <thead>\n",
+              "    <tr style=\"text-align: right;\">\n",
+              "      <th></th>\n",
+              "      <th>level_0</th>\n",
+              "      <th>index</th>\n",
+              "      <th>query</th>\n",
+              "      <th>document</th>\n",
+              "      <th>prediction</th>\n",
+              "      <th>logprobs</th>\n",
+              "      <th>probability</th>\n",
+              "      <th>yes_probability</th>\n",
+              "    </tr>\n",
+              "  </thead>\n",
+              "  <tbody>\n",
+              "    <tr>\n",
+              "      <th>0</th>\n",
+              "      <td>16</td>\n",
+              "      <td>16</td>\n",
+              "      <td>how do bi-encoders work for sentence embeddings</td>\n",
+              "      <td>In Search for Linear Relations in Sentence Emb...</td>\n",
+              "      <td>Yes</td>\n",
+              "      <td>-0.004824</td>\n",
+              "      <td>0.995187</td>\n",
+              "      <td>0.995187</td>\n",
+              "    </tr>\n",
+              "    <tr>\n",
+              "      <th>1</th>\n",
+              "      <td>8</td>\n",
+              "      <td>8</td>\n",
+              "      <td>how do bi-encoders work for sentence embeddings</td>\n",
+              "      <td>Vec2Sent: Probing Sentence Embeddings with Nat...</td>\n",
+              "      <td>Yes</td>\n",
+              "      <td>-0.004863</td>\n",
+              "      <td>0.995149</td>\n",
+              "      <td>0.995149</td>\n",
+              "    </tr>\n",
+              "    <tr>\n",
+              "      <th>2</th>\n",
+              "      <td>19</td>\n",
+              "      <td>19</td>\n",
+              "      <td>how do bi-encoders work for sentence embeddings</td>\n",
+              "      <td>Relational Sentence Embedding for Flexible Sem...</td>\n",
+              "      <td>Yes</td>\n",
+              "      <td>-0.038814</td>\n",
+              "      <td>0.961930</td>\n",
+              "      <td>0.961930</td>\n",
+              "    </tr>\n",
+              "    <tr>\n",
+              "      <th>3</th>\n",
+              "      <td>0</td>\n",
+              "      <td>0</td>\n",
+              "      <td>how do bi-encoders work for sentence embeddings</td>\n",
+              "      <td>SBERT studies Meaning Representations: Decompo...</td>\n",
+              "      <td>Yes</td>\n",
+              "      <td>-0.053264</td>\n",
+              "      <td>0.948130</td>\n",
+              "      <td>0.948130</td>\n",
+              "    </tr>\n",
+              "    <tr>\n",
+              "      <th>4</th>\n",
+              "      <td>15</td>\n",
+              "      <td>15</td>\n",
+              "      <td>how do bi-encoders work for sentence embeddings</td>\n",
+              "      <td>Sentence-T5: Scalable Sentence Encoders from P...</td>\n",
+              "      <td>No</td>\n",
+              "      <td>-0.291893</td>\n",
+              "      <td>0.746849</td>\n",
+              "      <td>0.253151</td>\n",
+              "    </tr>\n",
+              "    <tr>\n",
+              "      <th>5</th>\n",
+              "      <td>6</td>\n",
+              "      <td>6</td>\n",
+              "      <td>how do bi-encoders work for sentence embeddings</td>\n",
+              "      <td>How to Probe Sentence Embeddings in Low-Resour...</td>\n",
+              "      <td>No</td>\n",
+              "      <td>-0.015551</td>\n",
+              "      <td>0.984570</td>\n",
+              "      <td>0.015430</td>\n",
+              "    </tr>\n",
+              "    <tr>\n",
+              "      <th>6</th>\n",
+              "      <td>18</td>\n",
+              "      <td>18</td>\n",
+              "      <td>how do bi-encoders work for sentence embeddings</td>\n",
+              "      <td>Efficient and Flexible Topic Modeling using Pr...</td>\n",
+              "      <td>No</td>\n",
+              "      <td>-0.015296</td>\n",
+              "      <td>0.984820</td>\n",
+              "      <td>0.015180</td>\n",
+              "    </tr>\n",
+              "    <tr>\n",
+              "      <th>7</th>\n",
+              "      <td>9</td>\n",
+              "      <td>9</td>\n",
+              "      <td>how do bi-encoders work for sentence embeddings</td>\n",
+              "      <td>Non-Linguistic Supervision for Contrastive Lea...</td>\n",
+              "      <td>No</td>\n",
+              "      <td>-0.013869</td>\n",
+              "      <td>0.986227</td>\n",
+              "      <td>0.013773</td>\n",
+              "    </tr>\n",
+              "    <tr>\n",
+              "      <th>8</th>\n",
+              "      <td>12</td>\n",
+              "      <td>12</td>\n",
+              "      <td>how do bi-encoders work for sentence embeddings</td>\n",
+              "      <td>Character-based Neural Networks for Sentence P...</td>\n",
+              "      <td>No</td>\n",
+              "      <td>-0.012866</td>\n",
+              "      <td>0.987216</td>\n",
+              "      <td>0.012784</td>\n",
+              "    </tr>\n",
+              "    <tr>\n",
+              "      <th>9</th>\n",
+              "      <td>7</td>\n",
+              "      <td>7</td>\n",
+              "      <td>how do bi-encoders work for sentence embeddings</td>\n",
+              "      <td>Clustering and Network Analysis for the Embedd...</td>\n",
+              "      <td>No</td>\n",
+              "      <td>-0.012663</td>\n",
+              "      <td>0.987417</td>\n",
+              "      <td>0.012583</td>\n",
+              "    </tr>\n",
+              "  </tbody>\n",
+              "</table>\n",
+              "</div>"
+            ],
+            "text/plain": [
+              "   level_0  index                                            query  \\\n",
+              "0       16     16  how do bi-encoders work for sentence embeddings   \n",
+              "1        8      8  how do bi-encoders work for sentence embeddings   \n",
+              "2       19     19  how do bi-encoders work for sentence embeddings   \n",
+              "3        0      0  how do bi-encoders work for sentence embeddings   \n",
+              "4       15     15  how do bi-encoders work for sentence embeddings   \n",
+              "5        6      6  how do bi-encoders work for sentence embeddings   \n",
+              "6       18     18  how do bi-encoders work for sentence embeddings   \n",
+              "7        9      9  how do bi-encoders work for sentence embeddings   \n",
+              "8       12     12  how do bi-encoders work for sentence embeddings   \n",
+              "9        7      7  how do bi-encoders work for sentence embeddings   \n",
+              "\n",
+              "                                            document prediction  logprobs  \\\n",
+              "0  In Search for Linear Relations in Sentence Emb...        Yes -0.004824   \n",
+              "1  Vec2Sent: Probing Sentence Embeddings with Nat...        Yes -0.004863   \n",
+              "2  Relational Sentence Embedding for Flexible Sem...        Yes -0.038814   \n",
+              "3  SBERT studies Meaning Representations: Decompo...        Yes -0.053264   \n",
+              "4  Sentence-T5: Scalable Sentence Encoders from P...         No -0.291893   \n",
+              "5  How to Probe Sentence Embeddings in Low-Resour...         No -0.015551   \n",
+              "6  Efficient and Flexible Topic Modeling using Pr...         No -0.015296   \n",
+              "7  Non-Linguistic Supervision for Contrastive Lea...         No -0.013869   \n",
+              "8  Character-based Neural Networks for Sentence P...         No -0.012866   \n",
+              "9  Clustering and Network Analysis for the Embedd...         No -0.012663   \n",
+              "\n",
+              "   probability  yes_probability  \n",
+              "0     0.995187         0.995187  \n",
+              "1     0.995149         0.995149  \n",
+              "2     0.961930         0.961930  \n",
+              "3     0.948130         0.948130  \n",
+              "4     0.746849         0.253151  \n",
+              "5     0.984570         0.015430  \n",
+              "6     0.984820         0.015180  \n",
+              "7     0.986227         0.013773  \n",
+              "8     0.987216         0.012784  \n",
+              "9     0.987417         0.012583  "
+            ]
+          },
+          "execution_count": 13,
+          "metadata": {},
+          "output_type": "execute_result"
+        }
+      ],
+      "source": [
+        "# Return reranked results\n",
+        "reranked_df = output_df.sort_values(\n",
+        "    by=[\"yes_probability\"], ascending=False\n",
+        ").reset_index()\n",
+        "reranked_df.head(10)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 14,
+      "id": "610b2c7f",
+      "metadata": {},
+      "outputs": [
+        {
+          "data": {
+            "text/plain": [
+              "'In Search for Linear Relations in Sentence Embedding Spaces: We present an introductory investigation into continuous-space vector\\nrepresentations of sentences. We acquire pairs of very similar sentences\\ndiffering only by a small alterations (such as change of a noun, adding an\\nadjective, noun or punctuation) from datasets for natural language inference\\nusing a simple pattern method. We look into how such a small change within the\\nsentence text affects its representation in the continuous space and how such\\nalterations are reflected by some of the popular sentence embedding models. We\\nfound that vector differences of some embeddings actually reflect small changes\\nwithin a sentence.'"
+            ]
+          },
+          "execution_count": 14,
+          "metadata": {},
+          "output_type": "execute_result"
+        }
+      ],
+      "source": [
+        "# Inspect our new top document following reranking\n",
+        "reranked_df[\"document\"][0]"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "f372d311",
+      "metadata": {},
+      "source": [
+        "## Conclusion\n",
+        "\n",
+        "We've shown how to create a tailored cross-encoder to rerank academic papers. This approach will work best where there are domain-specific nuances that can be used to pick the most relevant corpus for your users, and where some pre-filtering has taken place to limit the amount of data the cross-encoder will need to process. \n",
+        "\n",
+        "A few typical use cases we've seen are:\n",
+        "- Returning a list of 100 most relevant stock reports, then re-ordering into a top 5 or 10 based on the detailed context of a particular set of customer portfolios\n",
+        "- Running after a classic rules-based search that gets the top 100 or 1000 most relevant results to prune it according to a specific user's context\n",
+        "\n",
+        "\n",
+        "### Taking this forward\n",
+        "\n",
+        "Taking the few-shot approach, as we have here, can work well when the domain is general enough that a small number of examples will cover most reranking cases. However, as the differences between documents become more specific you may want to consider the ```Fine-tuning``` endpoint to make a more elaborate cross-encoder with a wider variety of examples.\n",
+        "\n",
+        "There is also a latency impact of using ```text-davinci-003``` that you'll need to consider, with even our few examples above taking a couple seconds each - again, the ```Fine-tuning``` endpoint may help you here if you are able to get decent results from an ```ada``` or ```babbage``` fine-tuned model.\n",
+        "\n",
+        "We've used the ```Completions``` endpoint from OpenAI to build our cross-encoder, but this area is well-served by the open-source community. [Here](https://huggingface.co/jeffwan/mmarco-mMiniLMv2-L12-H384-v1) is an example from HuggingFace, for example.\n",
+        "\n",
+        "We hope you find this useful for tuning your search use cases, and look forward to seeing what you build."
+      ]
+    }
+  ],
+  "metadata": {
+    "kernelspec": {
+      "display_name": "openai_test",
+      "language": "python",
+      "name": "openai_test"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.10.11"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 5
+}

examples/Semantic_text_search_using_embeddings.ipynb

@@ -0,0 +1,204 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Semantic text search using embeddings\n",
+    "\n",
+    "We can search through all our reviews semantically in a very efficient manner and at very low cost, by embedding our search query, and then finding the most similar reviews. The dataset is created in the [Get_embeddings_from_dataset Notebook](Get_embeddings_from_dataset.ipynb)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import numpy as np\n",
+    "from ast import literal_eval\n",
+    "\n",
+    "datafile_path = \"data/fine_food_reviews_with_embeddings_1k.csv\"\n",
+    "\n",
+    "df = pd.read_csv(datafile_path)\n",
+    "df[\"embedding\"] = df.embedding.apply(literal_eval).apply(np.array)\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Here we compare the cosine similarity of the embeddings of the query and the documents, and show top_n best matches."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Delicious!:  I enjoy this white beans seasoning, it gives a rich flavor to the beans I just love it, my mother in law didn't know about this Zatarain's brand and now she is traying different seasoning\n",
+      "\n",
+      "Fantastic Instant Refried beans:  Fantastic Instant Refried Beans have been a staple for my family now for nearly 20 years.  All 7 of us love it and my grown kids are passing on the tradition.\n",
+      "\n",
+      "Delicious:  While there may be better coffee beans available, this is my first purchase and my first time grinding my own beans.  I read several reviews before purchasing this brand, and am extremely \n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "from utils.embeddings_utils import get_embedding, cosine_similarity\n",
+    "\n",
+    "# search through the reviews for a specific product\n",
+    "def search_reviews(df, product_description, n=3, pprint=True):\n",
+    "    product_embedding = get_embedding(\n",
+    "        product_description,\n",
+    "        model=\"text-embedding-3-small\"\n",
+    "    )\n",
+    "    df[\"similarity\"] = df.embedding.apply(lambda x: cosine_similarity(x, product_embedding))\n",
+    "\n",
+    "    results = (\n",
+    "        df.sort_values(\"similarity\", ascending=False)\n",
+    "        .head(n)\n",
+    "        .combined.str.replace(\"Title: \", \"\")\n",
+    "        .str.replace(\"; Content:\", \": \")\n",
+    "    )\n",
+    "    if pprint:\n",
+    "        for r in results:\n",
+    "            print(r[:200])\n",
+    "            print()\n",
+    "    return results\n",
+    "\n",
+    "\n",
+    "results = search_reviews(df, \"delicious beans\", n=3)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Tasty and Quick Pasta:  Barilla Whole Grain Fusilli with Vegetable Marinara is tasty and has an excellent chunky vegetable marinara.  I just wish there was more of it.  If you aren't starving or on a \n",
+      "\n",
+      "sooo good:  tastes so good. Worth the money. My boyfriend hates wheat pasta and LOVES this. cooks fast tastes great.I love this brand and started buying more of their pastas. Bulk is best.\n",
+      "\n",
+      "Bland and vaguely gamy tasting, skip this one:  As far as prepared dinner kits go, \"Barilla Whole Grain Mezze Penne with Tomato and Basil Sauce\" just did not do it for me...and this is coming from a p\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "results = search_reviews(df, \"whole wheat pasta\", n=3)\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can search through these reviews easily. To speed up computation, we can use a special algorithm, aimed at faster search through embeddings."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "great product, poor delivery:  The coffee is excellent and I am a repeat buyer.  Problem this time was with the UPS delivery.  They left the box in front of my garage door in the middle of the drivewa\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "results = search_reviews(df, \"bad delivery\", n=1)\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "As we can see, this can immediately deliver a lot of value. In this example we show being able to quickly find the examples of delivery failures."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Disappointed:  The metal cover has severely disformed. And most of the cookies inside have been crushed into small pieces. Shopping experience is awful. I'll never buy it online again.\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "results = search_reviews(df, \"spoilt\", n=1)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Great food!:  I wanted a food for a a dog with skin problems. His skin greatly improved with the switch, though he still itches some.  He loves the food. No recalls, American made with American ingred\n",
+      "\n",
+      "Great food!:  I wanted a food for a a dog with skin problems. His skin greatly improved with the switch, though he still itches some.  He loves the food. No recalls, American made with American ingred\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "results = search_reviews(df, \"pet food\", n=2)\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "openai",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.5"
+  },
+  "orig_nbformat": 4,
+  "vscode": {
+   "interpreter": {
+    "hash": "365536dcbde60510dc9073d6b991cd35db2d9bac356a11f5b64279a5e6708b97"
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}