comet-ml--opik/sdks/python/examples/agent_config_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7fb27b941602401d91542211134fc71a",
   "metadata": {},
   "source": [
    "# Agent Config API Demo\n",
    "\n",
    "This notebook walks through the Opik **Agent Config** API:\n",
    "\n",
    "1. `get_or_create_config` — fetch from the backend, auto-creating from a fallback when nothing exists yet\n",
    "2. `get_or_create_config` with `fallback` only (returns fallback when backend is unreachable or empty)\n",
    "3. `create_config` — unconditionally write a new config version\n",
    "4. `set_config_env` — tag a version with an environment name (e.g. `\"prod\"`, `\"staging\"`)\n",
    "5. Fetching by `env` and by explicit `version` name\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acae54e37e7d407bbb7b55eff062a284",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a63283cbaf04dbcab1f6479b197f3a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install opik --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8dd0d8092fe74a7c96281538738b07e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import uuid\n",
    "from typing import Optional\n",
    "\n",
    "import opik\n",
    "from opik.api_objects.agent_config.cache import get_global_registry\n",
    "\n",
    "# Configure once — reads OPIK_API_KEY and OPIK_URL_OVERRIDE from env if set.\n",
    "# opik.configure(use_local=True)  # swap for opik.configure() when using Opik Cloud"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "72eea5119410473aa328ad9291626812",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Project: agent-config-demo-e9013134\n"
     ]
    }
   ],
   "source": [
    "client = opik.Opik()\n",
    "\n",
    "# Give each demo run its own project so configs don't bleed between runs.\n",
    "PROJECT = f\"agent-config-demo-{uuid.uuid4().hex[:8]}\"\n",
    "print(f\"Project: {PROJECT}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8edb47106e1a46a883d545849b8ab81b",
   "metadata": {},
   "source": [
    "## Define a typed Config subclass\n",
    "\n",
    "`opik.Config` is a Pydantic-based model. Subclass it to declare the fields your agent needs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "10185d26023b46108eb7d9f57d49d2b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "class AgentConfig(opik.Config):\n",
    "    temperature: float\n",
    "    model: str\n",
    "    system_prompt: Optional[str] = None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8763a12b2bbd4a93a75aff182afb95dc",
   "metadata": {},
   "source": [
    "---\n",
    "## 1. `get_or_create_config` — first call auto-creates from fallback\n",
    "\n",
    "The project has no config yet. `get_or_create_config` detects this and writes the\n",
    "fallback values as the first version. The backend automatically tags the first\n",
    "version as `\"prod\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7623eae2785240b9bd12b16a66d81610",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "OPIK: Started logging traces to the \"agent-config-demo-e9013134\" project at https://test.dev.comet.com/opik/api/v1/session/redirect/projects/?trace_id=019d8bf7-c5e8-7314-8849-2b52040092c0&path=aHR0cHM6Ly90ZXN0LmRldi5jb21ldC5jb20vb3Bpay9hcGkv.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== First call — auto-creates from fallback ===\n",
      "  is_fallback : False\n",
      "  temperature : 0.5\n",
      "  model       : gpt-3.5-turbo\n",
      "  system_prompt: You are a helpful assistant.\n"
     ]
    }
   ],
   "source": [
    "FALLBACK_V1 = AgentConfig(\n",
    "    temperature=0.5,\n",
    "    model=\"gpt-3.5-turbo\",\n",
    "    system_prompt=\"You are a helpful assistant.\",\n",
    ")\n",
    "\n",
    "\n",
    "# get_or_create_config must be called from inside an @opik.track function.\n",
    "@opik.track(project_name=PROJECT)\n",
    "def run_agent(user_message: str):\n",
    "    cfg = client.get_or_create_config(\n",
    "        fallback=AgentConfig(\n",
    "            temperature=0.54,\n",
    "            model=\"gpt-3.5aaaa-turbo\",\n",
    "            system_prompt=\"You are a helpful assistant.\",\n",
    "        ),  # optional, but preferred\n",
    "        project_name=PROJECT,\n",
    "    )\n",
    "    # cfg is an AgentConfig instance because we passed a typed fallback.\n",
    "    print(f\"  is_fallback : {cfg.is_fallback}\")\n",
    "    print(f\"  temperature : {cfg.temperature}\")\n",
    "    print(f\"  model       : {cfg.model}\")\n",
    "    print(f\"  system_prompt: {cfg.system_prompt}\")\n",
    "    return cfg\n",
    "\n",
    "\n",
    "print(\"=== First call — auto-creates from fallback ===\")\n",
    "cfg_v1 = run_agent(\"Hello!\")\n",
    "assert cfg_v1.is_fallback is False, (\n",
    "    \"auto-created config should NOT be marked as fallback\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cdc8c89c7104fffa095e18ddfef8986",
   "metadata": {},
   "source": [
    "---\n",
    "## 2. `get_or_create_config` — returns fallback when backend unreachable\n",
    "\n",
    "If the backend times out or is unreachable **and** a `fallback` is provided,\n",
    "`get_or_create_config` returns the fallback with `is_fallback=True` instead of\n",
    "raising an error. We simulate this by passing an unreachable URL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b118ea5561624da68c537baed56e602f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== Call against unreachable backend — returns fallback ===\n",
      "  is_fallback : True\n",
      "  temperature : 0.5  (from local fallback)\n",
      "  model       : gpt-3.5-turbo\n"
     ]
    }
   ],
   "source": [
    "# Point a second client at a non-existent host to force a timeout.\n",
    "unreachable_client = opik.Opik(\n",
    "    host=\"http://127.0.0.1:19999\",  # nothing listening here\n",
    "    api_key=\"demo\",\n",
    ")\n",
    "\n",
    "\n",
    "@opik.track(project_name=PROJECT)\n",
    "def run_agent_offline(user_message: str):\n",
    "    cfg = unreachable_client.get_or_create_config(\n",
    "        fallback=FALLBACK_V1,\n",
    "        project_name=PROJECT,\n",
    "        timeout_in_seconds=2,\n",
    "    )\n",
    "    print(f\"  is_fallback : {cfg.is_fallback}\")\n",
    "    print(f\"  temperature : {cfg.temperature}  (from local fallback)\")\n",
    "    print(f\"  model       : {cfg.model}\")\n",
    "    return cfg\n",
    "\n",
    "\n",
    "print(\"=== Call against unreachable backend — returns fallback ===\")\n",
    "offline_cfg = run_agent_offline(\"Hello offline!\")\n",
    "assert offline_cfg.is_fallback is True, (\n",
    "    \"should be marked as fallback when backend is unreachable\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "938c804e27f84196a10c8828c723f798",
   "metadata": {},
   "source": [
    "---\n",
    "## 3. `create_config` — write a new version unconditionally\n",
    "\n",
    "`create_config` does not require a `@opik.track` context and always creates a new\n",
    "version. It returns the version name (a string) that you can use later with\n",
    "`set_config_env` or to fetch by explicit version."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "504fb2a444614c0babb325280ed9130a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created version: 'v2'\n"
     ]
    }
   ],
   "source": [
    "v2 = AgentConfig(\n",
    "    temperature=0.8,\n",
    "    model=\"gpt-4o\",\n",
    "    system_prompt=\"You are an expert assistant. Think step by step.\",\n",
    ")\n",
    "\n",
    "v2_name = client.create_config(\n",
    "    v2,\n",
    "    project_name=PROJECT,\n",
    "    description=\"Upgraded to gpt-4o with chain-of-thought prompt\",\n",
    ")\n",
    "\n",
    "print(f\"Created version: {v2_name!r}\")\n",
    "assert isinstance(v2_name, str) and v2_name != \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59bbdb311c014d738909a11f9e486628",
   "metadata": {},
   "source": [
    "---\n",
    "## 4. `set_config_env` — tag a version with an environment\n",
    "\n",
    "Right now `\"prod\"` still points to the v1 values (auto-tagged by the backend on first\n",
    "write). We promote v2 to `\"prod\"` with `set_config_env`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b43b363d81ae4b689946ece5c682cd59",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Version 'v2' is now tagged as 'prod'\n",
      "Version 'v2' is now also tagged as 'staging'\n"
     ]
    }
   ],
   "source": [
    "client.set_config_env(\n",
    "    project_name=PROJECT,\n",
    "    version=v2_name,\n",
    "    env=\"prod\",\n",
    ")\n",
    "print(f\"Version {v2_name!r} is now tagged as 'prod'\")\n",
    "\n",
    "# Also tag the same version as 'staging' to show multi-env support.\n",
    "client.set_config_env(\n",
    "    project_name=PROJECT,\n",
    "    version=v2_name,\n",
    "    env=\"staging\",\n",
    ")\n",
    "print(f\"Version {v2_name!r} is now also tagged as 'staging'\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a65eabff63a45729fe45fb5ade58bdc",
   "metadata": {},
   "source": [
    "---\n",
    "## 5. Fetch by `env` — confirm prod now returns v2 values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "c3933fab20d04ec698c2621248eb3be0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== Fetch env='prod' (should return v2 after set_config_env) ===\n",
      "  temperature : 0.8\n",
      "  model       : gpt-4o\n",
      "  system_prompt: You are an expert assistant. Think step by step.\n"
     ]
    }
   ],
   "source": [
    "# Clear cache so we hit the backend, not a locally cached copy.\n",
    "# get_global_registry().clear()\n",
    "\n",
    "\n",
    "@opik.track(project_name=PROJECT)\n",
    "def fetch_prod():\n",
    "    return client.get_or_create_config(\n",
    "        fallback=FALLBACK_V1,\n",
    "        project_name=PROJECT,\n",
    "        env=\"prod\",\n",
    "    )\n",
    "\n",
    "\n",
    "print(\"=== Fetch env='prod' (should return v2 after set_config_env) ===\")\n",
    "prod_cfg = fetch_prod()\n",
    "print(f\"  temperature : {prod_cfg.temperature}\")\n",
    "print(f\"  model       : {prod_cfg.model}\")\n",
    "print(f\"  system_prompt: {prod_cfg.system_prompt}\")\n",
    "\n",
    "assert prod_cfg.temperature == 0.8\n",
    "assert prod_cfg.model == \"gpt-4o\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dd4641cc4064e0191573fe9c69df29b",
   "metadata": {},
   "source": [
    "---\n",
    "## 6. Fetch by explicit `version` name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "8309879909854d7188b41380fd92a7c3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== Fetch by explicit version 'v2' ===\n",
      "  temperature : 0.8\n",
      "  model       : gpt-4o\n"
     ]
    }
   ],
   "source": [
    "@opik.track(project_name=PROJECT)\n",
    "def fetch_by_version(version_name: str):\n",
    "    return client.get_or_create_config(\n",
    "        fallback=FALLBACK_V1,\n",
    "        project_name=PROJECT,\n",
    "        version=version_name,\n",
    "    )\n",
    "\n",
    "\n",
    "print(f\"=== Fetch by explicit version {v2_name!r} ===\")\n",
    "by_name_cfg = fetch_by_version(v2_name)\n",
    "print(f\"  temperature : {by_name_cfg.temperature}\")\n",
    "print(f\"  model       : {by_name_cfg.model}\")\n",
    "\n",
    "assert by_name_cfg.temperature == 0.8\n",
    "assert by_name_cfg.model == \"gpt-4o\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ed186c9a28b402fb0bc4494df01f08d",
   "metadata": {},
   "source": [
    "---\n",
    "## 7. Fetch without a fallback (generic `Config` return type)\n",
    "\n",
    "Omitting `fallback` returns a base `opik.Config` instance. Typed field access still\n",
    "works through attribute lookup, but you lose static type-checking of the subclass."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "cb1e1581032b452c9409d6c6813c49d1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== Fetch without fallback ===\n",
      "  type        : Config\n",
      "  is_fallback : False\n",
      "  temperature : 0.8\n",
      "  model       : gpt-4o\n"
     ]
    }
   ],
   "source": [
    "get_global_registry().clear()\n",
    "\n",
    "\n",
    "@opik.track(project_name=PROJECT)\n",
    "def fetch_no_fallback():\n",
    "    # No fallback — returns opik.Config, raises ConfigNotFound if project is empty.\n",
    "    return client.get_or_create_config(project_name=PROJECT)\n",
    "\n",
    "\n",
    "print(\"=== Fetch without fallback ===\")\n",
    "no_fallback_cfg = fetch_no_fallback()\n",
    "print(f\"  type        : {type(no_fallback_cfg).__name__}\")\n",
    "print(f\"  is_fallback : {no_fallback_cfg.is_fallback}\")\n",
    "print(f\"  temperature : {no_fallback_cfg.temperature}\")\n",
    "print(f\"  model       : {no_fallback_cfg.model}\")\n",
    "\n",
    "assert type(no_fallback_cfg) is opik.Config\n",
    "assert no_fallback_cfg.is_fallback is False\n",
    "# Values come from the prod version (v2).\n",
    "assert no_fallback_cfg.temperature == by_name_cfg.temperature\n",
    "assert no_fallback_cfg.model == by_name_cfg.model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "379cbbc1e968416e875cc15c1202d7eb",
   "metadata": {},
   "source": [
    "---\n",
    "## Cleanup\n",
    "\n",
    "Delete the demo project so it doesn't clutter the workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "277c27b1587741f2af2001be3712ef0d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from opik.rest_api import core as rest_api_core\n",
    "\n",
    "try:\n",
    "    project_id = client.rest_client.projects.retrieve_project(name=PROJECT).id\n",
    "    client.rest_client.projects.delete_project_by_id(project_id)\n",
    "    print(f\"Deleted project {PROJECT!r}\")\n",
    "except rest_api_core.ApiError as e:\n",
    "    print(f\"Could not delete project: {e}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.19"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}