{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a1b2c3d4",
   "metadata": {},
   "source": [
    "# Lesson 10 - AI Agents in Production\n",
    "\n",
    "In this lesson you will learn **production patterns** for AI agents using the Microsoft Agent Framework (Python). We cover:\n",
    "\n",
    "- **Observability** — adding timing and logging to agent interactions\n",
    "- **Evaluation** — using an evaluator agent to score response quality\n",
    "- **Cost management** — strategies for token optimization and model selection\n",
    "\n",
    "The scenario is a **travel agent** that helps users plan trips, with monitoring and evaluation layered on top."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2c3d4e5",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3d4e5f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install agent-framework azure-ai-projects azure-identity python-dotenv -U -q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4e5f6a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "logging.getLogger(\"agent_framework.foundry\").setLevel(logging.ERROR)\n",
    "\n",
    "import os\n",
    "import asyncio\n",
    "import time\n",
    "import dotenv\n",
    "from typing import Annotated\n",
    "\n",
    "from agent_framework import tool\n",
    "from agent_framework.foundry import FoundryChatClient\n",
    "from azure.identity import DefaultAzureCredential\n",
    "\n",
    "dotenv.load_dotenv()\n",
    "\n",
    "endpoint = os.getenv(\"AZURE_AI_PROJECT_ENDPOINT\")\n",
    "deployment_name = os.getenv(\"AZURE_AI_MODEL_DEPLOYMENT_NAME\")\n",
    "\n",
    "missing = [k for k, v in {\n",
    "    \"AZURE_AI_PROJECT_ENDPOINT\": endpoint,\n",
    "    \"AZURE_AI_MODEL_DEPLOYMENT_NAME\": deployment_name\n",
    "}.items() if not v]\n",
    "\n",
    "if missing:\n",
    "    raise ValueError(\n",
    "        f\"Missing required environment variables: {', '.join(missing)}. \"\n",
    "        \"Please set them as environment variables (e.g., in your .env file or shell environment).\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5f6a7b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the Microsoft Foundry client\n",
    "client = FoundryChatClient(\n",
    "    project_endpoint=endpoint,\n",
    "    model=deployment_name,\n",
    "    credential=DefaultAzureCredential()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6a7b8c9",
   "metadata": {},
   "source": [
    "## Production Considerations\n",
    "\n",
    "Moving AI agents from prototypes to production requires careful attention to three pillars:\n",
    "\n",
    "1. **Observability** — You need visibility into what the agent is doing, how long it takes, and which tools it calls. Without tracing and logging, debugging production issues is nearly impossible.\n",
    "\n",
    "2. **Evaluation** — Automated quality checks ensure the agent's responses remain accurate, complete, and helpful over time. An evaluator agent can score responses against defined criteria.\n",
    "\n",
    "3. **Cost Management** — Token usage directly impacts cost. Strategies like prompt optimization, model selection, and caching help keep expenses under control without sacrificing quality."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7b8c9d0",
   "metadata": {},
   "source": [
    "## Building an Observable Agent\n",
    "\n",
    "We define travel tools and wrap the agent call with timing so we can monitor latency. In production you would integrate with OpenTelemetry or a similar tracing backend."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8c9d0e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "@tool(approval_mode=\"never_require\")\n",
    "def get_flight_info(destination: Annotated[str, \"The destination city\"]) -> str:\n",
    "    \"\"\"Get flight information for a destination.\"\"\"\n",
    "    flights = {\n",
    "        \"Paris\": \"BA 304, 08:30-11:45, $350\",\n",
    "        \"Tokyo\": \"JL 044, 11:00-07:00+1, $890\",\n",
    "        \"Barcelona\": \"VY 7821, 07:15-10:30, $280\",\n",
    "    }\n",
    "    return flights.get(destination, f\"No flights found to {destination}\")\n",
    "\n",
    "\n",
    "@tool(approval_mode=\"never_require\")\n",
    "def get_activity_suggestions(destination: Annotated[str, \"The destination city\"]) -> str:\n",
    "    \"\"\"Get activity suggestions for a destination.\"\"\"\n",
    "    activities = {\n",
    "        \"Paris\": \"Louvre Museum, Eiffel Tower, Seine River Cruise, Montmartre walking tour\",\n",
    "        \"Tokyo\": \"Senso-ji Temple, Tsukiji Market tour, Shibuya Crossing, teamLab Borderless\",\n",
    "        \"Barcelona\": \"Sagrada Familia, Park Güell, La Boqueria Market, Gothic Quarter walk\",\n",
    "    }\n",
    "    return activities.get(destination, f\"No activities found for {destination}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9d0e1f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "agent = client.as_agent(\n",
    "    tools=[get_flight_info, get_activity_suggestions],\n",
    "    name=\"TravelAgent\",\n",
    "    instructions=\"You are a helpful travel agent. Use the available tools to help users plan their trips. Provide comprehensive, actionable travel advice.\",\n",
    ")\n",
    "\n",
    "# Simple observability: track timing\n",
    "start_time = time.time()\n",
    "response = await agent.run(\n",
    "    \"I want to plan a day trip in Paris. What flights and activities do you recommend?\",\n",
    "    )\n",
    "elapsed = time.time() - start_time\n",
    "print(f\"Response ({elapsed:.2f}s):\\n{response}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0e1f2a3",
   "metadata": {},
   "source": [
    "## Evaluation Patterns\n",
    "\n",
    "A common production pattern is to use a second agent as an **evaluator**. The evaluator scores the primary agent's response against predefined criteria such as completeness, accuracy, and helpfulness.\n",
    "\n",
    "This enables:\n",
    "- Automated quality gates before responses reach users\n",
    "- Regression detection when prompts or models change\n",
    "- Continuous monitoring of agent performance over time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1f2a3b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluator = client.as_agent(\n",
    "    name=\"ResponseEvaluator\",\n",
    "    instructions=\"\"\"You evaluate travel agent responses on these criteria:\n",
    "1. Completeness (1-5): Did it cover flights AND activities?\n",
    "2. Accuracy (1-5): Is the information consistent?\n",
    "3. Helpfulness (1-5): Would a traveler find this actionable?\n",
    "4. Overall Score (1-5)\n",
    "Provide scores and a brief explanation for each.\"\"\",\n",
    ")\n",
    "\n",
    "evaluation = await evaluator.run(f\"Evaluate this travel agent response:\\n\\n{response}\")\n",
    "print(f\"Evaluation:\\n{evaluation}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2a3b4c5",
   "metadata": {},
   "source": [
    "## Cost Management Strategies\n",
    "\n",
    "Controlling costs is critical for production AI agents. Here are key strategies:\n",
    "\n",
    "| Strategy | Description |\n",
    "|---|---|\n",
    "| **Prompt optimization** | Keep system instructions concise. Remove redundant context to reduce input tokens. |\n",
    "| **Model selection** | Use smaller, cheaper models (e.g. GPT-4o-mini) for simple tasks like classification or extraction, and reserve larger models for complex reasoning. |\n",
    "| **Caching** | Cache tool results and frequent queries to avoid redundant API calls. |\n",
    "| **Token budgets** | Set `max_tokens` limits to prevent unexpectedly long responses. |\n",
    "| **Batching** | Group multiple user queries into a single API call where possible. |\n",
    "\n",
    "In practice, a tiered approach works well: route straightforward requests to a fast, inexpensive model and escalate only complex queries to a more capable one."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3b4c5d6",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "In this lesson you learned how to:\n",
    "\n",
    "1. **Add observability** to agent interactions with timing and logging, laying the groundwork for tracing and monitoring.\n",
    "2. **Evaluate agent responses** automatically using an evaluator agent that scores completeness, accuracy, and helpfulness.\n",
    "3. **Manage costs** through prompt optimization, model selection, caching, and token budgets.\n",
    "\n",
    "These production patterns help ensure your AI agents are reliable, measurable, and cost-effective at scale."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}