wehub-resource-sync
261 lines
9.2 KiB
Plaintext
261 lines
9.2 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a1b2c3d4",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Lesson 10 - AI Agents in Production\n",
|
|
"\n",
|
|
"In this lesson you will learn **production patterns** for AI agents using the Microsoft Agent Framework (Python). We cover:\n",
|
|
"\n",
|
|
"- **Observability** — adding timing and logging to agent interactions\n",
|
|
"- **Evaluation** — using an evaluator agent to score response quality\n",
|
|
"- **Cost management** — strategies for token optimization and model selection\n",
|
|
"\n",
|
|
"The scenario is a **travel agent** that helps users plan trips, with monitoring and evaluation layered on top."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b2c3d4e5",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Setup"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "c3d4e5f6",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install agent-framework azure-ai-projects azure-identity python-dotenv -U -q"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "d4e5f6a7",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import logging\n",
|
|
"logging.getLogger(\"agent_framework.foundry\").setLevel(logging.ERROR)\n",
|
|
"\n",
|
|
"import os\n",
|
|
"import asyncio\n",
|
|
"import time\n",
|
|
"import dotenv\n",
|
|
"from typing import Annotated\n",
|
|
"\n",
|
|
"from agent_framework import tool\n",
|
|
"from agent_framework.foundry import FoundryChatClient\n",
|
|
"from azure.identity import DefaultAzureCredential\n",
|
|
"\n",
|
|
"dotenv.load_dotenv()\n",
|
|
"\n",
|
|
"endpoint = os.getenv(\"AZURE_AI_PROJECT_ENDPOINT\")\n",
|
|
"deployment_name = os.getenv(\"AZURE_AI_MODEL_DEPLOYMENT_NAME\")\n",
|
|
"\n",
|
|
"missing = [k for k, v in {\n",
|
|
" \"AZURE_AI_PROJECT_ENDPOINT\": endpoint,\n",
|
|
" \"AZURE_AI_MODEL_DEPLOYMENT_NAME\": deployment_name\n",
|
|
"}.items() if not v]\n",
|
|
"\n",
|
|
"if missing:\n",
|
|
" raise ValueError(\n",
|
|
" f\"Missing required environment variables: {', '.join(missing)}. \"\n",
|
|
" \"Please set them as environment variables (e.g., in your .env file or shell environment).\"\n",
|
|
" )"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "e5f6a7b8",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Create the Microsoft Foundry client\n",
|
|
"client = FoundryChatClient(\n",
|
|
" project_endpoint=endpoint,\n",
|
|
" model=deployment_name,\n",
|
|
" credential=DefaultAzureCredential()\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f6a7b8c9",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Production Considerations\n",
|
|
"\n",
|
|
"Moving AI agents from prototypes to production requires careful attention to three pillars:\n",
|
|
"\n",
|
|
"1. **Observability** — You need visibility into what the agent is doing, how long it takes, and which tools it calls. Without tracing and logging, debugging production issues is nearly impossible.\n",
|
|
"\n",
|
|
"2. **Evaluation** — Automated quality checks ensure the agent's responses remain accurate, complete, and helpful over time. An evaluator agent can score responses against defined criteria.\n",
|
|
"\n",
|
|
"3. **Cost Management** — Token usage directly impacts cost. Strategies like prompt optimization, model selection, and caching help keep expenses under control without sacrificing quality."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a7b8c9d0",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Building an Observable Agent\n",
|
|
"\n",
|
|
"We define travel tools and wrap the agent call with timing so we can monitor latency. In production you would integrate with OpenTelemetry or a similar tracing backend."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "b8c9d0e1",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"@tool(approval_mode=\"never_require\")\n",
|
|
"def get_flight_info(destination: Annotated[str, \"The destination city\"]) -> str:\n",
|
|
" \"\"\"Get flight information for a destination.\"\"\"\n",
|
|
" flights = {\n",
|
|
" \"Paris\": \"BA 304, 08:30-11:45, $350\",\n",
|
|
" \"Tokyo\": \"JL 044, 11:00-07:00+1, $890\",\n",
|
|
" \"Barcelona\": \"VY 7821, 07:15-10:30, $280\",\n",
|
|
" }\n",
|
|
" return flights.get(destination, f\"No flights found to {destination}\")\n",
|
|
"\n",
|
|
"\n",
|
|
"@tool(approval_mode=\"never_require\")\n",
|
|
"def get_activity_suggestions(destination: Annotated[str, \"The destination city\"]) -> str:\n",
|
|
" \"\"\"Get activity suggestions for a destination.\"\"\"\n",
|
|
" activities = {\n",
|
|
" \"Paris\": \"Louvre Museum, Eiffel Tower, Seine River Cruise, Montmartre walking tour\",\n",
|
|
" \"Tokyo\": \"Senso-ji Temple, Tsukiji Market tour, Shibuya Crossing, teamLab Borderless\",\n",
|
|
" \"Barcelona\": \"Sagrada Familia, Park Güell, La Boqueria Market, Gothic Quarter walk\",\n",
|
|
" }\n",
|
|
" return activities.get(destination, f\"No activities found for {destination}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "c9d0e1f2",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"agent = client.as_agent(\n",
|
|
" tools=[get_flight_info, get_activity_suggestions],\n",
|
|
" name=\"TravelAgent\",\n",
|
|
" instructions=\"You are a helpful travel agent. Use the available tools to help users plan their trips. Provide comprehensive, actionable travel advice.\",\n",
|
|
")\n",
|
|
"\n",
|
|
"# Simple observability: track timing\n",
|
|
"start_time = time.time()\n",
|
|
"response = await agent.run(\n",
|
|
" \"I want to plan a day trip in Paris. What flights and activities do you recommend?\",\n",
|
|
" )\n",
|
|
"elapsed = time.time() - start_time\n",
|
|
"print(f\"Response ({elapsed:.2f}s):\\n{response}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d0e1f2a3",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Evaluation Patterns\n",
|
|
"\n",
|
|
"A common production pattern is to use a second agent as an **evaluator**. The evaluator scores the primary agent's response against predefined criteria such as completeness, accuracy, and helpfulness.\n",
|
|
"\n",
|
|
"This enables:\n",
|
|
"- Automated quality gates before responses reach users\n",
|
|
"- Regression detection when prompts or models change\n",
|
|
"- Continuous monitoring of agent performance over time"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "e1f2a3b4",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"evaluator = client.as_agent(\n",
|
|
" name=\"ResponseEvaluator\",\n",
|
|
" instructions=\"\"\"You evaluate travel agent responses on these criteria:\n",
|
|
"1. Completeness (1-5): Did it cover flights AND activities?\n",
|
|
"2. Accuracy (1-5): Is the information consistent?\n",
|
|
"3. Helpfulness (1-5): Would a traveler find this actionable?\n",
|
|
"4. Overall Score (1-5)\n",
|
|
"Provide scores and a brief explanation for each.\"\"\",\n",
|
|
")\n",
|
|
"\n",
|
|
"evaluation = await evaluator.run(f\"Evaluate this travel agent response:\\n\\n{response}\")\n",
|
|
"print(f\"Evaluation:\\n{evaluation}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f2a3b4c5",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Cost Management Strategies\n",
|
|
"\n",
|
|
"Controlling costs is critical for production AI agents. Here are key strategies:\n",
|
|
"\n",
|
|
"| Strategy | Description |\n",
|
|
"|---|---|\n",
|
|
"| **Prompt optimization** | Keep system instructions concise. Remove redundant context to reduce input tokens. |\n",
|
|
"| **Model selection** | Use smaller, cheaper models (e.g. GPT-4o-mini) for simple tasks like classification or extraction, and reserve larger models for complex reasoning. |\n",
|
|
"| **Caching** | Cache tool results and frequent queries to avoid redundant API calls. |\n",
|
|
"| **Token budgets** | Set `max_tokens` limits to prevent unexpectedly long responses. |\n",
|
|
"| **Batching** | Group multiple user queries into a single API call where possible. |\n",
|
|
"\n",
|
|
"In practice, a tiered approach works well: route straightforward requests to a fast, inexpensive model and escalate only complex queries to a more capable one."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a3b4c5d6",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Summary\n",
|
|
"\n",
|
|
"In this lesson you learned how to:\n",
|
|
"\n",
|
|
"1. **Add observability** to agent interactions with timing and logging, laying the groundwork for tracing and monitoring.\n",
|
|
"2. **Evaluate agent responses** automatically using an evaluator agent that scores completeness, accuracy, and helpfulness.\n",
|
|
"3. **Manage costs** through prompt optimization, model selection, caching, and token budgets.\n",
|
|
"\n",
|
|
"These production patterns help ensure your AI agents are reliable, measurable, and cost-effective at scale."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.12.13"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|