Files
2026-07-13 12:59:43 +08:00

642 lines
23 KiB
Plaintext

{
"cells": [
{
"cell_type": "markdown",
"id": "44b5899e",
"metadata": {},
"source": [
"# 🔍 Enterprise RAG with Microsoft Foundry (.NET)\n",
"\n",
"## 📋 Learning Objectives\n",
"\n",
"This notebook demonstrates how to build enterprise-grade Retrieval-Augmented Generation (RAG) systems using the Microsoft Agent Framework in .NET with Microsoft Foundry. You'll learn to create production-ready agents that can search through documents and provide accurate, context-aware responses with enterprise security and scalability.\n",
"\n",
"**Enterprise RAG Capabilities You'll Build:**\n",
"- 📚 **Document Intelligence**: Advanced document processing with Azure AI services\n",
"- 🔍 **Semantic Search**: High-performance vector search with enterprise features\n",
"- 🛡️ **Security Integration**: Role-based access and data protection patterns\n",
"- 🏢 **Scalable Architecture**: Production-ready RAG systems with monitoring\n",
"\n",
"## 🎯 Enterprise RAG Architecture\n",
"\n",
"### Core Enterprise Components\n",
"- **Microsoft Foundry**: Managed enterprise AI platform with security and compliance\n",
"- **Persistent Agents**: Stateful agents with conversation history and context management\n",
"- **Vector Store Management**: Enterprise-grade document indexing and retrieval\n",
"- **Identity Integration**: Azure AD authentication and role-based access control\n",
"\n",
"### .NET Enterprise Benefits\n",
"- **Type Safety**: Compile-time validation for RAG operations and data structures\n",
"- **Async Performance**: Non-blocking document processing and search operations\n",
"- **Memory Management**: Efficient resource utilization for large document collections\n",
"- **Integration Patterns**: Native Azure service integration with dependency injection\n",
"\n",
"## 🏗️ Technical Architecture\n",
"\n",
"### Enterprise RAG Pipeline\n",
"```csharp\n",
"Document Upload → Security Validation → Vector Processing → Index Creation\n",
" ↓ ↓ ↓\n",
"User Query → Authentication → Semantic Search → Context Ranking → AI Response\n",
"```\n",
"\n",
"### Core .NET Components\n",
"- **Azure.AI.Agents.Persistent**: Enterprise agent management with state persistence\n",
"- **Azure.Identity**: Integrated authentication for secure Azure service access\n",
"- **Microsoft.Agents.AI.AzureAI**: Azure-optimized agent framework implementation\n",
"- **System.Linq.Async**: High-performance asynchronous LINQ operations\n",
"\n",
"## 🔧 Enterprise Features & Benefits\n",
"\n",
"### Security & Compliance\n",
"- **Azure AD Integration**: Enterprise identity management and authentication\n",
"- **Role-Based Access**: Fine-grained permissions for document access and operations\n",
"- **Data Protection**: Encryption at rest and in transit for sensitive documents\n",
"- **Audit Logging**: Comprehensive activity tracking for compliance requirements\n",
"\n",
"### Performance & Scalability\n",
"- **Connection Pooling**: Efficient Azure service connection management\n",
"- **Async Processing**: Non-blocking operations for high-throughput scenarios\n",
"- **Caching Strategies**: Intelligent caching for frequently accessed documents\n",
"- **Load Balancing**: Distributed processing for large-scale deployments\n",
"\n",
"### Management & Monitoring\n",
"- **Health Checks**: Built-in monitoring for RAG system components\n",
"- **Performance Metrics**: Detailed analytics on search quality and response times\n",
"- **Error Handling**: Comprehensive exception management with retry policies\n",
"- **Configuration Management**: Environment-specific settings with validation\n",
"\n",
"## ⚙️ Prerequisites & Setup\n",
"\n",
"**Development Environment:**\n",
"- .NET 9.0 SDK or higher\n",
"- Visual Studio 2022 or VS Code with C# extension\n",
"- Azure subscription with AI Foundry access\n",
"\n",
"**Required NuGet Packages:**\n",
"```xml\n",
"<PackageReference Include=\"Microsoft.Extensions.AI\" Version=\"9.9.0\" />\n",
"<PackageReference Include=\"Azure.AI.Agents.Persistent\" Version=\"1.2.0-beta.5\" />\n",
"<PackageReference Include=\"Azure.Identity\" Version=\"1.15.0\" />\n",
"<PackageReference Include=\"System.Linq.Async\" Version=\"6.0.3\" />\n",
"<PackageReference Include=\"DotNetEnv\" Version=\"3.1.1\" />\n",
"```\n",
"\n",
"**Azure Authentication Setup:**\n",
"```bash\n",
"# Install Azure CLI and authenticate\n",
"az login\n",
"az account set --subscription \"your-subscription-id\"\n",
"```\n",
"\n",
"**Environment Configuration (.env file):**\n",
"```env\n",
"# Microsoft Foundry configuration (automatically handled via Azure CLI)\n",
"# Ensure you're authenticated to the correct Azure subscription\n",
"```\n",
"\n",
"## 📊 Enterprise RAG Patterns\n",
"\n",
"### Document Management Patterns\n",
"- **Bulk Upload**: Efficient processing of large document collections\n",
"- **Incremental Updates**: Real-time document addition and modification\n",
"- **Version Control**: Document versioning and change tracking\n",
"- **Metadata Management**: Rich document attributes and taxonomy\n",
"\n",
"### Search & Retrieval Patterns\n",
"- **Hybrid Search**: Combining semantic and keyword search for optimal results\n",
"- **Faceted Search**: Multi-dimensional filtering and categorization\n",
"- **Relevance Tuning**: Custom scoring algorithms for domain-specific needs\n",
"- **Result Ranking**: Advanced ranking with business logic integration\n",
"\n",
"### Security Patterns\n",
"- **Document-Level Security**: Fine-grained access control per document\n",
"- **Data Classification**: Automatic sensitivity labeling and protection\n",
"- **Audit Trails**: Comprehensive logging of all RAG operations\n",
"- **Privacy Protection**: PII detection and redaction capabilities\n",
"\n",
"## 🔒 Enterprise Security Features\n",
"\n",
"### Authentication & Authorization\n",
"```csharp\n",
"// Azure AD integrated authentication\n",
"var credential = new AzureCliCredential();\n",
"var agentsClient = new PersistentAgentsClient(endpoint, credential);\n",
"\n",
"// Role-based access validation\n",
"if (!await ValidateUserPermissions(user, documentId))\n",
"{\n",
" throw new UnauthorizedAccessException(\"Insufficient permissions\");\n",
"}\n",
"```\n",
"\n",
"### Data Protection\n",
"- **Encryption**: End-to-end encryption for documents and search indices\n",
"- **Access Controls**: Integration with Azure AD for user and group permissions\n",
"- **Data Residency**: Geographic data location controls for compliance\n",
"- **Backup & Recovery**: Automated backup and disaster recovery capabilities\n",
"\n",
"## 📈 Performance Optimization\n",
"\n",
"### Async Processing Patterns\n",
"```csharp\n",
"// Efficient async document processing\n",
"await foreach (var document in documentStream.AsAsyncEnumerable())\n",
"{\n",
" await ProcessDocumentAsync(document, cancellationToken);\n",
"}\n",
"```\n",
"\n",
"### Memory Management\n",
"- **Streaming Processing**: Handle large documents without memory issues\n",
"- **Resource Pooling**: Efficient reuse of expensive resources\n",
"- **Garbage Collection**: Optimized memory allocation patterns\n",
"- **Connection Management**: Proper Azure service connection lifecycle\n",
"\n",
"### Caching Strategies\n",
"- **Query Caching**: Cache frequently executed searches\n",
"- **Document Caching**: In-memory caching for hot documents\n",
"- **Index Caching**: Optimized vector index caching\n",
"- **Result Caching**: Intelligent caching of generated responses\n",
"\n",
"## 📊 Enterprise Use Cases\n",
"\n",
"### Knowledge Management\n",
"- **Corporate Wiki**: Intelligent search across company knowledge bases\n",
"- **Policy & Procedures**: Automated compliance and procedure guidance\n",
"- **Training Materials**: Intelligent learning and development assistance\n",
"- **Research Databases**: Academic and research paper analysis systems\n",
"\n",
"### Customer Support\n",
"- **Support Knowledge Base**: Automated customer service responses\n",
"- **Product Documentation**: Intelligent product information retrieval\n",
"- **Troubleshooting Guides**: Contextual problem-solving assistance\n",
"- **FAQ Systems**: Dynamic FAQ generation from document collections\n",
"\n",
"### Regulatory Compliance\n",
"- **Legal Document Analysis**: Contract and legal document intelligence\n",
"- **Compliance Monitoring**: Automated regulatory compliance checking\n",
"- **Risk Assessment**: Document-based risk analysis and reporting\n",
"- **Audit Support**: Intelligent document discovery for audits\n",
"\n",
"## 🚀 Production Deployment\n",
"\n",
"### Monitoring & Observability\n",
"- **Application Insights**: Detailed telemetry and performance monitoring\n",
"- **Custom Metrics**: Business-specific KPI tracking and alerting\n",
"- **Distributed Tracing**: End-to-end request tracking across services\n",
"- **Health Dashboards**: Real-time system health and performance visualization\n",
"\n",
"### Scalability & Reliability\n",
"- **Auto-Scaling**: Automatic scaling based on load and performance metrics\n",
"- **High Availability**: Multi-region deployment with failover capabilities\n",
"- **Load Testing**: Performance validation under enterprise load conditions\n",
"- **Disaster Recovery**: Automated backup and recovery procedures\n",
"\n",
"Ready to build enterprise-grade RAG systems that can handle sensitive documents at scale? Let's architect intelligent knowledge systems for the enterprise! 🏢📖✨"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"language_info": {
"name": "polyglot-notebook"
},
"polyglot_notebook": {
"kernelName": "csharp"
}
},
"outputs": [
{
"data": {
"text/html": [
"<div><div></div><div></div><div><strong>Installed Packages</strong><ul><li><span>Microsoft.Extensions.AI, 9.9.1</span></li></ul></div></div>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"#r \"nuget: Microsoft.Extensions.AI, 9.9.1\""
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "4ec1f0d1",
"metadata": {
"language_info": {
"name": "polyglot-notebook"
},
"polyglot_notebook": {
"kernelName": "csharp"
}
},
"outputs": [
{
"data": {
"text/html": [
"<div><div></div><div></div><div><strong>Installed Packages</strong><ul><li><span>Azure.AI.Agents.Persistent, 1.2.0-beta.5</span></li><li><span>Azure.Identity, 1.15.0</span></li><li><span>System.Linq.Async, 6.0.3</span></li></ul></div></div>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"#r \"nuget: Azure.AI.Agents.Persistent, 1.2.0-beta.5\"\n",
"#r \"nuget: Azure.Identity, 1.15.0\"\n",
"#r \"nuget: System.Linq.Async, 6.0.3\""
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "2363ae07",
"metadata": {
"language_info": {
"name": "polyglot-notebook"
},
"polyglot_notebook": {
"kernelName": "csharp"
}
},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "d10cec9d",
"metadata": {
"language_info": {
"name": "polyglot-notebook"
},
"polyglot_notebook": {
"kernelName": "csharp"
}
},
"outputs": [
{
"data": {
"text/html": [
"<div><div></div><div></div><div><strong>Installed Packages</strong><ul><li><span>Microsoft.Agents.AI.AzureAI, 1.0.0-preview.251001.2</span></li></ul></div></div>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"#r \"nuget: Microsoft.Agents.AI.AzureAI, 1.0.0-preview.251001.3\""
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "78199d1c",
"metadata": {
"language_info": {
"name": "polyglot-notebook"
},
"polyglot_notebook": {
"kernelName": "csharp"
}
},
"outputs": [
{
"data": {
"text/html": [
"<div><div></div><div></div><div><strong>Installed Packages</strong><ul><li><span>microsoft.agents.ai, 1.0.0-preview.251001.2</span></li></ul></div></div>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"#r \"nuget: Microsoft.Agents.AI, 1.0.0-preview.251001.3\""
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "7de4684a",
"metadata": {
"language_info": {
"name": "polyglot-notebook"
},
"polyglot_notebook": {
"kernelName": "csharp"
}
},
"outputs": [
{
"data": {
"text/html": [
"<div><div></div><div></div><div><strong>Installed Packages</strong><ul><li><span>DotNetEnv, 3.1.1</span></li></ul></div></div>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"#r \"nuget: DotNetEnv, 3.1.1\""
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "251efd31",
"metadata": {
"language_info": {
"name": "polyglot-notebook"
},
"polyglot_notebook": {
"kernelName": "csharp"
}
},
"outputs": [],
"source": [
"using System;\n",
"using System.Linq;\n",
"using Azure.AI.Agents.Persistent;\n",
"using Azure.Identity;\n",
"using Microsoft.Agents.AI;"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "a2e342f1",
"metadata": {
"language_info": {
"name": "polyglot-notebook"
},
"polyglot_notebook": {
"kernelName": "csharp"
}
},
"outputs": [],
"source": [
" using DotNetEnv;"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "a7a01653",
"metadata": {
"language_info": {
"name": "polyglot-notebook"
},
"polyglot_notebook": {
"kernelName": "csharp"
}
},
"outputs": [],
"source": [
"Env.Load(\"../../../.env\");"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "a42735d5",
"metadata": {
"language_info": {
"name": "polyglot-notebook"
},
"polyglot_notebook": {
"kernelName": "csharp"
}
},
"outputs": [],
"source": [
"var azure_foundry_endpoint = Environment.GetEnvironmentVariable(\"AZURE_AI_PROJECT_ENDPOINT\") ?? throw new InvalidOperationException(\"AZURE_AI_PROJECT_ENDPOINT is not set.\");\n",
"var azure_foundry_model_id = Environment.GetEnvironmentVariable(\"AZURE_AI_MODEL_DEPLOYMENT_NAME\") ?? \"gpt-4.1-mini\";"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "e29bdb58",
"metadata": {
"language_info": {
"name": "polyglot-notebook"
},
"polyglot_notebook": {
"kernelName": "csharp"
}
},
"outputs": [],
"source": [
"string pdfPath = \"./document.md\";"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "7351e12d",
"metadata": {
"language_info": {
"name": "polyglot-notebook"
},
"polyglot_notebook": {
"kernelName": "csharp"
}
},
"outputs": [],
"source": [
"using System.IO;\n",
"\n",
"async Task<Stream> OpenImageStreamAsync(string path)\n",
"{\n",
"\treturn await Task.Run(() => File.OpenRead(path));\n",
"}\n",
"\n",
"var pdfStream = await OpenImageStreamAsync(pdfPath);"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "0b6bf484",
"metadata": {
"language_info": {
"name": "polyglot-notebook"
},
"polyglot_notebook": {
"kernelName": "csharp"
}
},
"outputs": [],
"source": [
"var persistentAgentsClient = new PersistentAgentsClient(azure_foundry_endpoint, new AzureCliCredential());"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "81e0dddc",
"metadata": {
"language_info": {
"name": "polyglot-notebook"
},
"polyglot_notebook": {
"kernelName": "csharp"
}
},
"outputs": [],
"source": [
"PersistentAgentFileInfo fileInfo = await persistentAgentsClient.Files.UploadFileAsync(pdfStream, PersistentAgentFilePurpose.Agents, \"demo.md\");"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "f0c75d80",
"metadata": {
"language_info": {
"name": "polyglot-notebook"
},
"polyglot_notebook": {
"kernelName": "csharp"
}
},
"outputs": [],
"source": [
"PersistentAgentsVectorStore fileStore =\n",
" await persistentAgentsClient.VectorStores.CreateVectorStoreAsync(\n",
" [fileInfo.Id],\n",
" metadata: new Dictionary<string, string>() { { \"agentkey\", bool.TrueString } });"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "c77986c5",
"metadata": {
"language_info": {
"name": "polyglot-notebook"
},
"polyglot_notebook": {
"kernelName": "csharp"
}
},
"outputs": [],
"source": [
"PersistentAgent agentModel = await persistentAgentsClient.Administration.CreateAgentAsync(\n",
" azure_foundry_model_id,\n",
" name: \"DotNetRAGAgent\",\n",
" tools: [new FileSearchToolDefinition()],\n",
" instructions: \"\"\"\n",
" You are an AI assistant designed to answer user questions using only the information retrieved from the provided document(s).\n",
"\n",
" - If a user's question cannot be answered using the retrieved context, **you must clearly respond**: \n",
" \"I'm sorry, but the uploaded document does not contain the necessary information to answer that question.\"\n",
" - Do not answer from general knowledge or reasoning. Do not make assumptions or generate hypothetical explanations.\n",
" - Do not provide definitions, tutorials, or commentary that is not explicitly grounded in the content of the uploaded file(s).\n",
" - If a user asks a question like \"What is a Neural Network?\", and this is not discussed in the uploaded document, respond as instructed above.\n",
" - For questions that do have relevant content in the document (e.g., Contoso's travel insurance coverage), respond accurately, and cite the document explicitly.\n",
"\n",
" You must behave as if you have no external knowledge beyond what is retrieved from the uploaded document.\n",
" \"\"\",\n",
" toolResources: new()\n",
" {\n",
" FileSearch = new()\n",
" {\n",
" VectorStoreIds = { fileStore.Id },\n",
" }\n",
" },\n",
" metadata: new Dictionary<string, string>() { { \"agentkey\", bool.TrueString } });"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "282326cf",
"metadata": {
"language_info": {
"name": "polyglot-notebook"
},
"polyglot_notebook": {
"kernelName": "csharp"
}
},
"outputs": [],
"source": [
"AIAgent agent = await persistentAgentsClient.GetAIAgentAsync(agentModel.Id);"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "2067d313",
"metadata": {
"language_info": {
"name": "polyglot-notebook"
},
"polyglot_notebook": {
"kernelName": "csharp"
}
},
"outputs": [],
"source": [
"AgentThread thread = agent.GetNewThread();"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "454c4230",
"metadata": {
"language_info": {
"name": "polyglot-notebook"
},
"polyglot_notebook": {
"kernelName": "csharp"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Contoso's travel insurance coverage includes protection for medical emergencies, trip cancellations, and lost baggage. This ensures that travelers are supported in case of health-related issues during their trip, unforeseen cancellations, and the loss of their belongings while traveling【4:0†demo.md】.\r\n"
]
}
],
"source": [
"Console.WriteLine(await agent.RunAsync(\"Can you explain Contoso's travel insurance coverage?\", thread));"
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".NET (C#)",
"language": "C#",
"name": ".net-csharp"
},
"language_info": {
"name": "polyglot-notebook"
},
"polyglot_notebook": {
"kernelInfo": {
"defaultKernelName": "csharp",
"items": [
{
"aliases": [],
"name": "csharp"
}
]
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}