{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "44b5899e",
      "metadata": {},
      "source": [
        "# 🔍 Enterprise RAG with Microsoft Foundry (.NET)\n",
        "\n",
        "## 📋 Learning Objectives\n",
        "\n",
        "This notebook demonstrates how to build enterprise-grade Retrieval-Augmented Generation (RAG) systems using the Microsoft Agent Framework in .NET with Microsoft Foundry. You'll learn to create production-ready agents that can search through documents and provide accurate, context-aware responses with enterprise security and scalability.\n",
        "\n",
        "**Enterprise RAG Capabilities You'll Build:**\n",
        "- 📚 **Document Intelligence**: Advanced document processing with Azure AI services\n",
        "- 🔍 **Semantic Search**: High-performance vector search with enterprise features\n",
        "- 🛡️ **Security Integration**: Role-based access and data protection patterns\n",
        "- 🏢 **Scalable Architecture**: Production-ready RAG systems with monitoring\n",
        "\n",
        "## 🎯 Enterprise RAG Architecture\n",
        "\n",
        "### Core Enterprise Components\n",
        "- **Microsoft Foundry**: Managed enterprise AI platform with security and compliance\n",
        "- **Persistent Agents**: Stateful agents with conversation history and context management\n",
        "- **Vector Store Management**: Enterprise-grade document indexing and retrieval\n",
        "- **Identity Integration**: Azure AD authentication and role-based access control\n",
        "\n",
        "### .NET Enterprise Benefits\n",
        "- **Type Safety**: Compile-time validation for RAG operations and data structures\n",
        "- **Async Performance**: Non-blocking document processing and search operations\n",
        "- **Memory Management**: Efficient resource utilization for large document collections\n",
        "- **Integration Patterns**: Native Azure service integration with dependency injection\n",
        "\n",
        "## 🏗️ Technical Architecture\n",
        "\n",
        "### Enterprise RAG Pipeline\n",
        "```csharp\n",
        "Document Upload → Security Validation → Vector Processing → Index Creation\n",
        "                      ↓                    ↓                  ↓\n",
        "User Query → Authentication → Semantic Search → Context Ranking → AI Response\n",
        "```\n",
        "\n",
        "### Core .NET Components\n",
        "- **Azure.AI.Agents.Persistent**: Enterprise agent management with state persistence\n",
        "- **Azure.Identity**: Integrated authentication for secure Azure service access\n",
        "- **Microsoft.Agents.AI.AzureAI**: Azure-optimized agent framework implementation\n",
        "- **System.Linq.Async**: High-performance asynchronous LINQ operations\n",
        "\n",
        "## 🔧 Enterprise Features & Benefits\n",
        "\n",
        "### Security & Compliance\n",
        "- **Azure AD Integration**: Enterprise identity management and authentication\n",
        "- **Role-Based Access**: Fine-grained permissions for document access and operations\n",
        "- **Data Protection**: Encryption at rest and in transit for sensitive documents\n",
        "- **Audit Logging**: Comprehensive activity tracking for compliance requirements\n",
        "\n",
        "### Performance & Scalability\n",
        "- **Connection Pooling**: Efficient Azure service connection management\n",
        "- **Async Processing**: Non-blocking operations for high-throughput scenarios\n",
        "- **Caching Strategies**: Intelligent caching for frequently accessed documents\n",
        "- **Load Balancing**: Distributed processing for large-scale deployments\n",
        "\n",
        "### Management & Monitoring\n",
        "- **Health Checks**: Built-in monitoring for RAG system components\n",
        "- **Performance Metrics**: Detailed analytics on search quality and response times\n",
        "- **Error Handling**: Comprehensive exception management with retry policies\n",
        "- **Configuration Management**: Environment-specific settings with validation\n",
        "\n",
        "## ⚙️ Prerequisites & Setup\n",
        "\n",
        "**Development Environment:**\n",
        "- .NET 9.0 SDK or higher\n",
        "- Visual Studio 2022 or VS Code with C# extension\n",
        "- Azure subscription with AI Foundry access\n",
        "\n",
        "**Required NuGet Packages:**\n",
        "```xml\n",
        "<PackageReference Include=\"Microsoft.Extensions.AI\" Version=\"9.9.0\" />\n",
        "<PackageReference Include=\"Azure.AI.Agents.Persistent\" Version=\"1.2.0-beta.5\" />\n",
        "<PackageReference Include=\"Azure.Identity\" Version=\"1.15.0\" />\n",
        "<PackageReference Include=\"System.Linq.Async\" Version=\"6.0.3\" />\n",
        "<PackageReference Include=\"DotNetEnv\" Version=\"3.1.1\" />\n",
        "```\n",
        "\n",
        "**Azure Authentication Setup:**\n",
        "```bash\n",
        "# Install Azure CLI and authenticate\n",
        "az login\n",
        "az account set --subscription \"your-subscription-id\"\n",
        "```\n",
        "\n",
        "**Environment Configuration (.env file):**\n",
        "```env\n",
        "# Microsoft Foundry configuration (automatically handled via Azure CLI)\n",
        "# Ensure you're authenticated to the correct Azure subscription\n",
        "```\n",
        "\n",
        "## 📊 Enterprise RAG Patterns\n",
        "\n",
        "### Document Management Patterns\n",
        "- **Bulk Upload**: Efficient processing of large document collections\n",
        "- **Incremental Updates**: Real-time document addition and modification\n",
        "- **Version Control**: Document versioning and change tracking\n",
        "- **Metadata Management**: Rich document attributes and taxonomy\n",
        "\n",
        "### Search & Retrieval Patterns\n",
        "- **Hybrid Search**: Combining semantic and keyword search for optimal results\n",
        "- **Faceted Search**: Multi-dimensional filtering and categorization\n",
        "- **Relevance Tuning**: Custom scoring algorithms for domain-specific needs\n",
        "- **Result Ranking**: Advanced ranking with business logic integration\n",
        "\n",
        "### Security Patterns\n",
        "- **Document-Level Security**: Fine-grained access control per document\n",
        "- **Data Classification**: Automatic sensitivity labeling and protection\n",
        "- **Audit Trails**: Comprehensive logging of all RAG operations\n",
        "- **Privacy Protection**: PII detection and redaction capabilities\n",
        "\n",
        "## 🔒 Enterprise Security Features\n",
        "\n",
        "### Authentication & Authorization\n",
        "```csharp\n",
        "// Azure AD integrated authentication\n",
        "var credential = new AzureCliCredential();\n",
        "var agentsClient = new PersistentAgentsClient(endpoint, credential);\n",
        "\n",
        "// Role-based access validation\n",
        "if (!await ValidateUserPermissions(user, documentId))\n",
        "{\n",
        "    throw new UnauthorizedAccessException(\"Insufficient permissions\");\n",
        "}\n",
        "```\n",
        "\n",
        "### Data Protection\n",
        "- **Encryption**: End-to-end encryption for documents and search indices\n",
        "- **Access Controls**: Integration with Azure AD for user and group permissions\n",
        "- **Data Residency**: Geographic data location controls for compliance\n",
        "- **Backup & Recovery**: Automated backup and disaster recovery capabilities\n",
        "\n",
        "## 📈 Performance Optimization\n",
        "\n",
        "### Async Processing Patterns\n",
        "```csharp\n",
        "// Efficient async document processing\n",
        "await foreach (var document in documentStream.AsAsyncEnumerable())\n",
        "{\n",
        "    await ProcessDocumentAsync(document, cancellationToken);\n",
        "}\n",
        "```\n",
        "\n",
        "### Memory Management\n",
        "- **Streaming Processing**: Handle large documents without memory issues\n",
        "- **Resource Pooling**: Efficient reuse of expensive resources\n",
        "- **Garbage Collection**: Optimized memory allocation patterns\n",
        "- **Connection Management**: Proper Azure service connection lifecycle\n",
        "\n",
        "### Caching Strategies\n",
        "- **Query Caching**: Cache frequently executed searches\n",
        "- **Document Caching**: In-memory caching for hot documents\n",
        "- **Index Caching**: Optimized vector index caching\n",
        "- **Result Caching**: Intelligent caching of generated responses\n",
        "\n",
        "## 📊 Enterprise Use Cases\n",
        "\n",
        "### Knowledge Management\n",
        "- **Corporate Wiki**: Intelligent search across company knowledge bases\n",
        "- **Policy & Procedures**: Automated compliance and procedure guidance\n",
        "- **Training Materials**: Intelligent learning and development assistance\n",
        "- **Research Databases**: Academic and research paper analysis systems\n",
        "\n",
        "### Customer Support\n",
        "- **Support Knowledge Base**: Automated customer service responses\n",
        "- **Product Documentation**: Intelligent product information retrieval\n",
        "- **Troubleshooting Guides**: Contextual problem-solving assistance\n",
        "- **FAQ Systems**: Dynamic FAQ generation from document collections\n",
        "\n",
        "### Regulatory Compliance\n",
        "- **Legal Document Analysis**: Contract and legal document intelligence\n",
        "- **Compliance Monitoring**: Automated regulatory compliance checking\n",
        "- **Risk Assessment**: Document-based risk analysis and reporting\n",
        "- **Audit Support**: Intelligent document discovery for audits\n",
        "\n",
        "## 🚀 Production Deployment\n",
        "\n",
        "### Monitoring & Observability\n",
        "- **Application Insights**: Detailed telemetry and performance monitoring\n",
        "- **Custom Metrics**: Business-specific KPI tracking and alerting\n",
        "- **Distributed Tracing**: End-to-end request tracking across services\n",
        "- **Health Dashboards**: Real-time system health and performance visualization\n",
        "\n",
        "### Scalability & Reliability\n",
        "- **Auto-Scaling**: Automatic scaling based on load and performance metrics\n",
        "- **High Availability**: Multi-region deployment with failover capabilities\n",
        "- **Load Testing**: Performance validation under enterprise load conditions\n",
        "- **Disaster Recovery**: Automated backup and recovery procedures\n",
        "\n",
        "Ready to build enterprise-grade RAG systems that can handle sensitive documents at scale? Let's architect intelligent knowledge systems for the enterprise! 🏢📖✨"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "language_info": {
          "name": "polyglot-notebook"
        },
        "polyglot_notebook": {
          "kernelName": "csharp"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div><div></div><div></div><div><strong>Installed Packages</strong><ul><li><span>Microsoft.Extensions.AI, 9.9.1</span></li></ul></div></div>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "#r \"nuget: Microsoft.Extensions.AI, 9.9.1\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "4ec1f0d1",
      "metadata": {
        "language_info": {
          "name": "polyglot-notebook"
        },
        "polyglot_notebook": {
          "kernelName": "csharp"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div><div></div><div></div><div><strong>Installed Packages</strong><ul><li><span>Azure.AI.Agents.Persistent, 1.2.0-beta.5</span></li><li><span>Azure.Identity, 1.15.0</span></li><li><span>System.Linq.Async, 6.0.3</span></li></ul></div></div>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "#r \"nuget: Azure.AI.Agents.Persistent, 1.2.0-beta.5\"\n",
        "#r \"nuget: Azure.Identity, 1.15.0\"\n",
        "#r \"nuget: System.Linq.Async, 6.0.3\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "2363ae07",
      "metadata": {
        "language_info": {
          "name": "polyglot-notebook"
        },
        "polyglot_notebook": {
          "kernelName": "csharp"
        }
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d10cec9d",
      "metadata": {
        "language_info": {
          "name": "polyglot-notebook"
        },
        "polyglot_notebook": {
          "kernelName": "csharp"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div><div></div><div></div><div><strong>Installed Packages</strong><ul><li><span>Microsoft.Agents.AI.AzureAI, 1.0.0-preview.251001.2</span></li></ul></div></div>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "#r \"nuget: Microsoft.Agents.AI.AzureAI, 1.0.0-preview.251001.3\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "78199d1c",
      "metadata": {
        "language_info": {
          "name": "polyglot-notebook"
        },
        "polyglot_notebook": {
          "kernelName": "csharp"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div><div></div><div></div><div><strong>Installed Packages</strong><ul><li><span>microsoft.agents.ai, 1.0.0-preview.251001.2</span></li></ul></div></div>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "#r \"nuget: Microsoft.Agents.AI, 1.0.0-preview.251001.3\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "id": "7de4684a",
      "metadata": {
        "language_info": {
          "name": "polyglot-notebook"
        },
        "polyglot_notebook": {
          "kernelName": "csharp"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div><div></div><div></div><div><strong>Installed Packages</strong><ul><li><span>DotNetEnv, 3.1.1</span></li></ul></div></div>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "#r \"nuget: DotNetEnv, 3.1.1\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "id": "251efd31",
      "metadata": {
        "language_info": {
          "name": "polyglot-notebook"
        },
        "polyglot_notebook": {
          "kernelName": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "using System;\n",
        "using System.Linq;\n",
        "using Azure.AI.Agents.Persistent;\n",
        "using Azure.Identity;\n",
        "using Microsoft.Agents.AI;"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "id": "a2e342f1",
      "metadata": {
        "language_info": {
          "name": "polyglot-notebook"
        },
        "polyglot_notebook": {
          "kernelName": "csharp"
        }
      },
      "outputs": [],
      "source": [
        " using DotNetEnv;"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "id": "a7a01653",
      "metadata": {
        "language_info": {
          "name": "polyglot-notebook"
        },
        "polyglot_notebook": {
          "kernelName": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "Env.Load(\"../../../.env\");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "id": "a42735d5",
      "metadata": {
        "language_info": {
          "name": "polyglot-notebook"
        },
        "polyglot_notebook": {
          "kernelName": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var azure_foundry_endpoint = Environment.GetEnvironmentVariable(\"AZURE_AI_PROJECT_ENDPOINT\") ?? throw new InvalidOperationException(\"AZURE_AI_PROJECT_ENDPOINT is not set.\");\n",
        "var azure_foundry_model_id = Environment.GetEnvironmentVariable(\"AZURE_AI_MODEL_DEPLOYMENT_NAME\") ?? \"gpt-4.1-mini\";"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "id": "e29bdb58",
      "metadata": {
        "language_info": {
          "name": "polyglot-notebook"
        },
        "polyglot_notebook": {
          "kernelName": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "string pdfPath = \"./document.md\";"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "id": "7351e12d",
      "metadata": {
        "language_info": {
          "name": "polyglot-notebook"
        },
        "polyglot_notebook": {
          "kernelName": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "using System.IO;\n",
        "\n",
        "async Task<Stream> OpenImageStreamAsync(string path)\n",
        "{\n",
        "\treturn await Task.Run(() => File.OpenRead(path));\n",
        "}\n",
        "\n",
        "var pdfStream = await OpenImageStreamAsync(pdfPath);"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "id": "0b6bf484",
      "metadata": {
        "language_info": {
          "name": "polyglot-notebook"
        },
        "polyglot_notebook": {
          "kernelName": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "var persistentAgentsClient = new PersistentAgentsClient(azure_foundry_endpoint, new AzureCliCredential());"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "id": "81e0dddc",
      "metadata": {
        "language_info": {
          "name": "polyglot-notebook"
        },
        "polyglot_notebook": {
          "kernelName": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "PersistentAgentFileInfo fileInfo = await persistentAgentsClient.Files.UploadFileAsync(pdfStream, PersistentAgentFilePurpose.Agents, \"demo.md\");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "id": "f0c75d80",
      "metadata": {
        "language_info": {
          "name": "polyglot-notebook"
        },
        "polyglot_notebook": {
          "kernelName": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "PersistentAgentsVectorStore fileStore =\n",
        "            await persistentAgentsClient.VectorStores.CreateVectorStoreAsync(\n",
        "                [fileInfo.Id],\n",
        "                metadata: new Dictionary<string, string>() { { \"agentkey\", bool.TrueString } });"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "id": "c77986c5",
      "metadata": {
        "language_info": {
          "name": "polyglot-notebook"
        },
        "polyglot_notebook": {
          "kernelName": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "PersistentAgent agentModel = await persistentAgentsClient.Administration.CreateAgentAsync(\n",
        "            azure_foundry_model_id,\n",
        "            name: \"DotNetRAGAgent\",\n",
        "            tools: [new FileSearchToolDefinition()],\n",
        "            instructions: \"\"\"\n",
        "                You are an AI assistant designed to answer user questions using only the information retrieved from the provided document(s).\n",
        "\n",
        "                - If a user's question cannot be answered using the retrieved context, **you must clearly respond**: \n",
        "                \"I'm sorry, but the uploaded document does not contain the necessary information to answer that question.\"\n",
        "                - Do not answer from general knowledge or reasoning. Do not make assumptions or generate hypothetical explanations.\n",
        "                - Do not provide definitions, tutorials, or commentary that is not explicitly grounded in the content of the uploaded file(s).\n",
        "                - If a user asks a question like \"What is a Neural Network?\", and this is not discussed in the uploaded document, respond as instructed above.\n",
        "                - For questions that do have relevant content in the document (e.g., Contoso's travel insurance coverage), respond accurately, and cite the document explicitly.\n",
        "\n",
        "                You must behave as if you have no external knowledge beyond what is retrieved from the uploaded document.\n",
        "                \"\"\",\n",
        "            toolResources: new()\n",
        "            {\n",
        "                FileSearch = new()\n",
        "                {\n",
        "                    VectorStoreIds = { fileStore.Id },\n",
        "                }\n",
        "            },\n",
        "            metadata: new Dictionary<string, string>() { { \"agentkey\", bool.TrueString } });"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "id": "282326cf",
      "metadata": {
        "language_info": {
          "name": "polyglot-notebook"
        },
        "polyglot_notebook": {
          "kernelName": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "AIAgent agent = await persistentAgentsClient.GetAIAgentAsync(agentModel.Id);"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "id": "2067d313",
      "metadata": {
        "language_info": {
          "name": "polyglot-notebook"
        },
        "polyglot_notebook": {
          "kernelName": "csharp"
        }
      },
      "outputs": [],
      "source": [
        "AgentThread thread = agent.GetNewThread();"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "id": "454c4230",
      "metadata": {
        "language_info": {
          "name": "polyglot-notebook"
        },
        "polyglot_notebook": {
          "kernelName": "csharp"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Contoso's travel insurance coverage includes protection for medical emergencies, trip cancellations, and lost baggage. This ensures that travelers are supported in case of health-related issues during their trip, unforeseen cancellations, and the loss of their belongings while traveling【4:0†demo.md】.\r\n"
          ]
        }
      ],
      "source": [
        "Console.WriteLine(await agent.RunAsync(\"Can you explain Contoso's travel insurance coverage?\", thread));"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": ".NET (C#)",
      "language": "C#",
      "name": ".net-csharp"
    },
    "language_info": {
      "name": "polyglot-notebook"
    },
    "polyglot_notebook": {
      "kernelInfo": {
        "defaultKernelName": "csharp",
        "items": [
          {
            "aliases": [],
            "name": "csharp"
          }
        ]
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}