chore: import upstream snapshot with attribution
dotnet-build-and-test / dotnet-test-functions (push) Has been cancelled
dotnet-build-and-test / paths-filter (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Debug, windows-latest, net9.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Release, ubuntu-latest, net10.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Release, ubuntu-latest, net8.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Release, windows-latest, net472) (push) Has been cancelled
dotnet-build-and-test / dotnet-test (Release, integration, true, ubuntu-latest, net10.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-test (Release, integration, true, windows-latest, net472) (push) Has been cancelled
dotnet-build-and-test / dotnet-foundry-hosted-it (push) Has been cancelled
dotnet-build-and-test / dotnet-build-and-test-check (push) Has been cancelled
dotnet-build-and-test / Integration Test Report (push) Has been cancelled
CodeQL / Analyze (csharp) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
dotnet-build-and-test / dotnet-test-functions (push) Has been cancelled
dotnet-build-and-test / paths-filter (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Debug, windows-latest, net9.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Release, ubuntu-latest, net10.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Release, ubuntu-latest, net8.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Release, windows-latest, net472) (push) Has been cancelled
dotnet-build-and-test / dotnet-test (Release, integration, true, ubuntu-latest, net10.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-test (Release, integration, true, windows-latest, net472) (push) Has been cancelled
dotnet-build-and-test / dotnet-foundry-hosted-it (push) Has been cancelled
dotnet-build-and-test / dotnet-build-and-test-check (push) Has been cancelled
dotnet-build-and-test / Integration Test Report (push) Has been cancelled
CodeQL / Analyze (csharp) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,366 @@
|
||||
# FIDES Implementation Summary
|
||||
|
||||
## Overview
|
||||
|
||||
**FIDES** is a comprehensive deterministic prompt injection defense system for the agent framework. The implementation provides label-based security mechanisms to defend against prompt injection attacks by tracking integrity and confidentiality of content throughout agent execution.
|
||||
|
||||
**🚀 Key Features:**
|
||||
- **Context Provider Pattern** - `SecureAgentConfig` extends `ContextProvider`, injecting tools, instructions, and middleware automatically
|
||||
- **Automatic Variable Hiding** - UNTRUSTED content is automatically hidden without requiring manual intervention
|
||||
- **Per-Item Embedded Labels** - Tools return `list[Content]` with `Content.from_text()` for proper label propagation
|
||||
- **SecureMCPToolProxy Auto-Labeling** - MCP tools are labeled automatically from MCP `ToolAnnotations` hints
|
||||
- **MCP `_meta.ifc` Support** - Per-result IFC labels from servers (for example GitHub MCP with `X-MCP-Features: ifc_labels`) are parsed and enforced
|
||||
- **SecureAgentConfig** - One-line secure agent configuration via `context_providers=[config]`
|
||||
- **Data Exfiltration Prevention** - `max_allowed_confidentiality` prevents sensitive data leakage
|
||||
- **Message-Level Label Tracking** (Phase 1) - Track labels on every message in the conversation
|
||||
|
||||
## Architecture Components
|
||||
|
||||
The FIDES defense system consists of seven main components:
|
||||
|
||||
1. **Content Labeling Infrastructure** - Labels for tracking integrity and confidentiality
|
||||
2. **Label Tracking Middleware** - Automatically assigns, propagates labels, and hides untrusted content
|
||||
3. **Per-Item Embedded Labels** - Tools can return mixed-trust data with per-item security labels
|
||||
4. **Policy Enforcement Middleware** - Blocks tool calls that violate security policies
|
||||
5. **Security Tools** - Specialized tools for safe handling of untrusted content (`quarantined_llm`, `inspect_variable`)
|
||||
6. **SecureAgentConfig** - Context provider for easy secure agent configuration
|
||||
7. **Message-Level Label Tracking** - Track labels on every message in the conversation (Phase 1)
|
||||
8. **MCP Tool/Result Label Integration** - MCP hint-based tool labeling and `_meta.ifc` result label parsing
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### Files Created
|
||||
|
||||
1. **`python/packages/core/agent_framework/security.py`** (~2950 lines — all security primitives, middleware, tools, and configuration in a single public module)
|
||||
- `IntegrityLabel` enum (TRUSTED/UNTRUSTED)
|
||||
- `ConfidentialityLabel` enum (PUBLIC/PRIVATE/USER_IDENTITY)
|
||||
- `ContentLabel` class with serialization support
|
||||
- `combine_labels()` function for label composition
|
||||
- `ContentVariableStore` for client-side content storage
|
||||
- `VariableReferenceContent` for variable indirection
|
||||
- `LabeledMessage` class (inherits from `Message`) for message-level tracking
|
||||
- `check_confidentiality_allowed()` helper for data exfiltration prevention
|
||||
- `LabelTrackingFunctionMiddleware` - Tracks and propagates security labels
|
||||
- `PolicyEnforcementFunctionMiddleware` - Enforces security policies
|
||||
- `SecureAgentConfig` extends `ContextProvider` - automatic secure agent configuration
|
||||
- `quarantined_llm()` - Isolated LLM calls with labeled data
|
||||
- `inspect_variable()` - Controlled variable content inspection
|
||||
- `store_untrusted_content()` - Helper for manual variable indirection (legacy)
|
||||
- `get_security_tools()` - Returns list of security tools
|
||||
- `SECURITY_TOOL_INSTRUCTIONS` - Detailed guidance for agents
|
||||
|
||||
|
||||
2. **`FIDES_DEVELOPER_GUIDE.md`** (~1250 lines)
|
||||
- Located at `python/samples/02-agents/security/FIDES_DEVELOPER_GUIDE.md`
|
||||
- Complete documentation of the FIDES security system
|
||||
- Architecture overview and design rationale
|
||||
- Usage examples (6+ comprehensive scenarios)
|
||||
- Best practices and configuration options
|
||||
- API reference with full parameter documentation
|
||||
- Data exfiltration prevention documentation
|
||||
|
||||
3. **`python/packages/core/tests/test_security.py`** (~800+ lines)
|
||||
- Unit tests for ContentLabel and label operations
|
||||
- Tests for ContentVariableStore functionality
|
||||
- Tests for VariableReferenceContent
|
||||
- Middleware behavior tests (label tracking and policy enforcement)
|
||||
- Automatic hiding tests
|
||||
- Per-item embedded label tests
|
||||
- Context label tracking tests
|
||||
- Message-level tracking tests (Phase 1)
|
||||
- Data exfiltration prevention tests
|
||||
|
||||
4. **`docs/decisions/0024-prompt-injection-defense.md`**
|
||||
- Architecture Decision Record (ADR)
|
||||
- Design rationale and alternatives considered
|
||||
- Security properties and guarantees
|
||||
|
||||
5. **`python/samples/02-agents/security/README.md`**
|
||||
- Sample-focused entry point for the two runnable FIDES security samples
|
||||
- Prerequisites, run commands, and links to the developer guide for deeper details
|
||||
|
||||
### Files Modified
|
||||
|
||||
1. **`python/packages/core/agent_framework/__init__.py`**
|
||||
- Removed root-level security exports so `agent_framework.security` is the canonical import surface
|
||||
|
||||
## Core Features
|
||||
|
||||
### 1. Content Labeling Infrastructure
|
||||
|
||||
- **IntegrityLabel**: TRUSTED (user input) vs UNTRUSTED (AI-generated, external)
|
||||
- **ConfidentialityLabel**: PUBLIC, PRIVATE, USER_IDENTITY
|
||||
- **Label Combination**: Most restrictive policy (UNTRUSTED + metadata merging)
|
||||
- **Serialization**: Full support for `to_dict()` and `from_dict()`
|
||||
|
||||
### 2. Per-Item Embedded Labels
|
||||
|
||||
Tools returning mixed-trust data embed labels on individual items using `Content.from_text()`:
|
||||
|
||||
```python
|
||||
import json
|
||||
from agent_framework import Content, tool
|
||||
|
||||
@tool(description="Fetch emails from inbox")
|
||||
async def fetch_emails(count: int = 5) -> list[Content]:
|
||||
return [
|
||||
Content.from_text(
|
||||
json.dumps({
|
||||
"id": email["id"],
|
||||
"body": email["body"],
|
||||
}),
|
||||
additional_properties={
|
||||
"security_label": {
|
||||
"integrity": "trusted" if email["internal"] else "untrusted",
|
||||
"confidentiality": "private",
|
||||
}
|
||||
),
|
||||
)
|
||||
for email in emails
|
||||
]
|
||||
```
|
||||
|
||||
These embedded labels are automatically consumed by `LabelTrackingFunctionMiddleware`, which:
|
||||
- Extracts the `security_label` from `additional_properties`
|
||||
- Uses the embedded label as the highest-priority source for that item
|
||||
- Automatically hides UNTRUSTED items in the variable store
|
||||
- Replaces hidden items with `VariableReferenceContent` in the LLM context
|
||||
- Preserves TRUSTED items visible to the LLM without tainting the context label
|
||||
|
||||
This enables tools to return mixed-trust data where some items (internal emails) remain visible while untrusted items (external emails) are automatically hidden without manual intervention.
|
||||
},
|
||||
)
|
||||
for email in emails
|
||||
]
|
||||
```
|
||||
|
||||
### 3. Automatic Variable Hiding
|
||||
|
||||
This feature automatically hides any UNTRUSTED content returned by tools while keeping the hiding logic transparent to the developer. Developers do not need to manually call `store_untrusted_content()`. This allows the LLM /agent's context to remain clean and secure. Key aspects include:
|
||||
|
||||
- **Automatic Detection**: Middleware checks integrity label after each tool call
|
||||
- **Automatic Storage**: UNTRUSTED results/items stored in variable store
|
||||
- **Transparent Replacement**: LLM context receives `VariableReferenceContent`
|
||||
- **Context Label Protection**: Hidden content does NOT taint context label
|
||||
|
||||
### 4. Context Label Tracking
|
||||
|
||||
- Context label starts as TRUSTED + PUBLIC
|
||||
- Gets updated (tainted) when non-hidden untrusted content enters context
|
||||
- Policy enforcement uses context label for validation
|
||||
- Provides `get_context_label()` and `reset_context_label()` methods
|
||||
|
||||
### 5. Data Exfiltration Prevention
|
||||
|
||||
Tools declare `max_allowed_confidentiality` to prevent sensitive data leakage:
|
||||
|
||||
```python
|
||||
@tool(
|
||||
description="Post to public Slack channel",
|
||||
additional_properties={
|
||||
"max_allowed_confidentiality": "public", # Blocks PRIVATE data
|
||||
}
|
||||
)
|
||||
async def post_to_slack(channel: str, message: str) -> dict:
|
||||
return {"status": "posted"}
|
||||
```
|
||||
|
||||
### 6. SecureAgentConfig (Context Provider)
|
||||
|
||||
SecureAgentConfig extends `ContextProvider` for automatic secure agent configuration:
|
||||
|
||||
```python
|
||||
config = SecureAgentConfig(
|
||||
auto_hide_untrusted=True,
|
||||
allow_untrusted_tools={"search_web", "fetch_data"},
|
||||
block_on_violation=True,
|
||||
quarantine_chat_client=quarantine_client, # Optional: real LLM for quarantine
|
||||
)
|
||||
|
||||
# Context provider injects tools, instructions, and middleware automatically
|
||||
agent = Agent(
|
||||
client=client,
|
||||
name="secure_assistant",
|
||||
instructions="You are a helpful assistant.",
|
||||
tools=[my_tool],
|
||||
context_providers=[config], # That's it!
|
||||
)
|
||||
```
|
||||
|
||||
### 7. MCP Labeling Pipeline (Hints + `_meta.ifc`)
|
||||
|
||||
FIDES now secures remote MCP integration end-to-end:
|
||||
|
||||
- **Tool labels from hints**: `apply_mcp_security_labels(...)` maps MCP hints (`readOnlyHint`, `openWorldHint`) to FIDES tool properties.
|
||||
- **Safe sink defaults**: tools not explicitly marked `readOnlyHint=True` are treated as potential sinks and receive `max_allowed_confidentiality=public`.
|
||||
- **Result labels from metadata**: MCP result `_meta` is propagated via `__mcp_result_meta__`; `_meta.ifc` is parsed into `security_label` per result item.
|
||||
- **`SecureMCPToolProxy` convenience**: wraps MCP tools/URLs and applies this labeling automatically on connect.
|
||||
|
||||
This behavior is used with the GitHub MCP server when `X-MCP-Features: ifc_labels` is passed, which causes the server to return IFC labels in `_meta` (for example `{"ifc": {"integrity": "untrusted", "confidentiality": "public"}}`).
|
||||
|
||||
## Security Properties
|
||||
|
||||
### Deterministic Defense
|
||||
|
||||
1. **Tiered label propagation**: Every tool result receives a label via 3-tier priority (embedded > source_integrity > input labels join)
|
||||
2. **Context tracking**: Cumulative security state tracked across turns
|
||||
3. **Policy enforcement**: Violations blocked before execution
|
||||
4. **Content isolation**: Untrusted content stored as variables
|
||||
5. **Taint propagation**: Once context becomes UNTRUSTED, it stays UNTRUSTED
|
||||
6. **Data exfiltration prevention**: `max_allowed_confidentiality` gates output destinations
|
||||
7. **Audit trail**: All security events logged
|
||||
8. **No runtime guessing**: Deterministic label assignment
|
||||
|
||||
### Attack Prevention
|
||||
|
||||
- **Direct prompt injection**: Variables hide actual content from LLM
|
||||
- **Indirect prompt injection**: Labels track untrusted AI-generated calls
|
||||
- **Privilege escalation**: Policy blocks untrusted calls to privileged tools
|
||||
- **Data exfiltration**: Confidentiality labels + `max_allowed_confidentiality` enforced
|
||||
- **Tool misuse**: Only whitelisted tools accept untrusted inputs
|
||||
|
||||
## Configuration Options
|
||||
|
||||
### LabelTrackingFunctionMiddleware
|
||||
- `default_integrity`: Default label for unknown sources
|
||||
- `default_confidentiality`: Default confidentiality level
|
||||
- `auto_hide_untrusted`: Enable automatic variable hiding (default: True)
|
||||
- `hide_threshold`: Integrity level at which hiding occurs (default: UNTRUSTED)
|
||||
|
||||
### PolicyEnforcementFunctionMiddleware
|
||||
- `allow_untrusted_tools`: Set of tools accepting untrusted inputs
|
||||
- `block_on_violation`: Block vs warn on violations
|
||||
- `enable_audit_log`: Enable/disable audit logging
|
||||
|
||||
### Tool Metadata (via `additional_properties`)
|
||||
- `confidentiality`: Tool's output confidentiality level
|
||||
- `source_integrity`: Fallback integrity for unlabeled results (data-producing tools only)
|
||||
- `accepts_untrusted`: Explicit untrusted input permission
|
||||
- `max_allowed_confidentiality`: Maximum allowed input confidentiality (for sink tools)
|
||||
- `requires_approval`: Human-in-the-loop requirement
|
||||
|
||||
## Usage Pattern
|
||||
|
||||
### Recommended: SecureAgentConfig as Context Provider
|
||||
|
||||
```python
|
||||
from agent_framework.security import SecureAgentConfig
|
||||
|
||||
config = SecureAgentConfig(
|
||||
auto_hide_untrusted=True,
|
||||
allow_untrusted_tools={"search_web"},
|
||||
block_on_violation=True,
|
||||
)
|
||||
|
||||
# Context provider injects everything automatically
|
||||
agent = Agent(
|
||||
client=client,
|
||||
name="secure_assistant",
|
||||
instructions="You are a helpful assistant.",
|
||||
tools=[search_web],
|
||||
context_providers=[config], # Tools, instructions, and middleware injected via before_run()
|
||||
)
|
||||
```
|
||||
|
||||
### Processing Hidden Content with quarantined_llm
|
||||
|
||||
```python
|
||||
from agent_framework.security import quarantined_llm
|
||||
|
||||
# Agent automatically uses quarantined_llm with variable_ids
|
||||
result = await quarantined_llm(
|
||||
prompt="Summarize this data",
|
||||
variable_ids=["var_abc123"] # Reference hidden content by ID
|
||||
)
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
Comprehensive test suite with:
|
||||
- 115+ unit tests covering all components
|
||||
- Label creation, serialization, combination
|
||||
- Variable store operations
|
||||
- Middleware behavior (tracking and enforcement)
|
||||
- Automatic hiding with per-item labels
|
||||
- Context label tracking
|
||||
- Message-level tracking (Phase 1)
|
||||
- Data exfiltration prevention
|
||||
- Policy violation scenarios
|
||||
- Audit log verification
|
||||
|
||||
Run tests:
|
||||
```bash
|
||||
cd python/packages/core && ../../.venv/bin/pytest tests/test_security.py -v
|
||||
```
|
||||
|
||||
## Code Statistics
|
||||
|
||||
- **Total lines**: ~2,950+ lines (single `security.py` module)
|
||||
- **New modules**: 1 (`security.py` — consolidated from 3 original modules)
|
||||
- **Total tests**: 115+ unit tests
|
||||
- **Documentation**: 1,250+ lines in developer guide
|
||||
- **Examples**: 6+ comprehensive scenarios
|
||||
|
||||
## Deliverables Checklist
|
||||
|
||||
### Core Implementation
|
||||
✅ ContentLabel infrastructure with integrity and confidentiality
|
||||
✅ ContentVariableStore for variable indirection
|
||||
✅ VariableReferenceContent for safe context references
|
||||
✅ LabelTrackingFunctionMiddleware for automatic labeling
|
||||
✅ PolicyEnforcementFunctionMiddleware for policy enforcement
|
||||
✅ quarantined_llm tool for isolated processing
|
||||
✅ inspect_variable tool for controlled content access
|
||||
✅ store_untrusted_content helper for manual variable indirection
|
||||
|
||||
### Automatic Hiding Enhancement
|
||||
✅ Auto-hide UNTRUSTED content with `auto_hide_untrusted` flag
|
||||
✅ Per-middleware ContentVariableStore instances
|
||||
✅ Thread-local storage for middleware access from tools
|
||||
✅ Automatic UNTRUSTED content replacement
|
||||
|
||||
### Per-Item Embedded Labels
|
||||
✅ Support for `additional_properties.security_label` on individual items
|
||||
✅ Mixed-trust data handling (hide untrusted, keep trusted visible)
|
||||
✅ Fallback to `source_integrity` for unlabeled items
|
||||
|
||||
### Context Label Tracking
|
||||
✅ Cumulative context label tracking across turns
|
||||
✅ Hidden content does NOT taint context
|
||||
✅ `get_context_label()` and `reset_context_label()` methods
|
||||
✅ Policy enforcement uses context label
|
||||
|
||||
### Data Exfiltration Prevention
|
||||
✅ `max_allowed_confidentiality` tool property
|
||||
✅ `check_confidentiality_allowed()` helper function
|
||||
✅ Policy enforcement validates confidentiality flow
|
||||
|
||||
### SecureAgentConfig
|
||||
✅ Context provider pattern with `ContextProvider` base class
|
||||
✅ `before_run()` hook for automatic injection of tools, instructions, and middleware
|
||||
✅ One-line secure agent configuration via `context_providers=[config]`
|
||||
✅ `get_tools()`, `get_instructions()`, `get_middleware()` methods (for manual use)
|
||||
✅ `quarantine_chat_client` support for real LLM calls
|
||||
✅ `SECURITY_TOOL_INSTRUCTIONS` constant
|
||||
|
||||
### Documentation & Testing
|
||||
✅ Complete FIDES Developer Guide (~1250 lines)
|
||||
✅ Architecture Decision Record (ADR)
|
||||
✅ Quick Start Guide
|
||||
✅ Comprehensive test suite (115+ tests)
|
||||
✅ Example code with 6+ scenarios
|
||||
✅ 3 complete security examples (email, repo confidentiality, GitHub MCP labels)
|
||||
|
||||
## Summary
|
||||
|
||||
**FIDES** provides a comprehensive, deterministic defense against prompt injection attacks with:
|
||||
|
||||
- **Zero-effort protection**: Automatic variable hiding for developers
|
||||
- **Context provider pattern**: `SecureAgentConfig` extends `ContextProvider` for automatic setup
|
||||
- **Granular control**: Per-item embedded labels via `Content.from_text()` for mixed-trust data
|
||||
- **Easy configuration**: `SecureAgentConfig` for one-line setup
|
||||
- **Data safety**: Exfiltration prevention via confidentiality gates
|
||||
- **Full traceability**: Message-level label tracking
|
||||
- **Complete auditability**: All security events logged
|
||||
|
||||
The system ensures that untrusted content never directly reaches the LLM context and that all tool calls are policy-checked based on the cumulative security state before execution.
|
||||
@@ -0,0 +1,625 @@
|
||||
# CodeAct .NET implementation
|
||||
|
||||
This document describes the .NET realization of the CodeAct design in
|
||||
[`docs/decisions/0024-codeact-integration.md`](../../decisions/0024-codeact-integration.md).
|
||||
|
||||
This document is intentionally focused on the .NET design and public API surface.
|
||||
The initial public .NET type described here is `HyperlightCodeActProvider`. Future .NET backends, such as Monty, should follow the same conceptual model with their own concrete provider types rather than through a public abstract base class or a public executor parameter.
|
||||
|
||||
## What is the goal of this feature?
|
||||
|
||||
Goals:
|
||||
- .NET developers can enable CodeAct through an `AIContextProvider`-based integration.
|
||||
- Developers can configure a provider-owned CodeAct tool set that is separate from the agent's direct tool surface.
|
||||
- Developers can use the same `execute_code` concept for both tool-enabled CodeAct and a standard code interpreter tool implementation.
|
||||
- Developers can swap execution backends over time, starting with Hyperlight while keeping room for alternatives.
|
||||
- Developers can configure execution capabilities such as workspace mounts and outbound network allow lists in a portable way.
|
||||
|
||||
Success Metric:
|
||||
- .NET samples exist for both a tool-enabled CodeAct mode and a standard interpreter mode.
|
||||
|
||||
Implementation-free outcome:
|
||||
- A .NET developer can attach a backend-specific CodeAct provider, choose which tools are available inside CodeAct, and configure execution capabilities without rewriting the function invocation loop or ChatClient pipeline.
|
||||
|
||||
## What is the problem being solved?
|
||||
|
||||
The cross-SDK problem statement and decision rationale live in the [ADR](../../decisions/0024-codeact-integration.md). The items below narrow that statement to .NET-specific design concerns:
|
||||
|
||||
- Today, the easiest way to prototype CodeAct in .NET is to manually configure an `AIFunction` and wire instructions — this is fragile and requires understanding internal sandbox lifecycle details.
|
||||
- There is no first-class .NET design that simultaneously covers Hyperlight-backed CodeAct now, future backend-specific providers, and both tool-enabled and interpreter modes.
|
||||
- Sandbox capabilities such as mounted file access and outbound network access need a portable configuration model instead of ad hoc backend-specific wiring.
|
||||
- Approval behavior needs to be explicit and configurable, mapping to .NET's existing `ApprovalRequiredAIFunction` wrapper mechanism.
|
||||
|
||||
## API Changes
|
||||
|
||||
### CodeAct contract
|
||||
|
||||
#### Terminology
|
||||
|
||||
- **CodeAct** is the primary term.
|
||||
- `execute_code` is the model-facing tool name used by the initial .NET provider in this spec.
|
||||
- Tool-enabled versus interpreter behavior is derived from the presence of CodeAct-managed tools, not from a separate public profile object.
|
||||
|
||||
#### Provider-owned CodeAct tool registry
|
||||
|
||||
A concrete .NET CodeAct provider owns the set of tools available through `call_tool(...)` inside CodeAct.
|
||||
|
||||
Rules:
|
||||
- Only tools explicitly configured on the concrete provider instance are available inside CodeAct.
|
||||
- The provider must not infer its CodeAct-managed tool set from the agent's direct tool configuration (`ChatClientAgentOptions.Tools` or `AIContext.Tools`).
|
||||
- Exclusive versus mixed behavior is achieved by where tools are configured, not by rewriting the agent's direct tool list.
|
||||
|
||||
Implications:
|
||||
- **CodeAct-only tool**: configured on the concrete CodeAct provider only.
|
||||
- **Direct-only tool**: configured on the agent only.
|
||||
- **Tool available both ways**: configured on both the agent and the concrete CodeAct provider.
|
||||
|
||||
#### Managing tools and capabilities after provider construction
|
||||
|
||||
There is no separate runtime setup object in the .NET design. CodeAct tools, file mounts, and outbound network allow-list state are managed directly on the provider through CRUD-style registry methods.
|
||||
|
||||
Preferred pattern:
|
||||
- `AddTools(params AIFunction[] tools) -> void`
|
||||
- `GetTools() -> IReadOnlyList<AIFunction>`
|
||||
- `RemoveTools(params string[] names) -> void`
|
||||
- `ClearTools() -> void`
|
||||
- `AddFileMounts(params FileMount[] mounts) -> void`
|
||||
- `GetFileMounts() -> IReadOnlyList<FileMount>`
|
||||
- `RemoveFileMounts(params string[] mountPaths) -> void`
|
||||
- `ClearFileMounts() -> void`
|
||||
- `AddAllowedDomains(params AllowedDomain[] domains) -> void`
|
||||
- `GetAllowedDomains() -> IReadOnlyList<AllowedDomain>`
|
||||
- `RemoveAllowedDomains(params string[] targets) -> void`
|
||||
- `ClearAllowedDomains() -> void`
|
||||
|
||||
Requirements:
|
||||
- The provider-owned CodeAct tool registry is keyed by tool name (from `AIFunction.Name`).
|
||||
- `AddTools(...)` adds new tools and replaces an existing provider-owned registration when the same tool name is added again.
|
||||
- `GetTools()` returns the provider's current configured CodeAct tool registry.
|
||||
- `RemoveTools(...)` removes provider-owned CodeAct tools by name.
|
||||
- `ClearTools()` removes all provider-owned CodeAct tools.
|
||||
- File mounts are keyed by sandbox mount path.
|
||||
- `AddFileMounts(...)` adds new file mounts and replaces an existing mount when the same mount path is added again.
|
||||
- `GetFileMounts()` returns the provider's current configured file mounts.
|
||||
- `RemoveFileMounts(...)` removes file mounts by mount path.
|
||||
- `ClearFileMounts()` removes all configured file mounts.
|
||||
- Allowed domains are keyed by normalized target string.
|
||||
- `AddAllowedDomains(...)` adds allow-list entries and replaces an existing entry when the same target is added again.
|
||||
- `GetAllowedDomains()` returns the current outbound allow-list entries.
|
||||
- `RemoveAllowedDomains(...)` removes allow-list entries by target.
|
||||
- `ClearAllowedDomains()` removes all configured allow-list entries.
|
||||
- Tool, file-mount, and network-allow-list mutations affect subsequent runs only; runs already in progress keep the snapshot captured at run start.
|
||||
- The provider must snapshot its effective tool registry and capability state at the start of each run so concurrent execution remains deterministic.
|
||||
|
||||
#### Approval model
|
||||
|
||||
The initial .NET design follows the ADR's bundled approval decision and maps to the existing `ApprovalRequiredAIFunction` wrapper from `Microsoft.Extensions.AI.Abstractions`:
|
||||
|
||||
- The provider exposes a default `ApprovalMode` for `execute_code` (enum: `CodeActApprovalMode.AlwaysRequire` / `CodeActApprovalMode.NeverRequire`).
|
||||
|
||||
Effective `execute_code` approval is computed as follows:
|
||||
|
||||
- If the provider default is `AlwaysRequire`, `execute_code` requires approval.
|
||||
- If the provider default is `NeverRequire`, the provider evaluates the provider-owned CodeAct tool registry snapshot for that run.
|
||||
- If every provider-owned CodeAct tool in that snapshot is not an `ApprovalRequiredAIFunction`, `execute_code` does not require approval.
|
||||
- If any provider-owned CodeAct tool in that snapshot is an `ApprovalRequiredAIFunction`, `execute_code` requires approval, even if the generated code may not call that tool.
|
||||
- When the effective approval resolves to `AlwaysRequire`, the generated `execute_code` function is wrapped in `ApprovalRequiredAIFunction` before being added to the `AIContext.Tools`.
|
||||
- Provider-owned tool calls made through `call_tool(...)` during that execution run use the approval already determined for `execute_code`.
|
||||
- Direct-only agent tools are excluded from this calculation.
|
||||
- File and network capabilities do not create a separate runtime approval check in the initial model; configuring them on the provider is itself the approval for those capabilities.
|
||||
|
||||
This is intentionally conservative and matches the shape of the existing .NET function-tool approval flow, where `ApprovalRequiredAIFunction` signals to the `ChatClientAgent` that user approval is needed before invocation.
|
||||
|
||||
#### Shared execution flow
|
||||
|
||||
On each run:
|
||||
1. `ProvideAIContextAsync(...)` snapshots the current CodeAct-managed tool registry and capability settings.
|
||||
2. Computes the effective approval requirement for `execute_code` from the provider default plus the snapshotted tool registry.
|
||||
3. Builds provider-defined instructions.
|
||||
4. Builds a run-scoped `execute_code` `AIFunction` from the snapshot (optionally wrapped in `ApprovalRequiredAIFunction`).
|
||||
5. Returns an `AIContext` containing the instructions and `execute_code` tool.
|
||||
6. When `execute_code` is invoked by the model, the run-scoped function creates or reuses an execution environment.
|
||||
7. If the current provider mode exposes host tools, `call_tool(...)` is bound only to the provider-owned tool registry snapshot.
|
||||
8. Code is executed and results converted to a JSON result string.
|
||||
|
||||
Caching rules:
|
||||
- The Hyperlight backend supports snapshots: the provider caches a reusable clean snapshot after the first sandbox initialization.
|
||||
- No mutable per-run execution state may be shared across concurrent runs.
|
||||
- In-memory interpreter state does not persist across separate `execute_code` calls.
|
||||
- Configured workspace files, mounted files, and any writable artifact/output area are the supported persistence mechanism across calls when the backend exposes them.
|
||||
|
||||
### .NET public API
|
||||
|
||||
#### Core types
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Represents a host-to-sandbox file mount configuration.
|
||||
/// </summary>
|
||||
/// <param name="HostPath">Absolute or relative path on the host filesystem.</param>
|
||||
/// <param name="MountPath">Path inside the sandbox (e.g. "/input/data.csv").</param>
|
||||
public sealed record FileMount(string HostPath, string MountPath);
|
||||
|
||||
/// <summary>
|
||||
/// Represents an outbound network allow-list entry.
|
||||
/// </summary>
|
||||
/// <param name="Target">URL or domain (e.g. "https://api.github.com").</param>
|
||||
/// <param name="Methods">
|
||||
/// Optional HTTP methods to allow (e.g. ["GET", "POST"]).
|
||||
/// Null allows all methods supported by the backend.
|
||||
/// </param>
|
||||
public sealed record AllowedDomain(string Target, IReadOnlyList<string>? Methods = null);
|
||||
|
||||
/// <summary>
|
||||
/// Controls the approval behavior for execute_code invocations.
|
||||
/// </summary>
|
||||
public enum CodeActApprovalMode
|
||||
{
|
||||
/// <summary>execute_code always requires user approval.</summary>
|
||||
AlwaysRequire,
|
||||
|
||||
/// <summary>
|
||||
/// Approval is derived from the provider-owned tool registry:
|
||||
/// if any tool is an ApprovalRequiredAIFunction, execute_code requires approval.
|
||||
/// </summary>
|
||||
NeverRequire,
|
||||
}
|
||||
```
|
||||
|
||||
#### HyperlightCodeActProvider
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// An AIContextProvider that enables CodeAct execution through the
|
||||
/// Hyperlight sandbox backend.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// This provider injects an <c>execute_code</c> tool into the model-facing
|
||||
/// tool surface and builds CodeAct guidance instructions. Guest code executed
|
||||
/// through <c>execute_code</c> runs in an isolated Hyperlight sandbox with
|
||||
/// snapshot/restore for clean state per invocation.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// If no CodeAct-managed tools are configured, the provider uses
|
||||
/// interpreter-style behavior. If one or more CodeAct-managed tools are
|
||||
/// configured, the provider uses tool-enabled behavior and exposes
|
||||
/// <c>call_tool(...)</c> inside the sandbox bound to the configured tools.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class HyperlightCodeActProvider : AIContextProvider, IDisposable
|
||||
{
|
||||
/// <summary>
|
||||
/// Initializes a new HyperlightCodeActProvider.
|
||||
/// </summary>
|
||||
/// <param name="options">Configuration options for the provider.</param>
|
||||
public HyperlightCodeActProvider(HyperlightCodeActProviderOptions options);
|
||||
|
||||
// ----- Tool registry -----
|
||||
|
||||
/// <summary>Adds tools to the provider-owned CodeAct tool registry.</summary>
|
||||
public void AddTools(params AIFunction[] tools);
|
||||
|
||||
/// <summary>Returns the current CodeAct-managed tools.</summary>
|
||||
public IReadOnlyList<AIFunction> GetTools();
|
||||
|
||||
/// <summary>Removes tools by name from the CodeAct tool registry.</summary>
|
||||
public void RemoveTools(params string[] names);
|
||||
|
||||
/// <summary>Removes all CodeAct-managed tools.</summary>
|
||||
public void ClearTools();
|
||||
|
||||
// ----- File mounts -----
|
||||
|
||||
/// <summary>Adds file mount configurations.</summary>
|
||||
public void AddFileMounts(params FileMount[] mounts);
|
||||
|
||||
/// <summary>Returns the current file mount configurations.</summary>
|
||||
public IReadOnlyList<FileMount> GetFileMounts();
|
||||
|
||||
/// <summary>Removes file mounts by sandbox mount path.</summary>
|
||||
public void RemoveFileMounts(params string[] mountPaths);
|
||||
|
||||
/// <summary>Removes all file mount configurations.</summary>
|
||||
public void ClearFileMounts();
|
||||
|
||||
// ----- Network allow-list -----
|
||||
|
||||
/// <summary>Adds outbound network allow-list entries.</summary>
|
||||
public void AddAllowedDomains(params AllowedDomain[] domains);
|
||||
|
||||
/// <summary>Returns the current outbound allow-list entries.</summary>
|
||||
public IReadOnlyList<AllowedDomain> GetAllowedDomains();
|
||||
|
||||
/// <summary>Removes allow-list entries by target.</summary>
|
||||
public void RemoveAllowedDomains(params string[] targets);
|
||||
|
||||
/// <summary>Removes all outbound allow-list entries.</summary>
|
||||
public void ClearAllowedDomains();
|
||||
|
||||
// ----- Lifecycle -----
|
||||
|
||||
/// <summary>Releases the sandbox and all associated native resources.</summary>
|
||||
public void Dispose();
|
||||
}
|
||||
```
|
||||
|
||||
#### HyperlightCodeActProviderOptions
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Configuration options for <see cref="HyperlightCodeActProvider"/>.
|
||||
/// </summary>
|
||||
public sealed class HyperlightCodeActProviderOptions
|
||||
{
|
||||
/// <summary>
|
||||
/// The sandbox backend to use. Default is <c>Wasm</c>.
|
||||
/// </summary>
|
||||
public SandboxBackend Backend { get; set; } = SandboxBackend.Wasm;
|
||||
|
||||
/// <summary>
|
||||
/// Path to the guest module (.wasm or .aot file).
|
||||
/// Required for the Wasm backend; not needed for JavaScript.
|
||||
/// When null, the provider attempts to locate the default packaged
|
||||
/// Python guest module.
|
||||
/// </summary>
|
||||
public string? ModulePath { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Guest heap size. Accepts human-readable strings ("50Mi", "2Gi")
|
||||
/// or raw byte values. Null uses the backend default.
|
||||
/// </summary>
|
||||
public string? HeapSize { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Guest stack size. Accepts human-readable strings ("35Mi")
|
||||
/// or raw byte values. Null uses the backend default.
|
||||
/// </summary>
|
||||
public string? StackSize { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Initial set of CodeAct-managed tools available inside the sandbox.
|
||||
/// </summary>
|
||||
public IEnumerable<AIFunction>? Tools { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Default approval mode for the execute_code tool.
|
||||
/// Default is <see cref="CodeActApprovalMode.NeverRequire"/>.
|
||||
/// </summary>
|
||||
public CodeActApprovalMode ApprovalMode { get; set; } = CodeActApprovalMode.NeverRequire;
|
||||
|
||||
/// <summary>
|
||||
/// Optional workspace root directory on the host.
|
||||
/// When set, it is exposed as the sandbox's input directory.
|
||||
/// </summary>
|
||||
public string? WorkspaceRoot { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Initial file mount configurations.
|
||||
/// </summary>
|
||||
public IEnumerable<FileMount>? FileMounts { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Initial outbound network allow-list entries.
|
||||
/// </summary>
|
||||
public IEnumerable<AllowedDomain>? AllowedDomains { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// State key used to store provider state in AgentSession.StateBag.
|
||||
/// Defaults to "HyperlightCodeActProvider". Override when using
|
||||
/// multiple provider instances on the same agent.
|
||||
/// </summary>
|
||||
public string? StateKey { get; set; }
|
||||
}
|
||||
```
|
||||
|
||||
#### Provider implementation contract
|
||||
|
||||
The concrete provider plugs into the existing .NET `AIContextProvider` surface from `Microsoft.Agents.AI.Abstractions`.
|
||||
|
||||
Required override:
|
||||
- `ProvideAIContextAsync(InvokingContext, CancellationToken) -> ValueTask<AIContext>`
|
||||
|
||||
`ProvideAIContextAsync(...)` is responsible for:
|
||||
- snapshotting the current CodeAct-managed tool registry and capability settings for the run,
|
||||
- computing the effective approval requirement for `execute_code` from the provider default and the snapshotted tool registry,
|
||||
- building a short CodeAct guidance instruction string,
|
||||
- building a run-scoped `execute_code` `AIFunction` from the snapshot,
|
||||
- optionally wrapping it in `ApprovalRequiredAIFunction` when approval is required,
|
||||
- and returning an `AIContext` with `Instructions` and `Tools` set.
|
||||
|
||||
These steps run on every invocation rather than once at construction time because the provider supports CRUD mutations between runs, concurrent runs need independent snapshots, and the effective approval and instructions depend on the tool registry state captured at run start.
|
||||
|
||||
The provider overrides `StateKeys` to return the configured `StateKey` from options, enabling multiple provider instances on the same agent without key collisions.
|
||||
|
||||
Mutating the provider after `ProvideAIContextAsync(...)` has captured a run-scoped snapshot is allowed, but it affects subsequent runs only. Provider implementations synchronize state capture and CRUD operations so shared provider instances remain safe across concurrent runs.
|
||||
|
||||
#### AIFunction-to-sandbox tool bridging
|
||||
|
||||
The Hyperlight sandbox's `RegisterTool(name, Func<string, string>)` accepts a synchronous JSON-in / JSON-out delegate. Provider-owned CodeAct tools are `AIFunction` instances that are async and cancellation-aware.
|
||||
|
||||
Bridging strategy:
|
||||
- At sandbox initialization time, the provider registers each CodeAct-managed tool with the sandbox using the raw JSON overload: `RegisterTool(name, Func<string, string>)`.
|
||||
- When the sandbox guest calls `call_tool("name", ...)`, the bridge delegate:
|
||||
1. Deserializes the JSON arguments.
|
||||
2. Invokes `AIFunction.InvokeAsync(...)` synchronously (via `GetAwaiter().GetResult()`) since the sandbox FFI callback is inherently synchronous.
|
||||
3. Serializes the result back to JSON.
|
||||
- This sync-over-async bridge is a known pragmatic trade-off constrained by the Hyperlight FFI boundary. It is safe because:
|
||||
- Sandbox execution already runs on the thread pool (via `Task.Run`).
|
||||
- The FFI callback runs on a worker thread with no synchronization context.
|
||||
- If the Hyperlight .NET SDK later adds async tool registration, the bridge should migrate to that.
|
||||
|
||||
#### Runtime behavior
|
||||
|
||||
- `ProvideAIContextAsync(...)` adds a short CodeAct guidance block through `AIContext.Instructions`.
|
||||
- `ProvideAIContextAsync(...)` adds `execute_code` through `AIContext.Tools`.
|
||||
- The detailed `call_tool(...)`, sandbox-tool, and capability guidance is carried by the `execute_code` function's `Description`.
|
||||
- `execute_code` invokes the configured Hyperlight sandbox guest.
|
||||
- If the current CodeAct tool registry snapshot is non-empty, the runtime injects `call_tool(...)` bound to the provider-owned tool registry.
|
||||
- The provider does not inspect or mutate the agent's `ChatClientAgentOptions.Tools` or the incoming `AIContext.Tools` to determine its CodeAct tool set.
|
||||
- The provider snapshots the current CodeAct tool registry and capability state at run start, so later registry and allow-list mutations only affect future runs.
|
||||
- Interpreter versus tool-enabled behavior is derived from the presence of CodeAct-managed tools.
|
||||
- `execute_code` is traced like a normal tool invocation within the surrounding agent run.
|
||||
|
||||
#### Backend integration
|
||||
|
||||
Initial public provider:
|
||||
- `HyperlightCodeActProvider`
|
||||
|
||||
Backend-specific notes:
|
||||
- **Hyperlight**
|
||||
- The provider internally creates a `SandboxBuilder` from the options and uses the `Sandbox` API from `HyperlightSandbox.Api`.
|
||||
- The provider uses snapshot/restore to ensure clean execution state per `execute_code` invocation: a "warm" snapshot is taken after the first no-op initialization run, and restored before each subsequent execution.
|
||||
- File access maps to Hyperlight Sandbox's `WithInputDir()` / `WithOutputDir()` / `WithTempOutput()` capability model.
|
||||
- Network access is denied by default and is enabled through `Sandbox.AllowDomain(...)` per-target allow-list entries.
|
||||
- Guest module resolution: if `ModulePath` is null for the Wasm backend, the provider attempts to locate a packaged Python guest module (equivalent to the Python SDK's `python_guest.path` resolution).
|
||||
|
||||
#### Capability handling
|
||||
|
||||
Capabilities are first-class `HyperlightCodeActProviderOptions` properties and provider-managed CRUD surfaces:
|
||||
- `WorkspaceRoot`
|
||||
- `FileMounts`
|
||||
- `AllowedDomains`
|
||||
|
||||
Enabling access means:
|
||||
- Configuring `WorkspaceRoot` or any `FileMounts` enables the sandbox filesystem surface exposed through `/input` and `/output`.
|
||||
- Leaving both `WorkspaceRoot` and `FileMounts` unset means no filesystem surface is configured.
|
||||
- Adding any `AllowedDomains` entry enables outbound access only for the configured targets; leaving it empty means network access is disabled without a separate network mode flag.
|
||||
|
||||
Backends may implement stricter semantics than these top-level settings.
|
||||
|
||||
#### Execution output representation
|
||||
|
||||
Backend execution output maps to a JSON result string returned from the `execute_code` `AIFunction`:
|
||||
|
||||
```json
|
||||
{
|
||||
"stdout": "Hello world\n",
|
||||
"stderr": "",
|
||||
"exit_code": 0,
|
||||
"success": true
|
||||
}
|
||||
```
|
||||
|
||||
Execution failures should surface readable error text in the `stderr` field and a non-zero `exit_code`. Timeouts, out-of-memory conditions, backend crashes, and similar sandbox failures are all `execute_code` failures and should surface as structured error results. Partial textual or file outputs may be returned only when the backend can report them unambiguously.
|
||||
|
||||
#### `execute_code` input contract
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"code": {
|
||||
"type": "string",
|
||||
"description": "Code to execute using the provider's configured backend/runtime behavior."
|
||||
}
|
||||
},
|
||||
"required": ["code"]
|
||||
}
|
||||
```
|
||||
|
||||
#### Thread safety and concurrency
|
||||
|
||||
- All CRUD methods (`AddTools`, `RemoveTools`, `AddFileMounts`, etc.) are synchronized via an internal lock.
|
||||
- `ProvideAIContextAsync(...)` acquires the lock to snapshot current state, then releases it before building the run-scoped function. The run-scoped function closes over the immutable snapshot, not mutable provider state.
|
||||
- Concurrent `execute_code` invocations from different runs use independent sandbox instances or synchronized access to a shared sandbox with snapshot/restore.
|
||||
- Workspace directories (`WorkspaceRoot`, `FileMounts`) are external shared state: concurrent runs against the same workspace can race on files. This is the user's responsibility to manage (e.g., by using per-run output directories or separate provider instances).
|
||||
|
||||
### HyperlightExecuteCodeFunction
|
||||
|
||||
The provider package also exports a standalone `HyperlightExecuteCodeFunction` for direct-tool scenarios where a provider lifecycle is not needed. This is the .NET equivalent of the Python `HyperlightExecuteCodeTool`.
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// A standalone execute_code AIFunction backed by a Hyperlight sandbox.
|
||||
/// Use this for manual/static wiring when the AIContextProvider lifecycle
|
||||
/// is not needed.
|
||||
/// </summary>
|
||||
public sealed class HyperlightExecuteCodeFunction : IDisposable
|
||||
{
|
||||
/// <summary>
|
||||
/// Creates a new standalone code execution function.
|
||||
/// </summary>
|
||||
/// <param name="options">Configuration options.</param>
|
||||
public HyperlightExecuteCodeFunction(HyperlightCodeActProviderOptions options);
|
||||
|
||||
/// <summary>
|
||||
/// Returns this as an AIFunction for direct registration on an agent.
|
||||
/// When approval is required, the returned function is wrapped in
|
||||
/// ApprovalRequiredAIFunction.
|
||||
/// </summary>
|
||||
public AIFunction AsAIFunction();
|
||||
|
||||
/// <summary>
|
||||
/// Builds a CodeAct instruction string describing the available
|
||||
/// tools and capabilities.
|
||||
/// </summary>
|
||||
/// <param name="toolsVisibleToModel">
|
||||
/// When false, the instructions include full tool descriptions
|
||||
/// (for use when tools are only accessible through CodeAct).
|
||||
/// When true, instructions are abbreviated (tools are already
|
||||
/// visible to the model as direct tools).
|
||||
/// </param>
|
||||
public string BuildInstructions(bool toolsVisibleToModel = false);
|
||||
|
||||
/// <summary>Releases sandbox resources.</summary>
|
||||
public void Dispose();
|
||||
}
|
||||
```
|
||||
|
||||
### Internal implementation structure
|
||||
|
||||
The provider and standalone function share internal helpers:
|
||||
|
||||
```
|
||||
Microsoft.Agents.AI.Hyperlight/
|
||||
├── HyperlightCodeActProvider.cs // AIContextProvider implementation
|
||||
├── HyperlightCodeActProviderOptions.cs // Options record
|
||||
├── HyperlightExecuteCodeFunction.cs // Standalone AIFunction for manual wiring
|
||||
├── FileMount.cs // File mount record
|
||||
├── AllowedDomain.cs // Network allow-list record
|
||||
├── CodeActApprovalMode.cs // Approval enum
|
||||
├── Internal/
|
||||
│ ├── SandboxExecutor.cs // Manages sandbox lifecycle, snapshot/restore
|
||||
│ ├── InstructionBuilder.cs // Builds CodeAct instruction strings
|
||||
│ └── ToolBridge.cs // AIFunction ↔ Sandbox.RegisterTool adapter
|
||||
```
|
||||
|
||||
`SandboxExecutor` encapsulates:
|
||||
- Creating and configuring a `Sandbox` from options.
|
||||
- Performing the initial no-op warm-up and snapshot.
|
||||
- Registering bridged tools via `ToolBridge`.
|
||||
- Restoring to the clean snapshot before each execution.
|
||||
- Translating `ExecutionResult` to a JSON string.
|
||||
|
||||
`InstructionBuilder` generates:
|
||||
- A short CodeAct guidance block for `AIContext.Instructions`.
|
||||
- A detailed `execute_code` description including `call_tool(...)` signatures and capability documentation.
|
||||
|
||||
`ToolBridge` handles:
|
||||
- Reflecting `AIFunction` metadata to build the sandbox tool registration.
|
||||
- The sync-over-async invocation bridge.
|
||||
|
||||
## E2E Code Samples
|
||||
|
||||
### Tool-enabled CodeAct mode
|
||||
|
||||
```csharp
|
||||
var fetchDocs = AIFunctionFactory.Create(FetchDocs, name: "fetch_docs");
|
||||
var queryData = AIFunctionFactory.Create(QueryData, name: "query_data");
|
||||
var lookupUser = AIFunctionFactory.Create(LookupUser, name: "lookup_user");
|
||||
|
||||
var codeact = new HyperlightCodeActProvider(new HyperlightCodeActProviderOptions
|
||||
{
|
||||
Tools = [fetchDocs, queryData],
|
||||
WorkspaceRoot = "./workdir",
|
||||
AllowedDomains = [new AllowedDomain("api.github.com", ["GET"])],
|
||||
});
|
||||
codeact.AddTools(lookupUser);
|
||||
|
||||
var sendEmail = AIFunctionFactory.Create(SendEmail, name: "send_email");
|
||||
|
||||
var agent = chatClient.AsAIAgent(
|
||||
instructions: "You are a helpful assistant.",
|
||||
options: new ChatClientAgentOptions
|
||||
{
|
||||
Tools = [sendEmail], // direct-only tool
|
||||
AIContextProviders = [codeact],
|
||||
});
|
||||
|
||||
await using var session = await agent.CreateSessionAsync();
|
||||
var response = await agent.InvokeAsync("Analyze the latest docs", session);
|
||||
```
|
||||
|
||||
### Standard code interpreter mode
|
||||
|
||||
```csharp
|
||||
var codeact = new HyperlightCodeActProvider(new HyperlightCodeActProviderOptions
|
||||
{
|
||||
WorkspaceRoot = "./data",
|
||||
});
|
||||
|
||||
var agent = chatClient.AsAIAgent(
|
||||
instructions: "You are a code interpreter.",
|
||||
options: new ChatClientAgentOptions
|
||||
{
|
||||
AIContextProviders = [codeact],
|
||||
});
|
||||
```
|
||||
|
||||
### Manual static wiring (no provider lifecycle)
|
||||
|
||||
When the tool registry and capability configuration are fixed, the provider lifecycle can be skipped entirely. Build the `execute_code` function and instructions once and pass them directly to the agent:
|
||||
|
||||
```csharp
|
||||
using var executeCode = new HyperlightExecuteCodeFunction(
|
||||
new HyperlightCodeActProviderOptions
|
||||
{
|
||||
Tools = [fetchDocs, queryData],
|
||||
WorkspaceRoot = "./workdir",
|
||||
AllowedDomains = [new AllowedDomain("api.github.com", ["GET"])],
|
||||
});
|
||||
|
||||
var codeactInstructions = executeCode.BuildInstructions(toolsVisibleToModel: false);
|
||||
|
||||
var agent = chatClient.AsAIAgent(
|
||||
instructions: $"You are a helpful assistant.\n\n{codeactInstructions}",
|
||||
options: new ChatClientAgentOptions
|
||||
{
|
||||
Tools = [sendEmail, executeCode.AsAIFunction()],
|
||||
});
|
||||
```
|
||||
|
||||
### With approval required
|
||||
|
||||
```csharp
|
||||
var sensitiveAction = new ApprovalRequiredAIFunction(
|
||||
AIFunctionFactory.Create(DeleteRecords, name: "delete_records"));
|
||||
|
||||
var codeact = new HyperlightCodeActProvider(new HyperlightCodeActProviderOptions
|
||||
{
|
||||
Tools = [fetchDocs, sensitiveAction], // sensitiveAction triggers approval
|
||||
});
|
||||
|
||||
// execute_code will be wrapped in ApprovalRequiredAIFunction because
|
||||
// at least one managed tool (delete_records) requires approval.
|
||||
var agent = chatClient.AsAIAgent(
|
||||
instructions: "You are a helpful assistant.",
|
||||
options: new ChatClientAgentOptions
|
||||
{
|
||||
AIContextProviders = [codeact],
|
||||
});
|
||||
```
|
||||
|
||||
## Relationship to hyperlight-sandbox .NET SDK
|
||||
|
||||
This design depends on the .NET SDK being added in [hyperlight-dev/hyperlight-sandbox#46](https://github.com/hyperlight-dev/hyperlight-sandbox/pull/46). Key types consumed from that SDK:
|
||||
|
||||
| hyperlight-sandbox type | Used for |
|
||||
|---|---|
|
||||
| `Sandbox` | Core sandbox lifecycle: `Run()`, `RegisterTool()`, `AllowDomain()`, `Snapshot()`, `Restore()` |
|
||||
| `SandboxBuilder` | Fluent sandbox construction from provider options |
|
||||
| `SandboxBackend` | Backend selection (Wasm, JavaScript) |
|
||||
| `ExecutionResult` | Capturing stdout, stderr, exit code from guest execution |
|
||||
| `SandboxSnapshot` | Checkpoint/restore for clean state per execution |
|
||||
|
||||
The provider package (`Microsoft.Agents.AI.Hyperlight`) takes a NuGet dependency on `Hyperlight.HyperlightSandbox.Api` and `Microsoft.Extensions.AI.Abstractions`. It does **not** depend on `HyperlightSandbox.Extensions.AI` (`CodeExecutionTool`) — the provider implements its own sandbox lifecycle management with run-scoped snapshots to support concurrent invocations safely.
|
||||
|
||||
## Package structure
|
||||
|
||||
The CodeAct Hyperlight provider ships as an optional NuGet package:
|
||||
- **Package**: `Microsoft.Agents.AI.Hyperlight`
|
||||
- **Dependencies**:
|
||||
- `Microsoft.Agents.AI.Abstractions` (for `AIContextProvider`, `AIContext`)
|
||||
- `Microsoft.Extensions.AI.Abstractions` (for `AIFunction`, `ApprovalRequiredAIFunction`)
|
||||
- `Hyperlight.HyperlightSandbox.Api` (for sandbox API)
|
||||
- **Target framework**: `net8.0`
|
||||
|
||||
This keeps CodeAct and its native sandbox dependencies optional — users who do not need CodeAct do not take on the Hyperlight installation and dependency footprint.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **Guest module distribution**: How should the default Python guest module (`.aot` file) be distributed for .NET consumers? Options include a separate NuGet package with native assets, a runtime download, or requiring users to build/provide their own.
|
||||
2. **Async tool registration**: If the Hyperlight .NET SDK adds async tool callback support in a future release, the sync-over-async bridge should be replaced. This is tracked as a known technical debt item.
|
||||
3. **Output file access**: The Hyperlight sandbox exposes `GetOutputFiles()` and `OutputPath` for retrieving files written by guest code. The initial design returns these as part of the JSON result. A future iteration could surface output files as framework-native content (e.g., `DataContent` or URI references).
|
||||
4. **Multiple sandbox instances for concurrency**: The current design uses synchronized access to a single sandbox with snapshot/restore. An alternative pooling strategy (one sandbox per concurrent run) could improve throughput at the cost of memory. This is deferred to implementation time.
|
||||
@@ -0,0 +1,385 @@
|
||||
# CodeAct Python implementation
|
||||
|
||||
This document describes the Python realization of the CodeAct design in
|
||||
[`docs/decisions/0024-codeact-integration.md`](../../decisions/0024-codeact-integration.md).
|
||||
|
||||
This document is intentionally focused on the Python design and public API surface.
|
||||
The initial public Python type described here is `HyperlightCodeActProvider`. Future Python backends, such as Monty, should follow the same conceptual model with their own concrete provider types rather than through a public abstract base class or a public executor parameter.
|
||||
|
||||
## What is the goal of this feature?
|
||||
|
||||
Goals:
|
||||
- Python developers can enable CodeAct through a `ContextProvider`-based integration.
|
||||
- Developers can configure a provider-owned CodeAct tool set that is separate from the agent's direct `tools=` surface.
|
||||
- Developers can use the same `execute_code` concept for both tool-enabled CodeAct and a standard code interpreter tool implementation.
|
||||
- Developers can swap execution backends over time, starting with Hyperlight while keeping room for alternatives such as Pydantic's Monty.
|
||||
- Developers can configure execution capabilities such as workspace mounts and outbound network allow lists in a portable way.
|
||||
|
||||
Success Metric:
|
||||
- Python samples exist for both a tool-enabled CodeAct mode and a standard interpreter mode.
|
||||
|
||||
Implementation-free outcome:
|
||||
- A Python developer can attach a backend-specific CodeAct provider, choose which tools are available inside CodeAct, and configure execution capabilities without rewriting the function invocation loop.
|
||||
|
||||
## What is the problem being solved?
|
||||
|
||||
The cross-SDK problem statement and decision rationale live in the [ADR](../../decisions/0024-codeact-integration.md). The items below narrow that statement to Python-specific design concerns:
|
||||
|
||||
- Today, the easiest way to prototype CodeAct is to infer or reshape the agent's direct tool surface, which is fragile and hard to reason about.
|
||||
- In Python, inferring a CodeAct tool surface from generic agent tool configuration is fragile and hard to reason about.
|
||||
- There is no first-class Python design that simultaneously covers Hyperlight-backed CodeAct now, future backend-specific providers such as Monty, and both tool-enabled and interpreter modes.
|
||||
- Sandbox capabilities such as mounted file access and outbound network access need a portable configuration model instead of ad hoc backend-specific wiring.
|
||||
- Approval behavior needs to be explicit and configurable, especially when CodeAct and direct tool calling may both be available.
|
||||
|
||||
## API Changes
|
||||
|
||||
### CodeAct contract
|
||||
|
||||
#### Terminology
|
||||
|
||||
- **CodeAct** is the primary term.
|
||||
- **Code mode**, **codemode**, and **programmatic tool calling** refer to the same concept in this document.
|
||||
- `execute_code` is the model-facing tool name used by the initial Python providers in this spec.
|
||||
|
||||
#### Provider-owned CodeAct tool registry
|
||||
|
||||
A concrete Python CodeAct provider owns the set of tools available through `call_tool(...)` inside CodeAct.
|
||||
|
||||
Rules:
|
||||
- Only tools explicitly configured on the concrete provider instance are available inside CodeAct.
|
||||
- The provider must not infer its CodeAct-managed tool set from the agent's direct `tools=` configuration.
|
||||
- Exclusive versus mixed behavior is achieved by where tools are configured, not by rewriting the agent's direct tool list.
|
||||
|
||||
Implications:
|
||||
- **CodeAct-only tool**: configured on the concrete CodeAct provider only.
|
||||
- **Direct-only tool**: configured on the agent only.
|
||||
- **Tool available both ways**: configured on both the agent and the concrete CodeAct provider.
|
||||
|
||||
#### Managing tools and capabilities after provider construction
|
||||
|
||||
There is no separate runtime setup object in the Python design. CodeAct tools, file mounts, and outbound network allow-list state are managed directly on the provider through CRUD-style registry methods.
|
||||
|
||||
Preferred pattern:
|
||||
- `add_tools(...) -> None`
|
||||
- `get_tools() -> Sequence[ToolTypes]`
|
||||
- `remove_tool(...) -> None`
|
||||
- `clear_tools() -> None`
|
||||
- `add_file_mounts(...) -> None`
|
||||
- `get_file_mounts() -> Sequence[FileMount]`
|
||||
- `remove_file_mount(...) -> None`
|
||||
- `clear_file_mounts() -> None`
|
||||
- `add_allowed_domains(...) -> None`
|
||||
- `get_allowed_domains() -> Sequence[AllowedDomain]`
|
||||
- `remove_allowed_domain(...) -> None`
|
||||
- `clear_allowed_domains() -> None`
|
||||
|
||||
Requirements:
|
||||
- The provider-owned CodeAct tool registry is keyed by tool name.
|
||||
- `add_tools(...)` adds new tools and replaces an existing provider-owned registration when the same tool name is added again.
|
||||
- `get_tools()` returns the provider's current configured CodeAct tool registry.
|
||||
- `remove_tool(...)` removes provider-owned CodeAct tools by name.
|
||||
- `clear_tools()` removes all provider-owned CodeAct tools.
|
||||
- File mounts are keyed by sandbox mount path.
|
||||
- `add_file_mounts(...)` adds new file mounts and replaces an existing mount when the same mount path is added again.
|
||||
- `get_file_mounts()` returns the provider's current configured file mounts.
|
||||
- `remove_file_mount(...)` removes file mounts by mount path.
|
||||
- `clear_file_mounts()` removes all configured file mounts.
|
||||
- Allowed domains are keyed by normalized target string.
|
||||
- `add_allowed_domains(...)` adds allow-list entries and replaces an existing entry when the same target is added again.
|
||||
- `get_allowed_domains()` returns the current outbound allow-list entries.
|
||||
- `remove_allowed_domain(...)` removes allow-list entries by target.
|
||||
- `clear_allowed_domains()` removes all configured allow-list entries.
|
||||
- Tool, file-mount, and network-allow-list mutations affect subsequent runs only; runs already in progress keep the snapshot captured at run start.
|
||||
- The provider must snapshot its effective tool registry and capability state at the start of each run so concurrent execution remains deterministic.
|
||||
|
||||
#### Approval model
|
||||
|
||||
The initial Python design follows the ADR's initial approval decision and reuses the existing tool approval vocabulary from `agent_framework._tools`:
|
||||
|
||||
- `approval_mode="always_require"`
|
||||
- `approval_mode="never_require"`
|
||||
|
||||
The provider exposes a default `approval_mode` for `execute_code`.
|
||||
|
||||
Effective `execute_code` approval is computed as follows:
|
||||
|
||||
- If the provider default is `always_require`, `execute_code` requires approval.
|
||||
- If the provider default is `never_require`, the provider evaluates the provider-owned CodeAct tool registry snapshot for that run.
|
||||
- If every provider-owned CodeAct tool in that snapshot is `never_require`, `execute_code` is `never_require`.
|
||||
- If any provider-owned CodeAct tool in that snapshot is `always_require`, `execute_code` is `always_require`, even if the generated code may not call that tool.
|
||||
- Provider-owned tool calls made through `call_tool(...)` during that execution run use the approval already determined for `execute_code`.
|
||||
- Direct-only agent tools are excluded from this calculation.
|
||||
- File and network capabilities do not create a separate runtime approval check in the initial model; configuring them on the provider, including adding file mounts or outbound network allow-list entries, is itself the approval for those capabilities.
|
||||
|
||||
This is intentionally conservative and matches the shape of the current function-tool approval flow, where `FunctionTool` uses `always_require` / `never_require` and the auto-invocation loop escalates the whole batch if any called tool requires approval.
|
||||
|
||||
If one sensitive provider-owned tool causes `execute_code` to require approval more often than desired, the mitigation is to keep that tool direct-only or expose it through a different CodeAct provider/tool surface. The initial model does not try to infer whether generated code will actually call that tool before approval.
|
||||
|
||||
If the framework later standardizes pre-execution inspection or nested per-tool approvals, the Python provider surface can grow to expose that explicitly. The initial design does not assume that those extra modes are required.
|
||||
|
||||
#### Shared execution flow
|
||||
|
||||
On each run:
|
||||
1. Resolve the provider's backend/runtime behavior, capabilities, provider default `approval_mode`, and provider-owned tool registry.
|
||||
2. Compute the effective approval requirement for `execute_code` from the provider default plus the provider-owned tool registry snapshot.
|
||||
3. Build provider-defined instructions.
|
||||
4. Add `execute_code` to the model-facing tool surface.
|
||||
5. Invoke the underlying model.
|
||||
6. When `execute_code` is called, create or reuse an execution environment keyed by provider type, backend setup identity, capability configuration, and provider-owned tool signature.
|
||||
7. If the current provider mode exposes host tools, expose `call_tool(...)` bound only to the provider-owned tool registry.
|
||||
8. Execute code and convert results to framework-native content objects.
|
||||
|
||||
Caching rules:
|
||||
- Backends that support snapshots may cache a reusable clean snapshot.
|
||||
- Backends that do not support snapshots may still cache warm initialization artifacts.
|
||||
- No mutable per-run execution state may be shared across concurrent runs.
|
||||
- In-memory interpreter state does not persist across separate `execute_code` calls.
|
||||
- Configured workspace files, mounted files, and any writable artifact/output area are the supported persistence mechanism across calls when the backend exposes them.
|
||||
|
||||
### Python public API
|
||||
|
||||
#### Core types
|
||||
|
||||
```python
|
||||
class FileMount(NamedTuple):
|
||||
host_path: str | Path
|
||||
mount_path: str
|
||||
|
||||
FileMountInput = str | tuple[str | Path, str] | FileMount
|
||||
|
||||
|
||||
class AllowedDomain(NamedTuple):
|
||||
target: str
|
||||
methods: tuple[str, ...] | None = None
|
||||
|
||||
|
||||
AllowedDomainInput = str | tuple[str, str | Sequence[str]] | AllowedDomain
|
||||
|
||||
|
||||
class HyperlightCodeActProvider(ContextProvider):
|
||||
def __init__(
|
||||
self,
|
||||
source_id: str = "hyperlight_codeact",
|
||||
*,
|
||||
backend: str = "wasm",
|
||||
module: str | None = "python_guest.path",
|
||||
module_path: str | None = None,
|
||||
tools: ToolTypes | None = None,
|
||||
approval_mode: Literal["always_require", "never_require"] = "never_require",
|
||||
workspace_root: Path | None = None,
|
||||
file_mounts: Sequence[FileMountInput] = (),
|
||||
allowed_domains: Sequence[AllowedDomainInput] = (),
|
||||
) -> None: ...
|
||||
|
||||
def add_tools(self, tools: ToolTypes | Sequence[ToolTypes]) -> None: ...
|
||||
def get_tools(self) -> Sequence[ToolTypes]: ...
|
||||
def remove_tool(self, name: str) -> None: ...
|
||||
def clear_tools(self) -> None: ...
|
||||
def add_file_mounts(self, mounts: FileMountInput | Sequence[FileMountInput]) -> None: ...
|
||||
def get_file_mounts(self) -> Sequence[FileMount]: ...
|
||||
def remove_file_mount(self, mount_path: str) -> None: ...
|
||||
def clear_file_mounts(self) -> None: ...
|
||||
def add_allowed_domains(self, domains: AllowedDomainInput | Sequence[AllowedDomainInput]) -> None: ...
|
||||
def get_allowed_domains(self) -> Sequence[AllowedDomain]: ...
|
||||
def remove_allowed_domain(self, domain: str) -> None: ...
|
||||
def clear_allowed_domains(self) -> None: ...
|
||||
```
|
||||
|
||||
`file_mounts` accepts three equivalent input forms:
|
||||
- `"data/report.csv"` uses the same relative path on the host and in the sandbox.
|
||||
- `("fixtures/users.json", "data/users.json")` or `(Path("fixtures/users.json"), "data/users.json")` uses distinct host and sandbox paths.
|
||||
- `FileMount(Path("fixtures/users.json"), "data/users.json")` is the named-tuple form of the explicit pair.
|
||||
|
||||
`allowed_domains` accepts three equivalent input forms:
|
||||
- `"github.com"` allows that target with all backend-supported methods.
|
||||
- `("github.com", "GET")` or `("github.com", ["GET", "HEAD"])` uses an explicit per-target method list.
|
||||
- `AllowedDomain("github.com", ("GET", "HEAD"))` is the named-tuple form of the explicit entry.
|
||||
|
||||
No public abstract `CodeActContextProvider` base or public `executor=` parameter is required for the initial Python API.
|
||||
|
||||
The initial alpha package also exports a standalone `HyperlightExecuteCodeTool`
|
||||
for direct-tool scenarios where a provider is not needed. That standalone tool
|
||||
should advertise `call_tool(...)`, the registered sandbox tools, and capability
|
||||
state through its own `description` rather than requiring separate agent
|
||||
instructions.
|
||||
|
||||
Provider modes:
|
||||
- If no CodeAct-managed tools are configured, `HyperlightCodeActProvider` uses interpreter-style behavior.
|
||||
- If one or more CodeAct-managed tools are configured, `HyperlightCodeActProvider` uses tool-enabled behavior.
|
||||
|
||||
#### Python provider implementation contract
|
||||
|
||||
The concrete provider plugs into the existing Python `ContextProvider` surface from `agent_framework._sessions`.
|
||||
|
||||
The Hyperlight package also depends on a small set of core hooks that must remain available from `agent-framework-core`:
|
||||
- `ContextProvider.before_run(...)`
|
||||
- `SessionContext.extend_instructions(...)`
|
||||
- `SessionContext.extend_tools(...)`
|
||||
- per-run runtime tool access via `SessionContext.options["tools"]`
|
||||
- the shared `ApprovalMode` vocabulary used by `FunctionTool`
|
||||
|
||||
Required lifecycle hook:
|
||||
- `before_run(*, agent, session, context, state) -> None`
|
||||
|
||||
Optional lifecycle hook:
|
||||
- `after_run(*, agent, session, context, state) -> None`
|
||||
|
||||
`before_run(...)` is responsible for:
|
||||
- snapshotting the current CodeAct-managed tool registry and capability settings for the run,
|
||||
- computing the effective approval requirement for `execute_code` from the provider default and the snapshotted tool registry,
|
||||
- adding a short CodeAct guidance block,
|
||||
- adding `execute_code` to the run through `SessionContext.extend_tools(...)`,
|
||||
- and wiring any backend-specific execution state needed for the run.
|
||||
|
||||
These steps run on every invocation rather than once at construction time because the provider supports CRUD mutations between runs, concurrent runs need independent snapshots, and the effective approval and instructions depend on the tool registry state captured at run start. When the tool registry and capability configuration are fixed for the lifetime of the agent, the manual wiring pattern (see `codeact_manual_wiring.py`) can be used instead, which passes the tool and instructions directly to the `Agent` constructor and avoids the per-run provider lifecycle entirely.
|
||||
|
||||
If the provider stores anything in `state`, that value must stay JSON-serializable.
|
||||
|
||||
Mutating the provider after `before_run(...)` has captured a run-scoped snapshot is allowed, but it affects subsequent runs only. Provider implementations should synchronize state capture and CRUD operations so shared provider instances remain safe across concurrent runs.
|
||||
|
||||
`after_run(...)` is responsible for any backend-specific cleanup or post-processing that must happen after the model invocation completes.
|
||||
|
||||
If shared internal helpers are introduced later for multiple concrete providers, they should standardize responsibilities for:
|
||||
- building instructions,
|
||||
- computing effective approval,
|
||||
- configuring file access,
|
||||
- configuring network access,
|
||||
- preparing or restoring execution state,
|
||||
- executing code,
|
||||
- and converting backend output into framework-native `Content`.
|
||||
|
||||
#### Runtime behavior
|
||||
|
||||
- `before_run(...)` adds a short CodeAct guidance block through `SessionContext.extend_instructions(...)`.
|
||||
- `before_run(...)` adds `execute_code` through `SessionContext.extend_tools(...)`.
|
||||
- The detailed `call_tool(...)`, sandbox-tool, and capability guidance is carried by `execute_code.description`.
|
||||
- `execute_code` invokes the configured Hyperlight sandbox guest.
|
||||
- If the current CodeAct tool registry is non-empty, the runtime injects `call_tool(...)` bound to the provider-owned tool registry.
|
||||
- The provider does not inspect or mutate `Agent.default_options["tools"]` or `context.options["tools"]` to determine its CodeAct tool set.
|
||||
- The provider snapshots the current CodeAct tool registry and capability state at run start, so later registry and allow-list mutations only affect future runs.
|
||||
- Interpreter versus tool-enabled behavior is derived from the concrete provider and the presence of CodeAct-managed tools, not from a separate public profile object.
|
||||
- `execute_code` should be traced like a normal tool invocation within the surrounding agent run, and provider-owned tool calls executed through `call_tool(...)` should continue to emit ordinary tool invocation telemetry.
|
||||
|
||||
#### Backend integration
|
||||
|
||||
Initial public provider:
|
||||
- `HyperlightCodeActProvider`
|
||||
|
||||
Backend-specific notes:
|
||||
- **Hyperlight**
|
||||
- Provider construction needs a guest artifact via `module`, which may be a packaged guest module name or a path to a compiled guest artifact.
|
||||
- File access maps naturally to Hyperlight Sandbox's read-only `/input` and writable `/output` capability model.
|
||||
- Network access is denied by default and is enabled through per-target allow-list entries.
|
||||
- **Monty**
|
||||
- A future `MontyCodeActProvider` should be a separate public type rather than a `HyperlightCodeActProvider` mode.
|
||||
- Monty does not expose built-in filesystem or network access directly inside the interpreter.
|
||||
- File and URL access are mediated through host-provided external functions, so a Monty provider would need to translate provider settings into virtual files and allow-checked callbacks.
|
||||
- Monty setup may also include backend-specific inputs such as `script_name`, optional type-check stubs, or restored snapshots.
|
||||
|
||||
#### Capability handling
|
||||
|
||||
Capabilities are first-class `HyperlightCodeActProvider` init parameters and provider-managed CRUD surfaces:
|
||||
- `workspace_root`
|
||||
- `file_mounts`
|
||||
- `allowed_domains`
|
||||
|
||||
Concrete providers should normalize these settings internally. Hyperlight can map them directly to sandbox capabilities, while Monty must enforce them through host-mediated file and network functions and may apply stricter URL-level checks than the public provider surface expresses.
|
||||
|
||||
Expected management split:
|
||||
- `workspace_root` remains a direct configuration value on the provider,
|
||||
- file mounts are managed through provider CRUD methods,
|
||||
- outbound allow-list entries are managed through provider CRUD methods.
|
||||
|
||||
Enabling access means:
|
||||
- Configuring `workspace_root` or any `file_mounts` enables the sandbox filesystem surface exposed through `/input` and `/output`.
|
||||
- Leaving both `workspace_root` and `file_mounts` unset means no filesystem surface is configured.
|
||||
- Adding any `allowed_domains` entry enables outbound access only for the configured targets; leaving it empty means network access is disabled without a separate `network_mode` flag.
|
||||
- A string target allows all backend-supported methods for that target; an explicit tuple or `AllowedDomain` entry narrows the methods for that target.
|
||||
|
||||
Backends may implement stricter semantics than these top-level settings. For example, Hyperlight naturally maps file access to `/input` and `/output`, while Monty would enforce equivalent policy through host-provided callbacks rather than direct interpreter I/O.
|
||||
|
||||
#### Execution output representation
|
||||
|
||||
Backend execution output should be translated into existing AF `Content` values rather than a custom `CodeActExecutionResult` type.
|
||||
|
||||
Use the existing content model from `agent_framework._types`, for example:
|
||||
- `Content.from_code_interpreter_tool_result(outputs=[...])` to surface the overall result of sandboxed code execution,
|
||||
- `Content.from_text(...)` for plain textual output,
|
||||
- `Content.from_data(...)` or `Content.from_uri(...)` for generated files or binary artifacts,
|
||||
- `Content.from_error(...)` for execution failures,
|
||||
- and `Content.from_function_result(..., result=list[Content])` when surfacing the final result of `execute_code` through the normal tool result path.
|
||||
|
||||
#### `execute_code` input contract
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"code": {
|
||||
"type": "string",
|
||||
"description": "Code to execute using the provider's configured backend/runtime behavior."
|
||||
}
|
||||
},
|
||||
"required": ["code"]
|
||||
}
|
||||
```
|
||||
|
||||
Execution failures should surface readable error text and structured error `Content`, not a custom backend result object.
|
||||
|
||||
Timeouts, out-of-memory conditions, backend crashes, and similar sandbox failures are all `execute_code` failures and should surface as structured error content. Partial textual or file outputs may be returned only when the backend can report them unambiguously; callers should not rely on partial-output recovery as a portable contract.
|
||||
|
||||
## E2E Code Samples
|
||||
|
||||
### Tool-enabled CodeAct mode
|
||||
|
||||
```python
|
||||
codeact = HyperlightCodeActProvider(
|
||||
tools=[fetch_docs, query_data],
|
||||
workspace_root="./workdir",
|
||||
allowed_domains=[("api.github.com", "GET")],
|
||||
)
|
||||
codeact.add_tools([lookup_user])
|
||||
|
||||
agent = Agent(
|
||||
client=client,
|
||||
name="assistant",
|
||||
tools=[send_email], # direct-only tool
|
||||
context_providers=[codeact],
|
||||
)
|
||||
```
|
||||
|
||||
### Standard code interpreter mode
|
||||
|
||||
```python
|
||||
codeact = HyperlightCodeActProvider(
|
||||
workspace_root="./data",
|
||||
)
|
||||
|
||||
agent = Agent(
|
||||
client=client,
|
||||
name="interpreter",
|
||||
context_providers=[codeact],
|
||||
)
|
||||
```
|
||||
|
||||
### Manual static wiring (no per-run provider lifecycle)
|
||||
|
||||
When the tool registry and capability configuration are fixed, the provider lifecycle can be skipped entirely. Build the `execute_code` tool and instructions once and pass them directly to the agent:
|
||||
|
||||
```python
|
||||
execute_code = HyperlightExecuteCodeTool(
|
||||
tools=[fetch_docs, query_data],
|
||||
workspace_root="./workdir",
|
||||
allowed_domains=[("api.github.com", "GET")],
|
||||
approval_mode="never_require",
|
||||
)
|
||||
|
||||
codeact_instructions = execute_code.build_instructions(tools_visible_to_model=False)
|
||||
|
||||
agent = Agent(
|
||||
client=client,
|
||||
name="assistant",
|
||||
instructions=f"You are a helpful assistant.\n\n{codeact_instructions}",
|
||||
tools=[send_email, execute_code],
|
||||
)
|
||||
```
|
||||
@@ -0,0 +1,48 @@
|
||||
# AGENTS.md
|
||||
|
||||
Instructions for AI coding agents working on durable agents documentation.
|
||||
|
||||
## Scope
|
||||
|
||||
This directory contains feature documentation for the durable agents integration. The source code and samples live elsewhere:
|
||||
|
||||
- .NET implementation: `dotnet/src/Microsoft.Agents.AI.DurableTask/` and `dotnet/src/Microsoft.Agents.AI.Hosting.AzureFunctions/`
|
||||
- Python implementation: `python/packages/durabletask/` and `python/packages/azurefunctions/` (package `agent-framework-azurefunctions`)
|
||||
- .NET samples: `dotnet/samples/04-hosting/DurableAgents/`
|
||||
- Python samples: `python/samples/04-hosting/durabletask/`
|
||||
- Official docs (Microsoft Learn): <https://learn.microsoft.com/agent-framework/integrations/azure-functions>
|
||||
|
||||
## Document structure
|
||||
|
||||
| File | Purpose |
|
||||
| --- | --- |
|
||||
| `README.md` | Main technical overview: architecture, hosting models, orchestration patterns, and links to samples. |
|
||||
| `durable-agents-ttl.md` | Deep-dive on session Time-To-Live (TTL) configuration and behavior. |
|
||||
|
||||
Add new sibling documents when a topic is too detailed for the README (e.g., a new feature like reliable streaming or MCP tool exposure). Keep the README focused on orientation and link out to siblings for depth.
|
||||
|
||||
## Writing guidelines
|
||||
|
||||
- **Audience**: Developers already familiar with the Microsoft Agent Framework who want to understand what durability adds and how to use it.
|
||||
- **Host-agnostic first**: Durable agents work in console apps, Azure Functions, and any Durable Task–compatible host. Show host-agnostic patterns (plain orchestration functions, `IServiceCollection` registration) before Azure Functions–specific patterns. Avoid giving the impression that Azure Functions is the only hosting option.
|
||||
- **Both languages**: Always include C# and Python examples side by side. Keep them equivalent in functionality.
|
||||
- **Callout syntax**: Use GitHub-flavored callouts (`> [!NOTE]`, `> [!IMPORTANT]`, `> [!WARNING]`) rather than bold-text callouts (`> **Note:** ...`).
|
||||
- **Line length**: Do not wrap long lines. Rely on text viewers / renderers for line wrapping.
|
||||
- **Tables**: Use spaces around pipes in separator rows (`| --- |` not `|---|`).
|
||||
- **Code snippets**: Keep them minimal and self-contained. Omit boilerplate (using statements, environment variable reads) unless the snippet is specifically about setup.
|
||||
- **Cross-references**: Link to Microsoft Learn for conceptual background (Durable Entities, Durable Task Scheduler, Azure Functions). Link to sibling docs within this directory for feature deep-dives.
|
||||
|
||||
## Linting
|
||||
|
||||
Run markdownlint on all documents before committing, with line-length checks disabled:
|
||||
|
||||
```bash
|
||||
markdownlint docs/features/durable-agents/ --disable MD013
|
||||
```
|
||||
|
||||
## When to update these docs
|
||||
|
||||
- A new durable agent feature is added (e.g., a new orchestration pattern, hosting model, or configuration option).
|
||||
- The public API surface changes in a way that affects how developers use durable agents.
|
||||
- New sample directories are added — update the sample links in README.md.
|
||||
- The official Microsoft Learn documentation is restructured — update external links.
|
||||
@@ -0,0 +1,239 @@
|
||||
# Durable agents
|
||||
|
||||
## Overview
|
||||
|
||||
Durable agents extend the standard Microsoft Agent Framework with **durable state management** powered by the Durable Task framework. An ordinary Agent Framework agent runs in-process: its conversation history lives in memory and is lost when the process ends. A durable agent persists conversation history and execution state in external storage so that sessions survive process restarts, failures, and scale-out events.
|
||||
|
||||
| Capability | Ordinary agent | Durable agent |
|
||||
| --- | --- | --- |
|
||||
| Conversation history | In-memory only | Durably persisted |
|
||||
| Failure recovery | State lost on crash | Automatically resumed |
|
||||
| Multi-instance scale-out | Not supported | Any worker can resume a session |
|
||||
| Multi-agent orchestrations | Manual coordination | Deterministic, checkpointed workflows |
|
||||
| Human-in-the-loop | Must keep process alive | Can wait days/weeks with zero compute |
|
||||
| Hosting | Any process | Console app, Azure Functions, or any Durable Task–compatible host |
|
||||
|
||||
> [!NOTE]
|
||||
> For a step-by-step tutorial and deployment guidance, see [Azure Functions (Durable)](https://learn.microsoft.com/agent-framework/integrations/azure-functions) on Microsoft Learn.
|
||||
|
||||
## How durable agents work
|
||||
|
||||
Durable agents are implemented on top of [Durable Entities](https://learn.microsoft.com/azure/azure-functions/durable/durable-functions-entities) (also called "virtual actors"). Each **agent session** maps to one entity instance whose state contains the full conversation history. When you send a message to a durable agent, the following happens:
|
||||
|
||||
1. The message is dispatched to the entity identified by an `AgentSessionId` (a composite of the agent name and a unique session key).
|
||||
2. The entity loads its persisted `DurableAgentState`, which includes the complete conversation history.
|
||||
3. The entity invokes the underlying `AIAgent` with the full conversation history, collects the response, and appends both the request and the response to the state.
|
||||
4. The updated state is persisted back to durable storage automatically.
|
||||
|
||||
Because the entity framework serializes access to each entity instance, concurrent messages to the same session are processed one at a time, eliminating race conditions.
|
||||
|
||||
### Agent session identity
|
||||
|
||||
Every durable agent session is identified by an `AgentSessionId`, which has two components:
|
||||
|
||||
- **Name** – the registered name of the agent (case-insensitive).
|
||||
- **Key** – a unique session key (case-sensitive), typically a GUID.
|
||||
|
||||
The session ID is mapped to an underlying Durable Task entity ID with a `dafx-` prefix (e.g., `dafx-joker`). This naming convention is consistent across both .NET and Python implementations.
|
||||
|
||||
## Architecture
|
||||
|
||||
### .NET
|
||||
|
||||
The .NET implementation consists of two NuGet packages:
|
||||
|
||||
| Package | Purpose |
|
||||
| --- | --- |
|
||||
| `Microsoft.Agents.AI.DurableTask` | Core durable agent types: `DurableAIAgent`, `AgentEntity`, `DurableAgentSession`, `AgentSessionId`, `DurableAgentsOptions`, and the state model. |
|
||||
| `Microsoft.Agents.AI.Hosting.AzureFunctions` | Azure Functions hosting integration: auto-generated HTTP endpoints, MCP tool triggers, entity function triggers, and the `ConfigureDurableAgents` extension method on `FunctionsApplicationBuilder`. |
|
||||
|
||||
Key types:
|
||||
|
||||
- **`DurableAIAgent`** – A subclass of `AIAgent` used *inside orchestrations*. Obtained via `context.GetAgent("agentName")`, it routes `RunAsync` calls through the orchestration's entity APIs so that each call is checkpointed.
|
||||
- **`DurableAIAgentProxy`** – A subclass of `AIAgent` used *outside orchestrations* (e.g., from HTTP triggers or console apps). It signals the entity via `DurableTaskClient` and polls for the response.
|
||||
- **`AgentEntity`** – The `TaskEntity<DurableAgentState>` that hosts the real agent. It loads the registered `AIAgent` by name, wraps it in an `EntityAgentWrapper`, feeds it the full conversation history, and persists the result.
|
||||
- **`DurableAgentSession`** – An `AgentSession` subclass that carries the `AgentSessionId`.
|
||||
- **`DurableAgentsOptions`** – Builder for registering agents and configuring TTL.
|
||||
|
||||
### Python
|
||||
|
||||
The core Python implementation is in the `agent-framework-durabletask` package (`python/packages/durabletask`). Azure Functions hosting (including `AgentFunctionApp`) is in the separate `agent-framework-azurefunctions` package (`python/packages/azurefunctions`).
|
||||
|
||||
Key types:
|
||||
|
||||
- **`DurableAIAgent`** – A generic proxy (`DurableAIAgent[TaskT]`) implementing `SupportsAgentRun`. Returns a `TaskT` from `run()` — either an `AgentResponse` (client context) or a `DurableAgentTask` (orchestration context, must be `yield`ed).
|
||||
- **`DurableAIAgentWorker`** – Wraps a `TaskHubGrpcWorker` and registers agents as durable entities via `add_agent()`.
|
||||
- **`DurableAIAgentClient`** – Wraps a `TaskHubGrpcClient` for external callers. `get_agent()` returns a `DurableAIAgent[AgentResponse]`.
|
||||
- **`DurableAIAgentOrchestrationContext`** – Wraps an `OrchestrationContext` for use inside orchestrations. `get_agent()` returns a `DurableAIAgent[DurableAgentTask]`.
|
||||
- **`AgentEntity`** – Platform-agnostic agent execution logic that manages state, invokes the agent, handles streaming, and calls response callbacks.
|
||||
|
||||
## Hosting models
|
||||
|
||||
### Azure Functions
|
||||
|
||||
The recommended production hosting model. A single call to `ConfigureDurableAgents` (C#) or `AgentFunctionApp` (Python) automatically:
|
||||
|
||||
- Registers agent entities with the Durable Task worker.
|
||||
- Generates HTTP endpoints at `/api/agents/{agentName}/run` for each registered agent.
|
||||
- Supports `thread_id` query parameter / JSON field and the `x-ms-thread-id` response header for session continuity.
|
||||
- Supports fire-and-forget via the `x-ms-wait-for-response: false` header (returns HTTP 202).
|
||||
- Optionally exposes agents as MCP tools.
|
||||
|
||||
**C# example:**
|
||||
|
||||
```csharp
|
||||
using IHost app = FunctionsApplication
|
||||
.CreateBuilder(args)
|
||||
.ConfigureFunctionsWebApplication()
|
||||
.ConfigureDurableAgents(options => options.AddAIAgent(agent))
|
||||
.Build();
|
||||
app.Run();
|
||||
```
|
||||
|
||||
**Python example:**
|
||||
|
||||
```python
|
||||
app = AgentFunctionApp(agents=[agent])
|
||||
```
|
||||
|
||||
### Console apps / generic hosts
|
||||
|
||||
For self-hosted or non-serverless scenarios, register durable agents via `IServiceCollection.ConfigureDurableAgents` (.NET) or `DurableAIAgentWorker` (Python) with explicit Durable Task worker and client configuration.
|
||||
|
||||
**C# example:**
|
||||
|
||||
```csharp
|
||||
IHost host = Host.CreateDefaultBuilder(args)
|
||||
.ConfigureServices(services =>
|
||||
{
|
||||
services.ConfigureDurableAgents(
|
||||
options => options.AddAIAgent(agent),
|
||||
workerBuilder: b => b.UseDurableTaskScheduler(connectionString),
|
||||
clientBuilder: b => b.UseDurableTaskScheduler(connectionString));
|
||||
})
|
||||
.Build();
|
||||
```
|
||||
|
||||
**Python example:**
|
||||
|
||||
```python
|
||||
worker = DurableAIAgentWorker(TaskHubGrpcWorker(host_address="localhost:4001"))
|
||||
worker.add_agent(agent)
|
||||
worker.start()
|
||||
```
|
||||
|
||||
## Deterministic multi-agent orchestrations
|
||||
|
||||
Durable agents can be composed into deterministic, checkpointed workflows using Durable Task orchestrations. The orchestration framework replays orchestrator code on failure, so completed agent calls are not re-executed.
|
||||
|
||||
### Patterns
|
||||
|
||||
| Pattern | Description |
|
||||
| --- | --- |
|
||||
| **Sequential (chaining)** | Call agents one after another, passing outputs forward. |
|
||||
| **Parallel (fan-out/fan-in)** | Run multiple agents concurrently and aggregate results. |
|
||||
| **Conditional** | Branch orchestration logic based on structured agent output. |
|
||||
| **Human-in-the-loop** | Pause for external events (approvals, feedback) with optional timeouts. |
|
||||
|
||||
### Using agents in orchestrations
|
||||
|
||||
Inside an orchestration function, obtain a `DurableAIAgent` via the orchestration context. Each agent gets its own session (created with `CreateSessionAsync` / `create_session`), and you can call the same agent multiple times on the same session to maintain conversation context across sequential invocations.
|
||||
|
||||
**C#:**
|
||||
|
||||
```csharp
|
||||
static async Task<string> WritingOrchestration(TaskOrchestrationContext context)
|
||||
{
|
||||
// Get a durable agent reference — works in any host (console app, Azure Functions, etc.)
|
||||
DurableAIAgent writer = context.GetAgent("WriterAgent");
|
||||
|
||||
// Create a session to maintain conversation context across multiple calls
|
||||
AgentSession session = await writer.CreateSessionAsync();
|
||||
|
||||
// First call: generate an initial draft
|
||||
AgentResponse<TextResponse> draft = await writer.RunAsync<TextResponse>(
|
||||
message: "Write a concise inspirational sentence about learning.",
|
||||
session: session);
|
||||
|
||||
// Second call: refine the draft — the agent sees the full conversation history
|
||||
AgentResponse<TextResponse> refined = await writer.RunAsync<TextResponse>(
|
||||
message: $"Improve this further while keeping it under 25 words: {draft.Result.Text}",
|
||||
session: session);
|
||||
|
||||
return refined.Result.Text;
|
||||
}
|
||||
```
|
||||
|
||||
**Python:**
|
||||
|
||||
```python
|
||||
def writing_orchestration(context, _):
|
||||
agent_ctx = DurableAIAgentOrchestrationContext(context)
|
||||
|
||||
# Get a durable agent reference — works in any host (standalone worker, Azure Functions, etc.)
|
||||
writer = agent_ctx.get_agent("WriterAgent")
|
||||
|
||||
# Create a session to maintain conversation context across multiple calls
|
||||
session = writer.create_session()
|
||||
|
||||
# First call: generate an initial draft
|
||||
draft = yield writer.run(
|
||||
messages="Write a concise inspirational sentence about learning.",
|
||||
session=session,
|
||||
)
|
||||
|
||||
# Second call: refine the draft — the agent sees the full conversation history
|
||||
refined = yield writer.run(
|
||||
messages=f"Improve this further while keeping it under 25 words: {draft.text}",
|
||||
session=session,
|
||||
)
|
||||
|
||||
return refined.text
|
||||
```
|
||||
|
||||
> [!IMPORTANT]
|
||||
> In .NET, `DurableAIAgent.RunAsync<T>` deliberately avoids `ConfigureAwait(false)` because the Durable Task Framework uses a custom synchronization context — all continuations must run on the orchestration thread.
|
||||
|
||||
## Streaming and response callbacks
|
||||
|
||||
Durable agents do not support true end-to-end streaming because entity operations are request/response. However, **reliable streaming** is supported via response callbacks:
|
||||
|
||||
- **`IAgentResponseHandler`** (.NET) or **`AgentResponseCallbackProtocol`** (Python) – Implement this interface to receive streaming updates as the underlying agent generates them (e.g., push tokens to a Redis Stream for client consumption).
|
||||
- The entity still returns the complete `AgentResponse` after the stream is fully consumed.
|
||||
- Clients can reconnect and resume reading from a cursor-based stream (e.g., Redis Streams) without losing messages.
|
||||
|
||||
See the **Reliable Streaming** samples for a complete implementation using Redis Streams.
|
||||
|
||||
## Session TTL (Time-To-Live)
|
||||
|
||||
Durable agent sessions support automatic cleanup via configurable TTL. See [Session TTL](durable-agents-ttl.md) for details on configuration, behavior, and best practices.
|
||||
|
||||
## Observability
|
||||
|
||||
When using the [Durable Task Scheduler](https://learn.microsoft.com/azure/azure-functions/durable/durable-task-scheduler/durable-task-scheduler) as the durable backend, you get built-in observability through its dashboard:
|
||||
|
||||
- **Conversation history** – View complete chat history for each agent session.
|
||||
- **Orchestration visualization** – See multi-agent execution flows, including parallel branches and conditional logic.
|
||||
- **Performance metrics** – Monitor agent response times, token usage, and orchestration duration.
|
||||
- **Debugging** – Trace tool invocations and external event handling.
|
||||
|
||||
## Samples
|
||||
|
||||
- **.NET** – [Console app samples](../../../dotnet/samples/04-hosting/DurableAgents/ConsoleApps/) and [Azure Functions samples](../../../dotnet/samples/04-hosting/DurableAgents/AzureFunctions/) covering single-agent, chaining, concurrency, conditionals, human-in-the-loop, long-running tools, MCP tool exposure, and reliable streaming.
|
||||
- **Python** – [Durable Task samples](../../../python/samples/04-hosting/durabletask/) covering single-agent, multi-agent, streaming, chaining, concurrency, conditionals, and human-in-the-loop.
|
||||
|
||||
## Packages
|
||||
|
||||
| Language | Package | Source |
|
||||
| --- | --- | --- |
|
||||
| .NET | `Microsoft.Agents.AI.DurableTask` | [`dotnet/src/Microsoft.Agents.AI.DurableTask`](../../../dotnet/src/Microsoft.Agents.AI.DurableTask) |
|
||||
| .NET | `Microsoft.Agents.AI.Hosting.AzureFunctions` | [`dotnet/src/Microsoft.Agents.AI.Hosting.AzureFunctions`](../../../dotnet/src/Microsoft.Agents.AI.Hosting.AzureFunctions) |
|
||||
| Python | `agent-framework-durabletask` | [`python/packages/durabletask`](../../../python/packages/durabletask) |
|
||||
| Python | `agent-framework-azurefunctions` | [`python/packages/azurefunctions`](../../../python/packages/azurefunctions) |
|
||||
|
||||
## Further reading
|
||||
|
||||
- [Azure Functions (Durable) — Microsoft Learn](https://learn.microsoft.com/agent-framework/integrations/azure-functions)
|
||||
- [Durable Task Scheduler](https://learn.microsoft.com/azure/azure-functions/durable/durable-task-scheduler/durable-task-scheduler)
|
||||
- [Durable Entities](https://learn.microsoft.com/azure/azure-functions/durable/durable-functions-entities)
|
||||
- [Session TTL](durable-agents-ttl.md)
|
||||
@@ -0,0 +1,147 @@
|
||||
# Time-To-Live (TTL) for durable agent sessions
|
||||
|
||||
## Overview
|
||||
|
||||
The durable agents automatically maintain conversation history and state for each session. Without automatic cleanup, this state can accumulate indefinitely, consuming storage resources and increasing costs. The Time-To-Live (TTL) feature provides automatic cleanup of idle agent sessions, ensuring that sessions are automatically deleted after a period of inactivity.
|
||||
|
||||
## What is TTL?
|
||||
|
||||
Time-To-Live (TTL) is a configurable duration that determines how long an agent session state will be retained after its last interaction. When an agent session is idle (no messages sent to it) for longer than the TTL period, the session state is automatically deleted. Each new interaction with an agent resets the TTL timer, extending the session's lifetime.
|
||||
|
||||
## Benefits
|
||||
|
||||
- **Automatic cleanup**: No manual intervention required to clean up idle agent sessions
|
||||
- **Cost optimization**: Reduces storage costs by automatically removing unused session state
|
||||
- **Resource management**: Prevents unbounded growth of agent session state in storage
|
||||
- **Configurable**: Set TTL globally or per-agent type to match your application's needs
|
||||
|
||||
## Configuration
|
||||
|
||||
TTL can be configured at two levels:
|
||||
|
||||
1. **Global default TTL**: Applies to all agent sessions unless overridden
|
||||
2. **Per-agent type TTL**: Overrides the global default for specific agent types
|
||||
|
||||
Additionally, you can configure a **minimum deletion delay** that controls how frequently deletion operations are scheduled. The default value is 5 minutes, and the maximum allowed value is also 5 minutes.
|
||||
|
||||
> [!NOTE]
|
||||
> Reducing the minimum deletion delay below 5 minutes can be useful for testing or for ensuring rapid cleanup of short-lived agent sessions. However, this can also increase the load on the system and should be used with caution.
|
||||
|
||||
### Default values
|
||||
|
||||
- **Default TTL**: 14 days
|
||||
- **Minimum TTL deletion delay**: 5 minutes (maximum allowed value, subject to change in future releases)
|
||||
|
||||
### Configuration examples
|
||||
|
||||
#### .NET
|
||||
|
||||
```csharp
|
||||
// Configure global default TTL and minimum signal delay
|
||||
services.ConfigureDurableAgents(
|
||||
options =>
|
||||
{
|
||||
// Set global default TTL to 7 days
|
||||
options.DefaultTimeToLive = TimeSpan.FromDays(7);
|
||||
|
||||
// Add agents (will use global default TTL)
|
||||
options.AddAIAgent(myAgent);
|
||||
});
|
||||
|
||||
// Configure per-agent TTL
|
||||
services.ConfigureDurableAgents(
|
||||
options =>
|
||||
{
|
||||
options.DefaultTimeToLive = TimeSpan.FromDays(14); // Global default
|
||||
|
||||
// Agent with custom TTL of 1 day
|
||||
options.AddAIAgent(shortLivedAgent, timeToLive: TimeSpan.FromDays(1));
|
||||
|
||||
// Agent with custom TTL of 90 days
|
||||
options.AddAIAgent(longLivedAgent, timeToLive: TimeSpan.FromDays(90));
|
||||
|
||||
// Agent using global default (14 days)
|
||||
options.AddAIAgent(defaultAgent);
|
||||
});
|
||||
|
||||
// Disable TTL for specific agents by setting TTL to null
|
||||
services.ConfigureDurableAgents(
|
||||
options =>
|
||||
{
|
||||
options.DefaultTimeToLive = TimeSpan.FromDays(14);
|
||||
|
||||
// Agent with no TTL (never expires)
|
||||
options.AddAIAgent(permanentAgent, timeToLive: null);
|
||||
});
|
||||
```
|
||||
|
||||
## How TTL works
|
||||
|
||||
The following sections describe how TTL works in detail.
|
||||
|
||||
### Expiration tracking
|
||||
|
||||
Each agent session maintains an expiration timestamp in its internally managed state that is updated whenever the session processes a message:
|
||||
|
||||
1. When a message is sent to an agent session, the expiration time is set to `current time + TTL`
|
||||
2. The runtime schedules a delete operation for the expiration time (subject to minimum delay constraints)
|
||||
3. When the delete operation runs, if the current time is past the expiration time, the session state is deleted. Otherwise, the delete operation is rescheduled for the next expiration time.
|
||||
|
||||
### State deletion
|
||||
|
||||
When an agent session expires, its entire state is deleted, including:
|
||||
|
||||
- Conversation history
|
||||
- Any custom state data
|
||||
- Expiration timestamps
|
||||
|
||||
After deletion, if a message is sent to the same agent session, a new session is created with a fresh conversation history.
|
||||
|
||||
## Behavior examples
|
||||
|
||||
The following examples illustrate how TTL works in different scenarios.
|
||||
|
||||
### Example 1: Agent session expires after TTL
|
||||
|
||||
1. Agent configured with 30-day TTL
|
||||
2. User sends message at Day 0 → agent session created, expiration set to Day 30
|
||||
3. No further messages sent
|
||||
4. At Day 30 → Agent session is deleted
|
||||
5. User sends message at Day 31 → New agent session created with fresh conversation history
|
||||
|
||||
### Example 2: TTL reset on interaction
|
||||
|
||||
1. Agent configured with 30-day TTL
|
||||
2. User sends message at Day 0 → agent session created, expiration set to Day 30
|
||||
3. User sends message at Day 15 → Expiration reset to Day 45
|
||||
4. User sends message at Day 40 → Expiration reset to Day 70
|
||||
5. Agent session remains active as long as there are regular interactions
|
||||
|
||||
## Logging
|
||||
|
||||
The TTL feature includes comprehensive logging to track state changes:
|
||||
|
||||
- **Expiration time updated**: Logged when TTL expiration time is set or updated
|
||||
- **Deletion scheduled**: Logged when a deletion check signal is scheduled
|
||||
- **Deletion check**: Logged when a deletion check operation runs
|
||||
- **Session expired**: Logged when an agent session is deleted due to expiration
|
||||
- **TTL rescheduled**: Logged when a deletion signal is rescheduled
|
||||
|
||||
These logs help monitor TTL behavior and troubleshoot any issues.
|
||||
|
||||
## Best practices
|
||||
|
||||
1. **Choose appropriate TTL values**: Balance between storage costs and user experience. Too short TTLs may delete active sessions, while too long TTLs may accumulate unnecessary state.
|
||||
|
||||
2. **Use per-agent TTLs**: Different agents may have different usage patterns. Configure TTLs per-agent based on expected session lifetimes.
|
||||
|
||||
3. **Monitor expiration logs**: Review logs to understand TTL behavior and adjust configuration as needed.
|
||||
|
||||
4. **Test with short TTLs**: During development, use short TTLs (e.g., minutes) to verify TTL behavior without waiting for long periods.
|
||||
|
||||
## Limitations
|
||||
|
||||
- TTL is based on wall-clock time, not activity time. The expiration timer starts from the last message timestamp.
|
||||
- Deletion checks are durably scheduled operations and may have slight delays depending on system load.
|
||||
- Once an agent session is deleted, its conversation history cannot be recovered.
|
||||
- TTL deletion requires at least one worker to be available to process the deletion operation message.
|
||||
@@ -0,0 +1,390 @@
|
||||
# Vector Stores and Embeddings
|
||||
|
||||
## Overview
|
||||
|
||||
This feature ports the vector store abstractions, embedding generator abstractions, and their implementations from Semantic Kernel into Agent Framework. The ported code follows AF's coding standards, feels native to AF, and is structured to allow data models/schemas to be reusable across both frameworks. The embedding abstraction combines the best of SK's `EmbeddingGeneratorBase` and MEAI's `IEmbeddingGenerator<TInput, TEmbedding>`.
|
||||
|
||||
| Capability | Description |
|
||||
| --- | --- |
|
||||
| Embedding generation | Generic embedding client abstraction supporting text, image, and audio inputs |
|
||||
| Vector store collections | CRUD operations on vector store collections (upsert, get, delete) |
|
||||
| Vector search | Unified search interface with `search_type` parameter (`"vector"`, `"keyword_hybrid"`) |
|
||||
| Data model decorator | `@vectorstoremodel` decorator for defining vector store data models (supports Pydantic, dataclasses, plain classes, dicts) |
|
||||
| Agent tools | `create_search_tool`, `create_upsert_tool`, `create_get_tool`, `create_delete_tool` for agent-usable vector store operations |
|
||||
| In-memory store | Zero-dependency vector store for testing and development |
|
||||
| 13+ connectors | Azure AI Search, Qdrant, Redis, PostgreSQL, MongoDB, Cosmos DB, Pinecone, Chroma, Weaviate, Oracle, SQL Server, FAISS |
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
### Embedding Abstractions (combining SK + MEAI)
|
||||
- **Both Protocol and Base class** (matching AF's `SupportsChatGetResponse` + `BaseChatClient` pattern):
|
||||
- `SupportsGetEmbeddings` — Protocol for duck-typing
|
||||
- `BaseEmbeddingClient` — ABC base class for implementations (similar to `BaseChatClient`)
|
||||
- **Generic input type** (`EmbeddingInputT`, default `str`) from MEAI — allows image/audio embeddings in the future
|
||||
- **Generic output type** (`EmbeddingT`, default `list[float]`) from MEAI — supports `list[float]`, `list[int]`, `bytes`, etc.
|
||||
- **Generic order**: `[EmbeddingInputT, EmbeddingT, EmbeddingOptionsT]` — options last, matching MEAI's `IEmbeddingGenerator<TInput, TEmbedding>` with options appended
|
||||
- **TypeVar naming convention**: Use `SuffixT` per AF standard (e.g., `EmbeddingInputT`, `EmbeddingT`, `ModelT`, `KeyT`)
|
||||
- `EmbeddingGenerationOptions` TypedDict (inspired by MEAI, matching AF's `ChatOptions` pattern) — `total=False`, includes `dimensions`, `model_id`. No `additional_properties` since each implementation extends with its own fields.
|
||||
- Protocol and base class are generic over input, output, and options: `SupportsGetEmbeddings[EmbeddingInputT, EmbeddingT, OptionsContraT]`, `BaseEmbeddingClient[EmbeddingInputT, EmbeddingT, OptionsCoT]`
|
||||
- **`Embedding[EmbeddingT]` type** in `_types.py` — a lightweight generic class (not Pydantic) with `vector: EmbeddingT`, `model_id: str | None`, `dimensions: int | None` (explicit or computed from vector), `created_at: datetime | None`, `additional_properties: dict[str, Any]`
|
||||
- **`GeneratedEmbeddings[EmbeddingT, EmbeddingOptionsT]` type** — a list-like container of `Embedding[EmbeddingT]` objects with `options: EmbeddingOptionsT | None` (stores the options used to generate), `usage: dict[str, Any] | None`, `additional_properties: dict[str, Any]`
|
||||
- **No numpy dependency** — return `list[float]` by default; users cast as needed
|
||||
|
||||
### Vector Store Abstractions
|
||||
- **Port core abstractions without Pydantic for internal classes** — use plain classes
|
||||
- **Both Protocol and Base class** for vector store operations (matching AF pattern):
|
||||
- `SupportsVectorUpsert` / `SupportsVectorSearch` — Protocols for duck-typing (follows `Supports<Capability>` naming convention)
|
||||
- `BaseVectorCollection` / `BaseVectorSearch` — ABC base classes for implementations
|
||||
- `BaseVectorStore` — ABC base class for store operations (factory for collections, no protocol needed)
|
||||
- **TypeVar naming convention**: `ModelT`, `KeyT`, `FilterT` (suffix T, per AF standard)
|
||||
- **Support Pydantic for user-facing data models** — the `@vectorstoremodel` decorator and `VectorStoreCollectionDefinition` should work with Pydantic models, dataclasses, plain classes, and dicts
|
||||
- **Remove SK-specific dependencies** — no `KernelBaseModel`, `KernelFunction`, `KernelParameterMetadata`, `kernel_function`, `PromptExecutionSettings`
|
||||
- **Embedding types in `_types.py`**, embedding protocol/base class in `_clients.py`
|
||||
- **All vector store specific types, enums, protocols, base classes** in `_vectors.py`
|
||||
- **Error handling** uses AF's exception hierarchy (e.g., `IntegrationException` variants)
|
||||
|
||||
### Package Structure
|
||||
- **Embedding types** (`Embedding`, `GeneratedEmbeddings`, `EmbeddingGenerationOptions`) in `agent_framework/_types.py`
|
||||
- **Embedding protocol + base class** (`SupportsGetEmbeddings`, `BaseEmbeddingClient`) in `agent_framework/_clients.py`
|
||||
- **All vector store specific code** in a new `agent_framework/_vectors.py` module — this includes:
|
||||
- Enums: `FieldTypes`, `IndexKind`, `DistanceFunction`
|
||||
- `VectorStoreField`, `VectorStoreCollectionDefinition`
|
||||
- `SearchOptions`, `SearchResponse`, `RecordFilterOptions`
|
||||
- `@vectorstoremodel` decorator
|
||||
- Serialization/deserialization protocols
|
||||
- `VectorStoreRecordHandler`, `BaseVectorCollection`, `BaseVectorStore`, `BaseVectorSearch`
|
||||
- `SupportsVectorUpsert`, `SupportsVectorSearch` protocols
|
||||
- **OpenAI embeddings** in `agent_framework/openai/` (built into core, like OpenAI chat)
|
||||
- **Azure OpenAI embeddings** in `agent_framework/azure/` (built into core, follows `AzureOpenAIChatClient` pattern)
|
||||
- **Each vector store connector** in its own AF package under `packages/`
|
||||
- **In-memory store** in core (no external deps)
|
||||
- **TextSearch and its implementations** (Brave, Google) — last phase, separate work
|
||||
|
||||
## Naming: SK → AF
|
||||
|
||||
### Names that change
|
||||
|
||||
| SK Name | AF Name | Rationale |
|
||||
|---------|---------|-----------|
|
||||
| `VectorStoreCollection` | `BaseVectorCollection` | Drop redundant `Store`, add `Base` prefix per AF pattern |
|
||||
| `VectorStore` | `BaseVectorStore` | Add `Base` prefix per AF pattern |
|
||||
| `VectorSearch` | `BaseVectorSearch` | Add `Base` prefix per AF pattern |
|
||||
| `VectorSearchOptions` | `SearchOptions` | Shorter — context is already vector search |
|
||||
| `VectorSearchResult` | `SearchResponse` | Align with `ChatResponse`/`AgentResponse` |
|
||||
| `GetFilteredRecordOptions` | `RecordFilterOptions` | Shorter, more natural |
|
||||
| `EmbeddingGeneratorBase` | `BaseEmbeddingClient` | Matches AF `BaseChatClient` pattern |
|
||||
| `VectorStoreCollectionProtocol` | `SupportsVectorUpsert` | AF `Supports*` naming convention |
|
||||
| `VectorSearchProtocol` | `SupportsVectorSearch` | AF `Supports*` naming convention |
|
||||
| `__kernel_vectorstoremodel__` | `__vectorstoremodel__` | Drop SK `kernel` prefix |
|
||||
| `__kernel_vectorstoremodel_definition__` | `__vectorstoremodel_definition__` | Drop SK `kernel` prefix |
|
||||
| `search()` + `hybrid_search()` | `search(search_type=...)` | Single method with `Literal` parameter |
|
||||
| `SearchType` enum | `Literal["vector", "keyword_hybrid"]` | No enum, just a literal |
|
||||
| `KernelSearchResults` | `SearchResults` | Drop SK `Kernel` prefix (plural — container of `SearchResponse` items) |
|
||||
|
||||
### Names that stay the same
|
||||
|
||||
| Name | Location |
|
||||
|------|----------|
|
||||
| `@vectorstoremodel` | `_vectors.py` |
|
||||
| `VectorStoreField` | `_vectors.py` |
|
||||
| `VectorStoreCollectionDefinition` | `_vectors.py` |
|
||||
| `VectorStoreRecordHandler` | `_vectors.py` |
|
||||
| `FieldTypes` | `_vectors.py` |
|
||||
| `IndexKind` | `_vectors.py` |
|
||||
| `DistanceFunction` | `_vectors.py` |
|
||||
| `DISTANCE_FUNCTION_DIRECTION_HELPER` | `_vectors.py` |
|
||||
| `Embedding` | `_types.py` |
|
||||
| `GeneratedEmbeddings` | `_types.py` |
|
||||
| `EmbeddingGenerationOptions` | `_types.py` |
|
||||
| `SupportsGetEmbeddings` | `_clients.py` |
|
||||
|
||||
### New AF-only names (no SK equivalent)
|
||||
|
||||
| Name | Location | Purpose |
|
||||
|------|----------|---------|
|
||||
| `BaseEmbeddingClient` | `_clients.py` | ABC base for embedding implementations |
|
||||
| `EmbeddingInputT` | `_types.py` | TypeVar for generic embedding input (default `str`) |
|
||||
| `EmbeddingTelemetryLayer` | `observability.py` | MRO-based OTel tracing for embeddings |
|
||||
| `SupportsVectorUpsert` | `_vectors.py` | Protocol for collection CRUD |
|
||||
| `SupportsVectorSearch` | `_vectors.py` | Protocol for vector search |
|
||||
| `create_search_tool` | `_vectors.py` | Creates AF `FunctionTool` from vector search |
|
||||
|
||||
## Source Files Reference (SK → AF mapping)
|
||||
|
||||
### SK Source Files
|
||||
| SK File | Lines | Content |
|
||||
|---------|-------|---------|
|
||||
| `data/vector.py` | 2369 | All vector store abstractions, enums, decorator, search |
|
||||
| `data/_shared.py` | 184 | SearchOptions, KernelSearchResults, shared search types |
|
||||
| `data/text_search.py` | 349 | TextSearch base, TextSearchResult |
|
||||
| `connectors/ai/embedding_generator_base.py` | 50 | EmbeddingGeneratorBase ABC |
|
||||
| `connectors/in_memory.py` | 520 | InMemoryCollection, InMemoryStore |
|
||||
| `connectors/azure_ai_search.py` | 793 | Azure AI Search collection + store |
|
||||
| `connectors/azure_cosmos_db.py` | 1104 | Cosmos DB (Mongo + NoSQL) |
|
||||
| `connectors/redis.py` | 845 | Redis (Hashset + JSON) |
|
||||
| `connectors/qdrant.py` | 653 | Qdrant collection + store |
|
||||
| `connectors/postgres.py` | 987 | PostgreSQL collection + store |
|
||||
| `connectors/mongodb.py` | 633 | MongoDB Atlas collection + store |
|
||||
| `connectors/pinecone.py` | 691 | Pinecone collection + store |
|
||||
| `connectors/chroma.py` | 484 | Chroma collection + store |
|
||||
| `connectors/faiss.py` | 278 | FAISS (extends InMemory) |
|
||||
| `connectors/weaviate.py` | 804 | Weaviate collection + store |
|
||||
| `connectors/oracle.py` | 1267 | Oracle collection + store |
|
||||
| `connectors/sql_server.py` | 1132 | SQL Server collection + store |
|
||||
| `connectors/ai/open_ai/services/open_ai_text_embedding.py` | 91 | OpenAI embedding impl |
|
||||
| `connectors/ai/open_ai/services/open_ai_text_embedding_base.py` | 78 | OpenAI embedding base |
|
||||
| `connectors/brave.py` | ~200 | Brave TextSearch impl |
|
||||
| `connectors/google_search.py` | ~200 | Google TextSearch impl |
|
||||
|
||||
---
|
||||
|
||||
## Implementation Phases
|
||||
|
||||
### Phase 1: Core Embedding Abstractions & OpenAI Implementation ✅ DONE
|
||||
**Goal:** Establish the embedding generator abstraction and ship one working implementation.
|
||||
**Mergeable:** Yes — adds new types/protocols, no breaking changes.
|
||||
**Status:** Merged via PR #4153. Closes sub-issue #4163.
|
||||
|
||||
#### 1.1 — Embedding types in `_types.py`
|
||||
- `EmbeddingInputT` TypeVar (default `str`) — generic input type for embedding generation
|
||||
- `EmbeddingT` TypeVar (default `list[float]`) — generic output embedding vector type
|
||||
- `Embedding[EmbeddingT]` generic class: `vector: EmbeddingT`, `model_id: str | None`, `dimensions: int | None` (explicit param or computed from vector length), `created_at: datetime | None`, `additional_properties: dict[str, Any]`
|
||||
- `GeneratedEmbeddings[EmbeddingT, EmbeddingOptionsT]` generic class: list-like container of `Embedding[EmbeddingT]` objects with `options: EmbeddingOptionsT | None` (the options used to generate), `usage: dict[str, Any] | None`, `additional_properties: dict[str, Any]`
|
||||
- `EmbeddingGenerationOptions` TypedDict (`total=False`): `dimensions: int`, `model_id: str` — follows the same pattern as `ChatOptions`. No `additional_properties` needed since it's a TypedDict and each implementation can extend with its own fields.
|
||||
|
||||
#### 1.2 — Embedding generator protocol + base class in `_clients.py`
|
||||
- `SupportsGetEmbeddings(Protocol[EmbeddingInputT, EmbeddingT, OptionsContraT])`: generic over input, output, and options (all with defaults), `get_embeddings(values: Sequence[EmbeddingInputT], *, options: OptionsContraT | None = None) -> Awaitable[GeneratedEmbeddings[EmbeddingT]]`
|
||||
- `BaseEmbeddingClient(ABC, Generic[EmbeddingInputT, EmbeddingT, OptionsCoT])`: ABC base class mirroring `BaseChatClient` pattern
|
||||
- `__init__` with `additional_properties`, etc.
|
||||
- Abstract `get_embeddings(...)` for subclasses to implement directly (no `_inner_*` indirection — simpler than chat, no middleware needed)
|
||||
- `EmbeddingTelemetryLayer` in `observability.py` — MRO-based telemetry (no closure), `gen_ai.operation.name = "embeddings"`
|
||||
|
||||
#### 1.3 — OpenAI embedding generator in `agent_framework/openai/` and `agent_framework/azure/`
|
||||
- `RawOpenAIEmbeddingClient` — implements `get_embeddings` via `_ensure_client()` factory
|
||||
- `OpenAIEmbeddingClient(OpenAIConfigMixin, EmbeddingTelemetryLayer[str, list[float], OptionsT], RawOpenAIEmbeddingClient[OptionsT])` — full client with config + telemetry layers
|
||||
- `OpenAIEmbeddingOptions(EmbeddingGenerationOptions)` — extends with `encoding_format`, `user`
|
||||
- `AzureOpenAIEmbeddingClient` in `agent_framework/azure/` — follows `AzureOpenAIChatClient` pattern with `AzureOpenAIConfigMixin`, `load_settings`, Entra ID credential support
|
||||
- `AzureOpenAISettings` extended with `embedding_deployment_name` (env var: `AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME`)
|
||||
|
||||
#### 1.4 — Tests and samples
|
||||
- Unit tests for types, protocol, base class, OpenAI client, Azure OpenAI client
|
||||
- Integration tests for OpenAI and Azure OpenAI (gated behind credentials check, `@pytest.mark.flaky`)
|
||||
- Samples in `samples/02-agents/embeddings/` — `openai_embeddings.py`, `azure_openai_embeddings.py`
|
||||
|
||||
---
|
||||
|
||||
### Phase 2: Embedding Generators for Existing Providers
|
||||
**Goal:** Add embedding generators to all existing AF provider packages that have chat clients.
|
||||
**Mergeable:** Yes — each is independent, added to existing provider packages.
|
||||
|
||||
#### 2.1 — Foundry inference embedding (in `packages/foundry/`)
|
||||
#### 2.2 — Ollama embedding (in `packages/ollama/`)
|
||||
#### 2.3 — Anthropic embedding (in `packages/anthropic/`)
|
||||
#### 2.4 — Bedrock embedding (in `packages/bedrock/`)
|
||||
|
||||
---
|
||||
|
||||
### Phase 3: Core Vector Store Abstractions
|
||||
**Goal:** Establish all vector store types, enums, the decorator, collection definition, and base classes.
|
||||
**Mergeable:** Yes — adds new abstractions, no breaking changes.
|
||||
|
||||
#### 3.1 — Vector store enums and field types in `_vectors.py`
|
||||
- `FieldTypes` enum: `KEY`, `VECTOR`, `DATA`
|
||||
- `IndexKind` enum: `HNSW`, `FLAT`, `IVF_FLAT`, `DISK_ANN`, `QUANTIZED_FLAT`, `DYNAMIC`, `DEFAULT`
|
||||
- `DistanceFunction` enum: `COSINE_SIMILARITY`, `COSINE_DISTANCE`, `DOT_PROD`, `EUCLIDEAN_DISTANCE`, `EUCLIDEAN_SQUARED_DISTANCE`, `MANHATTAN`, `HAMMING`, `DEFAULT`
|
||||
- No `SearchType` enum — use `Literal["vector", "keyword_hybrid"]` instead, per AF convention of avoiding unnecessary imports
|
||||
- `VectorStoreField` plain class (not Pydantic)
|
||||
- `VectorStoreCollectionDefinition` class (not Pydantic internally, but supports Pydantic models as input)
|
||||
- `SearchOptions` plain class — includes `score_threshold: float | None` for filtering results by score (see note below)
|
||||
- `SearchResponse` generic class
|
||||
- `RecordFilterOptions` plain class
|
||||
- `DISTANCE_FUNCTION_DIRECTION_HELPER` dict
|
||||
|
||||
#### 3.2 — `@vectorstoremodel` decorator
|
||||
- Port from SK, works with dataclasses, Pydantic models, plain classes, and dicts
|
||||
- Sets `__vectorstoremodel__` and `__vectorstoremodel_definition__` on the class
|
||||
- Remove SK-specific `kernel` prefix (`__kernel_vectorstoremodel__` → `__vectorstoremodel__`)
|
||||
|
||||
#### 3.3 — Serialization/deserialization protocols
|
||||
- `SerializeMethodProtocol`, `ToDictFunctionProtocol`, `FromDictFunctionProtocol`, etc.
|
||||
- Port the record handler logic but without Pydantic base class — use plain class or ABC
|
||||
|
||||
#### 3.4 — Vector store base classes in `_vectors.py`
|
||||
- `VectorStoreRecordHandler` — internal base class that handles serialization/deserialization between user data models and store-specific formats, plus embedding generation for vector fields. Both `BaseVectorCollection` and `BaseVectorSearch` extend this.
|
||||
- `BaseVectorCollection(VectorStoreRecordHandler)` — base for collections
|
||||
- Uses `SupportsGetEmbeddings` instead of `EmbeddingGeneratorBase`
|
||||
- Not a Pydantic model — use `__init__` with explicit params
|
||||
- `upsert`, `get`, `delete`, `ensure_collection_exists`, `collection_exists`, `ensure_collection_deleted`
|
||||
- Async context manager support
|
||||
- `BaseVectorStore` — base for stores
|
||||
- `get_collection`, `list_collection_names`, `collection_exists`, `ensure_collection_deleted`
|
||||
- Async context manager support
|
||||
|
||||
#### 3.5 — Vector search base class
|
||||
- `BaseVectorSearch(VectorStoreRecordHandler)` — base for vector search
|
||||
- Single `search(search_type=...)` method with `search_type: Literal["vector", "keyword_hybrid"]` parameter — no enum, just a literal
|
||||
- `_inner_search` abstract method for implementations
|
||||
- Filter building with lambda parser (AST-based)
|
||||
- Vector generation from values using embedding generator
|
||||
|
||||
#### 3.6 — Protocols for type checking
|
||||
- `SupportsVectorUpsert` — Protocol for upsert/get/delete operations
|
||||
- `SupportsVectorSearch` — Protocol for vector search (single `search()` with `search_type` parameter)
|
||||
- No separate `SupportsVectorHybridSearch` — search type is a parameter, not a separate capability
|
||||
- No protocol for `VectorStore` — it's a factory for collections, not a capability to duck-type against
|
||||
|
||||
#### 3.7 — Exception types
|
||||
- Add vector store exceptions under `IntegrationException` or create new branch
|
||||
- `VectorStoreException`, `VectorStoreOperationException`, `VectorSearchException`, `VectorStoreModelException`, etc.
|
||||
|
||||
#### 3.8 — `create_search_tool` on `BaseVectorSearch`
|
||||
- Method on `BaseVectorSearch` that creates an AF `FunctionTool` from the vector search
|
||||
- Wraps the single `search()` method, passing `search_type` parameter
|
||||
- Accepts: `name`, `description`, `search_type`, `top`, `skip`, `filter`, `string_mapper`
|
||||
- The tool takes a query string, vectorizes it, searches, and returns results as strings
|
||||
- Can also be a standalone factory function in `_vectors.py`
|
||||
|
||||
#### 3.9 — Tests for all vector store abstractions
|
||||
- Unit tests for enums, field types, collection definition
|
||||
- Unit tests for decorator
|
||||
- Unit tests for serialization/deserialization
|
||||
- Unit tests for record handler
|
||||
|
||||
---
|
||||
|
||||
### Phase 4: In-Memory Vector Store
|
||||
**Goal:** Provide a zero-dependency vector store for testing and development.
|
||||
**Mergeable:** Yes — first usable vector store.
|
||||
|
||||
#### 4.1 — Port `InMemoryCollection` and `InMemoryStore` into core
|
||||
- Place in `agent_framework/_vectors.py` (alongside the abstractions)
|
||||
- Supports vector search (cosine similarity, etc.)
|
||||
- No external dependencies
|
||||
|
||||
#### 4.2 — Port FAISS extension (optional, can be separate package)
|
||||
- Extends InMemory with FAISS indexing
|
||||
|
||||
#### 4.3 — Tests and sample code
|
||||
|
||||
---
|
||||
|
||||
### Phase 5: Vector Store Connectors — Tier 1 (High Priority)
|
||||
**Goal:** Ship the most commonly used vector store connectors.
|
||||
**Mergeable:** Yes — each connector is independent.
|
||||
|
||||
Each connector follows the AF package structure:
|
||||
- New package under `packages/`
|
||||
- Own `pyproject.toml`, `tests/`, lazy loading in core
|
||||
|
||||
#### 5.1 — Azure AI Search (`packages/azure-ai-search/`)
|
||||
- May extend existing package or be new
|
||||
- `AzureAISearchCollection`, `AzureAISearchStore`
|
||||
|
||||
#### 5.2 — Qdrant (`packages/qdrant/`)
|
||||
- New package
|
||||
- `QdrantCollection`, `QdrantStore`
|
||||
|
||||
#### 5.3 — Redis (`packages/redis/`)
|
||||
- May extend existing redis package
|
||||
- `RedisCollection` (JSON + Hashset variants), `RedisStore`
|
||||
|
||||
#### 5.4 — PostgreSQL/pgvector (`packages/postgres/`)
|
||||
- New package
|
||||
- `PostgresCollection`, `PostgresStore`
|
||||
|
||||
---
|
||||
|
||||
### Phase 6: Vector Store Connectors — Tier 2
|
||||
**Goal:** Ship remaining vector store connectors.
|
||||
**Mergeable:** Yes — each connector is independent.
|
||||
|
||||
#### 6.1 — MongoDB Atlas (`packages/mongodb/`)
|
||||
#### 6.2 — Azure Cosmos DB (`packages/azure-cosmos-db/`)
|
||||
- Cosmos Mongo + Cosmos NoSQL
|
||||
#### 6.3 — Pinecone (`packages/pinecone/`)
|
||||
#### 6.4 — Chroma (`packages/chroma/`)
|
||||
#### 6.5 — Weaviate (`packages/weaviate/`)
|
||||
|
||||
---
|
||||
|
||||
### Phase 7: Vector Store Connectors — Tier 3
|
||||
**Goal:** Ship niche or less common connectors.
|
||||
**Mergeable:** Yes — each connector is independent.
|
||||
|
||||
#### 7.1 — Oracle (`packages/oracle/`)
|
||||
#### 7.2 — SQL Server (`packages/sql-server/`)
|
||||
#### 7.3 — FAISS (`packages/faiss/` or in core extending InMemory)
|
||||
|
||||
> **Note:** When implementing any SQL-based connector (PostgreSQL, SQL Server, SQLite, Cosmos DB), review the .NET MEVD changes made by @roji (Shay Rojansky) in SK for design patterns, query building, filter translation, and feature parity: https://github.com/microsoft/semantic-kernel/pulls?q=is%3Apr+author%3Aroji+is%3Aclosed
|
||||
|
||||
---
|
||||
|
||||
### Phase 8: Vector Store CRUD Tools
|
||||
**Goal:** Provide a full set of agent-usable tools for CRUD operations on vector store collections.
|
||||
**Mergeable:** Yes — adds tools without changing existing APIs.
|
||||
|
||||
#### 8.1 — `create_upsert_tool` — tool for upserting records into a collection
|
||||
#### 8.2 — `create_get_tool` — tool for retrieving records by key
|
||||
- Key-based lookup only (by primary key), not a search tool
|
||||
- Documentation must clearly distinguish this from `create_search_tool`: get_tool retrieves specific records by their known key, while search_tool performs similarity/filtered search across the collection
|
||||
- Consider if this overlaps with filtered search and document when to use which
|
||||
#### 8.3 — `create_delete_tool` — tool for deleting records by key
|
||||
#### 8.4 — Tests and samples for CRUD tools
|
||||
|
||||
---
|
||||
|
||||
### Phase 9: Additional Embedding Implementations (New Providers)
|
||||
**Goal:** Provide embedding generators for providers that don't yet have AF packages.
|
||||
**Mergeable:** Yes — each is independent, new packages.
|
||||
|
||||
#### 9.1 — HuggingFace/ONNX embedding (new package or lab)
|
||||
#### 9.2 — Mistral AI embedding (new package)
|
||||
#### 9.3 — Google AI / Vertex AI embedding (new package)
|
||||
#### 9.4 — Nvidia embedding (new package)
|
||||
|
||||
---
|
||||
|
||||
### Phase 10: TextSearch Abstractions & Implementations (Separate Work)
|
||||
**Goal:** Port text search (non-vector) abstractions and implementations.
|
||||
**Mergeable:** Yes — independent of vector stores.
|
||||
|
||||
#### 10.1 — TextSearch base class and types
|
||||
- `SearchOptions`, `SearchResponse`, `TextSearchResult`
|
||||
- `TextSearch` base class with `search()` method
|
||||
- `create_search_function()` for kernel integration (may need AF equivalent)
|
||||
|
||||
#### 10.2 — Brave Search implementation
|
||||
#### 10.3 — Google Search implementation
|
||||
#### 10.4 — Vector store text search bridge (connecting VectorSearch to TextSearch interface)
|
||||
|
||||
---
|
||||
|
||||
## Key Considerations
|
||||
|
||||
1. **No Pydantic for internal classes**: All AF internal classes should use plain classes. Pydantic is only used for user-facing input validation (e.g., vector store data models).
|
||||
|
||||
2. **Protocol + Base class**: Follow AF's pattern of both a `Protocol` for duck-typing and a `Base` ABC for implementation, matching how `SupportsChatGetResponse` + `BaseChatClient` works.
|
||||
|
||||
3. **Exception hierarchy**: Use AF's `IntegrationException` branch for vector store operations, since vector stores are external dependencies.
|
||||
|
||||
4. **`from __future__ import annotations`**: Required in all files per AF coding standard.
|
||||
|
||||
5. **No `**kwargs` escape hatches in public APIs**: For user-facing interfaces, use explicit named parameters per AF coding standard. Internal implementation details (e.g., cooperative multiple inheritance / MRO patterns) may use `**kwargs` where necessary, as long as they are not exposed in public signatures.
|
||||
|
||||
6. **Lazy loading**: Connector packages use `__getattr__` lazy loading in core provider folders.
|
||||
|
||||
7. **Reusable data models**: The `@vectorstoremodel` decorator and `VectorStoreCollectionDefinition` should be agnostic enough to work with both SK and AF. The core types (`FieldTypes`, `IndexKind`, `DistanceFunction`, `VectorStoreField`) should be identical or easily mapped.
|
||||
|
||||
8. **`create_search_tool`**: The AF-native equivalent of SK's `create_search_function`. Instead of creating a `KernelFunction`, this creates an AF `FunctionTool` (via the `@tool` decorator pattern) from a vector search. This allows agents to use vector search as a tool during conversations. Design:
|
||||
- `create_search_tool(name, description, search_type, ...)` → returns a `FunctionTool` that wraps `VectorSearch.search(search_type=...)`
|
||||
- The tool accepts a query string, performs embedding + vector search, and returns results as strings
|
||||
- Supports configurable string mappers, filter functions, top/skip defaults
|
||||
- Lives in `_vectors.py` as a method on `BaseVectorSearch` and/or as a standalone factory function
|
||||
|
||||
9. **CRUD tools**: A full set of create/read/update/delete tools for vector store collections, allowing agents to manage data in vector stores. Design:
|
||||
- `create_upsert_tool(...)` → tool for upserting records
|
||||
- `create_get_tool(...)` → tool for retrieving records by key
|
||||
- `create_delete_tool(...)` → tool for deleting records
|
||||
- These are separate from search and are placed in a later phase
|
||||
|
||||
10. **Score threshold filtering**: `SearchOptions` includes `score_threshold: float | None` to filter search results by relevance score (ref: [SK .NET PR #13501](https://github.com/microsoft/semantic-kernel/pull/13501)). The semantics depend on the distance function: for similarity functions (cosine similarity, dot product), results *below* the threshold are filtered out; for distance functions (cosine distance, euclidean), results *above* the threshold are filtered out. Use `DISTANCE_FUNCTION_DIRECTION_HELPER` to determine direction. Connectors should implement this natively where the database supports it, falling back to client-side post-filtering otherwise.
|
||||
Reference in New Issue
Block a user