chore: import upstream snapshot with attribution
CodeQL / Analyze (csharp) (push) Waiting to run
CodeQL / Analyze (python) (push) Waiting to run
dotnet-build-and-test / dotnet-test-functions (push) Has been cancelled
dotnet-build-and-test / paths-filter (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Debug, windows-latest, net9.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Release, ubuntu-latest, net10.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Release, ubuntu-latest, net8.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Release, windows-latest, net472) (push) Has been cancelled
dotnet-build-and-test / dotnet-test (Release, integration, true, ubuntu-latest, net10.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-test (Release, integration, true, windows-latest, net472) (push) Has been cancelled
dotnet-build-and-test / dotnet-foundry-hosted-it (push) Has been cancelled
dotnet-build-and-test / dotnet-build-and-test-check (push) Has been cancelled
dotnet-build-and-test / Integration Test Report (push) Has been cancelled
CodeQL / Analyze (csharp) (push) Waiting to run
CodeQL / Analyze (python) (push) Waiting to run
dotnet-build-and-test / dotnet-test-functions (push) Has been cancelled
dotnet-build-and-test / paths-filter (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Debug, windows-latest, net9.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Release, ubuntu-latest, net10.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Release, ubuntu-latest, net8.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Release, windows-latest, net472) (push) Has been cancelled
dotnet-build-and-test / dotnet-test (Release, integration, true, ubuntu-latest, net10.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-test (Release, integration, true, windows-latest, net472) (push) Has been cancelled
dotnet-build-and-test / dotnet-foundry-hosted-it (push) Has been cancelled
dotnet-build-and-test / dotnet-build-and-test-check (push) Has been cancelled
dotnet-build-and-test / Integration Test Report (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,13 @@
|
||||
# Azure OpenAI Configuration
|
||||
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
|
||||
AZURE_OPENAI_MODEL=your-deployment-name
|
||||
# Optional: Use Azure CLI authentication if not provided
|
||||
# AZURE_OPENAI_API_KEY=your-api-key
|
||||
|
||||
# Durable Task Scheduler Configuration
|
||||
ENDPOINT=http://localhost:8080
|
||||
TASKHUB=default
|
||||
|
||||
# Redis Configuration (for streaming tests)
|
||||
REDIS_CONNECTION_STRING=redis://localhost:6379
|
||||
REDIS_STREAM_TTL_MINUTES=10
|
||||
@@ -0,0 +1,110 @@
|
||||
# Sample Integration Tests
|
||||
|
||||
Integration tests that validate the Durable Agent Framework samples by running them against a Durable Task Scheduler (DTS) instance.
|
||||
|
||||
## Setup
|
||||
|
||||
### 1. Create `.env` file
|
||||
|
||||
Copy `.env.example` to `.env` and fill in your Azure credentials:
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
Required variables:
|
||||
- `AZURE_OPENAI_ENDPOINT`
|
||||
- `AZURE_OPENAI_MODEL`
|
||||
- `AZURE_OPENAI_API_KEY` (optional if using Azure CLI authentication)
|
||||
- `ENDPOINT` (default: http://localhost:8080)
|
||||
- `TASKHUB` (default: default)
|
||||
|
||||
Optional variables (for streaming tests):
|
||||
- `REDIS_CONNECTION_STRING` (default: redis://localhost:6379)
|
||||
- `REDIS_STREAM_TTL_MINUTES` (default: 10)
|
||||
|
||||
### 2. Start required services
|
||||
|
||||
**Durable Task Scheduler:**
|
||||
```bash
|
||||
docker run -d --name dts-emulator -p 8080:8080 -p 8082:8082 mcr.microsoft.com/dts/dts-emulator:latest
|
||||
```
|
||||
- Port 8080: gRPC endpoint (used by tests)
|
||||
- Port 8082: Web dashboard (optional, for monitoring)
|
||||
|
||||
**Redis (for streaming tests):**
|
||||
```bash
|
||||
docker run -d --name redis -p 6379:6379 redis:latest
|
||||
```
|
||||
- Port 6379: Redis server endpoint
|
||||
|
||||
## Running Tests
|
||||
|
||||
The tests automatically start and stop worker processes for each sample.
|
||||
|
||||
### Run all sample tests
|
||||
```bash
|
||||
uv run pytest packages/durabletask/tests/integration_tests -v
|
||||
```
|
||||
|
||||
### Run specific sample
|
||||
```bash
|
||||
uv run pytest packages/durabletask/tests/integration_tests/test_01_single_agent.py -v
|
||||
```
|
||||
|
||||
### Run with verbose output
|
||||
```bash
|
||||
uv run pytest packages/durabletask/tests/integration_tests -sv
|
||||
```
|
||||
|
||||
## How It Works
|
||||
|
||||
Each test file uses pytest markers to automatically configure and start the worker process:
|
||||
|
||||
```python
|
||||
pytestmark = [
|
||||
pytest.mark.sample("03_single_agent_streaming"),
|
||||
pytest.mark.integration_test,
|
||||
pytest.mark.requires_azure_openai,
|
||||
pytest.mark.requires_dts,
|
||||
pytest.mark.requires_redis,
|
||||
]
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Tests are skipped:**
|
||||
Ensure the required environment variables (e.g., `AZURE_OPENAI_ENDPOINT`) are set in your `.env` file.
|
||||
|
||||
**DTS connection failed:**
|
||||
Check that the DTS emulator container is running: `docker ps | grep dts-emulator`
|
||||
|
||||
**Redis connection failed:**
|
||||
Check that Redis is running: `docker ps | grep redis`
|
||||
|
||||
**Missing environment variables:**
|
||||
Ensure your `.env` file contains all required variables from `.env.example`.
|
||||
|
||||
**Tests timeout:**
|
||||
Check that Azure OpenAI credentials are valid and the service is accessible.
|
||||
|
||||
If you see "DTS emulator is not available":
|
||||
- Ensure Docker container is running: `docker ps | grep dts-emulator`
|
||||
- Check port 8080 is not in use by another process
|
||||
- Restart the container if needed
|
||||
|
||||
### Azure OpenAI Errors
|
||||
|
||||
If you see authentication or deployment errors:
|
||||
- Verify your `AZURE_OPENAI_ENDPOINT` is correct
|
||||
- Confirm `AZURE_OPENAI_MODEL` matches your deployment
|
||||
- If using API key, check `AZURE_OPENAI_API_KEY` is valid
|
||||
- If using Azure CLI, ensure you're logged in: `az login`
|
||||
|
||||
## CI/CD
|
||||
|
||||
For automated testing in CI/CD pipelines:
|
||||
|
||||
1. Use Docker Compose to start DTS emulator
|
||||
2. Set environment variables via CI/CD secrets
|
||||
3. Run tests with appropriate markers: `pytest -m integration_test`
|
||||
@@ -0,0 +1,512 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
"""Pytest configuration and fixtures for durabletask integration tests."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import uuid
|
||||
from collections.abc import Generator
|
||||
from pathlib import Path
|
||||
from typing import Any, Protocol, cast
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import pytest
|
||||
import redis.asyncio as aioredis
|
||||
from dotenv import load_dotenv
|
||||
from durabletask.azuremanaged.client import DurableTaskSchedulerClient
|
||||
from durabletask.client import OrchestrationStatus
|
||||
|
||||
from agent_framework_durabletask import DurableAIAgentClient, DurableWorkflowClient
|
||||
|
||||
# Load environment variables from .env file
|
||||
load_dotenv(Path(__file__).parent / ".env")
|
||||
|
||||
# Configure logging to reduce noise during tests
|
||||
logging.basicConfig(level=logging.WARNING)
|
||||
|
||||
|
||||
class AgentClientFactoryProtocol(Protocol):
|
||||
"""Protocol for the agent client factory fixture."""
|
||||
|
||||
@classmethod
|
||||
def create(cls, max_poll_retries: int = 90) -> tuple[DurableTaskSchedulerClient, DurableAIAgentClient]: ...
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Environment and Service Checks
|
||||
# =============================================================================
|
||||
|
||||
|
||||
def _get_dts_endpoint() -> str:
|
||||
"""Get the DTS endpoint from environment or use default."""
|
||||
return os.getenv("ENDPOINT", "http://localhost:8080")
|
||||
|
||||
|
||||
def _check_dts_available(endpoint: str | None = None) -> bool:
|
||||
"""Check if DTS emulator is available at the given endpoint."""
|
||||
try:
|
||||
resolved_endpoint: str = _get_dts_endpoint() if endpoint is None else endpoint
|
||||
parsed = urlparse(resolved_endpoint)
|
||||
host = parsed.hostname or "localhost"
|
||||
port = parsed.port or 8080
|
||||
|
||||
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
|
||||
sock.settimeout(2)
|
||||
return sock.connect_ex((host, port)) == 0
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def _check_redis_available() -> bool:
|
||||
"""Check if Redis is available at the default connection string."""
|
||||
try:
|
||||
|
||||
async def test_connection() -> bool:
|
||||
redis_url = os.getenv("REDIS_CONNECTION_STRING", "redis://localhost:6379")
|
||||
try:
|
||||
client = aioredis.from_url(redis_url, socket_timeout=2) # type: ignore[reportUnknownMemberType]
|
||||
await client.ping() # type: ignore[reportUnknownMemberType]
|
||||
await client.aclose() # type: ignore[reportUnknownMemberType]
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
return asyncio.run(test_connection())
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Client Factory Functions
|
||||
# =============================================================================
|
||||
|
||||
|
||||
def create_dts_client(endpoint: str, taskhub: str) -> DurableTaskSchedulerClient:
|
||||
"""Create a DurableTaskSchedulerClient with common configuration.
|
||||
|
||||
Args:
|
||||
endpoint: The DTS endpoint address
|
||||
taskhub: The task hub name
|
||||
|
||||
Returns:
|
||||
A configured DurableTaskSchedulerClient instance
|
||||
"""
|
||||
return DurableTaskSchedulerClient(
|
||||
host_address=endpoint,
|
||||
secure_channel=False,
|
||||
taskhub=taskhub,
|
||||
token_credential=None,
|
||||
)
|
||||
|
||||
|
||||
def create_agent_client(
|
||||
endpoint: str,
|
||||
taskhub: str,
|
||||
max_poll_retries: int = 90,
|
||||
) -> tuple[DurableTaskSchedulerClient, DurableAIAgentClient]:
|
||||
"""Create a DurableAIAgentClient with the underlying DTS client.
|
||||
|
||||
Args:
|
||||
endpoint: The DTS endpoint address
|
||||
taskhub: The task hub name
|
||||
max_poll_retries: Max poll retries for the agent client
|
||||
|
||||
Returns:
|
||||
A tuple of (DurableTaskSchedulerClient, DurableAIAgentClient)
|
||||
"""
|
||||
dts_client = create_dts_client(endpoint, taskhub)
|
||||
agent_client = DurableAIAgentClient(dts_client, max_poll_retries=max_poll_retries)
|
||||
return dts_client, agent_client
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Orchestration Helper Class
|
||||
# =============================================================================
|
||||
|
||||
|
||||
class OrchestrationHelper:
|
||||
"""Helper class for orchestration-related test operations."""
|
||||
|
||||
def __init__(self, dts_client: DurableTaskSchedulerClient):
|
||||
"""Initialize the orchestration helper.
|
||||
|
||||
Args:
|
||||
dts_client: The DurableTaskSchedulerClient instance to use
|
||||
"""
|
||||
self.client = dts_client
|
||||
|
||||
def wait_for_orchestration(
|
||||
self,
|
||||
instance_id: str,
|
||||
timeout: float = 60.0,
|
||||
) -> Any:
|
||||
"""Wait for an orchestration to complete.
|
||||
|
||||
Args:
|
||||
instance_id: The orchestration instance ID
|
||||
timeout: Maximum time to wait in seconds
|
||||
|
||||
Returns:
|
||||
The final OrchestrationMetadata
|
||||
|
||||
Raises:
|
||||
TimeoutError: If the orchestration doesn't complete within timeout
|
||||
RuntimeError: If the orchestration fails
|
||||
"""
|
||||
# Use the built-in wait_for_orchestration_completion method
|
||||
metadata = self.client.wait_for_orchestration_completion(
|
||||
instance_id=instance_id,
|
||||
timeout=int(timeout),
|
||||
)
|
||||
|
||||
if metadata is None:
|
||||
raise TimeoutError(f"Orchestration {instance_id} did not complete within {timeout} seconds")
|
||||
|
||||
# Check if failed or terminated
|
||||
if metadata.runtime_status == OrchestrationStatus.FAILED:
|
||||
raise RuntimeError(f"Orchestration {instance_id} failed: {metadata.serialized_custom_status}")
|
||||
if metadata.runtime_status == OrchestrationStatus.TERMINATED:
|
||||
raise RuntimeError(f"Orchestration {instance_id} was terminated")
|
||||
|
||||
return metadata
|
||||
|
||||
def wait_for_orchestration_with_output(
|
||||
self,
|
||||
instance_id: str,
|
||||
timeout: float = 60.0,
|
||||
) -> tuple[Any, Any]:
|
||||
"""Wait for an orchestration to complete and return its output.
|
||||
|
||||
Args:
|
||||
instance_id: The orchestration instance ID
|
||||
timeout: Maximum time to wait in seconds
|
||||
|
||||
Returns:
|
||||
A tuple of (OrchestrationMetadata, output)
|
||||
|
||||
Raises:
|
||||
TimeoutError: If the orchestration doesn't complete within timeout
|
||||
RuntimeError: If the orchestration fails
|
||||
"""
|
||||
metadata = self.wait_for_orchestration(instance_id, timeout)
|
||||
|
||||
# The output should be available in the metadata
|
||||
return metadata, metadata.serialized_output
|
||||
|
||||
def get_orchestration_status(self, instance_id: str) -> Any | None:
|
||||
"""Get the current status of an orchestration.
|
||||
|
||||
Args:
|
||||
instance_id: The orchestration instance ID
|
||||
|
||||
Returns:
|
||||
The OrchestrationMetadata or None if not found
|
||||
"""
|
||||
try:
|
||||
# Try to wait with a short timeout to get current status
|
||||
return self.client.wait_for_orchestration_completion(
|
||||
instance_id=instance_id,
|
||||
timeout=1, # Very short timeout, just checking status
|
||||
)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
def raise_event(
|
||||
self,
|
||||
instance_id: str,
|
||||
event_name: str,
|
||||
event_data: Any = None,
|
||||
) -> None:
|
||||
"""Raise an external event to an orchestration.
|
||||
|
||||
Args:
|
||||
instance_id: The orchestration instance ID
|
||||
event_name: The name of the event
|
||||
event_data: The event data payload
|
||||
"""
|
||||
self.client.raise_orchestration_event(instance_id, event_name, data=event_data)
|
||||
|
||||
def wait_for_notification(self, instance_id: str, timeout_seconds: int = 30) -> bool:
|
||||
"""Wait for the orchestration to reach a notification point.
|
||||
|
||||
Polls the orchestration status until it appears to be waiting for approval.
|
||||
|
||||
Args:
|
||||
instance_id: The orchestration instance ID
|
||||
timeout_seconds: Maximum time to wait
|
||||
|
||||
Returns:
|
||||
True if notification detected, False if timeout
|
||||
"""
|
||||
start_time = time.time()
|
||||
while time.time() - start_time < timeout_seconds:
|
||||
try:
|
||||
metadata = self.client.get_orchestration_state(
|
||||
instance_id=instance_id,
|
||||
)
|
||||
|
||||
if metadata:
|
||||
# Check if we're waiting for approval by examining custom status
|
||||
if metadata.serialized_custom_status:
|
||||
try:
|
||||
custom_status = json.loads(metadata.serialized_custom_status)
|
||||
# Handle both string and dict custom status
|
||||
status_str = custom_status if isinstance(custom_status, str) else str(custom_status)
|
||||
if status_str.lower().startswith("requesting human feedback"):
|
||||
return True
|
||||
except (json.JSONDecodeError, AttributeError):
|
||||
# If it's not JSON, treat as plain string
|
||||
if metadata.serialized_custom_status.lower().startswith("requesting human feedback"):
|
||||
return True
|
||||
|
||||
# Check for terminal states
|
||||
if metadata.runtime_status.name == "COMPLETED" or metadata.runtime_status.name == "FAILED":
|
||||
return False
|
||||
except Exception:
|
||||
# Silently ignore transient errors during polling (e.g., network issues, service unavailable).
|
||||
# The loop will retry until timeout, allowing the service to recover.
|
||||
pass
|
||||
|
||||
time.sleep(1)
|
||||
|
||||
return False
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Pytest Configuration
|
||||
# =============================================================================
|
||||
|
||||
|
||||
def pytest_configure(config: pytest.Config) -> None:
|
||||
"""Register custom markers."""
|
||||
config.addinivalue_line("markers", "integration_test: mark test as integration test")
|
||||
config.addinivalue_line("markers", "requires_dts: mark test as requiring DTS emulator")
|
||||
config.addinivalue_line("markers", "requires_azure_openai: mark test as requiring Azure OpenAI")
|
||||
config.addinivalue_line("markers", "requires_redis: mark test as requiring Redis")
|
||||
config.addinivalue_line(
|
||||
"markers",
|
||||
"sample(path): specify the sample directory name for the test (e.g., @pytest.mark.sample('01_single_agent'))",
|
||||
)
|
||||
|
||||
|
||||
def pytest_collection_modifyitems(config: pytest.Config, items: list[pytest.Item]) -> None:
|
||||
"""Skip tests based on markers and environment availability."""
|
||||
foundry_vars = ["FOUNDRY_PROJECT_ENDPOINT", "FOUNDRY_MODEL"]
|
||||
foundry_available = all(os.getenv(var) for var in foundry_vars)
|
||||
azure_openai_vars = ["AZURE_OPENAI_ENDPOINT", "AZURE_OPENAI_MODEL"]
|
||||
azure_openai_available = all(os.getenv(var) for var in azure_openai_vars)
|
||||
skip_foundry = pytest.mark.skip(reason=f"Missing required environment variables: {', '.join(foundry_vars)}")
|
||||
skip_azure_openai = pytest.mark.skip(
|
||||
reason=f"Missing required environment variables: {', '.join(azure_openai_vars)}"
|
||||
)
|
||||
|
||||
# Check DTS availability
|
||||
dts_available = _check_dts_available()
|
||||
skip_dts = pytest.mark.skip(reason=f"DTS emulator is not available at {_get_dts_endpoint()}")
|
||||
|
||||
# Check Redis availability
|
||||
redis_available = _check_redis_available()
|
||||
skip_redis = pytest.mark.skip(reason="Redis is not available at redis://localhost:6379")
|
||||
|
||||
for item in items:
|
||||
if "requires_azure_openai" in item.keywords and not foundry_available:
|
||||
item.add_marker(skip_foundry)
|
||||
sample_marker = item.get_closest_marker("sample")
|
||||
sample_name = sample_marker.args[0] if sample_marker and sample_marker.args else None
|
||||
if sample_name == "06_multi_agent_orchestration_conditionals" and not azure_openai_available:
|
||||
item.add_marker(skip_azure_openai)
|
||||
if "requires_dts" in item.keywords and not dts_available:
|
||||
item.add_marker(skip_dts)
|
||||
if "requires_redis" in item.keywords and not redis_available:
|
||||
item.add_marker(skip_redis)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Pytest Fixtures
|
||||
# =============================================================================
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def dts_endpoint() -> str:
|
||||
"""Get the DTS endpoint from environment or use default."""
|
||||
return _get_dts_endpoint()
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def dts_available(dts_endpoint: str) -> bool:
|
||||
"""Check if DTS emulator is available and responding."""
|
||||
if _check_dts_available(dts_endpoint):
|
||||
return True
|
||||
pytest.skip(f"DTS emulator is not available at {dts_endpoint}")
|
||||
return False
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def check_sample_env(request: pytest.FixtureRequest) -> None:
|
||||
"""Verify the environment variables required by the current sample are set."""
|
||||
sample_marker = request.node.get_closest_marker("sample") # type: ignore[union-attr]
|
||||
if not sample_marker:
|
||||
pytest.fail("Test class must have @pytest.mark.sample() marker")
|
||||
|
||||
sample_name = cast(str, sample_marker.args[0]) # type: ignore[union-attr]
|
||||
# Samples that host no AI agents need no model credentials (only the DTS emulator).
|
||||
no_llm_samples = {"12_subworkflow_hitl"}
|
||||
if sample_name in no_llm_samples:
|
||||
return
|
||||
if sample_name == "06_multi_agent_orchestration_conditionals":
|
||||
required_vars = ["AZURE_OPENAI_ENDPOINT", "AZURE_OPENAI_MODEL"]
|
||||
else:
|
||||
required_vars = ["FOUNDRY_PROJECT_ENDPOINT", "FOUNDRY_MODEL"]
|
||||
missing = [var for var in required_vars if not os.getenv(var)]
|
||||
|
||||
if missing:
|
||||
pytest.skip(f"Missing required environment variables: {', '.join(missing)}")
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def unique_taskhub() -> str:
|
||||
"""Generate a unique task hub name for test isolation."""
|
||||
# Use a shorter UUID to avoid naming issues
|
||||
return f"test-{uuid.uuid4().hex[:8]}"
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def worker_process(
|
||||
dts_available: bool,
|
||||
check_sample_env: None,
|
||||
dts_endpoint: str,
|
||||
unique_taskhub: str,
|
||||
request: pytest.FixtureRequest,
|
||||
) -> Generator[dict[str, Any], None, None]:
|
||||
"""Start a worker process for the current test module by running the sample worker.py.
|
||||
|
||||
This fixture:
|
||||
1. Determines which sample to run from @pytest.mark.sample()
|
||||
2. Starts the sample's worker.py as a subprocess
|
||||
3. Waits for the worker to be ready
|
||||
4. Tears down the worker after tests complete
|
||||
|
||||
Usage:
|
||||
@pytest.mark.sample("01_single_agent")
|
||||
class TestSingleAgent:
|
||||
...
|
||||
"""
|
||||
# Get sample path from marker
|
||||
sample_marker = request.node.get_closest_marker("sample") # type: ignore[union-attr]
|
||||
if not sample_marker:
|
||||
pytest.fail("Test class must have @pytest.mark.sample() marker")
|
||||
|
||||
sample_name: str = cast(str, sample_marker.args[0]) # type: ignore[union-attr]
|
||||
sample_path: Path = Path(__file__).parents[4] / "samples" / "04-hosting" / "durabletask" / sample_name
|
||||
worker_file: Path = sample_path / "worker.py"
|
||||
|
||||
if not worker_file.exists():
|
||||
pytest.fail(f"Sample worker not found: {worker_file}")
|
||||
|
||||
# Set up environment for worker subprocess
|
||||
env = os.environ.copy()
|
||||
env["ENDPOINT"] = dts_endpoint
|
||||
env["TASKHUB"] = unique_taskhub
|
||||
|
||||
# Start worker subprocess
|
||||
try:
|
||||
# On Windows, use CREATE_NEW_PROCESS_GROUP to allow proper termination
|
||||
# shell=True only on Windows to handle PATH resolution
|
||||
if sys.platform == "win32":
|
||||
process = subprocess.Popen(
|
||||
[sys.executable, str(worker_file)],
|
||||
cwd=str(sample_path),
|
||||
creationflags=subprocess.CREATE_NEW_PROCESS_GROUP,
|
||||
shell=True,
|
||||
env=env,
|
||||
text=True,
|
||||
)
|
||||
# On Unix, don't use shell=True to avoid shell wrapper issues
|
||||
else:
|
||||
process = subprocess.Popen(
|
||||
[sys.executable, str(worker_file)],
|
||||
cwd=str(sample_path),
|
||||
env=env,
|
||||
text=True,
|
||||
)
|
||||
except Exception as e:
|
||||
pytest.fail(f"Failed to start worker subprocess: {e}")
|
||||
|
||||
# Wait for worker to initialize
|
||||
# The worker needs time to:
|
||||
# 1. Start Python and import modules
|
||||
# 2. Create Azure OpenAI clients
|
||||
# 3. Register agents with the DTS worker
|
||||
# 4. Connect to DTS and be ready to receive signals
|
||||
#
|
||||
# We use a generous wait time because CI environments can be slow,
|
||||
# and the first test that runs depends on the worker being fully ready.
|
||||
time.sleep(8)
|
||||
|
||||
# Check if process is still running
|
||||
if process.poll() is not None:
|
||||
stderr_output = process.stderr.read() if process.stderr else ""
|
||||
pytest.fail(f"Worker process exited prematurely. stderr: {stderr_output}")
|
||||
|
||||
# Provide worker info to tests
|
||||
worker_info = {
|
||||
"process": process,
|
||||
"endpoint": dts_endpoint,
|
||||
"taskhub": unique_taskhub,
|
||||
}
|
||||
|
||||
try:
|
||||
yield worker_info
|
||||
finally:
|
||||
# Cleanup: terminate worker subprocess
|
||||
try:
|
||||
process.terminate()
|
||||
try:
|
||||
process.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
process.kill()
|
||||
process.wait()
|
||||
except Exception as e:
|
||||
logging.warning(f"Error during worker process cleanup: {e}")
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def orchestration_helper(worker_process: dict[str, Any]) -> OrchestrationHelper:
|
||||
"""Create an OrchestrationHelper for the current test module."""
|
||||
dts_client = create_dts_client(worker_process["endpoint"], worker_process["taskhub"])
|
||||
return OrchestrationHelper(dts_client)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def agent_client_factory(worker_process: dict[str, Any]) -> type[AgentClientFactoryProtocol]:
|
||||
"""Return a factory class for creating agent clients.
|
||||
|
||||
Usage in tests:
|
||||
def test_example(self, agent_client_factory):
|
||||
dts_client, agent_client = agent_client_factory.create(max_poll_retries=90)
|
||||
"""
|
||||
|
||||
class AgentClientFactory:
|
||||
"""Factory for creating DTS and Agent client pairs."""
|
||||
|
||||
endpoint = worker_process["endpoint"]
|
||||
taskhub = worker_process["taskhub"]
|
||||
|
||||
@classmethod
|
||||
def create(cls, max_poll_retries: int = 90) -> tuple[DurableTaskSchedulerClient, DurableAIAgentClient]:
|
||||
"""Create a DTS client and Agent client pair."""
|
||||
return create_agent_client(cls.endpoint, cls.taskhub, max_poll_retries)
|
||||
|
||||
return AgentClientFactory
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def workflow_client(worker_process: dict[str, Any]) -> DurableWorkflowClient:
|
||||
"""Create a DurableWorkflowClient bound to the current sample worker's task hub."""
|
||||
dts_client = create_dts_client(worker_process["endpoint"], worker_process["taskhub"])
|
||||
return DurableWorkflowClient(dts_client)
|
||||
@@ -0,0 +1,97 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
"""Integration tests for single agent functionality.
|
||||
|
||||
Tests basic agent operations including:
|
||||
- Agent registration and retrieval
|
||||
- Single agent interactions
|
||||
- Conversation continuity across multiple messages
|
||||
- Multi-threaded agent usage
|
||||
- Empty thread ID handling
|
||||
"""
|
||||
|
||||
from typing import Any, Protocol
|
||||
|
||||
import pytest
|
||||
|
||||
from agent_framework_durabletask import DurableAIAgentClient
|
||||
|
||||
|
||||
class AgentClientFactoryProtocol(Protocol):
|
||||
"""Protocol for the agent client factory fixture."""
|
||||
|
||||
@classmethod
|
||||
def create(cls, max_poll_retries: int = 90) -> tuple[Any, DurableAIAgentClient]: ...
|
||||
|
||||
|
||||
# Module-level markers - applied to all tests in this module
|
||||
pytestmark = [
|
||||
pytest.mark.flaky,
|
||||
pytest.mark.integration,
|
||||
pytest.mark.sample("01_single_agent"),
|
||||
pytest.mark.integration_test,
|
||||
pytest.mark.requires_azure_openai,
|
||||
pytest.mark.requires_dts,
|
||||
]
|
||||
|
||||
|
||||
class TestSingleAgent:
|
||||
"""Test suite for single agent functionality."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup(self, agent_client_factory: type[AgentClientFactoryProtocol]) -> None:
|
||||
"""Setup test fixtures."""
|
||||
# Create agent client using the factory fixture
|
||||
_, self.agent_client = agent_client_factory.create()
|
||||
|
||||
def test_agent_registration(self) -> None:
|
||||
"""Test that the Joker agent is registered and accessible."""
|
||||
agent = self.agent_client.get_agent("Joker")
|
||||
assert agent is not None
|
||||
assert agent.name == "Joker"
|
||||
|
||||
def test_single_interaction(self):
|
||||
"""Test a single interaction with the agent."""
|
||||
agent = self.agent_client.get_agent("Joker")
|
||||
session = agent.create_session()
|
||||
|
||||
response = agent.run("Tell me a short joke about programming.", session=session)
|
||||
|
||||
assert response is not None
|
||||
assert response.text is not None
|
||||
assert len(response.text) > 0
|
||||
|
||||
def test_conversation_continuity(self):
|
||||
"""Test that conversation context is maintained across turns."""
|
||||
agent = self.agent_client.get_agent("Joker")
|
||||
session = agent.create_session()
|
||||
|
||||
# First turn: Ask for a joke about a specific topic
|
||||
response1 = agent.run("Tell me a joke about cats.", session=session)
|
||||
assert response1 is not None
|
||||
assert len(response1.text) > 0
|
||||
|
||||
# Second turn: Ask a follow-up that requires context
|
||||
response2 = agent.run("Can you make it funnier?", session=session)
|
||||
assert response2 is not None
|
||||
assert len(response2.text) > 0
|
||||
|
||||
# The agent should understand "it" refers to the previous joke
|
||||
|
||||
def test_multiple_sessions(self):
|
||||
"""Test that different sessions maintain separate contexts."""
|
||||
agent = self.agent_client.get_agent("Joker")
|
||||
|
||||
# Create two separate sessions
|
||||
session1 = agent.create_session()
|
||||
session2 = agent.create_session()
|
||||
|
||||
assert session1.durable_session_id != session2.durable_session_id
|
||||
|
||||
# Send different messages to each session
|
||||
response1 = agent.run("Tell me a joke about dogs.", session=session1)
|
||||
response2 = agent.run("Tell me a joke about birds.", session=session2)
|
||||
|
||||
assert response1 is not None
|
||||
assert response2 is not None
|
||||
assert response1.text != response2.text
|
||||
@@ -0,0 +1,113 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
"""Integration tests for multi-agent functionality.
|
||||
|
||||
Tests operations with multiple specialized agents:
|
||||
- Multiple agent registration
|
||||
- Agent-specific tool usage
|
||||
- Independent thread management per agent
|
||||
- Concurrent agent operations
|
||||
- Agent isolation and tool routing
|
||||
"""
|
||||
|
||||
from typing import Any, Protocol
|
||||
|
||||
import pytest
|
||||
|
||||
from agent_framework_durabletask import DurableAIAgentClient
|
||||
|
||||
|
||||
class AgentClientFactoryProtocol(Protocol):
|
||||
"""Protocol for the agent client factory fixture."""
|
||||
|
||||
@classmethod
|
||||
def create(cls, max_poll_retries: int = 90) -> tuple[Any, DurableAIAgentClient]: ...
|
||||
|
||||
|
||||
# Agent names from the 02_multi_agent sample
|
||||
WEATHER_AGENT_NAME: str = "WeatherAgent"
|
||||
MATH_AGENT_NAME: str = "MathAgent"
|
||||
|
||||
# Module-level markers - applied to all tests in this module
|
||||
pytestmark = [
|
||||
pytest.mark.flaky,
|
||||
pytest.mark.integration,
|
||||
pytest.mark.sample("02_multi_agent"),
|
||||
pytest.mark.integration_test,
|
||||
pytest.mark.requires_azure_openai,
|
||||
pytest.mark.requires_dts,
|
||||
]
|
||||
|
||||
|
||||
class TestMultiAgent:
|
||||
"""Test suite for multi-agent functionality."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup(self, agent_client_factory: type[AgentClientFactoryProtocol]) -> None:
|
||||
"""Setup test fixtures."""
|
||||
# Create agent client using the factory fixture
|
||||
_, self.agent_client = agent_client_factory.create()
|
||||
|
||||
def test_multiple_agents_registered(self) -> None:
|
||||
"""Test that both agents are registered and accessible."""
|
||||
weather_agent = self.agent_client.get_agent(WEATHER_AGENT_NAME)
|
||||
math_agent = self.agent_client.get_agent(MATH_AGENT_NAME)
|
||||
|
||||
assert weather_agent is not None
|
||||
assert weather_agent.name == WEATHER_AGENT_NAME
|
||||
assert math_agent is not None
|
||||
assert math_agent.name == MATH_AGENT_NAME
|
||||
|
||||
@pytest.mark.skip(reason="Flaky in CI: times out / crashes the xdist runner; temporarily disabled.")
|
||||
def test_weather_agent_with_tool(self):
|
||||
"""Test weather agent with weather tool execution."""
|
||||
agent = self.agent_client.get_agent(WEATHER_AGENT_NAME)
|
||||
session = agent.create_session()
|
||||
|
||||
response = agent.run("What's the weather in Seattle?", session=session)
|
||||
|
||||
assert response is not None
|
||||
assert response.text is not None
|
||||
# Should contain weather information from the tool
|
||||
assert len(response.text) > 0
|
||||
|
||||
# Verify that the get_weather tool was actually invoked
|
||||
tool_calls = [
|
||||
content for msg in response.messages for content in msg.contents if content.type == "function_call"
|
||||
]
|
||||
assert len(tool_calls) > 0, "Expected at least one tool call"
|
||||
assert any(call.name == "get_weather" for call in tool_calls), "Expected get_weather tool to be called"
|
||||
|
||||
@pytest.mark.skip(reason="Flaky in CI: times out / crashes the xdist runner; temporarily disabled.")
|
||||
def test_math_agent_with_tool(self):
|
||||
"""Test math agent with calculation tool execution."""
|
||||
agent = self.agent_client.get_agent(MATH_AGENT_NAME)
|
||||
session = agent.create_session()
|
||||
|
||||
response = agent.run("Calculate a 20% tip on a $50 bill.", session=session)
|
||||
|
||||
assert response is not None
|
||||
assert response.text is not None
|
||||
# Should contain calculation results from the tool
|
||||
assert len(response.text) > 0
|
||||
|
||||
# Verify that the calculate_tip tool was actually invoked
|
||||
tool_calls = [
|
||||
content for msg in response.messages for content in msg.contents if content.type == "function_call"
|
||||
]
|
||||
assert len(tool_calls) > 0, "Expected at least one tool call"
|
||||
assert any(call.name == "calculate_tip" for call in tool_calls), "Expected calculate_tip tool to be called"
|
||||
|
||||
def test_multiple_calls_to_same_agent(self):
|
||||
"""Test multiple sequential calls to the same agent."""
|
||||
agent = self.agent_client.get_agent(WEATHER_AGENT_NAME)
|
||||
session = agent.create_session()
|
||||
|
||||
# Multiple weather queries
|
||||
response1 = agent.run("What's the weather in Chicago?", session=session)
|
||||
response2 = agent.run("And what about Los Angeles?", session=session)
|
||||
|
||||
assert response1 is not None
|
||||
assert response2 is not None
|
||||
assert len(response1.text) > 0
|
||||
assert len(response2.text) > 0
|
||||
+236
@@ -0,0 +1,236 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
"""
|
||||
Integration Tests for Reliable Streaming Sample
|
||||
|
||||
Tests the reliable streaming sample using Redis Streams for persistent message delivery.
|
||||
|
||||
The worker process is automatically started by the test fixture.
|
||||
|
||||
Prerequisites:
|
||||
- Azure OpenAI credentials configured (see packages/durabletask/tests/integration_tests/.env.example)
|
||||
- DTS emulator running (docker run -d -p 8080:8080 mcr.microsoft.com/durabletask/emulator:latest)
|
||||
- Redis running (docker run -d --name redis -p 6379:6379 redis:latest)
|
||||
|
||||
Usage:
|
||||
uv run pytest packages/durabletask/tests/integration_tests/test_03_single_agent_streaming.py -v
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from datetime import timedelta
|
||||
from pathlib import Path
|
||||
from typing import Any, Protocol
|
||||
|
||||
import pytest
|
||||
import redis.asyncio as aioredis
|
||||
|
||||
from agent_framework_durabletask import DurableAIAgentClient
|
||||
|
||||
|
||||
class AgentClientFactoryProtocol(Protocol):
|
||||
"""Protocol for the agent client factory fixture."""
|
||||
|
||||
@classmethod
|
||||
def create(cls, max_poll_retries: int = 90) -> tuple[Any, DurableAIAgentClient]: ...
|
||||
|
||||
|
||||
# Add sample directory to path to import RedisStreamResponseHandler
|
||||
SAMPLE_DIR = Path(__file__).parents[4] / "samples" / "04-hosting" / "durabletask" / "03_single_agent_streaming"
|
||||
sys.path.insert(0, str(SAMPLE_DIR))
|
||||
|
||||
from redis_stream_response_handler import ( # type: ignore[reportMissingImports] # pyrefly: ignore[missing-import] # ty: ignore[unresolved-import] # noqa: E402
|
||||
RedisStreamResponseHandler,
|
||||
)
|
||||
|
||||
# Module-level markers - applied to all tests in this file
|
||||
pytestmark = [
|
||||
pytest.mark.flaky,
|
||||
pytest.mark.integration,
|
||||
pytest.mark.sample("03_single_agent_streaming"),
|
||||
pytest.mark.integration_test,
|
||||
pytest.mark.requires_azure_openai,
|
||||
pytest.mark.requires_dts,
|
||||
pytest.mark.requires_redis,
|
||||
]
|
||||
|
||||
|
||||
class TestSampleReliableStreaming:
|
||||
"""Tests for 03_single_agent_streaming sample."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup(self, agent_client_factory: type[AgentClientFactoryProtocol], orchestration_helper) -> None:
|
||||
"""Setup test fixtures."""
|
||||
# Create agent client using the factory fixture
|
||||
_, self.agent_client = agent_client_factory.create()
|
||||
self.helper = orchestration_helper
|
||||
|
||||
# Redis configuration
|
||||
self.redis_connection_string = os.environ.get("REDIS_CONNECTION_STRING", "redis://localhost:6379")
|
||||
self.redis_stream_ttl_minutes = int(os.environ.get("REDIS_STREAM_TTL_MINUTES", "10"))
|
||||
|
||||
async def _get_stream_handler(self) -> RedisStreamResponseHandler: # type: ignore[reportMissingTypeStubs]
|
||||
"""Create a new Redis stream handler for each request."""
|
||||
redis_client = aioredis.from_url( # type: ignore[reportUnknownMemberType]
|
||||
self.redis_connection_string,
|
||||
encoding="utf-8",
|
||||
decode_responses=False,
|
||||
)
|
||||
return RedisStreamResponseHandler( # type: ignore[reportUnknownMemberType]
|
||||
redis_client=redis_client,
|
||||
stream_ttl=timedelta(minutes=self.redis_stream_ttl_minutes),
|
||||
)
|
||||
|
||||
async def _stream_from_redis(
|
||||
self,
|
||||
session_key: str,
|
||||
cursor: str | None = None,
|
||||
timeout: float = 30.0,
|
||||
) -> tuple[str, bool, str]:
|
||||
"""
|
||||
Stream responses from Redis using the sample's RedisStreamResponseHandler.
|
||||
|
||||
Args:
|
||||
session_key: The conversation/thread ID to stream from
|
||||
cursor: Optional cursor to resume from
|
||||
timeout: Maximum time to wait for stream completion
|
||||
|
||||
Returns:
|
||||
Tuple of (accumulated text, completion status, last entry_id)
|
||||
"""
|
||||
accumulated_text = ""
|
||||
is_complete = False
|
||||
last_entry_id = cursor if cursor else "0-0"
|
||||
start_time = time.time()
|
||||
|
||||
async with await self._get_stream_handler() as stream_handler: # type: ignore[reportUnknownMemberType]
|
||||
try:
|
||||
async for chunk in stream_handler.read_stream(session_key, cursor): # type: ignore[reportUnknownMemberType]
|
||||
if time.time() - start_time > timeout:
|
||||
break
|
||||
|
||||
last_entry_id = chunk.entry_id # type: ignore[reportUnknownMemberType]
|
||||
|
||||
if chunk.error: # type: ignore[reportUnknownMemberType]
|
||||
# Stream not found or timeout - this is expected if agent hasn't written yet
|
||||
# Don't raise an error, just return what we have
|
||||
break
|
||||
|
||||
if chunk.is_done: # type: ignore[reportUnknownMemberType]
|
||||
is_complete = True
|
||||
break
|
||||
|
||||
if chunk.text: # type: ignore[reportUnknownMemberType]
|
||||
accumulated_text += chunk.text # type: ignore[reportUnknownMemberType]
|
||||
|
||||
except Exception as ex:
|
||||
# For test purposes, we catch exceptions and return what we have
|
||||
if "timed out" not in str(ex).lower():
|
||||
raise
|
||||
|
||||
return accumulated_text, is_complete, last_entry_id # type: ignore[reportReturnType]
|
||||
|
||||
def test_agent_run_and_stream(self) -> None:
|
||||
"""Test agent execution with Redis streaming."""
|
||||
# Get the TravelPlanner agent
|
||||
travel_planner = self.agent_client.get_agent("TravelPlanner")
|
||||
assert travel_planner is not None
|
||||
assert travel_planner.name == "TravelPlanner"
|
||||
|
||||
# Create a new session
|
||||
session = travel_planner.create_session()
|
||||
assert session.durable_session_id is not None
|
||||
assert session.durable_session_id.key is not None
|
||||
session_key = str(session.durable_session_id.key)
|
||||
|
||||
# Start agent run with wait_for_response=False for non-blocking execution
|
||||
travel_planner.run(
|
||||
"Plan a 1-day trip to Seattle in 1 sentence", session=session, options={"wait_for_response": False}
|
||||
)
|
||||
|
||||
# Poll Redis stream with retries to handle race conditions
|
||||
# The agent may take a few seconds to process and start writing to Redis
|
||||
# We use cursor-based resumption to continue reading from where we left off
|
||||
max_retries = 20
|
||||
retry_count = 0
|
||||
accumulated_text = ""
|
||||
is_complete = False
|
||||
cursor: str | None = None
|
||||
|
||||
while retry_count < max_retries and not is_complete:
|
||||
text, is_complete, last_cursor = asyncio.run(
|
||||
self._stream_from_redis(session_key, cursor=cursor, timeout=10.0)
|
||||
)
|
||||
accumulated_text += text
|
||||
cursor = last_cursor # Resume from last position on next read
|
||||
|
||||
if is_complete:
|
||||
# Stream completed successfully
|
||||
break
|
||||
|
||||
if len(accumulated_text) > 0:
|
||||
# Got content but not completion marker yet - keep reading without delay
|
||||
# The agent may still be streaming or about to write completion marker
|
||||
continue
|
||||
|
||||
# No content yet - wait before retrying
|
||||
time.sleep(2)
|
||||
retry_count += 1
|
||||
|
||||
# Verify we got content
|
||||
assert len(accumulated_text) > 0, (
|
||||
f"Expected text content but got empty string for session_key: {session_key} after {retry_count} retries"
|
||||
)
|
||||
assert "seattle" in accumulated_text.lower(), f"Expected 'seattle' in response but got: {accumulated_text}"
|
||||
assert is_complete, "Expected stream to be complete"
|
||||
|
||||
def test_stream_with_cursor_resumption(self) -> None:
|
||||
"""Test streaming with cursor-based resumption."""
|
||||
# Get the TravelPlanner agent
|
||||
travel_planner = self.agent_client.get_agent("TravelPlanner")
|
||||
session = travel_planner.create_session()
|
||||
assert session.durable_session_id is not None
|
||||
assert session.durable_session_id.key is not None
|
||||
session_key = str(session.durable_session_id.key)
|
||||
|
||||
# Start agent run
|
||||
travel_planner.run("What's the weather like?", session=session, options={"wait_for_response": False})
|
||||
|
||||
# Wait for agent to start writing
|
||||
time.sleep(3)
|
||||
|
||||
# Read partial stream to get a cursor
|
||||
async def get_partial_stream() -> tuple[str, str]:
|
||||
async with await self._get_stream_handler() as stream_handler: # type: ignore[reportUnknownMemberType]
|
||||
accumulated_text = ""
|
||||
last_entry_id = "0-0"
|
||||
chunk_count = 0
|
||||
|
||||
# Read just first 2 chunks
|
||||
async for chunk in stream_handler.read_stream(session_key): # type: ignore[reportUnknownMemberType]
|
||||
last_entry_id = chunk.entry_id # type: ignore[reportUnknownMemberType]
|
||||
if chunk.text: # type: ignore[reportUnknownMemberType]
|
||||
accumulated_text += chunk.text # type: ignore[reportUnknownMemberType]
|
||||
chunk_count += 1
|
||||
if chunk_count >= 2:
|
||||
break
|
||||
|
||||
return accumulated_text, last_entry_id # type: ignore[reportReturnType]
|
||||
|
||||
partial_text, cursor = asyncio.run(get_partial_stream())
|
||||
|
||||
# Resume from cursor
|
||||
remaining_text, _, _ = asyncio.run(self._stream_from_redis(session_key, cursor=cursor))
|
||||
|
||||
# Verify we got some initial content
|
||||
assert len(partial_text) > 0
|
||||
|
||||
# Combined text should be coherent
|
||||
full_text = partial_text + remaining_text
|
||||
assert len(full_text) > 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
+111
@@ -0,0 +1,111 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
"""Integration tests for single agent orchestration with chaining.
|
||||
|
||||
Tests orchestration patterns with sequential agent calls:
|
||||
- Orchestration registration and execution
|
||||
- Sequential agent calls on same thread
|
||||
- Conversation continuity in orchestrations
|
||||
- Thread context preservation
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from typing import Any, Protocol
|
||||
|
||||
import pytest
|
||||
from durabletask.client import OrchestrationStatus
|
||||
|
||||
from agent_framework_durabletask import DurableAIAgentClient
|
||||
|
||||
|
||||
class AgentClientFactoryProtocol(Protocol):
|
||||
"""Protocol for the agent client factory fixture."""
|
||||
|
||||
@classmethod
|
||||
def create(cls, max_poll_retries: int = 90) -> tuple[Any, DurableAIAgentClient]: ...
|
||||
|
||||
|
||||
# Agent name from the 04_single_agent_orchestration_chaining sample
|
||||
WRITER_AGENT_NAME: str = "WriterAgent"
|
||||
|
||||
# Configure logging
|
||||
logging.basicConfig(level=logging.WARNING)
|
||||
|
||||
# Module-level markers - applied to all tests in this module
|
||||
pytestmark = [
|
||||
pytest.mark.flaky,
|
||||
pytest.mark.integration,
|
||||
pytest.mark.sample("04_single_agent_orchestration_chaining"),
|
||||
pytest.mark.integration_test,
|
||||
pytest.mark.requires_azure_openai,
|
||||
pytest.mark.requires_dts,
|
||||
]
|
||||
|
||||
|
||||
class TestSingleAgentOrchestrationChaining:
|
||||
"""Test suite for single agent orchestration with chaining."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup(self, agent_client_factory: type[AgentClientFactoryProtocol], orchestration_helper) -> None:
|
||||
"""Setup test fixtures."""
|
||||
# Create agent client using the factory fixture
|
||||
self.dts_client, self.agent_client = agent_client_factory.create()
|
||||
self.orch_helper = orchestration_helper
|
||||
|
||||
def test_agent_registered(self):
|
||||
"""Test that the Writer agent is registered."""
|
||||
agent = self.agent_client.get_agent(WRITER_AGENT_NAME)
|
||||
assert agent is not None
|
||||
assert agent.name == WRITER_AGENT_NAME
|
||||
|
||||
def test_chaining_context_preserved(self):
|
||||
"""Test that context is preserved across agent runs in orchestration."""
|
||||
# Start the orchestration
|
||||
instance_id = self.dts_client.schedule_new_orchestration(
|
||||
orchestrator="single_agent_chaining_orchestration",
|
||||
input="",
|
||||
)
|
||||
|
||||
# Wait for completion with output
|
||||
metadata, output = self.orch_helper.wait_for_orchestration_with_output(
|
||||
instance_id=instance_id,
|
||||
timeout=120.0,
|
||||
)
|
||||
|
||||
assert metadata is not None
|
||||
assert output is not None
|
||||
|
||||
# The final output should be a refined sentence
|
||||
final_text = json.loads(output)
|
||||
|
||||
# Should be a meaningful sentence (not empty or error message)
|
||||
assert len(final_text) > 10
|
||||
assert not final_text.startswith("Error")
|
||||
|
||||
def test_multiple_orchestration_instances(self):
|
||||
"""Test that multiple orchestration instances can run independently."""
|
||||
# Start two orchestrations
|
||||
instance_id_1 = self.dts_client.schedule_new_orchestration(
|
||||
orchestrator="single_agent_chaining_orchestration",
|
||||
input="",
|
||||
)
|
||||
instance_id_2 = self.dts_client.schedule_new_orchestration(
|
||||
orchestrator="single_agent_chaining_orchestration",
|
||||
input="",
|
||||
)
|
||||
|
||||
assert instance_id_1 != instance_id_2
|
||||
|
||||
# Both should complete
|
||||
metadata_1 = self.orch_helper.wait_for_orchestration(
|
||||
instance_id=instance_id_1,
|
||||
timeout=120.0,
|
||||
)
|
||||
metadata_2 = self.orch_helper.wait_for_orchestration(
|
||||
instance_id=instance_id_2,
|
||||
timeout=120.0,
|
||||
)
|
||||
|
||||
assert metadata_1.runtime_status == OrchestrationStatus.COMPLETED
|
||||
assert metadata_2.runtime_status == OrchestrationStatus.COMPLETED
|
||||
+87
@@ -0,0 +1,87 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
"""Integration tests for multi-agent orchestration with concurrency.
|
||||
|
||||
Tests concurrent execution patterns:
|
||||
- Parallel agent execution
|
||||
- Concurrent orchestration tasks
|
||||
- Independent thread management in parallel
|
||||
- Result aggregation from concurrent calls
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from typing import Any, Protocol
|
||||
|
||||
import pytest
|
||||
from durabletask.client import OrchestrationStatus
|
||||
|
||||
from agent_framework_durabletask import DurableAIAgentClient
|
||||
|
||||
|
||||
class AgentClientFactoryProtocol(Protocol):
|
||||
"""Protocol for the agent client factory fixture."""
|
||||
|
||||
@classmethod
|
||||
def create(cls, max_poll_retries: int = 90) -> tuple[Any, DurableAIAgentClient]: ...
|
||||
|
||||
|
||||
# Agent names from the 05_multi_agent_orchestration_concurrency sample
|
||||
PHYSICIST_AGENT_NAME: str = "PhysicistAgent"
|
||||
CHEMIST_AGENT_NAME: str = "ChemistAgent"
|
||||
|
||||
# Configure logging
|
||||
logging.basicConfig(level=logging.WARNING)
|
||||
|
||||
# Module-level markers
|
||||
pytestmark = [
|
||||
pytest.mark.flaky,
|
||||
pytest.mark.integration,
|
||||
pytest.mark.sample("05_multi_agent_orchestration_concurrency"),
|
||||
pytest.mark.integration_test,
|
||||
pytest.mark.requires_dts,
|
||||
]
|
||||
|
||||
|
||||
class TestMultiAgentOrchestrationConcurrency:
|
||||
"""Test suite for multi-agent orchestration with concurrency."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup(self, agent_client_factory: type[AgentClientFactoryProtocol], orchestration_helper) -> None:
|
||||
"""Setup test fixtures."""
|
||||
# Create agent client using the factory fixture
|
||||
self.dts_client, self.agent_client = agent_client_factory.create()
|
||||
self.orch_helper = orchestration_helper
|
||||
|
||||
def test_agents_registered(self):
|
||||
"""Test that both agents are registered."""
|
||||
physicist = self.agent_client.get_agent(PHYSICIST_AGENT_NAME)
|
||||
chemist = self.agent_client.get_agent(CHEMIST_AGENT_NAME)
|
||||
|
||||
assert physicist is not None
|
||||
assert physicist.name == PHYSICIST_AGENT_NAME
|
||||
assert chemist is not None
|
||||
assert chemist.name == CHEMIST_AGENT_NAME
|
||||
|
||||
def test_different_prompts(self):
|
||||
"""Test concurrent orchestration with different prompts."""
|
||||
prompts = [
|
||||
"What is temperature?",
|
||||
"Explain molecules.",
|
||||
]
|
||||
|
||||
for prompt in prompts:
|
||||
instance_id = self.dts_client.schedule_new_orchestration(
|
||||
orchestrator="multi_agent_concurrent_orchestration",
|
||||
input=prompt,
|
||||
)
|
||||
|
||||
metadata, output = self.orch_helper.wait_for_orchestration_with_output(
|
||||
instance_id=instance_id,
|
||||
timeout=120.0,
|
||||
)
|
||||
|
||||
assert metadata.runtime_status == OrchestrationStatus.COMPLETED
|
||||
result = json.loads(output)
|
||||
assert "physicist" in result
|
||||
assert "chemist" in result
|
||||
+86
@@ -0,0 +1,86 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
"""Integration tests for multi-agent orchestration with conditionals.
|
||||
|
||||
Tests conditional orchestration patterns:
|
||||
- Conditional branching in orchestrations
|
||||
- Agent-based decision making
|
||||
- Activity function execution
|
||||
- Structured output handling
|
||||
- Conditional routing based on agent responses
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Any, Protocol
|
||||
|
||||
import pytest
|
||||
from durabletask.client import OrchestrationStatus
|
||||
|
||||
from agent_framework_durabletask import DurableAIAgentClient
|
||||
|
||||
|
||||
class AgentClientFactoryProtocol(Protocol):
|
||||
"""Protocol for the agent client factory fixture."""
|
||||
|
||||
@classmethod
|
||||
def create(cls, max_poll_retries: int = 90) -> tuple[Any, DurableAIAgentClient]: ...
|
||||
|
||||
|
||||
# Agent names from the 06_multi_agent_orchestration_conditionals sample
|
||||
SPAM_AGENT_NAME: str = "SpamDetectionAgent"
|
||||
EMAIL_AGENT_NAME: str = "EmailAssistantAgent"
|
||||
|
||||
# Configure logging
|
||||
logging.basicConfig(level=logging.WARNING)
|
||||
|
||||
# Module-level markers
|
||||
pytestmark = [
|
||||
pytest.mark.flaky,
|
||||
pytest.mark.integration,
|
||||
pytest.mark.sample("06_multi_agent_orchestration_conditionals"),
|
||||
pytest.mark.integration_test,
|
||||
pytest.mark.requires_dts,
|
||||
]
|
||||
|
||||
|
||||
class TestMultiAgentOrchestrationConditionals:
|
||||
"""Test suite for multi-agent orchestration with conditionals."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup(self, agent_client_factory: type[AgentClientFactoryProtocol], orchestration_helper) -> None:
|
||||
"""Setup test fixtures."""
|
||||
# Create agent client using the factory fixture
|
||||
self.dts_client, self.agent_client = agent_client_factory.create()
|
||||
self.orch_helper = orchestration_helper
|
||||
|
||||
def test_agents_registered(self):
|
||||
"""Test that both agents are registered."""
|
||||
spam_agent = self.agent_client.get_agent(SPAM_AGENT_NAME)
|
||||
email_agent = self.agent_client.get_agent(EMAIL_AGENT_NAME)
|
||||
|
||||
assert spam_agent is not None
|
||||
assert spam_agent.name == SPAM_AGENT_NAME
|
||||
assert email_agent is not None
|
||||
assert email_agent.name == EMAIL_AGENT_NAME
|
||||
|
||||
@pytest.mark.skip(reason="Flaky in CI: times out / crashes the xdist runner; temporarily disabled.")
|
||||
def test_conditional_branching(self):
|
||||
"""Test that conditional branching works correctly."""
|
||||
# Test with obvious spam
|
||||
spam_payload = {
|
||||
"email_id": "spam-001",
|
||||
"email_content": "Buy cheap medications online! No prescription needed! Limited time offer!",
|
||||
}
|
||||
|
||||
spam_instance_id = self.dts_client.schedule_new_orchestration(
|
||||
orchestrator="spam_detection_orchestration",
|
||||
input=spam_payload,
|
||||
)
|
||||
|
||||
# Both should complete successfully (different branches)
|
||||
spam_metadata = self.orch_helper.wait_for_orchestration(
|
||||
instance_id=spam_instance_id,
|
||||
timeout=120.0,
|
||||
)
|
||||
|
||||
assert spam_metadata.runtime_status == OrchestrationStatus.COMPLETED
|
||||
+174
@@ -0,0 +1,174 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
"""Integration tests for single agent orchestration with human-in-the-loop.
|
||||
|
||||
Tests human-in-the-loop (HITL) patterns:
|
||||
- External event waiting and handling
|
||||
- Timeout handling in orchestrations
|
||||
- Iterative refinement with human feedback
|
||||
- Activity function integration
|
||||
- Approval workflow patterns
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Any, Protocol
|
||||
|
||||
import pytest
|
||||
from durabletask.client import OrchestrationStatus
|
||||
|
||||
from agent_framework_durabletask import DurableAIAgentClient
|
||||
|
||||
|
||||
class AgentClientFactoryProtocol(Protocol):
|
||||
"""Protocol for the agent client factory fixture."""
|
||||
|
||||
@classmethod
|
||||
def create(cls, max_poll_retries: int = 90) -> tuple[Any, DurableAIAgentClient]: ...
|
||||
|
||||
|
||||
# Constants from the 07_single_agent_orchestration_hitl sample
|
||||
WRITER_AGENT_NAME: str = "WriterAgent"
|
||||
HUMAN_APPROVAL_EVENT: str = "HumanApproval"
|
||||
|
||||
# Configure logging
|
||||
logging.basicConfig(level=logging.WARNING)
|
||||
|
||||
# Module-level markers
|
||||
pytestmark = [
|
||||
pytest.mark.flaky,
|
||||
pytest.mark.integration,
|
||||
pytest.mark.sample("07_single_agent_orchestration_hitl"),
|
||||
pytest.mark.integration_test,
|
||||
pytest.mark.requires_dts,
|
||||
]
|
||||
|
||||
|
||||
class TestSingleAgentOrchestrationHITL:
|
||||
"""Test suite for single agent orchestration with human-in-the-loop."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup(self, agent_client_factory: type[AgentClientFactoryProtocol], orchestration_helper) -> None:
|
||||
"""Setup test fixtures."""
|
||||
# Create agent client using the factory fixture
|
||||
self.dts_client, self.agent_client = agent_client_factory.create()
|
||||
self.orch_helper = orchestration_helper
|
||||
|
||||
def test_agent_registered(self):
|
||||
"""Test that the Writer agent is registered."""
|
||||
agent = self.agent_client.get_agent(WRITER_AGENT_NAME)
|
||||
assert agent is not None
|
||||
assert agent.name == WRITER_AGENT_NAME
|
||||
|
||||
def test_hitl_orchestration_with_approval(self):
|
||||
"""Test HITL orchestration with immediate approval."""
|
||||
payload = {
|
||||
"topic": "The benefits of continuous learning",
|
||||
"max_review_attempts": 3,
|
||||
"approval_timeout_seconds": 60,
|
||||
}
|
||||
|
||||
# Start the orchestration
|
||||
instance_id = self.dts_client.schedule_new_orchestration(
|
||||
orchestrator="content_generation_hitl_orchestration",
|
||||
input=payload,
|
||||
)
|
||||
|
||||
assert instance_id is not None
|
||||
|
||||
# Wait for orchestration to reach notification point
|
||||
notification_received = self.orch_helper.wait_for_notification(instance_id, timeout_seconds=90)
|
||||
assert notification_received, "Failed to receive notification from orchestration"
|
||||
|
||||
# Send approval event
|
||||
approval_data = {"approved": True, "feedback": ""}
|
||||
self.orch_helper.raise_event(
|
||||
instance_id=instance_id,
|
||||
event_name=HUMAN_APPROVAL_EVENT,
|
||||
event_data=approval_data,
|
||||
)
|
||||
|
||||
# Wait for completion
|
||||
metadata = self.orch_helper.wait_for_orchestration(
|
||||
instance_id=instance_id,
|
||||
timeout=90.0,
|
||||
)
|
||||
|
||||
assert metadata is not None
|
||||
assert metadata.runtime_status == OrchestrationStatus.COMPLETED
|
||||
|
||||
def test_hitl_orchestration_with_rejection_and_feedback(self):
|
||||
"""Test HITL orchestration with rejection and iterative refinement."""
|
||||
payload = {
|
||||
"topic": "Artificial Intelligence in healthcare",
|
||||
"max_review_attempts": 3,
|
||||
"approval_timeout_seconds": 60,
|
||||
}
|
||||
|
||||
# Start the orchestration
|
||||
instance_id = self.dts_client.schedule_new_orchestration(
|
||||
orchestrator="content_generation_hitl_orchestration",
|
||||
input=payload,
|
||||
)
|
||||
|
||||
# Wait for orchestration to reach notification point
|
||||
notification_received = self.orch_helper.wait_for_notification(instance_id, timeout_seconds=90)
|
||||
assert notification_received, "Failed to receive notification from orchestration"
|
||||
|
||||
# First rejection with feedback
|
||||
rejection_data = {
|
||||
"approved": False,
|
||||
"feedback": "Please make it more concise and add specific examples.",
|
||||
}
|
||||
self.orch_helper.raise_event(
|
||||
instance_id=instance_id,
|
||||
event_name=HUMAN_APPROVAL_EVENT,
|
||||
event_data=rejection_data,
|
||||
)
|
||||
|
||||
# Wait for orchestration to refine and reach notification point again
|
||||
notification_received = self.orch_helper.wait_for_notification(instance_id, timeout_seconds=90)
|
||||
assert notification_received, "Failed to receive notification after refinement"
|
||||
|
||||
# Second approval
|
||||
approval_data = {"approved": True, "feedback": ""}
|
||||
self.orch_helper.raise_event(
|
||||
instance_id=instance_id,
|
||||
event_name=HUMAN_APPROVAL_EVENT,
|
||||
event_data=approval_data,
|
||||
)
|
||||
|
||||
# Wait for completion
|
||||
metadata = self.orch_helper.wait_for_orchestration(
|
||||
instance_id=instance_id,
|
||||
timeout=90.0,
|
||||
)
|
||||
|
||||
assert metadata is not None
|
||||
assert metadata.runtime_status == OrchestrationStatus.COMPLETED
|
||||
|
||||
def test_hitl_orchestration_timeout(self):
|
||||
"""Test HITL orchestration timeout behavior."""
|
||||
payload = {
|
||||
"topic": "Cloud computing fundamentals",
|
||||
"max_review_attempts": 1,
|
||||
"approval_timeout_seconds": 0.1, # Short timeout for testing
|
||||
}
|
||||
|
||||
# Start the orchestration
|
||||
instance_id = self.dts_client.schedule_new_orchestration(
|
||||
orchestrator="content_generation_hitl_orchestration",
|
||||
input=payload,
|
||||
)
|
||||
|
||||
# Don't send any approval - let it timeout
|
||||
# The orchestration should fail due to timeout
|
||||
try:
|
||||
metadata = self.orch_helper.wait_for_orchestration(
|
||||
instance_id=instance_id,
|
||||
timeout=90.0,
|
||||
)
|
||||
# If it completes, it should be failed status due to timeout
|
||||
assert metadata.runtime_status == OrchestrationStatus.FAILED
|
||||
except (RuntimeError, TimeoutError):
|
||||
# Expected - orchestration should timeout and fail
|
||||
pass
|
||||
@@ -0,0 +1,90 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
"""Integration tests for the standalone durabletask workflow sample (08_workflow).
|
||||
|
||||
Exercises the standalone (non-Azure-Functions) workflow path:
|
||||
- ``DurableAIAgentWorker.configure_workflow`` auto-registers the agent entities,
|
||||
non-agent executor activities, and the workflow orchestrator.
|
||||
- A client starts the workflow by scheduling its ``dafx-{workflow_name}`` orchestration.
|
||||
- Conditional routing sends spam to a non-agent handler and legitimate email
|
||||
through a second agent and a sender executor.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Any, Protocol
|
||||
|
||||
import pytest
|
||||
from durabletask.client import OrchestrationStatus
|
||||
|
||||
from agent_framework_durabletask import DurableAIAgentClient, workflow_orchestrator_name
|
||||
|
||||
# Must match the workflow name in samples/04-hosting/durabletask/08_workflow/worker.py
|
||||
WORKFLOW_NAME = "email_triage"
|
||||
|
||||
logging.basicConfig(level=logging.WARNING)
|
||||
|
||||
|
||||
class AgentClientFactoryProtocol(Protocol):
|
||||
"""Protocol for the agent client factory fixture."""
|
||||
|
||||
@classmethod
|
||||
def create(cls, max_poll_retries: int = 90) -> tuple[Any, DurableAIAgentClient]: ...
|
||||
|
||||
|
||||
# Module-level markers
|
||||
pytestmark = [
|
||||
pytest.mark.flaky,
|
||||
pytest.mark.integration,
|
||||
pytest.mark.sample("08_workflow"),
|
||||
pytest.mark.integration_test,
|
||||
pytest.mark.requires_dts,
|
||||
]
|
||||
|
||||
|
||||
class TestStandaloneWorkflow:
|
||||
"""Standalone (non-Azure-Functions) workflow execution on a durabletask worker."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup(self, agent_client_factory: type[AgentClientFactoryProtocol], orchestration_helper) -> None:
|
||||
"""Provide a DTS client and orchestration helper for each test."""
|
||||
self.dts_client, self.agent_client = agent_client_factory.create()
|
||||
self.orch_helper = orchestration_helper
|
||||
|
||||
def test_legitimate_email_drafts_response(self) -> None:
|
||||
"""A legitimate email routes through the email agent and is 'sent'."""
|
||||
instance_id = self.dts_client.schedule_new_orchestration(
|
||||
orchestrator=workflow_orchestrator_name(WORKFLOW_NAME),
|
||||
input=(
|
||||
"Hi team, just a reminder about our sprint planning meeting tomorrow at 10 AM. "
|
||||
"Please review the agenda in Jira."
|
||||
),
|
||||
)
|
||||
|
||||
metadata, output = self.orch_helper.wait_for_orchestration_with_output(
|
||||
instance_id=instance_id,
|
||||
timeout=180.0,
|
||||
)
|
||||
|
||||
assert metadata.runtime_status == OrchestrationStatus.COMPLETED
|
||||
assert output is not None
|
||||
assert "Email sent" in str(output)
|
||||
|
||||
def test_spam_email_handled(self) -> None:
|
||||
"""A spam email routes to the non-agent spam handler."""
|
||||
instance_id = self.dts_client.schedule_new_orchestration(
|
||||
orchestrator=workflow_orchestrator_name(WORKFLOW_NAME),
|
||||
input="URGENT! You've won $1,000,000! Click here now to claim your prize! Limited time offer!",
|
||||
)
|
||||
|
||||
metadata, output = self.orch_helper.wait_for_orchestration_with_output(
|
||||
instance_id=instance_id,
|
||||
timeout=180.0,
|
||||
)
|
||||
|
||||
assert metadata.runtime_status == OrchestrationStatus.COMPLETED
|
||||
assert output is not None
|
||||
assert "spam" in str(output).lower()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
@@ -0,0 +1,112 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
"""Integration tests for the standalone durabletask HITL workflow sample (09_workflow_hitl).
|
||||
|
||||
Exercises the human-in-the-loop workflow path on a standalone durabletask worker:
|
||||
- The ``InputRouter`` start executor receives a typed ``ContentSubmission`` that the
|
||||
shared engine reconstructs from the client's JSON payload (no manual parsing).
|
||||
- An analysis agent produces a recommendation, then the workflow pauses for human
|
||||
approval via ``request_info``.
|
||||
- The client retrieves the pending request, replies with ``send_hitl_response``, and
|
||||
the workflow resumes to an approved/rejected outcome read via ``await_workflow_output``.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from agent_framework_durabletask import DurableWorkflowClient
|
||||
|
||||
logging.basicConfig(level=logging.WARNING)
|
||||
|
||||
# Must match the workflow name in samples/04-hosting/durabletask/09_workflow_hitl/worker.py
|
||||
WORKFLOW_NAME = "content_moderation"
|
||||
|
||||
# Module-level markers
|
||||
pytestmark = [
|
||||
pytest.mark.flaky,
|
||||
pytest.mark.integration,
|
||||
pytest.mark.sample("09_workflow_hitl"),
|
||||
pytest.mark.integration_test,
|
||||
pytest.mark.requires_dts,
|
||||
pytest.mark.requires_azure_openai,
|
||||
]
|
||||
|
||||
|
||||
def _wait_for_hitl_request(
|
||||
client: DurableWorkflowClient, instance_id: str, timeout_seconds: int = 90
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Poll until the workflow records at least one pending HITL request."""
|
||||
deadline = time.time() + timeout_seconds
|
||||
while time.time() < deadline:
|
||||
pending = client.get_pending_hitl_requests(instance_id, workflow_name=WORKFLOW_NAME)
|
||||
if pending:
|
||||
return pending
|
||||
time.sleep(2)
|
||||
raise AssertionError(f"Timed out waiting for a HITL request on instance {instance_id}")
|
||||
|
||||
|
||||
class TestStandaloneWorkflowHITL:
|
||||
"""Human-in-the-loop workflow execution on a standalone durabletask worker."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup(self, workflow_client: DurableWorkflowClient) -> None:
|
||||
"""Bind the DurableWorkflowClient for the current sample worker."""
|
||||
self.client = workflow_client
|
||||
|
||||
def _run_case(self, submission: dict[str, Any], *, approve: bool) -> Any:
|
||||
"""Start a moderation case, answer the HITL pause, and return the final output."""
|
||||
instance_id = self.client.start_workflow(input=submission, workflow_name=WORKFLOW_NAME)
|
||||
|
||||
pending = _wait_for_hitl_request(self.client, instance_id)
|
||||
request = pending[0]
|
||||
assert request["request_id"]
|
||||
assert request["source_executor_id"]
|
||||
|
||||
self.client.send_hitl_response(
|
||||
instance_id,
|
||||
request["request_id"],
|
||||
{"approved": approve, "reviewer_notes": "Looks good." if approve else "Violates content policy."},
|
||||
workflow_name=WORKFLOW_NAME,
|
||||
)
|
||||
|
||||
return self.client.await_workflow_output(instance_id, workflow_name=WORKFLOW_NAME, timeout_seconds=180)
|
||||
|
||||
def test_hitl_workflow_approval(self) -> None:
|
||||
"""Appropriate content is approved after the reviewer says yes."""
|
||||
output = self._run_case(
|
||||
{
|
||||
"content_id": "article-001",
|
||||
"title": "Introduction to AI in Healthcare",
|
||||
"body": (
|
||||
"Artificial intelligence is improving healthcare by enabling faster diagnosis, "
|
||||
"personalized treatment plans, and better patient outcomes."
|
||||
),
|
||||
"author": "Dr. Jane Smith",
|
||||
},
|
||||
approve=True,
|
||||
)
|
||||
|
||||
assert output is not None
|
||||
assert "APPROVED" in str(output).upper()
|
||||
|
||||
def test_hitl_workflow_rejection(self) -> None:
|
||||
"""Spammy content is rejected after the reviewer says no."""
|
||||
output = self._run_case(
|
||||
{
|
||||
"content_id": "article-002",
|
||||
"title": "Get Rich Quick",
|
||||
"body": "Click here NOW to make $10,000 overnight! GUARANTEED! Limited time offer!",
|
||||
"author": "Definitely Not Spam",
|
||||
},
|
||||
approve=False,
|
||||
)
|
||||
|
||||
assert output is not None
|
||||
assert "REJECTED" in str(output).upper()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
@@ -0,0 +1,72 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
"""Integration tests for the composed sub-workflow sample (11_subworkflow).
|
||||
|
||||
Exercises workflow *composition* on a standalone durabletask worker:
|
||||
- An outer ``review_pipeline`` embeds an inner ``sentiment_analysis`` workflow via a
|
||||
``WorkflowExecutor`` node (``sentiment_sub``).
|
||||
- ``DurableAIAgentWorker.configure_workflow`` walks the composition and registers a
|
||||
durable orchestration for each workflow; the inner workflow runs as a child
|
||||
orchestration when the outer reaches the ``WorkflowExecutor`` node.
|
||||
- The inner workflow's output (a sentiment summary) is forwarded to the outer
|
||||
``reporter`` executor, which produces the final result.
|
||||
|
||||
The inner workflow hosts an AI agent, so these tests require model credentials.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from agent_framework_durabletask import DurableWorkflowClient
|
||||
|
||||
logging.basicConfig(level=logging.WARNING)
|
||||
|
||||
# Must match the outer workflow name in samples/04-hosting/durabletask/11_subworkflow/worker.py
|
||||
WORKFLOW_NAME = "review_pipeline"
|
||||
|
||||
# Module-level markers
|
||||
pytestmark = [
|
||||
pytest.mark.flaky,
|
||||
pytest.mark.integration,
|
||||
pytest.mark.sample("11_subworkflow"),
|
||||
pytest.mark.integration_test,
|
||||
pytest.mark.requires_dts,
|
||||
pytest.mark.requires_azure_openai,
|
||||
]
|
||||
|
||||
|
||||
class TestSubworkflowComposition:
|
||||
"""Composed (outer + inner) workflow execution on a standalone durabletask worker."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup(self, workflow_client: DurableWorkflowClient) -> None:
|
||||
"""Bind the DurableWorkflowClient for the current sample worker."""
|
||||
self.client = workflow_client
|
||||
|
||||
def _run(self, review: str) -> Any:
|
||||
"""Run the composed workflow with a review and return its final output."""
|
||||
instance_id = self.client.start_workflow(input=review, workflow_name=WORKFLOW_NAME)
|
||||
return self.client.await_workflow_output(instance_id, workflow_name=WORKFLOW_NAME, timeout_seconds=180)
|
||||
|
||||
def test_positive_review_runs_through_subworkflow(self) -> None:
|
||||
"""A positive review flows through the embedded sentiment sub-workflow to a report."""
|
||||
output = self._run(
|
||||
"Absolutely love this espresso machine - it heats up fast and the coffee is consistently great."
|
||||
)
|
||||
|
||||
assert output is not None
|
||||
# The outer reporter wraps the inner sub-workflow's forwarded sentiment summary.
|
||||
assert "sentiment" in str(output).lower()
|
||||
|
||||
def test_negative_review_runs_through_subworkflow(self) -> None:
|
||||
"""A negative review also completes the composed pipeline end-to-end."""
|
||||
output = self._run("Disappointed. The device stopped working after two weeks and support never replied.")
|
||||
|
||||
assert output is not None
|
||||
assert "sentiment" in str(output).lower()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
@@ -0,0 +1,152 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
"""Integration tests for the composed sub-workflow HITL sample (12_subworkflow_hitl).
|
||||
|
||||
Exercises human-in-the-loop **inside a nested sub-workflow** on a standalone
|
||||
durabletask worker:
|
||||
- An outer ``moderation_pipeline`` embeds an inner ``human_review`` workflow via a
|
||||
``WorkflowExecutor`` node (``review_sub``); on the durable host the inner workflow
|
||||
runs as a child orchestration.
|
||||
- The inner ``review_gate`` pauses via ``request_info``. The pending request surfaces
|
||||
at the top-level instance with a **qualified** id ``review_sub~0~{requestId}`` (the
|
||||
``~{ordinal}~`` hop addresses the specific child the node dispatched).
|
||||
- The client responds with that qualified id against the *top-level* instance and the
|
||||
host routes it to the owning child orchestration, resuming to an approved/rejected
|
||||
outcome.
|
||||
|
||||
This sample hosts **no AI agents**, so it needs only the DTS emulator (no model
|
||||
credentials), which makes it a deterministic end-to-end check of the nested-HITL
|
||||
addressing.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from agent_framework_durabletask import DurableWorkflowClient
|
||||
from agent_framework_durabletask._workflows.naming import SUBWORKFLOW_REQUEST_SEPARATOR
|
||||
|
||||
logging.basicConfig(level=logging.WARNING)
|
||||
|
||||
# Must match the outer workflow name in samples/04-hosting/durabletask/12_subworkflow_hitl/worker.py
|
||||
WORKFLOW_NAME = "moderation_pipeline"
|
||||
# The WorkflowExecutor node id that embeds the inner HITL workflow.
|
||||
SUBWORKFLOW_NODE_ID = "review_sub"
|
||||
|
||||
# Module-level markers. No requires_azure_openai: the sample hosts no agents.
|
||||
pytestmark = [
|
||||
pytest.mark.flaky,
|
||||
pytest.mark.integration,
|
||||
pytest.mark.sample("12_subworkflow_hitl"),
|
||||
pytest.mark.integration_test,
|
||||
pytest.mark.requires_dts,
|
||||
]
|
||||
|
||||
|
||||
def _wait_for_hitl_request(
|
||||
client: DurableWorkflowClient, instance_id: str, timeout_seconds: int = 90
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Poll until the workflow (or a nested sub-workflow) records a pending HITL request."""
|
||||
deadline = time.time() + timeout_seconds
|
||||
while time.time() < deadline:
|
||||
pending = client.get_pending_hitl_requests(instance_id, workflow_name=WORKFLOW_NAME)
|
||||
if pending:
|
||||
return pending
|
||||
time.sleep(2)
|
||||
raise AssertionError(f"Timed out waiting for a nested HITL request on instance {instance_id}")
|
||||
|
||||
|
||||
class TestSubworkflowHITL:
|
||||
"""Nested (sub-workflow) human-in-the-loop on a standalone durabletask worker."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup(self, workflow_client: DurableWorkflowClient) -> None:
|
||||
"""Bind the DurableWorkflowClient for the current sample worker."""
|
||||
self.client = workflow_client
|
||||
|
||||
def _run_case(self, submission: dict[str, Any], *, approve: bool) -> tuple[dict[str, Any], Any]:
|
||||
"""Start a moderation case, answer the nested HITL pause, return (request, output)."""
|
||||
instance_id = self.client.start_workflow(input=submission, workflow_name=WORKFLOW_NAME)
|
||||
|
||||
pending = _wait_for_hitl_request(self.client, instance_id)
|
||||
request = pending[0]
|
||||
|
||||
self.client.send_hitl_response(
|
||||
instance_id,
|
||||
request["request_id"],
|
||||
{"approved": approve, "reviewer_notes": "Looks good." if approve else "Violates content policy."},
|
||||
workflow_name=WORKFLOW_NAME,
|
||||
)
|
||||
|
||||
output = self.client.await_workflow_output(instance_id, workflow_name=WORKFLOW_NAME, timeout_seconds=180)
|
||||
return request, output
|
||||
|
||||
def test_nested_request_id_is_qualified_with_ordinal(self) -> None:
|
||||
"""The nested pending request surfaces with a ``review_sub~0~{id}`` qualified id."""
|
||||
instance_id = self.client.start_workflow(
|
||||
input={
|
||||
"content_id": "article-100",
|
||||
"title": "Quarterly Roadmap",
|
||||
"body": "A summary of the upcoming features planned for the next quarter.",
|
||||
},
|
||||
workflow_name=WORKFLOW_NAME,
|
||||
)
|
||||
|
||||
pending = _wait_for_hitl_request(self.client, instance_id)
|
||||
|
||||
assert len(pending) == 1
|
||||
request = pending[0]
|
||||
# The qualifier carries the node id and the child's ordinal (0 for the single
|
||||
# dispatch), then the inner bare request id: ``review_sub~0~{requestId}``.
|
||||
expected_prefix = f"{SUBWORKFLOW_NODE_ID}{SUBWORKFLOW_REQUEST_SEPARATOR}0{SUBWORKFLOW_REQUEST_SEPARATOR}"
|
||||
assert request["request_id"].startswith(expected_prefix), request["request_id"]
|
||||
# The bare inner id is non-empty after the qualifier.
|
||||
assert request["request_id"][len(expected_prefix) :]
|
||||
# The originating executor is the inner workflow's review gate.
|
||||
assert request["source_executor_id"] == "review_gate"
|
||||
|
||||
# Drain the pause so the worker does not leave the instance hanging.
|
||||
self.client.send_hitl_response(
|
||||
instance_id,
|
||||
request["request_id"],
|
||||
{"approved": True, "reviewer_notes": "ok"},
|
||||
workflow_name=WORKFLOW_NAME,
|
||||
)
|
||||
self.client.await_workflow_output(instance_id, workflow_name=WORKFLOW_NAME, timeout_seconds=180)
|
||||
|
||||
def test_nested_hitl_approval(self) -> None:
|
||||
"""Responding 'approved' to the nested request resumes the outer workflow to APPROVED."""
|
||||
_request, output = self._run_case(
|
||||
{
|
||||
"content_id": "article-001",
|
||||
"title": "Introduction to AI in Healthcare",
|
||||
"body": (
|
||||
"Artificial intelligence is improving healthcare by enabling faster diagnosis, "
|
||||
"personalized treatment plans, and better patient outcomes."
|
||||
),
|
||||
},
|
||||
approve=True,
|
||||
)
|
||||
|
||||
assert output is not None
|
||||
assert "APPROVED" in str(output).upper()
|
||||
|
||||
def test_nested_hitl_rejection(self) -> None:
|
||||
"""Responding 'rejected' to the nested request resumes the outer workflow to REJECTED."""
|
||||
_request, output = self._run_case(
|
||||
{
|
||||
"content_id": "article-002",
|
||||
"title": "Get Rich Quick",
|
||||
"body": "Click here NOW to make $10,000 overnight! GUARANTEED! Limited time offer!",
|
||||
},
|
||||
approve=False,
|
||||
)
|
||||
|
||||
assert output is not None
|
||||
assert "REJECTED" in str(output).upper()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
Reference in New Issue
Block a user