chore: import upstream snapshot with attribution
dotnet-build-and-test / dotnet-test-functions (push) Has been cancelled
dotnet-build-and-test / paths-filter (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Debug, windows-latest, net9.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Release, ubuntu-latest, net10.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Release, ubuntu-latest, net8.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Release, windows-latest, net472) (push) Has been cancelled
dotnet-build-and-test / dotnet-test (Release, integration, true, ubuntu-latest, net10.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-test (Release, integration, true, windows-latest, net472) (push) Has been cancelled
dotnet-build-and-test / dotnet-foundry-hosted-it (push) Has been cancelled
dotnet-build-and-test / dotnet-build-and-test-check (push) Has been cancelled
dotnet-build-and-test / Integration Test Report (push) Has been cancelled
CodeQL / Analyze (csharp) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
dotnet-build-and-test / dotnet-test-functions (push) Has been cancelled
dotnet-build-and-test / paths-filter (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Debug, windows-latest, net9.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Release, ubuntu-latest, net10.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Release, ubuntu-latest, net8.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-build (Release, windows-latest, net472) (push) Has been cancelled
dotnet-build-and-test / dotnet-test (Release, integration, true, ubuntu-latest, net10.0) (push) Has been cancelled
dotnet-build-and-test / dotnet-test (Release, integration, true, windows-latest, net472) (push) Has been cancelled
dotnet-build-and-test / dotnet-foundry-hosted-it (push) Has been cancelled
dotnet-build-and-test / dotnet-build-and-test-check (push) Has been cancelled
dotnet-build-and-test / Integration Test Report (push) Has been cancelled
CodeQL / Analyze (csharp) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,10 @@
|
||||
# Azure OpenAI Configuration
|
||||
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
|
||||
AZURE_OPENAI_MODEL=your-deployment-name
|
||||
FUNCTIONS_WORKER_RUNTIME=python
|
||||
|
||||
# Azure Functions Configuration
|
||||
AzureWebJobsStorage=UseDevelopmentStorage=true
|
||||
DURABLE_TASK_SCHEDULER_CONNECTION_STRING=Endpoint=http://localhost:8080;Authentication=None
|
||||
|
||||
# Note: TASKHUB_NAME is not required for integration tests; it is auto-generated per test run.
|
||||
@@ -0,0 +1,81 @@
|
||||
# Sample Integration Tests
|
||||
|
||||
Integration tests that validate the Durable Agent Framework samples by running them as Azure Functions.
|
||||
|
||||
## Setup
|
||||
|
||||
### 1. Create `.env` file
|
||||
|
||||
Copy `.env.example` to `.env` and fill in your Azure credentials:
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
Required variables:
|
||||
- `AZURE_OPENAI_ENDPOINT`
|
||||
- `AZURE_OPENAI_MODEL`
|
||||
- `AZURE_OPENAI_API_KEY`
|
||||
- `AzureWebJobsStorage`
|
||||
- `DURABLE_TASK_SCHEDULER_CONNECTION_STRING`
|
||||
- `FUNCTIONS_WORKER_RUNTIME`
|
||||
|
||||
### 2. Start required services
|
||||
|
||||
**Azurite (for orchestration tests):**
|
||||
```bash
|
||||
docker run -d -p 10000:10000 -p 10001:10001 -p 10002:10002 mcr.microsoft.com/azure-storage/azurite
|
||||
```
|
||||
|
||||
**Durable Task Scheduler:**
|
||||
```bash
|
||||
docker run -d -p 8080:8080 -p 8082:8082 -e DTS_USE_DYNAMIC_TASK_HUBS=true mcr.microsoft.com/dts/dts-emulator:latest
|
||||
```
|
||||
|
||||
## Running Tests
|
||||
|
||||
The tests automatically start and stop the Azure Functions app for each sample.
|
||||
|
||||
### Run all sample tests
|
||||
```bash
|
||||
uv run pytest packages/azurefunctions/tests/integration_tests -v
|
||||
```
|
||||
|
||||
### Run specific sample
|
||||
```bash
|
||||
uv run pytest packages/azurefunctions/tests/integration_tests/test_01_single_agent.py -v
|
||||
```
|
||||
|
||||
### Run with verbose output
|
||||
```bash
|
||||
uv run pytest packages/azurefunctions/tests/integration_tests -sv
|
||||
```
|
||||
|
||||
## How It Works
|
||||
|
||||
Each test file uses pytest markers to automatically configure and start the function app:
|
||||
|
||||
```python
|
||||
pytestmark = [
|
||||
pytest.mark.sample("01_single_agent"),
|
||||
pytest.mark.usefixtures("function_app_for_test"),
|
||||
skip_if_azure_functions_integration_tests_disabled,
|
||||
]
|
||||
```
|
||||
|
||||
The `function_app_for_test` fixture:
|
||||
1. Loads environment variables from `.env`
|
||||
2. Validates required variables are present
|
||||
3. Starts the function app on a dynamically allocated port
|
||||
4. Waits for the app to be ready
|
||||
5. Runs your tests
|
||||
6. Tears down the function app
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
|
||||
**Missing environment variables:**
|
||||
Ensure your `.env` file contains all required variables from `.env.example`.
|
||||
|
||||
**Tests timeout:**
|
||||
Check that Azure OpenAI credentials are valid and the service is accessible.
|
||||
@@ -0,0 +1,613 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
"""
|
||||
Pytest configuration for Azure Functions integration tests.
|
||||
|
||||
This module provides fixtures, configuration, and test utilities for pytest.
|
||||
"""
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import uuid
|
||||
from collections.abc import Iterator, Mapping
|
||||
from contextlib import suppress
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
import requests
|
||||
|
||||
# =============================================================================
|
||||
# Configuration Constants
|
||||
# =============================================================================
|
||||
|
||||
TIMEOUT = 30 # seconds
|
||||
ORCHESTRATION_TIMEOUT = 180 # seconds for orchestrations
|
||||
_DEFAULT_HOST = "localhost"
|
||||
|
||||
# Emulator ports (match CI workflow configuration)
|
||||
_AZURITE_BLOB_PORT = 10000
|
||||
_DTS_EMULATOR_PORT = 8080
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Exceptions
|
||||
# =============================================================================
|
||||
|
||||
|
||||
class FunctionAppStartupError(RuntimeError):
|
||||
"""Raised when the Azure Functions host fails to start reliably."""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Environment and Service Checks
|
||||
# =============================================================================
|
||||
|
||||
|
||||
def _load_env_file_if_present() -> None:
|
||||
"""Load environment variables from the local .env file when available."""
|
||||
env_file = Path(__file__).parent / ".env"
|
||||
if not env_file.exists():
|
||||
return
|
||||
|
||||
try:
|
||||
from dotenv import load_dotenv
|
||||
|
||||
load_dotenv(env_file)
|
||||
except ImportError:
|
||||
# python-dotenv not available; rely on existing environment
|
||||
pass
|
||||
|
||||
|
||||
def _check_func_cli_available() -> bool:
|
||||
"""Check if Azure Functions Core Tools (func) is installed and available."""
|
||||
return shutil.which("func") is not None
|
||||
|
||||
|
||||
def _check_port_listening(port: int, host: str = _DEFAULT_HOST) -> bool:
|
||||
"""Check if a service is listening on the given port."""
|
||||
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
|
||||
sock.settimeout(1)
|
||||
return sock.connect_ex((host, port)) == 0
|
||||
|
||||
|
||||
def _check_azurite_available() -> bool:
|
||||
"""Check if Azurite (Azure Storage emulator) is available on the expected port."""
|
||||
return _check_port_listening(_AZURITE_BLOB_PORT)
|
||||
|
||||
|
||||
def _check_dts_emulator_available() -> bool:
|
||||
"""Check if Durable Task Scheduler emulator is available on the expected port."""
|
||||
return _check_port_listening(_DTS_EMULATOR_PORT)
|
||||
|
||||
|
||||
def _should_skip_azure_functions_integration_tests() -> tuple[bool, str]:
|
||||
"""Determine whether Azure Functions integration tests should be skipped."""
|
||||
_load_env_file_if_present()
|
||||
|
||||
# Check for Azure Functions Core Tools
|
||||
if not _check_func_cli_available():
|
||||
return (
|
||||
True,
|
||||
"Azure Functions Core Tools (func) not installed. Install with: npm install -g azure-functions-core-tools@4", # noqa: E501
|
||||
)
|
||||
|
||||
# Check for Azurite (Azure Storage emulator)
|
||||
if not _check_azurite_available():
|
||||
return (
|
||||
True,
|
||||
f"Azurite not running on port {_AZURITE_BLOB_PORT}. Start with: docker run -d -p 10000:10000 -p 10001:10001 -p 10002:10002 mcr.microsoft.com/azure-storage/azurite", # noqa: E501
|
||||
)
|
||||
|
||||
# Check for Durable Task Scheduler emulator
|
||||
if not _check_dts_emulator_available():
|
||||
return (
|
||||
True,
|
||||
f"Durable Task Scheduler emulator not running on port {_DTS_EMULATOR_PORT}. Start with: docker run -d -p 8080:8080 -p 8082:8082 mcr.microsoft.com/dts/dts-emulator:latest", # noqa: E501
|
||||
)
|
||||
|
||||
has_foundry_config = bool(os.getenv("FOUNDRY_PROJECT_ENDPOINT", "").strip()) and bool(
|
||||
os.getenv("FOUNDRY_MODEL", "").strip()
|
||||
)
|
||||
has_azure_openai_config = bool(os.getenv("AZURE_OPENAI_ENDPOINT", "").strip()) and bool(
|
||||
os.getenv("AZURE_OPENAI_MODEL", "").strip()
|
||||
)
|
||||
if not has_foundry_config and not has_azure_openai_config:
|
||||
return (
|
||||
True,
|
||||
"No real FOUNDRY_* or AZURE_OPENAI_* configuration provided; skipping integration tests.",
|
||||
)
|
||||
|
||||
return False, "Integration tests enabled."
|
||||
|
||||
|
||||
_SKIP_AZURE_FUNCTIONS_INTEGRATION_TESTS, _AZURE_FUNCTIONS_SKIP_REASON = _should_skip_azure_functions_integration_tests()
|
||||
|
||||
skip_if_azure_functions_integration_tests_disabled = pytest.mark.skipif(
|
||||
_SKIP_AZURE_FUNCTIONS_INTEGRATION_TESTS,
|
||||
reason=_AZURE_FUNCTIONS_SKIP_REASON,
|
||||
)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Test Helper Class
|
||||
# =============================================================================
|
||||
|
||||
|
||||
class SampleTestHelper:
|
||||
"""Helper class for testing samples."""
|
||||
|
||||
@staticmethod
|
||||
def post_json(url: str, data: dict[str, Any], timeout: int = TIMEOUT) -> requests.Response:
|
||||
"""POST JSON data to a URL."""
|
||||
return requests.post(url, json=data, headers={"Content-Type": "application/json"}, timeout=timeout)
|
||||
|
||||
@staticmethod
|
||||
def post_text(url: str, text: str, timeout: int = TIMEOUT) -> requests.Response:
|
||||
"""POST plain text to a URL."""
|
||||
return requests.post(url, data=text, headers={"Content-Type": "text/plain"}, timeout=timeout)
|
||||
|
||||
@staticmethod
|
||||
def get(url: str, timeout: int = TIMEOUT) -> requests.Response:
|
||||
"""GET request to a URL."""
|
||||
return requests.get(url, timeout=timeout)
|
||||
|
||||
@staticmethod
|
||||
def wait_for_orchestration(
|
||||
status_url: str, max_wait: int = ORCHESTRATION_TIMEOUT, poll_interval: int = 2
|
||||
) -> dict[str, Any]:
|
||||
"""Wait for an orchestration to complete.
|
||||
|
||||
Args:
|
||||
status_url: URL to poll for orchestration status
|
||||
max_wait: Maximum seconds to wait
|
||||
poll_interval: Seconds between polls
|
||||
|
||||
Returns:
|
||||
Final orchestration status
|
||||
|
||||
Raises:
|
||||
TimeoutError: If orchestration doesn't complete in time
|
||||
"""
|
||||
start_time = time.time()
|
||||
while time.time() - start_time < max_wait:
|
||||
response = requests.get(status_url, timeout=TIMEOUT)
|
||||
response.raise_for_status()
|
||||
status = response.json()
|
||||
|
||||
runtime_status = status.get("runtimeStatus", "")
|
||||
if runtime_status in ["Completed", "Failed", "Terminated"]:
|
||||
return status
|
||||
|
||||
time.sleep(poll_interval)
|
||||
|
||||
raise TimeoutError(f"Orchestration did not complete within {max_wait} seconds")
|
||||
|
||||
@staticmethod
|
||||
def wait_for_orchestration_with_output(
|
||||
status_url: str, max_wait: int = ORCHESTRATION_TIMEOUT, poll_interval: int = 2
|
||||
) -> dict[str, Any]:
|
||||
"""Wait for an orchestration to complete and have output available.
|
||||
|
||||
This is a specialized version of wait_for_orchestration that also
|
||||
ensures the output field is present, handling timing race conditions.
|
||||
|
||||
Args:
|
||||
status_url: URL to poll for orchestration status
|
||||
max_wait: Maximum seconds to wait
|
||||
poll_interval: Seconds between polls
|
||||
|
||||
Returns:
|
||||
Final orchestration status with output
|
||||
|
||||
Raises:
|
||||
TimeoutError: If orchestration doesn't complete with output in time
|
||||
"""
|
||||
start_time = time.time()
|
||||
while time.time() - start_time < max_wait:
|
||||
response = requests.get(status_url, timeout=TIMEOUT)
|
||||
response.raise_for_status()
|
||||
status = response.json()
|
||||
|
||||
runtime_status = status.get("runtimeStatus", "")
|
||||
if runtime_status in ["Failed", "Terminated"]:
|
||||
return status
|
||||
if runtime_status == "Completed" and status.get("output"):
|
||||
return status
|
||||
# If completed but no output, continue polling for a bit more to
|
||||
# handle the race condition where output has not been persisted yet.
|
||||
|
||||
time.sleep(poll_interval)
|
||||
|
||||
# Provide detailed error message based on final status
|
||||
final_response = requests.get(status_url, timeout=TIMEOUT)
|
||||
final_response.raise_for_status()
|
||||
final_status = final_response.json()
|
||||
final_runtime_status = final_status.get("runtimeStatus", "Unknown")
|
||||
|
||||
if final_runtime_status == "Completed":
|
||||
if "output" not in final_status:
|
||||
raise TimeoutError(
|
||||
"Orchestration completed but 'output' field is missing after "
|
||||
f"{max_wait} seconds. Final status: {final_status}"
|
||||
)
|
||||
if not final_status["output"]:
|
||||
raise TimeoutError(
|
||||
"Orchestration completed but output is empty after "
|
||||
f"{max_wait} seconds. Final status: {final_status}"
|
||||
)
|
||||
raise TimeoutError(
|
||||
"Orchestration completed with output but validation failed after "
|
||||
f"{max_wait} seconds. Final status: {final_status}"
|
||||
)
|
||||
raise TimeoutError(
|
||||
"Orchestration did not complete within "
|
||||
f"{max_wait} seconds. Final status: {final_runtime_status}, "
|
||||
f"Full status: {final_status}"
|
||||
)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Function App Lifecycle Management
|
||||
# =============================================================================
|
||||
|
||||
|
||||
def _resolve_repo_root() -> Path:
|
||||
"""Resolve the repository root, preferring GITHUB_WORKSPACE when available."""
|
||||
workspace = os.getenv("GITHUB_WORKSPACE")
|
||||
if workspace:
|
||||
candidate = Path(workspace).expanduser()
|
||||
if not (candidate / "samples").exists() and (candidate / "python" / "samples").exists():
|
||||
return (candidate / "python").resolve()
|
||||
return candidate.resolve()
|
||||
|
||||
# If `GITHUB_WORKSPACE` is not set,
|
||||
# go up from conftest.py -> integration_tests -> tests -> azurefunctions -> packages -> python
|
||||
return Path(__file__).resolve().parents[4]
|
||||
|
||||
|
||||
def _get_sample_path_from_marker(request: pytest.FixtureRequest) -> tuple[Path | None, str | None]:
|
||||
"""Get sample path from @pytest.mark.sample() marker.
|
||||
|
||||
Returns a tuple of (sample_path, error_message).
|
||||
If successful, error_message is None.
|
||||
If failed, sample_path is None and error_message contains the reason.
|
||||
"""
|
||||
marker = request.node.get_closest_marker("sample")
|
||||
|
||||
if not marker:
|
||||
return (
|
||||
None,
|
||||
(
|
||||
"No @pytest.mark.sample() marker found on test. Add pytestmark with "
|
||||
"@pytest.mark.sample('sample_name') to the test module."
|
||||
),
|
||||
)
|
||||
|
||||
if not marker.args:
|
||||
return (
|
||||
None,
|
||||
"@pytest.mark.sample() marker found but no sample name provided. Use @pytest.mark.sample('sample_name').",
|
||||
)
|
||||
|
||||
sample_name = marker.args[0]
|
||||
repo_root = _resolve_repo_root()
|
||||
sample_path = repo_root / "samples" / "04-hosting" / "azure_functions" / sample_name
|
||||
|
||||
if not sample_path.exists():
|
||||
return None, f"Sample directory does not exist: {sample_path}"
|
||||
|
||||
return sample_path, None
|
||||
|
||||
|
||||
def _find_available_port(host: str = _DEFAULT_HOST) -> int:
|
||||
"""Find an available TCP port on the given host."""
|
||||
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
|
||||
sock.bind((host, 0))
|
||||
return sock.getsockname()[1]
|
||||
|
||||
|
||||
def _build_base_url(port: int, host: str = _DEFAULT_HOST) -> str:
|
||||
"""Construct a base URL for the Azure Functions host."""
|
||||
return f"http://{host}:{port}"
|
||||
|
||||
|
||||
def _is_port_in_use(port: int, host: str = _DEFAULT_HOST) -> bool:
|
||||
"""Check if a port is already in use.
|
||||
|
||||
Returns True if the port is in use, False otherwise.
|
||||
"""
|
||||
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
|
||||
return sock.connect_ex((host, port)) == 0
|
||||
|
||||
|
||||
def _load_and_validate_env(sample_path: Path) -> None:
|
||||
"""Load .env file from current directory if it exists, then validate required environment variables.
|
||||
|
||||
Raises pytest.fail if required environment variables are missing.
|
||||
"""
|
||||
_load_env_file_if_present()
|
||||
# Required environment variables for Azure Functions samples
|
||||
required_env_vars = [
|
||||
"AzureWebJobsStorage",
|
||||
"DURABLE_TASK_SCHEDULER_CONNECTION_STRING",
|
||||
"FUNCTIONS_WORKER_RUNTIME",
|
||||
]
|
||||
# Samples that host no AI agents need no model credentials (only the DTS emulator
|
||||
# and Azurite). The suite-level gate still requires *some* LLM config to be present.
|
||||
no_llm_samples = {"13_subworkflow_hitl"}
|
||||
if sample_path.name in no_llm_samples:
|
||||
pass
|
||||
elif sample_path.name == "11_workflow_parallel":
|
||||
required_env_vars.extend(["AZURE_OPENAI_ENDPOINT", "AZURE_OPENAI_MODEL"])
|
||||
else:
|
||||
required_env_vars.extend(["FOUNDRY_PROJECT_ENDPOINT", "FOUNDRY_MODEL"])
|
||||
|
||||
# Check if required env vars are set
|
||||
missing_vars = [var for var in required_env_vars if not os.environ.get(var)]
|
||||
|
||||
if missing_vars:
|
||||
pytest.fail(
|
||||
f"Missing required environment variables: {', '.join(missing_vars)}. "
|
||||
"Please create a .env file in tests/integration_tests/ based on .env.example or "
|
||||
"set these variables in your environment."
|
||||
)
|
||||
|
||||
|
||||
def _start_function_app(sample_path: Path, port: int) -> subprocess.Popen[Any]:
|
||||
"""Start a function app in the specified sample directory.
|
||||
|
||||
Returns the subprocess.Popen object for the running process.
|
||||
"""
|
||||
env = os.environ.copy()
|
||||
# Use a unique TASKHUB_NAME for each test run to ensure test isolation.
|
||||
# This prevents conflicts between parallel or repeated test runs, as Durable Functions
|
||||
# use the task hub name to separate orchestration state.
|
||||
env["TASKHUB_NAME"] = f"test{uuid.uuid4().hex[:8]}"
|
||||
|
||||
# The Azure Functions Python worker's dependency isolation mechanism crashes
|
||||
# on Python 3.13 with a SIGSEGV in the protobuf C extension (google._upb).
|
||||
# Disabling isolation lets the worker load dependencies from the app's own
|
||||
# environment, which avoids the crash.
|
||||
# See: https://github.com/Azure/azure-functions-python-worker/issues/1797
|
||||
if sys.version_info >= (3, 13):
|
||||
env.setdefault("PYTHON_ISOLATE_WORKER_DEPENDENCIES", "0")
|
||||
|
||||
# On Windows, use CREATE_NEW_PROCESS_GROUP to allow proper termination
|
||||
# shell=True only on Windows to handle PATH resolution
|
||||
if sys.platform == "win32":
|
||||
return subprocess.Popen(
|
||||
["func", "start", "--port", str(port)],
|
||||
cwd=str(sample_path),
|
||||
creationflags=subprocess.CREATE_NEW_PROCESS_GROUP,
|
||||
shell=True,
|
||||
env=env,
|
||||
)
|
||||
# On Unix, use start_new_session=True to isolate the process group from the
|
||||
# pytest-xdist worker. Without this, signals (e.g. from test-timeout) can
|
||||
# propagate to the func host and vice-versa, potentially killing the worker.
|
||||
return subprocess.Popen(
|
||||
["func", "start", "--port", str(port)],
|
||||
cwd=str(sample_path),
|
||||
env=env,
|
||||
start_new_session=True,
|
||||
)
|
||||
|
||||
|
||||
def _wait_for_function_app_ready(func_process: subprocess.Popen[Any], port: int, max_wait: int = 60) -> None:
|
||||
"""Block until the Azure Functions host responds healthy or fail fast."""
|
||||
start_time = time.time()
|
||||
health_url = f"{_build_base_url(port)}/api/health"
|
||||
last_error: Exception | None = None
|
||||
|
||||
while time.time() - start_time < max_wait:
|
||||
# If the process exited early, capture any previously seen error and fail fast.
|
||||
if func_process.poll() is not None:
|
||||
raise FunctionAppStartupError(
|
||||
f"Function app process exited with code {func_process.returncode} before becoming healthy"
|
||||
) from last_error
|
||||
|
||||
if _is_port_in_use(port):
|
||||
try:
|
||||
response = requests.get(health_url, timeout=5)
|
||||
if response.status_code == 200:
|
||||
return
|
||||
last_error = RuntimeError(f"Health check returned {response.status_code}")
|
||||
except requests.RequestException as exc:
|
||||
last_error = exc
|
||||
|
||||
time.sleep(1)
|
||||
|
||||
raise FunctionAppStartupError(
|
||||
f"Function app did not become healthy on port {port} within {max_wait} seconds"
|
||||
) from last_error
|
||||
|
||||
|
||||
def _cleanup_function_app(func_process: subprocess.Popen[Any]) -> None:
|
||||
"""Clean up the function app process and all its children.
|
||||
|
||||
Uses psutil if available for more thorough cleanup, falls back to basic termination.
|
||||
"""
|
||||
try:
|
||||
import psutil
|
||||
|
||||
if func_process.poll() is None: # Process still running
|
||||
# Get parent process
|
||||
parent = psutil.Process(func_process.pid)
|
||||
|
||||
# Get all child processes recursively
|
||||
children = parent.children(recursive=True)
|
||||
|
||||
# Kill children first
|
||||
for child in children:
|
||||
with suppress(psutil.NoSuchProcess, psutil.AccessDenied):
|
||||
child.kill()
|
||||
|
||||
# Kill parent
|
||||
with suppress(psutil.NoSuchProcess, psutil.AccessDenied):
|
||||
parent.kill()
|
||||
|
||||
# Wait for all to terminate
|
||||
_gone, alive = psutil.wait_procs(children + [parent], timeout=3)
|
||||
|
||||
# Force kill any remaining
|
||||
for proc in alive:
|
||||
with suppress(psutil.NoSuchProcess, psutil.AccessDenied):
|
||||
proc.kill()
|
||||
except ImportError:
|
||||
# Fallback if psutil not available
|
||||
try:
|
||||
if func_process.poll() is None:
|
||||
func_process.kill()
|
||||
func_process.wait()
|
||||
except Exception:
|
||||
# Ignore all exceptions during fallback cleanup; best effort to terminate process.
|
||||
pass
|
||||
except Exception:
|
||||
pass # Best effort cleanup
|
||||
|
||||
# Give the port time to be released
|
||||
time.sleep(2)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Pytest Configuration
|
||||
# =============================================================================
|
||||
|
||||
|
||||
def pytest_configure(config: pytest.Config) -> None:
|
||||
"""Register custom markers."""
|
||||
config.addinivalue_line("markers", "orchestration: marks tests that use orchestrations (require Azurite)")
|
||||
config.addinivalue_line(
|
||||
"markers",
|
||||
"sample(path): specify the sample directory path for the test (e.g., @pytest.mark.sample('01_single_agent'))",
|
||||
)
|
||||
|
||||
|
||||
def pytest_collection_modifyitems(config: pytest.Config, items: list[pytest.Item]) -> None:
|
||||
"""Skip integration tests in this directory if prerequisites are not met."""
|
||||
should_skip, reason = _should_skip_azure_functions_integration_tests()
|
||||
if should_skip:
|
||||
skip_marker = pytest.mark.skip(reason=reason)
|
||||
for item in items:
|
||||
# Only skip items that are in this integration_tests directory
|
||||
if "integration_tests" in str(item.fspath):
|
||||
item.add_marker(skip_marker)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Pytest Fixtures
|
||||
# =============================================================================
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def function_app_running() -> bool:
|
||||
"""Check if the function app is running on localhost:7071.
|
||||
|
||||
This fixture can be used to skip tests if the function app is not available.
|
||||
"""
|
||||
try:
|
||||
response = requests.get("http://localhost:7071/api/health", timeout=2)
|
||||
return response.status_code == 200
|
||||
except requests.exceptions.RequestException:
|
||||
return False
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def skip_if_no_function_app(function_app_running: bool) -> None:
|
||||
"""Skip test if function app is not running."""
|
||||
if not function_app_running:
|
||||
pytest.skip("Function app is not running on http://localhost:7071")
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def function_app_for_test(request: pytest.FixtureRequest) -> Iterator[dict[str, int | str]]:
|
||||
"""Start the function app for the corresponding sample based on marker.
|
||||
|
||||
This fixture:
|
||||
1. Determines which sample to run from @pytest.mark.sample()
|
||||
2. Validates environment variables
|
||||
3. Starts the function app using 'func start'
|
||||
4. Waits for the app to be ready
|
||||
5. Tears down the app after tests complete
|
||||
|
||||
Usage:
|
||||
@pytest.mark.sample("01_single_agent")
|
||||
@pytest.mark.usefixtures("function_app_for_test")
|
||||
class TestSample01SingleAgent:
|
||||
...
|
||||
"""
|
||||
# Get sample path from marker
|
||||
sample_path, error_message = _get_sample_path_from_marker(request)
|
||||
if error_message:
|
||||
pytest.fail(error_message)
|
||||
|
||||
assert sample_path is not None, "Sample path must be resolved before starting the function app"
|
||||
|
||||
# Load .env file if it exists and validate required env vars
|
||||
_load_and_validate_env(sample_path)
|
||||
|
||||
max_attempts = 3
|
||||
# The overall budget MUST be shorter than the pytest-timeout value
|
||||
# (--timeout=120 by default) so that the fixture finishes cleanly instead
|
||||
# of being killed by os._exit() which crashes the xdist worker.
|
||||
overall_budget = 100 # seconds – leaves headroom below the 120 s test timeout
|
||||
last_error: Exception | None = None
|
||||
func_process: subprocess.Popen[Any] | None = None
|
||||
base_url = ""
|
||||
port = 0
|
||||
overall_start = time.monotonic()
|
||||
attempts_made = 0
|
||||
|
||||
for _ in range(max_attempts):
|
||||
remaining = overall_budget - (time.monotonic() - overall_start)
|
||||
if remaining < 10:
|
||||
# Not enough time for another attempt; bail out.
|
||||
break
|
||||
|
||||
attempts_made += 1
|
||||
port = _find_available_port()
|
||||
base_url = _build_base_url(port)
|
||||
func_process = _start_function_app(sample_path, port)
|
||||
|
||||
try:
|
||||
# Cap each attempt's wait to the remaining budget minus a small
|
||||
# buffer for cleanup.
|
||||
per_attempt_wait = min(60, int(remaining) - 5)
|
||||
_wait_for_function_app_ready(func_process, port, max_wait=max(per_attempt_wait, 10))
|
||||
last_error = None
|
||||
break
|
||||
except FunctionAppStartupError as exc:
|
||||
last_error = exc
|
||||
_cleanup_function_app(func_process)
|
||||
func_process = None
|
||||
|
||||
if func_process is None:
|
||||
elapsed = int(time.monotonic() - overall_start)
|
||||
error_message = f"Function app failed to start after {attempts_made} attempt(s) ({elapsed}s elapsed)."
|
||||
if last_error is not None:
|
||||
error_message += f" Last error: {last_error}"
|
||||
pytest.fail(error_message)
|
||||
|
||||
try:
|
||||
yield {"base_url": base_url, "port": port}
|
||||
finally:
|
||||
if func_process is not None:
|
||||
_cleanup_function_app(func_process)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def base_url(function_app_for_test: Mapping[str, int | str]) -> str:
|
||||
"""Expose the function app's base URL to tests."""
|
||||
return str(function_app_for_test["base_url"])
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def sample_helper() -> type[SampleTestHelper]:
|
||||
"""Provide the SampleTestHelper class for tests."""
|
||||
return SampleTestHelper
|
||||
@@ -0,0 +1,115 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
"""
|
||||
Integration Tests for Single Agent Sample
|
||||
|
||||
Tests the single agent sample with various message formats and session management.
|
||||
|
||||
The function app is automatically started by the test fixture.
|
||||
|
||||
Prerequisites:
|
||||
- Azure OpenAI credentials configured (see packages/azurefunctions/tests/integration_tests/.env.example)
|
||||
- Azurite or Azure Storage account configured
|
||||
|
||||
Usage:
|
||||
uv run pytest packages/azurefunctions/tests/integration_tests/test_01_single_agent.py -v
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from agent_framework_durabletask import THREAD_ID_HEADER
|
||||
|
||||
# Module-level markers - applied to all tests in this file
|
||||
pytestmark = [
|
||||
pytest.mark.flaky,
|
||||
pytest.mark.integration,
|
||||
pytest.mark.sample("01_single_agent"),
|
||||
pytest.mark.usefixtures("function_app_for_test"),
|
||||
]
|
||||
|
||||
|
||||
class TestSampleSingleAgent:
|
||||
"""Tests for 01_single_agent sample."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _setup(self, base_url: str, sample_helper) -> None:
|
||||
"""Provide agent-specific base URL and helper for the tests."""
|
||||
self.base_url = f"{base_url}/api/agents/Joker"
|
||||
self.helper = sample_helper
|
||||
|
||||
def test_health_check(self, base_url: str, sample_helper) -> None:
|
||||
"""Test health check endpoint."""
|
||||
response = sample_helper.get(f"{base_url}/api/health")
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "healthy"
|
||||
|
||||
def test_simple_message_json(self) -> None:
|
||||
"""Test sending a simple message with JSON payload."""
|
||||
response = self.helper.post_json(
|
||||
f"{self.base_url}/run",
|
||||
{"message": "Tell me a short joke about cloud computing.", "thread_id": "test-simple-json"},
|
||||
)
|
||||
# Agent can return 200 (immediate) or 202 (async with wait_for_response=false)
|
||||
assert response.status_code in [200, 202]
|
||||
data = response.json()
|
||||
|
||||
if response.status_code == 200:
|
||||
# Synchronous response - check result directly
|
||||
assert data["status"] == "success"
|
||||
assert "response" in data
|
||||
assert data["message_count"] >= 1
|
||||
else:
|
||||
# Async response - check we got correlation info
|
||||
assert "correlation_id" in data or "thread_id" in data
|
||||
|
||||
def test_simple_message_plain_text(self) -> None:
|
||||
"""Test sending a message with plain text payload."""
|
||||
response = self.helper.post_text(f"{self.base_url}/run", "Tell me a short joke about networking.")
|
||||
assert response.status_code in [200, 202]
|
||||
|
||||
# Agent responded with plain text when the request body was text/plain.
|
||||
assert response.text.strip()
|
||||
assert response.headers.get(THREAD_ID_HEADER) is not None
|
||||
|
||||
def test_thread_id_in_query(self) -> None:
|
||||
"""Test using thread_id in query parameter."""
|
||||
response = self.helper.post_text(
|
||||
f"{self.base_url}/run?thread_id=test-query-thread", "Tell me a short joke about weather in Texas."
|
||||
)
|
||||
assert response.status_code in [200, 202]
|
||||
|
||||
assert response.text.strip()
|
||||
assert response.headers.get(THREAD_ID_HEADER) == "test-query-thread"
|
||||
|
||||
def test_conversation_continuity(self) -> None:
|
||||
"""Test conversation context is maintained across requests."""
|
||||
thread_id = "test-continuity"
|
||||
|
||||
# First message
|
||||
response1 = self.helper.post_json(
|
||||
f"{self.base_url}/run",
|
||||
{"message": "Tell me a short joke about weather in Seattle.", "thread_id": thread_id},
|
||||
)
|
||||
assert response1.status_code in [200, 202]
|
||||
|
||||
if response1.status_code == 200:
|
||||
data1 = response1.json()
|
||||
assert data1["message_count"] == 2 # Initial + reply
|
||||
|
||||
# Second message in same session
|
||||
response2 = self.helper.post_json(
|
||||
f"{self.base_url}/run", {"message": "What about San Francisco?", "thread_id": thread_id}
|
||||
)
|
||||
assert response2.status_code == 200
|
||||
data2 = response2.json()
|
||||
assert data2["message_count"] == 4
|
||||
else:
|
||||
# In async mode, we can't easily test message count
|
||||
# Just verify we can make multiple calls
|
||||
response2 = self.helper.post_json(
|
||||
f"{self.base_url}/run", {"message": "What about Texas?", "thread_id": thread_id}
|
||||
)
|
||||
assert response2.status_code == 202
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
@@ -0,0 +1,65 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
"""
|
||||
Integration Tests for Multi-Agent Sample
|
||||
|
||||
Tests the multi-agent sample with different agent endpoints.
|
||||
|
||||
The function app is automatically started by the test fixture.
|
||||
|
||||
Prerequisites:
|
||||
- Azure OpenAI credentials configured (see packages/azurefunctions/tests/integration_tests/.env.example)
|
||||
- Azurite or Azure Storage account configured
|
||||
|
||||
Usage:
|
||||
uv run pytest packages/azurefunctions/tests/integration_tests/test_02_multi_agent.py -v
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
# Module-level markers - applied to all tests in this file
|
||||
pytestmark = [
|
||||
pytest.mark.flaky,
|
||||
pytest.mark.integration,
|
||||
pytest.mark.sample("02_multi_agent"),
|
||||
pytest.mark.usefixtures("function_app_for_test"),
|
||||
]
|
||||
|
||||
|
||||
class TestSampleMultiAgent:
|
||||
"""Tests for 02_multi_agent sample."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _setup(self, base_url: str, sample_helper) -> None:
|
||||
"""Configure base URLs for Weather and Math agents."""
|
||||
self.weather_base_url = f"{base_url}/api/agents/WeatherAgent"
|
||||
self.math_base_url = f"{base_url}/api/agents/MathAgent"
|
||||
self.helper = sample_helper
|
||||
|
||||
@pytest.mark.skip(reason="Flaky in CI: times out / crashes the xdist runner; temporarily disabled.")
|
||||
def test_weather_agent(self) -> None:
|
||||
"""Test WeatherAgent endpoint."""
|
||||
response = self.helper.post_json(
|
||||
f"{self.weather_base_url}/run",
|
||||
{"message": "What is the weather in Seattle?"},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "success"
|
||||
assert "response" in data
|
||||
|
||||
def test_math_agent(self) -> None:
|
||||
"""Test MathAgent endpoint."""
|
||||
response = self.helper.post_json(
|
||||
f"{self.math_base_url}/run",
|
||||
{"message": "Calculate a 20% tip on a $50 bill", "wait_for_response": False},
|
||||
)
|
||||
assert response.status_code == 202
|
||||
data = response.json()
|
||||
|
||||
assert data["status"] == "accepted"
|
||||
assert "correlation_id" in data
|
||||
assert "thread_id" in data
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
@@ -0,0 +1,121 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
"""
|
||||
Integration Tests for Reliable Streaming Sample
|
||||
|
||||
Tests the reliable streaming sample using Redis Streams for persistent message delivery.
|
||||
|
||||
The function app is automatically started by the test fixture.
|
||||
|
||||
Prerequisites:
|
||||
- Azure OpenAI credentials configured (see packages/azurefunctions/tests/integration_tests/.env.example)
|
||||
- Azurite or Azure Storage account configured
|
||||
- Redis running (docker run -d --name redis -p 6379:6379 redis:latest)
|
||||
|
||||
Usage:
|
||||
uv run pytest packages/azurefunctions/tests/integration_tests/test_03_reliable_streaming.py -v
|
||||
"""
|
||||
|
||||
import time
|
||||
|
||||
import pytest
|
||||
import requests
|
||||
|
||||
# Module-level markers - applied to all tests in this file
|
||||
pytestmark = [
|
||||
pytest.mark.flaky,
|
||||
pytest.mark.integration,
|
||||
pytest.mark.sample("03_reliable_streaming"),
|
||||
pytest.mark.usefixtures("function_app_for_test"),
|
||||
]
|
||||
|
||||
|
||||
class TestSampleReliableStreaming:
|
||||
"""Tests for 03_reliable_streaming sample."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _setup(self, base_url: str, sample_helper) -> None:
|
||||
"""Provide the base URL and helper for each test."""
|
||||
self.base_url = base_url
|
||||
self.agent_url = f"{base_url}/api/agents/TravelPlanner"
|
||||
self.stream_url = f"{base_url}/api/agent/stream"
|
||||
self.helper = sample_helper
|
||||
|
||||
def test_agent_run_and_stream(self) -> None:
|
||||
"""Test agent execution with Redis streaming."""
|
||||
# Start agent run
|
||||
response = self.helper.post_json(
|
||||
f"{self.agent_url}/run",
|
||||
{"message": "Plan a 1-day trip to Seattle in 1 sentence", "wait_for_response": False},
|
||||
)
|
||||
assert response.status_code == 202
|
||||
data = response.json()
|
||||
|
||||
thread_id = data.get("thread_id")
|
||||
|
||||
# Wait a moment for the agent to start writing to Redis
|
||||
time.sleep(2)
|
||||
|
||||
# Stream response from Redis with longer timeout to account for LLM latency
|
||||
stream_response = requests.get(
|
||||
f"{self.stream_url}/{thread_id}",
|
||||
headers={"Accept": "text/plain"},
|
||||
timeout=60,
|
||||
)
|
||||
assert stream_response.status_code == 200
|
||||
|
||||
def test_stream_with_sse_format(self) -> None:
|
||||
"""Test streaming with Server-Sent Events format."""
|
||||
# Start agent run
|
||||
response = self.helper.post_json(
|
||||
f"{self.agent_url}/run",
|
||||
{"message": "What's the weather like?", "wait_for_response": False},
|
||||
)
|
||||
assert response.status_code == 202
|
||||
data = response.json()
|
||||
thread_id = data.get("thread_id")
|
||||
|
||||
# Wait for agent to start writing
|
||||
time.sleep(2)
|
||||
|
||||
# Stream with SSE format
|
||||
stream_response = requests.get(
|
||||
f"{self.stream_url}/{thread_id}",
|
||||
headers={"Accept": "text/event-stream"},
|
||||
timeout=60,
|
||||
)
|
||||
assert stream_response.status_code == 200
|
||||
content_type = stream_response.headers.get("content-type", "")
|
||||
assert "text/event-stream" in content_type
|
||||
|
||||
# Check for SSE event markers if we got content
|
||||
content = stream_response.text
|
||||
if content:
|
||||
assert "event:" in content or "data:" in content
|
||||
|
||||
def test_stream_nonexistent_conversation(self) -> None:
|
||||
"""Test streaming from a non-existent conversation.
|
||||
|
||||
The endpoint will wait for data in Redis, but since the conversation
|
||||
doesn't exist, it will timeout. This is expected behavior.
|
||||
"""
|
||||
fake_id = "nonexistent-conversation-12345"
|
||||
|
||||
# Should timeout since the conversation doesn't exist
|
||||
with pytest.raises(requests.exceptions.ReadTimeout):
|
||||
requests.get(
|
||||
f"{self.stream_url}/{fake_id}",
|
||||
headers={"Accept": "text/plain"},
|
||||
timeout=10, # Short timeout for non-existent ID
|
||||
)
|
||||
|
||||
def test_health_endpoint(self) -> None:
|
||||
"""Test health check endpoint."""
|
||||
response = self.helper.get(f"{self.base_url}/api/health")
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "healthy"
|
||||
assert "agents" in data
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
+57
@@ -0,0 +1,57 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
"""
|
||||
Integration Tests for Orchestration Chaining Sample
|
||||
|
||||
Tests the orchestration chaining sample for sequential agent execution.
|
||||
|
||||
The function app is automatically started by the test fixture.
|
||||
|
||||
Prerequisites:
|
||||
- Azure OpenAI credentials configured (see packages/azurefunctions/tests/integration_tests/.env.example)
|
||||
- Azurite running for durable orchestrations (or Azure Storage account configured)
|
||||
|
||||
Usage:
|
||||
# Start Azurite (if not already running)
|
||||
azurite &
|
||||
|
||||
# Run tests
|
||||
uv run pytest packages/azurefunctions/tests/integration_tests/test_04_single_agent_orchestration_chaining.py -v
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
# Module-level markers - applied to all tests in this file
|
||||
pytestmark = [
|
||||
pytest.mark.flaky,
|
||||
pytest.mark.integration,
|
||||
pytest.mark.sample("04_single_agent_orchestration_chaining"),
|
||||
pytest.mark.usefixtures("function_app_for_test"),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.orchestration
|
||||
class TestSampleOrchestrationChaining:
|
||||
"""Tests for 04_single_agent_orchestration_chaining sample."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _setup(self, sample_helper) -> None:
|
||||
"""Provide the helper for each test."""
|
||||
self.helper = sample_helper
|
||||
|
||||
def test_orchestration_chaining(self, base_url: str) -> None:
|
||||
"""Test sequential agent calls in orchestration."""
|
||||
# Start orchestration
|
||||
response = self.helper.post_json(f"{base_url}/api/singleagent/run", {})
|
||||
assert response.status_code == 202
|
||||
data = response.json()
|
||||
assert "instanceId" in data
|
||||
assert "statusQueryGetUri" in data
|
||||
|
||||
# Wait for completion with output available
|
||||
status = self.helper.wait_for_orchestration_with_output(data["statusQueryGetUri"])
|
||||
assert status["runtimeStatus"] == "Completed"
|
||||
assert "output" in status
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
+59
@@ -0,0 +1,59 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
"""
|
||||
Integration Tests for MultiAgent Concurrency Sample
|
||||
|
||||
Tests the multi-agent concurrency sample for parallel agent execution.
|
||||
|
||||
The function app is automatically started by the test fixture.
|
||||
|
||||
Prerequisites:
|
||||
- Azure OpenAI credentials configured (see packages/azurefunctions/tests/integration_tests/.env.example)
|
||||
- Azurite running for durable orchestrations (or Azure Storage account configured)
|
||||
|
||||
Usage:
|
||||
# Start Azurite (if not already running)
|
||||
azurite &
|
||||
|
||||
# Run tests
|
||||
uv run pytest packages/azurefunctions/tests/integration_tests/test_05_multi_agent_orchestration_concurrency.py -v
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
# Module-level markers - applied to all tests in this file
|
||||
pytestmark = [
|
||||
pytest.mark.flaky,
|
||||
pytest.mark.integration,
|
||||
pytest.mark.orchestration,
|
||||
pytest.mark.sample("05_multi_agent_orchestration_concurrency"),
|
||||
pytest.mark.usefixtures("function_app_for_test"),
|
||||
]
|
||||
|
||||
|
||||
class TestSampleMultiAgentConcurrency:
|
||||
"""Tests for 05_multi_agent_orchestration_concurrency sample."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _setup(self, sample_helper) -> None:
|
||||
"""Provide the helper for each test."""
|
||||
self.helper = sample_helper
|
||||
|
||||
def test_concurrent_agents(self, base_url: str) -> None:
|
||||
"""Test multiple agents running concurrently."""
|
||||
# Start orchestration
|
||||
response = self.helper.post_text(f"{base_url}/api/multiagent/run", "What is temperature?")
|
||||
assert response.status_code == 202
|
||||
data = response.json()
|
||||
assert "instanceId" in data
|
||||
assert "statusQueryGetUri" in data
|
||||
|
||||
# Wait for completion
|
||||
status = self.helper.wait_for_orchestration(data["statusQueryGetUri"])
|
||||
assert status["runtimeStatus"] == "Completed"
|
||||
output = status["output"]
|
||||
assert "physicist" in output
|
||||
assert "chemist" in output
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
+77
@@ -0,0 +1,77 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
"""
|
||||
Integration Tests for MultiAgent Conditionals Sample
|
||||
|
||||
Tests the multi-agent conditionals sample for conditional orchestration logic.
|
||||
|
||||
The function app is automatically started by the test fixture.
|
||||
|
||||
Prerequisites:
|
||||
- Azure OpenAI credentials configured (see packages/azurefunctions/tests/integration_tests/.env.example)
|
||||
- Azurite running for durable orchestrations (or Azure Storage account configured)
|
||||
|
||||
Usage:
|
||||
# Start Azurite (if not already running)
|
||||
azurite &
|
||||
|
||||
# Run tests
|
||||
uv run pytest packages/azurefunctions/tests/integration_tests/test_06_multi_agent_orchestration_conditionals.py -v
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
# Module-level markers - applied to all tests in this file
|
||||
pytestmark = [
|
||||
pytest.mark.flaky,
|
||||
pytest.mark.integration,
|
||||
pytest.mark.orchestration,
|
||||
pytest.mark.sample("06_multi_agent_orchestration_conditionals"),
|
||||
pytest.mark.usefixtures("function_app_for_test"),
|
||||
]
|
||||
|
||||
|
||||
class TestSampleMultiAgentConditionals:
|
||||
"""Tests for 06_multi_agent_orchestration_conditionals sample."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _setup(self, sample_helper) -> None:
|
||||
"""Provide the helper for each test."""
|
||||
self.helper = sample_helper
|
||||
|
||||
def test_legitimate_email(self, base_url: str) -> None:
|
||||
"""Test conditional logic with legitimate email."""
|
||||
response = self.helper.post_json(
|
||||
f"{base_url}/api/spamdetection/run",
|
||||
{
|
||||
"email_id": "email-test-001",
|
||||
"email_content": "Hi John, I hope you are doing well. Can you send me the report?",
|
||||
},
|
||||
)
|
||||
assert response.status_code == 202
|
||||
data = response.json()
|
||||
assert "instanceId" in data
|
||||
assert "statusQueryGetUri" in data
|
||||
|
||||
# Wait for completion
|
||||
status = self.helper.wait_for_orchestration(data["statusQueryGetUri"])
|
||||
assert status["runtimeStatus"] == "Completed"
|
||||
assert "Email sent:" in status["output"]
|
||||
|
||||
def test_spam_email(self, base_url: str) -> None:
|
||||
"""Test conditional logic with spam email."""
|
||||
response = self.helper.post_json(
|
||||
f"{base_url}/api/spamdetection/run",
|
||||
{"email_id": "email-test-002", "email_content": "URGENT! You have won $1,000,000! Click here now!"},
|
||||
)
|
||||
assert response.status_code == 202
|
||||
data = response.json()
|
||||
assert "instanceId" in data
|
||||
|
||||
# Wait for completion
|
||||
status = self.helper.wait_for_orchestration(data["statusQueryGetUri"])
|
||||
assert status["runtimeStatus"] == "Completed"
|
||||
assert "Email marked as spam:" in status["output"]
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
+185
@@ -0,0 +1,185 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
"""
|
||||
Integration Tests for Human-in-the-Loop (HITL) Orchestration Sample
|
||||
|
||||
Tests the HITL orchestration sample for content generation with human approval workflow.
|
||||
|
||||
The function app is automatically started by the test fixture.
|
||||
|
||||
Prerequisites:
|
||||
- Azure OpenAI credentials configured (see packages/azurefunctions/tests/integration_tests/.env.example)
|
||||
- Azurite running for durable orchestrations (or Azure Storage account configured)
|
||||
|
||||
Usage:
|
||||
# Start Azurite (if not already running)
|
||||
azurite &
|
||||
|
||||
# Run tests
|
||||
uv run pytest packages/azurefunctions/tests/integration_tests/test_07_single_agent_orchestration_hitl.py -v
|
||||
"""
|
||||
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
# Module-level markers - applied to all tests in this file
|
||||
pytestmark = [
|
||||
pytest.mark.flaky,
|
||||
pytest.mark.integration,
|
||||
pytest.mark.sample("07_single_agent_orchestration_hitl"),
|
||||
pytest.mark.usefixtures("function_app_for_test"),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.orchestration
|
||||
class TestSampleHITLOrchestration:
|
||||
"""Tests for 07_single_agent_orchestration_hitl sample."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _setup(self, base_url: str, sample_helper) -> None:
|
||||
"""Provide the helper and base URL for each test."""
|
||||
self.hitl_base_url = f"{base_url}/api/hitl"
|
||||
self.helper = sample_helper
|
||||
|
||||
def test_hitl_orchestration_approval(self) -> None:
|
||||
"""Test HITL orchestration with human approval."""
|
||||
# Start orchestration
|
||||
response = self.helper.post_json(
|
||||
f"{self.hitl_base_url}/run",
|
||||
{"topic": "artificial intelligence", "max_review_attempts": 3, "approval_timeout_hours": 1.0},
|
||||
)
|
||||
assert response.status_code == 202
|
||||
data = response.json()
|
||||
assert "instanceId" in data
|
||||
assert "statusQueryGetUri" in data
|
||||
assert data["topic"] == "artificial intelligence"
|
||||
instance_id = data["instanceId"]
|
||||
|
||||
# Wait a bit for the orchestration to generate initial content
|
||||
time.sleep(5)
|
||||
|
||||
# Check status to ensure it's waiting for approval
|
||||
status_response = self.helper.get(data["statusQueryGetUri"])
|
||||
assert status_response.status_code == 200
|
||||
status = status_response.json()
|
||||
assert status["runtimeStatus"] in ["Running", "Pending"]
|
||||
|
||||
# Send approval
|
||||
approval_response = self.helper.post_json(
|
||||
f"{self.hitl_base_url}/approve/{instance_id}", {"approved": True, "feedback": ""}
|
||||
)
|
||||
assert approval_response.status_code == 200
|
||||
approval_data = approval_response.json()
|
||||
assert approval_data["approved"] is True
|
||||
|
||||
# Wait for orchestration to complete
|
||||
status = self.helper.wait_for_orchestration(data["statusQueryGetUri"])
|
||||
assert status["runtimeStatus"] == "Completed"
|
||||
assert "output" in status
|
||||
assert "content" in status["output"]
|
||||
|
||||
def test_hitl_orchestration_rejection_with_feedback(self) -> None:
|
||||
"""Test HITL orchestration with rejection and subsequent approval."""
|
||||
# Start orchestration
|
||||
response = self.helper.post_json(
|
||||
f"{self.hitl_base_url}/run",
|
||||
{"topic": "machine learning", "max_review_attempts": 3, "approval_timeout_hours": 1.0},
|
||||
)
|
||||
assert response.status_code == 202
|
||||
data = response.json()
|
||||
instance_id = data["instanceId"]
|
||||
|
||||
# Wait for initial content generation
|
||||
time.sleep(5)
|
||||
|
||||
# Send rejection with feedback
|
||||
rejection_response = self.helper.post_json(
|
||||
f"{self.hitl_base_url}/approve/{instance_id}",
|
||||
{"approved": False, "feedback": "Please make it more concise and focus on practical applications."},
|
||||
)
|
||||
assert rejection_response.status_code == 200
|
||||
|
||||
# Wait for regeneration
|
||||
time.sleep(5)
|
||||
|
||||
# Check status - should still be running
|
||||
status_response = self.helper.get(data["statusQueryGetUri"])
|
||||
assert status_response.status_code == 200
|
||||
status = status_response.json()
|
||||
assert status["runtimeStatus"] in ["Running", "Pending"]
|
||||
|
||||
# Now approve the revised content
|
||||
approval_response = self.helper.post_json(
|
||||
f"{self.hitl_base_url}/approve/{instance_id}", {"approved": True, "feedback": ""}
|
||||
)
|
||||
assert approval_response.status_code == 200
|
||||
|
||||
# Wait for completion
|
||||
status = self.helper.wait_for_orchestration(data["statusQueryGetUri"])
|
||||
assert status["runtimeStatus"] == "Completed"
|
||||
assert "output" in status
|
||||
|
||||
def test_hitl_orchestration_missing_topic(self) -> None:
|
||||
"""Test HITL orchestration with missing topic."""
|
||||
response = self.helper.post_json(f"{self.hitl_base_url}/run", {"max_review_attempts": 3})
|
||||
assert response.status_code == 400
|
||||
data = response.json()
|
||||
assert "error" in data
|
||||
|
||||
def test_hitl_get_status(self) -> None:
|
||||
"""Test getting orchestration status."""
|
||||
# Start orchestration
|
||||
response = self.helper.post_json(
|
||||
f"{self.hitl_base_url}/run",
|
||||
{"topic": "quantum computing", "max_review_attempts": 2, "approval_timeout_hours": 1.0},
|
||||
)
|
||||
assert response.status_code == 202
|
||||
data = response.json()
|
||||
instance_id = data["instanceId"]
|
||||
|
||||
# Get status
|
||||
status_response = self.helper.get(f"{self.hitl_base_url}/status/{instance_id}")
|
||||
assert status_response.status_code == 200
|
||||
status = status_response.json()
|
||||
assert "instanceId" in status
|
||||
assert "runtimeStatus" in status
|
||||
assert status["instanceId"] == instance_id
|
||||
|
||||
# Cleanup: approve to complete orchestration
|
||||
time.sleep(5)
|
||||
self.helper.post_json(f"{self.hitl_base_url}/approve/{instance_id}", {"approved": True, "feedback": ""})
|
||||
|
||||
def test_hitl_approval_invalid_payload(self) -> None:
|
||||
"""Test sending approval with invalid payload."""
|
||||
# Start orchestration first
|
||||
response = self.helper.post_json(
|
||||
f"{self.hitl_base_url}/run",
|
||||
{"topic": "test topic", "max_review_attempts": 1, "approval_timeout_hours": 1.0},
|
||||
)
|
||||
assert response.status_code == 202
|
||||
data = response.json()
|
||||
instance_id = data["instanceId"]
|
||||
|
||||
time.sleep(3)
|
||||
|
||||
# Send approval without 'approved' field
|
||||
approval_response = self.helper.post_json(
|
||||
f"{self.hitl_base_url}/approve/{instance_id}", {"feedback": "Some feedback"}
|
||||
)
|
||||
assert approval_response.status_code == 400
|
||||
error_data = approval_response.json()
|
||||
assert "error" in error_data
|
||||
|
||||
# Cleanup
|
||||
self.helper.post_json(f"{self.hitl_base_url}/approve/{instance_id}", {"approved": True, "feedback": ""})
|
||||
|
||||
def test_hitl_status_invalid_instance(self) -> None:
|
||||
"""Test getting status for non-existent instance."""
|
||||
response = self.helper.get(f"{self.hitl_base_url}/status/invalid-instance-id")
|
||||
assert response.status_code == 404
|
||||
data = response.json()
|
||||
assert "error" in data
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
+100
@@ -0,0 +1,100 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
"""
|
||||
Integration Tests for Workflow Shared State Sample
|
||||
|
||||
Tests the workflow shared state sample for conditional email processing
|
||||
with shared state management.
|
||||
|
||||
The function app is automatically started by the test fixture.
|
||||
|
||||
Prerequisites:
|
||||
- Azure OpenAI credentials configured (see packages/azurefunctions/tests/integration_tests/.env.example)
|
||||
- Azurite running for durable orchestrations (or Azure Storage account configured)
|
||||
|
||||
Usage:
|
||||
# Start Azurite (if not already running)
|
||||
azurite &
|
||||
|
||||
# Run tests
|
||||
uv run pytest packages/azurefunctions/tests/integration_tests/test_09_workflow_shared_state.py -v
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
# Must match the workflow name in samples/04-hosting/azure_functions/09_workflow_shared_state/function_app.py
|
||||
WORKFLOW_NAME = "email_triage_shared_state"
|
||||
|
||||
# Module-level markers - applied to all tests in this file
|
||||
pytestmark = [
|
||||
pytest.mark.flaky,
|
||||
pytest.mark.integration,
|
||||
pytest.mark.sample("09_workflow_shared_state"),
|
||||
pytest.mark.usefixtures("function_app_for_test"),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.orchestration
|
||||
class TestWorkflowSharedState:
|
||||
"""Tests for 09_workflow_shared_state sample."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _setup(self, base_url: str, sample_helper) -> None:
|
||||
"""Provide the helper and base URL for each test."""
|
||||
self.base_url = base_url
|
||||
self.helper = sample_helper
|
||||
|
||||
def test_workflow_with_spam_email(self) -> None:
|
||||
"""Test workflow with spam email content - should be detected and handled as spam."""
|
||||
spam_content = "URGENT! You have won $1,000,000! Click here to claim your prize now before it expires!"
|
||||
|
||||
# Start orchestration with spam email
|
||||
response = self.helper.post_text(f"{self.base_url}/api/workflow/{WORKFLOW_NAME}/run", spam_content)
|
||||
assert response.status_code == 202
|
||||
data = response.json()
|
||||
assert "instanceId" in data
|
||||
assert "statusQueryGetUri" in data
|
||||
|
||||
# Wait for completion
|
||||
status = self.helper.wait_for_orchestration_with_output(data["statusQueryGetUri"])
|
||||
assert status["runtimeStatus"] == "Completed"
|
||||
assert "output" in status
|
||||
|
||||
def test_workflow_with_legitimate_email(self) -> None:
|
||||
"""Test workflow with legitimate email content - should generate response."""
|
||||
legitimate_content = (
|
||||
"Hi team, just a reminder about the sprint planning meeting tomorrow at 10 AM. "
|
||||
"Please review the agenda items in Jira before the call."
|
||||
)
|
||||
|
||||
# Start orchestration with legitimate email
|
||||
response = self.helper.post_text(f"{self.base_url}/api/workflow/{WORKFLOW_NAME}/run", legitimate_content)
|
||||
assert response.status_code == 202
|
||||
data = response.json()
|
||||
assert "instanceId" in data
|
||||
assert "statusQueryGetUri" in data
|
||||
|
||||
# Wait for completion
|
||||
status = self.helper.wait_for_orchestration_with_output(data["statusQueryGetUri"])
|
||||
assert status["runtimeStatus"] == "Completed"
|
||||
assert "output" in status
|
||||
|
||||
def test_workflow_with_phishing_email(self) -> None:
|
||||
"""Test workflow with phishing email - should be detected as spam."""
|
||||
phishing_content = (
|
||||
"Dear Customer, Your account has been compromised! "
|
||||
"Click this link immediately to secure your account: http://totallylegit.suspicious.com/secure"
|
||||
)
|
||||
|
||||
# Start orchestration with phishing email
|
||||
response = self.helper.post_text(f"{self.base_url}/api/workflow/{WORKFLOW_NAME}/run", phishing_content)
|
||||
assert response.status_code == 202
|
||||
data = response.json()
|
||||
assert "instanceId" in data
|
||||
|
||||
# Wait for completion
|
||||
status = self.helper.wait_for_orchestration_with_output(data["statusQueryGetUri"])
|
||||
assert status["runtimeStatus"] == "Completed"
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
+116
@@ -0,0 +1,116 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
"""
|
||||
Integration Tests for Workflow No Shared State Sample
|
||||
|
||||
Tests the workflow sample that runs without shared state,
|
||||
demonstrating conditional routing with spam detection and email response.
|
||||
|
||||
The function app is automatically started by the test fixture.
|
||||
|
||||
Prerequisites:
|
||||
- Azure OpenAI credentials configured (see packages/azurefunctions/tests/integration_tests/.env.example)
|
||||
- Azurite running for durable orchestrations (or Azure Storage account configured)
|
||||
|
||||
Usage:
|
||||
# Start Azurite (if not already running)
|
||||
azurite &
|
||||
|
||||
# Run tests
|
||||
uv run pytest packages/azurefunctions/tests/integration_tests/test_10_workflow_no_shared_state.py -v
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
# Must match the workflow name in samples/04-hosting/azure_functions/10_workflow_no_shared_state/function_app.py
|
||||
WORKFLOW_NAME = "email_triage"
|
||||
|
||||
# Module-level markers - applied to all tests in this file
|
||||
pytestmark = [
|
||||
pytest.mark.flaky,
|
||||
pytest.mark.integration,
|
||||
pytest.mark.sample("10_workflow_no_shared_state"),
|
||||
pytest.mark.usefixtures("function_app_for_test"),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.orchestration
|
||||
class TestWorkflowNoSharedState:
|
||||
"""Tests for 10_workflow_no_shared_state sample."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _setup(self, base_url: str, sample_helper) -> None:
|
||||
"""Provide the helper and base URL for each test."""
|
||||
self.base_url = base_url
|
||||
self.helper = sample_helper
|
||||
|
||||
def test_workflow_with_spam_email(self) -> None:
|
||||
"""Test workflow with spam email - should detect and handle as spam."""
|
||||
payload = {
|
||||
"email_id": "email-test-001",
|
||||
"email_content": (
|
||||
"URGENT! You've won $1,000,000! Click here immediately to claim your prize! "
|
||||
"Limited time offer - act now!"
|
||||
),
|
||||
}
|
||||
|
||||
# Start orchestration
|
||||
response = self.helper.post_json(f"{self.base_url}/api/workflow/{WORKFLOW_NAME}/run", payload)
|
||||
assert response.status_code == 202
|
||||
data = response.json()
|
||||
assert "instanceId" in data
|
||||
assert "statusQueryGetUri" in data
|
||||
|
||||
# Wait for completion
|
||||
status = self.helper.wait_for_orchestration_with_output(data["statusQueryGetUri"])
|
||||
assert status["runtimeStatus"] == "Completed"
|
||||
assert "output" in status
|
||||
|
||||
def test_workflow_with_legitimate_email(self) -> None:
|
||||
"""Test workflow with legitimate email - should draft a response."""
|
||||
payload = {
|
||||
"email_id": "email-test-002",
|
||||
"email_content": (
|
||||
"Hi team, just a reminder about our sprint planning meeting tomorrow at 10 AM. "
|
||||
"Please review the agenda in Jira."
|
||||
),
|
||||
}
|
||||
|
||||
# Start orchestration
|
||||
response = self.helper.post_json(f"{self.base_url}/api/workflow/{WORKFLOW_NAME}/run", payload)
|
||||
assert response.status_code == 202
|
||||
data = response.json()
|
||||
assert "instanceId" in data
|
||||
assert "statusQueryGetUri" in data
|
||||
|
||||
# Wait for completion
|
||||
status = self.helper.wait_for_orchestration_with_output(data["statusQueryGetUri"])
|
||||
assert status["runtimeStatus"] == "Completed"
|
||||
assert "output" in status
|
||||
|
||||
def test_workflow_status_endpoint(self) -> None:
|
||||
"""Test that the status endpoint works correctly."""
|
||||
payload = {
|
||||
"email_id": "email-test-003",
|
||||
"email_content": "Quick question: When is the next team meeting scheduled?",
|
||||
}
|
||||
|
||||
# Start orchestration
|
||||
response = self.helper.post_json(f"{self.base_url}/api/workflow/{WORKFLOW_NAME}/run", payload)
|
||||
assert response.status_code == 202
|
||||
data = response.json()
|
||||
instance_id = data["instanceId"]
|
||||
|
||||
# Check status using the workflow status endpoint
|
||||
status_response = self.helper.get(f"{self.base_url}/api/workflow/{WORKFLOW_NAME}/status/{instance_id}")
|
||||
assert status_response.status_code == 200
|
||||
status = status_response.json()
|
||||
assert "instanceId" in status
|
||||
assert status["instanceId"] == instance_id
|
||||
assert "runtimeStatus" in status
|
||||
|
||||
# Wait for completion to clean up
|
||||
self.helper.wait_for_orchestration(data["statusQueryGetUri"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
@@ -0,0 +1,87 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
"""
|
||||
Integration Tests for Parallel Workflow Sample
|
||||
|
||||
Tests the parallel workflow execution sample demonstrating:
|
||||
- Two executors running concurrently (fan-out to activities)
|
||||
- Two agents running concurrently (fan-out to entities)
|
||||
- Mixed agent + executor running concurrently
|
||||
|
||||
The function app is automatically started by the test fixture.
|
||||
|
||||
Prerequisites:
|
||||
- Azure OpenAI credentials configured (see packages/azurefunctions/tests/integration_tests/.env.example)
|
||||
- Azurite running for durable orchestrations (or Azure Storage account configured)
|
||||
|
||||
Usage:
|
||||
# Start Azurite (if not already running)
|
||||
azurite &
|
||||
|
||||
# Run tests
|
||||
uv run pytest packages/azurefunctions/tests/integration_tests/test_11_workflow_parallel.py -v
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
# Must match the workflow name in samples/04-hosting/azure_functions/11_workflow_parallel/function_app.py
|
||||
WORKFLOW_NAME = "parallel_review"
|
||||
|
||||
# Module-level markers - applied to all tests in this file
|
||||
pytestmark = [
|
||||
pytest.mark.flaky,
|
||||
pytest.mark.integration,
|
||||
pytest.mark.sample("11_workflow_parallel"),
|
||||
pytest.mark.usefixtures("function_app_for_test"),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.orchestration
|
||||
class TestWorkflowParallel:
|
||||
"""Tests for 11_workflow_parallel sample."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _setup(self, base_url: str, sample_helper) -> None:
|
||||
"""Provide the helper and base URL for each test."""
|
||||
self.base_url = base_url
|
||||
self.helper = sample_helper
|
||||
|
||||
@pytest.mark.skip(reason="Flaky in CI: times out / crashes the xdist runner; temporarily disabled.")
|
||||
def test_parallel_workflow_end_to_end(self) -> None:
|
||||
"""Run the parallel workflow end-to-end: start, check status, verify completion.
|
||||
|
||||
Consolidated into a single test on purpose: the work-stealing xdist scheduler
|
||||
distributes tests (not modules) across workers, and the module-scoped
|
||||
``function_app_for_test`` fixture is created per worker -- so multiple tests in
|
||||
this module would each spawn a separate ``func`` host for this resource-heavy
|
||||
parallel sample. One test keeps it to a single host while still covering the
|
||||
fan-out path end-to-end.
|
||||
"""
|
||||
payload = {
|
||||
"document_id": "doc-test-001",
|
||||
"content": (
|
||||
"The quarterly earnings report shows strong growth in our cloud services division. "
|
||||
"Revenue increased by 25% compared to last year, driven by enterprise adoption. "
|
||||
"Customer satisfaction remains high at 92%."
|
||||
),
|
||||
}
|
||||
|
||||
# Start the orchestration.
|
||||
response = self.helper.post_json(f"{self.base_url}/api/workflow/{WORKFLOW_NAME}/run", payload)
|
||||
assert response.status_code == 202
|
||||
data = response.json()
|
||||
instance_id = data["instanceId"]
|
||||
assert "statusQueryGetUri" in data
|
||||
|
||||
# The status endpoint reflects the started instance.
|
||||
status_response = self.helper.get(f"{self.base_url}/api/workflow/{WORKFLOW_NAME}/status/{instance_id}")
|
||||
assert status_response.status_code == 200
|
||||
assert status_response.json()["instanceId"] == instance_id
|
||||
|
||||
# Fan-out to parallel processors and agents completes with an aggregated output.
|
||||
status = self.helper.wait_for_orchestration_with_output(data["statusQueryGetUri"], max_wait=300)
|
||||
assert status["runtimeStatus"] == "Completed"
|
||||
assert "output" in status
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
@@ -0,0 +1,219 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
"""
|
||||
Integration Tests for Workflow Human-in-the-Loop (HITL) Sample
|
||||
|
||||
Tests the workflow HITL sample demonstrating content moderation with human approval
|
||||
using the MAF request_info / @response_handler pattern.
|
||||
|
||||
The function app is automatically started by the test fixture.
|
||||
|
||||
Prerequisites:
|
||||
- Azure OpenAI credentials configured (see packages/azurefunctions/tests/integration_tests/.env.example)
|
||||
- Azurite running for durable orchestrations (or Azure Storage account configured)
|
||||
|
||||
Usage:
|
||||
# Start Azurite (if not already running)
|
||||
azurite &
|
||||
|
||||
# Run tests
|
||||
uv run pytest packages/azurefunctions/tests/integration_tests/test_12_workflow_hitl.py -v
|
||||
"""
|
||||
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
# Module-level markers - applied to all tests in this file
|
||||
pytestmark = [
|
||||
pytest.mark.flaky,
|
||||
pytest.mark.integration,
|
||||
pytest.mark.sample("12_workflow_hitl"),
|
||||
pytest.mark.usefixtures("function_app_for_test"),
|
||||
]
|
||||
|
||||
# Must match the workflow name in samples/04-hosting/azure_functions/12_workflow_hitl/function_app.py
|
||||
WORKFLOW_NAME = "content_moderation"
|
||||
|
||||
|
||||
@pytest.mark.orchestration
|
||||
class TestWorkflowHITL:
|
||||
"""Tests for 12_workflow_hitl sample."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _setup(self, base_url: str, sample_helper) -> None:
|
||||
"""Provide the helper and base URL for each test."""
|
||||
self.base_url = base_url
|
||||
self.helper = sample_helper
|
||||
|
||||
def _wait_for_hitl_request(self, instance_id: str, timeout: int = 40) -> dict:
|
||||
"""Polls for a pending HITL request."""
|
||||
start_time = time.time()
|
||||
while time.time() - start_time < timeout:
|
||||
status_response = self.helper.get(f"{self.base_url}/api/workflow/{WORKFLOW_NAME}/status/{instance_id}")
|
||||
if status_response.status_code == 200:
|
||||
status = status_response.json()
|
||||
pending_requests = status.get("pendingHumanInputRequests", [])
|
||||
if pending_requests:
|
||||
return status
|
||||
time.sleep(2)
|
||||
raise AssertionError(f"Timed out waiting for HITL request for instance {instance_id}")
|
||||
|
||||
def test_hitl_workflow_approval(self) -> None:
|
||||
"""Test HITL workflow with human approval."""
|
||||
payload = {
|
||||
"content_id": "article-test-001",
|
||||
"title": "Introduction to AI in Healthcare",
|
||||
"body": (
|
||||
"Artificial intelligence is revolutionizing healthcare by enabling faster diagnosis, "
|
||||
"personalized treatment plans, and improved patient outcomes. Machine learning algorithms "
|
||||
"can analyze medical images with remarkable accuracy."
|
||||
),
|
||||
"author": "Dr. Jane Smith",
|
||||
}
|
||||
|
||||
# Start orchestration
|
||||
response = self.helper.post_json(f"{self.base_url}/api/workflow/{WORKFLOW_NAME}/run", payload)
|
||||
assert response.status_code == 202
|
||||
data = response.json()
|
||||
assert "instanceId" in data
|
||||
assert "statusQueryGetUri" in data
|
||||
instance_id = data["instanceId"]
|
||||
|
||||
# Wait for the workflow to reach the HITL pause point
|
||||
status = self._wait_for_hitl_request(instance_id)
|
||||
|
||||
# Confirm status is valid
|
||||
assert status["runtimeStatus"] in ["Running", "Pending"]
|
||||
|
||||
# Get the request ID from pending requests
|
||||
pending_requests = status.get("pendingHumanInputRequests", [])
|
||||
assert len(pending_requests) > 0, "Expected pending HITL request"
|
||||
request_id = pending_requests[0]["requestId"]
|
||||
|
||||
# Send approval
|
||||
approval_response = self.helper.post_json(
|
||||
f"{self.base_url}/api/workflow/{WORKFLOW_NAME}/respond/{instance_id}/{request_id}",
|
||||
{"approved": True, "reviewer_notes": "Content is appropriate and well-written."},
|
||||
)
|
||||
assert approval_response.status_code == 200
|
||||
|
||||
# Wait for orchestration to complete
|
||||
final_status = self.helper.wait_for_orchestration(data["statusQueryGetUri"])
|
||||
assert final_status["runtimeStatus"] == "Completed"
|
||||
assert "output" in final_status
|
||||
|
||||
def test_hitl_workflow_rejection(self) -> None:
|
||||
"""Test HITL workflow with human rejection."""
|
||||
payload = {
|
||||
"content_id": "article-test-002",
|
||||
"title": "Get Rich Quick Scheme",
|
||||
"body": (
|
||||
"Click here NOW to make $10,000 overnight! This SECRET method is GUARANTEED to work! "
|
||||
"Limited time offer - act NOW before it's too late!"
|
||||
),
|
||||
"author": "Definitely Not Spam",
|
||||
}
|
||||
|
||||
# Start orchestration
|
||||
response = self.helper.post_json(f"{self.base_url}/api/workflow/{WORKFLOW_NAME}/run", payload)
|
||||
assert response.status_code == 202
|
||||
data = response.json()
|
||||
instance_id = data["instanceId"]
|
||||
|
||||
# Wait for the workflow to reach the HITL pause point
|
||||
status = self._wait_for_hitl_request(instance_id)
|
||||
|
||||
# Get the request ID from pending requests
|
||||
pending_requests = status.get("pendingHumanInputRequests", [])
|
||||
assert len(pending_requests) > 0, "Expected pending HITL request"
|
||||
request_id = pending_requests[0]["requestId"]
|
||||
|
||||
# Send rejection
|
||||
rejection_response = self.helper.post_json(
|
||||
f"{self.base_url}/api/workflow/{WORKFLOW_NAME}/respond/{instance_id}/{request_id}",
|
||||
{"approved": False, "reviewer_notes": "Content appears to be spam/scam material."},
|
||||
)
|
||||
assert rejection_response.status_code == 200
|
||||
|
||||
# Wait for orchestration to complete
|
||||
final_status = self.helper.wait_for_orchestration(data["statusQueryGetUri"])
|
||||
assert final_status["runtimeStatus"] == "Completed"
|
||||
assert "output" in final_status
|
||||
# The output should indicate rejection
|
||||
output = final_status["output"]
|
||||
assert "rejected" in str(output).lower()
|
||||
|
||||
def test_hitl_workflow_status_endpoint(self) -> None:
|
||||
"""Test that the workflow status endpoint shows pending HITL requests."""
|
||||
payload = {
|
||||
"content_id": "article-test-003",
|
||||
"title": "Test Article",
|
||||
"body": "This is a test article for checking status endpoint functionality.",
|
||||
"author": "Test Author",
|
||||
}
|
||||
|
||||
# Start orchestration
|
||||
response = self.helper.post_json(f"{self.base_url}/api/workflow/{WORKFLOW_NAME}/run", payload)
|
||||
assert response.status_code == 202
|
||||
data = response.json()
|
||||
instance_id = data["instanceId"]
|
||||
|
||||
# Wait for HITL pause
|
||||
status = self._wait_for_hitl_request(instance_id)
|
||||
|
||||
# Check status
|
||||
assert "instanceId" in status
|
||||
assert status["instanceId"] == instance_id
|
||||
assert "runtimeStatus" in status
|
||||
assert "pendingHumanInputRequests" in status
|
||||
|
||||
# Clean up: approve to complete
|
||||
pending_requests = status.get("pendingHumanInputRequests", [])
|
||||
if pending_requests:
|
||||
request_id = pending_requests[0]["requestId"]
|
||||
self.helper.post_json(
|
||||
f"{self.base_url}/api/workflow/{WORKFLOW_NAME}/respond/{instance_id}/{request_id}",
|
||||
{"approved": True, "reviewer_notes": ""},
|
||||
)
|
||||
|
||||
# Wait for completion
|
||||
self.helper.wait_for_orchestration(data["statusQueryGetUri"])
|
||||
|
||||
def test_hitl_workflow_with_neutral_content(self) -> None:
|
||||
"""Test HITL workflow with neutral content that should get medium risk."""
|
||||
payload = {
|
||||
"content_id": "article-test-004",
|
||||
"title": "Product Review",
|
||||
"body": (
|
||||
"This product works as advertised. The build quality is average and the price "
|
||||
"is reasonable. I would recommend it for basic use cases but not for professional work."
|
||||
),
|
||||
"author": "Regular User",
|
||||
}
|
||||
|
||||
# Start orchestration
|
||||
response = self.helper.post_json(f"{self.base_url}/api/workflow/{WORKFLOW_NAME}/run", payload)
|
||||
assert response.status_code == 202
|
||||
data = response.json()
|
||||
instance_id = data["instanceId"]
|
||||
|
||||
# Wait for HITL pause
|
||||
status = self._wait_for_hitl_request(instance_id)
|
||||
|
||||
pending_requests = status.get("pendingHumanInputRequests", [])
|
||||
assert len(pending_requests) > 0
|
||||
request_id = pending_requests[0]["requestId"]
|
||||
|
||||
# Approve
|
||||
self.helper.post_json(
|
||||
f"{self.base_url}/api/workflow/{WORKFLOW_NAME}/respond/{instance_id}/{request_id}",
|
||||
{"approved": True, "reviewer_notes": "Approved after review."},
|
||||
)
|
||||
|
||||
# Wait for completion
|
||||
final_status = self.helper.wait_for_orchestration(data["statusQueryGetUri"])
|
||||
assert final_status["runtimeStatus"] == "Completed"
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
+150
@@ -0,0 +1,150 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
"""
|
||||
Integration Tests for the Sub-workflow HITL Sample (13_subworkflow_hitl)
|
||||
|
||||
Tests nested human-in-the-loop through the Azure Functions host: the HITL pause
|
||||
lives inside an inner workflow embedded via ``WorkflowExecutor``, so the pending
|
||||
request surfaces at the top-level instance with a **qualified** request id
|
||||
(``review_sub~0~{requestId}``). The caller responds against the top-level instance
|
||||
and the host routes it to the owning child orchestration.
|
||||
|
||||
This sample hosts no AI agents, so it exercises the AF nested-HITL plumbing
|
||||
deterministically (no model latency / variability).
|
||||
|
||||
The function app is automatically started by the test fixture.
|
||||
|
||||
Prerequisites:
|
||||
- Azurite running for durable orchestrations
|
||||
- Durable Task Scheduler emulator running on localhost:8080
|
||||
|
||||
Usage:
|
||||
uv run pytest packages/azurefunctions/tests/integration_tests/test_13_workflow_subworkflow_hitl.py -v
|
||||
"""
|
||||
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
# Module-level markers - applied to all tests in this file
|
||||
pytestmark = [
|
||||
pytest.mark.flaky,
|
||||
pytest.mark.integration,
|
||||
pytest.mark.sample("13_subworkflow_hitl"),
|
||||
pytest.mark.usefixtures("function_app_for_test"),
|
||||
]
|
||||
|
||||
# Must match the outer workflow name in samples/.../13_subworkflow_hitl/function_app.py
|
||||
WORKFLOW_NAME = "moderation_pipeline"
|
||||
# The WorkflowExecutor node id that embeds the inner HITL workflow.
|
||||
SUBWORKFLOW_NODE_ID = "review_sub"
|
||||
|
||||
|
||||
@pytest.mark.orchestration
|
||||
class TestSubworkflowHITL:
|
||||
"""Tests for the 13_subworkflow_hitl sample (nested HITL behind one surface)."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _setup(self, base_url: str, sample_helper) -> None:
|
||||
"""Provide the helper and base URL for each test."""
|
||||
self.base_url = base_url
|
||||
self.helper = sample_helper
|
||||
|
||||
def _wait_for_hitl_request(self, instance_id: str, timeout: int = 40) -> dict:
|
||||
"""Poll the top-level status endpoint until a (nested) HITL request appears."""
|
||||
start_time = time.time()
|
||||
while time.time() - start_time < timeout:
|
||||
status_response = self.helper.get(f"{self.base_url}/api/workflow/{WORKFLOW_NAME}/status/{instance_id}")
|
||||
if status_response.status_code == 200:
|
||||
status = status_response.json()
|
||||
if status.get("pendingHumanInputRequests"):
|
||||
return status
|
||||
time.sleep(2)
|
||||
raise AssertionError(f"Timed out waiting for a nested HITL request for instance {instance_id}")
|
||||
|
||||
def _start(self, payload: dict) -> dict:
|
||||
"""Start the outer workflow and return the run response JSON."""
|
||||
response = self.helper.post_json(f"{self.base_url}/api/workflow/{WORKFLOW_NAME}/run", payload)
|
||||
assert response.status_code == 202
|
||||
return response.json()
|
||||
|
||||
def test_nested_request_surfaces_with_qualified_id(self) -> None:
|
||||
"""The nested pending request is surfaced with a ``review_sub~0~{id}`` qualified id."""
|
||||
data = self._start({
|
||||
"content_id": "article-100",
|
||||
"title": "Quarterly Roadmap",
|
||||
"body": "A summary of the upcoming features planned for the next quarter.",
|
||||
})
|
||||
instance_id = data["instanceId"]
|
||||
|
||||
status = self._wait_for_hitl_request(instance_id)
|
||||
pending = status.get("pendingHumanInputRequests", [])
|
||||
assert len(pending) == 1
|
||||
request_id = pending[0]["requestId"]
|
||||
|
||||
# The qualifier carries the node id and the child's ordinal (0 for the single
|
||||
# dispatch), then the inner bare request id: ``review_sub~0~{requestId}``.
|
||||
expected_prefix = f"{SUBWORKFLOW_NODE_ID}~0~"
|
||||
assert request_id.startswith(expected_prefix), request_id
|
||||
assert request_id[len(expected_prefix) :] # non-empty inner id
|
||||
|
||||
# The respondUrl always targets the top-level instance.
|
||||
assert f"/api/workflow/{WORKFLOW_NAME}/respond/{instance_id}/" in pending[0]["respondUrl"]
|
||||
|
||||
# Drain the pause so the instance does not hang.
|
||||
approve = self.helper.post_json(
|
||||
f"{self.base_url}/api/workflow/{WORKFLOW_NAME}/respond/{instance_id}/{request_id}",
|
||||
{"approved": True, "reviewer_notes": "ok"},
|
||||
)
|
||||
assert approve.status_code == 200
|
||||
self.helper.wait_for_orchestration(data["statusQueryGetUri"])
|
||||
|
||||
def test_nested_hitl_approval(self) -> None:
|
||||
"""Responding 'approved' to the nested request resumes the outer workflow to APPROVED."""
|
||||
data = self._start({
|
||||
"content_id": "article-001",
|
||||
"title": "Introduction to AI in Healthcare",
|
||||
"body": (
|
||||
"Artificial intelligence is improving healthcare by enabling faster diagnosis, "
|
||||
"personalized treatment plans, and better patient outcomes."
|
||||
),
|
||||
})
|
||||
instance_id = data["instanceId"]
|
||||
|
||||
status = self._wait_for_hitl_request(instance_id)
|
||||
request_id = status["pendingHumanInputRequests"][0]["requestId"]
|
||||
|
||||
approval = self.helper.post_json(
|
||||
f"{self.base_url}/api/workflow/{WORKFLOW_NAME}/respond/{instance_id}/{request_id}",
|
||||
{"approved": True, "reviewer_notes": "Looks good."},
|
||||
)
|
||||
assert approval.status_code == 200
|
||||
|
||||
final_status = self.helper.wait_for_orchestration(data["statusQueryGetUri"])
|
||||
assert final_status["runtimeStatus"] == "Completed"
|
||||
assert "APPROVED" in str(final_status.get("output")).upper()
|
||||
|
||||
def test_nested_hitl_rejection(self) -> None:
|
||||
"""Responding 'rejected' to the nested request resumes the outer workflow to REJECTED."""
|
||||
data = self._start({
|
||||
"content_id": "article-002",
|
||||
"title": "Get Rich Quick",
|
||||
"body": "Click here NOW to make $10,000 overnight! GUARANTEED! Limited time offer!",
|
||||
})
|
||||
instance_id = data["instanceId"]
|
||||
|
||||
status = self._wait_for_hitl_request(instance_id)
|
||||
request_id = status["pendingHumanInputRequests"][0]["requestId"]
|
||||
|
||||
rejection = self.helper.post_json(
|
||||
f"{self.base_url}/api/workflow/{WORKFLOW_NAME}/respond/{instance_id}/{request_id}",
|
||||
{"approved": False, "reviewer_notes": "Violates content policy."},
|
||||
)
|
||||
assert rejection.status_code == 200
|
||||
|
||||
final_status = self.helper.wait_for_orchestration(data["statusQueryGetUri"])
|
||||
assert final_status["runtimeStatus"] == "Completed"
|
||||
assert "REJECTED" in str(final_status.get("output")).upper()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
Reference in New Issue
Block a user