tracer-cloud--opensre/tests/e2e/AGENTS.md

Test Specification Principles

1. Separation of Concerns: Pure Business Logic

Principle: Pipeline business logic (use_case.py) must be completely isolated from test orchestration and observability code.

Why: Simulates real customer code that has no awareness of Tracer, RCA, or investigation infrastructure. Tests the agent's ability to investigate production-like failures.

Pattern:

# use_case.py - Pure business logic, no test infrastructure
def extract_and_validate(input_path: str) -> str:
    if not os.path.exists(input_path):
        raise FileNotFoundError(f"input file not present: {input_path}")
    return data

# test_orchestrator.py - All test orchestration separate
def main():
    try:
        result = use_case.main()  # Run pure business logic
    except Exception as e:
        # Orchestration: logging, alerting, investigation

Anti-pattern: Mixing test infrastructure with business logic.

---

2. Real End-to-End Testing: No Mocking

Principle: Tests must trigger real failures using actual AWS services, real APIs, and actual infrastructure. No mocked services or simulated failures.

Why: Validates the agent can investigate production-like scenarios with real AWS APIs, real error messages, and real data lineage.

Pattern:

# Trigger actual failure via HTTP to real Lambda
response = requests.post(
    UPSTREAM_DOWNSTREAM_CONFIG["ingester_api_url"],
    json={"correlation_id": correlation_id, "inject_schema_change": True},
)

# Query real CloudWatch logs
logs_client = boto3.client("logs")
response = logs_client.filter_log_events(
    logGroupName=log_group,
    filterPattern=correlation_id,
)

Requirements:

• Real AWS infrastructure (CloudWatch, S3, Lambda)
• Real HTTP endpoints that can fail
• Real data written to real buckets
• Real error messages from real services

---

3. Traceable Investigation Metadata

Principle: Every investigation must be decorated with @traceable and include comprehensive metadata for observability and debugging.

Why: Enables tracking investigation quality, debugging agent behavior, and measuring performance over time.

Pattern:

@traceable(
    name=f"Pipeline Investigation - {raw_alert['alert_id'][:8]}",
    metadata={
        "alert_id": raw_alert["alert_id"],
        "pipeline_name": pipeline_name,
        "correlation_id": failure_data["correlation_id"],
        "s3_key": failure_data["s3_key"],
    },
)
def run_investigation():
    return _run(
        alert_name=f"Pipeline failure: {pipeline_name}",
        pipeline_name=pipeline_name,
        severity="critical",
        raw_alert=raw_alert,
    )

Required Metadata:

• alert_id: Unique identifier for this investigation
• pipeline_name: Which pipeline is being investigated
• correlation_id or run_id: Trace the failure through logs
• Context-specific keys (s3_key, log_group, function_name, etc.)

---

4. Alert Factory Pattern: Standardized Alert Structure

Principle: All tests must use the create_alert factory to generate alerts with proper structure and annotations.

Why: Ensures consistent alert format, proper context source declarations, and complete metadata for investigation.

Pattern:

raw_alert = create_alert(
    pipeline_name=pipeline_name,
    run_name=run_id,
    status="failed",
    timestamp=datetime.now(UTC).isoformat(),
    annotations={
        "s3_bucket": failure_data["bucket"],
        "s3_key": failure_data["s3_key"],
        "correlation_id": failure_data["correlation_id"],
        "error": failure_data["error_message"],
        "lambda_log_group": failure_data["log_group"],
        "function_name": config["mock_dag_function_name"],
        "context_sources": "s3,lambda,cloudwatch",  # Declares available evidence
    },
)

Required Alert Fields:

• pipeline_name: Which pipeline failed
• run_name: Unique run identifier
• status: "failed" (or "success" for negative tests)
• timestamp: When the failure occurred
• annotations: All context needed for investigation
• annotations.context_sources: Comma-separated list of available evidence sources

---

5. Failure-First Test Design

Principle: Tests are designed to fail first, capture complete failure context, then invoke investigation. The failure is the test case.

Why: Validates the agent can investigate real failures with realistic incomplete information, not artificial success scenarios.

Test Flow Pattern:

def main():
    # Step 1: Trigger real failure
    failure_data = trigger_pipeline_failure()
    
    # Step 2: Capture failure context (logs, metrics, data)
    error_message = extract_error_from_cloudwatch(failure_data)
    
    # Step 3: Create alert with captured context
    raw_alert = create_alert(annotations={...failure_data...})
    
    # Step 4: Invoke investigation agent
    result = run_investigation(raw_alert)
    
    # Step 5: Validate investigation quality
    assert result.get('validity_score') > 0.7

Anti-pattern: Testing happy paths or artificially injecting failures after the fact.

---

6. Context Source Annotations: Investigation Strategy Hints

Principle: Alerts must explicitly declare which evidence sources are available via context_sources annotation.

Why: Guides the agent's investigation strategy by declaring upfront what data sources exist (CloudWatch logs, S3 objects, Lambda configs, etc.).

Pattern:

annotations={
    # Evidence source declarations
    "context_sources": "s3,lambda,cloudwatch",
    
    # S3 context
    "s3_bucket": "landing-bucket",
    "s3_key": "raw/data/2024/file.json",
    
    # Lambda context
    "function_name": "processor-function",
    "lambda_log_group": "/aws/lambda/processor",
    
    # CloudWatch context
    "cloudwatch_log_group": "/ecs/pipeline",
    "correlation_id": "run-123",
}

Valid Context Sources:

• cloudwatch: CloudWatch logs are available
• s3: S3 objects/metadata are available
• lambda: Lambda function configs/logs are available
• batch: AWS Batch job information
• tracer_web: Tracer platform pipeline metadata
• storage: General storage layer (S3, EFS, etc.)

Agent Behavior: The investigation node uses context_sources to determine which investigation_actions to execute, avoiding wasted API calls to unavailable services.

---