tracer-cloud--opensre/tests/synthetic/hermes

Hermes Synthetic Suite

Real-world Hermes log fixtures used to lock in the behavior of integrations.hermes.IncidentClassifier. Each scenario captures a production pattern observed on a Hermes deployment, ships the raw errors.log slice that triggered it, and declares — in answer.yml — the incidents the classifier is expected to emit.

The suite runs offline: there is no LLM, no live infrastructure, and no mocking. The classifier is rule-based, so the assertions are exact.

Layout

tests/synthetic/hermes/
├── __init__.py
├── README.md                  ← this file
├── scenario_loader.py         ← loads scenarios into typed fixtures
├── test_suite.py              ← parametrized pytest entrypoint
└── 000-telegram-polling-conflict/
    ├── README.md              ← scenario narrative + real-world cause
    ├── errors.log             ← raw Hermes log fixture
    ├── scenario.yml           ← metadata (id, source, evidence files)
    └── answer.yml             ← expected incidents

Scenario schema

scenario.yml

scenario_id: "000-telegram-polling-conflict"
title: "Two Hermes instances share a TELEGRAM_BOT_TOKEN"
source: "production-hermes-mac+ec2"
log_file: "errors.log"
classifier:
  warning_burst_threshold: 5
  warning_burst_window_s: 60
  traceback_followup_s: 5

The classifier block is optional; omit it to use the package defaults. Override only when a scenario specifically needs different thresholds.

answer.yml

expected_incidents:
  - rule: "warning_burst"
    severity: "medium"
    logger: "gateway.platforms.telegram"
    min_records: 5
expected_incident_count:
  warning_burst: ">=1"
  error_severity: "==0"
  traceback: "==0"

expected_incidents is an ordered list of partial matches: each entry must match at least one emitted incident in the order given (without consuming a previously matched incident). expected_incident_count asserts the total per-rule emission count using the operators ==, >=, <=, >, <.

Adding a new scenario

1. Capture a slice of the Hermes log that demonstrates the pattern. Strip irrelevant lines but keep timestamps intact — the classifier uses them for the warning-burst window.
2. Create a new numbered directory NNN-short-name/ under this folder.
3. Drop the log into errors.log and write the two YAML files.
4. Run uv run pytest tests/synthetic/hermes -q to confirm the classifier matches your answer key.
5. Commit the scenario and document the real-world cause in the per-scenario README.md.