Files
tracer-cloud--opensre/tests/e2e/rca/openclaw_gateway_crashed.md
T
wehub-resource-sync 4b6817381b
CI (OpenClaw E2E) / openclaw test (push) Has been cancelled
CI / coverage-report (push) Has been cancelled
CI / test-kubernetes (push) Has been cancelled
CI / should-run-thorough (push) Has been cancelled
CI / test-thorough (cloudwatch-demo) (push) Has been cancelled
CI / test-thorough (flink-ecs) (push) Has been cancelled
CI / test-thorough (upstream-lambda) (push) Has been cancelled
CI / test-thorough (prefect-ecs-fargate) (push) Has been cancelled
Release / build-binaries (zip, opensre.exe, onefile, windows-latest, windows-x64) (push) Has been cancelled
Benchmark image — build + push to ECR (any adapter) / build + push (push) Has been cancelled
CI / quality (ubuntu-latest) (push) Has been cancelled
CI / test (tools-runtime) (push) Has been cancelled
CI / test (e2e-general) (push) Has been cancelled
CI / test (cli-runtime) (push) Has been cancelled
CI / test (e2e-provider-and-openclaw) (push) Has been cancelled
CI / test (integrations-and-misc) (push) Has been cancelled
Release / verify (push) Has been cancelled
Release / build-python-dist (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, macos-15-intel, darwin-x64) (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, macos-latest, darwin-arm64) (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, ubuntu-22.04, linux-x64) (push) Has been cancelled
Release / publish-release (push) Has been cancelled
Release / publish-main-release (push) Has been cancelled
Interactive Shell Live (PR + post-merge) / turn-checks (no-LLM) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Interactive Shell Live (PR + post-merge) / turn-live shard ${{ matrix.shard_index }} (push) Has been cancelled
Release / prepare (push) Has been cancelled
Release / build-binaries (tar.gz, opensre, onedir, ubuntu-22.04-arm, linux-arm64) (push) Has been cancelled
Synthetic Deterministic Tests / Synthetic offline (deterministic) (push) Has been cancelled
chore: import upstream snapshot with attribution
2026-07-13 13:10:45 +08:00

4.5 KiB

Alert: [FIRING:1] OpenClaw Gateway Unavailable — MCP Bridge Down

Source

OpenClaw local host (stdio MCP transport)

Message

Firing

The OpenClaw MCP gateway process has stopped responding on the host machine. opensre integrations verify openclaw is returning:

OpenClaw bridge validation failed: Connection closed.
Hint: The `openclaw mcp serve` bridge needs a running OpenClaw Gateway.
Check `openclaw gateway status`, then start it with `openclaw gateway run`
(foreground) or `openclaw gateway install` followed by `openclaw gateway start`.

All three bridge tools (list_openclaw_tools, search_openclaw_conversations, call_openclaw_bridge_tool) are permanently unavailable for the duration of the outage. Investigation write-back to OpenClaw is also failing.

Labels:

  • alertname = OpenClawGatewayUnavailable
  • severity = critical
  • service = openclaw-gateway
  • environment = local
  • pipeline_name = openclaw_mcp

Annotations:

  • error = Connection closed
  • command = openclaw mcp serve
  • last_successful_heartbeat = 2026-05-11T08:34:00Z

Alert Metadata

{
  "title": "[FIRING:1] OpenClaw Gateway Unavailable — MCP Bridge Down",
  "state": "alerting",
  "commonLabels": {
    "alertname": "OpenClawGatewayUnavailable",
    "severity": "critical",
    "service": "openclaw-gateway",
    "environment": "local",
    "pipeline_name": "openclaw_mcp"
  },
  "commonAnnotations": {
    "summary": "OpenClaw MCP gateway process is not running. All MCP bridge calls returning 'Connection closed'.",
    "description": "The openclaw gateway process stopped responding at 08:42 UTC. opensre tried to connect via 'openclaw mcp serve' stdio transport and received 'Connection closed' on every attempt. Last successful heartbeat was 8 minutes ago. Engineers cannot use OpenClaw's AI assistant and no new MCP sessions can be established.",
    "error": "Connection closed",
    "command": "openclaw mcp serve",
    "transport": "stdio",
    "last_successful_heartbeat": "2026-05-11T08:34:00Z",
    "fault_injection_script": "openclaw gateway stop",
    "fix": "openclaw gateway start"
  },
  "version": "4",
  "groupKey": "{}:{alertname=\"OpenClawGatewayUnavailable\"}",
  "truncatedAlerts": 0,
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "OpenClawGatewayUnavailable",
        "severity": "critical",
        "service": "openclaw-gateway",
        "environment": "local",
        "instance": "openclaw-host-prod-01:18789"
      },
      "annotations": {
        "summary": "OpenClaw MCP gateway process is not running",
        "description": "Connection closed when attempting 'openclaw mcp serve' stdio transport.",
        "error": "Connection closed"
      },
      "startsAt": "2026-05-11T08:42:00Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "",
      "fingerprint": "openclaw-gateway-crashed-001"
    }
  ]
}

Fault Injection Script

The following script reproduces this alert against a real local OpenClaw install:

#!/usr/bin/env bash
# inject_gateway_crash.sh — stop the OpenClaw gateway to trigger the alert
set -euo pipefail

echo "[inject] Stopping OpenClaw gateway..."
openclaw gateway stop 2>/dev/null || pkill -f "openclaw gateway" 2>/dev/null || true

echo "[inject] Verifying fault is active..."
sleep 1
opensre integrations verify openclaw && echo "ERROR: gateway still up" && exit 1 || true

echo "[inject] Fault confirmed. Gateway is down."
echo "[inject] Run 'opensre investigate' with the alert JSON above to get RCA."
echo "[inject] To restore: openclaw gateway start"