siriusscan--sirius/documentation/dev/operations/README.api-key-operations.md

title, description, template, llm_context, categories, tags, related_docs

title	description	template	llm_context	categories	tags	related_docs
API Key Operations Runbook	Production runbook for stateless service API key rotation, recovery, and incident response.	TEMPLATE.guide	high	operations security deployment	apikey rotation incident-response valkey runbook	README.development.md README.docker-container-deployment.md README.auth-surface-matrix.md

API Key Operations Runbook

This runbook defines production-safe operations for Sirius service API keys.

Scope

Applies to service-to-service credential SIRIUS_API_KEY used by:

• sirius-api
• sirius-ui (server-side tRPC -> API calls)
• sirius-engine and agent/scanner API clients

Baseline Invariants

• SIRIUS_API_KEY must be non-empty in production.
• Go API must reject requests without a valid X-API-Key.
• Only /health is public.
• Root service key validation is stateless from runtime environment configuration.
• Valkey stores only dynamic/user-generated API key records.

Rotation Procedure (No Downtime)

1) Create new key

1. Authenticate as admin in UI.
2. Create a new API key in Settings.
3. Record:
   • key value (shown once)
   • key id/hash
   • label

2) Deploy new key to services

1. Update deployment secret store or environment value for SIRIUS_API_KEY with the new key.
2. Roll out services in order:
   1. sirius-api
   2. sirius-ui
   3. sirius-engine
3. Validate:
   • API 401 rate does not spike.
   • UI can read/write through tRPC.
   • scanner/agent host submissions continue.

3) Revoke old key

1. After rollout and validation period, revoke old key in UI.
2. Confirm old key returns 401.
3. Close rotation ticket.

Incident: Valkey Data Loss

Symptoms:

• sudden 401s for user-generated keys
• API key list empty
• dynamic key operations fail while root key requests still succeed

Recovery:

1. Restore Valkey from backup if available.
2. If no backup:
   • ensure production SIRIUS_API_KEY is correctly set for all services.
   • restart sirius-api first; root key path remains stateless.
3. Restart sirius-ui and sirius-engine.
4. Validate end-to-end operations and security harness results.

Incident: Root Key Mismatch (Configuration Drift)

Symptoms:

• services return 401 with old key after key rotation
• env values differ between sirius-ui, sirius-api, and sirius-engine

Recovery:

1. Confirm deployed SIRIUS_API_KEY value in runtime environment.
2. Restart sirius-api, then sirius-ui and sirius-engine.
3. Re-check key validation success for deployed root key.
4. If still failing, capture API logs and run security harness auth-surface and api suites.

Incident: Unauthorized Agent/Scanner API Writes

Symptoms:

• scanner/agent host submissions return 401

Recovery:

1. Verify SIRIUS_API_KEY is present in engine/scanner/agent runtime.
2. Verify clients send X-API-Key.
3. Validate key exists in API key list and is not revoked.
4. Roll service restart after secret correction.

Validation Commands

Run security checks from repository root:

cd testing/security
go run . --suite api
go run . --suite trpc
go run . --suite auth-surface

Operational Checklist

• SIRIUS_API_KEY secret exists and is non-empty in production.
• API middleware accepts configured root key on non-health routes.
• UI + engine are using the same active service key.
• Old keys are revoked only after rollout validation.
• Recovery procedure for config drift or Valkey loss is documented in incident ticket.