siriusscan--sirius/documentation/OPERATIONS.md

Sirius Operations & Troubleshooting Runbook

This document contains operational procedures, verification runbooks, and advanced troubleshooting for Sirius maintainers and operators. For getting started, see the README.

Clean Rollout Verification (Fresh Clone)

Use this sequence to validate a production rollout from a fresh checkout.

# 1) Start from a clean runtime state
docker compose down -v --remove-orphans

# 2) Generate runtime secrets/config
docker compose -f docker-compose.installer.yaml run --rm sirius-installer --non-interactive --no-print-secrets

# 3) Build from local source deterministically
docker compose -f docker-compose.yaml -f docker-compose.build.yaml up -d --build

# 4) Confirm all services are healthy
docker compose ps

# 5) Validate API auth behavior
curl -i http://localhost:9001/host/            # expect 401 (no key)
# Internal key: prefer the mounted secret file path inside the container, else env.
RUNTIME_KEY=$(docker exec sirius-api sh -lc 'if [ -r "${SIRIUS_API_KEY_FILE:-}" ]; then tr -d "\r\n" < "$SIRIUS_API_KEY_FILE"; elif [ -n "${SIRIUS_API_KEY:-}" ]; then printf %s "$SIRIUS_API_KEY"; fi')
curl -i -H "X-API-Key: ${RUNTIME_KEY}" http://localhost:9001/host/   # expect 200

# 6) Ensure startup regressions are absent
docker compose logs --no-color sirius-ui sirius-engine | rg -i "ENOTFOUND|permission denied|Failed to open log file"

If step 6 returns any lines, capture full logs and investigate before rollout.

Migrating existing .env deployments to file-based internal key

If docker compose up errors on the sirius_api_key secret file, create it once from your current key (same value as SIRIUS_API_KEY in .env):

mkdir -p secrets
printf '%s\n' "$SIRIUS_API_KEY" > secrets/sirius_api_key.txt
chmod 644 secrets/sirius_api_key.txt

Use 644 (owner read/write, world read) on secrets/sirius_api_key.txt, not 600. Compose bind-mounts that file into the container as /run/secrets/sirius_api_key; sirius-api runs as a non-root UID (1001) and must be able to open it. 600 keeps “other” from reading, so you get permission denied on /run/secrets/sirius_api_key even when SIRIUS_API_KEY is set in .env (older API builds failed before env fallback; current sirius-api falls back to SIRIUS_API_KEY if the file is unreadable when the env is set). The installer writes this file as 644 by default.

Re-run the installer to add SIRIUS_API_KEY_FILE=/run/secrets/sirius_api_key to .env if it is missing. Services accept either the file or SIRIUS_API_KEY during transition.

Release Image Propagation Verification

Use this path to validate what operators experience when running pulled release images.

# 1) Ensure local builds are not used
docker compose down -v --remove-orphans

# 2) Generate runtime secrets/config
docker compose -f docker-compose.installer.yaml run --rm sirius-installer --non-interactive --no-print-secrets

# 3) Validate the public release path for a published tag
export IMAGE_TAG=v0.4.1
bash scripts/validate-public-compose-path.sh "$IMAGE_TAG"

# 4) Verify running container image IDs match pulled release images
bash scripts/verify-release-images.sh

Expected result: the public-stack validator passes, all checks print a pass, and no service is running an unexpected local image.

If verify-ghcr-public-access.sh reports Anonymous access denied, treat it as a registry visibility failure and stop before rollout. If it reports Manifest missing, the selected release tag was not published successfully. If validate-public-compose-path.sh fails after pull succeeds, investigate runtime contract drift before rollout.

Runtime Auth Contract Verification

Use this check any time you run reset, switch between source/release mode, or see 401 and DB auth errors.

bash scripts/verify-runtime-auth-contract.sh

The script assumes default Docker Compose container names (sirius-ui, sirius-api, sirius-engine, sirius-postgres) and curls the API at http://localhost:9001. For a different Compose project name (non-default container_name pattern), set:

• SIRIUS_CONTRACT_CONTAINER_UI, SIRIUS_CONTRACT_CONTAINER_API, SIRIUS_CONTRACT_CONTAINER_ENGINE, SIRIUS_CONTRACT_CONTAINER_POSTGRES
• SIRIUS_API_PUBLIC_URL if the published API port differs

See the header comment in scripts/verify-runtime-auth-contract.sh.

If this script fails, do not start new scans until the mismatch is corrected.

If sirius-engine is still restarting after this passes, verify runtime preflight tooling:

docker exec sirius-engine sh -lc 'which psql && psql --version'

Expected result: prints /usr/bin/psql and a PostgreSQL client version. If missing, pull the corrected release image and recreate services.

Scan-Stuck Troubleshooting Runbook

If scans complete in backend logs but UI remains non-terminal, run:

# 0) Do NOT use command-scoped secret overrides for single-service restarts.
# Bad (causes key drift): SIRIUS_API_KEY=local-dev docker compose up -d sirius-engine
# Good: keep secrets in .env and recreate dependent services together.

# 1) Verify API key contract is consistent across services
docker inspect sirius-ui --format '{{range .Config.Env}}{{println .}}{{end}}' | rg '^SIRIUS_API_KEY='
docker inspect sirius-api --format '{{range .Config.Env}}{{println .}}{{end}}' | rg '^SIRIUS_API_KEY='
docker inspect sirius-engine --format '{{range .Config.Env}}{{println .}}{{end}}' | rg '^SIRIUS_API_KEY='

# 2) Check engine scanner warnings and terminal status persistence
docker compose logs --no-color sirius-engine | rg -i "source-aware|status|completed|failed|warning|401"

# 3) Check UI auth/session and API bridge logs
docker compose logs --no-color sirius-ui | rg -i "JWT_SESSION_ERROR|SIRIUS_API_KEY|fetch failed|401"

# 4) Verify DB credential consistency from runtime containers
docker compose logs --no-color sirius-postgres sirius-api sirius-engine | rg -i "password authentication failed|database connection not available"

# 5) Run contract verifier
bash scripts/verify-runtime-auth-contract.sh

# 6) Verify templates endpoint is populated and not in missing/empty state
API_KEY=$(docker inspect sirius-api --format '{{range .Config.Env}}{{println .}}{{end}}' | rg '^SIRIUS_API_KEY=' | sed 's/^SIRIUS_API_KEY=//')
curl -s -D - -o /tmp/sirius-templates.json -H "X-API-Key: ${API_KEY}" http://localhost:9001/templates | rg '^HTTP/|^X-Sirius-Template-State'
python3 -c 'import json; print(len(json.load(open("/tmp/sirius-templates.json"))))'

If any command surfaces key/secret mismatch, re-run installer and restart:

docker compose -f docker-compose.installer.yaml run --rm sirius-installer --non-interactive --no-print-secrets
docker compose up -d --force-recreate

Expected result for step 6: HTTP 200 and template count >= 1. If X-Sirius-Template-State: missing or empty appears, sirius-engine has not initialized template data yet.

Host Discovery Validation

# Confirm compose renders successfully and includes NET_RAW
docker compose config | rg "NET_RAW"

# Confirm scanner system template is canonicalized on startup
docker compose exec sirius-valkey valkey-cli GET template:quick | rg '"scan_types"'

# Run a scan from UI/API, then verify queue consumers and scan state
docker compose exec sirius-rabbitmq rabbitmqctl list_queues name consumers messages_ready messages_unacknowledged | rg "scan|scan_control"
docker compose exec sirius-valkey valkey-cli GET currentScan

Emergency Recovery

Complete System Reset

# Stop all services
docker compose down

# Remove all data (this deletes all scan data)
docker compose down -v

# Clean Docker system
docker system prune -a -f

# Recreate .env using installer (required after reset)
docker compose -f docker-compose.installer.yaml run --rm sirius-installer --non-interactive --no-print-secrets

# Fresh source rebuild
docker compose -f docker-compose.yaml -f docker-compose.build.yaml up -d --build

# Verify auth contract before interacting with UI
bash scripts/verify-runtime-auth-contract.sh

Backup Current Data

# Backup database
docker exec sirius-postgres pg_dump -U postgres sirius > backup.sql

# Backup scan results directory
docker cp sirius-engine:/opt/sirius/ ./sirius-backup/

Common Issues

Container Issues

Services fail to start:

docker compose ps              # Check service status
docker compose logs <service>  # View service logs
docker system df              # Check disk space

Infrastructure services don't start (dev overlay):

# The dev file is an OVERRIDE file, not standalone.
# Wrong:  docker compose -f docker-compose.dev.yaml up -d
# Correct:
docker compose -f docker-compose.yaml -f docker-compose.dev.yaml up -d

Port already in use:

lsof -i :3000
# Stop the conflicting process, then restart

Scanner Issues

Nmap errors or scanning failures:

docker logs sirius-engine | grep -i nmap
docker exec sirius-engine nmap --version
docker restart sirius-engine

Database Issues

Connection failures:

docker exec sirius-postgres pg_isready
docker logs sirius-postgres
docker exec sirius-postgres psql -U postgres -d sirius -c "SELECT version();"

Message Queue Issues

RabbitMQ connectivity or schema errors:

docker exec sirius-rabbitmq rabbitmqctl status
docker exec sirius-rabbitmq rabbitmqctl list_queues
# For schema integrity failures, remove old volumes:
docker compose down -v && docker compose up -d

Network Issues

Services can't communicate:

docker exec sirius-ui ping sirius-api
docker network inspect sirius

GHCR Pull Issues

docker compose pull returns unauthorized:

bash scripts/verify-ghcr-public-access.sh "${IMAGE_TAG:-latest}"

Expected result: all compose-rendered GHCR refs pass anonymously. If the script reports Anonymous access denied, the package visibility contract is broken and the GHCR publicization workflow or token scope needs maintenance.

docker compose pull returns manifest unknown or not found:

bash scripts/verify-ghcr-public-access.sh "${IMAGE_TAG:-latest}"

Expected result: the script identifies the missing tag explicitly. Re-run the publishing workflow for the missing release tag before retrying deployment.

Maintainer: GHCR distribution checklist (public operators)

A Git tag and GitHub Release do not create container tags on GHCR. Third parties need all six images (sirius-ui, sirius-api, sirius-engine, sirius-postgres, sirius-rabbitmq, sirius-valkey) published under the same semver tag, and each package must be public for anonymous docker pull.

1. Package visibility (per package) — In SiriusScan org packages, open each container image, set Package settings to Public, and link the package to this repository if needed. Partial visibility explains “works when logged in” vs failures for everyone else (see issue #119).
2. Publish semver tags — Run Publish Release Image Tags with source_tag = the tag that exists on GHCR for all six (usually latest) and target_tag = the release (e.g. v1.0.0). Confirm the workflow run succeeds end-to-end.
3. Verify anonymously — From a machine not logged in to ghcr.io:

bash scripts/verify-ghcr-public-access.sh v1.0.0

CI runs the same check on every published release and weekly via .github/workflows/verify-ghcr-release-tag.yml.