siriusscan--sirius/documentation/dev/deployment

title, description, template, version, last_updated, author, tags, categories, difficulty, prerequisites, related_docs, dependencies, llm_context, search_keywords

title	description	template	version	last_updated	author	tags	categories	difficulty	prerequisites	related_docs	dependencies	llm_context	search_keywords
GitHub Actions Workflow Architecture	Comprehensive guide to Sirius CI/CD pipeline structure, parallel build jobs, and deployment workflows	TEMPLATE.documentation-standard	1.0.0	2025-11-14	Sirius Team	ci-cd github-actions docker ghcr parallel-builds	deployment automation	intermediate	README.docker-container-deployment.md README.architecture.md	README.docker-container-deployment.md README.terraform-deployment.md README.cicd.md	.github/workflows/ci.yml	high	ci-cd github-actions workflow parallel build deploy ghcr container-registry

GitHub Actions Workflow Architecture

Purpose

This document describes the Sirius CI/CD pipeline structure, explaining how GitHub Actions workflows orchestrate parallel container builds, validation, testing, and deployment. It's designed for developers maintaining or modifying the CI/CD pipeline and for those troubleshooting build issues.

When to Use

• Modifying CI/CD pipeline: Making changes to build, test, or deployment processes
• Troubleshooting build failures: Understanding job dependencies and execution order
• Adding new services: Integrating new containers into the build pipeline
• Optimizing build times: Identifying parallelization opportunities
• Debugging deployment issues: Understanding how code reaches production

How to Use

Quick Reference

# Check workflow status
gh run list --workflow=ci.yml --limit 5

# Watch current run
gh run watch

# View workflow logs
gh run view <run-id> --log

# Manually trigger workflow
gh workflow run ci.yml --ref main

Understanding Workflow Execution

1. Code Push/PR: Triggers the workflow automatically
2. Change Detection: Determines which services need rebuilding
3. Parallel Builds: UI, API, and Engine build concurrently
4. Integration Tests: Validates built images work together
5. Deployment Dispatch: Triggers downstream deployments

What It Is

Workflow Overview

The Sirius CI/CD pipeline is implemented in .github/workflows/ci.yml and follows a parallel build architecture that significantly reduces build times compared to sequential builds.

Other workflows (not shown in the diagram below):

• .github/workflows/publish-release-image-tags.yml — manual retag of all six GHCR images from a source tag (e.g. latest) to a release tag (e.g. v1.0.0).
• .github/workflows/verify-ghcr-release-tag.yml — on each published GitHub Release, on a weekly schedule, and via workflow_dispatch, runs scripts/verify-ghcr-public-access.sh so the release tag cannot drift from what anonymous operators can pull (see OPERATIONS.md GHCR checklist).

Key characteristics:

• Parallel execution: UI, API, and Engine containers build simultaneously
• Smart change detection: Only rebuilds services with code changes
• Prebuild validation: Runs lints/tests before expensive Docker builds
• Multi-arch support: Builds for both amd64 and arm64 architectures
• Container registry integration: Pushes to GitHub Container Registry (GHCR)

Architecture Diagram

┌─────────────────┐
│ detect-changes  │ ← Determines what needs rebuilding
└────────┬────────┘
         │
         ├──────────┬──────────┬──────────┐
         ▼          ▼          ▼          ▼
    ┌────────┐ ┌────────┐ ┌──────────┐   │
    │build-ui│ │build-api│ │build-engine│ │  ← Run in parallel
    └───┬────┘ └───┬────┘ └─────┬────┘   │
        │          │            │         │
        └──────────┴────────────┴─────────┘
                   │
                   ▼
              ┌────────┐
              │  test  │ ← Integration testing
              └───┬────┘
                  │
        ┬─────────┴─────────┬
        ▼                   ▼
┌───────────────┐   ┌──────────────┐
│dispatch-demo- │   │dispatch-demo-│
│   deployment  │   │    canary    │
└───────────────┘   └──────────────┘

Workflow Jobs

1. detect-changes

Purpose: Analyzes commits to determine which services need rebuilding.

Triggers on:

• Direct pushes to main or sirius-demo branches
• Pull requests to main
• Repository dispatch events (submodule updates)

Outputs:

• sirius_ui_changes: Boolean indicating UI code changes
• sirius_api_changes: Boolean indicating API code changes
• sirius_engine_changes: Boolean indicating Engine code changes
• submodule_changes: Boolean indicating submodule updates

Change detection logic:

# UI changes: Any file in sirius-ui/
# API changes: Any file in sirius-api/
# Engine changes: Any file in sirius-engine/ or rabbitmq/
# Global changes: Dockerfile, docker-compose, .github/ → rebuild all

2. build-ui (Parallel)

Purpose: Validates and builds the Next.js UI container.

Runs when: detect-changes.outputs.sirius_ui_changes == 'true'

Steps:

1. Generate metadata: Determines image tag (latest, beta, pr-*)
2. Validate UI code: Runs npm ci && npm run lint
3. Set up Docker Buildx: Configures multi-arch builds
4. Log in to GHCR: Authenticates with container registry
5. Build and push: Creates and pushes amd64/arm64 images

Output: image_tag (e.g., latest, beta, pr-123)

Build platforms: linux/amd64, linux/arm64

Validation timing: ~2-3 minutes (lint before Docker build)

3. build-api (Parallel)

Purpose: Validates and builds the Go API container.

Runs when: detect-changes.outputs.sirius_api_changes == 'true'

Steps:

1. Generate metadata: Determines image tag and submodule SHAs
2. Set up Go: Installs Go 1.24
3. Validate API code: Runs go mod download && go test ./...
4. Set up Docker Buildx: Configures multi-arch builds
5. Log in to GHCR: Authenticates with container registry
6. Build and push: Creates and pushes amd64/arm64 images with submodule refs

Output: image_tag

Build args: GO_API_COMMIT_SHA (for go-api submodule)

Validation timing: ~1-2 minutes (tests before Docker build)

4. build-engine (Parallel)

Purpose: Validates and builds the Go Engine container.

Runs when: detect-changes.outputs.sirius_engine_changes == 'true'

Steps:

1. Generate metadata: Determines image tag and all submodule SHAs
2. Set up Go: Installs Go 1.24
3. Validate Engine code: Runs go mod download && go test ./...
4. Set up Docker Buildx: Configures multi-arch builds
5. Log in to GHCR: Authenticates with container registry
6. Build and push: Creates and pushes amd64/arm64 images with all submodule refs

Output: image_tag

Build args:

• GO_API_COMMIT_SHA
• APP_SCANNER_COMMIT_SHA
• APP_TERMINAL_COMMIT_SHA
• SIRIUS_NSE_COMMIT_SHA
• APP_AGENT_COMMIT_SHA

Validation timing: ~1-2 minutes (tests before Docker build)

5. test

Purpose: Validates that built containers work together in an integrated environment.

Runs when: At least one build job succeeds

Dependencies: [detect-changes, build-ui, build-api, build-engine]

Steps:

1. Determine image tag: Uses output from whichever build job ran
2. Create test environment: Generates docker-compose.test.yml with fresh images
3. Start infrastructure: Launches Postgres, RabbitMQ, Valkey
4. Start application services: Launches only the services that were built
5. Run smoke tests: Validates services are running and responsive
6. Cleanup: Tears down test environment

Test configuration:

• Uses in-memory Postgres (tmpfs) for speed
• Isolated test database (sirius_test)
• Debug-level logging for troubleshooting

6. dispatch-demo-deployment

Purpose: Triggers deployment to the demo environment when sirius-demo branch updates.

Runs when: Push to sirius-demo branch after successful build + test

Dependencies: [detect-changes, build-ui, build-api, build-engine, test]

Sends to: SiriusScan/sirius-demo repository with event type sirius-demo-updated

Payload includes:

• Source repo/branch/SHA
• Triggering actor
• Commit message

7. dispatch-demo-canary

Purpose: Triggers demo rebuild on every main branch push as a deployment canary.

Runs when: Push to main branch after successful build + test

Dependencies: [detect-changes, build-ui, build-api, build-engine, test]

Sends to: SiriusScan/sirius-demo repository with event type sirius-main-updated

Purpose of canary: Catches bad commits to main by immediately deploying to demo environment

Image Tagging Strategy

Tag Types

latest:

• Pushed on every main branch commit
• Also tagged as beta simultaneously
• Used by default in docker-compose.yaml

beta:

• Alias for latest (same image)
• Explicitly labeled for beta testing

pr-{number}:

• Unique tag for pull request builds
• Enables testing PRs in isolation

dev:

• Fallback for other branches/events

Tag Determination

# Pull request
TAG="pr-123"

# Push to main or repository_dispatch
TAG="latest"
also_tag_beta="true"

# Other events
TAG="dev"

Prebuild Validation

Each build job runs validation before Docker builds to fail fast and save time.

UI Validation

cd sirius-ui
npm ci                    # Install dependencies
npm run lint              # ESLint validation
# Docker build only if lint succeeds

Typical duration: 2-3 minutes
Catches: Import errors, syntax issues, unused variables

API Validation

cd sirius-api
go mod download           # Download dependencies
go test ./... -v          # Run tests
# Docker build only if tests pass

Typical duration: 1-2 minutes
Catches: Compilation errors, failing tests, import issues

Engine Validation

cd sirius-engine
go mod download           # Download dependencies
go test ./... -v          # Run tests
# Docker build only if tests pass

Typical duration: 1-2 minutes
Catches: Compilation errors, failing tests, integration issues

Timing Expectations

Before Parallelization (Sequential Builds)

Phase	Duration
Change detection	~30s
Build UI	~15-20 min
Build API	~20-25 min
Build Engine	~30-40 min
Test	~5 min
Total	~70-90 min

After Parallelization (Current)

Phase	Duration
Change detection	~30s
Prebuild validation (all parallel)	~2-3 min
Build UI, API, Engine (all parallel)	~30-40 min (longest wins)
Test	~5 min
Total	~40-50 min

Time savings: ~40-60% reduction (30-40 minutes faster)

Authentication & Secrets

Required Secrets

GHCR_PUSH_USER: GitHub username that generated the PAT
GHCR_PUSH_TOKEN: GitHub Personal Access Token with scopes:

• write:packages (push images)
• read:packages (pull existing layers)
• delete:packages (cleanup, optional)
• repo (access repository context)

GITHUB_TOKEN: Automatically provided by GitHub Actions

• Used for repository dispatch events
• Has limited package permissions (read-only)

Setting Up Secrets

# Repository secrets (Sirius repo)
Settings → Security → Secrets and variables → Actions

# Add GHCR_PUSH_USER (your GitHub username)
Name: GHCR_PUSH_USER
Value: your-username

# Add GHCR_PUSH_TOKEN (PAT with package scopes)
Name: GHCR_PUSH_TOKEN
Value: ghp_xxxxxxxxxxxx

Troubleshooting

Build Fails with "denied: installation not allowed"

Cause: Default GITHUB_TOKEN doesn't have package creation permission.

Fix: Use PAT-based authentication (already implemented with GHCR_PUSH_USER/GHCR_PUSH_TOKEN).

Build Fails with "403 Forbidden" during push

Cause: PAT missing required scopes or package visibility is restricted.

Fix:

1. Verify PAT has write:packages and read:packages scopes
2. Check package is public: https://github.com/orgs/SiriusScan/packages/container/sirius-{ui,api,engine}/settings
3. Set visibility to Public (one-time change)

Test job fails to pull images

Cause: Image tag mismatch or build job didn't complete.

Fix:

1. Check build job outputs: needs.build-{ui,api,engine}.outputs.image_tag
2. Verify images exist in GHCR: https://github.com/orgs/SiriusScan/packages
3. Check GHCR authentication in test job

Dispatch jobs don't trigger downstream

Cause: Missing GITHUB_TOKEN permission or incorrect event type.

Fix:

1. Verify GITHUB_TOKEN has workflow permissions
2. Check downstream repo workflow listens for correct event type
3. Confirm payload format matches expectations

Builds take longer than expected

Expected timings (parallel mode):

• Prebuild validation: 2-3 min
• Docker builds: 30-40 min (longest service)
• Integration test: 5 min

Investigate if:

• Build cache not working (check cache-from: type=gha)
• Network issues downloading dependencies
• Jobs running sequentially instead of parallel

Best Practices

When Modifying Workflows

1. Test in PR first: Use pr-* tags to test changes without affecting latest
2. Preserve outputs: Build jobs must output image_tag for downstream jobs
3. Use always(): Downstream jobs should use always() with result checks
4. Validate locally: Use act (GitHub Actions local runner) when possible
5. Check syntax: Run actionlint before committing

Adding New Services

1. Create build job: Copy pattern from build-ui/api/engine
2. Add change detection: Update detect-changes job logic
3. Update test job: Add service to docker-compose.test.yml
4. Update dispatch dependencies: Include new job in needs array

Optimizing Build Times

Already implemented:

• Parallel builds (saves 40-60%)
• Prebuild validation (fails fast)
• Docker layer caching (cache-from: type=gha)
• Multi-arch builds in one step

Future opportunities:

• Cache Go dependencies between runs
• Use npm ci --prefer-offline for UI builds
• Split test job into parallel service-specific tests

Related Documentation

• README.docker-container-deployment.md: Container registry deployment guide
• README.terraform-deployment.md: Terraform-based infrastructure deployment
• README.cicd.md: CI/CD architecture overview
• README.docker-architecture.md: Docker multi-stage builds and architecture

Additional Resources

• GitHub Actions Docs: https://docs.github.com/en/actions
• Docker Buildx: https://docs.docker.com/buildx/
• GHCR Documentation: https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry