chore: import upstream snapshot with attribution
Check engine pin consistency / Dockerfile / CI pin consistency (push) Successful in 8s
Sirius CI/CD Pipeline / Detect Changes (push) Successful in 23s
Validate Docker Configuration / Validate Docker Compose Configuration (push) Successful in 47s
Sirius CI/CD Pipeline / Build API (${{ matrix.platform }}) (push) Has been cancelled
Sirius CI/CD Pipeline / Build UI (${{ matrix.platform }}) (push) Has been cancelled
Sirius CI/CD Pipeline / Merge Engine Manifest (push) Has been cancelled
Sirius CI/CD Pipeline / Merge API Manifest (push) Has been cancelled
Sirius CI/CD Pipeline / Merge UI Manifest (push) Has been cancelled
Sirius CI/CD Pipeline / Build Engine (${{ matrix.platform }}) (push) Has been cancelled
Sirius CI/CD Pipeline / Build Infra (${{ matrix.service }}, ${{ matrix.platform }}) (push) Has been cancelled
Sirius CI/CD Pipeline / Merge Infra Manifest (sirius-postgres) (push) Has been cancelled
Sirius CI/CD Pipeline / Merge Infra Manifest (sirius-rabbitmq) (push) Has been cancelled
Sirius CI/CD Pipeline / Merge Infra Manifest (sirius-valkey) (push) Has been cancelled
Sirius CI/CD Pipeline / Integration Test (push) Has been cancelled
Sirius CI/CD Pipeline / Public Stack Contract (push) Has been cancelled
Sirius CI/CD Pipeline / Dispatch Demo Deployment (sirius-demo branch) (push) Has been cancelled
Sirius CI/CD Pipeline / Dispatch Demo Canary (main branch) (push) Has been cancelled
Sirius CI/CD Pipeline / Guard Registry Namespace (push) Has been cancelled
Check engine pin consistency / Dockerfile / CI pin consistency (push) Successful in 8s
Sirius CI/CD Pipeline / Detect Changes (push) Successful in 23s
Validate Docker Configuration / Validate Docker Compose Configuration (push) Successful in 47s
Sirius CI/CD Pipeline / Build API (${{ matrix.platform }}) (push) Has been cancelled
Sirius CI/CD Pipeline / Build UI (${{ matrix.platform }}) (push) Has been cancelled
Sirius CI/CD Pipeline / Merge Engine Manifest (push) Has been cancelled
Sirius CI/CD Pipeline / Merge API Manifest (push) Has been cancelled
Sirius CI/CD Pipeline / Merge UI Manifest (push) Has been cancelled
Sirius CI/CD Pipeline / Build Engine (${{ matrix.platform }}) (push) Has been cancelled
Sirius CI/CD Pipeline / Build Infra (${{ matrix.service }}, ${{ matrix.platform }}) (push) Has been cancelled
Sirius CI/CD Pipeline / Merge Infra Manifest (sirius-postgres) (push) Has been cancelled
Sirius CI/CD Pipeline / Merge Infra Manifest (sirius-rabbitmq) (push) Has been cancelled
Sirius CI/CD Pipeline / Merge Infra Manifest (sirius-valkey) (push) Has been cancelled
Sirius CI/CD Pipeline / Integration Test (push) Has been cancelled
Sirius CI/CD Pipeline / Public Stack Contract (push) Has been cancelled
Sirius CI/CD Pipeline / Dispatch Demo Deployment (sirius-demo branch) (push) Has been cancelled
Sirius CI/CD Pipeline / Dispatch Demo Canary (main branch) (push) Has been cancelled
Sirius CI/CD Pipeline / Guard Registry Namespace (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,428 @@
|
||||
---
|
||||
title: "Docker Container Deployment Guide"
|
||||
description: "Complete guide for deploying Sirius using prebuilt container images from GitHub Container Registry"
|
||||
template: "TEMPLATE.guide"
|
||||
version: "1.0.0"
|
||||
last_updated: "2026-04-08"
|
||||
author: "Development Team"
|
||||
tags: ["docker", "deployment", "containers", "ghcr", "registry", "production"]
|
||||
categories: ["deployment", "infrastructure"]
|
||||
difficulty: "beginner"
|
||||
prerequisites: ["docker", "docker-compose"]
|
||||
related_docs:
|
||||
- "README.terraform-deployment.md"
|
||||
- "README.development.md"
|
||||
- "README.docker-architecture.md"
|
||||
dependencies: ["docker-compose.yaml"]
|
||||
llm_context: "high"
|
||||
search_keywords:
|
||||
[
|
||||
"docker",
|
||||
"deployment",
|
||||
"containers",
|
||||
"ghcr",
|
||||
"registry",
|
||||
"production",
|
||||
"images",
|
||||
"compose",
|
||||
]
|
||||
---
|
||||
|
||||
# Docker Container Deployment Guide
|
||||
|
||||
## Purpose
|
||||
|
||||
This guide explains how to deploy Sirius using prebuilt container images from GitHub Container Registry (GHCR). This approach provides faster deployments (5-8 minutes vs 20-25 minutes) by eliminating on-instance Docker builds. The guide covers production deployments, image versioning, and fallback strategies.
|
||||
|
||||
## When to Use
|
||||
|
||||
- **Production deployments** - Deploying Sirius to production environments
|
||||
- **Demo environments** - Setting up demo instances quickly
|
||||
- **Staging environments** - Creating isolated testing environments
|
||||
- **Quick deployments** - When you need fast deployment without building from source
|
||||
- **CI/CD pipelines** - Automated deployments using prebuilt images
|
||||
|
||||
**Avoid when**:
|
||||
|
||||
- **Local development** - Use `docker-compose.dev.yaml` for local builds with hot reloading
|
||||
- **Custom modifications** - When you need to modify source code before deployment
|
||||
- **Offline environments** - When registry access is unavailable
|
||||
|
||||
## How to Use
|
||||
|
||||
### Quick Start
|
||||
|
||||
```bash
|
||||
# Clone repository
|
||||
git clone https://github.com/SiriusScan/Sirius.git
|
||||
cd Sirius
|
||||
|
||||
# Generate startup config and secrets
|
||||
docker compose -f docker-compose.installer.yaml run --rm sirius-installer
|
||||
|
||||
# Deploy with prebuilt images (default)
|
||||
docker compose up -d
|
||||
|
||||
# Or specify a version tag
|
||||
IMAGE_TAG=v0.4.1 docker compose up -d
|
||||
|
||||
# Optional: verify the public GHCR contract before pulling
|
||||
bash scripts/verify-ghcr-public-access.sh "${IMAGE_TAG:-latest}"
|
||||
```
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- **Docker** >= 20.10.0
|
||||
- **Docker Compose** >= 2.0.0
|
||||
- **Internet access** to GitHub Container Registry (ghcr.io)
|
||||
- **Git** (for cloning repository)
|
||||
|
||||
## Container Registry Integration
|
||||
|
||||
### GitHub Container Registry (GHCR)
|
||||
|
||||
Sirius images are automatically built and pushed to GitHub Container Registry on every push to the main branch. Images are available at:
|
||||
|
||||
- **UI**: `ghcr.io/siriusscan/sirius-ui:{tag}`
|
||||
- **API**: `ghcr.io/siriusscan/sirius-api:{tag}`
|
||||
- **Engine**: `ghcr.io/siriusscan/sirius-engine:{tag}`
|
||||
- **Postgres**: `ghcr.io/siriusscan/sirius-postgres:{tag}`
|
||||
- **RabbitMQ**: `ghcr.io/siriusscan/sirius-rabbitmq:{tag}`
|
||||
- **Valkey**: `ghcr.io/siriusscan/sirius-valkey:{tag}`
|
||||
|
||||
`docker-compose.yaml` is the public-image deployment path for Sirius. If an unauthenticated `docker pull` against one of the image refs above returns `unauthorized`, the GHCR public-visibility contract is broken and operators should stop before rollout.
|
||||
|
||||
### Image Tagging Strategy
|
||||
|
||||
Images are tagged with the following strategy:
|
||||
|
||||
| Tag | Description | When Created |
|
||||
| -------- | ------------------------ | ----------------------- |
|
||||
| `latest` | Latest main branch build | On every push to main |
|
||||
| `beta` | Beta release candidate | On main branch pushes |
|
||||
| `v0.4.1` | Version-specific tag | After the `Publish Release Image Tags` workflow succeeds |
|
||||
| `dev` | Development builds | On other branch pushes |
|
||||
| `pr-123` | Pull request builds | On PR creation/updates |
|
||||
|
||||
### Image Availability
|
||||
|
||||
- **Public images**: No authentication required
|
||||
- **Multi-architecture**: Images support `linux/amd64` and `linux/arm64`
|
||||
- **Automatic updates**: Latest images are built automatically by CI/CD
|
||||
- **Release contract**: A release tag is only valid for operators after the release-tag workflow publishes it, the anonymous GHCR verification step passes, and the public Compose smoke test succeeds. CI validates **`latest`** on every main push (`public-stack-contract` in `ci.yml`); **semver** tags are additionally checked when a GitHub Release is published and on a weekly schedule ([`verify-ghcr-release-tag.yml`](../../../.github/workflows/verify-ghcr-release-tag.yml)).
|
||||
|
||||
## Docker Compose Configuration
|
||||
|
||||
### Base Configuration (Production)
|
||||
|
||||
The default `docker-compose.yaml` uses prebuilt images:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
sirius-ui:
|
||||
image: ghcr.io/siriusscan/sirius-ui:${IMAGE_TAG:-latest}
|
||||
pull_policy: always
|
||||
# ... other configuration
|
||||
|
||||
sirius-api:
|
||||
image: ghcr.io/siriusscan/sirius-api:${IMAGE_TAG:-latest}
|
||||
pull_policy: always
|
||||
# ... other configuration
|
||||
|
||||
sirius-engine:
|
||||
image: ghcr.io/siriusscan/sirius-engine:${IMAGE_TAG:-latest}
|
||||
pull_policy: always
|
||||
# ... other configuration
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Control which images to use with the `IMAGE_TAG` environment variable:
|
||||
|
||||
```bash
|
||||
# Use latest images (default)
|
||||
docker compose up -d
|
||||
|
||||
# Use specific version
|
||||
IMAGE_TAG=v0.4.1 docker compose up -d
|
||||
|
||||
# Use beta release
|
||||
IMAGE_TAG=beta docker compose up -d
|
||||
|
||||
# Validate that the selected tag is publicly readable before rollout
|
||||
bash scripts/verify-ghcr-public-access.sh "${IMAGE_TAG:-latest}"
|
||||
```
|
||||
|
||||
`.env.production.example` leaves `IMAGE_TAG` blank so fresh installer runs inherit the Compose default (`latest`). Pin a release tag only after the publish workflow has validated that all six Sirius images exist for that tag.
|
||||
|
||||
### Development Override
|
||||
|
||||
For local development with source code changes, use the development override:
|
||||
|
||||
```bash
|
||||
# Build locally with hot reloading
|
||||
docker compose -f docker-compose.yaml -f docker-compose.dev.yaml up -d --build
|
||||
```
|
||||
|
||||
The `docker-compose.dev.yaml` file overrides the registry images with local builds, enabling:
|
||||
|
||||
- Hot reloading
|
||||
- Volume mounts for source code
|
||||
- Development-specific environment variables
|
||||
- Debug logging
|
||||
|
||||
## Deployment Workflow
|
||||
|
||||
### Standard Production Deployment
|
||||
|
||||
1. **Clone repository**:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/SiriusScan/Sirius.git
|
||||
cd Sirius
|
||||
```
|
||||
|
||||
2. **Configure environment**:
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.installer.yaml run --rm sirius-installer
|
||||
# Optional: pass explicit values
|
||||
# docker compose -f docker-compose.installer.yaml run --rm sirius-installer --non-interactive --no-print-secrets
|
||||
```
|
||||
|
||||
3. **Deploy services**:
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
4. **Verify deployment**:
|
||||
|
||||
```bash
|
||||
docker compose ps
|
||||
docker compose logs -f
|
||||
```
|
||||
|
||||
5. **Check health**:
|
||||
```bash
|
||||
curl http://localhost:9001/health # API
|
||||
curl http://localhost:3000/api/health # UI
|
||||
```
|
||||
|
||||
### Maintainer Validation Path
|
||||
|
||||
Use the shared validation script to exercise the same public Compose path that operators use:
|
||||
|
||||
```bash
|
||||
# Validate the default public stack (latest)
|
||||
bash scripts/validate-public-compose-path.sh latest
|
||||
|
||||
# Validate a published release tag
|
||||
bash scripts/validate-public-compose-path.sh v0.4.1
|
||||
```
|
||||
|
||||
### Version-Specific Deployment
|
||||
|
||||
Deploy a specific version:
|
||||
|
||||
```bash
|
||||
# Set version tag
|
||||
export IMAGE_TAG=v0.4.1
|
||||
|
||||
# Confirm the compose-rendered GHCR images are public and readable
|
||||
bash scripts/verify-ghcr-public-access.sh "$IMAGE_TAG"
|
||||
|
||||
# Pull and start services
|
||||
docker compose pull
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### Updating Deployment
|
||||
|
||||
To update to the latest version:
|
||||
|
||||
```bash
|
||||
# Pull latest images
|
||||
docker compose pull
|
||||
|
||||
# Restart services with new images
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
## Fallback Strategy
|
||||
|
||||
### When Registry is Unavailable
|
||||
|
||||
If GitHub Container Registry is unavailable or images fail to pull, you can fall back to local builds:
|
||||
|
||||
1. **Use the committed source-build override**:
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.yaml -f docker-compose.build.yaml up -d --build
|
||||
```
|
||||
|
||||
**Note**: Local builds take significantly longer (20-25 minutes vs 5-8 minutes) and require more system resources.
|
||||
|
||||
## Secrets Hardening Overlays
|
||||
|
||||
For hardened deployments, Sirius includes optional overlay manifests:
|
||||
|
||||
- `docker-compose.secrets.yaml` for Compose secrets mounted at `/run/secrets/*`
|
||||
- `docker-stack.swarm.yaml` for Swarm stack deployments
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
mkdir -p secrets
|
||||
printf '%s' "your-postgres-password" > secrets/postgres_password.txt
|
||||
printf '%s' "your-service-key" > secrets/sirius_api_key.txt
|
||||
chmod 644 secrets/sirius_api_key.txt
|
||||
printf '%s' "your-nextauth-secret" > secrets/nextauth_secret.txt
|
||||
printf '%s' "your-admin-password" > secrets/initial_admin_password.txt
|
||||
|
||||
docker compose -f docker-compose.yaml -f docker-compose.secrets.yaml up -d
|
||||
```
|
||||
|
||||
`sirius_api_key.txt` should stay **world-readable** (`644`) so bind-mounted `/run/secrets/sirius_api_key` is readable inside **sirius-api** / **sirius-ui** / **sirius-engine** (non-root UIDs).
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Images Not Pulling
|
||||
|
||||
**Problem**: `docker compose pull` fails with authentication or network errors.
|
||||
|
||||
**Solutions**:
|
||||
|
||||
- Verify internet connectivity: `curl -I https://ghcr.io`
|
||||
- Check image exists: Visit `https://github.com/SiriusScan/Sirius/pkgs/container/sirius-ui`
|
||||
- Try pulling manually: `docker pull ghcr.io/siriusscan/sirius-ui:latest`
|
||||
- Run the contract check: `bash scripts/verify-ghcr-public-access.sh "${IMAGE_TAG:-latest}"`
|
||||
- If the script reports `Anonymous access denied`, the package is not publicly readable and the GHCR visibility workflow or token scope needs attention
|
||||
- If the script reports `Manifest missing`, the requested tag was not published and you should verify the release-tag workflow completed successfully
|
||||
- If the public Compose smoke test fails after pull succeeds, run `bash scripts/validate-public-compose-path.sh "${IMAGE_TAG:-latest}"` to reproduce the operator path and inspect the runtime contract failure
|
||||
- Use fallback build strategy (see above)
|
||||
|
||||
### Wrong Version Deployed
|
||||
|
||||
**Problem**: Services are running an unexpected version.
|
||||
|
||||
**Solutions**:
|
||||
|
||||
- Check current IMAGE_TAG: `echo $IMAGE_TAG`
|
||||
- Verify image tags: `docker compose images`
|
||||
- Pull specific version: `IMAGE_TAG=v0.4.1 docker compose pull`
|
||||
- Restart services: `docker compose up -d`
|
||||
|
||||
### Services Not Starting
|
||||
|
||||
**Problem**: Containers fail to start after pulling images.
|
||||
|
||||
**Solutions**:
|
||||
|
||||
- Check logs: `docker compose logs`
|
||||
- Verify environment variables: `docker compose config`
|
||||
- Check image compatibility: Ensure architecture matches (amd64/arm64)
|
||||
- Verify dependencies: Ensure PostgreSQL, RabbitMQ, Valkey are running
|
||||
|
||||
### Performance Issues
|
||||
|
||||
**Problem**: Deployment is slower than expected.
|
||||
|
||||
**Solutions**:
|
||||
|
||||
- Check network speed: `docker pull` should be fast on good connections
|
||||
- Verify image sizes: Large images take longer to pull
|
||||
- Use specific version tags instead of `latest` for faster pulls
|
||||
- Consider using image caching strategies
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Security
|
||||
|
||||
- **Use specific version tags** in production (e.g., `v0.4.1`) instead of `latest`
|
||||
- **Regularly update images** to get security patches
|
||||
- **Scan images** for vulnerabilities using Docker security scanning
|
||||
- **Use private registries** for sensitive deployments (if needed)
|
||||
|
||||
### Version Management
|
||||
|
||||
- **Pin versions** in production environments
|
||||
- **Test updates** in staging before production
|
||||
- **Document versions** deployed in each environment
|
||||
- **Use semantic versioning** for releases
|
||||
|
||||
### Performance
|
||||
|
||||
- **Pre-pull images** before deployment to reduce startup time
|
||||
- **Use image caching** to avoid redundant pulls
|
||||
- **Monitor image sizes** and optimize Dockerfiles if needed
|
||||
- **Use multi-stage builds** to reduce final image sizes
|
||||
|
||||
### Monitoring
|
||||
|
||||
- **Track deployment times** to measure improvements
|
||||
- **Monitor registry availability** and fallback usage
|
||||
- **Log image versions** deployed for audit trails
|
||||
- **Alert on deployment failures** for quick response
|
||||
|
||||
## Comparison: Registry vs Local Builds
|
||||
|
||||
| Aspect | Registry Images | Local Builds |
|
||||
| --------------------- | ----------------- | -------------------- |
|
||||
| **Deployment Time** | 5-8 minutes | 20-25 minutes |
|
||||
| **Resource Usage** | Low (pull only) | High (compilation) |
|
||||
| **Network Required** | Yes (for pull) | No (after clone) |
|
||||
| **Customization** | Limited | Full |
|
||||
| **CI/CD Integration** | Automatic | Manual |
|
||||
| **Best For** | Production, demos | Development, offline |
|
||||
|
||||
## Integration with CI/CD
|
||||
|
||||
### GitHub Actions
|
||||
|
||||
Images are automatically built and pushed by GitHub Actions on:
|
||||
|
||||
- Push to `main` branch → `latest` and `beta` tags
|
||||
- Manual `Publish Release Image Tags` run → version-specific tag (e.g., `v0.4.1`)
|
||||
- Pull requests → `pr-{number}` tags
|
||||
|
||||
### Deployment Automation
|
||||
|
||||
Example GitHub Actions workflow for deployment:
|
||||
|
||||
```yaml
|
||||
name: Deploy Sirius
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
|
||||
jobs:
|
||||
deploy:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Deploy with registry images
|
||||
run: |
|
||||
docker compose pull
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Terraform Deployment Guide](README.terraform-deployment.md) - AWS deployment using Terraform
|
||||
- [Development Guide](../README.development.md) - Local development setup
|
||||
- [Docker Architecture Guide](../architecture/README.docker-architecture.md) - Container architecture details
|
||||
|
||||
## Support
|
||||
|
||||
For issues with container deployment:
|
||||
|
||||
1. Check the troubleshooting section above
|
||||
2. Review Docker logs: `docker compose logs`
|
||||
3. Verify image availability on GitHub Container Registry
|
||||
4. Create an issue in the Sirius repository
|
||||
|
||||
---
|
||||
|
||||
_This guide follows the Sirius Documentation Standard. For questions about documentation structure, see [ABOUT.documentation.md](../ABOUT.documentation.md)._
|
||||
@@ -0,0 +1,464 @@
|
||||
---
|
||||
title: "GitHub Actions Workflow Architecture"
|
||||
description: "Comprehensive guide to Sirius CI/CD pipeline structure, parallel build jobs, and deployment workflows"
|
||||
template: "TEMPLATE.documentation-standard"
|
||||
version: "1.0.0"
|
||||
last_updated: "2025-11-14"
|
||||
author: "Sirius Team"
|
||||
tags: ["ci-cd", "github-actions", "docker", "ghcr", "parallel-builds"]
|
||||
categories: ["deployment", "automation"]
|
||||
difficulty: "intermediate"
|
||||
prerequisites: ["README.docker-container-deployment.md", "README.architecture.md"]
|
||||
related_docs:
|
||||
- "README.docker-container-deployment.md"
|
||||
- "README.terraform-deployment.md"
|
||||
- "README.cicd.md"
|
||||
dependencies:
|
||||
- ".github/workflows/ci.yml"
|
||||
llm_context: "high"
|
||||
search_keywords: ["ci-cd", "github-actions", "workflow", "parallel", "build", "deploy", "ghcr", "container-registry"]
|
||||
---
|
||||
|
||||
# GitHub Actions Workflow Architecture
|
||||
|
||||
## Purpose
|
||||
|
||||
This document describes the Sirius CI/CD pipeline structure, explaining how GitHub Actions workflows orchestrate parallel container builds, validation, testing, and deployment. It's designed for developers maintaining or modifying the CI/CD pipeline and for those troubleshooting build issues.
|
||||
|
||||
## When to Use
|
||||
|
||||
- **Modifying CI/CD pipeline**: Making changes to build, test, or deployment processes
|
||||
- **Troubleshooting build failures**: Understanding job dependencies and execution order
|
||||
- **Adding new services**: Integrating new containers into the build pipeline
|
||||
- **Optimizing build times**: Identifying parallelization opportunities
|
||||
- **Debugging deployment issues**: Understanding how code reaches production
|
||||
|
||||
## How to Use
|
||||
|
||||
### Quick Reference
|
||||
|
||||
```bash
|
||||
# Check workflow status
|
||||
gh run list --workflow=ci.yml --limit 5
|
||||
|
||||
# Watch current run
|
||||
gh run watch
|
||||
|
||||
# View workflow logs
|
||||
gh run view <run-id> --log
|
||||
|
||||
# Manually trigger workflow
|
||||
gh workflow run ci.yml --ref main
|
||||
```
|
||||
|
||||
### Understanding Workflow Execution
|
||||
|
||||
1. **Code Push/PR**: Triggers the workflow automatically
|
||||
2. **Change Detection**: Determines which services need rebuilding
|
||||
3. **Parallel Builds**: UI, API, and Engine build concurrently
|
||||
4. **Integration Tests**: Validates built images work together
|
||||
5. **Deployment Dispatch**: Triggers downstream deployments
|
||||
|
||||
## What It Is
|
||||
|
||||
### Workflow Overview
|
||||
|
||||
The Sirius CI/CD pipeline is implemented in `.github/workflows/ci.yml` and follows a **parallel build architecture** that significantly reduces build times compared to sequential builds.
|
||||
|
||||
**Other workflows** (not shown in the diagram below):
|
||||
|
||||
- [`.github/workflows/publish-release-image-tags.yml`](../../../.github/workflows/publish-release-image-tags.yml) — manual retag of all six GHCR images from a source tag (e.g. `latest`) to a release tag (e.g. `v1.0.0`).
|
||||
- [`.github/workflows/verify-ghcr-release-tag.yml`](../../../.github/workflows/verify-ghcr-release-tag.yml) — on each **published** GitHub Release, on a **weekly** schedule, and via **workflow_dispatch**, runs `scripts/verify-ghcr-public-access.sh` so the release tag cannot drift from what anonymous operators can pull (see [OPERATIONS.md](../../../OPERATIONS.md) GHCR checklist).
|
||||
|
||||
**Key characteristics:**
|
||||
|
||||
- **Parallel execution**: UI, API, and Engine containers build simultaneously
|
||||
- **Smart change detection**: Only rebuilds services with code changes
|
||||
- **Prebuild validation**: Runs lints/tests before expensive Docker builds
|
||||
- **Multi-arch support**: Builds for both amd64 and arm64 architectures
|
||||
- **Container registry integration**: Pushes to GitHub Container Registry (GHCR)
|
||||
|
||||
### Architecture Diagram
|
||||
|
||||
```
|
||||
┌─────────────────┐
|
||||
│ detect-changes │ ← Determines what needs rebuilding
|
||||
└────────┬────────┘
|
||||
│
|
||||
├──────────┬──────────┬──────────┐
|
||||
▼ ▼ ▼ ▼
|
||||
┌────────┐ ┌────────┐ ┌──────────┐ │
|
||||
│build-ui│ │build-api│ │build-engine│ │ ← Run in parallel
|
||||
└───┬────┘ └───┬────┘ └─────┬────┘ │
|
||||
│ │ │ │
|
||||
└──────────┴────────────┴─────────┘
|
||||
│
|
||||
▼
|
||||
┌────────┐
|
||||
│ test │ ← Integration testing
|
||||
└───┬────┘
|
||||
│
|
||||
┬─────────┴─────────┬
|
||||
▼ ▼
|
||||
┌───────────────┐ ┌──────────────┐
|
||||
│dispatch-demo- │ │dispatch-demo-│
|
||||
│ deployment │ │ canary │
|
||||
└───────────────┘ └──────────────┘
|
||||
```
|
||||
|
||||
## Workflow Jobs
|
||||
|
||||
### 1. detect-changes
|
||||
|
||||
**Purpose**: Analyzes commits to determine which services need rebuilding.
|
||||
|
||||
**Triggers on**:
|
||||
- Direct pushes to `main` or `sirius-demo` branches
|
||||
- Pull requests to `main`
|
||||
- Repository dispatch events (submodule updates)
|
||||
|
||||
**Outputs**:
|
||||
- `sirius_ui_changes`: Boolean indicating UI code changes
|
||||
- `sirius_api_changes`: Boolean indicating API code changes
|
||||
- `sirius_engine_changes`: Boolean indicating Engine code changes
|
||||
- `submodule_changes`: Boolean indicating submodule updates
|
||||
|
||||
**Change detection logic**:
|
||||
```yaml
|
||||
# UI changes: Any file in sirius-ui/
|
||||
# API changes: Any file in sirius-api/
|
||||
# Engine changes: Any file in sirius-engine/ or rabbitmq/
|
||||
# Global changes: Dockerfile, docker-compose, .github/ → rebuild all
|
||||
```
|
||||
|
||||
### 2. build-ui (Parallel)
|
||||
|
||||
**Purpose**: Validates and builds the Next.js UI container.
|
||||
|
||||
**Runs when**: `detect-changes.outputs.sirius_ui_changes == 'true'`
|
||||
|
||||
**Steps**:
|
||||
1. **Generate metadata**: Determines image tag (`latest`, `beta`, `pr-*`)
|
||||
2. **Validate UI code**: Runs `npm ci && npm run lint`
|
||||
3. **Set up Docker Buildx**: Configures multi-arch builds
|
||||
4. **Log in to GHCR**: Authenticates with container registry
|
||||
5. **Build and push**: Creates and pushes amd64/arm64 images
|
||||
|
||||
**Output**: `image_tag` (e.g., `latest`, `beta`, `pr-123`)
|
||||
|
||||
**Build platforms**: `linux/amd64`, `linux/arm64`
|
||||
|
||||
**Validation timing**: ~2-3 minutes (lint before Docker build)
|
||||
|
||||
### 3. build-api (Parallel)
|
||||
|
||||
**Purpose**: Validates and builds the Go API container.
|
||||
|
||||
**Runs when**: `detect-changes.outputs.sirius_api_changes == 'true'`
|
||||
|
||||
**Steps**:
|
||||
1. **Generate metadata**: Determines image tag and submodule SHAs
|
||||
2. **Set up Go**: Installs Go 1.24
|
||||
3. **Validate API code**: Runs `go mod download && go test ./...`
|
||||
4. **Set up Docker Buildx**: Configures multi-arch builds
|
||||
5. **Log in to GHCR**: Authenticates with container registry
|
||||
6. **Build and push**: Creates and pushes amd64/arm64 images with submodule refs
|
||||
|
||||
**Output**: `image_tag`
|
||||
|
||||
**Build args**: `GO_API_COMMIT_SHA` (for go-api submodule)
|
||||
|
||||
**Validation timing**: ~1-2 minutes (tests before Docker build)
|
||||
|
||||
### 4. build-engine (Parallel)
|
||||
|
||||
**Purpose**: Validates and builds the Go Engine container.
|
||||
|
||||
**Runs when**: `detect-changes.outputs.sirius_engine_changes == 'true'`
|
||||
|
||||
**Steps**:
|
||||
1. **Generate metadata**: Determines image tag and all submodule SHAs
|
||||
2. **Set up Go**: Installs Go 1.24
|
||||
3. **Validate Engine code**: Runs `go mod download && go test ./...`
|
||||
4. **Set up Docker Buildx**: Configures multi-arch builds
|
||||
5. **Log in to GHCR**: Authenticates with container registry
|
||||
6. **Build and push**: Creates and pushes amd64/arm64 images with all submodule refs
|
||||
|
||||
**Output**: `image_tag`
|
||||
|
||||
**Build args**:
|
||||
- `GO_API_COMMIT_SHA`
|
||||
- `APP_SCANNER_COMMIT_SHA`
|
||||
- `APP_TERMINAL_COMMIT_SHA`
|
||||
- `SIRIUS_NSE_COMMIT_SHA`
|
||||
- `APP_AGENT_COMMIT_SHA`
|
||||
|
||||
**Validation timing**: ~1-2 minutes (tests before Docker build)
|
||||
|
||||
### 5. test
|
||||
|
||||
**Purpose**: Validates that built containers work together in an integrated environment.
|
||||
|
||||
**Runs when**: At least one build job succeeds
|
||||
|
||||
**Dependencies**: `[detect-changes, build-ui, build-api, build-engine]`
|
||||
|
||||
**Steps**:
|
||||
1. **Determine image tag**: Uses output from whichever build job ran
|
||||
2. **Create test environment**: Generates `docker-compose.test.yml` with fresh images
|
||||
3. **Start infrastructure**: Launches Postgres, RabbitMQ, Valkey
|
||||
4. **Start application services**: Launches only the services that were built
|
||||
5. **Run smoke tests**: Validates services are running and responsive
|
||||
6. **Cleanup**: Tears down test environment
|
||||
|
||||
**Test configuration**:
|
||||
- Uses in-memory Postgres (`tmpfs`) for speed
|
||||
- Isolated test database (`sirius_test`)
|
||||
- Debug-level logging for troubleshooting
|
||||
|
||||
### 6. dispatch-demo-deployment
|
||||
|
||||
**Purpose**: Triggers deployment to the demo environment when `sirius-demo` branch updates.
|
||||
|
||||
**Runs when**: Push to `sirius-demo` branch after successful build + test
|
||||
|
||||
**Dependencies**: `[detect-changes, build-ui, build-api, build-engine, test]`
|
||||
|
||||
**Sends to**: `SiriusScan/sirius-demo` repository with event type `sirius-demo-updated`
|
||||
|
||||
**Payload includes**:
|
||||
- Source repo/branch/SHA
|
||||
- Triggering actor
|
||||
- Commit message
|
||||
|
||||
### 7. dispatch-demo-canary
|
||||
|
||||
**Purpose**: Triggers demo rebuild on every `main` branch push as a deployment canary.
|
||||
|
||||
**Runs when**: Push to `main` branch after successful build + test
|
||||
|
||||
**Dependencies**: `[detect-changes, build-ui, build-api, build-engine, test]`
|
||||
|
||||
**Sends to**: `SiriusScan/sirius-demo` repository with event type `sirius-main-updated`
|
||||
|
||||
**Purpose of canary**: Catches bad commits to `main` by immediately deploying to demo environment
|
||||
|
||||
## Image Tagging Strategy
|
||||
|
||||
### Tag Types
|
||||
|
||||
**`latest`**:
|
||||
- Pushed on every `main` branch commit
|
||||
- Also tagged as `beta` simultaneously
|
||||
- Used by default in `docker-compose.yaml`
|
||||
|
||||
**`beta`**:
|
||||
- Alias for `latest` (same image)
|
||||
- Explicitly labeled for beta testing
|
||||
|
||||
**`pr-{number}`**:
|
||||
- Unique tag for pull request builds
|
||||
- Enables testing PRs in isolation
|
||||
|
||||
**`dev`**:
|
||||
- Fallback for other branches/events
|
||||
|
||||
### Tag Determination
|
||||
|
||||
```bash
|
||||
# Pull request
|
||||
TAG="pr-123"
|
||||
|
||||
# Push to main or repository_dispatch
|
||||
TAG="latest"
|
||||
also_tag_beta="true"
|
||||
|
||||
# Other events
|
||||
TAG="dev"
|
||||
```
|
||||
|
||||
## Prebuild Validation
|
||||
|
||||
Each build job runs validation **before** Docker builds to fail fast and save time.
|
||||
|
||||
### UI Validation
|
||||
|
||||
```bash
|
||||
cd sirius-ui
|
||||
npm ci # Install dependencies
|
||||
npm run lint # ESLint validation
|
||||
# Docker build only if lint succeeds
|
||||
```
|
||||
|
||||
**Typical duration**: 2-3 minutes
|
||||
**Catches**: Import errors, syntax issues, unused variables
|
||||
|
||||
### API Validation
|
||||
|
||||
```bash
|
||||
cd sirius-api
|
||||
go mod download # Download dependencies
|
||||
go test ./... -v # Run tests
|
||||
# Docker build only if tests pass
|
||||
```
|
||||
|
||||
**Typical duration**: 1-2 minutes
|
||||
**Catches**: Compilation errors, failing tests, import issues
|
||||
|
||||
### Engine Validation
|
||||
|
||||
```bash
|
||||
cd sirius-engine
|
||||
go mod download # Download dependencies
|
||||
go test ./... -v # Run tests
|
||||
# Docker build only if tests pass
|
||||
```
|
||||
|
||||
**Typical duration**: 1-2 minutes
|
||||
**Catches**: Compilation errors, failing tests, integration issues
|
||||
|
||||
## Timing Expectations
|
||||
|
||||
### Before Parallelization (Sequential Builds)
|
||||
|
||||
| Phase | Duration |
|
||||
|-------|----------|
|
||||
| Change detection | ~30s |
|
||||
| Build UI | ~15-20 min |
|
||||
| Build API | ~20-25 min |
|
||||
| Build Engine | ~30-40 min |
|
||||
| Test | ~5 min |
|
||||
| **Total** | **~70-90 min** |
|
||||
|
||||
### After Parallelization (Current)
|
||||
|
||||
| Phase | Duration |
|
||||
|-------|----------|
|
||||
| Change detection | ~30s |
|
||||
| Prebuild validation (all parallel) | ~2-3 min |
|
||||
| Build UI, API, Engine (all parallel) | ~30-40 min (longest wins) |
|
||||
| Test | ~5 min |
|
||||
| **Total** | **~40-50 min** |
|
||||
|
||||
**Time savings**: ~40-60% reduction (30-40 minutes faster)
|
||||
|
||||
## Authentication & Secrets
|
||||
|
||||
### Required Secrets
|
||||
|
||||
**`GHCR_PUSH_USER`**: GitHub username that generated the PAT
|
||||
**`GHCR_PUSH_TOKEN`**: GitHub Personal Access Token with scopes:
|
||||
- `write:packages` (push images)
|
||||
- `read:packages` (pull existing layers)
|
||||
- `delete:packages` (cleanup, optional)
|
||||
- `repo` (access repository context)
|
||||
|
||||
**`GITHUB_TOKEN`**: Automatically provided by GitHub Actions
|
||||
- Used for repository dispatch events
|
||||
- Has limited package permissions (read-only)
|
||||
|
||||
### Setting Up Secrets
|
||||
|
||||
```bash
|
||||
# Repository secrets (Sirius repo)
|
||||
Settings → Security → Secrets and variables → Actions
|
||||
|
||||
# Add GHCR_PUSH_USER (your GitHub username)
|
||||
Name: GHCR_PUSH_USER
|
||||
Value: your-username
|
||||
|
||||
# Add GHCR_PUSH_TOKEN (PAT with package scopes)
|
||||
Name: GHCR_PUSH_TOKEN
|
||||
Value: ghp_xxxxxxxxxxxx
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Build Fails with "denied: installation not allowed"
|
||||
|
||||
**Cause**: Default `GITHUB_TOKEN` doesn't have package creation permission.
|
||||
|
||||
**Fix**: Use PAT-based authentication (already implemented with `GHCR_PUSH_USER`/`GHCR_PUSH_TOKEN`).
|
||||
|
||||
### Build Fails with "403 Forbidden" during push
|
||||
|
||||
**Cause**: PAT missing required scopes or package visibility is restricted.
|
||||
|
||||
**Fix**:
|
||||
1. Verify PAT has `write:packages` and `read:packages` scopes
|
||||
2. Check package is public: `https://github.com/orgs/SiriusScan/packages/container/sirius-{ui,api,engine}/settings`
|
||||
3. Set visibility to Public (one-time change)
|
||||
|
||||
### Test job fails to pull images
|
||||
|
||||
**Cause**: Image tag mismatch or build job didn't complete.
|
||||
|
||||
**Fix**:
|
||||
1. Check build job outputs: `needs.build-{ui,api,engine}.outputs.image_tag`
|
||||
2. Verify images exist in GHCR: `https://github.com/orgs/SiriusScan/packages`
|
||||
3. Check GHCR authentication in test job
|
||||
|
||||
### Dispatch jobs don't trigger downstream
|
||||
|
||||
**Cause**: Missing `GITHUB_TOKEN` permission or incorrect event type.
|
||||
|
||||
**Fix**:
|
||||
1. Verify `GITHUB_TOKEN` has workflow permissions
|
||||
2. Check downstream repo workflow listens for correct event type
|
||||
3. Confirm payload format matches expectations
|
||||
|
||||
### Builds take longer than expected
|
||||
|
||||
**Expected timings** (parallel mode):
|
||||
- Prebuild validation: 2-3 min
|
||||
- Docker builds: 30-40 min (longest service)
|
||||
- Integration test: 5 min
|
||||
|
||||
**Investigate if**:
|
||||
- Build cache not working (check `cache-from: type=gha`)
|
||||
- Network issues downloading dependencies
|
||||
- Jobs running sequentially instead of parallel
|
||||
|
||||
## Best Practices
|
||||
|
||||
### When Modifying Workflows
|
||||
|
||||
1. **Test in PR first**: Use `pr-*` tags to test changes without affecting `latest`
|
||||
2. **Preserve outputs**: Build jobs must output `image_tag` for downstream jobs
|
||||
3. **Use `always()`**: Downstream jobs should use `always()` with result checks
|
||||
4. **Validate locally**: Use `act` (GitHub Actions local runner) when possible
|
||||
5. **Check syntax**: Run `actionlint` before committing
|
||||
|
||||
### Adding New Services
|
||||
|
||||
1. **Create build job**: Copy pattern from `build-ui/api/engine`
|
||||
2. **Add change detection**: Update `detect-changes` job logic
|
||||
3. **Update test job**: Add service to `docker-compose.test.yml`
|
||||
4. **Update dispatch dependencies**: Include new job in `needs` array
|
||||
|
||||
### Optimizing Build Times
|
||||
|
||||
**Already implemented**:
|
||||
- Parallel builds (saves 40-60%)
|
||||
- Prebuild validation (fails fast)
|
||||
- Docker layer caching (`cache-from: type=gha`)
|
||||
- Multi-arch builds in one step
|
||||
|
||||
**Future opportunities**:
|
||||
- Cache Go dependencies between runs
|
||||
- Use `npm ci --prefer-offline` for UI builds
|
||||
- Split test job into parallel service-specific tests
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- **[README.docker-container-deployment.md](README.docker-container-deployment.md)**: Container registry deployment guide
|
||||
- **[README.terraform-deployment.md](../operations/README.terraform-deployment.md)**: Terraform-based infrastructure deployment
|
||||
- **[README.cicd.md](../architecture/README.cicd.md)**: CI/CD architecture overview
|
||||
- **[README.docker-architecture.md](../architecture/README.docker-architecture.md)**: Docker multi-stage builds and architecture
|
||||
|
||||
## Additional Resources
|
||||
|
||||
- **GitHub Actions Docs**: https://docs.github.com/en/actions
|
||||
- **Docker Buildx**: https://docs.docker.com/buildx/
|
||||
- **GHCR Documentation**: https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry
|
||||
|
||||
Reference in New Issue
Block a user