siriusscan--sirius/documentation/dev/architecture/README.cicd.md

title, description, template, version, last_updated, author, tags, categories, difficulty, prerequisites, related_docs, dependencies, llm_context, search_keywords

title	description	template	version	last_updated	author	tags	categories	difficulty	prerequisites	related_docs	dependencies	llm_context	search_keywords
Sirius CI/CD Pipeline	Comprehensive guide to the Sirius Continuous Integration and Continuous Deployment pipeline, including workflows, testing strategies, and deployment processes.	TEMPLATE.architecture	1.0.0	2025-01-03	Development Team	cicd pipeline github-actions docker testing deployment	architecture operations	intermediate	docker github-actions git	README.docker-architecture.md README.developer-guide.md README.container-testing.md	.github/workflows/ci.yml testing/container-testing/ scripts/switch-env.sh	high	cicd pipeline github actions docker testing continuous integration deployment workflow

Sirius CI/CD Pipeline

Overview

The Sirius CI/CD pipeline provides automated testing, building, and deployment of the multi-service Sirius application. The pipeline is designed to be efficient, reliable, and developer-friendly while maintaining high quality standards.

Pipeline Architecture

Workflow Structure

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Quick Checks  │    │  Full Testing   │    │   Deployment    │
│   (Pre-commit)  │───▶│   (CI/CD)       │───▶│   (Production)  │
└─────────────────┘    └─────────────────┘    └─────────────────┘

Trigger Strategy

Event	Quick Checks	Full Testing	Deployment
Feature Branch Push	✅ Yes	❌ No	❌ No
Pull Request	✅ Yes	✅ Yes	❌ No
Main Branch Push	✅ Yes	✅ Yes	✅ Yes
Hotfix Push	✅ Yes	✅ Yes	✅ Yes

Quick Checks (Pre-commit)

Purpose: Fast validation to catch obvious issues before commit

Duration: ~30 seconds

What's Tested:

• Docker Compose configuration validation
• Documentation linting
• Basic syntax checks
• Code formatting

Commands:

# Pre-commit validation
make build-all          # Validate all Docker Compose configs
make lint-docs-quick    # Quick documentation checks

Full Testing (CI/CD)

Purpose: Comprehensive testing of all changes

Duration: ~5-10 minutes

What's Tested:

• Docker container builds (all services)
• Service health checks
• Integration testing
• Cross-service communication
• Production build validation

Commands:

# Full CI testing
make test-all           # Complete test suite
make validate-all       # Full validation with docs

Environment Strategy

Development Testing

Trigger: Feature branch pushes, local development

Configuration: Uses development Docker Compose setup

Testing Level: Quick validation only

Purpose: Catch basic issues early

Integration Testing

Trigger: Pull requests, main branch pushes

Configuration: Uses production Docker Compose setup

Testing Level: Full test suite

Purpose: Validate complete integration before merge

Production Deployment

Trigger: Main branch pushes, hotfixes

Configuration: Production-optimized builds

Testing Level: Full validation + deployment

Purpose: Deploy tested, validated code

Service Detection and Building

Change Detection

The CI pipeline intelligently detects which services have changed:

# Example change detection
if changes in sirius-ui/ → build sirius-ui
if changes in sirius-api/ → build sirius-api  
if changes in sirius-engine/ → build sirius-engine
if changes in docker-compose*.yaml → build all services

Build Strategy

Incremental Building: Only build services that have changed

Parallel Building: Build multiple services simultaneously

Caching: Use Docker layer caching for faster builds

Multi-Platform: Build for both AMD64 and ARM64 architectures

Testing Infrastructure

Test Categories

1. Build Tests

   • Individual container builds
   • Multi-stage build validation
   • Architecture compatibility

2. Health Tests

   • Service startup validation
   • Health endpoint checks
   • Resource availability

3. Integration Tests

   • Cross-service communication
   • Database connectivity
   • Message queue functionality

Test Environment

Configuration: Production-like environment

Database: PostgreSQL with test data

Cache: Valkey (Redis-compatible)

Message Queue: RabbitMQ

Network: Isolated Docker network

Docker Image Strategy

Image Tagging

Environment	Tag Pattern	Purpose
Development	sirius-*:dev	Development builds
Production	sirius-*:prod	Production builds
Base	sirius-*:latest	Default builds
CI	sirius-*:ci-{sha}	CI-specific builds

Registry Management

Primary Registry: GitHub Container Registry (ghcr.io)

Namespace: siriusscan

Image Naming: ghcr.io/siriusscan/sirius-{service}:{tag}

Workflow Jobs

1. Change Detection

Purpose: Determine which services need building

Inputs: Git diff, file changes

Outputs: Service build flags

Duration: ~30 seconds

2. Build and Push

Purpose: Build and push Docker images

Inputs: Change detection results

Outputs: Built and tagged images

Duration: ~3-5 minutes

3. Integration Testing

Purpose: Validate complete system integration

Inputs: Built images

Outputs: Test results

Duration: ~2-3 minutes

4. Deployment (Production)

Purpose: Deploy to production environment

Inputs: Tested images

Outputs: Deployed services

Duration: ~1-2 minutes

Configuration Management

Environment Variables

Development: Local .env files

CI: Generated .env.ci.* files

Production: Secure environment variables

Docker Compose Files

Base: docker-compose.yaml - Core configuration

Development: docker-compose.dev.yaml - Development overrides

Production: docker-compose.prod.yaml - Production overrides

CI: docker-compose.ci.yml - Generated CI configuration

Monitoring and Observability

Build Monitoring

• Build success/failure rates
• Build duration trends
• Resource utilization

Test Monitoring

• Test pass/fail rates
• Test duration trends
• Flaky test detection

Deployment Monitoring

• Deployment success rates
• Service health post-deployment
• Performance metrics

Troubleshooting

Common Issues

Issue	Symptoms	Solution
Build Failures	Docker build errors	Check Dockerfile syntax, dependencies
Test Failures	Integration test errors	Check service logs, network connectivity
Deployment Failures	Service startup errors	Check environment variables, resource limits
Cache Issues	Stale builds	Clear Docker cache, rebuild with --no-cache

Debugging Commands

# Check CI logs
gh run list
gh run view {run-id}

# Local testing
make test-all
make logs

# Docker debugging
docker compose ps
docker compose logs {service}

Best Practices

For Developers

1. Use feature branches for all development
2. Test locally before pushing
3. Keep commits focused and atomic
4. Write descriptive commit messages
5. Update documentation when changing CI

For CI/CD

1. Keep builds fast with proper caching
2. Use conditional execution based on changes
3. Monitor build metrics regularly
4. Update dependencies regularly
5. Test CI changes in feature branches

Migration from Legacy CI

What Changed

• Simplified structure: Removed complex submodule handling
• Modern testing: Integrated with current testing system
• Better caching: Improved Docker layer caching
• Cleaner workflows: Streamlined job structure

Migration Steps

1. Update workflows to use new Docker Compose structure
2. Integrate testing system with CI pipeline
3. Update image tagging strategy
4. Modernize deployment process
5. Update documentation to reflect changes

Future Enhancements

Planned Improvements

• Parallel testing across multiple environments
• Advanced caching strategies
• Performance testing integration
• Security scanning automation
• Rollback capabilities for failed deployments

Monitoring Enhancements

• Real-time notifications for build failures
• Performance dashboards for CI metrics
• Automated rollback triggers
• Cost optimization recommendations

---

This CI/CD documentation follows the Sirius Documentation Standard. For questions about the pipeline, see README.developer-guide.md.