siriusscan--sirius/documentation/dev/test/README.container-testing.md

title, description, template, version, last_updated, author, tags, categories, difficulty, prerequisites, related_docs, dependencies, llm_context, search_keywords

title	description	template	version	last_updated	author	tags	categories	difficulty	prerequisites	related_docs	dependencies	llm_context	search_keywords
Container Testing	Comprehensive container testing system for Sirius Scan, providing automated validation of Docker builds, service health, and integration before production deployment	TEMPLATE.documentation-standard	1.0.0	2025-01-03	Development Team	docker testing containers ci-cd validation	development testing infrastructure	intermediate	docker make bash	README.development.md ABOUT.documentation.md	docker-compose.yaml testing/container-testing/ testing/Makefile	high	container testing docker health-check validation ci-cd

Container Testing

Purpose

This document describes the comprehensive container testing system for SiriusScan, providing automated validation of Docker builds, service health, and integration before production deployment. It serves as the definitive guide for developers and DevOps engineers who need to ensure container reliability and system integrity.

When to Use

• Before every production deployment - Validate that all containers build and function correctly
• After making changes to Dockerfiles - Ensure modifications don't break the build process
• During development cycles - Catch container issues early in the development process
• When troubleshooting container issues - Use health checks and integration tests to diagnose problems
• In CI/CD pipelines - Automated validation of container builds and functionality
• When onboarding new team members - Understand the testing infrastructure and processes

How to Use

Quick Start

# Navigate to container testing directory
cd testing/container-testing

# Run complete test suite (includes runtime auth contract — see below)
make test-all

# Run specific test types
make test-build    # Build validation only
make test-health   # Health checks only
make test-integration  # Integration tests only
make test-runtime-contract  # SIRIUS_API_KEY parity, engine preflight script, API HTTP contract
make test-public-ghcr  # anonymous GHCR access for the compose-rendered IMAGE_TAG
make test-release-gates  # test-build + test-integration + test-runtime-contract (release bar)

# Test specific environment
make test-dev      # Development environment
make test-prod     # Production environment

Alternative: Run scripts directly from project root

# From project root directory
./testing/container-testing/test-integration.sh
./testing/container-testing/test-health.sh
./testing/container-testing/test-build.sh

Prerequisites

• Docker Engine 20.10.0+
• Docker Compose V2
• Make utility
• curl, nc (netcat), jq (for health checks)
• 4GB RAM minimum (8GB recommended)

Step-by-Step Process

1. Navigate to container testing directory: cd /path/to/Sirius/testing/container-testing
2. Run build tests: make test-build to validate all containers build successfully
3. Run health tests: make test-health to verify services start and respond correctly
4. Run integration tests: make test-integration to test service interactions
5. make test-all also runs test-runtime-contract, which starts the stack from the repo root and runs scripts/verify-runtime-auth-contract.sh (shared SIRIUS_API_KEY, postgres parity, authenticated GET /host/).
6. Review results: Check test output and logs in testing/logs/ directory

Alternative approach (from project root):

1. Navigate to project root: cd /path/to/Sirius
2. Run tests directly: ./testing/container-testing/test-integration.sh
3. Review results: Check test output and logs in testing/logs/ directory

What It Is

Architecture Overview

The container testing system consists of these main components:

• Build Validation: Tests individual container builds and Docker Compose configurations
• Health Checks: Validates service startup, connectivity, and basic functionality
• Integration Tests: Verifies inter-service communication and data flow
• Runtime Auth Contract (make test-runtime-contract): Runs scripts/verify-runtime-auth-contract.sh against a running stack. Default container names match container_name in docker-compose.yaml (sirius-ui, sirius-api, etc.). For a non-default Compose project (e.g. name: sirius-test), set SIRIUS_CONTRACT_CONTAINER_UI, SIRIUS_CONTRACT_CONTAINER_API, SIRIUS_CONTRACT_CONTAINER_ENGINE, and SIRIUS_CONTRACT_CONTAINER_POSTGRES before running the script. Use SIRIUS_API_PUBLIC_URL if the API is not on http://localhost:9001.
• Public GHCR Contract (make test-public-ghcr): Runs scripts/verify-ghcr-public-access.sh with the selected IMAGE_TAG and checks the exact image refs rendered by docker-compose.yaml. Failures distinguish unauthorized (visibility/token problem) from manifest unknown (tag publication problem).

Technical Details

Build Testing System

Purpose: Ensures all containers can be built successfully with correct configurations.

Components:

• testing/container-testing/test-build.sh - Main build validation script
• Tests all Docker Compose configurations (base, dev, prod)
• Validates individual container builds for all services
• Cleans up test images after completion

Services Tested:

• sirius-ui (Next.js frontend)
• sirius-api (Go REST API)
• sirius-engine (Multi-service container)
• sirius-postgres (Database)
• sirius-valkey (Cache)
• sirius-rabbitmq (Message queue)

Build Targets Validated:

• Production builds (production, runner, runtime stages)
• Development builds (development stage)
• Multi-stage Dockerfile validation

Health Check System

Purpose: Verifies that all services start correctly and respond to basic health checks.

Components:

• testing/container-testing/test-health.sh - Health validation script
• Service-specific health endpoints
• Database connectivity tests
• Port accessibility validation

Health Checks Performed:

1. Container Status: All 6 containers running and healthy
2. UI Health: Next.js application serving content on port 3000
3. API Health: Go API responding on port 9001 with /health endpoint
4. Engine Health: Process validation for sirius-engine container
5. Database Health: PostgreSQL connectivity and readiness
6. Cache Health: Valkey/Redis responding to ping commands
7. Queue Health: RabbitMQ broker status and connectivity
8. Port Accessibility: External port binding validation

Integration Testing System

Purpose: Tests inter-service communication and data flow between components.

Components:

• testing/container-testing/test-integration.sh - Integration validation script
• Service-to-service communication tests
• Database integration validation
• Message queue functionality tests

Integration Tests:

• API to Database connectivity
• UI to API communication
• Engine to Message Queue integration
• Cross-service data validation

Implementation Details

File Structure

testing/
├── container-testing/
│   ├── test-build.sh          # Build validation
│   ├── test-health.sh         # Health checks
│   ├── test-integration.sh    # Integration tests
│   └── Makefile               # Container testing commands
├── logs/                      # Test execution logs
│   ├── build_test_*.log
│   ├── health_test_*.log
│   └── integration_test_*.log
└── README.md                  # Testing documentation

Note: The Makefile is located in the container-testing/ directory and should be run from there. The test scripts can be run either from the container-testing/ directory via Makefile or directly from the project root.

Makefile Integration

The testing system uses a dedicated Makefile in the container-testing/ directory:

# Testing targets (run from testing/container-testing/)
test-all: test-build test-health test-integration
test-build: ./test-build.sh
test-health: ./test-health.sh
test-integration: ./test-integration.sh

# Environment-specific testing
test-dev: ./test-health.sh dev
test-prod: ./test-health.sh prod

Usage:

• Run from testing/container-testing/ directory: make test-all
• Or run scripts directly from project root: ./testing/container-testing/test-integration.sh

Logging and Reporting

Log Files: All test executions create timestamped log files in testing/logs/

• Format: {test_type}_test_{timestamp}.log
• Contains: Full command output, test results, error details
• Retention: Logs are kept for troubleshooting and analysis

Test Results: Each test provides clear pass/fail indicators:

• ✅ PASSED: Test completed successfully
• ❌ FAILED: Test failed with error details
• Summary: Total tests, passed, failed counts

Environment Support

Development Environment:

• Uses docker-compose.dev.yaml overrides
• Includes volume mounts for live development
• Accounts for longer startup times due to dependency downloads

Production Environment:

• Uses docker-compose.prod.yaml overrides
• Tests production-optimized builds
• Validates production configuration

Base Environment:

• Uses standard docker-compose.yaml
• Tests default configuration
• Validates core functionality

Advanced Topics

Custom Test Configuration

Environment Variables:

export TEST_RETRIES=120         # Retry attempts (default: 60, dev mode: 120)
export LOG_LEVEL=debug          # Logging verbosity (info, debug)

Timeout Configuration:

• Default: 60 retries × 2 seconds = 120 seconds total
• Development Mode: Automatically increases to 120 retries × 2 seconds = 240 seconds total
• Custom: Override with TEST_RETRIES environment variable

Custom Test Execution:

# Run with custom timeout (4 minutes)
TEST_RETRIES=120 ./testing/scripts/test-health.sh dev

# Run with debug logging
LOG_LEVEL=debug ./testing/scripts/test-health.sh dev

# Run with both custom timeout and debug logging
TEST_RETRIES=180 LOG_LEVEL=debug ./testing/scripts/test-health.sh dev

Performance Monitoring

Resource Usage Tracking:

• Container startup times
• Memory and CPU usage during tests
• Build time optimization
• Service response times

Optimization Opportunities:

• Parallel test execution
• Build cache utilization
• Service startup optimization
• Resource limit tuning

Security Considerations

Test Isolation:

• Tests run in isolated Docker networks
• No external network access during testing
• Cleanup procedures remove all test artifacts

Security Validation:

• Container security scanning (future enhancement)
• Vulnerability assessment (future enhancement)
• Secret management validation (future enhancement)

Troubleshooting

FAQ

Q: Why do health tests fail with "API not ready yet" errors? A: The sirius-api service takes time to download Go dependencies in development mode. The health check automatically increases timeout to 4 minutes for dev mode, and includes fallback validation for this scenario.

Q: Why does development mode take longer than production mode? A: Development mode downloads Go dependencies at runtime, which can take 2-4 minutes depending on your internet connection. Production mode pre-builds these dependencies, so it starts much faster.

Q: How do I run tests for a specific service only? A: Modify the test scripts to comment out unwanted services, or use Docker Compose to start only the services you need to test.

Q: What if build tests fail with "target stage not found" errors? A: Check that Docker Compose files reference correct Dockerfile stage names. Common stages are development, production, runtime, and runner.

Q: How can I increase test timeout for slow systems? A: Set the TEST_RETRIES environment variable: export TEST_RETRIES=120 && make test-health (120 retries = 4 minutes total)

Q: Why do integration tests fail with connection refused errors? A: Ensure all services are fully started before running integration tests. Use docker compose ps to verify service status.

Q: How do I debug a specific test failure? A: Check the timestamped log file in testing/logs/ for detailed error information and command output.

Q: Can I run tests in parallel? A: Currently tests run sequentially to avoid resource conflicts. Parallel execution is planned for future enhancement.

Q: What if I need to test with custom Docker images? A: Modify the test scripts to use your custom image tags, or update the Docker Compose files to reference your images.

Command Reference

# From testing/container-testing/ directory
make test-all
make test-build
make test-health
make test-integration
make test-public-ghcr
make test-dev
make test-prod

# Or from project root directory
./testing/container-testing/test-build.sh
./testing/container-testing/test-health.sh dev
./testing/container-testing/test-integration.sh prod

# Docker Compose validation
docker compose config --quiet
docker compose -f docker-compose.yaml -f docker-compose.dev.yaml config --quiet

# Service health checks
docker compose ps
docker compose logs sirius-api
curl -f http://localhost:9001/health

# Container debugging
docker exec -it sirius-engine /bin/bash
docker logs sirius-engine --tail 50 -f

# Cleanup
docker compose down -v
docker system prune -f

Common Issues and Solutions

Issue	Symptoms	Solution
Build failures	"target stage not found" errors	Check Dockerfile stage names in Compose files
Health check timeouts	"service not ready" errors	Increase timeout or check service logs
Port conflicts	"port already in use" errors	Stop conflicting services or change ports
Permission errors	"permission denied" in containers	Check file permissions and Docker daemon access
Network issues	"connection refused" errors	Verify Docker network configuration
Resource exhaustion	Out of memory errors	Increase Docker memory limits or close other applications

Debugging Steps

1. Check service status: docker compose ps to see which containers are running
2. Review service logs: docker compose logs [service-name] for error details
3. Verify network connectivity: docker network ls and docker network inspect sirius
4. Test individual components: Run health checks manually with curl commands
5. Check resource usage: docker stats to monitor CPU and memory usage
6. Validate configuration: docker compose config to check for syntax errors

Log Analysis

Build Test Logs:

# View build test results
tail -f testing/logs/build_test_*.log

# Check for specific errors
grep -i "error\|failed" testing/logs/build_test_*.log

Health Test Logs:

# Monitor health check progress
tail -f testing/logs/health_test_*.log

# Find failed tests
grep -i "failed\|error" testing/logs/health_test_*.log

Integration Test Logs:

# Check integration test results
cat testing/logs/integration_test_*.log

# Analyze service communication
grep -i "connection\|timeout" testing/logs/integration_test_*.log

Performance Troubleshooting

Slow Build Times:

• Check Docker BuildKit is enabled: export DOCKER_BUILDKIT=1
• Clear Docker cache: docker builder prune
• Use build cache mounts for dependencies

Slow Service Startup:

• Check system resources: docker stats
• Optimize Dockerfile layers
• Use multi-stage builds efficiently

Test Execution Time:

• Run tests in parallel (future enhancement)
• Optimize health check intervals
• Use faster base images where possible

Lessons Learned

2025-01-03: Implemented comprehensive container testing system to address build reliability issues. Key insight: Automated testing prevents deployment failures and improves development confidence.

2025-01-03: Created health check system with fallback validation for services with longer startup times. Lesson: Account for real-world service startup patterns in test design.

2025-01-03: Established layered testing approach (build → health → integration). Benefit: Catches issues at appropriate levels and provides clear failure isolation.

2025-01-03: Implemented timestamped logging system for all test executions. Advantage: Enables detailed troubleshooting and historical analysis of test failures.

LLM Context

[Additional context specifically for Large Language Models:]

• Key Concepts: Container testing involves build validation, health checks, and integration testing to ensure Docker containers work correctly before deployment
• Technical Context: Uses Docker Compose for orchestration, shell scripts for automation, and Make for command execution. Tests cover 6 services: UI, API, Engine, PostgreSQL, Valkey, and RabbitMQ
• Common Patterns: Layered testing approach (build → health → integration), automated cleanup, timestamped logging, environment-specific configurations
• Edge Cases: API dependency downloading in dev mode, service startup timing variations, port conflicts, resource constraints
• Integration Points: Connects with CI/CD pipelines, deployment processes, monitoring systems, and development workflows

---

This document follows the Sirius Documentation Standard. For questions about documentation structure, see ABOUT.documentation.md.