chore: import upstream snapshot with attribution

scripts/agent-identities/README.md

@@ -0,0 +1,385 @@
+# Agent Identity Generation System
+
+A TypeScript-based system for generating comprehensive agent identity files from documentation and code.
+
+## Overview
+
+This system combines **manual templates** (role summaries, best practices) with **auto-generated sections** (file structure, ports, dependencies, configs) to create comprehensive agent identities (~1200-1500 lines).
+
+## Quick Start
+
+```bash
+# Install dependencies
+npm install
+
+# Generate all agent identities
+npm run generate -- --all
+
+# Generate specific agent
+npm run generate -- --product=agent-engineer
+
+# Validate agent identities
+npm run validate
+
+# Check for stale identities
+npm run check-sync
+```
+
+## Commands
+
+### Generation
+
+```bash
+# Generate all agent identities
+npm run generate -- --all
+
+# Generate specific product
+npm run generate -- --product=<product-name>
+
+# Example
+npm run generate -- --product=agent-engineer
+```
+
+### Validation
+
+```bash
+# Full validation (YAML, content, sync, file paths)
+npm run validate
+
+# Check if identities need regeneration
+npm run check-sync
+```
+
+### Via Makefile
+
+```bash
+cd testing/container-testing
+
+# Generation
+make regenerate-agents                    # All
+make regenerate-agent PRODUCT=agent-engineer  # Specific
+
+# Validation
+make lint-agents                          # Full
+make check-agent-sync                     # Sync check
+```
+
+## Project Structure
+
+```
+scripts/agent-identities/
+├── package.json              # Project config
+├── tsconfig.json             # TypeScript config
+├── src/
+│   ├── generators/           # Data extractors
+│   │   ├── index.ts         # Main orchestrator
+│   │   ├── file-structure.ts
+│   │   ├── ports.ts
+│   │   ├── dependencies.ts
+│   │   ├── config-examples.ts
+│   │   ├── documentation.ts
+│   │   └── merge.ts
+│   ├── validators/           # Validation modules
+│   │   ├── yaml-validator.ts
+│   │   ├── content-validator.ts
+│   │   ├── sync-validator.ts
+│   │   └── file-path-validator.ts
+│   ├── types/                # Type definitions
+│   └── utils/                # Utilities
+├── lint-agents.ts            # Validation CLI
+├── generate-agents.ts        # Generation CLI
+└── check-sync.ts             # Sync check CLI
+```
+
+## Creating a New Agent Identity
+
+### 1. Create Template
+
+Create `.cursor/agents/templates/<name>.template.md`:
+
+```markdown
+---
+# YAML front matter (manually maintained)
+name: "My Agent"
+title: "My Agent (product/Tech)"
+# ... metadata
+---
+
+# My Agent
+
+<!-- MANUAL SECTION -->
+Manual narrative content
+<!-- END MANUAL SECTION -->
+
+## Key Documentation
+
+<!-- AUTO-GENERATED: documentation-links -->
+<!-- END AUTO-GENERATED -->
+
+## Project Location
+
+<!-- AUTO-GENERATED: file-structure -->
+<!-- END AUTO-GENERATED -->
+
+## Technology Stack
+
+<!-- AUTO-GENERATED: dependencies -->
+<!-- END AUTO-GENERATED -->
+```
+
+### 2. Create Config
+
+Create `.cursor/agents/config/<name>.config.yaml`:
+
+```yaml
+product: my-product
+product_path: /path/to/product
+docker_service_name: my-service
+
+metadata:
+  name: "My Agent"
+  title: "My Agent (product/Tech)"
+  description: "Brief description"
+  role_type: engineering
+  version: "1.0.0"
+  last_updated: "2025-10-25"
+  author: "Sirius Team"
+  specialization: ["skill1", "skill2"]
+  technology_stack: ["Tech1", "Tech2"]
+  system_integration_level: high
+  categories: ["category"]
+  tags: ["tag1", "tag2"]
+  related_docs: ["doc1.md", "doc2.md"]
+  dependencies: ["path/"]
+  llm_context: high
+  context_window_target: 1400
+
+generation:
+  include_file_structure: true
+  file_structure_depth: 3
+  file_structure_ignore: [node_modules, dist, bin]
+  
+  include_ports: true
+  ports_config:
+    "8080": "HTTP server"
+  
+  include_dependencies: true
+  dependencies_source: go.mod  # or package.json
+  dependencies_grouping:
+    "Category":
+      - "package-name"
+  
+  include_config_examples: true
+  config_files:
+    - path: config/server.yaml
+      title: "Server Configuration"
+  
+  extract_code_patterns_from_docs:
+    - documentation/dev/file.md
+
+template_path: .cursor/agents/templates/<name>.template.md
+output_path: .cursor/agents/<name>.agent.md
+```
+
+### 3. Generate
+
+```bash
+npm run generate -- --product=<name>
+```
+
+### 4. Validate
+
+```bash
+npm run validate
+```
+
+## Template Markers
+
+### AUTO-GENERATED Sections
+
+Sections between these markers are auto-generated:
+
+```markdown
+<!-- AUTO-GENERATED: section-name -->
+Content is replaced on each generation
+<!-- END AUTO-GENERATED -->
+```
+
+### MANUAL Sections
+
+Sections between these markers are preserved:
+
+```markdown
+<!-- MANUAL SECTION -->
+Content is preserved across generations
+<!-- END MANUAL SECTION -->
+```
+
+## Available Generators
+
+### file-structure
+Extracts directory tree from product path with max depth and ignore patterns.
+
+### ports
+Extracts port mappings from docker-compose.yaml for specified service.
+
+### dependencies
+Extracts and groups dependencies from go.mod or package.json.
+
+### config-examples
+Extracts and formats configuration file contents.
+
+### documentation-links
+Generates documentation links from related_docs list.
+
+### code-patterns
+Extracts code patterns, best practices, and common tasks from documentation.
+
+## Validation
+
+### YAML Front Matter
+- Required fields present
+- Enum values valid (role_type, llm_context, etc.)
+- Version format (semver: 1.0.0)
+- Date format (YYYY-MM-DD)
+
+### Content
+- Line count in range (150-2000)
+- Warning if outside target (800-1500)
+- Required sections present
+
+### Sync Status
+- Compares source file modification times
+- Detects when regeneration needed
+- Tracks source files via metadata
+
+### File Paths
+- Extracts file references from content
+- Verifies files exist
+- Reports missing files
+
+## Generation Metadata
+
+Generated files include metadata:
+
+```yaml
+---
+# ... standard metadata ...
+_generated_at: "2025-10-25T10:30:00.000Z"
+_source_files:
+  - "/path/to/source1"
+  - "/path/to/source2"
+---
+```
+
+This enables:
+- Staleness detection
+- Source tracking
+- Regeneration decisions
+
+## Development
+
+### Run Tests
+
+```bash
+npm test  # (when tests are added)
+```
+
+### Build
+
+```bash
+npm run build
+```
+
+### Type Check
+
+```bash
+npx tsc --noEmit
+```
+
+## Troubleshooting
+
+### "Cannot find module"
+
+Ensure dependencies are installed:
+```bash
+npm install
+```
+
+### "Config file not found"
+
+Check config path in command:
+```bash
+npm run generate -- --product=<exact-name>
+```
+
+Config must exist at `.cursor/agents/config/<exact-name>.config.yaml`
+
+### "Template not found"
+
+Check template_path in config file points to valid template.
+
+### Generation Fails
+
+Check:
+1. Product path exists and is accessible
+2. Source files (go.mod, docker-compose.yaml) exist
+3. Config YAML is valid
+4. Template has proper markers
+
+### Validation Fails
+
+Check:
+1. YAML front matter has all required fields
+2. Field values match enum options
+3. Version and date formats correct
+4. File is within line count limits
+
+## Integration with Workflow
+
+### Pre-Commit Hook
+
+Agent identities are validated on commit when source files change.
+
+### CI/CD
+
+Use in CI/CD pipelines:
+
+```bash
+cd scripts/agent-identities
+npm run validate || exit 1
+```
+
+### Makefile
+
+Integrated with project Makefile:
+
+```bash
+make regenerate-agents        # Generate all
+make regenerate-agent PRODUCT=name  # Generate specific
+make check-agent-sync         # Check staleness
+make lint-agents              # Validate all
+```
+
+## Best Practices
+
+1. **Manual Sections:** Use for narrative, philosophy, best practices
+2. **Auto-Generated Sections:** Use for structural info (files, ports, deps)
+3. **Regular Regeneration:** Regenerate after major changes
+4. **Validate Before Commit:** Run validation before committing
+5. **Keep Templates Clean:** Don't add manual content in auto-generated sections
+
+## Resources
+
+- **Plan:** `/agent-identity-system.plan.md`
+- **Implementation Summary:** `/AGENT-IDENTITY-GENERATION-IMPLEMENTED.md`
+- **Templates:** `.cursor/agents/templates/`
+- **Configs:** `.cursor/agents/config/`
+- **Generated Identities:** `.cursor/agents/*.agent.md`
+
+---
+
+**Version:** 1.0.0  
+**Last Updated:** 2025-10-25
+
+

scripts/agent-identities/lint-agent-index.sh

@@ -0,0 +1,248 @@
+#!/bin/bash
+
+# Sirius Agent Identity Index Linting
+# Validates agent index completeness and consistency
+
+set -e
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m' # No Color
+
+# Configuration
+AGENTS_DIR="../../.cursor/agents"
+INDEX_FILE="$AGENTS_DIR/docs/INDEX.agent-identities.md"
+SCRIPT_DIR="$(dirname "$0")"
+LOG_FILE="tmp/index-lint-$(date +%Y%m%d-%H%M%S).log"
+
+# Create tmp directory if it doesn't exist
+mkdir -p tmp
+
+# Counters
+ERRORS=0
+WARNINGS=0
+
+# Functions
+log() {
+    echo -e "${BLUE}[$(date +'%H:%M:%S')]${NC} $1" | tee -a "$LOG_FILE"
+}
+
+success() {
+    echo -e "${GREEN}✅ $1${NC}" | tee -a "$LOG_FILE"
+}
+
+warning() {
+    echo -e "${YELLOW}⚠️  $1${NC}" | tee -a "$LOG_FILE"
+    ((WARNINGS++))
+}
+
+error() {
+    echo -e "${RED}❌ $1${NC}" | tee -a "$LOG_FILE"
+    ((ERRORS++))
+}
+
+# Check if index file exists
+check_index_exists() {
+    if [ ! -f "$INDEX_FILE" ]; then
+        error "Index file not found: $INDEX_FILE"
+        return 1
+    fi
+    return 0
+}
+
+# Get all agent files
+get_agent_files() {
+    find "$AGENTS_DIR" -name "*.agent.md" -type f | sort
+}
+
+# Extract agents mentioned in index
+get_indexed_agents() {
+    if [ ! -f "$INDEX_FILE" ]; then
+        return 1
+    fi
+    
+    # Look for markdown links to .agent.md files
+    grep -oE '\[.*\]\([^)]*\.agent\.md\)' "$INDEX_FILE" | \
+        grep -oE '[^/]+\.agent\.md' | \
+        sort -u
+}
+
+# Check for orphaned agent files
+check_orphaned_files() {
+    log "Checking for orphaned agent files..."
+    
+    local orphaned=0
+    
+    while IFS= read -r agent_file; do
+        local basename=$(basename "$agent_file")
+        
+        # Skip template and system files
+        if [[ "$basename" == "TEMPLATE."* ]] || \
+           [[ "$basename" == "ABOUT."* ]] || \
+           [[ "$basename" == "INDEX."* ]] || \
+           [[ "$basename" == "GUIDE."* ]] || \
+           [[ "$basename" == "REFERENCE."* ]] || \
+           [[ "$basename" == "SPECIFICATION."* ]]; then
+            continue
+        fi
+        
+        # Check if mentioned in index
+        if ! grep -q "$basename" "$INDEX_FILE" 2>/dev/null; then
+            error "Agent file not in index: $basename"
+            ((orphaned++))
+        fi
+    done < <(get_agent_files)
+    
+    if [ $orphaned -eq 0 ]; then
+        success "No orphaned agent files found"
+        return 0
+    else
+        error "Found $orphaned orphaned agent file(s)"
+        return 1
+    fi
+}
+
+# Check for broken index references
+check_broken_references() {
+    log "Checking for broken index references..."
+    
+    local broken=0
+    
+    while IFS= read -r indexed_agent; do
+        local full_path="$AGENTS_DIR/$indexed_agent"
+        
+        if [ ! -f "$full_path" ]; then
+            error "Index references non-existent file: $indexed_agent"
+            ((broken++))
+        fi
+    done < <(get_indexed_agents)
+    
+    if [ $broken -eq 0 ]; then
+        success "No broken index references found"
+        return 0
+    else
+        error "Found $broken broken reference(s)"
+        return 1
+    fi
+}
+
+# Check index has required sections
+check_index_structure() {
+    log "Checking index structure..."
+    
+    local structure_errors=0
+    
+    if ! grep -q "^# Agent Identity Index" "$INDEX_FILE" 2>/dev/null; then
+        error "Index missing main heading"
+        ((structure_errors++))
+    fi
+    
+    if ! grep -q "## By Role Type" "$INDEX_FILE" 2>/dev/null; then
+        warning "Index missing 'By Role Type' section"
+    fi
+    
+    if ! grep -q "## Complete List" "$INDEX_FILE" 2>/dev/null; then
+        warning "Index missing 'Complete List' section"
+    fi
+    
+    if [ $structure_errors -eq 0 ]; then
+        success "Index structure is valid"
+        return 0
+    else
+        return 1
+    fi
+}
+
+# Verify metadata consistency
+check_metadata_consistency() {
+    log "Checking metadata consistency..."
+    
+    local inconsistencies=0
+    
+    while IFS= read -r agent_file; do
+        local basename=$(basename "$agent_file")
+        
+        # Skip system files
+        if [[ "$basename" == "TEMPLATE."* ]] || \
+           [[ "$basename" == "ABOUT."* ]] || \
+           [[ "$basename" == "INDEX."* ]] || \
+           [[ "$basename" == "GUIDE."* ]] || \
+           [[ "$basename" == "REFERENCE."* ]] || \
+           [[ "$basename" == "SPECIFICATION."* ]]; then
+            continue
+        fi
+        
+        # Extract name from YAML
+        local yaml_name=$(sed -n '/^---$/,/^---$/p' "$agent_file" | grep "^name:" | sed 's/name: *"\?\([^"]*\)"\?/\1/' | tr -d '"')
+        
+        if [ -n "$yaml_name" ]; then
+            # Check if name appears in index
+            if ! grep -q "$yaml_name" "$INDEX_FILE" 2>/dev/null; then
+                warning "Agent name '$yaml_name' from $basename not found in index"
+                ((inconsistencies++))
+            fi
+        fi
+    done < <(get_agent_files)
+    
+    if [ $inconsistencies -eq 0 ]; then
+        success "Metadata consistency check passed"
+        return 0
+    else
+        warning "Found $inconsistencies metadata inconsistenc(y|ies)"
+        return 0  # Just warnings, not errors
+    fi
+}
+
+# Count agents by type
+count_agents_by_type() {
+    log "Agent counts by role type:"
+    
+    for role_type in "engineering" "design" "product" "operations" "qa" "documentation"; do
+        local count=$(find "$AGENTS_DIR" -name "*.agent.md" -type f -exec grep -l "^role_type: *\"*${role_type}\"*" {} \; 2>/dev/null | wc -l)
+        log "  $role_type: $count"
+    done
+}
+
+# Main script
+log "Starting agent identity index validation..."
+log "Agents directory: $AGENTS_DIR"
+log "Index file: $INDEX_FILE"
+log "=================================================="
+
+# Check index exists
+if ! check_index_exists; then
+    error "Cannot continue without index file"
+    exit 1
+fi
+
+# Run all checks
+check_index_structure
+check_orphaned_files
+check_broken_references
+check_metadata_consistency
+
+echo ""
+count_agents_by_type
+echo ""
+
+# Summary
+log "=================================================="
+log "Index Validation Summary:"
+log "  Errors: $ERRORS"
+log "  Warnings: $WARNINGS"
+log "=================================================="
+
+if [ $ERRORS -eq 0 ]; then
+    success "Agent identity index validation passed!"
+    if [ $WARNINGS -gt 0 ]; then
+        warning "Validation passed with $WARNINGS warning(s)"
+    fi
+    exit 0
+else
+    error "Index validation failed with $ERRORS error(s)"
+    exit 1
+fi
+

scripts/agent-identities/lint-agents-quick.sh

@@ -0,0 +1,166 @@
+#!/bin/bash
+
+# Sirius Agent Identity Quick Linting
+# Fast validation for CI/CD - checks only critical items
+
+set -e
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m' # No Color
+
+# Configuration
+AGENTS_DIR="../../.cursor/agents"
+SCRIPT_DIR="$(dirname "$0")"
+
+# Required YAML front matter fields
+REQUIRED_FIELDS=("name" "title" "description" "role_type" "version" "last_updated" "llm_context" "context_window_target")
+
+# Length constraints
+MIN_LINES=150
+MAX_LINES=500
+
+# Counters
+TOTAL_FILES=0
+PASSED_FILES=0
+FAILED_FILES=0
+
+# Functions
+log() {
+    echo -e "${BLUE}[$(date +'%H:%M:%S')]${NC} $1"
+}
+
+success() {
+    echo -e "${GREEN}✅ $1${NC}"
+}
+
+error() {
+    echo -e "${RED}❌ $1${NC}"
+}
+
+# Check if file has YAML front matter
+check_yaml_front_matter() {
+    local file="$1"
+    
+    if ! head -n 1 "$file" | grep -q "^---$"; then
+        error "Missing YAML front matter in $(basename $file)"
+        return 1
+    fi
+    
+    return 0
+}
+
+# Extract YAML front matter
+extract_yaml() {
+    local file="$1"
+    local yaml_start=0
+    local yaml_end=0
+    local line_num=0
+    
+    while IFS= read -r line; do
+        ((line_num++))
+        if [ "$line" = "---" ]; then
+            if [ $yaml_start -eq 0 ]; then
+                yaml_start=$line_num
+            else
+                yaml_end=$line_num
+                break
+            fi
+        fi
+    done < "$file"
+    
+    if [ $yaml_start -gt 0 ] && [ $yaml_end -gt 0 ]; then
+        sed -n "${yaml_start},${yaml_end}p" "$file"
+    fi
+}
+
+# Quick check required fields
+check_required_fields() {
+    local file="$1"
+    local yaml_content
+    local missing_fields=()
+    
+    yaml_content=$(extract_yaml "$file")
+    if [ -z "$yaml_content" ]; then
+        error "Could not extract YAML from $(basename $file)"
+        return 1
+    fi
+    
+    for field in "${REQUIRED_FIELDS[@]}"; do
+        if ! echo "$yaml_content" | grep -q "^${field}:"; then
+            missing_fields+=("$field")
+        fi
+    done
+    
+    if [ ${#missing_fields[@]} -gt 0 ]; then
+        error "Missing fields in $(basename $file): ${missing_fields[*]}"
+        return 1
+    fi
+    
+    return 0
+}
+
+# Quick length check
+check_length() {
+    local file="$1"
+    local line_count=$(wc -l < "$file")
+    
+    if [ $line_count -lt $MIN_LINES ] || [ $line_count -gt $MAX_LINES ]; then
+        error "$(basename $file) length ($line_count) outside valid range ($MIN_LINES-$MAX_LINES)"
+        return 1
+    fi
+    
+    return 0
+}
+
+# Quick validation
+quick_validate() {
+    local file="$1"
+    local errors=0
+    
+    if ! check_yaml_front_matter "$file"; then
+        ((errors++))
+    fi
+    
+    if ! check_required_fields "$file"; then
+        ((errors++))
+    fi
+    
+    if ! check_length "$file"; then
+        ((errors++))
+    fi
+    
+    return $errors
+}
+
+# Main script
+log "Quick agent identity validation..."
+
+# Find all agent files
+for file in "$AGENTS_DIR"/*.agent.md; do
+    if [ -f "$file" ]; then
+        ((TOTAL_FILES++))
+        if quick_validate "$file"; then
+            ((PASSED_FILES++))
+        else
+            ((FAILED_FILES++))
+        fi
+    fi
+done
+
+# Summary
+echo ""
+log "Quick Validation Summary: $PASSED_FILES/$TOTAL_FILES passed"
+
+if [ $FAILED_FILES -eq 0 ]; then
+    success "Quick validation passed!"
+    exit 0
+else
+    error "Quick validation failed for $FAILED_FILES file(s)"
+    exit 1
+fi
+
+

scripts/agent-identities/lint-agents.sh

@@ -0,0 +1,379 @@
+#!/bin/bash
+
+# Sirius Agent Identity Linting System
+# Validates agent identity quality, consistency, and compliance
+
+set -e
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m' # No Color
+
+# Configuration
+AGENTS_DIR="../../.cursor/agents"
+SCRIPT_DIR="$(dirname "$0")"
+LOG_FILE="tmp/agent-lint-$(date +%Y%m%d-%H%M%S).log"
+
+# Create tmp directory if it doesn't exist
+mkdir -p tmp
+
+# Required YAML front matter fields
+REQUIRED_FIELDS=("name" "title" "description" "role_type" "version" "last_updated" "llm_context" "context_window_target")
+OPTIONAL_FIELDS=("author" "specialization" "technology_stack" "system_integration_level" "categories" "tags" "related_docs" "dependencies")
+
+# Valid values for specific fields
+VALID_ROLE_TYPES=("engineering" "design" "product" "operations" "qa" "documentation")
+VALID_LLM_CONTEXTS=("high" "medium" "low")
+VALID_INTEGRATION_LEVELS=("none" "low" "medium" "high")
+
+# Length constraints
+MIN_LINES=150
+MAX_LINES=500
+TARGET_MIN=200
+TARGET_MAX=400
+
+# Counters
+TOTAL_FILES=0
+PASSED_FILES=0
+FAILED_FILES=0
+WARNINGS=0
+
+# Functions
+log() {
+    echo -e "${BLUE}[$(date +'%H:%M:%S')]${NC} $1" | tee -a "$LOG_FILE"
+}
+
+success() {
+    echo -e "${GREEN}✅ $1${NC}" | tee -a "$LOG_FILE"
+}
+
+warning() {
+    echo -e "${YELLOW}⚠️  $1${NC}" | tee -a "$LOG_FILE"
+    ((WARNINGS++))
+}
+
+error() {
+    echo -e "${RED}❌ $1${NC}" | tee -a "$LOG_FILE"
+}
+
+# Check if file has YAML front matter
+check_yaml_front_matter() {
+    local file="$1"
+    local has_yaml=false
+    
+    if head -n 1 "$file" | grep -q "^---$"; then
+        has_yaml=true
+    fi
+    
+    if [ "$has_yaml" = false ]; then
+        error "Missing YAML front matter in $file"
+        return 1
+    fi
+    
+    return 0
+}
+
+# Extract YAML front matter
+extract_yaml() {
+    local file="$1"
+    local yaml_start=0
+    local yaml_end=0
+    local line_num=0
+    
+    while IFS= read -r line; do
+        ((line_num++))
+        if [ "$line" = "---" ]; then
+            if [ $yaml_start -eq 0 ]; then
+                yaml_start=$line_num
+            else
+                yaml_end=$line_num
+                break
+            fi
+        fi
+    done < "$file"
+    
+    if [ $yaml_start -gt 0 ] && [ $yaml_end -gt 0 ]; then
+        sed -n "${yaml_start},${yaml_end}p" "$file"
+    fi
+}
+
+# Check required fields
+check_required_fields() {
+    local file="$1"
+    local yaml_content
+    local missing_fields=()
+    
+    yaml_content=$(extract_yaml "$file")
+    if [ -z "$yaml_content" ]; then
+        return 1
+    fi
+    
+    for field in "${REQUIRED_FIELDS[@]}"; do
+        if ! echo "$yaml_content" | grep -q "^${field}:"; then
+            missing_fields+=("$field")
+        fi
+    done
+    
+    if [ ${#missing_fields[@]} -gt 0 ]; then
+        error "Missing required fields in $file: ${missing_fields[*]}"
+        return 1
+    fi
+    
+    return 0
+}
+
+# Validate enumerated field values
+validate_enum_fields() {
+    local file="$1"
+    local yaml_content
+    local errors=0
+    
+    yaml_content=$(extract_yaml "$file")
+    if [ -z "$yaml_content" ]; then
+        return 1
+    fi
+    
+    # Check role_type
+    local role_type=$(echo "$yaml_content" | grep "^role_type:" | cut -d: -f2- | sed 's/^[[:space:]]*//;s/[[:space:]]*$//' | tr -d '"')
+    if [ -n "$role_type" ]; then
+        local valid=false
+        for valid_type in "${VALID_ROLE_TYPES[@]}"; do
+            if [ "$role_type" = "$valid_type" ]; then
+                valid=true
+                break
+            fi
+        done
+        if [ "$valid" = false ]; then
+            error "Invalid role_type '$role_type' in $file. Must be one of: ${VALID_ROLE_TYPES[*]}"
+            ((errors++))
+        fi
+    fi
+    
+    # Check llm_context
+    local llm_context=$(echo "$yaml_content" | grep "^llm_context:" | cut -d: -f2- | sed 's/^[[:space:]]*//;s/[[:space:]]*$//' | tr -d '"')
+    if [ -n "$llm_context" ]; then
+        local valid=false
+        for valid_ctx in "${VALID_LLM_CONTEXTS[@]}"; do
+            if [ "$llm_context" = "$valid_ctx" ]; then
+                valid=true
+                break
+            fi
+        done
+        if [ "$valid" = false ]; then
+            error "Invalid llm_context '$llm_context' in $file. Must be one of: ${VALID_LLM_CONTEXTS[*]}"
+            ((errors++))
+        fi
+    fi
+    
+    # Check system_integration_level (optional)
+    local integration_level=$(echo "$yaml_content" | grep "^system_integration_level:" | cut -d: -f2- | sed 's/^[[:space:]]*//;s/[[:space:]]*$//' | tr -d '"')
+    if [ -n "$integration_level" ]; then
+        local valid=false
+        for valid_level in "${VALID_INTEGRATION_LEVELS[@]}"; do
+            if [ "$integration_level" = "$valid_level" ]; then
+                valid=true
+                break
+            fi
+        done
+        if [ "$valid" = false ]; then
+            error "Invalid system_integration_level '$integration_level' in $file. Must be one of: ${VALID_INTEGRATION_LEVELS[*]}"
+            ((errors++))
+        fi
+    fi
+    
+    return $errors
+}
+
+# Validate version format (semver)
+validate_version() {
+    local file="$1"
+    local yaml_content
+    
+    yaml_content=$(extract_yaml "$file")
+    if [ -z "$yaml_content" ]; then
+        return 1
+    fi
+    
+    local version=$(echo "$yaml_content" | grep "^version:" | cut -d: -f2- | sed 's/^[[:space:]]*//;s/[[:space:]]*$//' | tr -d '"')
+    if [ -n "$version" ]; then
+        if ! echo "$version" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+$'; then
+            error "Invalid version format '$version' in $file. Must be semver (e.g., 1.0.0)"
+            return 1
+        fi
+    fi
+    
+    return 0
+}
+
+# Validate date format (ISO 8601)
+validate_date() {
+    local file="$1"
+    local yaml_content
+    
+    yaml_content=$(extract_yaml "$file")
+    if [ -z "$yaml_content" ]; then
+        return 1
+    fi
+    
+    local date=$(echo "$yaml_content" | grep "^last_updated:" | cut -d: -f2- | sed 's/^[[:space:]]*//;s/[[:space:]]*$//' | tr -d '"')
+    if [ -n "$date" ]; then
+        if ! echo "$date" | grep -qE '^[0-9]{4}-[0-9]{2}-[0-9]{2}$'; then
+            error "Invalid date format '$date' in $file. Must be YYYY-MM-DD"
+            return 1
+        fi
+    fi
+    
+    return 0
+}
+
+# Check length constraints
+check_length() {
+    local file="$1"
+    local line_count=$(wc -l < "$file")
+    
+    if [ $line_count -lt $MIN_LINES ]; then
+        error "File $file is too short ($line_count lines). Minimum: $MIN_LINES"
+        return 1
+    elif [ $line_count -gt $MAX_LINES ]; then
+        error "File $file is too long ($line_count lines). Maximum: $MAX_LINES"
+        return 1
+    elif [ $line_count -lt $TARGET_MIN ] || [ $line_count -gt $TARGET_MAX ]; then
+        warning "File $file length ($line_count lines) outside target range ($TARGET_MIN-$TARGET_MAX)"
+    fi
+    
+    return 0
+}
+
+# Check context window target
+check_context_target() {
+    local file="$1"
+    local yaml_content
+    
+    yaml_content=$(extract_yaml "$file")
+    if [ -z "$yaml_content" ]; then
+        return 1
+    fi
+    
+    local target=$(echo "$yaml_content" | grep "^context_window_target:" | cut -d: -f2- | sed 's/^[[:space:]]*//;s/[[:space:]]*$//' | tr -d '"')
+    if [ -n "$target" ]; then
+        # Only check if it's a valid integer
+        if [[ "$target" =~ ^[0-9]+$ ]]; then
+            if [ "$target" -lt $MIN_LINES ] || [ "$target" -gt $MAX_LINES ]; then
+                warning "context_window_target $target in $file outside valid range ($MIN_LINES-$MAX_LINES)"
+            fi
+        fi
+    fi
+    
+    return 0
+}
+
+# Validate file naming
+check_filename() {
+    local file="$1"
+    local basename=$(basename "$file")
+    
+    if ! echo "$basename" | grep -qE '^[a-z0-9-]+\.agent\.md$'; then
+        error "Invalid filename format: $basename. Must match [a-z0-9-]+.agent.md"
+        return 1
+    fi
+    
+    return 0
+}
+
+# Main validation function
+validate_file() {
+    local file="$1"
+    local file_errors=0
+    
+    log "Validating $file..."
+    
+    # Check filename format
+    if ! check_filename "$file"; then
+        ((file_errors++))
+    fi
+    
+    # Check YAML front matter exists
+    if ! check_yaml_front_matter "$file"; then
+        ((file_errors++))
+        return 1  # Can't continue without YAML
+    fi
+    
+    # Check required fields
+    if ! check_required_fields "$file"; then
+        ((file_errors++))
+    fi
+    
+    # Validate enum fields
+    if ! validate_enum_fields "$file"; then
+        ((file_errors++))
+    fi
+    
+    # Validate version format
+    if ! validate_version "$file"; then
+        ((file_errors++))
+    fi
+    
+    # Validate date format
+    if ! validate_date "$file"; then
+        ((file_errors++))
+    fi
+    
+    # Check length constraints
+    if ! check_length "$file"; then
+        ((file_errors++))
+    fi
+    
+    # Check context window target
+    check_context_target "$file"
+    
+    if [ $file_errors -eq 0 ]; then
+        success "$(basename $file) passed validation"
+        return 0
+    else
+        error "$(basename $file) failed validation with $file_errors error(s)"
+        return 1
+    fi
+}
+
+# Main script
+log "Starting agent identity validation..."
+log "Agents directory: $AGENTS_DIR"
+log "=================================================="
+
+# Find all agent files
+for file in "$AGENTS_DIR"/*.agent.md; do
+    if [ -f "$file" ]; then
+        ((TOTAL_FILES++))
+        if validate_file "$file"; then
+            ((PASSED_FILES++))
+        else
+            ((FAILED_FILES++))
+        fi
+        echo ""
+    fi
+done
+
+# Summary
+log "=================================================="
+log "Validation Summary:"
+log "  Total files: $TOTAL_FILES"
+log "  Passed: $PASSED_FILES"
+log "  Failed: $FAILED_FILES"
+log "  Warnings: $WARNINGS"
+log "=================================================="
+
+if [ $FAILED_FILES -eq 0 ]; then
+    success "All agent identities passed validation!"
+    if [ $WARNINGS -gt 0 ]; then
+        warning "Validation passed with $WARNINGS warning(s)"
+        exit 0
+    fi
+    exit 0
+else
+    error "Validation failed for $FAILED_FILES agent identit(y|ies)"
+    exit 1
+fi
+

scripts/agent-identities/package.json

@@ -0,0 +1,25 @@
+{
+  "name": "@sirius/agent-identity-generator",
+  "version": "1.0.0",
+  "type": "module",
+  "scripts": {
+    "build": "tsc",
+    "generate": "tsx src/generate-agents.ts",
+    "validate": "tsx src/lint-agents.ts",
+    "check-sync": "tsx src/check-sync.ts"
+  },
+  "devDependencies": {
+    "@types/node": "^20.0.0",
+    "@types/js-yaml": "^4.0.9",
+    "typescript": "^5.3.0",
+    "tsx": "^4.7.0"
+  },
+  "dependencies": {
+    "js-yaml": "^4.1.0",
+    "chalk": "^5.3.0",
+    "glob": "^10.3.0",
+    "gray-matter": "^4.0.3"
+  }
+}
+
+

scripts/agent-identities/src/check-sync.ts

@@ -0,0 +1,50 @@
+import { glob } from 'glob';
+import chalk from 'chalk';
+import { checkIfStale } from './validators/sync-validator.js';
+
+async function main() {
+  const agentFiles = await glob('.cursor/agents/*.agent.md', { cwd: '../../' });
+
+  console.log(chalk.blue('\n🔍 Agent Identity Sync Check\n'));
+
+  let staleCount = 0;
+
+  for (const agentFile of agentFiles) {
+    const fullPath = `../../${agentFile}`;
+    const result = await checkIfStale(fullPath);
+
+    if (result.isStale) {
+      console.log(chalk.yellow(`⚠️  ${agentFile} is stale`));
+      console.log(chalk.yellow(`   Reason: ${result.reason}`));
+      if (result.staleFiles && result.staleFiles.length > 0) {
+        console.log(chalk.yellow(`   Modified files:`));
+        result.staleFiles.forEach(file => console.log(chalk.yellow(`     - ${file}`)));
+      }
+      console.log();
+      staleCount++;
+    } else {
+      console.log(chalk.green(`✅ ${agentFile} is in sync`));
+    }
+  }
+
+  console.log(chalk.blue(`\n📊 Summary:`));
+  console.log(`  Total files: ${agentFiles.length}`);
+  console.log(`  In sync: ${agentFiles.length - staleCount}`);
+  console.log(`  Stale: ${staleCount}`);
+
+  if (staleCount > 0) {
+    console.log(chalk.yellow(`\n⚠️  ${staleCount} agent identit${staleCount === 1 ? 'y' : 'ies'} need regeneration`));
+    console.log(chalk.blue('   Run: make regenerate-agents\n'));
+    process.exit(1);
+  } else {
+    console.log(chalk.green('\n✅ All agent identities are in sync!\n'));
+    process.exit(0);
+  }
+}
+
+main().catch(error => {
+  console.error(chalk.red(`\n❌ Fatal error: ${error}\n`));
+  process.exit(1);
+});
+
+

scripts/agent-identities/src/generate-agents.ts

@@ -0,0 +1,75 @@
+import { generateAgentIdentity } from "./generators/index.js";
+import { glob } from "glob";
+import chalk from "chalk";
+import { fileURLToPath } from "url";
+import { dirname } from "path";
+
+const __filename = fileURLToPath(import.meta.url);
+const __dirname = dirname(__filename);
+
+async function main() {
+  const args = process.argv.slice(2);
+
+  if (args.includes("--help")) {
+    console.log(`
+${chalk.blue("🤖 Agent Identity Generator")}
+
+Usage: npm run generate [options]
+
+Options:
+  --all              Generate all agent identities
+  --product=<name>   Generate specific product (e.g., app-agent)
+  --help             Show this help
+
+Examples:
+  npm run generate -- --all
+  npm run generate -- --product=agent-engineer
+    `);
+    return;
+  }
+
+  console.log(chalk.blue("\n🤖 Agent Identity Generation\n"));
+
+  try {
+    if (args.includes("--all")) {
+      // Find config files relative to the project root (2 levels up from src/)
+      const configs = await glob("../../.cursor/agents/config/*.config.yaml");
+
+      if (configs.length === 0) {
+        console.log(
+          chalk.yellow("⚠️  No config files found in .cursor/agents/config/")
+        );
+        return;
+      }
+
+      console.log(chalk.blue(`Found ${configs.length} config file(s)\n`));
+
+      for (const config of configs) {
+        await generateAgentIdentity(config);
+      }
+
+      console.log(
+        chalk.green("\n✅ All agent identities generated successfully!\n")
+      );
+    } else {
+      const product = args
+        .find((arg) => arg.startsWith("--product="))
+        ?.split("=")[1];
+      if (!product) {
+        console.error(chalk.red("❌ Error: Specify --product=<name> or --all"));
+        console.log(chalk.blue("Run with --help for usage information"));
+        process.exit(1);
+      }
+
+      const configPath = `../../.cursor/agents/config/${product}.config.yaml`;
+      await generateAgentIdentity(configPath);
+
+      console.log(chalk.green("\n✅ Agent identity generated successfully!\n"));
+    }
+  } catch (error) {
+    console.error(chalk.red(`\n❌ Error: ${error}\n`));
+    process.exit(1);
+  }
+}
+
+main();

scripts/agent-identities/src/generators/config-examples.ts

@@ -0,0 +1,31 @@
+import { readFileContent } from '../utils/file-reader.js';
+import { join } from 'path';
+import { codeBlock } from '../utils/markdown-builder.js';
+
+export async function extractConfigExamples(
+  productPath: string,
+  configFiles: Array<{ path: string; title: string }>
+): Promise<string> {
+  let result = '';
+
+  for (const configFile of configFiles) {
+    try {
+      const fullPath = join(productPath, configFile.path);
+      const content = await readFileContent(fullPath);
+      
+      // Determine language from file extension
+      const ext = configFile.path.split('.').pop();
+      const language = ext === 'yaml' || ext === 'yml' ? 'yaml' : ext === 'json' ? 'json' : '';
+      
+      result += `**${configFile.title}** (\`${configFile.path}\`):\n\n`;
+      result += codeBlock(content, language);
+      result += '\n\n';
+    } catch (error) {
+      result += `**${configFile.title}:** Error reading file - ${error}\n\n`;
+    }
+  }
+
+  return result.trim();
+}
+
+

scripts/agent-identities/src/generators/dependencies.ts

@@ -0,0 +1,119 @@
+import { readFileContent } from '../utils/file-reader.js';
+import { join } from 'path';
+
+export async function extractGoDependencies(
+  productPath: string,
+  grouping: Record<string, string[]> = {}
+): Promise<string> {
+  try {
+    const goModPath = join(productPath, 'go.mod');
+    const content = await readFileContent(goModPath);
+    
+    // Parse go.mod file
+    const requireRegex = /require\s+\(([^)]+)\)/s;
+    const match = content.match(requireRegex);
+    
+    if (!match) {
+      return 'No dependencies found in go.mod';
+    }
+
+    const requireBlock = match[1];
+    const dependencies: Array<{ name: string; version: string }> = [];
+    
+    // Parse each dependency line
+    const lines = requireBlock.split('\n');
+    for (const line of lines) {
+      const trimmed = line.trim();
+      if (!trimmed || trimmed.startsWith('//')) continue;
+      
+      const parts = trimmed.split(/\s+/);
+      if (parts.length >= 2) {
+        dependencies.push({
+          name: parts[0],
+          version: parts[1],
+        });
+      }
+    }
+
+    // Group dependencies if grouping is provided
+    if (Object.keys(grouping).length > 0) {
+      let result = '';
+      
+      for (const [category, patterns] of Object.entries(grouping)) {
+        result += `**${category}:**\n\n`;
+        
+        const categoryDeps = dependencies.filter(dep =>
+          patterns.some(pattern => dep.name.includes(pattern))
+        );
+        
+        for (const dep of categoryDeps) {
+          result += `- \`${dep.name}\`\n`;
+        }
+        
+        result += '\n';
+      }
+      
+      return result.trim();
+    }
+
+    // No grouping - return all dependencies
+    let result = '**Dependencies:**\n\n';
+    for (const dep of dependencies) {
+      result += `- \`${dep.name}\` (${dep.version})\n`;
+    }
+    
+    return result.trim();
+  } catch (error) {
+    return `Error extracting Go dependencies: ${error}`;
+  }
+}
+
+export async function extractNodeDependencies(
+  productPath: string,
+  grouping: Record<string, string[]> = {}
+): Promise<string> {
+  try {
+    const packageJsonPath = join(productPath, 'package.json');
+    const content = await readFileContent(packageJsonPath);
+    const packageJson = JSON.parse(content);
+    
+    const deps = packageJson.dependencies || {};
+    
+    if (Object.keys(deps).length === 0) {
+      return 'No dependencies found in package.json';
+    }
+
+    // Group dependencies if grouping is provided
+    if (Object.keys(grouping).length > 0) {
+      let result = '';
+      
+      for (const [category, patterns] of Object.entries(grouping)) {
+        result += `**${category}:**\n\n`;
+        
+        const categoryDeps = Object.entries(deps).filter(([name]) =>
+          patterns.some(pattern => name.includes(pattern))
+        );
+        
+        for (const [name, version] of categoryDeps) {
+          result += `- \`${name}\` (${version})\n`;
+        }
+        
+        result += '\n';
+      }
+      
+      return result.trim();
+    }
+
+    // No grouping - return all dependencies
+    let result = '**Dependencies:**\n\n';
+    for (const [name, version] of Object.entries(deps)) {
+      result += `- \`${name}\` (${version})\n`;
+    }
+    
+    return result.trim();
+  } catch (error) {
+    return `Error extracting Node dependencies: ${error}`;
+  }
+}
+
+

scripts/agent-identities/src/generators/documentation.ts

@@ -0,0 +1,64 @@
+import { readFileContent } from '../utils/file-reader.js';
+
+const PATTERN_SECTION_TITLES = [
+  'Code Patterns',
+  'Best Practices',
+  'Common Patterns',
+  'Development Workflow',
+  'Common Tasks',
+  'Troubleshooting',
+];
+
+function extractSection(content: string, sectionTitle: string): string | null {
+  // Try to find the section with various heading levels
+  const patterns = [
+    new RegExp(`## ${sectionTitle}\\s*\\n([\\s\\S]*?)(?=\\n## |\\n# |$)`, 'i'),
+    new RegExp(`### ${sectionTitle}\\s*\\n([\\s\\S]*?)(?=\\n### |\\n## |\\n# |$)`, 'i'),
+  ];
+
+  for (const pattern of patterns) {
+    const match = content.match(pattern);
+    if (match && match[1]) {
+      return match[1].trim();
+    }
+  }
+
+  return null;
+}
+
+export async function extractCodePatterns(docPaths: string[]): Promise<string> {
+  let result = '';
+
+  for (const docPath of docPaths) {
+    try {
+      const content = await readFileContent(docPath);
+      
+      // Try to extract each pattern section
+      for (const sectionTitle of PATTERN_SECTION_TITLES) {
+        const section = extractSection(content, sectionTitle);
+        if (section) {
+          result += `### ${sectionTitle}\n\n${section}\n\n`;
+        }
+      }
+    } catch (error) {
+      console.error(`Error reading doc ${docPath}:`, error);
+    }
+  }
+
+  return result.trim() || 'No code patterns found in documentation';
+}
+
+export async function extractDocumentationLinks(
+  relatedDocs: string[]
+): Promise<string> {
+  let result = '';
+  
+  for (const doc of relatedDocs) {
+    const docName = doc.replace(/^.*\//, '').replace(/\.md$/, '');
+    result += `- [${docName}](mdc:documentation/dev/${doc})\n`;
+  }
+  
+  return result.trim();
+}
+
+

scripts/agent-identities/src/generators/file-structure.ts

@@ -0,0 +1,142 @@
+import { readdir, stat } from 'fs/promises';
+import { join, relative, basename } from 'path';
+import { FileNode } from '../types/extraction.js';
+
+const FILE_COMMENTS: Record<string, string> = {
+  'main.go': 'Main application entry point',
+  'server.go': 'Server implementation',
+  'agent.go': 'Agent client implementation',
+  'types.go': 'Type definitions',
+  'config.yaml': 'Configuration file',
+  'docker-compose.yaml': 'Docker Compose configuration',
+  'package.json': 'Node.js package manifest',
+  'go.mod': 'Go module definition',
+  'README.md': 'Project documentation',
+  'Makefile': 'Build automation',
+};
+
+const DIRECTORY_COMMENTS: Record<string, string> = {
+  'cmd': 'Main applications',
+  'internal': 'Private application code',
+  'pkg': 'Public library code',
+  'src': 'Source code',
+  'proto': 'Protocol buffer definitions',
+  'config': 'Configuration files',
+  'templates': 'Template files',
+  'testing': 'Test files',
+  'scripts': 'Utility scripts',
+  'docs': 'Documentation',
+};
+
+async function buildTree(
+  rootPath: string,
+  currentDepth: number,
+  maxDepth: number,
+  ignorePatterns: string[],
+  basePath: string = rootPath
+): Promise<FileNode[]> {
+  if (currentDepth >= maxDepth) {
+    return [];
+  }
+
+  try {
+    const entries = await readdir(rootPath, { withFileTypes: true });
+    const nodes: FileNode[] = [];
+
+    for (const entry of entries) {
+      const fullPath = join(rootPath, entry.name);
+      
+      // Skip ignored patterns
+      if (ignorePatterns.some(pattern => entry.name.includes(pattern))) {
+        continue;
+      }
+
+      // Skip hidden files/directories
+      if (entry.name.startsWith('.') && entry.name !== '.cursor') {
+        continue;
+      }
+
+      const node: FileNode = {
+        name: entry.name,
+        path: relative(basePath, fullPath),
+        type: entry.isDirectory() ? 'directory' : 'file',
+      };
+
+      // Add comment if available
+      if (entry.isDirectory()) {
+        node.comment = DIRECTORY_COMMENTS[entry.name];
+      } else {
+        node.comment = FILE_COMMENTS[entry.name];
+      }
+
+      // Recursively build children for directories
+      if (entry.isDirectory()) {
+        node.children = await buildTree(
+          fullPath,
+          currentDepth + 1,
+          maxDepth,
+          ignorePatterns,
+          basePath
+        );
+      }
+
+      nodes.push(node);
+    }
+
+    // Sort: directories first, then files, alphabetically
+    nodes.sort((a, b) => {
+      if (a.type !== b.type) {
+        return a.type === 'directory' ? -1 : 1;
+      }
+      return a.name.localeCompare(b.name);
+    });
+
+    return nodes;
+  } catch (error) {
+    console.error(`Error reading directory ${rootPath}:`, error);
+    return [];
+  }
+}
+
+function formatTreeNode(node: FileNode, indent: string = '', isLast: boolean = true): string {
+  const prefix = isLast ? '└── ' : '├── ';
+  const name = node.type === 'directory' ? `${node.name}/` : node.name;
+  const comment = node.comment ? `  # ${node.comment}` : '';
+  
+  let result = `${indent}${prefix}${name}${comment}\n`;
+
+  if (node.children && node.children.length > 0) {
+    const childIndent = indent + (isLast ? '    ' : '│   ');
+    node.children.forEach((child, index) => {
+      const isLastChild = index === node.children!.length - 1;
+      result += formatTreeNode(child, childIndent, isLastChild);
+    });
+  }
+
+  return result;
+}
+
+function formatAsMarkdown(nodes: FileNode[], rootName: string): string {
+  let result = '```\n';
+  result += `${rootName}/\n`;
+  
+  nodes.forEach((node, index) => {
+    const isLast = index === nodes.length - 1;
+    result += formatTreeNode(node, '', isLast);
+  });
+  
+  result += '```';
+  return result;
+}
+
+export async function extractFileStructure(
+  rootPath: string,
+  maxDepth: number = 3,
+  ignorePatterns: string[] = ['node_modules', 'dist', '.git', 'bin', '__pycache__']
+): Promise<string> {
+  const tree = await buildTree(rootPath, 0, maxDepth, ignorePatterns);
+  const rootName = basename(rootPath);
+  return formatAsMarkdown(tree, rootName);
+}
+
+

scripts/agent-identities/src/generators/index.ts

@@ -0,0 +1,152 @@
+import { readFile, writeFile } from 'fs/promises';
+import matter from 'gray-matter';
+import { AgentIdentityConfig, GeneratedSection } from '../types/agent-identity.js';
+import { parseYamlFile } from '../utils/yaml-parser.js';
+import { logger } from '../utils/logger.js';
+import { extractFileStructure } from './file-structure.js';
+import { extractPorts } from './ports.js';
+import { extractGoDependencies, extractNodeDependencies } from './dependencies.js';
+import { extractConfigExamples } from './config-examples.js';
+import { extractCodePatterns, extractDocumentationLinks } from './documentation.js';
+import { mergeTemplate } from './merge.js';
+import { join } from 'path';
+
+async function loadConfig(configPath: string): Promise<AgentIdentityConfig> {
+  return await parseYamlFile(configPath);
+}
+
+export async function generateAgentIdentity(configPath: string): Promise<void> {
+  logger.info(`Loading configuration from ${configPath}`);
+  
+  // 1. Load configuration
+  const config = await loadConfig(configPath);
+
+  // 2. Load template
+  logger.info(`Loading template from ${config.template_path}`);
+  const templateContent = await readFile(config.template_path, 'utf8');
+  const { data: frontMatter, content: templateBody } = matter(templateContent);
+
+  // 3. Extract data from sources
+  const generatedSections: GeneratedSection[] = [];
+
+  // 3.1 File structure
+  if (config.generation.include_file_structure) {
+    logger.info('Extracting file structure...');
+    const fileStructure = await extractFileStructure(
+      config.product_path,
+      config.generation.file_structure_depth || 3,
+      config.generation.file_structure_ignore || []
+    );
+    generatedSections.push({
+      name: 'file-structure',
+      content: fileStructure,
+      source_files: [config.product_path],
+      last_generated: new Date(),
+    });
+  }
+
+  // 3.2 Ports
+  if (config.generation.include_ports && config.docker_service_name) {
+    logger.info('Extracting ports...');
+    const ports = await extractPorts(
+      'docker-compose.yaml',
+      config.docker_service_name,
+      config.generation.ports_config || {}
+    );
+    generatedSections.push({
+      name: 'ports',
+      content: ports,
+      source_files: ['docker-compose.yaml'],
+      last_generated: new Date(),
+    });
+  }
+
+  // 3.3 Dependencies
+  if (config.generation.include_dependencies) {
+    logger.info('Extracting dependencies...');
+    const dependencySource = config.generation.dependencies_source || 'go.mod';
+    let dependencies: string;
+
+    if (dependencySource === 'go.mod') {
+      dependencies = await extractGoDependencies(
+        config.product_path,
+        config.generation.dependencies_grouping || {}
+      );
+    } else if (dependencySource === 'package.json') {
+      dependencies = await extractNodeDependencies(
+        config.product_path,
+        config.generation.dependencies_grouping || {}
+      );
+    } else {
+      dependencies = `Unknown dependency source: ${dependencySource}`;
+    }
+
+    generatedSections.push({
+      name: 'dependencies',
+      content: dependencies,
+      source_files: [join(config.product_path, dependencySource)],
+      last_generated: new Date(),
+    });
+  }
+
+  // 3.4 Config examples
+  if (config.generation.include_config_examples && config.generation.config_files) {
+    logger.info('Extracting config examples...');
+    const configExamples = await extractConfigExamples(
+      config.product_path,
+      config.generation.config_files
+    );
+    generatedSections.push({
+      name: 'config-examples',
+      content: configExamples,
+      source_files: config.generation.config_files.map(cf => join(config.product_path, cf.path)),
+      last_generated: new Date(),
+    });
+  }
+
+  // 3.5 Code patterns from documentation
+  if (config.generation.extract_code_patterns_from_docs.length > 0) {
+    logger.info('Extracting code patterns from documentation...');
+    const codePatterns = await extractCodePatterns(
+      config.generation.extract_code_patterns_from_docs
+    );
+    generatedSections.push({
+      name: 'code-patterns',
+      content: codePatterns,
+      source_files: config.generation.extract_code_patterns_from_docs,
+      last_generated: new Date(),
+    });
+  }
+
+  // 3.6 Documentation links
+  if (config.metadata.related_docs.length > 0) {
+    logger.info('Generating documentation links...');
+    const docLinks = await extractDocumentationLinks(config.metadata.related_docs);
+    generatedSections.push({
+      name: 'documentation-links',
+      content: docLinks,
+      source_files: [],
+      last_generated: new Date(),
+    });
+  }
+
+  // 4. Merge template with generated sections
+  logger.info('Merging template with generated sections...');
+  const finalContent = mergeTemplate(templateBody, generatedSections);
+
+  // 5. Update front matter with generation metadata
+  const updatedFrontMatter = {
+    ...config.metadata,
+    last_updated: new Date().toISOString().split('T')[0],
+    _generated_at: new Date().toISOString(),
+    _source_files: generatedSections.flatMap(s => s.source_files),
+  };
+
+  // 6. Write output
+  const output = matter.stringify(finalContent, updatedFrontMatter);
+  await writeFile(config.output_path, output, 'utf8');
+
+  logger.success(`Generated ${config.output_path}`);
+}
+
+

scripts/agent-identities/src/generators/merge.ts

@@ -0,0 +1,40 @@
+import { GeneratedSection } from '../types/agent-identity.js';
+
+function escapeRegex(str: string): string {
+  return str.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
+}
+
+export function mergeTemplate(
+  template: string,
+  generatedSections: GeneratedSection[]
+): string {
+  let result = template;
+
+  // Find all <!-- AUTO-GENERATED: section-name --> blocks
+  // Replace content between markers with generated content
+  // Preserve manual sections
+
+  for (const section of generatedSections) {
+    const startMarker = `<!-- AUTO-GENERATED: ${section.name} -->`;
+    const endMarker = `<!-- END AUTO-GENERATED -->`;
+
+    const pattern = new RegExp(
+      `${escapeRegex(startMarker)}[\\s\\S]*?${escapeRegex(endMarker)}`,
+      'g'
+    );
+
+    const replacement =
+      `${startMarker}\n` +
+      `<!-- Generated: ${section.last_generated.toISOString()} -->\n` +
+      `<!-- Sources: ${section.source_files.join(', ')} -->\n\n` +
+      section.content +
+      '\n' +
+      endMarker;
+
+    result = result.replace(pattern, replacement);
+  }
+
+  return result;
+}
+
+

scripts/agent-identities/src/generators/ports.ts

@@ -0,0 +1,49 @@
+import { parseYamlFile } from '../utils/yaml-parser.js';
+
+interface DockerComposeService {
+  ports?: string[];
+  [key: string]: any;
+}
+
+interface DockerCompose {
+  services: Record<string, DockerComposeService>;
+}
+
+export async function extractPorts(
+  dockerComposePath: string,
+  serviceName: string,
+  portsConfig: Record<string, string> = {}
+): Promise<string> {
+  try {
+    const compose = await parseYamlFile(dockerComposePath) as DockerCompose;
+    
+    if (!compose.services || !compose.services[serviceName]) {
+      return `Service "${serviceName}" not found in docker-compose.yaml`;
+    }
+
+    const service = compose.services[serviceName];
+    
+    if (!service.ports || service.ports.length === 0) {
+      return 'No ports configured for this service';
+    }
+
+    // Format ports section
+    let result = '**Server Ports:**\n\n';
+    
+    for (const portMapping of service.ports) {
+      // Parse port mapping (e.g., "50051:50051" or "8080:80")
+      const match = portMapping.match(/(\d+):(\d+)/);
+      if (match) {
+        const [, hostPort, containerPort] = match;
+        const description = portsConfig[hostPort] || portsConfig[containerPort] || '';
+        result += `- \`${hostPort}\` - ${description || `Maps to container port ${containerPort}`}\n`;
+      }
+    }
+
+    return result.trim();
+  } catch (error) {
+    return `Error extracting ports: ${error}`;
+  }
+}
+
+

scripts/agent-identities/src/lint-agents.ts

@@ -0,0 +1,87 @@
+import { glob } from 'glob';
+import chalk from 'chalk';
+import { validateYaml, validateContent, checkIfStale, validateFilePaths } from './validators/index.js';
+
+async function main() {
+  const agentFiles = await glob('.cursor/agents/*.agent.md', { cwd: '../../' });
+
+  let errors = 0;
+  let warnings = 0;
+
+  console.log(chalk.blue('\n🤖 Agent Identity Validation\n'));
+
+  for (const agentFile of agentFiles) {
+    const fullPath = `../../${agentFile}`;
+    console.log(chalk.blue(`\nValidating ${agentFile}...`));
+
+    // YAML validation
+    const yamlResult = await validateYaml(fullPath);
+    if (!yamlResult.valid) {
+      console.log(chalk.red(`  ❌ YAML errors:`));
+      yamlResult.errors.forEach(err => console.log(chalk.red(`     - ${err}`)));
+      errors++;
+    }
+
+    // Content validation
+    const contentResult = await validateContent(fullPath);
+    if (!contentResult.valid) {
+      console.log(chalk.red(`  ❌ Content errors:`));
+      contentResult.errors.forEach(err => console.log(chalk.red(`     - ${err}`)));
+      errors++;
+    }
+    if (contentResult.warnings.length > 0) {
+      console.log(chalk.yellow(`  ⚠️  Content warnings:`));
+      contentResult.warnings.forEach(warn => console.log(chalk.yellow(`     - ${warn}`)));
+      warnings++;
+    }
+
+    // Sync validation
+    const syncResult = await checkIfStale(fullPath);
+    if (syncResult.isStale) {
+      console.log(chalk.yellow(`  ⚠️  Stale: ${syncResult.reason}`));
+      if (syncResult.staleFiles && syncResult.staleFiles.length > 0) {
+        console.log(chalk.yellow(`      Modified files:`));
+        syncResult.staleFiles.forEach(file => console.log(chalk.yellow(`        - ${file}`)));
+      }
+      console.log(chalk.yellow(`      Run: make regenerate-agents`));
+      warnings++;
+    }
+
+    // File path validation
+    const pathResult = await validateFilePaths(fullPath);
+    if (!pathResult.valid && pathResult.missingFiles.length > 0) {
+      console.log(chalk.yellow(`  ⚠️  Missing files:`));
+      pathResult.missingFiles.slice(0, 5).forEach(file =>
+        console.log(chalk.yellow(`     - ${file}`))
+      );
+      if (pathResult.missingFiles.length > 5) {
+        console.log(chalk.yellow(`     ... and ${pathResult.missingFiles.length - 5} more`));
+      }
+      warnings++;
+    }
+
+    if (yamlResult.valid && contentResult.valid) {
+      console.log(chalk.green(`  ✅ Valid`));
+    }
+  }
+
+  console.log(chalk.blue(`\n📊 Summary:`));
+  console.log(`  Total files: ${agentFiles.length}`);
+  console.log(`  Errors: ${errors}`);
+  console.log(`  Warnings: ${warnings}`);
+
+  if (errors === 0) {
+    console.log(chalk.green('\n✅ All validations passed!\n'));
+  } else {
+    console.log(chalk.red('\n❌ Validation failed with errors\n'));
+  }
+
+  process.exit(errors > 0 ? 1 : 0);
+}
+
+main().catch(error => {
+  console.error(chalk.red(`\n❌ Fatal error: ${error}\n`));
+  process.exit(1);
+});
+
+

scripts/agent-identities/src/types/agent-identity.ts

@@ -0,0 +1,49 @@
+export interface AgentIdentityMetadata {
+  name: string;
+  title: string;
+  description: string;
+  role_type: 'engineering' | 'design' | 'product' | 'operations' | 'qa' | 'documentation';
+  version: string;
+  last_updated: string;
+  author: string;
+  specialization: string[];
+  technology_stack: string[];
+  system_integration_level: 'high' | 'medium' | 'low' | 'none';
+  categories: string[];
+  tags: string[];
+  related_docs: string[];
+  dependencies: string[];
+  llm_context: 'high' | 'medium' | 'low';
+  context_window_target: number;
+}
+
+export interface AgentIdentityConfig {
+  product: string;
+  product_path: string;
+  docker_service_name?: string;
+  metadata: AgentIdentityMetadata;
+  generation: {
+    include_file_structure: boolean;
+    file_structure_depth?: number;
+    file_structure_ignore?: string[];
+    include_ports: boolean;
+    ports_config?: Record<string, string>;
+    include_dependencies: boolean;
+    dependencies_source?: string;
+    dependencies_grouping?: Record<string, string[]>;
+    include_config_examples: boolean;
+    config_files?: Array<{ path: string; title: string }>;
+    extract_code_patterns_from_docs: string[];
+  };
+  template_path: string;
+  output_path: string;
+}
+
+export interface GeneratedSection {
+  name: string;
+  content: string;
+  source_files: string[];
+  last_generated: Date;
+}
+
+

scripts/agent-identities/src/types/extraction.ts

@@ -0,0 +1,21 @@
+export interface FileNode {
+  name: string;
+  path: string;
+  type: 'file' | 'directory';
+  children?: FileNode[];
+  comment?: string;
+}
+
+export interface PortMapping {
+  service: string;
+  ports: Array<{ host: number; container: number; description: string }>;
+}
+
+export interface DependencyInfo {
+  name: string;
+  version?: string;
+  category?: string;
+  description?: string;
+}
+
+

scripts/agent-identities/src/types/template.ts

@@ -0,0 +1,14 @@
+export interface TemplateSection {
+  name: string;
+  type: 'manual' | 'generated';
+  startMarker?: string;
+  endMarker?: string;
+  content: string;
+}
+
+export interface TemplateMetadata {
+  frontMatter: Record<string, any>;
+  sections: TemplateSection[];
+}
+
+

scripts/agent-identities/src/utils/file-reader.ts

@@ -0,0 +1,21 @@
+import { readFile } from 'fs/promises';
+import { resolve } from 'path';
+
+export async function readFileContent(filePath: string): Promise<string> {
+  try {
+    const absolutePath = resolve(filePath);
+    return await readFile(absolutePath, 'utf8');
+  } catch (error) {
+    throw new Error(`Failed to read file ${filePath}: ${error}`);
+  }
+}
+
+export async function readFileIfExists(filePath: string): Promise<string | null> {
+  try {
+    return await readFileContent(filePath);
+  } catch {
+    return null;
+  }
+}
+
+

scripts/agent-identities/src/utils/logger.ts

@@ -0,0 +1,11 @@
+import chalk from 'chalk';
+
+export const logger = {
+  info: (message: string) => console.log(chalk.blue(`ℹ️  ${message}`)),
+  success: (message: string) => console.log(chalk.green(`✅ ${message}`)),
+  warning: (message: string) => console.log(chalk.yellow(`⚠️  ${message}`)),
+  error: (message: string) => console.log(chalk.red(`❌ ${message}`)),
+  debug: (message: string) => console.log(chalk.gray(`🔍 ${message}`)),
+};
+
+

scripts/agent-identities/src/utils/markdown-builder.ts

@@ -0,0 +1,37 @@
+export function codeBlock(content: string, language: string = ''): string {
+  return `\`\`\`${language}\n${content}\n\`\`\``;
+}
+
+export function heading(text: string, level: number = 1): string {
+  return `${'#'.repeat(level)} ${text}`;
+}
+
+export function list(items: string[], ordered: boolean = false): string {
+  return items
+    .map((item, index) =>
+      ordered ? `${index + 1}. ${item}` : `- ${item}`
+    )
+    .join('\n');
+}
+
+export function table(headers: string[], rows: string[][]): string {
+  const headerRow = `| ${headers.join(' | ')} |`;
+  const separatorRow = `| ${headers.map(() => '---').join(' | ')} |`;
+  const dataRows = rows.map(row => `| ${row.join(' | ')} |`).join('\n');
+  
+  return `${headerRow}\n${separatorRow}\n${dataRows}`;
+}
+
+export function link(text: string, url: string): string {
+  return `[${text}](${url})`;
+}
+
+export function bold(text: string): string {
+  return `**${text}**`;
+}
+
+export function italic(text: string): string {
+  return `*${text}*`;
+}
+
+

scripts/agent-identities/src/utils/yaml-parser.ts

@@ -0,0 +1,21 @@
+import yaml from 'js-yaml';
+import { readFileContent } from './file-reader.js';
+
+export async function parseYamlFile(filePath: string): Promise<any> {
+  const content = await readFileContent(filePath);
+  return yaml.load(content);
+}
+
+export function parseYamlString(content: string): any {
+  return yaml.load(content);
+}
+
+export function stringifyYaml(data: any): string {
+  return yaml.dump(data, {
+    indent: 2,
+    lineWidth: -1,
+    noRefs: true,
+  });
+}
+
+

scripts/agent-identities/src/validators/content-validator.ts

@@ -0,0 +1,55 @@
+import { readFileContent } from '../utils/file-reader.js';
+
+const MIN_LINES = 150;
+const MAX_LINES = 2000;
+const TARGET_MIN = 800;
+const TARGET_MAX = 1500;
+
+export async function validateContent(agentPath: string): Promise<{
+  valid: boolean;
+  errors: string[];
+  warnings: string[];
+}> {
+  const errors: string[] = [];
+  const warnings: string[] = [];
+
+  try {
+    const content = await readFileContent(agentPath);
+    const lines = content.split('\n').length;
+
+    // Check line count
+    if (lines < MIN_LINES) {
+      errors.push(`File too short (${lines} lines). Minimum: ${MIN_LINES}`);
+    } else if (lines > MAX_LINES) {
+      errors.push(`File too long (${lines} lines). Maximum: ${MAX_LINES}`);
+    } else if (lines < TARGET_MIN || lines > TARGET_MAX) {
+      warnings.push(`Line count (${lines}) outside target range (${TARGET_MIN}-${TARGET_MAX})`);
+    }
+
+    // Check for required sections
+    const requiredSections = [
+      '# ', // Title
+      '## Key Documentation',
+      '## Project Location',
+      '## Core Responsibilities',
+      '## Technology Stack',
+    ];
+
+    for (const section of requiredSections) {
+      if (!content.includes(section)) {
+        errors.push(`Missing required section: ${section}`);
+      }
+    }
+
+  } catch (error) {
+    errors.push(`Failed to read file: ${error}`);
+  }
+
+  return {
+    valid: errors.length === 0,
+    errors,
+    warnings,
+  };
+}
+
+

scripts/agent-identities/src/validators/file-path-validator.ts

@@ -0,0 +1,68 @@
+import { access } from 'fs/promises';
+import { readFileContent } from '../utils/file-reader.js';
+
+function extractFileReferences(content: string): string[] {
+  const references: string[] = [];
+
+  // Extract mdc: links
+  const mdcPattern = /\(mdc:([^)]+)\)/g;
+  let match;
+  while ((match = mdcPattern.exec(content)) !== null) {
+    references.push(match[1]);
+  }
+
+  // Extract code block file paths (comments indicating source)
+  // e.g., <!-- Source: app-agent/config/server.yaml -->
+  const sourcePattern = /<!--\s*Source:\s*([^-]+?)\s*-->/g;
+  while ((match = sourcePattern.exec(content)) !== null) {
+    const paths = match[1].split(',').map(p => p.trim());
+    references.push(...paths);
+  }
+
+  // Extract direct file path references in backticks
+  // e.g., `app-agent/cmd/server/main.go`
+  const backtickPattern = /`([a-z-]+\/[a-z-/.]+)`/g;
+  while ((match = backtickPattern.exec(content)) !== null) {
+    if (match[1].includes('/') && match[1].includes('.')) {
+      references.push(match[1]);
+    }
+  }
+
+  return [...new Set(references)]; // Remove duplicates
+}
+
+export async function validateFilePaths(agentPath: string): Promise<{
+  valid: boolean;
+  missingFiles: string[];
+}> {
+  try {
+    // 1. Parse agent identity content
+    const content = await readFileContent(agentPath);
+
+    // 2. Extract all file/directory references
+    const fileReferences = extractFileReferences(content);
+
+    // 3. Check each file exists
+    const missingFiles: string[] = [];
+
+    for (const filePath of fileReferences) {
+      try {
+        await access(filePath);
+      } catch {
+        missingFiles.push(filePath);
+      }
+    }
+
+    return {
+      valid: missingFiles.length === 0,
+      missingFiles,
+    };
+  } catch (error) {
+    return {
+      valid: false,
+      missingFiles: [`Error validating file paths: ${error}`],
+    };
+  }
+}
+
+

scripts/agent-identities/src/validators/index.ts

@@ -0,0 +1,6 @@
+export { validateYaml } from './yaml-validator.js';
+export { validateContent } from './content-validator.js';
+export { checkIfStale } from './sync-validator.js';
+export { validateFilePaths } from './file-path-validator.js';
+
+

scripts/agent-identities/src/validators/sync-validator.ts

@@ -0,0 +1,60 @@
+import { stat } from 'fs/promises';
+import { readFileContent } from '../utils/file-reader.js';
+import matter from 'gray-matter';
+
+export async function checkIfStale(agentPath: string): Promise<{
+  isStale: boolean;
+  reason?: string;
+  staleFiles?: string[];
+}> {
+  try {
+    // 1. Parse agent identity
+    const content = await readFileContent(agentPath);
+    const { data: frontMatter } = matter(content);
+
+    // 2. Get source files from metadata
+    const sourceFiles: string[] = frontMatter._source_files || [];
+    const generatedAt = frontMatter._generated_at
+      ? new Date(frontMatter._generated_at)
+      : new Date(0); // If no generation date, consider it stale
+
+    if (sourceFiles.length === 0) {
+      return {
+        isStale: true,
+        reason: 'No source files tracked (not generated or old format)',
+      };
+    }
+
+    // 3. Check if any source file is newer than generation time
+    const staleFiles: string[] = [];
+
+    for (const sourceFile of sourceFiles) {
+      try {
+        const stats = await stat(sourceFile);
+        if (stats.mtime > generatedAt) {
+          staleFiles.push(sourceFile);
+        }
+      } catch (err) {
+        // File doesn't exist or can't be read
+        staleFiles.push(`${sourceFile} (missing)`);
+      }
+    }
+
+    if (staleFiles.length > 0) {
+      return {
+        isStale: true,
+        reason: 'Source files modified since last generation',
+        staleFiles,
+      };
+    }
+
+    return { isStale: false };
+  } catch (error) {
+    return {
+      isStale: true,
+      reason: `Error checking staleness: ${error}`,
+    };
+  }
+}
+
+

scripts/agent-identities/src/validators/yaml-validator.ts

@@ -0,0 +1,65 @@
+import { readFileContent } from '../utils/file-reader.js';
+import matter from 'gray-matter';
+
+const REQUIRED_FIELDS = [
+  'name',
+  'title',
+  'description',
+  'role_type',
+  'version',
+  'last_updated',
+  'llm_context',
+  'context_window_target',
+];
+
+const ENUM_FIELDS: Record<string, string[]> = {
+  role_type: ['engineering', 'design', 'product', 'operations', 'qa', 'documentation'],
+  system_integration_level: ['high', 'medium', 'low', 'none'],
+  llm_context: ['high', 'medium', 'low'],
+};
+
+export async function validateYaml(agentPath: string): Promise<{
+  valid: boolean;
+  errors: string[];
+}> {
+  const errors: string[] = [];
+
+  try {
+    const content = await readFileContent(agentPath);
+    const { data: frontMatter } = matter(content);
+
+    // Check required fields
+    for (const field of REQUIRED_FIELDS) {
+      if (!(field in frontMatter)) {
+        errors.push(`Missing required field: ${field}`);
+      }
+    }
+
+    // Check enum fields
+    for (const [field, validValues] of Object.entries(ENUM_FIELDS)) {
+      if (field in frontMatter && !validValues.includes(frontMatter[field])) {
+        errors.push(`Invalid value for ${field}: ${frontMatter[field]}. Must be one of: ${validValues.join(', ')}`);
+      }
+    }
+
+    // Check version format (semver)
+    if (frontMatter.version && !/^\d+\.\d+\.\d+$/.test(frontMatter.version)) {
+      errors.push(`Invalid version format: ${frontMatter.version}. Must be semver (e.g., 1.0.0)`);
+    }
+
+    // Check date format (YYYY-MM-DD)
+    if (frontMatter.last_updated && !/^\d{4}-\d{2}-\d{2}$/.test(frontMatter.last_updated)) {
+      errors.push(`Invalid date format: ${frontMatter.last_updated}. Must be YYYY-MM-DD`);
+    }
+
+  } catch (error) {
+    errors.push(`Failed to parse YAML: ${error}`);
+  }
+
+  return {
+    valid: errors.length === 0,
+    errors,
+  };
+}
+
+

scripts/agent-identities/tsconfig.json

@@ -0,0 +1,16 @@
+{
+  "compilerOptions": {
+    "target": "ES2022",
+    "module": "ESNext",
+    "moduleResolution": "bundler",
+    "esModuleInterop": true,
+    "strict": true,
+    "skipLibCheck": true,
+    "outDir": "./dist",
+    "rootDir": "./src"
+  },
+  "include": ["src/**/*"],
+  "exclude": ["node_modules", "dist"]
+}
+
+

scripts/build-containers-with-monitor.sh

@@ -0,0 +1,77 @@
+#!/bin/bash
+
+# Build containers with updated SystemMonitor from main project
+# This script copies the SystemMonitor source to each container directory and builds
+
+set -e
+
+echo "🔨 Building containers with updated SystemMonitor..."
+
+# Get the directory of this script
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
+
+# Paths to the monitoring components
+SYSTEM_MONITOR_DIR="$PROJECT_ROOT/../minor-projects/app-system-monitor"
+
+# Check if main system monitor directory exists
+if [ ! -d "$SYSTEM_MONITOR_DIR" ]; then
+    echo "❌ System monitor directory not found: $SYSTEM_MONITOR_DIR"
+    exit 1
+fi
+
+# Function to copy system monitor to container directory
+copy_system_monitor() {
+    local container_dir="$1"
+    local container_name="$2"
+    
+    echo "📊 Copying SystemMonitor to $container_name..."
+    
+    if [ -d "$container_dir" ]; then
+        # Remove any existing app-system-monitor directory
+        rm -rf "$container_dir/app-system-monitor"
+        
+        # Copy the main system monitor
+        cp -r "$SYSTEM_MONITOR_DIR" "$container_dir/"
+        
+        echo "✅ SystemMonitor copied to $container_name"
+    else
+        echo "⚠️  Container directory not found: $container_dir"
+    fi
+}
+
+# Copy system monitor to each container directory
+copy_system_monitor "$PROJECT_ROOT/sirius-postgres" "sirius-postgres"
+copy_system_monitor "$PROJECT_ROOT/sirius-rabbitmq" "sirius-rabbitmq"
+copy_system_monitor "$PROJECT_ROOT/sirius-valkey" "sirius-valkey"
+
+# Update Dockerfiles to use local copy instead of relative path
+echo "🔧 Updating Dockerfiles to use local SystemMonitor copy..."
+
+# Update PostgreSQL Dockerfile
+sed -i.bak 's|COPY ../../minor-projects/app-system-monitor|COPY app-system-monitor|g' "$PROJECT_ROOT/sirius-postgres/Dockerfile"
+
+# Update RabbitMQ Dockerfile
+sed -i.bak 's|COPY ../../minor-projects/app-system-monitor|COPY app-system-monitor|g' "$PROJECT_ROOT/sirius-rabbitmq/Dockerfile"
+
+# Update Valkey Dockerfile
+sed -i.bak 's|COPY ../../minor-projects/app-system-monitor|COPY app-system-monitor|g' "$PROJECT_ROOT/sirius-valkey/Dockerfile"
+
+# Build the containers
+echo "🚀 Building containers..."
+docker-compose build sirius-postgres sirius-rabbitmq sirius-valkey
+
+# Restore original Dockerfiles
+echo "🔄 Restoring original Dockerfiles..."
+mv "$PROJECT_ROOT/sirius-postgres/Dockerfile.bak" "$PROJECT_ROOT/sirius-postgres/Dockerfile"
+mv "$PROJECT_ROOT/sirius-rabbitmq/Dockerfile.bak" "$PROJECT_ROOT/sirius-rabbitmq/Dockerfile"
+mv "$PROJECT_ROOT/sirius-valkey/Dockerfile.bak" "$PROJECT_ROOT/sirius-valkey/Dockerfile"
+
+# Clean up copied directories
+echo "🧹 Cleaning up temporary SystemMonitor copies..."
+rm -rf "$PROJECT_ROOT/sirius-postgres/app-system-monitor"
+rm -rf "$PROJECT_ROOT/sirius-rabbitmq/app-system-monitor"
+rm -rf "$PROJECT_ROOT/sirius-valkey/app-system-monitor"
+
+echo "✅ Container build completed successfully!"
+echo "🎉 All containers now have the updated SystemMonitor with CPU drift fix!"

scripts/build-system-monitor-universal.sh

@@ -0,0 +1,132 @@
+#!/bin/bash
+
+# Universal SystemMonitor build script
+# This script builds the SystemMonitor binary for all target architectures
+# and creates a universal binary that can be used across all containers
+
+set -e
+
+echo "🔨 Building universal SystemMonitor binary..."
+
+# Get the directory of this script
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
+
+# Paths
+SYSTEM_MONITOR_DIR="$PROJECT_ROOT/../minor-projects/app-system-monitor"
+BUILD_DIR="$PROJECT_ROOT/tmp/system-monitor-build"
+BINARY_DIR="$PROJECT_ROOT/tmp/system-monitor-binaries"
+
+# Clean and create build directories
+rm -rf "$BUILD_DIR" "$BINARY_DIR"
+mkdir -p "$BUILD_DIR" "$BINARY_DIR"
+
+# Copy SystemMonitor source to build directory
+echo "📦 Preparing SystemMonitor source..."
+cp -r "$SYSTEM_MONITOR_DIR"/* "$BUILD_DIR/"
+
+# Build for multiple architectures
+echo "🏗️  Building SystemMonitor for multiple architectures..."
+
+cd "$BUILD_DIR"
+
+# Build for linux/amd64 (most common)
+echo "  → Building for linux/amd64..."
+GOOS=linux GOARCH=amd64 go build -o "$BINARY_DIR/system-monitor-linux-amd64" .
+
+# Build for linux/arm64 (Apple Silicon, ARM servers)
+echo "  → Building for linux/arm64..."
+GOOS=linux GOARCH=arm64 go build -o "$BINARY_DIR/system-monitor-linux-arm64" .
+
+# Create a universal binary script that detects architecture
+echo "🔧 Creating universal binary script..."
+cat > "$BINARY_DIR/system-monitor" << 'EOF'
+#!/bin/bash
+
+# Universal SystemMonitor binary launcher
+# Detects the current architecture and runs the appropriate binary
+
+ARCH=$(uname -m)
+OS=$(uname -s)
+
+# Map architecture names
+case "$ARCH" in
+    x86_64|amd64)
+        BINARY_NAME="system-monitor-linux-amd64"
+        ;;
+    aarch64|arm64)
+        BINARY_NAME="system-monitor-linux-arm64"
+        ;;
+    *)
+        echo "❌ Unsupported architecture: $ARCH"
+        exit 1
+        ;;
+esac
+
+# Get the directory of this script
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+BINARY_PATH="$SCRIPT_DIR/$BINARY_NAME"
+
+# Check if the appropriate binary exists
+if [ ! -f "$BINARY_PATH" ]; then
+    echo "❌ SystemMonitor binary not found for architecture $ARCH: $BINARY_PATH"
+    exit 1
+fi
+
+# Make sure the binary is executable
+chmod +x "$BINARY_PATH"
+
+# Execute the appropriate binary with all arguments
+exec "$BINARY_PATH" "$@"
+EOF
+
+chmod +x "$BINARY_DIR/system-monitor"
+
+# Test the universal binary
+echo "🧪 Testing universal binary..."
+cd "$BINARY_DIR"
+if ./system-monitor --help 2>/dev/null || echo "Binary built successfully (no help flag)"; then
+    echo "✅ Universal SystemMonitor binary is working"
+else
+    echo "❌ Universal SystemMonitor binary test failed"
+    exit 1
+fi
+
+echo "✅ Universal SystemMonitor build completed successfully!"
+echo "📍 Binary location: $BINARY_DIR/system-monitor"
+echo "📦 Architecture-specific binaries:"
+echo "  - linux/amd64: $BINARY_DIR/system-monitor-linux-amd64"
+echo "  - linux/arm64: $BINARY_DIR/system-monitor-linux-arm64"
+
+# Copy to container directories for Docker builds
+echo "📋 Copying binaries to container directories..."
+
+# Copy to sirius-postgres
+if [ -d "$PROJECT_ROOT/sirius-postgres" ]; then
+    echo "  → Copying to sirius-postgres..."
+    rm -rf "$PROJECT_ROOT/sirius-postgres/system-monitor-binary"
+    cp -r "$BINARY_DIR" "$PROJECT_ROOT/sirius-postgres/system-monitor-binary"
+fi
+
+# Copy to sirius-rabbitmq
+if [ -d "$PROJECT_ROOT/sirius-rabbitmq" ]; then
+    echo "  → Copying to sirius-rabbitmq..."
+    rm -rf "$PROJECT_ROOT/sirius-rabbitmq/system-monitor-binary"
+    cp -r "$BINARY_DIR" "$PROJECT_ROOT/sirius-rabbitmq/system-monitor-binary"
+fi
+
+# Copy to sirius-valkey
+if [ -d "$PROJECT_ROOT/sirius-valkey" ]; then
+    echo "  → Copying to sirius-valkey..."
+    rm -rf "$PROJECT_ROOT/sirius-valkey/system-monitor-binary"
+    cp -r "$BINARY_DIR" "$PROJECT_ROOT/sirius-valkey/system-monitor-binary"
+fi
+
+# Copy to sirius-engine
+if [ -d "$PROJECT_ROOT/sirius-engine" ]; then
+    echo "  → Copying to sirius-engine..."
+    rm -rf "$PROJECT_ROOT/sirius-engine/system-monitor-binary"
+    cp -r "$BINARY_DIR" "$PROJECT_ROOT/sirius-engine/system-monitor-binary"
+fi
+
+echo "🎉 Universal SystemMonitor is ready for all containers!"

scripts/build-ui-monitoring.sh

@@ -0,0 +1,45 @@
+#!/bin/bash
+
+# Build script for UI container monitoring components
+# This script builds the system monitor and app administrator for the UI container
+
+set -e
+
+echo "🔧 Building monitoring components for UI container..."
+
+# Get the directory of this script
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
+
+# Paths to the monitoring components
+SYSTEM_MONITOR_DIR="$PROJECT_ROOT/../minor-projects/app-system-monitor"
+ADMINISTRATOR_DIR="$PROJECT_ROOT/../minor-projects/app-administrator"
+
+# Check if directories exist
+if [ ! -d "$SYSTEM_MONITOR_DIR" ]; then
+    echo "❌ System monitor directory not found: $SYSTEM_MONITOR_DIR"
+    exit 1
+fi
+
+if [ ! -d "$ADMINISTRATOR_DIR" ]; then
+    echo "❌ Administrator directory not found: $ADMINISTRATOR_DIR"
+    exit 1
+fi
+
+# Build system monitor
+echo "📊 Building system monitor..."
+cd "$SYSTEM_MONITOR_DIR"
+go mod download
+CGO_ENABLED=0 GOOS=linux go build -ldflags="-w -s" -o system-monitor main.go
+echo "✅ System monitor built successfully"
+
+# Build app administrator
+echo "🔧 Building app administrator..."
+cd "$ADMINISTRATOR_DIR"
+go mod download
+CGO_ENABLED=0 GOOS=linux go build -ldflags="-w -s" -o administrator main.go
+echo "✅ App administrator built successfully"
+
+echo "🎉 All monitoring components built successfully!"
+echo "📁 System monitor: $SYSTEM_MONITOR_DIR/system-monitor"
+echo "📁 App administrator: $ADMINISTRATOR_DIR/administrator"

scripts/cross-platform/find-password-files.sh

@@ -0,0 +1,232 @@
+#!/bin/bash
+
+# Script metadata
+VULNERABILITY_ID="CVE-2024-PASSWORD-001"
+SEVERITY="high"
+DESCRIPTION="Detects insecure password files (passwords.txt) in user home directories"
+AUTHOR="sirius-security-team"
+VERSION="1.0"
+
+# Configuration
+VERBOSE=false
+SPECIFIC_USER=""
+
+# Parse command line arguments
+while [[ $# -gt 0 ]]; do
+    case $1 in
+        --verbose|-v)
+            VERBOSE=true
+            shift
+            ;;
+        --user|-u)
+            SPECIFIC_USER="$2"
+            shift 2
+            ;;
+        --help|-h)
+            echo "Usage: $0 [--verbose] [--user <username>]"
+            echo "  --verbose, -v    Enable verbose output"
+            echo "  --user, -u       Check specific user's home directory"
+            echo "  --help, -h       Show this help message"
+            exit 0
+            ;;
+        *)
+            echo "Unknown option: $1"
+            exit 1
+            ;;
+    esac
+done
+
+# Logging function
+log_verbose() {
+    if [[ "$VERBOSE" == "true" ]]; then
+        echo "[VERBOSE] $1" >&2
+    fi
+}
+
+# Main detection function
+find_password_files() {
+    local vulnerable=false
+    local confidence=0.0
+    local evidence=()
+    local error=""
+    local total_files_found=0
+
+    log_verbose "Starting password file detection"
+
+    # Determine which directories to search
+    local search_dirs=()
+    
+    if [[ -n "$SPECIFIC_USER" ]]; then
+        # Check specific user's home directory
+        if [[ "$OSTYPE" == "msys" || "$OSTYPE" == "win32" ]]; then
+            # Windows
+            local user_home="/c/Users/$SPECIFIC_USER"
+        else
+            # Linux/macOS
+            local user_home="/home/$SPECIFIC_USER"
+            if [[ "$OSTYPE" == "darwin"* ]]; then
+                user_home="/Users/$SPECIFIC_USER"
+            fi
+        fi
+        
+        if [[ -d "$user_home" ]]; then
+            search_dirs+=("$user_home")
+        else
+            error="User home directory not found for user: $SPECIFIC_USER"
+        fi
+    else
+        # Search current user's home directory
+        if [[ -n "$HOME" ]]; then
+            search_dirs+=("$HOME")
+        else
+            error="HOME environment variable not set"
+        fi
+    fi
+
+    if [[ -n "$error" ]]; then
+        cat << EOF
+{
+    "vulnerability_id": "$VULNERABILITY_ID",
+    "vulnerable": null,
+    "confidence": 0.0,
+    "evidence": [],
+    "metadata": {},
+    "error": "$error"
+}
+EOF
+        return
+    fi
+
+    # Search for password files
+    for search_dir in "${search_dirs[@]}"; do
+        log_verbose "Searching directory: $search_dir"
+        
+        # Look for passwords.txt files (case insensitive)
+        local password_files
+        password_files=$(find "$search_dir" -type f -iname "passwords.txt" 2>/dev/null)
+        
+        if [[ -n "$password_files" ]]; then
+            while IFS= read -r password_file; do
+                if [[ -n "$password_file" ]]; then
+                    total_files_found=$((total_files_found + 1))
+                    vulnerable=true
+                    
+                    # Get file details
+                    local file_size
+                    local file_perms
+                    local file_owner
+                    local file_modified
+                    
+                    file_size=$(stat -c%s "$password_file" 2>/dev/null || stat -f%z "$password_file" 2>/dev/null || echo "unknown")
+                    file_perms=$(stat -c%a "$password_file" 2>/dev/null || stat -f%A "$password_file" 2>/dev/null || echo "unknown")
+                    file_owner=$(stat -c%U "$password_file" 2>/dev/null || stat -f%Su "$password_file" 2>/dev/null || echo "unknown")
+                    file_modified=$(stat -c%Y "$password_file" 2>/dev/null || stat -f%m "$password_file" 2>/dev/null || echo "unknown")
+                    
+                    # Check if file is world-readable
+                    local world_readable=false
+                    if [[ "$file_perms" =~ [0-9][0-9][4-7] ]]; then
+                        world_readable=true
+                    fi
+                    
+                    # Build evidence entry
+                    local evidence_entry
+                    evidence_entry=$(cat << EOF
+{
+    "type": "insecure_password_file",
+    "file_path": "$password_file",
+    "file_size": "$file_size",
+    "file_permissions": "$file_perms",
+    "file_owner": "$file_owner",
+    "last_modified": "$file_modified",
+    "world_readable": $world_readable,
+    "risk_level": "$(if [[ "$world_readable" == "true" ]]; then echo "critical"; else echo "high"; fi)"
+}
+EOF
+                    )
+                    
+                    evidence+=("$evidence_entry")
+                    
+                    log_verbose "FOUND: $password_file (size: $file_size, perms: $file_perms, world-readable: $world_readable)"
+                fi
+            done <<< "$password_files"
+        fi
+        
+        # Also look for other common password file patterns
+        local other_patterns=("password.txt" "pass.txt" "passwords.log" "creds.txt" "credentials.txt")
+        for pattern in "${other_patterns[@]}"; do
+            local found_files
+            found_files=$(find "$search_dir" -type f -iname "$pattern" 2>/dev/null)
+            
+            if [[ -n "$found_files" ]]; then
+                while IFS= read -r found_file; do
+                    if [[ -n "$found_file" ]]; then
+                        total_files_found=$((total_files_found + 1))
+                        vulnerable=true
+                        
+                        local file_size
+                        local file_perms
+                        file_size=$(stat -c%s "$found_file" 2>/dev/null || stat -f%z "$found_file" 2>/dev/null || echo "unknown")
+                        file_perms=$(stat -c%a "$found_file" 2>/dev/null || stat -f%A "$found_file" 2>/dev/null || echo "unknown")
+                        
+                        local evidence_entry
+                        evidence_entry=$(cat << EOF
+{
+    "type": "potential_password_file",
+    "file_path": "$found_file",
+    "file_size": "$file_size",
+    "file_permissions": "$file_perms",
+    "risk_level": "medium"
+}
+EOF
+                        )
+                        
+                        evidence+=("$evidence_entry")
+                        
+                        log_verbose "FOUND: $found_file (pattern: $pattern)"
+                    fi
+                done <<< "$found_files"
+            fi
+        done
+    done
+
+    # Calculate confidence
+    if [[ "$vulnerable" == "true" ]]; then
+        if [[ $total_files_found -ge 5 ]]; then
+            confidence=0.95
+        elif [[ $total_files_found -ge 3 ]]; then
+            confidence=0.85
+        elif [[ $total_files_found -ge 2 ]]; then
+            confidence=0.75
+        else
+            confidence=0.65
+        fi
+    fi
+
+    # Format evidence array
+    local evidence_json=""
+    if [[ ${#evidence[@]} -gt 0 ]]; then
+        evidence_json=$(printf '%s,' "${evidence[@]}" | sed 's/,$//')
+    fi
+
+    log_verbose "Password file detection complete: $total_files_found files found"
+
+    # Output JSON result
+    cat << EOF
+{
+    "vulnerability_id": "$VULNERABILITY_ID",
+    "vulnerable": $vulnerable,
+    "confidence": $confidence,
+    "evidence": [$evidence_json],
+    "metadata": {
+        "scan_type": "password_file_detection",
+        "total_files_found": $total_files_found,
+        "search_directories": "$(printf '%s,' "${search_dirs[@]}" | sed 's/,$//')",
+        "searched_user": "$(if [[ -n "$SPECIFIC_USER" ]]; then echo "$SPECIFIC_USER"; else echo "current"; fi)"
+    },
+    "error": null
+}
+EOF
+}
+
+# Execute detection
+find_password_files 

scripts/deploy.sh

@@ -0,0 +1,361 @@
+#!/bin/bash
+
+# Sirius Scan Deployment Script
+# Usage: ./scripts/deploy.sh [environment] [image_tag]
+# Example: ./scripts/deploy.sh production v1.2.3
+
+set -e  # Exit on any error
+
+# Configuration
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_DIR="$(dirname "$SCRIPT_DIR")"
+ENVIRONMENTS_DIR="$PROJECT_DIR/environments"
+
+# Default values
+ENVIRONMENT="${1:-development}"
+IMAGE_TAG="${2:-latest}"
+COMPOSE_PROJECT_NAME="sirius"
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m' # No Color
+
+# Logging functions
+log_info() {
+    echo -e "${BLUE}[INFO]${NC} $1"
+}
+
+log_success() {
+    echo -e "${GREEN}[SUCCESS]${NC} $1"
+}
+
+log_warning() {
+    echo -e "${YELLOW}[WARNING]${NC} $1"
+}
+
+log_error() {
+    echo -e "${RED}[ERROR]${NC} $1"
+}
+
+# Help function
+show_help() {
+    cat << EOF
+Sirius Scan Deployment Script
+
+Usage: $0 [ENVIRONMENT] [IMAGE_TAG]
+
+ENVIRONMENT:
+    development  - Local development (default)
+    staging      - Staging environment
+    production   - Production environment
+
+IMAGE_TAG:
+    Version tag for Docker images (default: latest)
+
+Examples:
+    $0                          # Deploy development with latest images
+    $0 staging                  # Deploy staging with latest images
+    $0 production v1.2.3        # Deploy production with specific version
+    $0 staging beta-2024.01.15  # Deploy staging with beta version
+
+Environment Configuration:
+    All environments use defaults in compose files with override capability:
+    - Development: Uses docker-compose.override.yaml
+    - Staging/Production: Uses sensible defaults with ${VAR:-default} syntax
+    - Override any variable: export POSTGRES_PASSWORD=mypassword
+
+Docker Compose Files:
+    - docker-compose.yaml (base configuration)
+    - docker-compose.override.yaml (development - auto-loaded)
+    - docker-compose.staging.yaml (staging environment)
+    - docker-compose.production.yaml (production environment)
+
+EOF
+}
+
+# Validate environment
+validate_environment() {
+    case "$ENVIRONMENT" in
+        development|staging|production)
+            log_info "Deploying to: $ENVIRONMENT"
+            ;;
+        help|--help|-h)
+            show_help
+            exit 0
+            ;;
+        *)
+            log_error "Invalid environment: $ENVIRONMENT"
+            log_error "Valid environments: development, staging, production"
+            exit 1
+            ;;
+    esac
+}
+
+# Check if environment file exists (now using defaults in compose files)
+check_env_file() {
+    if [[ "$ENVIRONMENT" == "development" ]]; then
+        log_info "Development environment uses docker-compose.override.yaml"
+        return 0
+    fi
+    
+    log_info "Using defaults from compose files (no separate env file needed)"
+    log_info "Override variables with: export POSTGRES_PASSWORD=yourpassword"
+}
+
+# Check if Docker Compose files exist
+check_compose_files() {
+    local base_file="$PROJECT_DIR/docker-compose.yaml"
+    local env_file=""
+    
+    if [[ ! -f "$base_file" ]]; then
+        log_error "Base docker-compose.yaml not found: $base_file"
+        exit 1
+    fi
+    
+    case "$ENVIRONMENT" in
+        development)
+            env_file="$PROJECT_DIR/docker-compose.override.yaml"
+            ;;
+        staging)
+            env_file="$PROJECT_DIR/docker-compose.staging.yaml"
+            ;;
+        production)
+            env_file="$PROJECT_DIR/docker-compose.production.yaml"
+            ;;
+    esac
+    
+    if [[ -n "$env_file" && ! -f "$env_file" ]]; then
+        log_error "Environment-specific compose file not found: $env_file"
+        exit 1
+    fi
+    
+    log_success "Docker Compose files validated"
+}
+
+# Pre-deployment checks
+pre_deployment_checks() {
+    log_info "Running pre-deployment checks..."
+    
+    # Check if Docker is running
+    if ! docker info >/dev/null 2>&1; then
+        log_error "Docker is not running. Please start Docker and try again."
+        exit 1
+    fi
+    
+    # Check if Docker Compose is available
+    if ! command -v docker >/dev/null 2>&1; then
+        log_error "Docker Compose is not installed or not in PATH"
+        exit 1
+    fi
+    
+    # Check available disk space (warn if less than 5GB)
+    local available_space=$(df "$PROJECT_DIR" | awk 'NR==2 {print $4}')
+    if [[ "$available_space" -lt 5242880 ]]; then  # 5GB in KB
+        log_warning "Low disk space detected. Consider cleaning up before deployment."
+    fi
+    
+    log_success "Pre-deployment checks passed"
+}
+
+# Build compose command
+build_compose_command() {
+    local cmd="docker compose"
+    
+    # Add base compose file
+    cmd="$cmd -f docker-compose.yaml"
+    
+    # Add environment-specific compose file
+    case "$ENVIRONMENT" in
+        staging)
+            cmd="$cmd -f docker-compose.staging.yaml"
+            ;;
+        production)
+            cmd="$cmd -f docker-compose.production.yaml"
+            ;;
+        development)
+            # docker-compose.override.yaml is loaded automatically
+            ;;
+    esac
+    
+    # Environment variables are handled with defaults in compose files
+    # No separate env files needed
+    
+    # Set project name
+    cmd="$cmd -p ${COMPOSE_PROJECT_NAME}_${ENVIRONMENT}"
+    
+    echo "$cmd"
+}
+
+# Pull images for production/staging
+pull_images() {
+    if [[ "$ENVIRONMENT" == "development" ]]; then
+        log_info "Skipping image pull for development environment"
+        return 0
+    fi
+    
+    log_info "Pulling Docker images for $ENVIRONMENT..."
+    
+    local compose_cmd=$(build_compose_command)
+    
+    # Export IMAGE_TAG for compose
+    export IMAGE_TAG="$IMAGE_TAG"
+    
+    if $compose_cmd pull; then
+        log_success "Images pulled successfully"
+    else
+        log_error "Failed to pull images"
+        exit 1
+    fi
+}
+
+# Deploy services
+deploy_services() {
+    log_info "Deploying services for $ENVIRONMENT environment..."
+    
+    local compose_cmd=$(build_compose_command)
+    
+    # Export IMAGE_TAG for compose
+    export IMAGE_TAG="$IMAGE_TAG"
+    
+    # Deploy based on environment
+    case "$ENVIRONMENT" in
+        development)
+            log_info "Starting development environment with build..."
+            if $compose_cmd up --build -d; then
+                log_success "Development environment started"
+            else
+                log_error "Failed to start development environment"
+                exit 1
+            fi
+            ;;
+        staging|production)
+            log_info "Starting $ENVIRONMENT environment..."
+            if $compose_cmd up -d; then
+                log_success "$ENVIRONMENT environment started"
+            else
+                log_error "Failed to start $ENVIRONMENT environment"
+                exit 1
+            fi
+            ;;
+    esac
+}
+
+# Health checks
+run_health_checks() {
+    log_info "Running health checks..."
+    
+    local compose_cmd=$(build_compose_command)
+    
+    # Wait for services to be ready
+    sleep 10
+    
+    # Check if all services are running
+    local running_services=$($compose_cmd ps --services --filter "status=running" | wc -l)
+    local total_services=$($compose_cmd ps --services | wc -l)
+    
+    if [[ "$running_services" -eq "$total_services" ]]; then
+        log_success "All services are running ($running_services/$total_services)"
+    else
+        log_warning "Some services may not be running ($running_services/$total_services)"
+        log_info "Service status:"
+        $compose_cmd ps
+    fi
+    
+    # Basic connectivity tests
+    case "$ENVIRONMENT" in
+        development)
+            log_info "Testing local endpoints..."
+            if curl -f http://localhost:3000 >/dev/null 2>&1; then
+                log_success "UI is responding on http://localhost:3000"
+            else
+                log_warning "UI may not be ready yet on http://localhost:3000"
+            fi
+            
+            if curl -f http://localhost:9001 >/dev/null 2>&1; then
+                log_success "API is responding on http://localhost:9001"
+            else
+                log_warning "API may not be ready yet on http://localhost:9001"
+            fi
+            ;;
+    esac
+}
+
+# Show deployment summary
+show_summary() {
+    log_success "Deployment completed successfully!"
+    echo
+    echo "=== Deployment Summary ==="
+    echo "Environment: $ENVIRONMENT"
+    echo "Image Tag: $IMAGE_TAG"
+    echo "Project Name: ${COMPOSE_PROJECT_NAME}_${ENVIRONMENT}"
+    echo
+    
+    case "$ENVIRONMENT" in
+        development)
+            echo "Access URLs:"
+            echo "  UI: http://localhost:3000"
+            echo "  API: http://localhost:9001"
+            echo "  RabbitMQ Management: http://localhost:15672"
+            echo "  PostgreSQL: localhost:5432"
+            echo
+            echo "Useful commands:"
+            echo "  View logs: docker compose logs -f"
+            echo "  Stop services: docker compose down"
+            echo "  Restart: docker compose restart"
+            ;;
+        staging|production)
+            echo "Useful commands:"
+            local compose_cmd=$(build_compose_command)
+            echo "  View logs: $compose_cmd logs -f"
+            echo "  Stop services: $compose_cmd down"
+            echo "  Restart: $compose_cmd restart"
+            echo "  Scale service: $compose_cmd up -d --scale sirius-api=3"
+            ;;
+    esac
+    
+    echo
+    echo "For more information, see: documentation/README.deployment-strategy.md"
+}
+
+# Cleanup on exit
+cleanup() {
+    if [[ $? -ne 0 ]]; then
+        log_error "Deployment failed. Check the logs above for details."
+        echo
+        echo "Troubleshooting tips:"
+        echo "1. Check Docker daemon is running"
+        echo "2. Verify environment file exists and is valid"
+        echo "3. Ensure sufficient disk space"
+        echo "4. Check network connectivity for image pulls"
+    fi
+}
+
+trap cleanup EXIT
+
+# Main execution
+main() {
+    echo "=== Sirius Scan Deployment Script ==="
+    echo
+    
+    validate_environment
+    check_env_file
+    check_compose_files
+    pre_deployment_checks
+    
+    if [[ "$ENVIRONMENT" != "development" ]]; then
+        pull_images
+    fi
+    
+    deploy_services
+    run_health_checks
+    show_summary
+}
+
+# Change to project directory
+cd "$PROJECT_DIR"
+
+# Run main function
+main "$@" 

scripts/dev-setup.sh

@@ -0,0 +1,221 @@
+#!/bin/bash
+
+# Sirius Development Setup Helper
+# This script helps set up local development environment
+
+set -e
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
+LOCAL_COMPOSE="$PROJECT_ROOT/docker-compose.local.yaml"
+EXAMPLE_COMPOSE="$PROJECT_ROOT/docker-compose.local.example.yaml"
+ENV_FILE="$PROJECT_ROOT/.env"
+INSTALLER_COMPOSE_FILE="$PROJECT_ROOT/docker-compose.installer.yaml"
+
+echo "🚀 Sirius Development Setup"
+echo "================================"
+
+# Function to show usage
+show_usage() {
+    echo "Usage: $0 [command]"
+    echo ""
+    echo "Commands:"
+    echo "  init              Create local docker-compose.local.yaml from template"
+    echo "  start             Start development environment"
+    echo "  start-extended    Start with local repository mounts (requires setup)"
+    echo "  stop              Stop development environment" 
+    echo "  clean             Clean development environment and volumes"
+    echo "  status            Show container status"
+    echo "  logs [service]    Show logs for all services or specific service"
+    echo "  shell <service>   Open shell in service container"
+    echo ""
+    echo "Examples:"
+    echo "  $0 init                    # Set up local development files"
+    echo "  $0 start                   # Standard development mode"
+    echo "  $0 start-extended          # Extended development with local repos"
+    echo "  $0 logs sirius-engine      # Show engine logs"
+    echo "  $0 shell sirius-ui         # Open shell in UI container"
+}
+
+# Function to check if local compose file exists
+check_local_compose() {
+    if [ ! -f "$LOCAL_COMPOSE" ]; then
+        echo "⚠️  Local compose file not found: $LOCAL_COMPOSE"
+        echo "💡 Run '$0 init' to create it from template"
+        return 1
+    fi
+    return 0
+}
+
+# Function to initialize local development setup
+init_dev() {
+    echo "📋 Setting up local development environment..."
+    
+    if [ -f "$LOCAL_COMPOSE" ]; then
+        echo "⚠️  $LOCAL_COMPOSE already exists"
+        read -p "Do you want to overwrite it? (y/N): " -n 1 -r
+        echo
+        if [[ ! $REPLY =~ ^[Yy]$ ]]; then
+            echo "ℹ️  Keeping existing file"
+            return 0
+        fi
+    fi
+    
+    if [ ! -f "$EXAMPLE_COMPOSE" ]; then
+        echo "❌ Template file not found: $EXAMPLE_COMPOSE"
+        return 1
+    fi
+    
+    cp "$EXAMPLE_COMPOSE" "$LOCAL_COMPOSE"
+    echo "✅ Created $LOCAL_COMPOSE"
+    echo ""
+    echo "📝 Next steps:"
+    echo "1. Edit $LOCAL_COMPOSE to uncomment the repositories you want to develop"
+    echo "2. Make sure you have the corresponding repositories in ../minor-projects/"
+    echo "3. Run '$0 start-extended' to start with local repository mounts"
+    echo ""
+    echo "🔧 For standard development (no local repos needed): '$0 start'"
+}
+
+# Ensure required env exists before startup.
+ensure_env_ready() {
+    if [ -f "$ENV_FILE" ]; then
+        return 0
+    fi
+
+    echo "⚠️  Missing $ENV_FILE"
+    if [ "${SIRIUS_AUTO_SETUP:-0}" = "1" ]; then
+        echo "🔧 Running installer container in non-interactive mode..."
+        docker compose -f "$INSTALLER_COMPOSE_FILE" run --rm sirius-installer --non-interactive --no-print-secrets
+        return 0
+    fi
+
+    echo "Run installer first:"
+    echo "  docker compose -f docker-compose.installer.yaml run --rm sirius-installer"
+    echo "Or rerun with SIRIUS_AUTO_SETUP=1 for automated setup."
+    return 1
+}
+
+# Function to start development environment
+start_dev() {
+    local mode=$1
+    
+    cd "$PROJECT_ROOT"
+    ensure_env_ready || return 1
+    
+    if [ "$mode" = "extended" ]; then
+        echo "🔧 Starting extended development environment..."
+        if ! check_local_compose; then
+            return 1
+        fi
+        echo "📁 Using local repository mounts from docker-compose.local.yaml"
+        docker compose -f docker-compose.yaml -f docker-compose.override.yaml -f docker-compose.local.yaml up -d
+    else
+        echo "🔧 Starting standard development environment..."
+        docker compose -f docker-compose.yaml -f docker-compose.dev.yaml up -d
+    fi
+    
+    echo ""
+    echo "✅ Development environment started!"
+    echo "🌐 UI: http://localhost:3000"
+    echo "🔧 API: http://localhost:9001"
+    echo "📊 RabbitMQ Management: http://localhost:15672 (guest/guest)"
+    
+    if command -v docker &> /dev/null; then
+        echo ""
+        echo "📋 Container Status:"
+        docker compose ps
+    fi
+}
+
+# Function to stop development environment
+stop_dev() {
+    echo "🛑 Stopping development environment..."
+    cd "$PROJECT_ROOT"
+    docker compose down
+    echo "✅ Development environment stopped"
+}
+
+# Function to clean development environment
+clean_dev() {
+    echo "🧹 Cleaning development environment..."
+    cd "$PROJECT_ROOT"
+    
+    read -p "This will remove all containers and volumes. Continue? (y/N): " -n 1 -r
+    echo
+    if [[ ! $REPLY =~ ^[Yy]$ ]]; then
+        echo "ℹ️  Cleanup cancelled"
+        return 0
+    fi
+    
+    docker compose down -v --remove-orphans
+    docker system prune -f
+    echo "✅ Development environment cleaned"
+}
+
+# Function to show container status
+show_status() {
+    echo "📋 Container Status:"
+    cd "$PROJECT_ROOT"
+    docker compose ps
+}
+
+# Function to show logs
+show_logs() {
+    local service=$1
+    cd "$PROJECT_ROOT"
+    
+    if [ -n "$service" ]; then
+        echo "📜 Showing logs for $service..."
+        docker compose logs -f "$service"
+    else
+        echo "📜 Showing logs for all services..."
+        docker compose logs -f
+    fi
+}
+
+# Function to open shell in container
+open_shell() {
+    local service=$1
+    
+    if [ -z "$service" ]; then
+        echo "❌ Service name required"
+        echo "Available services: sirius-ui, sirius-api, sirius-engine, sirius-postgres, sirius-rabbitmq, sirius-valkey"
+        return 1
+    fi
+    
+    cd "$PROJECT_ROOT"
+    echo "🐚 Opening shell in $service..."
+    docker compose exec "$service" /bin/bash || docker compose exec "$service" /bin/sh
+}
+
+# Main command handling
+case "$1" in
+    "init")
+        init_dev
+        ;;
+    "start")
+        start_dev
+        ;;
+    "start-extended")
+        start_dev extended
+        ;;
+    "stop")
+        stop_dev
+        ;;
+    "clean")
+        clean_dev
+        ;;
+    "status")
+        show_status
+        ;;
+    "logs")
+        show_logs "$2"
+        ;;
+    "shell")
+        open_shell "$2"
+        ;;
+    *)
+        show_usage
+        ;;
+esac 

scripts/documentation/lint-docs.sh

@@ -0,0 +1,400 @@
+#!/bin/bash
+
+# Sirius Documentation Linting System
+# Validates documentation quality, consistency, and compliance
+
+set -e
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m' # No Color
+
+# Configuration
+DOCS_DIR="../../documentation/dev"
+TEMPLATES_DIR="$DOCS_DIR/templates"
+SCRIPT_DIR="$(dirname "$0")"
+LOG_FILE="tmp/documentation-lint-$(date +%Y%m%d-%H%M%S).log"
+
+# Create tmp directory if it doesn't exist
+mkdir -p tmp
+
+# Required YAML front matter fields
+REQUIRED_FIELDS=("title" "description" "template" "version" "last_updated")
+OPTIONAL_FIELDS=("author" "tags" "categories" "difficulty" "prerequisites" "related_docs" "dependencies" "llm_context" "search_keywords")
+
+# Valid values for specific fields
+VALID_TEMPLATES=("TEMPLATE.documentation-standard" "TEMPLATE.guide" "TEMPLATE.troubleshooting" "TEMPLATE.about" "TEMPLATE.reference" "TEMPLATE.architecture" "TEMPLATE.api" "TEMPLATE.template" "TEMPLATE.custom")
+VALID_DIFFICULTIES=("beginner" "intermediate" "advanced")
+VALID_LLM_CONTEXTS=("high" "medium" "low")
+
+# Counters
+TOTAL_FILES=0
+PASSED_FILES=0
+FAILED_FILES=0
+WARNINGS=0
+
+# Functions
+log() {
+    echo -e "${BLUE}[$(date +'%H:%M:%S')]${NC} $1" | tee -a "$LOG_FILE"
+}
+
+success() {
+    echo -e "${GREEN}✅ $1${NC}" | tee -a "$LOG_FILE"
+}
+
+warning() {
+    echo -e "${YELLOW}⚠️  $1${NC}" | tee -a "$LOG_FILE"
+    ((WARNINGS++))
+}
+
+error() {
+    echo -e "${RED}❌ $1${NC}" | tee -a "$LOG_FILE"
+    ((FAILED_FILES++))
+}
+
+# Check if file has YAML front matter
+check_yaml_front_matter() {
+    local file="$1"
+    local has_yaml=false
+    
+    if head -n 1 "$file" | grep -q "^---$"; then
+        has_yaml=true
+    fi
+    
+    if [ "$has_yaml" = false ]; then
+        error "Missing YAML front matter in $file"
+        return 1
+    fi
+    
+    return 0
+}
+
+# Extract YAML front matter
+extract_yaml() {
+    local file="$1"
+    local yaml_start=0
+    local yaml_end=0
+    local line_num=0
+    
+    while IFS= read -r line; do
+        ((line_num++))
+        if [ "$line" = "---" ]; then
+            if [ $yaml_start -eq 0 ]; then
+                yaml_start=$line_num
+            else
+                yaml_end=$line_num
+                break
+            fi
+        fi
+    done < "$file"
+    
+    if [ $yaml_start -gt 0 ] && [ $yaml_end -gt 0 ]; then
+        sed -n "${yaml_start},${yaml_end}p" "$file"
+    fi
+}
+
+# Validate YAML syntax
+validate_yaml_syntax() {
+    local file="$1"
+    local yaml_content
+    local temp_yaml
+    
+    yaml_content=$(extract_yaml "$file")
+    if [ -z "$yaml_content" ]; then
+        error "Could not extract YAML from $file"
+        return 1
+    fi
+    
+    # Create temporary file for yq validation
+    temp_yaml=$(mktemp)
+    echo "$yaml_content" > "$temp_yaml"
+    
+    # Check if yq is available
+    if command -v yq >/dev/null 2>&1; then
+        if ! yq eval '.' "$temp_yaml" >/dev/null 2>&1; then
+            warning "YAML syntax validation skipped (yq compatibility issue)"
+        fi
+    else
+        warning "yq not available, skipping YAML syntax validation"
+    fi
+    
+    rm -f "$temp_yaml"
+    return 0
+}
+
+# Check required fields
+check_required_fields() {
+    local file="$1"
+    local yaml_content
+    local missing_fields=()
+    
+    yaml_content=$(extract_yaml "$file")
+    if [ -z "$yaml_content" ]; then
+        return 1
+    fi
+    
+    for field in "${REQUIRED_FIELDS[@]}"; do
+        if ! echo "$yaml_content" | grep -q "^${field}:"; then
+            missing_fields+=("$field")
+        fi
+    done
+    
+    if [ ${#missing_fields[@]} -gt 0 ]; then
+        error "Missing required fields in $file: ${missing_fields[*]}"
+        return 1
+    fi
+    
+    return 0
+}
+
+# Validate field values
+validate_field_values() {
+    local file="$1"
+    local yaml_content
+    local template
+    local difficulty
+    local llm_context
+    
+    yaml_content=$(extract_yaml "$file")
+    if [ -z "$yaml_content" ]; then
+        return 1
+    fi
+    
+    # Check template field
+    template=$(echo "$yaml_content" | grep "^template:" | sed 's/^template: *//' | tr -d ' "')
+    if [ -n "$template" ]; then
+        if ! printf '%s\n' "${VALID_TEMPLATES[@]}" | grep -q "^${template}$"; then
+            error "Invalid template value in $file: $template"
+            return 1
+        fi
+    fi
+    
+    # Check difficulty field
+    difficulty=$(echo "$yaml_content" | grep "^difficulty:" | sed 's/^difficulty: *//' | tr -d ' "')
+    if [ -n "$difficulty" ]; then
+        if ! printf '%s\n' "${VALID_DIFFICULTIES[@]}" | grep -q "^${difficulty}$"; then
+            error "Invalid difficulty value in $file: $difficulty"
+            return 1
+        fi
+    fi
+    
+    # Check llm_context field
+    llm_context=$(echo "$yaml_content" | grep "^llm_context:" | sed 's/^llm_context: *//' | tr -d ' "')
+    if [ -n "$llm_context" ]; then
+        if ! printf '%s\n' "${VALID_LLM_CONTEXTS[@]}" | grep -q "^${llm_context}$"; then
+            error "Invalid llm_context value in $file: $llm_context"
+            return 1
+        fi
+    fi
+    
+    return 0
+}
+
+# Check template compliance
+check_template_compliance() {
+    local file="$1"
+    local yaml_content
+    local template
+    local template_file
+    
+    yaml_content=$(extract_yaml "$file")
+    if [ -z "$yaml_content" ]; then
+        return 1
+    fi
+    
+    template=$(echo "$yaml_content" | grep "^template:" | sed 's/^template: *//' | tr -d ' "')
+    if [ -z "$template" ]; then
+        warning "No template specified in $file"
+        return 0
+    fi
+    
+    # Map template names to files
+    case "$template" in
+        "TEMPLATE.documentation-standard")
+            template_file="$TEMPLATES_DIR/TEMPLATE.documentation-standard.md"
+            ;;
+        "TEMPLATE.guide")
+            template_file="$TEMPLATES_DIR/TEMPLATE.guide.md"
+            ;;
+        "TEMPLATE.troubleshooting")
+            template_file="$TEMPLATES_DIR/TEMPLATE.troubleshooting.md"
+            ;;
+        "TEMPLATE.about")
+            template_file="$TEMPLATES_DIR/TEMPLATE.about.md"
+            ;;
+        "TEMPLATE.reference")
+            template_file="$TEMPLATES_DIR/TEMPLATE.reference.md"
+            ;;
+        "TEMPLATE.architecture")
+            template_file="$TEMPLATES_DIR/TEMPLATE.architecture.md"
+            ;;
+        "TEMPLATE.api")
+            template_file="$TEMPLATES_DIR/TEMPLATE.api.md"
+            ;;
+        "TEMPLATE.template")
+            template_file="$TEMPLATES_DIR/TEMPLATE.documentation-standard.md"
+            ;;
+        "TEMPLATE.custom")
+            template_file="$TEMPLATES_DIR/TEMPLATE.custom.md"
+            ;;
+        *)
+            error "Unknown template in $file: $template"
+            return 1
+            ;;
+    esac
+    
+    if [ ! -f "$template_file" ]; then
+        error "Template file not found: $template_file"
+        return 1
+    fi
+    
+    # Basic structure compliance check
+    local template_sections
+    local file_sections
+    
+    template_sections=$(grep "^## " "$template_file" | sed 's/^## //' | sort)
+    file_sections=$(grep "^## " "$file" | sed 's/^## //' | sort)
+    
+    # Check if file has all required sections from template
+    # Skip template compliance for custom documents and meta-documents
+    if [[ "$file" == *"ABOUT."* ]] || [[ "$template" == "TEMPLATE.custom" ]]; then
+        log "Skipping template compliance for custom/meta-document: $file"
+    else
+        while IFS= read -r section; do
+            if [ -n "$section" ]; then
+                if ! echo "$file_sections" | grep -q "^${section}$"; then
+                    warning "Missing section '$section' in $file (required by template $template)"
+                fi
+            fi
+        done <<< "$template_sections"
+    fi
+    
+    return 0
+}
+
+# Check internal links
+check_internal_links() {
+    local file="$1"
+    local broken_links=()
+    local link_pattern='\[([^\]]+)\]\(([^)]+)\)'
+    
+    while IFS= read -r line; do
+        if echo "$line" | grep -qE "$link_pattern"; then
+            # Extract link target
+            local link_target
+            link_target=$(echo "$line" | sed -E "s/.*$link_pattern.*/\2/")
+            
+            # Skip external links
+            if echo "$link_target" | grep -qE '^https?://'; then
+                continue
+            fi
+            
+            # Check if internal link exists
+            if [ ! -f "$link_target" ] && [ ! -f "$DOCS_DIR/$link_target" ]; then
+                broken_links+=("$link_target")
+            fi
+        fi
+    done < "$file"
+    
+    if [ ${#broken_links[@]} -gt 0 ]; then
+        error "Broken internal links in $file: ${broken_links[*]}"
+        return 1
+    fi
+    
+    return 0
+}
+
+# Lint a single file
+lint_file() {
+    local file="$1"
+    local file_passed=true
+    
+    log "Linting $file..."
+    
+    # Check YAML front matter
+    if ! check_yaml_front_matter "$file"; then
+        file_passed=false
+    fi
+    
+    # Validate YAML syntax
+    if ! validate_yaml_syntax "$file"; then
+        file_passed=false
+    fi
+    
+    # Check required fields
+    if ! check_required_fields "$file"; then
+        file_passed=false
+    fi
+    
+    # Validate field values
+    if ! validate_field_values "$file"; then
+        file_passed=false
+    fi
+    
+    # Check template compliance
+    if ! check_template_compliance "$file"; then
+        file_passed=false
+    fi
+    
+    # Check internal links
+    if ! check_internal_links "$file"; then
+        file_passed=false
+    fi
+    
+    if [ "$file_passed" = true ]; then
+        success "All checks passed for $file"
+        ((PASSED_FILES++))
+    else
+        error "Some checks failed for $file"
+    fi
+    
+    ((TOTAL_FILES++))
+}
+
+# Main execution
+main() {
+    log "Starting documentation linting..."
+    log "Log file: $LOG_FILE"
+    
+    # Find all markdown files in dev directory
+    local files
+    files=$(find "$DOCS_DIR" -name "*.md" -type f | sort)
+    
+    if [ -z "$files" ]; then
+        error "No markdown files found in $DOCS_DIR"
+        exit 1
+    fi
+    
+    # Lint each file
+    while IFS= read -r file; do
+        if [ -f "$file" ]; then
+            lint_file "$file"
+        fi
+    done <<< "$files"
+    
+    # Summary
+    log ""
+    log "=== LINTING SUMMARY ==="
+    log "Total files: $TOTAL_FILES"
+    success "Passed: $PASSED_FILES"
+    if [ $FAILED_FILES -gt 0 ]; then
+        error "Failed: $FAILED_FILES"
+    fi
+    if [ $WARNINGS -gt 0 ]; then
+        warning "Warnings: $WARNINGS"
+    fi
+    
+    if [ $FAILED_FILES -eq 0 ]; then
+        success "All documentation files passed linting!"
+        exit 0
+    else
+        error "Some documentation files failed linting. Check the log for details."
+        exit 1
+    fi
+}
+
+# Run main function
+main "$@"

scripts/documentation/lint-index.sh

@@ -0,0 +1,148 @@
+#!/bin/bash
+
+# Sirius Documentation Index Completeness Linter
+# Ensures all files in dev/ are referenced in the documentation index
+
+set -e
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m'
+
+# Configuration
+DOCS_DIR="../../documentation/dev"
+INDEX_FILE="../../documentation/README.documentation-index.md"
+LOG_FILE="tmp/index-lint-$(date +%Y%m%d-%H%M%S).log"
+
+# Create tmp directory if it doesn't exist
+mkdir -p tmp
+
+# Counters
+TOTAL_FILES=0
+MISSING_FILES=0
+EXTRA_FILES=0
+
+# Functions
+log() {
+    echo -e "${BLUE}[$(date +'%H:%M:%S')]${NC} $1" | tee -a "$LOG_FILE"
+}
+
+success() {
+    echo -e "${GREEN}✅ $1${NC}" | tee -a "$LOG_FILE"
+}
+
+warning() {
+    echo -e "${YELLOW}⚠️  $1${NC}" | tee -a "$LOG_FILE"
+}
+
+error() {
+    echo -e "${RED}❌ $1${NC}" | tee -a "$LOG_FILE"
+}
+
+# Get all markdown files in dev directory
+get_dev_files() {
+    find "$DOCS_DIR" -name "*.md" -type f | sed "s|^$DOCS_DIR/||" | sort
+}
+
+# Get files referenced in index
+get_indexed_files() {
+    grep -E "\[.*\]\(dev/.*\.md\)" "$INDEX_FILE" | grep -v "_This document follows" | sed 's/.*\[[^]]*\](dev\/\([^)]*\))/\1/' | sed 's/ - .*$//' | sed 's/^[[:space:]]*//' | sed 's/[[:space:]]*$//' | sort | uniq
+}
+
+# Check if file is in index
+check_file_in_index() {
+    local file="$1"
+    local indexed_files="$2"
+    
+    if echo "$indexed_files" | grep -q "^${file}$"; then
+        return 0
+    else
+        return 1
+    fi
+}
+
+# Check if indexed file exists
+check_indexed_file_exists() {
+    local indexed_file="$1"
+    local dev_files="$2"
+    
+    if echo "$dev_files" | grep -q "^${indexed_file}$"; then
+        return 0
+    else
+        return 1
+    fi
+}
+
+# Main linting function
+lint_index() {
+    log "Starting index completeness linting..."
+    log "Log file: $LOG_FILE"
+    
+    # Get file lists
+    local dev_files
+    local indexed_files
+    
+    dev_files=$(get_dev_files)
+    indexed_files=$(get_indexed_files)
+    
+    log "Found $(echo "$dev_files" | wc -l) files in dev directory"
+    log "Found $(echo "$indexed_files" | wc -l) files referenced in index"
+    
+    # Check for missing files (in dev but not in index)
+    log ""
+    log "Checking for files missing from index..."
+    
+    while IFS= read -r file; do
+        if [ -n "$file" ]; then
+            TOTAL_FILES=$((TOTAL_FILES + 1))
+            if ! check_file_in_index "$file" "$indexed_files"; then
+                error "File missing from index: $file"
+                MISSING_FILES=$((MISSING_FILES + 1))
+            else
+                success "File properly indexed: $file"
+            fi
+        fi
+    done <<< "$dev_files"
+    
+    # Check for extra files (in index but not in dev)
+    log ""
+    log "Checking for extra files in index..."
+    
+    while IFS= read -r indexed_file; do
+        if [ -n "$indexed_file" ]; then
+            if ! check_indexed_file_exists "$indexed_file" "$dev_files"; then
+                warning "File in index but not found in dev: $indexed_file"
+                EXTRA_FILES=$((EXTRA_FILES + 1))
+            fi
+        fi
+    done <<< "$indexed_files"
+    
+    # Summary
+    log ""
+    log "=== INDEX LINTING SUMMARY ==="
+    log "Total dev files: $TOTAL_FILES"
+    if [ $MISSING_FILES -gt 0 ]; then
+        error "Missing from index: $MISSING_FILES"
+    else
+        success "All dev files are indexed"
+    fi
+    if [ $EXTRA_FILES -gt 0 ]; then
+        warning "Extra files in index: $EXTRA_FILES"
+    else
+        success "No extra files in index"
+    fi
+    
+    if [ $MISSING_FILES -eq 0 ] && [ $EXTRA_FILES -eq 0 ]; then
+        success "Index is complete and accurate!"
+        exit 0
+    else
+        error "Index has issues. Check the log for details."
+        exit 1
+    fi
+}
+
+# Run main function
+lint_index "$@"

scripts/documentation/lint-quick.sh

@@ -0,0 +1,83 @@
+#!/bin/bash
+
+# Quick documentation linting for development
+# Faster, less comprehensive checks
+
+set -e
+
+# Colors
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m'
+
+DOCS_DIR="../../documentation/dev"
+
+# Quick checks
+quick_check() {
+    local file="$1"
+    local issues=0
+    
+    # Check if file has YAML front matter
+    if ! head -n 1 "$file" | grep -q "^---$"; then
+        echo -e "${RED}❌ $file: Missing YAML front matter${NC}"
+        issues=$((issues + 1))
+    fi
+    
+    # Check if file has title field
+    if ! grep -q "^title:" "$file"; then
+        echo -e "${RED}❌ $file: Missing title field${NC}"
+        issues=$((issues + 1))
+    fi
+    
+    # Check if file has description field
+    if ! grep -q "^description:" "$file"; then
+        echo -e "${RED}❌ $file: Missing description field${NC}"
+        issues=$((issues + 1))
+    fi
+    
+    # Check if file has template field
+    if ! grep -q "^template:" "$file"; then
+        echo -e "${RED}❌ $file: Missing template field${NC}"
+        issues=$((issues + 1))
+    fi
+    
+    if [ $issues -eq 0 ]; then
+        echo -e "${GREEN}✅ $file: Quick checks passed${NC}"
+    fi
+    
+    return $issues
+}
+
+# Main execution
+main() {
+    echo -e "${BLUE}📚 Running quick documentation checks...${NC}"
+    
+    local total_issues=0
+    local total_files=0
+    
+    # Find all markdown files in dev directory
+    while IFS= read -r -d '' file; do
+        total_files=$((total_files + 1))
+        if ! quick_check "$file"; then
+            total_issues=$((total_issues + 1))
+        fi
+    done < <(find "$DOCS_DIR" -name "*.md" -type f -print0)
+    
+    echo ""
+    echo -e "${BLUE}📊 Quick Check Summary:${NC}"
+    echo -e "  Files checked: $total_files"
+    echo -e "  Files with issues: $total_issues"
+    
+    if [ $total_issues -eq 0 ]; then
+        echo -e "${GREEN}✅ All quick checks passed!${NC}"
+        exit 0
+    else
+        echo -e "${RED}❌ $total_issues files have issues${NC}"
+        exit 1
+    fi
+}
+
+# Run main function
+main "$@"

scripts/git-hooks/pre-commit

@@ -0,0 +1,57 @@
+#!/bin/bash
+
+# Pre-commit hook to automatically comment out volume mounts in docker-compose.override.yaml
+# This prevents accidental commits of local development configuration
+
+echo "🔍 Checking docker-compose.override.yaml for uncommented volume mounts..."
+
+OVERRIDE_FILE="docker-compose.override.yaml"
+
+if [ ! -f "$OVERRIDE_FILE" ]; then
+    exit 0
+fi
+
+# Check if there are any uncommented volume mounts
+if grep -E "^\s*-\s+\.\./minor-projects/" "$OVERRIDE_FILE" > /dev/null; then
+    echo "⚠️  Found uncommented volume mounts in $OVERRIDE_FILE"
+    echo "🔧 Automatically commenting them out..."
+    
+    # Create backup
+    cp "$OVERRIDE_FILE" "${OVERRIDE_FILE}.backup"
+    
+    # Comment out the volume mounts
+    sed -i.tmp 's/^\(\s*\)-\s\+\.\./\1# - \.\./' "$OVERRIDE_FILE"
+    rm "${OVERRIDE_FILE}.tmp" 2>/dev/null || true
+    
+    # Re-stage the fixed file
+    git add "$OVERRIDE_FILE"
+    
+    echo "✅ Volume mounts have been commented out automatically"
+    echo "💡 Your local copy has been backed up to ${OVERRIDE_FILE}.backup"
+    echo "💡 Use docker-compose.local.yaml for local development overrides"
+    echo ""
+fi
+
+# Check for modified agent identity files
+echo ""
+echo "🤖 Checking modified agent identity files..."
+
+AGENTS_DIR=".cursor/agents"
+MODIFIED_AGENTS=$(git diff --cached --name-only --diff-filter=ACM | grep "^${AGENTS_DIR}/.*\.agent\.md$" || true)
+
+if [ -n "$MODIFIED_AGENTS" ]; then
+    echo "📝 Found modified agent files, running quick validation..."
+    
+    if ! bash scripts/agent-identities/lint-agents-quick.sh > /dev/null 2>&1; then
+        echo "❌ Agent identity validation failed!"
+        echo "💡 Run 'cd testing/container-testing && make lint-agents' for detailed errors"
+        exit 1
+    fi
+    
+    echo "✅ Agent identity validation passed"
+else
+    echo "ℹ️  No agent identity files modified"
+fi
+
+echo ""
+echo "✅ Pre-commit validation passed" 

scripts/git-hooks/pre-commit-modernized

@@ -0,0 +1,125 @@
+#!/bin/bash
+
+# Modernized pre-commit hook for Sirius
+# Performs quick validation only - full testing moved to CI
+
+echo "🔍 Running quick pre-commit validation..."
+
+# Check if we're in the right directory
+if [ ! -f "docker-compose.yaml" ]; then
+    echo "❌ Not in Sirius project root directory"
+    exit 1
+fi
+
+# 1. Docker Compose Configuration Validation
+echo "🐳 Validating Docker Compose configurations..."
+if ! docker compose config --quiet; then
+    echo "❌ Base Docker Compose configuration is invalid"
+    exit 1
+fi
+
+if ! docker compose -f docker-compose.yaml -f docker-compose.dev.yaml config --quiet; then
+    echo "❌ Development Docker Compose configuration is invalid"
+    exit 1
+fi
+
+if ! docker compose -f docker-compose.yaml -f docker-compose.prod.yaml config --quiet; then
+    echo "❌ Production Docker Compose configuration is invalid"
+    exit 1
+fi
+
+echo "✅ All Docker Compose configurations are valid"
+
+# 2. Quick Documentation Validation
+echo "📚 Running quick documentation checks..."
+if [ -d "testing/container-testing" ]; then
+    cd testing/container-testing
+    if ! make lint-docs-quick; then
+        echo "❌ Documentation validation failed"
+        exit 1
+    fi
+    if ! make lint-index; then
+        echo "❌ Documentation index validation failed"
+        exit 1
+    fi
+    cd ../..
+else
+    echo "⚠️  Testing directory not found, skipping documentation validation"
+fi
+
+echo "✅ Documentation validation passed"
+
+# 3. Check for uncommented volume mounts in docker-compose.override.yaml
+echo "🔍 Checking for uncommented volume mounts..."
+OVERRIDE_FILE="docker-compose.override.yaml"
+
+if [ -f "$OVERRIDE_FILE" ]; then
+    if grep -E "^\s*-\s+\.\./minor-projects/" "$OVERRIDE_FILE" > /dev/null; then
+        echo "⚠️  Found uncommented volume mounts in $OVERRIDE_FILE"
+        echo "🔧 Automatically commenting them out..."
+        
+        # Create backup
+        cp "$OVERRIDE_FILE" "${OVERRIDE_FILE}.backup"
+        
+        # Comment out the volume mounts
+        sed -i.tmp 's/^\(\s*\)-\s\+\.\./\1# - \.\./' "$OVERRIDE_FILE"
+        rm "${OVERRIDE_FILE}.tmp" 2>/dev/null || true
+        
+        # Re-stage the fixed file
+        git add "$OVERRIDE_FILE"
+        
+        echo "✅ Volume mounts have been commented out automatically"
+        echo "💡 Your local copy has been backed up to ${OVERRIDE_FILE}.backup"
+        echo "💡 Use docker-compose.local.yaml for local development overrides"
+    fi
+fi
+
+# 4. Check for accidentally committed local files
+echo "🔍 Checking for accidentally committed local files..."
+if [ -f "docker-compose.local.yaml" ]; then
+    echo "❌ docker-compose.local.yaml should not be committed"
+    echo "💡 This file is for local development only and should be git-ignored"
+    exit 1
+fi
+
+if ls docker-compose.*.local.yaml 2>/dev/null; then
+    echo "❌ Local docker-compose files found:"
+    ls docker-compose.*.local.yaml
+    echo "💡 These files should be git-ignored"
+    exit 1
+fi
+
+echo "✅ No local override files found in repository"
+
+# 5. Basic syntax checks for key files
+echo "🔍 Running basic syntax checks..."
+
+# Check if package.json is valid JSON
+if [ -f "sirius-ui/package.json" ]; then
+    if ! python3 -m json.tool sirius-ui/package.json > /dev/null 2>&1; then
+        echo "❌ sirius-ui/package.json is not valid JSON"
+        exit 1
+    fi
+fi
+
+# Check if go.mod is valid
+if [ -f "sirius-api/go.mod" ]; then
+    if ! go mod verify -modfile=sirius-api/go.mod > /dev/null 2>&1; then
+        echo "❌ sirius-api/go.mod is not valid"
+        exit 1
+    fi
+fi
+
+if [ -f "sirius-engine/go.mod" ]; then
+    if ! go mod verify -modfile=sirius-engine/go.mod > /dev/null 2>&1; then
+        echo "❌ sirius-engine/go.mod is not valid"
+        exit 1
+    fi
+fi
+
+echo "✅ Basic syntax checks passed"
+
+echo ""
+echo "🎉 Pre-commit validation completed successfully!"
+echo "💡 Full testing will run in CI/CD pipeline"
+echo "💡 For local testing, run: cd testing/container-testing && make test-all"

scripts/performance-test-suite.sh

@@ -0,0 +1 @@
+ 

scripts/prepare-monitoring.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+
+# Script to prepare monitoring components for container builds
+# This script validates that the main SystemMonitor project is available
+
+set -e
+
+echo "🔧 Preparing monitoring components for container builds..."
+
+# Get the directory of this script
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
+
+# Paths to the monitoring components
+SYSTEM_MONITOR_DIR="$PROJECT_ROOT/../minor-projects/app-system-monitor"
+
+# Check if main system monitor directory exists
+if [ ! -d "$SYSTEM_MONITOR_DIR" ]; then
+    echo "❌ System monitor directory not found: $SYSTEM_MONITOR_DIR"
+    exit 1
+fi
+
+# Validate that the main system monitor has the required files
+echo "📊 Validating main system monitor project..."
+
+if [ ! -f "$SYSTEM_MONITOR_DIR/main.go" ]; then
+    echo "❌ System monitor main.go not found"
+    exit 1
+fi
+
+if [ ! -f "$SYSTEM_MONITOR_DIR/go.mod" ]; then
+    echo "❌ System monitor go.mod not found"
+    exit 1
+fi
+
+# Test build the system monitor to ensure it compiles
+echo "🔨 Testing system monitor build..."
+cd "$SYSTEM_MONITOR_DIR"
+if go build -o /tmp/test-system-monitor .; then
+    echo "✅ System monitor builds successfully"
+    rm -f /tmp/test-system-monitor
+else
+    echo "❌ System monitor build failed"
+    exit 1
+fi
+
+echo "✅ Main system monitor project is ready for container builds!"
+echo "📝 Note: Containers will now build SystemMonitor from the main project instead of local copies"

scripts/switch-env.sh

@@ -0,0 +1,109 @@
+#!/bin/bash
+
+# Sirius Environment Switching Script
+# Usage: ./scripts/switch-env.sh [dev|prod|base]
+
+set -e
+
+ENV=${1:-"base"}
+PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+INSTALLER_COMPOSE_FILE="$PROJECT_ROOT/docker-compose.installer.yaml"
+ENV_FILE="$PROJECT_ROOT/.env"
+
+require_env_file() {
+    if [ -f "$ENV_FILE" ]; then
+        return 0
+    fi
+
+    echo "⚠️  Missing $ENV_FILE"
+    if [ "${SIRIUS_AUTO_SETUP:-0}" = "1" ]; then
+        echo "🔧 SIRIUS_AUTO_SETUP=1 detected; running installer container..."
+        docker compose -f "$INSTALLER_COMPOSE_FILE" run --rm sirius-installer --non-interactive --no-print-secrets
+        return 0
+    fi
+
+    echo "Run installer first:"
+    echo "  docker compose -f docker-compose.installer.yaml run --rm sirius-installer"
+    echo "Or rerun with SIRIUS_AUTO_SETUP=1 to auto-generate missing values."
+    exit 1
+}
+
+echo "🔄 Switching Sirius environment to: $ENV"
+
+# Function to clean up containers and images
+cleanup() {
+    echo "🧹 Cleaning up existing containers and images..."
+    docker compose down 2>/dev/null || true
+    docker compose -f docker-compose.yaml -f docker-compose.dev.yaml down 2>/dev/null || true
+    docker compose -f docker-compose.yaml -f docker-compose.prod.yaml down 2>/dev/null || true
+    
+    # Remove old UI images to force rebuild
+    docker rmi sirius-sirius-ui:latest 2>/dev/null || true
+    docker rmi sirius-sirius-ui:dev 2>/dev/null || true
+    docker rmi sirius-sirius-ui:prod 2>/dev/null || true
+}
+
+# Function to start development environment
+start_dev() {
+    echo "🚀 Starting development environment..."
+    docker compose -f docker-compose.yaml -f docker-compose.dev.yaml build --no-cache sirius-ui
+    docker compose -f docker-compose.yaml -f docker-compose.dev.yaml up -d
+    echo "✅ Development environment started!"
+    echo "🌐 UI: http://localhost:3000 (with hot reloading)"
+    echo "🔧 API: http://localhost:9001"
+}
+
+# Function to start production environment
+start_prod() {
+    echo "🚀 Starting production environment..."
+    docker compose -f docker-compose.yaml -f docker-compose.prod.yaml build --no-cache sirius-ui
+    docker compose -f docker-compose.yaml -f docker-compose.prod.yaml up -d
+    echo "✅ Production environment started!"
+    echo "🌐 UI: http://localhost:3000 (optimized build)"
+    echo "🔧 API: http://localhost:9001"
+}
+
+# Function to start base environment
+start_base() {
+    echo "🚀 Starting base environment..."
+    docker compose build --no-cache sirius-ui
+    docker compose up -d
+    echo "✅ Base environment started!"
+    echo "🌐 UI: http://localhost:3000"
+    echo "🔧 API: http://localhost:9001"
+}
+
+# Main logic
+cd "$PROJECT_ROOT"
+
+case $ENV in
+    "dev"|"development")
+        require_env_file
+        cleanup
+        start_dev
+        ;;
+    "prod"|"production")
+        require_env_file
+        cleanup
+        start_prod
+        ;;
+    "base"|"default")
+        require_env_file
+        cleanup
+        start_base
+        ;;
+    *)
+        echo "❌ Invalid environment: $ENV"
+        echo "Usage: $0 [dev|prod|base]"
+        echo ""
+        echo "Environments:"
+        echo "  dev     - Development with hot reloading and volume mounts"
+        echo "  prod    - Production with optimized builds"
+        echo "  base    - Base configuration (default)"
+        exit 1
+        ;;
+esac
+
+echo ""
+echo "📊 Container status:"
+docker compose ps

scripts/test-e2e-suite.sh

@@ -0,0 +1,416 @@
+#!/bin/bash
+
+# ============================================================================
+# Sirius E2E Test Suite
+# 
+# Comprehensive end-to-end testing for source attribution functionality
+# Tests complete workflows across all scanner types and frontend integration
+# ============================================================================
+
+set -e
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m' # No Color
+
+# Test configuration
+API_BASE_URL="${API_BASE_URL:-http://localhost:9001}"
+UI_BASE_URL="${UI_BASE_URL:-http://localhost:3000}"
+TEST_HOST_IP="192.168.1.100"
+TEST_HOST_IP_2="192.168.1.101"
+TEST_TIMESTAMP=$(date +%s)
+RESULTS_DIR="test-results-e2e-${TEST_TIMESTAMP}"
+
+# Create test results directory
+mkdir -p "${RESULTS_DIR}"
+
+echo -e "${BLUE}🚀 Starting Sirius E2E Test Suite${NC}"
+echo -e "${BLUE}================================================${NC}"
+echo -e "API Base URL: ${API_BASE_URL}"
+echo -e "UI Base URL: ${UI_BASE_URL}"
+echo -e "Results Directory: ${RESULTS_DIR}"
+echo ""
+
+# Test counters
+TESTS_TOTAL=0
+TESTS_PASSED=0
+TESTS_FAILED=0
+
+# Function to run a test
+run_test() {
+    local test_name="$1"
+    local test_function="$2"
+    
+    TESTS_TOTAL=$((TESTS_TOTAL + 1))
+    echo -e "${YELLOW}📋 Test ${TESTS_TOTAL}: ${test_name}${NC}"
+    
+    if $test_function; then
+        echo -e "${GREEN}✅ PASS: ${test_name}${NC}"
+        TESTS_PASSED=$((TESTS_PASSED + 1))
+        echo "PASS" > "${RESULTS_DIR}/test_${TESTS_TOTAL}_$(echo ${test_name} | tr ' ' '_').result"
+    else
+        echo -e "${RED}❌ FAIL: ${test_name}${NC}"
+        TESTS_FAILED=$((TESTS_FAILED + 1))
+        echo "FAIL" > "${RESULTS_DIR}/test_${TESTS_TOTAL}_$(echo ${test_name} | tr ' ' '_').result"
+    fi
+    echo ""
+}
+
+# Function to cleanup test data
+cleanup_test_data() {
+    echo -e "${YELLOW}🧹 Cleaning up test data...${NC}"
+    
+    # Clean up test hosts
+    curl -s -X DELETE "${API_BASE_URL}/host/${TEST_HOST_IP}" || true
+    curl -s -X DELETE "${API_BASE_URL}/host/${TEST_HOST_IP_2}" || true
+    
+    # Additional cleanup as needed
+    sleep 2
+}
+
+# Function to check service availability
+check_services() {
+    echo -e "${BLUE}🔍 Checking service availability...${NC}"
+    
+    # Check API service
+    if ! curl -s --max-time 10 "${API_BASE_URL}/health" > /dev/null 2>&1; then
+        echo -e "${RED}❌ API service not available at ${API_BASE_URL}${NC}"
+        exit 1
+    fi
+    
+    # Check UI service (optional)
+    if ! curl -s --max-time 10 "${UI_BASE_URL}" > /dev/null 2>&1; then
+        echo -e "${YELLOW}⚠️  UI service not available at ${UI_BASE_URL} (continuing with API tests)${NC}"
+    fi
+    
+    echo -e "${GREEN}✅ Services are available${NC}"
+    echo ""
+}
+
+# ============================================================================
+# Test Functions
+# ============================================================================
+
+# Test 1: Basic API Connectivity
+test_api_connectivity() {
+    local response=$(curl -s -w "%{http_code}" "${API_BASE_URL}/health")
+    local http_code="${response: -3}"
+    
+    if [[ "$http_code" == "200" ]]; then
+        return 0
+    else
+        echo "HTTP Status: $http_code"
+        return 1
+    fi
+}
+
+# Test 2: Legacy Host Submission (Backward Compatibility)
+test_legacy_host_submission() {
+    local payload='{
+        "hid": "e2e-test-legacy",
+        "os": "Ubuntu",
+        "osversion": "22.04",
+        "ip": "'${TEST_HOST_IP}'",
+        "hostname": "test-host-legacy",
+        "ports": [{"id": 22, "protocol": "tcp", "state": "open"}],
+        "vulnerabilities": [{"vid": "CVE-2023-E2E-LEGACY", "description": "E2E Test Legacy Vulnerability", "title": "Legacy Test CVE"}]
+    }'
+    
+    local response=$(curl -s -w "%{http_code}" -X POST \
+        -H "Content-Type: application/json" \
+        -d "$payload" \
+        "${API_BASE_URL}/host")
+    
+    local http_code="${response: -3}"
+    
+    if [[ "$http_code" == "200" ]]; then
+        echo "Legacy host submission successful"
+        return 0
+    else
+        echo "HTTP Status: $http_code"
+        echo "Response: ${response%???}"
+        return 1
+    fi
+}
+
+# Test 3: Source-Aware Network Scanner Submission
+test_network_scanner_submission() {
+    local payload='{
+        "hid": "e2e-test-nmap",
+        "os": "Linux",
+        "osversion": "Ubuntu 22.04",
+        "ip": "'${TEST_HOST_IP}'",
+        "hostname": "test-host-nmap",
+        "ports": [
+            {"id": 80, "protocol": "tcp", "state": "open"},
+            {"id": 443, "protocol": "tcp", "state": "open"}
+        ],
+        "vulnerabilities": [
+            {
+                "vid": "CVE-2023-E2E-NMAP",
+                "description": "E2E Test Network Vulnerability",
+                "title": "Network Scanner Test CVE"
+            }
+        ]
+    }'
+    
+    local response=$(curl -s -w "%{http_code}" -X POST \
+        -H "Content-Type: application/json" \
+        -H "User-Agent: nmap-scanner/7.94" \
+        -H "X-Scanner-Name: nmap" \
+        -H "X-Scanner-Version: 7.94" \
+        -d "$payload" \
+        "${API_BASE_URL}/host")
+    
+    local http_code="${response: -3}"
+    
+    if [[ "$http_code" == "200" ]]; then
+        echo "Network scanner submission successful"
+        return 0
+    else
+        echo "HTTP Status: $http_code"
+        echo "Response: ${response%???}"
+        return 1
+    fi
+}
+
+# Test 4: Source-Aware Agent Submission
+test_agent_submission() {
+    local payload='{
+        "hid": "e2e-test-agent",
+        "os": "Windows",
+        "osversion": "Server 2019",
+        "ip": "'${TEST_HOST_IP}'",
+        "hostname": "test-host-agent",
+        "vulnerabilities": [
+            {
+                "vid": "CVE-2023-E2E-AGENT",
+                "description": "E2E Test Agent Vulnerability",
+                "title": "Agent Scanner Test CVE"
+            }
+        ],
+        "users": ["testuser1", "testuser2"]
+    }'
+    
+    local response=$(curl -s -w "%{http_code}" -X POST \
+        -H "Content-Type: application/json" \
+        -H "User-Agent: sirius-agent/1.0.0" \
+        -H "X-Scanner-Name: agent" \
+        -H "X-Scanner-Version: 1.0.0" \
+        -d "$payload" \
+        "${API_BASE_URL}/host")
+    
+    local http_code="${response: -3}"
+    
+    if [[ "$http_code" == "200" ]]; then
+        echo "Agent submission successful"
+        return 0
+    else
+        echo "HTTP Status: $http_code"
+        echo "Response: ${response%???}"
+        return 1
+    fi
+}
+
+# Test 5: Multi-Source Data Integrity
+test_multi_source_integrity() {
+    # First, submit via network scanner
+    test_network_scanner_submission > /dev/null 2>&1
+    
+    # Then submit via agent (should merge, not replace)
+    test_agent_submission > /dev/null 2>&1
+    
+    # Verify both sources are preserved
+    local response=$(curl -s "${API_BASE_URL}/host/${TEST_HOST_IP}/sources")
+    
+    if echo "$response" | grep -q "nmap" && echo "$response" | grep -q "agent"; then
+        echo "Multi-source data integrity verified"
+        return 0
+    else
+        echo "Multi-source verification failed"
+        echo "Response: $response"
+        return 1
+    fi
+}
+
+# Test 6: Source Attribution API Endpoints
+test_source_attribution_endpoints() {
+    # Test host sources endpoint
+    local sources_response=$(curl -s "${API_BASE_URL}/host/${TEST_HOST_IP}/sources")
+    if ! echo "$sources_response" | grep -q "sources"; then
+        echo "Host sources endpoint failed"
+        return 1
+    fi
+    
+    # Test host history endpoint
+    local history_response=$(curl -s "${API_BASE_URL}/host/${TEST_HOST_IP}/history")
+    if ! echo "$history_response" | grep -q "history"; then
+        echo "Host history endpoint failed"
+        return 1
+    fi
+    
+    # Test source coverage endpoint
+    local coverage_response=$(curl -s "${API_BASE_URL}/host/source-coverage")
+    if ! echo "$coverage_response" | grep -q "coverage"; then
+        echo "Source coverage endpoint failed"
+        return 1
+    fi
+    
+    echo "All source attribution endpoints working"
+    return 0
+}
+
+# Test 7: Database Consistency Check
+test_database_consistency() {
+    # Get host data and verify structure
+    local host_response=$(curl -s "${API_BASE_URL}/host/${TEST_HOST_IP}")
+    
+    # Check if response contains expected fields
+    if echo "$host_response" | grep -q "ip" && \
+       echo "$host_response" | grep -q "vulnerabilities" && \
+       echo "$host_response" | grep -q "ports"; then
+        echo "Database consistency verified"
+        return 0
+    else
+        echo "Database consistency check failed"
+        echo "Response: $host_response"
+        return 1
+    fi
+}
+
+# Test 8: Performance Baseline Test
+test_performance_baseline() {
+    local start_time=$(date +%s%N)
+    
+    # Perform a standard API call
+    curl -s "${API_BASE_URL}/host" > /dev/null
+    
+    local end_time=$(date +%s%N)
+    local duration=$(((end_time - start_time) / 1000000)) # Convert to milliseconds
+    
+    # Check if response time is reasonable (< 2000ms)
+    if [[ $duration -lt 2000 ]]; then
+        echo "Performance baseline met: ${duration}ms"
+        return 0
+    else
+        echo "Performance baseline failed: ${duration}ms (should be < 2000ms)"
+        return 1
+    fi
+}
+
+# Test 9: Vulnerability Data Format Validation
+test_vulnerability_data_format() {
+    local response=$(curl -s "${API_BASE_URL}/host/${TEST_HOST_IP}/sources")
+    
+    # Check if vulnerability data has required fields
+    if echo "$response" | grep -q "vid" && \
+       echo "$response" | grep -q "description" && \
+       echo "$response" | grep -q "sources"; then
+        echo "Vulnerability data format validated"
+        return 0
+    else
+        echo "Vulnerability data format validation failed"
+        echo "Response: $response"
+        return 1
+    fi
+}
+
+# Test 10: Cross-Scanner Compatibility
+test_cross_scanner_compatibility() {
+    # Test with different scanner types on same host
+    local rustscan_payload='{
+        "hid": "e2e-test-rustscan",
+        "ip": "'${TEST_HOST_IP_2}'",
+        "hostname": "test-host-rustscan",
+        "ports": [{"id": 8080, "protocol": "tcp", "state": "open"}]
+    }'
+    
+    local naabu_payload='{
+        "hid": "e2e-test-naabu",
+        "ip": "'${TEST_HOST_IP_2}'",
+        "hostname": "test-host-naabu",
+        "ports": [{"id": 9090, "protocol": "tcp", "state": "open"}]
+    }'
+    
+    # Submit via rustscan
+    curl -s -X POST \
+        -H "Content-Type: application/json" \
+        -H "User-Agent: rustscan/2.1.1" \
+        -H "X-Scanner-Name: rustscan" \
+        -d "$rustscan_payload" \
+        "${API_BASE_URL}/host" > /dev/null
+    
+    # Submit via naabu
+    curl -s -X POST \
+        -H "Content-Type: application/json" \
+        -H "User-Agent: naabu/2.1.9" \
+        -H "X-Scanner-Name: naabu" \
+        -d "$naabu_payload" \
+        "${API_BASE_URL}/host" > /dev/null
+    
+    # Verify both scanners are tracked
+    local response=$(curl -s "${API_BASE_URL}/host/${TEST_HOST_IP_2}/sources")
+    
+    if echo "$response" | grep -q "rustscan" && echo "$response" | grep -q "naabu"; then
+        echo "Cross-scanner compatibility verified"
+        return 0
+    else
+        echo "Cross-scanner compatibility failed"
+        echo "Response: $response"
+        return 1
+    fi
+}
+
+# ============================================================================
+# Main Test Execution
+# ============================================================================
+
+main() {
+    echo -e "${BLUE}Starting E2E Test Suite Execution${NC}"
+    echo ""
+    
+    # Check service availability
+    check_services
+    
+    # Clean up any previous test data
+    cleanup_test_data
+    
+    # Run all tests
+    run_test "API Connectivity" test_api_connectivity
+    run_test "Legacy Host Submission" test_legacy_host_submission
+    run_test "Network Scanner Submission" test_network_scanner_submission
+    run_test "Agent Submission" test_agent_submission
+    run_test "Multi-Source Data Integrity" test_multi_source_integrity
+    run_test "Source Attribution Endpoints" test_source_attribution_endpoints
+    run_test "Database Consistency" test_database_consistency
+    run_test "Performance Baseline" test_performance_baseline
+    run_test "Vulnerability Data Format" test_vulnerability_data_format
+    run_test "Cross-Scanner Compatibility" test_cross_scanner_compatibility
+    
+    # Clean up test data
+    cleanup_test_data
+    
+    # Display results
+    echo -e "${BLUE}================================================${NC}"
+    echo -e "${BLUE}📊 E2E Test Suite Results${NC}"
+    echo -e "${BLUE}================================================${NC}"
+    echo -e "Total Tests: ${TESTS_TOTAL}"
+    echo -e "${GREEN}Passed: ${TESTS_PASSED}${NC}"
+    echo -e "${RED}Failed: ${TESTS_FAILED}${NC}"
+    
+    if [[ $TESTS_FAILED -eq 0 ]]; then
+        echo -e "${GREEN}🎉 All tests passed! System is ready for production.${NC}"
+        echo "PASS" > "${RESULTS_DIR}/e2e_suite_result.txt"
+        exit 0
+    else
+        echo -e "${RED}❌ Some tests failed. Please review and fix issues.${NC}"
+        echo "FAIL" > "${RESULTS_DIR}/e2e_suite_result.txt"
+        exit 1
+    fi
+}
+
+# Run main function
+main "$@" 

scripts/test-full-build.sh

@@ -0,0 +1,37 @@
+#!/bin/bash
+
+# Sirius Full Build Test Script
+# Runs complete build tests including Docker builds
+# Use this when you want to test full builds on feature branches
+
+set -e
+
+# Colors
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m'
+
+echo -e "${BLUE}🔨 Running full build tests...${NC}"
+
+# Check if we're in the right directory
+if [ ! -f "testing/container-testing/Makefile" ]; then
+    echo -e "${RED}❌ Not in Sirius project root${NC}"
+    echo -e "${YELLOW}💡 Run this script from the project root directory${NC}"
+    exit 1
+fi
+
+# Change to container testing directory
+cd testing/container-testing
+
+# Run full build tests
+echo -e "${BLUE}🧪 Running complete build test suite...${NC}"
+if make test-build; then
+    echo -e "${GREEN}✅ Full build tests passed!${NC}"
+    echo -e "${YELLOW}💡 All Docker images built successfully${NC}"
+else
+    echo -e "${RED}❌ Full build tests failed${NC}"
+    echo -e "${YELLOW}💡 Check the output above for specific errors${NC}"
+    exit 1
+fi

scripts/test-phase1-cleanup.sh

@@ -0,0 +1,139 @@
+#!/bin/bash
+
+# Test Cleanup Script for Phase 1 Validation
+# This script removes test data and test scripts after validation
+# RUN THIS SCRIPT after Phase 1 testing is complete
+
+set -e
+
+API_BASE="http://localhost:9001"
+echo "🧹 Starting Phase 1 Test Cleanup..."
+echo "API Base: $API_BASE"
+echo ""
+
+# Color codes for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m' # No Color
+
+# Function to make DELETE API calls
+delete_host() {
+    local ip=$1
+    local description=$2
+    
+    echo "🗑️  $description"
+    response=$(curl -s -w "\n%{http_code}" -X DELETE "$API_BASE/host/$ip")
+    
+    # Extract HTTP code
+    http_code=$(echo "$response" | tail -n1)
+    response_body=$(echo "$response" | head -n -1)
+    
+    if [ "$http_code" -ge 200 ] && [ "$http_code" -lt 300 ]; then
+        echo -e "   ${GREEN}✅ Deleted ($http_code)${NC}"
+    else
+        echo -e "   ${YELLOW}⚠️  May not exist or already deleted ($http_code)${NC}"
+    fi
+    echo ""
+}
+
+echo "🗑️  CLEANING UP TEST DATA"
+echo "========================"
+
+# Remove all test hosts
+delete_host "192.168.1.100" "Removing multi-source test host"
+delete_host "192.168.1.200" "Removing temporal tracking test host"
+delete_host "192.168.1.300" "Removing backward compatibility test host"
+delete_host "192.168.1.400" "Removing multi-source CVE test host"
+delete_host "192.168.1.500" "Removing port attribution test host"
+
+# Alternative: If there's a bulk delete endpoint
+echo "🧹 Attempting bulk cleanup (if supported)..."
+bulk_response=$(curl -s -w "\n%{http_code}" -X POST "$API_BASE/hosts/bulk-delete" \
+    -H "Content-Type: application/json" \
+    -d '{"ips": ["192.168.1.100", "192.168.1.200", "192.168.1.300", "192.168.1.400", "192.168.1.500"]}' 2>/dev/null || echo "not_supported\n404")
+
+bulk_code=$(echo "$bulk_response" | tail -n1)
+if [ "$bulk_code" = "200" ] || [ "$bulk_code" = "204" ]; then
+    echo -e "${GREEN}✅ Bulk cleanup successful${NC}"
+else
+    echo -e "${YELLOW}⚠️  Bulk cleanup not supported, individual deletes used${NC}"
+fi
+echo ""
+
+echo "📋 VERIFYING CLEANUP"
+echo "==================="
+
+# Verify hosts are gone
+echo "🔍 Checking if test hosts were removed..."
+for ip in "192.168.1.100" "192.168.1.200" "192.168.1.300" "192.168.1.400" "192.168.1.500"; do
+    check_response=$(curl -s -w "%{http_code}" "$API_BASE/host/$ip" -o /dev/null)
+    if [ "$check_response" = "404" ]; then
+        echo -e "   ${GREEN}✅ $ip - Successfully removed${NC}"
+    else
+        echo -e "   ${YELLOW}⚠️  $ip - Still exists (code: $check_response)${NC}"
+    fi
+done
+echo ""
+
+echo "🗂️  CLEANING UP TEST SCRIPTS"
+echo "============================"
+
+# Get the directory where this script is located
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+# Remove test scripts
+if [ -f "$SCRIPT_DIR/test-phase1-populate.sh" ]; then
+    rm "$SCRIPT_DIR/test-phase1-populate.sh"
+    echo -e "${GREEN}✅ Removed test-phase1-populate.sh${NC}"
+else
+    echo -e "${YELLOW}⚠️  test-phase1-populate.sh not found${NC}"
+fi
+
+if [ -f "$SCRIPT_DIR/test-phase1-verify.sh" ]; then
+    rm "$SCRIPT_DIR/test-phase1-verify.sh"
+    echo -e "${GREEN}✅ Removed test-phase1-verify.sh${NC}"
+else
+    echo -e "${YELLOW}⚠️  test-phase1-verify.sh not found${NC}"
+fi
+
+echo ""
+echo "🔄 OPTIONAL: Database Cleanup"
+echo "============================="
+echo "If you want to completely reset the database for testing:"
+echo ""
+echo "Option 1 - Reset specific test data (if supported by API):"
+echo "  curl -X POST $API_BASE/admin/reset-test-data"
+echo ""
+echo "Option 2 - Full database reset (CAUTION - removes ALL data):"
+echo "  docker-compose down"
+echo "  docker volume rm \$(docker volume ls -q | grep postgres)"
+echo "  docker-compose up -d"
+echo ""
+echo "Option 3 - Truncate test tables only:"
+if command -v psql &> /dev/null; then
+    echo "  psql -h localhost -U postgres -d sirius -c \"DELETE FROM host_vulnerabilities WHERE host_id IN (SELECT id FROM hosts WHERE ip LIKE '192.168.1.%');\""
+    echo "  psql -h localhost -U postgres -d sirius -c \"DELETE FROM hosts WHERE ip LIKE '192.168.1.%';\""
+else
+    echo "  (psql not available - use database admin tool)"
+fi
+
+echo ""
+echo -e "${GREEN}🎉 CLEANUP COMPLETED!${NC}"
+echo ""
+echo "📝 Summary:"
+echo "   • Test data removed from API"
+echo "   • Test scripts deleted"
+echo "   • System ready for production use"
+echo ""
+echo "🔄 Next Steps:"
+echo "   • Proceed to Phase 2 implementation"
+echo "   • Begin scanner integration work"
+echo "   • Update task status to completed"
+
+# Self-destruct this cleanup script
+echo ""
+echo "🔥 Self-destructing cleanup script..."
+rm "$0"
+echo -e "${GREEN}✅ Cleanup script removed${NC}" 

scripts/test-phase1-populate.sh

@@ -0,0 +1,287 @@
+#!/bin/bash
+
+# Test Data Population Script for Phase 1 Validation
+# This script populates test data to validate source-aware functionality
+# DELETE THIS SCRIPT after Phase 1 testing is complete
+
+set -e
+
+API_BASE="http://localhost:9001"
+echo "🧪 Starting Phase 1 Test Data Population..."
+echo "API Base: $API_BASE"
+echo ""
+
+# Function to make API calls with error handling
+make_api_call() {
+    local method=$1
+    local endpoint=$2
+    local data=$3
+    local description=$4
+    
+    echo "📡 $description"
+    echo "   Method: $method"
+    echo "   Endpoint: $endpoint"
+    
+    if [ "$method" = "POST" ]; then
+        response=$(curl -s -w "\n%{http_code}" -X POST "$API_BASE$endpoint" \
+            -H "Content-Type: application/json" \
+            -d "$data")
+    else
+        response=$(curl -s -w "\n%{http_code}" "$API_BASE$endpoint")
+    fi
+    
+    # Extract response body and HTTP code
+    http_code=$(echo "$response" | tail -n1)
+    response_body=$(echo "$response" | sed '$d')
+    
+    if [ "$http_code" -ge 200 ] && [ "$http_code" -lt 300 ]; then
+        echo "   ✅ Success ($http_code)"
+        echo "   Response: $(echo "$response_body" | jq -c . 2>/dev/null || echo "$response_body")"
+    else
+        echo "   ❌ Failed ($http_code)"
+        echo "   Response: $response_body"
+        return 1
+    fi
+    echo ""
+}
+
+# Test Case 1: Multi-Source Data Loss Prevention Test
+echo "🔥 TEST CASE 1: Multi-Source Data Loss Prevention"
+echo "Testing that different sources don't overwrite each other's data"
+echo ""
+
+# 1a. Add host with NMAP source
+nmap_data='{
+  "host": {
+    "ip": "192.168.1.100",
+    "hostname": "test-host-1",
+    "os": "Linux",
+    "osversion": "Ubuntu 22.04",
+    "vulnerabilities": [
+      {
+        "vid": "CVE-2017-0144",
+        "title": "EternalBlue SMB Vulnerability",
+        "description": "Microsoft SMB Remote Code Execution Vulnerability",
+        "riskscore": 8.1
+      },
+      {
+        "vid": "CVE-2019-0708",
+        "title": "BlueKeep RDP Vulnerability",
+        "description": "Remote Desktop Services Remote Code Execution Vulnerability",
+        "riskscore": 9.8
+      }
+    ]
+  },
+  "source": {
+    "name": "nmap",
+    "version": "7.94",
+    "config": "nmap -sV -sC --script vuln"
+  }
+}'
+
+make_api_call "POST" "/host/with-source" "$nmap_data" "Adding host with NMAP source"
+
+# 1b. Add same host with AGENT source (should NOT overwrite NMAP data)
+agent_data='{
+  "host": {
+    "ip": "192.168.1.100",
+    "hostname": "test-host-1",
+    "os": "Linux",
+    "osversion": "Ubuntu 22.04",
+    "vulnerabilities": [
+      {
+        "vid": "CVE-2021-34527",
+        "title": "PrintNightmare Vulnerability",
+        "description": "Windows Print Spooler Remote Code Execution Vulnerability",
+        "riskscore": 8.8
+      },
+      {
+        "vid": "CVE-2019-0708",
+        "title": "BlueKeep RDP Vulnerability",
+        "description": "Remote Desktop Services Remote Code Execution Vulnerability (Agent detected)",
+        "riskscore": 9.8
+      }
+    ]
+  },
+  "source": {
+    "name": "agent",
+    "version": "2.1.0",
+    "config": "full-system-scan"
+  }
+}'
+
+make_api_call "POST" "/host/with-source" "$agent_data" "Adding same host with AGENT source"
+
+# Test Case 2: Temporal Tracking Test
+echo "🕐 TEST CASE 2: Temporal Tracking"
+echo "Testing first_seen and last_seen timestamps"
+echo ""
+
+# 2a. Create initial scan
+temporal_data_1='{
+  "host": {
+    "ip": "192.168.1.200",
+    "hostname": "temporal-test",
+    "vulnerabilities": [
+      {
+        "vid": "CVE-2020-1472",
+        "title": "Zerologon Vulnerability",
+        "description": "Netlogon Remote Protocol (MS-NRPC) Elevation of Privilege Vulnerability",
+        "riskscore": 10.0
+      }
+    ]
+  },
+  "source": {
+    "name": "nmap",
+    "version": "7.94",
+    "config": "initial-scan"
+  }
+}'
+
+make_api_call "POST" "/host/with-source" "$temporal_data_1" "Initial scan for temporal tracking"
+
+# 2b. Wait and rescan with same source (should update last_seen)
+echo "⏳ Waiting 3 seconds to test temporal tracking..."
+sleep 3
+
+temporal_data_2='{
+  "host": {
+    "ip": "192.168.1.200",
+    "hostname": "temporal-test",
+    "vulnerabilities": [
+      {
+        "vid": "CVE-2020-1472",
+        "title": "Zerologon Vulnerability",
+        "description": "Netlogon Remote Protocol (MS-NRPC) Elevation of Privilege Vulnerability - rescan",
+        "riskscore": 10.0
+      }
+    ]
+  },
+  "source": {
+    "name": "nmap",
+    "version": "7.94",
+    "config": "rescan"
+  }
+}'
+
+make_api_call "POST" "/host/with-source" "$temporal_data_2" "Rescan for temporal tracking (should update last_seen)"
+
+# Test Case 3: Backward Compatibility Test
+echo "🔄 TEST CASE 3: Backward Compatibility"
+echo "Testing that old API still works with source attribution"
+echo ""
+
+legacy_data='{
+  "ip": "192.168.1.300",
+  "hostname": "legacy-test",
+  "os": "Windows",
+  "osversion": "Server 2019",
+  "vulnerabilities": [
+    {
+      "vid": "CVE-2018-8174",
+      "title": "VBScript Engine Memory Corruption",
+      "description": "VBScript Engine Remote Code Execution Vulnerability",
+      "riskscore": 7.5
+    }
+  ]
+}'
+
+make_api_call "POST" "/host" "$legacy_data" "Testing legacy API (should assign unknown source)"
+
+# Test Case 4: Multiple Sources on Same Vulnerability
+echo "🔀 TEST CASE 4: Multiple Sources on Same CVE"
+echo "Testing that same CVE from different sources creates separate records"
+echo ""
+
+# 4a. Manual source finds CVE-2023-SHARED
+manual_data='{
+  "host": {
+    "ip": "192.168.1.400",
+    "hostname": "multi-source-test",
+    "vulnerabilities": [
+      {
+        "vid": "CVE-2019-1182",
+        "title": "Windows RDP Vulnerability",
+        "description": "Remote Desktop Protocol Remote Code Execution Vulnerability",
+        "riskscore": 8.1
+      }
+    ]
+  },
+  "source": {
+    "name": "manual",
+    "version": "1.0.0",
+    "config": "manual-analysis"
+  }
+}'
+
+make_api_call "POST" "/host/with-source" "$manual_data" "Manual source finds shared CVE"
+
+# 4b. RustScan also finds the same CVE
+rustscan_data='{
+  "host": {
+    "ip": "192.168.1.400",
+    "hostname": "multi-source-test",
+    "vulnerabilities": [
+      {
+        "vid": "CVE-2019-1182",
+        "title": "Windows RDP Vulnerability",
+        "description": "Remote Desktop Protocol Remote Code Execution Vulnerability (RustScan detected)",
+        "riskscore": 8.1
+      }
+    ]
+  },
+  "source": {
+    "name": "rustscan",
+    "version": "2.0.1",
+    "config": "rustscan -p 1-65535"
+  }
+}'
+
+make_api_call "POST" "/host/with-source" "$rustscan_data" "RustScan finds same CVE (should create separate record)"
+
+# Test Case 5: Port Tracking with Source Attribution
+echo "🔌 TEST CASE 5: Port Source Attribution"
+echo "Testing port discovery with source attribution"
+echo ""
+
+port_data='{
+  "host": {
+    "ip": "192.168.1.500",
+    "hostname": "port-test",
+    "ports": [
+      {
+        "id": 80,
+        "protocol": "tcp",
+        "state": "open"
+      },
+      {
+        "id": 443,
+        "protocol": "tcp", 
+        "state": "open"
+      },
+      {
+        "id": 22,
+        "protocol": "tcp",
+        "state": "closed"
+      }
+    ]
+  },
+  "source": {
+    "name": "nmap",
+    "version": "7.94",
+    "config": "nmap -p 1-1000"
+  }
+}'
+
+make_api_call "POST" "/host/with-source" "$port_data" "Port discovery with source attribution"
+
+echo "✅ Test data population completed!"
+echo ""
+echo "📋 Summary of test data created:"
+echo "   • 192.168.1.100 - Multi-source vulnerability test (nmap + agent)"
+echo "   • 192.168.1.200 - Temporal tracking test (nmap rescan)"
+echo "   • 192.168.1.300 - Backward compatibility test (legacy API)"
+echo "   • 192.168.1.400 - Same CVE from multiple sources (manual + rustscan)"
+echo "   • 192.168.1.500 - Port source attribution test (nmap)"
+echo ""
+echo "🔍 Run 'test-phase1-verify.sh' to validate the results" 

scripts/test-phase1-verify.sh

@@ -0,0 +1,284 @@
+#!/bin/bash
+
+# Test Verification Script for Phase 1 Validation
+# This script validates that source-aware functionality is working correctly
+# DELETE THIS SCRIPT after Phase 1 testing is complete
+
+set -e
+
+API_BASE="http://localhost:9001"
+echo "🔍 Starting Phase 1 Verification Tests..."
+echo "API Base: $API_BASE"
+echo ""
+
+# Color codes for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m' # No Color
+
+# Test tracking
+TESTS_PASSED=0
+TESTS_FAILED=0
+TOTAL_TESTS=0
+
+# Function to run test cases
+run_test() {
+    local test_name="$1"
+    local test_command="$2"
+    local expected_result="$3"
+    
+    TOTAL_TESTS=$((TOTAL_TESTS + 1))
+    echo -e "${BLUE}🧪 TEST: $test_name${NC}"
+    
+    if eval "$test_command"; then
+        echo -e "   ${GREEN}✅ PASSED${NC}"
+        TESTS_PASSED=$((TESTS_PASSED + 1))
+    else
+        echo -e "   ${RED}❌ FAILED${NC}"
+        TESTS_FAILED=$((TESTS_FAILED + 1))
+    fi
+    echo ""
+}
+
+# Function to make API calls and parse JSON
+api_get() {
+    local endpoint=$1
+    curl -s "$API_BASE$endpoint" | jq . 2>/dev/null || curl -s "$API_BASE$endpoint"
+}
+
+# Function to check if JSON contains specific data
+json_contains() {
+    local json_data="$1"
+    local jq_query="$2"
+    echo "$json_data" | jq -e "$jq_query" > /dev/null 2>&1
+}
+
+# Function to count items in JSON array
+json_count() {
+    local json_data="$1"
+    local jq_query="$2"
+    echo "$json_data" | jq "$jq_query | length" 2>/dev/null || echo "0"
+}
+
+echo "🔥 VERIFICATION TEST 1: Multi-Source Data Loss Prevention"
+echo "Verifying that host 192.168.1.100 has vulnerabilities from both nmap and agent"
+
+# Get host data with source information
+host_data=$(api_get "/host/192.168.1.100/sources")
+echo "📋 Host data retrieved:"
+echo "$host_data" | jq . 2>/dev/null || echo "$host_data"
+echo ""
+
+# Test 1a: Host should have vulnerabilities from nmap source
+nmap_vuln_count=$(echo "$host_data" | jq '[.vulnerability_sources[] | select(.source == "nmap")] | length' 2>/dev/null || echo "0")
+run_test "NMAP source vulnerabilities present" \
+    "[ $nmap_vuln_count -gt 0 ]" \
+    "Host should have vulnerabilities from nmap source"
+
+# Test 1b: Host should have vulnerabilities from agent source  
+agent_vuln_count=$(echo "$host_data" | jq '[.vulnerability_sources[] | select(.source == "agent")] | length' 2>/dev/null || echo "0")
+run_test "Agent source vulnerabilities present" \
+    "[ $agent_vuln_count -gt 0 ]" \
+    "Host should have vulnerabilities from agent source"
+
+# Test 1c: Host should have the shared CVE from both sources
+nmap_shared_count=$(echo "$host_data" | jq '[.vulnerability_sources[] | select(.VID == "CVE-2019-0708" and .source == "nmap")] | length' 2>/dev/null || echo "0")
+agent_shared_count=$(echo "$host_data" | jq '[.vulnerability_sources[] | select(.VID == "CVE-2019-0708" and .source == "agent")] | length' 2>/dev/null || echo "0")
+
+run_test "Shared CVE from both sources" \
+    "[ $nmap_shared_count -gt 0 ] && [ $agent_shared_count -gt 0 ]" \
+    "CVE-2019-0708 should exist from both nmap and agent sources"
+
+# Test 1d: NMAP-specific vulnerability should only come from nmap
+nmap_specific=$(echo "$host_data" | jq '[.vulnerability_sources[] | select(.VID == "CVE-2017-0144")] | length' 2>/dev/null || echo "0")
+run_test "NMAP-specific vulnerability isolation" \
+    "[ $nmap_specific -gt 0 ]" \
+    "CVE-2017-0144 should exist from nmap source"
+
+echo "🕐 VERIFICATION TEST 2: Temporal Tracking"
+echo "Verifying first_seen and last_seen timestamps work correctly"
+
+temporal_data=$(api_get "/host/192.168.1.200/sources")
+echo "📋 Temporal test data:"
+echo "$temporal_data" | jq . 2>/dev/null || echo "$temporal_data"
+echo ""
+
+# Test 2a: Vulnerability should have first_seen timestamp
+run_test "First seen timestamp exists" \
+    "json_contains '$temporal_data' '.vulnerability_sources[0].first_seen'" \
+    "Vulnerability should have first_seen timestamp"
+
+# Test 2b: Vulnerability should have last_seen timestamp
+run_test "Last seen timestamp exists" \
+    "json_contains '$temporal_data' '.vulnerability_sources[0].last_seen'" \
+    "Vulnerability should have last_seen timestamp"
+
+# Test 2c: Last seen should be more recent than first seen (due to our 3-second delay)
+first_seen=$(echo "$temporal_data" | jq -r '.vulnerability_sources[0].first_seen' 2>/dev/null || echo "")
+last_seen=$(echo "$temporal_data" | jq -r '.vulnerability_sources[0].last_seen' 2>/dev/null || echo "")
+
+run_test "Temporal ordering correct" \
+    "[ '$last_seen' != '$first_seen' ]" \
+    "Last seen should be different from first seen due to rescan"
+
+echo "🔄 VERIFICATION TEST 3: Backward Compatibility"
+echo "Verifying legacy API assigns 'unknown' source correctly"
+
+legacy_data=$(api_get "/host/192.168.1.300")
+echo "📋 Legacy API data:"
+echo "$legacy_data" | jq . 2>/dev/null || echo "$legacy_data"
+echo ""
+
+# Test 3a: Legacy host should exist
+legacy_ip=$(echo "$legacy_data" | jq -r '.ip // .IP // empty' 2>/dev/null || echo "")
+run_test "Legacy host exists" \
+    "[ '$legacy_ip' = '192.168.1.300' ]" \
+    "Host created via legacy API should exist"
+
+# Test 3b: Legacy host vulnerabilities should have 'unknown' source when queried with source info
+legacy_with_sources=$(api_get "/host/192.168.1.300/sources")
+run_test "Legacy data has unknown source attribution" \
+    "json_contains '$legacy_with_sources' '.vulnerability_sources[] | select(.source == \"unknown\")'" \
+    "Legacy API data should be attributed to unknown source"
+
+echo "🔀 VERIFICATION TEST 4: Multiple Sources on Same CVE"
+echo "Verifying same CVE from different sources creates separate records"
+
+multi_source_data=$(api_get "/host/192.168.1.400/sources")
+echo "📋 Multi-source test data:"
+echo "$multi_source_data" | jq . 2>/dev/null || echo "$multi_source_data"
+echo ""
+
+# Test 4a: CVE should exist from manual source
+manual_count=$(echo "$multi_source_data" | jq '[.vulnerability_sources[] | select(.VID == "CVE-2019-1182" and .source == "manual")] | length' 2>/dev/null || echo "0")
+run_test "CVE from manual source" \
+    "[ $manual_count -gt 0 ]" \
+    "CVE-2019-1182 should exist from manual source"
+
+# Test 4b: CVE should exist from rustscan source
+rustscan_count=$(echo "$multi_source_data" | jq '[.vulnerability_sources[] | select(.VID == "CVE-2019-1182" and .source == "rustscan")] | length' 2>/dev/null || echo "0")
+run_test "CVE from rustscan source" \
+    "[ $rustscan_count -gt 0 ]" \
+    "CVE-2019-1182 should exist from rustscan source"
+
+# Test 4c: Total count should be at least 2 (one from each source, plus unknown)
+shared_total=$(echo "$multi_source_data" | jq '[.vulnerability_sources[] | select(.VID == "CVE-2019-1182")] | length' 2>/dev/null || echo "0")
+run_test "Separate records for same CVE" \
+    "[ $shared_total -ge 2 ]" \
+    "CVE-2019-1182 should have at least 2 separate records (manual + rustscan)"
+
+echo "🔌 VERIFICATION TEST 5: Port Source Attribution"
+echo "Verifying port discovery includes source attribution"
+
+port_data=$(api_get "/host/192.168.1.500/sources")
+echo "📋 Port attribution data:"
+echo "$port_data" | jq . 2>/dev/null || echo "$port_data"
+echo ""
+
+# Test 5a: Host should have ports with source attribution
+run_test "Ports have source attribution" \
+    "json_contains '$port_data' '.port_sources[] | select(.source == \"nmap\")'" \
+    "Ports should be attributed to nmap source"
+
+# Test 5b: Specific ports should be present
+port_80_count=$(echo "$port_data" | jq '[.port_sources[] | select(.ID == 80)] | length' 2>/dev/null || echo "0")
+run_test "Port 80 discovered" \
+    "[ $port_80_count -gt 0 ]" \
+    "Port 80 should be discovered and recorded"
+
+echo "🔍 VERIFICATION TEST 6: New API Endpoints"
+echo "Verifying new source-aware endpoints are available"
+
+# Test 6a: Source coverage endpoint
+echo "Testing source coverage endpoint..."
+coverage_data=$(api_get "/sources/coverage")
+run_test "Source coverage endpoint" \
+    "[ $? -eq 0 ]" \
+    "Source coverage endpoint should be available"
+
+# Test 6b: Host history endpoint  
+echo "Testing host history endpoint..."
+history_data=$(api_get "/host/192.168.1.100/history")
+run_test "Host history endpoint" \
+    "[ $? -eq 0 ]" \
+    "Host history endpoint should be available"
+
+echo "📊 VERIFICATION TEST 7: Database Schema Validation"
+echo "Verifying database has new source-aware schema"
+
+# Test 7a: Direct database query (if PostgreSQL is accessible)
+if command -v psql &> /dev/null; then
+    echo "Testing database schema directly..."
+    
+    # Check if host_vulnerabilities table has new columns
+    schema_check=$(psql -h localhost -U postgres -d sirius -t -c "\d host_vulnerabilities" 2>/dev/null | grep -E "(source|first_seen|last_seen)" | wc -l)
+    run_test "Database schema updated" \
+        "[ $schema_check -gt 0 ]" \
+        "host_vulnerabilities table should have new source-aware columns"
+else
+    echo "⚠️  PostgreSQL CLI not available, skipping direct database tests"
+fi
+
+echo "🧹 VERIFICATION TEST 8: Data Integrity"
+echo "Verifying overall data integrity and consistency"
+
+# Test 8a: All test hosts should exist
+all_hosts=$(api_get "/host")
+host_count=$(echo "$all_hosts" | jq 'length' 2>/dev/null || echo "0")
+run_test "All test hosts exist" \
+    "[ $host_count -ge 5 ]" \
+    "At least 5 test hosts should exist"
+
+# Test 8b: No duplicate host records for same IP
+test_host_data=$(api_get "/host/192.168.1.100")
+test_host_ip=$(echo "$test_host_data" | jq -r '.ip // .IP // empty' 2>/dev/null || echo "")
+run_test "No duplicate host records" \
+    "[ '$test_host_ip' = '192.168.1.100' ]" \
+    "Host should exist uniquely without duplication"
+
+echo ""
+echo "🎯 TEST SUMMARY"
+echo "=============="
+echo -e "Total Tests: $TOTAL_TESTS"
+echo -e "${GREEN}Passed: $TESTS_PASSED${NC}"
+echo -e "${RED}Failed: $TESTS_FAILED${NC}"
+
+if [ $TESTS_FAILED -eq 0 ]; then
+    echo ""
+    echo -e "${GREEN}🎉 ALL TESTS PASSED! Phase 1 implementation is working correctly.${NC}"
+    echo ""
+    echo "✅ Key validations confirmed:"
+    echo "   • Multi-source data is preserved (no overwriting)"
+    echo "   • Source attribution is working correctly"
+    echo "   • Temporal tracking (first_seen/last_seen) is functional"
+    echo "   • Backward compatibility is maintained"
+    echo "   • Same CVE from different sources creates separate records"
+    echo "   • Port source attribution is working"
+    echo "   • New API endpoints are available"
+    echo ""
+    echo "🚀 Phase 1 is ready for production!"
+else
+    echo ""
+    echo -e "${RED}❌ SOME TESTS FAILED. Phase 1 implementation needs attention.${NC}"
+    echo ""
+    echo "🔧 Review the failed tests above and check:"
+    echo "   • Database migration completed successfully"
+    echo "   • New API endpoints are implemented"
+    echo "   • Source-aware functions replace Association(...).Replace() calls"
+    echo "   • Junction tables have source attribution fields"
+fi
+
+echo ""
+echo "📝 Next Steps:"
+echo "   • If tests pass: Proceed to Phase 2 (Scanner Integration)"
+echo "   • If tests fail: Review implementation and fix issues"
+echo "   • After verification: Delete these test scripts"
+
+# Cleanup reminder
+echo ""
+echo "⚠️  REMEMBER: Delete test scripts after verification:"
+echo "   rm scripts/test-phase1-populate.sh"
+echo "   rm scripts/test-phase1-verify.sh"
+echo "   rm scripts/test-phase1-cleanup.sh" 

scripts/test-phase2-integration.sh

@@ -0,0 +1,568 @@
+#!/bin/bash
+
+# Phase 2 Integration Test Suite
+# Tests source attribution functionality across all scanner types
+# Validates multi-source scenarios and data integrity
+
+set -e
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m' # No Color
+
+# Test configuration
+API_BASE_URL="http://localhost:9001"
+TEST_HOST_IP="192.168.1.100"
+TEST_CVE_1="CVE-2017-0144"  # EternalBlue
+TEST_CVE_2="CVE-2019-0708"  # BlueKeep
+TEST_CVE_3="CVE-2021-34527" # PrintNightmare
+
+# Test counters
+TOTAL_TESTS=0
+PASSED_TESTS=0
+FAILED_TESTS=0
+
+# Logging functions
+log_info() {
+    echo -e "${BLUE}[INFO]${NC} $1"
+}
+
+log_success() {
+    echo -e "${GREEN}[SUCCESS]${NC} $1"
+    ((PASSED_TESTS++))
+}
+
+log_error() {
+    echo -e "${RED}[ERROR]${NC} $1"
+    ((FAILED_TESTS++))
+}
+
+log_warning() {
+    echo -e "${YELLOW}[WARNING]${NC} $1"
+}
+
+# Test helper functions
+run_test() {
+    local test_name="$1"
+    local test_function="$2"
+    
+    ((TOTAL_TESTS++))
+    log_info "Running test: $test_name"
+    
+    if $test_function; then
+        log_success "$test_name"
+    else
+        log_error "$test_name"
+    fi
+    echo ""
+}
+
+# API helper functions
+api_post() {
+    local endpoint="$1"
+    local data="$2"
+    local expected_status="${3:-200}"
+    
+    response=$(curl -s -w "HTTPSTATUS:%{http_code}" \
+        -X POST \
+        -H "Content-Type: application/json" \
+        -d "$data" \
+        "$API_BASE_URL$endpoint")
+    
+    http_code=$(echo "$response" | tr -d '\n' | sed -e 's/.*HTTPSTATUS://')
+    body=$(echo "$response" | sed -e 's/HTTPSTATUS:.*//g')
+    
+    if [[ "$http_code" -eq "$expected_status" ]]; then
+        echo "$body"
+        return 0
+    else
+        log_error "API call failed. Expected: $expected_status, Got: $http_code"
+        echo "$body"
+        return 1
+    fi
+}
+
+api_get() {
+    local endpoint="$1"
+    local expected_status="${2:-200}"
+    
+    response=$(curl -s -w "HTTPSTATUS:%{http_code}" \
+        -X GET \
+        -H "Content-Type: application/json" \
+        "$API_BASE_URL$endpoint")
+    
+    http_code=$(echo "$response" | tr -d '\n' | sed -e 's/.*HTTPSTATUS://')
+    body=$(echo "$response" | sed -e 's/HTTPSTATUS:.*//g')
+    
+    if [[ "$http_code" -eq "$expected_status" ]]; then
+        echo "$body"
+        return 0
+    else
+        log_error "API call failed. Expected: $expected_status, Got: $http_code"
+        echo "$body"
+        return 1
+    fi
+}
+
+# Test functions
+test_api_connectivity() {
+    log_info "Testing API connectivity..."
+    if api_get "/host" >/dev/null 2>&1; then
+        return 0
+    else
+        log_error "Cannot connect to API at $API_BASE_URL"
+        return 1
+    fi
+}
+
+test_legacy_host_submission() {
+    log_info "Testing legacy host submission (backward compatibility)..."
+    
+    local host_data='{
+        "ip": "'$TEST_HOST_IP'",
+        "hostname": "test-legacy-host",
+        "os": "Linux",
+        "ports": [
+            {
+                "port": 22,
+                "protocol": "tcp",
+                "state": "open",
+                "service": "ssh"
+            }
+        ],
+        "vulnerabilities": [
+            {
+                "vid": "'$TEST_CVE_1'",
+                "title": "EternalBlue SMB Vulnerability",
+                "description": "Legacy submission test",
+                "severity": "critical",
+                "cvss_score": 9.3,
+                "port": 445
+            }
+        ]
+    }'
+    
+    if api_post "/host" "$host_data" 200 >/dev/null; then
+        # Verify the host was created with legacy source
+        local host_response
+        host_response=$(api_get "/host/$TEST_HOST_IP")
+        if echo "$host_response" | grep -q "$TEST_HOST_IP"; then
+            return 0
+        else
+            log_error "Legacy host not found after submission"
+            return 1
+        fi
+    else
+        return 1
+    fi
+}
+
+test_source_aware_nmap_submission() {
+    log_info "Testing source-aware nmap submission..."
+    
+    local host_data='{
+        "host": {
+            "ip": "192.168.1.101",
+            "hostname": "test-nmap-host",
+            "os": "Linux",
+            "ports": [
+                {
+                    "port": 80,
+                    "protocol": "tcp",
+                    "state": "open",
+                    "service": "http"
+                }
+            ],
+            "vulnerabilities": [
+                {
+                    "vid": "'$TEST_CVE_2'",
+                    "title": "BlueKeep RDP Vulnerability",
+                    "description": "Nmap source test",
+                    "severity": "critical",
+                    "cvss_score": 9.8,
+                    "port": 3389
+                }
+            ]
+        },
+        "source": {
+            "name": "nmap",
+            "version": "7.94",
+            "config": "ports:1-1000;aggressive:true;template:comprehensive"
+        }
+    }'
+    
+    if api_post "/host/with-source" "$host_data" 200 >/dev/null; then
+        # Verify source attribution
+        local host_response
+        host_response=$(api_get "/host/192.168.1.101/sources")
+        if echo "$host_response" | grep -q "nmap" && echo "$host_response" | grep -q "7.94"; then
+            return 0
+        else
+            log_error "Nmap source attribution not found"
+            return 1
+        fi
+    else
+        return 1
+    fi
+}
+
+test_source_aware_agent_submission() {
+    log_info "Testing source-aware agent submission..."
+    
+    local host_data='{
+        "host": {
+            "ip": "192.168.1.102",
+            "hostname": "test-agent-host",
+            "os": "Ubuntu 20.04",
+            "ports": [
+                {
+                    "port": 443,
+                    "protocol": "tcp",
+                    "state": "open",
+                    "service": "https"
+                }
+            ],
+            "vulnerabilities": [
+                {
+                    "vid": "'$TEST_CVE_3'",
+                    "title": "PrintNightmare Vulnerability",
+                    "description": "Agent source test",
+                    "severity": "high",
+                    "cvss_score": 8.8,
+                    "port": 445
+                }
+            ]
+        },
+        "source": {
+            "name": "sirius-agent",
+            "version": "1.0.0",
+            "config": "os:linux;arch:amd64;go:go1.21.0;host:test-agent"
+        }
+    }'
+    
+    if api_post "/host/with-source" "$host_data" 200 >/dev/null; then
+        # Verify source attribution
+        local host_response
+        host_response=$(api_get "/host/192.168.1.102/sources")
+        if echo "$host_response" | grep -q "sirius-agent" && echo "$host_response" | grep -q "1.0.0"; then
+            return 0
+        else
+            log_error "Agent source attribution not found"
+            return 1
+        fi
+    else
+        return 1
+    fi
+}
+
+test_multi_source_same_host() {
+    log_info "Testing multiple sources for same host..."
+    
+    local test_ip="192.168.1.103"
+    
+    # Submit from nmap source
+    local nmap_data='{
+        "host": {
+            "ip": "'$test_ip'",
+            "hostname": "multi-source-host",
+            "os": "Linux",
+            "ports": [
+                {
+                    "port": 22,
+                    "protocol": "tcp",
+                    "state": "open",
+                    "service": "ssh"
+                }
+            ],
+            "vulnerabilities": [
+                {
+                    "vid": "'$TEST_CVE_1'",
+                    "title": "EternalBlue from Nmap",
+                    "description": "Multi-source test - nmap",
+                    "severity": "critical",
+                    "cvss_score": 9.3,
+                    "port": 445
+                }
+            ]
+        },
+        "source": {
+            "name": "nmap",
+            "version": "7.94",
+            "config": "multi-source-test"
+        }
+    }'
+    
+    # Submit from agent source
+    local agent_data='{
+        "host": {
+            "ip": "'$test_ip'",
+            "hostname": "multi-source-host",
+            "os": "Linux",
+            "ports": [
+                {
+                    "port": 80,
+                    "protocol": "tcp",
+                    "state": "open",
+                    "service": "http"
+                }
+            ],
+            "vulnerabilities": [
+                {
+                    "vid": "'$TEST_CVE_2'",
+                    "title": "BlueKeep from Agent",
+                    "description": "Multi-source test - agent",
+                    "severity": "critical",
+                    "cvss_score": 9.8,
+                    "port": 3389
+                }
+            ]
+        },
+        "source": {
+            "name": "sirius-agent",
+            "version": "1.0.0",
+            "config": "multi-source-test"
+        }
+    }'
+    
+    # Submit both sources
+    if api_post "/host/with-source" "$nmap_data" 200 >/dev/null && \
+       api_post "/host/with-source" "$agent_data" 200 >/dev/null; then
+        
+        # Verify both sources are present
+        local sources_response
+        sources_response=$(api_get "/host/$test_ip/sources")
+        if echo "$sources_response" | grep -q "nmap" && echo "$sources_response" | grep -q "sirius-agent"; then
+            # Verify both vulnerabilities are present
+            local host_response
+            host_response=$(api_get "/host/$test_ip")
+            if echo "$host_response" | grep -q "$TEST_CVE_1" && echo "$host_response" | grep -q "$TEST_CVE_2"; then
+                return 0
+            else
+                log_error "Not all vulnerabilities found from multiple sources"
+                return 1
+            fi
+        else
+            log_error "Not all sources found for multi-source host"
+            return 1
+        fi
+    else
+        return 1
+    fi
+}
+
+test_source_detection_headers() {
+    log_info "Testing automatic source detection from headers..."
+    
+    local host_data='{
+        "ip": "192.168.1.104",
+        "hostname": "header-detection-host",
+        "os": "Linux",
+        "ports": [
+            {
+                "port": 8080,
+                "protocol": "tcp",
+                "state": "open",
+                "service": "http-proxy"
+            }
+        ],
+        "vulnerabilities": []
+    }'
+    
+    # Submit with custom headers that should be detected
+    response=$(curl -s -w "HTTPSTATUS:%{http_code}" \
+        -X POST \
+        -H "Content-Type: application/json" \
+        -H "X-Scanner-Name: rustscan" \
+        -H "X-Scanner-Version: 2.1.1" \
+        -H "X-Scanner-Config: fast-scan" \
+        -d "$host_data" \
+        "$API_BASE_URL/host")
+    
+    http_code=$(echo "$response" | tr -d '\n' | sed -e 's/.*HTTPSTATUS://')
+    
+    if [[ "$http_code" -eq 200 ]]; then
+        # Check if source was detected and attributed
+        local host_response
+        host_response=$(api_get "/host/192.168.1.104?include_source=true")
+        if echo "$host_response" | grep -q "rustscan" && echo "$host_response" | grep -q "2.1.1"; then
+            return 0
+        else
+            log_error "Source detection from headers failed"
+            return 1
+        fi
+    else
+        return 1
+    fi
+}
+
+test_backward_compatibility_response_format() {
+    log_info "Testing backward compatibility response formats..."
+    
+    # Test legacy client detection
+    local host_data='{
+        "ip": "192.168.1.105",
+        "hostname": "legacy-client-host",
+        "os": "Windows",
+        "ports": [],
+        "vulnerabilities": []
+    }'
+    
+    # Submit with legacy User-Agent
+    response=$(curl -s \
+        -X POST \
+        -H "Content-Type: application/json" \
+        -H "User-Agent: legacy-client/1.0" \
+        -H "X-API-Version: 1.0" \
+        -d "$host_data" \
+        "$API_BASE_URL/host")
+    
+    # Check response format (should be legacy format)
+    if echo "$response" | grep -q "Host added successfully" && ! echo "$response" | grep -q "source"; then
+        return 0
+    else
+        log_error "Legacy response format not maintained"
+        return 1
+    fi
+}
+
+test_data_integrity_across_sources() {
+    log_info "Testing data integrity across multiple sources..."
+    
+    local test_ip="192.168.1.106"
+    
+    # Submit same vulnerability from different sources
+    local source1_data='{
+        "host": {
+            "ip": "'$test_ip'",
+            "hostname": "integrity-test-host",
+            "os": "Linux",
+            "ports": [],
+            "vulnerabilities": [
+                {
+                    "vid": "'$TEST_CVE_1'",
+                    "title": "EternalBlue",
+                    "description": "From source 1",
+                    "severity": "critical",
+                    "cvss_score": 9.3,
+                    "port": 445
+                }
+            ]
+        },
+        "source": {
+            "name": "nmap",
+            "version": "7.94",
+            "config": "integrity-test-1"
+        }
+    }'
+    
+    local source2_data='{
+        "host": {
+            "ip": "'$test_ip'",
+            "hostname": "integrity-test-host",
+            "os": "Linux",
+            "ports": [],
+            "vulnerabilities": [
+                {
+                    "vid": "'$TEST_CVE_1'",
+                    "title": "EternalBlue",
+                    "description": "From source 2",
+                    "severity": "critical",
+                    "cvss_score": 9.3,
+                    "port": 445
+                }
+            ]
+        },
+        "source": {
+            "name": "rustscan",
+            "version": "2.1.1",
+            "config": "integrity-test-2"
+        }
+    }'
+    
+    # Submit from both sources
+    if api_post "/host/with-source" "$source1_data" 200 >/dev/null && \
+       api_post "/host/with-source" "$source2_data" 200 >/dev/null; then
+        
+        # Verify both source records exist for the same vulnerability
+        local sources_response
+        sources_response=$(api_get "/host/$test_ip/sources")
+        
+        # Count occurrences of the CVE (should be 2 - one from each source)
+        local cve_count
+        cve_count=$(echo "$sources_response" | grep -o "$TEST_CVE_1" | wc -l)
+        
+        if [[ "$cve_count" -ge 2 ]]; then
+            return 0
+        else
+            log_error "Data integrity test failed - CVE not preserved across sources"
+            return 1
+        fi
+    else
+        return 1
+    fi
+}
+
+# Cleanup function
+cleanup_test_data() {
+    log_info "Cleaning up test data..."
+    
+    # Delete test hosts (if delete endpoint exists)
+    local test_ips=("192.168.1.100" "192.168.1.101" "192.168.1.102" "192.168.1.103" "192.168.1.104" "192.168.1.105" "192.168.1.106")
+    
+    for ip in "${test_ips[@]}"; do
+        curl -s -X POST \
+            -H "Content-Type: application/json" \
+            -d '{"ip": "'$ip'"}' \
+            "$API_BASE_URL/host/delete" >/dev/null 2>&1 || true
+    done
+    
+    log_info "Cleanup completed"
+}
+
+# Main test execution
+main() {
+    echo "=========================================="
+    echo "Phase 2 Integration Test Suite"
+    echo "Testing Source Attribution Functionality"
+    echo "=========================================="
+    echo ""
+    
+    # Pre-test cleanup
+    cleanup_test_data
+    
+    # Run all tests
+    run_test "API Connectivity" test_api_connectivity
+    run_test "Legacy Host Submission" test_legacy_host_submission
+    run_test "Source-Aware Nmap Submission" test_source_aware_nmap_submission
+    run_test "Source-Aware Agent Submission" test_source_aware_agent_submission
+    run_test "Multi-Source Same Host" test_multi_source_same_host
+    run_test "Source Detection from Headers" test_source_detection_headers
+    run_test "Backward Compatibility Response Format" test_backward_compatibility_response_format
+    run_test "Data Integrity Across Sources" test_data_integrity_across_sources
+    
+    # Post-test cleanup
+    cleanup_test_data
+    
+    # Test summary
+    echo "=========================================="
+    echo "Test Summary"
+    echo "=========================================="
+    echo "Total Tests: $TOTAL_TESTS"
+    echo -e "Passed: ${GREEN}$PASSED_TESTS${NC}"
+    echo -e "Failed: ${RED}$FAILED_TESTS${NC}"
+    echo ""
+    
+    if [[ $FAILED_TESTS -eq 0 ]]; then
+        echo -e "${GREEN}🎉 All tests passed! Phase 2 integration is working correctly.${NC}"
+        exit 0
+    else
+        echo -e "${RED}❌ Some tests failed. Please review the errors above.${NC}"
+        exit 1
+    fi
+}
+
+# Run main function
+main "$@" 

scripts/test-windows-registry.ps1

@@ -0,0 +1,110 @@
+# Windows Registry Vulnerability Detection Test Script
+# This script tests the Windows registry detection capabilities
+
+Write-Host "🔍 Starting Windows Registry Vulnerability Detection Test" -ForegroundColor Cyan
+Write-Host "===============================================" -ForegroundColor Cyan
+
+# Set up test registry entries
+Write-Host "`n📝 Setting up test registry entries..." -ForegroundColor Yellow
+
+try {
+    # Create test vulnerable app registry entry
+    Write-Host "   Creating test vulnerable app entry..."
+    reg add "HKLM\SOFTWARE\SiriusTest\VulnerableApp" /v "Version" /t REG_SZ /d "16.0.15330.20264" /f | Out-Null
+    
+    # Set UAC disabled (for testing)
+    Write-Host "   Setting UAC disabled (test)..."
+    reg add "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System" /v "EnableLUA" /t REG_DWORD /d 0 /f | Out-Null
+    
+    # Set Print Spooler enabled (for testing)
+    Write-Host "   Setting Print Spooler enabled (test)..."
+    reg add "HKLM\SYSTEM\CurrentControlSet\Services\Spooler" /v "Start" /t REG_DWORD /d 2 /f | Out-Null
+    
+    Write-Host "✅ Test registry entries created successfully" -ForegroundColor Green
+    
+} catch {
+    Write-Host "❌ Failed to create test registry entries: $($_.Exception.Message)" -ForegroundColor Red
+    exit 1
+}
+
+# Run the agent scan with registry templates
+Write-Host "`n🚀 Running agent scan with registry detection..." -ForegroundColor Yellow
+
+try {
+    # Set environment variables for standalone mode
+    $env:SERVER_ADDRESS = "localhost:50051"
+    $env:AGENT_ID = "windows-test-agent"
+    $env:HOST_ID = "windows-test-host"
+    $env:API_BASE_URL = "http://localhost:9001"
+    $env:ENABLE_SCRIPTING = "true"
+    
+    # Check if agent executable exists
+    $agentPath = ".\agent.exe"
+    if (-not (Test-Path $agentPath)) {
+        $agentPath = ".\agent-windows.exe"
+        if (-not (Test-Path $agentPath)) {
+            Write-Host "❌ Agent executable not found. Please build with: GOOS=windows GOARCH=amd64 go build -o agent-windows.exe ." -ForegroundColor Red
+            exit 1
+        }
+    }
+    
+    Write-Host "   Using agent: $agentPath"
+    Write-Host "   Templates directory: .\templates\"
+    
+    # Run the scan
+    Write-Host "`n📊 Executing registry vulnerability scan..."
+    & $agentPath scan 2>&1 | Tee-Object -FilePath "windows-registry-test.log"
+    
+    $scanResult = $LASTEXITCODE
+    
+    if ($scanResult -eq 0) {
+        Write-Host "✅ Agent scan completed successfully" -ForegroundColor Green
+    } else {
+        Write-Host "⚠️  Agent scan completed with exit code: $scanResult" -ForegroundColor Orange
+    }
+    
+} catch {
+    Write-Host "❌ Failed to run agent scan: $($_.Exception.Message)" -ForegroundColor Red
+}
+
+# Clean up test registry entries
+Write-Host "`n🧹 Cleaning up test registry entries..." -ForegroundColor Yellow
+
+try {
+    Write-Host "   Removing test vulnerable app entry..."
+    reg delete "HKLM\SOFTWARE\SiriusTest" /f 2>$null | Out-Null
+    
+    Write-Host "   Restoring UAC to enabled..."
+    reg add "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System" /v "EnableLUA" /t REG_DWORD /d 1 /f | Out-Null
+    
+    Write-Host "✅ Test registry entries cleaned up" -ForegroundColor Green
+    
+} catch {
+    Write-Host "⚠️  Failed to clean up some registry entries: $($_.Exception.Message)" -ForegroundColor Orange
+}
+
+# Display results
+Write-Host "`n📋 Test Results Summary:" -ForegroundColor Cyan
+Write-Host "===============================================" -ForegroundColor Cyan
+
+if (Test-Path "windows-registry-test.log") {
+    Write-Host "📄 Detailed log saved to: windows-registry-test.log"
+    Write-Host "`n🔍 Looking for registry detection results..."
+    
+    $logContent = Get-Content "windows-registry-test.log" -Raw
+    
+    if ($logContent -match "template.*registry" -or $logContent -match "vulnerable.*found") {
+        Write-Host "✅ Registry detection functionality appears to be working" -ForegroundColor Green
+    } else {
+        Write-Host "⚠️  Registry detection results not clearly visible in logs" -ForegroundColor Orange
+    }
+    
+    if ($logContent -match "error" -or $logContent -match "failed") {
+        Write-Host "⚠️  Some errors detected in logs - check windows-registry-test.log for details" -ForegroundColor Orange
+    }
+} else {
+    Write-Host "⚠️  Log file not found" -ForegroundColor Orange
+}
+
+Write-Host "`n🎯 Registry Detection Test Completed!" -ForegroundColor Cyan
+Write-Host "   Check the above output and windows-registry-test.log for detailed results" -ForegroundColor Gray 

scripts/update-go-api.sh

@@ -0,0 +1,126 @@
+#!/bin/bash
+#
+# Bulk Update go-api Dependency Across All Projects
+#
+# This script updates the github.com/SiriusScan/go-api dependency to the
+# latest version (or a specified version) across all projects in the workspace.
+#
+# Usage:
+#   ./scripts/update-go-api.sh [version]
+#
+# Examples:
+#   ./scripts/update-go-api.sh          # Update to latest
+#   ./scripts/update-go-api.sh v0.0.10  # Update to specific version
+#
+
+set -euo pipefail
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m' # No Color
+
+# Get the root directory of the repository
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+
+# Version to update to (default: latest)
+VERSION="${1:-latest}"
+
+echo -e "${BLUE}🔄 Bulk Go API Update Script${NC}"
+echo -e "${BLUE}================================${NC}\n"
+
+if [ "$VERSION" = "latest" ]; then
+    echo -e "${YELLOW}📦 Fetching latest version from GitHub...${NC}"
+    LATEST_TAG=$(curl -s https://api.github.com/repos/SiriusScan/go-api/releases/latest | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')
+    if [ -z "$LATEST_TAG" ]; then
+        echo -e "${RED}❌ Failed to fetch latest version${NC}"
+        exit 1
+    fi
+    VERSION="$LATEST_TAG"
+    echo -e "${GREEN}✅ Latest version: $VERSION${NC}\n"
+else
+    echo -e "${YELLOW}📌 Updating to specified version: $VERSION${NC}\n"
+fi
+
+# Find all go.mod files that use go-api
+echo -e "${BLUE}🔍 Finding projects that use go-api...${NC}"
+GO_MOD_FILES=$(find "$REPO_ROOT" -name "go.mod" -type f | grep -v "minor-projects/go-api" || true)
+
+if [ -z "$GO_MOD_FILES" ]; then
+    echo -e "${YELLOW}⚠️  No projects found using go-api${NC}"
+    exit 0
+fi
+
+# Filter for files that actually contain go-api dependency
+PROJECTS_TO_UPDATE=()
+while IFS= read -r go_mod; do
+    if grep -q "github.com/SiriusScan/go-api" "$go_mod"; then
+        PROJECTS_TO_UPDATE+=("$go_mod")
+    fi
+done <<< "$GO_MOD_FILES"
+
+if [ ${#PROJECTS_TO_UPDATE[@]} -eq 0 ]; then
+    echo -e "${YELLOW}⚠️  No projects found with go-api dependency${NC}"
+    exit 0
+fi
+
+echo -e "${GREEN}Found ${#PROJECTS_TO_UPDATE[@]} project(s) to update:${NC}"
+for project in "${PROJECTS_TO_UPDATE[@]}"; do
+    echo -e "  • $(dirname "$project" | sed "s|$REPO_ROOT/||")"
+done
+echo ""
+
+# Ask for confirmation
+read -p "$(echo -e ${YELLOW}"Continue with update? (y/N): "${NC})" -n 1 -r
+echo
+if [[ ! $REPLY =~ ^[Yy]$ ]]; then
+    echo -e "${RED}❌ Update cancelled${NC}"
+    exit 0
+fi
+
+echo ""
+
+# Update each project
+UPDATED=0
+FAILED=0
+for go_mod in "${PROJECTS_TO_UPDATE[@]}"; do
+    PROJECT_DIR=$(dirname "$go_mod")
+    PROJECT_NAME=$(basename "$PROJECT_DIR")
+    
+    echo -e "${BLUE}📦 Updating $PROJECT_NAME...${NC}"
+    
+    cd "$PROJECT_DIR"
+    
+    # Run go get and go mod tidy
+    if go get "github.com/SiriusScan/go-api@$VERSION" && go mod tidy; then
+        echo -e "${GREEN}✅ $PROJECT_NAME updated successfully${NC}\n"
+        ((UPDATED++))
+    else
+        echo -e "${RED}❌ Failed to update $PROJECT_NAME${NC}\n"
+        ((FAILED++))
+    fi
+done
+
+# Summary
+echo -e "${BLUE}================================${NC}"
+echo -e "${BLUE}📊 Update Summary${NC}"
+echo -e "${BLUE}================================${NC}"
+echo -e "${GREEN}✅ Successfully updated: $UPDATED${NC}"
+if [ $FAILED -gt 0 ]; then
+    echo -e "${RED}❌ Failed: $FAILED${NC}"
+fi
+echo ""
+
+if [ $UPDATED -gt 0 ]; then
+    echo -e "${YELLOW}💡 Next steps:${NC}"
+    echo -e "  1. Review the changes: ${BLUE}git status${NC}"
+    echo -e "  2. Test the updates: ${BLUE}docker compose down && docker compose up -d --build${NC}"
+    echo -e "  3. Commit the changes: ${BLUE}git add . && git commit -m 'chore(deps): update go-api to $VERSION'${NC}"
+    echo ""
+fi
+
+echo -e "${GREEN}✨ Done!${NC}"
+

scripts/validate-ci-overhaul.sh

@@ -0,0 +1,150 @@
+#!/bin/bash
+
+# Validate CI/CD container builds locally (no separate GHCR base images).
+#
+# Builds infra + application images and validates docker-compose configs.
+#
+# Usage: ./scripts/validate-ci-overhaul.sh [--full-compose]
+#   --full-compose  Also bring up infrastructure + one app service to verify startup
+
+set -e
+
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m'
+
+FULL_COMPOSE=false
+if [[ "${1:-}" == "--full-compose" ]]; then
+  FULL_COMPOSE=true
+fi
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
+LOG_DIR="$PROJECT_ROOT/testing/logs"
+TIMESTAMP=$(date +"%Y%m%d_%H%M%S")
+LOG_FILE="$LOG_DIR/validate_ci_overhaul_$TIMESTAMP.log"
+
+mkdir -p "$LOG_DIR"
+
+log() {
+  echo -e "$1" | tee -a "$LOG_FILE"
+}
+
+run_step() {
+  local step_name="$1"
+  local cmd="$2"
+  log "${BLUE}▶ $step_name${NC}"
+  log "  $cmd"
+  if eval "$cmd" >> "$LOG_FILE" 2>&1; then
+    log "${GREEN}  ✓ $step_name${NC}"
+    return 0
+  else
+    log "${RED}  ✗ $step_name FAILED${NC}"
+    log "  See $LOG_FILE for details"
+    return 1
+  fi
+}
+
+cd "$PROJECT_ROOT"
+
+log ""
+log "${BLUE}═══════════════════════════════════════════════════════════════${NC}"
+log "${BLUE}  CI/CD container local validation${NC}"
+log "${BLUE}═══════════════════════════════════════════════════════════════${NC}"
+log "Project Root: $PROJECT_ROOT"
+log "Log File: $LOG_FILE"
+log ""
+
+# ─────────────────────────────────────────────────────────────────────────────
+# Phase 1: Infrastructure containers
+# ─────────────────────────────────────────────────────────────────────────────
+log "${YELLOW}Phase 1: Infrastructure Containers${NC}"
+log ""
+
+run_step "Build sirius-postgres" \
+  "docker build -t sirius-postgres:test ./sirius-postgres/" || exit 1
+
+run_step "Build sirius-rabbitmq" \
+  "docker build -t sirius-rabbitmq:test ./sirius-rabbitmq/" || exit 1
+
+run_step "Build sirius-valkey" \
+  "docker build -t sirius-valkey:test ./sirius-valkey/" || exit 1
+
+log ""
+log "${YELLOW}Phase 2: Application Containers${NC}"
+log ""
+
+run_step "Build sirius-api (runner stage)" \
+  "docker build -t sirius-api:test ./sirius-api/ --target runner" || exit 1
+
+run_step "Build sirius-ui (production stage)" \
+  "docker build -t sirius-ui:test ./sirius-ui/ --target production" || exit 1
+
+run_step "Build sirius-engine (development stage)" \
+  "docker build -t sirius-engine:dev ./sirius-engine/ --target development" || exit 1
+
+run_step "Build sirius-engine (runtime stage)" \
+  "docker build -t sirius-engine:runtime ./sirius-engine/ --target runtime" || exit 1
+
+log ""
+log "${YELLOW}Phase 3: Runtime Contracts${NC}"
+log ""
+
+run_step "sirius-postgres entrypoint contract" \
+  "docker run --rm --entrypoint /bin/sh sirius-postgres:test -c 'test -x /usr/local/bin/start-with-monitor.sh'" || exit 1
+
+run_step "sirius-engine runtime binary contract (bash, nmap, rustscan, pwsh, psql)" \
+  "docker run --rm --entrypoint /bin/bash sirius-engine:runtime -c 'command -v bash >/dev/null && command -v nmap >/dev/null && command -v rustscan >/dev/null && command -v pwsh >/dev/null && command -v psql >/dev/null'" || exit 1
+
+log ""
+log "${YELLOW}Phase 4: Docker Compose Validation${NC}"
+log ""
+
+export POSTGRES_PASSWORD="${POSTGRES_PASSWORD:-test-postgres-password}"
+export NEXTAUTH_SECRET="${NEXTAUTH_SECRET:-test-nextauth-secret}"
+export INITIAL_ADMIN_PASSWORD="${INITIAL_ADMIN_PASSWORD:-test-admin-password}"
+mkdir -p secrets
+printf '%s\n' "${SIRIUS_INTERNAL_API_KEY_TEST_VALUE:-test-api-key}" > secrets/sirius_api_key.txt
+
+run_step "Base docker-compose config" \
+  "docker compose config --quiet" || exit 1
+
+run_step "Development docker-compose config" \
+  "docker compose -f docker-compose.yaml -f docker-compose.dev.yaml config --quiet" || exit 1
+
+log ""
+log "${GREEN}═══════════════════════════════════════════════════════════════${NC}"
+log "${GREEN}  All builds and validations passed!${NC}"
+log "${GREEN}═══════════════════════════════════════════════════════════════${NC}"
+log ""
+log "You can now confidently commit and push. CI builds application images on"
+log "native amd64/arm64 runners (ci.yml), merges manifests, and runs integration tests."
+log ""
+
+# Optional: bring up a minimal stack to verify runtime
+if $FULL_COMPOSE; then
+  log "${YELLOW}Phase 5: Minimal Compose Stack (--full-compose)${NC}"
+  log ""
+  log "Starting postgres, rabbitmq, valkey, api for 30s to verify startup..."
+  docker compose up -d sirius-postgres sirius-rabbitmq sirius-valkey 2>>"$LOG_FILE" || true
+  sleep 10
+  docker compose up -d sirius-api 2>>"$LOG_FILE" || true
+  sleep 20
+  if docker compose exec -T sirius-api wget -q -O /dev/null http://127.0.0.1:9001/health 2>/dev/null; then
+    log "${GREEN}  ✓ sirius-api health check passed${NC}"
+  else
+    log "${YELLOW}  ⚠ sirius-api may still be starting; check: docker compose logs sirius-api${NC}"
+  fi
+  docker compose down 2>>"$LOG_FILE" || true
+  log ""
+fi
+
+# Cleanup test tags (optional, comment out to keep for debugging)
+log "${YELLOW}Cleaning up test images...${NC}"
+docker rmi sirius-postgres:test sirius-rabbitmq:test sirius-valkey:test sirius-api:test sirius-ui:test sirius-engine:dev sirius-engine:runtime 2>/dev/null || true
+log ""
+
+log "Full log: $LOG_FILE"
+exit 0

scripts/validate-public-compose-path.sh

@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(dirname "${SCRIPT_DIR}")"
+
+IMAGE_TAG_VALUE="${1:-${IMAGE_TAG:-latest}}"
+: "${KEEP_PUBLIC_STACK_RUNNING:=0}"
+: "${SIRIUS_IMAGE_PULL_POLICY:=always}"
+
+cleanup() {
+  if [ "${KEEP_PUBLIC_STACK_RUNNING}" = "1" ]; then
+    return
+  fi
+
+  docker compose down --volumes --remove-orphans >/dev/null 2>&1 || true
+}
+
+trap cleanup EXIT
+
+cd "${PROJECT_ROOT}"
+
+echo "Resetting public compose stack state..."
+docker compose down --volumes --remove-orphans >/dev/null 2>&1 || true
+
+echo "Generating runtime config with installer..."
+# Run the installer container as the invoking host user so the resulting
+# .env / secrets/ files are readable by subsequent steps (e.g. `docker
+# compose config` reading .env). Without this, the installer writes
+# .env as root:root mode 0600 and the next compose invocation fails with
+# "open .env: permission denied" — silently producing zero image refs.
+SIRIUS_INSTALLER_UID="$(id -u)" SIRIUS_INSTALLER_GID="$(id -g)" \
+  docker compose -f docker-compose.installer.yaml run --rm --build sirius-installer --non-interactive --no-print-secrets
+
+# Defensive: even with the user override above, guarantee .env is readable
+# by the current user. Harmless if already readable.
+if [ -f "${PROJECT_ROOT}/.env" ] && [ ! -r "${PROJECT_ROOT}/.env" ]; then
+  echo "::warning::.env not readable by $(id -un); attempting chmod via sudo"
+  sudo chown "$(id -u):$(id -g)" "${PROJECT_ROOT}/.env" 2>/dev/null || true
+fi
+
+export IMAGE_TAG="${IMAGE_TAG_VALUE}"
+export SIRIUS_IMAGE_PULL_POLICY
+
+echo "Verifying anonymous GHCR access for IMAGE_TAG=${IMAGE_TAG}..."
+bash scripts/verify-ghcr-public-access.sh "${IMAGE_TAG}"
+
+echo "Pulling compose-rendered public images..."
+docker compose pull
+
+echo "Starting public compose stack..."
+docker compose up -d
+
+echo "Verifying runtime auth contract..."
+bash scripts/verify-runtime-auth-contract.sh
+
+echo "Public compose path validated for IMAGE_TAG=${IMAGE_TAG}."

scripts/verify-ghcr-public-access.sh

@@ -0,0 +1,94 @@
+#!/usr/bin/env bash
+# Anonymous registry checks use a temporary DOCKER_CONFIG inside this script.
+# Do not wrap the entire shell in DOCKER_CONFIG=... when invoking this script:
+# `docker compose config` needs your normal Docker config to resolve the compose file.
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(dirname "${SCRIPT_DIR}")"
+COMPOSE_FILE="${COMPOSE_FILE:-${PROJECT_ROOT}/docker-compose.yaml}"
+REGISTRY="${REGISTRY:-ghcr.io}"
+IMAGE_NAMESPACE="${IMAGE_NAMESPACE:-siriusscan}"
+
+if [ "$#" -gt 0 ]; then
+  TAGS=("$@")
+else
+  TAGS=("latest")
+fi
+
+export POSTGRES_PASSWORD="${POSTGRES_PASSWORD:-test-postgres-password}"
+export NEXTAUTH_SECRET="${NEXTAUTH_SECRET:-test-nextauth-secret}"
+export INITIAL_ADMIN_PASSWORD="${INITIAL_ADMIN_PASSWORD:-test-admin-password}"
+export DATABASE_URL="${DATABASE_URL:-postgresql://postgres:test-postgres-password@sirius-postgres:5432/sirius}"
+export SIRIUS_IMAGE_PULL_POLICY="${SIRIUS_IMAGE_PULL_POLICY:-always}"
+
+if [ ! -f "${PROJECT_ROOT}/secrets/sirius_api_key.txt" ]; then
+  mkdir -p "${PROJECT_ROOT}/secrets"
+  printf '%s\n' "${SIRIUS_INTERNAL_API_KEY_TEST_VALUE:-test-api-key}" > "${PROJECT_ROOT}/secrets/sirius_api_key.txt"
+fi
+
+anonymous_manifest_check() {
+  local image_ref="$1"
+  local docker_config
+  local err
+  local rc
+
+  docker_config="$(mktemp -d)"
+
+  set +e
+  err="$(DOCKER_CONFIG="${docker_config}" docker manifest inspect "${image_ref}" 2>&1 >/dev/null)"
+  rc=$?
+  set -e
+
+  rm -rf "${docker_config}"
+
+  if [ "${rc}" -eq 0 ]; then
+    echo "  PASS ${image_ref}"
+    return 0
+  fi
+
+  if printf '%s' "${err}" | grep -qiE 'unauthorized|denied|forbidden'; then
+    echo "::error::Anonymous access denied for ${image_ref}. GHCR package visibility is not public or token-based visibility enforcement failed."
+    return 1
+  fi
+
+  if printf '%s' "${err}" | grep -qiE 'manifest unknown|no such manifest|not found'; then
+    echo "::error::Manifest missing for ${image_ref}. The tag was not published or was published under a different name."
+    return 1
+  fi
+
+  echo "::error::Unable to verify ${image_ref}: ${err}"
+  return 1
+}
+
+for tag in "${TAGS[@]}"; do
+  echo "Verifying anonymous GHCR access for IMAGE_TAG=${tag}"
+
+  image_refs=()
+  while IFS= read -r image_ref; do
+    image_refs+=("${image_ref}")
+  done < <(
+    IMAGE_TAG="${tag}" docker compose -f "${COMPOSE_FILE}" config \
+      | awk '/^[[:space:]]+image:[[:space:]]/ {print $2}' \
+      | grep "^${REGISTRY}/${IMAGE_NAMESPACE}/" \
+      | sort -u
+  )
+
+  if [ "${#image_refs[@]}" -eq 0 ]; then
+    echo "::error::No GHCR image references found in ${COMPOSE_FILE} for IMAGE_TAG=${tag}"
+    exit 1
+  fi
+
+  failures=0
+  for image_ref in "${image_refs[@]}"; do
+    if ! anonymous_manifest_check "${image_ref}"; then
+      failures=1
+    fi
+  done
+
+  if [ "${failures}" -ne 0 ]; then
+    exit 1
+  fi
+done
+
+echo "All compose-rendered GHCR images are anonymously readable."

scripts/verify-release-images.sh

@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+IMAGE_TAG="${IMAGE_TAG:-latest}"
+REGISTRY="${REGISTRY:-ghcr.io/siriusscan}"
+
+declare -a SERVICES=("sirius-ui" "sirius-api" "sirius-engine")
+
+echo "Verifying release images for tag: ${IMAGE_TAG}"
+
+for service in "${SERVICES[@]}"; do
+  image_ref="${REGISTRY}/${service}:${IMAGE_TAG}"
+  container_name="${service}"
+
+  echo
+  echo "Pulling ${image_ref}..."
+  docker pull "${image_ref}" >/dev/null
+
+  expected_id="$(docker image inspect "${image_ref}" --format '{{.Id}}')"
+  running_image_ref="$(docker inspect "${container_name}" --format '{{.Config.Image}}' 2>/dev/null || true)"
+  running_id="$(docker inspect "${container_name}" --format '{{.Image}}' 2>/dev/null || true)"
+
+  if [ -z "${running_id}" ]; then
+    echo "❌ ${container_name} is not running"
+    exit 1
+  fi
+
+  echo "${container_name} config image: ${running_image_ref}"
+  echo "${container_name} running image id: ${running_id}"
+  echo "${container_name} expected image id: ${expected_id}"
+
+  if [ "${running_id}" != "${expected_id}" ]; then
+    echo "❌ ${container_name} is not running ${image_ref}"
+    exit 1
+  fi
+
+  echo "✅ ${container_name} matches ${image_ref}"
+done
+
+echo
+echo "All release image checks passed."

scripts/verify-runtime-auth-contract.sh

@@ -0,0 +1,198 @@
+#!/usr/bin/env bash
+# Runtime auth contract: internal API key comes from the mounted
+# SIRIUS_API_KEY_FILE secret, POSTGRES_PASSWORD parity, engine env, API HTTP
+# behavior.
+#
+# Container names default to docker-compose.yaml `container_name` values (project: sirius).
+# Override for other projects, e.g. CI:
+#   export SIRIUS_CONTRACT_CONTAINER_UI=sirius-test-sirius-ui-1
+#   export SIRIUS_CONTRACT_CONTAINER_API=sirius-test-sirius-api-1
+#   export SIRIUS_CONTRACT_CONTAINER_ENGINE=sirius-test-sirius-engine-1
+#   export SIRIUS_CONTRACT_CONTAINER_POSTGRES=sirius-test-sirius-postgres-1
+#
+# Host URL for curl (published API port on the host):
+#   export SIRIUS_API_PUBLIC_URL=http://localhost:9001
+#
+set -euo pipefail
+
+: "${SIRIUS_CONTRACT_CONTAINER_UI:=sirius-ui}"
+: "${SIRIUS_CONTRACT_CONTAINER_API:=sirius-api}"
+: "${SIRIUS_CONTRACT_CONTAINER_ENGINE:=sirius-engine}"
+: "${SIRIUS_CONTRACT_CONTAINER_POSTGRES:=sirius-postgres}"
+: "${SIRIUS_API_PUBLIC_URL:=http://localhost:9001}"
+SIRIUS_API_PUBLIC_URL="${SIRIUS_API_PUBLIC_URL%/}"
+
+CTR_UI="$SIRIUS_CONTRACT_CONTAINER_UI"
+CTR_API="$SIRIUS_CONTRACT_CONTAINER_API"
+CTR_ENGINE="$SIRIUS_CONTRACT_CONTAINER_ENGINE"
+CTR_POSTGRES="$SIRIUS_CONTRACT_CONTAINER_POSTGRES"
+
+required_containers=("$CTR_POSTGRES" "$CTR_API" "$CTR_ENGINE" "$CTR_UI")
+for name in "${required_containers[@]}"; do
+  if ! docker inspect "$name" >/dev/null 2>&1; then
+    echo "❌ Missing container: $name (override SIRIUS_CONTRACT_CONTAINER_* if using a non-default compose project)"
+    exit 1
+  fi
+done
+
+extract_env() {
+  local container="$1"
+  local key="$2"
+  # NOTE: use grep, not rg — ripgrep is not preinstalled on GitHub-hosted
+  # ubuntu-latest runners. The trailing `|| true` previously masked the
+  # `rg: command not found` failure, silently returning empty for every
+  # key and tripping the POSTGRES_PASSWORD parity check downstream.
+  docker inspect "$container" --format '{{range .Config.Env}}{{println .}}{{end}}' \
+    | grep "^${key}=" | sed "s/^${key}=//" || true
+}
+
+# Effective key inside the container (trimmed file contents only).
+resolve_file_backed_api_key() {
+  local container="$1"
+  docker exec "$container" sh -lc \
+    'if [ -r "${SIRIUS_API_KEY_FILE:-}" ]; then tr -d "\r\n" < "$SIRIUS_API_KEY_FILE"; fi'
+}
+
+mask_value() {
+  local value="$1"
+  if [ -z "$value" ]; then
+    echo "<empty>"
+    return
+  fi
+  local len=${#value}
+  if [ "$len" -le 10 ]; then
+    echo "${value:0:2}***"
+    return
+  fi
+  echo "${value:0:6}...${value: -4}"
+}
+
+assert_file_only_api_key_contract() {
+  local container="$1"
+  local label="$2"
+  local file_key
+  local configured_env_key
+
+  file_key="$(resolve_file_backed_api_key "$container")"
+  configured_env_key="$(extract_env "$container" SIRIUS_API_KEY)"
+
+  if [ -n "$configured_env_key" ]; then
+    echo "❌ ${label} still has legacy SIRIUS_API_KEY configured in container env"
+    exit 1
+  fi
+
+  if [ -z "$file_key" ]; then
+    echo "❌ ${label} is missing a readable file-backed internal API key"
+    exit 1
+  fi
+}
+
+assert_file_only_api_key_contract "$CTR_UI" "sirius-ui"
+assert_file_only_api_key_contract "$CTR_API" "sirius-api"
+assert_file_only_api_key_contract "$CTR_ENGINE" "sirius-engine"
+
+ui_key="$(resolve_file_backed_api_key "$CTR_UI")"
+api_key="$(resolve_file_backed_api_key "$CTR_API")"
+engine_key="$(resolve_file_backed_api_key "$CTR_ENGINE")"
+
+postgres_db_pw="$(extract_env "$CTR_POSTGRES" POSTGRES_PASSWORD)"
+api_db_pw="$(extract_env "$CTR_API" POSTGRES_PASSWORD)"
+engine_db_pw="$(extract_env "$CTR_ENGINE" POSTGRES_PASSWORD)"
+engine_api_base_url="$(extract_env "$CTR_ENGINE" API_BASE_URL)"
+engine_sirius_api_url="$(extract_env "$CTR_ENGINE" SIRIUS_API_URL)"
+engine_agent_id="$(extract_env "$CTR_ENGINE" AGENT_ID)"
+engine_host_id="$(extract_env "$CTR_ENGINE" HOST_ID)"
+
+echo "Runtime key fingerprints:"
+echo "  ui ($CTR_UI):     $(mask_value "$ui_key")"
+echo "  api ($CTR_API):    $(mask_value "$api_key")"
+echo "  engine ($CTR_ENGINE): $(mask_value "$engine_key")"
+
+if [ -z "$ui_key" ] || [ -z "$api_key" ] || [ -z "$engine_key" ]; then
+  echo "❌ One or more services are missing readable SIRIUS_API_KEY_FILE secret data"
+  exit 1
+fi
+
+echo "✅ File-only internal API key contract check passed"
+
+if [ "$ui_key" != "$api_key" ] || [ "$api_key" != "$engine_key" ]; then
+  echo "❌ Internal API key mismatch across services (key split-brain detected)"
+  exit 1
+fi
+echo "✅ Internal API key parity check passed"
+
+if [ -z "$postgres_db_pw" ] || [ -z "$api_db_pw" ] || [ -z "$engine_db_pw" ]; then
+  echo "❌ One or more services are missing POSTGRES_PASSWORD"
+  exit 1
+fi
+
+if [ "$postgres_db_pw" != "$api_db_pw" ] || [ "$api_db_pw" != "$engine_db_pw" ]; then
+  echo "❌ POSTGRES_PASSWORD mismatch across postgres/api/engine"
+  exit 1
+fi
+echo "✅ POSTGRES_PASSWORD parity check passed"
+
+if [ -z "$engine_sirius_api_url" ] || [ -z "$engine_agent_id" ] || [ -z "$engine_host_id" ]; then
+  echo "❌ sirius-engine is missing one or more required runtime env values (SIRIUS_API_URL, AGENT_ID, HOST_ID)"
+  exit 1
+fi
+if [ -n "$engine_api_base_url" ] && [ -n "$engine_sirius_api_url" ]; then
+  if [ "${engine_api_base_url%/}" != "${engine_sirius_api_url%/}" ]; then
+    echo "❌ API_BASE_URL must equal SIRIUS_API_URL on engine (got API_BASE_URL=${engine_api_base_url} SIRIUS_API_URL=${engine_sirius_api_url})"
+    exit 1
+  fi
+fi
+echo "✅ sirius-engine runtime env contract check passed"
+
+if docker exec "$CTR_POSTGRES" test -f /usr/local/bin/start-with-monitor.sh 2>/dev/null; then
+  if docker exec "$CTR_POSTGRES" sh -lc "grep -E -q 'psql[^\\n]*\\|\\| true|ALTER ROLE[^\\n]*\\|\\| true' /usr/local/bin/start-with-monitor.sh"; then
+    echo "❌ Running postgres entrypoint still contains permissive psql reconciliation fallback"
+    exit 1
+  fi
+  echo "✅ Postgres reconciliation script does not contain permissive psql fallback"
+else
+  echo "ℹ️  Skipping custom postgres entrypoint check (not Sirius postgres image)"
+fi
+
+if ! docker exec "$CTR_ENGINE" bash -lc 'for bin in bash curl psql pkill nmap rustscan pwsh; do command -v "$bin" >/dev/null || exit 1; done'; then
+  echo "❌ sirius-engine runtime binary contract check failed"
+  exit 1
+fi
+echo "✅ sirius-engine runtime binary contract check passed"
+
+if ! docker exec "$CTR_ENGINE" bash -lc '/bin/bash -n /start-enhanced.sh && grep -q "validate_required_binary \"psql\"" /start-enhanced.sh'; then
+  echo "❌ sirius-engine startup script contract check failed"
+  exit 1
+fi
+echo "✅ sirius-engine startup script contract check passed"
+
+HOST_PROBE="${SIRIUS_API_PUBLIC_URL}/host/"
+unauth_code="$(curl -sS --max-time 15 -o /tmp/sirius-auth-unauth.out -w '%{http_code}' "$HOST_PROBE" || true)"
+if [ "$unauth_code" != "401" ]; then
+  echo "❌ Expected unauthenticated /host/ to return 401, got $unauth_code (URL: $HOST_PROBE)"
+  exit 1
+fi
+echo "✅ Unauthenticated API contract check passed"
+
+auth_code="$(curl -sS --max-time 15 -o /tmp/sirius-auth-auth.out -w '%{http_code}' -H "X-API-Key: $api_key" "$HOST_PROBE" || true)"
+if [ "$auth_code" = "401" ] || [ "$auth_code" = "403" ] || [ "$auth_code" = "000" ]; then
+  echo "❌ Authenticated /host/ request failed with status $auth_code"
+  exit 1
+fi
+if ! [[ "$auth_code" =~ ^[0-9]+$ ]]; then
+  echo "❌ Unexpected curl HTTP code output: $auth_code"
+  exit 1
+fi
+if [ "$auth_code" -lt 200 ] || [ "$auth_code" -ge 300 ]; then
+  echo "❌ Authenticated /host/ expected HTTP 2xx, got $auth_code"
+  exit 1
+fi
+echo "✅ Authenticated API contract check passed (status $auth_code)"
+
+if ! docker exec "$CTR_POSTGRES" sh -lc 'PGPASSWORD="$POSTGRES_PASSWORD" psql -h 127.0.0.1 -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "SELECT 1;" >/dev/null'; then
+  echo "❌ Postgres runtime credential probe failed"
+  exit 1
+fi
+echo "✅ Postgres runtime credential probe passed"
+
+echo "✅ Runtime auth contract verification succeeded"