wehub/siriusscan--sirius
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
Check engine pin consistency / Dockerfile / CI pin consistency (push) Successful in 8s
Details
Sirius CI/CD Pipeline / Detect Changes (push) Successful in 23s
Details
Validate Docker Configuration / Validate Docker Compose Configuration (push) Successful in 47s
Details
Sirius CI/CD Pipeline / Build API (${{ matrix.platform }}) (push) Has been cancelled
Details
Sirius CI/CD Pipeline / Build UI (${{ matrix.platform }}) (push) Has been cancelled
Details
Sirius CI/CD Pipeline / Merge Engine Manifest (push) Has been cancelled
Details
Sirius CI/CD Pipeline / Merge API Manifest (push) Has been cancelled
Details
Sirius CI/CD Pipeline / Merge UI Manifest (push) Has been cancelled
Details
Sirius CI/CD Pipeline / Build Engine (${{ matrix.platform }}) (push) Has been cancelled
Details
Sirius CI/CD Pipeline / Build Infra (${{ matrix.service }}, ${{ matrix.platform }}) (push) Has been cancelled
Details
Sirius CI/CD Pipeline / Merge Infra Manifest (sirius-postgres) (push) Has been cancelled
Details
Sirius CI/CD Pipeline / Merge Infra Manifest (sirius-rabbitmq) (push) Has been cancelled
Details
Sirius CI/CD Pipeline / Merge Infra Manifest (sirius-valkey) (push) Has been cancelled
Details
Sirius CI/CD Pipeline / Integration Test (push) Has been cancelled
Details
Sirius CI/CD Pipeline / Public Stack Contract (push) Has been cancelled
Details
Sirius CI/CD Pipeline / Dispatch Demo Deployment (sirius-demo branch) (push) Has been cancelled
Details
Sirius CI/CD Pipeline / Dispatch Demo Canary (main branch) (push) Has been cancelled
Details
Sirius CI/CD Pipeline / Guard Registry Namespace (push) Has been cancelled
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:32:25 +08:00
commit 161ef94b4f
708 changed files with 189571 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/dev/ABOUT.documentation.md
+462
View File
@@ -0,0 +1,462 @@
---
title: "About Sirius Documentation"
description: "A comprehensive guide to the Sirius project's documentation system, including standards, conventions, and best practices for creating and maintaining high-quality, machine-readable documentation."
template: "TEMPLATE.custom"
version: "1.0.0"
last_updated: "2025-01-03"
author: "AI Assistant"
tags:
["documentation", "standards", "guidelines", "llm-friendly", "best-practices"]
categories: ["project-management", "development"]
difficulty: "beginner"
prerequisites: []
related_docs:
- "README.documentation-index.md"
- "TEMPLATE.documentation-standard.md"
- "deployment/README.workflows.md"
dependencies: []
llm_context: "high"
search_keywords:
[
"documentation standards",
"llm documentation",
"markdown conventions",
"sirius docs",
]
---
# ABOUT Documentation
> **📚 Documentation Index**: For a complete list of all documentation files, see [README.documentation-index.md](../README.documentation-index.md)
## Purpose
This document defines the documentation standards, conventions, and processes for the Sirius project. It serves as the definitive guide for anyone creating, maintaining, or using documentation within our ecosystem, with special emphasis on LLM-consumable structure and machine readability.
## When to Use This Guide
- **Before creating any new documentation** - Understand our standards first
- **When updating existing documentation** - Ensure consistency with our conventions
- **When troubleshooting documentation issues** - Reference our established patterns
- **During code reviews** - Verify documentation meets our standards
- **When onboarding new team members** - Provide clear documentation guidelines
- **For LLM context building** - Ensure documentation is optimally structured for AI consumption
## How to Use This Guide
1. **Start with the documentation index** - Review [README.documentation-index.md](../README.documentation-index.md) to see all available documentation
2. **Read the Lexicon** - Understand our naming conventions and terminology
3. **Review File Types** - Know which template to use for different documentation needs
4. **Follow the Standards** - Use our template structure for consistency
5. **Add Front Matter** - Include YAML metadata for machine readability
6. **Link Documents** - Establish relationships between related documentation
7. **Update as Needed** - Keep this guide current with project evolution
## What This Documentation System Is
### Core Philosophy
Our documentation system is designed around **layered information architecture** where the most critical information appears first, followed by increasingly detailed technical content. This ensures that users can quickly find what they need without wading through irrelevant details.
**LLM Optimization**: Every document is structured to provide maximum context to Large Language Models, with clear metadata, relationships, and machine-readable front matter.
### File Naming Conventions
#### Lexicon of File Types
| File Type | Prefix | Purpose | Example | Template Used |
| ------------------- | ------------------ | ------------------------------------ | ------------------------------------ | ------------------------------- |
| **ABOUT** | `ABOUT.` | Explains how to use documentation | `ABOUT.documentation.md` | TEMPLATE.about |
| **TEMPLATE** | `TEMPLATE.` | Defines structure for document types | `TEMPLATE.documentation-standard.md` | TEMPLATE.template |
| **README** | `README.` | Primary documentation for a topic | `README.container-testing.md` | TEMPLATE.documentation-standard |
| **GUIDE** | `GUIDE.` | Step-by-step instructions | `GUIDE.deployment.md` | TEMPLATE.guide |
| **REFERENCE** | `REFERENCE.` | Technical specifications | `REFERENCE.api-endpoints.md` | TEMPLATE.reference |
| **TROUBLESHOOTING** | `TROUBLESHOOTING.` | Problem-solving focused | `TROUBLESHOOTING.docker-issues.md` | TEMPLATE.troubleshooting |
| **ARCHITECTURE** | `ARCHITECTURE.` | System design and structure | `ARCHITECTURE.system-overview.md` | TEMPLATE.architecture |
| **API** | `API.` | API documentation and specifications | `API.endpoints.md` | TEMPLATE.api |
#### Naming Rules
1. **All prefixes are fully capitalized** (ABOUT, TEMPLATE, README, etc.)
2. **Descriptive suffixes** use kebab-case (container-testing, api-endpoints)
3. **File extensions** are always `.md` for Markdown files
4. **No spaces** in filenames - use hyphens instead
5. **Version suffixes** for multiple versions (e.g., `README.deployment-v2.md`)
### Directory Structure
```
documentation/
├── dev/ # Development documentation
│ ├── ABOUT.documentation.md # This file
│ ├── templates/ # Document templates
│ │ ├── TEMPLATE.about.md
│ │ ├── TEMPLATE.documentation-standard.md
│ │ ├── TEMPLATE.guide.md
│ │ ├── TEMPLATE.reference.md
│ │ ├── TEMPLATE.troubleshooting.md
│ │ ├── TEMPLATE.architecture.md
│ │ └── TEMPLATE.api.md
│ ├── architecture/ # System architecture docs
│ │ ├── ARCHITECTURE.system-overview.md
│ │ └── ARCHITECTURE.data-flow.md
│ ├── operations/ # Operations documentation
│ │ ├── README.git-operations.md
│ │ └── README.terraform-deployment.md
│ ├── test/ # Testing documentation
│ │ ├── README.container-testing.md
│ │ └── GUIDE.testing-workflows.md
│ ├── api/ # API documentation
│ │ ├── API.endpoints.md
│ │ └── REFERENCE.api-specs.md
│ └── dev-notes/ # Development notes
│ ├── TROUBLESHOOTING.docker-issues.md
│ └── GUIDE.development-setup.md
├── production/ # Production documentation
│ ├── README.deployment.md
│ ├── GUIDE.monitoring.md
│ └── TROUBLESHOOTING.production-issues.md
├── user/ # End-user documentation
│ ├── GUIDE.getting-started.md
│ └── GUIDE.user-workflows.md
└── archive/ # Historical documentation
└── README.legacy-systems.md
```
### YAML Front Matter Standards
Every documentation file MUST include YAML front matter for machine readability:
```yaml
---
title: "Document Title"
description: "Brief description of the document's purpose"
template: "TEMPLATE.documentation-standard"
version: "1.0.0"
last_updated: "2025-01-03"
author: "Document Author"
tags: ["docker", "testing", "containers"]
categories: ["development", "testing"]
difficulty: "intermediate"
prerequisites: ["docker", "make"]
related_docs:
- "README.deployment.md"
- "GUIDE.docker-setup.md"
dependencies:
- "docker-compose.yaml"
- "testing/scripts/"
llm_context: "high"
search_keywords: ["container", "testing", "docker", "health-check"]
---
```
#### Front Matter Fields
| Field | Required | Description | Example |
| ----------------- | -------- | -------------------------------- | ---------------------------------------- |
| `title` | ✅ | Human-readable document title | "Container Testing" |
| `description` | ✅ | Brief purpose description | "Comprehensive container testing system" |
| `template` | ✅ | Template used to create document | "TEMPLATE.documentation-standard" |
| `version` | ✅ | Document version | "1.0.0" |
| `last_updated` | ✅ | Last modification date | "2025-01-03" |
| `author` | ❌ | Document author | "Development Team" |
| `tags` | ❌ | Searchable tags | ["docker", "testing"] |
| `categories` | ❌ | Document categories | ["development", "testing"] |
| `difficulty` | ❌ | Complexity level | "beginner", "intermediate", "advanced" |
| `prerequisites` | ❌ | Required knowledge | ["docker", "make"] |
| `related_docs` | ❌ | Related documentation | ["README.deployment.md"] |
| `dependencies` | ❌ | Required files/systems | ["docker-compose.yaml"] |
| `llm_context` | ❌ | LLM relevance level | "high", "medium", "low" |
| `search_keywords` | ❌ | Search optimization | ["container", "testing"] |
### Document Structure Standards
Every documentation file follows our **Standard Documentation Template**:
1. **YAML Front Matter** - Machine-readable metadata
2. **Purpose** - What this document is for
3. **When to Use** - When to reference this document
4. **How to Use** - How to apply the information
5. **What It Is** - Detailed technical content
6. **Troubleshooting** - Problem-solving section
- **FAQ** - Frequently asked questions
- **Command Reference** - Common commands
- **Lessons Learned** - Timestamped insights
7. **Related Documentation** - Links to related docs
8. **LLM Context** - Additional context for AI systems
### Content Guidelines
#### Writing Style
- **Clear and Concise** - Get to the point quickly
- **Technical Accuracy** - Verify all technical details
- **Consistent Terminology** - Use our established lexicon
- **Actionable Content** - Provide specific steps, not vague guidance
- **LLM-Friendly** - Structure content for AI consumption
#### Information Layering
1. **Executive Summary** - High-level overview (2-3 sentences)
2. **Quick Reference** - Essential information for immediate use
3. **Detailed Content** - Comprehensive technical details
4. **Reference Material** - Commands, examples, troubleshooting
5. **LLM Context** - Additional context for AI systems
#### Code Examples
- **Always include working examples**
- **Use syntax highlighting** for code blocks
- **Provide context** for complex examples
- **Test examples** before including them
- **Include expected outputs** for verification
### Template System
#### Template Files
Template files define the structure for specific types of documentation. They serve as:
- **Consistency enforcers** - Ensure all docs follow the same pattern
- **Quality checklists** - Remind authors of required sections
- **Onboarding tools** - Help new contributors understand expectations
- **LLM context builders** - Provide structured context for AI systems
#### Template Types
1. **TEMPLATE.about** - For ABOUT documents
2. **TEMPLATE.documentation-standard** - For README documents
3. **TEMPLATE.guide** - For step-by-step guides
4. **TEMPLATE.reference** - For technical specifications
5. **TEMPLATE.troubleshooting** - For problem-solving docs
6. **TEMPLATE.architecture** - For system design docs
7. **TEMPLATE.api** - For API documentation
#### Using Templates
1. **Copy the appropriate template** for your document type
2. **Fill in YAML front matter** with complete metadata
3. **Fill in each section** according to the template structure
4. **Add LLM context** where appropriate
5. **Link related documentation** in the front matter
6. **Review against the template** before finalizing
### Document Relationships
#### Linking Strategy
- **Front Matter Links** - Use `related_docs` for primary relationships
- **Inline Links** - Use markdown links for specific references
- **Cross-References** - Link to specific sections when relevant
- **Dependency Tracking** - Use `dependencies` field for required files
#### Relationship Types
1. **Prerequisites** - Documents that should be read first
2. **Related Topics** - Documents covering similar subjects
3. **Dependencies** - Documents that depend on this one
4. **References** - Documents that reference this one
5. **Troubleshooting** - Documents that help solve problems
### LLM Optimization
#### Context Building
- **Rich Metadata** - Comprehensive front matter for AI understanding
- **Structured Content** - Consistent sections for predictable parsing
- **Clear Relationships** - Document links for context building
- **Search Keywords** - Optimized for AI search and retrieval
#### AI-Friendly Features
- **Consistent Formatting** - Predictable structure for parsing
- **Rich Descriptions** - Detailed explanations for AI understanding
- **Code Examples** - Working examples with expected outputs
- **Troubleshooting** - Comprehensive problem-solving information
### Maintenance Standards
#### Update Triggers
Documentation should be updated when:
- **New features are added** - Document new functionality
- **Bugs are fixed** - Update troubleshooting sections
- **Processes change** - Reflect new workflows
- **Issues are discovered** - Add to FAQ or lessons learned
- **Dependencies change** - Update related document links
#### Review Process
1. **Technical accuracy** - Verify all technical details
2. **Completeness** - Ensure all required sections are present
3. **Clarity** - Check for clear, understandable language
4. **Consistency** - Verify adherence to our standards
5. **LLM Readability** - Ensure AI-friendly structure
6. **Link Validation** - Verify all links are working
### Quality Assurance
#### Pre-Publication Checklist
- [ ] YAML front matter complete and accurate
- [ ] Follows appropriate template structure
- [ ] All code examples tested and working
- [ ] Technical details verified for accuracy
- [ ] Spelling and grammar checked
- [ ] Links and references validated
- [ ] Consistent with project terminology
- [ ] LLM context optimized
- [ ] Related documents linked
#### Post-Publication
- **Monitor usage** - Track which sections are referenced most
- **Collect feedback** - Note areas that need clarification
- **Update regularly** - Keep content current with project evolution
- **LLM Performance** - Monitor AI system usage and effectiveness
## Troubleshooting
### FAQ
**Q: Which template should I use for a new document?**
A: Check the template mapping in the File Types table. Most documents use `TEMPLATE.documentation-standard`, but specialized templates exist for specific document types.
**Q: How detailed should the YAML front matter be?**
A: Include all required fields and as many optional fields as relevant. More metadata improves LLM context and searchability.
**Q: What if I need to deviate from the standard template?**
A: Document the deviation in the document itself, update the front matter accordingly, and consider whether a new template is needed.
**Q: How do I establish document relationships?**
A: Use the `related_docs` field in front matter for primary relationships and inline markdown links for specific references.
**Q: What's the difference between tags and categories?**
A: Tags are specific keywords for search, categories are broader groupings for organization and navigation.
**Q: How often should I update the FAQ section?**
A: Update it every time you encounter a new question or problem related to the document's topic.
**Q: What's the purpose of the LLM context field?**
A: It helps AI systems understand the document's relevance and importance for different types of queries and tasks.
**Q: How do I handle version-specific information?**
A: Include version information in the front matter and clearly mark version-specific sections in the content.
**Q: What if I find an error in existing documentation?**
A: Fix it immediately, update the version number, and add an entry to the "Lessons Learned" section.
### Command Reference
```bash
# Create new documentation from template
cp documentation/dev/templates/TEMPLATE.documentation-standard.md documentation/new-doc.md
# Validate YAML front matter
yamllint documentation/*.md
# Validate Markdown syntax
markdownlint documentation/*.md
# Check for broken links
markdown-link-check documentation/*.md
# Generate table of contents
doctoc documentation/README.container-testing.md
# Search documentation by tags
grep -r "tags:" documentation/ | grep "docker"
# Find documents by category
grep -r "categories:" documentation/ | grep "testing"
# Extract LLM context documents
grep -r "llm_context: high" documentation/
# Validate document relationships
grep -r "related_docs:" documentation/
# Check template usage
grep -r "template:" documentation/
```
### Common Issues and Solutions
| Issue | Symptoms | Solution |
| --------------------- | ---------------------------------- | ------------------------------------------------------- |
| Missing front matter | No YAML metadata at document top | Add complete YAML front matter with required fields |
| Broken links | 404 errors in related documents | Update link paths and validate with markdown-link-check |
| Template mismatch | Document doesn't follow template | Compare document structure with specified template |
| Missing relationships | No related_docs in front matter | Add related_docs field with relevant document links |
| Inconsistent tags | Tags don't follow conventions | Standardize tags using established tag vocabulary |
| Outdated metadata | Last_updated doesn't match content | Update last_updated field when making changes |
### Debugging Steps
1. **Check front matter**: Verify YAML syntax and required fields
2. **Validate template usage**: Compare document structure with template
3. **Test links**: Use markdown-link-check to find broken links
4. **Verify relationships**: Check that related_docs links are valid
5. **Review LLM context**: Ensure content is optimized for AI consumption
6. **Check consistency**: Verify terminology and formatting standards
### Log Analysis
**Front Matter Validation**:
```bash
# Check for missing required fields
grep -L "title:" documentation/*.md
grep -L "template:" documentation/*.md
# Validate YAML syntax
find documentation/ -name "*.md" -exec head -20 {} \; | grep -A 20 "^---"
```
**Link Validation**:
```bash
# Check for broken internal links
grep -r "\[.*\](" documentation/ | grep -v "http"
# Validate related_docs links
grep -r "related_docs:" documentation/ | cut -d: -f3
```
**Template Usage**:
```bash
# Check template compliance
grep -r "template:" documentation/ | cut -d: -f3 | sort | uniq -c
```
### Performance Troubleshooting
**LLM Context Optimization**:
- Ensure rich metadata in front matter
- Use consistent structure across documents
- Include comprehensive troubleshooting information
- Maintain up-to-date relationships
**Search Optimization**:
- Use descriptive tags and keywords
- Include relevant categories
- Maintain consistent terminology
- Update search_keywords regularly
### Lessons Learned
**2025-01-03**: Established comprehensive documentation standards with LLM optimization. Key insight: YAML front matter and structured relationships significantly improve AI system effectiveness.
**2025-01-03**: Created template system with specialized templates for different document types. Lesson: Specialized templates improve consistency and reduce cognitive load for authors.
**2025-01-03**: Implemented document relationship tracking through front matter. Benefit: Enables AI systems to build comprehensive context and understand document dependencies.
**2025-01-03**: Added LLM-specific optimization features including context levels and search keywords. Advantage: Improves AI system performance and document discoverability.
---
_This document is the foundation of our documentation system. Keep it updated as our standards evolve._
documentation/dev/QUICK-REFERENCE.md
+162
View File
@@ -0,0 +1,162 @@
---
title: "Developer Quick Reference"
description: "Quick reference card for common development tasks and commands"
template: "TEMPLATE.reference"
version: "1.0.0"
last_updated: "2025-01-03"
author: "Development Team"
tags: ["quick reference", "commands", "development", "docker"]
categories: ["reference", "development"]
difficulty: "beginner"
prerequisites: ["docker", "git"]
related_docs:
- "README.developer-guide.md"
- "README.development.md"
dependencies: []
llm_context: "medium"
search_keywords: ["quick reference", "commands", "cheat sheet", "development"]
---
# Developer Quick Reference
## Environment Switching
```bash
# Switch to development mode (hot reloading)
./scripts/switch-env.sh dev
# Switch to production mode (optimized builds)
./scripts/switch-env.sh prod
# Switch to base mode (standard config)
./scripts/switch-env.sh base
```
## Common Commands
### Container Management
```bash
# View container status
docker compose ps
# View logs
docker compose logs -f
docker compose logs sirius-ui -f
docker compose logs sirius-api -f
docker compose logs sirius-engine -f
# Restart services
docker compose restart
docker compose restart sirius-ui
# Stop all services
docker compose down
```
### Development Workflow
```bash
# Start development
./scripts/switch-env.sh dev
# Make changes to code
# UI changes appear instantly (hot reloading)
# API/Engine changes require restart
# Test production build
./scripts/switch-env.sh prod
# Switch back to development
./scripts/switch-env.sh dev
```
### Container Access
```bash
# Open shell in container
docker compose exec sirius-ui sh
docker compose exec sirius-api sh
docker compose exec sirius-engine sh
# Run commands in container
docker compose exec sirius-ui npm run build
docker compose exec sirius-api go run main.go
```
### Health Checks
```bash
# Check service health
curl http://localhost:3000/api/health
curl http://localhost:9001/api/v1/health
curl http://localhost:5174/health
# Check database
docker compose exec sirius-postgres pg_isready -U postgres
```
### Troubleshooting
```bash
# Clean Docker cache
docker system prune -f
# Force rebuild
./scripts/switch-env.sh dev
# Check what's running
docker compose exec sirius-ui ps aux
# View detailed logs
docker compose logs sirius-ui --tail=50
```
## Service Ports
| Service | Port | Purpose |
|---------|------|---------|
| sirius-ui | 3000 | Web interface |
| sirius-api | 9001 | REST API |
| sirius-engine | 5174 | Engine interface |
| sirius-engine | 50051 | gRPC communication |
| sirius-postgres | 5432 | Database |
| sirius-valkey | 6379 | Cache |
| sirius-rabbitmq | 5672 | Message queue |
| sirius-rabbitmq | 15672 | Management UI |
## File Locations
| Component | Location | Purpose |
|-----------|----------|---------|
| UI Source | `sirius-ui/src/` | React components and pages |
| API Source | `sirius-api/` | Go REST API code |
| Engine Source | `sirius-engine/` | Multi-service engine code |
| Docker Config | `docker-compose*.yaml` | Environment configurations |
| Switch Script | `scripts/switch-env.sh` | Environment switching |
## Environment Differences
| Feature | Development | Production |
|---------|-------------|------------|
| UI Server | `npm run dev` | `npm start` |
| Database | SQLite | PostgreSQL |
| Hot Reloading | ✅ Yes | ❌ No |
| Volume Mounts | ✅ Yes | ❌ No |
| Debug Logging | ✅ Yes | ❌ No |
| Optimizations | ❌ No | ✅ Yes |
## Quick Troubleshooting
| Problem | Solution |
|---------|----------|
| Wrong environment running | `./scripts/switch-env.sh dev` |
| Hot reloading not working | Check you're in dev mode |
| Services not starting | Check logs: `docker compose logs` |
| Port conflicts | `lsof -i :3000` then kill process |
| Database issues | `docker compose restart sirius-postgres` |
| Cache issues | `docker system prune -f` |
---
_For detailed information, see [README.developer-guide.md](README.developer-guide.md)._
documentation/dev/QUICKREF.template-types.md
+290
View File
@@ -0,0 +1,290 @@
---
title: "Template System Type Reference"
description: "Quick reference for template types and module configurations"
template: "TEMPLATE.documentation-standard"
version: "1.0.0"
last_updated: "2025-10-25"
author: "Sirius Platform Team"
tags: ["templates", "types", "reference", "quick-guide"]
categories: ["reference", "templates"]
difficulty: "beginner"
related_docs:
- "architecture/README.template-ui-integration.md"
- "architecture/README.agent-system.md"
llm_context: "high"
search_keywords: ["template", "types", "config", "module", "reference"]
---
# Template System Type Reference
Quick reference guide for template types, module configurations, and validation rules.
## Detection Step Types
| Type | Module | Config Required | Platforms |
| -------------- | -------------------- | ---------------------- | --------- |
| `file-hash` | FileHashModule | path, hash | All |
| `file-content` | FileContentModule | path, patterns | All |
| `version-cmd` | CommandVersionModule | command, version_regex | All |
## Module Configurations
### file-hash
```yaml
type: file-hash
platforms: [linux, darwin, windows]
weight: 1.0
config:
path: "/etc/passwd" # REQUIRED: Absolute path
hash: "sha256:abc123..." # REQUIRED: Hash with prefix
algorithm: "sha256" # OPTIONAL: Algorithm
```
**Hash Formats**:
- SHA256: `sha256:` + 64 hex chars
- SHA1: `sha1:` + 40 hex chars
- MD5: `md5:` + 32 hex chars
- SHA512: `sha512:` + 128 hex chars
---
### file-content
```yaml
type: file-content
platforms: [linux, darwin]
weight: 0.8
config:
path: "/etc/apache2/apache2.conf" # REQUIRED: File to search
patterns: # REQUIRED: Regex patterns
- "ServerTokens.*Full"
- "ServerSignature.*On"
```
**Pattern Notes**:
- Go regex (RE2 syntax)
- Case-sensitive by default
- No lookahead/lookbehind support
- Array required (even for single pattern)
---
### version-cmd
```yaml
type: version-cmd
platforms: [linux, darwin]
weight: 1.0
config:
command: "httpd -v" # REQUIRED: Shell command
version_regex: "Apache/(\\d+\\.\\d+)" # REQUIRED: Version capture
vulnerable_versions: # OPTIONAL: Constraints
- "< 2.4.52"
- ">= 2.2.0, < 2.2.34"
expected_exit_code: 0 # OPTIONAL: Exit code
```
**Version Operators**:
- `<`, `<=`, `>`, `>=`, `=`, `!=`
- Combine with comma: `>= 2.0, < 3.0`
## Platform Values
```yaml
platforms: [linux] # Linux only
platforms: [darwin] # macOS only
platforms: [windows] # Windows only
platforms: [linux, darwin] # Unix-like systems
platforms: [] # All platforms (default)
```
## Severity Values
```yaml
severity: critical # Highest priority
severity: high
severity: medium
severity: low
severity: info # Lowest priority
```
## Detection Logic
```yaml
detection:
logic: all # AND - All steps must match
logic: any # OR - At least one step must match
```
## Complete Template Example
```yaml
id: apache-outdated
info:
name: Outdated Apache HTTP Server
author: security-team
severity: high
description: Detects Apache versions vulnerable to known CVEs
references:
- https://nvd.nist.gov/vuln/detail/CVE-2021-44228
cve:
- CVE-2021-44228
tags:
- apache
- web-server
- cve
version: "1.0"
detection:
logic: all
steps:
- type: version-cmd
platforms: [linux, darwin]
weight: 1.0
config:
command: "httpd -v"
version_regex: "Apache/(\\d+\\.\\d+\\.\\d+)"
vulnerable_versions:
- "< 2.4.52"
- type: file-content
platforms: [linux, darwin]
weight: 0.8
config:
path: "/etc/apache2/apache2.conf"
patterns:
- "ServerTokens.*Full"
```
## TypeScript Interfaces
```typescript
// Detection step
interface DetectionStep {
type: "file-hash" | "file-content" | "version-cmd";
platforms?: ("linux" | "darwin" | "windows")[];
weight?: number; // 0.0 to 1.0
config: FileHashConfig | FileContentConfig | CommandVersionConfig;
}
// File hash config
interface FileHashConfig {
path: string;
hash: string;
algorithm?: "sha256" | "sha1" | "md5" | "sha512";
}
// File content config
interface FileContentConfig {
path: string;
patterns: string[];
}
// Version command config
interface CommandVersionConfig {
command: string;
version_regex: string;
vulnerable_versions?: string[];
expected_exit_code?: number;
}
// Template info
interface TemplateInfo {
name: string;
author: string;
severity: "critical" | "high" | "medium" | "low" | "info";
description: string;
references?: string[];
cve?: string[];
tags?: string[];
version?: string;
}
// Detection config
interface DetectionConfig {
logic?: "all" | "any"; // defaults to "all"
steps: DetectionStep[];
}
// Complete template
interface TemplateContent {
id: string;
info: TemplateInfo;
detection: DetectionConfig;
}
```
## Validation Rules
### Template ID
- Format: `^[a-z0-9-]+$`
- Length: 3-50 characters
- Cannot start/end with dash
- Cannot be reserved keyword (new, create, edit, delete, standard, custom)
### CVE Format
- Pattern: `^CVE-\d{4}-\d{4,}$`
- Example: `CVE-2021-44228`
### File Paths
- Must be absolute
- Unix: Starts with `/`
- Windows: Starts with drive letter (e.g., `C:\`)
### Regex Patterns
- Use Go RE2 syntax
- No backreferences
- No lookahead/lookbehind
- Named groups: `(?P<name>pattern)`
### Weight Values
- Range: 0.0 to 1.0
- Default: 1.0
- Affects confidence calculation
## Common Errors
| Error | Cause | Fix |
| ------------------------ | ------------------ | -------------------------- |
| "unknown module type" | Wrong type name | Use dashes not underscores |
| "missing required field" | Config incomplete | Check module requirements |
| "invalid regex" | Bad pattern syntax | Test with Go regex tester |
| "template not found" | Wrong ID | Check exact ID in library |
## Quick Tips
✅ **DO**:
- Use lowercase IDs with dashes
- Test regex patterns before submission
- Specify platforms when not all apply
- Include CVE references when available
- Add meaningful descriptions
❌ **DON'T**:
- Use underscores in type names
- Forget algorithm prefix in hashes
- Use single string for patterns (must be array)
- Use JavaScript regex features (Go RE2 only)
- Create IDs with spaces or special chars
## Resources
- Full guide: [README.template-ui-integration.md](architecture/README.template-ui-integration.md)
- Agent system: [README.agent-system.md](architecture/README.agent-system.md)
- Module code: `app-agent/internal/modules/`
---
**Quick Reference Version**: 1.0.0
**Last Updated**: October 25, 2025
documentation/dev/README.developer-guide.md
+582
View File
@@ -0,0 +1,582 @@
---
title: "Sirius Developer Guide"
description: "Complete guide for developers working on Sirius, including environment setup, development workflows, and best practices."
template: "TEMPLATE.guide"
version: "1.0.0"
last_updated: "2025-01-03"
author: "Development Team"
tags: ["development", "guide", "workflow", "docker", "environment"]
categories: ["development", "guide"]
difficulty: "beginner"
prerequisites: ["docker", "git", "nodejs", "go"]
related_docs:
- "README.development.md"
- "README.docker-architecture.md"
- "README.container-testing.md"
dependencies:
- "scripts/switch-env.sh"
- "docker-compose.yaml"
- "docker-compose.dev.yaml"
- "docker-compose.prod.yaml"
llm_context: "high"
search_keywords:
[
"developer guide",
"development workflow",
"environment switching",
"docker development",
"hot reloading",
"development setup",
]
---
# Sirius Developer Guide
## Overview
This guide provides everything you need to know to develop effectively on the Sirius project. Whether you're working on the UI, API, engine, or any other component, this guide will help you get up and running quickly.
## Quick Start
```bash
# Clone the repository
git clone https://github.com/SiriusScan/Sirius.git
cd Sirius
# Start development environment
./scripts/switch-env.sh dev
# Access the application
open http://localhost:3000
```
## Development Environments
Sirius supports three distinct development environments, each optimized for different use cases:
### 🚀 Development Mode (Recommended)
**Best for: UI/API development, testing, most development tasks**
```bash
./scripts/switch-env.sh dev
```
**Features:**
- Hot reloading for instant code changes
- Volume mounts for live code updates
- Development-optimized builds
- SQLite database for faster development
- Debug logging enabled
**What's Running:**
- UI: `npm run dev` (Next.js development server)
- API: Go development mode with live reloading
- Engine: Development mode with volume mounts
### 🏭 Production Mode
**Best for: Testing production builds, performance testing, pre-deployment validation**
```bash
./scripts/switch-env.sh prod
```
**Features:**
- Optimized production builds
- PostgreSQL database
- Production authentication
- Performance optimizations
- Security hardening
**What's Running:**
- UI: `npm start` (Next.js production server)
- API: Pre-built Go binary
- Engine: Production-optimized runtime
### ⚙️ Base Mode
**Best for: Standard operations, CI/CD, basic testing**
```bash
./scripts/switch-env.sh base
```
**Features:**
- Default configuration
- Balanced performance
- Standard resource allocation
- Core functionality only
## Environment Switching
The `scripts/switch-env.sh` script handles seamless switching between environments:
```bash
# Switch to development
./scripts/switch-env.sh dev
# Switch to production
./scripts/switch-env.sh prod
# Switch to base
./scripts/switch-env.sh base
```
**What the script does:**
1. Stops all running containers
2. Removes old images to prevent cache conflicts
3. Builds the appropriate environment with correct build targets
4. Starts all services
5. Shows container status and access URLs
## Development Workflow
### Daily Development
1. **Start your day:**
```bash
./scripts/switch-env.sh dev
```
2. **Make your changes:**
- Edit UI code in `sirius-ui/src/`
- Edit API code in `sirius-api/`
- Edit engine code in `sirius-engine/`
3. **See changes instantly:**
- UI changes appear immediately (hot reloading)
- API changes require container restart
- Engine changes require container restart
4. **Test your changes:**
```bash
# Check container status
docker compose ps
# View logs
docker compose logs sirius-ui
docker compose logs sirius-api
```
### Testing Production Builds
1. **Switch to production mode:**
```bash
./scripts/switch-env.sh prod
```
2. **Test your changes:**
- Verify UI renders correctly
- Test API endpoints
- Check performance characteristics
3. **Switch back to development:**
```bash
./scripts/switch-env.sh dev
```
### Working with Different Components
#### Frontend Development (UI)
```bash
# Start development mode
./scripts/switch-env.sh dev
# Make changes to React components
# Changes appear instantly in browser
# Test production build
./scripts/switch-env.sh prod
```
#### Backend Development (API)
```bash
# Start development mode
./scripts/switch-env.sh dev
# Make changes to Go code
# Restart API container to see changes
docker compose restart sirius-api
# Check API logs
docker compose logs sirius-api -f
```
#### Engine Development
```bash
# Start development mode
./scripts/switch-env.sh dev
# Make changes to engine code
# Restart engine container to see changes
docker compose restart sirius-engine
# Check engine logs
docker compose logs sirius-engine -f
```
## Project Structure
```
Sirius/
├── sirius-ui/ # Next.js frontend
│ ├── src/ # React components and pages
│ ├── Dockerfile # Multi-stage build (dev/prod)
│ ├── start-dev.sh # Development startup script
│ └── start-prod.sh # Production startup script
├── sirius-api/ # Go REST API
│ ├── cmd/ # API entry point
│ ├── internal/ # Internal API code
│ └── Dockerfile # Multi-stage build (dev/runner)
├── sirius-engine/ # Multi-service engine
│ ├── cmd/ # Engine entry point
│ ├── internal/ # Internal engine code
│ └── Dockerfile # Multi-stage build (dev/runtime)
├── docker-compose.yaml # Base configuration
├── docker-compose.dev.yaml # Development overrides
├── docker-compose.prod.yaml # Production overrides
└── scripts/
└── switch-env.sh # Environment switching script
```
## Docker Architecture
### Multi-Stage Builds
Each service uses multi-stage Dockerfiles to optimize for different environments:
**sirius-ui:**
- `development`: Full dev environment with hot reloading
- `production`: Optimized production build
**sirius-api:**
- `development`: Go development with live reloading
- `runner`: Pre-built Go binary for production
**sirius-engine:**
- `development`: Full development environment
- `runtime`: Production-optimized runtime
### Image Tagging Strategy
- **Development**: `sirius-sirius-ui:dev`
- **Production**: `sirius-sirius-ui:prod`
- **Base**: `sirius-sirius-ui:latest`
This prevents Docker cache conflicts when switching environments.
## CI/CD Integration
### Pre-commit Validation
**Purpose**: Fast validation to catch obvious issues before commit
**Duration**: ~30 seconds
**What's Tested**:
- Docker Compose configuration validation
- Documentation linting
- Basic syntax checks
- Code formatting
**Commands**:
```bash
# Pre-commit validation (automatic)
git commit # Runs quick validation automatically
# Manual validation
cd testing/container-testing
make build-all # Validate all Docker Compose configs
make lint-docs-quick # Quick documentation checks
```
### CI/CD Pipeline
**Purpose**: Comprehensive testing of all changes
**Duration**: ~5-10 minutes
**What's Tested**:
- Docker container builds (all services)
- Service health checks
- Integration testing
- Cross-service communication
- Production build validation
**Triggers**:
- Pull requests to main branch
- Pushes to main branch
- Hotfix pushes
### Local Testing
**Purpose**: Manual testing during development
**Available Commands**:
```bash
# Full test suite
cd testing/container-testing
make test-all
# Individual tests
make test-build # Test Docker builds
make test-health # Test service health
make test-integration # Test integration
# Quick validation
make build-all # Validate configs
make lint-docs-quick # Quick docs check
```
### Testing Strategy
| Scenario | Pre-commit | CI/CD | Local Testing |
| ----------------------- | ------------------- | --------------- | --------------- |
| **Feature Development** | ✅ Quick validation | ❌ No | ✅ Full testing |
| **Pull Request** | ✅ Quick validation | ✅ Full testing | ✅ Full testing |
| **Main Branch** | ✅ Quick validation | ✅ Full testing | ✅ Full testing |
| **Documentation** | ✅ Quick validation | ❌ No | ✅ Full testing |
## Common Tasks
### Viewing Logs
```bash
# All services
docker compose logs -f
# Specific service
docker compose logs sirius-ui -f
docker compose logs sirius-api -f
docker compose logs sirius-engine -f
```
### Accessing Containers
```bash
# Open shell in container
docker compose exec sirius-ui sh
docker compose exec sirius-api sh
docker compose exec sirius-engine sh
```
### Restarting Services
```bash
# Restart specific service
docker compose restart sirius-ui
# Restart all services
docker compose restart
```
### Checking Service Health
```bash
# View container status
docker compose ps
# Check health endpoints
curl http://localhost:3000/api/health
curl http://localhost:9001/api/v1/health
curl http://localhost:5174/health
```
### Database Access
```bash
# Development (SQLite)
docker compose exec sirius-ui ls -la /app/dev.db
# Production (PostgreSQL)
docker compose exec sirius-postgres psql -U postgres -d sirius
```
## Troubleshooting
### Environment Switching Issues
**Problem**: Wrong environment running despite switching
**Solution**: The script automatically handles this, but if issues persist:
```bash
# Force clean rebuild
docker system prune -f
./scripts/switch-env.sh dev
```
### Hot Reloading Not Working
**Problem**: UI changes not appearing instantly
**Solution**: Ensure you're in development mode:
```bash
./scripts/switch-env.sh dev
# Check it's running: docker compose exec sirius-ui ps aux
```
### Services Not Starting
**Problem**: Containers failing to start
**Solution**: Check logs and restart:
```bash
docker compose logs sirius-ui
docker compose restart sirius-ui
```
### Port Conflicts
**Problem**: Port already in use
**Solution**: Stop conflicting services:
```bash
# Check what's using the port
lsof -i :3000
lsof -i :9001
# Stop conflicting services
sudo kill -9 <PID>
```
### Database Connection Issues
**Problem**: API can't connect to database
**Solution**: Check database health:
```bash
# Check database status
docker compose ps sirius-postgres
# Check database logs
docker compose logs sirius-postgres
# Restart database
docker compose restart sirius-postgres
```
## Best Practices
### Development Workflow
1. **Always start with development mode** for new features
2. **Test in production mode** before committing
3. **Use hot reloading** for UI development
4. **Check logs regularly** to catch issues early
5. **Switch environments** to test different scenarios
### Code Organization
1. **Keep components focused** and single-purpose
2. **Use TypeScript** for better type safety
3. **Follow naming conventions** consistently
4. **Write tests** for critical functionality
5. **Document complex logic** with comments
### Docker Usage
1. **Use the switch script** instead of manual docker commands
2. **Don't edit Docker Compose files** directly unless necessary
3. **Clean up unused images** regularly
4. **Monitor resource usage** during development
5. **Test in multiple environments** before deploying
### Git Workflow
1. **Create feature branches** for new work
2. **Test thoroughly** before merging
3. **Write descriptive commit messages**
4. **Keep commits focused** and atomic
5. **Use pull requests** for code review
## Advanced Usage
### Custom Environment Variables
Create a `.env.local` file for custom environment variables:
```bash
# .env.local
NODE_ENV=development
API_URL=http://localhost:9001
DEBUG=true
```
### Volume Mounts for Live Development
For advanced development scenarios, you can mount local directories:
```bash
# Edit docker-compose.dev.yaml to add volume mounts
volumes:
- ./sirius-ui/src:/app/src
- ./sirius-api:/app
```
### Debugging
Enable debug mode for more verbose logging:
```bash
# Set debug environment variable
export DEBUG=true
./scripts/switch-env.sh dev
```
### Performance Testing
Use production mode for performance testing:
```bash
./scripts/switch-env.sh prod
# Run your performance tests
```
## Getting Help
### Documentation
- [Docker Architecture](README.docker-architecture.md) - Detailed Docker setup
- [Development Setup](README.development.md) - Legacy development guide
- [Container Testing](README.container-testing.md) - Testing procedures
### Common Issues
- Check the troubleshooting section above
- Review container logs for error messages
- Ensure all prerequisites are installed
- Verify Docker is running properly
### Support
- Create an issue on GitHub for bugs
- Use discussions for questions
- Check existing issues before creating new ones
---
_This guide follows the Sirius Documentation Standard. For questions about documentation structure, see [ABOUT.documentation.md](ABOUT.documentation.md)._
documentation/dev/README.development.md
+260
View File
@@ -0,0 +1,260 @@
---
title: "Sirius Development Setup"
description: "Comprehensive guide for setting up and using the Sirius development environment, including standard and extended development modes with local repository integration."
template: "TEMPLATE.documentation-standard"
version: "1.0.0"
last_updated: "2026-04-07"
author: "Development Team"
tags: ["development", "setup", "docker", "workflow"]
categories: ["development", "setup"]
difficulty: "intermediate"
prerequisites: ["docker", "git"]
related_docs:
- "README.container-testing.md"
- "ABOUT.documentation.md"
dependencies:
- "docker-compose.yaml"
- "scripts/dev-setup.sh"
llm_context: "medium"
search_keywords:
["development setup", "docker compose", "local development", "workflow"]
---
# Sirius Development Setup
## Quick Start
**New in v1.0.0**: Use the improved environment switching system for easier development.
```bash
# Generate/merge required .env values once (installer-first)
docker compose -f docker-compose.installer.yaml run --rm sirius-installer
# Modern development workflow (recommended)
./scripts/switch-env.sh dev
# Legacy development (uses built-in repositories)
./scripts/dev-setup.sh start
# Extended development (with local repository mounts)
./scripts/dev-setup.sh init # Create local overrides
./scripts/dev-setup.sh start-extended # Start with local mounts
```
> **Note**: For the complete developer experience, see [README.developer-guide.md](README.developer-guide.md) for comprehensive development workflows and best practices.
## Development Modes
### 🎯 Standard Development
**Best for: UI/API development, testing, most development tasks**
```bash
./scripts/dev-setup.sh start
```
- Uses built-in repositories from Docker images
- No local repository setup required
- All services start with `go run` in development mode
- Live reloading for UI changes
- Uses secrets generated by `docker-compose.installer.yaml` (`.env`) for service auth/DB consistency
### 🔧 Extended Development
**Best for: Working on scanner, terminal, or agent code**
```bash
# 1. Set up local repositories (one-time)
mkdir -p ../minor-projects && cd ../minor-projects
git clone https://github.com/SiriusScan/app-scanner.git
git clone https://github.com/SiriusScan/app-terminal.git
git clone https://github.com/SiriusScan/app-agent.git
cd ../Sirius
# 2. Initialize local overrides
./scripts/dev-setup.sh init
# 3. Edit docker-compose.local.yaml (uncomment what you need)
nano docker-compose.local.yaml
# 4. Start extended development
./scripts/dev-setup.sh start-extended
```
**Hot reloading in extended mode (engine services):**
Each minor-project ships its own `.air.toml` (`app-agent/.air.toml`, `app-terminal/.air.toml`, `app-scanner/.air.toml`). When `GO_ENV=development` (the default for `docker-compose.dev.yaml`), `start-enhanced.sh` checks for `.air.toml` in the bind-mounted source and uses [Air](https://github.com/air-verse/air) for live rebuilds; if `.air.toml` is absent (or `air` is missing from the image) it falls back to plain `go run`. The matrix:
| Service | Bind mount path in container | Live-reload tool | Entry point |
| --- | --- | --- | --- |
| `app-agent` | `/app-agent` | `air` | `cmd/server/main.go` |
| `app-terminal` | `/app-terminal` | `air` | `cmd/main.go` |
| `app-scanner` | `/app-scanner` | `air` (`CGO_ENABLED=1`) | `main.go` |
Notes:
- Changes to bind-mounted source are picked up automatically by Air; you should see a rebuild log line in `docker logs sirius-engine` within ~1s of saving.
- `app-scanner` requires CGO and `libpcap`; the container ships both, so the in-container Air build works out of the box.
- The orphan `sirius-engine/.air.toml` that previously lived in the engine image was removed in the April 2026 dev-mode overhaul; per-service `.air.toml` is the only supported configuration.
- Changes are not tracked by the main Sirius repository (the bind mounts are git-ignored).
**When Air isn't enough — hot-swap from local source:**
Sometimes you want to validate a change against a *production-mode* engine without waiting for a full container rebuild (e.g. you're testing a binary against `docker-compose.prod.yaml`, or you want to confirm a release-style build before bumping a pin). Use the root `Makefile` targets:
```bash
make engine-mode # show whether the running engine is dev or prod
make agent-hot-swap-from-local
make terminal-hot-swap-from-local
make scanner-hot-swap-from-local # CGO; fails cleanly if host lacks libpcap
make engine-rebuild-from-local # all three, best-effort
```
What they do: cross-compile the binary on the host (`linux/$(uname -m)` by default; override with `HOST_GOARCH=amd64`), `docker cp` it into the engine container at the production binary path (e.g. `/app-agent-src/server`), and restart the engine.
Important guard rails:
- **The targets refuse to run in dev mode** (`GO_ENV=development`). In dev, `/app-terminal` and `/app-scanner` are bind-mounted, so `docker cp` would write through to your host repo; and Air owns the live binary at `./tmp/main`, so the swapped production-path binary wouldn't even be executed. The right loop in dev is *edit source → Air rebuilds*. Override with `FORCE_HOT_SWAP=1` only if you know what you want.
- **Scanner cross-compile may fail on macOS hosts**. The scanner uses `cgo` + `libpcap`; the macOS toolchain can't satisfy linux cgo headers. The target prints the actual `cgo` error and a copy-pasteable in-container build command. This is expected and not a Makefile bug.
- **These targets are an inner-loop convenience, not a release path.** Once the change is good, push it to the relevant minor-project and bump the engine pin per [README.engine-component-pinning.md](architecture/README.engine-component-pinning.md). The CI pipeline is the single source of truth for shipped images.
## File Structure
```
Sirius/
├── docker-compose.yaml # Base configuration
├── docker-compose.override.yaml # 🔒 Committed: Safe development defaults
├── docker-compose.local.example.yaml # 🔒 Committed: Template for local overrides
├── docker-compose.local.yaml # 🚫 Git-ignored: Your personal overrides
└── scripts/dev-setup.sh # 🔒 Committed: Development helper
```
## Safety Features
### 🛡️ Prevents Accidental Commits
- `docker-compose.local.yaml` is git-ignored
- CI/CD validates that volume mounts stay commented in `docker-compose.override.yaml`
- Pre-commit hook auto-fixes uncommented volume mounts
### 🔧 Developer-Friendly
- Easy setup with `./scripts/dev-setup.sh init`
- Template file shows all available options
- Helper script for common development tasks
## Available Commands
```bash
./scripts/dev-setup.sh init # Create local overrides from template
./scripts/dev-setup.sh start # Standard development mode
./scripts/dev-setup.sh start-extended # Extended development with local repos
./scripts/dev-setup.sh stop # Stop all services
./scripts/dev-setup.sh status # Show container status
./scripts/dev-setup.sh logs [service] # Show logs
./scripts/dev-setup.sh shell <service> # Open shell in container
./scripts/dev-setup.sh clean # Clean containers and volumes
```
## Troubleshooting
### Permission denied (Linux, Docker dev bind mounts)
`docker-compose.dev.yaml` runs **sirius-ui** and **sirius-api** as UID/GID **1001** (matching the image users). If you cloned the repo as **root** or files under `sirius-ui/` / `sirius-api/` are root-owned, processes inside the container get **EACCES** when Next.js or `go build` writes into mounted directories.
**Preferred:** fix ownership on the host (adjust path to your checkout):
```bash
sudo chown -R 1001:1001 sirius-ui sirius-api
# If you mount minor-projects into those containers:
sudo chown -R 1001:1001 ../minor-projects/go-api ../minor-projects/app-system-monitor ../minor-projects/app-administrator
```
**Quick dev-only workaround:** in `.env` set:
```bash
SIRIUS_DEV_CONTAINER_UID=0
SIRIUS_DEV_CONTAINER_GID=0
```
Then recreate containers: `docker compose -f docker-compose.yaml -f docker-compose.dev.yaml up --build -d --force-recreate`. This runs those two services as root inside the container (acceptable for a local lab, not for production images).
### Volume Mounts Not Working
```bash
# Check if local file exists
ls -la docker-compose.local.yaml
# Check if repositories exist
ls -la ../minor-projects/
# Verify you're using start-extended
./scripts/dev-setup.sh start-extended
```
### Git Commits Being Rejected
Your CI/CD is preventing commits with uncommented volume mounts:
```bash
# Fix automatically
git add docker-compose.override.yaml
git commit # Pre-commit hook will fix and re-stage
# Or fix manually - comment out volume mounts with #
nano docker-compose.override.yaml
```
### Services Not Starting
```bash
# Ensure installer-generated .env exists and is up to date
docker compose -f docker-compose.installer.yaml run --rm sirius-installer
# Check container status
./scripts/dev-setup.sh status
# View logs for specific service
./scripts/dev-setup.sh logs sirius-engine
# Restart clean
./scripts/dev-setup.sh stop
./scripts/dev-setup.sh clean
./scripts/dev-setup.sh start
```
## Best Practices
1. **Use standard mode** for most development work
2. **Only use extended mode** when working on engine/scanner/terminal/agent code
3. **Never commit** `docker-compose.local.yaml`
4. **Always test** your changes work in standard mode before committing
5. **Keep local overrides minimal** - only uncomment what you're actively developing
## Migration from Old Setup
If you were previously editing `docker-compose.override.yaml` directly:
```bash
# 1. Reset override file to clean state
git checkout docker-compose.override.yaml
# 2. Set up new local overrides
./scripts/dev-setup.sh init
# 3. Move your customizations to docker-compose.local.yaml
nano docker-compose.local.yaml
# 4. Start with new system
./scripts/dev-setup.sh start-extended
```
## CI/CD Integration
The repository includes automatic validation:
- **GitHub Actions**: Validates docker-compose files on PRs
- **Pre-commit Hook**: Auto-fixes volume mount issues
- **Git Ignore**: Prevents local files from being committed
This ensures the repository stays clean and deployments are predictable.
documentation/dev/ai-rules/ABOUT.ai-rules.md
+105
View File
@@ -0,0 +1,105 @@
---
title: "About AI Rules"
description: "Documentation for AI/LLM interaction rules and guidelines within the Sirius project, including cursor rules and context shaping strategies."
template: "TEMPLATE.about"
version: "1.0.0"
last_updated: "2025-01-03"
author: "AI Assistant"
tags: ["ai", "llm", "rules", "cursor", "context"]
categories: ["project-management", "development"]
difficulty: "intermediate"
prerequisites: ["ABOUT.documentation.md"]
related_docs:
- "ABOUT.documentation.md"
- "README.documentation-index.md"
dependencies: []
llm_context: "high"
search_keywords:
["ai rules", "llm context", "cursor rules", "documentation rules"]
---
# About AI Rules
## Purpose
This document explains how AI/LLM interaction rules and guidelines work within the Sirius project. These rules help shape how Large Language Models interact with our codebase and documentation system.
## When to Use
- **Setting up AI development environment** - Understanding how AI tools should interact with our project
- **Creating new AI rules** - When adding new guidelines for AI interaction
- **Troubleshooting AI context issues** - When AI tools aren't providing the right context
- **Optimizing AI performance** - When improving how AI systems understand our project
## How to Use
1. **Read the cursor rules** to understand current AI interaction guidelines
2. **Check the documentation system** for how AI should consume our docs
3. **Follow the context shaping strategies** for optimal AI performance
4. **Update rules as needed** when project structure changes
## What It Is
### Cursor Rules Integration
Our project uses cursor rules to guide AI interactions:
- **Code understanding** - How AI should interpret our codebase
- **Documentation consumption** - How AI should use our documentation system
- **Context building** - How AI should build comprehensive project context
- **Best practices** - Guidelines for AI-assisted development
### AI Context Optimization
Our documentation system is designed for optimal AI consumption:
- **YAML front matter** - Machine-readable metadata
- **Structured relationships** - Clear document connections
- **LLM context levels** - Prioritized information for AI
- **Search keywords** - Optimized for AI discovery
### Integration Points
- **Cursor IDE** - Primary AI development environment
- **Documentation system** - AI context source
- **Codebase structure** - AI understanding target
- **Development workflows** - AI assistance scope
## Troubleshooting
### FAQ
**Q: How do I update cursor rules for new documentation?**
A: Update the cursor rules file to include new documentation patterns and relationships.
**Q: Why isn't AI understanding our documentation structure?**
A: Check that YAML front matter is complete and relationships are properly defined.
**Q: How do I optimize AI context for specific tasks?**
A: Use the LLM context levels and search keywords in documentation front matter.
### Command Reference
| Command | Purpose | Example |
| ----------------------------- | ------------------------------ | -------------------------------------------- |
| `grep -r "llm_context: high"` | Find high-priority docs for AI | `grep -r "llm_context: high" documentation/` |
| `grep -r "template:"` | Check template usage | `grep -r "template:" documentation/` |
| `grep -r "related_docs:"` | Find document relationships | `grep -r "related_docs:" documentation/` |
### Common Issues
| Issue | Symptoms | Solution |
| ----------------------------- | ------------------------ | -------------------------------------------- |
| AI not finding relevant docs | Poor context building | Check LLM context levels and search keywords |
| AI misunderstanding structure | Incorrect template usage | Verify template compliance and relationships |
| AI missing dependencies | Broken document links | Validate related_docs and dependencies |
## Lessons Learned
### [Date] - [What was learned]
[Description of AI rules lesson learned and how it improved the system]
---
_This document follows the Sirius Documentation Standard. For questions about documentation structure, see [ABOUT.documentation.md](../ABOUT.documentation.md)._
documentation/dev/ai-rules/README.ai-identities.md
+1021
View File
File diff suppressed because it is too large Load Diff
documentation/dev/ai-rules/README.playwright.md
+577
View File
@@ -0,0 +1,577 @@
---
title: "Playwright Browser Testing Guide"
description: "Guide for using Playwright MCP for browser testing, troubleshooting, and validation"
template: "TEMPLATE.guide"
version: "1.0.0"
last_updated: "2025-10-18"
author: "Sirius Development Team"
tags: ["testing", "playwright", "browser", "e2e", "validation"]
categories: ["testing", "development", "troubleshooting"]
difficulty: "intermediate"
prerequisites: ["Docker", "MCP Server"]
related_docs:
- "README.container-testing.md"
- "CHECKLIST.testing-by-type.md"
llm_context: "high"
search_keywords: ["playwright", "browser testing", "e2e", "automation", "mcp"]
---
# Playwright Browser Testing Guide
## Overview
This guide covers using the Playwright MCP (Model Context Protocol) server for automated browser testing, troubleshooting, and implementation validation in the Sirius project.
## Docker Network Access
### Critical: Host Access from Docker
The Playwright MCP server runs inside Docker and cannot access `localhost` directly. Use Docker's special DNS name:
```
✅ CORRECT: http://host.docker.internal:3000
❌ INCORRECT: http://localhost:3000
❌ INCORRECT: http://127.0.0.1:3000
```
### Service URLs
| Service | Docker URL | Description |
| ------------------- | ----------------------------------- | ------------------- |
| UI | `http://host.docker.internal:3000` | Next.js frontend |
| API | `http://host.docker.internal:9001` | Go Fiber backend |
| RabbitMQ Management | `http://host.docker.internal:15672` | Queue management UI |
| PostgreSQL | `host.docker.internal:5432` | Database (not HTTP) |
## When to Use Playwright
### ✅ Use Playwright For:
1. **Implementation Validation**
- Verify new features work end-to-end
- Test user flows after major changes
- Confirm UI components render correctly
2. **Bug Troubleshooting**
- Reproduce user-reported issues
- Capture screenshots of error states
- Inspect console logs and network requests
- Debug form submissions and interactions
3. **Design Verification**
- Check responsive layouts
- Verify component positioning
- Test accessibility features
- Validate user experience flows
4. **Integration Testing**
- Test API integration points
- Verify data flow between UI and backend
- Check authentication flows
- Validate form submissions
5. **Regression Testing**
- Verify existing functionality still works
- Test critical user paths
- Check for breaking changes
### ❌ Don't Use Playwright For:
1. **Unit Testing** - Use Jest/Vitest instead
2. **Backend API Testing** - Use curl/API testing tools
3. **Performance Testing** - Use dedicated tools
4. **Load Testing** - Use k6 or similar tools
## Common Testing Scenarios
### 1. Testing Authentication Flow
```yaml
Example:
- Navigate to login page
- Fill in credentials
- Click submit
- Verify redirect to dashboard
- Check session state
```
### 2. Testing Form Submission
```yaml
Example:
- Navigate to scanner page
- Fill in target IP address
- Select template from dropdown
- Click "Start Scan"
- Verify API requests made
- Check UI updates
```
### 3. Testing API Integration
```yaml
Example:
- Navigate to page that loads data
- Wait for API calls to complete
- Check network requests
- Verify data displays correctly
- Test error handling
```
### 4. Debugging UI Issues
```yaml
Example:
- Navigate to problematic page
- Take screenshot of current state
- Check console for errors
- Inspect element states
- Verify network requests
```
## Playwright MCP Tools Reference
### Navigation
**`browser_navigate`** - Navigate to a URL
```
Use: Load a page
URL: http://host.docker.internal:3000/scanner
```
**`browser_navigate_back`** - Go to previous page
```
Use: Test back button functionality
```
### Interaction
**`browser_click`** - Click an element
```
Parameters:
- element: Human-readable description
- ref: Element reference from snapshot
Example: Click "Start Scan" button
```
**`browser_type`** - Type text into input
```
Parameters:
- element: Input field description
- ref: Element reference
- text: Text to type
- slowly: Type one character at a time (optional)
- submit: Press Enter after typing (optional)
```
**`browser_fill_form`** - Fill multiple form fields
```
Use: Fill entire forms efficiently
Fields: Array of {name, type, ref, value}
```
**`browser_select_option`** - Select dropdown option
```
Use: Choose from select/combobox
Values: Array of option values
```
### Inspection
**`browser_snapshot`** - Capture page state
```
Use: Get current page structure
Returns: Accessibility tree with element refs
```
**`browser_take_screenshot`** - Take visual screenshot
```
Use: Capture visual state
Options: fullPage, element, filename
Note: Cannot interact based on screenshot
```
**`browser_console_messages`** - Get console logs
```
Use: Debug JavaScript errors
Options: onlyErrors to filter
```
**`browser_network_requests`** - View network activity
```
Use: Verify API calls
Returns: All requests since page load
```
### Waiting
**`browser_wait_for`** - Wait for condition
```
Options:
- text: Wait for text to appear
- textGone: Wait for text to disappear
- time: Wait N seconds
```
### Advanced
**`browser_hover`** - Hover over element
```
Use: Test hover states, tooltips
```
**`browser_drag`** - Drag and drop
```
Use: Test drag interactions
```
**`browser_evaluate`** - Execute JavaScript
```
Use: Complex DOM manipulation or queries
```
**`browser_tabs`** - Manage browser tabs
```
Actions: list, new, close, select
```
## Testing Workflow Examples
### Example 1: Test Scanner Page
```yaml
Step 1: Navigate
- URL: http://host.docker.internal:3000/scanner
- Wait for: Page load
Step 2: Login (if needed)
- Fill username: admin
- Fill password: password
- Click: "Join the Pack"
- Wait for: Dashboard redirect
Step 3: Navigate to Scanner
- Click: Scanner link
- Wait for: Page load
Step 4: Add Target
- Type in IP field: 192.168.1.100
- Click: Add button
- Verify: Target appears in list
Step 5: Start Scan
- Select template: "High Risk Scan"
- Click: "Start Scan"
- Check network: queue.sendMsg called
- Check network: store.setValue called
- Verify: UI updates with scan details
```
### Example 2: Debug Form Not Submitting
```yaml
Step 1: Navigate to Form
- URL: http://host.docker.internal:3000/form-page
Step 2: Fill Form
- Fill all required fields
- Take screenshot: "before-submit.png"
Step 3: Submit
- Click: Submit button
- Get console messages
- Get network requests
Step 4: Analyze
- Check console for errors
- Verify API request was made
- Check API response status
- Take screenshot: "after-submit.png"
```
### Example 3: Verify Component Rendering
```yaml
Step 1: Navigate
- URL: http://host.docker.internal:3000/component-page
Step 2: Wait for Load
- Wait for: Expected text
- Wait: 2 seconds for async data
Step 3: Capture State
- Take snapshot
- Check for: Expected elements
- Verify: Element text content
Step 4: Test Interaction
- Click: Interactive element
- Verify: State change
- Take screenshot: Show result
```
## Best Practices
### 1. Always Use host.docker.internal
```yaml
✅ GOOD:
- http://host.docker.internal:3000/scanner
❌ BAD:
- http://localhost:3000/scanner
- http://127.0.0.1:3000/scanner
```
### 2. Wait for Async Operations
```yaml
✅ GOOD:
- Navigate to page
- Wait 2-3 seconds
- Check for loaded content
- Interact with elements
❌ BAD:
- Navigate to page
- Immediately interact (might not be ready)
```
### 3. Use Snapshots for Element Refs
```yaml
✅ GOOD:
- Take snapshot
- Get element ref (e.g., e187)
- Use ref for interaction
❌ BAD:
- Guess element selectors
- Use outdated refs
```
### 4. Check Network Requests
```yaml
✅ GOOD:
- Perform action
- Get network requests
- Verify expected API calls
❌ BAD:
- Assume API was called
- Skip verification
```
### 5. Capture Console Logs
```yaml
✅ GOOD:
- Check console after actions
- Look for errors or warnings
- Verify debug logs
❌ BAD:
- Ignore console output
- Miss JavaScript errors
```
## Troubleshooting
### Issue: Cannot Connect to Page
```
Error: net::ERR_CONNECTION_REFUSED
Solution: Use host.docker.internal instead of localhost
```
### Issue: Element Not Found
```
Problem: Element ref is stale
Solution: Take fresh snapshot before interaction
```
### Issue: Timeout Errors
```
Problem: Page loads slowly
Solution: Increase wait time or wait for specific condition
```
### Issue: Network Requests Not Visible
```
Problem: Requests made before navigation
Solution: Navigate first, then check requests
```
## Testing Checklist
### Pre-Test Setup
- [ ] Ensure Docker containers are running
- [ ] Verify services are healthy
- [ ] Check that UI is accessible at host.docker.internal:3000
### During Testing
- [ ] Use host.docker.internal URLs
- [ ] Wait for page loads
- [ ] Take snapshots before interactions
- [ ] Check console for errors
- [ ] Verify network requests
- [ ] Capture screenshots for visual issues
### Post-Test Analysis
- [ ] Review console messages
- [ ] Check network request logs
- [ ] Verify expected API calls
- [ ] Document any issues found
- [ ] Take final screenshots
## Common Test Scenarios
### Authentication
```
1. Navigate to login page
2. Fill credentials
3. Submit form
4. Verify session cookie
5. Check redirect
```
### Form Validation
```
1. Navigate to form
2. Submit without required fields
3. Verify error messages
4. Fill fields correctly
5. Submit and verify success
```
### Data Loading
```
1. Navigate to data page
2. Wait for loading state
3. Verify API calls made
4. Check data displays
5. Test error states
```
### Navigation
```
1. Test all nav links
2. Verify correct pages load
3. Check active states
4. Test back/forward
5. Verify breadcrumbs
```
## Integration with Development
### When to Run Tests
1. **After UI Changes**
- Verify components still render
- Check interactions work
- Test user flows
2. **Before Commits**
- Quick smoke test
- Verify critical paths
- Check for console errors
3. **During Debugging**
- Reproduce reported issues
- Capture error states
- Verify fixes
4. **During Code Review**
- Validate new features
- Check edge cases
- Verify error handling
## LLM Context for Testing
When the AI should use Playwright:
1. **User reports UI bug** → Use Playwright to reproduce
2. **Testing new feature** → Use Playwright to validate
3. **Verifying implementation** → Use Playwright to check
4. **Debugging form issues** → Use Playwright to inspect
5. **Checking API integration** → Use Playwright to verify network calls
When the AI should NOT use Playwright:
1. **Backend-only changes** → Use curl or API testing
2. **Unit test failures** → Check test files directly
3. **Build issues** → Check logs and configuration
4. **Database queries** → Use database tools
## Quick Reference
### Login Sequence
```
1. navigate → http://host.docker.internal:3000
2. wait 3 seconds
3. type username → "admin"
4. type password → "password"
5. click → "Join the Pack"
6. wait for redirect
```
### Scan Flow
```
1. navigate → http://host.docker.internal:3000/scanner
2. type target → "192.168.1.100"
3. click → Add button
4. select template → "High Risk Scan"
5. click → "Start Scan"
6. check network → verify queue.sendMsg
```
### Debug Flow
```
1. navigate → problematic page
2. wait for load
3. take snapshot → see structure
4. get console messages → check errors
5. get network requests → verify API
6. take screenshot → capture visual state
```
---
_For more testing information, see [README.container-testing.md](../test/README.container-testing.md) and [CHECKLIST.testing-by-type.md](../test/CHECKLIST.testing-by-type.md)._
documentation/dev/apps/agent/README.agent-system.md
+1240
View File
File diff suppressed because it is too large Load Diff
documentation/dev/apps/agent/README.agent-template-api.md
+512
View File
@@ -0,0 +1,512 @@
---
title: "Agent Template API Documentation"
description: "API endpoints for managing agent vulnerability detection templates"
template: "TEMPLATE.documentation-standard"
llm_context: "high"
categories: ["api", "agent", "templates"]
tags: ["api", "agent", "templates", "vulnerability", "detection"]
related_docs:
- "README.agent-template-ui.md"
- "ABOUT.documentation.md"
search_keywords:
["agent template api", "template endpoints", "vulnerability detection"]
---
# Agent Template API Documentation
## Overview
The Agent Template API provides endpoints for managing vulnerability detection templates used by Sirius agents. These templates define detection logic for identifying vulnerabilities on target systems.
## Base URL
```
http://localhost:9001/api
```
## Endpoints
### List All Templates
Get all agent templates (standard + custom).
**Endpoint:** `GET /agent-templates`
**Response:**
```json
[
{
"id": "CVE-2021-44228",
"name": "Log4Shell Detection",
"description": "Detects Log4j RCE vulnerability",
"type": "standard",
"severity": "critical",
"author": "Sirius Security Team",
"platforms": ["linux", "darwin", "windows"],
"version": "1.0.0",
"createdAt": "2024-01-15T10:00:00Z",
"updatedAt": "2024-01-15T10:00:00Z",
"source": {
"type": "standard",
"name": "sirius-templates",
"priority": 1
}
}
]
```
---
### Get Single Template
Get a specific template by ID, including its full YAML content.
**Endpoint:** `GET /agent-templates/:id`
**Parameters:**
- `id` (path): Template ID
**Response:**
```json
{
"id": "CVE-2021-44228",
"name": "Log4Shell Detection",
"description": "Detects Log4j RCE vulnerability",
"type": "standard",
"severity": "critical",
"author": "Sirius Security Team",
"platforms": ["linux"],
"version": "1.0.0",
"createdAt": "2024-01-15T10:00:00Z",
"updatedAt": "2024-01-15T10:00:00Z",
"content": "id: CVE-2021-44228\ninfo:\n name: Log4Shell Detection\n..."
}
```
**Error Responses:**
- `404 Not Found`: Template not found
---
### Upload Custom Template
Upload a new custom template from a YAML file.
**Endpoint:** `POST /agent-templates`
**Content-Type:** `multipart/form-data`
**Form Fields:**
- `file` (file, required): YAML template file (.yaml or .yml)
- `author` (string, optional): Override author name
**Response:**
```json
{
"id": "custom-template-123",
"message": "Template uploaded successfully and pushed to agents",
"validation": {
"valid": true,
"errors": [],
"warnings": ["Author field is recommended"]
}
}
```
**Error Responses:**
- `400 Bad Request`: Invalid file or validation failed
```json
{
"error": "Template validation failed",
"validation": {
"valid": false,
"errors": ["Missing required field: info.severity"],
"warnings": []
}
}
```
**Validation Rules:**
- File must be .yaml or .yml extension
- File size must be under 1MB
- Must include required fields: `id`, `info.name`, `info.severity`, `info.description`
- Severity must be one of: `critical`, `high`, `medium`, `low`, `info`
---
### Validate Template
Validate a template without saving it.
**Endpoint:** `POST /agent-templates/validate`
**Content-Type:** `application/json`
**Request Body:**
```json
{
"content": "id: test-template\ninfo:\n name: Test Template\n..."
}
```
**Response:**
```json
{
"valid": true,
"errors": [],
"warnings": ["Tags are recommended for better organization"]
}
```
---
### Update Custom Template
Update an existing custom template.
**Endpoint:** `PUT /agent-templates/:id`
**Parameters:**
- `id` (path): Template ID
**Content-Type:** `application/json`
**Request Body:**
```json
{
"content": "id: test-template\ninfo:\n name: Updated Test Template\n..."
}
```
**Response:**
```json
{
"message": "Template updated and pushed to agents"
}
```
**Error Responses:**
- `400 Bad Request`: Validation failed
- `404 Not Found`: Template not found
- `403 Forbidden`: Cannot update standard templates
---
### Delete Custom Template
Delete a custom template.
**Endpoint:** `DELETE /agent-templates/:id`
**Parameters:**
- `id` (path): Template ID
**Response:**
```json
{
"message": "Template deleted from agents"
}
```
**Error Responses:**
- `404 Not Found`: Template not found
- `403 Forbidden`: Cannot delete standard templates
---
### Test Template on Agent
Execute a template on a specific agent and get results.
**Endpoint:** `POST /agent-templates/:id/test`
**Parameters:**
- `id` (path): Template ID
**Content-Type:** `application/json`
**Request Body:**
```json
{
"agent_id": "agent-123"
}
```
**Response:**
```json
{
"message": "Template test initiated on agent",
"agent_id": "agent-123",
"template_id": "CVE-2021-44228"
}
```
**Note:** This endpoint initiates the test. Results are retrieved asynchronously via agent event logs or response queues.
---
### Deploy Template to Agents
Deploy a template to specific agents or all agents.
**Endpoint:** `POST /agent-templates/:id/deploy`
**Parameters:**
- `id` (path): Template ID
**Content-Type:** `application/json`
**Request Body:**
```json
{
"agent_ids": ["agent-123", "agent-456"]
}
```
**Note:** Empty `agent_ids` array deploys to all agents.
**Response:**
```json
{
"message": "Template deployed to agents",
"agent_count": 2
}
```
---
### Get Template Analytics
Get template effectiveness statistics.
**Endpoint:** `GET /agent-templates/analytics`
**Response:**
```json
{
"top_templates": [
{
"template_id": "CVE-2021-44228",
"template_name": "Log4Shell Detection",
"detection_count": 42,
"execution_count": 150,
"success_rate": 0.95,
"average_execution_time_ms": 245
}
],
"execution_stats": {
"total_executions": 1250,
"total_detections": 105,
"average_execution_time_ms": 195,
"success_rate": 0.91
},
"platform_distribution": {
"linux": 850,
"darwin": 250,
"windows": 150
}
}
```
**Note:** Analytics are aggregated from agent event logs.
---
### Get Template Results History
Get historical execution results for a template.
**Endpoint:** `GET /agent-templates/:id/results`
**Parameters:**
- `id` (path): Template ID
**Response:**
```json
{
"template_id": "CVE-2021-44228",
"results": [
{
"agent_id": "agent-123",
"timestamp": "2024-01-15T10:30:00Z",
"vulnerable": true,
"confidence": 0.95,
"execution_time_ms": 245
}
]
}
```
---
## Template YAML Schema
### Required Fields
```yaml
id: unique-template-id
info:
name: Human-readable template name
severity: critical|high|medium|low|info
description: What this template detects
detection:
steps:
- type: file_hash|file_content|command_version|script
config: {}
```
### Optional Fields
```yaml
info:
author: Author name
version: Template version (e.g., 1.0.0)
references:
- https://example.com/vuln-info
cve:
- CVE-2021-XXXXX
tags:
- apache
- rce
detection:
logic: all|any # Default: all
steps:
- platforms:
- linux
- darwin
weight: 0.8 # 0.0-1.0, affects confidence
```
### Detection Step Types
#### File Hash
```yaml
- type: file_hash
config:
path: /usr/bin/vulnerable-binary
hash: abc123def456...
algorithm: sha256 # sha256, sha1, md5, sha512
```
#### File Content
```yaml
- type: file_content
config:
path: /etc/apache2/apache2.conf
regex: "ServerTokens\\s+Full"
```
#### Command Version
```yaml
- type: command_version
config:
command: ["dpkg-query", "-W", "-f='${Version}'", "openssh-server"]
regex: "^6\\.5\\.1"
exit_code: 0 # optional
```
#### Script
```yaml
- type: script
config:
interpreter: bash|python|powershell
script: |
#!/bin/bash
dpkg-query -W -f='${Version}' openssh-server
regex: "^6\\.5\\.1"
exit_code: 0 # optional
```
---
## Error Codes
| Code | Description |
| ---- | ---------------------------------------------------------------------- |
| 400 | Bad Request - Invalid input or validation failed |
| 403 | Forbidden - Operation not allowed (e.g., modifying standard templates) |
| 404 | Not Found - Template does not exist |
| 500 | Internal Server Error - Server-side error occurred |
---
## Rate Limiting
Currently, no rate limiting is applied. This may change in future versions.
---
## Authentication
Currently, no authentication is required. All authenticated users can manage templates. Role-based access control will be added in a future version.
---
## Examples
### Upload a Custom Template
```bash
curl -X POST http://localhost:9001/api/agent-templates \
-F "file=@my-template.yaml" \
-F "author=Security Team"
```
### Get All Templates
```bash
curl http://localhost:9001/api/agent-templates
```
### Delete a Custom Template
```bash
curl -X DELETE http://localhost:9001/api/agent-templates/my-custom-template
```
### Test Template on Agent
```bash
curl -X POST http://localhost:9001/api/agent-templates/CVE-2021-44228/test \
-H "Content-Type: application/json" \
-d '{"agent_id": "agent-123"}'
```
---
## See Also
- [Agent Template UI Documentation](README.agent-template-ui.md)
- [Agent Template Types Definition](../../app-agent/internal/template/types/types.go)
- [Template System Notes](../../app-agent/project/BRAINSTORM.template-system-notes.md)
documentation/dev/apps/agent/README.agent-template-ui.md
+480
View File
@@ -0,0 +1,480 @@
---
title: "Agent Template UI Documentation"
description: "User interface workflows for managing agent vulnerability detection templates"
template: "TEMPLATE.documentation-standard"
llm_context: "high"
categories: ["ui", "agent", "templates"]
tags: ["ui", "agent", "templates", "vulnerability", "detection", "scanner"]
related_docs:
- "README.agent-template-api.md"
- "ABOUT.documentation.md"
search_keywords: ["agent template ui", "template management", "scanner ui"]
---
# Agent Template UI Documentation
## Overview
The Agent Template UI provides a comprehensive interface for managing vulnerability detection templates within the Sirius Scanner. Users can browse, upload, test, and analyze templates through an intuitive web interface.
## Accessing Template Management
**Navigation:** Scanner → Advanced → Agent → Templates Tab
The Agent Templates interface is located within the Scanner's Advanced configuration section, under the Agent settings.
---
## Interface Components
### 1. Template Browser
The main view for browsing and managing templates.
#### Features
- **Search**: Search templates by name, ID, description, or author
- **Filters**: Filter by type, severity, platform, and source
- **Template Cards**: Visual cards displaying template metadata
- **Actions**: View, edit (custom only), delete (custom only), test, and deploy
#### Template Card Information
Each template card displays:
- Template name and ID
- Description (truncated to 2 lines)
- Severity badge (color-coded)
- Type badge (standard/custom)
- Source information
- Supported platforms
- Author and version
#### Action Buttons
- **View**: Display full template details and YAML content
- **Edit**: Modify custom templates (not available for standard templates)
- **Delete**: Remove custom templates (not available for standard templates)
- **Test**: Execute template on a selected agent
---
### 2. Template Uploader
Upload custom vulnerability detection templates.
#### Upload Process
1. Click "Create Template" or "Upload Template" button
2. Drag and drop a YAML file or click to browse
3. File is automatically validated
4. Review validation results (errors and warnings)
5. Preview template content
6. Optionally override author field
7. Click "Upload Template" to save
#### File Requirements
- File format: `.yaml` or `.yml`
- Maximum size: 1MB
- Must include required YAML fields
- Must pass validation checks
#### Validation
**Required Fields:**
- `id`: Unique template identifier
- `info.name`: Human-readable template name
- `info.severity`: Severity level (critical/high/medium/low/info)
- `info.description`: Description of what the template detects
**Optional but Recommended:**
- `info.author`: Template author
- `info.tags`: Tags for organization
- `info.version`: Template version
- `info.references`: Links to vulnerability information
- `info.cve`: CVE identifiers
**Validation Feedback:**
- ✓ **Errors** (red): Must be fixed before upload
- ⚠ **Warnings** (yellow): Recommended improvements, not blocking
---
### 3. Template Tester
Test templates on live agents to verify detection logic.
#### Testing Process
1. Select a template to test
2. Choose a target agent from the list
3. Click "Run Test" button
4. View real-time execution progress
5. Review detailed test results
#### Agent Selection
**Filters:**
- Status: Online/Offline (only online agents can be selected)
- Platform: Filter by operating system
- Search: Find agents by hostname or IP
**Agent Information Displayed:**
- Hostname
- IP address
- Platform (Linux/macOS/Windows)
- Status (online/offline)
- Last seen timestamp
- Agent version
#### Test Results
**Summary Metrics:**
- **Vulnerability Status**: Vulnerable or Not Vulnerable
- **Confidence Score**: 0-100% (color-coded: green/yellow/orange/red)
- **Execution Time**: Template execution duration in milliseconds
- **Steps Completed**: Number of detection steps executed
**Detailed Results:**
- Evidence collected during detection
- Step-by-step execution results
- Individual step confidence scores
- Error messages (if any)
**Step Details (expandable):**
- Step type (file_hash, file_content, command_version, script)
- Match status (matched/not matched)
- Evidence data
- Error information
---
### 4. Template Analytics
View template effectiveness and execution statistics.
#### Summary Cards
- **Total Executions**: Total number of template runs across all agents
- **Vulnerabilities Found**: Total detections
- **Success Rate**: Percentage of successful executions
- **Average Execution Time**: Mean template execution time
#### Top Performing Templates
Ranked list of templates by detection count, showing:
- Template rank (top 5)
- Template name and ID
- Detection count
- Execution count
- Success rate percentage
- Average execution time
- Visual performance bar
#### Platform Distribution
Breakdown of template executions by platform:
- Linux
- macOS (Darwin)
- Windows
Displayed as:
- Platform icon
- Execution count
- Percentage of total
- Visual progress bar
---
## User Workflows
### Workflow 1: Uploading a Custom Template
1. Navigate to Scanner → Advanced → Agent → Templates
2. Ensure "Enable Agent Templates" is toggled ON
3. Click "Upload Template" button
4. Drag and drop your YAML file or click to browse
5. Wait for automatic validation
6. Review validation results:
- If errors exist, fix your template and re-upload
- Warnings are optional improvements
7. Review the template preview
8. (Optional) Override the author field
9. Click "Upload Template"
10. Template is saved and automatically distributed to all agents
11. Return to template browser to see your new template
**Success Indicators:**
- Green checkmark in validation
- Template appears in browser with "custom" badge
- Confirmation message displayed
---
### Workflow 2: Testing a Template
1. From the template browser, click the test icon on any template
2. Or click "Test Template" and select a template from dropdown
3. Review template information (description, platforms, etc.)
4. Scroll to "Select Target Agent" section
5. Use filters to find appropriate agents:
- Filter by platform if template is platform-specific
- Ensure agent is online (green status badge)
6. Click on an agent card to select it
7. Click "Run Test" button
8. Wait for test execution (typically 1-5 seconds)
9. Review results:
- Check vulnerability status
- Review confidence score
- Examine evidence
- Expand steps for detailed information
10. Test additional agents or return to browser
**Best Practices:**
- Test on multiple agents to verify detection logic
- Test on different platforms if template supports multiple
- Review confidence scores to validate template accuracy
---
### Workflow 3: Managing Templates
#### Viewing Template Details
1. Click "View" button on any template card
2. Review full template information:
- Complete metadata
- Full YAML content (with syntax highlighting)
3. Use "Test Template" to verify functionality
4. Use "Back to Templates" to return
#### Deleting Custom Templates
1. Locate the custom template in browser
2. Click the trash icon (red)
3. Confirm deletion in dialog
4. Template is removed from all agents immediately
**Note:** Standard templates cannot be deleted.
---
### Workflow 4: Monitoring Template Effectiveness
1. Click "Analytics" button in template browser
2. Review summary metrics at the top
3. Check "Top Performing Templates":
- Identify which templates find the most vulnerabilities
- Review success rates
- Identify slow-running templates
4. Review "Platform Distribution":
- Understand which platforms are scanned most
- Identify coverage gaps
5. Use insights to:
- Prioritize template improvements
- Add templates for underrepresented platforms
- Remove ineffective templates
---
## Settings
### Enable Agent Templates
Toggle agent template scanning on or off for all scans.
**Location:** Templates tab → Settings card
**Effect:** When disabled, agents will not execute vulnerability detection templates during scans.
### Template Priority
Control which templates are used based on severity level.
**Options:**
- **High**: Only critical and high severity templates
- **Medium**: Include medium severity templates
- **All**: Use all templates regardless of severity
**Use Cases:**
- **High**: Fast scans focusing on critical vulnerabilities
- **Medium**: Balanced scan coverage
- **All**: Comprehensive vulnerability assessment
---
## Template Card Color Coding
### Severity Colors
- **Critical**: Red (bg-red-500/20, text-red-400)
- **High**: Orange (bg-orange-500/20, text-orange-400)
- **Medium**: Yellow (bg-yellow-500/20, text-yellow-400)
- **Low**: Blue (bg-blue-500/20, text-blue-400)
- **Info**: Gray (bg-gray-500/20, text-gray-400)
### Type Colors
- **Standard**: Green (bg-green-500/20, text-green-400)
- **Custom**: Violet (bg-violet-500/20, text-violet-400)
- **Repository**: Blue (bg-blue-500/20, text-blue-400)
### Status Colors (Agents)
- **Online**: Green
- **Offline**: Gray
---
## Keyboard Shortcuts
Currently, no keyboard shortcuts are implemented. This may be added in a future version.
---
## Troubleshooting
### Template Upload Fails
**Symptoms:** Validation errors prevent upload
**Solutions:**
1. Review error messages carefully
2. Check required fields are present
3. Verify severity value is valid (critical/high/medium/low/info)
4. Ensure YAML syntax is correct
5. Check file size is under 1MB
6. Verify file extension is .yaml or .yml
### Template Test Hangs
**Symptoms:** Test never completes or times out
**Possible Causes:**
- Agent is offline or unreachable
- Template contains infinite loop or long-running command
- Network connectivity issues
**Solutions:**
1. Verify agent is online in agent list
2. Test a different template on the same agent
3. Test the same template on a different agent
4. Check agent logs for errors
5. Review template detection logic for long-running operations
### Template Not Appearing
**Symptoms:** Uploaded template doesn't appear in browser
**Solutions:**
1. Refresh the page
2. Clear any active filters
3. Check template was successfully uploaded (confirmation message)
4. Verify template wasn't deleted by another user
5. Check browser console for JavaScript errors
### Agents Not Listed in Test View
**Symptoms:** No agents available for testing
**Possible Causes:**
- No agents are currently online
- Platform filter is too restrictive
- Agents haven't registered yet
**Solutions:**
1. Check agent status in Terminal or Dashboard
2. Clear platform filter to see all agents
3. Wait for agents to come online
4. Verify agent connectivity
---
## Performance Considerations
### Template Browser
- Displays up to 1000 templates efficiently
- Filtering and search are client-side (instant)
- Template cards use lazy loading for large lists
### Template Testing
- Test results are displayed in real-time
- Step-by-step results are lazy-loaded
- Large evidence data is formatted for readability
### Analytics
- Analytics data is cached for 5 minutes
- Refresh manually to get latest statistics
- Chart rendering is optimized for large datasets
---
## Security Notes
### Template Upload
- All templates are validated before upload
- Malicious YAML is rejected
- File size limits prevent DoS attacks
- Templates are sandboxed during execution
### Template Testing
- Tests only execute on selected agents
- Test results are not persisted
- Agent permissions are enforced
### Template Deletion
- Only custom templates can be deleted
- Deletion is immediate and cannot be undone
- All agents are notified of deletion
---
## Future Enhancements
Planned features for future releases:
1. **Template Editor**: Visual form-based template builder
2. **Template Versioning**: Track and rollback template changes
3. **Template Collections**: Bundle related templates
4. **Template Marketplace**: Share templates with community
5. **Scheduled Scans**: Automatic periodic template execution
6. **Advanced Analytics**: Detection trends over time
7. **Role-Based Access**: Control who can upload/edit templates
8. **Bulk Operations**: Deploy/test multiple templates at once
---
## See Also
- [Agent Template API Documentation](README.agent-template-api.md)
- [Template System Notes](../../app-agent/project/BRAINSTORM.template-system-notes.md)
- [Scanner Advanced Configuration](../ui/README.scanner-advanced.md)
documentation/dev/apps/scanner/ARCHITECTURE.host-deduplication.md
+162
View File
@@ -0,0 +1,162 @@
---
title: "Host Deduplication and Multi-Source Attribution"
description: "Canonical host identity by IP, multi-source attribution (network/agent), scan_sources field, and UI merge/display with SourceIcon and SourceIconRow."
template: "TEMPLATE.custom"
version: "1.0.0"
last_updated: "2025-02-07"
author: "Sirius Team"
tags: ["scanner", "host", "deduplication", "scan_sources", "SourceIcon", "multi-source"]
categories: ["architecture", "scanner"]
difficulty: "intermediate"
prerequisites: ["README.scanner.md", "ARCHITECTURE.sub-scans.md"]
related_docs:
- "documentation/dev/apps/scanner/README.scanner.md"
- "documentation/dev/apps/scanner/ARCHITECTURE.scanner-data-flow.md"
- "documentation/dev/apps/scanner/ARCHITECTURE.sub-scans.md"
dependencies: []
llm_context: "high"
search_keywords:
- "host deduplication"
- "scan_sources"
- "multi-source"
- "SourceIcon"
- "SourceIconRow"
- "EnvironmentTableData"
- "canonical host IP"
---
# Host Deduplication and Multi-Source Attribution
Hosts in the Sirius scanner can be discovered by multiple scan methods (network and agent). This document describes how hosts are identified by a canonical key, how multiple sources are attributed, and how the UI merges and displays them.
## Canonical Host Identity: IP Address
The **primary key** for a host is its **IP address**. Regardless of whether a host was found by a network scan, an agent scan, or both, it is represented once per IP in the merged view.
- **Backend:** `HostEntry` (and persisted host records) use `ip` as the canonical identifier; `id` can be a unique row or document id; `hostname`, `aliases`, and `sources` are optional.
- **UI:** Tables and detail views key hosts by `ip` when merging data from different sub-scans or from the environment summary API.
This allows:
- One row per host in the environment/host table
- Correct aggregation of vulnerability counts and source badges per host
- Stable navigation to host detail (e.g. by IP) regardless of which scan discovered it
---
## Multi-Source Attribution
A host can be discovered by:
- **network** – app-scanner (Nmap, RustScan, etc.)
- **agent** – app-agent (template runs on remote host)
Other sources (e.g. cloud, application) can be added later. Each discovery path that reports a host should tag it with its **source identifier** (e.g. `"network"`, `"agent"`).
### HostEntry.sources (backend / ValKey)
In the live `ScanResult` stored in ValKey, each `HostEntry` can carry a list of sources:
```go
type HostEntry struct {
ID string `json:"id"`
IP string `json:"ip"`
Hostname string `json:"hostname,omitempty"`
Aliases []string `json:"aliases,omitempty"`
Sources []string `json:"sources,omitempty"` // e.g. ["network", "agent"]
}
```
When merging results from multiple sub-scans, the component that writes to `currentScan` should:
- Merge hosts by IP (one entry per IP)
- Set `sources` to the union of all sources that reported that IP (e.g. if both network and agent found it, `sources = ["network", "agent"]`)
---
## EnvironmentTableData.scan_sources (UI)
The UI uses **EnvironmentTableData** for the environment/host table. It includes:
- **scan_source** (optional, legacy): Single source string (e.g. `"network"` or `"agent"`).
- **scan_sources** (optional): Array of source strings for multi-source attribution.
```ts
interface EnvironmentTableData {
hostname: string;
ip: string;
os: string;
vulnerabilityCount: number;
maxCvss?: number;
groups: string[];
tags: string[];
scan_source?: ScanSource; // legacy single source
scan_sources?: string[]; // all discovery sources for this host
}
```
When mapping from API or from live scan results:
- Prefer **scan_sources** (array) when building the table (e.g. from `host.sources` or equivalent).
- Fall back to **scan_source** for older data or single-source responses, and convert to `scan_sources` for consistent display (e.g. `row.scan_sources ?? (row.scan_source ? [row.scan_source] : [])`).
---
## How the UI Merges Hosts from Different Sub-Scans
1. **Live scan results (ValKey):**
The `ScanResult.hosts` array may already be merged by the backend (app-scanner and agent path both updating the same `currentScan` and merging by IP with combined `sources`). The UI then maps `HostEntry[]` to `EnvironmentTableData[]` and sets `scan_sources = host.sources || []`.
2. **Environment summary (API):**
When the UI fetches the environment host list (e.g. `host.getEnvironmentSummary`), the API returns rows that may include a `sources` (or `scan_sources`) field per host. The UI maps that to `EnvironmentTableData.scan_sources`.
3. **Deduplication by IP:**
If the UI ever receives multiple rows for the same IP (e.g. from different endpoints), it should merge them into one row and combine `scan_sources` (union of all source arrays) so that the table shows one row per host with all sources that discovered it.
---
## Source Display: SourceIcon and SourceIconRow
The UI provides two components for showing scan source(s):
### SourceIcon (single source)
- **Purpose:** Renders one scan source (e.g. `"network"` or `"agent"`) as an icon with optional label and tooltip.
- **Usage:** `<SourceIcon source="agent" />`, `<SourceIcon source="network" showLabel />`.
- **Registry:** `SOURCE_ICON_REGISTRY` maps source keys to icon, color, and label (e.g. `agent` → Bot icon, cyan; `network` → Wifi icon, violet).
### SourceIconRow (multiple sources)
- **Purpose:** Renders a row of icons for all sources that discovered a host (or finding).
- **Usage:** `<SourceIconRow sources={["agent", "network"]} />`.
- **Behavior:** Renders one `SourceIcon` per entry in `sources`; tooltip/title can show a combined label (e.g. "Agent + Network").
**Example in host table column:**
- Accessor: `row.scan_sources ?? (row.scan_source ? [row.scan_source] : [])`.
- Cell: `<SourceIconRow sources={sources} />`.
This gives users a clear view of which scan methods discovered each host (network-only, agent-only, or both).
---
## Summary
- **Canonical host identity:** IP address.
- **Multi-source attribution:** Hosts carry `sources` (backend) / `scan_sources` (UI) listing every scan method that discovered them.
- **EnvironmentTableData:** Prefer `scan_sources: string[]`; support legacy `scan_source` by normalizing to an array.
- **UI merge:** One row per IP; `scan_sources` = union of sources for that IP.
- **Display:** Use **SourceIcon** for a single source and **SourceIconRow** for the list of sources on a host (or vulnerability) row.
---
**Related Documentation**
- [ARCHITECTURE.scanner-data-flow.md](./ARCHITECTURE.scanner-data-flow.md) – How host data flows from scanners into ValKey and to the UI
- [ARCHITECTURE.sub-scans.md](./ARCHITECTURE.sub-scans.md) – How network and agent sub-scans contribute hosts
---
**Last Updated:** 2025-02-07
**Version:** 1.0.0
**Maintainer:** Sirius Team
documentation/dev/apps/scanner/ARCHITECTURE.scanner-data-flow.md
+156
View File
@@ -0,0 +1,156 @@
---
title: "Scanner Data Flow - End-to-End Architecture"
description: "End-to-end data flow from UI scan initiation through network/agent scans to result display and persistence."
template: "TEMPLATE.custom"
version: "1.0.0"
last_updated: "2025-02-07"
author: "Sirius Team"
tags: ["scanner", "data-flow", "architecture", "valkey", "rabbitmq", "grpc"]
categories: ["architecture", "scanner"]
difficulty: "intermediate"
prerequisites: ["README.scanner.md"]
related_docs:
- "documentation/dev/apps/scanner/README.scanner.md"
- "documentation/dev/apps/scanner/ARCHITECTURE.sub-scans.md"
- "documentation/dev/apps/scanner/ARCHITECTURE.host-deduplication.md"
- "documentation/dev/architecture/README.architecture.md"
dependencies: []
llm_context: "high"
search_keywords:
- "scanner data flow"
- "scan initiation"
- "ValKey currentScan"
- "network scan flow"
- "agent scan flow"
- "scan persistence"
---
# Scanner Data Flow – End-to-End Architecture
This document describes the end-to-end data flow of the Sirius scanner system from the moment a user starts a scan in the UI until results are displayed and optionally persisted.
## Overview
The scanner system uses a **single live scan state** stored in ValKey under the key `currentScan`. The UI initiates scans via tRPC, triggers one or more sub-scans (network and/or agent), and polls ValKey for progress. When scans complete, results can be persisted from ValKey through go-api into PostgreSQL.
**Components involved:**
- **sirius-ui** (Next.js): Initiates scans, polls for results, displays hosts and vulnerabilities
- **go-api** (Go): REST API and persistence; can proxy queue/store or be called for cancel/persist
- **app-scanner** (Go): Network scanning; consumes from RabbitMQ, updates ValKey
- **app-agent** (Go): Agent-based scanning via gRPC; runs templates on remote hosts, results flow to ValKey
- **ValKey**: Stores live scan result as `currentScan` (base64-encoded JSON)
- **RabbitMQ**: Message queue between API/UI and app-scanner for network scan requests
---
## Network Scan Flow
1. **UI** – User configures targets and scan options, clicks “Start Scan”.
2. **tRPC** – UI calls `queue.sendMsg` (and optionally scanner/start endpoints) and `store.setValue` with key `currentScan` to write initial `ScanResult` (id, status, targets, empty hosts/vulnerabilities, `sub_scans.network`).
3. **go-api / Queue** – Scan request (id, targets, options, priority) is published to the **RabbitMQ** `scan` queue (either via go-api or via tRPC router in sirius-ui that publishes to RabbitMQ).
4. **app-scanner** – Consumes the message from the `scan` queue, expands targets to IPs, runs the scan pipeline (e.g. enumeration → discovery → vulnerability), and for each host/vulnerability discovered:
- Reads current `currentScan` from ValKey (or receives it in context),
- Merges new hosts/vulnerabilities and updates progress,
- Writes updated `ScanResult` back to ValKey key `currentScan`.
5. **UI polling** – `useScanResults` polls `store.getValue({ key: "currentScan" })` (e.g. every 3s). The tRPC store router reads from ValKey and returns the value; UI decodes the base64 JSON and updates hosts/vulnerabilities for display.
So: **UI → tRPC → go-api (or tRPC) → RabbitMQ → app-scanner → ValKey → UI polling (tRPC store.getValue → ValKey).**
---
## Agent Scan Flow
1. **UI** – Same scan start; if agent scan is enabled, UI builds `sub_scans.agent` (status `dispatching`, progress 0/0) and writes initial `ScanResult` to ValKey via `store.setValue` with key `currentScan`.
2. **tRPC** – UI calls `agentScan.dispatchAgentScan` with scan id and agent scan config.
3. **go-api / gRPC** – go-api (or backend invoked by tRPC) uses **gRPC** to communicate with **app-agent** to dispatch template scans to connected agents.
4. **app-agent** – Runs templates on remote hosts, collects results, and reports back (e.g. via gRPC or callback to API). The component that holds the “current scan” state (e.g. go-api or a worker) updates the `currentScan` value in **ValKey**: merges agent-discovered hosts and vulnerabilities, updates `sub_scans.agent` status and progress.
5. **UI polling** – Same as network: UI polls `store.getValue("currentScan")`, decodes the result, and displays agent-origin hosts and vulnerabilities (with source attribution).
So: **UI → tRPC → go-api → gRPC → app-agent → (results) → ValKey → UI polling.**
---
## Persistence: ValKey (Live) → go-api → PostgreSQL
- **Live state** – The only source of truth during an active scan is ValKey key `currentScan`. All sub-scans (network, agent) read-modify-write this key to add hosts, vulnerabilities, and progress.
- **Persistent state** – After a scan completes (or on demand), the API layer can:
- Read the final `ScanResult` from ValKey (`currentScan`),
- Map hosts and vulnerabilities to the persistent model,
- Write to **PostgreSQL** via go-api (hosts, vulnerabilities, scan_sources, etc.).
So: **ValKey (live) → go-api → PostgreSQL (persistent).** The UI can show live data from ValKey and, for history/reporting, data from go-api/PostgreSQL.
---
## Sequence Diagram (Mermaid)
```mermaid
sequenceDiagram
participant UI as sirius-ui
participant tRPC as tRPC (Next.js)
participant API as go-api
participant RMQ as RabbitMQ
participant Scanner as app-scanner
participant Valkey as ValKey
participant gRPC as gRPC
participant Agent as app-agent
participant DB as PostgreSQL
UI->>tRPC: Start scan (targets, options)
tRPC->>Valkey: setValue("currentScan", initial ScanResult)
tRPC->>API: (optional) start / queue
API->>RMQ: Publish scan message (network)
tRPC->>API: dispatchAgentScan (if agent enabled)
API->>gRPC: Dispatch to agents
gRPC->>Agent: Run templates on hosts
loop Network scan
Scanner->>RMQ: Consume scan message
Scanner->>Scanner: Expand targets, run phases
Scanner->>Valkey: Get currentScan
Scanner->>Valkey: Set currentScan (merge hosts/vulns)
end
loop Agent scan
Agent->>Agent: Execute templates
Agent->>API: Report results (or via gRPC)
API->>Valkey: Get currentScan
API->>Valkey: Set currentScan (merge agent results)
end
loop UI polling (e.g. every 3s)
UI->>tRPC: getValue("currentScan")
tRPC->>Valkey: GET currentScan
Valkey-->>tRPC: base64 ScanResult
tRPC-->>UI: ScanResult
UI->>UI: Decode, update hosts/vulnerabilities
end
note over API,DB: On completion or on demand
API->>Valkey: GET currentScan
Valkey-->>API: ScanResult
API->>DB: Persist hosts, vulnerabilities, sources
```
---
## Key Data Structures
- **ValKey key:** `currentScan` (string value = base64-encoded JSON).
- **ScanResult (JSON):** `id`, `status`, `targets`, `hosts` (array of `HostEntry` with `id`, `ip`, `hostname`, `sources`), `hosts_completed`, `vulnerabilities`, `start_time`, `end_time`, `sub_scans` (map of sub-scan key → `SubScan`).
- **SubScan:** `type`, `enabled`, `status`, `progress` (`completed`, `total`, `label`), `metadata` (optional). Used for both network and agent sub-scans.
---
## Related Documentation
- [ARCHITECTURE.sub-scans.md](./ARCHITECTURE.sub-scans.md) – Sub-scan lifecycle and progress aggregation
- [ARCHITECTURE.host-deduplication.md](./ARCHITECTURE.host-deduplication.md) – How hosts from multiple sources are merged and displayed
- [README.scanner.md](./README.scanner.md) – Scanner engine details (message format, strategies, ValKey usage)
---
**Last Updated:** 2025-02-07
**Version:** 1.0.0
**Maintainer:** Sirius Team
documentation/dev/apps/scanner/ARCHITECTURE.sub-scans.md
+185
View File
@@ -0,0 +1,185 @@
---
title: "Sub-Scans Architecture - Modular Scan Methods"
description: "Independent scan methods (network, agent) running in parallel, their data structure, lifecycle, progress aggregation, and cancellation."
template: "TEMPLATE.custom"
version: "1.0.0"
last_updated: "2025-02-07"
author: "Sirius Team"
tags: ["scanner", "sub-scan", "architecture", "progress", "cancellation"]
categories: ["architecture", "scanner"]
difficulty: "intermediate"
prerequisites: ["README.scanner.md"]
related_docs:
- "documentation/dev/apps/scanner/README.scanner.md"
- "documentation/dev/apps/scanner/ARCHITECTURE.scanner-data-flow.md"
- "documentation/dev/apps/scanner/ARCHITECTURE.host-deduplication.md"
dependencies: []
llm_context: "high"
search_keywords:
- "sub-scan"
- "sub_scans"
- "scan progress"
- "network agent parallel"
- "scan cancellation"
- "SubScan"
---
# Sub-Scans Architecture
The Sirius scanner uses a **sub-scan architecture**: each scan method (e.g. network, agent) is an independent **sub-scan** with its own status and progress. Sub-scans run in parallel and are tracked in a single `ScanResult` via a registry map.
## What Are Sub-Scans?
**Sub-scans** are independent scan methods that contribute to one logical “scan”:
- **network** – Traditional network scanning (e.g. Nmap, RustScan, Naabu) triggered via RabbitMQ and executed by app-scanner.
- **agent** – Agent-based scanning: templates run on remote hosts via app-agent over gRPC.
A scan can have zero, one, or both enabled. Each sub-scan has its own:
- Lifecycle state (e.g. dispatching → running → completed/failed/cancelled)
- Progress (e.g. completed/total hosts or agents)
- Optional metadata (e.g. agent list, per-agent status for agent sub-scan)
The overall scan state is the union of all sub-scan states and their results (hosts/vulnerabilities are merged at the scan level).
---
## Sub-Scan Data Structure
### Go (go-api / shared types)
```go
// SubScanProgress tracks completion progress for a sub-scan.
type SubScanProgress struct {
Completed int `json:"completed"`
Total int `json:"total"`
Label string `json:"label,omitempty"` // e.g. "hosts", "agents"
}
// SubScan represents a modular scanner contribution to a scan.
// Metadata is json.RawMessage so other scanners preserve it on read-modify-write.
type SubScan struct {
Type string `json:"type"` // "network", "agent"
Enabled bool `json:"enabled"`
Status string `json:"status"` // see Lifecycle states
Progress SubScanProgress `json:"progress"`
Metadata json.RawMessage `json:"metadata,omitempty"`
}
// ScanResult holds the top-level scan and a registry of sub-scans.
type ScanResult struct {
ID string
Status string
Targets []string
Hosts []HostEntry
HostsCompleted int
Vulnerabilities []VulnerabilitySummary
StartTime string
EndTime string
SubScans map[string]SubScan `json:"sub_scans,omitempty"`
}
```
### TypeScript (sirius-ui)
```ts
interface SubScanProgress {
completed: number;
total: number;
label?: string;
}
interface SubScan {
type: string;
enabled: boolean;
status: "pending" | "dispatching" | "running" | "completed" | "failed";
progress: SubScanProgress;
metadata?: Record<string, unknown>;
}
// ScanResult.sub_scans: Record<string, SubScan>
```
Agent-specific metadata in `SubScan.metadata` can include `mode`, `dispatched_agents`, `agent_statuses` (per-agent status, hosts/vulns found, etc.).
---
## Sub-Scan Registry Pattern
Sub-scans are stored in a **map keyed by scanner identifier**:
- `map[string]SubScan` in Go (`sub_scans` in JSON)
- `Record<string, SubScan>` in TypeScript
Common keys:
- `"network"` – Network scan (app-scanner).
- `"agent"` – Agent scan (app-agent).
Only enabled methods are present. When starting a scan, the UI (or API) builds this map (e.g. `subScans["network"]`, `subScans["agent"]`) and writes the initial `ScanResult` to ValKey. Consumers (app-scanner, go-api/agent path) update only their own key when merging back into `currentScan`, preserving other keys and `metadata` they don’t understand.
---
## Lifecycle States
Each sub-scan moves through a small set of states:
| State | Meaning |
|---------------|--------|
| `pending` | Not yet started (optional; may go straight to dispatching). |
| `dispatching` | Work is being dispatched (e.g. message to RabbitMQ, or gRPC dispatch to agents). |
| `running` | Actively running (targets being scanned, agents executing templates). |
| `completed` | Finished successfully. |
| `failed` | Finished with error. |
| `cancelled` | Stopped by user or system. |
The **overall scan** `status` is typically derived from sub-scans (e.g. “running” if any sub-scan is dispatching or running, “completed” when all are completed/failed/cancelled).
---
## Progress Aggregation
- **Per sub-scan:** `progress.completed`, `progress.total`, and optional `progress.label` (e.g. "hosts", "agents"). Each scanner updates its own sub-scan’s progress when writing to ValKey.
- **Overall scan:** The UI (or API) can compute an aggregate, for example:
- Total progress = sum of `completed` across sub-scans, divided by sum of `total` (or 0 if no total).
- Or: “running” if any sub-scan has `status === "running"` or `"dispatching"`, and overall completion when all sub-scans are in a terminal state.
The UI displays per–sub-scan progress (e.g. in `ScanStatus`) and can show an overall progress bar or status from these fields.
---
## Cancellation Handling
When the user cancels a scan:
1. UI/API calls the cancel endpoint (e.g. `POST /api/v1/scans/cancel` with optional `scan_id`).
2. go-api (or the component that owns the scan) sets the overall scan and/or sub-scans to a terminal state (e.g. `cancelled`) and writes the updated `ScanResult` back to ValKey.
3. **Network:** Cancellation can be implemented by app-scanner checking a shared “cancelled” flag (e.g. from ValKey or a separate key) or by receiving a cancel message, then stopping workers and updating its sub-scan status to `cancelled` in `currentScan`.
4. **Agent:** go-api (or agent coordinator) can signal agents to stop and then set `sub_scans.agent.status` to `cancelled` when updating ValKey.
So: **cancelling the scan** means ensuring all sub-scans are stopped and their status (and optionally the overall scan status) is set to `cancelled` in the same `currentScan` document.
---
## Summary
- **Sub-scans** = independent scan methods (network, agent) with their own status and progress.
- **Data structure:** `SubScan { type, enabled, status, progress, metadata }` in a `map[string]SubScan` (`sub_scans`).
- **Lifecycle:** dispatching → running → completed | failed | cancelled.
- **Progress:** per–sub-scan `completed`/`total`; overall progress can be aggregated from all sub-scans.
- **Cancellation:** cancel request stops all sub-scans and updates their status (and overall scan) in ValKey `currentScan`.
- **Registry:** Use a single `sub_scans` map; each scanner only updates its own key and preserves others’ `metadata`.
---
**Related Documentation**
- [ARCHITECTURE.scanner-data-flow.md](./ARCHITECTURE.scanner-data-flow.md) – How sub-scan results flow into ValKey and to the UI
- [ARCHITECTURE.host-deduplication.md](./ARCHITECTURE.host-deduplication.md) – How hosts from different sub-scans are merged by IP
---
**Last Updated:** 2025-02-07
**Version:** 1.0.0
**Maintainer:** Sirius Team
documentation/dev/apps/scanner/README.scanner.md
+2115
View File
File diff suppressed because it is too large Load Diff
documentation/dev/architecture/ADR.startup-secrets-model.md
+50
View File
@@ -0,0 +1,50 @@
---
title: "ADR: Startup and Secrets Model"
description: "Architectural decision record for installer-first startup and stateless infrastructure API key validation."
template: "TEMPLATE.reference"
llm_context: "high"
categories: ["architecture", "security", "operations"]
tags: ["adr", "startup", "secrets", "apikey", "docker", "installer"]
related_docs:
- "README.auth-surface-matrix.md"
- "README.api-key-operations.md"
- "README.docker-container-deployment.md"
---
# ADR: Startup and Secrets Model
## Status
Accepted
## Context
Sirius startup historically depended on manual environment setup and a bootstrap pattern that could create drift between runtime configuration and persistent key state. The platform also tolerated insecure defaults in startup and seeding paths.
## Decision
1. **Installer-first startup**: a first-run installer is the canonical setup path for generating and merging required secrets/config.
2. **Internal service key path**: `SIRIUS_API_KEY_FILE` (Docker secret file mount, preferred) with `SIRIUS_API_KEY` as a migration fallback — both resolve to the same installer-generated internal key for service-to-service root authentication.
3. **Dynamic key lifecycle in Valkey**: user-generated API keys continue to be managed in Valkey.
4. **Secure fail-fast runtime**: production startup and seeding flows must fail when required secrets are missing.
## Consequences
### Positive
- Deterministic behavior during restart and key rotation.
- Reduced operational complexity from bootstrap reconciliation state.
- Better first-time user experience with automated secret generation.
- Clear separation between infrastructure auth and user key lifecycle.
### Tradeoffs
- Stricter env requirements may break permissive legacy startup paths until migrated.
- CI/test and deployment scripts must be updated to provide required variables.
## Implementation Notes
- Use `docker-compose.installer.yaml` as the canonical installer entrypoint.
- Installer writes `secrets/sirius_api_key.txt` and sets `SIRIUS_API_KEY_FILE=/run/secrets/sirius_api_key` in `.env` alongside `SIRIUS_API_KEY` for compatibility.
- Ensure compose contracts explicitly require auth-critical variables (file and/or env).
- Update runbooks and deployment documentation in lockstep with runtime changes.
documentation/dev/architecture/ARCHITECTURE.nse-repository-management.md
+441
View File
@@ -0,0 +1,441 @@
---
title: "NSE Repository Management Architecture"
description: "Comprehensive architectural documentation for the NSE (Nmap Scripting Engine) repository management system, explaining how the scanner manages repositories dynamically and syncs to ValKey as the source of truth"
template: "TEMPLATE.architecture"
version: "1.0.0"
last_updated: "2025-10-25"
author: "Sirius Team"
tags:
[
"architecture",
"nse",
"repository-management",
"scanner",
"valkey",
"system-design",
]
categories: ["development", "architecture", "security"]
difficulty: "advanced"
prerequisites:
- "README.architecture.md"
- "README.scanner.md"
related_docs:
- "README.architecture.md"
- "../apps/scanner/README.scanner.md"
- "../README.development.md"
dependencies:
- "../../minor-projects/app-scanner/internal/nse/"
- "../../minor-projects/sirius-nse/"
llm_context: "high"
search_keywords:
[
"nse repository management",
"scanner architecture",
"valkey integration",
"dynamic repository",
"nmap scripts",
"runtime cloning",
]
---
# NSE Repository Management Architecture
## Purpose
This document defines the architectural design for NSE (Nmap Scripting Engine) repository management in Sirius Scanner. It clarifies component responsibilities, data flow, and integration boundaries to prevent architectural misunderstandings.
**Critical**: The scanner is the SOLE owner of NSE repository management. No other components should manage repositories via Dockerfile or file system mounts.
## When to Use
- **System Design**: When integrating with NSE scripts or scanner
- **Component Development**: When building UI/API features that consume NSE scripts
- **Code Review**: When reviewing scanner or UI/API integration code
- **Troubleshooting**: When diagnosing NSE script synchronization issues
- **Refactoring**: When making changes to repository management or script storage
## How to Use
### Quick Overview
1. **Start with System Overview** to understand the runtime repository management model
2. **Review Component Architecture** to understand scanner vs UI vs ValKey responsibilities
3. **Check Integration Points** to see how components access scripts
4. **Reference Design Decisions** for rationale on ValKey as source of truth
5. **Follow Anti-Patterns** to avoid common architectural violations
## What It Is
### System Overview
The NSE Repository Management system provides **dynamic, runtime management** of Nmap script repositories. The scanner clones and updates Git repositories at startup, synchronizes content to ValKey, and provides a read-only interface for UI and API components.
**Key Characteristics:**
- **Runtime Management**: Repositories cloned/updated at container startup
- **Multiple Repositories**: Supports arbitrary number of configured repositories
- **Single Source of Truth**: ValKey contains authoritative script data
- **Clean Boundaries**: Clear ownership prevents conflicts
### Architectural Principles
- **Separation of Concerns**: Scanner manages repos, ValKey stores data, UI/API read data
- **Single Source of Truth**: ValKey is the canonical source for all script data
- **Dynamic Configuration**: Add/remove repositories without rebuilding containers
- **Production Ready**: No host file system dependencies
### Component Architecture
#### Scanner (app-scanner)
- **Purpose**: Manages NSE script repositories and synchronizes to ValKey
- **Responsibilities**:
- Clone Git repositories at runtime
- Update repositories on startup
- Parse repository manifests
- Sync script metadata and content to ValKey
- **Dependencies**: ValKey (write), Git repositories (read)
- **Interfaces**: ValKey keys (`nse:*`)
- **Location**: `../minor-projects/app-scanner/internal/nse/`
**Key Components:**
```
internal/nse/
├── repo.go # RepoManager - Git clone/update operations
├── sync.go # SyncManager - ValKey synchronization
├── types.go # Data structures and ValKey keys
└── README.md # Implementation documentation
```
#### ValKey
- **Purpose**: Source of truth for NSE script data
- **Responsibilities**:
- Store repository manifest
- Store script manifest
- Store script content
- Provide read access to UI/API
- **Dependencies**: None (data store)
- **Interfaces**: Redis protocol
- **Keys**:
- `nse:repo-manifest` - Repository list configuration
- `nse:manifest` - Complete script manifest (612+ scripts)
- `nse:script:<id>` - Individual script content
#### UI (sirius-ui)
- **Purpose**: Display scripts and manage scan profiles
- **Responsibilities**:
- Read scripts from ValKey
- Display script information to users
- Allow script selection for scan profiles
- **Dependencies**: ValKey (read-only)
- **Interfaces**: tRPC procedures reading from ValKey
- **Location**: `sirius-ui/src/server/api/routers/store.ts`
**❌ NEVER**:
- Read from file system (`/sirius-nse/`)
- Clone or manage repositories
- Populate ValKey with script data
#### API (sirius-api)
- **Purpose**: Serve script data to external consumers
- **Responsibilities**:
- Read scripts from ValKey
- Provide REST endpoints for script data
- **Dependencies**: ValKey (read-only)
- **Interfaces**: REST API
**❌ NEVER**:
- Read from file system
- Manage repositories
### Data Flow
#### Repository Synchronization Flow
1. **Scanner Startup**: `main.go` creates `ScanManager`
2. **Initialize Managers**: Creates `RepoManager` and `SyncManager`
3. **Start Listening**: Calls `ListenForScans()`
4. **Sync Repositories**: `nseSync.Sync(ctx)` executes:
- Loads repository list from manifest or ValKey
- Clones repositories to `/opt/sirius/nse/<repo-name>`
- Reads local manifests from each repository
- Syncs to ValKey keys
5. **Ready**: Scripts available to UI/API via ValKey
#### UI Script Access Flow
1. **User Action**: Navigates to Scanner → Profiles
2. **UI Request**: Calls `getNseScripts` tRPC procedure
3. **ValKey Read**: Reads `nse:manifest` key
4. **Parse Manifest**: Extracts script list
5. **Display**: Shows 612 scripts to user
#### Scan Execution Flow
1. **User Action**: Creates scan with selected scripts
2. **Scanner Receives**: Gets scan request via RabbitMQ
3. **Script Selection**: Uses scripts from local repository
4. **Nmap Execution**: Runs Nmap with selected NSE scripts
5. **Results Storage**: Stores results in PostgreSQL
### Integration Points
#### Scanner → ValKey
- **Type**: Redis protocol (write)
- **Purpose**: Sync repository data to source of truth
- **Protocol**: Redis SET commands
- **Data Format**: JSON-serialized manifests and script content
- **Keys Written**:
- `nse:repo-manifest` - Repository configuration
- `nse:manifest` - Script manifest
- `nse:script:<id>` - Script content
#### UI → ValKey
- **Type**: Redis protocol (read-only)
- **Purpose**: Display scripts to users
- **Protocol**: Redis GET commands
- **Data Format**: JSON-serialized data
- **Keys Read**:
- `nse:manifest` - Script list
- `nse:script:<id>` - Script details
#### Scanner → Git Repositories
- **Type**: Git protocol
- **Purpose**: Clone and update script repositories
- **Protocol**: `git clone`, `git fetch`, `git reset --hard`
- **Data Format**: Git repository
- **Locations**: `/opt/sirius/nse/<repo-name>`
### Technology Stack
#### Backend
- **Go**: Scanner implementation language
- **Git**: Repository version control
- **ValKey/Redis**: Data storage and source of truth
#### Frontend
- **Next.js**: UI framework
- **tRPC**: Type-safe API layer
- **Valkey Client**: Redis client for script access
#### Infrastructure
- **Docker**: Container runtime
- **Docker Compose**: Multi-container orchestration
- **Volume Mounts**: Development-only, not for repositories
### Design Decisions
#### Decision 1: ValKey as Source of Truth
- **Context**: Multiple components need access to NSE script data
- **Decision**: Use ValKey as canonical source instead of shared file system
- **Rationale**:
- Clean separation of concerns
- No file system dependencies
- Easy to scale (ValKey cluster)
- Supports production deployments
- UI/API don't need repository knowledge
- **Consequences**:
- Scanner must sync on startup
- ValKey becomes critical dependency
- All components read from ValKey
#### Decision 2: Runtime Repository Management
- **Context**: Need to support multiple repositories and dynamic updates
- **Decision**: Clone repositories at runtime instead of Dockerfile
- **Rationale**:
- Add/remove repositories without rebuilds
- Support arbitrary number of repositories
- Update repositories on restart
- Clean container images
- **Consequences**:
- Startup time includes git clone
- Network required for cloning
- Scanner owns repository lifecycle
#### Decision 3: UI Read-Only Access
- **Context**: UI needs to display scripts for profile management
- **Decision**: UI only reads from ValKey, never manages repositories
- **Rationale**:
- Clear ownership (scanner manages, UI displays)
- No duplicate repository logic
- UI can't corrupt scanner state
- Simpler UI implementation
- **Consequences**:
- UI depends on scanner having run
- No offline UI script management
- Scanner must be healthy
### Security Considerations
- **Repository Trust**: Only clone from trusted Git repositories
- **ValKey Access**: ValKey should not be publicly accessible
- **Script Validation**: Scripts are trusted (from official Nmap repository)
- **File System Permissions**: Scanner runs with minimal permissions
### Performance Characteristics
- **Startup Time**: ~5-10 seconds for git clone + sync (612 scripts)
- **Sync Performance**: ~1 second to sync manifest to ValKey
- **UI Query Performance**: <100ms to read from ValKey
- **Memory Usage**: ~50MB for repository storage
### Scalability Considerations
- **Multiple Repositories**: Linear scaling with repository count
- **Script Count**: ValKey handles thousands of scripts efficiently
- **Concurrent Access**: ValKey supports multiple readers
- **Container Replicas**: Each scanner manages own repository copy
## Troubleshooting
### FAQ
**Q: Why doesn't the UI show any scripts?**
A: The scanner may not have started yet. The scanner syncs scripts to ValKey at startup via `ListenForScans()` → `Sync()`. Check scanner logs for sync messages.
**Q: Can I add a volume mount for sirius-nse to the UI?**
A: **No**. This violates the architecture. The UI should ONLY read from ValKey. Adding a volume mount creates unnecessary coupling and doesn't work in production.
**Q: How do I add a new NSE script repository?**
A: Update `app-scanner/internal/nse/manifest.json` or the `nse:repo-manifest` key in ValKey, then restart the scanner. It will automatically clone and sync the new repository.
### Command Reference
| Command | Purpose | Example | Notes |
| ------------------------------------------------------- | ------------------------ | ------------------------------------------------------------------------------- | ------------------------------- |
| `docker exec sirius-valkey valkey-cli GET nse:manifest` | Check script manifest | `docker exec sirius-valkey valkey-cli GET nse:manifest \| jq '.scripts.length'` | Returns 612 if synced correctly |
| `docker logs sirius-engine \| grep -i nse` | Check scanner sync logs | `docker logs sirius-engine \| grep -i "sync\|nse"` | Shows sync progress |
| `docker exec sirius-engine ls /opt/sirius/nse/` | List cloned repositories | `docker exec sirius-engine ls -la /opt/sirius/nse/sirius-nse` | Verifies repository exists |
| `docker restart sirius-engine` | Force repository re-sync | `docker restart sirius-engine` | Re-clones and syncs all repos |
| `docker exec sirius-engine rm -rf /opt/sirius/nse/` | Clear repository cache | `docker exec sirius-engine rm -rf /opt/sirius/nse/sirius-nse` | Forces fresh clone on restart |
### Common Issues
| Issue | Symptoms | Root Cause | Solution |
| ----------------------------- | ------------------------------------- | ---------------------------------------------- | ------------------------------------------------------------------------- |
| UI shows no scripts | Empty script list | Scanner hasn't synced or ValKey empty | Check scanner logs, restart scanner, verify ValKey keys |
| UI shows only 1 script | Fallback script displayed | UI tried to read file system instead of ValKey | Update UI code to read from ValKey (see `store.ts`) |
| Scanner fails to clone | "git clone failed" errors | Network issues or invalid repository URL | Check network, verify repository URL in manifest |
| Scripts not updating | Old scripts persist after repo update | Scanner cache not cleared | Restart scanner or manually delete `/opt/sirius/nse/` and restart |
| Production deployment failing | Container can't find scripts | Missing volume mount (wrong approach) | Remove volume mount, ensure scanner syncs to ValKey, UI reads from ValKey |
### Debugging Steps
1. **Check ValKey has scripts**:
```bash
docker exec sirius-valkey valkey-cli GET nse:manifest | jq '.scripts | length'
# Should return: 612
```
2. **Verify scanner synced**:
```bash
docker logs sirius-engine | grep -i "sync\|nse\|manifest"
# Should see: "Successfully initialized 612 NSE scripts"
```
3. **Test UI reading from ValKey**:
- Navigate to Scanner → Profiles
- Click "Initialize NSE Scripts"
- Should show: "Found 612 NSE scripts (synced by scanner)"
4. **Review scanner startup**:
```bash
docker logs sirius-engine --tail 100
# Check for sync errors during startup
```
## Anti-Patterns
### ❌ NEVER DO THESE
1. **Clone repositories in Dockerfile**:
```dockerfile
# ❌ WRONG - Breaks dynamic management
RUN git clone https://github.com/SiriusScan/sirius-nse.git /sirius-nse
```
2. **Add volume mount for UI to read repositories**:
```yaml
# ❌ WRONG - Violates architecture boundaries
sirius-ui:
volumes:
- ../minor-projects/sirius-nse:/sirius-nse
```
3. **Have UI read from file system**:
```typescript
// ❌ WRONG - UI should only read from ValKey
const manifestPath = "/sirius-nse/manifest.json";
const data = fs.readFileSync(manifestPath);
```
4. **Have UI populate ValKey**:
```typescript
// ❌ WRONG - Scanner owns ValKey population
await valkey.set("nse:manifest", JSON.stringify(manifest));
```
### ✅ ALWAYS DO THESE
1. **Let scanner manage repositories**:
```go
// ✅ CORRECT - Scanner manages at runtime
repoManager := nse.NewRepoManager("/opt/sirius/nse/sirius-nse", nse.NSERepoURL)
syncManager := nse.NewSyncManager(repoManager, kvStore)
```
2. **Have UI read from ValKey**:
```typescript
// ✅ CORRECT - UI reads from ValKey
const manifestData = await valkey.get("nse:manifest");
const manifest = JSON.parse(manifestData);
```
3. **Use ValKey as integration point**:
```
Scanner (File System) → ValKey → UI/API (Read-Only)
```
## Lessons Learned
### 2025-10-25 - UI Should Never Manage Repositories
**What happened**: UI was trying to read from file system (`/sirius-nse/manifest.json`) and populate ValKey, resulting in only 1 script being shown instead of 612.
**Root cause**: Misunderstanding of architecture - assumed UI should manage repositories.
**What we learned**: The scanner ALREADY populates ValKey with 612 scripts at startup. The UI should ONLY read from ValKey. This maintains clean separation of concerns and prevents duplication of repository management logic.
**Impact**:
- Removed unnecessary volume mount from UI
- Updated UI to read from ValKey only
- Added comprehensive architecture documentation
- Updated bot identity with clear integration boundaries
---
_This document follows the Sirius Documentation Standard. For questions about documentation structure, see [ABOUT.documentation.md](../ABOUT.documentation.md)._
documentation/dev/architecture/README.administrator.md
+41
View File
@@ -0,0 +1,41 @@
---
title: "Administrator Service Architecture"
description: "Architecture documentation for the Sirius administrator service"
template: "TEMPLATE.architecture"
llm_context: "medium"
categories: ["architecture", "services"]
tags: ["administrator", "admin", "management"]
related_docs:
- "README.system-monitor.md"
- "README.architecture.md"
---
# Administrator Service Architecture
This document describes the architecture and design of the Sirius administrator service.
## Overview
The administrator service provides administrative capabilities for the Sirius system, including system management, configuration, and monitoring controls.
## Components
- **Administrator Binary**: Standalone Go application for administrative operations
- **API Integration**: Connects to the main Sirius API for system management
- **Command Processing**: Handles administrative commands and operations
## Architecture
[Architecture details to be documented]
## Integration
The administrator service integrates with:
- Sirius API for system management
- System monitoring for operational insights
- Configuration management for system settings
## Deployment
[Deployment details to be documented]
documentation/dev/architecture/README.architecture-quick-reference.md
+279
View File
@@ -0,0 +1,279 @@
---
title: "SiriusScan Architecture Quick Reference"
description: "Concise architectural overview of SiriusScan - a microservices-based vulnerability scanner with Docker orchestration, providing essential context for LLM interactions."
template: "TEMPLATE.documentation-standard"
version: "1.0.0"
last_updated: "2025-01-03"
author: "AI Assistant"
tags:
[
"architecture",
"quick-reference",
"microservices",
"docker",
"vulnerability-scanner",
]
categories: ["architecture", "reference"]
difficulty: "beginner"
prerequisites: ["docker", "microservices"]
related_docs:
- "README.architecture.md"
- "README.development.md"
dependencies: []
llm_context: "high"
search_keywords:
[
"sirius architecture",
"vulnerability scanner",
"microservices",
"docker compose",
"quick reference",
]
---
# SiriusScan Architecture Quick Reference
> **📚 Full Architecture**: For comprehensive details, see [README.architecture.md](README.architecture.md)
## Purpose
This document provides a condensed architectural overview of SiriusScan for LLM context, focusing on core components, communication patterns, and essential setup information.
## What is SiriusScan
**SiriusScan** is an open-source, general-purpose vulnerability scanner built with a microservices architecture. It provides comprehensive vulnerability scanning capabilities through a web interface, with modular components for scanning, terminal access, and agent management.
## Core Architecture
### System Overview
```
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ sirius-ui │ │ sirius-api │ │sirius-engine│
│ (Next.js) │ │ (Go) │ │ (Go) │
│ Port 3000 │ │ Port 9001 │ │ Port 5174 │
└─────────────┘ └─────────────┘ └─────────────┘
│ │ │
└───────────────────┼───────────────────┘
│
┌──────────────────────┼──────────────────────┐
│ │ │
┌───▼────┐ ┌─────────────▼──┐ ┌─────────────▼──┐
│RabbitMQ│ │ PostgreSQL │ │ Valkey │
│ :5672 │ │ :5432 │ │ :6379 │
└────────┘ └────────────────┘ └────────────────┘
```
### Key Services
| Service | Technology | Port | Purpose |
| ------------------- | -------------------- | ---- | ---------------------- |
| **sirius-ui** | Next.js + TypeScript | 3000 | Web interface & BFF |
| **sirius-api** | Go + Fiber | 9001 | RESTful API backend |
| **sirius-engine** | Go | 5174 | Core processing engine |
| **sirius-rabbitmq** | RabbitMQ | 5672 | Message broker |
| **sirius-postgres** | PostgreSQL | 5432 | Relational database |
| **sirius-valkey** | Valkey (Redis) | 6379 | Key-value store |
## Component Details
### Frontend (sirius-ui)
- **Tech Stack**: Next.js, TypeScript, tRPC, Tailwind CSS, Shadcn/ui
- **Purpose**: User interface and Backend-for-Frontend (BFF)
- **Key Features**:
- Web-based vulnerability scanner interface
- Terminal component for command execution
- Agent management dashboard
- Real-time scan results display
### Backend API (sirius-api)
- **Tech Stack**: Go, Fiber web framework
- **Purpose**: RESTful API for data operations
- **Key Features**:
- Host management endpoints
- Vulnerability data handling
- Health check endpoints
- Database integration
### Processing Engine (sirius-engine)
- **Tech Stack**: Go with modular sub-applications
- **Purpose**: Core scanning and processing logic
- **Key Features**:
- Orchestrates vulnerability scans
- Manages deployed agents via gRPC
- Runs specialized sub-modules:
- `app-scanner`: Nmap-based vulnerability scanning
- `app-terminal`: PowerShell/command execution
- `app-agent`: Agent communication and management
### Data Layer
- **PostgreSQL**: Stores host information, vulnerabilities, scan results
- **Valkey**: Caching, session management, temporary data
- **RabbitMQ**: Asynchronous messaging between services
## Communication Patterns
### Frontend to Backend
- **tRPC**: Type-safe communication between Next.js frontend and BFF
- **REST**: Direct API calls to sirius-api for data operations
### Backend Services
- **RabbitMQ**: Asynchronous task queuing (UI → Engine)
- **gRPC**: Engine to agent communication
- **Database**: Direct connections to PostgreSQL and Valkey
## Docker Setup
### Quick Start
```bash
# Clone and start the system
git clone <repository>
cd Sirius
docker compose up -d
# Access services
# UI: http://localhost:3000
# API: http://localhost:9001
# Engine: http://localhost:5174
```
### Key Docker Files
- `docker-compose.yaml`: Main orchestration
- `sirius-ui/Dockerfile`: Next.js frontend
- `sirius-api/Dockerfile`: Go API service
- `sirius-engine/Dockerfile`: Core engine with sub-modules
## Project Structure
```
Sirius/
├── sirius-ui/ # Next.js frontend
│ ├── src/
│ │ ├── components/ # React components
│ │ ├── pages/ # Next.js pages & API routes
│ │ └── server/ # tRPC backend logic
│ └── prisma/ # Database schema
├── sirius-api/ # Go REST API
│ ├── handlers/ # HTTP handlers
│ └── routes/ # API routes
├── sirius-engine/ # Core processing engine
│ └── apps/ # Sub-modules
│ ├── app-scanner/
│ ├── app-terminal/
│ └── app-agent/
├── documentation/ # Project documentation
├── testing/ # Test suites
└── docker-compose.yaml
```
## Key Workflows
### Vulnerability Scanning
1. User initiates scan via UI
2. UI sends task to RabbitMQ
3. Engine processes scan using app-scanner
4. Results stored in PostgreSQL
5. UI displays results via real-time updates
### Agent Management
1. Agents connect to Engine via gRPC
2. Engine manages agent lifecycle
3. Commands executed on remote agents
4. Results streamed back to UI
### Terminal Access
1. User types commands in UI terminal
2. Commands routed through Engine
3. Executed via app-terminal (PowerShell)
4. Output streamed back to UI
## Development Context
### Backend Development
- **Location**: Inside `sirius-engine` container
- **Mount**: Local code mounted at `/app-agent` for dev mode
- **Commands**: Run via `docker exec sirius-engine <command>`
### Frontend Development
- **Hot Reload**: Enabled via volume mounts
- **Port**: 3000 (production)
- **Debugging**: Browser dev tools + React DevTools
### Testing
- **Location**: `testing/` directory
- **Commands**: `make test-all` for complete suite
- **Coverage**: Container health, integration, documentation
## Environment Variables
### Key Configuration
```bash
# Database
POSTGRES_HOST=sirius-postgres
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
POSTGRES_DB=sirius
# Messaging
RABBITMQ_URL=amqp://guest:guest@sirius-rabbitmq:5672/
# Services
SIRIUS_API_URL=http://sirius-api:9001
NEXTAUTH_URL=http://localhost:3000
```
## Troubleshooting
### Common Issues
- **Container startup**: Check health checks and dependencies
- **Database connection**: Verify PostgreSQL is healthy
- **Messaging**: Ensure RabbitMQ is running
- **Port conflicts**: Check if ports 3000, 9001, 5174 are available
### Debug Commands
```bash
# Check container status
docker compose ps
# View logs
docker compose logs -f sirius-ui
docker compose logs -f sirius-engine
# Health checks
curl http://localhost:3000/api/health
curl http://localhost:9001/api/v1/health
```
## LLM Context
This quick reference provides essential context for AI systems working with SiriusScan:
- **Architecture**: Microservices with Docker orchestration
- **Tech Stack**: Next.js frontend, Go backend, PostgreSQL/Valkey data layer
- **Communication**: tRPC, REST, RabbitMQ, gRPC
- **Purpose**: Vulnerability scanning with agent management
- **Development**: Container-based with hot reloading
- **Key Files**: docker-compose.yaml, service Dockerfiles, src/ directories
For detailed implementation, see the full [README.architecture.md](README.architecture.md) document.
---
_This quick reference provides essential architectural context for LLM interactions with the SiriusScan project._
documentation/dev/architecture/README.architecture.md
+2193
View File
File diff suppressed because it is too large Load Diff
documentation/dev/architecture/README.auth-surface-matrix.md
+130
View File
@@ -0,0 +1,130 @@
---
title: "Auth Surface Matrix"
description: "Authoritative authentication and operation policy matrix for all Sirius API surfaces."
template: "TEMPLATE.reference"
llm_context: "high"
categories: ["architecture", "security", "operations"]
tags: ["auth", "apikey", "trpc", "valkey", "rabbitmq", "agent"]
related_docs:
- "README.architecture.md"
- "README.development.md"
- "README.container-testing.md"
- "README.tasks.md"
---
# Auth Surface Matrix
This checklist is the canonical auth policy map for current Sirius architecture.
## Current Auth Model
- UI user auth: NextAuth session -> `protectedProcedure` in tRPC.
- Service auth: `X-API-Key` -> Go API middleware validation.
- Agent auth: gRPC token model for agent channel identity.
- Current platform model: single UI admin (no multi-user tenant isolation yet).
## Architecture Decision: Startup and Secret Strategy
### Decision Summary
- Startup onboarding is installer-first: generate/merge runtime config before compose startup.
- The infrastructure/root API key is validated statelessly from environment configuration.
- Valkey-backed key validation remains for dynamic, user-generated API keys only.
- Runtime auth and seed flows fail fast when required secrets are missing in production.
### Rationale
- Removes bootstrap drift between persistent key-value state and deployment configuration.
- Reduces first-run friction by automating secret generation and synchronization.
- Improves operational reliability for key rotation and service restarts.
- Aligns local and automated deployments with a single deterministic configuration model.
### Operational Implications
- Health endpoints remain public by explicit policy.
- Non-health API endpoints require `X-API-Key` and validate against either:
- environment root key (infrastructure path), or
- Valkey metadata (dynamic key path).
- Migration guidance is required for users moving from manual `.env` setup to installer flow.
## Policy Checklist
- [x] All sensitive tRPC procedures require `protectedProcedure`.
- [x] Go API middleware enforces API key for non-health routes.
- [x] UI -> Go API calls use shared authenticated client (`apiClient` / `apiFetch`).
- [x] Direct tRPC backends (Valkey/RabbitMQ) remain session-gated.
- [ ] Agent identity to HTTP/API actions is fully cryptographically bound.
- [x] Production key lifecycle is deterministic for root key validation and user-key lifecycle handling.
## tRPC Procedure Matrix (By Router)
Each row captures backend target and operation class.
| Router | Procedures | Backend Target | Operation Class | Required Auth |
|---|---|---|---|---|
| `apikeys` | `createKey`, `listKeys`, `revokeKey` | Go API `/api/v1/keys*` | Read/Write/Delete | Session + API key |
| `host` | `createHost`, `getHostList`, `updateHost`, `getHost`, `getHostStatistics`, `getEnvironmentSummary`, `getAllHosts`, `getSourceCoverage`, `getHostWithSources`, `getHostSoftwareInventory`, `getHostSoftwareStats`, `getHostSystemFingerprint`, `getEnhancedHostData`, `getEnvironmentSoftwareStats`, `getHostTemplateResults`, `getEnvironmentSoftwareInventory`, `getHostHistory`, `getVulnerabilityHistory` | Go API `/host*` | Read/Write | Session + API key |
| `vulnerability` | `addVulnerabilityToHost`, `getVulnerability`, `getAffectedHosts`, `getSoftwareDescription`, `getAllVulnerabilities` | Go API + external wiki lookup | Read/Write | Session (+ API key for Go API calls) |
| `scanner` | `getLatestScan`, `startScan`, `getScanStatus`, `cancelScan`, `forceStopScan`, `resetScanState` | mixed (local + Go API `/api/v1/scans*`) | Read/Write | Session (+ API key for Go API calls) |
| `templates` | `getTemplates`, `getTemplate`, `createTemplate`, `updateTemplate`, `deleteTemplate` | Go API `/templates*` via `apiFetch` | Read/Write/Delete | Session + API key |
| `scripts` | `getScripts`, `getScript`, `createScript`, `updateScript`, `deleteScript` | Go API `/scripts*` via `apiFetch` | Read/Write/Delete | Session + API key |
| `agentTemplates` | `getTemplates`, `getTemplate`, `uploadTemplate`, `validateTemplate`, `updateTemplate`, `deleteTemplate`, `testTemplate`, `deployTemplate`, `getAnalytics`, `getTemplateResults` | Go API `/api/agent-templates*` | Read/Write/Delete/Exec | Session + API key |
| `repositories` | `list`, `add`, `update`, `delete`, `sync`, `getSyncStatus` | Go API `/api/agent-templates/repositories*` | Read/Write/Delete | Session + API key |
| `events` | `getEvents`, `getEventStats`, `getRecentEvents`, `getEventsBySeverity` | Go API `/api/v1/events*` | Read | Session + API key |
| `statistics` | `createSnapshot`, `getVulnerabilityTrends`, `listSnapshots`, `getMostVulnerableHosts` | Go API `/api/v1/statistics*` | Read/Write | Session + API key |
| `logs` | `list`, `stats` | Go API `/api/v1/logs*` via `apiFetch` | Read | Session + internal API key (server-side only) |
| `store` | `initializeNseScripts`, `getValue`, `setValue`, `getNseScripts`, `getNseScript`, `updateNseScript`, `createNseScript`, `deleteNseScript`, `getNseRepositories`, `addNseRepository`, `removeNseRepository`, `initializeNseRepositories` | Direct Valkey | Read/Write/Delete | Session |
| `queue` | `sendMsg` | Direct RabbitMQ | Write/Exec | Session |
| `agent` | `listAgentsWithHosts`, `getAgentDetails`, `getTemplates`, `discoverTemplates`, `getTemplatesFromValKey`, `discoverTemplatesFromValKey`, `getTemplateContent`, `getScriptsFromValKey`, `discoverScriptsFromValKey`, `getScriptContent` | RabbitMQ + Go API + Valkey | Read/Exec | Session |
| `agentScan` | `dispatchAgentScan`, `getAgentScanStatus` | RabbitMQ + Valkey | Write/Read | Session |
| `terminal` | `executeCommand`, `getHistory`, `addHistoryEntry`, `deleteHistoryEntry`, `clearHistory` | RabbitMQ + Valkey | Exec/Read/Write/Delete | Session |
| `user` | `updateProfile`, `changePassword`, `getProfile` | Prisma DB | Read/Write | Session |
| `example` | `hello`, `getAll`, `getSecretMessage` | local/prisma demo | mixed | mixed (demo only) |
## Go API Endpoint Matrix (By Route Group)
All routes are under API key middleware except explicit health bypass.
| Group | Paths | Operation Class | Required Auth |
|---|---|---|---|
| health | `/health` | Read | Public (intentional) |
| system | `/api/v1/system/health`, `/api/v1/system/logs`, `/api/v1/system/resources` | Read | API key |
| admin | `/api/v1/admin/command` | Exec | API key |
| logs | `/api/v1/logs`, `/api/v1/logs/stats`, `/api/v1/logs/clear`, `/api/v1/logs/:logId` | Read/Write/Delete | Internal service API key (`SIRIUS_API_KEY_FILE` preferred, `SIRIUS_API_KEY` fallback). Browser UI uses tRPC `logs` router (session), not direct calls. |
| host | `/host/*`, `/vulnerability/:id/sources` | Read/Write/Delete | API key |
| vulnerability | `/vulnerability/:id`, `/vulnerability/`, `/vulnerability/delete` | Read/Write/Delete | API key |
| template | `/templates/*` | Read/Write/Delete | API key |
| agent template | `/api/agent-templates/*` | Read/Write/Delete/Exec | API key |
| repository | `/api/agent-templates/repositories/*` | Read/Write/Delete | API key |
| events | `/api/v1/events/*` | Read | API key |
| statistics | `/api/v1/statistics/*` | Read/Write | API key |
| scan control | `/api/v1/scans/*` | Read/Write | API key |
| api key mgmt | `/api/v1/keys/*` | Read/Write/Delete | API key |
| app passthrough | `/app/:appName` | Write/Exec | API key |
## Agent and NHI Boundary Matrix
| Boundary | Identity Mechanism | Current State | Required Control |
|---|---|---|---|
| Agent -> Engine (gRPC) | Agent token in stream messages | implemented | keep enforced |
| Engine/Agent -> Go API (HTTP) | Internal service key (file or env) | normalized via shared loader / logging client patch | keep `X-API-Key` on all internal HTTP clients |
| UI admin -> Agent dispatch | Session-gated tRPC + queue payloads | implemented but trust of client-supplied agent IDs needs hardening | validate agent existence/state on dispatch |
| Agent metadata ingestion | host source metadata | accepted from clients | validate schema and origin coupling |
## IDOR Opportunity Review (Current Stage)
Given single-admin model, current focus is authn + object selector hardening:
- selectors to validate consistently: `id`, `ip`, `vulnId`, `scanId`, `templateId`, `agentId`.
- malformed selectors should fail fast with `400`.
- non-existent resources should return `404`.
- unauthorized/unauthenticated should return `401` (and `403` when explicit deny model is later introduced).
## Verification Hooks
- Security harness must cover:
- Go API authn enforcement for all route families.
- tRPC session enforcement for all non-demo routers.
- direct Valkey/RabbitMQ procedure access controls.
- agent misuse cases (invalid/stale/mismatched agent IDs).
documentation/dev/architecture/README.cicd.md
+367
View File
@@ -0,0 +1,367 @@
---
title: "Sirius CI/CD Pipeline"
description: "Comprehensive guide to the Sirius Continuous Integration and Continuous Deployment pipeline, including workflows, testing strategies, and deployment processes."
template: "TEMPLATE.architecture"
version: "1.0.0"
last_updated: "2025-01-03"
author: "Development Team"
tags: ["cicd", "pipeline", "github-actions", "docker", "testing", "deployment"]
categories: ["architecture", "operations"]
difficulty: "intermediate"
prerequisites: ["docker", "github-actions", "git"]
related_docs:
- "README.docker-architecture.md"
- "README.developer-guide.md"
- "README.container-testing.md"
dependencies:
- ".github/workflows/ci.yml"
- "testing/container-testing/"
- "scripts/switch-env.sh"
llm_context: "high"
search_keywords:
[
"cicd",
"pipeline",
"github actions",
"docker testing",
"continuous integration",
"deployment",
"workflow"
]
---
# Sirius CI/CD Pipeline
## Overview
The Sirius CI/CD pipeline provides automated testing, building, and deployment of the multi-service Sirius application. The pipeline is designed to be efficient, reliable, and developer-friendly while maintaining high quality standards.
## Pipeline Architecture
### Workflow Structure
```
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Quick Checks │ │ Full Testing │ │ Deployment │
│ (Pre-commit) │───▶│ (CI/CD) │───▶│ (Production) │
└─────────────────┘ └─────────────────┘ └─────────────────┘
```
### Trigger Strategy
| Event | Quick Checks | Full Testing | Deployment |
|-------|-------------|--------------|------------|
| **Feature Branch Push** | ✅ Yes | ❌ No | ❌ No |
| **Pull Request** | ✅ Yes | ✅ Yes | ❌ No |
| **Main Branch Push** | ✅ Yes | ✅ Yes | ✅ Yes |
| **Hotfix Push** | ✅ Yes | ✅ Yes | ✅ Yes |
## Quick Checks (Pre-commit)
**Purpose**: Fast validation to catch obvious issues before commit
**Duration**: ~30 seconds
**What's Tested**:
- Docker Compose configuration validation
- Documentation linting
- Basic syntax checks
- Code formatting
**Commands**:
```bash
# Pre-commit validation
make build-all # Validate all Docker Compose configs
make lint-docs-quick # Quick documentation checks
```
## Full Testing (CI/CD)
**Purpose**: Comprehensive testing of all changes
**Duration**: ~5-10 minutes
**What's Tested**:
- Docker container builds (all services)
- Service health checks
- Integration testing
- Cross-service communication
- Production build validation
**Commands**:
```bash
# Full CI testing
make test-all # Complete test suite
make validate-all # Full validation with docs
```
## Environment Strategy
### Development Testing
**Trigger**: Feature branch pushes, local development
**Configuration**: Uses development Docker Compose setup
**Testing Level**: Quick validation only
**Purpose**: Catch basic issues early
### Integration Testing
**Trigger**: Pull requests, main branch pushes
**Configuration**: Uses production Docker Compose setup
**Testing Level**: Full test suite
**Purpose**: Validate complete integration before merge
### Production Deployment
**Trigger**: Main branch pushes, hotfixes
**Configuration**: Production-optimized builds
**Testing Level**: Full validation + deployment
**Purpose**: Deploy tested, validated code
## Service Detection and Building
### Change Detection
The CI pipeline intelligently detects which services have changed:
```yaml
# Example change detection
if changes in sirius-ui/ → build sirius-ui
if changes in sirius-api/ → build sirius-api
if changes in sirius-engine/ → build sirius-engine
if changes in docker-compose*.yaml → build all services
```
### Build Strategy
**Incremental Building**: Only build services that have changed
**Parallel Building**: Build multiple services simultaneously
**Caching**: Use Docker layer caching for faster builds
**Multi-Platform**: Build for both AMD64 and ARM64 architectures
## Testing Infrastructure
### Test Categories
1. **Build Tests**
- Individual container builds
- Multi-stage build validation
- Architecture compatibility
2. **Health Tests**
- Service startup validation
- Health endpoint checks
- Resource availability
3. **Integration Tests**
- Cross-service communication
- Database connectivity
- Message queue functionality
### Test Environment
**Configuration**: Production-like environment
**Database**: PostgreSQL with test data
**Cache**: Valkey (Redis-compatible)
**Message Queue**: RabbitMQ
**Network**: Isolated Docker network
## Docker Image Strategy
### Image Tagging
| Environment | Tag Pattern | Purpose |
|-------------|-------------|---------|
| **Development** | `sirius-*:dev` | Development builds |
| **Production** | `sirius-*:prod` | Production builds |
| **Base** | `sirius-*:latest` | Default builds |
| **CI** | `sirius-*:ci-{sha}` | CI-specific builds |
### Registry Management
**Primary Registry**: GitHub Container Registry (ghcr.io)
**Namespace**: siriusscan
**Image Naming**: `ghcr.io/siriusscan/sirius-{service}:{tag}`
## Workflow Jobs
### 1. Change Detection
**Purpose**: Determine which services need building
**Inputs**: Git diff, file changes
**Outputs**: Service build flags
**Duration**: ~30 seconds
### 2. Build and Push
**Purpose**: Build and push Docker images
**Inputs**: Change detection results
**Outputs**: Built and tagged images
**Duration**: ~3-5 minutes
### 3. Integration Testing
**Purpose**: Validate complete system integration
**Inputs**: Built images
**Outputs**: Test results
**Duration**: ~2-3 minutes
### 4. Deployment (Production)
**Purpose**: Deploy to production environment
**Inputs**: Tested images
**Outputs**: Deployed services
**Duration**: ~1-2 minutes
## Configuration Management
### Environment Variables
**Development**: Local `.env` files
**CI**: Generated `.env.ci.*` files
**Production**: Secure environment variables
### Docker Compose Files
**Base**: `docker-compose.yaml` - Core configuration
**Development**: `docker-compose.dev.yaml` - Development overrides
**Production**: `docker-compose.prod.yaml` - Production overrides
**CI**: `docker-compose.ci.yml` - Generated CI configuration
## Monitoring and Observability
### Build Monitoring
- Build success/failure rates
- Build duration trends
- Resource utilization
### Test Monitoring
- Test pass/fail rates
- Test duration trends
- Flaky test detection
### Deployment Monitoring
- Deployment success rates
- Service health post-deployment
- Performance metrics
## Troubleshooting
### Common Issues
| Issue | Symptoms | Solution |
|-------|----------|----------|
| **Build Failures** | Docker build errors | Check Dockerfile syntax, dependencies |
| **Test Failures** | Integration test errors | Check service logs, network connectivity |
| **Deployment Failures** | Service startup errors | Check environment variables, resource limits |
| **Cache Issues** | Stale builds | Clear Docker cache, rebuild with --no-cache |
### Debugging Commands
```bash
# Check CI logs
gh run list
gh run view {run-id}
# Local testing
make test-all
make logs
# Docker debugging
docker compose ps
docker compose logs {service}
```
## Best Practices
### For Developers
1. **Use feature branches** for all development
2. **Test locally** before pushing
3. **Keep commits focused** and atomic
4. **Write descriptive commit messages**
5. **Update documentation** when changing CI
### For CI/CD
1. **Keep builds fast** with proper caching
2. **Use conditional execution** based on changes
3. **Monitor build metrics** regularly
4. **Update dependencies** regularly
5. **Test CI changes** in feature branches
## Migration from Legacy CI
### What Changed
- **Simplified structure**: Removed complex submodule handling
- **Modern testing**: Integrated with current testing system
- **Better caching**: Improved Docker layer caching
- **Cleaner workflows**: Streamlined job structure
### Migration Steps
1. **Update workflows** to use new Docker Compose structure
2. **Integrate testing system** with CI pipeline
3. **Update image tagging** strategy
4. **Modernize deployment** process
5. **Update documentation** to reflect changes
## Future Enhancements
### Planned Improvements
- **Parallel testing** across multiple environments
- **Advanced caching** strategies
- **Performance testing** integration
- **Security scanning** automation
- **Rollback capabilities** for failed deployments
### Monitoring Enhancements
- **Real-time notifications** for build failures
- **Performance dashboards** for CI metrics
- **Automated rollback** triggers
- **Cost optimization** recommendations
---
_This CI/CD documentation follows the Sirius Documentation Standard. For questions about the pipeline, see [README.developer-guide.md](README.developer-guide.md)._
documentation/dev/architecture/README.docker-architecture.md
+895
View File
@@ -0,0 +1,895 @@
---
title: "Docker Architecture"
description: "Comprehensive guide to Sirius Docker setup, including all Docker Compose files, Dockerfiles, service architecture, and operational considerations"
template: "TEMPLATE.architecture"
version: "1.0.0"
last_updated: "2025-01-03"
author: "Development Team"
tags: ["docker", "architecture", "containers", "orchestration", "deployment"]
categories: ["architecture", "infrastructure", "deployment"]
difficulty: "intermediate"
prerequisites: ["docker", "docker-compose", "containerization"]
related_docs:
- "README.container-testing.md"
- "README.development.md"
dependencies:
- "docker-compose.yaml"
- "docker-compose.dev.yaml"
- "docker-compose.prod.yaml"
llm_context: "high"
search_keywords:
[
"docker",
"compose",
"containers",
"services",
"architecture",
"deployment",
"production",
"development",
]
---
# Docker Architecture
## Purpose
This document provides a comprehensive guide to the Sirius Docker architecture, including all Docker Compose configurations, service definitions, build processes, and operational considerations. It serves as the definitive reference for understanding, deploying, and maintaining the containerized Sirius application.
## When to Use
- **Before deploying Sirius** - Understand the complete container architecture
- **When troubleshooting Docker issues** - Reference service configurations and dependencies
- **During development setup** - Choose appropriate Docker Compose configuration
- **For production deployment** - Understand production-specific optimizations
- **When adding new services** - Follow established patterns and conventions
- **For capacity planning** - Understand resource requirements and scaling considerations
## How to Use
1. **Start with the overview** - Understand the overall architecture and service relationships
2. **Review Docker Compose files** - Understand the different environment configurations
3. **Examine Dockerfiles** - Understand build processes and multi-stage builds
4. **Check service dependencies** - Understand startup order and health checks
5. **Review operational considerations** - Understand monitoring, logging, and maintenance
6. **Reference troubleshooting** - Use the troubleshooting section for common issues
## What This Architecture Is
### Core Philosophy
The Sirius Docker architecture follows a **microservices-oriented containerization strategy** where each major component runs in its own container, with clear separation of concerns and well-defined interfaces. The architecture prioritizes:
- **Service isolation** - Each component runs independently with defined boundaries
- **Environment consistency** - Same container images work across development, staging, and production
- **Scalability** - Services can be scaled independently based on demand
- **Maintainability** - Clear separation makes debugging and updates easier
- **Resource efficiency** - Multi-stage builds and optimized base images minimize resource usage
### Architecture Overview
```
┌─────────────────────────────────────────────────────────────────┐
│ Sirius Docker Stack │
├─────────────────────────────────────────────────────────────────┤
│ Frontend Layer │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ sirius-ui │ │ sirius-api │ │
│ │ (Next.js) │◄──►│ (Go REST) │ │
│ │ Port: 3000 │ │ Port: 9001 │ │
│ └─────────────────┘ └─────────────────┘ │
│ │ │ │
│ │ │ │
│ Processing Layer │ │
│ ┌─────────────────┐ │ │
│ │ sirius-engine │◄─────────────┘ │
│ │ (Multi-service) │ │
│ │ Ports: 5174, │ │
│ │ 50051 │ │
│ └─────────────────┘ │
│ │ │
│ Data Layer │
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
│ │ sirius-postgres │ │ sirius-valkey │ │sirius-rabbitmq │ │
│ │ (PostgreSQL) │ │ (Redis Cache) │ │ (Message Queue) │ │
│ │ Port: 5432 │ │ Port: 6379 │ │ Ports: 5672, │ │
│ └─────────────────┘ └─────────────────┘ │ 15672 │ │
│ └─────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
```
## Service Architecture
### Frontend Services
#### sirius-ui (Next.js Frontend)
- **Purpose**: User interface and web application
- **Technology**: Next.js 14 with React, TypeScript, Tailwind CSS
- **Ports**: 3000 (HTTP), 3001 (Development)
- **Dependencies**: sirius-postgres (production), SQLite (development)
- **Health Check**: `http://localhost:3000/api/health`
- **Resource Limits**: 1GB RAM, 0.5 CPU
- **Build Stages**: development, production
**Key Features**:
- Multi-stage Dockerfile with optimized production build
- Hot reloading in development mode
- Volume mounts for source code in development
- Environment-specific database configuration (SQLite dev, PostgreSQL prod)
#### sirius-api (Go REST API)
- **Purpose**: RESTful API server and business logic
- **Technology**: Go 1.23 with Gin framework
- **Ports**: 9001 (HTTP)
- **Dependencies**: sirius-postgres, sirius-valkey, sirius-rabbitmq
- **Health Check**: `http://localhost:9001/api/v1/health`
- **Resource Limits**: 512MB RAM, 0.5 CPU
- **Build Stages**: development, runner
**Key Features**:
- Multi-stage build with optimized runner stage
- Volume mounts for development with live reloading
- Database connection pooling and health checks
- Message queue integration for async processing
### Processing Services
#### sirius-engine (Multi-Service Container)
- **Purpose**: Core processing engine with multiple integrated services
- **Technology**: Go 1.23 with integrated submodules
- **Ports**: 5174 (HTTP), 50051 (gRPC)
- **Dependencies**: sirius-rabbitmq, sirius-postgres, sirius-valkey
- **Health Check**: `http://localhost:5174/health`
- **Resource Limits**: 1GB RAM, 1.0 CPU
- **Build Stages**: development, runtime
**Integrated Services**:
- **Agent System**: Manages scanning agents and their lifecycle
- **Scanner Integration**: Integrates with nmap, rustscan, and custom scanners
- **Terminal Management**: Provides secure terminal access for agents
- **gRPC Server**: Handles agent communication on port 50051
**Key Features**:
- Multi-stage build with comprehensive dependency management
- Submodule integration for related projects
- Volume mounts for development with live code updates
- Resource-intensive processing capabilities
### Data Services
#### sirius-postgres (PostgreSQL Database)
- **Purpose**: Primary data storage and persistence
- **Technology**: PostgreSQL 15 Alpine
- **Ports**: 5432 (PostgreSQL)
- **Dependencies**: None (foundation service)
- **Health Check**: `pg_isready -U postgres`
- **Resource Limits**: 1GB RAM, 0.5 CPU
- **Data Persistence**: `postgres_data` volume
**Key Features**:
- Production-optimized configuration in prod mode
- Connection pooling and performance tuning
- Automated health checks and restart policies
- Persistent data storage with volume mounts
#### sirius-valkey (Redis Cache)
- **Purpose**: Caching and session storage
- **Technology**: Valkey (Redis-compatible)
- **Ports**: 6379 (Redis)
- **Dependencies**: None (foundation service)
- **Health Check**: `redis-cli ping`
- **Resource Limits**: 256MB RAM, 0.25 CPU
- **Data Persistence**: `valkey_data` volume
**Key Features**:
- Lightweight and fast caching layer
- Session storage for authentication
- Temporary data storage for processing
- Memory-optimized configuration
#### sirius-rabbitmq (Message Queue)
- **Purpose**: Asynchronous message processing and service communication
- **Technology**: RabbitMQ 3.7.3 with Management UI
- **Ports**: 5672 (AMQP), 15672 (Management UI)
- **Dependencies**: None (foundation service)
- **Health Check**: `rabbitmqctl status`
- **Resource Limits**: 512MB RAM, 0.5 CPU
- **Data Persistence**: `rabbitmq_data` volume
**Key Features**:
- Reliable message queuing for async processing
- Management UI for monitoring and debugging
- Dead letter queues and message routing
- Production authentication in prod mode
## Docker Compose Configurations
### Base Configuration (`docker-compose.yaml`)
The base configuration provides the core service definitions and is used as the foundation for all environments.
**Key Characteristics**:
- **Service Definitions**: All 6 services with production-ready defaults
- **Resource Limits**: Defined memory and CPU limits for each service
- **Health Checks**: Comprehensive health monitoring for all services
- **Dependencies**: Proper service startup order with health check conditions
- **Networking**: Custom bridge network named `sirius`
- **Volumes**: Persistent data storage for databases and message queue
**Usage**:
```bash
# Start all services with base configuration
docker compose up -d
# Stop all services
docker compose down
# View service status
docker compose ps
```
### Development Configuration (`docker-compose.dev.yaml`)
The development configuration overrides the base configuration to enable development features.
**Key Overrides**:
- **Volume Mounts**: Source code mounted for live reloading
- **Build Targets**: Uses development stages for all services
- **Environment Variables**: Development-specific settings
- **Database Configuration**: SQLite for UI, PostgreSQL for API
- **Logging**: Debug-level logging enabled
- **Ports**: Additional development ports exposed
**Key Features**:
- **Hot Reloading**: Source code changes reflected immediately
- **Debug Mode**: Enhanced logging and error reporting
- **Development Dependencies**: All dev dependencies included
- **Volume Persistence**: Node modules preserved to avoid architecture conflicts
**Usage**:
```bash
# Start development environment
docker compose -f docker-compose.yaml -f docker-compose.dev.yaml up -d
# Stop development environment
docker compose -f docker-compose.yaml -f docker-compose.dev.yaml down
```
### Production Configuration (`docker-compose.prod.yaml`)
The production configuration optimizes the base configuration for production deployment.
**Key Overrides**:
- **Build Targets**: Uses production/runner stages for all services
- **Environment Variables**: Production-specific settings
- **Resource Optimization**: Enhanced PostgreSQL and RabbitMQ configuration
- **Security**: Production authentication and secrets
- **Performance**: Optimized database and cache settings
**Key Features**:
- **Optimized Builds**: Multi-stage builds with minimal production images
- **Security Hardening**: Production authentication and secret management
- **Performance Tuning**: Database and cache optimization
- **Monitoring**: Enhanced health checks and resource monitoring
**Usage**:
```bash
# Start production environment
docker compose -f docker-compose.yaml -f docker-compose.prod.yaml up -d
# Stop production environment
docker compose -f docker-compose.yaml -f docker-compose.prod.yaml down
```
## Dockerfile Architecture
### Multi-Stage Build Strategy
All services use multi-stage Dockerfiles to optimize image size and build efficiency.
#### Image Tagging Strategy
To prevent Docker cache conflicts when switching between environments, each environment uses distinct image tags:
- **Development**: `sirius-sirius-ui:dev` - Built with development stage
- **Production**: `sirius-sirius-ui:prod` - Built with production stage
- **Base**: `sirius-sirius-ui:latest` - Built with default stage
This ensures that switching environments always uses the correct build target and prevents cached development images from being used in production mode.
#### sirius-ui Dockerfile
**Stages**:
1. **base**: Common dependencies and package installation
2. **development**: Full development environment with hot reloading
3. **production**: Optimized production build
**Key Features**:
- **Bun to npm conversion**: Automatic conversion for compatibility
- **Architecture support**: Multi-architecture builds
- **Optimized layers**: Cached dependency installation
- **Security**: Non-root user execution
#### sirius-api Dockerfile
**Stages**:
1. **development**: Development environment with volume mounts
2. **runner**: Production-optimized runtime
**Key Features**:
- **Go modules**: Efficient dependency management
- **Security**: Non-root user execution
- **Optimization**: Minimal production image
- **Health checks**: Built-in health monitoring
#### sirius-engine Dockerfile
**Stages**:
1. **builder**: Comprehensive build environment with all dependencies
2. **development**: Development environment with volume mounts
3. **runtime**: Production-optimized runtime
**Key Features**:
- **Submodule integration**: Automated cloning and building of related projects
- **Dependency management**: Comprehensive system and Go dependencies
- **Multi-service support**: Integrated agent, scanner, and terminal services
- **Resource optimization**: Efficient runtime with minimal dependencies
## Service Dependencies and Startup
### Dependency Chain
```
sirius-postgres (foundation)
├── sirius-api (depends on postgres health)
└── sirius-ui (depends on postgres health in production)
sirius-rabbitmq (foundation)
├── sirius-api (depends on rabbitmq health)
└── sirius-engine (depends on rabbitmq health)
sirius-valkey (foundation)
└── sirius-api (depends on valkey)
sirius-api (middleware)
└── sirius-ui (depends on api for data)
```
### Startup Sequence
1. **Foundation Services** (start first):
- sirius-postgres
- sirius-valkey
- sirius-rabbitmq
2. **Application Services** (start after foundation health checks):
- sirius-api
- sirius-engine
3. **Frontend Services** (start after application services):
- sirius-ui
### Health Check Strategy
- **Database Services**: Use native health check commands
- **Application Services**: HTTP health endpoints
- **Dependency Management**: Services wait for dependencies to be healthy
- **Retry Logic**: Configurable retry intervals and timeouts
## Resource Management
### Memory Allocation
| Service | Development | Production | Notes |
| --------------- | ----------- | ---------- | -------------------------- |
| sirius-ui | 512MB | 1GB | Frontend with build tools |
| sirius-api | 256MB | 512MB | REST API with caching |
| sirius-engine | 512MB | 1GB | Multi-service processing |
| sirius-postgres | 512MB | 1GB | Database with optimization |
| sirius-valkey | 128MB | 256MB | Lightweight cache |
| sirius-rabbitmq | 256MB | 512MB | Message queue with UI |
### CPU Allocation
| Service | Development | Production | Notes |
| --------------- | ----------- | ---------- | -------------------- |
| sirius-ui | 0.25 | 0.5 | Frontend processing |
| sirius-api | 0.25 | 0.5 | API request handling |
| sirius-engine | 0.5 | 1.0 | Intensive processing |
| sirius-postgres | 0.25 | 0.5 | Database operations |
| sirius-valkey | 0.1 | 0.25 | Cache operations |
| sirius-rabbitmq | 0.25 | 0.5 | Message processing |
## Networking Architecture
### Network Configuration
- **Network Name**: `sirius`
- **Driver**: Bridge
- **Scope**: Local
- **IPAM**: Default Docker IPAM
### Port Mapping
| Service | Internal Port | External Port | Protocol | Purpose |
| --------------- | ------------- | ------------- | -------- | ------------------- |
| sirius-ui | 3000 | 3000 | HTTP | Web interface |
| sirius-ui | 3001 | 3001 | HTTP | Development server |
| sirius-api | 9001 | 9001 | HTTP | REST API |
| sirius-engine | 5174 | 5174 | HTTP | Engine interface |
| sirius-engine | 50051 | 50051 | gRPC | Agent communication |
| sirius-postgres | 5432 | 5432 | TCP | Database |
| sirius-valkey | 6379 | 6379 | TCP | Cache |
| sirius-rabbitmq | 5672 | 5672 | AMQP | Message queue |
| sirius-rabbitmq | 15672 | 15672 | HTTP | Management UI |
### Service Communication
- **Internal Communication**: Services communicate using container names
- **External Access**: Ports exposed for external access
- **Health Checks**: Internal health check endpoints
- **Load Balancing**: Ready for external load balancer integration
## Data Persistence
### Volume Strategy
- **Database Data**: Persistent volumes for PostgreSQL
- **Cache Data**: Persistent volumes for Valkey
- **Message Queue Data**: Persistent volumes for RabbitMQ
- **Application Data**: Volume mounts for development
### Volume Configuration
```yaml
volumes:
postgres_data:
driver: local
valkey_data:
driver: local
rabbitmq_data:
driver: local
node_modules: # Development only
driver: local
```
### Backup Strategy
- **Database Backups**: Regular PostgreSQL dumps
- **Volume Backups**: Docker volume backup utilities
- **Configuration Backups**: Docker Compose file versioning
- **Disaster Recovery**: Multi-environment deployment strategy
## Environment Management
### Environment Variables
#### Required Variables
| Variable | Purpose | Default | Required |
| ----------------- | --------------------- | --------------------- | ---------- |
| POSTGRES_USER | Database user | postgres | No |
| POSTGRES_PASSWORD | Database password | postgres | No |
| POSTGRES_DB | Database name | sirius | No |
| NEXTAUTH_SECRET | Authentication secret | change-this-secret | Yes (prod) |
| NEXTAUTH_URL | Authentication URL | http://localhost:3000 | Yes (prod) |
#### Optional Variables
| Variable | Purpose | Default | Environment |
| ---------------- | ---------------- | ---------- | ----------- |
| NODE_ENV | Node environment | production | All |
| GO_ENV | Go environment | production | All |
| LOG_LEVEL | Logging level | info | All |
| API_PORT | API port | 9001 | All |
| ENGINE_MAIN_PORT | Engine port | 5174 | All |
### Configuration Files
- **Docker Compose**: Environment-specific overrides
- **Environment Files**: `.env` files for variable management
- **Dockerfile Args**: Build-time configuration
- **Service Configs**: Application-specific configuration
## Monitoring and Observability
### Health Checks
- **HTTP Endpoints**: Application health endpoints
- **Database Checks**: Native database health commands
- **Service Dependencies**: Automatic dependency health monitoring
- **Resource Monitoring**: Memory and CPU usage tracking
### Logging Strategy
- **Container Logs**: Docker native logging
- **Application Logs**: Service-specific logging
- **Log Levels**: Configurable per service
- **Log Aggregation**: Ready for external log aggregation
### Metrics Collection
- **Service Metrics**: Built-in health check endpoints
- **Resource Metrics**: Docker stats and resource usage
- **Application Metrics**: Service-specific metrics
- **External Monitoring**: Ready for Prometheus/Grafana integration
## Security Considerations
### Container Security
- **Non-root Users**: All services run as non-root users
- **Resource Limits**: Memory and CPU limits prevent resource exhaustion
- **Network Isolation**: Custom network with controlled access
- **Image Security**: Regular base image updates
### Data Security
- **Encryption**: Database and cache encryption at rest
- **Authentication**: Production authentication mechanisms
- **Secrets Management**: Environment variable-based secrets
- **Network Security**: Controlled port exposure
### Production Security
- **Secret Rotation**: Regular secret updates
- **Access Control**: Limited external port exposure
- **Audit Logging**: Comprehensive logging for security events
- **Vulnerability Scanning**: Regular container vulnerability scans
## Troubleshooting
### Common Issues
#### Service Startup Failures
**Symptoms**: Services fail to start or restart repeatedly
**Causes**:
- Resource constraints
- Dependency health check failures
- Configuration errors
- Port conflicts
**Solutions**:
```bash
# Check service status
docker compose ps
# View service logs
docker compose logs [service-name]
# Check resource usage
docker stats
# Restart specific service
docker compose restart [service-name]
```
#### Database Connection Issues
**Symptoms**: API or UI cannot connect to database
**Causes**:
- Database not ready
- Wrong connection parameters
- Network connectivity issues
**Solutions**:
```bash
# Check database health
docker compose exec sirius-postgres pg_isready -U postgres
# Check database logs
docker compose logs sirius-postgres
# Test connection
docker compose exec sirius-api curl -f http://localhost:9001/health
```
#### Build Failures
**Symptoms**: Docker build fails or takes too long
**Causes**:
- Insufficient resources
- Network connectivity issues
- Dockerfile errors
- Dependency resolution problems
**Solutions**:
```bash
# Clean build cache
docker builder prune
# Build with verbose output
docker compose build --no-cache --progress=plain
# Check build context
docker compose config
```
#### Volume Issues
**Symptoms**: Data not persisting or permission errors
**Causes**:
- Volume mount issues
- Permission problems
- Disk space issues
**Solutions**:
```bash
# Check volume status
docker volume ls
# Inspect volume
docker volume inspect sirius_postgres_data
# Clean up unused volumes
docker volume prune
```
#### Docker Layer Caching Issues
**Symptoms**:
- Production services running development code (e.g., `next dev` instead of `next start`)
- Wrong build targets being used despite correct Docker Compose configuration
- Services behaving differently than expected based on environment configuration
- React hook errors or development-specific errors in production mode
**Causes**:
- Docker layer caching preventing proper build stage selection
- Cached development images being used instead of production builds
- Multi-stage Dockerfile not building correct target stage
- Docker Compose not forcing rebuild of cached images
**Root Cause Analysis**:
This issue occurs when Docker's layer caching system retains development-stage images and reuses them even when production configurations are specified. The problem manifests when:
1. Development images are built first and cached
2. Production builds reference the same base layers
3. Docker reuses cached development layers instead of building production stages
4. The final image contains development code despite being tagged as production
**Solutions**:
**Immediate Fix**:
```bash
# Stop all services
docker compose down
# Clean Docker system cache
docker system prune -f
# Rebuild with no cache
docker compose build --no-cache
# Start services
docker compose up -d
```
**Prevention Strategies**:
```bash
# Always use --no-cache for production builds
docker compose -f docker-compose.yaml -f docker-compose.prod.yaml build --no-cache
# Clean build cache before switching environments
docker builder prune
# Verify correct build target is being used
docker compose config | grep -A 5 "target:"
```
**Verification Steps**:
```bash
# Check what command is actually running in container
docker compose exec sirius-ui ps aux
# Verify build stage in container
docker compose exec sirius-ui cat /app/start-prod.sh
# Check container logs for correct startup script
docker compose logs sirius-ui | grep -E "(next dev|next start|npm run)"
# Verify environment variables
docker compose exec sirius-ui env | grep NODE_ENV
```
**Long-term Prevention**:
1. **Use distinct image tags** for different environments
2. **Implement build stage validation** in CI/CD pipelines
3. **Regular cache cleanup** in automated builds
4. **Monitor container startup logs** for correct command execution
5. **Use multi-stage builds with explicit stage selection**
**Example Multi-Stage Build Validation**:
```dockerfile
# Ensure production stage is explicitly selected
FROM node:18-alpine AS production
# ... production-specific setup
# Development stage should be clearly separated
FROM node:18-alpine AS development
# ... development-specific setup
```
**Docker Compose Target Verification**:
```yaml
# Ensure correct target is specified
services:
sirius-ui:
build:
context: ./sirius-ui
target: production # Explicitly specify production stage
```
### Debugging Commands
```bash
# View all service status
docker compose ps
# View service logs
docker compose logs -f [service-name]
# Execute commands in container
docker compose exec [service-name] [command]
# View resource usage
docker stats
# Check network connectivity
docker compose exec [service-name] ping [target-service]
# View Docker Compose configuration
docker compose config
# Restart all services
docker compose restart
# Stop and remove all containers
docker compose down -v
```
### Performance Optimization
#### Resource Tuning
- **Memory Limits**: Adjust based on actual usage
- **CPU Limits**: Scale based on processing needs
- **Volume Performance**: Use local volumes for better performance
- **Network Optimization**: Optimize service communication
#### Build Optimization
- **Multi-stage Builds**: Use appropriate build stages
- **Layer Caching**: Optimize Dockerfile layer ordering
- **Dependency Management**: Minimize dependency changes
- **Image Size**: Use minimal base images
## Best Practices
### Development Workflow
1. **Use Development Configuration**: Always use dev overrides for development
2. **Volume Mounts**: Use volume mounts for live code updates
3. **Health Checks**: Wait for services to be healthy before testing
4. **Resource Monitoring**: Monitor resource usage during development
5. **Log Analysis**: Use logs for debugging and optimization
6. **Cache Management**: Clean Docker cache when switching between environments
7. **Build Verification**: Always verify correct build targets are being used
### Production Deployment
1. **Environment Variables**: Use proper secret management
2. **Resource Planning**: Plan resources based on expected load
3. **Monitoring Setup**: Implement comprehensive monitoring
4. **Backup Strategy**: Implement regular backup procedures
5. **Security Hardening**: Apply production security measures
6. **Build Validation**: Always use `--no-cache` for production builds
7. **Environment Isolation**: Ensure clean separation between dev and prod builds
### Environment Switching
**New in v1.0.0**: Use the `scripts/switch-env.sh` script for seamless environment switching.
#### Quick Environment Switching
```bash
# Switch to development mode (hot reloading, volume mounts)
./scripts/switch-env.sh dev
# Switch to production mode (optimized builds, PostgreSQL)
./scripts/switch-env.sh prod
# Switch to base mode (standard configuration)
./scripts/switch-env.sh base
```
#### What the Script Does
1. **Stops all containers** to ensure clean state
2. **Removes old images** to prevent cache conflicts
3. **Builds with correct target** for the specified environment
4. **Starts all services** with appropriate configuration
5. **Shows status and URLs** for easy access
#### Manual Environment Switching
If you need to switch environments manually:
1. **Clean Transitions**: Always clean Docker cache when switching environments
2. **Build Verification**: Verify correct build targets before starting services
3. **Log Monitoring**: Check startup logs to ensure correct commands are running
4. **Cache Management**: Use `docker system prune` between environment switches
5. **Target Validation**: Explicitly specify build targets in Docker Compose files
### Maintenance
1. **Regular Updates**: Keep base images and dependencies updated
2. **Log Rotation**: Implement log rotation and cleanup
3. **Volume Management**: Monitor and clean up unused volumes
4. **Security Scanning**: Regular vulnerability scanning
5. **Performance Monitoring**: Continuous performance monitoring
## Future Considerations
### Scaling Strategy
- **Horizontal Scaling**: Ready for service replication
- **Load Balancing**: External load balancer integration
- **Database Scaling**: Read replicas and connection pooling
- **Cache Scaling**: Redis cluster configuration
### Advanced Features
- **Service Mesh**: Istio or similar service mesh integration
- **Container Orchestration**: Kubernetes deployment readiness
- **CI/CD Integration**: Automated build and deployment pipelines
- **Monitoring Integration**: Prometheus and Grafana integration
---
_This document follows the Sirius Documentation Standard. For questions about documentation structure, see [ABOUT.documentation.md](../ABOUT.documentation.md)._
documentation/dev/architecture/README.engine-component-pinning.md
+167
View File
@@ -0,0 +1,167 @@
---
title: "Engine Component Pinning"
description: "How sirius-engine pins each minor-project at build time, where pins live, who is allowed to change them, and how to bump a component without breaking CI."
template: "TEMPLATE.documentation-standard"
llm_context: "high"
categories: ["architecture", "deployment", "operations"]
tags: ["docker", "submodules", "pinning", "release-engineering", "ci-cd"]
related_docs:
- "README.docker-architecture.md"
- "README.cicd.md"
- "../../dev-notes/SHA-AUDIT-2026-04.md"
- "../README.development.md"
---
# Engine Component Pinning
`sirius-engine` is built from six external SiriusScan repositories that
are cloned and compiled inside the engine image. To make every build
reproducible, each repository is pinned to a single commit SHA (or, for
`go-api`, a tag). This document describes where those pins live, the
rules that govern them, and the workflow to bump one safely.
## Why pin
Without explicit pins:
- A `docker compose build` on Monday and the same command on Wednesday
could produce different binaries (silent supply-chain drift).
- A bug introduced in a minor-project's `main` branch lands in the next
engine image without any signal in the Sirius repo.
- Rollbacks become impossible because there is no record of *which*
upstream commit produced a given engine image.
The `check-pin-consistency.yml` GitHub Action enforces these rules
mechanically (see `.github/workflows/check-pin-consistency.yml`).
## Pinned components
| ARG | Repository | Used by | Notes |
| --- | --- | --- | --- |
| `GO_API_COMMIT_SHA` | `SiriusScan/go-api` | All Go services in the engine + the standalone `sirius-api` image | Pinned by **tag** (`vX.Y.Z`). Tag is created in the `go-api` repo first, then bumped here. |
| `APP_SCANNER_COMMIT_SHA` | `SiriusScan/app-scanner` | Scanner binary (`/scanner`) | Full SHA. CGO build, depends on `libpcap` and `pingpp`. |
| `APP_TERMINAL_COMMIT_SHA` | `SiriusScan/app-terminal` | Terminal binary (`/terminal`) | Full SHA. |
| `SIRIUS_NSE_COMMIT_SHA` | `SiriusScan/sirius-nse` | NSE script repo bundled with the scanner | Full SHA. |
| `APP_AGENT_COMMIT_SHA` | `SiriusScan/app-agent` | Agent server (`/app-agent-src/server`) | Full SHA. |
| `PINGPP_COMMIT_SHA` | `SiriusScan/pingpp` | Fingerprinting library linked into `app-scanner` | Full SHA. |
## Where pins live
There are exactly **two** authoritative pin surfaces. They must always
agree.
1. **`sirius-engine/Dockerfile` — `ARG ..._COMMIT_SHA` defaults**
The Dockerfile is the source of truth. Local `docker compose build`
uses these defaults. The block lives at the top of the build stage
and carries a `# Pin policy` comment.
2. **`.github/workflows/ci.yml` — `build-args` fallbacks**
CI passes `${{ env.X_COMMIT_SHA || '<fallback>' }}` for every pin in
the `build-engine` and `build-api` jobs. The `<fallback>` literals
must match the Dockerfile defaults exactly. The `env.*` vars are
only populated when CI is triggered by a `repository_dispatch` from
a minor-project's release workflow (see "Bumping a pin via dispatch"
below).
There is also a per-build override:
3. **`docker compose build --build-arg X_COMMIT_SHA=...`**
Local engineers can override any pin to test a hot fix, but this
override must never be committed.
## Rules
1. **No floating refs.** `main`, `master`, `HEAD`, branch names, or
relative refs like `HEAD~1` are forbidden. Every pin must be either
a full 40-character SHA or a semver tag.
2. **Dockerfile and CI must agree.** The
`check-pin-consistency.yml` workflow fails any PR that drifts.
3. **Tag preferred for `go-api`.** Other Sirius services consume
`go-api` as a Go module via `go.mod`. Bumping the tag in two places
(its own `go.mod` and ours) is easier with a tag than a SHA.
4. **Bumps are atomic.** A pin bump PR must update the Dockerfile, the
CI fallback, and the audit doc (current: `SHA-AUDIT-2026-04.md`,
future: a successor file) in one change set.
5. **Every pin bump references the upstream change.** PR description
must link to the upstream commit or tag and summarize behavior
changes that affect the engine.
## Workflows
### Bumping a pin manually
```bash
# 1. Choose the new SHA (full 40-char) or tag.
NEW_SHA=ca1ef2fb75d2c422675eb41a27517da6aa5cf842
# 2. Edit sirius-engine/Dockerfile ARG block.
# 3. Edit .github/workflows/ci.yml build-engine fallback.
# 4. Append a row to documentation/dev-notes/SHA-AUDIT-2026-04.md
# (or the current audit doc) describing what changed and why.
# 5. Verify locally.
docker compose -f docker-compose.yaml -f docker-compose.dev.yaml build --no-cache sirius-engine
# 6. Open a PR. The check-pin-consistency.yml job will fail
# if Dockerfile and CI disagree, or if you used a floating ref.
```
### Bumping a pin via repository dispatch
When a minor-project finishes its own release, its
`Notify Sirius` workflow fires a `repository_dispatch` event with:
```json
{
"event_type": "submodule-update",
"client_payload": {
"submodule": "SiriusScan/app-agent",
"commit_sha": "<40-char SHA>"
}
}
```
The Sirius `ci.yml` workflow:
1. Derives the env-var name from `submodule` (e.g. `app-agent` →
`APP_AGENT_COMMIT_SHA`).
2. Sets `env.APP_AGENT_COMMIT_SHA = <commit_sha>` for the run.
3. Builds the engine image with the new SHA via
`${{ env.APP_AGENT_COMMIT_SHA || '<fallback>' }}`.
The result is published as `:latest` and `:beta`. The
**fallback in CI is intentionally not updated by the dispatch** — it
stays at the Dockerfile default until a human opens a PR. This means a
dispatch-triggered build is reproducible only as long as the dispatched
SHA exists; if you want the new SHA to be the long-lived floor, follow
the manual workflow above to land it in the Dockerfile.
### Local hot-swap (no rebuild)
For a fast iteration loop without rebuilding the image, see
`docker-compose.dev.yaml` (bind mounts) and the per-service `.air.toml`
files (live reload). See `../README.development.md` for the full
dev-mode workflow.
## Common failure modes
| Symptom | Likely cause | Fix |
| --- | --- | --- |
| Dashboard shows behavior that the latest source does not implement | Engine image is stale; pin is behind upstream `main` | Bump the relevant pin (manual or dispatch) and rebuild |
| `docker compose build` produces different binaries on different machines | A pin is using a floating ref like `main` | Replace with a full SHA; the guardrail will block this in CI |
| CI green on PR but runtime explodes after merge | Dockerfile and CI fallbacks disagree, and the merged build used the CI fallback (which differs from local) | Re-run guardrail; align both surfaces |
| `app-scanner` build fails with a `sed: pattern not found` error | An old branch still has the inline sed block; that block was removed in this overhaul | Rebase onto current `main`; the patches are now real source in `app-scanner` |
## See also
- `documentation/dev-notes/SHA-AUDIT-2026-04.md` — full audit and
per-component decision rationale captured at the time of the
overhaul.
- `documentation/dev/architecture/README.docker-architecture.md` —
overall engine container architecture.
- `documentation/dev/architecture/README.cicd.md` — CI/CD pipeline
overview.
- `documentation/dev/README.development.md` — dev-mode workflow.
documentation/dev/architecture/README.go-api-sdk.md
+750
View File
@@ -0,0 +1,750 @@
---
title: "Go API SDK Architecture"
description: "Comprehensive guide to the Sirius Go API SDK architecture, design patterns, and integration"
template: "TEMPLATE.documentation-standard"
llm_context: "high"
categories: ["architecture", "sdk", "backend"]
tags: ["go", "api", "sdk", "database", "orm", "gorm"]
related_docs:
- "README.architecture.md"
- "../operations/README.sdk-releases.md"
- "../../../minor-projects/go-api/README.md"
search_keywords: ["go-api", "sdk", "gorm", "database", "models", "postgres", "rabbitmq", "valkey"]
---
# Go API SDK Architecture
## Overview
The **Sirius Go API SDK** (`github.com/SiriusScan/go-api`) is a shared library that provides:
- **Core data models** (Host, Port, Vulnerability, Service)
- **Database operations** (PostgreSQL via GORM)
- **Message queue integration** (RabbitMQ)
- **Key-value store** (ValKey/Redis)
- **NVD API integration** (vulnerability data enrichment)
**Design Principles:**
- **Shared Types**: Single source of truth for data structures
- **Abstraction**: Database and queue operations hidden behind clean APIs
- **Source Attribution**: Track which tool found each piece of data
- **Flexibility**: Works across multiple projects (scanner, API server, agents)
## Package Structure
```
go-api/
├── sirius/ # Core functionality
│ ├── sirius.go # Core types (Host, Port, Vulnerability, Service)
│ ├── postgres/ # Database operations
│ │ ├── connection.go # DB initialization and connection management
│ │ ├── host_operations.go # Host CRUD operations
│ │ ├── vulnerability_operations.go
│ │ └── models/ # Database models (GORM)
│ │ ├── host.go # Host, Port, Service models
│ │ ├── vulnerability.go # Vulnerability, CVE models
│ │ └── scan_source.go # Source attribution models
│ ├── host/ # Host management SDK
│ │ ├── host.go # Basic host operations
│ │ └── source_aware.go # Source-attributed operations
│ ├── queue/ # RabbitMQ integration
│ │ └── queue.go # Publish/subscribe operations
│ ├── store/ # Key-value store (ValKey/Redis)
│ │ └── store.go # KV operations
│ └── logging/ # Logging infrastructure
│ ├── client.go # Logging client
│ └── api/ # Logging API server
├── nvd/ # NVD API integration
│ └── nvd.go # CVE data fetching
├── migrations/ # Database schema migrations
└── docs/ # SDK documentation
```
## Core Data Models
### Host Model
**File:** `sirius/sirius.go`
```go
type Host struct {
HID string // Host identifier
OS string // Operating system
OSVersion string // OS version
IP string // IP address
Hostname string // DNS hostname
Ports []Port // Open ports
Services []Service // Running services
Vulnerabilities []Vulnerability // Found vulnerabilities
CPE []string // Common Platform Enumeration
Agent *SiriusAgent // Optional agent info
}
```
### Port Model
**File:** `sirius/sirius.go` (Core), `sirius/postgres/models/host.go` (Database)
```go
// Core type (used by applications)
type Port struct {
Number int `json:"number"` // Port number (22, 80, 443, etc.)
Protocol string `json:"protocol"` // tcp, udp
State string `json:"state"` // open, closed, filtered
}
// Database model (used by GORM)
type Port struct {
gorm.Model // ID, CreatedAt, UpdatedAt, DeletedAt
Number int // Port number
Protocol string // Protocol
State string // State
Hosts []Host `gorm:"many2many:host_ports"`
HostPorts []HostPort `gorm:"foreignKey:PortID"`
}
```
**Key Design Decision:** Port numbers are stored in the `Number` field, NOT the auto-increment `ID` field. This allows the same port (e.g., port 22) to be associated with multiple hosts without conflicts.
### Vulnerability Model
**File:** `sirius/sirius.go` (Core), `sirius/postgres/models/vulnerability.go` (Database)
```go
// Core type
type Vulnerability struct {
VID string `json:"vid"` // CVE ID (CVE-2017-0144)
Title string `json:"title"` // Vulnerability title
Description string `json:"description"` // Description
RiskScore float64 `json:"risk_score"` // CVSS score (0-10)
}
// Database model
type Vulnerability struct {
gorm.Model
VID string `gorm:"column:v_id;uniqueIndex"` // Unique CVE ID
Description string
Title string
RiskScore float64
Hosts []Host `gorm:"many2many:host_vulnerabilities"`
HostVulnerabilities []HostVulnerability `gorm:"foreignKey:VulnerabilityID"`
}
```
## Source Attribution System
### Purpose
Track **which tool** found **which data** on **which host** at **what time**. This enables:
- **Audit trails**: Know where data came from
- **Tool comparison**: Compare effectiveness of different scanners
- **Historical tracking**: See when vulnerabilities first appeared
- **Confidence scoring**: Weight findings by source reliability
### Junction Tables with Source
**HostPort Junction Table:**
```go
type HostPort struct {
HostID uint `json:"host_id" gorm:"primaryKey"`
PortID uint `json:"port_id" gorm:"primaryKey"`
Source string `json:"source" gorm:"primaryKey"` // nmap, rustscan, naabu
SourceVersion string `json:"source_version"` // Tool version
FirstSeen time.Time `json:"first_seen"` // First detection
LastSeen time.Time `json:"last_seen"` // Last confirmation
Status string `json:"status" gorm:"default:active"` // active, resolved
Notes string `json:"notes,omitempty"` // Config details
}
```
**Primary Key:** `(host_id, port_id, source)` allows:
- Same port on same host tracked by multiple tools
- Comparison of results between tools
- Historical view of port states
**HostVulnerability Junction Table:**
Similar structure with additional fields:
- `Confidence` (0.0-1.0): How confident is this finding?
- `Port` (optional): Specific port where vulnerability found
- `ServiceInfo`: Service details
### Using Source Attribution
**Example: Add host with source:**
```go
import (
"github.com/SiriusScan/go-api/sirius"
"github.com/SiriusScan/go-api/sirius/host"
"github.com/SiriusScan/go-api/sirius/postgres/models"
)
// Create host data
hostData := sirius.Host{
IP: "192.168.1.100",
Ports: []sirius.Port{
{Number: 22, Protocol: "tcp", State: "open"},
{Number: 80, Protocol: "tcp", State: "open"},
},
}
// Create source metadata
source := models.ScanSource{
Name: "nmap",
Version: "7.94",
Config: "ports:1-1000;template:quick;timing:T4",
}
// Submit with source attribution
err := host.AddHostWithSource(hostData, source)
```
## Database Operations
### Connection Management
**File:** `sirius/postgres/connection.go`
```go
// Get database connection (singleton)
db := postgres.GetDB()
// Initialize schema
err := postgres.InitDB()
```
**Environment Variables:**
- `DATABASE_HOST` (default: `localhost`)
- `DATABASE_PORT` (default: `5432`)
- `DATABASE_USER` (default: `postgres`)
- `DATABASE_PASSWORD` (default: `postgres`)
- `DATABASE_NAME` (default: `sirius`)
### Host Operations
**Basic Operations (Legacy):**
```go
import "github.com/SiriusScan/go-api/sirius/host"
// Add or update host
err := host.AddHost(hostData)
// Get host by IP
hostData, err := host.GetHost("192.168.1.100")
// Get all hosts
hosts, err := host.GetAllHosts()
// Delete host
err := host.DeleteHost("192.168.1.100")
```
**Source-Aware Operations (Recommended):**
```go
// Add/update with source tracking
err := host.AddHostWithSource(hostData, source)
// Get host with all source attributions
hostWithSources, err := host.GetHostWithSources("192.168.1.100")
// Get vulnerability history by source
history, err := host.GetVulnerabilityHistory(hostID, vulnID)
// Get source coverage statistics
stats, err := host.GetSourceCoverageStats()
```
### Mapping Between Core and Database Types
**File:** `sirius/host/host.go`
```go
// Convert database model to core type
siriusHost := host.MapDBHostToSiriusHost(dbHost)
// Convert core type to database model
dbHost := host.MapSiriusHostToDBHost(siriusHost)
```
**Important:** These functions handle:
- Port.Number ↔ Port field mapping
- Relationship loading (ports, services, vulnerabilities)
- Type conversions
## Message Queue Integration
### RabbitMQ
**File:** `sirius/queue/queue.go`
```go
import "github.com/SiriusScan/go-api/sirius/queue"
// Publish message to queue
err := queue.Publish("scan", scanMessage)
// Listen for messages
queue.Listen("scan", func(msg string) {
// Process message
})
```
**Environment Variables:**
- `RABBITMQ_HOST` (default: `localhost`)
- `RABBITMQ_PORT` (default: `5672`)
- `RABBITMQ_USER` (default: `guest`)
- `RABBITMQ_PASSWORD` (default: `guest`)
**Common Queues:**
- `scan` - Scan requests
- `scanner_logs` - Scanner log events
- `agent_commands` - Agent commands
- `terminal` - Terminal commands
## Key-Value Store Integration
### ValKey/Redis
**File:** `sirius/store/store.go`
```go
import "github.com/SiriusScan/go-api/sirius/store"
// Set value
err := store.SetValue(ctx, "key", "value")
// Get value
result, err := store.GetValue(ctx, "key")
// Delete value
err := store.DeleteValue(ctx, "key")
// List keys
keys, err := store.ListKeys(ctx, "pattern:*")
```
**Environment Variables:**
- `VALKEY_HOST` (default: `localhost`)
- `VALKEY_PORT` (default: `6379`)
**Common Use Cases:**
- Real-time scan progress tracking
- Template storage (scan templates)
- Temporary data caching
## NVD Integration
### CVE Data Enrichment
**File:** `nvd/nvd.go`
```go
import "github.com/SiriusScan/go-api/nvd"
// Fetch CVE details from NVD
cveData, err := nvd.GetCVE("CVE-2017-0144")
// Access CVSS scores
score := cveData.Metrics.CvssMetricV31[0].CvssData.BaseScore
```
**Rate Limiting:** NVD API has rate limits. SDK respects these limits.
## Integration Patterns
### Pattern 1: Local Development with Replace Directive
**Use Case:** Developing changes to go-api and dependent project simultaneously
**go.mod:**
```go
module github.com/SiriusScan/app-scanner
replace github.com/SiriusScan/go-api => ../go-api // Local development
require (
github.com/SiriusScan/go-api v0.0.11 // Version for production
)
```
**Benefits:**
- Test SDK changes immediately
- No need to publish SDK for every test
- Easy debugging across projects
### Pattern 2: Production Use with Versioned Import
**Use Case:** Production deployments
**go.mod:**
```go
module github.com/SiriusScan/app-scanner
require (
github.com/SiriusScan/go-api v0.0.11 // Specific version
)
```
**Benefits:**
- Reproducible builds
- Version pinning
- Explicit dependency management
### Pattern 3: Container Development with Volume Mounts
**Use Case:** Docker-based development (sirius-engine, sirius-api)
**docker-compose.dev.yaml:**
```yaml
services:
sirius-engine:
volumes:
- ../minor-projects/go-api:/go-api:ro # Mount SDK source
- ../minor-projects/app-scanner:/app-scanner
```
**go.mod in container:**
```go
replace github.com/SiriusScan/go-api => /go-api
```
**Benefits:**
- Live code reload
- No rebuild for SDK changes
- Mirrors local development
## Version Management
### Semantic Versioning
**Format:** `v{MAJOR}.{MINOR}.{PATCH}`
**Rules:**
- **MAJOR** (0.x.x → 1.x.x): Breaking changes, incompatible API changes
- **MINOR** (x.0.x → x.1.x): New features, backward compatible
- **PATCH** (x.x.0 → x.x.1): Bug fixes, backward compatible
**Current:** v0.0.11 (pre-1.0, unstable API)
### Breaking Change Policy
**What Constitutes a Breaking Change:**
- ✅ Renaming exported fields (e.g., `Port.ID` → `Port.Number`)
- ✅ Changing function signatures
- ✅ Removing exported functions/types
- ✅ Changing database schema in non-backward-compatible way
- ❌ Adding new fields (backward compatible)
- ❌ Adding new functions (backward compatible)
- ❌ Internal implementation changes
**Communication:**
- Document in CHANGELOG.md with `BREAKING CHANGE:` prefix
- Include migration guide
- Increment version appropriately
- Notify dependent projects
### Compatibility Guarantees
**Before v1.0.0:**
- ⚠️ No API stability guaranteed
- Breaking changes may occur in patch releases
- Use exact version pinning in production
**After v1.0.0:**
- ✅ Semantic versioning strictly followed
- ✅ Breaking changes only in major versions
- ✅ Deprecation warnings before removal
## Development Workflow
### Making Changes to SDK
**1. Create feature branch:**
```bash
cd go-api
git checkout -b feature/my-change
```
**2. Make changes and test locally:**
```bash
# Edit files
go test ./...
go build ./...
```
**3. Test in dependent project:**
```bash
cd ../app-scanner
# Ensure replace directive points to ../go-api
go mod tidy
go build .
```
**4. Commit and push:**
```bash
cd ../go-api
git add .
git commit -m "feat: add new feature"
git push origin feature/my-change
```
**5. Create Pull Request**
**6. After merge to main:**
- CI/CD automatically creates new release
- Update dependent projects to new version
### Testing Changes
**Unit Tests:**
```bash
cd go-api
go test ./...
```
**Integration Tests:**
```bash
# Test with actual database
DATABASE_HOST=localhost go test ./sirius/postgres/...
# Test with actual RabbitMQ
RABBITMQ_HOST=localhost go test ./sirius/queue/...
```
**Manual Testing:**
```bash
# Use in dependent project with replace directive
cd app-scanner
go run main.go
```
### Contributing Guidelines
**Code Style:**
- Follow Go idioms and best practices
- Use `gofmt` for formatting
- Write meaningful commit messages (conventional commits)
- Add tests for new functionality
**Documentation:**
- Update CHANGELOG.md for user-facing changes
- Add godoc comments for exported types/functions
- Update SDK documentation for architectural changes
**Pull Requests:**
- Keep PRs focused and small
- Include test coverage
- Reference related issues
- Wait for CI/CD to pass
## Common Patterns & Best Practices
### Pattern: Bulk Operations
**Problem:** Need to process many hosts efficiently
**Solution:**
```go
// Use transactions for bulk operations
db := postgres.GetDB()
tx := db.Begin()
for _, hostData := range hosts {
dbHost := host.MapSiriusHostToDBHost(hostData)
if err := tx.Create(&dbHost).Error; err != nil {
tx.Rollback()
return err
}
}
tx.Commit()
```
### Pattern: Error Handling
**Problem:** Need consistent error handling
**Solution:**
```go
import "fmt"
// Wrap errors with context
if err != nil {
return fmt.Errorf("failed to add host %s: %w", ip, err)
}
// Check for specific errors
if errors.Is(err, gorm.ErrRecordNotFound) {
// Handle not found
}
```
### Pattern: Connection Pooling
**Problem:** Efficient database connections
**Solution:**
```go
// GetDB() returns singleton with connection pool
db := postgres.GetDB()
// Configure in connection.go:
sqlDB, _ := db.DB()
sqlDB.SetMaxOpenConns(25)
sqlDB.SetMaxIdleConns(5)
sqlDB.SetConnMaxLifetime(5 * time.Minute)
```
### Pattern: Graceful Shutdown
**Problem:** Clean up resources on exit
**Solution:**
```go
func main() {
// Initialize connections
db := postgres.GetDB()
// Handle shutdown
c := make(chan os.Signal, 1)
signal.Notify(c, os.Interrupt, syscall.SIGTERM)
<-c
// Cleanup
sqlDB, _ := db.DB()
sqlDB.Close()
}
```
## Troubleshooting
### Issue: "Module not found"
**Problem:**
```
go: github.com/SiriusScan/go-api@v0.0.11: invalid version: unknown revision v0.0.11
```
**Solutions:**
1. Check if version exists: `git ls-remote --tags https://github.com/SiriusScan/go-api`
2. Clear Go module cache: `go clean -modcache`
3. Use `replace` directive for local development
### Issue: "Undefined field after update"
**Problem:**
```
port.ID undefined (type sirius.Port has no field or method ID)
```
**Solution:**
- Check CHANGELOG.md for breaking changes
- Update code to use new field names (e.g., `port.Number`)
- Run `go mod tidy` after updating version
### Issue: Database connection fails
**Problem:**
```
failed to connect to database: connection refused
```
**Solution:**
1. Check environment variables are set correctly
2. Verify database is running: `docker ps | grep postgres`
3. Test connection: `psql -h localhost -U postgres -d sirius`
### Issue: "Duplicate key violation"
**Problem:**
```
ERROR: duplicate key value violates unique constraint "ports_pkey"
```
**Solution:**
- Ensure using SDK version ≥ v0.0.11 (fixes Port.ID conflict)
- Run migration 005_fix_critical_schema_issues
- Clear old data if necessary
## Migration Guides
### Migrating from v0.0.9/v0.0.10 to v0.0.11
**Breaking Changes:**
1. `Port.ID` → `Port.Number`
2. `CVEDataMeta.ID` → `CVEDataMeta.CVEIdentifier`
**Code Changes Required:**
**Before:**
```go
port := sirius.Port{
ID: 22,
Protocol: "tcp",
}
ports := make([]int, len(host.Ports))
for i, port := range host.Ports {
ports[i] = port.ID // ❌ No longer exists
}
```
**After:**
```go
port := sirius.Port{
Number: 22,
Protocol: "tcp",
}
ports := make([]int, len(host.Ports))
for i, port := range host.Ports {
ports[i] = port.Number // ✅ Correct
}
```
**Database Migration:**
```bash
# Run migration in container
docker exec sirius-engine bash -c "cd /tmp/migrations && go run 005_fix_critical_schema_issues/main.go"
```
**Verification:**
1. Update go.mod: `require github.com/SiriusScan/go-api v0.0.11`
2. Run: `go mod tidy`
3. Find all references: `grep -r "port\.ID" .`
4. Replace with `port.Number`
5. Test build: `go build ./...`
6. Run tests: `go test ./...`
## References
### Related Documentation
- [SDK Release Process](../operations/README.sdk-releases.md) - How to release new SDK versions
- [System Architecture](README.architecture.md) - Overall Sirius architecture
- [go-api README](../../../minor-projects/go-api/README.md) - SDK getting started guide
- [go-api CHANGELOG](../../../minor-projects/go-api/CHANGELOG.md) - Version history
### External Resources
- [GORM Documentation](https://gorm.io/docs/)
- [Go Module Reference](https://go.dev/ref/mod)
- [RabbitMQ Go Client](https://github.com/streadway/amqp)
- [ValKey Documentation](https://valkey.io/docs/)
---
**Last Updated:** 2025-10-26
**SDK Version:** v0.0.11
**Author:** Sirius Team
documentation/dev/architecture/README.scanner-storage.md
+196
View File
@@ -0,0 +1,196 @@
---
title: "Scanner Storage Contract"
description: "Single source of truth for the Valkey schema shared by app-scanner, sirius-api, app-agent, and sirius-ui for NSE scripts and agent templates."
template: "TEMPLATE.documentation-standard"
llm_context: "high"
categories: ["architecture", "storage", "scanner"]
tags: ["valkey", "templates", "nse", "schema", "contract", "go-api"]
related_docs:
- "README.go-api-sdk.md"
- "ARCHITECTURE.nse-repository-management.md"
- "../apps/agent/README.agent-template-api.md"
- "../apps/agent/README.agent-template-ui.md"
search_keywords: ["scanner storage", "template:custom", "template:meta", "nse:script", "templates package", "TemplateRecord", "NseScriptRecord", "canonical script id"]
---
# Scanner Storage Contract
## Overview
The Sirius scanner stack persists two record families in Valkey:
- **Agent templates** - YAML vulnerability detection templates produced by the
UI and the agent's GitHub sync, consumed by the agent runtime.
- **NSE scripts** - Nmap scripting engine entries produced by `app-scanner`'s
GitHub sync, consumed by the UI for browsing/editing and by the scanner at
scan time.
Every producer and consumer goes through the **`github.com/SiriusScan/go-api/sirius/store/templates`**
package. That package owns the key namespaces, JSON envelopes, and
canonicalization rules. If you are about to write code that reads or writes any
of the keys below, you must call into the helpers in that package - never
hand-roll the encoding.
> Drift policy: any change to a record shape, key namespace, or canonicalization
> rule requires a `go-api` version bump, an updated entry in this document, and
> an updated assertion in `Sirius/testing/integration/scanner-storage/contract_test.go`,
> all in the same change set.
## Producers, consumers, queues
```
┌──────────────────────────────────────┐
│ Valkey │
│ │
sirius-ui ──tRPC──▶│ template:custom:<id> │◀── app-agent (read)
│ template:meta:<id> │ (template runner)
│ template:standard:<id> │
sirius-api ────────▶│ template:manifest │
(uploads) │ template:repo-manifest │
│ │
app-scanner ───────▶│ nse:script:<canonical-id> │◀── sirius-ui (browse/edit)
(NSE sync) │ nse:meta:<canonical-id> │ via sirius-api
│ nse:manifest │
│ nse:repo-manifest │
└──────────────────────────────────────┘
┌──────────────────────────────────────┐
│ RabbitMQ │
│ │
sirius-api ────────▶│ agent.template.sync.jobs │── app-agent (consumer)
(notify_agents) │ engine.commands (legacy) │── app-agent (defense in depth)
└──────────────────────────────────────┘
```
The agent never polls the template manifest opportunistically; it only re-reads
when a sync notification lands on `agent.template.sync.jobs` (or the legacy
`engine.commands` mirror introduced in PR 5).
## Key namespaces
| Key | Defined as | Producer(s) | Consumer(s) |
| -------------------------------- | ------------------------------------------- | ---------------------- | ------------------- |
| `template:standard:<id>` | `templates.AgentTemplateKey(id, false)` | app-agent (sync) | app-agent, sirius-api |
| `template:custom:<id>` | `templates.AgentTemplateKey(id, true)` | sirius-api (uploads) | app-agent, sirius-api |
| `template:meta:<id>` | `templates.AgentTemplateMetaKey(id)` | sirius-api, app-agent | sirius-api (enumeration) |
| `template:manifest` | `templates.KeyAgentTemplateManifest` | app-agent | app-agent |
| `template:repo-manifest` | `templates.KeyAgentTemplateRepoManifest` | app-agent | app-agent |
| `template:version:<id>` | `templates.KeyAgentTemplateVersionPrefix` | app-agent | app-agent |
| `nse:script:<canonical-id>` | `templates.NseScriptKey(id)` | app-scanner | sirius-api, app-scanner |
| `nse:meta:<canonical-id>` | `templates.KeyNseScriptMetaPrefix` | app-scanner | sirius-api |
| `nse:manifest` | `templates.KeyNseManifest` | app-scanner | sirius-api, app-scanner |
| `nse:repo-manifest` | `templates.KeyNseRepoManifest` | app-scanner | app-scanner |
`<canonical-id>` is whatever `templates.CanonicalScriptID(id)` returns. The
canonicalizer strips the `.nse` suffix and lowercases nothing else. You should
never construct an NSE key by string concatenation; always go through
`templates.NseScriptKey`.
## Record shapes
### `TemplateRecord`
Source: [`go-api/sirius/store/templates/template_record.go`](mdc:../../../minor-projects/go-api/sirius/store/templates/template_record.go).
Field tags pinned by `TestContract_TemplateWireShape` in the contract suite:
| JSON field | Go type | Notes |
| ------------------- | ------------------ | ----- |
| `id` | string | canonical id, no extension |
| `version` | string | semver string supplied by the producer |
| `checksum` | string | hex SHA-256 of `content`; populate via `templates.SHA256Hex` |
| `size` | int64 | byte length of `content` |
| `severity` | string | `info`, `low`, `medium`, `high`, `critical` |
| `platforms` | []string | `linux`, `windows`, `darwin`, ... |
| `detection_type` | string | `agent` or `network` |
| `author` | string | optional |
| `created` | time.Time (RFC3339)| set on first upload |
| `updated` | time.Time (RFC3339)| bump on every write |
| `vulnerability_ids` | []string | CVE / advisory ids the template detects |
| `is_custom` | bool | true ⇒ persisted under `template:custom:<id>` |
| `content` | []byte (base64) | YAML body; omitted from `template:meta:<id>` |
| `metadata` | map[string]string | optional free-form labels |
The meta projection (used at `template:meta:<id>`) is always
`r.Meta()`/`templates.EncodeMeta(r)` - it strips `Content` and leaves every
other field intact, so an enumerator can list custom templates without paying
the YAML body cost.
### `NseScriptRecord`
Source: [`go-api/sirius/store/templates/nse_record.go`](mdc:../../../minor-projects/go-api/sirius/store/templates/nse_record.go).
| JSON field | Go type | Notes |
| ------------ | ------------------ | ----- |
| `content` | string | full Lua/NSE source |
| `metadata` | `NseScriptMeta` | `author`, `tags[]`, `description` |
| `updatedAt` | int64 | Unix seconds; producer-supplied |
`NseManifestEntry` (`name`, `path`, `protocol`) and `NseManifest`
(`name`, `version`, `description`, `scripts[]`) are also defined alongside
the script record. Manifest map keys are canonicalized on write by
`templates.WriteNseManifest`.
## Helpers you should be calling
```go
import "github.com/SiriusScan/go-api/sirius/store/templates"
// Templates
key := templates.AgentTemplateKey(id, isCustom) // template:standard|custom:<id>
mkey := templates.AgentTemplateMetaKey(id) // template:meta:<id>
err := templates.WriteTemplate(ctx, kv, rec) // envelope + meta, with rollback
rec, err := templates.ReadTemplate(ctx, kv, id) // tries custom then standard
meta, err := templates.ReadTemplateMeta(ctx, kv, id)
// NSE scripts
key := templates.NseScriptKey(id) // nse:script:<canonical-id>
err := templates.WriteNseScript(ctx, kv, id, rec)
rec, err := templates.ReadNseScript(ctx, kv, id) // canonicalizes id for you
err := templates.WriteNseManifest(ctx, kv, m) // canonicalizes map keys
m, err := templates.ReadNseManifest(ctx, kv)
// Canonicalization is exposed on its own for callers (e.g. the UI -> API
// path that must match what the producer wrote).
canonical := templates.CanonicalScriptID("http-shellshock.nse") // "http-shellshock"
```
## Contract test
`Sirius/testing/integration/scanner-storage/contract_test.go` is a self-contained
Go module that exercises every writer/reader pair through the shared package
against an in-memory KV. It runs as part of `make test-integration` and on every
PR via the Sirius CI Integration Test job.
Add a new pair (or a new record type) here whenever you onboard a new producer
or consumer. The suite is intentionally cheap so it can be the canary that
catches schema drift before any service redeploys.
## Operational notes
- **Reading custom vs standard** - prefer `templates.ReadTemplate`; it tries
the custom namespace first and falls through to standard, which matches the
precedence the agent uses at runtime.
- **Custom uploads** - sirius-api persists the envelope, then the meta record,
then publishes a `notify_agents` job to `agent.template.sync.jobs`. The
shared helper rolls back the envelope on a meta-write failure so the
enumerator never sees an orphaned custom template.
- **Engine commands listener** - app-agent additionally consumes the legacy
`engine.commands` queue and accepts `internal:template upload|delete` as a
defense-in-depth trigger for the same `NotifyAgents` flow. See
[`app-agent internal/server/engine_commands_consumer.go`](mdc:../../../minor-projects/app-agent/internal/server/engine_commands_consumer.go).
- **Never** persist NSE scripts with a `.nse` suffix in the key. The
canonicalization rule exists because the UI canonicalizes before lookup; if
the producer doesn't, the UI silently shows an empty body. PR 1 fixed the
original drift; the contract test guards against regressions.
## Related work
- PR 1 - canonicalize NSE script keys
([`app-scanner internal/nse/sync.go`](mdc:../../../minor-projects/app-scanner/internal/nse/sync.go))
- PR 6 - introduce the shared `templates` package in `go-api v0.0.18`
- PR 7 - migrate sirius-api, app-scanner, app-agent to that package
- PR 8 - this document plus the contract test
documentation/dev/architecture/README.system-monitor.md
+870
View File
@@ -0,0 +1,870 @@
---
title: "SystemMonitor Architecture and Operations Guide"
description: "Comprehensive guide to SystemMonitor implementation, deployment, and troubleshooting"
template: "TEMPLATE.documentation-standard"
version: "2.1.0"
last_updated: "2025-10-07"
author: "Sirius Development Team"
tags:
[
"system-monitor",
"monitoring",
"containers",
"troubleshooting",
"architecture",
"development",
]
categories: ["architecture", "operations", "monitoring"]
difficulty: "intermediate"
prerequisites: ["docker", "go", "container-monitoring", "cgroups"]
related_docs:
- "README.architecture.md"
- "README.container-testing.md"
- "README.development.md"
dependencies:
- "app-system-monitor/"
- "sirius-engine/start-enhanced.sh"
- "sirius-ui/start-dev.sh"
llm_context: "high"
search_keywords:
[
"system-monitor",
"cpu-monitoring",
"container-metrics",
"troubleshooting",
"deployment",
"development-mode",
"exec-format-error",
]
---
# SystemMonitor Architecture and Operations Guide
## Overview
The SystemMonitor is a critical component that provides real-time system resource monitoring for all Sirius containers. It collects CPU, memory, network, disk, and process metrics from within containerized environments and stores them in Valkey for centralized monitoring and alerting.
**Key Features:**
- Real-time metrics collection every 5 seconds
- Support for both development (`go run`) and production (binary) modes
- Multi-architecture support (amd64, arm64)
- CPU drift-free monitoring with time-based calculations
- Automatic fallback mechanisms for missing metrics
## Architecture
### Core Components
#### 1. **Main SystemMonitor Project**
- **Location**: `/minor-projects/app-system-monitor/`
- **Language**: Go 1.23
- **Purpose**: Single source of truth for all SystemMonitor implementations
- **Dependencies**: `github.com/SiriusScan/go-api` for Valkey connectivity
#### 2. **Deployment Modes**
**Development Mode:**
- Uses `go run main.go` to run SystemMonitor from source
- Automatically picks up code changes without rebuild
- Enabled when `GO_ENV=development` environment variable is set
- Used in sirius-engine and sirius-ui development containers
**Production Mode:**
- Uses pre-compiled binary built from GitHub source
- Compiled during Docker image build process
- Better performance and smaller runtime footprint
- Used in all production deployments
#### 3. **Container Integration**
**All Containers (PostgreSQL, RabbitMQ, Valkey, Engine, UI):**
- Build SystemMonitor from GitHub source during Docker image build
- Use multi-stage build to compile from `github.com/SiriusScan/app-system-monitor`
- Copy compiled binary to final image
- Start SystemMonitor alongside main service
**Development Mode (Engine, UI):**
- Volume-mount `../minor-projects/app-system-monitor` for live development
- Run with `go run main.go` instead of binary
- Enables rapid iteration without container rebuilds
## SystemMonitor Implementation
### CPU Monitoring (Fixed in v2.0)
The SystemMonitor uses a sophisticated CPU utilization calculation that eliminates drift:
```go
// Time-based CPU utilization calculation
cpuPercent := (float64(cpuDiff) / 1000000.0) / timeDiff / float64(sm.cpuCores) * 100.0
```
**Key Features:**
- **State Tracking**: Maintains previous CPU usage and timestamp
- **Core Detection**: Automatically detects available CPU cores from cgroups
- **Drift Prevention**: Uses time-based differences instead of cumulative values
- **Architecture Support**: Works across different CPU architectures
### Memory Monitoring
Uses cgroups v2 for accurate container memory metrics:
```go
// Read from cgroups v2
currentBytes := readCgroupFile("/sys/fs/cgroup/memory.current")
maxBytes := readCgroupFile("/sys/fs/cgroup/memory.max")
percent := float64(currentBytes) / float64(maxBytes) * 100.0
```
### Network Monitoring
Tracks network I/O from container interfaces:
```go
// Network statistics
rxBytes := readCgroupFile("/sys/class/net/eth0/statistics/rx_bytes")
txBytes := readCgroupFile("/sys/class/net/eth0/statistics/tx_bytes")
```
### Disk Monitoring
Calculates container filesystem usage:
```go
// Walk container filesystem (excluding mounted volumes)
filepath.WalkDir("/", func(path string, d os.DirEntry, err error) error {
// Skip mounted volumes and system directories
if strings.HasPrefix(path, "/api") || strings.HasPrefix(path, "/proc") {
return filepath.SkipDir
}
// Calculate total size
})
```
## Deployment Architecture
### Build Process
#### Production Builds (All Containers)
All containers use a multi-stage Docker build to compile SystemMonitor from source:
```dockerfile
# Builder stage - Clone and build SystemMonitor
FROM golang:1.23-alpine AS builder
RUN apk add --no-cache git
RUN git clone https://github.com/SiriusScan/app-system-monitor.git && \
cd app-system-monitor && \
go mod download && \
CGO_ENABLED=0 GOOS=linux go build -ldflags="-w -s" -o system-monitor main.go
# Runtime stage - Copy binary
FROM <base-image>
COPY --from=builder /go/app-system-monitor/system-monitor /usr/local/bin/system-monitor
RUN chmod +x /usr/local/bin/system-monitor
```
**Benefits:**
- Always builds from latest source code
- No pre-compiled binary distribution needed
- Architecture-specific compilation
- Smaller final image size
#### Development Mode (Engine & UI)
Development containers use `go run` for live development:
```yaml
# docker-compose.dev.yaml
volumes:
- ../minor-projects/app-system-monitor:/system-monitor
environment:
- GO_ENV=development
```
**Startup logic checks environment:**
```bash
if [ "$GO_ENV" = "development" ] && [ -f "/system-monitor/main.go" ]; then
CONTAINER_NAME=sirius-engine go run main.go &
elif [ -f "/system-monitor/system-monitor" ] && [ -x "/system-monitor/system-monitor" ]; then
CONTAINER_NAME=sirius-engine ./system-monitor &
fi
```
### Container Startup Integration
#### PostgreSQL/RabbitMQ/Valkey Pattern (Production Only)
```bash
# In start-with-monitor.sh
echo "📊 Starting system monitor..."
cd /usr/local/bin
CONTAINER_NAME=sirius-postgres ./system-monitor &
```
These containers always use the compiled binary from the Docker build.
#### Engine Pattern (Development & Production)
```bash
# In start-enhanced.sh
if [ -d "/system-monitor" ]; then
echo "Starting system monitor..."
cd /system-monitor
if [ "$GO_ENV" = "development" ] && [ -f "main.go" ]; then
echo "Running system monitor with go run (development mode)"
CONTAINER_NAME=sirius-engine go run main.go &
elif [ -f "./system-monitor" ] && [ -x "./system-monitor" ]; then
echo "Running system monitor binary (production mode)"
CONTAINER_NAME=sirius-engine ./system-monitor &
else
echo "Error: System monitor not found (neither main.go nor binary)"
exit 1
fi
SYSTEM_MONITOR_PID=$!
sleep 2
check_service "System Monitor" $SYSTEM_MONITOR_PID
fi
```
**Key Features:**
- Checks `GO_ENV` environment variable
- Prefers `go run` in development mode
- Falls back to binary in production
- Provides clear error messages if neither is available
#### UI Pattern (Development & Production)
```bash
# In start-dev.sh
if [ -d "/system-monitor" ] && [ -f "/system-monitor/main.go" ]; then
echo "📊 Starting System Monitor..."
cd /system-monitor
CONTAINER_NAME=sirius-ui go run main.go &
SYSTEM_MONITOR_PID=$!
echo "✅ System Monitor started with PID: $SYSTEM_MONITOR_PID"
cd /app
fi
```
```bash
# In start-prod.sh
if [ -f "/system-monitor/system-monitor" ] && [ -x "/system-monitor/system-monitor" ]; then
echo "📊 Starting system monitor..."
cd /system-monitor
CONTAINER_NAME=sirius-ui ./system-monitor &
SYSTEM_MONITOR_PID=$!
echo "✅ System monitor started with PID: $SYSTEM_MONITOR_PID"
cd /app
fi
```
## Configuration
### Environment Variables
| Variable | Purpose | Default | Example |
| ---------------- | ---------------------- | --------------- | ----------------- |
| `CONTAINER_NAME` | Container identifier | `unknown` | `sirius-postgres` |
| `VALKEY_HOST` | Valkey server hostname | `sirius-valkey` | `localhost` |
| `VALKEY_PORT` | Valkey server port | `6379` | `6379` |
### Metrics Storage
**Valkey Keys:**
- `system:metrics:{container_name}` - Latest metrics
- `system:logs:{container_name}:{timestamp}` - Log entries
**Metrics Structure:**
```json
{
"container_name": "sirius-postgres",
"timestamp": "2025-01-07T12:00:00Z",
"cpu_percent": 15.2,
"memory_usage_bytes": 67108864,
"memory_percent": 1.8,
"network_rx_bytes": 1048576,
"network_tx_bytes": 524288,
"disk_usage_bytes": 134217728,
"disk_percent": 1.3,
"process_count": 12,
"file_descriptors": 45,
"load_average_1m": 0.15,
"load_average_5m": 0.12,
"load_average_15m": 0.1,
"uptime_seconds": 3600,
"status": "running"
}
```
## Troubleshooting Guide
### Common Issues and Solutions
#### 1. **SystemMonitor Not Starting / Exec Format Error**
**Symptoms:**
- Container starts but no metrics appear in monitoring dashboard
- Logs show "System Monitor failed to start or crashed"
- **"cannot execute binary file: Exec format error"** (most common in development mode)
- Container stuck in crash loop restarting every few seconds
**Most Likely Causes:**
1. **Development Mode Binary Mismatch** (95% of cases):
- Volume-mounted `/system-monitor` contains pre-compiled binary for wrong architecture
- Development container trying to execute x86_64 binary on ARM64 (or vice versa)
- Solution: Use `go run` instead of pre-compiled binary
2. **Missing Binary**: SystemMonitor binary not present in container
3. **Permission Issues**: Binary not executable
4. **Go Not Available**: Development mode but Go not installed in container
**Diagnostic Steps:**
```bash
# Check if binary exists and architecture
docker exec <container> ls -la /system-monitor/
docker exec <container> file /system-monitor/system-monitor
# Check container architecture
docker exec <container> uname -m
# Check if Go is available (for development mode)
docker exec <container> which go
docker exec <container> go version
# Check environment mode
docker exec <container> env | grep GO_ENV
# Check container logs for startup mode
docker logs <container> | grep -i "system monitor\|go run\|development"
```
**Solutions:**
**For Development Mode (Engine/UI):**
1. **Verify startup script uses `go run`**:
```bash
# Check if start-enhanced.sh or start-dev.sh has:
if [ "$GO_ENV" = "development" ] && [ -f "main.go" ]; then
go run main.go &
fi
```
2. **Rebuild container to pick up updated startup script**:
```bash
docker-compose -f docker-compose.yaml -f docker-compose.dev.yaml build <service>
docker-compose -f docker-compose.yaml -f docker-compose.dev.yaml up -d <service>
```
3. **Verify development environment is set**:
```bash
# In docker-compose.dev.yaml
environment:
- GO_ENV=development
```
**For Production Mode:**
1. **Rebuild container** (rebuilds SystemMonitor from source):
```bash
docker-compose build <service>
docker-compose up -d <service>
```
2. **Verify Dockerfile has multi-stage build**:
```dockerfile
FROM golang:1.23-alpine AS builder
RUN git clone https://github.com/SiriusScan/app-system-monitor.git
# ... build steps ...
```
#### 2. **CPU Utilization Drift**
**Symptoms:**
- CPU usage gradually increases over time
- Metrics don't match Docker's performance data
- CPU percentages exceed 100%
**Root Cause:**
- Old SystemMonitor versions used cumulative CPU time incorrectly
- Missing time-based calculation logic
**Solution:**
- Ensure using SystemMonitor v2.0+ with proper CPU calculation
- Verify `lastCPUUsage` and `lastCPUTime` state tracking
#### 3. **Valkey Connection Issues**
**Symptoms:**
- SystemMonitor starts but metrics not stored
- "Error storing metrics" in logs
- Empty metrics in monitoring dashboard
**Diagnostic Steps:**
```bash
# Check Valkey connectivity
docker exec <container> nc -zv sirius-valkey 6379
# Check Valkey logs
docker logs sirius-valkey
# Test Valkey connection from container
docker exec <container> redis-cli -h sirius-valkey ping
```
**Solutions:**
1. **Network Issues**: Check Docker network connectivity
2. **Valkey Down**: Restart Valkey container
3. **Firewall**: Check port 6379 accessibility
#### 4. **Service Crash Loop Due to Scanner/Other Service Failures**
**Symptoms:**
- SystemMonitor starts successfully but then container restarts
- Logs show "Scanner failed to start or crashed"
- Container repeatedly cycles through startup sequence
- SystemMonitor unable to maintain connection
**Root Cause:**
- In early implementations, any service failure (like scanner) would trigger `cleanup` function
- This shutdown ALL services including SystemMonitor
- Created unnecessary crash loops when only one service had issues
**Solution (Implemented in v2.1):**
The scanner and other non-critical services are now non-fatal:
```bash
# In start-enhanced.sh
if [ -n "$SCANNER_PATH" ]; then
cd "$SCANNER_PATH"
echo "Starting scanner service..."
go run main.go &
SCANNER_PID=$!
sleep 2
# Check if scanner started, but don't fail if it didn't
if ! kill -0 $SCANNER_PID 2>/dev/null; then
echo "Warning: Scanner failed to start, but continuing with other services"
SCANNER_PID=""
fi
fi
```
**Key Changes:**
- Scanner failure no longer calls `cleanup` function
- Other services (SystemMonitor, Terminal, Agent) continue running
- Warning logged but container remains operational
- Allows investigation of scanner issues without affecting monitoring
#### 5. **Memory Metrics Inaccurate**
**Symptoms:**
- Memory usage doesn't match container limits
- Memory percentage calculations seem wrong
**Diagnostic Steps:**
```bash
# Check cgroups v2 files
docker exec <container> cat /sys/fs/cgroup/memory.current
docker exec <container> cat /sys/fs/cgroup/memory.max
# Compare with Docker stats
docker stats <container> --no-stream
```
**Solutions:**
1. **Cgroups v2**: Ensure container supports cgroups v2
2. **Fallback Values**: Check if using fallback memory estimates
3. **Container Limits**: Verify memory limits are set correctly
### Performance Issues
#### High CPU Usage from SystemMonitor
**Symptoms:**
- SystemMonitor itself consuming significant CPU
- Container performance degraded
**Causes:**
1. **Frequent Polling**: Metrics collection interval too short
2. **Inefficient Calculations**: Complex disk usage calculations
3. **Memory Leaks**: Go runtime issues
**Solutions:**
1. **Adjust Polling**: Increase collection interval (default: 5 seconds)
2. **Optimize Disk Scanning**: Limit filesystem traversal
3. **Memory Profiling**: Use Go profiling tools
#### Memory Leaks
**Symptoms:**
- SystemMonitor memory usage grows over time
- Container memory limits exceeded
**Diagnostic Steps:**
```bash
# Monitor SystemMonitor memory usage
docker exec <container> ps aux | grep system-monitor
# Check for goroutine leaks
docker exec <container> curl http://localhost:6060/debug/pprof/goroutine
```
### Log Analysis
#### SystemMonitor Logs
**Location**: Container stdout/stderr
**Format**: Standard Go log format
**Key Log Messages:**
- `"Starting system monitor for container: {name}"` - Successful startup
- `"Error reading cpu.stat: {error}"` - CPU monitoring issues
- `"Error storing metrics: {error}"` - Valkey connectivity problems
- `"Warning: metrics channel full, dropping metrics"` - Performance issues
#### Log Patterns for Issues
```bash
# CPU monitoring problems
grep -i "cpu.stat\|cpu.max" <container_logs>
# Memory monitoring issues
grep -i "memory.current\|memory.max" <container_logs>
# Network connectivity
grep -i "valkey\|redis" <container_logs>
# Performance warnings
grep -i "channel full\|dropping" <container_logs>
```
## Maintenance and Updates
### Updating SystemMonitor
#### For Development Mode (Local Changes)
1. **Modify Source Code**:
```bash
cd /minor-projects/app-system-monitor/
# Make changes to main.go
```
2. **Test Changes** (no rebuild needed in development mode):
```bash
# Restart container to pick up changes
docker-compose -f docker-compose.yaml -f docker-compose.dev.yaml restart sirius-engine
# SystemMonitor will recompile with go run on startup
```
3. **Verify Changes**:
```bash
# Check logs for SystemMonitor startup
docker logs sirius-engine | grep -i "system monitor"
# Verify metrics in Valkey
docker exec sirius-valkey redis-cli get "system:metrics:sirius-engine"
```
#### For Production Deployment
1. **Push Changes to GitHub**:
```bash
cd /minor-projects/app-system-monitor/
git add .
git commit -m "feat(monitoring): update SystemMonitor metrics"
git push origin main
```
2. **Rebuild All Containers** (they fetch latest from GitHub):
```bash
# Full rebuild with no cache to get latest source
docker-compose build --no-cache sirius-postgres sirius-rabbitmq sirius-valkey sirius-engine sirius-ui
```
3. **Deploy Updates**:
```bash
docker-compose up -d
```
#### Testing in Development Before Production
1. **Make changes locally** in `/minor-projects/app-system-monitor/`
2. **Test in development mode** with `docker-compose.dev.yaml`
3. **Verify metrics** are collecting correctly
4. **Commit and push** to GitHub
5. **Rebuild production images** to fetch from GitHub
6. **Deploy to production**
### Monitoring SystemMonitor Health
#### Health Check Endpoints
**Valkey Metrics Check**:
```bash
# Check if metrics are being stored
redis-cli -h sirius-valkey get "system:metrics:sirius-postgres"
```
**Container Process Check**:
```bash
# Verify SystemMonitor is running
docker exec <container> ps aux | grep system-monitor
```
#### Automated Monitoring
**Metrics Collection Frequency**: Every 5 seconds
**Log Generation**: Every 30 seconds
**Health Check**: Monitor Valkey key updates
### Development Guidelines
#### Adding New Metrics
1. **Update SystemMetrics struct**:
```go
type SystemMetrics struct {
// ... existing fields ...
NewMetric float64 `json:"new_metric"`
}
```
2. **Implement Collection Logic**:
```go
func (sm *SystemMonitor) readNewMetric() float64 {
// Implementation
}
```
3. **Update gatherSystemMetrics**:
```go
return SystemMetrics{
// ... existing fields ...
NewMetric: sm.readNewMetric(),
}
```
#### Testing Changes
1. **Local Testing**:
```bash
cd /minor-projects/app-system-monitor/
go run main.go
```
2. **Container Testing**:
```bash
# Build and test in container
docker run --rm -it <container_image> /usr/local/bin/system-monitor
```
3. **Integration Testing**:
```bash
# Deploy to development environment
docker-compose -f docker-compose.dev.yaml up -d
```
## Security Considerations
### Container Security
- **Non-root Execution**: SystemMonitor runs with container user permissions
- **Read-only Access**: Only reads system files, no write access
- **Network Isolation**: Communicates only with Valkey service
### Data Privacy
- **No Sensitive Data**: Only collects system resource metrics
- **Local Processing**: All calculations performed within container
- **Encrypted Transport**: Valkey connections should use TLS in production
## Performance Tuning
### Optimization Settings
**Collection Intervals**:
- **Metrics**: 5 seconds (configurable)
- **Logs**: 30 seconds (configurable)
**Resource Limits**:
- **CPU**: Minimal impact (< 0.1% typical)
- **Memory**: ~10-20MB per container
- **Network**: Minimal bandwidth usage
### Scaling Considerations
- **Container Count**: Linear scaling with container count
- **Valkey Load**: Minimal impact on Valkey performance
- **Network Overhead**: Negligible for typical deployments
## Support and Escalation
### First-Level Support
1. **Check Container Status**: `docker ps`
2. **Review Logs**: `docker logs <container>`
3. **Verify Network**: Test Valkey connectivity
4. **Check Metrics**: Verify Valkey key updates
### Escalation Path
1. **Development Team**: For SystemMonitor code issues
2. **Infrastructure Team**: For container/Docker issues
3. **Database Team**: For Valkey connectivity problems
### Emergency Procedures
**SystemMonitor Down**:
1. Check container health
2. Restart affected containers
3. Verify Valkey connectivity
4. Check for resource constraints
**Metrics Not Updating**:
1. Verify SystemMonitor process running
2. Check Valkey connectivity
3. Review container logs
4. Test manual metric collection
---
## Quick Reference
### Essential Commands
```bash
# Development Mode - Restart to pick up changes
docker-compose -f docker-compose.yaml -f docker-compose.dev.yaml restart sirius-engine sirius-ui
# Production Mode - Rebuild from GitHub source
docker-compose build --no-cache sirius-postgres sirius-rabbitmq sirius-valkey sirius-engine sirius-ui
# Check SystemMonitor status
docker exec <container> ps aux | grep -E "system-monitor|go run main.go"
# View metrics in Valkey
docker exec sirius-valkey redis-cli get "system:metrics:<container_name>"
# Check container logs for startup mode
docker logs <container> | grep -i "system monitor\|development\|go run"
# Verify development environment
docker exec <container> env | grep GO_ENV
# Check for exec format errors
docker logs <container> | grep -i "exec format error"
```
### Key Files
**Source Code:**
- **SystemMonitor**: `/minor-projects/app-system-monitor/main.go`
- **GitHub Repo**: `https://github.com/SiriusScan/app-system-monitor`
**Container Integration:**
- **Engine Startup**: `/sirius-engine/start-enhanced.sh`
- **UI Dev Startup**: `/sirius-ui/start-dev.sh`
- **UI Prod Startup**: `/sirius-ui/start-prod.sh`
- **Other Services**: `/<service>/start-with-monitor.sh`
**Dockerfiles:**
- **Engine**: `/sirius-engine/Dockerfile` (multi-stage build from GitHub)
- **UI**: `/sirius-ui/Dockerfile` (multi-stage build from GitHub)
- **PostgreSQL**: `/sirius-postgres/Dockerfile` (multi-stage build from GitHub)
- **RabbitMQ**: `/sirius-rabbitmq/Dockerfile` (multi-stage build from GitHub)
- **Valkey**: `/sirius-valkey/Dockerfile` (multi-stage build from GitHub)
### Monitoring Endpoints
- **Valkey**: `sirius-valkey:6379`
- **Metrics Key**: `system:metrics:{container_name}`
- **Logs Key**: `system:logs:{container_name}:{timestamp}`
### Development Mode Indicators
**Logs showing development mode:**
```
Running system monitor with go run (development mode)
go: downloading github.com/SiriusScan/go-api...
System Monitor started with PID: 7
```
**Logs showing production mode:**
```
Running system monitor binary (production mode)
System monitor started with PID: 8
```
### Common Issues Quick Fix
| Issue | Quick Fix |
| --------------------------------- | ----------------------------------------------------------------------------- |
| Exec format error | Rebuild with `docker-compose build <service>` |
| SystemMonitor not running | Check `docker logs <container>` for startup errors |
| Metrics not updating | Verify Valkey connection: `docker exec <container> nc -zv sirius-valkey 6379` |
| Container crash loop | Check if scanner failure is bringing down other services |
| Development changes not appearing | Restart container: `docker-compose restart <service>` |
---
_This documentation covers SystemMonitor v2.1+ with development mode support, CPU drift fixes, and multi-stage Docker builds. For older versions, refer to historical documentation or upgrade to the latest version._
documentation/dev/deployment/README.docker-container-deployment.md
+428
View File
@@ -0,0 +1,428 @@
---
title: "Docker Container Deployment Guide"
description: "Complete guide for deploying Sirius using prebuilt container images from GitHub Container Registry"
template: "TEMPLATE.guide"
version: "1.0.0"
last_updated: "2026-04-08"
author: "Development Team"
tags: ["docker", "deployment", "containers", "ghcr", "registry", "production"]
categories: ["deployment", "infrastructure"]
difficulty: "beginner"
prerequisites: ["docker", "docker-compose"]
related_docs:
- "README.terraform-deployment.md"
- "README.development.md"
- "README.docker-architecture.md"
dependencies: ["docker-compose.yaml"]
llm_context: "high"
search_keywords:
[
"docker",
"deployment",
"containers",
"ghcr",
"registry",
"production",
"images",
"compose",
]
---
# Docker Container Deployment Guide
## Purpose
This guide explains how to deploy Sirius using prebuilt container images from GitHub Container Registry (GHCR). This approach provides faster deployments (5-8 minutes vs 20-25 minutes) by eliminating on-instance Docker builds. The guide covers production deployments, image versioning, and fallback strategies.
## When to Use
- **Production deployments** - Deploying Sirius to production environments
- **Demo environments** - Setting up demo instances quickly
- **Staging environments** - Creating isolated testing environments
- **Quick deployments** - When you need fast deployment without building from source
- **CI/CD pipelines** - Automated deployments using prebuilt images
**Avoid when**:
- **Local development** - Use `docker-compose.dev.yaml` for local builds with hot reloading
- **Custom modifications** - When you need to modify source code before deployment
- **Offline environments** - When registry access is unavailable
## How to Use
### Quick Start
```bash
# Clone repository
git clone https://github.com/SiriusScan/Sirius.git
cd Sirius
# Generate startup config and secrets
docker compose -f docker-compose.installer.yaml run --rm sirius-installer
# Deploy with prebuilt images (default)
docker compose up -d
# Or specify a version tag
IMAGE_TAG=v0.4.1 docker compose up -d
# Optional: verify the public GHCR contract before pulling
bash scripts/verify-ghcr-public-access.sh "${IMAGE_TAG:-latest}"
```
### Prerequisites
- **Docker** >= 20.10.0
- **Docker Compose** >= 2.0.0
- **Internet access** to GitHub Container Registry (ghcr.io)
- **Git** (for cloning repository)
## Container Registry Integration
### GitHub Container Registry (GHCR)
Sirius images are automatically built and pushed to GitHub Container Registry on every push to the main branch. Images are available at:
- **UI**: `ghcr.io/siriusscan/sirius-ui:{tag}`
- **API**: `ghcr.io/siriusscan/sirius-api:{tag}`
- **Engine**: `ghcr.io/siriusscan/sirius-engine:{tag}`
- **Postgres**: `ghcr.io/siriusscan/sirius-postgres:{tag}`
- **RabbitMQ**: `ghcr.io/siriusscan/sirius-rabbitmq:{tag}`
- **Valkey**: `ghcr.io/siriusscan/sirius-valkey:{tag}`
`docker-compose.yaml` is the public-image deployment path for Sirius. If an unauthenticated `docker pull` against one of the image refs above returns `unauthorized`, the GHCR public-visibility contract is broken and operators should stop before rollout.
### Image Tagging Strategy
Images are tagged with the following strategy:
| Tag | Description | When Created |
| -------- | ------------------------ | ----------------------- |
| `latest` | Latest main branch build | On every push to main |
| `beta` | Beta release candidate | On main branch pushes |
| `v0.4.1` | Version-specific tag | After the `Publish Release Image Tags` workflow succeeds |
| `dev` | Development builds | On other branch pushes |
| `pr-123` | Pull request builds | On PR creation/updates |
### Image Availability
- **Public images**: No authentication required
- **Multi-architecture**: Images support `linux/amd64` and `linux/arm64`
- **Automatic updates**: Latest images are built automatically by CI/CD
- **Release contract**: A release tag is only valid for operators after the release-tag workflow publishes it, the anonymous GHCR verification step passes, and the public Compose smoke test succeeds. CI validates **`latest`** on every main push (`public-stack-contract` in `ci.yml`); **semver** tags are additionally checked when a GitHub Release is published and on a weekly schedule ([`verify-ghcr-release-tag.yml`](../../../.github/workflows/verify-ghcr-release-tag.yml)).
## Docker Compose Configuration
### Base Configuration (Production)
The default `docker-compose.yaml` uses prebuilt images:
```yaml
services:
sirius-ui:
image: ghcr.io/siriusscan/sirius-ui:${IMAGE_TAG:-latest}
pull_policy: always
# ... other configuration
sirius-api:
image: ghcr.io/siriusscan/sirius-api:${IMAGE_TAG:-latest}
pull_policy: always
# ... other configuration
sirius-engine:
image: ghcr.io/siriusscan/sirius-engine:${IMAGE_TAG:-latest}
pull_policy: always
# ... other configuration
```
### Environment Variables
Control which images to use with the `IMAGE_TAG` environment variable:
```bash
# Use latest images (default)
docker compose up -d
# Use specific version
IMAGE_TAG=v0.4.1 docker compose up -d
# Use beta release
IMAGE_TAG=beta docker compose up -d
# Validate that the selected tag is publicly readable before rollout
bash scripts/verify-ghcr-public-access.sh "${IMAGE_TAG:-latest}"
```
`.env.production.example` leaves `IMAGE_TAG` blank so fresh installer runs inherit the Compose default (`latest`). Pin a release tag only after the publish workflow has validated that all six Sirius images exist for that tag.
### Development Override
For local development with source code changes, use the development override:
```bash
# Build locally with hot reloading
docker compose -f docker-compose.yaml -f docker-compose.dev.yaml up -d --build
```
The `docker-compose.dev.yaml` file overrides the registry images with local builds, enabling:
- Hot reloading
- Volume mounts for source code
- Development-specific environment variables
- Debug logging
## Deployment Workflow
### Standard Production Deployment
1. **Clone repository**:
```bash
git clone https://github.com/SiriusScan/Sirius.git
cd Sirius
```
2. **Configure environment**:
```bash
docker compose -f docker-compose.installer.yaml run --rm sirius-installer
# Optional: pass explicit values
# docker compose -f docker-compose.installer.yaml run --rm sirius-installer --non-interactive --no-print-secrets
```
3. **Deploy services**:
```bash
docker compose up -d
```
4. **Verify deployment**:
```bash
docker compose ps
docker compose logs -f
```
5. **Check health**:
```bash
curl http://localhost:9001/health # API
curl http://localhost:3000/api/health # UI
```
### Maintainer Validation Path
Use the shared validation script to exercise the same public Compose path that operators use:
```bash
# Validate the default public stack (latest)
bash scripts/validate-public-compose-path.sh latest
# Validate a published release tag
bash scripts/validate-public-compose-path.sh v0.4.1
```
### Version-Specific Deployment
Deploy a specific version:
```bash
# Set version tag
export IMAGE_TAG=v0.4.1
# Confirm the compose-rendered GHCR images are public and readable
bash scripts/verify-ghcr-public-access.sh "$IMAGE_TAG"
# Pull and start services
docker compose pull
docker compose up -d
```
### Updating Deployment
To update to the latest version:
```bash
# Pull latest images
docker compose pull
# Restart services with new images
docker compose up -d
```
## Fallback Strategy
### When Registry is Unavailable
If GitHub Container Registry is unavailable or images fail to pull, you can fall back to local builds:
1. **Use the committed source-build override**:
```bash
docker compose -f docker-compose.yaml -f docker-compose.build.yaml up -d --build
```
**Note**: Local builds take significantly longer (20-25 minutes vs 5-8 minutes) and require more system resources.
## Secrets Hardening Overlays
For hardened deployments, Sirius includes optional overlay manifests:
- `docker-compose.secrets.yaml` for Compose secrets mounted at `/run/secrets/*`
- `docker-stack.swarm.yaml` for Swarm stack deployments
Example:
```bash
mkdir -p secrets
printf '%s' "your-postgres-password" > secrets/postgres_password.txt
printf '%s' "your-service-key" > secrets/sirius_api_key.txt
chmod 644 secrets/sirius_api_key.txt
printf '%s' "your-nextauth-secret" > secrets/nextauth_secret.txt
printf '%s' "your-admin-password" > secrets/initial_admin_password.txt
docker compose -f docker-compose.yaml -f docker-compose.secrets.yaml up -d
```
`sirius_api_key.txt` should stay **world-readable** (`644`) so bind-mounted `/run/secrets/sirius_api_key` is readable inside **sirius-api** / **sirius-ui** / **sirius-engine** (non-root UIDs).
## Troubleshooting
### Images Not Pulling
**Problem**: `docker compose pull` fails with authentication or network errors.
**Solutions**:
- Verify internet connectivity: `curl -I https://ghcr.io`
- Check image exists: Visit `https://github.com/SiriusScan/Sirius/pkgs/container/sirius-ui`
- Try pulling manually: `docker pull ghcr.io/siriusscan/sirius-ui:latest`
- Run the contract check: `bash scripts/verify-ghcr-public-access.sh "${IMAGE_TAG:-latest}"`
- If the script reports `Anonymous access denied`, the package is not publicly readable and the GHCR visibility workflow or token scope needs attention
- If the script reports `Manifest missing`, the requested tag was not published and you should verify the release-tag workflow completed successfully
- If the public Compose smoke test fails after pull succeeds, run `bash scripts/validate-public-compose-path.sh "${IMAGE_TAG:-latest}"` to reproduce the operator path and inspect the runtime contract failure
- Use fallback build strategy (see above)
### Wrong Version Deployed
**Problem**: Services are running an unexpected version.
**Solutions**:
- Check current IMAGE_TAG: `echo $IMAGE_TAG`
- Verify image tags: `docker compose images`
- Pull specific version: `IMAGE_TAG=v0.4.1 docker compose pull`
- Restart services: `docker compose up -d`
### Services Not Starting
**Problem**: Containers fail to start after pulling images.
**Solutions**:
- Check logs: `docker compose logs`
- Verify environment variables: `docker compose config`
- Check image compatibility: Ensure architecture matches (amd64/arm64)
- Verify dependencies: Ensure PostgreSQL, RabbitMQ, Valkey are running
### Performance Issues
**Problem**: Deployment is slower than expected.
**Solutions**:
- Check network speed: `docker pull` should be fast on good connections
- Verify image sizes: Large images take longer to pull
- Use specific version tags instead of `latest` for faster pulls
- Consider using image caching strategies
## Best Practices
### Security
- **Use specific version tags** in production (e.g., `v0.4.1`) instead of `latest`
- **Regularly update images** to get security patches
- **Scan images** for vulnerabilities using Docker security scanning
- **Use private registries** for sensitive deployments (if needed)
### Version Management
- **Pin versions** in production environments
- **Test updates** in staging before production
- **Document versions** deployed in each environment
- **Use semantic versioning** for releases
### Performance
- **Pre-pull images** before deployment to reduce startup time
- **Use image caching** to avoid redundant pulls
- **Monitor image sizes** and optimize Dockerfiles if needed
- **Use multi-stage builds** to reduce final image sizes
### Monitoring
- **Track deployment times** to measure improvements
- **Monitor registry availability** and fallback usage
- **Log image versions** deployed for audit trails
- **Alert on deployment failures** for quick response
## Comparison: Registry vs Local Builds
| Aspect | Registry Images | Local Builds |
| --------------------- | ----------------- | -------------------- |
| **Deployment Time** | 5-8 minutes | 20-25 minutes |
| **Resource Usage** | Low (pull only) | High (compilation) |
| **Network Required** | Yes (for pull) | No (after clone) |
| **Customization** | Limited | Full |
| **CI/CD Integration** | Automatic | Manual |
| **Best For** | Production, demos | Development, offline |
## Integration with CI/CD
### GitHub Actions
Images are automatically built and pushed by GitHub Actions on:
- Push to `main` branch → `latest` and `beta` tags
- Manual `Publish Release Image Tags` run → version-specific tag (e.g., `v0.4.1`)
- Pull requests → `pr-{number}` tags
### Deployment Automation
Example GitHub Actions workflow for deployment:
```yaml
name: Deploy Sirius
on:
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Deploy with registry images
run: |
docker compose pull
docker compose up -d
```
## Related Documentation
- [Terraform Deployment Guide](README.terraform-deployment.md) - AWS deployment using Terraform
- [Development Guide](../README.development.md) - Local development setup
- [Docker Architecture Guide](../architecture/README.docker-architecture.md) - Container architecture details
## Support
For issues with container deployment:
1. Check the troubleshooting section above
2. Review Docker logs: `docker compose logs`
3. Verify image availability on GitHub Container Registry
4. Create an issue in the Sirius repository
---
_This guide follows the Sirius Documentation Standard. For questions about documentation structure, see [ABOUT.documentation.md](../ABOUT.documentation.md)._
documentation/dev/deployment/README.workflows.md
+464
View File
@@ -0,0 +1,464 @@
---
title: "GitHub Actions Workflow Architecture"
description: "Comprehensive guide to Sirius CI/CD pipeline structure, parallel build jobs, and deployment workflows"
template: "TEMPLATE.documentation-standard"
version: "1.0.0"
last_updated: "2025-11-14"
author: "Sirius Team"
tags: ["ci-cd", "github-actions", "docker", "ghcr", "parallel-builds"]
categories: ["deployment", "automation"]
difficulty: "intermediate"
prerequisites: ["README.docker-container-deployment.md", "README.architecture.md"]
related_docs:
- "README.docker-container-deployment.md"
- "README.terraform-deployment.md"
- "README.cicd.md"
dependencies:
- ".github/workflows/ci.yml"
llm_context: "high"
search_keywords: ["ci-cd", "github-actions", "workflow", "parallel", "build", "deploy", "ghcr", "container-registry"]
---
# GitHub Actions Workflow Architecture
## Purpose
This document describes the Sirius CI/CD pipeline structure, explaining how GitHub Actions workflows orchestrate parallel container builds, validation, testing, and deployment. It's designed for developers maintaining or modifying the CI/CD pipeline and for those troubleshooting build issues.
## When to Use
- **Modifying CI/CD pipeline**: Making changes to build, test, or deployment processes
- **Troubleshooting build failures**: Understanding job dependencies and execution order
- **Adding new services**: Integrating new containers into the build pipeline
- **Optimizing build times**: Identifying parallelization opportunities
- **Debugging deployment issues**: Understanding how code reaches production
## How to Use
### Quick Reference
```bash
# Check workflow status
gh run list --workflow=ci.yml --limit 5
# Watch current run
gh run watch
# View workflow logs
gh run view <run-id> --log
# Manually trigger workflow
gh workflow run ci.yml --ref main
```
### Understanding Workflow Execution
1. **Code Push/PR**: Triggers the workflow automatically
2. **Change Detection**: Determines which services need rebuilding
3. **Parallel Builds**: UI, API, and Engine build concurrently
4. **Integration Tests**: Validates built images work together
5. **Deployment Dispatch**: Triggers downstream deployments
## What It Is
### Workflow Overview
The Sirius CI/CD pipeline is implemented in `.github/workflows/ci.yml` and follows a **parallel build architecture** that significantly reduces build times compared to sequential builds.
**Other workflows** (not shown in the diagram below):
- [`.github/workflows/publish-release-image-tags.yml`](../../../.github/workflows/publish-release-image-tags.yml) — manual retag of all six GHCR images from a source tag (e.g. `latest`) to a release tag (e.g. `v1.0.0`).
- [`.github/workflows/verify-ghcr-release-tag.yml`](../../../.github/workflows/verify-ghcr-release-tag.yml) — on each **published** GitHub Release, on a **weekly** schedule, and via **workflow_dispatch**, runs `scripts/verify-ghcr-public-access.sh` so the release tag cannot drift from what anonymous operators can pull (see [OPERATIONS.md](../../../OPERATIONS.md) GHCR checklist).
**Key characteristics:**
- **Parallel execution**: UI, API, and Engine containers build simultaneously
- **Smart change detection**: Only rebuilds services with code changes
- **Prebuild validation**: Runs lints/tests before expensive Docker builds
- **Multi-arch support**: Builds for both amd64 and arm64 architectures
- **Container registry integration**: Pushes to GitHub Container Registry (GHCR)
### Architecture Diagram
```
┌─────────────────┐
│ detect-changes │ ← Determines what needs rebuilding
└────────┬────────┘
│
├──────────┬──────────┬──────────┐
▼ ▼ ▼ ▼
┌────────┐ ┌────────┐ ┌──────────┐ │
│build-ui│ │build-api│ │build-engine│ │ ← Run in parallel
└───┬────┘ └───┬────┘ └─────┬────┘ │
│ │ │ │
└──────────┴────────────┴─────────┘
│
▼
┌────────┐
│ test │ ← Integration testing
└───┬────┘
│
┬─────────┴─────────┬
▼ ▼
┌───────────────┐ ┌──────────────┐
│dispatch-demo- │ │dispatch-demo-│
│ deployment │ │ canary │
└───────────────┘ └──────────────┘
```
## Workflow Jobs
### 1. detect-changes
**Purpose**: Analyzes commits to determine which services need rebuilding.
**Triggers on**:
- Direct pushes to `main` or `sirius-demo` branches
- Pull requests to `main`
- Repository dispatch events (submodule updates)
**Outputs**:
- `sirius_ui_changes`: Boolean indicating UI code changes
- `sirius_api_changes`: Boolean indicating API code changes
- `sirius_engine_changes`: Boolean indicating Engine code changes
- `submodule_changes`: Boolean indicating submodule updates
**Change detection logic**:
```yaml
# UI changes: Any file in sirius-ui/
# API changes: Any file in sirius-api/
# Engine changes: Any file in sirius-engine/ or rabbitmq/
# Global changes: Dockerfile, docker-compose, .github/ → rebuild all
```
### 2. build-ui (Parallel)
**Purpose**: Validates and builds the Next.js UI container.
**Runs when**: `detect-changes.outputs.sirius_ui_changes == 'true'`
**Steps**:
1. **Generate metadata**: Determines image tag (`latest`, `beta`, `pr-*`)
2. **Validate UI code**: Runs `npm ci && npm run lint`
3. **Set up Docker Buildx**: Configures multi-arch builds
4. **Log in to GHCR**: Authenticates with container registry
5. **Build and push**: Creates and pushes amd64/arm64 images
**Output**: `image_tag` (e.g., `latest`, `beta`, `pr-123`)
**Build platforms**: `linux/amd64`, `linux/arm64`
**Validation timing**: ~2-3 minutes (lint before Docker build)
### 3. build-api (Parallel)
**Purpose**: Validates and builds the Go API container.
**Runs when**: `detect-changes.outputs.sirius_api_changes == 'true'`
**Steps**:
1. **Generate metadata**: Determines image tag and submodule SHAs
2. **Set up Go**: Installs Go 1.24
3. **Validate API code**: Runs `go mod download && go test ./...`
4. **Set up Docker Buildx**: Configures multi-arch builds
5. **Log in to GHCR**: Authenticates with container registry
6. **Build and push**: Creates and pushes amd64/arm64 images with submodule refs
**Output**: `image_tag`
**Build args**: `GO_API_COMMIT_SHA` (for go-api submodule)
**Validation timing**: ~1-2 minutes (tests before Docker build)
### 4. build-engine (Parallel)
**Purpose**: Validates and builds the Go Engine container.
**Runs when**: `detect-changes.outputs.sirius_engine_changes == 'true'`
**Steps**:
1. **Generate metadata**: Determines image tag and all submodule SHAs
2. **Set up Go**: Installs Go 1.24
3. **Validate Engine code**: Runs `go mod download && go test ./...`
4. **Set up Docker Buildx**: Configures multi-arch builds
5. **Log in to GHCR**: Authenticates with container registry
6. **Build and push**: Creates and pushes amd64/arm64 images with all submodule refs
**Output**: `image_tag`
**Build args**:
- `GO_API_COMMIT_SHA`
- `APP_SCANNER_COMMIT_SHA`
- `APP_TERMINAL_COMMIT_SHA`
- `SIRIUS_NSE_COMMIT_SHA`
- `APP_AGENT_COMMIT_SHA`
**Validation timing**: ~1-2 minutes (tests before Docker build)
### 5. test
**Purpose**: Validates that built containers work together in an integrated environment.
**Runs when**: At least one build job succeeds
**Dependencies**: `[detect-changes, build-ui, build-api, build-engine]`
**Steps**:
1. **Determine image tag**: Uses output from whichever build job ran
2. **Create test environment**: Generates `docker-compose.test.yml` with fresh images
3. **Start infrastructure**: Launches Postgres, RabbitMQ, Valkey
4. **Start application services**: Launches only the services that were built
5. **Run smoke tests**: Validates services are running and responsive
6. **Cleanup**: Tears down test environment
**Test configuration**:
- Uses in-memory Postgres (`tmpfs`) for speed
- Isolated test database (`sirius_test`)
- Debug-level logging for troubleshooting
### 6. dispatch-demo-deployment
**Purpose**: Triggers deployment to the demo environment when `sirius-demo` branch updates.
**Runs when**: Push to `sirius-demo` branch after successful build + test
**Dependencies**: `[detect-changes, build-ui, build-api, build-engine, test]`
**Sends to**: `SiriusScan/sirius-demo` repository with event type `sirius-demo-updated`
**Payload includes**:
- Source repo/branch/SHA
- Triggering actor
- Commit message
### 7. dispatch-demo-canary
**Purpose**: Triggers demo rebuild on every `main` branch push as a deployment canary.
**Runs when**: Push to `main` branch after successful build + test
**Dependencies**: `[detect-changes, build-ui, build-api, build-engine, test]`
**Sends to**: `SiriusScan/sirius-demo` repository with event type `sirius-main-updated`
**Purpose of canary**: Catches bad commits to `main` by immediately deploying to demo environment
## Image Tagging Strategy
### Tag Types
**`latest`**:
- Pushed on every `main` branch commit
- Also tagged as `beta` simultaneously
- Used by default in `docker-compose.yaml`
**`beta`**:
- Alias for `latest` (same image)
- Explicitly labeled for beta testing
**`pr-{number}`**:
- Unique tag for pull request builds
- Enables testing PRs in isolation
**`dev`**:
- Fallback for other branches/events
### Tag Determination
```bash
# Pull request
TAG="pr-123"
# Push to main or repository_dispatch
TAG="latest"
also_tag_beta="true"
# Other events
TAG="dev"
```
## Prebuild Validation
Each build job runs validation **before** Docker builds to fail fast and save time.
### UI Validation
```bash
cd sirius-ui
npm ci # Install dependencies
npm run lint # ESLint validation
# Docker build only if lint succeeds
```
**Typical duration**: 2-3 minutes
**Catches**: Import errors, syntax issues, unused variables
### API Validation
```bash
cd sirius-api
go mod download # Download dependencies
go test ./... -v # Run tests
# Docker build only if tests pass
```
**Typical duration**: 1-2 minutes
**Catches**: Compilation errors, failing tests, import issues
### Engine Validation
```bash
cd sirius-engine
go mod download # Download dependencies
go test ./... -v # Run tests
# Docker build only if tests pass
```
**Typical duration**: 1-2 minutes
**Catches**: Compilation errors, failing tests, integration issues
## Timing Expectations
### Before Parallelization (Sequential Builds)
| Phase | Duration |
|-------|----------|
| Change detection | ~30s |
| Build UI | ~15-20 min |
| Build API | ~20-25 min |
| Build Engine | ~30-40 min |
| Test | ~5 min |
| **Total** | **~70-90 min** |
### After Parallelization (Current)
| Phase | Duration |
|-------|----------|
| Change detection | ~30s |
| Prebuild validation (all parallel) | ~2-3 min |
| Build UI, API, Engine (all parallel) | ~30-40 min (longest wins) |
| Test | ~5 min |
| **Total** | **~40-50 min** |
**Time savings**: ~40-60% reduction (30-40 minutes faster)
## Authentication & Secrets
### Required Secrets
**`GHCR_PUSH_USER`**: GitHub username that generated the PAT
**`GHCR_PUSH_TOKEN`**: GitHub Personal Access Token with scopes:
- `write:packages` (push images)
- `read:packages` (pull existing layers)
- `delete:packages` (cleanup, optional)
- `repo` (access repository context)
**`GITHUB_TOKEN`**: Automatically provided by GitHub Actions
- Used for repository dispatch events
- Has limited package permissions (read-only)
### Setting Up Secrets
```bash
# Repository secrets (Sirius repo)
Settings → Security → Secrets and variables → Actions
# Add GHCR_PUSH_USER (your GitHub username)
Name: GHCR_PUSH_USER
Value: your-username
# Add GHCR_PUSH_TOKEN (PAT with package scopes)
Name: GHCR_PUSH_TOKEN
Value: ghp_xxxxxxxxxxxx
```
## Troubleshooting
### Build Fails with "denied: installation not allowed"
**Cause**: Default `GITHUB_TOKEN` doesn't have package creation permission.
**Fix**: Use PAT-based authentication (already implemented with `GHCR_PUSH_USER`/`GHCR_PUSH_TOKEN`).
### Build Fails with "403 Forbidden" during push
**Cause**: PAT missing required scopes or package visibility is restricted.
**Fix**:
1. Verify PAT has `write:packages` and `read:packages` scopes
2. Check package is public: `https://github.com/orgs/SiriusScan/packages/container/sirius-{ui,api,engine}/settings`
3. Set visibility to Public (one-time change)
### Test job fails to pull images
**Cause**: Image tag mismatch or build job didn't complete.
**Fix**:
1. Check build job outputs: `needs.build-{ui,api,engine}.outputs.image_tag`
2. Verify images exist in GHCR: `https://github.com/orgs/SiriusScan/packages`
3. Check GHCR authentication in test job
### Dispatch jobs don't trigger downstream
**Cause**: Missing `GITHUB_TOKEN` permission or incorrect event type.
**Fix**:
1. Verify `GITHUB_TOKEN` has workflow permissions
2. Check downstream repo workflow listens for correct event type
3. Confirm payload format matches expectations
### Builds take longer than expected
**Expected timings** (parallel mode):
- Prebuild validation: 2-3 min
- Docker builds: 30-40 min (longest service)
- Integration test: 5 min
**Investigate if**:
- Build cache not working (check `cache-from: type=gha`)
- Network issues downloading dependencies
- Jobs running sequentially instead of parallel
## Best Practices
### When Modifying Workflows
1. **Test in PR first**: Use `pr-*` tags to test changes without affecting `latest`
2. **Preserve outputs**: Build jobs must output `image_tag` for downstream jobs
3. **Use `always()`**: Downstream jobs should use `always()` with result checks
4. **Validate locally**: Use `act` (GitHub Actions local runner) when possible
5. **Check syntax**: Run `actionlint` before committing
### Adding New Services
1. **Create build job**: Copy pattern from `build-ui/api/engine`
2. **Add change detection**: Update `detect-changes` job logic
3. **Update test job**: Add service to `docker-compose.test.yml`
4. **Update dispatch dependencies**: Include new job in `needs` array
### Optimizing Build Times
**Already implemented**:
- Parallel builds (saves 40-60%)
- Prebuild validation (fails fast)
- Docker layer caching (`cache-from: type=gha`)
- Multi-arch builds in one step
**Future opportunities**:
- Cache Go dependencies between runs
- Use `npm ci --prefer-offline` for UI builds
- Split test job into parallel service-specific tests
## Related Documentation
- **[README.docker-container-deployment.md](README.docker-container-deployment.md)**: Container registry deployment guide
- **[README.terraform-deployment.md](../operations/README.terraform-deployment.md)**: Terraform-based infrastructure deployment
- **[README.cicd.md](../architecture/README.cicd.md)**: CI/CD architecture overview
- **[README.docker-architecture.md](../architecture/README.docker-architecture.md)**: Docker multi-stage builds and architecture
## Additional Resources
- **GitHub Actions Docs**: https://docs.github.com/en/actions
- **Docker Buildx**: https://docs.docker.com/buildx/
- **GHCR Documentation**: https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry
documentation/dev/development/README.logging-conventions.md
+187
View File
@@ -0,0 +1,187 @@
---
title: "Logging Conventions"
description: "Project-wide logging standards, level guidelines, and configuration"
template: "TEMPLATE.documentation-standard"
llm_context: "high"
categories: ["development", "standards", "observability"]
tags: ["logging", "slog", "zap", "LOG_LEVEL", "structured-logging"]
related_docs:
- "README.development.md"
- "README.go-api-sdk.md"
- "README.docker-architecture.md"
search_keywords:
- "log level"
- "LOG_LEVEL"
- "slog"
- "zap"
- "structured logging"
- "debug"
- "verbosity"
---
# Logging Conventions
This document defines the project-wide logging standards for all Sirius services.
## Guiding Principles
1. **Structured logging only** -- all Go services use `log/slog` (SDK, sirius-api, app-scanner) or `go.uber.org/zap` (app-agent). Raw `log.Print*` and `fmt.Print*` are not used for service logging.
2. **Level-gated output** -- every service reads the `LOG_LEVEL` environment variable at startup. Only messages at or above the configured level are emitted.
3. **No sensitive data** -- connection strings, credentials, tokens, and full request/response bodies must never appear in logs.
4. **No debug dumps** -- printing entire structs (e.g., `%+v` on a host or vulnerability object) is prohibited. Log only the identifying fields needed for correlation (IP, ID, name).
5. **CLI output is separate** -- user-facing CLI tools (e.g., `template-cli`) use `fmt.Print*` for deliberate terminal output. This is distinct from service logging.
## LOG_LEVEL Configuration
### Supported Values
| Value | Description |
| ------- | -------------------------------------------------------- |
| `debug` | Everything, including per-host scan phases and cache ops |
| `info` | Startup, significant events, scan-level progress |
| `warn` | Non-fatal issues, degraded behavior, retries |
| `error` | Failures that affect results or require attention |
Default: **`info`** (when `LOG_LEVEL` is unset or unrecognized).
### Per-Environment Defaults
| Environment | `sirius-api` | `sirius-engine` (scanner) | `app-agent` |
| ------------------------------------------- | :----------: | :-----------------------: | :---------: |
| **Production** (`docker-compose.yaml`) | `error` | `info` | `info` |
| **Development** (`docker-compose.dev.yaml`) | `info` | `info` | `info` |
To temporarily enable debug output during development, override in `docker-compose.dev.yaml` or pass as an environment variable:
```bash
LOG_LEVEL=debug docker compose -f docker-compose.dev.yaml up
```
## Shared Initialization Helpers
### SDK (slog) -- `go-api/sirius/slogger`
Used by `sirius-api` and `app-scanner`. Call once at the top of `main()`:
```go
import "github.com/SiriusScan/go-api/sirius/slogger"
func main() {
slogger.Init() // configures slog.SetDefault() from LOG_LEVEL
// ...
}
```
Helper functions:
- `slogger.Level()` -- returns the current `slog.Level`
- `slogger.IsDebug()` -- returns true when debug logging is active
### app-agent (zap) -- `internal/config`
```go
import "github.com/SiriusScan/app-agent/internal/config"
func main() {
logger := config.NewLogger() // reads LOG_LEVEL, returns *zap.Logger
defer logger.Sync()
// ...
}
```
## Level Assignment Guidelines
### Debug
Use for output that is only useful when actively investigating a specific issue:
- Per-host scan phase progress (`Phase 0: Fingerprinting on 10.0.0.1`)
- Individual host submission confirmations
- Cache hit/miss details
- Queue message bodies
- HID generation, version detection
- Periodic flush counts
### Info
Use for meaningful, scan-level or service-level events:
- Service startup and configuration
- Scan started / scan completed (aggregate)
- Template resolution
- Significant state transitions
### Warn
Use when something unexpected happened but the service can continue:
- Failed to sync NSE scripts (will retry)
- Host appears to be down (skipping)
- Failed to update KV store with discovery data
- Deprecated configuration detected
### Error
Use when a requested operation failed:
- Failed to create KV store connection
- Invalid scan message (cannot be processed)
- Template not found
- Database query failures
- API submission failures
## Structured Key-Value Pairs
Always use structured fields instead of string interpolation:
```go
// Good
slog.Info("Scan completed", "scan_id", scanID, "hosts", hostCount, "vulns", vulnCount)
// Bad
slog.Info(fmt.Sprintf("Scan %s completed: %d hosts, %d vulns", scanID, hostCount, vulnCount))
```
### Common Field Names
Use consistent field names across the project:
| Field | Description |
| ------------- | ------------------- |
| `error` | Error value |
| `ip` | Host IP address |
| `scan_id` | Scan identifier |
| `host_count` | Number of hosts |
| `template_id` | Template identifier |
| `queue` | Queue name |
| `duration` | Operation duration |
| `port_count` | Number of ports |
| `script_id` | Script identifier |
## Anti-Patterns
### Do Not
- Use `log.Panicln` or `log.Fatalf` in HTTP handlers (return proper error responses instead)
- Log entire structs with `%+v` or `%#v`
- Log connection strings or credentials
- Use emoji in structured log messages (level already conveys severity)
- Use `fmt.Printf` for service-level logging (only for CLI user output)
- Create per-request loggers unless adding request-scoped context
### Exceptions
- `log.Fatalf` is acceptable in `main()` for unrecoverable startup errors (e.g., cannot connect to database)
- `fmt.Print*` is correct for CLI tools that produce user-facing terminal output
- Test files may use `t.Log` / `t.Logf` freely
## Migration Checklist
When working on a Go file that still uses `log.Print*` or `fmt.Printf` for logging:
1. Replace `"log"` import with `"log/slog"`
2. Convert `log.Printf("message: %s", val)` to `slog.Info("message", "key", val)`
3. Choose the correct level (see guidelines above)
4. Remove any `%+v` struct dumps
5. Remove emoji from log messages
6. Verify no sensitive data is logged
documentation/dev/development/README.sirius-event-log.md
+53
View File
@@ -0,0 +1,53 @@
---
title: "Sirius Event Log System"
description: "Documentation for the Sirius event logging system and event management"
template: "TEMPLATE.documentation-standard"
llm_context: "medium"
categories: ["development", "logging"]
tags: ["events", "logging", "monitoring"]
related_docs:
- "README.system-monitor.md"
- "README.architecture.md"
---
# Sirius Event Log System
This document describes the Sirius event logging system and how events are managed throughout the application.
## Overview
The Sirius event log system provides comprehensive event tracking and logging capabilities for monitoring system activities, user actions, and system events.
## Event Types
- **System Events**: Container health, service status, resource usage
- **User Events**: Authentication, authorization, user actions
- **Application Events**: API calls, database operations, external service interactions
- **Security Events**: Failed login attempts, permission violations, suspicious activities
## Event Structure
Events follow a standardized format with:
- Timestamp
- Event type and category
- Source service/component
- Event data and metadata
- Severity level
## Storage and Retrieval
Events are stored in Valkey for fast access and retrieval, with configurable retention policies.
## Integration
The event log system integrates with:
- System monitoring dashboard
- Log viewer components
- Alerting and notification systems
- Audit and compliance reporting
## Configuration
[Configuration details to be documented]
documentation/dev/development/README.ui-style-guide.md
+1104
View File
File diff suppressed because it is too large Load Diff
documentation/dev/operations/README.api-key-operations.md
+118
View File
@@ -0,0 +1,118 @@
---
title: "API Key Operations Runbook"
description: "Production runbook for stateless service API key rotation, recovery, and incident response."
template: "TEMPLATE.guide"
llm_context: "high"
categories: ["operations", "security", "deployment"]
tags: ["apikey", "rotation", "incident-response", "valkey", "runbook"]
related_docs:
- "README.development.md"
- "README.docker-container-deployment.md"
- "README.auth-surface-matrix.md"
---
# API Key Operations Runbook
This runbook defines production-safe operations for Sirius service API keys.
## Scope
Applies to service-to-service credential `SIRIUS_API_KEY` used by:
- `sirius-api`
- `sirius-ui` (server-side tRPC -> API calls)
- `sirius-engine` and agent/scanner API clients
## Baseline Invariants
- `SIRIUS_API_KEY` must be non-empty in production.
- Go API must reject requests without a valid `X-API-Key`.
- Only `/health` is public.
- Root service key validation is stateless from runtime environment configuration.
- Valkey stores only dynamic/user-generated API key records.
## Rotation Procedure (No Downtime)
### 1) Create new key
1. Authenticate as admin in UI.
2. Create a new API key in Settings.
3. Record:
- key value (shown once)
- key id/hash
- label
### 2) Deploy new key to services
1. Update deployment secret store or environment value for `SIRIUS_API_KEY` with the new key.
2. Roll out services in order:
1. `sirius-api`
2. `sirius-ui`
3. `sirius-engine`
3. Validate:
- API 401 rate does not spike.
- UI can read/write through tRPC.
- scanner/agent host submissions continue.
### 3) Revoke old key
1. After rollout and validation period, revoke old key in UI.
2. Confirm old key returns 401.
3. Close rotation ticket.
## Incident: Valkey Data Loss
Symptoms:
- sudden 401s for user-generated keys
- API key list empty
- dynamic key operations fail while root key requests still succeed
Recovery:
1. Restore Valkey from backup if available.
2. If no backup:
- ensure production `SIRIUS_API_KEY` is correctly set for all services.
- restart `sirius-api` first; root key path remains stateless.
3. Restart `sirius-ui` and `sirius-engine`.
4. Validate end-to-end operations and security harness results.
## Incident: Root Key Mismatch (Configuration Drift)
Symptoms:
- services return 401 with old key after key rotation
- env values differ between `sirius-ui`, `sirius-api`, and `sirius-engine`
Recovery:
1. Confirm deployed `SIRIUS_API_KEY` value in runtime environment.
2. Restart `sirius-api`, then `sirius-ui` and `sirius-engine`.
3. Re-check key validation success for deployed root key.
4. If still failing, capture API logs and run security harness `auth-surface` and `api` suites.
## Incident: Unauthorized Agent/Scanner API Writes
Symptoms:
- scanner/agent host submissions return 401
Recovery:
1. Verify `SIRIUS_API_KEY` is present in engine/scanner/agent runtime.
2. Verify clients send `X-API-Key`.
3. Validate key exists in API key list and is not revoked.
4. Roll service restart after secret correction.
## Validation Commands
Run security checks from repository root:
```bash
cd testing/security
go run . --suite api
go run . --suite trpc
go run . --suite auth-surface
```
## Operational Checklist
- [ ] `SIRIUS_API_KEY` secret exists and is non-empty in production.
- [ ] API middleware accepts configured root key on non-health routes.
- [ ] UI + engine are using the same active service key.
- [ ] Old keys are revoked only after rollout validation.
- [ ] Recovery procedure for config drift or Valkey loss is documented in incident ticket.
documentation/dev/operations/README.git-operations.md
+1000
View File
File diff suppressed because it is too large Load Diff
documentation/dev/operations/README.maintainer-ops-issue-review.md
+125
View File
@@ -0,0 +1,125 @@
---
title: "Maintainer Issue Review System"
description: "Issue and PR triage taxonomy, lifecycle, and ChatOps conventions for mobile-friendly Sirius maintenance."
template: "TEMPLATE.documentation-standard"
version: "1.0.0"
last_updated: "2026-02-22"
author: "Sirius Team"
tags: ["operations", "triage", "chatops", "github", "labels"]
categories: ["operations", "development"]
difficulty: "intermediate"
prerequisites: ["github", "github-actions", "sirius-workflows"]
related_docs:
- "README.git-operations.md"
- "README.api-key-operations.md"
- "../deployment/README.workflows.md"
- "../test/CHECKLIST.testing-by-type.md"
dependencies:
- ".github/labels.yml"
- ".github/workflows/issue-triage.yml"
- ".github/workflows/chatops-runner.yml"
llm_context: "high"
search_keywords: ["triage", "labels", "issue forms", "chatops", "slash commands", "mobile maintainer"]
---
# Maintainer Issue Review System
## Purpose
Define a deterministic, evidence-driven issue and PR review workflow that works well from mobile and minimizes maintainer back-and-forth.
## Label Taxonomy
### Status Labels (single active status)
- `status:needs-triage`
- `status:needs-info`
- `status:repro-ready`
- `status:confirmed`
- `status:in-progress`
- `status:blocked`
- `status:ready-to-merge`
- `status:done`
### Type Labels
- `type:bug`
- `type:enhancement`
- `type:security`
- `type:docs`
- `type:question`
### Area Labels
- `area:installer-secrets`
- `area:compose-dev`
- `area:compose-prod`
- `area:ui`
- `area:api`
- `area:engine`
- `area:rabbitmq`
- `area:postgres`
- `area:valkey`
- `area:auth`
### Severity Labels
- `sev:critical`
- `sev:high`
- `sev:medium`
- `sev:low`
## Issue Intake
Issue forms are required and live under `.github/ISSUE_TEMPLATE/`.
Current form set:
- `install-startup.yml`
- `auth-401.yml`
- `dev-overlay.yml`
- `windows-wsl.yml`
- `security-report.yml`
Intake quality baseline:
- run mode (standard/dev/prod overlay)
- runtime version/tag
- host OS + Docker version
- compose render validation outcome
- key service logs
- minimal reproduction steps
## Triage and ChatOps Lifecycle
1. New issue receives `status:needs-triage`.
2. Triage automation applies `type:*`, `area:*`, and severity hints from deterministic rules.
3. Triage Card comment is posted with missing-info checks and recommended commands.
4. Maintainer drives progression from comments:
- `/triage needs-info`
- `/triage repro-ready`
- `/triage confirmed`
5. Maintainer triggers evidence workflows from comments:
- `/test health`
- `/test integration`
- `/test security <suite>`
## PR Review Policy
PRs are path-labeled (`area:*`) and receive an automated review card summarizing:
- changed surfaces and risk zones
- recommended test checklist slices
- mobile-safe slash commands for additional validation
Required test guidance is sourced from `documentation/dev/test/CHECKLIST.testing-by-type.md`.
## Operational Notes
- Deterministic rules decide labels/states; AI summaries are optional and advisory.
- Security and confirmed issues are exempt from stale auto-close.
- Health and integration evidence should be posted in-thread for auditability.
---
_This document follows the Sirius Documentation Standard. For documentation governance, see `documentation/dev/ABOUT.documentation.md`._
documentation/dev/operations/README.new-project.md
+395
View File
@@ -0,0 +1,395 @@
---
title: "New Project Development Workflow"
description: "Structured approach for starting new development sprints with consistent project organization, task management, and Git workflows."
template: "TEMPLATE.documentation-standard"
version: "1.0.0"
last_updated: "2025-01-03"
author: "Development Team"
tags: ["project-management", "workflow", "git", "tasks", "development"]
categories: ["operations", "development"]
difficulty: "beginner"
prerequisites: ["git", "cursor", "project-structure"]
related_docs:
- "README.tasks.md"
- "README.git-operations.md"
dependencies: []
llm_context: "high"
search_keywords:
[
"new project",
"development sprint",
"task management",
"git workflow",
"project structure",
]
---
# New Project Development Workflow
> **📚 Task Management**: For detailed task system usage, see [README.tasks.md](README.tasks.md)
## Purpose
This document defines the standardized workflow for starting new development sprints, ensuring consistent project organization, task tracking, and Git practices across all development efforts.
## When to Use This Workflow
- **Starting any new development sprint** - Follow this process for every new project
- **Major feature development** - Use for significant new features or enhancements
- **Bug fix campaigns** - Apply for systematic bug fixing efforts
- **Infrastructure changes** - Use for deployment, CI/CD, or architectural work
- **Research and prototyping** - Apply for exploratory development work
## Project Structure Requirements
### Required Files for Every Project
Every new development sprint must create these files in the specified locations:
#### 1. Task Definition File
- **Location**: `tasks/{sprint-name}.json`
- **Purpose**: Detailed task breakdown and tracking
- **Format**: JSON array following established schema
- **Example**: `tasks/cicd.json` (see template below)
#### 2. Project Plan Document
- **Location**: `documentation/dev-notes/{sprint-name}-plan.md`
- **Purpose**: High-level project overview and context
- **Format**: Markdown with YAML front matter
- **Content**: Project goals, scope, timeline, and key decisions
#### 3. Cleanup Task
- **Requirement**: Every project must include a final cleanup task
- **Purpose**: Remove temporary files, update documentation, merge branches
- **ID Pattern**: `{last-id + 1}` (e.g., if last task is 5, cleanup is 6)
## Task File Template
Use this template structure for all `tasks/{sprint-name}.json` files:
```json
[
{
"id": "0",
"title": "PHASE 0: Project Foundation",
"description": "Brief description of the phase",
"details": "Detailed implementation notes and expected outputs",
"status": "pending",
"priority": "high",
"dependencies": [],
"subtasks": [
{
"id": "0.1",
"title": "Specific Task Title",
"description": "What this task accomplishes",
"details": "Implementation details, acceptance criteria, and context",
"status": "pending",
"priority": "high",
"dependencies": [],
"testStrategy": "How to verify this task is complete"
}
]
},
{
"id": "1",
"title": "PHASE 1: Core Implementation",
"description": "Main development work",
"details": "Key deliverables and technical approach",
"status": "pending",
"priority": "high",
"dependencies": ["0"],
"subtasks": [
{
"id": "1.1",
"title": "Implementation Task",
"description": "Specific implementation work",
"details": "Technical details and requirements",
"status": "pending",
"priority": "high",
"dependencies": ["0.1"],
"testStrategy": "Verification approach"
}
]
},
{
"id": "2",
"title": "PHASE 2: Cleanup & Documentation",
"description": "Project completion and cleanup",
"details": "Final tasks to close out the project",
"status": "pending",
"priority": "medium",
"dependencies": ["1"],
"subtasks": [
{
"id": "2.1",
"title": "Clean Up Project Files",
"description": "Remove temporary files and update documentation",
"details": "Delete temporary files, update README if needed, ensure all documentation is current",
"status": "pending",
"priority": "medium",
"dependencies": ["1.1"],
"testStrategy": "Verify no temporary files remain and documentation is up to date"
}
]
}
]
```
### Task Schema Requirements
- **id**: Unique identifier (string, e.g., "0", "1.1", "2.3")
- **title**: Clear, descriptive task name
- **description**: Brief summary of what the task accomplishes
- **details**: Comprehensive implementation notes and context
- **status**: "pending", "in_progress", "done", "blocked"
- **priority**: "high", "medium", "low"
- **dependencies**: Array of task IDs this task depends on
- **subtasks**: Array of subtask objects (for main tasks)
- **testStrategy**: How to verify task completion (for subtasks)
## Git Workflow for New Projects
### Standard Branching Strategy
For every new development sprint:
#### 1. Create Feature Branch
```bash
# Start from main branch
git checkout main
git pull origin main
# Create new branch for the sprint
git checkout -b feature/{sprint-name}
# Example: git checkout -b feature/agent-enhancements
```
#### 2. Branch Naming Convention
- **Format**: `feature/{sprint-name}`
- **Examples**:
- `feature/agent-enhancements`
- `feature/ci-cd-pipeline`
- `feature/ui-redesign`
- `feature/database-optimization`
#### 3. Work on Branch
- Make all commits on the feature branch
- Use descriptive commit messages
- Commit frequently with atomic changes
- Push branch regularly to backup work
#### 4. Merge Back to Main
```bash
# When sprint is complete
git checkout main
git pull origin main
git merge feature/{sprint-name}
git push origin main
# Clean up feature branch
git branch -d feature/{sprint-name}
git push origin --delete feature/{sprint-name}
```
### Exception Cases
**When NOT to create a feature branch:**
- Hotfixes requiring immediate deployment
- Documentation-only changes
- Minor configuration updates
- Emergency security patches
**Alternative approaches:**
- **Hotfixes**: Use `hotfix/{issue-description}` branch
- **Documentation**: Can be committed directly to main
- **Config changes**: Use `config/{change-description}` branch
## Project Plan Document Template
Create `documentation/dev-notes/{sprint-name}-plan.md` with this structure:
```markdown
---
title: "{Sprint Name} - Project Plan"
description: "High-level project overview and implementation strategy"
template: "TEMPLATE.documentation-standard"
version: "1.0.0"
last_updated: "2025-01-03"
author: "Development Team"
tags: ["project-plan", "sprint", "{sprint-name}"]
categories: ["development", "planning"]
difficulty: "intermediate"
prerequisites: []
related_docs:
- "README.tasks.md"
dependencies: []
llm_context: "medium"
search_keywords: ["{sprint-name}", "project-plan", "development"]
---
# {Sprint Name} - Project Plan
## Project Overview
**Goal**: [Primary objective of this sprint]
**Timeline**: [Expected duration]
**Scope**: [What's included and excluded]
## Key Deliverables
1. [Primary deliverable 1]
2. [Primary deliverable 2]
3. [Primary deliverable 3]
## Technical Approach
[High-level technical strategy and key decisions]
## Success Criteria
- [ ] [Measurable success criterion 1]
- [ ] [Measurable success criterion 2]
- [ ] [Measurable success criterion 3]
## Risk Assessment
**High Risk Items:**
- [Risk 1]: [Mitigation strategy]
- [Risk 2]: [Mitigation strategy]
**Dependencies:**
- [External dependency 1]
- [External dependency 2]
## Notes
[Any additional context, decisions, or considerations]
```
## Development Workflow Steps
### 1. Project Initialization
```bash
# 1. Create feature branch
git checkout main
git pull origin main
git checkout -b feature/{sprint-name}
# 2. Create task file
touch tasks/{sprint-name}.json
# [Fill in task structure using template]
# 3. Create project plan
touch documentation/dev-notes/{sprint-name}-plan.md
# [Fill in project plan using template]
# 4. Initial commit
git add tasks/{sprint-name}.json documentation/dev-notes/{sprint-name}-plan.md
git commit -m "feat: initialize {sprint-name} project
- Add task breakdown in tasks/{sprint-name}.json
- Add project plan in documentation/dev-notes/{sprint-name}-plan.md
- Create feature branch for development"
git push origin feature/{sprint-name}
```
### 2. Development Process
- **Start each work session** by reviewing current tasks
- **Update task status** as work progresses
- **Commit frequently** with descriptive messages
- **Reference task IDs** in commit messages when relevant
- **Ask for help** when blocked or uncertain
### 3. Project Completion
```bash
# 1. Complete all tasks (including cleanup)
# 2. Update project plan with final status
# 3. Merge to main
git checkout main
git pull origin main
git merge feature/{sprint-name}
git push origin main
# 4. Clean up
git branch -d feature/{sprint-name}
git push origin --delete feature/{sprint-name}
```
## Best Practices
### Task Management
- **Keep tasks small** - Each subtask should be completable in 1-4 hours
- **Be specific** - Clear acceptance criteria and test strategies
- **Update status** - Mark tasks as done immediately when completed
- **Use dependencies** - Properly sequence related tasks
### Git Practices
- **Atomic commits** - Each commit should represent one logical change
- **Descriptive messages** - Use conventional commit format
- **Regular pushes** - Backup work frequently
- **Clean history** - Squash commits if needed before merging
### Documentation
- **Keep plans current** - Update project plans as scope changes
- **Document decisions** - Record important technical decisions
- **Include context** - Help future developers understand choices
## Troubleshooting
### Common Issues
**Task file not found:**
- Verify file is in `tasks/` directory
- Check filename matches sprint name
- Ensure JSON syntax is valid
**Branch conflicts:**
- Pull latest main before creating feature branch
- Rebase feature branch if main has moved significantly
- Resolve conflicts before merging
**Missing project plan:**
- Create plan document in `documentation/dev-notes/`
- Use template structure provided
- Include all required sections
### Getting Help
- **Task questions**: Reference [README.tasks.md](README.tasks.md)
- **Git issues**: Reference [README.git-operations.md](README.git-operations.md)
- **General help**: Ask for clarification on requirements or approach
## Integration with Cursor Rules
This workflow integrates with our Cursor rules system:
- **Task context**: Cursor will include task files in context when working on projects
- **Git operations**: Cursor will follow branching and commit conventions
- **Documentation**: Cursor will maintain project plan and task documentation
- **Code quality**: Cursor will follow established coding standards
---
_This workflow ensures consistent, organized development practices across all projects. For questions about specific aspects, see the related documentation files._
documentation/dev/operations/README.release-closeout-v1.0.0.md
+85
View File
@@ -0,0 +1,85 @@
---
title: "Sirius v1.0.0 Release Closeout"
description: "Final push, merge, validation, and launch closeout evidence for the Sirius v1.0.0 major release."
template: "TEMPLATE.guide"
llm_context: "high"
categories: ["operations", "release", "deployment"]
tags: ["release", "v1.0.0", "closeout", "validation", "rollout"]
related_docs:
- "README.git-operations.md"
- "README.workflows.md"
- "README.docker-container-deployment.md"
- "README.api-key-operations.md"
- "../../OPERATIONS.md"
---
# Sirius v1.0.0 Release Closeout
This document records final push/merge completion evidence for the Sirius v1.0.0 major release.
## Launch References
- PR merge record: https://github.com/SiriusScan/Sirius/pull/90
- Release page: https://github.com/SiriusScan/Sirius/releases/tag/v1.0.0
- Launch closeout issue: https://github.com/SiriusScan/Sirius/issues/93
- Stabilization backlog: https://github.com/SiriusScan/Sirius/issues/92
## Release Integrity Evidence
- PR `#90` status: merged into `main` (`v1` -> `main`)
- Merge commit: `014c89b441ab697654afb205e1960dcef74bfac3`
- Main release commit: `221b8001de3a580b7741f912cffa1ab8199d63e0`
- Main repo tag `v1.0.0` points to release commit `221b8001de3a580b7741f912cffa1ab8199d63e0`
- Release notes published (non-draft, non-prerelease)
## Distribution Gate Evidence
### Container image availability
- `ghcr.io/siriusscan/sirius-ui:latest` available
- `ghcr.io/siriusscan/sirius-api:latest` available
- `ghcr.io/siriusscan/sirius-engine:latest` available
- At initial closeout, `v1.0.0` semver tags were not yet consistently published on GHCR for all stack images (historical `manifest unknown` on some application images; infrastructure images require the same [Publish Release Image Tags](../../../.github/workflows/publish-release-image-tags.yml) step as the app trio).
Note: direct retag-and-push attempt from local environment was blocked by GHCR token scope restrictions. Versioned tags for operators are published via the **Publish Release Image Tags** workflow using CI credentials.
### Post-closeout: third-party pulls and issue #119
External operators who set `IMAGE_TAG=v1.0.0` (or any semver) need **six** matching tags on GHCR and **public** package visibility. If any image lacks the tag or a package remains private, `docker compose pull` fails with `not found` / `manifest unknown` while maintainer machines may still succeed when authenticated to GHCR ([issue #119](https://github.com/SiriusScan/Sirius/issues/119)).
**Remediation (maintainers):** follow the [GHCR distribution checklist](../../../OPERATIONS.md#maintainer-ghcr-distribution-checklist-public-operators): set each org package to Public, run **Publish Release Image Tags** with `source_tag=latest` and `target_tag=v1.0.0`, then confirm anonymously with `bash scripts/verify-ghcr-public-access.sh v1.0.0`. The **Verify GHCR Release Tag** workflow ([verify-ghcr-release-tag.yml](../../../.github/workflows/verify-ghcr-release-tag.yml)) runs the same check on every published release and weekly.
**Suggested reply for issue #119 (after tags and visibility are fixed):** confirm all six anonymous pulls for `v1.0.0`, link the successful workflow run, and recommend leaving `IMAGE_TAG` blank for `latest` unless pinning.
### Runtime smoke validation
Release stack boot with required `SIRIUS_API_KEY` succeeded.
- `docker compose ps`: all core services healthy (`sirius-ui`, `sirius-api`, `sirius-engine`, `sirius-postgres`, `sirius-rabbitmq`, `sirius-valkey`)
- API health: `GET http://localhost:9001/health` returns healthy JSON with `version: "1.0.0"`
- UI health proxy: `GET http://localhost:3000/` returns `200 OK`
- Engine gRPC listener: `localhost:50051` reachable
## Repo Hygiene and Freeze
- Main repo working tree is clean except local planning artifacts under `.cursor/plans/`.
- Freeze policy for immediate post-release window:
- no feature merges to `main`
- only release-blocking hotfixes linked to stabilization backlog issue
- all hotfixes require rollback notes and validation evidence
## Cross-Repo Release Matrix
| Repository | Tag | Commit |
| --- | --- | --- |
| Sirius (main) | `v1.0.0` | `221b8001de3a580b7741f912cffa1ab8199d63e0` |
| app-scanner | `v1.0.0` | `7821c8ea73093f7fbc89bf4749e8e07a7c76f499` |
## Rollback Reference
If rollback is required:
1. redeploy previous known-good image tags in compose inputs
2. restart stack and confirm service health checks (`/health`, UI `200`, gRPC port)
3. capture incident details and owner in stabilization backlog issue
4. roll forward with a hotfix PR after validation
documentation/dev/operations/README.sdk-releases.md
+974
View File
@@ -0,0 +1,974 @@
---
title: "SDK Release Process"
description: "Comprehensive guide for releasing new versions of the Sirius Go API SDK and updating dependent projects"
template: "TEMPLATE.documentation-standard"
llm_context: "high"
categories: ["operations", "sdk", "releases"]
tags: ["go-api", "versioning", "github-actions", "ci-cd", "deployment"]
related_docs:
- "../architecture/README.go-api-sdk.md"
- "README.development.md"
- "../../../minor-projects/go-api/README.md"
search_keywords: ["sdk", "release", "version", "deployment", "go-api", "dependencies", "breaking-changes"]
---
# SDK Release Process
## Overview
This document describes how to release new versions of the **Sirius Go API SDK** (`github.com/SiriusScan/go-api`) and propagate updates to dependent projects.
**Key Concepts:**
- **Automated releases** via GitHub Actions CI/CD
- **Semantic versioning** for version management
- **Dependency updates** across multiple projects
- **Breaking change communication** via CHANGELOG
## Quick Reference
### Release New SDK Version
```bash
# Automatic release (recommended)
cd /path/to/go-api
git add .
git commit -m "feat: add new feature"
git push origin main
# → CI/CD creates new patch version automatically
# Manual version bump
git tag v0.0.12
git push origin v0.0.12
# → CI/CD creates GitHub release
```
### Update Dependent Project
```bash
cd /path/to/dependent-project
# Update go.mod version
# Change: github.com/SiriusScan/go-api v0.0.10
# To: github.com/SiriusScan/go-api v0.0.11
go mod tidy
go build ./... # Verify build
go test ./... # Run tests
git commit -am "chore: update go-api SDK to v0.0.11"
```
## Automated Release Process (Recommended)
### How It Works
The go-api repository has a GitHub Actions workflow (`.github/workflows/ci.yml`) that:
1. **Triggers on push to `main` branch**
2. **Runs tests and linting**
3. **Auto-increments patch version** (v0.0.10 → v0.0.11)
4. **Creates Git tag**
5. **Generates GitHub release** with auto-generated notes
6. **Notifies main Sirius repo** via repository_dispatch event
### Workflow File
**Location:** `go-api/.github/workflows/ci.yml`
**Jobs:**
- `test-and-lint`: Runs tests and linters
- `release`: Creates new version and release (only on main push)
- `notify-main-repo`: Sends notification to dependent repos
### Step-by-Step Process
#### Step 1: Make Changes to SDK
```bash
cd /path/to/go-api
git checkout -b feature/my-changes
# Make your changes
# - Edit code files
# - Add tests
# - Update documentation
# Test locally
go test ./...
go build ./...
```
#### Step 2: Update CHANGELOG.md
**Always update CHANGELOG.md** before releasing:
```markdown
## [Unreleased]
### Added
- New feature description
### Fixed
- Bug fix description
### Changed
- Modification description
### BREAKING CHANGE (if applicable)
- Description of breaking change
- Migration guide
```
**Format:** Follow [Keep a Changelog](https://keepachangelog.com/) standard
#### Step 3: Commit and Push
```bash
# Stage changes
git add .
# Commit with conventional commit message
git commit -m "feat: add new feature
Detailed description of changes...
BREAKING CHANGE: Field renamed
- OldField → NewField
- Update code to use NewField"
# Push to trigger CI/CD
git push origin main
```
#### Step 4: Monitor CI/CD
1. **Check GitHub Actions**: Go to `github.com/SiriusScan/go-api/actions`
2. **Wait for workflows to complete**:
- ✅ `test-and-lint` must pass
- ✅ `release` creates new tag
- ✅ `notify-main-repo` sends notification
3. **Verify new release**: Check `github.com/SiriusScan/go-api/releases`
#### Step 5: Verify Release
```bash
# Fetch new tags
git fetch --tags
# Check latest tag
git tag -l "v*" --sort=-version:refname | head -1
# Should show new version (e.g., v0.0.11)
# View release on GitHub
# https://github.com/SiriusScan/go-api/releases/latest
```
**Expected Results:**
- ✅ New Git tag created (e.g., `v0.0.11`)
- ✅ GitHub release with auto-generated notes
- ✅ Release includes commit history since last version
- ✅ CI/CD tests passed
## Manual Release Process
### When to Use
- **Major/minor version bumps** (e.g., v0.0.x → v0.1.0)
- **Pre-release versions** (e.g., v0.1.0-beta.1)
- **CI/CD failures** (automated release fails)
- **Hotfix releases** (skip normal workflow)
### Step 1: Determine New Version
**Semantic Versioning Rules:**
- **MAJOR** (x.0.0): Breaking changes, incompatible API
- **MINOR** (0.x.0): New features, backward compatible
- **PATCH** (0.0.x): Bug fixes, backward compatible
**Examples:**
- `v0.0.10` → `v0.0.11` (bug fix)
- `v0.0.11` → `v0.1.0` (new feature, major milestone)
- `v0.1.0` → `v1.0.0` (stable API, breaking changes)
### Step 2: Create Tag
```bash
cd /path/to/go-api
# Ensure main is up to date
git checkout main
git pull origin main
# Create annotated tag
git tag -a v0.1.0 -m "Release v0.1.0
New Features:
- Feature 1
- Feature 2
Bug Fixes:
- Fix 1
- Fix 2"
# Push tag
git push origin v0.1.0
```
### Step 3: Create GitHub Release
**Option A: Automatic (via CI/CD)**
- Pushing the tag triggers `release` workflow
- GitHub release auto-generated
**Option B: Manual**
1. Go to `github.com/SiriusScan/go-api/releases/new`
2. Select tag: `v0.1.0`
3. Set title: `v0.1.0`
4. Add release notes (copy from CHANGELOG.md)
5. Click "Publish release"
### Step 4: Verify
```bash
# Check tag exists
git ls-remote --tags origin | grep v0.1.0
# View release
curl -s https://api.github.com/repos/SiriusScan/go-api/releases/latest | jq .tag_name
```
## Updating Dependent Projects
### Identifying Dependents
**Known projects using go-api:**
- `app-scanner` - Port scanning engine
- `app-agent` - Agent management system
- `sirius-api` (via container) - REST API server
- `sirius-engine` (via container) - Core scanning engine
**Find all dependents:**
```bash
cd /path/to/Sirius-Project
grep -r "github.com/SiriusScan/go-api" . --include="go.mod"
```
### Update Process for Each Project
#### Step 1: Update go.mod
**File:** `project/go.mod`
**Before:**
```go
module github.com/SiriusScan/app-scanner
replace github.com/SiriusScan/go-api => ../go-api //Development
require (
github.com/SiriusScan/go-api v0.0.10 // Old version
// ... other dependencies
)
```
**After:**
```go
module github.com/SiriusScan/app-scanner
replace github.com/SiriusScan/go-api => ../go-api //Development
require (
github.com/SiriusScan/go-api v0.0.11 // New version
// ... other dependencies
)
```
**Notes:**
- Keep `replace` directive for local development
- Update only the version number in `require`
#### Step 2: Download Dependencies
```bash
cd /path/to/dependent-project
# Update dependencies
go mod tidy
# Verify go.sum updated
git diff go.sum
```
#### Step 3: Check for Breaking Changes
**Read CHANGELOG.md** in go-api:
```bash
cat ../go-api/CHANGELOG.md
```
**Look for:**
- `BREAKING CHANGE:` sections
- Renamed fields/functions
- Removed functionality
- Schema changes
#### Step 4: Update Code
**Example: Port.ID → Port.Number breaking change**
**Find all references:**
```bash
grep -rn "port\.ID" . --include="*.go"
```
**Update code:**
```go
// Before (v0.0.10)
ports := make([]int, len(host.Ports))
for i, port := range host.Ports {
ports[i] = port.ID // ❌ Field removed
}
// After (v0.0.11)
ports := make([]int, len(host.Ports))
for i, port := range host.Ports {
ports[i] = port.Number // ✅ New field name
}
```
#### Step 5: Test Build
```bash
# Build project
go build -o /dev/null .
# If build fails, fix compilation errors
# Usually related to renamed/removed fields
```
#### Step 6: Run Tests
```bash
# Run unit tests
go test ./...
# Run integration tests (if applicable)
go test ./... -tags=integration
```
#### Step 7: Commit Changes
```bash
git add go.mod go.sum
# If code changes needed
git add path/to/changed/files.go
# Commit
git commit -m "chore: update go-api SDK to v0.0.11
Updates for go-api v0.0.11 breaking changes:
- Fixed port.ID references to port.Number
- Updated dependency version
- Verified build and tests pass
Breaking changes in go-api v0.0.11:
- Port.ID renamed to Port.Number
- See go-api CHANGELOG.md for details"
git push origin main
```
### Container-Based Projects
**Projects:** `sirius-api`, `sirius-engine`
These projects run in Docker containers and use **replace directives with volume mounts**.
#### docker-compose.dev.yaml Configuration
```yaml
services:
sirius-engine:
volumes:
- ../minor-projects/go-api:/go-api:ro
- ../minor-projects/app-scanner:/app-scanner
```
#### go.mod in Container
```go
replace github.com/SiriusScan/go-api => /go-api
```
#### Update Process
**Changes to go-api are automatically visible** in containers due to volume mount. No version update needed during development.
**For production builds:**
1. Update go.mod version (same as above)
2. Rebuild Docker images
3. Deploy new images
**Restart containers to reload code:**
```bash
docker compose -f docker-compose.dev.yaml restart sirius-engine
docker compose -f docker-compose.dev.yaml restart sirius-api
```
## Breaking Changes
### Definition
**A breaking change is any modification that requires code changes in dependent projects.**
**Examples:**
- ✅ Renaming exported fields (e.g., `Port.ID` → `Port.Number`)
- ✅ Changing function signatures
- ✅ Removing exported types/functions
- ✅ Changing return types
- ✅ Database schema changes (non-backward-compatible)
- ❌ Adding new fields (backward compatible)
- ❌ Adding new functions (backward compatible)
- ❌ Internal implementation changes (not breaking)
### Communicating Breaking Changes
#### 1. Commit Message
Use **conventional commit format** with `BREAKING CHANGE:` footer:
```
feat: rename Port.ID to Port.Number
This resolves conflict with GORM auto-increment ID field.
BREAKING CHANGE: Port.ID renamed to Port.Number
- Update all references from port.ID to port.Number
- Port numbers now stored in Number field
- Database migration required (see migrations/005)
```
#### 2. CHANGELOG.md
Document in CHANGELOG with clear migration guide:
```markdown
## [0.0.11] - 2025-10-26
### Fixed
#### 🔴 CRITICAL: Port.ID Schema Conflict
- **BREAKING CHANGE:** Renamed `Port.ID` field to `Port.Number`
- `Port.ID int` → `Port.Number int`
- Resolves duplicate key violations
### Upgrading
**Before:**
```go
port := sirius.Port{ID: 22, Protocol: "tcp"}
```
**After:**
```go
port := sirius.Port{Number: 22, Protocol: "tcp"}
```
```
#### 3. GitHub Release Notes
Include breaking changes prominently:
```markdown
# v0.0.11
## ⚠️ BREAKING CHANGES
### Port.ID → Port.Number
The `Port.ID` field has been renamed to `Port.Number` to resolve database conflicts.
**Migration Required:**
- Update all `port.ID` references to `port.Number`
- Run database migration 005
- See CHANGELOG.md for details
## What's Changed
- Full list of changes...
```
#### 4. Migration Guide
Create detailed migration guide in SDK docs:
**File:** `go-api/docs/migrations/v0.0.10-to-v0.0.11.md`
```markdown
# Migrating from v0.0.10 to v0.0.11
## Breaking Changes
### 1. Port.ID → Port.Number
**Find affected code:**
```bash
grep -rn "port\.ID" . --include="*.go"
```
**Update pattern:**
- Replace `port.ID` with `port.Number`
- Replace `Port{ID:` with `Port{Number:`
## Database Migration
Run migration script:
```bash
docker exec sirius-engine bash -c "cd /tmp/migrations && go run 005_fix_critical_schema_issues/main.go"
```
## Verification
1. Build succeeds: `go build ./...`
2. Tests pass: `go test ./...`
3. No compilation errors
```
### Version Bumping Strategy
**Pre-1.0.0 (current):**
- Breaking changes can occur in patch releases
- Document clearly in CHANGELOG
- Notify dependent project maintainers
**Post-1.0.0 (future):**
- Breaking changes ONLY in major versions
- Deprecation warnings before removal
- Longer support windows for old versions
## Automation Scripts
### Script 1: Update Dependents
**File:** `go-api/scripts/update-dependents.sh`
```bash
#!/bin/bash
# Update go-api version across all dependent projects
set -e
NEW_VERSION=$1
if [ -z "$NEW_VERSION" ]; then
echo "Usage: $0 <version>"
echo "Example: $0 v0.0.11"
exit 1
fi
PROJECTS=("app-scanner" "app-agent")
BASE_DIR="$(cd "$(dirname "$0")/../.." && pwd)"
echo "🚀 Updating go-api to $NEW_VERSION across dependent projects..."
for project in "${PROJECTS[@]}"; do
PROJECT_PATH="$BASE_DIR/minor-projects/$project"
if [ ! -d "$PROJECT_PATH" ]; then
echo "⚠️ Project $project not found at $PROJECT_PATH"
continue
fi
echo ""
echo "📦 Updating $project..."
cd "$PROJECT_PATH"
# Update go.mod
sed -i '' "s|github.com/SiriusScan/go-api v[0-9.]*|github.com/SiriusScan/go-api $NEW_VERSION|g" go.mod
# Run go mod tidy
go mod tidy
# Test build
echo " 🔨 Testing build..."
if go build -o /dev/null .; then
echo " ✅ Build successful"
else
echo " ❌ Build failed - manual intervention required"
continue
fi
# Run tests
echo " 🧪 Running tests..."
if go test ./... -short; then
echo " ✅ Tests passed"
else
echo " ⚠️ Tests failed - review required"
fi
# Commit changes
git add go.mod go.sum
git commit -m "chore: update go-api SDK to $NEW_VERSION" || echo " ℹ️ No changes to commit"
echo " ✅ $project updated successfully"
done
echo ""
echo "✅ All projects updated to go-api $NEW_VERSION"
echo ""
echo "Next steps:"
echo " 1. Review and test changes manually"
echo " 2. Push commits: git push origin main"
echo " 3. Monitor dependent project CI/CD"
```
**Usage:**
```bash
cd /path/to/go-api
./scripts/update-dependents.sh v0.0.11
```
### Script 2: Check Versions
**File:** `go-api/scripts/check-versions.sh`
```bash
#!/bin/bash
# Check which version of go-api each dependent project is using
BASE_DIR="$(cd "$(dirname "$0")/../.." && pwd)"
PROJECTS=("app-scanner" "app-agent")
echo "📊 go-api version check across projects"
echo "========================================"
echo ""
# Get latest go-api version
cd "$BASE_DIR/minor-projects/go-api"
LATEST_VERSION=$(git tag -l "v*" --sort=-version:refname | head -1)
echo "Latest go-api version: $LATEST_VERSION"
echo ""
for project in "${PROJECTS[@]}"; do
PROJECT_PATH="$BASE_DIR/minor-projects/$project"
if [ ! -f "$PROJECT_PATH/go.mod" ]; then
echo "⚠️ $project: go.mod not found"
continue
fi
CURRENT_VERSION=$(grep "github.com/SiriusScan/go-api" "$PROJECT_PATH/go.mod" | grep -v "replace" | awk '{print $2}')
if [ "$CURRENT_VERSION" == "$LATEST_VERSION" ]; then
echo "✅ $project: $CURRENT_VERSION (up to date)"
else
echo "⚠️ $project: $CURRENT_VERSION (outdated, latest: $LATEST_VERSION)"
fi
done
echo ""
echo "To update a project:"
echo " ./scripts/update-dependents.sh $LATEST_VERSION"
```
**Usage:**
```bash
cd /path/to/go-api
./scripts/check-versions.sh
```
## Testing & Verification
### Pre-Release Testing
**Before releasing SDK:**
1. **Unit tests pass:**
```bash
cd go-api
go test ./...
```
2. **Lint checks pass:**
```bash
golangci-lint run
```
3. **Build succeeds:**
```bash
go build ./...
```
4. **Local integration test:**
```bash
cd ../app-scanner
# Ensure replace directive active
go build .
go test ./...
```
### Post-Release Verification
**After releasing SDK:**
1. **Version tag exists:**
```bash
git ls-remote --tags origin | grep v0.0.11
```
2. **GitHub release created:**
- Visit: `https://github.com/SiriusScan/go-api/releases/latest`
- Verify: Tag, release notes, timestamp
3. **Module available:**
```bash
go list -m github.com/SiriusScan/go-api@v0.0.11
```
4. **Dependent project builds:**
```bash
cd app-scanner
go get github.com/SiriusScan/go-api@v0.0.11
go build .
```
### Integration Testing
**Full scan pipeline test:**
```bash
# Start all services
docker compose -f docker-compose.dev.yaml up -d
# Submit test scan
docker exec sirius-rabbitmq rabbitmqadmin publish \
exchange=amq.default routing_key=scan \
payload='{"id":"sdk-test","targets":[{"value":"192.168.1.100","type":"single_ip"}],"options":{"template_id":"quick"},"priority":3}'
# Monitor logs
docker logs --follow sirius-engine | grep -i "error\|duplicate"
docker logs --follow sirius-postgres | grep -i "error\|duplicate"
# Check results
docker exec sirius-api curl http://localhost:8080/hosts
```
**Success criteria:**
- ✅ Scan completes without errors
- ✅ No database constraint violations
- ✅ Data stored correctly in database
- ✅ All containers running stably
## Troubleshooting
### Issue: CI/CD Release Fails
**Symptoms:**
- GitHub Actions workflow fails
- No new tag/release created
**Solutions:**
1. **Check workflow logs:**
- Go to `github.com/SiriusScan/go-api/actions`
- Click failed workflow
- Review error messages
2. **Common causes:**
- Tests failing → Fix tests
- Lint errors → Run `golangci-lint run` locally
- Permission issues → Check GitHub token permissions
3. **Manual workaround:**
- Create tag manually (see Manual Release Process)
- Skip automated release
### Issue: Dependent Project Won't Build
**Symptoms:**
```
go: github.com/SiriusScan/go-api@v0.0.11: invalid version: unknown revision v0.0.11
```
**Solutions:**
1. **Check version exists:**
```bash
git ls-remote --tags https://github.com/SiriusScan/go-api | grep v0.0.11
```
2. **Clear Go module cache:**
```bash
go clean -modcache
go mod download
```
3. **Use replace directive temporarily:**
```go
replace github.com/SiriusScan/go-api => ../go-api
```
### Issue: Breaking Changes Cause Errors
**Symptoms:**
```
port.ID undefined (type sirius.Port has no field or method ID)
```
**Solutions:**
1. **Read CHANGELOG:**
```bash
cat ../go-api/CHANGELOG.md | grep -A 10 "BREAKING CHANGE"
```
2. **Find all affected code:**
```bash
grep -rn "port\.ID" . --include="*.go"
```
3. **Apply migration:**
- Follow migration guide in CHANGELOG
- Update all references
- Test build: `go build ./...`
### Issue: Database Schema Mismatch
**Symptoms:**
```
ERROR: column "id" does not exist in table "ports"
```
**Solutions:**
1. **Run database migration:**
```bash
docker exec sirius-engine bash -c "cd /tmp/migrations && go run 005_fix_critical_schema_issues/main.go"
```
2. **Verify migration applied:**
```bash
docker exec sirius-postgres psql -U postgres -d sirius -c "\d ports"
```
3. **If migration fails:**
- Backup database
- Drop and recreate schema
- Re-run scans to populate data
## Rollback Procedures
### Rolling Back SDK Release
**If new release causes critical issues:**
1. **Identify previous stable version:**
```bash
git tag -l "v*" --sort=-version:refname | head -5
# Example: v0.0.11 (broken), v0.0.10 (last stable)
```
2. **Revert main branch:**
```bash
cd go-api
git revert <commit-hash> # Commit that introduced issues
git push origin main
```
3. **Create hotfix release:**
```bash
git tag v0.0.12
git push origin v0.0.12
# CI/CD creates new release with reverted changes
```
4. **Update dependent projects:**
```bash
./scripts/update-dependents.sh v0.0.12
```
### Rolling Back Dependent Project
**If update breaks a dependent project:**
1. **Revert go.mod:**
```bash
cd app-scanner
git checkout HEAD~1 go.mod go.sum
```
2. **Restore dependencies:**
```bash
go mod download
```
3. **Test:**
```bash
go build ./...
go test ./...
```
4. **Commit rollback:**
```bash
git commit -am "revert: rollback go-api to v0.0.10 due to issues"
git push origin main
```
## Best Practices
### Development
- ✅ **Always update CHANGELOG.md** before releasing
- ✅ **Use conventional commits** for clear history
- ✅ **Test locally** with replace directive first
- ✅ **Run full test suite** before releasing
- ✅ **Document breaking changes** prominently
- ✅ **Create migration guides** for major changes
- ❌ **Don't skip pre-release testing**
- ❌ **Don't make breaking changes without warning**
### Release Management
- ✅ **Use semantic versioning** correctly
- ✅ **Let CI/CD handle releases** (automated)
- ✅ **Verify release succeeds** before updating dependents
- ✅ **Update all dependents together** (avoid version drift)
- ✅ **Monitor dependent project CI/CD** after updates
- ❌ **Don't skip version numbers**
- ❌ **Don't release without testing**
### Communication
- ✅ **Announce breaking changes** in advance
- ✅ **Provide migration guides** for complex changes
- ✅ **Update documentation** with releases
- ✅ **Tag team members** in PRs with breaking changes
- ❌ **Don't surprise teams** with undocumented changes
## References
### Related Documentation
- [Go API SDK Architecture](../architecture/README.go-api-sdk.md) - SDK design and structure
- [Development Workflow](README.development.md) - General development practices
- [go-api README](../../../minor-projects/go-api/README.md) - SDK quick start
- [go-api CHANGELOG](../../../minor-projects/go-api/CHANGELOG.md) - Version history
### External Resources
- [Semantic Versioning](https://semver.org/)
- [Keep a Changelog](https://keepachangelog.com/)
- [Conventional Commits](https://www.conventionalcommits.org/)
- [GitHub Actions](https://docs.github.com/en/actions)
---
**Last Updated:** 2025-10-26
**Process Owner:** Sirius DevOps Team
**Review Schedule:** Quarterly
documentation/dev/operations/README.tasks.md
+366
View File
@@ -0,0 +1,366 @@
---
title: "Task Management System"
description: "Guidelines for managing development tasks using the JSON-based task system for project tracking and progress monitoring."
template: "TEMPLATE.documentation-standard"
version: "1.0.0"
last_updated: "2025-01-03"
author: "Development Team"
tags: ["task-management", "project-tracking", "development-workflow"]
categories: ["operations", "development"]
difficulty: "beginner"
prerequisites: ["json", "project-structure"]
related_docs:
- "README.new-project.md"
- "README.git-operations.md"
dependencies: []
llm_context: "high"
search_keywords:
[
"task management",
"tasks.json",
"project tracking",
"development workflow",
"task status",
]
---
# Task Management System
> **📚 New Projects**: For project initialization workflow, see [README.new-project.md](README.new-project.md)
## Purpose
This document provides guidelines for managing development tasks using our JSON-based task system. It ensures consistent task tracking, progress monitoring, and project completion across all development efforts.
## When to Use This System
- **Working on any project** with a `tasks/{project-name}.json` file
- **Tracking progress** on development sprints
- **Managing dependencies** between related tasks
- **Planning work** and estimating completion
- **Reporting status** to stakeholders
## Core Principles
### 1. Always Check Your Tasks
- **Start each work session** by reviewing the current task file
- **Understand dependencies** before starting work
- **Check task status** to see what's been completed
- **Identify next steps** based on available tasks
### 2. Mark Tasks Complete
- **Update status immediately** when a task is finished
- **Verify completion** using the test strategy
- **Update dependent tasks** if completion affects them
- **Commit changes** to the task file with your code
### 3. Ask for Support When Needed
- **Request clarification** on unclear requirements
- **Ask for help** when blocked or uncertain
- **Escalate issues** that prevent progress
- **Get approval** for significant scope changes
## Task Status Management
### Status Values
| Status | Description | When to Use |
| ------------- | ------------------------- | ------------------------------------ |
| `pending` | Task not started | Default for new tasks |
| `in_progress` | Currently being worked on | When actively working |
| `done` | Completed successfully | When task is finished |
| `blocked` | Cannot proceed | When waiting for external dependency |
### Status Updates
```json
{
"id": "1.2",
"title": "Implement User Authentication",
"status": "in_progress", // ← Update this as work progresses
"priority": "high",
"dependencies": ["1.1"]
}
```
**Update Pattern:**
1. **Start work**: Change to `in_progress`
2. **Complete work**: Change to `done`
3. **Hit blocker**: Change to `blocked`
4. **Resolve blocker**: Change back to `in_progress`
## Task Dependencies
### Understanding Dependencies
Tasks can depend on other tasks being completed first:
```json
{
"id": "2.1",
"title": "Create Database Schema",
"dependencies": ["1.1", "1.2"] // ← Must complete 1.1 AND 1.2 first
}
```
### Dependency Rules
- **Check dependencies** before starting any task
- **Only work on tasks** with completed dependencies
- **Update dependent tasks** when you complete a task
- **Resolve circular dependencies** immediately
### Dependency Resolution
```bash
# Check what tasks are available to work on
# Look for tasks with status "pending" and no incomplete dependencies
# Example: Task 2.1 depends on 1.1 and 1.2
# - If 1.1 is "done" and 1.2 is "done" → Task 2.1 is available
# - If 1.1 is "pending" or 1.2 is "blocked" → Task 2.1 is not available
```
## Task File Structure
### Main Task Structure
```json
{
"id": "1",
"title": "PHASE 1: Core Implementation",
"description": "Brief description of the phase",
"details": "Detailed implementation notes and expected outputs",
"status": "pending",
"priority": "high",
"dependencies": [],
"subtasks": [
{
"id": "1.1",
"title": "Specific Implementation Task",
"description": "What this task accomplishes",
"details": "Implementation details, acceptance criteria, and context",
"status": "pending",
"priority": "high",
"dependencies": [],
"testStrategy": "How to verify this task is complete"
}
]
}
```
### Subtask Structure
```json
{
"id": "1.2",
"title": "Implement User Authentication",
"description": "Add login/logout functionality to the application",
"details": "Create authentication endpoints, middleware, and UI components. Use JWT tokens for session management. Include password hashing and validation.",
"status": "pending",
"priority": "high",
"dependencies": ["1.1"],
"testStrategy": "Test login with valid/invalid credentials, verify JWT token generation, confirm logout clears session"
}
```
## Working with Tasks
### Daily Workflow
1. **Review task file** at start of work session
2. **Identify available tasks** (pending status, dependencies met)
3. **Select highest priority** available task
4. **Update status** to `in_progress`
5. **Work on task** following details and test strategy
6. **Mark complete** when finished
7. **Commit changes** to both code and task file
### Task Selection Guidelines
**Priority Order:**
1. **High priority** tasks first
2. **Medium priority** tasks second
3. **Low priority** tasks last
**Dependency Order:**
1. **No dependencies** first
2. **Fewer dependencies** before more complex ones
3. **Blocking tasks** before dependent tasks
### Example Work Session
```bash
# 1. Check current tasks
cat tasks/feature-development.json | jq '.[] | select(.status == "pending")'
# 2. Find available tasks (no incomplete dependencies)
# Look for tasks with status "pending" and all dependencies "done"
# 3. Start work on selected task
# Update status to "in_progress" in task file
# 4. Complete work
# Update status to "done" in task file
# Commit both code and task file changes
git add .
git commit -m "feat: implement user authentication
- Add JWT-based authentication system
- Create login/logout endpoints
- Update task 1.2 to done status"
```
## Task File Maintenance
### Regular Updates
- **Update status** as work progresses
- **Add details** if requirements change
- **Update dependencies** if new relationships discovered
- **Refine test strategies** based on implementation
### File Validation
Ensure task files maintain proper structure:
```json
// Valid task structure
{
"id": "string", // Required: Unique identifier
"title": "string", // Required: Clear task name
"description": "string", // Required: Brief summary
"details": "string", // Required: Implementation notes
"status": "string", // Required: pending|in_progress|done|blocked
"priority": "string", // Required: high|medium|low
"dependencies": ["array"], // Required: Array of task IDs
"subtasks": [
// Optional: For main tasks only
{
"id": "string",
"title": "string",
"description": "string",
"details": "string",
"status": "string",
"priority": "string",
"dependencies": ["array"],
"testStrategy": "string" // Required for subtasks
}
]
}
```
## Integration with Development Tools
### Cursor Integration
When working with tasks, Cursor will:
- **Include task files** in context for relevant projects
- **Reference task details** when discussing implementation
- **Update task status** when completing work
- **Check dependencies** before suggesting work
### Git Integration
- **Commit task changes** with code changes
- **Reference task IDs** in commit messages
- **Track progress** through commit history
- **Maintain task history** across branches
## Common Patterns
### Task Completion Pattern
```bash
# 1. Complete the work
# 2. Test using testStrategy
# 3. Update task status
# 4. Check dependent tasks
# 5. Commit changes
git add .
git commit -m "feat: complete task 1.2 - user authentication
- Implemented JWT-based auth system
- Added login/logout endpoints
- Updated task status to done
- Ready for dependent tasks"
```
### Dependency Resolution Pattern
```bash
# When completing a task that others depend on:
# 1. Mark current task as done
# 2. Check what tasks depend on this one
# 3. Update dependent tasks if needed
# 4. Notify team of newly available tasks
```
### Blocked Task Pattern
```bash
# When encountering a blocker:
# 1. Update task status to "blocked"
# 2. Add details about the blocker
# 3. Identify resolution steps
# 4. Work on other available tasks
# 5. Return when blocker is resolved
```
## Troubleshooting
### Common Issues
**Task file not found:**
- Verify file exists in `tasks/` directory
- Check filename matches project name
- Ensure JSON syntax is valid
**Invalid task status:**
- Use only: `pending`, `in_progress`, `done`, `blocked`
- Check for typos in status values
- Validate JSON structure
**Circular dependencies:**
- Review dependency chains
- Break circular references
- Restructure task dependencies
**Missing test strategy:**
- Add testStrategy field to subtasks
- Include specific verification steps
- Make tests measurable and clear
### Getting Help
- **Task questions**: Ask for clarification on requirements
- **Dependency issues**: Request help resolving conflicts
- **Status updates**: Confirm completion criteria
- **File problems**: Verify JSON syntax and structure
## Future Enhancements
This task management system will evolve based on team needs:
- **Automated status tracking** based on commit messages
- **Dependency visualization** tools
- **Progress reporting** dashboards
- **Integration** with project management tools
- **Notification system** for task updates
---
_This task management system ensures consistent progress tracking and project completion. Always check your tasks, mark them complete, and ask for support when needed._
documentation/dev/operations/README.terraform-deployment.md
+995
View File
@@ -0,0 +1,995 @@
---
title: "Terraform Cloud Deployment Guide"
description: "Complete guide for deploying Sirius to AWS using Terraform Infrastructure as Code"
template: "TEMPLATE.guide"
version: "1.0.0"
last_updated: "2025-01-03"
author: "Development Team"
tags: ["terraform", "aws", "deployment", "infrastructure", "cloud"]
categories: ["deployment", "infrastructure"]
difficulty: "intermediate"
prerequisites: ["docker", "aws-cli", "terraform"]
related_docs: ["README.development.md", "README.architecture.md"]
dependencies: ["docker-compose.yaml", "infrastructure/"]
llm_context: "high"
search_keywords:
["terraform", "aws", "deployment", "infrastructure", "cloud", "ec2", "docker"]
---
# Terraform Cloud Deployment Guide
## Purpose
This guide provides comprehensive instructions for deploying Sirius to AWS using Terraform Infrastructure as Code. It covers everything from initial setup to production deployment, based on the proven infrastructure patterns from the sirius-demo project.
## When to Use This Guide
- **Production deployments** - Deploying Sirius to AWS for production use
- **Staging environments** - Creating isolated testing environments
- **Demo environments** - Setting up temporary demo instances
- **Infrastructure automation** - Managing infrastructure as code
- **Cloud migration** - Moving from local Docker to cloud deployment
## Prerequisites
### Required Tools
- **Terraform** >= 1.0.0
- **AWS CLI** >= 2.0.0
- **Docker** >= 20.10.0 (for local testing)
- **Git** >= 2.0.0
### AWS Requirements
- AWS account with appropriate permissions
- EC2, VPC, IAM, and S3 access
- Default VPC or custom VPC/subnet configuration
### Knowledge Prerequisites
- Basic understanding of Terraform
- AWS fundamentals (EC2, VPC, Security Groups)
- Docker and Docker Compose
- Linux command line
## Quick Start
### 1. Clone and Setup
```bash
# Clone the repository
git clone https://github.com/SiriusScan/Sirius.git
cd Sirius
# Create infrastructure directory
mkdir -p infrastructure/aws
cd infrastructure/aws
```
### 2. Basic Terraform Configuration
Create `main.tf`:
```hcl
terraform {
required_version = ">= 1.0.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.0"
}
}
}
provider "aws" {
region = var.aws_region
default_tags {
tags = var.common_tags
}
}
# Data source for latest Ubuntu 22.04 LTS AMI
data "aws_ami" "ubuntu" {
most_recent = true
owners = ["099720109477"] # Canonical
filter {
name = "name"
values = ["ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*"]
}
filter {
name = "virtualization-type"
values = ["hvm"]
}
}
# Security Group for Sirius
resource "aws_security_group" "sirius" {
name = "sirius-sg"
description = "Security group for Sirius deployment"
vpc_id = var.vpc_id
# UI access
ingress {
description = "UI Access"
from_port = 3000
to_port = 3000
protocol = "tcp"
cidr_blocks = var.allowed_cidrs
}
# API access
ingress {
description = "API Access"
from_port = 9001
to_port = 9001
protocol = "tcp"
cidr_blocks = var.allowed_cidrs
}
# HTTP/HTTPS for load balancer
ingress {
description = "HTTP"
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = var.allowed_cidrs
}
ingress {
description = "HTTPS"
from_port = 443
to_port = 443
protocol = "tcp"
cidr_blocks = var.allowed_cidrs
}
# SSH access (optional)
ingress {
description = "SSH"
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = var.ssh_cidrs
}
# All outbound traffic
egress {
description = "All outbound traffic"
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = {
Name = "sirius-sg"
}
}
# IAM role for EC2 instance
resource "aws_iam_role" "sirius_instance" {
name = "sirius-instance-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = {
Service = "ec2.amazonaws.com"
}
}
]
})
tags = {
Name = "sirius-instance-role"
}
}
# Attach SSM managed policy
resource "aws_iam_role_policy_attachment" "ssm_managed_instance" {
role = aws_iam_role.sirius_instance.name
policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}
# Attach CloudWatch policy
resource "aws_iam_role_policy_attachment" "cloudwatch_agent" {
role = aws_iam_role.sirius_instance.name
policy_arn = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
}
# Instance profile
resource "aws_iam_instance_profile" "sirius" {
name = "sirius-instance-profile"
role = aws_iam_role.sirius_instance.name
tags = {
Name = "sirius-instance-profile"
}
}
# EC2 Instance
resource "aws_instance" "sirius" {
ami = data.aws_ami.ubuntu.id
instance_type = var.instance_type
subnet_id = var.subnet_id
vpc_security_group_ids = [aws_security_group.sirius.id]
iam_instance_profile = aws_iam_instance_profile.sirius.name
root_block_device {
volume_type = "gp3"
volume_size = var.root_volume_size
delete_on_termination = true
encrypted = true
}
user_data = templatefile("${path.module}/user_data.sh", {
sirius_repo_url = var.sirius_repo_url
sirius_branch = var.sirius_branch
environment = var.environment
})
user_data_replace_on_change = true
tags = {
Name = "sirius-${var.environment}"
}
}
```
### 3. Variables Configuration
Create `variables.tf`:
```hcl
variable "aws_region" {
description = "AWS region for deployment"
type = string
default = "us-east-1"
}
variable "instance_type" {
description = "EC2 instance type"
type = string
default = "t3.medium"
}
variable "root_volume_size" {
description = "Root EBS volume size in GB"
type = number
default = 30
}
variable "vpc_id" {
description = "VPC ID where Sirius will be deployed"
type = string
}
variable "subnet_id" {
description = "Subnet ID for Sirius instance"
type = string
}
variable "allowed_cidrs" {
description = "CIDR blocks allowed to access Sirius"
type = list(string)
default = ["0.0.0.0/0"]
}
variable "ssh_cidrs" {
description = "CIDR blocks allowed SSH access"
type = list(string)
default = ["0.0.0.0/0"]
}
variable "sirius_repo_url" {
description = "Sirius repository URL"
type = string
default = "https://github.com/SiriusScan/Sirius.git"
}
variable "sirius_branch" {
description = "Git branch to deploy"
type = string
default = "main"
}
variable "environment" {
description = "Environment name (dev, staging, prod)"
type = string
default = "dev"
}
variable "common_tags" {
description = "Common tags for all resources"
type = map(string)
default = {
Project = "Sirius"
Environment = "dev"
ManagedBy = "Terraform"
}
}
```
### 4. Outputs Configuration
Create `outputs.tf`:
```hcl
output "instance_id" {
description = "EC2 instance ID"
value = aws_instance.sirius.id
}
output "instance_public_ip" {
description = "Public IP address of Sirius instance"
value = aws_instance.sirius.public_ip
}
output "instance_public_dns" {
description = "Public DNS name of Sirius instance"
value = aws_instance.sirius.public_dns
}
output "ui_url" {
description = "URL to access Sirius UI"
value = "http://${aws_instance.sirius.public_ip}:3000"
}
output "api_url" {
description = "URL to access Sirius API"
value = "http://${aws_instance.sirius.public_ip}:9001"
}
output "ssh_command" {
description = "SSH command to connect to instance"
value = "ssh -i ~/.ssh/your-key.pem ubuntu@${aws_instance.sirius.public_ip}"
}
output "ssm_command" {
description = "AWS SSM Session Manager command"
value = "aws ssm start-session --target ${aws_instance.sirius.id} --region ${var.aws_region}"
}
```
### 5. Bootstrap Script
Create `user_data.sh`:
```bash
#!/bin/bash
set -e
# Log all output
exec > >(tee -a /var/log/sirius-bootstrap.log)
exec 2>&1
echo "========================================="
echo "Sirius Bootstrap Script"
echo "Started at: $(date)"
echo "Environment: ${environment}"
echo "========================================="
# Update system
echo "📦 Updating system packages..."
apt-get update -y
DEBIAN_FRONTEND=noninteractive apt-get upgrade -y
# Install dependencies
echo "📦 Installing dependencies..."
apt-get install -y \
git \
jq \
curl \
ca-certificates \
gnupg \
lsb-release
# Install Docker
echo "🐳 Installing Docker..."
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
chmod a+r /etc/apt/keyrings/docker.gpg
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
# Start Docker
echo "🐳 Starting Docker service..."
systemctl enable docker
systemctl start docker
usermod -aG docker ubuntu
# Verify Docker
echo "✅ Verifying Docker installation..."
docker --version
docker compose version
# Create application directory
echo "📁 Creating application directory..."
mkdir -p /opt/sirius
cd /opt/sirius
# Clone repository
echo "📥 Cloning Sirius repository..."
echo "Repository: ${sirius_repo_url}"
echo "Branch: ${sirius_branch}"
git clone --branch ${sirius_branch} ${sirius_repo_url} repo
cd repo
# Create environment configuration using installer-first flow
echo "⚙️ Configuring environment..."
docker compose -f docker-compose.installer.yaml run --rm sirius-installer --non-interactive --no-print-secrets
echo "✅ Environment configuration created with installer"
# Determine image tag from sirius_branch variable
if [[ "${sirius_branch}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
# Version tag (e.g., v0.4.1)
IMAGE_TAG="${sirius_branch}"
elif [ "${sirius_branch}" == "main" ]; then
# Main branch uses latest
IMAGE_TAG="latest"
else
# Default to latest
IMAGE_TAG="latest"
fi
export IMAGE_TAG
echo "📦 Using image tag: ${IMAGE_TAG}"
# Pull Docker images from registry
echo "🐳 Pulling prebuilt images from GitHub Container Registry..."
docker compose pull || echo "⚠️ Some images may need to be built (fallback)"
# Start services (uses prebuilt images by default)
echo "🚀 Starting Sirius services..."
echo "Starting services with prebuilt images (this should take 2-5 minutes)..."
docker compose up -d
# Wait for services
echo "⏳ Waiting for services to initialize..."
sleep 30
# Show running containers
echo "📊 Running containers:"
docker compose ps
echo "========================================="
echo "Bootstrap completed at: $(date)"
echo "========================================="
echo ""
echo "Access services:"
echo " UI: http://$(curl -s http://169.254.169.254/latest/meta-data/public-ipv4):3000"
echo " API: http://$(curl -s http://169.254.169.254/latest/meta-data/public-ipv4):9001"
echo "========================================="
```
### 6. Terraform Variables File
Create `terraform.tfvars`:
```hcl
# AWS Configuration
aws_region = "us-east-1"
# Network Configuration (REQUIRED)
vpc_id = "vpc-xxxxxxxxxxxxxxxxx" # Replace with your VPC ID
subnet_id = "subnet-xxxxxxxxxxxxxxxxx" # Replace with your subnet ID
# Instance Configuration
instance_type = "t3.medium"
root_volume_size = 30
# Access Control
allowed_cidrs = ["0.0.0.0/0"] # Restrict for production
ssh_cidrs = ["0.0.0.0/0"] # Restrict for production
# Repository Configuration
sirius_repo_url = "https://github.com/SiriusScan/Sirius.git"
sirius_branch = "main"
environment = "dev"
# Resource Tags
common_tags = {
Project = "Sirius"
Environment = "dev"
ManagedBy = "Terraform"
Owner = "YourTeam"
}
```
## Container Registry Deployment
Sirius now uses prebuilt container images from GitHub Container Registry (GHCR) by default, providing **60-75% faster deployments** (5-8 minutes vs 20-25 minutes). The base `docker-compose.yaml` automatically pulls images from `ghcr.io/siriusscan/sirius-{ui,api,engine}:{tag}`.
### Image Tagging
Images are automatically tagged based on the deployment:
- **Version tags** (e.g., `v0.4.1`): Use when deploying specific releases
- **Latest tag**: Default for main branch deployments
- **Beta tag**: Available for beta releases
### Environment Variable
Control which images to use with the `IMAGE_TAG` environment variable in your bootstrap script:
```bash
# In user_data.sh or .env file
export IMAGE_TAG="${sirius_branch:-latest}"
```
### Benefits
- **Faster deployments**: 5-8 minutes vs 20-25 minutes
- **Reduced resource usage**: No compilation on EC2 instance
- **Consistent builds**: Same images tested in CI/CD
- **Easy updates**: Pull latest images without rebuilding
### Fallback to Local Builds
If registry images are unavailable, you can fall back to local builds by using `docker compose -f docker-compose.yaml -f docker-compose.build.yaml up -d --build` instead of `docker compose up -d`.
For more details, see [Docker Container Deployment Guide](../deployment/README.docker-container-deployment.md).
## Deployment Process
### 1. Initialize Terraform
```bash
# Initialize Terraform
terraform init
# Validate configuration
terraform validate
# Review planned changes
terraform plan
```
### 2. Deploy Infrastructure
```bash
# Apply configuration
terraform apply
# Confirm with 'yes' when prompted
# Or use -auto-approve for automated deployment
terraform apply -auto-approve
```
### 3. Access Your Deployment
After deployment completes, Terraform will output the URLs:
```bash
# Get outputs
terraform output
# Access UI
open http://$(terraform output -raw instance_public_ip):3000
# Check API health
curl http://$(terraform output -raw instance_public_ip):9001/health
```
## Advanced Configuration
### Production Deployment
For production deployments, consider these enhancements:
#### 1. State Management
Create `backend.tf`:
```hcl
terraform {
backend "s3" {
bucket = "your-terraform-state-bucket"
key = "sirius/terraform.tfstate"
region = "us-east-1"
encrypt = true
dynamodb_table = "terraform-state-lock"
}
}
```
#### 2. Load Balancer Configuration
Add to `main.tf`:
```hcl
# Application Load Balancer
resource "aws_lb" "sirius" {
name = "sirius-alb"
internal = false
load_balancer_type = "application"
security_groups = [aws_security_group.alb.id]
subnets = var.subnet_ids
enable_deletion_protection = false
tags = {
Name = "sirius-alb"
}
}
# Target Group
resource "aws_lb_target_group" "sirius" {
name = "sirius-tg"
port = 3000
protocol = "HTTP"
vpc_id = var.vpc_id
health_check {
enabled = true
healthy_threshold = 2
interval = 30
matcher = "200"
path = "/"
port = "traffic-port"
protocol = "HTTP"
timeout = 5
unhealthy_threshold = 2
}
}
# ALB Listener
resource "aws_lb_listener" "sirius" {
load_balancer_arn = aws_lb.sirius.arn
port = "80"
protocol = "HTTP"
default_action {
type = "forward"
target_group_arn = aws_lb_target_group.sirius.arn
}
}
# Target Group Attachment
resource "aws_lb_target_group_attachment" "sirius" {
target_group_arn = aws_lb_target_group.sirius.arn
target_id = aws_instance.sirius.id
port = 3000
}
```
#### 3. Database Configuration
For production, consider using RDS:
```hcl
# RDS Instance
resource "aws_db_instance" "sirius" {
identifier = "sirius-db"
engine = "postgres"
engine_version = "15.4"
instance_class = "db.t3.micro"
allocated_storage = 20
max_allocated_storage = 100
storage_type = "gp2"
storage_encrypted = true
db_name = "sirius"
username = "postgres"
password = var.db_password
vpc_security_group_ids = [aws_security_group.rds.id]
db_subnet_group_name = aws_db_subnet_group.sirius.name
backup_retention_period = 7
backup_window = "03:00-04:00"
maintenance_window = "sun:04:00-sun:05:00"
skip_final_snapshot = true
tags = {
Name = "sirius-db"
}
}
```
### Environment-Specific Deployments
#### Development Environment
```hcl
# terraform.tfvars.dev
environment = "dev"
instance_type = "t3.small"
allowed_cidrs = ["0.0.0.0/0"]
```
#### Staging Environment
```hcl
# terraform.tfvars.staging
environment = "staging"
instance_type = "t3.medium"
allowed_cidrs = ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"]
```
#### Production Environment
```hcl
# terraform.tfvars.prod
environment = "prod"
instance_type = "t3.large"
allowed_cidrs = ["10.0.0.0/8"] # Restrict to internal networks
ssh_cidrs = ["10.0.0.0/8"] # Restrict SSH access
```
## Monitoring and Maintenance
### Health Checks
```bash
# Check instance status
aws ec2 describe-instances --instance-ids $(terraform output -raw instance_id)
# Check application health
curl -f http://$(terraform output -raw instance_public_ip):9001/health
# Check Docker containers
aws ssm start-session --target $(terraform output -raw instance_id) --region us-east-1
docker compose ps
```
### Log Management
```bash
# View bootstrap logs
aws ssm start-session --target $(terraform output -raw instance_id) --region us-east-1
sudo tail -f /var/log/sirius-bootstrap.log
# View application logs
docker compose logs -f
```
### Updates and Scaling
```bash
# Update instance type
terraform apply -var="instance_type=t3.large"
# Update to new branch
terraform apply -var="sirius_branch=feature/new-feature"
# Scale storage
terraform apply -var="root_volume_size=50"
```
## Security Considerations
### Network Security
- **Restrict access**: Use specific CIDR blocks instead of 0.0.0.0/0
- **Use private subnets**: Deploy in private subnets with NAT Gateway
- **Implement WAF**: Add AWS WAF for additional protection
- **VPN access**: Use VPN for secure access to private resources
### Instance Security
- **Key pairs**: Use EC2 key pairs for SSH access
- **Security groups**: Implement least-privilege security group rules
- **IAM roles**: Use IAM roles instead of access keys
- **Encryption**: Enable encryption for EBS volumes
### Application Security
- **Secrets management**: Use AWS Secrets Manager or Parameter Store
- **TLS certificates**: Implement SSL/TLS with ACM
- **Regular updates**: Keep base images and dependencies updated
- **Monitoring**: Implement CloudWatch monitoring and alerting
## Troubleshooting
### Common Issues
#### 1. Instance Not Starting
```bash
# Check instance status
aws ec2 describe-instances --instance-ids $(terraform output -raw instance_id)
# View system logs
aws ec2 get-console-output --instance-id $(terraform output -raw instance_id)
```
#### 2. Services Not Responding
```bash
# Connect to instance
aws ssm start-session --target $(terraform output -raw instance_id) --region us-east-1
# Check Docker status
sudo systemctl status docker
docker compose ps
# View logs
docker compose logs -f
```
#### 3. Network Connectivity Issues
```bash
# Check security groups
aws ec2 describe-security-groups --group-ids $(terraform output -raw security_group_id)
# Test connectivity
curl -v http://$(terraform output -raw instance_public_ip):3000
```
### Debugging Commands
```bash
# Terraform state inspection
terraform state list
terraform state show aws_instance.sirius
# Plan changes
terraform plan -var-file="terraform.tfvars.prod"
# Destroy and recreate
terraform destroy
terraform apply
```
## Cost Optimization
### Instance Sizing
- **Development**: t3.small (2 vCPU, 2GB RAM) - ~$15/month
- **Staging**: t3.medium (2 vCPU, 4GB RAM) - ~$30/month
- **Production**: t3.large (2 vCPU, 8GB RAM) - ~$60/month
### Storage Optimization
- **GP3 volumes**: More cost-effective than GP2
- **Right-sizing**: Monitor usage and adjust volume sizes
- **Cleanup**: Regular cleanup of unused resources
### Monitoring Costs
```bash
# Check current costs
aws ce get-cost-and-usage \
--time-period Start=2024-01-01,End=2024-01-31 \
--granularity MONTHLY \
--metrics BlendedCost
# Set up billing alerts
aws cloudwatch put-metric-alarm \
--alarm-name "Sirius-Monthly-Cost" \
--alarm-description "Alert when monthly cost exceeds $100" \
--metric-name EstimatedCharges \
--namespace AWS/Billing \
--statistic Maximum \
--period 86400 \
--threshold 100 \
--comparison-operator GreaterThanThreshold
```
## Cleanup
### Destroy Infrastructure
```bash
# Destroy all resources
terraform destroy
# Confirm with 'yes' when prompted
# Or use -auto-approve
terraform destroy -auto-approve
```
### Clean Up State
```bash
# Remove state file (if using local state)
rm terraform.tfstate*
# Clean up .terraform directory
rm -rf .terraform/
```
## Best Practices
### Infrastructure as Code
- **Version control**: Keep all Terraform files in version control
- **State management**: Use remote state with locking
- **Modular design**: Break down into reusable modules
- **Documentation**: Document all variables and outputs
### Security
- **Least privilege**: Use minimal required permissions
- **Secrets management**: Never hardcode secrets
- **Regular updates**: Keep Terraform and providers updated
- **Audit trails**: Enable CloudTrail for API calls
### Operations
- **Monitoring**: Implement comprehensive monitoring
- **Backup**: Regular backups of state and data
- **Testing**: Test changes in non-production first
- **Documentation**: Keep deployment procedures documented
## Integration with CI/CD
### GitHub Actions Example
```yaml
name: Deploy Sirius
on:
push:
branches: [main]
workflow_dispatch:
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-east-1
- name: Setup Terraform
uses: hashicorp/setup-terraform@v3
with:
terraform_version: 1.6.0
- name: Terraform Init
run: |
cd infrastructure/aws
terraform init
- name: Terraform Plan
run: |
cd infrastructure/aws
terraform plan
- name: Terraform Apply
run: |
cd infrastructure/aws
terraform apply -auto-approve
```
## Related Documentation
- [Development Guide](dev/README.development.md) - Local development setup
- [Architecture Overview](dev/architecture/README.architecture.md) - System architecture
- [Container Testing](dev/test/README.container-testing.md) - Testing procedures
## Support
For issues with Terraform deployment:
1. Check the troubleshooting section above
2. Review Terraform logs and AWS CloudTrail
3. Consult AWS documentation for specific services
4. Create an issue in the Sirius repository
---
_This guide follows the Sirius Documentation Standard. For questions about documentation structure, see [ABOUT.documentation.md](dev/ABOUT.documentation.md)._
documentation/dev/templates/TEMPLATE.about.md
+136
View File
@@ -0,0 +1,136 @@
---
title: "About [System/Component] Documentation Template"
description: "This template defines the standard structure and conventions for ABOUT documents within the Sirius project, ensuring clear explanations of how to use specific systems or documentation types."
template: "TEMPLATE.template"
version: "1.0.0"
last_updated: "2025-01-03"
author: "AI Assistant"
tags: ["template", "about", "explanation", "how-to-use"]
categories: ["project-management", "development"]
difficulty: "beginner"
prerequisites: ["ABOUT.documentation.md"]
related_docs:
- "ABOUT.documentation.md"
- "TEMPLATE.documentation-standard.md"
dependencies: []
llm_context: "high"
search_keywords:
["about template", "explanation template", "how-to-use template"]
---
# About [System/Component] Documentation
## Purpose
This document explains how to use [specific system/component/documentation type] within the Sirius project. ABOUT documents serve as guides for understanding and working with specific aspects of our system.
## When to Use
- **New Team Members**: When someone needs to understand how to use a specific system
- **Reference Material**: When you need to quickly understand the purpose and usage of a component
- **Onboarding**: When integrating new developers or contributors
- **System Changes**: When updating or modifying how a system works
## How to Use
### Quick Start
1. **Read the Purpose section** to understand what this system does
2. **Check the Prerequisites** to ensure you have required knowledge/tools
3. **Follow the Usage Guidelines** for step-by-step instructions
4. **Reference the Examples** for common use cases
5. **Use the Troubleshooting section** if you encounter issues
### Common Commands
```bash
# [Common command 1]
[command example]
# [Common command 2]
[command example]
```
## What It Is
### System Overview
[Detailed explanation of what the system/component is and how it works]
### Key Components
- **[Component 1]**: [Description and purpose]
- **[Component 2]**: [Description and purpose]
- **[Component 3]**: [Description and purpose]
### Architecture
[High-level architecture diagram or description]
### File Structure
```
[relevant directory structure]
├── [file1] - [purpose]
├── [file2] - [purpose]
└── [directory]/
└── [file3] - [purpose]
```
### Configuration
[How to configure the system, including environment variables, config files, etc.]
### Integration Points
[How this system integrates with other parts of the project]
## Troubleshooting
### FAQ
**Q: [Common question 1]**
A: [Answer with specific steps or explanations]
**Q: [Common question 2]**
A: [Answer with specific steps or explanations]
**Q: [Common question 3]**
A: [Answer with specific steps or explanations]
### Command Reference
| Command | Purpose | Example |
| ------------ | -------------- | ----------------- |
| `[command1]` | [What it does] | `[example usage]` |
| `[command2]` | [What it does] | `[example usage]` |
| `[command3]` | [What it does] | `[example usage]` |
### Common Issues
| Issue | Symptoms | Solution |
| --------- | ----------------- | ------------------ |
| [Issue 1] | [How to identify] | [Step-by-step fix] |
| [Issue 2] | [How to identify] | [Step-by-step fix] |
| [Issue 3] | [How to identify] | [Step-by-step fix] |
### Debugging Steps
1. **Check [specific thing]**: [How to check]
2. **Verify [specific thing]**: [How to verify]
3. **Test [specific thing]**: [How to test]
4. **Review [specific thing]**: [What to look for]
## Lessons Learned
### [Date] - [What was learned]
[Description of lesson learned and how it improved the system]
### [Date] - [What was learned]
[Description of lesson learned and how it improved the system]
---
_This document follows the Sirius Documentation Standard. For questions about documentation structure, see [ABOUT.documentation.md](../ABOUT.documentation.md)._
documentation/dev/templates/TEMPLATE.api.md
+272
View File
@@ -0,0 +1,272 @@
---
title: "API Documentation Template"
description: "This template defines the standard structure and conventions for API documents within the Sirius project, ensuring comprehensive API specifications and usage documentation."
template: "TEMPLATE.template"
version: "1.0.0"
last_updated: "2025-01-03"
author: "AI Assistant"
tags: ["template", "api", "endpoints", "specifications", "integration"]
categories: ["project-management", "development"]
difficulty: "intermediate"
prerequisites: ["ABOUT.documentation.md"]
related_docs:
- "ABOUT.documentation.md"
- "TEMPLATE.documentation-standard.md"
dependencies: []
llm_context: "high"
search_keywords:
["api template", "endpoints", "api documentation", "integration specs"]
---
# [API Name] API Documentation
## Purpose
This document provides comprehensive API documentation for [specific API/service] within the Sirius project. API documents serve as authoritative sources for endpoint specifications, request/response formats, authentication, and integration examples.
## When to Use
- **API Integration**: When implementing client applications
- **API Development**: When building or modifying API endpoints
- **Testing**: When writing tests for API functionality
- **Debugging**: When troubleshooting API issues
- **Code Review**: When reviewing API implementation
## How to Use
### Quick Start
1. **Check the Base URL** and authentication requirements
2. **Review the Common Patterns** for typical usage
3. **Find the specific endpoint** you need
4. **Follow the examples** for implementation
5. **Use the troubleshooting section** for common issues
### Base URL
```
http://localhost:9001/api/v1
```
### Authentication
[Description of authentication method - API key, JWT, OAuth, etc.]
```bash
# Example authentication
curl -H "Authorization: Bearer <token>" \
-H "Content-Type: application/json" \
http://localhost:9001/api/v1/endpoint
```
## What It Is
### API Overview
[Description of what this API does, its purpose, and key capabilities]
### Endpoints
#### [Endpoint 1]
- **Method**: `GET|POST|PUT|DELETE`
- **Path**: `/api/v1/endpoint1`
- **Description**: [What this endpoint does]
- **Authentication**: [Required/Optional]
- **Parameters**:
- `param1` (required, string): [Description]
- `param2` (optional, integer): [Description]
- **Request Body**: [If applicable]
- **Response**: [Response format and status codes]
- **Example Request**:
```bash
curl -X GET "http://localhost:9001/api/v1/endpoint1?param1=value&param2=123" \
-H "Authorization: Bearer <token>"
```
- **Example Response**:
```json
{
"status": "success",
"data": {
"field1": "value1",
"field2": "value2"
}
}
```
#### [Endpoint 2]
- **Method**: `GET|POST|PUT|DELETE`
- **Path**: `/api/v1/endpoint2`
- **Description**: [What this endpoint does]
- **Authentication**: [Required/Optional]
- **Request Body**:
```json
{
"field1": "string",
"field2": "integer",
"field3": {
"nested_field": "string"
}
}
```
- **Response**: [Response format and status codes]
- **Example Request**:
```bash
curl -X POST "http://localhost:9001/api/v1/endpoint2" \
-H "Authorization: Bearer <token>" \
-H "Content-Type: application/json" \
-d '{
"field1": "value1",
"field2": 123,
"field3": {
"nested_field": "nested_value"
}
}'
```
- **Example Response**:
```json
{
"status": "success",
"data": {
"id": "generated_id",
"field1": "value1",
"field2": 123
}
}
```
### Data Models
#### [Model 1]
```json
{
"id": "string",
"name": "string",
"description": "string",
"created_at": "datetime",
"updated_at": "datetime"
}
```
#### [Model 2]
```json
{
"id": "string",
"field1": "string",
"field2": "integer",
"field3": {
"nested_field": "string"
},
"field4": ["string"]
}
```
### Error Responses
#### Standard Error Format
```json
{
"status": "error",
"error": {
"code": "ERROR_CODE",
"message": "Human readable error message",
"details": "Additional error details"
}
}
```
#### Common Error Codes
| Code | HTTP Status | Description | Resolution |
| ----------------- | ----------- | ------------------------- | ---------------------------------- |
| `INVALID_REQUEST` | 400 | Request format is invalid | Check request body and parameters |
| `UNAUTHORIZED` | 401 | Authentication required | Provide valid authentication token |
| `FORBIDDEN` | 403 | Insufficient permissions | Check user permissions |
| `NOT_FOUND` | 404 | Resource not found | Verify resource ID and existence |
| `INTERNAL_ERROR` | 500 | Server error | Contact support or retry later |
### Rate Limiting
- **Limit**: [Requests per time period]
- **Headers**: [Rate limit headers returned]
- **Example**: `X-RateLimit-Limit: 1000, X-RateLimit-Remaining: 999`
### Pagination
- **Method**: [Offset-based, cursor-based, etc.]
- **Parameters**: [page, limit, offset, etc.]
- **Response Format**: [How pagination data is returned]
```json
{
"data": [...],
"pagination": {
"page": 1,
"limit": 20,
"total": 100,
"pages": 5
}
}
```
## Troubleshooting
### FAQ
**Q: [Common API question 1]**
A: [Answer with specific examples and solutions]
**Q: [Common API question 2]**
A: [Answer with specific examples and solutions]
**Q: [Common API question 3]**
A: [Answer with specific examples and solutions]
### Command Reference
| Command | Purpose | Example | Notes |
| -------------- | -------------- | ----------- | ------------------ |
| `curl -X GET` | [What it does] | `[example]` | [Additional notes] |
| `curl -X POST` | [What it does] | `[example]` | [Additional notes] |
| `curl -X PUT` | [What it does] | `[example]` | [Additional notes] |
### Common Issues
| Issue | Symptoms | Root Cause | Solution |
| --------- | ----------------- | ---------------- | ------------------ |
| [Issue 1] | [How to identify] | [Why it happens] | [Step-by-step fix] |
| [Issue 2] | [How to identify] | [Why it happens] | [Step-by-step fix] |
| [Issue 3] | [How to identify] | [Why it happens] | [Step-by-step fix] |
### Debugging Steps
1. **Check [specific thing]**: [How to check and what to look for]
2. **Verify [specific thing]**: [How to verify and expected results]
3. **Test [specific thing]**: [How to test and success criteria]
4. **Review [specific thing]**: [What to review and common patterns]
## Lessons Learned
### [Date] - [What was learned]
[Description of API lesson learned and how it improved the system]
### [Date] - [What was learned]
[Description of API lesson learned and how it improved the system]
---
_This document follows the Sirius Documentation Standard. For questions about documentation structure, see [ABOUT.documentation.md](../ABOUT.documentation.md)._
documentation/dev/templates/TEMPLATE.architecture.md
+227
View File
@@ -0,0 +1,227 @@
---
title: "Architecture Documentation Template"
description: "This template defines the standard structure and conventions for ARCHITECTURE documents within the Sirius project, ensuring comprehensive system design and structural documentation."
template: "TEMPLATE.template"
version: "1.0.0"
last_updated: "2025-01-03"
author: "AI Assistant"
tags: ["template", "architecture", "system-design", "structure", "overview"]
categories: ["project-management", "development"]
difficulty: "advanced"
prerequisites: ["ABOUT.documentation.md"]
related_docs:
- "ABOUT.documentation.md"
- "TEMPLATE.documentation-standard.md"
dependencies: []
llm_context: "high"
search_keywords:
[
"architecture template",
"system design",
"structural overview",
"technical architecture",
]
---
# [System/Component] Architecture
## Purpose
This document provides comprehensive architectural documentation for [specific system/component] within the Sirius project. ARCHITECTURE documents serve as authoritative sources for system design, component relationships, data flow, and structural decisions.
## When to Use
- **System Design**: When designing new systems or major components
- **Integration Planning**: When connecting systems or components
- **Code Review**: When reviewing implementation against architectural decisions
- **Onboarding**: When bringing new team members up to speed
- **Refactoring**: When making significant structural changes
## How to Use
### Quick Overview
1. **Start with the System Overview** to understand the big picture
2. **Review the Component Architecture** to understand individual parts
3. **Follow the Data Flow** to understand how information moves
4. **Check the Integration Points** to understand connections
5. **Reference the Design Decisions** for context on why choices were made
### Architecture Patterns
```mermaid
graph TB
A[Component A] --> B[Component B]
B --> C[Component C]
C --> D[Component D]
D --> A
```
## What It Is
### System Overview
[High-level description of the system, its purpose, and key characteristics]
### Architectural Principles
- **[Principle 1]**: [Description and rationale]
- **[Principle 2]**: [Description and rationale]
- **[Principle 3]**: [Description and rationale]
### Component Architecture
#### [Component 1]
- **Purpose**: [What this component does]
- **Responsibilities**: [Key responsibilities]
- **Dependencies**: [What it depends on]
- **Interfaces**: [How it communicates with other components]
#### [Component 2]
- **Purpose**: [What this component does]
- **Responsibilities**: [Key responsibilities]
- **Dependencies**: [What it depends on]
- **Interfaces**: [How it communicates with other components]
#### [Component 3]
- **Purpose**: [What this component does]
- **Responsibilities**: [Key responsibilities]
- **Dependencies**: [What it depends on]
- **Interfaces**: [How it communicates with other components]
### Data Flow
#### [Flow 1]
1. **[Step 1]**: [Description of what happens]
2. **[Step 2]**: [Description of what happens]
3. **[Step 3]**: [Description of what happens]
#### [Flow 2]
1. **[Step 1]**: [Description of what happens]
2. **[Step 2]**: [Description of what happens]
3. **[Step 3]**: [Description of what happens]
### Integration Points
#### [Integration 1]
- **Type**: [API, Database, Message Queue, etc.]
- **Purpose**: [Why this integration exists]
- **Protocol**: [How communication happens]
- **Data Format**: [What data is exchanged]
#### [Integration 2]
- **Type**: [API, Database, Message Queue, etc.]
- **Purpose**: [Why this integration exists]
- **Protocol**: [How communication happens]
- **Data Format**: [What data is exchanged]
### Technology Stack
#### Backend
- **[Technology 1]**: [Purpose and rationale]
- **[Technology 2]**: [Purpose and rationale]
- **[Technology 3]**: [Purpose and rationale]
#### Frontend
- **[Technology 1]**: [Purpose and rationale]
- **[Technology 2]**: [Purpose and rationale]
- **[Technology 3]**: [Purpose and rationale]
#### Infrastructure
- **[Technology 1]**: [Purpose and rationale]
- **[Technology 2]**: [Purpose and rationale]
- **[Technology 3]**: [Purpose and rationale]
### Design Decisions
#### [Decision 1]
- **Context**: [What situation led to this decision]
- **Decision**: [What was decided]
- **Rationale**: [Why this decision was made]
- **Consequences**: [What this decision means for the system]
#### [Decision 2]
- **Context**: [What situation led to this decision]
- **Decision**: [What was decided]
- **Rationale**: [Why this decision was made]
- **Consequences**: [What this decision means for the system]
### Security Considerations
- **[Security Aspect 1]**: [How it's addressed]
- **[Security Aspect 2]**: [How it's addressed]
- **[Security Aspect 3]**: [How it's addressed]
### Performance Characteristics
- **[Performance Aspect 1]**: [Expected behavior and constraints]
- **[Performance Aspect 2]**: [Expected behavior and constraints]
- **[Performance Aspect 3]**: [Expected behavior and constraints]
### Scalability Considerations
- **[Scalability Aspect 1]**: [How the system scales]
- **[Scalability Aspect 2]**: [How the system scales]
- **[Scalability Aspect 3]**: [How the system scales]
## Troubleshooting
### FAQ
**Q: [Common architectural question 1]**
A: [Answer with architectural context and rationale]
**Q: [Common architectural question 2]**
A: [Answer with architectural context and rationale]
**Q: [Common architectural question 3]**
A: [Answer with architectural context and rationale]
### Command Reference
| Command | Purpose | Example | Notes |
| ------------ | -------------- | ----------- | ----------------------- |
| `[command1]` | [What it does] | `[example]` | [Architectural context] |
| `[command2]` | [What it does] | `[example]` | [Architectural context] |
| `[command3]` | [What it does] | `[example]` | [Architectural context] |
### Common Issues
| Issue | Symptoms | Root Cause | Solution |
| --------- | ----------------- | -------------------------------- | ------------------ |
| [Issue 1] | [How to identify] | [Why it happens architecturally] | [Step-by-step fix] |
| [Issue 2] | [How to identify] | [Why it happens architecturally] | [Step-by-step fix] |
| [Issue 3] | [How to identify] | [Why it happens architecturally] | [Step-by-step fix] |
### Debugging Steps
1. **Check [specific thing]**: [How to check and what to look for]
2. **Verify [specific thing]**: [How to verify and expected results]
3. **Test [specific thing]**: [How to test and success criteria]
4. **Review [specific thing]**: [What to review and common patterns]
## Lessons Learned
### [Date] - [What was learned]
[Description of architectural lesson learned and how it improved the system]
### [Date] - [What was learned]
[Description of architectural lesson learned and how it improved the system]
---
_This document follows the Sirius Documentation Standard. For questions about documentation structure, see [ABOUT.documentation.md](../ABOUT.documentation.md)._
documentation/dev/templates/TEMPLATE.custom.md
+131
View File
@@ -0,0 +1,131 @@
---
title: "Custom Documentation Template"
description: "This template defines the structure for custom documents that don't need to follow standard templates, providing flexibility for meta-documents and special cases."
template: "TEMPLATE.template"
version: "1.0.0"
last_updated: "2025-01-03"
author: "AI Assistant"
tags: ["template", "custom", "flexible", "meta-document"]
categories: ["project-management", "development"]
difficulty: "beginner"
prerequisites: ["ABOUT.documentation.md"]
related_docs:
- "ABOUT.documentation.md"
- "TEMPLATE.documentation-standard.md"
dependencies: []
llm_context: "high"
search_keywords:
[
"custom template",
"flexible template",
"meta-document",
"non-standard template",
]
---
# Custom Documentation Template
## Purpose
This template provides a flexible structure for custom documents that don't need to follow standard templates. Use this for meta-documents, special cases, or documents that require unique structure.
## When to Use
- **Meta-documents** - Documents about the documentation system itself
- **Special cases** - Documents that don't fit standard templates
- **Flexible content** - When you need custom structure
- **Experimental docs** - When testing new documentation approaches
## How to Use
1. **Copy this template** for your custom document
2. **Modify the structure** as needed for your specific use case
3. **Keep YAML front matter** for consistency and LLM optimization
4. **Maintain basic standards** even with custom structure
5. **Document deviations** if they're significant
## What It Is
### Flexible Structure
Custom documents can have any structure that makes sense for their purpose, but should include:
- **YAML Front Matter** - For consistency and LLM optimization
- **Clear Purpose** - What the document is for
- **Logical Organization** - Information should be well-structured
- **Appropriate Sections** - Based on the document's purpose
### Custom Sections
Unlike standard templates, custom documents can include:
- **Any section structure** that makes sense
- **Unique formatting** for specific needs
- **Specialized content** for specific audiences
- **Experimental approaches** to documentation
### YAML Front Matter
Even custom documents should include complete YAML front matter:
```yaml
---
title: "Document Title"
description: "Brief description of purpose"
template: "TEMPLATE.custom"
version: "1.0.0"
last_updated: "YYYY-MM-DD"
author: "Document Author"
tags: ["relevant", "tags"]
categories: ["appropriate", "categories"]
difficulty: "beginner|intermediate|advanced"
prerequisites: ["any", "prerequisites"]
related_docs:
- "related-doc1.md"
- "related-doc2.md"
dependencies: ["any", "dependencies"]
llm_context: "high|medium|low"
search_keywords: ["relevant", "keywords"]
---
```
## Troubleshooting
### FAQ
**Q: When should I use a custom template instead of a standard one?**
A: Use custom when the document doesn't fit standard patterns, is a meta-document, or requires unique structure.
**Q: Do custom documents still need YAML front matter?**
A: Yes, for consistency, LLM optimization, and integration with our documentation system.
**Q: How much can I deviate from standard structure?**
A: As much as needed for your specific use case, but maintain basic organization and clarity.
**Q: Should custom documents still be linted?**
A: Yes, but with relaxed template compliance rules for custom documents.
### Command Reference
| Command | Purpose | Example |
| ---------------------------------- | ---------------------- | --------------------------------------------------- |
| `cp TEMPLATE.custom.md new-doc.md` | Create custom document | `cp TEMPLATE.custom.md documentation/custom-doc.md` |
| `grep -r "template: custom"` | Find custom documents | `grep -r "template: custom" documentation/` |
### Common Issues
| Issue | Symptoms | Solution |
| -------------------- | ------------------------ | ------------------------------------------ |
| Too much deviation | Hard to maintain | Consider if a standard template would work |
| Missing front matter | Poor LLM integration | Add complete YAML front matter |
| Poor organization | Hard to find information | Improve structure and add clear headings |
## Lessons Learned
### [Date] - [What was learned]
[Description of custom template lesson learned and how it improved the system]
---
_This document follows the Sirius Documentation Standard. For questions about documentation structure, see [ABOUT.documentation.md](../ABOUT.documentation.md)._
documentation/dev/templates/TEMPLATE.documentation-standard.md
+182
View File
@@ -0,0 +1,182 @@
---
title: "[DOCUMENT TITLE]"
description: "[Brief description of the document's purpose and scope]"
template: "TEMPLATE.documentation-standard"
version: "1.0.0"
last_updated: "[YYYY-MM-DD]"
author: "[Document Author]"
tags: ["[tag1]", "[tag2]", "[tag3]"]
categories: ["[category1]", "[category2]"]
difficulty: "intermediate"
prerequisites: ["[prerequisite1]", "[prerequisite2]"]
related_docs:
- "[related-doc1.md]"
- "[related-doc2.md]"
dependencies:
- "[required-file1]"
- "[required-directory/]"
llm_context: "high"
search_keywords: ["[keyword1]", "[keyword2]", "[keyword3]"]
---
# [DOCUMENT TITLE]
## Purpose
[Brief 2-3 sentence description of what this document is for and who should use it. Be specific about the document's scope and intended audience.]
## When to Use
[Clear list of scenarios when this document should be referenced. Include specific triggers, situations, or conditions that warrant consulting this documentation.]
- **Primary Use Case**: [Main reason to use this document]
- **Secondary Use Cases**: [Additional scenarios]
- **Edge Cases**: [Special circumstances]
- **Avoid When**: [When NOT to use this document]
## How to Use
[Step-by-step instructions for applying the information in this document. Include specific commands, procedures, or workflows that users should follow.]
### Quick Start
```bash
# Essential commands or steps
[Provide working examples]
```
### Prerequisites
- [List any requirements or dependencies]
- [Include system requirements if applicable]
### Step-by-Step Process
1. **[Step 1]**: [Description and command]
2. **[Step 2]**: [Description and command]
3. **[Continue as needed]**
## What It Is
[Comprehensive technical documentation that explains the system, process, or concept in detail. This is the "meat and potatoes" section that provides deep technical understanding.]
### Architecture Overview
[High-level system design, components, and relationships]
### Technical Details
[Detailed explanation of how the system works, including:]
- **Core Components**: [Key parts and their functions]
- **Data Flow**: [How information moves through the system]
- **Dependencies**: [What the system relies on]
- **Configuration**: [How to configure the system]
### Implementation Details
[Specific technical implementation information:]
- **File Structure**: [Relevant files and directories]
- **Code Examples**: [Working code samples with context]
- **Configuration Options**: [Available settings and their effects]
- **Integration Points**: [How this connects with other systems]
### Advanced Topics
[Complex or specialized information for advanced users:]
- **Customization**: [How to modify or extend the system]
- **Performance Considerations**: [Optimization and scaling]
- **Security Implications**: [Security considerations and best practices]
## Troubleshooting
### FAQ
**Q: [Common question about the topic]**
A: [Clear, concise answer with specific steps if applicable]
**Q: [Another common question]**
A: [Answer with examples or references to other sections]
**Q: [Continue with up to 10 frequently asked questions]**
A: [Keep answers updated as new issues are discovered]
### Command Reference
[Essential commands for troubleshooting and maintenance:]
```bash
# [Command description]
[command] [options] [arguments]
# [Another command description]
[command] [options] [arguments]
# [Continue with commonly used commands]
```
### Common Issues and Solutions
| Issue | Symptoms | Solution |
| -------------------- | -------------------- | -------------------- |
| [Problem 1] | [How to identify] | [Step-by-step fix] |
| [Problem 2] | [How to identify] | [Step-by-step fix] |
| [Continue as needed] | [Continue as needed] | [Continue as needed] |
### Debugging Steps
1. **[Check X]**: [How to verify this component]
2. **[Check Y]**: [How to verify this component]
3. **[Check Z]**: [How to verify this component]
### Log Analysis
[How to read and interpret relevant logs:]
```bash
# [Command to view logs]
[command] | grep [pattern]
# [What to look for in logs]
[explanation of log patterns and their meanings]
```
### Performance Troubleshooting
[Specific guidance for performance-related issues:]
- **Resource Usage**: [How to check CPU, memory, disk usage]
- **Bottlenecks**: [Common performance bottlenecks and solutions]
- **Monitoring**: [How to monitor system health]
### Lessons Learned
**YYYY-MM-DD**: [Description of what was learned, what problem it solved, and how it improves the documentation]
**YYYY-MM-DD**: [Another lesson learned with timestamp]
[Continue adding lessons learned entries as new insights are gained]
## Related Documentation
[Links to related documents and their relationships:]
- **[Related Doc 1]**: [Brief description of relationship]
- **[Related Doc 2]**: [Brief description of relationship]
- **[Related Doc 3]**: [Brief description of relationship]
## LLM Context
[Additional context specifically for Large Language Models:]
- **Key Concepts**: [Important concepts for AI understanding]
- **Technical Context**: [Technical background information]
- **Common Patterns**: [Frequently used patterns or approaches]
- **Edge Cases**: [Important edge cases to consider]
- **Integration Points**: [How this relates to other systems]
---
_This document follows the Sirius Documentation Standard. For questions about documentation structure, see [ABOUT.documentation.md](../ABOUT.documentation.md)._
documentation/dev/templates/TEMPLATE.guide.md
+231
View File
@@ -0,0 +1,231 @@
---
title: "[GUIDE TITLE]"
description: "[Brief description of the step-by-step guide]"
template: "TEMPLATE.guide"
version: "1.0.0"
last_updated: "[YYYY-MM-DD]"
author: "[Guide Author]"
tags: ["[tag1]", "[tag2]", "[tag3]"]
categories: ["[category1]", "[category2]"]
difficulty: "intermediate"
prerequisites: ["[prerequisite1]", "[prerequisite2]"]
related_docs:
- "[related-doc1.md]"
- "[related-doc2.md]"
dependencies:
- "[required-file1]"
- "[required-directory/]"
llm_context: "medium"
search_keywords: ["[keyword1]", "[keyword2]", "[keyword3]"]
estimated_time: "[X minutes|hours]"
---
# [GUIDE TITLE]
## Purpose
[Brief description of what this guide accomplishes and who should follow it. Focus on the end result or goal.]
## When to Use This Guide
[Specific scenarios when this guide should be followed:]
- **Primary Scenario**: [Main use case for this guide]
- **Alternative Scenarios**: [Other situations where this applies]
- **Prerequisites Met**: [When you have the required knowledge/tools]
## How to Use This Guide
[Instructions for following the guide effectively:]
- **Read through completely** before starting
- **Gather prerequisites** listed below
- **Follow steps in order** unless noted otherwise
- **Verify each step** before proceeding to the next
### Quick Start
```bash
# Essential commands to get started
[Provide working examples]
```
### Prerequisites
**Knowledge Required:**
- [Required knowledge or skills]
- [Familiarity with specific tools]
**Tools Required:**
- [Required software or tools]
- [System requirements]
**Access Required:**
- [Required permissions or access]
- [Required accounts or credentials]
## Step-by-Step Instructions
### Step 1: [Step Title]
**Objective**: [What this step accomplishes]
**Instructions**:
1. [Detailed instruction]
2. [Next instruction]
3. [Continue as needed]
**Verification**:
```bash
# Command to verify this step worked
[verification command]
```
**Expected Output**:
```
[Expected output or result]
```
**Troubleshooting**:
- **If X happens**: [Solution]
- **If Y happens**: [Solution]
### Step 2: [Step Title]
[Continue with detailed steps...]
### Step 3: [Step Title]
[Continue with detailed steps...]
## Verification and Testing
### Final Verification
[How to verify the entire process worked correctly:]
```bash
# Commands to verify everything is working
[verification commands]
```
### Testing Checklist
- [ ] [Test item 1]
- [ ] [Test item 2]
- [ ] [Test item 3]
- [ ] [Continue as needed]
## Common Variations
### Variation 1: [Scenario Name]
[When to use this variation and how it differs:]
**Additional Steps**:
1. [Additional step]
2. [Another additional step]
**Skip These Steps**:
- [Step to skip]
- [Another step to skip]
### Variation 2: [Scenario Name]
[Another common variation...]
## Troubleshooting
### FAQ
**Q: What if Step X fails?**
A: [Detailed solution with steps]
**Q: How do I know if I'm on the right track?**
A: [How to verify progress]
**Q: Can I skip Step Y?**
A: [When it's safe to skip and what to do instead]
**Q: What if I get error message Z?**
A: [Specific error resolution]
### Common Issues and Solutions
| Issue | Symptoms | Solution |
| -------------------- | -------------------- | -------------------- |
| [Common problem 1] | [How to identify] | [Step-by-step fix] |
| [Common problem 2] | [How to identify] | [Step-by-step fix] |
| [Continue as needed] | [Continue as needed] | [Continue as needed] |
### Rollback Instructions
[How to undo the changes if something goes wrong:]
1. **[Rollback Step 1]**: [How to undo this step]
2. **[Rollback Step 2]**: [How to undo this step]
3. **[Continue as needed]**: [Continue rollback steps]
### Getting Help
**If you're stuck:**
1. Check the troubleshooting section above
2. Review the prerequisites
3. Verify you followed each step exactly
4. Check the related documentation
5. [Contact information or escalation path]
## Advanced Topics
### Customization Options
[How to customize the process for specific needs:]
- **Option A**: [Description and how to implement]
- **Option B**: [Description and how to implement]
### Performance Optimization
[Tips for making the process faster or more efficient:]
- [Optimization tip 1]
- [Optimization tip 2]
### Integration with Other Systems
[How this process integrates with other workflows:]
- [Integration point 1]
- [Integration point 2]
## Related Documentation
[Links to related guides and documentation:]
- **[Related Guide 1]**: [Brief description of relationship]
- **[Related Guide 2]**: [Brief description of relationship]
- **[Reference Doc]**: [Technical reference for this topic]
## LLM Context
[Additional context for Large Language Models:]
- **Process Overview**: [High-level process description]
- **Key Decision Points**: [Important choices in the process]
- **Common Pitfalls**: [Frequent mistakes to avoid]
- **Success Criteria**: [How to know the process succeeded]
- **Integration Points**: [How this connects to other systems]
---
_This document follows the Sirius Documentation Standard. For questions about documentation structure, see [ABOUT.documentation.md](../ABOUT.documentation.md)._
documentation/dev/templates/TEMPLATE.reference.md
+222
View File
@@ -0,0 +1,222 @@
---
title: "Reference Documentation Template"
description: "This template defines the standard structure and conventions for REFERENCE documents within the Sirius project, ensuring comprehensive technical specifications and API documentation."
template: "TEMPLATE.template"
version: "1.0.0"
last_updated: "2025-01-03"
author: "AI Assistant"
tags: ["template", "reference", "technical", "specifications", "api"]
categories: ["project-management", "development"]
difficulty: "intermediate"
prerequisites: ["ABOUT.documentation.md"]
related_docs:
- "ABOUT.documentation.md"
- "TEMPLATE.documentation-standard.md"
dependencies: []
llm_context: "high"
search_keywords:
[
"reference template",
"technical specs",
"api documentation",
"specifications",
]
---
# [System/Component] Reference
## Purpose
This document provides comprehensive technical reference information for [specific system/component/API] within the Sirius project. REFERENCE documents serve as authoritative sources for technical specifications, API endpoints, configuration options, and implementation details.
## When to Use
- **API Development**: When implementing or consuming APIs
- **Integration Work**: When connecting systems or components
- **Configuration**: When setting up or modifying system parameters
- **Debugging**: When troubleshooting technical issues
- **Code Review**: When reviewing implementation against specifications
## How to Use
### Quick Reference
1. **Find the relevant section** for your specific need
2. **Check the prerequisites** and requirements
3. **Follow the examples** for implementation patterns
4. **Reference the specifications** for exact details
5. **Use the troubleshooting section** for common issues
### Common Patterns
```bash
# [Common pattern 1]
[example implementation]
# [Common pattern 2]
[example implementation]
```
## What It Is
### Technical Overview
[Detailed technical explanation of the system/component/API]
### Architecture
[Technical architecture with diagrams and component relationships]
### Data Models
#### [Model 1]
```json
{
"field1": "type",
"field2": "type",
"field3": {
"nested_field": "type"
}
}
```
#### [Model 2]
```yaml
field1: type
field2: type
field3:
nested_field: type
```
### API Endpoints
#### [Endpoint 1]
- **Method**: `GET|POST|PUT|DELETE`
- **Path**: `/api/v1/endpoint`
- **Description**: [What this endpoint does]
- **Parameters**:
- `param1` (required): [Description]
- `param2` (optional): [Description]
- **Response**: [Response format and status codes]
- **Example**:
```bash
curl -X GET "http://localhost:9001/api/v1/endpoint?param1=value"
```
#### [Endpoint 2]
- **Method**: `GET|POST|PUT|DELETE`
- **Path**: `/api/v1/endpoint2`
- **Description**: [What this endpoint does]
- **Request Body**: [Request format]
- **Response**: [Response format and status codes]
- **Example**:
```bash
curl -X POST "http://localhost:9001/api/v1/endpoint2" \
-H "Content-Type: application/json" \
-d '{"field1": "value", "field2": "value"}'
```
### Configuration Options
| Option | Type | Default | Description | Required |
| --------- | ------- | --------- | ------------- | -------- |
| `option1` | string | "default" | [Description] | Yes |
| `option2` | integer | 0 | [Description] | No |
| `option3` | boolean | false | [Description] | No |
### Environment Variables
| Variable | Description | Example | Required |
| ---------- | ------------- | -------- | -------- |
| `ENV_VAR1` | [Description] | `value1` | Yes |
| `ENV_VAR2` | [Description] | `value2` | No |
### File Formats
#### [Format 1]
```yaml
# YAML format example
key1: value1
key2:
nested_key: nested_value
list_item:
- item1
- item2
```
#### [Format 2]
```json
{
"key1": "value1",
"key2": {
"nested_key": "nested_value",
"list_item": ["item1", "item2"]
}
}
```
### Error Codes
| Code | HTTP Status | Description | Resolution |
| ------ | ----------- | ------------------- | ------------ |
| `E001` | 400 | [Error description] | [How to fix] |
| `E002` | 401 | [Error description] | [How to fix] |
| `E003` | 500 | [Error description] | [How to fix] |
## Troubleshooting
### FAQ
**Q: [Common technical question 1]**
A: [Technical answer with specific details]
**Q: [Common technical question 2]**
A: [Technical answer with specific details]
**Q: [Common technical question 3]**
A: [Technical answer with specific details]
### Command Reference
| Command | Purpose | Example | Notes |
| ------------ | -------------- | ----------- | ------------------ |
| `[command1]` | [What it does] | `[example]` | [Additional notes] |
| `[command2]` | [What it does] | `[example]` | [Additional notes] |
| `[command3]` | [What it does] | `[example]` | [Additional notes] |
### Common Issues
| Issue | Symptoms | Root Cause | Solution |
| --------- | ----------------- | ---------------- | ------------------ |
| [Issue 1] | [How to identify] | [Why it happens] | [Step-by-step fix] |
| [Issue 2] | [How to identify] | [Why it happens] | [Step-by-step fix] |
| [Issue 3] | [How to identify] | [Why it happens] | [Step-by-step fix] |
### Debugging Steps
1. **Check [specific thing]**: [How to check and what to look for]
2. **Verify [specific thing]**: [How to verify and expected results]
3. **Test [specific thing]**: [How to test and success criteria]
4. **Review [specific thing]**: [What to review and common patterns]
## Lessons Learned
### [Date] - [What was learned]
[Description of technical lesson learned and how it improved the system]
### [Date] - [What was learned]
[Description of technical lesson learned and how it improved the system]
---
_This document follows the Sirius Documentation Standard. For questions about documentation structure, see [ABOUT.documentation.md](../ABOUT.documentation.md)._
documentation/dev/templates/TEMPLATE.template.md
+191
View File
@@ -0,0 +1,191 @@
---
title: "Template Documentation Template"
description: "This template defines the standard structure and conventions for template documents within the Sirius project, ensuring consistency and clarity for all template files."
template: "TEMPLATE.template"
version: "1.0.0"
last_updated: "2025-01-03"
author: "AI Assistant"
tags: ["template", "meta-template", "structure", "standards"]
categories: ["project-management", "development"]
difficulty: "beginner"
prerequisites: ["ABOUT.documentation.md"]
related_docs:
- "ABOUT.documentation.md"
- "TEMPLATE.documentation-standard.md"
dependencies: []
llm_context: "medium"
search_keywords:
[
"template template",
"meta-template",
"template structure",
"documentation template",
]
---
# [Template Name] Template
## Purpose
This template defines the standard structure and conventions for [specific type] documents within the Sirius project. Template documents serve as blueprints for creating consistent, high-quality documentation.
## When to Use
- **Creating new templates** - When defining structure for new document types
- **Updating existing templates** - When improving template structure
- **Template maintenance** - When ensuring template consistency
- **Onboarding contributors** - When explaining template structure
## How to Use
### Quick Start
1. **Copy this template** for your new template type
2. **Update the title and description** to match your template's purpose
3. **Define the structure** with clear sections and subsections
4. **Add examples** for each section to guide users
5. **Include placeholders** for common content patterns
6. **Test the template** by creating a sample document
### Template Structure
```markdown
# [Document Title]
## Purpose
[What this document is for]
## When to Use
[When to reference this document]
## How to Use
[How to apply the information]
## What It Is
[Detailed technical content]
## Troubleshooting
[Problem-solving section]
### FAQ
[Frequently asked questions]
### Command Reference
[Common commands]
### Common Issues
[Common problems and solutions]
## Lessons Learned
[Timestamped insights]
```
## What It Is
### Template Components
- **[Component 1]**: [Description and purpose]
- **[Component 2]**: [Description and purpose]
- **[Component 3]**: [Description and purpose]
### Required Sections
1. **Purpose** - Clear explanation of what the template is for
2. **When to Use** - Specific scenarios for using this template
3. **How to Use** - Step-by-step instructions for using the template
4. **What It Is** - Detailed explanation of the template structure
5. **Troubleshooting** - Common issues and solutions
### Optional Sections
- **Examples** - Sample content for each section
- **Best Practices** - Guidelines for effective use
- **Common Patterns** - Recurring content patterns
- **Integration Notes** - How this template works with others
### Placeholder System
Use consistent placeholders throughout the template:
- `[Document Title]` - Replace with actual document title
- `[Section Name]` - Replace with specific section names
- `[Description]` - Replace with detailed descriptions
- `[Example]` - Replace with working examples
- `[Command]` - Replace with actual commands
### YAML Front Matter
Every template must include complete YAML front matter:
```yaml
---
title: "[Template Name] Template"
description: "Brief description of template purpose"
template: "TEMPLATE.template"
version: "1.0.0"
last_updated: "YYYY-MM-DD"
author: "Template Author"
tags: ["template", "specific-type", "structure"]
categories: ["project-management", "development"]
difficulty: "beginner"
prerequisites: ["ABOUT.documentation.md"]
related_docs:
- "ABOUT.documentation.md"
- "TEMPLATE.documentation-standard.md"
dependencies: []
llm_context: "medium"
search_keywords: ["template", "specific-type", "structure"]
---
```
## Troubleshooting
### FAQ
**Q: How do I create a new template?**
A: Copy this template, update the title and description, define the structure, and add examples for each section.
**Q: What if I need to deviate from the standard structure?**
A: Document the deviation clearly and consider whether a new template type is needed.
**Q: How detailed should template examples be?**
A: Include enough detail to guide users but keep them concise and focused.
**Q: Should templates include all possible sections?**
A: Include required sections and common optional sections, but don't over-complicate.
### Command Reference
| Command | Purpose | Example |
| -------------------------------- | ------------------- | ---------------------------------------------- |
| `cp template.md new-template.md` | Copy template | `cp TEMPLATE.template.md TEMPLATE.new-type.md` |
| `grep -r "template:"` | Find template usage | `grep -r "template:" documentation/` |
| `grep -r "TEMPLATE\."` | Find all templates | `grep -r "TEMPLATE\." documentation/` |
### Common Issues
| Issue | Symptoms | Solution |
| ------------------------- | ------------------------------ | ------------------------------------------------- |
| Template too complex | Users confused | Simplify structure and add more examples |
| Template too simple | Missing guidance | Add more detailed instructions and examples |
| Inconsistent placeholders | Confusion about replacements | Standardize placeholder format and document usage |
| Missing examples | Users don't know what to write | Add comprehensive examples for each section |
## Lessons Learned
### [Date] - [What was learned]
[Description of template lesson learned and how it improved the system]
---
_This document follows the Sirius Documentation Standard. For questions about documentation structure, see [ABOUT.documentation.md](../ABOUT.documentation.md)._
documentation/dev/templates/TEMPLATE.troubleshooting.md
+334
View File
@@ -0,0 +1,334 @@
---
title: "[TROUBLESHOOTING TITLE]"
description: "[Brief description of the troubleshooting focus area]"
template: "TEMPLATE.troubleshooting"
version: "1.0.0"
last_updated: "[YYYY-MM-DD]"
author: "[Document Author]"
tags: ["[tag1]", "[tag2]", "[tag3]"]
categories: ["[category1]", "[category2]"]
difficulty: "intermediate"
prerequisites: ["[prerequisite1]", "[prerequisite2]"]
related_docs:
- "[related-doc1.md]"
- "[related-doc2.md]"
dependencies:
- "[required-file1]"
- "[required-directory/]"
llm_context: "high"
search_keywords: ["[keyword1]", "[keyword2]", "[keyword3]"]
---
# [TROUBLESHOOTING TITLE]
## Purpose
[Brief description of what problems this troubleshooting guide addresses and who should use it. Focus on the specific issue domain.]
## When to Use This Guide
[Specific scenarios when this troubleshooting guide should be consulted:]
- **Primary Issues**: [Main problems this guide addresses]
- **Error Messages**: [Specific error messages this guide covers]
- **System States**: [When the system is in a problematic state]
- **Before Escalating**: [When to try this guide before seeking help]
## How to Use This Guide
[Instructions for effective troubleshooting:]
1. **Start with Quick Fixes** - Try the most common solutions first
2. **Follow the Flow** - Use the diagnostic flow chart if available
3. **Gather Information** - Collect relevant logs and system state
4. **Test Incrementally** - Apply fixes one at a time and test
5. **Document Results** - Note what worked and what didn't
### Quick Diagnostic
```bash
# Essential diagnostic commands
[Provide key diagnostic commands]
```
### Information Gathering
**Before troubleshooting, collect:**
- [Required information 1]
- [Required information 2]
- [Required information 3]
## Common Problems and Solutions
### Problem 1: [Problem Name]
**Symptoms:**
- [Symptom 1]
- [Symptom 2]
- [Symptom 3]
**Root Cause:**
[Explanation of what causes this problem]
**Solution:**
1. [Step 1 of solution]
2. [Step 2 of solution]
3. [Step 3 of solution]
**Verification:**
```bash
# Command to verify the fix worked
[verification command]
```
**Prevention:**
[How to prevent this problem in the future]
### Problem 2: [Problem Name]
[Continue with additional problems...]
## Diagnostic Flow Chart
```
Start
↓
[First diagnostic question]
↓
Yes → [Action A] → [Next question]
↓
No → [Action B] → [Next question]
↓
[Continue flow...]
↓
End
```
## Error Message Reference
### Error: [Error Message]
**Meaning**: [What this error means]
**Common Causes**:
- [Cause 1]
- [Cause 2]
- [Cause 3]
**Solutions**:
1. [Solution 1]
2. [Solution 2]
3. [Solution 3]
### Error: [Another Error Message]
[Continue with additional error messages...]
## Command Reference
### Diagnostic Commands
```bash
# [Command description]
[command] [options]
# [Another diagnostic command]
[command] [options]
# [Continue with diagnostic commands]
```
### Fix Commands
```bash
# [Fix command description]
[command] [options]
# [Another fix command]
[command] [options]
# [Continue with fix commands]
```
### Verification Commands
```bash
# [Verification command description]
[command] [options]
# [Another verification command]
[command] [options]
# [Continue with verification commands]
```
## Log Analysis
### Key Log Locations
- **[Log 1]**: [Location and purpose]
- **[Log 2]**: [Location and purpose]
- **[Log 3]**: [Location and purpose]
### Log Patterns to Look For
**Success Patterns:**
```
[Pattern that indicates success]
```
**Error Patterns:**
```
[Pattern that indicates errors]
```
**Warning Patterns:**
```
[Pattern that indicates warnings]
```
### Log Analysis Commands
```bash
# [Command to analyze logs]
[command] | grep [pattern]
# [Another log analysis command]
[command] | grep [pattern]
# [Continue with log analysis commands]
```
## System State Checks
### Health Check Commands
```bash
# [Health check description]
[command]
# [Another health check]
[command]
# [Continue with health checks]
```
### Resource Monitoring
```bash
# [Resource monitoring command]
[command]
# [Another resource check]
[command]
# [Continue with resource checks]
```
## Advanced Troubleshooting
### Deep Dive Diagnostics
[Advanced diagnostic techniques for complex issues:]
1. **[Advanced Technique 1]**: [Description and when to use]
2. **[Advanced Technique 2]**: [Description and when to use]
3. **[Advanced Technique 3]**: [Description and when to use]
### Performance Troubleshooting
[Specific guidance for performance-related issues:]
- **CPU Issues**: [How to diagnose and fix]
- **Memory Issues**: [How to diagnose and fix]
- **Disk Issues**: [How to diagnose and fix]
- **Network Issues**: [How to diagnose and fix]
### Integration Troubleshooting
[Issues related to system integration:]
- **Service A Integration**: [Common issues and solutions]
- **Service B Integration**: [Common issues and solutions]
- **External System Integration**: [Common issues and solutions]
## Escalation Procedures
### When to Escalate
[Criteria for when to escalate to higher support levels:]
- [Escalation criteria 1]
- [Escalation criteria 2]
- [Escalation criteria 3]
### Information to Include
[What information to gather before escalating:]
- [Required information 1]
- [Required information 2]
- [Required information 3]
### Escalation Contacts
- **Level 1**: [Contact information]
- **Level 2**: [Contact information]
- **Level 3**: [Contact information]
## Prevention Strategies
### Best Practices
[How to prevent these problems from occurring:]
- [Best practice 1]
- [Best practice 2]
- [Best practice 3]
### Monitoring and Alerting
[How to monitor for these issues:]
- [Monitoring approach 1]
- [Monitoring approach 2]
- [Monitoring approach 3]
### Regular Maintenance
[Maintenance tasks to prevent issues:]
- [Maintenance task 1]
- [Maintenance task 2]
- [Maintenance task 3]
## Related Documentation
[Links to related troubleshooting and reference documentation:]
- **[Related Troubleshooting Guide]**: [Brief description]
- **[System Reference]**: [Technical reference]
- **[Configuration Guide]**: [Configuration documentation]
## LLM Context
[Additional context for Large Language Models:]
- **Problem Patterns**: [Common problem patterns to recognize]
- **Solution Strategies**: [General approaches to problem-solving]
- **System Dependencies**: [How different components interact]
- **Failure Modes**: [Common ways the system can fail]
- **Recovery Procedures**: [How to recover from various failure states]
---
_This document follows the Sirius Documentation Standard. For questions about documentation structure, see [ABOUT.documentation.md](../ABOUT.documentation.md)._
documentation/dev/test/ABOUT.testing.md
+370
View File
@@ -0,0 +1,370 @@
---
title: "About Testing in Sirius"
description: "Comprehensive guide to the Sirius testing philosophy, infrastructure, and operator-first testing approach for integration and validation testing."
template: "TEMPLATE.about"
version: "1.0.0"
last_updated: "2025-01-03"
author: "Development Team"
tags: ["testing", "philosophy", "integration", "operator-first", "validation", "qa"]
categories: ["testing", "philosophy"]
difficulty: "intermediate"
prerequisites: ["docker", "testing concepts", "integration testing"]
related_docs:
- "README.container-testing.md"
- "README.documentation-testing.md"
- "README.cicd.md"
dependencies:
- "testing/container-testing/"
- "testing/documentation/"
- "scripts/git-hooks/"
llm_context: "high"
search_keywords:
[
"testing philosophy",
"operator-first testing",
"integration testing",
"validation testing",
"automated qa",
"standalone tests",
"testing infrastructure"
]
---
# About Testing in Sirius
## Overview
Testing in Sirius follows an **operator-first philosophy** that prioritizes real-world validation over traditional unit testing. Our testing approach focuses on proving that complete systems, processes, and integrations work correctly in realistic scenarios, rather than testing individual code units in isolation.
## Testing Philosophy
### Operator-First Approach
**Core Principle**: Tests should validate that operators can successfully use the system to accomplish real tasks.
**What This Means**:
- Tests simulate actual user workflows and operations
- Validation focuses on end-to-end functionality
- Tests prove that complete processes work as intended
- Results must be actionable and meaningful to operators
**Example**: Instead of testing a scanner function in isolation, we test that the scanner can be invoked standalone, process real data, and produce valid results that an operator would actually use.
### Integration-Focused Testing
**Primary Goal**: Validate that different components work together correctly.
**Testing Scope**:
- Cross-service communication
- Data flow between components
- End-to-end process validation
- Real-world scenario simulation
**Benefits**:
- Catches integration issues that unit tests miss
- Validates complete workflows
- Ensures system reliability for operators
- Provides confidence in production deployments
## Current Testing Infrastructure
### Container Testing
**Location**: `testing/container-testing/`
**Purpose**: Validate Docker container builds, health, and integration
**What's Tested**:
- Docker Compose configuration validation
- Individual container builds (all targets)
- Service health checks
- Cross-service communication
- Integration testing
**Automation**:
- `make test-all` - Complete test suite
- `make test-build` - Build validation
- `make test-health` - Health checks
- `make test-integration` - Integration testing
### Documentation Testing
**Location**: `testing/documentation/`
**Purpose**: Validate documentation completeness, accuracy, and consistency
**What's Tested**:
- YAML front matter completeness
- Template compliance
- Index completeness
- Link validation
- Metadata validity
**Automation**:
- `make lint-docs` - Full documentation validation
- `make lint-docs-quick` - Quick validation
- `make lint-index` - Index completeness check
### Pre-commit Validation
**Location**: `scripts/git-hooks/`
**Purpose**: Quick validation before commits
**What's Tested**:
- Docker Compose configuration validity
- Basic syntax checks
- Quick documentation validation
- Code formatting
## Testing Standards
### Test Structure
**Standalone Execution**: Every test must be able to run independently without dependencies on the main codebase.
**Real Data**: Tests should use realistic data and scenarios, not mocked or synthetic data.
**Actionable Results**: Test results must provide clear, actionable information for operators and developers.
**Automated Validation**: All tests must be automatable and runnable via command-line tools.
### Test Categories
#### 1. Build Validation Tests
**Purpose**: Ensure components can be built correctly
**Examples**:
- Docker container builds
- Multi-stage build validation
- Architecture compatibility
- Dependency resolution
**Validation Criteria**:
- Build completes without errors
- All build targets work correctly
- Images are properly tagged
- Dependencies are resolved
#### 2. Health Validation Tests
**Purpose**: Ensure services start and remain healthy
**Examples**:
- Service startup validation
- Health endpoint checks
- Resource availability
- Graceful shutdown
**Validation Criteria**:
- Services start within expected time
- Health endpoints respond correctly
- No critical errors in logs
- Proper resource utilization
#### 3. Integration Validation Tests
**Purpose**: Ensure components work together correctly
**Examples**:
- Cross-service communication
- Database connectivity
- Message queue functionality
- API endpoint integration
**Validation Criteria**:
- Services can communicate
- Data flows correctly
- APIs respond as expected
- Error handling works
#### 4. Process Validation Tests
**Purpose**: Ensure complete workflows function correctly
**Examples**:
- End-to-end scanning processes
- Data processing pipelines
- User authentication flows
- Deployment processes
**Validation Criteria**:
- Complete workflows execute successfully
- Results are valid and usable
- Error conditions are handled
- Performance meets requirements
## Future Testing Objectives
### Next 3 Months
**Priority 1**: Scanner Testing Infrastructure
- Standalone scanner execution tests
- Real vulnerability detection validation
- Output format verification
- Performance benchmarking
**Priority 2**: API Testing Infrastructure
- Endpoint functionality validation
- Authentication and authorization testing
- Data validation and error handling
- Performance and load testing
**Priority 3**: Engine Testing Infrastructure
- Agent communication testing
- Task execution validation
- Resource management testing
- Error recovery testing
### Long-term Vision
**Comprehensive Test Coverage**: Every major component and process will have dedicated testing infrastructure.
**Automated Validation**: All tests will be integrated into CI/CD pipelines and pre-commit hooks.
**Operator Confidence**: Tests will provide operators with confidence that the system works correctly in production.
**Continuous Improvement**: Testing infrastructure will evolve with the system to maintain high quality standards.
## Testing Best Practices
### For Test Developers
1. **Start with Real Scenarios**: Design tests around actual operator workflows
2. **Use Real Data**: Test with realistic data, not synthetic or mocked data
3. **Validate Complete Processes**: Test end-to-end functionality, not just individual components
4. **Make Tests Standalone**: Ensure tests can run independently without external dependencies
5. **Provide Clear Results**: Test output should be actionable and meaningful
### For System Developers
1. **Design for Testability**: Build components that can be tested independently
2. **Expose Health Endpoints**: Provide clear health and status indicators
3. **Use Standard Interfaces**: Follow consistent patterns for APIs and data formats
4. **Document Dependencies**: Clearly document what each component needs to function
5. **Plan for Validation**: Consider how operators will validate that components work correctly
### For Operators
1. **Run Tests Regularly**: Use testing infrastructure to validate system health
2. **Understand Test Results**: Learn to interpret test output for troubleshooting
3. **Report Issues**: Use test failures to identify and report problems
4. **Validate Changes**: Run tests after any system changes or updates
5. **Trust the Tests**: Use test results to make operational decisions
## Testing Tools and Commands
### Container Testing
```bash
# Full test suite
cd testing/container-testing
make test-all
# Individual tests
make test-build
make test-health
make test-integration
# Quick validation
make build-all
```
### Documentation Testing
```bash
# Full documentation validation
cd testing/documentation
make lint-docs
# Quick validation
make lint-docs-quick
make lint-index
```
### Pre-commit Validation
```bash
# Automatic validation
git commit # Runs quick validation automatically
# Manual validation
cd testing/container-testing
make build-all
make lint-docs-quick
```
## Integration with Development
### CI/CD Pipeline
**Pre-commit**: Quick validation (~30 seconds)
- Docker Compose configuration validation
- Basic syntax checks
- Quick documentation validation
**CI/CD**: Full testing (~5-10 minutes)
- Complete Docker builds
- Integration testing
- Health checks
- Cross-service communication
### Local Development
**Available Commands**: Developers can run full test suites locally when needed
**Testing Strategy**: Quick validation for commits, full testing for CI and local validation
## Troubleshooting
### Common Issues
| Issue | Symptoms | Solution |
|-------|----------|----------|
| **Test Failures** | Tests fail unexpectedly | Check logs, verify dependencies, run individual tests |
| **Build Issues** | Docker builds fail | Check Dockerfile syntax, verify base images |
| **Integration Issues** | Services can't communicate | Check network configuration, verify service health |
| **Documentation Issues** | Linting fails | Check YAML front matter, verify template compliance |
### Debugging Commands
```bash
# Check test logs
cd testing/container-testing
make logs
# Run individual tests
make test-build
make test-health
# Check documentation
cd testing/documentation
make lint-docs
```
## Getting Help
### Documentation
- [README.container-testing.md](README.container-testing.md) - Container testing guide
- [README.documentation-testing.md](README.documentation-testing.md) - Documentation testing guide
- [README.cicd.md](../architecture/README.cicd.md) - CI/CD pipeline guide
### Common Issues
- Check test logs for specific error messages
- Verify all dependencies are available
- Ensure proper environment configuration
- Run tests individually to isolate issues
### Support
- Review testing documentation for guidance
- Check CI/CD logs for automated test results
- Use local testing for debugging and validation
- Report issues with specific test output and logs
---
_This testing philosophy guide follows the Sirius Documentation Standard. For specific testing implementation details, see [README.container-testing.md](README.container-testing.md) and [README.documentation-testing.md](README.documentation-testing.md)._
documentation/dev/test/CHECKLIST.testing-by-type.md
+470
View File
@@ -0,0 +1,470 @@
---
title: "Testing Checklists by Issue Type"
description: "Comprehensive testing checklists for different types of changes to ensure thorough validation before merging"
template: "TEMPLATE.documentation-standard"
version: "1.0.0"
last_updated: "2025-10-11"
author: "Development Team"
tags: ["testing", "checklist", "validation", "quality-assurance"]
categories: ["testing", "development"]
difficulty: "beginner"
prerequisites: ["docker", "git", "development-environment"]
related_docs:
- "README.container-testing.md"
- "../operations/README.git-operations.md"
dependencies: []
llm_context: "high"
search_keywords: ["testing", "checklist", "validation", "qa", "issue-types"]
---
# Testing Checklists by Issue Type
## Purpose
This document provides specific testing checklists for different types of changes. Use the appropriate checklist based on what components your fix or feature affects.
## When to Use
- **Before committing** any code changes
- **Before merging** to main/demo branch
- **During code review** to ensure completeness
- **After deployment** for verification
## How to Use
1. Identify which type of change you're making
2. Follow the complete checklist for that type
3. Check off each item as you test
4. Document any failures or issues
5. Only proceed to merge after all checks pass
## Testing Checklists
### Frontend Changes (UI/UX)
**Applies to**: Changes in `sirius-ui/src/pages/`, `sirius-ui/src/components/`
- [ ] **Visual Testing**
- [ ] Component renders correctly in browser
- [ ] Responsive design works on mobile/tablet/desktop
- [ ] Dark mode (if applicable) displays correctly
- [ ] All buttons and interactive elements are clickable
- [ ] No layout breaking or overflow issues
- [ ] **Functional Testing**
- [ ] All forms submit correctly
- [ ] Validation messages display properly
- [ ] Error states are handled gracefully
- [ ] Loading states display when appropriate
- [ ] Success/error toasts appear as expected
- [ ] **Browser Testing**
- [ ] Chrome/Chromium (primary)
- [ ] Firefox (if significant changes)
- [ ] Safari (if significant changes)
- [ ] **Accessibility**
- [ ] Keyboard navigation works
- [ ] Screen reader friendly (if major UI change)
- [ ] Proper ARIA labels (if applicable)
- [ ] **Container Testing**
- [ ] Hot reload works in development mode
- [ ] Production build completes successfully
- [ ] No console errors in browser
**Container Commands**:
```bash
# Development mode test
docker compose up
# Production build test
docker compose -f docker-compose.yaml up --build
```
---
### Backend API Changes (TRPC/Routes)
**Applies to**: Changes in `sirius-ui/src/server/api/routers/`
- [ ] **API Endpoint Testing**
- [ ] Endpoint accepts correct input format
- [ ] Endpoint returns expected data structure
- [ ] Error cases return appropriate error messages
- [ ] HTTP status codes are correct
- [ ] Response time is acceptable
- [ ] **Data Validation**
- [ ] Input validation works (Zod schemas)
- [ ] Required fields are enforced
- [ ] Type coercion works correctly
- [ ] Invalid data is rejected with clear errors
- [ ] **Authentication/Authorization**
- [ ] Protected endpoints require authentication
- [ ] Session/JWT validation works
- [ ] Unauthorized access is blocked
- [ ] **Database Interaction**
- [ ] Queries return expected data
- [ ] No SQL injection vulnerabilities
- [ ] Transactions complete successfully
- [ ] Error handling for database failures
- [ ] **Container Testing**
- [ ] Check logs for errors: `docker compose logs sirius-ui`
- [ ] Verify no crashes or exceptions
- [ ] Test with development and production builds
**Testing Example**:
```bash
# Watch container logs during testing
docker compose logs -f sirius-ui
# Test endpoint via browser console or Postman
# Verify response structure and error handling
```
---
### Database Schema Changes
**Applies to**: Changes in `sirius-ui/prisma/schema.prisma`, migrations
- [ ] **Schema Validation**
- [ ] Prisma schema is valid: `npx prisma validate`
- [ ] Migration generates correctly: `npx prisma migrate dev`
- [ ] No conflicting field types
- [ ] Relationships are correctly defined
- [ ] **Migration Testing**
- [ ] Migration runs successfully on clean database
- [ ] Migration is reversible (if needed)
- [ ] Existing data is preserved (if applicable)
- [ ] Seed script works with new schema
- [ ] **Application Testing**
- [ ] All queries still work
- [ ] CRUD operations function correctly
- [ ] No type errors in TypeScript
- [ ] Prisma Client regenerates correctly
- [ ] **Data Integrity**
- [ ] Foreign keys are enforced
- [ ] Unique constraints work
- [ ] Default values are applied
- [ ] Required fields are enforced
**Container Commands**:
```bash
# Inside container
docker exec -it sirius-ui npx prisma db push --force-reset
docker exec -it sirius-ui npm run seed
# Verify schema
docker exec -it sirius-ui npx prisma validate
# Check generated client
docker exec -it sirius-ui npx prisma generate
```
---
### Authentication Changes
**Applies to**: Changes in `sirius-ui/src/server/auth.ts`, auth-related endpoints
- [ ] **Login/Logout Testing**
- [ ] Can log in with valid credentials
- [ ] Login fails with invalid credentials
- [ ] Logout clears session correctly
- [ ] Session persists across page refreshes
- [ ] **Session Management**
- [ ] Session timeout works correctly
- [ ] JWT tokens are generated properly
- [ ] Token expiration is handled
- [ ] Refresh tokens work (if implemented)
- [ ] **Password Security**
- [ ] Passwords are hashed (never stored plain text)
- [ ] Password reset works correctly
- [ ] Old password verification works
- [ ] Password requirements are enforced
- [ ] **Protected Routes**
- [ ] Unauthorized users are redirected to login
- [ ] Authorized users can access protected pages
- [ ] Session state is checked on route changes
- [ ] **Database State**
- [ ] User records are created/updated correctly
- [ ] Can reset database to default credentials
- [ ] Seed script recreates admin user
**Testing Commands**:
```bash
# Reset database to default credentials
docker exec -it sirius-ui npx prisma db push --force-reset
docker exec -it sirius-ui npm run seed
# Default credentials: admin / password
```
**Important**: After testing authentication changes, **reset the database** so other developers aren't affected:
```bash
docker exec -it sirius-ui npm run seed
```
---
### Docker/Container Changes
**Applies to**: Changes in `Dockerfile`, `docker-compose.yaml`, startup scripts
- [ ] **Build Testing**
- [ ] Development image builds successfully
- [ ] Production image builds successfully
- [ ] Build time is reasonable
- [ ] Image size is acceptable
- [ ] **Runtime Testing**
- [ ] Container starts successfully
- [ ] All processes run correctly
- [ ] Environment variables are loaded
- [ ] Healthchecks pass
- [ ] **Development Mode**
- [ ] Hot reload works
- [ ] Volume mounts work correctly
- [ ] Source code changes reflect immediately
- [ ] Debugging is possible
- [ ] **Production Mode**
- [ ] Optimized build runs correctly
- [ ] No development dependencies in final image
- [ ] Security scanning passes (if applicable)
- [ ] Startup time is acceptable
- [ ] **Multi-Container Testing**
- [ ] All containers start together
- [ ] Services can communicate
- [ ] Networking is configured correctly
- [ ] Dependencies start in correct order
**Testing Commands**:
```bash
# Test development build
docker compose up --build
# Test production build
docker compose -f docker-compose.yaml up --build
# Check container health
docker compose ps
# View logs for debugging
docker compose logs -f [service-name]
# Test rebuild after changes
docker compose down
docker compose up --build
```
---
### Documentation Changes
**Applies to**: Changes in `documentation/` directory
- [ ] **Content Quality**
- [ ] YAML front matter is complete and valid
- [ ] All required sections are present
- [ ] Examples are accurate and tested
- [ ] Links are valid and working
- [ ] Code snippets are correct
- [ ] **Structure Validation**
- [ ] Follows appropriate template
- [ ] Consistent formatting throughout
- [ ] Proper markdown syntax
- [ ] Headings are hierarchical
- [ ] **Documentation Index**
- [ ] File is added to `README.documentation-index.md`
- [ ] Categorized correctly
- [ ] Description is accurate
- [ ] **Pre-commit Checks**
- [ ] Quick documentation checks pass
- [ ] Index completeness validation passes
- [ ] No linting errors
**Testing Commands**:
```bash
cd testing
make lint-docs-quick # Quick validation
make lint-docs # Full validation
make lint-index # Index completeness check
```
---
### Git Operations Changes
**Applies to**: Changes in pre-commit hooks, git workflows
- [ ] **Pre-commit Hook Testing**
- [ ] Hook runs on commit attempt
- [ ] Checks execute correctly
- [ ] Failures block commit appropriately
- [ ] Success allows commit to proceed
- [ ] **Workflow Testing**
- [ ] Test on feature branch
- [ ] Test on main/demo branch
- [ ] Verify branch detection logic
- [ ] Check optimization flags work
- [ ] **Documentation**
- [ ] New workflow is documented
- [ ] Examples are provided
- [ ] Troubleshooting section added
**Testing Commands**:
```bash
# Test pre-commit hook
git add .
git commit -m "test: validate pre-commit hook"
# Test on feature branch
git checkout -b test/pre-commit-test
# Make a change and commit
# Test failure cases
# Intentionally create validation failure
```
---
## Post-Merge Verification
After merging to main/demo, verify:
- [ ] **Container Health**
- [ ] All containers start successfully
- [ ] No errors in logs
- [ ] Services respond to requests
- [ ] **Functionality Check**
- [ ] Core features still work
- [ ] No regressions introduced
- [ ] New feature/fix works as expected
- [ ] **Rollback Preparedness**
- [ ] Know the last good commit hash
- [ ] Can quickly revert if needed
- [ ] Team is notified of deployment
---
## Troubleshooting Testing Issues
### Container Won't Start
```bash
# Check logs
docker compose logs [service-name]
# Rebuild from scratch
docker compose down -v
docker compose build --no-cache
docker compose up
```
### Database Issues
```bash
# Reset database
docker exec -it sirius-ui npx prisma db push --force-reset
docker exec -it sirius-ui npm run seed
```
### Network Issues
```bash
# Check container networking
docker network ls
docker network inspect sirius_default
# Restart containers
docker compose restart
```
### Build Issues
```bash
# Clear Docker cache
docker builder prune -a
# Rebuild specific service
docker compose build --no-cache [service-name]
```
---
## Continuous Improvement
**Add new checklists** as you encounter new types of issues:
1. Document what testing was needed
2. Create a new checklist section
3. Include specific commands and examples
4. Update this document
5. Notify team of new checklist
**Update existing checklists** based on:
- Issues that slipped through testing
- New tools or testing approaches
- Feedback from team members
- Changes in technology stack
---
_This document follows the Sirius Documentation Standard. For questions about documentation structure, see [ABOUT.documentation.md](../ABOUT.documentation.md)._
documentation/dev/test/README.container-testing.md
+470
View File
@@ -0,0 +1,470 @@
---
title: "Container Testing"
description: "Comprehensive container testing system for Sirius Scan, providing automated validation of Docker builds, service health, and integration before production deployment"
template: "TEMPLATE.documentation-standard"
version: "1.0.0"
last_updated: "2025-01-03"
author: "Development Team"
tags: ["docker", "testing", "containers", "ci-cd", "validation"]
categories: ["development", "testing", "infrastructure"]
difficulty: "intermediate"
prerequisites: ["docker", "make", "bash"]
related_docs:
- "README.development.md"
- "ABOUT.documentation.md"
dependencies:
- "docker-compose.yaml"
- "testing/container-testing/"
- "testing/Makefile"
llm_context: "high"
search_keywords:
["container", "testing", "docker", "health-check", "validation", "ci-cd"]
---
# Container Testing
## Purpose
This document describes the comprehensive container testing system for SiriusScan, providing automated validation of Docker builds, service health, and integration before production deployment. It serves as the definitive guide for developers and DevOps engineers who need to ensure container reliability and system integrity.
## When to Use
- **Before every production deployment** - Validate that all containers build and function correctly
- **After making changes to Dockerfiles** - Ensure modifications don't break the build process
- **During development cycles** - Catch container issues early in the development process
- **When troubleshooting container issues** - Use health checks and integration tests to diagnose problems
- **In CI/CD pipelines** - Automated validation of container builds and functionality
- **When onboarding new team members** - Understand the testing infrastructure and processes
## How to Use
### Quick Start
```bash
# Navigate to container testing directory
cd testing/container-testing
# Run complete test suite (includes runtime auth contract — see below)
make test-all
# Run specific test types
make test-build # Build validation only
make test-health # Health checks only
make test-integration # Integration tests only
make test-runtime-contract # SIRIUS_API_KEY parity, engine preflight script, API HTTP contract
make test-public-ghcr # anonymous GHCR access for the compose-rendered IMAGE_TAG
make test-release-gates # test-build + test-integration + test-runtime-contract (release bar)
# Test specific environment
make test-dev # Development environment
make test-prod # Production environment
```
**Alternative: Run scripts directly from project root**
```bash
# From project root directory
./testing/container-testing/test-integration.sh
./testing/container-testing/test-health.sh
./testing/container-testing/test-build.sh
```
### Prerequisites
- Docker Engine 20.10.0+
- Docker Compose V2
- Make utility
- curl, nc (netcat), jq (for health checks)
- 4GB RAM minimum (8GB recommended)
### Step-by-Step Process
1. **Navigate to container testing directory**: `cd /path/to/Sirius/testing/container-testing`
2. **Run build tests**: `make test-build` to validate all containers build successfully
3. **Run health tests**: `make test-health` to verify services start and respond correctly
4. **Run integration tests**: `make test-integration` to test service interactions
5. **`make test-all` also runs `test-runtime-contract`**, which starts the stack from the repo root and runs `scripts/verify-runtime-auth-contract.sh` (shared `SIRIUS_API_KEY`, postgres parity, authenticated `GET /host/`).
6. **Review results**: Check test output and logs in `testing/logs/` directory
**Alternative approach** (from project root):
1. **Navigate to project root**: `cd /path/to/Sirius`
2. **Run tests directly**: `./testing/container-testing/test-integration.sh`
3. **Review results**: Check test output and logs in `testing/logs/` directory
## What It Is
### Architecture Overview
The container testing system consists of these main components:
- **Build Validation**: Tests individual container builds and Docker Compose configurations
- **Health Checks**: Validates service startup, connectivity, and basic functionality
- **Integration Tests**: Verifies inter-service communication and data flow
- **Runtime Auth Contract** (`make test-runtime-contract`): Runs [`scripts/verify-runtime-auth-contract.sh`](../../../scripts/verify-runtime-auth-contract.sh) against a running stack. Default container names match `container_name` in `docker-compose.yaml` (`sirius-ui`, `sirius-api`, etc.). For a non-default Compose project (e.g. `name: sirius-test`), set `SIRIUS_CONTRACT_CONTAINER_UI`, `SIRIUS_CONTRACT_CONTAINER_API`, `SIRIUS_CONTRACT_CONTAINER_ENGINE`, and `SIRIUS_CONTRACT_CONTAINER_POSTGRES` before running the script. Use `SIRIUS_API_PUBLIC_URL` if the API is not on `http://localhost:9001`.
- **Public GHCR Contract** (`make test-public-ghcr`): Runs [`scripts/verify-ghcr-public-access.sh`](../../../scripts/verify-ghcr-public-access.sh) with the selected `IMAGE_TAG` and checks the exact image refs rendered by `docker-compose.yaml`. Failures distinguish `unauthorized` (visibility/token problem) from `manifest unknown` (tag publication problem).
### Technical Details
#### Build Testing System
**Purpose**: Ensures all containers can be built successfully with correct configurations.
**Components**:
- `testing/container-testing/test-build.sh` - Main build validation script
- Tests all Docker Compose configurations (base, dev, prod)
- Validates individual container builds for all services
- Cleans up test images after completion
**Services Tested**:
- `sirius-ui` (Next.js frontend)
- `sirius-api` (Go REST API)
- `sirius-engine` (Multi-service container)
- `sirius-postgres` (Database)
- `sirius-valkey` (Cache)
- `sirius-rabbitmq` (Message queue)
**Build Targets Validated**:
- Production builds (`production`, `runner`, `runtime` stages)
- Development builds (`development` stage)
- Multi-stage Dockerfile validation
#### Health Check System
**Purpose**: Verifies that all services start correctly and respond to basic health checks.
**Components**:
- `testing/container-testing/test-health.sh` - Health validation script
- Service-specific health endpoints
- Database connectivity tests
- Port accessibility validation
**Health Checks Performed**:
1. **Container Status**: All 6 containers running and healthy
2. **UI Health**: Next.js application serving content on port 3000
3. **API Health**: Go API responding on port 9001 with `/health` endpoint
4. **Engine Health**: Process validation for sirius-engine container
5. **Database Health**: PostgreSQL connectivity and readiness
6. **Cache Health**: Valkey/Redis responding to ping commands
7. **Queue Health**: RabbitMQ broker status and connectivity
8. **Port Accessibility**: External port binding validation
#### Integration Testing System
**Purpose**: Tests inter-service communication and data flow between components.
**Components**:
- `testing/container-testing/test-integration.sh` - Integration validation script
- Service-to-service communication tests
- Database integration validation
- Message queue functionality tests
**Integration Tests**:
- API to Database connectivity
- UI to API communication
- Engine to Message Queue integration
- Cross-service data validation
### Implementation Details
#### File Structure
```
testing/
├── container-testing/
│ ├── test-build.sh # Build validation
│ ├── test-health.sh # Health checks
│ ├── test-integration.sh # Integration tests
│ └── Makefile # Container testing commands
├── logs/ # Test execution logs
│ ├── build_test_*.log
│ ├── health_test_*.log
│ └── integration_test_*.log
└── README.md # Testing documentation
```
**Note**: The Makefile is located in the `container-testing/` directory and should be run from there. The test scripts can be run either from the `container-testing/` directory via Makefile or directly from the project root.
#### Makefile Integration
The testing system uses a dedicated Makefile in the `container-testing/` directory:
```makefile
# Testing targets (run from testing/container-testing/)
test-all: test-build test-health test-integration
test-build: ./test-build.sh
test-health: ./test-health.sh
test-integration: ./test-integration.sh
# Environment-specific testing
test-dev: ./test-health.sh dev
test-prod: ./test-health.sh prod
```
**Usage**:
- Run from `testing/container-testing/` directory: `make test-all`
- Or run scripts directly from project root: `./testing/container-testing/test-integration.sh`
#### Logging and Reporting
**Log Files**: All test executions create timestamped log files in `testing/logs/`
- Format: `{test_type}_test_{timestamp}.log`
- Contains: Full command output, test results, error details
- Retention: Logs are kept for troubleshooting and analysis
**Test Results**: Each test provides clear pass/fail indicators:
- ✅ PASSED: Test completed successfully
- ❌ FAILED: Test failed with error details
- Summary: Total tests, passed, failed counts
#### Environment Support
**Development Environment**:
- Uses `docker-compose.dev.yaml` overrides
- Includes volume mounts for live development
- Accounts for longer startup times due to dependency downloads
**Production Environment**:
- Uses `docker-compose.prod.yaml` overrides
- Tests production-optimized builds
- Validates production configuration
**Base Environment**:
- Uses standard `docker-compose.yaml`
- Tests default configuration
- Validates core functionality
### Advanced Topics
#### Custom Test Configuration
**Environment Variables**:
```bash
export TEST_RETRIES=120 # Retry attempts (default: 60, dev mode: 120)
export LOG_LEVEL=debug # Logging verbosity (info, debug)
```
**Timeout Configuration**:
- **Default**: 60 retries × 2 seconds = 120 seconds total
- **Development Mode**: Automatically increases to 120 retries × 2 seconds = 240 seconds total
- **Custom**: Override with `TEST_RETRIES` environment variable
**Custom Test Execution**:
```bash
# Run with custom timeout (4 minutes)
TEST_RETRIES=120 ./testing/scripts/test-health.sh dev
# Run with debug logging
LOG_LEVEL=debug ./testing/scripts/test-health.sh dev
# Run with both custom timeout and debug logging
TEST_RETRIES=180 LOG_LEVEL=debug ./testing/scripts/test-health.sh dev
```
#### Performance Monitoring
**Resource Usage Tracking**:
- Container startup times
- Memory and CPU usage during tests
- Build time optimization
- Service response times
**Optimization Opportunities**:
- Parallel test execution
- Build cache utilization
- Service startup optimization
- Resource limit tuning
#### Security Considerations
**Test Isolation**:
- Tests run in isolated Docker networks
- No external network access during testing
- Cleanup procedures remove all test artifacts
**Security Validation**:
- Container security scanning (future enhancement)
- Vulnerability assessment (future enhancement)
- Secret management validation (future enhancement)
## Troubleshooting
### FAQ
**Q: Why do health tests fail with "API not ready yet" errors?**
A: The sirius-api service takes time to download Go dependencies in development mode. The health check automatically increases timeout to 4 minutes for dev mode, and includes fallback validation for this scenario.
**Q: Why does development mode take longer than production mode?**
A: Development mode downloads Go dependencies at runtime, which can take 2-4 minutes depending on your internet connection. Production mode pre-builds these dependencies, so it starts much faster.
**Q: How do I run tests for a specific service only?**
A: Modify the test scripts to comment out unwanted services, or use Docker Compose to start only the services you need to test.
**Q: What if build tests fail with "target stage not found" errors?**
A: Check that Docker Compose files reference correct Dockerfile stage names. Common stages are `development`, `production`, `runtime`, and `runner`.
**Q: How can I increase test timeout for slow systems?**
A: Set the `TEST_RETRIES` environment variable: `export TEST_RETRIES=120 && make test-health` (120 retries = 4 minutes total)
**Q: Why do integration tests fail with connection refused errors?**
A: Ensure all services are fully started before running integration tests. Use `docker compose ps` to verify service status.
**Q: How do I debug a specific test failure?**
A: Check the timestamped log file in `testing/logs/` for detailed error information and command output.
**Q: Can I run tests in parallel?**
A: Currently tests run sequentially to avoid resource conflicts. Parallel execution is planned for future enhancement.
**Q: What if I need to test with custom Docker images?**
A: Modify the test scripts to use your custom image tags, or update the Docker Compose files to reference your images.
### Command Reference
```bash
# From testing/container-testing/ directory
make test-all
make test-build
make test-health
make test-integration
make test-public-ghcr
make test-dev
make test-prod
# Or from project root directory
./testing/container-testing/test-build.sh
./testing/container-testing/test-health.sh dev
./testing/container-testing/test-integration.sh prod
# Docker Compose validation
docker compose config --quiet
docker compose -f docker-compose.yaml -f docker-compose.dev.yaml config --quiet
# Service health checks
docker compose ps
docker compose logs sirius-api
curl -f http://localhost:9001/health
# Container debugging
docker exec -it sirius-engine /bin/bash
docker logs sirius-engine --tail 50 -f
# Cleanup
docker compose down -v
docker system prune -f
```
### Common Issues and Solutions
| Issue | Symptoms | Solution |
| --------------------- | --------------------------------- | --------------------------------------------------------- |
| Build failures | "target stage not found" errors | Check Dockerfile stage names in Compose files |
| Health check timeouts | "service not ready" errors | Increase timeout or check service logs |
| Port conflicts | "port already in use" errors | Stop conflicting services or change ports |
| Permission errors | "permission denied" in containers | Check file permissions and Docker daemon access |
| Network issues | "connection refused" errors | Verify Docker network configuration |
| Resource exhaustion | Out of memory errors | Increase Docker memory limits or close other applications |
### Debugging Steps
1. **Check service status**: `docker compose ps` to see which containers are running
2. **Review service logs**: `docker compose logs [service-name]` for error details
3. **Verify network connectivity**: `docker network ls` and `docker network inspect sirius`
4. **Test individual components**: Run health checks manually with curl commands
5. **Check resource usage**: `docker stats` to monitor CPU and memory usage
6. **Validate configuration**: `docker compose config` to check for syntax errors
### Log Analysis
**Build Test Logs**:
```bash
# View build test results
tail -f testing/logs/build_test_*.log
# Check for specific errors
grep -i "error\|failed" testing/logs/build_test_*.log
```
**Health Test Logs**:
```bash
# Monitor health check progress
tail -f testing/logs/health_test_*.log
# Find failed tests
grep -i "failed\|error" testing/logs/health_test_*.log
```
**Integration Test Logs**:
```bash
# Check integration test results
cat testing/logs/integration_test_*.log
# Analyze service communication
grep -i "connection\|timeout" testing/logs/integration_test_*.log
```
### Performance Troubleshooting
**Slow Build Times**:
- Check Docker BuildKit is enabled: `export DOCKER_BUILDKIT=1`
- Clear Docker cache: `docker builder prune`
- Use build cache mounts for dependencies
**Slow Service Startup**:
- Check system resources: `docker stats`
- Optimize Dockerfile layers
- Use multi-stage builds efficiently
**Test Execution Time**:
- Run tests in parallel (future enhancement)
- Optimize health check intervals
- Use faster base images where possible
### Lessons Learned
**2025-01-03**: Implemented comprehensive container testing system to address build reliability issues. Key insight: Automated testing prevents deployment failures and improves development confidence.
**2025-01-03**: Created health check system with fallback validation for services with longer startup times. Lesson: Account for real-world service startup patterns in test design.
**2025-01-03**: Established layered testing approach (build → health → integration). Benefit: Catches issues at appropriate levels and provides clear failure isolation.
**2025-01-03**: Implemented timestamped logging system for all test executions. Advantage: Enables detailed troubleshooting and historical analysis of test failures.
## LLM Context
[Additional context specifically for Large Language Models:]
- **Key Concepts**: Container testing involves build validation, health checks, and integration testing to ensure Docker containers work correctly before deployment
- **Technical Context**: Uses Docker Compose for orchestration, shell scripts for automation, and Make for command execution. Tests cover 6 services: UI, API, Engine, PostgreSQL, Valkey, and RabbitMQ
- **Common Patterns**: Layered testing approach (build → health → integration), automated cleanup, timestamped logging, environment-specific configurations
- **Edge Cases**: API dependency downloading in dev mode, service startup timing variations, port conflicts, resource constraints
- **Integration Points**: Connects with CI/CD pipelines, deployment processes, monitoring systems, and development workflows
---
_This document follows the Sirius Documentation Standard. For questions about documentation structure, see [ABOUT.documentation.md](../ABOUT.documentation.md)._
documentation/dev/test/README.documentation-testing.md
+437
View File
@@ -0,0 +1,437 @@
---
title: "Documentation Testing"
description: "Comprehensive documentation testing system for Sirius project, providing automated validation of documentation quality, structure, and completeness using specialized linters."
template: "TEMPLATE.documentation-standard"
version: "1.0.0"
last_updated: "2025-01-03"
author: "Development Team"
tags: ["documentation", "testing", "linting", "validation", "quality"]
categories: ["development", "testing", "documentation"]
difficulty: "intermediate"
prerequisites: ["markdown", "yaml", "bash"]
related_docs:
- "README.container-testing.md"
- "ABOUT.documentation.md"
dependencies:
- "scripts/documentation/"
- "testing/Makefile"
llm_context: "high"
search_keywords:
["documentation", "testing", "linting", "validation", "yaml", "markdown"]
---
# Documentation Testing
## Purpose
This document describes the comprehensive documentation testing system for the Sirius project, providing automated validation of documentation quality, structure, and completeness. It serves as the definitive guide for developers and technical writers who need to ensure documentation meets our standards and is optimized for LLM consumption.
## When to Use
- **Before committing documentation changes** - Validate that all documentation meets our standards
- **During documentation reviews** - Ensure consistency and completeness across all docs
- **When creating new documentation** - Verify proper structure and metadata
- **In CI/CD pipelines** - Automated validation of documentation quality
- **When troubleshooting documentation issues** - Use linters to identify specific problems
- **For LLM optimization** - Ensure documentation is properly structured for AI consumption
## How to Use
### Quick Start
```bash
# Navigate to container testing directory
cd testing/container-testing
# Run complete documentation testing
make lint-docs
# Run quick documentation checks
make lint-docs-quick
# Check documentation index completeness
make lint-index
# Run all validation including documentation
make validate-all
```
### Prerequisites
- Bash shell environment
- `yq` utility (for full YAML validation)
- `grep`, `sed`, `awk` (standard Unix tools)
- Make utility
- Git (for pre-commit hooks)
### Step-by-Step Process
1. **Navigate to container testing directory**: `cd /path/to/Sirius/testing/container-testing`
2. **Run quick checks**: `make lint-docs-quick` for basic validation
3. **Run full linting**: `make lint-docs` for comprehensive validation
4. **Check index completeness**: `make lint-index` to verify all files are indexed
5. **Review results**: Check output and logs in `tmp/` directory
## What It Is
### Architecture Overview
The documentation testing system consists of three main components:
- **YAML Front Matter Validation**: Ensures all documents have complete and valid metadata
- **Template Compliance Checking**: Verifies documents follow their specified templates
- **Index Completeness Validation**: Ensures all documentation files are properly indexed
### Technical Details
#### YAML Front Matter Validation
**Purpose**: Ensures all documentation files have complete and valid YAML front matter.
**Components**:
- `scripts/documentation/lint-docs.sh` - Main validation script
- `scripts/documentation/lint-quick.sh` - Quick validation script
- Validates required and optional fields
- Checks field value validity
**Required Fields**:
- `title` - Human-readable document title
- `description` - Brief purpose description
- `template` - Template used to create document
- `version` - Document version
- `last_updated` - Last modification date
**Optional Fields**:
- `author` - Document author
- `tags` - Searchable tags
- `categories` - Document categories
- `difficulty` - Complexity level (beginner, intermediate, advanced)
- `prerequisites` - Required knowledge
- `related_docs` - Related documentation
- `dependencies` - Required files/systems
- `llm_context` - LLM relevance level (high, medium, low)
- `search_keywords` - Search optimization
**Validation Rules**:
- YAML syntax must be valid
- Required fields must be present
- Field values must match valid options
- Template references must exist
- Related documents must exist
#### Template Compliance Checking
**Purpose**: Verifies that documents follow their specified template structure.
**Components**:
- Template file parsing and section extraction
- Document section comparison
- Missing section detection
- Custom document support
**Supported Templates**:
- `TEMPLATE.documentation-standard` - Standard documentation
- `TEMPLATE.guide` - Step-by-step guides
- `TEMPLATE.troubleshooting` - Problem-solving docs
- `TEMPLATE.about` - About documents
- `TEMPLATE.reference` - Technical specifications
- `TEMPLATE.architecture` - System design docs
- `TEMPLATE.api` - API documentation
- `TEMPLATE.template` - Template documents
- `TEMPLATE.custom` - Custom documents (no compliance checking)
**Compliance Rules**:
- Documents must have all required sections from their template
- Section headings must match exactly
- Custom documents are exempt from compliance checking
- Meta-documents (ABOUT files) are exempt from compliance checking
#### Index Completeness Validation
**Purpose**: Ensures all documentation files are properly referenced in the documentation index.
**Components**:
- `scripts/documentation/lint-index.sh` - Index validation script
- File discovery in `documentation/dev/` directory
- Index parsing and link extraction
- Missing file detection
- Extra file detection
**Validation Rules**:
- All files in `dev/` must be referenced in the index
- All referenced files must exist in `dev/`
- No duplicate references allowed
- Footer links are excluded from validation
### Implementation Details
#### File Structure
```
scripts/documentation/
├── lint-docs.sh # Full documentation linting
├── lint-quick.sh # Quick documentation checks
└── lint-index.sh # Index completeness validation
testing/
├── Makefile # Testing commands
└── tmp/ # Linting logs and output
├── documentation-lint-*.log
└── index-lint-*.log
```
#### Makefile Integration
The documentation testing system integrates with the testing Makefile:
```makefile
# Documentation testing targets
lint-docs: ../scripts/documentation/lint-docs.sh
lint-docs-quick: ../scripts/documentation/lint-quick.sh
lint-index: ../scripts/documentation/lint-index.sh
# Full validation including documentation
validate-all: build-all test-all lint-docs
```
#### Logging and Reporting
**Log Files**: All linting executions create timestamped log files in `tmp/`
- Format: `{lint_type}-{timestamp}.log`
- Contains: Full validation output, errors, warnings
- Retention: Logs are kept for troubleshooting
**Validation Results**: Each validation provides clear indicators:
- ✅ PASSED: Validation completed successfully
- ❌ FAILED: Validation failed with error details
- ⚠️ WARNING: Non-critical issues found
- Summary: Total files, passed, failed counts
#### Pre-commit Integration
**Pre-commit Hook**: `.git/hooks/pre-commit` runs documentation validation:
1. **Quick documentation linting** - Basic validation
2. **Index completeness check** - Ensures all files are indexed
3. **Build tests** - Validates system still works
**Hook Behavior**:
- Runs from project root or testing directory
- Fails commit if validation fails
- Provides clear error messages
- Suggests fixes for common issues
### Advanced Topics
#### Custom Validation Rules
**Environment Variables**:
```bash
export DOCS_DIR="../documentation/dev" # Documentation directory
export TEMPLATES_DIR="../documentation/dev/templates" # Templates directory
export LOG_LEVEL="debug" # Logging verbosity
```
**Custom Validation**:
```bash
# Run with custom documentation directory
DOCS_DIR="../custom-docs" make lint-docs
# Run with debug logging
LOG_LEVEL="debug" make lint-docs
# Run with custom template directory
TEMPLATES_DIR="../custom-templates" make lint-docs
```
#### Template Development
**Creating New Templates**:
1. **Copy base template**: `cp TEMPLATE.template.md TEMPLATE.new-type.md`
2. **Define structure**: Add required sections for new document type
3. **Update validation**: Add new template to `VALID_TEMPLATES` array
4. **Test validation**: Run `make lint-docs` to verify template works
**Template Requirements**:
- Must include YAML front matter
- Must define clear section structure
- Must be self-documenting
- Must follow naming conventions
#### LLM Optimization
**Context Building**:
- Rich metadata in front matter
- Consistent structure across documents
- Clear relationships between documents
- Comprehensive search keywords
**AI-Friendly Features**:
- Structured content for predictable parsing
- Detailed descriptions for AI understanding
- Working examples with expected outputs
- Comprehensive troubleshooting information
## Troubleshooting
### FAQ
**Q: Why do I get "Template file not found" warnings?**
A: The quick linter looks for template files in the wrong location. This is a known issue and doesn't affect functionality. Use `make lint-docs` for accurate validation.
**Q: Why does index linting fail with "Missing from index" errors?**
A: Some documentation files are not referenced in the documentation index. Add them to `documentation/README.documentation-index.md` or remove unused files.
**Q: How do I fix "Invalid difficulty value" errors?**
A: Difficulty must be one of: "beginner", "intermediate", "advanced". Check your YAML front matter for typos.
**Q: Why do I get "Missing section" warnings?**
A: Your document is missing required sections from its template. Compare your document structure with the template file.
**Q: How do I add a new document type?**
A: Create a new template file, add it to the validation system, and update the documentation index.
**Q: Why does pre-commit hook fail?**
A: Check the error messages in the terminal. Common issues include missing YAML front matter, invalid field values, or missing index entries.
**Q: How do I disable validation for a specific document?**
A: Use `TEMPLATE.custom` template type, which skips template compliance checking.
**Q: Can I run validation on specific files only?**
A: Modify the linting scripts to target specific files, or use grep to filter the file list.
### Command Reference
```bash
# Complete documentation validation
make lint-docs
# Quick documentation checks
make lint-docs-quick
# Index completeness validation
make lint-index
# Full system validation
make validate-all
# Manual script execution
../scripts/documentation/lint-docs.sh
../scripts/documentation/lint-quick.sh
../scripts/documentation/lint-index.sh
# Check specific validation
grep -r "template:" ../documentation/dev/
grep -r "llm_context: high" ../documentation/dev/
grep -r "difficulty:" ../documentation/dev/
# Debug validation issues
LOG_LEVEL=debug make lint-docs
tail -f tmp/documentation-lint-*.log
```
### Common Issues and Solutions
| Issue | Symptoms | Solution |
| -------------------- | --------------------------------- | --------------------------------------------------- |
| Missing front matter | "No YAML metadata" errors | Add complete YAML front matter with required fields |
| Invalid field values | "Invalid difficulty value" errors | Check field values against valid options |
| Template compliance | "Missing section" warnings | Compare document structure with template |
| Index completeness | "Missing from index" errors | Add files to documentation index |
| Broken links | "File not found" errors | Fix file paths in related_docs |
| YAML syntax errors | "YAML parse error" messages | Fix YAML syntax in front matter |
### Debugging Steps
1. **Check log files**: Review timestamped logs in `tmp/` directory
2. **Run quick validation**: Use `make lint-docs-quick` for basic checks
3. **Validate YAML syntax**: Check front matter for syntax errors
4. **Compare with templates**: Ensure document follows template structure
5. **Check index entries**: Verify all files are properly indexed
6. **Test individual files**: Run validation on specific files
### Log Analysis
**Documentation Lint Logs**:
```bash
# View validation results
tail -f tmp/documentation-lint-*.log
# Check for specific errors
grep -i "error\|failed" tmp/documentation-lint-*.log
# Find warnings
grep -i "warning" tmp/documentation-lint-*.log
```
**Index Lint Logs**:
```bash
# View index validation results
cat tmp/index-lint-*.log
# Check missing files
grep "missing from index" tmp/index-lint-*.log
# Check extra files
grep "extra files in index" tmp/index-lint-*.log
```
### Performance Optimization
**Faster Validation**:
- Use `make lint-docs-quick` for development
- Run validation on specific files only
- Optimize template file parsing
- Cache validation results
**Resource Usage**:
- Monitor disk space for log files
- Clean up old log files regularly
- Use efficient file parsing methods
- Optimize regex patterns
### Lessons Learned
**2025-01-03**: Implemented comprehensive documentation testing system to ensure consistency and quality. Key insight: Automated validation prevents documentation drift and improves maintainability.
**2025-01-03**: Created template compliance checking with support for custom documents. Lesson: Flexibility is important for meta-documents and special cases.
**2025-01-03**: Established index completeness validation to ensure all documentation is discoverable. Benefit: Prevents orphaned documentation and improves navigation.
**2025-01-03**: Integrated documentation testing with pre-commit hooks for automated quality assurance. Advantage: Catches documentation issues before they reach the repository.
## LLM Context
[Additional context specifically for Large Language Models:]
- **Key Concepts**: Documentation testing involves YAML validation, template compliance, and index completeness to ensure documentation quality and consistency
- **Technical Context**: Uses shell scripts for automation, YAML parsing for metadata validation, and regex for content analysis. Tests cover 14 documentation files with 8 different templates
- **Common Patterns**: Layered validation approach (quick → full → index), automated cleanup, timestamped logging, pre-commit integration
- **Edge Cases**: Custom documents exempt from template compliance, meta-documents with unique structure, footer links excluded from index validation
- **Integration Points**: Connects with git hooks, CI/CD pipelines, development workflows, and documentation maintenance processes
---
_This document follows the Sirius Documentation Standard. For questions about documentation structure, see [ABOUT.documentation.md](../ABOUT.documentation.md)._