--- title: "Agent System Architecture" description: "Comprehensive architecture documentation for the Sirius distributed agent system, including gRPC communication, template synchronization, and vulnerability detection" template: "TEMPLATE.documentation-standard" version: "1.0.0" last_updated: "2025-10-25" author: "Sirius Platform Team" tags: [ "agent", "grpc", "architecture", "distributed-system", "template-system", "vulnerability-detection", ] categories: ["architecture", "agent-system", "distributed-systems"] difficulty: "advanced" prerequisites: - "Basic understanding of gRPC and Protocol Buffers" - "Familiarity with Go programming language" - "Understanding of distributed system concepts" - "Knowledge of Docker and containerization" related_docs: - "README.docker-architecture.md" - "README.development.md" - "../README.developer-guide.md" - "../apps/README.agent-template-api.md" dependencies: - "app-agent/" - "sirius-api/" - "proto/hello/" llm_context: "high" search_keywords: [ "agent", "grpc", "distributed", "template", "vulnerability", "detection", "synchronization", "bidirectional-stream", "protobuf", ] --- # Agent System Architecture ## Overview The Sirius Agent System is a distributed architecture for managing remote security agents that perform vulnerability detection and system scanning across heterogeneous environments. The system uses gRPC bidirectional streaming for real-time communication, a template-based detection engine, and a sophisticated synchronization mechanism for distributing vulnerability signatures. ### System Components ``` ┌──────────────────────────────────────────────────────┐ │ Sirius Platform │ │ ┌────────────────┐ ┌────────────────┐ │ │ │ sirius-api │◄────►│ sirius-ui │ │ │ │ (Go/Fiber) │ │ (Next.js) │ │ │ └────────┬───────┘ └────────────────┘ │ │ │ │ │ │ RabbitMQ (Command Queue) │ │ ▼ │ │ ┌────────────────┐ │ │ │ Agent Server │◄──────── Valkey/Redis │ │ │ (gRPC) │ (Template Storage) │ │ └────────┬───────┘ │ │ │ Bidirectional gRPC Stream │ │ │ │ └───────────┼───────────────────────────────────────────┘ │ ┌───────┴───────┬──────────┬──────────┐ │ │ │ │ ▼ ▼ ▼ ▼ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ │ Agent │ │ Agent │ │ Agent │ │ Agent │ │ Linux │ │ Win │ │ macOS │ │ Remote │ └────────┘ └────────┘ └────────┘ └────────┘ ``` ## Core Architecture ### 1. Agent Server **Location**: `app-agent/cmd/server/` and `app-agent/internal/server/` **Purpose**: Central hub for agent management, command distribution, and template synchronization. #### Server Components ```go type Server struct { // Core services logger *zap.Logger config *config.ServerConfig server *grpc.Server // Agent management agentsMutex sync.RWMutex agents map[string]pb.HelloService_ConnectStreamServer // Command tracking commandsMutex sync.RWMutex commands map[string]*CommandStatus // Template system templateManager *ServerTemplateManager valkeyClient valkey.Client // Response storage responseStore store.ResponseStore // Pending command correlation pendingCommandsMutex sync.Mutex pendingCommands map[string]string } ``` #### Key Responsibilities 1. **Agent Connection Management** - Accept and maintain bidirectional gRPC streams - Track connected agents with unique IDs - Handle agent disconnections gracefully - Monitor agent health via heartbeats 2. **Command Distribution** - Listen to RabbitMQ for commands from UI - Route commands to appropriate agents - Track command execution status - Store results in Valkey for UI retrieval 3. **Template Synchronization** - Sync templates from GitHub repository - Store templates in Valkey - Distribute templates to agents - Handle custom template uploads 4. **State Management** - Maintain agent connection state - Track command execution lifecycle - Coordinate template versioning - Manage concurrent access safely ### 2. Agent Client **Location**: `app-agent/cmd/agent/` and `app-agent/internal/agent/` **Purpose**: Remote endpoint that executes commands and performs vulnerability scans. #### Agent Components ```go type Agent struct { // gRPC communication logger *zap.Logger config *config.AgentConfig conn *grpc.ClientConn client pb.HelloServiceClient stream pb.HelloService_ConnectStreamClient // Agent information startTime time.Time agentInfo commands.AgentInfo // Scripting support powerShellPath string scriptingEnabled bool // Template management syncManager *templateagent.AgentSyncManager } ``` #### Key Responsibilities 1. **Server Communication** - Establish bidirectional gRPC stream - Send periodic heartbeats with system metrics - Receive and process server messages - Handle connection failures and reconnection 2. **Command Execution** - Process internal commands (status, help, scan) - Execute shell commands via PowerShell/bash - Capture command output and errors - Report execution results to server 3. **Template Synchronization** - Request template updates from server - Receive and cache templates locally - Validate template checksums - Maintain local template manifest 4. **Vulnerability Scanning** - Load and parse cached templates - Execute template detection steps - Evaluate detection logic - Report scan results ## Communication Protocol ### gRPC Service Definition ```protobuf service HelloService { // Health check rpc Ping(PingRequest) returns (PingResponse) {} // Bidirectional streaming for real-time communication rpc ConnectStream(stream AgentMessage) returns (stream ServerMessage) {} } enum MessageType { UNKNOWN = 0; HEARTBEAT = 1; // Agent → Server: Health status COMMAND = 2; // Server → Agent: Execute command RESULT = 3; // Agent → Server: Command result ACKNOWLEDGMENT = 4; // Server → Agent: Confirm receipt TEMPLATE_UPDATE = 5; // Server → Agent: Template data TEMPLATE_SYNC_REQUEST = 6; // Agent → Server: Request templates } ``` ### Message Flow Patterns #### Command Execution Flow ``` 1. UI/API → RabbitMQ └─ Queue: agent_commands └─ Message: {action, command, agentId, userId, timestamp} 2. Server consumes from RabbitMQ └─ Validates agent is connected └─ Sends ACK to agent_response queue └─ Generates commandId = "agentId:timestamp" └─ Stores in pendingCommands map └─ Forwards to agent via gRPC stream 3. Agent receives COMMAND message └─ Checks internal command registry └─ If not found, executes as shell command └─ Captures stdout, stderr, exit code └─ Sends RESULT message to server 4. Server receives RESULT message └─ Looks up commandId from pendingCommands └─ Creates CommandResponse object └─ Stores in Valkey: key = "cmd:result:{commandId}" └─ Logs to file └─ Sends ACKNOWLEDGMENT to agent 5. UI/API polls Valkey └─ Retrieves result using commandId └─ Displays to user ``` #### Template Sync Flow ``` 1. Agent sends TEMPLATE_SYNC_REQUEST └─ Includes lastSync timestamp └─ Sent via gRPC stream 2. Server processes sync request └─ Queries Valkey for templates updated after lastSync └─ Builds TemplateManifest (metadata only) └─ Sends manifest as first message 3. Server streams templates └─ For each template: ├─ Retrieves content from Valkey ├─ Calculates checksum ├─ Creates TEMPLATE_UPDATE message └─ Sends via stream 4. Agent receives templates └─ For each TEMPLATE_UPDATE: ├─ Writes to cache directory (atomic) ├─ Verifies checksum ├─ Updates local manifest └─ Ready for scanning 5. Agent completes sync └─ Updates lastSync timestamp └─ Templates available for execution ``` #### Heartbeat Flow ``` Agent (every 30 seconds): └─ Collects system metrics (CPU, memory) └─ Sends HEARTBEAT message └─ Includes timestamp, metrics Server receives HEARTBEAT: └─ Updates agent last-seen time └─ Logs metrics (debug level) └─ No response needed (fire-and-forget) ``` ## Template System Architecture ### Template Structure The template system uses YAML-based vulnerability definitions that are platform-agnostic and module-driven. #### Template Definition ```yaml id: apache-outdated info: name: Outdated Apache HTTP Server author: security-team severity: high description: Detects Apache versions with known CVEs cve: - CVE-2021-44228 tags: [apache, web-server, cve] version: "1.0" detection: logic: all # all (AND) or any (OR) steps: - type: version-cmd platforms: [linux, darwin] weight: 1.0 config: command: "httpd -v" version_regex: "Apache/(\\d+\\.\\d+\\.\\d+)" vulnerable_versions: - "< 2.4.52" - type: file-content platforms: [linux, darwin] weight: 0.8 config: path: "/etc/apache2/apache2.conf" patterns: ["ServerTokens.*Full"] ``` ### Template Execution Engine **Location**: `app-agent/internal/template/executor/` ``` ┌─────────────────────────────────────────────┐ │ Template Executor │ ├─────────────────────────────────────────────┤ │ │ │ 1. Filter Steps by Platform │ │ └─ Current OS: linux/darwin/windows │ │ │ │ 2. Execute Steps Sequentially │ │ ├─ Get module from registry │ │ ├─ Create timeout context │ │ ├─ Execute module │ │ └─ Collect results │ │ │ │ 3. Evaluate Detection Logic │ │ ├─ logic: all → All steps match (AND) │ │ └─ logic: any → Any step matches (OR) │ │ │ │ 4. Calculate Confidence │ │ ├─ logic: all → Min(weights) │ │ └─ logic: any → Max(weights) │ │ │ │ 5. Build Result │ │ └─ {matched, confidence, steps, errors} │ │ │ └─────────────────────────────────────────────┘ ``` ### Detection Modules Detection modules are pluggable components that implement specific detection mechanisms. #### Module Interface ```go type Module interface { Name() string Execute(ctx context.Context, config map[string]interface{}) (*ModuleResult, error) } type ModuleResult struct { Matched bool Evidence map[string]interface{} Error string } ``` #### Available Modules | Module | Type | Purpose | Platforms | | ------------- | ------------- | -------------------------- | ----------------- | | version-cmd | Command | Parse version from command | All | | file-hash | File System | Check file SHA256 hash | All | | file-content | File System | Search file for patterns | All | | registry-key | Windows | Check registry keys/values | Windows only | | service-check | System | Verify service exists/runs | All | | config-check | Configuration | Parse and validate configs | All | | script-exec | Custom | Execute custom detection | Platform-specific | #### Module Registration ```go // Module implementation type FileHashModule struct{} func (m *FileHashModule) Name() string { return "file-hash" } func (m *FileHashModule) Execute(ctx context.Context, config map[string]interface{}) (*ModuleResult, error) { // Implementation } // Registration (in init()) func init() { registry.Register(&FileHashModule{}) } ``` ### Template Storage and Distribution #### Server-Side Storage (Valkey) ``` Template Keys Structure: template:manifest └─ Global manifest JSON with all template metadata template:meta: └─ Individual template metadata JSON └─ {id, version, checksum, size, severity, platforms, etc.} template:standard: └─ Standard template YAML content └─ From sirius-agent-modules GitHub repository template:custom: └─ Custom template YAML content └─ User-uploaded via UI ``` #### Agent-Side Cache ``` Cache Directory Structure: Windows: C:\ProgramData\Sirius\templates\ Linux: /var/lib/sirius/templates/ macOS: /Library/Application Support/Sirius/templates/ Contents: ├── cache-manifest.json # Local cache metadata │ └─ {version, lastSync, templates{}, statistics{}} │ ├── standard/ # Standard templates │ ├── apache-outdated.yaml │ ├── nginx-vuln.yaml │ └── ssh-weak-ciphers.yaml │ └── custom/ # Custom templates └── org-policy-check.yaml ``` #### Template Synchronization Manager **Server Side** (`internal/server/template_manager.go`): ```go type ServerTemplateManager struct { valkeyClient valkey.Client storage *templatevalkey.ValKeyTemplateStorage githubSync *templatevalkey.GitHubSyncManager logger *zap.Logger config *TemplateConfig server *Server } // Key Methods: // - SyncFromGitHub() → Pull from sirius-agent-modules // - GetTemplatesForSync() → Prepare templates for agent // - StoreCustomTemplate() → Handle user uploads // - ValidateTemplate() → Security and syntax checks ``` **Agent Side** (`internal/template/agent/sync_manager.go`): ```go type AgentSyncManager struct { cacheDir string logger *zap.Logger serverURL string agentID string grpcStream pb.HelloService_ConnectStreamClient } // Key Methods: // - SyncFromServer() → Request template sync // - HandleTemplateUpdate() → Process incoming templates // - LoadTemplates() → Load cached templates for scanning // - ValidateCache() → Verify template checksums ``` ## Command System ### Command Architecture The agent supports both internal commands (built into the agent) and shell commands (executed via PowerShell/bash). ``` ┌─────────────────────────────────────────────┐ │ Command Dispatcher │ ├─────────────────────────────────────────────┤ │ │ │ 1. Receive Command String │ │ └─ From server via gRPC stream │ │ │ │ 2. Parse Command │ │ ├─ Extract command name │ │ └─ Parse arguments │ │ │ │ 3. Check Command Registry │ │ ├─ Internal command? → Execute handler │ │ └─ Not found? → Fall back to shell │ │ │ │ 4. Execute │ │ ├─ Internal: Call Go function │ │ └─ Shell: Execute via PowerShell/bash │ │ │ │ 5. Capture Result │ │ ├─ stdout, stderr │ │ ├─ exit code │ │ └─ execution time │ │ │ │ 6. Send Result to Server │ │ └─ RESULT message via gRPC stream │ │ │ └─────────────────────────────────────────────┘ ``` ### Internal Commands | Command | Purpose | Example | | --------------- | -------------------------------- | ------------------------------------ | | `help` | List available commands | `help` | | `status` | Agent status and capabilities | `status` | | `scan` | Execute vulnerability scan | `scan --targets=192.168.1.0/24` | | `template list` | List cached templates | `template list` | | `template sync` | Trigger template synchronization | `template sync` | | `templatescan` | Run specific template | `templatescan --template=apache-001` | ### Command Registration ```go // Command handler signature type CommandHandler func(ctx context.Context, info AgentInfo, args []string) (string, error) // Registration func Register(name string, handler CommandHandler) { commandsMutex.Lock() defer commandsMutex.Unlock() commands[name] = handler } // Dispatch func Dispatch(ctx context.Context, info AgentInfo, commandString string) (string, error) { cmdName, args := parseCommandLine(commandString) handler, exists := commands[cmdName] if !exists { return "", ErrUnknownCommand } return handler(ctx, info, args) } ``` ### Shell Command Execution When a command is not found in the internal registry, the agent attempts to execute it as a shell command. #### PowerShell Execution (Windows) ```go func ExecuteScript(ctx context.Context, psPath string, scriptContent string) (stdout, stderr string, exitCode int, err error) { // Create temp script file tempFile := filepath.Join(os.TempDir(), fmt.Sprintf("sirius-script-%d.ps1", time.Now().UnixNano())) os.WriteFile(tempFile, []byte(scriptContent), 0600) defer os.Remove(tempFile) // Execute via PowerShell cmd := exec.CommandContext(ctx, psPath, "-NoProfile", "-ExecutionPolicy", "Bypass", "-File", tempFile) // Capture output var stdoutBuf, stderrBuf bytes.Buffer cmd.Stdout = &stdoutBuf cmd.Stderr = &stderrBuf err = cmd.Run() return stdoutBuf.String(), stderrBuf.String(), cmd.ProcessState.ExitCode(), err } ``` #### Bash Execution (Linux/macOS) ```go func ExecuteCommand(ctx context.Context, command string) (stdout, stderr string, exitCode int, err error) { cmd := exec.CommandContext(ctx, "/bin/bash", "-c", command) var stdoutBuf, stderrBuf bytes.Buffer cmd.Stdout = &stdoutBuf cmd.Stderr = &stderrBuf err = cmd.Run() return stdoutBuf.String(), stderrBuf.String(), cmd.ProcessState.ExitCode(), err } ``` ## Data Storage Architecture ### Valkey (Redis) Schema The system uses Valkey for distributed state management and caching. #### Key Patterns ``` Command Results: cmd:result:{agentId}:{timestamp} └─ CommandResponse JSON └─ TTL: 1 hour └─ Fields: {commandId, agentId, command, status, output, error, exitCode} Template Storage: template:manifest └─ Global manifest with statistics template:meta:{templateId} └─ Template metadata template:standard:{templateId} └─ Standard template YAML template:custom:{templateId} └─ Custom template YAML Agent Metadata (future): agent:meta:{agentId} └─ Agent information └─ {id, hostname, platform, version, capabilities, lastSeen} ``` ### RabbitMQ Message Queues #### Queue Definitions ``` agent_commands └─ Purpose: Commands from UI/API to agents └─ Format: JSON └─ Schema: {action, command, agentId, userId, timestamp, target} └─ Consumers: Agent Server agent_response └─ Purpose: Acknowledgments and responses to UI/API └─ Format: JSON └─ Schema: {success, message, output, error} └─ Consumers: sirius-api ``` #### Message Examples **Command Message**: ```json { "action": "", "command": "systeminfo", "agentId": "", "userId": "user-123", "timestamp": "1698765432000", "target": { "type": "agent", "id": "agent-001" } } ``` **Response Message**: ```json { "success": true, "message": "Command received, forwarding to agent", "output": "", "error": "" } ``` ## Security Architecture ### Authentication and Authorization #### Agent Authentication ``` 1. Agent Registration: ├─ Agent ID generated on first startup ├─ ID stored in agent config └─ Server validates agent ID on connection 2. gRPC Metadata: ├─ agent_id: Unique identifier ├─ scripting_enabled: Capability flag └─ Future: JWT tokens, TLS certificates 3. Stream Authorization: ├─ Server maintains whitelist of allowed agent IDs ├─ Validates agent ID on ConnectStream └─ Rejects unauthorized connections ``` #### Command Authorization ``` 1. Command Source Validation: ├─ Commands only accepted from RabbitMQ ├─ Server validates target agent exists └─ Server checks agent is connected 2. Agent-Side Validation: ├─ Only processes commands from established stream ├─ Validates command format └─ Rejects malformed commands 3. Future Enhancements: ├─ Role-based access control (RBAC) ├─ Command whitelisting per agent └─ Audit logging of all commands ``` ### Template Security #### Template Validation ```go // Server-side validation before storage func (tm *ServerTemplateManager) ValidateTemplate(template *types.Template, content []byte) error { // 1. Syntax validation if err := parser.ParseTemplate(template); err != nil { return fmt.Errorf("syntax error: %w", err) } // 2. Security scan - dangerous patterns dangerousPatterns := []string{ "eval(", "exec(", "system(", "shell_exec(", "passthru(", } for _, pattern := range dangerousPatterns { if strings.Contains(string(content), pattern) { return fmt.Errorf("dangerous pattern detected: %s", pattern) } } // 3. Script injection check scriptPatterns := []string{ " MaxTemplateSize { return fmt.Errorf("template exceeds size limit") } return nil } ``` #### Checksum Verification ``` 1. Server calculates SHA256 on storage: └─ checksum = sha256(templateContent) 2. Checksum included in TEMPLATE_UPDATE: └─ {templateId, version, checksum, content} 3. Agent verifies on receipt: ├─ Calculate sha256(received content) ├─ Compare with provided checksum └─ Reject if mismatch ``` ### Network Security ``` 1. gRPC Transport Security: ├─ TLS encryption (configurable) ├─ Certificate validation └─ Mutual TLS support (future) 2. RabbitMQ Security: ├─ Authentication required ├─ Per-queue permissions └─ TLS connections 3. Valkey Security: ├─ Password authentication ├─ Network isolation └─ Redis ACLs (future) ``` ## Deployment Architecture ### Container Architecture ``` ┌─────────────────────────────────────────────────────┐ │ Docker Network: sirius │ │ │ │ ┌──────────────┐ ┌──────────────┐ ┌───────────┐ │ │ │ sirius-ui │ │ sirius-api │ │ postgres │ │ │ │ (Next.js) │ │ (Go/Fiber) │ │ │ │ │ │ Port: 3000 │ │ Port: 8080 │ │ Port:5432 │ │ │ └──────────────┘ └──────┬───────┘ └───────────┘ │ │ │ │ │ ┌──────────────┐ ┌──────▼───────┐ ┌───────────┐ │ │ │ valkey │ │ rabbitmq │ │ sirius- │ │ │ │ (Redis) │ │ │ │ engine │ │ │ │ Port: 6379 │ │ Port: 5672 │ │ (agent) │ │ │ └──────────────┘ └──────────────┘ └───────────┘ │ │ │ │ ┌─────────────────────────────────────────────────┐│ │ │ Agent Server (gRPC) ││ │ │ Port: 50051 ││ │ │ Mounts: app-agent/ → /app-agent ││ │ └─────────────────────────────────────────────────┘│ └─────────────────────────────────────────────────────┘ │ │ gRPC (external) │ ┌───────────────┴────────────────┐ │ │ ▼ ▼ ┌──────────────┐ ┌──────────────┐ │ Remote Agent │ │ Remote Agent │ │ (Linux) │ │ (Windows) │ │ │ │ │ │ External IP │ │ External IP │ └──────────────┘ └──────────────┘ ``` ### Agent Server Deployment **Development Mode** (`sirius-engine` container): ```yaml services: sirius-engine: build: context: sirius-engine dockerfile: Dockerfile volumes: - ./app-agent:/app-agent # Dev mode hot-reload ports: - "50051:50051" # gRPC port environment: - VALKEY_ADDRESS=valkey:6379 - RABBITMQ_URL=amqp://guest:guest@rabbitmq:5672/ depends_on: - valkey - rabbitmq ``` **Production Mode**: ```yaml services: agent-server: image: sirius-agent-server:latest ports: - "50051:50051" environment: - SERVER_ADDRESS=0.0.0.0:50051 - VALKEY_ADDRESS=valkey:6379 - RABBITMQ_URL=amqp://user:pass@rabbitmq:5672/ - LOG_LEVEL=info restart: unless-stopped ``` ### Remote Agent Deployment #### Linux Agent (systemd) ```ini [Unit] Description=Sirius Security Agent After=network-online.target Wants=network-online.target [Service] Type=simple User=sirius-agent Group=sirius-agent Environment="AGENT_ID=agent-linux-001" Environment="SERVER_ADDRESS=server.example.com:50051" Environment="ENABLE_SCRIPTING=true" ExecStart=/usr/local/bin/sirius-agent Restart=always RestartSec=10 [Install] WantedBy=multi-user.target ``` #### Windows Agent (Service) ```powershell # Install as Windows Service New-Service -Name "SiriusAgent" ` -BinaryPathName "C:\Program Files\Sirius\sirius-agent.exe" ` -DisplayName "Sirius Security Agent" ` -StartupType Automatic ` -Description "Sirius distributed security agent" # Set environment variables [Environment]::SetEnvironmentVariable("AGENT_ID", "agent-win-001", "Machine") [Environment]::SetEnvironmentVariable("SERVER_ADDRESS", "server.example.com:50051", "Machine") [Environment]::SetEnvironmentVariable("ENABLE_SCRIPTING", "true", "Machine") # Start service Start-Service -Name "SiriusAgent" ``` #### macOS Agent (launchd) ```xml Label com.sirius.agent ProgramArguments /usr/local/bin/sirius-agent EnvironmentVariables AGENT_ID agent-mac-001 SERVER_ADDRESS server.example.com:50051 ENABLE_SCRIPTING true RunAtLoad KeepAlive ``` ## Performance Considerations ### Scalability #### Horizontal Scaling ``` Server Scaling: ├─ Multiple agent server instances behind load balancer ├─ gRPC load balancing strategies ├─ Shared state in Valkey └─ RabbitMQ message distribution Agent Scaling: ├─ Thousands of agents per server instance ├─ Efficient stream multiplexing ├─ Minimal memory footprint per agent └─ Connection pooling ``` #### Vertical Scaling ``` Server Resources: ├─ CPU: Moderate (gRPC is efficient) ├─ Memory: 2GB base + 5MB per 1000 agents ├─ Network: High bandwidth for template distribution └─ Storage: Minimal (state in Valkey) Agent Resources: ├─ CPU: Low (idle), High (scanning) ├─ Memory: 50-100MB base ├─ Disk: 100MB for template cache └─ Network: Low (heartbeats), High (template sync) ``` ### Optimization Strategies #### Template Caching ``` Agent-Side: ├─ Templates cached on disk ├─ Only sync changed templates ├─ Checksum-based validation └─ Last-sync timestamp tracking Server-Side: ├─ Templates stored in Valkey ├─ Manifest cached in memory ├─ Incremental sync support └─ Compression for transfer (future) ``` #### Connection Management ``` gRPC Streams: ├─ Single bidirectional stream per agent ├─ Stream multiplexing (HTTP/2) ├─ Automatic reconnection with backoff └─ Heartbeat-based connection validation Command Execution: ├─ Async command processing ├─ Non-blocking stream operations ├─ Context-based cancellation └─ Timeout enforcement ``` ## Monitoring and Observability ### Metrics ``` Server Metrics: ├─ agent.connections.total (gauge) ├─ agent.connections.active (gauge) ├─ commands.executed.total (counter) ├─ commands.duration (histogram) ├─ templates.synced.total (counter) └─ grpc.stream.errors (counter) Agent Metrics: ├─ agent.uptime (gauge) ├─ agent.heartbeat.sent (counter) ├─ commands.received (counter) ├─ scans.executed (counter) ├─ templates.cached (gauge) └─ memory.usage (gauge) ``` ### Logging ``` Structured Logging (zap): Server: ├─ agent.connected: {agentId, timestamp} ├─ command.sent: {commandId, agentId, command} ├─ command.completed: {commandId, exitCode, duration} └─ template.synced: {templateId, agentId} Agent: ├─ connection.established: {serverId, timestamp} ├─ command.received: {command, timestamp} ├─ command.executed: {command, exitCode, duration} └─ template.updated: {templateId, checksum} ``` ### Health Checks ``` Server Health Endpoint: GET /health └─ { "status": "healthy", "agents": { "total": 42, "connected": 40 }, "services": { "valkey": "healthy", "rabbitmq": "healthy" } } Agent Health Check: ├─ Heartbeat interval: 30 seconds ├─ Server considers agent unhealthy after: 90 seconds └─ Agent reconnects automatically ``` ## Error Handling and Resilience ### Connection Failures ``` Agent Reconnection Strategy: 1. Detect connection loss (stream error) 2. Wait with exponential backoff: ├─ 1st retry: 1 second ├─ 2nd retry: 2 seconds ├─ 3rd retry: 4 seconds └─ Max: 60 seconds 3. Attempt reconnection 4. Resume from last sync point 5. Resend pending commands ``` ### Command Failures ``` Server Handling: ├─ Command timeout: 5 minutes ├─ Store error in CommandResponse ├─ Mark command as failed └─ Notify UI via Valkey Agent Handling: ├─ Capture all errors ├─ Include stderr in result ├─ Return non-zero exit code └─ Continue processing other commands ``` ### Template Sync Failures ``` Graceful Degradation: ├─ Agent continues using cached templates ├─ Retry sync on next request ├─ Log sync errors for investigation └─ Alert on repeated failures ``` ## Future Enhancements ### Planned Features 1. **Enhanced Security** - Mutual TLS authentication - JWT-based authorization - Command RBAC system - Encrypted template storage 2. **Advanced Template Features** - Template versioning system - Template dependency resolution - Conditional step execution - Dynamic module loading 3. **Scalability Improvements** - Agent clustering support - Multi-region deployment - Template CDN distribution - Result streaming for large scans 4. **Monitoring & Analytics** - Real-time dashboards - Agent performance metrics - Template effectiveness tracking - Automated alerting system ## LLM Context ### High-Level Architecture Understanding This document provides a comprehensive overview of the Sirius Agent System architecture. Key areas for AI assistants to understand: 1. **Bidirectional gRPC Communication**: The system uses persistent streams for real-time agent communication, not request-response patterns. 2. **Template-Based Detection**: Vulnerability detection is driven by YAML templates with pluggable modules, not hardcoded logic. 3. **Distributed State Management**: Valkey stores templates and results; RabbitMQ handles command distribution; gRPC handles real-time communication. 4. **Agent Autonomy**: Agents cache templates locally and can operate semi-independently, syncing periodically with the server. 5. **Security-First Design**: Multiple validation layers for templates, commands, and connections ensure system integrity. ### Code Navigation When working with this system: - **Server code**: `app-agent/internal/server/` - **Agent code**: `app-agent/internal/agent/` - **Template system**: `app-agent/internal/template/` - **Detection modules**: `app-agent/internal/detect/` - **Protocol definitions**: `app-agent/proto/hello/` ### Common Development Tasks - Adding detection modules: Implement `Module` interface, register in `internal/modules/registry` - Creating templates: Follow YAML structure, test with `templatescan` command - Extending protocol: Update `.proto` file, regenerate with `./scripts/generate_proto.sh` - Debugging communication: Enable gRPC logging, check server/agent logs, inspect Valkey keys --- **Related Documentation**: - [Docker Architecture](mdc:README.docker-architecture.md) - Container deployment details - [Developer Guide](mdc:../README.developer-guide.md) - Development workflow - [Agent Template API](mdc:../apps/README.agent-template-api.md) - API integration **For Developers**: See [Remote Agent Engineer Role](@mdc:../../.cursor/agents/remote-agent-engineer.agent.md) for detailed implementation guidance. **Last Updated**: October 25, 2025 **Version**: 1.0.0 **Maintained By**: Sirius Platform Team