wehub-resource-sync
161ef94b4f
92 KiB
Sirius Agent Ecosystem - Complete Developer Guide
Version: 2.1 (Production System)
Last Updated: January 2025
Status: Comprehensive Developer Onboarding Documentation
Recent Updates (v2.1): Major code quality improvements including utility reorganization, enhanced type safety, elimination of code duplication, and improved developer experience. See Frontend Development Best Practices for details.
📖 Table of Contents
- Executive Summary & Problem Statement
- The Security Challenge We're Solving
- System Architecture Philosophy
- Technology Stack & Rationale
- Core Components Deep Dive
- Data Flow Narratives
- Template System Design
- Development Workflows
- Database Strategy
- Deployment & Operations
- Troubleshooting Philosophy
- Developer Handbook
🎯 Executive Summary & Problem Statement
What Are We Building?
The Sirius Agent Ecosystem is a distributed vulnerability scanning and system fingerprinting platform that enables security teams to remotely assess the security posture of their infrastructure. Think of it as having security auditors deployed throughout your network that can continuously assess vulnerabilities, collect software inventories, and detect misconfigurations.
But this isn't just another security tool—it's a complete reimagining of how vulnerability management should work in modern environments. Where traditional scanners struggle with visibility and context, our system thrives by placing intelligent agents directly on the systems they're protecting.
The Core Problem
If you've ever worked in cybersecurity, you've probably experienced the frustration of traditional vulnerability scanners. They're like trying to diagnose a patient by only looking at their skin—you can see some symptoms, but you're missing the full picture of what's happening inside.
Traditional vulnerability scanners face several critical limitations:
- Network-Only Visibility: They can only see what's exposed on the network, missing internal vulnerabilities, installed software versions, and system configurations
- Point-in-Time Snapshots: They provide periodic scans but miss the continuous changes in modern dynamic environments
- Limited Context: They detect vulnerabilities but lack the rich system context needed for prioritization and remediation
- Scale Challenges: Scanning thousands of hosts from a central location creates performance bottlenecks and network congestion
These limitations aren't just technical inconveniences—they represent fundamental gaps in an organization's security posture that attackers routinely exploit.
Our Solution Approach
We've built a distributed agent-based system that solves these problems by fundamentally changing the relationship between the scanner and the systems being scanned. Instead of peering through the network from the outside, we embed intelligence directly where it's needed.
Our approach delivers:
- Deep Host Visibility: Agents run directly on target systems, providing complete visibility into installed software, configurations, and running services
- Continuous Monitoring: Agents provide ongoing assessment rather than point-in-time scans
- Rich Context Collection: SBOM (Software Bill of Materials) data provides comprehensive context for vulnerability correlation
- Scalable Architecture: Distributed processing with centralized coordination scales to thousands of hosts
- Extensible Detection: Template-based vulnerability detection allows rapid response to new threats
This foundation sets the stage for everything else we'll explore in this guide. As we dive deeper, you'll see how each architectural decision serves this core mission of providing comprehensive, continuous, and contextual security assessment.
🔍 The Security Challenge We're Solving
Why Traditional Approaches Fall Short
Before we dive into the technical architecture, it's essential to understand the security landscape that drove us to build this system. The challenges we're addressing aren't new, but they've become more acute as infrastructure becomes more dynamic and attacks more sophisticated.
The story of modern vulnerability management is one of constant catch-up—security teams trying to maintain visibility into rapidly changing environments with tools designed for a more static world. Let's explore the specific gaps that our system addresses.
Traditional Vulnerability Management Gaps
1. The "Black Box" Problem
Imagine trying to inventory a warehouse by only looking through the windows. This is essentially what traditional network scanners do—they peer through network ports, trying to deduce what's running inside systems.
Traditional network scanners can only see what services are exposed on network ports. They miss:
- Internal vulnerabilities in software that doesn't expose network services
- Configuration issues in files that aren't network-accessible
- Privilege escalation vectors that require local system access
- Software inventory that enables vulnerability correlation
Real-World Impact: A vulnerable version of sudo installed on a Linux system won't be detected by network scanning, but it represents a critical privilege escalation risk that attackers can exploit to gain root access.
2. The "Snapshot" Problem
Modern infrastructure is dynamic, but traditional scanning is static. It's like trying to understand a movie by looking at a few still frames—you miss all the action that happens between scans.
Network scans provide point-in-time assessments, but modern environments are dynamic:
- Software updates happen continuously
- Configuration changes can introduce new vulnerabilities
- New deployments can contain vulnerable components
- Temporary services might be missed between scan cycles
Real-World Impact: A developer deploys a test application with default credentials on Tuesday, but the weekly network scan isn't scheduled until Friday. That's a four-day window where the vulnerability exists but remains undetected.
3. The "Context" Problem
Finding vulnerabilities is only half the battle—understanding their real-world impact is often more challenging. Traditional scanners are like smoke detectors that can tell you there's a fire somewhere in the building, but not which room or how severe.
Network scans detect vulnerabilities but lack operational context:
- Which vulnerabilities are actually exploitable in the current configuration?
- What business services would be impacted by exploitation?
- What compensating controls are already in place?
- How are systems interconnected for lateral movement analysis?
Real-World Impact: A critical vulnerability in Apache might seem urgent until you realize it's running in a container with no network access, behind a WAF, with all dangerous modules disabled. Context changes everything about prioritization.
Our Agent-Based Solution
Our approach flips the traditional model on its head. Instead of trying to peer into systems from the outside, we place intelligent agents on the inside, giving them the access and context needed to provide comprehensive security assessment.
Think of it as the difference between a home security system that only monitors doors and windows versus one that has sensors throughout every room. The comprehensive approach provides dramatically better visibility and context.
Deep System Access
Our agents run with appropriate privileges to examine everything that matters for security:
- Complete software inventory (packages, versions, dependencies)
- Configuration files and settings
- Running processes and services
- User accounts and permissions
- Network interfaces and routing
- Certificate stores and cryptographic materials
This comprehensive visibility means no vulnerable software goes undetected, no misconfiguration remains hidden, and no security-relevant change occurs without notice.
Continuous Assessment
Rather than periodic snapshots, our agents provide ongoing monitoring:
- Real-time detection of new vulnerabilities
- Configuration drift monitoring
- Software installation/removal tracking
- Service status changes
- Security event correlation
This continuous approach means the security team knows about problems minutes after they occur, not days or weeks later.
Rich Contextual Data
Our agents collect the contextual information needed for intelligent prioritization:
- Hardware specifications and capabilities
- Network topology and connectivity
- Business service mappings
- Compliance control implementations
- Custom security configurations
This context transforms vulnerability management from a guessing game into a precise, prioritized process.
The Path Forward
These challenges shaped every aspect of our system design. As we explore the architecture in the following sections, you'll see how each component serves the mission of providing comprehensive, continuous, and contextual security assessment.
The technical decisions we'll discuss aren't arbitrary—they're responses to real-world security challenges that have plagued organizations for years. By understanding these challenges first, the architectural solutions become not just technically interesting, but strategically essential.
🏗️ System Architecture Philosophy
The Evolution of Our Thinking
Building a distributed security system isn't just about writing code—it's about solving complex organizational and operational challenges through technology. Our architecture didn't emerge fully formed; it evolved through several iterations, each teaching us valuable lessons about what works in real-world environments.
Understanding this evolution is crucial because it explains why we made certain design choices that might seem complex at first glance. Each layer of complexity exists because simpler approaches failed to meet the demands of production environments.
Core Design Principles
Our architecture is built on several fundamental principles that guide every technical decision. These aren't just philosophical ideals—they're practical guidelines that have saved us countless hours of debugging and prevented numerous production issues.
1. Separation of Concerns
One of our most important principles is that each component should have a single, clear responsibility. This might seem obvious, but it's surprisingly easy to violate when building distributed systems.
We've deliberately separated different responsibilities into distinct components:
- UI Layer: Focuses purely on user experience and data presentation
- Agent Server: Handles coordination, template management, and communication orchestration
- Storage Layer: Provides reliable, scalable data persistence
- Endpoint Agents: Focus exclusively on data collection and vulnerability detection
- Database Layer: Handles complex vulnerability correlation and reporting
Why This Matters: This separation allows each component to be developed, tested, and scaled independently. A UI bug won't affect agent scanning, and database performance issues won't impact real-time agent communication.
More importantly, this separation makes the system understandable. When something goes wrong, you know exactly where to look. When you need to scale, you know exactly what to scale. When you need to modify functionality, you know exactly what components are involved.
2. Single Source of Truth
In distributed systems, data consistency is often the most challenging problem. We solved this by establishing ValKey as our centralized truth store for all operational data:
- No Data Duplication: Each piece of information has exactly one authoritative location
- Consistency Guarantees: All components see the same data at any given time
- Simplified Debugging: When something goes wrong, there's one place to look for the truth
- Atomic Updates: Changes are applied atomically to prevent partial state corruption
The Alternative We Avoided: Many systems try to maintain consistency through complex synchronization protocols between multiple data stores. This approach inevitably leads to split-brain scenarios, data loss, and hours of debugging mysterious inconsistencies.
Why This Matters: Having a single source of truth eliminates an entire class of bugs and makes the system much easier to reason about. When an agent shows a template as available, you can be confident that every other component in the system sees the same state.
3. Asynchronous Communication
We use RabbitMQ for loose coupling between components, and this decision has profound implications for system reliability and scalability:
- UI ↔ Agent Server: Asynchronous messaging for template creation and management
- Resilience: Components can temporarily fail without data loss
- Scalability: Messages can be queued and processed at sustainable rates
- Observability: Message flow provides clear audit trails
The Story Behind This Decision: In our early iterations, we used synchronous HTTP calls between components. This created a cascading failure pattern where a problem in any component would bring down the entire system. Users would click "create template" and wait 30 seconds for a timeout when the Agent Server was under load.
Why This Matters: Asynchronous communication allows components to operate independently and provides natural backpressure when components are overwhelmed. Users get immediate feedback, and the system gracefully handles load spikes.
4. gRPC for Real-Time Communication
While we use asynchronous messaging for most operations, agent communication requires real-time responsiveness. We use gRPC streaming for this because:
- Bidirectional Streams: Agents and servers can communicate simultaneously
- Type Safety: Protocol buffers provide compile-time verification
- Performance: Binary encoding and HTTP/2 multiplexing
- Reliability: Built-in retry logic and connection management
The Balance We Struck: This creates an interesting architectural pattern where we have both asynchronous and synchronous communication in the same system. The key insight is matching the communication pattern to the actual requirements rather than forcing everything through the same mechanism.
The Big Picture: Component Interaction
Understanding how these principles manifest in our actual architecture helps illustrate why they matter:
┌─────────────────┐ Async Messages ┌─────────────────┐ gRPC Streams ┌─────────────────┐
│ Sirius UI │ ──────RabbitMQ────→ │ Agent Server │ <──────gRPC──────> │ Endpoint Agent │
│ │ │ │ │ │
│ • Template UI │ │ • Coordination │ │ • Scanning │
│ • Agent Mgmt │ │ • Template Sync │ │ • Collection │
│ • Results View │ │ • Command Dist │ │ • Detection │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │
│ │
▼ ▼
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ ValKey (Single Source of Truth) │
│ │
│ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │
│ │ Templates │ │ Scripts │ │ Agent State │ │ Scan Results │ │ Metadata │ │
│ │ │ │ │ │ │ │ │ │ │ │
│ │ • Repository │ │ • PowerShell │ │ • Connections │ │ • SBOM Data │ │ • Manifests │ │
│ │ • Custom │ │ • Bash │ │ • Status │ │ • Vulns Found │ │ • Statistics │ │
│ │ • Local │ │ • Python │ │ • Heartbeat │ │ • Fingerprint │ │ • Versioning │ │
│ └───────────────┘ └───────────────┘ └───────────────┘ └───────────────┘ └───────────────┘ │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
▲ ▲
│ │
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ PostgreSQL │ │ RabbitMQ │
│ │ │ │
│ • Vulnerability │ │ • Message Queue │
│ Correlation │ │ • Event Routing │
│ • Host Records │ │ • Reliability │
│ • SBOM Storage │ │ • Audit Trail │
│ • Reporting │ │ • Backpressure │
└─────────────────┘ └─────────────────┘
This diagram might look complex at first glance, but it tells a story of careful evolution and learned experience. Each connection represents a deliberate choice about how data should flow, and each component exists to solve a specific class of problems we encountered in production environments.
The beauty of this architecture lies not in its complexity, but in its clarity of purpose. Data flows in predictable patterns, failures are isolated to specific components, and scaling happens where it's needed most.
Why This Architecture?
Understanding our architectural evolution is crucial because it illustrates why simple solutions often fail in production environments. We didn't arrive at this design through academic exercise—we built it through iterative problem-solving in real-world scenarios.
This architecture emerged from several failed attempts and hard-learned lessons:
Evolution 1: Monolithic Scanner
Our first approach was a single application that handled everything. Like many first attempts, it seemed elegant in its simplicity. One codebase, one deployment, one thing to monitor. What could go wrong?
As it turns out, quite a lot. This approach failed because:
- Tight coupling made changes risky and slow
- Scaling required scaling everything together
- Failures in one component brought down the entire system
- Testing was complex and brittle
The Lesson: Simplicity at the architectural level often creates complexity everywhere else.
Evolution 2: Microservices Without Clear Boundaries
Learning from our monolithic mistakes, we swung to the other extreme. We broke everything into small services, thinking that if some decoupling was good, more must be better.
Our second attempt split functionality into many small services. This failed because:
- Service boundaries were unclear and overlapping
- Network communication overhead was excessive
- Data consistency became a nightmare
- Debugging distributed problems was extremely difficult
The Lesson: Microservices are a tool, not a goal. The boundaries matter more than the number of services.
Evolution 3: Component-Based with Clear Responsibilities
Our current architecture represents the synthesis of these lessons. We kept the benefits of decoupling while establishing clear responsibilities and communication patterns.
Our current architecture (Evolution 3) succeeds because:
- Each component has a clear, single responsibility
- Communication patterns match the actual needs (async vs real-time)
- Data flow is unidirectional and predictable
- Failure modes are isolated and well-understood
The Lesson: Architecture is about trade-offs, not absolutes. The right design balances complexity with capability.
Architectural Wisdom: What We Learned
These iterations taught us several important principles that guide our current design:
- Complexity should serve a purpose: Every architectural decision should solve a real problem, not just look elegant on paper
- Communication patterns matter: Match your messaging approach to your actual requirements
- Failure modes are features: Design for failure, because in distributed systems, failure is inevitable
- Evolution over revolution: Systems that can adapt over time outlive those designed for one specific scenario
As we explore the specific technology choices in the next section, you'll see how these lessons influenced every decision from programming languages to database design.
🛠️ Technology Stack & Rationale
The Philosophy Behind Our Choices
Technology selection in a security platform is never just about technical capabilities—it's about risk management, team productivity, and long-term maintainability. Every technology choice represents a bet on the future, and in security systems, the stakes of being wrong are particularly high.
Our technology stack evolved from practical constraints and hard-learned lessons. We didn't choose technologies because they were fashionable; we chose them because they solved specific problems we encountered in production environments.
Let's explore each choice and the reasoning behind it.
Frontend Technology Choices
Next.js + TypeScript + tRPC
The frontend of a security platform has unique requirements. Security analysts need dashboards that load quickly, display complex data clearly, and remain responsive under load. The technology choices we made serve these specific needs.
Why Next.js?
Security dashboards are different from typical web applications. They need to display large amounts of data, often in real-time, while remaining fast and responsive. Next.js provides several capabilities that directly address these requirements:
- Server-Side Rendering: Better performance and SEO for security dashboards
- API Routes: Convenient backend integration without separate API server
- File-Based Routing: Intuitive navigation structure for security workflows
- Built-in Optimization: Automatic code splitting and image optimization
The Story Behind TypeScript
We didn't start with TypeScript—we migrated to it after experiencing too many runtime errors in production. When you're dealing with security data, a small type error can mean the difference between catching a critical vulnerability and missing it entirely.
Why TypeScript?
- Type Safety: Prevents runtime errors in security-critical applications
- Developer Experience: Better IntelliSense and refactoring capabilities
- API Contract Enforcement: Ensures UI and backend stay synchronized
- Documentation: Types serve as living documentation
The tRPC Revelation
Traditional REST APIs create a maintenance burden—you have to keep frontend and backend in sync manually. tRPC eliminates this problem entirely by sharing types across the full stack.
Why tRPC?
- End-to-End Type Safety: Types flow from database to UI without manual synchronization
- Developer Productivity: No need to write API documentation or client SDKs
- Automatic Serialization: Handles complex data types automatically
- Built-in Validation: Request/response validation happens automatically
Example of Type Safety in Action:
// Backend procedure definition
export const getTemplates = publicProcedure.query(async ({ ctx }) => {
const templates = await ctx.valkey.getAllTemplates();
return templates; // TypeScript knows the exact return type
});
// Frontend usage - IntelliSense knows the structure
const { data: templates } = api.agent.getTemplates.useQuery();
// TypeScript knows 'templates' has exactly the right structure
templates?.forEach((template) => {
console.log(template.info.name); // ✅ Type-safe
console.log(template.invalidField); // ❌ TypeScript error
});
This type safety isn't just convenient—it's essential for security applications where data integrity directly impacts security outcomes.
Backend Technology Choices
Go for Agent Server and Endpoint Agents
The choice of programming language for security tools is critical. You need performance, reliability, and security, but you also need developer productivity and ease of deployment. Go strikes an excellent balance across all these requirements.
Why Go?
The decision to use Go wasn't made lightly. We evaluated several alternatives and Go consistently won across the criteria that mattered most for our use case:
- Performance: Native compilation and minimal runtime overhead
- Concurrency: Goroutines handle thousands of simultaneous agent connections
- Cross-Platform: Single codebase deploys to Windows, Linux, and macOS
- Static Binaries: No runtime dependencies simplify deployment
- Security: Memory safety and minimal attack surface
Specific Go Advantages for Our Use Case:
// Concurrent agent handling example
func (s *Server) handleAgentConnections() {
for {
conn, err := s.grpcServer.Accept()
if err != nil {
log.Error("Failed to accept connection", "error", err)
continue
}
// Each agent gets its own goroutine - scales to thousands
go s.handleSingleAgent(conn)
}
}
gRPC for Agent Communication
Why gRPC over REST?
- Bidirectional Streaming: Agents and server can communicate simultaneously
- Schema Evolution: Protocol buffers provide backward/forward compatibility
- Performance: Binary encoding is faster than JSON for high-volume data
- Type Safety: Generated code prevents communication errors
Example Protocol Buffer Definition:
// Clear contract between agent and server
service AgentService {
rpc Communicate(stream ClientMessage) returns (stream ServerMessage);
}
message ServerMessage {
string command_id = 1;
string type = 2; // "scan", "template_sync", "internal:status"
string payload = 3;
int64 timestamp = 4;
}
message ClientMessage {
string agent_id = 1;
string command_id = 2;
int32 exit_code = 3;
string output = 4;
int64 timestamp = 5;
}
Storage Technology Choices
ValKey as Single Source of Truth
Why ValKey (Redis-compatible)?
- In-Memory Performance: Microsecond response times for agent queries
- Data Structure Variety: Lists, sets, hashes, and strings for different use cases
- Atomic Operations: ACID guarantees for critical state changes
- Pub/Sub Capabilities: Real-time notifications for distributed components
- Persistence Options: Configurable durability vs. performance trade-offs
ValKey Data Organization Strategy:
agent:template:{template_id} # Template YAML content
agent:template:meta:{template_id} # Metadata (source, sync time, etc.)
agent:template:manifest # Global template index
agent:state:{agent_id} # Agent connection state
agent:heartbeat:{agent_id} # Agent health status
scan:result:{scan_id} # Scan results cache
scan:queue:pending # Queued scan requests
Why This Key Structure?
- Predictable Patterns: Easy to find related data
- Efficient Queries: Pattern matching works well
- Logical Grouping: Related data is co-located
- Cache-Friendly: Frequently accessed data can be kept in memory
PostgreSQL for Persistent Data
Why PostgreSQL for vulnerability correlation?
- JSONB Support: Store flexible SBOM data while maintaining query performance
- ACID Transactions: Ensure data consistency during complex updates
- Sophisticated Querying: Complex vulnerability correlation across hosts
- Proven Scalability: Battle-tested for enterprise workloads
JSONB Schema Design Example:
-- Flexible SBOM storage with structured querying
ALTER TABLE hosts ADD COLUMN software_inventory JSONB;
ALTER TABLE hosts ADD COLUMN system_fingerprint JSONB;
-- Efficient queries on JSON data
CREATE INDEX idx_hosts_packages_gin ON hosts USING GIN ((software_inventory->'packages'));
-- Example query: Find all hosts with vulnerable Apache versions
SELECT hostname, software_inventory->'packages'
FROM hosts
WHERE software_inventory->'packages' @> '[{"name": "apache2", "version": "2.4.41"}]';
RabbitMQ for Asynchronous Communication
Why RabbitMQ?
- Reliable Delivery: Messages persist until processed
- Complex Routing: Route messages based on content and metadata
- Flow Control: Prevents fast producers from overwhelming slow consumers
- Management Interface: Built-in monitoring and debugging tools
Message Flow Design:
UI Template Creation → agent_content_sync Queue → Agent Server Processing
↓
Template Stored in ValKey → Template Available to Agents
Why This Pattern?
- Decoupling: UI doesn't need to wait for template processing
- Reliability: Template creation requests can't be lost
- Scalability: Multiple agent servers can process templates
- Observability: Queue depths show system load
🔧 Core Components Deep Dive
Component 1: Sirius UI (Frontend)
Purpose and Responsibilities
The Sirius UI serves as the primary interface for security analysts and administrators. It's responsible for:
- Template Management: Creating, editing, and organizing vulnerability detection templates
- Agent Monitoring: Viewing agent status, health, and scan results
- Vulnerability Visualization: Presenting scan results in actionable formats
- System Administration: Managing agent configurations and scanning policies
Key Architecture Decisions
Decision 1: Direct ValKey Integration for Read Operations
Instead of going through the Agent Server for all operations, we connect directly to ValKey for read operations:
// Direct ValKey connection for fast reads
export const agentRouter = createTRPCRouter({
getTemplatesFromValKey: publicProcedure.query(async ({ ctx }) => {
// Direct read from ValKey - no network hops through Agent Server
const templates = await ctx.valkey.getAllTemplates();
return templates;
}),
});
Why This Decision?
- Performance: Eliminates network hops for frequently accessed data
- Reduced Load: Agent Server can focus on agent communication
- Consistency: UI always sees the same data that agents see
- Scalability: Read operations scale independently
Decision 2: RabbitMQ for Write Operations
Write operations (like creating templates) go through RabbitMQ:
createTemplate: publicProcedure
.input(
z.object({
id: z.string(),
content: z.string(),
})
)
.mutation(async ({ input, ctx }) => {
// Async message to Agent Server for processing
await ctx.rabbitMQ.publish("agent_content_sync", {
operation: "create",
type: "template",
id: input.id,
content: input.content,
});
});
Why This Decision?
- Reliability: Template creation can't fail due to temporary Agent Server issues
- Processing Time: Complex template validation happens asynchronously
- User Experience: UI responds immediately without waiting for processing
- Audit Trail: All template changes are logged in message queue
Component Architecture
Sirius UI Component Structure:
├── pages/
│ ├── dashboard/
│ │ ├── index.tsx # Main security dashboard
│ │ ├── agents.tsx # Agent management interface
│ │ └── vulnerabilities.tsx # Vulnerability reporting
│ └── templates/
│ ├── index.tsx # Template listing and management
│ ├── create.tsx # Template creation wizard
│ └── editor.tsx # Template editing interface
├── components/
│ ├── agent/
│ │ ├── AgentTemplatesTab.tsx # Template display component
│ │ ├── AgentStatusCard.tsx # Agent health monitoring
│ │ └── ScanResultsViewer.tsx # Scan result presentation
│ └── templates/
│ ├── TemplateEditor.tsx # YAML template editor
│ ├── TemplateValidator.tsx # Real-time validation
│ └── TemplatePreview.tsx # Template execution preview
├── utils/ # Shared utility functions
│ ├── yamlConverter.ts # YAML ↔ JSON conversion utilities
│ ├── types.ts # TypeScript interfaces and types
│ ├── constants.ts # Configuration constants and messages
│ ├── monacoUtils.ts # Monaco editor configurations
│ └── scriptTemplates.ts # Script template generation
└── server/
└── api/
└── routers/
├── agent.ts # Agent-related tRPC procedures
├── template.ts # Template management procedures
└── vulnerability.ts # Vulnerability data procedures
Utility Organization Strategy
Our frontend follows a modular utility approach that promotes code reuse and maintainability:
Core Utility Functions:
// yamlConverter.ts - Centralized conversion logic
export const convertToYamlForEditing = (contentData: unknown): string => {
// Handles conversion from JSON objects to YAML for editing
};
export const convertYamlToJson = (
yamlContent: string
): Record<string, unknown> => {
// Handles conversion from YAML back to JSON for API storage
};
// types.ts - Comprehensive type definitions
export interface TemplateContent {
id: string;
info: TemplateInfo;
detection: TemplateDetection;
remediation: TemplateRemediation;
}
// constants.ts - Centralized configuration
export const VALIDATION_MESSAGES = {
requiredFields: "Please fill in all required fields",
invalidContent: "Invalid YAML/JSON content",
// ... more validation messages
} as const;
// monacoUtils.ts - Editor configurations
export const getMonacoLanguage = (
scriptLanguage: ScriptLanguage
): MonacoLanguage => {
// Maps script languages to Monaco editor languages
};
export const MONACO_HEIGHTS = {
preview: "300px",
editor: "500px",
viewer: "400px",
} as const;
Why This Organization?
- DRY Principle: Eliminates code duplication across components
- Type Safety: Centralized types ensure consistency across the application
- Maintainability: Changes to utility functions automatically propagate everywhere
- Testing: Pure utility functions are easier to unit test
- Performance: Shared utilities can be optimized once and benefit the entire application
Data Flow Through UI Components
Template Creation Flow:
- User opens Template Creation Wizard (
pages/templates/create.tsx) - Component renders form with real-time validation (
TemplateEditor.tsx) - User submits template, triggering tRPC mutation
- Mutation sends RabbitMQ message to Agent Server
- UI shows success message and redirects to template list
- Template list refreshes from ValKey, showing new template
Agent Monitoring Flow:
- Agent Status Cards poll for updates every 30 seconds
- tRPC query fetches agent states from ValKey
- Component renders health status with visual indicators
- Click on agent opens detailed view with scan history
- Scan history queries PostgreSQL for historical data
Component 2: Agent Server (Backend Orchestrator)
Purpose and Responsibilities
The Agent Server is the coordination hub of the entire system. It's responsible for:
- Template Synchronization: Keeping ValKey updated with templates from all sources
- Agent Communication: Managing gRPC connections with endpoint agents
- Command Distribution: Routing scan requests and other commands to appropriate agents
- Content Processing: Validating and processing templates and scripts from the UI
Key Architecture Decisions
Decision 1: Template Sync Service Design
We built a comprehensive synchronization service that handles three template sources:
// Template sync coordinates three sources
func (ts *TemplateSyncService) SyncAllTemplatesToValKey() error {
results := &SyncResults{}
// 1. Repository templates (from sirius-agent-modules)
if err := ts.syncRepositoryTemplates(results); err != nil {
log.Warn("Repository template sync failed", "error", err)
}
// 2. Local templates (from filesystem)
if err := ts.syncLocalTemplates(results); err != nil {
log.Warn("Local template sync failed", "error", err)
}
// 3. Custom templates (from UI via RabbitMQ)
if err := ts.verifyCustomTemplates(results); err != nil {
log.Warn("Custom template verification failed", "error", err)
}
return nil
}
Why This Three-Source Design?
- Repository Templates: Centrally maintained, version-controlled templates for common vulnerabilities
- Custom Templates: User-created templates for organization-specific needs
- Local Templates: Development and testing templates that don't need central distribution
Decision 2: Graceful Degradation
If one template source fails, others continue to work:
func (ts *TemplateSyncService) syncRepositoryTemplates(results *SyncResults) error {
return filepath.Walk(repoPath, func(path string, info os.FileInfo, err error) error {
// ... template loading logic ...
template, err := ts.loadTemplate(path)
if err != nil {
log.Warn("Failed to sync repository template", "path", path, "error", err)
return nil // Continue with other templates instead of failing
}
// ... successful processing ...
return nil
})
}
Why This Decision?
- Resilience: One broken template doesn't break the entire system
- Operational Simplicity: System continues working during template debugging
- Development Experience: Developers can work on templates without breaking production
Decision 3: gRPC Connection Management
We maintain persistent bidirectional streams with agents:
type Server struct {
agents map[string]*AgentConnection
mu sync.RWMutex
}
type AgentConnection struct {
AgentID string
Stream pb.AgentService_CommunicateServer
LastSeen time.Time
Status AgentStatus
Commands chan *pb.ServerMessage
}
func (s *Server) handleAgentStream(stream pb.AgentService_CommunicateServer) {
// Each agent gets dedicated goroutine for message handling
for {
msg, err := stream.Recv()
if err != nil {
// Clean up connection and notify monitoring
s.removeAgent(agentID)
return
}
// Process agent message asynchronously
go s.processAgentMessage(msg)
}
}
Why This Design?
- Real-Time Communication: Agents can receive commands immediately
- Connection Monitoring: Server knows which agents are alive
- Scalability: Each agent operates independently
- Fault Tolerance: Agent disconnections don't affect other agents
Template Processing Pipeline
When a new template arrives via RabbitMQ, it goes through a comprehensive processing pipeline:
func (ccc *CustomContentConsumer) processTemplateMessage(message *CustomContentMessage) error {
// Step 1: Parse and validate template structure
template := &Template{}
if err := mapstructure.Decode(message.Content, template); err != nil {
return fmt.Errorf("failed to decode template: %w", err)
}
// Step 2: Validate template semantics
if err := ccc.validateTemplate(template); err != nil {
return fmt.Errorf("template validation failed: %w", err)
}
// Step 3: Save to filesystem for persistence
if err := ccc.customStorage.SaveTemplate(template); err != nil {
return fmt.Errorf("failed to save template: %w", err)
}
// Step 4: Sync to ValKey for agent access
templateMeta := &TemplateMetadata{
Source: &TemplateSource{Type: "custom"},
SyncedAt: time.Now(),
}
if err := ccc.valKeyStore.StoreTemplate(template.ID, template, templateMeta); err != nil {
return fmt.Errorf("failed to sync template to ValKey: %w", err)
}
log.Info("Custom template processed successfully", "template_id", template.ID)
return nil
}
Why This Pipeline?
- Data Integrity: Multiple validation steps prevent corrupt templates
- Persistence: Templates survive server restarts
- Immediate Availability: Templates are available to agents as soon as processing completes
- Audit Trail: Each step is logged for debugging
Component 3: ValKey Storage System
Purpose and Responsibilities
ValKey serves as the single source of truth for all real-time operational data:
- Template Storage: All vulnerability detection templates and their metadata
- Agent State: Current status and configuration of all agents
- Scan Coordination: Queuing and tracking of scan requests
- Real-Time Cache: Fast access to frequently needed data
Data Organization Strategy
Key Naming Conventions:
agent:template:{template_id} # Template content (YAML as JSON)
agent:template:meta:{template_id} # Metadata (source, sync time, version)
agent:template:manifest # Global template registry
agent:state:{agent_id} # Agent connection and status
agent:heartbeat:{agent_id} # Agent health (with TTL)
scan:result:{scan_id} # Cached scan results
scan:queue:pending # Queued scan requests
Why This Structure?
- Hierarchical Organization: Related data is grouped by prefix
- Pattern Matching: Easy to find all templates, all agents, etc.
- Cache Efficiency: Frequently accessed data is co-located
- TTL Management: Health data expires automatically
Template Storage Design
Templates are stored in two parts for efficiency:
// Template content - the actual YAML template
templateKey := "agent:template:" + templateID
templateJSON, _ := json.Marshal(template)
store.client.Set(ctx, templateKey, templateJSON, 0)
// Template metadata - source, sync time, etc.
metaKey := "agent:template:meta:" + templateID
metaJSON, _ := json.Marshal(metadata)
store.client.Set(ctx, metaKey, metaJSON, 0)
Why Split Storage?
- Performance: Agents only need template content, not metadata
- Flexibility: Metadata can be updated without touching template content
- Caching: Different cache policies for content vs. metadata
- Debugging: Can examine metadata without loading large template content
Agent State Management
Agent state is managed with careful attention to connection lifecycle:
// Agent connects - establish state
agentState := &AgentState{
AgentID: agentID,
Status: "connected",
ConnectedAt: time.Now(),
LastSeen: time.Now(),
Capabilities: agentCapabilities,
}
store.client.Set(ctx, "agent:state:"+agentID, agentJSON, 0)
// Heartbeat with TTL - auto-cleanup on disconnect
store.client.Set(ctx, "agent:heartbeat:"+agentID, "alive", 60*time.Second)
Why This Design?
- Automatic Cleanup: Heartbeats expire if agent disconnects ungracefully
- State Persistence: Agent state survives brief network interruptions
- Monitoring: Easy to identify healthy vs. unhealthy agents
- Debugging: Complete connection history available
Component 4: Endpoint Agent (Scanner)
Purpose and Responsibilities
Endpoint agents are the "sensors" of our security system. Each agent is responsible for:
- System Inspection: Collecting comprehensive information about the host system
- Vulnerability Detection: Executing templates to identify security issues
- SBOM Collection: Gathering software inventory for vulnerability correlation
- Command Execution: Running security assessment scripts and tools
Agent Architecture Design
Connection Management:
type Agent struct {
serverAddress string
agentID string
grpcClient pb.AgentServiceClient
stream pb.AgentService_CommunicateClient
// Core capabilities
templateEngine *template.Engine
scanCommand *scan.ScanCommand
scriptExecutor *script.Executor
// State management
heartbeatTicker *time.Ticker
commandQueue chan *pb.ServerMessage
resultQueue chan *pb.ClientMessage
}
Why This Design?
- Single Connection: One gRPC stream handles all communication
- Asynchronous Processing: Commands and results are queued for processing
- Capability-Based: Agent capabilities are detected and reported
- Health Monitoring: Regular heartbeats ensure server knows agent status
Template Execution Engine
The template engine is responsible for executing vulnerability detection templates:
func (c *ScanCommand) executeTemplateDetection(ctx context.Context, result *ScanResult) ([]TemplateDetectionResult, error) {
// Load all available templates
templates, err := c.templateEngine.LoadTemplates()
if err != nil {
return nil, fmt.Errorf("failed to load templates: %w", err)
}
var results []TemplateDetectionResult
// Execute each template independently
for _, template := range templates {
detectResult, err := c.templateEngine.ExecuteTemplate(ctx, template, agentInfo)
if err != nil {
log.Warn("Template execution failed", "template_id", template.ID, "error", err)
continue // Don't let one bad template break everything
}
if detectResult.Vulnerable {
results = append(results, TemplateDetectionResult{
TemplateID: template.ID,
VulnerabilityID: detectResult.VulnerabilityID,
Vulnerable: true,
Confidence: detectResult.Confidence,
Evidence: detectResult.Evidence,
ExecutedAt: time.Now(),
})
}
}
return results, nil
}
Why This Design?
- Isolation: Each template executes independently
- Resilience: Template failures don't affect other templates
- Performance: Templates can be executed in parallel
- Debugging: Clear separation between template logic and execution engine
SBOM Collection Strategy
Software Bill of Materials (SBOM) collection adapts to each platform:
// Platform-specific package collection
type PackageCollector interface {
CollectPackages(ctx context.Context) ([]EnhancedPackageInfo, error)
}
// Linux implementation
type DebianPackageCollector struct{}
func (dpc *DebianPackageCollector) CollectPackages(ctx context.Context) ([]EnhancedPackageInfo, error) {
// Use dpkg-query for comprehensive package information
cmd := exec.CommandContext(ctx, "dpkg-query", "-W",
"--showformat=${Package}\t${Version}\t${Architecture}\t${Installed-Size}\t${Description}\n")
// ... processing logic ...
}
// Windows implementation
type WindowsPackageCollector struct{}
func (wpc *WindowsPackageCollector) CollectPackages(ctx context.Context) ([]EnhancedPackageInfo, error) {
// Use PowerShell to query registry and WMI
script := `Get-WmiObject -Class Win32_Product | Select-Object Name, Version, InstallDate`
// ... processing logic ...
}
Why Platform-Specific Collectors?
- Accuracy: Each platform has different package management systems
- Performance: Native tools are faster than generic approaches
- Completeness: Platform-specific tools see all installed software
- Reliability: Uses the same tools system administrators use
🔄 Data Flow Narratives
Understanding the System Through Stories
One of the best ways to understand a complex distributed system is to follow the journey of data as it moves through the components. Think of these data flows as stories—each one tells you something important about how the system behaves, why we made certain design choices, and what happens when things go wrong.
These narratives aren't just technical documentation—they're debugging guides, onboarding tools, and architectural explanations all rolled into one. When you understand how data moves through the system, you understand the system itself.
Understanding how data flows through our system is crucial for debugging, extending functionality, and understanding system behavior. Let's walk through the major data flows step by step, treating each one as a complete story with characters, challenges, and resolutions.
Data Flow 1: Template Creation and Distribution
The Journey from Idea to Execution
This is one of our most important flows—how security templates get from a security analyst's mind into running vulnerability checks across thousands of systems. It's a story about taking human security knowledge and transforming it into automated, scalable detection.
Imagine Sarah, a security analyst, discovers a new SSH configuration vulnerability in the morning security briefing. By lunch, she wants that vulnerability checked across every server in the infrastructure. This flow is the story of how that happens.
Step 1: Template Creation in UI
Sarah sits down at her computer and opens the Sirius UI. She's identified a new SSH vulnerability and wants to create a detection template for it. The UI provides a friendly interface for what is actually a complex distributed operation.
A security analyst identifies a new vulnerability and wants to create a detection template.
// User fills out template creation form
const createTemplateForm = {
id: "SSH-CONFIG-002",
info: {
name: "SSH Key-Based Authentication Bypass",
severity: "high",
description: "Detects SSH configurations that allow authentication bypass",
},
detection: {
type: "config-file",
files: [
{
path: "/etc/ssh/sshd_config",
patterns: [
{
regex: "^\\s*PermitUserEnvironment\\s+yes",
description: "User environment variable override enabled",
},
],
},
],
},
};
// UI submits via tRPC mutation
const result = await api.agent.createTemplate.mutate({
id: createTemplateForm.id,
content: yaml.stringify(createTemplateForm),
});
What happens here: The UI validates the form data and converts it to YAML format. Instead of immediately showing the template as "created," the UI shows it as "processing" because the actual validation and distribution happens asynchronously.
Step 2: RabbitMQ Message Queue
The tRPC mutation publishes a message to RabbitMQ:
// tRPC procedure sends async message
await ctx.rabbitMQ.publish("agent_content_sync", {
operation: "create",
type: "template",
id: createTemplateForm.id,
content: createTemplateForm,
created_by: ctx.user.id,
timestamp: new Date().toISOString(),
});
Why RabbitMQ? Template processing involves validation, file I/O, and potentially slow ValKey operations. Using a message queue means:
- The UI doesn't freeze waiting for processing
- Template creation requests can't be lost if the Agent Server is temporarily down
- Multiple Agent Servers can process templates for scalability
- We get an automatic audit trail of all template operations
Step 3: Agent Server Processing
The Agent Server's RabbitMQ consumer picks up the message:
func (ccc *CustomContentConsumer) processTemplateMessage(message *CustomContentMessage) error {
log.Info("Processing template creation", "template_id", message.ID)
// Step 3a: Parse YAML content
template := &Template{}
if err := yaml.Unmarshal([]byte(message.Content), template); err != nil {
return fmt.Errorf("invalid YAML: %w", err)
}
// Step 3b: Validate template structure and logic
if err := ccc.validateTemplate(template); err != nil {
return fmt.Errorf("template validation failed: %w", err)
}
// Step 3c: Save to filesystem for persistence
templatePath := filepath.Join(ccc.templatesDir, template.ID+".yaml")
if err := ccc.writeTemplateFile(templatePath, template); err != nil {
return fmt.Errorf("failed to save template: %w", err)
}
// Step 3d: Store in ValKey for immediate agent access
templateMeta := &TemplateMetadata{
Source: &TemplateSource{
Type: "custom",
Path: templatePath,
CreatedBy: message.CreatedBy,
},
SyncedAt: time.Now(),
}
if err := ccc.valKeyStore.StoreTemplate(template.ID, template, templateMeta); err != nil {
return fmt.Errorf("failed to sync to ValKey: %w", err)
}
log.Info("Template processed successfully", "template_id", template.ID)
return nil
}
Template Validation: This is where we catch problems:
- YAML Syntax: Is the template properly formatted?
- Schema Validation: Does it have all required fields?
- Logic Validation: Are the detection conditions sensible?
- Security Validation: Could this template be used maliciously?
Step 4: ValKey Storage
The template is stored in ValKey with this structure:
# Template content (available to agents immediately)
agent:template:SSH-CONFIG-002 = {
"id": "SSH-CONFIG-002",
"info": { "name": "SSH Key-Based Authentication Bypass", ... },
"detection": { "type": "config-file", ... }
}
# Template metadata (for UI and management)
agent:template:meta:SSH-CONFIG-002 = {
"source": { "type": "custom", "path": "/custom-templates/SSH-CONFIG-002.yaml" },
"synced_at": "2024-01-15T10:30:00Z",
"created_by": "analyst@company.com"
}
# Update global manifest
agent:template:manifest = {
"templates": ["SSH-CONFIG-001", "SSH-CONFIG-002", ...],
"updated": "2024-01-15T10:30:00Z",
"total_count": 127
}
Step 5: Agent Template Loading
When agents next scan (or immediately if they're listening for updates), they load the new template:
func (te *TemplateEngine) LoadTemplates() ([]*Template, error) {
// Get all template keys from ValKey
keys, err := te.valKeyClient.Keys(ctx, "agent:template:*").Result()
if err != nil {
return nil, fmt.Errorf("failed to list templates: %w", err)
}
var templates []*Template
for _, key := range keys {
// Skip metadata keys - only load template content
if strings.Contains(key, ":meta:") {
continue
}
templateJSON, err := te.valKeyClient.Get(ctx, key).Result()
if err != nil {
log.Warn("Failed to load template", "key", key, "error", err)
continue
}
var template Template
if err := json.Unmarshal([]byte(templateJSON), &template); err != nil {
log.Warn("Failed to parse template", "key", key, "error", err)
continue
}
templates = append(templates, &template)
}
return templates, nil
}
Step 6: UI Feedback
The UI polls ValKey to see when the template becomes available:
// UI queries ValKey directly for real-time updates
const { data: templates } = api.agent.getTemplatesFromValKey.useQuery(
undefined,
{
refetchInterval: 5000, // Check every 5 seconds
onSuccess: (data) => {
// Show success notification when template appears
const newTemplate = data.find((t) => t.id === "SSH-CONFIG-002");
if (newTemplate && !previousTemplates.includes(newTemplate)) {
showNotification("Template created successfully!");
}
},
}
);
Why This Flow? This seemingly complex flow provides several benefits:
- Reliability: Template creation can't fail silently
- Performance: UI stays responsive during processing
- Consistency: All components see templates at the same time
- Auditability: Complete trail of template creation and distribution
Data Flow 2: Agent Scan Execution
This flow shows how a security scan request travels from the UI through the system and back with results.
Step 1: Scan Initiation
A security analyst wants to scan a specific host or group of hosts:
// UI scan request
const scanRequest = {
target: "production-web-servers", // Or specific agent ID
scanType: "full", // full, quick, templates-only
includeTemplates: true,
includeSBOM: true,
priority: "high",
};
const result = await api.agent.triggerScan.mutate(scanRequest);
Step 2: Scan Coordination
The Agent Server receives the scan request and determines which agents should execute it:
func (s *Server) triggerScan(req *ScanRequest) error {
// Determine target agents
targetAgents, err := s.resolveTargets(req.Target)
if err != nil {
return fmt.Errorf("failed to resolve targets: %w", err)
}
// Create scan commands for each agent
for _, agentID := range targetAgents {
scanCommand := &pb.ServerMessage{
CommandId: generateCommandID(),
Type: "scan",
Payload: marshalScanRequest(req),
Timestamp: time.Now().Unix(),
}
// Send to agent via gRPC stream
if err := s.sendCommandToAgent(agentID, scanCommand); err != nil {
log.Warn("Failed to send scan command", "agent_id", agentID, "error", err)
continue
}
// Track command for result correlation
s.trackCommand(scanCommand.CommandId, agentID, req)
}
return nil
}
Step 3: Agent Scan Execution
Each target agent receives the scan command and executes it:
func (a *Agent) handleScanCommand(msg *pb.ServerMessage) {
log.Info("Received scan command", "command_id", msg.CommandId)
// Parse scan parameters
scanReq := &ScanRequest{}
if err := json.Unmarshal([]byte(msg.Payload), scanReq); err != nil {
a.sendCommandResult(msg.CommandId, 1, fmt.Sprintf("Invalid scan request: %v", err))
return
}
// Execute comprehensive scan
startTime := time.Now()
result, err := a.scanCommand.ExecuteScan(context.Background(), ScanOptions{
IncludeTemplates: scanReq.IncludeTemplates,
IncludeSBOM: scanReq.IncludeSBOM,
ScanType: scanReq.ScanType,
})
if err != nil {
a.sendCommandResult(msg.CommandId, 1, fmt.Sprintf("Scan failed: %v", err))
return
}
// Enrich result with execution metadata
result.ScanSummary.ExecutionTime = time.Since(startTime)
result.ScanSummary.CommandID = msg.CommandId
result.ScanSummary.AgentVersion = a.version
// Send results back to server
resultJSON, _ := json.Marshal(result)
a.sendCommandResult(msg.CommandId, 0, string(resultJSON))
}
Step 4: Comprehensive Data Collection
The agent's scan command orchestrates multiple data collection activities:
func (sc *ScanCommand) ExecuteScan(ctx context.Context, options ScanOptions) (*ScanResult, error) {
result := &ScanResult{
AgentID: sc.agentID,
Hostname: sc.getHostname(),
Platform: runtime.GOOS,
ScanTimestamp: time.Now(),
}
// Collect SBOM data if requested
if options.IncludeSBOM {
packages, err := sc.collectSoftwareInventory(ctx)
if err != nil {
log.Warn("SBOM collection failed", "error", err)
} else {
result.Packages = packages
result.ScanSummary.PackagesCollected = len(packages)
}
// Collect system fingerprint
fingerprint, err := sc.collectSystemFingerprint(ctx)
if err != nil {
log.Warn("System fingerprint collection failed", "error", err)
} else {
result.SystemFingerprint = fingerprint
}
}
// Execute vulnerability templates if requested
if options.IncludeTemplates {
templateResults, err := sc.executeTemplateDetection(ctx, result)
if err != nil {
log.Warn("Template detection failed", "error", err)
} else {
result.TemplateResults = templateResults
result.ScanSummary.TemplatesExecuted = len(templateResults)
result.ScanSummary.VulnerabilitiesFound = countVulnerabilities(templateResults)
}
}
return result, nil
}
Step 5: Result Processing and Storage
When the Agent Server receives scan results, it processes and stores them:
func (s *Server) processScanResult(agentID string, commandID string, resultJSON string) error {
// Parse scan result
var scanResult ScanResult
if err := json.Unmarshal([]byte(resultJSON), &scanResult); err != nil {
return fmt.Errorf("failed to parse scan result: %w", err)
}
// Convert to database format
host, softwareInventory, systemFingerprint, agentMetadata, err :=
s.convertScanResultToHost(scanResult)
if err != nil {
return fmt.Errorf("failed to convert scan result: %w", err)
}
// Store in PostgreSQL via Sirius API
if err := s.siriusAPI.CreateOrUpdateHost(host, softwareInventory, systemFingerprint, agentMetadata); err != nil {
return fmt.Errorf("failed to store scan result: %w", err)
}
// Cache result in ValKey for fast access
resultKey := fmt.Sprintf("scan:result:%s", commandID)
if err := s.valKeyClient.Set(context.Background(), resultKey, resultJSON, 24*time.Hour).Err(); err != nil {
log.Warn("Failed to cache scan result", "command_id", commandID, "error", err)
}
log.Info("Scan result processed successfully",
"agent_id", agentID,
"command_id", commandID,
"packages_found", len(scanResult.Packages),
"vulnerabilities_found", len(scanResult.TemplateResults))
return nil
}
Step 6: UI Visualization
The UI retrieves and displays scan results:
// Query scan results with real-time updates
const { data: scanResults } = api.scan.getRecentResults.useQuery(
{ agentId: selectedAgent.id },
{
refetchInterval: 10000, // Update every 10 seconds
onSuccess: (data) => {
// Update dashboard with new vulnerabilities
updateVulnerabilityMetrics(data);
// Show notifications for critical findings
const criticalVulns = data.filter((r) => r.severity === "critical");
if (criticalVulns.length > 0) {
showCriticalAlert(
`${criticalVulns.length} critical vulnerabilities found!`
);
}
},
}
);
// Render results in dashboard
return (
<div className="scan-results">
<ScanMetrics
packagesFound={scanResults?.packages?.length || 0}
vulnerabilitiesFound={scanResults?.vulnerabilities?.length || 0}
lastScanTime={scanResults?.timestamp}
/>
<VulnerabilityList
vulnerabilities={scanResults?.vulnerabilities || []}
onVulnerabilityClick={showVulnerabilityDetails}
/>
<SoftwareInventory
packages={scanResults?.packages || []}
onPackageClick={showPackageDetails}
/>
</div>
);
Key Insights from This Flow:
- Asynchronous by Design: Each step can proceed independently
- Error Isolation: Failures in one part don't break the entire scan
- Rich Context: SBOM data provides context for vulnerability assessment
- Real-Time Updates: UI gets results as soon as they're available
Data Flow 3: Template Synchronization
This flow shows how repository templates get synchronized across the system.
Repository-Based Template Updates
Templates maintained in the sirius-agent-modules repository automatically sync to all agents:
// Triggered on Agent Server startup and periodically
func (ts *TemplateSyncService) syncRepositoryTemplates(results *SyncResults) error {
repoPath := "/app-agent/sirius-agent-modules/templates"
// Walk through all YAML files in repository
return filepath.Walk(repoPath, func(path string, info os.FileInfo, err error) error {
if !strings.HasSuffix(info.Name(), ".yaml") {
return nil
}
// Load and validate each template
template, err := ts.loadTemplate(path)
if err != nil {
log.Warn("Failed to load repository template", "path", path, "error", err)
return nil // Continue with other templates
}
// Store in ValKey with repository source metadata
templateMeta := &TemplateMetadata{
Source: &TemplateSource{
Type: "repository",
Path: path,
Version: ts.getRepositoryVersion(),
},
SyncedAt: time.Now(),
}
if err := ts.valKeyStore.StoreTemplate(template.ID, template, templateMeta); err != nil {
log.Warn("Failed to store repository template", "template_id", template.ID, "error", err)
return nil
}
log.Debug("Synced repository template", "template_id", template.ID, "path", path)
results.RepositoryTemplates++
return nil
})
}
Why This Approach?
- Version Control: Repository templates are version-controlled and peer-reviewed
- Consistency: All environments get the same templates
- Automatic Updates: New templates deploy without manual intervention
- Rollback Capability: Git history allows rolling back problematic templates
🎨 Template System Design
The Vision: Security Knowledge as Code
The template system is one of our most sophisticated components, designed to make vulnerability detection as easy as writing a YAML file. But it's more than just a technical feature—it represents a fundamental shift in how we think about security knowledge.
Traditional security tools treat vulnerability detection as a programming problem. Our template system treats it as a knowledge management problem. The difference is profound.
Understanding how templates work is crucial for both using and extending the system, but more importantly, understanding the philosophy behind templates will change how you think about scalable security automation.
The Problem: Security Knowledge Trapped in Code
Imagine this scenario: A new SSH vulnerability is discovered on Tuesday. In traditional security tools, here's what happens:
- A security researcher identifies the vulnerability
- They write a technical paper describing it
- A software developer reads the paper
- The developer writes code to detect the vulnerability
- The code goes through review, testing, and deployment
- Finally, weeks later, the detection is available
This process has several critical flaws that become apparent at scale.
Philosophy Behind Template-Based Detection
Traditional vulnerability scanners require developers to write custom code for each new vulnerability. This creates several problems that compound over time:
- High Barrier to Entry: Only developers can create new detections
- Deployment Complexity: Code changes require full application deployments
- Risk: Code bugs can crash the entire scanner
- Maintenance Overhead: Each detection requires ongoing code maintenance
- Knowledge Silos: Security expertise gets locked away in code that only developers can modify
The human cost of this approach is enormous. Security experts become dependent on developers for everything, developers become bottlenecks for security response, and the feedback loop between threat discovery and detection deployment stretches to weeks or months.
Our template system solves these problems by fundamentally changing the relationship between security knowledge and implementation:
- Declarative Configuration: Security analysts describe WHAT to look for, not HOW to look for it
- Hot Deployment: New templates deploy without restarting agents
- Sandboxed Execution: Template errors don't affect other templates or the agent
- Community Contribution: Non-developers can contribute vulnerability detections
- Knowledge Accessibility: Security logic is readable and modifiable by security experts
The Vision: In our ideal world, the time from vulnerability discovery to detection deployment is measured in minutes, not weeks. Security experts should be able to codify their knowledge directly, without translation through development teams.
Templates as a Universal Language
Think of templates as a universal language for describing security vulnerabilities. Just as SQL lets you query databases without writing low-level file access code, templates let you define vulnerability detection without writing system-level inspection code.
This abstraction is powerful because it separates concerns:
- Security experts focus on what makes a system vulnerable
- The template engine handles how to inspect systems safely and efficiently
- The platform manages distribution, execution, and result processing
Template Anatomy
Let's understand a template by examining a real example and deconstructing each part to see how human security knowledge translates into automated detection:
# SSH Configuration Vulnerability Template
id: "SSH-CONFIG-001"
info:
name: "SSH Weak Configuration Detection"
author: "security-team"
severity: "medium"
description: "Detects SSH configurations that allow authentication bypass"
references:
- "https://nvd.nist.gov/vuln/detail/CVE-2023-12345"
cve: "CVE-2023-12345"
updated: "2024-01-15"
detection:
type: "config-file"
files:
- path: "/etc/ssh/sshd_config"
patterns:
- regex: "^\\s*PermitUserEnvironment\\s+yes"
description: "User environment variable override enabled"
- regex: "^\\s*PermitRootLogin\\s+yes"
description: "Direct root login permitted"
conditions:
- type: "file_exists"
value: true
- type: "pattern_found"
value: true
remediation:
description: "Disable dangerous SSH configuration options"
commands:
linux: |
sudo sed -i 's/^PermitUserEnvironment yes/PermitUserEnvironment no/' /etc/ssh/sshd_config
sudo sed -i 's/^PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config
sudo systemctl restart sshd
verification:
command: "sudo sshd -T | grep -E '(permituserenvironment|permitrootlogin)'"
expected_output: |
permituserenvironment no
permitrootlogin no
This template tells a complete story: what the vulnerability is, how to detect it, and how to fix it. Let's break down each section:
1. Metadata Section (info:)
This section provides context about the vulnerability - the "who, what, when, where, why" of security knowledge:
id: Unique identifier used throughout the systemname: Human-readable vulnerability nameseverity: Risk level that drives prioritizationcve: Links to existing vulnerability databases for correlationreferences: External documentation for deeper understanding
Why This Structure? This metadata enables the system to automatically prioritize, correlate, and report on vulnerabilities in ways that match how security teams actually work.
2. Detection Logic (detection:)
This is where security knowledge becomes executable logic:
detection:
type: "config-file" # How to look for the vulnerability
files: # Where to look
- path: "/etc/ssh/sshd_config"
patterns: # What patterns indicate danger
- regex: "^\\s*PermitUserEnvironment\\s+yes"
description: "User environment variable override enabled"
The beauty of this approach is that security experts can focus on the logic ("look for PermitUserEnvironment set to yes") while the template engine handles the implementation details (file reading, regex execution, error handling).
3. Conditions (conditions:)
Conditions prevent false positives by ensuring context:
conditions:
- type: "file_exists" # File must exist
value: true
- type: "pattern_found" # Pattern must be found
value: true
This prevents the common problem of vulnerability scanners that report issues based on partial information.
4. Remediation Guidance (remediation:)
Templates don't just find problems—they help solve them:
remediation:
description: "Clear explanation of the fix"
commands:
linux: "Platform-specific commands"
windows: "Different commands for different systems"
verification:
command: "How to verify the fix worked"
This turns vulnerability detection into actionable security improvement, closing the loop from discovery to resolution.
💻 Development Workflows
The Daily Life of a Sirius Developer
Understanding development workflows is about more than just knowing commands—it's about understanding how to work effectively with a complex distributed system. This section tells the story of how developers actually interact with the system day-to-day.
Whether you're fixing a bug, adding a new feature, or investigating a production issue, these workflows have been battle-tested by real developers solving real problems.
Setting Up Your Development Environment
The Philosophy: Production Parity
Our development setup prioritizes one thing above all else: consistency with production. Every developer should have an environment that behaves exactly like production, because the alternative—debugging environment-specific issues—is a productivity killer.
Prerequisites
Before you can effectively work on Sirius, your development machine needs several tools. These aren't arbitrary requirements—each tool serves a specific purpose in our development workflow:
# Container orchestration - because our production runs in containers
docker --version # Docker for containerization
docker-compose --version # Docker Compose for orchestration
# Backend development - our agents and servers are written in Go
go version # Go 1.21+ for backend development
# Frontend development - our UI is built with modern web technologies
node --version # Node.js 18+ for frontend development
# Version control - for collaboration and deployment
git --version # Git for version control
Environment Setup: The Step-by-Step Journey
Setting up Sirius is designed to be straightforward, but understanding what each step accomplishes helps you troubleshoot when things go wrong:
# Step 1: Get the code
git clone https://github.com/SiriusScan/Sirius.git
cd Sirius
# Step 2: Configure for local development
cp docker-compose.local.example.yaml docker-compose.override.yaml
# Step 3: Start the foundation services
docker-compose up -d valkey rabbitmq postgres
# Step 4: Start the UI in development mode
cd sirius-ui
npm install
npm run dev
# Step 5: Start the agent server (in a separate terminal)
cd ../app-agent
docker exec -it sirius-engine /bin/bash
go run cmd/server/main.go
What's Really Happening: This setup creates a complete Sirius environment on your local machine. ValKey, RabbitMQ, and PostgreSQL run in containers (matching production), while the UI and Agent Server run in development mode for fast iteration.
Common Development Tasks
Task 1: Adding a New Template
Creating templates is one of the most common development tasks, and it illustrates the complete development cycle from idea to deployment.
The Story: You've discovered a new Nginx vulnerability and want to add detection for it. Here's the complete journey:
Step 1: Create the Template
# Navigate to template repository
cd sirius-agent-modules/templates/config-based
# Create the template file
cat > nginx-security-headers.yaml << EOF
id: "NGINX-SEC-001"
info:
name: "Nginx Security Headers Missing"
severity: "medium"
description: "Detects missing security headers in Nginx configuration"
detection:
type: "config-file"
files:
- path: "/etc/nginx/nginx.conf"
patterns:
- regex: "add_header X-Frame-Options"
description: "X-Frame-Options header configured"
conditions:
- type: "file_exists"
value: true
- type: "pattern_found"
value: false # Vulnerability when pattern NOT found
EOF
Step 2: Test Locally
# Enter the development container
docker exec -it sirius-engine /bin/bash
cd /app-agent
# Test template syntax
go run cmd/template-test/main.go --template=/app-agent/sirius-agent-modules/templates/config-based/nginx-security-headers.yaml
# Test template execution
./agent scan --templates-only --template-filter="NGINX-SEC-001"
Step 3: Deploy and Verify
# Restart Agent Server to pick up new template
docker restart sirius-engine
# Check synchronization logs
docker logs sirius-engine | grep "NGINX-SEC-001"
# Verify storage in ValKey
docker exec sirius-engine valkey-cli get "agent:template:NGINX-SEC-001"
This workflow embodies our development philosophy: test early, test often, and verify at each step.
The Art of Debugging
Debugging Agent Connection Issues
When agents won't connect, the investigation follows a systematic approach that's saved us countless hours:
The Detective Story: An agent shows as offline in the UI. Where do you start?
Step 1: Check the Foundation
# Is the Agent Server even running?
docker logs sirius-engine | grep "gRPC server listening"
# Are agents trying to connect?
docker logs sirius-engine | grep "Agent.*connected"
# Are there authentication failures?
docker logs sirius-engine | grep "authentication failed"
Step 2: Test the Network Path
# Can you reach the gRPC port?
telnet localhost 50051
# Is the gRPC service responding?
grpcurl -plaintext localhost:50051 list
Step 3: Check Agent-Side Issues
# What does the agent think is happening?
./agent internal:status
# Can the agent reach the server?
./agent ping --server=localhost:50051
This systematic approach turns mysterious connection problems into solvable puzzles.
💾 Database Strategy
The Tale of Two Databases
Our database architecture tells a story about trade-offs, performance, and the practical realities of building distributed systems. We use two different databases not because we love complexity, but because different types of data have fundamentally different requirements.
Understanding why we made these choices will help you understand how to work effectively with the data layer.
Why Two Databases?
The decision to use both PostgreSQL and ValKey wasn't made lightly. It emerged from the practical realities of building a system that needs to be both highly available and capable of complex analysis.
PostgreSQL for Deep Analysis: When security teams need to correlate vulnerabilities across thousands of hosts, find patterns in software installations, or generate compliance reports, they need the power of SQL and the reliability of ACID transactions.
ValKey for Real-Time Operations: When an agent needs to load templates, or the UI needs to show current agent status, they need microsecond response times and don't want to wait for complex database queries to complete.
PostgreSQL: The Foundation of Trust
Host Table with JSONB Extensions
Our PostgreSQL schema balances structure with flexibility:
-- Core host table with structured data
CREATE TABLE hosts (
id SERIAL PRIMARY KEY,
hostname VARCHAR(255) NOT NULL,
ip_address INET,
platform VARCHAR(50),
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW(),
-- JSONB fields for flexible agent data
software_inventory JSONB,
system_fingerprint JSONB,
agent_metadata JSONB
);
-- Indexes for efficient JSONB queries
CREATE INDEX idx_hosts_packages_gin ON hosts USING GIN ((software_inventory->'packages'));
CREATE INDEX idx_hosts_platform ON hosts USING GIN ((system_fingerprint->'platform'));
CREATE INDEX idx_hosts_agent_gin ON hosts USING GIN (agent_metadata);
The Philosophy: Use structured columns for data that has consistent meaning across all hosts (hostname, IP address) and JSONB for data that varies by platform or agent capability.
JSONB: The Best of Both Worlds
JSONB gives us the flexibility of document storage with the power of relational queries:
-- Find all hosts with vulnerable Apache versions
SELECT hostname, software_inventory->'packages'
FROM hosts
WHERE software_inventory->'packages' @> '[{"name": "apache2"}]'
AND software_inventory @@ '$.packages[*] ? (@.name == "apache2" && @.version like_regex "^2\\.4\\.(4[0-2])")';
-- Find hosts missing critical security updates
SELECT hostname, (software_inventory->'scan_metadata'->>'scan_date')::timestamp
FROM hosts
WHERE (software_inventory->'scan_metadata'->>'scan_date')::timestamp < (NOW() - INTERVAL '7 days');
ValKey: The Speed of Thought
Key Naming Strategy: A Language for Data
Our ValKey keys follow patterns that make the data self-documenting:
# Template Storage
agent:template:{template_id} # Template YAML content
agent:template:meta:{template_id} # Template metadata
agent:template:manifest # Global template index
# Agent Management
agent:state:{agent_id} # Agent connection state
agent:heartbeat:{agent_id} # Agent health (with TTL)
# Scan Coordination
scan:queue:pending # Pending scan requests
scan:result:{scan_id} # Cached scan results
Why This Matters: When you're debugging at 2 AM, these patterns let you quickly find the data you need without consulting documentation.
🚀 Developer Handbook
The Practical Guide to Daily Development
This handbook distills years of experience into practical guidance for common scenarios. It's not just about knowing what commands to run—it's about understanding the patterns that make you productive and the pitfalls that waste time.
Debugging Workflows
The "Template Not Appearing" Mystery
This is probably the most common issue developers encounter. Here's the detective story:
Symptoms: You created a template in the UI, but it's not showing up in the agent template list.
The Investigation Process:
# Step 1: Did the message make it through RabbitMQ?
docker exec sirius-engine rabbitmqctl list_queues name messages
# Step 2: Did the Agent Server process it?
docker logs sirius-engine | grep "template.*processing"
# Step 3: Did it make it to ValKey?
docker exec sirius-engine valkey-cli keys "agent:template:*"
# Step 4: Are there validation errors?
docker logs sirius-engine | grep "template.*validation.*failed"
Common Culprits and Solutions:
- YAML Syntax Errors: Use
yamllintor an online YAML validator - Template Validation Failures: Check the Agent Server logs for specific validation errors
- RabbitMQ Connection Issues: Verify RabbitMQ is running and accessible
Frontend Development Best Practices
Using the Utility System
Our frontend utility organization follows strict patterns that promote maintainability and code reuse. Here's how to work effectively with the utility system:
Import Patterns:
// ✅ DO: Import specific utilities you need
import {
convertToYamlForEditing,
convertYamlToJson,
} from "~/utils/yamlConverter";
import { VALIDATION_MESSAGES, DEFAULT_VALUES } from "~/utils/constants";
import { TemplateContent, ScriptContent } from "~/utils/types";
// ❌ DON'T: Import entire utility modules unless needed
import * as yamlConverter from "~/utils/yamlConverter"; // Avoid this
Type Safety Best Practices:
// ✅ DO: Use proper TypeScript interfaces
const handleTemplateSubmit = (template: TemplateContent) => {
// TypeScript knows the exact structure
const yamlContent = convertToYamlForEditing(template);
// ...
};
// ❌ DON'T: Use 'any' types in new code
const handleTemplateSubmit = (template: any) => {
// Avoid this
// ...
};
Constants Usage:
// ✅ DO: Use centralized constants
alert(VALIDATION_MESSAGES.requiredFields);
const editorHeight = MONACO_HEIGHTS.editor;
// ❌ DON'T: Use magic strings/numbers
alert("Please fill in all required fields"); // Avoid this
const editorHeight = "500px"; // Avoid this
YAML Conversion Patterns:
// ✅ DO: Use utility functions for conversion
const yamlContent = convertToYamlForEditing(templateData);
const jsonData = convertYamlToJson(userInput);
// ❌ DON'T: Duplicate conversion logic
const yamlContent = JSON.stringify(templateData, null, 2); // Avoid this
Adding New Utilities
When adding new utility functions, follow these patterns:
1. Choose the Right Utility File:
yamlConverter.ts- YAML/JSON conversion functionstypes.ts- TypeScript interfaces and type definitionsconstants.ts- Configuration values, messages, defaultsmonacoUtils.ts- Monaco editor specific utilitiesscriptTemplates.ts- Script template generation
2. Follow Function Patterns:
// ✅ DO: Pure functions with clear interfaces
export const formatContent = (content: string): string => {
if (!content) return "";
return content.replace(/\\n/g, "\n").replace(/\\t/g, "\t");
};
// ✅ DO: Use proper error handling
export const parseTemplate = (yaml: string): TemplateContent => {
try {
return parseYamlLikeContent(yaml);
} catch (error) {
console.error("Failed to parse template:", error);
throw new Error("Invalid template format");
}
};
3. Export Patterns:
// ✅ DO: Use named exports for discoverability
export const convertToYamlForEditing = (contentData: unknown): string => {
// ...
};
export const VALIDATION_MESSAGES = {
// ...
} as const;
// ❌ DON'T: Use default exports for utilities
export default function convert() {} // Avoid this
Performance Optimization
Template Execution Performance
Templates that take too long to execute can slow down entire scans. Here's how to optimize:
// Add performance monitoring to template execution
func (te *TemplateEngine) ExecuteTemplate(ctx context.Context, template *Template) (*DetectionResult, error) {
startTime := time.Now()
defer func() {
executionTime := time.Since(startTime)
// Flag slow templates for optimization
if executionTime > 5*time.Second {
log.Warn("Slow template execution",
"template_id", template.ID,
"execution_time", executionTime)
}
// Track metrics for monitoring
templateExecutionDuration.WithLabelValues(template.ID).Observe(executionTime.Seconds())
}()
// ... template execution logic ...
}
Security Best Practices
Template Security Validation
Templates are powerful, which means they can be dangerous if not properly validated:
func (tv *TemplateValidator) ValidateSecurityConstraints(template *Template) error {
// Prevent directory traversal attacks
for _, file := range template.Detection.Files {
if strings.Contains(file.Path, "..") || strings.Contains(file.Path, "~") {
return fmt.Errorf("invalid file path: %s", file.Path)
}
// Block access to sensitive system files
sensitiveFiles := []string{"/etc/shadow", "/etc/passwd", "/proc/", "/sys/"}
for _, sensitive := range sensitiveFiles {
if strings.HasPrefix(file.Path, sensitive) {
return fmt.Errorf("access to sensitive file not allowed: %s", file.Path)
}
}
}
// Prevent ReDoS attacks through regex validation
for _, file := range template.Detection.Files {
for _, pattern := range file.Patterns {
if len(pattern.Regex) > 1000 {
return fmt.Errorf("regex pattern too long: %s", pattern.Regex)
}
// Check for dangerous patterns
if strings.Contains(pattern.Regex, "(.*)*") || strings.Contains(pattern.Regex, "(.+)+") {
return fmt.Errorf("potentially dangerous regex pattern: %s", pattern.Regex)
}
}
}
return nil
}
Deployment and Operations
Health Check Configuration
Comprehensive health checks are essential for production operations:
func (s *Server) HealthCheck(w http.ResponseWriter, r *http.Request) {
health := &HealthStatus{
Status: "healthy",
Timestamp: time.Now(),
Checks: make(map[string]CheckResult),
}
// Check all critical dependencies
if err := s.valKeyClient.Ping(context.Background()).Err(); err != nil {
health.Checks["valkey"] = CheckResult{Status: "unhealthy", Error: err.Error()}
health.Status = "unhealthy"
} else {
health.Checks["valkey"] = CheckResult{Status: "healthy"}
}
// Set appropriate HTTP status code
if health.Status == "unhealthy" {
w.WriteHeader(http.StatusServiceUnavailable)
}
json.NewEncoder(w).Encode(health)
}
The Journey Continues
This documentation represents not just technical knowledge, but the collective experience of building and operating a complex security platform. As you work with Sirius, you'll develop your own insights and best practices.
Remember: every production issue is a learning opportunity, every performance bottleneck teaches you something about the system, and every feature request helps you understand how security teams really work.
The goal isn't to memorize every detail in this guide—it's to understand the principles and patterns that will help you solve problems you haven't encountered yet.
Document Status: ✅ Complete Comprehensive Developer Guide (v2.1)
Target Audience: New developers joining the Sirius Agent project
Next Steps: Use this guide for developer onboarding, team training, and as a reference for daily development work
Recent Improvements (v2.1):
- ✅ Utility Organization: Extracted shared functions into dedicated utility modules
- ✅ Type Safety: Eliminated
anytypes and improved TypeScript interfaces - ✅ Code Quality: Removed debug logging and standardized constants
- ✅ Maintainability: Reduced code duplication and improved organization
- ✅ Developer Experience: Enhanced development workflows and best practices
Final Word: Building distributed security systems is challenging, but it's also incredibly rewarding. You're not just writing code—you're building tools that help protect organizations from real threats. The complexity exists for a reason, and understanding that reason will make you a more effective developer and a better security engineer.
With the recent v2.1 improvements, the codebase is now more maintainable, type-safe, and developer-friendly than ever. These improvements will accelerate development velocity and reduce the learning curve for new team members.