Files
wehub-resource-sync 161ef94b4f
Check engine pin consistency / Dockerfile / CI pin consistency (push) Successful in 8s
Sirius CI/CD Pipeline / Detect Changes (push) Successful in 23s
Validate Docker Configuration / Validate Docker Compose Configuration (push) Successful in 47s
Sirius CI/CD Pipeline / Build API (${{ matrix.platform }}) (push) Has been cancelled
Sirius CI/CD Pipeline / Build UI (${{ matrix.platform }}) (push) Has been cancelled
Sirius CI/CD Pipeline / Merge Engine Manifest (push) Has been cancelled
Sirius CI/CD Pipeline / Merge API Manifest (push) Has been cancelled
Sirius CI/CD Pipeline / Merge UI Manifest (push) Has been cancelled
Sirius CI/CD Pipeline / Build Engine (${{ matrix.platform }}) (push) Has been cancelled
Sirius CI/CD Pipeline / Build Infra (${{ matrix.service }}, ${{ matrix.platform }}) (push) Has been cancelled
Sirius CI/CD Pipeline / Merge Infra Manifest (sirius-postgres) (push) Has been cancelled
Sirius CI/CD Pipeline / Merge Infra Manifest (sirius-rabbitmq) (push) Has been cancelled
Sirius CI/CD Pipeline / Merge Infra Manifest (sirius-valkey) (push) Has been cancelled
Sirius CI/CD Pipeline / Integration Test (push) Has been cancelled
Sirius CI/CD Pipeline / Public Stack Contract (push) Has been cancelled
Sirius CI/CD Pipeline / Dispatch Demo Deployment (sirius-demo branch) (push) Has been cancelled
Sirius CI/CD Pipeline / Dispatch Demo Canary (main branch) (push) Has been cancelled
Sirius CI/CD Pipeline / Guard Registry Namespace (push) Has been cancelled
chore: import upstream snapshot with attribution
2026-07-13 12:32:25 +08:00
..

title, description, template, version, last_updated, author, tags, categories, difficulty, prerequisites, related_docs, dependencies, llm_context, search_keywords
title description template version last_updated author tags categories difficulty prerequisites related_docs dependencies llm_context search_keywords
Sirius Scanner - Vulnerability Scanning Engine Comprehensive documentation for the Sirius vulnerability scanning engine TEMPLATE.documentation-standard 1.0.0 2025-10-25 Sirius Team
scanner
vulnerability
nmap
nse
go
rabbitmq
backend
security
scanning
advanced
Go 1.23+
Docker
RabbitMQ
Nmap
documentation/dev/architecture/README.architecture.md
documentation/dev/README.development.md
../minor-projects/app-scanner
../minor-projects/go-api
../minor-projects/sirius-nse
high
scanner
vulnerability
nmap
nse
rabbitmq
valkey
scanning-engine

Sirius Scanner - Vulnerability Scanning Engine

Overview

Sirius Scanner is a sophisticated, modular vulnerability scanning engine that orchestrates multiple security scanning tools (Nmap, RustScan, Naabu) through a message-driven architecture. Built in Go, it processes scan requests from RabbitMQ, executes multi-phase security assessments, and enriches vulnerability data with NVD information.

Key Features:

  • Message-Driven Architecture: RabbitMQ-based scan request processing
  • Multi-Phase Scanning: Enumeration → Discovery → Vulnerability assessment
  • Concurrent Processing: Worker pool pattern for parallel target scanning
  • NSE Script Management: Git-based synchronization with curated script repository
  • Template System: Pre-configured and custom scan templates
  • Source Attribution: Comprehensive tracking of scan origins and configurations
  • Real-Time Updates: ValKey integration for live scan progress monitoring

Repository Location: ../minor-projects/app-scanner


Table of Contents

  1. Core Architecture
  2. Scanning Strategies
  3. NSE Script Management
  4. Template System
  5. Scan Message Format
  6. Target Processing
  7. Source Attribution System
  8. State Management
  9. Docker Integration
  10. Configuration Files
  11. Development Workflow
  12. Key Files Reference
  13. ValKey Schema Reference
  14. Profile vs Template System
  15. RabbitMQ Message Schema

Core Architecture

Message-Driven Design

The scanner operates as a RabbitMQ consumer, listening on the scan queue for incoming scan requests. This design enables:

  • Asynchronous Processing: Scans don't block the API
  • Load Distribution: Multiple scanner instances can consume from the same queue
  • Fault Tolerance: Failed scans can be retried without data loss
  • Priority Handling: Scan requests include priority levels (1-5)

Entry Point:

// main.go
func main() {
    scanManager := scan.NewScanManager(kvStore, toolFactory, scanUpdater)
    scanManager.ListenForScans() // Blocks, listening for RabbitMQ messages
    select {} // Keep service running
}

Multi-Phase Scanning Workflow

Scans proceed through up to three phases based on scan_types configuration:

┌─────────────────┐
│  Enumeration    │  Naabu: Fast port enumeration (SYN scan)
│  (Optional)     │  Output: List of open ports
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Discovery      │  RustScan: Rapid host/port discovery
│  (Optional)     │  Output: Live hosts with open ports
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Vulnerability  │  Nmap + NSE: Deep vulnerability scanning
│  (Required)     │  Output: CVEs, service versions, OS detection
└─────────────────┘

Phase Selection:

{
  "options": {
    "scan_types": ["enumeration", "discovery", "vulnerability"]
  }
}

Worker Pool Pattern

The scanner uses a concurrent worker pool (default: 10 workers) to process multiple targets simultaneously:

// internal/scan/worker_pool.go
type WorkerPool struct {
    workerCount int
    taskQueue   chan ScanTask
    manager     *ScanManager
}

// Each worker runs in its own goroutine
func (wp *WorkerPool) worker(ctx context.Context, id int) {
    for {
        select {
        case task := <-wp.taskQueue:
            wp.manager.scanIP(task.IP) // Execute scan
        case <-ctx.Done():
            return
        }
    }
}

Benefits:

  • Parallel Execution: Scan multiple IPs concurrently
  • Resource Control: Limit concurrent scans to prevent system overload
  • Graceful Shutdown: Context-based cancellation

Strategy Pattern

The scanner implements the Strategy Pattern for pluggable scan tools:

// internal/scan/strategies.go
type ScanStrategy interface {
    Execute(target string) (sirius.Host, error)
}

// Implementations:
type NaabuStrategy struct { ... }      // Port enumeration
type RustScanStrategy struct { ... }   // Discovery
type NmapStrategy struct { ... }       // Vulnerability scanning

Advantages:

  • Extensibility: Add new scanning tools without modifying core logic
  • Testability: Mock strategies for unit testing
  • Flexibility: Swap implementations based on requirements

Factory Pattern

The ScanToolFactory dynamically creates appropriate strategies:

// internal/scan/factory.go
func (f *ScanToolFactory) CreateTool(toolType string) ScanStrategy {
    switch toolType {
    case "enumeration":
        return &NaabuStrategy{...}
    case "discovery":
        return &RustScanStrategy{}
    case "vulnerability":
        return &NmapStrategy{...}
    }
}

Scanning Strategies

NaabuStrategy (Port Enumeration)

Purpose: Fast, accurate port enumeration using ProjectDiscovery's Naabu.

Technology: SYN-based port scanning (requires root/CAP_NET_RAW)

Configuration:

type NaabuStrategy struct {
    Ports   string // Port range (e.g., "1-65535", "80,443,8080")
    Retries int    // Number of retry attempts
}

Implementation Details:

  • Uses github.com/projectdiscovery/naabu/v2 library
  • Default timeout: 5 seconds per host
  • Returns error ErrHostDown if no open ports found
  • Outputs sirius.Host with populated Ports array

Example Usage:

strategy := &NaabuStrategy{
    Ports:   "1-10000",
    Retries: 3,
}
host, err := strategy.Execute("192.168.1.100")
// host.Ports: [{ID: 80, Protocol: "tcp", State: "open"}, ...]

When to Use:

  • Need comprehensive port enumeration
  • Scanning large port ranges (1-65535)
  • Require retry logic for unstable networks
  • Want detailed port state information

RustScanStrategy (Discovery)

Purpose: Ultra-fast port discovery for identifying live hosts.

Technology: Adaptive multi-threaded port scanning (RustScan binary)

Configuration:

type RustScanStrategy struct {
    // Uses default RustScan settings
    // Automatically adjusts thread count based on available resources
}

Implementation Details:

  • Executes rustscan binary via exec.Command
  • Parses output to extract open ports
  • Filters results: hosts with no open ports are skipped
  • Much faster than Nmap for initial discovery (5-20x speedup)

Example Usage:

strategy := &RustScanStrategy{}
host, err := strategy.Execute("192.168.1.100")
// host.Ports: [{ID: 22, ...}, {ID: 80, ...}, {ID: 443, ...}]

When to Use:

  • Scanning large IP ranges (entire subnets)
  • Need rapid feedback on live hosts
  • Initial reconnaissance phase
  • Time-sensitive assessments

NmapStrategy (Vulnerability Scanning)

Purpose: Deep vulnerability assessment with NSE script execution.

Technology: Nmap with NSE (Nmap Scripting Engine) for CVE detection

Configuration:

type NmapStrategy struct {
    Protocols  []string // Legacy: protocol-based selection (e.g., ["smb", "http"])
    ScriptList []string // Explicit script names (overrides Protocols)
}

Implementation Details:

  • Script Selection:

    • Template-based: Uses EnabledScripts from template
    • Protocol-based: Selects scripts matching protocols (smb, http, ssh, ftp, rdp)
    • Wildcard: ["*"] runs all available NSE scripts
  • Nmap Command Construction:

nmap -T4 -sV -Pn -p <ports> --script <scripts> --script-args-file <args.txt> <target> -oX -
  • CVE Extraction: Parses Nmap XML output for CVE patterns
  • Vulnerability Enrichment: Fetches CVE details from NVD API
  • Fallback Mechanism: If script errors occur, retries with minimal safe scripts

Example Usage:

// Template-based (recommended)
strategy := &NmapStrategy{
    ScriptList: []string{"vulners", "smb-vuln-ms17-010", "http-shellshock"},
}

// Protocol-based
strategy := &NmapStrategy{
    Protocols: []string{"smb", "http"},
}

host, err := strategy.Execute("192.168.1.100")
// host.Vulnerabilities: [{VID: "CVE-2017-0143", RiskScore: 9.3, ...}, ...]

Script Processing:

  1. Execution: Nmap runs selected NSE scripts against target
  2. Parsing: Extract structured data from script output
  3. CVE Detection: Regex matching for CVE-YYYY-NNNNN patterns
  4. Enrichment: Query NVD API for descriptions, CVSS scores
  5. Deduplication: Remove duplicate CVEs from different scripts

Fallback Scan:

If NSE scripts fail (e.g., script syntax errors), scanner automatically retries with minimal safe scripts:

safeScripts := []string{"banner", "http-title", "ssl-cert"}

When to Use:

  • Deep security assessment required
  • Need CVE identification
  • Service version detection
  • OS fingerprinting
  • Protocol-specific vulnerability checks

NSE Script Management

The scanner maintains a curated collection of NSE scripts via the sirius-nse repository.

Repository Management (internal/nse/repo.go)

Purpose: Git-based synchronization of NSE scripts.

Repository URL: https://github.com/SiriusScan/sirius-nse.git

Local Path: /opt/sirius/nse/sirius-nse (in Docker container)

Operations:

type RepoManager struct {
    BasePath string // /opt/sirius/nse/sirius-nse
    RepoURL  string // GitHub repository
    gitOps   GitOperations
}

// Ensure repository exists and is up-to-date
func (rm *RepoManager) EnsureRepo() error {
    if !rm.isGitRepo() {
        // Clone on first run
        return rm.gitOps.Clone(rm.RepoURL, rm.BasePath)
    }
    // Update existing repository
    return rm.updateRepo()
}

// Update: fetch + reset to origin/main
func (rm *RepoManager) updateRepo() error {
    rm.gitOps.Fetch(rm.BasePath)
    rm.gitOps.Reset(rm.BasePath) // Hard reset to origin/main
    return nil
}

Sync Timing:

  • Startup: Syncs when ScanManager initializes
  • Before Scans: Automatic sync before first scan (via ListenForScans)
  • Manual: Can be triggered via management API (future)

Symlink Strategy:

Nmap's script directory is symlinked to sirius-nse:

# Dockerfile
RUN ln -sf /opt/sirius/nse/sirius-nse/scripts /usr/local/share/nmap/scripts

This ensures only curated scripts are available, preventing accidental execution of default Nmap scripts.

Script Selection (internal/nse/script_selector.go)

Purpose: Select appropriate NSE scripts based on protocols or explicit list.

Manifest Structure:

{
  "version": "1.0",
  "scripts": [
    {
      "name": "vulners",
      "category": "vuln",
      "protocols": ["*"]
    },
    {
      "name": "smb-vuln-ms17-010",
      "category": "vuln",
      "protocols": ["smb"]
    }
  ]
}

Script Selection Logic:

type ScriptSelector struct {
    manifest *Manifest
    blacklist map[string]bool
}

// Build Nmap --script flag
func (ss *ScriptSelector) BuildNmapScriptFlag(protocols ...string) (string, error) {
    if len(protocols) == 1 && protocols[0] == "*" {
        // Return all non-blacklisted scripts
        return ss.getAllScripts(), nil
    }

    // Filter by protocols
    scripts := []string{}
    for _, script := range ss.manifest.Scripts {
        if ss.matchesProtocol(script, protocols) && !ss.isBlacklisted(script.Name) {
            scripts = append(scripts, script.Name)
        }
    }

    return strings.Join(scripts, ","), nil
}

Protocol Matching:

  • * (wildcard): Script applies to all protocols
  • Exact match: ["smb"] matches scripts with protocols: ["smb"]
  • Multiple protocols: ["smb", "http"] matches scripts with either protocol

Example:

selector := nse.NewScriptSelector(manifest)
scriptFlag, _ := selector.BuildNmapScriptFlag("smb", "http")
// Result: "vulners,smb-vuln-ms17-010,smb-os-discovery,http-title,http-enum,..."

Script Blacklist (internal/nse/script_blacklist.go)

Purpose: Exclude problematic or slow scripts.

Blacklist Criteria:

  • False Positives: Scripts with high FP rates
  • Performance: Extremely slow scripts (>5 minutes per host)
  • Stability: Scripts that crash or hang frequently
  • Compatibility: Scripts incompatible with our environment

Example Blacklist:

var DefaultBlacklist = map[string]bool{
    "broadcast-dhcp-discover": true,  // Sends broadcasts
    "firewalk":                true,  // Very slow
    "http-slowloris-check":    true,  // DoS risk
}

Blacklist Management:

  • Centrally managed in script_blacklist.go
  • Can be overridden via environment variables (future)
  • Logged when scripts are excluded

Sync Manager (internal/nse/sync.go)

Purpose: Coordinate NSE repository updates with scan operations.

Responsibilities:

  1. Pre-Scan Sync: Ensure scripts are up-to-date before scanning
  2. Concurrency Control: Prevent multiple simultaneous syncs
  3. Error Handling: Log sync failures but don't block scans
  4. Context Awareness: Respect context cancellation

Implementation:

type SyncManager struct {
    repoManager *RepoManager
    kvStore     store.KVStore
    lastSync    time.Time
    syncMutex   sync.Mutex
}

func (sm *SyncManager) Sync(ctx context.Context) error {
    sm.syncMutex.Lock()
    defer sm.syncMutex.Unlock()

    // Skip if recently synced (within 1 hour)
    if time.Since(sm.lastSync) < time.Hour {
        return nil
    }

    // Perform sync
    if err := sm.repoManager.EnsureRepo(); err != nil {
        return fmt.Errorf("failed to sync NSE repo: %w", err)
    }

    sm.lastSync = time.Now()
    return nil
}

Sync Strategy:

  • Cooldown Period: 1 hour between syncs
  • Non-Blocking: Sync failures logged but don't prevent scans
  • Startup Sync: Always sync on scanner startup

Template System

Templates provide pre-configured scan profiles for common use cases.

Template Structure

type Template struct {
    ID             string          // Unique identifier (e.g., "high-risk")
    Name           string          // Human-readable name
    Description    string          // Usage description
    Type           TemplateType    // SystemTemplate or CustomTemplate
    EnabledScripts []string        // NSE scripts to run
    ScanOptions    TemplateOptions // Default scan options
    CreatedAt      time.Time
    UpdatedAt      time.Time
}

type TemplateOptions struct {
    ScanTypes    []string // ["enumeration", "discovery", "vulnerability"]
    PortRange    string   // "1-10000"
    Aggressive   bool     // Enable aggressive scanning
    MaxRetries   int      // Retry attempts
    Parallel     bool     // Parallel target scanning
    ExcludePorts []string // Ports to skip
}

System Templates

Pre-defined templates initialized on startup:

1. high-risk - Focused Critical Vulnerability Scan

Purpose: Balanced scan focusing on high-impact vulnerabilities.

Scripts: 10 carefully selected scripts

EnabledScripts: []string{
    "vulners",                // CVE detection (highest value)
    "smb-vuln-ms17-010",      // EternalBlue
    "http-shellshock",        // Shellshock vulnerability
    "http-vuln-cve2017-5638", // Apache Struts RCE
    "banner",                 // Service identification
    "http-title",             // HTTP identification
    "ssl-cert",               // SSL certificate info
    "http-enum",              // HTTP path enumeration
    "smb-os-discovery",       // SMB OS detection
    "ftp-anon",               // Anonymous FTP access
}

ScanOptions: {
    ScanTypes:  []string{"enumeration", "discovery", "vulnerability"},
    PortRange:  "1-10000",
    Aggressive: true,
    MaxRetries: 3,
    Parallel:   true,
}

Use Case: Default scan for most security assessments.

2. all - Comprehensive Scan

Purpose: Exhaustive scanning with all available NSE scripts.

Scripts: All non-blacklisted scripts (wildcard)

EnabledScripts: []string{"*"} // Special marker

ScanOptions: {
    ScanTypes:  []string{"enumeration", "discovery", "vulnerability"},
    PortRange:  "1-65535",
    Aggressive: true,
    MaxRetries: 3,
    Parallel:   false, // Sequential for thoroughness
}

Use Case: Deep penetration testing, compliance audits.

Warning: Can take hours per host. Use sparingly.

3. quick - Fast Scan

Purpose: Rapid assessment with essential scripts.

Scripts: 3 lightweight scripts

EnabledScripts: []string{
    "vulners",    // CVE detection
    "banner",     // Service identification
    "http-title", // HTTP identification
}

ScanOptions: {
    ScanTypes:  []string{"enumeration", "vulnerability"},
    PortRange:  "top500Ports", // Most common 500 ports
    Aggressive: false,
    MaxRetries: 2,
    Parallel:   true,
}

Use Case: Initial reconnaissance, time-sensitive scans.

Custom Templates

Users can create custom templates via UI (future) or API.

Creation Example:

template := &Template{
    ID:          "web-app-scan",
    Name:        "Web Application Scan",
    Description: "Focused scan for web application vulnerabilities",
    Type:        CustomTemplate,
    EnabledScripts: []string{
        "vulners",
        "http-enum",
        "http-shellshock",
        "http-sql-injection",
        "ssl-cert",
    },
    ScanOptions: TemplateOptions{
        ScanTypes:  []string{"discovery", "vulnerability"},
        PortRange:  "80,443,8080,8443",
        Aggressive: false,
        MaxRetries: 2,
        Parallel:   true,
    },
}

templateManager.CreateTemplate(ctx, template)

Template Operations:

// Get template
template, err := templateManager.GetTemplate(ctx, "high-risk")

// List all templates
templates, err := templateManager.ListTemplates(ctx)

// Update custom template (system templates are immutable)
templateManager.UpdateTemplate(ctx, template)

// Delete custom template
templateManager.DeleteTemplate(ctx, "my-template")

Template Resolution

When a scan message includes template_id, the scanner:

  1. Fetches Template: Retrieve from ValKey
  2. Applies Defaults: Use template's EnabledScripts and ScanOptions
  3. Merges User Options: User-provided options override template defaults
  4. Resolves Scripts: Convert template script list to Nmap --script flag

Resolution Logic:

func (sm *ScanManager) handleMessage(msg string) {
    var scanMsg ScanMessage
    json.Unmarshal([]byte(msg), &scanMsg)

    if scanMsg.Options.TemplateID != "" {
        template, _ := sm.templateManager.GetTemplate(ctx, scanMsg.Options.TemplateID)

        // User options override template defaults
        if scanMsg.Options.PortRange == "" {
            scanMsg.Options.PortRange = template.ScanOptions.PortRange
        }
        if len(scanMsg.Options.ScanTypes) == 0 {
            scanMsg.Options.ScanTypes = template.ScanOptions.ScanTypes
        }
        // ... merge other options
    }

    sm.processTarget(scanMsg.Targets[0])
}

Priority: User Options > Template Defaults > System Defaults


Scan Message Format

Scan requests are JSON messages sent to the scan RabbitMQ queue.

Message Structure

type ScanMessage struct {
    ID          string      // Unique scan identifier
    Targets     []Target    // Targets to scan
    Options     ScanOptions // Scan configuration
    Priority    int         // 1 (low) to 5 (high)
    CallbackURL string      // Optional webhook on completion
}

type Target struct {
    Value   string     // IP, range, CIDR, or hostname
    Type    TargetType // Target type identifier
    Timeout int        // Per-target timeout (seconds, optional)
}

type TargetType string
const (
    SingleIP    TargetType = "single_ip"    // 192.168.1.1
    IPRange     TargetType = "ip_range"     // 192.168.1.1-192.168.1.254
    CIDR        TargetType = "cidr"         // 192.168.1.0/24
    DNSName     TargetType = "dns_name"     // example.com
    DNSWildcard TargetType = "dns_wildcard" // *.example.com (TODO)
)

type ScanOptions struct {
    TemplateID   string   // Template to use (optional)
    PortRange    string   // "1-65535", "80,443,8080"
    Aggressive   bool     // Aggressive scanning mode
    ExcludePorts []string // Ports to skip
    ScanTypes    []string // ["enumeration", "discovery", "vulnerability"]
    MaxRetries   int      // Retry attempts
    Parallel     bool     // Parallel target scanning
}

Example Messages

Simple Single-IP Scan

{
  "id": "scan-001",
  "targets": [
    {
      "value": "192.168.1.100",
      "type": "single_ip"
    }
  ],
  "options": {
    "template_id": "high-risk",
    "scan_types": ["vulnerability"]
  },
  "priority": 3
}

CIDR Range with Custom Options

{
  "id": "scan-002",
  "targets": [
    {
      "value": "192.168.1.0/24",
      "type": "cidr"
    }
  ],
  "options": {
    "port_range": "1-10000",
    "aggressive": true,
    "scan_types": ["discovery", "vulnerability"],
    "max_retries": 3,
    "parallel": true
  },
  "priority": 4
}

Multi-Target Scan with Template

{
  "id": "scan-003",
  "targets": [
    {
      "value": "192.168.1.1",
      "type": "single_ip"
    },
    {
      "value": "192.168.1.100-192.168.1.110",
      "type": "ip_range"
    },
    {
      "value": "example.com",
      "type": "dns_name"
    }
  ],
  "options": {
    "template_id": "quick",
    "scan_types": ["enumeration", "vulnerability"]
  },
  "priority": 2,
  "callback_url": "https://api.example.com/scan-complete"
}

Validation Rules

Required Fields:

  • id: Must be unique (recommended: UUID)
  • targets: At least one target
  • options.scan_types: At least one scan type
  • priority: 1-5 (inclusive)

Optional Fields:

  • template_id: If omitted, uses system defaults
  • callback_url: If provided, webhook POST on completion

Validation Errors:

func (sm *ScanManager) validateScanMessage(msg *ScanMessage) error {
    if len(msg.Targets) == 0 {
        return fmt.Errorf("no targets specified")
    }
    if msg.Priority < 1 || msg.Priority > 5 {
        return fmt.Errorf("invalid priority: must be between 1 and 5")
    }
    return nil
}

Target Processing

Target Type Expansion

The scanner expands targets into individual IPs for worker pool processing.

Single IP

Format: 192.168.1.100

Processing: Direct pass-through

case SingleIP:
    if !validateIP(target.Value) {
        return nil, fmt.Errorf("invalid IP address: %s", target.Value)
    }
    return []string{target.Value}, nil

IP Range

Format: 192.168.1.1-192.168.1.254

Processing: Expand to all IPs in range

case IPRange:
    return expandIPRange(target.Value)
    // Returns: ["192.168.1.1", "192.168.1.2", ..., "192.168.1.254"]

Implementation:

func expandIPRange(rangeStr string) ([]string, error) {
    parts := strings.Split(rangeStr, "-")
    startIP := net.ParseIP(parts[0])
    endIP := net.ParseIP(parts[1])

    // Iterate from startIP to endIP
    ips := []string{}
    for ip := startIP; !ip.Equal(endIP); ip = incrementIP(ip) {
        ips = append(ips, ip.String())
    }
    return ips, nil
}

CIDR Block

Format: 192.168.1.0/24

Processing: Expand to all IPs in subnet (excluding network/broadcast)

case CIDR:
    if !validateCIDR(target.Value) {
        return nil, fmt.Errorf("invalid CIDR notation: %s", target.Value)
    }
    return expandCIDR(target.Value)
    // Returns: ["192.168.1.1", "192.168.1.2", ..., "192.168.1.254"]

Note: Skips network address (.0) and broadcast address (.255)

DNS Name

Format: example.com

Processing: DNS resolution to IP(s)

case DNSName:
    ips, err := net.LookupIP(target.Value)
    if err != nil {
        return nil, fmt.Errorf("DNS lookup failed: %v", err)
    }
    result := make([]string, len(ips))
    for i, ip := range ips {
        result[i] = ip.String()
    }
    return result, nil

Handles:

  • Multiple A/AAAA records (returns all IPs)
  • IPv4 and IPv6 addresses
  • DNS resolution failures (returns error)

DNS Wildcard (TODO)

Format: *.example.com

Status: Not yet implemented

Planned Behavior:

  1. Subdomain enumeration (via DNS brute-force or Certificate Transparency logs)
  2. Resolution of discovered subdomains to IPs
  3. Deduplication of IPs

Worker Pool Processing

After expansion, each IP is enqueued as a task:

func (sm *ScanManager) processTarget(target Target) {
    targetIPs, err := sm.prepareTarget(target)
    if err != nil {
        log.Printf("Failed to prepare target: %v", err)
        return
    }

    // Add each IP to worker pool queue
    for _, ip := range targetIPs {
        task := ScanTask{
            IP:      ip,
            Options: sm.currentScanOptions,
        }
        sm.workerPool.AddTask(task)
    }
}

Worker Pool Flow:

Target Expansion
       │
       ▼
┌─────────────────┐
│  Task Queue     │
│  (Channel)      │
└────────┬────────┘
         │
         ├─────► Worker 1 ──► scanIP(192.168.1.1)
         ├─────► Worker 2 ──► scanIP(192.168.1.2)
         ├─────► Worker 3 ──► scanIP(192.168.1.3)
         │  ...
         └─────► Worker 10 ─► scanIP(192.168.1.10)

Source Attribution System

The scanner implements comprehensive source attribution to track scan origins and configurations.

Purpose

  • Audit Trail: Who/what/when/how for each scan
  • Debugging: Troubleshoot scanning issues with full context
  • Compliance: Demonstrate due diligence in security assessments
  • Analytics: Understand scanning patterns and effectiveness

Source Metadata

type ScanSource struct {
    Name    string // Tool name (e.g., "nmap", "rustscan", "naabu")
    Version string // Tool version (e.g., "7.94")
    Config  string // Scan configuration (semicolon-separated key:value pairs)
}

Configuration Tracking

The scanner captures:

Scan Configuration:

  • ports: Port range scanned
  • aggressive: Aggressive mode enabled
  • types: Scan types executed
  • exclude: Excluded ports
  • template: Template ID used

System Information:

  • host: Hostname of scanning system
  • user: User running scanner
  • scanner_id: Unique scan identifier
  • go_version: Go runtime version

Example Config String:

ports:1-10000;aggressive:true;types:enumeration,discovery,vulnerability;template:high-risk;host:scanner-prod-01;user:siriususer;scanner_id:scan-abc123;go_version:go1.23.0

Version Detection

Automatically detects scanning tool versions:

func (sm *ScanManager) detectScannerVersion(toolName string) string {
    switch toolName {
    case "nmap":
        output, _ := exec.Command("nmap", "--version").Output()
        // Parse: "Nmap version 7.94 ( https://nmap.org )"
        return extractVersion(output)
    case "rustscan":
        output, _ := exec.Command("rustscan", "--version").Output()
        // Parse: "rustscan 2.1.1"
        return extractVersion(output)
    case "naabu":
        output, _ := exec.Command("naabu", "-version").Output()
        return extractVersion(output)
    }
}

API Submission

Results are submitted via source-aware API endpoint:

func (sm *ScanManager) submitHostWithSource(host sirius.Host, toolName string) error {
    source := sm.createScanSource(toolName)

    request := SourcedHostRequest{
        Host:   host,
        Source: source,
    }

    jsonData, _ := json.Marshal(request)
    url := fmt.Sprintf("%s/host/with-source", sm.apiBaseURL)
    resp, _ := http.Post(url, "application/json", bytes.NewBuffer(jsonData))

    return nil
}

API Endpoint: POST /host/with-source

Request Body:

{
  "host": {
    "ip": "192.168.1.100",
    "ports": [...],
    "vulnerabilities": [...]
  },
  "source": {
    "name": "nmap",
    "version": "7.94",
    "config": "ports:1-10000;aggressive:true;template:high-risk;host:scanner-01;..."
  }
}

Database Storage

The API stores source attribution in PostgreSQL:

Schema:

CREATE TABLE scan_sources (
    id SERIAL PRIMARY KEY,
    name VARCHAR(255),
    version VARCHAR(50),
    config TEXT,
    created_at TIMESTAMP DEFAULT NOW()
);

CREATE TABLE hosts (
    id SERIAL PRIMARY KEY,
    ip VARCHAR(45),
    scan_source_id INTEGER REFERENCES scan_sources(id),
    ...
);

This enables queries like:

  • "Show all vulnerabilities found by Nmap 7.94"
  • "List scans using the 'high-risk' template"
  • "Find hosts scanned from scanner-prod-01"

State Management

The scanner maintains real-time scan state using two systems:

ValKey Integration

Purpose: Live scan progress tracking for UI display.

Key Structure:

scan:<scan_id> = {
  "status": "running",
  "start_time": "2025-10-25T10:30:00Z",
  "end_time": null,
  "hosts": ["192.168.1.1", "192.168.1.2", ...],
  "hosts_completed": 45,
  "vulnerabilities": [
    {
      "id": "CVE-2017-0143",
      "severity": "critical",
      "title": "EternalBlue SMB RCE",
      "description": "Remote code execution..."
    }
  ]
}

Update Pattern:

type ScanUpdater struct {
    kvStore store.KVStore
}

func (su *ScanUpdater) Update(ctx context.Context, updateFn func(*ScanResult) error) error {
    // 1. Get current scan state from ValKey
    current, _ := su.kvStore.GetValue(ctx, "scan:"+scanID)

    // 2. Apply update function
    var scan ScanResult
    json.Unmarshal([]byte(current.Message.Value), &scan)
    updateFn(&scan)

    // 3. Write updated state back to ValKey
    updated, _ := json.Marshal(scan)
    su.kvStore.SetValue(ctx, "scan:"+scanID, string(updated))

    return nil
}

Update Events:

  • Discovery Complete: Add host to hosts array
  • Host Complete: Increment hosts_completed
  • Vulnerability Found: Add to vulnerabilities array
  • Scan Complete: Set status = "completed", set end_time

Example Update:

sm.scanUpdater.Update(ctx, func(scan *ScanResult) error {
    scan.HostsCompleted++
    if scan.HostsCompleted >= len(scan.Hosts) {
        scan.Status = "completed"
        scan.EndTime = time.Now().Format(time.RFC3339)
    }
    return nil
})

Logging System

Purpose: Structured audit trail for compliance and debugging.

Implementation: RabbitMQ-based logging to scanner_logs queue.

Log Client:

type LoggingClient struct {
    queueName string // "scanner_logs"
}

func (lc *LoggingClient) LogScanEvent(scanID, eventType, message string, metadata map[string]interface{}) {
    logEntry := map[string]interface{}{
        "timestamp":  time.Now().Format(time.RFC3339),
        "scan_id":    scanID,
        "event_type": eventType,
        "message":    message,
        "metadata":   metadata,
    }

    jsonData, _ := json.Marshal(logEntry)
    queue.Publish("scanner_logs", string(jsonData))
}

Event Types:

Event Type Description Metadata
scan_initiated Scan request received targets_count, priority, template_id
target_prepared Target expanded to IPs target_value, target_type, ips_generated
host_discovered Live host found host_ip, ports, tool
tool_execution Scanning tool completed tool, duration, success, ports_found
vulnerability_found CVE detected host_ip, cve_id, severity, tool
host_completed All phases done for host host_ip, vulnerabilities_found
scan_completed Entire scan finished total_hosts, vulnerabilities_found, duration
scan_error Error occurred host_ip, error_type, error_message

Example Log Entries:

// scan_initiated
{
  "timestamp": "2025-10-25T10:30:00Z",
  "scan_id": "scan-abc123",
  "event_type": "scan_initiated",
  "message": "Scan request received",
  "metadata": {
    "targets_count": 3,
    "priority": 4,
    "template_id": "high-risk"
  }
}

// vulnerability_found
{
  "timestamp": "2025-10-25T10:35:22Z",
  "scan_id": "scan-abc123",
  "event_type": "vulnerability_found",
  "message": "CVE detected",
  "metadata": {
    "host_ip": "192.168.1.100",
    "cve_id": "CVE-2017-0143",
    "severity": "critical",
    "tool": "nmap"
  }
}

Log Consumption:

  • Elasticsearch: Indexing for search/analytics
  • File: JSON lines for long-term storage
  • Monitoring: Real-time alerting on errors

Docker Integration

The scanner runs inside the sirius-engine container, which bundles multiple applications.

Dockerfile Architecture

Multi-Stage Build:

# Stage 1: Builder
FROM golang:1.23-bullseye AS builder

# Install build dependencies
RUN apt-get update && apt-get install -y \
    git ca-certificates build-essential libpcap-dev

# Clone and build app-scanner
RUN git clone https://github.com/SiriusScan/app-scanner.git && \
    cd app-scanner && \
    CGO_ENABLED=1 GOOS=linux go build -ldflags="-w -s" -o scanner main.go

# Stage 2: Runtime
FROM debian:bullseye-slim

# Install runtime dependencies
RUN apt-get update && apt-get install -y \
    nmap \           # Nmap binary
    rustscan \       # RustScan binary
    libpcap0.8 \     # Packet capture library
    git              # For NSE repo cloning

# Copy built scanner binary
COPY --from=builder /repos/app-scanner/scanner /app-scanner-src/scanner

# Create NSE directory
RUN mkdir -p /opt/sirius/nse/sirius-nse

# Symlink Nmap scripts to sirius-nse
RUN ln -sf /opt/sirius/nse/sirius-nse/scripts /usr/local/share/nmap/scripts

Volume Mounts

Development Mode:

# docker-compose.dev.yaml
services:
  sirius-engine:
    volumes:
      - ../minor-projects/app-scanner:/app-scanner # Live code reload
      - ../minor-projects/go-api:/go-api
      - ../minor-projects/sirius-nse:/sirius-nse

Production Mode:

No volume mounts. Uses pre-compiled binary at /app-scanner-src/scanner.

Startup Script

The scanner is started by start-enhanced.sh:

#!/bin/bash

# Development mode: run from source with live reload
if [ "$GO_ENV" = "development" ] && [ -d "/app-scanner" ]; then
    echo "Starting scanner in development mode..."
    cd /app-scanner
    air -c .air.toml &  # Live reload with air
fi

# Production mode: run pre-compiled binary
if [ ! "$GO_ENV" = "development" ]; then
    echo "Starting scanner in production mode..."
    /app-scanner-src/scanner &
fi

# Keep container running
wait

NSE Script Installation

Automatic on Startup:

The SyncManager automatically clones/updates the sirius-nse repository when ListenForScans() is called:

func (sm *ScanManager) ListenForScans() {
    // Sync NSE scripts before listening
    if err := sm.nseSync.Sync(sm.ctx); err != nil {
        log.Printf("Warning: failed to sync NSE scripts: %v", err)
        // Continue anyway - may use cached scripts
    }

    queue.Listen("scan", sm.handleMessage)
}

Fallback: If sync fails, uses existing scripts from /opt/sirius/nse/sirius-nse.

Environment Variables

# Scanner configuration
SIRIUS_API_URL=http://sirius-api:9001  # API endpoint for results
RABBITMQ_URL=amqp://guest:guest@sirius-rabbitmq:5672/
VALKEY_HOST=sirius-valkey
VALKEY_PORT=6379

# Development settings
GO_ENV=development  # Enable dev mode

Configuration Files

manifest.json

Location: /app-scanner/manifest.json

Purpose: Reference to external repositories (sirius-nse).

Structure:

{
  "repositories": [
    {
      "name": "sirius-nse",
      "url": "https://github.com/SiriusScan/sirius-nse.git"
    }
  ]
}

Usage: Read by installer scripts or future auto-update mechanisms.

nmap-args/args.txt

Location: /app-scanner/nmap-args/args.txt (or /opt/sirius/nse/sirius-nse/scripts/args.txt)

Purpose: Default arguments for NSE scripts.

Example Content:

# Vulners script configuration
vulners.showall=false

# HTTP scripts
http.useragent=Sirius Scanner/1.0

# SMB scripts
smbdomain=WORKGROUP
smbusername=guest
smbpassword=

# Timing
timeout=30000

Usage: Passed to Nmap via --script-args-file flag.

Path Resolution:

argsFilePaths := []string{
    "/opt/sirius/nse/sirius-nse/scripts/args.txt",
    "/app-scanner/nmap-args/args.txt",
    "/app-scanner-src/nmap-args/args.txt",
    "nmap-args/args.txt",
}

// Use first existing file
for _, path := range argsFilePaths {
    if _, err := os.Stat(path); err == nil {
        argsFilePath = path
        break
    }
}

.air.toml

Location: /app-scanner/.air.toml

Purpose: Live reload configuration for development.

Key Settings:

[build]
  cmd = "go build -o ./tmp/scanner main.go"
  bin = "./tmp/scanner"
  include_ext = ["go", "tpl", "tmpl", "html"]
  exclude_dir = ["tmp", "vendor", "testdata"]
  delay = 1000  # 1 second delay before rebuild

[log]
  time = true

Usage: Automatically watches for file changes and rebuilds/restarts scanner.


Development Workflow

Running Tests

Unit Tests:

cd /app-scanner
go test ./internal/scan/... -v

# Specific test
go test ./internal/scan/ -run TestScanToolFactory -v

# With coverage
go test ./internal/scan/... -cover -coverprofile=coverage.out
go tool cover -html=coverage.out

Integration Tests:

# Full scan test (requires running infrastructure)
go run cmd/scan-full-test/main.go

Manual Scan Testing

Test Commands:

# Direct Nmap test
cd cmd/direct-nmap-test
go run main.go

# NSE script test
cd cmd/nse-test
go run main.go

# Validate NSE script fixes
cd cmd/validate-nse-fix
go run main.go

Debugging NSE Scripts

Check Script Availability:

docker exec sirius-engine nmap --script-help vulners

Test Script Manually:

docker exec sirius-engine nmap -sV --script vulners 192.168.1.100

View NSE Repository:

docker exec sirius-engine ls -la /opt/sirius/nse/sirius-nse/scripts/

Check Symlink:

docker exec sirius-engine ls -la /usr/local/share/nmap/scripts
# Should point to: /opt/sirius/nse/sirius-nse/scripts

Live Reload Development

Development Mode:

  1. Edit code in ../minor-projects/app-scanner
  2. Air detects changes and rebuilds
  3. Scanner restarts automatically
  4. Logs visible in docker logs -f sirius-engine

Watch Logs:

# Scanner logs
docker logs -f sirius-engine | grep scanner

# RabbitMQ messages
docker exec sirius-rabbitmq rabbitmqctl list_queues name messages

# ValKey scan state
docker exec sirius-valkey valkey-cli GET scan:scan-abc123

Adding New Scan Strategies

Step 1: Implement Strategy Interface

// internal/scan/strategies.go
type MyNewStrategy struct {
    CustomOption string
}

func (s *MyNewStrategy) Execute(target string) (sirius.Host, error) {
    // Your scanning logic here
    return host, nil
}

Step 2: Update Factory

// internal/scan/factory.go
func (f *ScanToolFactory) CreateTool(toolType string) ScanStrategy {
    switch toolType {
    case "my-new-tool":
        return &MyNewStrategy{
            CustomOption: f.currentOptions.SomeOption,
        }
    // ... existing cases
    }
}

Step 3: Add to Scan Types

Update documentation and UI to include new scan type in scan_types options.

Step 4: Write Tests

// internal/scan/strategies_test.go
func TestMyNewStrategy(t *testing.T) {
    strategy := &MyNewStrategy{CustomOption: "test"}
    host, err := strategy.Execute("192.168.1.100")
    assert.NoError(t, err)
    assert.NotEmpty(t, host.IP)
}

Creating System Templates

Step 1: Define Template

// internal/scan/template_manager.go
func (tm *TemplateManager) InitializeSystemTemplates(ctx context.Context) error {
    templates := []Template{
        // ... existing templates
        {
            ID:          "my-template",
            Name:        "My Custom Template",
            Description: "Description of what this template does",
            Type:        SystemTemplate,
            EnabledScripts: []string{
                "script1",
                "script2",
            },
            ScanOptions: TemplateOptions{
                ScanTypes:  []string{"vulnerability"},
                PortRange:  "1-1000",
                Aggressive: false,
            },
        },
    }
    // ... create templates
}

Step 2: Rebuild Scanner

cd /app-scanner
go build -o scanner main.go

Step 3: Test Template

{
  "id": "test-scan",
  "targets": [{ "value": "192.168.1.100", "type": "single_ip" }],
  "options": { "template_id": "my-template" },
  "priority": 3
}

Optimizing Scan Performance

Strategies:

  1. Adjust Worker Pool Size:
// internal/scan/manager.go
const DEFAULT_WORKERS = 20  // Increase for more parallelism
  1. Reduce Port Range:
{ "port_range": "1-1000" } // Instead of 1-65535
  1. Limit NSE Scripts:
{ "template_id": "quick" } // Use quick template
  1. Enable Parallel Processing:
{ "parallel": true }
  1. Adjust Nmap Timing:
// modules/nmap/nmap.go
args := []string{
    "-T5",  // Insane timing (faster but less accurate)
    // ...
}

Key Files Reference

File Lines Purpose
main.go 37 Application entry point, initializes ScanManager
internal/scan/manager.go 783 Core scan orchestration, RabbitMQ listener, worker pool
internal/scan/strategies.go 164 ScanStrategy interface and implementations (Naabu, RustScan, Nmap)
internal/scan/factory.go 50 Strategy factory for creating scan tools
internal/scan/worker_pool.go ~150 Concurrent worker pool for parallel scanning
internal/scan/template_manager.go 413 Template CRUD operations, system template initialization
internal/scan/template_types.go ~100 Template data structures
internal/scan/updater.go ~150 ValKey state update management
internal/scan/logging.go ~200 Structured logging to RabbitMQ
internal/scan/helpers.go ~100 Utility functions (severity calculation, etc.)
internal/scan/network_helpers.go ~150 IP/CIDR expansion, validation
internal/nse/repo.go 138 Git repository management for NSE scripts
internal/nse/sync.go ~150 NSE sync coordination
internal/nse/script_selector.go ~200 Protocol-based NSE script selection
internal/nse/script_blacklist.go ~50 Problematic script exclusion
internal/nse/types.go ~100 NSE data structures (Manifest, Script)
modules/nmap/nmap.go 533 Nmap integration, XML parsing, CVE extraction
modules/rustscan/rustscan.go ~150 RustScan integration
modules/naabu/naabu.go ~150 Naabu integration

Total: ~3,800 lines of application code


Code Examples

Sending a Scan Request

From Go:

scanMsg := ScanMessage{
    ID: uuid.New().String(),
    Targets: []Target{
        {Value: "192.168.1.0/24", Type: CIDR},
    },
    Options: ScanOptions{
        TemplateID: "high-risk",
        ScanTypes:  []string{"discovery", "vulnerability"},
    },
    Priority: 4,
}

msgBytes, _ := json.Marshal(scanMsg)
queue.Publish("scan", string(msgBytes))

From CLI (RabbitMQ):

rabbitmqadmin publish exchange=amq.default \
  routing_key=scan \
  payload='{"id":"scan-123","targets":[{"value":"192.168.1.100","type":"single_ip"}],"options":{"template_id":"quick"},"priority":3}'

Monitoring Scan Progress

ValKey Query:

docker exec sirius-valkey valkey-cli GET scan:scan-123

Output:

{
  "status": "running",
  "start_time": "2025-10-25T10:30:00Z",
  "hosts": ["192.168.1.1", "192.168.1.2", "192.168.1.100"],
  "hosts_completed": 2,
  "vulnerabilities": [
    {
      "id": "CVE-2017-0143",
      "severity": "critical",
      "title": "EternalBlue SMB RCE"
    }
  ]
}

Troubleshooting

NSE Script Errors

Symptom: "NSE: failed to initialize the script engine"

Causes:

  • Script syntax errors
  • Missing script dependencies
  • Blacklisted scripts referenced

Solutions:

  1. Check script exists:
docker exec sirius-engine ls /opt/sirius/nse/sirius-nse/scripts/ | grep vulners
  1. Test script manually:
docker exec sirius-engine nmap --script-help vulners
  1. Check blacklist:

Review internal/nse/script_blacklist.go

  1. Force repository sync:
docker exec sirius-engine rm -rf /opt/sirius/nse/sirius-nse
docker restart sirius-engine  # Will re-clone on startup

Scan Timeouts

Symptom: Scans never complete, hosts stuck in "running" status

Causes:

  • Network unreachability
  • Firewall blocking scans
  • Extremely large port ranges

Solutions:

  1. Reduce port range:
{ "port_range": "1-1000" } // Instead of 1-65535
  1. Enable aggressive mode:
{ "aggressive": true } // Faster but more detectable
  1. Check network connectivity:
docker exec sirius-engine ping 192.168.1.100
  1. View scanner logs:
docker logs -f sirius-engine | grep "ERROR\|Failed"

Memory Issues

Symptom: Scanner crashes with OOM (Out of Memory) errors

Causes:

  • Too many concurrent workers
  • Large target ranges without pagination
  • Memory leaks in scanning tools

Solutions:

  1. Reduce worker pool size:
const DEFAULT_WORKERS = 5  // Down from 10
  1. Scan smaller ranges:

Split large CIDR blocks into smaller chunks.

  1. Increase Docker memory:
# docker-compose.yaml
services:
  sirius-engine:
    deploy:
      resources:
        limits:
          memory: 4G # Increase from 2G
  1. Monitor memory usage:
docker stats sirius-engine

RabbitMQ Disconnections

Symptom: "Failed to consume from queue: connection closed"

Causes:

  • RabbitMQ container crashed
  • Network partition
  • Connection timeout

Solutions:

  1. Check RabbitMQ status:
docker exec sirius-rabbitmq rabbitmqctl status
  1. Restart RabbitMQ:
docker restart sirius-rabbitmq
  1. Check connection string:
echo $RABBITMQ_URL
# Should be: amqp://guest:guest@sirius-rabbitmq:5672/
  1. Implement reconnection logic (future):

Add automatic reconnection with exponential backoff.


Performance Benchmarks

Environment: 4-core CPU, 8GB RAM, 1Gbps network

Scan Type Target Range Duration Hosts/sec
Quick (top 500 ports) /24 subnet (254 hosts) 3-5 min ~50-80
High-Risk /24 subnet 15-25 min ~10-15
All Scripts /24 subnet 2-4 hours ~1-2
Quick Single host 10-20 sec N/A
High-Risk Single host 2-5 min N/A

Optimization Tips:

  • Parallelism: Increase worker count for faster subnet scans
  • Port Range: Limit to relevant ports (80, 443, 22, 3389)
  • Script Selection: Use templates instead of wildcard *
  • Network: Faster networks = faster scans (avoid VPNs)

Security Considerations

Responsible Scanning

Always:

  • Get written authorization before scanning
  • Use reasonable rate limiting (avoid DoS)
  • Scan only approved IP ranges
  • Document all scanning activities

Never:

  • Scan production systems without approval
  • Use aggressive mode on fragile systems
  • Store vulnerability data in insecure locations
  • Share scan results with unauthorized parties

Rate Limiting

The scanner implements implicit rate limiting through:

  • Worker Pool Size: Limits concurrent scans
  • Nmap Timing: -T4 (balanced) by default, not -T5 (insane)
  • Connection Limits: Naabu respects system limits

Future Enhancements:

  • Configurable rate limits (scans per second)
  • Per-target rate limiting
  • Adaptive throttling based on network conditions

Future Enhancements

Planned Features:

  1. DNS Wildcard Support: Subdomain enumeration via CT logs
  2. Distributed Scanning: Multiple scanner instances coordinating via ValKey
  3. Scan Scheduling: Cron-based recurring scans
  4. Advanced Reporting: PDF/HTML reports with executive summaries
  5. Machine Learning: Anomaly detection for unusual scan results
  6. API Gateway Integration: Direct scan triggering via REST API
  7. WebSocket Updates: Real-time scan progress in UI
  8. Scan Comparison: Diff scans to identify new vulnerabilities

ValKey Schema Reference

ValKey (Redis-compatible) is used for live scan state. Key structure:

  • currentScan Main key storing the live scan result as JSON. Contains the full ScanResult object with hosts, vulnerabilities, sub-scans, and progress.
  • Encoding: JSON may be stored as a base64-encoded string in some contexts; decode with b64Decode from ~/utils/std.ts when reading from the frontend.
  • Polling: The frontend polls this key at regular intervals (5-second refetch) to display live progress.
  • TTL: No TTL is set; the key persists until overwritten by the next scan.

Profile vs Template System

  • Scan Profiles: High-level presets that control overall scan behavior (e.g., "quick-scan", "full-scan", "agent-only"). Stored as ScanProfile type. They determine which scan methods are enabled and how they are configured.
  • Agent Templates: Specific detection templates used by the agent scanner (e.g., checks for a specific CVE). Stored in the template system. They define detection steps such as file_content, file_hash, file_search, and version_cmd. The authoritative set is KnownDetectionTypes in app-agent/internal/modules/detection_types.go (drift-checked against the runtime registry by detection_types_registry_test.go).
  • Profiles may reference templates but are not the same thing: profiles define what to run and how; templates define the concrete detection logic.

RabbitMQ Message Schema

  • Queue: Scan queue (consumed by app-scanner).
  • Message format: JSON with target list and scan options.
  • Scan request contents: Includes targets, scan_types configuration, and agent_scan configuration.
  • Results: app-scanner writes results back to the ValKey currentScan key.

LLM Context

This documentation is optimized for AI assistant context. Key information:

Architecture: Message-driven (RabbitMQ), worker pool concurrency, strategy pattern for extensibility

Core Flow: Receive scan message → Expand targets → Distribute to workers → Execute strategies → Enrich results → Submit to API

Key Components: ScanManager (orchestration), ScanStrategies (tool integrations), TemplateManager (scan configs), NSE system (script management)

Integration Points: RabbitMQ (input), ValKey (state), PostgreSQL (results), NVD API (CVE enrichment)

Development: Go 1.23+, runs in sirius-engine container, live reload with Air, unit/integration tests

Common Tasks: Add strategies (implement interface + factory), create templates (InitializeSystemTemplates), debug NSE (check symlinks/manifest)


Last Updated: 2025-10-25
Version: 1.0.0
Maintainer: Sirius Team
Repository: https://github.com/SiriusScan/app-scanner