title, description, template, version, last_updated, author, tags, categories, difficulty, prerequisites, related_docs, dependencies, llm_context, search_keywords
| title | description | template | version | last_updated | author | tags | categories | difficulty | prerequisites | related_docs | dependencies | llm_context | search_keywords | |||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Sirius Scanner - Vulnerability Scanning Engine | Comprehensive documentation for the Sirius vulnerability scanning engine | TEMPLATE.documentation-standard | 1.0.0 | 2025-10-25 | Sirius Team |
|
|
advanced |
|
|
|
high |
|
Sirius Scanner - Vulnerability Scanning Engine
Overview
Sirius Scanner is a sophisticated, modular vulnerability scanning engine that orchestrates multiple security scanning tools (Nmap, RustScan, Naabu) through a message-driven architecture. Built in Go, it processes scan requests from RabbitMQ, executes multi-phase security assessments, and enriches vulnerability data with NVD information.
Key Features:
- Message-Driven Architecture: RabbitMQ-based scan request processing
- Multi-Phase Scanning: Enumeration → Discovery → Vulnerability assessment
- Concurrent Processing: Worker pool pattern for parallel target scanning
- NSE Script Management: Git-based synchronization with curated script repository
- Template System: Pre-configured and custom scan templates
- Source Attribution: Comprehensive tracking of scan origins and configurations
- Real-Time Updates: ValKey integration for live scan progress monitoring
Repository Location: ../minor-projects/app-scanner
Table of Contents
- Core Architecture
- Scanning Strategies
- NSE Script Management
- Template System
- Scan Message Format
- Target Processing
- Source Attribution System
- State Management
- Docker Integration
- Configuration Files
- Development Workflow
- Key Files Reference
- ValKey Schema Reference
- Profile vs Template System
- RabbitMQ Message Schema
Core Architecture
Message-Driven Design
The scanner operates as a RabbitMQ consumer, listening on the scan queue for incoming scan requests. This design enables:
- Asynchronous Processing: Scans don't block the API
- Load Distribution: Multiple scanner instances can consume from the same queue
- Fault Tolerance: Failed scans can be retried without data loss
- Priority Handling: Scan requests include priority levels (1-5)
Entry Point:
// main.go
func main() {
scanManager := scan.NewScanManager(kvStore, toolFactory, scanUpdater)
scanManager.ListenForScans() // Blocks, listening for RabbitMQ messages
select {} // Keep service running
}
Multi-Phase Scanning Workflow
Scans proceed through up to three phases based on scan_types configuration:
┌─────────────────┐
│ Enumeration │ Naabu: Fast port enumeration (SYN scan)
│ (Optional) │ Output: List of open ports
└────────┬────────┘
│
▼
┌─────────────────┐
│ Discovery │ RustScan: Rapid host/port discovery
│ (Optional) │ Output: Live hosts with open ports
└────────┬────────┘
│
▼
┌─────────────────┐
│ Vulnerability │ Nmap + NSE: Deep vulnerability scanning
│ (Required) │ Output: CVEs, service versions, OS detection
└─────────────────┘
Phase Selection:
{
"options": {
"scan_types": ["enumeration", "discovery", "vulnerability"]
}
}
Worker Pool Pattern
The scanner uses a concurrent worker pool (default: 10 workers) to process multiple targets simultaneously:
// internal/scan/worker_pool.go
type WorkerPool struct {
workerCount int
taskQueue chan ScanTask
manager *ScanManager
}
// Each worker runs in its own goroutine
func (wp *WorkerPool) worker(ctx context.Context, id int) {
for {
select {
case task := <-wp.taskQueue:
wp.manager.scanIP(task.IP) // Execute scan
case <-ctx.Done():
return
}
}
}
Benefits:
- Parallel Execution: Scan multiple IPs concurrently
- Resource Control: Limit concurrent scans to prevent system overload
- Graceful Shutdown: Context-based cancellation
Strategy Pattern
The scanner implements the Strategy Pattern for pluggable scan tools:
// internal/scan/strategies.go
type ScanStrategy interface {
Execute(target string) (sirius.Host, error)
}
// Implementations:
type NaabuStrategy struct { ... } // Port enumeration
type RustScanStrategy struct { ... } // Discovery
type NmapStrategy struct { ... } // Vulnerability scanning
Advantages:
- Extensibility: Add new scanning tools without modifying core logic
- Testability: Mock strategies for unit testing
- Flexibility: Swap implementations based on requirements
Factory Pattern
The ScanToolFactory dynamically creates appropriate strategies:
// internal/scan/factory.go
func (f *ScanToolFactory) CreateTool(toolType string) ScanStrategy {
switch toolType {
case "enumeration":
return &NaabuStrategy{...}
case "discovery":
return &RustScanStrategy{}
case "vulnerability":
return &NmapStrategy{...}
}
}
Scanning Strategies
NaabuStrategy (Port Enumeration)
Purpose: Fast, accurate port enumeration using ProjectDiscovery's Naabu.
Technology: SYN-based port scanning (requires root/CAP_NET_RAW)
Configuration:
type NaabuStrategy struct {
Ports string // Port range (e.g., "1-65535", "80,443,8080")
Retries int // Number of retry attempts
}
Implementation Details:
- Uses
github.com/projectdiscovery/naabu/v2library - Default timeout: 5 seconds per host
- Returns error
ErrHostDownif no open ports found - Outputs
sirius.Hostwith populatedPortsarray
Example Usage:
strategy := &NaabuStrategy{
Ports: "1-10000",
Retries: 3,
}
host, err := strategy.Execute("192.168.1.100")
// host.Ports: [{ID: 80, Protocol: "tcp", State: "open"}, ...]
When to Use:
- Need comprehensive port enumeration
- Scanning large port ranges (1-65535)
- Require retry logic for unstable networks
- Want detailed port state information
RustScanStrategy (Discovery)
Purpose: Ultra-fast port discovery for identifying live hosts.
Technology: Adaptive multi-threaded port scanning (RustScan binary)
Configuration:
type RustScanStrategy struct {
// Uses default RustScan settings
// Automatically adjusts thread count based on available resources
}
Implementation Details:
- Executes
rustscanbinary viaexec.Command - Parses output to extract open ports
- Filters results: hosts with no open ports are skipped
- Much faster than Nmap for initial discovery (5-20x speedup)
Example Usage:
strategy := &RustScanStrategy{}
host, err := strategy.Execute("192.168.1.100")
// host.Ports: [{ID: 22, ...}, {ID: 80, ...}, {ID: 443, ...}]
When to Use:
- Scanning large IP ranges (entire subnets)
- Need rapid feedback on live hosts
- Initial reconnaissance phase
- Time-sensitive assessments
NmapStrategy (Vulnerability Scanning)
Purpose: Deep vulnerability assessment with NSE script execution.
Technology: Nmap with NSE (Nmap Scripting Engine) for CVE detection
Configuration:
type NmapStrategy struct {
Protocols []string // Legacy: protocol-based selection (e.g., ["smb", "http"])
ScriptList []string // Explicit script names (overrides Protocols)
}
Implementation Details:
-
Script Selection:
- Template-based: Uses
EnabledScriptsfrom template - Protocol-based: Selects scripts matching protocols (smb, http, ssh, ftp, rdp)
- Wildcard:
["*"]runs all available NSE scripts
- Template-based: Uses
-
Nmap Command Construction:
nmap -T4 -sV -Pn -p <ports> --script <scripts> --script-args-file <args.txt> <target> -oX -
- CVE Extraction: Parses Nmap XML output for CVE patterns
- Vulnerability Enrichment: Fetches CVE details from NVD API
- Fallback Mechanism: If script errors occur, retries with minimal safe scripts
Example Usage:
// Template-based (recommended)
strategy := &NmapStrategy{
ScriptList: []string{"vulners", "smb-vuln-ms17-010", "http-shellshock"},
}
// Protocol-based
strategy := &NmapStrategy{
Protocols: []string{"smb", "http"},
}
host, err := strategy.Execute("192.168.1.100")
// host.Vulnerabilities: [{VID: "CVE-2017-0143", RiskScore: 9.3, ...}, ...]
Script Processing:
- Execution: Nmap runs selected NSE scripts against target
- Parsing: Extract structured data from script output
- CVE Detection: Regex matching for
CVE-YYYY-NNNNNpatterns - Enrichment: Query NVD API for descriptions, CVSS scores
- Deduplication: Remove duplicate CVEs from different scripts
Fallback Scan:
If NSE scripts fail (e.g., script syntax errors), scanner automatically retries with minimal safe scripts:
safeScripts := []string{"banner", "http-title", "ssl-cert"}
When to Use:
- Deep security assessment required
- Need CVE identification
- Service version detection
- OS fingerprinting
- Protocol-specific vulnerability checks
NSE Script Management
The scanner maintains a curated collection of NSE scripts via the sirius-nse repository.
Repository Management (internal/nse/repo.go)
Purpose: Git-based synchronization of NSE scripts.
Repository URL: https://github.com/SiriusScan/sirius-nse.git
Local Path: /opt/sirius/nse/sirius-nse (in Docker container)
Operations:
type RepoManager struct {
BasePath string // /opt/sirius/nse/sirius-nse
RepoURL string // GitHub repository
gitOps GitOperations
}
// Ensure repository exists and is up-to-date
func (rm *RepoManager) EnsureRepo() error {
if !rm.isGitRepo() {
// Clone on first run
return rm.gitOps.Clone(rm.RepoURL, rm.BasePath)
}
// Update existing repository
return rm.updateRepo()
}
// Update: fetch + reset to origin/main
func (rm *RepoManager) updateRepo() error {
rm.gitOps.Fetch(rm.BasePath)
rm.gitOps.Reset(rm.BasePath) // Hard reset to origin/main
return nil
}
Sync Timing:
- Startup: Syncs when
ScanManagerinitializes - Before Scans: Automatic sync before first scan (via
ListenForScans) - Manual: Can be triggered via management API (future)
Symlink Strategy:
Nmap's script directory is symlinked to sirius-nse:
# Dockerfile
RUN ln -sf /opt/sirius/nse/sirius-nse/scripts /usr/local/share/nmap/scripts
This ensures only curated scripts are available, preventing accidental execution of default Nmap scripts.
Script Selection (internal/nse/script_selector.go)
Purpose: Select appropriate NSE scripts based on protocols or explicit list.
Manifest Structure:
{
"version": "1.0",
"scripts": [
{
"name": "vulners",
"category": "vuln",
"protocols": ["*"]
},
{
"name": "smb-vuln-ms17-010",
"category": "vuln",
"protocols": ["smb"]
}
]
}
Script Selection Logic:
type ScriptSelector struct {
manifest *Manifest
blacklist map[string]bool
}
// Build Nmap --script flag
func (ss *ScriptSelector) BuildNmapScriptFlag(protocols ...string) (string, error) {
if len(protocols) == 1 && protocols[0] == "*" {
// Return all non-blacklisted scripts
return ss.getAllScripts(), nil
}
// Filter by protocols
scripts := []string{}
for _, script := range ss.manifest.Scripts {
if ss.matchesProtocol(script, protocols) && !ss.isBlacklisted(script.Name) {
scripts = append(scripts, script.Name)
}
}
return strings.Join(scripts, ","), nil
}
Protocol Matching:
*(wildcard): Script applies to all protocols- Exact match:
["smb"]matches scripts withprotocols: ["smb"] - Multiple protocols:
["smb", "http"]matches scripts with either protocol
Example:
selector := nse.NewScriptSelector(manifest)
scriptFlag, _ := selector.BuildNmapScriptFlag("smb", "http")
// Result: "vulners,smb-vuln-ms17-010,smb-os-discovery,http-title,http-enum,..."
Script Blacklist (internal/nse/script_blacklist.go)
Purpose: Exclude problematic or slow scripts.
Blacklist Criteria:
- False Positives: Scripts with high FP rates
- Performance: Extremely slow scripts (>5 minutes per host)
- Stability: Scripts that crash or hang frequently
- Compatibility: Scripts incompatible with our environment
Example Blacklist:
var DefaultBlacklist = map[string]bool{
"broadcast-dhcp-discover": true, // Sends broadcasts
"firewalk": true, // Very slow
"http-slowloris-check": true, // DoS risk
}
Blacklist Management:
- Centrally managed in
script_blacklist.go - Can be overridden via environment variables (future)
- Logged when scripts are excluded
Sync Manager (internal/nse/sync.go)
Purpose: Coordinate NSE repository updates with scan operations.
Responsibilities:
- Pre-Scan Sync: Ensure scripts are up-to-date before scanning
- Concurrency Control: Prevent multiple simultaneous syncs
- Error Handling: Log sync failures but don't block scans
- Context Awareness: Respect context cancellation
Implementation:
type SyncManager struct {
repoManager *RepoManager
kvStore store.KVStore
lastSync time.Time
syncMutex sync.Mutex
}
func (sm *SyncManager) Sync(ctx context.Context) error {
sm.syncMutex.Lock()
defer sm.syncMutex.Unlock()
// Skip if recently synced (within 1 hour)
if time.Since(sm.lastSync) < time.Hour {
return nil
}
// Perform sync
if err := sm.repoManager.EnsureRepo(); err != nil {
return fmt.Errorf("failed to sync NSE repo: %w", err)
}
sm.lastSync = time.Now()
return nil
}
Sync Strategy:
- Cooldown Period: 1 hour between syncs
- Non-Blocking: Sync failures logged but don't prevent scans
- Startup Sync: Always sync on scanner startup
Template System
Templates provide pre-configured scan profiles for common use cases.
Template Structure
type Template struct {
ID string // Unique identifier (e.g., "high-risk")
Name string // Human-readable name
Description string // Usage description
Type TemplateType // SystemTemplate or CustomTemplate
EnabledScripts []string // NSE scripts to run
ScanOptions TemplateOptions // Default scan options
CreatedAt time.Time
UpdatedAt time.Time
}
type TemplateOptions struct {
ScanTypes []string // ["enumeration", "discovery", "vulnerability"]
PortRange string // "1-10000"
Aggressive bool // Enable aggressive scanning
MaxRetries int // Retry attempts
Parallel bool // Parallel target scanning
ExcludePorts []string // Ports to skip
}
System Templates
Pre-defined templates initialized on startup:
1. high-risk - Focused Critical Vulnerability Scan
Purpose: Balanced scan focusing on high-impact vulnerabilities.
Scripts: 10 carefully selected scripts
EnabledScripts: []string{
"vulners", // CVE detection (highest value)
"smb-vuln-ms17-010", // EternalBlue
"http-shellshock", // Shellshock vulnerability
"http-vuln-cve2017-5638", // Apache Struts RCE
"banner", // Service identification
"http-title", // HTTP identification
"ssl-cert", // SSL certificate info
"http-enum", // HTTP path enumeration
"smb-os-discovery", // SMB OS detection
"ftp-anon", // Anonymous FTP access
}
ScanOptions: {
ScanTypes: []string{"enumeration", "discovery", "vulnerability"},
PortRange: "1-10000",
Aggressive: true,
MaxRetries: 3,
Parallel: true,
}
Use Case: Default scan for most security assessments.
2. all - Comprehensive Scan
Purpose: Exhaustive scanning with all available NSE scripts.
Scripts: All non-blacklisted scripts (wildcard)
EnabledScripts: []string{"*"} // Special marker
ScanOptions: {
ScanTypes: []string{"enumeration", "discovery", "vulnerability"},
PortRange: "1-65535",
Aggressive: true,
MaxRetries: 3,
Parallel: false, // Sequential for thoroughness
}
Use Case: Deep penetration testing, compliance audits.
Warning: Can take hours per host. Use sparingly.
3. quick - Fast Scan
Purpose: Rapid assessment with essential scripts.
Scripts: 3 lightweight scripts
EnabledScripts: []string{
"vulners", // CVE detection
"banner", // Service identification
"http-title", // HTTP identification
}
ScanOptions: {
ScanTypes: []string{"enumeration", "vulnerability"},
PortRange: "top500Ports", // Most common 500 ports
Aggressive: false,
MaxRetries: 2,
Parallel: true,
}
Use Case: Initial reconnaissance, time-sensitive scans.
Custom Templates
Users can create custom templates via UI (future) or API.
Creation Example:
template := &Template{
ID: "web-app-scan",
Name: "Web Application Scan",
Description: "Focused scan for web application vulnerabilities",
Type: CustomTemplate,
EnabledScripts: []string{
"vulners",
"http-enum",
"http-shellshock",
"http-sql-injection",
"ssl-cert",
},
ScanOptions: TemplateOptions{
ScanTypes: []string{"discovery", "vulnerability"},
PortRange: "80,443,8080,8443",
Aggressive: false,
MaxRetries: 2,
Parallel: true,
},
}
templateManager.CreateTemplate(ctx, template)
Template Operations:
// Get template
template, err := templateManager.GetTemplate(ctx, "high-risk")
// List all templates
templates, err := templateManager.ListTemplates(ctx)
// Update custom template (system templates are immutable)
templateManager.UpdateTemplate(ctx, template)
// Delete custom template
templateManager.DeleteTemplate(ctx, "my-template")
Template Resolution
When a scan message includes template_id, the scanner:
- Fetches Template: Retrieve from ValKey
- Applies Defaults: Use template's
EnabledScriptsandScanOptions - Merges User Options: User-provided options override template defaults
- Resolves Scripts: Convert template script list to Nmap
--scriptflag
Resolution Logic:
func (sm *ScanManager) handleMessage(msg string) {
var scanMsg ScanMessage
json.Unmarshal([]byte(msg), &scanMsg)
if scanMsg.Options.TemplateID != "" {
template, _ := sm.templateManager.GetTemplate(ctx, scanMsg.Options.TemplateID)
// User options override template defaults
if scanMsg.Options.PortRange == "" {
scanMsg.Options.PortRange = template.ScanOptions.PortRange
}
if len(scanMsg.Options.ScanTypes) == 0 {
scanMsg.Options.ScanTypes = template.ScanOptions.ScanTypes
}
// ... merge other options
}
sm.processTarget(scanMsg.Targets[0])
}
Priority: User Options > Template Defaults > System Defaults
Scan Message Format
Scan requests are JSON messages sent to the scan RabbitMQ queue.
Message Structure
type ScanMessage struct {
ID string // Unique scan identifier
Targets []Target // Targets to scan
Options ScanOptions // Scan configuration
Priority int // 1 (low) to 5 (high)
CallbackURL string // Optional webhook on completion
}
type Target struct {
Value string // IP, range, CIDR, or hostname
Type TargetType // Target type identifier
Timeout int // Per-target timeout (seconds, optional)
}
type TargetType string
const (
SingleIP TargetType = "single_ip" // 192.168.1.1
IPRange TargetType = "ip_range" // 192.168.1.1-192.168.1.254
CIDR TargetType = "cidr" // 192.168.1.0/24
DNSName TargetType = "dns_name" // example.com
DNSWildcard TargetType = "dns_wildcard" // *.example.com (TODO)
)
type ScanOptions struct {
TemplateID string // Template to use (optional)
PortRange string // "1-65535", "80,443,8080"
Aggressive bool // Aggressive scanning mode
ExcludePorts []string // Ports to skip
ScanTypes []string // ["enumeration", "discovery", "vulnerability"]
MaxRetries int // Retry attempts
Parallel bool // Parallel target scanning
}
Example Messages
Simple Single-IP Scan
{
"id": "scan-001",
"targets": [
{
"value": "192.168.1.100",
"type": "single_ip"
}
],
"options": {
"template_id": "high-risk",
"scan_types": ["vulnerability"]
},
"priority": 3
}
CIDR Range with Custom Options
{
"id": "scan-002",
"targets": [
{
"value": "192.168.1.0/24",
"type": "cidr"
}
],
"options": {
"port_range": "1-10000",
"aggressive": true,
"scan_types": ["discovery", "vulnerability"],
"max_retries": 3,
"parallel": true
},
"priority": 4
}
Multi-Target Scan with Template
{
"id": "scan-003",
"targets": [
{
"value": "192.168.1.1",
"type": "single_ip"
},
{
"value": "192.168.1.100-192.168.1.110",
"type": "ip_range"
},
{
"value": "example.com",
"type": "dns_name"
}
],
"options": {
"template_id": "quick",
"scan_types": ["enumeration", "vulnerability"]
},
"priority": 2,
"callback_url": "https://api.example.com/scan-complete"
}
Validation Rules
Required Fields:
id: Must be unique (recommended: UUID)targets: At least one targetoptions.scan_types: At least one scan typepriority: 1-5 (inclusive)
Optional Fields:
template_id: If omitted, uses system defaultscallback_url: If provided, webhook POST on completion
Validation Errors:
func (sm *ScanManager) validateScanMessage(msg *ScanMessage) error {
if len(msg.Targets) == 0 {
return fmt.Errorf("no targets specified")
}
if msg.Priority < 1 || msg.Priority > 5 {
return fmt.Errorf("invalid priority: must be between 1 and 5")
}
return nil
}
Target Processing
Target Type Expansion
The scanner expands targets into individual IPs for worker pool processing.
Single IP
Format: 192.168.1.100
Processing: Direct pass-through
case SingleIP:
if !validateIP(target.Value) {
return nil, fmt.Errorf("invalid IP address: %s", target.Value)
}
return []string{target.Value}, nil
IP Range
Format: 192.168.1.1-192.168.1.254
Processing: Expand to all IPs in range
case IPRange:
return expandIPRange(target.Value)
// Returns: ["192.168.1.1", "192.168.1.2", ..., "192.168.1.254"]
Implementation:
func expandIPRange(rangeStr string) ([]string, error) {
parts := strings.Split(rangeStr, "-")
startIP := net.ParseIP(parts[0])
endIP := net.ParseIP(parts[1])
// Iterate from startIP to endIP
ips := []string{}
for ip := startIP; !ip.Equal(endIP); ip = incrementIP(ip) {
ips = append(ips, ip.String())
}
return ips, nil
}
CIDR Block
Format: 192.168.1.0/24
Processing: Expand to all IPs in subnet (excluding network/broadcast)
case CIDR:
if !validateCIDR(target.Value) {
return nil, fmt.Errorf("invalid CIDR notation: %s", target.Value)
}
return expandCIDR(target.Value)
// Returns: ["192.168.1.1", "192.168.1.2", ..., "192.168.1.254"]
Note: Skips network address (.0) and broadcast address (.255)
DNS Name
Format: example.com
Processing: DNS resolution to IP(s)
case DNSName:
ips, err := net.LookupIP(target.Value)
if err != nil {
return nil, fmt.Errorf("DNS lookup failed: %v", err)
}
result := make([]string, len(ips))
for i, ip := range ips {
result[i] = ip.String()
}
return result, nil
Handles:
- Multiple A/AAAA records (returns all IPs)
- IPv4 and IPv6 addresses
- DNS resolution failures (returns error)
DNS Wildcard (TODO)
Format: *.example.com
Status: Not yet implemented
Planned Behavior:
- Subdomain enumeration (via DNS brute-force or Certificate Transparency logs)
- Resolution of discovered subdomains to IPs
- Deduplication of IPs
Worker Pool Processing
After expansion, each IP is enqueued as a task:
func (sm *ScanManager) processTarget(target Target) {
targetIPs, err := sm.prepareTarget(target)
if err != nil {
log.Printf("Failed to prepare target: %v", err)
return
}
// Add each IP to worker pool queue
for _, ip := range targetIPs {
task := ScanTask{
IP: ip,
Options: sm.currentScanOptions,
}
sm.workerPool.AddTask(task)
}
}
Worker Pool Flow:
Target Expansion
│
▼
┌─────────────────┐
│ Task Queue │
│ (Channel) │
└────────┬────────┘
│
├─────► Worker 1 ──► scanIP(192.168.1.1)
├─────► Worker 2 ──► scanIP(192.168.1.2)
├─────► Worker 3 ──► scanIP(192.168.1.3)
│ ...
└─────► Worker 10 ─► scanIP(192.168.1.10)
Source Attribution System
The scanner implements comprehensive source attribution to track scan origins and configurations.
Purpose
- Audit Trail: Who/what/when/how for each scan
- Debugging: Troubleshoot scanning issues with full context
- Compliance: Demonstrate due diligence in security assessments
- Analytics: Understand scanning patterns and effectiveness
Source Metadata
type ScanSource struct {
Name string // Tool name (e.g., "nmap", "rustscan", "naabu")
Version string // Tool version (e.g., "7.94")
Config string // Scan configuration (semicolon-separated key:value pairs)
}
Configuration Tracking
The scanner captures:
Scan Configuration:
ports: Port range scannedaggressive: Aggressive mode enabledtypes: Scan types executedexclude: Excluded portstemplate: Template ID used
System Information:
host: Hostname of scanning systemuser: User running scannerscanner_id: Unique scan identifiergo_version: Go runtime version
Example Config String:
ports:1-10000;aggressive:true;types:enumeration,discovery,vulnerability;template:high-risk;host:scanner-prod-01;user:siriususer;scanner_id:scan-abc123;go_version:go1.23.0
Version Detection
Automatically detects scanning tool versions:
func (sm *ScanManager) detectScannerVersion(toolName string) string {
switch toolName {
case "nmap":
output, _ := exec.Command("nmap", "--version").Output()
// Parse: "Nmap version 7.94 ( https://nmap.org )"
return extractVersion(output)
case "rustscan":
output, _ := exec.Command("rustscan", "--version").Output()
// Parse: "rustscan 2.1.1"
return extractVersion(output)
case "naabu":
output, _ := exec.Command("naabu", "-version").Output()
return extractVersion(output)
}
}
API Submission
Results are submitted via source-aware API endpoint:
func (sm *ScanManager) submitHostWithSource(host sirius.Host, toolName string) error {
source := sm.createScanSource(toolName)
request := SourcedHostRequest{
Host: host,
Source: source,
}
jsonData, _ := json.Marshal(request)
url := fmt.Sprintf("%s/host/with-source", sm.apiBaseURL)
resp, _ := http.Post(url, "application/json", bytes.NewBuffer(jsonData))
return nil
}
API Endpoint: POST /host/with-source
Request Body:
{
"host": {
"ip": "192.168.1.100",
"ports": [...],
"vulnerabilities": [...]
},
"source": {
"name": "nmap",
"version": "7.94",
"config": "ports:1-10000;aggressive:true;template:high-risk;host:scanner-01;..."
}
}
Database Storage
The API stores source attribution in PostgreSQL:
Schema:
CREATE TABLE scan_sources (
id SERIAL PRIMARY KEY,
name VARCHAR(255),
version VARCHAR(50),
config TEXT,
created_at TIMESTAMP DEFAULT NOW()
);
CREATE TABLE hosts (
id SERIAL PRIMARY KEY,
ip VARCHAR(45),
scan_source_id INTEGER REFERENCES scan_sources(id),
...
);
This enables queries like:
- "Show all vulnerabilities found by Nmap 7.94"
- "List scans using the 'high-risk' template"
- "Find hosts scanned from scanner-prod-01"
State Management
The scanner maintains real-time scan state using two systems:
ValKey Integration
Purpose: Live scan progress tracking for UI display.
Key Structure:
scan:<scan_id> = {
"status": "running",
"start_time": "2025-10-25T10:30:00Z",
"end_time": null,
"hosts": ["192.168.1.1", "192.168.1.2", ...],
"hosts_completed": 45,
"vulnerabilities": [
{
"id": "CVE-2017-0143",
"severity": "critical",
"title": "EternalBlue SMB RCE",
"description": "Remote code execution..."
}
]
}
Update Pattern:
type ScanUpdater struct {
kvStore store.KVStore
}
func (su *ScanUpdater) Update(ctx context.Context, updateFn func(*ScanResult) error) error {
// 1. Get current scan state from ValKey
current, _ := su.kvStore.GetValue(ctx, "scan:"+scanID)
// 2. Apply update function
var scan ScanResult
json.Unmarshal([]byte(current.Message.Value), &scan)
updateFn(&scan)
// 3. Write updated state back to ValKey
updated, _ := json.Marshal(scan)
su.kvStore.SetValue(ctx, "scan:"+scanID, string(updated))
return nil
}
Update Events:
- Discovery Complete: Add host to
hostsarray - Host Complete: Increment
hosts_completed - Vulnerability Found: Add to
vulnerabilitiesarray - Scan Complete: Set
status = "completed", setend_time
Example Update:
sm.scanUpdater.Update(ctx, func(scan *ScanResult) error {
scan.HostsCompleted++
if scan.HostsCompleted >= len(scan.Hosts) {
scan.Status = "completed"
scan.EndTime = time.Now().Format(time.RFC3339)
}
return nil
})
Logging System
Purpose: Structured audit trail for compliance and debugging.
Implementation: RabbitMQ-based logging to scanner_logs queue.
Log Client:
type LoggingClient struct {
queueName string // "scanner_logs"
}
func (lc *LoggingClient) LogScanEvent(scanID, eventType, message string, metadata map[string]interface{}) {
logEntry := map[string]interface{}{
"timestamp": time.Now().Format(time.RFC3339),
"scan_id": scanID,
"event_type": eventType,
"message": message,
"metadata": metadata,
}
jsonData, _ := json.Marshal(logEntry)
queue.Publish("scanner_logs", string(jsonData))
}
Event Types:
| Event Type | Description | Metadata |
|---|---|---|
scan_initiated |
Scan request received | targets_count, priority, template_id |
target_prepared |
Target expanded to IPs | target_value, target_type, ips_generated |
host_discovered |
Live host found | host_ip, ports, tool |
tool_execution |
Scanning tool completed | tool, duration, success, ports_found |
vulnerability_found |
CVE detected | host_ip, cve_id, severity, tool |
host_completed |
All phases done for host | host_ip, vulnerabilities_found |
scan_completed |
Entire scan finished | total_hosts, vulnerabilities_found, duration |
scan_error |
Error occurred | host_ip, error_type, error_message |
Example Log Entries:
// scan_initiated
{
"timestamp": "2025-10-25T10:30:00Z",
"scan_id": "scan-abc123",
"event_type": "scan_initiated",
"message": "Scan request received",
"metadata": {
"targets_count": 3,
"priority": 4,
"template_id": "high-risk"
}
}
// vulnerability_found
{
"timestamp": "2025-10-25T10:35:22Z",
"scan_id": "scan-abc123",
"event_type": "vulnerability_found",
"message": "CVE detected",
"metadata": {
"host_ip": "192.168.1.100",
"cve_id": "CVE-2017-0143",
"severity": "critical",
"tool": "nmap"
}
}
Log Consumption:
- Elasticsearch: Indexing for search/analytics
- File: JSON lines for long-term storage
- Monitoring: Real-time alerting on errors
Docker Integration
The scanner runs inside the sirius-engine container, which bundles multiple applications.
Dockerfile Architecture
Multi-Stage Build:
# Stage 1: Builder
FROM golang:1.23-bullseye AS builder
# Install build dependencies
RUN apt-get update && apt-get install -y \
git ca-certificates build-essential libpcap-dev
# Clone and build app-scanner
RUN git clone https://github.com/SiriusScan/app-scanner.git && \
cd app-scanner && \
CGO_ENABLED=1 GOOS=linux go build -ldflags="-w -s" -o scanner main.go
# Stage 2: Runtime
FROM debian:bullseye-slim
# Install runtime dependencies
RUN apt-get update && apt-get install -y \
nmap \ # Nmap binary
rustscan \ # RustScan binary
libpcap0.8 \ # Packet capture library
git # For NSE repo cloning
# Copy built scanner binary
COPY --from=builder /repos/app-scanner/scanner /app-scanner-src/scanner
# Create NSE directory
RUN mkdir -p /opt/sirius/nse/sirius-nse
# Symlink Nmap scripts to sirius-nse
RUN ln -sf /opt/sirius/nse/sirius-nse/scripts /usr/local/share/nmap/scripts
Volume Mounts
Development Mode:
# docker-compose.dev.yaml
services:
sirius-engine:
volumes:
- ../minor-projects/app-scanner:/app-scanner # Live code reload
- ../minor-projects/go-api:/go-api
- ../minor-projects/sirius-nse:/sirius-nse
Production Mode:
No volume mounts. Uses pre-compiled binary at /app-scanner-src/scanner.
Startup Script
The scanner is started by start-enhanced.sh:
#!/bin/bash
# Development mode: run from source with live reload
if [ "$GO_ENV" = "development" ] && [ -d "/app-scanner" ]; then
echo "Starting scanner in development mode..."
cd /app-scanner
air -c .air.toml & # Live reload with air
fi
# Production mode: run pre-compiled binary
if [ ! "$GO_ENV" = "development" ]; then
echo "Starting scanner in production mode..."
/app-scanner-src/scanner &
fi
# Keep container running
wait
NSE Script Installation
Automatic on Startup:
The SyncManager automatically clones/updates the sirius-nse repository when ListenForScans() is called:
func (sm *ScanManager) ListenForScans() {
// Sync NSE scripts before listening
if err := sm.nseSync.Sync(sm.ctx); err != nil {
log.Printf("Warning: failed to sync NSE scripts: %v", err)
// Continue anyway - may use cached scripts
}
queue.Listen("scan", sm.handleMessage)
}
Fallback: If sync fails, uses existing scripts from /opt/sirius/nse/sirius-nse.
Environment Variables
# Scanner configuration
SIRIUS_API_URL=http://sirius-api:9001 # API endpoint for results
RABBITMQ_URL=amqp://guest:guest@sirius-rabbitmq:5672/
VALKEY_HOST=sirius-valkey
VALKEY_PORT=6379
# Development settings
GO_ENV=development # Enable dev mode
Configuration Files
manifest.json
Location: /app-scanner/manifest.json
Purpose: Reference to external repositories (sirius-nse).
Structure:
{
"repositories": [
{
"name": "sirius-nse",
"url": "https://github.com/SiriusScan/sirius-nse.git"
}
]
}
Usage: Read by installer scripts or future auto-update mechanisms.
nmap-args/args.txt
Location: /app-scanner/nmap-args/args.txt (or /opt/sirius/nse/sirius-nse/scripts/args.txt)
Purpose: Default arguments for NSE scripts.
Example Content:
# Vulners script configuration
vulners.showall=false
# HTTP scripts
http.useragent=Sirius Scanner/1.0
# SMB scripts
smbdomain=WORKGROUP
smbusername=guest
smbpassword=
# Timing
timeout=30000
Usage: Passed to Nmap via --script-args-file flag.
Path Resolution:
argsFilePaths := []string{
"/opt/sirius/nse/sirius-nse/scripts/args.txt",
"/app-scanner/nmap-args/args.txt",
"/app-scanner-src/nmap-args/args.txt",
"nmap-args/args.txt",
}
// Use first existing file
for _, path := range argsFilePaths {
if _, err := os.Stat(path); err == nil {
argsFilePath = path
break
}
}
.air.toml
Location: /app-scanner/.air.toml
Purpose: Live reload configuration for development.
Key Settings:
[build]
cmd = "go build -o ./tmp/scanner main.go"
bin = "./tmp/scanner"
include_ext = ["go", "tpl", "tmpl", "html"]
exclude_dir = ["tmp", "vendor", "testdata"]
delay = 1000 # 1 second delay before rebuild
[log]
time = true
Usage: Automatically watches for file changes and rebuilds/restarts scanner.
Development Workflow
Running Tests
Unit Tests:
cd /app-scanner
go test ./internal/scan/... -v
# Specific test
go test ./internal/scan/ -run TestScanToolFactory -v
# With coverage
go test ./internal/scan/... -cover -coverprofile=coverage.out
go tool cover -html=coverage.out
Integration Tests:
# Full scan test (requires running infrastructure)
go run cmd/scan-full-test/main.go
Manual Scan Testing
Test Commands:
# Direct Nmap test
cd cmd/direct-nmap-test
go run main.go
# NSE script test
cd cmd/nse-test
go run main.go
# Validate NSE script fixes
cd cmd/validate-nse-fix
go run main.go
Debugging NSE Scripts
Check Script Availability:
docker exec sirius-engine nmap --script-help vulners
Test Script Manually:
docker exec sirius-engine nmap -sV --script vulners 192.168.1.100
View NSE Repository:
docker exec sirius-engine ls -la /opt/sirius/nse/sirius-nse/scripts/
Check Symlink:
docker exec sirius-engine ls -la /usr/local/share/nmap/scripts
# Should point to: /opt/sirius/nse/sirius-nse/scripts
Live Reload Development
Development Mode:
- Edit code in
../minor-projects/app-scanner - Air detects changes and rebuilds
- Scanner restarts automatically
- Logs visible in
docker logs -f sirius-engine
Watch Logs:
# Scanner logs
docker logs -f sirius-engine | grep scanner
# RabbitMQ messages
docker exec sirius-rabbitmq rabbitmqctl list_queues name messages
# ValKey scan state
docker exec sirius-valkey valkey-cli GET scan:scan-abc123
Adding New Scan Strategies
Step 1: Implement Strategy Interface
// internal/scan/strategies.go
type MyNewStrategy struct {
CustomOption string
}
func (s *MyNewStrategy) Execute(target string) (sirius.Host, error) {
// Your scanning logic here
return host, nil
}
Step 2: Update Factory
// internal/scan/factory.go
func (f *ScanToolFactory) CreateTool(toolType string) ScanStrategy {
switch toolType {
case "my-new-tool":
return &MyNewStrategy{
CustomOption: f.currentOptions.SomeOption,
}
// ... existing cases
}
}
Step 3: Add to Scan Types
Update documentation and UI to include new scan type in scan_types options.
Step 4: Write Tests
// internal/scan/strategies_test.go
func TestMyNewStrategy(t *testing.T) {
strategy := &MyNewStrategy{CustomOption: "test"}
host, err := strategy.Execute("192.168.1.100")
assert.NoError(t, err)
assert.NotEmpty(t, host.IP)
}
Creating System Templates
Step 1: Define Template
// internal/scan/template_manager.go
func (tm *TemplateManager) InitializeSystemTemplates(ctx context.Context) error {
templates := []Template{
// ... existing templates
{
ID: "my-template",
Name: "My Custom Template",
Description: "Description of what this template does",
Type: SystemTemplate,
EnabledScripts: []string{
"script1",
"script2",
},
ScanOptions: TemplateOptions{
ScanTypes: []string{"vulnerability"},
PortRange: "1-1000",
Aggressive: false,
},
},
}
// ... create templates
}
Step 2: Rebuild Scanner
cd /app-scanner
go build -o scanner main.go
Step 3: Test Template
{
"id": "test-scan",
"targets": [{ "value": "192.168.1.100", "type": "single_ip" }],
"options": { "template_id": "my-template" },
"priority": 3
}
Optimizing Scan Performance
Strategies:
- Adjust Worker Pool Size:
// internal/scan/manager.go
const DEFAULT_WORKERS = 20 // Increase for more parallelism
- Reduce Port Range:
{ "port_range": "1-1000" } // Instead of 1-65535
- Limit NSE Scripts:
{ "template_id": "quick" } // Use quick template
- Enable Parallel Processing:
{ "parallel": true }
- Adjust Nmap Timing:
// modules/nmap/nmap.go
args := []string{
"-T5", // Insane timing (faster but less accurate)
// ...
}
Key Files Reference
| File | Lines | Purpose |
|---|---|---|
main.go |
37 | Application entry point, initializes ScanManager |
internal/scan/manager.go |
783 | Core scan orchestration, RabbitMQ listener, worker pool |
internal/scan/strategies.go |
164 | ScanStrategy interface and implementations (Naabu, RustScan, Nmap) |
internal/scan/factory.go |
50 | Strategy factory for creating scan tools |
internal/scan/worker_pool.go |
~150 | Concurrent worker pool for parallel scanning |
internal/scan/template_manager.go |
413 | Template CRUD operations, system template initialization |
internal/scan/template_types.go |
~100 | Template data structures |
internal/scan/updater.go |
~150 | ValKey state update management |
internal/scan/logging.go |
~200 | Structured logging to RabbitMQ |
internal/scan/helpers.go |
~100 | Utility functions (severity calculation, etc.) |
internal/scan/network_helpers.go |
~150 | IP/CIDR expansion, validation |
internal/nse/repo.go |
138 | Git repository management for NSE scripts |
internal/nse/sync.go |
~150 | NSE sync coordination |
internal/nse/script_selector.go |
~200 | Protocol-based NSE script selection |
internal/nse/script_blacklist.go |
~50 | Problematic script exclusion |
internal/nse/types.go |
~100 | NSE data structures (Manifest, Script) |
modules/nmap/nmap.go |
533 | Nmap integration, XML parsing, CVE extraction |
modules/rustscan/rustscan.go |
~150 | RustScan integration |
modules/naabu/naabu.go |
~150 | Naabu integration |
Total: ~3,800 lines of application code
Code Examples
Sending a Scan Request
From Go:
scanMsg := ScanMessage{
ID: uuid.New().String(),
Targets: []Target{
{Value: "192.168.1.0/24", Type: CIDR},
},
Options: ScanOptions{
TemplateID: "high-risk",
ScanTypes: []string{"discovery", "vulnerability"},
},
Priority: 4,
}
msgBytes, _ := json.Marshal(scanMsg)
queue.Publish("scan", string(msgBytes))
From CLI (RabbitMQ):
rabbitmqadmin publish exchange=amq.default \
routing_key=scan \
payload='{"id":"scan-123","targets":[{"value":"192.168.1.100","type":"single_ip"}],"options":{"template_id":"quick"},"priority":3}'
Monitoring Scan Progress
ValKey Query:
docker exec sirius-valkey valkey-cli GET scan:scan-123
Output:
{
"status": "running",
"start_time": "2025-10-25T10:30:00Z",
"hosts": ["192.168.1.1", "192.168.1.2", "192.168.1.100"],
"hosts_completed": 2,
"vulnerabilities": [
{
"id": "CVE-2017-0143",
"severity": "critical",
"title": "EternalBlue SMB RCE"
}
]
}
Troubleshooting
NSE Script Errors
Symptom: "NSE: failed to initialize the script engine"
Causes:
- Script syntax errors
- Missing script dependencies
- Blacklisted scripts referenced
Solutions:
- Check script exists:
docker exec sirius-engine ls /opt/sirius/nse/sirius-nse/scripts/ | grep vulners
- Test script manually:
docker exec sirius-engine nmap --script-help vulners
- Check blacklist:
Review internal/nse/script_blacklist.go
- Force repository sync:
docker exec sirius-engine rm -rf /opt/sirius/nse/sirius-nse
docker restart sirius-engine # Will re-clone on startup
Scan Timeouts
Symptom: Scans never complete, hosts stuck in "running" status
Causes:
- Network unreachability
- Firewall blocking scans
- Extremely large port ranges
Solutions:
- Reduce port range:
{ "port_range": "1-1000" } // Instead of 1-65535
- Enable aggressive mode:
{ "aggressive": true } // Faster but more detectable
- Check network connectivity:
docker exec sirius-engine ping 192.168.1.100
- View scanner logs:
docker logs -f sirius-engine | grep "ERROR\|Failed"
Memory Issues
Symptom: Scanner crashes with OOM (Out of Memory) errors
Causes:
- Too many concurrent workers
- Large target ranges without pagination
- Memory leaks in scanning tools
Solutions:
- Reduce worker pool size:
const DEFAULT_WORKERS = 5 // Down from 10
- Scan smaller ranges:
Split large CIDR blocks into smaller chunks.
- Increase Docker memory:
# docker-compose.yaml
services:
sirius-engine:
deploy:
resources:
limits:
memory: 4G # Increase from 2G
- Monitor memory usage:
docker stats sirius-engine
RabbitMQ Disconnections
Symptom: "Failed to consume from queue: connection closed"
Causes:
- RabbitMQ container crashed
- Network partition
- Connection timeout
Solutions:
- Check RabbitMQ status:
docker exec sirius-rabbitmq rabbitmqctl status
- Restart RabbitMQ:
docker restart sirius-rabbitmq
- Check connection string:
echo $RABBITMQ_URL
# Should be: amqp://guest:guest@sirius-rabbitmq:5672/
- Implement reconnection logic (future):
Add automatic reconnection with exponential backoff.
Performance Benchmarks
Environment: 4-core CPU, 8GB RAM, 1Gbps network
| Scan Type | Target Range | Duration | Hosts/sec |
|---|---|---|---|
| Quick (top 500 ports) | /24 subnet (254 hosts) | 3-5 min | ~50-80 |
| High-Risk | /24 subnet | 15-25 min | ~10-15 |
| All Scripts | /24 subnet | 2-4 hours | ~1-2 |
| Quick | Single host | 10-20 sec | N/A |
| High-Risk | Single host | 2-5 min | N/A |
Optimization Tips:
- Parallelism: Increase worker count for faster subnet scans
- Port Range: Limit to relevant ports (80, 443, 22, 3389)
- Script Selection: Use templates instead of wildcard
* - Network: Faster networks = faster scans (avoid VPNs)
Security Considerations
Responsible Scanning
Always:
- ✅ Get written authorization before scanning
- ✅ Use reasonable rate limiting (avoid DoS)
- ✅ Scan only approved IP ranges
- ✅ Document all scanning activities
Never:
- ❌ Scan production systems without approval
- ❌ Use aggressive mode on fragile systems
- ❌ Store vulnerability data in insecure locations
- ❌ Share scan results with unauthorized parties
Rate Limiting
The scanner implements implicit rate limiting through:
- Worker Pool Size: Limits concurrent scans
- Nmap Timing:
-T4(balanced) by default, not-T5(insane) - Connection Limits: Naabu respects system limits
Future Enhancements:
- Configurable rate limits (scans per second)
- Per-target rate limiting
- Adaptive throttling based on network conditions
Future Enhancements
Planned Features:
- DNS Wildcard Support: Subdomain enumeration via CT logs
- Distributed Scanning: Multiple scanner instances coordinating via ValKey
- Scan Scheduling: Cron-based recurring scans
- Advanced Reporting: PDF/HTML reports with executive summaries
- Machine Learning: Anomaly detection for unusual scan results
- API Gateway Integration: Direct scan triggering via REST API
- WebSocket Updates: Real-time scan progress in UI
- Scan Comparison: Diff scans to identify new vulnerabilities
ValKey Schema Reference
ValKey (Redis-compatible) is used for live scan state. Key structure:
currentScan– Main key storing the live scan result as JSON. Contains the fullScanResultobject with hosts, vulnerabilities, sub-scans, and progress.- Encoding: JSON may be stored as a base64-encoded string in some contexts; decode with
b64Decodefrom~/utils/std.tswhen reading from the frontend. - Polling: The frontend polls this key at regular intervals (5-second refetch) to display live progress.
- TTL: No TTL is set; the key persists until overwritten by the next scan.
Profile vs Template System
- Scan Profiles: High-level presets that control overall scan behavior (e.g., "quick-scan", "full-scan", "agent-only"). Stored as
ScanProfiletype. They determine which scan methods are enabled and how they are configured. - Agent Templates: Specific detection templates used by the agent scanner (e.g., checks for a specific CVE). Stored in the template system. They define detection steps such as
file_content,file_hash,file_search, andversion_cmd. The authoritative set isKnownDetectionTypesinapp-agent/internal/modules/detection_types.go(drift-checked against the runtime registry bydetection_types_registry_test.go). - Profiles may reference templates but are not the same thing: profiles define what to run and how; templates define the concrete detection logic.
RabbitMQ Message Schema
- Queue: Scan queue (consumed by app-scanner).
- Message format: JSON with target list and scan options.
- Scan request contents: Includes
targets,scan_typesconfiguration, andagent_scanconfiguration. - Results: app-scanner writes results back to the ValKey
currentScankey.
LLM Context
This documentation is optimized for AI assistant context. Key information:
Architecture: Message-driven (RabbitMQ), worker pool concurrency, strategy pattern for extensibility
Core Flow: Receive scan message → Expand targets → Distribute to workers → Execute strategies → Enrich results → Submit to API
Key Components: ScanManager (orchestration), ScanStrategies (tool integrations), TemplateManager (scan configs), NSE system (script management)
Integration Points: RabbitMQ (input), ValKey (state), PostgreSQL (results), NVD API (CVE enrichment)
Development: Go 1.23+, runs in sirius-engine container, live reload with Air, unit/integration tests
Common Tasks: Add strategies (implement interface + factory), create templates (InitializeSystemTemplates), debug NSE (check symlinks/manifest)
Last Updated: 2025-10-25
Version: 1.0.0
Maintainer: Sirius Team
Repository: https://github.com/SiriusScan/app-scanner