siriusscan--sirius/documentation/dev-notes/archive/IMPLEMENTATION-PLAN-SCAN-TRACKING.md

Implementation Plan: Scan Source Tracking & History System

Project Overview

This implementation plan addresses the critical issue where multiple scan sources (network scanner, agent scanner, manual submissions) overwrite each other's results in the database. The solution implements source-tagged scan history to maintain comprehensive vulnerability data from all sources.

Goals

1. Eliminate Data Loss: Prevent scan results from different sources overwriting each other
2. Source Attribution: Track which scanner/tool reported each finding
3. Historical Tracking: Maintain timeline of when vulnerabilities were first/last seen
4. Holistic View: Present unified vulnerability assessment combining all sources
5. Backward Compatibility: Ensure existing API consumers continue working

Technical Approach

Database Schema Evolution

Current Problem: Junction tables use Association(...).Replace() which completely overwrites all relationships, losing data from other sources.

Solution: Enhanced junction tables with source attribution and temporal tracking.

New Schema Design

// Enhanced junction table with source tracking
type HostVulnerability struct {
    gorm.Model
    HostID          uint      `json:"host_id"`
    VulnerabilityID uint      `json:"vulnerability_id"`
    Source          string    `json:"source"`          // "nmap", "agent", "manual", "rustscan"
    SourceVersion   string    `json:"source_version"`  // Scanner version/build
    FirstSeen       time.Time `json:"first_seen"`      // When first detected
    LastSeen        time.Time `json:"last_seen"`       // When last confirmed
    Status          string    `json:"status"`          // "active", "resolved", "false_positive"
    Confidence      float64   `json:"confidence"`      // 0.0-1.0 confidence score
    Port            *int      `json:"port,omitempty"`  // Specific port if applicable
    ServiceInfo     string    `json:"service_info,omitempty"` // Service details
    Notes           string    `json:"notes,omitempty"` // Additional context
}

// Similarly for ports
type HostPort struct {
    gorm.Model
    HostID        uint      `json:"host_id"`
    PortID        uint      `json:"port_id"`
    Source        string    `json:"source"`
    SourceVersion string    `json:"source_version"`
    FirstSeen     time.Time `json:"first_seen"`
    LastSeen      time.Time `json:"last_seen"`
    Status        string    `json:"status"`
    Notes         string    `json:"notes,omitempty"`
}

API Architecture Changes

Source-Aware Data Structures

type ScanSource struct {
    Name    string `json:"name"`    // "nmap", "agent", "rustscan", "manual"
    Version string `json:"version"` // Tool version
    Config  string `json:"config"`  // Scan configuration used
}

type SourcedHost struct {
    sirius.Host
    Source ScanSource `json:"source"`
}

New Core Functions

Replace the problematic Association(...).Replace() calls with source-aware operations:

// Core function signatures
func AddHostWithSource(host sirius.Host, source ScanSource) error
func UpdateVulnerabilitiesWithSource(hostID uint, vulns []sirius.Vulnerability, source ScanSource) error
func UpdatePortsWithSource(hostID uint, ports []sirius.Port, source ScanSource) error
func GetHostWithSources(ip string) (HostWithSources, error)
func GetVulnerabilityHistory(hostID uint, vulnID uint) ([]SourceAttribution, error)

Scanner Integration Strategy

Network Scanner (app-scanner/)

File: app-scanner/internal/scan/manager.go

Changes:

• Add source metadata to all database calls
• Include scanner version and configuration details
• Update result processing to use source-aware API functions

// Example integration
source := ScanSource{
    Name:    "nmap",
    Version: getNmapVersion(),
    Config:  scanConfig.String(),
}
err := host.AddHostWithSource(discoveredHost, source)

Agent Scanner (app-agent/)

File: app-agent/internal/commands/scan/scan_command.go

Changes:

• Include agent version and scan type in API submissions
• Modify HTTP POST payload to include source information
• Update result structures to carry source metadata

API Handlers (sirius-api/)

File: sirius-api/handlers/host_handler.go

Changes:

• Detect source information from request headers or payload
• Route to appropriate source-aware functions
• Maintain backward compatibility for existing clients

Database Migration Strategy

Since existing data preservation is not required, we'll implement a clean migration:

1. Schema Update: Add new fields to junction tables
2. Index Creation: Add indexes for efficient source-based queries
3. Data Migration: Mark existing data with "unknown" source
4. Constraint Addition: Add foreign key constraints and validation

Frontend Enhancement Plan

Enhanced Display Components

Vulnerability Tables:

• Source attribution columns
• Historical timeline view
• Source filtering capabilities
• Confidence-based sorting

Host Details Views:

• Per-source vulnerability breakdown
• Scanner coverage matrix
• Historical scan timeline

New API Endpoints for Frontend

GET /host/{ip}/sources           - Get all sources that scanned this host
GET /host/{ip}/history          - Get scan history timeline
GET /vulnerability/{id}/sources  - Get which sources reported this CVE
GET /sources/coverage           - Get coverage statistics per source

Implementation Phases

Phase 1: Core Database & API Changes (Week 1)

Priority: High Goal: Fix the data overwriting issue

1. Update database schema with source tracking fields
2. Create database migration scripts
3. Implement source-aware core functions
4. Replace Association(...).Replace() calls
5. Add comprehensive unit tests

Phase 2: Scanner Integration (Week 2)

Priority: High Goal: Integrate source attribution into scanners

1. Update network scanner to use source-aware API
2. Update agent scanner to include source metadata
3. Modify API handlers for source detection
4. Implement backward compatibility layer
5. Add integration tests

Phase 3: Frontend Enhancements (Week 3)

Priority: Medium Goal: Present source-attributed data to users

1. Create enhanced vulnerability display components
2. Add source filtering and comparison features
3. Implement historical timeline views
4. Update existing pages to show source information
5. Add source coverage dashboards

Phase 4: Testing & Documentation (Week 4)

Priority: Medium Goal: Ensure reliability and maintainability

1. Comprehensive end-to-end testing
2. Performance testing with multiple sources
3. Documentation updates
4. Deployment and monitoring setup

Risk Mitigation

Technical Risks

1. Performance Impact: Additional fields and joins may slow queries
   • Mitigation: Add strategic indexes, implement query optimization
2. Backward Compatibility: Existing API clients may break
   • Mitigation: Implement compatibility layer, gradual deprecation
3. Data Consistency: Complex source attribution logic may introduce bugs
   • Mitigation: Comprehensive testing, atomic transactions

Development Risks

1. Scope Creep: Feature complexity may expand during development
   • Mitigation: Clear phase boundaries, regular stakeholder reviews
2. Testing Complexity: Multiple sources create exponential test scenarios
   • Mitigation: Automated test suites, containerized test environments

Success Metrics

Technical Metrics

• Zero data loss incidents between scan sources
• All vulnerabilities properly attributed to sources
• Response time degradation < 20% after source tracking
• 100% API backward compatibility maintained

Functional Metrics

• Users can see which scanner found each vulnerability
• Historical vulnerability tracking works across all sources
• Unified vulnerability view combines all source data
• Source-specific filtering and reporting functions

Quality Metrics

• Unit test coverage > 85% for new functionality
• Integration test coverage for all scanner combinations
• End-to-end testing validates complete workflows
• Documentation covers all new features and APIs

Dependencies

Internal Dependencies

• Database migration capabilities
• Scanner development environments
• Frontend development stack
• Testing infrastructure

External Dependencies

• PostgreSQL database availability
• Scanner tool compatibility (nmap, rustscan, etc.)
• Agent deployment and connectivity
• Container orchestration for testing

Rollback Strategy

If critical issues arise during implementation:

1. Phase 1 Rollback: Revert database schema, restore original Association(...).Replace() logic
2. Phase 2 Rollback: Disable source-aware scanner calls, use legacy API endpoints
3. Phase 3 Rollback: Hide source attribution UI, revert to original display components
4. Data Recovery: Full database backup before each phase, automated restore procedures

Post-Implementation

Monitoring

• Database query performance metrics
• API response time monitoring
• Source attribution accuracy tracking
• User adoption of new features

Maintenance

• Regular source attribution data quality checks
• Scanner version tracking and updates
• Historical data cleanup and archival
• Performance optimization based on usage patterns

---

This implementation plan provides a structured approach to solving the scan result overwriting issue while maintaining system reliability and user experience. The phased approach allows for incremental delivery and risk management throughout the development process.