siriusscan--sirius/documentation/dev/architecture/README.go-api-sdk.md

title, description, template, llm_context, categories, tags, related_docs, search_keywords

title	description	template	llm_context	categories	tags	related_docs	search_keywords
Go API SDK Architecture	Comprehensive guide to the Sirius Go API SDK architecture, design patterns, and integration	TEMPLATE.documentation-standard	high	architecture sdk backend	go api sdk database orm gorm	README.architecture.md ../operations/README.sdk-releases.md ../../../minor-projects/go-api/README.md	go-api sdk gorm database models postgres rabbitmq valkey

Go API SDK Architecture

Overview

The Sirius Go API SDK (github.com/SiriusScan/go-api) is a shared library that provides:

• Core data models (Host, Port, Vulnerability, Service)
• Database operations (PostgreSQL via GORM)
• Message queue integration (RabbitMQ)
• Key-value store (ValKey/Redis)
• NVD API integration (vulnerability data enrichment)

Design Principles:

• Shared Types: Single source of truth for data structures
• Abstraction: Database and queue operations hidden behind clean APIs
• Source Attribution: Track which tool found each piece of data
• Flexibility: Works across multiple projects (scanner, API server, agents)

Package Structure

go-api/
├── sirius/               # Core functionality
│   ├── sirius.go        # Core types (Host, Port, Vulnerability, Service)
│   ├── postgres/        # Database operations
│   │   ├── connection.go        # DB initialization and connection management
│   │   ├── host_operations.go  # Host CRUD operations
│   │   ├── vulnerability_operations.go
│   │   └── models/      # Database models (GORM)
│   │       ├── host.go          # Host, Port, Service models
│   │       ├── vulnerability.go # Vulnerability, CVE models
│   │       └── scan_source.go   # Source attribution models
│   ├── host/           # Host management SDK
│   │   ├── host.go            # Basic host operations
│   │   └── source_aware.go    # Source-attributed operations
│   ├── queue/          # RabbitMQ integration
│   │   └── queue.go           # Publish/subscribe operations
│   ├── store/          # Key-value store (ValKey/Redis)
│   │   └── store.go           # KV operations
│   └── logging/        # Logging infrastructure
│       ├── client.go          # Logging client
│       └── api/               # Logging API server
├── nvd/                # NVD API integration
│   └── nvd.go                 # CVE data fetching
├── migrations/         # Database schema migrations
└── docs/               # SDK documentation

Core Data Models

Host Model

File: sirius/sirius.go

type Host struct {
    HID             string          // Host identifier
    OS              string          // Operating system
    OSVersion       string          // OS version
    IP              string          // IP address
    Hostname        string          // DNS hostname
    Ports           []Port          // Open ports
    Services        []Service       // Running services
    Vulnerabilities []Vulnerability // Found vulnerabilities
    CPE             []string        // Common Platform Enumeration
    Agent           *SiriusAgent    // Optional agent info
}

Port Model

File: sirius/sirius.go (Core), sirius/postgres/models/host.go (Database)

// Core type (used by applications)
type Port struct {
    Number   int    `json:"number"`   // Port number (22, 80, 443, etc.)
    Protocol string `json:"protocol"` // tcp, udp
    State    string `json:"state"`    // open, closed, filtered
}

// Database model (used by GORM)
type Port struct {
    gorm.Model              // ID, CreatedAt, UpdatedAt, DeletedAt
    Number    int           // Port number
    Protocol  string        // Protocol
    State     string        // State
    Hosts     []Host        `gorm:"many2many:host_ports"`
    HostPorts []HostPort    `gorm:"foreignKey:PortID"`
}

Key Design Decision: Port numbers are stored in the Number field, NOT the auto-increment ID field. This allows the same port (e.g., port 22) to be associated with multiple hosts without conflicts.

Vulnerability Model

File: sirius/sirius.go (Core), sirius/postgres/models/vulnerability.go (Database)

// Core type
type Vulnerability struct {
    VID         string  `json:"vid"`         // CVE ID (CVE-2017-0144)
    Title       string  `json:"title"`       // Vulnerability title
    Description string  `json:"description"` // Description
    RiskScore   float64 `json:"risk_score"`  // CVSS score (0-10)
}

// Database model
type Vulnerability struct {
    gorm.Model
    VID         string  `gorm:"column:v_id;uniqueIndex"` // Unique CVE ID
    Description string
    Title       string
    RiskScore   float64
    Hosts       []Host              `gorm:"many2many:host_vulnerabilities"`
    HostVulnerabilities []HostVulnerability `gorm:"foreignKey:VulnerabilityID"`
}

Source Attribution System

Purpose

Track which tool found which data on which host at what time. This enables:

• Audit trails: Know where data came from
• Tool comparison: Compare effectiveness of different scanners
• Historical tracking: See when vulnerabilities first appeared
• Confidence scoring: Weight findings by source reliability

Junction Tables with Source

HostPort Junction Table:

type HostPort struct {
    HostID        uint      `json:"host_id" gorm:"primaryKey"`
    PortID        uint      `json:"port_id" gorm:"primaryKey"`
    Source        string    `json:"source" gorm:"primaryKey"`      // nmap, rustscan, naabu
    SourceVersion string    `json:"source_version"`                 // Tool version
    FirstSeen     time.Time `json:"first_seen"`                     // First detection
    LastSeen      time.Time `json:"last_seen"`                      // Last confirmation
    Status        string    `json:"status" gorm:"default:active"`   // active, resolved
    Notes         string    `json:"notes,omitempty"`                // Config details
}

Primary Key: (host_id, port_id, source) allows:

• Same port on same host tracked by multiple tools
• Comparison of results between tools
• Historical view of port states

HostVulnerability Junction Table:

Similar structure with additional fields:

• Confidence (0.0-1.0): How confident is this finding?
• Port (optional): Specific port where vulnerability found
• ServiceInfo: Service details

Using Source Attribution

Example: Add host with source:

import (
    "github.com/SiriusScan/go-api/sirius"
    "github.com/SiriusScan/go-api/sirius/host"
    "github.com/SiriusScan/go-api/sirius/postgres/models"
)

// Create host data
hostData := sirius.Host{
    IP: "192.168.1.100",
    Ports: []sirius.Port{
        {Number: 22, Protocol: "tcp", State: "open"},
        {Number: 80, Protocol: "tcp", State: "open"},
    },
}

// Create source metadata
source := models.ScanSource{
    Name:    "nmap",
    Version: "7.94",
    Config:  "ports:1-1000;template:quick;timing:T4",
}

// Submit with source attribution
err := host.AddHostWithSource(hostData, source)

Database Operations

Connection Management

File: sirius/postgres/connection.go

// Get database connection (singleton)
db := postgres.GetDB()

// Initialize schema
err := postgres.InitDB()

Environment Variables:

• DATABASE_HOST (default: localhost)
• DATABASE_PORT (default: 5432)
• DATABASE_USER (default: postgres)
• DATABASE_PASSWORD (default: postgres)
• DATABASE_NAME (default: sirius)

Host Operations

Basic Operations (Legacy):

import "github.com/SiriusScan/go-api/sirius/host"

// Add or update host
err := host.AddHost(hostData)

// Get host by IP
hostData, err := host.GetHost("192.168.1.100")

// Get all hosts
hosts, err := host.GetAllHosts()

// Delete host
err := host.DeleteHost("192.168.1.100")

Source-Aware Operations (Recommended):

// Add/update with source tracking
err := host.AddHostWithSource(hostData, source)

// Get host with all source attributions
hostWithSources, err := host.GetHostWithSources("192.168.1.100")

// Get vulnerability history by source
history, err := host.GetVulnerabilityHistory(hostID, vulnID)

// Get source coverage statistics
stats, err := host.GetSourceCoverageStats()

Mapping Between Core and Database Types

File: sirius/host/host.go

// Convert database model to core type
siriusHost := host.MapDBHostToSiriusHost(dbHost)

// Convert core type to database model
dbHost := host.MapSiriusHostToDBHost(siriusHost)

Important: These functions handle:

• Port.Number ↔ Port field mapping
• Relationship loading (ports, services, vulnerabilities)
• Type conversions

Message Queue Integration

RabbitMQ

File: sirius/queue/queue.go

import "github.com/SiriusScan/go-api/sirius/queue"

// Publish message to queue
err := queue.Publish("scan", scanMessage)

// Listen for messages
queue.Listen("scan", func(msg string) {
    // Process message
})

Environment Variables:

• RABBITMQ_HOST (default: localhost)
• RABBITMQ_PORT (default: 5672)
• RABBITMQ_USER (default: guest)
• RABBITMQ_PASSWORD (default: guest)

Common Queues:

• scan - Scan requests
• scanner_logs - Scanner log events
• agent_commands - Agent commands
• terminal - Terminal commands

Key-Value Store Integration

ValKey/Redis

File: sirius/store/store.go

import "github.com/SiriusScan/go-api/sirius/store"

// Set value
err := store.SetValue(ctx, "key", "value")

// Get value
result, err := store.GetValue(ctx, "key")

// Delete value
err := store.DeleteValue(ctx, "key")

// List keys
keys, err := store.ListKeys(ctx, "pattern:*")

Environment Variables:

• VALKEY_HOST (default: localhost)
• VALKEY_PORT (default: 6379)

Common Use Cases:

• Real-time scan progress tracking
• Template storage (scan templates)
• Temporary data caching

NVD Integration

CVE Data Enrichment

File: nvd/nvd.go

import "github.com/SiriusScan/go-api/nvd"

// Fetch CVE details from NVD
cveData, err := nvd.GetCVE("CVE-2017-0144")

// Access CVSS scores
score := cveData.Metrics.CvssMetricV31[0].CvssData.BaseScore

Rate Limiting: NVD API has rate limits. SDK respects these limits.

Integration Patterns

Pattern 1: Local Development with Replace Directive

Use Case: Developing changes to go-api and dependent project simultaneously

go.mod:

module github.com/SiriusScan/app-scanner

replace github.com/SiriusScan/go-api => ../go-api  // Local development

require (
    github.com/SiriusScan/go-api v0.0.11  // Version for production
)

Benefits:

• Test SDK changes immediately
• No need to publish SDK for every test
• Easy debugging across projects

Pattern 2: Production Use with Versioned Import

Use Case: Production deployments

go.mod:

module github.com/SiriusScan/app-scanner

require (
    github.com/SiriusScan/go-api v0.0.11  // Specific version
)

Benefits:

• Reproducible builds
• Version pinning
• Explicit dependency management

Pattern 3: Container Development with Volume Mounts

Use Case: Docker-based development (sirius-engine, sirius-api)

docker-compose.dev.yaml:

services:
  sirius-engine:
    volumes:
      - ../minor-projects/go-api:/go-api:ro  # Mount SDK source
      - ../minor-projects/app-scanner:/app-scanner

go.mod in container:

replace github.com/SiriusScan/go-api => /go-api

Benefits:

• Live code reload
• No rebuild for SDK changes
• Mirrors local development

Version Management

Semantic Versioning

Format: v{MAJOR}.{MINOR}.{PATCH}

Rules:

• MAJOR (0.x.x → 1.x.x): Breaking changes, incompatible API changes
• MINOR (x.0.x → x.1.x): New features, backward compatible
• PATCH (x.x.0 → x.x.1): Bug fixes, backward compatible

Current: v0.0.11 (pre-1.0, unstable API)

Breaking Change Policy

What Constitutes a Breaking Change:

• ✅ Renaming exported fields (e.g., Port.ID → Port.Number)
• ✅ Changing function signatures
• ✅ Removing exported functions/types
• ✅ Changing database schema in non-backward-compatible way
• ❌ Adding new fields (backward compatible)
• ❌ Adding new functions (backward compatible)
• ❌ Internal implementation changes

Communication:

• Document in CHANGELOG.md with BREAKING CHANGE: prefix
• Include migration guide
• Increment version appropriately
• Notify dependent projects

Compatibility Guarantees

Before v1.0.0:

• ⚠️ No API stability guaranteed
• Breaking changes may occur in patch releases
• Use exact version pinning in production

After v1.0.0:

• ✅ Semantic versioning strictly followed
• ✅ Breaking changes only in major versions
• ✅ Deprecation warnings before removal

Development Workflow

Making Changes to SDK

1. Create feature branch:

cd go-api
git checkout -b feature/my-change

2. Make changes and test locally:

# Edit files
go test ./...
go build ./...

3. Test in dependent project:

cd ../app-scanner
# Ensure replace directive points to ../go-api
go mod tidy
go build .

4. Commit and push:

cd ../go-api
git add .
git commit -m "feat: add new feature"
git push origin feature/my-change

5. Create Pull Request

6. After merge to main:

• CI/CD automatically creates new release
• Update dependent projects to new version

Testing Changes

Unit Tests:

cd go-api
go test ./...

Integration Tests:

# Test with actual database
DATABASE_HOST=localhost go test ./sirius/postgres/...

# Test with actual RabbitMQ
RABBITMQ_HOST=localhost go test ./sirius/queue/...

Manual Testing:

# Use in dependent project with replace directive
cd app-scanner
go run main.go

Contributing Guidelines

Code Style:

• Follow Go idioms and best practices
• Use gofmt for formatting
• Write meaningful commit messages (conventional commits)
• Add tests for new functionality

Documentation:

• Update CHANGELOG.md for user-facing changes
• Add godoc comments for exported types/functions
• Update SDK documentation for architectural changes

Pull Requests:

• Keep PRs focused and small
• Include test coverage
• Reference related issues
• Wait for CI/CD to pass

Common Patterns & Best Practices

Pattern: Bulk Operations

Problem: Need to process many hosts efficiently

Solution:

// Use transactions for bulk operations
db := postgres.GetDB()
tx := db.Begin()

for _, hostData := range hosts {
    dbHost := host.MapSiriusHostToDBHost(hostData)
    if err := tx.Create(&dbHost).Error; err != nil {
        tx.Rollback()
        return err
    }
}

tx.Commit()

Pattern: Error Handling

Problem: Need consistent error handling

Solution:

import "fmt"

// Wrap errors with context
if err != nil {
    return fmt.Errorf("failed to add host %s: %w", ip, err)
}

// Check for specific errors
if errors.Is(err, gorm.ErrRecordNotFound) {
    // Handle not found
}

Pattern: Connection Pooling

Problem: Efficient database connections

Solution:

// GetDB() returns singleton with connection pool
db := postgres.GetDB()

// Configure in connection.go:
sqlDB, _ := db.DB()
sqlDB.SetMaxOpenConns(25)
sqlDB.SetMaxIdleConns(5)
sqlDB.SetConnMaxLifetime(5 * time.Minute)

Pattern: Graceful Shutdown

Problem: Clean up resources on exit

Solution:

func main() {
    // Initialize connections
    db := postgres.GetDB()
    
    // Handle shutdown
    c := make(chan os.Signal, 1)
    signal.Notify(c, os.Interrupt, syscall.SIGTERM)
    
    <-c
    
    // Cleanup
    sqlDB, _ := db.DB()
    sqlDB.Close()
}

Troubleshooting

Issue: "Module not found"

Problem:

go: github.com/SiriusScan/go-api@v0.0.11: invalid version: unknown revision v0.0.11

Solutions:

1. Check if version exists: git ls-remote --tags https://github.com/SiriusScan/go-api
2. Clear Go module cache: go clean -modcache
3. Use replace directive for local development

Issue: "Undefined field after update"

Problem:

port.ID undefined (type sirius.Port has no field or method ID)

Solution:

• Check CHANGELOG.md for breaking changes
• Update code to use new field names (e.g., port.Number)
• Run go mod tidy after updating version

Issue: Database connection fails

Problem:

failed to connect to database: connection refused

Solution:

1. Check environment variables are set correctly
2. Verify database is running: docker ps | grep postgres
3. Test connection: psql -h localhost -U postgres -d sirius

Issue: "Duplicate key violation"

Problem:

ERROR: duplicate key value violates unique constraint "ports_pkey"

Solution:

• Ensure using SDK version ≥ v0.0.11 (fixes Port.ID conflict)
• Run migration 005_fix_critical_schema_issues
• Clear old data if necessary

Migration Guides

Migrating from v0.0.9/v0.0.10 to v0.0.11

Breaking Changes:

1. Port.ID → Port.Number
2. CVEDataMeta.ID → CVEDataMeta.CVEIdentifier

Code Changes Required:

Before:

port := sirius.Port{
    ID:       22,
    Protocol: "tcp",
}

ports := make([]int, len(host.Ports))
for i, port := range host.Ports {
    ports[i] = port.ID  // ❌ No longer exists
}

After:

port := sirius.Port{
    Number:   22,
    Protocol: "tcp",
}

ports := make([]int, len(host.Ports))
for i, port := range host.Ports {
    ports[i] = port.Number  // ✅ Correct
}

Database Migration:

# Run migration in container
docker exec sirius-engine bash -c "cd /tmp/migrations && go run 005_fix_critical_schema_issues/main.go"

Verification:

1. Update go.mod: require github.com/SiriusScan/go-api v0.0.11
2. Run: go mod tidy
3. Find all references: grep -r "port\.ID" .
4. Replace with port.Number
5. Test build: go build ./...
6. Run tests: go test ./...

References

Related Documentation

• SDK Release Process - How to release new SDK versions
• System Architecture - Overall Sirius architecture
• go-api README - SDK getting started guide
• go-api CHANGELOG - Version history

External Resources

• GORM Documentation
• Go Module Reference
• RabbitMQ Go Client
• ValKey Documentation

---

Last Updated: 2025-10-26
SDK Version: v0.0.11
Author: Sirius Team