chore: import upstream snapshot with attribution

documentation/dev/apps/scanner/ARCHITECTURE.host-deduplication.md

@@ -0,0 +1,162 @@
+---
+title: "Host Deduplication and Multi-Source Attribution"
+description: "Canonical host identity by IP, multi-source attribution (network/agent), scan_sources field, and UI merge/display with SourceIcon and SourceIconRow."
+template: "TEMPLATE.custom"
+version: "1.0.0"
+last_updated: "2025-02-07"
+author: "Sirius Team"
+tags: ["scanner", "host", "deduplication", "scan_sources", "SourceIcon", "multi-source"]
+categories: ["architecture", "scanner"]
+difficulty: "intermediate"
+prerequisites: ["README.scanner.md", "ARCHITECTURE.sub-scans.md"]
+related_docs:
+  - "documentation/dev/apps/scanner/README.scanner.md"
+  - "documentation/dev/apps/scanner/ARCHITECTURE.scanner-data-flow.md"
+  - "documentation/dev/apps/scanner/ARCHITECTURE.sub-scans.md"
+dependencies: []
+llm_context: "high"
+search_keywords:
+  - "host deduplication"
+  - "scan_sources"
+  - "multi-source"
+  - "SourceIcon"
+  - "SourceIconRow"
+  - "EnvironmentTableData"
+  - "canonical host IP"
+---
+
+# Host Deduplication and Multi-Source Attribution
+
+Hosts in the Sirius scanner can be discovered by multiple scan methods (network and agent). This document describes how hosts are identified by a canonical key, how multiple sources are attributed, and how the UI merges and displays them.
+
+## Canonical Host Identity: IP Address
+
+The **primary key** for a host is its **IP address**. Regardless of whether a host was found by a network scan, an agent scan, or both, it is represented once per IP in the merged view.
+
+- **Backend:** `HostEntry` (and persisted host records) use `ip` as the canonical identifier; `id` can be a unique row or document id; `hostname`, `aliases`, and `sources` are optional.
+- **UI:** Tables and detail views key hosts by `ip` when merging data from different sub-scans or from the environment summary API.
+
+This allows:
+
+- One row per host in the environment/host table
+- Correct aggregation of vulnerability counts and source badges per host
+- Stable navigation to host detail (e.g. by IP) regardless of which scan discovered it
+
+---
+
+## Multi-Source Attribution
+
+A host can be discovered by:
+
+- **network** – app-scanner (Nmap, RustScan, etc.)
+- **agent** – app-agent (template runs on remote host)
+
+Other sources (e.g. cloud, application) can be added later. Each discovery path that reports a host should tag it with its **source identifier** (e.g. `"network"`, `"agent"`).
+
+### HostEntry.sources (backend / ValKey)
+
+In the live `ScanResult` stored in ValKey, each `HostEntry` can carry a list of sources:
+
+```go
+type HostEntry struct {
+    ID       string   `json:"id"`
+    IP       string   `json:"ip"`
+    Hostname string   `json:"hostname,omitempty"`
+    Aliases  []string `json:"aliases,omitempty"`
+    Sources  []string `json:"sources,omitempty"` // e.g. ["network", "agent"]
+}
+```
+
+When merging results from multiple sub-scans, the component that writes to `currentScan` should:
+
+- Merge hosts by IP (one entry per IP)
+- Set `sources` to the union of all sources that reported that IP (e.g. if both network and agent found it, `sources = ["network", "agent"]`)
+
+---
+
+## EnvironmentTableData.scan_sources (UI)
+
+The UI uses **EnvironmentTableData** for the environment/host table. It includes:
+
+- **scan_source** (optional, legacy): Single source string (e.g. `"network"` or `"agent"`).
+- **scan_sources** (optional): Array of source strings for multi-source attribution.
+
+```ts
+interface EnvironmentTableData {
+  hostname: string;
+  ip: string;
+  os: string;
+  vulnerabilityCount: number;
+  maxCvss?: number;
+  groups: string[];
+  tags: string[];
+  scan_source?: ScanSource;   // legacy single source
+  scan_sources?: string[];    // all discovery sources for this host
+}
+```
+
+When mapping from API or from live scan results:
+
+- Prefer **scan_sources** (array) when building the table (e.g. from `host.sources` or equivalent).
+- Fall back to **scan_source** for older data or single-source responses, and convert to `scan_sources` for consistent display (e.g. `row.scan_sources ?? (row.scan_source ? [row.scan_source] : [])`).
+
+---
+
+## How the UI Merges Hosts from Different Sub-Scans
+
+1. **Live scan results (ValKey):**  
+   The `ScanResult.hosts` array may already be merged by the backend (app-scanner and agent path both updating the same `currentScan` and merging by IP with combined `sources`). The UI then maps `HostEntry[]` to `EnvironmentTableData[]` and sets `scan_sources = host.sources || []`.
+
+2. **Environment summary (API):**  
+   When the UI fetches the environment host list (e.g. `host.getEnvironmentSummary`), the API returns rows that may include a `sources` (or `scan_sources`) field per host. The UI maps that to `EnvironmentTableData.scan_sources`.
+
+3. **Deduplication by IP:**  
+   If the UI ever receives multiple rows for the same IP (e.g. from different endpoints), it should merge them into one row and combine `scan_sources` (union of all source arrays) so that the table shows one row per host with all sources that discovered it.
+
+---
+
+## Source Display: SourceIcon and SourceIconRow
+
+The UI provides two components for showing scan source(s):
+
+### SourceIcon (single source)
+
+- **Purpose:** Renders one scan source (e.g. `"network"` or `"agent"`) as an icon with optional label and tooltip.
+- **Usage:** `<SourceIcon source="agent" />`, `<SourceIcon source="network" showLabel />`.
+- **Registry:** `SOURCE_ICON_REGISTRY` maps source keys to icon, color, and label (e.g. `agent` → Bot icon, cyan; `network` → Wifi icon, violet).
+
+### SourceIconRow (multiple sources)
+
+- **Purpose:** Renders a row of icons for all sources that discovered a host (or finding).
+- **Usage:** `<SourceIconRow sources={["agent", "network"]} />`.
+- **Behavior:** Renders one `SourceIcon` per entry in `sources`; tooltip/title can show a combined label (e.g. "Agent + Network").
+
+**Example in host table column:**
+
+- Accessor: `row.scan_sources ?? (row.scan_source ? [row.scan_source] : [])`.
+- Cell: `<SourceIconRow sources={sources} />`.
+
+This gives users a clear view of which scan methods discovered each host (network-only, agent-only, or both).
+
+---
+
+## Summary
+
+- **Canonical host identity:** IP address.
+- **Multi-source attribution:** Hosts carry `sources` (backend) / `scan_sources` (UI) listing every scan method that discovered them.
+- **EnvironmentTableData:** Prefer `scan_sources: string[]`; support legacy `scan_source` by normalizing to an array.
+- **UI merge:** One row per IP; `scan_sources` = union of sources for that IP.
+- **Display:** Use **SourceIcon** for a single source and **SourceIconRow** for the list of sources on a host (or vulnerability) row.
+
+---
+
+**Related Documentation**
+
+- [ARCHITECTURE.scanner-data-flow.md](./ARCHITECTURE.scanner-data-flow.md) – How host data flows from scanners into ValKey and to the UI
+- [ARCHITECTURE.sub-scans.md](./ARCHITECTURE.sub-scans.md) – How network and agent sub-scans contribute hosts
+
+---
+
+**Last Updated:** 2025-02-07  
+**Version:** 1.0.0  
+**Maintainer:** Sirius Team

documentation/dev/apps/scanner/ARCHITECTURE.scanner-data-flow.md

@@ -0,0 +1,156 @@
+---
+title: "Scanner Data Flow - End-to-End Architecture"
+description: "End-to-end data flow from UI scan initiation through network/agent scans to result display and persistence."
+template: "TEMPLATE.custom"
+version: "1.0.0"
+last_updated: "2025-02-07"
+author: "Sirius Team"
+tags: ["scanner", "data-flow", "architecture", "valkey", "rabbitmq", "grpc"]
+categories: ["architecture", "scanner"]
+difficulty: "intermediate"
+prerequisites: ["README.scanner.md"]
+related_docs:
+  - "documentation/dev/apps/scanner/README.scanner.md"
+  - "documentation/dev/apps/scanner/ARCHITECTURE.sub-scans.md"
+  - "documentation/dev/apps/scanner/ARCHITECTURE.host-deduplication.md"
+  - "documentation/dev/architecture/README.architecture.md"
+dependencies: []
+llm_context: "high"
+search_keywords:
+  - "scanner data flow"
+  - "scan initiation"
+  - "ValKey currentScan"
+  - "network scan flow"
+  - "agent scan flow"
+  - "scan persistence"
+---
+
+# Scanner Data Flow – End-to-End Architecture
+
+This document describes the end-to-end data flow of the Sirius scanner system from the moment a user starts a scan in the UI until results are displayed and optionally persisted.
+
+## Overview
+
+The scanner system uses a **single live scan state** stored in ValKey under the key `currentScan`. The UI initiates scans via tRPC, triggers one or more sub-scans (network and/or agent), and polls ValKey for progress. When scans complete, results can be persisted from ValKey through go-api into PostgreSQL.
+
+**Components involved:**
+
+- **sirius-ui** (Next.js): Initiates scans, polls for results, displays hosts and vulnerabilities
+- **go-api** (Go): REST API and persistence; can proxy queue/store or be called for cancel/persist
+- **app-scanner** (Go): Network scanning; consumes from RabbitMQ, updates ValKey
+- **app-agent** (Go): Agent-based scanning via gRPC; runs templates on remote hosts, results flow to ValKey
+- **ValKey**: Stores live scan result as `currentScan` (base64-encoded JSON)
+- **RabbitMQ**: Message queue between API/UI and app-scanner for network scan requests
+
+---
+
+## Network Scan Flow
+
+1. **UI** – User configures targets and scan options, clicks “Start Scan”.
+2. **tRPC** – UI calls `queue.sendMsg` (and optionally scanner/start endpoints) and `store.setValue` with key `currentScan` to write initial `ScanResult` (id, status, targets, empty hosts/vulnerabilities, `sub_scans.network`).
+3. **go-api / Queue** – Scan request (id, targets, options, priority) is published to the **RabbitMQ** `scan` queue (either via go-api or via tRPC router in sirius-ui that publishes to RabbitMQ).
+4. **app-scanner** – Consumes the message from the `scan` queue, expands targets to IPs, runs the scan pipeline (e.g. enumeration → discovery → vulnerability), and for each host/vulnerability discovered:
+   - Reads current `currentScan` from ValKey (or receives it in context),
+   - Merges new hosts/vulnerabilities and updates progress,
+   - Writes updated `ScanResult` back to ValKey key `currentScan`.
+5. **UI polling** – `useScanResults` polls `store.getValue({ key: "currentScan" })` (e.g. every 3s). The tRPC store router reads from ValKey and returns the value; UI decodes the base64 JSON and updates hosts/vulnerabilities for display.
+
+So: **UI → tRPC → go-api (or tRPC) → RabbitMQ → app-scanner → ValKey → UI polling (tRPC store.getValue → ValKey).**
+
+---
+
+## Agent Scan Flow
+
+1. **UI** – Same scan start; if agent scan is enabled, UI builds `sub_scans.agent` (status `dispatching`, progress 0/0) and writes initial `ScanResult` to ValKey via `store.setValue` with key `currentScan`.
+2. **tRPC** – UI calls `agentScan.dispatchAgentScan` with scan id and agent scan config.
+3. **go-api / gRPC** – go-api (or backend invoked by tRPC) uses **gRPC** to communicate with **app-agent** to dispatch template scans to connected agents.
+4. **app-agent** – Runs templates on remote hosts, collects results, and reports back (e.g. via gRPC or callback to API). The component that holds the “current scan” state (e.g. go-api or a worker) updates the `currentScan` value in **ValKey**: merges agent-discovered hosts and vulnerabilities, updates `sub_scans.agent` status and progress.
+5. **UI polling** – Same as network: UI polls `store.getValue("currentScan")`, decodes the result, and displays agent-origin hosts and vulnerabilities (with source attribution).
+
+So: **UI → tRPC → go-api → gRPC → app-agent → (results) → ValKey → UI polling.**
+
+---
+
+## Persistence: ValKey (Live) → go-api → PostgreSQL
+
+- **Live state** – The only source of truth during an active scan is ValKey key `currentScan`. All sub-scans (network, agent) read-modify-write this key to add hosts, vulnerabilities, and progress.
+- **Persistent state** – After a scan completes (or on demand), the API layer can:
+  - Read the final `ScanResult` from ValKey (`currentScan`),
+  - Map hosts and vulnerabilities to the persistent model,
+  - Write to **PostgreSQL** via go-api (hosts, vulnerabilities, scan_sources, etc.).
+
+So: **ValKey (live) → go-api → PostgreSQL (persistent).** The UI can show live data from ValKey and, for history/reporting, data from go-api/PostgreSQL.
+
+---
+
+## Sequence Diagram (Mermaid)
+
+```mermaid
+sequenceDiagram
+  participant UI as sirius-ui
+  participant tRPC as tRPC (Next.js)
+  participant API as go-api
+  participant RMQ as RabbitMQ
+  participant Scanner as app-scanner
+  participant Valkey as ValKey
+  participant gRPC as gRPC
+  participant Agent as app-agent
+  participant DB as PostgreSQL
+
+  UI->>tRPC: Start scan (targets, options)
+  tRPC->>Valkey: setValue("currentScan", initial ScanResult)
+  tRPC->>API: (optional) start / queue
+  API->>RMQ: Publish scan message (network)
+  tRPC->>API: dispatchAgentScan (if agent enabled)
+  API->>gRPC: Dispatch to agents
+  gRPC->>Agent: Run templates on hosts
+
+  loop Network scan
+    Scanner->>RMQ: Consume scan message
+    Scanner->>Scanner: Expand targets, run phases
+    Scanner->>Valkey: Get currentScan
+    Scanner->>Valkey: Set currentScan (merge hosts/vulns)
+  end
+
+  loop Agent scan
+    Agent->>Agent: Execute templates
+    Agent->>API: Report results (or via gRPC)
+    API->>Valkey: Get currentScan
+    API->>Valkey: Set currentScan (merge agent results)
+  end
+
+  loop UI polling (e.g. every 3s)
+    UI->>tRPC: getValue("currentScan")
+    tRPC->>Valkey: GET currentScan
+    Valkey-->>tRPC: base64 ScanResult
+    tRPC-->>UI: ScanResult
+    UI->>UI: Decode, update hosts/vulnerabilities
+  end
+
+  note over API,DB: On completion or on demand
+  API->>Valkey: GET currentScan
+  Valkey-->>API: ScanResult
+  API->>DB: Persist hosts, vulnerabilities, sources
+```
+
+---
+
+## Key Data Structures
+
+- **ValKey key:** `currentScan` (string value = base64-encoded JSON).
+- **ScanResult (JSON):** `id`, `status`, `targets`, `hosts` (array of `HostEntry` with `id`, `ip`, `hostname`, `sources`), `hosts_completed`, `vulnerabilities`, `start_time`, `end_time`, `sub_scans` (map of sub-scan key → `SubScan`).
+- **SubScan:** `type`, `enabled`, `status`, `progress` (`completed`, `total`, `label`), `metadata` (optional). Used for both network and agent sub-scans.
+
+---
+
+## Related Documentation
+
+- [ARCHITECTURE.sub-scans.md](./ARCHITECTURE.sub-scans.md) – Sub-scan lifecycle and progress aggregation
+- [ARCHITECTURE.host-deduplication.md](./ARCHITECTURE.host-deduplication.md) – How hosts from multiple sources are merged and displayed
+- [README.scanner.md](./README.scanner.md) – Scanner engine details (message format, strategies, ValKey usage)
+
+---
+
+**Last Updated:** 2025-02-07  
+**Version:** 1.0.0  
+**Maintainer:** Sirius Team

documentation/dev/apps/scanner/ARCHITECTURE.sub-scans.md

@@ -0,0 +1,185 @@
+---
+title: "Sub-Scans Architecture - Modular Scan Methods"
+description: "Independent scan methods (network, agent) running in parallel, their data structure, lifecycle, progress aggregation, and cancellation."
+template: "TEMPLATE.custom"
+version: "1.0.0"
+last_updated: "2025-02-07"
+author: "Sirius Team"
+tags: ["scanner", "sub-scan", "architecture", "progress", "cancellation"]
+categories: ["architecture", "scanner"]
+difficulty: "intermediate"
+prerequisites: ["README.scanner.md"]
+related_docs:
+  - "documentation/dev/apps/scanner/README.scanner.md"
+  - "documentation/dev/apps/scanner/ARCHITECTURE.scanner-data-flow.md"
+  - "documentation/dev/apps/scanner/ARCHITECTURE.host-deduplication.md"
+dependencies: []
+llm_context: "high"
+search_keywords:
+  - "sub-scan"
+  - "sub_scans"
+  - "scan progress"
+  - "network agent parallel"
+  - "scan cancellation"
+  - "SubScan"
+---
+
+# Sub-Scans Architecture
+
+The Sirius scanner uses a **sub-scan architecture**: each scan method (e.g. network, agent) is an independent **sub-scan** with its own status and progress. Sub-scans run in parallel and are tracked in a single `ScanResult` via a registry map.
+
+## What Are Sub-Scans?
+
+**Sub-scans** are independent scan methods that contribute to one logical “scan”:
+
+- **network** – Traditional network scanning (e.g. Nmap, RustScan, Naabu) triggered via RabbitMQ and executed by app-scanner.
+- **agent** – Agent-based scanning: templates run on remote hosts via app-agent over gRPC.
+
+A scan can have zero, one, or both enabled. Each sub-scan has its own:
+
+- Lifecycle state (e.g. dispatching → running → completed/failed/cancelled)
+- Progress (e.g. completed/total hosts or agents)
+- Optional metadata (e.g. agent list, per-agent status for agent sub-scan)
+
+The overall scan state is the union of all sub-scan states and their results (hosts/vulnerabilities are merged at the scan level).
+
+---
+
+## Sub-Scan Data Structure
+
+### Go (go-api / shared types)
+
+```go
+// SubScanProgress tracks completion progress for a sub-scan.
+type SubScanProgress struct {
+    Completed int    `json:"completed"`
+    Total     int    `json:"total"`
+    Label     string `json:"label,omitempty"` // e.g. "hosts", "agents"
+}
+
+// SubScan represents a modular scanner contribution to a scan.
+// Metadata is json.RawMessage so other scanners preserve it on read-modify-write.
+type SubScan struct {
+    Type     string          `json:"type"`     // "network", "agent"
+    Enabled  bool            `json:"enabled"`
+    Status   string          `json:"status"`   // see Lifecycle states
+    Progress SubScanProgress `json:"progress"`
+    Metadata json.RawMessage `json:"metadata,omitempty"`
+}
+
+// ScanResult holds the top-level scan and a registry of sub-scans.
+type ScanResult struct {
+    ID              string
+    Status          string
+    Targets         []string
+    Hosts           []HostEntry
+    HostsCompleted  int
+    Vulnerabilities []VulnerabilitySummary
+    StartTime       string
+    EndTime         string
+    SubScans        map[string]SubScan `json:"sub_scans,omitempty"`
+}
+```
+
+### TypeScript (sirius-ui)
+
+```ts
+interface SubScanProgress {
+  completed: number;
+  total: number;
+  label?: string;
+}
+
+interface SubScan {
+  type: string;
+  enabled: boolean;
+  status: "pending" | "dispatching" | "running" | "completed" | "failed";
+  progress: SubScanProgress;
+  metadata?: Record<string, unknown>;
+}
+
+// ScanResult.sub_scans: Record<string, SubScan>
+```
+
+Agent-specific metadata in `SubScan.metadata` can include `mode`, `dispatched_agents`, `agent_statuses` (per-agent status, hosts/vulns found, etc.).
+
+---
+
+## Sub-Scan Registry Pattern
+
+Sub-scans are stored in a **map keyed by scanner identifier**:
+
+- `map[string]SubScan` in Go (`sub_scans` in JSON)
+- `Record<string, SubScan>` in TypeScript
+
+Common keys:
+
+- `"network"` – Network scan (app-scanner).
+- `"agent"` – Agent scan (app-agent).
+
+Only enabled methods are present. When starting a scan, the UI (or API) builds this map (e.g. `subScans["network"]`, `subScans["agent"]`) and writes the initial `ScanResult` to ValKey. Consumers (app-scanner, go-api/agent path) update only their own key when merging back into `currentScan`, preserving other keys and `metadata` they don’t understand.
+
+---
+
+## Lifecycle States
+
+Each sub-scan moves through a small set of states:
+
+| State         | Meaning |
+|---------------|--------|
+| `pending`     | Not yet started (optional; may go straight to dispatching). |
+| `dispatching` | Work is being dispatched (e.g. message to RabbitMQ, or gRPC dispatch to agents). |
+| `running`     | Actively running (targets being scanned, agents executing templates). |
+| `completed`   | Finished successfully. |
+| `failed`      | Finished with error. |
+| `cancelled`   | Stopped by user or system. |
+
+The **overall scan** `status` is typically derived from sub-scans (e.g. “running” if any sub-scan is dispatching or running, “completed” when all are completed/failed/cancelled).
+
+---
+
+## Progress Aggregation
+
+- **Per sub-scan:** `progress.completed`, `progress.total`, and optional `progress.label` (e.g. "hosts", "agents"). Each scanner updates its own sub-scan’s progress when writing to ValKey.
+- **Overall scan:** The UI (or API) can compute an aggregate, for example:
+  - Total progress = sum of `completed` across sub-scans, divided by sum of `total` (or 0 if no total).
+  - Or: “running” if any sub-scan has `status === "running"` or `"dispatching"`, and overall completion when all sub-scans are in a terminal state.
+
+The UI displays per–sub-scan progress (e.g. in `ScanStatus`) and can show an overall progress bar or status from these fields.
+
+---
+
+## Cancellation Handling
+
+When the user cancels a scan:
+
+1. UI/API calls the cancel endpoint (e.g. `POST /api/v1/scans/cancel` with optional `scan_id`).
+2. go-api (or the component that owns the scan) sets the overall scan and/or sub-scans to a terminal state (e.g. `cancelled`) and writes the updated `ScanResult` back to ValKey.
+3. **Network:** Cancellation can be implemented by app-scanner checking a shared “cancelled” flag (e.g. from ValKey or a separate key) or by receiving a cancel message, then stopping workers and updating its sub-scan status to `cancelled` in `currentScan`.
+4. **Agent:** go-api (or agent coordinator) can signal agents to stop and then set `sub_scans.agent.status` to `cancelled` when updating ValKey.
+
+So: **cancelling the scan** means ensuring all sub-scans are stopped and their status (and optionally the overall scan status) is set to `cancelled` in the same `currentScan` document.
+
+---
+
+## Summary
+
+- **Sub-scans** = independent scan methods (network, agent) with their own status and progress.
+- **Data structure:** `SubScan { type, enabled, status, progress, metadata }` in a `map[string]SubScan` (`sub_scans`).
+- **Lifecycle:** dispatching → running → completed | failed | cancelled.
+- **Progress:** per–sub-scan `completed`/`total`; overall progress can be aggregated from all sub-scans.
+- **Cancellation:** cancel request stops all sub-scans and updates their status (and overall scan) in ValKey `currentScan`.
+- **Registry:** Use a single `sub_scans` map; each scanner only updates its own key and preserves others’ `metadata`.
+
+---
+
+**Related Documentation**
+
+- [ARCHITECTURE.scanner-data-flow.md](./ARCHITECTURE.scanner-data-flow.md) – How sub-scan results flow into ValKey and to the UI
+- [ARCHITECTURE.host-deduplication.md](./ARCHITECTURE.host-deduplication.md) – How hosts from different sub-scans are merged by IP
+
+---
+
+**Last Updated:** 2025-02-07  
+**Version:** 1.0.0  
+**Maintainer:** Sirius Team