chore: import upstream snapshot with attribution

2026-07-13 12:32:25 +08:00
commit 161ef94b4f
708 changed files with 189571 additions and 0 deletions
@@ -0,0 +1,512 @@
+---
+title: "Agent Template API Documentation"
+description: "API endpoints for managing agent vulnerability detection templates"
+template: "TEMPLATE.documentation-standard"
+llm_context: "high"
+categories: ["api", "agent", "templates"]
+tags: ["api", "agent", "templates", "vulnerability", "detection"]
+related_docs:
+  - "README.agent-template-ui.md"
+  - "ABOUT.documentation.md"
+search_keywords:
+  ["agent template api", "template endpoints", "vulnerability detection"]
+---
+
+# Agent Template API Documentation
+
+## Overview
+
+The Agent Template API provides endpoints for managing vulnerability detection templates used by Sirius agents. These templates define detection logic for identifying vulnerabilities on target systems.
+
+## Base URL
+
+```
+http://localhost:9001/api
+```
+
+## Endpoints
+
+### List All Templates
+
+Get all agent templates (standard + custom).
+
+**Endpoint:** `GET /agent-templates`
+
+**Response:**
+
+```json
+[
+  {
+    "id": "CVE-2021-44228",
+    "name": "Log4Shell Detection",
+    "description": "Detects Log4j RCE vulnerability",
+    "type": "standard",
+    "severity": "critical",
+    "author": "Sirius Security Team",
+    "platforms": ["linux", "darwin", "windows"],
+    "version": "1.0.0",
+    "createdAt": "2024-01-15T10:00:00Z",
+    "updatedAt": "2024-01-15T10:00:00Z",
+    "source": {
+      "type": "standard",
+      "name": "sirius-templates",
+      "priority": 1
+    }
+  }
+]
+```
+
+---
+
+### Get Single Template
+
+Get a specific template by ID, including its full YAML content.
+
+**Endpoint:** `GET /agent-templates/:id`
+
+**Parameters:**
+
+- `id` (path): Template ID
+
+**Response:**
+
+```json
+{
+  "id": "CVE-2021-44228",
+  "name": "Log4Shell Detection",
+  "description": "Detects Log4j RCE vulnerability",
+  "type": "standard",
+  "severity": "critical",
+  "author": "Sirius Security Team",
+  "platforms": ["linux"],
+  "version": "1.0.0",
+  "createdAt": "2024-01-15T10:00:00Z",
+  "updatedAt": "2024-01-15T10:00:00Z",
+  "content": "id: CVE-2021-44228\ninfo:\n  name: Log4Shell Detection\n..."
+}
+```
+
+**Error Responses:**
+
+- `404 Not Found`: Template not found
+
+---
+
+### Upload Custom Template
+
+Upload a new custom template from a YAML file.
+
+**Endpoint:** `POST /agent-templates`
+
+**Content-Type:** `multipart/form-data`
+
+**Form Fields:**
+
+- `file` (file, required): YAML template file (.yaml or .yml)
+- `author` (string, optional): Override author name
+
+**Response:**
+
+```json
+{
+  "id": "custom-template-123",
+  "message": "Template uploaded successfully and pushed to agents",
+  "validation": {
+    "valid": true,
+    "errors": [],
+    "warnings": ["Author field is recommended"]
+  }
+}
+```
+
+**Error Responses:**
+
+- `400 Bad Request`: Invalid file or validation failed
+  ```json
+  {
+    "error": "Template validation failed",
+    "validation": {
+      "valid": false,
+      "errors": ["Missing required field: info.severity"],
+      "warnings": []
+    }
+  }
+  ```
+
+**Validation Rules:**
+
+- File must be .yaml or .yml extension
+- File size must be under 1MB
+- Must include required fields: `id`, `info.name`, `info.severity`, `info.description`
+- Severity must be one of: `critical`, `high`, `medium`, `low`, `info`
+
+---
+
+### Validate Template
+
+Validate a template without saving it.
+
+**Endpoint:** `POST /agent-templates/validate`
+
+**Content-Type:** `application/json`
+
+**Request Body:**
+
+```json
+{
+  "content": "id: test-template\ninfo:\n  name: Test Template\n..."
+}
+```
+
+**Response:**
+
+```json
+{
+  "valid": true,
+  "errors": [],
+  "warnings": ["Tags are recommended for better organization"]
+}
+```
+
+---
+
+### Update Custom Template
+
+Update an existing custom template.
+
+**Endpoint:** `PUT /agent-templates/:id`
+
+**Parameters:**
+
+- `id` (path): Template ID
+
+**Content-Type:** `application/json`
+
+**Request Body:**
+
+```json
+{
+  "content": "id: test-template\ninfo:\n  name: Updated Test Template\n..."
+}
+```
+
+**Response:**
+
+```json
+{
+  "message": "Template updated and pushed to agents"
+}
+```
+
+**Error Responses:**
+
+- `400 Bad Request`: Validation failed
+- `404 Not Found`: Template not found
+- `403 Forbidden`: Cannot update standard templates
+
+---
+
+### Delete Custom Template
+
+Delete a custom template.
+
+**Endpoint:** `DELETE /agent-templates/:id`
+
+**Parameters:**
+
+- `id` (path): Template ID
+
+**Response:**
+
+```json
+{
+  "message": "Template deleted from agents"
+}
+```
+
+**Error Responses:**
+
+- `404 Not Found`: Template not found
+- `403 Forbidden`: Cannot delete standard templates
+
+---
+
+### Test Template on Agent
+
+Execute a template on a specific agent and get results.
+
+**Endpoint:** `POST /agent-templates/:id/test`
+
+**Parameters:**
+
+- `id` (path): Template ID
+
+**Content-Type:** `application/json`
+
+**Request Body:**
+
+```json
+{
+  "agent_id": "agent-123"
+}
+```
+
+**Response:**
+
+```json
+{
+  "message": "Template test initiated on agent",
+  "agent_id": "agent-123",
+  "template_id": "CVE-2021-44228"
+}
+```
+
+**Note:** This endpoint initiates the test. Results are retrieved asynchronously via agent event logs or response queues.
+
+---
+
+### Deploy Template to Agents
+
+Deploy a template to specific agents or all agents.
+
+**Endpoint:** `POST /agent-templates/:id/deploy`
+
+**Parameters:**
+
+- `id` (path): Template ID
+
+**Content-Type:** `application/json`
+
+**Request Body:**
+
+```json
+{
+  "agent_ids": ["agent-123", "agent-456"]
+}
+```
+
+**Note:** Empty `agent_ids` array deploys to all agents.
+
+**Response:**
+
+```json
+{
+  "message": "Template deployed to agents",
+  "agent_count": 2
+}
+```
+
+---
+
+### Get Template Analytics
+
+Get template effectiveness statistics.
+
+**Endpoint:** `GET /agent-templates/analytics`
+
+**Response:**
+
+```json
+{
+  "top_templates": [
+    {
+      "template_id": "CVE-2021-44228",
+      "template_name": "Log4Shell Detection",
+      "detection_count": 42,
+      "execution_count": 150,
+      "success_rate": 0.95,
+      "average_execution_time_ms": 245
+    }
+  ],
+  "execution_stats": {
+    "total_executions": 1250,
+    "total_detections": 105,
+    "average_execution_time_ms": 195,
+    "success_rate": 0.91
+  },
+  "platform_distribution": {
+    "linux": 850,
+    "darwin": 250,
+    "windows": 150
+  }
+}
+```
+
+**Note:** Analytics are aggregated from agent event logs.
+
+---
+
+### Get Template Results History
+
+Get historical execution results for a template.
+
+**Endpoint:** `GET /agent-templates/:id/results`
+
+**Parameters:**
+
+- `id` (path): Template ID
+
+**Response:**
+
+```json
+{
+  "template_id": "CVE-2021-44228",
+  "results": [
+    {
+      "agent_id": "agent-123",
+      "timestamp": "2024-01-15T10:30:00Z",
+      "vulnerable": true,
+      "confidence": 0.95,
+      "execution_time_ms": 245
+    }
+  ]
+}
+```
+
+---
+
+## Template YAML Schema
+
+### Required Fields
+
+```yaml
+id: unique-template-id
+info:
+  name: Human-readable template name
+  severity: critical|high|medium|low|info
+  description: What this template detects
+detection:
+  steps:
+    - type: file_hash|file_content|command_version|script
+      config: {}
+```
+
+### Optional Fields
+
+```yaml
+info:
+  author: Author name
+  version: Template version (e.g., 1.0.0)
+  references:
+    - https://example.com/vuln-info
+  cve:
+    - CVE-2021-XXXXX
+  tags:
+    - apache
+    - rce
+detection:
+  logic: all|any # Default: all
+  steps:
+    - platforms:
+        - linux
+        - darwin
+      weight: 0.8 # 0.0-1.0, affects confidence
+```
+
+### Detection Step Types
+
+#### File Hash
+
+```yaml
+- type: file_hash
+  config:
+    path: /usr/bin/vulnerable-binary
+    hash: abc123def456...
+    algorithm: sha256 # sha256, sha1, md5, sha512
+```
+
+#### File Content
+
+```yaml
+- type: file_content
+  config:
+    path: /etc/apache2/apache2.conf
+    regex: "ServerTokens\\s+Full"
+```
+
+#### Command Version
+
+```yaml
+- type: command_version
+  config:
+    command: ["dpkg-query", "-W", "-f='${Version}'", "openssh-server"]
+    regex: "^6\\.5\\.1"
+    exit_code: 0 # optional
+```
+
+#### Script
+
+```yaml
+- type: script
+  config:
+    interpreter: bash|python|powershell
+    script: |
+      #!/bin/bash
+      dpkg-query -W -f='${Version}' openssh-server
+    regex: "^6\\.5\\.1"
+    exit_code: 0 # optional
+```
+
+---
+
+## Error Codes
+
+| Code | Description                                                            |
+| ---- | ---------------------------------------------------------------------- |
+| 400  | Bad Request - Invalid input or validation failed                       |
+| 403  | Forbidden - Operation not allowed (e.g., modifying standard templates) |
+| 404  | Not Found - Template does not exist                                    |
+| 500  | Internal Server Error - Server-side error occurred                     |
+
+---
+
+## Rate Limiting
+
+Currently, no rate limiting is applied. This may change in future versions.
+
+---
+
+## Authentication
+
+Currently, no authentication is required. All authenticated users can manage templates. Role-based access control will be added in a future version.
+
+---
+
+## Examples
+
+### Upload a Custom Template
+
+```bash
+curl -X POST http://localhost:9001/api/agent-templates \
+  -F "file=@my-template.yaml" \
+  -F "author=Security Team"
+```
+
+### Get All Templates
+
+```bash
+curl http://localhost:9001/api/agent-templates
+```
+
+### Delete a Custom Template
+
+```bash
+curl -X DELETE http://localhost:9001/api/agent-templates/my-custom-template
+```
+
+### Test Template on Agent
+
+```bash
+curl -X POST http://localhost:9001/api/agent-templates/CVE-2021-44228/test \
+  -H "Content-Type: application/json" \
+  -d '{"agent_id": "agent-123"}'
+```
+
+---
+
+## See Also
+
+- [Agent Template UI Documentation](README.agent-template-ui.md)
+- [Agent Template Types Definition](../../app-agent/internal/template/types/types.go)
+- [Template System Notes](../../app-agent/project/BRAINSTORM.template-system-notes.md)
+
@@ -0,0 +1,480 @@
+---
+title: "Agent Template UI Documentation"
+description: "User interface workflows for managing agent vulnerability detection templates"
+template: "TEMPLATE.documentation-standard"
+llm_context: "high"
+categories: ["ui", "agent", "templates"]
+tags: ["ui", "agent", "templates", "vulnerability", "detection", "scanner"]
+related_docs:
+  - "README.agent-template-api.md"
+  - "ABOUT.documentation.md"
+search_keywords: ["agent template ui", "template management", "scanner ui"]
+---
+
+# Agent Template UI Documentation
+
+## Overview
+
+The Agent Template UI provides a comprehensive interface for managing vulnerability detection templates within the Sirius Scanner. Users can browse, upload, test, and analyze templates through an intuitive web interface.
+
+## Accessing Template Management
+
+**Navigation:** Scanner → Advanced → Agent → Templates Tab
+
+The Agent Templates interface is located within the Scanner's Advanced configuration section, under the Agent settings.
+
+---
+
+## Interface Components
+
+### 1. Template Browser
+
+The main view for browsing and managing templates.
+
+#### Features
+
+- **Search**: Search templates by name, ID, description, or author
+- **Filters**: Filter by type, severity, platform, and source
+- **Template Cards**: Visual cards displaying template metadata
+- **Actions**: View, edit (custom only), delete (custom only), test, and deploy
+
+#### Template Card Information
+
+Each template card displays:
+
+- Template name and ID
+- Description (truncated to 2 lines)
+- Severity badge (color-coded)
+- Type badge (standard/custom)
+- Source information
+- Supported platforms
+- Author and version
+
+#### Action Buttons
+
+- **View**: Display full template details and YAML content
+- **Edit**: Modify custom templates (not available for standard templates)
+- **Delete**: Remove custom templates (not available for standard templates)
+- **Test**: Execute template on a selected agent
+
+---
+
+### 2. Template Uploader
+
+Upload custom vulnerability detection templates.
+
+#### Upload Process
+
+1. Click "Create Template" or "Upload Template" button
+2. Drag and drop a YAML file or click to browse
+3. File is automatically validated
+4. Review validation results (errors and warnings)
+5. Preview template content
+6. Optionally override author field
+7. Click "Upload Template" to save
+
+#### File Requirements
+
+- File format: `.yaml` or `.yml`
+- Maximum size: 1MB
+- Must include required YAML fields
+- Must pass validation checks
+
+#### Validation
+
+**Required Fields:**
+
+- `id`: Unique template identifier
+- `info.name`: Human-readable template name
+- `info.severity`: Severity level (critical/high/medium/low/info)
+- `info.description`: Description of what the template detects
+
+**Optional but Recommended:**
+
+- `info.author`: Template author
+- `info.tags`: Tags for organization
+- `info.version`: Template version
+- `info.references`: Links to vulnerability information
+- `info.cve`: CVE identifiers
+
+**Validation Feedback:**
+
+- ✓ **Errors** (red): Must be fixed before upload
+- ⚠ **Warnings** (yellow): Recommended improvements, not blocking
+
+---
+
+### 3. Template Tester
+
+Test templates on live agents to verify detection logic.
+
+#### Testing Process
+
+1. Select a template to test
+2. Choose a target agent from the list
+3. Click "Run Test" button
+4. View real-time execution progress
+5. Review detailed test results
+
+#### Agent Selection
+
+**Filters:**
+
+- Status: Online/Offline (only online agents can be selected)
+- Platform: Filter by operating system
+- Search: Find agents by hostname or IP
+
+**Agent Information Displayed:**
+
+- Hostname
+- IP address
+- Platform (Linux/macOS/Windows)
+- Status (online/offline)
+- Last seen timestamp
+- Agent version
+
+#### Test Results
+
+**Summary Metrics:**
+
+- **Vulnerability Status**: Vulnerable or Not Vulnerable
+- **Confidence Score**: 0-100% (color-coded: green/yellow/orange/red)
+- **Execution Time**: Template execution duration in milliseconds
+- **Steps Completed**: Number of detection steps executed
+
+**Detailed Results:**
+
+- Evidence collected during detection
+- Step-by-step execution results
+- Individual step confidence scores
+- Error messages (if any)
+
+**Step Details (expandable):**
+
+- Step type (file_hash, file_content, command_version, script)
+- Match status (matched/not matched)
+- Evidence data
+- Error information
+
+---
+
+### 4. Template Analytics
+
+View template effectiveness and execution statistics.
+
+#### Summary Cards
+
+- **Total Executions**: Total number of template runs across all agents
+- **Vulnerabilities Found**: Total detections
+- **Success Rate**: Percentage of successful executions
+- **Average Execution Time**: Mean template execution time
+
+#### Top Performing Templates
+
+Ranked list of templates by detection count, showing:
+
+- Template rank (top 5)
+- Template name and ID
+- Detection count
+- Execution count
+- Success rate percentage
+- Average execution time
+- Visual performance bar
+
+#### Platform Distribution
+
+Breakdown of template executions by platform:
+
+- Linux
+- macOS (Darwin)
+- Windows
+
+Displayed as:
+
+- Platform icon
+- Execution count
+- Percentage of total
+- Visual progress bar
+
+---
+
+## User Workflows
+
+### Workflow 1: Uploading a Custom Template
+
+1. Navigate to Scanner → Advanced → Agent → Templates
+2. Ensure "Enable Agent Templates" is toggled ON
+3. Click "Upload Template" button
+4. Drag and drop your YAML file or click to browse
+5. Wait for automatic validation
+6. Review validation results:
+   - If errors exist, fix your template and re-upload
+   - Warnings are optional improvements
+7. Review the template preview
+8. (Optional) Override the author field
+9. Click "Upload Template"
+10. Template is saved and automatically distributed to all agents
+11. Return to template browser to see your new template
+
+**Success Indicators:**
+
+- Green checkmark in validation
+- Template appears in browser with "custom" badge
+- Confirmation message displayed
+
+---
+
+### Workflow 2: Testing a Template
+
+1. From the template browser, click the test icon on any template
+2. Or click "Test Template" and select a template from dropdown
+3. Review template information (description, platforms, etc.)
+4. Scroll to "Select Target Agent" section
+5. Use filters to find appropriate agents:
+   - Filter by platform if template is platform-specific
+   - Ensure agent is online (green status badge)
+6. Click on an agent card to select it
+7. Click "Run Test" button
+8. Wait for test execution (typically 1-5 seconds)
+9. Review results:
+   - Check vulnerability status
+   - Review confidence score
+   - Examine evidence
+   - Expand steps for detailed information
+10. Test additional agents or return to browser
+
+**Best Practices:**
+
+- Test on multiple agents to verify detection logic
+- Test on different platforms if template supports multiple
+- Review confidence scores to validate template accuracy
+
+---
+
+### Workflow 3: Managing Templates
+
+#### Viewing Template Details
+
+1. Click "View" button on any template card
+2. Review full template information:
+   - Complete metadata
+   - Full YAML content (with syntax highlighting)
+3. Use "Test Template" to verify functionality
+4. Use "Back to Templates" to return
+
+#### Deleting Custom Templates
+
+1. Locate the custom template in browser
+2. Click the trash icon (red)
+3. Confirm deletion in dialog
+4. Template is removed from all agents immediately
+
+**Note:** Standard templates cannot be deleted.
+
+---
+
+### Workflow 4: Monitoring Template Effectiveness
+
+1. Click "Analytics" button in template browser
+2. Review summary metrics at the top
+3. Check "Top Performing Templates":
+   - Identify which templates find the most vulnerabilities
+   - Review success rates
+   - Identify slow-running templates
+4. Review "Platform Distribution":
+   - Understand which platforms are scanned most
+   - Identify coverage gaps
+5. Use insights to:
+   - Prioritize template improvements
+   - Add templates for underrepresented platforms
+   - Remove ineffective templates
+
+---
+
+## Settings
+
+### Enable Agent Templates
+
+Toggle agent template scanning on or off for all scans.
+
+**Location:** Templates tab → Settings card
+
+**Effect:** When disabled, agents will not execute vulnerability detection templates during scans.
+
+### Template Priority
+
+Control which templates are used based on severity level.
+
+**Options:**
+
+- **High**: Only critical and high severity templates
+- **Medium**: Include medium severity templates
+- **All**: Use all templates regardless of severity
+
+**Use Cases:**
+
+- **High**: Fast scans focusing on critical vulnerabilities
+- **Medium**: Balanced scan coverage
+- **All**: Comprehensive vulnerability assessment
+
+---
+
+## Template Card Color Coding
+
+### Severity Colors
+
+- **Critical**: Red (bg-red-500/20, text-red-400)
+- **High**: Orange (bg-orange-500/20, text-orange-400)
+- **Medium**: Yellow (bg-yellow-500/20, text-yellow-400)
+- **Low**: Blue (bg-blue-500/20, text-blue-400)
+- **Info**: Gray (bg-gray-500/20, text-gray-400)
+
+### Type Colors
+
+- **Standard**: Green (bg-green-500/20, text-green-400)
+- **Custom**: Violet (bg-violet-500/20, text-violet-400)
+- **Repository**: Blue (bg-blue-500/20, text-blue-400)
+
+### Status Colors (Agents)
+
+- **Online**: Green
+- **Offline**: Gray
+
+---
+
+## Keyboard Shortcuts
+
+Currently, no keyboard shortcuts are implemented. This may be added in a future version.
+
+---
+
+## Troubleshooting
+
+### Template Upload Fails
+
+**Symptoms:** Validation errors prevent upload
+
+**Solutions:**
+
+1. Review error messages carefully
+2. Check required fields are present
+3. Verify severity value is valid (critical/high/medium/low/info)
+4. Ensure YAML syntax is correct
+5. Check file size is under 1MB
+6. Verify file extension is .yaml or .yml
+
+### Template Test Hangs
+
+**Symptoms:** Test never completes or times out
+
+**Possible Causes:**
+
+- Agent is offline or unreachable
+- Template contains infinite loop or long-running command
+- Network connectivity issues
+
+**Solutions:**
+
+1. Verify agent is online in agent list
+2. Test a different template on the same agent
+3. Test the same template on a different agent
+4. Check agent logs for errors
+5. Review template detection logic for long-running operations
+
+### Template Not Appearing
+
+**Symptoms:** Uploaded template doesn't appear in browser
+
+**Solutions:**
+
+1. Refresh the page
+2. Clear any active filters
+3. Check template was successfully uploaded (confirmation message)
+4. Verify template wasn't deleted by another user
+5. Check browser console for JavaScript errors
+
+### Agents Not Listed in Test View
+
+**Symptoms:** No agents available for testing
+
+**Possible Causes:**
+
+- No agents are currently online
+- Platform filter is too restrictive
+- Agents haven't registered yet
+
+**Solutions:**
+
+1. Check agent status in Terminal or Dashboard
+2. Clear platform filter to see all agents
+3. Wait for agents to come online
+4. Verify agent connectivity
+
+---
+
+## Performance Considerations
+
+### Template Browser
+
+- Displays up to 1000 templates efficiently
+- Filtering and search are client-side (instant)
+- Template cards use lazy loading for large lists
+
+### Template Testing
+
+- Test results are displayed in real-time
+- Step-by-step results are lazy-loaded
+- Large evidence data is formatted for readability
+
+### Analytics
+
+- Analytics data is cached for 5 minutes
+- Refresh manually to get latest statistics
+- Chart rendering is optimized for large datasets
+
+---
+
+## Security Notes
+
+### Template Upload
+
+- All templates are validated before upload
+- Malicious YAML is rejected
+- File size limits prevent DoS attacks
+- Templates are sandboxed during execution
+
+### Template Testing
+
+- Tests only execute on selected agents
+- Test results are not persisted
+- Agent permissions are enforced
+
+### Template Deletion
+
+- Only custom templates can be deleted
+- Deletion is immediate and cannot be undone
+- All agents are notified of deletion
+
+---
+
+## Future Enhancements
+
+Planned features for future releases:
+
+1. **Template Editor**: Visual form-based template builder
+2. **Template Versioning**: Track and rollback template changes
+3. **Template Collections**: Bundle related templates
+4. **Template Marketplace**: Share templates with community
+5. **Scheduled Scans**: Automatic periodic template execution
+6. **Advanced Analytics**: Detection trends over time
+7. **Role-Based Access**: Control who can upload/edit templates
+8. **Bulk Operations**: Deploy/test multiple templates at once
+
+---
+
+## See Also
+
+- [Agent Template API Documentation](README.agent-template-api.md)
+- [Template System Notes](../../app-agent/project/BRAINSTORM.template-system-notes.md)
+- [Scanner Advanced Configuration](../ui/README.scanner-advanced.md)
+
@@ -0,0 +1,162 @@
+---
+title: "Host Deduplication and Multi-Source Attribution"
+description: "Canonical host identity by IP, multi-source attribution (network/agent), scan_sources field, and UI merge/display with SourceIcon and SourceIconRow."
+template: "TEMPLATE.custom"
+version: "1.0.0"
+last_updated: "2025-02-07"
+author: "Sirius Team"
+tags: ["scanner", "host", "deduplication", "scan_sources", "SourceIcon", "multi-source"]
+categories: ["architecture", "scanner"]
+difficulty: "intermediate"
+prerequisites: ["README.scanner.md", "ARCHITECTURE.sub-scans.md"]
+related_docs:
+  - "documentation/dev/apps/scanner/README.scanner.md"
+  - "documentation/dev/apps/scanner/ARCHITECTURE.scanner-data-flow.md"
+  - "documentation/dev/apps/scanner/ARCHITECTURE.sub-scans.md"
+dependencies: []
+llm_context: "high"
+search_keywords:
+  - "host deduplication"
+  - "scan_sources"
+  - "multi-source"
+  - "SourceIcon"
+  - "SourceIconRow"
+  - "EnvironmentTableData"
+  - "canonical host IP"
+---
+
+# Host Deduplication and Multi-Source Attribution
+
+Hosts in the Sirius scanner can be discovered by multiple scan methods (network and agent). This document describes how hosts are identified by a canonical key, how multiple sources are attributed, and how the UI merges and displays them.
+
+## Canonical Host Identity: IP Address
+
+The **primary key** for a host is its **IP address**. Regardless of whether a host was found by a network scan, an agent scan, or both, it is represented once per IP in the merged view.
+
+- **Backend:** `HostEntry` (and persisted host records) use `ip` as the canonical identifier; `id` can be a unique row or document id; `hostname`, `aliases`, and `sources` are optional.
+- **UI:** Tables and detail views key hosts by `ip` when merging data from different sub-scans or from the environment summary API.
+
+This allows:
+
+- One row per host in the environment/host table
+- Correct aggregation of vulnerability counts and source badges per host
+- Stable navigation to host detail (e.g. by IP) regardless of which scan discovered it
+
+---
+
+## Multi-Source Attribution
+
+A host can be discovered by:
+
+- **network** – app-scanner (Nmap, RustScan, etc.)
+- **agent** – app-agent (template runs on remote host)
+
+Other sources (e.g. cloud, application) can be added later. Each discovery path that reports a host should tag it with its **source identifier** (e.g. `"network"`, `"agent"`).
+
+### HostEntry.sources (backend / ValKey)
+
+In the live `ScanResult` stored in ValKey, each `HostEntry` can carry a list of sources:
+
+```go
+type HostEntry struct {
+    ID       string   `json:"id"`
+    IP       string   `json:"ip"`
+    Hostname string   `json:"hostname,omitempty"`
+    Aliases  []string `json:"aliases,omitempty"`
+    Sources  []string `json:"sources,omitempty"` // e.g. ["network", "agent"]
+}
+```
+
+When merging results from multiple sub-scans, the component that writes to `currentScan` should:
+
+- Merge hosts by IP (one entry per IP)
+- Set `sources` to the union of all sources that reported that IP (e.g. if both network and agent found it, `sources = ["network", "agent"]`)
+
+---
+
+## EnvironmentTableData.scan_sources (UI)
+
+The UI uses **EnvironmentTableData** for the environment/host table. It includes:
+
+- **scan_source** (optional, legacy): Single source string (e.g. `"network"` or `"agent"`).
+- **scan_sources** (optional): Array of source strings for multi-source attribution.
+
+```ts
+interface EnvironmentTableData {
+  hostname: string;
+  ip: string;
+  os: string;
+  vulnerabilityCount: number;
+  maxCvss?: number;
+  groups: string[];
+  tags: string[];
+  scan_source?: ScanSource;   // legacy single source
+  scan_sources?: string[];    // all discovery sources for this host
+}
+```
+
+When mapping from API or from live scan results:
+
+- Prefer **scan_sources** (array) when building the table (e.g. from `host.sources` or equivalent).
+- Fall back to **scan_source** for older data or single-source responses, and convert to `scan_sources` for consistent display (e.g. `row.scan_sources ?? (row.scan_source ? [row.scan_source] : [])`).
+
+---
+
+## How the UI Merges Hosts from Different Sub-Scans
+
+1. **Live scan results (ValKey):**  
+   The `ScanResult.hosts` array may already be merged by the backend (app-scanner and agent path both updating the same `currentScan` and merging by IP with combined `sources`). The UI then maps `HostEntry[]` to `EnvironmentTableData[]` and sets `scan_sources = host.sources || []`.
+
+2. **Environment summary (API):**  
+   When the UI fetches the environment host list (e.g. `host.getEnvironmentSummary`), the API returns rows that may include a `sources` (or `scan_sources`) field per host. The UI maps that to `EnvironmentTableData.scan_sources`.
+
+3. **Deduplication by IP:**  
+   If the UI ever receives multiple rows for the same IP (e.g. from different endpoints), it should merge them into one row and combine `scan_sources` (union of all source arrays) so that the table shows one row per host with all sources that discovered it.
+
+---
+
+## Source Display: SourceIcon and SourceIconRow
+
+The UI provides two components for showing scan source(s):
+
+### SourceIcon (single source)
+
+- **Purpose:** Renders one scan source (e.g. `"network"` or `"agent"`) as an icon with optional label and tooltip.
+- **Usage:** `<SourceIcon source="agent" />`, `<SourceIcon source="network" showLabel />`.
+- **Registry:** `SOURCE_ICON_REGISTRY` maps source keys to icon, color, and label (e.g. `agent` → Bot icon, cyan; `network` → Wifi icon, violet).
+
+### SourceIconRow (multiple sources)
+
+- **Purpose:** Renders a row of icons for all sources that discovered a host (or finding).
+- **Usage:** `<SourceIconRow sources={["agent", "network"]} />`.
+- **Behavior:** Renders one `SourceIcon` per entry in `sources`; tooltip/title can show a combined label (e.g. "Agent + Network").
+
+**Example in host table column:**
+
+- Accessor: `row.scan_sources ?? (row.scan_source ? [row.scan_source] : [])`.
+- Cell: `<SourceIconRow sources={sources} />`.
+
+This gives users a clear view of which scan methods discovered each host (network-only, agent-only, or both).
+
+---
+
+## Summary
+
+- **Canonical host identity:** IP address.
+- **Multi-source attribution:** Hosts carry `sources` (backend) / `scan_sources` (UI) listing every scan method that discovered them.
+- **EnvironmentTableData:** Prefer `scan_sources: string[]`; support legacy `scan_source` by normalizing to an array.
+- **UI merge:** One row per IP; `scan_sources` = union of sources for that IP.
+- **Display:** Use **SourceIcon** for a single source and **SourceIconRow** for the list of sources on a host (or vulnerability) row.
+
+---
+
+**Related Documentation**
+
+- [ARCHITECTURE.scanner-data-flow.md](./ARCHITECTURE.scanner-data-flow.md) – How host data flows from scanners into ValKey and to the UI
+- [ARCHITECTURE.sub-scans.md](./ARCHITECTURE.sub-scans.md) – How network and agent sub-scans contribute hosts
+
+---
+
+**Last Updated:** 2025-02-07  
+**Version:** 1.0.0  
+**Maintainer:** Sirius Team
@@ -0,0 +1,156 @@
+---
+title: "Scanner Data Flow - End-to-End Architecture"
+description: "End-to-end data flow from UI scan initiation through network/agent scans to result display and persistence."
+template: "TEMPLATE.custom"
+version: "1.0.0"
+last_updated: "2025-02-07"
+author: "Sirius Team"
+tags: ["scanner", "data-flow", "architecture", "valkey", "rabbitmq", "grpc"]
+categories: ["architecture", "scanner"]
+difficulty: "intermediate"
+prerequisites: ["README.scanner.md"]
+related_docs:
+  - "documentation/dev/apps/scanner/README.scanner.md"
+  - "documentation/dev/apps/scanner/ARCHITECTURE.sub-scans.md"
+  - "documentation/dev/apps/scanner/ARCHITECTURE.host-deduplication.md"
+  - "documentation/dev/architecture/README.architecture.md"
+dependencies: []
+llm_context: "high"
+search_keywords:
+  - "scanner data flow"
+  - "scan initiation"
+  - "ValKey currentScan"
+  - "network scan flow"
+  - "agent scan flow"
+  - "scan persistence"
+---
+
+# Scanner Data Flow – End-to-End Architecture
+
+This document describes the end-to-end data flow of the Sirius scanner system from the moment a user starts a scan in the UI until results are displayed and optionally persisted.
+
+## Overview
+
+The scanner system uses a **single live scan state** stored in ValKey under the key `currentScan`. The UI initiates scans via tRPC, triggers one or more sub-scans (network and/or agent), and polls ValKey for progress. When scans complete, results can be persisted from ValKey through go-api into PostgreSQL.
+
+**Components involved:**
+
+- **sirius-ui** (Next.js): Initiates scans, polls for results, displays hosts and vulnerabilities
+- **go-api** (Go): REST API and persistence; can proxy queue/store or be called for cancel/persist
+- **app-scanner** (Go): Network scanning; consumes from RabbitMQ, updates ValKey
+- **app-agent** (Go): Agent-based scanning via gRPC; runs templates on remote hosts, results flow to ValKey
+- **ValKey**: Stores live scan result as `currentScan` (base64-encoded JSON)
+- **RabbitMQ**: Message queue between API/UI and app-scanner for network scan requests
+
+---
+
+## Network Scan Flow
+
+1. **UI** – User configures targets and scan options, clicks “Start Scan”.
+2. **tRPC** – UI calls `queue.sendMsg` (and optionally scanner/start endpoints) and `store.setValue` with key `currentScan` to write initial `ScanResult` (id, status, targets, empty hosts/vulnerabilities, `sub_scans.network`).
+3. **go-api / Queue** – Scan request (id, targets, options, priority) is published to the **RabbitMQ** `scan` queue (either via go-api or via tRPC router in sirius-ui that publishes to RabbitMQ).
+4. **app-scanner** – Consumes the message from the `scan` queue, expands targets to IPs, runs the scan pipeline (e.g. enumeration → discovery → vulnerability), and for each host/vulnerability discovered:
+   - Reads current `currentScan` from ValKey (or receives it in context),
+   - Merges new hosts/vulnerabilities and updates progress,
+   - Writes updated `ScanResult` back to ValKey key `currentScan`.
+5. **UI polling** – `useScanResults` polls `store.getValue({ key: "currentScan" })` (e.g. every 3s). The tRPC store router reads from ValKey and returns the value; UI decodes the base64 JSON and updates hosts/vulnerabilities for display.
+
+So: **UI → tRPC → go-api (or tRPC) → RabbitMQ → app-scanner → ValKey → UI polling (tRPC store.getValue → ValKey).**
+
+---
+
+## Agent Scan Flow
+
+1. **UI** – Same scan start; if agent scan is enabled, UI builds `sub_scans.agent` (status `dispatching`, progress 0/0) and writes initial `ScanResult` to ValKey via `store.setValue` with key `currentScan`.
+2. **tRPC** – UI calls `agentScan.dispatchAgentScan` with scan id and agent scan config.
+3. **go-api / gRPC** – go-api (or backend invoked by tRPC) uses **gRPC** to communicate with **app-agent** to dispatch template scans to connected agents.
+4. **app-agent** – Runs templates on remote hosts, collects results, and reports back (e.g. via gRPC or callback to API). The component that holds the “current scan” state (e.g. go-api or a worker) updates the `currentScan` value in **ValKey**: merges agent-discovered hosts and vulnerabilities, updates `sub_scans.agent` status and progress.
+5. **UI polling** – Same as network: UI polls `store.getValue("currentScan")`, decodes the result, and displays agent-origin hosts and vulnerabilities (with source attribution).
+
+So: **UI → tRPC → go-api → gRPC → app-agent → (results) → ValKey → UI polling.**
+
+---
+
+## Persistence: ValKey (Live) → go-api → PostgreSQL
+
+- **Live state** – The only source of truth during an active scan is ValKey key `currentScan`. All sub-scans (network, agent) read-modify-write this key to add hosts, vulnerabilities, and progress.
+- **Persistent state** – After a scan completes (or on demand), the API layer can:
+  - Read the final `ScanResult` from ValKey (`currentScan`),
+  - Map hosts and vulnerabilities to the persistent model,
+  - Write to **PostgreSQL** via go-api (hosts, vulnerabilities, scan_sources, etc.).
+
+So: **ValKey (live) → go-api → PostgreSQL (persistent).** The UI can show live data from ValKey and, for history/reporting, data from go-api/PostgreSQL.
+
+---
+
+## Sequence Diagram (Mermaid)
+
+```mermaid
+sequenceDiagram
+  participant UI as sirius-ui
+  participant tRPC as tRPC (Next.js)
+  participant API as go-api
+  participant RMQ as RabbitMQ
+  participant Scanner as app-scanner
+  participant Valkey as ValKey
+  participant gRPC as gRPC
+  participant Agent as app-agent
+  participant DB as PostgreSQL
+
+  UI->>tRPC: Start scan (targets, options)
+  tRPC->>Valkey: setValue("currentScan", initial ScanResult)
+  tRPC->>API: (optional) start / queue
+  API->>RMQ: Publish scan message (network)
+  tRPC->>API: dispatchAgentScan (if agent enabled)
+  API->>gRPC: Dispatch to agents
+  gRPC->>Agent: Run templates on hosts
+
+  loop Network scan
+    Scanner->>RMQ: Consume scan message
+    Scanner->>Scanner: Expand targets, run phases
+    Scanner->>Valkey: Get currentScan
+    Scanner->>Valkey: Set currentScan (merge hosts/vulns)
+  end
+
+  loop Agent scan
+    Agent->>Agent: Execute templates
+    Agent->>API: Report results (or via gRPC)
+    API->>Valkey: Get currentScan
+    API->>Valkey: Set currentScan (merge agent results)
+  end
+
+  loop UI polling (e.g. every 3s)
+    UI->>tRPC: getValue("currentScan")
+    tRPC->>Valkey: GET currentScan
+    Valkey-->>tRPC: base64 ScanResult
+    tRPC-->>UI: ScanResult
+    UI->>UI: Decode, update hosts/vulnerabilities
+  end
+
+  note over API,DB: On completion or on demand
+  API->>Valkey: GET currentScan
+  Valkey-->>API: ScanResult
+  API->>DB: Persist hosts, vulnerabilities, sources
+```
+
+---
+
+## Key Data Structures
+
+- **ValKey key:** `currentScan` (string value = base64-encoded JSON).
+- **ScanResult (JSON):** `id`, `status`, `targets`, `hosts` (array of `HostEntry` with `id`, `ip`, `hostname`, `sources`), `hosts_completed`, `vulnerabilities`, `start_time`, `end_time`, `sub_scans` (map of sub-scan key → `SubScan`).
+- **SubScan:** `type`, `enabled`, `status`, `progress` (`completed`, `total`, `label`), `metadata` (optional). Used for both network and agent sub-scans.
+
+---
+
+## Related Documentation
+
+- [ARCHITECTURE.sub-scans.md](./ARCHITECTURE.sub-scans.md) – Sub-scan lifecycle and progress aggregation
+- [ARCHITECTURE.host-deduplication.md](./ARCHITECTURE.host-deduplication.md) – How hosts from multiple sources are merged and displayed
+- [README.scanner.md](./README.scanner.md) – Scanner engine details (message format, strategies, ValKey usage)
+
+---
+
+**Last Updated:** 2025-02-07  
+**Version:** 1.0.0  
+**Maintainer:** Sirius Team
@@ -0,0 +1,185 @@
+---
+title: "Sub-Scans Architecture - Modular Scan Methods"
+description: "Independent scan methods (network, agent) running in parallel, their data structure, lifecycle, progress aggregation, and cancellation."
+template: "TEMPLATE.custom"
+version: "1.0.0"
+last_updated: "2025-02-07"
+author: "Sirius Team"
+tags: ["scanner", "sub-scan", "architecture", "progress", "cancellation"]
+categories: ["architecture", "scanner"]
+difficulty: "intermediate"
+prerequisites: ["README.scanner.md"]
+related_docs:
+  - "documentation/dev/apps/scanner/README.scanner.md"
+  - "documentation/dev/apps/scanner/ARCHITECTURE.scanner-data-flow.md"
+  - "documentation/dev/apps/scanner/ARCHITECTURE.host-deduplication.md"
+dependencies: []
+llm_context: "high"
+search_keywords:
+  - "sub-scan"
+  - "sub_scans"
+  - "scan progress"
+  - "network agent parallel"
+  - "scan cancellation"
+  - "SubScan"
+---
+
+# Sub-Scans Architecture
+
+The Sirius scanner uses a **sub-scan architecture**: each scan method (e.g. network, agent) is an independent **sub-scan** with its own status and progress. Sub-scans run in parallel and are tracked in a single `ScanResult` via a registry map.
+
+## What Are Sub-Scans?
+
+**Sub-scans** are independent scan methods that contribute to one logical “scan”:
+
+- **network** – Traditional network scanning (e.g. Nmap, RustScan, Naabu) triggered via RabbitMQ and executed by app-scanner.
+- **agent** – Agent-based scanning: templates run on remote hosts via app-agent over gRPC.
+
+A scan can have zero, one, or both enabled. Each sub-scan has its own:
+
+- Lifecycle state (e.g. dispatching → running → completed/failed/cancelled)
+- Progress (e.g. completed/total hosts or agents)
+- Optional metadata (e.g. agent list, per-agent status for agent sub-scan)
+
+The overall scan state is the union of all sub-scan states and their results (hosts/vulnerabilities are merged at the scan level).
+
+---
+
+## Sub-Scan Data Structure
+
+### Go (go-api / shared types)
+
+```go
+// SubScanProgress tracks completion progress for a sub-scan.
+type SubScanProgress struct {
+    Completed int    `json:"completed"`
+    Total     int    `json:"total"`
+    Label     string `json:"label,omitempty"` // e.g. "hosts", "agents"
+}
+
+// SubScan represents a modular scanner contribution to a scan.
+// Metadata is json.RawMessage so other scanners preserve it on read-modify-write.
+type SubScan struct {
+    Type     string          `json:"type"`     // "network", "agent"
+    Enabled  bool            `json:"enabled"`
+    Status   string          `json:"status"`   // see Lifecycle states
+    Progress SubScanProgress `json:"progress"`
+    Metadata json.RawMessage `json:"metadata,omitempty"`
+}
+
+// ScanResult holds the top-level scan and a registry of sub-scans.
+type ScanResult struct {
+    ID              string
+    Status          string
+    Targets         []string
+    Hosts           []HostEntry
+    HostsCompleted  int
+    Vulnerabilities []VulnerabilitySummary
+    StartTime       string
+    EndTime         string
+    SubScans        map[string]SubScan `json:"sub_scans,omitempty"`
+}
+```
+
+### TypeScript (sirius-ui)
+
+```ts
+interface SubScanProgress {
+  completed: number;
+  total: number;
+  label?: string;
+}
+
+interface SubScan {
+  type: string;
+  enabled: boolean;
+  status: "pending" | "dispatching" | "running" | "completed" | "failed";
+  progress: SubScanProgress;
+  metadata?: Record<string, unknown>;
+}
+
+// ScanResult.sub_scans: Record<string, SubScan>
+```
+
+Agent-specific metadata in `SubScan.metadata` can include `mode`, `dispatched_agents`, `agent_statuses` (per-agent status, hosts/vulns found, etc.).
+
+---
+
+## Sub-Scan Registry Pattern
+
+Sub-scans are stored in a **map keyed by scanner identifier**:
+
+- `map[string]SubScan` in Go (`sub_scans` in JSON)
+- `Record<string, SubScan>` in TypeScript
+
+Common keys:
+
+- `"network"` – Network scan (app-scanner).
+- `"agent"` – Agent scan (app-agent).
+
+Only enabled methods are present. When starting a scan, the UI (or API) builds this map (e.g. `subScans["network"]`, `subScans["agent"]`) and writes the initial `ScanResult` to ValKey. Consumers (app-scanner, go-api/agent path) update only their own key when merging back into `currentScan`, preserving other keys and `metadata` they don’t understand.
+
+---
+
+## Lifecycle States
+
+Each sub-scan moves through a small set of states:
+
+| State         | Meaning |
+|---------------|--------|
+| `pending`     | Not yet started (optional; may go straight to dispatching). |
+| `dispatching` | Work is being dispatched (e.g. message to RabbitMQ, or gRPC dispatch to agents). |
+| `running`     | Actively running (targets being scanned, agents executing templates). |
+| `completed`   | Finished successfully. |
+| `failed`      | Finished with error. |
+| `cancelled`   | Stopped by user or system. |
+
+The **overall scan** `status` is typically derived from sub-scans (e.g. “running” if any sub-scan is dispatching or running, “completed” when all are completed/failed/cancelled).
+
+---
+
+## Progress Aggregation
+
+- **Per sub-scan:** `progress.completed`, `progress.total`, and optional `progress.label` (e.g. "hosts", "agents"). Each scanner updates its own sub-scan’s progress when writing to ValKey.
+- **Overall scan:** The UI (or API) can compute an aggregate, for example:
+  - Total progress = sum of `completed` across sub-scans, divided by sum of `total` (or 0 if no total).
+  - Or: “running” if any sub-scan has `status === "running"` or `"dispatching"`, and overall completion when all sub-scans are in a terminal state.
+
+The UI displays per–sub-scan progress (e.g. in `ScanStatus`) and can show an overall progress bar or status from these fields.
+
+---
+
+## Cancellation Handling
+
+When the user cancels a scan:
+
+1. UI/API calls the cancel endpoint (e.g. `POST /api/v1/scans/cancel` with optional `scan_id`).
+2. go-api (or the component that owns the scan) sets the overall scan and/or sub-scans to a terminal state (e.g. `cancelled`) and writes the updated `ScanResult` back to ValKey.
+3. **Network:** Cancellation can be implemented by app-scanner checking a shared “cancelled” flag (e.g. from ValKey or a separate key) or by receiving a cancel message, then stopping workers and updating its sub-scan status to `cancelled` in `currentScan`.
+4. **Agent:** go-api (or agent coordinator) can signal agents to stop and then set `sub_scans.agent.status` to `cancelled` when updating ValKey.
+
+So: **cancelling the scan** means ensuring all sub-scans are stopped and their status (and optionally the overall scan status) is set to `cancelled` in the same `currentScan` document.
+
+---
+
+## Summary
+
+- **Sub-scans** = independent scan methods (network, agent) with their own status and progress.
+- **Data structure:** `SubScan { type, enabled, status, progress, metadata }` in a `map[string]SubScan` (`sub_scans`).
+- **Lifecycle:** dispatching → running → completed | failed | cancelled.
+- **Progress:** per–sub-scan `completed`/`total`; overall progress can be aggregated from all sub-scans.
+- **Cancellation:** cancel request stops all sub-scans and updates their status (and overall scan) in ValKey `currentScan`.
+- **Registry:** Use a single `sub_scans` map; each scanner only updates its own key and preserves others’ `metadata`.
+
+---
+
+**Related Documentation**
+
+- [ARCHITECTURE.scanner-data-flow.md](./ARCHITECTURE.scanner-data-flow.md) – How sub-scan results flow into ValKey and to the UI
+- [ARCHITECTURE.host-deduplication.md](./ARCHITECTURE.host-deduplication.md) – How hosts from different sub-scans are merged by IP
+
+---
+
+**Last Updated:** 2025-02-07  
+**Version:** 1.0.0  
+**Maintainer:** Sirius Team