siriusscan--sirius/documentation/dev/architecture/README.architecture.md

title, description, template, version, last_updated, author, tags, categories, difficulty, prerequisites, related_docs, dependencies, llm_context, search_keywords

title	description	template	version	last_updated	author	tags	categories	difficulty	prerequisites	related_docs	dependencies	llm_context	search_keywords
SiriusScan Architecture	Comprehensive architectural documentation for SiriusScan, covering system design, component relationships, data flow, and technical implementation details.	TEMPLATE.architecture	1.0.0	2025-01-03	Development Team	architecture system-design overview technical	architecture	advanced	docker microservices vulnerability-scanning	README.container-testing.md README.development.md		high	architecture system design microservices vulnerability scanner sirius

Sirius Scan Architecture Document

Table of Contents

A. Getting Started & Development Workflow - A.1. System Prerequisites - A.2. Repository Setup - A.3. Running the Full Stack - A.4. Common Development Tasks - A.5. Testing Strategy B. Key Architectural Workflows (End-to-End Examples) - B.1. User Initiates a Vulnerability Scan - B.2. Agent Interaction & Management - B.3. Terminal Command Execution

1. Introduction

   • 1.1. Purpose of Sirius Scan
   • 1.2. High-Level Architecture Overview

2. Core Infrastructure (Docker Compose)

   • 2.1. docker-compose.yaml Overview
   • 2.2. Service Orchestration
     • 2.2.1. sirius-rabbitmq (Message Broker)
     • 2.2.2. sirius-postgres (Relational Database)
     • 2.2.3. sirius-valkey (Key-Value Store)
     • 2.2.4. Networking

3. Application Services

   • 3.1. sirius-ui (User Interface & BFF)
     • 3.1.1. Overview & Purpose
     • 3.1.2. Technology Stack
     • 3.1.3. Docker Configuration (sirius-ui/Dockerfile)
     • 3.1.4. UI Theming (globals.css)
     • 3.1.5. Extensibility (Marketplace Concept)
     • 3.1.6. Communication
     • 3.1.7. Key Directory Structure
     • 3.1.8. Local Development & Debugging
     • 3.1.9. Environment Variables
   • 3.2. sirius-api (Backend API)
     • 3.2.1. Overview & Purpose
     • 3.2.2. Technology Stack
     • 3.2.3. Docker Configuration (sirius-api/Dockerfile)
     • 3.2.4. Key Functionalities (High-Level)
     • 3.2.5. Communication
     • 3.2.6. Key Directory Structure
     • 3.2.7. Local Development & Debugging
     • 3.2.8. Environment Variables
   • 3.3. sirius-engine (Core Processing Engine)
     • 3.3.1. Overview & Purpose
     • 3.3.2. Technology Stack
     • 3.3.3. Docker Configuration (sirius-engine/Dockerfile)
     • 3.3.4. Modular Design (Loading Sub-Applications)
     • 3.3.5. Communication
     • 3.3.6. Key Directory Structure
     • 3.3.7. Local Development & Debugging
     • 3.3.8. Environment Variables
   • 3.4. go-api (Shared Backend SDK)
     • 3.4.1. Overview & Purpose
     • 3.4.2. Technology Stack
     • 3.4.3. Key Functionalities
     • 3.4.4. Usage & Integration
     • 3.4.5. Development & Testing of go-api

4. Sub-Modules & Agents

   • 4.1. app-scanner
     • 4.1.1. Purpose and Functionality
     • 4.1.2. NSE (Nmap Script Engine) Management
   • 4.2. app-terminal
     • 4.2.1. Purpose and Functionality
   • 4.3. app-agent
     • 4.3.1. Purpose and Functionality
     • 4.3.2. gRPC Communication with sirius-engine

5. Data Persistence & Storage

   • 5.1. PostgreSQL (sirius-postgres)
     • 5.1.1. Role and Data Stored
     • 5.1.2. Schema Overview (Conceptual)
   • 5.2. Valkey (sirius-valkey)
     • 5.2.1. Role and Data Stored

6. Inter-Service Communication

   • 6.1. RabbitMQ (Asynchronous Messaging)
   • 6.2. tRPC (UI Frontend to Next.js Backend)
   • 6.3. RESTful APIs (sirius-api)
   • 6.4. gRPC (Engine to Agents)

7. Development & Deployment

   • 7.1. Dockerized Environment
   • 7.2. Hot Reloading (air)
   • 7.3. Build Processes (Dockerfiles)

8. Conclusion

   • 8.1. Summary of Architecture
   • 8.2. Future Directions

A. Getting Started & Development Workflow

A.1. System Prerequisites

To contribute to and run Sirius Scan, you will need the following software installed on your system:

• Docker and Docker Compose: Essential for running the containerized services. Ensure Docker Desktop (or equivalent for your OS) is installed and running.
• Git: For cloning the repository and managing versions.
• Node.js and npm/yarn/bun: Required for sirius-ui development (Next.js, TypeScript). Bun is used in the Dockerfile, so having it locally can be beneficial for consistency.
• Go (Golang): Required for sirius-api and sirius-engine development.
• A code editor: Such as VS Code, GoLand, WebStorm, etc.
• Basic command-line tools: curl, wget, etc., for various scripts and setups.

It's recommended to have reasonably up-to-date versions of these tools.

A.2. Repository Setup

1. Clone the Main Repository: Start by cloning the main Sirius repository:

   git clone https://github.com/SiriusScan/Sirius.git
   cd Sirius
   

2. Sub-Module Management: Sirius Scan utilizes several sub-modules (e.g., app-scanner, app-terminal, go-api, app-agent) that are developed in separate repositories. The sirius-engine Dockerfile clones some of these repositories (app-scanner, app-terminal, go-api) directly during its build process. This ensures that the engine has the necessary code for these integrated applications.

   For local development of these sub-modules, the docker-compose.yaml file includes commented-out volume mounts. For example:

   # In docker-compose.yaml, under sirius-engine service:
   volumes:
     - ./sirius-engine:/engine
     - ../minor-projects/app-agent:/app-agent
     # - ../minor-projects/go-api:/go-api # Local Development
     # - ../minor-projects/app-scanner:/app-scanner # Local Development
     # - ../minor-projects/app-terminal:/app-terminal # Local Development
   

   To develop a sub-module locally (e.g., app-scanner):

   • Clone the app-scanner repository into a directory relative to the main Sirius project, for instance, at the same level as Sirius in a minor-projects directory: ../minor-projects/app-scanner.
   • Uncomment the corresponding volume mount in docker-compose.yaml.
   • This will mount your local app-scanner code into the sirius-engine container, overriding the version cloned by the Dockerfile. Changes you make locally will be reflected due to tools like air (for Go applications) configured for hot reloading.

3. Environment Variables: Some services might require .env files. For example, sirius-ui copies a .env file in its Dockerfile. Check individual service directories or their respective Dockerfiles for any specific .env setup instructions. Often, default or example environment files are provided (e.g., .env.example) which can be copied and customized.

A.3. Running the Full Stack

Once you have the prerequisites installed and the repository set up (including any sub-modules you intend to develop locally):

1. Navigate to the Root Directory: Ensure your terminal is in the root of the Sirius project directory (where docker-compose.yaml is located).

2. Start All Services: Run the following command to build (if not already built) and start all services defined in docker-compose.yaml:

   docker compose up
   

   • Add the -d flag (docker compose up -d) to run containers in detached mode (in the background).
   • If you want to force a rebuild of the images (e.g., after changing a Dockerfile), use the --build flag: docker compose up --build.

3. Verify Services: After the command completes and services are starting up, you can check their status:

   docker compose ps
   

   You should see all services (e.g., sirius-ui, sirius-api, sirius-engine, sirius-postgres, sirius-rabbitmq, sirius-valkey) running.

4. Accessing Services:

   • sirius-ui: Open your web browser and navigate to http://localhost:3000 (or http://localhost:3001 for the development port, check docker-compose.yaml).
   • sirius-rabbitmq Management: Accessible at http://localhost:15672 (default credentials are often guest/guest).
   • Other services like sirius-api (port 9001), sirius-postgres (port 5432), etc., expose ports as defined in docker-compose.yaml for direct interaction if needed (e.g., with a database client or API tool).

5. Viewing Logs:

   • To view logs for all services in real-time (if not detached): docker compose logs -f
   • To view logs for a specific service (e.g., sirius-engine): docker compose logs -f sirius-engine

6. Stopping Services:

   • If running in the foreground, press Ctrl+C in the terminal where docker compose up is running.
   • If running in detached mode, or from another terminal:

     docker compose down
     

   • To stop and remove volumes (useful for a clean restart, be cautious as this deletes data in Docker volumes like your database): docker compose down -v.

A.4. Common Development Tasks

This section outlines general workflows for common development tasks. Detailed steps can vary and might require deeper understanding of the specific service's codebase and frameworks (Next.js/React for UI, Go/Fiber for API, Go for Engine).

A.4.1. Contributing a new UI Page/Component to sirius-ui

1. Locate Relevant Directories:
   • Pages are typically in sirius-ui/src/pages/.
   • Reusable components are in sirius-ui/src/components/.
   • tRPC procedures (API endpoints for the UI) are in sirius-ui/src/server/api/routers/.
2. Create New Files: For a new page, create a new .tsx file in the pages directory (e.g., sirius-ui/src/pages/new-feature.tsx). For a component, create it in an appropriate subdirectory under components.
3. Develop the Component/Page: Use React, TypeScript, and Tailwind CSS. Leverage existing components from Shadcn/ui (sirius-ui/src/components/lib/ui/) where possible.
4. Data Fetching (if needed):
   • If data is needed from the backend, define a new tRPC procedure in a relevant router file (e.g., sirius-ui/src/server/api/routers/yourRouter.ts).
   • Use api.yourRouter.yourProcedure.useQuery() or .useMutation() hooks in your frontend component to interact with the tRPC endpoint.
5. Styling: Use Tailwind CSS utility classes directly in your JSX. For global styles or complex base styles, consider sirius-ui/src/styles/globals.css.
6. Routing: Next.js uses a file-system based router. A file like pages/new-feature.tsx will be accessible at /new-feature.
7. Testing: Add relevant unit or integration tests.
8. Local Development: With docker compose up running, changes to files in sirius-ui/ (mounted into the container) should trigger hot reloading, and you can see your changes in the browser at http://localhost:3000.

A.4.2. Adding a new API endpoint to sirius-api

1. Understand Fiber Framework: sirius-api uses Fiber. Familiarize yourself with its routing and handler concepts.

2. Define Route: In sirius-api/main.go or a dedicated routes file (e.g., inside a handlers or routes directory if structured that way), add a new route using app.Get(), app.Post(), etc.

   // Example in sirius-api/main.go or a routes setup function
   // import "github.com/gofiber/fiber/v2"
   // import "your-project/sirius-api/handlers"
   
   // app is a *fiber.App instance
   // app.Get("/api/v1/new-endpoint", handlers.HandleNewEndpoint)
   

3. Implement Handler Function: Create the Go function that will handle requests to this endpoint (e.g., HandleNewEndpoint in a handlers package).

   // Example in sirius-api/handlers/new_handler.go
   package handlers
   
   import "github.com/gofiber/fiber/v2"
   
   func HandleNewEndpoint(c *fiber.Ctx) error {
       // Your logic here: parse request, interact with database (via go-api package),
       // communicate with sirius-engine via RabbitMQ, etc.
       return c.JSON(fiber.Map{"status": "success", "message": "New endpoint reached"})
   }
   

4. Database/Service Interaction: If your endpoint needs to interact with PostgreSQL or Valkey, use the functions and types provided by the shared go-api library (cloned into the sirius-engine container, but its source code should be available in your ../minor-projects/go-api if you set up local dev for it, or you might need to ensure sirius-api has access to it via Go modules).

5. Testing: Write unit or integration tests for your new endpoint and handler logic.

6. Local Development: The sirius-api service uses air for hot reloading. With docker compose up running and if sirius-api/ is volume-mounted (as it is by default in docker-compose.yaml), changes to Go files should trigger a restart of the API service. You can test your endpoint using tools like curl or Postman against http://localhost:9001/api/v1/new-endpoint.

A.4.3. Modifying or adding a module within sirius-engine

Development within sirius-engine is more complex due to its role as an orchestrator and its integration of sub-applications like app-scanner and app-terminal.

1. Understand Engine Structure: The main engine logic is in Go. Sub-applications (app-scanner, app-terminal) are also typically Go projects cloned into the engine's container during build (or volume-mounted for local dev).
2. Local Development Setup:
   • To modify a sub-application like app-scanner, clone its repository locally (e.g., ../minor-projects/app-scanner).
   • Uncomment the relevant volume mount in docker-compose.yaml for sirius-engine (e.g., - ../minor-projects/app-scanner:/app-scanner).
   • This mounts your local code into the /app-scanner directory inside the sirius-engine container.
3. Making Changes:
   • Engine Core Logic: Modify Go files directly within the sirius-engine/ directory.
   • Sub-Application Logic (e.g., app-scanner): Modify Go files in your local clone (e.g., ../minor-projects/app-scanner/).
4. start.sh Script: The sirius-engine/Dockerfile uses start.sh as its ENTRYPOINT. This script is responsible for starting the main engine and potentially building/running the sub-applications. It might use air for hot reloading these sub-apps if they have .air.toml configurations.
5. Communication:
   • Modules within the engine might communicate via Go channels, direct function calls, or local IPC if start.sh runs them as separate processes.
   • The engine communicates with sirius-ui/sirius-api via RabbitMQ and with app-agent instances via gRPC.
6. Testing: This can be complex. Unit tests for individual Go packages are standard. Integration testing might involve sending messages via RabbitMQ or making gRPC calls to test specific flows.
7. Local Development & Hot Reloading:
   • If air is configured for the Go components you are working on (either the main engine or a sub-application like app-scanner via its own air setup invoked by start.sh), changes to mounted Go files should trigger a rebuild/restart of that component within the sirius-engine container.
   • Observe logs using docker compose logs -f sirius-engine to see restarts and any errors.

A.5. Testing Strategy

Sirius Scan aims for comprehensive testing, though the specific strategies and depth can vary per module.

• General Approach:

  • Unit Tests: Each module/service should have unit tests for its core logic, especially for Go and TypeScript code. Standard testing libraries for each language/framework are used (e.g., Go's testing package, Jest/Vitest for TypeScript/React).
  • Integration Tests: Testing interactions between components within a service (e.g., API endpoint to database logic in sirius-api) or between services where feasible (e.g., UI making a call that results in a RabbitMQ message).
  • End-to-End (E2E) Tests: For critical user flows, E2E tests can simulate user interactions from the UI through the backend services. (Frameworks like Cypress or Playwright might be considered for UI E2E testing).

• sirius-ui:

  • Component testing using React Testing Library or similar.
  • tRPC procedure testing.

• sirius-api:

  • Handler function unit tests.
  • Endpoint integration tests (e.g., using Go's httptest package).

• sirius-engine:

  • Unit tests for Go packages.
  • Testing RabbitMQ message handling and gRPC service interactions can be complex and might involve mock servers or specific test environments.
  • Sub-applications like app-scanner and app-terminal should have their own testing strategies within their repositories.

• app-agent:

  • Unit tests for agent logic.
  • Testing gRPC communication with a mock sirius-engine server.

• Existing Documentation:

  • The document Sirius/documentation/README.testing.md provides specific test cases and scenarios, particularly focused on data relationships and expected outcomes for vulnerability scanning. This document should be consulted for understanding expected system behavior and for designing new tests related to core scanning functionalities.

Developers are encouraged to write tests for new features and bug fixes. CI/CD pipelines (if set up) would ideally run these tests automatically.

B. Key Architectural Workflows (End-to-End Examples)

B.1. User Initiates a Vulnerability Scan

This workflow describes the sequence of events when a user initiates a vulnerability scan from the sirius-ui.

1. User Interaction (sirius-ui):

   • The user navigates to the scan configuration page in the sirius-ui.
   • They input scan parameters such as target IP addresses/ranges, scan type (e.g., full scan, specific port scan), Nmap script selection (if applicable), and other options.
   • The user clicks a "Start Scan" button.

2. UI Frontend to BFF (sirius-ui - tRPC):

   • The frontend component makes a tRPC mutation call to its Next.js backend (BFF).
   • The payload includes the scan configuration details.

3. BFF Processing (sirius-ui - Next.js Backend):

   • The tRPC procedure in the Next.js backend receives the scan request.
   • It may perform initial validation or enrichment of the scan parameters.
   • It then constructs a message (e.g., a JSON payload) representing the scan job.

4. Task Queuing (sirius-ui to sirius-rabbitmq):

   • The Next.js backend (BFF) publishes this scan job message to a specific queue in sirius-rabbitmq (e.g., scan_jobs_queue).
   • This decouples the UI/BFF from the sirius-engine, allowing the UI to remain responsive even if the engine is busy.

5. Scan Processing (sirius-engine):

   • sirius-engine (Go application) is subscribed to the scan_jobs_queue in RabbitMQ.
   • It consumes the scan job message.
   • The engine parses the job details and determines the appropriate action.
   • This might involve:
     • Interacting with sirius-postgres (via go-api library) to retrieve host information or store scan metadata.
     • Orchestrating app-scanner (which is part of its container) to perform the actual network scanning using tools like Nmap and RustScan, leveraging the managed NSE scripts.
     • If the scan involves remote agents, it might delegate parts of the scan to specific app-agent instances via gRPC (see Workflow B.2).

6. Execution (app-scanner within sirius-engine):

   • app-scanner executes the scan commands (e.g., Nmap with specified options and scripts).
   • It collects the raw scan output (e.g., Nmap XML output).

7. Result Handling & Persistence (sirius-engine, go-api, sirius-postgres, sirius-valkey):

   • sirius-engine (or app-scanner itself) parses the raw scan results.
   • It extracts meaningful information: open ports, services, identified vulnerabilities (e.g., by matching script output to CVEs or known issues).
   • This processed data is then persisted into sirius-postgres (e.g., updating hosts, ports, vulnerabilities, and their join tables) likely using the go-api library functions.
   • Intermediate status or partial results might be cached in sirius-valkey.

8. Status Updates & Notifications (RabbitMQ):

   • During the scan, sirius-engine may publish progress updates or status messages to other RabbitMQ queues (e.g., scan_status_updates_queue).
   • sirius-ui (via its BFF and potentially WebSockets or polling tRPC queries) can subscribe to these updates to display real-time scan progress to the user.

9. Scan Completion & Final Results (sirius-ui):

   • Once the scan is complete and results are persisted, sirius-engine might publish a final completion message.
   • The user can then view the detailed scan results and vulnerabilities in the sirius-ui, which fetches this data from sirius-api (which in turn queries sirius-postgres) or directly via tRPC calls that access the database.

B.2. Agent Interaction & Management

This workflow covers how app-agent instances might register (conceptually, as the current app-agent is basic) and how the sirius-engine executes commands on them.

Conceptual Agent Registration/Discovery (Future Enhancement): While the current app-agent (from app-agent/cmd/agent/README.md) doesn't detail a formal registration process, a production system would typically involve agents announcing their presence or the engine discovering them.

1. Agent Startup (app-agent): An app-agent instance starts on a target host. It knows the sirius-engine's gRPC address (SERVER_ADDRESS).
2. Initial Contact/Heartbeat (app-agent to sirius-engine - gRPC):
   • The agent could make an initial Register or Heartbeat gRPC call to sirius-engine, providing its AGENT_ID (e.g., hostname), capabilities (OS, available tools), and IP address.
   • sirius-engine records or updates the agent's status and availability, perhaps in sirius-valkey or sirius-postgres.
3. Periodic Pings: Agents would periodically send Ping requests to the engine to maintain active status. The engine would mark agents as offline if pings cease.

Command Execution via Agent:

1. Trigger for Command:

   • A user might initiate a command for a specific agent via sirius-ui (e.g., through an agent management panel or the terminal, see Workflow B.3).
   • A scan job in sirius-engine might require executing a command on a specific agent (e.g., running a local enumeration script).

2. Request to Engine:

   • If from UI, the request flows: sirius-ui (frontend) -> tRPC -> sirius-ui (BFF) -> RabbitMQ -> sirius-engine.
   • The message to sirius-engine would specify the target AGENT_ID and the command to execute.

3. Engine Locates Agent (sirius-engine):

   • sirius-engine looks up the target agent's details (e.g., its gRPC connection status/details, if connections are persistent, or just its known address).

4. Command Delegation (sirius-engine to app-agent - gRPC):

   • sirius-engine makes an ExecuteCommand gRPC call to the specified app-agent instance.
   • The gRPC request payload contains the command string (e.g., "ls -la /tmp" or "nmap -sV target.com").

5. Command Execution on Agent (app-agent):

   • The app-agent receives the ExecuteCommand request.
   • It executes the command on the underlying operating system using its shell execution capabilities.
   • Security Note: As per app-agent/cmd/agent/README.md, the current example agent lacks command validation and sanitization, which is a critical security concern in a production system.

6. Result Streaming/Return (app-agent to sirius-engine - gRPC):

   • The app-agent captures the standard output (stdout) and standard error (stderr) of the executed command.
   • It sends these results back to sirius-engine via the gRPC response. This could be a single response upon command completion or a stream of data if the command produces continuous output.
   • The agent README mentions it "periodically executes 'ls -la' and sends results to the server," implying it can handle periodic/streaming results.

7. Engine Processes Results (sirius-engine):

   • sirius-engine receives the command output from the agent.
   • It processes the results as needed:
     • If part of a larger scan, the results are incorporated into the scan findings.
     • If a direct command from UI, the results are formatted.
   • Results might be stored in sirius-postgres or cached in sirius-valkey.

8. Forwarding to User/UI (sirius-engine to RabbitMQ to sirius-ui):

   • sirius-engine publishes the command results (or a notification of completion) to a RabbitMQ queue.
   • sirius-ui (BFF) consumes this message and relays the information to the user's interface (e.g., displayed in a terminal window or results panel).

B.3. Terminal Command Execution

This workflow outlines how a command typed into the terminal in sirius-ui reaches and is executed by app-terminal within the sirius-engine.

1. User Input (sirius-ui - Terminal Component):

   • The user types a command into the terminal interface provided by sirius-ui (e.g., XTerm.js integrated component).
   • The command might be a system-level command or a special internal command (e.g., use agent <id>).

2. UI Frontend to BFF (sirius-ui - tRPC):

   • The terminal component captures the command string.
   • It makes a tRPC mutation call (e.g., executeTerminalCommand) to its Next.js backend (BFF).
   • The payload includes the command string and potentially the current context (e.g., if a specific agent is targeted via a previous use agent command).

3. BFF Processing (sirius-ui - Next.js Backend):

   • The tRPC procedure in the Next.js backend receives the command request.
   • It may perform initial parsing or validation. For example, it might differentiate between internal UI commands (like switching agent context locally in the UI state) and commands to be sent to the backend engine.
   • For commands destined for app-terminal, it constructs a message (e.g., a JSON payload) including the command and any relevant context (like target agent ID if the terminal is proxying to an agent, or if it's a command for the engine itself).

4. Task Queuing (sirius-ui to sirius-rabbitmq):

   • The Next.js backend (BFF) publishes this terminal command message to a specific queue in sirius-rabbitmq (e.g., terminal_commands_queue).

5. Command Routing & Processing (sirius-engine):

   • sirius-engine (Go application) subscribes to the terminal_commands_queue.
   • It consumes the command message.
   • The engine parses the command and context. It determines if the command is for:
     • app-terminal directly: For general commands to be executed in the engine's environment (potentially within a PowerShell session managed by app-terminal).
     • An app-agent via app-terminal as a proxy: If the terminal context is set to an agent, the engine might instruct app-terminal to relay this command, or the engine itself might make a gRPC call to the target app-agent (similar to Workflow B.2, step 4).
     • Internal engine commands: For special commands handled by the engine itself.

6. Execution via app-terminal (sirius-engine -> app-terminal):

   • If the command is for app-terminal, sirius-engine passes the command to its integrated app-terminal module.
   • app-terminal manages command execution environments (e.g., it could maintain PowerShell sessions).
   • It executes the command within the appropriate session/environment.
   • app-terminal captures the stdout and stderr of the command.
   • Note: The sirius-engine/Dockerfile installs PowerShell, and app-terminal is the likely consumer of this, providing a PowerShell backend for the terminal feature.

7. Result Return (app-terminal to sirius-engine):

   • app-terminal returns the command output (stdout, stderr) back to the main sirius-engine logic that invoked it.

8. Forwarding to User/UI (sirius-engine to RabbitMQ to sirius-ui):

   • sirius-engine takes the output received from app-terminal (or directly from an app-agent if it was a proxied command).
   • It publishes this output to a specific RabbitMQ queue intended for terminal output (e.g., terminal_output_queue). The message should correlate with the original command or session.
   • sirius-ui (BFF) subscribes to this queue.
   • The BFF receives the output and uses a mechanism (like WebSockets or Server-Sent Events connected to the tRPC subscription) to stream the output back to the specific user's terminal component in the UI.

9. Display Output (sirius-ui - Terminal Component):

   • The terminal component in the UI receives the streamed output and displays it to the user.

1. Introduction

1.1. Purpose of Sirius Scan

Sirius Scan is an open-source, general-purpose vulnerability scanner designed to leverage community-driven security intelligence. It aims to provide a comprehensive platform for identifying, managing, and remediating security vulnerabilities across an organization's digital assets. The system is built with modularity and extensibility in mind, allowing for continuous enhancement and adaptation to the evolving cybersecurity landscape.

(Reference: Sirius Scan GitHub README)

1.2. High-Level Architecture Overview

Sirius Scan employs a microservices architecture, with its various components containerized and orchestrated using Docker Compose. This approach promotes scalability, resilience, and maintainability. The system comprises several key services that work in concert:

• sirius-ui: A Next.js-based web interface providing the primary user interaction point.
• sirius-api: A Go-based RESTful API service that handles backend logic and data operations.
• sirius-engine: A Go-based core processing engine responsible for orchestrating scans, managing agents, and running various specialized sub-applications that provide core functionalities like scanning (app-scanner) and terminal access (app-terminal). It receives tasks primarily via RabbitMQ from sirius-ui or sirius-api and coordinates the execution of these tasks using its sub-modules and agents.
• Data Stores:
  • sirius-postgres: A PostgreSQL database for persistent storage of relational data (e.g., host information, vulnerabilities).
  • sirius-valkey: A Valkey (Redis-like) key-value store for caching, session management, and other non-relational data.
• Messaging:
  • sirius-rabbitmq: A RabbitMQ message broker facilitating asynchronous communication between services, particularly between the UI/API and the Engine.

The following diagram from the main project README illustrates the interconnected services:

Service	Description	Port(s)
sirius-ui	Web interface (Next.js)	3000 (HTTP), 3001 (Dev)
sirius-api	Backend API service	9001
sirius-engine	Scanning engine	5174
sirius-rabbitmq	Message broker	5672 (AMQP), 15672 (Management)
sirius-postgres	Database	5432
sirius-valkey	Key-value store	6379

(Source: Adapted from Sirius/README.md)

Communication between the frontend, API, and engine is carefully managed, with tRPC used for efficient data transfer between the UI's frontend and its Next.js backend, REST for the sirius-api, and RabbitMQ for asynchronous tasks delegated to the sirius-engine. The sirius-engine also communicates with deployed agents via gRPC.

2. Core Infrastructure (Docker Compose)

Sirius Scan leverages Docker Compose for defining and running its multi-container application environment. This ensures consistency across development, testing, and deployment, and simplifies the management of the various microservices.

2.1. docker-compose.yaml Overview

The main configuration is found in Sirius/docker-compose.yaml. It defines the services, networks, and volumes required for the application to run. The name: sirius directive at the top sets a project name for easier management of Docker resources.

name: sirius

services:
  # ... service definitions below ...

(Source: Sirius/docker-compose.yaml)

2.2. Service Orchestration

The following services form the backbone of the Sirius Scan infrastructure:

2.2.1. sirius-rabbitmq (Message Broker)

RabbitMQ serves as the central message bus for asynchronous communication between different services, particularly for decoupling long-running tasks and distributing workloads.

• Purpose: Facilitates asynchronous communication, task queuing, and event-driven interactions.
• Image: rabbitmq:3.7.3-management (Includes the RabbitMQ management plugin).
• Ports:
  • 5672:5672 (AMQP protocol for messaging)
  • 15672:15672 (Management UI)
• Configuration: Uses a custom rabbitmq.conf for tailored settings.
• Healthcheck: Ensures the RabbitMQ service is operational.

sirius-rabbitmq:
  image: rabbitmq:3.7.3-management
  restart: always
  container_name: sirius-rabbitmq
  hostname: sirius-rabbitmq
  ports:
    - "5672:5672"
    - "15672:15672"
  volumes:
    - ./rabbitmq/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf:ro
  healthcheck:
    test: ["CMD", "rabbitmqctl", "status"]
    interval: 30s
    timeout: 15s
    retries: 5

(Source: Sirius/docker-compose.yaml)

2.2.2. sirius-postgres (Relational Database)

PostgreSQL is used as the primary relational database for storing structured data such as host information, vulnerability details, scan results, and user data.

• Purpose: Persistent storage for relational application data.
• Image: postgres:15-alpine.
• Environment Variables: Sets up default user, password, and database name.
• Ports: 5432:5432 (Standard PostgreSQL port).
• Healthcheck: Verifies the database server is ready to accept connections.

sirius-postgres:
  image: postgres:15-alpine
  restart: always
  container_name: sirius-postgres
  hostname: sirius-postgres
  environment:
    POSTGRES_USER: postgres
    POSTGRES_PASSWORD: postgres
    POSTGRES_DB: sirius
  ports:
    - "5432:5432"
  healthcheck:
    test: ["CMD-SHELL", "pg_isready -U postgres"]
    interval: 10s
    timeout: 5s
    retries: 5

(Source: Sirius/docker-compose.yaml)

2.2.3. sirius-valkey (Key-Value Store)

Valkey, a fork of Redis, provides a high-performance in-memory key-value store. It's used for caching, session management, storing temporary data, and potentially for managing NSE script manifests as indicated in scanner documentation.

• Purpose: In-memory data storage for caching, session data, and other non-relational, fast-access needs.
• Image: valkey/valkey:latest.
• Ports: 6379:6379 (Standard Valkey/Redis port).

sirius-valkey:
  image: valkey/valkey:latest
  restart: always
  container_name: sirius-valkey
  hostname: sirius-valkey
  ports:
    - "6379:6379"

(Source: Sirius/docker-compose.yaml)

2.2.4. Networking

Docker Compose sets up a default network for all services defined in the docker-compose.yaml file. This allows services to discover and communicate with each other using their service names as hostnames.

• Default Network: Unless specified otherwise, all services join a network named <project_name>_default (e.g., sirius_default since name: sirius is at the top of docker-compose.yaml).
• Service Discovery: Within this default network, services can reach each other using their names. For example, sirius-api can connect to sirius-postgres using the hostname sirius-postgres and port 5432.

  // Example connection string for sirius-api to connect to postgres
  // dsn := "host=sirius-postgres user=postgres password=postgres dbname=sirius port=5432 sslmode=disable"
  

• Port Mapping: The ports directive in docker-compose.yaml (e.g., "3000:3000" for sirius-ui) maps ports from the host machine to the container. The first port is the host port, and the second is the container port. This makes services accessible from the host machine (e.g., localhost:3000). Internal communication between containers typically uses the container port and service name.
• Custom Networks: While not explicitly used in the current docker-compose.yaml, Docker Compose allows for defining custom bridge networks for more complex networking scenarios or to isolate groups of services. This could be a future consideration if needed.
• Dependencies (depends_on): The depends_on directive can control the startup order of services, but it only waits until the dependency has started, not until it's ready to accept connections. For readiness, healthchecks (as used for sirius-rabbitmq and sirius-postgres) are a more robust solution to ensure a service is fully operational before dependent services try to connect to it.

3. Application Services

Beyond the core infrastructure, Sirius Scan is composed of three main application services: sirius-ui for the frontend and user interaction, sirius-api for backend RESTful services, and sirius-engine for core scanning and processing tasks. The go-api library serves as a critical shared SDK for the Go-based backend services.

3.1. sirius-ui (User Interface & BFF)

3.1.1. Overview & Purpose

The sirius-ui service provides the web-based graphical user interface (GUI) for Sirius Scan. It allows users to manage scans, view results, configure settings, and interact with all aspects of the vulnerability scanner. It also acts as a Backend-For-Frontend (BFF), handling some UI-specific logic before communicating with other backend services.

As defined in docker-compose.yaml:

sirius-ui:
  build:
    context: ./sirius-ui/
    dockerfile: Dockerfile
    args:
      NEXT_PUBLIC_CLIENTVAR: "clientvar"
  container_name: sirius-ui
  hostname: sirius-ui
  restart: always
  image: sirius-ui
  working_dir: /app
  ports:
    - "3000:3000"
    - "3001:3001"
  volumes:
    - ./sirius-ui:/app
    - /app/node_modules

(Source: Sirius/docker-compose.yaml)

3.1.2. Technology Stack

sirius-ui is built with a modern JavaScript/TypeScript stack:

• Next.js: A React framework for server-side rendering (SSR), static site generation (SSG), and client-side applications. Chosen for its robust features and development experience, including routing and API routes (which contribute to its BFF capabilities).
• TypeScript: For static typing, improving code quality and maintainability.
• tRPC: Enables end-to-end typesafe APIs between the Next.js frontend and its backend (API routes). This simplifies data fetching and ensures type consistency.
• Tailwind CSS: A utility-first CSS framework for rapid UI development and styling.
• Shadcn/ui: A collection of re-usable UI components built with Radix UI and Tailwind CSS, used for building the user interface.
• npm: Used for package management and dependency installation, with Node.js as the JavaScript runtime.

3.1.3. Docker Configuration (sirius-ui/Dockerfile)

The sirius-ui Docker image is built using node:18-alpine as the base, using npm for package management and running the Next.js application.

Key steps in the Sirius/sirius-ui/Dockerfile include:

# sirius-ui Dockerfile - Multi-stage build

# Base stage with common dependencies
FROM node:18-alpine AS base

# Install system dependencies
RUN apk add --no-cache libc6-compat openssl ca-certificates

WORKDIR /app

# Copy package files and install dependencies
COPY package*.json ./
RUN npm install

# Copy source code and build configuration
COPY . .

# Generate Prisma client
RUN npx prisma generate

# Development stage
FROM base AS development
ENV NODE_ENV=development
ENV NEXT_TELEMETRY_DISABLED=1

EXPOSE 3000
CMD ["/app/start-dev.sh"]

(Source: Sirius/sirius-ui/Dockerfile)

This Dockerfile sets up the environment, installs dependencies using bun install, copies the application source, and runs the development server. It also exposes ports 3000 (likely for the application) and 3001 (possibly for a development or auxiliary service).

3.1.4. UI Theming (globals.css)

The visual theme of sirius-ui is defined in Sirius/sirius-ui/src/styles/globals.css. It utilizes Tailwind CSS and defines CSS custom properties (variables) for theming, including support for a dark mode.

Example of theme variables (light mode):

@layer base {
  :root {
    --background: 220, 13%, 100%, 1;
    --foreground: 222.2 84% 4.9%;

    --card: 0 0% 100%;
    --card-foreground: 222.2 84% 4.9%;

    --primary: 222.2 47.4% 11.2%;
    --primary-foreground: 210 40% 98%;

    /* ... other variables ... */
  }

  .dark {
    --background: 234, 26%, 15%, 1;
    --foreground: 210 40% 98%;

    /* ... other dark mode variables ... */
  }
}

(Source: Sirius/sirius-ui/src/styles/globals.css)

The file also includes styles for specific components and decorative elements like the hexgrad background.

3.1.5. Extensibility (Marketplace Concept)

A key design consideration for choosing Next.js was to facilitate future development of a marketplace-style system. This implies that the UI is structured to allow third-party developers to contribute additional pages, modules, or plugins, enhancing the platform's capabilities over time. While not yet implemented, this architectural foresight influences the UI's design towards modularity.

3.1.6. Communication

• Frontend to BFF (Next.js Backend): Uses tRPC for type-safe API calls from the client-side components to the Next.js API routes.
• BFF to other services: The Next.js backend part of sirius-ui communicates with sirius-engine via RabbitMQ for asynchronous operations and potentially with sirius-api for other data needs (though direct REST calls from client components to sirius-api are also possible, tRPC typically centralizes data access through the BFF).

3.1.7. Key Directory Structure

The sirius-ui service, being a Next.js application, follows a conventional directory structure. Here are some key directories within Sirius/sirius-ui/:

• src/: The primary directory for application source code.
  • src/pages/: Contains Next.js pages. Each .tsx or .ts file in this directory (excluding those in api/) becomes a route.
    • src/pages/index.tsx: The home page.
    • src/pages/api/**: API routes for the Next.js backend (BFF).
      • src/pages/api/trpc/[trpc].ts: Handles all tRPC requests.
  • src/components/: Reusable React components.
    • src/components/lib/ui/: Shadcn/ui components (often imported as ~/components/lib/ui/...).
    • Custom components are usually organized into subdirectories based on feature or type (e.g., agent/, scanner/, terminal/).
  • src/server/: Backend-specific code for the Next.js application.
    • src/server/api/:
      • src/server/api/root.ts: Defines the root tRPC router, merging other routers.
      • src/server/api/routers/: Contains individual tRPC routers (e.g., terminal.ts, agent.ts).
  • src/styles/: Styling files.
    • src/styles/globals.css: Global styles and Tailwind CSS base configuration, including theme variables.
  • src/utils/: Utility functions and helpers.
  • src/hooks/: Custom React hooks.
  • src/types/: TypeScript type definitions and interfaces.
  • src/services/: May contain client-side service integrations if not directly handled by tRPC hooks.
• public/: Static assets that are served directly from the root (e.g., images, favicons).
• prisma/: Contains Prisma schema (schema.prisma) and migration files, used for database interaction by the Next.js backend (though the primary database interaction seems to be driven by sirius-api). If sirius-ui needs direct DB access for some BFF tasks, Prisma might be used here.
• Dockerfile: Defines how the sirius-ui service is built as a Docker image.
• package-lock.json, package.json, tsconfig.json: Project configuration and dependency management files (using npm).
• .env / .env.example: Environment variable definitions.

3.1.8. Local Development & Debugging

Developing and debugging sirius-ui typically involves the following:

• Prerequisites: Ensure Node.js and npm are installed locally.
• Running with Docker Compose:
  • The easiest way is to use docker compose up. The sirius-ui service definition in docker-compose.yaml mounts the ./sirius-ui:/app directory into the container.
  • The Dockerfile for sirius-ui uses CMD ["/app/start-dev.sh"], which starts the Next.js development server with hot reloading.
  • Changes made to your local Sirius/sirius-ui/src/ files will be reflected in the container, and the Next.js dev server will automatically reload the application in your browser (typically http://localhost:3000).
• Browser Developer Tools:
  • Use your browser's developer tools (e.g., Chrome DevTools, Firefox Developer Tools) extensively for debugging frontend issues: inspecting HTML/CSS, debugging JavaScript (React components, tRPC calls), checking network requests, and viewing console logs.
  • React Developer Tools browser extension is highly recommended for inspecting React component hierarchy, props, and state.
• tRPC Debugging:
  • Network tab in browser dev tools can show tRPC requests (usually to /api/trpc/...).
  • For server-side tRPC procedure debugging, you can add console.log statements in your tRPC router functions (sirius-ui/src/server/api/routers/*.ts). These logs will appear in the sirius-ui container logs (docker compose logs -f sirius-ui).
• VS Code Debugging (Next.js): You can configure VS Code to attach to the Node.js process running inside the Docker container for more advanced debugging of the Next.js backend (BFF) part. This usually involves:
  • Ensuring the Node.js process in the container is started with the --inspect flag (might require modifying the dev script in package.json and Dockerfile CMD if not already set up for this).
  • Configuring port forwarding in docker-compose.yaml for the inspect port (e.g., 9229).
  • Setting up a launch.json configuration in VS Code to attach to the debugger.
• Viewing Logs: Use docker compose logs -f sirius-ui to see server-side logs from the Next.js application, including any errors or console output from tRPC procedures or server components.

3.1.9. Environment Variables

sirius-ui utilizes environment variables for configuration. These are typically managed via an .env file in the Sirius/sirius-ui/ directory.

• .env File: The Sirius/sirius-ui/Dockerfile copies a .env file into the image: COPY .env ./. You should create this file if it doesn't exist, possibly by copying from an .env.example if provided.

• Next.js Environment Variables:

  • Variables prefixed with NEXT_PUBLIC_ are exposed to the browser (client-side JavaScript).
  • Variables without this prefix are only available on the server-side (Next.js backend/BFF).

• Key Variables (Examples - actual variables may differ):

  • DATABASE_URL: If Prisma or another ORM is used directly by the UI's BFF for some tasks (though primary DB interaction is via sirius-api).
  • NEXTAUTH_URL: For NextAuth.js if used for authentication.
  • NEXTAUTH_SECRET: Secret key for NextAuth.js.
  • RABBITMQ_URL: URL for connecting to the RabbitMQ service (e.g., amqp://guest:guest@sirius-rabbitmq:5672/). Used by the BFF to send messages to sirius-engine.
  • SIRIUS_API_URL: Base URL for the sirius-api service (e.g., http://sirius-api:9001/api/v1) if the BFF needs to call it directly.
  • NEXT_PUBLIC_CLIENTVAR: An example variable shown in docker-compose.yaml build args, exposed to the client.
  • SKIP_ENV_VALIDATION=1: As seen in the Dockerfile, this is set to bypass environment variable validation, likely from tRPC or a similar library that checks for expected env vars.

• Loading in Docker: The Dockerfile copies the .env file. During local development with docker compose up, the volume mount - ./sirius-ui:/app ensures that your local .env file is used.

• Build Arguments: The docker-compose.yaml for sirius-ui shows an example of passing build arguments:

  args:
    NEXT_PUBLIC_CLIENTVAR: "clientvar"
  

  These can be used to set environment variables at build time.

3.2. sirius-api (Backend API)

3.2.1. Overview & Purpose

The sirius-api service is a backend application written in Go. It exposes a RESTful API that serves as a primary interface for data manipulation and core business logic not directly handled by the UI's BFF or the specialized sirius-engine. It interacts with the data stores (PostgreSQL, Valkey) and communicates with other services like sirius-engine via RabbitMQ.

As defined in docker-compose.yaml:

sirius-api:
  build: ./sirius-api/
  container_name: sirius-api
  hostname: sirius-api
  restart: always
  image: sirius-api
  ports:
    - "9001:9001"
  volumes:
    - ./sirius-api:/api
    # - ../minor-projects/go-api:/go-api # Local Development

(Source: Sirius/docker-compose.yaml)

The commented-out volume ../minor-projects/go-api:/go-api suggests that go-api is a shared library used by this service, likely containing common Go code for database interaction, models, etc.

3.2.2. Technology Stack

• Go (Golang): The primary programming language, chosen for its performance and concurrency features.
• Fiber: A Go web framework inspired by Express.js, used for building the RESTful API. It's known for its speed and low memory footprint.
• Go Modules: For dependency management.
• Air: Used for live reloading during development, as indicated by the Dockerfile.

3.2.3. Docker Configuration (sirius-api/Dockerfile)

The Sirius/sirius-api/Dockerfile sets up the Go environment and uses air for development, enabling hot reloading of the application upon code changes.

# sirius-api Dockerfile

FROM golang:latest
WORKDIR /api

COPY .air.toml .air.toml

# Dependencies
RUN apt-get update

WORKDIR /api
#DEV Dependencies
RUN go install github.com/air-verse/air@latest


# Invoke air to run the server after volume mount
CMD ["air", "-c", ".air.toml"]

EXPOSE 9001

(Source: Sirius/sirius-api/Dockerfile)

The Dockerfile copies an .air.toml configuration file, installs air, and sets the CMD to run the API using air. Port 9001 is exposed for the API service.

3.2.4. Key Functionalities (High-Level)

Based on the go-api/docs/documentation.md (which describes the github.com/SiriusScan/go-api package likely used by this service), the sirius-api would be responsible for functionalities such as:

• Host Management: CRUD operations for hosts, retrieving host details, and statistics.
• Vulnerability Management: Adding, retrieving, and checking for vulnerabilities.
• Database Interaction: Managing connections and operations with PostgreSQL via GORM (as suggested by go-api documentation).
• Key-Value Store Interaction: Interfacing with Valkey for caching or other purposes.
• NVD Integration: Potentially fetching CVE information from the National Vulnerability Database.

Example from go-api/docs/documentation.md showing host management functions:

// GetHost retrieves a host by IP address
func GetHost(ip string) (sirius.Host, error)

// GetAllHosts retrieves all hosts from the database
func GetAllHosts() ([]sirius.Host, error)

// AddHost adds or updates a host in the database
func AddHost(host sirius.Host) error

(Source: go-api/docs/documentation.md)

3.2.5. Communication

• Client Interaction: Exposes RESTful endpoints on port 9001 for consumption by sirius-ui (either directly or via its BFF) or other potential clients.
• Internal Communication: Communicates with sirius-engine via RabbitMQ for delegating tasks or sending notifications.
• Data Stores: Directly interacts with sirius-postgres and sirius-valkey.

3.2.6. Key Directory Structure

The sirius-api service, being a Go application using the Fiber framework, typically has a structure that might include:

• main.go: The entry point of the application. Initializes the Fiber app, sets up middleware, defines routes, and starts the server.
• handlers/ (or controllers/): Contains Go files with handler functions for different API routes. Each handler processes incoming requests, interacts with services or databases, and returns responses.
• routes/: Might contain files that define route groups and associate them with handlers, helping to organize routing logic if main.go becomes too crowded.
• models/: Could contain Go struct definitions representing data entities (e.g., Host, Vulnerability). However, much of this might be centralized in the shared go-api library if it's used extensively.
• middleware/: For custom Fiber middleware (e.g., for authentication, logging, CORS).
• config/: For application configuration loading (e.g., reading database credentials, port numbers from environment variables or config files).
• Dockerfile: Defines how the sirius-api Go application is built into a Docker image.
• .air.toml: Configuration file for air, the live-reloading tool used for development. Specifies build commands, run commands, and files to watch.
• go.mod and go.sum: Go module files managing project dependencies.
• Shared go-api Library: The docker-compose.yaml mentions a commented-out volume mount for ../minor-projects/go-api. This external library likely contains shared database models (GORM structs), database interaction logic (PostgreSQL, Valkey), RabbitMQ client utilities, and other common functionalities used by both sirius-api and potentially sirius-engine.
  • When developing sirius-api, if it relies on go-api, ensure this library is accessible in your Go module path or use the volume mount for local development of go-api itself.

3.2.7. Local Development & Debugging

Developing and debugging the Go-based sirius-api service involves these common practices:

• Prerequisites: Ensure Go is installed locally.

• Running with Docker Compose & Air:

  • The sirius-api service in docker-compose.yaml mounts the local ./sirius-api:/api directory.
  • Its Dockerfile sets air -c .air.toml as the command. air watches for file changes in the mounted directory and automatically recompiles and restarts the Go application.
  • Start services with docker compose up. Changes to .go files in your local Sirius/sirius-api/ directory will trigger a reload within the container.

• Viewing Logs: Use docker compose logs -f sirius-api to see the output from air (compilation status, errors) and any logs generated by the Fiber application (e.g., request logs, fmt.Println or logger output).

• Testing Endpoints: Use tools like curl, Postman, or Insomnia to send HTTP requests to your API endpoints (e.g., http://localhost:9001/your-endpoint).

• Go Debugging (Delve): For step-through debugging of Go code:

  • Install Delve: You might need Delve installed locally or ensure it's available within the Docker container if you intend to debug inside it.

  • Modify .air.toml for Debugging: You can adjust the cmd in .air.toml to start your application with Delve listening for a debugger. For example:

    # .air.toml example modification for delve
    [build]
    cmd = "go build -o ./tmp/main ."
    # ... other build settings
    
    [run]
    # cmd = "./tmp/main" # Original run command
    cmd = "dlv --listen=:2345 --headless=true --api-version=2 --accept-multiclient exec ./tmp/main" # Delve command
    # ... other run settings
    

  • Expose Delve Port: In docker-compose.yaml, expose the Delve port (e.g., 2345:2345) for the sirius-api service.

  • VS Code/GoLand Debug Configuration: Configure your IDE (VS Code with Go extension, or GoLand) to attach to the remote Delve debugger running in the Docker container. Create a debug configuration (e.g., Go Remote in VS Code's launch.json) pointing to localhost:2345.

  • Set breakpoints in your Go code and start the debugging session from your IDE.

• Database Interaction: If debugging issues related to database queries (using go-api library and GORM), GORM has logging capabilities that can be enabled to see the exact SQL queries being executed. This logging would appear in the sirius-api container logs.

3.2.8. Environment Variables

The Go-based sirius-api service is configured using environment variables. While a dedicated .env file isn't explicitly copied in its Dockerfile (unlike sirius-ui), environment variables can be supplied to Docker containers in several ways:

• docker-compose.yaml: The most common method for development is to set environment variables directly in the docker-compose.yaml file for the sirius-api service.

  # Example in docker-compose.yaml
  sirius-api:
    # ... other settings ...
    environment:
      - API_PORT=9001
      - POSTGRES_HOST=sirius-postgres
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_DB=sirius
      - POSTGRES_PORT=5432
      - VALKEY_HOST=sirius-valkey
      - VALKEY_PORT=6379
      - RABBITMQ_URL=amqp://guest:guest@sirius-rabbitmq:5672/
      # - GO_ENV=development # Or production
  

• Operating System Environment: Docker containers can inherit environment variables from the shell session where docker compose up is executed, though this is less explicit for service-specific configuration.

• Go Code: The Go application would typically use the os package (e.g., os.Getenv("API_PORT")) or a configuration library (like Viper) to read these environment variables at runtime.

• Key Variables (Examples - derived from typical Go API needs and inter-service communication within Sirius Scan):

  • API_PORT: Port on which the Fiber API server listens (e.g., 9001).
  • POSTGRES_HOST: Hostname for the PostgreSQL service (e.g., sirius-postgres).
  • POSTGRES_USER: Username for PostgreSQL.
  • POSTGRES_PASSWORD: Password for PostgreSQL.
  • POSTGRES_DB: Database name in PostgreSQL (e.g., sirius).
  • POSTGRES_PORT: Port for PostgreSQL (e.g., 5432).
  • VALKEY_HOST: Hostname for the Valkey service (e.g., sirius-valkey).
  • VALKEY_PORT: Port for Valkey (e.g., 6379).
  • RABBITMQ_URL: Connection string for RabbitMQ, used for publishing tasks or messages to sirius-engine.
  • GO_ENV or GIN_MODE (if Fiber uses similar conventions to Gin): To set a development or production mode, which might affect logging levels or error handling (e.g., development, production).

Check the sirius-api Go source code (especially main.go or any config package) to identify the exact environment variables it expects.

3.3. sirius-engine (Core Processing Engine)

3.3.1. Overview & Purpose

The sirius-engine is the central nervous system of Sirius Scan. Written in Go, it is responsible for orchestrating scanning tasks, managing distributed agents (app-agent), and running various specialized sub-applications that provide core functionalities like scanning (app-scanner) and terminal access (app-terminal). It receives tasks primarily via RabbitMQ from sirius-ui or sirius-api and coordinates the execution of these tasks using its sub-modules and agents.

As defined in docker-compose.yaml:

sirius-engine:
  build: ./sirius-engine/
  container_name: sirius-engine
  hostname: sirius-engine
  restart: always
  image: sirius-engine
  ports:
    - "5174:5174" # Likely a primary port for the engine itself
    - "50051:50051" # gRPC port for agent communication
  volumes:
    - ./sirius-engine:/engine
    - ../minor-projects/app-agent:/app-agent
    # - ../minor-projects/go-api:/go-api # Local Development
    # - ../minor-projects/app-scanner:/app-scanner # Local Development
    # - ../minor-projects/app-terminal:/app-terminal # Local Development
    # - ../minor-projects/nmap-db:/nmap-db # Local Development
  # depends_on:
  #   - rabbitmq # Dependency is implied for message consumption

(Source: Sirius/docker-compose.yaml)

The volume mounts for app-agent (and commented out ones for other minor-projects) indicate that these sub-applications are developed as potentially separate projects but are integrated into the engine's runtime environment.

3.3.2. Technology Stack

• Go (Golang): The primary language for its performance, concurrency, and suitability for system-level programming.
• gRPC: Used for communication with app-agent instances.
• RabbitMQ Client: For receiving tasks and publishing results/status.
• Air: For live reloading during development, as suggested by the .air.toml file and common Go development practices (though air itself is installed in the app-scanner context within the sirius-engine Dockerfile).

3.3.3. Docker Configuration (sirius-engine/Dockerfile)

The Sirius/sirius-engine/Dockerfile is responsible for setting up a comprehensive environment that not only runs the engine's Go code but also prepares and includes several key tools and sub-applications.

Key aspects of the Dockerfile:

# sirius-engine Dockerfile

FROM golang:latest
WORKDIR /engine

COPY .air.toml .air.toml # Air configuration for hot-reloading (likely for sub-apps)

RUN apt-get update -y
RUN apt-get upgrade -y
RUN apt-get install -y libpcap-dev libicu-dev # System dependencies

# NMAP Installation
RUN apt-get install -y nmap

# Rust Scan Installation
WORKDIR /tmp
RUN curl https://sh.rustup.rs -sSf | bash -s -- -y
ENV PATH="/root/.cargo/bin:${PATH}"
RUN cargo install --git https://github.com/RustScan/RustScan.git --branch master

# PowerShell Installation
WORKDIR /opt/microsoft/powershell
RUN ARCH=$(uname -m) && \
    case "$ARCH" in \
    "aarch64") \
    echo "Installing ARM64 version" && \
    wget https://github.com/PowerShell/PowerShell/releases/download/v7.5.0/powershell-7.5.0-linux-arm64.tar.gz && \
    tar -xvf powershell-7.5.0-linux-arm64.tar.gz \
    ;; \
    "x86_64") \
    echo "Installing AMD64 version" && \
    wget https://github.com/PowerShell/PowerShell/releases/download/v7.5.0/powershell-7.5.0-linux-x64.tar.gz && \
    tar -xvf powershell-7.5.0-linux-x64.tar.gz \
    ;; \
    *) \
    echo "Unsupported architecture: $ARCH" && exit 1 \
    ;; \
    esac
RUN chmod +x /opt/microsoft/powershell/pwsh
RUN ln -s /opt/microsoft/powershell/pwsh /usr/bin/pwsh

# Set up NSE directory structure (for Nmap Scripting Engine used by app-scanner)
RUN mkdir -p /opt/sirius/nse && \
    chown -R 1000:1000 /opt/sirius && \
    chmod -R 755 /opt/sirius

# App Repos - Cloning sub-application source code
WORKDIR /
RUN git clone https://github.com/SiriusScan/app-scanner.git
RUN git clone https://github.com/SiriusScan/app-terminal.git
RUN git clone https://github.com/SiriusScan/go-api.git # go-api likely contains shared code/models

WORKDIR /app-scanner # Setting WORKDIR for app-scanner specific setup
#DEV Dependencies for app-scanner (Air)
RUN go install github.com/air-verse/air@latest

# Add startup script and make it the entrypoint
RUN apt install dos2unix
COPY start.sh /
RUN dos2unix /start.sh
RUN chmod +x /start.sh

ENTRYPOINT ["/start.sh"]

(Source: Sirius/sirius-engine/Dockerfile)

Key takeaways from the Dockerfile:

• Tooling: Installs essential scanning tools (Nmap, RustScan) and PowerShell, making them available within the engine's container environment, likely for use by sub-applications.
• Sub-Module Integration: It directly clones the repositories for app-scanner, app-terminal, and go-api. This means the source code for these modules is present within the sirius-engine container.
• start.sh Script: This is the crucial ENTRYPOINT. The start.sh script (not provided but its actions can be inferred) is responsible for:
  • Initializing the main sirius-engine Go application.
  • Potentially building and running the cloned sub-applications (app-scanner, app-terminal). It might use air for hot reloading these sub-apps if they are Go-based and have an .air.toml configurations.
  • Managing the lifecycle of these internal applications.

3.3.4. Modular Design (Loading Sub-Applications)

The sirius-engine achieves its modularity by incorporating other specialized applications (app-scanner, app-terminal) directly into its operational environment. As indicated by the Dockerfile and docker-compose.yaml:

• app-scanner: Cloned into the container. Likely provides scanning capabilities, potentially invoked by the main engine process. The setup of /opt/sirius/nse suggests it heavily uses Nmap and its scripting engine.
• app-terminal: Cloned into the container. Provides terminal functionalities.
• app-agent: While its source might be volume-mounted from ../minor-projects/app-agent (for local dev), the engine is designed to communicate with app-agent instances via gRPC. The engine itself might not run the app-agent code directly but orchestrates and communicates with separate agent processes (which could also be running in other containers or on remote hosts).
• go-api (cloned repository): This is likely a shared Go library containing common code, models, and utilities used by both sirius-api (via its own build) and potentially sirius-engine or its sub-modules like app-scanner.

The start.sh script is the primary mechanism that brings these components to life within the sirius-engine container, managing their execution and potentially their inter-process communication if they run as separate processes within the container.

3.3.5. Communication

• Task Reception: Listens to RabbitMQ queues for tasks (e.g., scan requests, command execution requests) from sirius-ui (via its BFF) and sirius-api.
• Agent Communication: Communicates with app-agent instances via gRPC on port 50051. This is used for sending commands to agents and receiving results or status updates.
• Internal Orchestration: Manages and communicates with app-scanner and app-terminal (details depend on start.sh implementation - could be via IPC, local network calls if they expose ports, or direct function calls if they are run as libraries).
• Data Stores: Interacts with sirius-postgres and sirius-valkey for persisting and retrieving data related to scans, engine state, and agent information, likely using the shared go-api library.
• Result/Status Publishing: Publishes results, progress, and status updates back to RabbitMQ for consumption by other services.

3.3.6. Key Directory Structure

The Sirius/sirius-engine/ directory itself contains the core Go code for the engine. However, its operational environment inside the Docker container is more complex due to the integration of other tools and sub-applications as defined by its Dockerfile.

• Sirius/sirius-engine/ (Local Project Directory):

  • Contains the main Go source files for the sirius-engine application (e.g., main.go, packages for gRPC services, RabbitMQ listeners, task orchestration logic).
  • Dockerfile: Defines the build process for the sirius-engine image.
  • .air.toml: Potentially used if air is run directly for the main engine code, though start.sh manages the overall startup.
  • start.sh: The entrypoint script copied into the Docker image. This script is crucial as it initializes the engine and potentially manages the cloned sub-applications.

• Inside the Docker Container (as per sirius-engine/Dockerfile):

  • /engine/: The WORKDIR where the Sirius/sirius-engine/ contents are copied.
  • /app-scanner/: Cloned Git repository for app-scanner.
  • /app-terminal/: Cloned Git repository for app-terminal.
  • /go-api/: Cloned Git repository for the shared go-api library.
  • /opt/sirius/nse/: Directory structure for Nmap NSE scripts, managed by app-scanner.
  • Standard paths for installed tools like Nmap (/usr/bin/nmap), RustScan (/root/.cargo/bin/rustscan), PowerShell (/usr/bin/pwsh).

• Sub-Module Interaction: The sirius-engine Go code, running from /engine/, interacts with:

  • Cloned sub-applications (e.g., by executing their binaries if start.sh builds them, or via IPC if start.sh runs them as separate processes).
  • Tools like Nmap/PowerShell via system calls.
  • The go-api library as a Go module dependency.

Understanding the sirius-engine/Dockerfile and the start.sh script (even if its content needs to be inferred based on Dockerfile actions) is key to grasping the engine's internal structure and operation.

3.3.7. Local Development & Debugging

Developing sirius-engine and its integrated sub-applications (app-scanner, app-terminal) involves understanding its Dockerized environment and the start.sh script.

• Prerequisites: Go installed locally.
• Local Development of Sub-Modules (app-scanner, app-terminal, go-api):
  • Clone the respective sub-module repository locally (e.g., into ../minor-projects/app-scanner).
  • In Sirius/docker-compose.yaml, uncomment the volume mount for that sub-module under the sirius-engine service definition (e.g., - ../minor-projects/app-scanner:/app-scanner). This mounts your local code into the engine's container, overriding the version cloned by the sirius-engine/Dockerfile.
  • Changes to your local Go files for that sub-module will be picked up. If the sub-module has an .air.toml and start.sh is configured to use air for it, it should hot-reload within the sirius-engine container.
• Local Development of sirius-engine Core:
  • The Sirius/sirius-engine/ directory is already mounted by default in docker-compose.yaml (- ./sirius-engine:/engine).
  • If the main engine Go application (started by start.sh) is also configured with air (e.g., via the .air.toml in /engine/), changes to its core Go files will trigger a reload.
• start.sh Script: This script is central. It controls how the main engine and its sub-modules are built (if necessary, e.g., go build for sub-apps) and run inside the container. You might need to inspect or modify start.sh to understand or alter the development workflow for components within the engine.
• Viewing Logs: Crucial for debugging. Use docker compose logs -f sirius-engine. This will show output from the main engine, air (for any components it manages), and the sub-applications.
• Go Debugging (Delve): Similar to sirius-api, you can use Delve for debugging Go code within sirius-engine or its Go-based sub-modules.
  • You'd need to ensure Delve is available in the container and modify start.sh (or an .air.toml used by start.sh) to launch the target Go application with Delve.
  • Expose Delve's port (e.g., 2345) from the sirius-engine container in docker-compose.yaml.
  • Attach your IDE's debugger to the exposed port.
• Interacting with Tools: The engine has Nmap, RustScan, PowerShell, etc. The start.sh might make these available on the PATH within the container, or sub-modules might call them with full paths. You can use docker exec -it sirius-engine bash to get a shell inside the running container to manually test these tools or explore the environment.

3.3.8. Environment Variables

The sirius-engine (a Go application) is configured using environment variables. These are typically set in the docker-compose.yaml file for the sirius-engine service or passed to the Docker container at runtime.

• Setting via docker-compose.yaml: This is the most common method for development.

  # Example in docker-compose.yaml for sirius-engine
  services:
    sirius-engine:
      # ... other settings ...
      environment:
        - ENGINE_MAIN_PORT=5174 # Example port for engine-specific services
        - GRPC_AGENT_PORT=50051 # Port for gRPC communication with app-agents
        - RABBITMQ_URL=amqp://guest:guest@sirius-rabbitmq:5672/
        - POSTGRES_HOST=sirius-postgres
        - POSTGRES_USER=postgres
        - POSTGRES_PASSWORD=postgres
        - POSTGRES_DB=sirius
        - VALKEY_HOST=sirius-valkey
        # GO_ENV can be used to control behavior (e.g., logging level)
        - GO_ENV=development
        # Variables for sub-modules like app-scanner if they are configured via env vars
        # - SCANNER_DEFAULT_TIMEOUT=3600
  

• Reading in Go: The engine's Go code (and its sub-modules if they are also Go-based and configured via environment) would use the os package (e.g., os.Getenv("GRPC_AGENT_PORT")) or a configuration library to read these values.

• Key Environment Variables (Examples for sirius-engine):

  • ENGINE_MAIN_PORT: A primary port the engine might use for its own services (if any, besides gRPC).
  • GRPC_AGENT_PORT: The port on which the engine's gRPC server listens for connections from app-agent instances (e.g., 50051).
  • RABBITMQ_URL: Connection string for RabbitMQ, used for consuming tasks and publishing results/status.
  • POSTGRES_HOST, POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB: For connecting to the PostgreSQL database (likely via the shared go-api library).
  • VALKEY_HOST: For connecting to the Valkey key-value store.
  • GO_ENV: To control execution mode (e.g., development, production), which might affect logging, debugging features, etc.
  • Sub-module specific variables: If sub-applications like app-scanner or app-terminal are configured via environment variables, those would also be passed through sirius-engine's environment if start.sh propagates them.

Consult the sirius-engine's main.go, start.sh script, and any configuration packages or sub-module documentation to identify the precise set of environment variables required and how they are consumed.

3.4. go-api (Shared Backend SDK)

3.4.1. Overview & Purpose

The go-api project (github.com/SiriusScan/go-api) is a crucial shared Go library, or Software Development Kit (SDK), designed to be used by all Go-based backend services within the Sirius Scan ecosystem, primarily sirius-api and sirius-engine (including its sub-modules like app-scanner).

Its main purpose is to provide a centralized and standardized way to interact with shared backend infrastructure components, abstracting away common functionalities and ensuring consistency across the platform. This SDK acts as the single source of truth for critical backend operations, offering several benefits:

• Code Reusability: Reduces code duplication significantly by providing common functions for database access, message queuing, caching, etc.
• Consistency: Ensures that all backend services interact with infrastructure components (like databases and message queues) in a uniform way, using the same data models and patterns. This minimizes integration errors and simplifies debugging.
• Maintainability: Centralizes the logic for core interactions. If a database schema changes or a message queue protocol needs an update, modifications can often be made primarily within go-api, with consuming services benefiting from these updates by simply updating their dependency.
• Developer Productivity: Speeds up development of new backend features and services by providing ready-to-use, tested components for common backend tasks. Developers can focus on business logic rather than boilerplate infrastructure code.
• Simplified Onboarding: New developers working on backend services can quickly understand how to interact with core infrastructure by learning the go-api interfaces.

Key areas of abstraction include:

• Database Abstraction: Simplifies interactions with PostgreSQL through GORM.
• Message Queue Utilities: Provides helpers for robust RabbitMQ communication.
• Key-Value Store Access: Offers a standardized interface for Valkey.
• Core Data Models: Defines common Go structs representing data entities used across services (e.g., hosts, vulnerabilities).
• NVD Integration: Facilitates fetching and using data from the National Vulnerability Database to enrich scan findings.

By centralizing these functions, go-api is fundamental to the stability, consistency, and agile development of Sirius Scan's backend capabilities.

3.4.2. Technology Stack

The go-api library is built using a selection of robust and widely-used technologies in the Go ecosystem:

• Go (Golang): The primary programming language, chosen for its performance, concurrency features, and strong standard library, making it well-suited for backend SDK development.
• GORM (gorm.io): A popular Go ORM (Object-Relational Mapper) used for interacting with the PostgreSQL database. GORM simplifies database operations by allowing developers to work with Go structs instead of writing raw SQL queries, increases developer productivity, and provides features like schema migration, transaction management, and association handling.
• RabbitMQ Client Libraries: Standard Go libraries for the AMQP 0-9-1 protocol (e.g., github.com/streadway/amqp or a similar actively maintained fork) are used to interact with sirius-rabbitmq. These libraries provide the foundational tools for publishing messages and subscribing to queues.
• Valkey Client Libraries: Go libraries specifically designed for interacting with Valkey (or Redis-compatible stores) are used to connect to and perform operations against the sirius-valkey key-value store.
• Go Modules: go-api is structured as a Go module, which allows it to manage its own dependencies (like GORM and AMQP clients) and to be easily consumed as a versioned dependency by other Go projects (sirius-api in particular).

3.4.3. Key Functionalities

Based on its documentation (go-api/docs/documentation.md) and observed usage patterns, go-api provides the following key functionalities, typically organized into distinct packages:

• Database Layer (sirius/postgres):

  • Connection Management: Initializes and provides access to the global GORM database instance (*gorm.DB). This includes:
    • Handling of database connection parameters (e.g., host, port, user, password, DB name) typically read from environment variables.
    • Automatic connection retry mechanisms with exponential backoff to handle transient network issues.
    • Environment-aware configuration (e.g., different connection settings for Docker vs. local development environments).
    • Automatic schema initialization and migration capabilities using GORM's migration features, ensuring the database schema is up-to-date with model definitions.
    • Efficient connection pooling managed by GORM to optimize database resource usage.
  • Data Models (sirius/postgres/models):
    • Defines a comprehensive set of GORM structs representing the core data entities of Sirius Scan. These include Host, Vulnerability, Port, Service, Note, CPE (Common Platform Enumeration), and various CVE-related structures (e.g., for storing information from NVD).
    • These models define table names, columns, data types, relationships (e.g., one-to-many, many-to-many), and database constraints, serving as the ORM mapping layer.

• Business Logic Modules (Database Interactors):

  • Host Management (sirius/host): Provides an API for performing CRUD (Create, Read, Update, Delete) operations on host records stored in the database. For example:
    • GetHost(ip string) (sirius.Host, error): Retrieves a host by its IP address.
    • GetAllHosts() ([]sirius.Host, error): Fetches all host records.
    • AddHost(host sirius.Host) error: Adds a new host or updates an existing one.
    • Additional functionalities might include calculating host vulnerability statistics, managing associated ports and services, and computing risk scores or other metrics related to hosts.
  • Vulnerability Management (sirius/vulnerability): Offers functions for managing vulnerability data within the system:
    • GetVulnerability(vid string) (nvd.CveItem, error): Retrieves detailed vulnerability information, potentially by CVE ID.
    • CheckVulnerabilityExists(vid string) bool: Checks if a specific vulnerability is already known to the system.
    • AddVulnerability(vuln sirius.Vulnerability) error: Adds new vulnerability findings to the database.

• Infrastructure Integration Modules:

  • Queue System (sirius/queue): Encapsulates interactions with RabbitMQ.
    • MessageProcessor func(msg string): Defines a type for callback functions that process incoming messages.
    • Listen(qName string, messageProcessor MessageProcessor): Subscribes to a specified RabbitMQ queue and invokes the messageProcessor for each received message. This often involves running processors in separate goroutines for concurrent handling.
    • Send(qName string, message string) error: Publishes a message to a specified RabbitMQ queue.
    • Handles details like automatic queue declaration (ensuring queues exist before use), connection management to RabbitMQ, and basic error handling/logging for queue operations.
  • Key-Value Store (sirius/store): Provides a standardized way to interact with the Valkey cache.
    • KVStore interface: Defines a contract for key-value operations, promoting testability and potential future backend swaps.
    • SetValue(ctx context.Context, key, value string) error: Stores a value.
    • GetValue(ctx context.Context, key string) (ValkeyResponse, error): Retrieves a value. ValkeyResponse likely encapsulates the result and any metadata.
    • Close() error: Closes the connection to the Valkey store.
    • The implementation uses a Valkey client to connect to sirius-valkey.
  • NVD Integration (nvd): Contains logic for interacting with the National Vulnerability Database (NVD) API.
    • This is crucial for fetching up-to-date details about CVEs (Common Vulnerabilities and Exposures), such as severity scores (CVSS), descriptions, and references. This data enriches the vulnerability information identified by local scans.

• Utility Functions (utils) (if present):

  • This package would typically house common, reusable helper functions that don't fit neatly into the other categories. Examples could include:
    • Configuration loading helpers (e.g., reading specific environment variables with defaults).
    • Custom error handling types or wrappers.
    • Common logging setup or utility functions.
    • Data validation or sanitization routines.

3.4.4. Usage & Integration

The go-api SDK is a cornerstone of Sirius Scan's backend architecture, used extensively by sirius-api and sirius-engine:

• sirius-api (Backend API Service):

  • Integration Method: Consumes go-api as a standard Go module dependency, specified in its go.mod file. This allows sirius-api to depend on specific, versioned releases of go-api, promoting stable builds.
  • Usage Examples:
    • When an HTTP request to create a new host arrives at a sirius-api endpoint, the handler function would call host.AddHost(newHostInfo) from go-api to persist the host.
    • Endpoints for retrieving vulnerability details would use functions like vulnerability.GetVulnerability(cveID).
    • It utilizes go-api for all its interactions with PostgreSQL (via GORM models and connection management provided by go-api), Valkey (via the store package), and potentially for publishing messages to RabbitMQ if it needs to delegate long-running tasks or notify sirius-engine (e.g., via queue.Send()).
  • Local Development: The docker-compose.yaml for sirius-api includes a commented-out volume mount: - ../minor-projects/go-api:/go-api. Uncommenting this allows developers to mount a local clone of the go-api repository directly into the sirius-api container. This is invaluable when making changes to go-api and wanting to test them immediately with sirius-api without needing to publish a new version of go-api first.

• sirius-engine (Core Processing Engine & its Sub-Modules like app-scanner):

  • Integration Method: The sirius-engine/Dockerfile explicitly clones the go-api repository (e.g., git clone https://github.com/SiriusScan/go-api.git /go-api) into its build context. The main engine Go application and its Go-based sub-modules (like app-scanner, which is also cloned into the engine's image) then use go-api as a local library by adjusting their Go module paths or using replace directives in their go.mod files to point to this local copy.
    • This approach ensures that sirius-engine and its tightly coupled sub-modules are built with a specific, consistent version of go-api that is bundled directly within the engine's Docker image.
    • While simpler for the bundled environment of sirius-engine, it requires careful coordination if go-api undergoes breaking changes, as the sirius-engine image would need to be rebuilt to pick up the new go-api version.
  • Usage Examples:
    • When sirius-engine receives a scan task from RabbitMQ (via queue.Listen() from go-api), it might first retrieve target details using host.GetHost().
    • After app-scanner (a sub-module of the engine) completes a scan, it would use go-api functions (e.g., host.AddHost to update host details, port.AddPort, service.AddService, vulnerability.AddVulnerability) to parse and store the scan results in the PostgreSQL database.
    • The engine might use store.SetValue() from go-api to cache intermediate scan states or frequently accessed data in Valkey.
    • It relies on go-api's queue package for consuming tasks from RabbitMQ and potentially for publishing status updates or results back to other queues.
  • Local Development: Similar to sirius-api, the docker-compose.yaml for sirius-engine has a commented-out volume mount for go-api. Enabling this mount allows developers to test local changes to go-api directly within the sirius-engine's operational environment.

This shared SDK strategy is pivotal. It ensures that all backend Go components operate on the same data models and use consistent, tested methods for accessing shared infrastructure resources, which is critical for the overall system's reliability and maintainability.

3.4.5. Development & Testing of go-api

Developing and maintaining go-api as a standalone SDK involves its own set of practices to ensure its quality and reliability:

• Repository Structure: go-api is maintained in its own Git repository (github.com/SiriusScan/go-api), allowing for independent versioning and development lifecycle.
• Unit Testing: Each package and significant function within go-api (e.g., database interaction logic, queue sending/receiving, NVD API calls) should be covered by comprehensive unit tests. Go's built-in testing package is used for this. Mocks and interfaces are employed to isolate components during testing (e.g., mocking database connections or RabbitMQ interactions).
• Integration Testing: Beyond unit tests, go-api likely requires integration tests that verify its interaction with actual external services (PostgreSQL, RabbitMQ, Valkey). These tests might be run in CI/CD pipelines against test instances of these services, or locally using Docker Compose to spin up the necessary dependencies.
• Versioning: go-api should follow semantic versioning (SemVer). New versions are tagged in its repository (e.g., v1.0.0, v1.1.0). Consuming services like sirius-api can then specify the version of go-api they depend on in their go.mod file.
• Dependency Management: go-api itself manages its dependencies (like GORM, AMQP clients) using Go Modules. These are specified in its own go.mod file.
• Documentation: Maintaining clear documentation (like the go-api/docs/documentation.md) is essential for developers of consuming services to understand how to use the SDK, its available functions, data models, and any conventions.

Careful testing and versioning of go-api are crucial because changes to it can impact multiple backend services. Breaking changes must be managed through major version increments and communicated clearly to the teams working on consuming services.

4. Sub-Modules & Agents

4.1. app-scanner

4.1.1. Purpose and Functionality

The app-scanner sub-module, running within the sirius-engine container, is the primary component responsible for executing network scans. It leverages well-known scanning tools like Nmap and RustScan to discover hosts, open ports, services, and potential vulnerabilities based on Nmap Scripting Engine (NSE) scripts.

Key Functionalities:

• Port Scanning: Utilizes tools like Nmap and RustScan for fast and comprehensive port scanning to identify open TCP and UDP ports on target hosts.
• Service Detection: Employs Nmap's version detection capabilities (-sV) to identify the services running on discovered open ports and their versions.
• OS Detection: Uses Nmap's OS detection (-O) to attempt to identify the operating system of target hosts.
• NSE Script Execution: Manages and executes Nmap NSE scripts for vulnerability detection, further enumeration, or specific checks. This is a core feature for identifying potential issues.
• Output Parsing: Parses the XML output from Nmap (and potentially other tools) to extract structured data about hosts, ports, services, and script results.
• Integration with sirius-engine: Receives scan tasks from the main engine (which itself gets them from RabbitMQ), executes them, and returns structured results to the engine for persistence and further processing.

It is typically a Go application, cloned into the sirius-engine container during its build process (or volume-mounted for local development). The sirius-engine/Dockerfile installs Nmap and RustScan, making them available for app-scanner to use. The start.sh script within sirius-engine likely manages the execution of app-scanner.

4.1.2. NSE (Nmap Script Engine) Management

The Nmap Scripting Engine (NSE) is a powerful Nmap feature allowing users to write and use scripts (in Lua) to automate a wide array of networking tasks. Within Sirius Scan, app-scanner leverages NSE for advanced vulnerability detection, deeper service analysis, and more targeted information gathering beyond standard Nmap checks.

The management of NSE scripts is a sophisticated process handled by the app-scanner/internal/nse Go package. This system ensures that NSE scripts are consistently managed, versioned, and synchronized across the platform.

Key Aspects of NSE Management in app-scanner:

• Centralized Script Storage & Repositories:

  • The primary directory for NSE scripts within the sirius-engine Docker container is /opt/sirius/nse/. This path is configured during the sirius-engine Docker build.
  • Scripts are organized into repositories (e.g., sirius-nse cloned from https://github.com/SiriusScan/sirius-nse.git). Each repository can contain multiple Lua scripts and has its own manifest file.

• Manifest System for Script Definition & Discovery:

  • Repository List Manifest (/opt/sirius/nse/manifest.json): This top-level JSON file lists all external NSE script repositories that app-scanner should manage (containing their names and Git URLs).
  • Repository Manifest (<repo_name>/manifest.json): Each individual script repository contains its own manifest. This file is the source of truth for scripts within that repository, detailing metadata for each script, such as its name, path within the repository, supported protocols, description, author, and tags (e.g., "vulners": {"name": "vulners", "path": "scripts/vulners.nse", ...}).
  • Global Manifests in Valkey: To ensure consistency across potentially multiple sirius-engine instances or for dynamic updates, app-scanner uses Valkey as a central store for global script information:
    • nse:repo-manifest: Stores the global list of repositories, taking precedence over the local file-based repository list.
    • nse:manifest: Stores a consolidated global manifest of all scripts from all repositories, including their metadata.
  • Local In-Memory Manifests: During runtime, app-scanner maintains local, in-memory copies of these manifests for efficient access during scan configuration and execution.

• Synchronization Mechanism:

  • app-scanner implements a robust synchronization system to keep scripts and their manifests up-to-date. The order of precedence is: Global Valkey Manifests > Local Filesystem Manifests > Built-in Defaults.
  • The SyncManager within the nse package handles this synchronization. It clones/pulls script repositories from Git, reads local manifest files, and reconciles them with the global manifests stored in Valkey.
  • This ensures that app-scanner always uses the most current and authoritative versions of scripts and their definitions.

• Script Content Storage in Valkey:

  • Beyond manifests, the actual Lua code of NSE scripts can also be stored in Valkey under keys like nse:script:<script_name>.
  • This allows for dynamic updates to script content without needing to rebuild Docker images or manually distribute script files. When Nmap is invoked, app-scanner can provide it with the script content retrieved from Valkey.

• Usage During Scans:

  • When a user configures a scan that includes NSE scripts, sirius-engine passes this configuration to app-scanner.
  • app-scanner, leveraging its synchronized script manifests, selects the appropriate NSE scripts.
  • It constructs Nmap command-line arguments to include these scripts (e.g., --script <script_name_or_category>) and their arguments.
  • Nmap executes these scripts against the target(s).

• Processing NSE Results:

  • The output from NSE scripts (usually embedded in Nmap's XML output) is parsed by app-scanner.
  • This output often contains crucial vulnerability information, detailed service enumerations, or specific findings relevant to the script's purpose.
  • app-scanner then transforms these findings into a structured format suitable for storage in sirius-postgres (via go-api) and display in sirius-ui.

• Configuration & Extensibility:

  • The system is designed for extensibility, allowing new NSE script repositories to be added by updating the nse:repo-manifest in Valkey or the local /opt/sirius/nse/manifest.json.
  • Users can select specific scripts or categories of scripts for their scans, providing fine-grained control over the scanning process.

This comprehensive NSE management system ensures that Sirius Scan can effectively utilize the power of Nmap's scripting capabilities for in-depth security assessments, with a flexible and synchronized approach to script handling.

4.2. app-terminal

4.2.1. Purpose and Functionality

The app-terminal sub-module, developed as a Go application within the minor-projects/app-terminal repository, provides a web-based PowerShell terminal interface accessible through sirius-ui. It is designed to enable users to execute commands securely on the system where sirius-engine is running, or potentially on remote systems if command execution is proxied through agents in the future.

Key Functionalities (as per app-terminal/README.md):

• PowerShell Execution Backend: Leverages PowerShell Core (pwsh) for command execution. PowerShell is installed within the sirius-engine Docker container specifically for use by app-terminal.
• Secure Command Handling: Aims to execute commands in isolated PowerShell sessions, with session cleanup after execution.
• Message Queue Integration: Communicates asynchronously with the broader Sirius Scan ecosystem (primarily sirius-ui via sirius-engine) using RabbitMQ.
  • Receives command execution requests from a RabbitMQ queue.
  • Sends command output (stdout, stderr) and status back via RabbitMQ queues.
• Command Management: internal/terminal/manager.go is responsible for handling incoming command requests from the queue and managing their lifecycle.
• PowerShell Interface: internal/terminal/powershell.go provides the abstraction layer for interacting with the PowerShell runtime, executing commands, and capturing output.
• Output Formatting: Processes and formats the raw output from PowerShell to be suitable for display in the sirius-ui terminal (which uses xterm.js).
• Error Handling: Implements error handling for PowerShell execution issues, queue communication problems, and session management.

Integration with sirius-engine:

• app-terminal is cloned into the sirius-engine Docker container during its build process (from https://github.com/SiriusScan/app-terminal.git).
• The start.sh script within sirius-engine is responsible for launching and managing the app-terminal Go application.
• sirius-engine acts as a router for terminal commands: when a user enters a command in sirius-ui's terminal, the request (typically via tRPC to the UI's BFF) is put onto a RabbitMQ queue. sirius-engine consumes this message and, if it's a command for the local engine environment, forwards it to app-terminal for execution.

The app-terminal provides a crucial capability for direct interaction with the engine's environment or potentially managed endpoints, supporting tasks like manual enumeration, configuration checks, or diagnostic actions.

4.3. app-agent

4.3.1. Purpose and Functionality

The app-agent is a Go application designed to run on remote hosts, acting as a client to the sirius-engine's gRPC server. Its primary purpose is to extend the capabilities of Sirius Scan by allowing sirius-engine to delegate tasks, such as command execution or localized scanning, to these distributed agents. This enables Sirius Scan to interact with and assess systems that may not be directly accessible from the central sirius-engine instance.

Key Functionalities (based on app-agent/cmd/agent/README.md):

• Remote Command Execution: The core feature is its ability to receive shell command strings from sirius-engine via a gRPC call (ExecuteCommand) and execute these commands on the host where the agent is running. It then captures the standard output and standard error of the command and sends them back to the engine.
• Connectivity Testing: Implements a Ping gRPC method that sirius-engine can use to check the agent's reachability and operational status.
• Persistent Connection: Establishes a gRPC connection to the sirius-engine (server address configured via SERVER_ADDRESS environment variable, defaulting to localhost:50051).
• Identification: Identifies itself to the engine using an AGENT_ID (configurable via environment variable, defaults to the hostname), allowing the engine to manage and target specific agents.
• Periodic Updates (Example): The example agent demonstrates the capability to perform actions periodically (e.g., running ls -la every 10 seconds and sending results), suggesting a model for continuous monitoring or data collection tasks.
• Structured Logging: Utilizes zap for structured logging, aiding in diagnostics and monitoring of the agent's behavior.
• Graceful Shutdown: Designed to handle termination signals for a clean shutdown process.

Deployment and Development Context:

• app-agent is developed in the minor-projects/app-agent repository.
• Unlike app-scanner or app-terminal, app-agent is not typically cloned into the sirius-engine Docker image. Instead, it's intended to be built and deployed as a separate binary or container on target machines where remote operations are needed.
• The docker-compose.yaml for sirius-engine includes a volume mount for ../minor-projects/app-agent:/app-agent. This is primarily for a development scenario where sirius-engine might run an instance of app-agent locally for testing gRPC interactions, or if the engine itself needs to act as an agent for some reason, though its primary role is server.

Security Considerations: The provided app-agent README explicitly states it's a basic example and lacks production-grade security features such as TLS for gRPC, authentication/authorization, command validation, and sandboxing. These would be critical additions for a secure deployment.

4.3.2. gRPC Communication with sirius-engine

Communication between app-agent instances and sirius-engine is facilitated by gRPC, a high-performance, open-source universal RPC framework.

• Roles:

  • sirius-engine: Acts as the gRPC server. It defines and implements the gRPC services (e.g., AgentService with methods like Ping and ExecuteCommand). It listens for incoming gRPC connections from agents, typically on port 50051 (as suggested by agent defaults and engine's Docker Compose port mapping).
  • app-agent: Acts as the gRPC client. It initiates a connection to the sirius-engine's gRPC server address.

• Protocol Definition (.proto files):

  • The gRPC services, methods, and message structures are defined in Protocol Buffer (.proto) files. The app-agent repository contains proto/hello/hello.proto (an example name) which would define services like:

    service AgentService {
      rpc Ping (PingRequest) returns (PingResponse);
      rpc ExecuteCommand (CommandRequest) returns (stream CommandResponse); // Example: streaming for output
    }
    
    message CommandRequest {
      string agent_id = 1;
      string command = 2;
    }
    
    message CommandResponse {
      string output = 1;
      string error = 2;
      bool completed = 3;
    }
    // ... other message definitions
    

  • Both sirius-engine and app-agent would generate gRPC server/client stubs and data types from these .proto files in Go using protoc and Go gRPC plugins. This ensures type safety and consistency in communication.

• Communication Flow (Example: ExecuteCommand):

  1. sirius-engine decides to execute a command on a specific agent (identified by AGENT_ID).
  2. The engine, using its gRPC client stub for the AgentService (or by managing connections to agents), makes an ExecuteCommand RPC call to the target app-agent. The CommandRequest would contain the command string and potentially other parameters.
  3. The app-agent (gRPC client) receives this request via its gRPC server implementation of ExecuteCommand.
  4. The agent executes the command on its host system.
  5. The agent captures stdout/stderr and streams CommandResponse messages back to sirius-engine. This streaming is useful for long-running commands or commands that produce continuous output.
  6. sirius-engine receives the stream of responses and processes the output (e.g., stores it, forwards it to the UI via RabbitMQ).

• Connection Management: Agents establish a connection to the engine. The engine needs to manage connections from multiple agents if it is to orchestrate a fleet of them. This might involve keeping track of active agents, their capabilities, and their gRPC channel information.

• Advantages of gRPC: Chosen for its efficiency (uses HTTP/2 and Protocol Buffers), strong typing, support for streaming, and suitability for microservice communication.

This gRPC interface is fundamental for sirius-engine to control and leverage distributed app-agent instances for various scanning and operational tasks.

5. Data Persistence & Storage

Sirius Scan relies on two primary data stores for persistence and caching: PostgreSQL for structured relational data and Valkey for key-value storage, often used for caching, session management, and managing dynamic configurations like NSE script manifests.

5.1. PostgreSQL (sirius-postgres)

PostgreSQL serves as the main relational database, storing the core structured data of the application. This includes information about scanned assets, vulnerabilities, scan configurations, and potentially user data.

5.1.1. Role and Data Stored

• Primary Datastore: The authoritative source for persistent, relational data.
• Hosts Information: Details about discovered and scanned hosts (IP addresses, hostnames, operating systems, MAC addresses, etc.).
• Ports & Services: Information about open ports on hosts and the services running on them (port number, protocol, service name, version, state).
• Vulnerabilities: Detailed records of identified vulnerabilities (CVE IDs, descriptions, severity, affected hosts/ports, remediation status).
• Scan Configurations & Results: Parameters for scans performed and the summarized results associated with them.
• User and Access Management: (If implemented) User accounts, roles, and permissions.
• Audit Trails: Logs of significant actions performed within the system.

The go-api SDK, particularly its GORM models, defines and interacts with this schema.

5.1.2. Schema Overview (Conceptual)

While a full, detailed Entity-Relationship Diagram (ERD) is extensive, the conceptual schema revolves around several key entities and their relationships, managed via GORM models in go-api:

• Hosts: The central entity representing a scanned asset.
  • Attributes: ID, IPAddress, Hostname, OS, MACAddress, Status (e.g., up, down), CreatedAt, UpdatedAt.
• Ports: Represents network ports discovered on hosts.
  • Attributes: ID, PortNumber, Protocol (TCP/UDP), State (open, closed, filtered).
• Services: Describes services running on specific ports.
  • Attributes: ID, Name, Version, Product, ExtraInfo, Banner.
• Vulnerabilities: Stores information about identified vulnerabilities.
  • Attributes: ID, CVEID (e.g., "CVE-2023-1234"), Name/Title, Description, Severity (e.g., Critical, High, Medium, Low, Info), CVSSScore, References (URLs to advisories).
• HostPorts (Join Table for Hosts ↔ Ports/Services):
  • Establishes a many-to-many relationship between Hosts and Ports.
  • Often, service information is directly linked here or through a specific HostPortService record if a single port can host multiple services (less common for basic TCP/UDP, but possible with virtual hosting or more complex protocols).
  • Likely contains HostID, PortID, and potentially ServiceID if services are modeled as distinct entities associated with a host-port pair.
• HostVulnerabilities (Join Table for Hosts ↔ Vulnerabilities):
  • Establishes a many-to-many relationship between Hosts and Vulnerabilities.
  • Attributes: HostID, VulnerabilityID, Port (the specific port on the host where the vulnerability applies, if applicable), Status (e.g., new, confirmed, false positive, remediated), FirstSeen, LastSeen.
• CPEs (Common Platform Enumeration): Stores CPE identifiers found on hosts/services, which can be linked to vulnerabilities.
• Notes: Allows users to add notes or comments to hosts, vulnerabilities, or other entities.
• ScanMetadata / ScanJobs: Tables to store information about scan configurations (targets, options used) and their execution status/history.

The go-api library uses GORM to define these models as Go structs and manages their migrations, ensuring the database schema aligns with the application's data structures. The Sirius/documentation/README.testing.md document highlights the importance of these many-to-many relationships (Hosts-Ports, Hosts-Vulnerabilities) and provides context on how they are structured and tested.

5.2. Valkey (sirius-valkey)

Valkey (a fork of Redis) is employed as a high-performance in-memory key-value store. It is primarily used for caching, session management, and storing transient or frequently accessed data that doesn't require the relational structure of PostgreSQL.

5.2.1. Role and Data Stored

• Caching: Caching frequently accessed database query results to reduce load on PostgreSQL and improve response times (e.g., host details, vulnerability information that doesn't change often).
• Session Management: If sirius-ui or sirius-api implement user sessions, Valkey is a suitable store for session data.
• Task/Job Queuing (Metadata/Status): While RabbitMQ handles the primary task queuing, Valkey might be used to store metadata about jobs, their current status, or intermediate results for quick lookups before final persistence in PostgreSQL.
• Rate Limiting: Storing counters for API rate limiting or other attempt-based security features.
• Distributed Locks: If needed for coordinating actions between multiple instances of a service.
• NSE Script Management (as detailed in app-scanner/internal/nse/README.md):
  • Global Repository Manifest (nse:repo-manifest): Stores a JSON string representing the list of all NSE script repositories (names, Git URLs). This allows dynamic updates to the repositories app-scanner should manage.
  • Global Script Manifest (nse:manifest): Stores a JSON string representing a consolidated manifest of all scripts from all repositories, including their metadata (name, path, description, tags). This is the primary source of truth for available scripts.
  • Individual Script Content (nse:script:<script_name>): Stores the actual Lua code of individual NSE scripts. This enables app-scanner to fetch and use the latest script versions without requiring image rebuilds or file deployments, as scripts can be updated directly in Valkey. For example, the content of vulners.nse would be stored under the key nse:script:vulners.
• Agent Status/Heartbeats: sirius-engine might use Valkey to keep track of connected app-agent instances, their last heartbeat times, and current status for quick lookups.

The go-api SDK provides an interface and implementation for interacting with Valkey, abstracting the direct client library calls for consuming services.

6. Inter-Service Communication

Sirius Scan employs a variety of communication patterns and technologies to enable its microservices to collaborate effectively. These include asynchronous messaging via RabbitMQ, type-safe APIs with tRPC, traditional RESTful APIs, and high-performance RPC with gRPC.

6.1. RabbitMQ (Asynchronous Messaging)

RabbitMQ is the backbone for asynchronous communication and task delegation within Sirius Scan. It decouples services, improves resilience (e.g., if a consuming service is temporarily down, messages remain queued), and enables scalable processing of background tasks like vulnerability scans and command executions.

Key Roles & Usage:

• Task Queuing: Long-running or resource-intensive tasks initiated by users (e.g., via sirius-ui) are typically offloaded to sirius-engine by publishing messages to specific RabbitMQ queues. This allows the user-facing services (sirius-ui, sirius-api) to remain responsive.
• Decoupling Services: Producers of messages don't need to know the exact location or number of consumers. Consumers process messages at their own pace.
• Load Balancing: If multiple instances of a consumer service (e.g., sirius-engine) are running, RabbitMQ can distribute messages among them.
• Event Notifications: Services can publish events (e.g., scan completed, new vulnerability found) to RabbitMQ, allowing other interested services to subscribe and react accordingly.

Conceptual Queues and Message Flows (Payloads are typically JSON):

1. Scan Initiation & Management:

   • scan_jobs_queue (Producer: sirius-ui BFF -> Consumer: sirius-engine):
     • Purpose: To submit new vulnerability scan requests to the engine.
     • Example Payload:

       {
         "scan_id": "uuid-scan-123",
         "user_id": "user-abc",
         "targets": ["192.168.1.0/24", "testserver.com"],
         "scan_type": "full_scan_aggressive", // or specific profile name
         "nmap_options": {
           "scripts": ["vulners", "http-enum"],
           "timing_template": "T4",
           "service_version": true
         },
         "rustscan_options": {
           "port_range": "1-65535"
         }
       }
       

   • scan_control_queue (Producer: sirius-ui BFF -> Consumer: sirius-engine):
     • Purpose: To send control commands for ongoing scans (e.g., pause, resume, stop).
     • Example Payload:

       {
         "scan_id": "uuid-scan-123",
         "action": "pause" // "resume", "stop"
       }
       

   • scan_status_updates_queue (Producer: sirius-engine -> Consumer: sirius-ui BFF via WebSocket/SSE relay):
     • Purpose: To provide real-time progress updates for ongoing scans to the UI.
     • Example Payload:

       {
         "scan_id": "uuid-scan-123",
         "status": "running", // "pending", "paused", "completed", "error"
         "progress_percentage": 45,
         "current_target": "192.168.1.10",
         "message": "Scanning host 10 of 254..."
       }
       

   • scan_results_notification_queue (Producer: sirius-engine -> Consumer: sirius-ui BFF and/or sirius-api):
     • Purpose: To notify that scan results (or significant partial results) are available for a given scan, often indicating data has been persisted to PostgreSQL.
     • Example Payload:

       {
         "scan_id": "uuid-scan-123",
         "status": "completed",
         "result_summary_url": "/api/v1/scans/uuid-scan-123/summary", // Optional
         "message": "Scan completed. Results available."
       }
       

       (The UI would then typically fetch detailed results via tRPC/API calls to sirius-api or its own BFF which queries the DB).

2. Terminal Command Execution:

   • terminal_commands_queue (Producer: sirius-ui BFF -> Consumer: sirius-engine):
     • Purpose: To send commands entered in the UI terminal to the engine for execution by app-terminal (or an agent).
     • Example Payload:

       {
         "session_id": "uuid-session-xyz",
         "user_id": "user-abc",
         "command_string": "ls -la /tmp",
         "target_context": {
           // Optional, for routing to agent etc.
           "type": "agent",
           "agent_id": "agent-007"
         }
       }
       

   • terminal_output_queue (Producer: sirius-engine (from app-terminal or agent) -> Consumer: sirius-ui BFF via WebSocket/SSE relay):
     • Purpose: To stream command output back to the specific UI terminal session.
     • Example Payload (could be a series of messages for one command):

       {
         "session_id": "uuid-session-xyz",
         "data_type": "stdout", // "stderr", "status"
         "payload": "total 4\ndrwxr-xr-x 2 root root 4096 Jan 1 10:00 .\n",
         "is_final_chunk": false
       }
       

3. Agent Tasking (Conceptual - if not solely gRPC from engine):

   • While direct gRPC is primary for engine-to-agent, RabbitMQ could be used by the engine to broadcast tasks to groups of agents or for agents to report asynchronous results if direct gRPC streaming isn't suitable for a very long-running agent task.
   • agent_tasks_queue_group_<group_id> or agent_task_queue_<agent_id> (Producer: sirius-engine -> Consumer: app-agent instances)
   • agent_results_queue (Producer: app-agent instances -> Consumer: sirius-engine)

Implementation Details:

• go-api SDK: Go-based services (sirius-engine, sirius-api, and sub-modules like app-scanner, app-terminal) utilize the sirius/queue package from the go-api SDK. This package provides Send() and Listen() functions, abstracting RabbitMQ connection management, channel handling, queue declaration, and message publishing/consumption logic.
• Exchanges and Bindings: While not explicitly detailed in all component READMEs, a typical RabbitMQ setup involves exchanges (e.g., direct, topic, fanout) to route messages from producers to appropriate queues based on routing keys and binding rules. For instance, scan_jobs_queue might be bound to a direct exchange where sirius-ui publishes messages with a specific routing key.
• Message Durability & Persistence: For critical tasks like scan jobs, messages are likely published as persistent, and queues are declared as durable to survive RabbitMQ restarts. Status updates might be non-persistent if losing a few is acceptable.
• Error Handling & Retries: Robust consumers implement error handling, potentially using dead-letter exchanges (DLX) and retry mechanisms for messages that fail processing.

RabbitMQ's role is crucial for the scalability and resilience of Sirius Scan's distributed architecture, enabling efficient handling of potentially numerous and long-duration backend operations.

6.2. tRPC (UI Frontend to Next.js Backend)

tRPC is utilized within the sirius-ui service to facilitate highly efficient and type-safe communication between its React frontend components and its Next.js backend (which acts as a Backend-for-Frontend, or BFF). This approach significantly improves developer experience and reduces runtime errors related to client-server data contracts.

Key Roles & Benefits:

• End-to-End Type Safety: This is the primary advantage. TypeScript types defined on the server (Next.js API routes) are automatically inferred on the client (React components). This means if a server-side API changes its request parameters or response structure, TypeScript will immediately flag errors in the frontend code during development, preventing mismatches that would otherwise lead to runtime failures.
• Improved Developer Experience: Developers get autocompletion for API procedures and their inputs/outputs directly in their frontend code. Refactoring server-side procedures (e.g., renaming, changing parameters) can often be done with IDE support that also updates client-side usages.
• No Code Generation Needed (for client-server calls): Unlike GraphQL or OpenAPI-based approaches that often require a code generation step to create client-side SDKs or type definitions, tRPC achieves type safety through TypeScript inference, simplifying the build process.
• Simplified Data Fetching: The tRPC client provides hooks (e.g., useQuery for data fetching, useMutation for data modification) that integrate seamlessly with React, handling loading states, errors, and caching with minimal boilerplate.

Implementation in sirius-ui:

• Defining Procedures (Backend):

  • tRPC procedures are defined in the sirius-ui/src/server/api/routers/ directory. Each file (e.g., terminal.ts, agent.ts, scan.ts) typically exports a router defining a set of related procedures.

  • Each procedure specifies its input validation (often using libraries like Zod) and the resolver function that contains the server-side logic.

  • Example conceptual procedure in sirius-ui/src/server/api/routers/scan.ts:

    import { z } from "zod";
    import { createTRPCRouter, publicProcedure } from "~/server/api/trpc";
    // import { rabbitMQService } from '~/server/services/rabbitmq'; // Example service
    
    export const scanRouter = createTRPCRouter({
      startScan: publicProcedure
        .input(
          z.object({
            targets: z.array(z.string()),
            scanProfile: z.string(),
          })
        )
        .mutation(async ({ input, ctx }) => {
          // ctx might contain db access, session info, etc.
          console.log("Received scan request:", input);
          // 1. Validate input further if necessary
          // 2. Construct a message for sirius-engine
          const scanJobMessage = {
            scan_id: `uuid-scan-${Date.now()}`,
            user_id: ctx.session?.user?.id, // Example: if auth is used
            targets: input.targets,
            scan_type: input.scanProfile,
            // ... other parameters
          };
          // 3. Publish to RabbitMQ (example call)
          // await rabbitMQService.publishToQueue('scan_jobs_queue', scanJobMessage);
          // 4. Return a confirmation or initial status
          return {
            message: "Scan job submitted successfully",
            scan_id: scanJobMessage.scan_id,
          };
        }),
      // ... other procedures like getScanStatus, getScanResults etc.
    });
    

• Root Router: These individual routers are merged into a single appRouter in sirius-ui/src/server/api/root.ts.

• API Handler: An API route, typically sirius-ui/src/pages/api/trpc/[trpc].ts, handles all incoming tRPC requests and routes them to the appropriate procedure in the appRouter.

• Consuming Procedures (Frontend):

  • React components in sirius-ui/src/components/ or sirius-ui/src/pages/ use hooks provided by the tRPC client to call backend procedures.

  • Example usage in a React component:

    import { api } from "~/utils/api"; // Path to the tRPC client
    
    function ScanComponent() {
      const startScanMutation = api.scan.startScan.useMutation({
        onSuccess: (data) => {
          console.log("Scan submitted! ID:", data.scan_id);
          // navigate to results page or show notification
        },
        onError: (error) => {
          console.error("Failed to start scan:", error.message);
        },
      });
    
      const handleStartScan = () => {
        startScanMutation.mutate({
          targets: ["127.0.0.1"],
          scanProfile: "quick_scan",
        });
      };
    
      return (
        <button
          onClick={handleStartScan}
          disabled={startScanMutation.isLoading}
        >
          {startScanMutation.isLoading ? "Submitting..." : "Start Scan"}
        </button>
      );
    }
    

Role in Orchestration:

The Next.js backend (BFF) part of sirius-ui, powered by these tRPC procedures, often acts as an orchestrator:

• It receives requests from the frontend.
• Performs initial validation and UI-specific logic.
• For tasks requiring asynchronous processing (like starting a scan or executing a terminal command that might take time), it publishes messages to RabbitMQ queues for sirius-engine to consume.
• For fetching data that sirius-api owns (e.g., detailed host information, historical scan results), the tRPC procedure might make a REST call to sirius-api and then transform or relay the data to the frontend.
• It can also directly interact with sirius-postgres using Prisma (as suggested by sirius-ui/prisma/schema.prisma) for UI-specific data needs that don't fit the sirius-api domain, or for caching/aggregating data.

tRPC provides a clean, type-safe, and efficient communication layer that is internal to the sirius-ui service, streamlining its frontend-to-BFF interactions.

6.3. RESTful APIs (sirius-api)

The sirius-api service (Sirius/sirius-api) functions as the primary backend API gateway for core data operations and business logic that is not directly handled by the UI's BFF or the specialized sirius-engine. It provides a set of RESTful HTTP endpoints for creating, reading, updating, and deleting (CRUD) key data entities like hosts, vulnerabilities, scan results, and potentially user management if centralized here.

Key Roles & Characteristics:

• Centralized Data Access Logic: Provides a consistent API layer over the primary data stores (sirius-postgres and sirius-valkey).
• Business Logic Hub: Implements business rules and data processing related to the core entities (e.g., aggregating vulnerability data for a host, calculating risk scores, managing scan result lifecycles).
• Technology Stack: Built using Go and the Fiber web framework, chosen for its high performance and Express.js-like API, making it relatively easy to define routes and handlers.
• Database Interaction: Heavily utilizes the go-api SDK (specifically packages like sirius/host, sirius/vulnerability, sirius/postgres) for all interactions with sirius-postgres (via GORM) and sirius/store for sirius-valkey.
• RESTful Principles: Adheres to REST principles by:
  • Using standard HTTP methods: GET (retrieve), POST (create), PUT (update/replace), DELETE (remove), PATCH (partial update).
  • Employing resource-based URLs (e.g., /api/v1/hosts, /api/v1/hosts/{host_id}/vulnerabilities).
  • Using standard HTTP status codes to indicate outcomes (e.g., 200 OK, 201 Created, 400 Bad Request, 404 Not Found, 500 Internal Server Error).
  • Typically exchanging data in JSON format.
• Authentication & Authorization: This service is a logical place for enforcing authentication (e.g., validating JWT tokens passed in Authorization headers) and authorization (checking if the authenticated user has permission to perform the requested action on the resource).
• API Versioning: Endpoint paths often include a version (e.g., /api/v1/) to allow for future API evolution without breaking existing clients.
• Port: Runs on port 9001 by default, as configured in docker-compose.yaml.

Interaction with Other Services:

• sirius-ui: The UI (either its BFF or client-side components, depending on design choices) is a primary consumer of sirius-api. It makes HTTP requests to sirius-api to fetch data for display (e.g., list of hosts, details of a vulnerability) and to submit changes (e.g., adding a note to a host, updating a vulnerability's status).
  • The sirius-ui's BFF (tRPC procedures) might call sirius-api endpoints server-side to aggregate data or perform actions before relaying results to the frontend.
• sirius-engine: While the engine primarily receives tasks via RabbitMQ, it might interact with sirius-api for specific needs:
  • To fetch detailed configuration or historical data that it doesn't store itself.
  • To notify sirius-api about the completion of certain tasks if sirius-api needs to update its own state or trigger further workflows (though this is often also handled by sirius-api consuming RabbitMQ messages produced by the engine).
• Other Potential Clients: Any future services or third-party tools needing programmatic access to Sirius Scan's core data would likely interact via sirius-api.

Example Conceptual Endpoint (in Fiber/Go):

package routes

import (
	"github.com/gofiber/fiber/v2"
	"github.com/SiriusScan/Sirius/go-api/sirius/host" // Assuming go-api is accessible
	// ... other imports
)

// SetupHostRoutes configures routes related to hosts
func SetupHostRoutes(app *fiber.App) {
	api := app.Group("/api/v1") // API versioning
	hostRoutes := api.Group("/hosts")

	hostRoutes.Get("/", func(c *fiber.Ctx) error {
		// Logic to fetch all hosts using go-api's host.GetAllHosts()
		// Example: hosts, err := host.GetAllHosts( /* db_instance_from_go_api */ )
		// if err != nil { return c.Status(fiber.StatusInternalServerError).JSON(...) }
		// return c.JSON(hosts)
		return c.SendString("GET /api/v1/hosts - List all hosts")
	})

	hostRoutes.Get("/:hostId", func(c *fiber.Ctx) error {
		hostId := c.Params("hostId")
		// Logic to fetch a specific host by ID using go-api's host.GetHost(hostId)
		// Example: h, err := host.GetHost(hostId, /* db_instance */)
		// if err != nil { /* handle error, e.g., 404 if not found */ }
		// return c.JSON(h)
		return c.SendString("GET /api/v1/hosts/" + hostId)
	})

	hostRoutes.Post("/", func(c *fiber.Ctx) error {
		// var newHostPayload struct { /* ... fields ... */ }
		// if err := c.BodyParser(&newHostPayload); err != nil { return c.Status(fiber.StatusBadRequest).JSON(...) }
		// Logic to add a new host using go-api's host.AddHost(...)
		// return c.Status(fiber.StatusCreated).JSON( /* created_host_info */ )
		return c.SendString("POST /api/v1/hosts - Create new host")
	})
	// ... other routes for PUT, DELETE etc.
}

This service acts as the canonical source for much of the platform's data, ensuring that access and modification are managed centrally and consistently.

6.4. gRPC (Engine to Agents)

gRPC (gRPC Remote Procedure Calls) is the chosen framework for direct, high-performance communication between the sirius-engine and any deployed app-agent instances. This pattern is essential for scenarios requiring low-latency command execution, control, and data exchange with remote agents.

Key Characteristics & Roles:

• Roles: As detailed in section 4.3.2. gRPC Communication with sirius-engine:
  • sirius-engine: Acts as the gRPC server. It defines and implements the gRPC services that agents can call, and also makes calls to agent-exposed services if the model is bidirectional for certain features. Typically, the engine listens on a specific port (e.g., 50051) for incoming agent connections.
  • app-agent: Acts as the gRPC client. It initiates a connection to the sirius-engine's gRPC server address.
• Purpose: To enable sirius-engine to:
  • Discover and register agents.
  • Send commands (e.g., shell commands, requests to initiate local scans by the agent) to specific agents.
  • Receive results, status updates, or streams of data back from agents.
  • Perform health checks or retrieve agent capabilities (Ping method).
• Technology: Utilizes Protocol Buffers (.proto files) for defining service interfaces and message structures. This ensures strong typing and efficient serialization/deserialization of data over HTTP/2.
• Communication Style: Supports both unary RPCs (simple request-response) and streaming RPCs (client-streaming, server-streaming, or bidirectional-streaming). Streaming is particularly useful for long-running commands where app-agent can stream output back to sirius-engine as it becomes available.

Core Interactions (refer to 4.3.2 for more details on .proto examples and app-agent implementation):

• Agent Registration (Conceptual): Upon startup, an app-agent might register itself with sirius-engine by calling a specific gRPC endpoint on the engine, providing its AGENT_ID and capabilities.
• Command Execution (ExecuteCommand method): sirius-engine invokes the ExecuteCommand RPC on a target app-agent, sending the command string. The app-agent executes it and streams back the output and completion status.
• Health Checks (Ping method): sirius-engine can periodically call a Ping method on agents to ensure they are responsive.

Advantages in the Sirius Scan Context:

• Performance: gRPC is generally more performant than REST/JSON for inter-service communication due to its use of HTTP/2 and binary serialization (Protocol Buffers).
• Streaming: Native support for streaming is ideal for tasks like returning large outputs from commands or continuous data feeds.
• Strong Typing & API Contracts: .proto definitions provide a clear, language-agnostic contract between the engine and agents.
• Network Efficiency: Well-suited for potentially network-constrained environments where agents might be deployed.

This gRPC communication channel is vital for extending sirius-engine's operational reach and enabling sophisticated interactions with remote app-agent instances, forming a key part of the distributed task execution framework. For a detailed look at the agent-side implementation and an example .proto definition, please see section 4.3.2. gRPC Communication with sirius-engine.

6.5. Workflow Sequence Diagrams

To better illustrate how these communication mechanisms are used in concert, this section provides sequence diagrams for the key architectural workflows outlined in Section B.

6.5.1. B.1 User Initiates a Vulnerability Scan

sequenceDiagram
    actor User
    participant SF as sirius-ui (Frontend)
    participant SBFF as sirius-ui (BFF/tRPC)
    participant RMQ as RabbitMQ
    participant SE as sirius-engine
    participant AS as app-scanner
    participant SDB as sirius-postgres
    participant SAPI as sirius-api

    User->>SF: 1. Configure & Start Scan
    SF->>SBFF: 2. tRPC call (scan.startScan)
    SBFF->>RMQ: 3. Publish Scan Job (scan_jobs_queue)
    activate RMQ
    RMQ-->>SE: 4. Consume Scan Job
    deactivate RMQ
    activate SE
    SE->>AS: 5. Dispatch Scan Task
    activate AS
    Note over AS,SE: Intermediate Status Updates (Optional)
    SE->>RMQ: 6a. Publish Scan Status (scan_status_updates_queue)
    activate RMQ
    RMQ-->>SBFF: 6b. Consume Scan Status
    deactivate RMQ
    SBFF->>SF: 6c. Relay Status (e.g., WebSocket)

    AS->>SE: 7. Return Scan Results
    deactivate AS
    SE->>SDB: 8. Persist Results (via go-api)
    activate SDB
    deactivate SDB
    SE->>RMQ: 9. Publish Result Notification (scan_results_notification_queue)
    deactivate SE
    activate RMQ
    RMQ-->>SBFF: 10. Consume Notification
    deactivate RMQ

    User->>SF: 11. View Scan Results
    SF->>SBFF: 12. tRPC call (scan.getResults)
    SBFF->>SAPI: 13. Request Scan Results (REST API)
    activate SAPI
    SAPI->>SDB: 14. Query Database
    activate SDB
    SDB-->>SAPI: 15. Return Data
    deactivate SDB
    SAPI-->>SBFF: 16. Return Results
    deactivate SAPI
    SBFF-->>SF: 17. Return Results to UI

6.5.2. B.2 Agent Interaction & Management (e.g., Remote Command Execution)

sequenceDiagram
    actor User
    participant SF as sirius-ui (Frontend)
    participant SBFF as sirius-ui (BFF/tRPC)
    participant RMQ as RabbitMQ
    participant SE as sirius-engine (gRPC Server)
    participant AGENT as app-agent (gRPC Client)

    User->>SF: 1. Initiate Agent Action (e.g., Execute Command)
    SF->>SBFF: 2. tRPC call (e.g., agent.executeRemoteCommand)
    SBFF->>RMQ: 3. Publish Agent Command Task (e.g., agent_commands_queue)
    activate RMQ
    RMQ-->>SE: 4. Consume Agent Command Task
    deactivate RMQ
    activate SE
    SE->>AGENT: 5. gRPC Call (e.g., ExecuteCommand RPC)
    activate AGENT
    Note over AGENT: Agent executes command on remote host
    AGENT-->>SE: 6. Stream/Return Command Output (gRPC)
    deactivate AGENT
    SE->>RMQ: 7. Publish Agent Result/Status (e.g., agent_results_queue)
    deactivate SE
    activate RMQ
    RMQ-->>SBFF: 8. Consume Agent Result/Status
    deactivate RMQ
    SBFF->>SF: 9. Relay Result/Status to UI (e.g., WebSocket)
    SF->>User: 10. Display Result/Status

6.5.3. B.3 Terminal Command Execution (via app-terminal)

sequenceDiagram
    actor User
    participant SF as sirius-ui (Frontend/xterm.js)
    participant SBFF as sirius-ui (BFF/tRPC)
    participant RMQ as RabbitMQ
    participant SE as sirius-engine
    participant AT as app-terminal

    User->>SF: 1. Type Command in UI Terminal
    SF->>SBFF: 2. tRPC call (terminal.sendCommand)
    SBFF->>RMQ: 3. Publish Terminal Command (terminal_commands_queue)
    activate RMQ
    RMQ-->>SE: 4. Consume Terminal Command
    deactivate RMQ
    activate SE
    SE->>AT: 5. Dispatch Command to app-terminal
    activate AT
    Note over AT: app-terminal executes command (e.g., PowerShell)
    AT->>SE: 6. Return Command Output Stream/Chunks
    deactivate AT
    SE->>RMQ: 7. Publish Terminal Output (terminal_output_queue)
    deactivate SE
    activate RMQ
    RMQ-->>SBFF: 8. Consume Terminal Output
    deactivate RMQ
    SBFF->>SF: 9. Relay Output to UI Terminal (e.g., WebSocket)
    SF->>User: 10. Display Command Output

7. Development & Deployment

The development and deployment strategy for Sirius Scan centers around containerization with Docker and Docker Compose, enabling consistent environments and streamlined workflows. Hot reloading is utilized for Go and Next.js applications to enhance developer productivity.

7.1. Dockerized Environment (docker-compose.yaml)

The primary mechanism for orchestrating the various microservices during development (and potentially for simple deployments) is the Sirius/docker-compose.yaml file. This file defines:

• Services: Each core component of Sirius Scan runs as a distinct service:
  • sirius-ui: The Next.js frontend and BFF.
  • sirius-api: The Go-based backend API (Fiber framework).
  • sirius-engine: The Go-based core processing engine.
  • sirius-postgres: PostgreSQL database instance.
  • sirius-valkey: Valkey key-value store instance.
  • sirius-rabbitmq: RabbitMQ message broker instance.
• Build Contexts & Dockerfiles: For services like sirius-ui, sirius-api, and sirius-engine, the build instruction points to their respective directories containing their Dockerfile.
• Image Usage: For infrastructure services (sirius-postgres, sirius-valkey, sirius-rabbitmq), pre-built official Docker images are used.
• Service Dependencies (depends_on): Explicitly defines startup order, ensuring that infrastructure services like databases and message queues are likely to be available before application services attempt to connect to them. However, depends_on does not guarantee that the dependent service is fully initialized and ready, so applications should implement connection retry logic (as go-api does).
• Volume Mounts for Local Development: Crucial for an efficient development loop. For example:
  • sirius-ui: Mounts ./sirius-ui:/app allowing local code changes to be reflected immediately by the Next.js development server.
  • sirius-api: Mounts ./sirius-api:/app for Go code, used by air for hot reloading.
  • sirius-engine: Mounts ./sirius-engine:/engine. It also has commented-out volume mounts for go-api and sub-modules like app-scanner, app-terminal, and app-agent (e.g., - ../minor-projects/app-scanner:/app-scanner). Uncommenting these allows developers to work on these shared libraries or sub-modules locally and have their changes reflected inside the sirius-engine container, overriding the versions cloned by the engine's Dockerfile.
• Port Mappings (ports): Exposes service ports to the host machine (e.g., sirius-ui on 3000, sirius-api on 9001, sirius-engine on 5174 and 50051 for gRPC, PostgreSQL on 54321, RabbitMQ on 5672 and 15672 for management UI).
• Environment Variables (environment): A primary way to configure services. Connection strings, API keys, operational parameters (e.g., GO_ENV=development) are set here. For sirius-ui, build-time arguments (args under build) are also used for NEXT_PUBLIC_ variables.
• Networking: By default, Docker Compose sets up a single default bridge network. Services can reach each other using their defined service names as hostnames (e.g., sirius-api can connect to sirius-postgres using the hostname sirius-postgres).

To run the entire stack: docker compose up --build -d. To view logs: docker compose logs -f <service_name>. To stop: docker compose down.

7.2. Local Development & Hot Reloading

Rapid iteration is supported by hot reloading mechanisms for the primary application services:

• sirius-ui (Next.js):
  • The Next.js development server (typically started by npm run dev within the container, which is the default CMD in its Dockerfile) provides fast refresh and hot module replacement (HMR) out of the box. Changes to React components or Next.js pages are reflected in the browser almost instantly without a full page reload where possible.
• sirius-api (Go/Fiber):
  • Uses air for live reloading. The sirius-api/Dockerfile installs air, and its CMD is typically ["air", "-c", ".air.toml"].
  • The sirius-api/.air.toml configuration file tells air which files/directories to watch (e.g., *.go), what build commands to run (go build -o ./tmp/main .), and how to run the compiled binary (./tmp/main). When Go source files change, air automatically recompiles and restarts the sirius-api server.
• sirius-engine (Go) & Go Sub-Modules (app-scanner, app-terminal):
  • The main sirius-engine Go application and its Go-based sub-modules (which run within the sirius-engine container) can also benefit from air if configured.
  • The sirius-engine/Dockerfile installs air (both globally and within the /app-scanner context as an example).
  • The sirius-engine/start.sh script, which is the ENTRYPOINT for the sirius-engine container, plays a crucial role. This script is responsible for:
    • Starting the main sirius-engine Go application. It could use air (e.g., air -c .air.toml pointing to an .air.toml in the /engine directory) if desired for hot reloading of the core engine code.
    • Managing the Go-based sub-applications cloned into the image (like app-scanner, app-terminal). If these sub-apps have their own .air.toml files (e.g., app-scanner/.air.toml), the start.sh script might navigate to their directories and launch them using their respective air configurations.
  • When local source code for a sub-module (e.g., ../minor-projects/app-scanner) is volume-mounted into the sirius-engine container (e.g., to /app-scanner), changes to those local files will be visible inside the container. If start.sh runs that sub-module via air, hot reloading will occur for that specific sub-module.

7.3. Build Processes (Dockerfiles & Scripts)

Each application service (sirius-ui, sirius-api, sirius-engine) has its own Dockerfile defining its build process:

• sirius-ui/Dockerfile:
  • Typically uses a Node.js base image.
  • Copies package.json and package-lock.json (or yarn.lock), installs dependencies (npm install).
  • Copies the rest of the application source code.
  • Builds the Next.js application for production (npm run build) if the NODE_ENV is production, though for development it often just installs deps and relies on npm run dev as the CMD.
  • Exposes the port (e.g., 3000).
  • Sets the CMD (e.g., npm run dev or npm start).
• sirius-api/Dockerfile:
  • Uses a Go base image.
  • Sets the working directory (e.g., /app).
  • Copies go.mod and go.sum, downloads Go module dependencies (go mod download).
  • Installs air for live reloading.
  • Copies the rest of the application source code.
  • The .air.toml usually defines the build command (go build -o ./tmp/main .).
  • Exposes the application port (e.g., 9001).
  • Sets the CMD to run air (e.g., ["air", "-c", ".air.toml"]).
• sirius-engine/Dockerfile:
  • More complex, as it sets up the environment for the core engine and its bundled sub-applications/tools.
  • Uses a Go base image.
  • Installs system dependencies: git, curl, build tools, and crucially, scanning tools like nmap, rustscan (built from source), and powershell.
  • Sets up directories (e.g., /opt/sirius/nse for Nmap scripts).
  • Clones repositories for sub-applications directly into the image: app-scanner, app-terminal, go-api.
  • Installs air (potentially in multiple contexts if sub-apps use it independently).
  • Copies its own source code (main engine code) and the start.sh script.
  • Makes start.sh executable and sets it as the ENTRYPOINT.
• sirius-engine/start.sh (Entrypoint Script):
  • This script is pivotal for the sirius-engine's operation.
  • It is responsible for the final setup and launching of processes within the container after it starts.
  • Its tasks likely include:
    • Initializing or configuring any services or environment aspects needed at runtime.
    • Starting the main sirius-engine Go application (potentially using air if an .air.toml for the engine core is present).
    • Navigating into the directories of cloned Go sub-applications (e.g., /app-scanner, /app-terminal) and launching them. This might involve running go build for them first (if not already built by air) and then executing their binaries, or running them via air if they have their own .air.toml configurations for hot reloading during development (when volume mounts are active).
    • Managing the lifecycle or backgrounding of these multiple processes if they are meant to run concurrently within the engine container.

Understanding these build and runtime mechanisms is key to developing for and deploying Sirius Scan.

8. Conclusion

This document has outlined the comprehensive architecture of the Sirius Scan platform, detailing its distributed microservices, data management strategies, inter-service communication patterns, and development workflows. The architecture is designed to be modular, scalable, and resilient, providing a robust foundation for a versatile vulnerability scanning and management system.

8.1. Summary of Architecture

Sirius Scan operates as a cohesive system of containerized microservices orchestrated by Docker Compose, promoting consistency across development and deployment environments. The key components and their interactions are summarized below:

• User Interface & BFF (sirius-ui): A Next.js application serves as the primary user interaction point. It includes a React frontend for rich user experience and a Next.js backend (BFF) that handles UI-specific logic. Communication between the frontend and its BFF is streamlined using type-safe tRPC calls.

• Backend API (sirius-api): A Go-based service built with the Fiber framework, exposing RESTful APIs. It acts as the central gateway for CRUD operations on core data entities (hosts, vulnerabilities, etc.) and implements key business logic. It leverages the go-api SDK for database and cache interactions.

• Core Processing Engine (sirius-engine): Another Go application, this is the workhorse of the system. It orchestrates scanning tasks, manages agents, and integrates several specialized sub-applications:

  • app-scanner: For performing network and vulnerability scans using tools like Nmap and RustScan, with sophisticated NSE script management.
  • app-terminal: Provides PowerShell terminal access within the engine's environment. The engine receives tasks primarily via RabbitMQ and uses the go-api SDK for data operations.

• Shared Backend SDK (go-api): A critical Go library providing standardized modules for database access (PostgreSQL with GORM), message queuing (RabbitMQ), key-value store interaction (Valkey), and core data models. It is consumed by sirius-api, sirius-engine, and its sub-modules, ensuring consistency and reusability.

• Remote Agents (app-agent): Go applications designed to run on remote hosts, extending the engine's reach for command execution and potentially distributed scanning tasks. These communicate with sirius-engine via gRPC.

• Data Persistence & Storage:

  • sirius-postgres (PostgreSQL): The primary relational database for storing structured data like host information, scan results, and vulnerabilities.
  • sirius-valkey (Valkey): An in-memory key-value store used for caching, session management, and dynamic configurations like NSE script manifests.

• Inter-Service Communication: A mix of patterns is employed:

  • RabbitMQ: For asynchronous task queuing and event-driven communication, decoupling services like sirius-ui (BFF) from sirius-engine.
  • tRPC: Ensures type-safe API calls within sirius-ui (frontend to BFF).
  • RESTful APIs: Exposed by sirius-api for general backend data and business logic access.
  • gRPC: Facilitates efficient, low-latency communication between sirius-engine and app-agent instances.

This layered and decoupled architecture allows for independent development, scaling, and maintenance of components, while the well-defined communication interfaces ensure effective collaboration between services.

8.2. Future Directions

Sirius Scan is envisioned as an evolving platform. Potential future enhancements and architectural considerations include:

• Enhanced Scalability & Orchestration: Moving beyond Docker Compose for production deployments to more robust orchestration platforms like Kubernetes for better scaling, self-healing, and resource management.
• Advanced Agent Capabilities: Expanding app-agent functionalities to include more sophisticated local scanning capabilities, evidence collection, and potentially persistent monitoring features.
• Expanded Scan Tool Integration: Incorporating a wider variety of security assessment tools and scanners into sirius-engine or via agents.
• Improved Security Posture: Further hardening all services, implementing end-to-end encryption for all sensitive data in transit and at rest, and adopting more granular authentication and authorization mechanisms (e.g., OAuth2/OIDC).
• Workflow Automation & Customization: Allowing users to define more complex, chained scanning and assessment workflows.
• Data Analytics & Reporting: Integrating more advanced data analytics features and customizable reporting capabilities to provide deeper insights into security posture.
• Configuration Management: Implementing a centralized configuration management system for all services, potentially integrated with secrets management solutions.
• Service Mesh: For larger deployments, exploring a service mesh (e.g., Istio, Linkerd) to manage inter-service communication, observability, and security policies more effectively.

By pursuing these directions, Sirius Scan aims to continuously improve its utility, performance, and security, serving as a valuable tool for the cybersecurity community.