mirrors/vxcontrol--pentagi
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:28:13 +08:00
commit 0878425be3
1160 changed files with 491311 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/docs/analytics_api.md
+1005
View File
File diff suppressed because it is too large Load Diff
backend/docs/chain_ast.md
+660
View File
@@ -0,0 +1,660 @@
# ChainAST Documentation
## Table of Contents
- [ChainAST Documentation](#chainast-documentation)
- [Table of Contents](#table-of-contents)
- [Introduction](#introduction)
- [Structure Overview](#structure-overview)
- [Constants and Default Values](#constants-and-default-values)
- [Size Tracking Features](#size-tracking-features)
- [Creating and Using ChainAST](#creating-and-using-chainast)
- [Basic Creation](#basic-creation)
- [Using Constructors](#using-constructors)
- [Getting Messages](#getting-messages)
- [Body Pair Validation](#body-pair-validation)
- [Common Validation Rules](#common-validation-rules)
- [Modifying Message Chains](#modifying-message-chains)
- [Adding Elements](#adding-elements)
- [Adding Human Messages](#adding-human-messages)
- [Working with Tool Calls](#working-with-tool-calls)
- [Testing Utilities](#testing-utilities)
- [Predefined Test Chains](#predefined-test-chains)
- [Generating Custom Test Chains](#generating-custom-test-chains)
- [Message Chain Structure in LLM Providers](#message-chain-structure-in-llm-providers)
- [Message Roles](#message-roles)
- [Message Content](#message-content)
- [Provider-Specific Requirements](#provider-specific-requirements)
- [Reasoning Signatures](#reasoning-signatures)
- [Gemini (Google AI)](#gemini-google-ai)
- [Anthropic (Claude)](#anthropic-claude)
- [Kimi/Moonshot (OpenAI-compatible)](#kimimoonshot-openai-compatible)
- [Helper Functions](#helper-functions)
- [Best Practices](#best-practices)
- [Common Use Cases](#common-use-cases)
- [1. Chain Validation and Repair](#1-chain-validation-and-repair)
- [2. Chain Summarization](#2-chain-summarization)
- [3. Adding Tool Responses](#3-adding-tool-responses)
- [4. Building a Conversation](#4-building-a-conversation)
- [5. Using Summarization in Conversation](#5-using-summarization-in-conversation)
- [Example Usage](#example-usage)
## Introduction
ChainAST is a structured representation of message chains used in Large Language Model (LLM) conversations. It organizes conversations into a logical hierarchy, making it easier to analyze, modify, and validate message sequences, especially when they involve tool calls and their responses.
The structure helps address common issues in LLM conversations such as:
- Validating proper conversation flow
- Managing tool calls and their responses
- Handling conversation sections and state changes
- Ensuring consistent conversation structure
- Efficient size tracking for summarization and context management
## Structure Overview
ChainAST represents a message chain as an abstract syntax tree with the following components:
```
ChainAST
├── Sections[] (ChainSection)
├── Header
│ ├── SystemMessage (optional)
│ ├── HumanMessage (optional)
│ └── sizeBytes (total header size in bytes)
├── sizeBytes (total section size in bytes)
└── Body[] (BodyPair)
├── Type (RequestResponse, Completion, or Summarization)
├── AIMessage
├── ToolMessages[] (for RequestResponse and Summarization types)
└── sizeBytes (total body pair size in bytes)
```
Components:
- **ChainAST**: The root structure containing an array of sections
- **ChainSection**: A logical unit of conversation, starting with a header and containing multiple body pairs
- Includes `sizeBytes` tracking total section size in bytes
- **Header**: Contains system and/or human messages that initiate a section
- Includes `sizeBytes` tracking total header size in bytes
- **BodyPair**: Represents an AI response, which may include tool calls and their responses
- Includes `sizeBytes` tracking total body pair size in bytes
- **RequestResponse**: A type of body pair where the AI message contains tool calls requiring responses
- **Completion**: A simple AI message without tool calls
- **Summarization**: A special type of body pair containing a tool call to the summarization tool
## Constants and Default Values
ChainAST provides several important constants:
- `fallbackRequestArgs`: Default arguments (`{}`) for tool calls without specified arguments
- `FallbackResponseContent`: Default response content ("the call was not handled, please try again") for missing tool responses when using force=true
- `SummarizationToolName`: Name of the special summarization tool ("execute_task_and_return_summary")
- `SummarizationToolArgs`: Default arguments for the summarization tool (`{"question": "delegate and execute the task, then return the summary of the result"}`)
## Size Tracking Features
ChainAST includes built-in size tracking to support efficient summarization algorithms and context management:
```go
// Get the size of a section in bytes
sizeInBytes := section.Size()
// Get the size of a body pair in bytes
sizeInBytes := bodyPair.Size()
// Get the size of a header in bytes
sizeInBytes := header.Size()
// Get the total size of the entire ChainAST
totalSize := ast.Size()
```
Size calculation considers all content types including:
- Text content (string length)
- Image URLs (URL string length)
- Binary data (byte count)
- Tool calls (ID, type, name, and arguments length)
- Tool call responses (ID, name, and content length)
The `sizeBytes` values are automatically maintained when:
- Creating a new ChainAST from a message chain
- Appending human messages
- Adding tool responses
- Creating elements with constructors
## Creating and Using ChainAST
### Basic Creation
```go
// Create from an existing message chain
ast, err := NewChainAST(messageChain, false)
if err != nil {
// Handle validation error
}
// Get messages (flattened chain)
flatChain := ast.Messages()
```
The `force` parameter in `NewChainAST` determines how the function handles inconsistencies:
- `force=false`: Strict validation, returns errors for any inconsistency
- `force=true`: Attempts to repair problems by:
- Merging consecutive human messages into a single message with multiple content parts
- Adding missing tool responses with placeholder content ("the call was not handled, please try again")
- Skipping invalid messages like unexpected tool messages without preceding AI messages
During creation, the size of all components is calculated automatically.
### Using Constructors
ChainAST provides constructors to create elements with automatic size calculation:
```go
// Create a header
header := NewHeader(systemMsg, humanMsg)
// Create a body pair (automatically determines type based on content)
bodyPair := NewBodyPair(aiMsg, toolMsgs)
// Create a body pair from a slice of messages
bodyPair, err := NewBodyPairFromMessages(messages)
// Create a chain section
section := NewChainSection(header, bodyPairs)
// Create a completion body pair with text
completionPair := NewBodyPairFromCompletion("This is a response")
// Create a summarization body pair with text
// The third parameter (addFakeSignature) should be true if the original content
// contained ToolCall reasoning signatures (required for providers like Gemini)
// The fourth parameter (reasoningMsg) preserves reasoning TextContent before ToolCall
// (required for providers like Kimi/Moonshot)
summarizationPair := NewBodyPairFromSummarization("This is a summary of the conversation", tcIDTemplate, false, nil)
// Create a summarization body pair with fake reasoning signature (Gemini)
// This is necessary when summarizing content that originally had ToolCall reasoning
// to satisfy provider requirements (e.g., Gemini's thought_signature)
summarizationWithSignature := NewBodyPairFromSummarization("Summary with signature", tcIDTemplate, true, nil)
// Extract reasoning message for Kimi/Moonshot compatibility
// Returns the first AI message with TextContent containing reasoning (or nil)
reasoningMsg := ExtractReasoningMessage(messages)
// Create summarization with preserved reasoning message (Kimi/Moonshot)
summarizationWithReasoning := NewBodyPairFromSummarization("Summary", tcIDTemplate, false, reasoningMsg)
// Create summarization with BOTH fake signature AND reasoning message
// Required when original had both ToolCall.Reasoning and TextContent.Reasoning
summarizationFull := NewBodyPairFromSummarization("Summary", tcIDTemplate, true, reasoningMsg)
// Check if messages contain reasoning signatures in ToolCall parts
// This is useful for determining if summarized content should include fake signatures
// Only checks ToolCall.Reasoning (not TextContent.Reasoning)
hasToolCallReasoning := ContainsToolCallReasoning(messages)
// Check if a message contains tool calls
hasCalls := HasToolCalls(aiMessage)
```
### Getting Messages
Each component provides a method to get its messages in the correct order:
```go
// Get all messages from a header (system first, then human)
headerMsgs := header.Messages()
// Get all messages from a body pair (AI first, then tools)
bodyPairMsgs := bodyPair.Messages()
// Get all messages from a section
sectionMsgs := section.Messages()
// Get all messages from the ChainAST
allMsgs := ast.Messages()
```
### Body Pair Validation
The `IsValid()` method checks if a BodyPair follows the structure rules:
```go
// Check if a body pair is valid
isValid := bodyPair.IsValid()
```
Validation rules depend on the body pair type:
- For **Completion**: No tool messages allowed
- For **RequestResponse**: Must have at least one tool message
- For **Summarization**: Must have exactly one tool message
- For all types: All tool calls must have matching responses and vice versa
The `GetToolCallsInfo()` method returns detailed information about tool calls:
```go
// Get information about tool calls and responses
toolCallsInfo := bodyPair.GetToolCallsInfo()
// Check for pending or unmatched tool calls
if len(toolCallsInfo.PendingToolCallIDs) > 0 {
// There are tool calls without responses
}
if len(toolCallsInfo.UnmatchedToolCallIDs) > 0 {
// There are tool responses without matching tool calls
}
// Access completed tool calls
for id, pair := range toolCallsInfo.CompletedToolCalls {
// Use tool call and response information
toolCall := pair.ToolCall
response := pair.Response
}
```
### Common Validation Rules
When `force=false`, NewChainAST enforces these rules:
1. First message must be System or Human
2. No consecutive Human messages
3. Tool calls must have matching responses
4. Tool responses must reference valid tool calls
5. System messages can't appear in the middle of a chain
6. AI messages with tool calls must have responses before another AI message
7. Summarization body pairs must have exactly one tool message
## Modifying Message Chains
### Adding Elements
```go
// Add a section to the ChainAST
ast.AddSection(section)
// Add a body pair to a section
section.AddBodyPair(bodyPair)
```
### Adding Human Messages
```go
// Append a new human message
ast.AppendHumanMessage("Tell me more about this topic")
```
The function follows these rules:
1. If chain is empty: Creates a new section with this message as HumanMessage
2. If the last section has body pairs (AI responses): Creates a new section with this message
3. If the last section has no body pairs and no HumanMessage: Adds this message to that section
4. If the last section has no body pairs but has HumanMessage: Appends content to the existing message
Section and header sizes are automatically updated when human messages are added or modified.
### Working with Tool Calls
```go
// Add a response to a tool call
err := ast.AddToolResponse("tool-call-id", "tool-name", "Response content")
if err != nil {
// Handle error (tool call not found)
}
// Find all responses for a specific tool call
responses := ast.FindToolCallResponses("tool-call-id")
```
The `AddToolResponse` function:
- Searches for the specified tool call ID in AI messages
- If the tool call is found and already has a response, updates the existing response content
- If the tool call is found but doesn't have a response, adds a new response
- If the tool call is not found, returns an error
Body pair and section sizes are automatically updated when tool responses are added or modified.
## Testing Utilities
ChainAST comes with utilities for generating test message chains to validate your code.
### Predefined Test Chains
Several test chains are available in the package for common scenarios:
```go
// Basic chains
emptyChain // Empty message chain
systemOnlyChain // Only a system message
humanOnlyChain // Only a human message
systemHumanChain // System + human messages
basicConversationChain // System + human + AI response
// Tool-related chains
chainWithTool // Chain with a tool call, no response
chainWithSingleToolResponse // Chain with a tool call and response
chainWithMultipleTools // Chain with multiple tool calls
chainWithMultipleToolResponses // Chain with multiple tool calls and responses
// Complex chains
chainWithMultipleSections // Multiple conversation turns
chainWithConsecutiveHumans // Chain with error: consecutive human messages
chainWithMissingToolResponse // Chain with error: missing tool response
chainWithUnexpectedTool // Chain with error: unexpected tool message
// Summarization chains
chainWithSummarization // Chain with summarization as the only body pair
chainWithSummarizationAndOtherPairs // Chain with summarization followed by other body pairs
```
### Generating Custom Test Chains
For more complex testing, use the chain generators:
```go
// Simple configuration
config := DefaultChainConfig() // Creates a simple chain with system + human + AI
// Custom configuration
config := ChainConfig{
IncludeSystem: true,
Sections: 3, // 3 conversation turns
BodyPairsPerSection: []int{1, 2, 1}, // Number of AI responses per section
ToolsForBodyPairs: []bool{false, true, false}, // Which responses have tool calls
ToolCallsPerBodyPair: []int{0, 2, 0}, // How many tool calls per response
IncludeAllToolResponses: true, // Whether to include responses for all tools
}
// Generate chain based on config
chain := GenerateChain(config)
// For more complex scenarios with missing responses
complexChain := GenerateComplexChain(
5, // Number of sections
3, // Number of tool calls per tool-using response
7 // Number of missing tool responses
)
```
The `ChainConfig` struct allows fine-grained control over generated test chains:
- `IncludeSystem`: Whether to add a system message at the start
- `Sections`: Number of conversation turns (each with a human message)
- `BodyPairsPerSection`: Number of AI responses per section
- `ToolsForBodyPairs`: Which AI responses should include tool calls
- `ToolCallsPerBodyPair`: Number of tool calls to include in each tool-using response
- `IncludeAllToolResponses`: Whether to add responses for all tool calls
## Message Chain Structure in LLM Providers
ChainAST is designed to work with message chains that follow common conventions in LLM providers:
### Message Roles
- **System**: Provides context or instructions to the model
- **Human/User**: User input messages
- **AI/Assistant**: Model responses
- **Tool**: Results of tool calls executed by the system
### Message Content
Messages can contain different types of content:
- **TextContent**: Simple text messages
- **ToolCall**: Function call requests from the model
- **ToolCallResponse**: Results returned from executing tools
## Provider-Specific Requirements
### Reasoning Signatures
Different LLM providers have specific requirements for reasoning content in function calls:
#### Gemini (Google AI)
Gemini requires **thought signatures** (`thought_signature`) for function calls, especially in multi-turn conversations with tool use. These signatures:
- Are cryptographic representations of the model's internal reasoning process
- Are strictly validated only for the **current turn** (defined as all messages after the last user message with text content)
- Must be preserved when summarizing content that contains them
- Can use fake signatures when creating summarized content: `"skip_thought_signature_validator"`
**Example:**
```go
// Check if original content had reasoning
hasReasoning := ContainsReasoning(originalMessages)
// Create summarized content with fake signature if needed
summaryPair := NewBodyPairFromSummarization(summaryText, tcIDTemplate, hasReasoning)
```
#### Anthropic (Claude)
Anthropic uses **extended thinking** with cryptographic signatures that:
- Are automatically removed from previous turns (not counted in context window)
- Are only required for the current tool use loop
#### Kimi/Moonshot (OpenAI-compatible)
Kimi reasoning models require **reasoning_content in TextContent** before ToolCall:
- Reasoning must be present in a TextContent part before any ToolCall when thinking is enabled
- Error: "thinking is enabled but reasoning_content is missing in assistant tool call message"
- Use `ExtractReasoningMessage()` to preserve reasoning TextContent when summarizing
- Combine with fake ToolCall signatures for full multi-provider compatibility
**Example structure:**
```go
AIMessage.Parts = [
TextContent{Text: "...", Reasoning: {Content: "..."}}, // Required by Kimi
ToolCall{..., Reasoning: {Signature: []byte("...")}}, // Required by Gemini
]
```
**Critical Rule:** Never summarize the last body pair in a section, as this preserves reasoning signatures required by Gemini, Anthropic, and Kimi.
### Helper Functions
```go
// Check if messages contain reasoning signatures in ToolCall parts
// Returns true if any message contains Reasoning in ToolCall (NOT TextContent)
// This is specific to function calling scenarios which require thought_signature
hasToolCallReasoning := ContainsToolCallReasoning(messages)
// Extract reasoning message from AI messages
// Returns the first AI message with TextContent containing reasoning (or nil)
// Useful for preserving reasoning content for providers like Kimi (Moonshot)
reasoningMsg := ExtractReasoningMessage(messages)
// Create summarization with conditional fake signature and reasoning message
addFakeSignature := ContainsToolCallReasoning(originalMessages)
reasoningMsg := ExtractReasoningMessage(originalMessages)
summaryPair := NewBodyPairFromSummarization(summaryText, tcIDTemplate, addFakeSignature, reasoningMsg)
```
## Best Practices
1. **Validation First**: Use `NewChainAST` with `force=false` to validate chains before processing
2. **Defensive Programming**: Always check for errors from ChainAST functions
3. **Complete Tool Calls**: Ensure all tool calls have corresponding responses before sending to an LLM
4. **Section Management**: Use sections to organize conversation turns logically
5. **Testing**: Use the provided generators to test code that manipulates message chains
6. **Size Management**: Leverage size tracking to maintain efficient context windows
7. **Reasoning Preservation**:
- Use `ContainsToolCallReasoning()` to check if fake signatures are needed (checks only ToolCall.Reasoning)
- Use `ExtractReasoningMessage()` to preserve reasoning TextContent for Kimi/Moonshot
8. **Last Pair Protection**: Never summarize the last (most recent) body pair in a section to preserve reasoning signatures
9. **Multi-Provider Support**: When summarizing for current turn, preserve both ToolCall and TextContent reasoning for maximum compatibility
## Common Use Cases
### 1. Chain Validation and Repair
```go
// Try to parse with strict validation
ast, err := NewChainAST(chain, false)
if err != nil {
// If validation fails, try with repair enabled
ast, err = NewChainAST(chain, true)
if err != nil {
// Handle severe structural errors
}
// Log that the chain was repaired
}
```
### 2. Chain Summarization
```go
// Create AST from chain
ast, _ := NewChainAST(chain, true)
// Analyze total size and section sizes
totalSize := ast.Size()
if totalSize > maxContextSize {
// Select sections to summarize
oldestSections := ast.Sections[:len(ast.Sections)-1] // Keep last section
// Summarize sections
summaryText := generateSummary(oldestSections)
// Create a new AST with the summary
newAST := &ChainAST{Sections: []*ChainSection{}}
// Copy system message if exists
var systemMsg *llms.MessageContent
if len(ast.Sections) > 0 && ast.Sections[0].Header.SystemMessage != nil {
systemMsgCopy := *ast.Sections[0].Header.SystemMessage
systemMsg = &systemMsgCopy
}
// Create header and section
header := NewHeader(systemMsg, nil)
section := NewChainSection(header, []*BodyPair{})
newAST.AddSection(section)
// Add summarization body pair
summaryPair := NewBodyPairFromSummarization(summaryText)
section.AddBodyPair(summaryPair)
// Copy the most recent section
lastSection := ast.Sections[len(ast.Sections)-1]
// Add appropriate logic to copy the last section
// Get the summarized chain
summarizedChain := newAST.Messages()
}
```
### 3. Adding Tool Responses
```go
// Parse a chain with tool calls
ast, _ := NewChainAST(chain, false)
// Find unresponded tool calls and add responses
for _, section := range ast.Sections {
for _, bodyPair := range section.Body {
if bodyPair.Type == RequestResponse {
for _, part := range bodyPair.AIMessage.Parts {
if toolCall, ok := part.(llms.ToolCall); ok {
// Execute the tool
result := executeToolCall(toolCall)
// Add the response
ast.AddToolResponse(toolCall.ID, toolCall.FunctionCall.Name, result)
}
}
}
}
}
// Get the updated chain
updatedChain := ast.Messages()
```
### 4. Building a Conversation
```go
// Create an empty AST
ast := &ChainAST{Sections: []*ChainSection{}}
// Add system message
sysMsg := &llms.MessageContent{
Role: llms.ChatMessageTypeSystem,
Parts: []llms.ContentPart{llms.TextContent{Text: "You are a helpful assistant"}},
}
header := NewHeader(sysMsg, nil)
section := NewChainSection(header, []*BodyPair{})
ast.AddSection(section)
// Add a human message
ast.AppendHumanMessage("Hello, how can you help me?")
// Add an AI response
aiMsg := &llms.MessageContent{
Role: llms.ChatMessageTypeAI,
Parts: []llms.ContentPart{llms.TextContent{Text: "I can answer questions, help with tasks, and more."}},
}
bodyPair := NewBodyPair(aiMsg, nil)
section.AddBodyPair(bodyPair)
// Continue the conversation
ast.AppendHumanMessage("Can you help me find information?")
// Get the message chain
chain := ast.Messages()
```
### 5. Using Summarization in Conversation
```go
// Create an empty AST
ast := &ChainAST{Sections: []*ChainSection{}}
// Create a new header with a system message
sysMsg := &llms.MessageContent{
Role: llms.ChatMessageTypeSystem,
Parts: []llms.ContentPart{llms.TextContent{Text: "You are a helpful assistant."}},
}
header := NewHeader(sysMsg, nil)
// Create a new section with the header
section := NewChainSection(header, []*BodyPair{})
ast.AddSection(section)
// Add a human message requesting a summary
ast.AppendHumanMessage("Can you summarize our discussion?")
// Create a summarization body pair
summaryPair := NewBodyPairFromSummarization("This is a summary of our previous conversation about weather and travel plans.")
section.AddBodyPair(summaryPair)
// Get the message chain
chain := ast.Messages()
```
## Example Usage
```go
// Parse a conversation chain with summarization
ast, err := NewChainAST(conversationChain, true)
if err != nil {
log.Fatalf("Failed to parse chain: %v", err)
}
// Check if any body pairs are summarization pairs
for _, section := range ast.Sections {
for _, bodyPair := range section.Body {
if bodyPair.Type == Summarization {
fmt.Println("Found a summarization body pair")
// Extract the summary text from the tool response
for _, toolMsg := range bodyPair.ToolMessages {
for _, part := range toolMsg.Parts {
if resp, ok := part.(llms.ToolCallResponse); ok &&
resp.Name == SummarizationToolName {
fmt.Printf("Summary content: %s\n", resp.Content)
}
}
}
}
}
}
```
backend/docs/chain_summary.md
+1366
View File
File diff suppressed because it is too large Load Diff
backend/docs/charm.md
+1734
View File
File diff suppressed because it is too large Load Diff
backend/docs/config.md
+1921
View File
File diff suppressed because it is too large Load Diff
backend/docs/controller.md
+1161
View File
File diff suppressed because it is too large Load Diff
backend/docs/database.md
+1487
View File
File diff suppressed because it is too large Load Diff
backend/docs/docker.md
+688
View File
@@ -0,0 +1,688 @@
# Docker Client Package Documentation
## Table of Contents
- [Overview](#overview)
- [Architecture](#architecture)
- [Configuration](#configuration)
- [Core Interfaces](#core-interfaces)
- [Container Lifecycle Management](#container-lifecycle-management)
- [Security and Isolation](#security-and-isolation)
- [Integration with PentAGI](#integration-with-pentagi)
- [Usage Examples](#usage-examples)
- [Error Handling](#error-handling)
- [Best Practices](#best-practices)
## Overview
The Docker client package (`backend/pkg/docker`) provides a secure and isolated containerized environment for PentAGI's AI agents to execute penetration testing operations. This package serves as a wrapper around the official Docker SDK, offering specialized functionality for managing containers that AI agents use to perform security testing tasks.
### Key Features
- **Secure Isolation**: All operations are performed in sandboxed Docker containers with complete isolation
- **AI Agent Integration**: Specifically designed to support AI agent workflows and terminal operations
- **Container Lifecycle Management**: Comprehensive container creation, execution, and cleanup
- **Port Management**: Automatic port allocation for flow-specific containers
- **File Operations**: Safe file transfer, path metadata lookup, and non-recursive directory listing between host and containers
- **Network Isolation**: Configurable network policies for security
- **Resource Management**: Memory and CPU limits for controlled execution
- **Volume Management**: Persistent and temporary storage solutions
### Role in PentAGI Ecosystem
The Docker client is a critical component that enables PentAGI's core promise of secure, isolated penetration testing. It provides the foundation for:
- **Terminal Access**: AI agents execute commands in isolated environments
- **Tool Execution**: Professional pentesting tools run in dedicated containers
- **File Management**: Secure file operations and artifact storage
- **Environment Preparation**: Dynamic container setup based on task requirements
- **Resource Cleanup**: Automatic cleanup of completed or failed operations
## Architecture
### Core Components
The Docker client package consists of several key components:
```
backend/pkg/docker/
├── client.go # Main Docker client implementation
└── (future files) # Additional Docker utilities
```
### Key Constants and Configuration
```go
const WorkFolderPathInContainer = "/work" // Standard working directory in containers
const BaseContainerPortsNumber = 28000 // Starting port number for dynamic allocation
const defaultImage = "debian:latest" // Fallback image if custom image fails
const containerPortsNumber = 2 // Number of ports allocated per container
const limitContainerPortsNumber = 2000 // Maximum port range for allocation
const containerListWorkers = 20 // Parallel stat workers for directory listing
```
### Port Allocation Strategy
PentAGI uses a deterministic port allocation algorithm to ensure each flow gets unique, predictable ports:
```go
func GetPrimaryContainerPorts(flowID int64) []int {
ports := make([]int, containerPortsNumber)
for i := 0; i < containerPortsNumber; i++ {
delta := (int(flowID)*containerPortsNumber + i) % limitContainerPortsNumber
ports[i] = BaseContainerPortsNumber + delta
}
return ports
}
```
This ensures that:
- Each flow gets consistent port numbers across restarts
- Port conflicts are avoided between different flows
- Ports are within a controlled range (28000-30000)
## Configuration
### Environment Variables
The Docker client is configured through several environment variables defined in the main configuration:
| Variable | Default | Description |
|----------|---------|-------------|
| `DOCKER_HOST` | `unix:///var/run/docker.sock` | Docker daemon connection |
| `DOCKER_INSIDE` | `false` | Whether PentAGI communicates with host Docker daemon from containers |
| `DOCKER_NET_ADMIN` | `false` | Whether PentAGI grants the primary container NET_ADMIN capability for advanced networking. |
| `DOCKER_SOCKET` | `/var/run/docker.sock` | Path to Docker socket on host |
| `DOCKER_NETWORK` | | Docker network for container communication (bridge mode) or `host` for host network mode |
| `DOCKER_PUBLIC_IP` | `0.0.0.0` | Public IP for port binding (bridge mode only) |
| `DOCKER_WORK_DIR` | | Custom work directory path on host |
| `DOCKER_DEFAULT_IMAGE` | `debian:latest` | Fallback image if AI-selected image fails |
| `DOCKER_DEFAULT_IMAGE_FOR_PENTEST` | `vxcontrol/kali-linux` | Default Docker image for penetration testing tasks |
| `DATA_DIR` | `./data` | Local data directory for file operations |
### Configuration Structure
```go
type Config struct {
// Docker (terminal) settings
DockerInside bool `env:"DOCKER_INSIDE" envDefault:"false"`
DockerNetAdmin bool `env:"DOCKER_NET_ADMIN" envDefault:"false"`
DockerSocket string `env:"DOCKER_SOCKET"`
DockerNetwork string `env:"DOCKER_NETWORK"`
DockerPublicIP string `env:"DOCKER_PUBLIC_IP" envDefault:"0.0.0.0"`
DockerWorkDir string `env:"DOCKER_WORK_DIR"`
DockerDefaultImage string `env:"DOCKER_DEFAULT_IMAGE" envDefault:"debian:latest"`
DockerDefaultImageForPentest string `env:"DOCKER_DEFAULT_IMAGE_FOR_PENTEST" envDefault:"vxcontrol/kali-linux"`
DataDir string `env:"DATA_DIR" envDefault:"./data"`
}
```
### NET_ADMIN Capability Configuration
The `DOCKER_NET_ADMIN` option controls whether PentAGI containers are granted the `NET_ADMIN` Linux capability, which provides advanced networking permissions essential for many penetration testing operations.
#### Network Administration Capabilities
When `DOCKER_NET_ADMIN=true`, containers receive the following networking capabilities:
- **Network Interface Management**: Create, modify, and delete network interfaces
- **Routing Control**: Manipulate routing tables and network routes
- **Firewall Rules**: Configure iptables, netfilter, and other firewall systems
- **Traffic Shaping**: Implement QoS (Quality of Service) and bandwidth controls
- **Bridge Operations**: Create and manage network bridges
- **VLAN Configuration**: Set up and modify VLAN configurations
- **Packet Capture**: Enhanced access to raw sockets and packet capture mechanisms
#### Security Implications
**Enabling NET_ADMIN (`DOCKER_NET_ADMIN=true`)**:
- **Benefits**: Enables full-featured network penetration testing tools
- **Risks**: Containers can potentially modify host network configuration
- **Use Cases**: Network scanning, traffic interception, custom routing setups
- **Tools Enabled**: Advanced nmap features, tcpdump, wireshark, custom networking tools
**Disabling NET_ADMIN (`DOCKER_NET_ADMIN=false`)**:
- **Benefits**: Enhanced security isolation from host networking
- **Limitations**: Some advanced networking tools may not function fully (nmap)
- **Use Cases**: Application-level testing, web security assessment
- **Recommended**: For environments where network-level testing is not required
#### Container Capability Assignment
The NET_ADMIN capability is applied differently based on container type and configuration:
```go
// Primary containers (when DOCKER_NET_ADMIN=true)
hostConfig := &container.HostConfig{
CapAdd: []string{"NET_RAW", "NET_ADMIN"}, // Full networking capabilities
// ... other configurations
}
// Primary containers (when DOCKER_NET_ADMIN=false)
hostConfig := &container.HostConfig{
CapAdd: []string{"NET_RAW"}, // Basic raw socket access only
// ... other configurations
}
```
### Docker-in-Docker Support
PentAGI supports running inside Docker containers while still managing other containers. This is controlled by the `DOCKER_INSIDE` setting:
- **`DOCKER_INSIDE=false`**: PentAGI runs on host, manages containers directly
- **`DOCKER_INSIDE=true`**: PentAGI runs in container, mounts Docker socket to manage sibling containers
### Network Configuration
PentAGI supports two network modes for container isolation:
#### Bridge Network Mode (Default)
When `DOCKER_NETWORK` is set to a custom network name (e.g., `pentagi-network`), containers are connected to an isolated bridge network:
- **Isolated Communication**: Containers communicate only within the defined network
- **Port Mapping**: Container ports are mapped to host ports for external access
- **Service Discovery**: Enables internal DNS-based service discovery
- **Enhanced Security**: Network-level isolation from other containers
#### Host Network Mode
When `DOCKER_NETWORK` is set to the special value `host`, containers use the host's network stack directly:
- **Direct Network Access**: Container shares the host's network interfaces
- **No Port Mapping**: Ports are directly accessible on host interfaces (no NAT)
- **Performance**: Eliminates network virtualization overhead
- **Use Cases**: Advanced network testing, raw packet manipulation, network monitoring
**Security Consideration**: Host network mode reduces isolation. Use only when necessary for penetration testing tasks requiring direct host network access.
## Core Interfaces
### DockerClient Interface
The main interface defines all Docker operations available to PentAGI components:
```go
type DockerClient interface {
// Container lifecycle management
RunContainer(ctx context.Context, containerName string, containerType database.ContainerType,
flowID int64, config *container.Config, hostConfig *container.HostConfig) (database.Container, error)
StopContainer(ctx context.Context, containerID string, dbID int64) error
RemoveContainer(ctx context.Context, containerID string, dbID int64) error
IsContainerRunning(ctx context.Context, containerID string) (bool, error)
// Command execution
ContainerExecCreate(ctx context.Context, container string, config container.ExecOptions) (container.ExecCreateResponse, error)
ContainerExecAttach(ctx context.Context, execID string, config container.ExecAttachOptions) (types.HijackedResponse, error)
ContainerExecInspect(ctx context.Context, execID string) (container.ExecInspect, error)
// File operations
ContainerStatPath(ctx context.Context, containerID string, path string) (container.PathStat, error)
ListContainerDir(ctx context.Context, containerID string, dirPath string) ([]container.PathStat, error)
CopyToContainer(ctx context.Context, containerID string, dstPath string, content io.Reader, options container.CopyToContainerOptions) error
CopyFromContainer(ctx context.Context, containerID string, srcPath string) (io.ReadCloser, container.PathStat, error)
// Utility methods
Cleanup(ctx context.Context) error
GetDefaultImage() string
}
```
### Implementation Structure
```go
type dockerClient struct {
db database.Querier // Database for container state management
logger *logrus.Logger // Structured logging
dataDir string // Local data directory
hostDir string // Host-mapped data directory
client *client.Client // Docker SDK client
inside bool // Running inside Docker
defImage string // Default fallback image
socket string // Docker socket path
network string // Docker network name
publicIP string // Public IP for port binding
}
```
## Container Lifecycle Management
### Container Creation Process
The `RunContainer` method handles the complete container creation workflow:
1. **Preparation**:
- Creates flow-specific work directory
- Generates unique container name
- Records container in database with "starting" status
2. **Image Management**:
- Attempts to pull requested image
- Falls back to default image if pull fails
- Updates database with actual image used
3. **Container Configuration**:
- Sets hostname based on container name hash
- Configures working directory to `/work`
- Sets up restart policy (`on-failure`, maximum 5 retries)
- Configures logging (JSON driver with rotation)
4. **Storage Setup**:
- Creates dedicated volume or bind mount
- Mounts work directory to `/work` in container
- Optionally mounts Docker socket for Docker-in-Docker
5. **Network and Ports**:
- **Bridge Mode**: Assigns flow-specific ports using deterministic algorithm, binds to public IP
- **Host Mode** (`DOCKER_NETWORK=host`): Uses host network stack, skips port bindings
- Connects to specified Docker network (unless host mode)
6. **Container Startup**:
- Creates container with all configurations
- Starts container
- Updates database status to "running"
### Example Container Configuration
```go
containerConfig := &container.Config{
Image: "kali:latest", // AI-selected or default image
Hostname: "a1b2c3d4", // Generated from container name
WorkingDir: "/work", // Standard working directory
Entrypoint: []string{"tail", "-f", "/dev/null"}, // Keep container running
ExposedPorts: nat.PortSet{
"28000/tcp": {}, // Flow-specific ports
"28001/tcp": {},
},
}
hostConfig := &container.HostConfig{
CapAdd: []string{"NET_RAW"}, // Required capabilities for network tools
RestartPolicy: container.RestartPolicy{
Name: "on-failure", // Restart failed containers only
MaximumRetryCount: 5,
},
Binds: []string{
"/host/data/flow-123:/work", // Work directory mount
"/var/run/docker.sock:/var/run/docker.sock", // Docker socket (if inside Docker)
},
PortBindings: nat.PortMap{
"28000/tcp": []nat.PortBinding{{HostIP: "0.0.0.0", HostPort: "28000"}},
"28001/tcp": []nat.PortBinding{{HostIP: "0.0.0.0", HostPort: "28001"}},
},
}
```
### Container States and Transitions
PentAGI tracks container states in the database:
- **`Starting`**: Container creation in progress
- **`Running`**: Container is active and available
- **`Stopped`**: Container has been stopped but not removed
- **`Failed`**: Container creation or startup failed
- **`Deleted`**: Container has been removed
### Container Naming Convention
Containers follow a specific naming pattern for easy identification:
```go
func PrimaryTerminalName(flowID int64) string {
return fmt.Sprintf("pentagi-terminal-%d", flowID)
}
```
This creates names like `pentagi-terminal-123` for flow ID 123, making it easy to:
- Identify containers belonging to specific flows
- Perform flow-based cleanup operations
- Debug container-related issues
### Cleanup Operations
The `Cleanup` method performs comprehensive cleanup:
1. **Flow State Assessment**:
- Identifies flows that should be terminated
- Marks incomplete flows as failed
- Preserves running flows that should continue
2. **Container Cleanup**:
- Stops all containers for terminated flows
- Removes stopped containers and their volumes
- Updates database to reflect current state
3. **Parallel Processing**:
- Uses goroutines for concurrent container deletion
- Ensures cleanup doesn't block system operation
## Security and Isolation
### Container Security Model
PentAGI implements a multi-layered security approach for container isolation:
#### Network Isolation
- **Custom Networks**: Containers run in dedicated Docker networks
- **Port Control**: Only specific ports are exposed to the host
- **Host Protection**: Container cannot access host network by default
#### File System Isolation
- **Read-Only Root**: Base container filesystem is immutable
- **Controlled Mounts**: Only specific directories are writable
- **Volume Separation**: Each flow gets isolated storage space
#### Capability Management
```go
hostConfig := &container.HostConfig{
CapAdd: []string{"NET_RAW"}, // Required for network scanning tools
// Other dangerous capabilities are not granted
}
```
#### Process Isolation
- **User Namespaces**: Containers run with isolated user space
- **PID Isolation**: Container processes are isolated from host
- **Resource Limits**: Memory and CPU usage are controlled
### Security Best Practices Implemented
1. **Image Validation**: All images are pulled and verified before use
2. **Fallback Strategy**: Safe default image used if custom image fails
3. **State Tracking**: All container operations are logged and monitored
4. **Automatic Cleanup**: Failed or abandoned containers are automatically removed
5. **Socket Security**: Docker socket is only mounted when explicitly required
## Integration with PentAGI
### Tool Integration
The Docker client integrates with PentAGI's tool system to provide terminal access:
```go
type terminal struct {
flowID int64
containerID int64
containerLID string
dockerClient docker.DockerClient
tlp TermLogProvider
}
```
The terminal tool uses the Docker client for:
- **Command Execution**: Running shell commands in isolated containers
- **File Operations**: Reading and writing files safely
- **Result Capture**: Collecting command output and artifacts
### Flow File Integration
Flow files are managed by the REST API in `pkg/server/services/flow_files.go` and use Docker client file APIs for synchronization with the running primary container.
PentAGI keeps two different storage areas for flow files:
- **Local cache**: `{DATA_DIR}/flow-{id}-data/uploads` and `{DATA_DIR}/flow-{id}-data/container`
- **Container workspace**: `/work` inside the primary container
This separation is intentional. It supports both single-node deployments and remote worker-node deployments where the backend host filesystem is not the same filesystem used by Docker workers.
The current behavior is:
- User uploads are saved to the local cache under `uploads/`.
- If the primary container is running, uploaded files are pushed best-effort to `/work/uploads`.
- When the primary container starts or is reused, cached uploads are synchronized into `/work/uploads`; the cache is the source of truth.
- Files pulled from the container are stored under `container/` using their normalized full container path, for example:
- `/etc/nginx/nginx.conf` -> `container/etc/nginx/nginx.conf`
- `/work/test.md` -> `container/work/test.md`
- Deleting cached upload files is allowed even when the container is not running. The next container start will resynchronize `/work/uploads` from cache.
The flow files API also exposes a non-recursive live container directory listing endpoint. It uses `ContainerStatPath` to determine whether the requested path is a file or directory:
- If the path is a file, it returns that file metadata directly.
- If the path is a directory, it calls `ListContainerDir`.
- If the path is omitted, it defaults to `/work`.
### Provider Integration
The provider system uses Docker client for environment preparation:
```go
// In providers.go
type flowProvider struct {
// ... other fields
docker docker.DockerClient
publicIP string
}
```
Providers use the Docker client to:
- **Image Selection**: AI agents choose appropriate container images
- **Environment Setup**: Prepare containers for specific tasks
- **Resource Management**: Allocate and deallocate containers as needed
### Database Integration
Container states are persisted in the PostgreSQL database:
```sql
-- Container state tracking
CREATE TABLE containers (
id SERIAL PRIMARY KEY,
flow_id INTEGER REFERENCES flows(id),
name VARCHAR NOT NULL,
image VARCHAR NOT NULL,
status container_status NOT NULL,
local_id VARCHAR,
local_dir VARCHAR,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
```
### Observability Integration
All Docker operations are instrumented with:
- **Structured Logging**: JSON logs with context and metadata
- **Error Tracking**: Comprehensive error capture and reporting
- **Performance Metrics**: Container creation and execution timing
- **Resource Monitoring**: CPU, memory, and network usage tracking
## Usage Examples
### Basic Container Creation
```go
// Initialize Docker client
dockerClient, err := docker.NewDockerClient(ctx, db, cfg)
if err != nil {
return fmt.Errorf("failed to create docker client: %w", err)
}
// Create container for a flow
containerName := docker.PrimaryTerminalName(flowID)
container, err := dockerClient.RunContainer(
ctx,
containerName,
database.ContainerTypePrimary,
flowID,
&container.Config{
Image: "kali:latest",
Entrypoint: []string{"tail", "-f", "/dev/null"},
},
&container.HostConfig{
CapAdd: []string{"NET_RAW", "NET_ADMIN"},
},
)
```
### Command Execution
```go
// Execute command in container
createResp, err := dockerClient.ContainerExecCreate(ctx, containerName, container.ExecOptions{
Cmd: []string{"sh", "-c", "nmap -sS 192.168.1.1"},
AttachStdout: true,
AttachStderr: true,
WorkingDir: "/work",
Tty: true,
})
// Attach to execution
resp, err := dockerClient.ContainerExecAttach(ctx, createResp.ID, container.ExecAttachOptions{
Tty: true,
})
// Read output
output, err := io.ReadAll(resp.Reader)
```
### File Operations
```go
// Write file to container
content := "#!/bin/bash\necho 'Hello from container'"
archive := createTarArchive("script.sh", content)
err := dockerClient.CopyToContainer(ctx, containerID, "/work", archive, container.CopyToContainerOptions{})
// Read file from container
reader, stats, err := dockerClient.CopyFromContainer(ctx, containerID, "/work/results.txt")
defer reader.Close()
// Extract content from tar
content := extractFromTar(reader)
// Stat a file or directory in the container
stat, err := dockerClient.ContainerStatPath(ctx, containerID, "/work/results.txt")
// List direct entries in a container directory
entries, err := dockerClient.ListContainerDir(ctx, containerID, "/work")
```
### Container Directory Listing
`ListContainerDir` performs a non-recursive directory listing inside a running container:
1. Uses `ContainerStatPath` to verify that `dirPath` exists and is a directory.
2. Executes `ls -1 -- <dirPath>` inside the container to get direct entry names.
3. Calls `ContainerStatPath` for every entry to return Docker `container.PathStat` metadata.
4. Runs entry stat calls through `pkg/queue` with `containerListWorkers = 20` workers to reduce latency for large directories.
The method returns `[]container.PathStat`. The caller is responsible for joining the returned entry name with the requested base path when it needs full paths.
If `dirPath` is empty, it defaults to `WorkFolderPathInContainer` (`/work`).
### Cleanup and Resource Management
```go
// Check if container is running
isRunning, err := dockerClient.IsContainerRunning(ctx, containerID)
// Stop container
err = dockerClient.StopContainer(ctx, containerID, dbID)
// Remove container and volumes
err = dockerClient.RemoveContainer(ctx, containerID, dbID)
// Global cleanup (usually called on startup)
err = dockerClient.Cleanup(ctx)
```
### Error Handling
```go
// The client implements comprehensive error handling
container, err := dockerClient.RunContainer(ctx, name, containerType, flowID, config, hostConfig)
if err != nil {
// Errors include:
// - Image pull failures (handled with fallback)
// - Container creation failures
// - Network configuration issues
// - Database update failures
// The client automatically:
// - Updates database with failure status
// - Cleans up partially created resources
// - Logs detailed error information
return fmt.Errorf("container creation failed: %w", err)
}
```
## Error Handling
### Error Categories
The Docker client handles several categories of errors:
1. **Docker Daemon Errors**:
- Connection failures to Docker daemon
- API version mismatches
- Permission issues
2. **Image-Related Errors**:
- Image pull failures (network, authentication)
- Invalid image names or tags
- Image compatibility issues
3. **Container Runtime Errors**:
- Container creation failures
- Container startup issues
- Resource allocation problems
4. **Network and Storage Errors**:
- Port binding conflicts
- Volume mount failures
- Network configuration issues
### Error Recovery Strategies
1. **Image Fallback**:
```go
if err := dc.pullImage(ctx, config.Image); err != nil {
logger.WithError(err).Warnf("failed to pull image '%s', using default", config.Image)
config.Image = dc.defImage
// Retry with default image
}
```
2. **Container Cleanup**:
```go
if containerCreationFails {
defer updateContainerInfo(database.ContainerStatusFailed, containerID)
// Clean up any partially created resources
}
```
3. **State Synchronization**:
- Database state always reflects actual container state
- Failed operations are marked appropriately
- Orphaned resources are cleaned up automatically
## Best Practices
### Resource Management
- Always use the `Cleanup()` method on application startup
- Monitor container resource usage through observability tools
- Set appropriate timeouts for long-running operations
- Use deterministic port allocation to avoid conflicts
### Security Considerations
- Regularly update base images used for containers
- Minimize capabilities granted to containers
- Use dedicated networks for container communication
- Monitor and audit all container operations
### Development and Debugging
- Use structured logging for all Docker operations
- Implement comprehensive error handling with context
- Test container operations in isolated environments
- Use the ftester utility for debugging specific operations
### Performance Optimization
- Reuse containers when possible instead of creating new ones
- Implement efficient cleanup to prevent resource leaks
- Use appropriate container restart policies
- Monitor container startup times and optimize configurations
### Integration Guidelines
- Always use the DockerClient interface instead of direct Docker SDK calls
- Integrate with PentAGI's database for state management
- Use the provided logging and observability infrastructure
- Follow the established naming conventions for containers
backend/docs/flow_execution.md
+1263
View File
File diff suppressed because it is too large Load Diff
backend/docs/gemini.md
+237
View File
@@ -0,0 +1,237 @@
# Google AI (Gemini) Provider
The Google AI provider enables PentAGI to use Google's Gemini language models through the Generative AI API. This provider supports advanced features like function calling, streaming responses, and competitive pricing.
## Features
- **Multi-model Support**: Access to Gemini 2.5 Flash, Gemini 2.5 Pro, and other Google AI models
- **Function Calling**: Full support for tool usage and function calls
- **Streaming Responses**: Real-time response streaming for better user experience
- **Competitive Pricing**: Cost-effective inference with transparent pricing
- **Proxy Support**: HTTP proxy support for enterprise environments
- **Advanced Configuration**: Fine-tuned parameters for different agent types
## Configuration
### Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `GEMINI_API_KEY` | - | Your Google AI API key (required) |
| `GEMINI_SERVER_URL` | `https://generativelanguage.googleapis.com` | Google AI API base URL |
### Getting API Key
1. Visit [Google AI Studio](https://aistudio.google.com/app/apikey)
2. Create a new API key
3. Set it as `GEMINI_API_KEY` environment variable
## Available Models
| Model | Context Window | Max Output | Input Price* | Output Price* | Best For |
|-------|----------------|------------|--------------|---------------|----------|
| gemini-2.5-flash | 1M tokens | 65K tokens | $0.15 | $0.60 | General tasks, fast responses |
| gemini-2.5-pro | 1M tokens | 65K tokens | $2.50 | $10.00 | Complex reasoning, analysis |
| gemini-2.0-flash | 1M tokens | 8K tokens | $0.15 | $0.60 | High-frequency tasks |
| gemini-1.5-flash | 1M tokens | 8K tokens | $0.075 | $0.30 | Legacy model (deprecated) |
| gemini-1.5-pro | 2M tokens | 8K tokens | $1.25 | $5.00 | Legacy model (deprecated) |
*Prices per 1M tokens (USD)
## Agent Configuration
Each agent type is optimized with specific parameters for Google AI models:
### Basic Agents
- **Simple**: General-purpose tasks with balanced settings
- **Simple JSON**: Structured output generation with JSON formatting
- **Primary Agent**: Core reasoning with moderate creativity
- **Assistant (A)**: User interaction with contextual responses
### Specialized Agents
- **Generator**: Creative content with higher temperature
- **Refiner**: Content improvement with focused parameters
- **Adviser**: Strategic guidance with extended context
- **Reflector**: Analysis and evaluation tasks
- **Searcher**: Information retrieval with precise settings
- **Enricher**: Data enhancement and augmentation
- **Coder**: Programming tasks with minimal temperature
- **Installer**: System setup with deterministic responses
- **Pentester**: Security testing with balanced creativity
## Usage Examples
### Basic Setup
```bash
# Set environment variables
export GEMINI_API_KEY="your_api_key_here"
export GEMINI_SERVER_URL="https://generativelanguage.googleapis.com"
# Test the provider
docker run --rm \
-v $(pwd)/.env:/opt/pentagi/.env \
vxcontrol/pentagi /opt/pentagi/bin/ctester -type gemini
```
### Custom Configuration
```yaml
# gemini-custom.yml
simple:
model: "gemini-2.5-pro"
temperature: 0.3
top_p: 0.4
max_tokens: 8000
price:
input: 2.50
output: 10.00
coder:
model: "gemini-2.5-flash"
temperature: 0.05
top_p: 0.1
max_tokens: 16000
price:
input: 0.15
output: 0.60
```
### Docker Usage
```bash
# Using pre-configured Gemini provider
docker run --rm \
-v $(pwd)/.env:/opt/pentagi/.env \
vxcontrol/pentagi /opt/pentagi/bin/ctester \
-config /opt/pentagi/conf/gemini.provider.yml
# Using custom configuration
docker run --rm \
-v $(pwd)/.env:/opt/pentagi/.env \
-v $(pwd)/gemini-custom.yml:/opt/pentagi/gemini-custom.yml \
vxcontrol/pentagi /opt/pentagi/bin/ctester \
-type gemini \
-config /opt/pentagi/gemini-custom.yml
```
## Integration with PentAGI
### Environment File (.env)
```bash
# Google AI Configuration
GEMINI_API_KEY=your_api_key_here
GEMINI_SERVER_URL=https://generativelanguage.googleapis.com
# Optional: Proxy settings
PROXY_URL=http://your-proxy:port
```
### Provider Selection
The Google AI provider is automatically available when `GEMINI_API_KEY` is set. You can use it for:
- **Flow Execution**: Autonomous penetration testing workflows
- **Assistant Mode**: Interactive chat and analysis
- **Custom Tasks**: Specialized security assessments
- **API Integration**: Programmatic access to Google AI models
## Best Practices
### Model Selection
- Use **gemini-2.5-flash** for general tasks and fast responses
- Use **gemini-2.5-pro** for complex reasoning and detailed analysis
- Avoid deprecated models (1.5 series) for new projects
### Performance Optimization
- Set appropriate `max_tokens` limits based on your use case
- Use lower `temperature` values for deterministic tasks
- Configure `top_p` to balance creativity and consistency
### Cost Management
- Monitor token usage through PentAGI's cost tracking
- Use cheaper models for simple tasks
- Implement request batching where possible
### Security Considerations
- Store API keys securely (environment variables, secrets management)
- Use HTTPS for all API communications
- Implement rate limiting to prevent abuse
- Monitor API usage and costs regularly
## Troubleshooting
### Common Issues
1. **API Key Issues**
```
Error: failed to create gemini provider: invalid API key
```
- Verify your API key is correct
- Check API key permissions in Google AI Studio
- Ensure the key hasn't expired
2. **Model Not Found**
```
Error: model "gemini-x.x-xxx" not found
```
- Use supported model names from the table above
- Check for typos in model names
- Verify model availability in your region
3. **Rate Limiting**
```
Error: quota exceeded
```
- Implement exponential backoff
- Reduce request frequency
- Check your quota limits in Google AI Studio
4. **Network Issues**
```
Error: connection timeout
```
- Check internet connectivity
- Verify proxy settings if applicable
- Check firewall rules for outbound HTTPS
### Testing Provider
```bash
# Test basic functionality
docker run --rm \
-v $(pwd)/.env:/opt/pentagi/.env \
vxcontrol/pentagi /opt/pentagi/bin/ctester \
-type gemini \
-agent simple \
-prompt "Hello, world!"
# Test JSON functionality
docker run --rm \
-v $(pwd)/.env:/opt/pentagi/.env \
vxcontrol/pentagi /opt/pentagi/bin/ctester \
-type gemini \
-agent simple_json \
-prompt "Generate a JSON object with name and age fields"
# Test all agents
docker run --rm \
-v $(pwd)/.env:/opt/pentagi/.env \
vxcontrol/pentagi /opt/pentagi/bin/ctester \
-type gemini
```
## Support and Resources
- [Google AI Documentation](https://ai.google.dev/docs)
- [Gemini API Reference](https://ai.google.dev/api)
- [PentAGI Documentation](https://docs.pentagi.com)
- [Issue Tracker](https://github.com/vxcontrol/pentagi/issues)
For provider-specific issues, include:
- Provider type: `gemini`
- Model name used
- Configuration snippet (without API keys)
- Error messages and logs
- Environment details (Docker, OS, etc.)
backend/docs/installer.md
+1382
View File
File diff suppressed because it is too large Load Diff
backend/docs/installer/charm-architecture-patterns.md
+662
View File
@@ -0,0 +1,662 @@
# Charm.sh Production Architecture Patterns
> Essential patterns for building robust, scalable TUI applications with the Charm ecosystem.
## 🏗️ **Centralized Styles & Dimensions**
### **Styles Singleton Pattern**
**CRITICAL**: Never store width/height in models - use styles singleton
```go
// ✅ CORRECT: Centralized in styles
type Styles struct {
width int
height int
renderer *glamour.TermRenderer
// Core styles
Header lipgloss.Style
Footer lipgloss.Style
Content lipgloss.Style
Title lipgloss.Style
Subtitle lipgloss.Style
Paragraph lipgloss.Style
// Form styles
FormLabel lipgloss.Style
FormInput lipgloss.Style
FormPlaceholder lipgloss.Style
FormHelp lipgloss.Style
// Status styles
Success lipgloss.Style
Error lipgloss.Style
Warning lipgloss.Style
Info lipgloss.Style
}
func New() *Styles {
renderer, _ := glamour.NewTermRenderer(
glamour.WithAutoStyle(),
glamour.WithWordWrap(80),
)
styles := &Styles{
renderer: renderer,
width: 80,
height: 24,
}
styles.updateStyles()
return styles
}
func (s *Styles) SetSize(width, height int) {
s.width = width
s.height = height
s.updateStyles() // Recalculate responsive styles
}
func (s *Styles) GetSize() (int, int) {
return s.width, s.height
}
func (s *Styles) GetWidth() int {
return s.width
}
func (s *Styles) GetHeight() int {
return s.height
}
func (s *Styles) GetRenderer() *glamour.TermRenderer {
return s.renderer
}
// Update styles based on current dimensions
func (s *Styles) updateStyles() {
// Base styles
s.Header = lipgloss.NewStyle().
Bold(true).
Foreground(lipgloss.Color("205"))
s.Footer = lipgloss.NewStyle().
Width(s.width).
Background(lipgloss.Color("240")).
Foreground(lipgloss.Color("255")).
Padding(0, 1, 0, 1)
s.Content = lipgloss.NewStyle().
Width(s.width).
Padding(1, 2, 1, 2)
// Form styles with responsive sizing
s.FormInput = lipgloss.NewStyle().
Border(lipgloss.RoundedBorder()).
BorderForeground(lipgloss.Color("240")).
Padding(0, 1, 0, 1)
// Status styles
s.Success = lipgloss.NewStyle().Foreground(lipgloss.Color("10"))
s.Error = lipgloss.NewStyle().Foreground(lipgloss.Color("9"))
s.Warning = lipgloss.NewStyle().Foreground(lipgloss.Color("11"))
s.Info = lipgloss.NewStyle().Foreground(lipgloss.Color("12"))
}
// Models use styles for dimensions
func (m *Model) updateViewport() {
width, height := m.styles.GetSize()
if width <= 0 || height <= 0 {
return // Graceful handling
}
// ... viewport setup
}
```
## 🏗️ **Unified Header/Footer Management**
### **App-Level Layout Control**
```go
// app.go - Central layout control
func (a *App) View() string {
header := a.renderHeader()
footer := a.renderFooter()
content := a.currentModel.View()
contentHeight := max(height - headerHeight - footerHeight, 0)
contentArea := a.styles.Content.Height(contentHeight).Render(content)
return lipgloss.JoinVertical(lipgloss.Left, header, contentArea, footer)
}
func (a *App) renderHeader() string {
switch a.navigator.Current() {
case WelcomeScreen:
return a.styles.RenderASCIILogo()
default:
title := a.getScreenTitle(a.navigator.Current())
return a.styles.Header.Render(title)
}
}
func (a *App) renderFooter() string {
actions := a.buildFooterActions()
footerText := strings.Join(actions, " • ")
return a.styles.Footer.Render(footerText)
}
func (a *App) buildFooterActions() []string {
actions := []string{"Esc: Back", "Ctrl+C: Exit"}
// Screen-specific actions
switch a.navigator.Current().GetScreen() {
case "eula":
if a.isEULAScrolledToEnd() {
actions = append(actions, "Y: Accept", "N: Reject")
} else {
actions = append(actions, "↑↓: Scroll", "PgUp/PgDn: Page")
}
case "form":
actions = append(actions, "Tab: Complete", "Ctrl+S: Save", "Enter: Save & Return")
case "menu":
actions = append(actions, "Enter: Select")
}
return actions
}
```
### **Background Footer Approach (Production)**
```go
// ✅ PRODUCTION READY: Background approach (always 1 line)
func (s *Styles) createFooter(width int, text string) string {
return lipgloss.NewStyle().
Width(width).
Background(lipgloss.Color("240")).
Foreground(lipgloss.Color("255")).
Padding(0, 1, 0, 1).
Render(text)
}
// ❌ WRONG: Border approach (height inconsistency)
func createFooterWrong(width int, text string) string {
return lipgloss.NewStyle().
Width(width).
Height(1).
Border(lipgloss.Border{Top: true}).
Render(text)
}
```
## 🏗️ **Component Initialization Pattern**
### **Standard Component Architecture**
```go
// Standard component initialization
func NewModel(styles *styles.Styles, window *window.Window, controller *controllers.StateController) *Model {
viewport := viewport.New(window.GetContentSize())
viewport.Style = lipgloss.NewStyle() // Clean style prevents conflicts
return &Model{
styles: styles,
window: window,
controller: controller,
viewport: viewport,
initialized: false,
}
}
func (m *Model) Init() tea.Cmd {
// ALWAYS reset ALL state
m.content = ""
m.ready = false
m.error = nil
m.initialized = false
logger.Log("[Model] INIT: starting initialization")
return m.loadContent
}
// Window size handling
func (m *Model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
switch msg := msg.(type) {
case tea.WindowSizeMsg:
// Update through styles, not directly
// m.styles.SetSize() called by app.go
m.updateViewport()
case ContentLoadedMsg:
m.content = msg.Content
m.ready = true
m.initialized = true
m.viewport.SetContent(m.content)
return m, nil
case tea.KeyMsg:
return m.handleKeyMsg(msg)
}
return m, nil
}
```
## 🏗️ **Model State Management**
### **Complete State Reset Pattern**
```go
// Model interface implementation
type Model struct {
styles *styles.Styles // ALWAYS use shared styles
window *window.Window
// Core state
content string
ready bool
error error
initialized bool
// Component state
viewport viewport.Model
// Navigation state
args []string
}
func (m *Model) Init() tea.Cmd {
logger.Log("[Model] INIT")
// ALWAYS reset ALL state completely
m.content = ""
m.ready = false
m.error = nil
m.initialized = false
// Reset component state
m.viewport.GotoTop()
m.viewport.SetContent("")
return m.loadContent
}
// Force re-render after async operations
func (m *Model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
switch msg := msg.(type) {
case ContentLoadedMsg:
m.content = msg.Content
m.ready = true
m.initialized = true
m.viewport.SetContent(m.content)
// Force view update with no-op command
return m, func() tea.Msg { return nil }
}
return m, nil
}
```
### **Graceful Error Handling**
```go
func (m *Model) View() string {
width, height := m.styles.GetSize()
if width <= 0 || height <= 0 {
return "Loading..." // Graceful fallback
}
if m.error != nil {
return m.styles.Error.Render("Error: " + m.error.Error())
}
if !m.ready {
return m.styles.Info.Render("Loading content...")
}
return m.viewport.View()
}
```
## 🏗️ **Responsive Layout Architecture**
### **Breakpoint-Based Design**
```go
// Layout Constants
const (
SmallScreenThreshold = 30 // Height threshold for viewport mode
MinTerminalWidth = 80 // Minimum width for horizontal layout
MinPanelWidth = 25 // Panel width constraints
WelcomeHeaderHeight = 8 // Fixed by ASCII Art Logo (8 lines)
EULAHeaderHeight = 3 // Title + subtitle + spacing
FooterHeight = 1 // Always 1 line with background approach
)
// Responsive layout detection
func (m *Model) isVerticalLayout() bool {
width := m.styles.GetWidth()
return width < MinTerminalWidth
}
func (m *Model) isSmallScreen() bool {
height := m.styles.GetHeight()
return height < SmallScreenThreshold
}
// Adaptive content rendering
func (m *Model) View() string {
width, height := m.styles.GetSize()
leftPanel := m.renderContent()
rightPanel := m.renderInfo()
if m.isVerticalLayout() {
return m.renderVerticalLayout(leftPanel, rightPanel, width, height)
}
return m.renderHorizontalLayout(leftPanel, rightPanel, width, height)
}
func (m *Model) renderVerticalLayout(leftPanel, rightPanel string, width, height int) string {
verticalStyle := lipgloss.NewStyle().Width(width).Padding(0, 2, 0, 2)
leftStyled := verticalStyle.Render(leftPanel)
rightStyled := verticalStyle.Render(rightPanel)
// Hide right panel if both don't fit
if lipgloss.Height(leftStyled)+lipgloss.Height(rightStyled)+2 < height {
return lipgloss.JoinVertical(lipgloss.Left,
leftStyled,
verticalStyle.Height(1).Render(""),
rightStyled,
)
}
// Show only essential left panel
return leftStyled
}
func (m *Model) renderHorizontalLayout(leftPanel, rightPanel string, width, height int) string {
leftWidth := width * 2 / 3
rightWidth := width - leftWidth - 4
if leftWidth < MinPanelWidth {
leftWidth = MinPanelWidth
rightWidth = width - leftWidth - 4
}
leftStyled := lipgloss.NewStyle().Width(leftWidth).Padding(0, 2, 0, 2).Render(leftPanel)
rightStyled := lipgloss.NewStyle().Width(rightWidth).PaddingLeft(2).Render(rightPanel)
return lipgloss.JoinHorizontal(lipgloss.Top, leftStyled, rightStyled)
}
```
## 🏗️ **Content Loading Architecture**
### **Async Content Loading**
```go
// Content loading pattern
func (m *Model) loadContent() tea.Cmd {
return func() tea.Msg {
logger.Log("[Model] LOAD: start")
content, err := m.loadFromSource()
if err != nil {
logger.Errorf("[Model] LOAD: error: %v", err)
return ErrorMsg{err}
}
logger.Log("[Model] LOAD: success (%d chars)", len(content))
return ContentLoadedMsg{content}
}
}
// Safe content loading with fallbacks
func (m *Model) loadFromSource() (string, error) {
// Try embedded filesystem first
if content, err := embedded.GetContent("file.md"); err == nil {
return content, nil
}
// Fallback to direct file access
if content, err := os.ReadFile("file.md"); err == nil {
return string(content), nil
}
// Final fallback
return "Content not available", fmt.Errorf("could not load content")
}
// Handle loading results
func (m *Model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
switch msg := msg.(type) {
case ContentLoadedMsg:
m.content = msg.Content
m.ready = true
// Render with glamour (safe pattern)
rendered, err := m.styles.GetRenderer().Render(m.content)
if err != nil {
// Fallback to plain text
rendered = fmt.Sprintf("# Content\n\n%s\n\n*Render error: %v*", m.content, err)
}
m.viewport.SetContent(rendered)
return m, func() tea.Msg { return nil } // Force re-render
case ErrorMsg:
m.error = msg.Error
m.ready = true
return m, nil
}
return m, nil
}
```
## 🏗️ **Window Management Pattern**
### **Window Integration**
```go
type Window struct {
width int
height int
}
func NewWindow() *Window {
return &Window{width: 80, height: 24}
}
func (w *Window) SetSize(width, height int) {
w.width = width
w.height = height
}
func (w *Window) GetSize() (int, int) {
return w.width, w.height
}
func (w *Window) GetContentSize() (int, int) {
// Account for padding and borders
contentWidth := max(w.width-4, 0)
contentHeight := max(w.height-6, 0) // Header + footer + padding
return contentWidth, contentHeight
}
func (w *Window) GetContentWidth() int {
width, _ := w.GetContentSize()
return width
}
func (w *Window) GetContentHeight() int {
_, height := w.GetContentSize()
return height
}
// Integration with models
func (m *Model) updateDimensions() {
width, height := m.window.GetContentSize()
if width <= 0 || height <= 0 {
return
}
m.viewport.Width = width
m.viewport.Height = height
}
```
## 🏗️ **Controller Integration Pattern**
### **StateController Bridge**
```go
type StateController struct {
state *state.State
}
func NewStateController(state *state.State) *StateController {
return &StateController{state: state}
}
// Environment variable management
func (c *StateController) GetVar(name string) (loader.EnvVar, error) {
return c.state.GetVar(name)
}
func (c *StateController) SetVar(name, value string) error {
return c.state.SetVar(name, value)
}
// Configuration management
func (c *StateController) GetLLMProviders() map[string]ProviderConfig {
// Implementation specific to controller
return c.state.GetLLMProviders()
}
func (c *StateController) SaveConfiguration() error {
return c.state.Save()
}
// Model integration
type Model struct {
controller *StateController
// ... other fields
}
func (m *Model) loadConfiguration() {
configs := m.controller.GetLLMProviders()
// Use configs to populate model state
}
```
## 🏗️ **Essential Key Handling**
### **Universal Key Patterns**
```go
func (m *Model) handleKeyMsg(msg tea.KeyMsg) (tea.Model, tea.Cmd) {
switch msg.String() {
// Universal scroll controls
case "up":
m.viewport.ScrollUp(1)
case "down":
m.viewport.ScrollDown(1)
case "left", "h":
m.viewport.ScrollLeft(2) // 2 steps for faster horizontal
case "right", "l":
m.viewport.ScrollRight(2)
// Page controls
case "pgup":
m.viewport.ScrollUp(m.viewport.Height)
case "pgdown":
m.viewport.ScrollDown(m.viewport.Height)
case "home":
m.viewport.GotoTop()
case "end":
m.viewport.GotoBottom()
// Universal actions (handled at app level)
case "esc":
// Handled by app.go - returns to welcome
return m, nil
case "ctrl+c":
// Handled by app.go - quits application
return m, nil
default:
// Screen-specific handling
return m.handleScreenSpecificKeys(msg)
}
return m, nil
}
func (m *Model) handleScreenSpecificKeys(msg tea.KeyMsg) (tea.Model, tea.Cmd) {
// Override in specific models
return m, nil
}
```
## 🎯 **Architecture Best Practices**
### **Separation of Concerns**
```go
// ✅ CORRECT: Clean separation
// app.go - Navigation, layout, global state
// models/ - Screen-specific logic, local state
// styles/ - Styling, dimensions, rendering
// controller/ - Business logic, state management
// window/ - Terminal size management
// Model responsibilities
type Model struct {
// ONLY screen-specific state
content string
ready bool
// Dependencies (injected)
styles *styles.Styles // Shared styling
window *window.Window // Size management
controller *Controller // Business logic
}
// App responsibilities
type App struct {
// Global state
navigator *Navigator
currentModel tea.Model
// Shared resources
styles *styles.Styles
window *window.Window
controller *Controller
}
```
### **Resource Management**
```go
// ✅ CORRECT: Shared resources
func NewApp() *App {
styles := styles.New()
window := window.New()
controller := controller.New()
return &App{
styles: styles, // Single instance
window: window, // Single instance
controller: controller, // Single instance
navigator: navigator.New(),
}
}
// ❌ WRONG: Resource duplication
func NewModelWrong() *Model {
styles := styles.New() // Multiple instances!
renderer, _ := glamour.New() // Multiple renderers!
return &Model{styles: styles}
}
```
This architecture provides:
- **Scalability**: Clean separation enables easy feature additions
- **Maintainability**: Centralized resources reduce coupling
- **Performance**: Shared instances prevent resource waste
- **Consistency**: Unified patterns across all components
- **Reliability**: Proper error handling and state management
backend/docs/installer/charm-best-practices.md
+649
View File
@@ -0,0 +1,649 @@
# Charm.sh Best Practices & Performance Guide
> Proven patterns and anti-patterns for building high-performance TUI applications.
## 🚀 **Performance Best Practices**
### **Single Glamour Renderer (Critical)**
**Problem**: Multiple renderer instances cause freezing and memory leaks
**Solution**: Create once, reuse everywhere
```go
// ✅ CORRECT: Single renderer instance
type Styles struct {
renderer *glamour.TermRenderer
}
func New() *Styles {
renderer, _ := glamour.NewTermRenderer(
glamour.WithAutoStyle(),
glamour.WithWordWrap(80),
)
return &Styles{renderer: renderer}
}
func (s *Styles) GetRenderer() *glamour.TermRenderer {
return s.renderer
}
// Usage: Always use shared renderer
rendered, _ := m.styles.GetRenderer().Render(content)
// ❌ WRONG: Creating new renderer each time
func renderContent(content string) string {
renderer, _ := glamour.NewTermRenderer(...) // Memory leak + freeze risk!
return renderer.Render(content)
}
```
### **Centralized Dimension Management**
**Problem**: Dimensions stored in models cause sync issues
**Solution**: Single source of truth in styles
```go
// ✅ CORRECT: Centralized dimensions
type Styles struct {
width int
height int
}
func (s *Styles) SetSize(width, height int) {
s.width = width
s.height = height
s.updateStyles() // Recalculate responsive styles
}
func (s *Styles) GetSize() (int, int) {
return s.width, s.height
}
// Models use styles for dimensions
func (m *Model) updateViewport() {
width, height := m.styles.GetSize()
// ... safe dimension usage
}
// ❌ WRONG: Models managing dimensions
type Model struct {
width, height int // Will get out of sync!
}
```
### **Efficient Viewport Usage**
#### **Permanent vs Temporary Viewports**
```go
// ✅ CORRECT: Permanent viewport for forms (preserves scroll state)
type FormModel struct {
viewport viewport.Model // Permanent - keeps scroll position
}
func (m *FormModel) ensureFocusVisible() {
// Scroll calculations use permanent viewport state
if focusY < m.viewport.YOffset {
m.viewport.YOffset = focusY
}
}
// ✅ CORRECT: Temporary viewport for layout rendering
func (m *Model) renderHorizontalLayout(left, right string, width, height int) string {
content := lipgloss.JoinHorizontal(lipgloss.Top, leftStyled, rightStyled)
// Create viewport just for rendering
vp := viewport.New(width, height-PaddingHeight)
vp.SetContent(content)
return vp.View()
}
// ❌ WRONG: Creating viewport in View() - loses scroll state
func (m *FormModel) View() string {
vp := viewport.New(width, height) // State lost on re-render!
return vp.View()
}
```
### **Content Loading Optimization**
```go
// ✅ CORRECT: Lazy loading with caching
type Model struct {
contentCache map[string]string
loadOnce sync.Once
}
func (m *Model) loadContent() tea.Cmd {
return func() tea.Msg {
// Check cache first
if cached, exists := m.contentCache[m.contentKey]; exists {
return ContentLoadedMsg{cached}
}
// Load and cache
content, err := m.loadFromSource()
if err != nil {
return ErrorMsg{err}
}
m.contentCache[m.contentKey] = content
return ContentLoadedMsg{content}
}
}
// ❌ WRONG: Loading on every Init()
func (m *Model) Init() tea.Cmd {
return m.loadContent // Reloads every time!
}
```
## 🎯 **Architecture Best Practices**
### **Clean Separation of Concerns**
```go
// ✅ CORRECT: Clear responsibilities
// app.go - Global navigation, layout management, shared resources
type App struct {
navigator *Navigator // Navigation state
currentModel tea.Model // Current screen
styles *Styles // Shared styling
window *Window // Dimension management
controller *Controller // Business logic
}
// models/ - Screen-specific logic only
type ScreenModel struct {
// Screen-specific state only
content string
ready bool
// Injected dependencies
styles *Styles
window *Window
controller *Controller
}
// controller/ - Business logic, no UI concerns
type Controller struct {
state *State
}
func (c *Controller) GetConfiguration() Config {
// Pure business logic, no UI dependencies
}
// ❌ WRONG: Mixed responsibilities
type Model struct {
// UI state
viewport viewport.Model
// Business logic (should be in controller)
database *sql.DB
apiClient *http.Client
// Global state (should be in app)
allScreens map[string]tea.Model
}
```
### **Resource Management**
```go
// ✅ CORRECT: Dependency injection
func NewApp() *App {
// Create shared resources once
styles := styles.New()
window := window.New()
controller := controller.New()
return &App{
styles: styles,
window: window,
controller: controller,
}
}
func (a *App) createModelForScreen(screenID ScreenID) tea.Model {
// Inject shared dependencies
return NewScreenModel(a.styles, a.window, a.controller)
}
// ❌ WRONG: Resource duplication
func NewScreenModel() *ScreenModel {
return &ScreenModel{
styles: styles.New(), // Multiple instances!
controller: controller.New(), // Multiple instances!
}
}
```
## 🎯 **State Management Best Practices**
### **Complete State Reset Pattern**
```go
// ✅ CORRECT: Reset ALL state in Init()
func (m *Model) Init() tea.Cmd {
// Reset UI state
m.content = ""
m.ready = false
m.error = nil
m.initialized = false
// Reset component state
m.viewport.GotoTop()
m.viewport.SetContent("")
// Reset form state
m.focusedIndex = 0
m.hasChanges = false
for i := range m.fields {
m.fields[i].Input.Blur()
}
// Reset navigation args
m.selectedIndex = m.getSelectedIndexFromArgs()
return m.loadContent
}
// ❌ WRONG: Partial state reset
func (m *Model) Init() tea.Cmd {
m.content = "" // Only resetting some fields!
return m.loadContent
}
```
### **Args-Based Construction**
```go
// ✅ CORRECT: Selection from constructor args
func NewModel(args []string) *Model {
selectedIndex := 0
if len(args) > 0 && args[0] != "" {
for i, item := range items {
if item.ID == args[0] {
selectedIndex = i
break
}
}
}
return &Model{
selectedIndex: selectedIndex,
args: args,
}
}
// No separate SetSelected methods needed
func (m *Model) Init() tea.Cmd {
// Selection already set in constructor
return m.loadData
}
// ❌ WRONG: Separate setter methods
func (m *Model) SetSelectedItem(itemID string) {
// Adds complexity, sync issues
for i, item := range m.items {
if item.ID == itemID {
m.selectedIndex = i
break
}
}
}
```
## 🎯 **Navigation Best Practices**
### **Type-Safe Navigation**
```go
// ✅ CORRECT: Type-safe constants and helpers
type ScreenID string
const (
WelcomeScreen ScreenID = "welcome"
MenuScreen ScreenID = "menu"
)
func CreateScreenID(screen string, args ...string) ScreenID {
if len(args) == 0 {
return ScreenID(screen)
}
return ScreenID(screen + "§" + strings.Join(args, "§"))
}
// Usage
return NavigationMsg{Target: CreateScreenID("form", "provider", "openai")}
// ❌ WRONG: String-based navigation
return NavigationMsg{Target: "form/provider/openai"} // Typo-prone!
```
### **GoBack Navigation Pattern**
```go
// ✅ CORRECT: Use GoBack to prevent loops
func (m *FormModel) saveAndReturn() (tea.Model, tea.Cmd) {
if err := m.saveConfiguration(); err != nil {
return m, nil // Stay on form if save fails
}
return m, func() tea.Msg {
return NavigationMsg{GoBack: true} // Return to previous screen
}
}
// ❌ WRONG: Direct navigation creates loops
func (m *FormModel) saveAndReturn() (tea.Model, tea.Cmd) {
m.saveConfiguration()
return m, func() tea.Msg {
return NavigationMsg{Target: ProvidersScreen} // Creates navigation loop!
}
}
```
## 🎯 **Form Best Practices**
### **Dynamic Width Management**
```go
// ✅ CORRECT: Calculate width dynamically
func (m *FormModel) updateFormContent() {
inputWidth := m.getInputWidth()
for i, field := range m.fields {
// Apply width during rendering, not initialization
field.Input.Width = inputWidth - 3
field.Input.SetValue(field.Input.Value()) // Trigger width update
}
}
func (m *FormModel) getInputWidth() int {
viewportWidth, _ := m.getViewportSize()
inputWidth := viewportWidth - 6 // Account for padding
if m.isVerticalLayout() {
inputWidth = viewportWidth - 4 // Less padding in vertical
}
return inputWidth
}
// ❌ WRONG: Fixed width at initialization
func (m *FormModel) createField() {
input := textinput.New()
input.Width = 50 // Breaks responsive design!
}
```
### **Environment Variable Integration**
```go
// ✅ CORRECT: Track initial state for cleanup
func (m *FormModel) buildForm() {
m.initiallySetFields = make(map[string]bool)
for _, fieldConfig := range m.fieldConfigs {
envVar, _ := m.controller.GetVar(fieldConfig.EnvVarName)
// Track if field was initially set
m.initiallySetFields[fieldConfig.Key] = envVar.IsPresent()
field := m.createFieldFromEnvVar(fieldConfig, envVar)
m.fields = append(m.fields, field)
}
}
func (m *FormModel) saveConfiguration() error {
// First pass: Remove cleared fields
for _, field := range m.fields {
value := strings.TrimSpace(field.Input.Value())
if value == "" && m.initiallySetFields[field.Key] {
// Field was set but now empty - remove from environment
m.controller.SetVar(field.EnvVarName, "")
}
}
// Second pass: Save non-empty values
for _, field := range m.fields {
value := strings.TrimSpace(field.Input.Value())
if value != "" {
m.controller.SetVar(field.EnvVarName, value)
}
}
return nil
}
// ❌ WRONG: No cleanup tracking
func (m *FormModel) saveConfiguration() error {
for _, field := range m.fields {
// Always sets value, even if it should be removed
m.controller.SetVar(field.EnvVarName, field.Input.Value())
}
}
```
## 🎯 **Layout Best Practices**
### **Responsive Breakpoints**
```go
// ✅ CORRECT: Consistent breakpoint logic
const (
MinTerminalWidth = 80
MinMenuWidth = 38
MaxMenuWidth = 66
MinInfoWidth = 34
PaddingWidth = 8
)
func (m *Model) isVerticalLayout() bool {
contentWidth := m.window.GetContentWidth()
return contentWidth < (MinMenuWidth + MinInfoWidth + PaddingWidth)
}
func (m *Model) renderHorizontalLayout(leftPanel, rightPanel string, width, height int) string {
leftWidth, rightWidth := MinMenuWidth, MinInfoWidth
extraWidth := width - leftWidth - rightWidth - PaddingWidth
if extraWidth > 0 {
leftWidth = min(leftWidth+extraWidth/2, MaxMenuWidth) // Cap at max
rightWidth = width - leftWidth - PaddingWidth/2
}
// ... render with calculated widths
}
// ❌ WRONG: Arbitrary breakpoints
func (m *Model) isVerticalLayout() bool {
return m.width < 85 // Magic number!
}
```
### **Content Hiding Strategy**
```go
// ✅ CORRECT: Graceful content hiding
func (m *Model) renderVerticalLayout(leftPanel, rightPanel string, width, height int) string {
leftStyled := verticalStyle.Render(leftPanel)
rightStyled := verticalStyle.Render(rightPanel)
// Show both panels if they fit
if lipgloss.Height(leftStyled)+lipgloss.Height(rightStyled)+2 < height {
return lipgloss.JoinVertical(lipgloss.Left, leftStyled, "", rightStyled)
}
// Hide right panel if insufficient space - show only essential content
return leftStyled
}
// ❌ WRONG: Always showing all content
func (m *Model) renderVerticalLayout(leftPanel, rightPanel string, width, height int) string {
// Forces both panels even if they don't fit
return lipgloss.JoinVertical(lipgloss.Left, leftPanel, rightPanel)
}
```
## 🚀 **Performance Optimization**
### **Memory Management**
```go
// ✅ CORRECT: Efficient string building
func (m *Model) buildLargeContent() string {
var builder strings.Builder
builder.Grow(1024) // Pre-allocate capacity
for _, section := range m.sections {
builder.WriteString(section)
builder.WriteString("\n")
}
return builder.String()
}
// ❌ WRONG: String concatenation in loop
func (m *Model) buildLargeContent() string {
content := ""
for _, section := range m.sections {
content += section + "\n" // Creates new string each iteration!
}
return content
}
```
### **Viewport Content Updates**
```go
// ✅ CORRECT: Update content only when changed
func (m *Model) updateViewportContent() {
newContent := m.buildContent()
// Only update if content changed
if newContent != m.lastContent {
m.viewport.SetContent(newContent)
m.lastContent = newContent
}
}
// ❌ WRONG: Always updating content
func (m *Model) View() string {
content := m.buildContent()
m.viewport.SetContent(content) // Updates every render!
return m.viewport.View()
}
```
## 🚀 **Error Handling Best Practices**
### **Graceful Degradation**
```go
// ✅ CORRECT: Multiple fallback levels
func (m *Model) View() string {
width, height := m.styles.GetSize()
if width <= 0 || height <= 0 {
return "Loading..." // Dimension fallback
}
if m.error != nil {
return m.styles.Error.Render("Error: " + m.error.Error()) // Error fallback
}
if !m.ready {
return m.styles.Info.Render("Loading content...") // Loading fallback
}
return m.viewport.View() // Normal rendering
}
// ❌ WRONG: No fallbacks
func (m *Model) View() string {
return m.viewport.View() // Crashes if viewport not initialized!
}
```
### **Safe Async Operations**
```go
// ✅ CORRECT: Safe async with error handling
func (m *Model) loadContent() tea.Cmd {
return func() tea.Msg {
defer func() {
if r := recover(); r != nil {
return ErrorMsg{fmt.Errorf("panic in loadContent: %v", r)}
}
}()
content, err := m.loadFromSource()
if err != nil {
return ErrorMsg{err}
}
return ContentLoadedMsg{content}
}
}
// ❌ WRONG: No error handling
func (m *Model) loadContent() tea.Cmd {
return func() tea.Msg {
content, _ := m.loadFromSource() // Ignores errors!
return ContentLoadedMsg{content}
}
}
```
## 🎯 **Key Anti-Patterns to Avoid**
### **❌ Don't Do These**
```go
// ❌ NEVER: Console output in TUI
fmt.Println("debug")
log.Println("debug")
// ❌ NEVER: Multiple glamour renderers
renderer1 := glamour.NewTermRenderer(...)
renderer2 := glamour.NewTermRenderer(...)
// ❌ NEVER: Dimensions in models
type Model struct {
width, height int
}
// ❌ NEVER: Direct navigation creating loops
return NavigationMsg{Target: PreviousScreen}
// ❌ NEVER: Fixed input widths
input.Width = 50
// ❌ NEVER: Partial state reset
func (m *Model) Init() tea.Cmd {
m.content = "" // Missing other state!
}
// ❌ NEVER: ClearScreen during navigation
return tea.Batch(cmd, tea.ClearScreen)
// ❌ NEVER: String-based navigation
return NavigationMsg{Target: "screen_name"}
```
### **✅ Always Do These**
```go
// ✅ ALWAYS: File-based logging
logger.Log("[Component] EVENT: %v", msg)
// ✅ ALWAYS: Single shared renderer
rendered := m.styles.GetRenderer().Render(content)
// ✅ ALWAYS: Centralized dimensions
width, height := m.styles.GetSize()
// ✅ ALWAYS: GoBack navigation
return NavigationMsg{GoBack: true}
// ✅ ALWAYS: Dynamic input sizing
input.Width = m.getInputWidth()
// ✅ ALWAYS: Complete state reset
func (m *Model) Init() tea.Cmd {
m.resetAllState()
return m.loadContent
}
// ✅ ALWAYS: Clean model initialization
return a, a.currentModel.Init()
// ✅ ALWAYS: Type-safe navigation
return NavigationMsg{Target: CreateScreenID("screen", "arg")}
```
This guide ensures:
- **Performance**: Efficient resource usage and rendering
- **Reliability**: Robust error handling and state management
- **Maintainability**: Clean architecture and consistent patterns
- **User Experience**: Responsive design and graceful degradation
backend/docs/installer/charm-core-libraries.md
+431
View File
@@ -0,0 +1,431 @@
# Charm.sh Core Libraries Reference
> Comprehensive guide to the core libraries in the Charm ecosystem for building TUI applications.
## 📦 **Core Libraries Overview**
### Core Packages
- **`bubbletea`**: Event-driven TUI framework (MVU pattern)
- **`lipgloss`**: Styling and layout engine
- **`bubbles`**: Pre-built components (viewport, textinput, etc.)
- **`huh`**: Advanced form builder
- **`glamour`**: Markdown renderer
## 🫧 **BubbleTea (MVU Pattern)**
### Model-View-Update Lifecycle
```go
// Model holds all state
type Model struct {
content string
ready bool
}
// Update handles events and returns new state
func (m Model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
switch msg := msg.(type) {
case tea.WindowSizeMsg:
// Handle resize
return m, nil
case tea.KeyMsg:
// Handle keyboard input
return m, nil
}
return m, nil
}
// View renders current state
func (m Model) View() string {
return "content"
}
// Init returns initial command
func (m Model) Init() tea.Cmd {
return nil
}
```
### Commands and Messages
```go
// Commands return future messages
func loadDataCmd() tea.Msg {
return DataLoadedMsg{data: "loaded"}
}
// Async operations
return m, tea.Cmd(func() tea.Msg {
time.Sleep(time.Second)
return TimerMsg{}
})
```
### Critical Patterns
```go
// Model interface implementation
type Model struct {
styles *styles.Styles // ALWAYS use shared styles
}
func (m Model) Init() tea.Cmd {
// ALWAYS reset state completely
m.content = ""
m.ready = false
return m.loadContent
}
func (m Model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
switch msg := msg.(type) {
case tea.WindowSizeMsg:
// NEVER store dimensions in model - use styles.SetSize()
// Model gets dimensions via m.styles.GetSize()
case tea.KeyMsg:
switch msg.String() {
case "enter": return m, navigateCmd
}
}
}
```
## 🎨 **Lipgloss (Styling & Layout)**
**Purpose**: CSS-like styling for terminal interfaces
**Key Insight**: Height() vs MaxHeight() behavior difference!
### Critical Height Control
```go
// ❌ WRONG: Height() sets MINIMUM height (can expand!)
style := lipgloss.NewStyle().Height(1).Border(lipgloss.NormalBorder())
// ✅ CORRECT: MaxHeight() + Inline() for EXACT height
style := lipgloss.NewStyle().MaxHeight(1).Inline(true)
// ✅ PRODUCTION: Background approach for consistent 1-line footers
footer := lipgloss.NewStyle().
Width(width).
Background(borderColor).
Foreground(textColor).
Padding(0, 1, 0, 1). // Only horizontal padding
Render(text)
// FOOTER APPROACH - PRODUCTION READY (✅ PROVEN SOLUTION)
// ❌ WRONG: Border approach (inconsistent height)
style.BorderTop(true).Height(1)
// ✅ CORRECT: Background approach (always 1 line)
style.Background(color).Foreground(textColor).Padding(0,1,0,1)
```
### Layout Patterns
```go
// LAYOUT COMPOSITION
lipgloss.JoinVertical(lipgloss.Left, header, content, footer)
lipgloss.JoinHorizontal(lipgloss.Top, left, right)
lipgloss.Place(width, height, lipgloss.Center, lipgloss.Top, content)
// Horizontal layout
left := lipgloss.NewStyle().Width(leftWidth).Render(leftContent)
right := lipgloss.NewStyle().Width(rightWidth).Render(rightContent)
combined := lipgloss.JoinHorizontal(lipgloss.Top, left, right)
// Vertical layout with consistent spacing
sections := []string{header, content, footer}
combined := lipgloss.JoinVertical(lipgloss.Left, sections...)
// Centering content
centered := lipgloss.Place(width, height, lipgloss.Center, lipgloss.Center, content)
// Responsive design
verticalStyle := lipgloss.NewStyle().Width(width).Padding(0, 2, 0, 2)
if width < 80 {
// Vertical layout for narrow screens
}
```
### Responsive Patterns
```go
// Breakpoint-based layout
width, height := m.styles.GetSize() // ALWAYS from styles
if width < 80 {
return lipgloss.JoinVertical(lipgloss.Left, panels...)
} else {
return lipgloss.JoinHorizontal(lipgloss.Top, panels...)
}
// Dynamic width allocation
leftWidth := width / 3
rightWidth := width - leftWidth - 4
```
## 📺 **Bubbles (Interactive Components)**
**Purpose**: Pre-built interactive components
**Key Components**: viewport, textinput, list, table
### Viewport - Critical for Scrolling
```go
import "github.com/charmbracelet/bubbles/viewport"
// Setup
viewport := viewport.New(width, height)
viewport.Style = lipgloss.NewStyle() // Clean style prevents conflicts
// Modern scroll methods (use these!)
viewport.ScrollUp(1) // Replaces LineUp()
viewport.ScrollDown(1) // Replaces LineDown()
viewport.ScrollLeft(2) // Horizontal, 2 steps for forms
viewport.ScrollRight(2)
// Deprecated (avoid)
vp.LineUp(lines) // ❌ Deprecated
vp.LineDown(lines) // ❌ Deprecated
// Status tracking
viewport.ScrollPercent() // 0.0 to 1.0
viewport.AtBottom() // bool
viewport.AtTop() // bool
// State checking
isScrollable := !(vp.AtTop() && vp.AtBottom())
progress := vp.ScrollPercent()
// Content management
viewport.SetContent(content)
viewport.View() // Renders visible portion
// Update in message handling
var cmd tea.Cmd
m.viewport, cmd = m.viewport.Update(msg)
```
### TextInput
```go
import "github.com/charmbracelet/bubbles/textinput"
ti := textinput.New()
ti.Placeholder = "Enter text..."
ti.Focus()
ti.EchoMode = textinput.EchoPassword // For masked input
ti.CharLimit = 100
```
## 📝 **Huh (Forms)**
**Purpose**: Advanced form builder for complex user input
```go
import "github.com/charmbracelet/huh"
form := huh.NewForm(
huh.NewGroup(
huh.NewInput().
Key("api_key").
Title("API Key").
Password(). // Masked input
Validate(func(s string) error {
if len(s) < 10 {
return errors.New("API key too short")
}
return nil
}),
huh.NewSelect[string]().
Key("provider").
Title("Provider").
Options(
huh.NewOption("OpenAI", "openai"),
huh.NewOption("Anthropic", "anthropic"),
),
),
)
// Integration with bubbletea
func (m Model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
var cmd tea.Cmd
m.form, cmd = m.form.Update(msg)
if m.form.State == huh.StateCompleted {
// Form submitted - access values
apiKey := m.form.GetString("api_key")
provider := m.form.GetString("provider")
}
return m, cmd
}
```
## ✨ **Glamour (Markdown Rendering)**
**Purpose**: Beautiful markdown rendering in terminal
**CRITICAL**: Create renderer ONCE in styles.New(), reuse everywhere
```go
// ✅ CORRECT: Single renderer instance (prevents freezing)
// styles.go
type Styles struct {
renderer *glamour.TermRenderer
}
func New() *Styles {
renderer, _ := glamour.NewTermRenderer(
glamour.WithAutoStyle(),
glamour.WithWordWrap(80),
)
return &Styles{renderer: renderer}
}
func (s *Styles) GetRenderer() *glamour.TermRenderer {
return s.renderer
}
// Usage in models
rendered, err := m.styles.GetRenderer().Render(markdown)
// ❌ WRONG: Creating new renderer each time (can freeze!)
renderer, _ := glamour.NewTermRenderer(...)
```
### Safe Rendering with Fallback
```go
// Safe rendering with fallback
rendered, err := renderer.Render(content)
if err != nil {
// Fallback to plain text
rendered = fmt.Sprintf("# Content\n\n%s\n\n*Render error: %v*", content, err)
}
```
## 🏗️ **Core Integration Patterns**
### Centralized Styles & Dimensions
**CRITICAL**: Never store width/height in models - use styles singleton
```go
// ✅ CORRECT: Centralized in styles
type Styles struct {
width int
height int
renderer *glamour.TermRenderer
// ... all styles
}
func (s *Styles) SetSize(width, height int) {
s.width = width
s.height = height
s.updateStyles() // Recalculate responsive styles
}
func (s *Styles) GetSize() (int, int) {
return s.width, s.height
}
// Models use styles for dimensions
func (m *Model) updateViewport() {
width, height := m.styles.GetSize()
if width <= 0 || height <= 0 {
return // Graceful handling
}
// ... viewport setup
}
```
### Component Initialization Pattern
```go
// Standard component initialization
func NewModel(styles *styles.Styles, window *window.Window) *Model {
viewport := viewport.New(window.GetContentSize())
viewport.Style = lipgloss.NewStyle() // Clean style
return &Model{
styles: styles,
window: window,
viewport: viewport,
}
}
func (m *Model) Init() tea.Cmd {
// ALWAYS reset ALL state
m.content = ""
m.ready = false
m.error = nil
return m.loadContent
}
```
### Essential Key Handling
```go
func (m *Model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
switch msg := msg.(type) {
case tea.WindowSizeMsg:
// Update through styles, not directly
// m.styles.SetSize() called by app.go
m.updateViewport()
case tea.KeyMsg:
switch msg.String() {
case "up":
m.viewport.ScrollUp(1)
case "down":
m.viewport.ScrollDown(1)
case "left":
m.viewport.ScrollLeft(2)
case "right":
m.viewport.ScrollRight(2)
case "pgup":
m.viewport.ScrollUp(m.viewport.Height)
case "pgdown":
m.viewport.ScrollDown(m.viewport.Height)
case "home":
m.viewport.GotoTop()
case "end":
m.viewport.GotoBottom()
}
}
return m, nil
}
```
## 🔧 **Common Integration Patterns**
### Content Loading
```go
// Load content asynchronously
func (m *Model) loadContent() tea.Cmd {
return func() tea.Msg {
content, err := loadFromSource()
if err != nil {
return ErrorMsg{err}
}
return ContentLoadedMsg{content}
}
}
// Handle loading in Update
case ContentLoadedMsg:
m.content = msg.Content
m.ready = true
m.viewport.SetContent(m.content)
return m, nil
```
### Error Handling
```go
// Graceful error handling
func (m *Model) View() string {
width, height := m.styles.GetSize()
if width <= 0 || height <= 0 {
return "Loading..." // Graceful fallback
}
if m.error != nil {
return m.styles.Error.Render("Error: " + m.error.Error())
}
if !m.ready {
return m.styles.Info.Render("Loading content...")
}
return m.viewport.View()
}
```
This reference provides the foundation for building robust TUI applications with the Charm ecosystem. Each library serves a specific purpose and when combined correctly, creates powerful terminal interfaces.
backend/docs/installer/charm-debugging-guide.md
+623
View File
@@ -0,0 +1,623 @@
# Charm.sh Debugging & Troubleshooting Guide
> Comprehensive guide to debugging TUI applications and avoiding common pitfalls.
## 🔧 **TUI-Safe Logging System**
### **File-Based Logger Pattern**
**Problem**: fmt.Printf breaks TUI rendering
**Solution**: File-based logger with structured output
```go
// logger.go - TUI-safe logging implementation
package logger
import (
"encoding/json"
"os"
"time"
)
type LogEntry struct {
Timestamp string `json:"timestamp"`
Level string `json:"level"`
Component string `json:"component"`
Message string `json:"message"`
Data any `json:"data,omitempty"`
}
var logFile *os.File
func init() {
var err error
logFile, err = os.OpenFile("log.json", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0644)
if err != nil {
panic(err)
}
}
func Log(format string, args ...any) {
writeLog("INFO", fmt.Sprintf(format, args...))
}
func Errorf(format string, args ...any) {
writeLog("ERROR", fmt.Sprintf(format, args...))
}
func Debugf(format string, args ...any) {
writeLog("DEBUG", fmt.Sprintf(format, args...))
}
func LogWithData(message string, data any) {
entry := LogEntry{
Timestamp: time.Now().Format(time.RFC3339),
Level: "INFO",
Message: message,
Data: data,
}
jsonData, _ := json.Marshal(entry)
logFile.Write(append(jsonData, '\n'))
}
func writeLog(level, message string) {
entry := LogEntry{
Timestamp: time.Now().Format(time.RFC3339),
Level: level,
Message: message,
}
jsonData, _ := json.Marshal(entry)
logFile.Write(append(jsonData, '\n'))
}
```
### **Development Monitoring**
```bash
# Monitor logs in separate terminal during development
tail -f log.json | jq '.'
# Filter by component
tail -f log.json | jq 'select(.component == "FormModel")'
# Filter by level
tail -f log.json | jq 'select(.level == "ERROR")'
# Real-time pretty printing
tail -f log.json | jq -r '"\(.timestamp) [\(.level)] \(.message)"'
```
### **Safe Debug Output**
```go
// ❌ NEVER: Breaks TUI rendering
fmt.Println("debug")
log.Println("debug")
os.Stdout.WriteString("debug")
// ✅ ALWAYS: File-based logging
logger.Log("[Component] Event: %v", msg)
logger.Log("[Model] UPDATE: key=%s", msg.String())
logger.Log("[Model] VIEWPORT: %dx%d ready=%v", width, height, m.ready)
logger.Errorf("[Model] ERROR: %v", err)
// Development pattern
func (m *Model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
logger.Log("[%s] UPDATE: %T", m.componentName, msg)
switch msg := msg.(type) {
case tea.KeyMsg:
logger.Log("[%s] KEY: %s", m.componentName, msg.String())
return m.handleKeyMsg(msg)
}
return m, nil
}
```
## 🔧 **Key Debugging Techniques**
### **Dimension Debugging**
```go
func (m *Model) debugDimensions() {
width, height := m.styles.GetSize()
contentWidth, contentHeight := m.window.GetContentSize()
logger.LogWithData("Dimensions Debug", map[string]interface{}{
"terminal_size": fmt.Sprintf("%dx%d", width, height),
"content_size": fmt.Sprintf("%dx%d", contentWidth, contentHeight),
"viewport_size": fmt.Sprintf("%dx%d", m.viewport.Width, m.viewport.Height),
"viewport_offset": m.viewport.YOffset,
"viewport_percent": m.viewport.ScrollPercent(),
"is_vertical": m.isVerticalLayout(),
})
}
func (m *Model) View() string {
width, height := m.styles.GetSize()
if width <= 0 || height <= 0 {
logger.Log("[%s] VIEW: invalid dimensions %dx%d", m.componentName, width, height)
return "Loading..." // Graceful fallback
}
// Debug dimensions on resize
if m.lastWidth != width || m.lastHeight != height {
m.debugDimensions()
m.lastWidth, m.lastHeight = width, height
}
return m.viewport.View()
}
```
### **Navigation Stack Debugging**
```go
func (n *Navigator) debugStack() {
stackInfo := make([]string, len(n.stack))
for i, screenID := range n.stack {
stackInfo[i] = string(screenID)
}
logger.LogWithData("Navigation Stack", map[string]interface{}{
"stack": stackInfo,
"current": string(n.Current()),
"depth": len(n.stack),
})
}
func (n *Navigator) Push(screenID ScreenID) {
logger.Log("[Navigator] PUSH: %s", string(screenID))
n.stack = append(n.stack, screenID)
n.debugStack()
n.persistState()
}
func (n *Navigator) Pop() ScreenID {
if len(n.stack) <= 1 {
logger.Log("[Navigator] POP: cannot pop last screen")
return n.stack[0]
}
popped := n.stack[len(n.stack)-1]
n.stack = n.stack[:len(n.stack)-1]
logger.Log("[Navigator] POP: %s -> %s", string(popped), string(n.Current()))
n.debugStack()
n.persistState()
return popped
}
```
### **Form State Debugging**
```go
func (m *FormModel) debugFormState() {
fields := make([]map[string]interface{}, len(m.fields))
for i, field := range m.fields {
fields[i] = map[string]interface{}{
"key": field.Key,
"value": field.Input.Value(),
"placeholder": field.Input.Placeholder,
"focused": i == m.focusedIndex,
"width": field.Input.Width,
}
}
logger.LogWithData("Form State", map[string]interface{}{
"focused_index": m.focusedIndex,
"has_changes": m.hasChanges,
"show_values": m.showValues,
"field_count": len(m.fields),
"fields": fields,
})
}
func (m *FormModel) validateField(index int) {
logger.Log("[FormModel] VALIDATE: field %d (%s)", index, m.fields[index].Key)
// ... validation logic ...
if hasError {
logger.Log("[FormModel] VALIDATE: field %s failed - %s",
m.fields[index].Key, errorMsg)
}
}
```
### **Content Loading Debugging**
```go
func (m *Model) loadContent() tea.Cmd {
return func() tea.Msg {
logger.Log("[%s] LOAD: starting content load", m.componentName)
// Try multiple sources with detailed logging
sources := []func() (string, error){
m.loadFromEmbedded,
m.loadFromFile,
m.loadFromFallback,
}
for i, loadFunc := range sources {
logger.Log("[%s] LOAD: trying source %d", m.componentName, i+1)
content, err := loadFunc()
if err != nil {
logger.Errorf("[%s] LOAD: source %d failed: %v", m.componentName, i+1, err)
continue
}
logger.Log("[%s] LOAD: source %d success (%d chars)",
m.componentName, i+1, len(content))
return ContentLoadedMsg{content}
}
logger.Errorf("[%s] LOAD: all sources failed", m.componentName)
return ErrorMsg{fmt.Errorf("failed to load content")}
}
}
```
## 🔧 **Common Pitfalls & Solutions**
### **1. Glamour Renderer Freezing**
**Problem**: Creating new renderer instances can freeze
**Solution**: Single shared renderer in styles.New()
```go
// ❌ WRONG: New renderer each time
func (m *Model) renderMarkdown(content string) string {
renderer, _ := glamour.NewTermRenderer(...) // Can freeze!
return renderer.Render(content)
}
// ✅ CORRECT: Shared renderer instance
func (m *Model) renderMarkdown(content string) string {
rendered, err := m.styles.GetRenderer().Render(content)
if err != nil {
logger.Errorf("[%s] RENDER: glamour error: %v", m.componentName, err)
// Fallback to plain text
return fmt.Sprintf("# Content\n\n%s\n\n*Render error: %v*", content, err)
}
return rendered
}
// Debug renderer creation
func NewStyles() *Styles {
logger.Log("[Styles] Creating glamour renderer")
renderer, err := glamour.NewTermRenderer(
glamour.WithAutoStyle(),
glamour.WithWordWrap(80),
)
if err != nil {
logger.Errorf("[Styles] Failed to create renderer: %v", err)
panic(err)
}
logger.Log("[Styles] Glamour renderer created successfully")
return &Styles{renderer: renderer}
}
```
### **2. Footer Height Inconsistency**
**Problem**: Border-based footers vary in height
**Solution**: Background approach with padding
```go
// ❌ WRONG: Border approach (height varies)
func createFooterWrong(width int, text string) string {
logger.Log("[Footer] Using border approach - height may vary")
return lipgloss.NewStyle().
Height(1).
Border(lipgloss.Border{Top: true}).
Render(text)
}
// ✅ CORRECT: Background approach (exactly 1 line)
func createFooter(width int, text string) string {
logger.Log("[Footer] Using background approach - consistent height")
return lipgloss.NewStyle().
Background(lipgloss.Color("240")).
Foreground(lipgloss.Color("255")).
Padding(0, 1, 0, 1).
Render(text)
}
// Debug footer height
func (a *App) View() string {
header := a.renderHeader()
footer := a.renderFooter()
content := a.currentModel.View()
headerHeight := lipgloss.Height(header)
footerHeight := lipgloss.Height(footer)
logger.LogWithData("Layout Heights", map[string]interface{}{
"header_height": headerHeight,
"footer_height": footerHeight,
"total_height": a.styles.GetHeight(),
})
contentHeight := max(a.styles.GetHeight() - headerHeight - footerHeight, 0)
contentArea := a.styles.Content.Height(contentHeight).Render(content)
return lipgloss.JoinVertical(lipgloss.Left, header, contentArea, footer)
}
```
### **3. Dimension Synchronization Issues**
**Problem**: Models store their own width/height, get out of sync
**Solution**: Centralize dimensions in styles singleton
```go
// ❌ WRONG: Models managing their own dimensions
type ModelWrong struct {
width, height int // Will get out of sync!
}
func (m *ModelWrong) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
switch msg := msg.(type) {
case tea.WindowSizeMsg:
m.width, m.height = msg.Width, msg.Height // Inconsistent!
logger.Log("[Model] Direct dimension update: %dx%d", m.width, m.height)
}
}
// ✅ CORRECT: Centralized dimension management
type Model struct {
styles *styles.Styles // Access via styles.GetSize()
}
func (m *Model) updateViewport() {
width, height := m.styles.GetSize()
logger.Log("[%s] Using centralized dimensions: %dx%d", m.componentName, width, height)
if width <= 0 || height <= 0 {
logger.Log("[%s] Invalid dimensions, skipping update", m.componentName)
return
}
// ... safe viewport update
}
```
### **4. TUI Rendering Corruption**
**Problem**: Console output breaks rendering
**Solution**: File-based logger, never fmt.Printf
```go
// ❌ NEVER: Use tea.ClearScreen during navigation
func (a *App) handleNavigation() (tea.Model, tea.Cmd) {
logger.Log("[App] Navigation: using ClearScreen")
return a, tea.Batch(cmd, tea.ClearScreen) // Corrupts rendering!
}
// ✅ CORRECT: Let model Init() handle clean state
func (a *App) handleNavigation() (tea.Model, tea.Cmd) {
logger.Log("[App] Navigation: clean model initialization")
return a, a.currentModel.Init()
}
// Debug rendering corruption
func (m *Model) View() string {
view := m.viewport.View()
// Debug view corruption
if strings.Contains(view, "\x1b[2J") || strings.Contains(view, "\x1b[H") {
logger.Errorf("[%s] VIEW: detected ANSI clear sequences", m.componentName)
}
logger.Log("[%s] VIEW: rendered %d chars", m.componentName, len(view))
return view
}
```
### **5. Navigation State Issues**
**Problem**: Models retain state between visits
**Solution**: Complete state reset in Init()
```go
// ❌ WRONG: Partial state reset
func (m *Model) Init() tea.Cmd {
logger.Log("[%s] INIT: partial reset", m.componentName)
m.content = "" // Only resetting some fields!
return m.loadContent
}
// ✅ CORRECT: Complete state reset
func (m *Model) Init() tea.Cmd {
logger.Log("[%s] INIT: complete state reset", m.componentName)
// Reset ALL state fields
m.content = ""
m.ready = false
m.error = nil
m.initialized = false
m.scrolled = false
m.focusedIndex = 0
m.hasChanges = false
// Reset component state
m.viewport.GotoTop()
m.viewport.SetContent("")
// Reset form state if applicable
for i := range m.fields {
m.fields[i].Input.Blur()
}
logger.Log("[%s] INIT: state reset complete", m.componentName)
return m.loadContent
}
```
## 🔧 **Performance Debugging**
### **Viewport Performance**
```go
func (m *Model) debugViewportPerformance() {
start := time.Now()
// Measure viewport operations
m.viewport.SetContent(m.content)
setContentDuration := time.Since(start)
start = time.Now()
view := m.viewport.View()
viewDuration := time.Since(start)
logger.LogWithData("Viewport Performance", map[string]interface{}{
"content_size": len(m.content),
"rendered_size": len(view),
"set_content_ms": setContentDuration.Milliseconds(),
"view_render_ms": viewDuration.Milliseconds(),
"viewport_height": m.viewport.Height,
"total_lines": strings.Count(m.content, "\n"),
"scroll_percent": m.viewport.ScrollPercent(),
})
}
```
### **Memory Usage Tracking**
```go
import "runtime"
func (m *Model) debugMemoryUsage(operation string) {
var memStats runtime.MemStats
runtime.ReadMemStats(&memStats)
logger.LogWithData("Memory Usage", map[string]interface{}{
"operation": operation,
"alloc_mb": memStats.Alloc / 1024 / 1024,
"total_alloc_mb": memStats.TotalAlloc / 1024 / 1024,
"sys_mb": memStats.Sys / 1024 / 1024,
"num_gc": memStats.NumGC,
})
}
// Usage in critical operations
func (m *Model) updateFormContent() {
m.debugMemoryUsage("form_update_start")
// ... form update logic ...
m.debugMemoryUsage("form_update_end")
}
```
## 🔧 **Error Recovery Patterns**
### **Graceful Degradation**
```go
func (m *Model) View() string {
defer func() {
if r := recover(); r != nil {
logger.Errorf("[%s] VIEW: panic recovered: %v", m.componentName, r)
}
}()
// Multi-level fallbacks
width, height := m.styles.GetSize()
if width <= 0 || height <= 0 {
logger.Log("[%s] VIEW: invalid dimensions, using fallback", m.componentName)
return "Loading..."
}
if m.error != nil {
logger.Log("[%s] VIEW: error state, showing error message", m.componentName)
return m.styles.Error.Render("Error: " + m.error.Error())
}
if !m.ready {
logger.Log("[%s] VIEW: not ready, showing loading", m.componentName)
return m.styles.Info.Render("Loading content...")
}
return m.viewport.View()
}
```
### **State Recovery**
```go
func (m *Model) recoverFromError(err error) tea.Cmd {
logger.Errorf("[%s] ERROR: %v", m.componentName, err)
// Try to recover state
m.error = err
m.ready = true
// Attempt graceful recovery
return func() tea.Msg {
logger.Log("[%s] RECOVERY: attempting state recovery", m.componentName)
// Try to reload content
if content, loadErr := m.loadFallbackContent(); loadErr == nil {
logger.Log("[%s] RECOVERY: fallback content loaded", m.componentName)
return ContentLoadedMsg{content}
}
logger.Log("[%s] RECOVERY: using minimal content", m.componentName)
return ContentLoadedMsg{"# Error\n\nContent temporarily unavailable."}
}
}
```
## 🔧 **Testing Strategies**
### **Manual Testing Checklist**
```go
// Test dimensions
// 1. Resize terminal to various sizes
// 2. Test minimum dimensions (80x24)
// 3. Test very narrow terminals (< 80 cols)
// 4. Test very short terminals (< 24 rows)
func (m *Model) testDimensions() {
testSizes := []struct{ width, height int }{
{80, 24}, // Standard
{40, 12}, // Small
{120, 40}, // Large
{20, 10}, // Tiny
}
for _, size := range testSizes {
m.styles.SetSize(size.width, size.height)
view := m.View()
logger.LogWithData("Dimension Test", map[string]interface{}{
"test_size": fmt.Sprintf("%dx%d", size.width, size.height),
"view_length": len(view),
"has_ansi": strings.Contains(view, "\x1b["),
"line_count": strings.Count(view, "\n"),
})
}
}
```
### **Navigation Testing**
```go
func testNavigationFlow() {
// Test complete navigation flow
testSteps := []struct {
action string
expected string
}{
{"start", "welcome"},
{"continue", "main_menu"},
{"select_providers", "llm_providers"},
{"select_openai", "llm_provider_form§openai"},
{"go_back", "llm_providers"},
{"esc", "welcome"},
}
for _, step := range testSteps {
logger.LogWithData("Navigation Test", map[string]interface{}{
"action": step.action,
"expected": step.expected,
"actual": string(navigator.Current()),
})
}
}
```
This debugging guide provides comprehensive tools for:
- **Safe Development**: TUI-compatible logging without rendering corruption
- **State Inspection**: Real-time monitoring of component state
- **Performance Analysis**: Memory and viewport performance tracking
- **Error Recovery**: Graceful degradation and state recovery patterns
- **Testing Strategies**: Systematic approaches to manual testing
backend/docs/installer/charm-form-patterns.md
+557
View File
@@ -0,0 +1,557 @@
# Charm.sh Advanced Form Patterns
> Comprehensive guide to building sophisticated forms using Charm ecosystem libraries.
## 🎯 **Advanced Form Field Patterns**
### **Boolean Fields with Tab Completion**
**Innovation**: Auto-completion for boolean values with suggestions
```go
import "github.com/charmbracelet/bubbles/textinput"
func createBooleanField() textinput.Model {
input := textinput.New()
input.Prompt = ""
input.ShowSuggestions = true
input.SetSuggestions([]string{"true", "false"}) // Enable tab completion
// Show default value in placeholder
input.Placeholder = "true (default)" // Or "false (default)"
return input
}
// Tab completion handler in Update()
func (m *FormModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
switch msg := msg.(type) {
case tea.KeyMsg:
switch msg.String() {
case "tab":
// Complete boolean suggestion
if m.focusedField.Input.ShowSuggestions {
suggestion := m.focusedField.Input.CurrentSuggestion()
if suggestion != "" {
m.focusedField.Input.SetValue(suggestion)
m.focusedField.Input.CursorEnd()
return m, nil
}
}
}
}
return m, nil
}
```
### **Integer Fields with Range Validation**
**Innovation**: Real-time validation with human-readable formatting
```go
type IntegerFieldConfig struct {
Key string
Title string
Description string
Min int
Max int
Default int
}
func (m *FormModel) addIntegerField(config IntegerFieldConfig) {
input := textinput.New()
input.Prompt = ""
input.PlaceholderStyle = m.styles.FormPlaceholder
// Human-readable placeholder with default
input.Placeholder = fmt.Sprintf("%s (%s default)",
formatNumber(config.Default), formatBytes(config.Default))
// Add validation range to description
fullDescription := fmt.Sprintf("%s (Range: %s - %s)",
config.Description, formatBytes(config.Min), formatBytes(config.Max))
field := FormField{
Key: config.Key,
Title: config.Title,
Description: fullDescription,
Input: input,
Min: config.Min,
Max: config.Max,
}
m.fields = append(m.fields, field)
}
// Real-time validation
func (m *FormModel) validateIntegerField(field *FormField) {
value := field.Input.Value()
if value == "" {
field.Input.Placeholder = fmt.Sprintf("%s (default)", formatNumber(field.Default))
return
}
if intVal, err := strconv.Atoi(value); err != nil {
field.Input.Placeholder = "Enter a valid number or leave empty for default"
} else {
if intVal < field.Min || intVal > field.Max {
field.Input.Placeholder = fmt.Sprintf("Range: %s - %s",
formatBytes(field.Min), formatBytes(field.Max))
} else {
field.Input.Placeholder = "" // Clear error
}
}
}
```
### **Value Formatting Utilities**
**Critical**: Consistent formatting across all forms
```go
// Universal byte formatting for configuration values
func formatBytes(bytes int) string {
if bytes >= 1048576 {
return fmt.Sprintf("%.1fMB", float64(bytes)/1048576)
} else if bytes >= 1024 {
return fmt.Sprintf("%.1fKB", float64(bytes)/1024)
}
return fmt.Sprintf("%d bytes", bytes)
}
// Universal number formatting for display
func formatNumber(num int) string {
if num >= 1000000 {
return fmt.Sprintf("%.1fM", float64(num)/1000000)
} else if num >= 1000 {
return fmt.Sprintf("%.1fK", float64(num)/1000)
}
return strconv.Itoa(num)
}
// Usage in forms and info panels
sections = append(sections, fmt.Sprintf("• Memory Limit: %s", formatBytes(memoryLimit)))
sections = append(sections, fmt.Sprintf("• Estimated tokens: ~%s", formatNumber(tokenCount)))
```
## 🎯 **Advanced Form Scrolling with Viewport**
### Auto-Scrolling Forms Pattern
**Problem**: Forms with many fields don't fit on smaller terminals, focused fields go off-screen
**Solution**: Viewport component with automatic scroll-to-focus behavior
```go
import "github.com/charmbracelet/bubbles/viewport"
type FormModel struct {
fields []FormField
focusedIndex int
viewport viewport.Model
formContent string
fieldHeights []int // Heights of each field for scroll calculation
}
// Initialize viewport
func New() *FormModel {
return &FormModel{
viewport: viewport.New(0, 0),
}
}
// Update viewport dimensions on resize
func (m *FormModel) updateViewport() {
contentWidth, contentHeight := m.getContentSize()
m.viewport.Width = contentWidth - 4 // padding
m.viewport.Height = contentHeight - 2 // header/footer space
m.viewport.SetContent(m.formContent)
}
// Render form content and track field positions
func (m *FormModel) updateFormContent() {
var sections []string
m.fieldHeights = []int{}
for i, field := range m.fields {
fieldHeight := 4 // title + description + input + spacing
m.fieldHeights = append(m.fieldHeights, fieldHeight)
sections = append(sections, field.Title)
sections = append(sections, field.Description)
sections = append(sections, field.Input.View())
sections = append(sections, "") // spacing
}
m.formContent = strings.Join(sections, "\n")
m.viewport.SetContent(m.formContent)
}
// Auto-scroll to focused field
func (m *FormModel) ensureFocusVisible() {
if m.focusedIndex >= len(m.fieldHeights) {
return
}
// Calculate Y position of focused field
focusY := 0
for i := 0; i < m.focusedIndex; i++ {
focusY += m.fieldHeights[i]
}
visibleRows := m.viewport.Height
offset := m.viewport.YOffset
// Scroll up if field is above visible area
if focusY < offset {
m.viewport.YOffset = focusY
}
// Scroll down if field is below visible area
if focusY+m.fieldHeights[m.focusedIndex] >= offset+visibleRows {
m.viewport.YOffset = focusY + m.fieldHeights[m.focusedIndex] - visibleRows + 1
}
}
// Navigation with auto-scroll
func (m *FormModel) focusNext() {
m.fields[m.focusedIndex].Input.Blur()
m.focusedIndex = (m.focusedIndex + 1) % len(m.fields)
m.fields[m.focusedIndex].Input.Focus()
m.updateFormContent()
m.ensureFocusVisible() // Key addition!
}
// Render scrollable form
func (m *FormModel) View() string {
return m.viewport.View() // Viewport handles clipping and scrolling
}
```
### Key Benefits of Viewport Forms
- **Automatic Clipping**: Viewport handles content that exceeds available space
- **Smooth Scrolling**: Fields slide into view without jarring jumps
- **Focus Preservation**: Focused field always remains visible
- **No Extra Hotkeys**: Uses standard navigation (Tab, arrows)
- **Terminal Friendly**: Works on any terminal size
### Critical Implementation Details
1. **Field Height Tracking**: Must calculate actual rendered height of each field
2. **Scroll Timing**: Call `ensureFocusVisible()` after every focus change
3. **Content Updates**: Re-render form content when input values change
4. **Viewport Sizing**: Account for padding, headers, footers in size calculation
## 🎯 **Environment Variable Integration Pattern**
**Innovation**: Direct EnvVar integration with presence detection
```go
// EnvVar wrapper (from loader package)
type EnvVar struct {
Key string
Value string // Current value in environment
Default string // Default value from config
}
func (e EnvVar) IsPresent() bool {
return e.Value != "" // Check if actually set in environment
}
// Form field creation from EnvVar
func (m *FormModel) addFieldFromEnvVar(envVarName, fieldKey, title, description string) {
envVar, _ := m.controller.GetVar(envVarName)
// Track initially set fields for cleanup logic
m.initiallySetFields[fieldKey] = envVar.IsPresent()
input := textinput.New()
input.Prompt = ""
// Show default in placeholder if not set
if !envVar.IsPresent() {
input.Placeholder = fmt.Sprintf("%s (default)", envVar.Default)
} else {
input.SetValue(envVar.Value) // Set current value
}
field := FormField{
Key: fieldKey,
Title: title,
Description: description,
Input: input,
EnvVarName: envVarName,
}
m.fields = append(m.fields, field)
}
```
### **Smart Field Cleanup Pattern**
**Innovation**: Environment variable cleanup for empty values
```go
func (m *FormModel) saveConfiguration() error {
// First pass: Remove cleared fields from environment
for _, field := range m.fields {
value := strings.TrimSpace(field.Input.Value())
// If field was initially set but now empty, remove it
if value == "" && m.initiallySetFields[field.Key] {
if err := m.controller.SetVar(field.EnvVarName, ""); err != nil {
return fmt.Errorf("failed to clear %s: %w", field.EnvVarName, err)
}
logger.Log("[FormModel] SAVE: cleared %s", field.EnvVarName)
}
}
// Second pass: Save only non-empty values
for _, field := range m.fields {
value := strings.TrimSpace(field.Input.Value())
if value == "" {
continue // Skip empty - use defaults
}
// Validate before saving
if err := m.validateFieldValue(field, value); err != nil {
return fmt.Errorf("validation failed for %s: %w", field.Key, err)
}
// Save validated value
if err := m.controller.SetVar(field.EnvVarName, value); err != nil {
return fmt.Errorf("failed to set %s: %w", field.EnvVarName, err)
}
logger.Log("[FormModel] SAVE: set %s=%s", field.EnvVarName, value)
}
return nil
}
```
## 🎯 **Resource Estimation Pattern**
**Innovation**: Real-time calculation of resource usage
```go
func (m *ConfigFormModel) calculateResourceEstimate() string {
// Get current form values or defaults
maxMemory := m.getIntValueOrDefault("max_memory")
maxConnections := m.getIntValueOrDefault("max_connections")
cacheSize := m.getIntValueOrDefault("cache_size")
// Algorithm-specific calculations
var estimatedMemory int
switch m.configType {
case "database":
estimatedMemory = maxMemory + (maxConnections * 1024) + cacheSize
case "worker":
estimatedMemory = maxMemory * maxConnections
default:
estimatedMemory = maxMemory
}
// Convert to human-readable format
return fmt.Sprintf("~%s RAM", formatBytes(estimatedMemory))
}
// Helper to get form value or default
func (m *FormModel) getIntValueOrDefault(fieldKey string) int {
// First check current form input
for _, field := range m.fields {
if field.Key == fieldKey {
if value := strings.TrimSpace(field.Input.Value()); value != "" {
if intVal, err := strconv.Atoi(value); err == nil {
return intVal
}
}
}
}
// Fall back to environment default
envVar, _ := m.controller.GetVar(m.getEnvVarName(fieldKey))
if defaultVal, err := strconv.Atoi(envVar.Default); err == nil {
return defaultVal
}
return 0
}
// Display in form content
func (m *FormModel) updateFormContent() {
// ... form fields ...
// Resource estimation section
sections = append(sections, "")
sections = append(sections, m.styles.Subtitle.Render("Resource Estimation"))
sections = append(sections, m.styles.Paragraph.Render("Estimated usage: "+m.calculateResourceEstimate()))
m.formContent = strings.Join(sections, "\n")
m.viewport.SetContent(m.formContent)
}
```
## 🎯 **Current Configuration Preview Pattern**
**Innovation**: Live display of current settings in info panel
```go
func (m *TypeSelectionModel) renderConfigurationPreview() string {
selectedType := m.types[m.selectedIndex]
var sections []string
// Helper to get current environment values
getValue := func(suffix string) string {
envVar, _ := m.controller.GetVar(m.getEnvVarName(selectedType.ID, suffix))
if envVar.Value != "" {
return envVar.Value
}
return envVar.Default + " (default)"
}
getIntValue := func(suffix string) int {
envVar, _ := m.controller.GetVar(m.getEnvVarName(selectedType.ID, suffix))
if envVar.Value != "" {
if val, err := strconv.Atoi(envVar.Value); err == nil {
return val
}
}
if val, err := strconv.Atoi(envVar.Default); err == nil {
return val
}
return 0
}
// Display current configuration
sections = append(sections, m.styles.Subtitle.Render("Current Configuration"))
sections = append(sections, "")
maxMemory := getIntValue("MAX_MEMORY")
timeout := getIntValue("TIMEOUT")
enabled := getValue("ENABLED")
sections = append(sections, fmt.Sprintf("• Max Memory: %s", formatBytes(maxMemory)))
sections = append(sections, fmt.Sprintf("• Timeout: %d seconds", timeout))
sections = append(sections, fmt.Sprintf("• Enabled: %s", enabled))
// Type-specific configuration
if selectedType.ID == "advanced" {
retries := getIntValue("MAX_RETRIES")
sections = append(sections, fmt.Sprintf("• Max Retries: %d", retries))
}
return strings.Join(sections, "\n")
}
```
## 🎯 **Type-Based Dynamic Forms**
**Innovation**: Conditional field generation based on selection
```go
func (m *FormModel) buildDynamicForm() {
m.fields = []FormField{} // Reset
// Common fields for all types
m.addFieldFromEnvVar("ENABLED", "enabled", "Enable Service", "Enable or disable this service")
m.addFieldFromEnvVar("MAX_MEMORY", "max_memory", "Memory Limit", "Maximum memory usage in bytes")
// Type-specific fields
switch m.configType {
case "database":
m.addFieldFromEnvVar("MAX_CONNECTIONS", "max_connections", "Max Connections", "Maximum database connections")
m.addFieldFromEnvVar("CACHE_SIZE", "cache_size", "Cache Size", "Database cache size in bytes")
case "worker":
m.addFieldFromEnvVar("WORKER_COUNT", "worker_count", "Worker Count", "Number of worker processes")
m.addFieldFromEnvVar("QUEUE_SIZE", "queue_size", "Queue Size", "Maximum queue size")
case "api":
m.addFieldFromEnvVar("RATE_LIMIT", "rate_limit", "Rate Limit", "API requests per minute")
m.addFieldFromEnvVar("TIMEOUT", "timeout", "Request Timeout", "Request timeout in seconds")
}
// Set focus on first field
if len(m.fields) > 0 {
m.fields[0].Input.Focus()
}
}
// Environment variable naming helper
func (m *FormModel) getEnvVarName(configType, suffix string) string {
prefix := strings.ToUpper(configType) + "_"
return prefix + suffix
}
```
## 🏗️ **Form Architecture Best Practices**
### Viewport Usage Patterns
#### **Forms: Permanent Viewport Property**
```go
// ✅ For forms with user interaction and scroll state
type FormModel struct {
viewport viewport.Model // Permanent - preserves scroll position
}
func (m *FormModel) ensureFocusVisible() {
// Auto-scroll to focused field
focusY := m.calculateFieldPosition(m.focusedIndex)
if focusY < m.viewport.YOffset {
m.viewport.YOffset = focusY
}
// ... scroll logic
}
```
#### **Layout: Temporary Viewport Creation**
```go
// ✅ For final layout rendering only
func (m *Model) renderHorizontalLayout(left, right string, width, height int) string {
content := lipgloss.JoinHorizontal(lipgloss.Top, leftStyled, rightStyled)
// Create viewport just for layout rendering
vp := viewport.New(width, height-PaddingHeight)
vp.SetContent(content)
return vp.View()
}
```
### Form Field State Management
```go
type FormField struct {
Key string
Title string
Description string
Input textinput.Model
Value string
Required bool
Masked bool
Min int // For integer validation
Max int // For integer validation
EnvVarName string
}
// Dynamic width application
func (m *FormModel) updateFormContent() {
inputWidth := m.getInputWidth()
for i, field := range m.fields {
// Apply dynamic width to input
field.Input.Width = inputWidth - 3 // Account for borders
field.Input.SetValue(field.Input.Value()) // Trigger width update
// Render with consistent styling
inputStyle := m.styles.FormInput.Width(inputWidth)
if i == m.focusedIndex {
inputStyle = inputStyle.BorderForeground(styles.Primary)
}
renderedInput := inputStyle.Render(field.Input.View())
sections = append(sections, renderedInput)
}
}
```
These advanced patterns enable:
- **Smart Validation**: Real-time feedback with user-friendly error messages
- **Resource Awareness**: Live estimation of memory, CPU, or token usage
- **Environment Integration**: Proper handling of defaults, presence detection, and cleanup
- **Type Safety**: Compile-time validation and runtime error handling
- **User Experience**: Auto-completion, formatting, and intuitive navigation
backend/docs/installer/charm-navigation-patterns.md
+535
View File
@@ -0,0 +1,535 @@
# Charm.sh Navigation Patterns
> Comprehensive guide to implementing robust navigation systems in TUI applications.
## 🎯 **Type-Safe Navigation with Composite ScreenIDs**
### **Composite ScreenID Pattern**
**Problem**: Need to pass parameters to screens (e.g., which provider to configure)
**Solution**: Composite ScreenIDs with `§` separator
```go
// Format: "screen§arg1§arg2§..."
type ScreenID string
// Methods for parsing composite IDs
func (s ScreenID) GetScreen() string {
parts := strings.Split(string(s), "§")
return parts[0]
}
func (s ScreenID) GetArgs() []string {
parts := strings.Split(string(s), "§")
if len(parts) <= 1 {
return []string{}
}
return parts[1:]
}
// Helper for creating composite IDs
func CreateScreenID(screen string, args ...string) ScreenID {
if len(args) == 0 {
return ScreenID(screen)
}
parts := append([]string{screen}, args...)
return ScreenID(strings.Join(parts, "§"))
}
```
### **Usage Examples**
```go
// Simple screen (no arguments)
welcome := WelcomeScreen // "welcome"
// Composite screen (with arguments)
providerForm := CreateScreenID("llm_provider_form", "openai") // "llm_provider_form§openai"
// Navigation with arguments
return m, func() tea.Msg {
return NavigationMsg{
Target: CreateScreenID("llm_provider_form", "anthropic"),
Data: FormData{ProviderID: "anthropic"},
}
}
// In createModelForScreen - extract arguments
func (a *App) createModelForScreen(screenID ScreenID, data any) tea.Model {
baseScreen := screenID.GetScreen()
args := screenID.GetArgs()
switch ScreenID(baseScreen) {
case LLMProviderFormScreen:
providerID := "openai" // default
if len(args) > 0 {
providerID = args[0]
}
return NewLLMProviderFormModel(providerID, ...)
}
}
```
### **State Persistence**
```go
// Stack automatically preserves composite IDs
navigator.Push(CreateScreenID("llm_provider_form", "gemini"))
// State contains: ["welcome", "main_menu", "llm_providers", "llm_provider_form§gemini"]
// On restore: user returns to Gemini provider form, not default OpenAI
```
## 🎯 **Navigation Message Pattern**
### **NavigationMsg Structure**
```go
type NavigationMsg struct {
Target ScreenID // Can be simple or composite
GoBack bool // Return to previous screen
Data any // Optional data to pass
}
// Type-safe constants
type ScreenID string
const (
WelcomeScreen ScreenID = "welcome"
EULAScreen ScreenID = "eula"
MainMenuScreen ScreenID = "main_menu"
LLMProviderFormScreen ScreenID = "llm_provider_form"
)
```
### **Navigation Commands**
```go
// Simple navigation
return m, func() tea.Msg {
return NavigationMsg{Target: EULAScreen}
}
// Navigation with parameters
return m, func() tea.Msg {
return NavigationMsg{Target: CreateScreenID("llm_provider_form", "openai")}
}
// Go back to previous screen
return m, func() tea.Msg {
return NavigationMsg{GoBack: true}
}
// Navigation with data passing
return m, func() tea.Msg {
return NavigationMsg{
Target: CreateScreenID("config_form", "database"),
Data: ConfigData{Type: "database", Settings: currentSettings},
}
}
```
## 🎯 **Navigator Implementation**
### **Navigation Stack Management**
```go
type Navigator struct {
stack []ScreenID
stateManager StateManager
}
func NewNavigator(stateManager StateManager) *Navigator {
return &Navigator{
stack: []ScreenID{WelcomeScreen},
stateManager: stateManager,
}
}
func (n *Navigator) Push(screenID ScreenID) {
n.stack = append(n.stack, screenID)
n.persistState()
}
func (n *Navigator) Pop() ScreenID {
if len(n.stack) <= 1 {
return n.stack[0] // Can't pop last screen
}
popped := n.stack[len(n.stack)-1]
n.stack = n.stack[:len(n.stack)-1]
n.persistState()
return popped
}
func (n *Navigator) Current() ScreenID {
if len(n.stack) == 0 {
return WelcomeScreen
}
return n.stack[len(n.stack)-1]
}
func (n *Navigator) Replace(screenID ScreenID) {
if len(n.stack) == 0 {
n.stack = []ScreenID{screenID}
} else {
n.stack[len(n.stack)-1] = screenID
}
n.persistState()
}
func (n *Navigator) persistState() {
stringStack := make([]string, len(n.stack))
for i, screenID := range n.stack {
stringStack[i] = string(screenID)
}
n.stateManager.SetStack(stringStack)
}
func (n *Navigator) RestoreState() {
stringStack := n.stateManager.GetStack()
if len(stringStack) == 0 {
n.stack = []ScreenID{WelcomeScreen}
return
}
n.stack = make([]ScreenID, len(stringStack))
for i, s := range stringStack {
n.stack[i] = ScreenID(s)
}
}
```
## 🎯 **Universal ESC Behavior**
### **Global Navigation Handling**
```go
func (a *App) handleGlobalNavigation(msg tea.KeyMsg) (tea.Model, tea.Cmd) {
switch msg.String() {
case "esc":
// Universal ESC: ALWAYS returns to Welcome screen
if a.navigator.Current().GetScreen() != string(WelcomeScreen) {
a.navigator.stack = []ScreenID{WelcomeScreen}
a.navigator.persistState()
a.currentModel = a.createModelForScreen(WelcomeScreen, nil)
return a, a.currentModel.Init()
}
case "ctrl+c":
// Global quit
return a, tea.Quit
}
return a, nil
}
// In main Update loop
func (a *App) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
switch msg := msg.(type) {
case tea.KeyMsg:
// Handle global navigation first
if newModel, cmd := a.handleGlobalNavigation(msg); cmd != nil {
return newModel, cmd
}
// Then pass to current model
var cmd tea.Cmd
a.currentModel, cmd = a.currentModel.Update(msg)
return a, cmd
case NavigationMsg:
return a.handleNavigationMsg(msg)
}
// Delegate to current model
var cmd tea.Cmd
a.currentModel, cmd = a.currentModel.Update(msg)
return a, cmd
}
```
## 🎯 **Navigation Message Handling**
### **App-Level Navigation**
```go
func (a *App) handleNavigationMsg(msg NavigationMsg) (tea.Model, tea.Cmd) {
if msg.GoBack {
if len(a.navigator.stack) > 1 {
a.navigator.Pop()
currentScreen := a.navigator.Current()
a.currentModel = a.createModelForScreen(currentScreen, msg.Data)
return a, a.currentModel.Init()
}
// Can't go back further, stay on current screen
return a, nil
}
// Forward navigation
a.navigator.Push(msg.Target)
a.currentModel = a.createModelForScreen(msg.Target, msg.Data)
return a, a.currentModel.Init()
}
func (a *App) createModelForScreen(screenID ScreenID, data any) tea.Model {
baseScreen := screenID.GetScreen()
args := screenID.GetArgs()
switch ScreenID(baseScreen) {
case WelcomeScreen:
return NewWelcomeModel(a.controller, a.styles, a.window)
case EULAScreen:
return NewEULAModel(a.controller, a.styles, a.window)
case MainMenuScreen:
selectedItem := ""
if len(args) > 0 {
selectedItem = args[0]
}
return NewMainMenuModel(a.controller, a.styles, a.window, []string{selectedItem})
case LLMProviderFormScreen:
providerID := "openai"
if len(args) > 0 {
providerID = args[0]
}
return NewLLMProviderFormModel(a.controller, a.styles, a.window, []string{providerID})
default:
// Fallback to welcome screen
return NewWelcomeModel(a.controller, a.styles, a.window)
}
}
```
## 🎯 **Args-Based Model Construction**
### **Model Constructor Pattern**
```go
// Model constructor receives args from composite ScreenID
func NewModel(
controller *controllers.StateController, styles *styles.Styles,
window *window.Window, args []string,
) *Model {
// Initialize with selection from args
selectedIndex := 0
if len(args) > 1 && args[1] != "" {
// Find matching item and set selectedIndex
for i, item := range items {
if item.ID == args[1] {
selectedIndex = i
break
}
}
}
return &Model{
controller: controller,
selectedIndex: selectedIndex,
args: args,
}
}
// No separate SetSelected* methods needed
func (m *Model) Init() tea.Cmd {
logger.Log("[Model] INIT: args=%s", strings.Join(m.args, " § "))
// Selection already set in constructor from args
m.loadData()
return nil
}
```
### **Selection Preservation Pattern**
```go
// Navigation from menu with argument preservation
func (m *MenuModel) handleSelection() (tea.Model, tea.Cmd) {
selectedItem := m.getSelectedItem()
// Create composite ScreenID with current selection for stack preservation
return m, func() tea.Msg {
return NavigationMsg{
Target: CreateScreenID(string(targetScreen), selectedItem.ID),
}
}
}
// Form navigation back - use GoBack to avoid stack loops
func (m *FormModel) saveAndReturn() (tea.Model, tea.Cmd) {
model, cmd := m.saveConfiguration()
if cmd != nil {
return model, cmd
}
// ✅ CORRECT: Use GoBack to return to previous screen
return m, func() tea.Msg {
return NavigationMsg{GoBack: true}
}
}
// ❌ WRONG: Direct navigation creates stack loops
return m, func() tea.Msg {
return NavigationMsg{Target: LLMProvidersScreen} // Creates loop!
}
```
## 🎯 **Data Passing Pattern**
### **Structured Data Transfer**
```go
// Define data structures for navigation
type FormData struct {
ProviderID string
Settings map[string]string
}
type ConfigData struct {
Type string
Settings map[string]interface{}
}
// Pass data through navigation
func (m *MenuModel) openConfiguration() (tea.Model, tea.Cmd) {
return m, func() tea.Msg {
return NavigationMsg{
Target: CreateScreenID("config_form", "database"),
Data: ConfigData{
Type: "database",
Settings: m.getCurrentSettings(),
},
}
}
}
// Receive data in target model
func NewConfigFormModel(
controller *controllers.StateController, styles *styles.Styles,
window *window.Window, args []string, data any,
) *ConfigFormModel {
configType := "default"
if len(args) > 0 {
configType = args[0]
}
var settings map[string]interface{}
if configData, ok := data.(ConfigData); ok {
settings = configData.Settings
}
return &ConfigFormModel{
configType: configType,
settings: settings,
// ...
}
}
```
## 🎯 **Navigation Anti-Patterns & Solutions**
### **❌ Common Mistakes**
```go
// ❌ WRONG: Direct navigation creates loops
func (m *FormModel) saveAndReturn() (tea.Model, tea.Cmd) {
m.saveConfiguration()
return m, func() tea.Msg {
return NavigationMsg{Target: LLMProvidersScreen} // Loop!
}
}
// ❌ WRONG: Separate SetSelected methods
func (m *Model) SetSelectedProvider(providerID string) {
// Complexity - removed in favor of args-based construction
}
// ❌ WRONG: String-based navigation (typo-prone)
return NavigationMsg{Target: "main_menu"}
// ❌ WRONG: Manual string concatenation for arguments
return NavigationMsg{Target: ScreenID("llm_provider_form/openai")}
```
### **✅ Correct Patterns**
```go
// ✅ CORRECT: GoBack navigation
func (m *FormModel) saveAndReturn() (tea.Model, tea.Cmd) {
if err := m.saveConfiguration(); err != nil {
return m, nil // Stay on form if save fails
}
return m, func() tea.Msg {
return NavigationMsg{GoBack: true} // Return to previous screen
}
}
// ✅ CORRECT: Args-based selection
func NewModel(..., args []string) *Model {
selectedIndex := 0
if len(args) > 1 && args[1] != "" {
// Set selection from args during construction
for i, item := range items {
if item.ID == args[1] {
selectedIndex = i
break
}
}
}
return &Model{selectedIndex: selectedIndex, args: args}
}
// ✅ CORRECT: Type-safe constants
return NavigationMsg{Target: MainMenuScreen}
// ✅ CORRECT: Composite ScreenID with helper
return NavigationMsg{Target: CreateScreenID("llm_provider_form", "openai")}
```
## 🎯 **Navigation Stack Examples**
### **Typical Navigation Flow**
```go
// Stack progression example:
// 1. Start: ["welcome"]
// 2. Continue: ["welcome", "main_menu"]
// 3. LLM Providers: ["welcome", "main_menu§llm_providers", "llm_providers"]
// 4. OpenAI Form: ["welcome", "main_menu§llm_providers", "llm_providers§openai", "llm_provider_form§openai"]
// 5. GoBack: ["welcome", "main_menu§llm_providers", "llm_providers§openai"]
// 6. ESC: ["welcome"]
func demonstrateNavigation() {
nav := NewNavigator(stateManager)
// Initial state
current := nav.Current() // "welcome"
// Navigate to main menu
nav.Push(CreateScreenID("main_menu", "llm_providers"))
current = nav.Current() // "main_menu§llm_providers"
// Navigate to providers list
nav.Push(CreateScreenID("llm_providers", "openai"))
current = nav.Current() // "llm_providers§openai"
// Navigate to form
nav.Push(CreateScreenID("llm_provider_form", "openai"))
current = nav.Current() // "llm_provider_form§openai"
// Go back
nav.Pop()
current = nav.Current() // "llm_providers§openai"
// ESC to home (clear stack)
nav.stack = []ScreenID{WelcomeScreen}
current = nav.Current() // "welcome"
}
```
### **State Restoration**
```go
// On app restart, navigation stack is restored with all parameters
func (a *App) initializeNavigation() {
a.navigator.RestoreState()
// User returns to exact screen with preserved selection
// e.g., "llm_provider_form§anthropic" restores Anthropic form
currentScreen := a.navigator.Current()
a.currentModel = a.createModelForScreen(currentScreen, nil)
}
```
This navigation system provides:
- **Type Safety**: Compile-time validation of screen IDs
- **Parameter Preservation**: Arguments maintained across navigation
- **Stack Management**: Proper back navigation without loops
- **State Persistence**: Complete navigation state restoration
- **Universal Behavior**: Consistent ESC and global navigation
backend/docs/installer/checker-test-scenarios.md
+155
View File
@@ -0,0 +1,155 @@
# Checker Test Scenarios
This document outlines test scenarios for the installer's system checking functionality, focusing on failure modes and their detection.
## Test Scenarios
### 1. Docker Not Installed
**Setup**: Remove Docker from the system
**Expected**:
- `DockerErrorType`: "not_installed"
- `DockerApiAccessible`: false
- `DockerInstalled`: false
- UI shows: "Docker Not Installed" with installation instructions
### 2. Docker Daemon Not Running
**Setup**: Install Docker but stop the daemon (e.g., quit Docker Desktop on macOS)
**Expected**:
- `DockerErrorType`: "not_running"
- `DockerApiAccessible`: false
- `DockerInstalled`: true
- UI shows: "Docker Daemon Not Running" with start instructions
### 3. Docker Permission Denied
**Setup**: Run installer as non-docker user on Linux
**Expected**:
- `DockerErrorType`: "permission"
- `DockerApiAccessible`: false
- `DockerInstalled`: true
- UI shows: "Docker Permission Denied" with usermod instructions
### 4. Remote Docker Connection Failed
**Setup**: Set DOCKER_HOST to invalid address
**Expected**:
- `DockerErrorType`: "api_error"
- `DockerApiAccessible`: false
- UI shows: "Docker API Connection Failed" with DOCKER_HOST troubleshooting
### 5. Write Permissions Denied
**Setup**: Run installer in read-only directory
**Expected**:
- `EnvDirWritable`: false
- UI shows: "Write Permissions Required" with chmod instructions
### 6. Network Issues - DNS Failure
**Setup**: Block DNS resolution (modify /etc/hosts or firewall)
**Expected**:
- `SysNetworkFailures`: ["• DNS resolution failed for docker.io"]
- UI shows specific DNS failure with resolution steps
### 7. Network Issues - HTTPS Blocked
**Setup**: Block outbound HTTPS (port 443)
**Expected**:
- `SysNetworkFailures`: ["• Cannot reach external services via HTTPS"]
- UI shows HTTPS failure with proxy configuration info
### 8. Network Issues - Docker Registry Blocked
**Setup**: Block docker.io specifically
**Expected**:
- `SysNetworkFailures`: ["• Cannot pull Docker images from registry"]
- UI shows registry access failure
### 9. Behind Corporate Proxy
**Setup**: Network requires proxy, but not configured
**Expected**:
- Multiple network failures
- UI shows proxy configuration instructions for HTTP_PROXY/HTTPS_PROXY
### 10. Low Memory
**Setup**: System with < 2GB available RAM
**Expected**:
- `SysMemoryOK`: false
- `SysMemoryAvailable`: < 2.0
- UI shows memory requirements with specific numbers
### 11. Low Disk Space
**Setup**: System with < 25GB free space
**Expected**:
- `SysDiskFreeSpaceOK`: false
- `SysDiskAvailable`: < 25.0
- UI shows disk requirements with cleanup suggestions
### 12. Worker Docker Environment Issues
**Setup**: Configure DOCKER_HOST for remote, but remote unavailable
**Expected**:
- `WorkerEnvApiAccessible`: false
- UI shows worker environment troubleshooting
## Environment Variable Tests
### 1. HTTP_PROXY Auto-Detection
**Setup**: Set HTTP_PROXY before running installer
**Expected**: PROXY_URL in .env automatically populated
### 2. DOCKER_HOST Inheritance
**Setup**: Set DOCKER_HOST, DOCKER_TLS_VERIFY, DOCKER_CERT_PATH
**Expected**:
- Values synchronized to .env on first run via DoSyncNetworkSettings()
- DOCKER_CERT_PATH migrated to PENTAGI_DOCKER_CERT_PATH (host path) + DOCKER_CERT_PATH set to /opt/pentagi/docker/ssl (container path)
## Edge Cases
### 1. Docker Version Too Old
**Setup**: Docker 19.x installed
**Expected**:
- `DockerVersionOK`: false
- UI shows version upgrade instructions
### 2. Docker Compose Missing
**Setup**: Docker installed without Compose
**Expected**:
- `DockerComposeInstalled`: false
- UI shows Compose installation instructions
### 3. Multiple Failures
**Setup**: No Docker + network issues + low resources
**Expected**: All issues shown in priority order:
1. Environment file
2. Write permissions
3. Docker issues
4. Resource issues
5. Network issues
## Testing Commands
```bash
# Simulate Docker not running (macOS)
osascript -e 'quit app "Docker"'
# Simulate permission issues (Linux)
sudo gpasswd -d $USER docker
# Simulate network issues
sudo iptables -A OUTPUT -p tcp --dport 443 -j DROP
# Simulate DNS issues
echo "127.0.0.1 docker.io" | sudo tee -a /etc/hosts
# Test with proxy
export HTTP_PROXY=http://proxy:3128
export HTTPS_PROXY=http://proxy:3128
# Test remote Docker
export DOCKER_HOST=tcp://remote:2376
export DOCKER_TLS_VERIFY=1
export DOCKER_CERT_PATH=/path/to/certs # auto-migrated to PENTAGI_DOCKER_CERT_PATH on startup
```
## Verification
Each scenario should:
1. Be detected correctly by the checker
2. Show appropriate error message in UI
3. Provide actionable fix instructions
4. Not block other checks unnecessarily
5. Work under both privileged and unprivileged users
backend/docs/installer/checker.md
+179
View File
@@ -0,0 +1,179 @@
# Checker Package Documentation
## Overview
The `checker` package is responsible for gathering system facts and verifying installation prerequisites for PentAGI. It performs comprehensive system analysis to determine the current state of the installation and what operations are available.
## Architecture
### Core Design Principles
1. **Delegation Pattern**: Uses a `CheckHandler` interface to delegate information gathering logic, allowing for flexible implementations and testing
2. **Parallel Information Gathering**: Collects information from multiple sources (Docker, filesystem, network) concurrently
3. **Fail-Safe Approach**: Returns sensible defaults when checks cannot be performed, avoiding false negatives
4. **Context-Aware**: All operations support context for cancellation and timeouts
### Key Components
#### CheckResult Structure
Central data structure that holds all system check results:
- Installation status for each component (PentAGI, Langfuse, Observability)
- System resource availability (CPU, memory, disk)
- Docker environment status
- Network connectivity status
- Update availability information
- Computed values for UI display:
- CPU count
- Required and available memory in GB
- Required and available disk space in GB
- Detailed network failure messages
- Docker error type (not_installed, not_running, permission, api_error)
- Write permissions for configuration directory
#### CheckHandler Interface
```go
type CheckHandler interface {
GatherAllInfo(ctx context.Context, c *CheckResult) error
GatherDockerInfo(ctx context.Context, c *CheckResult) error
GatherWorkerInfo(ctx context.Context, c *CheckResult) error
GatherPentagiInfo(ctx context.Context, c *CheckResult) error
GatherLangfuseInfo(ctx context.Context, c *CheckResult) error
GatherObservabilityInfo(ctx context.Context, c *CheckResult) error
GatherSystemInfo(ctx context.Context, c *CheckResult) error
GatherUpdatesInfo(ctx context.Context, c *CheckResult) error
}
```
## Check Categories
### 1. Docker Environment Checks
- **Docker API Accessibility**: Verifies connection to Docker daemon
- **Docker Error Detection**: Identifies specific Docker issues (not installed, not running, permission denied)
- **Docker Version**: Ensures Docker version >= 20.0.0
- **Docker Compose Version**: Ensures Docker Compose version >= 1.25.0
- **Worker Environment**: Checks separate Docker environment for pentesting tools (supports remote Docker hosts)
### 2. Component Installation Checks
- **File Existence**: Verifies presence of docker-compose files
- **Container Status**: Checks if containers exist and their running state
- **Script Installation**: Verifies PentAGI CLI script in /usr/local/bin
### 3. System Resource Checks
- **Write Permissions**: Verifies write access to configuration directory
- **CPU**: Minimum 2 CPU cores required
- **Memory**: Dynamic calculation based on components to be installed
- Base: 0.5GB free
- PentAGI: +0.5GB
- Langfuse: +1.5GB
- Observability: +1.5GB
- **Disk Space**: Context-aware requirements
- Worker images not present: 25GB (for large pentesting images)
- Components to install: 10GB + 2GB per component
- Already installed: 5GB minimum
### 4. Network Connectivity Checks
Three-tier verification process:
1. **DNS Resolution**: Tests ability to resolve docker.io
2. **HTTP Connectivity**: Verifies HTTPS access (proxy-aware)
3. **Docker Pull Test**: Attempts to pull `debian:latest` (the default worker image) when both Docker clients are available
#### Restricted Network Troubleshooting
The current checker validates Docker Hub reachability by resolving `docker.io`, making an HTTPS connectivity check, and — when both Docker clients are available — attempting a pull of `debian:latest` (the default worker image). This means the installer can fail network validation even when the host has general internet access but Docker itself is not configured for the target network.
Recommended remediation order:
1. Confirm general internet access and DNS resolution for `docker.io`
2. If your environment requires an outbound proxy for installer or PentAGI HTTP traffic, set the `PROXY_URL` environment variable. To route Docker image pulls through a proxy, configure the Docker daemon or Docker Desktop proxy separately — Docker does not use `PROXY_URL` for registry access.
3. If Docker Hub is blocked or rate-limited, configure an organization-approved Docker registry mirror or registry proxy at the Docker daemon / Docker Desktop level
4. Restart Docker and rerun the installer checks
PentAGI variables such as `PENTAGI_IMAGE`, `DOCKER_DEFAULT_IMAGE`, and `DOCKER_DEFAULT_IMAGE_FOR_PENTEST` do not replace Docker daemon registry configuration. They only influence the PentAGI application image or worker image selection after Docker is already able to pull the required images. Note that the main Compose stack already includes a service from `quay.io` (`postgres-exporter`), and the optional observability stack includes an image from `gcr.io`. A Docker Hub mirror alone is therefore not sufficient for a full deployment — those registries also need to be reachable or individually mirrored.
See Docker's official documentation for [registry mirrors](https://docs.docker.com/docker-hub/image-library/mirror/) and [daemon proxy configuration](https://docs.docker.com/engine/daemon/proxy/).
### 5. Update Availability Checks
- Communicates with update server to check latest versions
- Sends current component versions and configuration
- Supports proxy configuration
- Checks updates for: Installer, PentAGI, Langfuse, Observability, Worker images
## Public API
### Main Entry Points
```go
// Gather performs all system checks using provided application state
func Gather(ctx context.Context, appState state.State) (CheckResult, error)
// GatherWithHandler allows custom CheckHandler implementation
func GatherWithHandler(ctx context.Context, handler CheckHandler) (CheckResult, error)
```
### Availability Helper Methods
The CheckResult provides helper methods to determine available operations:
```go
func (c *CheckResult) IsReadyToContinue() bool // Pre-installation checks passed
func (c *CheckResult) CanInstallAll() bool // Can perform installation
func (c *CheckResult) CanStartAll() bool // Can start services
func (c *CheckResult) CanStopAll() bool // Can stop services
func (c *CheckResult) CanUpdateAll() bool // Updates available
func (c *CheckResult) CanFactoryReset() bool // Can reset installation
```
## Implementation Details
### OS-Specific Implementations
- **Memory Checks**:
- Linux: Reads /proc/meminfo for MemAvailable
- macOS: Parses vm_stat output for free + inactive + purgeable pages
- **Disk Space Checks**:
- Uses `df` command with appropriate flags per OS
### Docker Integration
- Supports both local and remote Docker environments
- Handles TLS configuration for secure remote connections
- Compatible with Docker contexts and environment variables
### Error Handling Philosophy
- Network failures are treated as "assume OK" to avoid blocking on transient issues
- Missing system information defaults to "sufficient resources"
- Only critical failures (missing env file, Docker API inaccessible) prevent continuation
### Version Parsing
- Flexible regex-based extraction from various version output formats
- Semantic version comparison for compatibility checks
- Handles both docker-compose and docker compose command variants
### Image Information Extraction
- Parses complex Docker image references (registry/namespace/name:tag@hash)
- Handles various edge cases in image naming conventions
- Extracts version information for update comparison
### Helper Functions for Code Reusability
To avoid code duplication, the package provides several shared helper functions:
- **calculateRequiredMemoryGB**: Calculates total memory requirements based on components that need to be started
- **calculateRequiredDiskGB**: Computes disk space requirements considering worker images and local components
- **countLocalComponentsToInstall**: Counts how many components need local installation
- **determineComponentNeeds**: Determines which components need to be started based on their current state
- **getAvailableMemoryGB**: Platform-specific memory availability detection
- **getAvailableDiskGB**: Platform-specific disk space availability detection
- **getNetworkFailures**: Collects detailed network connectivity failure messages
- **getProxyURL**: Centralized proxy URL retrieval from application state
- **getDockerErrorType**: Identifies specific Docker error types (not installed, not running, permission issues)
- **checkDirIsWritable**: Tests write permissions by creating a temporary file
These functions ensure consistent calculations across different parts of the codebase and make maintenance easier.
## Constants and Thresholds
Key configuration values are defined as constants for easy adjustment:
- Container names for each service
- Minimum resource requirements
- Default endpoints for services
- Update server configuration
- Version compatibility thresholds
## Thread Safety
The default implementation uses mutex protection for Docker client management, ensuring safe concurrent access during information gathering operations.
backend/docs/installer/installer-architecture-design.md
+590
View File
@@ -0,0 +1,590 @@
# PentAGI Installer Architecture & Design Patterns
> Architecture patterns, design decisions, and implementation strategies specific to the PentAGI installer.
## 🏗️ **Unified App Architecture**
### **Central Orchestrator Pattern**
The installer implements a centralized app controller that manages all global concerns:
```go
// File: wizard/app.go
type App struct {
// Navigation state
navigator *Navigator
currentModel tea.Model
// Shared resources (injected into all models)
controller *controllers.StateController
styles *styles.Styles
window *window.Window
// Global state
eulaAccepted bool
systemReady bool
}
func (a *App) View() string {
header := a.renderHeader() // Screen-specific header
footer := a.renderFooter() // Dynamic footer with actions
content := a.currentModel.View() // Content only from model
// App.go enforces layout constraints
contentWidth, contentHeight := a.window.GetContentSize()
contentArea := a.styles.Content.
Width(contentWidth).
Height(contentHeight).
Render(content)
return lipgloss.JoinVertical(lipgloss.Left, header, contentArea, footer)
}
```
### **Responsibilities Separation**
- **App Layer**: Navigation, layout, global state, resource management
- **Model Layer**: Screen-specific logic, user interaction, content rendering
- **Controller Layer**: Business logic, environment variables, configuration
- **Styles Layer**: Presentation, theming, responsive calculations
- **Window Layer**: Terminal size management, dimension coordination
## 🏗️ **Navigation Architecture**
### **Composite ScreenID System**
**Innovation**: Parameters embedded in screen identifiers for type-safe navigation
```go
// Screen ID structure: "screen§arg1§arg2§..."
type ScreenID string
// Helper methods for parsing composite IDs
func (s ScreenID) GetScreen() string {
parts := strings.Split(string(s), "§")
return parts[0]
}
func (s ScreenID) GetArgs() []string {
parts := strings.Split(string(s), "§")
if len(parts) <= 1 {
return []string{}
}
return parts[1:]
}
// Type-safe creation
func CreateScreenID(screen string, args ...string) ScreenID {
if len(args) == 0 {
return ScreenID(screen)
}
return ScreenID(screen + "§" + strings.Join(args, "§"))
}
```
### **Navigator Implementation**
```go
type Navigator struct {
stack []ScreenID
stateManager StateManager // Persists stack across sessions
}
func (n *Navigator) Push(screenID ScreenID) {
n.stack = append(n.stack, screenID)
n.persistState()
}
func (n *Navigator) Pop() ScreenID {
if len(n.stack) <= 1 {
return n.stack[0] // Can't pop welcome screen
}
popped := n.stack[len(n.stack)-1]
n.stack = n.stack[:len(n.stack)-1]
n.persistState()
return popped
}
// Universal ESC behavior
func (a *App) handleGlobalNavigation(msg tea.KeyMsg) (tea.Model, tea.Cmd) {
switch msg.String() {
case "esc":
if a.navigator.Current().GetScreen() != string(WelcomeScreen) {
a.navigator.stack = []ScreenID{WelcomeScreen}
a.navigator.persistState()
a.currentModel = a.createModelForScreen(WelcomeScreen, nil)
return a, a.currentModel.Init()
}
}
return a, nil
}
```
### **Args-Based Model Construction**
```go
func (a *App) createModelForScreen(screenID ScreenID, data any) tea.Model {
baseScreen := screenID.GetScreen()
args := screenID.GetArgs()
switch ScreenID(baseScreen) {
case LLMProviderFormScreen:
providerID := "openai" // default
if len(args) > 0 {
providerID = args[0]
}
return NewLLMProviderFormModel(a.controller, a.styles, a.window, []string{providerID})
case SummarizerFormScreen:
summarizerType := "general" // default
if len(args) > 0 {
summarizerType = args[0]
}
return NewSummarizerFormModel(a.controller, a.styles, a.window, []string{summarizerType})
}
}
```
## 🏗️ **Adaptive Layout Strategy**
### **Responsive Design Pattern**
The installer implements a sophisticated responsive design that adapts to terminal capabilities:
```go
// Layout constants define breakpoints
const (
MinTerminalWidth = 80 // Minimum for horizontal layout
MinMenuWidth = 38 // Minimum left panel width
MaxMenuWidth = 66 // Maximum left panel width (prevents too wide forms)
MinInfoWidth = 34 // Minimum right panel width
PaddingWidth = 8 // Total horizontal padding
)
// Layout decision logic
func (m *Model) isVerticalLayout() bool {
contentWidth := m.window.GetContentWidth()
return contentWidth < (MinMenuWidth + MinInfoWidth + PaddingWidth)
}
// Dynamic width allocation
func (m *Model) renderHorizontalLayout(leftPanel, rightPanel string, width, height int) string {
leftWidth, rightWidth := MinMenuWidth, MinInfoWidth
extraWidth := width - leftWidth - rightWidth - PaddingWidth
// Distribute extra space intelligently, but cap left panel
if extraWidth > 0 {
leftWidth = min(leftWidth+extraWidth/2, MaxMenuWidth)
rightWidth = width - leftWidth - PaddingWidth/2
}
leftStyled := lipgloss.NewStyle().Width(leftWidth).Padding(0, 2, 0, 2).Render(leftPanel)
rightStyled := lipgloss.NewStyle().Width(rightWidth).PaddingLeft(2).Render(rightPanel)
return lipgloss.JoinHorizontal(lipgloss.Top, leftStyled, rightStyled)
}
```
### **Content Hiding Strategy**
```go
func (m *Model) renderVerticalLayout(leftPanel, rightPanel string, width, height int) string {
verticalStyle := lipgloss.NewStyle().Width(width).Padding(0, 4, 0, 2)
leftStyled := verticalStyle.Render(leftPanel)
rightStyled := verticalStyle.Render(rightPanel)
// Show both panels if they fit
if lipgloss.Height(leftStyled)+lipgloss.Height(rightStyled)+2 < height {
return lipgloss.JoinVertical(lipgloss.Left,
leftStyled,
verticalStyle.Height(1).Render(""),
rightStyled,
)
}
// Hide right panel if insufficient space - show only essential content
return leftStyled
}
```
## 🏗️ **Form Architecture Patterns**
### **Production Form Model Structure**
```go
type FormModel struct {
// Standard dependencies (injected)
controller *controllers.StateController
styles *styles.Styles
window *window.Window
// Form state
fields []FormField
focusedIndex int
showValues bool
hasChanges bool
// Environment integration
configType string
typeName string
initiallySetFields map[string]bool // Track for cleanup
// Navigation state
args []string // From composite ScreenID
// Viewport as permanent property (preserves scroll state)
viewport viewport.Model
formContent string
fieldHeights []int
}
```
### **Dynamic Field Generation Pattern**
```go
func (m *FormModel) buildForm() {
m.fields = []FormField{}
m.initiallySetFields = make(map[string]bool)
// Helper function for consistent field creation
addFieldFromEnvVar := func(suffix, key, title, description string) {
envVar, _ := m.controller.GetVar(m.getEnvVarName(suffix))
// Track initial state for cleanup
m.initiallySetFields[key] = envVar.IsPresent()
if key == "preserve_last" || key == "use_qa" {
m.addBooleanField(key, title, description, envVar)
} else {
// Determine validation ranges
var min, max int
switch key {
case "last_sec_bytes", "max_qa_bytes":
min, max = 1024, 1048576 // 1KB to 1MB
case "max_bp_bytes":
min, max = 1024, 524288 // 1KB to 512KB
default:
min, max = 0, 999999
}
m.addIntegerField(key, title, description, envVar, min, max)
}
}
// Type-specific field generation
switch m.configType {
case "general":
addFieldFromEnvVar("USE_QA", "use_qa", locale.SummarizerFormUseQA, locale.SummarizerFormUseQADesc)
addFieldFromEnvVar("SUM_MSG_HUMAN_IN_QA", "sum_human_in_qa", locale.SummarizerFormSumHumanInQA, locale.SummarizerFormSumHumanInQADesc)
case "assistant":
// Assistant-specific fields
}
// Common fields for all types
addFieldFromEnvVar("PRESERVE_LAST", "preserve_last", locale.SummarizerFormPreserveLast, locale.SummarizerFormPreserveLastDesc)
addFieldFromEnvVar("LAST_SEC_BYTES", "last_sec_bytes", locale.SummarizerFormLastSecBytes, locale.SummarizerFormLastSecBytesDesc)
}
```
### **Environment Variable Integration**
```go
// Environment variable naming pattern
func (m *FormModel) getEnvVarName(suffix string) string {
var prefix string
switch m.configType {
case "assistant":
prefix = "ASSISTANT_SUMMARIZER_"
default:
prefix = "SUMMARIZER_"
}
return prefix + suffix
}
// Smart cleanup pattern
func (m *FormModel) saveConfiguration() (tea.Model, tea.Cmd) {
// First pass: Handle fields that were cleared (remove from environment)
for _, field := range m.fields {
value := strings.TrimSpace(field.Input.Value())
// If field was initially set but now empty, remove it
if value == "" && m.initiallySetFields[field.Key] {
envVarName := m.getEnvVarName(getEnvSuffixFromKey(field.Key))
if err := m.controller.SetVar(envVarName, ""); err != nil {
logger.Errorf("[FormModel] SAVE: error clearing %s: %v", envVarName, err)
return m, nil
}
logger.Log("[FormModel] SAVE: cleared %s", envVarName)
}
}
// Second pass: Save only non-empty values
for _, field := range m.fields {
value := strings.TrimSpace(field.Input.Value())
if value == "" {
continue // Skip empty values - use defaults
}
envVarName := m.getEnvVarName(getEnvSuffixFromKey(field.Key))
if err := m.controller.SetVar(envVarName, value); err != nil {
logger.Errorf("[FormModel] SAVE: error setting %s: %v", envVarName, err)
return m, nil
}
}
return m, func() tea.Msg {
return NavigationMsg{GoBack: true}
}
}
```
## 🏗️ **Advanced Form Field Patterns**
### **Boolean Field with Auto-completion**
```go
func (m *FormModel) addBooleanField(key, title, description string, envVar loader.EnvVar) {
input := textinput.New()
input.Prompt = ""
input.PlaceholderStyle = m.styles.FormPlaceholder
input.ShowSuggestions = true
input.SetSuggestions([]string{"true", "false"})
// Show default in placeholder
if envVar.Default == "true" {
input.Placeholder = "true (default)"
} else {
input.Placeholder = "false (default)"
}
// Set value only if actually present in environment
if envVar.Value != "" && envVar.IsPresent() {
input.SetValue(envVar.Value)
}
field := FormField{
Key: key,
Title: title,
Description: description,
Input: input,
Type: "boolean",
}
m.fields = append(m.fields, field)
}
```
### **Integer Field with Validation**
```go
func (m *FormModel) addIntegerField(key, title, description string, envVar loader.EnvVar, min, max int) {
input := textinput.New()
input.Prompt = ""
input.PlaceholderStyle = m.styles.FormPlaceholder
// Parse and format default value
defaultValue := 0
if envVar.Default != "" {
if val, err := strconv.Atoi(envVar.Default); err == nil {
defaultValue = val
}
}
// Human-readable placeholder with default
input.Placeholder = fmt.Sprintf("%s (%s default)",
m.formatNumber(defaultValue), m.formatBytes(defaultValue))
// Set value only if present
if envVar.Value != "" && envVar.IsPresent() {
input.SetValue(envVar.Value)
}
// Add validation range to description
fullDescription := fmt.Sprintf("%s (Range: %s - %s)",
description, m.formatBytes(min), m.formatBytes(max))
field := FormField{
Key: key,
Title: title,
Description: fullDescription,
Input: input,
Type: "integer",
Min: min,
Max: max,
}
m.fields = append(m.fields, field)
}
```
## 🏗️ **Controller Integration Pattern**
### **StateController Bridge**
```go
type StateController struct {
state *state.State
}
func NewStateController(state *state.State) *StateController {
return &StateController{state: state}
}
// Environment variable management
func (c *StateController) GetVar(name string) (loader.EnvVar, error) {
return c.state.GetVar(name)
}
func (c *StateController) SetVar(name, value string) error {
return c.state.SetVar(name, value)
}
// Higher-level configuration management
func (c *StateController) GetLLMProviders() map[string]ProviderConfig {
// Aggregate multiple environment variables into structured config
providers := make(map[string]ProviderConfig)
for _, providerID := range []string{"openai", "anthropic", "gemini", "bedrock", "deepseek", "glm", "kimi", "qwen", "ollama", "custom"} {
config := c.loadProviderConfig(providerID)
providers[providerID] = config
}
return providers
}
func (c *StateController) loadProviderConfig(providerID string) ProviderConfig {
prefix := strings.ToUpper(providerID) + "_"
apiKey, _ := c.GetVar(prefix + "API_KEY")
baseURL, _ := c.GetVar(prefix + "BASE_URL")
return ProviderConfig{
ID: providerID,
Configured: apiKey.IsPresent() && baseURL.IsPresent(),
APIKey: apiKey.Value,
BaseURL: baseURL.Value,
}
}
```
## 🏗️ **Resource Estimation Architecture**
### **Token Calculation Pattern**
```go
func (m *FormModel) calculateTokenEstimate() string {
// Get current form values or defaults
useQAVal := m.getBoolValueOrDefault("use_qa")
lastSecBytesVal := m.getIntValueOrDefault("last_sec_bytes")
maxQABytesVal := m.getIntValueOrDefault("max_qa_bytes")
keepQASectionsVal := m.getIntValueOrDefault("keep_qa_sections")
var estimatedBytes int
// Algorithm-specific calculations
switch m.configType {
case "assistant":
estimatedBytes = keepQASectionsVal * lastSecBytesVal
default: // general
if useQAVal {
basicSize := keepQASectionsVal * lastSecBytesVal
if basicSize > maxQABytesVal {
estimatedBytes = maxQABytesVal
} else {
estimatedBytes = basicSize
}
} else {
estimatedBytes = keepQASectionsVal * lastSecBytesVal
}
}
// Convert to tokens with overhead
estimatedTokens := int(float64(estimatedBytes) * 1.1 / 4) // 4 bytes per token + 10% overhead
return fmt.Sprintf("~%s tokens", m.formatNumber(estimatedTokens))
}
// Helper methods to get form values or environment defaults
func (m *FormModel) getBoolValueOrDefault(key string) bool {
// First check form field value
for _, field := range m.fields {
if field.Key == key && field.Input.Value() != "" {
return field.Input.Value() == "true"
}
}
// Return default value from EnvVar
envVar, _ := m.controller.GetVar(m.getEnvVarName(getEnvSuffixFromKey(key)))
return envVar.Default == "true"
}
```
## 🏗️ **Auto-Scrolling Form Architecture**
### **Viewport-Based Scrolling**
```go
func (m *FormModel) ensureFocusVisible() {
if m.focusedIndex >= len(m.fieldHeights) {
return
}
// Calculate Y position of focused field
focusY := 0
for i := 0; i < m.focusedIndex; i++ {
focusY += m.fieldHeights[i]
}
visibleRows := m.viewport.Height
offset := m.viewport.YOffset
// Scroll up if field is above visible area
if focusY < offset {
m.viewport.YOffset = focusY
}
// Scroll down if field is below visible area
if focusY+m.fieldHeights[m.focusedIndex] >= offset+visibleRows {
m.viewport.YOffset = focusY + m.fieldHeights[m.focusedIndex] - visibleRows + 1
}
}
// Enhanced field navigation with auto-scroll
func (m *FormModel) focusNext() {
if len(m.fields) == 0 {
return
}
m.fields[m.focusedIndex].Input.Blur()
m.focusedIndex = (m.focusedIndex + 1) % len(m.fields)
m.fields[m.focusedIndex].Input.Focus()
m.updateFormContent()
m.ensureFocusVisible() // Key addition for auto-scroll
}
```
## 🏗️ **Layout Integration Architecture**
### **Content Area Management**
```go
// Models handle ONLY content area
func (m *Model) View() string {
leftPanel := m.renderForm()
rightPanel := m.renderHelp()
// Adaptive layout decision
if m.isVerticalLayout() {
return m.renderVerticalLayout(leftPanel, rightPanel, width, height)
}
return m.renderHorizontalLayout(leftPanel, rightPanel, width, height)
}
// App.go handles complete layout structure
func (a *App) View() string {
header := a.renderHeader() // Screen-specific header (logo or title)
footer := a.renderFooter() // Dynamic actions based on screen
content := a.currentModel.View() // Content from model
// Calculate content area size
contentWidth, contentHeight := a.window.GetContentSize()
contentArea := a.styles.Content.
Width(contentWidth).
Height(contentHeight).
Render(content)
return lipgloss.JoinVertical(lipgloss.Left, header, contentArea, footer)
}
```
This architecture provides:
- **Clean Separation**: Each layer has clear responsibilities
- **Type Safety**: Compile-time navigation validation
- **State Persistence**: Complete session restoration
- **Responsive Design**: Adaptive to terminal capabilities
- **Resource Awareness**: Real-time estimation and optimization
- **User Experience**: Professional interaction patterns
backend/docs/installer/installer-base-screen.md
+462
View File
@@ -0,0 +1,462 @@
# BaseScreen Architecture Guide
> Practical guide for implementing new installer screens and migrating existing ones using the BaseScreen architecture.
## 🏗️ **Architecture Overview**
The `BaseScreen` provides a unified foundation for all installer form screens, encapsulating:
- **State Management**: `initialized`, `hasChanges`, `focusedIndex`, `showValues`
- **Form Handling**: `fields []FormField`, viewport management, auto-scrolling
- **Navigation**: Composite ScreenID support, GoBack patterns
- **Layout**: Responsive horizontal/vertical layouts
- **Lists**: Optional dropdown lists with delegates
### **Core Components**
```go
type BaseScreen struct {
// Dependencies (injected)
controller *controllers.StateController
styles *styles.Styles
window *window.Window
// State
args []string
initialized bool
hasChanges bool
focusedIndex int
showValues bool
// Form data
fields []FormField
fieldHeights []int
// UI components
viewport viewport.Model
formContent string
// Handlers (must be implemented)
handler BaseScreenHandler
listHandler BaseListHandler // optional
}
```
### **Required Interfaces**
```go
type BaseScreenHandler interface {
BuildForm()
GetFormTitle() string
GetHelpContent() string
HandleSave() error
HandleReset()
OnFieldChanged(fieldIndex int, oldValue, newValue string)
GetFormFields() []FormField
SetFormFields(fields []FormField)
}
type BaseListHandler interface { // Optional
GetList() *list.Model
OnListSelectionChanged(oldSelection, newSelection string)
GetListHeight() int
}
```
## 🚀 **Creating New Screens**
### **1. Basic Form Screen**
```go
// example_form.go
type ExampleFormModel struct {
*BaseScreen
config *controllers.ExampleConfig
}
func NewExampleFormModel(
controller *controllers.StateController,
styles *styles.Styles,
window *window.Window,
args []string,
) *ExampleFormModel {
m := &ExampleFormModel{
config: controller.GetExampleConfig(),
}
m.BaseScreen = NewBaseScreen(controller, styles, window, args, m, nil)
return m
}
// Required interface implementations
func (m *ExampleFormModel) BuildForm() {
fields := []FormField{}
// Text field
apiKeyInput := textinput.New()
apiKeyInput.Placeholder = "Enter API key"
apiKeyInput.EchoMode = textinput.EchoPassword
apiKeyInput.SetValue(m.config.APIKey)
fields = append(fields, FormField{
Key: "api_key",
Title: "API Key",
Description: "Your service API key",
Required: true,
Masked: true,
Input: apiKeyInput,
Value: apiKeyInput.Value(),
})
// Boolean field
enabledInput := textinput.New()
enabledInput.Placeholder = "true/false"
enabledInput.ShowSuggestions = true
enabledInput.SetSuggestions([]string{"true", "false"})
enabledInput.SetValue(fmt.Sprintf("%t", m.config.Enabled))
fields = append(fields, FormField{
Key: "enabled",
Title: "Enabled",
Description: "Enable or disable service",
Required: false,
Masked: false,
Input: enabledInput,
Value: enabledInput.Value(),
})
m.SetFormFields(fields)
}
func (m *ExampleFormModel) GetFormTitle() string {
return "Example Service Configuration"
}
func (m *ExampleFormModel) GetHelpContent() string {
return "Configure your Example service settings here."
}
func (m *ExampleFormModel) HandleSave() error {
fields := m.GetFormFields()
for _, field := range fields {
switch field.Key {
case "api_key":
m.config.APIKey = field.Input.Value()
case "enabled":
m.config.Enabled = field.Input.Value() == "true"
}
}
if m.config.APIKey == "" {
return fmt.Errorf("API key is required")
}
return m.GetController().UpdateExampleConfig(m.config)
}
func (m *ExampleFormModel) HandleReset() {
m.config = m.GetController().GetExampleConfig()
m.BuildForm()
}
func (m *ExampleFormModel) OnFieldChanged(fieldIndex int, oldValue, newValue string) {
// Additional validation logic if needed
}
func (m *ExampleFormModel) GetFormFields() []FormField {
return m.BaseScreen.fields
}
func (m *ExampleFormModel) SetFormFields(fields []FormField) {
m.BaseScreen.fields = fields
}
// Update method with field input handling
func (m *ExampleFormModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
switch msg := msg.(type) {
case tea.KeyMsg:
switch msg.String() {
case "tab":
// Handle tab completion for boolean fields
return m.handleTabCompletion()
default:
// Handle field input
if cmd := m.HandleFieldInput(msg); cmd != nil {
return m, cmd
}
}
}
// Base screen handling
return m.BaseScreen.Update(msg)
}
```
### **2. Screen with List Selection**
```go
// list_form.go
type ListFormModel struct {
*BaseScreen
config *controllers.ListConfig
selectionList list.Model
delegate *ExampleDelegate
}
func NewListFormModel(...) *ListFormModel {
m := &ListFormModel{
config: controller.GetListConfig(),
}
m.initializeList()
m.BaseScreen = NewBaseScreen(controller, styles, window, args, m, m) // Both handlers
return m
}
func (m *ListFormModel) initializeList() {
items := []list.Item{
ExampleOption("Option 1"),
ExampleOption("Option 2"),
}
m.delegate = &ExampleDelegate{
style: m.GetStyles().FormLabel,
width: MinMenuWidth - 6,
}
m.selectionList = list.New(items, m.delegate, MinMenuWidth-6, 3)
m.selectionList.SetShowStatusBar(false)
m.selectionList.SetFilteringEnabled(false)
m.selectionList.SetShowHelp(false)
m.selectionList.SetShowTitle(false)
}
// BaseListHandler implementation
func (m *ListFormModel) GetList() *list.Model {
return &m.selectionList
}
func (m *ListFormModel) OnListSelectionChanged(oldSelection, newSelection string) {
m.config.SelectedOption = newSelection
m.BuildForm() // Rebuild form based on selection
}
func (m *ListFormModel) GetListHeight() int {
return 5
}
func (m *ListFormModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
switch msg := msg.(type) {
case tea.KeyMsg:
// Handle list input first
if cmd := m.HandleListInput(msg); cmd != nil {
return m, cmd
}
// Then field input
if cmd := m.HandleFieldInput(msg); cmd != nil {
return m, cmd
}
}
return m.BaseScreen.Update(msg)
}
```
## 🔄 **Migrating Existing Screens**
### **Step 1: Analyze Current Screen**
From existing screen, identify:
- Form fields and their types
- List components (if any)
- Special keyboard handling
- Save/reset logic
### **Step 2: Refactor Structure**
```go
// Before:
type OldFormModel struct {
controller *controllers.StateController
styles *styles.Styles
window *window.Window
args []string
initialized bool
hasChanges bool
focusedIndex int
showValues bool
fields []FormField
viewport viewport.Model
formContent string
fieldHeights []int
// ... config-specific fields
}
// After:
type NewFormModel struct {
*BaseScreen // Embedded base screen
// Only config-specific fields
config *controllers.Config
list list.Model // If needed
}
```
### **Step 3: Update Constructor**
```go
// Before:
func NewOldFormModel(...) *OldFormModel {
return &OldFormModel{
controller: controller,
styles: styles,
// ... lots of boilerplate
}
}
// After:
func NewNewFormModel(...) *NewFormModel {
m := &NewFormModel{
config: controller.GetConfig(),
}
// Initialize list if needed
m.initializeList()
m.BaseScreen = NewBaseScreen(controller, styles, window, args, m, m)
return m
}
```
### **Step 4: Implement Required Interfaces**
Move existing methods to interface implementations:
```go
// Move: buildForm() → BuildForm()
// Move: save logic → HandleSave()
// Move: reset logic → HandleReset()
// Move: help content → GetHelpContent()
```
### **Step 5: Remove Redundant Methods**
Delete these methods from migrated screens:
- `getInputWidth()`, `getViewportSize()`, `updateViewport()`
- `focusNext()`, `focusPrev()`, `toggleShowValues()`
- `renderVerticalLayout()`, `renderHorizontalLayout()`
- `ensureFocusVisible()`
## 📋 **Environment Variable Integration**
Follow existing patterns for environment variable handling:
```go
func (m *FormModel) BuildForm() {
// Track initially set fields for cleanup
m.initiallySetFields = make(map[string]bool)
for _, fieldConfig := range m.fieldConfigs {
envVar, _ := m.GetController().GetVar(fieldConfig.EnvVarName)
m.initiallySetFields[fieldConfig.Key] = envVar.IsPresent()
field := m.createFieldFromEnvVar(fieldConfig, envVar)
fields = append(fields, field)
}
m.SetFormFields(fields)
}
func (m *FormModel) HandleSave() error {
// First pass: Remove cleared fields
for _, field := range m.GetFormFields() {
value := strings.TrimSpace(field.Input.Value())
if value == "" && m.initiallySetFields[field.Key] {
m.GetController().SetVar(field.EnvVarName, "")
}
}
// Second pass: Save non-empty values
for _, field := range m.GetFormFields() {
value := strings.TrimSpace(field.Input.Value())
if value != "" {
m.GetController().SetVar(field.EnvVarName, value)
}
}
return nil
}
```
## 🎯 **Navigation Integration**
### **Screen Registration**
Add new screen to navigation system:
```go
// In types.go
const (
ExampleFormScreen ScreenID = "example_form"
)
// In app.go createModelForScreen()
case ExampleFormScreen:
return NewExampleFormModel(a.controller, a.styles, a.window, args)
```
### **Navigation Usage**
```go
// Navigate to screen with parameters
return m, func() tea.Msg {
return NavigationMsg{
Target: CreateScreenID("example_form", "config_type"),
}
}
// Return to previous screen
return m, func() tea.Msg {
return NavigationMsg{GoBack: true}
}
```
## 🔧 **Interface Validation**
Add compile-time interface checks:
```go
// Ensure interfaces are implemented
var _ BaseScreenHandler = (*ExampleFormModel)(nil)
var _ BaseListHandler = (*ListFormModel)(nil)
```
## 📊 **Benefits Summary**
- **Code Reduction**: 50-60% less boilerplate per screen
- **Consistency**: Unified behavior across all forms
- **Maintainability**: Centralized bug fixes and improvements
- **Development Speed**: Faster new screen implementation
## 🎯 **Quick Reference**
### **Screen Types**
1. **Simple Form**: Inherit BaseScreen, implement BaseScreenHandler
2. **Form with List**: Inherit BaseScreen, implement both handlers
3. **Menu Screen**: Use existing patterns without BaseScreen
### **Required Methods**
- `BuildForm()` - Create form fields
- `HandleSave()` - Save configuration with validation
- `HandleReset()` - Reset to defaults
- `GetFormTitle()` - Screen title
- `GetHelpContent()` - Right panel content
### **Optional Methods**
- `OnFieldChanged()` - Real-time validation
- List handler methods (if using lists)
This architecture enables rapid development of new installer screens while maintaining consistency and reducing code duplication.
backend/docs/installer/installer-overview.md
+277
View File
@@ -0,0 +1,277 @@
# PentAGI Installer Overview
> Comprehensive guide to the PentAGI installer - a robust Terminal User Interface (TUI) for configuring and deploying PentAGI services.
## 🎯 **Project Overview**
The PentAGI installer provides a modern, interactive Terminal User Interface for configuring and deploying the PentAGI autonomous penetration testing platform. Built using the [Charm](https://charm.sh/) tech stack, it implements responsive design patterns optimized for terminal environments.
### **Core Purpose**
- **Configuration Management**: Interactive setup of LLM providers, monitoring, and security settings
- **Environment Setup**: Automated configuration of Docker services and environment variables
- **User Experience**: Professional TUI with intuitive navigation and real-time validation
- **Production Ready**: Robust error handling, state persistence, and graceful degradation
### **Build Command**
```bash
# From backend/ directory
go build -o ../build/installer ./cmd/installer/main.go
# Monitor debug output
tail -f log.json | jq '.'
```
## 🏗️ **Technology Stack**
### **Core Technologies**
- **TUI Framework**: BubbleTea (Model-View-Update pattern)
- **Styling**: Lipgloss (CSS-like styling for terminals)
- **Components**: Bubbles (viewport, textinput, etc.)
- **Markdown**: Glamour (markdown rendering)
- **Language**: Go 1.21+
### **Architecture Components**
- **Navigation**: Type-safe screen routing with parameter passing
- **State Management**: Persistent configuration with environment variable integration
- **Layout System**: Responsive design with breakpoint-based layouts
- **Form System**: Dynamic forms with validation and auto-completion
- **Controller Layer**: Business logic abstraction from UI components
## 🎯 **Key Features**
### **Responsive Design**
- **Adaptive Layout**: Automatically adjusts to terminal size
- **Breakpoint System**: Horizontal/vertical layouts based on terminal width
- **Content Hiding**: Graceful degradation when space is insufficient
- **Dynamic Sizing**: Form fields and panels resize automatically
### **Interactive Configuration**
- **LLM Providers**: Support for OpenAI, Anthropic, Gemini, Bedrock, DeepSeek, GLM, Kimi, Qwen, Ollama, Custom endpoints
- **Monitoring Setup**: Langfuse integration for LLM observability
- **Observability**: Complete monitoring stack with Grafana, VictoriaMetrics, Jaeger
- **Summarization**: Advanced context management for LLM interactions
### **Professional UX**
- **Auto-Scrolling Forms**: Fields automatically scroll into view when focused
- **Tab Completion**: Boolean fields offer `true`/`false` suggestions
- **Real-time Validation**: Immediate feedback with human-readable error messages
- **Resource Estimation**: Live calculation of token usage and memory requirements
- **State Persistence**: Navigation and form state preserved across sessions
## 🏗️ **Architecture Overview**
### **Directory Structure**
```
backend/cmd/installer/
├── main.go # Application entry point
├── wizard/
│ ├── app.go # Main application controller
│ ├── controller/ # Business logic layer
│ │ └── controller.go
│ ├── locale/ # Localization constants
│ │ └── locale.go
│ ├── logger/ # TUI-safe logging
│ │ └── logger.go
│ ├── models/ # Screen implementations
│ │ ├── welcome.go # Welcome screen
│ │ ├── eula.go # EULA acceptance
│ │ ├── main_menu.go # Main navigation
│ │ ├── llm_providers.go
│ │ ├── llm_provider_form.go
│ │ ├── summarizer.go
│ │ ├── summarizer_form.go
│ │ └── types.go # Shared types
│ ├── styles/ # Styling and layout
│ │ └── styles.go
│ └── window/ # Terminal size management
│ └── window.go
```
### **Component Responsibilities**
#### **App Layer** (`app.go`)
- Global navigation management
- Screen lifecycle (creation, initialization, cleanup)
- Unified header and footer rendering
- Window size distribution to models
- Global event handling (ESC, Ctrl+C, resize)
#### **Models Layer** (`models/`)
- Screen-specific logic and state
- User interaction handling
- Content rendering (content area only)
- Local state management
#### **Controller Layer** (`controller/`)
- Business logic abstraction
- Environment variable management
- Configuration persistence
- State validation
#### **Styles Layer** (`styles/`)
- Centralized styling and theming
- Dimension management (singleton pattern)
- Shared glamour renderer (prevents freezing)
- Responsive style calculations
#### **Window Layer** (`window/`)
- Terminal size management
- Content area size calculations
- Dimension change coordination
## 🎯 **Navigation System**
### **Composite ScreenID Architecture**
The installer implements a sophisticated navigation system using composite screen IDs:
```go
// Format: "screen§arg1§arg2§..."
type ScreenID string
// Examples:
"welcome" // Simple screen
"main_menu§llm_providers" // Menu with selection
"llm_provider_form§openai" // Form with provider type
"summarizer_form§general" // Form with configuration type
```
### **Navigation Features**
- **Parameter Preservation**: Arguments maintained across navigation
- **Stack Management**: Proper back navigation without loops
- **State Persistence**: Complete navigation state restoration
- **Universal ESC**: Always returns to welcome screen
- **Type Safety**: Compile-time validation of screen IDs
### **Navigation Flow Example**
```
1. Start: ["welcome"]
2. Continue: ["welcome", "main_menu"]
3. LLM Providers: ["welcome", "main_menu§llm_providers", "llm_providers"]
4. OpenAI Form: [..., "llm_provider_form§openai"]
5. GoBack: [..., "llm_providers§openai"]
6. ESC: ["welcome"]
```
## 🎯 **Form System Architecture**
### **Advanced Form Patterns**
- **Boolean Fields**: Tab completion with `true`/`false` suggestions
- **Integer Fields**: Range validation with human-readable formatting
- **Environment Integration**: Direct EnvVar integration with presence detection
- **Smart Cleanup**: Automatic removal of cleared environment variables
- **Resource Estimation**: Real-time calculation of token/memory usage
### **Dynamic Field Generation**
Forms adapt based on configuration type:
```go
// Type-specific field generation
switch m.configType {
case "general":
m.addBooleanField("use_qa", "Use QA Pairs", envVar)
m.addIntegerField("max_sections", "Max Sections", envVar, 1, 50)
case "assistant":
m.addIntegerField("keep_sections", "Keep Sections", envVar, 1, 10)
}
```
### **Viewport-Based Scrolling**
Forms automatically scroll to keep focused fields visible:
- **Auto-scroll**: Focused field automatically stays visible
- **Smart positioning**: Calculates field heights for precise scroll positioning
- **No extra hotkeys**: Uses existing navigation keys
## 🎯 **Configuration Management**
### **Supported Configurations**
#### **LLM Providers**
- **OpenAI**: GPT-4, GPT-3.5-turbo with API key configuration
- **Anthropic**: Claude-3, Claude-2 with API key configuration
- **Google Gemini**: Gemini Pro, Ultra with API key configuration
- **AWS Bedrock**: Multi-model support with AWS credentials
- **DeepSeek/GLM/Kimi/Qwen**: Base URL + API Key + Provider Name (optional, for LiteLLM)
- **Ollama**: Local model server integration
- **Custom**: OpenAI-compatible endpoint configuration
#### **Monitoring & Observability**
- **Langfuse**: LLM observability (embedded or external)
- **Observability Stack**: Grafana, VictoriaMetrics, Jaeger, Loki
- **Performance Monitoring**: System metrics and health checks
#### **Summarization Settings**
- **General**: Global conversation context management
- **Assistant**: Specialized settings for AI assistant contexts
- **Token Estimation**: Real-time calculation of context size
## 🎯 **Localization Architecture**
### **Centralized Constants**
All user-visible text stored in `locale/locale.go`:
```go
// Screen-specific constants
const (
WelcomeTitle = "PentAGI Installer"
WelcomeGreeting = "Welcome to PentAGI!"
// Form help text with practical guidance
LLMFormOpenAIHelp = `OpenAI provides access to GPT models...
Get your API key from:
https://platform.openai.com/api-keys`
)
```
### **Multi-line Help Text**
Detailed guidance integrated into forms:
- Provider-specific setup instructions
- Configuration recommendations
- Troubleshooting tips
- Best practices
## 🎯 **Error Handling & Recovery**
### **Graceful Degradation**
- **Dimension Fallbacks**: Handles invalid terminal sizes
- **Content Fallbacks**: Shows loading states and error messages
- **Network Resilience**: Offline operation support
- **State Recovery**: Automatic restoration from corrupted state
### **User-Friendly Error Messages**
- **Validation Errors**: Real-time feedback with clear guidance
- **System Errors**: Plain English explanations with suggested fixes
- **Network Errors**: Offline alternatives and retry mechanisms
## 🎯 **Performance Considerations**
### **Optimizations**
- **Lazy Loading**: Content loaded on-demand when screens accessed
- **Single Renderer**: Shared glamour instance prevents freezing
- **Efficient Scrolling**: Viewport-based rendering for large content
- **Memory Management**: Proper cleanup and resource sharing
### **Responsive Performance**
- **Breakpoint-Based**: Layout decisions based on terminal capabilities
- **Content Adaptation**: Hide non-essential content on small screens
- **Progressive Enhancement**: Full features on capable terminals
## 🎯 **Development Workflow**
### **File Organization**
- **One Model Per File**: Clear separation of screen logic
- **Shared Constants**: Type definitions in `types.go`
- **Centralized Locale**: All text in `locale.go`
- **Clean Dependencies**: Business logic isolated in controllers
### **Code Style**
- **Compact Syntax**: Where appropriate for readability
- **Expanded Logic**: For complex business rules
- **Comments**: Explain "why" and "how", not "what"
- **Error Handling**: Graceful degradation with user guidance
### **Testing Strategy**
- **Build Testing**: Successful compilation verification
- **Manual Testing**: Interactive validation on various terminal sizes
- **Dimension Testing**: Minimum (80x24) to large terminal support
- **Navigation Testing**: Complete flow validation
This overview provides the foundation for understanding the PentAGI installer's architecture, features, and development approach. The system prioritizes user experience, maintainability, and production reliability.
backend/docs/installer/installer-troubleshooting.md
+596
View File
@@ -0,0 +1,596 @@
# PentAGI Installer Troubleshooting Guide
> Comprehensive troubleshooting guide including recent fixes, performance optimization, and common issues.
## 🚨 **Development-Specific Issues**
### **TUI Application Constraints**
**Problem**: Running installer breaks terminal session during development
**Solution**: Build-only development workflow
```bash
# ✅ CORRECT: Build and test separately
cd backend/
go build -o ../build/installer ./cmd/installer/main.go
# Test in separate terminal session
cd ../build/
./installer
# ❌ WRONG: Running during development
cd backend/
go run ./cmd/installer/main.go # Breaks active terminal!
```
**Debug Monitoring**:
```bash
# Monitor debug output during development
tail -f log.json | jq '.'
# Filter by component
tail -f log.json | jq 'select(.component == "FormModel")'
# Pretty print timestamps
tail -f log.json | jq -r '"\(.timestamp) [\(.level)] \(.message)"'
```
## 🔧 **Recent Fixes & Improvements**
### ✅ **Composite ScreenID Navigation System**
**Problem**: Need to preserve selected menu items and provider selections across navigation
**Solution**: Implemented composite ScreenIDs with `§` separator for parameter passing
**Before** (❌ Problematic):
```go
// Lost selection on navigation
func (m *MenuModel) handleSelection() (tea.Model, tea.Cmd) {
return NavigationMsg{Target: LLMProvidersScreen} // No context preserved
}
```
**After** (✅ Fixed):
```go
// Preserves selection context
func (m *MenuModel) handleSelection() (tea.Model, tea.Cmd) {
selectedItem := m.getSelectedItem()
return NavigationMsg{
Target: CreateScreenID("llm_providers", selectedItem.ID),
}
}
// Results in: "llm_providers§openai" - selection preserved
```
**Benefits**:
- Type-safe parameter passing via `GetScreen()`, `GetArgs()`, `CreateScreenID()`
- Automatic state restoration - user returns to exact selection after ESC
- Clean navigation stack with full context preservation
### ✅ **Complete Localization Architecture**
**Problem**: Hardcoded strings scattered throughout UI components
**Solution**: Centralized all user-visible text in `locale.go` with structured constants
**Implementation**:
```go
// Multi-line text stored as single constants
const MainMenuLLMProvidersInfo = `Configure AI language model providers for PentAGI.
Supported providers:
• OpenAI (GPT-4, GPT-3.5-turbo)
• Anthropic (Claude-3, Claude-2)
...`
// Usage in components
sections = append(sections, m.styles.Paragraph.Render(locale.MainMenuLLMProvidersInfo))
```
**Coverage**: 100% of user-facing text moved to locale constants
- Menu descriptions and help text
- Form labels and error messages
- Provider-specific documentation
- Keyboard shortcuts and hints
### ✅ **Viewport-Based Form Scrolling**
**Problem**: Forms with many fields don't fit on smaller terminals
**Solution**: Implemented auto-scrolling viewport with focus tracking
**Key Features**:
- **Auto-scroll**: Focused field automatically stays visible
- **Smart positioning**: Calculates field heights for precise scroll positioning
- **Seamless navigation**: Tab/Shift+Tab scroll form as needed
- **No extra hotkeys**: Uses existing navigation keys
**Technical Implementation**:
```go
// Auto-scroll on field focus change
func (m *FormModel) ensureFocusVisible() {
focusY := m.calculateFieldPosition(m.focusedIndex)
if focusY < m.viewport.YOffset {
m.viewport.YOffset = focusY // Scroll up
}
if focusY >= m.viewport.YOffset + m.viewport.Height {
m.viewport.YOffset = focusY - m.viewport.Height + 1 // Scroll down
}
}
```
### ✅ **Enhanced Provider Configuration**
**Problem**: Missing configuration fields for several LLM providers
**Solution**: Added complete field sets for all supported providers
**Provider-Specific Field Sets:**
- **OpenAI/Anthropic/Gemini**: Base URL + API Key
- **AWS Bedrock**: Region + Default Auth OR Bearer Token OR (Access Key + Secret Key + Session Token) + Base URL
- **DeepSeek**: Base URL + API Key + Provider Name (for LiteLLM prefix, e.g., 'deepseek')
- **GLM**: Base URL + API Key + Provider Name (for LiteLLM prefix, e.g., 'zai')
- **Kimi**: Base URL + API Key + Provider Name (for LiteLLM prefix, e.g., 'moonshot')
- **Qwen**: Base URL + API Key + Provider Name (for LiteLLM prefix, e.g., 'dashscope')
- **Ollama**: Base URL + API Key (cloud only) + Model + Config Path + Pull/Load settings
- Local scenario: No API key needed
- Cloud scenario: API key required from https://ollama.com/settings/keys
- **Custom**: Base URL + API Key + Model + Config Path + Provider Name + Reasoning options
**Dynamic Form Generation**: Forms adapt based on provider type with appropriate validation and help text.
## 🔧 **Common Issues & Solutions**
### **Navigation Issues**
#### **Navigation Stack Corruption**
**Symptoms**: User gets stuck on screens, ESC doesn't work, back navigation fails
**Cause**: Circular navigation patterns or corrupted navigation stack
**Debug**:
```go
func (n *Navigator) debugStack() {
stackInfo := make([]string, len(n.stack))
for i, screenID := range n.stack {
stackInfo[i] = string(screenID)
}
logger.LogWithData("Navigation Stack", map[string]interface{}{
"stack": stackInfo,
"current": string(n.Current()),
"depth": len(n.stack),
})
}
```
**Solution**:
```go
// ✅ CORRECT: Use GoBack to prevent loops
func (m *FormModel) saveAndReturn() (tea.Model, tea.Cmd) {
if err := m.saveConfiguration(); err != nil {
return m, nil // Stay on form if save fails
}
return m, func() tea.Msg {
return NavigationMsg{GoBack: true} // Return to previous screen
}
}
// ❌ WRONG: Direct navigation creates loops
func (m *FormModel) saveAndReturn() (tea.Model, tea.Cmd) {
m.saveConfiguration()
return m, func() tea.Msg {
return NavigationMsg{Target: ProvidersScreen} // Creates navigation loop!
}
}
```
#### **Lost Selection State**
**Symptoms**: Menu selections reset, provider choices forgotten, configuration lost
**Cause**: Models not constructed with proper args
**Solution**:
```go
// ✅ CORRECT: Args-based construction
func NewModel(controller *StateController, styles *Styles,
window *Window, args []string) *Model {
selectedIndex := 0
if len(args) > 0 && args[0] != "" {
// Restore selection from navigation args
for i, item := range items {
if item.ID == args[0] {
selectedIndex = i
break
}
}
}
return &Model{selectedIndex: selectedIndex, args: args}
}
```
### **Form Issues**
#### **Form Field Width Problems**
**Symptoms**: Input fields too narrow/wide, don't adapt to terminal size
**Cause**: Fixed width assignments during field creation
**Debug**:
```go
func (m *FormModel) debugFormDimensions() {
width, height := m.styles.GetSize()
viewportWidth, viewportHeight := m.getViewportSize()
inputWidth := m.getInputWidth()
logger.LogWithData("Form Dimensions", map[string]interface{}{
"terminal_size": fmt.Sprintf("%dx%d", width, height),
"viewport_size": fmt.Sprintf("%dx%d", viewportWidth, viewportHeight),
"input_width": inputWidth,
"is_vertical": m.isVerticalLayout(),
"field_count": len(m.fields),
})
}
```
**Solution**:
```go
// ✅ CORRECT: Dynamic width calculation
func (m *FormModel) updateFormContent() {
inputWidth := m.getInputWidth()
for i, field := range m.fields {
// Apply width during rendering, not initialization
field.Input.Width = inputWidth - 3
field.Input.SetValue(field.Input.Value()) // Trigger width update
}
}
// ❌ WRONG: Fixed width at creation
func (m *FormModel) addField() {
input := textinput.New()
input.Width = 50 // Breaks responsive design!
}
```
#### **Form Scrolling Issues**
**Symptoms**: Can't reach all fields, focused field goes off-screen
**Cause**: Missing auto-scroll implementation or incorrect field height calculation
**Debug**:
```go
func (m *FormModel) debugScrollState() {
logger.LogWithData("Scroll State", map[string]interface{}{
"focused_index": m.focusedIndex,
"viewport_offset": m.viewport.YOffset,
"viewport_height": m.viewport.Height,
"content_height": lipgloss.Height(m.formContent),
"field_heights": m.fieldHeights,
"total_fields": len(m.fields),
})
}
```
**Solution**:
```go
// ✅ CORRECT: Auto-scroll implementation
func (m *FormModel) focusNext() {
m.fields[m.focusedIndex].Input.Blur()
m.focusedIndex = (m.focusedIndex + 1) % len(m.fields)
m.fields[m.focusedIndex].Input.Focus()
m.updateFormContent()
m.ensureFocusVisible() // Critical for auto-scroll
}
```
### **Environment Variable Issues**
#### **Configuration Not Persisting**
**Symptoms**: Settings lost between sessions, environment variables not saved
**Cause**: Not calling controller save methods or incorrect cleanup logic
**Debug**:
```go
func (m *FormModel) debugEnvVarState() {
for _, field := range m.fields {
envVar, _ := m.controller.GetVar(m.getEnvVarName(getEnvSuffixFromKey(field.Key)))
logger.LogWithData("Field State", map[string]interface{}{
"field_key": field.Key,
"input_value": field.Input.Value(),
"env_var_name": m.getEnvVarName(getEnvSuffixFromKey(field.Key)),
"env_var_value": envVar.Value,
"env_var_default": envVar.Default,
"is_present": envVar.IsPresent(),
"initially_set": m.initiallySetFields[field.Key],
})
}
}
```
**Solution**:
```go
// ✅ CORRECT: Proper save implementation
func (m *FormModel) saveConfiguration() error {
// First pass: Remove cleared fields
for _, field := range m.fields {
value := strings.TrimSpace(field.Input.Value())
if value == "" && m.initiallySetFields[field.Key] {
// Field was set but now empty - remove from environment
if err := m.controller.SetVar(field.EnvVarName, ""); err != nil {
return fmt.Errorf("failed to clear %s: %w", field.EnvVarName, err)
}
logger.Log("[FormModel] SAVE: cleared %s", field.EnvVarName)
}
}
// Second pass: Save non-empty values
for _, field := range m.fields {
value := strings.TrimSpace(field.Input.Value())
if value != "" {
if err := m.controller.SetVar(field.EnvVarName, value); err != nil {
return fmt.Errorf("failed to set %s: %w", field.EnvVarName, err)
}
logger.Log("[FormModel] SAVE: set %s=%s", field.EnvVarName, value)
}
}
return nil
}
```
### **Layout Issues**
#### **Content Not Adapting to Terminal Size**
**Symptoms**: Content cut off, panels don't resize, horizontal scrolling
**Cause**: Missing responsive layout logic or incorrect dimension handling
**Debug**:
```go
func (m *Model) debugLayoutState() {
width, height := m.styles.GetSize()
contentWidth, contentHeight := m.window.GetContentSize()
logger.LogWithData("Layout State", map[string]interface{}{
"terminal_size": fmt.Sprintf("%dx%d", width, height),
"content_size": fmt.Sprintf("%dx%d", contentWidth, contentHeight),
"is_vertical": m.isVerticalLayout(),
"min_terminal": MinTerminalWidth,
"min_menu_width": MinMenuWidth,
"min_info_width": MinInfoWidth,
})
}
```
**Solution**:
```go
// ✅ CORRECT: Responsive layout implementation
func (m *Model) View() string {
width, height := m.styles.GetSize()
leftPanel := m.renderContent()
rightPanel := m.renderInfo()
if m.isVerticalLayout() {
return m.renderVerticalLayout(leftPanel, rightPanel, width, height)
}
return m.renderHorizontalLayout(leftPanel, rightPanel, width, height)
}
func (m *Model) isVerticalLayout() bool {
contentWidth := m.window.GetContentWidth()
return contentWidth < (MinMenuWidth + MinInfoWidth + PaddingWidth)
}
```
#### **Footer Height Inconsistency**
**Symptoms**: Footer takes more/less space than expected, layout calculations wrong
**Cause**: Using border-based footer approach instead of background approach
**Solution**:
```go
// ✅ CORRECT: Background approach (always 1 line)
func (a *App) renderFooter() string {
actions := a.buildFooterActions()
footerText := strings.Join(actions, " • ")
return a.styles.Footer.Render(footerText)
}
// In styles.go
func (s *Styles) updateStyles() {
s.Footer = lipgloss.NewStyle().
Width(s.width).
Background(lipgloss.Color("240")).
Foreground(lipgloss.Color("255")).
Padding(0, 1, 0, 1)
}
// ❌ WRONG: Border approach (height varies)
footer := lipgloss.NewStyle().
Height(1).
Border(lipgloss.Border{Top: true}).
Render(text)
```
## 🔧 **Performance Issues**
### **Slow Rendering**
**Symptoms**: Laggy UI, delayed responses to keystrokes
**Cause**: Multiple glamour renderers, excessive content updates
**Debug**:
```go
func (m *Model) debugRenderPerformance() {
start := time.Now()
content := m.buildContent()
buildDuration := time.Since(start)
start = time.Now()
m.viewport.SetContent(content)
setContentDuration := time.Since(start)
start = time.Now()
view := m.viewport.View()
viewDuration := time.Since(start)
logger.LogWithData("Render Performance", map[string]interface{}{
"content_size": len(content),
"rendered_size": len(view),
"build_ms": buildDuration.Milliseconds(),
"set_content_ms": setContentDuration.Milliseconds(),
"view_render_ms": viewDuration.Milliseconds(),
})
}
```
**Solution**:
```go
// ✅ CORRECT: Single shared renderer
// In styles.go
func New() *Styles {
renderer, _ := glamour.NewTermRenderer(
glamour.WithAutoStyle(),
glamour.WithWordWrap(80),
)
return &Styles{renderer: renderer}
}
// Usage
rendered, err := m.styles.GetRenderer().Render(content)
// ❌ WRONG: Multiple renderers
func (m *Model) renderMarkdown(content string) string {
renderer, _ := glamour.NewTermRenderer(...) // Performance killer!
return renderer.Render(content)
}
```
### **Memory Leaks**
**Symptoms**: Increasing memory usage, application becomes sluggish over time
**Cause**: Not properly cleaning up resources, creating multiple renderer instances
**Solution**:
```go
// ✅ CORRECT: Complete state reset
func (m *Model) Init() tea.Cmd {
// Reset ALL state completely
m.content = ""
m.ready = false
m.error = nil
m.initialized = false
// Reset component state
m.viewport.GotoTop()
m.viewport.SetContent("")
// Reset form state
m.focusedIndex = 0
m.hasChanges = false
for i := range m.fields {
m.fields[i].Input.Blur()
}
return m.loadContent
}
```
## 🔧 **Error Recovery Patterns**
### **Graceful State Recovery**
```go
func (m *Model) recoverFromError(err error) tea.Cmd {
logger.Errorf("[%s] ERROR: %v", m.componentName, err)
// Try to recover state
m.error = err
m.ready = true
// Attempt graceful recovery
return func() tea.Msg {
logger.Log("[%s] RECOVERY: attempting state recovery", m.componentName)
// Try to reload content
if content, loadErr := m.loadFallbackContent(); loadErr == nil {
logger.Log("[%s] RECOVERY: fallback content loaded", m.componentName)
return ContentLoadedMsg{content}
}
logger.Log("[%s] RECOVERY: using minimal content", m.componentName)
return ContentLoadedMsg{"# Error\n\nContent temporarily unavailable."}
}
}
```
### **Safe Async Operations**
```go
func (m *Model) loadContent() tea.Cmd {
return func() tea.Msg {
defer func() {
if r := recover(); r != nil {
logger.Errorf("[%s] PANIC: recovered from panic: %v", m.componentName, r)
return ErrorMsg{fmt.Errorf("panic in loadContent: %v", r)}
}
}()
content, err := m.loadFromSource()
if err != nil {
return ErrorMsg{err}
}
return ContentLoadedMsg{content}
}
}
```
## 🔧 **Testing Strategies**
### **Manual Testing Checklist**
```go
// Test dimensions
// 1. Resize terminal to various sizes
// 2. Test minimum dimensions (80x24)
// 3. Test very narrow terminals (< 80 cols)
// 4. Test very short terminals (< 24 rows)
func testDimensions() {
testSizes := []struct{ width, height int }{
{80, 24}, // Standard
{40, 12}, // Small
{120, 40}, // Large
{20, 10}, // Tiny
}
for _, size := range testSizes {
logger.LogWithData("Dimension Test", map[string]interface{}{
"test_size": fmt.Sprintf("%dx%d", size.width, size.height),
"layout_mode": getLayoutMode(size.width, size.height),
})
}
}
```
### **Navigation Flow Testing**
```go
func testNavigationFlow() {
testSteps := []struct {
action string
expected string
}{
{"start", "welcome"},
{"continue", "main_menu"},
{"select_providers", "llm_providers"},
{"select_openai", "llm_provider_form§openai"},
{"go_back", "llm_providers§openai"},
{"esc", "welcome"},
}
for _, step := range testSteps {
logger.LogWithData("Navigation Test", map[string]interface{}{
"action": step.action,
"expected": step.expected,
"actual": string(navigator.Current()),
})
}
}
```
This troubleshooting guide provides comprehensive solutions for:
- **Development Workflow**: TUI-safe development patterns
- **Navigation Issues**: Stack management and state preservation
- **Form Problems**: Responsive design and scrolling
- **Configuration**: Environment variable management
- **Performance**: Optimization and resource management
- **Recovery**: Graceful error handling and state restoration
backend/docs/installer/processor-implementation.md
+119
View File
@@ -0,0 +1,119 @@
# Processor Implementation Summary
## Overview
Processor package implements the operational engine for PentAGI installer operations per [processor.md](processor.md). Core lifecycle flows, file integrity logic, Docker/Compose orchestration, and Bubble Tea integration are implemented. Installer self-update flows are stubbed and intentionally not finalized yet.
## Implementation Notes
### Architecture Decisions
- **Interface-based design**: Internal interfaces per operation type (`fileSystemOperations`, `dockerOperations`, `composeOperations`, `updateOperations`) enable separation of concerns and testability
- **Two-track execution**: Docker API SDK for worker environment; Compose stacks via console commands with live output streaming
- **OperationOption pattern**: Functional options applied to an internal `operationState` support force mode and embedded terminal integration via `WithForce` and `WithTerminal`
- **State machine logic**: `ApplyChanges` implements three-phase stack management (Observability → Langfuse → PentAGI) with integrity validation; the wizard performs a pre-phase interactive integrity check with Y/N decision for force mode
- **Single-responsibility operations**: business logic delegates to compose layer; strict purge of images is implemented as `purgeImagesStack` alongside other compose operations
### Key Features Implemented
1. **File System Operations** (`fs.go`):
- Ensure/verify stack file integrity with force mode support
- Handle embedded directory trees (observability) and compose files
- YAML validation and automatic file recovery
- Support deployment modes (embedded/external/disabled) for applicable stacks
- Excluded files policy for integrity verification: `observability/otel/config.yml`, `observability/grafana/config/grafana.ini`, `example.custom.provider.yml`, `example.ollama.provider.yml` (presence ensured, content changes tolerated)
2. **Docker Operations** (`docker.go`):
- Worker and default image management with progress reporting
- Worker container lifecycle management (removal, purging)
- Support for custom Docker configuration via environment variables
3. **Compose Operations** (`compose.go`):
- Stack lifecycle management with dependency ordering
- Rolling updates with health checks
- Live output streaming to TUI callbacks
- Environment variable injection for compose commands
- `purgeStack` (down -v) and `purgeImagesStack` (down --rmi all -v) placed together for clarity
4. **Update Operations** (`update.go`):
- Update server communication and binary replacement helpers are scaffolded (checksum, atomic replace/backup)
- Installer update/download/remove operations are currently stubs and return "not implemented"; network calls use placeholder logic for now
5. **Remove/Purge Operations**:
- Soft removal (preserve data) vs purge (complete cleanup); strict image purge via compose in `purgeImagesStack`
- Proper cleanup ordering and external/existing deployment handling
### Critical Implementation Details
- **Three-phase execution**: Observability → Langfuse → PentAGI with state validation after each phase
- **Force mode behavior**: Aggressive file overwriting and state correction when explicitly requested
- **File integrity logic**: `ensureStackIntegrity` for missing files, `verifyStackIntegrity` for existing files; modified files are explicitly skipped and logged when `force=false`; excluded files are ensured to exist but not overwritten when modified
- **State consistency**: `Gather*Info` calls after each phase validate operation success
- **Error isolation**: Phase failures don't affect other stacks, partial state preserved
- **Compose environment tweaks**: `COMPOSE_IGNORE_ORPHANS=1`, `PYTHONUNBUFFERED=1`; ANSI disabled on narrow terminals via `COMPOSE_ANSI=never`
### Testing Strategy
Comprehensive tests include:
- Mock implementations for external dependencies (state, checker, files)
- Unit tests for file system integrity operations (ensure/verify/cleanup, excluded files policy, YAML validation)
- Validation tests for operation applicability
- Factory reset, lifecycle, and ordering behavior at logic level
### Integration Points
- **State management**: Integrates with `state.State` for configuration and environment variables
- **System assessment**: Uses `checker.CheckResult` for current system state analysis
- **File handling**: Integrates with `files.Files` for embedded content extraction
- **TUI integration**: Bubble Tea integration via `ProcessorModel` with message polling; wizard performs pre-phase integrity scan (Enter → scan; Y/N → overwrite decision; Ctrl+C → cancel integrity stage)
## Files Created/Modified
### Core Implementation
- `processor.go` - Processor interface, options, and synchronous operations entry points
- `model.go` - Bubble Tea `ProcessorModel` with `HandleMsg` polling
- `logic.go` - Business logic (ApplyChanges, lifecycle operations, factory reset)
- `fs.go` - File system operations and integrity verification
- `docker.go` - Docker API/CLI operations and worker image/volumes management
- `compose.go` - Docker Compose stack lifecycle management (including `purgeImagesStack`)
- `update.go` - Scaffolding for update mechanisms (stubs for installer update flows)
### Testing
- `mock_test.go` - Mocks for interfaces with call tracking
- `logic_test.go` - Business logic tests (state machine and sequencing)
- `fs_test.go` - File system operations tests (including excluded files policy)
## Status
✅ **MOSTLY COMPLETE** - Core processor functionality implemented and tested
- Lifecycle, file integrity, Docker/Compose orchestration are production-ready
- Bubble Tea integration via `ProcessorModel` is complete
- Installer self-update flows (download/update/remove) are stubbed and not enabled yet
- All current tests pass; additional tests will be added once update flows are finalized
## Current Architecture
### ProcessorModel Integration
The processor integrates with Bubble Tea through `ProcessorModel` that wraps operations as `tea.Cmd` and provides a polling handler:
```go
// ProcessorModel provides tea.Cmd wrappers for all operations and a polling handler
type ProcessorModel interface {
ApplyChanges(ctx context.Context, opts ...OperationOption) tea.Cmd
CheckFiles(ctx context.Context, stack ProductStack, opts ...OperationOption) tea.Cmd
FactoryReset(ctx context.Context, opts ...OperationOption) tea.Cmd
Install(ctx context.Context, opts ...OperationOption) tea.Cmd
Update(ctx context.Context, stack ProductStack, opts ...OperationOption) tea.Cmd
Download(ctx context.Context, stack ProductStack, opts ...OperationOption) tea.Cmd
Remove(ctx context.Context, stack ProductStack, opts ...OperationOption) tea.Cmd
Purge(ctx context.Context, stack ProductStack, opts ...OperationOption) tea.Cmd
Start(ctx context.Context, stack ProductStack, opts ...OperationOption) tea.Cmd
Stop(ctx context.Context, stack ProductStack, opts ...OperationOption) tea.Cmd
Restart(ctx context.Context, stack ProductStack, opts ...OperationOption) tea.Cmd
HandleMsg(msg tea.Msg) tea.Cmd
}
```
### Message Types
Processor operations communicate via messages:
- `ProcessorStartedMsg` - operation started
- `ProcessorOutputMsg` - command output (partial or full)
- `ProcessorFilesCheckMsg` - file statuses computed during `CheckFiles`
- `ProcessorCompletionMsg` - operation completed
- `ProcessorWaitMsg` - polling tick
### Terminal Integration
Operations support real-time terminal output through `WithTerminal(term terminal.Terminal)`; ANSI is auto-disabled for narrow terminals. Compose commands inherit current env plus `COMPOSE_IGNORE_ORPHANS=1` and `PYTHONUNBUFFERED=1`.
backend/docs/installer/processor-logic-implementation.md
+241
View File
@@ -0,0 +1,241 @@
# Processor Business Logic Implementation
## Core Concepts
The processor package manages PentAGI stack lifecycle through state-oriented approach. Main idea: maintain consistency between desired user state (state.State) and actual system state (checker.CheckResult).
## Key Principles
1. **State-Driven Operations**: All operations based on comparing current and target state
2. **Stack Independence**: Each stack (observability, langfuse, pentagi) managed independently
3. **Force Mode**: Aggressive state correction ignoring warnings
4. **Idempotency**: Repeated operation calls do not cause side effects
5. **User-Facing Automation**: Installer automates manual Docker/file operations with real-time feedback
## Stack Architecture
### ProductStack Hierarchy
```
ProductStackAll
├── ProductStackObservability (optional, embedded/external/disabled)
├── ProductStackLangfuse (optional, embedded/external/disabled)
└── ProductStackPentagi (mandatory, always embedded)
```
### Deployment Modes
- **Embedded**: Full local stack with Docker Compose files
- **External**: Using external service (configuration only)
- **Disabled**: Functionality turned off
## ApplyChanges Algorithm
### Purpose
Bring system to state matching user configuration. Main installation/update/configuration function.
### Prerequisites
- docker accessibility is validated via `checker.CheckResult` flags
- required compose networks are ensured via `ensureMainDockerNetworks` before stack operations
### Implementation Strategy
#### Pre-phase: interactive file integrity check (wizard)
- before starting ApplyChanges, the wizard runs an integrity scan using processor file checking helpers
- if outdated or missing files are detected, the user is prompted to choose:
- proceed with updates (force=true) – modified files will be overwritten from embedded content;
- proceed without updates (force=false) – installer will try to apply changes without touching modified files.
- hotkeys on the screen: Enter (start integrity scan), then Y/N to choose scenario; Ctrl+C cancels the integrity stage and returns to the initial prompt.
#### Phase 1: Observability Stack Management
```go
if p.isEmbeddedDeployment(ProductStackObservability) {
// user wants embedded observability
if !p.checker.ObservabilityExtracted {
// extract files (docker-compose + observability directory)
p.fsOps.ensureStackIntegrity(ctx, ProductStackObservability, state)
} else {
// verify file integrity, update if force=true
p.fsOps.verifyStackIntegrity(ctx, ProductStackObservability, state)
}
// update/start containers
p.composeOps.updateStack(ctx, ProductStackObservability, state)
} else {
// user wants external/disabled observability
if p.checker.ObservabilityInstalled {
// remove containers but keep files (user might re-enable)
p.composeOps.removeStack(ctx, ProductStackObservability, state)
}
}
// refresh state to verify operation success
p.checker.GatherObservabilityInfo(ctx)
```
**Rationale**: Observability processed first as most complex stack (directory + compose file). Force mode used for file conflict resolution. For external/disabled modes containers are removed but files are preserved.
#### Phase 2: Langfuse Stack Management
```go
if p.isEmbeddedDeployment(ProductStackLangfuse) {
// only docker-compose-langfuse.yml file
p.fsOps.ensureStackIntegrity(ctx, ProductStackLangfuse, state)
p.composeOps.updateStack(ctx, ProductStackLangfuse, state)
} else {
if p.checker.LangfuseInstalled {
p.composeOps.removeStack(ctx, ProductStackLangfuse, state)
}
}
p.checker.GatherLangfuseInfo(ctx)
```
**Rationale**: Langfuse simpler than observability (single file only), but follows same logic. As a precondition for local start, configuration must be connected (see checker `LangfuseConnected`).
#### Phase 3: PentAGI Stack Management
```go
// PentAGI always embedded, always required
p.fsOps.ensureStackIntegrity(ctx, ProductStackPentagi, state)
p.composeOps.updateStack(ctx, ProductStackPentagi, state)
p.checker.GatherPentagiInfo(ctx)
```
**Rationale**: PentAGI - main stack, always installed, only file integrity check.
### Critical Implementation Details
#### File System Integrity
- **ensureStackIntegrity**: creates missing files from embed, overwrites with force=true
- **verifyStackIntegrity**: checks existence, updates with force=true
- **Embedded Provider**: uses `files.Files` for embedded content access
- correctness of directory checks: when modified files are detected and force=false, we log skip explicitly and keep files intact; the final directory log reflects whether modified files were present
- excluded files policy: `observability/otel/config.yml`, `observability/grafana/config/grafana.ini`, `example.custom.provider.yml`, `example.ollama.provider.yml` are ensured to exist but not overwritten if modified
#### Container State Management
- `updateStack`: executes `docker compose up -d` for rolling update
- `removeStack`: executes `docker compose down` without removing volumes
- `purgeStack`: executes `docker compose down -v`
- `purgeImagesStack`: executes `docker compose down --rmi all -v`
- dependency ordering: observability → langfuse → pentagi
- environment: `COMPOSE_IGNORE_ORPHANS=1`, `PYTHONUNBUFFERED=1`; ANSI disabled on narrow terminals via `COMPOSE_ANSI=never`
#### State Consistency
- After each phase corresponding `Gather*Info()` method called
- CheckResult updated for next decisions
- On errors state remains partially updated (no rollback)
- optimization with `state.IsDirty()`: optional early exit can be used by the UI to avoid unnecessary work; installer remains consistent without it.
## Force Mode Behavior
### Normal Mode (force=false)
- Does not overwrite existing files
- Stops on filesystem conflicts
- Conservative approach, minimal changes
### Force Mode (force=true)
- Overwrites any files without warnings
- Ignores validation errors
- Maximum effort to reach target state
- Used on explicit user request
### Disabled branches and YAML validation
- when a stack is configured as disabled, compose operations are skipped and file system changes are not required (except prior installation remains preserved);
- YAML validation is performed on compose files during integrity ensuring to fail fast in case of syntax errors.
## Error Handling Strategy
### Fail-Fast Principle
- Each phase can interrupt execution
- Partial state preserved (no rollback)
- Errors bubbled up with context
### Recovery Scenarios
- User can repeat operation with force=true
- Partial installation can be completed
- Remove/Purge operations for complete cleanup
## Operation Algorithms
### Update Operation
- checks `checker.*IsUpToDate` flags for compose stacks
- compose stacks: download then `docker compose up -d`, gather info
- worker: pull images only, gather info
- installer: stubbed (download/update/remove return not implemented), checksum and replace helpers exist
- refreshes updates info at the end of successful flows
### FactoryReset Operation
Correct cleanup sequence ensuring no dangling resources:
1. `purgeStack(all)` - removes all compose containers, networks, volumes
2. Remove worker containers/volumes (Docker API managed)
3. Remove residual networks (fallback cleanup)
4. Restore default .env from embedded
5. Restore all stack files with force=true
6. Refresh checker state
### Install Operation
- ensures docker networks exist before any stack operations
- checks `*Installed` flags to skip already installed components
- follows the same three-phase approach as ApplyChanges
- designed for fresh system setup
### Remove vs Purge
- **Remove**: Soft operation preserving user data (`docker compose down`)
- **Purge**: Complete cleanup including volumes (`docker compose down -v`)
- Files preserved in both cases for potential re-enablement
## Code Organization
### Clean Architecture
- Each operation delegates to specialized handlers (compose/docker/fs/update)
- No duplicate logic - single responsibility principle
- Force mode propagated through operationState
- Global mutex prevents concurrent modifications
## Integration Points
### State Management
- `state.State.IsDirty()` determines need for operations
- `state.State.Commit()` commits changes
- Environment variables control deployment modes
### Checker Integration
- `checker.CheckResult` contains current state
- Gather methods update state after operations
- Boolean flags optimize decision logic
### Files Integration
- `files.Files` provides embedded content access
- Fallback to filesystem when embedded missing
- Copy operations with rewrite flag for force mode
## Testing Strategy
### Unit Tests Focus
- State transition logic (current → target)
- Force mode behavior verification
- Error handling and partial state recovery
- Integration between components
### Mock Requirements
- files.Files for embedded content control
- checker.CheckResult with mockCheckHandler for state simulation
- baseMockFileSystemOperations, baseMockDockerOperations, baseMockComposeOperations with call tracking
- mockCheckHandler configured via mockCheckConfig for various scenarios
### Test Implementation
- Base mocks provide call logging and positive scenarios
- Error injection through setError() method on base mocks
- CheckResult states controlled via mockCheckHandler configuration
- Helper functions: testState(), testOperationState(), assertNoError(), assertError()
## Performance Characteristics
### Time Complexity
- O(1) for each stack (processed in parallel)
- File operations: O(n) where n = number of files in stack
- Container operations: depend on Docker/network latency
### Memory Usage
- Minimal: only state metadata
- Files read streaming without full memory loading
- CheckResult caches results until explicit refresh
### Network Impact
- Docker image pulls only when necessary
- Container updates use efficient rolling strategy
- External service connections minimal (checks only)
backend/docs/installer/processor-wizard-integration.md
+332
View File
@@ -0,0 +1,332 @@
# Processor-Wizard Integration Guide
> Technical documentation for embedded terminal integration in PentAGI installer wizard using Bubble Tea and pseudoterminals.
## Architecture Decision: Pseudoterminal vs Built-in Bubble Tea
After extensive research of `tea.ExecProcess` and alternative approaches, **pseudoterminal solution was chosen** for the following reasons:
**Why Not `tea.ExecProcess`:**
- ❌ Fullscreen takeover - cannot be embedded in viewport regions
- ❌ Blocking execution - pauses entire Bubble Tea program
- ❌ No real-time output streaming to specific UI components
- ❌ Cannot handle interactive Docker commands within constrained areas
**Built-in Approach Limitations:**
- Limited to `os/exec` with pipes (no true terminal semantics)
- Manual ANSI escape sequence handling required
- Reduced interactivity (no Ctrl+C, terminal properties)
- Complex input/output coordination
**Pseudoterminal Advantages:**
- ✅ Embedded in specific viewport regions
- ✅ Real-time command output with ANSI colors/formatting
- ✅ Full interactivity (stdin/stdout/stderr, Ctrl+C)
- ✅ Professional terminal experience within TUI
- ✅ Docker compatibility (`docker exec -it`, progress bars)
## Core Architecture
### Key Components
**`Processor`** interface (defined in [`processor.go`](../../cmd/installer/processor/processor.go)):
```go
type Processor interface {
ApplyChanges(ctx context.Context, opts ...OperationOption) error
CheckFiles(ctx context.Context, stack ProductStack, opts ...OperationOption) (FilesCheckResult, error)
FactoryReset(ctx context.Context, opts ...OperationOption) error
Install(ctx context.Context, opts ...OperationOption) error
Update(ctx context.Context, stack ProductStack, opts ...OperationOption) error
Download(ctx context.Context, stack ProductStack, opts ...OperationOption) error
Remove(ctx context.Context, stack ProductStack, opts ...OperationOption) error
Purge(ctx context.Context, stack ProductStack, opts ...OperationOption) error
Start(ctx context.Context, stack ProductStack, opts ...OperationOption) error
Stop(ctx context.Context, stack ProductStack, opts ...OperationOption) error
Restart(ctx context.Context, stack ProductStack, opts ...OperationOption) error
ResetPassword(ctx context.Context, stack ProductStack, opts ...OperationOption) error
}
```
**Bubble Tea integration** in [`model.go`](../../cmd/installer/processor/model.go), [`state.go`](../../cmd/installer/processor/state.go), and [`logic.go`](../../cmd/installer/processor/logic.go):
- Wraps processor operations as `tea.Cmd` values through `ProcessorModel`
- Streams operation updates through `ProcessorStartedMsg`, `ProcessorOutputMsg`, `ProcessorFilesCheckMsg`, and `ProcessorCompletionMsg`
- Executes commands through the shared `terminal.Terminal` abstraction when available
- Falls back to non-ANSI compose output on small terminals and Windows by setting `COMPOSE_ANSI=never`
### Integration Pattern
**Wizard Screen Integration** (see [`apply_changes.go`](../../cmd/installer/wizard/models/apply_changes.go)):
```go
type ApplyChangesFormModel struct {
processor processor.ProcessorModel
running bool
terminal terminal.Terminal
}
// Create terminal model for the screen
m.terminal = terminal.NewTerminal(width, height, terminal.WithAutoScroll(), terminal.WithAutoPoll())
// Start operation through the processor model
return m.processor.ApplyChanges(context.Background(), processor.WithTerminal(m.terminal))
```
## Message Flow Architecture
The integration uses a **buffered channel-based approach** for real-time terminal output streaming:
```mermaid
graph TD
A[User Action - Enter] --> B[ProcessorModel Command]
B --> C[Operation State]
C --> D[Terminal Abstraction]
D --> E[Execute Command - docker compose]
E --> F[Terminal Update Loop]
F --> G[Processor Messages]
G --> H[HandleMsg Polling]
H --> I[ProcessorOutputMsg]
I --> J[appendOutput + Viewport Update]
J --> K[UI Refresh - Real-time Display]
L[User Input] --> M[handleTerminalInput]
M --> N{Key Type?}
N -->|Scroll Keys| O[Viewport Handling]
N -->|Other Keys| P[keyToTerminalSequence]
P --> Q[Direct PTY Write]
subgraph "Real-time Flow"
F
G
H
I
J
end
subgraph "Input Handling"
M
N
O
P
Q
end
subgraph "Terminal Display"
C
R[Blue Border Only]
S[Maximized Content]
T[Auto-scroll Bottom]
end
```
## Apply Changes integrity pre-check (Wizard)
Before invoking `processor.ApplyChanges()`, the Apply Changes screen performs an embedded files integrity scan:
- Enter: start async scan using `GetStackFilesStatus(files, ProductStackAll, workingDir)`
- If outdated/missing files found: prompt user to update (Y) or proceed without updates (N)
- Ctrl+C: cancel the integrity stage and return to initial instruction screen
Hotkeys on this screen:
- Initial: Enter
- During scan: Ctrl+C
- When prompt is shown: Y/N, Ctrl+C
Depending on choice, `processor.ApplyChanges()` is called with/without `WithForce()`. This keeps user in control of overwriting modified files while still allowing a smooth path when no updates are required.
Note: the integrity prompt lists only modified files; missing files are considered normal on a fresh installation and are not shown to the user.
**Key Improvements:**
- **50ms polling** via `waitForOutput()` for responsive UI updates
- **Buffered channel** (100 messages) prevents blocking
- **Direct key mapping** - no intermediate input buffers
- **Viewport delegation** for scrolling (PageUp/PageDown, mouse wheel)
## Implementation Guide
### 1. Screen Model Setup
Add terminal integration to any wizard screen following this pattern:
```go
type YourFormModel struct {
*BaseScreen
processor processor.ProcessorModel
terminal terminal.Terminal
}
func (m *YourFormModel) BuildForm() tea.Cmd {
contentWidth, contentHeight := m.getViewportFormSize()
if m.terminal == nil {
m.terminal = terminal.NewTerminal(
contentWidth-2,
contentHeight-1,
terminal.WithAutoScroll(),
terminal.WithAutoPoll(),
terminal.WithCurrentEnv(),
)
}
return m.terminal.Init()
}
```
### 2. Update Method Integration
Handle terminal model updates with proper type assertion and input delegation:
```go
func (m *YourFormModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
var cmds []tea.Cmd
// Update terminal model first (handles real-time output)
if m.terminal != nil {
updatedModel, terminalCmd := m.terminal.Update(msg)
if terminalModel := terminal.RestoreModel(updatedModel); terminalModel != nil {
m.terminal = terminalModel
}
if terminalCmd != nil {
cmds = append(cmds, terminalCmd)
}
}
switch msg := msg.(type) {
case tea.WindowSizeMsg:
// Update terminal size dynamically
if m.terminal != nil {
contentWidth, contentHeight := m.getViewportFormSize()
m.terminal.SetSize(contentWidth-2, contentHeight-1)
}
case tea.KeyMsg:
// Terminal takes priority when operation is running
if m.terminal != nil && m.terminal.IsRunning() {
return m, tea.Batch(cmds...) // Terminal already processed input
}
// Your screen-specific hotkeys here...
}
return m, tea.Batch(cmds...)
}
```
### 3. Operation Triggers
Start processor operations through terminal model:
```go
// In your action handler
func (m *YourFormModel) startProcess() tea.Cmd {
return m.processor.Install(
context.Background(),
processor.WithTerminal(m.terminal),
)
}
```
### 4. Display Integration
Render terminal within your screen layout - simplified approach:
```go
func (m *YourFormModel) renderMainPanel() string {
if m.terminal != nil {
// Terminal model returns complete styled content with blue border
return m.terminal.View()
}
return m.GetStyles().Error.Render(locale.ApplyChangesTerminalIsNotInitialized)
}
```
**Terminal View Structure:**
- **No header/footer** - maximized content space
- **Blue border only** - clean visual boundaries
- **Auto-scrolling viewport** - always shows latest output
- **Responsive sizing** - adapts to window changes via `SetSize()`
## Features & Capabilities
### Interactive Terminal
- **Real-time output**: 50ms polling via buffered channel (100 messages)
- **Native input handling**: Direct key-to-terminal-sequence conversion
- **ANSI support**: Colors, formatting, progress bars rendered correctly
- **Smart scrolling**: PageUp/PageDown and mouse wheel for viewport, all other keys to PTY
### Docker Integration
- **Interactive commands**: `docker exec -it container bash`
- **Progress visualization**: Docker pull progress bars with colors
- **Color output**: Docker's colored status messages preserved
- **Signal handling**: Ctrl+C, Ctrl+D, Ctrl+Z properly handled
### UI Features
- **Maximized space**: No headers, input lines, or inner borders
- **Blue border styling**: Clean visual boundaries with `lipgloss.Color("62")`
- **Dynamic resizing**: `SetSize()` method for window changes
- **Toggle mode**: Ctrl+T switches between embedded terminal and message logs
- **Auto-scroll**: Always shows latest output, bottom-aligned
## Command Configuration
Processor operations support flexible configuration via [`processor.go`](../../cmd/installer/processor/processor.go):
```go
// Available options
processor.WithForce() // Skip validation checks
processor.WithTerminal(terminal) // Terminal-backed integration
processor.WithPasswordValue(password) // Reset password operation input
```
Choose integration method based on screen requirements:
- **Embedded terminal**: Interactive operations, real-time output, full PTY support
- **Processor model**: Bubble Tea command wrappers and message polling for wizard screens (see [`model.go`](../../cmd/installer/processor/model.go))
**Alternative Integration:** For simpler use cases or Windows compatibility, the current processor model uses `ProcessorOutputMsg` events from [`state.go`](../../cmd/installer/processor/state.go) and disables Docker Compose ANSI output when needed in [`logic.go`](../../cmd/installer/processor/logic.go).
## Limitations & Considerations
### Performance
- **Memory usage**: Output buffer auto-managed, channel limited to 100 messages
- **Goroutine management**: Automatic cleanup via `defer close(outputChan)`
- **Resource overhead**: Minimal - one goroutine per operation, 1ms throttling
- **Update frequency**: 50ms polling prevents UI blocking
### Platform Compatibility
- **Unix/Linux/macOS**: Full pseudoterminal support via `github.com/creack/pty v1.1.21`
- **Windows**: Compose ANSI output is disabled in the command runner when the terminal is small or the host is Windows
### UI Constraints
- **Minimum size**: `width-2, height-2` for border space
- **Input delegation**: Terminal captures all input except scroll keys when running
- **Layout integration**: Single terminal per screen, full area utilization
## Testing & Debugging
### Processor Model Testing
Processor model behavior can be tested independently of wizard integration (see [`logic_test.go`](../../cmd/installer/processor/logic_test.go) and [`mock_test.go`](../../cmd/installer/processor/mock_test.go) for current mock patterns).
### Debug Features
- **Ctrl+T toggle**: Switch to message-based mode for debugging
- **Output logging**: All terminal output available for inspection
- **Error reporting**: Terminal errors bubble up to UI state
## Future Enhancements
### Potential Improvements
- **Session recording**: Capture terminal sessions for replay/debugging
- **Multiple terminals**: Support for concurrent operations with tabs
- **Terminal themes**: Customizable color schemes via lipgloss
- **Buffer persistence**: Save terminal output between screen switches
- **Copy/paste**: Terminal text selection and clipboard integration
### Integration Opportunities
- **Service management**: Real-time `docker ps` monitoring in terminal
- **Log streaming**: Live log viewing with `docker logs -f`
- **Interactive debugging**: Terminal-based troubleshooting tools
- **Configuration editing**: Embedded editors for compose files
### Architecture Extensions
- **Message broadcasting**: Share terminal output across multiple UI components
- **Operation queuing**: Sequential command execution with progress tracking
- **Error recovery**: Automatic retry mechanisms with user confirmation
This clean, channel-based architecture provides a maintainable foundation for embedding professional terminal functionality within Bubble Tea applications while maximizing screen real estate and user experience.
backend/docs/installer/processor.md
+426
View File
@@ -0,0 +1,426 @@
# Processor Package Architecture
## Overview
The `processor` package serves as the core orchestrator for PentAGI installer operations, managing system interactions, Docker environments, and file system operations across different product stacks. It acts as the operational engine that executes user configuration changes determined through the TUI wizard interface.
## Installer Integration Architecture
```mermaid
graph TB
subgraph "Installer Application"
Main[main.go<br/>Entry Point]
State[state package<br/>Configuration Management]
Checker[checker package<br/>System Assessment]
Files[files package<br/>Embedded Content]
Wizard[wizard package<br/>TUI Interface]
Processor[processor package<br/>Operation Engine]
end
subgraph "External Systems"
Docker[Docker Engine<br/>Container Management]
FS[File System<br/>Host OS]
UpdateServer[Update Server<br/>pentagi.com]
Compose[Docker Compose<br/>Stack Orchestration]
end
Main --> State
Main --> Checker
Main --> Wizard
State --> Wizard
Checker --> Wizard
Wizard --> Processor
Processor --> State
Processor --> Checker
Processor --> Files
Processor --> Docker
Processor --> FS
Processor --> UpdateServer
Processor --> Compose
classDef core fill:#f9f,stroke:#333,stroke-width:2px
classDef external fill:#bbf,stroke:#333,stroke-width:2px
class Main,State,Checker,Files,Wizard,Processor core
class Docker,FS,UpdateServer,Compose external
```
## Terms and Definitions
- **ProductStack**: Logical grouping of services that can be managed as a unit (pentagi, langfuse, observability, worker, installer, all)
- **Deployment Modes**: embedded (full local stack), external (existing service), disabled (no functionality)
- **State**: Persistent configuration storage including .env variables and wizard navigation stack
- **Checker**: System environment assessment providing current installation status and capabilities
- **TUI Wizard**: Terminal User Interface providing guided configuration flow
- **Embedded Content**: Docker compose files and configurations bundled within installer binary via go:embed
- **Worker Images**: Container images for AI agent tasks (default: debian:latest, pentest: vxcontrol/kali-linux)
- **ApplyChanges**: Deterministic state machine that transitions system from current to target configuration
- **Stack "all"**: Context-dependent operation affecting applicable stacks (excludes worker/installer for lifecycle operations)
- **Terminal Model**: Embedded pseudoterminal (`github.com/creack/pty`) providing real-time interactive command execution within TUI
- **Message Integration**: Channel-based processor integration via `ProcessorMessage` events for simple operations
## Architecture
### Main Components
- **processor.go**: Processor interface, options (`WithForce`, `WithTerminal`), and synchronous operation entry points
- **model.go**: Bubble Tea `ProcessorModel` with command wrappers and `HandleMsg` polling
- **compose.go**: Docker Compose operations and YAML file management, including strict purge `purgeImagesStack`
- **docker.go**: Docker API/CLI interactions for images, containers, networks, and volumes (worker + main)
- **fs.go**: File system operations and embedded content extraction with excluded files policy
- **logic.go**: Business logic (ApplyChanges state machine, lifecycle operations, factory reset)
- **update.go**: Update/download/remove scaffolding (installer flows currently stubbed)
- **state.go**: Operation state, messages, and terminal integration helpers
### ProductStack Types
```go
type ProductStack string
const (
StackPentAGI ProductStack = "pentagi" // Main stack (docker-compose.yml)
StackLangfuse ProductStack = "langfuse" // LLM observability (docker-compose-langfuse.yml)
StackObservability ProductStack = "observability" // System monitoring (docker-compose-observability.yml)
StackWorker ProductStack = "worker" // Docker images for AI agent tasks
StackInstaller ProductStack = "installer" // Installer binary itself
StackAll ProductStack = "all" // Context-dependent multi-stack operation
)
```
## Detailed Operation Scenarios
### Lifecycle Management
- **Start(stack)**:
- pentagi/langfuse/observability: `docker compose ... up -d` (honors embedded mode for non-destructive ops)
- all: sequential start in order observability → langfuse → pentagi
- worker/installer: not applicable
- **Stop(stack)**:
- pentagi/langfuse/observability: `docker compose ... stop`
- all: sequential stop in reverse order (pentagi → langfuse → observability)
- worker/installer: not applicable
- **Restart(stack)**:
- implemented as stop + small delay + start to avoid dependency race
- worker/installer: not applicable
### Installation & Content Management
- **Download(stack)**:
- pentagi/langfuse/observability: `docker compose pull`
- worker: `docker pull ${DOCKER_DEFAULT_IMAGE_FOR_PENTEST}` (default 6GB+)
- installer: stubbed (not implemented fully yet)
- all: download all applicable stacks
- **Install(stack)**:
- pentagi: extract compose file and example provider config
- langfuse: extract compose file (embedded mode only)
- observability: extract compose file and directory tree (embedded mode only)
- worker: download images
- installer: not applicable
- all: install all configured stacks
- **Update(stack)**:
- pentagi/langfuse/observability: download → `docker compose up -d`
- worker: download only (no forced restart)
- installer: stubbed (checksum/replace helpers exist, flow returns not implemented)
- all: sequence with dependency ordering
### Removal Operations
- **Remove(stack)**:
- pentagi/langfuse/observability: `docker compose down` (keep volumes/images)
- worker: remove images via Docker API and related containers
- installer: remove flow stubbed
- all: remove all stacks
- **Purge(stack)**:
- pentagi/langfuse/observability: `down --rmi all -v` for strict purge; standard purge `down -v` is also available
- worker: remove all containers, images, and volumes in worker environment
- installer: complete removal flow stubbed
- all: purge all stacks and remove custom networks
### State Management
- **ApplyChanges()**:
- pre-phase (wizard): integrity scan, user selects overwrite (force) or keep (no force)
- phase 1: observability (ensure/verify files → update stack or remove if external/disabled)
- phase 2: langfuse (same logic; local start requires `LangfuseConnected`)
- phase 3: pentagi (always embedded; ensure/verify → update)
- refresh checker state after each phase
- **ResetChanges()**:
- Call `state.State.Reset()` to discard pending configuration changes
- Preserve committed state from previous successful operations
- Reset wizard navigation stack to last stable point
## Implementation Strategy
### High-Level Method Organization
Each specialized file should contain business-logic level methods that directly support processor interface operations, not low-level utilities.
### Stack-Specific Operations
- **compose.go**:
- `installPentAGI()`, `installLangfuse()`, `installObservability()` - extract compose files with environment patching
- `startStack(stack)`, `stopStack(stack)`, `restartStack(stack)` - orchestrate docker compose commands
- `updateStack(stack)` - rolling updates with health checks
- **docker.go**:
- `pullWorkerImage()` - download pentest image (DOCKER_DEFAULT_IMAGE_FOR_PENTEST: vxcontrol/kali-linux)
- `pullDefaultImage()` - download general image (DOCKER_DEFAULT_IMAGE: debian:latest)
- `removeWorkerContainers()` - cleanup running worker containers (ports 28000-32000 range)
- `purgeWorkerImages()` - complete image removal including fallback images
- **fs.go**:
- `ensureStackIntegrity(stack, force)` - create missing files, force update existing ones
- `verifyStackIntegrity(stack, force)` - validate existing files, update if force=true
- `cleanupStackFiles(stack)` - remove extracted files and directories
- File integrity validation with YAML syntax checking
- Embedded directory tree handling for observability stack
- **update.go**:
- `checkUpdates()` - communicate with pentagi.com update server
- `downloadInstaller()` - atomic binary replacement with backup
- `updateStackImages(stack)` - orchestrate image updates with rollback
- **remove.go**:
- `removeStack(stack)` - soft removal preserving data
- `purgeStack(stack)` - complete cleanup including volumes and images
### Critical Implementation Details
- **Two-Track Command Execution**:
- Worker stack: Docker API SDK (uses DOCKER_HOST, PENTAGI_DOCKER_CERT_PATH, DOCKER_TLS_VERIFY from config)
- Compose stacks: Console commands with live output streaming to TUI
- **TUI Integration Modes**:
- **Embedded Terminal**: Real-time pseudoterminal integration (`ProcessorTerminalModel`) via `github.com/creack/pty`
- **Message Channel**: Simple progress tracking via `ProcessorMessage` events through channels
- **Toggle Support**: Ctrl+T switches between modes for debugging/compatibility
- **Deployment Mode Handling**:
- Langfuse: embedded (full stack), external (existing server), disabled (no analytics); local start guarded by `LangfuseConnected`
- Observability: embedded (full stack), external (OTEL collector), disabled (no monitoring)
- **Environment Variable Handling**: Compose files use --env-file parameter for environment variables, only special cases require file patching
- **Progress Tracking**: Worker downloads (vxcontrol/kali-linux 6GB+ → 13GB disk) with real-time progress via terminal
- **Docker Configuration**: Support NET_ADMIN capability for network scanning, Docker socket access for container management
- **Dependency Ordering**: PentAGI must start before Langfuse/Observability for network creation
- **State Persistence**: All operations update checker.CheckResult and state.State for consistency
- **Atomic Operations**: Install/Update operations must be reversible on failure
### Integration Patterns
- **Wizard → Processor**: Called via `wizard.controllers.StateController` on user action
- **State Coordination**: Processor updates both `state.State` (configuration) and internal state tracking
- **File System Layout**: Working directory contains .env + extracted compose files + .state/ subdirectory
- **Container Naming**: Follows patterns in checker constants (PentagiContainerName, etc.)
### Key Environment Variables
- **LLM providers**: OPEN_AI_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, BEDROCK_*, DEEPSEEK_*, GLM_*, KIMI_*, QWEN_*, OLLAMA_SERVER_URL
- **Provider configs**: PENTAGI_LLM_SERVER_CONFIG_PATH (host path), PENTAGI_OLLAMA_SERVER_CONFIG_PATH (host path)
- **Monitoring**: LANGFUSE_BASE_URL, LANGFUSE_PROJECT_ID, OTEL_HOST
- **Docker config**: DOCKER_HOST, PENTAGI_DOCKER_CERT_PATH (host path), DOCKER_TLS_VERIFY, DOCKER_CERT_PATH (container path, managed)
- **Deployment modes**: envs determine embedded vs external vs disabled
- **Worker images**: DOCKER_DEFAULT_IMAGE (debian:latest), DOCKER_DEFAULT_IMAGE_FOR_PENTEST (vxcontrol/kali-linux)
- **Path migration**: DoMigrateSettings() migrates old DOCKER_CERT_PATH/LLM_SERVER_CONFIG_PATH/OLLAMA_SERVER_CONFIG_PATH to PENTAGI_* variants on startup
### Error Handling Strategy
- **Validation errors**: validate stack applicability before operation
- **System errors**: Docker API failures, file permissions, network issues
- **State conflicts**: partial installations, conflicting configs, version mismatches
- **Rollback logic**: atomic replace helpers for installer binary; compose operations are fail-fast without rollback
## Integration Points
### Dependencies
- **checker package**: System state assessment via `checker.CheckResult`
- **state package**: Configuration management via `state.State`
- **files package**: Embedded content access via `files.Files`
- **loader package**: .env file operations
### External Systems
- Docker Compose CLI for stack orchestration
- Docker API for container/image management
- HTTP client for installer updates from pentagi.com
- File system for content extraction and cleanup
## ApplyChanges State Machine
Deterministic operation sequence based on current vs target configuration:
```mermaid
graph TD
Start([ApplyChanges Called]) --> Assess[Assess Current State]
Assess --> Compare[Compare with Target State]
Compare --> Plan[Generate Operation Plan]
Plan --> NeedInstall{Need Install?}
NeedInstall -->|Yes| Install[Install Stack Files]
NeedInstall -->|No| NeedDownload{Need Download?}
Install --> NeedDownload
NeedDownload -->|Yes| Download[Download Images/Binaries]
NeedDownload -->|No| NeedUpdate{Need Update?}
Download --> NeedUpdate
NeedUpdate -->|Yes| Update[Update Running Services]
NeedUpdate -->|No| NeedStart{Need Start?}
Update --> NeedStart
NeedStart -->|Yes| Start[Start Services]
NeedStart -->|No| Success[Operation Complete]
Start --> Success
Install --> Rollback{Error?}
Download --> Rollback
Update --> Rollback
Start --> Rollback
Rollback -->|Yes| Cleanup[Rollback Changes]
Rollback -->|No| Success
Cleanup --> Failure[Operation Failed]
```
### Decision Matrix
- **Fresh Install**: Install → Download → Start
- **Update Available**: Download → Update
- **Configuration Change**: Install → Update → Restart
- **Multi-Stack Setup**: Sequential operations with dependency handling
## User Scenarios & Integration
### Primary Use Cases
1. **First-time Installation**: User runs installer, configures via TUI, calls ApplyChanges to deploy complete stack
2. **Configuration Updates**: User modifies .env settings via TUI, ApplyChanges determines minimal required operations
3. **Stack Management**: User enables/disables Langfuse or Observability, system installs/removes appropriate components
4. **System Updates**: Periodic update checks trigger Download/Update operations for newer versions
5. **Troubleshooting**: Remove/Install cycles for component reset, Purge for complete cleanup
### Wizard Integration Flow
1. `main.go` initializes state, checker, launches wizard
2. `wizard.App` provides TUI for configuration changes
3. `wizard.controllers.StateController` manages state modifications
4. User triggers "Apply Changes" → wizard runs integrity scan (Enter), prompts user for update decision (Y/N), then executes `processor.ApplyChanges()` with/without force
5. Operations execute with real-time feedback to TUI; Ctrl+C cancels integrity stage only
6. `state.Commit()` persists successful changes, `state.Reset()` on failure
### Processor Usage in Wizard
**Controller Integration**: StateController creates processor instance and delegates operations
```go
// wizard/controllers/state_controller.go
type StateController struct {
state *state.State
checker *checker.CheckResult
processor processor.Processor
}
func (c *StateController) ApplyUserChanges() error {
return c.processor.ApplyChanges(context.Background(),
processor.WithForce(), // User explicitly requested changes
)
}
```
**Screen-Level Operations**: Modern approach with embedded terminal model
```go
// wizard/models/apply_changes.go (current implementation)
type ApplyChangesFormModel struct {
processor processor.Processor
terminalModel processor.ProcessorTerminalModel
useEmbeddedTerminal bool
}
func (m *ApplyChangesFormModel) startApplyProcess() tea.Cmd {
if m.useEmbeddedTerminal && m.terminalModel != nil {
return m.terminalModel.StartOperation("ApplyChanges", processor.ProductStackAll)
}
// Fallback for message-based integration
messageChan := make(chan processor.ProcessorMessage, 100)
return processor.CreateApplyChangesCommand(m.processor,
processor.WithForce(),
processor.WithTea(messageChan),
)
}
// Terminal model integration in Update method
func (m *ApplyChangesFormModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
if m.useEmbeddedTerminal && m.terminalModel != nil {
updatedModel, terminalCmd := m.terminalModel.Update(msg)
if terminalModel, ok := updatedModel.(processor.ProcessorTerminalModel); ok {
m.terminalModel = terminalModel
}
return m, terminalCmd
}
// Handle other cases...
}
```
**Real-Time Integration**: Terminal model provides native real-time feedback
```go
// No manual polling required - terminal model handles real-time updates automatically
func (m *ApplyChangesFormModel) renderTerminalPanel() string {
if m.useEmbeddedTerminal && m.terminalModel != nil {
return m.terminalModel.View() // Complete terminal interface
}
return m.renderFallbackPanel() // Message-based fallback
}
// Dynamic resizing support
case tea.WindowSizeMsg:
if m.useEmbeddedTerminal && m.terminalModel != nil {
contentWidth, contentHeight := m.getViewportFormSize()
m.terminalModel.SetSize(contentWidth-2, contentHeight-2)
}
```
## Implementation Requirements
### CommandOption Design
```go
type commandConfig struct {
// Execution control
Force bool // Skip validation checks and attempt maximum operations
// TUI Integration - two integration modes available
Tea chan ProcessorMessage // Message-based integration via channel
TerminalModel ProcessorTerminalModel // Embedded terminal model integration
}
// Simplified option pattern (only essential options)
type CommandOption func(*commandConfig)
func WithForce() CommandOption {
return func(c *commandConfig) { c.Force = true }
}
func WithTea(messageChan chan ProcessorMessage) CommandOption {
return func(c *commandConfig) { c.Tea = messageChan }
}
func WithTerminalModel(terminal ProcessorTerminalModel) CommandOption {
return func(c *commandConfig) { c.TerminalModel = terminal }
}
```
### Update Server Protocol
- Uses existing `checker.CheckUpdatesRequest`/`CheckUpdatesResponse` structures
- Binary download with SHA256 verification
- Atomic replacement via temporary file + rename
- Post-update exit with restart instruction message
### Critical Safety Measures
- **Pre-flight Checks**: Validate system resources before major operations
- **Backup Strategy**: Create backups before destructive operations (Purge)
- **Network Isolation**: Respect proxy settings from environment configuration
- **Permission Handling**: Graceful handling of Docker socket access requirements
### Current Architecture
**Core Components**:
- **processor.go**: Interface implementation and delegation
- **model.go**: ProcessorModel for tea.Cmd wrapping
- **logic.go**: Business logic (ApplyChanges, lifecycle operations)
- **state.go**: Operation state management
- **compose.go, docker.go, fs.go, update.go**: Specialized operations
**Testing Infrastructure**:
- **mock_test.go**: Comprehensive mocks with call tracking
- **logic_test.go**: Business logic tests
- **fs_test.go**: File system operation tests
- **Mock CheckHandler**: Flexible system state simulation
**Integration Points**:
- **Wizard**: Uses ProcessorModel with terminal integration
- **Checker**: Uses CheckHandler interface for state assessment
- **State**: Configuration management and persistence
backend/docs/installer/reference-config-pattern.md
+438
View File
@@ -0,0 +1,438 @@
# Reference Configuration Pattern
> Reference implementation of configuration using ScraperConfig as an example for all future configurations in StateController.
## 🎯 **Core Principles**
### 1. **Use loader.EnvVar for direct form mapping**
```go
type ReferenceConfig struct {
// ✅ Direct form mapping - use loader.EnvVar
DirectField1 loader.EnvVar // SOME_ENV_VAR
DirectField2 loader.EnvVar // ANOTHER_ENV_VAR
// ✅ Computed fields - simple types
ComputedMode string // computed from DirectField1
// ✅ Temporary data for processing
TempData string // not saved, used for logic
}
```
### 2. **Reference configuration structure**
```go
// ScraperConfig represents scraper configuration settings
type ScraperConfig struct {
// direct form field mappings using loader.EnvVar
// these fields directly correspond to environment variables and form inputs (not computed)
PublicURL loader.EnvVar // SCRAPER_PUBLIC_URL
PrivateURL loader.EnvVar // SCRAPER_PRIVATE_URL
LocalUsername loader.EnvVar // LOCAL_SCRAPER_USERNAME
LocalPassword loader.EnvVar // LOCAL_SCRAPER_PASSWORD
MaxConcurrentSessions loader.EnvVar // LOCAL_SCRAPER_MAX_CONCURRENT_SESSIONS
// computed fields (not directly mapped to env vars)
// these are derived from the above EnvVar fields
Mode string // "embedded", "external", "disabled" - computed from PrivateURL
// parsed credentials for external mode (extracted from URLs)
PublicUsername string
PublicPassword string
PrivateUsername string
PrivatePassword string
}
```
### 3. **Constants for default values**
```go
const (
DefaultScraperBaseURL = "https://scraper/"
DefaultScraperDomain = "scraper"
DefaultScraperSchema = "https"
)
```
## 🔧 **Configuration Methods**
### 1. **GetConfig() - Retrieve configuration**
```go
func (c *StateController) GetScraperConfig() *ScraperConfig {
// get all environment variables using the state controller
publicURL, _ := c.GetVar("SCRAPER_PUBLIC_URL")
privateURL, _ := c.GetVar("SCRAPER_PRIVATE_URL")
localUsername, _ := c.GetVar("LOCAL_SCRAPER_USERNAME")
localPassword, _ := c.GetVar("LOCAL_SCRAPER_PASSWORD")
maxSessions, _ := c.GetVar("LOCAL_SCRAPER_MAX_CONCURRENT_SESSIONS")
config := &ScraperConfig{
PublicURL: publicURL,
PrivateURL: privateURL,
LocalUsername: localUsername,
LocalPassword: localPassword,
MaxConcurrentSessions: maxSessions,
}
// compute derived fields using multiple inputs
config.Mode = c.determineScraperMode(privateURL.Value, publicURL.Value)
// for external mode, extract credentials from URLs
if config.Mode == "external" {
config.PublicUsername, config.PublicPassword = c.extractCredentialsFromURL(publicURL.Value)
config.PrivateUsername, config.PrivatePassword = c.extractCredentialsFromURL(privateURL.Value)
}
return config
}
```
### 2. **UpdateConfig() - Update configuration**
```go
func (c *StateController) UpdateScraperConfig(config *ScraperConfig) error {
if config == nil {
return fmt.Errorf("config cannot be nil")
}
switch config.Mode {
case "disabled":
// clear scraper URLs, preserve local settings
if err := c.SetVar("SCRAPER_PUBLIC_URL", ""); err != nil {
return fmt.Errorf("failed to clear SCRAPER_PUBLIC_URL: %w", err)
}
if err := c.SetVar("SCRAPER_PRIVATE_URL", ""); err != nil {
return fmt.Errorf("failed to clear SCRAPER_PRIVATE_URL: %w", err)
}
case "external":
// construct URLs with credentials if provided
publicURL := config.PublicURL.Value
if config.PublicUsername != "" && config.PublicPassword != "" {
publicURL = c.addCredentialsToURL(config.PublicURL.Value, config.PublicUsername, config.PublicPassword)
}
privateURL := config.PrivateURL.Value
if config.PrivateUsername != "" && config.PrivatePassword != "" {
privateURL = c.addCredentialsToURL(config.PrivateURL.Value, config.PrivateUsername, config.PrivatePassword)
}
if err := c.SetVar("SCRAPER_PUBLIC_URL", publicURL); err != nil {
return fmt.Errorf("failed to set SCRAPER_PUBLIC_URL: %w", err)
}
if err := c.SetVar("SCRAPER_PRIVATE_URL", privateURL); err != nil {
return fmt.Errorf("failed to set SCRAPER_PRIVATE_URL: %w", err)
}
case "embedded":
// handle embedded mode with credential mapping
publicURL := config.PublicURL.Value
if config.PublicUsername != "" && config.PublicPassword != "" {
// fallback to private URL if public URL is not set
if privateURL := config.PrivateURL.Value; privateURL != "" && publicURL == "" {
publicURL = privateURL
}
publicURL = c.addCredentialsToURL(publicURL, config.PublicUsername, config.PublicPassword)
}
privateURL := config.PrivateURL.Value
if config.PrivateUsername != "" && config.PrivatePassword != "" {
privateURL = c.addCredentialsToURL(privateURL, config.PrivateUsername, config.PrivatePassword)
}
// update all relevant variables
if err := c.SetVar("SCRAPER_PUBLIC_URL", publicURL); err != nil {
return fmt.Errorf("failed to set SCRAPER_PUBLIC_URL: %w", err)
}
if err := c.SetVar("SCRAPER_PRIVATE_URL", privateURL); err != nil {
return fmt.Errorf("failed to set SCRAPER_PRIVATE_URL: %w", err)
}
// map credentials to local settings
if err := c.SetVar("LOCAL_SCRAPER_USERNAME", config.PrivateUsername); err != nil {
return fmt.Errorf("failed to set LOCAL_SCRAPER_USERNAME: %w", err)
}
if err := c.SetVar("LOCAL_SCRAPER_PASSWORD", config.PrivatePassword); err != nil {
return fmt.Errorf("failed to set LOCAL_SCRAPER_PASSWORD: %w", err)
}
if err := c.SetVar("LOCAL_SCRAPER_MAX_CONCURRENT_SESSIONS", config.MaxConcurrentSessions.Value); err != nil {
return fmt.Errorf("failed to set LOCAL_SCRAPER_MAX_CONCURRENT_SESSIONS: %w", err)
}
}
return nil
}
```
### 3. **ResetConfig() - Reset to defaults**
```go
func (c *StateController) ResetScraperConfig() *ScraperConfig {
// reset all scraper-related environment variables to their defaults
vars := []string{
"SCRAPER_PUBLIC_URL",
"SCRAPER_PRIVATE_URL",
"LOCAL_SCRAPER_USERNAME",
"LOCAL_SCRAPER_PASSWORD",
"LOCAL_SCRAPER_MAX_CONCURRENT_SESSIONS",
}
if err := c.ResetVars(vars); err != nil {
return nil
}
return c.GetScraperConfig()
}
```
## 🧩 **Helper Methods**
### 1. **Mode determination with multiple inputs**
```go
func (c *StateController) determineScraperMode(privateURL, publicURL string) string {
if privateURL == "" && publicURL == "" {
return "disabled"
}
parsedURL, err := url.Parse(privateURL)
if err != nil {
return "external"
}
if parsedURL.Scheme == DefaultScraperSchema && parsedURL.Hostname() == DefaultScraperDomain {
return "embedded"
}
return "external"
}
```
### 2. **URL credential handling with defaults**
```go
func (c *StateController) addCredentialsToURL(urlStr, username, password string) string {
if username == "" || password == "" {
return urlStr
}
if urlStr == "" {
urlStr = DefaultScraperBaseURL
}
parsedURL, err := url.Parse(urlStr)
if err != nil {
return urlStr
}
// set user info
parsedURL.User = url.UserPassword(username, password)
return parsedURL.String()
}
```
### 3. **Public method for safe display**
```go
// RemoveCredentialsFromURL removes credentials from URL - public method for form display
func (c *StateController) RemoveCredentialsFromURL(urlStr string) string {
if urlStr == "" {
return urlStr
}
parsedURL, err := url.Parse(urlStr)
if err != nil {
return urlStr
}
parsedURL.User = nil
return parsedURL.String()
}
```
## 📋 **Form Integration**
### 1. **Field creation**
```go
func (m *FormModel) createURLField(key, title, description, placeholder string) FormField {
input := textinput.New()
input.Placeholder = placeholder
var value string
switch key {
case "public_url":
value = m.config.PublicURL.Value
case "private_url":
value = m.config.PrivateURL.Value
}
if value != "" {
input.SetValue(value)
}
return FormField{
Key: key,
Title: title,
Description: description,
Input: input,
Value: input.Value(),
}
}
```
### 2. **Save handling with validation**
```go
func (m *ScraperFormModel) HandleSave() error {
mode := m.getSelectedMode()
fields := m.GetFormFields()
// create a working copy of the current config to modify
newConfig := &controllers.ScraperConfig{
Mode: mode,
// copy current EnvVar fields - they preserve metadata like Line, IsPresent, etc.
PublicURL: m.config.PublicURL,
PrivateURL: m.config.PrivateURL,
LocalUsername: m.config.LocalUsername,
LocalPassword: m.config.LocalPassword,
MaxConcurrentSessions: m.config.MaxConcurrentSessions,
}
// update field values based on form input
for _, field := range fields {
value := strings.TrimSpace(field.Input.Value())
switch field.Key {
case "public_url":
newConfig.PublicURL.Value = value
case "private_url":
newConfig.PrivateURL.Value = value
case "local_username":
newConfig.LocalUsername.Value = value
case "local_password":
newConfig.LocalPassword.Value = value
case "max_sessions":
// validate numeric input
if value != "" {
if _, err := strconv.Atoi(value); err != nil {
return fmt.Errorf("invalid number for max concurrent sessions: %s", value)
}
}
newConfig.MaxConcurrentSessions.Value = value
}
}
// set defaults for embedded mode if needed
if mode == "embedded" {
if newConfig.LocalUsername.Value == "" {
newConfig.LocalUsername.Value = "someuser"
}
if newConfig.LocalPassword.Value == "" {
newConfig.LocalPassword.Value = "somepass"
}
if newConfig.MaxConcurrentSessions.Value == "" {
newConfig.MaxConcurrentSessions.Value = "10"
}
}
// save the configuration
if err := m.GetController().UpdateScraperConfig(newConfig); err != nil {
return err
}
// reload config to get updated state
m.config = m.GetController().GetScraperConfig()
return nil
}
```
### 3. **Safe display in overview**
```go
func (m *ScraperFormModel) GetFormOverview() string {
config := m.GetController().GetScraperConfig()
var sections []string
sections = append(sections, "Current Configuration:")
switch config.Mode {
case "embedded":
sections = append(sections, "• Mode: Embedded")
if config.PublicURL.Value != "" {
sections = append(sections, "• Public URL: " + config.PublicURL.Value)
}
if config.LocalUsername.Value != "" {
sections = append(sections, "• Local Username: " + config.LocalUsername.Value)
}
case "external":
sections = append(sections, "• Mode: External")
if config.PublicURL.Value != "" {
// show clean URL without credentials for security
cleanURL := m.GetController().RemoveCredentialsFromURL(config.PublicURL.Value)
sections = append(sections, "• Public URL: " + cleanURL)
}
if config.PrivateURL.Value != "" {
cleanURL := m.GetController().RemoveCredentialsFromURL(config.PrivateURL.Value)
sections = append(sections, "• Private URL: " + cleanURL)
}
case "disabled":
sections = append(sections, "• Mode: Disabled")
}
return strings.Join(sections, "\n")
}
```
## ✅ **Benefits of Reference Approach**
1. **State tracking**: `loader.EnvVar` tracks changes, file presence, default values
2. **Metadata preservation**: Information about changes, presence, defaults
3. **Security**: Public methods for safe display of sensitive data
4. **Consistency**: Uniform behavior across all configurations
5. **Reliability**: Minimizes errors in different usage scenarios
6. **URL handling**: Uses `net/url` package for robust URL parsing
7. **Default management**: Constants for maintainable default values
## 🔄 **Key Patterns**
### 1. **Data Types**
| Field Type | Data Type | Usage |
|------------|-----------|-------|
| **Direct mapping** | `loader.EnvVar` | Form fields, env variables |
| **Computed** | `string`/`bool`/`int` | Modes, status, flags |
| **Temporary** | `string` | Parsing, processing |
### 2. **Method signatures**
- `GetConfig() *Config` - retrieves with metadata
- `UpdateConfig(config *Config) error` - saves with validation
- `ResetConfig() *Config` - resets to defaults
- `PublicMethod()` - exported for form usage
### 3. **Error handling**
- Always validate input parameters
- Use `fmt.Errorf` with context
- Handle URL parsing errors gracefully
- Provide meaningful error messages
## 🚀 **Creating New Configurations**
```go
// 1. Define structure
type NewServiceConfig struct {
// direct fields
APIKey loader.EnvVar // NEW_SERVICE_API_KEY
BaseURL loader.EnvVar // NEW_SERVICE_BASE_URL
Enabled loader.EnvVar // NEW_SERVICE_ENABLED
// computed fields
IsConfigured bool
}
// 2. Add constants
const (
DefaultNewServiceURL = "https://api.newservice.com"
)
// 3. Implement methods
func (c *StateController) GetNewServiceConfig() *NewServiceConfig { /* ... */ }
func (c *StateController) UpdateNewServiceConfig(config *NewServiceConfig) error { /* ... */ }
func (c *StateController) ResetNewServiceConfig() *NewServiceConfig { /* ... */ }
// 4. Create form following ScraperFormModel pattern
```
This reference approach ensures reliable and consistent operation of all configurations in the system.
backend/docs/installer/terminal-wizard-integration.md
+561
View File
@@ -0,0 +1,561 @@
# Terminal Integration Guide for Wizard Screens
This guide covers integration of the `terminal` package into wizard configuration screens, providing command execution capabilities with real-time UI updates.
## Core Architecture
The terminal system consists of three layers:
- **Virtual Terminal (VT)**: Low-level ANSI parsing and screen management (`terminal/vt/`)
- **Terminal Interface**: High-level command execution with PTY/pipe support (`terminal/`)
- **Wizard Integration**: Screen-specific integration patterns (`wizard/models/`)
## Terminal Modes
### PTY Mode (Default on Unix)
- Full pseudoterminal emulation via `creack/pty`
- ANSI escape sequence processing through VT layer
- Interactive command support (vim, less, etc.)
- Proper terminal environment variables
### Pipe Mode (Windows/NoPty)
- Standard stdin/stdout/stderr pipes
- Line-by-line output processing
- Simpler but limited interactivity
- Plain text output handling
## Configuration Options
Terminal behavior is controlled via functional options:
```go
// Essential options for wizard integration
terminal.NewTerminal(width, height,
terminal.WithAutoScroll(), // Auto-scroll to bottom on updates
terminal.WithAutoPoll(), // Continuous update polling
terminal.WithCurrentEnv(), // Inherit process environment
)
// Advanced options
terminal.WithNoStyled() // Disable ANSI styling (PTY only)
terminal.WithNoPty() // Force pipe mode
terminal.WithStyle(lipgloss.Style) // Custom viewport styling
```
## Integration Patterns
### Complete Integration Template
```go
type YourFormModel struct {
*BaseScreen
terminal terminal.Terminal
// other screen-specific fields
}
func NewYourFormModel(controller *controllers.StateController, styles *styles.Styles, window *window.Window, args []string) *YourFormModel {
m := &YourFormModel{}
m.BaseScreen = NewBaseScreen(controller, styles, window, args, m, nil)
return m
}
// Required BaseScreenHandler implementation
func (m *YourFormModel) BuildForm() tea.Cmd {
contentWidth, contentHeight := m.getViewportFormSize()
// Initialize or reset terminal
if m.terminal == nil {
m.terminal = terminal.NewTerminal(
contentWidth-4, // Account for border + padding
contentHeight-1, // Account for border
terminal.WithAutoScroll(),
terminal.WithAutoPoll(),
terminal.WithCurrentEnv(),
)
} else {
m.terminal.Clear()
}
// Set initial content
m.terminal.Append("Terminal initialized...")
// CRITICAL: Return terminal init for update subscription (idempotent)
// repeated calls to Init() are safe: only a single waiter will receive
// the next TerminalUpdateMsg; others will return nil quietly
return m.terminal.Init()
}
```
### Sizing Calculations
The sizing adjustments account for UI elements:
- **Width -4**: Left border (1) + left padding (1) + right padding (1) + right border (1)
- **Height -1**: Top/bottom borders, content area needs space for text
- Use `m.getViewportFormSize()` from BaseScreen for consistent calculations
- Handle dynamic resizing in Update() method
### Event Flow Architecture
The system uses a single-waiter update notifier for real-time updates:
1. **Update Notifier**: Manages single-waiter update notifications (`teacmd.go`)
2. **Update Messages**: `TerminalUpdateMsg` carries terminal ID
3. **Subscription Model**: Commands wait for `release()` signalling the next update
4. **Auto-polling**: Continuous listening when `WithAutoPoll()` enabled
5. **Single-waiter guarantee**: For a given `Terminal`, at most one pending waiter is active at any time. Multiple `Init()` calls are safe; only one will receive the next `TerminalUpdateMsg` after `release()`, others return nil.
### Complete Update Method Implementation
```go
func (m *YourFormModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
// Helper function to handle terminal delegation
handleTerminal := func(msg tea.Msg) (tea.Model, tea.Cmd) {
if m.terminal == nil {
return m, nil
}
updatedModel, cmd := m.terminal.Update(msg)
if terminalModel := terminal.RestoreModel(updatedModel); terminalModel != nil {
m.terminal = terminalModel
}
return m, cmd
}
switch msg := msg.(type) {
case tea.WindowSizeMsg:
// Update terminal size first
contentWidth, contentHeight := m.getViewportFormSize()
if m.terminal != nil {
m.terminal.SetSize(contentWidth-4, contentHeight-1)
}
// Update viewports (BaseScreen functionality)
m.updateViewports()
return m, nil
case terminal.TerminalUpdateMsg:
// Terminal content updated - delegate and continue listening
// Only a single update message will be emitted per content change,
// even if Init() was invoked multiple times
return handleTerminal(msg)
case tea.KeyMsg:
// Route keys based on terminal state
if m.terminal != nil && m.terminal.IsRunning() {
// Command is running - all keys go to terminal
return handleTerminal(msg)
}
// Terminal is idle - handle screen-specific hotkeys first
switch msg.String() {
case "enter":
// Your screen-specific action
if m.terminal != nil && !m.terminal.IsRunning() {
m.executeCommands()
return m, nil
}
case "ctrl+r":
// Reset functionality
m.handleReset()
return m, nil
}
// Pass remaining keys to terminal for scrolling
return handleTerminal(msg)
default:
// Other messages (like custom commands) - delegate to terminal
return handleTerminal(msg)
}
}
```
### Event Processing Order
Process events in this strict order to ensure proper functionality:
1. **Window Resize**: Update terminal dimensions before any other processing
2. **Terminal Updates**: Handle `TerminalUpdateMsg` immediately for real-time updates
3. **Key Routing**: Route based on `IsRunning()` state
- **Running commands**: All keys forwarded to terminal for interaction
- **Idle terminal**: Screen hotkeys first, then terminal scrolling
4. **Other Messages**: Delegate to terminal for potential internal handling
## Command Execution
### Single Command with Error Handling
```go
func (m *YourFormModel) executeCommand() {
cmd := exec.Command("echo", "hello")
err := m.terminal.Execute(cmd)
if err != nil {
// Display error to user through terminal
m.terminal.Append(fmt.Sprintf("❌ Command failed: %v", err))
m.terminal.Append("Please check the command and try again.")
return
}
// Wait for completion if needed
go func() {
for m.terminal.IsRunning() {
time.Sleep(100 * time.Millisecond)
}
m.terminal.Append("✅ Command completed successfully")
}()
}
```
### Sequential Commands with Robust Error Handling
```go
func (m *YourFormModel) executeCommands() {
if m.terminal.IsRunning() {
m.terminal.Append("⚠️ Another command is already running")
return
}
commands := []struct {
cmd []string
desc string
canFail bool
}{
{[]string{"echo", "Starting process..."}, "Initialize", false},
{[]string{"docker", "--version"}, "Check Docker", true},
{[]string{"docker-compose", "up", "-d"}, "Start services", false},
}
go func() {
for i, cmdDef := range commands {
if i > 0 {
time.Sleep(500 * time.Millisecond)
}
m.terminal.Append(fmt.Sprintf("🔄 Step %d: %s", i+1, cmdDef.desc))
cmd := exec.Command(cmdDef.cmd[0], cmdDef.cmd[1:]...)
err := m.terminal.Execute(cmd)
if err != nil {
m.terminal.Append(fmt.Sprintf("❌ Failed: %v", err))
if !cmdDef.canFail {
m.terminal.Append("💥 Critical error - stopping execution")
return
}
m.terminal.Append("⚠️ Non-critical error - continuing...")
continue
}
// Wait for command completion
timeout := time.After(30 * time.Second)
ticker := time.NewTicker(100 * time.Millisecond)
completed := false
for !completed {
select {
case <-timeout:
m.terminal.Append("⏰ Command timeout - terminating")
return
case <-ticker.C:
if !m.terminal.IsRunning() {
completed = true
}
}
}
ticker.Stop()
}
m.terminal.Append("🎉 All commands completed successfully!")
}()
}
### Interactive Commands
Terminal automatically handles:
- Stdin forwarding in both PTY and pipe modes
- Key-to-input conversion (`key2uv.go`)
- ANSI escape sequence processing (PTY mode)
## Key Input Handling
### PTY Mode
- Full ANSI key sequence support via Ultraviolet conversion
- Vim-style navigation (arrows, page up/down, home/end)
- Control sequences (Ctrl+C, Ctrl+D, etc.)
- Alt+key combinations
### Pipe Mode
- Basic key mapping to stdin bytes
- Enter, space, tab, backspace support
- Control characters (Ctrl+C → \x03)
### Viewport Scrolling
Keys not consumed by running commands are passed to viewport:
- Page Up/Down, Home/End for navigation
- Preserved when terminal is idle
## Lifecycle Management
### Terminal Creation and Cleanup
```go
// Terminal lifecycle follows screen lifecycle
func (m *YourFormModel) BuildForm() tea.Cmd {
// Create terminal once per screen instance
if m.terminal == nil {
m.terminal = terminal.NewTerminal(...)
} else {
// Reset content when re-entering screen
m.terminal.Clear()
}
return m.terminal.Init()
}
// No manual cleanup needed - handled by finalizers
// Terminal will be cleaned up when screen model is garbage collected
```
### Screen Navigation Considerations
- **Terminal Persistence**: Terminal remains active during screen navigation
- **Content Reset**: Use `Clear()` when re-entering screens to avoid content buildup
- **Resource Cleanup**: Automatic via finalizers when screen model is destroyed
- **State Preservation**: Terminal state (size, options) persists across `BuildForm()` calls
### State Checking and Debugging
```go
// Essential state checks
if m.terminal == nil {
// Terminal not initialized - call BuildForm()
}
if m.terminal.IsRunning() {
// Command is executing - avoid new commands
// Show spinner or disable UI elements
}
// Debugging helpers
terminalID := m.terminal.ID() // Unique identifier for logging
width, height := m.terminal.GetSize() // Current dimensions
view := m.terminal.View() // Current rendered content
// For debugging terminal content
if DEBUG {
log.Printf("Terminal %s: %dx%d, running=%t",
terminalID, width, height, m.terminal.IsRunning())
}
```
### Resource Management Details
Resources managed via Go finalizers (`terminal.go:131-142`):
- **PTY file descriptors**: Automatically closed when terminal is garbage collected
- **Process termination**: Running processes killed during cleanup
- **Notifier shutdown**: Wait channel closed and state reset
- **Mutex-protected cleanup**: Thread-safe resource cleanup
- **No manual Close()**: Resources cleaned automatically, no explicit cleanup needed
## Virtual Terminal Capabilities
The VT layer provides advanced features:
- **Screen Buffer**: Main and alternate screen support
- **Scrollback**: Configurable history buffer
- **ANSI Processing**: Full VT100/xterm compatibility
- **Color Support**: 256-color palette + true color
- **Cursor Modes**: Various cursor styles and visibility
- **Character Sets**: GL/GR charset switching
## Testing Strategies
Key test patterns from `terminal_test.go`:
- **Command Output**: Verify content appears in `View()`
- **Interactive Input**: Simulate key sequences via `Update()`
- **Resource Cleanup**: Manual finalizer calls for verification
- **Concurrent Access**: Multiple goroutines with same terminal
- **Error Handling**: Invalid commands and process failures
## Concurrency and Threading
### Thread Safety
```go
// Terminal methods are thread-safe for these operations:
m.terminal.Append("message") // Safe from any goroutine
m.terminal.IsRunning() // Safe to check from any goroutine
m.terminal.ID() // Safe to call from any goroutine
// UI operations must be on main thread:
m.terminal.Update(msg) // Only from main BubbleTea thread
m.terminal.View() // Only from main rendering thread
m.terminal.SetSize(w, h) // Only from main thread
```
### Command Execution Patterns
```go
// CORRECT: Run commands in separate goroutine
go func() {
m.terminal.Append("Starting long operation...")
cmd := exec.Command("long-running-command")
err := m.terminal.Execute(cmd)
// Error handling...
}()
// INCORRECT: Blocking main thread
cmd := exec.Command("long-running-command")
m.terminal.Execute(cmd) // Will block UI updates
```
### AutoPoll vs Manual Updates
- **WithAutoPoll()**: Continuous listening, higher CPU but immediate updates; still single-waiter per terminal ensures no message storm. Updates are triggered internally via `release()` when content changes.
- **Manual polling**: Call `terminal.Init()` only when needed, lower resource usage
- **Use AutoPoll**: For active terminal screens with frequent updates
- **Skip AutoPoll**: For background or rarely updated terminals
## Troubleshooting Guide
### Terminal Not Updating
**Problem**: Terminal content doesn't appear or update
**Solutions**:
1. Ensure `terminal.Init()` is returned from `BuildForm()`
2. Check `TerminalUpdateMsg` handling in `Update()` method — it should return next wait command to continue listening
3. Verify `handleTerminal()` function calls `RestoreModel()`
4. Add debug logging to track message flow
```go
case terminal.TerminalUpdateMsg:
log.Printf("Received terminal update: %s", msg.ID)
return handleTerminal(msg)
```
### Commands Not Executing
**Problem**: `Execute()` returns nil but nothing happens
**Solutions**:
1. Check if previous command is still running: `m.terminal.IsRunning()`
2. Verify command path and arguments
3. Check terminal initialization
4. Add error logging and terminal output
```go
if m.terminal.IsRunning() {
m.terminal.Append("⚠️ Previous command still running")
return
}
```
### UI Freezing During Commands
**Problem**: Interface becomes unresponsive
**Solutions**:
1. Always run `Execute()` in goroutines for long commands
2. Use `WithAutoPoll()` for real-time updates
3. Implement proper key routing in `Update()`
### Resource Leaks
**Problem**: Memory or file descriptor leaks
**Solutions**:
1. Avoid creating multiple terminals unnecessarily
2. Let finalizers handle cleanup (don't try manual cleanup)
3. Check for goroutine leaks in command execution
### Size and Layout Issues
**Problem**: Terminal appears cut off or incorrectly sized
**Solutions**:
1. Use proper sizing calculations (width-4, height-1)
2. Handle `tea.WindowSizeMsg` correctly
3. Call `m.updateViewports()` after size changes
## Best Practices
### Initialization Checklist
- ✅ Return `terminal.Init()` from `BuildForm()` (idempotent, single-waiter)
- ✅ Use `WithAutoPoll()` for active terminals
- ✅ Set appropriate dimensions with border adjustments
- ✅ Initialize once per screen, clear content on re-entry
### Event Processing Checklist
- ✅ Handle `TerminalUpdateMsg` first in `Update()` and return next wait command
- ✅ Properly restore terminal models after updates
- ✅ Route keys based on `IsRunning()` state
- ✅ Update terminal size on window resize
### Command Management Checklist
- ✅ Run long operations in goroutines
- ✅ Check `IsRunning()` before new commands
- ✅ Use `Append()` for progress and error messages
- ✅ Implement timeouts for long-running commands
- ✅ Handle both critical and non-critical errors
### Performance Optimization
- **VT Layer**: Automatically caches rendered lines for efficiency
- **Notifier**: Single-waiter, release-based notifications to prevent message storms and deadlocks
- **Resource Cleanup**: Deferred via finalizers to avoid blocking
- **AutoPoll Usage**: Enable only for active terminals requiring real-time updates
## Integration Examples
### Progress Display (`apply_changes.go`)
Shows terminal integration in configuration screen with:
- Dynamic content based on configuration state
- Command execution with progress feedback
- Proper event routing and error handling
### Test Scenarios (`terminal_test.go`)
Demonstrates various usage patterns:
- Simple command output verification
- Interactive input simulation
- Concurrent command execution prevention
- Resource lifecycle management
## Platform Considerations
### Unix Systems
- PTY mode provides full terminal emulation
- ANSI sequences processed through VT layer
- Interactive commands work naturally
### Windows
- Pipe mode used automatically
- Limited interactivity compared to PTY
- Plain text output processing
### Environment Variables
- `TERM=xterm-256color` set automatically in PTY mode
- Current process environment inherited with `WithCurrentEnv()`
- Custom environment via `exec.Cmd.Env`
## Quick Reference
### Essential Methods
```go
// Creation and lifecycle
terminal.NewTerminal(width, height, options...)
m.terminal.Init() // Subscribe to updates
m.terminal.Clear() // Reset content
// Command execution
m.terminal.Execute(cmd) // Run command
m.terminal.IsRunning() // Check execution status
m.terminal.Append(text) // Add content
// UI integration
m.terminal.Update(msg) // Handle messages
m.terminal.View() // Render content
m.terminal.SetSize(width, height) // Update dimensions
```
### Common Error Patterns to Avoid
- ❌ Creating multiple terminals per screen
- ❌ Running `Execute()` on main thread for long commands
- ❌ Forgetting to return `terminal.Init()` from `BuildForm()`
- ❌ Not handling `TerminalUpdateMsg` in `Update()`
- ❌ Calling UI methods from background goroutines
- ❌ Manual resource cleanup (use finalizers instead)
### Integration Checklist for New Screens
1. ✅ Add `terminal terminal.Terminal` to model struct
2. ✅ Initialize in `BuildForm()` with proper sizing
3. ✅ Return `terminal.Init()` from `BuildForm()`
4. ✅ Handle `TerminalUpdateMsg` first in `Update()`
5. ✅ Implement proper key routing based on `IsRunning()`
6. ✅ Handle window resize events
7. ✅ Run commands in goroutines with error handling
8. ✅ Add progress feedback via `Append()`
This comprehensive architecture provides robust terminal integration with wizard screens while maintaining proper resource management, real-time UI updates, and cross-platform compatibility.
backend/docs/langfuse.md
+1088
View File
File diff suppressed because it is too large Load Diff
backend/docs/llms_how_to.md
+1863
View File
File diff suppressed because it is too large Load Diff
backend/docs/observability.md
+1011
View File
File diff suppressed because it is too large Load Diff
backend/docs/ollama.md
+147
View File
@@ -0,0 +1,147 @@
# Ollama Provider
The Ollama provider enables PentAGI to use local language models through the [Ollama](https://ollama.ai/) server.
## Installation
1. Install Ollama server on your system following the [official installation guide](https://ollama.ai/download)
2. Start the Ollama server (usually runs on `http://localhost:11434`)
3. Pull required models: `ollama pull gemma3:1b`
## Configuration
Configure the Ollama provider using environment variables:
### Required Variables
```bash
# Ollama server URL (default: http://localhost:11434)
OLLAMA_SERVER_URL=http://localhost:11434
```
### Optional Variables
```bash
# Default model for inference (optional, default: llama3.1:8b-instruct-q8_0)
OLLAMA_SERVER_MODEL=llama3.1:8b-instruct-q8_0
# Path to custom config file (optional)
OLLAMA_SERVER_CONFIG_PATH=/path/to/ollama_config.yml
# Model management settings (optional)
OLLAMA_SERVER_PULL_MODELS_TIMEOUT=600 # Timeout for model downloads in seconds
OLLAMA_SERVER_PULL_MODELS_ENABLED=false # Auto-download models on startup
OLLAMA_SERVER_LOAD_MODELS_ENABLED=false # Load model list from server
# Proxy URL if needed
PROXY_URL=http://proxy:8080
```
### Advanced Configuration
Control how PentAGI interacts with your Ollama server:
**Model Management:**
- **Auto-pull Models** (`OLLAMA_SERVER_PULL_MODELS_ENABLED=true`): Automatically downloads models specified in config file on startup
- **Pull Timeout** (`OLLAMA_SERVER_PULL_MODELS_TIMEOUT`): Maximum time to wait for model downloads (default: 600 seconds)
- **Load Models List** (`OLLAMA_SERVER_LOAD_MODELS_ENABLED=true`): Queries Ollama server for available models via API
**Performance Note:** Enabling `OLLAMA_SERVER_LOAD_MODELS_ENABLED` adds startup latency as PentAGI queries the Ollama API. Disable if you only need specific models from config file.
**Recommended Settings:**
```bash
# Fast startup (static config)
OLLAMA_SERVER_MODEL=llama3.1:8b-instruct-q8_0
OLLAMA_SERVER_PULL_MODELS_ENABLED=false
OLLAMA_SERVER_LOAD_MODELS_ENABLED=false
# Auto-discovery (dynamic config)
OLLAMA_SERVER_PULL_MODELS_ENABLED=true
OLLAMA_SERVER_PULL_MODELS_TIMEOUT=900
OLLAMA_SERVER_LOAD_MODELS_ENABLED=true
```
## Supported Models
The provider **dynamically loads models** from your local Ollama server. Available models depend on what you have installed locally.
**Popular model families include:**
- **Gemma models**: `gemma3:1b`, `gemma3:2b`, `gemma3:7b`, `gemma3:27b`
- **Llama models**: `llama3.1:7b`, `llama3.1:8b`, `llama3.1:8b-instruct-q8_0`, `llama3.1:8b-instruct-fp16`, `llama3.1:70b`, `llama3.2:1b`, `llama3.2:3b`, `llama3.2:90b`
- **Qwen models**: `qwen2.5:1.5b`, `qwen2.5:3b`, `qwen2.5:7b`, `qwen2.5:14b`, `qwen2.5:32b`, `qwen2.5:72b`
- **DeepSeek models**: `deepseek-r1:1.5b`, `deepseek-r1:7b`, `deepseek-r1:8b`, `deepseek-r1:14b`, `deepseek-r1:32b`
- **Embedding models**: `nomic-embed-text`
To see available models on your system: `ollama list`
To download new models: `ollama pull <model-name>`
## Features
- **Dynamic model discovery**: Automatically detects models installed on your Ollama server (when enabled)
- **Model caching**: Use only configured models without API calls (when load disabled)
- **Local inference**: No API keys required, models run locally
- **Auto model pulling**: Models are automatically downloaded when needed (when enabled)
- **Agent specialization**: Different agent types (assistant, coder, pentester) with optimized settings
- **Tool support**: Supports function calling for compatible models
- **Streaming**: Real-time response streaming
- **Custom configuration**: Override default settings with YAML config files
- **Zero pricing**: Local models have no usage costs
## Agent Types
The provider supports all PentAGI agent types with optimized configurations:
- `simple`: General purpose chat (temperature: 0.2)
- `assistant`: AI assistant tasks (temperature: 0.2)
- `coder`: Code generation (temperature: 0.1, max tokens: 6000)
- `pentester`: Security testing (temperature: 0.3, max tokens: 8000)
- `generator`: Content generation (temperature: 0.4)
- `refiner`: Content refinement (temperature: 0.3)
- `searcher`: Information searching (temperature: 0.2, max tokens: 3000)
- And more...
## Custom Configuration
Create a custom config file to override default settings:
```yaml
simple:
model: "llama3.1:8b-instruct-q8_0"
temperature: 0.2
top_p: 0.3
n: 1
max_tokens: 4000
coder:
model: "deepseek-r1:8b"
temperature: 0.1
top_p: 0.2
n: 1
max_tokens: 8000
```
Then set `OLLAMA_SERVER_CONFIG_PATH` to the file path.
## Pricing
Ollama provides free local inference - no usage costs or API limits.
## Example Usage
```bash
# Set environment variables
export OLLAMA_SERVER_URL=http://localhost:11434
# Start PentAGI with Ollama provider
./pentagi
```
## Troubleshooting
1. **Connection errors**: Ensure Ollama server is running and accessible
2. **Model not found**: Pull the model first with `ollama pull <model-name>`
3. **Performance issues**: Use smaller models for faster inference or upgrade hardware
4. **Memory issues**: Monitor system memory usage with larger models
backend/docs/prompt_engineering_openai.md
+214
View File
@@ -0,0 +1,214 @@
# A Comprehensive Guide to Writing Effective Prompts for AI Agents
## Introduction
This guide provides essential principles and best practices for creating high-performing prompts for AI agent systems, with a particular focus on the latest generation models. Based on extensive research and testing, these recommendations will help you design prompts that elicit optimal AI responses across various use cases.
## Core Principles of Effective Prompt Engineering
### 1. Structure and Organization
**Clear Hierarchical Structure**
- Use meaningful sections with clear hierarchical organization (titles, subtitles)
- Start with role definition and objectives, followed by specific instructions
- Place instructions at both the beginning and end of long context prompts
- Example framework:
```
# Role and Objective
# Instructions
## Sub-categories for detailed instructions
# Reasoning Steps
# Output Format
# Examples
# Context
# Final instructions
```
**Effective Delimiters**
- Use Markdown for general purposes (titles, code blocks, lists)
- Use XML for precise wrapping of sections and nested content
- Use JSON for highly structured data, especially in coding contexts
- Avoid JSON format for large document collections
### 2. Instruction Clarity and Specificity
**Be Explicit and Unambiguous**
- Modern AI models follow instructions more literally than previous generations
- Make instructions specific, clear, and unequivocal
- Use active voice and directive language
- If behavior deviates from expectations, a single clear clarifying instruction is usually sufficient
**Provide Complete Context**
- Include all necessary information for the agent to understand the task
- Clearly define the scope and boundaries of what the agent should and should not do
- Specify any constraints or requirements for the output
### 3. Agent Workflow Guidance
**Enable Persistence and Autonomy**
- Instruct the agent to continue until the task is fully resolved
- Include explicit instructions to prevent premature termination of the process
- Example: "You are an agent - please keep going until the user's query is completely resolved, before ending your turn and yielding back to the user."
**Encourage Tool Usage**
- Direct the agent to use available tools rather than guessing or hallucinating
- Provide clear descriptions of each tool and its parameters
- Example: "If you are not sure about information pertaining to the user's request, use your tools to gather the relevant information: do NOT guess or make up an answer."
**Induce Planning**
- Prompt the agent to plan and reflect before and after each action
- Encourage step-by-step thinking and analysis
- Example: "You MUST plan extensively before each function call, and reflect extensively on the outcomes of the previous function calls."
### 4. Reasoning and Problem-Solving
**Chain-of-Thought Prompting**
- Instruct the agent to think step-by-step for complex problems
- Request explicit reasoning before arriving at conclusions
- Use phrases like "think through this carefully" or "break this down"
- Basic instruction example: "First, think carefully step by step about what is needed to answer the query."
**Structured Problem-Solving Approach**
- Guide the agent through a specific methodology:
1. Analysis: Understanding the problem and requirements
2. Planning: Creating a strategy to approach the problem
3. Execution: Performing the necessary steps
4. Verification: Checking the solution for correctness
5. Iteration: Improving the solution if needed
### 5. Output Control and Formatting
**Define Expected Output Format**
- Provide clear instructions on how the output should be structured
- Use examples to demonstrate desired formatting
- Specify any required sections, headers, or organizational elements
**Set Response Parameters**
- Define tone, style, and level of detail expected
- Specify any technical requirements (e.g., code formatting, citation style)
- Indicate whether to include explanations, summaries, or step-by-step breakdowns
## Special Considerations for Specific Use Cases
### 1. Coding and Technical Tasks
**Precise Tool Definitions**
- Use API-parsed tool descriptions rather than manual injection
- Name tools clearly to indicate their purpose
- Provide detailed descriptions in the tool's "description" field
- Keep parameter descriptions thorough but concise
- Place usage examples in a dedicated examples section
**Working with Code**
- Provide clear context about the codebase structure
- Specify the programming language and any framework requirements
- For file operations, use relative paths and specify the expected format
- For code changes, explain both what to change and why
- For diffs and patches, use context-based formats rather than line numbers
**Diff Generation Best Practices**
- Use formats that include both original and replacement code
- Provide sufficient context (3 lines before/after) to locate code precisely
- Use clear delimiters between old and new code
- For complex files, include class/method identifiers with @@ operator
### 2. Long Context Handling
**Context Size Management**
- Optimize for best performance at 1M token context window
- Be aware that performance may degrade as more items need to be retrieved
- For complex reasoning across large contexts, break tasks into smaller chunks
**Context Reliance Settings**
- Specify whether to use only provided context or blend with model knowledge
- For strict adherence to provided information: "Only use the documents in the provided External Context to answer. If you don't know the answer based on this context, respond 'I don't have the information needed to answer that'"
- For flexible approach: "By default, use the provided external context, but if other basic knowledge is needed, and you're confident in the answer, you can use some of your own knowledge"
### 3. Customer-Facing Applications
**Voice and Tone Control**
- Define the personality and communication style
- Provide sample phrases to guide tone while avoiding repetition
- Include instructions for handling difficult or prohibited topics
**Interaction Flow**
- Specify greeting and closing formats
- Detail how to maintain conversation continuity
- Include instructions for when to ask follow-up questions vs. ending the interaction
## Troubleshooting and Optimization
### Common Issues and Solutions
**Instruction Conflicts**
- Check for contradictory instructions in your prompt
- Remember that instructions placed later in the prompt may take precedence
- Ensure examples align with written rules
**Over-Compliance**
- If the agent follows instructions too rigidly, add flexibility clauses
- Include conditional statements: "If you don't have enough information, ask the user"
- Add permission to use judgment: "Use your best judgment when..."
**Repetitive Outputs**
- Instruct the agent to vary phrases and expressions
- Avoid providing exact quotes the agent might repeat
- Include diversity instructions: "Ensure responses are varied and not repetitive"
### Iterative Improvement Process
1. Start with a basic prompt following the structure guidelines
2. Test with representative examples of your use case
3. Identify patterns in suboptimal responses
4. Address specific issues with targeted instructions
5. Validate improvements through testing
6. Continue refining based on performance
## Implementation Example
Below is a sample prompt template for an AI agent tasked with prompt engineering:
```
# Role and Objective
You are a specialized AI Prompt Engineer responsible for creating and optimizing prompts that guide AI systems to perform specific tasks effectively. Your goal is to craft prompts that are clear, comprehensive, and designed to elicit optimal performance from AI models.
# Instructions
- Analyze the task requirements thoroughly before designing the prompt
- Structure prompts with clear sections and hierarchical organization
- Make instructions explicit, unambiguous, and comprehensive
- Include appropriate context and examples to guide the AI
- Specify the desired output format, style, and level of detail
- Test and refine prompts based on performance feedback
- Ensure prompts are efficient and do not contain unnecessary content
- Consider edge cases and potential misinterpretations
- Always optimize for the specific AI model being targeted
## Prompt Design Principles
- Start with clear role definition and objectives
- Use hierarchical structure with markdown headings
- Separate instructions into logical categories
- Include examples that demonstrate desired behavior
- Specify output format clearly
- End with final instructions that reinforce key requirements
# Reasoning Steps
1. Analyze the task requirements and constraints
2. Identify the critical information needed in the prompt
3. Draft the initial prompt structure following best practices
4. Review for completeness, clarity, and potential ambiguities
5. Test the prompt with sample inputs
6. Refine based on performance and feedback
# Output Format
Your output should include:
1. A complete, ready-to-use prompt
2. Brief explanation of key design choices
3. Suggestions for testing and refinement
# Final Instructions
When creating prompts, think step-by-step about how the AI will interpret and act on each instruction. Ensure all requirements are clearly specified and the prompt structure guides the AI through a logical workflow.
```
## Conclusion
Effective prompt engineering is both an art and a science. By following these guidelines and continuously refining your approach based on results, you can create prompts that consistently produce high-quality outputs from AI agent systems. Remember that the field is evolving rapidly, and staying current with best practices will help you maximize the capabilities of the latest AI models.
backend/docs/prompt_engineering_pentagi.md
+424
View File
@@ -0,0 +1,424 @@
# PentAGI Prompt Engineering Guide
A comprehensive framework for designing high-performance prompts within the PentAGI penetration testing system. This guide provides specialized principles for creating prompts that leverage the multi-agent architecture, memory systems, security tools, and specific operational context of PentAGI.
## Understanding Cognitive Aspects of Language Models
**Model Processing Fundamentals**
- Language models process information via attention mechanisms, giving higher weight to specific parts of the input.
- Position matters: Content at the beginning and end of prompts receives more attention and is processed more thoroughly.
- LLMs follow instructions more literally than humans expect; be explicit rather than implicit.
- Task decomposition improves performance: Break complex tasks into simpler, sequential steps.
- Models have no actual memory or consciousness; simulate these through explicit context and instructions.
**Priming and Contextual Influence**
- Information provided early shapes how later information is interpreted and processed.
- Set expectations clearly at the beginning to guide the model's approach to the entire task.
- Use consistent terminology throughout to avoid confusing the model with synonym switching.
- Brief examples often provide clearer guidance than lengthy explanations.
- Be aware that unintended priming can occur through choice of words, examples, or framing.
## Core Principles for PentAGI Prompts
### 1. Structure and Organization
**Clear Hierarchical Structure**
- Use Markdown headings (`#`, `##`, `###`) for clear visual hierarchy and logical grouping of instructions. Ensure a logical flow from high-level role definition to specific protocols and requirements.
- Begin with a clear definition of the agent's specific **role** (e.g., Orchestrator, Pentester, Searcher), its primary **objective** within the PentAGI workflow, and any overarching **security focus**.
- Place critical **operational constraints** (security, environment) early in the prompt for high visibility.
- Use separate, clearly marked sections for key areas:
- `CORE CAPABILITIES / KNOWLEDGE BASE`
- `OPERATIONAL ENVIRONMENT` (including `<container_constraints>`)
- `COMMAND & TOOL EXECUTION RULES` (including `<terminal_protocol>`, `<tool_usage_rules>`)
- `MEMORY SYSTEM INTEGRATION` (including `<memory_protocol>`)
- `TEAM COLLABORATION & DELEGATION` (including `<team_specialists>`, `<delegation_rules>`)
- `SUMMARIZATION AWARENESS PROTOCOL` (including `<summarized_content_handling>`)
- `EXECUTION CONTEXT` (detailing use of `{{.ExecutionContext}}`)
- `COMPLETION REQUIREMENTS`
- Ensure instructions are **specific**, **unambiguous**, use **active voice**, and are directly relevant to the agent's function within PentAGI.
**Semantic XML Delimiters**
- Use descriptive XML tags (e.g., `<container_constraints>`, `<terminal_protocol>`, `<memory_protocol>`, `<team_specialists>`, `<summarized_content_handling>`) to logically group related instructions, especially for complex protocols and constraints requiring precise adherence by the LLM.
- Maintain **consistent tag naming and structure** across all agent prompts for shared concepts (like summarization handling or team specialists) to ensure predictability.
- Use nesting appropriately (e.g., defining individual `<specialist>` tags within `<team_specialists>`). Refer to existing templates like `primary_agent.tmpl` for examples.
**Context Window Optimization**
- Prioritize information based on importance; place critical instructions at the beginning and end.
- Use compression techniques for lengthy information: summarize when possible, link to references instead of full inclusion.
- Break down extremely complex prompts into logical, manageable sections with clear transitions.
- For recurring boilerplate sections, consider using shorter references to standardized protocols.
- Use consistent formatting and avoid redundant information that consumes token space.
*Example Structure:*
```markdown
# [AGENT SPECIALIST TITLE]
[Role definition, primary objective, and security focus relevant to PentAGI]
## CORE CAPABILITIES / KNOWLEDGE BASE
[Agent-specific skills, knowledge areas relevant to PentAGI tasks]
## OPERATIONAL ENVIRONMENT
<container_constraints>...</container_constraints>
## COMMAND & TOOL EXECUTION RULES
<terminal_protocol>...</terminal_protocol>
<tool_usage_rules>...</tool_usage_rules>
## MEMORY SYSTEM INTEGRATION
<memory_protocol>...</memory_protocol>
## TEAM COLLABORATION & DELEGATION
<team_specialists>...</team_specialists>
<delegation_rules>...</delegation_rules>
## SUMMARIZATION AWARENESS PROTOCOL
<summarized_content_handling>...</summarized_content_handling>
## EXECUTION CONTEXT
[Explain how to use {{.ExecutionContext}} for Flow/Task/SubTask details]
## COMPLETION REQUIREMENTS
[Numbered list: Output format, final tool usage, language, reporting needs]
{{.ToolPlaceholder}}
```
### 2. Agent-Specific Instructions
**Role-Based Customization**
- Tailor instructions, tone, knowledge references, and complexity directly to the agent's specialized role within the PentAGI system (Orchestrator, Pentester, Searcher, Developer, Adviser, Memorist, Installer). Explicitly reference `ai-concepts.mdc` for role definitions.
- Enforce stricter command protocols and safety measures for agents with direct system/tool access (Pentester, Maintenance/Installer).
- Include references to specialized knowledge bases or toolsets relevant to the agent's function (e.g., specific security tools from `security-tools.mdc` for Pentester; search strategies and tool priorities for Searcher).
- Clearly define inter-agent communication protocols, especially delegation criteria and the expected format/content of information exchange between agents.
**Security and Operational Boundaries**
- Explicitly state the **scope** of permitted actions and **security constraints**. Reference `security-tools.mdc` for general tool security context.
- For engagement-level boundaries, start from the reusable [scope-of-work pentest prompt template](../../examples/prompts/scope_of_work_pentest.md) and adapt the allowed targets, out-of-scope targets, stop conditions, and evidence expectations before the flow starts.
- Define **Docker container limitations** within `<container_constraints>`, populated by template variables like `{{.DockerImage}}`, `{{.Cwd}}`, `{{.ContainerPorts}}`. Specify restrictions clearly (e.g., "No direct host access," "No GUI applications," "No UDP scanning").
- Specify **forbidden actions** clearly. Use **ALL CAPS** for critical security warnings, permissions, or prohibitions (e.g., "DO NOT attempt to install new software packages," "ONLY execute commands related to the current SubTask").
- Emphasize working **strictly within the scope of the current `SubTask`**. The agent must understand its current objective based on `{{.ExecutionContext}}` and not attempt actions related to other SubTasks or the overall Flow goal unless explicitly instructed within the current SubTask. Reference `data-models.mdc` and `controller.md` for task/subtask relationships.
**Ethical Boundaries and Safety**
- Explicitly include ethics guidance relevant to penetration testing context: legal compliance, responsible disclosure, data protection.
- Specify techniques for identifying and mitigating potential risks in generated prompts.
- Establish explicit guidelines for avoiding harmful outputs, jailbreaking, or prompt injection vulnerabilities.
- Include a verification step requiring agents to review outputs for potentially harmful consequences.
- Create clear escalation paths for handling edge cases requiring human judgment.
### 3. Agentic Capabilities and Persistence
**Agent Persistence Protocol**
- Include **explicit instructions** about persistence: "You are an agent - continue working until the subtask is fully completed. Do not prematurely end your turn or yield control back to the user/orchestrator until you have achieved the specific objective of your current subtask."
- Emphasize the agent's responsibility to **drive the interaction forward** autonomously and maintain momentum until a definitive result (success or failure with clear explanation) is achieved.
- Provide clear termination criteria so the agent knows precisely when its work on the subtask is considered complete.
**Planning and Reasoning**
- Instruct agents to **explicitly plan before acting**, especially for complex security operations or tool usage: "Before executing commands or invoking tools, develop a clear step-by-step plan. Think through each stage of execution, potential failure points, and contingency approaches."
- Encourage **chain-of-thought reasoning**: "When analyzing complex security issues or ambiguous results, think step-by-step through your reasoning process. Break down problems into components, consider alternatives, and justify your approach before moving to execution."
- For critical security tasks, mandate a **validation step**: "After obtaining results, verify they are correct and complete before proceeding. Cross-check findings using alternative methods when possible."
**Chain-of-Thought Engineering**
- Structure reasoning processes explicitly: problem analysis → decomposition → solution of subproblems → synthesis.
- Encourage splitting complex reasoning into discrete, traceable steps with clear transitions.
- Implement verification checkpoints throughout reasoning chains to validate intermediate conclusions.
- For complex decisions, instruct the model to evaluate multiple approaches before selecting one.
- Include prompts for explicit reflection on assumptions made during reasoning processes.
**Error Handling and Adaptation**
- Provide explicit guidance on **handling unexpected errors**: "If a command fails, do not simply repeat the same exact command. Analyze the error message, modify your approach based on the specific error, and try an alternative method if necessary."
- Define a **maximum retry threshold** (typically 3 attempts) for similar approaches before pivoting to a completely different strategy.
- Include instructions for **graceful degradation**: "If the optimal approach fails, fall back to simpler or more reliable alternatives rather than abandoning the task entirely."
**Metacognitive Processes**
- Instruct agents to periodically evaluate their own reasoning and progress toward goals.
- Include explicit steps for identifying and questioning assumptions made during problem-solving.
- Implement self-verification protocols: "After formulating a solution, critically review it for flaws or edge cases."
- Encourage steelmanning opposing viewpoints to strengthen reasoning and avoid blind spots.
- Provide mechanisms for agents to express confidence levels in their conclusions or recommendations.
### 4. Memory System Integration
**Memory Operations Protocol (`<memory_protocol>`)**
- Provide explicit, actionable instructions on *when* and *how* to interact with PentAGI's vector memory system. Reference `ai-concepts.mdc` (Memory section).
- **Crucially, specify the primary action:** Agents MUST **always attempt to retrieve relevant information from memory first** using retrieval tools (e.g., `{{.SearchGuideToolName}}`, `{{.SearchAnswerToolName}}`) *before* performing external actions like web searches or running discovery tools.
- Define clear criteria for *storing* new information: Only store valuable, novel, and reusable knowledge (e.g., confirmed vulnerabilities, successful complex command sequences, effective troubleshooting steps, reusable code snippets) using storage tools (e.g., `{{.StoreGuideToolName}}`, `{{.StoreAnswerToolName}}`). Avoid cluttering memory with trivial or intermediate results.
- Specify the exact tool names (`{{.ToolName}}`) for memory interaction.
**Vector Database Awareness**
- Guide agents on formulating effective **semantic search queries** for memory retrieval, leveraging keywords and concepts relevant to the current task context.
- If applicable, define knowledge categorization or metadata usage for more precise memory storage and retrieval (e.g., types like 'guide', 'vulnerability', 'tool_usage', 'code_snippet').
### 5. Multi-Agent Team Collaboration
**Team Specialist Definition (`<team_specialists>`)**
- Include a complete, accurate roster of **all available specialist agents** within PentAGI (searcher, pentester, developer, adviser, memorist, installer).
- For each specialist, clearly define:
- `skills`: Core competencies.
- `use_cases`: Specific situations or types of problems they should be delegated.
- `tools`: General categories of tools they utilize (not the specific invocation tool name).
- `tool_name`: The **exact tool name variable** (e.g., `{{.SearchToolName}}`, `{{.PentesterToolName}}`) used to invoke/delegate to this specialist.
- Ensure this section is consistently defined, especially in the Orchestrator prompt and any other agent prompts that allow delegation.
**Delegation Rules (`<delegation_rules>`)**
- Define clear, unambiguous criteria for *when* an agent should delegate versus attempting a task independently. A common rule is: "Attempt independent solution using your own tools/knowledge first. Delegate ONLY if the task clearly falls outside your core skills OR if a specialist agent is demonstrably better equipped to handle it efficiently and accurately."
- Mandate that **COMPREHENSIVE context** MUST be provided with every delegation request. This includes: background information, the specific objective of the delegated task, relevant data/findings gathered so far, constraints, and the expected format/content of the specialist's output.
- Instruct the delegating agent on how to handle, verify, and integrate the results received from specialists into its own workflow.
### 6. Tool-Specific Execution Rules
**Terminal Command Protocol (`<terminal_protocol>`)**
- Reinforce that commands execute within an isolated Docker container (`{{.DockerImage}}`) and that the **working directory (`{{.Cwd}}`) is NOT persistent between tool calls**.
- Mandate **explicit directory changes (`cd /path/to/dir && command`)** within a single tool call if a specific path context is required for `command`.
- Require **absolute paths** for file operations (reading, writing, listing) whenever possible to avoid ambiguity.
- Specify **timeout handling** (if controllable via parameters) and output redirection (`> file.log 2>&1`) for potentially long-running commands.
- **Limit repetition of *identical* failed commands** (e.g., maximum 3 attempts). Encourage trying variations or different approaches upon failure.
- Encourage the use of non-interactive flags (e.g., `-y`, `--assume-yes`, `--non-interactive`) where safe and appropriate to avoid hangs.
- Define when to use `detach` mode if available/applicable for background tasks.
**Tool Definition and Invocation Best Practices**
- Name tools clearly to indicate their purpose and function (e.g., `SearchGuide`, not just `Search`)
- Provide detailed yet concise descriptions in the tool's documentation
- For complex tools, include parameter examples showing proper usage
- Emphasize that **all actions MUST use structured tool calls** - the system operates exclusively through proper tool invocation
- Explicitly prohibit "simulating" or "describing" tool usage
**Search Tool Prioritization (`<search_tools>`)**
- Define an explicit **hierarchy or selection logic** for using different search tools (Internal Memory first, then potentially Browser for specific URLs, Google/DuckDuckGo for general discovery, Tavily/Perplexity/Traversaal for complex research/synthesis). Refer to `searcher.tmpl` for a good example matrix structure.
- Include tool-specific guidance (e.g., "Use `browser` tool only for accessing specific known URLs, not for general web searching," "Use `tavily` for in-depth technical research questions").
- Define **action economy rules:** Limit the total number of search tool calls per query/subtask (e.g., 3-5 max). Instruct the agent to **stop searching as soon as sufficient information is found** to fulfill the request or subtask objective. Do not exhaust all search tools unnecessarily.
**Mandatory Result Delivery Tools**
- Clearly specify the **exact final tool** (e.g., `{{.HackResultToolName}}` for Pentester, `{{.SearchResultToolName}}` for Searcher, `{{.FinalyToolName}}` for Orchestrator) that an agent **MUST** use to deliver its final output, report success/failure, and signify the completion of its current subtask.
- Define the expected structure of the output within this final tool call (e.g., "result" field contains the detailed findings/answer, "message" field contains a concise summary or status update). This signals completion to the controlling system (`controller.md`).
### 7. Context Preservation and Summarization
**Summarization Awareness Protocol (`<summarized_content_handling>`)**
- **This entire protocol section, as defined in `primary_agent.tmpl`, `pentester.tmpl`, etc., MUST be included verbatim in *all* agent prompts.**
- **Emphasize Key Points:**
- Clearly define the two forms of system-generated summaries (Tool Call Summary via `{{.SummarizationToolName}}`, Prefixed Summary via `{{.SummarizedContentPrefix}}`).
- Instruct agents to treat summaries *strictly* as **historical records of actual past events, tool executions, and their results**. They are *not* examples to be copied.
- Mandate extracting useful information from summaries (past commands, successes, failures, errors, findings) to inform current strategy and **avoid redundant actions**.
- **Strictly prohibit** agents from: mimicking summary formats, using the `{{.SummarizedContentPrefix}}`, or calling the `{{.SummarizationToolName}}` tool.
- **Reinforce:** The PentAGI system operates **exclusively via structured tool calls.** Any attempt to simulate actions or results in plain text will fail.
**Execution Context Awareness**
- Instruct agents to **actively utilize the information provided in the `{{.ExecutionContext}}` variable.**
- Explain that this variable contains structured details about the current **Flow, Task, and SubTask** (IDs, Status, Titles, Descriptions), as managed by the `controller` package (`backend/docs/controller.md`).
- Agents *must* use this context to understand their precise current objective, operational scope, relationship to parent tasks/flows, and potentially relevant history within the current operational branch.
### 8. Environment Awareness
**Container Constraints (`<container_constraints>`)**
- Clearly define the **Docker runtime environment** using template variables: `{{.DockerImage}}` (image name), `{{.Cwd}}` (working directory), `{{.ContainerPorts}}` (available ports).
- Specify **resource limitations** (e.g., default command timeouts) and **operational restrictions** derived from PentAGI's secure execution model (No GUI, No host access, No UDP scanning, No arbitrary software installation). Reference `security-tools.mdc`.
**Available Tools (`<tools>`)**
- For agents like the Pentester, explicitly **list the specific security testing tools** confirmed to be available within their container environment. Reference the list in `pentester.tmpl` and cross-check with `security-tools.mdc`.
- Provide version-specific guidance or known limitations if necessary.
## Effective Few-Shot Learning
**Example Selection and Structure**
- Include diverse, representative examples that demonstrate expected behavior across different scenarios.
- Structure examples consistently: input conditions → reasoning process → output format.
- Order examples from simple to complex to establish foundational patterns before edge cases.
- When space is limited, prioritize examples that demonstrate difficult or non-obvious aspects of the task.
- Ensure examples demonstrate all critical behaviors mentioned in the instructions.
**Example Implementation**
- Format examples using clear delimiters like XML tags, markdown blocks, or consistent headings.
- For each example, explicitly show both the process (reasoning, planning) and the outcome.
- Include examples of both successful operations and appropriate error handling.
- If possible, annotate examples with brief explanations of why specific approaches were taken.
- Ensure examples reflect the exact output format requirements.
## Handling Ambiguity and Uncertainty
**Ambiguity Resolution Strategies**
- Establish clear protocols for handling incomplete or ambiguous information.
- Define a hierarchy of information sources to consult when clarification is needed.
- Include explicit instructions for requesting additional information when necessary.
- Specify how to present multiple interpretations when a definitive answer isn't possible.
- Mandate expression of confidence levels for conclusions based on uncertain data.
**Conflict Resolution**
- Define a clear hierarchy of priorities for resolving conflicting requirements.
- Establish explicit rules for handling contradictory information from different sources.
- Include a protocol for identifying and surfacing contradictions rather than making assumptions.
- Specify when to defer to specific authorities (documentation, security policies) in case of conflicts.
- Provide a framework for transparently documenting resolution decisions when conflicts are encountered.
## Language Model Optimization
**Structured Tool Invocation is Mandatory**
- **Reiterate:** *All* actions, queries, commands, memory operations, delegations, and final result reporting **MUST** be performed via **structured tool calls** using the correct tool name variable (e.g., `{{.ToolName}}`).
- **Explicitly state:** Plain text descriptions or simulations of actions (e.g., writing "Running command `nmap -sV target.com`") **will not be executed** by the system.
- Use consistent template variables for tool names (see list below).
- Ensure prompts clearly specify expected parameters for critical tool calls.
**Completion Requirements Section**
- Always end prompts with a clearly marked section (e.g., `## COMPLETION REQUIREMENTS`) containing a **numbered list** of final instructions.
- Include a reminder about language: Respond/report in the user's/manager's preferred language (`{{.Lang}}`).
- Specify the required **final output format** and the **mandatory final tool** to use for delivery (e.g., `MUST use "{{.HackResultToolName}}" to deliver the final report`).
- **Crucially, place the `{{.ToolPlaceholder}}` variable at the very end of the prompt.** This allows the system backend to correctly inject tool definitions for the LLM.
### LLM Instruction Following Characteristics
**Modern LLM Instruction Following**
- Understand that newer LLMs (like those used in PentAGI) follow instructions **more literally and precisely** than previous generations. Make instructions explicit and unambiguous, avoiding indirect or implied guidance.
- Use **directive language** rather than suggestions: "DO X" instead of "You might want to do X" when the action is truly required.
- For critical behaviors, use **clear, unequivocal instructions** rather than lengthy explanations. A single direct statement is often more effective than paragraphs of background.
- When creating prompts, remember that if agent behavior deviates from expectations, a single clear corrective instruction is usually sufficient to guide it back on track.
**Literal Adherence vs. Intent Inference**
- Design prompts with the understanding that PentAGI agents will **follow the letter of instructions** rather than attempting to infer unstated intent.
- Make all critical behaviors explicit rather than relying on the agent to infer them from context or examples.
- If you need the agent to reason through problems rather than following a rigid process, explicitly instruct it to "think step-by-step" or "consider alternatives before deciding."
### Prompt Template Variables
**Essential Context Variables**
- Ensure prompts utilize essential context variables provided by the PentAGI backend:
- `{{.ExecutionContext}}`: **Critical.** Provides structured details (IDs, status, titles, descriptions) about the current `Flow`, `Task`, and `SubTask`. Essential for scope and objective understanding.
- `{{.Lang}}`: Specifies the preferred language for agent responses and reports.
- `{{.CurrentTime}}`: Provides the execution timestamp for context.
- `{{.DockerImage}}`: Name of the Docker image the agent operates within.
- `{{.Cwd}}`: Default working directory inside the Docker container.
- `{{.ContainerPorts}}`: Available/mapped ports within the container environment.
**Standardized Tool Name Variables**
- Use the consistent naming pattern for all tool invocation variables:
- *Specialist Invocation:*
- `{{.SearchToolName}}`
- `{{.PentesterToolName}}`
- `{{.CoderToolName}}`
- `{{.AdviceToolName}}`
- `{{.MemoristToolName}}`
- `{{.MaintenanceToolName}}`
- *Memory Operations:*
- `{{.SearchGuideToolName}}` (Retrieve Guide)
- `{{.StoreGuideToolName}}` (Store Guide)
- `{{.SearchAnswerToolName}}` (Retrieve Answer/General)
- `{{.StoreAnswerToolName}}` (Store Answer/General)
- `{{.SearchCodeToolName}}` (*Likely needed*) (Retrieve Code Snippet)
- `{{.StoreCodeToolName}}` (*Likely needed*) (Store Code Snippet)
- *Result Delivery:*
- `{{.HackResultToolName}}` (Pentester Final Report)
- `{{.SearchResultToolName}}` (Searcher Final Report)
- `{{.FinalyToolName}}` (Orchestrator Subtask Completion Report)
- *System & Environment Tools:*
- `{{.SummarizationToolName}}` (**System Use Only** - Marker for historical summaries)
- `{{.TerminalToolName}}` (*Assumed name for terminal function*)
- `{{.FileToolName}}` (*Assumed name for file operations function*)
- `{{.BrowserToolName}}` (*Assumed name for browser/scraping function*)
- *Ensure this list is kept synchronized with the actual tool names defined and passed by the backend.*
## Prompt Patterns and Anti-Patterns
**Effective Patterns**
- **Progressive Disclosure**: Introduce concepts in layers of increasing complexity.
- **Explicit Ordering**: Number steps or use clear sequence markers for sequential operations.
- **Task Decomposition**: Break complex tasks into clearly defined subtasks with their own guidelines.
- **Parameter Validation**: Include instructions for validating inputs before proceeding with operations.
- **Fallback Chains**: Define explicit alternatives when primary approaches fail.
**Common Anti-Patterns**
- **Overspecification**: Providing too many constraints that paralyze decision-making.
- **Conflicting Priorities**: Giving contradictory guidance without clear hierarchy.
- **Vague Success Criteria**: Failing to define when a task is considered complete.
- **Implicit Assumptions**: Relying on unstated knowledge or context.
- **Tool Ambiguity**: Unclear guidance on which tools to use for specific situations.
## Iterative Prompt Improvement
**Systematic Diagnosis**
- When prompts underperform, systematically isolate the issue: is it in task definition, reasoning guidance, tool usage, or output formatting?
- Document specific patterns of failure to address in revisions.
- Use controlled testing with identical inputs to validate improvements.
- Maintain version history with clear annotations about changes and their effects.
- Focus on targeted, minimal changes rather than wholesale rewrites when refining.
**Improvement Metrics**
- Define objective success criteria for prompt performance before making changes.
- Measure improvements across specific dimensions: accuracy, completeness, efficiency, robustness.
- Test prompts against edge cases and unusual inputs to ensure generalizability.
- Compare performance across different LLM providers to ensure consistency.
- Document both successful and unsuccessful prompt modifications to build institutional knowledge.
## Multimodal Integration
**Text-Visual Integration**
- When referencing visual elements, use precise descriptive language and spatial relationships.
- Define protocols for describing and referencing images, diagrams, or visualizations.
- For security-relevant visual information, instruct agents to extract and document specific details systematically.
- Establish clear formats for describing visual evidence in reports and documentation.
- Include guidance on when to request visual confirmation versus relying on textual descriptions.
## Agent-Specific Guidelines Summary
### Primary Agent (Orchestrator)
- **Focus**: Task decomposition, delegation orchestration, context management across subtasks, final subtask result aggregation.
- **Key Sections**: `TEAM CAPABILITIES`, `OPERATIONAL PROTOCOLS` (esp. Task Analysis, Boundaries, Delegation Efficiency), `DELEGATION PROTOCOL`, `SUMMARIZATION AWARENESS PROTOCOL`, `COMPLETION REQUIREMENTS` (using `{{.FinalyToolName}}`).
- **Critical Instructions**: Gather context *before* delegating, strictly enforce current subtask scope, provide *full* context upon delegation, manage execution attempts/failures, report subtask completion status and comprehensive results using `{{.FinalyToolName}}`.
### Pentester Agent
- **Focus**: Hands-on security testing, execution of tools (`nmap`, `sqlmap`, etc.), vulnerability exploitation, evidence collection and documentation.
- **Key Sections**: `KNOWLEDGE MANAGEMENT` (Memory Protocol), `OPERATIONAL ENVIRONMENT` (Container Constraints), `COMMAND EXECUTION RULES` (Terminal Protocol), `PENETRATION TESTING TOOLS` (list available), `TEAM COLLABORATION`, `DELEGATION PROTOCOL`, `SUMMARIZATION AWARENESS PROTOCOL`, `COMPLETION REQUIREMENTS` (using `{{.HackResultToolName}}`).
- **Critical Instructions**: Check memory first, strictly adhere to terminal rules & container constraints, use only listed available tools, delegate appropriately (e.g., exploit development to Coder), provide detailed, evidence-backed exploitation reports using `{{.HackResultToolName}}`.
#### Pentesting Methodology Checklist for Prompt Authors
- Encode authorization boundaries explicitly. Prompts should remind the agent to test only approved targets, respect engagement scope, and avoid destructive actions unless the task requires them.
- Start with coverage before exploitation. Instruct the agent to map routes, roles, inputs, file handling, integrations, and trust boundaries before choosing attack paths.
- Organize testing by attack surface. Good prompts group checks around authentication, access control, injection, cross-site scripting, server-side request forgery, file processing, and business logic instead of presenting a random payload dump.
- Prefer low-risk validation first. Reflection markers, controlled payloads, timing checks, and out-of-band verification should be used deliberately to confirm hypotheses before deeper exploitation.
- Require evidence at every stage. Prompts should ask for captured requests, responses, tool output, prerequisites, and impact notes so confirmed findings can move directly into a report.
- Use memory and iteration intentionally. The agent should record confirmed dead ends, revisit promising leads with new context, and avoid repeating the same failed checks.
- End with actionable reporting. A strong pentesting prompt tells the agent to summarize what was confirmed, what remains unverified, how the issue can be reproduced, and which follow-up actions are justified.
#### Recommended Reference Material
- Use public methodology resources such as [HackTricks](https://book.hacktricks.wiki/en/index.html) and [Pentest Book](https://pentestbook.six2dez.com/) as inspiration for attack-surface coverage and testing depth.
- Translate those references into concise phases, priorities, and verification rules for the agent instead of copying long checklists into the system prompt verbatim.
- Keep prompt examples aligned with live PentAGI assets such as [`backend/pkg/templates/prompts/pentester.tmpl`](../pkg/templates/prompts/pentester.tmpl) and [`examples/prompts/base_web_pentest.md`](../../examples/prompts/base_web_pentest.md).
### Searcher Agent
- **Focus**: Highly efficient information retrieval (internal memory & external sources), source evaluation and prioritization, synthesis of findings.
- **Key Sections**: `CORE CAPABILITIES` (Action Economy, Search Optimization), `SEARCH TOOL DEPLOYMENT MATRIX`, `OPERATIONAL PROTOCOLS` (Search Efficiency, Query Engineering), `SUMMARIZATION AWARENESS PROTOCOL`, `SEARCH RESULT DELIVERY` (using `{{.SearchResultToolName}}`).
- **Critical Instructions**: **Always prioritize memory search** (`{{.SearchAnswerToolName}}`), strictly limit the number of search actions, use the right tool for the query complexity (Matrix), **stop searching once sufficient information is gathered**, deliver concise yet comprehensive synthesized results via `{{.SearchResultToolName}}`.
*(Guidelines for Developer, Adviser, Memorist, Installer agents should be developed following this structure, focusing on their unique roles, tools, and interactions based on their specific implementations and prompt templates).*
## Prompt Maintenance and Evolution
### Version Control and Documentation
- Store all prompt templates consistently within the `backend/pkg/templates/prompts/` directory.
- Use a clear and consistent naming pattern: `<agent_role>[_optional_specifier].tmpl`.
- Include version information or brief changelog comments within the templates themselves or in associated documentation.
- Document the purpose, expected template variables (`{{.Variable}}`), and the general input/output behavior for each prompt template. Ensure this documentation stays synchronized with the backend code that populates the variables.
### Testing and Refinement
- Utilize the `ctester` utility (`backend/cmd/ctester/`) for validating LLM provider compatibility and basic prompt adherence (e.g., JSON formatting, function calling capabilities) for different agent types. Reference `development-workflow.mdc` / `README.md`.
- Employ the `ftester` utility (`backend/cmd/ftester/`) for **in-depth testing** of specific agent functions and prompt behaviors within realistic contexts (Flow/Task/SubTask). This is crucial for debugging complex interactions and prompt logic.
- Actively analyze agent performance, errors, and interaction traces using observability tools like **Langfuse**. Identify patterns where prompts are misunderstood, lead to inefficient actions, or violate protocols.
- Refine prompts iteratively based on `ctester`, `ftester`, and Langfuse analysis. Test changes thoroughly before deployment.
- Verify prompt changes across different supported LLM providers to ensure consistent behavior.
- Regularly validate that XML structures are well-formed and consistently applied across prompts.
### Prompt Evolution Workflow
- Document successful vs. unsuccessful prompt patterns to build institutional knowledge
- Identify areas where agents commonly misunderstand instructions or violate protocols
- Focus refinement efforts on critical sections with highest impact on performance
- Test prompt changes systematically with controlled variables
- When adding new agent types or specializations, adapt existing templates rather than creating entirely new structures
### Prompt Debugging Guide
- When agents act incorrectly, first check: Are instructions contradictory? Are priorities clear? Is context sufficient?
- For reasoning failures, examine if the problem has been properly decomposed and if verification steps exist.
- For tool usage errors, verify tool descriptions and examples are clear and parameters well-defined.
- When memory usage is suboptimal, check memory protocol clarity and retrieval/storage guidance.
- Document common failure modes to address in future prompt revisions.
## Implementation Examples
*(Refer to the actual, up-to-date files in `backend/pkg/templates/prompts/` such as `primary_agent.tmpl`, `pentester.tmpl`, and `searcher.tmpl` for concrete implementation patterns that follow these guidelines.)*