botpress--botpress/packages/llmz/DOCS.md

LLMz Documentation

LLMz: A Revolutionary TypeScript AI Agent Framework

Stop chaining tools. Start generating real code.

---

Table of Contents

1. Introduction
2. Core Philosophy
3. Quick Start
4. Core Concepts
5. Execution Modes
6. Tools
7. Objects and Variables
8. Execution Results
9. Hooks System
10. Advanced Features
11. API Reference
12. Examples

---

Introduction

LLMz is a revolutionary TypeScript AI agent framework that fundamentally changes how AI agents work. Like other agent frameworks, LLMz calls LLM models in a loop to achieve desired outcomes with access to tools and memory. However, LLMz is code-first – meaning it generates and runs TypeScript code in a sandbox rather than using traditional JSON tool calling.

What Makes LLMz Different

Traditional agent frameworks rely on JSON tool calling, which has significant limitations:

• Hard-to-parse JSON schemas for LLMs
• Incapable of complex logic like loops and conditionals
• Multiple expensive roundtrips for each tool call
• Unreliable beyond simple scenarios

LLMz leverages the fact that models have been trained extensively on millions of TypeScript codebases, making them incredibly reliable at generating working code. This enables:

• Complex logic and multi-tool orchestration in one call
• Native LLM thinking via comments and code structure
• Complete type safety and predictable schemas
• Seamless scaling in production environments

Battle-Tested at Scale

LLMz operates as an LLM-native TypeScript VM built on top of Zui (Botpress's internal schema library), battle-tested in production powering millions of AI agents worldwide.

---

Core Philosophy

Code Generation > Tool Calling

Traditional tool-calling agents are fundamentally limited by the JSON interface between the LLM and tools. This requires multiple roundtrips for complex tasks and cannot handle conditional logic, loops, or sophisticated data processing.

LLMz solves this by letting LLMs do what they do best: generate code. Since models are trained extensively on code, they can reliably generate TypeScript that:

• Calls multiple tools in sequence
• Handles conditional logic and error cases
• Processes and transforms data between tool calls
• Implements complex business logic
• Maintains type safety throughout execution

Example: Traditional vs LLMz

Traditional Tool Calling:

LLM → JSON: {"tool": "getPrice", "params": {"from": "quebec", "to": "new york"}}
System → Response: {"price": 600}
LLM → JSON: {"tool": "checkBudget", "params": {"amount": 600}}
System → Response: {"canAfford": false}
LLM → JSON: {"tool": "notifyUser", "params": {"message": "Price too high"}}

LLMz Code Generation:

// Check the ticket price and user's budget in one go
const price = await getTicketPrice({ from: 'quebec', to: 'new york' })
const budget = await getUserBudget()

if (price > budget) {
  await notifyUser({ message: `Price $${price} exceeds budget $${budget}` })
  return { action: 'budget_exceeded', price, budget }
} else {
  const ticketId = await buyTicket({ from: 'quebec', to: 'new york' })
  return { action: 'done', result: ticketId }
}

---

Quick Start

Installation

npm install llmz @botpress/client

Basic Example (Worker Mode)

import { execute } from 'llmz'
import { Client } from '@botpress/client'

const client = new Client({
  botId: process.env.BOTPRESS_BOT_ID!,
  token: process.env.BOTPRESS_TOKEN!,
})

const result = await execute({
  instructions: 'Calculate the sum of numbers 1 to 100',
  client,
})

if (result.isSuccess()) {
  console.log('Result:', result.output)
  console.log('Generated code:', result.iteration.code)
}

Basic Example (Chat Mode)

import { execute } from 'llmz'
import { Client } from '@botpress/client'
import { CLIChat } from './utils/cli-chat'

const client = new Client({
  botId: process.env.BOTPRESS_BOT_ID!,
  token: process.env.BOTPRESS_TOKEN!,
})

const chat = new CLIChat()

while (await chat.iterate()) {
  await execute({
    instructions: 'You are a helpful assistant',
    chat,
    client,
  })
}

---

Core Concepts

Execution Loop

At its core, LLMz exposes a single method (execute) that runs in a loop until one of these conditions is met:

1. An Exit is returned - Agent completes with structured result
2. Agent waits for user input (Chat Mode) - Returns control to user
3. Maximum iterations reached - Safety limit to prevent infinite loops

The loop automatically handles:

• Tool calling and result processing
• Thinking about outputs and context
• Error recovery and retry logic
• Variable state persistence across iterations

Generated Code Structure

Every LLMz code block follows a predictable structure that LLMs can reliably generate:

Return Statement (Required)

At minimum, an LLMz response must contain a return statement with an Exit:

// Chat mode - give turn back to user
return { action: 'listen' }

// Worker mode - complete with result
return { action: 'done', result: calculatedValue }

Tool Calls with Logic

Unlike traditional tool calling, LLMz enables complex logic impossible with JSON:

// Complex conditional logic and error handling
const price = await getTicketPrice({ from: 'quebec', to: 'new york' })

if (price > 500) {
  throw new Error('Price too high')
} else {
  const ticketId = await buyTicket({ from: 'quebec', to: 'new york' })
  return { action: 'done', result: ticketId }
}

Comments for Planning

Comments help LLMs think step-by-step and plan ahead:

// Check user's budget first before proceeding with purchase
const budget = await getUserBudget()

// Only proceed if we have enough funds
if (budget >= price) {
  // Purchase the ticket
  const ticket = await buyTicket(ticketDetails)
}

React Components (Chat Mode Only)

In Chat Mode, agents can yield React components for rich user interaction:

// Multi-line text support
yield <Text>
Hello, world!
This is a second line.
</Text>

// Composed/nested components
yield <Message>
  <Text>What do you prefer?</Text>
  <Button>Cats</Button>
  <Button>Dogs</Button>
</Message>

return { action: 'listen' }

Compilation Pipeline

LLMz uses a sophisticated Babel-based compilation system to transform generated code:

1. AST Parsing: TypeScript/JSX code parsed into Abstract Syntax Tree
2. Plugin Transformation: Custom plugins modify the AST for execution
3. Code Generation: Modified AST compiled back to executable JavaScript
4. Source Maps: Generated for debugging and error tracking

Key transformations include:

• Tool call instrumentation for monitoring
• Variable extraction and tracking
• JSX component handling
• Line number preservation for stack traces

Virtual Machine Execution

LLMz supports multiple execution environments:

• Production: Uses isolated-vm for security isolation
• CI/Development: Falls back to Node.js VM for compatibility
• Browser: Uses standard JavaScript execution

The VM provides:

• Memory isolation and limits
• Execution timeouts
• Secure context separation
• Stack trace sanitization

---

Execution Modes

LLMz operates in two distinct modes depending on whether a chat interface is provided:

Chat Mode

Enabled when: chat parameter is provided to execute()

Chat Mode is designed for interactive conversational agents that need to:

• Maintain conversation history
• Respond to user messages
• Yield UI components for rich interaction
• Handle turn-taking between agent and user

Key characteristics:

• Agent can yield React components to user
• Special ListenExit automatically available
• Transcript management for conversation history
• Turn-based execution flow

const result = await execute({
  instructions: 'You are a helpful assistant',
  chat: myChatInstance,
  tools: [searchTool, calculatorTool],
  client,
})

if (result.is(ListenExit)) {
  // Agent is waiting for user input
}

Worker Mode

Enabled when: chat parameter is omitted from execute()

Worker Mode is designed for automated execution environments that need to:

• Process data and perform computations
• Execute multi-step workflows
• Return structured results
• Run without human interaction

Key characteristics:

• Focus on computational tasks and data processing
• Uses DefaultExit if no custom exits provided
• Sandboxed execution with security isolation
• Automated completion without user interaction

const result = await execute({
  instructions: 'Process the customer data and generate insights',
  tools: [dataProcessorTool, analyticseTool],
  exits: [dataProcessedExit],
  client,
})

if (result.is(dataProcessedExit)) {
  console.log('Analysis complete:', result.output)
}

Mode Comparison

Feature	Chat Mode	Worker Mode
User Interaction	✅ Interactive	❌ Automated
UI Components	✅ React components	❌ No UI
Conversation History	✅ Full transcript	❌ No history
Default Exits	ListenExit	DefaultExit
Primary Use Case	Conversational AI	Data processing
Execution Pattern	Turn-based	Continuous

---

Tools

Tools are the primary way to extend LLMz agents with external capabilities. Unlike traditional agent frameworks, LLMz tools are called through generated TypeScript code, enabling complex orchestration and error handling.

Tool Definition

Tools are defined using Zui schemas for complete type safety:

import { Tool } from 'llmz'
import { z } from '@bpinternal/zui'

const weatherTool = new Tool({
  name: 'getWeather',
  description: 'Get current weather for a location',
  input: z.object({
    location: z.string().describe('City name or coordinates'),
    units: z.enum(['celsius', 'fahrenheit']).optional().default('celsius'),
  }),
  output: z.object({
    temperature: z.number(),
    conditions: z.string(),
    humidity: z.number(),
  }),
  handler: async ({ location, units }) => {
    // Implementation here
    return {
      temperature: 22,
      conditions: 'sunny',
      humidity: 65,
    }
  },
})

Tool Usage in Generated Code

The LLM generates TypeScript code that calls tools naturally:

// Simple tool call
const weather = await getWeather({ location: 'New York' })

// Complex logic with multiple tools
const weather = await getWeather({ location: userLocation })
if (weather.temperature < 0) {
  const clothing = await getSuggestions({ type: 'winter', temperature: weather.temperature })
  yield <Text>It's {weather.temperature}°C! {clothing.suggestion}</Text>
} else {
  yield <Text>Nice weather! {weather.conditions} at {weather.temperature}°C</Text>
}

return { action: 'listen' }

Advanced Tool Features

Tool Aliases

Tools can have multiple names for flexible calling:

const tool = new Tool({
  name: 'calculatePrice',
  aliases: ['getPrice', 'checkCost'],
  // ... rest of definition
})

// All of these work in generated code:
// await calculatePrice(params)
// await getPrice(params)
// await checkCost(params)

Static Inputs

Force specific inputs to be always included:

const tool = new Tool({
  name: 'logEvent',
  input: z.object({
    event: z.string(),
    userId: z.string(),
    timestamp: z.number(),
  }),
  staticInputs: {
    userId: 'user-123',
    timestamp: () => Date.now(), // Dynamic static input
  },
  handler: async ({ event, userId, timestamp }) => {
    // userId and timestamp are automatically provided
  },
})

Tool Wrapping and Cloning

Clone and modify existing tools:

const originalTool = new Tool({
  /* definition */
})

const wrappedTool = originalTool.clone({
  name: 'wrappedVersion',
  description: 'Enhanced version with logging',
  handler: async (input) => {
    console.log('Tool called with:', input)
    const result = await originalTool.execute(input)
    console.log('Tool returned:', result)
    return result
  },
})

Tool Type Generation

Use tool.getTypings() to see the TypeScript definitions generated for the LLM:

console.log(weatherTool.getTypings())
// Output:
// /**
//  * Get current weather for a location
//  */
// declare function getWeather(input: {
//   location: string; // City name or coordinates
//   units?: "celsius" | "fahrenheit";
// }): Promise<{
//   temperature: number;
//   conditions: string;
//   humidity: number;
// }>;

Best Practices

1. Descriptive Schemas: Detailed descriptions help LLMs generate better code
2. Type Safety: Use strict Zui schemas for predictable behavior
3. Error Handling: Tools should handle errors gracefully
4. Performance: Keep tool execution fast to avoid timeouts
5. Documentation: Clear descriptions improve code generation quality

---

Objects and Variables

Objects in LLMz provide namespaced containers for related tools and variables, enabling sophisticated state management and data organization.

Object Definition

Objects group related functionality and provide scoped variables:

import { ObjectInstance } from 'llmz'
import { z } from '@bpinternal/zui'

const userObject = new ObjectInstance({
  name: 'user',
  properties: [
    {
      name: 'name',
      value: 'John Doe',
      writable: true,
      type: z.string(),
    },
    {
      name: 'age',
      value: 30,
      writable: false, // Read-only
      type: z.number(),
    },
    {
      name: 'preferences',
      value: { theme: 'dark', language: 'en' },
      writable: true,
      type: z.object({
        theme: z.enum(['light', 'dark']),
        language: z.string(),
      }),
    },
  ],
  tools: [
    new Tool({
      name: 'updateProfile',
      input: z.object({ name: z.string() }),
      handler: async ({ name }) => {
        // This tool is scoped to the user object
        return { success: true }
      },
    }),
  ],
})

Variables in Generated Code

The LLM can read and write object properties in generated code:

// Reading variables
const userName = user.name // "John Doe"
const userAge = user.age // 30

// Writing to writable variables
user.name = 'Jane Smith' // ✅ Succeeds
user.preferences = { theme: 'light', language: 'es' } // ✅ Succeeds

// Attempting to write read-only variables
user.age = 25 // ❌ Throws AssignmentError

Type Safety and Validation

Variables are validated against their schemas:

// Valid assignment
user.preferences = { theme: 'dark', language: 'fr' } // ✅

// Invalid assignment - wrong type
user.preferences = { theme: 'blue', language: 'fr' } // ❌ Throws validation error

// Invalid assignment - missing required fields
user.preferences = { theme: 'dark' } // ❌ Missing language field

Mutation Tracking

LLMz automatically tracks changes to object properties:

// In generated code
user.name = 'Updated Name'
user.preferences.theme = 'light'

// After execution, mutations are available
console.log(result.iteration.mutations)
// [
//   {
//     object: 'user',
//     property: 'name',
//     before: 'John Doe',
//     after: 'Updated Name'
//   },
//   {
//     object: 'user',
//     property: 'preferences',
//     before: { theme: 'dark', language: 'en' },
//     after: { theme: 'light', language: 'en' }
//   }
// ]

Namespaced Tools

Tools within objects are called with object namespace:

// Tool is scoped to the user object
await user.updateProfile({ name: 'New Name' })

// This automatically updates the user object's properties
// and is tracked as a mutation

Object Sealing and Protection

Objects are automatically sealed to prevent unauthorized modifications:

// In generated code - these will throw errors
user.newProperty = 'value' // ❌ Cannot add new properties
delete user.name // ❌ Cannot delete properties

// Only predefined properties can be modified (if writable)
user.name = 'New Name' // ✅ Allowed if writable: true

Variable Persistence

Variables persist across iterations and thinking cycles:

// Iteration 1: Set a variable
user.preferences = { theme: 'dark', language: 'es' }
return { action: 'think' } // Trigger thinking

// Iteration 2: Variable is still available
const currentTheme = user.preferences.theme // 'dark'

Best Practices

1. Meaningful Names: Use descriptive object and property names
2. Appropriate Scope: Group related functionality together
3. Write Protection: Mark properties as read-only when appropriate
4. Type Safety: Use strict schemas for predictable behavior
5. Mutation Tracking: Leverage mutation tracking for audit trails

---

Execution Results

Every call to execute() returns an ExecutionResult that provides type-safe access to the execution outcome. LLMz execution can result in three different types of results.

Result Types

SuccessExecutionResult

Agent completed successfully with an Exit. Contains the structured data produced by the agent.

const result = await execute({
  instructions: 'Calculate the sum',
  client,
})

if (result.isSuccess()) {
  console.log('Output:', result.output)
  console.log('Exit used:', result.exit.name)
  console.log('Generated code:', result.iteration.code)
}

ErrorExecutionResult

Execution failed with an unrecoverable error:

if (result.isError()) {
  console.error('Error:', result.error)
  console.error('Failed iteration:', result.iteration?.error)

  // Analyze failure progression
  result.iterations.forEach((iter, i) => {
    console.log(`Iteration ${i + 1}: ${iter.status.type}`)
  })
}

PartialExecutionResult

Execution was interrupted by a SnapshotSignal for pauseable operations:

if (result.isInterrupted()) {
  console.log('Interrupted:', result.signal.message)

  // Save snapshot for later resumption
  const serialized = result.snapshot.toJSON()
  await database.saveSnapshot(serialized)
}

Type-Safe Exit Checking

Use result.is(exit) for type-safe access to specific exit data:

const successExit = new Exit({
  name: 'success',
  schema: z.object({
    recordsProcessed: z.number(),
    processingTime: z.number(),
  }),
})

const errorExit = new Exit({
  name: 'error',
  schema: z.object({
    errorCode: z.string(),
    details: z.string(),
  }),
})

const result = await execute({
  instructions: 'Process the data',
  exits: [successExit, errorExit],
  client,
})

// Type-safe exit handling with automatic output typing
if (result.is(successExit)) {
  // TypeScript knows result.output has the success schema
  console.log(`Processed ${result.output.recordsProcessed} records`)
  console.log(`Processing took ${result.output.processingTime}ms`)
} else if (result.is(errorExit)) {
  // TypeScript knows result.output has the error schema
  console.error(`Error ${result.output.errorCode}: ${result.output.details}`)
}

Built-in Exits

import { ListenExit, DefaultExit, ThinkExit } from 'llmz'

// Check for built-in exits
if (result.is(ListenExit)) {
  console.log('Agent is waiting for user input')
}

if (result.is(DefaultExit)) {
  // DefaultExit has success/failure discriminated union
  if (result.output.success) {
    console.log('Completed successfully:', result.output.result)
  } else {
    console.error('Completed with error:', result.output.error)
  }
}

if (result.is(ThinkExit)) {
  console.log('Agent requested thinking time')
  console.log('Current variables:', result.output.variables)
}

Accessing Execution Details

Iterations and Execution Flow

// Access the final iteration
const lastIteration = result.iteration
if (lastIteration) {
  console.log('Generated code:', lastIteration.code)
  console.log('Status:', lastIteration.status.type)
  console.log('Duration:', lastIteration.duration)
}

// Access all iterations to see full execution flow
result.iterations.forEach((iteration, index) => {
  console.log(`Iteration ${index + 1}:`)
  console.log('  Status:', iteration.status.type)
  console.log('  Code length:', iteration.code?.length || 0)
  console.log('  Variables:', Object.keys(iteration.variables).length)
})

Variables and Declarations

// If agent generates: const hello = '1234'
const lastIteration = result.iteration
if (lastIteration) {
  console.log(lastIteration.variables.hello) // '1234'

  // Access all variables from the final iteration
  Object.entries(lastIteration.variables).forEach(([name, value]) => {
    console.log(`Variable ${name}:`, value)
  })
}

Tool Calls and Traces

// Access tool calls from all iterations
const allToolCalls = result.iterations.flatMap((iter) => iter.traces.filter((trace) => trace.type === 'tool_call'))

console.log('Total tool calls:', allToolCalls.length)

// Access other trace types
const lastIteration = result.iteration
if (lastIteration) {
  const yields = lastIteration.traces.filter((trace) => trace.type === 'yield')
  const comments = lastIteration.traces.filter((trace) => trace.type === 'comment')
  const propertyAccess = lastIteration.traces.filter((trace) => trace.type === 'property')
}

Context and Metadata

if (result.isSuccess()) {
  // Access original execution parameters
  console.log('Instructions:', result.context.instructions)
  console.log('Loop limit:', result.context.loop)
  console.log('Temperature:', result.context.temperature)
  console.log('Model:', result.context.model)

  // Access tools and exits that were available
  console.log(
    'Available tools:',
    result.context.tools?.map((t) => t.name)
  )
  console.log(
    'Available exits:',
    result.context.exits?.map((e) => e.name)
  )
}

Error Analysis

if (result.isError()) {
  console.error('Execution failed:', result.error)

  // Analyze the failure progression
  const failedIteration = result.iteration
  if (failedIteration) {
    switch (failedIteration.status.type) {
      case 'execution_error':
        console.error('Code execution failed:', failedIteration.status.execution_error.message)
        console.error('Stack trace:', failedIteration.status.execution_error.stack)
        console.error('Failed code:', failedIteration.code)
        break

      case 'generation_error':
        console.error('LLM generation failed:', failedIteration.status.generation_error.message)
        break

      case 'invalid_code_error':
        console.error('Invalid code generated:', failedIteration.status.invalid_code_error.message)
        console.error('Invalid code:', failedIteration.code)
        break

      case 'aborted':
        console.error('Execution aborted:', failedIteration.status.aborted.reason)
        break
    }
  }

  // Review all iterations to understand failure progression
  console.log('Iterations before failure:', result.iterations.length)
  result.iterations.forEach((iter, i) => {
    console.log(`Iteration ${i + 1}: ${iter.status.type}`)
  })
}

Snapshot Handling

Handle interrupted executions with snapshot resumption:

const result = await execute({
  instructions: 'Process large dataset with pauseable operation',
  tools: [snapshotCapableTool],
  client,
})

if (result.isInterrupted()) {
  console.log('Execution paused:', result.signal.message)

  // Serialize snapshot for persistence
  const serialized = result.snapshot.toJSON()
  await database.saveSnapshot('execution-123', serialized)

  // Later, resume from snapshot
  const snapshot = Snapshot.fromJSON(serialized)
  snapshot.resolve({ resumeData: 'Operation completed' })

  const continuation = await execute({
    snapshot,
    instructions: result.context.instructions,
    tools: result.context.tools,
    exits: result.context.exits,
    client,
  })

  if (continuation.isSuccess()) {
    console.log('Resumed execution completed:', continuation.output)
  }
}

---

Hooks System

LLMz provides a comprehensive hook system that allows you to inject custom logic at various points during execution. Hooks are categorized as either blocking (execution waits) or non-blocking, and either mutation (can modify data) or non-mutation.

Hook Types Overview

Hook	Blocking	Mutation	Called When
onTrace	❌	❌	Each trace generated
onIterationEnd	✅	❌	After iteration completion
onExit	✅	❌	When exit is reached
onBeforeExecution	✅	✅	Before code execution
onBeforeTool	✅	✅	Before tool execution
onAfterTool	✅	✅	After tool execution

onTrace (Non-blocking, Non-mutation)

Called for each trace generated during iteration. Useful for logging, debugging, or monitoring execution progress.

await execute({
  onTrace: ({ trace, iteration }) => {
    console.log(`Iteration ${iteration}: ${trace.type}`, trace)

    // Log specific trace types
    if (trace.type === 'tool_call') {
      console.log(`Tool ${trace.tool_name} called with:`, trace.input)
    }
  },
  // ... other props
})

Available Trace Types:

• abort_signal: Abort signal received
• comment: Comment found in generated code
• llm_call_success: LLM generation completed successfully
• property: Object property accessed or modified
• think_signal: ThinkSignal thrown
• tool_call: Tool executed
• yield: Component yielded in chat mode
• log: General logging event

onIterationEnd (Blocking, Non-mutation)

Called after each iteration ends, regardless of status. Useful for logging, cleanup, or controlling iteration timing.

await execute({
  onIterationEnd: async (iteration, controller) => {
    console.log(`Iteration ${iteration.id} ended with status: ${iteration.status.type}`)

    // Add delays, cleanup, or conditional logic
    if (iteration.status.type === 'execution_error') {
      await logError(iteration.error)

      // Add delay before retry
      await new Promise((resolve) => setTimeout(resolve, 1000))
    }

    // Can use controller to abort execution if needed
    if (shouldAbort(iteration)) {
      controller.abort('Custom abort reason')
    }
  },
  // ... other props
})

onExit (Blocking, Non-mutation)

Called when an exit is reached. Useful for logging, notifications, or implementing guardrails by throwing errors to prevent exit.

await execute({
  onExit: async (result) => {
    console.log(`Exiting with: ${result.exit.name}`, result.result)

    // Implement guardrails
    if (result.exit.name === 'approve_loan' && result.result.amount > 10000) {
      throw new Error('Manager approval required for loans over $10,000')
    }

    // Send notifications
    await notifyStakeholders(result)

    // Log to audit trail
    await auditLog.record({
      action: result.exit.name,
      data: result.result,
      timestamp: Date.now(),
    })
  },
  // ... other props
})

onBeforeExecution (Blocking, Mutation)

Called after LLM generates code but before execution. Allows code modification and guardrails implementation.

await execute({
  onBeforeExecution: async (iteration, controller) => {
    console.log('Generated code:', iteration.code)

    // Code modification
    if (iteration.code?.includes('dangerousOperation')) {
      return {
        code: iteration.code.replace('dangerousOperation', 'safeOperation'),
      }
    }

    // Guardrails - throw to prevent execution
    if (iteration.code?.includes('forbidden')) {
      throw new Error('Forbidden operation detected')
    }

    // Add security checks
    const securityIssues = await scanCodeForSecurity(iteration.code)
    if (securityIssues.length > 0) {
      throw new Error(`Security issues found: ${securityIssues.join(', ')}`)
    }

    // Log code generation for audit
    await auditCodeGeneration(iteration.code)
  },
  // ... other props
})

onBeforeTool (Blocking, Mutation)

Called before any tool execution. Allows input modification and tool execution control.

await execute({
  onBeforeTool: async ({ iteration, tool, input, controller }) => {
    console.log(`Executing tool: ${tool.name}`, input)

    // Input modification
    if (tool.name === 'sendEmail') {
      return {
        input: {
          ...input,
          subject: `[Automated] ${input.subject}`, // Add prefix
          from: 'noreply@company.com', // Override sender
        },
      }
    }

    // Access control
    if (tool.name === 'deleteFile' && !hasPermission(input.path)) {
      throw new Error('Insufficient permissions to delete file')
    }

    // Rate limiting
    const rateLimit = await checkRateLimit(tool.name)
    if (rateLimit.exceeded) {
      throw new Error(`Rate limit exceeded for ${tool.name}`)
    }

    // Validation
    await validateToolUsage(tool, input)
  },
  // ... other props
})

onAfterTool (Blocking, Mutation)

Called after tool execution. Allows output modification and post-processing.

await execute({
  onAfterTool: async ({ iteration, tool, input, output, controller }) => {
    console.log(`Tool ${tool.name} completed`, { input, output })

    // Output modification
    if (tool.name === 'fetchUserData') {
      return {
        output: {
          ...output,
          // Remove sensitive data before LLM sees it
          ssn: undefined,
          creditCard: undefined,
          // Add metadata
          fetchedAt: Date.now(),
        },
      }
    }

    // Result enhancement
    if (tool.name === 'calculatePrice') {
      return {
        output: {
          ...output,
          currency: 'USD',
          timestamp: Date.now(),
          exchangeRate: await getCurrentExchangeRate(),
        },
      }
    }

    // Logging and caching
    await Promise.all([
      cacheResult(tool.name, input, output),
      logToolExecution(tool.name, input, output),
      updateMetrics(tool.name, Date.now() - tool.startTime),
    ])
  },
  // ... other props
})

Hook Execution Order

For each iteration:

1. onTrace: Throughout execution (non-blocking)
2. onBeforeExecution: After code generation, before execution
3. onBeforeTool: Before each tool call
4. onAfterTool: After each tool call
5. onExit: When exit is reached
6. onIterationEnd: After iteration completes

Advanced Hook Patterns

Conditional Hook Logic

await execute({
  onBeforeTool: async ({ tool, input }) => {
    // Apply different logic based on tool
    switch (tool.name) {
      case 'payment':
        return await handlePaymentValidation(input)
      case 'notification':
        return await handleNotificationThrottling(input)
      default:
        return // No modification
    }
  },
})

Error Recovery in Hooks

await execute({
  onExit: async (result) => {
    try {
      await criticalPostProcessing(result)
    } catch (error) {
      // Log error but don't fail the entire execution
      console.error('Post-processing failed:', error)

      // Optionally throw to retry the iteration
      if (error.retryable) {
        throw new Error('Retrying due to recoverable error')
      }
    }
  },
})

Hook State Management

let executionMetrics = { toolCalls: 0, totalTime: 0 }

await execute({
  onBeforeTool: async ({ tool }) => {
    executionMetrics.toolCalls++
    tool.startTime = Date.now()
  },

  onAfterTool: async ({ tool }) => {
    executionMetrics.totalTime += Date.now() - tool.startTime
  },

  onIterationEnd: async () => {
    console.log('Execution metrics:', executionMetrics)
  },
})

Best Practices

1. Error Handling: Always wrap hook logic in try-catch for production
2. Performance: Keep hooks lightweight, especially onTrace
3. Security: Use onBeforeExecution and onBeforeTool for security validation
4. Debugging: Leverage onTrace for comprehensive execution monitoring
5. Guardrails: Implement business logic validation in onExit
6. Data Transformation: Use onBeforeTool/onAfterTool for input/output processing
7. Async Operations: All hooks support async/await for external API calls
8. State Management: Use closures or external state for cross-hook data sharing

---

Advanced Features

Snapshots (Pauseable Execution)

Snapshots allow you to pause and resume LLMz execution, enabling long-running workflows that can be interrupted and continued later.

SnapshotSignal

Inside a tool, throw a SnapshotSignal to halt execution and create a serializable snapshot:

import { SnapshotSignal, Tool } from 'llmz'

const longRunningTool = new Tool({
  name: 'processLargeDataset',
  input: z.object({ datasetId: z.string() }),
  async handler({ datasetId }) {
    // Start processing
    const dataset = await loadDataset(datasetId)

    // At any point, pause execution for later resumption
    if (dataset.size > LARGE_THRESHOLD) {
      throw new SnapshotSignal(
        'Dataset is large, pausing for background processing',
        'Processing will continue once background job completes'
      )
    }

    return { processed: true }
  },
})

Snapshot Handling

const result = await execute({
  instructions: 'Process the uploaded dataset',
  tools: [longRunningTool],
  client,
})

if (result.isInterrupted()) {
  console.log('Execution paused:', result.signal.message)

  // Serialize snapshot for persistence
  const serialized = result.snapshot.toJSON()
  await database.saveSnapshot('job-123', serialized)

  // Start background processing
  await backgroundJobQueue.add('process-dataset', {
    snapshotId: 'job-123',
    datasetId: result.signal.toolCall?.input.datasetId,
  })
}

Resuming from Snapshot

// Later, when background job completes
const serialized = await database.getSnapshot('job-123')
const snapshot = Snapshot.fromJSON(serialized)

// Resolve the snapshot with the result
snapshot.resolve({
  processed: true,
  recordCount: 1000000,
  processingTime: 3600000,
})

// Continue execution from where it left off
const continuation = await execute({
  snapshot,
  instructions: 'Process the uploaded dataset', // Same as original
  tools: [longRunningTool], // Same tools
  client,
})

if (continuation.isSuccess()) {
  console.log('Processing completed:', continuation.output)
}

Snapshot Rejection

// If background processing fails
const snapshot = Snapshot.fromJSON(serialized)
snapshot.reject(new Error('Background processing failed'))

const continuation = await execute({
  snapshot,
  // ... same parameters
})

// The agent will receive the error and can handle it

Thinking (Agent Reflection)

The thinking system allows agents to pause and reflect on variables and context before proceeding.

ThinkSignal (Tool-Initiated)

Tools can force thinking by throwing a ThinkSignal:

const analysisTool = new Tool({
  name: 'analyzeData',
  input: z.object({ data: z.array(z.number()) }),
  async handler({ data }) {
    const result = performAnalysis(data)

    // Force the agent to think about the results before responding
    throw new ThinkSignal(
      'Analysis complete, consider the implications',
      `Found ${result.anomalies.length} anomalies and ${result.patterns.length} patterns`
    )
  },
})

Agent-Initiated Thinking

Agents can request thinking time in generated code:

// In generated code
const analysisResult = await analyzeData({ data: userInputData })

// Think about the results before responding to user
return { action: 'think' }

Thinking with Variables

Pass specific variables for reflection:

// In generated code
const price = await calculatePrice({ items: cartItems })
const budget = await getUserBudget()

// Think about pricing vs budget with specific context
return {
  action: 'think',
  price,
  budget,
  recommendation: price > budget ? 'deny' : 'approve',
}

Handling Think Results

const result = await execute({
  instructions: 'Analyze the user data and provide recommendations',
  tools: [analysisTool],
  client,
})

if (result.is(ThinkExit)) {
  console.log('Agent is thinking about:', result.output.variables)

  // Continue execution after thinking
  const continuation = await execute({
    instructions: result.context.instructions,
    tools: result.context.tools,
    // Variables from thinking are automatically preserved
    client,
  })
}

Citations (RAG Support)

CitationsManager provides standardized source tracking and referencing for RAG (Retrieval-Augmented Generation) systems.

Core Concepts

Citations use rare Unicode symbols (【】) as markers that are unlikely to appear in natural text. The system supports:

• Source Registration: Register any object as a citation source
• Tag Generation: Automatic creation of unique citation tags like 【0】, 【1】
• Content Processing: Extract and clean citation tags from text
• Multiple Citations: Support for multi-source citations like 【0,1,3】

Basic Usage

import { CitationsManager } from 'llmz'

const citations = new CitationsManager()

// Register sources and get citation tags
const source1 = citations.registerSource({
  file: 'document.pdf',
  page: 5,
  title: 'Company Policy',
})

const source2 = citations.registerSource({
  url: 'https://example.com/article',
  title: 'Best Practices',
})

console.log(source1.tag) // "【0】"
console.log(source2.tag) // "【1】"

// Use tags in content
const content = `The policy states employees must arrive on time${source1.tag}. However, best practices suggest flexibility${source2.tag}.`

RAG Implementation Example

const ragTool = new Tool({
  name: 'search',
  description: 'Searches in the knowledge base for relevant information.',
  input: z.string().describe('The query to search in the knowledge base.'),
  async handler(query) {
    // Perform semantic search
    const { passages } = await client.searchFiles({
      query,
      limit: 20,
      contextDepth: 3,
    })

    if (!passages.length) {
      throw new ThinkSignal(
        'No results found',
        'No results were found in the knowledge base. Try rephrasing your question.'
      )
    }

    // Build response with citations
    let message: string[] = ['Here are the search results:']
    let { tag: example } = chat.citations.registerSource({}) // Example citation

    // Register each passage as a source
    for (const passage of passages) {
      const { tag } = chat.citations.registerSource({
        file: passage.file.key,
        title: passage.file.tags.title,
      })

      message.push(`<${tag} file="${passage.file.key}">`)
      message.push(`**${passage.file.tags.title}**`)
      message.push(passage.content)
      message.push(`</${tag}>`)
    }

    // Provide context with citation instructions
    throw new ThinkSignal(
      `Got search results. When answering, you MUST add inline citations (eg: "The price is $10${example} ...")`,
      message.join('\n').trim()
    )
  },
})

Chat Integration

class CLIChat extends Chat {
  public citations: CitationsManager = new CitationsManager()

  private async sendMessage(input: RenderedComponent) {
    if (input.type === 'Text') {
      let sources: string[] = []

      // Extract citations and format them for display
      const { cleaned } = this.citations.extractCitations(input.text, (citation) => {
        let idx = chalk.bgGreenBright.black.bold(` ${sources.length + 1} `)
        sources.push(`${idx}: ${JSON.stringify(citation.source)}`)
        return `${idx}` // Replace 【0】 with [1]
      })

      // Display cleaned text and sources
      console.log(`🤖 Agent: ${cleaned}`)

      if (sources.length) {
        console.log(chalk.dim('Citations'))
        console.log(chalk.dim('========='))
        console.log(chalk.dim(sources.join('\n')))
      }
    }
  }
}

Advanced Citation Features

Multiple Citation Support:

// Agent can reference multiple sources in one citation
const content = 'This fact is supported by multiple studies【0,1,3】'

const { cleaned, citations } = manager.extractCitations(content)
// citations array contains entries for sources 0, 1, and 3

Object Citation Processing:

// Remove citations from complex objects
const dataWithCitations = {
  summary: 'The report shows positive trends【0】',
  details: {
    revenue: 'Increased by 15%【1】',
    costs: 'Reduced by 8%【2】',
  },
}

const [cleanData, extractedCitations] = manager.removeCitationsFromObject(dataWithCitations)
// cleanData has citations removed, extractedCitations contains path + citation info

Citation Stripping:

// Remove all citation tags from content
const textWithCitations = 'This statement【0】 has multiple【1,2】 citations.'
const cleaned = CitationsManager.stripCitationTags(textWithCitations)
// Result: "This statement has multiple citations."

Dynamic Context

LLMz supports dynamic evaluation of most parameters, allowing context-aware configuration:

await execute({
  // Dynamic instructions based on context
  instructions: (ctx) => {
    const timeOfDay = new Date().getHours()
    const greeting = timeOfDay < 12 ? 'Good morning' : 'Good afternoon'
    return `${greeting}! You are a helpful assistant with access to ${ctx.tools?.length || 0} tools.`
  },

  // Dynamic tools based on user permissions
  tools: async (ctx) => {
    const userPermissions = await getUserPermissions(ctx.userId)
    return allTools.filter((tool) => userPermissions.includes(tool.permission))
  },

  // Dynamic objects with current state
  objects: async (ctx) => {
    const userPreferences = await loadUserPreferences(ctx.userId)
    return [
      new ObjectInstance({
        name: 'user',
        properties: [{ name: 'preferences', value: userPreferences, writable: true }],
      }),
    ]
  },

  client,
})

---

API Reference

Core Functions

execute(props: ExecutionProps): Promise

Main execution function that runs LLMz agents in either Chat Mode or Worker Mode.

Parameters:

• props.client - Botpress Client or Cognitive Client instance for LLM generation
• props.instructions - System prompt/instructions for the LLM (static string or dynamic function)
• props.chat - Optional Chat instance to enable Chat Mode with user interaction
• props.tools - Array of Tool instances available to the agent (static or dynamic)
• props.objects - Array of ObjectInstance for namespaced tools and variables (static or dynamic)
• props.exits - Array of Exit definitions for structured completion (static or dynamic)
• props.snapshot - Optional Snapshot to resume paused execution
• props.signal - Optional AbortSignal to cancel execution
• props.options - Optional execution options (loop limit, temperature, model, timeout)
• props.onTrace - Optional non-blocking hook for monitoring traces during execution
• props.onIterationEnd - Optional blocking hook called after each iteration
• props.onExit - Optional blocking hook called when an exit is reached
• props.onBeforeExecution - Optional blocking hook to modify code before VM execution
• props.onBeforeTool - Optional blocking hook to modify tool inputs before execution
• props.onAfterTool - Optional blocking hook to modify tool outputs after execution

Returns: Promise<ExecutionResult> - Result containing success/error/interrupted status with type-safe exit checking

Tool Class

new Tool(config: ToolConfig)

Creates a new tool definition with type-safe schemas.

Properties:

• name: string - Tool name used in generated code
• description?: string - Description for LLM understanding
• input?: ZuiSchema - Input validation schema
• output?: ZuiSchema - Output validation schema
• handler: (input: any) => Promise<any> | any - Tool implementation
• aliases?: string[] - Alternative names for the tool
• staticInputs?: Record<string, any> - Force specific input values

Methods:

• execute(input: any, context?: ToolContext): Promise<any> - Execute the tool
• getTypings(): string - Get TypeScript definitions for LLM
• clone(overrides: Partial<ToolConfig>): Tool - Create a modified copy

Exit Class

new Exit(config: ExitConfig)

Defines a structured exit point for agent execution.

Properties:

• name: string - Exit name used in generated code
• description?: string - Description for LLM understanding
• schema?: ZuiSchema - Output validation schema
• aliases?: string[] - Alternative names for the exit

ObjectInstance Class

new ObjectInstance(config: ObjectConfig)

Creates a namespaced container for tools and variables.

Properties:

• name: string - Object name used in generated code
• properties?: PropertyConfig[] - Object properties/variables
• tools?: Tool[] - Tools scoped to this object

PropertyConfig:

• name: string - Property name
• value: any - Initial value
• writable: boolean - Whether property can be modified
• type?: ZuiSchema - Validation schema

ExecutionResult Types

SuccessExecutionResult

Properties:

• isSuccess(): boolean - Type guard for success
• output: any - The result data from the exit
• exit: Exit - The exit that was used
• iteration: Iteration - Final iteration details
• iterations: Iteration[] - All iterations
• context: Context - Execution context
• is(exit: Exit): boolean - Type-safe exit checking

ErrorExecutionResult

Properties:

• isError(): boolean - Type guard for error
• error: Error | string - The error that occurred
• iteration?: Iteration - Failed iteration details
• iterations: Iteration[] - All iterations before failure
• context: Context - Execution context

PartialExecutionResult

Properties:

• isInterrupted(): boolean - Type guard for interruption
• signal: SnapshotSignal - The signal that caused interruption
• snapshot: Snapshot - Serializable execution state
• iterations: Iteration[] - All iterations before interruption
• context: Context - Execution context

Chat Class

Abstract base class for implementing chat interfaces.

Abstract Methods:

• getTranscript(): Promise<Transcript.Message[]> | Transcript.Message[] - Get conversation history
• getComponents(): Promise<ComponentDefinition[]> | ComponentDefinition[] - Get available UI components
• handler(component: RenderedComponent): Promise<void> - Handle agent messages

CitationsManager Class

Manages source citations for RAG systems.

Methods:

• registerSource(source: any): { tag: string, id: number } - Register a source and get citation tag
• extractCitations(text: string, replacer?: (citation) => string): { cleaned: string, citations: Citation[] } - Extract and process citations
• removeCitationsFromObject(obj: any): [cleanedObj: any, citations: Citation[]] - Remove citations from objects
• static stripCitationTags(text: string): string - Remove all citation tags

Snapshot Class

Manages pauseable execution state.

Methods:

• toJSON(): string - Serialize snapshot
• static fromJSON(json: string): Snapshot - Deserialize snapshot
• resolve(data: any): void - Resume with success
• reject(error: Error): void - Resume with error

Built-in Exits

• ListenExit - Automatically available in Chat Mode for user interaction
• DefaultExit - Default exit for Worker Mode with success/failure discrimination
• ThinkExit - Used when agent requests thinking time

Signals

• SnapshotSignal - Thrown to pause execution for later resumption
• ThinkSignal - Thrown to request agent reflection time
• LoopExceededError - Thrown when maximum iterations reached

Environment Variables

• VM_DRIVER: 'isolated-vm' | 'node' - Choose VM execution environment
• CI: boolean - Automatically detected, affects VM driver selection

---

Examples

The LLMz repository includes 20 comprehensive examples demonstrating different patterns and capabilities:

Chat Examples (Interactive Patterns)

1. 01_chat_basic - Basic conversational agent setup
2. 02_chat_exits - Custom exits for structured conversations
3. 03_chat_conditional_tool - Conditional tool usage based on context
4. 04_chat_small_models - Optimizations for smaller language models
5. 05_chat_web_search - Integration with web search capabilities
6. 06_chat_confirm_tool - User confirmation patterns for sensitive operations
7. 07_chat_guardrails - Safety mechanisms and content filtering
8. 08_chat_multi_agent - Multi-agent orchestration and delegation
9. 09_chat_variables - Object variables and state management
10. 10_chat_components - Rich UI components and interactive elements

Worker Examples (Automated Patterns)

11. 11_worker_minimal - Simplest worker mode execution
12. 12_worker_fs - File system operations and data processing
13. 13_worker_sandbox - Security isolation and sandboxing
14. 14_worker_snapshot - Pauseable execution and resumption
15. 15_worker_stacktraces - Error handling and debugging
16. 16_worker_tool_chaining - Complex multi-tool workflows
17. 17_worker_error_recovery - Graceful error recovery patterns
18. 18_worker_security - Security best practices and validation
19. 19_worker_wrap_tool - Tool modification and enhancement
20. 20_chat_rag - Retrieval-Augmented Generation with citations

Running Examples

# Install dependencies
pnpm install

# Set up environment variables
cp .env.example .env
# Edit .env with your Botpress credentials

# Run a specific example
pnpm start 01_chat_basic
pnpm start chat_basic
pnpm start 01

# List all available examples
pnpm start

Example Environment Setup

Create a .env file in the examples directory:

BOTPRESS_BOT_ID=your_bot_id_here
BOTPRESS_TOKEN=your_token_here

Key Learning Paths

Getting Started:

• Start with 01_chat_basic and 11_worker_minimal
• Understand the difference between Chat and Worker modes
• Learn basic tool integration patterns

Intermediate Concepts:

• Explore 09_chat_variables for state management
• Study 16_worker_tool_chaining for complex workflows
• Review 14_worker_snapshot for pauseable execution

Advanced Patterns:

• Examine 08_chat_multi_agent for orchestration
• Learn from 20_chat_rag for knowledge integration
• Study 18_worker_security for production deployment

Each example includes detailed comments explaining the concepts and implementation patterns, making them excellent learning resources for understanding LLMz capabilities.

---

This documentation covers the complete LLMz framework. For the latest updates and community contributions, visit the LLMz repository.