promptfoo--promptfoo/site/docs/usage/node-api-reference.md

sidebar_label, sidebar_position, title, description

sidebar_label	sidebar_position	title	description
Node API Reference	21	Node API reference	Reference guide for promptfoo's Node.js APIs, including evals, providers, assertions, caching, guardrails, red teaming, and utilities.

Node Module API Reference

This guide documents the Node.js module API for advanced programmatic usage of promptfoo. It covers all publicly exported functions, namespaces, and types.

:::info For standard YAML-based evaluation configuration, see Configuration Guide. This API reference is for users building programmatic evaluation workflows directly in JavaScript/TypeScript. :::

Overview

The promptfoo Node module provides programmatic access to:

• Core evaluation engine (evaluate())
• Provider management (load and resolve LLM providers)
• Assertion execution (run assertions independently)
• Cache management (control caching behavior)
• Security guardrails (PII, harm, moderation checks)
• Adversarial testing (red team evaluation)

Core API

evaluate(testSuite, options?)

The main entry point for running evaluations programmatically.

async function evaluate(testSuite: EvaluateTestSuite, options?: EvaluateOptions): Promise<Eval>;

Parameters:

• testSuite: Configuration object containing prompts, providers, and tests
• options: Optional evaluation settings (caching, output, concurrency, etc.)

Returns: Eval record. Call toEvaluateSummary() when you need the serializable results summary.

Example:

import { evaluate } from 'promptfoo';

const evalRecord = await evaluate({
  prompts: ['What is 2+2?'],
  providers: ['openai:chat:gpt-5.5', 'anthropic:messages:claude-opus-4-7'],
  tests: [
    {
      vars: { query: 'math question' },
      assert: [
        {
          type: 'contains',
          value: '4',
          metric: 'is_correct',
        },
      ],
    },
  ],
});
const results = await evalRecord.toEvaluateSummary();

console.log(`Passed: ${results.stats.successes}/${results.results.length}`);
console.log(`Shareable URL: ${evalRecord.shareableUrl}`);

Common Options:

interface EvaluateOptions {
  // Output and format
  outputPath?: string | string[];
  formatOutput?: boolean;

  // Evaluation behavior
  maxConcurrency?: number;
  nunjucksFilters?: Record<string, Function>;

  // Caching
  cache?: boolean;

  // Sharing and persistence
  sharing?: boolean;
  writeLatestResults?: boolean;

  // Progress tracking
  onTestComplete?: (result: EvaluateResult) => void;
}

---

Provider APIs

loadApiProvider(providerPath, context?)

Load a single provider instance by path or identifier.

async function loadApiProvider(
  providerPath: string,
  context?: LoadApiProviderContext,
): Promise<ApiProvider>;

Parameters:

• providerPath: Provider identifier (e.g., 'openai:chat:gpt-5.5', 'anthropic:messages:claude-opus-4-7', or 'file://./custom-provider.js')
• context: Optional context with environment overrides

Returns: Configured ApiProvider instance ready to call

Example:

import { loadApiProvider } from 'promptfoo';

const openaiProvider = await loadApiProvider('openai:chat:gpt-5.5', {
  env: { OPENAI_API_KEY: process.env.MY_SECRET_KEY },
});

const response = await openaiProvider.callApi('Hello, world!');

console.log(response.output);

Supported Provider Types:

• openai:* (GPT models)
• anthropic:* (Claude models)
• azure:*
• bedrock:* (AWS Bedrock)
• vertex:* (Google Vertex AI)
• cohere:*
• replicate:*
• huggingface:*
• Custom files: file://./path/to/provider.js
• Custom functions via inline JavaScript

---

loadApiProviders(providers, options?)

Load multiple providers in parallel. Accepts the same ProvidersConfig shape that evaluate() uses internally, so you can mix string identifiers with inline config objects or functions.

async function loadApiProviders(
  providers: ProvidersConfig,
  options?: { env?: Record<string, string> },
): Promise<ApiProvider[]>;

Parameters:

• providers: Array of provider identifiers or config objects
• options: Optional environment overrides

Returns: Array of configured provider instances

Example:

import { loadApiProviders } from 'promptfoo';

const providers = await loadApiProviders([
  'openai:chat:gpt-5.5',
  'anthropic:messages:claude-opus-4-7',
  {
    id: 'custom-provider',
    config: {
      model: 'my-model',
      temperature: 0.7,
    },
  },
]);

for (const provider of providers) {
  const result = await provider.callApi('Test');
  console.log(`${provider.id()}: ${result.output}`);
}

---

Assertions API

assertions.runAssertion(params)

Execute a single assertion against provider output. Powerful for custom evaluation logic.

async function runAssertion({
  prompt?: string;
  provider?: ApiProvider;
  assertion: Assertion;
  test: AtomicTestCase;
  vars?: Record<string, VarValue>;
  latencyMs?: number;
  providerResponse: ProviderResponse;
  traceId?: string;
  traceData?: TraceData | null;
}): Promise<GradingResult>

Parameters:

• assertion: The assertion to run (e.g., { type: 'contains', value: 'expected' })
• test: The test case context
• providerResponse: The output from the provider to evaluate
• vars: Template variables from the test
• provider: Provider instance (optional, for context)
• traceData: Distributed trace data for debugging (optional)

Returns:

interface GradingResult {
  pass: boolean; // Did the assertion pass?
  score?: number; // 0-1 score
  reason?: string; // Explanation
  assertion: Assertion; // The original assertion
  metric?: string; // Metric name
  error?: string; // Error message if failed
}

Example: Custom Assertion Logic

import { assertions } from 'promptfoo';

const result = await assertions.runAssertion({
  assertion: {
    type: 'javascript',
    value: (output, context) => {
      // Custom grading logic
      const score = output.includes('yes') ? 1.0 : 0.0;
      return {
        pass: score >= 0.8,
        score,
        reason: `Output contains required keyword`,
      };
    },
  },
  test: {
    vars: { question: 'Is the sky blue?' },
    assert: [], // populated with assertion
  },
  providerResponse: {
    output: 'Yes, the sky is blue in most places.',
    tokenUsage: { total: 15 },
  },
});

console.log(`Pass: ${result.pass}, Score: ${result.score}`);

Assertion Value Function Context:

interface AssertionValueFunctionContext {
  // Test and evaluation metadata
  prompt: string | undefined;
  vars: Record<string, unknown>; // Template variables
  test: AtomicTestCase; // Full test context

  // Provider info
  provider: ApiProvider | undefined;
  providerResponse: ProviderResponse | undefined;

  // Output metadata
  logProbs?: number[]; // Token log probabilities

  // Advanced: Tracing
  trace?: TraceData; // Distributed trace with spans
  config?: Record<string, any>; // Custom assertion config
}

Using Trace Data:

import { assertions } from 'promptfoo';

const result = await assertions.runAssertion({
  assertion: {
    type: 'javascript',
    value: (output, context) => {
      // Access trace data for latency analysis
      if (context.trace?.spans) {
        const ttft = context.trace.spans.find((s) => s.name === 'time_to_first_token');
        console.log(`Time to first token: ${ttft?.duration}ms`);
      }
      return { pass: true };
    },
  },
  test: { vars: {} },
  providerResponse: { output: 'test' },
  traceId: 'trace-123',
  traceData: {
    spans: [
      {
        name: 'time_to_first_token',
        startTime: Date.now(),
        duration: 250,
      },
    ],
  },
});

---

assertions.runAssertions(params)

Execute multiple assertions in batch against provider output.

async function runAssertions({
  assertions: (Assertion | AssertionSet)[];
  prompt?: string;
  test: AtomicTestCase;
  provider?: ApiProvider;
  vars?: Record<string, VarValue>;
  providerResponse: ProviderResponse;
  latencyMs?: number;
  traceData?: TraceData | null;
}): Promise<AssertionsResult>

Returns:

interface GradingResult {
  pass: boolean;
  score: number; // Aggregate score across all assertions
  reason?: string;
  componentResults?: GradingResult[]; // Per-assertion results
  namedScores?: Record<string, number>;
  tokensUsed?: {
    total: number;
    prompt: number;
    completion: number;
    cached: number;
    numRequests: number;
  };
}

Example:

import { assertions } from 'promptfoo';

const result = await assertions.runAssertions({
  assertions: [
    { type: 'contains', value: '4' },
    { type: 'regex', value: '^The answer is \\d+$' },
    { type: 'not-regex', value: '(?i)error|failed' },
  ],
  test: { vars: { question: 'What is 2+2?' } },
  providerResponse: { output: 'The answer is 4.' },
});

console.log(`All passed: ${result.pass}`);
console.log(`Average score: ${result.score}`);
result.componentResults?.forEach((r) => console.log(`  ${r.assertion?.type}: ${r.pass}`));

---

Cache API

Control promptfoo's caching layer for LLM provider calls.

enableCache()

Enable caching for provider calls (default).

export function enableCache(): void;

Example:

import { cache, evaluate } from 'promptfoo';

cache.enableCache();

---

disableCache()

Disable caching. Useful during development/testing to always hit the provider.

export function disableCache(): void;

Example:

import { cache, evaluate } from 'promptfoo';

// Disable for fresh results
cache.disableCache();
const evalRecord = await evaluate(testSuite);
cache.enableCache();

---

isCacheEnabled()

Check if caching is currently enabled.

export function isCacheEnabled(): boolean;

---

clearCache()

Clear all cached results.

export async function clearCache(): Promise<void>;

Example:

import { cache, evaluate } from 'promptfoo';

// Clear old cache
await cache.clearCache();
await evaluate(testSuite); // Will refetch all provider calls

---

withCacheNamespace(namespace, fn)

Run evaluation with isolated cache namespace.

export function withCacheNamespace<T>(
  namespace: string | undefined,
  fn: () => Promise<T>,
): Promise<T>;

Use Cases:

• Isolate caches for different test runs
• Prevent cache collisions across environments
• Version-specific caching

Example:

import { cache, evaluate } from 'promptfoo';

// Run v1 and v2 evals with separate caches
const v1Results = await cache.withCacheNamespace('v1', async () => {
  return evaluate(testSuiteV1);
});

const v2Results = await cache.withCacheNamespace('v2', async () => {
  return evaluate(testSuiteV2);
});

---

getCache()

Get the underlying cache instance for advanced operations.

export function getCache(): Cache;

interface Cache {
  get(key: string): Promise<unknown>;
  set(key: string, value: unknown, ttl?: number): Promise<void>;
  del(key: string): Promise<void>;
  clear(): Promise<void>;
}

---

fetchWithCache(url, options?, timeout?, format?, bustOrOptions?, maxRetries?)

Fetch with caching enabled. The fifth argument accepts either the legacy boolean cache-bust flag or an options object.

export async function fetchWithCache<T>(
  url: string,
  options?: RequestInit,
  timeout?: number,
  format?: 'json' | 'text',
  bustOrOptions?: boolean | { bust?: boolean; repeatIndex?: number },
  maxRetries?: number,
): Promise<FetchWithCacheResult<T>>;

Example:

import { cache } from 'promptfoo';

const result = await cache.fetchWithCache(
  'https://api.example.com/data',
  { method: 'GET' },
  undefined,
  'json',
);

console.log(result.cached); // true if from cache
console.log(result.data); // the fetched data

Configuration: Set cache location and TTL via environment variables:

# Cache directory (default: ~/.promptfoo/cache)
export PROMPTFOO_CACHE_PATH=/path/to/cache

# Cache TTL in seconds (default: 1209600 = 14 days)
export PROMPTFOO_CACHE_TTL=1209600

# Disable cache via env
export PROMPTFOO_CACHE_ENABLED=false

---

Guardrails API

Content safety and security layer. Use guardrails to detect PII, harmful content, and other safety concerns.

guardrails.guard(input)

Run general content moderation.

async function guard(input: string): Promise<GuardResult>;

Returns:

interface GuardResult {
  model: string;
  results: Array<{
    categories: Record<string, boolean>; // e.g., { hate: false, violence: true }
    category_scores: Record<string, number>; // Scores 0-1
    flagged: boolean; // Any category flagged?
  }>;
}

Example:

import { guardrails } from 'promptfoo';

const result = await guardrails.guard('This is a test message');

if (result.results[0].flagged) {
  console.log('Content flagged for safety issues:');
  Object.entries(result.results[0].categories).forEach(([name, flagged]) => {
    if (flagged) {
      console.log(`  ${name}: ${result.results[0].category_scores[name]}`);
    }
  });
}

---

guardrails.pii(input)

Detect personally identifiable information (PII).

async function pii(input: string): Promise<GuardResult>;

Returns: GuardResult with PII detection details

Example:

import { guardrails } from 'promptfoo';

const result = await guardrails.pii('John Doe, john@example.com, SSN: 123-45-6789');

if (result.results[0].flagged) {
  console.log('PII detected:');
  if (result.results[0].payload?.pii) {
    result.results[0].payload.pii.forEach((item) => {
      console.log(`  ${item.type}: ${item.value}`);
    });
  }
}

---

guardrails.harm(input)

Detect harmful content (violence, hate speech, etc.).

async function harm(input: string): Promise<GuardResult>;

---

guardrails.adaptive(request)

Run adaptive guardrails with custom configuration.

async function adaptive(request: AdaptiveRequest): Promise<AdaptiveResult>;

---

Red Team API

Adversarial testing framework for finding vulnerabilities in LLM systems.

redteam.generate(config)

Generate adversarial test cases.

async function generate(config: RedteamGenerateConfig): Promise<RedteamGenerateResult>;

See Red Teaming Guide for full documentation.

---

redteam.run(config)

Run full red team evaluation against a system.

async function run(config: RedteamRunConfig): Promise<RedteamRunResult>;

---

redteam.Plugins

Registry of available red team attack plugins.

interface Plugins {
  [pluginName: string]: typeof RedteamPluginBase;
}

// Example plugins
Plugins['indirect-prompt-injection']; // Indirect prompt injection
Plugins['hijacking']; // Hijacking attempts
Plugins['rbac']; // RBAC bypasses
Plugins['sql-injection']; // SQL injection
// ... and many more

Creating Custom Plugins:

import { redteam } from 'promptfoo';

class MyCustomPlugin extends redteam.Base.Plugin {
  async run(params: RedteamPluginRunParams): Promise<RedteamPluginResult> {
    // Custom attack logic
    return {
      generated: ['attack1', 'attack2'],
      stats: { duration: 100 },
    };
  }
}

// Register and use
export default MyCustomPlugin;

---

redteam.Strategies

Registry of red team strategies for composing attacks.

interface Strategies {
  [strategyName: string]: RedteamStrategy;
}

See Red Teaming Strategies for details.

---

redteam.Extractors

Extract system information for targeted attacks.

const {
  extractEntities, // Extract named entities from system
  extractMcpToolsInfo, // Extract MCP tool metadata
  extractSystemPurpose, // Extract system purpose and goal
} = redteam.Extractors;

Example:

import { redteam } from 'promptfoo';

const purpose = await redteam.Extractors.extractSystemPurpose({
  prompt: 'You are a helpful assistant...',
});

console.log(`System purpose: ${purpose}`);

---

redteam.Base.Plugin

Base class for custom red team plugins.

abstract class RedteamPluginBase {
  async run(params: {
    target: Prompt;
    injectVar?: string;
    conversationHistory?: Message[];
    options?: Record<string, unknown>;
  }): Promise<RedteamPluginResult>;
}

See Plugin Development Guide (below) for examples.

---

redteam.Base.Grader

Base class for custom red team graders.

abstract class RedteamGraderBase {
  async grade(params: { prompt: string; output: string; rubric: string }): Promise<GradingResult>;
}

---

Utilities

generateTable(evaluateTable, tableCellMaxLength?, maxRows?)

Format evaluation results as a display table.

export function generateTable(
  evaluateTable: EvaluateTable,
  tableCellMaxLength?: number,
  maxRows?: number,
): string;

Example:

import { evaluate, generateTable } from 'promptfoo';

const evalRecord = await evaluate(testSuite);
const table = await evalRecord.getTable();
console.log(generateTable(table));

---

isTransformFunction(value)

Check if a value is a transform function (for custom prompt/variable transforms).

export function isTransformFunction(value: unknown): value is TransformFunction;

Example:

import { isTransformFunction } from 'promptfoo';

const maybeTransform = (x) => x.toUpperCase();
console.log(isTransformFunction(maybeTransform)); // true

---

Advanced Patterns

Pattern 1: Programmatic Test Suite Generation

Dynamically generate test cases:

import { evaluate } from 'promptfoo';

const generateTests = (questions) => {
  return questions.map((q) => ({
    vars: { question: q },
    assert: [
      { type: 'contains', value: 'verified-fact' },
      {
        type: 'custom-llm-rubric',
        value: 'Is the answer accurate?',
      },
    ],
  }));
};

const evalRecord = await evaluate({
  prompts: ['Answer: {{ question }}'],
  providers: ['openai:chat:gpt-5.5'],
  tests: generateTests([
    'What is the capital of France?',
    'What is 2+2?',
    'What is photosynthesis?',
  ]),
});

---

Pattern 2: Custom Grading with Context

Use provider response data in assertions:

import { evaluate } from 'promptfoo';

const evalRecord = await evaluate({
  prompts: ['Analyze: {{ text }}'],
  providers: ['openai:chat:gpt-5.5'],
  tests: [
    {
      vars: { text: 'Sample text' },
      assert: [
        {
          type: 'javascript',
          value: (output, context) => {
            // Access token usage from provider response
            const tokens = context.providerResponse?.tokenUsage?.total || 0;
            return {
              pass: tokens < 100,
              reason: `Used ${tokens} tokens`,
            };
          },
        },
      ],
    },
  ],
});

---

Pattern 3: Batch Provider Testing

Load providers and test in parallel:

import { assertions, loadApiProviders } from 'promptfoo';

const providers = await loadApiProviders([
  'openai:chat:gpt-5.5',
  'anthropic:messages:claude-opus-4-7',
]);

const testCases = ['2+2=?', 'What is AI?'];

for (const provider of providers) {
  console.log(`Testing ${provider.id()}:`);
  for (const test of testCases) {
    const response = await provider.callApi(test);

    const result = await assertions.runAssertion({
      provider,
      assertion: { type: 'contains', value: 'answer' },
      test: { vars: { prompt: test } },
      providerResponse: response,
    });

    console.log(`  ${test}: ${result.pass ? '✓' : '✗'}`);
  }
}

---

Pattern 4: Cache Isolation for A/B Testing

Compare two model versions with isolated caches:

import { cache, evaluate } from 'promptfoo';

async function compareModels(oldVersion, newVersion, testSuite) {
  const oldEval = await cache.withCacheNamespace(`model-${oldVersion}`, () =>
    evaluate({
      ...testSuite,
      providers: [`openai:chat:${oldVersion}`],
    }),
  );

  const newEval = await cache.withCacheNamespace(`model-${newVersion}`, () =>
    evaluate({
      ...testSuite,
      providers: [`openai:chat:${newVersion}`],
    }),
  );

  const oldResults = await oldEval.toEvaluateSummary();
  const newResults = await newEval.toEvaluateSummary();

  return {
    oldPass: oldResults.stats.successes,
    newPass: newResults.stats.successes,
    improvement: newResults.stats.successes - oldResults.stats.successes,
  };
}

---

Type Definitions

All types are exported from promptfoo and available in TypeScript:

import type {
  // Main types
  EvaluateTestSuite,
  EvaluateOptions,
  EvaluateSummary,
  EvaluateResult,

  // Test configuration
  TestCase,
  AtomicTestCase,
  Assertion,
  AssertionSet,

  // Provider types
  ApiProvider,
  ProviderResponse,
  ProviderConfig,

  // Evaluation types
  GradingResult,
  AssertionValueFunctionContext,

  // Transform types
  TransformFunction,
  TransformContext,

  // Extension hooks
  BeforeAllExtensionHookContext,
  AfterEachExtensionHookContext,
  // ... and more
} from 'promptfoo';

---

API Stability

Stable APIs (Safe for production)

• evaluate()
• loadApiProvider(), loadApiProviders()
• assertions.runAssertion(), assertions.runAssertions()
• Cache namespace functions
• Guardrails namespace

Experimental APIs (May change)

• redteam.* (plugin/strategy system)
• Trace data structures

Internal APIs (Don't use directly)

• Functions not exported from main promptfoo module
• Test utilities
• CLI-specific functions

---

Troubleshooting

Cache issues?

// Clear cache if stale
import { cache } from 'promptfoo';
await cache.clearCache();

// Or disable for development
cache.disableCache();

Type errors with custom assertions?

import type { AssertionValueFunctionContext } from 'promptfoo';

// Type-safe context usage
value: (output, context: AssertionValueFunctionContext) => {
  // Full autocomplete and type checking
};

Need to debug evaluation flow?

// Enable verbose logging
process.env.LOG_LEVEL = 'debug';

// Or check trace data in assertions
const result = await assertions.runAssertion({
  // ...
  traceData: trace, // Access timing information
});

---

Examples Repository

Find runnable examples in the promptfoo examples directory.

---

See Also

• Configuration Guide - YAML-based setup
• Red Teaming Guide - Adversarial testing
• Assertions Reference - Assertion types
• Providers Reference - LLM provider setup