promptfoo--promptfoo/site/docs/providers/custom-api.md

---
sidebar_label: Custom Javascript
description: Configure custom JavaScript providers to integrate any API or service with promptfoo's testing framework using TypeScript, CommonJS, or ESM modules
---

# Javascript Provider

Custom Javascript providers let you create providers in JavaScript or TypeScript to integrate with any API or service not already built into promptfoo.

## Supported File Formats and Examples

promptfoo supports multiple JavaScript module formats. Complete working examples are available on GitHub:

- [CommonJS Provider](https://github.com/promptfoo/promptfoo/tree/main/examples/provider-custom/basic) - (`.js`, `.cjs`) - Uses `module.exports` and `require()`
- [ESM Provider](https://github.com/promptfoo/promptfoo/tree/main/examples/provider-custom/mjs) - (`.mjs`, `.js` with `"type": "module"`) - Uses `import`/`export`
- [TypeScript Provider](https://github.com/promptfoo/promptfoo/tree/main/examples/provider-custom/typescript) - (`.ts`) - Provides type safety with interfaces
- [Embeddings Provider](https://github.com/promptfoo/promptfoo/tree/main/examples/provider-custom/embeddings) (commonjs)

## Provider Interface

At minimum, a custom provider must implement an `id` method and a `callApi` method.

```javascript title="echoProvider.mjs"
// Save as echoProvider.mjs for ES6 syntax, or echoProvider.js for CommonJS
export default class EchoProvider {
  id = () => 'echo';

  callApi = async (prompt, context, options) => {
    return {
      output: `Echo: ${prompt}`,
    };
  };
}
```

You can optionally use a constructor to initialize the provider, for example:

```javascript title="openaiProvider.js"
const promptfoo = require('promptfoo').default;

module.exports = class OpenAIProvider {
  constructor(options) {
    this.providerId = options.id || 'openai-custom';
    this.config = options.config;
  }

  id() {
    return this.providerId;
  }

  async callApi(prompt, context, options) {
    const { data } = await promptfoo.cache.fetchWithCache(
      'https://api.openai.com/v1/chat/completions',
      {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
          Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
        },
        body: JSON.stringify({
          model: this.config?.model || 'gpt-5-mini',
          messages: [{ role: 'user', content: prompt }],
          max_completion_tokens: this.config?.max_tokens || 1024,
          temperature: this.config?.temperature || 1,
        }),
      },
    );

    return {
      output: data.choices[0].message.content,
      tokenUsage: data.usage,
    };
  }
};
```

`callApi` returns a `ProviderResponse` object. The `ProviderResponse` object format:

```javascript
{
  // main response shown to users
  output: "Model response - can be text or structured data",
  error: "Error message if applicable",
  prompt: "The actual prompt sent to the LLM", // Optional: reported prompt
  tokenUsage: {
    total: 100,
    prompt: 50,
    completion: 50,
  },
  cost: 0.002,
  cached: false,
  conversationEnded: false, // Optional: set true to stop multi-turn redteam gracefully
  conversationEndReason: 'thread_closed', // Optional reason when conversationEnded=true
  metadata: {}, // Additional data
  ...
}
```

### Context Parameter

The `context` parameter provides test case information and utility objects:

```javascript
{
  vars: {},              // Test case variables
  prompt: {},            // Prompt template (raw, label, config)
  test: {                // Full test case object
    vars: {},
    metadata: {
      pluginId: '...',   // Redteam plugin (e.g. "promptfoo:redteam:harmful:hate")
      strategyId: '...',  // Redteam strategy (e.g. "jailbreak", "jailbreak-templates")
    },
  },
  originalProvider: {},  // Original provider when overridden
  logger: {},            // Winston logger instance
}
```

For redteam evals, use `context.test.metadata.pluginId` and `context.test.metadata.strategyId` to identify which plugin and strategy generated the test case.

### Reporting the Actual Prompt

If your provider dynamically generates or modifies prompts, you can report the actual prompt sent to the LLM using the `prompt` field in your response. This is useful for:

- Frameworks like GenAIScript that generate prompts dynamically
- Agent frameworks that build multi-turn conversations
- Providers that add system instructions or modify the prompt

```javascript title="dynamicPromptProvider.mjs"
export default class DynamicPromptProvider {
  id = () => 'dynamic-prompt';

  callApi = async (prompt, context) => {
    // Generate a different prompt dynamically
    const generatedPrompt = `System: You are helpful.\nUser: ${prompt}`;

    // Call the LLM with the generated prompt
    const response = await callLLM(generatedPrompt);

    return {
      output: response,
      prompt: generatedPrompt, // Report what was actually sent
    };
  };
}
```

The reported prompt is used for:

- **Display**: Shown as "Actual Prompt Sent" in the web UI
- **Assertions**: Prompt-based assertions like `moderation` check this value
- **Debugging**: Helps understand what was actually sent to the LLM

See the [vercel-ai-sdk example](https://github.com/promptfoo/promptfoo/tree/main/examples/integration-vercel/ai-sdk) for a complete working example.

### Two-Stage Provider

```javascript title="twoStageProvider.js"
const promptfoo = require('promptfoo').default;

module.exports = class TwoStageProvider {
  constructor(options) {
    this.providerId = options.id || 'two-stage';
    this.config = options.config;
  }

  id() {
    return this.providerId;
  }

  async callApi(prompt) {
    // First stage: fetch additional data
    const secretData = await this.fetchSecret(this.config.secretKey);

    // Second stage: call LLM with enriched prompt
    const enrichedPrompt = `${prompt}\nContext: ${secretData}`;
    const llmResponse = await this.callLLM(enrichedPrompt);

    return {
      output: llmResponse.output,
      metadata: { secretUsed: true },
    };
  }

  async fetchSecret(key) {
    // Fetch some external data needed for processing
    return `Secret information for ${key}`;
  }

  async callLLM(prompt) {
    const { data } = await promptfoo.cache.fetchWithCache(
      'https://api.openai.com/v1/chat/completions',
      {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
          Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
        },
        body: JSON.stringify({
          model: 'gpt-5-mini',
          messages: [{ role: 'user', content: prompt }],
        }),
      },
    );

    return {
      output: data.choices[0].message.content,
    };
  }
};
```

### TypeScript Implementation

```typescript title="typedProvider.ts"
import promptfoo from 'promptfoo';
import type {
  ApiProvider,
  ProviderOptions,
  ProviderResponse,
  CallApiContextParams,
} from 'promptfoo';

export default class TypedProvider implements ApiProvider {
  protected providerId: string;
  public config: Record<string, any>;

  constructor(options: ProviderOptions) {
    this.providerId = options.id || 'typed-provider';
    this.config = options.config || {};
  }

  id(): string {
    return this.providerId;
  }

  async callApi(prompt: string, context?: CallApiContextParams): Promise<ProviderResponse> {
    const username = (context?.vars?.username as string) || 'anonymous';

    return {
      output: `Hello, ${username}! You said: "${prompt}"`,
      tokenUsage: {
        total: prompt.length,
        prompt: prompt.length,
        completion: 0,
      },
    };
  }
}
```

### TypeScript Providers in Frontend Projects

Promptfoo loads TypeScript providers in Node.js, not through your frontend bundler. If your provider imports app code from a Vite, Next.js, or Webpack project, make sure the imports are also valid from Node.

For path aliases such as `@/utils`, define the alias in `tsconfig.json`:

```json
{
  "compilerOptions": {
    "baseUrl": ".",
    "paths": {
      "@/*": ["src/*"]
    }
  }
}
```

Run `promptfoo eval` from the project root so the TypeScript loader can find that `tsconfig.json`.

If your provider depends on bundler-only aliases, browser-only globals, CSS imports, or frontend plugins, compile or bundle the provider to JavaScript first and reference the built file:

```yaml
providers:
  - file://dist/promptfoo-provider.js
```

## Additional Capabilities

### Embeddings API

```javascript title="embeddingProvider.js"
async callEmbeddingApi(text) {
  const response = await fetch('https://api.openai.com/v1/embeddings', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
    },
    body: JSON.stringify({
      model: 'text-embedding-3-small',
      input: text,
    }),
  });

  const data = await response.json();

  return {
    embedding: data.data[0].embedding,
    tokenUsage: {
      total: data.usage.total_tokens,
      prompt: data.usage.prompt_tokens,
      completion: 0,
    },
  };
}
```

### Classification API

```javascript title="classificationProvider.js"
async callClassificationApi(text) {
  return {
    classification: {
      positive: 0.75,
      neutral: 0.20,
      negative: 0.05,
    },
  };
}
```

### Handling Multimodal Content

Custom providers handle multimodal content the same way whether the media comes from a standard eval or a red team strategy: read the media variable from `context.vars` and translate it into the target API's expected payload shape.

For standard evals, provide the media value through `tests[].vars`, `defaultTest.vars`, a dataset column, or a dynamic variable:

```yaml title="promptfooconfig.yaml"
prompts:
  - '{{image}} {{question}}'

tests:
  - vars:
      image: 'data:image/png;base64,iVBORw0KGgo...'
      question: Describe this image.
```

In this case, `context.vars.image` contains the configured value. It may be raw base64, a `data:` URL, an external URL, or another representation your provider knows how to forward.

For red team runs, [image](/docs/red-team/strategies/image), [audio](/docs/red-team/strategies/audio), and [video](/docs/red-team/strategies/video) strategies generate media and store it in the template variable named by `redteam.injectVar`. The rendered `prompt` also contains the media value, but `context.vars` is safer because it preserves variable boundaries and avoids parsing a very long prompt.

| Red team strategy | `context.vars[redteam.injectVar]`                        | Extra context                                                   | Forwarding notes                                                                                                                                                                                                                   |
| ----------------- | -------------------------------------------------------- | --------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `image`           | Raw PNG base64, no `data:` prefix                        | `context.vars.image_text`, `context.test.metadata.originalText` | Wrap as `data:image/png;base64,...` for APIs that expect data URLs.                                                                                                                                                                |
| `audio`           | Raw MP3 base64 from remote generation, no `data:` prefix | `context.test.metadata.originalText`                            | Requires remote generation. Forward with MIME type `audio/mpeg` or your provider's equivalent audio format.                                                                                                                        |
| `video`           | Raw MP4 base64 when local FFmpeg generation succeeds     | `context.vars.video_text`, `context.test.metadata.originalText` | Install FFmpeg and set `PROMPTFOO_DISABLE_REMOTE_GENERATION=true` or `PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION=true` for real MP4 bytes. If generation falls back, the value may decode to the original text instead of an MP4. |

Audio and video have opposite generation requirements today: audio requires remote generation, while real MP4 video requires the local FFmpeg path. Run separate scans if you need to verify both remote audio and local MP4 handling.

```javascript title="multimodalProvider.js"
module.exports = class MultimodalProvider {
  id() {
    return 'multimodal-provider';
  }

  async callApi(prompt, context) {
    const apiKey = process.env.OPENAI_API_KEY;
    if (!apiKey) {
      return { error: 'OPENAI_API_KEY is required' };
    }

    const imageBase64 = context.vars.image || '';
    const question = context.vars.question || 'Describe this image';

    // Red team image runs provide raw PNG base64. Eval vars may already provide a URL.
    const imageUrl = /^(data:|https?:\/\/)/.test(imageBase64)
      ? imageBase64
      : `data:image/png;base64,${imageBase64}`;

    const response = await fetch('https://api.openai.com/v1/chat/completions', {
      method: 'POST',
      headers: {
        Authorization: `Bearer ${apiKey}`,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({
        model: 'gpt-5',
        messages: [
          {
            role: 'user',
            content: [
              { type: 'image_url', image_url: { url: imageUrl } },
              { type: 'text', text: question },
            ],
          },
        ],
      }),
    });

    if (!response.ok) {
      return { error: `OpenAI API error ${response.status}: ${await response.text()}` };
    }

    const result = await response.json();
    const output = result.choices?.[0]?.message?.content;
    return output
      ? { output }
      : { error: `OpenAI API returned no output: ${JSON.stringify(result)}` };
  }
};
```

:::note

`injectVar` defaults to the **last** template variable in your prompt. With `{{image}} {{question}}`, it defaults to `question` — not `image`. Always set `injectVar` explicitly when using media strategies.

:::

Avoid logging full media strings; screenshots, audio, and video can be large or sensitive. For debugging, log length, detected MIME type, a hash, or the first few bytes after decoding instead of the full base64 payload.

See the [Python provider multimodal docs](/docs/providers/python#handling-multimodal-content) for a Python example.

## Cache System

The built-in caching system helps avoid redundant API calls:

```javascript title="cacheExample.js"
// Get the cache instance
const cache = promptfoo.cache.getCache();

// Store and retrieve data
await cache.set('my-key', 'cached-value', { ttl: 3600 }); // TTL in seconds
const value = await cache.get('my-key');

// Fetch with cache wrapper
const { data, cached } = await promptfoo.cache.fetchWithCache(
  'https://api.example.com/endpoint',
  {
    method: 'POST',
    body: JSON.stringify({ query: 'data' }),
  },
  5000, // timeout in ms
);
```

## Configuration

### Provider Configuration

```yaml title="promptfooconfig.yaml"
providers:
  - id: file://./myProvider.mjs # ES6 modules
    label: 'My Custom API' # Display name in UI
    config:
      model: 'gpt-5'
      temperature: 0.7
      max_tokens: 2000
      custom_parameter: 'custom value'
  # - id: file://./myProvider.js   # CommonJS modules
```

### Link to Cloud Target

:::info Promptfoo Cloud Feature
Available in [Promptfoo Cloud](/docs/enterprise) deployments.
:::

Link your local provider configuration to a cloud target using `linkedTargetId`:

```yaml
providers:
  - id: file://./myProvider.mjs
    config:
      linkedTargetId: 'promptfoo://provider/12345678-1234-1234-1234-123456789abc'
```

See [Linking Local Targets to Cloud](/docs/red-team/troubleshooting/linking-targets/) for setup instructions.

### Multiple Instances

```yaml title="multiple-providers.yaml"
providers:
  - id: file:///path/to/provider.js
    label: high-temperature
    config:
      temperature: 0.9
  - id: file:///path/to/provider.js
    label: low-temperature
    config:
      temperature: 0.1
```

## See Also

- [Browser Provider](/docs/providers/browser/)
- [Custom Provider Examples](https://github.com/promptfoo/promptfoo/tree/main/examples)
- [Custom Script Provider](/docs/providers/custom-script/)
- [Go Provider](/docs/providers/go/)
- [HTTP Provider](/docs/providers/http/)
- [Python Provider](/docs/providers/python/)