promptfoo--promptfoo/plugins/promptfoo/skills/promptfoo-provider-setup/references/provider-patterns.md

Provider Setup Patterns

Use these as starting points; keep secrets in env vars.

Live HTTP JSON endpoint

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: HTTP endpoint smoke test

prompts:
  - '{{message}}'

providers:
  - id: https
    label: live-chat-api
    config:
      url: '{{env.CHAT_API_URL}}'
      method: POST
      stateful: false
      headers:
        Content-Type: application/json
        Authorization: 'Bearer {{env.CHAT_API_TOKEN}}'
      body:
        message: '{{prompt}}'
      transformResponse: json.output

tests:
  - description: endpoint returns text
    vars:
      message: Say exactly PONG.
    assert:
      - type: contains
        value: PONG

Smoke with promptfoo eval -c promptfooconfig.yaml -o output.json --no-cache --no-share. Use stateful: false for stateless targets. If the app maintains conversation state, omit it and map {{sessionId}} into the request or configure sessionParser.

Live HTTP GET endpoint

Use queryParams instead of hand-building query strings when the target accepts GET parameters.

providers:
  - id: https
    label: search-api
    config:
      url: '{{env.SEARCH_API_URL}}'
      method: GET
      stateful: false
      headers:
        Authorization: 'Bearer {{env.SEARCH_API_TOKEN}}'
      queryParams:
        q: '{{prompt}}'
        user_id: '{{user_id}}'
      transformResponse: json.answer

OpenAI-compatible chat endpoint

Use an HTTP provider when the endpoint is OpenAI-shaped but not one of Promptfoo's native providers.

providers:
  - id: https
    label: openai-compatible-chat
    config:
      url: '{{env.CHAT_COMPLETIONS_URL}}'
      method: POST
      stateful: false
      headers:
        Content-Type: application/json
        Authorization: 'Bearer {{env.CHAT_COMPLETIONS_TOKEN}}'
      body:
        model: '{{env.CHAT_COMPLETIONS_MODEL}}'
        messages:
          - role: system
            content: Return concise answers.
          - role: user
            content: '{{prompt}}'
      transformResponse: json.choices[0].message.content

Text response endpoint

Use text in transformResponse when the target returns plain text.

providers:
  - id: https
    label: text-chat-api
    config:
      url: '{{env.TEXT_CHAT_API_URL}}'
      method: POST
      stateful: false
      headers:
        Content-Type: text/plain
      body: '{{prompt}}'
      transformResponse: text.replace(/^Assistant:\s*/, '')

OpenAPI operation to HTTP provider

Map one operation at a time: base URL to an env var, path parameters into the URL with urlencode, request/header/query fields into body/headers/queryParams, and the first successful response schema into transformResponse (prefer 200, otherwise the lowest explicit 2xx status).

The bundled scripts/openapi-operation-to-config.mjs helper supports local OpenAPI $refs plus allOf and first-variant oneOf/anyOf schemas. It lets operation parameters override path parameters, URL-encodes path/form values, skips readOnly request and writeOnly response fields even through $ref/composed schemas, keeps wire names intact, creates safe vars, preserves headers, and maps prompt fields (message, question, input, text, q, query) to {{prompt}}. With --token-env, it infers Bearer/OAuth2/OpenID/header/query/cookie API-key auth, uses parameter/media examples (including example-only bodies), +json media, text request bodies, form-url-encoded request bodies, structured multipart request bodies with generated file parts, typed/format schema samples from const/defaults/enums, root JSON array request bodies, schema/example-derived response transforms, health/status message vars, and --auth-header X-API-Key --auth-prefix none.

providers:
  - id: https
    config:
      url: '{{env.INVOICE_API_BASE_URL}}/v1/invoices/{{invoice_id}}/chat'
      method: POST
      headers:
        Authorization: 'Bearer {{env.INVOICE_API_TOKEN}}'
      body: { user_id: '{{user_id}}', message: '{{prompt}}' }
      transformResponse: json.output

Static-code-derived local wrappers

Use this for direct local code or custom setup. Pick JavaScript for Node apps and Python for Python app modules.

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Local agent provider smoke test
prompts:
  - '{{message}}'
providers:
  - id: file://provider.py:call_api # use file://provider.js for Node wrappers
    config:
      workers: 1
      timeout: 30000
tests:
  - description: local provider returns text
    vars:
      message: Say exactly PONG.
    assert:
      - type: contains
        value: PONG

export default class LocalAgentProvider {
  constructor(options = {}) {
    this.config = options.config || {};
  }

  id() {
    return 'local-agent';
  }

  async callApi(prompt, context = {}) {
    const vars = context.vars || {};
    const result = await callAgent({
      message: prompt,
      userId: vars.user_id || this.config.defaultUserId || 'validate-user',
    });

    return typeof result?.output === 'string'
      ? { output: result.output }
      : { error: 'Agent returned no string output' };
  }
}

import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
from app.invoice_agent import call_agent  # noqa: E402
def call_api(prompt: str, options: dict, context: dict) -> dict:
    config = options.get("config", {}) if isinstance(options, dict) else {}
    vars = context.get("vars", {}) if isinstance(context, dict) else {}
    user_id = vars.get("user_id") or config.get("defaultUserId") or "validate-user"
    result = call_agent(message=prompt, user_id=user_id)
    if not isinstance(result.get("output"), str):
        return {"error": "Agent returned no string output"}
    return {"output": result["output"]}

Python providers implement call_api(prompt, options, context) unless the id uses a custom file://provider.py:function_name suffix. Anchor sys.path to the provider directory before nearby app imports. Use PROMPTFOO_PYTHON or config.pythonExecutable for a venv, PROMPTFOO_PYTHON_WORKERS or config.workers for concurrency, and config.timeout for slow SDK calls. validate target may call providers without vars, so wrappers need harmless defaults.

Redteam target with named inputs

Use targets and preserve the app's real input fields. This gives redteam plugins access to the actual authorization and injection surface.

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Multi-input redteam target

targets:
  - id: https
    label: invoice-agent
    config:
      url: '{{env.INVOICE_AGENT_URL}}'
      method: POST
      stateful: false
      headers:
        Content-Type: application/json
        Authorization: 'Bearer {{env.INVOICE_AGENT_TOKEN}}'
      body:
        vendor_id: '{{vendor_id}}'
        invoice_id: '{{invoice_id}}'
        message: '{{message}}'
      transformResponse: json.output
    inputs:
      vendor_id: Vendor identifier for the signed-in user.
      invoice_id: Invoice identifier being discussed.
      message: User message to the assistant.

redteam:
  purpose: >-
    Invoice support assistant that answers invoice questions only for the
    authenticated vendor and must not reveal or modify other vendors' invoices.
  numTests: 3
  plugins:
    - bola
    - rbac
    - indirect-prompt-injection
  strategies:
    - basic

Do not set redteam.injectVar for multi-input mode. Define inputs on the target; Promptfoo automatically creates the internal combined __prompt value for generation and grading.

Hybrid discovery notes

Use this when static code and a live endpoint are both available. Record the contract before writing YAML:

• Static source: route/handler/client file and line range.
• Expected request: method, path, headers, body/query fields, and auth env var.
• Safe live probe: exact non-mutating payload and observed response field.
• Promptfoo mapping: vars to request fields and transformResponse.
• Open questions: mutations, session behavior, rate limits, or missing auth.

Prefer a local wrapper when the static path exposes app logic directly; prefer id: https when the live endpoint contract is simple and safely probeable.

Static discovery checklist

rg -n "app\\.(get|post|put|patch)|router\\.(get|post|put|patch)|fetch\\(|axios\\." .
rg -n "openapi|swagger|routes|controller|handler|chat|completion|agent" .
rg -n "Authorization|Bearer|apiKey|x-api-key|transformResponse|callApi" .