promptfoo--promptfoo/site/docs/contributing.md

---
title: Contributing to promptfoo
sidebar_label: Contributing
description: Contribute to promptfoo by submitting code, documentation, providers, and features following our development guidelines
---

We welcome contributions from the community to help make promptfoo better. This guide will help you get started. If you have any questions, please reach out to us on [Discord](https://discord.gg/promptfoo) or through a [GitHub issue](https://github.com/promptfoo/promptfoo/issues/new).

## Project Overview

Promptfoo is an MIT-licensed tool for testing and evaluating LLM apps.

### How to Contribute

There are several ways to contribute to promptfoo:

1. **Submit Pull Requests**: Anyone can contribute by forking the repository and submitting pull requests. You don't need to be a collaborator to contribute code or documentation changes.

2. **Report Issues**: Help us by reporting bugs or suggesting improvements through GitHub issues or [Discord](https://discord.gg/promptfoo).

3. **Improve Documentation**: Documentation improvements are always welcome, including fixing typos, adding examples, or writing guides.

We particularly welcome contributions in the following areas:

- Bug fixes
- Documentation updates, including examples and guides
- Updates to providers including new models, new capabilities (tool use, function calling, JSON mode, file uploads, etc.)
- Features that improve the user experience of promptfoo, especially relating to RAGs, Agents, and synthetic data generation.

## Getting Started

1. Fork the repository on GitHub by clicking the "Fork" button at the top right of the [promptfoo repository](https://github.com/promptfoo/promptfoo).
2. Clone your fork locally:

   ```bash
   git clone https://github.com/[your-username]/promptfoo.git
   cd promptfoo
   ```

3. Set up your development environment:

   3.1. Setup locally

   ```bash
   # Use the Node.js version specified in .nvmrc
   nvm use

   # If you're using npm, match the major used in CI so release-age policy is applied
   npm install -g npm@11

   # Install dependencies (CI uses npm; pnpm and yarn also work for local development)
   npm install
   ```

   3.2 Set up using `devcontainer` (requires Docker and VSCode)

   Open the repository in VSCode and click on the "Reopen in Container" button. This will build a Docker container with all the necessary dependencies.

   Now install node based dependencies:

   ```bash
   npm install -g npm@11
   npm install
   ```

4. Run the tests to make sure everything is working:

   ```bash
   npm test
   ```

5. Build the project:

   ```bash
   npm run build
   ```

6. Run the project:

   ```bash
   npm run dev
   ```

   This will run the express server on port 15500 and the web UI on port 3000.
   Both the API and UI will be automatically reloaded when you make changes.

   :::info

   The development experience is a little bit different than how it runs in production. In development, the web UI is served using a Vite server. In all other environments, the front end is built and served as a static site via the Express server.

   :::

If you're not sure where to start, check out our [good first issues](https://github.com/promptfoo/promptfoo/issues?q=state%3Aopen%20label%3Agood-first-issue%20is%3Aissue) or join our [Discord community](https://discord.gg/promptfoo) for guidance.

## Development Workflow

1. Create a new branch for your feature or bug fix:

   ```bash
   git checkout -b feature/your-feature-name
   ```

2. Make your changes and commit them. We follow the [Conventional Commits](https://www.conventionalcommits.org/) specification for PR titles when merging into `main`. Individual commits can use any format, since we squash merge all PRs with a conventional commit message.

   :::note

   All pull requests are squash-merged with a conventional commit message.

   :::

3. Push your branch to your fork:

   ```bash
   git push origin your-branch-name
   ```

4. [Open a pull request](https://github.com/promptfoo/promptfoo/compare) (PR) against the `main` branch of the promptfoo repository.

When opening a pull request:

- Keep changes small and focused. Avoid mixing refactors with new features.
- Ensure test coverage for new code or bug fixes.
- Provide clear instructions on how to reproduce the problem or test the new feature.
- Be responsive to feedback and be prepared to make changes if requested.
- Ensure your tests are passing and your code is properly linted and formatted. You can do this by running `npm run lint -- --fix` and `npm run format` respectively.

:::tip

If you're unsure about how to implement something, feel free to open a draft PR to get early feedback.

:::

Don't hesitate to ask for help. We're here to support you. If you're worried about whether your PR will be accepted, please talk to us first (see [Getting Help](#getting-help)).

## Tests

### Running Tests

We use both Vitest and Jest. To run the test suite:

```bash
npm test
```

To run tests in watch mode:

```bash
npm run test:watch
```

You can also run specific tests with:

```bash
# Vitest
npx vitest [pattern]

# Jest
npx jest [pattern]

# Example:
# Runs all provider tests
npx vitest providers
```

### Writing Tests

When writing tests, please:

- **Use Vitest for new test files.** When modifying existing files, use whichever framework that file uses.
- Ensure proper test isolation by:
  - Using `beforeEach` and `afterEach` to set up and clean up mocks
  - Calling `vi.clearAllMocks()` or `vi.restoreAllMocks()` for Vitest (or `jest.clearAllMocks()` for Jest)
  - Avoiding shared state between tests
- Check the coverage report to ensure your changes are covered.
- Avoid adding additional logs to the console.

## Linting and Formatting

We use Biome for JavaScript/TypeScript linting and formatting, and Prettier for CSS/HTML/Markdown. Before submitting a pull request, please run:

```bash
npm run format
npm run lint
```

It's a good idea to run the lint command as `npm run lint -- --fix` to automatically fix some linting errors.

## Building the Project

To build the project:

```bash
npm run build
```

For continuous building of the API during development:

```bash
npm run build:watch
```

## Contributing to the CLI

### Running the CLI During Development

We recommend using `npm link` to link your local `promptfoo` package to the global `promptfoo` command:

```bash
npm link
promptfoo eval --config examples/provider-cloudflare/ai/chat_config.yaml
```

Alternatively, you can use `npm run local` to run your local build directly:

```bash
npm run local -- eval --config examples/provider-cloudflare/ai/chat_config.yaml
```

**Important:** Always use `--` before flags so they're passed to promptfoo, not npm.

When working on a new feature, we recommend setting up a local `promptfooconfig.yaml` that tests your feature. Think of this as an end-to-end test for your feature.

Here's a simple example:

```yaml title="promptfooconfig.yaml"
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
providers:
  - id: openai:gpt-5
prompts:
  - Translate "{{input}}" to {{language}}
tests:
  - vars:
      input: 'Hello, world!'
      language: 'English'
    assert:
      - type: new-assertion-type
```

## Adding a New Provider

Providers are defined in TypeScript. We also provide language bindings for Python and Go.

Before writing code, check whether the service already works through the `openai` provider. For an OpenAI-compatible endpoint, set `apiBaseUrl`, `OPENAI_API_BASE_URL`, or `OPENAI_BASE_URL`. If that is all the service needs, contribute docs or an example that uses the generic path; do not add a new provider prefix or registry entry.

A _dedicated provider_ adds its own `id:` prefix and registry entry. Promptfoo maintains that code, and a dedicated name signals a baseline of trust to users. To keep the registry useful, dedicated providers must meet the following bar.

### Provider eligibility

- **Legitimate model access.** The provider must serve models it owns, models its users are licensed to run, or models it can access through a first-party API, partner agreement, or reseller agreement. We do not accept unauthorized relays that resell or proxy another provider's models. If you resell third-party models, be ready to confirm that you are permitted to do so.
- **Accountable ownership.** A hosted service must have an identifiable operator, a maintenance contact, and its own terms of service and privacy policy. For a self-hosted open-source project, established maintainers and a clear maintenance or security contact satisfy this requirement. Anonymous services with no identifiable owner are not a good fit.
- **Evidence it will last.** We look for a practical signal that the provider will be maintained, such as funding, paying customers, an established track record, or substantial _organic_ adoption (real usage, not inflated stars or followers). Brand-new services may be asked to wait or start with the generic `openai` path.
- **Clear data handling.** Hosted services must document whether they log prompt or response content, how long they retain it, and whether they use it for training. Self-hosted projects must document whether they send data outside the user's deployment and what telemetry they collect.
- **Value beyond the generic path.** A dedicated provider should add behavior that would otherwise require provider-specific setup or code, such as non-OpenAI authentication, required headers, custom routing, or provider-specific endpoints and response handling. A base URL and API-key environment variable alone are not enough.

We review aggregators, gateways, and resellers case by case against this same bar. Meeting these criteria doesn't guarantee a merge, and we may decline or later remove a provider that stops meeting them.

To contribute a dedicated provider:

1. Confirm it meets the eligibility criteria above and doesn't already exist in Promptfoo.

2. Implement the provider in `src/providers/yourProviderName.ts` following our [Custom API Provider Docs](/docs/providers/custom-api/). Please use our cache `src/cache.ts` to store responses. If your provider requires a new dependency, please add it as an optional dependency.

3. Write unit tests in `test/providers/yourProviderName.test.ts` and create an example in the `examples/` directory.

4. Document your provider in `site/docs/providers/yourProviderName.md`, including a description, setup instructions, configuration options, and usage examples. You can also add examples to the `examples/` directory. Consider writing a guide comparing your provider to others or highlighting unique features or benefits.

5. Update `src/providers/registry.ts` and `site/docs/providers/index.md` to include your new provider. Update `src/envars.ts` to include any new environment variables your provider may need.

6. Ensure all tests pass (`npm test`) and fix any linting issues (`npm run lint`).

## Adding a New Assertion

Assertions define different ways to compare and validate the output of an LLM against expected results. To contribute a new assertion:

1. **Define the Assertion Type**:
   - Add your new assertion type to the `BaseAssertionTypesSchema` enum in `src/types/index.ts`.
   - Run `npm run jsonSchema:generate` to update the JSON schema located at `site/static/config-schema.json`

2. **Implement the Assertion Handler**:
   - Create a new file in `src/assertions/` for your assertion logic.
   - Implement a handler function that takes `AssertionParams` and returns a `GradingResult`.

   Basic handler structure:

   ```typescript
   import type { AssertionParams, GradingResult } from '../types';
   import invariant from '../util/invariant';

   export function handleYourAssertion({
     assertion,
     inverse,
     outputString,
     renderedValue,
     provider, // Use if your assertion needs provider-specific logic
     test,     // Access to test case data
   }: AssertionParams): GradingResult {
     // Validate inputs
     invariant(
       typeof renderedValue === 'string' || Array.isArray(renderedValue),
       '"your-assertion" assertion must have a string or array value'
     );

     // Implementation logic
     const threshold = assertion.threshold ?? 0.5; // Set a sensible default

     // Calculate the score
     const score = /* your scoring logic */;

     // Determine if test passes
     const pass = (score >= threshold) !== inverse;

     return {
       pass,
       score: inverse ? 1 - score : score,
       reason: pass
         ? 'Assertion passed'
         : `Your assertion scored ${score.toFixed(2)} vs threshold ${threshold}`,
       assertion,
     };
   }
   ```

3. **Register the Assertion Handler**:
   - In `src/assertions/index.ts`, import your handler function and add it to the handlers mapping.

   ```typescript
   import { handleYourAssertion } from './yourAssertion';

   // In the handlers mapping
   'your-assertion': handleYourAssertion,
   ```

4. **Document Your Assertion**:
   - Update the appropriate documentation files:
     - For standard assertions, add details to `site/docs/configuration/expected-outputs/deterministic.md`
     - Include your assertion in the reference table in `site/docs/configuration/expected-outputs/index.md`

   - For model-graded assertions:
     - Add an entry to the list in `site/docs/configuration/expected-outputs/model-graded/index.md`
     - Create a dedicated documentation page at `site/docs/configuration/expected-outputs/model-graded/your-assertion.md`

5. **Write Tests**:
   - Create a test file in `test/assertions/yourAssertion.test.ts`.
   - Test scenarios including:
     - Standard use cases
     - Edge cases and error handling
     - Provider-specific behavior (if applicable)
     - Schema validation (if applicable)
     - Backward compatibility (if refactoring existing assertions)

## Contributing to the Web UI

The web UI is written as a React app. It is exported as a static site and hosted by a local express server when bundled.

To run the web UI in dev mode:

```bash
npm run dev
```

This will host the web UI at http://localhost:3000. This allows you to hack on the React app quickly (with fast refresh). If you want to run the web UI without the express server, you can run:

```bash
npm run dev:app
```

To test the entire thing end-to-end, build the project and run it:

```bash
npm run build
promptfoo view
# Or: npm run dev
```

:::note

This will not update the web UI if you make further changes to the code. You have to run `npm run build` again.

:::

## Python Contributions

While promptfoo is primarily written in TypeScript, we support custom Python prompts, providers, asserts, and many examples in Python. We strive to keep our Python codebase simple and minimal, without external dependencies. Please adhere to these guidelines:

- Use Python 3.9 or later
- For linting and formatting, use `ruff`. Run `ruff check --fix` and `ruff format` before submitting changes
- Follow the [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html)
- Use type hints to improve code readability and catch potential errors
- Write unit tests for new Python functions using the built-in `unittest` module
- When adding new Python dependencies to an example, update the relevant `requirements.txt` file

## Documentation

If you're adding new features or changing existing ones, please update the relevant documentation. We use [Docusaurus](https://docusaurus.io/) for our documentation. We strongly encourage examples and guides as well.

### Documentation Standards

Our documentation follows several standards to ensure accessibility:

- **Human-readable**: Clean, well-structured markdown with clear navigation
- **LLM-friendly**: Automated generation of [LLMs.txt files](https://llmstxt.org) for AI tool integration
- **Searchable**: Proper headings, tags, and cross-references
- **Example-driven**: Real-world examples and use cases

### Development Workflow

To run the documentation in development mode:

```bash
cd site
npm start
```

This will start the Docusaurus development server on port 3100 by default (or a custom port if you set the `PORT` environment variable). You can then view the documentation at http://localhost:3100.

To build the documentation for production:

```bash
cd site
npm run build
```

This will generate static content in the `build` directory that can be served using any static content hosting service. Building the documentation may occasionally catch errors that do not surface when running `npm start`.

## Advanced Topics

### Database

Promptfoo uses SQLite as its default database, managed through the Drizzle ORM. By default, the database is stored in `~/.promptfoo/`. You can override this location by setting `PROMPTFOO_CONFIG_DIR`. The database schema is defined in `src/database/tables.ts` and migrations are stored in `drizzle/`. Note that the migrations are all generated and you should not access these files directly.

#### Main Tables

- `evals`: Stores eval details including results and configuration.
- `prompts`: Stores information about different prompts.
- `datasets`: Stores dataset information and test configurations.
- `evalsToPrompts`: Manages the relationship between evals and prompts.
- `evalsToDatasets`: Manages the relationship between evals and datasets.

You can view the contents of each of these tables by running `npx drizzle-kit studio`, which will start a web server.

#### Adding a Migration

1. **Modify Schema**: Make changes to your schema in `src/database/tables.ts`.
2. **Generate Migration**: Run the command to create a new migration:

   ```bash
   npm run db:generate
   ```

   This command will create a new SQL file in the `drizzle/` directory.

3. **Review Migration**: Inspect the generated migration file to ensure it captures your intended changes.
4. **Apply Migration**: Apply the migration with:

   ```bash
   npm run db:migrate
   ```

### Release Process

This project uses [release-please](https://github.com/googleapis/release-please) for automated releases:

1. Merge PRs to main with conventional commit messages
2. release-please creates/updates a release PR with changelog entries, version bumps, and updated `package.json`
3. When merged, a GitHub release is created and the npm package is published

For urgent releases, contact maintainers on [Discord](https://discord.gg/promptfoo).

## AI-Assisted Contributions

We welcome AI-assisted development. Use whatever tools help you ship better work.

We don't judge contributions by how they were produced. We judge them by the quality of the result. The same standards apply regardless of whether you wrote every character by hand or used Copilot, Cursor, Claude Code, or any other tool.

:::tip You own the change

If you open a pull request, you are responsible for what you submit. AI can help you write code faster, but it cannot take accountability for correctness, security, or maintainability. That stays with you.

:::

### Prove the change works

A pull request should include clear instructions so that a reviewer can verify correctness by running the code locally. Don't make reviewers reverse-engineer how to test your change.

**What to include:**

- What changed and why (a few sentences is fine)
- Steps to reproduce or verify the change (commands, config files, expected output)
- Any tradeoffs or edge cases you considered

If the change is hard to demonstrate, include something concrete: a minimal repro, a log excerpt, a screenshot, or a short recording.

### Be able to explain the diff

You don't need to have typed every character. You do need to be able to answer reviewer questions about your changes:

- Why is this implemented this way?
- What alternatives did you consider?
- What did you test?
- What could break?

If you can't explain a non-trivial part of your PR, revise it until you can.

### No hypothetical platform support

Don't submit changes for environments you cannot actually run or test. If you want to add support for a platform you don't have access to, open an issue or discussion describing what you need and what you can verify today.

### Maintain agent guidance files

This repository uses `AGENTS.md` files to provide context to AI coding assistants. If your changes affect how agents should work in a particular area (new patterns, conventions, or gotchas), update the relevant `AGENTS.md` file following existing patterns. See the root `AGENTS.md` for the directory structure and conventions.

### Disclosure

Disclosure of AI usage is optional. We won't try to guess whether you used AI.

That said, if a maintainer asks about your process, please answer candidly. If an agent produced a large portion of the change, a short note describing how you validated the output can help reviewers and speed up the process.

## Changelog

**Do not manually edit `CHANGELOG.md`.** release-please generates entries from conventional commit messages.

Use [Conventional Commits](https://www.conventionalcommits.org/) format for PR titles (they become squash-merge commit messages).

## Getting Help

If you need help or have questions, you can:

- Open an issue on GitHub.
- Join our [Discord community](https://discord.gg/promptfoo).

## Code of Conduct

We follow the [Contributor Covenant Code of Conduct](https://www.contributor-covenant.org/). Please read and adhere to it in all interactions within our community.