promptfoo--promptfoo/site/docs/usage/troubleshooting.md

---
sidebar_position: 60
description: Debug and resolve common promptfoo issues with solutions for memory optimization, API configuration, Node.js errors, native builds, and network/proxy setup in your LLM testing pipeline
---

# Troubleshooting

## Log Files and Debugging

Before troubleshooting specific issues, you can access detailed logs to help diagnose problems:

- **View logs directly**: Log files are stored in your config directory at `~/.promptfoo/logs` by default
- **Custom log directory**: Set the `PROMPTFOO_LOG_DIR` environment variable to write logs to a different directory (e.g., `PROMPTFOO_LOG_DIR=./logs promptfoo eval`)
- **Export logs for sharing**: Use `promptfoo export logs` to create a compressed archive of your log files for debugging or support

### Live Debug Toggle

During `promptfoo redteam run`, you can toggle debug logging on and off in real-time without restarting:

- **Press `v`** at any time to toggle verbose/debug output
- Works only in interactive terminal mode (not in CI or when output is piped)
- Useful for investigating issues mid-run without overwhelming log output

When you start a scan, you'll see:

```
  Tip: Press v to toggle debug output
```

Press `v` to enable debug logs and see detailed request metadata, provider response summaries, and grading status. Sensitive payloads, credentials, and raw response bodies may be redacted or summarized. Press `v` again to return to clean output.

:::tip
This is especially helpful when a scan seems stuck or you want to understand what's happening with a specific test case.
:::

## Out of memory error

If you have a large number of tests or your tests have large outputs, you may encounter an out of memory error. There are several ways to handle this:

### Basic setup

Follow **all** of these steps:

1. Do not use the `--no-write` flag. We need to write to disk to avoid memory issues.
2. Use the `--no-table` flag.
3. **Use JSONL format**: `--output results.jsonl`

:::tip
JSONL format processes results in batches, avoiding memory limits that cause JSON export to fail on large datasets.
:::

### Granular memory optimization

You can selectively strip out heavy data from the results using environment variables:

```bash
# Strip prompt text (useful if your prompts contain large amounts of text or images)
export PROMPTFOO_STRIP_PROMPT_TEXT=true

# Strip model outputs (useful if your model generates large responses)
export PROMPTFOO_STRIP_RESPONSE_OUTPUT=true

# Strip test variables (useful if your test cases contain large datasets)
export PROMPTFOO_STRIP_TEST_VARS=true

# Strip grading results (useful if you're using model-graded assertions)
export PROMPTFOO_STRIP_GRADING_RESULT=true

# Strip metadata (useful if you're storing large amounts of custom metadata)
export PROMPTFOO_STRIP_METADATA=true
```

You can use any combination of these variables to optimize memory usage while preserving the data you need.

### Increase Node.js memory limit

If you're still encountering memory issues after trying the above options, you can increase the amount of memory available to promptfoo by setting the `NODE_OPTIONS` environment variable:

```bash
# 8192 MB is 8 GB. Set this to an appropriate value for your machine.
NODE_OPTIONS="--max-old-space-size=8192" npx promptfoo eval
```

## Object template handling

When working with complex data structures in templates, you might encounter issues with how objects are displayed or accessed in your prompts and grading rubrics.

### `[object Object]` appears in outputs

If you see `[object Object]` in your LLM outputs or grading results, this means JavaScript objects are being converted to their string representation without proper serialization. By default, promptfoo automatically stringifies objects to prevent this issue.

**Example problem:**

```yaml
prompts:
  - 'Product: {{product}}'
tests:
  - vars:
      product:
        name: 'Headphones'
        price: 99.99
# Results in: "Product: [object Object]" in outputs
```

**Default solution:** Objects are automatically converted to JSON strings:

```text
Product: {"name":"Headphones","price":99.99}
```

### Accessing object properties in templates

If you need to access specific properties of objects in your templates (like `{{ product.name }}`), you can enable direct object access:

```bash
export PROMPTFOO_DISABLE_OBJECT_STRINGIFY=true
promptfoo eval
```

With this setting enabled, you can use object property access in templates:

```yaml
prompts:
  - 'Product: {{ product.name }} costs ${{ product.price }}'
# Results in: "Product: Headphones costs $99.99"
```

### When to use each approach

**Use default behavior (stringified objects) when:**

- You want objects as JSON strings in your prompts
- Working with existing templates that expect JSON strings
- You need maximum compatibility and don't want to see `[object Object]`

**Use object property access (`PROMPTFOO_DISABLE_OBJECT_STRINGIFY=true`) when:**

- You need to access specific properties like `{{ product.name }}`
- Building new templates designed for object navigation
- Working with complex nested data structures

<a id="nodejs-version-mismatch-error"></a>

## libsql binding not found

You may see this error if the libsql platform binding for your OS/architecture is
missing or wasn't installed:

```text
Error: Cannot find module '@libsql/darwin-arm64'
Require stack:
- /path/to/node_modules/libsql/index.js
```

libsql ships prebuilt N-API bindings as optional peer packages
(`@libsql/darwin-arm64`, `@libsql/linux-x64-gnu`, `@libsql/win32-x64-msvc`, etc.).
N-API is ABI-stable across Node.js versions, so this is almost always a packaging
issue (npm skipped optional deps, the cache is corrupt, or the platform isn't yet
supported), not a Node.js version mismatch.

Use the repair path that matches how you run promptfoo.

**Project checkout**

```bash
npm install
```

If `npm install` keeps skipping the binding, force optional dependencies on:

```bash
npm install --include=optional
```

**Global npm install**

```bash
npm install -g promptfoo@latest
```

**npx**

Remove the cached npx install, then re-run. With newer npm versions, drop only the
matching cache entry:

```bash
npm cache npx ls
npm cache npx rm <key>
npx -y promptfoo@latest
```

Use the key shown next to `promptfoo@latest` in `npm cache npx ls`.

**Unsupported platform**

If your `<platform>-<arch>` is not listed in the
[libsql release matrix](https://github.com/tursodatabase/libsql-js/releases),
file an issue at
[promptfoo/promptfoo](https://github.com/promptfoo/promptfoo/issues) with your
platform and architecture in the title.

## Native build failures

Some dependencies include native code that may need to compile locally. Ensure your machine has a C/C++ build toolchain:

- **Ubuntu/Debian**: `sudo apt-get install build-essential`
- **macOS**: `xcode-select --install`
- **Windows**: [Visual Studio Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/)

If the prebuilt binaries fail, force a local build:

```bash
npm install --build-from-source
# or
npm rebuild
```

## Network and proxy issues

If you're behind a corporate proxy or firewall and having trouble connecting to LLM APIs:

### Configure proxy settings

Set standard proxy environment variables before running promptfoo:

```bash
# Set proxy for HTTPS requests (most common)
export HTTPS_PROXY=http://proxy.company.com:8080

# With authentication if needed
export HTTPS_PROXY=http://username:password@proxy.company.com:8080

# Exclude specific hosts from proxying
export NO_PROXY=localhost,127.0.0.1,internal.domain.com
```

### Custom CA certificates

For environments with custom certificate authorities:

```bash
export PROMPTFOO_CA_CERT_PATH=/path/to/ca-bundle.crt
```

### Verify your configuration

Run `promptfoo debug` to see detected proxy settings and verify your network configuration is correct.

See the [FAQ](/docs/faq/#how-do-i-configure-promptfoo-for-corporate-networks-or-proxies) for complete proxy and SSL configuration details.

## OpenAI API key is not set

If you're using OpenAI, set the `OPENAI_API_KEY` environment variable or add `apiKey` to the provider config.

For default text-only model grading and synthesis, you can also install `@openai/codex-sdk`, sign in through the Codex CLI, and let Promptfoo use `openai:codex-sdk` automatically when no higher-priority API credentials are set. This does not cover embedding or moderation providers.

If you're not using OpenAI but still receiving this message, you probably have some [model-graded metric](/docs/configuration/expected-outputs/model-graded/) such as `similar` or `moderation` that requires you to [override the grader](/docs/configuration/expected-outputs/model-graded/#overriding-the-llm-grader) or configure an embedding/moderation provider explicitly.

Follow the instructions to override the grader, e.g., using the `defaultTest` property:

```yaml
defaultTest:
  options:
    provider:
      text:
        id: azureopenai:chat:gpt-4o-deployment
        config:
          apiHost: xxx.openai.azure.com
      embedding:
        id: azureopenai:embeddings:text-embedding-ada-002-deployment
        config:
          apiHost: xxx.openai.azure.com
```

## Python/JavaScript tool files require function name

If you see errors like `Python files require a function name` when loading tools from Python or JavaScript files, you need to specify the function name that returns the tool definitions.

### Solution

Python and JavaScript tool files must specify a function name using the `file://path:function_name` format:

```yaml title="promptfooconfig.yaml"
providers:
  - id: openai:chat:gpt-4.1-mini
    config:
      # Correct - specifies function name
      tools: file://./tools.py:get_tools
      # or for JavaScript/TypeScript
      tools: file://./tools.js:getTools
```

The function must return a tool definitions array (can be synchronous or asynchronous):

```python title="tools.py"
def get_tools():
    return [
        {
            "type": "function",
            "function": {
                "name": "get_current_weather",
                "description": "Get the current weather in a given location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "The city and state, e.g. San Francisco, CA"
                        }
                    },
                    "required": ["location"]
                }
            }
        }
    ]
```

```javascript title="tools.js"
function getTools() {
  return [
    {
      type: 'function',
      function: {
        name: 'get_current_weather',
        description: 'Get the current weather in a given location',
        parameters: {
          type: 'object',
          properties: {
            location: {
              type: 'string',
              description: 'The city and state, e.g. San Francisco, CA',
            },
          },
          required: ['location'],
        },
      },
    },
  ];
}

module.exports = { getTools };
```

## How to triage stuck evals

When running evals, you may encounter timeout errors, especially when using local providers or when running many concurrent requests. Here's how to fix them:

**Common use cases:**

- Ensure evaluations complete within a time limit (useful for CI/CD)
- Handle custom providers or providers that get stuck
- Prevent runaway costs from long-running evaluations

You can control two settings: timeout for individual test cases and timeout for the entire evaluation.

### Quick fixes

**Set timeouts for individual requests and total evaluation time:**

```bash
export PROMPTFOO_EVAL_TIMEOUT_MS=30000  # 30 seconds per request
export PROMPTFOO_MAX_EVAL_TIME_MS=300000  # 5 minutes total limit

npx promptfoo eval
```

You can also set these values in your `.env` file or Promptfoo config file:

```yaml title="promptfooconfig.yaml"
env:
  PROMPTFOO_EVAL_TIMEOUT_MS: 30000
  PROMPTFOO_MAX_EVAL_TIME_MS: 300000
```

## Debugging Python

When using custom Python providers, prompts, hooks, assertions, etc., you may need to debug your Python code. Here are some tips to help you troubleshoot issues:

### Viewing Python output

To see the output from your Python script, including print statements, set the `LOG_LEVEL` environment variable to `debug` when running your eval:

```bash
LOG_LEVEL=debug npx promptfoo eval
```

Alternatively, you can use the `--verbose` flag:

```bash
npx promptfoo eval --verbose
```

### Using the Python debugger (pdb)

Promptfoo now supports native Python debugging with pdb. To enable it:

```bash
export PROMPTFOO_PYTHON_DEBUG_ENABLED=true
```

Then add breakpoints in your Python code:

```python
import pdb

def call_api(prompt, options, context):
    pdb.set_trace()  # Debugger will pause here
    # Your code...
```

### Python Installation and Path Issues

If you encounter errors like `spawn py -3 ENOENT` or `Python 3 not found`, promptfoo cannot locate your Python installation. Here's how to resolve this:

#### Setting a Custom Python Path

Use the `PROMPTFOO_PYTHON` environment variable to specify your Python executable:

```bash
# Windows (if Python is installed at a custom location)
export PROMPTFOO_PYTHON=C:\Python\3_11\python.exe

# macOS/Linux
export PROMPTFOO_PYTHON=/usr/local/bin/python3

# Then run your evaluation
npx promptfoo eval
```

#### Per-Provider Python Configuration

You can also set the Python path for specific providers in your config:

```yaml
providers:
  - id: 'file://my_provider.py'
    config:
      pythonExecutable: /path/to/specific/python
```

#### Windows-Specific Issues

On Windows, promptfoo tries to detect Python in this order:

1. Provider-specific `pythonExecutable` config (if set)
2. `PROMPTFOO_PYTHON` environment variable (if set)
3. **Windows smart detection**: Uses `where python` command and filters out Microsoft Store stubs
4. `python -c "import sys; print(sys.executable)"` (to get the actual Python path)
5. Common fallback commands: `python`, `python3`, `py -3`, `py`

If you don't have the Python launcher (`py.exe`) installed but have Python directly, make sure the `python` command works from your command line. If not, either:

- Add your Python installation directory to your PATH
- Set `PROMPTFOO_PYTHON` to the full path of your `python.exe`

**Common Windows Python locations:**

- Microsoft Store: `%USERPROFILE%\AppData\Local\Microsoft\WindowsApps\python.exe`
- Direct installer: `C:\Python3X\python.exe` (where X is the version)
- Anaconda: `C:\Users\YourName\anaconda3\python.exe`

#### Testing Your Python Configuration

To verify your Python is correctly configured:

```bash
# Test that promptfoo can find your Python
python -c "import sys; print(sys.executable)"

# If this works but promptfoo still has issues, set PROMPTFOO_PYTHON:
export PROMPTFOO_PYTHON=$(python -c "import sys; print(sys.executable)")
```

### Handling errors

If you encounter errors in your Python script, the error message and stack trace will be displayed in the promptfoo output. Make sure to check this information for clues about what might be going wrong in your code.

Remember that promptfoo runs your Python script in a separate process, so some standard debugging techniques may not work as expected. Using logging and remote debugging as described above are the most reliable ways to troubleshoot issues in your Python providers.

## Debugging the Database

1. Set environment variables:

   ```bash
   export PROMPTFOO_ENABLE_DATABASE_LOGS=true
   export LOG_LEVEL=debug
   ```

2. Run your command:

   ```bash
   npx promptfoo eval
   ```

3. Disable logging when done:

   ```bash
   unset PROMPTFOO_ENABLE_DATABASE_LOGS
   ```

## Finding log files

Promptfoo logs errors and debug logs to `~/.promptfoo/logs` by default.

To inspect them from the CLI:

```bash
promptfoo logs --list
promptfoo logs <filename>
```

Change the location by setting `PROMPTFOO_LOG_DIR` to a different directory.

For each run an error log and a debug log will be created.