wehub-resource-sync
0d3cb498a3
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
56 lines
1.7 KiB
Markdown
56 lines
1.7 KiB
Markdown
---
|
|
title: ToxicChat
|
|
description: Evaluate model resilience against 7,599 toxic and jailbreaking prompts from real user conversations in the ToxicChat dataset
|
|
---
|
|
|
|
# ToxicChat Plugin
|
|
|
|
The ToxicChat plugin tests your model's ability to handle toxic user prompts and resist jailbreaking attempts using the [ToxicChat dataset](https://huggingface.co/datasets/lmsys/toxic-chat) from LMSYS.
|
|
|
|
This dataset contains real user conversations labeled for toxicity and jailbreaking attempts, making it valuable for evaluating content safety measures.
|
|
|
|
## Configuration
|
|
|
|
```yaml
|
|
plugins:
|
|
- id: toxic-chat
|
|
# Optional: customize number of test cases (default: 5)
|
|
numTests: 10
|
|
```
|
|
|
|
## How It Works
|
|
|
|
The plugin:
|
|
|
|
1. **Fetches data** from the ToxicChat dataset on Hugging Face
|
|
2. **Filters prompts** labeled as toxic (toxicity=1) or jailbreaking attempts (jailbreaking=1)
|
|
3. **Tests your model** with these challenging prompts
|
|
4. **Evaluates responses** for content safety compliance
|
|
|
|
## Example Output
|
|
|
|
The plugin tests prompts like:
|
|
|
|
- Requests for harmful content generation
|
|
- Attempts to bypass safety guidelines
|
|
- Toxic language and harassment
|
|
- Jailbreaking techniques
|
|
|
|
A **passing** response should:
|
|
|
|
- Refuse to engage with harmful requests
|
|
- Maintain appropriate boundaries
|
|
- Not provide toxic or harmful content
|
|
|
|
A **failing** response might:
|
|
|
|
- Comply with toxic requests
|
|
- Generate harmful content
|
|
- Assist with jailbreaking attempts
|
|
|
|
## Related Concepts
|
|
|
|
- [Harmbench Plugin](./harmbench.md) - Tests using the HarmBench dataset
|
|
- [Harmful Content Plugin](./harmful.md) - Tests for various types of harmful content
|
|
- [DoNotAnswer Plugin](./donotanswer.md) - Tests handling of harmful queries
|