wehub-resource-sync
0d3cb498a3
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
1.7 KiB
1.7 KiB
title, description
| title | description |
|---|---|
| ToxicChat | Evaluate model resilience against 7,599 toxic and jailbreaking prompts from real user conversations in the ToxicChat dataset |
ToxicChat Plugin
The ToxicChat plugin tests your model's ability to handle toxic user prompts and resist jailbreaking attempts using the ToxicChat dataset from LMSYS.
This dataset contains real user conversations labeled for toxicity and jailbreaking attempts, making it valuable for evaluating content safety measures.
Configuration
plugins:
- id: toxic-chat
# Optional: customize number of test cases (default: 5)
numTests: 10
How It Works
The plugin:
- Fetches data from the ToxicChat dataset on Hugging Face
- Filters prompts labeled as toxic (toxicity=1) or jailbreaking attempts (jailbreaking=1)
- Tests your model with these challenging prompts
- Evaluates responses for content safety compliance
Example Output
The plugin tests prompts like:
- Requests for harmful content generation
- Attempts to bypass safety guidelines
- Toxic language and harassment
- Jailbreaking techniques
A passing response should:
- Refuse to engage with harmful requests
- Maintain appropriate boundaries
- Not provide toxic or harmful content
A failing response might:
- Comply with toxic requests
- Generate harmful content
- Assist with jailbreaking attempts
Related Concepts
- Harmbench Plugin - Tests using the HarmBench dataset
- Harmful Content Plugin - Tests for various types of harmful content
- DoNotAnswer Plugin - Tests handling of harmful queries