chore: import upstream snapshot with attribution
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Waiting to run
Test and Publish Multi-arch Docker Image / test (push) Waiting to run
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Blocked by required conditions
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Blocked by required conditions
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Blocked by required conditions
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Blocked by required conditions
Validate Renovate Config / Validate Renovate Configuration (push) Waiting to run
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled

This commit is contained in:
wehub-resource-sync
2026-07-13 13:24:08 +08:00
commit 0d3cb498a3
5438 changed files with 1316560 additions and 0 deletions
@@ -0,0 +1,12 @@
# integration-docker/code-generation-sandbox (Docker Code Generation Sandbox)
You can run this example with:
```bash
npx promptfoo@latest init --example integration-docker/code-generation-sandbox
cd integration-docker/code-generation-sandbox
```
## Usage
See https://promptfoo.dev/docs/guides/sandboxed-code-evals
@@ -0,0 +1,7 @@
You are a Python code generator. Write a Python function to solve the following problem:
{{problem}}
Use the following function name: {{function_name}}
Only provide the function code, without any explanations or additional text. Wrap your code in triple backticks.
@@ -0,0 +1,31 @@
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: 'Generate code with Claude in a Docker sandbox'
prompts:
- file://code_generation_prompt.txt
providers:
- anthropic:claude-sonnet-4-6
# - openai:gpt-4.1
tests:
- vars:
problem: 'Write a Python function to calculate the factorial of a number'
function_name: 'factorial'
test_input: '5'
expected_output: '120'
- vars:
problem: 'Write a Python function to check if a string is a palindrome'
function_name: 'is_palindrome'
test_input: "'racecar'"
expected_output: 'True'
- vars:
problem: 'Write a Python function to find the largest element in a list'
function_name: 'find_largest'
test_input: '[1, 5, 3, 9, 2]'
expected_output: '9'
defaultTest:
assert:
- type: python
value: file://validate_and_run_code.py
@@ -0,0 +1,10 @@
certifi==2025.10.5
charset-normalizer==3.3.2
docker==7.1.0
epicbox==1.1.0
idna==3.15
python-dateutil==2.9.0.post0
requests==2.33.0
six==1.16.0
structlog==24.2.0
urllib3==2.7.0
@@ -0,0 +1,60 @@
import re
import epicbox
# Replace with your preferred Docker image
DOCKER_IMAGE = "python:3.9-alpine"
def get_assert(output, context):
# Extract the Python function from the LLM output
function_match = re.search(r"```python\s*\n(def\s+.*?)\n```", output, re.DOTALL)
if not function_match:
return {"pass": False, "score": 0, "reason": "No function definition found"}
function_code = function_match.group(1)
# Configure epicbox
epicbox.configure(profiles=[epicbox.Profile("python", DOCKER_IMAGE)])
# Get the function name, test input, and expected output from the context
function_name = context["vars"]["function_name"]
test_input = context["vars"]["test_input"]
expected_output = context["vars"]["expected_output"]
# Prepare the code to run in the sandbox
test_code = f"""
{function_code}
# Test the function
result = {function_name}({test_input})
print(result)
"""
files = [{"name": "main.py", "content": test_code.encode("utf-8")}]
limits = {"cputime": 1, "memory": 64}
# Run the code in the sandbox
result = epicbox.run("python", "python main.py", files=files, limits=limits)
if result["exit_code"] != 0:
return {
"pass": False,
"score": 0,
"reason": f"Execution error: {result['stderr'].decode('utf-8')}",
}
# Compare the output with the expected result
actual_output = result["stdout"].decode("utf-8").strip()
if actual_output == str(expected_output):
return {
"pass": True,
"score": 1,
"reason": f"Correct output: got {expected_output}",
}
else:
return {
"pass": False,
"score": 0,
"reason": f"Incorrect output. Expected: {expected_output}, Got: {actual_output}",
}