chore: import upstream snapshot with attribution
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Waiting to run
Test and Publish Multi-arch Docker Image / test (push) Waiting to run
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Blocked by required conditions
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Blocked by required conditions
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Blocked by required conditions
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Blocked by required conditions
Validate Renovate Config / Validate Renovate Configuration (push) Waiting to run
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Waiting to run
Test and Publish Multi-arch Docker Image / test (push) Waiting to run
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Blocked by required conditions
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Blocked by required conditions
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Blocked by required conditions
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Blocked by required conditions
Validate Renovate Config / Validate Renovate Configuration (push) Waiting to run
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,12 @@
|
||||
# integration-docker/code-generation-sandbox (Docker Code Generation Sandbox)
|
||||
|
||||
You can run this example with:
|
||||
|
||||
```bash
|
||||
npx promptfoo@latest init --example integration-docker/code-generation-sandbox
|
||||
cd integration-docker/code-generation-sandbox
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
See https://promptfoo.dev/docs/guides/sandboxed-code-evals
|
||||
@@ -0,0 +1,7 @@
|
||||
You are a Python code generator. Write a Python function to solve the following problem:
|
||||
|
||||
{{problem}}
|
||||
|
||||
Use the following function name: {{function_name}}
|
||||
|
||||
Only provide the function code, without any explanations or additional text. Wrap your code in triple backticks.
|
||||
@@ -0,0 +1,31 @@
|
||||
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
|
||||
description: 'Generate code with Claude in a Docker sandbox'
|
||||
|
||||
prompts:
|
||||
- file://code_generation_prompt.txt
|
||||
|
||||
providers:
|
||||
- anthropic:claude-sonnet-4-6
|
||||
# - openai:gpt-4.1
|
||||
|
||||
tests:
|
||||
- vars:
|
||||
problem: 'Write a Python function to calculate the factorial of a number'
|
||||
function_name: 'factorial'
|
||||
test_input: '5'
|
||||
expected_output: '120'
|
||||
- vars:
|
||||
problem: 'Write a Python function to check if a string is a palindrome'
|
||||
function_name: 'is_palindrome'
|
||||
test_input: "'racecar'"
|
||||
expected_output: 'True'
|
||||
- vars:
|
||||
problem: 'Write a Python function to find the largest element in a list'
|
||||
function_name: 'find_largest'
|
||||
test_input: '[1, 5, 3, 9, 2]'
|
||||
expected_output: '9'
|
||||
|
||||
defaultTest:
|
||||
assert:
|
||||
- type: python
|
||||
value: file://validate_and_run_code.py
|
||||
@@ -0,0 +1,10 @@
|
||||
certifi==2025.10.5
|
||||
charset-normalizer==3.3.2
|
||||
docker==7.1.0
|
||||
epicbox==1.1.0
|
||||
idna==3.15
|
||||
python-dateutil==2.9.0.post0
|
||||
requests==2.33.0
|
||||
six==1.16.0
|
||||
structlog==24.2.0
|
||||
urllib3==2.7.0
|
||||
@@ -0,0 +1,60 @@
|
||||
import re
|
||||
|
||||
import epicbox
|
||||
|
||||
# Replace with your preferred Docker image
|
||||
DOCKER_IMAGE = "python:3.9-alpine"
|
||||
|
||||
|
||||
def get_assert(output, context):
|
||||
# Extract the Python function from the LLM output
|
||||
function_match = re.search(r"```python\s*\n(def\s+.*?)\n```", output, re.DOTALL)
|
||||
if not function_match:
|
||||
return {"pass": False, "score": 0, "reason": "No function definition found"}
|
||||
|
||||
function_code = function_match.group(1)
|
||||
|
||||
# Configure epicbox
|
||||
epicbox.configure(profiles=[epicbox.Profile("python", DOCKER_IMAGE)])
|
||||
|
||||
# Get the function name, test input, and expected output from the context
|
||||
function_name = context["vars"]["function_name"]
|
||||
test_input = context["vars"]["test_input"]
|
||||
expected_output = context["vars"]["expected_output"]
|
||||
|
||||
# Prepare the code to run in the sandbox
|
||||
test_code = f"""
|
||||
{function_code}
|
||||
|
||||
# Test the function
|
||||
result = {function_name}({test_input})
|
||||
print(result)
|
||||
"""
|
||||
|
||||
files = [{"name": "main.py", "content": test_code.encode("utf-8")}]
|
||||
limits = {"cputime": 1, "memory": 64}
|
||||
|
||||
# Run the code in the sandbox
|
||||
result = epicbox.run("python", "python main.py", files=files, limits=limits)
|
||||
|
||||
if result["exit_code"] != 0:
|
||||
return {
|
||||
"pass": False,
|
||||
"score": 0,
|
||||
"reason": f"Execution error: {result['stderr'].decode('utf-8')}",
|
||||
}
|
||||
|
||||
# Compare the output with the expected result
|
||||
actual_output = result["stdout"].decode("utf-8").strip()
|
||||
if actual_output == str(expected_output):
|
||||
return {
|
||||
"pass": True,
|
||||
"score": 1,
|
||||
"reason": f"Correct output: got {expected_output}",
|
||||
}
|
||||
else:
|
||||
return {
|
||||
"pass": False,
|
||||
"score": 0,
|
||||
"reason": f"Incorrect output. Expected: {expected_output}, Got: {actual_output}",
|
||||
}
|
||||
Reference in New Issue
Block a user