chore: import upstream snapshot with attribution

examples/integration-e2b/README.md

@@ -0,0 +1,71 @@
+# integration-e2b (E2B Code Evaluation)
+
+## What This Example Demonstrates
+
+This example shows a complete prompt→LLM→sandboxed-execution→metric pipeline using:
+
+- `promptfoo` to run LLM prompts and manage evaluation cases.
+- An LLM provider to generate Python functions from a short problem prompt.
+- e2b sandboxes (via `e2b-code-interpreter`) to run generated code safely.
+- OpenAI step to generate small verification unit tests and re-run them in the sandbox.
+- Per-run JSON metrics written to .promptfoo_results/ and a human-friendly markdown report produced by report.py.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example integration-e2b
+cd integration-e2b
+```
+
+## Environment Variables
+
+Set these in your shell before running the example.
+
+```bash
+# Required
+export E2B_API_KEY="e2b_xxx_your_key_here"        # e2b sandbox API key
+export OPENAI_API_KEY="sk_xxx_your_key_here"     # OpenAI key (or your chosen LLM provider)
+
+# Recommended
+export PROMPTFOO_PYTHON="$(pwd)/.venv/bin/python"  # tell promptfoo which Python/venv to use
+```
+
+- If you use a different provider name in promptfooconfig.yaml, add that provider's key instead.
+
+## Prerequisites
+
+Install and prepare a Python virtual environment, and install the required packages.
+
+```bash
+# create & activate venv
+python -m venv .venv
+source .venv/bin/activate
+
+# install Python packages
+pip install --upgrade pip
+pip install e2b-code-interpreter
+npm i -g promptfoo
+```
+
+## Running the Example
+
+Activate venv and ensure env vars are set:
+
+```bash
+source .venv/bin/activate
+export E2B_API_KEY="e2b_xxx"
+export OPENAI_API_KEY="sk_xxx"
+export PROMPTFOO_PYTHON="$(pwd)/.venv/bin/python"
+```
+
+Run the evaluation:
+
+```bash
+promptfoo eval
+```
+
+Open the interactive viewer:
+
+```bash
+promptfoo view
+```

examples/integration-e2b/code_generation_prompt_fs.txt

@@ -0,0 +1,14 @@
+You are a careful Python engineer. Produce only the Python function definition (no explanation) wrapped in triple backticks.
+
+Requirements:
+- Use the exact function name: {{function_name}}
+- Avoid external network calls or reading system files.
+- Prefer idiomatic, clear Python.
+- Raise ValueError for clearly invalid inputs when applicable.
+- Keep runtime reasonable for typical inputs.
+
+Few-shot examples:
+Problem: "Sum a list of integers"
+```python
+def sum_integers(nums):
+    return sum(nums)

examples/integration-e2b/generate_unit_test_prompt.txt

@@ -0,0 +1,3 @@
+You are a Python test author. Given the function below, write 3 short `assert` tests (not using pytest) that exercise edge cases and typical cases. Return only the tests (no explanations). Function:
+
+{{function_code}}

examples/integration-e2b/metrics.py

@@ -0,0 +1,27 @@
+# metrics.py
+import json
+import os
+import time
+
+
+def write_metrics(
+    task_id, provider, model, success, runtime_s, memory_mb=None, extra=None
+):
+    metrics = {
+        "task_id": str(task_id),
+        "provider": provider,
+        "model": model,
+        "success": bool(success),
+        "runtime_s": float(runtime_s),
+        "memory_mb": memory_mb,
+        "extra": extra or {},
+        "timestamp": int(time.time()),
+    }
+    out_dir = os.getenv("PROMPTFOO_RESULTS_DIR", ".promptfoo_results")
+    os.makedirs(out_dir, exist_ok=True)
+    fn = f"{metrics['task_id']}_{provider}_{model}.json".replace("/", "_")
+    path = os.path.join(out_dir, fn)
+    with open(path, "w") as f:
+        # <- this makes ExecutionError, Exception, etc. become strings
+        json.dump(metrics, f, indent=2, default=str)
+    return path

examples/integration-e2b/patch_generation_prompt.txt

@@ -0,0 +1,13 @@
+You are a careful engineer. Given a failing test log and repository file contents, produce a minimal unified diff patch that fixes the failing test.
+
+Format:
+- Output strictly a unified diff (git apply-ready).
+- Do not include explanation text.
+
+Inputs:
+
+Failing test log:
+{{fail_log}}
+
+Relevant files (path => content):
+{{relevant_files}}

examples/integration-e2b/promptfooconfig.yaml

@@ -0,0 +1,55 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'E2B Code Eval'
+
+prompts: file://code_generation_prompt_fs.txt
+
+providers:
+  - openai:gpt-4.1
+
+tests:
+  # 1. Normal factorial test (existing)
+  - vars:
+      problem: 'Write a Python function to calculate the factorial of a number'
+      function_name: 'factorial'
+      test_input: '5'
+      expected_output: '120'
+
+  # 2. Palindrome check (existing)
+  - vars:
+      problem: 'Write a Python function to check if a string is a palindrome'
+      function_name: 'is_palindrome'
+      test_input: "'racecar'"
+      expected_output: 'True'
+
+  # 3. Find largest element (existing)
+  - vars:
+      problem: 'Write a Python function to find the largest element in a list'
+      function_name: 'find_largest'
+      test_input: '[1, 5, 3, 9, 2]'
+      expected_output: '9'
+
+  # ⚠️ 4. Unsafe behavior test — should be blocked by validation
+  - vars:
+      problem: 'Write a Python function that lists files in the current directory'
+      function_name: 'list_files'
+      test_input: "'.'"
+      expected_output: 'SAFE_ERROR_DETECTED' # The validator should block this
+
+  # 🧪 5. Invalid input test — checks if ValueError is raised properly
+  - vars:
+      problem: 'Write a Python function to calculate the factorial of a number'
+      function_name: 'factorial'
+      test_input: '-3' # Invalid input
+      expected_output: 'ValueError' # Function should raise ValueError
+
+  # ❌ 6. Failing test case — checks if system flags wrong logic
+  - vars:
+      problem: 'Write a Python function to reverse a string'
+      function_name: 'reverse_string'
+      test_input: "'hello'"
+      expected_output: "'olleh'" # If model fails here, system should mark it failed
+
+defaultTest:
+  assert:
+    - type: python
+      value: file://validate_and_run_code_e2b.py

examples/integration-e2b/report.py

@@ -0,0 +1,31 @@
+# report.py
+import glob
+import json
+import os
+
+
+def gen_report(out_dir=".promptfoo_results", out_md="promptfoo_report.md"):
+    files = glob.glob(os.path.join(out_dir, "*.json"))
+    rows = []
+    for f in files:
+        try:
+            with open(f, encoding="utf-8") as result_file:
+                j = json.load(result_file)
+            rows.append(j)
+        except Exception:
+            continue
+    lines = [
+        "| task | provider | model | success | runtime_s |",
+        "|---|---|---|---|---|",
+    ]
+    for r in rows:
+        lines.append(
+            f"|{r.get('task_id', '-')}|{r.get('provider', '-')}|{r.get('model', '-')}|{r.get('success')}|{r.get('runtime_s')}|"
+        )
+    with open(out_md, "w", encoding="utf-8") as report_file:
+        report_file.write("\n".join(lines))
+    print("Wrote", out_md)
+
+
+if __name__ == "__main__":
+    gen_report()

examples/integration-e2b/swe_runner.py

@@ -0,0 +1,136 @@
+# swe_runner.py (e2b-only skeleton)
+import os
+import shutil
+import subprocess
+import tempfile
+from textwrap import dedent
+
+from e2b_code_interpreter import Sandbox
+
+# Optional: use OpenAI if available for real patch generation
+try:
+    from openai import OpenAI  # pip install openai
+except Exception:
+    OpenAI = None
+
+
+def model_call_patch(fail_log: str, relevant_files: dict) -> str:
+    """
+    Minimal working example:
+    - If OPENAI_API_KEY is set and the OpenAI SDK is available, ask the model for a unified diff.
+    - Otherwise, return a tiny, valid unified diff against the first relevant file (demo fallback).
+    """
+
+    # 1) Try LLM-backed patch generation (only if configured)
+    if OpenAI and os.getenv("OPENAI_API_KEY"):
+        client = OpenAI()
+        model = os.getenv("PATCH_MODEL", "gpt-4o-mini")
+
+        sys_msg = (
+            "You are a code repair assistant. Output ONLY a unified diff patch that "
+            "can be applied with `git apply -p0`. Do NOT include any explanations."
+        )
+        files_blob = "\n\n".join(
+            f"{path}:\n{content}" for path, content in relevant_files.items()
+        )
+        user_msg = (
+            "Given the failing test log and the repository files, produce a minimal fix:\n\n"
+            f"Failing test log:\n{fail_log}\n\n"
+            f"Relevant files (path => content):\n{files_blob}\n"
+        )
+
+        resp = client.chat.completions.create(
+            model=model,
+            temperature=0,
+            messages=[
+                {"role": "system", "content": sys_msg},
+                {"role": "user", "content": user_msg},
+            ],
+        )
+        if not resp.choices or resp.choices[0].message is None:
+            raise ValueError("LLM returned empty or filtered response")
+        patch = resp.choices[0].message.content.strip()
+        return patch
+
+    # 2) Fallback: return a harmless, valid unified diff so the example runs without any API keys
+    if not relevant_files:
+        demo_path = "README.md"
+        old = ""
+    else:
+        demo_path = next(iter(relevant_files.keys()))
+        old = relevant_files[demo_path]
+
+    # Minimal unified diff; uses -p0 friendly paths (a/ and b/)
+    fallback_patch = dedent(f"""\
+    --- a/{demo_path}
+    +++ b/{demo_path}
+    @@
+    {old or ""}
+    +# patched-by-demo
+    """)
+    return fallback_patch
+
+
+def run_task_local(
+    repo_url: str, failing_test_cmd: str, relevant_paths: list, attempts=3
+):
+    tmp = tempfile.mkdtemp()
+    try:
+        subprocess.check_call(["git", "clone", repo_url, tmp])
+        cwd = os.getcwd()
+        os.chdir(tmp)
+        # capture failing test
+        p = subprocess.run(failing_test_cmd, shell=True, capture_output=True, text=True)
+        fail_log = p.stdout + "\n" + p.stderr
+        relevant_files = {}
+        for path in relevant_paths:
+            try:
+                with open(os.path.join(tmp, path), "r") as f:
+                    relevant_files[path] = f.read()
+            except Exception:
+                relevant_files[path] = ""
+        success = False
+        for attempt in range(attempts):
+            try:
+                patch = model_call_patch(fail_log, relevant_files)
+            except Exception as e:
+                print("Model call failed:", e)
+                break
+            # apply patch locally
+            proc = subprocess.run(
+                ["git", "apply", "-"], input=patch, text=True, capture_output=True
+            )
+            if proc.returncode != 0:
+                print("Patch apply failed:", proc.stderr)
+                continue
+            # run tests inside e2b sandbox (safe)
+            with Sandbox.create() as sbx:
+                test_script = f"import subprocess; print(subprocess.run('{failing_test_cmd}', shell=True, capture_output=True).returncode)"
+                res = sbx.run_code(
+                    code=test_script,
+                    language="python",
+                    limits={"cputime": 10, "wall_time": 30, "memory": 512},
+                    allow_network=False,
+                )
+            rc = None
+            try:
+                if getattr(res, "results", None):
+                    r0 = res.results[0]
+                    val = r0.get("stdout") or r0.get("output") or r0.get("text") or ""
+                    if isinstance(val, list):
+                        val = "".join(map(str, val))
+                    rc = int(str(val).strip())
+                elif getattr(res, "text", None):
+                    rc = int(str(res.text).strip())
+            except Exception:
+                rc = None
+            if rc == 0:
+                success = True
+                print("Patch fixed tests on attempt", attempt + 1)
+                break
+            else:
+                print("Patch did not fix tests; rc:", rc)
+        os.chdir(cwd)
+        return {"success": success}
+    finally:
+        shutil.rmtree(tmp, ignore_errors=True)

examples/integration-e2b/validate_and_run_code_e2b.py

@@ -0,0 +1,284 @@
+# validate_and_run_code_e2b.py
+import json
+import logging
+import re
+import time
+
+from e2b_code_interpreter import Sandbox
+from metrics import write_metrics
+
+# Robust fenced code regex
+FENCE_RE = re.compile(r"```(?:\s*python)?\s*\r?\n(.*?)```", re.DOTALL | re.IGNORECASE)
+
+# Unsafe patterns: pre-exec static scanner
+UNSAFE_PATTERNS = [
+    r"\bimport\s+socket\b",
+    r"\bimport\s+requests\b",
+    r"\bimport\s+urllib\b",
+    r"\bos\.system\b",
+    r"\bsubprocess\b",
+    r"\beval\s*\(",
+    r"\bexec\s*\(",
+    r"open\(\s*['\"]\/etc",
+    r"open\(\s*['\"]\/proc",
+    r"__import__\(",
+]
+
+logger = logging.getLogger(__name__)
+
+
+def is_unsafe(code: str) -> bool:
+    """Return True when generated code contains blocked imports or APIs."""
+    for p in UNSAFE_PATTERNS:
+        if re.search(p, code):
+            return True
+    return False
+
+
+def _extract_function(output: str, fn_name: str) -> str | None:
+    """Extract the generated function from fenced code or plain text output."""
+    m = FENCE_RE.search(output)
+    if m:
+        return m.group(1).strip()
+
+    by_name = re.search(
+        rf"(def\s+{re.escape(fn_name)}\s*\(.*?\)\s*:[\s\S]*?)(?=\n\s*\n|^```|^class\s+|^def\s+)",
+        output,
+        re.IGNORECASE | re.MULTILINE,
+    )
+    if by_name:
+        return by_name.group(1).strip()
+
+    any_def = re.search(r"(def\s+\w+\s*\(.*?\)\s*:[\s\S]*)", output)
+    if any_def:
+        return any_def.group(1).strip()
+    return None
+
+
+def _try_parse_logs_obj(logs_obj):
+    """Try multiple ways to extract stdout/stderr from logs object or string."""
+    if isinstance(logs_obj, (dict, list)):
+        if isinstance(logs_obj, dict) and "stdout" in logs_obj:
+            out = logs_obj.get("stdout")
+            err = logs_obj.get("stderr", "")
+            if isinstance(out, list):
+                out = "".join(map(str, out))
+            if isinstance(err, list):
+                err = "".join(map(str, err))
+            return str(out).strip(), str(err).strip()
+        if isinstance(logs_obj, list):
+            outs, errs = [], []
+            for it in logs_obj:
+                if isinstance(it, dict):
+                    if it.get("stdout"):
+                        outs.append(it.get("stdout"))
+                    if it.get("stderr"):
+                        errs.append(it.get("stderr"))
+            if outs or errs:
+                outs = "".join(map(str, outs))
+                errs = "".join(map(str, errs))
+                return str(outs).strip(), str(errs).strip()
+
+    if hasattr(logs_obj, "to_json"):
+        try:
+            j = logs_obj.to_json()
+            return _try_parse_logs_obj(j)
+        except Exception:
+            logger.debug("Failed to parse sandbox logs via to_json()", exc_info=True)
+
+    if hasattr(logs_obj, "stdout") or hasattr(logs_obj, "stderr"):
+        try:
+            out = getattr(logs_obj, "stdout", "")
+            err = getattr(logs_obj, "stderr", "")
+            if isinstance(out, list):
+                out = "".join(map(str, out))
+            if isinstance(err, list):
+                err = "".join(map(str, err))
+            return str(out).strip(), str(err).strip()
+        except Exception:
+            logger.debug("Failed to parse sandbox log stream fields", exc_info=True)
+
+    if isinstance(logs_obj, str):
+        try:
+            parsed = json.loads(logs_obj)
+            return _try_parse_logs_obj(parsed)
+        except Exception:
+            logger.debug("Failed to decode sandbox logs as JSON", exc_info=True)
+            m = re.search(r"stdout:\s*\[([^\]]*)\]", logs_obj)
+            if m:
+                inner = m.group(1).strip()
+                items = re.findall(r"'(.*?)'|\"(.*?)\"", inner)
+                strs = []
+                for a, b in items:
+                    if a:
+                        strs.append(a)
+                    elif b:
+                        strs.append(b)
+                out = "".join(strs)
+                return out.strip(), ""
+    return "", ""
+
+
+def _stdout_from_result(res) -> tuple[str, str]:
+    """Return stdout and stderr strings from an E2B execution result."""
+    if hasattr(res, "results") and res.results:
+        try:
+            first = res.results[0]
+            if isinstance(first, dict):
+                out = (
+                    first.get("stdout")
+                    or first.get("output")
+                    or first.get("text")
+                    or ""
+                )
+                err = first.get("stderr") or ""
+                if isinstance(out, list):
+                    out = "".join(map(str, out))
+                if isinstance(err, list):
+                    err = "".join(map(str, err))
+                if out or err:
+                    return str(out).strip(), str(err).strip()
+        except Exception:
+            logger.debug("Failed to parse sandbox result entries", exc_info=True)
+
+    logs_field = getattr(res, "logs", None)
+    if logs_field is not None:
+        out, err = _try_parse_logs_obj(logs_field)
+        if out or err:
+            return out, err
+
+    if getattr(res, "text", None):
+        return str(res.text).strip(), ""
+
+    if hasattr(res, "to_json"):
+        try:
+            j = res.to_json()
+            return _try_parse_logs_obj(j)
+        except Exception:
+            logger.debug(
+                "Failed to serialize sandbox result via to_json()", exc_info=True
+            )
+
+    return "", ""
+
+
+def _run_code_in_sandbox(sbx, code: str):
+    """Try a few run_code signatures safely (SDKs vary)."""
+    # Preferred: named args (limits, disable network)
+    try:
+        return sbx.run_code(
+            code=code,
+            language="python",
+            limits={"cputime": 1, "wall_time": 5, "memory": 128},
+            allow_network=False,
+        )
+    except TypeError:
+        pass
+    except Exception:
+        logger.debug("Preferred sandbox run_code signature failed", exc_info=True)
+    # Try alternate signatures
+    try:
+        return sbx.run_code(
+            code, "python", {"cputime": 1, "wall_time": 5, "memory": 128}
+        )
+    except Exception:
+        logger.debug("Positional sandbox run_code signature failed", exc_info=True)
+    # Last resort
+    return sbx.run_code(code)
+
+
+def get_assert(output, context):
+    """Execute the generated function in E2B and compare stdout with expected output."""
+    task_id = context.get("id", str(time.time()))
+    provider = context.get("provider", "unknown")
+    model = context.get("model", "unknown")
+
+    fn_name = context["vars"]["function_name"]
+    test_input = context["vars"]["test_input"]
+    expected = str(context["vars"]["expected_output"])
+
+    function_code = _extract_function(output, fn_name)
+    if not function_code:
+        snippet = output.strip()[:300].replace("\n", " ")
+        write_metrics(
+            task_id, provider, model, False, 0.0, extra={"reason": "no_code_found"}
+        )
+        return {
+            "pass": False,
+            "score": 0,
+            "reason": f"No Python code block found (first 300 chars: {snippet})",
+        }
+
+    # Pre-exec safety
+    if is_unsafe(function_code):
+        write_metrics(
+            task_id, provider, model, False, 0.0, extra={"reason": "unsafe_pattern"}
+        )
+        return {
+            "pass": False,
+            "score": 0,
+            "reason": "Unsafe pattern detected in generated code",
+        }
+
+    test_program = f"""{function_code}
+
+print({fn_name}({test_input}))
+"""
+
+    start = time.time()
+    with Sandbox.create() as sbx:
+        try:
+            res = _run_code_in_sandbox(sbx, test_program)
+        except Exception as e:
+            duration = time.time() - start
+            write_metrics(
+                task_id, provider, model, False, duration, extra={"error": str(e)}
+            )
+            return {
+                "pass": False,
+                "score": 0,
+                "reason": f"Sandbox execution error: {e}",
+            }
+
+    duration = time.time() - start
+    stdout, stderr = _stdout_from_result(res)
+    err_obj = getattr(res, "error", None)
+
+    if err_obj or stderr:
+        dbg = ""
+        try:
+            dbg = (
+                getattr(res, "to_json")()
+                if hasattr(res, "to_json")
+                else str(getattr(res, "logs", ""))[:800]
+            )
+        except Exception:
+            dbg = str(getattr(res, "logs", ""))[:800]
+        write_metrics(
+            task_id,
+            provider,
+            model,
+            False,
+            duration,
+            extra={"error": err_obj or stderr, "debug": dbg},
+        )
+        return {
+            "pass": False,
+            "score": 0,
+            "reason": f"Execution error: {err_obj or stderr} | debug: {dbg}",
+        }
+
+    success = stdout == expected
+    write_metrics(
+        task_id, provider, model, success, duration, extra={"stdout": stdout[:500]}
+    )
+    if success:
+        return {"pass": True, "score": 1, "reason": f"Correct output: {stdout}"}
+    logs_preview = getattr(res, "logs", None)
+    if isinstance(logs_preview, str) and len(logs_preview) > 400:
+        logs_preview = logs_preview[:400] + "...(truncated)"
+    return {
+        "pass": False,
+        "score": 0,
+        "reason": f"Expected {expected}, got {stdout or '(empty)'} | logs: {logs_preview}",
+    }