chore: import upstream snapshot with attribution

examples/integration-inspect-osworld/.gitignore

@@ -0,0 +1,9 @@
+inspect_logs/
+*.eval
+*.json
+*.jsonl
+screenshots/
+*.png
+*.jpg
+*.jpeg
+*.webp

examples/integration-inspect-osworld/README.md

@@ -0,0 +1,260 @@
+# integration-inspect-osworld (OSWorld via Inspect)
+
+This example runs a real [OSWorld](https://github.com/xlang-ai/OSWorld) task through promptfoo by wrapping the Inspect-native implementation in [`inspect_evals/osworld`](https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/osworld). OSWorld is a multimodal computer-use benchmark where an agent observes an Ubuntu desktop via screenshots, acts with mouse and keyboard tools, and is graded by task-specific checks against VM state. The benchmark is described in [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://arxiv.org/abs/2404.07972).
+
+This is an orchestration wrapper, not a from-scratch promptfoo-native computer-use agent loop. Inspect owns the Docker sandbox, `basic_agent` solver, `computer` tool, screenshots, model calls, and OSWorld scorer. Promptfoo starts one Inspect eval, dumps the `.eval` log to JSON, parses the final score, and applies a normal promptfoo assertion.
+
+## Prerequisites
+
+You need:
+
+- Docker Engine 24.0.6 or newer, running and usable by your current user.
+- Docker Compose V2 available as `docker compose`. Inspect validates this with
+  `docker compose version --format json`; a standalone `docker-compose` binary
+  is not enough unless your `docker` command exposes it as `docker compose`.
+- Python with Inspect's OSWorld dependencies, Promptfoo's Python OpenTelemetry
+  dependencies, and the SDK for whichever model provider you choose. This
+  installs both SDKs used below:
+
+  ```bash
+  pip install 'inspect-evals[osworld]' openai anthropic opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
+  ```
+
+- A computer-use-capable model and API key. For the default config, export
+  `OPENAI_API_KEY`. To use Anthropic instead, export `ANTHROPIC_API_KEY` and set
+  `vars.model` or `providers[0].config.defaultModel` to an Inspect model such as
+  `anthropic/claude-sonnet-4-5`.
+- Disk and time for Inspect's OSWorld Docker image. The first run builds an image of roughly 8GB and can take several minutes before the sample starts.
+- Budget for a non-trivial model run. Start with one exact sample before expanding
+  to a larger subset or the full suite.
+
+The default config uses `inspect_evals/osworld_small`, the smaller OSWorld corpus
+supported by Inspect. `promptfooconfig.full.yaml` switches to
+`inspect_evals/osworld` with `include_connected=true`, which loads every
+Inspect-supported full-corpus sample. In the Inspect version used for this
+example, that is 246 samples, not the 369-task upstream OSWorld paper corpus.
+
+## Run
+
+For the first real verification from the repository root, run one exact sample:
+
+```bash
+PROMPTFOO_ENABLE_OTEL=true OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 \
+  npm run local -- eval -c examples/integration-inspect-osworld/promptfooconfig.yaml --no-cache \
+    --filter-metadata sample_id=42e0a640-4f19-4b28-973d-729602b5a4a7
+```
+
+Or, after copying the example with `npx promptfoo@latest init --example integration-inspect-osworld`, run:
+
+```bash
+PROMPTFOO_ENABLE_OTEL=true OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 \
+  promptfoo eval -c promptfooconfig.yaml --no-cache \
+    --filter-metadata sample_id=42e0a640-4f19-4b28-973d-729602b5a4a7
+```
+
+After that succeeds, broaden to an app subset:
+
+```bash
+PROMPTFOO_ENABLE_OTEL=true OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 \
+  promptfoo eval -c promptfooconfig.yaml --no-cache \
+    --filter-metadata app=libreoffice_calc --max-concurrency 1 \
+    -o osworld-libreoffice-calc.json
+```
+
+App filters are still multi-sample runs. In the current `osworld_small` set,
+`app=libreoffice_calc` selects three samples; in one local GPT-5.5 verification
+on April 29, 2026, that sequential subset took 12m31s and used 533,101 total
+tokens. Treat that as scale guidance, not a fixed benchmark.
+
+To run the full supported small suite, remove the metadata filter and set a
+concurrency appropriate for your machine:
+
+```bash
+PROMPTFOO_ENABLE_OTEL=true OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 \
+  promptfoo eval -c promptfooconfig.yaml --no-cache --max-concurrency 6 \
+    -o osworld-results.json
+```
+
+To run Inspect's full supported corpus through Promptfoo, use the dedicated
+full-suite config:
+
+```bash
+PROMPTFOO_ENABLE_OTEL=true OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 \
+  promptfoo eval -c promptfooconfig.full.yaml --no-cache --max-concurrency 3 \
+    -o osworld-full-results.json
+```
+
+That config keeps the same wrapper but switches both moving pieces that define
+the run:
+
+```yaml
+providers:
+  - id: file://provider.py
+    config:
+      task: inspect_evals/osworld
+      taskParameters:
+        include_connected: true
+
+tests: file://osworld_tests.py:generate_full_tests
+```
+
+Because the full config includes connected samples, it is more sensitive to the
+runtime network environment than the default small-suite config.
+
+The full config also uses larger timeouts than the small config:
+
+- `timeout: 7500000` gives Promptfoo's Python worker a little over two hours.
+- `timeoutSeconds: 7200` gives the inner Inspect subprocess two hours.
+
+Some full-suite Writer rows can exceed the small config's 30-minute timeout
+budget, so keep the full-suite timeouts larger than the exact-sample and
+small-suite defaults.
+
+`promptfooconfig.yaml` keeps the shared assertion and tracing metadata in
+`defaultTest`, then asks `osworld_tests.py` to generate the OSWorld rows:
+
+```yaml
+defaultTest:
+  metadata:
+    tracingEnabled: true
+  assert:
+    - type: python
+      value: file://assertion.py
+
+tests: file://osworld_tests.py:generate_tests
+```
+
+The loader calls Inspect's `osworld_small().dataset` or
+`osworld(include_connected=True).dataset` and returns one Promptfoo test case per
+supported sample. Each row sets `vars.prompt`, `vars.app`, `vars.sample_id`, and
+matching filterable metadata. Because Inspect supplies the sample ids, updating
+`inspect-evals` updates the generated row list without maintaining a local copy.
+
+The default config runs the full Inspect-supported `osworld_small` suite, which
+is 21 samples in the version used for the reference run below. To run a broader
+subset after the exact-sample check, filter by app metadata:
+
+```bash
+PROMPTFOO_ENABLE_OTEL=true OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 \
+  promptfoo eval -c promptfooconfig.yaml --no-cache --filter-metadata app=libreoffice_calc
+```
+
+To run the smallest real end-to-end validation, filter by `sample_id`:
+
+```bash
+PROMPTFOO_ENABLE_OTEL=true OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 \
+  promptfoo eval -c promptfooconfig.yaml --no-cache \
+    --filter-metadata sample_id=42e0a640-4f19-4b28-973d-729602b5a4a7
+```
+
+For custom subsets, filter by metadata at the CLI. The generated metadata uses
+OSWorld app ids, normalizes VS Code to `vscode`, and keeps multi-app tasks under
+`multi_apps`.
+
+Use the run scopes intentionally:
+
+1. `mockllm/model --limit 0` checks the Inspect CLI shape without model spend.
+2. `--filter-metadata sample_id=...` is the smallest real end-to-end validation.
+3. `--filter-metadata app=...` is a broader app slice and may include multiple samples.
+4. No filter on `promptfooconfig.yaml` runs the full small suite.
+5. `promptfooconfig.full.yaml` runs Inspect's full supported corpus and is the benchmark-style configuration.
+
+The example config sets two timeouts because both layers need enough time:
+
+- `providers[0].config.timeout` is promptfoo's Python worker timeout in
+  milliseconds.
+- `providers[0].config.timeoutSeconds` is the inner Inspect subprocess timeout
+  in seconds.
+
+## Expected output
+
+The provider returns text like:
+
+```text
+Sample <id> on app libreoffice_calc: score=1.0 status=pass
+
+Final answer: <agent final message if Inspect logged one>
+```
+
+It also returns metadata for the promptfoo UI and assertions:
+
+```json
+{
+  "inspect_log_path": "/absolute/path/to/examples/integration-inspect-osworld/inspect_logs/.../*.eval",
+  "score": 1.0,
+  "status": "pass",
+  "sample_id": "...",
+  "model": "openai/gpt-5.5",
+  "num_messages": 42,
+  "duration_seconds": 600.0
+}
+```
+
+The Python assertion passes when `metadata.score >= 1.0` or `metadata.status == "pass"`.
+
+If Inspect exits before a scored sample is available, or if the selected sample
+has no OSWorld scorer result, the provider returns an error instead of converting
+that condition into a benchmark failure. For subprocess failures, Promptfoo stores
+only a concise error plus the local log path/status/duration; inspect the local
+Inspect logs when you need the detailed trajectory or raw tool output.
+
+## Reference GPT-5.5 run
+
+A local traced run of the generated 21-sample suite with exact `sample_id`
+selectors and `--max-concurrency 6` completed in 20m 9s. GPT-5.5 passed 13
+samples and produced 7 scored failures. One concurrent run hit an Inspect
+computer-tool runtime error before scoring; rerunning that exact `sample_id`
+alone with `--max-concurrency 1` completed normally with score `0.0`. After
+that rerun, the report had 13 passes, 8 scored failures, 0 provider errors, and
+mean OSWorld score `0.665`. Promptfoo recorded 21 trace records and 21
+Python provider spans for the concurrent run.
+
+For larger benchmark reports, rerun provider-error samples by exact `sample_id`
+before publishing a pass rate. Count reruns that produce an OSWorld score as
+normal passes or failures, and keep repeated provider errors separate from
+scored benchmark failures.
+
+For the full supported corpus, a local GPT-5.5 run on April 30, 2026 used
+`promptfooconfig.full.yaml`, `--max-concurrency 3`, and a 6-vCPU / 16-GiB
+Colima VM. The 246-sample run took 5h27m5s and used 54,421,072 total tokens.
+The raw run ended at 138 passes, 101 scored failures, and 7 provider errors.
+Rerunning those seven rows one at a time recovered one pass and two ordinary
+scored failures; four rows repeated as provider errors. The reconciled report
+was therefore 139 passes, 103 scored failures, 4 provider errors, and mean
+OSWorld score `0.594` across the 242 scored rows. The seven targeted reruns
+added 1,917,890 tokens.
+
+The repeated provider errors were not model failures: one row reproduced an
+Inspect computer-tool runtime error, one row reproduced an OSWorld scorer
+missing-image-artifact error, and two VLC rows reproduced an OSWorld scorer
+environment error. Keep those rows outside the scored denominator unless a
+later rerun produces an OSWorld score.
+
+## Inspect logs and traces
+
+Inspect writes `.eval` files under `examples/integration-inspect-osworld/inspect_logs/`. They are ignored by git because they can include screenshots, trajectories, tool calls, model outputs, and other large run artifacts.
+
+For trace-level visibility into the OSWorld desktop trajectory, use Inspect's
+viewer:
+
+```bash
+inspect view --log-dir examples/integration-inspect-osworld/inspect_logs
+```
+
+The example config enables Promptfoo OpenTelemetry tracing. Set
+`PROMPTFOO_ENABLE_OTEL=true` for Python provider spans. This records the
+Python provider call and links it to the eval result, but it does not translate
+Inspect's internal screenshots, mouse moves, keyboard actions, or scorer events
+into Promptfoo trajectory spans. Use Inspect's `.eval` log for those steps.
+
+## Smoke test without model spend
+
+To check the Inspect CLI shape without running a full OSWorld sample:
+
+```bash
+inspect eval inspect_evals/osworld_small --model mockllm/model --limit 0 --log-dir <dir>
+inspect log dump <file.eval>
+```
+
+A real end-to-end OSWorld run still requires Docker, the first-run image build,
+and provider credentials. Use one exact sample before spending on larger slices.

examples/integration-inspect-osworld/assertion.py

@@ -0,0 +1,24 @@
+"""Promptfoo assertion for the OSWorld Inspect wrapper."""
+
+from __future__ import annotations
+
+from typing import Any
+
+
+def get_assert(output: str, context: dict[str, Any]) -> dict[str, Any]:
+    del output
+    provider_response = context.get("providerResponse") or {}
+    metadata = provider_response.get("metadata") or {}
+    status = metadata.get("status")
+    score = metadata.get("score")
+
+    passed = status == "pass" or (isinstance(score, (int, float)) and score >= 1.0)
+    sample_id = metadata.get("sample_id", "unknown sample")
+    log_path = metadata.get("inspect_log_path", "unknown log")
+    reason = f"OSWorld sample {sample_id} score={score} status={status}; Inspect log: {log_path}"
+
+    return {
+        "pass": passed,
+        "score": 1.0 if passed else 0.0,
+        "reason": reason,
+    }

examples/integration-inspect-osworld/osworld_tests.py

@@ -0,0 +1,86 @@
+"""Generate Promptfoo test cases from Inspect's OSWorld dataset."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+CONTAINER_EXAMPLE_PATH = "/tmp/osworld/desktop_env/example.json"
+
+
+def generate_tests():
+    """Return one Promptfoo test case per Inspect-supported small-suite sample."""
+
+    dataset = _osworld_small_dataset()
+    return [_test_case(dataset[index]) for index in range(len(dataset))]
+
+
+def generate_full_tests():
+    """Return one Promptfoo test case per Inspect-supported full-suite sample."""
+
+    dataset = _osworld_full_dataset()
+    return [_test_case(dataset[index]) for index in range(len(dataset))]
+
+
+def _osworld_small_dataset():
+    try:
+        from inspect_evals.osworld import osworld_small
+    except ImportError as exc:
+        raise RuntimeError(
+            "Could not import inspect_evals.osworld. Install prerequisites with "
+            "`pip install 'inspect-evals[osworld]'` before loading OSWorld tests."
+        ) from exc
+
+    return osworld_small().dataset
+
+
+def _osworld_full_dataset():
+    try:
+        from inspect_evals.osworld import osworld
+    except ImportError as exc:
+        raise RuntimeError(
+            "Could not import inspect_evals.osworld. Install prerequisites with "
+            "`pip install 'inspect-evals[osworld]'` before loading OSWorld tests."
+        ) from exc
+
+    return osworld(include_connected=True).dataset
+
+
+def _test_case(sample):
+    sample_id = str(sample.id)
+    instruction = str(sample.input)
+    app = _normalize_app(_example_path(sample).parent.name)
+    return {
+        "description": f"{app} - {_short_label(instruction)}",
+        "vars": {
+            "prompt": instruction,
+            "app": app,
+            "sample_id": sample_id,
+        },
+        "metadata": {
+            "app": app,
+            "sample_id": sample_id,
+            "testCaseId": f"osworld-{app.replace('_', '-')}-{sample_id.split('-', 1)[0]}",
+        },
+    }
+
+
+def _example_path(sample):
+    try:
+        return Path(str(sample.files[CONTAINER_EXAMPLE_PATH]))
+    except (AttributeError, KeyError, TypeError) as exc:
+        raise RuntimeError(
+            f"Inspect OSWorld sample {getattr(sample, 'id', '<unknown>')} did not "
+            f"include {CONTAINER_EXAMPLE_PATH} in sample.files."
+        ) from exc
+
+
+def _short_label(instruction, max_length=80):
+    label = " ".join(str(instruction).split())
+    if len(label) <= max_length:
+        return label
+    return f"{label[: max_length - 3].rstrip()}..."
+
+
+def _normalize_app(app):
+    normalized = str(app).strip().lower().replace("-", "_").replace(" ", "_")
+    return "vscode" if normalized == "vs_code" else normalized

examples/integration-inspect-osworld/promptfooconfig.full.yaml

@@ -0,0 +1,33 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Full OSWorld via Inspect benchmark with tracing
+prompts:
+  - '{{prompt}}'
+providers:
+  - id: file://provider.py
+    label: OSWorld via Inspect
+    config:
+      defaultModel: openai/gpt-5.5
+      task: inspect_evals/osworld
+      taskParameters:
+        include_connected: true
+      # Promptfoo's Python worker timeout, in milliseconds.
+      timeout: 7500000
+      # Inspect subprocess timeout, in seconds.
+      timeoutSeconds: 7200
+defaultTest:
+  metadata:
+    tracingEnabled: true
+  assert:
+    - type: python
+      value: file://assertion.py
+tests: file://osworld_tests.py:generate_full_tests
+tracing:
+  enabled: true
+  otlp:
+    http:
+      enabled: true
+      port: 4318
+      host: 127.0.0.1
+      acceptFormats:
+        - json
+        - protobuf

examples/integration-inspect-osworld/promptfooconfig.yaml

@@ -0,0 +1,30 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: OSWorld via Inspect wrapper with tracing
+prompts:
+  - '{{prompt}}'
+providers:
+  - id: file://provider.py
+    label: OSWorld via Inspect
+    config:
+      defaultModel: openai/gpt-5.5
+      # Promptfoo's Python worker timeout, in milliseconds.
+      timeout: 1800000
+      # Inspect subprocess timeout, in seconds.
+      timeoutSeconds: 1800
+defaultTest:
+  metadata:
+    tracingEnabled: true
+  assert:
+    - type: python
+      value: file://assertion.py
+tests: file://osworld_tests.py:generate_tests
+tracing:
+  enabled: true
+  otlp:
+    http:
+      enabled: true
+      port: 4318
+      host: 127.0.0.1
+      acceptFormats:
+        - json
+        - protobuf

examples/integration-inspect-osworld/provider.py

@@ -0,0 +1,478 @@
+"""Promptfoo provider that orchestrates inspect_evals' OSWorld task.
+
+This is intentionally wrapper-shaped: Inspect owns the computer-use agent loop,
+Docker sandbox, screenshots, tool calls, and OSWorld scorer. Promptfoo starts one
+Inspect eval, dumps the Inspect `.eval` log to JSON, and exposes the final score.
+
+Observed `inspect log dump` shape with Inspect 0.3.213:
+  data["results"]["scores"][0]["metrics"]["accuracy"]["value"] -> aggregate pass rate
+  data["samples"][0]["scores"][<scorer_name>]["value"] -> per-sample score
+  data["samples"][0]["output"]["completion"] -> final assistant text when present
+  data["stats"]["model_usage"][<model>] -> token counts
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import shlex
+import shutil
+import subprocess
+import time
+import uuid
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+
+DEFAULT_MODEL = "openai/gpt-5.5"
+DEFAULT_TIMEOUT_SECONDS = 1800
+DEFAULT_TASK = "inspect_evals/osworld_small"
+OSWORLD_SCORER = "osworld_scorer"
+
+
+def call_api(prompt: str, options: dict, context: dict) -> dict:
+    """Run one OSWorld sample through inspect_evals and return Promptfoo output."""
+    del prompt
+    config = options.get("config") or {}
+    vars_ = context.get("vars") or {}
+    requested_sample_id = vars_.get("sample_id")
+    app = vars_.get("app")
+    if not requested_sample_id:
+        return {
+            "error": (
+                "OSWorld sample_id is required. Use osworld_tests.py or set "
+                "vars.sample_id."
+            )
+        }
+
+    model = vars_.get("model") or config.get("defaultModel") or DEFAULT_MODEL
+    timeout_seconds = _int_config(config.get("timeoutSeconds"), DEFAULT_TIMEOUT_SECONDS)
+    base_path = _resolve_base_path(config)
+    log_root = _resolve_log_root(config, base_path)
+    log_dir = _new_log_dir(log_root, str(app or requested_sample_id))
+    inspect_cmd = _inspect_command(config)
+    if not inspect_cmd:
+        return {
+            "error": (
+                "Inspect CLI command is empty. Set providers[0].config.inspectCommand "
+                "or PROMPTFOO_OSWORLD_INSPECT_COMMAND."
+            )
+        }
+
+    task = config.get("task") or DEFAULT_TASK
+    task_parameters = _task_parameters(config.get("taskParameters"))
+
+    eval_cmd = [
+        *inspect_cmd,
+        "eval",
+        task,
+        *task_parameters,
+        "--model",
+        str(model),
+        "--log-dir",
+        str(log_dir),
+        "--sample-id",
+        str(requested_sample_id),
+    ]
+
+    started = time.monotonic()
+    try:
+        eval_result = subprocess.run(
+            eval_cmd,
+            check=False,
+            capture_output=True,
+            text=True,
+            timeout=timeout_seconds,
+            cwd=str(base_path),
+        )
+    except FileNotFoundError:
+        return {"error": _missing_inspect_message(inspect_cmd[0])}
+    except subprocess.TimeoutExpired:
+        return {
+            "error": (
+                f"Inspect OSWorld run timed out after {timeout_seconds}s. "
+                "Increase providers[0].config.timeoutSeconds for real runs."
+            ),
+            "metadata": {
+                "inspect_log_path": str(log_dir),
+                "status": "error",
+                "duration_seconds": round(time.monotonic() - started, 3),
+            },
+        }
+
+    duration = time.monotonic() - started
+    if eval_result.returncode != 0:
+        return {
+            "error": _humanize_inspect_failure(eval_result),
+            "metadata": {
+                "inspect_log_path": str(log_dir),
+                "status": "error",
+                "duration_seconds": round(duration, 3),
+            },
+        }
+
+    eval_log = _find_eval_log(log_dir)
+    if eval_log is None:
+        return {
+            "error": f"Inspect completed but no .eval log was found in {log_dir}.",
+            "metadata": {"inspect_log_path": str(log_dir), "status": "error"},
+        }
+
+    dump_cmd = [*inspect_cmd, "log", "dump", str(eval_log)]
+    try:
+        dump_result = subprocess.run(
+            dump_cmd,
+            check=False,
+            capture_output=True,
+            text=True,
+            timeout=300,
+            cwd=str(base_path),
+        )
+    except FileNotFoundError:
+        return {"error": _missing_inspect_message(inspect_cmd[0])}
+    except subprocess.TimeoutExpired:
+        return {
+            "error": f"Timed out while dumping Inspect log {eval_log} to JSON.",
+            "metadata": {"inspect_log_path": str(eval_log), "status": "error"},
+        }
+
+    if dump_result.returncode != 0:
+        return {
+            "error": _humanize_log_dump_failure(dump_result, eval_log),
+            "metadata": {"inspect_log_path": str(eval_log), "status": "error"},
+        }
+
+    try:
+        log_json = json.loads(dump_result.stdout)
+    except json.JSONDecodeError as exc:
+        return {
+            "error": f"Could not parse Inspect log dump JSON for {eval_log}: {exc}",
+            "metadata": {"inspect_log_path": str(eval_log), "status": "error"},
+        }
+
+    parsed = _parse_inspect_log(log_json)
+    parsed_sample_id = parsed.get("sample_id")
+    sample_id = parsed_sample_id or "unknown sample"
+    if parsed_sample_id and sample_id != str(requested_sample_id):
+        return {
+            "error": (
+                f"Inspect returned sample {sample_id}, but Promptfoo requested "
+                f"{requested_sample_id}."
+            ),
+            "metadata": {
+                "inspect_log_path": str(eval_log),
+                "status": "error",
+                "sample_id": sample_id,
+                "requested_sample_id": str(requested_sample_id),
+                "duration_seconds": round(duration, 3),
+                "task": task,
+                "app": str(app or "unknown"),
+            },
+        }
+
+    score = parsed.get("score")
+    status = "pass" if isinstance(score, (int, float)) and score >= 1.0 else "fail"
+    metadata = {
+        "inspect_log_path": str(eval_log),
+        "score": score,
+        "status": status,
+        "sample_id": sample_id,
+        "model": str(model),
+        "num_messages": parsed.get("num_messages", 0),
+        "duration_seconds": round(duration, 3),
+        "task": task,
+        "app": str(app or "unknown"),
+        "requested_sample_id": str(requested_sample_id),
+    }
+    token_usage = parsed.get("token_usage")
+
+    inspect_error = _inspect_log_error(parsed)
+    if inspect_error:
+        metadata["status"] = "error"
+        metadata["inspect_status"] = parsed.get("inspect_status")
+        metadata["inspect_error"] = inspect_error
+        result: dict[str, Any] = {
+            "error": (
+                f"Inspect OSWorld run did not produce a score for sample {sample_id}: "
+                f"{inspect_error}"
+            ),
+            "metadata": metadata,
+        }
+        if token_usage:
+            result["tokenUsage"] = token_usage
+        return result
+
+    final_answer = (
+        parsed.get("final_answer") or "(no final assistant text found in Inspect log)"
+    )
+    output = (
+        f"Sample {sample_id} on app {app or 'unknown'}: "
+        f"score={score if score is not None else 'unknown'} "
+        f"status={status}\n\nFinal answer: {final_answer}"
+    )
+
+    result: dict[str, Any] = {"output": output, "metadata": metadata}
+    if token_usage:
+        result["tokenUsage"] = token_usage
+    return result
+
+
+def _inspect_command(config: dict) -> list[str]:
+    env_command = os.environ.get("PROMPTFOO_OSWORLD_INSPECT_COMMAND")
+    command = config.get("inspectCommand") or env_command or "inspect"
+    if isinstance(command, list):
+        return [str(part) for part in command]
+    return shlex.split(str(command))
+
+
+def _task_parameters(configured: Any) -> list[str]:
+    if not isinstance(configured, dict):
+        return []
+
+    parameters: list[str] = []
+    for key, value in configured.items():
+        if value is None:
+            continue
+        parameters.extend(["-T", f"{key}={_format_task_parameter_value(value)}"])
+    return parameters
+
+
+def _format_task_parameter_value(value: Any) -> str:
+    if isinstance(value, bool):
+        return str(value).lower()
+    return str(value)
+
+
+def _resolve_base_path(config: dict) -> Path:
+    configured = config.get("basePath")
+    if configured:
+        return Path(configured).expanduser().resolve()
+    return Path(__file__).resolve().parent
+
+
+def _resolve_log_root(config: dict, base_path: Path) -> Path:
+    configured = config.get("logRoot")
+    path = Path(configured).expanduser() if configured else base_path / "inspect_logs"
+    if not path.is_absolute():
+        path = base_path / path
+    return path.resolve()
+
+
+def _new_log_dir(log_root: Path, app: str) -> Path:
+    safe_app = "".join(ch if ch.isalnum() or ch in ("-", "_") else "-" for ch in app)
+    stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
+    path = log_root / f"{stamp}-{safe_app}-{os.getpid()}-{uuid.uuid4().hex[:8]}"
+    path.mkdir(parents=True, exist_ok=False)
+    return path
+
+
+def _find_eval_log(log_dir: Path) -> Path | None:
+    logs = sorted(
+        log_dir.rglob("*.eval"), key=lambda path: path.stat().st_mtime, reverse=True
+    )
+    return logs[0] if logs else None
+
+
+def _parse_inspect_log(data: dict[str, Any]) -> dict[str, Any]:
+    samples = data.get("samples") or []
+    sample = samples[0] if samples else {}
+    sample_score, score_error = _sample_score(sample) if sample else (None, None)
+    score = None if score_error else sample_score
+    return {
+        "score": score,
+        "score_error": score_error,
+        "sample_id": sample.get("id") or sample.get("uuid"),
+        "final_answer": _final_answer(sample),
+        "num_messages": len(sample.get("messages") or []),
+        "token_usage": _token_usage(data, sample),
+        "inspect_status": data.get("status"),
+        "sample_error": sample.get("error"),
+    }
+
+
+def _sample_score(sample: dict[str, Any]) -> tuple[float | None, str | None]:
+    scores = sample.get("scores") or {}
+    preferred_score = scores.get(OSWORLD_SCORER)
+    if preferred_score is not None:
+        numeric = _score_value(preferred_score)
+        if numeric is not None:
+            return numeric, None
+        return (
+            None,
+            f"Inspect recorded {OSWORLD_SCORER}, but its value was not numeric.",
+        )
+
+    if scores:
+        return (
+            None,
+            f"Inspect did not record {OSWORLD_SCORER} for the selected sample.",
+        )
+    return None, None
+
+
+def _score_value(score: Any) -> float | None:
+    if not isinstance(score, dict):
+        return None
+    return _coerce_score(score.get("value"))
+
+
+def _coerce_score(value: Any) -> float | None:
+    if isinstance(value, bool):
+        return 1.0 if value else 0.0
+    if isinstance(value, (int, float)):
+        return float(value)
+    if isinstance(value, str):
+        normalized = value.strip().upper()
+        if normalized == "C":
+            return 1.0
+        if normalized == "I":
+            return 0.0
+        try:
+            return float(value)
+        except ValueError:
+            return None
+    return None
+
+
+def _final_answer(sample: dict[str, Any]) -> str | None:
+    output = sample.get("output") or {}
+    completion = output.get("completion")
+    if isinstance(completion, str) and completion.strip():
+        return completion.strip()
+
+    for message in reversed(sample.get("messages") or []):
+        if message.get("role") == "assistant":
+            content = message.get("content")
+            if isinstance(content, str) and content.strip():
+                return content.strip()
+    return None
+
+
+def _token_usage(data: dict[str, Any], sample: dict[str, Any]) -> dict[str, int] | None:
+    usage_sources = [
+        sample.get("model_usage"),
+        (data.get("stats") or {}).get("model_usage"),
+    ]
+    totals = {"prompt": 0, "completion": 0, "total": 0}
+    found = False
+    for usage_by_model in usage_sources:
+        if not isinstance(usage_by_model, dict):
+            continue
+        for usage in usage_by_model.values():
+            if not isinstance(usage, dict):
+                continue
+            totals["prompt"] += int(
+                usage.get("input_tokens") or usage.get("prompt_tokens") or 0
+            )
+            totals["completion"] += int(
+                usage.get("output_tokens") or usage.get("completion_tokens") or 0
+            )
+            totals["total"] += int(usage.get("total_tokens") or 0)
+            found = True
+        if found:
+            break
+    if found and totals["total"] == 0:
+        totals["total"] = totals["prompt"] + totals["completion"]
+    return totals if found else None
+
+
+def _inspect_log_error(parsed: dict[str, Any]) -> str | None:
+    score_error = parsed.get("score_error")
+    if score_error:
+        return str(score_error)
+
+    sample_error = parsed.get("sample_error")
+    if isinstance(sample_error, dict):
+        message = sample_error.get("message") or sample_error.get("traceback")
+        if message:
+            return _summarize_inspect_error(str(message))
+    elif sample_error:
+        return _summarize_inspect_error(str(sample_error))
+
+    if parsed.get("inspect_status") == "error":
+        return "Inspect log status is error and no scorer result was recorded."
+
+    if parsed.get("score") is None:
+        sample_id = parsed.get("sample_id")
+        if sample_id:
+            return (
+                f"Inspect completed sample {sample_id}, but no OSWorld scorer result "
+                "was recorded."
+            )
+        return (
+            "Inspect completed, but selected zero samples or recorded no scorer result. "
+            "Check vars.sample_id."
+        )
+
+    return None
+
+
+def _summarize_inspect_error(message: str) -> str:
+    if "Failure executing command:" in message:
+        return (
+            "Inspect computer tool failed while executing a model-requested desktop "
+            "command. See the Inspect log for the full traceback and command output."
+        )
+
+    if "image_original.png" in message:
+        return (
+            "OSWorld scorer could not read an expected image artifact. See the Inspect "
+            "log for the full traceback."
+        )
+
+    if "'OSWorldDesktopEnv' object has no attribute 'vm_ip'" in message:
+        return (
+            "OSWorld scorer could not read VLC desktop state. See the Inspect log for "
+            "the full traceback."
+        )
+
+    return "Inspect reported an unscored sample error. See the Inspect log for details."
+
+
+def _int_config(value: Any, default: int) -> int:
+    try:
+        return int(value)
+    except (TypeError, ValueError):
+        return default
+
+
+def _missing_inspect_message(command: str) -> str:
+    if shutil.which(command) is None:
+        return (
+            f"Could not find Inspect CLI command '{command}'. Install prerequisites with "
+            "`pip install 'inspect-evals[osworld]' openai anthropic` or set "
+            "providers[0].config.inspectCommand."
+        )
+    return f"Could not execute Inspect CLI command '{command}'."
+
+
+def _humanize_inspect_failure(result: subprocess.CompletedProcess) -> str:
+    tail = _tail(result.stderr) or _tail(result.stdout) or "(no output)"
+    hint = ""
+    lowered = tail.lower()
+    if "specified dataset is empty" in lowered or "no samples" in lowered:
+        hint = " Check vars.sample_id. Inspect selected no OSWorld samples for this id."
+    elif "docker compose" in lowered or "compose" in lowered:
+        hint = " Check that Docker Compose V2 is installed and available as `docker compose`."
+    elif "docker" in lowered:
+        hint = " Check that Docker is installed, running, and usable by your user."
+    elif "requires optional dependencies" in lowered:
+        hint = " Install the selected Inspect model provider SDK, such as `openai` or `anthropic`."
+    elif "api_key" in lowered or "api key" in lowered or "authentication" in lowered:
+        hint = " Check that the selected model provider API key is exported."
+    elif "model" in lowered:
+        hint = " Check providers[0].config.defaultModel or vars.model."
+    return f"Inspect eval failed with exit code {result.returncode}.{hint}"
+
+
+def _humanize_log_dump_failure(
+    result: subprocess.CompletedProcess, eval_log: Path
+) -> str:
+    return f"Inspect log dump failed for {eval_log} with exit code {result.returncode}."
+
+
+def _tail(text: Any, limit: int = 4000) -> str:
+    if not text:
+        return ""
+    text = text.decode(errors="replace") if isinstance(text, bytes) else str(text)
+    return text[-limit:]