opensandbox-group--opensandbox/docs/examples/harbor-evaluation.md

title, description

title	description
Harbor Evaluation	Run a Harbor agent evaluation on OpenSandbox, provisioning one sandbox per trial.

Harbor Evaluation on OpenSandbox

Run a Harbor agent evaluation on OpenSandbox infrastructure. Harbor provisions one OpenSandbox container per trial, runs the agent inside it, executes the task's verifier, and collects the reward plus logs and artifacts.

Harbor gained native OpenSandbox support in harbor-framework/harbor#2054, which adds the opensandbox environment backend and the harbor[opensandbox] install extra.

This example ships a minimal, self-contained task (hello-opensandbox) plus a job config that selects the opensandbox environment. The task uses a prebuilt image and an offline verifier so it runs anywhere without extra dependencies.

Start OpenSandbox server [local]

Start a local OpenSandbox server (Docker runtime):

uv pip install opensandbox-server
opensandbox-server init-config ~/.sandbox.toml --example docker
opensandbox-server

Install Harbor

The OpenSandbox backend was merged into Harbor's main branch (#2054) but is not yet in a published release. Until a release ships with it, install Harbor from git:

uv pip install "harbor[opensandbox] @ git+https://github.com/harbor-framework/harbor"

Once a released Harbor version includes the backend, the plain extra will work:

uv pip install "harbor[opensandbox]"   # after the next Harbor release

Configure the connection

Harbor reads the server connection from environment variables (or from the domain / api_key kwargs in config.yaml):

export OPENSANDBOX_DOMAIN="localhost:8080"   # OpenSandbox server address
export OPENSANDBOX_API_KEY=""                # API key, if your server requires one

OPENSANDBOX_API_KEY may be left empty for a local / no-auth server.

Run the evaluation

# From examples/harbor-evaluation
harbor run -c config.yaml

Harbor creates an OpenSandbox sandbox from the task's prebuilt image (ubuntu:24.04), runs the oracle agent (which executes the reference solution), runs the verifier, records the reward, and tears the sandbox down.

Environment Variables

Variable	Default	Description
OPENSANDBOX_DOMAIN	localhost:8080	OpenSandbox server address
OPENSANDBOX_API_KEY	(optional)	API key if your server requires authentication

Outputs

Results are written under the jobs_dir from config.yaml (jobs/opensandbox/<job-id>/<trial-id>/), including agent/ logs, verifier/ output (reward.txt, ctrf.json), collected artifacts/, and results.json. A successful run produces reward = 1.0.

Use your own task

The included hello-opensandbox task is checked into this repository only to make the example self-contained and runnable without publishing or downloading anything first. For real evaluations, prefer tasks from an external Harbor task registry and override the example task from the command line:

harbor run -c config.yaml --task <org>/<task>@latest

During local task development, you can also point Harbor at a task directory:

harbor run -c config.yaml --path /path/to/task

Evaluate a real agent

Swap the oracle agent for a real agent + model:

harbor run -c config.yaml -a claude-code -m "anthropic/claude-sonnet-4-5"

OpenSandbox-specific notes

• Prebuilt images only — every task must set [environment].docker_image in task.toml; the OpenSandbox backend does not build Dockerfiles. Use image_auth kwargs for private registries.
• Working directory — set [environment].workdir (this task uses /app) so the agent's relative paths resolve where the verifier expects them.
• Resources — cpus and memory_mb map to hard sandbox limits; storage_mb is ignored; GPUs are supported via gpus / gpu_types.
• Artifacts — files written under /logs/artifacts are downloaded into the trial's artifacts/ directory.
• Multi-container — docker-compose.yaml tasks are not supported; use single-container tasks.

References

• Source code on GitHub
• Harbor