This folder contains a lightweight vision eval harness plus example runners for image generation and image editing. The code mirrors the structure of examples/evals/realtime_evals/ so you can adapt it quickly.

Directory layout

vision_harness/: minimal shared library (types, runners, graders, evaluate loop)
generation_harness/: text-to-image evals (UI mockups + marketing flyer)
editing_harness/: image-edit evals (virtual try-on + logo edit)
shared/: reporting and optional rendering helpers

Quickstart

Python 3.9+ required.

pip install -r requirements.txt
export OPENAI_API_KEY="your_api_key"

Run a harness:

Generation: python generation_harness/run_imagegen_evals.py
Editing: python editing_harness/run_imagegen_evals.py

What the harness does

Builds a small set of TestCase objects (prompt + criteria).
Runs the image model for each case.
Grades each output with an LLM judge using a strict JSON schema.
Writes results and artifacts to results/<run_id>/.

The harness is intentionally small so you can copy/paste parts into your own production eval setup.

Example cases

Generation cases:

ui_checkout_mockup: mobile checkout screen with strict text + layout rules
coffee_flyer_generation: marketing flyer with exact copy constraints

Editing cases:

vto_jacket_tryon: virtual try-on with reference person + garment
logo_year_edit: precision logo text edit

Required assets (editing harness)

The editing harness expects these files in images/:

images/base_woman.png
images/jacket.png
images/logo_generation_1.png

Results layout

Each harness writes into its own results/ folder:

results/<run_id>/results.json: per-example outputs and grades
results/<run_id>/results.csv: tabular results
results/<run_id>/summary.json: aggregated metrics
results/<run_id>/artifacts/*.png: generated or edited images

Tip: keep results/ out of git (delete after runs or add to .gitignore).

Common CLI flags

Both harnesses accept a shared core set of flags:

--results-dir, --run-name
--model (image model under test), --judge-model (LLM judge)
--num-images (images per case)
--cases (comma-separated list of case ids)

Generation harness adds --image-size and --run-ocr (text extraction check).

Suggested workflows

Quick smoke test:
- python generation_harness/run_imagegen_evals.py --cases ui_checkout_mockup
- python editing_harness/run_imagegen_evals.py --cases vto_jacket_tryon
Compare model variants: run once per model name and compare summary.json.
Use --run-ocr on the flyer case to validate exact text rendering.

Adapting for your use case

Replace prompts/criteria in the harness scripts.
Update JSON schemas to match your rubric (and parse them into Score values).
Add new cases by creating new TestCase objects and graders.
Swap the output store to change where artifacts are written.

Notes

This harness uses the OpenAI Images API and a judge model. Costs scale with the number of images and judge calls.
Judge outputs are best treated as signals, not ground truth. Keep a small human review loop for high-stakes decisions.