wehub-resource-sync
2.2 KiB
Deplying Agent S3 in OSWorld
Step 1: Set up Agent S3
Follow the README.md to set up Agent S3.
Step 2: Copying Over Run Files
If you haven't already, please follow the OSWorld environment setup. We've provided the relevant OSWorld run files for evaluation in this osworld_setup folder. Please copy this over to your OSWorld folder. run_local.py is for if you want to run locally on VMWare and run.py and lib_run_single.py are for if you want to run on AWS. All run commands in order are provided in the run.sh. Copy over the files in osworld_setup/s3/bbon as well.
Step 3: Switch the AMI
Switch image AMI for the AWS provider in desktop_env/providers/aws/manager.py is set to "ami-0b505e9d0d99ba88c".
Step 4: Generating Facts
After completing your OSWorld runs and having result directories, run generate_facts.py to generate fact captions for screenshot pairs:
python osworld_setup/s3/bbon/generate_facts.py \
--results-dirs \
results1/pyautogui/screenshot/gpt-5-2025-08-07 \
results2/pyautogui/screenshot/gpt-5-2025-08-07 \
--model "gpt-5-2025-08-07" \
--engine-type "openai" \
--temperature 1.0
This will populate your result directories with fact_captions.jsonl files containing behavioral descriptions of screenshot differences.
Step 5: Run the Judge
Finally, run run_judge.py to evaluate the trajectories using the generated fact captions:
python osworld_setup/s3/bbon/run_judge.py \
--results-dirs \
results1/pyautogui/screenshot/gpt-5-2025-08-07 \
results2/pyautogui/screenshot/gpt-5-2025-08-07 \
--output-dir "judge_results" \
--examples-path "evaluation_examples/examples" \
--model "gpt-5-2025-08-07" \
--engine-type "openai" \
--temperature 1.0
This will:
- Compare trajectories across different result directories
- Use the facts to judge which trajectory performs better
- Generate evaluation results
- Save results to the specified output directory
The judge will create files like BoN2.json, BoN3.json, etc., showing the performance comparison as you add more trajectories.