wehub-resource-sync
55 lines
2.2 KiB
Markdown
55 lines
2.2 KiB
Markdown
# Deplying Agent S3 in OSWorld
|
|
|
|
# Step 1: Set up Agent S3
|
|
|
|
Follow the [README.md](https://github.com/simular-ai/Agent-S/blob/main/README.md) to set up Agent S3.
|
|
|
|
# Step 2: Copying Over Run Files
|
|
|
|
If you haven't already, please follow the [OSWorld environment setup](https://github.com/xlang-ai/OSWorld/blob/main/README.md). We've provided the relevant OSWorld run files for evaluation in this `osworld_setup` folder. Please copy this over to your OSWorld folder. `run_local.py` is for if you want to run locally on VMWare and `run.py` and `lib_run_single.py` are for if you want to run on AWS. All run commands in order are provided in the `run.sh`. Copy over the files in `osworld_setup/s3/bbon` as well.
|
|
|
|
# Step 3: Switch the AMI
|
|
|
|
Switch image AMI for the AWS provider in `desktop_env/providers/aws/manager.py` is set to `"ami-0b505e9d0d99ba88c"`.
|
|
|
|
# Step 4: Generating Facts
|
|
|
|
After completing your OSWorld runs and having result directories, run `generate_facts.py` to generate fact captions for screenshot pairs:
|
|
|
|
```bash
|
|
python osworld_setup/s3/bbon/generate_facts.py \
|
|
--results-dirs \
|
|
results1/pyautogui/screenshot/gpt-5-2025-08-07 \
|
|
results2/pyautogui/screenshot/gpt-5-2025-08-07 \
|
|
--model "gpt-5-2025-08-07" \
|
|
--engine-type "openai" \
|
|
--temperature 1.0
|
|
```
|
|
|
|
This will populate your result directories with `fact_captions.jsonl` files containing behavioral descriptions of screenshot differences.
|
|
|
|
# Step 5: Run the Judge
|
|
|
|
Finally, run `run_judge.py` to evaluate the trajectories using the generated fact captions:
|
|
|
|
```bash
|
|
python osworld_setup/s3/bbon/run_judge.py \
|
|
--results-dirs \
|
|
results1/pyautogui/screenshot/gpt-5-2025-08-07 \
|
|
results2/pyautogui/screenshot/gpt-5-2025-08-07 \
|
|
--output-dir "judge_results" \
|
|
--examples-path "evaluation_examples/examples" \
|
|
--model "gpt-5-2025-08-07" \
|
|
--engine-type "openai" \
|
|
--temperature 1.0
|
|
```
|
|
|
|
This will:
|
|
- Compare trajectories across different result directories
|
|
- Use the facts to judge which trajectory performs better
|
|
- Generate evaluation results
|
|
- Save results to the specified output directory
|
|
|
|
The judge will create files like `BoN2.json`, `BoN3.json`, etc., showing the performance comparison as you add more trajectories.
|
|
|