chore: import upstream snapshot with attribution

2026-07-13 12:23:35 +08:00
commit c8c954c85d
127 changed files with 22519 additions and 0 deletions
@@ -0,0 +1,54 @@
+# Deplying Agent S3 in OSWorld
+
+# Step 1: Set up Agent S3
+
+Follow the [README.md](https://github.com/simular-ai/Agent-S/blob/main/README.md) to set up Agent S3.
+
+# Step 2: Copying Over Run Files
+
+If you haven't already, please follow the [OSWorld environment setup](https://github.com/xlang-ai/OSWorld/blob/main/README.md). We've provided the relevant OSWorld run files for evaluation in this `osworld_setup` folder. Please copy this over to your OSWorld folder. `run_local.py` is for if you want to run locally on VMWare and `run.py` and `lib_run_single.py` are for if you want to run on AWS. All run commands in order are provided in the `run.sh`. Copy over the files in `osworld_setup/s3/bbon` as well. 
+
+# Step 3: Switch the AMI 
+
+Switch image AMI for the AWS provider in `desktop_env/providers/aws/manager.py` is set to `"ami-0b505e9d0d99ba88c"`.
+
+# Step 4: Generating Facts
+
+After completing your OSWorld runs and having result directories, run `generate_facts.py` to generate fact captions for screenshot pairs:
+
+```bash
+python osworld_setup/s3/bbon/generate_facts.py \
+  --results-dirs \
+    results1/pyautogui/screenshot/gpt-5-2025-08-07 \
+    results2/pyautogui/screenshot/gpt-5-2025-08-07 \
+  --model "gpt-5-2025-08-07" \
+  --engine-type "openai" \
+  --temperature 1.0
+```
+
+This will populate your result directories with `fact_captions.jsonl` files containing behavioral descriptions of screenshot differences.
+
+# Step 5: Run the Judge
+
+Finally, run `run_judge.py` to evaluate the trajectories using the generated fact captions:
+
+```bash
+python osworld_setup/s3/bbon/run_judge.py \
+  --results-dirs \
+    results1/pyautogui/screenshot/gpt-5-2025-08-07 \
+    results2/pyautogui/screenshot/gpt-5-2025-08-07 \
+  --output-dir "judge_results" \
+  --examples-path "evaluation_examples/examples" \
+  --model "gpt-5-2025-08-07" \
+  --engine-type "openai" \
+  --temperature 1.0
+```
+
+This will:
+- Compare trajectories across different result directories
+- Use the facts to judge which trajectory performs better
+- Generate evaluation results
+- Save results to the specified output directory
+
+The judge will create files like `BoN2.json`, `BoN3.json`, etc., showing the performance comparison as you add more trajectories.
+