wehub-resource-sync
104 lines
4.7 KiB
Markdown
104 lines
4.7 KiB
Markdown
# Output files
|
|
|
|
The [`trajectories/`](https://github.com/SWE-agent/SWE-agent/tree/main/trajectories) folder is the default location that experiment results (for example invocations of [`sweagent run`](cl_tutorial.md)) will be written to.
|
|
|
|
## Trajectories
|
|
|
|
The main output file is `<instance_id>.traj`, which is a `.json` formatted file containing the (thought, action, observation) turns generated by SWE-agent towards solving `<instance_id>`.
|
|
|
|
Under the `trajectory` key, you can see information for every step of the agent. It looks something like this:
|
|
|
|
```json
|
|
{
|
|
# This is the output of the LM
|
|
"response": "We are indeed seeing the same output as the issue. The issue suggests that we should look at line 1474 of the `fields.py`...",
|
|
# We then parse it into thoughts and actions
|
|
"thought": "We are indeed seeing the same output as the issue. The issue suggests that we should look at line 1474 of the `fields.py`..."
|
|
"action": "ls -F\n",
|
|
# And execute the action, resulting in the output
|
|
"observation": "AUTHORS.rst\nCHANGELOG.rst\nCODE_OF_CONDUCT.md...",
|
|
# In addition, after the action was executed, state can be extracted from the environment
|
|
"state": "{\"open_file\": \"/marshmallow-code__marshmallow/reproduce.py\", \"working_dir\": \"/marshmallow-code__marshmallow\"}\n",
|
|
# For debugging, we also keep all messages that were shown to the LM
|
|
"query": [{"role": "system", "content": "You are a helpful assistant ..."}, ...]
|
|
},
|
|
```
|
|
|
|
!!! warning "Query and message field"
|
|
Prior to SWE-agent 1.1.0, we had a `message` field which corresponded (approximately) to the input
|
|
for the LM for the _next_ step. This was replaced with `query`, which shows the exact input
|
|
at the current step.
|
|
|
|
Here's a full example:
|
|
|
|
<details>
|
|
<summary>Example trajectory</summary>
|
|
|
|
Note: This trajectory is from SWE-agent v0.7.0. It might look different in newer versions.
|
|
|
|
```json
|
|
--8<-- "trajectories/demonstrations/replay__marshmallow-code__marshmallow-1867__default__t-0.20__p-0.95__c-2.00__install-1___install_from_source/marshmallow-code__marshmallow-1867.traj"
|
|
```
|
|
</details>
|
|
|
|
!!! tip "Viewing trajectories"
|
|
We provide a [trajectory viewer](inspector.md) for an easy viewing of trajectories.
|
|
If you want to view the json, we recommend [jsoneditoronline](https://jsoneditoronline.org/),
|
|
because of the way it handles long lines.
|
|
|
|
Trajectories can also be turned into custom demonstrations for SWE-agent ([more information](../config/demonstrations.md)).
|
|
|
|
## Other output files
|
|
|
|
Together with the trajectory, SWE-agent will also generate the following files per instance:
|
|
|
|
* `config.yaml`: A yaml file containing the configuration used for the experiment. You can directly repeat the run of this
|
|
exact instance by running `sweagent run --config config.yaml`.
|
|
* `*.log`: Log files at different levels
|
|
|
|
At a high level, the experiments folder is organized in the following manner:
|
|
|
|
```
|
|
trajectories:
|
|
user 1:
|
|
experiment 1:
|
|
run_batch.config.yaml
|
|
preds.json
|
|
run_batch.debug.log
|
|
run_batch.info.log
|
|
run_batch.warning.log
|
|
run_batch_exit_statuses.yaml
|
|
instance_1:
|
|
instance_1.traj
|
|
instance_1.config.yaml
|
|
instance_1.debug.log
|
|
instance_1.info.log
|
|
instance_1.warning.log
|
|
instance 2:
|
|
...
|
|
experiment 2:
|
|
...
|
|
user 2:
|
|
...
|
|
```
|
|
|
|
Where every experiment follows the pattern `trajectories/<user name>/<experiment name>`. The `<user name>` is automatically inferred from your system, and the `experiment name` is inferred from the model names and other settings to the run command.
|
|
|
|
|
|
Here are a few more files that are generated with `sweagent run-batch`:
|
|
|
|
* `run_batch.config.yaml`: A yaml file containing the configuration used for the experiment. You can directly repeat the run of this
|
|
exact experiment by running `sweagent run-batch --config run_batch.config.yaml`.
|
|
* `preds.json`: All predictions (i.e., patches produced by the agent) from all instances of the experiment.
|
|
If this file goes out of date, you can regenerate it by running `sweagent merge-preds`.
|
|
* `run_batch.*.log`: Log files at different levels.
|
|
* `run_batch_exit_statuses.yaml`: A yaml file containing the exit statuses of all instances in the experiment (for this run of `sweagent run-batch`).
|
|
|
|
|
|
!!! tip
|
|
* Evaluation is not completed by `sweagent run-batch`, it is a separate step (see [batch mode](batch_mode.md))
|
|
* `preds.json` can be referenced directly into SWE-bench to run evaluation
|
|
* Trajectories can be turned into custom demonstrations for SWE-agent ([more information](../config/demonstrations.md)).
|
|
|
|
{% include-markdown "../_footer.md" %}
|