chore: import upstream snapshot with attribution
Deploy Documentation / deploy (push) Has been cancelled
CPU Test / Test (Utilities, legacy, Python 3.10) (push) Has been cancelled
CPU Test / Test (LLM proxy, stable, Python 3.11) (push) Has been cancelled
CPU Test / Test (Others, stable, Python 3.11) (push) Has been cancelled
CPU Test / Test (Store, stable, Python 3.11) (push) Has been cancelled
CPU Test / Test (Utilities, stable, Python 3.11) (push) Has been cancelled
CPU Test / Test (Weave, stable, Python 3.11) (push) Has been cancelled
CPU Test / Test (AgentOps, stable, Python 3.12) (push) Has been cancelled
CPU Test / Test (LLM proxy, stable, Python 3.12) (push) Has been cancelled
CPU Test / Test (Others, stable, Python 3.12) (push) Has been cancelled
CPU Test / Test (Weave, latest, Python 3.13) (push) Has been cancelled
Dashboard / Chromatic (push) Has been cancelled
CPU Test / Lint - fast (push) Has been cancelled
CPU Test / Lint - next (push) Has been cancelled
CPU Test / Lint - slow (push) Has been cancelled
CPU Test / Lint - JavaScript (push) Has been cancelled
CPU Test / Build documentation (push) Has been cancelled
CPU Test / Test (AgentOps, legacy, Python 3.10) (push) Has been cancelled
CPU Test / Test (LLM proxy, legacy, Python 3.10) (push) Has been cancelled
CPU Test / Test (Others, legacy, Python 3.10) (push) Has been cancelled
CPU Test / Test (Store, legacy, Python 3.10) (push) Has been cancelled
CPU Test / Test (Weave, legacy, Python 3.10) (push) Has been cancelled
CPU Test / Test (AgentOps, stable, Python 3.11) (push) Has been cancelled
CPU Test / Test (Store, stable, Python 3.12) (push) Has been cancelled
CPU Test / Test (Utilities, stable, Python 3.12) (push) Has been cancelled
CPU Test / Test (Weave, stable, Python 3.12) (push) Has been cancelled
CPU Test / Test (AgentOps, latest, Python 3.13) (push) Has been cancelled
CPU Test / Test (LLM proxy, latest, Python 3.13) (push) Has been cancelled
CPU Test / Test (Others, latest, Python 3.13) (push) Has been cancelled
CPU Test / Test (Store, latest, Python 3.13) (push) Has been cancelled
CPU Test / Test (Utilities, latest, Python 3.13) (push) Has been cancelled
CPU Test / Test (JavaScript) (push) Has been cancelled
Deploy Documentation / deploy (push) Has been cancelled
CPU Test / Test (Utilities, legacy, Python 3.10) (push) Has been cancelled
CPU Test / Test (LLM proxy, stable, Python 3.11) (push) Has been cancelled
CPU Test / Test (Others, stable, Python 3.11) (push) Has been cancelled
CPU Test / Test (Store, stable, Python 3.11) (push) Has been cancelled
CPU Test / Test (Utilities, stable, Python 3.11) (push) Has been cancelled
CPU Test / Test (Weave, stable, Python 3.11) (push) Has been cancelled
CPU Test / Test (AgentOps, stable, Python 3.12) (push) Has been cancelled
CPU Test / Test (LLM proxy, stable, Python 3.12) (push) Has been cancelled
CPU Test / Test (Others, stable, Python 3.12) (push) Has been cancelled
CPU Test / Test (Weave, latest, Python 3.13) (push) Has been cancelled
Dashboard / Chromatic (push) Has been cancelled
CPU Test / Lint - fast (push) Has been cancelled
CPU Test / Lint - next (push) Has been cancelled
CPU Test / Lint - slow (push) Has been cancelled
CPU Test / Lint - JavaScript (push) Has been cancelled
CPU Test / Build documentation (push) Has been cancelled
CPU Test / Test (AgentOps, legacy, Python 3.10) (push) Has been cancelled
CPU Test / Test (LLM proxy, legacy, Python 3.10) (push) Has been cancelled
CPU Test / Test (Others, legacy, Python 3.10) (push) Has been cancelled
CPU Test / Test (Store, legacy, Python 3.10) (push) Has been cancelled
CPU Test / Test (Weave, legacy, Python 3.10) (push) Has been cancelled
CPU Test / Test (AgentOps, stable, Python 3.11) (push) Has been cancelled
CPU Test / Test (Store, stable, Python 3.12) (push) Has been cancelled
CPU Test / Test (Utilities, stable, Python 3.12) (push) Has been cancelled
CPU Test / Test (Weave, stable, Python 3.12) (push) Has been cancelled
CPU Test / Test (AgentOps, latest, Python 3.13) (push) Has been cancelled
CPU Test / Test (LLM proxy, latest, Python 3.13) (push) Has been cancelled
CPU Test / Test (Others, latest, Python 3.13) (push) Has been cancelled
CPU Test / Test (Store, latest, Python 3.13) (push) Has been cancelled
CPU Test / Test (Utilities, latest, Python 3.13) (push) Has been cancelled
CPU Test / Test (JavaScript) (push) Has been cancelled
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,55 @@
|
||||
# APO
|
||||
|
||||
!!! tip "Shortcut"
|
||||
|
||||
You can use the shortcut `agl.APO(...)` to create an APO instance.
|
||||
|
||||
```python
|
||||
import agentlightning as agl
|
||||
|
||||
agl.APO(...)
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pip install agentlightning[apo]
|
||||
```
|
||||
|
||||
## Scope of Current Implementation
|
||||
|
||||
APO is currently scoped to optimize a single prompt template. Optimizing multiple prompt templates is not supported yet.
|
||||
|
||||
There is however no restriction on the number of variable placeholders in the prompt template (can range from zero to many). It's possible that invalid prompts are created during the optimization process. It is up to the agent developer to ensure that the prompt template is valid for the agent's task.
|
||||
|
||||
## Initial Prompt
|
||||
|
||||
APO expects the initial prompt to be provided in the `initial_resources` dictionary. This can be done in two approaches:
|
||||
|
||||
1. Pass to the [Trainer][agentlightning.Trainer] constructor:
|
||||
|
||||
```python
|
||||
trainer = agl.Trainer(
|
||||
algorithm=agl.APO(...),
|
||||
initial_resources={"main_prompt": agl.PromptTemplate(template="You are a helpful assistant.", engine="f-string")},
|
||||
)
|
||||
```
|
||||
|
||||
2. Pass to the `[APO][agentlightning.algorithm.apo.APO].set_initial_resources()` method:
|
||||
|
||||
```python
|
||||
algo = agl.APO(...)
|
||||
algo.set_initial_resources(
|
||||
{"this_is_also_valid_key": agl.PromptTemplate(template="You are a helpful assistant.", engine="f-string")}
|
||||
)
|
||||
```
|
||||
|
||||
The resource key can be arbitrary, which is used to identify the prompt template in [class-based implementations](../tutorials/write-agents.md) when you have multiple resources. When the key changes, the agent developer needs to update the key in the `rollout` method.
|
||||
|
||||
## Tutorials Using APO
|
||||
|
||||
- [Train the First Agent with APO](../how-to/train-first-agent.md) - A step-by-step guide to training your first agent using APO.
|
||||
|
||||
## References
|
||||
|
||||
::: agentlightning.algorithm.apo
|
||||
@@ -0,0 +1,10 @@
|
||||
# Algorithm Zoo
|
||||
|
||||
AgentLightning includes several popular and frequently requested algorithms in its built-in library, allowing agent developers to use them directly. These algorithms are designed to be compatible with most agent scenarios.
|
||||
|
||||
For customizing algorithms, see [Algorithm-side References](../reference/algorithm.md).
|
||||
|
||||
| Algorithm | Optimizing Resources | Description |
|
||||
| --------- | ------------------- | ----------- |
|
||||
| [APO](./apo.md) | `{<initial_prompt_key>: [PromptTemplate][agentlightning.PromptTemplate]}` | Automatic Prompt Optimization (APO) algorithm using textual gradients and beam search. |
|
||||
| [VERL](./verl.md) | `{"main_llm": [LLM][agentlightning.LLM]}` | Reinforcement Learning with [VERL framework](https://github.com/volcengine/verl). |
|
||||
@@ -0,0 +1,58 @@
|
||||
# VERL
|
||||
|
||||
!!! tip "Shortcut"
|
||||
|
||||
You can use the shortcut `agl.VERL(...)` to create a VERL instance.
|
||||
|
||||
```python
|
||||
import agentlightning as agl
|
||||
|
||||
agl.VERL(...)
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pip install agentlightning[verl]
|
||||
```
|
||||
|
||||
!!! warning
|
||||
|
||||
To avoid various compatibility issues, follow the steps in the [installation guide](../tutorials/installation.md) to set up VERL and its dependencies. Installing VERL directly with `pip install agentlightning[verl]` can cause issues unless you already have a compatible version of PyTorch installed.
|
||||
|
||||
!!! note "Notes for Readers"
|
||||
|
||||
[VERL][agentlightning.algorithm.verl.VERL] in this article refers to a wrapper, provided by Agent-lightning, of the [VERL framework](https://github.com/volcengine/verl). It's a subclass of [agentlightning.Algorithm][]. To differentiate it from the VERL framework, all references to the VERL framework shall use the term "VERL framework", and all references to the Agent-lightning wrapper shall be highlighted with a link.
|
||||
|
||||
## Resources
|
||||
|
||||
[VERL][agentlightning.algorithm.verl.VERL] expects no initial resources. The first LLM endpoint is directly deployed from the VERL configuration (`.actor_rollout_ref.model.path`). The resource key is always `main_llm`.
|
||||
|
||||
[VERL][agentlightning.algorithm.verl.VERL] currently does not support optimizing multiple [LLM][agentlightning.LLM]s together.
|
||||
|
||||
!!! note
|
||||
|
||||
The resource type created by [VERL][agentlightning.algorithm.verl.VERL] is actually a [ProxyLLM][agentlightning.ProxyLLM], a subclass of the [LLM][agentlightning.LLM] type. This object contains a **URL template** provided by [VERL][agentlightning.algorithm.verl.VERL], with placeholders for rollout and attempt IDs. When a rollout begins on the agent side, the framework uses the current `rollout_id` and `attempt_id` to format this template, generating a final, unique endpoint URL. This URL points to [VERL][agentlightning.algorithm.verl.VERL]'s internal proxy, allowing it to intercept and log all traffic for that specific attempt, for tracing and load balancing purposes. For agents created with the `@rollout` decorator, this resolution of the template is handled automatically ("auto-stripped"). Class-based agents will need to manually resolve the [`ProxyLLM`][agentlightning.ProxyLLM] using the rollout context.
|
||||
|
||||
```python
|
||||
proxy_llm = resources["main_llm"]
|
||||
proxy_llm.get_base_url(rollout.rollout_id, rollout.attempt.attempt_id)
|
||||
```
|
||||
|
||||
## Customization
|
||||
|
||||
Internally, [VERL][agentlightning.algorithm.verl.VERL] decomposes each agent execution into prompt–response pairs via the [Adapter][agentlightning.Adapter] and associates them with their corresponding reward signals as [Triplet][agentlightning.Triplet] objects. The final scalar reward, derived from the last triplet in the trajectory, is propagated to all preceding triplets following the [identical assignment strategy](https://arxiv.org/abs/2508.03680). This ensures that each triplet receives an identical reward signal and can be independently optimized as a valid RLHF trajectory within the VERL framework.
|
||||
|
||||
At present, [VERL][agentlightning.algorithm.verl.VERL] does not expose fine-grained control over its reward propagation or credit assignment mechanisms. Users requiring customized reward shaping or trajectory decomposition are advised to clone and modify the [VERL][agentlightning.algorithm.verl.VERL] source implementation directly.
|
||||
|
||||
## Tutorials Using VERL
|
||||
|
||||
- [Train SQL Agent with RL](../how-to/train-sql-agent.md) - A practical example of training a SQL agent using VERL.
|
||||
|
||||
## References - Entrypoint
|
||||
|
||||
::: agentlightning.algorithm.verl
|
||||
|
||||
## References - Implementation
|
||||
|
||||
::: agentlightning.verl
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 536 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 275 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 247 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 102 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1003 B |
Binary file not shown.
|
After Width: | Height: | Size: 1.8 KiB |
@@ -0,0 +1,4 @@
|
||||
<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
|
||||
<path d="M8.06935 0.740967C8.46471 0.740967 8.78513 1.06143 8.78513 1.45675C8.78513 1.73028 8.6317 1.96783 8.40619 2.08833V2.6357C8.60265 2.66378 9.18471 2.67955 9.92197 3.01465C10.8483 3.4357 11.121 3.77803 11.4378 4.2357C11.6095 4.48376 11.7205 4.74914 11.7909 5.0357H11.9009C12.273 5.0357 12.5746 5.33732 12.5746 5.70938V6.38307C12.5746 6.75514 12.273 7.05675 11.9009 7.05675H11.8192C11.7591 7.40026 11.6762 7.69328 11.6062 7.85675C11.413 8.30755 11.129 8.48833 10.9746 8.53044C11.0869 8.57254 11.4395 8.67604 11.6483 8.82517C11.943 9.0357 12.2378 9.39174 12.2378 9.75149C12.2378 10.0462 12.1957 10.4673 11.9851 10.6778C11.7074 10.9556 11.2272 11.4778 10.9746 11.6883L6.34303 15.8568L7.35356 13.4989L10.3851 9.54096H8.02724L8.86934 6.59359L8.19658 7.34393L8.19567 7.35149L5.2483 10.762H7.6904L7.10093 12.7831L5.62724 11.8989L4.91146 11.4357C4.53251 11.1831 4.32198 11.0989 4.06935 10.6778C3.91617 10.4225 3.90093 10.0462 3.90093 9.75149C3.90093 9.39174 4.19567 9.0357 4.4904 8.82517C4.69915 8.67604 4.78514 8.61465 4.99567 8.53044C4.82724 8.44623 4.7257 8.30755 4.53251 7.85675C4.46245 7.69328 4.37956 7.40026 4.31951 7.05675H4.23777C3.8657 7.05675 3.56409 6.75514 3.56409 6.38307V5.70938C3.56409 5.33732 3.8657 5.0357 4.23777 5.0357H4.34788C4.41815 4.74914 4.5292 4.48376 4.70093 4.2357C5.01778 3.77803 5.2904 3.4357 6.21672 3.01465C6.95396 2.67955 7.53602 2.66378 7.73251 2.6357V2.08833C7.50704 1.96783 7.35356 1.73028 7.35356 1.45675C7.35356 1.06143 7.67403 0.740967 8.06935 0.740967ZM6.80619 5.0357C6.50389 5.0357 6.25882 5.28077 6.25882 5.58307C6.25882 5.88538 6.50389 6.13044 6.80619 6.13044C7.1085 6.13044 7.35356 5.88538 7.35356 5.58307C7.35356 5.28077 7.1085 5.0357 6.80619 5.0357ZM9.3325 5.0357C9.03018 5.0357 8.78513 5.28077 8.78513 5.58307C8.78513 5.88538 9.03018 6.13044 9.3325 6.13044C9.63481 6.13044 9.87987 5.88538 9.87987 5.58307C9.87987 5.28077 9.63481 5.0357 9.3325 5.0357Z" fill="black"/>
|
||||
<path d="M12.2279 9.63738C12.2342 9.67527 12.2378 9.71342 12.2378 9.75165C12.2378 10.0464 12.1957 10.4674 11.9851 10.678C11.7074 10.9558 11.2272 11.478 10.9746 11.6885L6.34305 15.8569L7.35357 13.499L7.9831 12.677C8.20912 12.6273 8.41543 12.5774 8.57462 12.5306C9.29041 12.3201 10.1325 11.6885 10.7641 11.099C11.2418 10.6531 11.9076 9.97136 12.2279 9.63738ZM9.62725 3.77271C10.3248 3.77271 10.8904 4.33825 10.8904 5.03586V6.80428C10.8904 7.50191 10.3248 8.06744 9.62725 8.06744H8.4483L8.86935 6.59376L8.19659 7.34409L8.19568 7.35165L7.57479 8.06744H6.59568C5.89805 8.06744 5.33252 7.50191 5.33252 6.80428V5.03586C5.33252 4.33825 5.89805 3.77271 6.59568 3.77271H9.62725ZM6.8062 5.03586C6.5039 5.03586 6.25884 5.28093 6.25884 5.58323C6.25884 5.88554 6.5039 6.1306 6.8062 6.1306C7.10851 6.1306 7.35357 5.88554 7.35357 5.58323C7.35357 5.28093 7.10851 5.03586 6.8062 5.03586ZM9.33251 5.03586C9.0302 5.03586 8.78514 5.28093 8.78514 5.58323C8.78514 5.88554 9.0302 6.1306 9.33251 6.1306C9.63483 6.1306 9.87988 5.88554 9.87988 5.58323C9.87988 5.28093 9.63483 5.03586 9.33251 5.03586Z" fill="black"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.0 KiB |
@@ -0,0 +1,4 @@
|
||||
<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
|
||||
<path d="M8.06935 0.740967C8.46471 0.740967 8.78513 1.06143 8.78513 1.45675C8.78513 1.73028 8.6317 1.96783 8.40619 2.08833V2.6357C8.60265 2.66378 9.18471 2.67955 9.92197 3.01465C10.8483 3.4357 11.121 3.77803 11.4378 4.2357C11.6095 4.48376 11.7205 4.74914 11.7909 5.0357H11.9009C12.273 5.0357 12.5746 5.33732 12.5746 5.70938V6.38307C12.5746 6.75514 12.273 7.05675 11.9009 7.05675H11.8192C11.7591 7.40026 11.6762 7.69328 11.6062 7.85675C11.413 8.30755 11.129 8.48833 10.9746 8.53044C11.0869 8.57254 11.4395 8.67604 11.6483 8.82517C11.943 9.0357 12.2378 9.39174 12.2378 9.75149C12.2378 10.0462 12.1957 10.4673 11.9851 10.6778C11.7074 10.9556 11.2272 11.4778 10.9746 11.6883L6.34303 15.8568L7.35356 13.4989L10.3851 9.54096H8.02724L8.86934 6.59359L8.19658 7.34393L8.19567 7.35149L5.2483 10.762H7.6904L7.10093 12.7831L5.62724 11.8989L4.91146 11.4357C4.53251 11.1831 4.32198 11.0989 4.06935 10.6778C3.91617 10.4225 3.90093 10.0462 3.90093 9.75149C3.90093 9.39174 4.19567 9.0357 4.4904 8.82517C4.69915 8.67604 4.78514 8.61465 4.99567 8.53044C4.82724 8.44623 4.7257 8.30755 4.53251 7.85675C4.46245 7.69328 4.37956 7.40026 4.31951 7.05675H4.23777C3.8657 7.05675 3.56409 6.75514 3.56409 6.38307V5.70938C3.56409 5.33732 3.8657 5.0357 4.23777 5.0357H4.34788C4.41815 4.74914 4.5292 4.48376 4.70093 4.2357C5.01778 3.77803 5.2904 3.4357 6.21672 3.01465C6.95396 2.67955 7.53602 2.66378 7.73251 2.6357V2.08833C7.50704 1.96783 7.35356 1.73028 7.35356 1.45675C7.35356 1.06143 7.67403 0.740967 8.06935 0.740967ZM6.80619 5.0357C6.50389 5.0357 6.25882 5.28077 6.25882 5.58307C6.25882 5.88538 6.50389 6.13044 6.80619 6.13044C7.1085 6.13044 7.35356 5.88538 7.35356 5.58307C7.35356 5.28077 7.1085 5.0357 6.80619 5.0357ZM9.3325 5.0357C9.03018 5.0357 8.78513 5.28077 8.78513 5.58307C8.78513 5.88538 9.03018 6.13044 9.3325 6.13044C9.63481 6.13044 9.87987 5.88538 9.87987 5.58307C9.87987 5.28077 9.63481 5.0357 9.3325 5.0357Z" fill="white"/>
|
||||
<path d="M12.2279 9.63738C12.2342 9.67527 12.2378 9.71342 12.2378 9.75165C12.2378 10.0464 12.1957 10.4674 11.9851 10.678C11.7074 10.9558 11.2272 11.478 10.9746 11.6885L6.34305 15.8569L7.35357 13.499L7.9831 12.677C8.20912 12.6273 8.41543 12.5774 8.57462 12.5306C9.29041 12.3201 10.1325 11.6885 10.7641 11.099C11.2418 10.6531 11.9076 9.97136 12.2279 9.63738ZM9.62725 3.77271C10.3248 3.77271 10.8904 4.33825 10.8904 5.03586V6.80428C10.8904 7.50191 10.3248 8.06744 9.62725 8.06744H8.4483L8.86935 6.59376L8.19659 7.34409L8.19568 7.35165L7.57479 8.06744H6.59568C5.89805 8.06744 5.33252 7.50191 5.33252 6.80428V5.03586C5.33252 4.33825 5.89805 3.77271 6.59568 3.77271H9.62725ZM6.8062 5.03586C6.5039 5.03586 6.25884 5.28093 6.25884 5.58323C6.25884 5.88554 6.5039 6.1306 6.8062 6.1306C7.10851 6.1306 7.35357 5.88554 7.35357 5.58323C7.35357 5.28093 7.10851 5.03586 6.8062 5.03586ZM9.33251 5.03586C9.0302 5.03586 8.78514 5.28093 8.78514 5.58323C8.78514 5.88554 9.0302 6.1306 9.33251 6.1306C9.63483 6.1306 9.87988 5.88554 9.87988 5.58323C9.87988 5.28093 9.63483 5.03586 9.33251 5.03586Z" fill="white"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.0 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 527 B |
@@ -0,0 +1,4 @@
|
||||
<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
|
||||
<path d="M8.06935 0.740967C8.46471 0.740967 8.78513 1.06143 8.78513 1.45675C8.78513 1.73028 8.6317 1.96783 8.40619 2.08833V2.6357C8.60265 2.66378 9.18471 2.67955 9.92197 3.01465C10.8483 3.4357 11.121 3.77803 11.4378 4.2357C11.6095 4.48376 11.7205 4.74914 11.7909 5.0357H11.9009C12.273 5.0357 12.5746 5.33732 12.5746 5.70938V6.38307C12.5746 6.75514 12.273 7.05675 11.9009 7.05675H11.8192C11.7591 7.40026 11.6762 7.69328 11.6062 7.85675C11.413 8.30755 11.129 8.48833 10.9746 8.53044C11.0869 8.57254 11.4395 8.67604 11.6483 8.82517C11.943 9.0357 12.2378 9.39174 12.2378 9.75149C12.2378 10.0462 12.1957 10.4673 11.9851 10.6778C11.7074 10.9556 11.2272 11.4778 10.9746 11.6883L6.34303 15.8568L7.35356 13.4989L10.3851 9.54096H8.02724L8.86934 6.59359L8.19658 7.34393L8.19567 7.35149L5.2483 10.762H7.6904L7.10093 12.7831L5.62724 11.8989L4.91146 11.4357C4.53251 11.1831 4.32198 11.0989 4.06935 10.6778C3.91617 10.4225 3.90093 10.0462 3.90093 9.75149C3.90093 9.39174 4.19567 9.0357 4.4904 8.82517C4.69915 8.67604 4.78514 8.61465 4.99567 8.53044C4.82724 8.44623 4.7257 8.30755 4.53251 7.85675C4.46245 7.69328 4.37956 7.40026 4.31951 7.05675H4.23777C3.8657 7.05675 3.56409 6.75514 3.56409 6.38307V5.70938C3.56409 5.33732 3.8657 5.0357 4.23777 5.0357H4.34788C4.41815 4.74914 4.5292 4.48376 4.70093 4.2357C5.01778 3.77803 5.2904 3.4357 6.21672 3.01465C6.95396 2.67955 7.53602 2.66378 7.73251 2.6357V2.08833C7.50704 1.96783 7.35356 1.73028 7.35356 1.45675C7.35356 1.06143 7.67403 0.740967 8.06935 0.740967ZM6.80619 5.0357C6.50389 5.0357 6.25882 5.28077 6.25882 5.58307C6.25882 5.88538 6.50389 6.13044 6.80619 6.13044C7.1085 6.13044 7.35356 5.88538 7.35356 5.58307C7.35356 5.28077 7.1085 5.0357 6.80619 5.0357ZM9.3325 5.0357C9.03018 5.0357 8.78513 5.28077 8.78513 5.58307C8.78513 5.88538 9.03018 6.13044 9.3325 6.13044C9.63481 6.13044 9.87987 5.88538 9.87987 5.58307C9.87987 5.28077 9.63481 5.0357 9.3325 5.0357Z" fill="#F69047"/>
|
||||
<path d="M12.2279 9.63738C12.2342 9.67527 12.2378 9.71342 12.2378 9.75165C12.2378 10.0464 12.1957 10.4674 11.9851 10.678C11.7074 10.9558 11.2272 11.478 10.9746 11.6885L6.34305 15.8569L7.35357 13.499L7.9831 12.677C8.20912 12.6273 8.41543 12.5774 8.57462 12.5306C9.29041 12.3201 10.1325 11.6885 10.7641 11.099C11.2418 10.6531 11.9076 9.97136 12.2279 9.63738ZM9.62725 3.77271C10.3248 3.77271 10.8904 4.33825 10.8904 5.03586V6.80428C10.8904 7.50191 10.3248 8.06744 9.62725 8.06744H8.4483L8.86935 6.59376L8.19659 7.34409L8.19568 7.35165L7.57479 8.06744H6.59568C5.89805 8.06744 5.33252 7.50191 5.33252 6.80428V5.03586C5.33252 4.33825 5.89805 3.77271 6.59568 3.77271H9.62725ZM6.8062 5.03586C6.5039 5.03586 6.25884 5.28093 6.25884 5.58323C6.25884 5.88554 6.5039 6.1306 6.8062 6.1306C7.10851 6.1306 7.35357 5.88554 7.35357 5.58323C7.35357 5.28093 7.10851 5.03586 6.8062 5.03586ZM9.33251 5.03586C9.0302 5.03586 8.78514 5.28093 8.78514 5.58323C8.78514 5.88554 9.0302 6.1306 9.33251 6.1306C9.63483 6.1306 9.87988 5.88554 9.87988 5.58323C9.87988 5.28093 9.63483 5.03586 9.33251 5.03586Z" fill="#C45259"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.0 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 4.3 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 4.5 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 202 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 529 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 9.5 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 1.9 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 310 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 235 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 817 KiB |
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 128 KiB |
@@ -0,0 +1,211 @@
|
||||
# Changelog
|
||||
|
||||
## Agent-lightning v0.3.0 (12/24/2025)
|
||||
|
||||
Agent-lightning v0.3.0 is a major release that introduces several new features and bug fixes. The release is a collaborative effort between Agent-lightning core teams and the community. Thanks to all the contributors who made this release possible.
|
||||
|
||||
### Highlights
|
||||
|
||||
* **Tinker integration**: Support Tinker as an alternative backend for Reinforcement Learning (#226 #245 #264 #269 #327). See [example code](https://github.com/microsoft/agent-lightning/tree/v0.3.0/examples/tinker), [blog 1](https://medium.com/@yugez/tuning-any-ai-agent-with-tinker-agent-lightning-part-1-1d8c9a397f0e) and [blog 2](https://medium.com/@yugez/tuning-any-ai-agent-with-tinker-agent-lightning-part-2-332c5437f0dc).
|
||||
* **Azure OpenAI integration**: Support Azure OpenAI as a backend for LLM inference and supervised fine-tuning (#256 #327). [Example code](https://github.com/microsoft/agent-lightning/tree/v0.3.0/examples/azure).
|
||||
* **MongoDB-based Lightning Store** is added as an alternative backend for Lightning Store (#323). [Documentation](https://microsoft.github.io/agent-lightning/0.3.0/tutorials/parallelize/#parallelizing-lightningstore).
|
||||
* **Contrib package**: Add contrib package for community projects. Search-R1 is integrated as a contrib recipe. More coming. (#239 #396 #410 #412 #417).
|
||||
* **RESTful API**: Stabilize and document RESTful API for Lightning Store (#241 #275). [Documentation](https://microsoft.github.io/agent-lightning/0.3.0/reference/restful/).
|
||||
* **OTel Semantic Conventions** that are specifically designed for Agent-optimization areas (#340). [Documentation](https://microsoft.github.io/agent-lightning/0.3.0/reference/semconv/).
|
||||
* *[Preview]* **Agent-lightning Dashboard** is now available (#288 #289 #291 #296 #371 #375). It's the official web application for inspecting and debugging Agent-lightning experiments. See details [here](https://microsoft.github.io/agent-lightning/0.3.0/tutorials/debug/).
|
||||
* *[Preview]* **Multi-modality example** featuring VERL and a LangGraph agent on ChartQA dataset (#379). [Example code](https://github.com/microsoft/agent-lightning/tree/v0.3.0/examples/chartqa).
|
||||
* *[Preview]* Integrate **Claude Code** as a LitAgent and support training on SWE-Bench (#332 #346 #348). [Example code](https://github.com/microsoft/agent-lightning/tree/v0.3.0/examples/claude_code).
|
||||
* *[Preview]* **Weave tracer** as a substitute for AgentOps tracer (#277 #411 #420 #423). [Documentation](https://microsoft.github.io/agent-lightning/0.3.0/tutorials/traces/#weave-tracer-experimental).
|
||||
* *[Preview]* **Trajectory Level Aggregation** for more efficient training with VERL. See [blog](https://agent-lightning.github.io/posts/trajectory_level_aggregation/) and [documentation](https://microsoft.github.io/agent-lightning/0.3.0/algorithm-zoo/verl/).
|
||||
|
||||
### Store Benchmark
|
||||
|
||||
In this release, the Lightning Store core was redesigned for significantly greater efficiency and scalability (#315 #318 #328 #342 #344 #356 #380 #388 #418 #421). The benchmark results below demonstrate the impact: with large numbers of concurrent runners, v0.3.0 delivers up to a 15x increase in throughput compared to v0.2.2.
|
||||
|
||||
| Throughput (\#rollout/sec) | v0.2.2 | v0.3.0 (in-memory) | v0.3.0 (Mongo) |
|
||||
| :---- | :---- | :---- | :---- |
|
||||
| Minimal (batch, #runner=32, #turns=6) | 8.73 | 9.06 | 8.71 |
|
||||
| Medium (batch, #runners=100, #turns=10) | 12.03 | 23.26 | 32.79 |
|
||||
| Mid-high (batch, #runners=300, #turns=6) | 10.61 | 24.42 | 40.24 |
|
||||
| Large (batch, #runners=1000, #turns=3) | 3.36 | 14.60 | 50.05 |
|
||||
| Long queue (queue, #runners=256, #turns=4) | 7.42 | 30.86 | 57.01 |
|
||||
| Heavy trace (queue, #runners=512, #turns=20) | 5.93 | 13.28 | 29.41 |
|
||||
|
||||
*Notes:*
|
||||
|
||||
1. Benchmarks were run on a single Standard_D32as_v4 Azure VM (Large and heavy trace tests used Standard_D64ads_v5), executed via GitHub Actions.
|
||||
2. Two algorithm patterns are evaluated: the batch pattern submits a group of rollouts and waits for all to finish before starting the next group, while the queue pattern maintains a set number of in-flight rollouts, submitting new ones as soon as capacity frees up. Configuration details are available [here](https://github.com/microsoft/agent-lightning/blob/v0.3.0/.github/workflows/benchmark.yml).
|
||||
3. The number of turns is directly proportional to the number of spans each rollout generates.
|
||||
|
||||
### Maintenance and Bug fixes
|
||||
|
||||
#### Core (Store, Interfaces, etc.)
|
||||
|
||||
* Add Trainer port option for client-server strategies (#198)
|
||||
* Fix store port conflict handling (#227)
|
||||
* Unified PythonServerLauncher (#286 #292 #303)
|
||||
* Make health timeout configurable (#305)
|
||||
* Refactor logging (#306)
|
||||
* Support OTLP in LightningStore (#313)
|
||||
* Centralized metrics helper (#368)
|
||||
* Fix redundant cancel tracebacks on Ctrl+C (#370)
|
||||
|
||||
#### Proxy, Adapters and Algorithms
|
||||
|
||||
* Fix training metrics before and after processing in VERL (#145)
|
||||
* Forward streaming requests for Anthropic and OpenAI APIs (as non-streaming requests) (#299)
|
||||
* Check traces with reward for VERL (#317)
|
||||
* Patch LiteLLM root span (#341)
|
||||
* Handle ref_in_actor flag for LoRA compatibility (#386)
|
||||
* Support `with_llm_proxy` and `with_store` in algorithms (#398)
|
||||
* Support image URL export in TracerTraceToTriplets (#400)
|
||||
* Fix match_rewards assign_to elements in TraceTree (#403)
|
||||
* Support customizing trainer and daemon in VERL (#407)
|
||||
|
||||
#### Runners, Tracers and Agents
|
||||
|
||||
* Refactor tracer initialization (#321)
|
||||
* Fix OpenAI Agents 0.6 compatibility (#322)
|
||||
* `emit_operation`, `emit_annotation`, tags and links (#359)
|
||||
* Sunset HTTP tracer (#402)
|
||||
|
||||
#### Examples
|
||||
|
||||
* Fix typos in train-first-agent.md (#263)
|
||||
* Fix room_selector example which always runs the first task (#270)
|
||||
* Fix typo in SQL agent example (#285)
|
||||
* Add the README and script files for training SQL agent on NPU (#272)
|
||||
* Examples Catalog and Refine Contribution Guide (#331)
|
||||
* Upgrade LangChain to 1.x (#364)
|
||||
* Update RAG example to Agent-lightning v0.2.x (#349)
|
||||
|
||||
#### Miscellaneous
|
||||
|
||||
* DeepWiki Badge (#263)
|
||||
* Add AGENTS.md (#374)
|
||||
|
||||
### New Contributors
|
||||
|
||||
Warm welcome to our first-time contributors: @cptnm3, @TerryChan, @genji970, @zxgx, @xiaochulaoban, @lspinheiro, @Kwanghoon-Choi, @Vasuk12, @totoluo, @jinghuan-Chen 🎉
|
||||
|
||||
**Full Changelog**: https://github.com/microsoft/agent-lightning/compare/v0.2.0...v0.3.0
|
||||
|
||||
---
|
||||
|
||||
## Agent-lightning v0.2.2 (11/12/2025)
|
||||
|
||||
Agent-lightning v0.2.2 is a stabilization release for v0.2.1. It introduces several bug fixes.
|
||||
|
||||
* Fix compatibility issues with VERL 0.6.0.
|
||||
* Fix model name for pre-downloaded models in VERL.
|
||||
* Fix preparing status transition on rollout when creating attempts.
|
||||
* Fix OpenAI Agents SDK compatibility issues.
|
||||
|
||||
**Full Changelog**: https://github.com/microsoft/agent-lightning/compare/v0.2.1...v0.2.2
|
||||
|
||||
---
|
||||
|
||||
## Agent-lightning v0.2.1 (10/30/2025)
|
||||
|
||||
Agent-lightning v0.2.1 is a stabilization release for v0.2.0. It introduces several bug fixes and new features, plus a number of unlisted CI improvements.
|
||||
|
||||
### Bug fixes
|
||||
|
||||
* Fix LiteLLM issues when restarting the proxy multiple times in the same process (#174 #206)
|
||||
* Fix LiteLLM model name selection when multiple servers use the same model (#197)
|
||||
* Fix store port conflict handling (#227)
|
||||
|
||||
### New Features
|
||||
|
||||
* Add trainer port option for client-server strategies (#198)
|
||||
|
||||
### Documentation
|
||||
|
||||
* Add tutorial for launching workers on separate machines (#213)
|
||||
* Add link to VERL framework (#210)
|
||||
* Add link to vLLM blog (#215)
|
||||
* Fix a couple of typos and avoid emacs backup files (#237)
|
||||
|
||||
### New Contributors
|
||||
|
||||
A warm welcome to our first-time contributors: @scott-vsi, @ddsfda99, @jeis4wpi 🎉
|
||||
|
||||
**Full Changelog**: https://github.com/microsoft/agent-lightning/compare/v0.2.0...v0.2.1
|
||||
|
||||
---
|
||||
|
||||
## Agent-lightning v0.2.0 (10/22/2025)
|
||||
|
||||
Agent-Lightning v0.2.0 introduces major framework improvements, new execution strategies, expanded documentation, and enhanced reliability across the agent training and deployment workflow. This release includes **78 pull requests** since v0.1.2.
|
||||
|
||||
### Core Enhancements
|
||||
|
||||
* **Lightning Store**: Added unified interface and implementation for Agent-lightning's core storage.
|
||||
* **Emitter**: Emitting any objects as spans to the store.
|
||||
* **Adapter** and **Tracer**: Adapting to OpenAI-like messages, and OpenTelemetry dummy tracer.
|
||||
* **LLM Proxy**: Added LLM Proxy as the first-class citizen in Agent-lightning.
|
||||
* **Agent Runner**: New version providing a more modular and robust runner design.
|
||||
* **Embedded Algorithms**: Algorithms are now embedded directly into trainers for simplicity.
|
||||
* **New Execution Strategies**: Introduced *Client-Server* and *Shared Memory* execution models.
|
||||
* **Trainer Updates**: Integrated v0.2 interfaces and FastAlgorithm validation.
|
||||
|
||||
### Documentation & Examples
|
||||
|
||||
* Revamped documentation with new guides for **agent creation**, **training**, **debugging**, and **store concepts**.
|
||||
* Improved quickstart tutorials, clarified installation and new deep-dive articles.
|
||||
* Added and updated examples: *SQL Agent*, *Calc-X*, *Local SFT*, *Search-R1*, and *APO algorithm*.
|
||||
|
||||
### Developer Experience
|
||||
|
||||
* Migrated build and CI pipelines to **1ES**, split workflows and aggregate badges for clarity.
|
||||
* Adopted **uv** as the dependency manager.
|
||||
* Added GPU-based pytest workflows for full test coverage.
|
||||
* Enhanced debugging UX, pre-commit configs, and linting (Pyright fixes, import sorting).
|
||||
|
||||
### Ecosystem & Integrations
|
||||
|
||||
* Added support for agents built with [**Agent-framework**](https://github.com/microsoft/agent-framework).
|
||||
* Added new community listings: [*DeepWerewolf*](https://github.com/af-74413592/DeepWerewolf) and [*AgentFlow*](https://agentflow.stanford.edu/).
|
||||
|
||||
### New Contributors
|
||||
|
||||
A warm welcome to our first-time contributors:
|
||||
@hzy46, @lunaqiu, @syeehyn, @linhx1999, @SiyunZhao, and @acured 🎉
|
||||
|
||||
**Full changelog:** [v0.1.2 → v0.2.0](https://github.com/microsoft/agent-lightning/compare/v0.1.2...v0.2.0)
|
||||
|
||||
---
|
||||
|
||||
## Agent-lightning v0.1.2 (08/12/2025)
|
||||
|
||||
### What's Changed
|
||||
* Add basic documentation in https://github.com/microsoft/agent-lightning/pull/33
|
||||
* RAG example by @wizardlancet in https://github.com/microsoft/agent-lightning/pull/21
|
||||
|
||||
### New Contributors
|
||||
* @wizardlancet made their first contribution in https://github.com/microsoft/agent-lightning/pull/21
|
||||
|
||||
**Full Changelog**: https://github.com/microsoft/agent-lightning/compare/v0.1.1...v0.1.2
|
||||
|
||||
---
|
||||
|
||||
## Agent-lightning v0.1.1 (08/06/2025)
|
||||
|
||||
### What's Changed
|
||||
* Disable HTTP tracer tests and bump to 0.1.1 in https://github.com/microsoft/agent-lightning/pull/26
|
||||
* Fix trainer bugs in v0.1 in https://github.com/microsoft/agent-lightning/pull/24
|
||||
|
||||
**Full Changelog**: https://github.com/microsoft/agent-lightning/compare/v0.1...v0.1.1
|
||||
|
||||
---
|
||||
|
||||
## Agent-lightning v0.1.0 (08/04/2025)
|
||||
|
||||
The first release of Agent-lightning!
|
||||
|
||||
- Turn your agent into an optimizable beast with **ZERO CODE CHANGE** (almost)! 💤
|
||||
- Build with **ANY** agent framework (LangChain, OpenAI Agent SDK, AutoGen, CrewAI, ...); or even WITHOUT agent framework (Python OpenAI). You name it! 🤖
|
||||
- **Selectively** optimize one or more agents in a multi-agent system. 🎯
|
||||
- Embraces Reinforcement Learning, Automatic Prompt Optimization and more **algorithms**. 🤗
|
||||
|
||||
Install via `pip install agentlightning`.
|
||||
@@ -0,0 +1,253 @@
|
||||
# Contributing Guide
|
||||
|
||||
Agent Lightning gets better every time someone files a clear bug, polishes docs, improves tests, or lands a new feature. This guide collects the expectations, checklists, and tips that help you go from “I have an idea” to “my pull request just merged.”
|
||||
|
||||
## Before You Start
|
||||
|
||||
Agent-lightning is built by a small Microsoft Research team with limited reviewer hours and GPU budget. For any sizeable change (new algorithm, example, or API surface) please first discuss scope with us in [Discord](https://discord.gg/RYk7CdvDR7). Early alignment keeps your effort from being blocked late in the process.
|
||||
|
||||
## Where You Can Help
|
||||
|
||||
Pick a lane, or combine several. Just keep the discussion-first principle in mind for anything non-trivial.
|
||||
|
||||
### Documentation Improvements
|
||||
|
||||
Documentation improvements are the easiest way to get started. You can find more about how to write good documentations and organize documentations in the following sections. Here are some general contribution points we can think of:
|
||||
|
||||
- Tighten language, fix typos, clarify confusing sections, or add missing links. Fresh eyes catch docs gaps best.
|
||||
- Organize content using the directories listed below so readers can actually find it.
|
||||
- Avoid duplicate prose, unrelated “how-to” guides, or translations (we cannot maintain them today).
|
||||
|
||||
!!! note "Changes that are usually rejected"
|
||||
|
||||
- Copy/pasting existing docs with shallow edits.
|
||||
- Adding a `how-to` guide that is not tied to a new example.
|
||||
- Adding doc translations to other languages (no capacity to review/maintain yet).
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
Bug fixes are the fastest way to get familiar with the codebase. To get started, you can:
|
||||
|
||||
- Browse the ["help wanted"](https://github.com/microsoft/agent-lightning/labels/help%20wanted) and ["bug"](https://github.com/microsoft/agent-lightning/labels/bug) labels; drop a comment before you start so we can mark it as taken.
|
||||
- For fresh bugs, open an issue with reproduction steps, logs, and expected behavior before submitting a fix.
|
||||
- Keep each pull request focused, ideally avoiding breaking API changes. Larger refactors should be discussed via RFC or maintainer sync.
|
||||
|
||||
### New Examples
|
||||
|
||||
Examples must be curated so that we can maintain them. We generally merge only those that meet at least one (ideally several) of these criteria:
|
||||
|
||||
- Demonstrates an agent framework or workflow that is materially different from what already exists. ([LangChain](https://www.langchain.com/) vs. [LlamaIndex](https://www.llamaindex.ai/) is not different enough; [LangChain](https://www.langchain.com/) vs. [n8n](https://n8n.io/) or [Vercel AI SDK](https://ai-sdk.dev/) is, because they either have different orchestration paradigms or differ in programming languages.)
|
||||
- Shows measurable performance gains on a **real-world** problem with a **real-world** dataset, such as tuning a search agent with Google Search API or improving a coding agent’s (e.g., Claude Code) SWE-Bench score.
|
||||
- Integrates a new algorithm, training backend, or serving stack (see “New Algorithms” below).
|
||||
- Validates scenarios that are rarely tested, such as multi-modality agents or long-lived memory/workflow agents.
|
||||
|
||||
Bonus points for examples that:
|
||||
|
||||
- Ship CI or self-test coverage so we know they still work as the core evolves. **Otherwise, we would have to mark the example as unmaintained because we won't be able to test the examples manually before each release.**
|
||||
- Include a [`docs/how-to/`]({{ src("docs/how-to/") }}) guide (or a detailed README if no how-to exists) without duplicating content in multiple places.
|
||||
- Favor simple, dependency-light code over heavy abstractions.
|
||||
- Ship a README that documents smoke-test instructions and includes an "Included Files" section summarizing every file and its role; keep the runnable module self-contained with a module-level docstring explaining CLI usage, plus targeted docstrings or inline comments for educational functions/classes.
|
||||
|
||||
!!! warning "Please discuss first"
|
||||
|
||||
Examples tend to be the most time-consuming contributions for both you and reviewers. Sync with us on Discord or through an issue before diving into a new one.
|
||||
|
||||
### Fresh Implementations of Core Modules
|
||||
|
||||
If you are looking to extend [`Runner`][agentlightning.Runner], [`Tracer`][agentlightning.Tracer], [`Adapter`][agentlightning.Adapter], [`LightningStore`][agentlightning.LightningStore], or another core interface, here are the steps:
|
||||
|
||||
1. File an issue or proposal first.
|
||||
2. Explain which interface you are extending, why existing implementations are insufficient, and how you intend to test compatibility with the rest of the stack (unit tests, documentation updates, example refreshes, etc.).
|
||||
3. Any API changes must be reviewed up front. DO NOT begin coding large changes before the discussion lands!
|
||||
|
||||
### New Algorithms
|
||||
|
||||
If you are integrating a new training/serving backend, check whether it already lives in the [Algorithm Zoo](../algorithm-zoo/index.md) or is covered in the [Examples Catalog](../how-to/examples-catalog.md). We especially welcome:
|
||||
|
||||
- Currently unsupported or under-tested algorithms such as Supervised Fine-tuning (SFT), Direct Policy Optimization (DPO), or Monte Carlo Tree Search (MCTS).
|
||||
- Tuning [Resource][agentlightning.Resource]s that are not supported yet, such as workflows or memory.
|
||||
- Expansions of supported stacks, e.g., adding multi-modality to APO or multi-agent prompt tuning.
|
||||
- Reinforcement-learning integrations beyond our current stack of [VERL](https://github.com/volcengine/verl), [vLLM](https://vllm.ai/), [Azure OpenAI](https://azure.microsoft.com/en-us/products/ai-foundry/models/openai), and [Tinker](https://tinker-docs.thinkingmachines.ai/). Contributions using [SGLang](https://github.com/sgl-project/sglang), [TRL](https://github.com/huggingface/trl), [SkyRL](https://github.com/NovaSky-AI/SkyRL), [RLinf](https://github.com/RLinf/RLinf), [litgpt](https://github.com/Lightning-AI/litgpt), or similar are welcome.
|
||||
|
||||
Most brand-new algorithms ultimately land as “new examples,” so read that section too. Post an issue or design doc to scope the work, reuse existing utilities, and avoid duplicating efforts. Mature, battle-tested examples graduate into the [Algorithm Zoo](../algorithm-zoo/index.md).
|
||||
|
||||
### Ecosystem Projects
|
||||
|
||||
Have a project that builds on Agent-lightning but does not belong in the main repo? Fork it or depend on it externally, then let us know. We can showcase notable projects in [Community Projects](../index.md) and the main [README]({{ src("README.md") }}).
|
||||
|
||||
### Agent-lightning Contrib
|
||||
|
||||
[`contrib/`]({{ src("contrib") }}) is where work-in-progress or third-party integrations, and curated recipes live before they are hardened enough for the core runtime tree. Think of it as an incubator: additions should remain easy to consume, clearly owned, and scoped so downstream users can vendor them with minimal risk.
|
||||
|
||||
The following types of contributions are welcome in the contrib area:
|
||||
|
||||
- **Recipes** that assemble multiple Agent Lightning components for a narrow task (`contrib/recipes/<topic>/`). Each recipe must be self-contained, include running instructions and result reports.
|
||||
- **Runtime extensions** that would bloat the primary `agentlightning/` namespace (`contrib/agentlightning/contrib/<feature>/`). These should mirror the published wheel layout so that `import agentlightning.contrib.<feature>` works out of the box.
|
||||
- **Supporting scripts and assets** (`contrib/scripts/`) that automate dataset downloads, environment preparation, or benchmarks required by contrib modules.
|
||||
|
||||
If you are unsure where a contribution should live, start a thread in Discord or open an issue before writing code. The [contrib README]({{ src("contrib/README.md") }}) also lists the directory expectations.
|
||||
|
||||
A quick checklist for contributions to be accepted:
|
||||
|
||||
1. **Document everything.** Include configuration steps, environment variables, and sample commands so contributors can reproduce the results without guesswork. Pin to a specific version of Agent-lightning and other dependencies to avoid unexpected changes if you don't want to update the recipe frequently.
|
||||
2. **Keep quality predictable.** Match the repo’s style guide, apply exhaustive type hints, and run `uv run --no-sync pyright` plus targeted `pytest` suites for any Python module you touch.
|
||||
3. **Ship reproducibility artifacts.** Store only scripts or instructions for downloading datasets, weights, or binaries. Never upload large artifacts or credentials directly.
|
||||
4. **Update ownership.** Add `CODEOWNERS` entries when new directories appear so maintainers know who can review follow-up fixes.
|
||||
|
||||
Contrib entries do not need the same maturity level as core code, but they must still meet the baseline above. Submissions that lack documentation, hide ownership, or depend on untracked assets are typically rejected until those gaps are resolved.
|
||||
|
||||
### Other Contribution Ideas
|
||||
|
||||
- **Tests.** Add or improve cases in [`tests/`]({{ src("tests") }}) (unit, integration, or end-to-end).
|
||||
- **Benchmarks.** Expand [`tests/benchmark`]({{ src("tests/benchmark") }}) to stress large-scale training or rollouts.
|
||||
- **Issue triage.** Reproduce bugs, confirm whether they reproduce on `main`, or suggest short-term mitigations so maintainers can prioritize.
|
||||
|
||||
## Contribution Workflow
|
||||
|
||||
The steps below keep changes reviewable and CI-friendly. Follow them in order; rerun the relevant pieces if you revisit a branch later.
|
||||
|
||||
### 1. Prepare Your Environment
|
||||
|
||||
Minimum tooling:
|
||||
|
||||
- **Python** 3.10+ (3.12 recommended).
|
||||
- **uv** for dependency and virtual-environment management. Install it using the [official uv docs](https://docs.astral.sh/uv/getting-started/installation/).
|
||||
- **Git** configured with your GitHub credentials.
|
||||
|
||||
Clone your fork and point `upstream` at the official repo:
|
||||
|
||||
```bash
|
||||
git clone git@github.com:<your-username>/agent-lightning.git
|
||||
cd agent-lightning
|
||||
git remote add upstream https://github.com/microsoft/agent-lightning.git
|
||||
```
|
||||
|
||||
Install the default development stack:
|
||||
|
||||
```bash
|
||||
uv sync --group dev
|
||||
```
|
||||
|
||||
Need GPU extras or specific optional dependencies? Lock them in with one command:
|
||||
|
||||
```bash
|
||||
uv sync --frozen \
|
||||
--extra apo \
|
||||
--extra verl \
|
||||
--group dev \
|
||||
--group torch-cpu \
|
||||
--group torch-stable \
|
||||
--group agents \
|
||||
--no-default-groups
|
||||
```
|
||||
|
||||
After `uv sync`, run commands via `uv run ...` (add `--no-sync` once the environment is locked) or activate `.venv/`.
|
||||
|
||||
### 2. Install and Run Pre-commit
|
||||
|
||||
Formatting and linting are enforced through [pre-commit](https://pre-commit.com/). Install once, then run before each push:
|
||||
|
||||
```bash
|
||||
uv run --no-sync pre-commit install
|
||||
uv run --no-sync pre-commit run --all-files --show-diff-on-failure --color=always
|
||||
```
|
||||
|
||||
Once installed, the hooks run automatically on every `git commit`. Running the pre-commit hooks locally keeps CI green and diffs manageable.
|
||||
|
||||
### 3. Branch from Fresh `main` and Code
|
||||
|
||||
Start all work from the latest upstream state:
|
||||
|
||||
```bash
|
||||
git fetch upstream
|
||||
git checkout main
|
||||
git merge upstream/main
|
||||
```
|
||||
|
||||
Branch naming convention:
|
||||
|
||||
- `feature/<short-description>` for new features.
|
||||
- `fix/<short-description>` for bug fixes.
|
||||
- `docs/<short-description>` for documentation-only updates.
|
||||
- `chore/<short-description>` for tooling or maintenance.
|
||||
|
||||
Use lowercase with hyphens, e.g., `feature/async-runner-hooks`.
|
||||
|
||||
!!! note "Where should docs or examples live?"
|
||||
|
||||
Many new contributors get confused about what to put in the `docs/how-to/` directory and what to put in the `examples/` directory (particularly README files). Here is a quick reference you can refer to:
|
||||
|
||||
| Location | Description |
|
||||
| --- | --- |
|
||||
| `docs/algorithm-zoo/` | Documentation for **built-in algorithms** shipped with Agent-lightning. |
|
||||
| `docs/how-to/` | Step-by-step **how-to guides**, usually tied to an example in `examples/`. |
|
||||
| `docs/tutorials/` | Conceptual walkthroughs for components or workflows. See [debugging](../tutorials/debug.md) or [parallelization](../tutorials/parallelize.md) for examples. |
|
||||
| `docs/deep-dive/` | Advanced explanations and in-depth concepts. |
|
||||
| `examples/<name>/README.md` | Example-specific README. If any related how-to if that exists, link to it avoid duplicating the same instructions twice; write only brief instructions on how to install and run the example. Otherwise, you can make the README more detailed and self-explanatory. |
|
||||
|
||||
Remember to register new docs in [`mkdocs.yml`]({{ src("mkdocs.yml") }}), add examples to [examples/README]({{ src("examples/README.md") }}), and update the [Examples Catalog](../how-to/examples-catalog.md).
|
||||
|
||||
Before you start coding, bring the shared coding conventions with you:
|
||||
|
||||
- Target `requires-python >= 3.10`, four-space indentation, ~120-character lines (docstrings may run longer), and formatter-owned diffs (Black + isort with the `black` profile).
|
||||
- Use `snake_case` for modules, functions, and variables; `PascalCase` for classes and React components; lowercase hyphenation for CLI flags, branch names, and TypeScript filenames.
|
||||
- Maintain exhaustive type hints (pyright enforces them), write succinct Google-style docstrings (with `[][]` cross-references).
|
||||
- Prefer dataclasses or Pydantic models from `agentlightning.types`.
|
||||
- Log via `logging.getLogger(__name__)` with targeted DEBUG/INFO/WARNING/ERROR calls—especially for long multi-step functions or broad `try/except` blocks.
|
||||
|
||||
### 4. Test and Validate
|
||||
|
||||
Most contributions require automated checks. Once `uv sync` locks dependencies, prefix commands with `uv run --no-sync ...` so they share the same environment as CI.
|
||||
|
||||
**Full test suite**
|
||||
|
||||
```bash
|
||||
uv run --no-sync pytest -v
|
||||
```
|
||||
|
||||
**Targeted tests**
|
||||
|
||||
```bash
|
||||
uv run --no-sync pytest tests/path/to/test_file.py -k test_name
|
||||
```
|
||||
|
||||
**Optional/gated tests:** GPU-specific suites or API-dependent tests run automatically when the required hardware or environment variables (such as `OPENAI_API_KEY`) are present.
|
||||
|
||||
**Static analysis:**
|
||||
|
||||
```bash
|
||||
uv run --no-sync pyright
|
||||
```
|
||||
|
||||
If you have touched code under `examples/`, you should run the example-specific smoke tests. Each directory includes a README with example-specific smoke tests—run those too.
|
||||
|
||||
!!! note "Build documentation when needed"
|
||||
|
||||
Keep API references under [docs/reference]({{ src("docs/reference/") }}) up to date. Doc-only changes should still build cleanly:
|
||||
|
||||
```bash
|
||||
uv run --no-sync mkdocs serve --strict # live reload
|
||||
uv run --no-sync mkdocs build --strict # CI-equivalent
|
||||
```
|
||||
|
||||
`--strict` elevates warnings to errors so you catch issues before CI.
|
||||
|
||||
Before opening a PR, double-check the basics:
|
||||
|
||||
- Run `uv lock` if you changed dependencies.
|
||||
- Run `uv run --no-sync pre-commit run --all-files --show-diff-on-failure` (hooks installed via `pre-commit install` run automatically on `git commit`, but rerun them if you amended history).
|
||||
- Execute the relevant commands from the test list above.
|
||||
- Validate each affected example via its README instructions.
|
||||
|
||||
### 5. Open a Pull Request
|
||||
|
||||
1. Push your branch:
|
||||
```bash
|
||||
git push origin <branch-name>
|
||||
```
|
||||
2. Open a PR against `microsoft/agent-lightning:main`.
|
||||
3. Fill out the template with a concise summary, the commands/tests you ran, and linked issues (use `Fixes #123` syntax to auto-close).
|
||||
4. Include screenshots or logs if they clarify behavior.
|
||||
5. Address review feedback promptly. Follow-up tweaks work best as focused commits; `git commit --fixup` is handy for reviewer-suggested edits.
|
||||
|
||||
Thanks for contributing! every improvement strengthens the Agent Lightning community!
|
||||
@@ -0,0 +1,86 @@
|
||||
# Maintainer Guide
|
||||
|
||||
This guide describes the day-to-day responsibilities for Agent Lightning maintainers—how to bump versions, run release ceremonies, interact with CI, and backport fixes safely.
|
||||
|
||||
## Release Workflow
|
||||
|
||||
Follow this checklist throughout each release cycle.
|
||||
|
||||
### Immediately After Shipping
|
||||
|
||||
Agent Lightning uses a **bump-first** strategy. As soon as a release is published:
|
||||
|
||||
1. Update version metadata:
|
||||
- `pyproject.toml`: bump the `version`.
|
||||
- `agentlightning/__init__.py`: update `__version__` if it exists.
|
||||
- `uv.lock`: refresh the lock file after the bump.
|
||||
2. Refresh dependency pins as needed:
|
||||
```bash
|
||||
uv lock --upgrade
|
||||
```
|
||||
|
||||
3. For a new minor or major release, create a stable branch from `main`:
|
||||
```bash
|
||||
git checkout main
|
||||
git pull upstream main
|
||||
git checkout -b stable/v2.0.x # adjust to the new series
|
||||
git push upstream stable/v2.0.x
|
||||
```
|
||||
|
||||
All future changes to the stable branch must land via pull requests.
|
||||
|
||||
### Preparing the Next Release
|
||||
|
||||
When it is time to publish the next version:
|
||||
|
||||
1. **Draft release notes** in `docs/changelog.md`, collecting every notable change since the previous tag.
|
||||
2. **Open a release PR** targeting `main` (for minor/major) or the relevant stable branch (for patch releases). Use the title `[Release] vX.Y.Z`.
|
||||
3. **Run extended CI** by labeling the PR with `ci-all` and commenting `/ci`. Investigate and resolve any failures.
|
||||
4. **Merge the release PR** once notes are final and CI is green.
|
||||
5. **Tag the release** from the branch you just merged into:
|
||||
|
||||
```bash
|
||||
git checkout main # minor/major releases
|
||||
git checkout stable/vX.Y.Z # patch releases
|
||||
|
||||
git pull
|
||||
git tag vX.Y.Z -m "Release vX.Y.Z"
|
||||
git push upstream vX.Y.Z
|
||||
```
|
||||
|
||||
Pushing the tag publishes to PyPI and deploys the documentation.
|
||||
|
||||
6. **Publish the GitHub release** using the drafted notes, and confirm the docs site and PyPI listing reflect the new version.
|
||||
|
||||
## Working with CI Labels and `/ci`
|
||||
|
||||
GPU suites and example end-to-end runs are opt-in. To trigger them on a pull request:
|
||||
|
||||
1. Apply the appropriate labels before issuing the command:
|
||||
- `ci-all` for every repository-dispatch workflow.
|
||||
- `ci-gpu` for GPU integration tests (`tests-full.yml`).
|
||||
- `ci-apo`, `ci-calc-x`, `ci-spider`, `ci-unsloth`, `ci-compat` for the individual example pipelines.
|
||||
2. Comment `/ci` on the PR. The `issue-comment` workflow acknowledges the request and tracks job results inline.
|
||||
3. Remove the labels once you have the signal to avoid accidental re-runs.
|
||||
|
||||
Use `/ci` whenever changes touch shared infrastructure, dependencies, or training loops that require coverage beyond the default PR checks.
|
||||
|
||||
!!! note
|
||||
|
||||
`/ci` always executes the workflow definitions on the current `main` branch, then checks out the PR diff. If you need to test workflow modifications, push the changes to a branch in the upstream repo and run:
|
||||
|
||||
```bash
|
||||
gh workflow run examples-xxx.yml --ref your-branch-name
|
||||
```
|
||||
|
||||
## Backporting Pull Requests
|
||||
|
||||
Supported stable branches rely on automated backports:
|
||||
|
||||
1. Identify the target branch (for example `stable/v0.2.x`).
|
||||
2. Before merging the original PR into `main`, add the matching `stable/<series>` label (e.g. `stable/v0.2.x`).
|
||||
3. The `backport.yml` workflow opens a follow-up PR named `backport/<original-number>/<target-branch>` authored by `agent-lightning-bot`.
|
||||
4. Review the generated PR, ensure CI is green, and merge into the stable branch.
|
||||
5. Resolve conflicts by pushing manual fixes to the backport branch and re-running `/ci` if required.
|
||||
|
||||
Keep stable branches healthy by cherry-picking only critical fixes and ensuring documentation and example metadata stay aligned with each release line.
|
||||
@@ -0,0 +1,539 @@
|
||||
# The Bird's Eye View of Agent-lightning
|
||||
|
||||
!!! warning "High Volume of Information Ahead"
|
||||
|
||||
This article provides an in-depth exploration of the Agent-lightning architecture.
|
||||
It is not intended as a beginner’s guide or usage tutorial.
|
||||
|
||||
This article summarizes how Agent-lightning (as of v0.2) wires the [Algorithm][agentlightning.Algorithm], [Runner][agentlightning.Runner], and [LightningStore][agentlightning.LightningStore] loop together and shows where auxiliary components (the [Tracer][agentlightning.Tracer], [Adapter][agentlightning.Adapter], and [LLM Proxy][agentlightning.LLMProxy]) plug into the loop. Each section provides a diagram for a different perspective of the system.
|
||||
|
||||
## Algorithm ↔ Runner ↔ Store data flow
|
||||
|
||||
At its heart, Agent-lightning is built on three main components that work in a coordinated loop:
|
||||
|
||||
* **[Algorithm][agentlightning.Algorithm]:** The "brain" of the system. It decides what tasks to run, learns from the results, and updates resources (like AI models or prompts).
|
||||
* **[Runner][agentlightning.Runner]:** The "worker" of the system. It executes tasks assigned by the algorithm, runs the agent, and records the results.
|
||||
* **[LightningStore][agentlightning.LightningStore]:** The central "database" and message queue. It acts as the single source of truth, storing tasks, results, and resources, and enabling communication between the Algorithm and Runner.
|
||||
|
||||
The typical data flow in a training loop is as follows: The **[Algorithm][agentlightning.Algorithm]** enqueues tasks (called **[Rollouts][agentlightning.Rollout]**) into the **[LightningStore][agentlightning.LightningStore]**. A **[Runner][agentlightning.Runner]** then dequeues a task, executes it, and streams the results (called **[Spans][agentlightning.Span]**) back to the store. Once the task is complete, the algorithm can query the new data from the store to learn and update its resources.
|
||||
|
||||
The diagram below shows this fundamental interaction in a simple, non-parallel setup.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant Algo as Algorithm
|
||||
participant Store as LightningStore
|
||||
participant Runner
|
||||
participant Agent
|
||||
|
||||
loop Over the dataset
|
||||
Algo-->>Store: add_resources + enqueue_rollout
|
||||
Store-->>Runner: dequeue_rollout → AttemptedRollout
|
||||
Store-->>Runner: get_latest_resources
|
||||
Runner-->>Store: update_attempt("running", worker_id)
|
||||
Runner->>Agent: rollout + resources
|
||||
Agent->>Runner: reward / spans
|
||||
Runner-->>Store: add_span or add_otel_span
|
||||
Runner-->>Store: update_attempt("finished", status)
|
||||
Store-->>Algo: query_rollouts + spans
|
||||
Algo-->>Algo: Update resources (optional)
|
||||
end
|
||||
```
|
||||
|
||||
Solid lines represent direct calls, while dashed lines are asynchronous or long-running operations.
|
||||
|
||||
### Key Terminology
|
||||
|
||||
We define the following terms, which may be helpful for understanding the diagram above.
|
||||
|
||||
* **[Resources][agentlightning.Resource]:** A collection of assets to be tuned or trained. Agents perform rollouts against resources and collect span data. Algorithms use those data to update the resources. In RL training, the resources are a tunable model. In prompt tuning, the resources are prompt templates.
|
||||
* **[Rollout][agentlightning.Rollout]:** A unit of work that an agent performs against a resource. A rollout (noun) can be incomplete, in which case it is also known as a **task**, **sample**, or **job** (these terms are used interchangeably). The agent executes its own defined workflow against the rollout — the process is also called "to rollout" (verb). After execution, the rollout (noun) is considered *complete*.
|
||||
* **[Attempt][agentlightning.Attempt]:** A single execution of a rollout. One rollout can have multiple attempts in case of failures or timeouts.
|
||||
* **[Span][agentlightning.Span]:** During the rollout, the agent can generate multiple spans (also known as "traces" or "events"). The recorded spans are collected in the store, which is crucial for understanding agent behavior and optimizing agents.
|
||||
* **[Reward][agentlightning.emit_reward]:** A special span that is defined as a number judging the quality of the rollout during some period of the rollout.
|
||||
* **[Dataset][agentlightning.Dataset]:** A collection of incomplete rollouts (i.e., tasks) for the agent to process. The dual datasets (train, val) serve as the initial input for the algorithm to enqueue the first batch of rollouts.
|
||||
|
||||
### Store
|
||||
|
||||
As discussed previously, the [LightningStore][agentlightning.LightningStore] is the central hub for all data in Agent-lightning. The store exposes a set of APIs for algorithms and runners to interact with the data; the most important ones are:
|
||||
|
||||
```python
|
||||
from agentlightning.types import AttemptedRollout, ResourcesUpdate, Span, TaskInput
|
||||
|
||||
class LightningStore:
|
||||
|
||||
async def enqueue_rollout(self, input: TaskInput, ...) -> Rollout: ...
|
||||
|
||||
async def dequeue_rollout(self) -> AttemptedRollout | None: ...
|
||||
|
||||
async def add_span(self, span: Span) -> Span: ...
|
||||
|
||||
async def get_latest_resources(self) -> Optional[ResourcesUpdate]: ...
|
||||
|
||||
async def wait_for_rollouts(self, rollout_ids: List[str], ...): ...
|
||||
|
||||
async def query_spans(self, rollout_id: str, ...): ...
|
||||
|
||||
async def update_attempt(self, rollout_id: str, attempt_id: str, status: str, ...): ...
|
||||
|
||||
...
|
||||
```
|
||||
|
||||
These interfaces operate on [`AttemptedRollout`][agentlightning.AttemptedRollout], [`ResourcesUpdate`][agentlightning.ResourcesUpdate], [`Span`][agentlightning.Span], and [`TaskInput`][agentlightning.TaskInput] instances from `agentlightning.types`.
|
||||
|
||||
As the APIs show, the store essentially provides a queue for rollouts and storage for resources, spans, and attempts. Developers should implement the store carefully to ensure data integrity and consistency, especially when multiple runners work in parallel across multiple attempts.
|
||||
|
||||
The store is designed to be extensible. Users can implement their own store by inheriting from [`LightningStore`][agentlightning.LightningStore] and overriding methods. Agent-lightning provides a few reference implementations, such as [`InMemoryLightningStore`][agentlightning.InMemoryLightningStore] (default) and `SqliteLightningStore` (under construction). When parallelized, the store may need special wrappers to ensure thread/process safety or delegate computation to a store in another process or machine.
|
||||
|
||||
## Supporting Components in the Loop
|
||||
|
||||
While the core loop is simple, Agent-lightning provides several components to make development easier and more powerful.
|
||||
|
||||
### Tracer
|
||||
|
||||
The [`Tracer`][agentlightning.Tracer] is a component within the [`Runner`][agentlightning.Runner] that records detailed spans (events) during an agent's execution and sends them to the [`LightningStore`][agentlightning.LightningStore]. Instead of requiring the agent to manually log every span, the tracer automatically instruments key methods (e.g., LLM calls) and captures their inputs, outputs, and metadata. This provides a detailed log of the agent's behavior with minimal effort.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant Store
|
||||
participant Runner
|
||||
participant Tracer
|
||||
participant Agent
|
||||
|
||||
Note over Runner,Tracer: Runner manages tracer as member
|
||||
|
||||
Tracer->>Agent: Apply instrumentation
|
||||
loop Until no more rollouts
|
||||
Store-->>Runner: dequeue_rollout → AttemptedRollout
|
||||
Store-->>Runner: get_latest_resources
|
||||
Runner->>Agent: training_rollout / validation_rollout
|
||||
loop For each finished span
|
||||
Agent-->>Tracer: openai.chat.completion invoked<br>agent.execute invoked<br>...
|
||||
Agent->>Tracer: emit intermediate reward
|
||||
Tracer-->>Store: add_otel_span(rollout_id, attempt_id, span)
|
||||
end
|
||||
Agent->>Runner: final reward + extra spans (if any)
|
||||
Runner-->>Store: add_span(rollout_id, attempt_id, span)
|
||||
Runner-->>Store: update_attempt(status)
|
||||
end
|
||||
Tracer->>Agent: Unapply instrumentation
|
||||
```
|
||||
|
||||
The above diagram shows the overall data flow between store, tracer and agent. In realistic, it's a bit more complicated than that. Spans are not emitted actively by the agent; they are intercepted by the tracer by hooking and instrumenting key methods used in the agents. The tracer uses a callback (called exporter) to monitor events and log to the store. Before a rollout starts, the runner enters a [`trace_context`][agentlightning.Tracer.trace_context] before invoking the agent, wiring store identifiers into the tracer. Each span completion streams back to the store through `LightningSpanProcessor`, so the agent’s instrumentation lands in [`add_otel_span`][agentlightning.LightningStore.add_otel_span]. If the agent’s rollout method returns a numeric reward, the runner emits one more OpenTelemetry span before finalizing the attempt.
|
||||
|
||||
### Hooks
|
||||
|
||||
[`Hook`][agentlightning.Hook] implementations are user-defined callback functions that allow you to augment a [`Runner`][agentlightning.Runner]'s behavior at specific points in its lifecycle. You can use hooks to add custom logging, set up resources before a rollout begins, or tear them down after it ends. Hooks can be triggered at four key moments: `on_rollout_start`, `on_trace_start`, `on_trace_end`, and `on_rollout_end`.
|
||||
|
||||
Users should pay special attention to the difference between `on_trace_end` and `on_rollout_end`. The former is called right before the tracer exits the trace context, while the latter is called after the runner processes the final leftover rewards and spans, and finalizes the attempt in the store.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant Store
|
||||
participant Hooks
|
||||
participant Runner
|
||||
participant Tracer
|
||||
participant Agent
|
||||
|
||||
Note over Runner,Hooks: Runner manages hooks as member
|
||||
|
||||
loop Until no more rollouts
|
||||
Store-->>Runner: dequeue_rollout → AttemptedRollout
|
||||
Store-->>Runner: get_latest_resources
|
||||
|
||||
Runner->>Hooks: on_rollout_start(agent, runner, rollout)
|
||||
Runner->>Agent: training_rollout / validation_rollout
|
||||
Tracer->>Agent: enter_trace_context
|
||||
activate Tracer
|
||||
Runner->>Hooks: on_trace_start(agent, runner, tracer, rollout)
|
||||
Note over Runner,Agent: Agent rollout omitted
|
||||
Runner->>Hooks: on_trace_end(agent, runner, tracer, rollout)
|
||||
Tracer->>Agent: exit_trace_context
|
||||
deactivate Tracer
|
||||
Agent->>Runner: final reward + extra spans (if any)
|
||||
Runner-->>Store: add_span(rollout_id, attempt_id, span)
|
||||
Runner->>Hooks: on_rollout_end(agent, runner, rollout, status)
|
||||
end
|
||||
```
|
||||
|
||||
### Adapter
|
||||
|
||||
The [`Adapter`][agentlightning.Adapter] is a component used by the [`Algorithm`][agentlightning.Algorithm] to transform raw data from the [`LightningStore`][agentlightning.LightningStore] into a format suitable for learning. Runners stream raw spans into the store during execution. Later, the algorithm queries these spans and uses an adapter to convert them into structured data, like training examples for a reinforcement learning model.
|
||||
|
||||
For instance, the [`TracerTraceToTriplet`][agentlightning.TracerTraceToTriplet] processes OpenTelemetry spans to create `(prompt, response, reward)` triplets, which are the fundamental data structure for many RL fine-tuning algorithms.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Runner -- (1) add_otel_span --> Store
|
||||
Store -- (2) query_spans --> Algorithm
|
||||
Algorithm -- (3) spans --> Adapter
|
||||
Adapter -- (4) transformed data --> Algorithm
|
||||
```
|
||||
|
||||
### LLM Proxy
|
||||
|
||||
The [`LLMProxy`][agentlightning.LLMProxy] is an optional bridge component that sits between an agent and the algorithms' resources. It acts as a centralized endpoint for all LLM calls. Usually the proxy URL is added to the store as a special resource, so that the [`Runner`][agentlightning.Runner] can fetch it along with other resources when dequeuing a rollout. During rollouts, the runner invokes the proxy's HTTP endpoint instead of calling a model backend directly.
|
||||
|
||||
This design offers several benefits:
|
||||
|
||||
1. **Instrumentation:** It automatically captures detailed traces of LLM interactions (prompts, responses, metadata) and sends them to the store, complementing the tracer, especially when the agent's code is hard to instrument directly.
|
||||
2. **Backend Abstraction:** It provides a unified interface for various LLM backends (OpenAI, Anthropic, local models) and can add features like retry logic, rate limiting, and caching.
|
||||
3. **Resource Management:** The algorithm can dynamically update which LLM the agent uses (e.g., swapping to a newly fine-tuned model) by simply swapping the backend model the proxy is using, without interrupting the agent's code.
|
||||
|
||||
The benefits above seem to be all discussed within the context of model fine-tuning. As a matter of fact, the proxy can be useful for prompt tuning as well. The algorithm can register one of the following two types of endpoints into the proxy:
|
||||
|
||||
1. **Endpoint served by the algorithm:** If the algorithm is internally updating the LLM weights (e.g., RL), it can launch an LLM inference engine (i.e., a model server) and register the endpoint URL with the proxy. The proxy then forwards all LLM calls to that endpoint.
|
||||
2. **Third-party LLM endpoint:** If the algorithm is not updating the LLM weights (e.g., prompt tuning), it can register a third-party LLM endpoint into the proxy.
|
||||
|
||||
We show a diagram below that illustrates how the proxy fits into the overall data flow.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant Algo as Algorithm
|
||||
participant LLMProxy as LLM Proxy
|
||||
participant Store
|
||||
participant Runner
|
||||
participant Agent
|
||||
|
||||
Note over Algo,LLMProxy: Algorithm manages LLMProxy as member
|
||||
|
||||
loop Over the Dataset
|
||||
Algo->>Algo: Launch LLM Inference Engine<br>(optional)
|
||||
Algo->>LLMProxy: Register Inference Engine<br>(optional)
|
||||
Algo-->>Store: enqueue_rollout
|
||||
LLMProxy->>Store: Proxy URL added as Resource
|
||||
Store-->>Runner: dequeue_rollout → AttemptedRollout
|
||||
Store-->>Runner: get_latest_resources
|
||||
Runner->>Agent: rollout + resources<br>(LLM Proxy URL as resource)
|
||||
loop Defined by Agent
|
||||
Agent-->>LLMProxy: LLM calls
|
||||
activate LLMProxy
|
||||
LLMProxy-->>Store: add_span or add_otel_span
|
||||
LLMProxy-->>Agent: LLM responses
|
||||
deactivate LLMProxy
|
||||
Agent-->>Runner: rewards
|
||||
Runner-->>Store: add_span or add_otel_span
|
||||
end
|
||||
Runner-->>Store: update_attempt("finished", status)
|
||||
Store-->>Algo: query_rollouts + spans
|
||||
Algo-->>Algo: Update LLM Weights<br>(optional)
|
||||
end
|
||||
```
|
||||
|
||||
In this diagram, the store receives spans from both the proxy and the runner. We will see a problem later with parallelism where the proxy and runner are in different machines, and spans need to obtain a special counter from the store to ensure the ordering of spans.
|
||||
|
||||
### Trainer
|
||||
|
||||
The [Trainer][agentlightning.Trainer] is the high-level orchestrator that initializes and connects all major components -- [Algorithm][agentlightning.Algorithm], [Runner][agentlightning.Runner], [LightningStore][agentlightning.LightningStore], [Tracer][agentlightning.Tracer], [Adapter][agentlightning.Adapter], [LLM Proxy][agentlightning.LLMProxy], and [Hook][agentlightning.Hook]. The components can have a lifecycle as long as the trainer. The trainer manages their lifecycles and handles dependency injection, ensuring that every part of the system operates within a consistent and shared environment.
|
||||
|
||||
Below, we demonstrate how the components relate to each other and their roles. We first clarify the roles and relationships shown in the diagram:
|
||||
|
||||
1. **Owns:** components that the trainer constructs and manages directly (e.g., runner, tracer).
|
||||
2. **Injects:** components passed into others as dependencies.
|
||||
3. **References:** weak links for coordination without ownership.
|
||||
4. **Uses:** components that are temporarily interacted with.
|
||||
|
||||
For example, the [LightningStore][agentlightning.LightningStore] is injected into the [Algorithm][agentlightning.Algorithm] and [Runner][agentlightning.Runner]. The [Tracer][agentlightning.Tracer] and [LitAgent][agentlightning.LitAgent] are injected into the runner. The [Adapter][agentlightning.Adapter] and [LLM Proxy][agentlightning.LLMProxy] are injected into the algorithm. The store is further injected into the tracer, adapter and LLM proxy by the runner and algorithm respectively.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
%% === Left side: Algorithm domain ===
|
||||
subgraph L["Algorithm Side"]
|
||||
Algorithm["Algorithm<br>(no default)"]
|
||||
Adapter["Adapter<br>(TracerTraceToTriplet*)"]
|
||||
LLMProxy["LLM Proxy<br>(no default)"]
|
||||
Algorithm -.injects.-> Adapter
|
||||
Algorithm -.injects.-> LLMProxy
|
||||
end
|
||||
linkStyle 0,1 stroke:#896978,stroke-width:2px;
|
||||
|
||||
%% === Middle: Core trainer and store ===
|
||||
subgraph M["Core"]
|
||||
Trainer["Trainer"]
|
||||
Store["LightningStore<br>(InMemory* default)"]
|
||||
Trainer --has--> Algorithm
|
||||
Trainer --has--> Store
|
||||
Trainer --has--> Adapter
|
||||
Trainer --has--> LLMProxy
|
||||
end
|
||||
linkStyle 2,3,4,5 stroke:#839791,stroke-width:2px;
|
||||
|
||||
%% === Right side: Runner side ===
|
||||
subgraph R["Runner Side"]
|
||||
Runner["Runner<br>(LitAgentRunner* default)"]
|
||||
Tracer["Tracer<br>(AgentOpsTracer*)"]
|
||||
Hooks["Hooks (empty default)"]
|
||||
Agent["Agent<br>(LitAgent*)"]
|
||||
Runner -.injects.-> Tracer
|
||||
Runner -.injects.-> Store
|
||||
Runner -.injects.-> Agent
|
||||
Runner -.injects.-> Hooks
|
||||
Tracer -.injects.-> Store
|
||||
Hooks -.uses.-> Runner
|
||||
Hooks -.uses.-> Agent
|
||||
Hooks -.uses.-> Tracer
|
||||
end
|
||||
linkStyle 6,7,8,9,10 stroke:#896978,stroke-width:2px;
|
||||
linkStyle 11,12,13 stroke:#7a89c2,stroke-width:2px;
|
||||
|
||||
%% === Cross-section connections ===
|
||||
Trainer --has--> Runner
|
||||
Trainer --has--> Tracer
|
||||
Trainer --has--> Hooks
|
||||
Trainer --uses--> Agent
|
||||
Algorithm -.injects.-> Store
|
||||
LLMProxy -.injects.-> Store
|
||||
Agent -.references.-> Trainer
|
||||
Runner -.references.-> Trainer
|
||||
Algorithm -.references.-> Trainer
|
||||
linkStyle 14,15,16 stroke:#839791,stroke-width:2px;
|
||||
linkStyle 17,20,21,22 stroke:#7a89c2,stroke-width:2px;
|
||||
linkStyle 18,19 stroke:#896978,stroke-width:2px;
|
||||
|
||||
style L fill:none;
|
||||
style M fill:none;
|
||||
style R fill:none;
|
||||
```
|
||||
|
||||
## Putting It All Together: A Reinforcement Learning Example (VERL)
|
||||
|
||||
[](){ #birds-eye-view-verl-example }
|
||||
|
||||
VERL shows how an algorithm consumes the shared infrastructure. For historical reasons, code lives in `agentlightning.algorithm.verl` and `agentlightning.verl`. The latter is legacy and reuses terms like `Trainer` in confusing ways. The former is a thin wrapper that conforms to the new algorithm interface. Future versions will merge the two.
|
||||
|
||||
Reinforcement learning aims to learn a policy that takes actions in states to maximize expected reward. For agents, the policy is usually a language model. Inputs are prompts (state). Outputs are generated text (action). A numeric score judges quality (reward). The `(state, action, reward)` **triplet** is the basic learning unit.
|
||||
|
||||
In Agent-lightning, the environment is implicit in the agent’s workflow, which orchestrates one or more LLM calls and often self-judges using rules or additional model calls. During a rollout, the agent emits spans that contain everything needed for RL training, including LLM call traces and numeric judge/reward signals. The "algorithm", on the other hand, have more responsibilities.
|
||||
|
||||
1. Providing a language model deployment that is currently learning and improving for the agent to interact with;
|
||||
2. Preparing the tasks that the agents will perform;
|
||||
3. Querying the spans generated, extracting triplets, and converting them into a format that the underlying RL library can consume;
|
||||
4. Updating the language model based on the learning signals.
|
||||
|
||||
In the VERL integration, the algorithm launches a chat completion endpoint using `vLLM` and wraps training with `FSDP` for distributed optimization. It enqueues tasks from the dataset. After rollouts finish, it queries spans and converts them to triplets with `TracerTraceToTriplet`. VERL’s native training loop then consumes these triplets to update model weights. The workflow can be summarized in the following diagram.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant vLLM as vLLM Chat<br>Completion Endpoint
|
||||
participant FSDP as FSDP / Megatron<br>Weights Optimizer
|
||||
participant Algo as Algorithm<br>Main Controller<br>(Main Process)
|
||||
participant Adapter as TracerTraceToTriplet
|
||||
participant LLMProxy as LLM Proxy
|
||||
participant Store as LightningStore
|
||||
participant Runner as Runner + Agent
|
||||
|
||||
Note over Algo,LLMProxy: LLMProxy and Adapter are injected by Trainer as member
|
||||
Note over vLLM,Algo: Algorithm creates and owns vLLM and FSDP
|
||||
|
||||
loop Over the Dataset in Batches
|
||||
Algo->>vLLM: Create Chat Completion Endpoint
|
||||
activate vLLM
|
||||
vLLM->>LLMProxy: Registered as Backend Endpoint
|
||||
LLMProxy->>Store: Proxy URL added as Resource
|
||||
par Over data samples in the batch
|
||||
Algo-->>Store: enqueue_rollout
|
||||
Store-->>Runner: Dequeue Rollout +<br>Resources (i.e., URL)
|
||||
loop One Rollout Attempt
|
||||
Runner-->>LLMProxy: LLM calls
|
||||
LLMProxy-->>vLLM: Forwarded LLM calls
|
||||
vLLM-->>LLMProxy: LLM responses
|
||||
LLMProxy-->>Store: add_span / add_otel_span
|
||||
LLMProxy-->>Runner: Forwarded LLM responses
|
||||
Runner-->>Store: add_span / add_otel_span <br> (by tracer, including rewards)
|
||||
end
|
||||
Runner-->>Store: update_attempt("finished", status)
|
||||
end
|
||||
Algo-->>Store: Poll for completed rollouts + spans
|
||||
Algo->>vLLM: Chat Completion Endpoint Sleeps
|
||||
deactivate vLLM
|
||||
Algo->>Adapter: adapt(spans)
|
||||
Adapter->>FSDP: Triplets (state, action, reward)
|
||||
activate FSDP
|
||||
FSDP-->>Algo: Updated LLM weights
|
||||
deactivate FSDP
|
||||
end
|
||||
```
|
||||
|
||||
**Notes:**
|
||||
|
||||
1. There are interactions between different components injected into or owned by algorithms in the diagram, such as the output of the adapter feeding into the FSDP optimizer. This is for simplicity of illustration and slightly different from the actual implementation, where it's the algorithm main controller that orchestrates the data flow between components.
|
||||
|
||||
2. **On mapping to VERL.** VERL uses a classic RLHF setup where each action is a single token, the state is the full conversation history up to that token, and reward is given at the end. This is very different from our setup where each action is actually a chunk of text, although they are both called RL! Therefore, after the adapter produces triplets, the algorithm converts each `(state, action, reward)` into a VERL trajectory (`DataProto`) with keys like `input_ids`, `position_ids`, `attention_mask`, and `token_level_scores`. That conversion happens after triplet generation and is not shown in the diagram.
|
||||
|
||||
## Execution Strategies and Parallelism
|
||||
|
||||
Readers might have observed from the diagram above that there is absolutely no communication between (1) runner and agents and (2) algorithm. The only overlap of them is the [Trainer][agentlightning.Trainer] and [LightningStore][agentlightning.LightningStore]. This observation is very clear with the diagram within the trainer section. This design allows us to flexibly scale the runner and algorithm independently, which is crucial for large-scale training.
|
||||
|
||||
Agent-lightning packages two executable bundles: a runner bundle ([Runner][agentlightning.Runner], [Tracer][agentlightning.Tracer], [Hook][agentlightning.Hook], [LitAgent][agentlightning.LitAgent]) and an algorithm bundle ([Algorithm][agentlightning.Algorithm], [Adapter][agentlightning.Adapter], [LLM Proxy][agentlightning.LLMProxy]). Both share the [LightningStore][agentlightning.LightningStore]. The trainer initializes and connects the bundles.
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph Runner_Side["Runner Bundle"]
|
||||
direction LR
|
||||
R[Runner] --- T[Tracer] --- H[Hooks] --- A1[Agent]
|
||||
end
|
||||
|
||||
subgraph Algorithm_Side["Algorithm Bundle"]
|
||||
direction LR
|
||||
ALG[Algorithm] --- AD[Adapter] --- LLM[LLM Proxy]
|
||||
end
|
||||
|
||||
S[(Store)]
|
||||
TR[Trainer]
|
||||
|
||||
Runner_Side <--> S
|
||||
Algorithm_Side <--> S
|
||||
TR --> Runner_Side
|
||||
TR --> Algorithm_Side
|
||||
|
||||
linkStyle 0,1,2,3,4 opacity:0;
|
||||
```
|
||||
|
||||
An [execution strategy][agentlightning.ExecutionStrategy], defined and owned by the trainer, governs how algorithm and runner bundles are placed, connected, scaled, and aborted. It serves four primary purposes.
|
||||
|
||||
Execution strategies first determine **bundle placement** — whether the two bundles run in the same thread, process, machine, or across separate machines. They also define **store management**, wrapping the store and specifying how data is shared between bundles.
|
||||
|
||||
In terms of **scalability**, the strategy can replicate the runner bundle across multiple threads, processes, or machines to expand throughput on the runner side. The algorithm side remains single-process due to the complexity of parallelization. Mature frameworks such as *DeepSpeed* and *Megatron* already support distributed model training, so scaling of the algorithm bundle is delegated to those implementations.
|
||||
|
||||
**Abort handling** is another core responsibility. Aborts may be triggered by normal exits, failures in either bundle, or user interrupts. The trainer must include cancellation interfaces for the bundles so that bundles can be cleanly aborted. When the algorithm bundle exits normally, the strategy signals the runner bundle to terminate. If the runner exits first, no signal is sent to the algorithm, as it may still be processing completed rollouts. In cases of failure or user interruption, the strategy signals both bundles to abort; if a bundle fails to respond, the strategy should attempt a forceful termination.
|
||||
|
||||
Agent-lightning currently provides two execution strategies: **shared-memory** and **client-server**, described in the following sections.
|
||||
|
||||
### Shared-memory Strategy
|
||||
|
||||
[`SharedMemoryExecutionStrategy`][agentlightning.SharedMemoryExecutionStrategy] runs algorithm and runner bundles as threads in one process. The strategy wraps the store with [`LightningStoreThreaded`][agentlightning.LightningStoreThreaded], which guards calls with a lock for safe concurrency.
|
||||
|
||||
This is good for lightweight debugging because components share one Python heap and avoid serialization. It is not suitable for heavy RL training or compute-intensive agents.
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph MainProcess
|
||||
direction TB
|
||||
subgraph AlgorithmThread [Thread 0]
|
||||
Algorithm[Algorithm bundle]
|
||||
end
|
||||
subgraph RunnerThread1 [Thread 1]
|
||||
Runner1[Runner bundle #1]
|
||||
end
|
||||
subgraph RunnerThread2 [Thread 2]
|
||||
Runner2[Runner bundle #2]
|
||||
end
|
||||
subgraph RunnerThread3 [Thread 3]
|
||||
RunnerN[Runner bundle #N]
|
||||
end
|
||||
LightningStoreFacade[LightningStoreThreaded]
|
||||
BaseStore[Underlying LightningStore]
|
||||
end
|
||||
Algorithm -- async calls --> LightningStoreFacade
|
||||
Runner1 -- async calls --> LightningStoreFacade
|
||||
Runner2 -- async calls --> LightningStoreFacade
|
||||
RunnerN -- async calls --> LightningStoreFacade
|
||||
LightningStoreFacade -->|thread-safe delegates| BaseStore
|
||||
```
|
||||
|
||||
You can configure which role runs on the main thread. If the main thread runs the algorithm, it is able to spawn multiple runner threads. If it runs a runner, `n_runners` must be 1 and the runner lives on the main thread.
|
||||
|
||||
### Client-server Strategy
|
||||
|
||||
[](){ #birds-eye-view-client-server-strategy }
|
||||
|
||||
[`ClientServerExecutionStrategy`][agentlightning.ClientServerExecutionStrategy] splits concerns across processes. The algorithm bundle starts a [`LightningStoreServer`][agentlightning.LightningStoreServer] (HTTP API) that wraps the underlying store. Runners connect via [`LightningStoreClient`][agentlightning.LightningStoreClient] to call the same interface over REST. The server embeds a client to support algorithm-launched subprocesses (e.g., an LLM proxy worker) that need to talk back to the algorithm’s process through the same API.
|
||||
|
||||
Currently this design introduces an extra wrapper in the Server side (as shown in the diagram), which helps debugging and improves fault tolerance. We might revisit this design in the future and enforce the client to be the only way to communicate with the store.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph Algorithm Process Group
|
||||
subgraph StoreServer[LightningStoreServer]
|
||||
StoreHttpClient[HTTP Client]
|
||||
StoreHttpServer[HTTP Server]
|
||||
StoreWrapper[LightningStore Wrapper]
|
||||
StoreHttpClient -- HTTP --> StoreHttpServer
|
||||
end
|
||||
subgraph Algorithm Bundle
|
||||
Algorithm[Algorithm Main Process]
|
||||
subgraph Another subprocess
|
||||
LLMProxy[LLM Proxy]
|
||||
end
|
||||
end
|
||||
LLMProxy -- async calls --> StoreHttpClient
|
||||
Algorithm -- async calls --> StoreWrapper
|
||||
end
|
||||
subgraph RunnerSide ["Runner Side"]
|
||||
subgraph Runner Process 1
|
||||
Runner1[Runner bundle #1]
|
||||
Runner1 -- async calls --> LightningStoreClient1
|
||||
LightningStoreClient1[LightningStoreClient]
|
||||
end
|
||||
subgraph Runner Process 2
|
||||
Runner2[Runner bundle #2]
|
||||
Runner2 -- async calls --> LightningStoreClient2
|
||||
LightningStoreClient2[LightningStoreClient]
|
||||
end
|
||||
subgraph Runner Process N
|
||||
RunnerN[Runner bundle #N]
|
||||
RunnerN -- async calls --> LightningStoreClientN
|
||||
LightningStoreClientN[LightningStoreClient]
|
||||
end
|
||||
end
|
||||
LocalStore[Underlying LightningStore]
|
||||
StoreHttpServer -->|delegates| StoreWrapper
|
||||
StoreWrapper -->|delegates| LocalStore
|
||||
LightningStoreClient1 -- HTTP --> StoreHttpServer
|
||||
LightningStoreClient2 -- HTTP --> StoreHttpServer
|
||||
LightningStoreClientN -- HTTP --> StoreHttpServer
|
||||
|
||||
style RunnerSide fill:none;
|
||||
```
|
||||
|
||||
## Online/Continuous Learning
|
||||
|
||||
Continuous learning keeps the algorithm loop running while runners report tasks and spans opportunistically. Key differences from batch mode:
|
||||
|
||||
1. The algorithm does not enqueue rollouts from a fixed dataset. Runners report tasks/rollouts and spans spontaneously.
|
||||
2. The algorithm can wait for rollouts with a expected set of rollout IDs, but more often polls for new rollouts and spans or waits for a count to arrive.
|
||||
3. The [`Runner`][agentlightning.Runner] processes one rollout at a time via [`step(task)`][agentlightning.Runner.step] instead of exhausting a task queue. It notifies the store when starting a rollout so the store records it.
|
||||
4. A user or higher-level loop controls which resources the next step uses and when to retry.
|
||||
|
||||
[Spans][agentlightning.Span], [Adapter][agentlightning.Adapter] implementations, and the [LLM Proxy][agentlightning.LLMProxy] work the same way.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
actor User
|
||||
participant Runner
|
||||
participant Agent
|
||||
participant Store as LightningStore
|
||||
participant Algorithm
|
||||
|
||||
Note over Algorithm: Algorithm is long-running and loops continuously
|
||||
|
||||
loop Continuous Learning Loop
|
||||
activate User
|
||||
opt Decide what to do next
|
||||
User-->>Store: get_resources_by_id
|
||||
Store-->>User: Resources
|
||||
User-->>User: Prepare input for next step
|
||||
end
|
||||
User->>Runner: step(input, resources)
|
||||
activate Runner
|
||||
Runner-->>Store: Notify: start_rollout(input)
|
||||
Runner->>Agent: rollout(input, resources)
|
||||
Agent-->>Runner: add_span / reward spans
|
||||
Runner-->>Store: add_span or add_otel_span
|
||||
Runner-->>Store: update_attempt(status="finished")
|
||||
deactivate Runner
|
||||
deactivate User
|
||||
Algorithm->>Store: poll for new rollouts and spans
|
||||
opt If there is enough new data
|
||||
Store-->>Algorithm: new spans
|
||||
Algorithm->>Algorithm: adapt spans → learning signal
|
||||
Algorithm->>Store: update_resources
|
||||
end
|
||||
end
|
||||
```
|
||||
@@ -0,0 +1,135 @@
|
||||
# Serving LLMs under Agent-lightning
|
||||
|
||||
Agent-lightning focuses on data, learning signals, and control flow — **not** on running model inference. This deep dive explains how to **serve** a model alongside Agent-lightning so runners can call it reliably, how the **LLM Proxy** fits into the loop, and why **token IDs** matter if you care about correctness in training and evaluation.
|
||||
|
||||
## General background on LLM serving
|
||||
|
||||
[](){ #general-llm-serving-background }
|
||||
|
||||
Serving a model is essential if you want to train it, especially when you use the model’s own generations as training data. We’ll briefly review the general background to ensure all readers are aligned.
|
||||
|
||||
Modern LLM servers solve a difficult scheduling problem: keeping GPUs fully utilized while handling prompts of different lengths, streaming tokens as they arrive, and fitting large KV caches into limited memory. Techniques like [**continuous batching**](https://www.anyscale.com/blog/continuous-batching-llm-inference) and [**paged attention**](https://arxiv.org/abs/2309.06180) address these challenges. Continuous batching interleaves decoding across requests to reuse weights efficiently; with careful memory planning, it achieves major throughput gains without increasing latency. PagedAttention reduces KV-cache fragmentation so batching remains effective as sequences grow. See [vLLM’s PagedAttention paper](https://arxiv.org/abs/2309.06180) and [industry analyses](https://www.baseten.co/blog/continuous-vs-dynamic-batching-for-ai-inference/) for details. Balancing inference correctness and efficiency is difficult — a [recent blog](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/) from Thinking Machines Labs highlights how inference nondeterminism ultimately affects training.
|
||||
|
||||
Beyond scheduling, servers expose an HTTP API, often **OpenAI-compatible** (`/v1/chat/completions` and `/v1/responses`), which is itself a complex stack. In addition to text prompts and chat messages, the API defines many parameters and response fields such as [tool calls](https://platform.openai.com/docs/guides/function-calling), [structured output](https://platform.openai.com/docs/guides/structured-outputs), and [multimodal support](https://platform.openai.com/docs/guides/images-vision). Much effort has been put into implementing all these parameters for many frameworks. Popular engines like **vLLM** and [**SGLang**](https://github.com/sgl-project/sglang) ship with OpenAI-compatible frontends so you can reuse existing client code. [Ollama](https://ollama.com/blog/openai-compatibility) and [llama.cpp](https://llama-cpp-python.readthedocs.io/en/latest/server/) provide similar capabilities. However, because models differ internally, each framework interprets and implements the API slightly differently. Even with identical requests, the tokens passed to the model can vary substantially across frameworks.
|
||||
|
||||
## What Agent-lightning expects from a served LLM
|
||||
|
||||
Most of the issues above either have workarounds or remain open research problems. Keep them in mind, but the key question is: what does Agent-lightning expect from a served LLM? The answer includes at least two things:
|
||||
|
||||
* An OpenAI-compatible **Chat Completions** or **Responses** endpoint the agent can call during rollouts.
|
||||
* Optional training and debugging signals: **logprobs**, **usage**, and ideally **token IDs**. (OpenAI’s public API exposes usage and logprobs, but **not** token IDs — more on [why IDs matter][token-ids-matter] later.)
|
||||
|
||||
## Launching a serving framework
|
||||
|
||||
For many algorithms, you’ll start an engine (e.g., **vLLM** or **SGLang**) before rollouts, then shut it down afterward to free GPU memory. Most frameworks provide a one-line “serve” command to launch the OpenAI-compatible server. You can use those to bring up `/v1/chat/completions` with your checkpoint, ensuring streaming and any required tool-calling features are enabled. A working example is shown in [Unsloth SFT](../how-to/unsloth-sft.md).
|
||||
|
||||
Weight updates — which occur after each training step — are trickier. Some frameworks like [vLLM](https://vllm.ai/) support hot-updating model weights, but it’s usually simpler and more reliable to restart the engine to load new weights. For medium-sized tasks (hundreds of rollouts taking 10+ minutes), the restart overhead (under 30 seconds) is typically negligible.
|
||||
|
||||
If you’re using Agent-lightning’s [**VERL**][agentlightning.algorithm.verl.VERL] integration, the algorithm can **manage the server automatically**. The [VERL framework](https://github.com/volcengine/verl) intelligently allocates compute resources and wraps vLLM/SGLang behind an `AsyncLLMServer` abstraction. You can directly use this as the LLM endpoint for agents. Since VERL can spawn multiple vLLM replicas, using [`LLMProxy`][agentlightning.LLMProxy] to manage them adds an additional safety layer.
|
||||
|
||||
A full sequence diagram of how [VERL][agentlightning.algorithm.verl.VERL] interacts with the LLM server and proxy is available [here][birds-eye-view-verl-example].
|
||||
|
||||
## LLM Proxy
|
||||
|
||||
The **LLM Proxy** is a utility class in Agent-lightning, built on [LiteLLM](https://docs.litellm.ai/), that sits between runners and your backend engine(s) or server(s). In Agent-lightning it acts as a single URL registered as a [`Resource`][agentlightning.Resource] in the store, offering three key benefits:
|
||||
|
||||
1. **Unified endpoint & hot-swaps.** You can redirect traffic between OpenAI, Anthropic, local vLLM/SGLang, or canary checkpoints without modifying agent code — simply repoint the proxy.
|
||||
2. **First-class tracing.** The proxy emits **OpenTelemetry** spans for every call and sends them to the [`LightningStore`][agentlightning.LightningStore]. It includes rollout and attempt identifiers in request headers so spans are correctly attributed. Sequence numbers are allocated monotonically via the store to [prevent clock-skew issues][distributed-tracing] and allow reliable reconstruction of execution trees.
|
||||
3. **Token IDs.** The proxy can return prompt and response token IDs along with the model output. More details are available in the [next section][token-ids-matter].
|
||||
|
||||
Operationally, running the proxy alongside the algorithm works best: the algorithm registers the backend (e.g., the vLLM URL) via [`LLMProxy.update_model_list`][agentlightning.LLMProxy.update_model_list], publishes the proxy URL as a resource via [`LightningStore.add_resources`][agentlightning.LightningStore.add_resources], and runners simply use that URL during rollouts. This mirrors many production client–server setups.
|
||||
|
||||
## Token IDs and why they matter
|
||||
|
||||
[](){ #token-ids-matter }
|
||||
|
||||
This section explains how Agent-lightning handles and uses token IDs — a subtle but important detail for training stability and accuracy.
|
||||
|
||||
Most agents interact with LLMs via **Chat Completion APIs**, exchanging chat messages. There are two main approaches to collecting training data from such agents.
|
||||
|
||||
!!! note
|
||||
|
||||
Tokenization here refers to the process of converting **Chat Messages** into **Token IDs**. Detokenization is the reverse process of converting **Token IDs** back to **Chat Messages**. Normally, the tokenizer is published along with the pretrained model, which includes a vocabulary, special tokens, and a chat template to dealing with chat messages.
|
||||
|
||||
**1. Retokenizing chat messages.**
|
||||
In this approach, you store chat messages as text and let training algorithms **retokenize** them later, as done in many SFT workflows (e.g., [HuggingFace SFT](https://huggingface.co/docs/trl/sft_trainer)).
|
||||
In practice, we’ve found this method unstable and less accurate. The chart below compares training results. The retokenization approach is run twice. All settings are the same except for the retokenization approach.
|
||||
|
||||
<div style="height:400px">
|
||||
<canvas data-chart='{
|
||||
"type": "line",
|
||||
"data": {
|
||||
"labels": [0.0, 32.0, 64.0, 96.0, 128.0, 160.0, 192.0, 224.0, 256.0, 288.0, 320.0, 352.0, 384.0, 416.0, 448.0, 480.0],
|
||||
"datasets": [
|
||||
{
|
||||
"label": "With Token IDs from Retokenization",
|
||||
"data": [0.49, 0.512, 0.54, 0.532, 0.54, 0.466, 0.328, 0.358, 0.348, 0.35, 0.346, 0.372, 0.346, 0.33, 0.346, 0.332],
|
||||
"spanGaps": true
|
||||
},
|
||||
{
|
||||
"label": "Retokenization (Second Run)",
|
||||
"data": [0.494, 0.526, 0.536, 0.554, 0.544, 0.556, 0.568, 0.552, 0.45, 0.466, 0.474, 0.47, 0.464, 0.476, 0.488, 0.432],
|
||||
"spanGaps": true
|
||||
},
|
||||
{
|
||||
"label": "With Token IDs from Engine",
|
||||
"data": [0.494, 0.522, 0.514, 0.538, 0.53, 0.564, 0.564, 0.586, 0.594, 0.604, 0.618, 0.584, 0.606, 0.558, 0.612, 0.588],
|
||||
"spanGaps": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"options": {
|
||||
"interaction": {
|
||||
"mode": "nearest",
|
||||
"intersect": false
|
||||
},
|
||||
"plugins": {
|
||||
"legend": {
|
||||
"display": true,
|
||||
"position": "top"
|
||||
},
|
||||
"title": {
|
||||
"display": true,
|
||||
"text": "Agent Training Results Comparison"
|
||||
}
|
||||
},
|
||||
"scales": {
|
||||
"x": {
|
||||
"title": {
|
||||
"display": true,
|
||||
"text": "Step"
|
||||
}
|
||||
},
|
||||
"y": {
|
||||
"title": {
|
||||
"display": true,
|
||||
"text": "Reward"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}'></canvas>
|
||||
</div>
|
||||
|
||||
This instability has three causes. Firstly, chat template used in different frameworks could be slightly different. For example, one single LLaMA model can work with multiple chat templates (multiple in [vLLM](https://github.com/vllm-project/vllm/tree/1d165d6d859d3c50720f0c07209db2363c4fd33b/examples) and one in [HuggingFace](https://huggingface.co/meta-llama)). It's possible that the chat template used in detokenization is different from the one used in tokenization (this is actually an implementation bug).
|
||||
|
||||
Secondly, a word might be generated as two tokens (e.g., `H + AVING`) but later retokenized as `HAV + ING`. The text looks identical, but the token IDs differ from what the model originally produced.
|
||||
|
||||
Thirdly, a generated tool call text like `<tool_call>{ "name": ... }</tool_call>` is parsed by tool call parser into an object that is required by chat completion API. Later, the object is rendered back to `<tool_call>{ "name": ... }</tool_call>` and retokenized again, tool call parsing and re-rendering might cause changes in whitespace and formatting. In some situations, JSON errors may even be auto-corrected by the tool call parser — masking the model’s true generation errors and preventing them from being trained away.
|
||||
|
||||
----
|
||||
|
||||
**2. Saving token IDs directly.**
|
||||
The alternative is to save the token IDs generated by the model, as done in RL setups like [Tinker](https://thinkingmachines.ai/tinker/). This requires a training pipeline that treats tokens as first-class entities, meaning agents must communicate with the inference engine at the token level.
|
||||
|
||||
However, most agents — especially those built with frameworks like LangChain — rely on OpenAI-compatible APIs and can’t tokenize or detokenize themselves. As mentioned [earlier][general-llm-serving-background], implementing this layer manually is complex and error-prone. Some frameworks implement custom solutions (e.g., [VERL Agent Loop](https://github.com/volcengine/verl/blob/4da0d3d3188072772cb2ec817b3d6cf4a463821f/recipe/langgraph_agent/chat_model.py), [Tinker Renderer](https://github.com/thinking-machines-lab/tinker-cookbook/blob/34a6588d7055040c259985d98e71c0140b389ba7/tinker_cookbook/renderers.py)), while others leave it to users (e.g., [SkyRL Search-R1](https://novasky-ai.notion.site/skyrl-searchr1)).
|
||||
|
||||
----
|
||||
|
||||
A better solution is to use an **OpenAI-compatible API that returns token IDs directly.** This lets agents continue using familiar APIs while capturing token IDs via [tracing](../tutorials/traces.md) for training. The limitation, of course, is that the serving framework must actually support this capability.
|
||||
|
||||
When Agent-lightning was first released, we implemented an [instrumented vLLM server](https://github.com/microsoft/agent-lightning/blob/v0.1/agentlightning/instrumentation/vllm.py) that monkey-patched vLLM’s OpenAI server to return token IDs. Since then, the Agent-lightning and vLLM teams have collaborated to add this feature directly to [vLLM core](https://github.com/vllm-project/vllm/pull/22587). Starting with **vLLM v0.10.2**, the OpenAI-compatible API includes a [`return_token_ids` parameter](https://docs.vllm.ai/en/v0.10.2/serving/openai_compatible_server.html#api-reference), allowing token IDs to be requested alongside chat messages. SGLang has tracked [similar feature requests](https://github.com/sgl-project/sglang/issues/2634), though its OpenAI-compatible layer doesn’t yet support them.
|
||||
|
||||
In short, when using vLLM v0.10.2 or newer, [`LLMProxy`][agentlightning.LLMProxy] automatically adds `return_token_ids` to each request so the engine includes token IDs in its response. For older vLLM versions, you still need the instrumented version (via `agl vllm` CLI command).
|
||||
|
||||
Finally, if you only save token IDs in spans, it will have its own limitations — if you train one model using spans from another model with a different tokenizer, incompatibilities can arise. In practice, though, spans in Agent-lightning always store both chat messages and token IDs (actually the full request and response objects), allowing you to fall back to retokenization when necessary.
|
||||
@@ -0,0 +1,410 @@
|
||||
# Understanding Store
|
||||
|
||||
The **[`LightningStore`][agentlightning.LightningStore]** is the central coordination point for Agent-lightning. It holds the task queue, rollouts, attempts, spans, and versioned resources, and exposes a small API both Runners and Algorithms use to communicate. This document explains what's in the store, how statuses transition, how spans are recorded, and the concurrency model (threads & processes).
|
||||
|
||||
## What's in the Store?
|
||||
|
||||
{ .center }
|
||||
|
||||
At a high level:
|
||||
|
||||
* **Task Queue** – [`enqueue_rollout`][agentlightning.LightningStore.enqueue_rollout] adds work; workers poll with [`dequeue_rollout`][agentlightning.LightningStore.dequeue_rollout]. When a rollout is dequeued, it automatically creates a new attempt associated with itself.
|
||||
* **Rollouts** – A rollout is one unit of work. It has input, metadata, links to resources, and a lifecycle (`queuing → preparing → running → ...`). Valid [RolloutStatus][agentlightning.RolloutStatus] are **`queuing`**, `preparing`, `running`, `succeeded`, `failed`, **`requeuing`**, **`cancelled`**. For algorithms and runners, the rollout can be seen as a whole, without worrying about the internal attempts.
|
||||
* **Attempts** – Each rollout can have multiple executions (retries). Attempts track [`status`][agentlightning.Attempt.status], [`start_time`][agentlightning.Attempt.start_time], [`end_time`][agentlightning.Attempt.end_time], [`last_heartbeat_time`][agentlightning.Attempt.last_heartbeat_time] and link to spans. Valid [AttemptStatus][agentlightning.AttemptStatus] are `preparing`, `running`, `succeeded`, `failed`, `requeuing`, `cancelled`.
|
||||
* **Spans** – Structured trace events produced by the Tracer during an attempt. Spans are ordered by a **monotonic sequence id** per `(rollout_id, attempt_id)`.
|
||||
* **Resources** – Versioned, named bundles (e.g., prompt templates) referenced by rollouts.
|
||||
* **Workers** – Metadata about runner instances: heartbeat timestamps, current assignment, and status.
|
||||
|
||||
Rollout and Task share the same surface in practice: [`Rollout.input`][agentlightning.types.Rollout] is the task input. The queue stores rollouts that are not yet running; [Runners][agentlightning.Runner] dequeue them and update the same rollout's status as work progresses.
|
||||
|
||||
Before we look at status transitions, it helps to keep in mind that rollouts are the "outside view," while attempts are the "inside view." Attempts are what actually run; rollouts summarize the latest attempt plus a small set of control actions like queueing and cancellation.
|
||||
|
||||
## Attempt Status Transitions
|
||||
|
||||
The status model is intentionally small and operationally clear.
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
direction LR
|
||||
|
||||
[*] --> preparing: <b>Runner calls</b> dequeue_rollout()<br>or start_rollout()<br>or start_attempt()
|
||||
preparing --> running: <b>Runner calls</b><br>add_[otel_]span()<br>for the first time
|
||||
|
||||
state c_runner <<choice>>
|
||||
state c_watch <<choice>>
|
||||
|
||||
preparing --> c_runner: <b>Runner calls</b><br>update_attempt(...)</b>
|
||||
running --> c_runner: <b>Runner calls</b><br>update_attempt(...)
|
||||
running --> c_watch: <b>Watchdog checks</b>
|
||||
preparing --> c_watch: <b>Watchdog checks</b>
|
||||
|
||||
state "Client-set outcome" as Client {
|
||||
direction TB
|
||||
succeeded
|
||||
failed
|
||||
}
|
||||
state "Watchdog / policy" as Watch {
|
||||
direction TB
|
||||
timeout
|
||||
unresponsive
|
||||
}
|
||||
|
||||
c_runner --> succeeded: update_attempt(status=succeeded)
|
||||
c_runner --> failed: update_attempt(status=failed)
|
||||
|
||||
c_watch --> timeout: now - start_time > timeout_seconds
|
||||
c_watch --> unresponsive: now - last_heartbeat > unresponsive_seconds
|
||||
|
||||
unresponsive --> running: <b>Runner calls</b><br>add_[otel_]span()
|
||||
```
|
||||
|
||||
Each attempt begins in **preparing**, created either when a rollout is dequeued or explicitly started. It transitions to **running** the first time a span is recorded. From there, a few clear rules govern how it can change:
|
||||
|
||||
* When the runner explicitly marks completion, the attempt becomes **succeeded** or **failed** (when the runner catches exception thrown out by the agent).
|
||||
* When the watchdog detects that the total elapsed time since start exceeds the configured limit, it marks the attempt as **timeout**.
|
||||
* If heartbeats stop arriving for too long, the watchdog marks it **unresponsive**.
|
||||
* A new span from the runner can immediately revive an **unresponsive** attempt back to **running**.
|
||||
|
||||
!!! info "What's a Watchdog?"
|
||||
|
||||
The watchdog enforces timing and liveness rules defined by each rollout’s [`RolloutConfig`][agentlightning.RolloutConfig]. It’s not a separate thread or service, but a function periodically invoked (e.g., before store mutations) to keep attempts healthy and consistent.
|
||||
|
||||
This simple model allows the system to distinguish between normal termination, abnormal stalling, and recoverable interruption without additional state flags.
|
||||
|
||||
## Worker Telemetry
|
||||
|
||||
Workers track runner-level activity timestamps (`last_heartbeat_time`, `last_dequeue_time`, `last_busy_time`, `last_idle_time`) plus their current rollout assignment. Those fields are now derived automatically:
|
||||
|
||||
- [`dequeue_rollout(worker_id=...)`][agentlightning.LightningStore.dequeue_rollout] records which worker polled the queue and refreshes `last_dequeue_time`.
|
||||
- [`update_attempt(..., worker_id=...)`][agentlightning.LightningStore.update_attempt] drives the worker status machine. Assigning an attempt marks the worker **busy** and stamps `last_busy_time`; finishing with `status in {"succeeded","failed"}` switches to **idle**, while watchdog transitions such as `timeout`/`unresponsive` make the worker **unknown** and clear `current_rollout_id` / `current_attempt_id`.
|
||||
- [`update_worker(...)`][agentlightning.LightningStore.update_worker] is reserved for heartbeats. It snapshots optional `heartbeat_stats` and always updates `last_heartbeat_time`.
|
||||
|
||||
Because every transition flows through these APIs, worker status is derived automatically from rollout execution and heartbeats. Note, however, that calling `update_worker` with a new `worker_id` will create a new worker record with status "unknown" if one does not exist. Thus, while manual status changes are not allowed, new worker records can be created externally via heartbeats.
|
||||
|
||||
## Rollout Transition Map
|
||||
|
||||
Rollout status is an **aggregated view** of its latest attempt’s status, with additional transitions for queueing and explicit cancellation.
|
||||
|
||||
A rollout’s retry behavior is controlled by [`Rollout.config`][agentlightning.types.Rollout] (a [`RolloutConfig`][agentlightning.types.RolloutConfig]). The key fields are:
|
||||
|
||||
* [`timeout_seconds`][agentlightning.RolloutConfig.timeout_seconds] – maximum wall-clock time for an attempt before it is marked `timeout`.
|
||||
* [`unresponsive_seconds`][agentlightning.RolloutConfig.unresponsive_seconds] – maximum silence between heartbeats before an attempt is marked `unresponsive`.
|
||||
* [`max_attempts`][agentlightning.RolloutConfig.max_attempts] – total number of attempts allowed for the rollout (including the first).
|
||||
* [`retry_condition`][agentlightning.RolloutConfig.retry_condition] – which attempt terminal statuses should trigger a retry (e.g., `["failed", "timeout", "unresponsive"]`).
|
||||
|
||||
**How it plays out:** The runner works on attempt `k`. If the attempt ends in a status that is listed in `retry_condition`, and `k < max_attempts`, the rollout moves to **requeuing** and the store creates attempt `k+1`. Otherwise, the rollout becomes **failed** (or **succeeded** if the runner marked it so). `timeout_seconds` and `unresponsive_seconds` are enforced by the watchdog and feed into the same decision flow.
|
||||
|
||||
A minimal example of how to use `RolloutConfig`:
|
||||
|
||||
```python
|
||||
from agentlightning import RolloutConfig
|
||||
|
||||
# Retry on explicit failures or timeouts, up to 3 attempts in total.
|
||||
cfg = RolloutConfig(
|
||||
timeout_seconds=600,
|
||||
unresponsive_seconds=120,
|
||||
max_attempts=3,
|
||||
retry_condition=["failed", "timeout"]
|
||||
)
|
||||
|
||||
# When creating/enqueuing a rollout, attach this config.
|
||||
# The store will propagate attempt outcomes according to cfg.
|
||||
rollout = await store.enqueue_rollout(input, config=cfg)
|
||||
```
|
||||
|
||||
| Latest attempt status | Rollout transition | Notes / guards |
|
||||
| ------------------------------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------- |
|
||||
| N/A | `queuing` | Created by `enqueue_rollout()`. |
|
||||
| `preparing` | `queuing/requeuing` → `preparing` | Typically `dequeue_rollout()` or `start_rollout()`/`start_attempt()` creates a new attempt. |
|
||||
| `running` | `preparing/queuing/requeuing` → `running` | First `add_[otel_]span()` flips the attempt to `running`; rollout follows via `rollout_status_from_attempt`. |
|
||||
| `succeeded` | `*` → `succeeded` | Terminal. Rollout `end_time` set. |
|
||||
| `failed` / `timeout` / `unresponsive` | `*` → `requeuing` | **Only if** `status ∈ retry_condition ∧ sequence_id < max_attempts`. |
|
||||
| `failed` / `timeout` / `unresponsive` | `*` → `failed` | Otherwise (no retries left or retries disabled). |
|
||||
| `*` | `*` → `cancelled` | Explicitly set by `update_rollout(status=cancelled)`. |
|
||||
|
||||
!!! note "Why aggregation?"
|
||||
|
||||
In code, we use `rollout_status_from_attempt()` which actively updates the rollout based on the latest attempt. Reading the table above is usually easier than reverse-engineering the propagation logic in the code: think of the rollout’s transitions as *callbacks* on attempt state changes, plus queue/cancel paths.
|
||||
|
||||
## Spans
|
||||
|
||||
Every traceable operation in a rollout is stored as a [Span][agentlightning.Span]. Spans not only capture fine-grained instrumentation but also act as periodic heartbeats that demonstrate liveness. The first span marks activation; each subsequent one both extends the trace and refreshes the attempt’s [`last_heartbeat_time`][agentlightning.Attempt.last_heartbeat_time]. If no span arrives within the configured [`unresponsive_seconds`][agentlightning.RolloutConfig.unresponsive_seconds], the watchdog downgrades the attempt to **unresponsive** until activity resumes.
|
||||
|
||||
Spans are indexed by `(rollout_id, attempt_id, sequence_id)` where the sequence is monotonic. Tracing analysis tools like [Adapter][agentlightning.Adapter] usually rely on "time order" to reconstruct the trace. However, in a distributed system, the recorded start time and end time of a span are not necessarily in the right order when they aggregated into a central store. Therefore, we enforce every span creation to retrieve a monotonically increasing [`sequence_id`][agentlightning.Span.sequence_id] from the store before adding the span.
|
||||
|
||||
!!! note
|
||||
|
||||
In practice, one `sequence_id` can be used to create multiple spans. In that case, the orders between the multiple spans are determined by the order of `start_time` and `end_time` of the spans.
|
||||
|
||||
### OpenTelemetry conversion
|
||||
|
||||
Runners often produce [OpenTelemetry `ReadableSpan`](https://opentelemetry.io/docs/concepts/signals/traces/#attributes) objects directly. The store normalizes them into [`Span`][agentlightning.Span] as follows:
|
||||
|
||||
1. The runner first requests [`get_next_span_sequence_id`][agentlightning.LightningStore.get_next_span_sequence_id] via `sequence_id = await store.get_next_span_sequence_id(rollout_id, attempt_id)`. This guarantees ordering within the attempt regardless of clock skew.
|
||||
2. `trace_id`, `span_id`, `parent_id`, `name`, `status`, timestamps, attributes, events, links, and resource are copied from the OTEL span. Timestamps are auto-normalized to seconds (nanoseconds are converted).
|
||||
3. OTEL `SpanContext` and parent context are preserved so downstream tools can correlate traces across systems.
|
||||
4. Any additional serializable fields present on the `ReadableSpan` are retained in the stored span (after safe JSON serialization), which keeps the representation forward-compatible.
|
||||
|
||||
Programmatically this is encapsulated by [`Span.from_opentelemetry(readable_span, rollout_id, attempt_id, sequence_id)`][agentlightning.Span.from_opentelemetry]; [`store.add_otel_span(...)`][agentlightning.LightningStore.add_otel_span] simply wraps the fetch-then-add flow. The end result is a store span that is stable to sort, merge, and query, while still preserving the richness of the original OTEL payload.
|
||||
|
||||
!!! tip
|
||||
|
||||
[`add_span`][agentlightning.LightningStore.add_span] or [`add_otel_span`][agentlightning.LightningStore.add_otel_span] both appends a span *and* acts as a heartbeat that can revive `unresponsive` → `running`.
|
||||
|
||||
### OTLP Compatibility
|
||||
|
||||
Some of the LightningStore implementations support exporting traces via the [OTLP/HTTP specification](https://opentelemetry.io/docs/specs/otlp/). For example, [`LightningStoreServer`][agentlightning.LightningStoreServer] exposes `/v1/traces` endpoint, it implements the binary Protobuf variant defined by the spec, including the required `Content-Type: application/x-protobuf`, optional `Content-Encoding: gzip`, and status responses encoded as `google.rpc.Status`. Agent-lightning helps parsing `ExportTraceServiceRequest` messages, validate identifiers, normalize resource metadata, and allocate sequence numbers so store implementations only need to persist [`Span`][agentlightning.Span] objects in order.
|
||||
|
||||
Because the interface speaks standard OTLP, any OpenTelemetry-compatible SDK or collector can emit spans directly to a LightningStore OTLP endpoint without custom shims. The server responds according to the OTLP contract (status code, encoding, and error payloads), which keeps Agent-lightning interoperable with existing observability tooling. This compatibility serves as a strong complement to the OpenTelemetry conversion discussed above.
|
||||
|
||||
Check whether the store supports OTLP traces via the [`capabilities["otlp_traces"]`][agentlightning.LightningStore.capabilities] property.
|
||||
|
||||
## Implementation Overview
|
||||
|
||||
The `agentlightning.store` module is organized into two distinct layers plus optional wrappers:
|
||||
|
||||
```mermaid
|
||||
classDiagram
|
||||
direction TB
|
||||
|
||||
class LightningStore {
|
||||
<<abstract>>
|
||||
+enqueue_rollout()
|
||||
+dequeue_rollout()
|
||||
+update_attempt()
|
||||
+add_span()
|
||||
+query_rollouts()
|
||||
...
|
||||
}
|
||||
|
||||
class LightningCollections {
|
||||
<<abstract>>
|
||||
+rollouts: Collection
|
||||
+attempts: Collection
|
||||
+spans: Collection
|
||||
+resources: Collection
|
||||
+workers: Collection
|
||||
+rollout_queue: Queue
|
||||
+span_sequence_ids: KeyValue
|
||||
+atomic()
|
||||
}
|
||||
|
||||
class CollectionBasedLightningStore~T~ {
|
||||
+collections: T
|
||||
-healthcheck_before()
|
||||
-tracked()
|
||||
}
|
||||
|
||||
class InMemoryLightningStore
|
||||
class MongoLightningStore
|
||||
class InMemoryLightningCollections
|
||||
class MongoLightningCollections
|
||||
|
||||
class LightningStoreServer {
|
||||
+store: LightningStore
|
||||
+start()
|
||||
+stop()
|
||||
}
|
||||
class LightningStoreClient {
|
||||
+server_address: str
|
||||
}
|
||||
class LightningStoreThreaded {
|
||||
+store: LightningStore
|
||||
}
|
||||
|
||||
LightningStore <|-- CollectionBasedLightningStore
|
||||
LightningStore <|-- LightningStoreServer
|
||||
LightningStore <|-- LightningStoreClient
|
||||
LightningStore <|-- LightningStoreThreaded
|
||||
|
||||
CollectionBasedLightningStore <|-- InMemoryLightningStore
|
||||
CollectionBasedLightningStore <|-- MongoLightningStore
|
||||
|
||||
LightningCollections <|-- InMemoryLightningCollections
|
||||
LightningCollections <|-- MongoLightningCollections
|
||||
|
||||
InMemoryLightningStore ..> InMemoryLightningCollections : uses
|
||||
MongoLightningStore ..> MongoLightningCollections : uses
|
||||
|
||||
LightningStoreServer o-- LightningStore : wraps
|
||||
LightningStoreThreaded o-- LightningStore : wraps
|
||||
```
|
||||
|
||||
1. **Collections Layer** – Low-level storage primitives ([`LightningCollections`][agentlightning.store.collection.LightningCollections]) providing CRUD operations via [`Collection`][agentlightning.store.collection.Collection], [`Queue`][agentlightning.store.collection.Queue], and [`KeyValue`][agentlightning.store.collection.KeyValue] interfaces. Each backend (in-memory, MongoDB) implements these primitives.
|
||||
|
||||
2. **Store Layer** – All [`LightningStore`][agentlightning.LightningStore] implementations must inherit from [`LightningStore`][agentlightning.LightningStore] and override the methods to implement the storage logic. [`CollectionBasedLightningStore`][agentlightning.CollectionBasedLightningStore] builds on collections to implement the full [`LightningStore`][agentlightning.LightningStore] API, including business logic like status transitions, watchdog health checks, and retry policies.
|
||||
|
||||
3. **Wrappers** – Cross-cutting concerns live in thin wrappers:
|
||||
- [`LightningStoreThreaded`][agentlightning.LightningStoreThreaded] adds mutex-based thread safety.
|
||||
- [`LightningStoreServer`][agentlightning.LightningStoreServer] / [`LightningStoreClient`][agentlightning.LightningStoreClient] enable multi-process access over HTTP.
|
||||
|
||||
## Collections
|
||||
|
||||
The collections layer provides storage primitives that [`CollectionBasedLightningStore`][agentlightning.CollectionBasedLightningStore] builds upon. This separation keeps business logic (status transitions, watchdog, retries) in the store layer while allowing different backends to focus purely on persistence.
|
||||
|
||||
The off-the-shelf implementations are [`InMemoryLightningCollections`][agentlightning.store.collection.InMemoryLightningCollections] and [`MongoLightningCollections`][agentlightning.store.collection.mongo.MongoLightningCollections], which are the underlying collections for [`InMemoryLightningStore`][agentlightning.InMemoryLightningStore] and [`MongoLightningStore`][agentlightning.store.mongo.MongoLightningStore], respectively.
|
||||
|
||||
### Collection Primitives
|
||||
|
||||
[`LightningCollections`][agentlightning.store.collection.LightningCollections] bundles three primitive types:
|
||||
|
||||
| Primitive | Purpose | Methods |
|
||||
|-----------|---------|---------|
|
||||
| [`Collection[T]`][agentlightning.store.collection.Collection] | Indexed storage with primary keys | [`query()`][agentlightning.store.collection.Collection.query], [`get()`][agentlightning.store.collection.Collection.get], [`insert()`][agentlightning.store.collection.Collection.insert], [`update()`][agentlightning.store.collection.Collection.update], [`upsert()`][agentlightning.store.collection.Collection.upsert], [`delete()`][agentlightning.store.collection.Collection.delete] |
|
||||
| [`Queue[T]`][agentlightning.store.collection.Queue] | FIFO queue for task scheduling | [`enqueue()`][agentlightning.store.collection.Queue.enqueue], [`dequeue()`][agentlightning.store.collection.Queue.dequeue], [`peek()`][agentlightning.store.collection.Queue.peek], [`size()`][agentlightning.store.collection.Queue.size] |
|
||||
| [`KeyValue[K, V]`][agentlightning.store.collection.KeyValue] | Simple key-value store | [`get()`][agentlightning.store.collection.KeyValue.get], [`set()`][agentlightning.store.collection.KeyValue.set], [`inc()`][agentlightning.store.collection.KeyValue.inc], [`chmax()`][agentlightning.store.collection.KeyValue.chmax], [`pop()`][agentlightning.store.collection.KeyValue.pop] |
|
||||
|
||||
Every [`LightningCollections`][agentlightning.store.collection.LightningCollections] instance exposes these named collections:
|
||||
|
||||
- `rollouts` – [`Collection[Rollout]`][agentlightning.store.collection.Collection] keyed by `rollout_id`
|
||||
- `attempts` – [`Collection[Attempt]`][agentlightning.store.collection.Collection] keyed by `(rollout_id, attempt_id)`
|
||||
- `spans` – [`Collection[Span]`][agentlightning.store.collection.Collection] keyed by `(rollout_id, attempt_id, span_id)`
|
||||
- `resources` – [`Collection[ResourcesUpdate]`][agentlightning.store.collection.Collection] keyed by `resources_id`
|
||||
- `workers` – [`Collection[Worker]`][agentlightning.store.collection.Collection] keyed by `worker_id`
|
||||
- `rollout_queue` – [`Queue[str]`][agentlightning.store.collection.Queue] holding rollout IDs awaiting execution
|
||||
- `span_sequence_ids` – [`KeyValue[str, int]`][agentlightning.store.collection.KeyValue] tracking monotonic sequence counters
|
||||
|
||||
### Atomic Operations
|
||||
|
||||
Collections support atomic operations through the [`atomic()`][agentlightning.store.collection.LightningCollections.atomic] context manager:
|
||||
|
||||
```python
|
||||
async with collections.atomic(mode="rw", labels=["rollouts", "attempts"]) as ctx:
|
||||
rollout = await ctx.rollouts.get(filter={"rollout_id": {"exact": rollout_id}})
|
||||
# modify and update within the same transaction
|
||||
await ctx.rollouts.update([updated_rollout])
|
||||
```
|
||||
|
||||
The arguments passed to [`atomic()`][agentlightning.store.collection.LightningCollections.atomic] are quite arbitrary and flexible. Different implementations may have different interpretations of the arguments. For example, to [`InMemoryLightningCollections`][agentlightning.store.collection.InMemoryLightningCollections], the `mode` parameter controls locking behavior (`"r"` for read-only, `"rw"` for read-write), while `labels` specifies which collections to lock. Acquiring locks in sorted order prevents deadlocks when multiple operations run concurrently.
|
||||
|
||||
### Implementing a Custom Backend
|
||||
|
||||
To add a new storage backend, implement [`LightningCollections`][agentlightning.store.collection.LightningCollections]:
|
||||
|
||||
```python
|
||||
from agentlightning.store.collection import LightningCollections, Collection, Queue, KeyValue
|
||||
|
||||
class MyLightningCollections(LightningCollections):
|
||||
@property
|
||||
def rollouts(self) -> Collection[Rollout]:
|
||||
return self._rollouts # your implementation
|
||||
|
||||
@property
|
||||
def rollout_queue(self) -> Queue[str]:
|
||||
return self._queue # your implementation
|
||||
|
||||
# ... implement remaining properties
|
||||
|
||||
async def atomic(self, *, mode, snapshot=False, labels=None, **kwargs):
|
||||
# provide transaction / locking semantics
|
||||
...
|
||||
```
|
||||
|
||||
Then instantiate your store:
|
||||
|
||||
```python
|
||||
from agentlightning.store.collection_based import CollectionBasedLightningStore
|
||||
|
||||
store = CollectionBasedLightningStore(collections=MyLightningCollections())
|
||||
```
|
||||
|
||||
The store layer handles all business logic; your collections just need to provide correct CRUD semantics.
|
||||
|
||||
## Collection-based Store Implementations
|
||||
|
||||
Agent-lightning ships with two collection-based store implementations:
|
||||
|
||||
### InMemoryLightningStore
|
||||
|
||||
[`InMemoryLightningStore`][agentlightning.InMemoryLightningStore] uses [`InMemoryLightningCollections`][agentlightning.store.collection.InMemoryLightningCollections] backed by Python data structures. It supports **fast startup** with zero external dependencies—ideal for local development, CI, and unit tests. It also provides two lock modes, configurable between `"asyncio"` (single-thread, multiple coroutines) and `"thread"` (multi-threaded via [aiologic](https://github.com/x42005e1f/aiologic)).
|
||||
[`InMemoryLightningCollections`][agentlightning.store.collection.InMemoryLightningCollections] use nested dictionaries for O(1) primary-key lookup and `deque` for the task queue.
|
||||
|
||||
### MongoLightningStore
|
||||
|
||||
[`MongoLightningStore`][agentlightning.store.mongo.MongoLightningStore] uses [`MongoLightningCollections`][agentlightning.store.collection.mongo.MongoLightningCollections] backed by MongoDB. It supports **persistent storage** suitable for production deployments and **multi-process safe** via database-level atomicity. It also supports **partition support** via `partition_id` for running multiple trainers against the same database.
|
||||
|
||||
```python
|
||||
from agentlightning.store.mongo import MongoLightningStore
|
||||
|
||||
store = MongoLightningStore(
|
||||
mongo_uri="mongodb://localhost:27017/?replicaSet=rs0",
|
||||
database_name="agentlightning",
|
||||
partition_id="trainer-1", # optional: isolate data per trainer
|
||||
)
|
||||
```
|
||||
|
||||
!!! note
|
||||
|
||||
[`MongoLightningStore`][agentlightning.store.mongo.MongoLightningStore] requires the `mongo` optional dependency. Install with `pip install agentlightning[mongo]`.
|
||||
|
||||
### Capabilities
|
||||
|
||||
[](){ #store-capabilities }
|
||||
|
||||
Different stores have different capabilities. Check the [`capabilities`][agentlightning.LightningStore.capabilities] property to understand what a store supports:
|
||||
|
||||
| Capability | Description | InMemory | Mongo | Server | Client |
|
||||
|------------|-------------|----------|-------|--------|--------|
|
||||
| `thread_safe` | Safe for concurrent access from multiple threads | configurable | ✓ | ✓ | ✓ |
|
||||
| `async_safe` | Safe for concurrent access from multiple coroutines | ✓ | ✓ | ✓ | ✓ |
|
||||
| `zero_copy` | Can be shared across processes without serialization | ✗ | ✓ | ✓ | ✓ |
|
||||
| `otlp_traces` | Exposes an OTLP-compatible `/v1/traces` endpoint | ✗ | ✗ | ✓ | ✓ |
|
||||
|
||||
## Thread Safety
|
||||
|
||||
Thread safety can be achieved at different layers:
|
||||
|
||||
**At the collections layer**: [`InMemoryLightningCollections`][agentlightning.store.collection.InMemoryLightningCollections] accepts a `lock_type` parameter:
|
||||
|
||||
- `"asyncio"` – Uses per-event-loop `asyncio.Lock` for single-threaded, multi-coroutine scenarios.
|
||||
- `"thread"` – Uses `aiologic.Lock` for true multi-threaded access.
|
||||
|
||||
**At the store layer**: [`LightningStoreThreaded`][agentlightning.LightningStoreThreaded] wraps any [`LightningStore`][agentlightning.LightningStore] to add mutex-based thread safety:
|
||||
|
||||
* Methods like [`start_rollout`][agentlightning.LightningStore.start_rollout], [`enqueue_rollout`][agentlightning.LightningStore.enqueue_rollout], [`update_attempt`][agentlightning.LightningStore.update_attempt], [`add_span`][agentlightning.LightningStore.add_span], etc. are guarded by a lock.
|
||||
* Non-mutating, potentially blocking calls remain pass-through by design (e.g., [`wait_for_rollouts`][agentlightning.LightningStore.wait_for_rollouts]), as they don't modify shared state and should not hold the lock for long periods.
|
||||
|
||||
Database-based stores like [`MongoLightningStore`][agentlightning.store.mongo.MongoLightningStore] are inherently thread-safe through database atomicity guarantees.
|
||||
|
||||
## Process Safety and Client-server Store
|
||||
|
||||
**[`LightningStoreServer`][agentlightning.LightningStoreServer]** wraps another underlying store and runs a FastAPI app to expose the store API over HTTP. [`LightningStoreClient`][agentlightning.LightningStoreClient] is a small [`LightningStore`][agentlightning.LightningStore] implementation that talks to the HTTP API.
|
||||
|
||||
!!! warning
|
||||
|
||||
The server HTTP API is not considered a stable API at this moment. Users are encouraged to use the [`LightningStoreClient`][agentlightning.LightningStoreClient] to communicate with the server as a stable interface.
|
||||
|
||||
The server tracks the creator PID. In the owner process it delegates directly to the in-memory store; in other processes it lazily constructs a [`LightningStoreClient`][agentlightning.LightningStoreClient] to talk to the HTTP API. This prevents accidental cross-process mutation of the wrong memory image. When the server is pickled (e.g., via `multiprocessing`), only the minimal fields are serialized, but **NOT** the FastAPI/uvicorn objects. Subprocesses won’t accidentally carry live server state. Forked subprocess should also use [`LightningStoreClient`][agentlightning.LightningStoreClient] to communicate with the server in the main process.
|
||||
|
||||
On the client side, the client retries network/5xx failures using a small backoff, and probes `/v1/agl/health` between attempts. Application exceptions inside the server are wrapped as HTTP 400 with a traceback—these are **not retried**. The client also maintains a **per-event-loop** `aiohttp.ClientSession` map so that tracer callbacks (often on separate loops/threads) don’t hang by reusing a session from another loop.
|
||||
|
||||
Minimal lifecycle:
|
||||
|
||||
```python
|
||||
import agentlightning as agl
|
||||
|
||||
# Server (owner process)
|
||||
in_memory_store = agl.InMemoryLightningStore()
|
||||
server = agl.LightningStoreServer(store=in_memory_store, host="0.0.0.0", port=4747)
|
||||
await server.start() # starts uvicorn in a daemon thread and waits for /health
|
||||
# or keep your own event loop and stop via await server.stop()
|
||||
# await server.run_forever()
|
||||
|
||||
# Client (same or different process)
|
||||
client = agl.LightningStoreClient("http://localhost:4747")
|
||||
|
||||
print(await client.query_rollouts(status_in=["queuing"]))
|
||||
|
||||
await client.close()
|
||||
await server.stop()
|
||||
```
|
||||
|
||||
Another approach is to use a dedicated command line to start a long running server process, possibly sharable across multiple processes. In the main process, you can always use [`LightningStoreClient`][agentlightning.LightningStoreClient] to communicate with the server.
|
||||
|
||||
```bash
|
||||
agl store --port 4747
|
||||
```
|
||||
|
||||
!!! note
|
||||
|
||||
[`LightningStoreClient.wait_for_rollouts`][agentlightning.LightningStoreClient.wait_for_rollouts] intentionally enforces a tiny timeout (≤ 0.1s) to avoid blocking event loops. Poll with short timeouts or compose with `asyncio.wait_for` at a higher layer.
|
||||
@@ -0,0 +1,89 @@
|
||||
# Examples Catalog
|
||||
|
||||
!!! tip "Want to Contribute?"
|
||||
|
||||
We welcome contributions to the examples catalog! Please refer to the [Contributing](../community/contributing.md) guide for more details.
|
||||
|
||||
<div class="grid cards" markdown>
|
||||
|
||||
- :material-robot:{ .lg .middle } __APO room selector__
|
||||
|
||||
---
|
||||
|
||||
Prompt-optimize a room-booking agent with the built-in APO algorithm, then contrast it with the write-your-own algorithm and debugging workflows in the tutorials. Pairs well with the [Train the First Agent how-to]({{ src("docs/how-to/train-first-agent.md") }}) and the [Write the First Algorithm guide]({{ src("docs/how-to/write-first-algorithm.md") }}).
|
||||
|
||||
[:octicons-repo-24: Browse source]({{ src("examples/apo") }})
|
||||
|
||||
- :material-cloud-sync:{ .lg .middle } __Azure OpenAI SFT__
|
||||
|
||||
---
|
||||
|
||||
Run a supervised fine-tuning loop against Azure OpenAI: roll out the capital-lookup agent, turn traces into JSONL, launch fine-tunes, and redeploy the resulting checkpoints through Azure CLI.
|
||||
|
||||
[:octicons-repo-24: Browse source]({{ src("examples/azure") }})
|
||||
|
||||
- :material-calculator:{ .lg .middle } __Calc-X VERL math__
|
||||
|
||||
---
|
||||
|
||||
VERL-based reinforcement learning setup for a math-reasoning agent that uses AutoGen plus an MCP calculator tool to solve Calc-X problems end to end.
|
||||
|
||||
[:octicons-repo-24: Browse source]({{ src("examples/calc_x") }})
|
||||
|
||||
- :material-chart-box:{ .lg .middle } __ChartQA vision-language RL__
|
||||
|
||||
---
|
||||
|
||||
LangGraph-powered workflow for answering chart questions end to end: rollout the multi-modality agent with GPT or vLLM, and train with VERL/GRPO plus self-refinement loops.
|
||||
|
||||
[:octicons-repo-24: Browse source]({{ src("examples/chartqa") }})
|
||||
|
||||
- :material-code-braces:{ .lg .middle } __Claude Code SWE-bench__
|
||||
|
||||
---
|
||||
|
||||
Instrumented driver that runs Anthropic's Claude Code workflow on SWE-bench instances while streaming traces through Agent-lightning—supports hosted vLLM, official Anthropic, or any OpenAI-compatible backend and emits datasets for downstream tuning.
|
||||
|
||||
[:octicons-repo-24: Browse source]({{ src("examples/claude_code") }})
|
||||
|
||||
- :material-view-grid:{ .lg .middle } __Minimal building blocks__
|
||||
|
||||
---
|
||||
|
||||
Bite-sized scripts that isolate Agent-lightning primitives (e.g., LightningStore usage, LLM proxying, minimal vLLM host) so you can study each part before composing larger workflows.
|
||||
|
||||
[:octicons-repo-24: Browse source]({{ src("examples/minimal") }})
|
||||
|
||||
- :material-book-open-page-variant:{ .lg .middle } __RAG (MuSiQue)__
|
||||
|
||||
---
|
||||
|
||||
Retrieval-Augmented Generation pipeline that preps a Wikipedia retriever via MCP and trains a MuSiQue QA agent with GRPO. Documented for historical reference (verified on Agent-lightning v0.1.x).
|
||||
|
||||
[:octicons-repo-24: Browse source]({{ src("examples/rag") }})
|
||||
|
||||
- :material-database:{ .lg .middle } __Spider SQL agent__
|
||||
|
||||
---
|
||||
|
||||
LangGraph-powered text-to-SQL workflow for the Spider benchmark, combining LangChain tooling with Agent-lightning rollouts; follow along with the [how-to for training SQL agents]({{ src("docs/how-to/train-sql-agent.md") }}).
|
||||
|
||||
[:octicons-repo-24: Browse source]({{ src("examples/spider") }})
|
||||
|
||||
- :material-thought-bubble:{ .lg .middle } __Tinker integration__
|
||||
|
||||
---
|
||||
|
||||
Adapter package ([`agl_tinker`]({{ src("examples/tinker/agl_tinker") }})) with Tinker plus sample CrewAI/OpenAI agents that feed Agent-lightning traces into Tinker’s reinforcement-learning backend for both toy and 20-Questions-style workflows.
|
||||
|
||||
[:octicons-repo-24: Browse source]({{ src("examples/tinker") }})
|
||||
|
||||
- :material-fast-forward:{ .lg .middle } __Unsloth SFT__
|
||||
|
||||
---
|
||||
|
||||
Supervised fine-tuning loop that ranks math-agent rollouts, fine-tunes with Unsloth’s 4-bit LoRA stack, and mirrors the [Fine-tune with Unsloth recipe]({{ src("docs/how-to/unsloth-sft.md") }}).
|
||||
|
||||
[:octicons-repo-24: Browse source]({{ src("examples/unsloth") }})
|
||||
|
||||
</div>
|
||||
@@ -0,0 +1,229 @@
|
||||
# Train the First Agent with Agent-lightning
|
||||
|
||||
Welcome! This tutorial is your first step into making AI agents smarter using the **Agent-lightning** framework. We'll show you how to take a simple agent and automatically improve its performance through a process called [**Automatic Prompt Optimization (APO)**](../algorithm-zoo/apo.md).
|
||||
|
||||
The main goal of Agent-lightning is to provide a structured way to **train your agents**. Just like you train a machine learning model on data, you can train an agent on a task dataset. This could involve using Reinforcement Learning (RL) to teach it new behaviors or, as we'll do today, optimizing its prompts to make it more accurate and reliable.
|
||||
|
||||
!!! tip
|
||||
|
||||
You can open the sample code [room_selector_apo.py]({{ src("examples/apo/room_selector_apo.py") }}) and [room_selector.py]({{ src("examples/apo/room_selector.py") }}) as you go through this tutorial.
|
||||
|
||||
## Our Example: The Room Selector Agent
|
||||
|
||||
Today, we'll work with an agent whose job is to book a meeting room. It's a common but tricky task with multiple constraints.
|
||||
|
||||
Here's how the agent works:
|
||||
|
||||
- **Input:** It receives a task with specific requirements, like "`Find a room for 4 people at 10:00 AM with a whiteboard.`"
|
||||
- **Action:** The agent uses a Large Language Model (LLM) to understand the request. It can also use tools, which are pre-defined functions it can call, to get more information, such as checking room availability in an external database.
|
||||
- **Output:** Its final decision is the ID of the best room it found, like "`A103`".
|
||||
- **Reward:** After the agent makes its choice, a separate "grader" function scores its performance on a scale of 0 to 1. This score is called its **reward**. A perfect choice gets a 1.0, while a wrong one gets a 0.0.
|
||||
|
||||
The agent's logic is sound, but its performance heavily depends on its initial prompt. A poorly worded prompt will confuse the LLM, leading to bad decisions. Our goal is to use Agent-lightning to find the best possible prompt automatically.
|
||||
|
||||
!!! tip "A Closer Look at the Agent's Logic"
|
||||
|
||||
Modern LLMs can do more than just generate text; they can decide to call functions you provide. This is often called tool use or function calling. Our agent uses this capability to make informed decisions. If you're new to this concept, you can read more about it in [OpenAI's documentation](https://platform.openai.com/docs/guides/function-calling).
|
||||
|
||||
Here is a sketch of the agent's logic, adhering closely to the OpenAI API:
|
||||
|
||||
```python
|
||||
# Pseudo-code for the Room Selector agent
|
||||
|
||||
import openai
|
||||
import json
|
||||
|
||||
def room_selector_agent(task, prompt):
|
||||
client = openai.OpenAI()
|
||||
messages = [{"role": "user", "content": prompt.format(**task)}]
|
||||
tools = [ ... ] # Tool definition for the LLM
|
||||
|
||||
# 1. First LLM call to decide if a tool is needed.
|
||||
response = client.chat.completions.create(
|
||||
model="gpt-5-mini",
|
||||
messages=messages,
|
||||
tools=tools,
|
||||
tool_choice="auto",
|
||||
)
|
||||
response_message = response.choices[0].message
|
||||
tool_calls = response_message.tool_calls
|
||||
|
||||
# 2. Check if the LLM wants to use a tool.
|
||||
if tool_calls:
|
||||
messages.append(response_message) # Append assistant's reply
|
||||
|
||||
# 3. Execute the tool and get the real-world data.
|
||||
for tool_call in tool_calls:
|
||||
function_name = tool_call.function.name
|
||||
if function_name == "get_rooms_and_availability":
|
||||
function_args = json.loads(tool_call.function.arguments)
|
||||
# Query the local room database
|
||||
function_response = get_rooms_and_availability(
|
||||
date=function_args.get("date"),
|
||||
time_str=function_args.get("time"),
|
||||
duration_min=function_args.get("duration_min"),
|
||||
)
|
||||
messages.append({
|
||||
"tool_call_id": tool_call.id,
|
||||
"role": "tool",
|
||||
"name": function_name,
|
||||
"content": json.dumps(function_response),
|
||||
})
|
||||
|
||||
# 4. Second LLM call with the tool's output to get a final choice.
|
||||
second_response = client.chat.completions.create(
|
||||
model="gpt-5-mini",
|
||||
messages=messages,
|
||||
)
|
||||
final_choice = second_response.choices[0].message.content
|
||||
else:
|
||||
final_choice = response_message.content
|
||||
|
||||
# 5. Grade the final choice to get a reward.
|
||||
reward = grade_the_choice(final_choice, task["expected_choice"])
|
||||
return reward
|
||||
```
|
||||
|
||||
In Agent-lightning, you wrap this logic in a Python function marked with the [`@rollout`][agentlightning.rollout] decorator, so that the agent can be managed and tuned by Agent-lightning's runner and trainer. The `prompt_template` that the APO algorithm tunes is passed in as an argument:
|
||||
|
||||
```python
|
||||
import agentlightning as agl
|
||||
|
||||
@agl.rollout
|
||||
def room_selector(task: RoomSelectionTask, prompt_template: agl.PromptTemplate) -> float:
|
||||
# ... agent logic using the prompt_template ...
|
||||
|
||||
# The final reward is determined by a grader function
|
||||
reward = room_selection_grader(client, final_message, task["expected_choice"])
|
||||
return reward
|
||||
```
|
||||
|
||||
## Core Concepts: Tasks, Rollouts, Spans, and Prompt Templates
|
||||
|
||||
To understand how Agent-lightning works, you need to know these key terms.
|
||||
|
||||
### Task
|
||||
|
||||
A task is a specific input or problem statement given to the agent. It defines what the agent needs to accomplish.
|
||||
|
||||
!!! example "Analogy: Task"
|
||||
|
||||
If the agent is a chef, a task is the recipe request: "Bake a chocolate cake."
|
||||
|
||||
### Rollout
|
||||
|
||||
A rollout is a single, complete execution of an agent attempting to solve a given **task**. It's the entire story from receiving the task to producing a final result and receiving a reward. A rollout captures a full trace of the agent's execution.
|
||||
|
||||
!!! example "Analogy: Rollout"
|
||||
|
||||
A rollout is one full attempt by the chef to bake the chocolate cake, from gathering ingredients to the final taste test.
|
||||
|
||||
### Span
|
||||
|
||||
A span represents a single unit of work or an operation within a **rollout**. Spans are the building blocks of a trace. They have a start and end time and contain details about the specific operation, like an LLM call, a tool execution, or a reward calculation. For a more precise definition, see the [OpenTelemetry documentation](https://opentelemetry.io/docs/concepts/signals/traces/).
|
||||
|
||||
!!! example "Analogy: Span"
|
||||
|
||||
If the rollout is "baking a cake," a span could be "preheating the oven," "mixing flour and sugar," or "adding frosting." Each is a distinct step or unit of work.
|
||||
|
||||
The picture below from [ADK](https://google.github.io/adk-docs/observability/cloud-trace/) shows a typical rollout, where each rectangle in the waterfall visualizes a span. As can be seen in the visualization, spans can be sequential, parallel or nested among each other. In other frameworks, the terminology might be slightly different. Agent-lightning follows the terminologies used by OpenTelemetry to avoid confusion.
|
||||
|
||||
|
||||
|
||||
### Prompt Template
|
||||
|
||||
A prompt template is a reusable instruction for the agent, often containing placeholders that can be filled in with specific details from a task. It is a key **"resource"** that the algorithm learns and improves over time.
|
||||
|
||||
!!! example "Analogy: Resource (Prompt Template)"
|
||||
|
||||
If the task is the recipe request, the prompt template is the master recipe card that the chef follows. The algorithm's job is to edit this recipe card to make the instructions clearer and the final dish better.
|
||||
|
||||
## The Training Loop: How the Magic Happens
|
||||
|
||||
Training in Agent-lightning revolves around a clear, managed loop, orchestrated by the **Trainer**. The diagram below illustrates this core interaction:
|
||||
|
||||
{ .center }
|
||||
|
||||
**The Loop Explained:**
|
||||
|
||||
- **Algorithm to Agent (via Trainer):** The **Algorithm** (the "brain") creates an improved **Prompt Template** and selects **Tasks**. The Trainer then sends both to the Agent.
|
||||
- **Agent to Algorithm (via Trainer):** For each task it receives, the Agent uses the provided prompt template to perform a Rollout, executing its logic and potentially using tools. During this rollout, the runner that runs the agent captures Spans that detail every step. The agent also calculates a Reward for its performance on the task. These spans and rewards are then sent back to the Algorithm via the Trainer.
|
||||
- **Algorithm Learning:** The Algorithm then analyzes these spans and rewards to learn how to improve the agent's behavior, for example, by generating a better prompt. This improved prompt is then used in the next iteration of tasks.
|
||||
|
||||
This cycle continues, allowing the agent to continuously learn and get better at solving tasks.
|
||||
|
||||
!!! note
|
||||
|
||||
In the next tutorial, we will see that the "via Trainer" here is not accurate. It's actually via the runner and store.
|
||||
|
||||
### The Algorithm
|
||||
|
||||
The algorithm is the smart part of the system that drives the improvement. In this tutorial, we use [**APO**][agentlightning.algorithm.apo.APO] (Automatic Prompt Optimization). It works in a few steps:
|
||||
|
||||
1. **Evaluate:** The algorithm first asks for rollouts to be run using the current prompt template to see how well it performs.
|
||||
2. **Critique:** It then looks at the detailed spans from those rollouts. Using a powerful LLM (`gpt-5-mini`), it generates a "textual gradient", which is a natural language critique of the prompt. For example: "The prompt is ambiguous about how to handle tie-breakers for equally good rooms."
|
||||
3. **Rewrite:** Finally, it gives the critique and the original prompt to another LLM (`gpt-4.1-mini`) and asks it to apply the edits, generating a new, improved prompt template.
|
||||
|
||||
This cycle repeats, with each round producing a slightly better prompt. To use it, you simply initialize the APO class with your desired hyperparameters.
|
||||
|
||||
```python
|
||||
# In the main training script: run_apo.py
|
||||
from openai import AsyncOpenAI
|
||||
|
||||
openai = AsyncOpenAI()
|
||||
algo = agl.APO(openai)
|
||||
```
|
||||
|
||||
!!! tip
|
||||
|
||||
Make sure you have `OPENAI_API_KEY` set in your environment variables.
|
||||
|
||||
### The Trainer
|
||||
|
||||
The Trainer is the central component you'll interact with. It connects everything and manages the entire workflow by running the loop described above. You configure the Trainer, providing the algorithm, the number of parallel runners, and the initial prompt. A single call to [`trainer.fit()`][agentlightning.Trainer.fit] kicks off the entire process!
|
||||
|
||||
```python
|
||||
# 1. Configure the Trainer with the algorithm and initial prompt
|
||||
trainer = agl.Trainer(
|
||||
algorithm=algo,
|
||||
n_runners=8, # Run 8 agents in parallel to try out the prompts
|
||||
initial_resources={
|
||||
# The initial prompt template to be tuned
|
||||
"prompt_template": prompt_template_baseline()
|
||||
},
|
||||
# This is used to convert the span data into a message format consumable by APO algorithm
|
||||
adapter=agl.TraceToMessages(),
|
||||
)
|
||||
|
||||
# 2. Load datasets: They can be list of task objects consumable by `room_selector`.
|
||||
dataset_train, dataset_val = ...
|
||||
|
||||
# 3. Start the training process!
|
||||
trainer.fit(
|
||||
agent=room_selector,
|
||||
train_dataset=dataset_train,
|
||||
val_dataset=dataset_val
|
||||
)
|
||||
```
|
||||
|
||||
!!! tip
|
||||
|
||||
[`TraceToMessages`][agentlightning.TraceToMessages] is a convenience adapter that converts spans into OpenAI chat messages. It requires `openai >= 1.100.0` to be installed.
|
||||
|
||||
## Training Results
|
||||
|
||||
The APO algorithm successfully improved the agent's performance. We ran the example with the following hyper-parameters:
|
||||
|
||||
- `val_batch_size` = 10
|
||||
- `gradient_batch_size` = 4
|
||||
- `beam_width` = 2
|
||||
- `branch_factor` = 2
|
||||
- `beam_rounds` = 2
|
||||
|
||||
The validation accuracy on the 29 samples of datasets steadily increase from 0.569 (baseline) to **0.721** (after round 2). The tuning takes around 10 minutes with 8 runners. We ran twice, and the results are shown in the chart below.
|
||||
|
||||
<div style="height:400px">
|
||||
<canvas data-chart='{ "type": "line", "data": { "labels": ["Baseline", "After round 1", "After round 2"], "datasets": [ { "label": "Run #1", "data": [0.569, 0.638, 0.721], "spanGaps": true }, { "label": "Run #2", "data": [0.534, 0.628, 0.645], "spanGaps": true } ] }, "options": { "interaction": { "mode": "nearest", "intersect": false }, "plugins": { "legend": { "display": true, "position": "top" }, "title": { "display": true, "text": "Validation Accuracy Across Rounds" } }, "scales": { "x": { "title": { "display": true, "text": "Round" } }, "y": { "title": { "display": true, "text": "Accuracy" } } } } }'></canvas>
|
||||
</div>
|
||||
|
||||
This demonstrates how Agent-lightning can efficiently and automatically enhance your agent's capabilities with just a few lines of code.
|
||||
@@ -0,0 +1,394 @@
|
||||
# Train SQL Agent with Agent-lightning and VERL
|
||||
|
||||
This walkthrough builds upon the **Agent-lightning SQL Agent** example and explains how the system components integrate: a **LangGraph-based SQL agent** wrapped as a [`LitAgent`][agentlightning.LitAgent], the **[`VERL`][agentlightning.algorithm.verl.VERL] reinforcement learning (RL) algorithm**, and the **[`Trainer`][agentlightning.Trainer]**, which coordinates both training and debugging.
|
||||
|
||||
The command-line interface in [`examples/spider/train_sql_agent.py`]({{ src("examples/spider/train_sql_agent.py") }}) provides a complete runnable example. However, this document focuses on understanding the underlying architecture so you can effectively adapt the workflow to your own agents.
|
||||
|
||||
## SQL Agent Architecture
|
||||
|
||||
Agent-lightning integrates seamlessly with various orchestration frameworks, including [Agent Framework](https://github.com/microsoft/agent-framework), [AutoGen](https://github.com/microsoft/autogen), [CrewAI](https://www.crewai.com/), [LangGraph](https://github.com/langchain-ai/langgraph), and the [OpenAI Agents SDK](https://github.com/openai/openai-agents-python). It can also interoperate with custom Python logic.
|
||||
|
||||
In this example, **LangGraph** defines a cyclic workflow that mirrors an analyst’s iterative SQL development process. The following graph (rendered directly from [`sql_agent.py`]({{ src("examples/spider/sql_agent.py") }})) illustrates how the agent drafts, executes, critiques, and refines queries until a satisfactory result is achieved.
|
||||
|
||||
```mermaid
|
||||
---
|
||||
config:
|
||||
flowchart:
|
||||
curve: linear
|
||||
---
|
||||
graph LR;
|
||||
__start__([<p>__start__</p>]):::first
|
||||
write_query(write_query)
|
||||
execute_query(execute_query)
|
||||
check_query(check_query)
|
||||
rewrite_query(rewrite_query)
|
||||
__end__([<p>__end__</p>]):::last
|
||||
__start__ --> write_query;
|
||||
check_query -.-> __end__;
|
||||
check_query -.-> rewrite_query;
|
||||
execute_query --> check_query;
|
||||
rewrite_query --> execute_query;
|
||||
write_query --> execute_query;
|
||||
classDef default fill:#f2f2f2,line-height:1.2
|
||||
classDef first fill-opacity:0
|
||||
classDef last fill:#cccccc
|
||||
```
|
||||
|
||||
!!! note
|
||||
|
||||
The workflow proceeds through the following stages:
|
||||
|
||||
1. **write_query** – Generates an initial SQL query from the user’s question and the database schema.
|
||||
2. **execute_query** – Executes the generated query against the target database.
|
||||
3. **check_query** – Evaluates the query and its results (or errors) using a specialized prompt (`CHECK_QUERY_PROMPT`) to detect issues.
|
||||
4. **rewrite_query** – If issues are identified, the agent rewrites the query using feedback from the previous step and re-enters the loop.
|
||||
5. **END** – The cycle terminates when the query is validated or the maximum iteration count (`max_turns`) is reached. Each *turn* consists of one full loop through the `write_query`, `execute_query`, `check_query`, and (if applicable) `rewrite_query` stages.
|
||||
|
||||
In this tutorial, **reinforcement learning (RL)** is used to optimize the `write_query` and `rewrite_query` stages. While the `check_query` step shares the same underlying LLM weights, its trace data is not used for learning.
|
||||
|
||||
To keep the design modular and maintainable, it is recommended to define the LangGraph-based SQL Agent in a separate file and expose it via a builder function such as:
|
||||
|
||||
```python
|
||||
def build_langgraph_sql_agent(
|
||||
database_path: str,
|
||||
openai_base_url: str,
|
||||
model: str,
|
||||
sampling_parameters: Dict[str, Any],
|
||||
max_turns: int,
|
||||
truncate_length: int
|
||||
):
|
||||
builder = StateGraph(State)
|
||||
builder.add_node(write_query)
|
||||
...
|
||||
|
||||
builder.add_edge(START, "write_query")
|
||||
...
|
||||
|
||||
return builder.compile().graph()
|
||||
```
|
||||
|
||||
This approach isolates your LangGraph logic from Agent-lightning version changes, improving both readability and debuggability.
|
||||
|
||||
## Bridging LangGraph and Agent-lightning
|
||||
|
||||
!!! tip
|
||||
|
||||
Keep [`sql_agent.py`]({{ src("examples/spider/sql_agent.py") }}) open on the side while reading this section. This will help you understand how the code snippets shown here work in practice.
|
||||
|
||||
The **`LitSQLAgent`** class defined in [`sql_agent.py`]({{ src("examples/spider/sql_agent.py") }}) acts as the bridge. It subclasses [`agl.LitAgent`][agentlightning.LitAgent], allowing the runner to provision shared resources (e.g., [LLMs][agentlightning.LLM]) for each rollout.
|
||||
|
||||
Below is a simplified illustration of the key logic (note: this is conceptual pseudocode; the actual implementation includes dataset-specific details):
|
||||
|
||||
```python
|
||||
class LitSQLAgent(agl.LitAgent[Dict[str, Any]]):
|
||||
|
||||
def __init__(self, max_turns: int, truncate_length: int):
|
||||
# Every turn here refers to a full cycle of write/exe/check/rewrite
|
||||
self.max_turns = max_turns
|
||||
self.truncate_length = truncate_length
|
||||
|
||||
def rollout(
|
||||
self,
|
||||
task: Dict[str, Any],
|
||||
resources: agl.NamedResources,
|
||||
rollout: agl.Rollout
|
||||
) -> float | None:
|
||||
llm: agl.LLM = resources["main_llm"]
|
||||
agent = build_langgraph_sql_agent(
|
||||
database_path="sqlite:///" + task["db_id"],
|
||||
max_turns=self.max_turns,
|
||||
truncate_length=self.truncate_length,
|
||||
openai_base_url=llm.get_base_url(rollout.rollout_id, rollout.attempt.attempt_id),
|
||||
model=llm.model,
|
||||
sampling_parameters=llm.sampling_parameters,
|
||||
)
|
||||
result = agent.invoke({"question": question}, {
|
||||
"callbacks": [self.tracer.get_langchain_handler()],
|
||||
"recursion_limit": 100,
|
||||
})
|
||||
reward = evaluate_query(result["query"], ground_truth, db_path, raise_on_error=False)
|
||||
return reward
|
||||
```
|
||||
|
||||
The `LitSQLAgent` serves as a lightweight wrapper around the LangGraph agent, providing the correct interface for the [`rollout`][agentlightning.LitAgent.rollout] method. It constructs the LangGraph agent, invokes it, and returns the evaluation result as a reward signal.
|
||||
|
||||
The `"main_llm"` resource key is a convention between the agent and [VERL][agentlightning.algorithm.verl.VERL]. It is used to inject an OpenAI-compatible endpoint from the [VERL][agentlightning.algorithm.verl.VERL] algorithm during rollout. Two approaches are supported to use this [agentlightning.LLM][] resource:
|
||||
|
||||
1. **Direct access** – Use [`llm.endpoint`][agentlightning.LLM.endpoint] for a simple integration (identical to the v0.1 example).
|
||||
2. **Context-aware access** – Use [`get_base_url`][agentlightning.ProxyLLM.get_base_url] with [`rollout.rollout_id`][agentlightning.Rollout.rollout_id] and [`rollout.attempt.attempt_id`][agentlightning.Attempt.attempt_id].
|
||||
This approach enables per-caller trace attribution, improving trace collection per rollout or attempt when runner-side tracers are unavailable. For details, see [Working with Traces](../tutorials/traces.md).
|
||||
|
||||
## Reward Signal and Evaluation
|
||||
|
||||
The `evaluate_query` function provides the reward mechanism for RL training. In agent training, obtaining a consistent and meaningful reward signal is often challenging. Fortunately, this is simplified when using the [**Spider dataset**](https://yale-lily.github.io/spider). The dataset includes ~8k samples containing natural-language questions, database schemas, and ground-truth SQL queries.
|
||||
|
||||
Using the [**Spider evaluator**](https://github.com/taoyds/test-suite-sql-eval), the agent's generated query is executed and compared to the ground-truth query on the target database. The two queries are considered equivalent if they produce identical execution results.
|
||||
|
||||
!!! attention
|
||||
|
||||
The ground-truth queries must **never** be exposed to the agent during training to prevent data leakage.
|
||||
|
||||
In this setup, the reward is returned directly from the [`rollout`][agentlightning.LitAgent.rollout] method, enabling the runner to forward it back to the RL algorithm.
|
||||
|
||||
!!! warning
|
||||
|
||||
Avoid using [`emit_reward`][agentlightning.emit_reward] in conjunction with returning a reward value. Doing both will cause the algorithm to receive duplicate reward signals, leading to inconsistent training behavior.
|
||||
|
||||
## Configuring VERL for Reinforcement Learning
|
||||
|
||||
View [`examples/spider/train_sql_agent.py`]({{ src("examples/spider/train_sql_agent.py") }}) for a full reinforcement learning configuration, which is a plain Python dictionary. It mirrors (and actually *is*) the [shell arguments](https://verl.readthedocs.io/en/latest/index.html) used to launch training in the VERL framework but is easier to tweak programmatically:
|
||||
|
||||
```python
|
||||
verl_config: Dict[str, Any] = {
|
||||
"algorithm": {"adv_estimator": "grpo", "use_kl_in_reward": False},
|
||||
"data": {
|
||||
# train_files and val_files are no longer needed here
|
||||
# because data are read in agl.Trainer
|
||||
...,
|
||||
# Controls how many tasks are pooled per step
|
||||
# (multiplied by actor_rollout_ref.rollout.n)
|
||||
"train_batch_size": 32,
|
||||
# Prompt and responses larger than these lengths are truncated
|
||||
"max_prompt_length": 4096,
|
||||
"max_response_length": 2048,
|
||||
},
|
||||
"actor_rollout_ref": {
|
||||
"rollout": {
|
||||
# Only vLLM is supported currently
|
||||
"name": "vllm",
|
||||
# Equals to group size of GRPO
|
||||
"n": 4,
|
||||
# Used to enable tool call parser in vLLM
|
||||
"multi_turn": {"format": "hermes"},
|
||||
...
|
||||
},
|
||||
"actor": {"ppo_mini_batch_size": 32, "optim": {"lr": 1e-6}, ...},
|
||||
"model": {
|
||||
# Config your preferred LLM here
|
||||
"path": "Qwen/Qwen2.5-Coder-1.5B-Instruct",
|
||||
...
|
||||
},
|
||||
},
|
||||
"trainer": {
|
||||
"n_gpus_per_node": 1,
|
||||
# Validation once before training starts
|
||||
"val_before_train": True,
|
||||
# Validation every N training steps
|
||||
"test_freq": 32,
|
||||
# Save checkpoints every N training steps
|
||||
"save_freq": 64,
|
||||
# Go through the train dataset this many times
|
||||
"total_epochs": 2
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
This is equivalent to the following CLI invocation:
|
||||
|
||||
```bash
|
||||
python3 -m verl.trainer.main_ppo \
|
||||
algorithm.adv_estimator=grpo \
|
||||
algorithm.use_kl_in_reward=False \
|
||||
data.train_batch_size=32 \
|
||||
data.max_prompt_length=4096 \
|
||||
data.max_response_length=2048 \
|
||||
actor_rollout_ref.rollout.name=vllm \
|
||||
actor_rollout_ref.rollout.n=4 \
|
||||
actor_rollout_ref.rollout.multi_turn.format=hermes \
|
||||
actor_rollout_ref.actor.ppo_mini_batch_size=32 \
|
||||
actor_rollout_ref.actor.optim.lr=1e-6 \
|
||||
actor_rollout_ref.model.path=Qwen/Qwen2.5-Coder-1.5B-Instruct \
|
||||
trainer.n_gpus_per_node=1 \
|
||||
trainer.val_before_train=True \
|
||||
trainer.test_freq=32 \
|
||||
trainer.save_freq=64 \
|
||||
trainer.total_epochs=2
|
||||
```
|
||||
|
||||
!!! warning
|
||||
We used to provide a CLI called `python -m agentlightning.verl` to launch training in v0.1. This is no longer the recommended approach. Instead, use [`agl.Trainer`][agentlightning.Trainer] to run VERL and agent runners together, or follow the [debugging tutorial](../tutorials/debug.md) if you want an isolated experience similar to v0.1.
|
||||
## Orchestrating Training with [`Trainer`][agentlightning.Trainer]
|
||||
|
||||
[`Trainer`][agentlightning.Trainer] is the high-level orchestrator that integrates the agent, algorithm, dataset, and distributed runners. The key benefits of using the [`Trainer`][agentlightning.Trainer] are:
|
||||
|
||||
1. It allows you to launch everything with a single line of code: `trainer.fit(...)`.
|
||||
2. It exposes configuration options such as `n_runners` to control parallelism and `adapter` to define how algorithms interpret the trace data produced by the agent.
|
||||
|
||||
An example usage is shown below:
|
||||
|
||||
```python
|
||||
import agentlightning as agl
|
||||
|
||||
agent = LitSQLAgent()
|
||||
algorithm = agl.VERL(verl_config)
|
||||
trainer = agl.Trainer(
|
||||
n_runners=10,
|
||||
algorithm=algorithm,
|
||||
adapter={"agent_match": active_agent},
|
||||
)
|
||||
train_data = pd.read_parquet("data/train_spider.parquet").to_dict("records")
|
||||
val_data = pd.read_parquet("data/test_dev_500.parquet").to_dict("records")
|
||||
trainer.fit(agent, train_dataset=train_data, val_dataset=val_data)
|
||||
```
|
||||
|
||||
First, `agl.VERL(verl_config)` launches the [`VERL`][agentlightning.algorithm.verl.VERL] algorithm and its OpenAI-compatible proxy. The `train_data` and `val_data` are passed into [`VERL`][agentlightning.algorithm.verl.VERL], which enqueues tasks to a centralized task queue managed by the [`LightningStore`][agentlightning.LightningStore], accessible to all runners.
|
||||
|
||||
When [`Trainer.fit`][agentlightning.Trainer.fit] is called, it launches 10 concurrent runners (as specified by `n_runners=10`). Each runner pulls tasks from the centralized task queue, executes the agent’s [`rollout`][agentlightning.LitAgent.rollout] method, collects traces, and returns rewards to VERL for training.
|
||||
|
||||
The [`Adapter`][agentlightning.Adapter], as discussed earlier, is used at the algorithm side, and receives the traces emitted by the agent and runners. The `agent_match` parameter ensures [`VERL`][agentlightning.algorithm.verl.VERL] only ingests spans from the specific agent you want to optimize.
|
||||
In the example above, there are at least three agents—`write_query`, `rewrite_query`, and `check_query`. By setting `agent_match` to a regex like `"write"`, both `write_query` and `rewrite_query` agents are optimized simultaneously. You can also set it to `"write|check"` or `None` to include all agents if desired.
|
||||
|
||||
## Dry-Run the Pipeline with [`Trainer.dev`][agentlightning.Trainer.dev]
|
||||
|
||||
Before committing hours of GPU time, you can **dry-run** the agent with [`Trainer.dev()`][agentlightning.Trainer.dev]. This method swaps in the lightweight [`Baseline`][agentlightning.Baseline] algorithm, enqueues up to ten tasks, and prints every span emitted by the agent. Because it uses the same runner stack as full training, it’s ideal for verifying database connections and LangGraph control flow.
|
||||
|
||||
To begin, the agent needs a valid OpenAI-compatible endpoint since VERL is not active in this mode. You can use OpenAI’s official API or your own local LLM endpoint. Wrap it as follows:
|
||||
|
||||
```python
|
||||
trainer = agl.Trainer(
|
||||
n_workers=1,
|
||||
initial_resources={
|
||||
"main_llm": agl.LLM(
|
||||
endpoint=os.environ["OPENAI_API_BASE"],
|
||||
model="gpt-4.1-nano",
|
||||
sampling_parameters={"temperature": 0.7},
|
||||
)
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
Then, call [`trainer.dev(...)`][agentlightning.Trainer.dev] with a small number of tasks:
|
||||
|
||||
```python
|
||||
dev_data = pd.read_parquet("data/test_dev_500.parquet").to_dict("records")[:10]
|
||||
trainer.dev(agent, dev_dataset=dev_data)
|
||||
```
|
||||
|
||||
Run this in a Python session or adapt your script to include a `--dev` flag. Once the spans appear healthy and the rewards are non-zero, switch back to [`trainer.fit(...)`][agentlightning.Trainer.fit] for full RL training. See the [debugging tutorial](../tutorials/debug.md) for more tips on how to debug the agent.
|
||||
|
||||
## Running the Sample Code
|
||||
|
||||
The following tutorial explains how to run the complete example in [`examples/spider`]({{ src("examples/spider") }}).
|
||||
|
||||
### Dataset
|
||||
|
||||
The trainer expects three Parquet files inside `examples/spider/data`:
|
||||
`train_spider.parquet`, `test_dev_500.parquet`, and `test_dev.parquet`.
|
||||
|
||||
Download the curated dataset bundle provided with the repository:
|
||||
|
||||
```bash
|
||||
cd examples/spider
|
||||
pip install gdown # included in the 'experiment' optional dependency
|
||||
gdown --fuzzy https://drive.google.com/file/d/1oi9J1jZP9TyM35L85CL3qeGWl2jqlnL6/view
|
||||
unzip -q spider-data.zip -d data
|
||||
rm spider-data.zip
|
||||
```
|
||||
|
||||
If you prefer to generate the files yourself, download [Spider 1.0](https://yale-lily.github.io/spider) and run:
|
||||
|
||||
```bash
|
||||
python spider_eval/convert_dataset.py
|
||||
```
|
||||
|
||||
Set `VERL_SPIDER_DATA_DIR` if you store the dataset outside the default `data` directory.
|
||||
|
||||
### Dependencies
|
||||
|
||||
Create a clean virtual environment, activate it, and install Agent-lightning with the VERL extras required by [this tutorial](../tutorials/installation.md). Install LangChain-related dependencies as needed.
|
||||
|
||||
For full training profiles, plan to use a GPU with at least **40 GB** of memory.
|
||||
|
||||
### Launch Training
|
||||
|
||||
From [`examples/spider`]({{ src("examples/spider") }}), run one of the helper scripts depending on your model preference:
|
||||
|
||||
```bash
|
||||
python train_sql_agent.py qwen # Default Qwen-2.5-Coder-1.5B run
|
||||
python train_sql_agent.py llama # LLaMA-3.2-1B with llama3_json tool parser
|
||||
```
|
||||
|
||||
The script instantiates `LitSQLAgent` and launches [`trainer.fit`][agentlightning.Trainer.fit].
|
||||
Provide `--active-agent my_agent_variant` if you only want to train one of the agents in the graph.
|
||||
|
||||
For the LLaMA profile, export an `HF_TOKEN` before running so VERL can download the model weights.
|
||||
|
||||
!!! tip "Troubleshooting"
|
||||
|
||||
If you have got some Ray worker errors on either `WANDB_API_KEY` not set, or `HF_TOKEN` not set, or data not found, please try to restart the Ray cluster with the helper script: [scripts/restart_ray.sh]({{ src("scripts/restart_ray.sh") }}), which essentially stops the ray cluster if any, and starts a new one:
|
||||
|
||||
```bash
|
||||
env RAY_DEBUG=legacy HYDRA_FULL_ERROR=1 VLLM_USE_V1=1 ray start --head --dashboard-host=0.0.0.0
|
||||
```
|
||||
|
||||
!!! note "Launching Training with NPUs"
|
||||
|
||||
The example also supports running with **Huawei Ascend NPUs**. This feature is contributed by [Teams from Huawei](https://github.com/microsoft/agent-lightning/pull/272). To use it, resort to the function `config_train_npu` in the script.
|
||||
|
||||
**Hardware Supported:** Atlas 200T A2 Box16, Atlas 900 A2 PODc, Atlas 800T A3. At least **a single 40GB NPU** is required to run the **Qwen2.5-Coder-1.5B-Instruct** model.
|
||||
|
||||
**Environment Setup:** Python 3.11.13, CANN 8.2.RC1, torch 2.7.1+cpu, torch_npu 2.7.1.dev20250724. For basic environment preparation, please refer to this [document](https://gitcode.com/Ascend/pytorch).
|
||||
|
||||
Before installing dependencies, configure the following pip mirrors:
|
||||
|
||||
```bash
|
||||
pip config set global.index-url http://repo.huaweicloud.com/repository/pypi/simple
|
||||
pip config set global.extra-index-url "https://download.pytorch.org/whl/cpu/ https://mirrors.huaweicloud.com/ascend/repos/pypi"
|
||||
```
|
||||
|
||||
Then install vLLM, vLLM-Ascend and VERL:
|
||||
|
||||
```bash
|
||||
pip install vllm==0.10.0 --trusted-host repo.huaweicloud.com
|
||||
pip install vllm-Ascend==0.10.0rc1 --trusted-host repo.huaweicloud.com
|
||||
pip install verl==0.5.0
|
||||
```
|
||||
|
||||
To ensure the VERL framework runs correctly on NPU, add the following lines to `verl/utils/vllm_utils.py`:
|
||||
|
||||
```python
|
||||
from vllm_ascend.patch import platform
|
||||
from vllm_ascend.patch import worker
|
||||
```
|
||||
|
||||
See the following reference for more details: [https://github.com/vllm-project/vllm-ascend/issues/1776](https://github.com/vllm-project/vllm-ascend/issues/1776).
|
||||
|
||||
After the above dependencies have been installed, from [`examples/spider`]({{ src("examples/spider") }}) run the following script command:
|
||||
|
||||
```bash
|
||||
python train_sql_agent.py npu
|
||||
```
|
||||
|
||||
### Debugging the Agent without VERL
|
||||
|
||||
[`sql_agent.py`]({{ src("examples/spider/sql_agent.py") }}) also provides a `debug_sql_agent()` helper to run the LangGraph workflow directly against a local or hosted OpenAI-compatible endpoint before using VERL.
|
||||
|
||||
Set the following environment variables, then execute the file:
|
||||
|
||||
```bash
|
||||
export OPENAI_API_BASE=<your_api_base>
|
||||
export OPENAI_API_KEY=<your_api_key>
|
||||
cd examples/spider
|
||||
python sql_agent.py
|
||||
```
|
||||
|
||||
This allows you to verify that the workflow and prompts behave as expected before reinforcement learning is introduced.
|
||||
|
||||
### Evaluation
|
||||
|
||||
The following results were obtained by running `python train_sql_agent.py qwen` on a single 80 GB GPU.
|
||||
Training completes in approximately **12 hours**.
|
||||
The training curves below are smoothed by aggregating every 16 steps for better visualization.
|
||||
|
||||
Additional evaluation results were collected with a legacy version — Agent-lightning v0.1.1, `verl==0.5.0`, and `vllm==0.10.0`.
|
||||
You can find them in this write-up:
|
||||
[Training AI Agents to Write and Self-Correct SQL with Reinforcement Learning](https://medium.com/@yugez/training-ai-agents-to-write-and-self-correct-sql-with-reinforcement-learning-571ed31281ad)
|
||||
|
||||
|
||||
<div style="height:400px">
|
||||
<canvas data-chart='{"type": "line", "data": {"labels": [0.0, 16.0, 32.0, 48.0, 64.0, 80.0, 96.0, 112.0, 128.0, 144.0, 160.0, 176.0, 192.0, 208.0, 224.0, 240.0, 256.0, 272.0, 288.0, 304.0, 320.0, 336.0, 352.0, 368.0, 384.0, 400.0, 416.0, 432.0], "datasets": [{"label": "Training", "data": [0.4609375, 0.5041666666666667, 0.5790441176470589, 0.6015625, 0.6070772058823529, 0.6208333333333333, 0.6668198529411765, 0.66875, 0.6709558823529411, 0.6708333333333333, 0.6847426470588235, 0.6791666666666667, 0.6819852941176471, 0.690625, 0.7008272058823529, 0.7453125, 0.7398897058823529, 0.7119791666666667, 0.7224264705882353, 0.7114583333333333, 0.7431066176470589, 0.7427083333333333, 0.75, 0.7302083333333333, 0.7247242647058824, 0.7390625, 0.7463235294117647, 0.7376302083333334], "spanGaps": true}, {"label": "Validation", "data": [0.342, null, 0.594, null, 0.642, null, 0.66, null, 0.676, null, 0.676, null, 0.694, null, 0.712, null, 0.702, null, 0.678, null, 0.702, null, 0.702, null, 0.674, null, 0.734, 0.722], "spanGaps": true}]}, "options": {"interaction": {"mode": "nearest", "intersect": false}, "plugins": {"legend": {"display": true, "position": "top"}, "title": {"display": true, "text": "SQL Agent Training Result (agent_match = write)"}}, "scales": {"x": {"title": {"display": true, "text": "Step (aggregated)"}}, "y": {"title": {"display": true, "text": "Accuracy"}}}}}'></canvas>
|
||||
</div>
|
||||
|
||||
<div style="height:400px">
|
||||
<canvas data-chart='{"type": "line", "data": {"labels": [0.0, 16.0, 32.0, 48.0, 64.0, 80.0, 96.0, 112.0, 128.0, 144.0, 160.0, 176.0, 192.0, 208.0, 224.0, 240.0, 256.0, 272.0, 288.0, 304.0, 320.0, 336.0, 352.0, 368.0, 384.0, 400.0, 416.0, 432.0], "datasets": [{"label": "Training", "data": [0.4560546875, 0.578125, 0.6167279411764706, 0.6401041666666667, 0.6461397058823529, 0.6598958333333333, 0.6838235294117647, 0.69375, 0.6916360294117647, 0.6833333333333333, 0.6893382352941176, 0.6921875, 0.6838235294117647, 0.70625, 0.7045036764705882, 0.7442708333333333, 0.7288602941176471, 0.7317708333333334, 0.7311580882352942, 0.7286458333333333, 0.7316176470588235, 0.7359375, 0.7366727941176471, 0.7208333333333333, 0.7118566176470589, 0.7296875, 0.7389705882352942, 0.7350260416666666], "spanGaps": true}, {"label": "Validation", "data": [0.33, null, 0.62, null, 0.662, null, 0.682, null, 0.696, null, 0.7, null, 0.708, null, 0.692, null, 0.72, null, 0.7, null, 0.7, null, 0.702, null, 0.694, null, 0.702, 0.682], "spanGaps": true}]}, "options": {"interaction": {"mode": "nearest", "intersect": false}, "plugins": {"legend": {"display": true, "position": "top"}, "title": {"display": true, "text": "SQL Agent Training Result (agent_match = null)"}}, "scales": {"x": {"title": {"display": true, "text": "Step (aggregated)"}}, "y": {"title": {"display": true, "text": "Value"}}}}}'></canvas>
|
||||
</div>
|
||||
@@ -0,0 +1,405 @@
|
||||
# Fine-tune with Unsloth SFT
|
||||
|
||||
!!! note "Prerequisites"
|
||||
|
||||
Please make sure you have read [Write the First Algorithm](./write-first-algorithm.md). Although that recipe is based on a simple prompt tuning algorithm, it introduces the core concepts of Agent-lightning and you should be familiar with them before proceeding.
|
||||
|
||||
This recipe builds on [Write the First Algorithm](./write-first-algorithm.md). Instead of iterating on a prompt, we will fine-tune a large language model with [Unsloth](https://docs.unsloth.ai/)'s SFT Trainer and keep the whole loop inside Agent-lightning. The new pieces you will meet are the **LLM proxy**, the **trace-to-triplet adapter**, a [vLLM](https://github.com/vllm-project/vllm) inference endpoint, and an agent implemented with the [OpenAI Agents SDK](https://openai.github.io/openai-agents-python/). The full sample code is available in the [`examples/unsloth`]({{ src("examples/unsloth") }}) folder.
|
||||
|
||||
!!! warning
|
||||
|
||||
You need a GPU that can host the Unsloth base model and run vLLM. The sample defaults to `unsloth/Qwen3-4B-Instruct-2507`, which requires at least 16GB of GPU memory under 4-bit quantization.
|
||||
|
||||
## The Data and Serving Loop
|
||||
|
||||
To tune a large language model in Supervised Fine-Tuning (SFT), we commonly need a dataset with input/output samples. For example, the [TRL SFT Trainer](https://huggingface.co/docs/trl/sft_trainer) expects a dataset with samples like the following:
|
||||
|
||||
```json
|
||||
{"messages": [{"role": "user", "content": "What color is the sky?"},
|
||||
{"role": "assistant", "content": "It is blue."}]}
|
||||
```
|
||||
|
||||
With supervised fine-tuning, the LLM learns to generate the "assistant" response as close as possible to the completion in the dataset.
|
||||
|
||||
Typically, the dataset used in SFT should be a curated set of samples. The samples can be either hand-written by humans, or generated by a more powerful model, which is known as [data distillation](https://docs.nvidia.com/nemo-framework/user-guide/24.12/modelalignment/knowledge-distillation.html). However, in this recipe, we use a different setup that relies on samples generated by the model itself. We use the reward emitted by the agent to select the top-performing samples.
|
||||
|
||||
Overall, the flow of the algorithm is an iteration of the following steps:
|
||||
|
||||
1. Serve the current checkpoint (with vLLM).
|
||||
2. Publish the vLLM endpoint through the LLM proxy and let runners roll out some tasks with the current model.
|
||||
3. Collect the traces from the rollouts and transform the highest-rewarded ones into a dataset that is acceptable for Unsloth SFT Trainer.
|
||||
4. Launch Unsloth to fine-tune on the dataset and save a new checkpoint.
|
||||
|
||||
You will find the full source code of this iteration in `sft_one_iter` in [sft_algorithm.py]({{ src("examples/unsloth/sft_algorithm.py") }}). We will elaborate on each part below.
|
||||
|
||||
### Serving the Model with vLLM and Proxy
|
||||
|
||||
Most modern agents do not use the model directly; instead, they use an API like the OpenAI chat completions API to interact with the model. Therefore, we need a vLLM-based inference server launched before rollouts. The serving code looks like the following. See the `vllm_server` function in [sft_algorithm.py]({{ src("examples/unsloth/sft_algorithm.py") }}) if you want to see a more robust version.
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
vllm_process = subprocess.Popen([
|
||||
"vllm", "serve", model_path, "--port", str(port),
|
||||
"--enable-auto-tool-choice", "--tool-call-parser", "hermes"
|
||||
])
|
||||
|
||||
# Wait for the server to be ready
|
||||
url = f"http://localhost:{port}/health"
|
||||
start = time.time()
|
||||
client = httpx.Client()
|
||||
|
||||
while True:
|
||||
if client.get(url).status_code == 200:
|
||||
break
|
||||
|
||||
server_address = f"http://localhost:{port}/v1"
|
||||
|
||||
# Try using the vLLM server
|
||||
openai = OpenAI(base_url=server_address)
|
||||
...
|
||||
```
|
||||
|
||||
In this recipe, we do not expose the server address directly to the agent runners, because we want to install a "middleware" to collect the prompts and responses of all the requests. In general, it's up to you to decide whether to hide the vLLM server behind a proxy or not.
|
||||
|
||||
The "middleware" here is [`LLMProxy`][agentlightning.LLMProxy], which is an independent [LiteLLM](https://docs.litellm.ai/) server that forwards the requests to the vLLM server. It also exposes an OpenAI-compatible API that the runners can target without caring about where the model lives. The benefits of using the proxy are:
|
||||
|
||||
1. **Traces:** The proxy automatically logs the prompts and responses of all the requests into the store.
|
||||
2. **Token IDs:** The proxy augments the requests so that the vLLM server can return the prompt and response token IDs (see more details in [Serving LLM](../deep-dive/serving-llm.md)).
|
||||
|
||||
The [`LLMProxy`][agentlightning.LLMProxy] accepts a list of model configurations, in the same syntax as LiteLLM's [`model_list`](https://docs.litellm.ai/docs/proxy/configs). Include a `hosted_vllm/` prefix to the models to activate LiteLLM's [vLLM integration](https://docs.litellm.ai/docs/providers/vllm).
|
||||
|
||||
```python
|
||||
import agentlightning as agl
|
||||
|
||||
llm_proxy = agl.LLMProxy(port=port, store=store)
|
||||
model_list = [
|
||||
{
|
||||
"model_name": "Qwen3-4B-Instruct",
|
||||
"litellm_params": {"model": f"hosted_vllm/{model_path}", "api_base": server_address},
|
||||
}
|
||||
]
|
||||
llm_proxy.update_model_list(model_list)
|
||||
# If the proxy is not running, it will start automatically.
|
||||
await llm_proxy.restart()
|
||||
# Add the proxy as a resource to the store so that the runners can access it via URL.
|
||||
resource_update = await store.add_resources({"main_llm": llm_proxy.as_resource()})
|
||||
```
|
||||
|
||||
### Spawn Rollout and Collect Spans
|
||||
|
||||
Once the proxy is registered as a resource, the algorithm schedules work for the rollout runners. Each problem from a training dataset becomes a rollout with the proxy baked into its resources:
|
||||
|
||||
```python
|
||||
rollouts: list[Rollout] = []
|
||||
for sample in train_dataset:
|
||||
rollouts.append(
|
||||
await store.enqueue_rollout(
|
||||
input=sample,
|
||||
mode="train",
|
||||
resources_id=resources_update.resources_id,
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
`resources_id` ties every rollout to the `main_llm` proxy resource we just uploaded. The runners on the other side poll the store ([`LitAgentRunner.iter()`][agentlightning.LitAgentRunner.iter]) and execute the agent for each rollout. On the algorithm side we wait for completions with a non-blocking polling loop:
|
||||
|
||||
```python
|
||||
completed_rollouts: list[Rollout] = []
|
||||
while True:
|
||||
completed_rollouts = await store.wait_for_rollouts(
|
||||
rollout_ids=[r.rollout_id for r in rollouts],
|
||||
timeout=0.0,
|
||||
)
|
||||
if len(completed_rollouts) == len(rollouts):
|
||||
break
|
||||
await asyncio.sleep(5.0)
|
||||
```
|
||||
|
||||
!!! note
|
||||
|
||||
The `timeout=0.0` is needed here because this example uses a [`LightningStoreClient`][agentlightning.LightningStoreClient], and `wait_for_rollouts` establishes an HTTP connection to that store. Currently, only non-blocking wait requests are supported, which avoids holding the store connection open.
|
||||
|
||||
Once the rollouts complete, we terminate the vLLM server to free up GPU memory.
|
||||
|
||||
```python
|
||||
vllm_process.terminate()
|
||||
vllm_process.join(timeout=10.0)
|
||||
```
|
||||
|
||||
### Adapt the Spans to HuggingFace Dataset
|
||||
|
||||
[`LlmProxyTraceToTriplet`][agentlightning.LlmProxyTraceToTriplet] converts the proxy’s spans (which might be dozens to hundreds per rollout) into [`Triplet`][agentlightning.Triplet] objects that contain prompt/response token IDs plus an optional reward. The adapter may return multiple triplets per rollout (one per chat-completion call). To bias training toward successful reasoning chains the algorithm walks the triplets in reverse order, keeps the most recent reward, and turns each prompt/response pair into Hugging Face dataset rows:
|
||||
|
||||
```python
|
||||
all_triplets = []
|
||||
data_adapter = agl.LlmProxyTraceToTriplet()
|
||||
|
||||
for rollout in completed_rollouts:
|
||||
spans = await store.query_spans(rollout.rollout_id, "latest")
|
||||
triplets = data_adapter.adapt(spans)
|
||||
|
||||
recent_reward = None
|
||||
for triplet in reversed(triplets):
|
||||
if triplet.reward is not None:
|
||||
recent_reward = triplet.reward
|
||||
if recent_reward is None:
|
||||
continue
|
||||
|
||||
input_ids = triplet.prompt["token_ids"] + triplet.response["token_ids"]
|
||||
# We don't train on prompt tokens, so they are masked out by setting to -100.
|
||||
labels = [-100] * len(triplet.prompt["token_ids"]) + triplet.response["token_ids"]
|
||||
# This matches the dataset format required by the Unsloth SFT trainer.
|
||||
all_triplets.append(
|
||||
{
|
||||
"input_ids": input_ids,
|
||||
"attention_mask": [1] * len(input_ids),
|
||||
"labels": labels,
|
||||
"reward": recent_reward,
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
!!! note
|
||||
|
||||
You might notice that the dataset format used here differs from the format described in the **SFT Trainer** documentation. According to the documentation, dataset samples should be provided as plain text strings or message objects.
|
||||
|
||||
As a matter of fact, this example leverages some [undocumented behavior](https://github.com/huggingface/trl/blob/e0eec055b412c48ad754149c475a87a8fca34fb4/trl/trainer/sft_trainer.py#L887) in the SFT Trainer implementation. When the dataset already includes a `"input_ids"` column, the Trainer automatically marks it as `is_processed` and skips the internal tokenization step.
|
||||
|
||||
Since we already have spans with token IDs generated by the [`LLMProxy`][agentlightning.LLMProxy], providing them directly avoids unnecessary [**re-tokenization** and related complications](../deep-dive/serving-llm.md). This approach will both save processing time and increase consistency between training and inference.
|
||||
|
||||
After aggregating every rollout we shuffle, sort by reward, and keep the top fraction (e.g., 50%) before shuffling again. The resulting list feeds directly into `datasets.Dataset.from_list`, which is the format Unsloth’s SFT trainer expects.
|
||||
|
||||
```python
|
||||
from datasets import Dataset as HuggingFaceDataset
|
||||
|
||||
random.shuffle(all_triplets)
|
||||
all_triplets.sort(key=lambda x: x["reward"], reverse=True)
|
||||
sliced_triplets = all_triplets[: max(1, int(len(all_triplets) * triplet_fraction))]
|
||||
# Shuffle the sliced triplets again
|
||||
random.shuffle(sliced_triplets)
|
||||
|
||||
sft_dataset = HuggingFaceDataset.from_list(sliced_triplets)
|
||||
```
|
||||
|
||||
### Launch Unsloth Training
|
||||
|
||||
The heavy lifting happens in [`trl.SFTTrainer`](https://huggingface.co/docs/trl/sft_trainer) (see [unsloth_helper.py]({{ src("examples/unsloth/unsloth_helper.py") }}) on how it's used). We launch it in a fresh process created with `multiprocessing.get_context("spawn")` so CUDA memory is reliably reclaimed when training ends. Launching it in the same process will also work for the first iteration, but we found that the memory won't be freed properly for subsequent vLLM serving.
|
||||
|
||||
```python
|
||||
context = multiprocessing.get_context("spawn")
|
||||
unsloth_process = context.Process(
|
||||
target=unsloth_training,
|
||||
args=(model_path, sft_dataset, next_model_path),
|
||||
daemon=True,
|
||||
)
|
||||
unsloth_process.start()
|
||||
unsloth_process.join(timeout=600.0)
|
||||
```
|
||||
|
||||
Inside the `unsloth_training` subprocess, Unsloth loads the previous checkpoint in 4-bit, applies LoRA adapters, and forwards the Hugging Face dataset to [`trl.SFTTrainer`](https://huggingface.co/docs/trl/sft_trainer) with the configuration defined in [`SFTConfig`](https://huggingface.co/docs/trl/sft_trainer#trl.SFTConfig) (batch size, accumulation steps, learning rate, etc.). The merged 16-bit weights are saved under `models/version_<iteration + 1>` so the next iteration can immediately serve them with vLLM.
|
||||
|
||||
```python
|
||||
from unsloth import FastLanguageModel
|
||||
# TRL is patched by unsloth.
|
||||
from trl import SFTConfig, SFTTrainer
|
||||
|
||||
model, tokenizer = FastLanguageModel.from_pretrained(
|
||||
model_name=model_path,
|
||||
load_in_4bit=True, # 4 bit quantization to reduce memory
|
||||
)
|
||||
|
||||
# Config the model to use LoRA
|
||||
model = FastLanguageModel.get_peft_model(
|
||||
model,
|
||||
r=32,
|
||||
...
|
||||
)
|
||||
|
||||
trainer = SFTTrainer(
|
||||
model=model,
|
||||
tokenizer=tokenizer,
|
||||
train_dataset=sft_dataset,
|
||||
...
|
||||
)
|
||||
|
||||
# This is the heaviest step.
|
||||
trainer_stats = trainer.train()
|
||||
|
||||
# Save in 16-bit for vLLM inference later
|
||||
model.save_pretrained_merged(next_model_path, tokenizer, save_method="merged_16bit")
|
||||
```
|
||||
|
||||
## Math Agent: OpenAI Agents SDK with MCP
|
||||
|
||||
We build an agent with the [OpenAI Agents SDK](https://openai.github.io/openai-agents-python/) to wire a calculator MCP tool and an OpenAI-compatible chat completion model together. The agent aims to solve a math problem and returns a reward indicating whether the answer is correct or not. The runner injects the `LLM` resource supplied by the algorithm side:
|
||||
|
||||
```python
|
||||
import os
|
||||
from typing import TypedDict
|
||||
|
||||
import agentlightning as agl
|
||||
from agents import Agent, ModelSettings, OpenAIChatCompletionsModel, Runner as OpenAIRunner
|
||||
from agents.mcp import MCPServerStdio
|
||||
from openai import AsyncOpenAI
|
||||
|
||||
class GsmProblem(TypedDict):
|
||||
input: str
|
||||
target: float
|
||||
|
||||
def compute_reward(result: str, target: float) -> float:
|
||||
...
|
||||
|
||||
@agl.rollout
|
||||
async def math_agent(task: GsmProblem, llm: agl.LLM) -> float:
|
||||
async with MCPServerStdio(
|
||||
name="Calculator via uvx",
|
||||
params={"command": "uvx", "args": ["mcp-server-calculator"]},
|
||||
) as server:
|
||||
agent = Agent(
|
||||
name="Assistant",
|
||||
instructions=(
|
||||
"Use the calculator tool for every question. "
|
||||
"Return only the numeric answer wrapped like ### <answer> ###."
|
||||
),
|
||||
mcp_servers=[server],
|
||||
model=OpenAIChatCompletionsModel(
|
||||
model=llm.model,
|
||||
openai_client=AsyncOpenAI(
|
||||
base_url=llm.endpoint,
|
||||
api_key=llm.api_key or "dummy",
|
||||
),
|
||||
),
|
||||
model_settings=ModelSettings(
|
||||
temperature=llm.sampling_parameters.get("temperature", 0.0),
|
||||
),
|
||||
)
|
||||
result = await OpenAIRunner.run(agent, task["input"])
|
||||
return compute_reward(result.final_output, task["target"])
|
||||
```
|
||||
|
||||
!!! tip
|
||||
|
||||
You can test the agent with a dry run:
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
|
||||
llm = agl.LLM(
|
||||
endpoint=os.environ["OPENAI_BASE_URL"],
|
||||
api_key=os.environ["OPENAI_API_KEY"],
|
||||
model="gpt-4.1-mini",
|
||||
)
|
||||
asyncio.run(math_agent({"input": "What is 1 + 1?", "target": 2.0}, llm))
|
||||
```
|
||||
|
||||
## Run this Recipe
|
||||
|
||||
The full runnable script for this recipe resides in [`examples/unsloth`]({{ src("examples/unsloth") }}) folder.
|
||||
|
||||
Before running this example, install `unsloth`, `vllm`, and the other libraries used in the examples (the project uses CUDA tooling, TRL, rich, datasets, etc.). We tested with `unsloth==2025.10.1`. `unsloth==2025.10.2` and `2025.10.3` are not working because of an [issue](https://github.com/unslothai/unsloth/issues/3451) we have been investigating with the unsloth team.
|
||||
|
||||
It's recommended to download the base model before running the example, such that the first iteration and subsequent iterations can both load from local checkpoints.
|
||||
|
||||
```bash
|
||||
hf download unsloth/Qwen3-4B-Instruct-2507 --local-dir models/version_0
|
||||
```
|
||||
|
||||
The repository already contains `examples/unsloth/data_gsmhard.jsonl` (which is a very small subset of the [GSM-hard math dataset](https://huggingface.co/datasets/reasoning-machines/gsm-hard) for demonstration purposes).
|
||||
|
||||
### Run Manually
|
||||
|
||||
Similar to the [Write the First Algorithm](./write-first-algorithm.md) recipe, you can open three terminals and start each component in parallel.
|
||||
|
||||
```bash
|
||||
agl store --port 4747
|
||||
python examples/unsloth/sft_rollout_runners.py
|
||||
python examples/unsloth/sft_algorithm.py
|
||||
```
|
||||
|
||||
In this case, [`sft_rollout_runners.py`]({{ src("examples/unsloth/sft_rollout_runners.py") }}) is a simple spawner implemented in Python that spawns 4 runners in parallel. The runners all connect to the same store server executing in another terminal.
|
||||
|
||||
```python
|
||||
import agentlightning as agl
|
||||
|
||||
def run_rollout(store: agl.LightningStore, worker_id: int) -> None:
|
||||
# Since the server side has already used LiteLLM proxy to collect traces,
|
||||
# a simple OtelTracer to collect the rewards is enough.
|
||||
tracer = agl.OtelTracer()
|
||||
|
||||
runner = agl.LitAgentRunner(tracer=tracer)
|
||||
|
||||
with runner.run_context(agent=math_agent, store=store, worker_id=worker_id):
|
||||
asyncio.run(runner.iter())
|
||||
|
||||
|
||||
def spawn_runners(store: agl.LightningStore, n_runners: int) -> None:
|
||||
runners = [
|
||||
multiprocessing.Process(target=run_rollout, args=(store, worker_id))
|
||||
for worker_id in range(n_runners)
|
||||
]
|
||||
for runner in runners:
|
||||
runner.start()
|
||||
|
||||
for runner in runners:
|
||||
runner.join()
|
||||
|
||||
|
||||
store = agl.LightningStoreClient("http://localhost:4747")
|
||||
spawn_runners(store=store, n_runners=4)
|
||||
```
|
||||
|
||||
!!! tip
|
||||
|
||||
Try to swap [`OtelTracer`][agentlightning.OtelTracer] in the runners with other tracers like [`AgentOpsTracer`][agentlightning.AgentOpsTracer]. Try to use a different adapter at the algorithm side such as [`TracerTraceToTriplet`][agentlightning.TracerTraceToTriplet] to see what happens.
|
||||
|
||||
### Run Everything with Trainer
|
||||
|
||||
We also show how to wrap everything into a single script using [`Trainer`][agentlightning.Trainer]. [`sft_allinone.py`]({{ src("examples/unsloth/sft_allinone.py") }}) wires the same components together, replacing the manual management of runners above.
|
||||
|
||||
```python
|
||||
class UnslothSupervisedFinetuning(agl.Algorithm):
|
||||
|
||||
async def run(
|
||||
self,
|
||||
train_dataset: Optional[Dataset[GsmProblem]] = None,
|
||||
val_dataset: Optional[Dataset[GsmProblem]] = None,
|
||||
):
|
||||
# Use the store, llm_proxy, and adapter from the trainer
|
||||
store = self.get_store()
|
||||
llm_proxy = self.get_llm_proxy()
|
||||
data_adapter = self.get_adapter()
|
||||
|
||||
for iteration in range(self.max_iterations):
|
||||
... # Same logic as sft_algorithm.py
|
||||
|
||||
algo = UnslothSupervisedFinetuning(
|
||||
max_iterations=2,
|
||||
vllm_port=12316,
|
||||
train_triplet_fraction=0.5,
|
||||
initial_model_path="models/version_0",
|
||||
)
|
||||
|
||||
# The LLM proxy can be created before Trainer
|
||||
trainer = Trainer(
|
||||
n_runners=4,
|
||||
algorithm=algo,
|
||||
llm_proxy=LLMProxy(port=12358),
|
||||
)
|
||||
|
||||
trainer.fit(math_agent, load_math_dataset())
|
||||
```
|
||||
|
||||
You might wonder where the initialization of [`Adapter`][agentlightning.Adapter] happens in this code. It turns out that [`TracerTraceToTriplet`][agentlightning.TracerTraceToTriplet] is the default adapter in [`Trainer`][agentlightning.Trainer], so we don't need to create one manually.
|
||||
|
||||
Now you can run the example with:
|
||||
|
||||
```bash
|
||||
python examples/unsloth/sft_allinone.py
|
||||
```
|
||||
|
||||
It starts an [`InMemoryLighningStore`][agentlightning.InMemoryLightningStore] for you, launches four worker processes, iterates the SFT loop, and prints the final checkpoint path when done. Adjust `max_iterations`, `train_triplet_fraction`, `n_runners`, or the proxy port to match your hardware or training goals. If you already run an external store or proxy you can also pass those objects into [`Trainer`][agentlightning.Trainer] instead of relying on the [Trainer-managed defaults][debug-with-external-store].
|
||||
|
||||
!!! info
|
||||
|
||||
As a future plan, we might graduate this example into a more powerful SFT algorithm bundled into [Algorithm Zoo](../algorithm-zoo/index.md). Currently, this `UnslothSupervisedFinetuning` is still for demo purposes.
|
||||
@@ -0,0 +1,279 @@
|
||||
# Write the First Algorithm with Agent-lightning
|
||||
|
||||
In the [first tutorial](./train-first-agent.md), "Train the First Agent," we introduced the [Trainer][agentlightning.Trainer] and showed how to use a pre-built algorithm like **Automatic Prompt Optimization (APO)** to improve an agent's performance. The [Trainer][agentlightning.Trainer] handled all the complex interactions, letting us focus on the agent's logic.
|
||||
|
||||
Now, we'll go a step deeper. What if you have a unique training idea that doesn't fit a standard algorithm? This tutorial will show you how to write your own custom algorithm from scratch. We'll build a simple algorithm that systematically tests a list of prompt templates and identifies the one with the highest reward.
|
||||
|
||||
By the end, you'll understand the core mechanics of how the **Algorithm**, **Runner**, and a new component, the **Store**, work together to create the powerful training loop at the heart of Agent-lightning.
|
||||
|
||||
!!! tip
|
||||
This tutorial helps you build a basic understanding of how to interact with Agent-lightning's core components. It's recommended that all users customizing algorithms should read this tutorial, even for those who are not planning to do prompt optimization.
|
||||
|
||||
## Core Concepts for Training
|
||||
|
||||
Before diving into the [LightningStore][agentlightning.LightningStore], let's define two key concepts that are central to any training process in Agent-lightning: **Resources** and the **Tracer**.
|
||||
|
||||
### Resources: The Tunable Assets
|
||||
|
||||
[](){ #introduction-to-resources }
|
||||
|
||||
**Resources** are the assets your algorithm is trying to improve. Think of them as the "recipe" an agent uses to perform its task. This recipe can be:
|
||||
|
||||
* A **prompt template** that guides an LLM.
|
||||
* The **weights** of a machine learning model.
|
||||
* Any other configuration or data your agent needs.
|
||||
|
||||
The algorithm's job is to run experiments and iteratively update these resources to find the best-performing version.
|
||||
|
||||
### Tracer: The Data Collector
|
||||
|
||||
How does the algorithm know if a change was an improvement? It needs data. This is where the **Tracer** comes in.
|
||||
|
||||
The Tracer automatically **instruments** (aka modifies / patches) the agent's code. This means it watches for important events, like an LLM call, a tool being used, or **reward signals**, and records a detailed log of what happened. Each of these logs is called a **Span** (which has already been introduced in the [last tutorial](./train-first-agent.md)).
|
||||
|
||||
A collection of spans from a single task execution gives the algorithm a complete, step-by-step trace of the agent's behavior, which is essential for learning and making improvements. Our default tracer is built on the AgentOps SDK to support instrumenting code written in various Agent/non-agent frameworks.
|
||||
|
||||
## The Central Hub: The LightningStore
|
||||
|
||||
Now, where do all these resources, tasks, and spans live? They are all managed by the **LightningStore**.
|
||||
|
||||
The LightningStore acts as the central database and message queue for the entire system. It's the single source of truth that decouples the Algorithm from the Runners.
|
||||
|
||||
!!! note
|
||||
|
||||
In the [last tutorial](./train-first-agent.md) we simplified the training loop, saying the Algorithm and Agent communicate "via the Trainer." That's true at a high level, but the component that makes it all possible is actually the **LightningStore**.
|
||||
|
||||
* The **Algorithm** connects to the Store to `enqueue_rollout` (tasks) and `update_resources` (like new prompt templates). It also queries the Store to retrieve the resulting spans and rewards from completed rollouts.
|
||||
* The **Runners** connect to the Store to `dequeue_rollout` (polling for available tasks). After executing a task, they use the `Tracer` to write the resulting spans and status updates back to the Store.
|
||||
|
||||
This architecture is key to Agent-lightning's scalability. Since the Algorithm and Runners only talk to the Store, they can run in different processes or even on different machines.
|
||||
|
||||
{ .center }
|
||||
|
||||
!!! tip "A Mental Model of What the Store Contains"
|
||||
|
||||
The [LightningStore][agentlightning.LightningStore] isn't just a simple database; it's an organized system for managing the entire training lifecycle. Here's what it keeps track of:
|
||||
|
||||
* **Task Queue**: A queue of pending **Rollouts** waiting for a Runner to pick them up, interactable via `enqueue_rollout` and `dequeue_rollout`.
|
||||
* **Rollouts**: The record of a single task. A rollout contains metadata about the task and tracks all **Attempts** to complete it, interactable via `query_rollouts` and `wait_for_rollouts`.
|
||||
* **Attempts**: A single execution of a rollout. If an attempt fails (e.g., due to a network error), the Store can automatically schedules a retry if it's configured. Each attempt is linked to its parent rollout and contains the status and timing information. The rollout status is [synced](../deep-dive/store.md) with its children's status. **For beginners, you can assume each rollout has only one attempt unless you have explicitly configure the retry.**
|
||||
* **Spans**: The detailed, structured logs generated by the `Tracer` during an attempt. Each span is linked to its parent attempt and rollout.
|
||||
* **Resources**: A versioned collection of the assets (like prompt templates) that the algorithm creates. Each rollout is linked to the specific version of the resources it should use.
|
||||
|
||||
## Building a Custom Algorithm
|
||||
|
||||
Let's build an algorithm that finds the best system prompt from a predefined list. The logic is straightforward:
|
||||
|
||||
1. Start with a list of candidate prompt templates.
|
||||
2. For each template, create a "resource" bundle in the Store.
|
||||
3. Enqueue a rollout (a task), telling the Runner to use this specific resource.
|
||||
4. Wait for a Runner to pick up the task and complete it.
|
||||
5. Query the Store to get the final reward from the rollout's spans.
|
||||
6. After testing all templates, compare the rewards and declare the best one.
|
||||
|
||||
We can implement this as a simple Python function that interacts directly with the [LightningStore][agentlightning.LightningStore].
|
||||
|
||||
```python
|
||||
async def find_best_prompt(store, prompts_to_test, task_input):
|
||||
"""A simple algorithm to find the best prompt from a list."""
|
||||
results = []
|
||||
|
||||
# Iterate through each prompt to test it
|
||||
for prompt in prompts_to_test:
|
||||
print(f"[Algo] Updating prompt template to: '{prompt}'")
|
||||
|
||||
# 1. Update the resources in the store with the new prompt
|
||||
resources_update = await store.add_resources(
|
||||
resources={"prompt_template": prompt}
|
||||
)
|
||||
|
||||
# 2. Enqueue a rollout task for a runner to execute
|
||||
print("[Algo] Queuing task for clients...")
|
||||
rollout = await store.enqueue_rollout(
|
||||
input=task_input,
|
||||
resources_id=resources_update.resources_id,
|
||||
)
|
||||
print(f"[Algo] Task '{rollout.rollout_id}' is now available for clients.")
|
||||
|
||||
# 3. Wait for the rollout to be completed by a runner
|
||||
await store.wait_for_rollouts([rollout.rollout_id])
|
||||
|
||||
# 4. Query the completed rollout and its spans
|
||||
completed_rollout = await store.get_rollout_by_id(rollout.rollout_id)
|
||||
print(f"[Algo] Received Result: {completed_rollout.model_dump_json(indent=None)}")
|
||||
|
||||
spans = await store.query_spans(rollout.rollout_id)
|
||||
# We expect at least two spans: one for the LLM call and one for the final reward
|
||||
print(f"[Algo] Queried Spans:\n - " + "\n - ".join(str(span) for span in spans))
|
||||
# find_final_reward is a helper function to extract the reward span
|
||||
final_reward = find_final_reward(spans)
|
||||
print(f"[Algo] Final reward: {final_reward}\n")
|
||||
|
||||
results.append((prompt, final_reward))
|
||||
|
||||
# 5. Find and print the best prompt based on the collected rewards
|
||||
print(f"[Algo] All prompts and their rewards: {results}")
|
||||
best_prompt, best_reward = max(results, key=lambda item: item[1])
|
||||
print(f"[Algo] Best prompt found: '{best_prompt}' with reward {best_reward}")
|
||||
```
|
||||
|
||||
!!! note "Asynchronous Operations"
|
||||
|
||||
You'll notice the `async` and `await` keywords. Agent-lightning is built on asyncio to handle concurrent operations efficiently. All interactions with the store are asynchronous network calls, so they must be awaited.
|
||||
|
||||
## The Agent and Runner
|
||||
|
||||
Our algorithm needs an **agent** to execute the tasks and a **runner** to manage the process.
|
||||
|
||||
The runner is a long-lived worker process. Its job is simple:
|
||||
|
||||
1. Connect to the [LightningStore][agentlightning.LightningStore] via a [LightningStoreClient][agentlightning.LightningStoreClient].
|
||||
2. Enter a loop, constantly asking the [LightningStore][agentlightning.LightningStore] for new tasks (`dequeue_rollout`).
|
||||
3. When it gets a task, it runs the `simple_agent` function.
|
||||
4. Crucially, the runner wraps the agent execution with a **Tracer**. The tracer automatically captures all the important events (like the LLM call and the final reward) as spans and sends them back to the [LightningStore][agentlightning.LightningStore].
|
||||
|
||||
```python
|
||||
# Connecting to Store
|
||||
store = agl.LightningStoreClient("http://localhost:4747") # or some other address
|
||||
runner = LitAgentRunner[str](tracer=AgentOpsTracer())
|
||||
with runner.run_context(agent=simple_agent, store=store): # <-- where the wrapping and instrumentation happens
|
||||
await runner.iter() # polling for new tasks forever
|
||||
```
|
||||
|
||||
For this example, the agent's job is to take the prompt from the resources, use it to ask an LLM a question, and return a score.
|
||||
|
||||
```python
|
||||
def simple_agent(task: str, prompt_template: PromptTemplate) -> float:
|
||||
"""An agent that answers a question and gets judged by an LLM."""
|
||||
client = OpenAI()
|
||||
|
||||
# Generate a response using the provided prompt template
|
||||
prompt = prompt_template.format(any_question=task)
|
||||
response = client.chat.completions.create(
|
||||
model="gpt-4.1-nano", messages=[{"role": "user", "content": prompt}]
|
||||
)
|
||||
llm_output = response.choices[0].message.content
|
||||
print(f"[Rollout] LLM returned: {llm_output}")
|
||||
|
||||
# This llm_output and the final score are automatically logged as spans by the Tracer
|
||||
score = random.uniform(0, 1) # Replace with actual scoring logic if needed
|
||||
return score
|
||||
```
|
||||
|
||||
## Running the Example
|
||||
|
||||
To see everything in action, you'll need three separate terminal windows.
|
||||
|
||||
!!! tip
|
||||
|
||||
If you want to follow along, you can find the complete code for this example in the [apo_custom_algorithm.py]({{ src("examples/apo/apo_custom_algorithm.py") }}) file.
|
||||
|
||||
**1. Start the Store:**
|
||||
In the first terminal, start the LightningStore server. This component will wait for connections from the algorithm and the runner. The store will be listening on port `4747` ⚡ by default.
|
||||
|
||||
```bash
|
||||
agl store
|
||||
```
|
||||
|
||||
**2. Start the Runner:**
|
||||
In the second terminal, start the runner process. It will connect to the store and wait for tasks.
|
||||
|
||||
The code to start the runner looks like the following:
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY=sk-... # Your OpenAI API key
|
||||
python apo_custom_algorithm.py runner
|
||||
```
|
||||
|
||||
You will see output indicating the runner has started and is waiting for rollouts.
|
||||
|
||||
```text
|
||||
2025-10-14 22:23:41,339 [INFO] ... [Worker 0] Setting up tracer...
|
||||
2025-10-14 22:23:41,343 [INFO] ... [Worker 0] Instrumentation applied.
|
||||
2025-10-14 22:23:41,494 [INFO] ... [Worker 0] AgentOps client initialized.
|
||||
2025-10-14 22:23:41,494 [INFO] ... [Worker 0] Started async rollouts (max: unlimited).
|
||||
```
|
||||
|
||||
**3. Start the Algorithm:**
|
||||
In the third terminal, run the algorithm. This will kick off the entire process.
|
||||
|
||||
For example, we run the algorithm code shown above with the following parameters:
|
||||
|
||||
```python
|
||||
prompts_to_test = [
|
||||
"You are a helpful assistant. {any_question}",
|
||||
"You are a knowledgeable AI. {any_question}",
|
||||
"You are a friendly chatbot. {any_question}",
|
||||
]
|
||||
task_input = "Why is the sky blue?"
|
||||
store = agl.LightningStoreClient("http://localhost:4747")
|
||||
find_best_prompt(store, prompts_to_test, task_input)
|
||||
```
|
||||
|
||||
Or you can simply use our pre-written script to try out:
|
||||
|
||||
```bash
|
||||
python apo_custom_algorithm.py algo
|
||||
```
|
||||
|
||||
### Understanding the Output
|
||||
|
||||
As the algorithm runs, you'll see logs appear across all three terminals, showing the components interacting in real-time.
|
||||
|
||||
**Algorithm Output:**
|
||||
The algorithm terminal shows the main control flow: updating prompts, queuing tasks, and receiving the final results. You can also see the raw span data it retrieves from the store.
|
||||
|
||||
```text
|
||||
[Algo] Updating prompt template to: 'You are a helpful assistant. {any_question}'
|
||||
[Algo] Queuing task for clients...
|
||||
[Algo] Task 'ro-1d18988581cd' is now available for clients.
|
||||
[Algo] Received Result: rollout_id='ro-1d18988581cd' ... status='succeeded' ...
|
||||
[Algo] Queried Spans:
|
||||
- Span(name='openai.chat.completion', attributes={'gen_ai.prompt.0.content': 'You are a helpful assistant...', 'gen_ai.completion.0.content': 'The sky appears blue...'})
|
||||
- Span(name='reward', attributes={'value': 0.95})
|
||||
[Algo] Final reward: 0.95
|
||||
|
||||
[Algo] Updating prompt template to: 'You are a knowledgeable AI. {any_question}'
|
||||
...
|
||||
[Algo] Final reward: 0.95
|
||||
|
||||
[Algo] Updating prompt template to: 'You are a friendly chatbot. {any_question}'
|
||||
...
|
||||
[Algo] Final reward: 1.0
|
||||
|
||||
[Algo] All prompts and their rewards: [('You are a helpful assistant. {any_question}', 0.95), ('You are a knowledgeable AI. {any_question}', 0.95), ('You are a friendly chatbot. {any_question}', 1.0)]
|
||||
[Algo] Best prompt found: 'You are a friendly chatbot. {any_question}' with reward 1.0
|
||||
```
|
||||
|
||||
**Runner Output:**
|
||||
The runner terminal shows it picking up each task, executing the agent logic, and reporting the completion.
|
||||
|
||||
```text
|
||||
[Rollout] LLM returned: The sky appears blue due to Rayleigh scattering...
|
||||
2025-10-14 22:25:50,803 [INFO] ... [Worker 0 | Rollout ro-a9f54ac19af5] Completed in 4.24s. ...
|
||||
|
||||
[Rollout] LLM returned: The sky looks blue because of a process called Rayleigh scattering...
|
||||
2025-10-14 22:25:59,863 [INFO] ... [Worker 0 | Rollout ro-c67eaa9016b6] Completed in 4.06s. ...
|
||||
```
|
||||
|
||||
**Store Server Output:**
|
||||
The store terminal shows a detailed log of every interaction, confirming its role as the central hub. You can see requests to enqueue and dequeue rollouts, add spans, and update statuses.
|
||||
|
||||
```text
|
||||
... "POST /enqueue_rollout HTTP/1.1" 200 ...
|
||||
... "GET /dequeue_rollout HTTP/1.1" 200 ...
|
||||
... "POST /add_span HTTP/1.1" 200 ...
|
||||
... "POST /update_attempt HTTP/1.1" 200 ...
|
||||
... "POST /wait_for_rollouts HTTP/1.1" 200 ...
|
||||
... "GET /query_spans/ro-c67eaa9016b6 HTTP/1.1" 200 ...
|
||||
```
|
||||
|
||||
!!! info "So Where is Trainer?"
|
||||
|
||||
You might be wondering why the [last tutorial](./train-first-agent.md) focused on the [Trainer][agentlightning.Trainer] class, but we haven't used it here.
|
||||
|
||||
Think of the [Trainer][agentlightning.Trainer] as a convenient wrapper that manages the entire training process for you. It's perfect when you want to apply a pre-built algorithm to your agent without worrying about the underlying mechanics. The [Trainer][agentlightning.Trainer] handles starting the [LightningStore][agentlightning.LightningStore], coordinating the [Runners][agentlightning.Runner], managing their lifecycles, and handling errors.
|
||||
|
||||
In this tutorial, however, our goal is to *build a new algorithm*. To do that, we need to interact directly with the core components: the [Store][agentlightning.LightningStore], the [Runner][agentlightning.Runner], and the algorithm logic itself. Running them separately gives you more control and clearer, isolated logs, which is ideal for development and debugging.
|
||||
|
||||
Once your custom algorithm is mature, you can package it to comply with our standard interface ([@algo][agentlightning.algo] or [Algorithm][agentlightning.Algorithm]). This allows you to use it with the [Trainer][agentlightning.Trainer] again, getting all the benefits of automated lifecycle management while using your own custom logic. A sample code doing this is available in [apo_custom_algorithm_trainer.py]({{ src("examples/apo/apo_custom_algorithm_trainer.py") }}).
|
||||
@@ -0,0 +1,59 @@
|
||||
# Agent Lightning
|
||||
|
||||
Agent Lightning is the absolute trainer to light up AI agents.
|
||||
|
||||
[Join our Discord community](https://discord.gg/RYk7CdvDR7) to connect with other users and contributors.
|
||||
|
||||
## Features
|
||||
|
||||
- Turn your agent into an optimizable beast with **ZERO CODE CHANGE** (almost)! 💤
|
||||
- Build with **ANY** agent framework (LangChain, OpenAI Agent SDK, AutoGen, CrewAI, Microsoft Agent Framework...); or even WITHOUT agent framework (Python OpenAI). You name it! 🤖
|
||||
- **Selectively** optimize one or more agents in a multi-agent system. 🎯
|
||||
- Embraces **Algorithms** like Reinforcement Learning, Automatic Prompt Optimization, Supervised Fine-tuning and more. 🤗
|
||||
|
||||
## How to Read this Documentation
|
||||
|
||||
This documentation is organized into the following parts:
|
||||
|
||||
- [Installation](tutorials/installation.md) - Get started with Agent Lightning
|
||||
- How-to Recipes (e.g., [Train SQL Agent with RL](how-to/train-sql-agent.md)) - Practical examples of training agents and customizing algorithms.
|
||||
- Learning More (e.g., [Debugging](tutorials/debug.md)) - Guides on specific topics like debugging or parallelization.
|
||||
- Algorithm Zoo (e.g., [APO](algorithm-zoo/apo.md)) - References for built-in algorithms.
|
||||
- Deep Dive (e.g., [Bird's Eye View](deep-dive/birds-eye-view.md)) - For a deeper understanding of what Agent-lightning is doing under the hood.
|
||||
- API References (e.g., [Agent](reference/agent.md)) - References for the Agent-lightning Python API.
|
||||
|
||||
## Resources
|
||||
|
||||
- 11/4/2025 [Tuning ANY AI agent with Tinker ✕ Agent-lightning](https://medium.com/@yugez/tuning-any-ai-agent-with-tinker-agent-lightning-part-1-1d8c9a397f0e) Medium. See also [Part 2](https://medium.com/@yugez/tuning-any-ai-agent-with-tinker-agent-lightning-part-2-332c5437f0dc).
|
||||
- 10/22/2025 [No More Retokenization Drift: Returning Token IDs via the OpenAI Compatible API Matters in Agent RL](https://blog.vllm.ai/2025/10/22/agent-lightning.html) vLLM blog. See also [Zhihu writeup](https://zhuanlan.zhihu.com/p/1965067274642785725).
|
||||
- 8/11/2025 [Training AI Agents to Write and Self-correct SQL with Reinforcement Learning](https://medium.com/@yugez/training-ai-agents-to-write-and-self-correct-sql-with-reinforcement-learning-571ed31281ad) Medium.
|
||||
- 8/5/2025 [Agent Lightning: Train ANY AI Agents with Reinforcement Learning](https://arxiv.org/abs/2508.03680) arXiv paper.
|
||||
- 7/26/2025 [We discovered an approach to train any AI agent with RL, with (almost) zero code changes.](https://www.reddit.com/r/LocalLLaMA/comments/1m9m670/we_discovered_an_approach_to_train_any_ai_agent/) Reddit.
|
||||
- 6/6/2025 [Agent Lightning - Microsoft Research](https://www.microsoft.com/en-us/research/project/agent-lightning/) Project page.
|
||||
|
||||
## Community Projects
|
||||
|
||||
- [DeepWerewolf](https://github.com/af-74413592/DeepWerewolf) — A case study of agent RL training for the Chinese Werewolf game built with AgentScope and Agent Lightning.
|
||||
- [AgentFlow](https://agentflow.stanford.edu/) — A modular multi-agent framework that combines planner, executor, verifier, and generator agents with the Flow-GRPO algorithm to tackle long-horizon, sparse-reward tasks.
|
||||
- [Youtu-Agent](https://github.com/TencentCloudADP/Youtu-agent) — Youtu-Agent lets you build and train your agent with ease. Built with [a modified branch](https://github.com/microsoft/agent-lightning/tree/contrib/youtu-agent-lightning) of Agent Lightning, Youtu-Agent has verified up to 128 GPUs RL training on maths/code and search capabilities with steady convergence. Also check [the recipe](https://github.com/TencentCloudADP/youtu-agent/tree/rl/agl) and their blog [*Stop Wrestling with Your Agent RL: How Youtu-Agent Achieved Stable, 128-GPU Scaling Without Breaking a Sweat*](https://spotted-coconut-df8.notion.site/Stop-Wrestling-with-Your-Agent-RL-How-Youtu-Agent-Achieved-Stable-128-GPU-Scaling-Without-Breaking-2ca5e8f089ba80539a98c582b65e0233).
|
||||
|
||||
|
||||
## Citation
|
||||
|
||||
If you find Agent Lightning useful in your research or projects, please cite our paper:
|
||||
|
||||
```bibtex
|
||||
@misc{luo2025agentlightningtrainai,
|
||||
title={Agent Lightning: Train ANY AI Agents with Reinforcement Learning},
|
||||
author={Xufang Luo and Yuge Zhang and Zhiyuan He and Zilong Wang and Siyun Zhao and Dongsheng Li and Luna K. Qiu and Yuqing Yang},
|
||||
year={2025},
|
||||
eprint={2508.03680},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.AI},
|
||||
url={https://arxiv.org/abs/2508.03680},
|
||||
}
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
See the [LICENSE](https://github.com/microsoft/agent-lightning/blob/main/LICENSE) file for details.
|
||||
@@ -0,0 +1,225 @@
|
||||
// Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
// ---- CSS helpers ---------------------------------------------------------
|
||||
function matVar(name) {
|
||||
return getComputedStyle(document.body).getPropertyValue(name).trim();
|
||||
}
|
||||
function toRGBA(color, a = 1) {
|
||||
if (!color) return `rgba(0,0,0,${a})`;
|
||||
const m = color.match(/^#?([\da-f]{3}|[\da-f]{6})$/i);
|
||||
if (m) {
|
||||
const hex = m[1].length === 3 ? m[1].split("").map((x) => x + x).join("") : m[1];
|
||||
const r = parseInt(hex.slice(0, 2), 16);
|
||||
const g = parseInt(hex.slice(2, 4), 16);
|
||||
const b = parseInt(hex.slice(4, 6), 16);
|
||||
return `rgba(${r}, ${g}, ${b}, ${a})`;
|
||||
}
|
||||
const nums = color.match(/[\d.]+/g) || [0, 0, 0, 1];
|
||||
const [r, g, b] = nums.map(Number);
|
||||
return `rgba(${r | 0}, ${g | 0}, ${b | 0}, ${a})`;
|
||||
}
|
||||
|
||||
// ---- Theme defaults (pulled from MkDocs Material CSS vars) ---------------
|
||||
function applyThemeDefaults() {
|
||||
const font = matVar("--md-text-font").replace(/['"]/g, "") || "Roboto, sans-serif";
|
||||
const text = matVar("--md-default-fg-color") || "#1f2937";
|
||||
const border = "rgba(128, 128, 128, 0.1)"
|
||||
const bg = "#777777";
|
||||
|
||||
Chart.defaults.font.family = font;
|
||||
Chart.defaults.font.size = 16;
|
||||
Chart.defaults.color = text;
|
||||
Chart.defaults.borderColor = border;
|
||||
Chart.defaults.backgroundColor = bg;
|
||||
|
||||
Chart.defaults.scale.grid.color = border;
|
||||
Chart.defaults.scale.ticks.color = text;
|
||||
Chart.defaults.scale.title.color = text;
|
||||
|
||||
Chart.defaults.plugins.legend.labels.color = text;
|
||||
Chart.defaults.plugins.tooltip.titleColor = text;
|
||||
Chart.defaults.plugins.tooltip.bodyColor = text;
|
||||
Chart.defaults.plugins.tooltip.backgroundColor = toRGBA(bg, 0.5);
|
||||
Chart.defaults.plugins.tooltip.borderColor = border;
|
||||
Chart.defaults.plugins.tooltip.borderWidth = 1;
|
||||
|
||||
Chart.defaults.responsive = true;
|
||||
Chart.defaults.maintainAspectRatio = false;
|
||||
if (window.matchMedia("(prefers-reduced-motion: reduce)").matches) {
|
||||
Chart.defaults.animation = false;
|
||||
}
|
||||
}
|
||||
|
||||
// ---- Dataset color defaults (Material primary/accent) --------------------
|
||||
const colorScheme = ["#c45259", "#5276c4", "#f69047", "#7cc452", "#c2b00a"];
|
||||
|
||||
function applyDatasetDefaults(config) {
|
||||
if (!config.data || !Array.isArray(config.data.datasets)) return;
|
||||
config.data.datasets = config.data.datasets.map((ds, index) => {
|
||||
const color = colorScheme[index % colorScheme.length];
|
||||
return {
|
||||
...ds,
|
||||
borderColor: toRGBA(color, 0.8),
|
||||
backgroundColor: toRGBA(color, 0.3),
|
||||
pointBackgroundColor: color,
|
||||
pointBorderColor: color,
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
// ---- Deep merge (config JSON + our defaults) ----------------------------
|
||||
function deepMerge(target, src) {
|
||||
if (!src || typeof src !== "object") return target;
|
||||
for (const k of Object.keys(src)) {
|
||||
const v = src[k];
|
||||
if (v && typeof v === "object" && !Array.isArray(v)) {
|
||||
target[k] = deepMerge(target[k] || {}, v);
|
||||
} else {
|
||||
target[k] = v;
|
||||
}
|
||||
}
|
||||
return target;
|
||||
}
|
||||
|
||||
// ---- Build final config for a canvas ------------------------------------
|
||||
function buildConfig(baseCfg) {
|
||||
const globalDefaults = {
|
||||
options: {
|
||||
responsive: true,
|
||||
maintainAspectRatio: false,
|
||||
interaction: { mode: "index", intersect: false },
|
||||
plugins: {
|
||||
legend: { position: "top" },
|
||||
tooltip: { enabled: true },
|
||||
},
|
||||
layout: { padding: { top: 8, right: 8, bottom: 0, left: 0 } },
|
||||
normalized: true,
|
||||
alignToPixels: true,
|
||||
animations: {
|
||||
y: {
|
||||
from: (ctx) => 300,
|
||||
duration: 1500,
|
||||
easing: "easeOutCubic",
|
||||
},
|
||||
radius: {
|
||||
from: 0,
|
||||
to: 3,
|
||||
duration: 300,
|
||||
delay: (ctx) => ctx.dataIndex * 30,
|
||||
},
|
||||
},
|
||||
elements: { line: { tension: 0.3 } },
|
||||
},
|
||||
};
|
||||
const merged = deepMerge({}, globalDefaults);
|
||||
applyDatasetDefaults(baseCfg);
|
||||
deepMerge(merged, baseCfg); // user config wins
|
||||
return merged;
|
||||
}
|
||||
|
||||
(function () {
|
||||
// registry stores per-canvas state: { chart, cfg }
|
||||
const registry = new WeakMap(); // canvas -> { chart, cfg }
|
||||
|
||||
// IntersectionObserver to (re)animate when visible
|
||||
const io = new IntersectionObserver(
|
||||
(entries) => {
|
||||
entries.forEach((entry) => {
|
||||
if (!entry.isIntersecting) return;
|
||||
|
||||
const canvas = entry.target;
|
||||
const state = registry.get(canvas);
|
||||
if (!state || !state.cfg) return;
|
||||
|
||||
// Respect reduced-motion: if disabled, just update without animation
|
||||
const prefersReduced =
|
||||
window.matchMedia("(prefers-reduced-motion: reduce)").matches;
|
||||
|
||||
// Destroy & rebuild to guarantee a fresh animation
|
||||
if (state.chart) {
|
||||
try {
|
||||
state.chart.destroy();
|
||||
} catch (_) {}
|
||||
}
|
||||
const ctx = canvas.getContext("2d");
|
||||
const cfg = buildConfig(JSON.parse(JSON.stringify(state.cfg)));
|
||||
|
||||
// If reduced motion, skip animations
|
||||
if (prefersReduced) {
|
||||
cfg.options = cfg.options || {};
|
||||
cfg.options.animation = false;
|
||||
}
|
||||
|
||||
state.chart = new Chart(ctx, cfg);
|
||||
registry.set(canvas, state);
|
||||
});
|
||||
},
|
||||
{ threshold: 0.3 } // animate when ~30% visible
|
||||
);
|
||||
|
||||
// ---- Render all canvases with data-chart JSON ---------------------------
|
||||
function renderAll() {
|
||||
document.querySelectorAll("canvas[data-chart]").forEach((canvas) => {
|
||||
let parsedCfg;
|
||||
try {
|
||||
parsedCfg = JSON.parse(canvas.getAttribute("data-chart"));
|
||||
} catch (e) {
|
||||
console.error("Invalid data-chart JSON:", e, canvas);
|
||||
return;
|
||||
}
|
||||
|
||||
// store config; chart will be created by IntersectionObserver when visible
|
||||
if (!registry.get(canvas)) {
|
||||
registry.set(canvas, { chart: null, cfg: parsedCfg });
|
||||
io.observe(canvas);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// ---- Retheme on scheme/primary/accent change ----------------------------
|
||||
function retheme() {
|
||||
applyThemeDefaults();
|
||||
// Update visible charts without forcing animation
|
||||
document.querySelectorAll("canvas[data-chart]").forEach((c) => {
|
||||
const state = registry.get(c);
|
||||
if (state?.chart) {
|
||||
// Fix the issue that scale options color are not updated when theme changes
|
||||
const scaleOptions = state.chart.options.scales;
|
||||
for (const key of Object.keys(scaleOptions)) {
|
||||
scaleOptions[key].ticks.color = Chart?.defaults?.scale?.ticks?.color;
|
||||
scaleOptions[key].title.color = Chart?.defaults?.scale?.title?.color;
|
||||
}
|
||||
state.chart.update("none");
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// Initial theme + render (works on hard refresh)
|
||||
function boot() {
|
||||
applyThemeDefaults();
|
||||
renderAll();
|
||||
}
|
||||
|
||||
if (document.readyState === "loading") {
|
||||
document.addEventListener("DOMContentLoaded", boot);
|
||||
} else {
|
||||
boot();
|
||||
}
|
||||
|
||||
// Observe theme flips. Attributes might be on <html> or <body>.
|
||||
const attrs = ["data-md-color-scheme", "data-md-color-primary", "data-md-color-accent"];
|
||||
const obs = new MutationObserver(retheme);
|
||||
obs.observe(document.documentElement, {
|
||||
attributes: true,
|
||||
attributeFilter: attrs,
|
||||
subtree: true,
|
||||
});
|
||||
|
||||
// Re-scan on SPA navigations (Material)
|
||||
if (typeof document$ !== "undefined" && document$.subscribe) {
|
||||
document$.subscribe(() => {
|
||||
renderAll(); // new canvases
|
||||
retheme(); // keep colors in sync
|
||||
});
|
||||
}
|
||||
})();
|
||||
@@ -0,0 +1,12 @@
|
||||
// Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
document$.subscribe(({ body }) => {
|
||||
renderMathInElement(body, {
|
||||
delimiters: [
|
||||
{ left: "$$", right: "$$", display: true },
|
||||
{ left: "$", right: "$", display: false },
|
||||
{ left: "\\(", right: "\\)", display: false },
|
||||
{ left: "\\[", right: "\\]", display: true },
|
||||
],
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,11 @@
|
||||
// Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
document.addEventListener('DOMContentLoaded', function () {
|
||||
const container = document.querySelector('.md-content__inner');
|
||||
if (!container) return;
|
||||
const firstH1 = container.querySelector('h1');
|
||||
const meta = container.querySelector('.md-source-file'); // the block rendered by source-file.html
|
||||
if (firstH1 && meta && meta.previousElementSibling !== firstH1) {
|
||||
firstH1.insertAdjacentElement('afterend', meta);
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,72 @@
|
||||
# Copyright (c) Microsoft. All rights reserved.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import html
|
||||
import os
|
||||
from typing import Any, Dict
|
||||
|
||||
|
||||
def define_env(env: Any):
|
||||
"""Expose {{ src('path/to/file.py') }} to link to a file in the repo.
|
||||
|
||||
Behavior:
|
||||
|
||||
- Builds URL from repo_url + extra.source_commit in mkdocs.yml
|
||||
- Verifies that the file exists at build time
|
||||
- If file missing: logs a WARNING and returns a visible marker.
|
||||
With `mkdocs build --strict`, warnings become errors → build fails.
|
||||
|
||||
Examples:
|
||||
```
|
||||
[`apo_debug.py`]({{ src('examples/apo/apo_debug.py') }})
|
||||
```
|
||||
|
||||
or
|
||||
|
||||
```
|
||||
{{ src('examples/apo/apo_debug.py') }}
|
||||
```
|
||||
"""
|
||||
|
||||
cfg: Dict[str, Any] = env.conf or {}
|
||||
repo_url = cfg.get("repo_url", "").rstrip("/")
|
||||
extra: Dict[str, Any] = cfg.get("extra", {}) or {}
|
||||
default_commit = extra.get("source_commit", "main")
|
||||
project_dir = env.project_dir
|
||||
|
||||
if not repo_url:
|
||||
raise RuntimeError("repo_url must be set in mkdocs.yml for src() macro to work.")
|
||||
|
||||
logger = getattr(env, "logger", None)
|
||||
|
||||
def _warn(msg: str):
|
||||
if logger and hasattr(logger, "warning"):
|
||||
logger.warning(f"[macros:src] {msg}")
|
||||
else:
|
||||
print(f"[macros:src] WARNING: {msg}")
|
||||
|
||||
def src(path: str, text: str | None = None, commit: str | None = None) -> str:
|
||||
commit = commit or default_commit
|
||||
abs_root = os.path.abspath(project_dir)
|
||||
abs_path = os.path.abspath(os.path.join(project_dir, path))
|
||||
|
||||
# Prevent escaping project root
|
||||
if not abs_path.startswith(abs_root + os.sep) and abs_path != abs_root:
|
||||
raise ValueError(f"Invalid path outside project: {path}")
|
||||
|
||||
# Build the GitHub tree URL (folder or file both work for our use case)
|
||||
url = f"{repo_url}/tree/{commit}/{path}"
|
||||
|
||||
if not os.path.exists(abs_path):
|
||||
_warn(f"Source path not found: {path}. Rendering a visible broken-link marker.")
|
||||
label = html.escape(text or path)
|
||||
return (
|
||||
f'<span class="broken-source-link" title="Missing: {html.escape(url)}">' f"{label} (missing)" f"</span>"
|
||||
)
|
||||
|
||||
if text:
|
||||
return f"[{text}]({url})"
|
||||
return url
|
||||
|
||||
env.macro(src)
|
||||
@@ -0,0 +1,26 @@
|
||||
{% extends "base.html" %}
|
||||
|
||||
{% block content %}
|
||||
<div id="version-warning" class="version-warning" style="display: none;">
|
||||
⚠️ You are viewing development documentation. For stable documentation, please visit
|
||||
<a href="https://microsoft.github.io/agent-lightning/stable/">https://microsoft.github.io/agent-lightning/stable/</a>
|
||||
</div>
|
||||
{{ super() }}
|
||||
{% endblock %}
|
||||
|
||||
{% block scripts %}
|
||||
{{ super() }}
|
||||
<script>
|
||||
// Show warning banner if not viewing stable documentation
|
||||
document.addEventListener("DOMContentLoaded", function() {
|
||||
const currentURL = window.location.href;
|
||||
const warningDiv = document.getElementById("version-warning");
|
||||
|
||||
// Check if we're on the stable docs or the redirect page
|
||||
if (currentURL.includes('agent-lightning/latest/')) {
|
||||
// Show warning for explicitly latest docs
|
||||
warningDiv.style.display = "block";
|
||||
}
|
||||
});
|
||||
</script>
|
||||
{% endblock %}
|
||||
@@ -0,0 +1,27 @@
|
||||
<!-- Copied and modified from https://github.com/squidfunk/mkdocs-material/blob/master/src/templates/partials/content.html -->
|
||||
|
||||
<!-- Tags -->
|
||||
{% include "partials/tags.html" %}
|
||||
|
||||
<!-- Actions -->
|
||||
{% include "partials/actions.html" %}
|
||||
|
||||
<!--
|
||||
Hack: check whether the content contains a h1 headline. If it doesn't, the
|
||||
page title (or respectively site name) is used as the main headline.
|
||||
-->
|
||||
{% if "\u003ch1" not in page.content %}
|
||||
<h1>{{ page.title | d(config.site_name, true)}}</h1>
|
||||
{% endif %}
|
||||
|
||||
<!-- Source file information -->
|
||||
{% include "partials/source-file.html" %}
|
||||
|
||||
<!-- Page content -->
|
||||
{{ page.content }}
|
||||
|
||||
<!-- Was this page helpful? -->
|
||||
{% include "partials/feedback.html" %}
|
||||
|
||||
<!-- Comment system -->
|
||||
{% include "partials/comments.html" %}
|
||||
@@ -0,0 +1,51 @@
|
||||
# Agent Developer APIs
|
||||
|
||||
## Agent Decorators
|
||||
|
||||
!!! tip
|
||||
|
||||
These are convenient helpers for creating agents from functions. First-time users are recommended to use these decorators to create agents.
|
||||
|
||||
::: agentlightning.rollout
|
||||
|
||||
!!! warning
|
||||
|
||||
The following two decorators are implementations of [`agentlightning.rollout`][agentlightning.rollout]. They are not recommended for new users.
|
||||
|
||||
::: agentlightning.llm_rollout
|
||||
|
||||
::: agentlightning.prompt_rollout
|
||||
|
||||
## Class-based Agents
|
||||
|
||||
::: agentlightning.LitAgent
|
||||
|
||||
## Emitter
|
||||
|
||||
::: agentlightning.operation
|
||||
|
||||
::: agentlightning.emit_annotation
|
||||
|
||||
::: agentlightning.emit_reward
|
||||
|
||||
::: agentlightning.emit_message
|
||||
|
||||
::: agentlightning.emit_object
|
||||
|
||||
::: agentlightning.emit_exception
|
||||
|
||||
## Emitter Helpers
|
||||
|
||||
::: agentlightning.get_message_value
|
||||
|
||||
::: agentlightning.get_object_value
|
||||
|
||||
::: agentlightning.find_final_reward
|
||||
|
||||
::: agentlightning.find_reward_spans
|
||||
|
||||
::: agentlightning.get_reward_value
|
||||
|
||||
::: agentlightning.get_rewards_from_span
|
||||
|
||||
::: agentlightning.is_reward_span
|
||||
@@ -0,0 +1,38 @@
|
||||
## Algorithm-side References
|
||||
|
||||
!!! note
|
||||
|
||||
This reference covers APIs that are designed to be used at "Algorithm Side".
|
||||
For built-in algorithms, see [Algorithm Zoo](../algorithm-zoo/index.md).
|
||||
|
||||
## Base Class and Decorators
|
||||
|
||||
::: agentlightning.Algorithm
|
||||
|
||||
::: agentlightning.algo
|
||||
|
||||
## Fast Algorithms (for Debugging)
|
||||
|
||||
::: agentlightning.FastAlgorithm
|
||||
|
||||
::: agentlightning.Baseline
|
||||
|
||||
## Adapter
|
||||
|
||||
::: agentlightning.Adapter
|
||||
|
||||
::: agentlightning.TraceAdapter
|
||||
|
||||
::: agentlightning.OtelTraceAdapter
|
||||
|
||||
::: agentlightning.TraceToTripletBase
|
||||
|
||||
::: agentlightning.TracerTraceToTriplet
|
||||
|
||||
::: agentlightning.LlmProxyTraceToTriplet
|
||||
|
||||
::: agentlightning.TraceToMessages
|
||||
|
||||
## LLM Proxy
|
||||
|
||||
::: agentlightning.LLMProxy
|
||||
@@ -0,0 +1,118 @@
|
||||
# Command Line Interface
|
||||
|
||||
!!! warning
|
||||
|
||||
This document is a work in progress and might not be updated with the latest changes.
|
||||
Try to use `agl -h` to get the latest help message.
|
||||
|
||||
!!! tip
|
||||
|
||||
Agent-lightning also provides utilities to help you build your own CLI for [LitAgent][agentlightning.LitAgent] and [Trainer][agentlightning.Trainer]. See [Trainer](./trainer.md) for references.
|
||||
|
||||
## agl
|
||||
|
||||
```text
|
||||
usage: agl [-h] {vllm,store,prometheus,agentops}
|
||||
|
||||
Agent Lightning CLI entry point.
|
||||
|
||||
Available subcommands:
|
||||
vllm Run the vLLM CLI with Agent Lightning instrumentation.
|
||||
store Run a LightningStore server.
|
||||
prometheus Serve Prometheus metrics from the multiprocess registry.
|
||||
agentops Start the AgentOps server manager.
|
||||
|
||||
positional arguments:
|
||||
{vllm,store,prometheus,agentops}
|
||||
Subcommand to run.
|
||||
|
||||
options:
|
||||
-h, --help show this help message and exit
|
||||
```
|
||||
|
||||
## agl vllm
|
||||
|
||||
Agent-lightning's instrumented vLLM CLI.
|
||||
|
||||
```text
|
||||
usage: agl vllm [-h] [-v] {chat,complete,serve,bench,collect-env,run-batch} ...
|
||||
|
||||
vLLM CLI
|
||||
|
||||
positional arguments:
|
||||
{chat,complete,serve,bench,collect-env,run-batch}
|
||||
chat Generate chat completions via the running API server.
|
||||
complete Generate text completions based on the given prompt via the running API server.
|
||||
collect-env Start collecting environment information.
|
||||
run-batch Run batch prompts and write results to file.
|
||||
|
||||
options:
|
||||
-h, --help show this help message and exit
|
||||
-v, --version show program's version number and exit
|
||||
|
||||
For full list: vllm [subcommand] --help=all
|
||||
For a section: vllm [subcommand] --help=ModelConfig (case-insensitive)
|
||||
For a flag: vllm [subcommand] --help=max-model-len (_ or - accepted)
|
||||
Documentation: https://docs.vllm.ai
|
||||
```
|
||||
|
||||
## agl store
|
||||
|
||||
Agent-lightning's LightningStore CLI. Use it to start an independent LightningStore server.
|
||||
|
||||
Currently the store data are stored in memory and will be lost when the server is stopped.
|
||||
|
||||
```text
|
||||
usage: agl store [-h] [--host HOST] [--port PORT] [--cors-origin CORS_ORIGINS] [--log-level {DEBUG,INFO,WARNING,ERROR}] [--tracker {prometheus,console} [{prometheus,console} ...]] [--n-workers N_WORKERS] [--backend {memory,mongo}]
|
||||
[--mongo-uri MONGO_URI]
|
||||
|
||||
Run a LightningStore server
|
||||
|
||||
options:
|
||||
-h, --help show this help message and exit
|
||||
--host HOST Host to bind the server to
|
||||
--port PORT Port to run the server on
|
||||
--cors-origin CORS_ORIGINS
|
||||
Allowed CORS origin. Repeat for multiple origins. Use '*' to allow all origins.
|
||||
--log-level {DEBUG,INFO,WARNING,ERROR}
|
||||
Configure the logging level for the store.
|
||||
--tracker {prometheus,console} [{prometheus,console} ...]
|
||||
Enable metrics tracking. Repeat for multiple trackers.
|
||||
--n-workers N_WORKERS
|
||||
Number of workers to run in the server. When it's greater than 1, the server will be run using `mp` launch mode. Only applicable for zero-copy stores such as MongoDB backend.
|
||||
--backend {memory,mongo}
|
||||
Backend to use for the store.
|
||||
--mongo-uri MONGO_URI
|
||||
MongoDB URI to use for the store. Applicable only if --backend is 'mongo'.
|
||||
```
|
||||
|
||||
!!! tip
|
||||
|
||||
After launching the store via CLI, you can tell the [`Trainer`][agentlightning.Trainer] to use the store by passing the store address to the trainer.
|
||||
|
||||
```python
|
||||
store_client = agl.LightningStoreClient("http://localhost:4747")
|
||||
trainer = agl.Trainer(store=store_client, ...)
|
||||
```
|
||||
|
||||
See [using external store][debug-with-external-store] for more details.
|
||||
|
||||
## agl prometheus
|
||||
|
||||
Expose the Prometheus multiprocess registry on a dedicated FastAPI server. This is useful when the main LightningStore service is under heavy load; exporters can scrape this auxiliary endpoint instead.
|
||||
|
||||
```text
|
||||
usage: agl prometheus [-h] [--host HOST] [--port PORT] [--metrics-path METRICS_PATH] [--log-level {DEBUG,INFO,WARNING,ERROR}] [--access-log]
|
||||
|
||||
Serve Prometheus metrics outside the LightningStore server.
|
||||
|
||||
options:
|
||||
-h, --help show this help message and exit
|
||||
--host HOST Host to bind the metrics server to.
|
||||
--port PORT Port to expose the Prometheus metrics on.
|
||||
--metrics-path METRICS_PATH
|
||||
HTTP path used to expose metrics. Must start with '/' and not be the root path.
|
||||
--log-level {DEBUG,INFO,WARNING,ERROR}
|
||||
Configure the logging level for the metrics server.
|
||||
--access-log Enable uvicorn access logs. Disabled by default to reduce noise.
|
||||
```
|
||||
@@ -0,0 +1,21 @@
|
||||
# Instrumentation API
|
||||
|
||||
::: agentlightning.instrumentation.instrument_all
|
||||
|
||||
::: agentlightning.instrumentation.uninstrument_all
|
||||
|
||||
## AgentOps LangChain
|
||||
|
||||
::: agentlightning.instrumentation.agentops_langchain
|
||||
|
||||
## AgentOps
|
||||
|
||||
::: agentlightning.instrumentation.agentops
|
||||
|
||||
## LiteLLM
|
||||
|
||||
::: agentlightning.instrumentation.litellm
|
||||
|
||||
## vLLM
|
||||
|
||||
::: agentlightning.instrumentation.vllm
|
||||
@@ -0,0 +1,77 @@
|
||||
# Internal API References
|
||||
|
||||
!!! danger
|
||||
|
||||
The following APIs should be used with extra caution because they are very likely to change in the future.
|
||||
|
||||
## Algorithms and Adapters
|
||||
|
||||
::: agentlightning.adapter.messages.OpenAIMessages
|
||||
|
||||
::: agentlightning.adapter.triplet.TraceTree
|
||||
|
||||
::: agentlightning.adapter.triplet.Transition
|
||||
|
||||
::: agentlightning.adapter.triplet.RewardMatchPolicy
|
||||
|
||||
::: agentlightning.algorithm.decorator.FunctionalAlgorithm
|
||||
|
||||
## LitAgent
|
||||
|
||||
::: agentlightning.litagent.decorator.FunctionalLitAgent
|
||||
|
||||
::: agentlightning.litagent.decorator.llm_rollout
|
||||
|
||||
::: agentlightning.litagent.decorator.prompt_rollout
|
||||
|
||||
::: agentlightning.emitter.annotation.OperationContext
|
||||
|
||||
## LLM Proxy
|
||||
|
||||
::: agentlightning.llm_proxy.ModelConfig
|
||||
|
||||
::: agentlightning.llm_proxy.LightningSpanExporter
|
||||
|
||||
::: agentlightning.llm_proxy.LightningOpenTelemetry
|
||||
|
||||
::: agentlightning.llm_proxy.AddReturnTokenIds
|
||||
|
||||
::: agentlightning.llm_proxy.StreamConversionMiddleware
|
||||
|
||||
::: agentlightning.llm_proxy.MessageInspectionMiddleware
|
||||
|
||||
::: agentlightning.llm_proxy.RolloutAttemptMiddleware
|
||||
|
||||
## Store
|
||||
|
||||
::: agentlightning.store.base.UNSET
|
||||
|
||||
::: agentlightning.store.utils.rollout_status_from_attempt
|
||||
|
||||
::: agentlightning.store.utils.scan_unhealthy_rollouts
|
||||
|
||||
## Tracing and OpenTelemetry
|
||||
|
||||
::: agentlightning.tracer.otel.LightningSpanProcessor
|
||||
|
||||
## Deprecated APIs
|
||||
|
||||
::: agentlightning.emitter.reward.reward
|
||||
|
||||
::: agentlightning.server.AgentLightningServer
|
||||
|
||||
::: agentlightning.server.ServerDataStore
|
||||
|
||||
::: agentlightning.client.AgentLightningClient
|
||||
|
||||
::: agentlightning.client.DevTaskLoader
|
||||
|
||||
::: agentlightning.Task
|
||||
|
||||
::: agentlightning.TaskInput
|
||||
|
||||
::: agentlightning.TaskIfAny
|
||||
|
||||
::: agentlightning.RolloutRawResultLegacy
|
||||
|
||||
::: agentlightning.RolloutLegacy
|
||||
@@ -0,0 +1,19 @@
|
||||
# RESTful API References
|
||||
|
||||
!!! note
|
||||
|
||||
Shown in the following is the RESTful API for Lightning Store.
|
||||
|
||||
<div id="swagger-ui"></div>
|
||||
<link rel="stylesheet" href="https://unpkg.com/swagger-ui-dist/swagger-ui.css" />
|
||||
<script src="https://unpkg.com/swagger-ui-dist/swagger-ui-bundle.js"></script>
|
||||
<script>
|
||||
window.onload = () => {
|
||||
window.ui = SwaggerUIBundle({
|
||||
url: '../../assets/store-openapi.json',
|
||||
dom_id: '#swagger-ui',
|
||||
presets: [SwaggerUIBundle.presets.apis],
|
||||
layout: "BaseLayout"
|
||||
});
|
||||
}
|
||||
</script>
|
||||
@@ -0,0 +1,31 @@
|
||||
# Runner-side References
|
||||
|
||||
!!! note
|
||||
|
||||
This reference covers APIs that are designed to be used at "Runner Side".
|
||||
|
||||
## Runners
|
||||
|
||||
::: agentlightning.LitAgentRunner
|
||||
|
||||
::: agentlightning.Runner
|
||||
|
||||
## Tracer
|
||||
|
||||
::: agentlightning.AgentOpsTracer
|
||||
|
||||
::: agentlightning.OtelTracer
|
||||
|
||||
::: agentlightning.Tracer
|
||||
|
||||
::: agentlightning.tracer.weave.WeaveTracer
|
||||
|
||||
::: agentlightning.DummyTracer
|
||||
|
||||
::: agentlightning.set_active_tracer
|
||||
|
||||
::: agentlightning.get_active_tracer
|
||||
|
||||
::: agentlightning.clear_active_tracer
|
||||
|
||||
::: agentlightning.tracer.weave.WeaveTracer
|
||||
@@ -0,0 +1,3 @@
|
||||
# Semantic Conventions
|
||||
|
||||
::: agentlightning.semconv
|
||||
@@ -0,0 +1,53 @@
|
||||
# Store References
|
||||
|
||||
::: agentlightning.LightningStore
|
||||
|
||||
::: agentlightning.LightningStoreCapabilities
|
||||
|
||||
## Store Implementations
|
||||
|
||||
::: agentlightning.InMemoryLightningStore
|
||||
|
||||
::: agentlightning.store.mongo.MongoLightningStore
|
||||
|
||||
::: agentlightning.CollectionBasedLightningStore
|
||||
|
||||
## Client-Server and Thread-safe Wrappers
|
||||
|
||||
::: agentlightning.LightningStoreServer
|
||||
|
||||
::: agentlightning.LightningStoreClient
|
||||
|
||||
::: agentlightning.LightningStoreThreaded
|
||||
|
||||
## Collections and Collection Implementations
|
||||
|
||||
::: agentlightning.store.collection.AtomicMode
|
||||
|
||||
::: agentlightning.store.collection.AtomicLabels
|
||||
|
||||
::: agentlightning.store.collection.Collection
|
||||
|
||||
::: agentlightning.store.collection.Queue
|
||||
|
||||
::: agentlightning.store.collection.KeyValue
|
||||
|
||||
::: agentlightning.store.collection.LightningCollections
|
||||
|
||||
::: agentlightning.store.collection.ListBasedCollection
|
||||
|
||||
::: agentlightning.store.collection.DequeBasedQueue
|
||||
|
||||
::: agentlightning.store.collection.DictBasedKeyValue
|
||||
|
||||
::: agentlightning.store.collection.InMemoryLightningCollections
|
||||
|
||||
::: agentlightning.store.collection.mongo.MongoBasedCollection
|
||||
|
||||
::: agentlightning.store.collection.mongo.MongoBasedQueue
|
||||
|
||||
::: agentlightning.store.collection.mongo.MongoBasedKeyValue
|
||||
|
||||
::: agentlightning.store.collection.mongo.MongoClientPool
|
||||
|
||||
::: agentlightning.store.collection.mongo.MongoLightningCollections
|
||||
@@ -0,0 +1,33 @@
|
||||
# Agent-lightning Trainer
|
||||
|
||||
::: agentlightning.Trainer
|
||||
|
||||
::: agentlightning.build_component
|
||||
|
||||
## Execution Strategy
|
||||
|
||||
::: agentlightning.ExecutionStrategy
|
||||
|
||||
::: agentlightning.ClientServerExecutionStrategy
|
||||
|
||||
::: agentlightning.SharedMemoryExecutionStrategy
|
||||
|
||||
## Events
|
||||
|
||||
::: agentlightning.ExecutionEvent
|
||||
|
||||
::: agentlightning.ThreadingEvent
|
||||
|
||||
::: agentlightning.MultiprocessingEvent
|
||||
|
||||
## CLI Builder
|
||||
|
||||
::: agentlightning.lightning_cli
|
||||
|
||||
## Logging
|
||||
|
||||
::: agentlightning.configure_logger
|
||||
|
||||
::: agentlightning.setup_module_logging
|
||||
|
||||
::: agentlightning.setup_logging
|
||||
@@ -0,0 +1,97 @@
|
||||
# Type References
|
||||
|
||||
## Core Types
|
||||
|
||||
::: agentlightning.Triplet
|
||||
|
||||
::: agentlightning.RolloutRawResult
|
||||
|
||||
::: agentlightning.RolloutMode
|
||||
|
||||
::: agentlightning.GenericResponse
|
||||
|
||||
::: agentlightning.ParallelWorkerBase
|
||||
|
||||
::: agentlightning.Dataset
|
||||
|
||||
::: agentlightning.AttemptStatus
|
||||
|
||||
::: agentlightning.RolloutStatus
|
||||
|
||||
::: agentlightning.RolloutConfig
|
||||
|
||||
::: agentlightning.Rollout
|
||||
|
||||
::: agentlightning.EnqueueRolloutRequest
|
||||
|
||||
::: agentlightning.Attempt
|
||||
|
||||
::: agentlightning.AttemptedRollout
|
||||
|
||||
::: agentlightning.Worker
|
||||
|
||||
::: agentlightning.WorkerStatus
|
||||
|
||||
::: agentlightning.Hook
|
||||
|
||||
::: agentlightning.PaginatedResult
|
||||
|
||||
::: agentlightning.FilterOptions
|
||||
|
||||
::: agentlightning.SortOptions
|
||||
|
||||
::: agentlightning.FilterField
|
||||
|
||||
## Resources
|
||||
|
||||
::: agentlightning.Resource
|
||||
|
||||
::: agentlightning.LLM
|
||||
|
||||
::: agentlightning.ProxyLLM
|
||||
|
||||
::: agentlightning.PromptTemplate
|
||||
|
||||
::: agentlightning.ResourceUnion
|
||||
|
||||
::: agentlightning.NamedResources
|
||||
|
||||
::: agentlightning.ResourcesUpdate
|
||||
|
||||
## Traces
|
||||
|
||||
::: agentlightning.AttributeValue
|
||||
|
||||
::: agentlightning.Attributes
|
||||
|
||||
::: agentlightning.TraceState
|
||||
|
||||
::: agentlightning.SpanContext
|
||||
|
||||
::: agentlightning.TraceStatus
|
||||
|
||||
::: agentlightning.Event
|
||||
|
||||
::: agentlightning.Link
|
||||
|
||||
::: agentlightning.Resource
|
||||
|
||||
::: agentlightning.Span
|
||||
|
||||
::: agentlightning.SpanAttributeNames
|
||||
|
||||
::: agentlightning.SpanLike
|
||||
|
||||
::: agentlightning.SpanCoreFields
|
||||
|
||||
::: agentlightning.SpanRecordingContext
|
||||
|
||||
## Environment Variables
|
||||
|
||||
::: agentlightning.LightningEnvVar
|
||||
|
||||
::: agentlightning.resolve_bool_env_var
|
||||
|
||||
::: agentlightning.resolve_int_env_var
|
||||
|
||||
::: agentlightning.resolve_str_env_var
|
||||
@@ -0,0 +1,75 @@
|
||||
# Utility References
|
||||
|
||||
## ID
|
||||
|
||||
::: agentlightning.utils.id.generate_id
|
||||
|
||||
## Metrics
|
||||
|
||||
::: agentlightning.utils.metrics.MetricsBackend
|
||||
|
||||
::: agentlightning.utils.metrics.ConsoleMetricsBackend
|
||||
|
||||
::: agentlightning.utils.metrics.PrometheusMetricsBackend
|
||||
|
||||
::: agentlightning.utils.metrics.MultiMetricsBackend
|
||||
|
||||
::: agentlightning.utils.metrics.setup_multiprocess_prometheus
|
||||
|
||||
::: agentlightning.utils.metrics.get_prometheus_registry
|
||||
|
||||
::: agentlightning.utils.metrics.shutdown_metrics
|
||||
|
||||
## Server Launcher
|
||||
|
||||
::: agentlightning.utils.server_launcher.PythonServerLauncher
|
||||
|
||||
::: agentlightning.utils.server_launcher.PythonServerLauncherArgs
|
||||
|
||||
::: agentlightning.utils.server_launcher.LaunchMode
|
||||
|
||||
## OpenTelemetry
|
||||
|
||||
::: agentlightning.utils.otel.full_qualified_name
|
||||
|
||||
::: agentlightning.utils.otel.get_tracer_provider
|
||||
|
||||
::: agentlightning.utils.otel.get_tracer
|
||||
|
||||
::: agentlightning.utils.otel.make_tag_attributes
|
||||
|
||||
::: agentlightning.utils.otel.extract_tags_from_attributes
|
||||
|
||||
::: agentlightning.utils.otel.make_link_attributes
|
||||
|
||||
::: agentlightning.utils.otel.query_linked_spans
|
||||
|
||||
::: agentlightning.utils.otel.extract_links_from_attributes
|
||||
|
||||
::: agentlightning.utils.otel.filter_attributes
|
||||
|
||||
::: agentlightning.utils.otel.filter_and_unflatten_attributes
|
||||
|
||||
::: agentlightning.utils.otel.flatten_attributes
|
||||
|
||||
::: agentlightning.utils.otel.unflatten_attributes
|
||||
|
||||
::: agentlightning.utils.otel.sanitize_attribute_value
|
||||
|
||||
::: agentlightning.utils.otel.sanitize_attributes
|
||||
|
||||
::: agentlightning.utils.otel.sanitize_list_attribute_sanity
|
||||
|
||||
::: agentlightning.utils.otel.check_attributes_sanity
|
||||
|
||||
::: agentlightning.utils.otel.format_exception_attributes
|
||||
|
||||
## OTLP
|
||||
|
||||
::: agentlightning.utils.otlp.handle_otlp_export
|
||||
|
||||
::: agentlightning.utils.otlp.spans_from_proto
|
||||
|
||||
## System Snapshot
|
||||
|
||||
::: agentlightning.utils.system_snapshot.system_snapshot
|
||||
@@ -0,0 +1,111 @@
|
||||
.md-grid {
|
||||
max-width: 88rem;
|
||||
}
|
||||
|
||||
@media screen and (min-width: 100em) {
|
||||
html {
|
||||
font-size: 130%;
|
||||
}
|
||||
}
|
||||
|
||||
@media screen and (min-width: 125em) {
|
||||
html {
|
||||
font-size: 135%;
|
||||
}
|
||||
}
|
||||
|
||||
.md-typeset .admonition {
|
||||
font-size: 0.72rem;
|
||||
}
|
||||
|
||||
.md-typeset h4 {
|
||||
font-size: 1.15em;
|
||||
}
|
||||
|
||||
.md-typeset h5, .md-typeset h6 {
|
||||
font-size: 1em;
|
||||
}
|
||||
|
||||
/* Increase spacing between API references */
|
||||
.doc-class, .doc-function, .doc-attribute {
|
||||
padding-bottom: 2em;
|
||||
margin-bottom: 3em;
|
||||
border-bottom: 1px solid #77777777;
|
||||
}
|
||||
|
||||
.doc-class .doc-function:not(:last-child), .doc-class .doc-attribute:not(:last-child) {
|
||||
padding-bottom: 0;
|
||||
margin-bottom: 2.5em;
|
||||
border-bottom: none;
|
||||
}
|
||||
|
||||
.doc-class .doc-function:last-child, .doc-class .doc-attribute:last-child {
|
||||
padding-bottom: 0;
|
||||
margin-bottom: 0;
|
||||
border-bottom: none;
|
||||
}
|
||||
|
||||
[data-md-color-primary="agl"] {
|
||||
--md-primary-fg-color: #c45259;
|
||||
--md-primary-fg-color--light: #e8b4b7;
|
||||
--md-primary-fg-color--dark: #9a3038;
|
||||
--md-hue: 356;
|
||||
}
|
||||
|
||||
[data-md-color-accent="agl"] {
|
||||
--md-accent-fg-color: #f69047;
|
||||
--md-accent-fg-color--light: #fcc59e;
|
||||
--md-accent-fg-color--dark: #da6005;
|
||||
}
|
||||
|
||||
[data-md-color-scheme="slate"][data-md-color-primary="agl"] {
|
||||
--md-default-fg-color: hsla(var(--md-hue), 15%, 90%, 0.88);
|
||||
--md-default-bg-color: hsla(var(--md-hue), 6%, 4%, 1);
|
||||
--md-code-bg-color: hsla(var(--md-hue), 5%, 20%, 0.5);
|
||||
}
|
||||
|
||||
/* Documentation version warning banner */
|
||||
.version-warning {
|
||||
background-color: #FFD15D18;
|
||||
padding: 1em;
|
||||
font-weight: 500;
|
||||
position: relative;
|
||||
z-index: 1000;
|
||||
border: 2px solid #FFD15D;
|
||||
border-radius: 0.25em;
|
||||
margin-bottom: 2em;
|
||||
}
|
||||
|
||||
/* To center images */
|
||||
.center {
|
||||
display: block;
|
||||
margin-left: auto;
|
||||
margin-right: auto;
|
||||
}
|
||||
|
||||
/* Charts */
|
||||
canvas[data-chart] {
|
||||
width: 100%;
|
||||
display: block;
|
||||
}
|
||||
|
||||
/* Grid behavior */
|
||||
.md-typeset .grid {
|
||||
grid-template-columns: repeat(auto-fit, minmax(24rem, 1fr));
|
||||
}
|
||||
|
||||
/* Make cards fill equal height and push footer link to bottom */
|
||||
.md-typeset .grid.cards > ul > li {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 0;
|
||||
}
|
||||
|
||||
.md-typeset .grid.cards > ul > li > hr {
|
||||
margin: 0.5em 0;
|
||||
}
|
||||
|
||||
.md-typeset .grid.cards > ul > li > :last-child {
|
||||
margin-top: auto; /* pushes the last element (Browse source) to bottom */
|
||||
padding-top: 0.5em;
|
||||
}
|
||||
@@ -0,0 +1,326 @@
|
||||
# Debugging and Troubleshooting
|
||||
|
||||
When you train your own agent with Agent-lightning, most failures surface because the agent logic is brittle or simply incorrect. Debugging becomes easier when you peel back the stack: start by driving the rollout logic on its own, dry-run the trainer loop, and only then bring the full algorithm and runner topology online. The [`examples/apo/apo_debug.py`]({{ src("examples/apo/apo_debug.py") }}) script demonstrates these techniques; this guide expands on each approach and helps you decide when to reach for them.
|
||||
|
||||
## Debugging with Dashboard
|
||||
|
||||
When you launch an experiment with [`Trainer.fit`][agentlightning.Trainer.fit] or start an isolated store via [`agl store`](../reference/cli.md), the terminal prints a message similar to:
|
||||
|
||||
```text
|
||||
INFO Agent-lightning dashboard will be available at http://192.168.0.107:4747
|
||||
```
|
||||
|
||||
Visit that URL, and you will see the Agent-lightning dashboard:
|
||||
|
||||
|
||||
|
||||
The dashboard surfaces everything stored inside [the store](../deep-dive/store.md). Because the store mediates interactions between algorithms and runners, inspecting it often reveals which side is causing issues such as stale rollouts, unresponsive workers, or empty traces.
|
||||
|
||||
For example, the VERL algorithm may receive no token IDs and emit `cannot reshape tensor of 0 elements into shape [1, 0, -1, 128] because the unspecified dimension size -1 can be any value and is ambiguous` ([Issue #50](https://github.com/microsoft/agent-lightning/issues/50), [Issue #76](https://github.com/microsoft/agent-lightning/issues/76)). Several scenarios can produce that error: the runner might not produce trace spans at all, it might produce spans without token IDs, or the IDs may be present but formatted incorrectly. Inspecting the dashboard traces helps you pinpoint which condition applies.
|
||||
|
||||
|
||||
|
||||
By checking whether the trace span is empty and whether token IDs appear in the span attributes, you can narrow the issue to either the runner (agent) side or the algorithm side. Then apply the techniques below to debug the faulty component.
|
||||
|
||||
## Debug-level Logging
|
||||
|
||||
Starting from v0.3, detailed signals such as store server access logs, runner lifecycle logs, and span payloads only appear when the log level is `DEBUG` so the default output stays readable. Enable debug-level logging by adding the following snippet near the top of your script:
|
||||
|
||||
```python
|
||||
import agentlightning as agl
|
||||
|
||||
agl.setup_logging("DEBUG")
|
||||
```
|
||||
|
||||
Set the log level on every process if your setup involves multiple workers. For example, when [running stores in isolation][debug-with-external-store], configure the store process explicitly:
|
||||
|
||||
```bash
|
||||
agl store --port 4747 --log-level DEBUG
|
||||
```
|
||||
|
||||
## Using [`Runner`][agentlightning.Runner] in Isolation
|
||||
|
||||
[`Runner`][agentlightning.Runner] is a long-lived worker that wraps your [`LitAgent`][agentlightning.LitAgent], coordinates tracing, and talks to the [`LightningStore`][agentlightning.LightningStore]. In typical training flows the trainer manages runners for you, but being able to spin one up manually is invaluable while debugging.
|
||||
|
||||
If you define rollout logic with [`@rollout`][agentlightning.rollout] or implement a [`LitAgent`][agentlightning.LitAgent] directly, you will get a [`LitAgent`][agentlightning.LitAgent] instance and you should be able to execute it with [`LitAgentRunner`][agentlightning.LitAgentRunner], which is a subclass of [`Runner`][agentlightning.Runner]. The runner needs but does not instantiate a [`Tracer`][agentlightning.Tracer], so supply one yourself. See [Working with Traces](./traces.md) for a walkthrough of tracer options.
|
||||
|
||||
[`Runner.run_context`][agentlightning.Runner.run_context] prepares the runner to execute a particular agent. Besides the agent and tracer you must provide a store that will collect spans and rollouts. [`InMemoryLightningStore`][agentlightning.InMemoryLightningStore] keeps everything in-process, which is perfect for debugging sessions.
|
||||
|
||||
```python
|
||||
import agentlightning as agl
|
||||
|
||||
tracer = agl.OtelTracer()
|
||||
runner = agl.LitAgentRunner(tracer)
|
||||
store = agl.InMemoryLightningStore()
|
||||
|
||||
with runner.run_context(agent=apo_rollout, store=store):
|
||||
...
|
||||
```
|
||||
|
||||
Inside the [`run_context`][agentlightning.Runner.run_context] block you can call [`runner.step(...)`][agentlightning.Runner.step] to execute a single rollout. The payload includes the task input and any [`NamedResources`][agentlightning.NamedResources] the agent expects. Read [introduction to Resources][introduction-to-resources] and [NamedResources][introduction-to-named-resources] for more details. For example, if your agent references a [`PromptTemplate`][agentlightning.PromptTemplate], pass it through the `resources` argument:
|
||||
|
||||
```python
|
||||
with runner.run_context(agent=apo_rollout, store=store):
|
||||
resource = agl.PromptTemplate(template="You are a helpful assistant. {any_question}", engine="f-string")
|
||||
rollout = await runner.step(
|
||||
"Explain why the sky appears blue using principles of light scattering in 100 words.",
|
||||
resources={"main_prompt": resource},
|
||||
)
|
||||
```
|
||||
|
||||
You can do as many things as you want within the [`Runner.run_context`][agentlightning.Runner.run_context] block. After the rollout finishes you can query the store to inspect what happened:
|
||||
|
||||
```python
|
||||
print(await store.query_rollouts())
|
||||
print(await store.query_spans(rollout.rollout_id))
|
||||
```
|
||||
|
||||
Example output (with a reward span captured):
|
||||
|
||||
```python
|
||||
[Rollout(rollout_id='ro-519769241af8', input='Explain why the sky appears blue using principles of light scattering in 100 words.', start_time=1760706315.6996238, ..., status='succeeded')]
|
||||
[Span(rollout_id='ro-519769241af8', attempt_id='at-a6b62caf', sequence_id=1, ..., name='agentlightning.annotation', attributes={'agentlightning.reward.0.value': 0.95}, ...)]
|
||||
```
|
||||
|
||||
Swap in an [`AgentOpsTracer`][agentlightning.AgentOpsTracer] instead of [`OtelTracer`][agentlightning.OtelTracer] to see the underlying LLM spans alongside reward information:
|
||||
|
||||
```python
|
||||
[
|
||||
Span(rollout_id='ro-519769241af8', attempt_id='at-a6b62caf', sequence_id=1, ..., name='openai.chat.completion', attributes={..., 'gen_ai.prompt.0.role': 'user', 'gen_ai.prompt.0.content': 'You are a helpful assistant. Explain why the sky appears blue using principles of light scattering in 100 words.', ...}),
|
||||
Span(rollout_id='ro-519769241af8', attempt_id='at-a6b62caf', sequence_id=2, ..., name='openai.chat.completion', attributes={..., 'gen_ai.prompt.0.role': 'user', 'gen_ai.prompt.0.content': 'Evaluate how well the output fulfills the task...', ...}),
|
||||
Span(rollout_id='ro-519769241af8', attempt_id='at-a6b62caf', sequence_id=3, ..., name='agentlightning.annotation', attributes={'agentlightning.reward.0.value': 0.95}, ...)
|
||||
]
|
||||
```
|
||||
|
||||
!!! tip
|
||||
|
||||
Spans too difficult to read? Try using [`Adapter`][agentlightning.Adapter] to convert them into a [more readable format](./traces.md).
|
||||
|
||||
[`Runner.step`][agentlightning.Runner.step] executes a full rollout even though it is named "step". The companion method [`Runner.iter`][agentlightning.Runner.iter] executes multiple "steps" by continuously pulling new rollout inputs from the store until a stop event is set. Use `iter` once you are confident the single-step path works and you have another worker [`enqueue_rollout`][agentlightning.LightningStore.enqueue_rollout] to the store.
|
||||
|
||||
!!! tip
|
||||
|
||||
You can also call [`Runner.step`][agentlightning.Runner.step] to inject ad-hoc rollouts into a running store being used by another algorithm, so that the rollouts can be consumed by the algorithms. This is very recently known as the paradigm of ["online RL"](https://cursor.com/blog/tab-rl). At the moment, no algorithm in the [algorithm zoo](../algorithm-zoo/index.md) consumes externally generated rollouts, but the data flow is available there if you need it.
|
||||
|
||||
## Debug with LLM Proxy
|
||||
|
||||
If you are dealing with LLM optimization like Reinforcement Learning, we generally recommend using an online stable LLM service for your debugging purposes, like `openai/gpt-4.1-nano`. After the debugging is done, you can switch to a local training endpoint.
|
||||
|
||||
However, if you want to use a local LLM features like [getting the token IDs](../deep-dive/serving-llm.md), you can also manually start a local vLLM server by:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen2.5-0.5B-Instruct --port 8080
|
||||
```
|
||||
|
||||
Then start the LLM proxy via the following script:
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import aiohttp
|
||||
import agentlightning as agl
|
||||
|
||||
async def serve_llm_proxy():
|
||||
store = agl.InMemoryLightningStore()
|
||||
store_server = agl.LightningStoreServer(store, "127.0.0.1", 8081)
|
||||
await store_server.start()
|
||||
|
||||
llm_proxy = agl.LLMProxy(
|
||||
port=8082,
|
||||
model_list=[
|
||||
{
|
||||
"model_name": "Qwen/Qwen2.5-0.5B-Instruct",
|
||||
"litellm_params": {
|
||||
"model": "hosted_vllm/Qwen/Qwen2.5-0.5B-Instruct",
|
||||
"api_base": "http://localhost:8080/v1",
|
||||
},
|
||||
}
|
||||
],
|
||||
store=store_server,
|
||||
)
|
||||
|
||||
await llm_proxy.start()
|
||||
await asyncio.sleep(1000000)
|
||||
```
|
||||
|
||||
Test the served LLM proxy with a client like:
|
||||
|
||||
```python
|
||||
async def test_llm_proxy():
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.post("http://localhost:8082/v1/chat/completions", json={
|
||||
"model": "Qwen/Qwen2.5-0.5B-Instruct",
|
||||
"messages": [{"role": "user", "content": "Hello, world!"}],
|
||||
}) as response:
|
||||
print(await response.json())
|
||||
```
|
||||
|
||||
You can now use the LLM proxy by specifying environment variables:
|
||||
|
||||
```bash
|
||||
export OPENAI_API_BASE=http://localhost:8081/v1
|
||||
export OPENAI_API_KEY=dummy
|
||||
```
|
||||
|
||||
You might see warnings about `Missing or invalid rollout_id, attempt_id, or sequence_id` in the LLM proxy logs. This is fine because you don't have a rollout and attempt yet when you are debugging. When you started the training, the algorithm will create the rollouts for you and the warnings will go away.
|
||||
|
||||
## Hook into Runner's Lifecycle
|
||||
|
||||
[`Runner.run_context`][agentlightning.Runner.run_context] accepts a `hooks` argument so you can observe or augment lifecycle events without editing your agent. Hooks subclass [`Hook`][agentlightning.Hook] and can respond to four asynchronous callbacks: [`on_trace_start`][agentlightning.Hook.on_trace_start], [`on_rollout_start`][agentlightning.Hook.on_rollout_start], [`on_rollout_end`][agentlightning.Hook.on_rollout_end], and [`on_trace_end`][agentlightning.Hook.on_trace_end]. This is useful for:
|
||||
|
||||
- Capturing raw OpenTelemetry spans before they hit the store and before the [`LitAgentRunner`][agentlightning.LitAgentRunner] do postprocessing on the rollout
|
||||
- Inspecting the tracer instance after they are activated
|
||||
- Logging rollout inputs before they are processed by the agent
|
||||
|
||||
The `hook` mode in [`examples/apo/apo_debug.py`]({{ src("examples/apo/apo_debug.py") }}) prints every span collected during a rollout:
|
||||
|
||||
```python
|
||||
import agentlightning as agl
|
||||
|
||||
# ... Same as previous example
|
||||
|
||||
class DebugHook(agl.Hook):
|
||||
async def on_trace_end(self, *, agent, runner, tracer, rollout):
|
||||
trace = tracer.get_last_trace()
|
||||
print("Trace spans collected during the rollout:")
|
||||
for span in trace:
|
||||
print(f"- {span.name} (status: {span.status}):\n {span.attributes}")
|
||||
|
||||
with runner.run_context(
|
||||
agent=apo_rollout,
|
||||
store=store,
|
||||
hooks=[DebugHook()],
|
||||
):
|
||||
await runner.step(
|
||||
"Explain why the sky appears blue using principles of light scattering in 100 words.",
|
||||
resources={"main_prompt": resource},
|
||||
)
|
||||
```
|
||||
|
||||
Because hooks run inside the runner process you can also attach debuggers or breakpoints directly in the callback implementations.
|
||||
|
||||
!!! note
|
||||
|
||||
For a better understanding of where hooks are called, we show a pseudo code of Runner's working flow below:
|
||||
|
||||
```python
|
||||
resources = await store.get_latest_resources()
|
||||
rollout = ...
|
||||
try:
|
||||
# <-- on_rollout_start
|
||||
with tracer.trace_context(...):
|
||||
# <--- on_trace_start
|
||||
result = await agent.rollout(...)
|
||||
# <--- on_trace_end
|
||||
post_process_result(result)
|
||||
except Exception:
|
||||
# <-- on_rollout_end
|
||||
await store.update_attempt(status=...)
|
||||
```
|
||||
|
||||
## Dry-Run the Trainer Loop
|
||||
|
||||
Once single rollouts behave, switch to the trainer’s dry-run mode. [`Trainer.dev`][agentlightning.Trainer.dev] spins up a lightweight fast algorithm — [`agentlightning.Baseline`][agentlightning.Baseline] by default — so you can exercise the same infrastructure as [`Trainer.fit`][agentlightning.Trainer.fit] without standing up complex stacks like RL or SFT.
|
||||
|
||||
!!! warning
|
||||
When you enable multiple runners via `n_runners`, the trainer may execute them in separate worker processes. Attaching a debugger such as `pdb` is only practical when `n_runners=1`, and even then the runner might not live in the main process.
|
||||
|
||||
```python
|
||||
import agentlightning as agl
|
||||
|
||||
dataset: agl.Dataset[str] = [
|
||||
"Explain why the sky appears blue using principles of light scattering in 100 words.",
|
||||
"What's the capital of France?",
|
||||
]
|
||||
resource = agl.PromptTemplate(template="You are a helpful assistant. {any_question}", engine="f-string")
|
||||
|
||||
trainer = agl.Trainer(
|
||||
n_runners=1,
|
||||
initial_resources={"main_prompt": resource},
|
||||
)
|
||||
trainer.dev(apo_rollout, dataset)
|
||||
```
|
||||
|
||||
Just like [`Runner.run_context`][agentlightning.Runner.run_context], [`Trainer.dev`][agentlightning.Trainer.dev] requires the [`NamedResources`][agentlightning.NamedResources] your agent expects. The key difference is that resources are attached to the trainer rather than the runner.
|
||||
|
||||
[`Trainer.dev`][agentlightning.Trainer.dev] uses an almost switchable interface from [`Trainer.fit`][agentlightning.Trainer.fit]. It also needs a dataset to iterate over, similar to [`fit`][agentlightning.Trainer.fit]. Under the hood [`dev`][agentlightning.Trainer.dev] uses the same implementation as [`fit`][agentlightning.Trainer.fit], which means you can spin up multiple runners, observe scheduler behavior, and validate how algorithms adapt rollouts. The default [`Baseline`][agentlightning.Baseline] logs detailed traces so you can see each rollout as the algorithm perceives it:
|
||||
|
||||
```text
|
||||
21:20:30 Initial resources set: {'main_prompt': PromptTemplate(resource_type='prompt_template', template='You are a helpful assistant. {any_question}', engine='f-string')}
|
||||
21:20:30 Proceeding epoch 1/1.
|
||||
21:20:30 Enqueued rollout ro-302fb202bd85 in train mode with sample: Explain why the sky appears blue using principles of light scattering in 100 words.
|
||||
21:20:30 Enqueued rollout ro-e65a3ffaa540 in train mode with sample: What's the capital of France?
|
||||
21:20:30 Waiting for 2 harvest tasks to complete...
|
||||
21:20:30 [Rollout ro-302fb202bd85] Status is initialized to queuing.
|
||||
21:20:30 [Rollout ro-e65a3ffaa540] Status is initialized to queuing.
|
||||
21:20:35 [Rollout ro-302fb202bd85] Finished with status succeeded in 3.80 seconds.
|
||||
21:20:35 [Rollout ro-302fb202bd85 | Attempt 1] ID: at-f84ad21c. Status: succeeded. Worker: Worker-0
|
||||
21:20:35 [Rollout ro-302fb202bd85 | Attempt at-f84ad21c | Span 3a286a856af6bea8] #1 (openai.chat.completion) ... 1.95 seconds. Attribute keys: ['gen_ai.request.type', 'gen_ai.system', ...]
|
||||
21:20:35 [Rollout ro-302fb202bd85 | Attempt at-f84ad21c | Span e2f44b775e058dd6] #2 (openai.chat.completion) ... 1.24 seconds. Attribute keys: ['gen_ai.request.type', 'gen_ai.system', ...]
|
||||
21:20:35 [Rollout ro-302fb202bd85 | Attempt at-f84ad21c | Span 45ee3c94fa1070ec] #3 (agentlightning.annotation) ... 0.00 seconds. Attribute keys: ['agentlightning.reward.0.value']
|
||||
21:20:35 [Rollout ro-302fb202bd85] Adapted data: [Triplet(prompt={'token_ids': []}, response={'token_ids': []}, reward=None, metadata={'response_id': '...', 'agent_name': ''}), Triplet(prompt={'token_ids': []}, response={'token_ids': []}, reward=0.95, metadata={'response_id': '...', 'agent_name': ''})]
|
||||
21:20:35 Finished 1 rollouts.
|
||||
21:20:35 [Rollout ro-e65a3ffaa540] Status changed to preparing.
|
||||
21:20:40 [Rollout ro-e65a3ffaa540] Finished with status succeeded in 6.39 seconds.
|
||||
21:20:40 [Rollout ro-e65a3ffaa540 | Attempt 1] ID: at-eaefa5d4. Status: succeeded. Worker: Worker-0
|
||||
21:20:40 [Rollout ro-e65a3ffaa540 | Attempt at-eaefa5d4 | Span 901dd6acc0f50147] #1 (openai.chat.completion) ... 1.30 seconds. Attribute keys: ['gen_ai.request.type', 'gen_ai.system', ...]
|
||||
21:20:40 [Rollout ro-e65a3ffaa540 | Attempt at-eaefa5d4 | Span 52e0aa63e02be611] #2 (openai.chat.completion) ... 1.26 seconds. Attribute keys: ['gen_ai.request.type', 'gen_ai.system', ...]
|
||||
21:20:40 [Rollout ro-e65a3ffaa540 | Attempt at-eaefa5d4 | Span 6c452de193fbffd3] #3 (agentlightning.annotation) ... 0.00 seconds. Attribute keys: ['agentlightning.reward.0.value']
|
||||
21:20:40 [Rollout ro-e65a3ffaa540] Adapted data: [Triplet(prompt={'token_ids': []}, response={'token_ids': []}, reward=None, metadata={'response_id': '...', 'agent_name': ''}), Triplet(prompt={'token_ids': []}, response={'token_ids': []}, reward=1.0, metadata={'response_id': '...', 'agent_name': ''})]
|
||||
21:20:40 Finished 2 rollouts.
|
||||
```
|
||||
|
||||
The only limitation is that resources remain static and components like [`LLMProxy`][agentlightning.LLMProxy] are not wired in. For richer dry runs you can subclass [`FastAlgorithm`][agentlightning.FastAlgorithm] and override the pieces you care about.
|
||||
|
||||
## Debug the Algorithm-Runner Boundary
|
||||
|
||||
[](){ #debug-with-external-store }
|
||||
|
||||
Debugging algorithms in Agent-Lightning is often more challenging than debugging agents. Algorithms are typically **stateful** and depend on several moving parts — runners, stores, and trainers — which makes it difficult to isolate and inspect their behavior. Even mocking an agent to cooperate with an algorithm can be costly and error-prone. To simplify this, Agent-Lightning provides a way to run algorithms in isolation so you can attach a debugger and inspect internal state without interference from other components.
|
||||
|
||||
By default, [`Trainer.fit`][agentlightning.Trainer.fit] runs the algorithm in the main process and thread, but its logs are interleaved with those from the store and runners, making it hard to follow what’s happening inside the algorithm itself. In [*Write Your First Algorithm*](../how-to/write-first-algorithm.md), we covered how to stand up a store, algorithm, and runner in isolation for your own implementations. This section extends that approach to cover two common questions:
|
||||
|
||||
1. How can I run built-in or class-based algorithms (inheriting from [`Algorithm`][agentlightning.Algorithm]) in isolation?
|
||||
2. How can I still use [`Trainer`][agentlightning.Trainer] features like `n_runners`, `adapter`, or `llm_proxy` while debugging?
|
||||
|
||||
The solution is to keep using a [`Trainer`][agentlightning.Trainer] instance but **manage the store yourself**, running the algorithm and runner roles separately. This approach mirrors the internal process orchestration of [`Trainer.fit`][agentlightning.Trainer.fit], but with more visibility and control. Below, we show a step-by-step guide to achieve this with the [`calc_agent` example]({{ src("examples/calc_x/train_calc_agent.py") }}).
|
||||
|
||||
**1. Launch the store manually.**
|
||||
In a separate terminal, start the store:
|
||||
|
||||
```bash
|
||||
agl store --port 4747
|
||||
```
|
||||
|
||||
Add `--log-level DEBUG` to the command to see the detailed logs.
|
||||
|
||||
Then, in your training script, create a [`LightningStoreClient`][agentlightning.LightningStoreClient] and pass it to the trainer:
|
||||
|
||||
```python
|
||||
client = agl.LightningStoreClient("http://localhost:4747")
|
||||
trainer = agl.Trainer(store=client, ...)
|
||||
```
|
||||
|
||||
Set the environment variable `AGL_MANAGED_STORE=0` so the trainer doesn't attempt to manage the store automatically.
|
||||
|
||||
**2. Start the runner and algorithm processes separately.**
|
||||
Each process should run the same training script, but with different environment variables specifying the current role.
|
||||
This setup faithfully mirrors how [`Trainer.fit`][agentlightning.Trainer.fit] orchestrates these components behind the scenes.
|
||||
|
||||
```bash
|
||||
# Terminal 2 – Runner process
|
||||
AGL_MANAGED_STORE=0 AGL_CURRENT_ROLE=runner \
|
||||
python train_calc_agent.py --external-store-address http://localhost:4747 --val-file data/test_mini.parquet
|
||||
|
||||
# Terminal 3 – Algorithm process
|
||||
AGL_MANAGED_STORE=0 AGL_CURRENT_ROLE=algorithm \
|
||||
python train_calc_agent.py --external-store-address http://localhost:4747 --val-file data/test_mini.parquet
|
||||
```
|
||||
|
||||
**3. Reuse your existing trainer configuration.**
|
||||
You can continue using the same datasets, adapters, and proxies as usual. Because the store is now external, you can:
|
||||
|
||||
* Attach debuggers to either the algorithm or runner process
|
||||
* Add fine-grained logging or tracing
|
||||
* Simulate partial failures or latency in individual components
|
||||
|
||||
This setup provides a faithful reproduction of the algorithm–runner interaction while keeping the store visible for inspection. Once you’ve resolved the issue, simply set `AGL_MANAGED_STORE=1` (or omit it) to return to the standard managed training workflow.
|
||||
@@ -0,0 +1,219 @@
|
||||
# Using Emitters
|
||||
|
||||
[](){ #using-emitter }
|
||||
|
||||
While returning a single float for the final reward is sufficient for many algorithm-agent combinations, some advanced scenarios require richer feedback. For instance, an algorithm might learn more effectively if it receives intermediate rewards throughout a multi-step task, or if the agent needs to emit additional spans for debugging or analysis.
|
||||
|
||||
Agent-lightning provides an **emitter** module for recording custom spans inside your agent logic. Just as [Tracer][agentlightning.Tracer] automatically instruments common operations (for example, LLM calls), each emitter helper sends a [Span][agentlightning.Span] that captures Agent-lightning-specific work so downstream algorithms can query it later. See [Working with Traces](./traces.md) for more details.
|
||||
|
||||
For multi-step routines such as function calls, tools, or adapters, wrap code with [`operation`][agentlightning.operation] — either as a decorator or a context manager — to capture inputs, outputs, and metadata on a dedicated [`operation`][agentlightning.operation] span. This makes it easier to correlate downstream annotations (like rewards or messages) with the higher-level work that produced them.
|
||||
|
||||
You can find the emitter functions in [`agentlightning.emitter`](../reference/agent.md).
|
||||
|
||||
## Emitting Rewards, Messages, and More
|
||||
|
||||
Here are the primary emitter functions:
|
||||
|
||||
* [`emit_reward(value: float)`][agentlightning.emit_reward]: Records an intermediate/final reward, which is a convenient wrapper of [`emit_annotation`][agentlightning.emit_annotation].
|
||||
* [`emit_annotation(attributes: Dict[str, Any])`][agentlightning.emit_annotation]: Records arbitrary metadata as a span.
|
||||
* [`emit_message(message: str)`][agentlightning.emit_message]: Records a simple log message as a span.
|
||||
* [`emit_exception(exception: BaseException)`][agentlightning.emit_exception]: Records a Python exception, including its type, message, and stack trace.
|
||||
* [`emit_object(obj: Any)`][agentlightning.emit_object]: Records any JSON-serializable object, perfect for structured data.
|
||||
|
||||
Let's first see an example of an agent using these emitters to provide detailed feedback.
|
||||
|
||||
```python
|
||||
import agentlightning as agl
|
||||
|
||||
@agl.rollout
|
||||
def multi_step_agent(task: dict, prompt_template: PromptTemplate) -> float:
|
||||
try:
|
||||
# Step 1: Initial planning
|
||||
agl.emit_message("Starting planning phase.")
|
||||
plan = generate_plan(task, prompt_template)
|
||||
agl.emit_object({"plan_steps": len(plan), "first_step": plan[0]})
|
||||
|
||||
# Award a small reward for a valid plan
|
||||
plan_reward = grade_plan(plan)
|
||||
agl.emit_reward(plan_reward)
|
||||
|
||||
# Step 2: Execute the plan
|
||||
agl.emit_message(f"Executing {len(plan)}-step plan.")
|
||||
execution_result = execute_plan(plan)
|
||||
|
||||
# Step 3: Final evaluation
|
||||
final_reward = custom_grade_final_result(execution_result, task["expected_output"])
|
||||
|
||||
# The return value is treated as the final reward for the rollout
|
||||
return final_reward
|
||||
|
||||
except ValueError as e:
|
||||
# Record the specific error and return a failure reward
|
||||
agl.emit_exception(e)
|
||||
return 0.0
|
||||
```
|
||||
|
||||
Each helper accepts nested `attributes` (or keyword arguments for [`operation`][agentlightning.operation]) and automatically flattens/sanitizes them into dotted OpenTelemetry keys. This means you can pass ordinary dictionaries/lists without pre-processing and still get consistent attribute names such as `meta.any_attribute` across all emitter operations. Agent-lightning does not restrict the attributes you supply, but it is best to consult [OpenTelemetry's semantic conventions](https://opentelemetry.io/docs/specs/semconv/) for recommended names. Agent-lightning also defines [specific semconv](../reference/semconv.md) for its own use cases. The pattern looks like this:
|
||||
|
||||
```python
|
||||
from opentelemetry.semconv.attributes import server_attributes
|
||||
from agentlightning import emit_object
|
||||
|
||||
emit_object({
|
||||
"name": "John Doe",
|
||||
"age": 30,
|
||||
"email": "john.doe@example.com",
|
||||
}, attributes={
|
||||
server_attributes.SERVER_ADDRESS: "127.0.0.1",
|
||||
server_attributes.SERVER_PORT: 8080,
|
||||
})
|
||||
```
|
||||
|
||||
Running the above code sends the following span to the backend if you have a tracer active:
|
||||
|
||||
```text
|
||||
Span(
|
||||
name='agentlightning.object',
|
||||
attributes={
|
||||
'agentlightning.object.type': 'dict',
|
||||
'agentlightning.object.json': '{"name": "John Doe", "age": 30, "email": "john.doe@example.com"}',
|
||||
'server.address': '127.0.0.1',
|
||||
'server.port': 8080
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
!!! tip
|
||||
|
||||
If you don't have a tracer active, the above code will raise the following error:
|
||||
|
||||
```text
|
||||
RuntimeError: No active tracer found. Cannot emit object span.
|
||||
```
|
||||
|
||||
By default, emitter helpers delegate to the active tracer to create and export spans (specifically via [`Tracer.create_span`][agentlightning.Tracer.create_span]). If you want to emit spans without an active tracer, set `propagate=False` to keep the span local — a useful option for offline tests. The default `True` streams spans through the active tracer/exporters.
|
||||
|
||||
When working with [agentlightning.semconv](../reference/semconv.md), you typically use utilities such as [`make_tag_attributes`][agentlightning.utils.otel.make_tag_attributes] and [`make_link_attributes`][agentlightning.utils.otel.make_link_attributes] to build the attributes dictionary. For example:
|
||||
|
||||
```python
|
||||
from agentlightning.utils.otel import make_tag_attributes
|
||||
|
||||
emit_annotation(make_tag_attributes(["tool", "calculator", "fast", "good"]))
|
||||
```
|
||||
|
||||
The above code will send a span with the following attributes to the backend:
|
||||
|
||||
```json
|
||||
{
|
||||
"agentlightning.tag.0": "tool",
|
||||
"agentlightning.tag.1": "calculator",
|
||||
"agentlightning.tag.2": "fast",
|
||||
"agentlightning.tag.3": "good"
|
||||
}
|
||||
```
|
||||
|
||||
A counterpart utility function [`extract_tags_from_attributes`][agentlightning.utils.otel.extract_tags_from_attributes] is also available to extract the tags from the attributes dictionary.
|
||||
|
||||
## Operations
|
||||
|
||||
The [`operation`][agentlightning.operation] helper tracks logical units of work within your agent, capturing inputs, outputs, timing, and success/failure status. Unlike point-in-time emitters, operations create a span representing a time interval. Use operations for tool calls, multi-step workflows, debugging, and performance monitoring. [`operation`][agentlightning.operation] works as either a decorator or a context manager.
|
||||
|
||||
The decorator automatically captures function arguments as inputs and the return value as output:
|
||||
|
||||
```python
|
||||
import agentlightning as agl
|
||||
|
||||
@agl.operation
|
||||
def search_documents(query: str, max_results: int = 10) -> list[dict]:
|
||||
results = perform_search(query, max_results)
|
||||
return results
|
||||
|
||||
@agl.operation(category="tool", priority="high")
|
||||
def execute_calculation(expression: str) -> float:
|
||||
return eval_safely(expression)
|
||||
```
|
||||
|
||||
The example above emits a span with `{"category": "tool", "priority": "high"}` attributes. It also records the function input and output via [OPERATION_INPUT][agentlightning.semconv.LightningSpanAttributes.OPERATION_INPUT] and [OPERATION_OUTPUT][agentlightning.semconv.LightningSpanAttributes.OPERATION_OUTPUT]. It works with async functions too:
|
||||
|
||||
```python
|
||||
@agl.operation
|
||||
async def async_api_call(endpoint: str, payload: dict) -> dict:
|
||||
response = await http_client.post(endpoint, json=payload)
|
||||
return response.json()
|
||||
```
|
||||
|
||||
Override the operation name if needed:
|
||||
|
||||
```python
|
||||
@agl.operation(name="custom-name")
|
||||
def any_weird_name_i_dont_want():
|
||||
pass
|
||||
```
|
||||
|
||||
For more control, [`operation`][agentlightning.operation] can also be used as a context manager to explicitly record inputs and outputs:
|
||||
|
||||
```python
|
||||
with agl.operation(tool_name="web_search") as op:
|
||||
op.set_input(query="latest AI research", filters={"date": "2024"})
|
||||
results = search_web("latest AI research", {"date": "2024"})
|
||||
op.set_output({"result_count": len(results), "top_result": results[0]})
|
||||
```
|
||||
|
||||
The `propagate=False` flag also applies to [`operation`][agentlightning.operation] when you want to keep operations local without requiring an active tracer:
|
||||
|
||||
```python
|
||||
@agl.operation(propagate=False)
|
||||
def local_test():
|
||||
return "Not sent to backend"
|
||||
```
|
||||
|
||||
## Linking to Other Spans
|
||||
|
||||
Sometimes a span should explicitly point back to another span that produced the input it is working on (for example, linking a reward annotation to the [`agentlightning.operation`][agentlightning.operation] span that generated a response). Agent-lightning encodes these relationships through flattened link attributes. The helper [`make_link_attributes`][agentlightning.utils.otel.make_link_attributes] converts a dictionary of keys such as `trace_id`, `span_id`, or any custom attribute into the `"agentlightning.link.*"` ([LightningSpanAttributes.LINK][agentlightning.semconv.LightningSpanAttributes.LINK]) fields expected by the backend. Later, [`query_linked_spans`][agentlightning.utils.otel.query_linked_spans] can recover the original span(s) from those link descriptors.
|
||||
|
||||
```python
|
||||
import opentelemetry.trace as trace_api
|
||||
from agentlightning import emit_annotation, operation
|
||||
from agentlightning.utils.otel import make_link_attributes, make_tag_attributes
|
||||
|
||||
with operation(conversation_id="chat-42") as op:
|
||||
# ... perform the work ...
|
||||
link_attrs = make_link_attributes({
|
||||
"conversation_id": "chat-42",
|
||||
})
|
||||
|
||||
emit_annotation(
|
||||
{
|
||||
**link_attrs,
|
||||
**make_tag_attributes(["reward", "good"]),
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
When analyzing in adapters, pass the extracted link models to [`query_linked_spans`][agentlightning.utils.otel.query_linked_spans] to retrieve the matching span(s):
|
||||
|
||||
```python
|
||||
from agentlightning.utils.otel import extract_links_from_attributes, query_linked_spans
|
||||
|
||||
annotation_span = ... # Span from your trace store
|
||||
operation_spans = [...] # list of spans you want to search
|
||||
|
||||
link_models = extract_links_from_attributes(annotation_span.attributes)
|
||||
matches = query_linked_spans(operation_spans, link_models)
|
||||
assert matches # Contains the original operation span
|
||||
```
|
||||
|
||||
!!! tip "Correlating Rewards with LLM Requests"
|
||||
|
||||
[Tracer](./traces.md) instruments each request/response as its own span. You can link to the [`gen_ai.response.id`](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-events/) attribute, which comes from the LLM response ID.
|
||||
|
||||
```python
|
||||
from agentlightning import emit_reward
|
||||
from agentlightning.utils.otel import make_link_attributes
|
||||
|
||||
result = call_llm(prompt)
|
||||
reward_links = make_link_attributes({"gen_ai.response.id": result.id})
|
||||
emit_reward(0.9, attributes=reward_links)
|
||||
```
|
||||
|
||||
Later, use the same `gen_ai.response.id` key inside `query_linked_spans` to find the reward(s) that reference that specific LLM request span.
|
||||
@@ -0,0 +1,190 @@
|
||||
# Installation Guide
|
||||
|
||||
This guide explains how to install **Agent-Lightning**. You can install it from **PyPI** (the Python Package Index) for general use or directly from the **source code** if you plan to contribute or need fine-grained control over dependencies.
|
||||
|
||||
!!! info "Platform and Hardware Requirements"
|
||||
Agent-Lightning is officially supported on **Linux distributions** (Ubuntu 22.04 or later is recommended).
|
||||
At the moment **macOS and Windows** (outside of WSL2) are not supported.
|
||||
|
||||
The Python runtime must be **Python 3.10 or newer**. We recommend using the latest patch release of Python 3.10, 3.11, or 3.12 to pick up performance and security updates.
|
||||
|
||||
A **GPU is optional**—you only need CUDA-capable hardware if you plan to fine-tune model weights or run GPU-accelerated workloads. CPU-only environments are fully supported for evaluation and inference.
|
||||
|
||||
## Installing from PyPI
|
||||
|
||||
The easiest way to get started is by installing Agent-Lightning directly from PyPI. This ensures you get the latest **stable release** of the package, tested for compatibility and reliability.
|
||||
|
||||
### Install the Stable Release
|
||||
|
||||
Run the following command in your terminal:
|
||||
|
||||
```bash
|
||||
pip install --upgrade agentlightning
|
||||
```
|
||||
|
||||
This installs or upgrades Agent-Lightning to the newest stable version.
|
||||
|
||||
!!! tip
|
||||
|
||||
If you intend to use **Agent-Lightning** with [**VERL**](../algorithm-zoo/verl.md) or run any of its **example scripts**, you’ll need to install some additional dependencies.
|
||||
See the sections on [Algorithm-specific installation](#algorithm-specific-installation) and [Example-specific installation](#example-specific-installation) for details.
|
||||
|
||||
### Install the Nightly Build (Latest Features)
|
||||
|
||||
Agent-Lightning also publishes **nightly builds**, which contain the latest experimental features and improvements from the main branch. These are available via **Test PyPI**.
|
||||
|
||||
```bash
|
||||
pip install --upgrade --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ --pre agentlightning
|
||||
```
|
||||
|
||||
!!! warning
|
||||
The nightly builds are cutting-edge but may include unstable or untested changes.
|
||||
Use them **at your own risk**, especially in production environments.
|
||||
|
||||
## Algorithm-specific Installation
|
||||
|
||||
Agent-Lightning supports multiple learning algorithms. Some of them like [APO](../algorithm-zoo/apo.md) or [VERL](../algorithm-zoo/verl.md) require extra dependencies. You can install them automatically using **optional extras** or manually if you prefer finer control.
|
||||
|
||||
### Installing APO
|
||||
|
||||
[APO](../algorithm-zoo/apo.md) is an algorithm module that depends on libraries such as [POML](https://github.com/microsoft/POML).
|
||||
You can install Agent-Lightning with APO support by running:
|
||||
|
||||
```bash
|
||||
pip install agentlightning[apo]
|
||||
```
|
||||
|
||||
!!! warning
|
||||
APO also depends on the [OpenAI Python SDK](https://github.com/openai/openai-python), version **2.0 or newer**.
|
||||
Ensure your SDK version is up to date to avoid compatibility issues.
|
||||
|
||||
### Installing VERL
|
||||
|
||||
[VERL](../algorithm-zoo/verl.md) integrates with libraries like **PyTorch**, **vLLM**, and **VERL framework**.
|
||||
Although you *can* install all dependencies automatically, we recommend doing it manually to avoid version conflicts.
|
||||
|
||||
```bash
|
||||
pip install agentlightning[verl]
|
||||
```
|
||||
|
||||
!!! tip "Recommended Manual Setup (More Stable)"
|
||||
Automated installation may cause issues if you don’t have a compatible **PyTorch** or **CUDA** version preinstalled.
|
||||
For a more stable setup, install dependencies step-by-step:
|
||||
|
||||
```bash
|
||||
pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu128
|
||||
pip install flash-attn --no-build-isolation
|
||||
pip install vllm==0.10.2
|
||||
pip install verl==0.5.0
|
||||
```
|
||||
|
||||
This approach ensures compatibility with CUDA 12.8 and minimizes dependency conflicts.
|
||||
|
||||
## Example-specific Installation
|
||||
|
||||
Each example in the `examples/` directory may have its own additional dependencies.
|
||||
Please refer to the **README** file of each example for detailed setup instructions:
|
||||
|
||||
[See Example READMEs]({{ src("examples") }}).
|
||||
|
||||
## Installing from Source (for Developers and Contributors)
|
||||
|
||||
If you plan to contribute to Agent-Lightning or prefer to work with the latest development code, install it directly from the **source repository**.
|
||||
|
||||
### Why Install from Source?
|
||||
|
||||
* You want to **modify or contribute** to the project.
|
||||
* You prefer an **isolated development environment**.
|
||||
* You want to test unreleased features or fix bugs locally.
|
||||
|
||||
### Using `uv` for Dependency Management
|
||||
|
||||
Starting with version **0.2**, Agent-Lightning uses [`uv`](https://docs.astral.sh/uv/) as its **default dependency manager**.
|
||||
|
||||
`uv` is a fast and safe alternative to `pip` that:
|
||||
|
||||
* Installs packages **in seconds** (instead of minutes),
|
||||
* Prevents **dependency conflicts**,
|
||||
* Supports **grouped dependencies** for optional features.
|
||||
|
||||
Before proceeding, make sure `uv` is installed.
|
||||
|
||||
### Minimal Developer Installation
|
||||
|
||||
```bash
|
||||
git clone https://github.com/microsoft/agent-lightning
|
||||
cd agent-lightning
|
||||
uv sync --group dev
|
||||
```
|
||||
|
||||
This command sets up a clean development environment with only the essential dependencies.
|
||||
|
||||
### Installing All Extras (CPU or GPU)
|
||||
|
||||
`uv sync` can also handle algorithm-specific and example-specific dependencies in one step.
|
||||
|
||||
For a CPU-only machine:
|
||||
|
||||
```bash
|
||||
uv sync --frozen \
|
||||
--extra apo \
|
||||
--extra verl \
|
||||
--group dev \
|
||||
--group torch-cpu \
|
||||
--group torch-stable \
|
||||
--group trl \
|
||||
--group agents \
|
||||
--no-default-groups
|
||||
```
|
||||
|
||||
For a GPU-equipped machine that is CUDA 12.8 compatible:
|
||||
|
||||
```bash
|
||||
uv sync --frozen \
|
||||
--extra apo \
|
||||
--extra verl \
|
||||
--group dev \
|
||||
--group torch-gpu-stable \
|
||||
--group trl \
|
||||
--group agents \
|
||||
--no-default-groups
|
||||
```
|
||||
|
||||
Read more about Agent-lightning managed dependency groups [here]({{ src("pyproject.toml") }}).
|
||||
|
||||
### Building the Dashboard
|
||||
|
||||
The Agent-Lightning dashboard is built using [Vite](https://vite.dev/). To build the dashboard, run the following command:
|
||||
|
||||
```bash
|
||||
cd dashboard
|
||||
npm ci
|
||||
npm run build
|
||||
```
|
||||
|
||||
Some HTML and JavaScript assets will be generated in the `agentlightning/dashboard` directory.
|
||||
|
||||
### Activating Your Environment
|
||||
|
||||
After syncing dependencies, `uv` automatically creates a virtual environment inside the `.venv/` directory.
|
||||
|
||||
You can use it in two ways:
|
||||
|
||||
```bash
|
||||
# Option 1: Prefix commands with uv run
|
||||
uv run python your_script.py
|
||||
|
||||
# Option 2: Activate the virtual environment
|
||||
source .venv/bin/activate
|
||||
python your_script.py
|
||||
```
|
||||
|
||||
!!! warning "Before Contributing"
|
||||
|
||||
Agent-Lightning enforces code style and linting rules via **pre-commit hooks**.
|
||||
Installing them early prevents many avoidable formatting issues.
|
||||
|
||||
```bash
|
||||
uv run pre-commit install
|
||||
uv run pre-commit run --all-files --show-diff-on-failure --color=always
|
||||
```
|
||||
@@ -0,0 +1,334 @@
|
||||
# Scaling out Agent-lightning
|
||||
|
||||
Agent-lightning splits training into an **algorithm bundle** and a **runner bundle** that exchange work through the [`LightningStore`][agentlightning.LightningStore]. This tutorial shows how to increase rollout throughput, place bundles across processes or machines, and keep the algorithm side scalable with external frameworks.
|
||||
|
||||
## Parallelizing Rollouts with [`Trainer`][agentlightning.Trainer]
|
||||
|
||||
Before we dive into the details of the bundles and execution strategies, let's first revisit how to parallelize rollouts with [`Trainer`][agentlightning.Trainer].
|
||||
|
||||
[`Trainer`][agentlightning.Trainer] is the quickest way to dial up parallelism. Even when `n_runners = 1`, calling [`Trainer.fit`][agentlightning.Trainer.fit] runs the algorithm and runners in parallel. The algorithm enqueues rollouts; runners dequeue them and execute your [`LitAgent`][agentlightning.LitAgent], and the algorithm collects spans via its [`Adapter`][agentlightning.Adapter] before scheduling the next batch.
|
||||
|
||||
!!! note
|
||||
|
||||
One of the most important features of [`Trainer`][agentlightning.Trainer] is the ability to abort things gracefully. For example, if you press `Ctrl+C` in the terminal, the algorithm will abort and the runners will stop executing. If the algorithm crashes, the runners will also stop executing.
|
||||
|
||||
Increase throughput by setting `n_runners` when constructing the trainer. The following example comes from [train_calc_agent.py]({{ src("examples/calc_x/train_calc_agent.py") }}). Since backend LLMs usually use techniques like [continuous batching](https://docs.vllm.ai/en/latest/) to increase throughput, you do not have to worry about overwhelming the backend with too many requests.
|
||||
|
||||
```python
|
||||
import agentlightning as agl
|
||||
from datasets import Dataset as HFDataset
|
||||
from calc_agent import calc_agent
|
||||
|
||||
train_dataset = HFDataset.from_parquet("data/train.parquet").to_list()
|
||||
val_dataset = HFDataset.from_parquet("data/test.parquet").to_list()
|
||||
|
||||
algorithm = agl.VERL(verl_config)
|
||||
|
||||
trainer = agl.Trainer(
|
||||
algorithm=algorithm,
|
||||
n_runners=8, # launch eight rollout workers
|
||||
tracer=agl.OtelTracer(),
|
||||
adapter=agl.LlmProxyTraceToTriplet(),
|
||||
)
|
||||
|
||||
trainer.fit(calc_agent, train_dataset=train_dataset, val_dataset=val_dataset)
|
||||
```
|
||||
|
||||
In [`Trainer`][agentlightning.Trainer], there are multiple other initialization parameters that you can use to customize the training process. For example, you can use `max_rollouts` to keep smoke tests short. Pass a concrete [`LightningStore`][agentlightning.LightningStore] instance when you need persistence or want to share the queue across multiple scripts.
|
||||
|
||||
!!! tip
|
||||
|
||||
Before scaling out, run [`Trainer.dev()`][agentlightning.Trainer.dev] with `n_runners=1` to verify the rollout logic and spans without burning GPU hours.
|
||||
|
||||
## Bundles and Execution Strategies
|
||||
|
||||
When [`Trainer`][agentlightning.Trainer] starts, it packages its configuration into two callable **bundles**:
|
||||
|
||||
|
||||
|
||||
The **algorithm bundle** wraps your [`Algorithm`][agentlightning.Algorithm], adapter, and any LLM proxy into a single callable that can be aborted via a signal event.
|
||||
|
||||
```python
|
||||
async def algorithm_bundle(store: LightningStore, event: ExecutionEvent) -> None:
|
||||
...
|
||||
```
|
||||
|
||||
The **runner bundle** wraps the [`Runner`][agentlightning.Runner], tracer, hooks, and agent into a single callable that can be aborted via a signal event. Unlike the algorithm bundle, the runner bundle is expected to be replicated.
|
||||
|
||||
```python
|
||||
async def runner_bundle(store: LightningStore, worker_id: int, event: ExecutionEvent) -> None:
|
||||
...
|
||||
```
|
||||
|
||||
An **execution strategy** then decides where those bundles are placed (threads vs processes vs multiple machines), how many runner replicas to launch, and how lifecycle events such as shutdown are coordinated.
|
||||
|
||||
By default, the trainer builds an [`InMemoryLightningStore`][agentlightning.InMemoryLightningStore] if you do not provide one. Because that store has no locking or cross-process transport, the execution strategy is the component that wraps it in thread-safe or HTTP-safe facades ([`LightningStoreThreaded`][agentlightning.LightningStoreThreaded], [`LightningStoreServer`][agentlightning.LightningStoreServer]) before handing it to bundles. For a deeper look at these facades, see [Understanding the Store](../deep-dive/store.md) and [Birds' Eye View](../deep-dive/birds-eye-view.md).
|
||||
|
||||
Agent-lightning provides two built-in execution strategies: [`SharedMemoryExecutionStrategy`][agentlightning.SharedMemoryExecutionStrategy] and [`ClientServerExecutionStrategy`][agentlightning.ClientServerExecutionStrategy]. You can pass a string alias, a configuration dictionary, or a pre-built strategy instance:
|
||||
|
||||
```python
|
||||
import agentlightning as agl
|
||||
|
||||
algorithm = agl.Baseline()
|
||||
|
||||
# Short alias for the shared-memory strategy.
|
||||
# Because the runner lives on the main thread in this mode,
|
||||
# n_runners must be 1 unless you move the algorithm to the main thread.
|
||||
trainer = agl.Trainer(algorithm=algorithm, n_runners=1, strategy="shm")
|
||||
|
||||
# Dict with overrides; keep the algorithm on the main thread so multiple runner threads can spawn.
|
||||
# Specifying `n_runners` inside strategy is equivalent to passing `n_runners` to the trainer.
|
||||
trainer = agl.Trainer(
|
||||
algorithm=algorithm,
|
||||
strategy={
|
||||
"type": "shm",
|
||||
"n_runners": 8,
|
||||
"main_thread": "algorithm",
|
||||
},
|
||||
)
|
||||
|
||||
# Pass an existing strategy instance – Trainer respects the strategy's own `n_runners`.
|
||||
strategy = agl.SharedMemoryExecutionStrategy(main_thread="algorithm", n_runners=4)
|
||||
trainer = agl.Trainer(algorithm=algorithm, strategy=strategy)
|
||||
```
|
||||
|
||||
If you omit the strategy, the trainer defaults to `ClientServerExecutionStrategy(n_runners=trainer.n_runners)`. You can still re-specify the client-server strategy through aliases or configuration to tweak ports and other settings:
|
||||
|
||||
```python
|
||||
trainer = agl.Trainer(
|
||||
algorithm=algorithm,
|
||||
n_runners=8,
|
||||
strategy={"type": "cs", "server_port": 9999},
|
||||
)
|
||||
```
|
||||
|
||||
Environment variables give you another layer of control. For example:
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
os.environ["AGL_SERVER_PORT"] = "10000"
|
||||
os.environ["AGL_CURRENT_ROLE"] = "algorithm"
|
||||
os.environ["AGL_MANAGED_STORE"] = "0"
|
||||
|
||||
trainer = agl.Trainer(algorithm=algorithm, n_runners=8, strategy="cs")
|
||||
```
|
||||
|
||||
The resulting [`ClientServerExecutionStrategy`][agentlightning.ClientServerExecutionStrategy] picks up the port, role, and managed-store flag from the environment.
|
||||
|
||||
!!! tip
|
||||
|
||||
The same configuration patterns apply to other trainer components. For example,
|
||||
```python
|
||||
trainer = agl.Trainer(algorithm=algorithm, tracer=agl.OtelTracer())
|
||||
```
|
||||
wires in a custom tracer, while
|
||||
```python
|
||||
trainer = agl.Trainer(algorithm=algorithm, adapter="agentlightning.adapter.TraceToMessages")
|
||||
```
|
||||
swaps in a different adapter. Passing a dict lets you tweak the init parameters of defaults without naming the class explicitly:
|
||||
|
||||
```python
|
||||
trainer = agl.Trainer(
|
||||
algorithm=algorithm,
|
||||
adapter={"agent_match": "plan_agent", "repair_hierarchy": False},
|
||||
)
|
||||
```
|
||||
|
||||
The next sections walk through the two built-in strategies and how they affect placement and store access.
|
||||
|
||||
## Client-server Architecture
|
||||
|
||||
The default [`ClientServerExecutionStrategy`][agentlightning.ClientServerExecutionStrategy] starts a [`LightningStoreServer`][agentlightning.LightningStoreServer] alongside the algorithm and spawns runner processes that talk to it through [`LightningStoreClient`][agentlightning.LightningStoreClient]. All runners share the HTTP endpoint, so the queue and spans stay consistent across processes or machines.
|
||||
|
||||
If you simply instantiate [`Trainer`][agentlightning.Trainer] (as above), it will send the algorithm bundle and runner bundle to [`ClientServerExecutionStrategy`][agentlightning.ClientServerExecutionStrategy], which will then:
|
||||
|
||||
1. Launch \(N+1\) processes: \(N\) runner processes and 1 algorithm process (one of them could live in the main process).
|
||||
2. The algorithm process will take the store received from [`Trainer`][agentlightning.Trainer], wrap it in a [`LightningStoreServer`][agentlightning.LightningStoreServer], and start serving it over HTTP.
|
||||
3. The runner processes discard the store and create a new store, which is a client that connects to the algorithm process through [`LightningStoreClient`][agentlightning.LightningStoreClient], and start executing the runner bundle.
|
||||
4. The strategy automatically escalates shutdown (cooperative stop → `SIGINT` → `terminate()` → `kill()`) so long-running runners do not linger.
|
||||
|
||||
You can override server placement or ports, and whether to automatically wrap the store, through constructor arguments or environment variables:
|
||||
|
||||
```python
|
||||
trainer = agl.Trainer(
|
||||
algorithm=algorithm,
|
||||
n_runners=1,
|
||||
strategy={
|
||||
"type": "cs",
|
||||
"server_host": "0.0.0.0",
|
||||
"server_port": 9999,
|
||||
"main_process": "runner",
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
Set `AGL_SERVER_HOST` and `AGL_SERVER_PORT` if you prefer environment-based configuration. You can also use `AGL_MANAGED_STORE` if you do not want the execution strategy to wrap the store for you. An example is shown in [Debugging with External Store][debug-with-external-store].
|
||||
|
||||
Algorithms sometimes require heterogeneous computation resources, such as GPU accelerators, while runners sometimes require a specific environment to run because many agent frameworks are fragile in their dependencies. A role-based launch pattern helps you place the algorithm on a dedicated machine with more GPU memory, while runners can live on another machine with more flexible dependencies. This is possible via `AGL_CURRENT_ROLE="algorithm"` or `AGL_CURRENT_ROLE="runner"` environment variables. When running on different machines, you also need to set `AGL_SERVER_HOST` and `AGL_SERVER_PORT` to the IP address and port of the algorithm machine. You might recognize that this convention is very similar to `MASTER_ADDR` and `MASTER_PORT` in [PyTorch distributed training](https://docs.pytorch.org/docs/stable/notes/ddp.html).
|
||||
|
||||
### Launching Algorithm and Runner Roles on Separate Machines
|
||||
|
||||
When you want to stretch the algorithm onto a GPU-rich machine and keep rollout workers close to the data source (or on machines with a more permissive dependency stack), launch the same training script in different terminals with role-specific environment variables. The client–server strategy will route each process to the right side of the queue as long as they share the same `AGL_SERVER_HOST`/`AGL_SERVER_PORT` pair.
|
||||
|
||||
**1. Pick an address and port for the store.** Decide which machine will host the algorithm. Choose a TCP port that can be reached by the runner machines (for example, open it in your firewall configuration). In this example we will use `10.0.0.4:4747`.
|
||||
|
||||
**2. Start the algorithm process.** On the machine that should run the algorithm, expose the store by binding to all network interfaces and mark the role as `algorithm`.
|
||||
|
||||
```bash
|
||||
export AGL_SERVER_HOST=0.0.0.0
|
||||
export AGL_SERVER_PORT=4747
|
||||
export AGL_CURRENT_ROLE=algorithm
|
||||
|
||||
python train_calc_agent.py
|
||||
```
|
||||
|
||||
Leaving `AGL_MANAGED_STORE` unset (or setting it to `1`) lets the strategy create the [`LightningStoreServer`][agentlightning.LightningStoreServer] for you. Otherwise, you can use the method in the previous section to create a store on your own.
|
||||
|
||||
**3. Start rollout workers on remote machines.** Every runner machine should point to the algorithm host and declare itself as the `runner` role. You can start multiple processes per machine or repeat the command on additional hosts.
|
||||
|
||||
```bash
|
||||
export AGL_SERVER_HOST=10.0.0.4
|
||||
export AGL_SERVER_PORT=4747
|
||||
export AGL_CURRENT_ROLE=runner
|
||||
python train_calc_agent.py --n-runners 4
|
||||
```
|
||||
|
||||
The runner process automatically connects via [`LightningStoreClient`][agentlightning.LightningStoreClient]. Adjust `--n-runners` to spawn the desired number of worker processes on that machine.
|
||||
|
||||
**4. Scale out as needed.** Repeat step 3 on as many machines as you need. When you are done, stop the algorithm process. However, since the runners are on different machines, the strategy WILL NOT send a cooperative stop signal to the connected runners. So you need to kill the runners on your own.
|
||||
|
||||
This role-based launch mirrors what [`Trainer.fit`][agentlightning.Trainer.fit] does inside a single machine while letting you spread work across a fleet. Because every process shares the same training script, you keep a single source of truth for dataset loading, adapters, and tracers, but you can tune compute resources independently for the algorithm and rollout workers.
|
||||
|
||||
### Shared-memory Strategy
|
||||
|
||||
[`SharedMemoryExecutionStrategy`][agentlightning.SharedMemoryExecutionStrategy] keeps everything inside one process. The runner runs on the main thread (by default) while the algorithm lives on a Python thread guarded by [`LightningStoreThreaded`][agentlightning.LightningStoreThreaded].
|
||||
|
||||
Use it when you want easier debugging with shared breakpoints and no serialization overhead, or minimal startup time for unit tests. It's not a good choice for many algorithms that require heavy model training because [`LightningStoreThreaded`][agentlightning.LightningStoreThreaded] does not work for multiprocessing. Using it with multiprocessing algorithms will lead to undefined behavior.
|
||||
|
||||
Sample configuration:
|
||||
|
||||
```python
|
||||
trainer = agl.Trainer(
|
||||
algorithm=algorithm,
|
||||
strategy="shm",
|
||||
)
|
||||
```
|
||||
|
||||
You can further customize the init parameters of [`SharedMemoryExecutionStrategy`][agentlightning.SharedMemoryExecutionStrategy]. With `main_thread="runner"`, the runner occupies the main thread and `n_runners` must be `1`. The strategy respects `AGL_MANAGED_STORE`; set it to `0` to opt out of the `LightningStoreThreaded` wrapper.
|
||||
|
||||
## Parallelizing Algorithms
|
||||
|
||||
Runner parallelism scales rollout throughput, but the algorithm loop remains a single-process loop inside the execution strategy. We understand that many algorithms have parallelization built in, but that's outside the parallelization scope of Agent-lightning.
|
||||
|
||||
Agent-lightning strives to make algorithms’ own parallelization work well under our execution strategies. The biggest challenge turns out to come from the store. For example, [`VERL`][agentlightning.algorithm.verl.VERL] uses [Ray](https://www.ray.io/) and launches [FSDP](https://docs.pytorch.org/tutorials/intermediate/FSDP_tutorial.html) and [vLLM](https://vllm.ai/) components internally. [`ClientServerExecutionStrategy`][agentlightning.ClientServerExecutionStrategy] has to make sure that the server is not simultaneously serving in multiple processes or Ray workers, and that there is only one single authoritative source of truth for all subprocesses to connect to. Subprocesses connect to the store via a small [`LightningStoreClient`][agentlightning.LightningStoreClient] bundled within [`LightningStoreServer`][agentlightning.LightningStoreServer].
|
||||
|
||||
!!! note
|
||||
|
||||
The [birds' eye view][birds-eye-view-client-server-strategy] illustrates how adapters, proxies, and stores interact when the algorithm spawns additional workers. Use that diagram as a checklist when introducing new distributed components.
|
||||
|
||||
## Parallelizing [`LightningStore`][agentlightning.LightningStore]
|
||||
|
||||
By default, Agent-lightning persists rollouts and spans in an in-memory store. [`Trainer.fit`][agentlightning.Trainer.fit] spins it up automatically, or you can launch it yourself via the [`agl store` command](../reference/cli.md). [`InMemoryLightningStore`][agentlightning.InMemoryLightningStore] keeps all state inside the current process, which makes local iteration fast but introduces two production constraints:
|
||||
|
||||
1. Spans are evicted once the process crosses its memory cap, so long runs risk data loss unless the host has abundant RAM.
|
||||
2. Although the store is well optimized via asynchronous programming, the store lives in a single process and remains bound by the GIL, preventing it from saturating multi-core machines.
|
||||
|
||||
!!! note "General note for all server-client stores"
|
||||
|
||||
If your algorithm and runners communicate through HTTP protocol (which should be the default for 99% of the cases), you need to ensure the file limit is sufficiently large to avoid the "Too many open files" error. You can set the file limit by running the following command:
|
||||
|
||||
```bash
|
||||
ulimit -n 100000
|
||||
```
|
||||
|
||||
For resilient runs, switch to a persistent backend such as [`MongoLightningStore`][agentlightning.store.mongo.MongoLightningStore], which writes data to MongoDB instead of local RAM. Agent-lightning relies on [pymongo](https://pymongo.readthedocs.io/en/stable/) to interact with MongoDB, which can be installed via:
|
||||
|
||||
```bash
|
||||
pip install agentlightning[mongo]
|
||||
```
|
||||
|
||||
To use the MongoDB store, you need to pass the MongoDB URI to the store constructor. The URI should be in the format of `mongodb://<host>:<port>/<database>?replicaSet=<replicaSet>`.
|
||||
|
||||
```python
|
||||
from agentlightning.store.mongo import MongoLightningStore
|
||||
|
||||
trainer = agl.Trainer(
|
||||
algorithm=algorithm,
|
||||
store=MongoLightningStore(mongo_uri="mongodb://localhost:27017/?replicaSet=rs0"),
|
||||
)
|
||||
```
|
||||
|
||||
!!! tip "Setting up MongoDB"
|
||||
|
||||
MongoDB is a popular document-oriented database. Before running Agent-lightning with [`MongoLightningStore`][agentlightning.store.mongo.MongoLightningStore], make sure that you've already had a MongoDB instance running. Setting up can be conveniently done via Docker Compose via [compose.mongo.yml]({{ src("docker/compose.mongo.yml") }}). Unless targeting serious production use, we recommend creating the data folders and setting them to `777` permission to avoid permission issues.
|
||||
|
||||
```bash
|
||||
mkdir -p data/mongo-host
|
||||
chmod 777 data/mongo-host
|
||||
docker compose -f compose.mongo.yml up -d
|
||||
```
|
||||
|
||||
Alternatively, you can also install MongoDB manually following the [official documentation](https://www.mongodb.com/docs/manual/installation/). If you installed MongoDB manually, an important note is that you need to ensure that the MongoDB instance has enabled replica set feature, since Agent-lightning uses the transactional operations internally. The simplest approach is to use the following script (executed in the MongoDB shell) to initialize the replica set:
|
||||
|
||||
```javascript
|
||||
rs.initiate({
|
||||
_id: "rs0",
|
||||
members: [{ _id: 0, host: "localhost:27017" }],
|
||||
});
|
||||
```
|
||||
|
||||
To scale out further, launch the store server via [`agl store --backend mongo`](../reference/cli.md) (see [Debugging with External Store][debug-with-external-store]). The CLI accepts `--n-workers`, which starts the server under `gunicorn` with multiple worker processes so concurrent runners can push and pull at higher throughput. This option applies only to persistent backends; an in-memory store, on the other hand, cannot be sharded across workers because its state lives inside one process.
|
||||
|
||||
!!! note
|
||||
|
||||
The `--n-workers` here is the number of worker processes for the store server, NOT related to the number of rollout runners.
|
||||
|
||||
## Increasing Throughput of LLM Proxy
|
||||
|
||||
Agent-lightning includes an optional [`LLMProxy`][agentlightning.LLMProxy] that wraps [LiteLLM](https://docs.litellm.ai/) to provide a unified OpenAI-compatible endpoint for your agents. When rollout throughput increases, the proxy can become a bottleneck. You can scale it out using the same pattern as the store server.
|
||||
|
||||
To increase proxy throughput, pass `num_workers` when constructing the proxy:
|
||||
|
||||
```python
|
||||
import agentlightning as agl
|
||||
|
||||
proxy = agl.LLMProxy(
|
||||
port=4000,
|
||||
launch_mode="mp", # multiprocessing mode
|
||||
num_workers=4, # four gunicorn workers handle concurrent requests
|
||||
)
|
||||
```
|
||||
|
||||
You can also configure the proxy through [`Trainer`][agentlightning.Trainer]:
|
||||
|
||||
```python
|
||||
trainer = agl.Trainer(
|
||||
algorithm=algorithm,
|
||||
n_runners=8, # The runners here is the rollout runners, not related to LLM proxy replicas
|
||||
llm_proxy={"port": 4000, "num_workers": 4}, # launch mode is actually mp by default
|
||||
)
|
||||
```
|
||||
|
||||
When `num_workers > 1`, the launcher starts gunicorn with the specified number of worker processes. Each worker runs its own event loop, allowing the proxy to handle many concurrent LLM requests without being blocked by Python's GIL.
|
||||
|
||||
!!! tip
|
||||
|
||||
When using `mp` launch mode, [`LLMProxy`][agentlightning.LLMProxy] will start the server in a separate process. To make sure the proxy is still accessing the same store as the main process, you need to set the store to be [zero-copy compatible][store-capabilities], which means, either the store is a native zero-copy store like [`MongoLightningStore`][agentlightning.store.mongo.MongoLightningStore] or the store is wrapped via [`LightningStoreServer`][agentlightning.LightningStoreServer] or [`LightningStoreClient`][agentlightning.LightningStoreClient].
|
||||
|
||||
!!! note "Shared Server Infrastructure"
|
||||
|
||||
Both [`LightningStoreServer`][agentlightning.LightningStoreServer] and [`LLMProxy`][agentlightning.LLMProxy] rely on a common utility called [`PythonServerLauncherArgs`][agentlightning.utils.server_launcher.PythonServerLauncherArgs]. This dataclass captures the settings needed to launch a FastAPI application:
|
||||
|
||||
```python
|
||||
from agentlightning.utils import PythonServerLauncherArgs
|
||||
|
||||
args = PythonServerLauncherArgs(
|
||||
port=8000,
|
||||
host="0.0.0.0",
|
||||
n_workers=4, # spawn 4 gunicorn workers
|
||||
launch_mode="thread", # or "mp" for multiprocessing, "asyncio" for in-loop
|
||||
)
|
||||
```
|
||||
|
||||
Under the hood, [`PythonServerLauncher`][agentlightning.utils.server_launcher.PythonServerLauncher] reads these arguments and chooses between uvicorn (single worker) and gunicorn (multiple workers) automatically.
|
||||
@@ -0,0 +1,133 @@
|
||||
# Working with Traces
|
||||
|
||||
Tracing is the secret capability that lets Agent-lightning train almost any agent without rewriting its core logic. The idea was born in observability tooling inside LLMOps workflows and, in Agent-lightning, evolved into a first-class primitive inside the learning loop. Beyond helping you understand what happened inside a rollout, traces provide reward spans and other learning signals that power reinforcement learning and fine-tuning algorithms.
|
||||
|
||||
|
||||
|
||||
Agent-lightning stores every recorded operation as a [`Span`][agentlightning.Span] inside a [`LightningStore`][agentlightning.LightningStore]. The naming comes from [OpenTelemetry spans](https://opentelemetry.io/docs/concepts/signals/traces/), shown in the screenshot above. A span can represent an LLM call, a tool invocation, a graph edge, an explicit reward emission, or an arbitrary Python code block. Spans form a tree where parent spans describe higher-level steps and children record the detailed work. The sections below walk through how spans are produced and how to interpret them once they reach the store.
|
||||
|
||||
## Writing Spans
|
||||
|
||||
Most [`Runner`][agentlightning.Runner] implementations wire a [`Tracer`][agentlightning.Tracer] into the agent’s lifecycle. The tracer is responsible for installing instrumentation, buffering OpenTelemetry spans, and committing them to the [`LightningStore`][agentlightning.LightningStore]. When a runner executes a rollout, it allocates a store-backed tracing context:
|
||||
|
||||
```python
|
||||
async with tracer.trace_context(
|
||||
name="my-rollout",
|
||||
store=store,
|
||||
rollout_id=rollout.rollout_id,
|
||||
attempt_id=attempt.attempt_id,
|
||||
):
|
||||
await run_agent_logic()
|
||||
```
|
||||
|
||||
The context manager then requests sequence numbers from the store, converts OpenTelemetry spans into [`Span`][agentlightning.Span] objects, and persists them in the middle or at the end of the attempt, depending on the tracer implementation. Agent-lightning ships two tracers out of the box; both rely on [OpenTelemetry Traces](https://opentelemetry.io/docs/concepts/signals/traces/) and ignore metrics or logs.
|
||||
|
||||
!!! tip "What's instrumentation?"
|
||||
|
||||
In simple terms, *instrumentation* means adding "patches" or hooks inside your code so you can observe what it’s doing while it runs. Think of it like putting flight recorders in an airplane — instrumentation records key actions, inputs, outputs, and timings without changing how the code behaves. In Agent-lightning tracers, this instrumentation automatically creates spans (small, structured records of work) that show what each part of an agent did, how long it took, and how different steps connect together.
|
||||
|
||||
### AgentOps Tracer
|
||||
|
||||
[`AgentOpsTracer`][agentlightning.AgentOpsTracer] will be the default tracer when [`Trainer`][agentlightning.Trainer] is used but no tracer is explicitly specified. It bootstraps the [AgentOps SDK](https://www.agentops.ai/) locally, installs the supplied instrumentation hooks (LangChain, LangGraph, LiteLLM, FastAPI, and others) provided by the [AgentOps Python SDK](https://github.com/AgentOps-AI/agentops), and forwards everything through a local OpenTelemetry [`TracerProvider`](https://opentelemetry.io/docs/specs/otel/trace/api/). [`AgentOpsTracer`][agentlightning.AgentOpsTracer] never calls the hosted AgentOps service; instead, it attaches a `LightningSpanProcessor` implemented by the Agent-lightning team so that spans are captured and shipped straight into the store.
|
||||
|
||||
Because it shares the AgentOps instrumentation surface, any framework supported by AgentOps automatically gains tracing in Agent-lightning. We layer additional hooks on top of AgentOps to capture features that the SDK misses today:
|
||||
|
||||
1. Certain providers emit extra metadata — for example, [token IDs returned by vLLM](../deep-dive/serving-llm.md) — that are not recorded by the stock SDK. We augment those spans with the missing payloads.
|
||||
2. AgentOps constructs parent-child relationships on a best-effort basis, but mixed instrumentation (for example, OpenAI Agent SDK alongside direct OpenAI Chat Completion calls) can leave segments disconnected. Our implementation (actually implemented in the [`TracerTraceToTriplet`][agentlightning.TracerTraceToTriplet] adapter) repairs those relationships when the hierarchy can be inferred from rollout context.
|
||||
3. Some versions of downstream frameworks simply do not emit spans for critical events (LangGraph node entrances are a common example). The tracer installs lightweight shims so those spans appear consistently.
|
||||
|
||||
If a vendor integration behaves unexpectedly, users are encouraged to combine the tracer with [Hooks](./debug.md) to inspect the raw spans or diagnostics, and/or implement a specialized tracer for the framework in question.
|
||||
|
||||
### OpenTelemetry Tracer
|
||||
|
||||
[`OtelTracer`][agentlightning.OtelTracer] is a minimal implementation that initializes a vanilla [`TracerProvider`](https://opentelemetry.io/docs/specs/otel/trace/api/) and gives you direct control over span creation through the standard `opentelemetry.trace` API. Use it when you already have explicit instrumentation in your agent, when the AgentOps SDK does not support your framework, or when you want to emit custom spans from business logic.
|
||||
|
||||
!!! note
|
||||
|
||||
[Microsoft Agent Framework](https://github.com/microsoft/agent-framework) is a typical example with built-in OpenTelemetry support. Once you set `OBSERVABILITY_SETTINGS.enable_otel = True`, the framework will automatically emit OpenTelemetry spans, and [`OtelTracer`][agentlightning.OtelTracer] will be able to capture them. No extra instrumentation is needed.
|
||||
|
||||
Inside your agent you can call `opentelemetry.trace.get_trace_provider().get_tracer("my-agent")` and use that tracer to [create spans](https://opentelemetry.io/docs/languages/python/cookbook/) exactly as you would in any OpenTelemetry application. The Lightning span processor attached by [`OtelTracer`][agentlightning.OtelTracer] guarantees that every span is sequenced, converted, and written to the store. The same applies for emitted rewards ([`emit_reward`][agentlightning.emit_reward]) and other emitter signals, which are just a special case of manually-created spans.
|
||||
|
||||
### Weave Tracer (Experimental)
|
||||
|
||||
[`WeaveTracer`][agentlightning.tracer.weave.WeaveTracer] is an experimental tracer that integrates with the [Weave Python SDK](https://docs.wandb.ai/weave). Use it as a substitute for [`AgentOpsTracer`][agentlightning.AgentOpsTracer] when the AgentOps SDK does not fit your environment.
|
||||
|
||||
The Weave SDK instruments LLM calls and agent libraries directly. Unlike [`AgentOpsTracer`][agentlightning.AgentOpsTracer], Weave does not rely on OpenTelemetry to export spans; it routes everything through a dedicated Weave Trace Server. Agent-lightning implements a custom Weave Trace Server so every call captured by the Weave SDK can be persisted to the [`LightningStore`][agentlightning.LightningStore].
|
||||
|
||||
!!! warning
|
||||
|
||||
[`WeaveTracer`][agentlightning.tracer.weave.WeaveTracer] remains experimental and has not been tested as thoroughly as [`AgentOpsTracer`][agentlightning.AgentOpsTracer]. It may conflict with libraries that ship OpenTelemetry instrumentation by default (for example, LiteLLM-based LLM proxies). Use the tracer with caution and report any issues to the Agent-lightning team.
|
||||
|
||||
### LLM Proxy
|
||||
|
||||
Sometimes the runner can’t observe the agent directly — because it’s in another language or running remotely. [`LLMProxy`][agentlightning.LLMProxy] bridges that gap by instrumenting the server side of LLM calls. It wraps [LiteLLM](https://docs.litellm.ai/) and adds middleware that accepts prefixed routes like `/rollout/{rid}/attempt/{aid}/v1/chat/completions`. Before forwarding, the middleware rewrites the path to `/v1/chat/completions`, fetches a monotonic `sequence_id` from the `LightningStore`, injects `x-rollout-id`, `x-attempt-id`, and `x-sequence-id` into the request headers, and then forwards the request to the backend LLM endpoint.
|
||||
|
||||
LiteLLM produces OpenTelemetry spans for the request/response. A custom `LightningSpanExporter` reads the rollout/attempt/sequence identifiers from the recorded request headers and persists each span to the store. Because the `sequence_id` is allocated at the start of the request, traces stay in strict order even across machines with skewed clocks or asynchronous responses.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Agent
|
||||
participant Proxy as LLM Proxy
|
||||
participant Backend as LLM Backend
|
||||
participant Store as LightningStore
|
||||
|
||||
Agent->>Proxy: POST /rollout/{rid}/attempt/{aid}/v1/chat/completions
|
||||
Proxy->>Store: get_next_span_sequence_id(rid, aid)
|
||||
Store-->>Proxy: sequence_id
|
||||
Proxy->>Backend: Forward /v1/chat/completions<br>(headers: rid, aid, sid)
|
||||
Backend-->>Proxy: Response (tokens, usage, token_ids)
|
||||
Proxy->>Store: Export OTEL spans (rid, aid, sequence_id)
|
||||
Proxy-->>Agent: OpenAI-compatible response
|
||||
```
|
||||
|
||||
[`LLMProxy`][agentlightning.LLMProxy] actually provides more functionalities than just the middleware for tracing. Read [Serving LLM](../deep-dive/serving-llm.md) for more details.
|
||||
|
||||
[](){ #distributed-tracing }
|
||||
|
||||
!!! note "Distributed Tracing"
|
||||
|
||||
Agent-lightning enforces deterministic span ordering by assigning a monotonic [`sequence_id`][agentlightning.Span.sequence_id] to every span within an attempt. Before calling [`LightningStore.add_span`][agentlightning.LightningStore.add_span] or [`LightningStore.add_otel_span`][agentlightning.LightningStore.add_otel_span], tracers are expected to call [`LightningStore.get_next_span_sequence_id`][agentlightning.LightningStore.get_next_span_sequence_id] to get the next sequence id. This removes clock skew and merges spans produced on different machines or threads. If you implement a custom tracer or exporter, make sure you do this (or respect the one provided in headers by components such as [`LLMProxy`][agentlightning.LLMProxy]); otherwise, adapters will struggle to properly reconstruct the execution tree.
|
||||
|
||||
### Custom Tracer
|
||||
|
||||
If none of the built-in tracers fit your environment, the first option to consider is to [return the spans](./write-agents.md) directly from your agent implementation. If that's not possible, or you want to support multiple agents in a unified effort, you can implement your own tracer by subclassing [`Tracer`][agentlightning.Tracer].
|
||||
|
||||
Custom tracers must implement at least [`trace_context`][agentlightning.Tracer.trace_context]. The [`trace_context`][agentlightning.Tracer.trace_context] coroutine should install or activate whatever instrumentation you need, then yield a span processor that ultimately adds spans to the store. You can reuse the `LightningSpanProcessor` if you produce OpenTelemetry `ReadableSpan` objects, or call [`LightningStore.add_span`][agentlightning.LightningStore.add_span] directly if you generate [`Span`][agentlightning.Span] instances yourself.
|
||||
|
||||
Advanced tracers often run auxiliary services (for example, starting a telemetry daemon or attaching to a container runtime) inside `init_worker` and tear them down in `teardown_worker`. The [`ParallelWorkerBase`][agentlightning.ParallelWorkerBase] lifecycle that `Tracer` inherits from ensures those hooks are executed in every runner subprocess.
|
||||
|
||||
## Reading Traces
|
||||
|
||||
Generally, there are two approaches to reading traces. When you only need a quick look, [`Tracer.get_last_trace`][agentlightning.Tracer.get_last_trace] returns the raw OpenTelemetry spans captured most recently. For historical analysis, use the [`LightningStore.query_spans`][agentlightning.LightningStore.query_spans] API, which yields normalized [`Span`][agentlightning.Span] objects keyed by rollout ID and attempt ID. Combine those queries with [`LightningStore.query_rollouts`][agentlightning.LightningStore.query_rollouts] to align spans with rollout status, retries, and timing information.
|
||||
|
||||
Spans arrive asynchronously, originate from different processes, and form hierarchies rather than simple lists. The attributes of each span are tedious and unfriendly to human readers. This combination makes raw traces time-consuming to inspect, especially when you only care about specific signals such as rewards, LLM prompts, responses, or tool outputs. Understanding how the store exposes traces and how adapters reshape them will save hours when debugging or training.
|
||||
|
||||
!!! note "Why traces can be difficult to read?"
|
||||
|
||||
The trace tree for a single rollout typically mixes multiple abstraction layers: a planner span may contain several LLM spans, each of which contains tool execution spans that can themselves trigger nested agent invocations. There are also instrumentations at different levels. For example, when a request delegates to another library (e.g., from LangChain to OpenAI), two libraries might emit spans for the same request. At the top level, there could be concurrently running agents that may flush spans slightly out of order. Sorting by `sequence_id` restores the chronological view, but interpreting the tree requires additional context about parent-child relationships and rollout metadata.
|
||||
|
||||
### Adapter
|
||||
|
||||
[Adapters][agentlightning.Adapter] transform lists of spans into higher-level data structures that training algorithms can consume directly. Agent-lightning provides several adapters out of the box:
|
||||
|
||||
* [`TracerTraceToTriplet`][agentlightning.TracerTraceToTriplet] converts spans into `(prompt, response, reward)` triplets, which power reinforcement-learning algorithms such as [VERL](../algorithm-zoo/verl.md) and connect trace data to gradient updates.
|
||||
* [`TraceToMessages`][agentlightning.TraceToMessages] rewrites spans into OpenAI chat message JSON suitable for supervised fine-tuning or evaluation harnesses.
|
||||
* [`LlmProxyTraceToTriplet`][agentlightning.LlmProxyTraceToTriplet] mirrors [`TracerTraceToTriplet`][agentlightning.TracerTraceToTriplet] but understands spans emitted by [LLMProxy][agentlightning.LLMProxy]. It is experimental and might be merged with [`TracerTraceToTriplet`][agentlightning.TracerTraceToTriplet] in the future.
|
||||
|
||||
Adapters are regular Python callable instances, so you can plug them into [`Trainer`][agentlightning.Trainer] via the `adapter` argument, or call them manually during exploration. When used in [`Trainer`][agentlightning.Trainer], adapters are bundled into the [`Algorithm`][agentlightning.Algorithm] before the algorithm runs, through the [`Algorithm.set_adapter`][agentlightning.Algorithm.set_adapter] method.
|
||||
|
||||
You can also customize an [`Adapter`][agentlightning.Adapter] by extending the implementations above or subclassing the base class. If you need a bespoke format, subclass [`TraceAdapter`][agentlightning.TraceAdapter] (for store spans) or [`OtelTraceAdapter`][agentlightning.OtelTraceAdapter] (for raw OpenTelemetry spans) and implement `adapt` (these two classes can usually share the same implementation).
|
||||
|
||||
### Reading Rewards
|
||||
|
||||
Rewards are recorded as dedicated spans named [`agentlightning.annotation`][agentlightning.semconv.AGL_ANNOTATION]. Emitting a reward through [`emit_reward`][agentlightning.emit_reward] or [`emit_annotation`][agentlightning.emit_annotation] ensures the value is stored in the span’s `attributes`. To audit rewards, fetch spans from the store and use the helper utilities in [`agentlightning.emitter`](../reference/agent.md):
|
||||
|
||||
```python
|
||||
from agentlightning.emitter import find_final_reward
|
||||
|
||||
spans = await store.query_spans(rollout_id)
|
||||
reward = find_final_reward(spans)
|
||||
print(f"Final reward: {reward}")
|
||||
```
|
||||
|
||||
[`find_reward_spans`][agentlightning.find_reward_spans] returns every reward span so you can visualize intermediate shaping signals, while [`find_final_reward`][agentlightning.find_final_reward] extracts the last non-null reward per attempt. While these helpers are convenient, they may not help you fully understand the chronological or hierarchical relationships between reward spans and other spans. Using an [`Adapter`][agentlightning.Adapter] — especially the same one used in the algorithm you’re working with — remains the recommended way to inspect your generated spans.
|
||||
@@ -0,0 +1,204 @@
|
||||
# Writing Agents
|
||||
|
||||
This tutorial will focus on the heart of the system: the agent itself, guiding you through the different ways to define an agent's logic in Agent-lightning.
|
||||
|
||||
The basic requirements for any agent are:
|
||||
|
||||
1. It must accept a single **task** as input.
|
||||
2. It must accept a set of tunable **resources** (like a [PromptTemplate][agentlightning.PromptTemplate] or [LLM][agentlightning.LLM]).
|
||||
3. It must **emit** trace span data so that algorithms can understand its behavior and learn from it. *The simplest way to do this is by returning a final reward.*
|
||||
|
||||
In practice, please also bear in mind that tasks, resources, and spans have extra requirements, in order to make it *trainable* within Agent-lightning:
|
||||
|
||||
1. You will need a training dataset containing a set of tasks, of the same type that your agent expects as input.
|
||||
2. The tunable resources are related to the algorithm. For example, the APO algorithm we've seen tunes a [PromptTemplate][agentlightning.PromptTemplate]. Other algorithms might tune model weights or other configurations.
|
||||
3. The type of spans an algorithm can use varies. Almost all algorithms support a single, final reward span at the end of a rollout. However, not all algorithms support rewards emitted mid-rollout, let alone other kinds of spans like exceptions or log messages.
|
||||
|
||||
This tutorial will show you how to write an agent that can handle various tasks and resources and emit all kinds of spans. However, you should understand that agents and algorithms are often co-designed. Supporting new types of resources or spans in an algorithm is often much more complex than just adding them to an agent.
|
||||
|
||||
## [`@rollout`][agentlightning.rollout] Decorator
|
||||
|
||||
The simplest way to create an agent is by writing a standard Python function and marking it with the [@rollout][agentlightning.rollout] decorator. This approach is perfect for agents with straightforward logic that doesn't require complex state management.
|
||||
|
||||
Agent-lightning automatically inspects your function's signature and injects the required resources. For example, if your function has a parameter named `prompt_template`, Agent-lightning will find the [PromptTemplate][agentlightning.PromptTemplate] resource for the current rollout and pass it in.
|
||||
|
||||
Let's revisit the `room_selector` agent from the first tutorial:
|
||||
|
||||
```python
|
||||
from typing import TypedDict
|
||||
from agentlightning import PromptTemplate, rollout
|
||||
|
||||
# Define a data structure for the task input
|
||||
class RoomSelectionTask(TypedDict):
|
||||
# ... fields for the task ...
|
||||
pass
|
||||
|
||||
@rollout
|
||||
def room_selector(task: RoomSelectionTask, prompt_template: PromptTemplate) -> float:
|
||||
# 1. Use the injected prompt_template to format the input for the LLM
|
||||
prompt = prompt_template.format(**task)
|
||||
|
||||
# 2. Execute the agent's logic (e.g., call an LLM, use tools)
|
||||
# ...
|
||||
|
||||
# 3. Grade the final choice to get a reward
|
||||
reward = room_selection_grader(final_message, task["expected_choice"])
|
||||
|
||||
# 4. Return the final reward as a float
|
||||
return reward
|
||||
```
|
||||
|
||||
When you train this agent, the dataset is expected to be a list of `RoomSelectionTask` objects:
|
||||
|
||||
```python
|
||||
from agentlightning import Dataset, Trainer
|
||||
|
||||
dataset: Dataset[RoomSelectionTask] = [
|
||||
RoomSelectionTask(date="2025-10-15", time="10:00", duration_min=60, attendees=10),
|
||||
RoomSelectionTask(date="2025-10-16", time="10:00", duration_min=60, attendees=10),
|
||||
]
|
||||
|
||||
Trainer().fit(agent=room_selector, train_dataset=dataset)
|
||||
```
|
||||
|
||||
Behind the scenes, the [`@rollout`][agentlightning.rollout] decorator wraps your function in a `FunctionalLitAgent` object, which is a subclass of [LitAgent][agentlightning.LitAgent] introduced below, making it compatible with the [Trainer][agentlightning.Trainer] and [Runner][agentlightning.Runner]. It supports parameters like `task`, `prompt_template`, `llm`, and `rollout`, giving you flexible access to the execution context.
|
||||
|
||||
Here is another example with more advanced usage with `llm` and `rollout` as parameters. The `llm` parameter gives you an OpenAI-compatible LLM endpoint to interact with, which can be tuned under the hood by algorithms. The `rollout` parameter gives you the full [Rollout][agentlightning.Rollout] object, which contains the rollout ID, rollout mode (training or validation), etc.
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
from agentlightning import LLM, Rollout
|
||||
|
||||
class FlightBookingTask(TypedDict):
|
||||
request: str
|
||||
expected_booking: dict
|
||||
|
||||
@rollout
|
||||
def flight_assistant(task: FlightBookingTask, llm: LLM, rollout: Rollout) -> float:
|
||||
print(f"Rollout ID: {rollout.rollout_id}")
|
||||
print(f"Rollout Mode: {rollout.mode}")
|
||||
|
||||
# Use the tuned LLM resource to create an OpenAI client
|
||||
client = OpenAI(
|
||||
# This endpoint could be a proxy to a proxy to a proxy ...
|
||||
# It could be different every time `flight_assistant` is called
|
||||
# But it should be OpenAI-API compatible
|
||||
base_url=llm.endpoint,
|
||||
|
||||
# Use a dummy key if not provided
|
||||
# Usually this does not matter because the training LLM is often not guarded by an API key
|
||||
# But you can use `or os.environ["OPENAI_API_KEY"]` to make the function compatible with 3rd-party LLMs
|
||||
api_key=llm.api_key or "dummy-key",
|
||||
)
|
||||
|
||||
# Make an API call with the specified model
|
||||
response = client.chat.completions.create(
|
||||
model=llm.model,
|
||||
messages=[{"role": "user", "content": task["request"]}],
|
||||
)
|
||||
# Whether the API supports features like streaming, tool calls, etc. depends on
|
||||
# the endpoint that algorithms are serving to you.
|
||||
final_message = response.choices[0].message.content
|
||||
|
||||
# Grade the result and return a reward
|
||||
reward = grade_flight_booking(final_message, task["expected_booking"])
|
||||
return reward
|
||||
```
|
||||
|
||||
## Return Values from Agents
|
||||
|
||||
The value your agent function returns (i.e., the return value of the function decorated by [`@rollout`][agentlightning.rollout]) is crucial, as it's the primary way to report the outcome of a rollout. Agent-lightning supports several return types to accommodate different scenarios, from simple rewards to detailed, custom traces.
|
||||
|
||||
* **`float`**: This is the simplest and most common return type. The `float` is treated as the **final reward** for the entire rollout. Agent-lightning automatically creates a final reward span based on this value.
|
||||
|
||||
* **`None`**: Returning `None` tells the runner that trace collection is being handled entirely by the [Tracer][agentlightning.Tracer] through auto-instrumentation (e.g., via AgentOps). In this case, the runner will simply retrieve the spans that the tracer has already captured.
|
||||
|
||||
!!! important "Emitting the Final Reward"
|
||||
|
||||
When returning `None`, you must still ensure a final reward is logged. You can do this by using the [`emit_reward`][agentlightning.emit_reward] function (covered in the [Use Emitters](./emitter.md) documentation). Wrapping your reward calculation function with the `@reward` decorator is NOT the recommended approach any more.
|
||||
|
||||
* **`list[ReadableSpan]`**, **`list[SpanCoreFields]`**, or **`list[Span]`**: For advanced use cases, you can manually construct and return a complete list of all spans for the rollout. This gives you full control over the trace data. You can return either a list of OpenTelemetry `ReadableSpan` objects or Agent-lightning's native `Span` objects.
|
||||
|
||||
For most users, returning a **`float`** for simple agents or returning **`None`** and using the emitter for more complex ones are the recommended approaches.
|
||||
|
||||
## Class-based Agents
|
||||
|
||||
For more complex agents that require state, helper methods, or distinct logic for training versus validation, you can create a class that inherits from [`LitAgent`][agentlightning.LitAgent]. This object-oriented approach provides more structure and control over the agent's lifecycle.
|
||||
|
||||
To create a class-based agent, you subclass [agentlightning.LitAgent][] and implement its [`rollout`][agentlightning.LitAgent.rollout] method.
|
||||
|
||||
[](){ #introduction-to-named-resources }
|
||||
|
||||
Here's how the `room_selector` could be implemented as a class. The rollout method has a slightly different signature than the function-based agent, mainly in how it handles the resources. Putting it simply, algorithms do not just send a [PromptTemplate][agentlightning.PromptTemplate] to the agents, they instead send [NamedResources][agentlightning.NamedResources], which is a mapping from resource key to [Resource][agentlightning.Resource]. This design is to allow for more advanced features like multi-resource tuning.
|
||||
|
||||
With [`@rollout`][agentlightning.rollout] decorator, the resource with correctly matched type will be automatically injected into the rollout method. However, when you use a class-based agent, you need to manually access the resource from the `resources` dictionary. Built-in algorithms listed their resource key naming conventions [here](../algorithm-zoo/index.md).
|
||||
|
||||
```python
|
||||
import agentlightning as agl
|
||||
|
||||
class RoomSelectorAgent(agl.LitAgent[RoomSelectionTask]):
|
||||
def rollout(self, task: RoomSelectionTask, resources: agl.NamedResources, rollout: agl.Rollout) -> float:
|
||||
# 1. Access the prompt_template from the resources dictionary
|
||||
prompt_template = resources["prompt_template"]
|
||||
|
||||
# 2. Execute the agent's logic
|
||||
prompt = prompt_template.format(**task)
|
||||
# ...
|
||||
|
||||
# 3. Grade the final choice
|
||||
reward = room_selection_grader(final_message, task["expected_choice"])
|
||||
|
||||
# 4. Return the final reward
|
||||
return reward
|
||||
|
||||
# To use it with the trainer:
|
||||
# agent = RoomSelectorAgent()
|
||||
# trainer.fit(agent=agent, ...)
|
||||
```
|
||||
|
||||
The [`LitAgent`][agentlightning.LitAgent] class provides several methods you can override for more fine-grained control:
|
||||
|
||||
* [`rollout()`][agentlightning.LitAgent.rollout]: The primary method for the agent's logic. It's called for both training and validation by default.
|
||||
* [`training_rollout()`][agentlightning.LitAgent.training_rollout] / [`validation_rollout()`][agentlightning.LitAgent.validation_rollout]: Implement these if you need different behavior during training (e.g., with exploration) and validation (e.g., with deterministic choices).
|
||||
* [`rollout_async()`][agentlightning.LitAgent.rollout_async] / [`training_rollout_async()`][agentlightning.LitAgent.training_rollout_async] / [`validation_rollout_async()`][agentlightning.LitAgent.validation_rollout_async]: Implement the asynchronous versions of these methods if your agent uses `asyncio`.
|
||||
|
||||
!!! note
|
||||
|
||||
Rollout is always executed in an asynchronous context no matter whether the agent is asynchronous or synchronous. If your synchronous agent contains some `asyncio.run()` calls, it might raise an error that there is already an event loop running. To avoid blocking the event loop, it's recommended to offload the inner async operations to a separate thread. Here is a sample code:
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import queue
|
||||
import threading
|
||||
|
||||
def run_sync_ephemeral(coro) -> Any:
|
||||
"""
|
||||
Run an async coroutine from sync code.
|
||||
- If no loop in this thread: use asyncio.run() directly.
|
||||
- If already in an event loop: spawn a worker thread that calls asyncio.run()
|
||||
(which creates and closes a brand-new event loop per call).
|
||||
"""
|
||||
try:
|
||||
asyncio.get_running_loop()
|
||||
except RuntimeError:
|
||||
# No running loop in this thread; safe to use asyncio.run
|
||||
return asyncio.run(coro)
|
||||
|
||||
# Already in a running loop -> execute in a worker thread
|
||||
q = queue.Queue[Any]()
|
||||
|
||||
def worker():
|
||||
try:
|
||||
result = asyncio.run(coro) # creates & closes its own loop
|
||||
q.put((True, result))
|
||||
except BaseException as e:
|
||||
q.put((False, e))
|
||||
|
||||
t = threading.Thread(target=worker, daemon=True)
|
||||
t.start()
|
||||
ok, payload = q.get()
|
||||
t.join()
|
||||
if ok:
|
||||
return payload
|
||||
raise payload
|
||||
```
|
||||
Reference in New Issue
Block a user