chore: import upstream snapshot with attribution
Deploy Documentation / deploy (push) Has been cancelled
CPU Test / Test (Utilities, legacy, Python 3.10) (push) Has been cancelled
CPU Test / Test (LLM proxy, stable, Python 3.11) (push) Has been cancelled
CPU Test / Test (Others, stable, Python 3.11) (push) Has been cancelled
CPU Test / Test (Store, stable, Python 3.11) (push) Has been cancelled
CPU Test / Test (Utilities, stable, Python 3.11) (push) Has been cancelled
CPU Test / Test (Weave, stable, Python 3.11) (push) Has been cancelled
CPU Test / Test (AgentOps, stable, Python 3.12) (push) Has been cancelled
CPU Test / Test (LLM proxy, stable, Python 3.12) (push) Has been cancelled
CPU Test / Test (Others, stable, Python 3.12) (push) Has been cancelled
CPU Test / Test (Weave, latest, Python 3.13) (push) Has been cancelled
Dashboard / Chromatic (push) Has been cancelled
CPU Test / Lint - fast (push) Has been cancelled
CPU Test / Lint - next (push) Has been cancelled
CPU Test / Lint - slow (push) Has been cancelled
CPU Test / Lint - JavaScript (push) Has been cancelled
CPU Test / Build documentation (push) Has been cancelled
CPU Test / Test (AgentOps, legacy, Python 3.10) (push) Has been cancelled
CPU Test / Test (LLM proxy, legacy, Python 3.10) (push) Has been cancelled
CPU Test / Test (Others, legacy, Python 3.10) (push) Has been cancelled
CPU Test / Test (Store, legacy, Python 3.10) (push) Has been cancelled
CPU Test / Test (Weave, legacy, Python 3.10) (push) Has been cancelled
CPU Test / Test (AgentOps, stable, Python 3.11) (push) Has been cancelled
CPU Test / Test (Store, stable, Python 3.12) (push) Has been cancelled
CPU Test / Test (Utilities, stable, Python 3.12) (push) Has been cancelled
CPU Test / Test (Weave, stable, Python 3.12) (push) Has been cancelled
CPU Test / Test (AgentOps, latest, Python 3.13) (push) Has been cancelled
CPU Test / Test (LLM proxy, latest, Python 3.13) (push) Has been cancelled
CPU Test / Test (Others, latest, Python 3.13) (push) Has been cancelled
CPU Test / Test (Store, latest, Python 3.13) (push) Has been cancelled
CPU Test / Test (Utilities, latest, Python 3.13) (push) Has been cancelled
CPU Test / Test (JavaScript) (push) Has been cancelled
Deploy Documentation / deploy (push) Has been cancelled
CPU Test / Test (Utilities, legacy, Python 3.10) (push) Has been cancelled
CPU Test / Test (LLM proxy, stable, Python 3.11) (push) Has been cancelled
CPU Test / Test (Others, stable, Python 3.11) (push) Has been cancelled
CPU Test / Test (Store, stable, Python 3.11) (push) Has been cancelled
CPU Test / Test (Utilities, stable, Python 3.11) (push) Has been cancelled
CPU Test / Test (Weave, stable, Python 3.11) (push) Has been cancelled
CPU Test / Test (AgentOps, stable, Python 3.12) (push) Has been cancelled
CPU Test / Test (LLM proxy, stable, Python 3.12) (push) Has been cancelled
CPU Test / Test (Others, stable, Python 3.12) (push) Has been cancelled
CPU Test / Test (Weave, latest, Python 3.13) (push) Has been cancelled
Dashboard / Chromatic (push) Has been cancelled
CPU Test / Lint - fast (push) Has been cancelled
CPU Test / Lint - next (push) Has been cancelled
CPU Test / Lint - slow (push) Has been cancelled
CPU Test / Lint - JavaScript (push) Has been cancelled
CPU Test / Build documentation (push) Has been cancelled
CPU Test / Test (AgentOps, legacy, Python 3.10) (push) Has been cancelled
CPU Test / Test (LLM proxy, legacy, Python 3.10) (push) Has been cancelled
CPU Test / Test (Others, legacy, Python 3.10) (push) Has been cancelled
CPU Test / Test (Store, legacy, Python 3.10) (push) Has been cancelled
CPU Test / Test (Weave, legacy, Python 3.10) (push) Has been cancelled
CPU Test / Test (AgentOps, stable, Python 3.11) (push) Has been cancelled
CPU Test / Test (Store, stable, Python 3.12) (push) Has been cancelled
CPU Test / Test (Utilities, stable, Python 3.12) (push) Has been cancelled
CPU Test / Test (Weave, stable, Python 3.12) (push) Has been cancelled
CPU Test / Test (AgentOps, latest, Python 3.13) (push) Has been cancelled
CPU Test / Test (LLM proxy, latest, Python 3.13) (push) Has been cancelled
CPU Test / Test (Others, latest, Python 3.13) (push) Has been cancelled
CPU Test / Test (Store, latest, Python 3.13) (push) Has been cancelled
CPU Test / Test (Utilities, latest, Python 3.13) (push) Has been cancelled
CPU Test / Test (JavaScript) (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,55 @@
|
||||
# APO
|
||||
|
||||
!!! tip "Shortcut"
|
||||
|
||||
You can use the shortcut `agl.APO(...)` to create an APO instance.
|
||||
|
||||
```python
|
||||
import agentlightning as agl
|
||||
|
||||
agl.APO(...)
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pip install agentlightning[apo]
|
||||
```
|
||||
|
||||
## Scope of Current Implementation
|
||||
|
||||
APO is currently scoped to optimize a single prompt template. Optimizing multiple prompt templates is not supported yet.
|
||||
|
||||
There is however no restriction on the number of variable placeholders in the prompt template (can range from zero to many). It's possible that invalid prompts are created during the optimization process. It is up to the agent developer to ensure that the prompt template is valid for the agent's task.
|
||||
|
||||
## Initial Prompt
|
||||
|
||||
APO expects the initial prompt to be provided in the `initial_resources` dictionary. This can be done in two approaches:
|
||||
|
||||
1. Pass to the [Trainer][agentlightning.Trainer] constructor:
|
||||
|
||||
```python
|
||||
trainer = agl.Trainer(
|
||||
algorithm=agl.APO(...),
|
||||
initial_resources={"main_prompt": agl.PromptTemplate(template="You are a helpful assistant.", engine="f-string")},
|
||||
)
|
||||
```
|
||||
|
||||
2. Pass to the `[APO][agentlightning.algorithm.apo.APO].set_initial_resources()` method:
|
||||
|
||||
```python
|
||||
algo = agl.APO(...)
|
||||
algo.set_initial_resources(
|
||||
{"this_is_also_valid_key": agl.PromptTemplate(template="You are a helpful assistant.", engine="f-string")}
|
||||
)
|
||||
```
|
||||
|
||||
The resource key can be arbitrary, which is used to identify the prompt template in [class-based implementations](../tutorials/write-agents.md) when you have multiple resources. When the key changes, the agent developer needs to update the key in the `rollout` method.
|
||||
|
||||
## Tutorials Using APO
|
||||
|
||||
- [Train the First Agent with APO](../how-to/train-first-agent.md) - A step-by-step guide to training your first agent using APO.
|
||||
|
||||
## References
|
||||
|
||||
::: agentlightning.algorithm.apo
|
||||
@@ -0,0 +1,10 @@
|
||||
# Algorithm Zoo
|
||||
|
||||
AgentLightning includes several popular and frequently requested algorithms in its built-in library, allowing agent developers to use them directly. These algorithms are designed to be compatible with most agent scenarios.
|
||||
|
||||
For customizing algorithms, see [Algorithm-side References](../reference/algorithm.md).
|
||||
|
||||
| Algorithm | Optimizing Resources | Description |
|
||||
| --------- | ------------------- | ----------- |
|
||||
| [APO](./apo.md) | `{<initial_prompt_key>: [PromptTemplate][agentlightning.PromptTemplate]}` | Automatic Prompt Optimization (APO) algorithm using textual gradients and beam search. |
|
||||
| [VERL](./verl.md) | `{"main_llm": [LLM][agentlightning.LLM]}` | Reinforcement Learning with [VERL framework](https://github.com/volcengine/verl). |
|
||||
@@ -0,0 +1,58 @@
|
||||
# VERL
|
||||
|
||||
!!! tip "Shortcut"
|
||||
|
||||
You can use the shortcut `agl.VERL(...)` to create a VERL instance.
|
||||
|
||||
```python
|
||||
import agentlightning as agl
|
||||
|
||||
agl.VERL(...)
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pip install agentlightning[verl]
|
||||
```
|
||||
|
||||
!!! warning
|
||||
|
||||
To avoid various compatibility issues, follow the steps in the [installation guide](../tutorials/installation.md) to set up VERL and its dependencies. Installing VERL directly with `pip install agentlightning[verl]` can cause issues unless you already have a compatible version of PyTorch installed.
|
||||
|
||||
!!! note "Notes for Readers"
|
||||
|
||||
[VERL][agentlightning.algorithm.verl.VERL] in this article refers to a wrapper, provided by Agent-lightning, of the [VERL framework](https://github.com/volcengine/verl). It's a subclass of [agentlightning.Algorithm][]. To differentiate it from the VERL framework, all references to the VERL framework shall use the term "VERL framework", and all references to the Agent-lightning wrapper shall be highlighted with a link.
|
||||
|
||||
## Resources
|
||||
|
||||
[VERL][agentlightning.algorithm.verl.VERL] expects no initial resources. The first LLM endpoint is directly deployed from the VERL configuration (`.actor_rollout_ref.model.path`). The resource key is always `main_llm`.
|
||||
|
||||
[VERL][agentlightning.algorithm.verl.VERL] currently does not support optimizing multiple [LLM][agentlightning.LLM]s together.
|
||||
|
||||
!!! note
|
||||
|
||||
The resource type created by [VERL][agentlightning.algorithm.verl.VERL] is actually a [ProxyLLM][agentlightning.ProxyLLM], a subclass of the [LLM][agentlightning.LLM] type. This object contains a **URL template** provided by [VERL][agentlightning.algorithm.verl.VERL], with placeholders for rollout and attempt IDs. When a rollout begins on the agent side, the framework uses the current `rollout_id` and `attempt_id` to format this template, generating a final, unique endpoint URL. This URL points to [VERL][agentlightning.algorithm.verl.VERL]'s internal proxy, allowing it to intercept and log all traffic for that specific attempt, for tracing and load balancing purposes. For agents created with the `@rollout` decorator, this resolution of the template is handled automatically ("auto-stripped"). Class-based agents will need to manually resolve the [`ProxyLLM`][agentlightning.ProxyLLM] using the rollout context.
|
||||
|
||||
```python
|
||||
proxy_llm = resources["main_llm"]
|
||||
proxy_llm.get_base_url(rollout.rollout_id, rollout.attempt.attempt_id)
|
||||
```
|
||||
|
||||
## Customization
|
||||
|
||||
Internally, [VERL][agentlightning.algorithm.verl.VERL] decomposes each agent execution into prompt–response pairs via the [Adapter][agentlightning.Adapter] and associates them with their corresponding reward signals as [Triplet][agentlightning.Triplet] objects. The final scalar reward, derived from the last triplet in the trajectory, is propagated to all preceding triplets following the [identical assignment strategy](https://arxiv.org/abs/2508.03680). This ensures that each triplet receives an identical reward signal and can be independently optimized as a valid RLHF trajectory within the VERL framework.
|
||||
|
||||
At present, [VERL][agentlightning.algorithm.verl.VERL] does not expose fine-grained control over its reward propagation or credit assignment mechanisms. Users requiring customized reward shaping or trajectory decomposition are advised to clone and modify the [VERL][agentlightning.algorithm.verl.VERL] source implementation directly.
|
||||
|
||||
## Tutorials Using VERL
|
||||
|
||||
- [Train SQL Agent with RL](../how-to/train-sql-agent.md) - A practical example of training a SQL agent using VERL.
|
||||
|
||||
## References - Entrypoint
|
||||
|
||||
::: agentlightning.algorithm.verl
|
||||
|
||||
## References - Implementation
|
||||
|
||||
::: agentlightning.verl
|
||||
Reference in New Issue
Block a user