wehub/conductor-oss--conductor
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
CI / unit-test (push) Has been cancelled
Details
CI / detect-changes (push) Has been cancelled
Details
CI / build (push) Has been cancelled
Details
Publish docs via GitHub Pages / Deploy docs (push) Has been cancelled
Details
CI / test-harness (push) Has been cancelled
Details
CI / generate-e2e-matrix (push) Has been cancelled
Details
CI / e2e (push) Has been cancelled
Details
CI / build-ui (push) Has been cancelled
Details
Release Drafter / update_release_draft (push) Has been cancelled
Details
UI v2 Integration CI / E2E (Integration) (push) Has been cancelled
Details
UI v2 CI / Lint, Format & Test (push) Has been cancelled
Details
UI v2 CI / E2E (Mocked) (push) Has been cancelled
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:37:56 +08:00
commit a9cd7750f4
3727 changed files with 627706 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/devguide/ai/a2a-integration.md
+620
View File
@@ -0,0 +1,620 @@
---
description: "A2A (Agent2Agent) integration with Conductor — call remote agents as durable workflow tasks, and expose Conductor workflows as A2A agents. Crash-safe, resumable, observable."
---
# A2A integration
[A2A (Agent2Agent)](https://a2a-protocol.org/) is an open protocol for agents to talk to one another over HTTP/JSON-RPC. Conductor integrates A2A in **both directions**:
- **Client** — call a remote A2A agent from a workflow as a durable system task (`AGENT`, `GET_AGENT_CARD`, `CANCEL_AGENT`).
- **Server** — expose any Conductor workflow as an A2A agent that other A2A clients (Google ADK, CrewAI, LangGraph, another Conductor) can discover and invoke.
The integration is **durable**: a remote agent call survives a server crash, restart, or redeploy, and resumes from where it left off — the call's state lives in the workflow execution, not in a thread.
## What is A2A
A2A standardizes three things: an **Agent Card** (a `/.well-known/agent-card.json` document describing an agent's skills), a **JSON-RPC** surface (`message/send`, `tasks/get`, `tasks/cancel`, …), and a **task lifecycle** (`submitted → working → input-required → completed/failed/canceled`). An agent runs a long task; the client polls (or is pushed) until it reaches a terminal state.
Conductor maps this lifecycle onto its own durable task model, so a remote agent task behaves like any other Conductor task — retried, timed out, observed, and resumed by the engine.
Conductor speaks A2A in **both directions**: a workflow can *call* remote agents (client), and a workflow can *be* an agent that external A2A clients call (server).
```mermaid
flowchart LR
ExtClient["External A2A client<br/>(Google ADK · CrewAI · LangGraph · another Conductor)"]
Remote["Remote A2A agent"]
subgraph C["Conductor"]
WF["Workflow execution<br/>(durable · resumable · observable)"]
end
ExtClient -->|"server: message/send starts the workflow"| WF
WF -->|"client: AGENT task sends message/send"| Remote
```
## Call a remote agent from a workflow (client)
*Direction A — Conductor is the A2A client.* These tasks require the AI integration to be enabled:
```properties
conductor.integrations.ai.enabled=true
```
Each task takes an **`agentType`** input that selects the agent runtime. It defaults to `"a2a"` (Agent2Agent — the only runtime in OSS today); native runtimes such as LangGraph and OpenAI are planned, and the field is the extension point for them. An unrecognized `agentType` fails the task with a clear error.
### AGENT — send a message to an agent
Sends an A2A `message/send` and works the resulting agent task to a terminal state. Non-blocking: a fast reply completes immediately; long-running work moves to `IN_PROGRESS` and is polled at a cadence (no worker thread is held).
```mermaid
sequenceDiagram
autonumber
participant WF as Conductor workflow
participant T as AGENT task
participant R as Remote A2A agent
WF->>T: schedule { agentUrl, message }
T->>R: message/send (idempotencyKey = deterministic messageId)
R-->>T: Task { state: working }
alt poll (default) / push backstop
loop until terminal or input-required
T->>R: tasks/get
R-->>T: Task { working → completed }
end
else streaming
R-->>T: SSE status-update / artifact-update …
end
T-->>WF: artifacts + state as task output
```
```json
{
"name": "call_currency_agent",
"taskReferenceName": "agent",
"type": "AGENT",
"inputParameters": {
"agentType": "a2a",
"agentUrl": "https://currency-agent.example.com",
"text": "convert 100 USD to EUR",
"pollIntervalSeconds": 5,
"headers": {
"Authorization": "Bearer ${workflow.input.agentToken}"
}
}
}
```
**Key inputs** (see `A2ACallRequest`):
| Field | Description |
|---|---|
| `agentType` | Agent runtime to use — defaults to `"a2a"`. Reserved for native runtimes (e.g. `langgraph`, `openai`) coming later; any other value is rejected today. |
| `agentUrl` | Base URL of the remote agent (required). |
| `text` / `prompt` | Convenience for a single text part. |
| `parts` / `message` | A full A2A message (multi-part / data parts) instead of `text`. |
| `contextId`, `taskId` | Continue an existing conversation / resume an agent task (multi-turn). |
| `headers` | Per-call HTTP headers (e.g. auth). Reference credentials via workflow inputs/secrets rather than hardcoding them. |
| `pollIntervalSeconds` | Poll cadence in poll mode (default 5). |
| `streaming` | `true` → consume `message/stream` (SSE) and aggregate to completion. |
| `pushNotification` | `true` → the agent calls back our webhook on completion (see below). |
| `maxDurationSeconds` | Absolute deadline (default 86400). |
| `maxPollFailures` | Consecutive transient poll failures tolerated before failing (default 30). |
**Output** (`agent.output`): `state` (the A2A task state), `taskId` and `contextId` (for resumption), `artifacts`, `text` (extracted text), `agentMessage`, and the full `task` object. For a completed call it looks like:
```json
{
"state": "completed",
"taskId": "task-7f3a",
"contextId": "ctx-7f3a",
"text": "100 USD = 92.40 EUR",
"artifacts": [
{ "artifactId": "result", "parts": [ { "kind": "text", "text": "100 USD = 92.40 EUR" } ] }
]
}
```
Downstream tasks read these with `${agent.output.text}`, `${agent.output.taskId}`, etc.
#### Three execution modes
- **Poll** (default) — the task is `IN_PROGRESS` and polled via `tasks/get` at `pollIntervalSeconds`. No thread is held between polls; the call survives restarts.
- **Streaming** (`streaming: true`) — consumes the agent's SSE stream and aggregates events. Requires `capabilities.streaming=true` on the agent card. Holds a thread for the stream's duration.
- **Push** (`pushNotification: true`) — the agent calls Conductor's webhook when the task finishes, so nothing polls in the meantime. Requires `conductor.a2a.callback.url`. A slow **backstop poll** still runs (`pushBackstopPollSeconds`, default 300) so a lost webhook can't hang the task.
#### Push notifications — end to end
**1. Configure the externally-reachable callback base URL** (where the agent can reach Conductor):
```properties
conductor.integrations.ai.enabled=true
conductor.a2a.callback.url=https://conductor.example.com
```
**2. Ask for push on the task:**
```json
{
"name": "call_research_agent",
"taskReferenceName": "agent",
"type": "AGENT",
"inputParameters": {
"agentUrl": "https://research-agent.example.com",
"text": "research durable agent protocols",
"pushNotification": true,
"pushBackstopPollSeconds": 300
}
}
```
**3. What Conductor sends** — the `message/send` carries a `pushNotificationConfig` pointing at a
per-task webhook, with a single-use bearer token (a `{uuid}:{expiryEpochMillis}` value, 24h TTL):
```json
{
"method": "message/send",
"params": {
"message": { "role": "user", "messageId": "a2a-...", "parts": [ { "kind": "text", "text": "research durable agent protocols" } ] },
"configuration": {
"pushNotificationConfig": {
"url": "https://conductor.example.com/api/a2a/callback/<conductorTaskId>",
"token": "3f9c…:1750300000000",
"authentication": { "schemes": ["Bearer"], "credentials": "3f9c…:1750300000000" }
}
}
}
}
```
The `AGENT` task then **waits** (holds no worker thread) until the webhook arrives; the backstop
poll runs only as a safety net.
**4. The agent calls back** when the task reaches a terminal/interrupted state — Conductor verifies
the token (constant-time + expiry), fetches the final task via `tasks/get`, and completes the
workflow task:
```bash
curl -X POST https://conductor.example.com/api/a2a/callback/<conductorTaskId> \
-H 'Authorization: Bearer 3f9c…:1750300000000' \
-H 'Content-Type: application/json' \
-d '{ "taskId": "<remoteAgentTaskId>", "status": { "state": "completed" } }'
# → 200 OK; the AGENT task is now COMPLETED with the agent's output.
```
Agents that don't support the `authentication` field fall back to a `?token=` query parameter on the
callback URL, which the endpoint still accepts (with a deprecation warning, since tokens in URLs land
in access logs).
### GET_AGENT_CARD — discover an agent
```json
{
"name": "discover_agent",
"taskReferenceName": "discover",
"type": "GET_AGENT_CARD",
"inputParameters": { "agentUrl": "https://currency-agent.example.com" }
}
```
Resolves the agent card from `/.well-known/agent-card.json` (falling back to the legacy `/.well-known/agent.json`) and returns the parsed skills/capabilities — feed it to an LLM so it can pick a skill at runtime.
### CANCEL_AGENT — cancel a running agent task
```json
{
"name": "cancel_agent_task",
"taskReferenceName": "cancel",
"type": "CANCEL_AGENT",
"inputParameters": {
"agentUrl": "https://currency-agent.example.com",
"taskId": "${agent.output.taskId}"
}
}
```
### Multi-turn (input-required)
When a remote task reaches `input-required` (or `auth-required`), `AGENT` **completes** and surfaces the agent's question plus the `taskId`/`contextId` in its output. The workflow branches on that state and issues another `AGENT` task with the **same `taskId` and `contextId`** carrying the answer — resuming the same remote task rather than starting a new conversation:
```json
{
"name": "branch_on_state",
"taskReferenceName": "branch",
"type": "SWITCH",
"evaluatorType": "value-param",
"expression": "state",
"inputParameters": { "state": "${ask.output.state}" },
"decisionCases": {
"input-required": [
{
"name": "answer_agent", "taskReferenceName": "answer", "type": "AGENT",
"inputParameters": {
"agentUrl": "${workflow.input.agentUrl}",
"text": "${workflow.input.answer}",
"contextId": "${ask.output.contextId}",
"taskId": "${ask.output.taskId}"
}
}
]
},
"defaultCase": []
}
```
Full example: `ai/examples/29-a2a-client-multi-turn.json`.
### Orchestrating multiple agents
Because each `AGENT` is an ordinary durable task, you compose agents with the usual Conductor operators — e.g. **`FORK_JOIN`** to call several agents in parallel, **`JOIN`** to gather results. Every branch is independently crash-safe: if Conductor restarts mid-flight, each in-flight agent call resumes from persisted state (`ai/examples/27-a2a-multi-agent.json`). To let an LLM pick which skill to use, chain `GET_AGENT_CARD → LLM_CHAT_COMPLETE → AGENT` (`ai/examples/28-a2a-llm-pick-skill.json`).
```mermaid
flowchart LR
Start([Workflow]) --> Fork{{FORK_JOIN}}
Fork --> A1[AGENT → agent A]
Fork --> A2[AGENT → agent B]
Fork --> A3[AGENT → agent C]
A1 --> Join{{JOIN}}
A2 --> Join
A3 --> Join
Join --> Next([aggregate results])
```
### Error handling & retries
`AGENT` maps remote outcomes onto Conductor task statuses, so the engine's normal retry/timeout machinery applies. Retryable failures become `FAILED` (the engine retries per the task def's `retryCount`); permanent failures become `FAILED_WITH_TERMINAL_ERROR` (no retry):
| Condition | Task status | Retried? |
|---|---|---|
| HTTP 408/429/5xx, connect/read timeout, dropped/empty stream | `FAILED` | yes |
| JSON-RPC transient error (e.g. `-32603` internal) | `FAILED` | yes |
| Remote agent task ends `failed` / `rejected` | `FAILED` | yes |
| HTTP 4xx (except 408/429) | `FAILED_WITH_TERMINAL_ERROR` | no |
| JSON-RPC terminal codes (`-32700/-32600/-32601/-32602/-3200{1..5,7}`) | `FAILED_WITH_TERMINAL_ERROR` | no |
| Missing `agentUrl` / empty message / **SSRF-blocked** URL | `FAILED_WITH_TERMINAL_ERROR` | no |
| Exceeds `maxDurationSeconds`, or `maxPollFailures` consecutive poll failures | `FAILED_WITH_TERMINAL_ERROR` | no |
Retries reuse the deterministic `messageId`, so agents that dedupe on it get effectively-once delivery. The failure reason is on `task.reasonForIncompletion`.
**Troubleshooting**
| Symptom | Cause / fix |
|---|---|
| `… SSRF blocked` | `agentUrl` resolves to a private/loopback/metadata address. Use a public URL, or set `conductor.a2a.client.allow-private-network=true` for trusted/dev (cloud-metadata stays blocked). |
| `streaming: true` behaves like poll | The agent card has `capabilities.streaming=false`; the client only streams when the agent advertises it. |
| Fails after N poll failures | The agent is unreachable — raise `maxPollFailures` or check connectivity. |
| Hangs, then fails at the deadline | The agent never reached a terminal state within `maxDurationSeconds`. |
## Expose a workflow as an A2A agent (server)
*Direction B — Conductor is the A2A server.* Any Conductor workflow can be published as an A2A agent
that other A2A clients (Google ADK, CrewAI, LangGraph, another Conductor) discover and invoke. The
workflow execution **is** the durable, resumable A2A task — that's the native fit.
```mermaid
sequenceDiagram
autonumber
participant Client as External A2A client
participant S as A2AServerResource
participant A as A2AWorkflowAgent
participant E as Conductor engine
Client->>S: GET …/.well-known/agent-card.json
S-->>Client: Agent Card (one skill = the workflow)
Client->>S: POST message/send
S->>A: sendMessage
A->>E: startWorkflow (idempotencyKey = A2A messageId)
E-->>A: workflowId
A-->>Client: Task { id = workflowId, state: working }
loop tasks/get until terminal
Client->>S: tasks/get
S->>E: getExecutionStatus
E-->>S: RUNNING → COMPLETED
S-->>Client: Task { state, artifacts }
end
note over Client,E: blocked on HUMAN/WAIT → input-required;<br/>a follow-up message/send resumes the same execution
```
Enable the server and opt the workflow in:
```properties
conductor.a2a.server.enabled=true
# Expose by name…
conductor.a2a.server.exposed-workflows=order_pizza,book_appointment
```
…or per-workflow via `WorkflowDef.metadata`:
```json
{
"name": "order_pizza",
"version": 1,
"metadata": { "a2a.enabled": true, "a2a.tags": ["ordering"] },
"tasks": [ ... ]
}
```
**Routing: one agent per workflow.** Each exposed workflow is its own focused agent at `{basePath}/{workflow}` (default basePath `/a2a`):
| Method & path | Purpose |
|---|---|
| `GET /a2a/{workflow}/.well-known/agent-card.json` | Agent Card (also `/agent.json`). |
| `POST /a2a/{workflow}` | JSON-RPC: `message/send`, `message/stream` (SSE), `tasks/get`, `tasks/cancel`. |
| `GET /a2a` | Convenience listing of exposed agents (non-spec). |
Exposed agents advertise `capabilities.streaming=true`.
### Discover and call
```bash
# 1. Discover
curl http://localhost:8080/a2a/order_pizza/.well-known/agent-card.json
# 2. Start a task (message/send → starts the workflow)
curl -X POST http://localhost:8080/a2a/order_pizza \
-H 'Content-Type: application/json' \
-d '{
"jsonrpc": "2.0", "id": 1, "method": "message/send",
"params": { "message": {
"role": "user", "messageId": "m-1",
"parts": [ { "kind": "text", "text": "one large pepperoni" } ]
} }
}'
# → result is an A2A Task: { "id": "<workflowId>", "contextId": ..., "status": { "state": "working" } }
# 3. Poll
curl -X POST http://localhost:8080/a2a/order_pizza \
-H 'Content-Type: application/json' \
-d '{ "jsonrpc": "2.0", "id": 2, "method": "tasks/get", "params": { "id": "<workflowId>" } }'
```
The inbound A2A message is injected into the workflow input as `_a2a_text`, `_a2a_message_id`, `_a2a_context_id` (plus any data parts), and `contextId` becomes the workflow `correlationId`.
### Streaming (message/stream)
Use `message/stream` instead of `message/send` for a Server-Sent Events stream: the initial `Task`,
then `status-update` events as the workflow's A2A state changes and `artifact-update` events as
output is produced, ending with a `final` status-update at a terminal / input-required state.
```bash
curl -N -X POST http://localhost:8080/a2a/order_pizza \
-H 'Content-Type: application/json' \
-d '{ "jsonrpc":"2.0", "id":1, "method":"message/stream",
"params": { "message": { "role":"user", "messageId":"m-1",
"parts":[ {"kind":"text","text":"one large pepperoni"} ] } } }'
```
```text
data: {"jsonrpc":"2.0","id":1,"result":{"kind":"task","id":"wf-7f3a","status":{"state":"working"}}}
data: {"jsonrpc":"2.0","id":1,"result":{"kind":"artifact-update","taskId":"wf-7f3a","artifact":{"artifactId":"workflow-output","parts":[{"kind":"data","data":{"orderId":"ORD-42"}}]}}}
data: {"jsonrpc":"2.0","id":1,"result":{"kind":"status-update","taskId":"wf-7f3a","status":{"state":"completed"},"final":true}}
```
The stream is a live view of the durable execution — if the connection drops, resume tracking with
`tasks/get`. Tuning: `conductor.a2a.server.stream-poll-interval-millis` (default 500) and
`conductor.a2a.server.stream-max-duration-seconds` (default 300).
### Durable, idempotent start
`message/send` starts the workflow with `idempotencyKey = {workflow}:{messageId}` and `RETURN_EXISTING`, so a client's **retried** `message/send` returns the **existing** execution rather than starting a duplicate — server-side effectively-once. The execution's durability (crash-safe, resumable) is inherited from the engine.
### Status mapping
| Conductor workflow | A2A task state |
|---|---|
| RUNNING, blocked on a `HUMAN`/`WAIT` task | `input-required` |
| RUNNING (not blocked) / PAUSED | `working` |
| COMPLETED | `completed` (output → an artifact) |
| FAILED / TIMED_OUT | `failed` |
| TERMINATED | `canceled` |
### Multi-turn resume — worked example
If the workflow blocks on a `HUMAN`/`WAIT` task, the agent reports `input-required`. A follow-up
`message/send` carrying that task's `id` (the workflow id) **resumes** the paused execution — the
message content completes the pending task and the workflow continues. No duplicate workflow is
started; if the workflow is already terminal or not awaiting input, its current state is returned
unchanged.
Take this exposed workflow (`ai/examples/25-a2a-server-multi-turn.json`) — it asks a question, then
confirms:
```json
{
"name": "book_appointment",
"version": 1,
"metadata": { "a2a.enabled": true },
"tasks": [
{ "name": "ask_preferred_time", "taskReferenceName": "ask", "type": "HUMAN" },
{
"name": "confirm_appointment", "taskReferenceName": "confirm", "type": "INLINE",
"inputParameters": {
"evaluatorType": "graaljs",
"expression": "({ status: 'confirmed', when: $.when })",
"when": "${ask.output._a2a_text}"
}
}
]
}
```
**Turn 1 — start.** The workflow reaches the `HUMAN` task and parks at `input-required`:
```bash
curl -X POST http://localhost:8080/a2a/book_appointment \
-H 'Content-Type: application/json' \
-d '{ "jsonrpc":"2.0", "id":1, "method":"message/send",
"params": { "message": { "role":"user", "messageId":"m-1",
"parts":[ {"kind":"text","text":"Book me a dentist appointment"} ] } } }'
```
```json
{
"jsonrpc": "2.0", "id": 1,
"result": {
"kind": "task",
"id": "wf-7f3a91",
"contextId": "wf-7f3a91",
"status": {
"state": "input-required",
"message": { "role": "agent", "parts": [ { "kind": "text",
"text": "Workflow is awaiting input. Send another message/send carrying this task's id to provide the input and resume the execution." } ] }
}
}
}
```
**Turn 2 — resume.** Send the answer with the **same `taskId`** (`= result.id`); the `HUMAN` task
completes with the message as its input and the workflow finishes:
```bash
curl -X POST http://localhost:8080/a2a/book_appointment \
-H 'Content-Type: application/json' \
-d '{ "jsonrpc":"2.0", "id":2, "method":"message/send",
"params": { "message": { "role":"user", "messageId":"m-2", "taskId":"wf-7f3a91",
"parts":[ {"kind":"text","text":"Tuesday at 3pm"} ] } } }'
```
```json
{
"jsonrpc": "2.0", "id": 2,
"result": {
"kind": "task",
"id": "wf-7f3a91",
"contextId": "wf-7f3a91",
"status": { "state": "completed" },
"artifacts": [
{ "artifactId": "workflow-output", "name": "output",
"parts": [ { "kind": "data", "data": { "status": "confirmed", "when": "Tuesday at 3pm" } } ] }
]
}
}
```
The answer (`Tuesday at 3pm`) arrives at the workflow as `${ask.output._a2a_text}`, exactly as if the
`HUMAN` task had been completed through the Conductor API.
## Durability
The "durable A2A" claim rests on a few concrete mechanisms:
- **Deterministic message id.** `AGENT` derives the A2A `messageId` from `workflowInstanceId + referenceTaskName + iteration` — stable across task retries and server restarts, distinct per `DO_WHILE` iteration. Agents that dedupe on `messageId` get effectively-once delivery despite at-least-once retries.
- **State in the execution, not the thread.** Poll mode holds no thread; the remote `taskId`, deadline, and poll-failure count live in the persisted task output, so a restart resumes the poll loop.
- **Liveness guards.** An absolute deadline (`maxDurationSeconds`) and a consecutive-poll-failure bound (`maxPollFailures`) ensure a dead or stuck agent can't hang a task forever.
- **Push backstop.** Push mode still backstop-polls, so a lost webhook degrades to polling rather than hanging.
## Security
- **SSRF guard.** Outbound `agentUrl`s that resolve to loopback, private (RFC-1918), link-local, IPv6 unique-local (`fc00::/7`), or cloud-metadata addresses are rejected. Cloud-metadata addresses are blocked **always**. To allow private-network agents (e.g. localhost in dev):
```properties
conductor.a2a.client.allow-private-network=true
```
Cloud metadata stays blocked even with this on. For production, prefer a network-layer egress firewall.
- **Server auth.** Like OSS Conductor REST, the A2A server is **open by default**. Front it with a gateway/firewall (or mTLS) to control access. Inbound authentication (API keys, OAuth/OIDC, mTLS, per-skill scopes, signed Agent Cards) is provided by the **enterprise** build.
- **Push tokens.** Push callbacks carry a single-use bearer token with an embedded 24h expiry, validated constant-time by the callback endpoint (`POST /api/a2a/callback/{taskId}`).
## Observability
A2A code paths emit metrics through the shared Conductor metrics registry and set MDC keys for log correlation.
**Metrics:** `a2a_client_calls{result}`, `a2a_client_poll_failures`, `a2a_rpc_errors{method,terminal}`, `a2a_ssrf_blocked`, `a2a_server_requests{method}`, `a2a_server_resumes`.
**MDC keys** (greppable in logs): `a2aWorkflowId`, `a2aTaskId`, `a2aRef`, `a2aRemoteTaskId`, `a2aContextId`, `a2aMessageId`, `a2aAgent`, `a2aMethod`.
## Configuration reference
| Property | Default | Purpose |
|---|---|---|
| `conductor.integrations.ai.enabled` | `false` | Enables the client tasks (`AGENT`, …). |
| `conductor.a2a.callback.url` | — | Externally-reachable base URL for push callbacks. |
| `conductor.a2a.client.allow-private-network` | `false` | Allow agent URLs on private/loopback networks (metadata still blocked). |
| `conductor.a2a.server.enabled` | `false` | Enables the A2A server endpoints. |
| `conductor.a2a.server.basePath` | `/a2a` | Base path for exposed agents. |
| `conductor.a2a.server.exposed-workflows` | — | Comma-separated workflow names to expose. |
| `conductor.a2a.server.public-url` | request-derived | Base URL advertised in the agent card. |
| `conductor.a2a.server.provider-organization` | `Conductor` | `provider.organization` on the card. |
## Examples
### A complete workflow: discover then call
This workflow discovers a remote agent's card, then calls it — passing the agent URL and prompt as
workflow inputs so the same definition works against any A2A agent:
```json
{
"name": "a2a_interop_echo",
"version": 1,
"schemaVersion": 2,
"description": "Discover a remote A2A agent, then call it.",
"ownerEmail": "a2a@example.com",
"tasks": [
{
"name": "discover_agent",
"taskReferenceName": "discover",
"type": "GET_AGENT_CARD",
"inputParameters": { "agentUrl": "${workflow.input.agentUrl}" }
},
{
"name": "call_agent",
"taskReferenceName": "call",
"type": "AGENT",
"inputParameters": {
"agentUrl": "${workflow.input.agentUrl}",
"text": "${workflow.input.prompt}",
"pollIntervalSeconds": 2
}
}
]
}
```
Register and run it (the AI integration must be enabled — `conductor.integrations.ai.enabled=true`;
for a localhost agent in dev also set `conductor.a2a.client.allow-private-network=true`):
```bash
# register
curl -X POST localhost:8080/api/metadata/workflow \
-H 'Content-Type: application/json' -d @a2a_interop_echo.json
# run against a reachable A2A agent
curl -X POST localhost:8080/api/workflow/a2a_interop_echo \
-H 'Content-Type: application/json' \
-d '{"agentUrl":"http://localhost:9999","prompt":"convert 100 USD to EUR"}'
```
### Run it end to end (showcase demos)
Two self-contained demos under `ai/src/test/resources/a2a/` boot a real agent + Conductor and run a
workflow against it — no API keys:
```bash
# Interop: Conductor calls the official a2a-sdk reference agent (a real, non-Conductor A2A server)
ai/src/test/resources/a2a/interop-demo/run-interop-demo.sh
# Durability: kill the Conductor server mid-call; the workflow resumes and completes after restart
ai/src/test/resources/a2a/durable-demo/run-durable-demo.sh
```
### Example library
Runnable workflow definitions live in [`ai/examples/`](https://github.com/conductor-oss/conductor/tree/main/ai/examples):
| File | Shows |
|---|---|
| `10-a2a-call-agent.json` | Call a remote agent (poll mode) |
| `11-a2a-get-agent-card.json` | Discover an agent's skills |
| `12-a2a-server-workflow.json` | Expose a workflow as an A2A agent |
| `23-a2a-streaming.json` | Streaming (SSE) call |
| `24-a2a-push.json` | Push-notification mode |
| `25-a2a-server-multi-turn.json` | Multi-turn server agent (HUMAN task → resume) |
| `26-a2a-cancel.json` | Start then cancel a remote agent task |
| `27-a2a-multi-agent.json` | Call multiple agents in parallel (FORK_JOIN → JOIN) |
| `28-a2a-llm-pick-skill.json` | Discover → LLM picks the prompt → call |
| `29-a2a-client-multi-turn.json` | Client multi-turn (branch on input-required, re-call) |
docs/devguide/ai/durable-agents.md
+178
View File
@@ -0,0 +1,178 @@
---
description: What makes a durable AI agent — persisted state, crash recovery, and why JSON workflow definitions are AI-native for agent orchestration.
---
# Durable agents
An agent that runs in a single process is fragile. A crashed pod replays every LLM call from the beginning — burning tokens and money. A human approval that took three days is lost because a deploy bounced the server. A multi-hour research pipeline fails at step 47 and starts over from step 1.
Conductor eliminates all of this. Every step of a durable agent workflow is persisted to storage as it completes. If the process dies, the agent resumes from the last completed step — not from the beginning.
## What gets persisted
- The **workflow definition snapshot** (immutable for this execution).
- Each **LLM call**: input prompt, model response, token usage, latency.
- Each **tool call**: input, output, status, retry count.
- Each **wait state**: when it started, what it's waiting for, the resume payload when it arrives.
- Each **human decision**: who approved, when, with what data.
- The **loop state**: iteration count, intermediate results, exit condition evaluation.
No LLM calls are repeated unless a task explicitly failed and needs retry. A human approval that completed on Tuesday is still there on Wednesday, even if the cluster was replaced overnight. This is what makes Conductor-based agents production-ready.
## JSON is AI-native
LLMs natively produce JSON. Conductor natively executes JSON. This means an agent can generate its own execution plan as a workflow definition and Conductor will execute it immediately — no compilation, no deployment, no code generation step.
```
LLM generates plan → JSON workflow definition → Conductor executes it
```
This is not a workaround. It is the intended design, and it makes Conductor uniquely suited to agent orchestration:
**Runtime generation.** An LLM or planner emits a workflow definition as JSON, your code passes it to the [StartWorkflowRequest API](../../documentation/api/startworkflow.md), and Conductor validates, persists, and executes it immediately — without pre-registration. The workflow itself becomes a first-class output of the agent's planning step.
**Inspectability.** Every agent run is a JSON document you can query, diff, and audit. You can see exactly what the LLM decided, what tools were called, what the human approved, and in what order. No opaque framework state — just data.
**Versioning.** Workflow definitions are versioned. Run multiple agent versions concurrently, A/B test different tool configurations, and roll back without affecting running executions.
**SDK/UI/API parity.** The same workflow can be defined via JSON file, SDK code, API call, or the Conductor UI. All paths produce the same stored JSON definition. An agent that generates workflows programmatically and a human who designs them in the UI are using the same runtime.
## Error handling and compensation
Agents don't just read data — they take actions. They send emails, create tickets, charge cards, update databases. When a step fails after earlier steps have already produced side effects, you need compensation: the ability to undo or mitigate what was already done.
Conductor provides this through the `failureWorkflow` field and the saga compensation pattern:
```json
{
"name": "booking_agent",
"failureWorkflow": "booking_agent_compensation",
"tasks": [
{ "name": "reserve_flight", "type": "HTTP", "taskReferenceName": "flight" },
{ "name": "reserve_hotel", "type": "HTTP", "taskReferenceName": "hotel" },
{ "name": "charge_payment", "type": "HTTP", "taskReferenceName": "payment" }
]
}
```
If `charge_payment` fails, the `booking_agent_compensation` workflow runs automatically. It receives the full execution state — including the outputs of `reserve_flight` and `reserve_hotel` — so it can cancel the flight, release the hotel reservation, and notify the user.
This is not error handling you bolt on later. It is built into the execution model:
- **`failureWorkflow`** runs a separate workflow on failure, with full access to the failed execution's state.
- **Retry policies** on individual tasks (fixed, exponential backoff, linear) with configurable limits.
- **Timeout policies** that fail or alert when an LLM call or tool takes too long.
- **`TERMINATE` task** to end execution early with a specific status and output when the agent detects an unrecoverable condition.
Most AI frameworks have no concept of compensation. If your LangChain agent sends an email in step 3 and crashes in step 5, the email is already sent and there is no built-in mechanism to undo it. Conductor's failure workflows solve this.
## Multi-agent composition
Real-world AI systems rarely run as a single agent. A research agent delegates to specialist sub-agents. A customer service agent escalates to a billing agent. A planning agent spawns parallel analysis agents and synthesizes their results.
Conductor models this with `SUB_WORKFLOW` tasks inside a `FORK`/`JOIN` for parallel execution:
```json
{
"name": "research_coordinator",
"tasks": [
{
"name": "plan",
"type": "LLM_CHAT_COMPLETE",
"taskReferenceName": "plan",
"inputParameters": {
"llmProvider": "anthropic",
"model": "claude-sonnet-4-20250514",
"messages": [
{ "role": "user", "message": "Break this research task into sub-tasks: ${workflow.input.topic}" }
]
}
},
{
"name": "fork_sub_agents",
"type": "FORK_JOIN",
"taskReferenceName": "fork",
"forkTasks": [
[
{
"name": "run_web_researcher",
"type": "SUB_WORKFLOW",
"taskReferenceName": "web_research",
"subWorkflowParam": { "name": "web_research_agent", "version": 1 },
"inputParameters": { "query": "${plan.output.result.webQuery}" }
}
],
[
{
"name": "run_data_analyst",
"type": "SUB_WORKFLOW",
"taskReferenceName": "data_analysis",
"subWorkflowParam": { "name": "data_analysis_agent", "version": 1 },
"inputParameters": { "dataset": "${plan.output.result.dataset}" }
}
]
]
},
{
"name": "join_sub_agents",
"type": "JOIN",
"taskReferenceName": "join",
"joinOn": ["web_research", "data_analysis"]
},
{
"name": "synthesize",
"type": "LLM_CHAT_COMPLETE",
"taskReferenceName": "synthesize",
"inputParameters": {
"llmProvider": "anthropic",
"model": "claude-sonnet-4-20250514",
"messages": [
{ "role": "user", "message": "Synthesize these findings:\n\nWeb research: ${web_research.output}\n\nData analysis: ${data_analysis.output}" }
]
}
}
],
"failureWorkflow": "research_coordinator_cleanup"
}
```
Both sub-agents run concurrently. The `JOIN` waits for both to complete before the synthesize step runs. If you don't know the number of sub-agents ahead of time, use `DYNAMIC_FORK` instead — the LLM's plan output determines how many sub-agents to spawn.
**What you get from multi-agent composition in Conductor:**
- **Parallel execution.** Sub-agents run concurrently via `FORK`/`JOIN` or `DYNAMIC_FORK`. The join collects all results before the next step proceeds.
- **Full observability across the agent tree.** The parent workflow shows the status of each sub-agent. You can drill into any sub-workflow to see its individual LLM calls, tool calls, and decisions.
- **Failure isolation.** A failing sub-agent does not crash the parent. The parent can catch the failure, retry with different parameters, or route to a fallback agent.
- **Failure propagation with compensation.** If a sub-agent fails and the parent should also fail, `failureWorkflow` runs compensation across the entire agent tree.
- **Independent scaling.** Each sub-agent type can have its own workers scaled independently. A CPU-heavy data analysis agent doesn't compete for resources with a lightweight web research agent.
## Observability
Every agent execution in Conductor is fully observable — not through external logging you have to set up, but as a built-in property of the execution model. Because every step is persisted, the observability is automatic and complete.
**What you can see for every agent run:**
- **Task-by-task execution timeline.** Each task shows its status (scheduled, in progress, completed, failed), start time, end time, and duration. You see exactly where an agent is in its workflow at any moment.
- **Every LLM prompt and response.** The full input messages, model response, token usage (prompt tokens, completion tokens), and latency for each `LLM_CHAT_COMPLETE` or `LLM_TEXT_COMPLETE` call. You can inspect exactly what the agent decided and why.
- **Every tool call with input/output.** For `CALL_MCP_TOOL`, `HTTP`, and custom worker tasks: the exact arguments sent, the response received, and how many retry attempts were needed.
- **Human approval audit trail.** For `HUMAN` tasks: when the task was created, who completed it, when they completed it, and what data they provided. This is an immutable audit record.
- **Loop iteration history.** For `DO_WHILE` agent loops: the iteration count, the result of each iteration, and the exit condition evaluation. You can trace the agent's reasoning across its entire plan/act/observe cycle.
- **Sub-agent drill-down.** For `SUB_WORKFLOW` tasks: click through to the child workflow's full execution view. The parent shows the sub-agent's overall status; the child shows every step within it.
- **Retry and failure history.** Every retry attempt is recorded with its input, output, and failure reason. If a task failed three times before succeeding, all four attempts are visible.
This observability applies to every workflow — including workflows [generated dynamically by an LLM](dynamic-workflows.md). A workflow that was created 30 seconds ago by an agent's planning step gets the same execution visibility as one that was registered months ago.
For programmatic access, the [Workflow API](../../documentation/api/workflow.md) and [Task API](../../documentation/api/task.md) provide the same data via REST: query execution status, retrieve task inputs/outputs, and search across executions.
## Next steps
- **[Human-in-the-Loop](human-in-the-loop.md)** &mdash; Pre-execution review, conditional approval, and LLM-as-judge patterns.
- **[Dynamic Workflows](dynamic-workflows.md)** &mdash; Agent loops, dynamic workflow generation, and tool use examples.
- **[LLM Orchestration](llm-orchestration.md)** &mdash; Native LLM providers, vector databases, and content generation.
- **[Durable Execution Semantics](../../architecture/durable-execution.md)** &mdash; Failure matrix, state transitions, and exactly what persists.
docs/devguide/ai/dynamic-workflows.md
+255
View File
@@ -0,0 +1,255 @@
---
description: Dynamic workflow execution for AI agents — agents that build their own plans as JSON workflow definitions, agent loops with DO_WHILE, and tool use with MCP. Full durability, observability, and retry support.
---
# Dynamic workflows for agents
Conductor supports three levels of agent dynamism, from simple tool use to fully self-generating agents.
## Agent loop: plan/act/observe with DO_WHILE
The defining pattern of an autonomous agent is the loop: call an LLM, execute a tool, observe the result, decide whether to continue. Conductor models this with `DO_WHILE`:
```json
{
"name": "autonomous_agent",
"description": "Agent that loops until the task is complete",
"version": 1,
"schemaVersion": 2,
"tasks": [
{
"name": "agent_loop",
"taskReferenceName": "loop",
"type": "DO_WHILE",
"loopCondition": "if ($.loop['think'].output.result.done == true) { false; } else { true; }",
"loopOver": [
{
"name": "think",
"taskReferenceName": "think",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "anthropic",
"model": "claude-sonnet-4-20250514",
"messages": [
{
"role": "system",
"message": "You are an agent. Available tools: ${workflow.input.tools}. Previous results: ${loop.output.results}. Respond with JSON: {\"action\": \"tool_name\", \"arguments\": {}, \"done\": false} or {\"answer\": \"...\", \"done\": true}"
},
{
"role": "user",
"message": "${workflow.input.task}"
}
],
"temperature": 0.1
}
},
{
"name": "act",
"taskReferenceName": "act",
"type": "SWITCH",
"evaluatorType": "javascript",
"expression": "$.think.output.result.done ? 'done' : 'call_tool'",
"decisionCases": {
"call_tool": [
{
"name": "execute_tool",
"taskReferenceName": "tool_call",
"type": "CALL_MCP_TOOL",
"inputParameters": {
"mcpServer": "${workflow.input.mcpServerUrl}",
"method": "${think.output.result.action}",
"arguments": "${think.output.result.arguments}"
}
}
]
},
"defaultCase": []
}
]
}
],
"outputParameters": {
"answer": "${loop.output.think.output.result.answer}",
"iterations": "${loop.output.iteration}"
}
}
```
**What makes this durable:**
- Each iteration of the loop is a persisted checkpoint. If the agent crashes at iteration 12, it resumes from iteration 12 — not from iteration 1.
- Every LLM call (prompt, response, token usage) is recorded. You can inspect exactly what the agent decided at each step.
- Every tool call (input, output, status) is tracked. If a tool call fails, it retries according to the task's retry policy without re-running the LLM.
- The loop counter and all intermediate state survive server restarts.
## Dynamic workflow generation: agents that build their own plans
Conductor supports dynamic workflow execution where the complete workflow definition is provided at start time, without pre-registration. This is the most powerful form of agent dynamism — the LLM generates the entire execution plan as JSON, and Conductor runs it immediately.
1. An LLM generates a plan as a JSON workflow definition.
2. Your code passes that definition directly to the `StartWorkflowRequest`.
3. Conductor validates, persists, and executes it immediately.
4. Every step is durable, observable, and retryable — even though the workflow was generated at runtime.
```json
{
"name": "dynamic_agent_planner",
"version": 1,
"schemaVersion": 2,
"tasks": [
{
"name": "generate_plan",
"taskReferenceName": "planner",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "anthropic",
"model": "claude-sonnet-4-20250514",
"messages": [
{
"role": "system",
"message": "You are a workflow planner. Given a user task, generate a Conductor workflow definition as JSON. Available task types: LLM_CHAT_COMPLETE, CALL_MCP_TOOL, LIST_MCP_TOOLS, HTTP, HUMAN, LLM_SEARCH_INDEX. The workflow must include a 'name', 'tasks' array, and 'outputParameters'."
},
{
"role": "user",
"message": "${workflow.input.task}"
}
],
"temperature": 0.2
}
},
{
"name": "review_plan",
"taskReferenceName": "approval",
"type": "HUMAN",
"inputParameters": {
"generatedWorkflow": "${planner.output.result}"
}
},
{
"name": "execute_plan",
"taskReferenceName": "execution",
"type": "START_WORKFLOW",
"inputParameters": {
"startWorkflow": {
"workflowDefinition": "${planner.output.result}",
"input": "${workflow.input.taskInput}"
}
}
}
],
"outputParameters": {
"generatedPlan": "${planner.output.result}",
"executionId": "${execution.output.workflowId}"
}
}
```
**What happens:**
1. `planner` &mdash; `LLM_CHAT_COMPLETE` generates an entire workflow definition as JSON based on the user's task description.
2. `approval` &mdash; `HUMAN` task pauses the workflow so a reviewer can inspect the generated plan before it runs. This is critical — you don't want an LLM-generated workflow executing unsupervised.
3. `execution` &mdash; `START_WORKFLOW` launches the generated workflow definition directly. Conductor validates it, persists it, and executes it with full durability. No pre-registration needed.
The generated child workflow gets all the same guarantees as any Conductor workflow: persisted state, retry policies, failure handling, full observability. The fact that it was generated by an LLM 30 seconds ago doesn't matter — it runs on the same durable execution engine.
Combined with `DYNAMIC` tasks (where the task type is resolved at runtime based on input) and `DYNAMIC_FORK` (where the number and type of parallel tasks is determined at runtime), this enables agents that create, modify, and execute their own plans.
## Example: MCP agent with tool use and human approval
A more focused example — an agent that discovers tools, plans, gets approval, and executes. Every step uses a built-in system task.
```json
{
"name": "mcp_agent_with_approval",
"description": "Discover tools, plan, execute with approval, summarize",
"version": 1,
"schemaVersion": 2,
"tasks": [
{
"name": "list_available_tools",
"taskReferenceName": "discover_tools",
"type": "LIST_MCP_TOOLS",
"inputParameters": {
"mcpServer": "${workflow.input.mcpServerUrl}"
}
},
{
"name": "decide_which_tools_to_use",
"taskReferenceName": "plan",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "anthropic",
"model": "claude-sonnet-4-20250514",
"messages": [
{
"role": "system",
"message": "You are an AI agent. Available tools: ${discover_tools.output.tools}. User wants to: ${workflow.input.task}"
},
{
"role": "user",
"message": "Which tool should I use and what parameters? Respond with JSON: {\"method\": \"string\", \"arguments\": {}}"
}
],
"temperature": 0.1,
"maxTokens": 500
}
},
{
"name": "human_review",
"taskReferenceName": "approval",
"type": "HUMAN",
"inputParameters": {
"plannedAction": "${plan.output.result}"
}
},
{
"name": "execute_tool",
"taskReferenceName": "execute",
"type": "CALL_MCP_TOOL",
"inputParameters": {
"mcpServer": "${workflow.input.mcpServerUrl}",
"method": "${plan.output.result.method}",
"arguments": "${plan.output.result.arguments}"
}
},
{
"name": "summarize_result",
"taskReferenceName": "summarize",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "anthropic",
"model": "claude-sonnet-4-20250514",
"messages": [
{
"role": "user",
"message": "The user asked: ${workflow.input.task}\n\nTool result: ${execute.output.content}\n\nSummarize this result for the user."
}
],
"maxTokens": 500
}
}
],
"outputParameters": {
"plan": "${plan.output.result}",
"toolResult": "${execute.output.content}",
"summary": "${summarize.output.result}",
"approvedBy": "${approval.output.reviewer}"
}
}
```
Every task type here — `LIST_MCP_TOOLS`, `LLM_CHAT_COMPLETE`, `CALL_MCP_TOOL`, `HUMAN` — is a native Conductor system task. No custom workers, no external frameworks.
See the full set of examples in the [`ai/examples/`](https://github.com/conductor-oss/conductor/tree/main/ai/examples) directory.
## Next steps
- **[Durable Agents](durable-agents.md)** &mdash; What persists, what gets retried, and why JSON is AI-native.
- **[LLM Orchestration](llm-orchestration.md)** &mdash; Native LLM providers, vector databases, and content generation.
- **[Dynamic Fork](../../documentation/configuration/workflowdef/operators/dynamic-fork-task.md)** &mdash; Runtime-determined parallel execution.
- **[DO_WHILE](../../documentation/configuration/workflowdef/operators/do-while-task.md)** &mdash; Loop operator for agent iterations.
- **[HUMAN task](../../documentation/configuration/workflowdef/systemtasks/human-task.md)** &mdash; Human-in-the-loop approval.
docs/devguide/ai/failure-semantics.md
+281
View File
@@ -0,0 +1,281 @@
---
description: "The exact failure contract for AI agents on Conductor — what happens when LLM calls fail, tools timeout, humans don't respond, callbacks arrive twice, branches partially complete, versions change mid-flight, and workers deploy during active executions."
---
# Failure semantics for AI agents
This page defines exactly what happens when things go wrong in an agent workflow. Not "Conductor is durable" — but the precise behavior under every failure scenario an agent can encounter.
## LLM task failure
**Scenario:** The `LLM_CHAT_COMPLETE` task calls an LLM provider and the call fails (rate limit, timeout, provider outage, malformed response).
**What happens:**
1. The task moves to `FAILED`.
2. Conductor checks the task's retry configuration (`retryCount`, `retryLogic`, `retryDelaySeconds`).
3. A new task execution is created with an incremented retry count.
4. The task is requeued after the configured delay.
5. If all retries are exhausted, the task moves to `FAILED` terminal state.
6. The workflow's failure handling kicks in: `failureWorkflow` runs if configured, or the workflow moves to `FAILED`.
**What is preserved:** The prompt, the error response, the retry count, and the timing of each attempt. You can inspect every failed attempt in the UI.
**What is NOT re-executed:** Nothing upstream. Only the failed LLM call retries. All previously completed tasks retain their outputs.
**Configuration:**
```json
{
"name": "plan_action",
"retryCount": 3,
"retryLogic": "EXPONENTIAL_BACKOFF",
"retryDelaySeconds": 5,
"responseTimeoutSeconds": 60
}
```
This retries the LLM call up to 3 times with exponential backoff (5s, 10s, 20s). If the LLM doesn't respond within 60 seconds, the task times out and retries.
## LLM returns malformed output
**Scenario:** The LLM responds, but the output is not valid JSON or doesn't match the expected schema (e.g., missing `action` field).
**What happens:**
The `LLM_CHAT_COMPLETE` task completes successfully — the LLM did respond. The malformed output propagates to the next task. What happens next depends on the downstream task:
- If the next task references `${plan.output.result.action}` and `action` doesn't exist, the task fails with an input resolution error.
- The task retries according to its retry policy.
- The LLM is **not** re-called (it already completed).
**How to handle it:** Add a `SWITCH` or `INLINE` task after the LLM call to validate the output before acting on it:
```json
{
"name": "validate_plan",
"taskReferenceName": "validate",
"type": "INLINE",
"inputParameters": {
"plan": "${plan.output.result}",
"evaluatorType": "graaljs",
"expression": "(function() { var p = $.plan; if (!p || !p.action) { return {valid: false, error: 'Missing action field'}; } return {valid: true, plan: p}; })()"
}
}
```
If validation fails, use a `SWITCH` to re-run the LLM with a corrective prompt, or fail the workflow.
## Tool call timeout
**Scenario:** A `CALL_MCP_TOOL` or `HTTP` task calls an external tool and the tool doesn't respond within the configured timeout.
**What happens:**
1. `responseTimeoutSeconds` fires. The task moves to `TIMED_OUT`.
2. If retries are configured, the task is retried. A new request is sent to the tool.
3. The original (timed-out) request may still be in flight. The tool may eventually process it.
**Critical implication:** The tool call may execute more than once. **Tool workers and MCP tools should be idempotent.** Use the task's `taskId` or a correlation ID as an idempotency key.
**What is preserved:** The timed-out attempt is recorded with its input, the timeout event, and the timing. Every retry attempt is separately recorded.
## Tool call fails after side effects
**Scenario:** A tool call sends an email, then the worker crashes before reporting completion. The task is retried, and the email is sent again.
**What happens:**
1. The worker polls the task, begins execution, and sends the email.
2. The worker crashes before calling `POST /api/tasks` to report completion.
3. `responseTimeoutSeconds` fires. The task moves to `TIMED_OUT`, then `SCHEDULED` (retry).
4. A new worker picks up the task and sends the email again.
**This is at-least-once delivery.** Conductor guarantees the task will execute at least once, but it may execute more than once if the worker fails after performing side effects.
**How to handle it:**
- Make side-effecting operations idempotent. Use an idempotency key (the `taskId` is unique per attempt).
- Use the task's `updateTime` to detect redelivery — if the task was already processed, skip the side effect.
- For irreversible side effects, configure a `failureWorkflow` with compensation tasks.
## Human never responds
**Scenario:** A `HUMAN` task is waiting for approval, and nobody responds. Hours pass. Days pass.
**What happens:**
The `HUMAN` task remains `IN_PROGRESS` in durable storage indefinitely. It does not timeout unless you explicitly configure `timeoutSeconds` on the task definition.
- The workflow consumes no compute resources while waiting. No polling, no timers, no threads.
- The task survives server restarts, deploys, and infrastructure changes.
- The task is visible in the UI and queryable via API.
**If you want a timeout:** Set `timeoutSeconds` and `timeoutPolicy` on the task definition:
```json
{
"name": "human_approval",
"timeoutSeconds": 86400,
"timeoutPolicy": "TIME_OUT_WF"
}
```
This times out after 24 hours and fails the workflow. Alternatively, use `timeoutPolicy: "ALERT_ONLY"` to log a timeout without failing.
**If you want escalation:** Use a parallel `WAIT` + `HUMAN` pattern:
```json
{
"type": "FORK",
"forkTasks": [
[{"type": "HUMAN", "taskReferenceName": "approval"}],
[{"type": "WAIT", "inputParameters": {"duration": "4 hours"}},
{"type": "LLM_CHAT_COMPLETE", "taskReferenceName": "escalation_notify"}]
]
}
```
## Callback delivered twice
**Scenario:** An external system calls the Task Update API to complete a `HUMAN` task, but the network is flaky and the call is retried. Conductor receives the completion signal twice.
**What happens:**
The first call moves the task from `IN_PROGRESS` to `COMPLETED` and advances the workflow. The second call arrives for a task that is already in a terminal state.
- Conductor rejects the update. The task is already `COMPLETED`.
- No duplicate execution occurs. The workflow does not advance twice.
- The second call returns an error indicating the task is already in a terminal state.
**This is safe by default.** Conductor's task state machine enforces that a task can only transition to a terminal state once. Duplicate callbacks are harmless.
## Branch partially completes in a FORK/JOIN
**Scenario:** A `FORK/JOIN` runs three parallel branches. Branch 1 completes. Branch 2 fails. Branch 3 is still running.
**What happens:**
1. Branch 2 fails. Its task moves to `FAILED` and retries according to its retry policy.
2. Branch 3 continues executing independently.
3. The `JOIN` task waits for all branches to reach a terminal state.
4. If branch 2 exhausts its retries and moves to terminal `FAILED`, the `JOIN` task fails.
5. Branch 3 may still be running — it is not automatically canceled (unless the workflow is terminated).
6. The workflow's failure handling kicks in.
**What is preserved:** Each branch's completed tasks retain their outputs. If you retry the workflow from the failed task, only the failed branch re-executes. Successful branches are not re-run.
## Workflow definition changes mid-flight
**Scenario:** You update the workflow definition (add a task, change a parameter) while executions are running.
**What happens:**
Running executions are **not affected**. Each execution uses an immutable snapshot of the definition taken at start time. The snapshot is embedded in the execution record.
- New executions use the updated definition.
- Running executions continue with their original definition.
- You can have multiple versions running concurrently.
**If you want to apply the new definition:** Use [restart with latest definitions](../../architecture/durable-execution.md#replay-and-recovery). This re-executes the workflow from the beginning using the updated definition.
## Worker deploy during active executions
**Scenario:** You deploy a new version of your worker code. Old worker instances are shut down, new instances start up. Tasks are in-flight.
**What happens:**
1. Old workers are shut down. Tasks they were processing are abandoned.
2. `responseTimeoutSeconds` fires for abandoned tasks. Tasks move to `TIMED_OUT`, then `SCHEDULED` (retry).
3. New worker instances poll for tasks and pick up the requeued tasks.
4. Execution continues.
**Window of vulnerability:** The time between old worker shutdown and `responseTimeoutSeconds` firing. During this window, the task appears `IN_PROGRESS` but no worker is processing it.
**How to minimize impact:**
- Keep `responseTimeoutSeconds` short (10-60 seconds for most tasks).
- Use graceful shutdown in your workers — complete in-progress tasks before stopping.
- For the Conductor server itself: the sweeper service re-evaluates in-progress workflows on startup and requeues stalled tasks.
**What is never lost:** Completed task outputs. The workflow state. The execution history. Only the in-progress task is affected, and it is automatically retried.
## Dynamic task type no longer exists
**Scenario:** A `DYNAMIC` task resolves to a task type based on LLM output. The LLM returns a task name that doesn't exist (not registered, was deleted, or is misspelled).
**What happens:**
The `DYNAMIC` task fails with a resolution error — the specified task type cannot be found. The task moves to `FAILED` and retries according to its retry policy.
**How to handle it:** Validate the LLM output before the `DYNAMIC` task. Use an `INLINE` or `SWITCH` task to check that the resolved task name is in a known allowlist.
## Network partition between worker and server
**Scenario:** A worker is executing a task (e.g., an LLM call). A network partition occurs. The worker completes the task but cannot report the result to the Conductor server.
**What happens:**
1. The worker completes the LLM call and receives the response.
2. The worker attempts to report `COMPLETED` to the server. The request fails due to the network partition.
3. The worker retries the status update (SDK-level retry).
4. If the partition persists longer than `responseTimeoutSeconds`, the server marks the task as `TIMED_OUT` and requeues it.
5. When the partition heals, a worker (possibly the same one) picks up the task and re-executes the LLM call.
**Tokens are consumed twice in this scenario.** The original LLM call succeeded but the result was lost. This is the cost of at-least-once delivery. For long-running or expensive LLM calls, consider implementing client-side caching in your worker to avoid re-execution.
## Long-running agent loops over hours/days/weeks
**Scenario:** An autonomous agent loop runs for hours or days, with `WAIT` pauses, `HUMAN` approvals, and periodic LLM calls.
**What happens:**
This is a normal operating mode for Conductor. The workflow stays `RUNNING` with individual tasks in `IN_PROGRESS` (for active work) or `COMPLETED` (for finished steps).
- `WAIT` tasks consume no resources. The durable timer fires when the duration elapses, even across deploys.
- `HUMAN` tasks consume no resources. They persist until the signal arrives.
- The `DO_WHILE` loop counter and all intermediate state survive indefinitely.
- Server restarts, worker deploys, and infrastructure changes do not affect the execution.
**Practical limits:**
- Execution data grows linearly with the number of completed tasks. For very long loops (thousands of iterations), consider offloading large payloads to external storage and storing only pointers in task output. See [external payload storage](../../documentation/advanced/externalpayloadstorage.md).
- Workflow-level `timeoutSeconds` applies to the total execution. Set it high enough for your expected duration, or omit it for unlimited execution time.
## Summary: the failure contract
| Failure | What Conductor does | What you should do |
|---------|--------------------|--------------------|
| LLM call fails | Retries with configured backoff | Set retry policy on task definition |
| LLM returns bad output | Downstream task fails on input resolution | Add a validation step after LLM calls |
| Tool call times out | Retries after `responseTimeoutSeconds` | Make tools idempotent |
| Tool call has side effects, then crashes | Retries — side effect may execute twice | Use idempotency keys |
| Human never responds | Task stays `IN_PROGRESS` forever | Set `timeoutSeconds` or build escalation |
| Duplicate callback | Second call rejected, no duplicate execution | Safe by default |
| FORK branch fails | JOIN waits for all branches; workflow fails if branch exhausts retries | Configure retry policies per branch |
| Definition changes while running | Running executions unaffected (snapshot) | Use restart to apply new definitions |
| Worker deploy | In-flight tasks requeued after response timeout | Keep response timeouts short; use graceful shutdown |
| Dynamic task doesn't exist | Task fails, retries | Validate LLM output before DYNAMIC resolution |
| Network partition | Task requeued after timeout, may re-execute | Make workers idempotent; consider client-side caching |
| Multi-day execution | Normal operation, fully durable | Offload large payloads; set appropriate timeouts |
## Next steps
- **[Production Agent Architecture](production-agent-architecture.md)** — The canonical end-to-end agent pattern.
- **[Durable Execution Semantics](../../architecture/durable-execution.md)** — The full persistence model, task state machine, and retry configuration.
- **[Why Conductor for Agents](why-conductor.md)** — What Conductor gives you out of the box for agentic workflows.
- **[Token Efficiency](token-efficiency.md)** — How durable execution saves tokens across all these failure scenarios.
docs/devguide/ai/first-ai-agent.md
+294
View File
@@ -0,0 +1,294 @@
---
description: "Build your first AI agent with Conductor in 5 minutes. Step-by-step tutorial: discover MCP tools, call an LLM, execute tools, add human approval, and make it autonomous — all with durable execution guarantees."
---
# Build your first AI agent
**Build a durable AI agent in 5 minutes.** Your agent will discover tools, plan actions, execute them, and summarize results — with full crash recovery, observability, and human approval built in.
**Prerequisites:**
- Conductor running locally (`conductor server start`)
- An LLM provider API key (OpenAI or Anthropic)
- An MCP server running (we'll use a simple example below)
## Step 1: Start an MCP server
Your agent needs tools to call. MCP (Model Context Protocol) is the open standard for connecting AI agents to tools. Start a test MCP server — or use any MCP server you already have running.
```bash
pip install mcp-testkit
mcp-testkit --transport http
```
This starts an MCP server at `http://localhost:3001/mcp` with deterministic tools for testing. You'll use this URL in the workflow definition.
!!! tip "Any MCP server works"
Conductor connects to any MCP-compatible server. Use community MCP servers for GitHub, Slack, databases, or any API — or build your own. See the [MCP integration guide](mcp-guide.md) for details.
## Step 2: Configure your LLM provider
Set your API key as an environment variable before starting the server:
```bash
# Choose one (or both):
export OPENAI_API_KEY=sk-your-openai-key
export ANTHROPIC_API_KEY=sk-ant-your-anthropic-key
```
Then start (or restart) the server. Conductor auto-enables providers when their API key is set.
## Step 3: Create the agent workflow
Save this as `my_first_agent.json`. This is a complete AI agent in four tasks — no custom code, no workers, no framework:
```json
{
"name": "my_first_agent",
"description": "AI agent that discovers tools, plans, executes, and summarizes",
"version": 1,
"schemaVersion": 2,
"inputParameters": ["task"],
"tasks": [
{
"name": "discover_tools",
"taskReferenceName": "discover",
"type": "LIST_MCP_TOOLS",
"inputParameters": {
"mcpServer": "http://localhost:3001/mcp"
}
},
{
"name": "plan_action",
"taskReferenceName": "plan",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "openai",
"model": "gpt-4o-mini",
"messages": [
{
"role": "system",
"message": "You are an AI agent. Available tools: ${discover.output.tools}. The user wants to: ${workflow.input.task}. Decide which tool to use. Respond with JSON: {\"method\": \"tool_name\", \"arguments\": {}}"
},
{
"role": "user",
"message": "${workflow.input.task}"
}
],
"temperature": 0.1,
"maxTokens": 500
}
},
{
"name": "execute_tool",
"taskReferenceName": "execute",
"type": "CALL_MCP_TOOL",
"inputParameters": {
"mcpServer": "http://localhost:3001/mcp",
"method": "${plan.output.result.method}",
"arguments": "${plan.output.result.arguments}"
}
},
{
"name": "summarize_result",
"taskReferenceName": "summarize",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "openai",
"model": "gpt-4o-mini",
"messages": [
{
"role": "user",
"message": "The user asked: ${workflow.input.task}\n\nTool result: ${execute.output.content}\n\nSummarize this clearly for the user."
}
],
"maxTokens": 500
}
}
],
"outputParameters": {
"plan": "${plan.output.result}",
"toolResult": "${execute.output.content}",
"summary": "${summarize.output.result}"
}
}
```
**What each task does:**
| Task | Type | Purpose |
|------|------|---------|
| `discover` | `LIST_MCP_TOOLS` | Queries the MCP server to discover available tools |
| `plan` | `LLM_CHAT_COMPLETE` | Sends the tool list + user task to the LLM, which picks a tool and arguments |
| `execute` | `CALL_MCP_TOOL` | Calls the selected tool on the MCP server |
| `summarize` | `LLM_CHAT_COMPLETE` | Summarizes the raw tool output for the user |
Every task is a native Conductor system task. No workers to write, no code to deploy.
## Step 4: Register and run
```bash
# Register the workflow
conductor workflow create my_first_agent.json
# Run the agent synchronously — output prints directly to your terminal
curl -s -X POST 'http://localhost:8080/api/workflow/execute/my_first_agent/1' \
-H 'Content-Type: application/json' \
-d '{
"task": "What is the weather in San Francisco?"
}' | jq .
```
Or using the CLI:
```bash
conductor workflow start -w my_first_agent --sync --input '{"task": "What is the weather in San Francisco?"}'
```
Open [http://localhost:8080](http://localhost:8080) to see the execution. Click into the workflow to see each task's input, output, and timing.
!!! success "What just happened"
Your agent discovered tools from an MCP server, asked an LLM to pick the right one, executed it, and summarized the result. Every step was persisted — if the server had crashed at any point, execution would have resumed from the last completed task. No tokens wasted, no progress lost.
## Step 5: Add human approval
Real agents need guardrails. Add a `HUMAN` task between planning and execution so a person reviews the agent's plan before it acts.
Update `my_first_agent.json` — insert this task between `plan_action` and `execute_tool`:
```json
{
"name": "human_review",
"taskReferenceName": "approval",
"type": "HUMAN",
"inputParameters": {
"plannedAction": "${plan.output.result}",
"userTask": "${workflow.input.task}"
}
}
```
Now when you run the agent, it pauses after planning and waits for human approval. Approve it via the UI or API:
```bash
# Approve the plan (replace TASK_ID with the actual task ID from the execution)
curl -X POST 'http://localhost:8080/api/tasks' \
-H 'Content-Type: application/json' \
-d '{
"workflowInstanceId": "WORKFLOW_ID",
"taskId": "TASK_ID",
"status": "COMPLETED",
"outputData": {"approved": true, "reviewer": "you"}
}'
```
The approval is durable — the workflow stays paused indefinitely, even across server restarts and deploys, until someone approves it.
## Step 6: Make it autonomous
Turn your agent into an autonomous loop that keeps working until the task is done. Replace the linear workflow with a `DO_WHILE` loop:
```json
{
"name": "autonomous_agent",
"description": "Agent that loops until the task is complete",
"version": 1,
"schemaVersion": 2,
"inputParameters": ["task"],
"tasks": [
{
"name": "discover_tools",
"taskReferenceName": "discover",
"type": "LIST_MCP_TOOLS",
"inputParameters": {
"mcpServer": "http://localhost:3001/mcp"
}
},
{
"name": "agent_loop",
"taskReferenceName": "loop",
"type": "DO_WHILE",
"loopCondition": "if ($.loop['think'].output.result.done == true) { false; } else { true; }",
"loopOver": [
{
"name": "think",
"taskReferenceName": "think",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "openai",
"model": "gpt-4o-mini",
"messages": [
{
"role": "system",
"message": "You are an autonomous agent. Available tools: ${discover.output.tools}. Previous results: ${loop.output.results}. Respond with JSON: {\"action\": \"tool_name\", \"arguments\": {}, \"done\": false} when you need to use a tool, or {\"answer\": \"final answer\", \"done\": true} when the task is complete."
},
{
"role": "user",
"message": "${workflow.input.task}"
}
],
"temperature": 0.1
}
},
{
"name": "act",
"taskReferenceName": "act",
"type": "SWITCH",
"evaluatorType": "javascript",
"expression": "$.think.output.result.done ? 'done' : 'call_tool'",
"decisionCases": {
"call_tool": [
{
"name": "execute_tool",
"taskReferenceName": "tool_call",
"type": "CALL_MCP_TOOL",
"inputParameters": {
"mcpServer": "http://localhost:3001/mcp",
"method": "${think.output.result.action}",
"arguments": "${think.output.result.arguments}"
}
}
]
},
"defaultCase": []
}
]
}
],
"outputParameters": {
"answer": "${loop.output.think.output.result.answer}",
"iterations": "${loop.output.iteration}"
}
}
```
Each iteration of the loop is a durable checkpoint. If the agent crashes at iteration 12, it resumes from iteration 12 — not from the beginning. Every LLM call and tool call is persisted and observable.
## What you built
In 5 minutes, you built an AI agent that:
- **Discovers tools** from any MCP server at runtime
- **Plans actions** using an LLM
- **Executes tools** with full retry and error handling
- **Supports human approval** as a durable pause
- **Loops autonomously** until the task is complete
- **Survives crashes** without losing progress or re-running LLM calls
- **Is fully observable** — every prompt, response, tool call, and decision is recorded
All of this with zero custom code. The entire agent is a JSON workflow definition that Conductor executes with durable execution guarantees.
## Next steps
- **[MCP Integration](mcp-guide.md)** — Connect to any MCP server, expose workflows as MCP tools.
- **[Human-in-the-Loop](human-in-the-loop.md)** — Advanced approval patterns: conditional review, LLM-as-judge.
- **[Dynamic Workflows](dynamic-workflows.md)** — Agents that generate their own execution plans as JSON.
- **[Token Efficiency](token-efficiency.md)** — How durable execution saves tokens and reduces LLM costs.
- **[LLM Orchestration](llm-orchestration.md)** — 12 native LLM providers, vector databases, content generation.
docs/devguide/ai/human-in-the-loop.md
+184
View File
@@ -0,0 +1,184 @@
---
description: Human-in-the-loop patterns for AI agents — pre-execution approval, conditional post-execution review, LLM-as-judge automated review, and durable human oversight that survives server restarts.
---
# Human-in-the-loop
Production agents need oversight. Conductor's `HUMAN` task is a durable pause — the workflow stops, persists its state, and resumes only when a human responds via the Task Update API. This pause survives server restarts, deploys, and infrastructure changes. Whether the reviewer responds in 5 seconds or 5 days, the workflow state is preserved and execution resumes exactly where it left off.
Conductor supports two distinct patterns for human oversight, plus LLM-as-judge for automated review.
## Pre-execution review
The LLM plans an action and a human reviews it **before** it executes. The agent cannot proceed without approval.
```json
[
{
"name": "plan_action",
"type": "LLM_CHAT_COMPLETE",
"taskReferenceName": "plan",
"inputParameters": {
"llmProvider": "anthropic",
"model": "claude-sonnet-4-20250514",
"messages": [
{ "role": "user", "message": "Decide what action to take for: ${workflow.input.task}" }
]
}
},
{
"name": "human_approval",
"type": "HUMAN",
"taskReferenceName": "approval",
"inputParameters": {
"plannedAction": "${plan.output.result}",
"reason": "Review before executing tool call"
}
},
{
"name": "execute_action",
"type": "CALL_MCP_TOOL",
"taskReferenceName": "execute",
"inputParameters": {
"mcpServer": "${workflow.input.mcpServerUrl}",
"method": "${plan.output.result.method}",
"arguments": "${plan.output.result.arguments}"
}
}
]
```
Use this when the action has real-world consequences (sending emails, modifying data, making purchases) and you want a human gate before anything happens.
## Conditional post-execution review
The tool executes, but the result goes to a human for review **only when a condition is met** — for example, when the confidence is low, the amount exceeds a threshold, or the output affects sensitive data.
```json
[
{
"name": "execute_action",
"type": "CALL_MCP_TOOL",
"taskReferenceName": "execute",
"inputParameters": {
"mcpServer": "${workflow.input.mcpServerUrl}",
"method": "${workflow.input.method}",
"arguments": "${workflow.input.arguments}"
}
},
{
"name": "check_if_review_needed",
"type": "SWITCH",
"taskReferenceName": "review_gate",
"evaluatorType": "javascript",
"expression": "($.execute.output.confidence < 0.8 || $.execute.output.amount > 1000) ? 'needs_review' : 'auto_approve'",
"decisionCases": {
"needs_review": [
{
"name": "human_review",
"type": "HUMAN",
"taskReferenceName": "review",
"inputParameters": {
"toolResult": "${execute.output}",
"reason": "Low confidence or high-value action"
}
}
]
},
"defaultCase": []
}
]
```
Use this when most actions are safe to auto-approve but certain conditions require human oversight. The `SWITCH` task evaluates the condition; the `HUMAN` task only triggers when needed.
## LLM-as-judge: automated review
Instead of (or in addition to) a human reviewer, you can add an LLM task to evaluate the output of another LLM or tool call. This is useful for quality checks, safety screening, or validating structured output before it proceeds.
```json
[
{
"name": "generate_response",
"type": "LLM_CHAT_COMPLETE",
"taskReferenceName": "response",
"inputParameters": {
"llmProvider": "anthropic",
"model": "claude-sonnet-4-20250514",
"messages": [
{ "role": "user", "message": "Draft a customer reply for: ${workflow.input.complaint}" }
]
}
},
{
"name": "judge_response",
"type": "LLM_CHAT_COMPLETE",
"taskReferenceName": "judge",
"inputParameters": {
"llmProvider": "openai",
"model": "gpt-4o",
"messages": [
{
"role": "system",
"message": "You are a quality reviewer. Evaluate the response for tone, accuracy, and policy compliance. Respond with JSON: {\"approved\": true/false, \"reason\": \"...\"}"
},
{
"role": "user",
"message": "Customer complaint: ${workflow.input.complaint}\n\nDraft response: ${response.output.result}"
}
],
"temperature": 0.1
}
},
{
"name": "check_approval",
"type": "SWITCH",
"taskReferenceName": "gate",
"evaluatorType": "javascript",
"expression": "$.judge.output.result.approved ? 'approved' : 'rejected'",
"decisionCases": {
"rejected": [
{
"name": "escalate_to_human",
"type": "HUMAN",
"taskReferenceName": "escalation",
"inputParameters": {
"draftResponse": "${response.output.result}",
"judgeReason": "${judge.output.result.reason}"
}
}
]
},
"defaultCase": []
}
]
```
**What happens:**
1. The first LLM generates a response.
2. A second LLM (potentially a different provider or model) reviews it for quality, tone, or policy compliance.
3. If approved, the workflow continues. If rejected, it escalates to a `HUMAN` task with the judge's reasoning attached.
You can use different models for generation and review — for example, a fast model for drafting and a more capable model for judging. You can also chain multiple judges, or combine LLM-as-judge with human review as a final gate. Because each LLM call is a separate persisted task, the generation is never re-run if the judge or human review step fails.
## Combining patterns
These patterns compose naturally. A single workflow can use all three:
1. **LLM-as-judge** screens every output automatically.
2. **Conditional HITL** escalates to a human only when the judge rejects or confidence is low.
3. **Pre-execution review** gates high-stakes actions regardless of judge outcome.
Because each review step is a separate persisted task, no upstream work is repeated if a review step fails or takes time. The LLM generation that took 10 seconds and cost tokens is preserved — only the review decision needs to happen.
## Next steps
- **[Durable Agents](durable-agents.md)** &mdash; What persists, what gets retried, error handling, and multi-agent composition.
- **[Dynamic Workflows](dynamic-workflows.md)** &mdash; Agent loops, dynamic workflow generation, and tool use examples.
- **[HUMAN task reference](../../documentation/configuration/workflowdef/systemtasks/human-task.md)** &mdash; Full configuration options for the HUMAN system task.
docs/devguide/ai/index.md
+91
View File
@@ -0,0 +1,91 @@
---
description: AI agent orchestration and LLM orchestration with Conductor — LLM tasks with function calling, tool use via MCP, human-in-the-loop approval, dynamic workflows, vector database workflows, and saga pattern compensation. The open source workflow engine for AI agents.
---
# AI Cookbook
Conductor is not an AI framework. It is a durable execution engine that provides AI agent orchestration and LLM orchestration by solving the hard infrastructure problems that AI agents create: long-running processes, unreliable external calls, function calling and tool use, human-in-the-loop approval, structured output, and the need to survive failures across any of these steps. Conductor makes every agent a durable agent — one that survives crashes, retries, and infrastructure failures without losing progress.
## The problem agents create
An AI agent is a long-running process that:
1. **Calls an LLM** to decide what to do next.
2. **Calls tools** (APIs, databases, other services) to take action.
3. **Waits** for external events, human approval, or time-based delays.
4. **Loops** through plan/act/observe cycles until a goal is reached.
5. **Returns structured output** to the caller or another system.
Each of these steps can fail, take minutes to hours, or require intervention. Running this in a single process means any crash loses all progress. Running it in a queue means building your own state machine, retry logic, and observability. Conductor provides all of this out of the box.
## How it works
```mermaid
graph LR
A[Your Agent Code] -->|start workflow| B[Conductor Server]
B -->|schedule tasks| C[Task Queue]
C -->|poll| D[LLM Worker]
C -->|poll| E[Tool Worker]
C -->|poll| F[MCP Worker]
B -->|persist every step| G[(Durable Storage)]
B -->|pause & resume| H[HUMAN / WAIT]
H -->|API call or signal| B
D -->|result| B
E -->|result| B
F -->|result| B
```
Your agent code starts a workflow. Conductor schedules each step as a task, persists every input and output to durable storage, and manages retries, timeouts, and pauses. Workers (LLM calls, tool calls, MCP calls) poll for tasks, execute them, and return results. If any worker or the server itself crashes, execution resumes from the last completed step.
## How Conductor's primitives map to agent patterns
| Agent pattern | Conductor primitive | What happens mechanically |
|---|---|---|
| **LLM call** | `LLM_CHAT_COMPLETE` / `LLM_TEXT_COMPLETE` system task | Native LLM task. Configure provider and model as parameters. Retried on failure. Prompt, response, and token usage persisted. Supports built-in tools: web search, code execution, file search, extended thinking. |
| **Embeddings** | `LLM_GENERATE_EMBEDDINGS` system task | Generate vector embeddings using any configured provider. Output stored and passed to downstream tasks. |
| **Tool call / function calling** | `CALL_MCP_TOOL` system task, or `SIMPLE` / `HTTP` task | Call tools on any MCP server, or implement custom tool workers. Each call is tracked, retried on failure, and fully auditable. |
| **Tool discovery** | `LIST_MCP_TOOLS` system task | Discover available tools from an MCP server at runtime. Feed the tool list to an LLM for dynamic tool selection. |
| **RAG / semantic search** | `LLM_INDEX_TEXT` + `LLM_SEARCH_INDEX` system tasks | Index documents and run semantic search against Pinecone, pgvector, or MongoDB Atlas. No external RAG framework needed. |
| **Wait for human approval** | `HUMAN` task | Workflow pauses. Remains `IN_PROGRESS` in persistent storage. Resumes when the Task Update API is called with approval/rejection. Survives deploys. |
| **Wait for external event** | `WAIT` task (time-based) or `HUMAN` task with event handler | Durable pause. Timer or signal resolution survives server restarts. |
| **Wait for webhook** | `HUMAN` task + webhook endpoint | External system calls the Task Update API with payload. Workflow resumes with that payload as task output. |
| **Plan/act/observe loop** | `DO_WHILE` operator | Loop until a condition is met. Each iteration is a persisted step. The loop counter and state survive failures. |
| **Dynamic tool selection** | `DYNAMIC` task or `DYNAMIC_FORK` | The LLM output determines which task(s) to run next. Conductor resolves the task type at runtime. |
| **Multi-agent / sub-agent** | `SUB_WORKFLOW` task | Spawn a child agent as a sub-workflow. Parent waits for completion. Failure in a child can trigger compensation in the parent. Full observability across the entire agent tree. |
| **Rollback on failure** | `failureWorkflow` + compensation pattern | When an agent fails after taking real-world actions, a failure workflow runs compensating tasks (undo API calls, send notifications, release resources). |
| **Structured output** | Workflow `outputParameters` | Map task outputs to a structured JSON response using Conductor's expression syntax. |
| **Expose as API** | Conductor REST API: `POST /api/workflow/{name}` | Any workflow is callable via HTTP. Start synchronously or asynchronously. Get structured output back. |
| **Expose as MCP tool** | MCP Gateway integration | Register any workflow as an MCP tool. LLMs and agents invoke it directly via `LIST_MCP_TOOLS` / `CALL_MCP_TOOL` and receive structured output. |
## What you'd have to build without Conductor
If you run agents on a framework like LangChain, CrewAI, or LangGraph without a durable execution backend, you are responsible for:
- **State persistence** &mdash; Checkpointing agent progress so crashes don't restart from zero.
- **Retry logic** &mdash; Retrying failed LLM and tool calls with backoff, deduplication, and timeout handling.
- **Human-in-the-loop** &mdash; Building a pause/resume mechanism that survives process restarts and deploys.
- **Compensation** &mdash; Rolling back side effects (sent emails, created records, charged payments) when a downstream step fails.
- **Observability** &mdash; Logging every LLM prompt, response, tool call, and decision in a queryable, auditable format.
- **Multi-agent coordination** &mdash; Managing parent-child lifecycle, failure propagation, and shared state across sub-agents.
- **Scalability** &mdash; Distributing work across multiple worker processes and scaling them independently.
Conductor provides all of this as infrastructure. Your agent code focuses on the logic — what to ask the LLM, which tools to call, what to do with the results.
## Next steps
- **[Build Your First AI Agent](first-ai-agent.md)** &mdash; Step-by-step: discover MCP tools, call an LLM, execute, add human approval, make it autonomous. 5 minutes.
- **[AI & LLM Recipes](../cookbook/ai-llm.md)** &mdash; Ready-to-use recipes: chat completion, RAG, MCP agents, web search, code execution, coding agents, extended thinking, and more.
- **[LLM Orchestration](llm-orchestration.md)** &mdash; Native LLM providers, built-in tools, vector databases, and content generation.
- **[MCP Integration](mcp-guide.md)** &mdash; Connect to any MCP server, expose workflows as MCP tools, multi-server agents.
- **[Production Agent Architecture](production-agent-architecture.md)** &mdash; The canonical reference architecture for a durable production agent. End-to-end pattern with every primitive mapped.
- **[Failure Semantics for AI Agents](failure-semantics.md)** &mdash; The exact failure contract: what happens under crashes, retries, duplicates, long waits, and partial side effects.
- **[Why Conductor for Agents](why-conductor.md)** &mdash; What Conductor gives you out of the box for agentic workflows.
- **[Durable Agents](durable-agents.md)** &mdash; What persists, what gets retried, and why JSON is AI-native.
- **[Human-in-the-Loop](human-in-the-loop.md)** &mdash; Pre-execution review, conditional approval, and LLM-as-judge patterns.
- **[Dynamic Workflows](dynamic-workflows.md)** &mdash; Agent loops, dynamic workflow generation, and tool use examples.
- **[Token Efficiency](token-efficiency.md)** &mdash; How durable execution saves tokens and reduces LLM costs.
docs/devguide/ai/llm-orchestration.md
+231
View File
@@ -0,0 +1,231 @@
---
description: Native LLM orchestration with Conductor — supported LLM providers, vector database integration for RAG pipelines, and multimodal content generation tasks.
---
# LLM orchestration
Conductor provides native system tasks for LLM orchestration and integration. No external frameworks or custom workers required — configure a provider and use it in any workflow. Each provider supports function calling via MCP tool integration.
## Supported LLM providers
| Provider | Chat Completion | Text Completion | Embeddings |
|---|---|---|---|
| Anthropic (Claude) | ✓ | ✓ | — |
| OpenAI (GPT) | ✓ | ✓ | ✓ |
| Azure OpenAI | ✓ | ✓ | ✓ |
| Google Gemini | ✓ | ✓ | ✓ |
| AWS Bedrock | ✓ | ✓ | ✓ |
| Mistral | ✓ | ✓ | ✓ |
| Cohere | ✓ | ✓ | ✓ |
| HuggingFace | ✓ | ✓ | ✓ |
| Ollama | ✓ | ✓ | ✓ |
| Perplexity | ✓ | — | — |
| Grok (xAI) | ✓ | ✓ | — |
| StabilityAI | — | — | — |
No other open source workflow engine provides native LLM orchestration at this breadth. Each provider is a configuration — switch models by changing a parameter, not your code.
## Built-in tools & advanced capabilities
Conductor supports provider-native tools that run on the provider's infrastructure — no MCP server or custom worker needed. Enable them with a single parameter in the `LLM_CHAT_COMPLETE` task.
| Capability | Parameter | OpenAI | Anthropic | Google Gemini |
|---|---|---|---|---|
| Web Search | `webSearch: true` | ✓ | ✓ | ✓ |
| Code Execution | `codeInterpreter: true` | ✓ (code_interpreter) | ✓ (code_execution) | ✓ (code_execution) |
| File Search | `fileSearchVectorStoreIds: [...]` | ✓ | — | — |
| Extended Thinking | `thinkingTokenLimit: N` | — | ✓ | ✓ |
| Reasoning Effort | `reasoningEffort: "high"` | ✓ | — | — |
| Google Search | `googleSearchRetrieval: true` | — | — | ✓ |
| Custom Functions | `tools: [...]` | ✓ | ✓ | ✓ |
### Web search
The LLM can search the web for real-time information during chat completion. Enable it with `"webSearch": true`:
```json
{
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "openai",
"model": "gpt-4o-mini",
"messages": [{"role": "user", "message": "What happened in tech news today?"}],
"webSearch": true
}
}
```
Works with OpenAI, Anthropic, and Google Gemini. Each provider uses its own native web search implementation.
### Code execution
The LLM can write and execute code in a sandboxed environment. Enable it with `"codeInterpreter": true`:
```json
{
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "google_gemini",
"model": "gemini-2.5-flash",
"messages": [{"role": "user", "message": "Calculate the first 100 prime numbers and plot them"}],
"codeInterpreter": true
}
}
```
Use this for data analysis, chart generation, mathematical computation, or any task that benefits from running code.
### Extended thinking
Give the LLM a token budget for step-by-step reasoning before it responds. Useful for complex problems that benefit from chain-of-thought reasoning:
```json
{
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "anthropic",
"model": "claude-sonnet-4-20250514",
"messages": [{"role": "user", "message": "Prove that there are infinitely many primes"}],
"thinkingTokenLimit": 10000,
"maxTokens": 16000
}
}
```
Supported by Anthropic and Google Gemini.
## Vector database workflows
Built-in vector database integration enables RAG (retrieval-augmented generation) pipelines as standard vector database workflows.
| Vector Database | Store Embeddings | Index Text | Semantic Search |
|---|---|---|---|
| Pinecone | ✓ | ✓ | ✓ |
| pgvector (PostgreSQL) | ✓ | ✓ | ✓ |
| MongoDB Atlas Vector Search | ✓ | ✓ | ✓ |
### Example: RAG pipeline
A complete RAG workflow using native system tasks — index documents, search, and generate an answer. No custom workers required.
```json
{
"name": "rag_pipeline",
"description": "Index documents, search, and generate RAG answer",
"version": 1,
"schemaVersion": 2,
"tasks": [
{
"name": "index_document",
"taskReferenceName": "index_ref",
"type": "LLM_INDEX_TEXT",
"inputParameters": {
"vectorDB": "postgres-prod",
"index": "knowledge_base",
"namespace": "docs",
"docId": "${workflow.input.docId}",
"text": "${workflow.input.text}",
"embeddingModelProvider": "openai",
"embeddingModel": "text-embedding-3-small",
"dimensions": 1536,
"metadata": "${workflow.input.metadata}"
}
},
{
"name": "search_index",
"taskReferenceName": "search_ref",
"type": "LLM_SEARCH_INDEX",
"inputParameters": {
"vectorDB": "postgres-prod",
"index": "knowledge_base",
"namespace": "docs",
"query": "${workflow.input.question}",
"embeddingModelProvider": "openai",
"embeddingModel": "text-embedding-3-small",
"dimensions": 1536,
"maxResults": 3
}
},
{
"name": "generate_answer",
"taskReferenceName": "answer_ref",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "openai",
"model": "gpt-4o-mini",
"messages": [
{
"role": "system",
"message": "Answer the question using only the provided context."
},
{
"role": "user",
"message": "Context:\n${search_ref.output.result}\n\nQuestion: ${workflow.input.question}"
}
],
"temperature": 0.2
}
}
],
"outputParameters": {
"searchResults": "${search_ref.output.result}",
"answer": "${answer_ref.output.result}"
}
}
```
Every task type — `LLM_INDEX_TEXT`, `LLM_SEARCH_INDEX`, `LLM_CHAT_COMPLETE` — is a native Conductor system task. The vector database, embedding model, and LLM provider are all configuration parameters. Switch from pgvector to Pinecone or from OpenAI to Anthropic by changing a parameter value.
## Content generation
Native system tasks for multimodal content generation:
| Task | Type | Description |
|---|---|---|
| Generate Image | `GENERATE_IMAGE` | Text-to-image generation via AI models |
| Generate Audio | `GENERATE_AUDIO` | Text-to-speech synthesis |
| Generate Video | `GENERATE_VIDEO` | Text/image-to-video generation (async) |
| Generate PDF | `GENERATE_PDF` | Markdown-to-PDF document conversion |
## Examples
Ready-to-use workflow definitions for every AI task type. Each example is a complete JSON workflow you can register and run directly.
| Example | Task types used |
|---|---|
| [Chat Completion](https://github.com/conductor-oss/conductor/blob/main/ai/examples/01-chat-completion.json) | `LLM_CHAT_COMPLETE` |
| [Generate Embeddings](https://github.com/conductor-oss/conductor/blob/main/ai/examples/02-generate-embeddings.json) | `LLM_GENERATE_EMBEDDINGS` |
| [Image Generation](https://github.com/conductor-oss/conductor/blob/main/ai/examples/03-image-generation.json) | `GENERATE_IMAGE` |
| [Audio Generation](https://github.com/conductor-oss/conductor/blob/main/ai/examples/04-audio-generation.json) | `GENERATE_AUDIO` |
| [Semantic Search](https://github.com/conductor-oss/conductor/blob/main/ai/examples/05-semantic-search.json) | `LLM_SEARCH_INDEX` |
| [RAG Basic](https://github.com/conductor-oss/conductor/blob/main/ai/examples/06-rag-basic.json) | `LLM_SEARCH_INDEX`, `LLM_CHAT_COMPLETE` |
| [RAG Complete](https://github.com/conductor-oss/conductor/blob/main/ai/examples/07-rag-complete.json) | `LLM_INDEX_TEXT`, `LLM_SEARCH_INDEX`, `LLM_CHAT_COMPLETE` |
| [MCP List Tools](https://github.com/conductor-oss/conductor/blob/main/ai/examples/08-mcp-list-tools.json) | `LIST_MCP_TOOLS` |
| [MCP Call Tool](https://github.com/conductor-oss/conductor/blob/main/ai/examples/09-mcp-call-tool.json) | `CALL_MCP_TOOL` |
| [MCP AI Agent](https://github.com/conductor-oss/conductor/blob/main/ai/examples/10-mcp-ai-agent.json) | `LIST_MCP_TOOLS`, `LLM_CHAT_COMPLETE`, `CALL_MCP_TOOL` |
| [Video — OpenAI Sora](https://github.com/conductor-oss/conductor/blob/main/ai/examples/11-video-openai-sora.json) | `GENERATE_VIDEO` |
| [Video — Gemini Veo](https://github.com/conductor-oss/conductor/blob/main/ai/examples/12-video-gemini-veo.json) | `GENERATE_VIDEO` |
| [Image-to-Video Pipeline](https://github.com/conductor-oss/conductor/blob/main/ai/examples/13-image-to-video-pipeline.json) | `GENERATE_IMAGE`, `GENERATE_VIDEO` |
| [StabilityAI Image](https://github.com/conductor-oss/conductor/blob/main/ai/examples/14-stabilityai-image.json) | `GENERATE_IMAGE` |
| [PDF Generation](https://github.com/conductor-oss/conductor/blob/main/ai/examples/15-pdf-generation.json) | `GENERATE_PDF` |
| [LLM-to-PDF Pipeline](https://github.com/conductor-oss/conductor/blob/main/ai/examples/16-llm-to-pdf-pipeline.json) | `LLM_CHAT_COMPLETE`, `GENERATE_PDF` |
| [Web Search](https://github.com/conductor-oss/conductor/blob/main/ai/examples/17-web-search.json) | `LLM_CHAT_COMPLETE` (web search) |
| [Code Execution](https://github.com/conductor-oss/conductor/blob/main/ai/examples/18-code-execution.json) | `LLM_CHAT_COMPLETE` (code execution) |
| [Coding Agent](https://github.com/conductor-oss/conductor/blob/main/ai/examples/19-coding-agent.json) | `LLM_CHAT_COMPLETE` (code_interpreter) |
| [Extended Thinking](https://github.com/conductor-oss/conductor/blob/main/ai/examples/20-extended-thinking.json) | `LLM_CHAT_COMPLETE` (thinking) |
| [Web Research Agent](https://github.com/conductor-oss/conductor/blob/main/ai/examples/21-web-search-research-agent.json) | `LLM_CHAT_COMPLETE` (web search + thinking), `GENERATE_PDF` |
| [Multi-Turn Chain](https://github.com/conductor-oss/conductor/blob/main/ai/examples/22-multi-turn-chain.json) | `LLM_CHAT_COMPLETE` (previousResponseId) |
Browse all examples: [`ai/examples/`](https://github.com/conductor-oss/conductor/tree/main/ai/examples)
## Next steps
- **[Durable Agents](durable-agents.md)** &mdash; What persists, what gets retried, and why JSON is AI-native.
- **[Dynamic Workflows](dynamic-workflows.md)** &mdash; Agents that build their own execution plans at runtime.
- **[AI & LLM Recipes](../cookbook/ai-llm.md)** &mdash; Practical recipes for common LLM workflow patterns.
docs/devguide/ai/mcp-guide.md
+245
View File
@@ -0,0 +1,245 @@
---
description: "MCP (Model Context Protocol) integration with Conductor — connect AI agents to external tools, discover tools at runtime, execute with durable retry, and expose workflows as MCP tools."
---
# MCP integration
MCP (Model Context Protocol) is the open standard for connecting AI agents to tools and data sources. Conductor provides native MCP integration — discover tools, call them with full durability, and expose your own workflows as MCP tools.
## What is MCP
MCP defines a protocol for how AI agents discover and use tools. Instead of hardcoding API integrations, your agent asks an MCP server "what tools do you have?" and gets back a structured list. The agent (or the LLM) picks the right tool, and the MCP server executes it.
**Without MCP:** Every tool integration is custom code — different auth, different schemas, different error handling.
**With MCP:** Tools are standardized. Connect once, use any MCP-compatible tool server.
Conductor supports MCP as a first-class integration with two native system tasks.
## Native MCP system tasks
### LIST_MCP_TOOLS — discover available tools
Queries an MCP server and returns the list of tools it offers, including names, descriptions, and parameter schemas.
```json
{
"name": "discover_tools",
"taskReferenceName": "discover",
"type": "LIST_MCP_TOOLS",
"inputParameters": {
"mcpServer": "${workflow.input.mcpServerUrl}"
}
}
```
**Output:** A structured list of tools with their schemas. Pass this directly to an LLM so it can decide which tool to call.
**Why this matters:** Tool discovery happens at runtime. Your agent doesn't need to know which tools exist at design time — it discovers them dynamically. Add a new tool to the MCP server, and every agent using it gains that capability immediately.
### CALL_MCP_TOOL — execute a tool
Calls a specific tool on an MCP server with the given arguments.
```json
{
"name": "execute_tool",
"taskReferenceName": "execute",
"type": "CALL_MCP_TOOL",
"inputParameters": {
"mcpServer": "${workflow.input.mcpServerUrl}",
"method": "${plan.output.result.method}",
"arguments": "${plan.output.result.arguments}"
}
}
```
**What Conductor adds on top of raw MCP:**
- **Durable execution** — if the tool call fails, Conductor retries according to the task's retry policy. The retry is automatic and configurable (fixed delay, exponential backoff, linear backoff).
- **Full audit trail** — every tool call is persisted: the method, arguments, response, timing, and retry history. You can inspect exactly what your agent did.
- **Crash recovery** — if the server crashes between tool calls, the workflow resumes from the last completed step. The tool call is never silently lost.
- **Timeout handling** — configure `responseTimeoutSeconds` to prevent stuck tool calls from blocking your agent.
## Connecting to MCP servers
Conductor connects to any MCP server via HTTP. Pass the server URL as a workflow input or hardcode it in the task definition.
```json
{
"mcpServer": "http://localhost:3001/mcp"
}
```
### Using multiple MCP servers
An agent can connect to multiple MCP servers in the same workflow. Discover tools from each server, combine the tool lists, and let the LLM choose across all of them:
```json
{
"name": "multi_tool_agent",
"version": 1,
"schemaVersion": 2,
"tasks": [
{
"name": "discover_github_tools",
"taskReferenceName": "github_tools",
"type": "LIST_MCP_TOOLS",
"inputParameters": {
"mcpServer": "http://localhost:3001/mcp"
}
},
{
"name": "discover_db_tools",
"taskReferenceName": "db_tools",
"type": "LIST_MCP_TOOLS",
"inputParameters": {
"mcpServer": "http://localhost:3002/mcp"
}
},
{
"name": "plan_with_all_tools",
"taskReferenceName": "plan",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "openai",
"model": "gpt-4o-mini",
"messages": [
{
"role": "system",
"message": "Available tools: GitHub: ${github_tools.output.tools}, Database: ${db_tools.output.tools}. User task: ${workflow.input.task}. Pick the best tool. Respond with JSON: {\"server\": \"github\" or \"db\", \"method\": \"tool_name\", \"arguments\": {}}"
}
],
"temperature": 0.1
}
}
]
}
```
## Exposing workflows as MCP tools
Any Conductor workflow can be exposed as an MCP tool via the MCP Gateway. This means other agents and LLMs can discover and invoke your workflows using the MCP protocol.
```
Agent → LIST_MCP_TOOLS → discovers your workflow
Agent → CALL_MCP_TOOL → starts your workflow
Conductor → executes with full durability
Agent → receives structured output
```
Your workflow's `inputParameters` become the tool's input schema, and `outputParameters` become the tool's output. The workflow runs with full durable execution guarantees — retries, persistence, compensation — while appearing to the calling agent as a simple tool call.
This creates a composable architecture: workflows call MCP tools, and workflows *are* MCP tools. Agents can invoke other agents' workflows without knowing they're workflows.
## MCP vs HTTP vs custom workers
| Approach | When to use |
|----------|-------------|
| **MCP** (`LIST_MCP_TOOLS` + `CALL_MCP_TOOL`) | Tools exposed via MCP servers. Dynamic tool discovery. Agent decides which tool to call at runtime. |
| **HTTP** (`HTTP` system task) | Direct API calls with known endpoints. No tool discovery needed. |
| **Custom workers** (`SIMPLE` task) | Complex business logic that needs custom code. Multi-step processing. |
MCP is the best choice when your agent needs to **discover tools dynamically** or when you want to **standardize tool access** across multiple agents. Use HTTP for simple, known API calls. Use custom workers for logic that doesn't fit into a single API call.
## Complete example: MCP agent with approval
A production-ready agent that discovers tools, plans, gets human approval, executes, and summarizes:
```json
{
"name": "mcp_agent_with_approval",
"description": "Discover tools, plan, execute with approval, summarize",
"version": 1,
"schemaVersion": 2,
"inputParameters": ["task", "mcpServerUrl"],
"tasks": [
{
"name": "list_available_tools",
"taskReferenceName": "discover_tools",
"type": "LIST_MCP_TOOLS",
"inputParameters": {
"mcpServer": "${workflow.input.mcpServerUrl}"
}
},
{
"name": "decide_which_tools_to_use",
"taskReferenceName": "plan",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "anthropic",
"model": "claude-sonnet-4-20250514",
"messages": [
{
"role": "system",
"message": "You are an AI agent. Available tools: ${discover_tools.output.tools}. User wants to: ${workflow.input.task}"
},
{
"role": "user",
"message": "Which tool should I use and what parameters? Respond with JSON: {\"method\": \"string\", \"arguments\": {}}"
}
],
"temperature": 0.1,
"maxTokens": 500
}
},
{
"name": "human_review",
"taskReferenceName": "approval",
"type": "HUMAN",
"inputParameters": {
"plannedAction": "${plan.output.result}"
}
},
{
"name": "execute_tool",
"taskReferenceName": "execute",
"type": "CALL_MCP_TOOL",
"inputParameters": {
"mcpServer": "${workflow.input.mcpServerUrl}",
"method": "${plan.output.result.method}",
"arguments": "${plan.output.result.arguments}"
}
},
{
"name": "summarize_result",
"taskReferenceName": "summarize",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "anthropic",
"model": "claude-sonnet-4-20250514",
"messages": [
{
"role": "user",
"message": "The user asked: ${workflow.input.task}\n\nTool result: ${execute.output.content}\n\nSummarize this result for the user."
}
],
"maxTokens": 500
}
}
],
"outputParameters": {
"plan": "${plan.output.result}",
"toolResult": "${execute.output.content}",
"summary": "${summarize.output.result}",
"approvedBy": "${approval.output.reviewer}"
}
}
```
Every task type here — `LIST_MCP_TOOLS`, `LLM_CHAT_COMPLETE`, `CALL_MCP_TOOL`, `HUMAN` — is a native Conductor system task. No custom code needed.
## Next steps
- **[Build Your First AI Agent](first-ai-agent.md)** — Step-by-step tutorial using MCP.
- **[Dynamic Workflows](dynamic-workflows.md)** — Agents that generate their own execution plans.
- **[Human-in-the-Loop](human-in-the-loop.md)** — Approval patterns for MCP tool calls.
- **[LLM Orchestration](llm-orchestration.md)** — 12 native LLM providers, vector databases, content generation.
docs/devguide/ai/production-agent-architecture.md
+451
View File
@@ -0,0 +1,451 @@
---
description: "The canonical reference architecture for building production AI agents on Conductor — end-to-end pattern with planner, tool selection, execution, retry, memory, human approval, long waits, reflection loops, budget caps, and full observability."
---
# Production agent architecture
This is the reference architecture for a durable AI agent on Conductor. Not a toy. Not a feature list. This is the exact pattern for an agent that plans, acts, waits, recovers, and runs in production.
## Architecture diagram
<div style="margin: 2rem 0;">
<svg viewBox="0 0 720 820" xmlns="http://www.w3.org/2000/svg" style="max-width: 720px; width: 100%; height: auto;">
<defs>
<marker id="pa-arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z" fill="#4a5568"/></marker>
<marker id="pa-arrow-teal" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z" fill="#06d6a0"/></marker>
<marker id="pa-arrow-red" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z" fill="#dc2626"/></marker>
<marker id="pa-arrow-blue" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z" fill="#3b82f6"/></marker>
</defs>
<!-- Background for loop region -->
<rect x="30" y="215" width="660" height="480" rx="12" fill="rgba(6,214,160,0.06)" stroke="#06d6a0" stroke-width="1.5" stroke-dasharray="6,4"/>
<text x="50" y="240" font-size="11" font-weight="600" fill="#06d6a0" font-family="sans-serif">DO_WHILE — Agent Loop (checkpointed per iteration)</text>
<!-- Start -->
<circle cx="360" cy="30" r="22" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/>
<text x="360" y="35" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">Start</text>
<!-- Discover Tools -->
<line x1="360" y1="52" x2="360" y2="80" stroke="#4a5568" stroke-width="1.5" marker-end="url(#pa-arrow)"/>
<rect x="245" y="80" width="230" height="40" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/>
<text x="360" y="98" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif" font-weight="600">Discover Tools</text>
<text x="360" y="112" text-anchor="middle" font-size="9" fill="#4a5568" font-family="sans-serif">LIST_MCP_TOOLS</text>
<!-- Init Memory -->
<line x1="360" y1="120" x2="360" y2="148" stroke="#4a5568" stroke-width="1.5" marker-end="url(#pa-arrow)"/>
<rect x="245" y="148" width="230" height="40" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/>
<text x="360" y="166" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif" font-weight="600">Initialize Memory</text>
<text x="360" y="180" text-anchor="middle" font-size="9" fill="#4a5568" font-family="sans-serif">SET_VARIABLE</text>
<!-- Arrow into loop -->
<line x1="360" y1="188" x2="360" y2="260" stroke="#4a5568" stroke-width="1.5" marker-end="url(#pa-arrow)"/>
<!-- Plan (LLM) -->
<rect x="245" y="260" width="230" height="45" rx="6" fill="#3b82f6" stroke="#2563eb" stroke-width="1.5"/>
<text x="360" y="280" text-anchor="middle" font-size="11" fill="#fff" font-weight="600" font-family="sans-serif">Plan Next Action</text>
<text x="360" y="296" text-anchor="middle" font-size="9" fill="rgba(255,255,255,0.8)" font-family="sans-serif">LLM_CHAT_COMPLETE</text>
<!-- Switch: done / needs_approval / execute -->
<line x1="360" y1="305" x2="360" y2="335" stroke="#4a5568" stroke-width="1.5" marker-end="url(#pa-arrow)"/>
<polygon points="360,335 400,365 360,395 320,365" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/>
<text x="360" y="362" text-anchor="middle" font-size="9" fill="#2e3545" font-family="sans-serif" font-weight="600">SWITCH</text>
<text x="360" y="374" text-anchor="middle" font-size="8" fill="#4a5568" font-family="sans-serif">done?</text>
<!-- Done branch (exits loop) — goes right and down to end -->
<line x1="400" y1="365" x2="620" y2="365" stroke="#06d6a0" stroke-width="1.5"/>
<text x="500" y="358" text-anchor="middle" font-size="9" fill="#06d6a0" font-family="sans-serif" font-weight="600">done = true</text>
<line x1="620" y1="365" x2="620" y2="770" stroke="#06d6a0" stroke-width="1.5" marker-end="url(#pa-arrow-teal)"/>
<!-- Needs approval branch — goes left -->
<line x1="320" y1="365" x2="140" y2="365" stroke="#4a5568" stroke-width="1.5"/>
<text x="230" y="358" text-anchor="middle" font-size="9" fill="#4a5568" font-family="sans-serif">needs_approval</text>
<line x1="140" y1="365" x2="140" y2="420" stroke="#4a5568" stroke-width="1.5" marker-end="url(#pa-arrow)"/>
<!-- Human Approval -->
<rect x="55" y="420" width="170" height="45" rx="6" fill="#f59e0b" stroke="#d97706" stroke-width="1.5"/>
<text x="140" y="440" text-anchor="middle" font-size="11" fill="#fff" font-weight="600" font-family="sans-serif">Human Approval</text>
<text x="140" y="456" text-anchor="middle" font-size="9" fill="rgba(255,255,255,0.8)" font-family="sans-serif">HUMAN (durable pause)</text>
<!-- Arrow from approval to tool -->
<line x1="140" y1="465" x2="140" y2="500" stroke="#4a5568" stroke-width="1.5"/>
<line x1="140" y1="500" x2="360" y2="500" stroke="#4a5568" stroke-width="1.5"/>
<!-- Execute branch — goes straight down -->
<line x1="360" y1="395" x2="360" y2="490" stroke="#4a5568" stroke-width="1.5" marker-end="url(#pa-arrow)"/>
<text x="375" y="440" font-size="9" fill="#4a5568" font-family="sans-serif">execute</text>
<!-- Execute Tool -->
<rect x="265" y="490" width="190" height="45" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/>
<text x="360" y="510" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif" font-weight="600">Execute Tool</text>
<text x="360" y="526" text-anchor="middle" font-size="9" fill="#4a5568" font-family="sans-serif">CALL_MCP_TOOL</text>
<!-- Retry badge on tool -->
<circle cx="465" cy="500" r="12" fill="#f59e0b" stroke="#d97706" stroke-width="1"/>
<text x="465" y="504" text-anchor="middle" font-size="9" fill="#fff" font-weight="bold" font-family="sans-serif">!</text>
<text x="485" y="504" font-size="8" fill="#4a5568" font-family="sans-serif">auto-retry</text>
<!-- Update Memory -->
<line x1="360" y1="535" x2="360" y2="570" stroke="#4a5568" stroke-width="1.5" marker-end="url(#pa-arrow)"/>
<rect x="255" y="570" width="210" height="40" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/>
<text x="360" y="588" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif" font-weight="600">Update Memory</text>
<text x="360" y="602" text-anchor="middle" font-size="9" fill="#4a5568" font-family="sans-serif">SET_VARIABLE</text>
<!-- Budget check -->
<line x1="360" y1="610" x2="360" y2="640" stroke="#4a5568" stroke-width="1.5" marker-end="url(#pa-arrow)"/>
<polygon points="360,640 400,665 360,690 320,665" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/>
<text x="360" y="662" text-anchor="middle" font-size="8" fill="#2e3545" font-family="sans-serif" font-weight="600">Budget</text>
<text x="360" y="674" text-anchor="middle" font-size="8" fill="#4a5568" font-family="sans-serif">check</text>
<!-- Loop back arrow -->
<line x1="320" y1="665" x2="80" y2="665" stroke="#06d6a0" stroke-width="1.5"/>
<line x1="80" y1="665" x2="80" y2="282" stroke="#06d6a0" stroke-width="1.5"/>
<line x1="80" y1="282" x2="245" y2="282" stroke="#06d6a0" stroke-width="1.5" marker-end="url(#pa-arrow-teal)"/>
<text x="68" y="480" text-anchor="middle" font-size="9" fill="#06d6a0" font-family="sans-serif" font-weight="600" transform="rotate(-90 68 480)">next iteration</text>
<!-- Budget exceeded — exit loop -->
<line x1="400" y1="665" x2="620" y2="665" stroke="#dc2626" stroke-width="1.5"/>
<text x="510" y="658" text-anchor="middle" font-size="9" fill="#dc2626" font-family="sans-serif" font-weight="600">budget exceeded</text>
<line x1="620" y1="665" x2="620" y2="770" stroke="#dc2626" stroke-width="1.5"/>
<!-- End -->
<line x1="360" y1="695" x2="360" y2="770" stroke="#4a5568" stroke-width="1.5" marker-end="url(#pa-arrow)"/>
<circle cx="360" cy="792" r="22" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/>
<text x="360" y="797" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">End</text>
<!-- Compensation annotation -->
<rect x="490" y="748" width="180" height="42" rx="6" fill="#fff" stroke="#dc2626" stroke-width="1" stroke-dasharray="4,3"/>
<text x="580" y="765" text-anchor="middle" font-size="9" fill="#dc2626" font-family="sans-serif" font-weight="600">On failure:</text>
<text x="580" y="780" text-anchor="middle" font-size="9" fill="#dc2626" font-family="sans-serif">failureWorkflow runs</text>
<text x="580" y="790" text-anchor="middle" font-size="9" fill="#dc2626" font-family="sans-serif">compensation</text>
<line x1="490" y1="770" x2="385" y2="785" stroke="#dc2626" stroke-width="1" stroke-dasharray="3,3"/>
<!-- Persistence annotation -->
<rect x="500" y="260" width="150" height="50" rx="6" fill="#fff" stroke="#3b82f6" stroke-width="1" stroke-dasharray="4,3"/>
<text x="575" y="278" text-anchor="middle" font-size="9" fill="#3b82f6" font-family="sans-serif" font-weight="600">Every step persisted</text>
<text x="575" y="292" text-anchor="middle" font-size="9" fill="#3b82f6" font-family="sans-serif">Prompt, response,</text>
<text x="575" y="304" text-anchor="middle" font-size="9" fill="#3b82f6" font-family="sans-serif">tokens, timing</text>
<line x1="500" y1="285" x2="475" y2="282" stroke="#3b82f6" stroke-width="1" stroke-dasharray="3,3"/>
</svg>
</div>
## The canonical agent pattern
A production agent has these concerns. Each one maps to a specific Conductor primitive:
| Agent concern | Conductor primitive | How it works |
|---|---|---|
| **Plan next action** | `LLM_CHAT_COMPLETE` | LLM receives goal + context + tool list, returns structured plan |
| **Select tool at runtime** | `DYNAMIC` task | LLM output determines which task type executes next |
| **Execute tool** | `CALL_MCP_TOOL`, `HTTP`, or `SIMPLE` worker | Tool runs with retry policy, timeout, and full I/O recording |
| **Retry with backoff** | Task definition `retryLogic` | `FIXED`, `EXPONENTIAL_BACKOFF`, or `LINEAR_BACKOFF` — no code needed |
| **Parallel tool calls** | `FORK/JOIN` or `DYNAMIC_FORK` | Fan out to N tools in parallel, join when all complete |
| **Memory / context handoff** | `SET_VARIABLE` + workflow variables | Accumulate results across loop iterations; pass to next LLM call |
| **Human approval gate** | `HUMAN` task | Durable pause. Survives restarts and deploys. Resumes on API signal. |
| **Long wait (hours/days)** | `WAIT` task | Timer-based durable pause. Survives server restarts. |
| **Resume from external event** | `HUMAN` task + webhook/API | External system calls Task Update API. Workflow resumes with payload. |
| **Reflection / evaluation loop** | `DO_WHILE` with LLM-as-judge | Second LLM evaluates output quality; loop continues if below threshold |
| **Budget / iteration cap** | `DO_WHILE` `loopCondition` | `iteration < maxIterations` or token/cost check in loop condition |
| **Termination criteria** | `DO_WHILE` exit + `SWITCH` | LLM sets `done: true`, or evaluator decides goal is met |
| **Delegate to specialist** | `SUB_WORKFLOW` or `START_WORKFLOW` | Spawn child agent. Parent waits. Failure propagates. Full observability across the tree. |
| **Compensation on failure** | `failureWorkflow` | Undo side effects: revoke API calls, send notifications, release resources |
| **Audit trail** | Automatic | Every task's input, output, timing, retry count, and worker ID is persisted |
## End-to-end workflow
Here is the complete agent as a single Conductor workflow. Every step is a native system task or operator — no custom code, no external framework.
```json
{
"name": "production_agent",
"description": "Reference architecture: durable production agent",
"version": 1,
"schemaVersion": 2,
"inputParameters": ["goal", "mcpServerUrl", "maxIterations"],
"tasks": [
{
"name": "discover_tools",
"taskReferenceName": "discover",
"type": "LIST_MCP_TOOLS",
"inputParameters": {
"mcpServer": "${workflow.input.mcpServerUrl}"
}
},
{
"name": "initialize_memory",
"taskReferenceName": "init_memory",
"type": "SET_VARIABLE",
"inputParameters": {
"context": [],
"actions_taken": []
}
},
{
"name": "agent_loop",
"taskReferenceName": "loop",
"type": "DO_WHILE",
"loopCondition": "if ($.loop['plan'].output.result.done == true) { false; } else if ($.loop['plan'].output.iteration >= $.maxIterations) { false; } else { true; }",
"inputParameters": {
"maxIterations": "${workflow.input.maxIterations}"
},
"loopOver": [
{
"name": "plan_next_action",
"taskReferenceName": "plan",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "anthropic",
"model": "claude-sonnet-4-20250514",
"messages": [
{
"role": "system",
"message": "You are a production AI agent. Goal: ${workflow.input.goal}\n\nAvailable tools: ${discover.output.tools}\n\nPrevious actions and results: ${workflow.variables.context}\n\nDecide the next action. Respond with JSON:\n- To use a tool: {\"action\": \"tool_name\", \"arguments\": {}, \"reasoning\": \"why\", \"needs_approval\": true/false, \"done\": false}\n- To finish: {\"answer\": \"final answer\", \"done\": true}"
}
],
"temperature": 0.1,
"maxTokens": 1000
}
},
{
"name": "check_if_done",
"taskReferenceName": "done_check",
"type": "SWITCH",
"evaluatorType": "javascript",
"expression": "$.plan.output.result.done ? 'done' : ($.plan.output.result.needs_approval ? 'needs_approval' : 'execute')",
"decisionCases": {
"needs_approval": [
{
"name": "human_approval",
"taskReferenceName": "approval",
"type": "HUMAN",
"inputParameters": {
"plannedAction": "${plan.output.result.action}",
"arguments": "${plan.output.result.arguments}",
"reasoning": "${plan.output.result.reasoning}",
"goal": "${workflow.input.goal}"
}
},
{
"name": "execute_approved_tool",
"taskReferenceName": "approved_tool_call",
"type": "CALL_MCP_TOOL",
"inputParameters": {
"mcpServer": "${workflow.input.mcpServerUrl}",
"method": "${plan.output.result.action}",
"arguments": "${plan.output.result.arguments}"
}
},
{
"name": "update_memory_approved",
"taskReferenceName": "mem_update_approved",
"type": "SET_VARIABLE",
"inputParameters": {
"context": "${workflow.variables.context.concat([{action: plan.output.result.action, result: approved_tool_call.output.content, approved: true}])}"
}
}
],
"execute": [
{
"name": "execute_tool",
"taskReferenceName": "tool_call",
"type": "CALL_MCP_TOOL",
"inputParameters": {
"mcpServer": "${workflow.input.mcpServerUrl}",
"method": "${plan.output.result.action}",
"arguments": "${plan.output.result.arguments}"
}
},
{
"name": "update_memory",
"taskReferenceName": "mem_update",
"type": "SET_VARIABLE",
"inputParameters": {
"context": "${workflow.variables.context.concat([{action: plan.output.result.action, result: tool_call.output.content}])}"
}
}
]
},
"defaultCase": []
}
]
}
],
"outputParameters": {
"answer": "${loop.output.plan.output.result.answer}",
"iterations": "${loop.output.iteration}",
"actions_taken": "${workflow.variables.context}"
},
"failureWorkflow": "agent_compensation_workflow"
}
```
## What makes this production-ready
### Every step is a durable checkpoint
Each iteration of `DO_WHILE` is persisted before the next begins. If the agent crashes at iteration 15 of 20, it resumes from iteration 15 — not from scratch. Every LLM prompt, response, tool call, and human decision is recorded.
### Human approval is a durable gate
The `HUMAN` task pauses the workflow indefinitely. The pause survives server restarts, deploys, and infrastructure changes. When a reviewer approves via the API or UI, the workflow resumes with the approval payload as task output. No polling, no timeouts (unless you configure one), no lost approvals.
### Retry is automatic and configurable
Every tool call (`CALL_MCP_TOOL`, `HTTP`, `SIMPLE`) inherits retry behavior from its [task definition](../../documentation/configuration/taskdef.md):
```json
{
"name": "execute_tool",
"retryCount": 3,
"retryLogic": "EXPONENTIAL_BACKOFF",
"retryDelaySeconds": 2,
"responseTimeoutSeconds": 30
}
```
If the MCP server is down, Conductor retries with exponential backoff. The LLM is **not** re-called — only the failed tool call retries.
### Memory persists across iterations
`SET_VARIABLE` stores accumulated context in workflow variables. These variables are persisted to durable storage and available to every subsequent task. The LLM receives the full history of actions and results on each iteration.
### Budget cap prevents runaway agents
The `loopCondition` checks both the agent's `done` flag and an iteration cap. You can also check token usage or cost in the condition. The agent terminates cleanly when the budget is exhausted.
### Compensation handles side effects
If the agent fails after taking real-world actions (sent an email, created a record, charged a payment), the `failureWorkflow` runs compensating tasks automatically. The compensation workflow receives the full execution context: which actions succeeded, which failed, and why.
### Observability is automatic
Open the Conductor UI to see:
- The exact task graph for this execution
- Every LLM prompt and response (click any `LLM_CHAT_COMPLETE` task)
- Every tool call with input, output, and timing
- Every human approval with who approved and when
- The iteration count and loop state
- Retry history for any failed task
- The full workflow input, output, and variables
## Extending the pattern
### Add parallel research
Replace a single tool call with `DYNAMIC_FORK` to fan out to multiple tools in parallel:
```json
{
"name": "parallel_research",
"taskReferenceName": "research",
"type": "DYNAMIC_FORK",
"inputParameters": {
"dynamicTasks": "${plan.output.result.parallel_tasks}",
"dynamicTasksInput": "${plan.output.result.task_inputs}"
},
"dynamicForkTasksParam": "dynamicTasks",
"dynamicForkTasksInputParamName": "dynamicTasksInput"
}
```
The LLM decides how many tools to call in parallel and with what inputs. Conductor creates the branches at runtime.
### Add a reflection / evaluation step
Insert an LLM-as-judge after tool execution to evaluate output quality:
```json
{
"name": "evaluate_result",
"taskReferenceName": "evaluator",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "anthropic",
"model": "claude-sonnet-4-20250514",
"messages": [
{
"role": "system",
"message": "Evaluate this result against the goal. Is it sufficient? Respond with JSON: {\"quality\": \"good\" or \"insufficient\", \"feedback\": \"...\"}"
},
{
"role": "user",
"message": "Goal: ${workflow.input.goal}\nResult: ${tool_call.output.content}"
}
]
}
}
```
If the evaluator returns `insufficient`, the loop continues with the feedback as context for the next planning step.
### Add long waits
Insert a `WAIT` task for time-based pauses (rate limiting, cooldown periods, scheduled actions):
```json
{
"name": "wait_before_retry",
"taskReferenceName": "cooldown",
"type": "WAIT",
"inputParameters": {
"duration": "1 hour"
}
}
```
The wait is durable. The workflow does not consume resources while waiting. After 1 hour — even if the server restarted during that time — the workflow resumes.
### Delegate to specialist agents
Use `SUB_WORKFLOW` to spawn a child agent for a specialized task:
```json
{
"name": "delegate_to_researcher",
"taskReferenceName": "research_agent",
"type": "SUB_WORKFLOW",
"inputParameters": {
"name": "research_agent_workflow",
"version": 1,
"input": {
"topic": "${plan.output.result.research_topic}",
"mcpServerUrl": "${workflow.input.mcpServerUrl}"
}
}
}
```
The parent agent waits for the child to complete. If the child fails, the parent's failure handling kicks in. The entire agent tree is observable in the UI — drill from parent to child to sub-child.
## The primitives, mapped
| "I need my agent to..." | Use this | Why |
|---|---|---|
| Wait for a tool callback | `HUMAN` task or async completion | Durable pause. Resumes on API signal with payload. |
| Sleep until a retry window | `WAIT` task | Timer-based durable pause. Zero resource consumption. |
| Pick the next tool at runtime | `DYNAMIC` task | LLM output determines task type. Resolved at execution time. |
| Call multiple tools in parallel | `FORK/JOIN` or `DYNAMIC_FORK` | Static or runtime-determined parallelism. Join waits for all. |
| Loop until goal is met | `DO_WHILE` | Checkpointed loop. Each iteration persisted. |
| Delegate to a specialist agent | `SUB_WORKFLOW` or `START_WORKFLOW` | Child workflow with full lifecycle management. |
| Accumulate context across steps | `SET_VARIABLE` | Workflow variables persisted to durable storage. |
| Evaluate output quality | `LLM_CHAT_COMPLETE` as evaluator | LLM-as-judge pattern inside the loop. |
| Cap iterations or cost | `DO_WHILE` `loopCondition` | Check iteration count, token usage, or cost. |
| Undo side effects on failure | `failureWorkflow` | Compensation tasks run automatically on workflow failure. |
| Pause for human review | `HUMAN` task | Indefinite durable pause. Survives restarts and deploys. |
| Resume on external event | `HUMAN` task + API/webhook | External system calls Task Update API with payload. |
| Post-process structured output | `INLINE` (JavaScript) or `JSON_JQ_TRANSFORM` | Server-side transforms without a worker. |
## Next steps
- **[Failure Semantics for AI Agents](failure-semantics.md)** — The exact failure contract: what happens under crashes, retries, duplicates, and long waits.
- **[Why Conductor for Agents](why-conductor.md)** — What Conductor gives you out of the box for agentic workflows.
- **[Build Your First AI Agent](first-ai-agent.md)** — Start simple and build up to this architecture in 5 minutes.
- **[MCP Integration](mcp-guide.md)** — Connect to any MCP server, expose workflows as MCP tools.
- **[Token Efficiency](token-efficiency.md)** — How durable execution saves tokens and reduces LLM costs.
docs/devguide/ai/token-efficiency.md
+117
View File
@@ -0,0 +1,117 @@
---
description: "How durable execution saves LLM tokens and reduces AI costs — crash recovery without re-execution, replay without re-running LLM calls, and the real cost of non-durable agent frameworks."
---
# Token efficiency with durable execution
LLM calls are expensive. Every token costs money, and every re-execution burns tokens that were already paid for. Durable execution eliminates wasted tokens by ensuring that completed work is never lost.
## The cost of crashes without durability
Consider an autonomous agent that runs a 20-step loop. Each iteration calls an LLM (planning) and a tool (execution). The agent is on iteration 18 when the process crashes.
**Without durable execution:**
The agent restarts from iteration 1. Iterations 1-17 must re-execute — 17 LLM calls that produce the exact same output as before. The tokens are burned again, the tool calls re-execute (potentially causing duplicate side effects), and the user waits for work that was already done.
**With Conductor:**
The agent resumes from iteration 18. Iterations 1-17 are already persisted — their LLM outputs, tool results, and state are all in durable storage. Zero tokens wasted. Zero duplicate tool calls. The agent picks up exactly where it left off.
## Where tokens are saved
### 1. Crash recovery
Every LLM call in a Conductor workflow is persisted at completion. The prompt, response, token usage, and model are all recorded. If the server, worker, or network fails:
- Completed LLM calls are **never re-executed**. Their outputs are read from storage.
- Only the in-progress call is retried — and only that single call.
- The workflow resumes from the last persisted state.
**Token savings:** Proportional to how far the agent progressed before the crash. An agent that crashes at step 18 of 20 saves 17 LLM calls worth of tokens.
### 2. Retry from failed task
When a workflow fails (e.g., a tool call returns an error after the LLM planned successfully), you can [retry from the failed task](../../architecture/durable-execution.md#replay-and-recovery). Conductor reuses the outputs of all previously completed tasks.
**Example:** A 5-task agent workflow fails at task 4 (tool execution). Tasks 1-3 included two LLM calls that consumed 8,000 tokens total. Retry from task 4:
- Tasks 1-3 are **not re-executed**. Their outputs (including LLM responses) are reused from storage.
- Only task 4 (and anything after it) re-executes.
- **8,000 tokens saved** per retry.
### 3. Rerun from a specific task
When you fix a bug in a task definition and [rerun from that task](../../architecture/durable-execution.md#replay-and-recovery), all tasks before it keep their persisted outputs. Upstream LLM calls are not re-executed.
### 4. Loop checkpointing
Agent loops (`DO_WHILE`) checkpoint every iteration. If the loop runs 50 iterations and the agent crashes at iteration 48:
- Iterations 1-47 are persisted with all their LLM calls and tool results.
- Only iteration 48 re-executes.
- **47 iterations of LLM tokens saved.**
Without durability, the entire loop restarts from iteration 1.
## Real-world cost impact
Here's a concrete example using typical LLM pricing:
| Scenario | Without durability | With Conductor | Savings |
|----------|-------------------|----------------|---------|
| 20-step agent, crash at step 18 | Re-run all 20 steps: ~40K tokens | Resume from step 18: ~4K tokens | **~36K tokens ($0.04-$0.40)** |
| RAG pipeline fails at PDF generation | Re-run embedding + LLM: ~12K tokens | Retry only PDF step: 0 LLM tokens | **~12K tokens ($0.01-$0.12)** |
| 100-iteration loop, crash at 95 | Re-run all 100: ~200K tokens | Resume from 95: ~10K tokens | **~190K tokens ($0.19-$1.90)** |
| Agent with human approval, reviewer slow | Process may timeout and restart | HUMAN task persists indefinitely | **All upstream tokens preserved** |
These are per-execution savings. Multiply by thousands of daily executions and the cost difference becomes significant.
At scale — thousands of agent executions per day — even a 5% crash/retry rate translates to substantial token waste without durability. With Conductor, that waste drops to near zero.
## Token savings beyond crashes
Durable execution saves tokens in scenarios beyond crashes:
**Long-running agents with human-in-the-loop.** A HUMAN task can pause a workflow for hours or days. Without durability, the process might timeout or be killed, requiring a full restart (and re-running all upstream LLM calls). With Conductor, the pause is durable — the workflow resumes exactly where it stopped, with all LLM outputs preserved.
**Deployment and scaling.** When you deploy a new version of your workers or scale down instances, in-flight workflows survive. No LLM calls are lost. Without durability, scaling events can kill processes mid-execution, wasting all tokens consumed so far.
**Debugging and iteration.** When debugging a failed agent, you can inspect every LLM prompt and response without re-running the agent. Rerun from a specific task to test a fix without re-executing (and re-paying for) upstream LLM calls.
## How it works mechanically
Conductor persists LLM task outputs the same way it persists any task output:
1. The `LLM_CHAT_COMPLETE` task is scheduled and a worker (or the server itself) executes it.
2. The LLM response is received — prompt, completion, token usage, model, and latency are all recorded.
3. The task moves to `COMPLETED` and its output is **written to durable storage** before the next task is scheduled.
4. If anything fails after this point, the LLM output is already persisted. It is never re-executed.
This is the same persistence model that applies to every task in Conductor — the [durable execution semantics](../../architecture/durable-execution.md) guarantee that completed work is never lost.
## Comparison: durable vs non-durable frameworks
| | Non-durable (LangChain, CrewAI, custom) | Durable (Conductor) |
|---|---|---|
| **Crash at step N of M** | Restart from step 1. All N tokens re-consumed. | Resume from step N. Zero tokens wasted. |
| **Retry after tool failure** | Re-run entire chain including LLM calls. | Retry only the failed task. LLM outputs preserved. |
| **Long pause (human review)** | Process may die. Full restart required. | Durable pause. Resume with all state intact. |
| **Debugging** | Re-run the agent to reproduce. More tokens. | Inspect persisted outputs. Rerun from any task. |
| **Deploy/scale** | In-flight work may be lost. | Workflows survive scaling events. |
The bottom line: **durable execution is a cost optimization**, not just a reliability feature. Every crash, retry, pause, or debugging session that would re-execute LLM calls in a non-durable framework is free in Conductor — because the work was already persisted.
## Next steps
- **[Durable Agents](durable-agents.md)** — What persists, what gets retried, and why JSON is AI-native.
- **[Durable Execution Semantics](../../architecture/durable-execution.md)** — The full persistence and recovery model.
- **[Build Your First AI Agent](first-ai-agent.md)** — Step-by-step tutorial with durable execution built in.
- **[LLM Orchestration](llm-orchestration.md)** — 14+ native LLM providers, vector databases, content generation.
docs/devguide/ai/why-conductor.md
+324
View File
@@ -0,0 +1,324 @@
---
description: "Why Conductor for AI agents — native LLM tasks, MCP tool calling, deterministic JSON definitions, durable human-in-the-loop, and dynamic runtime execution. Show-don't-tell with code examples."
---
# Why Conductor for agents
Conductor is the original durable workflow orchestration engine — born at Netflix to run microservices at internet scale, now powering AI agents with the same battle-tested execution model. Other engines give you generic primitives and say "build your agent infrastructure yourself." Conductor gives you the agent infrastructure. Here's what that looks like in practice.
## Call an LLM — zero boilerplate
Other engines treat LLM calls as generic function calls. You build the abstraction: prompt construction, provider switching, response parsing, token tracking, retry logic. On Conductor, an LLM call is a system task:
```json
{
"name": "plan_action",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "anthropic",
"model": "claude-sonnet-4-20250514",
"messages": [
{"role": "system", "message": "You are a planning agent. Tools: ${tools.output}"},
{"role": "user", "message": "${workflow.input.goal}"}
],
"temperature": 0.1,
"maxTokens": 1000
}
}
```
That's it. No SDK wrapper, no worker code, no retry logic. Conductor executes it, persists the prompt, response, token usage, model, and latency. Switch providers by changing `llmProvider` — from `anthropic` to `openai` to `bedrock` — with zero code changes. 14+ providers supported natively.
On other engines, this same task requires:
- A worker/activity function that constructs the HTTP request
- Provider-specific SDK initialization and auth
- Response parsing and error handling
- Custom logging for prompt/response/token tracking
- Retry configuration in your code, not the orchestrator
Every team builds this differently. Every implementation has different bugs.
## Discover and call tools — native MCP
MCP (Model Context Protocol) is the open standard for agent tool use. On Conductor, tool discovery and execution are system tasks:
```json
[
{
"name": "discover",
"type": "LIST_MCP_TOOLS",
"inputParameters": {
"mcpServer": "http://localhost:3001/mcp"
}
},
{
"name": "execute",
"type": "CALL_MCP_TOOL",
"inputParameters": {
"mcpServer": "http://localhost:3001/mcp",
"method": "${plan.output.result.method}",
"arguments": "${plan.output.result.arguments}"
}
}
]
```
The agent discovers tools at runtime, the LLM picks the right one, and Conductor executes it with automatic retry, timeout, and full audit trail. Connect to any MCP server — GitHub, Slack, databases, custom APIs — with no wrapper code.
On other engines, you write a "Durable MCP" wrapper: a custom activity/worker that connects to the MCP server, marshals requests, handles errors, and logs results. For every MCP server. For every tool type.
## Human-in-the-loop — one line, durable forever
An agent needs human approval before a risky action. On Conductor:
```json
{
"name": "approval_gate",
"type": "HUMAN",
"inputParameters": {
"action": "${plan.output.result.action}",
"reasoning": "${plan.output.result.reasoning}"
}
}
```
The workflow pauses. The pause survives server restarts, deploys, infrastructure changes — indefinitely. When someone approves via the API or UI, the workflow resumes with the approval payload. No polling, no timer hacks, no external state.
On other engines, you implement `wait_condition()` with signal handlers, write the signal routing code, and build the approval UI integration yourself. The pause mechanism is in your workflow code, not in the platform.
## Agent loops — checkpointed per iteration
An autonomous agent loops: plan, act, observe, repeat. On Conductor, each iteration is a durable checkpoint:
```json
{
"name": "agent_loop",
"type": "DO_WHILE",
"loopCondition": "if ($.loop['think'].output.result.done == true) { false; } else if ($.loop['think'].output.iteration >= 20) { false; } else { true; }",
"loopOver": [
{
"name": "think",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "anthropic",
"model": "claude-sonnet-4-20250514",
"messages": [
{"role": "system", "message": "Goal: ${workflow.input.goal}. Previous results: ${workflow.variables.context}. Respond with {action, arguments, done}."}
]
}
},
{
"name": "act",
"type": "CALL_MCP_TOOL",
"inputParameters": {
"mcpServer": "${workflow.input.mcpServerUrl}",
"method": "${think.output.result.action}",
"arguments": "${think.output.result.arguments}"
}
},
{
"name": "remember",
"type": "SET_VARIABLE",
"inputParameters": {
"context": "${workflow.variables.context.concat([{action: think.output.result.action, result: act.output.content}])}"
}
}
]
}
```
If the agent crashes at iteration 18 of 20, it resumes from iteration 18. Not from scratch. The 17 completed LLM calls and tool executions are already persisted — zero tokens wasted, zero duplicate side effects. The loop condition enforces an iteration cap so the agent can't run forever.
On other engines, you build the loop in your workflow code. If the process crashes, you either restart from the beginning (burning all tokens again) or build your own checkpointing mechanism.
## Dynamic workflows — LLMs generate execution plans
This is the capability no other engine can match. An LLM generates a complete workflow definition as JSON, and Conductor executes it immediately:
```json
{
"name": "execute_agent_plan",
"type": "START_WORKFLOW",
"inputParameters": {
"startWorkflow": {
"workflowDefinition": "${planner_llm.output.result}",
"input": "${workflow.input.taskInput}"
}
}
}
```
The LLM's output is a Conductor workflow definition. No code generation. No compilation. No deployment pipeline. The generated workflow runs with the same durable execution guarantees as any hand-written workflow — persistence, retries, observability, replay.
Combined with `DYNAMIC` tasks (resolve which task to run at runtime) and `DYNAMIC_FORK` (create N parallel branches at runtime), Conductor is more dynamic than code-based engines. Not despite using JSON — because of it. Data is easier to generate, transform, and compose than code.
On code-based engines, dynamic workflows require generating source code, compiling it, deploying it, and then executing it. That friction fundamentally limits how dynamically an AI system can operate.
## RAG pipelines — native vector database support
Retrieval-augmented generation as two system tasks, no external framework:
```json
[
{
"name": "search",
"type": "LLM_SEARCH_INDEX",
"inputParameters": {
"vectorDB": "postgres-prod",
"namespace": "kb",
"index": "articles",
"embeddingModelProvider": "openai",
"embeddingModel": "text-embedding-3-small",
"query": "${workflow.input.question}"
}
},
{
"name": "answer",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "anthropic",
"model": "claude-sonnet-4-20250514",
"messages": [
{"role": "system", "message": "Answer based on: ${search.output.result}"},
{"role": "user", "message": "${workflow.input.question}"}
]
}
}
]
```
Pinecone, pgvector, and MongoDB Atlas are supported natively. No LangChain, no custom retrieval workers, no framework dependencies.
## Multi-agent delegation — sub-workflows with lifecycle
A parent agent delegates to specialist agents. Each specialist is a sub-workflow with full lifecycle management:
```json
{
"name": "parallel_research",
"type": "DYNAMIC_FORK",
"inputParameters": {
"dynamicTasks": "${planner.output.result.research_tasks}",
"dynamicTasksInput": "${planner.output.result.task_inputs}"
},
"dynamicForkTasksParam": "dynamicTasks",
"dynamicForkTasksInputParamName": "dynamicTasksInput"
}
```
The LLM decides how many research agents to spawn and what each one investigates. Conductor creates the branches at runtime, runs them in parallel, and joins the results. If one branch fails, it retries independently without affecting the others. The parent agent sees the full execution tree — drill from parent to child to sub-child in the UI.
## Long-running workflows — evolve without breaking
An agent workflow runs for days. Midway through, you need to fix a bug or add a step. On code-based engines, this is where things get painful — you end up littering your workflow code with version guards and `if/else` branches to keep old executions replaying correctly while new ones pick up the change. Every change adds a permanent branch that can never be removed. After a year of iteration, the workflow is an archaeology site of version checks.
Conductor eliminates this entirely. Each execution snapshots its definition at start time:
```json
{
"name": "agent_workflow",
"version": 2,
"tasks": [
{"name": "plan", "type": "LLM_CHAT_COMPLETE", "...": "..."},
{"name": "validate", "type": "INLINE", "...": "..."},
{"name": "execute", "type": "CALL_MCP_TOOL", "...": "..."}
]
}
```
Running executions continue with their original definition. New executions pick up the updated definition. No version guards. No branching. No archaeology. Update the definition, register it, and move on. If you need to apply the new definition to a running execution, [restart it](../../architecture/durable-execution.md#replay-and-recovery) — Conductor re-executes the workflow with the latest definition from the beginning.
This is not a minor convenience. For AI agents that run for hours or days — iterating through plan/act/observe loops, waiting for human approvals, pausing for external events — the ability to evolve the workflow definition without version branching is the difference between a maintainable system and a fragile one.
## Guaranteed execution — failure is not a choice
Conductor was built as a state machine engine at Netflix to orchestrate microservices at internet scale. The execution model is designed around one principle: **every task will be executed to completion, or every failure will be explicitly handled.** There is no silent failure mode.
The guarantees:
- **At-least-once task delivery** — Every task is persisted to durable storage before execution. If a worker crashes, the task is automatically requeued and delivered to another worker. Tasks do not disappear.
- **Sweeper recovery** — A background sweeper service continuously scans for stalled tasks. If a task is `IN_PROGRESS` but its worker has gone silent (no heartbeat, past `responseTimeoutSeconds`), the sweeper requeues it. If the Conductor server itself restarts, the sweeper recovers all in-flight work on startup.
- **Configurable retry policies** — Every task has retry count, delay, and backoff strategy. Retries are managed by the engine, not your code. Exponential backoff, fixed delay, and linear backoff are built in.
- **Failure workflows** — When a workflow fails after exhausting retries, a `failureWorkflow` runs automatically. This is where you put compensation logic: undo API calls, release resources, send alerts. The failure workflow has the full context of what failed and why.
- **Terminal state is always reached** — A workflow always reaches `COMPLETED`, `FAILED`, or `TERMINATED`. There is no limbo state. You can query, alert, and act on any terminal state.
```json
{
"name": "critical_agent",
"failureWorkflow": "agent_failure_handler",
"tasks": [
{
"name": "risky_action",
"type": "CALL_MCP_TOOL",
"retryCount": 5,
"retryLogic": "EXPONENTIAL_BACKOFF",
"retryDelaySeconds": 10,
"responseTimeoutSeconds": 30,
"timeoutPolicy": "RETRY"
}
]
}
```
This task retries 5 times with exponential backoff (10s, 20s, 40s, 80s, 160s). If the worker doesn't respond within 30 seconds, the task is timed out and retried. If all retries are exhausted, the workflow fails and `agent_failure_handler` runs with full context. At no point does the task silently disappear.
These guarantees apply uniformly across the entire workflow graph — including sub-workflows, dynamic forks, and agent loops. You configure them declaratively in the definition. The engine enforces them.
## Deterministic by construction
JSON workflow definitions cannot have side effects. There is no ambient state, no thread-local context, no hidden mutation. Given the same inputs, a Conductor workflow schedules the same tasks in the same order, every time. This is why [replay](../../architecture/durable-execution.md#replay-and-recovery) works unconditionally — restart a workflow from three months ago and it re-executes the same graph.
When workflow logic lives in code, developers must manually enforce determinism constraints: no system clocks, no random numbers, no uncontrolled I/O. Violating these constraints causes subtle replay bugs that are hard to detect and harder to debug. Conductor eliminates this entire class of bugs by construction — JSON cannot have side effects.
## Observability — automatic, not opt-in
Every `LLM_CHAT_COMPLETE` task automatically records:
- The full prompt (every message in the conversation)
- The complete response
- Token usage (prompt tokens, completion tokens, total)
- Model and provider
- Latency
- Retry history (if any)
Every `CALL_MCP_TOOL` task records the method, arguments, response, and timing. Every `HUMAN` task records who approved, when, and with what payload. All of this is queryable via API and visible in the UI.
On other engines, you build this logging yourself. Every team does it differently, with different coverage and different gaps.
## The agent use case matrix
Every agentic pattern maps to a specific Conductor primitive:
| Use case | Conductor pattern |
|---|---|
| **Tool-calling agent** | `LLM_CHAT_COMPLETE` + `CALL_MCP_TOOL` |
| **Approval-gated actions** | `HUMAN` task + `SWITCH` for timeout |
| **Planner/executor loop** | `DO_WHILE` + `SET_VARIABLE` |
| **Multi-agent delegation** | `SUB_WORKFLOW` or `DYNAMIC_FORK` |
| **Long wait for external system** | `HUMAN` or `WAIT` task |
| **High fan-out research** | `DYNAMIC_FORK` + `JOIN` |
| **RAG pipeline** | `LLM_SEARCH_INDEX` + `LLM_CHAT_COMPLETE` |
| **Content generation** | `GENERATE_IMAGE` / `GENERATE_AUDIO` / `GENERATE_VIDEO` / `GENERATE_PDF` |
| **Agent that builds its own plan** | `LLM_CHAT_COMPLETE` + `START_WORKFLOW` with inline definition |
| **Deterministic post-processing** | `INLINE` (JavaScript) or `JSON_JQ_TRANSFORM` |
## Next steps
- **[Production Agent Architecture](production-agent-architecture.md)** — The canonical end-to-end agent pattern, fully wired.
- **[Failure Semantics for AI Agents](failure-semantics.md)** — The exact failure contract under every scenario.
- **[Build Your First AI Agent](first-ai-agent.md)** — From zero to a running agent in 5 minutes.
- **[Token Efficiency](token-efficiency.md)** — How durable execution saves tokens and reduces LLM costs.
docs/devguide/architecture/PollTimeoutSeconds.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 75 KiB

docs/devguide/architecture/ResponseTimeoutSeconds.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 28 KiB

docs/devguide/architecture/TaskFailure.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 125 KiB

docs/devguide/architecture/TimeoutSeconds.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 60 KiB

docs/devguide/architecture/conductor-architecture.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 736 KiB

docs/devguide/architecture/dag_workflow.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 64 KiB

docs/devguide/architecture/dag_workflow2.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 17 KiB

docs/devguide/architecture/directed-acyclic-graph.md
+54
View File
@@ -0,0 +1,54 @@
---
description: "Directed Acyclic Graph (DAG) — understand how Conductor models workflows as DAGs for reliable task orchestration."
---
# Directed Acyclic Graph (DAG)
All Conductor workflows are directed acyclic graphs (DAGs). A directed acyclic graph (DAG) is a set of vertices where the connections are unidirectional without any repetition. DAG workflows can only "move forward" and cannot redo a step (or series of steps).
Here is a breakdown of what DAG means:
- **Graph**
For DAGs, a graph refers to "a collection of vertices (or points) and edges (or lines) that indicate connections between the vertices."
<img alt="A regular graph (source: Wikipedia)." src="regular_graph.png" width="300">
Imagine that each vertex in the graph above is a microservice. The lines represent a dependency relation between each microservice. However, this graph is not a directed graph, as there is no direction given to each dependency.
- **Directed**
A directed graph means that there is a direction to each connection. For example, this graph is directed:
<img alt="A directed graph." src="directed_graph.png" width="300">
Each line has a direction. In the example above, Point N can proceed directly to B, but B cannot proceed directly to N.
- **Acyclic**
Acyclic means without circular or cyclic paths. The example shown above contains directed cyclic graphs, such as A -> B -> D -> A. In contrast, a directed acyclic graph can only begin at one point and end at a different point (A -> B -> D).
## Workflows as DAGs
Since a Conductor workflow is a series of tasks that can connect in only a specific direction and cannot loop, it is a directed acyclic graph:
![A Conductor workflow.](dag_workflow2.png)
The flow of tasks is specified in a `tasks` array in a JSON file called a workflow definition, which can also be written in code (Python, Java, JavaScript, C#, Go, Clojure).
### Can a workflow contain loops and still be a DAG?
Yes. Take the following Conductor workflow, which contains Do While loops, for example:
![A Conductor workflow with Do While loop.](dag_workflow.png)
This workflow is still a DAG because the loop is just a simplified representation for running multiple instances of the same tasks repeatedly. For example, if the 2nd loop in the above workflow is run three times, the workflow path will be:
1. zero_offset_fix_1
2. post_to_orbit_ref_1
3. zero_offset_fix_2
4. post_to_orbit_ref_2
5. zero_offset_fix_3
6. post_to_orbit_ref_3
The path is directed forward to different task instances, each with its own unique inputs and outputs. The Do While loop simply makes it easier to represent this path.
docs/devguide/architecture/directed_graph.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 58 KiB

docs/devguide/architecture/index.md
+49
View File
@@ -0,0 +1,49 @@
---
description: "Conductor system architecture — worker-task queue model, state machine evaluator, pluggable data stores, and RPC-based polling for durable code execution."
---
# Architecture Overview
This diagram showcases an overview of Conductor's system architecture:
![Conductor's Architecture diagram.](conductor-architecture.png)
In Conductor, workflows are executed on a worker-task queue architecture, where each task type (HTTP, Event, Wait, *example_simple_task* and so on) has its own dedicated task queue. The key components of Conductor’s core orchestration engine include:
* **State machine evaluator**—Orchestrates workflows by scheduling tasks to their relevant queues and assigning them to active workers when polled. Monitors each task's state and ensures it is completed, retried, or failed as required.
* **Task queues**—Distributed queues for each task type, where tasks are completed on a first-in-first-out basis.
* **Task workers**—Poll the Conductor server via HTTP or gRPC for tasks, execute tasks, and update the server on the task status. Each worker is responsible for carrying out a specific task type.
* **Data stores** (Redis by default)—High-availability persistence stores that maintain workflow and task metadata, task queues, and execution history
* **APIs**—REST APIs for programmatic access to the Conductor server.
By default, Conductor uses Redis as its data store, with Elasticsearch used for its indexing backend. These [storage layers are pluggable](../../documentation/advanced/extend.md), allowing you to work with alternative backends and queue service providers.
## Task execution
With a worker-task queue architecture, Conductor schedules and assigns tasks to its designated task queues based on its task type. Conductor follows an RPC-based communication model where task workers run on a separate machine from the server and communicate over HTTP-based endpoints with the server.
The workers employ a polling model for managing their designated queues, and update Conductor with the task status.
![Runtime Model of Conductor.](overview.png)
### Worker-server polling mechanism
Each worker declares beforehand what task(s) it can execute. At runtime, task workers poll its designated task queue(s) to receive and execute scheduled work. Conductor passes task inputs to the worker for execution and collects the task outputs, continuing the process according to the workflow definition.
By default, workers infinitely poll Conductor every 100ms. The polling interval value for each type of worker can be adjusted accordingly based on factors like workload. Here is the polling mechanism in detail:
1. The application starts a workflow execution by interacting with Orkes Conductor, which returns a workflow (execution) ID. It can be used to track the workflow's progress and manage its execution.
2. Conductor schedules the first task in the workflow to its task queue.
3. The workers responsible for executing the first task within the workflow are polling Orkes Conductor for tasks to execute via HTTP or gRPC. When a task is scheduled, Conductor sends it to the next available worker, which then performs the required work.
4. Periodically, the worker returns the task status to Conductor (e.g. IN PROGRESS, FAILED, COMPLETED, etc).
5. Once the first task in the workflow instance is completed, the worker returns the task output to the server, and Conductor schedules the next set of tasks to be performed.
Conductor manages and maintains the workflow state, keeping track of which tasks have been completed and which are still pending. This ensures that the workflow is executed correctly, with each task triggered precisely at the right time.
Using the workflow ID, the application can check the Conductor server for the workflow status at any time. This is particularly useful for asynchronous or long-running workflows, as it allows the application to monitor the workflow's progress and take appropriate action, such as pausing or terminating the workflow if needed.
docs/devguide/architecture/overview.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 48 KiB

docs/devguide/architecture/pirate_graph.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 23 KiB

docs/devguide/architecture/regular_graph.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 25 KiB

docs/devguide/architecture/task_states.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 37 KiB

docs/devguide/architecture/tasklifecycle.md
+183
View File
@@ -0,0 +1,183 @@
---
description: "Understand the task lifecycle in Conductor — state transitions, retries, timeouts, and failure handling for durable workflow execution."
---
# Task Lifecycle
During a workflow execution, each task transitions through a series of states. Understanding these transitions is key to configuring retries, timeouts, and error handling correctly.
## State diagram
```mermaid
stateDiagram-v2
[*] --> SCHEDULED
SCHEDULED --> IN_PROGRESS : Worker polls task
SCHEDULED --> TIMED_OUT : Poll timeout exceeded
SCHEDULED --> CANCELED : Workflow terminated
IN_PROGRESS --> COMPLETED : Worker reports success
IN_PROGRESS --> FAILED : Worker reports failure
IN_PROGRESS --> FAILED_WITH_TERMINAL_ERROR : Non-retryable failure
IN_PROGRESS --> TIMED_OUT : Response/task timeout exceeded
IN_PROGRESS --> COMPLETED_WITH_ERRORS : Optional task fails
SCHEDULED --> SKIPPED : Skip Task API called
FAILED --> SCHEDULED : Retry (after delay)
TIMED_OUT --> SCHEDULED : Retry (after delay)
COMPLETED --> [*]
FAILED --> [*] : Retries exhausted or totalTimeoutSeconds exceeded
FAILED_WITH_TERMINAL_ERROR --> [*]
TIMED_OUT --> [*] : Retries exhausted or totalTimeoutSeconds exceeded
CANCELED --> [*]
SKIPPED --> [*]
COMPLETED_WITH_ERRORS --> [*]
```
## Task statuses
| Status | Description |
| :--- | :--- |
| `SCHEDULED` | Task is queued and waiting for a worker to poll it. |
| `IN_PROGRESS` | A worker has picked up the task and is executing it. |
| `COMPLETED` | Task completed successfully. |
| `FAILED` | Task failed due to an error. Conductor will retry based on the task definition's retry configuration. |
| `FAILED_WITH_TERMINAL_ERROR` | Task failed with a non-retryable error. No retries will be attempted. |
| `TIMED_OUT` | Task exceeded its configured timeout. Conductor will retry based on the retry configuration. |
| `CANCELED` | Task was canceled because the workflow was terminated. |
| `SKIPPED` | Task was skipped via the Skip Task API. The workflow continues to the next task. |
| `COMPLETED_WITH_ERRORS` | Task failed but is marked as optional in the workflow definition. The workflow continues. |
## Retry behavior
When a task fails with a retryable error, Conductor automatically reschedules it after the configured delay.
```mermaid
sequenceDiagram
participant W as Worker
participant C as Conductor Server
C->>W: Task T1 available for polling
W->>C: Poll task T1
C-->>W: Return T1 (IN_PROGRESS)
W->>W: Process task...
W->>C: Report FAILED (after 10s)
C->>C: Persist failed execution
Note over C: Wait retryDelaySeconds (5s)
C->>C: Schedule new T1 execution
C->>W: T1 available for polling again
W->>C: Poll task T1
C-->>W: Return T1 (IN_PROGRESS)
W->>W: Process task...
W->>C: Report COMPLETED
```
Retry behavior is controlled by the task definition:
| Parameter | Description |
| :--- | :--- |
| `retryCount` | Maximum number of retry attempts. |
| `retryLogic` | `FIXED`, `EXPONENTIAL_BACKOFF`, or `LINEAR_BACKOFF`. See [Retry Logic](../../../documentation/configuration/taskdef.md#retry-logic). |
| `retryDelaySeconds` | Base delay between retries. |
| `maxRetryDelaySeconds` | Caps the computed delay. Prevents exponential growth from becoming arbitrarily large. |
| `backoffJitterMs` | Adds random milliseconds to each delay to spread concurrent retries over time. |
| `totalTimeoutSeconds` | Hard wall-clock budget across all attempts. See [Total timeout](#total-timeout). |
## Timeout scenarios
### Poll timeout
If no worker polls the task within `pollTimeoutSeconds`, it is marked as `TIMED_OUT`.
```mermaid
sequenceDiagram
participant W as Worker
participant C as Conductor Server
C->>C: Schedule task T1
Note over C,W: No worker polls within 60s
C->>C: Mark T1 as TIMED_OUT
C->>C: Schedule retry (if retries remain)
```
This typically indicates a backlogged task queue or insufficient workers.
### Response timeout
If a worker polls a task but doesn't report back within `responseTimeoutSeconds`, the task is marked as `TIMED_OUT`. This handles cases where a worker crashes mid-execution.
```mermaid
sequenceDiagram
participant W as Worker
participant C as Conductor Server
C->>W: Task T1 available
W->>C: Poll T1
C-->>W: Return T1 (IN_PROGRESS)
W->>W: Processing...
Note over W: Worker crashes
Note over C: responseTimeoutSeconds (20s) elapsed
C->>C: Mark T1 as TIMED_OUT
Note over C: Wait retryDelaySeconds (5s)
C->>C: Schedule new T1 execution
```
Workers can extend the response timeout by sending `IN_PROGRESS` status updates with a `callbackAfterSeconds` value.
### Task timeout
`timeoutSeconds` is the overall SLA for task completion. Even if a worker keeps sending `IN_PROGRESS` updates, the task is marked as `TIMED_OUT` once this duration is exceeded.
```mermaid
sequenceDiagram
participant W as Worker
participant C as Conductor Server
C->>W: Task T1 available
W->>C: Poll T1
C-->>W: Return T1 (IN_PROGRESS)
W->>W: Processing...
W->>C: IN_PROGRESS (callback: 9s)
Note over C: Task back in queue, invisible 9s
W->>C: Poll T1 again
W->>C: IN_PROGRESS (callback: 9s)
Note over C: Cycle repeats...
Note over C: timeoutSeconds (30s) elapsed
C->>C: Mark T1 as TIMED_OUT
C->>C: Schedule retry (if retries remain)
W->>C: Report COMPLETED (at 32s)
Note over C: Ignored — T1 already terminal
```
### Total timeout
`totalTimeoutSeconds` limits the total wall-clock time across **all** retry attempts. Once this budget is consumed, no further retries are scheduled regardless of how many remain in `retryCount`.
```mermaid
sequenceDiagram
participant W as Worker
participant C as Conductor Server
Note over C: totalTimeoutSeconds = 30s
C->>W: Task T1 (attempt 1)
W->>C: FAILED (at t=5s)
Note over C: Retry delay 5s
C->>W: Task T1 (attempt 2, at t=10s)
W->>C: FAILED (at t=20s)
Note over C: Retry delay 5s
C->>W: Task T1 (attempt 3, at t=25s)
W->>C: FAILED (at t=28s)
Note over C: t=28s ≥ 30s → total budget exhausted
C->>C: Mark workflow FAILED — no more retries
```
This is useful when you need a hard SLA on how long a task can run across all its attempts, independent of how many retries are configured.
## Timeout configuration summary
| Parameter | Description | Default |
| :--- | :--- | :--- |
| `pollTimeoutSeconds` | Max time for a worker to poll the task. | No timeout |
| `responseTimeoutSeconds` | Max time for a worker to respond after polling. | 600s |
| `timeoutSeconds` | SLA per individual attempt (from first `IN_PROGRESS` to terminal). | No timeout |
| `totalTimeoutSeconds` | Hard budget across all attempts combined. Overrides `retryCount`. | No timeout |
| `timeoutPolicy` | Action on timeout: `RETRY`, `TIME_OUT_WF` (fail workflow), or `ALERT_ONLY`. | `TIME_OUT_WF` |
docs/devguide/bestpractices.md
+274
View File
@@ -0,0 +1,274 @@
---
description: "Production best practices for Conductor — idempotency, retry logic with exponential backoff, timeouts, payload management, horizontal scaling of workers, saga patterns, and deployment strategies for durable execution at scale."
---
# Best Practices
This guide covers production best practices for running Conductor as a durable execution engine at scale. Every recommendation here comes from real-world operational experience.
## Idempotent workers
Conductor guarantees **at-least-once** task delivery. Network partitions, worker restarts, and response timeouts can all cause a task to be delivered more than once. Your workers must be idempotent — executing the same task twice should produce the same result without side effects.
**Patterns for idempotency:**
| Pattern | When to use |
| :--- | :--- |
| **Idempotency key** | Pass a unique key (e.g., `workflowId + taskId`) to downstream services. The service deduplicates on this key. |
| **Upsert instead of insert** | Use `INSERT ... ON CONFLICT UPDATE` or equivalent so repeated writes converge to the same state. |
| **Check-then-act** | Query current state before performing the action. Skip if already completed. |
| **Idempotent HTTP methods** | Prefer PUT over POST when the downstream API supports it. |
```python
from conductor.client.worker.worker_task import worker_task
@worker_task(task_definition_name="charge_payment")
def charge_payment(workflow_id: str, task_id: str, amount: float, currency: str) -> dict:
idempotency_key = f"{workflow_id}-{task_id}"
# Check if this charge was already processed
existing = payment_gateway.get_charge(idempotency_key)
if existing:
return {"chargeId": existing.id, "status": "already_processed"}
charge = payment_gateway.create_charge(
amount=amount, currency=currency, idempotency_key=idempotency_key
)
return {"chargeId": charge.id, "status": "charged"}
```
The `workflowId` and `taskId` combination is unique per task execution attempt, making it an ideal idempotency key.
## Timeout configuration
Every task definition should have explicit timeouts. A task without timeouts can block a workflow indefinitely.
**The rule:** `responseTimeoutSeconds` < `timeoutSeconds`. The response timeout detects unresponsive workers; the overall timeout enforces the SLA.
### Recommended configurations
| Task pattern | `responseTimeoutSeconds` | `timeoutSeconds` | `timeoutPolicy` | `retryCount` |
| :--- | :--- | :--- | :--- | :--- |
| API call (< 5s expected) | 10 | 30 | `RETRY` | 3 |
| ML inference | 120 | 300 | `RETRY` | 1 |
| Human approval | 0 (disabled) | 86400 | `ALERT_ONLY` | 0 |
| Batch processing | 600 | 3600 | `TIME_OUT_WF` | 0 |
| Quick data transform | 5 | 15 | `RETRY` | 3 |
### Timeout policies
| Policy | Behavior | Use when |
| :--- | :--- | :--- |
| `RETRY` | Retries the task up to `retryCount` times. | Transient failures are expected (network calls, external APIs). |
| `TIME_OUT_WF` | Fails the entire workflow immediately. | The task is critical and retrying won't help (e.g., expired batch window). |
| `ALERT_ONLY` | Marks the task as timed out but keeps the workflow running. | Human-in-the-loop tasks or tasks with external completion signals. |
!!! warning
Setting `responseTimeoutSeconds` to 0 disables the response timeout. Only do this for tasks that are completed externally (e.g., [WAIT](../documentation/configuration/workflowdef/systemtasks/wait-task.md) or [Human](../documentation/configuration/workflowdef/systemtasks/human-task.md) tasks).
See [Task Definitions](../documentation/configuration/taskdef.md) for the full parameter reference.
## Payload management
Conductor stores task inputs and outputs in its database. Large payloads degrade performance and increase storage costs.
### Size guidelines
| Payload | Recommended limit | Hard limit (configurable) |
| :--- | :--- | :--- |
| Task input | < 64 KB | 1 MB |
| Task output | < 64 KB | 1 MB |
| Workflow input | < 64 KB | 1 MB |
### External payload storage
For payloads exceeding 64 KB, use external payload storage. Conductor supports S3 out of the box:
```json
{
"conductor.external-payload-storage.type": "s3",
"conductor.external-payload-storage.s3.bucket-name": "my-conductor-payloads",
"conductor.external-payload-storage.s3.region": "us-east-1",
"conductor.external-payload-storage.s3.signed-url-expiration-seconds": 300
}
```
### Do's and don'ts
| Do | Don't |
| :--- | :--- |
| Return only data that downstream tasks need. | Dump entire API responses into task output. |
| Store large files in S3/GCS and pass the URI. | Pass file contents as base64 in payloads. |
| Use `inputTemplate` to set default values on the task definition. | Duplicate static config in every workflow definition. |
| Keep payload keys flat and descriptive. | Nest payloads 5 levels deep with ambiguous keys. |
## Workflow design
### Small, focused tasks over monolithic workers
Break work into small tasks that each do one thing. This gives you:
- **Granular retries** — only the failed step retries, not the entire pipeline.
- **Reusability** — small tasks compose into different workflows.
- **Visibility** — each step is independently observable in the Conductor UI.
### Sub-workflows vs inline tasks
| Approach | When to use |
| :--- | :--- |
| [Sub-workflow](../documentation/configuration/workflowdef/operators/sub-workflow-task.md) | Reusable logic shared across multiple parent workflows. Independently versioned and testable. |
| Inline tasks in a single workflow | Logic specific to one workflow. Fewer indirections to debug. |
Use sub-workflows when a group of tasks represents a **bounded business capability** (e.g., "process payment", "send notification bundle"). Don't create sub-workflows for a single task — the overhead isn't worth it.
### DYNAMIC_FORK vs sequential loops
| Pattern | When to use |
| :--- | :--- |
| [DYNAMIC_FORK](../documentation/configuration/workflowdef/operators/dynamic-fork-task.md) | Process N items in parallel. Use when items are independent and parallelism improves throughput. |
| [DO_WHILE](../documentation/configuration/workflowdef/operators/do-while-task.md) | Process items sequentially when ordering matters or a shared resource requires serialization. |
!!! tip
Keep DYNAMIC_FORK fan-out under 500 concurrent tasks per workflow. Beyond that, consider batching items into chunks and forking over the chunks.
## Worker scaling
Workers are stateless and scale horizontally. Tune these parameters to match your workload.
### Polling interval
The polling interval controls how frequently workers check for new tasks. Shorter intervals reduce latency; longer intervals reduce server load.
| Workload | Recommended polling interval |
| :--- | :--- |
| Low-latency (< 1s SLA) | 100-250 ms |
| Standard processing | 500 ms - 1s |
| Background / batch | 5-10s |
### Thread pool sizing
Each worker instance runs a configurable number of polling threads. Start with:
```
threads = (target_throughput * avg_task_duration_seconds) / num_worker_instances
```
For example, 100 tasks/sec with 2s average execution across 5 instances: `(100 * 2) / 5 = 40 threads` per instance.
### Rate limiting and concurrency
Use task definition settings to protect downstream services:
```json
{
"name": "call_external_api",
"rateLimitPerFrequency": 50,
"rateLimitFrequencyInSeconds": 1,
"concurrentExecLimit": 20
}
```
This limits the task to 50 executions per second globally, with at most 20 running concurrently.
### Domain isolation
Use [task domains](../documentation/api/taskdomains.md) to route tasks to specific worker pools. Common use cases:
- **Environment isolation** — dev workers only pick up dev tasks.
- **Priority lanes** — premium customers routed to dedicated capacity.
- **Regional affinity** — route tasks to workers closest to the data.
See [Scaling Workers](how-tos/Workers/scaling-workers.md) for more detail.
## Error handling patterns
### Retries vs terminal failure
By default, a failed task is retried according to `retryCount` and `retryLogic` (`FIXED`, `EXPONENTIAL_BACKOFF`, or `LINEAR_BACKOFF`). For errors that should **not** be retried, set the task status to `FAILED_WITH_TERMINAL_ERROR`:
```python
from conductor.client.http.models import TaskResult, TaskResultStatus
@worker_task(task_definition_name="validate_order")
def validate_order(order_id: str, items: list) -> TaskResult:
if not items:
result = TaskResult()
result.status = TaskResultStatus.FAILED_WITH_TERMINAL_ERROR
result.reason_for_incompletion = "Order has no items — not retryable"
return result
# ... validation logic
return {"valid": True}
```
| Error type | Strategy |
| :--- | :--- |
| Transient (network timeout, 503) | Let Conductor retry with backoff. |
| Client error (400, validation failure) | Return `FAILED_WITH_TERMINAL_ERROR`. |
| Partial failure in batch | Return partial results as output; use workflow logic to handle remainder. |
### Compensation and saga patterns
For workflows that span multiple services, design compensation tasks to undo completed steps when a later step fails.
**Forward compensation** — Fix the problem and continue. Use a [SWITCH](../documentation/configuration/workflowdef/operators/switch-task.md) after the failed task to route to a recovery path.
**Backward compensation** — Undo completed work in reverse order. Model this as a separate workflow triggered by the [failure workflow](how-tos/Workflows/handling-errors.md) mechanism:
1. The main workflow fails at step 3.
2. Conductor invokes the configured `failureWorkflow`.
3. The failure workflow runs compensating tasks: undo step 2, then undo step 1.
!!! tip
Store compensation metadata (transaction IDs, resource handles) in each task's output so the failure workflow has everything it needs to roll back.
## Versioning and deployments
Conductor supports [workflow versioning](how-tos/Workflows/versioning-workflows.md) natively. Use this for safe deployments.
### Blue-green with versions
1. Deploy workflow version N+1 with your changes.
2. Start new executions on version N+1.
3. Let existing version N executions drain to completion.
4. Once all version N executions are complete, deprecate or remove it.
### Migrating running executions
Running workflows continue on the version they were started with. You cannot migrate a running execution to a new version. Plan for this:
- **Short-lived workflows** — Wait for drain. Most complete within minutes.
- **Long-running workflows** — If a critical fix is needed, terminate and restart on the new version. Use the [Terminate](../documentation/configuration/workflowdef/operators/terminate-task.md) API with a reason, then re-trigger.
### Safe rollback
If version N+1 has issues:
1. Stop starting new executions on N+1 (route traffic back to N).
2. Let N+1 executions fail or terminate them.
3. Resume on version N, which was never modified.
Because workers are decoupled from workflow definitions, you can roll back the workflow version independently of worker deployments.
## Monitoring
Track these metrics to maintain healthy Conductor operations:
| Metric | What it tells you | Alert threshold |
| :--- | :--- | :--- |
| Task queue depth | Backlog of unprocessed tasks. | Growing consistently over 5 minutes. |
| Task poll count (per task type) | Whether workers are actively polling. | Drops to zero. |
| Workflow failure rate | Percentage of workflows ending in FAILED state. | > 5% over a 15-minute window. |
| Task response time (p99) | How close workers are to the response timeout. | > 80% of `responseTimeoutSeconds`. |
| Worker thread utilization | Whether workers are saturated. | > 90% sustained for 10 minutes. |
| External payload storage errors | S3/GCS write failures blocking tasks. | Any non-zero count. |
See [Monitoring and Scaling Workers](how-tos/Workers/scaling-workers.md) for built-in monitoring tools.
docs/devguide/concepts/conductor.md
+117
View File
@@ -0,0 +1,117 @@
---
description: "Why use Conductor? An open source workflow engine for workflow orchestration, microservice orchestration, and AI agent orchestration. Durable execution, polyglot workers, LLM orchestration, workflow automation, and self-hosted deployment — a developer-first alternative to Temporal, Step Functions, and Airflow."
---
# Why Conductor
Conductor is an open source workflow engine built for workflow orchestration at scale. It orchestrates distributed workflows across services, languages, and infrastructure — tracking every state transition, retrying failures automatically, and giving you full visibility into what happened and why. Whether you need microservice orchestration, AI agent orchestration, or workflow automation, Conductor provides a self-hosted, code-first platform with no vendor lock-in.
## The problem
Distributed systems fail. Services crash, networks drop, deployments roll mid-flight. Without a workflow orchestration platform, you end up writing retry logic, state tracking, timeout handling, and compensation flows into every service. That logic is scattered, inconsistent, and invisible.
**Choreography** (peer-to-peer events) makes this worse at scale:
- Business processes are implicit — embedded across dozens of services with no single view of the flow.
- Tight coupling through assumed message contracts makes changes risky.
- "How far along is order #12345?" requires querying every service in the chain.
- Debugging a failure means correlating logs across services, queues, and time.
**Orchestration** centralizes the flow definition while keeping execution distributed. Conductor is the orchestrator — your workers stay stateless and independent.
## What Conductor gives you
### Durable execution
Conductor is a durable execution engine — every workflow execution is persisted. If a task fails, Conductor retries it with configurable backoff including exponential backoff. If a worker crashes, the task is rescheduled. If the server restarts, execution resumes exactly where it left off. Your code doesn't need to handle retry logic — Conductor provides it out of the box. This same durable execution guarantee powers durable agents that survive infrastructure failures.
### Language-agnostic workers
Write workers in Python, Java, Go, JavaScript, C#, or Clojure. Each task in a workflow can use a different language — pick the best tool for each job. Workers communicate with Conductor via REST or gRPC and can run anywhere: containers, VMs, serverless, or your laptop.
### Built-in system tasks
HTTP calls, inline JavaScript execution, JSON transforms, event publishing, wait timers, and human approval gates — all available without writing a single worker. See [System Tasks](../../documentation/configuration/workflowdef/systemtasks/index.md).
### Flow control operators
Fork/join for parallelism, switch for conditional branching, do-while for loops, sub-workflows for composition, and dynamic tasks resolved at runtime. See [Operators](../../documentation/configuration/workflowdef/operators/index.md).
### AI agent orchestration and LLM orchestration
Conductor provides LLM orchestration and AI agent orchestration as native system tasks — no external frameworks required. Supported providers include Anthropic (Claude), OpenAI (GPT), Azure OpenAI, Google Gemini, AWS Bedrock, Mistral, Cohere, HuggingFace, Ollama, Perplexity, Grok, and StabilityAI — 14+ providers available out of the box for chat completion, text completion, and embedding generation.
MCP (Model Context Protocol) integration is built in: use `LIST_MCP_TOOLS` to discover available tools and `CALL_MCP_TOOL` to invoke them — enabling function calling and tool use within workflows with full retry and state tracking.
For RAG pipelines, Conductor supports three vector databases natively — Pinecone, pgvector, and MongoDB Atlas — so you can index embeddings, run similarity search, and feed results to an LLM in a single workflow definition.
Content generation tasks cover image, audio, video, and PDF creation using AI models. Every AI task runs with the same durability guarantees as any other Conductor task: automatic retries, timeout handling, and a complete audit trail.
### Event-driven workflows
Publish to and consume from Kafka, NATS, AMQP (RabbitMQ), and SQS. Trigger workflows from external events or emit events from within workflows. See [Event Bus Orchestration](../how-tos/event-bus.md).
### Full operational control
Pause, resume, restart, retry, and terminate any workflow execution. Search and filter executions by status, time, correlation ID, or custom tags. Every task has a complete audit trail — inputs, outputs, timestamps, retry history, and worker identity.
### Horizontal scaling
Conductor scales horizontally to millions of concurrent workflow executions. Workers scale independently — add more instances and Conductor distributes tasks automatically. Rate limits and concurrency caps prevent overload. This workflow engine scalability makes Conductor suitable for production deployments at any scale.
## When to use Conductor
| Use case | Example |
| :--- | :--- |
| **Microservice orchestration** | Order processing: payment → inventory → shipping → notification |
| **Workflow automation** | Automate business processes with durable execution, retries, and full observability |
| **Durable agents** | Multi-step LLM chains with function calling, tool use, RAG, and human-in-the-loop — durable agents that survive crashes |
| **Long-running workflows** | Insurance claims, loan approvals, onboarding flows spanning days or weeks — async workflows that survive deploys |
| **Event-driven automation** | React to Kafka events, trigger workflows, publish results back |
| **Batch processing** | Fan-out work across thousands of parallel workers with dynamic fork |
| **Saga pattern** | Distributed transactions with compensation on failure |
| **RAG applications** | Build retrieval-augmented generation pipelines with vector search, embedding generation, and LLM completion as workflow tasks |
| **Content generation pipelines** | Generate images, audio, video, and PDFs using AI models orchestrated as durable workflows |
## What sets Conductor apart
No other open source workflow engine matches this combination:
- **14+ native LLM providers as system tasks** — Anthropic, OpenAI, Azure OpenAI, Gemini, Bedrock, Mistral, Cohere, HuggingFace, Ollama, Perplexity, Grok, StabilityAI, and more. No wrappers, no plugins — first-class support.
- **MCP (Model Context Protocol) native integration** — discover and call tools directly from workflow definitions.
- **3 vector databases for built-in RAG** — Pinecone, pgvector, MongoDB Atlas. Embed, index, search, and generate in one workflow.
- **Content generation tasks** — image, audio, video, and PDF generation as system tasks.
- **6 message brokers** — Kafka, NATS, NATS Streaming, SQS, AMQP (RabbitMQ), and internal queuing.
- **5 persistence backends** — Redis, PostgreSQL, MySQL, Cassandra, and SQLite.
- **7+ language SDKs** — Java, Python, Go, JavaScript, C#, Clojure, Ruby, and Rust.
- **Battle-tested at scale** — proven in production at Netflix, Tesla, LinkedIn, and JP Morgan.
- **JSON-native and code-first workflow definitions** — define workflows as JSON or as code using SDKs. Workflow as code for developers who want type safety; JSON for runtime generation and LLM-driven workflows.
- **Self-hosted with no vendor lock-in** — deploy Conductor on your own infrastructure. Apache 2.0 licensed, fully open source.
- **Human-in-the-loop as a first-class task type** — pause execution for approvals, reviews, or manual intervention with built-in timeout and escalation.
## How it works
```mermaid
graph TD
subgraph Workers
A["Worker A<br/>(Python)"]
B["Worker B<br/>(Java)"]
C["Worker C<br/>(Go)"]
D["Worker D<br/>(C#)"]
end
subgraph Server["Conductor Server"]
S["Scheduling · State · Retries<br/>Persistence · Queuing"]
end
subgraph Storage["Persistence"]
DB["Redis / PostgreSQL / MySQL / Cassandra"]
end
A -- "poll / complete" --> S
B -- "poll / complete" --> S
C -- "poll / complete" --> S
D -- "poll / complete" --> S
S --> DB
```
Workers poll for tasks, execute business logic, and report results. Conductor handles everything else — scheduling, retries, timeouts, state persistence, and flow control. See [Architecture](../architecture/index.md) for details.
## Next steps
- [Quickstart](../../quickstart/index.md) — run your first workflow in 2 minutes
- [Workflows](workflows.md) — how workflow definitions work
- [Tasks](tasks.md) — task types and configuration
- [Workers](workers.md) — building workers in any language
docs/devguide/concepts/index.md
+363
View File
@@ -0,0 +1,363 @@
---
description: "Core concepts of Conductor — an open source workflow orchestration engine for distributed workflows, microservice orchestration, AI agent orchestration, and workflow automation with code-first and JSON-native definitions and polyglot workers."
---
# Basic Concepts
Conductor is an open source workflow orchestration engine that orchestrates distributed workflows. You define
workflows as code or as JSON, write workers in any language, and let Conductor handle state persistence,
retries, timeouts, and flow control. Every step is durably recorded, so processes survive crashes,
restarts, and network partitions without losing progress.
Workflow definitions are JSON-native — you can version them in source control, diff changes across
releases, generate them programmatically, or let LLMs create and modify them at runtime. Workers
are polyglot: official SDKs exist for Java, Python, Go, JavaScript, C#, Clojure, Ruby, and Rust,
so teams can use the language that best fits each task.
Built-in system tasks handle common operations like HTTP calls, event publishing, inline transforms,
and sub-workflow orchestration without writing custom code. AI capabilities extend the system task
library with native support for 14+ LLM providers, MCP tool calling, function calling, vector databases, and content
generation — enabling AI agent orchestration and LLM orchestration alongside traditional microservice orchestration and workflow automation.
## What can Conductor do?
<div class="wcc-widget" role="tablist">
<div class="wcc-left">
<div class="wcc-item wcc-active" data-wcc="0" role="tab" aria-selected="true" tabindex="0">
<div class="wcc-header"><span class="wcc-title">Create Workflows</span><span class="wcc-chevron"></span></div>
<div class="wcc-body">Define workflows consisting of multiple tasks that are executed in a specific order. <a href="../../documentation/configuration/workflowdef/index.html">Learn more</a></div>
</div>
<div class="wcc-item" data-wcc="1" role="tab" aria-selected="false" tabindex="0">
<div class="wcc-header"><span class="wcc-title">Branch Your Flows</span><span class="wcc-chevron"></span></div>
<div class="wcc-body">Use switch-case operators to make branching decisions. <a href="../../documentation/configuration/workflowdef/operators/switch-task.html">Learn more</a></div>
</div>
<div class="wcc-item" data-wcc="2" role="tab" aria-selected="false" tabindex="0">
<div class="wcc-header"><span class="wcc-title">Run Loops</span><span class="wcc-chevron"></span></div>
<div class="wcc-body">Use the Do-While loop operator to iterate through a set of tasks. <a href="../../documentation/configuration/workflowdef/operators/do-while-task.html">Learn more</a></div>
</div>
<div class="wcc-item" data-wcc="3" role="tab" aria-selected="false" tabindex="0">
<div class="wcc-header"><span class="wcc-title">Parallelize Your Tasks</span><span class="wcc-chevron"></span></div>
<div class="wcc-body">Execute tasks in parallel using either static or dynamic forks. <a href="../../documentation/configuration/workflowdef/operators/fork-task.html">Learn more</a></div>
</div>
<div class="wcc-item" data-wcc="4" role="tab" aria-selected="false" tabindex="0">
<div class="wcc-header"><span class="wcc-title">Run Your Tasks Externally</span><span class="wcc-chevron"></span></div>
<div class="wcc-body">Implement tasks using external workers in microservices, serverless functions, or applications. <a href="workers.html">Workers</a> · <a href="../../documentation/clientsdks/index.html">SDKs</a></div>
</div>
<div class="wcc-item" data-wcc="5" role="tab" aria-selected="false" tabindex="0">
<div class="wcc-header"><span class="wcc-title">Use Built-In Tasks</span><span class="wcc-chevron"></span></div>
<div class="wcc-body">Use built-in tasks for common actions such as calling HTTP endpoints, writing to event queues, and executing inline code. <a href="../../documentation/configuration/workflowdef/systemtasks/index.html">Learn more</a></div>
</div>
<div class="wcc-item" data-wcc="6" role="tab" aria-selected="false" tabindex="0">
<div class="wcc-header"><span class="wcc-title">Use LLM Tasks</span><span class="wcc-chevron"></span></div>
<div class="wcc-body">Use LLM tasks to build AI-powered workflows, including agentic workflows. <a href="../ai/index.html">Learn more</a></div>
</div>
<div class="wcc-item" data-wcc="7" role="tab" aria-selected="false" tabindex="0">
<div class="wcc-header"><span class="wcc-title">Human in the Loop</span><span class="wcc-chevron"></span></div>
<div class="wcc-body">Plug in manual steps in your workflows using Human tasks. <a href="../../documentation/configuration/workflowdef/systemtasks/human-task.html">Human tasks</a> · <a href="../../documentation/configuration/workflowdef/systemtasks/wait-task.html">Wait tasks</a></div>
</div>
<div class="wcc-item" data-wcc="8" role="tab" aria-selected="false" tabindex="0">
<div class="wcc-header"><span class="wcc-title">Handle Failures</span><span class="wcc-chevron"></span></div>
<div class="wcc-body">Set timeouts and rate limits to manage failures for tasks and workflows. <a href="../how-tos/Workflows/handling-errors.html">Learn more</a></div>
</div>
<div class="wcc-item" data-wcc="9" role="tab" aria-selected="false" tabindex="0">
<div class="wcc-header"><span class="wcc-title">Replay Any Workflow</span><span class="wcc-chevron"></span></div>
<div class="wcc-body">Replay completed or failed workflows from the beginning, from any task, or retry just the failed step — even months later. Full execution history is always preserved. <a href="../how-tos/Workflows/debugging-workflows.html">Learn more</a></div>
</div>
<div class="wcc-item" data-wcc="10" role="tab" aria-selected="false" tabindex="0">
<div class="wcc-header"><span class="wcc-title">Integrate With Applications</span><span class="wcc-chevron"></span></div>
<div class="wcc-body">Connect Conductor to your ecosystem with event-driven triggers using Kafka, NATS, SQS, AMQP, and webhooks. <a href="../cookbook/event-driven.html">Learn more</a></div>
</div>
<div class="wcc-item" data-wcc="11" role="tab" aria-selected="false" tabindex="0">
<div class="wcc-header"><span class="wcc-title">Debug Visually</span><span class="wcc-chevron"></span></div>
<div class="wcc-body">Track and debug workflows from Conductor UI. View inputs, pull logs, and restart from any point. <a href="../../quickstart/index.html">Get started</a></div>
</div>
<div class="wcc-item" data-wcc="12" role="tab" aria-selected="false" tabindex="0">
<div class="wcc-header"><span class="wcc-title">Scale Horizontally</span><span class="wcc-chevron"></span></div>
<div class="wcc-body">Run multiple server instances behind a load balancer with shared backends for high availability. <a href="../running/deploy.html">Deployment guide</a></div>
</div>
</div>
<div class="wcc-right" role="tabpanel">
<!-- 0: Create Workflows — linear -->
<svg class="wcc-diagram wcc-visible" data-wcc-diagram="0" viewBox="0 0 220 420" xmlns="http://www.w3.org/2000/svg">
<circle cx="110" cy="30" r="22" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="110" y="35" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">Start</text>
<line x1="110" y1="52" x2="110" y2="80" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="35" y="80" width="150" height="40" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="110" y="105" text-anchor="middle" font-size="12" fill="#2e3545" font-family="sans-serif">Task A</text>
<line x1="110" y1="120" x2="110" y2="155" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="35" y="155" width="150" height="40" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="110" y="180" text-anchor="middle" font-size="12" fill="#2e3545" font-family="sans-serif">Task B</text>
<line x1="110" y1="195" x2="110" y2="230" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="35" y="230" width="150" height="40" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="110" y="255" text-anchor="middle" font-size="12" fill="#2e3545" font-family="sans-serif">Task C</text>
<line x1="110" y1="270" x2="110" y2="305" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<circle cx="110" cy="327" r="22" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="110" y="332" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">End</text>
<defs><marker id="wcc-arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z" fill="#a0aec0"/></marker></defs>
</svg>
<!-- 1: Branch — switch -->
<svg class="wcc-diagram" data-wcc-diagram="1" viewBox="0 0 260 420" xmlns="http://www.w3.org/2000/svg">
<circle cx="130" cy="30" r="22" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="130" y="35" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">Start</text>
<line x1="130" y1="52" x2="130" y2="80" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="55" y="80" width="150" height="40" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="130" y="105" text-anchor="middle" font-size="12" fill="#2e3545" font-family="sans-serif">Task A</text>
<line x1="130" y1="120" x2="130" y2="150" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<polygon points="130,150 170,185 130,220 90,185" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="130" y="188" text-anchor="middle" font-size="10" fill="#2e3545" font-family="sans-serif">Switch</text><text x="130" y="200" text-anchor="middle" font-size="10" fill="#2e3545" font-family="sans-serif">Case</text>
<line x1="90" y1="185" x2="50" y2="185" stroke="#a0aec0" stroke-width="1.5"/><line x1="50" y1="185" x2="50" y2="260" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<line x1="170" y1="185" x2="210" y2="185" stroke="#a0aec0" stroke-width="1.5"/><line x1="210" y1="185" x2="210" y2="260" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="0" y="260" width="100" height="40" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="50" y="285" text-anchor="middle" font-size="12" fill="#2e3545" font-family="sans-serif">Task B</text>
<rect x="160" y="260" width="100" height="40" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="210" y="285" text-anchor="middle" font-size="12" fill="#2e3545" font-family="sans-serif">Task C</text>
<line x1="50" y1="300" x2="50" y2="340" stroke="#a0aec0" stroke-width="1.5"/><line x1="50" y1="340" x2="130" y2="340" stroke="#a0aec0" stroke-width="1.5"/>
<line x1="210" y1="300" x2="210" y2="340" stroke="#a0aec0" stroke-width="1.5"/><line x1="210" y1="340" x2="130" y2="340" stroke="#a0aec0" stroke-width="1.5"/>
<line x1="130" y1="340" x2="130" y2="360" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<circle cx="130" cy="382" r="22" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="130" y="387" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">End</text>
</svg>
<!-- 2: Run Loops — do-while highlighted -->
<svg class="wcc-diagram" data-wcc-diagram="2" viewBox="0 0 280 440" xmlns="http://www.w3.org/2000/svg">
<circle cx="110" cy="30" r="22" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="110" y="35" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">Start</text>
<line x1="110" y1="52" x2="110" y2="80" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="35" y="80" width="150" height="40" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="110" y="105" text-anchor="middle" font-size="12" fill="#2e3545" font-family="sans-serif">Task A</text>
<line x1="110" y1="120" x2="110" y2="150" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<polygon points="110,150 150,185 110,220 70,185" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="110" y="188" text-anchor="middle" font-size="10" fill="#2e3545" font-family="sans-serif">Switch</text><text x="110" y="200" text-anchor="middle" font-size="10" fill="#2e3545" font-family="sans-serif">Case</text>
<line x1="70" y1="185" x2="30" y2="185" stroke="#a0aec0" stroke-width="1.5"/><line x1="30" y1="185" x2="30" y2="270" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="-20" y="270" width="100" height="40" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="30" y="295" text-anchor="middle" font-size="12" fill="#2e3545" font-family="sans-serif">Task B</text>
<line x1="150" y1="185" x2="210" y2="185" stroke="#a0aec0" stroke-width="1.5"/><line x1="210" y1="185" x2="210" y2="250" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="155" y="250" width="120" height="75" rx="8" fill="rgba(6,214,160,0.10)" stroke="#06d6a0" stroke-width="2"/><text x="215" y="272" text-anchor="middle" font-size="10" font-weight="600" fill="#06d6a0" font-family="sans-serif">Do While Loop</text>
<rect x="170" y="282" width="90" height="32" rx="5" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="215" y="303" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">Task C</text>
<path d="M 260 298 C 280 298 280 268 260 268" stroke="#06d6a0" stroke-width="1.5" fill="none" marker-end="url(#wcc-arrow-teal)"/>
<line x1="30" y1="310" x2="30" y2="370" stroke="#a0aec0" stroke-width="1.5"/><line x1="30" y1="370" x2="110" y2="370" stroke="#a0aec0" stroke-width="1.5"/>
<line x1="215" y1="325" x2="215" y2="370" stroke="#a0aec0" stroke-width="1.5"/><line x1="215" y1="370" x2="110" y2="370" stroke="#a0aec0" stroke-width="1.5"/>
<line x1="110" y1="370" x2="110" y2="390" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<circle cx="110" cy="412" r="22" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="110" y="417" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">End</text>
<defs><marker id="wcc-arrow-teal" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z" fill="#06d6a0"/></marker></defs>
</svg>
<!-- 3: Parallelize — fork/join -->
<svg class="wcc-diagram" data-wcc-diagram="3" viewBox="0 0 320 440" xmlns="http://www.w3.org/2000/svg">
<circle cx="140" cy="30" r="22" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="140" y="35" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">Start</text>
<line x1="140" y1="52" x2="140" y2="80" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="65" y="80" width="150" height="40" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="140" y="105" text-anchor="middle" font-size="12" fill="#2e3545" font-family="sans-serif">Task A</text>
<line x1="140" y1="120" x2="140" y2="148" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<polygon points="140,148 180,178 140,208 100,178" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="140" y="181" text-anchor="middle" font-size="10" fill="#2e3545" font-family="sans-serif">Switch</text><text x="140" y="193" text-anchor="middle" font-size="10" fill="#2e3545" font-family="sans-serif">Case</text>
<line x1="100" y1="178" x2="40" y2="178" stroke="#a0aec0" stroke-width="1.5"/><line x1="40" y1="178" x2="40" y2="240" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<line x1="140" y1="208" x2="140" y2="240" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<line x1="180" y1="178" x2="255" y2="178" stroke="#a0aec0" stroke-width="1.5"/><line x1="255" y1="178" x2="255" y2="230" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="-10" y="240" width="100" height="40" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="40" y="265" text-anchor="middle" font-size="12" fill="#2e3545" font-family="sans-serif">Task B</text>
<rect x="90" y="240" width="100" height="40" rx="6" fill="#dc2626" stroke="#dc2626" stroke-width="1.5"/><text x="140" y="265" text-anchor="middle" font-size="12" fill="#fff" font-family="sans-serif" font-weight="600">Task D</text>
<rect x="200" y="230" width="120" height="70" rx="8" fill="rgba(6,214,160,0.10)" stroke="#06d6a0" stroke-width="2"/><text x="260" y="250" text-anchor="middle" font-size="10" font-weight="600" fill="#06d6a0" font-family="sans-serif">Do While Loop</text>
<rect x="215" y="258" width="90" height="32" rx="5" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="260" y="279" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">Task C</text>
<line x1="40" y1="280" x2="40" y2="340" stroke="#a0aec0" stroke-width="1.5"/><line x1="40" y1="340" x2="140" y2="340" stroke="#a0aec0" stroke-width="1.5"/>
<line x1="140" y1="280" x2="140" y2="340" stroke="#a0aec0" stroke-width="1.5"/>
<line x1="260" y1="300" x2="260" y2="340" stroke="#a0aec0" stroke-width="1.5"/><line x1="260" y1="340" x2="140" y2="340" stroke="#a0aec0" stroke-width="1.5"/>
<line x1="140" y1="340" x2="140" y2="370" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<circle cx="140" cy="392" r="22" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="140" y="397" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">End</text>
</svg>
<!-- 4: External Workers -->
<svg class="wcc-diagram" data-wcc-diagram="4" viewBox="0 0 320 440" xmlns="http://www.w3.org/2000/svg">
<circle cx="90" cy="30" r="22" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="90" y="35" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">Start</text>
<line x1="90" y1="52" x2="90" y2="80" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="30" y="80" width="120" height="40" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="90" y="105" text-anchor="middle" font-size="12" fill="#2e3545" font-family="sans-serif">Task A</text>
<line x1="150" y1="100" x2="200" y2="100" stroke="#a0aec0" stroke-width="1.5" stroke-dasharray="5,3" marker-end="url(#wcc-arrow)"/>
<rect x="200" y="80" width="110" height="40" rx="6" fill="#06d6a0" stroke="#05c792" stroke-width="1.5"/><text x="255" y="98" text-anchor="middle" font-size="10" fill="#fff" font-weight="600" font-family="sans-serif">Worker A</text><text x="255" y="112" text-anchor="middle" font-size="9" fill="#fff" font-family="sans-serif">Microservice</text>
<line x1="90" y1="120" x2="90" y2="160" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="30" y="160" width="120" height="40" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="90" y="185" text-anchor="middle" font-size="12" fill="#2e3545" font-family="sans-serif">Task B</text>
<line x1="150" y1="180" x2="200" y2="180" stroke="#a0aec0" stroke-width="1.5" stroke-dasharray="5,3" marker-end="url(#wcc-arrow)"/>
<rect x="200" y="160" width="110" height="40" rx="6" fill="#f59e0b" stroke="#d97706" stroke-width="1.5"/><text x="255" y="178" text-anchor="middle" font-size="10" fill="#fff" font-weight="600" font-family="sans-serif">Worker B</text><text x="255" y="192" text-anchor="middle" font-size="9" fill="#fff" font-family="sans-serif">Serverless</text>
<line x1="90" y1="200" x2="90" y2="240" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="30" y="240" width="120" height="40" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="90" y="265" text-anchor="middle" font-size="12" fill="#2e3545" font-family="sans-serif">Task C</text>
<line x1="150" y1="260" x2="200" y2="260" stroke="#a0aec0" stroke-width="1.5" stroke-dasharray="5,3" marker-end="url(#wcc-arrow)"/>
<rect x="200" y="240" width="110" height="40" rx="6" fill="#3b82f6" stroke="#2563eb" stroke-width="1.5"/><text x="255" y="258" text-anchor="middle" font-size="10" fill="#fff" font-weight="600" font-family="sans-serif">Worker C</text><text x="255" y="272" text-anchor="middle" font-size="9" fill="#fff" font-family="sans-serif">Legacy App</text>
<line x1="90" y1="280" x2="90" y2="320" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<circle cx="90" cy="342" r="22" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="90" y="347" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">End</text>
</svg>
<!-- 5: Built-in Tasks -->
<svg class="wcc-diagram" data-wcc-diagram="5" viewBox="0 0 260 420" xmlns="http://www.w3.org/2000/svg">
<circle cx="130" cy="30" r="22" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="130" y="35" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">Start</text>
<line x1="130" y1="52" x2="130" y2="80" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="30" y="80" width="200" height="40" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="130" y="105" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">HTTP: Call API endpoint</text>
<line x1="130" y1="120" x2="130" y2="160" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="30" y="160" width="200" height="40" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="130" y="185" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">Event: Write to Kafka</text>
<line x1="130" y1="200" x2="130" y2="240" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="30" y="240" width="200" height="40" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="130" y="265" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">Inline: Execute JS</text>
<line x1="130" y1="280" x2="130" y2="320" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<circle cx="130" cy="342" r="22" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="130" y="347" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">End</text>
</svg>
<!-- 6: LLM Tasks -->
<svg class="wcc-diagram" data-wcc-diagram="6" viewBox="0 0 240 340" xmlns="http://www.w3.org/2000/svg">
<circle cx="120" cy="30" r="22" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="120" y="35" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">Start</text>
<line x1="120" y1="52" x2="120" y2="80" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="20" y="80" width="200" height="40" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="120" y="105" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">Search News Index</text>
<line x1="120" y1="120" x2="120" y2="160" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="20" y="160" width="200" height="40" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="120" y="185" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">Get Contextual Answer</text>
<line x1="120" y1="200" x2="120" y2="240" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<circle cx="120" cy="262" r="22" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="120" y="267" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">End</text>
</svg>
<!-- 7: Human in the Loop -->
<svg class="wcc-diagram" data-wcc-diagram="7" viewBox="0 0 280 380" xmlns="http://www.w3.org/2000/svg">
<circle cx="140" cy="30" r="22" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="140" y="35" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">Start</text>
<line x1="140" y1="52" x2="140" y2="80" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<polygon points="140,80 180,115 140,150 100,115" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="140" y="118" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">Switch</text>
<line x1="100" y1="115" x2="50" y2="115" stroke="#a0aec0" stroke-width="1.5"/><line x1="50" y1="115" x2="50" y2="190" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<line x1="180" y1="115" x2="230" y2="115" stroke="#a0aec0" stroke-width="1.5"/><line x1="230" y1="115" x2="230" y2="190" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="0" y="190" width="100" height="45" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="50" y="210" text-anchor="middle" font-size="10" fill="#2e3545" font-family="sans-serif">Default</text><text x="50" y="224" text-anchor="middle" font-size="10" fill="#2e3545" font-family="sans-serif">Approval</text>
<rect x="180" y="190" width="100" height="45" rx="6" fill="#f59e0b" stroke="#d97706" stroke-width="1.5"/><text x="230" y="210" text-anchor="middle" font-size="10" fill="#fff" font-weight="600" font-family="sans-serif">Human</text><text x="230" y="224" text-anchor="middle" font-size="10" fill="#fff" font-weight="600" font-family="sans-serif">Approval</text>
<line x1="50" y1="235" x2="50" y2="280" stroke="#a0aec0" stroke-width="1.5"/><line x1="50" y1="280" x2="140" y2="280" stroke="#a0aec0" stroke-width="1.5"/>
<line x1="230" y1="235" x2="230" y2="280" stroke="#a0aec0" stroke-width="1.5"/><line x1="230" y1="280" x2="140" y2="280" stroke="#a0aec0" stroke-width="1.5"/>
<line x1="140" y1="280" x2="140" y2="310" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<circle cx="140" cy="332" r="22" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="140" y="337" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">End</text>
</svg>
<!-- 8: Handle Failures -->
<svg class="wcc-diagram" data-wcc-diagram="8" viewBox="0 0 300 420" xmlns="http://www.w3.org/2000/svg">
<circle cx="110" cy="30" r="22" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="110" y="35" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">Start</text>
<line x1="110" y1="52" x2="110" y2="80" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="35" y="80" width="150" height="40" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="110" y="105" text-anchor="middle" font-size="12" fill="#2e3545" font-family="sans-serif">Task A</text>
<circle cx="200" cy="90" r="12" fill="#f59e0b" stroke="#d97706" stroke-width="1.5"/><text x="200" y="94" text-anchor="middle" font-size="10" fill="#fff" font-weight="bold" font-family="sans-serif">!</text>
<text x="220" y="94" font-size="9" fill="#4a5568" font-family="sans-serif">Retry on failure</text>
<line x1="110" y1="120" x2="110" y2="160" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="35" y="160" width="150" height="40" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="110" y="185" text-anchor="middle" font-size="12" fill="#2e3545" font-family="sans-serif">Task B</text>
<text x="200" y="175" font-size="14" fill="#4a5568" font-family="sans-serif">&#9201;</text>
<text x="220" y="178" font-size="9" fill="#4a5568" font-family="sans-serif">Timeout after</text><text x="220" y="190" font-size="9" fill="#4a5568" font-family="sans-serif">x seconds</text>
<line x1="110" y1="200" x2="110" y2="240" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="35" y="240" width="150" height="40" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="110" y="265" text-anchor="middle" font-size="12" fill="#2e3545" font-family="sans-serif">Task C</text>
<line x1="185" y1="260" x2="220" y2="260" stroke="#dc2626" stroke-width="1.5" stroke-dasharray="5,3" marker-end="url(#wcc-arrow-red)"/>
<text x="228" y="258" font-size="9" fill="#dc2626" font-family="sans-serif">On failure</text>
<line x1="110" y1="280" x2="110" y2="320" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<circle cx="110" cy="342" r="22" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="110" y="347" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">End</text>
<defs><marker id="wcc-arrow-red" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z" fill="#dc2626"/></marker></defs>
</svg>
<!-- 9: Replay Any Workflow -->
<svg class="wcc-diagram" data-wcc-diagram="9" viewBox="0 0 340 420" xmlns="http://www.w3.org/2000/svg">
<circle cx="110" cy="30" r="22" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="110" y="35" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">Start</text>
<line x1="110" y1="52" x2="110" y2="80" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="35" y="80" width="150" height="40" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="110" y="105" text-anchor="middle" font-size="12" fill="#2e3545" font-family="sans-serif">Task A</text>
<line x1="110" y1="120" x2="110" y2="160" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="35" y="160" width="150" height="40" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="110" y="185" text-anchor="middle" font-size="12" fill="#2e3545" font-family="sans-serif">Task B</text>
<line x1="110" y1="200" x2="110" y2="240" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="35" y="240" width="150" height="40" rx="6" fill="#dc2626" stroke="#dc2626" stroke-width="1.5"/><text x="110" y="265" text-anchor="middle" font-size="12" fill="#fff" font-weight="600" font-family="sans-serif">Task C</text>
<text x="110" y="300" text-anchor="middle" font-size="10" fill="#dc2626" font-family="sans-serif" font-weight="600">FAILED</text>
<!-- Restart arrow -->
<path d="M 35 260 C -20 260 -20 30 60 30" stroke="#06d6a0" stroke-width="2" fill="none" stroke-dasharray="5,3" marker-end="url(#wcc-arrow-teal)"/>
<text x="-8" y="150" text-anchor="middle" font-size="9" fill="#06d6a0" font-family="sans-serif" font-weight="600" transform="rotate(-90 -8 150)">Restart</text>
<!-- Rerun arrow -->
<path d="M 185 260 C 240 260 240 180 185 180" stroke="#3b82f6" stroke-width="2" fill="none" stroke-dasharray="5,3" marker-end="url(#wcc-arrow-blue)"/>
<text x="248" y="220" text-anchor="start" font-size="9" fill="#3b82f6" font-family="sans-serif" font-weight="600">Rerun</text>
<!-- Retry arrow -->
<path d="M 185 250 C 300 250 300 240 185 240" stroke="#f59e0b" stroke-width="2" fill="none" stroke-dasharray="5,3" marker-end="url(#wcc-arrow-amber)"/>
<text x="290" y="258" text-anchor="start" font-size="9" fill="#f59e0b" font-family="sans-serif" font-weight="600">Retry</text>
<defs>
<marker id="wcc-arrow-blue" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z" fill="#3b82f6"/></marker>
<marker id="wcc-arrow-amber" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path d="M 0 0 L 10 5 L 0 10 z" fill="#f59e0b"/></marker>
</defs>
</svg>
<!-- 10: Integrate -->
<svg class="wcc-diagram" data-wcc-diagram="10" viewBox="0 0 300 350" xmlns="http://www.w3.org/2000/svg">
<rect x="75" y="20" width="150" height="50" rx="8" fill="#06d6a0" stroke="#05c792" stroke-width="1.5"/><text x="150" y="42" text-anchor="middle" font-size="11" fill="#fff" font-weight="600" font-family="sans-serif">Conductor</text><text x="150" y="58" text-anchor="middle" font-size="10" fill="#fff" font-family="sans-serif">Workflow Engine</text>
<line x1="75" y1="45" x2="20" y2="115" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<line x1="120" y1="70" x2="90" y2="115" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<line x1="180" y1="70" x2="210" y2="115" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<line x1="225" y1="45" x2="280" y2="115" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="-10" y="115" width="70" height="36" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="25" y="138" text-anchor="middle" font-size="10" fill="#2e3545" font-family="sans-serif">Kafka</text>
<rect x="70" y="115" width="70" height="36" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="105" y="138" text-anchor="middle" font-size="10" fill="#2e3545" font-family="sans-serif">NATS</text>
<rect x="170" y="115" width="70" height="36" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="205" y="138" text-anchor="middle" font-size="10" fill="#2e3545" font-family="sans-serif">SQS</text>
<rect x="250" y="115" width="70" height="36" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="285" y="138" text-anchor="middle" font-size="10" fill="#2e3545" font-family="sans-serif">AMQP</text>
<line x1="150" y1="70" x2="150" y2="200" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="80" y="200" width="140" height="36" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="150" y="223" text-anchor="middle" font-size="10" fill="#2e3545" font-family="sans-serif">Webhooks</text>
</svg>
<!-- 11: Debug Visually -->
<svg class="wcc-diagram" data-wcc-diagram="11" viewBox="0 0 300 420" xmlns="http://www.w3.org/2000/svg">
<circle cx="110" cy="30" r="22" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="110" y="35" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">Start</text>
<line x1="110" y1="52" x2="110" y2="80" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="35" y="80" width="150" height="40" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="110" y="105" text-anchor="middle" font-size="12" fill="#2e3545" font-family="sans-serif">Task A</text>
<line x1="110" y1="120" x2="110" y2="160" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<polygon points="110,160 150,195 110,230 70,195" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="110" y="198" text-anchor="middle" font-size="10" fill="#2e3545" font-family="sans-serif">Switch</text>
<line x1="110" y1="230" x2="110" y2="260" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="35" y="260" width="150" height="40" rx="6" fill="#dc2626" stroke="#dc2626" stroke-width="1.5"/><text x="110" y="285" text-anchor="middle" font-size="12" fill="#fff" font-weight="600" font-family="sans-serif">Task D</text>
<rect x="180" y="165" width="110" height="60" rx="8" fill="#fff" stroke="#a0aec0" stroke-width="1" stroke-dasharray="4,3"/>
<text x="235" y="183" text-anchor="middle" font-size="9" fill="#4a5568" font-family="sans-serif">View inputs,</text>
<text x="235" y="195" text-anchor="middle" font-size="9" fill="#4a5568" font-family="sans-serif">pull logs,</text>
<text x="235" y="207" text-anchor="middle" font-size="9" fill="#4a5568" font-family="sans-serif">restart from</text>
<text x="235" y="219" text-anchor="middle" font-size="9" fill="#4a5568" font-family="sans-serif">here</text>
<line x1="180" y1="195" x2="155" y2="195" stroke="#a0aec0" stroke-width="1" stroke-dasharray="3,3"/>
<line x1="110" y1="300" x2="110" y2="340" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<circle cx="110" cy="362" r="22" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="110" y="367" text-anchor="middle" font-size="11" fill="#2e3545" font-family="sans-serif">End</text>
</svg>
<!-- 12: Scale -->
<svg class="wcc-diagram" data-wcc-diagram="12" viewBox="0 0 320 300" xmlns="http://www.w3.org/2000/svg">
<rect x="90" y="10" width="140" height="40" rx="8" fill="#06d6a0" stroke="#05c792" stroke-width="1.5"/><text x="160" y="35" text-anchor="middle" font-size="11" fill="#fff" font-weight="600" font-family="sans-serif">Load Balancer</text>
<line x1="120" y1="50" x2="60" y2="80" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<line x1="200" y1="50" x2="260" y2="80" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<rect x="10" y="80" width="100" height="55" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="60" y="102" text-anchor="middle" font-size="10" fill="#2e3545" font-weight="600" font-family="sans-serif">Instance 1</text><text x="60" y="116" text-anchor="middle" font-size="9" fill="#4a5568" font-family="sans-serif">API Server</text><text x="60" y="128" text-anchor="middle" font-size="9" fill="#4a5568" font-family="sans-serif">+ Sweeper</text>
<rect x="210" y="80" width="100" height="55" rx="6" fill="#e2e8f0" stroke="#4a5568" stroke-width="1.5"/><text x="260" y="102" text-anchor="middle" font-size="10" fill="#2e3545" font-weight="600" font-family="sans-serif">Instance 2</text><text x="260" y="116" text-anchor="middle" font-size="9" fill="#4a5568" font-family="sans-serif">API Server</text><text x="260" y="128" text-anchor="middle" font-size="9" fill="#4a5568" font-family="sans-serif">+ Sweeper</text>
<line x1="60" y1="135" x2="60" y2="175" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<line x1="260" y1="135" x2="260" y2="175" stroke="#a0aec0" stroke-width="1.5" marker-end="url(#wcc-arrow)"/>
<line x1="160" y1="195" x2="160" y2="195" stroke="#a0aec0" stroke-width="1.5"/>
<rect x="30" y="175" width="260" height="55" rx="8" fill="rgba(6,214,160,0.08)" stroke="#06d6a0" stroke-width="1.5"/>
<text x="160" y="195" text-anchor="middle" font-size="10" font-weight="600" fill="#06d6a0" font-family="sans-serif">Shared Backends</text>
<text x="160" y="215" text-anchor="middle" font-size="9" fill="#4a5568" font-family="sans-serif">Database · Queue · Index · Lock</text>
</svg>
</div>
</div>
<style>
.wcc-widget{display:flex;gap:0;border:1px solid var(--c-cloud,#e2e8f0);border-radius:var(--r-md,10px);overflow:hidden;margin:1.5rem 0 2rem;min-height:420px;background:var(--c-white,#fff)}
.wcc-left{flex:0 0 52%;border-right:1px solid var(--c-cloud,#e2e8f0);overflow-y:auto;max-height:520px}
.wcc-right{flex:1;display:flex;align-items:center;justify-content:center;padding:2rem;background:var(--c-snow,#f8fafc)}
.wcc-item{border-bottom:1px solid var(--c-cloud,#e2e8f0);cursor:pointer;transition:background .15s}
.wcc-item:last-child{border-bottom:none}
.wcc-item:hover{background:var(--c-fog,#f1f4f8)}
.wcc-item.wcc-active{background:var(--c-fog,#f1f4f8)}
.wcc-header{display:flex;align-items:center;justify-content:space-between;padding:.7rem 1rem}
.wcc-title{font-family:var(--font-body,sans-serif);font-size:.78rem;font-weight:500;color:var(--c-charcoal,#2e3545)}
.wcc-active .wcc-title{color:var(--c-teal,#06d6a0);font-weight:600}
.wcc-chevron{width:10px;height:10px;border-right:2px solid var(--c-muted,#718096);border-bottom:2px solid var(--c-muted,#718096);transform:rotate(45deg);transition:transform .2s;flex-shrink:0}
.wcc-active .wcc-chevron{transform:rotate(-135deg)}
.wcc-body{max-height:0;overflow:hidden;transition:max-height .25s ease,padding .25s ease;padding:0 1rem;font-size:.72rem;line-height:1.55;color:var(--c-slate,#4a5568)}
.wcc-active .wcc-body{max-height:120px;padding:0 1rem .7rem}
.wcc-body a{color:var(--c-teal,#06d6a0);text-decoration:none;font-weight:500}
.wcc-body a:hover{text-decoration:underline}
.wcc-diagram{display:none;max-width:100%;max-height:400px;width:auto;height:auto}
.wcc-diagram.wcc-visible{display:block}
@media(max-width:768px){.wcc-widget{flex-direction:column}.wcc-left{flex:none;border-right:none;border-bottom:1px solid var(--c-cloud,#e2e8f0);max-height:300px}.wcc-right{min-height:300px}}
</style>
<script>
document.addEventListener("DOMContentLoaded",function(){var items=document.querySelectorAll(".wcc-item");var diagrams=document.querySelectorAll(".wcc-diagram");items.forEach(function(item){item.addEventListener("click",function(){items.forEach(function(i){i.classList.remove("wcc-active");i.setAttribute("aria-selected","false")});item.classList.add("wcc-active");item.setAttribute("aria-selected","true");var idx=item.getAttribute("data-wcc");diagrams.forEach(function(d){d.classList.remove("wcc-visible")});var target=document.querySelector('[data-wcc-diagram="'+idx+'"]');if(target)target.classList.add("wcc-visible")});item.addEventListener("keydown",function(e){if(e.key==="Enter"||e.key===" "){e.preventDefault();item.click()}})})});
</script>
## Core building blocks
- **[Workflows](workflows.md)** — The blueprint of a process flow. A workflow is a JSON document
that describes a directed graph of tasks, their dependencies, input/output mappings, and failure
handling policies.
- **[Tasks](tasks.md)** — The basic building blocks of a Conductor workflow. Tasks can be system
tasks (executed by the engine) or worker tasks (executed by external workers polling for work).
- **[Workers](workers.md)** — The code that executes tasks in a Conductor workflow. Workers are
language-agnostic processes that poll the Conductor server, execute business logic, and report
results back.
## Key differentiators
These are the facts that matter when comparing workflow and orchestration engines:
- **Durable execution** — every step is persisted, automatic retries with configurable policies,
and workflows survive crashes and restarts without losing state.
- **Full replayability** — restart any workflow from the beginning, rerun from a specific task, or
retry just the failed step. Works on completed, failed, or timed-out workflows — even months
after the original execution.
- **Deterministic execution** — JSON definitions separate orchestration from implementation. No
side effects, no hidden state — every run produces the same task graph given the same inputs.
Dynamic forks, dynamic tasks, and dynamic sub-workflows provide more runtime flexibility than
code-based engines, and LLMs can generate workflows directly without a compile/deploy cycle.
- **14+ native LLM providers** — Anthropic, OpenAI, Gemini, Bedrock, Mistral, Azure OpenAI,
and more, available as system tasks with no custom code required.
- **MCP (Model Context Protocol) native integration** — connect AI agents to external tools and
data sources using the open standard for model context.
- **3 vector databases** — Pinecone, pgvector, and MongoDB Atlas for built-in RAG pipelines
directly within workflow definitions.
- **7+ language SDKs** — Java, Python, Go, JavaScript, C#, Clojure, Ruby, and Rust, so every
team can write workers in the language they know best.
- **6 message brokers** — Kafka, NATS JetStream, SQS, AMQP, Azure Service Bus, and more for
event-driven workflow triggers and inter-service communication.
- **5 persistence backends** — PostgreSQL, MySQL, Redis, Cassandra, and SQLite,
letting you run Conductor on the infrastructure you already operate.
- **Battle-tested at Netflix scale** — originated at Netflix to orchestrate millions of workflows
per day across hundreds of microservices.
## Deep dives
- [Architecture](../architecture/index.md) — system design and components
- [Durable Execution](../../architecture/durable-execution.md) — failure semantics and state persistence
- [Agents & AI](../ai/index.md) — LLM orchestration patterns and agentic workflows
docs/devguide/concepts/tasks.md
+145
View File
@@ -0,0 +1,145 @@
---
description: "Learn about tasks in Conductor — the reusable building blocks of workflows, including system tasks, worker tasks, operators, LLM tasks with 14+ AI providers, and MCP tool calling."
---
# Tasks
A task is the basic building block of a Conductor workflow. They are reusable and modular, representing steps in your application like processing data files, calling an AI model, or executing some logic.
In Conductor, tasks can be defined, configured, and then executed. Learn more about the distinct but related concepts, **task definition**, **task configuration**, and **task execution** below.
## Types of tasks
Tasks are categorized into three types, enabling you to flexibly build workflows using pre-built tasks, custom logic, or a combination of both:
### System tasks
Conductor ships with 20+ [system tasks](../../documentation/configuration/workflowdef/systemtasks/index.md) — built-in, general-purpose tasks designed for common uses like calling an HTTP endpoint, publishing events, or running AI inference.
System tasks are managed by Conductor and executed within its server's JVM, allowing you to get started without having to write custom workers.
| Category | Tasks |
|---|---|
| **Core** | HTTP, Inline (script), Event, Wait, Human, Kafka Publish, JSON JQ Transform, No Op |
| **Flow Control** | Fork/Join, Dynamic Fork, Join, Switch, Do While, Sub Workflow, Start Workflow, Set Variable, Terminate, Dynamic |
| **AI / LLM** | Chat Completion, Text Completion, Embeddings, Vector Search, Content Generation, MCP Tool Calling |
### Worker tasks
Worker tasks (`SIMPLE`) can be used to implement custom logic outside the scope of Conductor's system tasks. Also known as Simple tasks, Worker tasks are implemented by your task workers that run in a separate environment from Conductor.
A minimal worker task configuration and its corresponding Python worker:
```json
{
"name": "process_payment",
"taskReferenceName": "process_payment_ref",
"type": "SIMPLE",
"inputParameters": {
"orderId": "${workflow.input.orderId}",
"amount": "${workflow.input.amount}"
}
}
```
```python
@worker_task(task_definition_name="process_payment")
def process_payment(orderId: str, amount: float) -> dict:
result = payment_gateway.charge(orderId, amount)
return {"transactionId": result.id, "status": result.status}
```
### Operators
[Operators](../../documentation/configuration/workflowdef/operators/index.md) are built-in control flow primitives similar to programming language constructs like loops, switch cases, or fork/joins. Like system tasks, operators are also managed by Conductor.
## Task definition
[Task definitions](../../documentation/configuration/taskdef.md) are used to define a task's default parameters, like inputs and output keys, timeouts, and retries. This provides reusability across workflows, as the registered task definition will be referenced when a task is configured in a workflow definition.
```json
{
"name": "process_payment",
"retryCount": 3,
"retryLogic": "EXPONENTIAL_BACKOFF",
"retryDelaySeconds": 5,
"maxRetryDelaySeconds": 60,
"backoffJitterMs": 2000,
"totalTimeoutSeconds": 300,
"timeoutSeconds": 120,
"responseTimeoutSeconds": 60,
"pollTimeoutSeconds": 30
}
```
- **retryCount / retryLogic / retryDelaySeconds** — How many times to retry a failed task, the backoff strategy, and the initial delay between retries.
- **maxRetryDelaySeconds** — Caps the computed backoff delay. Prevents exponential growth from becoming arbitrarily large.
- **backoffJitterMs** — Adds random milliseconds to each retry delay to spread concurrent retries over time (thundering herd prevention).
- **totalTimeoutSeconds** — Hard wall-clock budget across all retry attempts combined. Once exceeded, no further retries are attempted regardless of `retryCount`.
- **timeoutSeconds** — Maximum wall-clock time per individual attempt before the task is marked `TIMED_OUT`.
- **responseTimeoutSeconds** — Maximum time to wait for a worker to respond after picking up a task. Useful for detecting unresponsive workers.
- **pollTimeoutSeconds** — Maximum time a worker can hold a long-poll connection before the server releases it.
When using Worker tasks (`SIMPLE`), its task definition must be registered to the Conductor server before it can execute in a workflow. Because system tasks are managed by Conductor, it is not necessary to add a task definition for system tasks unless you wish to customize its default parameters.
## Task configuration
Stored in the `tasks` array of a [workflow definition](workflows.md#workflow-definition), task configurations make up the workflow-specific blueprint that describes:
- The order and control flow of tasks.
- How data is passed from one task to another through task inputs and outputs.
- Other workflow-specific behavior, like optionality, caching, and schema enforcement.
The specific configuration for each task differs depending on the task type. For system tasks and operators, the task configuration will contain important parameters that control the behavior of the task. For example, the task configuration of an HTTP task will specify an endpoint URL and its templatized payload that will be used when the task executes.
Data is passed between tasks using `${...}` expression syntax. This allows a task to reference outputs from a previous task, workflow inputs, or other context variables:
```json
{
"name": "send_notification",
"taskReferenceName": "send_notification_ref",
"type": "SIMPLE",
"inputParameters": {
"recipient": "${workflow.input.email}",
"paymentId": "${process_payment_ref.output.transactionId}",
"status": "${process_payment_ref.output.status}"
}
}
```
For Worker tasks (`SIMPLE`), the configuration will simply contain its inputs/outputs and a reference to its task definition name, because the logic of its behavior will already be specified in the worker code of your application.
There must be at least one task configured in each workflow definition.
## Task execution
A task execution object is created during runtime when an input is passed into a configured task. This object has a unique ID and represents the result of the task operation, including the task status, start time, and inputs/outputs.
## AI and LLM tasks
Conductor includes first-class support for building AI-powered workflows through its AI/LLM [system tasks](../../documentation/configuration/workflowdef/systemtasks/index.md).
### Supported LLM providers
Conductor integrates with **14+ LLM providers** out of the box:
Anthropic, OpenAI, Azure OpenAI, Google Gemini, AWS Bedrock, Mistral, Cohere, HuggingFace, Ollama, Perplexity, Grok, StabilityAI, and more.
Each provider is configured once at the server level; workflows reference them by name, making it straightforward to swap models without changing workflow logic.
### MCP tool calling
The **LIST_MCP_TOOLS** and **CALL_MCP_TOOL** system tasks let your workflows discover and invoke tools exposed by any MCP-compatible server. This enables LLM agents to interact with external APIs, databases, and services through a standardized protocol.
### Vector databases and RAG
For retrieval-augmented generation (RAG), Conductor supports vector stores including **Pinecone**, **pgvector**, and **MongoDB Atlas**. The Embeddings and Vector Search system tasks handle the embedding generation and similarity search steps so that RAG pipelines can be expressed as standard workflows.
### Content generation
Beyond text, Conductor's AI tasks support generating images, audio, video, and PDFs — useful for workflows that produce rich media from LLM outputs.
For end-to-end AI agent patterns that combine LLM reasoning with tool use, see the [agents documentation](../ai/index.md).
docs/devguide/concepts/workers.md
+51
View File
@@ -0,0 +1,51 @@
---
description: "Learn about workers in Conductor — the code that executes tasks in workflows, written in any language and hosted anywhere you choose."
---
# Workers
A worker is responsible for executing a task in a workflow. Each type of worker implements the core functionality of each task, handling the logic as defined in its code.
System task workers are managed by Conductor within its JVM, while `SIMPLE` task workers are to be implemented by yourself. These workers can be implemented in any programming language of your choice (Python, Java, JavaScript, C#, Go, and Clojure) and hosted anywhere outside the Conductor environment.
!!! Note
Conductor provides a set of worker frameworks in its SDKs. These frameworks come with comes with features like polling threads, metrics, and server communication, making it easy to create custom workers.
These workers communicate with the Conductor server via REST/gRPC, allowing them to poll for tasks and update the task status. Learn more in [Architecture](../architecture/index.md).
## How workers work
1. **Poll** — The worker polls the Conductor server for tasks of a specific type.
2. **Execute** — The worker receives a task, executes the business logic, and produces an output.
3. **Report** — The worker reports the task result (COMPLETED or FAILED) back to the server.
Conductor handles scheduling, retries, and state persistence. Your worker just focuses on business logic.
## Worker configuration
Workers are configured through the task definition on the Conductor server. Key settings:
| Parameter | Description |
| :--- | :--- |
| `retryCount` | Number of times Conductor retries a failed task. |
| `retryDelaySeconds` | Delay between retries. |
| `responseTimeoutSeconds` | Max time for a worker to respond after polling. |
| `timeoutSeconds` | Overall SLA for task completion. |
| `pollTimeoutSeconds` | Max time for a worker to poll before timeout. |
| `rateLimitPerFrequency` | Max task executions per frequency window. |
| `concurrentExecLimit` | Max concurrent executions across all workers. |
See [Task Definitions](../../documentation/configuration/taskdef.md) for the full reference.
## Scaling task workers
Workers can be scaled independently of the Conductor server:
- **Horizontal scaling** — Run multiple instances of the same worker. Conductor distributes tasks across all polling workers automatically.
- **Rate limiting** — Use `rateLimitPerFrequency` to control throughput per task type.
- **Concurrency limits** — Use `concurrentExecLimit` to cap parallel executions.
- **Domain isolation** — Use [task domains](../../documentation/api/taskdomains.md) to route tasks to specific worker groups.
See [Scaling Workers](../how-tos/Workers/scaling-workers.md) for detailed guidance.
docs/devguide/concepts/workflows.md
+158
View File
@@ -0,0 +1,158 @@
---
description: "Understand workflows in Conductor — JSON workflow definition, dynamic workflows, distributed workflow execution, and long-running async workflows that power durable code execution across distributed services."
---
# Workflows
A workflow is a sequence of tasks with a defined order and execution. Each workflow encapsulates a specific process, such as:
- Classifying documents
- Ordering from a self-checkout service
- Upgrading cloud infrastructure
- Transcoding videos
- Approving expenses
In Conductor, workflows can be defined and then executed. Learn more about the two distinct but related concepts, **workflow definition** and **workflow execution**, below.
## What makes Conductor workflows different
Conductor workflows stand apart from traditional orchestration approaches in several key ways:
- **Durable execution** — Workflows survive process failures, restarts, and infrastructure outages. Conductor persists state at every step, so a long-running workflow or async workflow picks up exactly where it left off — even after days or weeks.
- **JSON-native definitions** — Every workflow is a JSON workflow definition you can store in version control, diff across releases, and generate programmatically. No compiled DSL or proprietary format required.
- **Dynamic workflows** — Workflows can be created and modified at runtime as code-first or JSON definitions, enabling use cases where the task graph is not known ahead of time (for example, when the number of parallel branches depends on an API response).
- **Versioned** — Each workflow definition carries an explicit version number so you can roll out changes incrementally and run multiple versions side by side.
- **Language-agnostic** — Workers that execute tasks can be written in any language — Java, Python, Go, JavaScript, C#, or Clojure — and deployed anywhere. The workflow definition itself is decoupled from implementation.
## Workflow definition
The workflow definition describes the flow and behavior of your business logic. Think of it as a blueprint specifying how it should execute at runtime until it reaches a terminal state. The workflow definition includes:
- The workflow's input/output keys.
- A collection of [task configurations](tasks.md#task-configuration) that specify the task conditions, sequence, and data flow until the workflow is completed.
- The workflow's runtime behavior, such as the timeout policy and compensation flow.
### Example JSON workflow definition
Below is a realistic three-task workflow that fetches data from an API, transforms it with an inline script, and then delegates the result to a worker task for further processing.
```json
{
"name": "process_order",
"description": "Fetch order details, enrich them, and hand off to fulfillment",
"version": 1,
"schemaVersion": 2,
"ownerEmail": "team-platform@example.com",
"timeoutPolicy": "ALERT_ONLY",
"timeoutSeconds": 3600,
"restartable": true,
"failureWorkflow": "handle_order_failure",
"inputParameters": ["orderId"],
"outputParameters": {
"enrichedOrder": "${enrich_order.output.result}",
"fulfillmentStatus": "${fulfill_order.output.status}"
},
"tasks": [
{
"name": "fetch_order",
"taskReferenceName": "fetch_order",
"type": "HTTP",
"inputParameters": {
"http_request": {
"uri": "https://api.example.com/orders/${workflow.input.orderId}",
"method": "GET",
"connectionTimeOut": 5000,
"readTimeOut": 5000
}
}
},
{
"name": "enrich_order",
"taskReferenceName": "enrich_order",
"type": "INLINE",
"inputParameters": {
"order": "${fetch_order.output.response.body}",
"evaluatorType": "graaljs",
"expression": "(function() { var o = $.order; o.region = o.country === 'US' ? 'domestic' : 'international'; return o; })()"
}
},
{
"name": "fulfill_order",
"taskReferenceName": "fulfill_order",
"type": "SIMPLE",
"inputParameters": {
"enrichedOrder": "${enrich_order.output.result}"
}
}
]
}
```
### Workflow definition parameters
| Parameter | Type | Description |
|---|---|---|
| **name** | `string` | A unique name identifying the workflow. Used when starting executions. |
| **version** | `integer` | The version of the workflow definition. Allows multiple versions to coexist. |
| **tasks** | `array[object]` | An ordered list of [task configurations](tasks.md#task-configuration) that define the workflow's execution graph. |
| **inputParameters** | `array[string]` | List of input keys the workflow expects when triggered. |
| **outputParameters** | `object` | Mapping of output keys to expressions that extract values from task outputs. |
| **failureWorkflow** | `string` | Name of a workflow to trigger when this workflow transitions to FAILED. Useful for compensation or alerting. |
| **timeoutPolicy** | `string` | Policy to apply when the workflow exceeds `timeoutSeconds`. Supported values: `TIME_OUT_WF` (fail the workflow) or `ALERT_ONLY` (mark timed out but keep running). |
| **timeoutSeconds** | `integer` | Maximum time (in seconds) the workflow is allowed to run before the timeout policy is applied. Set to `0` for no timeout. |
| **restartable** | `boolean` | Whether the workflow can be restarted after completion or failure. Defaults to `true`. |
| **ownerEmail** | `string` | Email address of the workflow owner. Used for notifications and audit tracking. |
| **schemaVersion** | `integer` | Schema version of the workflow definition format. Current version is `2`. |
## Workflow execution
A workflow execution is the execution instance of a workflow definition.
Whenever a workflow definition is invoked with a given input, a new workflow execution with a unique ID is created. The workflow is governed by a defined state (like RUNNING or COMPLETED), which makes it intuitive to track the workflow.
### Workflow execution states
Each workflow execution transitions through a set of well-defined states:
| State | Description |
|---|---|
| **RUNNING** | The workflow is actively executing tasks. |
| **COMPLETED** | All tasks finished successfully and the workflow reached its terminal state. |
| **FAILED** | One or more tasks failed and the workflow could not recover. If a `failureWorkflow` is configured, it will be triggered. |
| **TIMED_OUT** | The workflow exceeded its configured `timeoutSeconds` and the `timeoutPolicy` was set to `TIME_OUT_WF`. |
| **TERMINATED** | The workflow was explicitly stopped by an API call or system action. |
| **PAUSED** | The workflow has been paused and will not schedule new tasks until resumed. |
The following diagram illustrates how a workflow transitions between states:
```mermaid
stateDiagram-v2
[*] --> RUNNING
RUNNING --> COMPLETED : all tasks succeed
RUNNING --> FAILED : task failure (unrecoverable)
RUNNING --> TIMED_OUT : timeout exceeded
RUNNING --> TERMINATED : API termination
RUNNING --> PAUSED : pause requested
PAUSED --> RUNNING : resume requested
PAUSED --> TERMINATED : API termination
FAILED --> RUNNING : retry
TIMED_OUT --> RUNNING : retry
TERMINATED --> RUNNING : restart (if restartable)
COMPLETED --> [*]
FAILED --> [*]
TIMED_OUT --> [*]
TERMINATED --> [*]
```
## Next steps
- [Tasks](tasks.md) — Learn about the building blocks that make up a workflow, including system tasks, worker tasks, and operators.
- [Workers](workers.md) — Understand how to implement task workers in any programming language.
- [Handling errors](../how-tos/Workflows/handling-errors.md) — Configure retries, failure workflows, and compensation strategies.
docs/devguide/cookbook/ai-llm.md
+714
View File
@@ -0,0 +1,714 @@
---
description: "LLM orchestration cookbook — AI agent orchestration recipes for chat completion, RAG pipelines, MCP agents with function calling, web search, code execution, coding agents, extended thinking, image generation, LLM-to-PDF, and provider configuration."
---
# AI & LLM orchestration recipes
Build durable agents and LLM workflows with Conductor's native AI capabilities. Every recipe below runs with full durable execution guarantees — retries, state persistence, and crash recovery.
### Chat completion
A single-step workflow that sends a question to an LLM and returns the answer.
```json
{
"name": "chat_workflow",
"version": 1,
"schemaVersion": 2,
"tasks": [
{
"name": "chat_task",
"taskReferenceName": "chat",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "openai",
"model": "gpt-4o-mini",
"messages": [
{"role": "system", "message": "You are a helpful assistant."},
{"role": "user", "message": "${workflow.input.question}"}
],
"temperature": 0.7,
"maxTokens": 500
}
}
],
"inputParameters": ["question"],
"outputParameters": {
"answer": "${chat.output.result}"
}
}
```
**Register and run:**
```shell
curl -X POST 'http://localhost:8080/api/metadata/workflow' \
-H 'Content-Type: application/json' \
-d @chat_workflow.json
curl -X POST 'http://localhost:8080/api/workflow/chat_workflow' \
-H 'Content-Type: application/json' \
-d '{"question": "What is workflow orchestration?"}'
```
---
### RAG pipeline with vector database (search + answer)
A vector database workflow for retrieval-augmented generation: vector search retrieves relevant documents, then an LLM generates an answer grounded in those results.
```json
{
"name": "rag_workflow",
"version": 1,
"schemaVersion": 2,
"inputParameters": ["question"],
"tasks": [
{
"name": "search_knowledge_base",
"taskReferenceName": "search",
"type": "LLM_SEARCH_INDEX",
"inputParameters": {
"vectorDB": "postgres-prod",
"namespace": "kb",
"index": "articles",
"embeddingModelProvider": "openai",
"embeddingModel": "text-embedding-3-small",
"query": "${workflow.input.question}",
"llmMaxResults": 3
}
},
{
"name": "generate_answer",
"taskReferenceName": "answer",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "anthropic",
"model": "claude-sonnet-4-20250514",
"messages": [
{"role": "system", "message": "Answer based on the following context: ${search.output.result}"},
{"role": "user", "message": "${workflow.input.question}"}
],
"temperature": 0.3
}
}
],
"outputParameters": {
"answer": "${answer.output.result}",
"sources": "${search.output.result}"
}
}
```
**Register and run:**
```shell
curl -X POST 'http://localhost:8080/api/metadata/workflow' \
-H 'Content-Type: application/json' \
-d @rag_workflow.json
curl -X POST 'http://localhost:8080/api/workflow/rag_workflow' \
-H 'Content-Type: application/json' \
-d '{"question": "How do I configure retry policies?"}'
```
!!! note "Prerequisites"
Requires a vector database (pgvector, Pinecone, or MongoDB Atlas) configured as a Conductor integration, plus at least one LLM provider. See [AI provider configuration](#ai-provider-configuration) below.
---
### MCP AI agent with function calling
A four-step agentic workflow demonstrating AI agent orchestration with function calling: discover available tools via MCP, ask an LLM to pick the right tool, execute it via tool use, and summarize the result.
```json
{
"name": "mcp_ai_agent_workflow",
"version": 1,
"schemaVersion": 2,
"inputParameters": ["task"],
"tasks": [
{
"name": "list_available_tools",
"taskReferenceName": "discover_tools",
"type": "LIST_MCP_TOOLS",
"inputParameters": {
"mcpServer": "http://localhost:3001/mcp"
}
},
{
"name": "decide_which_tools_to_use",
"taskReferenceName": "plan",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "anthropic",
"model": "claude-sonnet-4-20250514",
"messages": [
{"role": "system", "message": "You are an AI agent. Available tools: ${discover_tools.output.tools}. User wants to: ${workflow.input.task}"},
{"role": "user", "message": "Which tool should I use and what parameters? Respond with JSON: {method: string, arguments: object}"}
],
"temperature": 0.1,
"maxTokens": 500
}
},
{
"name": "execute_tool",
"taskReferenceName": "execute",
"type": "CALL_MCP_TOOL",
"inputParameters": {
"mcpServer": "http://localhost:3001/mcp",
"method": "${plan.output.result.method}",
"arguments": "${plan.output.result.arguments}"
}
},
{
"name": "summarize_result",
"taskReferenceName": "summarize",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "openai",
"model": "gpt-4o-mini",
"messages": [
{"role": "user", "message": "Summarize this result for the user: ${execute.output.content}"}
],
"maxTokens": 200
}
}
],
"outputParameters": {
"summary": "${summarize.output.result}",
"rawToolOutput": "${execute.output.content}"
}
}
```
**Register and run:**
```shell
curl -X POST 'http://localhost:8080/api/metadata/workflow' \
-H 'Content-Type: application/json' \
-d @mcp_ai_agent_workflow.json
curl -X POST 'http://localhost:8080/api/workflow/mcp_ai_agent_workflow' \
-H 'Content-Type: application/json' \
-d '{"task": "Look up the latest order status for customer 42"}'
```
---
### Image generation
Generate images from a text prompt using DALL-E or another supported provider.
```json
{
"name": "image_gen_workflow",
"version": 1,
"schemaVersion": 2,
"inputParameters": ["prompt"],
"tasks": [
{
"name": "generate_image",
"taskReferenceName": "image",
"type": "GENERATE_IMAGE",
"inputParameters": {
"llmProvider": "openai",
"model": "dall-e-3",
"prompt": "${workflow.input.prompt}",
"width": 1024,
"height": 1024,
"n": 1,
"style": "vivid"
}
}
],
"outputParameters": {
"imageUrl": "${image.output.result}"
}
}
```
**Register and run:**
```shell
curl -X POST 'http://localhost:8080/api/metadata/workflow' \
-H 'Content-Type: application/json' \
-d @image_gen_workflow.json
curl -X POST 'http://localhost:8080/api/workflow/image_gen_workflow' \
-H 'Content-Type: application/json' \
-d '{"prompt": "A futuristic city skyline at sunset, digital art"}'
```
---
### LLM report to PDF pipeline
An LLM generates a structured markdown report, then Conductor converts it to a downloadable PDF.
```json
{
"name": "llm_to_pdf_pipeline",
"description": "LLM generates a markdown report, then converts it to PDF",
"version": 1,
"schemaVersion": 2,
"inputParameters": ["topic", "audience"],
"tasks": [
{
"name": "generate_report_markdown",
"taskReferenceName": "llm_report",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "openai",
"model": "gpt-4o-mini",
"messages": [
{"role": "system", "message": "You are a professional report writer. Generate well-structured markdown reports."},
{"role": "user", "message": "Write a detailed report about: ${workflow.input.topic}\nTarget audience: ${workflow.input.audience}"}
],
"temperature": 0.7,
"maxTokens": 2000
}
},
{
"name": "convert_to_pdf",
"taskReferenceName": "pdf_output",
"type": "GENERATE_PDF",
"inputParameters": {
"markdown": "${llm_report.output.result}",
"pageSize": "A4",
"theme": "default",
"baseFontSize": 11,
"pdfMetadata": {
"title": "${workflow.input.topic}",
"author": "Conductor AI Pipeline"
}
}
}
],
"outputParameters": {
"reportMarkdown": "${llm_report.output.result}",
"pdfLocation": "${pdf_output.output.result.location}"
}
}
```
**Register and run:**
```shell
curl -X POST 'http://localhost:8080/api/metadata/workflow' \
-H 'Content-Type: application/json' \
-d @llm_to_pdf_pipeline.json
curl -X POST 'http://localhost:8080/api/workflow/llm_to_pdf_pipeline' \
-H 'Content-Type: application/json' \
-d '{"topic": "Microservices observability best practices", "audience": "Platform engineering team"}'
```
---
### Web search — real-time information retrieval
Enable the LLM's built-in web search to answer questions about current events or find up-to-date information. No MCP server or external tool needed — the provider handles the search natively.
```json
{
"name": "web_search_workflow",
"version": 1,
"schemaVersion": 2,
"inputParameters": ["question"],
"tasks": [
{
"name": "web_search_chat",
"taskReferenceName": "chat",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "openai",
"model": "gpt-4o-mini",
"messages": [
{"role": "system", "message": "Use web search to find current information."},
{"role": "user", "message": "${workflow.input.question}"}
],
"webSearch": true,
"maxTokens": 1000
}
}
],
"outputParameters": {
"answer": "${chat.output.result}"
}
}
```
**Register and run:**
```shell
curl -X POST 'http://localhost:8080/api/metadata/workflow' \
-H 'Content-Type: application/json' \
-d @web_search_workflow.json
curl -X POST 'http://localhost:8080/api/workflow/web_search_workflow' \
-H 'Content-Type: application/json' \
-d '{"question": "What are the latest developments in AI regulation?"}'
```
!!! note "Provider support"
Web search is supported by OpenAI, Anthropic, and Google Gemini. Set `"webSearch": true` — the same parameter works across all providers.
---
### Code execution — sandboxed code interpreter
Let the LLM write and run code in a sandboxed environment. Useful for data analysis, calculations, chart generation, and tasks that benefit from executable code.
```json
{
"name": "code_execution_workflow",
"version": 1,
"schemaVersion": 2,
"inputParameters": ["task"],
"tasks": [
{
"name": "code_chat",
"taskReferenceName": "chat",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "google_gemini",
"model": "gemini-2.5-flash",
"messages": [
{"role": "system", "message": "Use code execution to compute results and analyze data."},
{"role": "user", "message": "${workflow.input.task}"}
],
"codeInterpreter": true,
"maxTokens": 2000
}
}
],
"outputParameters": {
"result": "${chat.output.result}"
}
}
```
**Register and run:**
```shell
curl -X POST 'http://localhost:8080/api/metadata/workflow' \
-H 'Content-Type: application/json' \
-d @code_execution_workflow.json
curl -X POST 'http://localhost:8080/api/workflow/code_execution_workflow' \
-H 'Content-Type: application/json' \
-d '{"task": "Calculate the first 100 prime numbers and find the average gap between consecutive primes"}'
```
!!! note "Provider support"
Code execution is supported by OpenAI (`code_interpreter`), Anthropic (`code_execution`), and Google Gemini (`codeExecution`). Set `"codeInterpreter": true` — the same parameter works across all providers.
---
### Coding agent — plan, code, and review
A three-step agent that plans an implementation, writes and executes the code using the code interpreter, and reviews the result. This pattern is useful for automated code generation tasks.
```json
{
"name": "coding_agent",
"version": 1,
"schemaVersion": 2,
"inputParameters": ["task"],
"tasks": [
{
"name": "plan",
"taskReferenceName": "plan",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "openai",
"model": "gpt-4o",
"messages": [
{"role": "system", "message": "Break down the coding task into clear numbered steps."},
{"role": "user", "message": "${workflow.input.task}"}
],
"temperature": 0.2,
"maxTokens": 1000
}
},
{
"name": "write_and_run",
"taskReferenceName": "code",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "openai",
"model": "gpt-4o",
"messages": [
{"role": "system", "message": "Write the code, run it, verify the output, and fix any errors."},
{"role": "user", "message": "Plan:\n${plan.output.result}\n\nTask: ${workflow.input.task}"}
],
"codeInterpreter": true,
"temperature": 0.1,
"maxTokens": 4000
}
},
{
"name": "review",
"taskReferenceName": "review",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "openai",
"model": "gpt-4o-mini",
"messages": [
{"role": "system", "message": "Review the implementation for correctness and code quality."},
{"role": "user", "message": "Task: ${workflow.input.task}\n\nCode:\n${code.output.result}"}
],
"maxTokens": 1000
}
}
],
"outputParameters": {
"code": "${code.output.result}",
"review": "${review.output.result}"
}
}
```
**Register and run:**
```shell
curl -X POST 'http://localhost:8080/api/metadata/workflow' \
-H 'Content-Type: application/json' \
-d @coding_agent.json
curl -X POST 'http://localhost:8080/api/workflow/coding_agent' \
-H 'Content-Type: application/json' \
-d '{"task": "Write a Python function that converts Roman numerals to integers, with unit tests"}'
```
---
### Extended thinking — complex reasoning
Give the LLM a token budget for step-by-step reasoning before generating its final response. Useful for math, logic, code review, and complex analysis.
```json
{
"name": "extended_thinking_workflow",
"version": 1,
"schemaVersion": 2,
"inputParameters": ["problem"],
"tasks": [
{
"name": "think_deeply",
"taskReferenceName": "think",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "anthropic",
"model": "claude-sonnet-4-20250514",
"messages": [
{"role": "user", "message": "${workflow.input.problem}"}
],
"thinkingTokenLimit": 10000,
"maxTokens": 16000
}
}
],
"outputParameters": {
"answer": "${think.output.result}"
}
}
```
**Register and run:**
```shell
curl -X POST 'http://localhost:8080/api/metadata/workflow' \
-H 'Content-Type: application/json' \
-d @extended_thinking_workflow.json
curl -X POST 'http://localhost:8080/api/workflow/extended_thinking_workflow' \
-H 'Content-Type: application/json' \
-d '{"problem": "Prove that the square root of 2 is irrational."}'
```
!!! note "Provider support"
Extended thinking is supported by Anthropic (`thinkingTokenLimit`) and Google Gemini (`thinkingBudgetTokens`). OpenAI uses `"reasoningEffort": "high"` for a similar effect.
---
### Multi-turn conversation chaining with previousResponseId
Chain multiple LLM calls as a conversation without resending the full message history. The first call returns a `responseId`; pass it as `previousResponseId` to the next call. OpenAI's Responses API stores the conversation server-side, saving tokens and latency.
```json
{
"name": "multi_turn_chain",
"description": "Two-step conversation using previousResponseId to avoid resending history",
"version": 1,
"schemaVersion": 2,
"inputParameters": ["topic"],
"tasks": [
{
"name": "first_turn",
"taskReferenceName": "turn1",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "openai",
"model": "gpt-4o",
"messages": [
{"role": "system", "message": "You are a technical architect. Be concise."},
{"role": "user", "message": "Design a high-level architecture for: ${workflow.input.topic}"}
],
"temperature": 0.3,
"maxTokens": 2000
}
},
{
"name": "follow_up",
"taskReferenceName": "turn2",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "openai",
"model": "gpt-4o",
"messages": [
{"role": "user", "message": "Now list the key risks and mitigations for this architecture."}
],
"previousResponseId": "${turn1.output.responseId}",
"temperature": 0.3,
"maxTokens": 2000
}
}
],
"outputParameters": {
"architecture": "${turn1.output.result}",
"risks": "${turn2.output.result}"
}
}
```
**Register and run:**
```shell
curl -X POST 'http://localhost:8080/api/metadata/workflow' \
-H 'Content-Type: application/json' \
-d @multi_turn_chain.json
curl -X POST 'http://localhost:8080/api/workflow/multi_turn_chain' \
-H 'Content-Type: application/json' \
-d '{"topic": "Real-time collaborative document editor"}'
```
The second call sends only the new user message — OpenAI already has the full conversation context from `previousResponseId`. This is especially useful for long agent loops where resending the full history each iteration would be expensive.
!!! note "Provider support"
`previousResponseId` is supported by OpenAI and Azure OpenAI (Responses API). Other providers require sending the full message history in each call.
---
### Web research agent — search, synthesize, PDF
A multi-step agent that uses web search to gather information, an LLM with extended thinking to synthesize a report, and converts it to PDF. Combines three built-in capabilities in a single workflow.
```json
{
"name": "web_research_agent",
"version": 1,
"schemaVersion": 2,
"inputParameters": ["topic"],
"tasks": [
{
"name": "gather_information",
"taskReferenceName": "research",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "openai",
"model": "gpt-4o",
"messages": [
{"role": "system", "message": "Use web search to find comprehensive, current information. Search for multiple perspectives and recent developments."},
{"role": "user", "message": "Research this topic thoroughly: ${workflow.input.topic}"}
],
"webSearch": true,
"temperature": 0.3,
"maxTokens": 3000
}
},
{
"name": "synthesize_report",
"taskReferenceName": "report",
"type": "LLM_CHAT_COMPLETE",
"inputParameters": {
"llmProvider": "anthropic",
"model": "claude-sonnet-4-20250514",
"messages": [
{"role": "system", "message": "Synthesize the research into a well-structured markdown report with sections, key findings, and citations."},
{"role": "user", "message": "Topic: ${workflow.input.topic}\n\nResearch:\n${research.output.result}\n\nWrite a comprehensive report."}
],
"thinkingTokenLimit": 5000,
"maxTokens": 8000
}
},
{
"name": "convert_to_pdf",
"taskReferenceName": "pdf",
"type": "GENERATE_PDF",
"inputParameters": {
"markdown": "${report.output.result}",
"pageSize": "A4",
"pdfMetadata": {
"title": "${workflow.input.topic}",
"author": "Conductor Research Agent"
}
}
}
],
"outputParameters": {
"report": "${report.output.result}",
"pdf": "${pdf.output.result.location}"
}
}
```
**Register and run:**
```shell
curl -X POST 'http://localhost:8080/api/metadata/workflow' \
-H 'Content-Type: application/json' \
-d @web_research_agent.json
curl -X POST 'http://localhost:8080/api/workflow/web_research_agent' \
-H 'Content-Type: application/json' \
-d '{"topic": "The state of WebAssembly adoption in 2026"}'
```
---
### AI provider configuration
Set environment variables before starting the server. Conductor auto-enables providers when their API key is present.
```bash
# OpenAI (required for most examples)
export OPENAI_API_KEY=sk-your-openai-api-key
# Anthropic (for RAG, extended thinking examples)
export ANTHROPIC_API_KEY=sk-ant-your-anthropic-key
# Google Gemini — API key (simplest)
export GEMINI_API_KEY=your-gemini-api-key
# Or Vertex AI (for enterprise/GCP) — set project and location in application.properties
```
For vector database and other advanced configuration, add to `application.properties`:
```properties
# PostgreSQL Vector DB (for RAG examples)
conductor.vectordb.instances[0].name=postgres-prod
conductor.vectordb.instances[0].type=postgres
conductor.vectordb.instances[0].postgres.datasourceURL=jdbc:postgresql://localhost:5432/vectors
conductor.vectordb.instances[0].postgres.user=conductor
conductor.vectordb.instances[0].postgres.password=secret
conductor.vectordb.instances[0].postgres.dimensions=1536
```
---
## More examples
For additional AI workflow definitions, see the [AI workflow examples on GitHub](https://github.com/conductor-oss/conductor/tree/main/ai/examples).
docs/devguide/cookbook/dynamic-parallelism.md
+156
View File
@@ -0,0 +1,156 @@
---
description: "Conductor cookbook — dynamic parallelism recipes with Dynamic Fork for different tasks per branch, fan-out with same task, and parallel sub-workflows."
---
# Dynamic parallelism
### Run different tasks in parallel (Dynamic Fork)
Use `dynamicForkTasksParam` + `dynamicForkTasksInputParamName` when each parallel branch runs a **different** task. The task list is determined at runtime by a preceding step.
```json
{
"name": "dynamic_fork_different_tasks",
"version": 1,
"schemaVersion": 2,
"tasks": [
{
"name": "prepare_tasks",
"taskReferenceName": "prepare",
"type": "INLINE",
"inputParameters": {
"evaluatorType": "graaljs",
"expression": "(function() { return { dynamicTasks: [{name: 'HTTP', taskReferenceName: 'fetch_weather', type: 'HTTP'}, {name: 'HTTP', taskReferenceName: 'fetch_news', type: 'HTTP'}], dynamicTasksInput: { fetch_weather: { http_request: {uri: 'https://api.weather.gov/points/39.7456,-104.9994', method: 'GET'}}, fetch_news: { http_request: {uri: 'https://hacker-news.firebaseio.com/v0/topstories.json', method: 'GET'}}}}; })()"
}
},
{
"name": "fork_join_dynamic",
"taskReferenceName": "dynamic_fork",
"type": "FORK_JOIN_DYNAMIC",
"inputParameters": {
"dynamicTasks": "${prepare.output.result.dynamicTasks}",
"dynamicTasksInput": "${prepare.output.result.dynamicTasksInput}"
},
"dynamicForkTasksParam": "dynamicTasks",
"dynamicForkTasksInputParamName": "dynamicTasksInput"
},
{
"name": "join",
"taskReferenceName": "join_ref",
"type": "JOIN"
}
]
}
```
`dynamicTasks` is an array of task definitions (each with `name`, `taskReferenceName`, and `type`). `dynamicTasksInput` is a map keyed by each task's `taskReferenceName` containing its input payload.
**Register and run:**
```shell
curl -X POST 'http://localhost:8080/api/metadata/workflow' \
-H 'Content-Type: application/json' \
-d @dynamic_fork_different_tasks.json
curl -X POST 'http://localhost:8080/api/workflow/dynamic_fork_different_tasks' \
-H 'Content-Type: application/json' \
-d '{}'
```
---
### Run same task in parallel (fan-out)
Use `forkTaskName` + `forkTaskInputs` when running the **same** task type across multiple inputs.
```json
{
"name": "fan_out_http_calls",
"version": 1,
"schemaVersion": 2,
"tasks": [
{
"name": "fork_join_dynamic",
"taskReferenceName": "parallel_fetch",
"type": "FORK_JOIN_DYNAMIC",
"inputParameters": {
"forkTaskName": "HTTP",
"forkTaskInputs": [
{"http_request": {"uri": "https://jsonplaceholder.typicode.com/posts/1", "method": "GET"}},
{"http_request": {"uri": "https://jsonplaceholder.typicode.com/posts/2", "method": "GET"}},
{"http_request": {"uri": "https://jsonplaceholder.typicode.com/posts/3", "method": "GET"}}
]
}
},
{
"name": "join",
"taskReferenceName": "join_ref",
"type": "JOIN"
}
]
}
```
!!! tip
Conductor injects `__index` into each fork's input so you can track the position of each parallel branch in the results.
**Register and run:**
```shell
curl -X POST 'http://localhost:8080/api/metadata/workflow' \
-H 'Content-Type: application/json' \
-d @fan_out_http_calls.json
curl -X POST 'http://localhost:8080/api/workflow/fan_out_http_calls' \
-H 'Content-Type: application/json' \
-d '{}'
```
---
### Run sub-workflows in parallel
Use `forkTaskWorkflow` + `forkTaskInputs` to fan out across instances of another workflow.
```json
{
"name": "parallel_sub_workflows",
"version": 1,
"schemaVersion": 2,
"tasks": [
{
"name": "fork_join_dynamic",
"taskReferenceName": "parallel_regions",
"type": "FORK_JOIN_DYNAMIC",
"inputParameters": {
"forkTaskWorkflow": "process_region",
"forkTaskWorkflowVersion": 1,
"forkTaskInputs": [
{"region": "us-east-1", "data": "batch_a"},
{"region": "eu-west-1", "data": "batch_b"},
{"region": "ap-southeast-1", "data": "batch_c"}
]
}
},
{
"name": "join",
"taskReferenceName": "join_ref",
"type": "JOIN"
}
]
}
```
Each element in `forkTaskInputs` spawns one instance of the `process_region` workflow. All results are collected at the JOIN task.
**Register and run:**
```shell
curl -X POST 'http://localhost:8080/api/metadata/workflow' \
-H 'Content-Type: application/json' \
-d @parallel_sub_workflows.json
curl -X POST 'http://localhost:8080/api/workflow/parallel_sub_workflows' \
-H 'Content-Type: application/json' \
-d '{}'
```
docs/devguide/cookbook/dynamic-workflows.md
+365
View File
@@ -0,0 +1,365 @@
---
description: "Workflow as code — build code-first workflows dynamically in Python using the Conductor SDK. Conditional branching, loops, parallel execution, and runtime-generated dynamic workflows."
---
# Dynamic workflows in code
## Workflow as code
Conductor supports a code-first workflow approach — build workflows programmatically using the Python SDK instead of writing JSON by hand. This workflow as code pattern lets you chain tasks with the `>>` operator, add conditional logic, loops, and parallel branches — all in Python. Code-first workflows are ideal for dynamic workflows where the task graph is determined at runtime.
### Simple sequential workflow
Chain tasks with the `>>` operator. Worker functions decorated with `@worker_task` become reusable task building blocks.
```python
from conductor.client.workflow.conductor_workflow import ConductorWorkflow
from conductor.client.worker.worker_task import worker_task
@worker_task(task_definition_name='fetch_order')
def fetch_order(order_id: str) -> dict:
return {'order_id': order_id, 'amount': 99.99, 'item': 'Widget'}
@worker_task(task_definition_name='process_payment')
def process_payment(order_id: str, amount: float) -> dict:
return {'transaction_id': 'txn_abc123', 'status': 'charged'}
@worker_task(task_definition_name='ship_order')
def ship_order(order_id: str, transaction_id: str) -> dict:
return {'tracking': 'TRACK-456', 'carrier': 'FedEx'}
workflow = ConductorWorkflow(name='order_fulfillment', version=1, executor=executor)
fetch = fetch_order(task_ref_name='fetch', order_id=workflow.input('order_id'))
pay = process_payment(
task_ref_name='pay',
order_id=workflow.input('order_id'),
amount=fetch.output('amount'),
)
ship = ship_order(
task_ref_name='ship',
order_id=workflow.input('order_id'),
transaction_id=pay.output('transaction_id'),
)
workflow >> fetch >> pay >> ship
workflow.output_parameters({
'tracking': ship.output('tracking'),
'transaction_id': pay.output('transaction_id'),
})
workflow.register(overwrite=True)
```
---
### Conditional branching with Switch
Route execution based on task output or workflow input. Each case gets its own task chain.
```python
from conductor.client.workflow.conductor_workflow import ConductorWorkflow
from conductor.client.workflow.task.switch_task import SwitchTask
workflow = ConductorWorkflow(name='route_by_priority', version=1, executor=executor)
classify = classify_ticket(
task_ref_name='classify',
description=workflow.input('description'),
)
switch = SwitchTask(task_ref_name='priority_router', case_expression=classify.output('priority'))
# Each case is a list of tasks to execute
switch.switch_case('critical', [
page_oncall(task_ref_name='page', ticket_id=workflow.input('ticket_id')),
escalate(task_ref_name='escalate', ticket_id=workflow.input('ticket_id')),
])
switch.switch_case('high', [
assign_senior(task_ref_name='assign', ticket_id=workflow.input('ticket_id')),
])
switch.default_case([
add_to_backlog(task_ref_name='backlog', ticket_id=workflow.input('ticket_id')),
])
workflow >> classify >> switch
workflow.register(overwrite=True)
```
---
### Parallel execution with Fork/Join
Run independent tasks in parallel and wait for all to complete.
```python
from conductor.client.workflow.conductor_workflow import ConductorWorkflow
from conductor.client.workflow.task.fork_task import ForkTask
from conductor.client.workflow.task.join_task import JoinTask
workflow = ConductorWorkflow(name='parallel_enrichment', version=1, executor=executor)
# Define independent tasks
credit_check = check_credit(task_ref_name='credit', customer_id=workflow.input('customer_id'))
fraud_check = check_fraud(task_ref_name='fraud', customer_id=workflow.input('customer_id'))
kyc_check = check_kyc(task_ref_name='kyc', customer_id=workflow.input('customer_id'))
# Fork runs all branches in parallel
fork = ForkTask(
task_ref_name='parallel_checks',
forked_tasks=[
[credit_check],
[fraud_check],
[kyc_check],
],
)
# Join waits for all branches
join = JoinTask(task_ref_name='wait_all', join_on=['credit', 'fraud', 'kyc'])
# Merge results
decide = make_decision(
task_ref_name='decide',
credit_score=credit_check.output('score'),
fraud_risk=fraud_check.output('risk_level'),
kyc_status=kyc_check.output('status'),
)
workflow >> fork >> join >> decide
workflow.output_parameters({'decision': decide.output('result')})
workflow.register(overwrite=True)
```
---
### Loops with Do/While
Repeat a set of tasks until a condition is met — useful for polling, retries, or iterative AI agent loops.
```python
from conductor.client.workflow.conductor_workflow import ConductorWorkflow
from conductor.client.workflow.task.do_while_task import DoWhileTask
workflow = ConductorWorkflow(name='agent_loop', version=1, executor=executor)
# The task(s) to repeat each iteration
think = call_llm(
task_ref_name='think',
prompt=workflow.input('goal'),
)
act = execute_tool(
task_ref_name='act',
tool=think.output('tool'),
args=think.output('args'),
)
# Loop until the LLM says it's done (max 10 iterations)
loop = DoWhileTask(
task_ref_name='agent_loop',
termination_condition='if ($.act["output"]["done"] == true) { false; } else { true; }',
tasks=[think, act],
)
loop.input_parameters.update({'max_iterations': 10})
summarize = summarize_results(task_ref_name='summarize', results=act.output('results'))
workflow >> loop >> summarize
workflow.register(overwrite=True)
```
---
### HTTP + system tasks mixed with workers
Combine built-in system tasks (HTTP, Wait, JQ Transform) with custom workers — no extra deployment needed for system tasks.
{% raw %}
```python
from conductor.client.workflow.conductor_workflow import ConductorWorkflow
from conductor.client.workflow.task.http_task import HttpTask
from conductor.client.workflow.task.json_jq_task import JsonJQTask
from conductor.client.workflow.task.wait_task import WaitTask
workflow = ConductorWorkflow(name='data_pipeline', version=1, executor=executor)
# HTTP task — fetch data from an external API (no worker needed)
fetch = HttpTask(task_ref_name='fetch_data', http_input={
'uri': 'https://api.example.com/records',
'method': 'GET',
'headers': {'Authorization': ['Bearer ${workflow.input.api_key}']},
})
# JQ Transform — reshape the response (no worker needed)
transform = JsonJQTask(
task_ref_name='transform',
script='.body.records | map({id: .id, value: .metrics.total})',
)
transform.input_parameters.update({
'records': fetch.output('response.body'),
})
# Custom worker — run business logic
enrich = enrich_records(
task_ref_name='enrich',
records=transform.output('result'),
)
# Wait — pause for 5 seconds before the next step
cooldown = WaitTask(task_ref_name='cooldown', wait_for_seconds=5)
# Custom worker — store results
store = save_to_database(task_ref_name='store', records=enrich.output('enriched'))
workflow >> fetch >> transform >> enrich >> cooldown >> store
workflow.output_parameters({'stored': store.output('count')})
workflow.register(overwrite=True)
```
{% endraw %}
---
### Sub-workflows
Break large workflows into reusable pieces. A parent workflow invokes child workflows as tasks.
```python
from conductor.client.workflow.conductor_workflow import ConductorWorkflow
from conductor.client.workflow.task.sub_workflow_task import SubWorkflowTask
# Child workflow (registered separately)
child = ConductorWorkflow(name='process_single_item', version=1, executor=executor)
validate = validate_item(task_ref_name='validate', item=child.input('item'))
transform = transform_item(task_ref_name='transform', item=validate.output('validated'))
child >> validate >> transform
child.output_parameters({'result': transform.output('transformed')})
child.register(overwrite=True)
# Parent workflow invokes the child
parent = ConductorWorkflow(name='batch_processor', version=1, executor=executor)
prepare = prepare_batch(task_ref_name='prepare', batch_id=parent.input('batch_id'))
run_child = SubWorkflowTask(
task_ref_name='process_item',
workflow_name='process_single_item',
version=1,
)
run_child.input_parameters.update({'item': prepare.output('first_item')})
aggregate = aggregate_results(
task_ref_name='aggregate',
result=run_child.output('result'),
)
parent >> prepare >> run_child >> aggregate
parent.register(overwrite=True)
```
---
### Runtime-generated dynamic workflow
Build a workflow definition at runtime and execute it without pre-registration. This runtime workflow pattern enables dynamic workflows where the task graph is generated on-the-fly — useful for AI agents, data pipelines, and any scenario where the steps are not known ahead of time.
{% raw %}
```python
from conductor.client.configuration.configuration import Configuration
from conductor.client.orkes_clients import OrkesClients
from conductor.client.http.models import StartWorkflowRequest
config = Configuration()
clients = OrkesClients(configuration=config)
executor = clients.get_workflow_executor()
# Build the workflow definition dynamically
steps = ['validate', 'enrich', 'store'] # determined at runtime
tasks = []
for i, step in enumerate(steps):
tasks.append({
'name': step,
'taskReferenceName': f'{step}_{i}',
'type': 'SIMPLE',
'inputParameters': {
'data': '${workflow.input.data}' if i == 0 else f'${{{steps[i-1]}_{i-1}.output.result}}',
},
})
# Start with inline definition — no pre-registration needed
request = StartWorkflowRequest(
name='dynamic_pipeline',
workflow_def={
'name': 'dynamic_pipeline',
'version': 1,
'tasks': tasks,
'outputParameters': {
'result': f'${{{steps[-1]}_{len(steps)-1}.output.result}}',
},
},
input={'data': {'key': 'value'}},
)
workflow_id = executor.start_workflow(request)
print(f'Started dynamic workflow: {workflow_id}')
```
{% endraw %}
This pattern is powerful for AI agents that generate execution plans at runtime — the LLM produces the list of steps, your code builds the workflow definition, and Conductor executes it with full durability, retries, and observability.
---
### Execute and wait for result
Run a workflow synchronously and get the result inline — useful for APIs and interactive applications.
```python
from conductor.client.configuration.configuration import Configuration
from conductor.client.orkes_clients import OrkesClients
config = Configuration()
clients = OrkesClients(configuration=config)
executor = clients.get_workflow_executor()
# Execute synchronously — blocks until the workflow completes
run = executor.execute(
name='order_fulfillment',
version=1,
workflow_input={'order_id': 'ORD-789'},
)
print(f'Status: {run.status}')
print(f'Output: {run.output}')
print(f'View: {config.ui_host}/execution/{run.workflow_id}')
```
---
## Setup
All examples above assume a `WorkflowExecutor` instance. Here is the standard setup:
```python
from conductor.client.configuration.configuration import Configuration
from conductor.client.orkes_clients import OrkesClients
config = Configuration() # reads CONDUCTOR_SERVER_URL from env
clients = OrkesClients(configuration=config)
executor = clients.get_workflow_executor()
```
```shell
pip install conductor-python
export CONDUCTOR_SERVER_URL=http://localhost:8080/api
```
For more Python SDK examples, see the [Python SDK documentation](../../documentation/clientsdks/python-sdk.md) and the [examples on GitHub](https://github.com/conductor-oss/python-sdk/tree/main/examples).
docs/devguide/cookbook/event-driven.md
+250
View File
@@ -0,0 +1,250 @@
---
description: "Conductor cookbook — event-driven workflow recipes for publishing to Kafka, NATS, RabbitMQ, SQS, triggering workflows from events, and completing tasks from external events."
---
# Event-driven recipes
### Publish events to Kafka, NATS, and RabbitMQ
Use the `EVENT` task type to publish messages. The `sink` field determines the destination.
**Kafka:**
```json
{
"name": "publish_to_kafka",
"taskReferenceName": "kafka_event",
"type": "EVENT",
"sink": "kafka:order-events",
"inputParameters": {
"orderId": "${workflow.input.orderId}",
"status": "PROCESSED"
}
}
```
**NATS:**
```json
{
"name": "publish_to_nats",
"taskReferenceName": "nats_event",
"type": "EVENT",
"sink": "nats:order-events",
"inputParameters": {
"orderId": "${workflow.input.orderId}",
"status": "PROCESSED"
}
}
```
**RabbitMQ (AMQP):**
```json
{
"name": "publish_to_rabbitmq",
"taskReferenceName": "amqp_event",
"type": "EVENT",
"sink": "amqp_exchange:order-events",
"inputParameters": {
"orderId": "${workflow.input.orderId}",
"status": "PROCESSED"
}
}
```
**Sink format reference:**
| Sink | Format |
|---|---|
| Kafka | `kafka:topic-name` |
| NATS | `nats:subject-name` |
| RabbitMQ queue | `amqp:queue-name` |
| RabbitMQ exchange | `amqp_exchange:exchange-name` |
| SQS | `sqs:queue-name` |
| Conductor internal | `conductor` |
---
### Listen for events to trigger workflows
Register event handlers to start workflows automatically when messages arrive on a queue or topic.
**Kafka event handler:**
```json
{
"name": "kafka_order_handler",
"event": "kafka:order-events",
"condition": "$.status == 'NEW'",
"actions": [
{
"action": "start_workflow",
"start_workflow": {
"name": "process_order",
"input": {
"orderId": "${orderId}",
"payload": "${$}"
}
}
}
],
"active": true
}
```
**NATS event handler:**
```json
{
"name": "nats_notification_handler",
"event": "nats:notifications",
"actions": [
{
"action": "start_workflow",
"start_workflow": {
"name": "handle_notification",
"input": { "data": "${$}" }
}
}
],
"active": true
}
```
**AMQP event handler:**
```json
{
"name": "amqp_task_handler",
"event": "amqp:task-queue",
"actions": [
{
"action": "start_workflow",
"start_workflow": {
"name": "process_task",
"input": { "taskData": "${$}" }
}
}
],
"active": true
}
```
**Register an event handler:**
```shell
curl -X POST 'http://localhost:8080/api/event' \
-H 'Content-Type: application/json' \
-d @handler.json
```
---
### Complete a task from an external event
Use a WAIT task to pause a workflow until an external system sends an event. An event handler listens for that event and completes the task, resuming the workflow.
**Workflow with WAIT task:**
```json
{
"name": "order_with_approval",
"version": 1,
"schemaVersion": 2,
"tasks": [
{
"name": "process_order",
"taskReferenceName": "process",
"type": "SIMPLE"
},
{
"name": "wait_for_approval",
"taskReferenceName": "approval_wait",
"type": "WAIT"
},
{
"name": "ship_order",
"taskReferenceName": "ship",
"type": "SIMPLE"
}
]
}
```
**Event handler to complete the WAIT task:**
```json
{
"name": "approval_event_handler",
"event": "kafka:approval-events",
"condition": "$.approved == true",
"actions": [
{
"action": "complete_task",
"complete_task": {
"workflowId": "${workflowId}",
"taskRefName": "approval_wait",
"output": {
"approvedBy": "${approvedBy}",
"approvedAt": "${timestamp}"
}
}
}
],
"active": true
}
```
When a message with `approved: true` arrives on the `approval-events` Kafka topic, the handler completes the WAIT task and the workflow continues to `ship_order`.
**Register both:**
```shell
# Register the workflow
curl -X POST 'http://localhost:8080/api/metadata/workflow' \
-H 'Content-Type: application/json' \
-d @order_with_approval.json
# Register the event handler
curl -X POST 'http://localhost:8080/api/event' \
-H 'Content-Type: application/json' \
-d @approval_event_handler.json
```
---
### Server configuration for event buses
Add the relevant properties to your `application.properties` to enable each event bus.
**Kafka:**
```properties
conductor.event-queues.kafka.enabled=true
conductor.event-queues.kafka.bootstrap-servers=kafka:9092
```
**NATS:**
```properties
conductor.event-queues.nats.enabled=true
conductor.event-queues.nats.url=nats://localhost:4222
```
**AMQP (RabbitMQ):**
```properties
conductor.event-queues.amqp.enabled=true
conductor.event-queues.amqp.hosts=rabbitmq
conductor.event-queues.amqp.port=5672
conductor.event-queues.amqp.username=guest
conductor.event-queues.amqp.password=guest
```
**SQS:**
```properties
conductor.event-queues.sqs.enabled=true
# Uses AWS default credential chain (env vars, IAM role, etc.)
```
docs/devguide/cookbook/files-api-usecase.md
+314
View File
@@ -0,0 +1,314 @@
# Conductor OSS — File Management Use Cases
Five real-world scenarios where Conductor orchestrates file creation, processing, and delivery across workflow stages.
---
## 1. Returns & Refund Document Processing
A customer initiates a product return. Conductor orchestrates the intake of return photos/documents, validates eligibility, generates an RMA (Return Merchandise Authorization) form, and produces the final refund receipt — all as a single traceable workflow.
### Workflow
```mermaid
flowchart TD
A["Customer Submits<br/>Return Request"] --> B["HTTP Task:<br/>Fetch Order Details"]
B --> C["INLINE Task:<br/>Validate Return Window"]
C --> D{"SWITCH:<br/>Eligible?"}
D -- No --> E["Generate Denial<br/>Letter PDF"]
E --> E1["Email Denial<br/>to Customer"]
D -- Yes --> F["HUMAN Task:<br/>Agent Reviews Photos"]
F --> G{"SWITCH:<br/>Condition Check"}
G -- Damaged --> H["Generate RMA Form<br/>+ Prepaid Shipping Label"]
G -- Wrong Item --> H
G -- Other --> I["HUMAN Task:<br/>Escalate to Supervisor"]
I --> H
H --> J["FORK"]
J --> K["Branch 1:<br/>Process Refund via<br/>Payment Gateway"]
J --> L["Branch 2:<br/>Generate Refund<br/>Receipt PDF"]
J --> M["Branch 3:<br/>Update Inventory<br/>System"]
K --> N["JOIN"]
L --> N
M --> N
N --> O["Email RMA + Receipt<br/>+ Shipping Label<br/>to Customer"]
O --> P["Archive All Docs<br/>to S3"]
style A fill:#4CAF50,color:#fff
style D fill:#FF9800,color:#fff
style G fill:#FF9800,color:#fff
style J fill:#2196F3,color:#fff
style N fill:#2196F3,color:#fff
style P fill:#9C27B0,color:#fff
```
### Files Produced
| Stage | File | Format |
|-------|------|--------|
| RMA Generation | `rma_RET-9001.pdf` | PDF |
| Shipping Label | `label_RET-9001.png` | 4×6 ZPL/PNG |
| Refund Receipt | `receipt_RET-9001.pdf` | PDF |
| Denial Letter | `denial_RET-9001.pdf` | PDF (if ineligible) |
### Conductor Primitives
SWITCH, HUMAN, FORK/JOIN, HTTP, INLINE, SUB_WORKFLOW
---
## 2. AI-Powered Knowledge Base Builder (RAG Pipeline)
An organization ingests documents (PDFs, Word files, web pages) into an AI-ready knowledge base. Conductor orchestrates crawling, extraction, chunking, embedding generation, and vector store indexing — enabling retrieval-augmented generation (RAG) for chatbots and search.
### Workflow
```mermaid
flowchart TD
A["Trigger:<br/>New Docs Uploaded<br/>to S3 Bucket"] --> B["DO_WHILE:<br/>Process Each Document"]
B --> C{"SWITCH:<br/>File Type?"}
C -- PDF --> D["Extract Text<br/>via Apache Tika"]
C -- DOCX --> E["Extract Text<br/>via python-docx"]
C -- HTML --> F["Scrape & Clean<br/>via BeautifulSoup"]
C -- Other --> G["OCR via<br/>Tesseract"]
D --> H["INLINE Task:<br/>Chunk Text<br/>(512 tokens, 50 overlap)"]
E --> H
F --> H
G --> H
H --> I["DYNAMIC_FORK:<br/>Generate Embeddings<br/>(1 per chunk)"]
I --> J["LLM_TEXT_COMPLETE:<br/>Create Embedding Vector"]
J --> K["JOIN:<br/>Collect All Vectors"]
K --> L["HTTP Task:<br/>Upsert to Vector DB<br/>(Pinecone / Weaviate)"]
L --> M["Generate Metadata<br/>Index JSON"]
M --> N{"More Docs?"}
N -- Yes --> B
N -- No --> O["Write Master<br/>Index Manifest"]
O --> P["Upload Manifest<br/>+ Logs to S3"]
style A fill:#4CAF50,color:#fff
style C fill:#FF9800,color:#fff
style I fill:#2196F3,color:#fff
style K fill:#2196F3,color:#fff
style N fill:#FF9800,color:#fff
style P fill:#9C27B0,color:#fff
```
### Files Produced
| Stage | File | Format |
|-------|------|--------|
| Extracted Text | `extracted_{doc_id}.txt` | Plain text |
| Chunk Manifest | `chunks_{doc_id}.jsonl` | JSONL |
| Embedding Vectors | `embeddings_{doc_id}.npy` | NumPy binary |
| Metadata Index | `index_{doc_id}.json` | JSON |
| Master Manifest | `kb_manifest_{run_id}.json` | JSON |
| Pipeline Log | `pipeline_log_{run_id}.txt` | Text |
### Conductor Primitives
DO_WHILE, SWITCH, DYNAMIC_FORK, LLM_TEXT_COMPLETE, HTTP, INLINE
---
## 3. Multi-Format Media Transcoding & Publishing
A media company uploads a master video file. Conductor fans out transcoding jobs to produce multiple resolutions and formats, generates thumbnails, extracts subtitles via speech-to-text, and publishes everything to a CDN — all in parallel where possible.
### Workflow
```mermaid
flowchart TD
A["Master Video<br/>Uploaded (4K ProRes)"] --> B["INLINE Task:<br/>Validate & Extract<br/>Media Metadata"]
B --> C["FORK (3 Branches)"]
C --> D["Branch 1:<br/>DYNAMIC_FORK<br/>Transcode Variants"]
D --> D1["1080p H.264 MP4"]
D --> D2["720p H.264 MP4"]
D --> D3["480p H.264 MP4"]
D --> D4["1080p WebM VP9"]
D --> D5["HLS Adaptive<br/>Playlist (.m3u8)"]
C --> E["Branch 2:<br/>Thumbnail Generation"]
E --> E1["Extract Keyframes<br/>(every 30s)"]
E1 --> E2["Resize to<br/>320×180 JPG"]
E2 --> E3["Generate Poster<br/>Image 1920×1080"]
C --> F["Branch 3:<br/>Speech-to-Text"]
F --> F1["LLM_TEXT_COMPLETE:<br/>Transcribe Audio"]
F1 --> F2["Generate SRT<br/>Subtitle File"]
F2 --> F3["Generate VTT<br/>Subtitle File"]
D1 --> G["JOIN"]
D2 --> G
D3 --> G
D4 --> G
D5 --> G
E3 --> G
F3 --> G
G --> H["Generate<br/>Manifest JSON"]
H --> I["HTTP Task:<br/>Upload All Assets<br/>to CDN"]
I --> J["HTTP Task:<br/>Update CMS<br/>with URLs"]
J --> K["Notify Editorial<br/>Team via Slack"]
style A fill:#4CAF50,color:#fff
style C fill:#2196F3,color:#fff
style D fill:#FF5722,color:#fff
style G fill:#2196F3,color:#fff
style K fill:#9C27B0,color:#fff
```
### Files Produced
| Stage | File | Format |
|-------|------|--------|
| Transcoded Videos | `video_{res}.mp4`, `video_1080p.webm` | MP4, WebM |
| HLS Playlist | `stream.m3u8` + segment `.ts` files | HLS |
| Thumbnails | `thumb_{timestamp}.jpg` | JPEG |
| Poster Image | `poster.jpg` | JPEG 1920×1080 |
| Subtitles | `subs_en.srt`, `subs_en.vtt` | SRT, VTT |
| Manifest | `publish_manifest.json` | JSON |
### Conductor Primitives
FORK/JOIN, DYNAMIC_FORK, LLM_TEXT_COMPLETE, HTTP, INLINE
---
## 4. Order Invoice, Packing Slip & Shipping Label Generation
An e-commerce order triggers Conductor to fetch order data, then fan out in parallel to generate three documents — a customer-facing invoice, a warehouse packing slip (no pricing), and a carrier shipping label — before bundling and distributing them.
### Workflow
```mermaid
flowchart TD
A["Order Placed<br/>(Webhook)"] --> B["HTTP Task:<br/>Fetch Order +<br/>Customer Profile"]
B --> C["INLINE Task:<br/>Calculate Totals<br/>(tax, discounts, shipping)"]
C --> D["FORK (3 Branches)"]
D --> E["Branch 1:<br/>Generate Invoice PDF"]
E --> E1["Apply Branding<br/>(logo, colors, footer)"]
E1 --> E2["Format Line Items<br/>+ Tax Breakdown"]
E2 --> E3["Render PDF<br/>invoice_ORD-12345.pdf"]
D --> F["Branch 2:<br/>Generate Packing Slip"]
F --> F1["Strip Pricing Info"]
F1 --> F2["Add Pick Locations<br/>+ Bin Numbers"]
F2 --> F3["Add Warehouse<br/>Barcode"]
F3 --> F4["Render PDF<br/>packslip_ORD-12345.pdf"]
D --> G["Branch 3:<br/>Generate Shipping Label"]
G --> G1{"SWITCH:<br/>Carrier?"}
G1 -- FedEx --> G2["Call FedEx API"]
G1 -- UPS --> G3["Call UPS API"]
G1 -- USPS --> G4["Call USPS API"]
G2 --> G5["Receive Tracking #<br/>+ Label Image"]
G3 --> G5
G4 --> G5
G5 --> G6["Render Label<br/>label_ORD-12345.png"]
E3 --> H["JOIN"]
F4 --> H
G6 --> H
H --> I["Bundle 3 Files<br/>into Order Package"]
I --> J["Upload to S3<br/>orders/ORD-12345/"]
J --> K["FORK (2 Branches)"]
K --> L["Email Invoice<br/>to Customer"]
K --> M["Send Slip + Label<br/>to Warehouse Printer"]
L --> N["JOIN"]
M --> N
N --> O["Update Order Status:<br/>Ready to Ship"]
style A fill:#4CAF50,color:#fff
style D fill:#2196F3,color:#fff
style G1 fill:#FF9800,color:#fff
style H fill:#2196F3,color:#fff
style K fill:#2196F3,color:#fff
style N fill:#2196F3,color:#fff
style O fill:#9C27B0,color:#fff
```
### Files Produced
| Stage | File | Format |
|-------|------|--------|
| Invoice | `invoice_ORD-12345.pdf` | PDF |
| Packing Slip | `packslip_ORD-12345.pdf` | PDF |
| Shipping Label | `label_ORD-12345.png` | 4×6 ZPL/PNG |
### Conductor Primitives
FORK/JOIN, SWITCH, HTTP, INLINE, SUB_WORKFLOW
---
## 5. Enterprise Video Surveillance Archival & Alert Pipeline
A network of security cameras streams footage to edge servers. Conductor orchestrates the pipeline: ingest video segments, run AI-based anomaly detection, generate alert clips with annotations, archive raw footage with retention policies, and produce daily summary reports.
### Workflow
```mermaid
flowchart TD
A["Camera Feed:<br/>60s Segment Arrives<br/>on Edge Server"] --> B["INLINE Task:<br/>Extract Metadata<br/>(camera ID, timestamp,<br/>resolution)"]
B --> C["Upload Raw Segment<br/>to Cold Storage<br/>(S3 Glacier)"]
C --> D["HTTP Task:<br/>AI Anomaly Detection<br/>Model Inference"]
D --> E{"SWITCH:<br/>Anomaly Detected?"}
E -- No --> F["Log: Normal<br/>Update Daily Counter"]
E -- Yes --> G["FORK (3 Branches)"]
G --> H["Branch 1:<br/>Clip 30s Around<br/>Anomaly Timestamp"]
H --> H1["Overlay Bounding<br/>Boxes + Labels"]
H1 --> H2["Render Alert Clip<br/>alert_CAM04_1712345678.mp4"]
G --> I["Branch 2:<br/>Generate Alert<br/>Snapshot"]
I --> I1["Extract Best Frame"]
I1 --> I2["Annotate with<br/>Detection Metadata"]
I2 --> I3["Save Snapshot<br/>alert_CAM04_1712345678.jpg"]
G --> J["Branch 3:<br/>Create Incident<br/>Report"]
J --> J1["LLM_TEXT_COMPLETE:<br/>Summarize Event"]
J1 --> J2["Generate PDF<br/>incident_1712345678.pdf"]
H2 --> K["JOIN"]
I3 --> K
J2 --> K
K --> L["Upload Alert Bundle<br/>to Hot Storage (S3)"]
L --> M["HTTP Task:<br/>Push Notification<br/>to Security Team"]
M --> N["Log Incident<br/>to SIEM"]
F --> O["TIMER:<br/>End of Day?"]
N --> O
O --> P["DO_WHILE:<br/>Aggregate All<br/>Camera Logs"]
P --> Q["Generate Daily<br/>Summary Report PDF"]
Q --> R["Apply Retention<br/>Policy (90-day<br/>hot → cold → delete)"]
R --> S["Email Daily Report<br/>to Facility Manager"]
style A fill:#4CAF50,color:#fff
style E fill:#FF9800,color:#fff
style G fill:#2196F3,color:#fff
style K fill:#2196F3,color:#fff
style O fill:#FF5722,color:#fff
style S fill:#9C27B0,color:#fff
```
### Files Produced
| Stage | File | Format |
|-------|------|--------|
| Raw Segment | `raw_CAM04_1712345678.mp4` | MP4 (60s) |
| Alert Clip | `alert_CAM04_1712345678.mp4` | MP4 (30s, annotated) |
| Alert Snapshot | `alert_CAM04_1712345678.jpg` | JPEG (annotated) |
| Incident Report | `incident_1712345678.pdf` | PDF |
| Daily Summary | `daily_report_2026-04-08.pdf` | PDF |
### Conductor Primitives
FORK/JOIN, SWITCH, DO_WHILE, TIMER, LLM_TEXT_COMPLETE, HTTP, INLINE
---
*Generated for Conductor OSS file management use case exploration.*
docs/devguide/cookbook/index.md
+43
View File
@@ -0,0 +1,43 @@
---
description: "Conductor cookbook — copy-paste workflow orchestration recipes for microservice orchestration, dynamic parallelism, event-driven patterns, AI agent orchestration, LLM orchestration, workflow automation, and RAG pipelines."
---
# Cookbook
Production-ready workflow recipes. Each recipe includes the complete JSON workflow definition and commands to register and run it.
<div class="grid cards" markdown>
- **[Microservice orchestration](microservice-orchestration.md)**
HTTP service chains, conditional branching, parallel HTTP calls with Fork/Join.
- **[Dynamic parallelism](dynamic-parallelism.md)**
Dynamic forks — different tasks per branch, fan-out with same task, parallel sub-workflows.
- **[Wait and timer patterns](wait-and-timers.md)**
Fixed delays, scheduled execution, external signals, and human-in-the-loop approvals.
- **[Task timeouts and retries](task-timeouts-and-retries.md)**
Exponential backoff with cap and jitter, lease extension for long-running workers, hard SLA with totalTimeoutSeconds, and thundering herd prevention.
- **[Scheduled workflows](workflow-scheduling.md)**
Cron-triggered execution, catchup after downtime, bounded time windows, input parameterization, and concurrent execution handling.
- **[Event-driven recipes](event-driven.md)**
Publish to Kafka/NATS/RabbitMQ/SQS, event handlers to trigger workflows, complete tasks from events.
- **[AI & LLM orchestration recipes](ai-llm.md)**
Chat completion, RAG pipelines, MCP agents with function calling, image generation, LLM-to-PDF, and provider configuration.
- **[Dynamic workflows as code](dynamic-workflows.md)**
Workflow as code in Python — sequential chains, conditional branching, parallel execution, loops, sub-workflows, and runtime-generated definitions.
</div>
docs/devguide/cookbook/microservice-orchestration.md
+256
View File
@@ -0,0 +1,256 @@
---
description: "Conductor cookbook — microservice orchestration recipes with HTTP service chains, conditional branching, and parallel HTTP calls using Fork/Join."
---
# Microservice orchestration
### HTTP service chain
A common pattern: call a series of HTTP endpoints where each step uses output from the previous one. No custom workers needed — Conductor handles it with built-in HTTP tasks.
```json
{
"name": "order_processing",
"description": "Validate order, charge payment, reserve inventory, send confirmation",
"version": 1,
"schemaVersion": 2,
"inputParameters": ["orderId", "customerId", "amount", "items"],
"tasks": [
{
"name": "validate_order",
"taskReferenceName": "validate",
"type": "HTTP",
"inputParameters": {
"http_request": {
"uri": "https://api.example.com/orders/${workflow.input.orderId}/validate",
"method": "POST",
"body": {
"customerId": "${workflow.input.customerId}",
"items": "${workflow.input.items}"
},
"connectionTimeOut": 5000,
"readTimeOut": 5000
}
}
},
{
"name": "charge_payment",
"taskReferenceName": "payment",
"type": "HTTP",
"inputParameters": {
"http_request": {
"uri": "https://api.example.com/payments/charge",
"method": "POST",
"body": {
"orderId": "${workflow.input.orderId}",
"amount": "${workflow.input.amount}",
"customerId": "${workflow.input.customerId}"
},
"connectionTimeOut": 10000,
"readTimeOut": 10000
}
}
},
{
"name": "reserve_inventory",
"taskReferenceName": "inventory",
"type": "HTTP",
"inputParameters": {
"http_request": {
"uri": "https://api.example.com/inventory/reserve",
"method": "POST",
"body": {
"orderId": "${workflow.input.orderId}",
"items": "${workflow.input.items}",
"paymentId": "${payment.output.response.body.paymentId}"
},
"connectionTimeOut": 5000,
"readTimeOut": 5000
}
}
},
{
"name": "send_confirmation",
"taskReferenceName": "notify",
"type": "HTTP",
"inputParameters": {
"http_request": {
"uri": "https://api.example.com/notifications/send",
"method": "POST",
"body": {
"customerId": "${workflow.input.customerId}",
"orderId": "${workflow.input.orderId}",
"paymentId": "${payment.output.response.body.paymentId}",
"reservationId": "${inventory.output.response.body.reservationId}"
}
}
}
}
],
"outputParameters": {
"paymentId": "${payment.output.response.body.paymentId}",
"reservationId": "${inventory.output.response.body.reservationId}"
},
"failureWorkflow": "order_compensation",
"timeoutPolicy": "TIME_OUT_WF",
"timeoutSeconds": 120
}
```
Each task passes data forward using `${taskReferenceName.output.response.body.field}` expressions. If any step fails, Conductor retries it (configurable) and can trigger the `failureWorkflow` for compensation.
**Register and run:**
```shell
curl -X POST 'http://localhost:8080/api/metadata/workflow' \
-H 'Content-Type: application/json' \
-d @order_processing.json
curl -X POST 'http://localhost:8080/api/workflow/order_processing' \
-H 'Content-Type: application/json' \
-d '{"orderId": "ORD-123", "customerId": "CUST-456", "amount": 99.99, "items": ["SKU-A", "SKU-B"]}'
```
---
### HTTP with conditional branching
Use a SWITCH operator to route workflow execution based on a previous task's output.
```json
{
"name": "user_onboarding",
"version": 1,
"schemaVersion": 2,
"inputParameters": ["userId"],
"tasks": [
{
"name": "get_user_profile",
"taskReferenceName": "profile",
"type": "HTTP",
"inputParameters": {
"http_request": {
"uri": "https://api.example.com/users/${workflow.input.userId}",
"method": "GET"
}
}
},
{
"name": "route_by_tier",
"taskReferenceName": "tier_switch",
"type": "SWITCH",
"evaluatorType": "javascript",
"expression": "$.tier == 'enterprise' ? 'enterprise' : 'standard'",
"inputParameters": {
"tier": "${profile.output.response.body.tier}"
},
"decisionCases": {
"enterprise": [
{
"name": "assign_account_manager",
"taskReferenceName": "assign_am",
"type": "HTTP",
"inputParameters": {
"http_request": {
"uri": "https://api.example.com/account-managers/assign",
"method": "POST",
"body": {"userId": "${workflow.input.userId}"}
}
}
}
],
"standard": [
{
"name": "send_welcome_email",
"taskReferenceName": "welcome",
"type": "HTTP",
"inputParameters": {
"http_request": {
"uri": "https://api.example.com/emails/welcome",
"method": "POST",
"body": {"userId": "${workflow.input.userId}"}
}
}
}
]
}
}
]
}
```
---
### Parallel HTTP calls with Fork/Join
When tasks are independent, run them in parallel with a static fork.
```json
{
"name": "enrich_customer_data",
"version": 1,
"schemaVersion": 2,
"inputParameters": ["customerId"],
"tasks": [
{
"name": "parallel_enrichment",
"taskReferenceName": "fork",
"type": "FORK_JOIN",
"forkTasks": [
[
{
"name": "get_credit_score",
"taskReferenceName": "credit",
"type": "HTTP",
"inputParameters": {
"http_request": {
"uri": "https://api.example.com/credit/${workflow.input.customerId}",
"method": "GET"
}
}
}
],
[
{
"name": "get_purchase_history",
"taskReferenceName": "purchases",
"type": "HTTP",
"inputParameters": {
"http_request": {
"uri": "https://api.example.com/purchases/${workflow.input.customerId}",
"method": "GET"
}
}
}
],
[
{
"name": "get_support_tickets",
"taskReferenceName": "tickets",
"type": "HTTP",
"inputParameters": {
"http_request": {
"uri": "https://api.example.com/support/${workflow.input.customerId}",
"method": "GET"
}
}
}
]
]
},
{
"name": "join_results",
"taskReferenceName": "join",
"type": "JOIN",
"joinOn": ["credit", "purchases", "tickets"]
}
],
"outputParameters": {
"creditScore": "${credit.output.response.body}",
"purchases": "${purchases.output.response.body}",
"tickets": "${tickets.output.response.body}"
}
}
```
All three HTTP calls execute simultaneously. The JOIN waits for all to complete before the workflow continues.
docs/devguide/cookbook/task-timeouts-and-retries.md
+187
View File
@@ -0,0 +1,187 @@
---
description: "Conductor cookbook — task timeout and retry recipes covering responseTimeout with lease extension, totalTimeoutSeconds, exponential backoff with cap and jitter, and thundering herd prevention."
---
# Task timeouts and retries
Practical recipes for making workers resilient. Each recipe is a complete task definition you can register with `POST /api/metadata/taskdefs`.
---
### Exponential backoff with a cap
Retries with exponential backoff for a task that calls an external API. The cap prevents the delay from growing indefinitely; jitter prevents multiple failing workers from hammering the API at the same time.
```json
{
"name": "call_payment_api",
"ownerEmail": "payments@example.com",
"retryCount": 6,
"retryLogic": "EXPONENTIAL_BACKOFF",
"retryDelaySeconds": 2,
"maxRetryDelaySeconds": 60,
"backoffJitterMs": 3000,
"responseTimeoutSeconds": 30,
"timeoutSeconds": 600,
"timeoutPolicy": "RETRY"
}
```
**Delay schedule** (`retryDelaySeconds=2`, `maxRetryDelaySeconds=60`, `backoffJitterMs=3000`):
| Attempt | Base delay | After cap | Actual range |
| :--- | :--- | :--- | :--- |
| 1 | 2s | 2s | 2.0 – 5.0s |
| 2 | 4s | 4s | 4.0 – 7.0s |
| 3 | 8s | 8s | 8.0 – 11.0s |
| 4 | 16s | 16s | 16.0 – 19.0s |
| 5 | 32s | 32s | 32.0 – 35.0s |
| 6 | 64s | **60s** | 60.0 – 63.0s |
---
### Lease extension for long-running workers
`responseTimeoutSeconds` is the heartbeat window: if the worker doesn't report back within this duration, Conductor marks the task `TIMED_OUT` and retries it. For tasks that take longer than the heartbeat window, workers extend the lease by posting an `IN_PROGRESS` update with `callbackAfterSeconds`.
**Task definition**
```json
{
"name": "transcode_video",
"ownerEmail": "media@example.com",
"retryCount": 2,
"retryLogic": "FIXED",
"retryDelaySeconds": 10,
"responseTimeoutSeconds": 30,
"timeoutSeconds": 3600,
"timeoutPolicy": "RETRY"
}
```
`responseTimeoutSeconds: 30` — Conductor will reschedule the task if the worker is silent for 30 seconds.
`timeoutSeconds: 3600` — the task itself can take up to 1 hour across all heartbeats.
**Worker: extend the lease every 25 seconds**
```python
import time
from conductor.client.http.models import TaskResult
def transcode_video(task):
task_id = task.task_id
workflow_id = task.workflow_instance_id
for chunk in video_chunks(task.input_data["file_url"]):
transcode_chunk(chunk)
# Extend the lease before responseTimeoutSeconds (30s) expires.
# callbackAfterSeconds tells Conductor to leave this task invisible
# in the queue for another 25s — resetting the response clock.
heartbeat = TaskResult(
task_id=task_id,
workflow_instance_id=workflow_id,
status="IN_PROGRESS",
callback_after_seconds=25,
output_data={"progress": chunk.index / len(video_chunks)}
)
conductor_client.update_task(heartbeat)
return TaskResult(
task_id=task_id,
workflow_instance_id=workflow_id,
status="COMPLETED",
output_data={"output_url": upload_result.url}
)
```
**What happens without a heartbeat:**
```
t=0s Worker polls task → IN_PROGRESS
t=30s responseTimeoutSeconds expires → TIMED_OUT → retry scheduled
t=40s Worker finishes (too late, task already terminated)
```
**What happens with a heartbeat every 25s:**
```
t=0s Worker polls task → IN_PROGRESS
t=25s Worker: POST IN_PROGRESS, callbackAfterSeconds=25 → clock resets
t=50s Worker: POST IN_PROGRESS, callbackAfterSeconds=25 → clock resets
...
t=90s Worker: POST COMPLETED → task done
```
---
### Hard SLA with `totalTimeoutSeconds`
Use `totalTimeoutSeconds` when you need a guaranteed upper bound on how long a task can take across all of its retries. This is independent of `retryCount` — whichever limit is hit first wins.
```json
{
"name": "sync_crm_record",
"ownerEmail": "crm@example.com",
"retryCount": 20,
"retryLogic": "FIXED",
"retryDelaySeconds": 5,
"totalTimeoutSeconds": 120,
"responseTimeoutSeconds": 15,
"timeoutPolicy": "TIME_OUT_WF"
}
```
`retryCount: 20` — would normally allow 20 retries.
`totalTimeoutSeconds: 120` — but if the 2-minute wall-clock budget is consumed first, no more retries are queued and the workflow is failed.
This is useful for SLA-sensitive tasks where you need to know that, regardless of transient failures, the workflow will either succeed or surface as failed within a bounded time window.
**Timeline example** (`retryDelaySeconds=5`, `totalTimeoutSeconds=30`):
```
t=0s Attempt 1 → FAILED
t=5s Attempt 2 → FAILED
t=10s Attempt 3 → FAILED
t=15s Attempt 4 → FAILED
t=20s Attempt 5 → FAILED
t=25s Attempt 6 → FAILED
t=30s totalTimeoutSeconds exceeded → workflow FAILED, no more retries
(10 retries still remained in retryCount)
```
---
### Thundering herd prevention
When hundreds of tasks fail simultaneously (e.g., a downstream service goes down), all retries are scheduled at the same time. Without jitter, they all hit the recovering service at once. `backoffJitterMs` spreads them across a time window.
```json
{
"name": "send_webhook",
"ownerEmail": "platform@example.com",
"retryCount": 5,
"retryLogic": "EXPONENTIAL_BACKOFF",
"retryDelaySeconds": 1,
"maxRetryDelaySeconds": 30,
"backoffJitterMs": 5000,
"responseTimeoutSeconds": 10,
"concurrentExecLimit": 200
}
```
With `backoffJitterMs: 5000`, 500 tasks that all fail at `t=0` will retry at uniformly random times between `t=1s` and `t=6s` — spreading the retry load across 5 seconds instead of hitting the service in a single burst.
---
### Choosing the right combination
| Scenario | Recommended config |
| :--- | :--- |
| External API with rate limits | `EXPONENTIAL_BACKOFF` + `maxRetryDelaySeconds` + `backoffJitterMs` |
| Long-running processing job | `responseTimeoutSeconds` (short) + heartbeats from worker + `timeoutSeconds` (long) |
| SLA-bounded task | `totalTimeoutSeconds` + `FIXED` or `EXPONENTIAL_BACKOFF` |
| High fan-out with many concurrent failures | `backoffJitterMs` + `concurrentExecLimit` |
| Non-retryable error | Return `FAILED_WITH_TERMINAL_ERROR` from the worker |
See the [Task Definition reference](../../documentation/configuration/taskdef.md) for all available parameters.
docs/devguide/cookbook/wait-and-timers.md
+152
View File
@@ -0,0 +1,152 @@
---
description: "Conductor cookbook — wait and timer pattern recipes for fixed delays, scheduled execution, external signals, and human-in-the-loop approvals."
---
# Wait and timer patterns
### Wait for a fixed delay
Introduce a delay between workflow steps — useful for rate limiting, cool-down periods, or retry backoff.
```json
{
"name": "delayed_notification",
"version": 1,
"schemaVersion": 2,
"tasks": [
{
"name": "process_event",
"taskReferenceName": "process",
"type": "SIMPLE"
},
{
"name": "wait_before_retry",
"taskReferenceName": "cooldown",
"type": "WAIT",
"inputParameters": {
"duration": "5 minutes"
}
},
{
"name": "send_notification",
"taskReferenceName": "notify",
"type": "HTTP",
"inputParameters": {
"uri": "https://api.example.com/notify",
"method": "POST",
"body": {"eventId": "${process.output.eventId}"}
}
}
]
}
```
The `duration` field supports human-readable formats: `30 seconds`, `5 minutes`, `2 hours`, `1 days`, or short forms like `30s`, `5m`, `2h`, `1d`. You can also combine them: `2 hours 30 minutes`.
---
### Wait until a specific time
Schedule workflow continuation for a specific date/time — useful for scheduled releases, SLA deadlines, or business-hours processing.
```json
{
"name": "scheduled_report",
"version": 1,
"schemaVersion": 2,
"inputParameters": ["reportDate"],
"tasks": [
{
"name": "prepare_report",
"taskReferenceName": "prepare",
"type": "SIMPLE"
},
{
"name": "wait_until_publish_time",
"taskReferenceName": "schedule_wait",
"type": "WAIT",
"inputParameters": {
"until": "${workflow.input.reportDate}"
}
},
{
"name": "publish_report",
"taskReferenceName": "publish",
"type": "HTTP",
"inputParameters": {
"uri": "https://api.example.com/reports/publish",
"method": "POST",
"body": {"reportId": "${prepare.output.reportId}"}
}
}
]
}
```
The `until` field supports formats: `yyyy-MM-dd HH:mm z` (e.g., `2025-06-15 09:00 GMT+00:00`), `yyyy-MM-dd HH:mm`, or `yyyy-MM-dd`.
**Register and run:**
```shell
curl -X POST 'http://localhost:8080/api/metadata/workflow' \
-H 'Content-Type: application/json' \
-d @scheduled_report.json
curl -X POST 'http://localhost:8080/api/workflow/scheduled_report' \
-H 'Content-Type: application/json' \
-d '{"reportDate": "2025-06-15 09:00 GMT+00:00"}'
```
---
### Wait for an external signal
Pause a workflow until an external system (or human) completes the task via API — useful for approvals, manual QA, or third-party callbacks.
```json
{
"name": "order_with_manual_approval",
"version": 1,
"schemaVersion": 2,
"inputParameters": ["orderId", "amount"],
"tasks": [
{
"name": "validate_order",
"taskReferenceName": "validate",
"type": "HTTP",
"inputParameters": {
"uri": "https://api.example.com/orders/${workflow.input.orderId}/validate",
"method": "GET"
}
},
{
"name": "wait_for_approval",
"taskReferenceName": "approval",
"type": "WAIT"
},
{
"name": "fulfill_order",
"taskReferenceName": "fulfill",
"type": "HTTP",
"inputParameters": {
"uri": "https://api.example.com/orders/${workflow.input.orderId}/fulfill",
"method": "POST",
"body": {
"approvedBy": "${approval.output.approvedBy}"
}
}
}
]
}
```
Complete the WAIT task externally (e.g., from a UI or webhook):
```shell
# Complete the wait task and resume the workflow
curl -X POST 'http://localhost:8080/api/tasks/{workflowId}/approval/COMPLETED/sync' \
-H 'Content-Type: application/json' \
-d '{"approvedBy": "manager@example.com"}'
```
The output data you pass when completing the task is available in subsequent tasks via `${approval.output.approvedBy}`.
docs/devguide/cookbook/workflow-scheduling.md
+364
View File
@@ -0,0 +1,364 @@
---
description: "Conductor cookbook — scheduled workflow recipes for cron-triggered execution, catchup after downtime, bounded time windows, parallel scheduled tasks, input parameterization, and concurrent execution handling."
---
# Scheduled workflow recipes
### Run a workflow every minute
The simplest schedule — trigger a workflow on a fixed interval.
```json
{
"name": "every-minute-demo-schedule",
"cronExpression": "0 * * * * *",
"zoneId": "UTC",
"startWorkflowRequest": {
"name": "daily_report_workflow",
"version": 1,
"correlationId": "demo-${scheduledTime}"
},
"runCatchupScheduleInstances": false,
"paused": false
}
```
The workflow:
```json
{
"name": "daily_report_workflow",
"version": 1,
"schemaVersion": 2,
"tasks": [
{
"name": "fetch_report_data",
"taskReferenceName": "fetch_report_data_ref",
"type": "HTTP",
"inputParameters": {
"http_request": {
"uri": "https://jsonplaceholder.typicode.com/todos?userId=1",
"method": "GET",
"connectionTimeOut": 3000,
"readTimeOut": 3000
}
}
}
],
"outputParameters": {
"statusCode": "${fetch_report_data_ref.output.response.statusCode}",
"itemCount": "${fetch_report_data_ref.output.response.body.length()}"
},
"timeoutPolicy": "TIME_OUT_WF",
"timeoutSeconds": 120
}
```
**Register and schedule:**
```shell
# Register workflow
curl -X PUT 'http://localhost:8080/api/metadata/workflow' \
-H 'Content-Type: application/json' \
-d @daily-report-workflow.json
# Create schedule
curl -X POST 'http://localhost:8080/api/scheduler/schedules' \
-H 'Content-Type: application/json' \
-d @every-minute-schedule.json
# Watch executions
curl 'http://localhost:8080/api/scheduler/search/executions?freeText=every-minute-demo-schedule&size=10'
```
---
### Weekday business-hours schedule
Trigger a report workflow at 9 AM Eastern on weekdays only.
```json
{
"name": "daily-report-schedule",
"cronExpression": "0 0 9 * * MON-FRI",
"zoneId": "America/New_York",
"startWorkflowRequest": {
"name": "daily_report_workflow",
"version": 1,
"correlationId": "daily-report-${scheduledTime}"
},
"runCatchupScheduleInstances": false,
"paused": false
}
```
The `zoneId` ensures the schedule respects daylight saving time transitions.
---
### Catch up missed executions after downtime
When the scheduler restarts after being offline, `runCatchupScheduleInstances: true` fires all missed cron slots. Use this for workflows where every execution matters (billing, compliance, ETL).
```json
{
"name": "catchup-demo-schedule",
"cronExpression": "0 * * * * *",
"zoneId": "UTC",
"runCatchupScheduleInstances": true,
"paused": false,
"startWorkflowRequest": {
"name": "catchup_demo_workflow",
"version": 1,
"input": {}
}
}
```
If the scheduler was down for 5 minutes, it will fire 5 workflow executions on restart — one per missed minute.
!!! warning
Catchup executions fire in rapid succession. Make sure your workflow and downstream systems can handle the burst.
---
### Bounded schedule with a time window
Restrict a schedule to fire only within a time window using `scheduleStartTime` and `scheduleEndTime` (epoch milliseconds).
```shell
# Compute a 5-minute window starting now
START_MS=$(date +%s000)
END_MS=$(( $(date +%s) + 300 ))000
curl -X POST 'http://localhost:8080/api/scheduler/schedules' \
-H 'Content-Type: application/json' \
-d "{
\"name\": \"bounded-demo-schedule\",
\"cronExpression\": \"0 * * * * *\",
\"zoneId\": \"UTC\",
\"scheduleStartTime\": $START_MS,
\"scheduleEndTime\": $END_MS,
\"startWorkflowRequest\": {
\"name\": \"bounded_demo_workflow\",
\"version\": 1,
\"input\": {}
}
}"
```
The schedule fires every minute but only within the 5-minute window, then stops automatically.
---
### Pass input parameters to scheduled workflows
The scheduler automatically injects `_scheduledTime` and `_executedTime` into every execution. You can also provide static input that gets merged:
Schedule definition:
```json
{
"name": "input-param-demo-schedule",
"cronExpression": "0 * * * * *",
"zoneId": "UTC",
"startWorkflowRequest": {
"name": "input_param_demo_workflow",
"version": 1,
"input": {
"reportOwner": "platform-team",
"alertThreshold": 100
}
}
}
```
Workflow that uses the injected timestamps to compute a 24-hour report window:
```json
{
"name": "input_param_demo_workflow",
"version": 1,
"schemaVersion": 2,
"tasks": [
{
"name": "compute_report_window",
"taskReferenceName": "compute_report_window",
"type": "INLINE",
"inputParameters": {
"scheduledTime": "${workflow.input._scheduledTime}",
"executionTime": "${workflow.input._executedTime}",
"evaluatorType": "javascript",
"expression": "function toISO(ms) { return new Date(ms).toISOString(); } ({ reportWindowStart: toISO($.scheduledTime - 86400000), reportWindowEnd: toISO($.scheduledTime), scheduledAt: toISO($.scheduledTime), triggeredAt: toISO($.executionTime) })"
}
}
],
"outputParameters": {
"reportWindowStart": "${compute_report_window.output.result.reportWindowStart}",
"reportWindowEnd": "${compute_report_window.output.result.reportWindowEnd}",
"scheduledAt": "${compute_report_window.output.result.scheduledAt}",
"triggeredAt": "${compute_report_window.output.result.triggeredAt}"
},
"timeoutPolicy": "ALERT_ONLY",
"timeoutSeconds": 30
}
```
---
### Schedule a parallel (FORK/JOIN) workflow
A scheduled workflow can use any Conductor construct. This example fetches two timezones in parallel using FORK_JOIN:
```json
{
"name": "multistep_demo_workflow",
"version": 3,
"schemaVersion": 2,
"tasks": [
{
"name": "fork_parallel_calls",
"taskReferenceName": "fork_parallel_calls",
"type": "FORK_JOIN",
"forkTasks": [
[
{
"name": "fetch_utc_time",
"taskReferenceName": "fetch_utc_time",
"type": "HTTP",
"inputParameters": {
"http_request": {
"uri": "https://timeapi.io/api/time/current/zone?timeZone=UTC",
"method": "GET"
}
}
}
],
[
{
"name": "fetch_ny_time",
"taskReferenceName": "fetch_ny_time",
"type": "HTTP",
"inputParameters": {
"http_request": {
"uri": "https://timeapi.io/api/time/current/zone?timeZone=America/New_York",
"method": "GET"
}
}
}
]
]
},
{
"name": "join_results",
"taskReferenceName": "join_results",
"type": "JOIN",
"joinOn": ["fetch_utc_time", "fetch_ny_time"]
}
],
"outputParameters": {
"utcTime": "${fetch_utc_time.output.response.body.dateTime}",
"newYorkTime": "${fetch_ny_time.output.response.body.dateTime}"
},
"timeoutPolicy": "ALERT_ONLY",
"timeoutSeconds": 60
}
```
Schedule it:
```json
{
"name": "multistep-demo-schedule",
"cronExpression": "0 * * * * *",
"zoneId": "UTC",
"startWorkflowRequest": {
"name": "multistep_demo_workflow",
"version": 3
}
}
```
---
### Handle concurrent executions
The scheduler fires on every cron tick regardless of whether the previous execution has completed. If a workflow takes 90 seconds and the schedule fires every 60 seconds, executions will overlap:
```json
{
"name": "concurrent_demo_workflow",
"version": 1,
"schemaVersion": 2,
"tasks": [
{
"name": "fetch_start_time",
"taskReferenceName": "fetch_start_time",
"type": "HTTP",
"inputParameters": {
"http_request": {
"uri": "https://timeapi.io/api/time/current/zone?timeZone=UTC",
"method": "GET"
}
}
},
{
"name": "wait_90s",
"taskReferenceName": "wait_90s",
"type": "WAIT",
"inputParameters": { "duration": "90s" }
},
{
"name": "fetch_end_time",
"taskReferenceName": "fetch_end_time",
"type": "HTTP",
"inputParameters": {
"http_request": {
"uri": "https://timeapi.io/api/time/current/zone?timeZone=UTC",
"method": "GET"
}
}
}
],
"outputParameters": {
"startedAt": "${fetch_start_time.output.response.body.dateTime}",
"finishedAt": "${fetch_end_time.output.response.body.dateTime}"
},
"timeoutPolicy": "ALERT_ONLY",
"timeoutSeconds": 300
}
```
!!! note "Design for overlap"
If concurrent runs are a problem, either increase the cron interval so it exceeds the workflow duration, or make your workflow idempotent so overlapping runs don't produce duplicate side effects.
---
### Manage a schedule lifecycle
Complete lifecycle in one session — create, verify, pause, resume, delete:
```shell
# Create
curl -X POST 'http://localhost:8080/api/scheduler/schedules' \
-H 'Content-Type: application/json' \
-d @daily-report-schedule.json
# Preview next 5 execution times
curl 'http://localhost:8080/api/scheduler/nextFewSchedules?cronExpression=0+0+9+*+*+MON-FRI&limit=5'
# Check execution history
curl 'http://localhost:8080/api/scheduler/search/executions?freeText=daily-report-schedule&size=10'
# Pause
curl -X PUT 'http://localhost:8080/api/scheduler/schedules/daily-report-schedule/pause?reason=maintenance'
# Verify paused state
curl 'http://localhost:8080/api/scheduler/schedules/daily-report-schedule'
# Resume
curl -X PUT 'http://localhost:8080/api/scheduler/schedules/daily-report-schedule/resume'
# Delete
curl -X DELETE 'http://localhost:8080/api/scheduler/schedules/daily-report-schedule'
```
docs/devguide/faq.md
+199
View File
@@ -0,0 +1,199 @@
---
description: "Frequently asked questions about Conductor — open source workflow engine, self-hosted deployment, AI agent orchestration, LLM orchestration, workflow automation, durable execution, microservice orchestration, saga pattern, scaling, and how Conductor compares to Temporal, Airflow, and Step Functions."
---
# Frequently Asked Questions
## General
### Is Conductor open source?
Yes. Conductor is a fully open source workflow engine, released under the Apache 2.0 license. You can self-host it on your own infrastructure — there is no vendor lock-in, no proprietary runtime, and no cloud dependency. The self-hosted workflow engine supports 5 persistence backends, 6 message brokers, and runs anywhere Docker or a JVM runs.
### Is this the same as Netflix Conductor?
Yes. Conductor OSS is the continuation of the original Netflix Conductor repository after Netflix contributed the project to the open-source foundation.
### Is Netflix Conductor abandoned?
No. The original Netflix repository has transitioned to Conductor OSS, which is the new home for the project. Active development and maintenance continues here.
### Is this project actively maintained?
Yes. Orkes is the primary maintainer of this repository and offers an enterprise SaaS platform for Conductor across all major cloud providers.
### Is Orkes Conductor compatible with Conductor OSS?
100% compatible. Orkes Conductor is built on top of Conductor OSS, ensuring full compatibility between the open-source version and the enterprise offering.
### Are workflows always asynchronous?
No. While Conductor excels at asynchronous orchestration, it also supports synchronous workflow execution when immediate results are required.
### Do I need to use a Conductor-specific framework?
Not at all. Conductor is language and framework agnostic. Use your preferred language and framework — SDKs provide native integration for Java, Python, JavaScript, Go, C#, and more.
### Is Conductor a low-code/no-code platform?
No. Conductor is designed for developers who write code. While workflows can be defined in JSON, the power comes from building workers and tasks in your preferred programming language.
### Can Conductor handle complex workflows?
Yes. Conductor supports advanced patterns including nested loops, dynamic branching, sub-workflows, and workflows with thousands of tasks.
## How does Conductor compare to other workflow engines?
Conductor combines durable execution, 14+ native LLM providers, JSON-native workflow definitions, 7+ language SDKs, and battle-tested scale (Netflix, Tesla, LinkedIn, JP Morgan). It's the only open source workflow engine with native AI/LLM task types, MCP integration, and built-in vector database support.
### Isn't JSON too limited for complex workflows?
No — JSON makes workflows *more* capable, not less. A JSON workflow definition is pure orchestration: it describes what runs, in what order, with what inputs. It cannot open connections, mutate state, or produce side effects. This means every execution is deterministic by construction — given the same inputs, the same task graph executes every time. That is why replay, restart, and retry work unconditionally.
Code-based workflow engines embed orchestration logic alongside business logic, which means your workflow code can introduce non-determinism (system clocks, random values, uncontrolled I/O). These engines must impose restrictions on what your code is allowed to do — and bugs from violating those restrictions are subtle and hard to debug.
Conductor's dynamic primitives — [DYNAMIC tasks](../documentation/configuration/workflowdef/operators/dynamic-task.md), [DYNAMIC_FORK](../documentation/configuration/workflowdef/operators/dynamic-fork-task.md), and [dynamic sub-workflows](../documentation/configuration/workflowdef/operators/sub-workflow-task.md) — provide more runtime flexibility than code-based definitions. An LLM can generate a complete workflow definition as JSON and Conductor executes it immediately, with full durability and observability. No code generation, no compilation, no deployment. See [JSON + Code Native](../architecture/json-native.md) for the full picture.
### How is Conductor different from Temporal?
Both are durable execution engines, but with fundamentally different approaches. Conductor's JSON-native definitions separate orchestration from implementation, making workflows deterministic by construction — no side-effect restrictions to remember, no non-determinism bugs to debug. Temporal embeds orchestration in code, which requires developers to avoid non-deterministic operations (system clocks, random values, uncontrolled I/O) or risk subtle replay failures.
Conductor is fully open source (Apache 2.0) with no proprietary server components. It provides native LLM orchestration for 14+ providers, MCP tool calling, and vector database support out of the box — capabilities Temporal does not offer. Conductor's JSON definitions can be generated and modified at runtime by LLMs or APIs without a compile/deploy cycle.
### How is Conductor different from AWS Step Functions?
Step Functions is a proprietary, cloud-locked service. Conductor is an open source, self-hosted workflow engine you can run on any infrastructure. Conductor supports 7+ language SDKs, 5 persistence backends, and provides native AI agent orchestration — none of which Step Functions offers. If you need an open source Step Functions alternative with no cloud lock-in, Conductor is a strong fit.
### How is Conductor different from Airflow?
Airflow is a DAG-based batch scheduler designed for data pipelines. Conductor is a real-time workflow orchestration engine designed for microservice orchestration, event-driven workflows, and AI agent orchestration. Conductor provides durable execution with sub-second task scheduling, while Airflow is optimized for scheduled batch jobs. If you need a real-time workflow engine rather than a job scheduler, Conductor is the better choice.
### Can I use Conductor for workflow automation?
Yes. Conductor is a developer-first workflow automation platform — not a low-code drag-and-drop tool, but a code-first workflow engine where you define workflows as code or JSON and implement task workers in any language. It is well suited for automating business processes, data pipelines, and multi-service workflows that need durable execution and full observability.
## Can Conductor orchestrate AI agents?
Yes. Conductor provides native AI agent orchestration with LLM tasks (chat completion, text completion), MCP tool calling and function calling (LIST_MCP_TOOLS, CALL_MCP_TOOL), human-in-the-loop approval (HUMAN task), and dynamic workflows that agents can generate at runtime. Every agent built on Conductor is a durable agent — LLM orchestration runs with the same durable execution guarantees as any other workflow, so agents survive crashes, retries, and infrastructure failures without losing progress.
## Does Conductor support MCP (Model Context Protocol)?
Yes. LIST_MCP_TOOLS discovers available tools from any MCP server, and CALL_MCP_TOOL executes them. Workflows can also be exposed as MCP tools via the MCP Gateway.
## What LLM providers does Conductor support?
14+ providers natively: Anthropic (Claude), OpenAI (GPT), Azure OpenAI, Google Gemini, AWS Bedrock, Mistral, Cohere, HuggingFace, Ollama, Perplexity, Grok, StabilityAI, and more. All accessible as workflow system tasks with built-in function calling and tool use via MCP integration.
## Does Conductor support vector databases and RAG?
Yes. Built-in support for Pinecone, pgvector, and MongoDB Atlas Vector Search. System tasks handle embedding generation, storage, indexing, and semantic search — enabling RAG pipelines as standard workflows.
## Is Conductor a durable execution engine?
Yes. Every workflow execution is persisted at each step. If a task fails, it's retried with configurable backoff. If a worker crashes, the task is rescheduled. If the server restarts, execution resumes exactly where it left off. See [Durable Execution](../architecture/durable-execution.md).
## Can Conductor handle millions of workflows?
Yes. Originally built at Netflix to handle massive scale, Conductor scales horizontally across multiple server instances. Workers scale independently, and the server supports millions of concurrent workflow executions across multiple persistence backends. This horizontal scaling architecture makes Conductor suitable for production workflow deployments at any scale.
## Does Conductor support the saga pattern?
Yes. Configure a `failureWorkflow` that runs compensation logic when the main workflow fails. Combined with task-level retries and timeout policies, Conductor provides full saga pattern support for distributed transactions. See [Handling Errors](how-tos/Workflows/handling-errors.md).
## Can I create workflows at runtime?
Yes. Workflow definitions are JSON and can be created, modified, and started dynamically via the API or SDKs. LLMs can generate workflow definitions that Conductor executes immediately without pre-registration.
## Does Conductor support human-in-the-loop?
Yes. The HUMAN task type pauses workflow execution until an external signal (approval, rejection, or data input) is received via API. The pause survives server restarts and deploys.
## What persistence backends are supported?
Redis, PostgreSQL, MySQL, Cassandra, and SQLite. Choose based on your scale and operational requirements.
## What message brokers are supported?
Kafka, NATS, NATS Streaming, AMQP (RabbitMQ), SQS, and Conductor's internal queue. Use them for event-driven workflows and external system integration.
## How do you schedule a task to be put in the queue after some time (e.g. 1 hour, 1 day etc.)
After polling for the task update the status of the task to `IN_PROGRESS` and set the `callbackAfterSeconds` value to the desired time. The task will remain in the queue until the specified second before worker polling for it will receive it again.
If there is a timeout set for the task, and the `callbackAfterSeconds` exceeds the timeout value, it will result in task being TIMED_OUT.
## How long can a workflow be in running state? Can I have a workflow that keeps running for days or months?
Yes. As long as the timeouts on the tasks are set to handle long running workflows, it will stay in running state.
## My workflow fails to start with missing task error
Ensure all the tasks are registered via `/metadata/taskdefs` APIs. Add any missing task definition (as reported in the error) and try again.
## Where does my worker run? How does conductor run my tasks?
Conductor does not run the workers. When a task is scheduled, it is put into the queue maintained by Conductor. Workers are required to poll for tasks using `/tasks/poll` API at periodic interval, execute the business logic for the task and report back the results using `POST {{ api_prefix }}/tasks` API call.
Conductor, however will run [system tasks](../documentation/configuration/workflowdef/systemtasks/index.md) on the Conductor server.
## How can I schedule workflows to run at a specific time?
Conductor itself does not provide any scheduling mechanism. But there is a community project [_Schedule Conductor Workflows_](https://github.com/jas34/scheduledwf) which provides workflow scheduling capability as a pluggable module as well as workflow server.
Other way is you can use any of the available scheduling systems to make REST calls to Conductor to start a workflow. Alternatively, publish a message to a supported eventing system like SQS to trigger a workflow.
More details about [eventing](../documentation/configuration/eventhandlers.md).
## Can I use Conductor with Ruby / Go / Python / JavaScript / C# / Rust?
Yes. Workers can be written in any language as long as they can poll and update the task results via HTTP endpoints. Conductor provides official and community SDKs for many languages:
- **Java** — [conductor-oss/java-sdk](https://github.com/conductor-oss/java-sdk)
- **Python** — [conductor-oss/python-sdk](https://github.com/conductor-oss/python-sdk)
- **Go** — [conductor-oss/go-sdk](https://github.com/conductor-oss/go-sdk)
- **JavaScript** — [conductor-oss/javascript-sdk](https://github.com/conductor-oss/javascript-sdk)
- **C#** — [conductor-oss/csharp-sdk](https://github.com/conductor-oss/csharp-sdk)
- **Ruby** — [conductor-oss/ruby-sdk](https://github.com/conductor-oss/ruby-sdk)
- **Rust** — [conductor-oss/rust-sdk](https://github.com/conductor-oss/rust-sdk)
## The same task is scheduled twice, both showing "attempt 0". What causes this?
This is almost always caused by running multiple Conductor server instances without distributed locking enabled. When locking is off, two server instances can each pick up the same workflow and independently schedule the same task — producing two identical entries, both at attempt 0, with neither aware of the other.
**To fix it**, enable distributed locking so only one server processes a given workflow at a time:
```properties
conductor.app.workflowExecutionLockEnabled=true
conductor.workflow-execution-lock.type=redis # or zookeeper
```
See [Locking](running/deploy.md#locking) for the full configuration, including Redis and Zookeeper options.
If you are running a single server instance, the cause is more likely the sweeper and an event or callback both triggering a `decide` on the same workflow simultaneously. The locking setting above resolves this case as well.
## My workflow is running and the task is SCHEDULED but it is not being processed.
Make sure that the worker is actively polling for this task. Navigate to the `Task Queues` tab on the Conductor UI and select your task name in the search box. Ensure that `Last Poll Time` for this task is current.
In Conductor 3.x, ```conductor.redis.availabilityZone``` defaults to ```us-east-1c```. Ensure that this matches where your workers are, and that it also matches```conductor.redis.hosts```.
## How do I configure a notification when my workflow completes or fails?
When a workflow fails, you can configure a "failure workflow" to run using the```failureWorkflow``` parameter. By default, three parameters are passed:
* reason
* workflowId: use this to pull the details of the failed workflow.
* failureStatus
You can also use the Workflow Status Listener:
* Set the workflowStatusListenerEnabled field in your workflow definition to true which enables [notifications](../documentation/configuration/workflowdef/index.md#workflow-status-listener).
* Add a custom implementation of the Workflow Status Listener. Refer to the [Workflow Status Listener extension guide](../documentation/advanced/extend.md#workflow-status-listener).
* This notification can be implemented in such a way as to either send a notification to an external system or to send an event on the conductor queue to complete/fail another task in another workflow as described in the [event handlers documentation](../documentation/configuration/eventhandlers.md).
Refer to this [documentation](../documentation/configuration/workflowdef/index.md#workflow-status-listener) to extend conductor to send out events/notifications upon workflow completion/failure.
## I want my worker to stop polling and executing tasks when the process is being terminated. (Java client)
In a `PreDestroy` block within your application, call the `shutdown()` method on the `TaskRunnerConfigurer` instance that you have created to facilitate a graceful shutdown of your worker in case the process is being terminated.
## Can I exit early from a task without executing the configured automatic retries in the task definition?
Set the status to `FAILED_WITH_TERMINAL_ERROR` in the TaskResult object within your worker. This would mark the task as FAILED and fail the workflow without retrying the task as a fail-fast mechanism.
docs/devguide/how-tos/Tasks/choosing-tasks.md
+108
View File
@@ -0,0 +1,108 @@
---
description: "Choose the right task type for your Conductor workflow — system tasks, operators, and worker tasks for microservice orchestration and workflow automation."
---
# Choosing Tasks
Tasks are the building blocks of Conductor workflows. In this guide, familiarise yourself with the tasks available in Conductor OSS and the differences between each of them.
## Built-in tasks
Built-in tasks allow you to easily run common tasks on the Conductor server without needing to build and deploy your own task workers. Here is an introduction of the built-in tasks available in Conductor:
* **[System tasks](../../../documentation/configuration/workflowdef/systemtasks/index.md)** common tasks that allow you to get started quickly without needing custom workers.
* **[Operators](../../../documentation/configuration/workflowdef/operators/index.md)** enable you to declaratively design the workflow's control flow and logic with minimal code required.
### System tasks
Here are the system tasks available in Conductor OSS for common use:
| System Task | Description |
| :-------------------- | :----------------------------------- |
| [Event](../../../documentation/configuration/workflowdef/systemtasks/event-task.md) | Publish events to an external eventing system (AMQP, SQS, Kafka, and so on). |
| [HTTP](../../../documentation/configuration/workflowdef/systemtasks/http-task.md) | Call an API or HTTP endpoint. |
| [Human](../../../documentation/configuration/workflowdef/systemtasks/human-task.md) | Wait for an external signal. |
| [Inline](../../../documentation/configuration/workflowdef/systemtasks/inline-task.md) | Execute lightweight JavaScript code inline. |
| [No Op](../../../documentation/configuration/workflowdef/systemtasks/noop-task.md) | Do nothing. |
| [JSON JQ Transform](../../../documentation/configuration/workflowdef/systemtasks/json-jq-transform-task.md) | Clean or transform JSON data using jq. |
| [Kafka Publish](../../../documentation/configuration/workflowdef/systemtasks/kafka-publish-task.md) | Publish messages to Kafka. |
| [Wait](../../../documentation/configuration/workflowdef/systemtasks/wait-task.md) | Wait until a set time or duration has passed. |
### Operators
Here are the operators available in Conductor OSS for managing the flow of execution:
| Operator | Description |
| -------------------------- | ----------------------------------------- |
| [Do While](../../../documentation/configuration/workflowdef/operators/do-while-task.md) | Execute tasks repeatedly, like a _do…while…_ statement. |
| [Dynamic](../../../documentation/configuration/workflowdef/operators/dynamic-task.md) | Execute a task dynamically, like a function pointer. |
| [Dynamic Fork](../../../documentation/configuration/workflowdef/operators/dynamic-fork-task.md) | Execute a dynamic number of tasks in parallel. |
| [Fork](../../../documentation/configuration/workflowdef/operators/fork-task.md) | Execute a static number of tasks in parallel. |
| [Join](../../../documentation/configuration/workflowdef/operators/join-task.md) | Join the forks after a Fork or Dynamic Fork before proceeding to the next task. |
| [Set Variable](../../../documentation/configuration/workflowdef/operators/set-variable-task.md) | Create or update workflow variables. |
| [Start Workflow](../../../documentation/configuration/workflowdef/operators/start-workflow-task.md) | Asynchronously start another workflow, like an entry point. |
| [Sub Workflow](../../../documentation/configuration/workflowdef/operators/sub-workflow-task.md) | Synchronously start another workflow, like a subroutine. |
| [Switch](../../../documentation/configuration/workflowdef/operators/switch-task.md) | Execute tasks conditionally, like an _if…else…_ statement. |
| [Terminate](../../../documentation/configuration/workflowdef/operators/terminate-task.md) | Terminate the current workflow, like a _return_ statement. |
## Custom tasks
If you need to implement custom logic beyond the scope of Conductor's system tasks, you can use Worker (`SIMPLE`) tasks instead. Unlike a built-in task, a Worker task requires setting up a worker outside the Conductor environment that polls for and executes the task.
## Task comparison
To help you decide on which tasks to use, here is a detailed comparison of similar tasks available in Conductor.
### Inline vs Worker tasks
The [Inline task](../../../documentation/configuration/workflowdef/systemtasks/inline-task.md) is used to execute custom JavaScript code directly within the workflow. It’s ideal for lightweight operations like **simple data transformations, conditional checks, or small calculations**. Because the code executes within the Conductor JVM, Inline tasks benefit from low latency, no network overhead, and easier debugging. However, it also has limitations on using other languages, custom libraries, frameworks, or stacks.
The Worker task is handled by external task workers that execute a custom function or service
is an external custom function or service that performs a specific task in a workflow. Written in any language of choice (Python, Java, etc), it can execute **complex business logic, custom algorithms, or long-running operations**. Worker tasks run outside the Conductor server, meaning they require additional infrastructure set-up and logging mechanisms.
### Event vs Kafka Publish tasks
If you only need to publish messages to a Kafka topic for external services to use, the [Kafka Publish](../../../documentation/configuration/workflowdef/systemtasks/kafka-publish-task.md) task is simpler to set up.
In contrast, the [Event](../../../documentation/configuration/workflowdef/systemtasks/event-task.md) task supports more involved set-ups, such as using events to start a Conductor workflow, or having Conductor consume messages. It also supports a wider range of event brokers across AMQP, NATS, SQS, Kafka, and Conductor's own internal queue.
### Wait vs Human tasks
The [Wait](../../../documentation/configuration/workflowdef/systemtasks/wait-task.md) task and [Human](../../../documentation/configuration/workflowdef/systemtasks/human-task.md) task both support waiting until a specific condition is met. Use the Wait task for cases when the workflow needs to wait for specific wait duration or timestamp, and use the Human task when the workflow needs to wait for an external trigger.
### Start Workflow vs Sub Workflow tasks
Both [Start Workflow](../../../documentation/configuration/workflowdef/operators/start-workflow-task.md) and [Sub Workflow](../../../documentation/configuration/workflowdef/operators/sub-workflow-task.md) tasks are useful for starting another workflow within a workflow. However, the Start Workflow task starts another workflow and proceeds to the next task without waiting for the started workflow to complete, while the Sub Workflow task will wait for the subworkflow to reach terminal state before proceeding to the next task.
The Sub Workflow task provides a tighter coupling between the parent workflow and the subworkflow. This is useful for cases when you need to associate workflow progress and states, or if you need to pass the output of the subworkflow back into the parent workflow.
### Fork vs Dynamic Fork tasks
Both [Fork](../../../documentation/configuration/workflowdef/operators/fork-task.md) and [Dynamic Fork](../../../documentation/configuration/workflowdef/operators/dynamic-fork-task.md) facilitate parallel execution of tasks. The Fork task executes a predetermined number of forks, while the Dynamic Fork executes a variable number of forks at runtime.
If each fork must run a different set of tasks, it is best to use the Fork task, because Dynamic Forks can only run the same task for all its forks.
### Dynamic vs Switch tasks
Both the [Switch](../../../documentation/configuration/workflowdef/operators/switch-task.md) task and the [Dynamic](../../../documentation/configuration/workflowdef/operators/dynamic-task.md) task are useful in situations when the specific task to run is determined only at runtime. Using the Switch task allows you to easily predefine and set the specific conditions for each switch case, while using the Dynamic task allows to to mark a dynamic point in the workflow without having to pre-set all the case options into the workflow definition beforehand.
In the workflow diagram, the Dynamic task will produce a more simplified view, as it will only display the selected task. Meanwhile, the Switch task will produce a more comprehensive view that shows all possible paths that the workflow could have taken.
Here are some scenarios for deciding between a Dynamic task and a Switch task:
| Scenario | Task to Use |
| -------------------------- | ----------------------------------------- |
| You have a huge number of case options or the specific case options are not yet determined. | Dynamic |
| You need a default case option. | Switch |
| Each case option involves multiple tasks. | Switch |
| The conditions for each switch case is relatively straightforward. | Switch |
| The conditions for each switch case is constantly changing, or requires more complicated logic. | Dynamic |
If you opt for the Dynamic task, you must set up the control flow for how the task to run will be determined at runtime. For example, using a preceding task that must pass the task name into the Dynamic task.
docs/devguide/how-tos/Tasks/creating-tasks.md
+128
View File
@@ -0,0 +1,128 @@
---
description: "Create and update task definitions in Conductor to configure timeouts, retries, rate limits, and input templates for worker and system tasks."
---
# Creating / Updating Task Definitions
A [task definition](../../../documentation/configuration/taskdef.md) specifies a task’s general implementation details:
- Timeout policy
- Retry logic
- Rate limit and execution limit
- Input/output keys
- Input template
This definition applies to all instances of the task across workflows.
You can create task definitions using the Conductor UI or APIs for the following scenarios:
- **Worker tasks**—All Worker tasks (`SIMPLE`) must be registered to the Conductor server as a task definition before it can execute in a workflow.
- **System tasks**—System tasks don't require a task definition, but you can create one with the same name to customize retry, timeout, and rate limit behavior.
## Using Conductor UI
With the UI, you can create or update task definitions visually.
### Creating task definitions
**To create a task definition:**
1. In [**Executions** > **Tasks**](http://localhost:8080/taskDefs), select **+ New Task Definition**.
2. Configure the task definition JSON. Refer to [Task Definitions](../../../documentation/configuration/taskdef.md) for the full parameters.
3. Select **Save** > **Save**.
### Updating task definitions
**To update a task definition:**
1. In [**Executions** > **Tasks**](http://localhost:8080/taskDefs), select the task definition to be updated.
2. Modify the task definition JSON. Refer to [Task Definitions](../../../documentation/configuration/taskdef.md) for the full parameters.
3. Select **Save** > **Save**.
## Using the CLI
You can create task definitions using the Conductor CLI. Save your task definitions to a JSON file and run:
```bash
conductor task create tasks.json
```
The file should contain an array of task definitions. Refer to [Task Definitions](../../../documentation/configuration/taskdef.md) for a reference guide on the full parameters.
## Using APIs
Refer to [Task Definitions](../../../documentation/configuration/taskdef.md) for a reference guide on the full parameters.
### Creating task definitions
You can also create task definitions using the Create Task Definition API (`POST api/metadata/taskdefs`). The API accepts an array of task definitions, allowing you to create them in bulk.
??? note "Example using cURL"
```shell
curl '{{ server_host }}/api/metadata/taskdefs' \
-H 'accept: */*' \
-H 'content-type: application/json' \
--data-raw '[{"createdBy":"user","name":"sample_task_name_1","description":"This is a sample task for demo","responseTimeoutSeconds":10,"timeoutSeconds":30,"inputKeys":[],"outputKeys":[],"timeoutPolicy":"TIME_OUT_WF","retryCount":3,"retryLogic":"FIXED","retryDelaySeconds":5,"inputTemplate":{},"rateLimitPerFrequency":0,"rateLimitFrequencyInSeconds":1}]'
```
### Updating task definitions
You can update task definitions using the Update Task Definition API (`PUT api/metadata/taskdefs`). This API can only be used to update a single task definition at a time.
??? note "Example using cURL"
```shell
curl '{{ server_host }}/api/metadata/taskdefs' \
-X 'PUT' \
-H 'accept: */*' \
-H 'content-type: application/json' \
--data-raw '{"createdBy":"user","name":"sample_task_name_1","description":"This is a sample task for demo","responseTimeoutSeconds":10,"timeoutSeconds":30,"inputKeys":[],"outputKeys":[],"timeoutPolicy":"TIME_OUT_WF","retryCount":3,"retryLogic":"FIXED","retryDelaySeconds":5,"inputTemplate":{},"rateLimitPerFrequency":0,"rateLimitFrequencyInSeconds":1}'
```
## Using SDKs
Conductor offers client SDKs for popular languages which have library methods for making the API call. Refer to the SDK documentation to configure a client in your selected language to create or update task definitions.
Refer to [Task Definitions](../../../documentation/configuration/taskdef.md) for a reference guide on the full parameters.
### Creating task definitions - Example using JavaScript
In this example, the JavaScript Fetch API is used to create the task definition `sample_task_name_1`.
```javascript
fetch("{{ server_host }}/api/metadata/taskdefs", {
"headers": {
"accept": "*/*",
"content-type": "application/json",
},
"body": "[{\"createdBy\":\"user\",\"name\":\"sample_task_name_1\",\"description\":\"This is a sample task for demo\",\"responseTimeoutSeconds\":10,\"timeoutSeconds\":30,\"inputKeys\":[],\"outputKeys\":[],\"timeoutPolicy\":\"TIME_OUT_WF\",\"retryCount\":3,\"retryLogic\":\"FIXED\",\"retryDelaySeconds\":5,\"inputTemplate\":{},\"rateLimitPerFrequency\":0,\"rateLimitFrequencyInSeconds\":1}]",
"method": "POST"
});
```
### Updating task definitions - Example using JavaScript
In this example, the JavaScript Fetch API is used to update the task definition `sample_task_name_1`.
```javascript
fetch("{{ server_host }}/api/metadata/taskdefs", {
"headers": {
"accept": "*/*",
"content-type": "application/json",
},
"body": "{\"createdBy\":\"user\",\"name\":\"sample_task_name_1\",\"description\":\"This is a sample task for demo\",\"responseTimeoutSeconds\":10,\"timeoutSeconds\":30,\"inputKeys\":[],\"outputKeys\":[],\"timeoutPolicy\":\"TIME_OUT_WF\",\"retryCount\":3,\"retryLogic\":\"FIXED\",\"retryDelaySeconds\":5,\"inputTemplate\":{},\"rateLimitPerFrequency\":0,\"rateLimitFrequencyInSeconds\":1}",
"method": "PUT"
});
```
## Reusing tasks
Once a task is defined in Conductor, it can be reused numerous times:
- **In the same workflow** — use the same task with different task reference names.
- **Across workflows** — any workflow can reference any registered task definition.
When reusing tasks in a multi-tenant system, all work assigned to a task goes into the same queue by default. If a noisy neighbor causes polling delays, you can scale up the number of workers or use [task-to-domain](../../../documentation/api/taskdomains.md) to route task load into separate queues.
docs/devguide/how-tos/Tasks/task-inputs.md
+314
View File
@@ -0,0 +1,314 @@
---
description: "Wire task inputs in Conductor workflows — reference workflow inputs, task outputs, and variables using dynamic expressions in this open source workflow orchestration engine."
---
# Wiring Task Inputs
In Conductor, task inputs can be provided in the workflow definition in multiple ways:
- As a hard-coded value –
```
"taskInputA": true
```
- As a dynamic reference to the workflow inputs, workflow variables, or the inputs/outputs of prior tasks –
```
"taskInputA": "${workflow.input.someValue}
```
## Syntax for dynamic references
All dynamic references are formatted as the following expression:
```
"${type.jsonpath}"
```
These dynamic references are formatted as dot-notation expressions, taking after [JSONPath syntax](https://goessner.net/articles/JsonPath/).
| Component | Description |
| -------------------- | ----------------------------------------------------------------------------------------------------- |
| `${...}` | The root notation indicating that the variable will be dynamically replaced at runtime. |
| type | The type of reference. Supported values:<ul><li>**workflow**—Refers to the current workflow instance.</li> <li>**workflow.input**—Refers to the workflow’s input parameters.</li> <li>**workflow.output**—Refers to the workflow’s output parameters.</li> <li>**workflow.variables**—Refers to the workflow variables set in the workflow using the [Set Variable](../../../documentation/configuration/workflowdef/operators/set-variable-task.md) task.</li> <li>**_taskReferenceName_**—Refers to a task in the current workflow instance by its reference name. (For example, “http_ref”).</li> <li>**_taskReferenceName_.input**—Refers to the task’s input parameters.</li> <li>**_taskReferenceName_.output**—Refers to the task’s output parameters.</li></ul> |
| jsonpath | The [JSONPath](https://goessner.net/articles/JsonPath/) expression in dot-notation. |
### Sample expressions
Here is a non-exhaustive list of dynamic references you can use:
- To reference a task’s input payload –
```
${<taskReferenceName>.input}
```
- To reference a task’s output payload –
```
${<taskReferenceName>.output}
```
- To reference a task’s input parameter –
```
${<taskReferenceName>.input.<someKey>}
```
- To reference a task’s output parameter –
```
${<taskReferenceName>.output.<someKey>}
```
- To reference the workflow's input payload –
```
${workflow.input}
```
- To reference the workflow's output payload –
```
${workflow.output}
```
- To reference the workflow's input parameter –
```
${workflow.input.<someKey>}
```
- To reference the workflow's output parameter –
```
${workflow.output.<someKey>}
```
- To reference the workflow's current status (RUNNING, PAUSED, TIMED_OUT, TERMINATED, FAILED, or COMPLETED) –
```
${workflow.status}
```
- To reference the workflow's (execution) ID –
```
${workflow.workflowId}
```
- (Used in sub-workflows) To reference the parent workflow (execution) ID –
```
${workflow.parentWorkflowId}
```
- (Used in sub-workflows) To reference the task execution ID for the Sub Workflow task in the parent workflow –
```
${workflow.parentWorkflowTaskId}
```
- To reference the workflow's name –
```
${workflow.workflowType}
```
- To reference the workflow's version –
```
${workflow.version}
```
- To reference the start time of the workflow execution –
```
${workflow.createTime}
```
- To reference the workflow's correlation ID –
```
${workflow.correlationId}
```
- To reference the workflow’s domain name that was invoked during its execution –
```
${workflow.taskToDomain.<domainName>}
```
- To reference the workflow's variable created using the Set Variable task –
```
${workflow.variables.<someKey>}
```
## Examples
Here are some examples for using dynamic references in workflows.
<details>
<summary>Referencing workflow inputs​​</summary>
For the given workflow input:
```json
{
"userID": 1,
"userName": "SAMPLE",
"userDetails": {
"country": "nestedValue",
"age": 50
}
}
```
You can reference these workflow inputs elsewhere using the following expressions:
```json
{
"user": "${workflow.input.userName}",
"userAge": "${workflow.input.userDetails.age}"
}
```
At runtime, the parameters will be:
```json
{
"user": "SAMPLE",
"userAge": 50
}
```
</details>
<details>
<summary>Referencing other task outputs​​</summary>
If a task <code>previousTaskReference</code> produced the following output:
```json
{
"taxZone": "A",
"productDetails": {
"nestedKey1": "outputValue-1",
"nestedKey2": "outputValue-2"
}
}
```
You can reference these task outputs elsewhere using the following expressions:
```json
{
"nextTaskInput1": "${previousTaskReference.output.taxZone}",
"nextTaskInput2": "${previousTaskReference.output.productDetails.nestedKey1}"
}
```
At runtime, the parameters will be:
```json
{
"nextTaskInput1": "A",
"nextTaskInput2": "outputValue-1"
}
```
</details>
<details>
<summary>Referencing workflow variables</summary>
If a workflow variable is set using the Set Variable task:
```json
{
"name": "Ipsum"
}
```
The variable can be referenced in the same workflow using the following expression:
```json
{
"user": "${workflow.variables.name}"
}
```
<b>Note:</b> Workflow variables cannot be re-referenced across workflows, even between a parent workflow and a sub-workflow.
</details>
<details>
<summary>Referencing data between parent workflow and sub-workflow​</summary>
To pass parameters from a parent workflow into its sub-workflow, you must declare them as input parameters for the Sub Workflow task. If needed, these inputs can then be set as workflow variables within the sub-workflow definition itself using a Set Variable task.
```
// parent workflow definition with task configuration
{
"createTime": 1733980872607,
"updateTime": 0,
"name": "testParent",
"description": "workflow with subworkflow",
"version": 1,
"tasks": [
{
"name": "get_item",
"taskReferenceName": "get_item_ref",
"inputParameters": {
"uri": "https://example.com/api",
"method": "GET",
"accept": "application/json",
"contentType": "application/json",
"encode": true
},
"type": "HTTP",
},
{
"name": "sub_workflow",
"taskReferenceName": "sub_workflow_ref",
"inputParameters": {
"user": "${workflow.variables.name}",
"item": "${previous_task_ref.output.item[0]}"
},
"type": "SUB_WORKFLOW",
"subWorkflowParam": {
"name": "testSub",
"version": 1
}
}
],
"inputParameters": [],
"outputParameters": {}
}
```
To pass parameters from a sub-workflow back to its parent workflow, you must pass them as the sub-workflow’s output parameters in the sub-workflow definition.
```
// sub-workflow definition
{
"createTime": 1726651838873,
"updateTime": 1733983507294,
"name": "testSub",
"description": "subworkflow for parent workflow",
"version": 1,
"tasks": [
{
"name": "get-user",
"taskReferenceName": "get-user_ref",
"inputParameters": {
"uri": "https://example.com/api",
"method": "GET",
"accept": "application/json",
"contentType": "application/json",
"encode": true
},
"type": "HTTP",
},
{
"name": "send-notification",
"taskReferenceName": "send-notification_ref",
"inputParameters": {
"uri": "https://example.com/api",
"method": "GET",
"accept": "application/json",
"contentType": "application/json",
"encode": true
},
"type": "HTTP",
}
],
"inputParameters": [],
"outputParameters": {
"location": "${get-user_ref.output.response.body.results[0].location.country}",
"isNotif": "${send-notification_ref.output}"
}
}
```
In the parent workflow, these sub-workflow outputs can be referenced using the expression format `${<sub_workflow_ref>.output.<someKey>}`.
</details>
## Troubleshooting
You can verify if the data was passed correctly by checking the input/output values of the task execution in the UI. Common errors:
- If the reference expression is incorrectly formatted, the referencing parameter value may end up with the wrong data or a null value.
- If the referenced value (such as a task output) has not resolved at the point when it is referenced, the referencing parameter value will be null.
docs/devguide/how-tos/Workers/scaling-workers.md
+140
View File
@@ -0,0 +1,140 @@
---
description: "Monitor task queues and scale Conductor workers — queue depth, poll data, Prometheus metrics, autoscaling policies, and performance tuning."
---
# Scaling Task Workers
Workers execute business logic outside the Conductor server. Keeping them healthy requires two things: **monitoring** queue and worker state, and **scaling** based on what the data tells you.
## Monitoring task queues
Conductor tracks queue size and worker poll activity for every task type. Use this data to detect backlogs, stalled workers, and capacity issues.
### Using the UI
Navigate to **Home > Task Queues** (or `<your UI server URL>/taskQueue`). For each task, the UI shows:
- **Queue Size** — tasks waiting to be picked up.
- **Workers** — count and instance details of workers polling this task.
### Using the CLI
```bash
# List all tasks with queue info
conductor task list
# Get details for a specific task
conductor task get <TASK_NAME>
```
### Using APIs
Get the number of tasks waiting in a queue:
```shell
curl '{{ server_host }}{{ api_prefix }}/tasks/queue/sizes?taskType=<TASK_NAME>' \
-H 'accept: */*'
```
Get worker poll data (which workers are polling, last poll time):
```shell
curl '{{ server_host }}{{ api_prefix }}/tasks/queue/polldata?taskType=<TASK_NAME>' \
-H 'accept: */*'
```
!!! note
Replace `<TASK_NAME>` with your task name.
## Prometheus metrics
Conductor publishes metrics that feed dashboards, alerts, and autoscaling policies. All metrics include `taskType` as a tag so you can monitor per-task.
### Queue depth (Gauge)
```promql
max(task_queue_depth{taskType="my_task"})
```
- Keep queue depth stable. It doesn't need to be zero (especially for long-running tasks), but sustained growth means workers can't keep up.
- Alert on queue depth increasing over a sustained period and use it to trigger autoscaling.
### Task completion rate (Counter)
```promql
rate(task_completed_seconds_count{taskType="my_task"}[$__rate_interval])
```
- Measures throughput — tasks completed per second.
- A sudden drop indicates workers are struggling, failing, or have stopped polling.
- Set a minimum throughput threshold and alert when it drops below.
### Queue wait time
```promql
max(task_queue_wait_time_seconds{quantile="0.99", taskType="my_task"})
```
How long tasks sit in the queue before a worker picks them up. If this is more than a few seconds:
1. **Check worker count** — if all workers are busy, add more instances.
2. **Check polling interval** — reduce it if workers aren't polling frequently enough.
!!! warning
Reducing the polling interval increases API requests to the server. Balance responsiveness against server load.
## Scaling strategies
### When to scale
| Signal | Action |
|---|---|
| Queue depth growing steadily | Add worker instances |
| Queue wait time > 5s at p99 | Add worker instances or reduce polling interval |
| Throughput dropping while queue grows | Investigate worker health (CPU, memory, downstream dependencies) |
| Queue consistently empty, workers idle | Scale down to save resources |
### Horizontal scaling
Add more worker instances. Conductor distributes tasks automatically — every worker polling the same task type competes for work from the same queue. No configuration changes needed on the Conductor server.
### Polling interval tuning
The polling interval controls how frequently workers check for new tasks. Shorter intervals mean lower latency but higher server load.
| Scenario | Recommended interval |
|---|---|
| Latency-sensitive tasks | 100–500ms |
| Standard processing | 1–5s |
| Batch / background work | 5–30s |
### Thread pool sizing
Each worker instance can run multiple polling threads. A good starting point:
```
threads = (task_throughput × avg_task_duration) / num_worker_instances
```
For I/O-bound tasks (HTTP calls, database queries), use more threads than CPU cores. For CPU-bound tasks, match thread count to available cores.
### Rate limiting
If downstream services have rate limits, configure task-level rate limits to prevent workers from overwhelming them:
```json
{
"name": "call_external_api",
"rateLimitPerFrequency": 100,
"rateLimitFrequencyInSeconds": 60
}
```
This limits the task to 100 executions per 60-second window across all workers.
### Domain isolation
Use [task-to-domain](../../../documentation/api/taskdomains.md) to route tasks to specific worker pools. This prevents noisy neighbors — a high-volume workflow won't starve workers serving a latency-sensitive one.
docs/devguide/how-tos/Workflows/creating-workflows.md
+78
View File
@@ -0,0 +1,78 @@
---
description: "Create and update workflow definitions in Conductor using the UI, CLI, REST APIs, or client SDKs. Supports versioning and JSON configuration."
---
# Creating / Updating Workflows
You can create and update workflows using the Conductor UI, APIs, or SDKs. These workflows can be versioned, which is useful for [a variety of cases](versioning-workflows.md#when-to-version-workflows).
If your workflow definition contains any new tasks, you must also register the task definitions to Conductor before running the workflow.
## Using Conductor UI
With the UI, you can create or update workflow definitions visually.
### Creating workflows
**To create a workflow definition:**
1. In **[Definitions](http://localhost:8080/workflowDefs)**, select **+ New Workflow Definition**.
2. Configure the workflow definition JSON. Refer to [Workflow Definition](../../../documentation/configuration/workflowdef/index.md) for the reference guide on the full parameters.
3. Select **Save** > **Save**.
### Updating workflows
**To update a workflow definition:**
1. In **[Definitions](http://localhost:8080/workflowDefs**)**, select the workflow to be updated.
2. Modify the workflow definition JSON. Refer to [Workflow Definition](../../../documentation/configuration/workflowdef/index.md) for the reference guide on the full parameters.
3. Select **Save**. The workflow version will automatically increment by 1.
4. (Optional) Clear the **Automatically set version** checkbox to save the updated workflow definition without creating a new version.
5. Select **Save** again to confirm.
## Using the CLI
You can create or update workflow definitions using the Conductor CLI. Save your workflow definition to a JSON file and run:
```bash
conductor workflow create workflow.json
```
Refer to [Workflow Definition](../../../documentation/configuration/workflowdef/index.md) for the reference guide on the full parameters.
## Using APIs
You can also create or update workflow definitions using the Update Workflow Definition API (`PUT api/metadata/workflow`).
Refer to [Workflow Definition](../../../documentation/configuration/workflowdef/index.md) for the reference guide on the full parameters.
??? note "Example using cURL"
```shell
curl '{{ server_host }}/api/metadata/workflow' \
-X 'PUT' \
-H 'accept: */*' \
-H 'content-type: application/json' \
--data-raw '[{"name":"sample_workflow","description":"shipping","version":1,"tasks":[{"name":"ship_via","taskReferenceName":"ship_via","type":"SIMPLE","inputParameters":{"service":"${workflow.input.service}"}}],"inputParameters":["service"],"outputParameters":{},"schemaVersion":2, "ownerEmail": "example@email.com"}]'
```
## Using SDKs
Conductor offers client SDKs for popular languages which have library methods for making the API call. Refer to the SDK documentation to configure a client in your selected language to invoke workflow executions.
Refer to [Workflow Definition](../../../documentation/configuration/workflowdef/index.md) for the reference guide on the full parameters.
### Example using JavaScript
In this example, the JavaScript Fetch API is used to create the workflow `sample_workflow`.
```javascript
fetch("{{ server_host }}/api/metadata/workflow", {
"headers": {
"accept": "*/*",
"content-type": "application/json"
},
"body": "[{\"name\":\"sample_workflow\",\"description\":\"shipping\",\"version\":1,\"tasks\":[{\"name\":\"ship_via\",\"taskReferenceName\":\"ship_via\",\"type\":\"SIMPLE\",\"inputParameters\":{\"service\":\"${workflow.input.service}\"}}],\"inputParameters\":[\"service\"],\"outputParameters\":{},\"schemaVersion\":2,\"ownerEmail\": \"example@email.com\"}]",
"method": "PUT"
});
```
docs/devguide/how-tos/Workflows/debugging-workflows.md
+61
View File
@@ -0,0 +1,61 @@
---
description: "Debugging Workflows — identify and resolve failed Conductor workflow executions using the UI diagram and task details."
---
# Debugging Workflows
The [workflow execution views](viewing-workflow-executions.md) in the Conductor UI are useful for debugging workflow issues. Learn how to debug failed executions and rerun them.
## Debug procedure
When you view the workflow execution details, the cause of the workflow failure will be stated at the top. Go to the **Tasks > Diagram** tab to quickly identify the failed task, which is marked in red. You can select the failed task to investigate the details of the failure.
The following tab views or fields in the task details are useful for debugging:
| Field or Tab Name | Description |
|-------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------|
| _Reason for Incompletion_ in **Task Detail** > **Summary** | Contains the exception message thrown by the task worker. |
| _Worker_ in **Task Detail** > **Summary** | Contains the worker instance ID where the failure occurred. Useful for digging up detailed logs, if it has not already captured by Conductor. |
| **Task Detail** > **Input** | Useful for verifying if the task inputs were correctly computed and provided to the task. |
| **Task Detail** > **Output** | Useful for verifying what the task produced as output. |
| **Task Detail** > **Logs** | Contains the task logs, if supplied by the task worker. |
| **Task Detail** > **Retried Task - Select an instance** | (If the task has been retried multiple times) Contains all retry attempts in a dropdown list. Each list item contains the task details for a particular attempt. |
![Debugging Workflow Execution](workflow_debugging.png)
## Recovering from failure
Once you have resolved the underlying issue for the execution failure, you can manually restart or retry the failed workflow execution using the Conductor UI or APIs.
Here are the recovery options:
| Recovery Action | Description |
|---------------------|----------------------------|
| Restart with Current Definitions | Restart the workflow from the beginning using the same workflow definition that was used in the original execution. This option is useful if the workflow definition has changed and you want to run the execution instance using the original definition. |
| Restart with Latest Definitions | Restart the workflow from the beginning using the latest workflow definition. This option is useful if changes were made to the workflow definition and you want to run the execution instance with the latest definition. |
| Rerun from a specific task | Re-execute the workflow from a specific task, reusing the outputs of all prior tasks. This option is useful when a task in the middle of the workflow failed and you want to fix and re-run it without re-executing everything before it. |
| Retry - From failed task | Retry the workflow from the last failed task. |
!!! Note
You can set tasks to be retried automatically in case of transient failures. Refer to [Task Definition](../../../documentation/configuration/taskdef.md) for more information.
### Using Conductor UI
**To recover from failure**:
1. In the workflow execution details page, select **Actions** in the top right corner.
2. Select one of the following options:
- Restart with Current Definitions
- Restart with Latest Definitions
- Rerun from a specific task
- Retry - From failed task
### Using APIs
You can restart workflow executions using the Restart Workflow API (`POST api/workflow/{workflowId}/restart`) or the Bulk Restart Workflow API (`POST api/workflow/bulk/restart`).
You can rerun a workflow from a specific task using the Rerun Workflow API (`POST api/workflow/{workflowId}/rerun`) with a request body specifying the `reRunFromTaskId`.
Likewise, you can retry workflow executions from the last failed task using the Retry Workflow API (`POST api/workflow/{workflowId}/retry`) or the Bulk Retry Workflow API (`POST api/workflow/bulk/retry`).
All three recovery operations — restart, rerun, and retry — work on workflows in any terminal state (COMPLETED, FAILED, TIMED_OUT, TERMINATED) and are available indefinitely. Conductor preserves the full execution history, so you can replay any workflow even months after the original run.
docs/devguide/how-tos/Workflows/execution_path.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 90 KiB

docs/devguide/how-tos/Workflows/handling-errors.md
+357
View File
@@ -0,0 +1,357 @@
---
description: "Handle workflow errors in Conductor using the saga pattern with compensation flows, retry strategies, task-level error handling, timeout policies, and workflow status listener notifications."
---
# Handling Workflow Errors
In production microservice architectures, failures are inevitable. Conductor provides multiple layers of error handling so you can build resilient, self-healing workflows:
* **Saga pattern** — run a compensation flow to undo completed steps when a workflow fails.
* **Retry strategies** — automatically retry failed tasks with configurable backoff.
* **Task-level error handling** — mark tasks as optional, fail immediately on terminal errors, or set per-task timeouts.
* **Timeout policies** — control what happens when a task or workflow exceeds its time limit.
* **Workflow status listener** — send notifications to external systems on workflow completion or failure.
## Saga pattern: compensation on failure
The saga pattern is a well-established approach for managing distributed transactions across microservices. Instead of a single atomic transaction that spans multiple services, a saga breaks the work into a sequence of local transactions. Each step has a corresponding **compensating action** that undoes its effect. When any step in the sequence fails, the previously completed steps are rolled back in reverse order by executing their compensating actions.
This pattern is essential in microservice architectures where two-phase commits are impractical. Because each service owns its own data, you cannot rely on a traditional database transaction to maintain consistency across services. The saga pattern gives you eventual consistency with explicit rollback logic, making failures predictable and recoverable.
### Configuring a failure workflow
You can configure a workflow to automatically run upon failure by adding the `failureWorkflow` parameter to your main workflow definition.
Additionally, you may also specify the _version_ of it by using the `failureWorkflowVersion` parameter.
```json
"failureWorkflow": "<Name of your compensation flow>",
"failureWorkflowVersion": 2,
```
If your main workflow fails, Conductor will trigger this failure workflow. By default, the following parameters are passed to the failure workflow as input:
* **`reason`** — The reason for the workflow's failure.
* **`workflowId`** — The failed workflow's execution ID.
* **`failureStatus`** — The failed workflow's status.
* **`failureTaskId`** — The execution ID for the task that failed in the workflow.
* **`failedWorkflow`** — The full workflow execution JSON for the failed workflow.
You can use these parameters to implement compensation actions in the failure workflow, such as notification alerts, resource clean-up, or reversing completed transactions.
### Example: Slack notification on failure
Here is a failure workflow that sends a Slack message when the main workflow fails. It posts the `reason` and `workflowId` so the team can debug the failure:
```json
{
"name": "shipping_failure",
"description": "Notification workflow for shipping workflow failures",
"version": 1,
"tasks": [
{
"name": "slack_message",
"taskReferenceName": "send_slack_message",
"inputParameters": {
"http_request": {
"headers": {
"Content-type": "application/json"
},
"uri": "https://hooks.slack.com/services/<_unique_Slack_generated_key_>",
"method": "POST",
"body": {
"text": "workflow: ${workflow.input.workflowId} failed. ${workflow.input.reason}"
},
"connectionTimeOut": 5000,
"readTimeOut": 5000
}
},
"type": "HTTP",
"retryCount": 3
}
],
"restartable": true,
"workflowStatusListenerEnabled": false,
"ownerEmail": "conductor@example.com",
"timeoutPolicy": "ALERT_ONLY"
}
```
### Example: saga compensation for order processing
A realistic saga implementation involves a main workflow that processes an order through multiple services and a compensation workflow that reverses each completed step if any step fails.
**Main workflow** — `order_processing` processes a customer order through three stages: charge the payment, reserve inventory, and arrange shipping.
```json
{
"name": "order_processing",
"description": "Process a customer order through payment, inventory, and shipping",
"version": 1,
"failureWorkflow": "order_compensation",
"tasks": [
{
"name": "charge_payment",
"taskReferenceName": "charge_payment_ref",
"inputParameters": {
"orderId": "${workflow.input.orderId}",
"customerId": "${workflow.input.customerId}",
"amount": "${workflow.input.totalAmount}"
},
"type": "SIMPLE",
"retryCount": 2,
"retryLogic": "EXPONENTIAL_BACKOFF",
"retryDelaySeconds": 5
},
{
"name": "reserve_inventory",
"taskReferenceName": "reserve_inventory_ref",
"inputParameters": {
"orderId": "${workflow.input.orderId}",
"items": "${workflow.input.items}",
"paymentTransactionId": "${charge_payment_ref.output.transactionId}"
},
"type": "SIMPLE",
"retryCount": 2,
"retryLogic": "FIXED",
"retryDelaySeconds": 3
},
{
"name": "arrange_shipping",
"taskReferenceName": "arrange_shipping_ref",
"inputParameters": {
"orderId": "${workflow.input.orderId}",
"shippingAddress": "${workflow.input.shippingAddress}",
"items": "${workflow.input.items}",
"inventoryReservationId": "${reserve_inventory_ref.output.reservationId}"
},
"type": "SIMPLE",
"retryCount": 1,
"retryLogic": "FIXED",
"retryDelaySeconds": 10
}
],
"restartable": true,
"workflowStatusListenerEnabled": true,
"ownerEmail": "order-team@example.com",
"timeoutPolicy": "TIME_OUT_WF",
"timeoutSeconds": 600
}
```
**Compensation workflow** — `order_compensation` reverses each completed step in reverse order: cancel the shipment, restore inventory, and refund the payment.
```json
{
"name": "order_compensation",
"description": "Undo completed order steps when order_processing fails",
"version": 1,
"tasks": [
{
"name": "cancel_shipment",
"taskReferenceName": "cancel_shipment_ref",
"inputParameters": {
"orderId": "${workflow.input.failedWorkflow.input.orderId}",
"shipmentId": "${workflow.input.failedWorkflow.tasks[arrange_shipping_ref].output.shipmentId}"
},
"type": "SIMPLE",
"optional": true,
"retryCount": 3,
"retryLogic": "FIXED",
"retryDelaySeconds": 5
},
{
"name": "restore_inventory",
"taskReferenceName": "restore_inventory_ref",
"inputParameters": {
"orderId": "${workflow.input.failedWorkflow.input.orderId}",
"reservationId": "${workflow.input.failedWorkflow.tasks[reserve_inventory_ref].output.reservationId}"
},
"type": "SIMPLE",
"optional": true,
"retryCount": 3,
"retryLogic": "FIXED",
"retryDelaySeconds": 5
},
{
"name": "refund_payment",
"taskReferenceName": "refund_payment_ref",
"inputParameters": {
"orderId": "${workflow.input.failedWorkflow.input.orderId}",
"transactionId": "${workflow.input.failedWorkflow.tasks[charge_payment_ref].output.transactionId}",
"amount": "${workflow.input.failedWorkflow.input.totalAmount}"
},
"type": "SIMPLE",
"retryCount": 5,
"retryLogic": "EXPONENTIAL_BACKOFF",
"retryDelaySeconds": 10
}
],
"restartable": true,
"workflowStatusListenerEnabled": false,
"ownerEmail": "order-team@example.com",
"timeoutPolicy": "ALERT_ONLY",
"timeoutSeconds": 1200
}
```
Notice that compensation tasks are marked `optional: true` for steps that may not have completed before the failure occurred. The refund task uses aggressive retries with exponential backoff because it is critical that the customer receives their money back.
## Retry strategies
When a task fails, Conductor can automatically retry it according to the retry logic configured on the task definition. You control the retry behavior with three parameters:
* **`retryCount`** — Maximum number of retry attempts.
* **`retryLogic`** — The backoff strategy between retries.
* **`retryDelaySeconds`** — The base delay between retries, in seconds.
### FIXED
Retries at a constant interval. Every retry waits the same amount of time.
```json
{
"retryCount": 3,
"retryLogic": "FIXED",
"retryDelaySeconds": 5
}
```
This retries up to 3 times, waiting exactly 5 seconds between each attempt.
### EXPONENTIAL_BACKOFF
Each retry waits exponentially longer than the previous one. The delay is calculated as `retryDelaySeconds * 2^(attemptNumber)`. This reduces load on downstream services that may be experiencing pressure.
```json
{
"retryCount": 4,
"retryLogic": "EXPONENTIAL_BACKOFF",
"retryDelaySeconds": 2
}
```
This retries up to 4 times with delays of approximately 2, 4, 8, and 16 seconds.
### LINEAR_BACKOFF
Each retry waits incrementally longer by a fixed amount. The delay is calculated as `retryDelaySeconds * attemptNumber`. This provides a gentler ramp-up than exponential backoff.
```json
{
"retryCount": 4,
"retryLogic": "LINEAR_BACKOFF",
"retryDelaySeconds": 5
}
```
This retries up to 4 times with delays of approximately 5, 10, 15, and 20 seconds.
### Choosing a retry strategy
| Strategy | Delay pattern | Best for |
|---|---|---|
| `FIXED` | Constant (e.g., 5s, 5s, 5s) | Predictable transient failures like brief network blips or short-lived lock contention. |
| `EXPONENTIAL_BACKOFF` | Doubling (e.g., 2s, 4s, 8s, 16s) | Rate-limited APIs, overloaded services, or any case where you want to reduce pressure on a struggling dependency. |
| `LINEAR_BACKOFF` | Incremental (e.g., 5s, 10s, 15s, 20s) | Moderate recovery scenarios where you need longer waits over time but exponential growth would be too aggressive. |
## Task-level error handling
Beyond retries, Conductor provides several task-level controls for managing failures within a running workflow.
### Optional tasks
Setting `optional` to `true` on a task tells Conductor to continue the workflow even if that task fails after exhausting all retries. The workflow will proceed to the next task rather than failing entirely.
```json
{
"name": "send_analytics_event",
"taskReferenceName": "send_analytics_ref",
"type": "SIMPLE",
"optional": true,
"retryCount": 2,
"retryLogic": "FIXED",
"retryDelaySeconds": 3
}
```
Use optional tasks for non-critical side effects like logging, analytics, or notifications where a failure should not block the primary business logic.
### Failing immediately with terminal errors
When a worker encounters an error that no amount of retrying will fix, such as invalid input data or a business rule violation, it should return a `FAILED_WITH_TERMINAL_ERROR` status. This tells Conductor to skip all remaining retries and fail the task immediately.
Workers signal this by setting the task status to `FAILED_WITH_TERMINAL_ERROR` in the task result. This avoids wasting time on retries when the failure is deterministic. For example, if a payment is declined due to insufficient funds, retrying the same charge will never succeed.
### Per-task timeout configuration
You can set timeouts on individual tasks to prevent them from blocking the workflow indefinitely:
```json
{
"name": "call_external_api",
"taskReferenceName": "call_api_ref",
"type": "SIMPLE",
"timeoutSeconds": 120,
"responseTimeoutSeconds": 60,
"timeoutPolicy": "RETRY"
}
```
* **`timeoutSeconds`** — Maximum total time for the task, including all retries.
* **`responseTimeoutSeconds`** — Maximum time to wait for a worker to pick up and respond to the task. If a worker does not update the task within this window, Conductor marks it as timed out.
## Timeout policies
Timeout policies determine what Conductor does when a task exceeds its `timeoutSeconds` or `responseTimeoutSeconds` limit.
### RETRY
Re-queue the task for another attempt. The retry counts against the task's `retryCount`.
```json
{
"timeoutPolicy": "RETRY",
"timeoutSeconds": 60,
"retryCount": 3
}
```
### TIME_OUT_WF
Fail the entire workflow immediately when the task times out. Use this for tasks where a timeout indicates a critical problem that makes continuing the workflow pointless.
```json
{
"timeoutPolicy": "TIME_OUT_WF",
"timeoutSeconds": 300
}
```
### ALERT_ONLY
Log an alert but allow the task to continue running. The task is not terminated or retried. This is useful for long-running tasks where you want visibility into slow execution without interrupting work.
```json
{
"timeoutPolicy": "ALERT_ONLY",
"timeoutSeconds": 600
}
```
### Choosing a timeout policy
| Policy | Behavior on timeout | Best for |
|---|---|---|
| `RETRY` | Retries the task (counts against `retryCount`) | Tasks that may hang due to transient issues like network timeouts or unresponsive workers. |
| `TIME_OUT_WF` | Fails the entire workflow | Critical tasks where a timeout means the workflow cannot produce a valid result. |
| `ALERT_ONLY` | Logs an alert, task keeps running | Long-running or best-effort tasks where you want monitoring without enforcement. |
## Implement a Workflow Status Listener
Using a Workflow Status Listener, you can send a notification to an external system or an event to Conductor's internal queue upon failure. Here is the high-level overview for using a Workflow Status Listener:
1. Set the `workflowStatusListenerEnabled` parameter to true in your main workflow definition:
```json
"workflowStatusListenerEnabled": true,
```
2. Implement the [WorkflowStatusListener interface](https://github.com/conductor-oss/conductor/blob/1be02a711dc20682718c6111c09d2b02ce7edde2/core/src/main/java/com/netflix/conductor/core/listener/WorkflowStatusListener.java#L20) to plug into a custom notification or eventing system upon workflow failure.
docs/devguide/how-tos/Workflows/restarting-workflows-at-runtime.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 302 KiB

docs/devguide/how-tos/Workflows/scheduling-workflows.md
+196
View File
@@ -0,0 +1,196 @@
---
description: "Schedule workflows to run on a cron expression using Conductor's built-in scheduler. Create, pause, resume, and delete schedules via the REST API."
---
# Scheduling Workflows
Conductor includes a built-in scheduler that triggers workflow executions on a cron schedule. Schedules are managed through the REST API — no external cron daemon or job scheduler is needed.
## How it works
A **schedule** binds a cron expression to a `StartWorkflowRequest`. On every cron tick the scheduler starts a new workflow execution with the configured input. Two timestamps are automatically injected into every triggered workflow's input:
| Input key | Description |
|---|---|
| `_scheduledTime` | The exact cron slot time (epoch ms) |
| `_executedTime` | The actual dispatch time (epoch ms) |
## Cron expression format
Conductor uses Spring's 6-field cron format with **second-level precision**:
```
┌─────────────── second (0-59)
│ ┌───────────── minute (0-59)
│ │ ┌─────────── hour (0-23)
│ │ │ ┌───────── day of month (1-31)
│ │ │ │ ┌─────── month (1-12 or JAN-DEC)
│ │ │ │ │ ┌───── day of week (0-7 or MON-SUN)
│ │ │ │ │ │
* * * * * *
```
| Expression | Meaning |
|---|---|
| `0 * * * * *` | Every minute |
| `0 0 9 * * MON-FRI` | Weekdays at 9 AM |
| `0 0 0 1 * *` | First day of every month at midnight |
| `*/10 * * * * *` | Every 10 seconds |
## Creating a schedule
**1. Register the workflow** (if not already registered):
```shell
curl -X PUT 'http://localhost:8080/api/metadata/workflow' \
-H 'Content-Type: application/json' \
-d '[{
"name": "daily_report_workflow",
"version": 1,
"schemaVersion": 2,
"tasks": [{
"name": "fetch_report_data",
"taskReferenceName": "fetch_report_data_ref",
"type": "HTTP",
"inputParameters": {
"http_request": {
"uri": "https://jsonplaceholder.typicode.com/todos?userId=1",
"method": "GET"
}
}
}],
"timeoutPolicy": "TIME_OUT_WF",
"timeoutSeconds": 120
}]'
```
**2. Create the schedule:**
```shell
curl -X POST 'http://localhost:8080/api/scheduler/schedules' \
-H 'Content-Type: application/json' \
-d '{
"name": "daily-report-schedule",
"cronExpression": "0 0 9 * * MON-FRI",
"zoneId": "America/New_York",
"startWorkflowRequest": {
"name": "daily_report_workflow",
"version": 1,
"correlationId": "daily-report-${scheduledTime}"
},
"runCatchupScheduleInstances": false,
"paused": false
}'
```
The response returns the saved schedule object including its computed `nextRunTime`.
## Schedule definition fields
| Field | Type | Required | Description |
|---|---|---|---|
| `name` | string | Yes | Unique schedule identifier |
| `cronExpression` | string | Yes | 6-field Spring cron expression |
| `zoneId` | string | No | Timezone (default: `UTC`) |
| `startWorkflowRequest` | object | Yes | Workflow to trigger — includes `name`, `version`, `input`, `correlationId` |
| `runCatchupScheduleInstances` | boolean | No | Fire missed slots if the scheduler was offline (default: `false`) |
| `paused` | boolean | No | Create in paused state (default: `false`) |
| `scheduleStartTime` | long | No | Earliest time the schedule fires (epoch ms) |
| `scheduleEndTime` | long | No | Latest time the schedule fires (epoch ms) |
| `description` | string | No | Free-text description |
## Previewing execution times
Before creating a schedule, preview when it will fire:
```shell
curl 'http://localhost:8080/api/scheduler/nextFewSchedules?cronExpression=0+0+9+*+*+MON-FRI&limit=5'
```
Returns an array of epoch-millisecond timestamps for the next 5 execution times.
## Pausing and resuming
```shell
# Pause
curl -X PUT 'http://localhost:8080/api/scheduler/schedules/daily-report-schedule/pause'
# Pause with a reason
curl -X PUT 'http://localhost:8080/api/scheduler/schedules/daily-report-schedule/pause?reason=maintenance+window'
# Resume
curl -X PUT 'http://localhost:8080/api/scheduler/schedules/daily-report-schedule/resume'
```
## Listing and searching schedules
```shell
# List all schedules
curl 'http://localhost:8080/api/scheduler/schedules'
# Filter by workflow name
curl 'http://localhost:8080/api/scheduler/schedules?workflowName=daily_report_workflow'
# Search with pagination
curl 'http://localhost:8080/api/scheduler/schedules/search?workflowName=daily_report_workflow&size=10'
```
## Viewing execution history
```shell
curl 'http://localhost:8080/api/scheduler/search/executions?freeText=daily-report-schedule&size=20'
```
Returns a `SearchResult` with `totalHits` and a list of execution records, each containing the `scheduledTime`, `executionTime`, `workflowId`, and `state` (`POLLED`, `EXECUTED`, or `FAILED`).
## Deleting a schedule
```shell
curl -X DELETE 'http://localhost:8080/api/scheduler/schedules/daily-report-schedule'
```
## Passing input to scheduled workflows
Static input parameters are merged with the auto-injected `_scheduledTime` and `_executedTime`:
```json
{
"name": "input-param-demo-schedule",
"cronExpression": "0 * * * * *",
"zoneId": "UTC",
"startWorkflowRequest": {
"name": "input_param_demo_workflow",
"version": 1,
"input": {
"reportOwner": "platform-team",
"alertThreshold": 100
}
}
}
```
Inside the workflow, access all values via `${workflow.input.*}`:
- `${workflow.input.reportOwner}` — your static input
- `${workflow.input._scheduledTime}` — injected cron slot time
- `${workflow.input._executedTime}` — injected actual dispatch time
## Configuration
The scheduler is configured under the `conductor.scheduler` prefix in your application properties:
| Property | Default | Description |
|---|---|---|
| `conductor.scheduler.enabled` | `true` | Enable/disable the scheduler |
| `conductor.scheduler.pollingInterval` | `100` | Poll interval in milliseconds |
| `conductor.scheduler.pollBatchSize` | `5` | Schedules processed per poll cycle |
| `conductor.scheduler.pollingThreadCount` | `1` | Number of polling threads |
| `conductor.scheduler.schedulerTimeZone` | `UTC` | Default timezone |
| `conductor.scheduler.initialDelayMs` | `15000` | Startup delay before first poll |
| `conductor.scheduler.maxScheduleJitterMs` | `1000` | Random jitter added to dispatch times to smooth load |
!!! note "Catchup mode"
When `runCatchupScheduleInstances` is `true`, the scheduler fires all cron slots that were missed while it was offline. Use this for workflows where every execution matters (e.g., billing, compliance). Leave it `false` (default) for dashboards or monitoring where only the latest run matters.
!!! warning "Concurrent executions"
The scheduler fires on every cron tick regardless of whether the previous execution has completed. If your workflow takes longer than the cron interval, multiple instances will run concurrently. Design your workflows to handle this, or use a longer interval.
docs/devguide/how-tos/Workflows/searching-workflows.md
+53
View File
@@ -0,0 +1,53 @@
---
description: "Searching Workflows — find Conductor workflow executions by name, status, time range, or task parameters in the UI."
---
# Searching Workflows
The Conductor UI provides a convenient interface for searching workflow executions. There are two modes of searching:
* **Workflows** tab — Search using workflow parameters.
* **Tasks** tab — Search workflows by tasks.
**To search workflow executions:**
1. Go to **[Executions](http://localhost:8080/executions)** in the Conductor UI.
2. Configure the [search parameters](#search-parameters).
3. Select **Search**.
Once the search results are displayed, you can sort the results by different column values and select additional columns to display.
## Search parameters
Here are the search parameters for each search mode.
### Search by workflows
The following fields are available for searching workflows in the **Workflows** tab.
| Search Field Name | Description |
|-------------------|---------------------------------------------------------------------------------------------------------|
| Workflow Name | Filters workflow executions by its name. |
| Workflow ID | Filters to a specific workflow execution by its execution ID. |
| Status | Filters workflow executions by its status (RUNNING, COMPLETED, FAILED, TIMED_OUT, TERMINATED, PAUSED). |
| Start Time - From | Filters workflow executions that started on or after the specified time. |
| Start Time - To | Filters workflow executions that started on or before the specified time. |
| Lookback (days) | Filters workflow executions that ran in the last given number of days. |
| Lucene-syntax Query (Double-quote strings for Free Text) | (If indexing is enabled) Filters workflow executions by querying workflow input and output values. |
### Search workflows by tasks
The following fields are available for searching workflows by its tasks in the **Tasks** tab.
| Search Field Name | Description |
|--------------------|--------------------------------------------------------------------------------------------------------------|
| Task Name | Filters workflow executions by its task name. |
| Task ID | Filters to a specific workflow execution that contains this task execution ID. |
| Task Status | Filters workflow executions by its task status (IN_PROGRESS, CANCELED, FAILED, FAILED_WITH_TERMINAL_ERROR, COMPLETED, COMPLETED_WITH_ERRORS, SCHEDULED, TIMED_OUT, SKIPPED). |
| Task Type | Filters workflow executions by its task type. |
| Workflow Name | Filters workflow executions by its workflow name. |
| Update Time - From | Filters workflow executions by tasks that started on or after the specified time. |
| Update Time - To | Filters workflow executions by tasks that started on or before the specified time. |
| Lookback (days) | Filters workflow executions by tasks that ran in the last given number of days. |
| Lucene-syntax Query (Double-quote strings for Free Text) | (If indexing is enabled) Filters workflow executions by querying task input and output values. |
docs/devguide/how-tos/Workflows/starting-workflows.md
+68
View File
@@ -0,0 +1,68 @@
---
description: "Start workflow executions in Conductor using the UI, CLI, REST APIs, or client SDKs. Pass inputs and track executions with a unique workflow ID."
---
# Starting Workflows
In Conductor, workflows can be started using the Conductor UI, APIs, or SDKs.
## Using Conductor UI
The Conductor UI is useful for sandbox testing before deploying the workflows to production using the APIs or SDKs.
**To start a workflow:**
1. Go to [Workbench](http://localhost:8080/workbench) in the Conductor UI.
2. Select the **Workflow Name** and **Workflow version**.
3. If required, provide the workflow inputs in **Input (JSON)**.
4. (Optional) Specify the **Correlation ID** and **Task to Domain (JSON)** for the execution.
5. Select the ▶ icon (Execute Workflow) at the top to run the workflow.
Once the workflow has started, you can view the ongoing execution by selecting the Workflow ID hyperlink in the **Execution History** side panel on the right.
## Using the CLI
You can start workflow executions using the Conductor CLI.
### Example using the CLI
In this example, the CLI is used to invoke the workflow `sample_workflow` with the input `service` specified as `fedex`.
```bash
conductor workflow start -w sample_workflow -i '{"service":"fedex"}'
```
## Using APIs
You can also start workflow executions using the Start Workflow API (`POST api/workflow/{name}`). `{name}` is the placeholder for the workflow name, and the request body contains the workflow inputs if any.
??? note "Example using cURL"
In this example, a cURL request is used to invoke the workflow `sample_workflow` with the input `service` specified as `fedex`.
```bash
curl '{{ server_host }}/api/workflow/sample_workflow' \
-H 'accept: text/plain' \
-H 'content-type: application/json' \
--data-raw '{"service":"fedex"}'
```
## Using SDKs
Conductor offers client SDKs for popular languages which have library methods for making the Start Workflow API call. Refer to the SDK documentation to configure a client in your selected language to invoke workflow executions.
### Example using JavaScript
In this example, the JavaScript Fetch API is used to invoke the workflow `sample_workflow` with the input `service` specified as `fedex`.
```javascript
fetch("{{ server_host }}/api/workflow/sample_workflow", {
"headers": {
"accept": "text/plain",
"content-type": "application/json",
},
"body": "{\"service\":\"fedex\"}",
"method": "POST",
});
```
docs/devguide/how-tos/Workflows/versioning-workflows.md
+65
View File
@@ -0,0 +1,65 @@
---
description: "Versioning Workflows — safely run multiple Conductor workflow versions side by side without disrupting production."
---
# Versioning Workflows
Conductor allows you to safely run different workflow versions without disrupting ongoing or scheduled workflow executions in production.
Refer to [Updating workflows](creating-workflows.md#updating-workflows) for more information on modifying a workflow and saving it as a new version.
## When to version workflows
Workflow versioning is useful for various scenarios, like gradually upgrading a process, or rolling out different workflow versions to different user bases.
### Example
For example, a new version of your core workflow will add a capability that is required for _customerA_. However, _customerB_ will not be ready to implement this code for another 6 months.
With workflow versioning, you can begin transitioning traffic onto version 2 for _customerA_, while _customerB_ remains on version 1. 6 months later, _customerB_ can begin transitioning traffic to version 2 as well.
## Runtime behavior with multiple workflow versions
At runtime, all Conductor workflows will reference a snapshot of the workflow definition at the start of its invocation. In other words, all changes to a workflow definition are decoupled from all of its ongoing workflow executions.
Here is an illustration of workflow versions at runtime, when you run workflows based on the latest version, versus when you run workflows based on a specific version.
![Diagram of a workflow definition's versions compared to its execution version at different points in time.](workflow-versioning-at-runtime.jpg)
In the illustration above, the workflow with version V1 is executed at timestamp T1 and thus uses the workflow definition at that time.
At T2, a new workflow version V2 is created. From that point on, any newly-triggered executions using the latest version will run based on V2, even if V1 gets updated later on at T3.
At T3, even when the V1 workflow definition is updated, all existing V1 executions will continue based on the definition at timestamp T1. From that point on, any newly-triggered executions using version V1 will run based on the V1 definition at timestamp T3.
### Runtime behavior during restarts
Likewise, by default all workflow restarts, workflow retries, and task reruns will be executed based on the snapshot of the workflow definition at the start of the _first_ execution attempt. If required, you can choose to restart workflows with the latest definitions.
Here is an illustration of workflow versions at runtime, when you restart workflows using the current definitions versus using the latest definitions.
![Diagram of workflow versions at runtime when restarting executions.](restarting-workflows-at-runtime.jpg)
In the illustration above, if a V1 execution is restarted with the current definitions after a new version V2 has been created, the restarted execution will still run based on the V1 definition at T1. This applies even if the same execution is restarted after the V1 definition itself has been updated at T3.
At T2, if a V1 execution is restarted with the latest definitions, the V1 execution will restart using the V2 definition instead. This also applies even if the same execution is restarted after the V1 definition itself has been updated at T3.
## Upgrading running workflows
Since any changes to a workflow definition will not impact its ongoing executions, running workflows need to be explicitly upgraded if required.
Using the Conductor UI or APIs, you can upgrade a running workflow by terminating the execution and restarting it with the latest definition.
### Using Conductor UI
**To upgrade a running workflow**:
1. In **[Executions](http://localhost:8080/executions)**, select an ongoing workflow to upgrade.
2. In the top right, select **Actions** > **Terminate**.
3. Once terminated, select **Actions** > **Restart with Latest Definitions**.
### Using Conductor APIs
The API approach allows you to upgrade running workflows in bulk. Use the Bulk Terminate API (`POST /api/workflow/bulk/terminate`) to specify a list of ongoing workflows. Then, use the Bulk Restart API (`POST /api/workflow/bulk/restart`) to restart the terminated workflows.
docs/devguide/how-tos/Workflows/viewing-workflow-executions.md
+57
View File
@@ -0,0 +1,57 @@
---
description: "Viewing Workflow Executions — inspect Conductor workflow runs with visual diagrams, task timelines, and input/output data."
---
# Viewing Workflow Executions
The Conductor UI provides a convenient interface for viewing workflow executions as visual diagrams. You can view workflow executions:
- In **[Executions](http://localhost:8080/executions)**, after [searching for workflows](searching-workflows.md).
- In **[Workbench](http://localhost:8080/workbench)** > **Execution History**
**To view a workflow execution:**
In **[Executions](http://localhost:8080/executions)** or **[Workbench](http://localhost:8080/workbench)**, select the Workflow ID hyperlink.
## Workflow execution details
The following tabs are available for each workflow execution:
| Tab Name | Description |
|----------------------------|-------------------------------------------------------------------------------------------------------------------|
| **Tasks** > **Diagram** | Visual diagram of the workflow and its tasks. |
| **Tasks** > **Task List** | List of the task executions in this workflow, including details like the task name, task ID, status, and so on. |
| **Tasks** > **Timeline** | Timeline showcasing the duration and sequence of each task in the workflow. |
| **Summary** | Summary view of the workflow execution, which includes the workflow ID, status, duration, and so on. |
| **Workflow Input/Output** | View of the JSON payload for the workflow inputs, outputs, and variables. |
| **JSON** | View of the full workflow execution JSON, including all tasks, inputs, outputs, and so on. |
### Workflow diagram view
In **Tasks** > **Diagram**, you can view the workflow's exact execution path. The executed paths are shown in green and while other alternative paths are greyed out.
![Workflow diagram in the Conductor UI.](execution_path.png)
Each task status will also be clearly marked, highlighting any task errors.
![Task statuses are visually represented in the workflow diagram.](workflow-task-states.jpg)
### Task execution details
You can also view a task's execution details by selecting a task from the following tabs:
- **Tasks** > **Diagram**
- **Tasks** > **Task List**
- **Tasks** > **Timeline**
This action opens a left-side panel that contains the following tabs:
| Tab Name | Description |
|------------|-----------------------------------------------------------------------------------------------------------------------------------------------------|
| **Summary** | Summary view of the task execution, which includes the task execution ID, status, duration, and so |
| **Input** | View of the JSON payload for the task inputs. |
| **Output** | View of the JSON payload for the task outputs. |
| **Logs** | View of the log messages logged by the task, if any. |
| **JSON** | View of the full task execution JSON, including retry count, start time, worker ID, and so on. |
| **Definition** | View of the task configuration used when executing the task. |
docs/devguide/how-tos/Workflows/workflow-task-states.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 184 KiB

docs/devguide/how-tos/Workflows/workflow-versioning-at-runtime.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 298 KiB

docs/devguide/how-tos/Workflows/workflow_debugging.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 499 KiB

docs/devguide/how-tos/conductor-skills.md
+283
View File
@@ -0,0 +1,283 @@
---
description: "Conductor Skills — teach your AI coding agent to create, run, monitor, and manage Conductor workflows. Works with Claude Code, Cursor, Copilot, Gemini CLI, and more."
---
# Build with AI agents
Conductor Skills teaches your AI coding agent to create, run, monitor, and manage Conductor workflows. Instead of writing JSON definitions and CLI commands by hand, describe what you want in natural language and your agent builds it for you — complete workflows, workers, error handling, and monitoring.
Works with Claude Code, Cursor, GitHub Copilot, Gemini CLI, Codex, Windsurf, Cline, Amazon Q, Aider, Roo Code, Amp, and OpenCode.
## Install
One command installs for all detected agents on your system:
=== "macOS / Linux"
```bash
curl -sSL https://conductor-oss.github.io/conductor-skills/install.sh | bash -s -- --all
```
=== "Windows (PowerShell)"
```powershell
irm https://conductor-oss.github.io/conductor-skills/install.ps1 -OutFile install.ps1; .\install.ps1 -All
```
To install for a specific agent only:
```bash
curl -sSL https://conductor-oss.github.io/conductor-skills/install.sh | bash -s -- --agent claude
```
## Connect to your server
After installing, tell your agent where your Conductor server is:
> *"Connect to my Conductor server at http://localhost:8080/api"*
Or set the environment variable directly:
```bash
export CONDUCTOR_SERVER_URL=http://localhost:8080/api
```
## What your agent can do
Once installed, your AI agent can:
| Capability | What you say | What happens |
|---|---|---|
| **Create workflows** | *"Create a workflow that calls the GitHub API and sends a Slack notification"* | Agent generates the full workflow definition with HTTP tasks, input expressions, and output parameters |
| **Run workflows** | *"Run my-workflow with input userId 123"* | Agent starts the execution and returns the execution ID |
| **Monitor executions** | *"Show me all failed workflows from the last hour"* | Agent searches executions by status, time, or correlation ID |
| **Debug failures** | *"What went wrong with execution abc-123?"* | Agent retrieves the execution, identifies the failed task, and shows the error |
| **Retry and recover** | *"Retry all failed executions of order-processing"* | Agent batch-retries failed executions |
| **Manage lifecycle** | *"Pause execution xyz-456"* | Agent pauses, resumes, terminates, or restarts workflows |
| **Signal tasks** | *"Approve the payment wait task in execution abc-123"* | Agent signals WAIT or HUMAN tasks to advance the workflow |
| **Write workers** | *"Write a Python worker that validates email addresses"* | Agent generates worker code using the appropriate SDK |
| **Visualize** | *"Show me a diagram of the order-processing workflow"* | Agent renders a Mermaid diagram of the workflow |
## Walkthrough: build an order processing system
This walkthrough shows how to build a complete application using Conductor as the backend — entirely through natural language prompts to your AI agent.
### Step 1: Create the workflow
> *"Create an order processing workflow with these steps: validate the order, check inventory, charge payment, and fulfill the order. If payment fails, compensate by releasing the inventory hold. Add a WAIT task before payment so a human can review high-value orders."*
Your agent creates the workflow definition:
```json
{
"name": "order_processing",
"description": "Process customer orders with inventory check, payment, and fulfillment",
"version": 1,
"schemaVersion": 2,
"inputParameters": ["orderId", "customerId", "items", "totalAmount"],
"tasks": [
{
"name": "validate_order",
"taskReferenceName": "validate",
"type": "HTTP",
"inputParameters": {
"http_request": {
"uri": "https://api.example.com/orders/${workflow.input.orderId}/validate",
"method": "POST",
"body": { "items": "${workflow.input.items}" }
}
}
},
{
"name": "check_inventory",
"taskReferenceName": "inventory",
"type": "HTTP",
"inputParameters": {
"http_request": {
"uri": "https://api.example.com/inventory/hold",
"method": "POST",
"body": { "items": "${workflow.input.items}" }
}
}
},
{
"name": "review_gate",
"taskReferenceName": "review_gate",
"type": "SWITCH",
"evaluatorType": "javascript",
"expression": "$.workflow.input.totalAmount > 500 ? 'needs_review' : 'auto_approve'",
"decisionCases": {
"needs_review": [
{
"name": "human_review",
"taskReferenceName": "review",
"type": "WAIT",
"inputParameters": {
"orderId": "${workflow.input.orderId}",
"amount": "${workflow.input.totalAmount}"
}
}
]
},
"defaultCase": []
},
{
"name": "charge_payment",
"taskReferenceName": "payment",
"type": "HTTP",
"inputParameters": {
"http_request": {
"uri": "https://api.example.com/payments/charge",
"method": "POST",
"body": {
"customerId": "${workflow.input.customerId}",
"amount": "${workflow.input.totalAmount}",
"holdId": "${inventory.output.response.body.holdId}"
}
}
}
},
{
"name": "fulfill_order",
"taskReferenceName": "fulfillment",
"type": "HTTP",
"inputParameters": {
"http_request": {
"uri": "https://api.example.com/fulfillment/ship",
"method": "POST",
"body": {
"orderId": "${workflow.input.orderId}",
"items": "${workflow.input.items}",
"paymentId": "${payment.output.response.body.paymentId}"
}
}
}
}
],
"failureWorkflow": "order_processing_compensation",
"outputParameters": {
"orderId": "${workflow.input.orderId}",
"paymentId": "${payment.output.response.body.paymentId}",
"trackingNumber": "${fulfillment.output.response.body.trackingNumber}"
}
}
```
The agent registers the workflow automatically.
### Step 2: Create the compensation workflow
> *"Create the compensation workflow for order_processing. It should release the inventory hold and refund the payment if it was charged."*
Your agent creates `order_processing_compensation` with the reverse operations.
### Step 3: Write a custom worker
> *"Write a Python worker that validates orders by checking that all items exist and quantities are positive"*
Your agent generates the worker code using the Conductor Python SDK:
```python
from conductor.client.worker.worker_task import worker_task
@worker_task(task_definition_name="validate_order")
def validate_order(task):
items = task.input_data.get("items", [])
for item in items:
if not item.get("productId"):
return {"valid": False, "reason": f"Missing productId"}
if item.get("quantity", 0) <= 0:
return {"valid": False, "reason": f"Invalid quantity for {item['productId']}"}
return {"valid": True, "itemCount": len(items)}
```
### Step 4: Run the workflow
> *"Run order_processing with orderId ORD-001, customerId CUST-42, items [{productId: SKU-100, quantity: 2}], totalAmount 750"*
```
Workflow started.
- Execution ID: f8a2b3c4-d5e6-7890-abcd-ef1234567890
- Status: RUNNING
- The order total ($750) exceeds $500, so it's waiting for human review.
```
### Step 5: Approve the review
> *"Approve the review task in execution f8a2b3c4"*
```
Task signaled: review → COMPLETED
Workflow is now executing charge_payment.
```
### Step 6: Monitor and debug
> *"Show me all failed order_processing executions from today"*
```
Found 2 failed executions:
1. exec-abc — Failed at charge_payment (HTTP 402: Insufficient funds)
2. exec-def — Failed at check_inventory (HTTP 409: Item SKU-200 out of stock)
```
> *"Retry exec-abc"*
```
Execution exec-abc retried. Status: RUNNING.
```
### Step 7: Visualize
> *"Show me a diagram of order_processing"*
Your agent renders:
```mermaid
graph LR
A[validate_order] --> B[check_inventory]
B --> C{totalAmount > 500?}
C -->|Yes| D[human_review WAIT]
C -->|No| E[charge_payment]
D --> E
E --> F[fulfill_order]
```
## Supported agents
| Agent | Install flag | Global install | Project install |
|---|---|---|---|
| [Claude Code](https://docs.anthropic.com/en/docs/claude-code) | `claude` | Native skill | — |
| [Codex CLI](https://github.com/openai/codex) | `codex` | `~/.codex/AGENTS.md` | `AGENTS.md` |
| [Gemini CLI](https://github.com/google-gemini/gemini-cli) | `gemini` | `~/.gemini/GEMINI.md` | `GEMINI.md` |
| [Cursor](https://cursor.com) | `cursor` | `~/.cursor/skills/` | `.cursor/rules/` |
| [Windsurf](https://codeium.com/windsurf) | `windsurf` | `~/.codeium/windsurf/` | `.windsurfrules` |
| [GitHub Copilot](https://github.com/features/copilot) | `copilot` | — | `.github/copilot-instructions.md` |
| [Cline](https://github.com/cline/cline) | `cline` | — | `.clinerules` |
| [Amazon Q](https://aws.amazon.com/q/developer/) | `amazonq` | — | `.amazonq/rules/` |
| [Aider](https://aider.chat) | `aider` | `~/.conductor-skills/` | `.conductor-skills/` |
| [Roo Code](https://github.com/RooVetGit/Roo-Code) | `roo` | `~/.roo/rules/` | `.roo/rules/` |
| [Amp](https://ampcode.com) | `amp` | `~/.config/AGENTS.md` | `.amp/instructions.md` |
| [OpenCode](https://opencode.ai) | `opencode` | `~/.config/opencode/skills/` | `AGENTS.md` |
## Upgrade
```bash
curl -sSL https://conductor-oss.github.io/conductor-skills/install.sh | bash -s -- --all --upgrade
```
## Next steps
- **[conductor-skills repository](https://github.com/conductor-oss/conductor-skills)** &mdash; Full documentation, more examples, and source code.
- **[Quickstart](../../quickstart/index.md)** &mdash; Get a Conductor server running to use with your agent.
- **[AI & Agents](../ai/index.md)** &mdash; Build durable AI agent workflows on Conductor.
- **[Client SDKs](../../documentation/clientsdks/index.md)** &mdash; Language SDKs for writing workers and programmatic access.
docs/devguide/how-tos/event-bus.md
+234
View File
@@ -0,0 +1,234 @@
---
description: "Orchestrate event-driven workflows with Conductor using Kafka, NATS, AMQP (RabbitMQ), and SQS as event buses. Configure event handlers to trigger workflows, complete tasks, or fail tasks on incoming events."
---
# Event Bus Orchestration
Conductor integrates with external messaging systems to enable event-driven workflow orchestration. You can publish events from workflows and react to external events — starting workflows, completing tasks, or failing tasks based on incoming messages.
## Supported event buses
| System | Sink prefix | Module | Use case |
| :--- | :--- | :--- | :--- |
| **Kafka** | `kafka` | `kafka` | High-throughput, durable event streaming |
| **NATS** | `nats` | `nats` | Lightweight, low-latency messaging |
| **NATS Streaming** | `nats-stream` | `nats-streaming` | Durable NATS with replay (legacy) |
| **NATS JetStream** | `nats` | `nats` | Modern durable NATS streaming |
| **AMQP (RabbitMQ)** | `amqp`, `amqp_queue`, `amqp_exchange` | `amqp` | Traditional message queuing with routing |
| **SQS** | `sqs` | `sqs` | AWS-native message queuing |
| **Conductor** | `conductor` | built-in | Internal event routing between workflows |
## How it works
Event bus orchestration has two sides:
1. **Publishing** — Use the [Event task](../../documentation/configuration/workflowdef/systemtasks/event-task.md) or [Kafka Publish task](../../documentation/configuration/workflowdef/systemtasks/kafka-publish-task.md) to send messages from a workflow.
2. **Consuming** — Register [event handlers](../../documentation/configuration/eventhandlers.md) that listen for messages and trigger actions.
```
┌──────────────┐ Event Task ┌──────────────┐ Event Handler ┌──────────────┐
│ Workflow A │ ──────────────────► │ Event Bus │ ──────────────────► │ Workflow B │
│ │ (publish) │ (Kafka/NATS/ │ (start_workflow) │ (triggered) │
│ │ │ AMQP/SQS) │ │ │
└──────────────┘ └──────────────┘ └──────────────┘
```
## Publishing events
### Event task
The [Event task](../../documentation/configuration/workflowdef/systemtasks/event-task.md) publishes a message to any supported event bus. The `sink` parameter determines the target:
```json
{
"name": "notify_downstream",
"taskReferenceName": "notify_ref",
"type": "EVENT",
"sink": "kafka:order-events",
"inputParameters": {
"orderId": "${workflow.input.orderId}",
"status": "PROCESSED"
}
}
```
### Kafka Publish task
For Kafka-specific features (custom headers, key, serializers), use the dedicated [Kafka Publish task](../../documentation/configuration/workflowdef/systemtasks/kafka-publish-task.md):
```json
{
"name": "publish_to_kafka",
"taskReferenceName": "kafka_ref",
"type": "KAFKA_PUBLISH",
"inputParameters": {
"kafka_request": {
"topic": "order-events",
"value": "${workflow.input.orderData}",
"bootStrapServers": "kafka:9092",
"headers": {
"X-Correlation-Id": "${workflow.correlationId}"
}
}
}
}
```
### Sink format
The `sink` parameter follows the format `prefix:queue_name`:
| Example | System |
| :--- | :--- |
| `kafka:order-events` | Kafka topic `order-events` |
| `nats:notifications` | NATS subject `notifications` |
| `amqp:task-queue` | AMQP queue `task-queue` |
| `amqp_exchange:events` | AMQP exchange `events` |
| `sqs:my-queue` | SQS queue `my-queue` |
| `conductor` | Conductor internal queue |
| `conductor:workflow_name:queue_name` | Conductor internal, specific queue |
## Consuming events
### Event handlers
Event handlers listen for messages on an event bus and execute actions when a matching event arrives. Register them via the `/api/event` API.
```json
{
"name": "order_event_handler",
"event": "kafka:order-events",
"condition": "$.status == 'PROCESSED'",
"actions": [
{
"action": "start_workflow",
"start_workflow": {
"name": "fulfillment_workflow",
"input": {
"orderId": "${orderId}"
}
}
}
]
}
```
### Supported actions
| Action | Description |
| :--- | :--- |
| `start_workflow` | Start a new workflow execution with the event payload as input. |
| `complete_task` | Complete a waiting task (e.g., a `WAIT` or `HUMAN` task) in a running workflow. |
| `fail_task` | Fail a task in a running workflow. |
### Conditions
The `condition` field supports JavaScript-like expressions evaluated against the event payload:
| Expression | Result |
| :--- | :--- |
| `$.version > 1` | true if `version` field > 1 |
| `$.metadata.codec == 'aac'` | true if nested field matches |
| `$.status == 'COMPLETED'` | true if status is COMPLETED |
Actions execute only when the condition evaluates to `true`. If no condition is specified, actions execute for every event.
## Patterns
### Event-driven workflow chaining
Decouple workflows using events instead of sub-workflows:
```json
{
"name": "order_pipeline",
"tasks": [
{
"name": "process_order",
"taskReferenceName": "process_ref",
"type": "SIMPLE"
},
{
"name": "notify_fulfillment",
"taskReferenceName": "notify_ref",
"type": "EVENT",
"sink": "kafka:fulfillment-requests",
"inputParameters": {
"orderId": "${workflow.input.orderId}",
"items": "${process_ref.output.items}"
}
}
]
}
```
A separate event handler starts the fulfillment workflow when the event arrives.
### Wait for external event
Combine a `WAIT` task with an event handler to pause a workflow until an external system signals completion:
```json
{
"name": "wait_for_approval",
"taskReferenceName": "approval_ref",
"type": "WAIT"
}
```
Register an event handler that completes the task when an approval event arrives:
```json
{
"name": "approval_handler",
"event": "kafka:approval-events",
"condition": "$.approved == true",
"actions": [
{
"action": "complete_task",
"complete_task": {
"workflowId": "${workflowId}",
"taskRefName": "approval_ref",
"output": {
"approvedBy": "${approvedBy}"
}
}
}
]
}
```
## Configuration
Each event bus module requires its own configuration. Enable the modules you need in your Conductor server configuration:
### Kafka
```properties
conductor.event-queues.kafka.enabled=true
conductor.event-queues.kafka.bootstrap-servers=kafka:9092
```
### NATS
```properties
conductor.event-queues.nats.enabled=true
conductor.event-queues.nats.url=nats://localhost:4222
```
### AMQP (RabbitMQ)
```properties
conductor.event-queues.amqp.enabled=true
conductor.event-queues.amqp.hosts=rabbitmq
conductor.event-queues.amqp.port=5672
conductor.event-queues.amqp.username=guest
conductor.event-queues.amqp.password=guest
```
Refer to the module source code for the full set of configuration properties.
docs/devguide/labs/eventhandlers.md
+177
View File
@@ -0,0 +1,177 @@
---
description: "Event Handlers Lab — hands-on tutorial for publishing events and triggering Conductor workflows with event handlers."
---
# Events and Event Handlers
In this exercise, we shall:
* Publish an Event to Conductor using `Event` task.
* Subscribe to Events, and perform actions:
* Start a Workflow
* Complete Task
Conductor supports eventing with two Interfaces:
* [Event Task](../../documentation/configuration/workflowdef/systemtasks/event-task.md)
* [Event Handlers](../../documentation/configuration/eventhandlers.md)
## Create Workflow Definitions
Let's create two workflows:
* `test_workflow_for_eventHandler` which will have an `Event` task to start another workflow, and a `WAIT` System task that will be completed by an event.
* `test_workflow_startedBy_eventHandler` which will have an `Event` task to generate an event to complete `WAIT` task in the above workflow.
Send `POST` requests to `/metadata/workflow` endpoint with below payloads:
```json
{
"name": "test_workflow_for_eventHandler",
"description": "A test workflow to start another workflow with EventHandler",
"version": 1,
"tasks": [
{
"name": "test_start_workflow_event",
"taskReferenceName": "start_workflow_with_event",
"type": "EVENT",
"sink": "conductor"
},
{
"name": "test_task_tobe_completed_by_eventHandler",
"taskReferenceName": "test_task_tobe_completed_by_eventHandler",
"type": "WAIT"
}
]
}
```
```json
{
"name": "test_workflow_startedBy_eventHandler",
"description": "A test workflow which is started by EventHandler, and then goes on to complete task in another workflow.",
"version": 1,
"tasks": [
{
"name": "test_complete_task_event",
"taskReferenceName": "complete_task_with_event",
"inputParameters": {
"sourceWorkflowId": "${workflow.input.sourceWorkflowId}"
},
"type": "EVENT",
"sink": "conductor"
}
]
}
```
### Event Tasks in Workflow
`EVENT` task is a System task, and we shall define it just like other Tasks in Workflow, with `sink` parameter. Also, `EVENT` task doesn't have to be registered before using in Workflow. This is also true for the `WAIT` task.
Hence, we will not be registering any tasks for these workflows.
### Events are sent, but they're not handled (yet)
Once you try to start `test_workflow_for_eventHandler` workflow, you would notice that the event is sent successfully, but the second worflow `test_workflow_startedBy_eventHandler` is not started. We have sent the Events, but we also need to define `Event Handlers` for Conductor to take any `actions` based on the Event. Let's create `Event Handlers`.
## Create Event Handlers
Event Handler definitions are pretty much like Task or Workflow definitions. We start by name:
```json
{
"name": "test_start_workflow"
}
```
Event Handler should know the Queue it has to listen to. This should be defined in `event` parameter.
When using Conductor queues, define `event` with format:
```conductor:{workflow_name}:{taskReferenceName}```
And when using SQS, define with format:
```sqs:{my_sqs_queue_name}```
```json
{
"name": "test_start_workflow",
"event": "conductor:test_workflow_for_eventHandler:start_workflow_with_event"
}
```
Event Handler can perform a list of actions defined in `actions` array parameter, for this particular `event` queue.
```json
{
"name": "test_start_workflow",
"event": "conductor:test_workflow_for_eventHandler:start_workflow_with_event",
"actions": [
"<insert-actions-here>"
],
"active": true
}
```
Let's define `start_workflow` action. We shall pass the name of workflow we would like to start. The `start_workflow` parameter can use any of the values from the general [Start Workflow Request](../../documentation/api/startworkflow.md). Here we are passing in the workflowId, so that the Complete Task Event Handler can use it.
```json
{
"action": "start_workflow",
"start_workflow": {
"name": "test_workflow_startedBy_eventHandler",
"input": {
"sourceWorkflowId": "${workflowInstanceId}"
}
}
}
```
Send a `POST` request to `/event` endpoint:
```json
{
"name": "test_start_workflow",
"event": "conductor:test_workflow_for_eventHandler:start_workflow_with_event",
"actions": [
{
"action": "start_workflow",
"start_workflow": {
"name": "test_workflow_startedBy_eventHandler",
"input": {
"sourceWorkflowId": "${workflowInstanceId}"
}
}
}
],
"active": true
}
```
Similarly, create another Event Handler to complete task.
```json
{
"name": "test_complete_task_event",
"event": "conductor:test_workflow_startedBy_eventHandler:complete_task_with_event",
"actions": [
{
"action": "complete_task",
"complete_task": {
"workflowId": "${sourceWorkflowId}",
"taskRefName": "test_task_tobe_completed_by_eventHandler"
}
}
],
"active": true
}
```
## Summary
After wiring all of the above, starting the `test_workflow_for_eventHandler` should:
1. Start `test_workflow_startedBy_eventHandler` workflow.
2. Sets `test_task_tobe_completed_by_eventHandler` WAIT task `IN_PROGRESS`.
3. `test_workflow_startedBy_eventHandler` event task would publish an Event to complete the WAIT task above.
4. Both the workflows would move to `COMPLETED` state.
docs/devguide/labs/first-workflow.md
+151
View File
@@ -0,0 +1,151 @@
---
description: "First Workflow Lab — step-by-step tutorial to create and run your first Conductor workflow using built-in HTTP tasks."
---
# A First Workflow
In this article we will explore how we can run a really simple workflow that runs without deploying any new microservice.
Conductor can orchestrate HTTP services out of the box without implementing any code. We will use that to create and run the first workflow.
See [System Task](../../documentation/configuration/workflowdef/systemtasks/index.md) for the list of such built-in tasks.
Using system tasks is a great way to run a lot of our code in production.
## Configuring our First Workflow
This is a sample workflow that we can leverage for our test.
```json
{
"name": "first_sample_workflow",
"description": "First Sample Workflow",
"version": 1,
"tasks": [
{
"name": "get_population_data",
"taskReferenceName": "get_population_data",
"inputParameters": {
"http_request": {
"uri": "https://datausa.io/api/data?drilldowns=Nation&measures=Population",
"method": "GET"
}
},
"type": "HTTP"
}
],
"inputParameters": [],
"outputParameters": {
"data": "${get_population_data.output.response.body.data}",
"source": "${get_population_data.output.response.body.source}"
},
"schemaVersion": 2,
"restartable": true,
"workflowStatusListenerEnabled": false,
"ownerEmail": "example@email.com",
"timeoutPolicy": "ALERT_ONLY",
"timeoutSeconds": 0
}
```
This is an example workflow that queries a publicly available JSON API to retrieve some data. This workflow doesn’t
require any worker implementation as the tasks in this workflow are managed by the system itself. This is an awesome
feature of Conductor. For a lot of typical work, we won’t have to write any code at all.
Let's talk about this workflow a little more so that we can gain some context.
```json
"name" : "first_sample_workflow"
```
This line here is how we name our workflow. In this case our workflow name is `first_sample_workflow`
This workflow contains just one worker. The workers are defined under the key `tasks`. Here is the worker definition
with the most important values:
```json
{
"name": "get_population_data",
"taskReferenceName": "get_population_data",
"inputParameters": {
"http_request": {
"uri": "https://datausa.io/api/data?drilldowns=Nation&measures=Population",
"method": "GET"
}
},
"type": "HTTP"
}
```
Here is a list of fields and what it does:
1. `"name"` : Name of our worker
2. `"taskReferenceName"` : This is a reference to this worker in this specific workflow implementation. We can have multiple
workers of the same name in our workflow, but we will need a unique task reference name for each of them. Task
reference name should be unique across our entire workflow.
3. `"inputParameters"` : These are the inputs into our worker. We can hard code inputs as we have done here. We can
also provide dynamic inputs such as from the workflow input or based on the output of another worker. We can find
examples of this in our documentation.
4. `"type"` : This is what defines what the type of worker is. In our example - this is `HTTP`. There are more task
types which we can find in the Conductor documentation.
5. `"http_request"` : This is an input that is required for tasks of type `HTTP`. In our example we have provided a well
known internet JSON API url and the type of HTTP method to invoke - `GET`
We haven't talked about the other fields that we can use in our definitions as these are either just
metadata or more advanced concepts which we can learn more in the detailed documentation.
Ok, now that we have walked through our workflow details, let's run this and see how it works.
To configure the workflow, head over to the swagger API of conductor server and access the metadata workflow create API:
[http://{{ server_host }}/swagger-ui/index.html?configUrl=/api-docs/swagger-config#/metadata-resource/create](http://{{ server_host }}/swagger-ui/index.html?configUrl=/api-docs/swagger-config#/metadata-resource/create)
If the link doesn’t open the right Swagger section, we can navigate to Metadata-Resource
→ `POST {{ api_prefix }}/metadata/workflow`
![Swagger UI - Metadata - Workflow](metadataWorkflowPost.png)
Paste the workflow payload into the Swagger API and hit Execute.
Now if we head over to the UI, we can see this workflow definition created:
![Conductor UI - Workflow Definition](uiWorkflowDefinition.png)
If we click through we can see a visual representation of the workflow:
![Conductor UI - Workflow Definition - Visual Flow](uiWorkflowDefinitionVisual.png)
## Running our First Workflow
Let’s run this workflow. To do that we can use the swagger API under the workflow-resources
[http://{{ server_host }}/swagger-ui/index.html?configUrl=/api-docs/swagger-config#/workflow-resource/startWorkflow_1](http://{{ server_host }}/swagger-ui/index.html?configUrl=/api-docs/swagger-config#/workflow-resource/startWorkflow_1)
![Swagger UI - Metadata - Workflow - Run](metadataWorkflowRun.png)
Hit **Execute**!
Conductor will return a workflow id. We will need to use this id to load this up on the UI. If our UI installation has
search enabled we wouldn't need to copy this. If we don't have search enabled (using Elasticsearch) copy it from the
Swagger UI.
![Swagger UI - Metadata - Workflow - Run](workflowRunIdCopy.png)
Ok, we should see this running and get completed soon. Let’s go to the UI to see what happened.
To load the workflow directly, use this URL format:
```
http://localhost:5000/execution/<WORKFLOW_ID>
```
Replace `<WORKFLOW_ID>` with our workflow id from the previous step. We should see a screen like below. Click on the
different tabs to see all inputs and outputs and task list etc. Explore away!
![Conductor UI - Workflow Run](workflowLoaded.png)
## Summary
In this article — we learned how to run a sample workflow in our Conductor installation. Concepts we touched on:
1. Workflow creation
2. System tasks such as HTTP
3. Running a workflow via API
docs/devguide/labs/img/EventHandlerCycle.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 83 KiB

docs/devguide/labs/img/bgnr_complete_workflow.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 27 KiB

docs/devguide/labs/img/bgnr_state_scheduled.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 25 KiB

docs/devguide/labs/img/bgnr_systask_state.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 36 KiB

docs/devguide/labs/index.md
+26
View File
@@ -0,0 +1,26 @@
---
description: "Guided Tutorial — learn Conductor step by step with hands-on labs covering task workers, definitions, and workflows."
---
# Guided Tutorial
## High Level Steps
Generally, these are the steps necessary in order to put Conductor to work for your business workflow:
1. Create task worker(s) that poll for scheduled tasks at regular interval
2. Create task definitions for these workers and register them.
3. Create the workflow definition
## Before We Begin
Ensure you have a Conductor instance up and running. This includes both the Server and the UI. We recommend following the [Docker Instructions](../running/deploy.md).
## Tools
For the purpose of testing and issuing API calls, the following tools are useful
- Linux cURL command
- [Postman](https://www.postman.com) or similar REST client
## Let's Go
We will begin by defining a simple workflow that utilizes System Tasks.
[Next](first-workflow.md)
docs/devguide/labs/kitchensink.md
+280
View File
@@ -0,0 +1,280 @@
---
description: "Run the Conductor kitchen sink example workflow that demonstrates forks, sub-workflows, decisions, dynamic tasks, and HTTP tasks in one definition."
---
# Kitchen Sink
An example kitchensink workflow that demonstrates the usage of all the schema constructs.
### Definition
```json
{
"name": "kitchensink",
"description": "kitchensink workflow",
"version": 1,
"tasks": [
{
"name": "task_1",
"taskReferenceName": "task_1",
"inputParameters": {
"mod": "${workflow.input.mod}",
"oddEven": "${workflow.input.oddEven}"
},
"type": "SIMPLE"
},
{
"name": "event_task",
"taskReferenceName": "event_0",
"inputParameters": {
"mod": "${workflow.input.mod}",
"oddEven": "${workflow.input.oddEven}"
},
"type": "EVENT",
"sink": "conductor"
},
{
"name": "dyntask",
"taskReferenceName": "task_2",
"inputParameters": {
"taskToExecute": "${workflow.input.task2Name}"
},
"type": "DYNAMIC",
"dynamicTaskNameParam": "taskToExecute"
},
{
"name": "oddEvenDecision",
"taskReferenceName": "oddEvenDecision",
"inputParameters": {
"oddEven": "${task_2.output.oddEven}"
},
"type": "DECISION",
"caseValueParam": "oddEven",
"decisionCases": {
"0": [
{
"name": "task_4",
"taskReferenceName": "task_4",
"inputParameters": {
"mod": "${task_2.output.mod}",
"oddEven": "${task_2.output.oddEven}"
},
"type": "SIMPLE"
},
{
"name": "dynamic_fanout",
"taskReferenceName": "fanout1",
"inputParameters": {
"dynamicTasks": "${task_4.output.dynamicTasks}",
"input": "${task_4.output.inputs}"
},
"type": "FORK_JOIN_DYNAMIC",
"dynamicForkTasksParam": "dynamicTasks",
"dynamicForkTasksInputParamName": "input"
},
{
"name": "dynamic_join",
"taskReferenceName": "join1",
"type": "JOIN"
}
],
"1": [
{
"name": "fork_join",
"taskReferenceName": "forkx",
"type": "FORK_JOIN",
"forkTasks": [
[
{
"name": "task_10",
"taskReferenceName": "task_10",
"type": "SIMPLE"
},
{
"name": "sub_workflow_x",
"taskReferenceName": "wf3",
"inputParameters": {
"mod": "${task_1.output.mod}",
"oddEven": "${task_1.output.oddEven}"
},
"type": "SUB_WORKFLOW",
"subWorkflowParam": {
"name": "sub_flow_1",
"version": 1
}
}
],
[
{
"name": "task_11",
"taskReferenceName": "task_11",
"type": "SIMPLE"
},
{
"name": "sub_workflow_x",
"taskReferenceName": "wf4",
"inputParameters": {
"mod": "${task_1.output.mod}",
"oddEven": "${task_1.output.oddEven}"
},
"type": "SUB_WORKFLOW",
"subWorkflowParam": {
"name": "sub_flow_1",
"version": 1
}
}
]
]
},
{
"name": "join",
"taskReferenceName": "join2",
"type": "JOIN",
"joinOn": [
"wf3",
"wf4"
]
}
]
}
},
{
"name": "search_elasticsearch",
"taskReferenceName": "get_es_1",
"inputParameters": {
"http_request": {
"uri": "http://localhost:9200/conductor/_search?size=10",
"method": "GET"
}
},
"type": "HTTP"
},
{
"name": "task_30",
"taskReferenceName": "task_30",
"inputParameters": {
"statuses": "${get_es_1.output..status}",
"workflowIds": "${get_es_1.output..workflowId}"
},
"type": "SIMPLE"
}
],
"outputParameters": {
"statues": "${get_es_1.output..status}",
"workflowIds": "${get_es_1.output..workflowId}"
},
"ownerEmail": "example@email.com",
"schemaVersion": 2
}
```
### Visual Flow
![img](kitchensink.png)
### Running Kitchensink Workflow
1. If you are running Conductor locally, use the `-DloadSample=true` Java system property when launching the server. This will create a kitchensink workflow,
related task definitions and kick off an instance of kitchensink workflow. Otherwise, you can create a new Workflow Definition in the UI by copying the sample above.
2. Once the workflow has started, the first task remains in the `SCHEDULED` state. This is because no workers are currently polling for the task.
3. We will use the REST endpoints directly to poll for tasks and updating the status.
#### Start workflow execution
Start the execution of the kitchensink workflow:
```bash
conductor workflow start -w kitchensink -i '{"task2Name": "task_5"}'
```
The response is a text string identifying the workflow instance id.
??? note "Using cURL"
```shell
curl -X POST --header 'Content-Type: application/json' --header 'Accept: text/plain' '{{ server_host }}{{ api_prefix }}/workflow/kitchensink' -d '
{
"task2Name": "task_5"
}
'
```
#### Poll for the first task:
```bash
conductor task poll task_1
```
??? note "Using cURL"
```shell
curl {{ server_host }}{{ api_prefix }}/tasks/poll/task_1
```
The response should look something like:
```json
{
"taskType": "task_1",
"status": "IN_PROGRESS",
"inputData": {
"mod": null,
"oddEven": null
},
"referenceTaskName": "task_1",
"retryCount": 0,
"seq": 1,
"pollCount": 1,
"taskDefName": "task_1",
"scheduledTime": 1486580932471,
"startTime": 1486580933869,
"endTime": 0,
"updateTime": 1486580933902,
"startDelayInSeconds": 0,
"retried": false,
"callbackFromWorker": true,
"responseTimeoutSeconds": 3600,
"workflowInstanceId": "b0d1a935-3d74-46fd-92b2-0ca1e388659f",
"taskId": "b9eea7dd-3fbd-46b9-a9ff-b00279459476",
"callbackAfterSeconds": 0,
"polledTime": 1486580933902,
"queueWaitTime": 1398
}
```
#### Update the task status
* Note the values for ```taskId``` and ```workflowInstanceId``` fields from the poll response
* Update the status of the task as ```COMPLETED``` as below:
```bash
conductor task update-execution --workflow-id b0d1a935-3d74-46fd-92b2-0ca1e388659f --task-ref-name task_1 --status COMPLETED --output '{"mod":5,"taskToExecute":"task_1","oddEven":0,"dynamicTasks":[{"name":"task_1","taskReferenceName":"task_1_1","type":"SIMPLE"},{"name":"sub_workflow_4","taskReferenceName":"wf_dyn","type":"SUB_WORKFLOW","subWorkflowParam":{"name":"sub_flow_1"}}],"inputs":{"task_1_1":{},"wf_dyn":{}}}'
```
??? note "Using cURL"
```json
curl -H 'Content-Type:application/json' -H 'Accept:application/json' -X POST {{ server_host }}{{ api_prefix }}/tasks/ -d '
{
"taskId": "b9eea7dd-3fbd-46b9-a9ff-b00279459476",
"workflowInstanceId": "b0d1a935-3d74-46fd-92b2-0ca1e388659f",
"status": "COMPLETED",
"outputData": {
"mod": 5,
"taskToExecute": "task_1",
"oddEven": 0,
"dynamicTasks": [
{
"name": "task_1",
"taskReferenceName": "task_1_1",
"type": "SIMPLE"
},
{
"name": "sub_workflow_4",
"taskReferenceName": "wf_dyn",
"type": "SUB_WORKFLOW",
"subWorkflowParam": {
"name": "sub_flow_1"
}
}
],
"inputs": {
"task_1_1": {},
"wf_dyn": {}
}
}
}'
```
This will mark the task_1 as completed and schedule ```task_5``` as the next task.
Repeat the same process for the subsequently scheduled tasks until the completion.
docs/devguide/labs/kitchensink.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 79 KiB

docs/devguide/labs/metadataWorkflowPost.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 98 KiB

docs/devguide/labs/metadataWorkflowRun.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 157 KiB

docs/devguide/labs/uiWorkflowDefinition.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 245 KiB

docs/devguide/labs/uiWorkflowDefinitionVisual.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 337 KiB

docs/devguide/labs/workflowLoaded.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 325 KiB

docs/devguide/labs/workflowRunIdCopy.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 147 KiB

docs/devguide/running/conductorUI.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 173 KiB

docs/devguide/running/deploy.md
+639
View File
@@ -0,0 +1,639 @@
---
description: "Deploy Conductor as a self-hosted workflow engine in production — architecture overview, horizontal scaling, database, queue, indexing, and lock configuration, workflow monitoring, and recommended production deployment settings for this open source workflow orchestration platform."
---
# Self-hosted deployment guide
Conductor is a self-hosted, open source workflow engine that you deploy on your own infrastructure. This production deployment guide covers everything you need to run Conductor at scale: architecture, backend configuration, horizontal scaling, workflow monitoring, and tuning.
## Architecture overview
A Conductor deployment consists of these components:
![Conductor Architecture](../architecture/conductor-architecture.png)
**What each component does:**
| Component | Role |
|:--|:--|
| **API Server** | Exposes REST and gRPC endpoints for workflow and task operations. |
| **Decider** | The core state machine. Evaluates workflow state and schedules the next set of tasks. |
| **Sweeper** | Background process that polls for running workflows and triggers the decider to evaluate them. Required for progress on long-running workflows. |
| **System Task Workers** | Execute built-in task types (HTTP, Event, Wait, Inline, JSON_JQ, etc.) within the server JVM. |
| **Event Processor** | Listens to configured event buses and triggers workflows or completes tasks based on incoming events. |
| **Database** | Persists workflow definitions, execution state, task state, and poll data. |
| **Queue** | Manages task scheduling — pending tasks, delayed tasks, and the sweeper's own work queue. |
| **Index** | Powers workflow and task search in the UI and via the search API. |
| **Lock** | Distributed lock that prevents concurrent decider evaluations of the same workflow. **Required in production.** |
---
## Quick start with Docker Compose
For local development and evaluation:
```shell
git clone https://github.com/conductor-oss/conductor
cd conductor
docker compose -f docker/docker-compose.yaml up
```
This starts Conductor with Redis (database + queue), Elasticsearch (indexing), and the server with UI on port **8080**.
| URL | Description |
|:----|:---|
| `http://localhost:8080` | Conductor UI |
| `http://localhost:8080/swagger-ui/index.html` | REST API docs |
| `http://localhost:8080/api/` | API base URL |
Pre-built compose files for other backend combinations:
| Compose file | Database | Queue | Index |
|:--|:--|:--|:--|
| `docker-compose.yaml` | Redis | Redis | Elasticsearch 7 |
| `docker-compose-es8.yaml` | Redis | Redis | Elasticsearch 8 |
| `docker-compose-postgres.yaml` | PostgreSQL | PostgreSQL | PostgreSQL |
| `docker-compose-postgres-es7.yaml` | PostgreSQL | PostgreSQL | Elasticsearch 7 |
| `docker-compose-mysql.yaml` | MySQL | Redis | Elasticsearch 7 |
| `docker-compose-redis-os2.yaml` | Redis | Redis | OpenSearch 2 |
| `docker-compose-redis-os3.yaml` | Redis | Redis | OpenSearch 3 |
```shell
# Example: PostgreSQL for everything
docker compose -f docker/docker-compose-postgres.yaml up
# Example: Redis + Elasticsearch 8
docker compose -f docker/docker-compose-es8.yaml up
# Example: Redis + OpenSearch 3
docker compose -f docker/docker-compose-redis-os3.yaml up
```
For Elasticsearch 8, set `conductor.indexing.type=elasticsearch8` and use
`config-redis-es8.properties` or an equivalent custom config.
---
## Production configuration
All configuration is done via Spring Boot properties in `application.properties` or environment variables. Properties can also be mounted as a Docker volume.
### Database
The database stores workflow definitions, execution state, task state, and event handler definitions.
```properties
conductor.db.type=postgres
```
**Supported database backends:**
| Backend | Property value | When to use | Notes |
|:--|:--|:--|:--|
| PostgreSQL | `postgres` | **Recommended for production.** ACID, battle-tested, supports indexing too. | Requires `spring.datasource.*` config. |
| MySQL | `mysql` | Production alternative if your team already runs MySQL. | Requires `spring.datasource.*` config. Needs separate queue backend (Redis). |
| Redis | `redis_standalone` | Fast, simple. Good for moderate scale. | Requires `conductor.redis.*` config. |
| Cassandra | `cassandra` | High write throughput, multi-region. | Requires `conductor.cassandra.*` config. |
| SQLite | `sqlite` | **Local development only.** Single-file, zero config. | Default. Not for production. |
#### PostgreSQL
```properties
conductor.db.type=postgres
conductor.external-payload-storage.type=postgres
spring.datasource.url=jdbc:postgresql://db-host:5432/conductor
spring.datasource.username=conductor
spring.datasource.password=<password>
# Optional tuning
conductor.postgres.deadlockRetryMax=3
conductor.postgres.taskDefCacheRefreshInterval=60s
conductor.postgres.asyncMaxPoolSize=12
conductor.postgres.asyncWorkerQueueSize=100
```
#### MySQL
```properties
conductor.db.type=mysql
spring.datasource.url=jdbc:mysql://db-host:3306/conductor
spring.datasource.username=conductor
spring.datasource.password=<password>
# Optional tuning
conductor.mysql.deadlockRetryMax=3
conductor.mysql.taskDefCacheRefreshInterval=60s
```
#### Redis
```properties
conductor.db.type=redis_standalone
# Format: host:port:rack (semicolon-separated for multiple hosts)
conductor.redis.hosts=redis-host:6379:us-east-1c
conductor.redis.workflowNamespacePrefix=conductor
conductor.redis.queueNamespacePrefix=conductor_queues
conductor.redis.taskDefCacheRefreshInterval=1s
# Connection pool
conductor.redis.maxIdleConnections=8
conductor.redis.minIdleConnections=5
# SSL
conductor.redis.ssl=false
# Auth (password is taken from the first host entry: host:port:rack:password)
# Or set conductor.redis.username and conductor.redis.password directly
```
---
### Queue
The queue backend manages task scheduling — it tracks which tasks are pending, delayed, or ready for execution. The sweeper and system task workers all depend on it.
```properties
conductor.queue.type=postgres
```
**Supported queue backends:**
| Backend | Property value | When to use |
|:--|:--|:--|
| PostgreSQL | `postgres` | Use when database is also PostgreSQL. Simplest stack. |
| Redis | `redis_standalone` | Use when database is Redis or MySQL. Fast, low-latency. |
| SQLite | `sqlite` | Local development only. |
!!! tip "Match your queue backend to your database"
PostgreSQL database + PostgreSQL queue is the simplest production stack — one fewer dependency. If you use MySQL for the database, pair it with Redis for the queue.
---
### Indexing
The indexing backend powers workflow and task search in the UI and via the `/api/workflow/search` and `/api/tasks/search` endpoints.
```properties
conductor.indexing.enabled=true
conductor.indexing.type=postgres
```
**Supported indexing backends:**
| Backend | Property value | When to use | Notes |
|:--|:--|:--|:--|
| PostgreSQL | `postgres` | Simplest stack when database is also PostgreSQL. | Set `conductor.elasticsearch.version=0` to disable ES client. |
| Elasticsearch 7 | `elasticsearch` | Best search performance at scale. Full-text search. | Set `conductor.elasticsearch.version=7`. |
| Elasticsearch 8 | `elasticsearch8` | Use when running the ES8 persistence module. | Set `conductor.elasticsearch.version=8`. |
| OpenSearch 2 | `opensearch2` | Open-source ES alternative. | Compatible with ES 7 queries. |
| OpenSearch 3 | `opensearch3` | Latest OpenSearch. | |
| SQLite | `sqlite` | Local development only. | |
| Disabled | N/A | Set `conductor.indexing.enabled=false`. UI search won't work. | |
#### PostgreSQL indexing
```properties
conductor.indexing.enabled=true
conductor.indexing.type=postgres
# Disable Elasticsearch client
conductor.elasticsearch.version=0
```
#### Elasticsearch 7
```properties
conductor.indexing.enabled=true
conductor.elasticsearch.url=http://es-host:9200
conductor.elasticsearch.version=7
conductor.elasticsearch.indexName=conductor
conductor.elasticsearch.clusterHealthColor=yellow
# Performance tuning
conductor.elasticsearch.indexBatchSize=1
conductor.elasticsearch.asyncMaxPoolSize=12
conductor.elasticsearch.asyncWorkerQueueSize=100
conductor.elasticsearch.asyncBufferFlushTimeout=10s
conductor.elasticsearch.indexShardCount=5
conductor.elasticsearch.indexReplicasCount=1
# Auth (if using security)
conductor.elasticsearch.username=elastic
conductor.elasticsearch.password=<password>
```
#### Elasticsearch 8
```properties
conductor.indexing.enabled=true
conductor.indexing.type=elasticsearch8
conductor.elasticsearch.url=http://es-host:9200
conductor.elasticsearch.version=8
conductor.elasticsearch.indexName=conductor
conductor.elasticsearch.clusterHealthColor=yellow
```
#### OpenSearch
```properties
conductor.indexing.enabled=true
conductor.indexing.type=opensearch2 # or opensearch3
conductor.opensearch.url=http://os-host:9200
conductor.opensearch.indexPrefix=conductor
conductor.opensearch.clusterHealthColor=yellow
conductor.opensearch.indexReplicasCount=0
```
#### Async indexing
For high-throughput deployments, enable async indexing to decouple the indexing path from the workflow execution path:
```properties
conductor.app.asyncIndexingEnabled=true
conductor.app.asyncUpdateShortRunningWorkflowDuration=30s
conductor.app.asyncUpdateDelay=60s
```
#### Indexing toggles
Control what gets indexed:
```properties
conductor.app.taskIndexingEnabled=true
conductor.app.taskExecLogIndexingEnabled=true
conductor.app.eventMessageIndexingEnabled=true
conductor.app.eventExecutionIndexingEnabled=true
```
---
### Locking
!!! warning "Required for production"
Distributed locking prevents race conditions when multiple server instances evaluate the same workflow concurrently. **Always enable locking in production with a distributed lock provider** (Redis or Zookeeper).
```properties
conductor.workflow-execution-lock.type=redis
conductor.app.workflowExecutionLockEnabled=true
```
**Supported lock providers:**
| Provider | Property value | When to use |
|:--|:--|:--|
| Redis | `redis` | **Recommended.** Use when Redis is already in the stack. |
| Zookeeper | `zookeeper` | Use when Zookeeper is available (e.g. Kafka deployments). |
| Local | `local_only` | Single-instance development only. **Not safe for multi-instance.** |
#### Redis lock
```properties
conductor.workflow-execution-lock.type=redis
conductor.app.workflowExecutionLockEnabled=true
conductor.app.lockLeaseTime=60000 # lock held for max 60s
conductor.app.lockTimeToTry=500 # wait up to 500ms to acquire
conductor.redis-lock.serverType=SINGLE # SINGLE, CLUSTER, or SENTINEL
conductor.redis-lock.serverAddress=redis://redis-host:6379
# conductor.redis-lock.serverPassword=<password>
# conductor.redis-lock.serverMasterName=master # for Sentinel
# conductor.redis-lock.namespace=conductor # key prefix
conductor.redis-lock.ignoreLockingExceptions=false
```
> **Sentinel with multiple endpoints:** When using `SENTINEL` server type, you can provide
> multiple sentinel addresses separated by semicolons for improved high availability:
> ```properties
> conductor.redis-lock.serverType=SENTINEL
> conductor.redis-lock.serverAddress=redis://sentinel-0:26379;redis://sentinel-1:26379
> conductor.redis-lock.serverMasterName=mymaster
> ```
> This ensures the lock client can discover the master even if one sentinel node is down.
#### Zookeeper lock
```properties
conductor.workflow-execution-lock.type=zookeeper
conductor.app.workflowExecutionLockEnabled=true
conductor.app.lockLeaseTime=60000
conductor.app.lockTimeToTry=500
conductor.zookeeper-lock.connectionString=zk1:2181,zk2:2181,zk3:2181
# conductor.zookeeper-lock.sessionTimeoutMs=60000
# conductor.zookeeper-lock.connectionTimeoutMs=15000
# conductor.zookeeper-lock.namespace=conductor
```
---
### Sweeper
The sweeper is a background process that monitors running workflows. It polls the queue for workflows that need evaluation and triggers the decider. Without the sweeper, long-running workflows will not make progress.
The sweeper runs automatically as part of the Conductor server. Tune the thread count based on your workflow volume:
```properties
# Number of sweeper threads (default: availableProcessors * 2)
conductor.app.sweeperThreadCount=8
# How long to wait when polling the sweep queue (default: 2000ms)
conductor.app.sweeperWorkflowPollTimeout=2000
# Batch size per sweep poll (default: 2)
conductor.app.sweeper.sweepBatchSize=2
# Queue pop timeout in ms (default: 100)
conductor.app.sweeper.queuePopTimeout=100
```
!!! tip "Sweeper sizing"
Start with `sweeperThreadCount = 2 * CPU cores`. If you see workflows stuck in RUNNING state, increase it. If CPU usage is high on idle, decrease it.
---
### System task workers
System task workers execute built-in task types (HTTP, Event, Wait, Inline, JSON_JQ_TRANSFORM, etc.) inside the Conductor server JVM. They poll internal queues for scheduled system tasks and execute them.
```properties
# Number of system task worker threads (default: availableProcessors * 2)
conductor.app.systemTaskWorkerThreadCount=20
# Max number of tasks to poll at once (default: same as thread count)
conductor.app.systemTaskMaxPollCount=20
# Poll interval (default: 50ms)
conductor.app.systemTaskWorkerPollInterval=50ms
# Callback duration — how often to re-check async system tasks (default: 30s)
conductor.app.systemTaskWorkerCallbackDuration=30s
# Queue pop timeout (default: 100ms)
conductor.app.systemTaskQueuePopTimeout=100ms
```
#### Running system task workers separately
In large deployments, you may want to run system task workers on dedicated instances, separate from the API server. Use the **execution namespace** to isolate which instance handles system tasks:
```properties
# On API-only instances — set a namespace that no system task worker listens on
conductor.app.systemTaskWorkerExecutionNamespace=api-only
conductor.app.systemTaskWorkerThreadCount=0
# On dedicated system task worker instances — match the namespace
conductor.app.systemTaskWorkerExecutionNamespace=worker-pool-1
conductor.app.systemTaskWorkerThreadCount=40
conductor.app.systemTaskMaxPollCount=40
```
#### Isolated system task workers
For task domain isolation (routing specific tasks to specific worker groups):
```properties
# Threads per isolation group (default: 1)
conductor.app.isolatedSystemTaskWorkerThreadCount=4
```
#### Postpone threshold
When a system task has been polled many times without completing (e.g. a Join waiting for branches), Conductor progressively delays re-evaluation to avoid busy-polling:
```properties
# After this many polls, begin exponential backoff (default: 200)
conductor.app.systemTaskPostponeThreshold=200
```
---
### Event processing
The event processor listens to configured event buses and triggers workflows or completes tasks based on incoming events.
```properties
# Thread count for event processing (default: 2)
conductor.app.eventProcessorThreadCount=4
# Event queue polling
conductor.app.eventQueueSchedulerPollThreadCount=4 # default: CPU cores
conductor.app.eventQueuePollInterval=100ms
conductor.app.eventQueuePollCount=10
conductor.app.eventQueueLongPollTimeout=1000ms
```
See the [Event-driven recipes](../cookbook/event-driven.md) for configuring Kafka, NATS, AMQP, and SQS event queues.
---
### Payload size limits
Conductor enforces payload size limits to prevent oversized data from degrading performance. When a payload exceeds the threshold, it is automatically stored in external payload storage (S3, PostgreSQL, or Azure Blob).
```properties
# Workflow input/output — threshold to move to external storage (default: 5120 KB)
conductor.app.workflowInputPayloadSizeThreshold=5120KB
conductor.app.workflowOutputPayloadSizeThreshold=5120KB
# Workflow input/output — hard limit, fails the workflow (default: 10240 KB)
conductor.app.maxWorkflowInputPayloadSizeThreshold=10240KB
conductor.app.maxWorkflowOutputPayloadSizeThreshold=10240KB
# Task input/output — threshold to move to external storage (default: 3072 KB)
conductor.app.taskInputPayloadSizeThreshold=3072KB
conductor.app.taskOutputPayloadSizeThreshold=3072KB
# Task input/output — hard limit, fails the task (default: 10240 KB)
conductor.app.maxTaskInputPayloadSizeThreshold=10240KB
conductor.app.maxTaskOutputPayloadSizeThreshold=10240KB
# Workflow variables — hard limit (default: 256 KB)
conductor.app.maxWorkflowVariablesPayloadSizeThreshold=256KB
```
For external payload storage configuration, see [External Payload Storage](../../documentation/advanced/externalpayloadstorage.md).
---
### Workflow monitoring and observability
Conductor exposes Prometheus-compatible metrics out of the box for workflow monitoring and observability:
```properties
conductor.metrics-prometheus.enabled=true
management.endpoints.web.exposure.include=health,info,prometheus
management.metrics.web.server.request.autotime.percentiles=0.50,0.75,0.90,0.95,0.99
management.endpoint.health.show-details=always
```
Scrape `http://<conductor-host>:8080/actuator/prometheus` with Prometheus.
For details on available metrics, see [Server Metrics](../../documentation/metrics/server.md) and [Client Metrics](../../documentation/metrics/client.md).
---
## Recommended production configurations
### PostgreSQL stack (simplest)
One database for everything — fewest moving parts.
```properties
# Database
conductor.db.type=postgres
conductor.queue.type=postgres
conductor.external-payload-storage.type=postgres
spring.datasource.url=jdbc:postgresql://db-host:5432/conductor
spring.datasource.username=conductor
spring.datasource.password=<password>
# Indexing (use PostgreSQL, no Elasticsearch needed)
conductor.indexing.enabled=true
conductor.indexing.type=postgres
conductor.elasticsearch.version=0
# Locking (use Redis — lightweight, fast)
conductor.workflow-execution-lock.type=redis
conductor.app.workflowExecutionLockEnabled=true
conductor.redis-lock.serverAddress=redis://redis-host:6379
# Sweeper
conductor.app.sweeperThreadCount=8
# System task workers
conductor.app.systemTaskWorkerThreadCount=20
conductor.app.systemTaskMaxPollCount=20
# Metrics
conductor.metrics-prometheus.enabled=true
management.endpoints.web.exposure.include=health,info,prometheus
```
### Redis + Elasticsearch stack (high throughput)
Best search performance and lowest latency for queue operations.
```properties
# Database + Queue
conductor.db.type=redis_standalone
conductor.queue.type=redis_standalone
conductor.redis.hosts=redis-host:6379:us-east-1c
conductor.redis.workflowNamespacePrefix=conductor
conductor.redis.queueNamespacePrefix=conductor_queues
# Indexing
conductor.indexing.enabled=true
conductor.elasticsearch.url=http://es-host:9200
conductor.elasticsearch.version=7
conductor.elasticsearch.indexName=conductor
conductor.elasticsearch.clusterHealthColor=yellow
conductor.app.asyncIndexingEnabled=true
# Locking
conductor.workflow-execution-lock.type=redis
conductor.app.workflowExecutionLockEnabled=true
conductor.redis-lock.serverAddress=redis://redis-host:6379
# Sweeper
conductor.app.sweeperThreadCount=16
# System task workers
conductor.app.systemTaskWorkerThreadCount=40
conductor.app.systemTaskMaxPollCount=40
# Metrics
conductor.metrics-prometheus.enabled=true
management.endpoints.web.exposure.include=health,info,prometheus
```
---
## Running with Docker
### Using Docker Compose
```shell
git clone https://github.com/conductor-oss/conductor
cd conductor
docker compose -f docker/docker-compose.yaml up
```
To use a different backend, swap the compose file:
```shell
docker compose -f docker/docker-compose-postgres.yaml up
```
### Using the standalone image
```shell
docker run -p 8080:8080 conductoross/conductor:latest
```
### Custom configuration via volume mount
Mount your own properties file to override the defaults without rebuilding the image:
```shell
docker run -p 8080:8080 \
-v /path/to/my-config.properties:/app/config/config.properties \
conductoross/conductor:latest
```
### Accessing Conductor
| URL | Description |
|:----|:---|
| `http://localhost:8080` | Conductor UI |
| `http://localhost:8080/swagger-ui/index.html` | REST API docs |
### Shutting down
```shell
# Ctrl+C to stop, then:
docker compose down
```
---
## Multi-instance deployment and horizontal scaling
For high availability and horizontal scaling, run multiple Conductor server instances behind a load balancer. All instances share the same database, queue, index, and lock backends. This architecture enables workflow engine scalability to millions of concurrent executions.
**Requirements:**
- **Distributed locking must be enabled** (`redis` or `zookeeper`). Without it, concurrent decider evaluations on the same workflow will cause race conditions.
- All instances must point to the same database, queue, and indexing backends.
- The load balancer should use round-robin or least-connections routing.
**Optional: separate API and worker instances:**
```
┌──────────────────┐ ┌──────────────────┐
│ API Instance 1 │ │ API Instance 2 │ ← handle REST/gRPC, low system task threads
│ (systemTask=0) │ │ (systemTask=0) │
└────────┬─────────┘ └────────┬─────────┘
│ │
┌────┴────────────────────────┴────┐
│ Load Balancer │
└────┬────────────────────────┬────┘
│ │
┌────────┴──────────┐ ┌───────┴───────────┐
│ Worker Instance │ │ Worker Instance │ ← high system task threads, sweeper
│ (systemTask=40) │ │ (systemTask=40) │
└───────────────────┘ └───────────────────┘
```
---
## Troubleshooting
| Issue | Fix |
|:--|:--|
| Out of memory or slow performance | Check JVM heap usage and adjust `-Xms` / `-Xmx` as necessary. Monitor with `jstat` or the `/actuator/health` endpoint. |
| Elasticsearch stuck in yellow health | Set `conductor.elasticsearch.clusterHealthColor=yellow` or add more ES nodes for green. |
| Workflows stuck in RUNNING | Check sweeper is running and `sweeperThreadCount > 0`. Check lock provider is reachable. |
| System tasks not executing | Verify `systemTaskWorkerThreadCount > 0` and the queue backend is reachable. |
| Config changes not taking effect | Properties are baked into the Docker image at build time. Mount a volume instead of rebuilding. |
docs/devguide/running/hosted.md
+20
View File
@@ -0,0 +1,20 @@
---
description: "Hosted Solutions — run Conductor in the cloud with Orkes, offering enterprise-grade hosting and managed infrastructure."
---
# Hosted Solutions
## Orkes
[Orkes](https://orkes.io) offers a cloud-hosted, enterprise-grade version of Conductor, enabling teams to get started with minimal operational overhead. Besides full compatibility with Conductor OSS, Orkes Conductor provides [additional features](https://www.orkes.io/platform/conductor-oss-vs-orkes) not available in the open source release.
Here are the options for using Conductor via Orkes:
- Developer Edition
- Cloud Hosting Plans
Orkes also operates a [Discourse](https://community.orkes.io/) forum for the community to discuss and share how to use Conductor.
### Developer Edition
The free Orkes Developer Edition for Conductor is available at [developer.orkescloud.com](https://developer.orkescloud.com/). The Developer Edition comes with all of Orkes' enterprise features, including a visual workflow editor, AI orchestration suite, event-driven connectors, human-in-the-loop tasks, and more. You can create and execute workflows from the UI or API.
### Cloud Hosted Conductor
Orkes provides multiple options of hosted Conductor clusters in the cloud (AWS, Azure, and GCP, in addition to private clouds) with enterprise support provided by the Orkes team. Learn more about [Orkes Cloud here](https://orkes.io/cloud).
docs/devguide/running/source.md
+83
View File
@@ -0,0 +1,83 @@
---
description: "Building from Source — build and run the Conductor server and UI locally from source for development and testing."
---
# Building from source
Build and run Conductor server and UI locally from source. The default configuration uses in-memory persistence with no indexing — all data is lost when the server stops. This setup is for development and testing only.
For persistent backends, use [Docker Compose](deploy.md) or configure a database backend.
## Prerequisites
- Java (JDK) 21+
- (Optional) [Docker](https://www.docker.com/get-started/) for running tests
## Building and running the server
1. Clone the repository:
```shell
git clone https://github.com/conductor-oss/conductor.git
cd conductor
```
2. Run with Gradle:
```shell
cd server
../gradlew bootRun
```
To use a custom configuration file:
```shell
CONFIG_PROP=config.properties ../gradlew bootRun
```
3. The server is now running:
| URL | Description |
|:----|:---|
| `http://localhost:8080` | Conductor UI |
| `http://localhost:8080/swagger-ui/index.html` | REST API docs |
| `http://localhost:8080/api/` | API base URL |
## Running from a pre-compiled JAR
As an alternative to building from source, download and run the pre-compiled JAR:
```shell
export CONDUCTOR_VER=3.21.10
export REPO_URL=https://repo1.maven.org/maven2/org/conductoross/conductor-server
curl $REPO_URL/$CONDUCTOR_VER/conductor-core-$CONDUCTOR_VER-boot.jar \
--output conductor-core-$CONDUCTOR_VER-boot.jar
java -jar conductor-core-$CONDUCTOR_VER-boot.jar
```
## Running the UI from source
### Prerequisites
- A running Conductor server on port 8080
- [Node.js](https://nodejs.org) v18+
- [Yarn](https://classic.yarnpkg.com/en/docs/install)
### Steps
```shell
cd ui
yarn install
yarn run start
```
The UI is accessible at [http://localhost:5000](http://localhost:5000).
To build compiled assets for production hosting:
```shell
yarn build
```
docs/devguide/running/swagger.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 299 KiB