a9cd7750f4
CI / unit-test (push) Has been cancelled
CI / detect-changes (push) Has been cancelled
CI / build (push) Has been cancelled
Publish docs via GitHub Pages / Deploy docs (push) Has been cancelled
CI / test-harness (push) Has been cancelled
CI / generate-e2e-matrix (push) Has been cancelled
CI / e2e (push) Has been cancelled
CI / build-ui (push) Has been cancelled
Release Drafter / update_release_draft (push) Has been cancelled
UI v2 Integration CI / E2E (Integration) (push) Has been cancelled
UI v2 CI / Lint, Format & Test (push) Has been cancelled
UI v2 CI / E2E (Mocked) (push) Has been cancelled
235 lines
7.1 KiB
Markdown
235 lines
7.1 KiB
Markdown
---
|
|
description: "Orchestrate event-driven workflows with Conductor using Kafka, NATS, AMQP (RabbitMQ), and SQS as event buses. Configure event handlers to trigger workflows, complete tasks, or fail tasks on incoming events."
|
|
---
|
|
|
|
# Event Bus Orchestration
|
|
|
|
Conductor integrates with external messaging systems to enable event-driven workflow orchestration. You can publish events from workflows and react to external events — starting workflows, completing tasks, or failing tasks based on incoming messages.
|
|
|
|
## Supported event buses
|
|
|
|
| System | Sink prefix | Module | Use case |
|
|
| :--- | :--- | :--- | :--- |
|
|
| **Kafka** | `kafka` | `kafka` | High-throughput, durable event streaming |
|
|
| **NATS** | `nats` | `nats` | Lightweight, low-latency messaging |
|
|
| **NATS Streaming** | `nats-stream` | `nats-streaming` | Durable NATS with replay (legacy) |
|
|
| **NATS JetStream** | `nats` | `nats` | Modern durable NATS streaming |
|
|
| **AMQP (RabbitMQ)** | `amqp`, `amqp_queue`, `amqp_exchange` | `amqp` | Traditional message queuing with routing |
|
|
| **SQS** | `sqs` | `sqs` | AWS-native message queuing |
|
|
| **Conductor** | `conductor` | built-in | Internal event routing between workflows |
|
|
|
|
|
|
## How it works
|
|
|
|
Event bus orchestration has two sides:
|
|
|
|
1. **Publishing** — Use the [Event task](../../documentation/configuration/workflowdef/systemtasks/event-task.md) or [Kafka Publish task](../../documentation/configuration/workflowdef/systemtasks/kafka-publish-task.md) to send messages from a workflow.
|
|
2. **Consuming** — Register [event handlers](../../documentation/configuration/eventhandlers.md) that listen for messages and trigger actions.
|
|
|
|
```
|
|
┌──────────────┐ Event Task ┌──────────────┐ Event Handler ┌──────────────┐
|
|
│ Workflow A │ ──────────────────► │ Event Bus │ ──────────────────► │ Workflow B │
|
|
│ │ (publish) │ (Kafka/NATS/ │ (start_workflow) │ (triggered) │
|
|
│ │ │ AMQP/SQS) │ │ │
|
|
└──────────────┘ └──────────────┘ └──────────────┘
|
|
```
|
|
|
|
|
|
## Publishing events
|
|
|
|
### Event task
|
|
|
|
The [Event task](../../documentation/configuration/workflowdef/systemtasks/event-task.md) publishes a message to any supported event bus. The `sink` parameter determines the target:
|
|
|
|
```json
|
|
{
|
|
"name": "notify_downstream",
|
|
"taskReferenceName": "notify_ref",
|
|
"type": "EVENT",
|
|
"sink": "kafka:order-events",
|
|
"inputParameters": {
|
|
"orderId": "${workflow.input.orderId}",
|
|
"status": "PROCESSED"
|
|
}
|
|
}
|
|
```
|
|
|
|
### Kafka Publish task
|
|
|
|
For Kafka-specific features (custom headers, key, serializers), use the dedicated [Kafka Publish task](../../documentation/configuration/workflowdef/systemtasks/kafka-publish-task.md):
|
|
|
|
```json
|
|
{
|
|
"name": "publish_to_kafka",
|
|
"taskReferenceName": "kafka_ref",
|
|
"type": "KAFKA_PUBLISH",
|
|
"inputParameters": {
|
|
"kafka_request": {
|
|
"topic": "order-events",
|
|
"value": "${workflow.input.orderData}",
|
|
"bootStrapServers": "kafka:9092",
|
|
"headers": {
|
|
"X-Correlation-Id": "${workflow.correlationId}"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### Sink format
|
|
|
|
The `sink` parameter follows the format `prefix:queue_name`:
|
|
|
|
| Example | System |
|
|
| :--- | :--- |
|
|
| `kafka:order-events` | Kafka topic `order-events` |
|
|
| `nats:notifications` | NATS subject `notifications` |
|
|
| `amqp:task-queue` | AMQP queue `task-queue` |
|
|
| `amqp_exchange:events` | AMQP exchange `events` |
|
|
| `sqs:my-queue` | SQS queue `my-queue` |
|
|
| `conductor` | Conductor internal queue |
|
|
| `conductor:workflow_name:queue_name` | Conductor internal, specific queue |
|
|
|
|
|
|
## Consuming events
|
|
|
|
### Event handlers
|
|
|
|
Event handlers listen for messages on an event bus and execute actions when a matching event arrives. Register them via the `/api/event` API.
|
|
|
|
```json
|
|
{
|
|
"name": "order_event_handler",
|
|
"event": "kafka:order-events",
|
|
"condition": "$.status == 'PROCESSED'",
|
|
"actions": [
|
|
{
|
|
"action": "start_workflow",
|
|
"start_workflow": {
|
|
"name": "fulfillment_workflow",
|
|
"input": {
|
|
"orderId": "${orderId}"
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
### Supported actions
|
|
|
|
| Action | Description |
|
|
| :--- | :--- |
|
|
| `start_workflow` | Start a new workflow execution with the event payload as input. |
|
|
| `complete_task` | Complete a waiting task (e.g., a `WAIT` or `HUMAN` task) in a running workflow. |
|
|
| `fail_task` | Fail a task in a running workflow. |
|
|
|
|
### Conditions
|
|
|
|
The `condition` field supports JavaScript-like expressions evaluated against the event payload:
|
|
|
|
| Expression | Result |
|
|
| :--- | :--- |
|
|
| `$.version > 1` | true if `version` field > 1 |
|
|
| `$.metadata.codec == 'aac'` | true if nested field matches |
|
|
| `$.status == 'COMPLETED'` | true if status is COMPLETED |
|
|
|
|
Actions execute only when the condition evaluates to `true`. If no condition is specified, actions execute for every event.
|
|
|
|
|
|
## Patterns
|
|
|
|
### Event-driven workflow chaining
|
|
|
|
Decouple workflows using events instead of sub-workflows:
|
|
|
|
```json
|
|
{
|
|
"name": "order_pipeline",
|
|
"tasks": [
|
|
{
|
|
"name": "process_order",
|
|
"taskReferenceName": "process_ref",
|
|
"type": "SIMPLE"
|
|
},
|
|
{
|
|
"name": "notify_fulfillment",
|
|
"taskReferenceName": "notify_ref",
|
|
"type": "EVENT",
|
|
"sink": "kafka:fulfillment-requests",
|
|
"inputParameters": {
|
|
"orderId": "${workflow.input.orderId}",
|
|
"items": "${process_ref.output.items}"
|
|
}
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
A separate event handler starts the fulfillment workflow when the event arrives.
|
|
|
|
### Wait for external event
|
|
|
|
Combine a `WAIT` task with an event handler to pause a workflow until an external system signals completion:
|
|
|
|
```json
|
|
{
|
|
"name": "wait_for_approval",
|
|
"taskReferenceName": "approval_ref",
|
|
"type": "WAIT"
|
|
}
|
|
```
|
|
|
|
Register an event handler that completes the task when an approval event arrives:
|
|
|
|
```json
|
|
{
|
|
"name": "approval_handler",
|
|
"event": "kafka:approval-events",
|
|
"condition": "$.approved == true",
|
|
"actions": [
|
|
{
|
|
"action": "complete_task",
|
|
"complete_task": {
|
|
"workflowId": "${workflowId}",
|
|
"taskRefName": "approval_ref",
|
|
"output": {
|
|
"approvedBy": "${approvedBy}"
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
|
|
## Configuration
|
|
|
|
Each event bus module requires its own configuration. Enable the modules you need in your Conductor server configuration:
|
|
|
|
### Kafka
|
|
|
|
```properties
|
|
conductor.event-queues.kafka.enabled=true
|
|
conductor.event-queues.kafka.bootstrap-servers=kafka:9092
|
|
```
|
|
|
|
### NATS
|
|
|
|
```properties
|
|
conductor.event-queues.nats.enabled=true
|
|
conductor.event-queues.nats.url=nats://localhost:4222
|
|
```
|
|
|
|
### AMQP (RabbitMQ)
|
|
|
|
```properties
|
|
conductor.event-queues.amqp.enabled=true
|
|
conductor.event-queues.amqp.hosts=rabbitmq
|
|
conductor.event-queues.amqp.port=5672
|
|
conductor.event-queues.amqp.username=guest
|
|
conductor.event-queues.amqp.password=guest
|
|
```
|
|
|
|
Refer to the module source code for the full set of configuration properties.
|