conductor-oss--conductor/es8-persistence

ES8 Persistence

This module provides Elasticsearch 8.x persistence for indexing workflows and tasks.

It uses the Elasticsearch Java API Client (elasticsearch-java) aligned with ES8.

Index management (ES8 best practices)

This module uses composable index templates, write aliases, and an ILM policy:

• ILM policy: ${indexPrefix}-default-ilm-policy
• Rollover conditions (hot phase): max_primary_shard_size=50gb
• Write aliases: ${indexPrefix}_workflow, ${indexPrefix}_task, ${indexPrefix}_task_log, ${indexPrefix}_message, ${indexPrefix}_event
• Initial indices: ${alias}-000001 (created automatically when index management is enabled)
• Refresh interval: defaults to 30s for all indices (configurable)

Build and usage

1) Build-time module selection

When building the server, select the ES8 persistence module to avoid Lucene conflicts:

./gradlew :conductor-server:bootJar -PindexingBackend=elasticsearch8

-PindexingBackend=es8 is also accepted.

2) Runtime configuration

Select the Elasticsearch 8 backend in your configuration:

conductor.indexing.type=elasticsearch8

Elasticsearch version compatibility

The ES8 module uses elasticsearch-java client version 8.19.11. For local Docker-based setups, use Elasticsearch 8.19.x (the provided compose file pins 8.19.11).

All other conductor.elasticsearch.* properties are shared with the ES7 module.

Configuration

(Default values shown below)

# A comma separated list of scheme/host/port of the ES nodes to communicate with.
# Scheme can be `http` or `https`. If scheme is omitted then `http` will be used.
conductor.elasticsearch.url=localhost:9200

# The name of the workflow and task index.
conductor.elasticsearch.indexPrefix=conductor

# Default refresh interval applied via the component template.
conductor.elasticsearch.indexRefreshInterval=30s

# Path to a PEM-encoded certificate to trust for HTTPS connections.
conductor.elasticsearch.trustCertPath=

# Worker queue size used in executor service for async methods in IndexDao.
conductor.elasticsearch.asyncWorkerQueueSize=100

# Maximum thread pool size in executor service for async methods in IndexDao
conductor.elasticsearch.asyncMaxPoolSize=12

# Timeout (in seconds) for the in-memory to be flushed if not explicitly indexed
conductor.elasticsearch.asyncBufferFlushTimeout=10

BASIC Authentication

If you need to pass user/password to connect to ES, add the following properties to your config file:

conductor.elasticsearch.username=someusername
conductor.elasticsearch.password=somepassword