wehub-resource-sync
a9cd7750f4
CI / unit-test (push) Has been cancelled
CI / detect-changes (push) Has been cancelled
CI / build (push) Has been cancelled
Publish docs via GitHub Pages / Deploy docs (push) Has been cancelled
CI / test-harness (push) Has been cancelled
CI / generate-e2e-matrix (push) Has been cancelled
CI / e2e (push) Has been cancelled
CI / build-ui (push) Has been cancelled
Release Drafter / update_release_draft (push) Has been cancelled
UI v2 Integration CI / E2E (Integration) (push) Has been cancelled
UI v2 CI / Lint, Format & Test (push) Has been cancelled
UI v2 CI / E2E (Mocked) (push) Has been cancelled
83 lines
2.6 KiB
Markdown
83 lines
2.6 KiB
Markdown
# ES8 Persistence
|
|
|
|
This module provides Elasticsearch 8.x persistence for indexing workflows and tasks.
|
|
|
|
It uses the Elasticsearch Java API Client (`elasticsearch-java`) aligned with ES8.
|
|
|
|
## Index management (ES8 best practices)
|
|
|
|
This module uses composable index templates, write aliases, and an ILM policy:
|
|
|
|
- **ILM policy:** `${indexPrefix}-default-ilm-policy`
|
|
- **Rollover conditions (hot phase):** `max_primary_shard_size=50gb`
|
|
- **Write aliases:** `${indexPrefix}_workflow`, `${indexPrefix}_task`, `${indexPrefix}_task_log`,
|
|
`${indexPrefix}_message`, `${indexPrefix}_event`
|
|
- **Initial indices:** `${alias}-000001` (created automatically when index management is enabled)
|
|
- **Refresh interval:** defaults to `30s` for all indices (configurable)
|
|
|
|
## Build and usage
|
|
|
|
### 1) Build-time module selection
|
|
|
|
When building the server, select the ES8 persistence module to avoid Lucene conflicts:
|
|
|
|
```sh
|
|
./gradlew :conductor-server:bootJar -PindexingBackend=elasticsearch8
|
|
```
|
|
|
|
`-PindexingBackend=es8` is also accepted.
|
|
|
|
### 2) Runtime configuration
|
|
|
|
Select the Elasticsearch 8 backend in your configuration:
|
|
|
|
```properties
|
|
conductor.indexing.type=elasticsearch8
|
|
```
|
|
|
|
### Elasticsearch version compatibility
|
|
|
|
The ES8 module uses `elasticsearch-java` client version `8.19.11`.
|
|
For local Docker-based setups, use Elasticsearch `8.19.x` (the provided compose file pins
|
|
`8.19.11`).
|
|
|
|
All other `conductor.elasticsearch.*` properties are shared with the ES7 module.
|
|
|
|
### Configuration
|
|
|
|
(Default values shown below)
|
|
|
|
```properties
|
|
# A comma separated list of scheme/host/port of the ES nodes to communicate with.
|
|
# Scheme can be `http` or `https`. If scheme is omitted then `http` will be used.
|
|
conductor.elasticsearch.url=localhost:9200
|
|
|
|
# The name of the workflow and task index.
|
|
conductor.elasticsearch.indexPrefix=conductor
|
|
|
|
# Default refresh interval applied via the component template.
|
|
conductor.elasticsearch.indexRefreshInterval=30s
|
|
|
|
# Path to a PEM-encoded certificate to trust for HTTPS connections.
|
|
conductor.elasticsearch.trustCertPath=
|
|
|
|
# Worker queue size used in executor service for async methods in IndexDao.
|
|
conductor.elasticsearch.asyncWorkerQueueSize=100
|
|
|
|
# Maximum thread pool size in executor service for async methods in IndexDao
|
|
conductor.elasticsearch.asyncMaxPoolSize=12
|
|
|
|
# Timeout (in seconds) for the in-memory to be flushed if not explicitly indexed
|
|
conductor.elasticsearch.asyncBufferFlushTimeout=10
|
|
```
|
|
|
|
### BASIC Authentication
|
|
|
|
If you need to pass user/password to connect to ES, add the following properties to your
|
|
config file:
|
|
|
|
```
|
|
conductor.elasticsearch.username=someusername
|
|
conductor.elasticsearch.password=somepassword
|
|
```
|