wehub/open-metadata--openmetadata
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
Integration Tests - MySQL + Elasticsearch / Detect Changes (push) Has been cancelled
Details
Integration Tests - MySQL + Elasticsearch / integration-tests-mysql-elasticsearch (push) Has been cancelled
Details
Integration Tests - PostgreSQL + Elasticsearch + Redis / Detect Changes (push) Has been cancelled
Details
Integration Tests - PostgreSQL + Elasticsearch + Redis / integration-tests-postgres-elasticsearch-redis (push) Has been cancelled
Details
Integration Tests - PostgreSQL + OpenSearch / Detect Changes (push) Has been cancelled
Details
Integration Tests - PostgreSQL + OpenSearch / integration-tests-postgres-opensearch (push) Has been cancelled
Details
Java Checkstyle / java-checkstyle (push) Has been cancelled
Details
Maven Collate Tests / maven-collate-ci (push) Has been cancelled
Details
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests-status (push) Has been cancelled
Details
Publish Package to Maven Central Repository / publish-maven-packages (push) Has been cancelled
Details
OpenMetadata Service Unit Tests / Detect Changes (push) Has been cancelled
Details
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests (push) Has been cancelled
Details
OpenMetadata Service Unit Tests / k8s_operator-unit-tests (push) Has been cancelled
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:35:45 +08:00
commit bf2343b7e4
16049 changed files with 3531137 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/assets/memory-primitives.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 208 KiB

docs/assets/open-context-layer-architecture.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 301 KiB

docs/assets/open-context-layer-graph.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 165 KiB

docs/assets/open-context-layer-hero.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 265 KiB

docs/auto-classification/add-support-for-another-entity.md
+1390
View File
File diff suppressed because it is too large Load Diff
docs/company-context-knowledge-pills.md
+620
View File
@@ -0,0 +1,620 @@
# Company Context — File-Extracted Knowledge Pills
- **Date:** 2026-06-12
- **Status:** Implemented (as-built spec for branch `pmbrull/companycontext-entity`)
- **Author:** pmbrull (with Claude)
- **Scope:** 72 files, ~+2818/−81. Backend (Java), schema (JSON), ingestion of
file content into reusable memories, embedding/search, MCP tools, UI badge.
This document describes the feature **as shipped**, not the earlier
pre-implementation design. Where the implementation diverged from that design,
the as-built behavior is recorded here and flagged in
[§13 Known limitations](#13-known-limitations--follow-ups).
---
## 1. Summary
The Context Center ("Drive") lets users upload documents. Previously an upload
was stored, its raw text extracted into `ContextFile.extractedText`, and that was
the end. This feature turns that raw text into structured, retrievable
**knowledge pills** — short question/answer facts about the company — that are
embedded, indexed, linked back to the originating file, and reachable by any
MCP-connected agent.
Pills are **not a new entity**. They are `ContextMemory` rows tagged
`sourceType=FileExtraction` with a `sourceFile` back-reference, riding the
entity's existing embedding + search plumbing.
End-to-end:
```
upload ──▶ store blob (object storage / AssetService)
──▶ extract text (status: Analyzing)
──▶ LLM extracts knowledge pills (status: ExtractingContext)
──▶ pills stored as ContextMemory rows, linked to the file
──▶ pills auto-embedded + indexed on create
──▶ pills retrievable via MCP + searchable by source filename
file carries a processing status surfaced in the UI (status: Processed)
```
## 2. Motivation
`ContextMemory` already existed — "Reusable context memory for Context Center
and AI-assisted retrieval", a question/answer pill that is already
vector-embedded (`contextMemory` is in `AvailableEntityTypes`, has
`ContextMemoryBodyTextContributor` + `ContextMemoryIndex`, and its index carries
parent aliases `["all", "dataAssetEmbeddings"]`). It needed only a new source
type and a file back-reference to become the backing store for file-extracted
knowledge, avoiding a second overlapping "knowledge" concept and a full new
entity stack.
The real new infrastructure is a **generic LLM completion layer** — there was no
chat/completion client before (only `EmbeddingClient` and a NoOp `NLQService`).
It is built decoupled from Context Center so other features (e.g. MCP Chat) can
reuse it.
## 3. Architecture & data flow
```
POST /v1/contextCenter/drive/files (ContextFileResource:208)
└─ store asset (AssetServiceFactory.getService().upload), persist ContextFile + content
└─ extractionService.submit(fileId, contentId) (ContextFileResource:287)
ContextFileProcessingService.process(fileId, contentId) (drive/ContextFileProcessingService.java)
guard: contentId == file.headContentId (else abandon — stale re-upload)
1. markAnalyzing() → status Analyzing (file + content)
2. extractText() [DEFAULT_EXECUTOR, CPU] → read blob via AssetService, text-extract
├─ failure → status Failed (+ processingError on content), extractedText cleared
└─ success → text status (Processed / Unsupported)
3. if text Processed AND LLMClientHolder.isEnabled():
status ExtractingContext (file) [content snapshot stays Processed]
submitMemoryExtraction() [LLM_EXECUTOR, network]
runMemoryExtraction():
deleteExtractedMemories(file, hardDelete=true) ← wholesale replace
ContextMemoryExtractor.extract(file, canonicalText)
chunk → per-chunk LLM call → dedupe → create ContextMemory rows
{sourceType:FileExtraction, sourceFile:<ref>, status:Active,
visibility:Shared} └─ VectorEmbeddingHandler auto-embeds
status Processed (file)
├─ LLM failure → status Failed (+ processingError), extractedText RETAINED
└─ LLM queue full → status Failed ("…queue is full. Please retry later.")
else (LLM disabled): text Processed is terminal; LLM stage never enqueued
```
Two **separate** thread pools by design (`ContextFileProcessingService`):
| Pool | Thread-name prefix | Work | Sizing |
|------|--------------------|------|--------|
| `DEFAULT_EXECUTOR` | `context-file-extraction-` | CPU-bound text extraction | `threads = max(2, cores/2)`, `ArrayBlockingQueue(max(64, threads*8))`, `AbortPolicy`, daemon |
| `LLM_EXECUTOR` | `context-memory-extraction-` | network-bound LLM calls | identical sizing |
Mixing seconds-long LLM calls into the text pool would starve text extraction;
the pools are kept separate (the class javadoc spells this out). They are
currently sized identically — the only difference is the thread-name prefix.
## 4. Key design decisions
| Decision | Choice |
|----------|--------|
| Entity | **Reuse `ContextMemory`** (new `sourceType=FileExtraction` + `sourceFile`); no new entity, no DDL migration |
| LLM layer | New **generic** `LLMCompletionClient` abstraction + per-provider concretes + factory + process-wide holder, modeled on `EmbeddingClient` |
| LLM→entity boundary | A narrow `KnowledgePill` DTO is the anti-corruption layer between untrusted model JSON and the entity (see [§9](#9-the-knowledgepill-dto-boundary)) |
| File↔pill link | `Relationship.MENTIONED_IN` edge, `ContextFile → ContextMemory`; `memoryCount` surfaced on the file |
| Reprocess | **Wholesale hard-delete + recreate** of a file's pills on every extraction run (no versioning, no checksum skip) |
| Async | Reuse + rename the live `ContextFileExtractionService` → `ContextFileProcessingService`; LLM step on its own executor |
| Status | Extend the single `ProcessingStatus` enum: insert `ExtractingContext` between `Analyzing` and `Processed` |
| Embedding | Free — `ContextMemory` already auto-embeds on create; this PR only threads `sourceFile` through the index/body-text |
| MCP | Two purpose-built tools `search_company_context` + `get_company_context` |
| Storage guardrail | New `/system/validate` "Object Storage" step + startup warn — uploads are useless if object storage is NoOp/broken |
## 5. Generic LLM completion layer
Package `org.openmetadata.service.llm` (all new).
### 5.1 Config schema
`openmetadata-spec/.../json/schema/configuration/llmConfiguration.json` →
`org.openmetadata.schema.configuration.LLMConfiguration`.
- `enabled` — boolean, default `false`.
- `provider` — enum `["bedrock", "openai", "azureOpenAI", "google", "anthropic", "noop"]`, default `noop` (Java enum `LLMProvider`).
- `maxConcurrentRequests` — integer, default `5`, minimum `1`.
- Per-provider objects (each `additionalProperties:false`):
| Object | Java type | Auth (masked?) | Notable fields / defaults |
|--------|-----------|----------------|---------------------------|
| `bedrock` | `LLMBedrockConfig` | `awsConfig` → `awsBaseConfig.json` | `modelId` `anthropic.claude-3-5-sonnet-20240620-v1:0`, temp 0.0, maxTokens 4096, timeout 60s |
| `openai` | `LLMOpenAIConfig` | `apiKey` **`mask:true`** | `modelId` `gpt-4o-mini`, `endpoint`, `deploymentName`, `apiVersion` `2024-02-01` (Azure), temp 0.0, maxTokens 4096, timeout 60s |
| `google` | `LLMGoogleConfig` | `apiKey` **`mask:true`** | `modelId` `gemini-2.5-flash`, `endpoint`, temp 0.0, maxTokens 4096, timeout 60s |
| `anthropic` | `LLMAnthropicConfig` | `apiKey` **`mask:true`** | `modelId` `claude-3-5-sonnet-20240620`, `baseUrl` `https://api.anthropic.com`, temp 0.0, maxTokens 4096, timeout 60s |
Masked API keys round-trip through the Secrets Manager. Bedrock delegates
secrecy to `awsBaseConfig.json`.
`conf/openmetadata.yaml` carries an `llmConfiguration` block, fully env-var
overridable (`LLM_ENABLED`, `LLM_PROVIDER`, `LLM_MAX_CONCURRENT_REQUESTS`, and
per-provider `LLM_<PROVIDER>_*` keys — see [§12](#12-configuration-reference)).
Its header comment states plainly that **uploaded document content is sent to the
configured provider when enabled**.
### 5.2 Client hierarchy
- **`LLMCompletionClient`** (abstract, `@Slf4j`): owns a `Semaphore`
(`maxConcurrentRequests` permits; ctor rejects `< 1`), the `complete()` /
`completeStructured()` template methods, JSON-array parsing, and code-fence
stripping. Abstract surface: `doComplete(systemPrompt, userPrompt)` and
`getModelId()`. `DEFAULT_MAX_CONCURRENT_REQUESTS = 5`.
- **Concretes** — `OpenAICompletionClient` (OpenAI + Azure OpenAI),
`AnthropicCompletionClient`, `BedrockCompletionClient` (AWS SDK v2,
`AutoCloseable`), `GoogleCompletionClient`, and `NoopCompletionClient`
(returns `"[]"`, model id `"noop"`).
- **`LLMCompletionClientFactory.create(LLMConfiguration)`** — switch on
`provider` → concrete; null config/provider → Noop. Note: the factory keys off
`provider` only; it does **not** check `enabled` (the holder does).
- **`LLMClientHolder`** — process-wide holder of the single shared client.
`initialize(config)`: `enabled = config != null && Boolean.TRUE.equals(getEnabled())`;
`instance = enabled ? factory.create(config) : new NoopCompletionClient()`. So
a real client is built **only** when `enabled == true`. `get()` never returns
null; `isEnabled()` gates the pipeline; `setForTesting()` is the test seam.
`volatile` fields + `synchronized` mutators.
### 5.3 `completeStructured` contract
```java
<T> List<T> completeStructured(String systemPrompt, String userPrompt, Class<T> elementType)
```
- Acquires a permit, calls `doComplete`, releases in `finally`.
- Strips a leading/trailing ` ``` ` code fence (drops the language tag line).
- Parses with a default `new ObjectMapper()` (so `FAIL_ON_UNKNOWN_PROPERTIES`
is **on**) into `List<elementType>`. Parse failure → `LLMCompletionException`.
- **One retry** on `LLMCompletionException` (logged `warn`), then propagates. The
retry catches `LLMCompletionException` broadly, so transport errors that surface
as `LLMCompletionException` also get one retry.
### 5.4 Providers
All providers use plain chat/messages JSON mode (**no tool/function calling**);
non-200 → `LLMCompletionException("<provider> API returned status …")`.
- **OpenAI / Azure**: JDK `HttpClient`. Azure detected when `endpoint` **and**
`deploymentName` are set → `.../openai/deployments/<dep>/chat/completions?api-version=<v>`
with `api-key` header; else `Authorization: Bearer`. Reads `choices[0].message.content`.
- **Anthropic**: JDK `HttpClient`, `<baseUrl>/v1/messages`, headers `x-api-key` +
`anthropic-version: 2023-06-01`. Reads `content[0].text`.
- **Bedrock**: AWS SDK v2 `BedrockRuntimeClient` (credentials via
`AwsCredentialsUtil.buildCredentialsProvider`, region required). Body uses
`anthropic_version: bedrock-2023-05-31`. Reads `content[0].text`. `close()`
exists but the holder never calls it.
- **Google Gemini**: JDK `HttpClient`,
`.../models/<modelId>:generateContent?key=<apiKey>` (**API key in the URL
query string**). Reads `candidates[0].content.parts[0].text`.
### 5.5 Startup wiring
`OpenMetadataApplicationConfig` gains
`@JsonProperty("llmConfiguration") private LLMConfiguration llmConfiguration`.
`OpenMetadataApplication` startup calls
`LLMClientHolder.initialize(catalogConfig.getLlmConfiguration())` (new), then the
pre-existing `LlmConfigHolder.initialize(...)` (publishes raw config for MCP
Chat). Two holders, same config source.
## 6. Schema & persistence changes
No DDL / Flyway migration — `ContextMemory` and `ContextFile` use the generic
JSON-column entity tables, and the file↔pill link lives in the generic
`entity_relationship` table.
### 6.1 Schemas
- **`contextMemory.json`** — `sourceType` enum gains `FileExtraction`
(`FILE_EXTRACTION`); new property `sourceFile` (`entityReference`, not required).
- **`createContextMemory.json`** — new create-time `sourceFile` (`entityReference`).
- **`contextFile.json`** — `ProcessingStatus` enum gains `ExtractingContext` (3rd,
between `Analyzing` and `Processed`); new derived property `memoryCount`
(integer, default 0).
### 6.2 Repositories
- **`ContextMemoryRepository`** — `FIELD_SOURCE_FILE = "sourceFile"` added to
`PATCH_FIELDS`/`UPDATE_FIELDS`. `storeRelationships` adds an
`addRelationship(sourceFile.id /*from*/, memory.id /*to*/, CONTEXT_FILE,
CONTEXT_MEMORY, Relationship.MENTIONED_IN)` edge. `setFields` resolves it with
`findFrom(memory.id, CONTEXT_MEMORY, MENTIONED_IN, CONTEXT_FILE)` (bulk path via
`findFromBatch`). Patch handled via `updateFromRelationship(...)`.
- **Collision-free by construction:** each relationship hierarchy uses a
distinct `Relationship` — `rootMemory`→`CONTAINS`, `parentMemory`→`PARENT_OF`,
`domains`→`HAS`, `sourceFile`→**`MENTIONED_IN`** — so the four resolvers query
disjoint `(relation, fromType)` tuples.
- **`ContextFileRepository`** — `memoryCount` computed in `setFields` as
`findTo(file.id, CONTEXT_FILE, MENTIONED_IN, CONTEXT_MEMORY).size()`. New
`postDelete` cascades: `deleteExtractedMemories(file, hardDelete)` deletes each
linked pill, propagating the file delete's `hardDelete` flag (soft→soft,
hard→hard). The same method is reused by the reprocess path with `hardDelete=true`.
- **`ContextMemoryMapper`** — passthrough of `create.getSourceFile()`.
## 7. Processing pipeline & status
### 7.1 Status state machine (`ProcessingStatus`)
```
Uploaded ─▶ Analyzing ─▶ (text result) ─▶ Processed ─▶ ExtractingContext ─▶ Processed
│ ▲ (LLM enabled) (final)
├─▶ Failed └─ if LLM disabled: terminal here
└─▶ Unsupported (passthrough from text extractor)
any stage failure ─▶ Failed
```
- The file and its content snapshot carry status independently. When the file
advances to `ExtractingContext`, the **content snapshot keeps `Processed`** (it
received the raw text status).
- `Unsupported` exists in the enum but the service never writes it — it is passed
through verbatim from `ContextFileTextExtractor`.
### 7.2 Service (`ContextFileProcessingService`, was `ContextFileExtractionService`)
- Only call site: `ContextFileResource` upload path →
`extractionService.submit(fileId, contentId)`.
- `submit` wraps `executor.execute(() -> process(...))` and catches
`RejectedExecutionException` (queue full → `Failed`, "Processing queue is full").
- Every stage re-reads from the repo and re-checks `headContentId`; a newer
upload silently abandons the stale content's writes (concurrent re-upload guard).
- `canonicalText` feeds the LLM the **content snapshot's** extracted text
(capped ~1M chars) in preference to the file's indexed text (capped ~200K).
### 7.3 Error handling
- `applyFailure` sets both file and content to `Failed`; `processingError` is
written to the **content snapshot only**, not the file entity.
- **Text-stage** failures clear `extractedText` (+ `pageCount`). **LLM-stage**
failures **retain** `extractedText` (so the file stays indexed and is retriable
by re-upload).
- `extractText` catches `Throwable` but re-throws `VirtualMachineError`.
- No automatic retry anywhere; "retry later" means re-POST the file.
## 8. Knowledge-pill extraction (`ContextMemoryExtractor`)
- `extract(file, text)`: `chunkText` → per-chunk `completeStructured(SYSTEM_PROMPT,
chunk, KnowledgePill.class)` → `dedupe` → `memoryRepository.create(toMemory(pill, fileRef))`.
- **Chunking:** `MAX_PROMPT_CHARS = 60_000`, `MAX_CHUNKS = 8`. Chunk ends snap
back to the last `\n\n` / `\n` / ` ` boundary in the second half of the budget,
else hard-cut at the char cap. **Text past 8 chunks (~480K chars) is dropped
with a warning** — even though `canonicalText` may supply up to ~1M chars.
- **Dedupe:** `LinkedHashMap` keyed by `question.trim().toLowerCase(ROOT)`,
first-wins, insertion order preserved.
- **`isValid`:** `question` and `answer` both non-blank (title/summary/memoryType
optional).
- **`toMemory`** sets the full server-owned envelope: `id` (random UUID), `name`
(`<fileName>-<uuid>`), `fullyQualifiedName`, `title`, `question`, `answer`,
`summary`, `memoryType` (via `parseType`), `status=ACTIVE`,
`sourceType=FILE_EXTRACTION`, `sourceFile=<fileRef>`,
`shareConfig.visibility=SHARED`, `updatedBy=admin`, `updatedAt=now`.
- **`parseType`:** case-insensitive match against `ContextMemoryType`
(`Preference | UseCase | Note | Runbook | Faq`); default `NOTE`.
- **`SYSTEM_PROMPT`:** *"You extract reusable company knowledge from a document as
a JSON array. Each element is an object with keys: title, question, answer,
summary, memoryType (one of Faq, Note, Runbook, UseCase, Preference). Capture
durable facts, definitions, policies, and how-to guidance. Return ONLY the JSON
array, no prose."*
### Reprocess / idempotency
Every LLM run first calls `deleteExtractedMemories(file, hardDelete=true)`,
**hard-deleting all of the file's existing pills**, then recreates from scratch
with fresh random IDs/FQNs. There is **no supersede/versioning** and **no
checksum short-circuit** — re-uploading identical bytes re-runs text extraction
and the LLM and regenerates pills. (Rationale in code: machine-generated pills
are replaced wholesale, so soft-deleted rows would only accumulate with no
restore path.)
## 9. The `KnowledgePill` DTO boundary
`KnowledgePill` (`service/llm/KnowledgePill.java`) is a 5-field record
(`title, question, answer, summary, memoryType`) used **only** as the LLM
deserialization target — deliberately not the `ContextMemory` entity. It is the
anti-corruption boundary between untrusted model JSON and the trusted entity
model. Parsing model output straight into `ContextMemory` is avoided because:
1. **Strict unknown fields.** `completeStructured` uses a default `ObjectMapper`
(`FAIL_ON_UNKNOWN_PROPERTIES` on) and `contextMemory.json` is
`additionalProperties:false` — one stray model key would fail the whole array
parse. `KnowledgePill` carries `@JsonIgnoreProperties(ignoreUnknown = true)`.
2. **Enum leniency.** `ContextMemory.memoryType` is the `ContextMemoryType` enum
(Jackson throws on a non-exact value); `KnowledgePill.memoryType` is a raw
String, so casing/variant forms degrade to `NOTE` in `parseType`.
3. **Server-owned identity.** `contextMemory.json` requires `id`/`name`, and the
whole envelope (FQN, status, sourceType, sourceFile, shareConfig,
updatedBy/At) is set server-side in `toMemory` — the model neither produces
nor should influence these.
4. **Minimal prompt contract.** The DTO is exactly the five fields the prompt
asks for; targeting the entity would expose its nested schema to the model.
`toMemory` + `parseType` is the single mapping layer; validation and dedupe
operate on the DTO before any entity is built.
## 10. Embedding & search
`contextMemory` was already vector-indexable and auto-embedded on create
(`AvailableEntityTypes`, `VectorEmbeddingHandler`, parent aliases
`["all", "dataAssetEmbeddings"]`) — **unchanged here**. This PR threads the new
`sourceFile` reference through three layers so file-derived pills are searchable
by their origin filename:
- `ContextMemoryIndex` — `doc.put("sourceFile", getEntityWithDisplayName(...))`.
- ES/OS mapping — a new `sourceFile` object (id/type/name/displayName/fqn/deleted
as keywords) added identically to all four
`elasticsearch/{en,jp,ru,zh}/context_memory_search_index.json` files (the
per-language copies are byte-identical; no language-specific analyzers).
- `ContextMemoryBodyTextContributor` — appends `source file: <name>` to the
embedding body text.
## 11. Object-storage validation (guardrail)
Uploaded file content lives in object storage (S3 / Azure / in-memory) via
`AssetService`. If that backend is disabled or NoOp, uploads silently "succeed"
but their bytes are discarded and extraction fails with **"Unable to read file
content from object storage"** — producing zero pills. To surface this *before*
users hit it:
- **`SystemRepository.getObjectStorageValidation(config)`** adds an **"Object
Storage"** step to the existing `GET /api/v1/system/status` validation. It fails
fast when object storage is disabled/missing or the factory holds a
`NoOpAssetService`; otherwise it runs a **live write→read→delete round-trip
probe** (tiny `text/plain` asset, 10s per-op timeout, asserts the bytes match,
cleans up in `finally`).
- **`AssetServiceFactory.init`** logs a `warn` when storage is disabled,
naming Context Center Drive and the exact failure string.
- Test (`SystemRepositoryObjectStorageValidationTest`) exercises the real
`AssetServiceFactory` + `InMemoryAssetService`: disabled, missing config,
NoOp-after-config-drift, and a passing live in-memory round-trip.
This reuses the existing `/system/status` framework (no new endpoint) but exists
solely for the Drive → object-storage → extraction pipeline.
## 12. MCP tools
Two tools in `openmetadata-mcp/.../tools/`, registered in `tools.json` and
dispatched by new `DefaultToolContext` switch cases (via the 3-arg, no-limits
`execute` — neither tool records usage or enforces limits).
### `search_company_context`
- **Input:** `query` (string, required); `size` (integer, default 10, clamped 1..50).
- **Query:** `OpenSearchVectorService.search(query, filters, size, from=0, k=100,
threshold=0.0)` with filters pinned to `entityType=contextMemory`,
`sourceType=FileExtraction`, **and `visibility=Shared`**.
- **Output:** `{query, results:[{fullyQualifiedName, name, title, question,
answer, summary, sourceFile, similarityScore}], returnedCount}`.
- **Auth:** global `authorize(CONTEXT_MEMORY, VIEW_ALL)`. Requires vector
embedding enabled, else returns an error payload.
### `get_company_context`
- **Input:** `fqn` (string, required — the `fullyQualifiedName` from a search hit).
- **Lookup:** `Entity.getEntityByName(CONTEXT_MEMORY, fqn,
"sourceFile,owners,tags,domains", null)`.
- **Exposability gate:** returns the pill only if `sourceType=FILE_EXTRACTION`
**and** `shareConfig.visibility=SHARED`, else an error.
- **Output:** `{fullyQualifiedName, name, title, question, answer, summary,
memoryType, sourceFile (as FQN)}`.
- **Auth:** same global `VIEW_ALL`.
Both gate on a single global `VIEW_ALL` plus the `Shared`-visibility scope; there
is no per-pill owner re-check beyond that.
## 13. UI
- **`DocumentStatusBadge`** (new) renders a `ui-core-components` `Badge` (size
`sm`, no icon) from a `ProcessingStatus`, returning `null` when status is
absent. Mapping:
| Status | Color | Label key |
|--------|-------|-----------|
| `Uploaded` | gray | `label.uploaded` |
| `Analyzing` | blue | `label.analyzing` |
| `ExtractingContext` | indigo | `label.extracting-context` |
| `Processed` | success | `label.processed` |
| `Failed` | error | `label.failed` |
| `Unsupported` | warning | `label.unsupported` |
- **`DocumentsView`** renders the badge inline next to each document's filename,
driven by `file.processingStatus`.
- **i18n:** 4 new `en-us` keys (`label.analyzing`, `label.extracting-context`,
`label.unsupported`, `label.uploaded`); `label.processed`/`label.failed`
pre-existed. Synced across all locales.
## 14. Security & privacy
- LLM provider API keys are `mask:true` config through the Secrets Manager —
never logged, never returned in API responses.
- **File content is sent to the configured LLM provider** during extraction. This
is an explicit, admin-enabled action (`llmConfiguration.enabled`), and the
provider is the admin's configured choice — documented in the yaml header.
- Generated pills default to `visibility=Shared` (org-readable). MCP retrieval
enforces the `Shared` scope and a `VIEW_ALL` authorize on every call.
- No new unauthenticated surface; MCP tools ride existing OAuth/JWT auth.
## 15. Testing
**Java unit (`openmetadata-service`)**
- `drive/ContextFileProcessingServiceTest` — status machine: analyzing→processed,
LLM-enabled extraction, LLM-rejection/failure keeps text, storage-unavailable &
null-stream failures, executor rejection, head-content skip, VME rethrow.
- `drive/ContextMemoryExtractorTest` — one-memory-per-pill, dedupe + skip-invalid,
skip-when-no-text, multi-chunk dedupe, chunk cap.
- `jdbi3/SystemRepositoryObjectStorageValidationTest` — the four storage-validation
scenarios.
- `llm/LLMClientHolderTest`, `llm/LLMCompletionClientFactoryTest`,
`llm/LLMCompletionClientTest` — holder stability + disabled-builds-no-client,
factory dispatch, structured parsing / code-fence / concurrency guard.
**MCP (`openmetadata-mcp`)**
- `tools/GetCompanyContextToolTest` — missing/blank fqn, auth-denied, shared-pill
projection, non-file & private rejection.
- `tools/SearchCompanyContextToolTest` — missing/blank query, auth-denied (input
guards; does not exercise the vector path).
**Integration (`openmetadata-integration-tests`)**
- `it/drive/CompanyContextPipelineIT` — end-to-end: upload `.txt`, poll
(Awaitility, ≤45s) to `Processed`, `assumeTrue(memoryCount > 0)` to skip
gracefully when no LLM provider is configured; otherwise asserts
`memoryCount == pills.size()` and per-pill `sourceType=FILE_EXTRACTION`, non-null
`sourceFile` matching the file id, non-blank question/answer.
**UI** — `DocumentStatusBadge.test.tsx` (renders nothing without status; `it.each`
over the six status→label→color rows). No new Playwright spec.
## 16. Configuration reference
```yaml
llmConfiguration:
enabled: ${LLM_ENABLED:-false}
provider: ${LLM_PROVIDER:-noop} # noop | openai | azureOpenAI | bedrock | google | anthropic
maxConcurrentRequests: ${LLM_MAX_CONCURRENT_REQUESTS:-5}
openai:
apiKey: ${LLM_OPENAI_API_KEY:-""}
modelId: ${LLM_OPENAI_MODEL_ID:-"gpt-4o-mini"}
endpoint: ${LLM_OPENAI_ENDPOINT:-""}
deploymentName:${LLM_OPENAI_DEPLOYMENT:-""}
apiVersion: ${LLM_OPENAI_API_VERSION:-"2024-02-01"}
maxTokens: ${LLM_OPENAI_MAX_TOKENS:-4096}
bedrock:
awsConfig:
awsRegion: ${AWS_DEFAULT_REGION:-""}
# IAM or static keys via AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN
modelId: ${LLM_BEDROCK_MODEL_ID:-"eu.anthropic.claude-haiku-4-5-20251001-v1:0"}
maxTokens: ${LLM_BEDROCK_MAX_TOKENS:-4096}
google:
apiKey: ${LLM_GOOGLE_API_KEY:-""}
modelId: ${LLM_GOOGLE_MODEL_ID:-"gemini-2.5-flash"}
maxTokens: ${LLM_GOOGLE_MAX_TOKENS:-4096}
anthropic:
apiKey: ${LLM_ANTHROPIC_API_KEY:-""}
modelId: ${LLM_ANTHROPIC_MODEL_ID:-"claude-3-5-sonnet-20240620"}
baseUrl: ${LLM_ANTHROPIC_BASE_URL:-"https://api.anthropic.com"}
maxTokens: ${LLM_ANTHROPIC_MAX_TOKENS:-4096}
```
## 17. Known limitations & follow-ups
1. **Chunk tail dropped.** Documents over ~480K chars (8 × 60K) lose their tail to
the LLM, even though `canonicalText` supplies up to ~1M chars. Consider raising
`MAX_CHUNKS` or summarizing the remainder.
2. **`memoryCount` not in bulk path.** It is populated only in single-entity
`setFields`, not `setFieldsInBulk` — list endpoints requesting `memoryCount`
may return it unset.
3. **`processingError` is content-only.** The file entity flips to `Failed` with
no error string; clients must read the content snapshot for the reason.
4. **No checksum short-circuit.** Identical re-uploads pay the full text + LLM
cost; the captured checksum is stored but never compared.
5. **Wholesale reprocess.** Pills are hard-deleted and regenerated with new
UUIDs/FQNs each run — no continuity/versioning for an unchanged pill.
6. **Bedrock client never closed.** `BedrockCompletionClient` is `AutoCloseable`
but `LLMClientHolder` never calls `close()`; a re-`initialize` would leak the
prior SDK client. Low impact (single startup init).
7. **Factory ignores `enabled`.** Only the holder gates on it; calling the factory
directly with `enabled=false` + a real provider would still build a live client.
8. **MCP authorization is coarse.** A single global `VIEW_ALL` + ES
`visibility=Shared` filter, with no per-pill owner re-check.
9. **`Unsupported` status** is in the enum but never written by the service
(passthrough from the text extractor only).
10. **Per-language ES index files are byte-identical** — the jp/ru/zh
`sourceFile` mappings carry no CJK/Russian-specific analyzers.
## 18. File touch-list
**New — backend**
- `openmetadata-spec/.../json/schema/configuration/llmConfiguration.json`
- `openmetadata-service/.../service/llm/` — `LLMCompletionClient`,
`LLMCompletionClientFactory`, `LLMClientHolder`, `LLMCompletionException`,
`KnowledgePill`, `NoopCompletionClient`, `OpenAICompletionClient`,
`AnthropicCompletionClient`, `BedrockCompletionClient`, `GoogleCompletionClient`
- `openmetadata-service/.../service/drive/ContextMemoryExtractor.java`
- `openmetadata-mcp/.../tools/SearchCompanyContextTool.java`, `GetCompanyContextTool.java`
**New — frontend**
- `.../components/.../ContextCenter/DocumentStatusBadge/` (component, interface, test)
**Modified — schema**
- `entity/context/contextMemory.json`, `api/context/createContextMemory.json`,
`entity/data/contextFile.json` (+ generated TS mirrors)
**Modified — backend**
- `OpenMetadataApplication.java`, `OpenMetadataApplicationConfig.java`,
`conf/openmetadata.yaml`
- `service/drive/ContextFileExtractionService.java` → **renamed**
`ContextFileProcessingService.java`
- `resources/drive/ContextFileResource.java`
- `jdbi3/ContextMemoryRepository.java`, `jdbi3/ContextFileRepository.java`,
`jdbi3/SystemRepository.java`, `attachments/AssetServiceFactory.java`
- `resources/context/ContextMemoryMapper.java`
- `search/indexes/ContextMemoryIndex.java`,
`search/vector/ContextMemoryBodyTextContributor.java`,
`elasticsearch/{en,jp,ru,zh}/context_memory_search_index.json`
- `openmetadata-mcp/.../tools/DefaultToolContext.java`,
`openmetadata-mcp/.../resources/json/data/mcp/tools.json`
**Modified — frontend**
- `.../DocumentsView/DocumentsView.component.tsx`, locale files (17), generated TS
**Tests** — across `openmetadata-service`, `openmetadata-mcp`,
`openmetadata-integration-tests`, and UI (see [§15](#15-testing)).
**Codegen after schema edits:** `make generate`, `mvn spotless:apply`, UI
checkstyle for generated TS, `npx tsc --noEmit`.
---
## 19. Follow-on: AISettings + Memory Agent (designed)
**Status:** Designed, not yet built. Full design:
`docs/superpowers/specs/2026-06-18-ai-settings-memory-agent-design.md`. Built on
branch `pmbrull/ottawa`. Two capabilities layered on top of the pill pipeline above.
### 19.1 AISettings (config plane)
A SearchSettings-style settings entity (`configuration/aiSettings.json`,
`SettingsType.AI_SETTINGS`, seeded + merged via `SettingsCache` / a new
`AISettingsHandler`, read/written through the generic `SystemResource`
GET/PUT/reset, UI page under Preferences → AI). Shape:
- master `enabled` kill-switch (gates extraction **and** the agent)
- `memoryExtraction.{fromFiles, fromPages}`
- `memoryAgent.{enabled, deriveGlossaryTerms, deriveMetrics, deletionPolicy}`
(`deletionPolicy` ∈ `cascade | orphan | deprecate`, default `cascade`)
- `prompts.{memoryExtraction, memoryAgent}.systemPrompt`
**Deltas to this doc:**
- The hardcoded `ContextMemoryExtractor.SYSTEM_PROMPT` ([§8](#8-knowledge-pill-extraction-contextmemoryextractor))
moves to `AISettings.prompts.memoryExtraction.systemPrompt` — seed JSON carries
the default, the code constant is kept only as the cache-miss fallback, and the
extractor reads it at run time. This addresses the un-tunable-prompt gap.
- File extraction ([§7.2](#72-service-contextfileprocessingservice-was-contextfileextractionservice)
`submit`) and page extraction now gate **additionally** on AISettings
(`enabled && memoryExtraction.fromFiles` / `fromPages`), not only on
`LLMClientHolder.isEnabled()`.
### 19.2 Memory Agent (memory → ontology)
On `ContextMemory` create/update, an async, throttled agent reconciles the memory
against the **existing** instance ontology and derives Glossary Terms / Metrics —
reusing what already covers the memory, creating only what is missing.
- New **`DERIVED_FROM`** relationship (appended last in `entityRelationship.json`;
`derivedEntity → sourceMemory`).
- `MemoryExtractor` (pure `derive`) + `MemoryReconciler` (owns writes) +
`MemoryProcessingEngine` (orchestrator) in the `drive` package — siblings to
this pill pipeline, same hash-gate / throttle / reconcile idioms — triggered
from new `ContextMemoryRepository.postCreate` / `postUpdate` hooks.
- Per memory, two **independent** axes (Term, Metric), each:
REUSE (`RELATED_TO` edge to an existing entity, not owned) /
CREATE (`provider=automation`, `DERIVED_FROM` edge, agent-owned; mints a
glossary when none fits) / SKIP. Grounding is top-K search over the existing
term/metric indexes (no full-ontology load).
- **Ownership lifecycle:** adopt-on-touch flips `automation→user` on a human PATCH
(mirrors the pills Manual-guard); `deletionPolicy` governs memory-delete cleanup
of owned entities (`cascade` hard-deletes, `orphan` releases, `deprecate` flags).
Agent writes as a dedicated `memory-bot`.
- **Schema:** new `ContextMemory.memoryStats` (hash-gate + derived/reused
counts); new `provider` field on `Metric` ([§6.1](#61-schemas)) for uniform
ownership across Terms/Metrics/Glossaries.
- Renders the full **File → Memory → Term/Metric** provenance chain in the UI.
docs/csv-relation-types-plan.md
+275
View File
@@ -0,0 +1,275 @@
# CSV Import/Export Enhancement for Glossary Term Relations
## Problem Statement
Currently, the glossary CSV import/export only captures related term FQNs without the relation type:
- **Export**: Only exports FQNs like `Glossary.Term1;Glossary.Term2`
- **Import**: Hardcodes all relations to `"relatedTo"`
This causes data loss when:
1. A term has `synonym`, `broader`, `narrower`, or custom relation types
2. CSV is exported and re-imported - all relation types become `"relatedTo"`
## Proposed Solution
### New CSV Format
**Format**: `relationType:termFQN` pairs separated by semicolons
**Examples**:
```csv
# New format with relation types
relatedTerms
synonym:Finance.Revenue;broader:Finance.Income;narrower:Finance.Net Revenue
# Backward compatible - no prefix defaults to "relatedTo"
relatedTerms
Finance.Revenue;Finance.Income
# Mixed format (new and legacy)
relatedTerms
synonym:Finance.Revenue;Finance.Income;broader:Finance.Gross Income
```
### Parsing Rules
1. If a value contains `:` and the part before `:` is a valid relation type → use that relation type
2. If no `:` or the prefix is not a valid relation type → default to `"relatedTo"`
3. Valid relation types are determined by checking `glossaryTermRelationSettings` or using defaults
### Default Relation Types
| Relation Type | Description |
|---------------|-------------|
| `relatedTo` | Generic related term (default) |
| `synonym` | Equivalent term |
| `broader` | More general term |
| `narrower` | More specific term |
| `antonym` | Opposite meaning |
| `partOf` | Component of |
| `hasPart` | Contains |
## Implementation Plan
### Phase 1: Backend Changes
#### 1.1 CsvUtil.java - Export Enhancement
**File**: `openmetadata-service/src/main/java/org/openmetadata/csv/CsvUtil.java`
**Current** (line 253-263):
```java
public static List<String> addTermRelations(
List<String> csvRecord, List<TermRelation> termRelations) {
csvRecord.add(
nullOrEmpty(termRelations)
? null
: termRelations.stream()
.map(tr -> tr.getTerm().getFullyQualifiedName())
.sorted()
.collect(Collectors.joining(FIELD_SEPARATOR)));
return csvRecord;
}
```
**New**:
```java
public static List<String> addTermRelations(
List<String> csvRecord, List<TermRelation> termRelations) {
csvRecord.add(
nullOrEmpty(termRelations)
? null
: termRelations.stream()
.map(tr -> {
String relationType = tr.getRelationType();
String fqn = tr.getTerm().getFullyQualifiedName();
// Only include relation type prefix if not the default "relatedTo"
if (relationType != null && !relationType.equals("relatedTo")) {
return relationType + ":" + fqn;
}
return fqn;
})
.sorted()
.collect(Collectors.joining(FIELD_SEPARATOR)));
return csvRecord;
}
```
#### 1.2 GlossaryRepository.java - Import Enhancement
**File**: `openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/GlossaryRepository.java`
**Current** (line 315-327):
```java
private List<TermRelation> getTermRelationsFromCsv(
CSVPrinter printer, CSVRecord csvRecord, int fieldNumber) throws IOException {
List<EntityReference> entityRefs =
getEntityReferences(printer, csvRecord, fieldNumber, GLOSSARY_TERM);
if (entityRefs == null) {
return null;
}
List<TermRelation> termRelations = new ArrayList<>();
for (EntityReference ref : entityRefs) {
termRelations.add(new TermRelation().withTerm(ref).withRelationType("relatedTo"));
}
return termRelations;
}
```
**New**:
```java
private static final Set<String> VALID_RELATION_TYPES = Set.of(
"relatedTo", "synonym", "broader", "narrower", "antonym", "partOf", "hasPart"
);
private List<TermRelation> getTermRelationsFromCsv(
CSVPrinter printer, CSVRecord csvRecord, int fieldNumber) throws IOException {
String fieldValue = csvRecord.get(fieldNumber);
if (nullOrEmpty(fieldValue)) {
return null;
}
List<TermRelation> termRelations = new ArrayList<>();
String[] entries = fieldValue.split(FIELD_SEPARATOR);
for (String entry : entries) {
String relationType = "relatedTo"; // Default
String termFqn = entry.trim();
// Check for relationType:fqn format
int colonIndex = entry.indexOf(':');
if (colonIndex > 0) {
String prefix = entry.substring(0, colonIndex).trim();
String suffix = entry.substring(colonIndex + 1).trim();
// Validate if prefix is a known relation type
if (VALID_RELATION_TYPES.contains(prefix) || isCustomRelationType(prefix)) {
relationType = prefix;
termFqn = suffix;
}
// If prefix is not a valid relation type, treat entire string as FQN
// (handles FQNs that contain colons like "Database:Schema.Table")
}
EntityReference termRef = getEntityReference(printer, csvRecord, GLOSSARY_TERM, termFqn);
if (termRef != null) {
termRelations.add(new TermRelation().withTerm(termRef).withRelationType(relationType));
}
}
return termRelations.isEmpty() ? null : termRelations;
}
private boolean isCustomRelationType(String relationType) {
// Check against glossaryTermRelationSettings for custom relation types
try {
// Fetch from settings cache or use default list
return false; // Implement based on settings lookup
} catch (Exception e) {
return false;
}
}
```
#### 1.3 Documentation Update
**File**: `openmetadata-service/src/main/resources/json/data/glossary/glossaryCsvDocumentation.json`
Update the `relatedTerms` field documentation:
```json
{
"name": "relatedTerms",
"required": false,
"description": "Related glossary terms with optional relation types. Format: 'relationType:FQN' or just 'FQN'. Multiple values separated by ';'. Valid relation types: relatedTo (default), synonym, broader, narrower, antonym, partOf, hasPart. Example: 'synonym:Glossary.Term1;broader:Glossary.Term2;Glossary.Term3'",
"examples": [
"Glossary.Term1;Glossary.Term2",
"synonym:Glossary.Term1;broader:Glossary.Term2",
"synonym:Glossary.Revenue;Glossary.Income;narrower:Glossary.Net Revenue"
]
}
```
### Phase 2: Testing
#### 2.1 Unit Tests
**File**: `openmetadata-service/src/test/java/org/openmetadata/csv/CsvUtilTest.java`
```java
@Test
void testAddTermRelationsWithRelationType() {
// Test that relation types are included in export
}
@Test
void testAddTermRelationsDefaultRelationType() {
// Test that "relatedTo" terms don't include prefix
}
```
#### 2.2 Integration Tests
**File**: `openmetadata-service/src/test/java/org/openmetadata/service/resources/glossary/GlossaryTermResourceTest.java`
```java
@Test
void testGlossaryTermCsvImportWithRelationTypes() {
// Test importing CSV with relation type prefixes
}
@Test
void testGlossaryTermCsvExportWithRelationTypes() {
// Test exporting terms with various relation types
}
@Test
void testGlossaryTermCsvBackwardCompatibility() {
// Test importing old format CSV (no relation types)
}
@Test
void testGlossaryTermCsvRoundTripWithRelationTypes() {
// Test that export -> import preserves relation types
}
```
### Phase 3: Edge Cases
1. **FQN contains colon**: Handle cases like `Database:Schema.Term` by validating the prefix against known relation types
2. **Invalid relation type**: If prefix is not a valid relation type, treat entire string as FQN with default `relatedTo`
3. **Empty relation type**: `":Glossary.Term"` should default to `relatedTo`
4. **Custom relation types**: Check against `glossaryTermRelationSettings` for user-defined relation types
### Backward Compatibility
| CSV Format | Import Behavior |
|------------|----------------|
| `Glossary.Term1;Glossary.Term2` | All relations → `relatedTo` |
| `synonym:Glossary.Term1;Glossary.Term2` | First → `synonym`, Second → `relatedTo` |
| `synonym:Glossary.Term1;broader:Glossary.Term2` | Preserves both relation types |
### Files to Modify
| File | Change |
|------|--------|
| `CsvUtil.java` | Update `addTermRelations()` to include relation type prefix |
| `GlossaryRepository.java` | Update `getTermRelationsFromCsv()` to parse relation types |
| `glossaryCsvDocumentation.json` | Update field documentation and examples |
| `GlossaryTermResourceTest.java` | Add tests for new format |
| `CsvUtilTest.java` | Add unit tests for parsing |
### Migration Notes
- **No database migration needed**: The database already stores relation types correctly
- **Existing CSVs**: Will continue to work (all imported as `relatedTo`)
- **New exports**: Will include relation type prefixes for non-default relations
## Summary
This enhancement:
1. ✅ Preserves relation types during CSV export/import
2. ✅ Maintains backward compatibility with existing CSVs
3. ✅ Defaults to `relatedTo` when no relation type specified
4. ✅ Follows existing OpenMetadata CSV patterns (`type:value`)
5. ✅ Supports custom relation types via settings
docs/impersonation-design.md
+1687
View File
File diff suppressed because it is too large Load Diff
docs/perf/cdn-deployment-guide.md
+295
View File
@@ -0,0 +1,295 @@
# Shipping Collate / OpenMetadata releases through CloudFront
Each customer gets a Collate deployment at their own host —
`acme.getcolate.io`, `widgets.getcolate.io`, `globex.getcolate.io` — and each customer can
be on a different release. This is the AWS-only design for serving the UI bundle from
CloudFront in that model, and the coordination story when a request lands at one of those
hosts.
## What we want
- **One CloudFront distribution** for every customer (not one per customer).
- **One S3 bucket** for every release. Releases are immutable; promotion is a separate
step from upload.
- **Per-customer version pinning** that updates atomically — no DNS change, no CloudFront
redeploy.
- **Customer's own ALB** continues to serve `/api/*`; CloudFront only handles the UI bundle.
## What we explicitly do NOT want
- A new external data store to maintain (DynamoDB, an extra RDS, a separate Redis). The
customer-version mapping is small (a few hundred entries, two tiny strings each) and
changes rarely (a few writes per week, even at peak). Standing up a data store for that
buys nothing and adds backup, monitoring, IAM, and cost surface.
- Per-customer CloudFront distributions. They give clean isolation but at N customers we
have N distributions to manage, N caches that share no edge state across customers, and
hit the AWS 200-distributions-per-account cap by default. The savings from edge cache
sharing (a thousand customers on v1.12.0 hit the same cached chunk) are the entire
reason the shared model is worth using.
- A lookup that requires Lambda@Edge. The cold start and per-request cost is real
($1+/M, plus 30-60 ms when cold) and we don't need the SDK access Lambda@Edge gives.
## The architecture
```
┌──────────────────────────────────────┐
acme.getcolate.io ───┐ │ CloudFront distribution │
widgets.getcolate.io ───┼─────►│ d1234abc.cloudfront.net │
globex.getcolate.io ───┘ │ │
│ ┌─ behavior: /* ──────────────────┐ │
│ │ origin: S3 │ │ ┌─────────────────────────────┐
│ │ viewer-request: host_router.js │─┼────►│ S3: collate-cdn │
│ │ rewrites /foo → │ │ │ release/v1.11.2/index.html │
│ │ /release/<version>/foo │ │ │ release/v1.12.0/index.html │
│ └──────────────────────────────────┘ │ │ release/v1.13.0-beta/... │
│ │ └─────────────────────────────┘
│ ┌─ behavior: /api/* ──────────────┐ │
│ │ bypass: same host's per- │ │ ┌─────────────────────────────┐
│ │ customer ALB (Option A below) │─┼────►│ Each customer's own ALB │
│ └──────────────────────────────────┘ │ └─────────────────────────────┘
└──────────────────────────────────────┘
```
The CloudFront Function holds the customer→version routing table **as JavaScript object
literal**. Source of truth is the Function's source code in our git repo. Promotion is
a Function code update.
## The Function (no external lookup)
```js
// host_router.js — CloudFront Function v2.0 (no Lambda@Edge, no KVS, no DynamoDB)
//
// Source of truth for which release each customer is pinned to. Edit, commit, deploy.
// CI propagates a change to every edge POP in ~60 s.
const CUSTOMER_VERSIONS = {
acme: 'v1.12.0',
widgets: 'v1.11.2',
globex: 'v1.13.0-beta',
// … N customers
};
// Hosts that don't match a customer slug (apex, www., staging) fall back to the latest
// stable release. Bump this in lockstep with every GA release so new customers that
// haven't been added to CUSTOMER_VERSIONS yet still get a current build.
const DEFAULT_VERSION = 'v1.12.0';
function handler(event) {
const request = event.request;
// /api/* lives on a separate behavior with the customer's own ALB as origin.
// The Function should never see these requests under the current behavior config,
// but guard anyway.
if (request.uri.startsWith('/api/')) {
return request;
}
const host = (request.headers.host && request.headers.host.value) || '';
// Convention: customer slug is the first label of the host.
// acme.getcolate.io -> 'acme'
const slug = host.split('.')[0];
const version = CUSTOMER_VERSIONS[slug] || DEFAULT_VERSION;
// /assets/foo.js -> /release/v1.12.0/assets/foo.js
request.uri = '/release/' + version + request.uri;
return request;
}
```
Function v2.0 has a 10 KB code limit. At ~30 bytes per entry that's ~300 customers
comfortably; well beyond that the design needs revisiting — but if you ever reach 300+
customers on this product, the operational economics of standing up KVS or DynamoDB
will have shifted significantly anyway.
## Promotion flow
1. Edit `CUSTOMER_VERSIONS` in the Function source.
2. Commit, push, open PR. The PR diff IS the promotion record — reviewable, auditable,
git-blame'd.
3. CI runs on merge: pushes the new Function code via `aws cloudfront update-function`
and `publish-function`.
4. ~60 s of edge propagation. Every POP picks up the new code.
A typical promotion PR looks like one line changed:
```diff
const CUSTOMER_VERSIONS = {
- acme: 'v1.12.0',
+ acme: 'v1.12.1',
widgets: 'v1.11.2',
globex: 'v1.13.0-beta',
};
```
That's the entire surface area of a promotion. No DynamoDB write. No KVS API call. No
extra IAM role. No backup story. Just a code change reviewed like any other.
Rollback is symmetric: revert the commit. Canary is "promote one slug first, watch error
metrics, then PR the next batch." Roll-forward on a regression is the same revert.
### Release upload (independent of promotion)
The bundle bytes go to S3 separately, on every release tag, regardless of which customer
ends up using them:
```bash
VERSION="v1.12.0"
aws s3 sync openmetadata-ui/src/main/resources/ui/dist/assets/ \
s3://collate-cdn/release/${VERSION}/assets/ \
--cache-control "public, max-age=31536000, immutable"
aws s3 cp openmetadata-ui/src/main/resources/ui/dist/index.html \
s3://collate-cdn/release/${VERSION}/index.html \
--cache-control "no-cache, must-revalidate" \
--content-type "text/html; charset=utf-8"
```
After this, the release exists in S3 but no customer is using it. Promotion (the PR
above) is what flips customers to it. The decoupling matters: you can sit on a release
in S3 for a week, watching it on staging, before promoting any customer to it.
## Why the Function code is a fine routing table
Honest comparison of the three approaches:
| | Function-embedded (this design) | CloudFront KeyValueStore | DynamoDB + Lambda@Edge |
|---|---|---|---|
| New AWS service to monitor / back up | none | KVS | DynamoDB + Lambda |
| Read latency at edge | ~0 (in-function) | ~1 ms | ~10 ms (warm Lambda) |
| Cold start | none | none | 30-60 ms |
| Per-request cost | $0.10/M Function | $0.10/M Function + $0.04/M KVS | $0.10/M + $1+/M Lambda + DynamoDB reads |
| Promotion surface | git PR | API call (`put-key`) | API call (`update-item`) |
| Audit trail | git history | CloudWatch + KVS audit logs | CloudWatch + DDB streams |
| Capacity ceiling | ~300 customers (10 KB code limit) | millions | millions |
| Concurrent promotion safety | git merge serializes | `IfMatch` ETag | conditional writes |
| Operational ownership | "this is in the repo" | "who paged on this last quarter?" | "who paged on this last quarter?" |
For a product that ships per-customer clusters and reaches dozens-to-low-hundreds of
customers, "the routing table is a file in the repo" wins on every operational axis that
matters. It only loses on capacity ceiling, and the day that becomes a problem we already
have a clear migration target (KVS) without changing anything else in the design.
## API routing — two options, pick one
The Function above only handles UI bundle requests. `/api/*` still has to reach the
customer's own ALB.
### Option A — Separate API host (recommended)
```
acme.getcolate.io → CNAME → CloudFront distribution (this design)
api-acme.getcolate.io → CNAME → acme's ALB
```
SPA's API base URL is derived from the page host at runtime: `https://api-{slug}.getcolate.io/api`.
Pros: CloudFront does one thing well (static delivery). No Lambda@Edge anywhere. Failure
modes are easy to reason about. Cons: SPA has a cookie/CORS story that knows about two
hosts; we already handle this for various integrations.
### Option B — Same host, Lambda@Edge for `/api/*`
CloudFront's `/api/*` behavior runs a Lambda@Edge on origin-request that reads the host
header and rewrites the origin to the right ALB.
Pros: single host per customer. Cons: now we DO have Lambda@Edge (which we explicitly
chose to avoid for routing), and the operational cost is per-customer-API-request, not
just per-promotion. We strongly prefer Option A.
## S3 bucket layout
```
collate-cdn/
└── release/
├── v1.11.5/
│ ├── index.html no-cache, must-revalidate
│ ├── assets/index-Z3O_FBkA.js immutable
│ ├── assets/index-Z3O_FBkA.js.br immutable
│ ├── assets/index-Z3O_FBkA.js.gz immutable
│ ├── assets/vendor-antd-BgrjOjhB.js immutable
│ └── ...
├── v1.12.0/ ← acme + widgets currently here
│ └── ...
└── v1.13.0-beta/ ← globex currently here (canary)
└── ...
```
Releases are immutable once uploaded. The promotion step never modifies S3 contents —
only the Function code that maps `slug → /release/<v>/`.
Disk cost is small: a typical OM bundle is ~12 MB on disk after content-hash dedup,
Brotli+gzip siblings add ~25%, call it 15 MB per release. 100 releases × 15 MB =
1.5 GB. S3 standard rates put that at a few cents per month — keep many releases live
for instant rollback and don't bother with aggressive lifecycle pruning.
## CloudFront cache behaviors
| Path pattern (after Function rewrite) | Edge TTL | Notes |
|---|---|---|
| `/release/<v>/assets/*` | 1 year | Content-addressed; bytes can't change |
| `/release/<v>/index.html`, `/release/<v>/` | 30 s | Concurrent users in one region share one origin hit; ETag layer takes over after 30 s |
| `/api/*` | bypass | Separate behavior to customer ALB (Option A: not via CloudFront at all) |
30 s on the shell is the sweet spot: long enough to dedupe a thousand concurrent reloads
to one origin fetch, short enough that a promotion lands at all customers within ~90 s
end-to-end (60 s Function propagation + 30 s residual edge cache).
## Per-customer branding (without per-customer bundles)
If a customer needs a different logo or accent colour, the right move is to keep one
universal bundle and overlay branding assets at request time:
- Universal default: `/release/v1.12.0/images/logo.png` in S3.
- Per-customer override (optional, only when needed): the Function checks for
`s3://collate-cdn/customer-overrides/<slug>/logo.png` first and rewrites if it exists.
Branding stays out of the build artifact, which means one bundle still serves every
customer and the cache-sharing argument holds.
## Verification after promotion
Two synthetic checks worth running automatically after a promotion PR merges:
```bash
SLUG=acme
EXPECTED_VERSION=v1.12.0
# 1. CloudFront serves the right release for this slug
RESPONSE=$(curl -s "https://${SLUG}.getcolate.io/?nocache=$(uuidgen)")
echo "$RESPONSE" | grep -oE 'index-[A-Za-z0-9_-]+\.js' | sort -u
# Should match the hash from the v1.12.0 build manifest
# 2. The HTML shell is being served fresh from the right S3 prefix
curl -sI "https://${SLUG}.getcolate.io/" \
| grep -i 'x-amz-cf-pop\|via\|x-cache'
# Should show an edge POP near the test runner, and either "Miss from cloudfront"
# (first request after promotion) or "Hit from cloudfront" (within the 30 s edge TTL)
```
CI runs this on every promotion PR after the Function deploys, and fails loud if the
served bundle doesn't match the version we just pinned.
## What's not in this design
- **Per-customer API origin selection inside CloudFront**. Option A keeps `/api/*` off
the CloudFront path entirely. If a customer ever needs single-host behavior, that's
the moment to revisit Option B and accept Lambda@Edge.
- **Multi-region S3 origin failover**. Single bucket in one region; CloudFront's edge
caching handles regional reach. If you want CRR + origin groups, add them; the cost
is straightforward but rarely justified for a UI bundle.
- **WAF / Shield Advanced**. Add separately if your security posture requires them.
## What this design is good for and what would push it elsewhere
- **Good for**: dozens to low-hundreds of customers, infrequent promotion (a few per
week), engineering ownership over the routing table.
- **Push toward KVS** when: customer count grows past a few hundred (function size
pressure) OR promotions happen via a non-engineering UI (a customer-success dashboard
that flips slugs without a git PR).
- **Push toward Lambda@Edge** when: routing decisions stop being a slug→version map and
start needing per-request information not available in the host header (e.g. A/B
testing by user ID, geo-routing, header-derived feature flags).
When those days come, the migration path from this design is small — the Function code
becomes a `kvs.get(slug)` instead of a hash lookup, and the rest of the architecture
(S3 layout, distribution behaviors, ALB routing) is identical.
docs/plans/2026-01-27-search-indexing-stats-redesign.md
+384
View File
@@ -0,0 +1,384 @@
# Search Indexing Stats Redesign
## Overview
Redesign the SearchIndexingApp stats tracking to simplify the current complex implementation and add support for vector embedding statistics.
## Goals
1. **Simplify stats building** - Replace multi-source stats with single pipeline model
2. **Add vector embedding stats** - Track vector indexing separately without affecting overall job status
3. **Per-entity index promotion** - Promote staged indexes immediately per entity type
4. **Alias management from indexMapping.json** - Use configuration instead of reading from old index
5. **Payload-aware vector bulk processor** - Respect payload size limits for vector chunks
## Design
### 1. Simplified Stats Architecture
Replace the current multi-source stats with a **single pipeline model**:
```
Read Stage → Process Stage → Sink Stage ──→ Vector Stage
↓ ↓ ↓ ↓
ReaderStats ProcessStats SinkStats VectorStats
↓ ↓ ↓ ↓
└──────────────┴──────────────┴──────────────┘
↓
search_index_server_stats (single source of truth)
```
#### Stats Structure (per entity type, per server)
```java
public class PipelineStats {
// Reader: Database read operations
int readerSuccess; // Entities read successfully
int readerFailed; // Critical read errors (DB issues)
int readerWarnings; // Non-critical (stale references, still processed)
// Process: Entity → SearchDoc conversion
int processSuccess; // Docs built successfully
int processFailed; // Build failures (EntityNotFoundException, schema errors)
int processWarnings; // Non-critical processing issues
// Sink: Elasticsearch/OpenSearch write
int sinkSuccess; // Docs indexed successfully
int sinkFailed; // Index failures (rejected, mapping errors)
int sinkWarnings; // Partial success (some fields skipped)
// Vector: Vector embedding indexing (Collate-specific)
int vectorSuccess; // Embeddings indexed successfully
int vectorFailed; // Embedding failures (API errors, chunk issues)
int vectorWarnings; // Non-critical (fingerprint match, skipped regeneration)
}
```
#### Key Points
- **No reconciliation needed** - each stage reports its own accurate counts
- **Vector stats are independent** - don't affect overall job success/failure
- **Single DB table** as source of truth, updated incrementally
### 2. Database Schema Updates
#### Update `search_index_server_stats` Table
```sql
ALTER TABLE search_index_server_stats ADD COLUMN (
-- Process stage (new)
processSuccess INT DEFAULT 0,
processFailed INT DEFAULT 0,
processWarnings INT DEFAULT 0,
-- Vector stage (new - Collate specific)
vectorSuccess INT DEFAULT 0,
vectorFailed INT DEFAULT 0,
vectorWarnings INT DEFAULT 0
);
```
#### Update `search_index_failures` Table
```sql
ALTER TABLE search_index_failures
MODIFY COLUMN failureStage ENUM(
'READER', -- DB read failure
'READER_EXCEPTION', -- Non-critical read issue
'PROCESS', -- Entity → Doc conversion failure
'SINK', -- ES/OpenSearch write failure
'VECTOR_SINK' -- Vector embedding failure
);
```
#### Migration Strategy
- Add new columns with defaults (non-breaking)
- New code writes to new columns
- Old `entityBuildFailures` mapped to `processFailed` during aggregation (temporary)
- Clean removal in future release
### 3. Simplified Stats Tracking Code
#### New `StageStatsTracker` Class
```java
public class StageStatsTracker {
private final String jobId;
private final String serverId;
private final String entityType;
// Atomic counters per stage
private final StageCounter reader = new StageCounter();
private final StageCounter process = new StageCounter();
private final StageCounter sink = new StageCounter();
private final StageCounter vector = new StageCounter();
// Record success/failure/warning for each stage
public void recordReader(Result result) { reader.record(result); }
public void recordProcess(Result result) { process.record(result); }
public void recordSink(Result result) { sink.record(result); }
public void recordVector(Result result) { vector.record(result); }
// Flush to DB periodically (every N operations or time interval)
public void flush() {
searchIndexStatsRepository.upsert(jobId, serverId, entityType,
reader, process, sink, vector);
}
}
public class StageCounter {
private final AtomicInteger success = new AtomicInteger();
private final AtomicInteger failed = new AtomicInteger();
private final AtomicInteger warnings = new AtomicInteger();
public void record(Result result) {
switch (result) {
case SUCCESS -> success.incrementAndGet();
case FAILED -> failed.incrementAndGet();
case WARNING -> warnings.incrementAndGet();
}
}
}
```
#### Usage in Pipeline
```java
// In SearchIndexExecutor
for (Entity entity : batch) {
// Read stage
try {
entity = readEntity(id);
tracker.recordReader(SUCCESS);
} catch (EntityNotFoundException e) {
tracker.recordReader(WARNING); // Non-critical, continue
continue;
} catch (Exception e) {
tracker.recordReader(FAILED); // Critical
recordFailure(entity, READER, e);
continue;
}
// Process stage
try {
doc = entity.buildSearchIndex();
tracker.recordProcess(SUCCESS);
} catch (Exception e) {
tracker.recordProcess(FAILED);
recordFailure(entity, PROCESS, e);
continue;
}
// Sink stage - handled by BulkSink callback
bulkSink.add(doc, entity, tracker);
}
```
### 4. Immediate Per-Entity Index Promotion
#### Current Flow (Wait for All)
```
Reindex table → Reindex dashboard → Reindex pipeline → ... → Promote ALL at once
```
#### New Flow (Promote Immediately)
```
Reindex table → Promote table immediately
Reindex dashboard → Promote dashboard immediately
Reindex pipeline → Promote pipeline immediately
```
#### Code Changes in `DefaultRecreateHandler`
```java
public class DefaultRecreateHandler implements RecreateHandler {
// Called after EACH entity type completes (not at the end)
public void promoteEntityIndex(String entityType, boolean success) {
ReindexContext context = getContext();
String stagedIndex = context.getStagedIndex(entityType);
String canonicalIndex = context.getCanonicalIndex(entityType);
if (!success) {
// Delete failed staged index, keep old index active
deleteIndex(stagedIndex);
LOG.warn("Reindex failed for {}, keeping old index", entityType);
return;
}
// Get aliases from indexMapping.json (not from old index)
Set<String> aliases = getAliasesFromMapping(entityType);
// Delete old indices with this prefix (except staged)
deleteOldIndices(canonicalIndex, stagedIndex);
// Promote: attach all aliases to staged index
attachAliases(stagedIndex, aliases);
LOG.info("Promoted {} -> {}", entityType, stagedIndex);
}
// Read aliases from indexMapping.json
private Set<String> getAliasesFromMapping(String entityType) {
IndexMapping mapping = indexMappings.get(entityType);
Set<String> aliases = new HashSet<>();
// Add parent aliases (e.g., "all", "dataAsset")
aliases.addAll(mapping.getParentAliases());
// Add short alias (e.g., "table")
aliases.add(mapping.getAlias());
// Add canonical index name as alias (e.g., "table_search_index")
aliases.add(mapping.getIndexName());
return aliases;
}
}
```
### 5. Vector Bulk Processor with Payload Size Handling
```java
public class VectorBulkProcessor {
private final List<BulkOperation> buffer = new ArrayList<>();
private final AtomicLong currentPayloadBytes = new AtomicLong(0);
private final int maxBulkActions; // e.g., 500 chunks
private final long maxPayloadSizeBytes; // e.g., 50MB (conservative for vectors)
public void addChunk(VectorChunk chunk, StageStatsTracker tracker) {
long chunkSize = estimateChunkSize(chunk);
// Flush if adding this chunk would exceed limits
if (shouldFlush(chunkSize)) {
flush();
}
buffer.add(toBulkOperation(chunk));
currentPayloadBytes.addAndGet(chunkSize);
}
private boolean shouldFlush(long incomingSize) {
return buffer.size() >= maxBulkActions
|| (currentPayloadBytes.get() + incomingSize) > maxPayloadSizeBytes;
}
private long estimateChunkSize(VectorChunk chunk) {
// Vector: dimensions × 4 bytes (float32)
long vectorSize = chunk.getEmbedding().length * 4L;
// Metadata: estimate JSON overhead
long metadataSize = chunk.getMetadataJson().length();
// Buffer for ES overhead
return (long) ((vectorSize + metadataSize) * 1.2);
}
public void flush() {
if (buffer.isEmpty()) return;
try {
BulkResponse response = client.bulk(buffer);
processResponse(response); // Update stats via tracker
} finally {
buffer.clear();
currentPayloadBytes.set(0);
}
}
}
```
### 6. Unified Failure Recording
```java
public class IndexingFailureRecorder {
private final List<SearchIndexFailure> buffer = new ArrayList<>();
private static final int BATCH_SIZE = 100;
public enum FailureStage {
READER, // DB read failure
READER_EXCEPTION, // Non-critical read issue
PROCESS, // Entity → Doc conversion failure
SINK, // ES/OpenSearch write failure
VECTOR_SINK // Vector embedding failure
}
public void recordFailure(
String jobId,
String entityType,
String entityId,
String entityFqn,
FailureStage stage,
Exception error) {
SearchIndexFailure failure = SearchIndexFailure.builder()
.jobId(jobId)
.serverId(getServerId())
.entityType(entityType)
.entityId(entityId)
.entityFqn(entityFqn)
.failureStage(stage)
.errorMessage(truncate(error.getMessage(), 65000))
.stackTrace(truncate(getStackTrace(error), 65000))
.timestamp(System.currentTimeMillis())
.build();
synchronized (buffer) {
buffer.add(failure);
if (buffer.size() >= BATCH_SIZE) {
flush();
}
}
}
public void flush() {
synchronized (buffer) {
if (!buffer.isEmpty()) {
repository.batchInsert(buffer);
buffer.clear();
}
}
}
}
```
## Implementation Plan
### Files to Modify
**OpenMetadata Submodule:**
| File | Changes |
|------|---------|
| `SearchIndexApp.java` | Simplify stats aggregation, remove reconciliation |
| `SearchIndexExecutor.java` | Use `StageStatsTracker`, clean pipeline flow |
| `DefaultRecreateHandler.java` | Per-entity promotion, alias from indexMapping.json |
| `OpenSearchBulkSink.java` | Integrate with `StageStatsTracker` |
| `StatsReconciler.java` | Remove or deprecate |
| DB migration | Add process/vector columns to stats table, update failure stage enum |
**Collate:**
| File | Changes |
|------|---------|
| `SearchRepositoryExt.java` | Initialize vector stats tracking |
| `OpenSearchBulkSinkExt.java` | Add payload-aware vector bulk processor |
| `ElasticSearchBulkSinkExt.java` | Same as above for ES |
| `RecreateWithEmbeddings.java` | Per-entity promotion for vector index |
### New Files to Create
| File | Purpose |
|------|---------|
| `StageStatsTracker.java` | Clean stats tracking per stage |
| `StageCounter.java` | Atomic counter for success/failed/warnings |
| `VectorBulkProcessor.java` | Payload-aware bulk processor for vectors |
| `PipelineStats.java` | Stats data model |
### Implementation Order
1. DB migrations (add columns, backward compatible)
2. `StageStatsTracker` and `StageCounter` (new code, no breaking changes)
3. Update `SearchIndexExecutor` to use new tracker
4. Update `DefaultRecreateHandler` for per-entity promotion
5. Add `VectorBulkProcessor` in Collate
6. Update `OpenSearchBulkSinkExt` for vector stats
7. Remove old reconciliation code
docs/plans/2026-06-22-bulk-deletion-redesign.md
+358
View File
@@ -0,0 +1,358 @@
# Bulk Recursive Deletion Redesign (Service-Level, At Scale)
**Status:** Proposed
**Date:** 2026-06-22
**Supersedes / replaces:** the FQN-prefix approach on branch `mohit/35dc-improve-deletion`
**Related code:** `EntityRepository`, `PrefixDeletionService`, `CollectionDAO.RelationshipDAO`, `HierarchicalLockManager`, `DeletionLockDAO`, `SearchRepository`
## Overview
Hard-deleting a service (e.g. a `databaseService` with 100k–1M descendant tables/columns) currently takes **2–6 hours** and frequently leaves **orphaned `entity_relationship` rows** behind. This document specifies a deletion subsystem that is **fast** (set-based, not per-entity), **orphan-free by construction** (deletes by entity **id-set**, immune to NULL hashes and renames), **atomic and resumable** (chunked transactions with a durable job/tombstone), and **safe under concurrent ingestion** (creates under a deleting subtree are rejected).
The design deliberately reuses primitives that **already exist** in the codebase rather than inventing new SQL.
## Implementation status against current `main` (2026-06-22)
This doc was first written against an April snapshot + the `mohit/35dc-improve-deletion`
prefix-deletion branch. **Latest `main` has since converged on most of this design's core**, which
materially narrows the remaining work. Verified against `EntityRepository` on `main`:
- ✅ **Per-level, per-type batched deletion** (`bulkHardDeleteSubtree` / `bulkSoftDeleteSubtree` /
`bulkRestoreSubtree`, dispatched from `deleteChildren`). Replaces the old per-entity
`cleanup()`-per-descendant transaction loop — the comments cite ~120k round-trips collapsed for a
12k-table DB.
- ✅ **Relationships deleted by entity id-set** via `RelationshipDAO.batchDeleteRelationships(ids,
type)` (`DELETE … WHERE fromId IN(…) … OR toId IN(…)`), i.e. the NULL-immune key this design
argued for — **not** fqnHash prefixes. The prefix-branch approach is obsolete.
- ✅ Entity-row deletes chunked at `MAX_IN_LIST_CHUNK_SIZE = 30_000` (`EntityDAO.deleteByIds`).
- ✅ Both `tag_usage` sides cleaned (`deleteTagLabelsByTargetPrefix` + `deleteTagLabelsByFqn`),
cache invalidation + NotFoundCache markers for every deleted descendant, per-entity `postDelete`
+ `deleteFromSearch`.
**Remaining gaps (what this design still drives):**
1. **Bounded memory — DONE in this change.** `bulkHardDeleteSubtree` loaded an entire tree level
(`loadForBulk(ids, ALL)`) before deleting — a 1M-table service OOMs on the load. Now the level is
processed in `BULK_HARD_DELETE_TXN_CHUNK_SIZE`-sized chunks (load → recurse children → purge),
bounding peak heap to ~chunk × tree-depth hydrated entities. (Soft-delete / restore share the same
ceiling and remain a follow-up.)
2. **Per-chunk transaction — follow-up.** The chunk purge is still per-DAO-call autocommit (matching
prior behavior). Wrapping each chunk in one `flushInOneTransaction` (from PR #28675) gives
per-chunk atomicity + deadlock-retry; the deletes are idempotent so it is safe to add.
3. **Concurrency race — deferred (separate "accuracy" PR).** The lock gate is **dormant on main**:
`LockManagerInitializer.initialize()` has no caller, so `lockManager` is null and
`checkModificationAllowed` is a no-op; even if enabled, `loadLockedFqnPrefixes()` is still a stub
and there is no stale-lock reaper (`cleanupStaleLocks()` has no caller). Closing the race safely
requires: wire startup init, implement `loadLockedFqnPrefixes` via `DeletionLockDAO` (cached),
and schedule the reaper — otherwise a crashed delete blocks ingestion under the prefix forever.
4. **Per-entity satellite + search loops — DONE in this change (the headline speedup).** The bulk
recursion still ran, *per descendant*: `field_relationship.deleteAllByPrefix`,
`tagUsageDAO.deleteTagLabelsByTargetPrefix` + `deleteTagLabelsByFqn`, `usageDAO.delete(id)`, and
`deleteFromSearch(entity)` (which serializes a snapshot + submits a lane task each). For an
N-entity subtree that is ~3N satellite round-trips + N search dispatches — which a local 100k
benchmark showed dominated the wall-clock (see below). Fixed via a capability field
`descendantsCoveredByAncestorCascade` (declared on `EntityRepository`, default `false`, set in
the constructor like `supportsSearch`; enabled across all service-rooted asset trees —
database (`Database`/`DatabaseSchema`/`Table`/`StoredProcedure`), dashboard
(`Dashboard`/`Chart`/`DashboardDataModel`), messaging (`Topic`), pipeline (`Pipeline`),
mlmodel (`MlModel`), search (`SearchIndex`), storage (`Container`), drive
(`Directory`/`File`/`Spreadsheet`/`Worksheet`), and api (`APICollection`/`APIEndpoint`)). When
set, the bulk path:
- **skips per-entity `deleteFromSearch`** — the root's own `deleteFromSearch` already fires
`SearchRepository.deleteOrUpdateChildren`, which deletes *all* descendant docs in one
delete-by-query by `service.id` / parent-id;
- **skips per-entity `field_relationship` + `tag_usage`** — the root's `cleanup()` already
prefix-deletes the whole FQN subtree in one statement each;
- **batches `usage`** by id-set (`deleteByIds` IN-list per chunk; usage is id-keyed so the root's
FQN-prefix cleanup doesn't cover descendants).
Default `false` keeps flat-FQN / non-cascade-covered types (Team, User, Role, Policy, …) and
reference types whose deletion scrubs refs out of surviving docs (Tag, GlossaryTerm, Domain,
DataProduct, TestSuite — `deleteOrUpdateChildren` `updateChildren` cases) on the safe per-entity
path. Enabling the api tree also required adding `Entity.API_SERVICE` to the `service.id` case in
`SearchRepository.deleteOrUpdateChildren` (it had fallen through to the default `apiService.id`
branch, a field api docs don't carry — so the cascade had silently skipped api children).
### Measured result (local Docker, 1 GB heap, MySQL + Elasticsearch)
100k tables under one schema (one service → db → schema), recursive hard-delete via
`DELETE /v1/services/databaseServices/{id}?hardDelete=true&recursive=true`:
| | baseline (per-level batched, current `main` + bounded-memory) | + per-entity satellite/search batching |
|---|---|---|
| **wall-clock** | **1643 s (~27 min)** | **59 s** (~28× faster, ~1700 tbl/s) |
| **peak heap** | 546 MB (median 394) | 493 MB (mean 384) — no OOM at 1 GB |
| **correctness** | subtree gone | subtree gone; `entity_relationship` 100088 → 86 (all 100,002 subtree edges removed, no orphans); ES table docs for the service = 0 (search clean) |
Extrapolated: ~1M tables would go from the reported multi-hour range to **~10 min** at this rate.
Still open as follow-ups: per-chunk `flushInOneTransaction` atomicity (#2); the concurrency race
(#3); extending the capability flag to the other service trees; and applying the same skips to
`bulkSoftDeleteSubtree` / `bulkRestoreSubtree`.
## Problem Statement & Root Causes
### Why it is slow (2–6 hours)
The cascade walks the tree one entity at a time:
`EntityResource.deleteByIdAsync → EntityRepository.delete → deleteChildren → batchDeleteChildren → processDeletionBatch → cleanup()`
The dominant cost is **N independent transactions**: `cleanup()` (`EntityRepository.java:3763`) wraps *each* entity's full cleanup in its own `Jdbi.inTransaction(...)`, and the recursion re-queries children at each level (batches of 50, threshold 100). For ~1M descendants this is millions of transactions + millions of per-entity search calls + millions of per-entity change events. Transaction overhead — not row volume — is the wall.
### Why relationships are orphaned
1. **Cross-cutting edges are never reached by the walk.** The recursive walk follows `CONTAINS`/`PARENT_OF` edges. Non-hierarchical edges (lineage `UPSTREAM`, ownership `OWNS`, `HAS` domain, `FOLLOWS`, dataProduct, tags) that point *into* the subtree from outside are only cleaned if the in-subtree endpoint is individually reached and `cleanup()` runs `deleteAll(id, type)` for it. Any entity missed (see #2) leaves its edges dangling.
2. **The concurrency window.** During the multi-hour walk, ingestion can re-create children that the walk already passed. Those new entities — and their relationships — survive as orphans. The `HierarchicalLockManager` was introduced to stop this but its create-path gate is **not actually wired** (see Appendix B).
## Goals
1. **Speed:** delete a 1M-entity service in **minutes**, bounded by *hundreds* of SQL statements, not millions of transactions.
2. **Orphan-free by construction:** after deletion, **zero** `entity_relationship` / `field_relationship` / `tag_usage` / `entity_extension` / time-series / feed rows reference any deleted entity — regardless of relationship type, hash population, or rename history.
3. **Atomic & resumable:** a crash/restart mid-delete never leaves a *live* entity stripped of its dependencies; the operation resumes and completes.
4. **Concurrency-safe:** ingestion/create under a subtree being deleted is rejected (or queued), closing the orphan race.
5. **Faithful side-effects:** change events, audit log, search index, RDF, alerts/governance, and per-type cleanup behave as if each entity were deleted.
6. **Bounded blast radius:** the bulk path is available only on hierarchical, FQN-nesting roots.
## Non-Goals
- Changing **soft-delete** semantics. Soft delete keeps the existing tree-walk (it must preserve relationships for restore).
- Adding database-level foreign keys. `entity_relationship` references ~60 `*_entity` tables polymorphically via `(fromId, fromEntity)`; `ON DELETE CASCADE` is not expressible, and the schema is FK-free by design.
- A general distributed job framework. We reuse the existing async executor + `entity_deletion_lock` table.
## Core Design Decision: delete by **id-set**, not by FQN-hash prefix
The single most important decision is the **deletion key**.
- **FQN-hash prefix is the wrong key.** `entity_relationship.fromFQNHash/toFQNHash` is populated on only a handful of `CONTAINS` code paths; bulk-ingestion (`bulkInsertTo`) and every legacy/lineage/ownership/domain `addRelationship` overload write **NULL**. NULL never matches `= :hash` or `LIKE :hash.%`, so those rows survive. A one-time backfill cannot fix rows created *after* it. Hashes also go stale on rename and are blind to flat-FQN hierarchies (sub-teams).
- **The entity id-set is the right key.** `entity_relationship.fromId`/`toId` (and the `id` column of every entity-keyed table) are **always populated** and **stable across renames**. Deleting `WHERE fromId IN (subtree) OR toId IN (subtree)` catches every edge touching the subtree, whatever its type or hash state.
**Rule:** delete by **id-set** wherever a table stores entity ids; delete by **bounded fqnHash prefix** (`hash + "." + %`, plus exact-match for the root) only for satellite tables that are *keyed by FQN hash* and have no id column.
| Table | Key column(s) | Deletion strategy |
|---|---|---|
| `<type>_entity` | `id` | id-set, chunked |
| `entity_relationship` | `fromId`, `toId` | **id-set** via `batchDeleteFrom`+`batchDeleteTo` per type (NULL-immune) |
| `entity_extension` | `id` | id-set, chunked |
| `entity_usage` | `id` | id-set, chunked |
| `thread_entity` (feed) | `entityId` (about) | id-set via `findByEntityIds` → delete threads |
| `field_relationship` | `fromFQNHash`, `toFQNHash` | bounded fqnHash prefix (already `.`-anchored) |
| `tag_usage` | `targetFQNHash` **and** `tagFQNHash` (source) | target by prefix; **source** by `deleteTagLabelsByFqn` per deleted tag/term |
| `*_time_series` (profiler, test results, query cost, etc.) | `entityFQNHash` | bounded fqnHash prefix |
| search index (ES/OS) | doc `fullyQualifiedName` | bounded prefix delete-by-query + exact root + reverse-reference scrub |
| RDF triple store | entity IRI | bulk SPARQL delete by subtree |
Existing primitives we reuse: `RelationshipDAO.batchDeleteRelationships(ids, type)` / `batchDeleteFrom` / `batchDeleteTo` (`CollectionDAO.java:2409-2433`, chunked), `EntityTimeSeriesDAO.deleteByFqnHashPrefix`, `FieldRelationshipDAO.deleteAllByPrefix`, `FeedDAO.findByEntityIds`.
## Architecture
Two cooperating pieces, both backed by the existing `entity_deletion_lock` table (used as the durable job record):
```
DELETE /services/.../prefix/{id}
│ (synchronous, O(1))
▼
┌─────────────────────────┐ ┌──────────────────────────────┐
│ 1. Acquire tombstone │ │ BulkDeletionExecutor │
│ (DELETE_IN_PROGRESS │ │ (async, resumable) │
│ lock on root FQN) │──────▶ │ - collect id-set (cursor) │
│ 2. Persist job record │ │ - per-chunk TXN: deps+rows │
│ 3. Return 202 + jobId │ │ - bulk hooks per type │
└─────────────────────────┘ │ - search/RDF/events │
│ │ - update cursor in lock row │
▼ │ - release lock on success │
create/update path └──────────────────────────────┘
checks tombstone → 409 ▲
│ │ resume on restart
└──────────────────────────────────────┘ (StaleLockReaper / boot)
```
### 1. The tombstone closes the race (synchronously, before any work)
On request, **before** collecting any ids, acquire a `DELETE_IN_PROGRESS` lock on the root FQN (`HierarchicalLockManager.acquireDeletionLock`, which already writes `entity_deletion_lock`). This is the gate; deletion proceeds only if the lock is acquired (no best-effort “continue without lock”).
The create/update/bulk-upsert paths already *call* `checkModificationAllowed(...)`. The fix is to make that check actually consult the DB:
- **Wire the gate.** Route `checkModificationAllowed` through the already-correct, **currently-callerless** `checkModificationAllowedByFqn(fqn)` (`HierarchicalLockManager.java:169`), which runs `findParentLocks` (`entityFqn = :fqn OR :fqn LIKE entityFqn || '.%'`). For the bulk path, batch it. Replace the dead `loadLockedFqnPrefixes()` stub (returns `new HashSet<>()`, `:312`) with a real `DeletionLockDAO` query, cached in the existing Caffeine cache (~30s TTL, invalidated on acquire/release) for the hot ingestion path.
Result: any insert/upsert under a deleting prefix is rejected with `EntityLockedException` → the snapshot-then-orphan race is closed.
### 2. The executor: collect → chunked-transactional purge → finalize
```
bulkDelete(root):
job = lock row for root (DELETE_IN_PROGRESS), with progress cursor in metadata
# Phase A — collect (read-only, resumable). FQN-hash prefix is fine HERE:
# we only use it to FIND descendants; we DELETE by id.
idsByType = {}
for type in fqnHashKeyedTypes(): # skip flat-FQN types
ids = dao(type).findIdsByFqnHashPrefix(hash(root.fqn)) # + the root id
if ids: idsByType[type] = ids
totalIds = flatten(idsByType) + root.id
# Phase B — purge in chunks; EACH CHUNK IS ONE TRANSACTION.
for chunk in partition(totalIds, CHUNK=25_000): # tune per engine
inTransaction:
# satellite tables keyed by FQN hash for entities in this chunk
# (or once up-front, see "scaling" note)
relationshipDAO.batchDeleteFrom(chunk, type) / batchDeleteTo(chunk, type) # id-set
entityExtensionDAO.deleteBatch(chunk)
usageDAO.deleteBatch(chunk)
feedRepository.deleteByAboutBatch(chunk)
dao(type).deleteBatch(chunk) # entity rows — CHUNKED (≤ 50k)
job.cursor = advance(chunk); persist(job) # durable progress
# FQN-hash-keyed satellites (bounded prefix, one pass — no id chunking needed)
fieldRelationshipDAO.deleteAllByPrefix(root.fqn)
for type: timeSeriesDAO(type).deleteByFqnHashPrefix(hash(root.fqn))
tagUsageDAO.deleteTagLabelsByTargetPrefix(root.fqn) # target side
# source side: for each deleted tag/glossaryTerm id → deleteTagLabelsByFqn(fqn)
# Phase C — per-type bulk hooks (replaces dropped cleanup()/preDelete/postDelete)
for type: repo(type).bulkCleanup(idsByType[type]) # see "Per-type side-effects"
# Phase D — finalize
searchRepo.deleteSubtree(root) # anchored prefix + root + reverse scrub
rdf.deleteSubtree(root) # bulk SPARQL
changeEventDAO.insert(ENTITY_DELETED for root) # + summary count
invalidateCache(ALL deleted ids) # not just root
releaseDeletionLock(root)
websocket: COMPLETED (or FAILED with detail on any aggregated error)
```
Key properties:
- **Atomicity at chunk granularity.** Dependency rows and entity rows **for the same id chunk** commit together. A crash leaves a clean *prefix* of fully-deleted chunks; never a live entity with destroyed dependencies. (Contrast: the prefix PR deletes *all* dependency tables first and *then* entity rows, with no transaction — a crash in between corrupts the whole subtree.)
- **Resumability.** The cursor lives in the lock row (`entity_deletion_lock.metadata` JSON). On restart, a reaper (or boot scan) finds `DELETE_IN_PROGRESS` locks and **resumes from the cursor**. Re-running a chunk is idempotent (`DELETE … WHERE id IN` of already-gone ids is a no-op).
- **No swallowed fatal errors.** Per-chunk failures are retried with backoff; unrecoverable failures mark the job `FAILED`, send the FAILED websocket/notification, and leave the lock for the reaper — the API never reports “completed” on partial failure.
## Per-Type Side-Effects (the dropped `cleanup()` work)
Raw `deleteBatch` skips `entitySpecificCleanup`, `preDelete` guards, and `postDelete`. Introduce one bulk hook on `EntityRepository`:
```java
protected void bulkCleanup(List<UUID> ids) { /* default no-op */ }
```
Overrides (mirroring today's per-entity logic, but set-based):
- **TestCase / TestSuite:** delete `data_quality_data_time_series` results.
- **IngestionPipeline / Workflow / WorkflowDefinition:** delete external pipelines/secrets — **batch** the external calls; do not `find(id)` per entity in a loop (the prefix PR re-introduced N+1 here).
- **Pipeline / StoredProcedure:** `deleteLineageBySourcePipeline` (lineage edges keyed by JSON `$.pipeline.id`, which an id-IN on `fromId/toId` will **not** catch).
- **Team:** sub-team reparenting / membership; **Role/Team:** `PolicyConditionUpdater` SpEL cleanup.
- **Tag / DataProduct:** the `IN_REVIEW` reviewer guard must be **checked**, not bypassed (governance). Also source-side `tag_usage` (`deleteTagLabelsByFqn`).
- **Base `postDelete`:** RDF removal via a **bulk** SPARQL delete for the subtree (not a per-entity loop).
`preDelete` system-protection guards (system policies/roles, the `organization` team) must run against the **root** and be enforced before Phase B.
## Endpoint Scope
Expose the bulk path **only** on hierarchical, FQN-nesting roots: `DatabaseService`, `Database`, `DatabaseSchema`, and the analogous `*Service`/container roots (dashboard, pipeline, messaging, mlmodel, storage, search, api, drive). **Do not** expose it on flat-FQN, `nameHash`-keyed types (`Team`, `User`, `Role`, `Persona`, `TestDefinition`, `Tag`, `Glossary`, `Domain`, `Policy`): for those `findIdsByFqnHashPrefix` returns `List.of()`, so a bulk delete degenerates to a raw root delete that skips required cleanup — strictly worse than the existing recursive path. Those types keep the existing `cleanup()` path.
## Search Index Strategy
The catalog (DB) is the source of truth; search must converge without divergence:
1. **Anchor the prefix.** Delete-by-query uses `fullyQualifiedName` prefixed with `root.fqn + Entity.SEPARATOR` (the `.`) so deleting `prod` does **not** wipe `prod_backup`. Delete the root doc separately by exact FQN/id. (The prefix PR passed the raw, unanchored `rootFqn`.)
2. **Reverse-reference scrub.** Run `deleteOrUpdateChildren`-equivalent for every deleted entity that may be *referenced inside other docs* (tags/terms/domain/dataProduct/owners/lineage on surviving assets), not just the root.
3. **Child docs.** Column-level / field-level docs under the deleted entities are covered by the same anchored prefix.
4. **Ordering & idempotency.** Search delete runs in Phase D after DB purge; it is idempotent and safe to re-run on resume.
## Change Events, Audit, Governance
The bulk path must not silently skip the eventing pipeline:
- Emit a real `ENTITY_DELETED` `change_event` for the **root** (so `EventSubscription` alerts, the audit log, and governance delete-workflows fire), carrying a **summary** (descendant counts by type, jobId).
- For very large subtrees, do **not** emit one event per descendant (that re-creates the millions-of-events cost). The root “subtree deleted” event + counts is the contract; document this as an intentional change from per-entity events. Consumers that need per-entity granularity subscribe to the job-summary payload.
## Relationship to PR #28675 (one-transaction write path) — build on it, don't reinvent
PR #28675 ("perf: one-transaction flush + async indexing write path", merged to `main` 2026-06-05) is **not** about deletion, but it landed exactly the infrastructure this design needs. **This deletion branch was last synced with `main` on 2026-04-13, so it does not yet contain #28675** — step 0 of any implementation is to rebase/merge `main`. The pieces and how the deletion path reuses each:
| #28675 primitive (location) | What it does for writes | How bulk deletion reuses it |
|---|---|---|
| `EntityRepository.flushInOneTransaction(Runnable)` (`EntityRepository.java:5029`) + `DeadlockRetry.execute(Supplier)` (`jdbi3/DeadlockRetry.java`) | Wraps the create/update/patch flush in `DeadlockRetry.execute(() -> jdbi.inTransaction(...))` — retry OUTER (fresh handle per replay), `inTransaction` INNER; collapses 5–7 commits → 1, atomic + deadlock-replay-safe | **The per-chunk atomic boundary in Phase B.** Each id-chunk purge runs inside `flushInOneTransaction` instead of a hand-rolled transaction. `DeadlockRetry` replays the whole chunk body — and `DELETE … WHERE id IN (…)` is idempotent, so replay is safe. Do not invent a new retry/transaction wrapper. |
| Deferred external-side-effect collectors: `DEFERRED_CACHE_INVALIDATIONS` + `beginCacheInvalidationDeferral`/`drainCacheInvalidations`; `RdfTagUpdater.beginDeferral`; `LineageUtil.drainLineageDeferred`; `SearchRepository.beginSearchWriteDeferral`/`drainSearchWriteDeferred` | Captures cache-L2 invalidation, RDF/SPARQL, lineage-ES and search writes *during* the flush and drains them **post-commit**, so the held DB connection makes **zero network round trips** | **Solves "no I/O while holding the delete transaction."** Inside each chunk transaction, only DB deletes run; record a cache invalidation for **every deleted descendant id** (fixes the only-root-invalidated bug) into `DEFERRED_CACHE_INVALIDATIONS`, and defer search/RDF deletes. Drain per chunk on the deletion worker thread (the "request thread" in #28675 just means "the thread that opened the scope"). |
| `EntityLifecycleEventDispatcher.onEntityDeleted` → `OrderedLaneExecutor` (per-entity-id lanes) → `SearchIndexHandler.onEntityDeleted` → `searchRepository.deleteEntityIndex`, failures → **`SearchIndexRetryQueue`** durable outbox | Entity-delete search/RDF/lineage propagation is already **async + per-entity-ordered + durable** | **Replaces the prefix PR's synchronous, best-effort `cleanSearchIndex`** (which silently diverges on ES failure). Route search/RDF cleanup through this hub so it inherits durable retry. |
| `SearchRepository.deleteEntityByFQNPrefix(EntityInterface)` (`SearchRepository.java:2573`) + `SearchIndexRetryQueue.failureReason(...)` | Prefix delete-by-query for search docs, with durable retry on failure (already used for `Entity.PAGE`) | The subtree search cleanup. **Note the signature differs** from this branch's `deleteByEntityTypeFqnPrefix(type, fqn)` — resolve the merge in favor of `main`'s durable variant; anchor with `Entity.SEPARATOR` (see Search Index Strategy). |
| Consistency contract: GET-by-id/name real-time (DB + **synchronous** cache write-through post-commit); `/search`, RDF, lineage **eventually consistent** with durable retry | — | **Adopt verbatim for delete.** The tombstone is the delete-time analog of #28675's synchronous cache write-through: it makes the subtree invisible/locked at the DB+cache layer *immediately*, while search/RDF converge asynchronously and durably. No new contract to invent. |
### The one adaptation that matters: granularity
#28675's dispatcher is **per-entity** — one `OrderedLaneExecutor` lane task per entity id (correct for writes, where per-entity ordering prevents a stale create clobbering a newer update). A bulk delete must **not** fan out `onEntityDeleted` across 1M descendants — that re-creates the millions-of-tasks cost this redesign exists to kill. Instead:
- Enqueue **one** subtree search delete-by-query (`deleteEntityByFQNPrefix` on the anchored root) as a single durable task after the DB purge — not N per-entity deletes.
- Emit **one** root `ENTITY_DELETED` summary event through the dispatcher (descendant counts by type), not one per descendant.
- Per-entity lane ordering is unnecessary here: the subtree is tombstone-locked, so no concurrent index-write for a deleted id can be in flight to race the delete.
Net: deletion reuses #28675's transaction wrapper, deferred-collector discipline, durable search outbox, and consistency contract, but operates at **subtree granularity** on the propagation side.
## Scaling Analysis
Let **N** = descendant count, **T** = number of entity types present, **C** = chunk size.
- **Statements:** collection = `T` SELECTs; purge = `~ceil(N/C) × (tables-per-chunk)`; satellites = `O(T)` prefix deletes. For N=1M, C=25k → ~40 chunks → low **hundreds** of statements total. (Baseline: ~N transactions.)
- **Transactions:** `~ceil(N/C)` (≈40) vs **~N** (≈1M). This is the headline win.
- **Memory:** the id-set is materialized once. 1M UUIDs ≈ tens of MB as `UUID`/`String`. Acceptable, but **stream the collection cursor-style** (page by `fqnHash` ranges) for >2–3M to bound heap; the cursor already supports paging.
- **IN-list limits:** entity-row and id-keyed deletes **must be chunked ≤ ~50k** to stay under PostgreSQL's 65,535 bind-parameter limit and MySQL `max_allowed_packet`. (The prefix PR left `EntityDAO.deleteBatch` un-chunked — a deterministic failure at the exact scale it targets.)
- **Index usage:** id-set deletes on `entity_relationship` use the existing `from_index (fromId,relation)` / `to_index (toId,relation)` on **both** engines — no dependency on the new fqnHash LIKE indexes (which on default-locale PostgreSQL require `varchar_pattern_ops` to serve `LIKE 'prefix%'` at all). Keep id-set off the hottest table's LIKE path entirely.
- **Lock/bloat:** per-chunk transactions keep lock duration and WAL/undo growth bounded; a single giant transaction over 1M rows would bloat and risk lock timeouts.
- **Collection note:** for FQN-hash-keyed satellites we can either delete per-chunk by FQN (more queries) or once up-front by bounded prefix (fewer queries, but those rows are deleted before the corresponding entity rows). Prefer the **one-pass bounded-prefix** delete *inside the final chunk's transaction window* or as an explicitly-resumable Phase, since these tables have no cross-entity integrity that a mid-run crash could corrupt beyond what the cursor already protects.
## Failure Handling & Resumability
| Failure | Behavior |
|---|---|
| Chunk SQL error (timeout, deadlock) | retry chunk w/ backoff; chunk TXN rolled back, cursor not advanced |
| Unrecoverable chunk error | job → `FAILED`, FAILED notification, lock retained for reaper/operator |
| JVM restart mid-job | `DELETE_IN_PROGRESS` lock + cursor survive; **StaleLockReaper** (Quartz) resumes from cursor |
| Abandoned/stale lock | reaper (wire the existing, **uncalled** `cleanupStaleLocks()` / `STALE_LOCK_CHECK_INTERVAL_MINUTES=5`) resumes or, past TTL, force-releases |
| Concurrent second delete of same root | rejected — lock already held |
| Concurrent ingestion under root | rejected with `EntityLockedException` (gate wired) |
## Observability
- Job record (`entity_deletion_lock` + metadata): `phase`, `cursor`, `deletedByType`, `startedAt`, `lastHeartbeat`, `error`.
- Metrics: rows deleted per table, chunk latency, total duration, retries; expose via `getLockStatistics()` and a status endpoint (`GET …/prefix/{jobId}/status`).
- Websocket progress events (already used by the prefix PR) reporting `% complete` from the cursor.
## Migration & Rollout
0. **Phase 0a (rebase):** merge/rebase `main` into the branch to pick up **PR #28675** (`flushInOneTransaction`, `DeadlockRetry`, deferred collectors, `OrderedLaneExecutor`, `SearchIndexRetryQueue`, `deleteEntityByFQNPrefix`). Resolve the heavily-overlapping `EntityRepository` changes and the `SearchRepository` prefix-delete signature in favor of `main`'s durable variants. Everything below builds on this.
1. **Phase 0b (correctness now / quick wins):**
- Add id-set relationship sweep (`batchDeleteRelationships(ids, type)`) to the purge — closes the entire NULL-hash orphan class with an existing primitive.
- Chunk `EntityDAO.deleteBatch` at 50k.
- Wrap each id-chunk purge in `flushInOneTransaction` (from #28675); stop reporting “completed” on partial failure.
- Anchor the search prefix with `.`; restrict the endpoint to hierarchical roots only.
2. **Phase 1 (race):** wire `checkModificationAllowed` → real DB gate (`checkModificationAllowedByFqn` / `loadLockedFqnPrefixes`), with Caffeine caching; tombstone before collection; acquire-or-abort (no best-effort).
3. **Phase 2 (durability):** cursor in lock metadata + `StaleLockReaper` (Quartz) for resume; status endpoint.
4. **Phase 3 (fidelity):** `bulkCleanup` overrides; route search/RDF cleanup through `EntityLifecycleEventDispatcher` + `SearchIndexRetryQueue` (one subtree task, not per-entity); record deferred cache invalidations for all deleted ids; reverse-reference search scrub; root `change_event` with summary.
5. **Deprecate / keep** the `fromFQNHash`/`toFQNHash` columns: they are **no longer required** for deletion (id-set replaces them). Decide separately whether to keep them for other features; if dropped, revert the v1.13.0 backfill (which is itself a perf risk — see Appendix A).
## Testing Strategy
Integration tests that **fail without the fix**:
1. **NULL-hash orphan:** bulk-ingest a service via the batch import path, prefix-delete it, assert **zero** surviving `entity_relationship` rows referencing any deleted id (proves id-set ≠ fqnHash).
2. **Cross-cutting edges:** add lineage/ownership/domain/follows from *outside* the subtree into it; after delete, assert all such edges are gone and the *external* entities survive.
3. **Crash atomicity:** inject a failure mid-purge; assert the subtree is either fully intact or fully gone — never a live entity with missing dependencies — and that resume completes it.
4. **Postgres scale:** a service with >65,535 descendants of one type deletes successfully (no bind-param failure).
5. **Concurrency:** insert-during-delete under the root is rejected; insert outside the root succeeds.
6. **Search convergence:** sibling docs (`prod_backup`) survive; subtree docs and reverse references are gone.
7. **Governance:** an `IN_REVIEW` tag inside the subtree blocks/honors the reviewer guard rather than being silently bypassed.
8. **Benchmark:** assert wall-clock for a seeded large service is within target (minutes), tracked over time.
## Alternatives Considered
- **A. FQN-hash prefix delete (the current PR).** Rejected as the primary mechanism: NULL-hash blindness re-creates orphans, stale on rename, blind to flat hierarchies, sibling collisions in search. The prefix is still useful for *collecting* descendants and for FQN-keyed satellite tables.
- **B. FK `ON DELETE CASCADE`.** Rejected: polymorphic `(fromId, fromEntity)` references prevent DB-level cascade; FK-free schema by design; huge migration to add/validate FKs on existing data.
- **C. Soft-tombstone + scheduled GC purge.** Strong for perceived latency (API is O(1), GC purges later) and folds naturally into this design’s tombstone. Adopted as an *option*: the tombstone already makes the subtree invisible/locked; whether the purge runs immediately (this doc’s default) or via a GC app is a deployment choice. The id-set purge mechanics are identical either way.
## Appendix A — Migration perf risk in the current PR
`MigrationUtil.backfillRelationshipFqnHashes` (v1.13.0) runs a correlated-subquery `UPDATE … SET fromFQNHash = (SELECT CAST(t.<hashcol> AS CHAR(768)) FROM <table> t WHERE CAST(t.id AS CHAR(36)) = entity_relationship.fromId)` once per entity type per direction (~120 statements). The `CAST(t.id AS CHAR(36))` **defeats the primary-key index** on the entity table, risking a full scan per ER row on instances with tens of millions of relationships. Since the redesign deletes by id-set, this backfill (and the columns) can be dropped. If retained, rewrite as an indexed join without casts.
## Appendix B — The race the current PR does not close
`createInternal`/`createOrUpdate`/bulk-upsert *do* call `lockManager.checkModificationAllowed(...)`, but it short-circuits on `isFqnLocked(fqn)` → `loadLockedFqnPrefixes()`, which is a stub returning `new HashSet<>()` (`HierarchicalLockManager.java:312-316`). So no FQN is ever considered locked and ingestion is never blocked. The correct, cache-free gate `checkModificationAllowedByFqn` (`:169`) exists but has **zero callers**. Wiring this (Phase 1) is the single change that makes the lock actually prevent concurrent-ingestion orphans.
docs/rdf-local-development.md
+357
View File
@@ -0,0 +1,357 @@
# RDF/Apache Jena Local Development Guide
This guide documents how to set up RDF/Knowledge Graph support for local development with OpenMetadata and Apache Jena Fuseki.
## Overview
OpenMetadata supports RDF (Resource Description Framework) for knowledge graph capabilities using Apache Jena Fuseki as the triple store. This enables:
- SPARQL queries against metadata
- JSON-LD serialization of entities
- Semantic search and graph exploration
## Architecture
```
┌─────────────────────┐ ┌─────────────────────┐
│ OpenMetadata │ │ Apache Jena │
│ Server (IntelliJ) │────▶│ Fuseki (Docker) │
│ Port: 8585 │ │ Port: 3030 │
└─────────────────────┘ └─────────────────────┘
```
## Prerequisites
- Docker and Docker Compose installed
- IntelliJ IDEA with the project imported
- MySQL or PostgreSQL running (for OpenMetadata backend)
- Elasticsearch running (for search)
## Quick Start
### Step 1: Choose the Right Startup Mode
The standard local Docker flow does not enable RDF or start Fuseki:
```bash
cd /path/to/OpenMetadata
./docker/run_local_docker.sh -d mysql
```
For PostgreSQL-based development:
```bash
./docker/run_local_docker.sh -d postgresql
```
Use the RDF-specific startup script when you want the full Docker stack with Fuseki enabled:
```bash
./docker/run_local_docker_rdf.sh -d mysql
```
For PostgreSQL-based RDF development:
```bash
./docker/run_local_docker_rdf.sh -d postgresql
```
This RDF startup path starts OpenMetadata, the backing database, search, ingestion services, and Fuseki with:
- **Port**: 3030
- **Admin Password**: admin
- **Dataset**: openmetadata
- **Memory**: 2-4GB allocated
### Step 2: Verify Fuseki is Running
```bash
# Check Fuseki health
curl -s http://localhost:3030/$/ping
# Access Fuseki UI in browser
open http://localhost:3030
```
The Fuseki web UI is available at `http://localhost:3030` with credentials:
- Username: `admin`
- Password: `admin`
### Step 3: Configure IntelliJ Run Configuration
If you are running the full RDF Docker stack with `run_local_docker_rdf.sh`, the Docker services already receive the RDF environment variables automatically.
If you want to run the OpenMetadata server directly from IntelliJ while keeping Fuseki in Docker, start Fuseki separately:
```bash
docker compose -f docker/development/docker-compose.yml -f docker/development/docker-compose-fuseki.yml up -d fuseki
```
If your local backend uses PostgreSQL, swap `docker-compose.yml` for `docker-compose-postgres.yml`.
Create or modify your IntelliJ run configuration for `OpenMetadataApplication` with these environment variables only when you want to run the OpenMetadata server directly from IntelliJ while keeping Fuseki in Docker:
```
RDF_ENABLED=true
RDF_STORAGE_TYPE=FUSEKI
RDF_BASE_URI=https://open-metadata.org/
RDF_ENDPOINT=http://localhost:3030/openmetadata
RDF_REMOTE_USERNAME=admin
RDF_REMOTE_PASSWORD=admin
RDF_DATASET=openmetadata
```
#### Setting Environment Variables in IntelliJ:
1. Open **Run** → **Edit Configurations**
2. Select your `OpenMetadataApplication` configuration
3. Click on **Modify options** → **Environment variables**
4. Add the environment variables above (semicolon-separated or using the dialog)
Example environment variables string:
```
RDF_ENABLED=true;RDF_STORAGE_TYPE=FUSEKI;RDF_BASE_URI=https://open-metadata.org/;RDF_ENDPOINT=http://localhost:3030/openmetadata;RDF_REMOTE_USERNAME=admin;RDF_REMOTE_PASSWORD=admin;RDF_DATASET=openmetadata
```
### Step 4: Start OpenMetadata Server
Run `OpenMetadataApplication` from IntelliJ. On startup, you should see in the logs:
```
INFO [main] o.o.s.OpenMetadataApplication - RDF knowledge graph support initialized
```
### Step 5: Verify RDF is Enabled
```bash
# Check RDF status
curl http://localhost:8585/api/v1/rdf/status
# Expected response:
# {"enabled": true}
```
## Configuration Reference
### Server Configuration (conf/openmetadata.yaml)
The RDF configuration section in `openmetadata.yaml`:
```yaml
rdf:
enabled: ${RDF_ENABLED:-false}
baseUri: ${RDF_BASE_URI:-"https://open-metadata.org/"}
storageType: ${RDF_STORAGE_TYPE:-"FUSEKI"}
remoteEndpoint: ${RDF_ENDPOINT:-"http://localhost:3030/openmetadata"}
username: ${RDF_REMOTE_USERNAME:-"admin"}
password: ${RDF_REMOTE_PASSWORD:-"admin"}
dataset: ${RDF_DATASET:-"openmetadata"}
```
### Environment Variables
| Variable | Description | Default |
|----------|-------------|---------|
| `RDF_ENABLED` | Enable/disable RDF support | `false` |
| `RDF_STORAGE_TYPE` | Storage backend type | `FUSEKI` |
| `RDF_BASE_URI` | Base URI for RDF resources | `https://open-metadata.org/` |
| `RDF_ENDPOINT` | Fuseki SPARQL endpoint URL | `http://localhost:3030/openmetadata` |
| `RDF_REMOTE_USERNAME` | Fuseki admin username | `admin` |
| `RDF_REMOTE_PASSWORD` | Fuseki admin password | `admin` |
| `RDF_DATASET` | Fuseki dataset name | `openmetadata` |
### Docker Compose Configuration
The Fuseki container (`docker/development/docker-compose-fuseki.yml`):
```yaml
services:
fuseki:
image: stain/jena-fuseki:5.0.0
container_name: openmetadata-fuseki
ports:
- "3030:3030"
environment:
- ADMIN_PASSWORD=admin
- JVM_ARGS=-Xmx4g -Xms2g
- FUSEKI_BASE=/fuseki
volumes:
- fuseki-data:/fuseki
```
## API Endpoints
Once RDF is enabled, these endpoints are available:
### Check RDF Status
```bash
GET /api/v1/rdf/status
```
### Get Entity as RDF
```bash
# Get entity in JSON-LD format (default)
GET /api/v1/rdf/entity/{entityType}/{id}
# Get entity in Turtle format
GET /api/v1/rdf/entity/{entityType}/{id}?format=turtle
# Get entity in RDF/XML format
GET /api/v1/rdf/entity/{entityType}/{id}?format=rdfxml
# Get entity in N-Triples format
GET /api/v1/rdf/entity/{entityType}/{id}?format=ntriples
```
### Execute SPARQL Query
```bash
POST /api/v1/rdf/sparql
Content-Type: application/json
{
"query": "SELECT ?s ?p ?o WHERE { ?s ?p ?o } LIMIT 10"
}
```
### Get Glossary Term Relationship Graph
```bash
# Get the full glossary term graph
GET /api/v1/rdf/glossary/graph
# Filter primary terms to a glossary
GET /api/v1/rdf/glossary/graph?glossaryId=<glossary-id>
# Filter to a glossary term and its direct incoming/outgoing neighbors
GET /api/v1/rdf/glossary/graph?glossaryTermId=<glossary-term-id>
# Require the selected term to belong to a glossary, while still returning
# direct cross-glossary neighbors when relationships cross glossary boundaries
GET /api/v1/rdf/glossary/graph?glossaryId=<glossary-id>&glossaryTermId=<glossary-term-id>
```
Optional query parameters:
| Parameter | Description |
|-----------|-------------|
| `glossaryId` | Filter primary terms to a glossary. |
| `glossaryTermId` | Filter to a selected glossary term and its direct incoming/outgoing glossary-term relations. |
| `relationTypes` | Comma-separated relation types to include. |
| `limit` | Maximum number of terms to return. Default: `500`. |
| `offset` | Pagination offset. Default: `0`. |
| `includeIsolated` | Include terms without relations. Default: `true`. |
### Example Queries
```bash
# Check if RDF is enabled
curl -s http://localhost:8585/api/v1/rdf/status | jq
# Get a table entity as JSON-LD
curl -s -H "Authorization: Bearer <token>" \
"http://localhost:8585/api/v1/rdf/entity/table/<table-id>" | jq
# Execute a SPARQL query
curl -s -X POST \
-H "Authorization: Bearer <token>" \
-H "Content-Type: application/json" \
-d '{"query": "SELECT ?s ?p ?o WHERE { ?s ?p ?o } LIMIT 10"}' \
http://localhost:8585/api/v1/rdf/sparql | jq
# Get a selected glossary term graph
curl -s -H "Authorization: Bearer <token>" \
"http://localhost:8585/api/v1/rdf/glossary/graph?glossaryId=<glossary-id>&glossaryTermId=<glossary-term-id>" | jq
```
## Indexing Entities to RDF
### Manual Reindexing
Trigger the RDF indexing application to populate the triple store with existing entities:
```bash
curl -X POST \
-H "Authorization: Bearer <admin-token>" \
-H "Content-Type: application/json" \
-d '{"entities": [], "recreateIndex": true, "batchSize": 100}' \
http://localhost:8585/api/v1/apps/trigger/RdfIndexApp
```
### Automatic Indexing
When RDF is enabled, new entities are automatically indexed to the triple store on create/update/delete operations.
## Fuseki Web UI
The Fuseki web interface provides:
- **Dataset Management**: View and manage datasets at `http://localhost:3030/#/manage`
- **SPARQL Query Interface**: Execute queries at `http://localhost:3030/#/dataset/openmetadata/query`
- **Data Upload**: Upload RDF data at `http://localhost:3030/#/dataset/openmetadata/upload`
## Troubleshooting
### Fuseki Connection Issues
1. Verify Fuseki is running:
```bash
docker ps | grep fuseki
curl http://localhost:3030/$/ping
```
2. Check Fuseki logs:
```bash
docker logs openmetadata-fuseki
```
3. Ensure the dataset exists:
```bash
curl -u admin:admin http://localhost:3030/$/datasets
```
### RDF Not Enabled in Server
1. Verify environment variables are set correctly in IntelliJ
2. Check server logs for RDF initialization message
3. Confirm configuration in `openmetadata.yaml`
### SPARQL Query Errors
1. Check Fuseki is accessible from OpenMetadata server
2. Verify the dataset name matches (`openmetadata`)
3. Check Fuseki logs for query errors
### Reset Fuseki Data
To clear all RDF data and start fresh:
```bash
# Stop Fuseki
docker compose -f docker/development/docker-compose-fuseki.yml down
# Remove volume
docker volume rm openmetadata_fuseki-data
# Restart Fuseki
docker compose -f docker/development/docker-compose-fuseki.yml up -d
```
## Full Stack with Docker Script
For a complete local environment with RDF enabled (server running in Docker, not IntelliJ):
```bash
./docker/run_local_docker_rdf.sh -m ui -d mysql -f true
```
Options:
- `-m ui|no-ui` - Include UI or not
- `-d mysql|postgresql` - Database type
- `-f true|false` - Start Fuseki for RDF support
- `-s true|false` - Skip Maven build
- `-x true|false` - Enable JVM debug on port 5005
## Related Files
- **Docker Compose**: `docker/development/docker-compose-fuseki.yml`
- **Server Config**: `conf/openmetadata.yaml`
- **RDF Java Code**: `openmetadata-service/src/main/java/org/openmetadata/service/rdf/`
- **Ontology**: `openmetadata-spec/src/main/resources/rdf/ontology/openmetadata.ttl`
- **RDF Index App**: `openmetadata-service/src/main/java/org/openmetadata/service/apps/bundles/rdf/RdfIndexApp.java`
docs/session-management-multi-node-design.md
+518
View File
@@ -0,0 +1,518 @@
# Multi-Node Session and WebSocket Session Management Design
## 1. Status
This document describes the current server-side session and websocket session design for
OpenMetadata issue [#21971](https://github.com/open-metadata/OpenMetadata/issues/21971).
The implementation is centered on these files:
- `openmetadata-service/src/main/java/org/openmetadata/service/security/session/SessionService.java`
- `openmetadata-service/src/main/java/org/openmetadata/service/security/session/SessionStore.java`
- `openmetadata-service/src/main/java/org/openmetadata/service/security/session/JdbcSessionStore.java`
- `openmetadata-service/src/main/java/org/openmetadata/service/security/session/RedisSessionStore.java`
- `openmetadata-service/src/main/java/org/openmetadata/service/security/session/SessionStoreFactory.java`
- `openmetadata-service/src/main/java/org/openmetadata/service/security/JwtFilter.java`
- `openmetadata-service/src/main/java/org/openmetadata/service/security/jwt/JWTTokenGenerator.java`
- `openmetadata-service/src/main/java/org/openmetadata/service/socket/SocketAddressFilter.java`
- `openmetadata-service/src/main/java/org/openmetadata/service/socket/WebSocketManager.java`
- `openmetadata-service/src/main/java/org/openmetadata/service/OpenMetadataApplication.java`
- `openmetadata-service/src/main/java/org/openmetadata/service/cache/CacheBundle.java`
## 2. Problem
Server-managed login state used to depend on pod-local servlet state. That breaks in a multi-node
deployment because login, callback, refresh, logout, and websocket reconnects can land on different
pods.
The key failure modes are:
- A login or OIDC/SAML callback starts on one node and completes on another.
- Refresh state is unavailable when the request is routed to a different node.
- Logout or session revocation is not visible to all nodes.
- A websocket can remain connected after the browser session is revoked.
- A secure websocket handshake can be spoofed if the server trusts a client-supplied `userId`.
## 3. Goals
1. Store server-managed user sessions in a shared, authoritative backend.
2. Support both JDBC-backed sessions and Redis-backed sessions.
3. Bind newly issued browser access JWTs to the server-side session that issued them.
4. Keep provider refresh tokens and OpenMetadata refresh tokens server-side.
5. Make refresh safe under cross-node concurrency.
6. Make logout and revocation visible to API and websocket paths.
7. Reject websocket handshakes whose token principal, cookie session, or requested socket user do not match.
8. Close websocket connections for revoked or expired sessions.
## 4. Non-Goals
1. Changing personal access token or bot token semantics.
2. Replacing the browser-managed public OIDC flow.
3. Making legacy JWTs without a `sessionId` claim retroactively session-bound.
4. Guaranteeing instant cross-node websocket disconnect without pub/sub. Without pub/sub, remote
sockets are closed by periodic validation.
## 5. Architecture
### 5.1 Component Model
| Component | Responsibility |
| --- | --- |
| `SessionService` | Creates, activates, refreshes, revokes, expires, and prunes sessions. Owns the Caffeine near-cache and revocation listeners. |
| `SessionStore` | Shared persistence contract used by both JDBC and Redis stores. |
| `JdbcSessionStore` | Default store backed by the `user_session` table through `SessionRepository`. |
| `RedisSessionStore` | Optional Redis store with key TTLs, per-user status indexes, and Lua compare-and-set on `version` plus index maintenance. |
| `SessionStoreFactory` | Selects Redis when Redis cache is configured and available; otherwise uses JDBC. Refuses Redis-to-JDBC fallback when Redis is configured but unavailable. |
| `SessionCookieUtil` | Reads, writes, validates, and clears the opaque `OM_SESSION` cookie. |
| `JWTTokenGenerator` | Issues OpenMetadata JWTs and can include the `sessionId` claim for session-backed auth flows. |
| `JwtFilter` | Validates JWTs. If the JWT has `sessionId`, it reloads the session from the store and requires an active, unexpired, username-matching session. |
| `SocketAddressFilter` | Validates websocket handshake identity and session state before Socket.IO sees the connection. |
| `WebSocketManager` | Tracks sockets by user and by session, sends events, and disconnects revoked or inactive session sockets. |
| `OpenMetadataApplication` | Wires `SessionService` into auth handlers, websockets, revocation listeners, and the websocket session validator. |
| `CacheBundle` | Handles cache invalidation pub/sub. Session invalidation messages also disconnect sockets on remote pods. |
### 5.2 Storage Selection
`SessionStoreFactory` chooses the store at application startup:
- If `cache.provider = redis` and Redis is available, sessions use `RedisSessionStore`.
- If Redis is configured but unavailable, startup fails closed.
- If Redis is not configured, sessions use `JdbcSessionStore`.
The system does not fail over live from Redis to JDBC. Mixing stores would split active sessions
across backends and make revocation unpredictable.
### 5.3 Session Cache
Each pod keeps a local Caffeine near-cache:
- maximum size: `10_000`
- expire after access: `10s`
The cache is a performance optimization, not a correctness boundary. Security-sensitive checks use
fresh reloads where revocation must be observed immediately.
## 6. User Session Management
### 6.1 Session ID
`UserSession.id` is an opaque bearer secret carried in the `OM_SESSION` cookie.
It is generated by `SessionIdGenerator` from secure random bytes and base64url encoded without
padding. It is not a UUID.
### 6.2 Session Types
Current session types:
- `AUTH`: browser or interactive user auth session.
- `MCP`: reserved for future interactive MCP session support.
### 6.3 Session Status
`SessionStatus` values:
- `PENDING`: login started, callback not completed.
- `ACTIVE`: usable session.
- `REFRESHING`: one node holds the refresh lease.
- `REVOKED`: logout or session-limit revocation.
- `EXPIRED`: timeout reached.
### 6.4 Session Fields
The important logical fields are:
```json
{
"id": "opaque-session-id",
"type": "AUTH",
"provider": "openmetadata",
"status": "ACTIVE",
"userId": "uuid",
"username": "alice",
"email": "alice@example.com",
"omRefreshToken": "fernet:encrypted-token",
"providerRefreshToken": "fernet:encrypted-provider-token",
"redirectUri": "https://ui.example.com/callback",
"state": "oidc-state",
"nonce": "oidc-nonce",
"pkceVerifier": "pkce-verifier",
"version": 7,
"refreshLeaseUntil": 1741300000000,
"createdAt": 1741200000000,
"updatedAt": 1741200005000,
"lastAccessedAt": 1741200005000,
"expiresAt": 1743792000000,
"idleExpiresAt": 1741804800000
}
```
Refresh tokens are encrypted before persistence with `Fernet.encryptIfApplies(...)`. If the Fernet
key is not configured, session creation fails instead of writing plaintext refresh tokens.
### 6.5 Session Creation
Basic, LDAP, and OpenMetadata login create an `ACTIVE` session directly:
1. Validate credentials.
2. Resolve the provisioned OpenMetadata user.
3. Persist or receive the OpenMetadata refresh token.
4. Create an `ACTIVE` `AUTH` session.
5. Encrypt and store the refresh token in the session.
6. Write the `OM_SESSION` cookie.
7. Return an OpenMetadata-signed JWT with a `sessionId` claim.
If user lookup or session creation fails after a refresh token is created, the refresh token is
deleted.
### 6.6 Pending Session Activation
SAML and confidential OIDC use pending sessions:
1. Login creates a `PENDING` `AUTH` session containing redirect state, OIDC state, nonce, and PKCE
verifier when applicable.
2. The callback loads the pending session from the shared store.
3. The user is created or updated.
4. The OpenMetadata refresh token is inserted.
5. `activatePendingSession` expires the pending session.
6. A brand-new active session ID is generated and stored.
7. The active session cookie replaces the pending cookie.
8. The browser receives an OpenMetadata-signed JWT with the active session ID.
Issuing a new active session ID during activation is the session fixation defense. The pre-auth
cookie value is never reused for the authenticated session.
If activation fails, the newly inserted refresh token is deleted and no JWT is issued.
### 6.7 Refresh
Refresh is guarded by an optimistic lease:
1. Load the session from `OM_SESSION`.
2. Reject missing, expired, pending, revoked, or already expired sessions.
3. If another node holds a non-stale `REFRESHING` lease, return retry guidance through
`SessionRefreshInProgressException`.
4. Acquire the lease by writing `REFRESHING`, setting `refreshLeaseUntil`, and incrementing
`version` with compare-and-set.
5. The winning node decrypts the stored refresh token.
6. The provider or OpenMetadata refresh token is rotated as needed.
7. `completeRefresh` writes the refreshed session back to `ACTIVE`, clears the lease, updates idle
expiry without extending beyond the absolute session expiry, and increments `version`.
8. The response contains a new OpenMetadata-signed JWT bound to the same session ID.
Lease duration is currently `15s`.
### 6.8 Logout and Revocation
Logout calls `SessionService.revokeSession(request, response)`:
1. Read `OM_SESSION`.
2. Reload the session from the authoritative store.
3. Write `REVOKED` with compare-and-set.
4. Clear `refreshLeaseUntil`.
5. Clear the `OM_SESSION` cookie.
6. Notify local revocation listeners.
Session limit enforcement also uses `revokeSession` for least-recently-used active sessions.
The limit is configured by `authenticationConfiguration.maxActiveSessionsPerUser`, exposed through
`AUTHENTICATION_MAX_ACTIVE_SESSIONS_PER_USER` in `openmetadata.yaml`. The default is `5`; values
below `1` fall back to the default.
### 6.9 Expiration and Cleanup
`SessionService` runs cleanup every `15m`:
- mark expired sessions as `EXPIRED`
- prune `REVOKED` and `EXPIRED` rows after `7d`
- process in bounded batches
For Redis, primary keys have TTLs and cleanup methods are no-ops. Session correctness still relies
on in-process status and expiry checks.
Default timeouts:
- pending session timeout: `10m`
- authenticated session expiry: `authenticationConfiguration.sessionExpiry`, default `7d`
- refresh lease: `15s`
- cleanup retention: `7d`
The `OM_SESSION` cookie max age is rewritten during refresh lease acquisition and is capped at the
remaining effective session lifetime.
## 7. Session-Bound JWTs
Server-managed auth flows return OpenMetadata-signed access JWTs with:
```json
{
"sub": "alice",
"tokenType": "OM_USER",
"sessionId": "opaque-session-id"
}
```
`JwtFilter` handles the claim as follows:
1. Validate the JWT signature, expiry, token type, principal, and token-specific rules.
2. If there is no `sessionId` claim, preserve existing stateless behavior.
3. If `sessionId` exists, call `SessionService.getFreshSessionById(sessionId)`.
4. Require:
- session exists
- status is `ACTIVE`
- session is not expired
- session username matches the JWT principal
5. Reject the token when any check fails.
This means session-backed browser API requests now consult the shared session store. That is an
intentional tradeoff in the current implementation: revocation is observed on the next request
instead of waiting for access-token expiry. PATs, bot tokens, and legacy JWTs without `sessionId`
remain stateless.
## 8. WebSocket Session Management
### 8.1 Handshake Validation
`SocketAddressFilter` runs before the Socket.IO server receives the connection.
When secure websocket connections are enabled:
1. Extract and validate the `Authorization` header.
2. Resolve the token principal from JWT claims.
3. Resolve the principal's user UUID server-side.
4. Reject the request if the query `userId` is present and does not match the resolved user UUID.
5. Inject the server-resolved `UserId` header for `WebSocketManager`.
6. If the JWT has `sessionId`, inject a `SessionId` header.
7. Validate `OM_SESSION` when present:
- reload the session fresh
- require `ACTIVE`
- require not expired
- require session username to match the token principal
- require cookie session ID to match token `sessionId` when both are present
If no `OM_SESSION` cookie is present:
- session-bound JWTs are accepted because `JwtFilter` has already validated the session ID
against `SessionService`
- legacy secure JWTs without `sessionId` are rejected with `401 Session is required`
- non-secure websocket mode remains compatible with existing query-based behavior
The filter no longer forwards trust from the user-supplied `userId` query parameter when secure
mode is enabled.
### 8.2 Socket Tracking
`WebSocketManager` maintains two local maps per pod:
- `activityFeedEndpoints`: `userId -> socketId -> SocketIoSocket`
- `socketSessionIds`: `socketId -> sessionId`
On connection:
1. Read `UserId` from the injected header, falling back to query only for legacy/non-secure paths.
2. Read `SessionId` from the injected header, falling back to query `sessionId` only for legacy
paths.
3. Store the socket in the user's local socket map.
4. Store the socket-to-session mapping when a session ID is available.
On disconnect, both maps are cleaned up.
The connection log records user and remote address only. It does not log initial headers, so bearer
tokens are not written to logs.
### 8.3 Revocation-Driven Disconnect
`SessionService` exposes revocation listeners. `OpenMetadataApplication` registers a listener that:
1. Converts the revoked session's `userId` to UUID.
2. Calls `WebSocketManager.disconnectForSession(userId, sessionId)` on the local pod.
3. Publishes a `"session"` invalidation message through cache invalidation pub/sub when available.
`CacheBundle` handles remote `"session"` invalidation messages:
- if the message has a session ID, call `disconnectForSession(userId, sessionId)`
- if no session ID is present, fall back to `disconnectAllForUser(userId)` for backward
compatibility
This gives targeted disconnects. Logging out one browser session does not force-close other
sessions for the same user.
### 8.4 Periodic WebSocket Validation
`OpenMetadataApplication.WebSocketSessionValidator` runs every `60s` by default. Operators can tune
the interval with the `openmetadata.websocketSessionValidationIntervalSeconds` system property or
the `WEBSOCKET_SESSION_VALIDATION_INTERVAL_SECONDS` environment variable. Values below `15s` are
clamped to `15s`.
Each run calls `WebSocketManager.disconnectInactiveSessions(sessionService, intervalMillis)`, which:
1. Iterates local sockets with known `sessionId`.
2. Reloads a socket's session fresh through `SessionService.getFreshSessionById` only when that
socket's revalidation interval is due.
3. Disconnects sockets whose session is missing, not `ACTIVE`, expired, or owned by a different
user.
This is the fallback when there is no cross-pod pub/sub. With JDBC and no pub/sub, a socket on a
remote node is closed within the validator interval instead of immediately.
## 9. End-to-End Flow
```mermaid
flowchart TD
Browser["Browser / UI"]
NodeA["OpenMetadata Pod A"]
NodeB["OpenMetadata Pod B"]
AuthHandlers["Auth handlers<br/>Basic / LDAP / SAML / OIDC"]
SessionServiceA["SessionService A"]
SessionServiceB["SessionService B"]
Store[("Shared SessionStore<br/>JDBC user_session or Redis")]
JwtFilter["JwtFilter"]
SocketFilter["SocketAddressFilter"]
WebSocketManagerA["WebSocketManager A"]
WebSocketManagerB["WebSocketManager B"]
PubSub["Cache invalidation pub/sub<br/>optional"]
Validator["WebSocketSessionValidator<br/>default 60s, min 15s"]
Browser -->|"login / callback"| NodeA
NodeA --> AuthHandlers
AuthHandlers --> SessionServiceA
SessionServiceA -->|"create PENDING or ACTIVE<br/>activate pending<br/>encrypt refresh tokens"| Store
SessionServiceA -->|"Set-Cookie: OM_SESSION"| Browser
AuthHandlers -->|"JWT with sessionId"| Browser
Browser -->|"API request<br/>Bearer JWT(sessionId)"| NodeB
NodeB --> JwtFilter
JwtFilter --> SessionServiceB
SessionServiceB -->|"fresh session lookup"| Store
JwtFilter -->|"allow only ACTIVE + unexpired + matching username"| NodeB
Browser -->|"websocket handshake<br/>Authorization + OM_SESSION"| NodeB
NodeB --> SocketFilter
SocketFilter -->|"validate JWT principal<br/>resolve user UUID<br/>validate session"| SessionServiceB
SocketFilter -->|"inject UserId + SessionId"| WebSocketManagerB
WebSocketManagerB -->|"track user -> sockets<br/>track socket -> session"| WebSocketManagerB
Browser -->|"logout / revoke"| NodeA
NodeA --> SessionServiceA
SessionServiceA -->|"mark REVOKED"| Store
SessionServiceA -->|"local disconnectForSession"| WebSocketManagerA
SessionServiceA -->|"publish session invalidation"| PubSub
PubSub -->|"remote disconnectForSession"| WebSocketManagerB
Validator --> WebSocketManagerB
WebSocketManagerB -->|"fresh validate socket sessions"| SessionServiceB
SessionServiceB --> Store
WebSocketManagerB -->|"disconnect inactive sockets"| Browser
```
## 10. Consistency Model
### 10.1 API Requests
For tokens with `sessionId`, the session store is authoritative. A revoked or expired session is
rejected on the next API request that uses that token.
For tokens without `sessionId`, existing stateless behavior is preserved.
### 10.2 Refresh
Refresh uses optimistic compare-and-set on `version`, so only one node can hold the refresh lease
for a session at a time.
JDBC implements this through the session repository update path. Redis implements it with a Lua CAS
script over the stored session JSON. The Redis script also removes the session ID from all
non-terminal per-user status indexes and adds it to the target non-terminal index before returning,
so the JSON write and index movement succeed or fail together.
### 10.3 WebSockets
Websocket consistency has two layers:
- event-driven disconnect through local revocation listeners and optional cache invalidation pub/sub
- polling-based validation with a `60s` default interval and `15s` minimum
The event path is immediate when revocation occurs on the same pod or pub/sub delivers the remote
event. The polling path bounds staleness when pub/sub is unavailable, and each socket is fresh
loaded at most once per validation interval.
## 11. Operational Characteristics
| Path | Store behavior |
| --- | --- |
| Login | create active or pending session |
| OIDC/SAML callback | fresh load pending session, expire pending session, create active session |
| Session-bound API request | fresh load session by `sessionId` |
| Refresh | load session, acquire CAS lease, complete CAS update |
| Logout | fresh load session, CAS revoke, clear cookie |
| WebSocket handshake | validate JWT, optionally fresh load cookie session |
| WebSocket validator | throttled fresh load for each tracked socket with `sessionId` |
Redis deployments should monitor Redis availability as auth-critical infrastructure. When Redis is
configured for sessions, the service refuses to start without it.
## 12. Security Properties
1. `OM_SESSION` is opaque and high entropy.
2. `OM_SESSION` is written as an HTTP-only cookie.
3. Provider refresh tokens and OpenMetadata refresh tokens are encrypted at rest.
4. Refresh tokens are not returned to the browser by server-managed auth flows.
5. Pending-session activation issues a brand-new active session ID.
6. Session-bound JWTs are invalid once the backing session is revoked, expired, deleted, or owned by
a different user.
7. Secure websocket mode derives socket user identity from the JWT principal, not from query params.
8. Websocket logs do not include initial headers or bearer tokens.
9. Revocation targets the revoked session instead of disconnecting every socket for the user.
## 13. Test Coverage
Relevant unit coverage includes:
- `SessionServiceTest`
- `SessionCookieUtilTest`
- `SessionTimeoutResolverTest`
- `SessionStoreContractTest`
- `RedisSessionStoreTest`
- `JwtFilterTest`
- `BasicAuthServletHandlerTest`
- `LdapAuthServletHandlerTest`
- `SamlAuthServletHandlerTest`
- `AuthenticationCodeFlowHandlerTest`
- `SocketAddressFilterTest`
- `WebSocketManagerTest`
Relevant integration coverage includes:
- `SessionMultiNodeIT`
- `SessionRedisMultiNodeIT`
- `SessionMultiNodeCluster`
Important scenarios covered or expected from this suite:
- login on one node and refresh/logout on another
- pending OIDC/SAML callback state loaded from shared session storage
- refresh lease contention
- stale cache behavior after revocation
- Redis-backed cross-node sessions
- websocket principal binding
- per-session websocket disconnect
- session-bound JWT rejection for revoked sessions
## 14. Tradeoff Resolutions
1. Session-bound browser API requests intentionally reload session state on JWT validation. This is
the chosen correctness boundary: logout and revocation are observed on the next browser API
request instead of waiting for access-token expiry.
2. Tokens without `sessionId` remain on existing JWT semantics. This preserves PAT, bot, public OIDC,
and rolling-upgrade compatibility. New server-managed auth responses include `sessionId`.
3. Non-secure websocket mode remains query-param based only for backward compatibility. Production
deployments should keep secure websocket connections enabled so `SocketAddressFilter` derives the
socket user from the JWT principal and records the session ID.
4. The active-session cap is now configurable with
`authenticationConfiguration.maxActiveSessionsPerUser` and
`AUTHENTICATION_MAX_ACTIVE_SESSIONS_PER_USER`; the default remains `5`.
5. Secure, session-managed websocket handshakes now record a session ID from the JWT claim or
`OM_SESSION` cookie. The validator checks those sockets on a configurable interval with a `60s`
default and `15s` minimum; sockets without session IDs are legacy/non-secure compatibility cases.
6. Cross-pod websocket revocation has two paths: cache invalidation pub/sub for immediate targeted
disconnects when available, and the configurable websocket validator as the bounded-staleness
fallback for JDBC-only deployments.
docs/streamable-logs.md
+259
View File
@@ -0,0 +1,259 @@
# Streamable Ingestion Logs
This document describes the end-to-end design of OpenMetadata's streamable ingestion-pipeline log system: how logs flow from a running connector to durable S3 storage, how the UI reads them while a run is in progress, and how the system handles long idle gaps, restarts, and abandoned runs.
## Overview
Ingestion pipelines (metadata, profiler, lineage, usage, dbt, etc.) emit logs as they run. Operators need to:
- Watch logs **live** while a pipeline is running, including for long-running connectors that can take hours.
- Read logs **after the run ends**, with a single canonical artifact per run.
- Recover gracefully from server restarts, network blips, and connector idle gaps.
OpenMetadata addresses this with a server-side log storage abstraction backed by S3 (or any S3-compatible store like MinIO). The connector pushes log batches over HTTP; the server persists them and serves both live and post-run reads.
## Architecture
```
┌──────────────────────┐
│ Python ingestion │ POST /logs/{fqn}/{runId} (append)
│ connector │ POST /logs/{fqn}/{runId}/close (finalize)
│ (logs_mixin.py) │
└──────────┬───────────┘
│ HTTP
▼
┌──────────────────────┐
│ OpenMetadata server │
│ IngestionPipeline │
│ Resource │
└──────────┬───────────┘
│ LogStorageInterface
▼
┌──────────────────────┐ ┌──────────────────────┐
│ S3LogStorage │────────▶│ S3 / MinIO bucket │
│ (streaming, in-mem │ │ partial.txt │
│ buffers, sweeper) │ │ logs.txt │
└──────────┬───────────┘ └──────────────────────┘
│ SSE / GET (paginated / download)
▼
┌──────────────────────┐
│ OpenMetadata UI │
│ (live tail + history)│
└──────────────────────┘
```
The `LogStorageInterface` abstraction supports multiple backends:
| Backend | Purpose |
|---------|---------|
| `S3LogStorage` | Production: stores logs durably in S3 / MinIO. The focus of this document. |
| `DefaultLogStorage` | Backward-compat: delegates to the pipeline service client (Airflow / Argo). No first-class storage. |
This document covers the `S3LogStorage` implementation.
## Storage Layout
Each pipeline run is identified by a `(fqn, runId)` tuple. On S3 the layout is:
```
{bucket}/{prefix}/ # prefix defaults to "pipeline-logs"
{sanitizedFQN}/{runId}/
partial.txt # readable view during the run
logs.txt # final artifact, materialized at /close
.active/{sanitizedFQN}/{runId}/{serverId} # heartbeat marker
```
**`partial.txt`** is the durable, readable view of an in-progress run. It is updated periodically as the connector appends batches. It carries durable offset state in S3 user-defined metadata:
| Metadata key | Purpose |
|--------------|---------|
| `x-amz-meta-last-flushed-line` | Logical line counter at the moment of this PUT. Drives retry idempotency and post-restart recovery. |
| `x-amz-meta-total-bytes` | Cross-check on body size; helps detect drift. |
| `x-amz-meta-writer-epoch` | Bumped each time a fresh OM-server instance picks up the stream after a restart. |
| `x-amz-meta-writer-version` | Identifies the writer code version. Useful during migration windows. |
**`logs.txt`** is the canonical post-run artifact. It is created **only** at `/close` (or by the abandoned-run sweeper), as a server-side S3 copy of the final `partial.txt`. Content matches `partial.txt` exactly at the moment of close.
**`.active/...`** markers are dropped as a side effect of `appendLogs`. They have no functional role in correctness; they are operational hints for diagnostics ("which OM-server instance most recently saw this run").
A bucket lifecycle policy ensures cleanup:
- `expirationDays` (default 30) on the `pipeline-logs/` prefix expires all logs after the retention window.
## Run Lifecycle
### 1. Connector emits a batch
The Python ingestion runner buffers log lines and POSTs batches to the server:
```
POST /api/v1/services/ingestionPipelines/logs/{fqn}/{runId}
Content-Type: application/json
"<raw log content>" OR
{
"logs": "<base64-gzipped log content>",
"connectorId": "...",
"compressed": true
}
```
`IngestionPipelineResource.writePipelineLogs` decodes the body and calls `repository.appendLogs(fqn, runId, content)`, which delegates to `S3LogStorage.appendLogs`.
### 2. Server-side append
`S3LogStorage.appendLogs` does five things, all in memory, all under a per-stream `ReentrantLock`:
1. **Increments `totalLinesAppended`**, the monotonic logical line counter that anchors retry idempotency.
2. **Appends to `SimpleLogBuffer`** (in-memory ring, capacity 1000 lines). This is the source for the SSE/WebSocket live-tail UI experience. It is bounded; oldest lines evict on overflow. It is **not** load-bearing for durability.
3. **Appends to `pendingFlush`** (in-memory queue, no fixed cap, byte-tracked). This is the durable-pending-write queue and survives until the next successful PUT.
4. **Notifies SSE listeners**, fanning out the new lines to any open live-tail HTTP connections.
5. **Schedules an early flush** if `pendingFlush` exceeds `earlyFlushWatermarkBytes` (default 5 MB). This protects against memory bloat under bursty writes.
A single-threaded `cleanupExecutor` schedules the periodic flush, the abandoned-run sweeper, and metrics updates.
### 3. Periodic flush to `partial.txt`
Every `partialFlushIntervalMinutes` (default 2) and on demand from the early-flush watermark, `writePartialLogsForStream` runs under the per-stream lock:
1. Snapshot `pendingFlush` and clear it.
2. If empty, no-op (idle streams cost nothing).
3. `GetObject partial.txt` → reads `Content-Length` and metadata from the response headers. On 404, treat as empty.
4. Build new metadata (`last-flushed-line`, `total-bytes`, `writer-epoch`, `writer-version`).
5. **If existing body < 5 MB** — read the body, build merged body = existing + `\n`-joined snapshot, `PutObject` atomically.
6. **If existing body ≥ 5 MB** — abort the body stream and concatenate server-side via Multipart Upload: `CreateMultipartUpload`, `UploadPartCopy` (existing body as part 1), `UploadPart` (new content as part 2, the last part has no 5 MB minimum), `CompleteMultipartUpload`. The merged body never enters JVM heap and is not re-uploaded.
7. On failure, abort any in-flight multipart upload, re-merge the snapshot to the head of `pendingFlush`, and try again next tick. No data loss.
Because `pendingFlush` is unbounded by the `SimpleLogBuffer` cap, no line is ever evicted before being flushed.
### 4. Live read while running
The UI's "live logs" view does two things in parallel:
- **HTTP GET** `/logs/{fqn}/{runId}?after={cursor}` for paginated history. The server reads `partial.txt` from S3 and concatenates the in-memory `pendingFlush` snapshot for the most-recent-tail bytes that haven't yet been flushed. The cursor is a line offset.
- **Server-Sent Events (SSE)** for live tail. The endpoint registers a `LogStreamListener` against the stream key and pushes new lines as `notifyListeners` fires from each `appendLogs`.
This gives the user "everything written so far" via GET and "everything written in real time from now on" via SSE.
### 5. `/close` finalization
When the connector terminates (success, graceful failure, or graceful abort), it calls:
```
POST /api/v1/services/ingestionPipelines/logs/{fqn}/{runId}/close
```
`S3LogStorage.closeStream` runs under the per-stream lock:
1. **Final flush**: drain remaining `pendingFlush` to `partial.txt` (same path as the periodic flush).
2. **Server-side copy** `partial.txt` → `logs.txt`. Bytes do not transit through OM. Cheap and constant-time regardless of log size.
3. **Delete `partial.txt`**.
4. **Best-effort delete** the `.active/{fqn}/{runId}/{serverId}` marker.
5. Drop in-memory state for the stream (`activeStreams`, `pendingFlush`, `totalLinesAppended`, `recentLogsCache`, the per-stream lock).
`/close` is idempotent. A second call finds no `partial.txt` and no in-memory state; it is a graceful no-op. A `/close` that arrives after the abandoned-run sweeper already finalized the stream behaves the same way.
### 6. Post-`/close` reads
Once `/close` completes, `logs.txt` is the canonical artifact. `getLogs(fqn, runId)` reads it directly. Pagination is by line offset; the response includes `after` (next cursor) and `total` (total bytes / lines).
There is also a download endpoint that streams the full file (or composes from segments / partial in legacy fallbacks).
## Read Paths
| Endpoint | Pre-`/close` | Post-`/close` |
|----------|-------------|---------------|
| `GET /logs/{fqn}/{runId}` | Reads `partial.txt` + appends `pendingFlush` snapshot. Apply cursor pagination. | Reads `logs.txt`. |
| `GET /logs/{fqn}/{runId}/download` | Streams `partial.txt`. | Streams `logs.txt`. |
| `GET /logs/{fqn}/stream/{runId}` (SSE) | Registers a listener; replays last 100 buffered lines, then live-streams new lines. | (Not used post-close; the run is over.) |
Legacy `partial.txt` files written by older code (without S3 metadata) read normally; the new flush logic treats them as "no prior offset" and merges any new content correctly.
## Abandoned-Run Recovery
Connectors can die without calling `/close` — process killed, OOM, network partition, infrastructure failure. To bound resource use and still produce a final `logs.txt`, a sweeper runs periodically:
- **Schedule**: every `cleanupIntervalMinutes` (default 60).
- **Threshold**: `streamTimeoutMinutes` since last `appendLogs` (default 1440 = 24h).
For each expired stream, the sweeper does the same finalization steps as `/close` (final flush, copy to `logs.txt`, delete `partial.txt`, drop in-memory state). The end result is identical: an abandoned run produces a finalized `logs.txt` artifact that the UI can read, just delayed.
The 24h default is intentionally lenient: typical idle gaps in slow connectors (waiting on source queries, batch boundaries, queues) are minutes-to-hours, not days. Operators can tune the threshold downward in deployments where memory pressure from many parallel runs requires more aggressive reclamation.
## Failure Modes & Recovery
| Failure | Recovery |
|---------|----------|
| S3 PUT fails during periodic flush | `pendingFlush` snapshot is restored under the lock. Next tick retries. No data loss. |
| OM-server restart mid-run | All in-memory state lost. `partial.txt` on S3 retains all previously-flushed content. The next `appendLogs` re-creates state; the first flush after restart reads `partial.txt` (with metadata) and resumes from `last-flushed-line`. Worst-case loss: lines that were in `pendingFlush` at restart time, bounded above by `partialFlushIntervalMinutes`. |
| Connector dies without `/close` | Abandoned-run sweeper finalizes the run after `streamTimeoutHours`. `logs.txt` is materialized from the most recent `partial.txt`. |
| `/close` retries after partial success | All steps are idempotent. Second call finds no `partial.txt` and no in-memory state; no-op. |
| Concurrent `appendLogs` and cleanup | The per-stream lock serializes them. Cleanup finds the stream "fresh" again and skips it next tick. |
| Bucket lifecycle expires `partial.txt` mid-run | Should not happen at default `expirationDays = 30`. If misconfigured (very low retention), the next flush would treat it as a fresh `partial.txt` and start over. Recommended floor: 7 days. |
## Configuration
All settings live under `LogStorageConfiguration` in `openmetadata.yaml`:
| Field | Default | Description |
|-------|---------|-------------|
| `bucketName` | (required) | S3 bucket for log storage. |
| `prefix` | `pipeline-logs` | Key prefix within the bucket. |
| `enableServerSideEncryption` | `true` | Apply SSE on every PUT. |
| `sseAlgorithm` | `AES_256` | Or `AWS_KMS` (requires `kmsKeyId`). |
| `storageClass` | `STANDARD_IA` | S3 storage class for log objects. |
| `expirationDays` | 30 | Bucket lifecycle: expire all logs after this many days. |
| `streamTimeoutMinutes` | 1440 | Idle threshold (in minutes) before the abandoned-run sweeper finalizes a stream. |
| `cleanupIntervalMinutes` | 60 | How often the sweeper wakes up to check for abandoned streams. |
| `partialFlushIntervalMinutes` | 2 | Periodic `pendingFlush` → `partial.txt` cadence. |
| `earlyFlushWatermarkBytes` | 5242880 (5 MB) | Triggers an out-of-band flush when `pendingFlush` exceeds this size. |
| `pendingFlushAlertAfterFailures` | 10 | Emit an alerting metric after this many consecutive failed flushes for a stream. |
| `maxConcurrentStreams` | 100 | Bound on in-flight pipeline runs per OM-server instance. |
| `awsConfig.*` | — | AWS credentials / region / endpoint (also supports IAM role + custom endpoints for MinIO). |
## Concurrency Model
Coordination is a per-stream lock keyed by `streamKey = fqn + "/" + runId`. The lock is held for the duration of `appendLogs`, periodic flush, abandoned-run cleanup, and `/close`. Locks are backed by a Guava `Striped<Lock>` with a fixed stripe count, so memory does not grow with completed-run accumulation; the same key always maps to the same lock instance, eliminating the acquire-vs-remove race that a per-key map would have. False contention across stripes is bounded by `maxConcurrentStreams << stripe count`.
A single-threaded `ScheduledExecutorService` (`cleanupExecutor`) drives:
- Periodic flushes (`writePartialLogs`)
- Abandoned-run sweeper (`cleanupAbandonedStreams`)
- Metrics updates (`updateStreamMetrics`)
- One-shot early flushes scheduled by the watermark trigger
Under sustained burst load, scheduled tasks queue on this single thread. This is intentional: it bounds resource use and avoids unbounded thread creation under spikes. If a deployment regularly sees queue backlog, the watermark or flush interval can be tuned.
## Observability
Key metrics exposed by `StreamableLogsMetrics`:
- `om_streamable_logs_log_shipment_*` — distribution of append latencies.
- `om_streamable_logs_logs_sent` / `logs_failed` — counter of successful and failed appends.
- `om_streamable_logs_batch_size` — distribution of lines per batch.
- `om_streamable_logs_s3_*` — distribution of S3 read/write latencies and counters of S3 errors.
- `om_streamable_logs_pending_part_uploads` — gauge for monitoring queue backlog (legacy, will be retired with multipart removal).
- `om_streamable_logs_multipart_uploads` — gauge for active multipart uploads (legacy, will be retired).
- `om_streamable_logs_pending_flush_bytes` — gauge for in-memory `pendingFlush` size per stream (new).
- `om_streamable_logs_consecutive_flush_failures` — gauge per stream (new).
Recommended alerts:
- `pending_flush_bytes` > 50 MB sustained → memory pressure or persistent S3 failures.
- `consecutive_flush_failures` ≥ 10 → S3 connectivity or auth issue.
- `s3_errors` rate > 1/min → S3 health degradation.
## Multi-Server Topology
The design assumes single-writer-per-run: an ALB / load balancer enforces sticky sessions for `(fqn, runId)` via the `PIPELINE_SESSION` cookie set on the first `appendLogs` response. All subsequent requests for the same run land on the same OM-server instance for the lifetime of the run.
If stickiness is broken (cookie stripped by a proxy, multi-cluster routing without coordination), two OM-server instances could write to the same `partial.txt` and clobber each other. This is **out of scope** for the current design. A future iteration could move offset state to the database for cross-server coordination.
## References
- Source files:
- `openmetadata-service/src/main/java/org/openmetadata/service/logstorage/S3LogStorage.java`
- `openmetadata-service/src/main/java/org/openmetadata/service/logstorage/LogStorageFactory.java`
- `openmetadata-spec/src/main/java/org/openmetadata/service/logstorage/LogStorageInterface.java`
- `openmetadata-service/src/main/java/org/openmetadata/service/resources/services/ingestionpipelines/IngestionPipelineResource.java`
- `ingestion/src/metadata/utils/streamable_logger.py`
- `ingestion/src/metadata/ingestion/ometa/mixins/logs_mixin.py`
- Related PRs: #23590, #24198, #24287, #24410