open-metadata--openmetadata/docs/company-context-knowledge-pills.md

Company Context — File-Extracted Knowledge Pills

• Date: 2026-06-12
• Status: Implemented (as-built spec for branch pmbrull/companycontext-entity)
• Author: pmbrull (with Claude)
• Scope: 72 files, ~+2818/−81. Backend (Java), schema (JSON), ingestion of file content into reusable memories, embedding/search, MCP tools, UI badge.

This document describes the feature as shipped, not the earlier pre-implementation design. Where the implementation diverged from that design, the as-built behavior is recorded here and flagged in §13 Known limitations.

---

1. Summary

The Context Center ("Drive") lets users upload documents. Previously an upload was stored, its raw text extracted into ContextFile.extractedText, and that was the end. This feature turns that raw text into structured, retrievable knowledge pills — short question/answer facts about the company — that are embedded, indexed, linked back to the originating file, and reachable by any MCP-connected agent.

Pills are not a new entity. They are ContextMemory rows tagged sourceType=FileExtraction with a sourceFile back-reference, riding the entity's existing embedding + search plumbing.

End-to-end:

upload ──▶ store blob (object storage / AssetService)
       ──▶ extract text                       (status: Analyzing)
       ──▶ LLM extracts knowledge pills        (status: ExtractingContext)
       ──▶ pills stored as ContextMemory rows, linked to the file
       ──▶ pills auto-embedded + indexed on create
       ──▶ pills retrievable via MCP + searchable by source filename
       file carries a processing status surfaced in the UI    (status: Processed)

2. Motivation

ContextMemory already existed — "Reusable context memory for Context Center and AI-assisted retrieval", a question/answer pill that is already vector-embedded (contextMemory is in AvailableEntityTypes, has ContextMemoryBodyTextContributor + ContextMemoryIndex, and its index carries parent aliases ["all", "dataAssetEmbeddings"]). It needed only a new source type and a file back-reference to become the backing store for file-extracted knowledge, avoiding a second overlapping "knowledge" concept and a full new entity stack.

The real new infrastructure is a generic LLM completion layer — there was no chat/completion client before (only EmbeddingClient and a NoOp NLQService). It is built decoupled from Context Center so other features (e.g. MCP Chat) can reuse it.

3. Architecture & data flow

POST /v1/contextCenter/drive/files                         (ContextFileResource:208)
  └─ store asset (AssetServiceFactory.getService().upload), persist ContextFile + content
  └─ extractionService.submit(fileId, contentId)           (ContextFileResource:287)

ContextFileProcessingService.process(fileId, contentId)    (drive/ContextFileProcessingService.java)
  guard: contentId == file.headContentId   (else abandon — stale re-upload)
  1. markAnalyzing()                         → status Analyzing (file + content)
  2. extractText()  [DEFAULT_EXECUTOR, CPU]  → read blob via AssetService, text-extract
        ├─ failure → status Failed (+ processingError on content), extractedText cleared
        └─ success → text status (Processed / Unsupported)
  3. if text Processed AND LLMClientHolder.isEnabled():
        status ExtractingContext (file)      [content snapshot stays Processed]
        submitMemoryExtraction()  [LLM_EXECUTOR, network]
          runMemoryExtraction():
            deleteExtractedMemories(file, hardDelete=true)   ← wholesale replace
            ContextMemoryExtractor.extract(file, canonicalText)
              chunk → per-chunk LLM call → dedupe → create ContextMemory rows
                {sourceType:FileExtraction, sourceFile:<ref>, status:Active,
                 visibility:Shared}            └─ VectorEmbeddingHandler auto-embeds
            status Processed (file)
          ├─ LLM failure → status Failed (+ processingError), extractedText RETAINED
          └─ LLM queue full → status Failed ("…queue is full. Please retry later.")
     else (LLM disabled): text Processed is terminal; LLM stage never enqueued

Two separate thread pools by design (ContextFileProcessingService):

Pool	Thread-name prefix	Work	Sizing
DEFAULT_EXECUTOR	context-file-extraction-	CPU-bound text extraction	threads = max(2, cores/2), ArrayBlockingQueue(max(64, threads*8)), AbortPolicy, daemon
LLM_EXECUTOR	context-memory-extraction-	network-bound LLM calls	identical sizing

Mixing seconds-long LLM calls into the text pool would starve text extraction; the pools are kept separate (the class javadoc spells this out). They are currently sized identically — the only difference is the thread-name prefix.

4. Key design decisions

Decision	Choice
Entity	Reuse ContextMemory (new sourceType=FileExtraction + sourceFile); no new entity, no DDL migration
LLM layer	New generic LLMCompletionClient abstraction + per-provider concretes + factory + process-wide holder, modeled on EmbeddingClient
LLM→entity boundary	A narrow KnowledgePill DTO is the anti-corruption layer between untrusted model JSON and the entity (see §9)
File↔pill link	Relationship.MENTIONED_IN edge, ContextFile → ContextMemory; memoryCount surfaced on the file
Reprocess	Wholesale hard-delete + recreate of a file's pills on every extraction run (no versioning, no checksum skip)
Async	Reuse + rename the live ContextFileExtractionService → ContextFileProcessingService; LLM step on its own executor
Status	Extend the single ProcessingStatus enum: insert ExtractingContext between Analyzing and Processed
Embedding	Free — ContextMemory already auto-embeds on create; this PR only threads sourceFile through the index/body-text
MCP	Two purpose-built tools search_company_context + get_company_context
Storage guardrail	New /system/validate "Object Storage" step + startup warn — uploads are useless if object storage is NoOp/broken

5. Generic LLM completion layer

Package org.openmetadata.service.llm (all new).

5.1 Config schema

openmetadata-spec/.../json/schema/configuration/llmConfiguration.json → org.openmetadata.schema.configuration.LLMConfiguration.

• enabled — boolean, default false.

• provider — enum ["bedrock", "openai", "azureOpenAI", "google", "anthropic", "noop"], default noop (Java enum LLMProvider).

• maxConcurrentRequests — integer, default 5, minimum 1.

• Per-provider objects (each additionalProperties:false):

  Object	Java type	Auth (masked?)	Notable fields / defaults
  bedrock	LLMBedrockConfig	awsConfig → awsBaseConfig.json	modelId anthropic.claude-3-5-sonnet-20240620-v1:0, temp 0.0, maxTokens 4096, timeout 60s
  openai	LLMOpenAIConfig	apiKey mask:true	modelId gpt-4o-mini, endpoint, deploymentName, apiVersion 2024-02-01 (Azure), temp 0.0, maxTokens 4096, timeout 60s
  google	LLMGoogleConfig	apiKey mask:true	modelId gemini-2.5-flash, endpoint, temp 0.0, maxTokens 4096, timeout 60s
  anthropic	LLMAnthropicConfig	apiKey mask:true	modelId claude-3-5-sonnet-20240620, baseUrl https://api.anthropic.com, temp 0.0, maxTokens 4096, timeout 60s

  Masked API keys round-trip through the Secrets Manager. Bedrock delegates secrecy to awsBaseConfig.json.

conf/openmetadata.yaml carries an llmConfiguration block, fully env-var overridable (LLM_ENABLED, LLM_PROVIDER, LLM_MAX_CONCURRENT_REQUESTS, and per-provider LLM_<PROVIDER>_* keys — see §12). Its header comment states plainly that uploaded document content is sent to the configured provider when enabled.

5.2 Client hierarchy

• LLMCompletionClient (abstract, @Slf4j): owns a Semaphore (maxConcurrentRequests permits; ctor rejects < 1), the complete() / completeStructured() template methods, JSON-array parsing, and code-fence stripping. Abstract surface: doComplete(systemPrompt, userPrompt) and getModelId(). DEFAULT_MAX_CONCURRENT_REQUESTS = 5.
• Concretes — OpenAICompletionClient (OpenAI + Azure OpenAI), AnthropicCompletionClient, BedrockCompletionClient (AWS SDK v2, AutoCloseable), GoogleCompletionClient, and NoopCompletionClient (returns "[]", model id "noop").
• LLMCompletionClientFactory.create(LLMConfiguration) — switch on provider → concrete; null config/provider → Noop. Note: the factory keys off provider only; it does not check enabled (the holder does).
• LLMClientHolder — process-wide holder of the single shared client. initialize(config): enabled = config != null && Boolean.TRUE.equals(getEnabled()); instance = enabled ? factory.create(config) : new NoopCompletionClient(). So a real client is built only when enabled == true. get() never returns null; isEnabled() gates the pipeline; setForTesting() is the test seam. volatile fields + synchronized mutators.

5.3 completeStructured contract

<T> List<T> completeStructured(String systemPrompt, String userPrompt, Class<T> elementType)

• Acquires a permit, calls doComplete, releases in finally.
• Strips a leading/trailing ``` code fence (drops the language tag line).
• Parses with a default new ObjectMapper() (so FAIL_ON_UNKNOWN_PROPERTIES is on) into List<elementType>. Parse failure → LLMCompletionException.
• One retry on LLMCompletionException (logged warn), then propagates. The retry catches LLMCompletionException broadly, so transport errors that surface as LLMCompletionException also get one retry.

5.4 Providers

All providers use plain chat/messages JSON mode (no tool/function calling); non-200 → LLMCompletionException("<provider> API returned status …").

• OpenAI / Azure: JDK HttpClient. Azure detected when endpoint and deploymentName are set → .../openai/deployments/<dep>/chat/completions?api-version=<v> with api-key header; else Authorization: Bearer. Reads choices[0].message.content.
• Anthropic: JDK HttpClient, <baseUrl>/v1/messages, headers x-api-key + anthropic-version: 2023-06-01. Reads content[0].text.
• Bedrock: AWS SDK v2 BedrockRuntimeClient (credentials via AwsCredentialsUtil.buildCredentialsProvider, region required). Body uses anthropic_version: bedrock-2023-05-31. Reads content[0].text. close() exists but the holder never calls it.
• Google Gemini: JDK HttpClient, .../models/<modelId>:generateContent?key=<apiKey> (API key in the URL query string). Reads candidates[0].content.parts[0].text.

5.5 Startup wiring

OpenMetadataApplicationConfig gains @JsonProperty("llmConfiguration") private LLMConfiguration llmConfiguration. OpenMetadataApplication startup calls LLMClientHolder.initialize(catalogConfig.getLlmConfiguration()) (new), then the pre-existing LlmConfigHolder.initialize(...) (publishes raw config for MCP Chat). Two holders, same config source.

6. Schema & persistence changes

No DDL / Flyway migration — ContextMemory and ContextFile use the generic JSON-column entity tables, and the file↔pill link lives in the generic entity_relationship table.

6.1 Schemas

• contextMemory.json — sourceType enum gains FileExtraction (FILE_EXTRACTION); new property sourceFile (entityReference, not required).
• createContextMemory.json — new create-time sourceFile (entityReference).
• contextFile.json — ProcessingStatus enum gains ExtractingContext (3rd, between Analyzing and Processed); new derived property memoryCount (integer, default 0).

6.2 Repositories

• ContextMemoryRepository — FIELD_SOURCE_FILE = "sourceFile" added to PATCH_FIELDS/UPDATE_FIELDS. storeRelationships adds an addRelationship(sourceFile.id /*from*/, memory.id /*to*/, CONTEXT_FILE, CONTEXT_MEMORY, Relationship.MENTIONED_IN) edge. setFields resolves it with findFrom(memory.id, CONTEXT_MEMORY, MENTIONED_IN, CONTEXT_FILE) (bulk path via findFromBatch). Patch handled via updateFromRelationship(...).
  • Collision-free by construction: each relationship hierarchy uses a distinct Relationship — rootMemory→CONTAINS, parentMemory→PARENT_OF, domains→HAS, sourceFile→MENTIONED_IN — so the four resolvers query disjoint (relation, fromType) tuples.
• ContextFileRepository — memoryCount computed in setFields as findTo(file.id, CONTEXT_FILE, MENTIONED_IN, CONTEXT_MEMORY).size(). New postDelete cascades: deleteExtractedMemories(file, hardDelete) deletes each linked pill, propagating the file delete's hardDelete flag (soft→soft, hard→hard). The same method is reused by the reprocess path with hardDelete=true.
• ContextMemoryMapper — passthrough of create.getSourceFile().

7. Processing pipeline & status

7.1 Status state machine (ProcessingStatus)

Uploaded ─▶ Analyzing ─▶ (text result) ─▶ Processed ─▶ ExtractingContext ─▶ Processed
                          │                  ▲ (LLM enabled)                    (final)
                          ├─▶ Failed         └─ if LLM disabled: terminal here
                          └─▶ Unsupported (passthrough from text extractor)
                       any stage failure ─▶ Failed

• The file and its content snapshot carry status independently. When the file advances to ExtractingContext, the content snapshot keeps Processed (it received the raw text status).
• Unsupported exists in the enum but the service never writes it — it is passed through verbatim from ContextFileTextExtractor.

7.2 Service (ContextFileProcessingService, was ContextFileExtractionService)

• Only call site: ContextFileResource upload path → extractionService.submit(fileId, contentId).
• submit wraps executor.execute(() -> process(...)) and catches RejectedExecutionException (queue full → Failed, "Processing queue is full").
• Every stage re-reads from the repo and re-checks headContentId; a newer upload silently abandons the stale content's writes (concurrent re-upload guard).
• canonicalText feeds the LLM the content snapshot's extracted text (capped ~1M chars) in preference to the file's indexed text (capped ~200K).

7.3 Error handling

• applyFailure sets both file and content to Failed; processingError is written to the content snapshot only, not the file entity.
• Text-stage failures clear extractedText (+ pageCount). LLM-stage failures retain extractedText (so the file stays indexed and is retriable by re-upload).
• extractText catches Throwable but re-throws VirtualMachineError.
• No automatic retry anywhere; "retry later" means re-POST the file.

8. Knowledge-pill extraction (ContextMemoryExtractor)

• extract(file, text): chunkText → per-chunk completeStructured(SYSTEM_PROMPT, chunk, KnowledgePill.class) → dedupe → memoryRepository.create(toMemory(pill, fileRef)).
• Chunking: MAX_PROMPT_CHARS = 60_000, MAX_CHUNKS = 8. Chunk ends snap back to the last \n\n / \n / boundary in the second half of the budget, else hard-cut at the char cap. Text past 8 chunks (~480K chars) is dropped with a warning — even though canonicalText may supply up to ~1M chars.
• Dedupe: LinkedHashMap keyed by question.trim().toLowerCase(ROOT), first-wins, insertion order preserved.
• isValid: question and answer both non-blank (title/summary/memoryType optional).
• toMemory sets the full server-owned envelope: id (random UUID), name (<fileName>-<uuid>), fullyQualifiedName, title, question, answer, summary, memoryType (via parseType), status=ACTIVE, sourceType=FILE_EXTRACTION, sourceFile=<fileRef>, shareConfig.visibility=SHARED, updatedBy=admin, updatedAt=now.
• parseType: case-insensitive match against ContextMemoryType (Preference | UseCase | Note | Runbook | Faq); default NOTE.
• SYSTEM_PROMPT: "You extract reusable company knowledge from a document as a JSON array. Each element is an object with keys: title, question, answer, summary, memoryType (one of Faq, Note, Runbook, UseCase, Preference). Capture durable facts, definitions, policies, and how-to guidance. Return ONLY the JSON array, no prose."

Reprocess / idempotency

Every LLM run first calls deleteExtractedMemories(file, hardDelete=true), hard-deleting all of the file's existing pills, then recreates from scratch with fresh random IDs/FQNs. There is no supersede/versioning and no checksum short-circuit — re-uploading identical bytes re-runs text extraction and the LLM and regenerates pills. (Rationale in code: machine-generated pills are replaced wholesale, so soft-deleted rows would only accumulate with no restore path.)

9. The KnowledgePill DTO boundary

KnowledgePill (service/llm/KnowledgePill.java) is a 5-field record (title, question, answer, summary, memoryType) used only as the LLM deserialization target — deliberately not the ContextMemory entity. It is the anti-corruption boundary between untrusted model JSON and the trusted entity model. Parsing model output straight into ContextMemory is avoided because:

1. Strict unknown fields. completeStructured uses a default ObjectMapper (FAIL_ON_UNKNOWN_PROPERTIES on) and contextMemory.json is additionalProperties:false — one stray model key would fail the whole array parse. KnowledgePill carries @JsonIgnoreProperties(ignoreUnknown = true).
2. Enum leniency. ContextMemory.memoryType is the ContextMemoryType enum (Jackson throws on a non-exact value); KnowledgePill.memoryType is a raw String, so casing/variant forms degrade to NOTE in parseType.
3. Server-owned identity. contextMemory.json requires id/name, and the whole envelope (FQN, status, sourceType, sourceFile, shareConfig, updatedBy/At) is set server-side in toMemory — the model neither produces nor should influence these.
4. Minimal prompt contract. The DTO is exactly the five fields the prompt asks for; targeting the entity would expose its nested schema to the model.

toMemory + parseType is the single mapping layer; validation and dedupe operate on the DTO before any entity is built.

10. Embedding & search

contextMemory was already vector-indexable and auto-embedded on create (AvailableEntityTypes, VectorEmbeddingHandler, parent aliases ["all", "dataAssetEmbeddings"]) — unchanged here. This PR threads the new sourceFile reference through three layers so file-derived pills are searchable by their origin filename:

• ContextMemoryIndex — doc.put("sourceFile", getEntityWithDisplayName(...)).
• ES/OS mapping — a new sourceFile object (id/type/name/displayName/fqn/deleted as keywords) added identically to all four elasticsearch/{en,jp,ru,zh}/context_memory_search_index.json files (the per-language copies are byte-identical; no language-specific analyzers).
• ContextMemoryBodyTextContributor — appends source file: <name> to the embedding body text.

11. Object-storage validation (guardrail)

Uploaded file content lives in object storage (S3 / Azure / in-memory) via AssetService. If that backend is disabled or NoOp, uploads silently "succeed" but their bytes are discarded and extraction fails with "Unable to read file content from object storage" — producing zero pills. To surface this before users hit it:

• SystemRepository.getObjectStorageValidation(config) adds an "Object Storage" step to the existing GET /api/v1/system/status validation. It fails fast when object storage is disabled/missing or the factory holds a NoOpAssetService; otherwise it runs a live write→read→delete round-trip probe (tiny text/plain asset, 10s per-op timeout, asserts the bytes match, cleans up in finally).
• AssetServiceFactory.init logs a warn when storage is disabled, naming Context Center Drive and the exact failure string.
• Test (SystemRepositoryObjectStorageValidationTest) exercises the real AssetServiceFactory + InMemoryAssetService: disabled, missing config, NoOp-after-config-drift, and a passing live in-memory round-trip.

This reuses the existing /system/status framework (no new endpoint) but exists solely for the Drive → object-storage → extraction pipeline.

12. MCP tools

Two tools in openmetadata-mcp/.../tools/, registered in tools.json and dispatched by new DefaultToolContext switch cases (via the 3-arg, no-limits execute — neither tool records usage or enforces limits).

search_company_context

• Input: query (string, required); size (integer, default 10, clamped 1..50).
• Query: OpenSearchVectorService.search(query, filters, size, from=0, k=100, threshold=0.0) with filters pinned to entityType=contextMemory, sourceType=FileExtraction, and visibility=Shared.
• Output: {query, results:[{fullyQualifiedName, name, title, question, answer, summary, sourceFile, similarityScore}], returnedCount}.
• Auth: global authorize(CONTEXT_MEMORY, VIEW_ALL). Requires vector embedding enabled, else returns an error payload.

get_company_context

• Input: fqn (string, required — the fullyQualifiedName from a search hit).
• Lookup: Entity.getEntityByName(CONTEXT_MEMORY, fqn, "sourceFile,owners,tags,domains", null).
• Exposability gate: returns the pill only if sourceType=FILE_EXTRACTION and shareConfig.visibility=SHARED, else an error.
• Output: {fullyQualifiedName, name, title, question, answer, summary, memoryType, sourceFile (as FQN)}.
• Auth: same global VIEW_ALL.

Both gate on a single global VIEW_ALL plus the Shared-visibility scope; there is no per-pill owner re-check beyond that.

13. UI

• DocumentStatusBadge (new) renders a ui-core-components Badge (size sm, no icon) from a ProcessingStatus, returning null when status is absent. Mapping:

  Status	Color	Label key
  Uploaded	gray	label.uploaded
  Analyzing	blue	label.analyzing
  ExtractingContext	indigo	label.extracting-context
  Processed	success	label.processed
  Failed	error	label.failed
  Unsupported	warning	label.unsupported

• DocumentsView renders the badge inline next to each document's filename, driven by file.processingStatus.

• i18n: 4 new en-us keys (label.analyzing, label.extracting-context, label.unsupported, label.uploaded); label.processed/label.failed pre-existed. Synced across all locales.

14. Security & privacy

• LLM provider API keys are mask:true config through the Secrets Manager — never logged, never returned in API responses.
• File content is sent to the configured LLM provider during extraction. This is an explicit, admin-enabled action (llmConfiguration.enabled), and the provider is the admin's configured choice — documented in the yaml header.
• Generated pills default to visibility=Shared (org-readable). MCP retrieval enforces the Shared scope and a VIEW_ALL authorize on every call.
• No new unauthenticated surface; MCP tools ride existing OAuth/JWT auth.

15. Testing

Java unit (openmetadata-service)

• drive/ContextFileProcessingServiceTest — status machine: analyzing→processed, LLM-enabled extraction, LLM-rejection/failure keeps text, storage-unavailable & null-stream failures, executor rejection, head-content skip, VME rethrow.
• drive/ContextMemoryExtractorTest — one-memory-per-pill, dedupe + skip-invalid, skip-when-no-text, multi-chunk dedupe, chunk cap.
• jdbi3/SystemRepositoryObjectStorageValidationTest — the four storage-validation scenarios.
• llm/LLMClientHolderTest, llm/LLMCompletionClientFactoryTest, llm/LLMCompletionClientTest — holder stability + disabled-builds-no-client, factory dispatch, structured parsing / code-fence / concurrency guard.

MCP (openmetadata-mcp)

• tools/GetCompanyContextToolTest — missing/blank fqn, auth-denied, shared-pill projection, non-file & private rejection.
• tools/SearchCompanyContextToolTest — missing/blank query, auth-denied (input guards; does not exercise the vector path).

Integration (openmetadata-integration-tests)

• it/drive/CompanyContextPipelineIT — end-to-end: upload .txt, poll (Awaitility, ≤45s) to Processed, assumeTrue(memoryCount > 0) to skip gracefully when no LLM provider is configured; otherwise asserts memoryCount == pills.size() and per-pill sourceType=FILE_EXTRACTION, non-null sourceFile matching the file id, non-blank question/answer.

UI — DocumentStatusBadge.test.tsx (renders nothing without status; it.each over the six status→label→color rows). No new Playwright spec.

16. Configuration reference

llmConfiguration:
  enabled: ${LLM_ENABLED:-false}
  provider: ${LLM_PROVIDER:-noop}          # noop | openai | azureOpenAI | bedrock | google | anthropic
  maxConcurrentRequests: ${LLM_MAX_CONCURRENT_REQUESTS:-5}
  openai:
    apiKey:        ${LLM_OPENAI_API_KEY:-""}
    modelId:       ${LLM_OPENAI_MODEL_ID:-"gpt-4o-mini"}
    endpoint:      ${LLM_OPENAI_ENDPOINT:-""}
    deploymentName:${LLM_OPENAI_DEPLOYMENT:-""}
    apiVersion:    ${LLM_OPENAI_API_VERSION:-"2024-02-01"}
    maxTokens:     ${LLM_OPENAI_MAX_TOKENS:-4096}
  bedrock:
    awsConfig:
      awsRegion:       ${AWS_DEFAULT_REGION:-""}
      # IAM or static keys via AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN
    modelId:   ${LLM_BEDROCK_MODEL_ID:-"eu.anthropic.claude-haiku-4-5-20251001-v1:0"}
    maxTokens: ${LLM_BEDROCK_MAX_TOKENS:-4096}
  google:
    apiKey:    ${LLM_GOOGLE_API_KEY:-""}
    modelId:   ${LLM_GOOGLE_MODEL_ID:-"gemini-2.5-flash"}
    maxTokens: ${LLM_GOOGLE_MAX_TOKENS:-4096}
  anthropic:
    apiKey:    ${LLM_ANTHROPIC_API_KEY:-""}
    modelId:   ${LLM_ANTHROPIC_MODEL_ID:-"claude-3-5-sonnet-20240620"}
    baseUrl:   ${LLM_ANTHROPIC_BASE_URL:-"https://api.anthropic.com"}
    maxTokens: ${LLM_ANTHROPIC_MAX_TOKENS:-4096}

17. Known limitations & follow-ups

1. Chunk tail dropped. Documents over ~480K chars (8 × 60K) lose their tail to the LLM, even though canonicalText supplies up to ~1M chars. Consider raising MAX_CHUNKS or summarizing the remainder.
2. memoryCount not in bulk path. It is populated only in single-entity setFields, not setFieldsInBulk — list endpoints requesting memoryCount may return it unset.
3. processingError is content-only. The file entity flips to Failed with no error string; clients must read the content snapshot for the reason.
4. No checksum short-circuit. Identical re-uploads pay the full text + LLM cost; the captured checksum is stored but never compared.
5. Wholesale reprocess. Pills are hard-deleted and regenerated with new UUIDs/FQNs each run — no continuity/versioning for an unchanged pill.
6. Bedrock client never closed. BedrockCompletionClient is AutoCloseable but LLMClientHolder never calls close(); a re-initialize would leak the prior SDK client. Low impact (single startup init).
7. Factory ignores enabled. Only the holder gates on it; calling the factory directly with enabled=false + a real provider would still build a live client.
8. MCP authorization is coarse. A single global VIEW_ALL + ES visibility=Shared filter, with no per-pill owner re-check.
9. Unsupported status is in the enum but never written by the service (passthrough from the text extractor only).
10. Per-language ES index files are byte-identical — the jp/ru/zh sourceFile mappings carry no CJK/Russian-specific analyzers.

18. File touch-list

New — backend

• openmetadata-spec/.../json/schema/configuration/llmConfiguration.json
• openmetadata-service/.../service/llm/ — LLMCompletionClient, LLMCompletionClientFactory, LLMClientHolder, LLMCompletionException, KnowledgePill, NoopCompletionClient, OpenAICompletionClient, AnthropicCompletionClient, BedrockCompletionClient, GoogleCompletionClient
• openmetadata-service/.../service/drive/ContextMemoryExtractor.java
• openmetadata-mcp/.../tools/SearchCompanyContextTool.java, GetCompanyContextTool.java

New — frontend

• .../components/.../ContextCenter/DocumentStatusBadge/ (component, interface, test)

Modified — schema

• entity/context/contextMemory.json, api/context/createContextMemory.json, entity/data/contextFile.json (+ generated TS mirrors)

Modified — backend

• OpenMetadataApplication.java, OpenMetadataApplicationConfig.java, conf/openmetadata.yaml
• service/drive/ContextFileExtractionService.java → renamed ContextFileProcessingService.java
• resources/drive/ContextFileResource.java
• jdbi3/ContextMemoryRepository.java, jdbi3/ContextFileRepository.java, jdbi3/SystemRepository.java, attachments/AssetServiceFactory.java
• resources/context/ContextMemoryMapper.java
• search/indexes/ContextMemoryIndex.java, search/vector/ContextMemoryBodyTextContributor.java, elasticsearch/{en,jp,ru,zh}/context_memory_search_index.json
• openmetadata-mcp/.../tools/DefaultToolContext.java, openmetadata-mcp/.../resources/json/data/mcp/tools.json

Modified — frontend

• .../DocumentsView/DocumentsView.component.tsx, locale files (17), generated TS

Tests — across openmetadata-service, openmetadata-mcp, openmetadata-integration-tests, and UI (see §15).

Codegen after schema edits: make generate, mvn spotless:apply, UI checkstyle for generated TS, npx tsc --noEmit.

---

19. Follow-on: AISettings + Memory Agent (designed)

Status: Designed, not yet built. Full design: docs/superpowers/specs/2026-06-18-ai-settings-memory-agent-design.md. Built on branch pmbrull/ottawa. Two capabilities layered on top of the pill pipeline above.

19.1 AISettings (config plane)

A SearchSettings-style settings entity (configuration/aiSettings.json, SettingsType.AI_SETTINGS, seeded + merged via SettingsCache / a new AISettingsHandler, read/written through the generic SystemResource GET/PUT/reset, UI page under Preferences → AI). Shape:

• master enabled kill-switch (gates extraction and the agent)
• memoryExtraction.{fromFiles, fromPages}
• memoryAgent.{enabled, deriveGlossaryTerms, deriveMetrics, deletionPolicy} (deletionPolicy ∈ cascade | orphan | deprecate, default cascade)
• prompts.{memoryExtraction, memoryAgent}.systemPrompt

Deltas to this doc:

• The hardcoded ContextMemoryExtractor.SYSTEM_PROMPT (§8) moves to AISettings.prompts.memoryExtraction.systemPrompt — seed JSON carries the default, the code constant is kept only as the cache-miss fallback, and the extractor reads it at run time. This addresses the un-tunable-prompt gap.
• File extraction (§7.2 submit) and page extraction now gate additionally on AISettings (enabled && memoryExtraction.fromFiles / fromPages), not only on LLMClientHolder.isEnabled().

19.2 Memory Agent (memory → ontology)

On ContextMemory create/update, an async, throttled agent reconciles the memory against the existing instance ontology and derives Glossary Terms / Metrics — reusing what already covers the memory, creating only what is missing.

• New DERIVED_FROM relationship (appended last in entityRelationship.json; derivedEntity → sourceMemory).
• MemoryExtractor (pure derive) + MemoryReconciler (owns writes) + MemoryProcessingEngine (orchestrator) in the drive package — siblings to this pill pipeline, same hash-gate / throttle / reconcile idioms — triggered from new ContextMemoryRepository.postCreate / postUpdate hooks.
• Per memory, two independent axes (Term, Metric), each: REUSE (RELATED_TO edge to an existing entity, not owned) / CREATE (provider=automation, DERIVED_FROM edge, agent-owned; mints a glossary when none fits) / SKIP. Grounding is top-K search over the existing term/metric indexes (no full-ontology load).
• Ownership lifecycle: adopt-on-touch flips automation→user on a human PATCH (mirrors the pills Manual-guard); deletionPolicy governs memory-delete cleanup of owned entities (cascade hard-deletes, orphan releases, deprecate flags). Agent writes as a dedicated memory-bot.
• Schema: new ContextMemory.memoryStats (hash-gate + derived/reused counts); new provider field on Metric (§6.1) for uniform ownership across Terms/Metrics/Glossaries.
• Renders the full File → Memory → Term/Metric provenance chain in the UI.