chore: import upstream snapshot with attribution
Integration Tests - MySQL + Elasticsearch / Detect Changes (push) Has been cancelled
Integration Tests - MySQL + Elasticsearch / integration-tests-mysql-elasticsearch (push) Has been cancelled
Integration Tests - PostgreSQL + Elasticsearch + Redis / Detect Changes (push) Has been cancelled
Integration Tests - PostgreSQL + Elasticsearch + Redis / integration-tests-postgres-elasticsearch-redis (push) Has been cancelled
Integration Tests - PostgreSQL + OpenSearch / Detect Changes (push) Has been cancelled
Integration Tests - PostgreSQL + OpenSearch / integration-tests-postgres-opensearch (push) Has been cancelled
Java Checkstyle / java-checkstyle (push) Has been cancelled
Maven Collate Tests / maven-collate-ci (push) Has been cancelled
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests-status (push) Has been cancelled
Publish Package to Maven Central Repository / publish-maven-packages (push) Has been cancelled
OpenMetadata Service Unit Tests / Detect Changes (push) Has been cancelled
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests (push) Has been cancelled
OpenMetadata Service Unit Tests / k8s_operator-unit-tests (push) Has been cancelled
Integration Tests - MySQL + Elasticsearch / Detect Changes (push) Has been cancelled
Integration Tests - MySQL + Elasticsearch / integration-tests-mysql-elasticsearch (push) Has been cancelled
Integration Tests - PostgreSQL + Elasticsearch + Redis / Detect Changes (push) Has been cancelled
Integration Tests - PostgreSQL + Elasticsearch + Redis / integration-tests-postgres-elasticsearch-redis (push) Has been cancelled
Integration Tests - PostgreSQL + OpenSearch / Detect Changes (push) Has been cancelled
Integration Tests - PostgreSQL + OpenSearch / integration-tests-postgres-opensearch (push) Has been cancelled
Java Checkstyle / java-checkstyle (push) Has been cancelled
Maven Collate Tests / maven-collate-ci (push) Has been cancelled
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests-status (push) Has been cancelled
Publish Package to Maven Central Repository / publish-maven-packages (push) Has been cancelled
OpenMetadata Service Unit Tests / Detect Changes (push) Has been cancelled
OpenMetadata Service Unit Tests / openmetadata-service-unit-tests (push) Has been cancelled
OpenMetadata Service Unit Tests / k8s_operator-unit-tests (push) Has been cancelled
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,82 @@
|
||||
# Kubernetes Integration Tests
|
||||
|
||||
## Overview
|
||||
|
||||
OpenMetadata has two types of Kubernetes integration tests:
|
||||
|
||||
1. **K8sIngestionPipelineResourceIT** - Tests ingestion pipelines using native Kubernetes Jobs/CronJobs
|
||||
2. **K8sOMJobOperatorIT** - Tests the custom OMJob operator (requires operator deployment)
|
||||
|
||||
## Key Configuration
|
||||
|
||||
TestSuiteBootstrap automatically configures the K8s pipeline client with:
|
||||
- `useOMJobOperator=false` - Uses native Kubernetes Jobs and CronJobs instead of custom OMJob CRDs
|
||||
|
||||
This means all K8s tests use standard Kubernetes resources by default, no custom operators needed!
|
||||
|
||||
## Running K8s Tests
|
||||
|
||||
### K8sIngestionPipelineResourceIT (Native Jobs/CronJobs)
|
||||
|
||||
This test uses standard Kubernetes resources (Jobs and CronJobs):
|
||||
|
||||
```bash
|
||||
# Run with K8s enabled
|
||||
ENABLE_K8S_TESTS=true mvn test -pl :openmetadata-integration-tests -Dtest=K8sIngestionPipelineResourceIT
|
||||
|
||||
# Or using system property
|
||||
mvn test -pl :openmetadata-integration-tests -DENABLE_K8S_TESTS=true -Dtest=K8sIngestionPipelineResourceIT
|
||||
```
|
||||
|
||||
The test will:
|
||||
- Start K3s using TestContainers
|
||||
- Use native Kubernetes Jobs and CronJobs (not custom CRDs)
|
||||
- Test pipeline deployment, triggering, deletion, etc.
|
||||
|
||||
### K8sOMJobOperatorIT (Custom OMJob Operator)
|
||||
|
||||
This test is for the custom OMJob operator and is disabled by default:
|
||||
- Has `@Disabled` annotation
|
||||
- Requires the OMJob operator to be deployed in the cluster
|
||||
- Tests custom CRD functionality
|
||||
|
||||
## What Happens When K8s is Enabled
|
||||
|
||||
1. TestSuiteBootstrap starts a K3s container using TestContainers
|
||||
2. Creates namespace `openmetadata-pipelines`
|
||||
3. Configures the pipeline service client with K3s kubeconfig
|
||||
4. **Sets `useOMJobOperator=false` by default** (native Jobs/CronJobs)
|
||||
5. OpenMetadata server can now deploy pipelines to K8s
|
||||
|
||||
## Requirements
|
||||
|
||||
- Docker must be running
|
||||
- TestContainers will automatically pull the K3s image
|
||||
- Tests take ~30-60 seconds longer due to K3s startup
|
||||
|
||||
## Test Output
|
||||
|
||||
When K8s is enabled, you'll see:
|
||||
```
|
||||
K8s tests enabled: true
|
||||
Starting K3s container with image: rancher/k3s:v1.28.5-k3s1
|
||||
K3s container started
|
||||
K8s pipeline service client configured and ready
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Tests skip with "K8s tests disabled"
|
||||
- Set `ENABLE_K8S_TESTS=true` environment variable
|
||||
- The K8s-only integration tests check the explicit `ENABLE_K8S_TESTS` flag before running
|
||||
- When enabled, each K8s test class calls `TestSuiteBootstrap.setupK8s()` to initialize its backend
|
||||
|
||||
### NullPointerException for pipelineServiceClient
|
||||
- This means the OpenMetadata app started before the K8s pipeline client was applied to the live
|
||||
ingestion resource
|
||||
- `TestSuiteBootstrap.setupK8s()` now refreshes the registered `IngestionPipelineResource` when
|
||||
K8s is enabled on-demand
|
||||
|
||||
### Tests timeout
|
||||
- K3s container startup can take 30-60 seconds
|
||||
- Ensure Docker has sufficient resources allocated
|
||||
@@ -0,0 +1,195 @@
|
||||
# OpenMetadata Integration Tests
|
||||
|
||||
This module contains SDK-based integration tests that run against a real OpenMetadata server using Testcontainers. Tests execute in parallel and are isolated using `TestNamespace`.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Run all tests with MySQL + Elasticsearch (default)
|
||||
mvn test -pl :openmetadata-integration-tests
|
||||
|
||||
# Run with PostgreSQL + OpenSearch
|
||||
mvn test -pl :openmetadata-integration-tests -Ppostgres-opensearch
|
||||
|
||||
# Run a specific test
|
||||
mvn test -pl :openmetadata-integration-tests -Dtest="TableResourceIT"
|
||||
```
|
||||
|
||||
## Available Profiles
|
||||
|
||||
| Profile | Database | Search Engine |
|
||||
|---------|----------|---------------|
|
||||
| `mysql-elasticsearch` (default) | MySQL 8.3.0 | Elasticsearch 8.11.4 |
|
||||
| `postgres-opensearch` | PostgreSQL 15 | OpenSearch 2.19.0 |
|
||||
| `postgres-elasticsearch` | PostgreSQL 15 | Elasticsearch 8.11.4 |
|
||||
| `mysql-opensearch` | MySQL 8.3.0 | OpenSearch 2.19.0 |
|
||||
|
||||
## Writing a New Integration Test
|
||||
|
||||
### 1. Create the Test Class
|
||||
|
||||
Extend `BaseEntityIT` for entity CRUD tests or create a standalone test class:
|
||||
|
||||
```java
|
||||
package org.openmetadata.it.tests;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertNotNull;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.openmetadata.it.util.TestNamespace;
|
||||
import org.openmetadata.schema.api.data.CreateTable;
|
||||
import org.openmetadata.schema.entity.data.Table;
|
||||
|
||||
public class MyFeatureIT extends BaseEntityIT<Table, CreateTable> {
|
||||
|
||||
@Override
|
||||
protected String getEntityType() {
|
||||
return "table";
|
||||
}
|
||||
|
||||
@Override
|
||||
protected Table createEntity(CreateTable request) {
|
||||
return SdkClients.adminClient().tables().create(request);
|
||||
}
|
||||
|
||||
@Override
|
||||
protected CreateTable createRequest(String name, TestNamespace ns) {
|
||||
return new CreateTable()
|
||||
.withName(name)
|
||||
.withDatabaseSchema(SharedEntities.getSchema().getFullyQualifiedName())
|
||||
.withColumns(List.of(new Column().withName("id").withDataType(ColumnDataType.INT)));
|
||||
}
|
||||
|
||||
@Test
|
||||
void myCustomTest(TestNamespace ns) throws Exception {
|
||||
CreateTable request = createRequest(ns.prefix("myTable"), ns);
|
||||
Table table = createEntity(request);
|
||||
|
||||
assertNotNull(table.getId());
|
||||
assertEquals(request.getName(), table.getName());
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Key Concepts
|
||||
|
||||
#### TestNamespace
|
||||
|
||||
Every test method receives a `TestNamespace` parameter that provides unique prefixes for entity names:
|
||||
|
||||
```java
|
||||
@Test
|
||||
void myTest(TestNamespace ns) {
|
||||
String uniqueName = ns.prefix("myEntity"); // e.g., "abc123_myEntity"
|
||||
}
|
||||
```
|
||||
|
||||
This ensures tests don't conflict when running in parallel.
|
||||
|
||||
#### SdkClients
|
||||
|
||||
Get pre-configured SDK clients for different users:
|
||||
|
||||
```java
|
||||
OpenMetadataClient adminClient = SdkClients.adminClient();
|
||||
OpenMetadataClient user1Client = SdkClients.user1Client();
|
||||
OpenMetadataClient botClient = SdkClients.botClient();
|
||||
```
|
||||
|
||||
#### SharedEntities
|
||||
|
||||
Access pre-created entities for tests:
|
||||
|
||||
```java
|
||||
DatabaseService service = SharedEntities.getService();
|
||||
Database database = SharedEntities.getDatabase();
|
||||
DatabaseSchema schema = SharedEntities.getSchema();
|
||||
User adminUser = SharedEntities.getAdminUser();
|
||||
```
|
||||
|
||||
### 3. BaseEntityIT Features
|
||||
|
||||
When extending `BaseEntityIT`, you get these tests automatically:
|
||||
|
||||
| Test | Description |
|
||||
|------|-------------|
|
||||
| `post_entityCreate_200` | Create entity successfully |
|
||||
| `get_entity_200_OK` | Get entity by ID |
|
||||
| `get_entityByName_200` | Get entity by FQN |
|
||||
| `get_entityNotFound_404` | Get non-existent entity |
|
||||
| `put_entityCreate_200` | Create via PUT |
|
||||
| `patch_entityAttributes_200` | Patch entity attributes |
|
||||
| `delete_entityAsAdmin_200` | Delete entity |
|
||||
| `get_entityListWithPagination_200` | List with pagination |
|
||||
| `test_sdkCRUDOperations` | Full CRUD via SDK |
|
||||
| ... and 30+ more |
|
||||
|
||||
### 4. Controlling Test Behavior
|
||||
|
||||
Use flags to customize which inherited tests run:
|
||||
|
||||
```java
|
||||
public class MyEntityIT extends BaseEntityIT<MyEntity, CreateMyEntity> {
|
||||
{
|
||||
supportsPatch = true; // Enable PATCH tests
|
||||
supportsTags = true; // Enable tag tests
|
||||
supportsOwner = true; // Enable owner tests
|
||||
supportsSearchIndex = true; // Enable search tests
|
||||
supportsDomains = true; // Enable domain tests
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 5. Best Practices
|
||||
|
||||
1. **Use `TestNamespace.prefix()`** for all entity names to ensure uniqueness
|
||||
2. **Don't clean up entities** - TestNamespace isolation handles this
|
||||
3. **Use specific imports** - No wildcard imports (`import static ....*`)
|
||||
4. **Keep tests independent** - Don't rely on order of execution
|
||||
5. **Use Awaitility for async operations** - Not `Thread.sleep()`
|
||||
|
||||
```java
|
||||
Awaitility.await()
|
||||
.atMost(Duration.ofSeconds(30))
|
||||
.pollInterval(Duration.ofMillis(500))
|
||||
.until(() -> someCondition());
|
||||
```
|
||||
|
||||
6. **Avoid single-line comments** - Write self-documenting code
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
openmetadata-integration-tests/
|
||||
├── src/test/java/org/openmetadata/it/
|
||||
│ ├── auth/ # JWT token generation
|
||||
│ ├── env/ # Test infrastructure (TestSuiteBootstrap)
|
||||
│ ├── factories/ # Entity factory classes
|
||||
│ ├── tests/ # Integration test classes
|
||||
│ └── util/ # Utilities (SdkClients, TestNamespace)
|
||||
└── src/test/resources/
|
||||
├── openmetadata-secure-test.yaml # Test config
|
||||
└── *.der # JWT keys
|
||||
```
|
||||
|
||||
## Test Infrastructure
|
||||
|
||||
Tests use `TestSuiteBootstrap` (a JUnit `LauncherSessionListener`) that:
|
||||
|
||||
1. Starts database container (MySQL or PostgreSQL)
|
||||
2. Starts search container (Elasticsearch or OpenSearch)
|
||||
3. Starts Fuseki SPARQL container (for RDF tests)
|
||||
4. Starts the OpenMetadata application
|
||||
5. Initializes `SharedEntities`
|
||||
|
||||
All containers are started **once** per test run and shared across all tests.
|
||||
|
||||
## Running in CI
|
||||
|
||||
GitHub workflows run these tests on every PR:
|
||||
|
||||
- `integration-tests-mysql-elasticsearch.yml` - MySQL + Elasticsearch
|
||||
- `integration-tests-postgres-opensearch.yml` - PostgreSQL + OpenSearch
|
||||
|
||||
Tests require the "safe to test" label on PRs.
|
||||
@@ -0,0 +1,244 @@
|
||||
# Test Migration Tracker: openmetadata-service → openmetadata-integration-tests
|
||||
|
||||
**Last Updated**: 2025-12-26
|
||||
**Status**: All Tests Passing - Migration Complete
|
||||
**Branch**: faster_tests_2
|
||||
|
||||
## Recent Progress (2025-12-26)
|
||||
|
||||
### MySQL Deadlock Fix ✅
|
||||
- Fixed MySQL deadlock issue in `TagUsageDAO.applyTagsBatchInternal`
|
||||
- Changed from `INSERT IGNORE` to `INSERT ... ON DUPLICATE KEY UPDATE`
|
||||
- PostgreSQL unchanged (`ON CONFLICT DO NOTHING` - no deadlock issue)
|
||||
|
||||
### Flaky Test Fixes ✅
|
||||
- **AppsResourceIT**: Changed to `SAME_THREAD` execution mode
|
||||
- Multiple tests trigger `SearchIndexingApplication` which is a shared resource
|
||||
- Added `waitForAppJobCompletion()` to tests that trigger apps
|
||||
- **TableResourceIT.test_multipleDomainInheritance**: Used Awaitility for search index wait
|
||||
- Replaced `Thread.sleep(2000)` with proper Awaitility polling
|
||||
- Waits up to 30 seconds for table to appear in search index
|
||||
|
||||
### Maven Profiles Added ✅
|
||||
- `mysql-elasticsearch` (default)
|
||||
- `postgres-opensearch`
|
||||
- `postgres-elasticsearch`
|
||||
- `mysql-opensearch`
|
||||
|
||||
### GitHub Workflows Created ✅
|
||||
- `integration-tests-mysql-elasticsearch.yml`
|
||||
- `integration-tests-postgres-opensearch.yml`
|
||||
|
||||
### Code Cleanup ✅
|
||||
- Removed wildcard imports from 85+ files
|
||||
- Added comprehensive README.md documentation
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
| Metric | Count |
|
||||
|--------|-------|
|
||||
| Total Test Classes | 91 |
|
||||
| Total @Test Methods | ~2,100 |
|
||||
| Base Class Tests (BaseEntityIT) | 119 |
|
||||
|
||||
---
|
||||
|
||||
## Migration Status By Entity Class
|
||||
|
||||
### Legend
|
||||
- ✅ **Complete**: All tests migrated and passing
|
||||
- 🔄 **Partial**: Some tests need attention
|
||||
- ⚠️ **Stub**: Only BaseEntityIT inherited tests
|
||||
|
||||
### Entity Resource Tests (Sorted by Test Count)
|
||||
|
||||
| Target Class | Tests | Status |
|
||||
|--------------|-------|--------|
|
||||
| BaseEntityIT | 119 | ✅ |
|
||||
| TestCaseResourceIT | 95 | ✅ |
|
||||
| TableResourceIT | 92 | ✅ |
|
||||
| DataContractResourceIT | 92 | ✅ |
|
||||
| GlossaryTermResourceIT | 71 | ✅ |
|
||||
| UserResourceIT | 66 | ✅ |
|
||||
| SearchResourceIT | 55 | ✅ |
|
||||
| NotificationTemplateResourceIT | 52 | ✅ |
|
||||
| EventSubscriptionResourceIT | 48 | ✅ |
|
||||
| FeedResourceIT | 47 | ✅ |
|
||||
| TopicResourceIT | 38 | ✅ |
|
||||
| MlModelResourceIT | 38 | ✅ |
|
||||
| PipelineResourceIT | 36 | ✅ |
|
||||
| GlossaryResourceIT | 35 | ✅ |
|
||||
| ContainerResourceIT | 34 | ✅ |
|
||||
| SystemResourceIT | 33 | ✅ |
|
||||
| TeamResourceIT | 32 | ✅ |
|
||||
| DashboardResourceIT | 29 | ✅ |
|
||||
| SpreadsheetResourceIT | 27 | ✅ |
|
||||
| IngestionPipelineResourceIT | 27 | ✅ |
|
||||
| ColumnResourceIT | 27 | ✅ |
|
||||
| AppsResourceIT | 27 | ✅ |
|
||||
| TestSuiteResourceIT | 26 | ✅ |
|
||||
| LineageResourceIT | 25 | ✅ |
|
||||
| ChartResourceIT | 25 | ✅ |
|
||||
| WorkflowDefinitionResourceIT | 24 | ✅ |
|
||||
| SuggestionsResourceIT | 22 | ✅ |
|
||||
| QueryResourceIT | 21 | ✅ |
|
||||
| OpenLineageResourceIT | 21 | ✅ |
|
||||
| MetricResourceIT | 20 | ✅ |
|
||||
| WorksheetResourceIT | 19 | ✅ |
|
||||
| TagResourceIT | 19 | ✅ |
|
||||
| PromptTemplateResourceIT | 19 | ✅ |
|
||||
| TypeResourceIT | 17 | ✅ |
|
||||
| SearchIndexResourceIT | 17 | ✅ |
|
||||
| AIApplicationResourceIT | 17 | ✅ |
|
||||
| UsageResourceIT | 16 | ✅ |
|
||||
| LLMModelResourceIT | 16 | ✅ |
|
||||
| DatabaseSchemaResourceIT | 16 | ✅ |
|
||||
| WorkflowResourceIT | 15 | ✅ |
|
||||
| DirectoryResourceIT | 15 | ✅ |
|
||||
| StoredProcedureResourceIT | 14 | ✅ |
|
||||
| PermissionsResourceIT | 14 | ✅ |
|
||||
| FileResourceIT | 14 | ✅ |
|
||||
| EntityProfileResourceIT | 14 | ✅ |
|
||||
| DatabaseResourceIT | 14 | ✅ |
|
||||
| PolicyResourceIT | 13 | ✅ |
|
||||
| DomainResourceIT | 13 | ✅ |
|
||||
| DashboardDataModelResourceIT | 13 | ✅ |
|
||||
| AIGovernancePolicyResourceIT | 13 | ✅ |
|
||||
| DatabaseServiceResourceIT | 12 | ✅ |
|
||||
| ClassificationResourceIT | 12 | ✅ |
|
||||
| AppMarketPlaceResourceIT | 12 | ✅ |
|
||||
| SecurityServiceResourceIT | 11 | ✅ |
|
||||
| RoleResourceIT | 11 | ✅ |
|
||||
| PersonaResourceIT | 11 | ✅ |
|
||||
| MetadataServiceResourceIT | 11 | ✅ |
|
||||
| LLMServiceResourceIT | 11 | ✅ |
|
||||
| KpiResourceIT | 11 | ✅ |
|
||||
| DriveServiceResourceIT | 11 | ✅ |
|
||||
| DocStoreResourceIT | 11 | ✅ |
|
||||
| DataProductResourceIT | 11 | ✅ |
|
||||
| DataInsightChartResourceIT | 11 | ✅ |
|
||||
| DataContractPermissionIT | 11 | ✅ |
|
||||
| ConfigResourceIT | 11 | ✅ |
|
||||
| ChangeEventParserResourceIT | 10 | ✅ |
|
||||
| APIEndpointResourceIT | 10 | ✅ |
|
||||
| K8sIngestionPipelineResourceIT | 9 | ✅ |
|
||||
| IngestionPipelineLogStreamingResourceIT | 9 | ✅ |
|
||||
| APICollectionResourceIT | 9 | ✅ |
|
||||
| UserMetricsResourceIT | 8 | ✅ |
|
||||
| TestConnectionDefinitionResourceIT | 8 | ✅ |
|
||||
| SearchServiceResourceIT | 8 | ✅ |
|
||||
| DashboardServiceResourceIT | 8 | ✅ |
|
||||
| AlertsRuleEvaluatorResourceIT | 8 | ✅ |
|
||||
| WebAnalyticEventResourceIT | 7 | ✅ |
|
||||
| MessagingServiceResourceIT | 7 | ✅ |
|
||||
| APIServiceResourceIT | 7 | ✅ |
|
||||
| StorageServiceResourceIT | 6 | ✅ |
|
||||
| ReportDataResourceIT | 6 | ✅ |
|
||||
| PipelineServiceResourceIT | 6 | ✅ |
|
||||
| MlModelServiceResourceIT | 6 | ✅ |
|
||||
| TestDefinitionResourceIT | 5 | ✅ |
|
||||
| BotResourceIT | 5 | ✅ |
|
||||
| AgentExecutionResourceIT | 5 | ✅ |
|
||||
| RdfResourceIT | 4 | ✅ |
|
||||
| PrometheusResourceIT | 3 | ✅ |
|
||||
| PaginationIT | 1 | ✅ |
|
||||
| DatabaseSmokeIT | 1 | ✅ |
|
||||
| DatabaseHierarchyIT | 1 | ✅ |
|
||||
| BaseServiceIT | 1 | ✅ |
|
||||
|
||||
---
|
||||
|
||||
## Test Infrastructure
|
||||
|
||||
### Available Profiles
|
||||
|
||||
| Profile | Database | Search Engine |
|
||||
|---------|----------|---------------|
|
||||
| `mysql-elasticsearch` (default) | MySQL 8.3.0 | Elasticsearch 8.11.4 |
|
||||
| `postgres-opensearch` | PostgreSQL 15 | OpenSearch 2.19.0 |
|
||||
| `postgres-elasticsearch` | PostgreSQL 15 | Elasticsearch 8.11.4 |
|
||||
| `mysql-opensearch` | MySQL 8.3.0 | OpenSearch 2.19.0 |
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
# Run all tests with MySQL + Elasticsearch (default)
|
||||
mvn test -pl :openmetadata-integration-tests
|
||||
|
||||
# Run with PostgreSQL + OpenSearch
|
||||
mvn test -pl :openmetadata-integration-tests -Ppostgres-opensearch
|
||||
|
||||
# Run a specific test
|
||||
mvn test -pl :openmetadata-integration-tests -Dtest="TableResourceIT"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## SDK Fluent APIs Added
|
||||
|
||||
The following fluent API classes were added to support the integration tests:
|
||||
|
||||
| SDK Class | Location |
|
||||
|-----------|----------|
|
||||
| `Columns` | `openmetadata-sdk/.../fluent/Columns.java` |
|
||||
| `DataContracts` | `openmetadata-sdk/.../fluent/DataContracts.java` |
|
||||
| `TestCases` | `openmetadata-sdk/.../fluent/TestCases.java` |
|
||||
| `Usage` | `openmetadata-sdk/.../fluent/Usage.java` |
|
||||
|
||||
### SDK Services Added
|
||||
|
||||
| Service | Purpose |
|
||||
|---------|---------|
|
||||
| `TestCaseResolutionStatusService` | Test case resolution status operations |
|
||||
| `TestCaseResultService` | Test case result operations |
|
||||
| `DataContractService` | Data contract CRUD + bulk operations |
|
||||
|
||||
---
|
||||
|
||||
## Key Files
|
||||
|
||||
- `README.md` - Comprehensive documentation on writing tests
|
||||
- `BaseEntityIT.java` - Base class with 119 inherited tests
|
||||
- `TestSuiteBootstrap.java` - Test infrastructure (Testcontainers)
|
||||
- `TestNamespace.java` - Test isolation utility
|
||||
- `SdkClients.java` - Pre-configured SDK clients
|
||||
|
||||
---
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
GitHub workflows run on every PR:
|
||||
|
||||
- `integration-tests-mysql-elasticsearch.yml` - MySQL + Elasticsearch
|
||||
- `integration-tests-postgres-opensearch.yml` - PostgreSQL + OpenSearch
|
||||
|
||||
Tests require the "safe to test" label on PRs (uses `pull_request_target`).
|
||||
|
||||
---
|
||||
|
||||
## Performance
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Full test suite | ~20 minutes locally |
|
||||
| Parallel execution | Yes (JUnit 5 parallel) |
|
||||
| Test isolation | TestNamespace prefixes |
|
||||
| Container startup | ~30 seconds |
|
||||
|
||||
---
|
||||
|
||||
## Recent Fixes
|
||||
|
||||
### MySQL Deadlock Fix (2025-12-26)
|
||||
Changed `TagUsageDAO.applyTagsBatchInternal` from:
|
||||
```sql
|
||||
INSERT IGNORE INTO tag_usage ...
|
||||
```
|
||||
To:
|
||||
```sql
|
||||
INSERT INTO tag_usage ... ON DUPLICATE KEY UPDATE ...
|
||||
```
|
||||
|
||||
This prevents deadlocks from MySQL's gap locking behavior during concurrent tag updates.
|
||||
@@ -0,0 +1,137 @@
|
||||
# UI test conventions
|
||||
|
||||
Read this before writing or porting any `*UIIT.java`. It exists so the suite stays
|
||||
consistent as we migrate 258 specs.
|
||||
|
||||
## Layering — strict, top-only depends down
|
||||
|
||||
```
|
||||
playwright.scenarios.<domain>.*UIIT.java — tests
|
||||
↓
|
||||
playwright.ui.pages.*Page.java — Page Objects (locators + actions)
|
||||
↓
|
||||
playwright.ui.{SessionBrowser,UiSession,UiSessionExtension,TraceRecorder}
|
||||
it.auth.{AuthBackend,AuthSession,BasicJwtBackend,OidcBackend,…}
|
||||
↓
|
||||
it.server.*, it.search.* — server lifecycle + SDK helpers
|
||||
```
|
||||
|
||||
**Rule:** prefer keeping `Locator` / `Page` / `BrowserContext` use inside Page Objects.
|
||||
Page Objects may expose `Locator`-returning accessors when a test legitimately needs to
|
||||
make a Playwright-level assertion on a specific element, and `PageObject.rawPage()` is a
|
||||
documented escape hatch for URL assertions and SSO redirect flows. If a test reaches for
|
||||
Playwright primitives for routine interactions (clicks, typing, navigation), promote the
|
||||
interaction into a Page Object method.
|
||||
|
||||
## Test skeleton
|
||||
|
||||
```java
|
||||
@ExtendWith({UiSessionExtension.class, TestNamespaceExtension.class})
|
||||
class FooUIIT {
|
||||
|
||||
@BeforeAll
|
||||
static void setup() {
|
||||
SdkClients.useFluentApis(UiTestServer.get().sdk());
|
||||
Apps.setDefaultClient(UiTestServer.get().sdk());
|
||||
}
|
||||
|
||||
@Test
|
||||
void scenario(final UiSession ui, final TestNamespace ns) {
|
||||
Table t = TableTestFactory.createSimple(ns); // SDK setup
|
||||
FooPage page = FooPage.open(ui, t.getFullyQualifiedName()); // page object navigation
|
||||
page.doSomething(); // page object action
|
||||
assertThat(page.someResult()).isVisible(); // assertion
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Setup via SDK. Cleanup via `TestNamespace`. Never click through the UI to seed state.
|
||||
|
||||
## Page Objects
|
||||
|
||||
- One class per page, ≤ 200 lines. Split if bigger.
|
||||
- Static `open(...)` factory navigates and returns a loaded instance.
|
||||
- Actions return `this` (or another `PageObject`) for chaining.
|
||||
- Locators: `getByTestId` > `getByRole`/`getByLabel` > `getByPlaceholder` > text > CSS.
|
||||
- Define every selector as `private static final` — no inline literals.
|
||||
- `waitForLoaded()` overridden per page — the readiness signal must be page-specific
|
||||
(a known testid visible, an API response done) not a generic `networkidle` if a better
|
||||
signal exists.
|
||||
|
||||
## Auth
|
||||
|
||||
The whole suite is parameterized by an `AuthBackend`. Pick one with `-Djpw.auth=<name>`
|
||||
(or `JPW_AUTH=<name>`); the same UI tests run unchanged regardless of which is active.
|
||||
|
||||
| `jpw.auth` | OM auth provider | Token acquired by |
|
||||
|---|---|---|
|
||||
| `basic` *(default)* | OM built-in JWT | `JwtAuthProvider` (admin signing key) |
|
||||
| `sso-google-public` | Google + public client (`response_type=id_token`) | mock IdP `/google/token` (password grant) |
|
||||
| `sso-google-confidential` | Google + confidential client (`response_type=code`) | same |
|
||||
| `sso-okta-public` | Okta + public client | mock IdP `/okta/token` |
|
||||
| `sso-okta-confidential` | Okta + confidential client | same |
|
||||
| `sso-custom-oidc-public` | Custom OIDC + public | mock IdP `/custom-oidc/token` |
|
||||
| `sso-custom-oidc-confidential` | Custom OIDC + confidential | same |
|
||||
|
||||
Per-test override (only when the test explicitly drives the sign-in surface and doesn't
|
||||
want a token preloaded):
|
||||
|
||||
```java
|
||||
@Test
|
||||
@NoPreloadAuth
|
||||
void clickingSignInWithGoogleCompletesLogin(UiSession ui) { ... }
|
||||
```
|
||||
|
||||
Tests that only make sense under one backend gate themselves with
|
||||
`AuthAssumptions.onlyWhenBackendIs("sso-google-confidential")`. They skip cleanly when the
|
||||
suite is running under any other backend.
|
||||
|
||||
A daemon `TokenRefresher` keeps `AuthSession.current()` valid across long runs; tests
|
||||
read the current token at `@BeforeEach`, so refresh is transparent.
|
||||
|
||||
## Parallelism
|
||||
|
||||
- Per-method parallel by default; classes serial. Configured in pom failsafe.
|
||||
- Tests that mutate global state must be tagged:
|
||||
- `@ResourceLock("GLOBAL_SETTINGS")` — settings, themes, login config
|
||||
- `@ResourceLock("SEARCH_INDEX_APP")` — re-triggers the global reindex app
|
||||
- `@ResourceLock("APPS")` — install/uninstall apps
|
||||
- `@ResourceLock("ALERTS")` — global alert config
|
||||
- `@Execution(SAME_THREAD)` — inherently sequential UI flows (Tour, SSO renewal)
|
||||
- Default: assume parallel-safe. `TestNamespace` keeps entity names unique.
|
||||
|
||||
## Java code rules (recap from project CLAUDE.md)
|
||||
|
||||
- Methods ≤ 15 lines, ≤ 3 nesting, ≤ 5 params, ≤ 10 cyclomatic.
|
||||
- `final` on every parameter and local that doesn't change.
|
||||
- No comments stating *what* — only *why*. Name the code so it reads.
|
||||
- No magic strings in `equals`/`contains`/`switch` — define a constant or an enum.
|
||||
- No deep `else if` chains — refactor to `switch`, `Map` dispatch, or named predicate.
|
||||
- No empty catches, no `printStackTrace`, no `catch(Exception)` for control flow.
|
||||
- `mvn spotless:apply` before every commit.
|
||||
|
||||
## Headed debugging
|
||||
|
||||
Knobs (all available as env var or `-D<name>=...` system property):
|
||||
|
||||
| Knob | Effect |
|
||||
|---|---|
|
||||
| `PW_HEADED=true` | Visible Chromium |
|
||||
| `PW_SLOWMO=<ms>` | Inter-action delay; defaults to 250ms in headed, 0 in headless |
|
||||
| `PW_VIDEO=true` | Records every test, saves to `target/playwright-videos/<class>-<test>-<ts>.webm` |
|
||||
|
||||
Trace zips for **failed** runs are saved to `target/playwright-traces/`. Replay any trace:
|
||||
```
|
||||
npx playwright show-trace target/playwright-traces/trace-<class>-<test>-<ts>.zip
|
||||
```
|
||||
|
||||
Typical local debug recipe — slow it down enough to follow, capture video for review:
|
||||
```
|
||||
PW_HEADED=true PW_SLOWMO=500 PW_VIDEO=true mvn verify -P ui-it \
|
||||
-pl :openmetadata-integration-tests -Dit.test='SimpleReindexTriggerUIIT'
|
||||
```
|
||||
|
||||
## When in doubt
|
||||
|
||||
- Read `REINDEX_TEST_PLAN.md` at the module root for the reindex scenario coverage map.
|
||||
- Read `SimpleReindexTriggerUIIT` as the canonical reference test.
|
||||
File diff suppressed because it is too large
Load Diff
+272
@@ -0,0 +1,272 @@
|
||||
/*
|
||||
* Copyright 2026 Collate
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.attachments;
|
||||
|
||||
import static jakarta.ws.rs.core.Response.Status.CREATED;
|
||||
import static org.awaitility.Awaitility.await;
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertNotNull;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
|
||||
import com.fasterxml.jackson.core.type.TypeReference;
|
||||
import io.dropwizard.jackson.Jackson;
|
||||
import io.dropwizard.jersey.jackson.JacksonFeature;
|
||||
import jakarta.ws.rs.client.Client;
|
||||
import jakarta.ws.rs.client.ClientBuilder;
|
||||
import jakarta.ws.rs.client.Entity;
|
||||
import jakarta.ws.rs.client.WebTarget;
|
||||
import jakarta.ws.rs.core.MediaType;
|
||||
import jakarta.ws.rs.core.MultivaluedHashMap;
|
||||
import jakarta.ws.rs.core.MultivaluedMap;
|
||||
import jakarta.ws.rs.core.Response;
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.Duration;
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
import org.glassfish.jersey.client.ClientProperties;
|
||||
import org.glassfish.jersey.media.multipart.FormDataMultiPart;
|
||||
import org.glassfish.jersey.media.multipart.MultiPartFeature;
|
||||
import org.glassfish.jersey.media.multipart.file.StreamDataBodyPart;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.extension.ExtendWith;
|
||||
import org.openmetadata.schema.api.data.CreateGlossary;
|
||||
import org.openmetadata.schema.attachments.Asset;
|
||||
import org.openmetadata.schema.entity.data.Glossary;
|
||||
import org.openmetadata.schema.utils.JsonUtils;
|
||||
import org.openmetadata.sdk.test.util.RestClient;
|
||||
import org.openmetadata.sdk.test.util.SdkClients;
|
||||
import org.openmetadata.sdk.test.util.TestNamespace;
|
||||
import org.openmetadata.sdk.test.util.TestNamespaceExtension;
|
||||
|
||||
@ExtendWith(TestNamespaceExtension.class)
|
||||
class AttachmentListIT {
|
||||
|
||||
private static final String ATTACHMENTS_PATH = "v1/attachments";
|
||||
private static String serverBaseUrl;
|
||||
private static Client multipartClient;
|
||||
private static WebTarget uploadTarget;
|
||||
|
||||
@BeforeAll
|
||||
static void setup() {
|
||||
String itBaseUrl =
|
||||
System.getProperty(
|
||||
"IT_BASE_URL",
|
||||
System.getenv().getOrDefault("IT_BASE_URL", "http://localhost:8585/api"));
|
||||
serverBaseUrl = itBaseUrl.endsWith("/api") ? itBaseUrl : itBaseUrl + "/api";
|
||||
|
||||
multipartClient = ClientBuilder.newClient();
|
||||
multipartClient.register(MultiPartFeature.class);
|
||||
multipartClient.register(new JacksonFeature(Jackson.newObjectMapper()));
|
||||
|
||||
uploadTarget =
|
||||
multipartClient
|
||||
.target(serverBaseUrl + "/" + ATTACHMENTS_PATH + "/upload")
|
||||
.property(ClientProperties.CONNECT_TIMEOUT, 30000)
|
||||
.property(ClientProperties.READ_TIMEOUT, 30000);
|
||||
}
|
||||
|
||||
@AfterAll
|
||||
static void tearDown() {
|
||||
if (multipartClient != null) {
|
||||
multipartClient.close();
|
||||
multipartClient = null;
|
||||
}
|
||||
}
|
||||
|
||||
private static MultivaluedMap<String, Object> adminAuthHeaders() {
|
||||
String token = SdkClients.getAdminToken();
|
||||
MultivaluedMap<String, Object> headers = new MultivaluedHashMap<>();
|
||||
headers.add("Authorization", "Bearer " + token);
|
||||
return headers;
|
||||
}
|
||||
|
||||
private Glossary createGlossary(RestClient rest, TestNamespace ns, String slug) {
|
||||
try {
|
||||
return rest.create(
|
||||
"v1/glossaries",
|
||||
new CreateGlossary().withName(ns.prefix(slug)).withDescription("attachment IT glossary"),
|
||||
Glossary.class);
|
||||
} catch (Exception e) {
|
||||
throw new AssertionError("Failed to create glossary " + slug, e);
|
||||
}
|
||||
}
|
||||
|
||||
private Asset uploadAttachment(String entityLink, String fileName, String body) {
|
||||
try (FormDataMultiPart multipart = new FormDataMultiPart()) {
|
||||
multipart.field("entityLink", entityLink);
|
||||
multipart.field("assetType", "External");
|
||||
multipart.bodyPart(
|
||||
new StreamDataBodyPart(
|
||||
"file",
|
||||
new ByteArrayInputStream(body.getBytes(StandardCharsets.UTF_8)),
|
||||
fileName,
|
||||
MediaType.APPLICATION_OCTET_STREAM_TYPE));
|
||||
|
||||
Response response =
|
||||
uploadTarget
|
||||
.request()
|
||||
.headers(adminAuthHeaders())
|
||||
.post(Entity.entity(multipart, multipart.getMediaType()));
|
||||
|
||||
String responseBody = response.readEntity(String.class);
|
||||
assertEquals(
|
||||
CREATED.getStatusCode(), response.getStatus(), "Asset upload failed: " + responseBody);
|
||||
return JsonUtils.readValue(responseBody, Asset.class);
|
||||
} catch (Exception e) {
|
||||
throw new AssertionError("Failed to upload attachment " + fileName, e);
|
||||
}
|
||||
}
|
||||
|
||||
private List<Asset> listExternalAttachments(String fqn, String queryString) {
|
||||
RestClient rest = RestClient.admin();
|
||||
String path = ATTACHMENTS_PATH + "/fqn/" + fqn + "/External";
|
||||
if (queryString != null && !queryString.isEmpty()) {
|
||||
path = path + "?" + queryString;
|
||||
}
|
||||
try (Response response = rest.rawGet(path)) {
|
||||
assertEquals(200, response.getStatus(), "Listing attachments failed");
|
||||
String body = response.readEntity(String.class);
|
||||
return JsonUtils.readValue(body, new TypeReference<List<Asset>>() {});
|
||||
}
|
||||
}
|
||||
|
||||
private String entityLink(Glossary glossary) {
|
||||
return "<#E::glossary::" + glossary.getFullyQualifiedName() + ">";
|
||||
}
|
||||
|
||||
private static void awaitClockPast(long timestamp) {
|
||||
await()
|
||||
.pollInterval(Duration.ofMillis(2))
|
||||
.atMost(Duration.ofSeconds(2))
|
||||
.until(() -> System.currentTimeMillis() > timestamp);
|
||||
}
|
||||
|
||||
@Test
|
||||
void testListAttachmentsSortByUpdatedAtDesc(TestNamespace ns) {
|
||||
Glossary glossary = createGlossary(RestClient.admin(), ns, "attach-sort-updated");
|
||||
String link = entityLink(glossary);
|
||||
|
||||
Asset older = uploadAttachment(link, "older.txt", "older");
|
||||
awaitClockPast(older.getUpdatedAt());
|
||||
Asset middle = uploadAttachment(link, "middle.txt", "middle");
|
||||
awaitClockPast(middle.getUpdatedAt());
|
||||
Asset newer = uploadAttachment(link, "newer.txt", "newer");
|
||||
|
||||
List<Asset> assets =
|
||||
listExternalAttachments(
|
||||
glossary.getFullyQualifiedName(), "sortBy=updatedAt&sortOrder=desc");
|
||||
|
||||
List<UUID> ids = assets.stream().map(a -> UUID.fromString(a.getId())).toList();
|
||||
assertEquals(
|
||||
List.of(
|
||||
UUID.fromString(newer.getId()),
|
||||
UUID.fromString(middle.getId()),
|
||||
UUID.fromString(older.getId())),
|
||||
ids,
|
||||
"Expected newest-first ordering");
|
||||
}
|
||||
|
||||
@Test
|
||||
void testListAttachmentsSortByNameAsc(TestNamespace ns) {
|
||||
Glossary glossary = createGlossary(RestClient.admin(), ns, "attach-sort-name");
|
||||
String link = entityLink(glossary);
|
||||
|
||||
Asset zebra = uploadAttachment(link, "zzz.txt", "z");
|
||||
Asset apple = uploadAttachment(link, "aaa.txt", "a");
|
||||
Asset mango = uploadAttachment(link, "mmm.txt", "m");
|
||||
|
||||
List<Asset> assets =
|
||||
listExternalAttachments(glossary.getFullyQualifiedName(), "sortBy=name&sortOrder=asc");
|
||||
|
||||
List<UUID> ids = assets.stream().map(a -> UUID.fromString(a.getId())).toList();
|
||||
assertEquals(
|
||||
List.of(
|
||||
UUID.fromString(apple.getId()),
|
||||
UUID.fromString(mango.getId()),
|
||||
UUID.fromString(zebra.getId())),
|
||||
ids,
|
||||
"Expected name-ascending ordering");
|
||||
}
|
||||
|
||||
@Test
|
||||
void testListAttachmentsCreatedAtAliasesUpdatedAt(TestNamespace ns) {
|
||||
Glossary glossary = createGlossary(RestClient.admin(), ns, "attach-sort-created");
|
||||
String link = entityLink(glossary);
|
||||
|
||||
Asset first = uploadAttachment(link, "first.txt", "1");
|
||||
awaitClockPast(first.getUpdatedAt());
|
||||
Asset second = uploadAttachment(link, "second.txt", "2");
|
||||
|
||||
List<Asset> assets =
|
||||
listExternalAttachments(
|
||||
glossary.getFullyQualifiedName(), "sortBy=createdAt&sortOrder=desc");
|
||||
List<UUID> ids = assets.stream().map(a -> UUID.fromString(a.getId())).toList();
|
||||
assertEquals(
|
||||
List.of(UUID.fromString(second.getId()), UUID.fromString(first.getId())),
|
||||
ids,
|
||||
"createdAt should alias to updatedAt and return newest first");
|
||||
}
|
||||
|
||||
@Test
|
||||
void testListAttachmentsWithLimitAndOffset(TestNamespace ns) {
|
||||
Glossary glossary = createGlossary(RestClient.admin(), ns, "attach-paginate");
|
||||
String link = entityLink(glossary);
|
||||
|
||||
uploadAttachment(link, "alpha.txt", "1");
|
||||
uploadAttachment(link, "bravo.txt", "2");
|
||||
uploadAttachment(link, "charlie.txt", "3");
|
||||
uploadAttachment(link, "delta.txt", "4");
|
||||
|
||||
List<Asset> page =
|
||||
listExternalAttachments(
|
||||
glossary.getFullyQualifiedName(), "sortBy=name&sortOrder=asc&offset=1&limit=2");
|
||||
assertEquals(2, page.size(), "Expected page size 2");
|
||||
assertEquals("bravo.txt", page.get(0).getFileName());
|
||||
assertEquals("charlie.txt", page.get(1).getFileName());
|
||||
}
|
||||
|
||||
@Test
|
||||
void testListAttachmentsRejectsUnknownSortBy(TestNamespace ns) {
|
||||
Glossary glossary = createGlossary(RestClient.admin(), ns, "attach-bad-sort");
|
||||
String link = entityLink(glossary);
|
||||
uploadAttachment(link, "only.txt", "only");
|
||||
|
||||
RestClient rest = RestClient.admin();
|
||||
String path =
|
||||
ATTACHMENTS_PATH
|
||||
+ "/fqn/"
|
||||
+ glossary.getFullyQualifiedName()
|
||||
+ "/External?sortBy=bogusField";
|
||||
try (Response response = rest.rawGet(path)) {
|
||||
assertTrue(
|
||||
response.getStatus() >= 400 && response.getStatus() < 500,
|
||||
"Expected 4xx for unknown sortBy, got " + response.getStatus());
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void testListAttachmentsNoSortReturnsAll(TestNamespace ns) {
|
||||
Glossary glossary = createGlossary(RestClient.admin(), ns, "attach-no-sort");
|
||||
String link = entityLink(glossary);
|
||||
Asset a = uploadAttachment(link, "one.txt", "1");
|
||||
Asset b = uploadAttachment(link, "two.txt", "2");
|
||||
|
||||
List<Asset> assets = listExternalAttachments(glossary.getFullyQualifiedName(), null);
|
||||
assertEquals(2, assets.size(), "Expected both attachments returned without sort/pagination");
|
||||
assertNotNull(a.getId());
|
||||
assertNotNull(b.getId());
|
||||
}
|
||||
}
|
||||
+44
@@ -0,0 +1,44 @@
|
||||
package org.openmetadata.it.auth;
|
||||
|
||||
import com.microsoft.playwright.BrowserContext;
|
||||
import java.util.Locale;
|
||||
|
||||
/**
|
||||
* Builds and applies the JS init-script that seeds the OM UI's
|
||||
* {@code localStorage.app_state} (and the equivalent IndexedDB store) with a token
|
||||
* before any application code runs. This skips the UI login flow regardless of which
|
||||
* {@link AuthBackend} produced the token — Playwright runs this script at every page
|
||||
* navigation, so each request goes straight into the authenticated app.
|
||||
*/
|
||||
final class AppStateInjection {
|
||||
|
||||
private static final String INIT_SCRIPT_TEMPLATE =
|
||||
"""
|
||||
(function() {
|
||||
const APP_STATE_KEY = 'app_state';
|
||||
const stateJson = JSON.stringify({ primary: "%s" });
|
||||
try { localStorage.setItem(APP_STATE_KEY, stateJson); } catch (e) {}
|
||||
try {
|
||||
const open = indexedDB.open('AppDataStore', 1);
|
||||
open.onupgradeneeded = () => {
|
||||
const db = open.result;
|
||||
if (!db.objectStoreNames.contains('keyValueStore')) {
|
||||
db.createObjectStore('keyValueStore');
|
||||
}
|
||||
};
|
||||
open.onsuccess = () => {
|
||||
const db = open.result;
|
||||
const tx = db.transaction(['keyValueStore'], 'readwrite');
|
||||
tx.objectStore('keyValueStore').put(stateJson, APP_STATE_KEY);
|
||||
};
|
||||
} catch (e) {}
|
||||
})();
|
||||
""";
|
||||
|
||||
private AppStateInjection() {}
|
||||
|
||||
static void seed(final BrowserContext context, final String tokenForLocalStorage) {
|
||||
final String escaped = tokenForLocalStorage.replace("\\", "\\\\").replace("\"", "\\\"");
|
||||
context.addInitScript(String.format(Locale.ROOT, INIT_SCRIPT_TEMPLATE, escaped));
|
||||
}
|
||||
}
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
package org.openmetadata.it.auth;
|
||||
|
||||
import static org.junit.jupiter.api.Assumptions.assumeTrue;
|
||||
|
||||
import java.util.Arrays;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
|
||||
/**
|
||||
* Utility for sign-in flow tests that are only meaningful under specific auth backends.
|
||||
*
|
||||
* <p>The whole UI suite can be run under any {@code jpw.auth} backend with a single Maven
|
||||
* command. Auth-mode-specific tests (e.g. a {@code GoogleSsoSignInUIIT} that drives the
|
||||
* Google login button) call {@link #onlyWhenBackendIs} so they skip with an
|
||||
* {@code Assumption} rather than fail when the suite is running under a different
|
||||
* backend.
|
||||
*/
|
||||
public final class AuthAssumptions {
|
||||
|
||||
private AuthAssumptions() {}
|
||||
|
||||
/**
|
||||
* Skip the current test unless the active backend's name matches one of the supplied
|
||||
* names. Names are compared case-insensitively against {@link AuthBackend#name()}.
|
||||
*/
|
||||
public static void onlyWhenBackendIs(final String... backendNames) {
|
||||
final List<String> wanted =
|
||||
Arrays.stream(backendNames).map(name -> name.toLowerCase(Locale.ROOT)).toList();
|
||||
final String actual = AuthSession.backend().name().toLowerCase(Locale.ROOT);
|
||||
assumeTrue(
|
||||
wanted.contains(actual),
|
||||
() ->
|
||||
"Test only runs under jpw.auth in " + wanted + "; current backend is '" + actual + "'");
|
||||
}
|
||||
}
|
||||
+41
@@ -0,0 +1,41 @@
|
||||
package org.openmetadata.it.auth;
|
||||
|
||||
import com.microsoft.playwright.BrowserContext;
|
||||
import java.util.Map;
|
||||
import org.openmetadata.it.server.ServerHandle;
|
||||
import org.openmetadata.it.server.sso.MockOidcServer;
|
||||
import org.openmetadata.it.server.sso.SsoProfile;
|
||||
|
||||
/**
|
||||
* One auth backend = one way OM is configured to authenticate, plus how tokens are
|
||||
* acquired/refreshed/used by tests.
|
||||
*
|
||||
* <p>Implementations are stateless and thread-safe; one instance per JVM.
|
||||
*
|
||||
* <p>Lifecycle inside {@code UiTestServer.get()}:
|
||||
* <ol>
|
||||
* <li>{@link #serverEnv} — env vars OM needs to boot under this backend
|
||||
* <li>OM starts; if {@link #requiresIdp} is true, a {@link MockOidcServer} is also booted
|
||||
* <li>{@link #acquireToken} — get the initial {@link TokenSet} for the suite
|
||||
* <li>{@link TokenRefresher} ticks; calls {@link #refresh} as expiry approaches
|
||||
* <li>Each {@code @Test} starts: {@link #injectIntoBrowser} writes the current token
|
||||
* into a fresh {@link BrowserContext}
|
||||
* </ol>
|
||||
*/
|
||||
public sealed interface AuthBackend permits BasicJwtBackend, OidcBackend {
|
||||
|
||||
String name();
|
||||
|
||||
boolean requiresIdp();
|
||||
|
||||
/** SSO profile used to boot the OM container; {@code null} for non-IdP backends. */
|
||||
SsoProfile ssoProfile();
|
||||
|
||||
Map<String, String> serverEnv(MockOidcServer idp, int omHostPort);
|
||||
|
||||
TokenSet acquireToken(ServerHandle server, MockOidcServer idp);
|
||||
|
||||
TokenSet refresh(TokenSet current, ServerHandle server, MockOidcServer idp);
|
||||
|
||||
void injectIntoBrowser(BrowserContext context, TokenSet tokens);
|
||||
}
|
||||
+54
@@ -0,0 +1,54 @@
|
||||
package org.openmetadata.it.auth;
|
||||
|
||||
import java.util.Locale;
|
||||
import org.openmetadata.it.server.sso.ClientType;
|
||||
import org.openmetadata.it.server.sso.CustomOidcProfile;
|
||||
import org.openmetadata.it.server.sso.GoogleProfile;
|
||||
import org.openmetadata.it.server.sso.OktaProfile;
|
||||
|
||||
/**
|
||||
* Resolves the JVM's active {@link AuthBackend} from the {@code jpw.auth} system property
|
||||
* (or {@code JPW_AUTH} env var). Defaults to {@link BasicJwtBackend} when nothing is set.
|
||||
*
|
||||
* <p>Adding a new provider × client-type combination:
|
||||
* <ol>
|
||||
* <li>Implement {@code SsoProfile} for the new provider (record).
|
||||
* <li>Update {@code SsoProfile}'s {@code permits} clause.
|
||||
* <li>Add a switch case below; the existing {@link OidcBackend} handles all OIDC flows.
|
||||
* </ol>
|
||||
*
|
||||
* <p>Names follow {@code sso-<provider>-<clienttype>} so they're greppable and obvious.
|
||||
*/
|
||||
public final class AuthBackends {
|
||||
|
||||
private AuthBackends() {}
|
||||
|
||||
public static AuthBackend resolve() {
|
||||
final String name = lookup().toLowerCase(Locale.ROOT);
|
||||
return switch (name) {
|
||||
case "", "basic" -> new BasicJwtBackend();
|
||||
case "sso-google-public" -> new OidcBackend(new GoogleProfile(ClientType.PUBLIC));
|
||||
case "sso-google-confidential" -> new OidcBackend(new GoogleProfile(ClientType.CONFIDENTIAL));
|
||||
case "sso-okta-public" -> new OidcBackend(new OktaProfile(ClientType.PUBLIC));
|
||||
case "sso-okta-confidential" -> new OidcBackend(new OktaProfile(ClientType.CONFIDENTIAL));
|
||||
case "sso-custom-oidc-public" -> new OidcBackend(new CustomOidcProfile(ClientType.PUBLIC));
|
||||
case "sso-custom-oidc-confidential" -> new OidcBackend(
|
||||
new CustomOidcProfile(ClientType.CONFIDENTIAL));
|
||||
default -> throw new IllegalArgumentException(
|
||||
"Unknown jpw.auth='"
|
||||
+ name
|
||||
+ "'. Known: basic, sso-google-public, sso-google-confidential, "
|
||||
+ "sso-okta-public, sso-okta-confidential, "
|
||||
+ "sso-custom-oidc-public, sso-custom-oidc-confidential.");
|
||||
};
|
||||
}
|
||||
|
||||
private static String lookup() {
|
||||
final String prop = System.getProperty("jpw.auth");
|
||||
if (prop != null && !prop.isBlank()) {
|
||||
return prop;
|
||||
}
|
||||
final String env = System.getenv("JPW_AUTH");
|
||||
return (env != null && !env.isBlank()) ? env : "";
|
||||
}
|
||||
}
|
||||
+50
@@ -0,0 +1,50 @@
|
||||
package org.openmetadata.it.auth;
|
||||
|
||||
import java.util.Optional;
|
||||
import java.util.concurrent.atomic.AtomicReference;
|
||||
|
||||
/**
|
||||
* JVM-wide holder for the suite's active {@link AuthBackend} and current {@link TokenSet}.
|
||||
*
|
||||
* <p>Tests never construct one — {@code UiTestServer} initializes the singleton once, and
|
||||
* test extensions read {@link #current()} per test to pick up post-refresh tokens.
|
||||
*/
|
||||
public final class AuthSession {
|
||||
|
||||
private static final AtomicReference<AuthBackend> BACKEND = new AtomicReference<>();
|
||||
private static final AtomicReference<TokenSet> CURRENT = new AtomicReference<>();
|
||||
|
||||
private AuthSession() {}
|
||||
|
||||
public static synchronized void initialize(final AuthBackend backend, final TokenSet initial) {
|
||||
BACKEND.set(backend);
|
||||
CURRENT.set(initial);
|
||||
}
|
||||
|
||||
public static AuthBackend backend() {
|
||||
final AuthBackend backend = BACKEND.get();
|
||||
if (backend == null) {
|
||||
throw new IllegalStateException(
|
||||
"AuthSession not initialized — UiTestServer.get() must run first");
|
||||
}
|
||||
return backend;
|
||||
}
|
||||
|
||||
public static TokenSet current() {
|
||||
final TokenSet tokens = CURRENT.get();
|
||||
if (tokens == null) {
|
||||
throw new IllegalStateException(
|
||||
"AuthSession has no token — UiTestServer.get() must run first");
|
||||
}
|
||||
return tokens;
|
||||
}
|
||||
|
||||
public static Optional<TokenSet> currentOptional() {
|
||||
return Optional.ofNullable(CURRENT.get());
|
||||
}
|
||||
|
||||
/** Replace the current token set after a refresh. */
|
||||
public static void update(final TokenSet refreshed) {
|
||||
CURRENT.set(refreshed);
|
||||
}
|
||||
}
|
||||
+69
@@ -0,0 +1,69 @@
|
||||
package org.openmetadata.it.auth;
|
||||
|
||||
import com.microsoft.playwright.BrowserContext;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.Map;
|
||||
import org.openmetadata.it.server.ServerHandle;
|
||||
import org.openmetadata.it.server.sso.MockOidcServer;
|
||||
import org.openmetadata.it.server.sso.SsoProfile;
|
||||
|
||||
/**
|
||||
* Default backend — OM runs with its built-in JWT auth and tests use admin tokens minted
|
||||
* by {@link JwtAuthProvider} (the same provider {@code SdkClients} uses elsewhere).
|
||||
*
|
||||
* <p>"Refresh" is just minting a new token from the same key; no IdP required, no real
|
||||
* OAuth2 flow. The OM UI accepts these tokens because the test harness shares the
|
||||
* provider's signing keys with the server (mounted as {@code public_key.der} /
|
||||
* {@code private_key.der} in {@code ContainerizedServer}).
|
||||
*/
|
||||
public final class BasicJwtBackend implements AuthBackend {
|
||||
|
||||
static final String NAME = "basic";
|
||||
private static final String ADMIN_PRINCIPAL = "admin@open-metadata.org";
|
||||
private static final String[] ADMIN_ROLES = {"admin"};
|
||||
private static final long TOKEN_TTL_SECONDS = 24L * 60 * 60;
|
||||
|
||||
@Override
|
||||
public String name() {
|
||||
return NAME;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean requiresIdp() {
|
||||
return false;
|
||||
}
|
||||
|
||||
@Override
|
||||
public SsoProfile ssoProfile() {
|
||||
return null;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Map<String, String> serverEnv(final MockOidcServer idp, final int omHostPort) {
|
||||
return Map.of();
|
||||
}
|
||||
|
||||
@Override
|
||||
public TokenSet acquireToken(final ServerHandle server, final MockOidcServer idp) {
|
||||
return mint();
|
||||
}
|
||||
|
||||
@Override
|
||||
public TokenSet refresh(
|
||||
final TokenSet current, final ServerHandle server, final MockOidcServer idp) {
|
||||
return mint();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void injectIntoBrowser(final BrowserContext context, final TokenSet tokens) {
|
||||
AppStateInjection.seed(context, tokens.accessToken());
|
||||
}
|
||||
|
||||
private static TokenSet mint() {
|
||||
final String jwt =
|
||||
JwtAuthProvider.tokenFor(ADMIN_PRINCIPAL, ADMIN_PRINCIPAL, ADMIN_ROLES, TOKEN_TTL_SECONDS);
|
||||
final Instant expiresAt = Instant.now().plus(Duration.ofSeconds(TOKEN_TTL_SECONDS));
|
||||
return new TokenSet(jwt, null, jwt, expiresAt);
|
||||
}
|
||||
}
|
||||
+48
@@ -0,0 +1,48 @@
|
||||
package org.openmetadata.it.auth;
|
||||
|
||||
import com.auth0.jwt.JWT;
|
||||
import com.auth0.jwt.algorithms.Algorithm;
|
||||
import java.io.InputStream;
|
||||
import java.security.KeyFactory;
|
||||
import java.security.PrivateKey;
|
||||
import java.security.spec.PKCS8EncodedKeySpec;
|
||||
import java.time.Instant;
|
||||
|
||||
public class JwtAuthProvider {
|
||||
|
||||
private static PrivateKey privateKey;
|
||||
|
||||
private static PrivateKey loadPrivateKey() {
|
||||
if (privateKey != null) return privateKey;
|
||||
try {
|
||||
InputStream is =
|
||||
JwtAuthProvider.class.getClassLoader().getResourceAsStream("private_key.der");
|
||||
if (is == null) throw new IllegalStateException("private_key.der not found in resources");
|
||||
byte[] keyBytes = is.readAllBytes();
|
||||
PKCS8EncodedKeySpec spec = new PKCS8EncodedKeySpec(keyBytes);
|
||||
KeyFactory kf = KeyFactory.getInstance("RSA");
|
||||
privateKey = kf.generatePrivate(spec);
|
||||
return privateKey;
|
||||
} catch (Exception e) {
|
||||
throw new RuntimeException("Failed to load private key", e);
|
||||
}
|
||||
}
|
||||
|
||||
public static String tokenFor(String subject, String email, String[] roles, long ttlSeconds) {
|
||||
Algorithm alg =
|
||||
Algorithm.RSA256(null, (java.security.interfaces.RSAPrivateKey) loadPrivateKey());
|
||||
Instant now = Instant.now();
|
||||
com.auth0.jwt.JWTCreator.Builder b =
|
||||
JWT.create()
|
||||
.withIssuer("open-metadata.org")
|
||||
.withKeyId("test-key")
|
||||
.withIssuedAt(java.util.Date.from(now))
|
||||
.withExpiresAt(java.util.Date.from(now.plusSeconds(ttlSeconds)))
|
||||
.withSubject(subject)
|
||||
.withClaim("email", email);
|
||||
if (roles != null && roles.length > 0) {
|
||||
b.withArrayClaim("roles", roles);
|
||||
}
|
||||
return b.sign(alg);
|
||||
}
|
||||
}
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
package org.openmetadata.it.auth;
|
||||
|
||||
import java.lang.annotation.ElementType;
|
||||
import java.lang.annotation.Retention;
|
||||
import java.lang.annotation.RetentionPolicy;
|
||||
import java.lang.annotation.Target;
|
||||
|
||||
/**
|
||||
* Skip the per-test auth injection done by {@code UiSessionExtension}, so the test starts
|
||||
* with a clean {@code BrowserContext} that lands on the OM sign-in page instead of an
|
||||
* authenticated home view.
|
||||
*
|
||||
* <p>Use only on tests that explicitly drive the sign-in UI (basic-auth credentials, the
|
||||
* "Sign in with <provider>" button, callback handling). Every other test should
|
||||
* accept the default token-injected fast path.
|
||||
*/
|
||||
@Target(ElementType.METHOD)
|
||||
@Retention(RetentionPolicy.RUNTIME)
|
||||
public @interface NoPreloadAuth {}
|
||||
+159
@@ -0,0 +1,159 @@
|
||||
package org.openmetadata.it.auth;
|
||||
|
||||
import com.microsoft.playwright.BrowserContext;
|
||||
import java.net.URI;
|
||||
import java.net.URLEncoder;
|
||||
import java.net.http.HttpClient;
|
||||
import java.net.http.HttpRequest;
|
||||
import java.net.http.HttpRequest.BodyPublishers;
|
||||
import java.net.http.HttpResponse;
|
||||
import java.net.http.HttpResponse.BodyHandlers;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.Locale;
|
||||
import java.util.Map;
|
||||
import java.util.regex.Matcher;
|
||||
import java.util.regex.Pattern;
|
||||
import org.openmetadata.it.server.ServerHandle;
|
||||
import org.openmetadata.it.server.sso.MockOidcServer;
|
||||
import org.openmetadata.it.server.sso.SsoProfile;
|
||||
|
||||
/**
|
||||
* Single OIDC backend — works for any {@link SsoProfile} (Google, Okta, custom OIDC) and
|
||||
* either {@code ClientType}. Boots OM with the profile's env vars and acquires tokens
|
||||
* silently against the mock IdP using OAuth2 password / refresh-token grants.
|
||||
*
|
||||
* <p>Browser-based sign-in flows are tested by separate {@code *SignInUIIT} classes; this
|
||||
* backend is what every other test reads from when running under SSO.
|
||||
*/
|
||||
public final class OidcBackend implements AuthBackend {
|
||||
|
||||
private static final String CLIENT_ID = "om-test-client";
|
||||
private static final String CLIENT_SECRET = "om-test-secret";
|
||||
private static final String SCOPE = "openid email profile";
|
||||
private static final String DEFAULT_USER = "admin@open-metadata.org";
|
||||
private static final String DEFAULT_PASSWORD = "test-password";
|
||||
private static final Duration HTTP_TIMEOUT = Duration.ofSeconds(15);
|
||||
private static final Pattern ACCESS_TOKEN_PATTERN =
|
||||
Pattern.compile("\"access_token\"\\s*:\\s*\"([^\"]+)\"");
|
||||
private static final Pattern REFRESH_TOKEN_PATTERN =
|
||||
Pattern.compile("\"refresh_token\"\\s*:\\s*\"([^\"]+)\"");
|
||||
private static final Pattern ID_TOKEN_PATTERN =
|
||||
Pattern.compile("\"id_token\"\\s*:\\s*\"([^\"]+)\"");
|
||||
private static final Pattern EXPIRES_IN_PATTERN =
|
||||
Pattern.compile("\"expires_in\"\\s*:\\s*(\\d+)");
|
||||
|
||||
private final SsoProfile profile;
|
||||
private final HttpClient http = HttpClient.newBuilder().connectTimeout(HTTP_TIMEOUT).build();
|
||||
|
||||
public OidcBackend(final SsoProfile profile) {
|
||||
this.profile = profile;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String name() {
|
||||
return String.format(
|
||||
Locale.ROOT,
|
||||
"sso-%s-%s",
|
||||
profile.providerSlug(),
|
||||
profile.clientType().name().toLowerCase(Locale.ROOT));
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean requiresIdp() {
|
||||
return true;
|
||||
}
|
||||
|
||||
@Override
|
||||
public SsoProfile ssoProfile() {
|
||||
return profile;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Map<String, String> serverEnv(final MockOidcServer idp, final int omHostPort) {
|
||||
return profile.serverEnv(idp, omHostPort);
|
||||
}
|
||||
|
||||
@Override
|
||||
public TokenSet acquireToken(final ServerHandle server, final MockOidcServer idp) {
|
||||
return requestTokens(
|
||||
idp,
|
||||
"grant_type=password&username=%s&password=%s&scope=%s"
|
||||
.formatted(urlEncode(DEFAULT_USER), urlEncode(DEFAULT_PASSWORD), urlEncode(SCOPE)));
|
||||
}
|
||||
|
||||
@Override
|
||||
public TokenSet refresh(
|
||||
final TokenSet current, final ServerHandle server, final MockOidcServer idp) {
|
||||
if (current.refreshToken() == null) {
|
||||
return acquireToken(server, idp);
|
||||
}
|
||||
return requestTokens(
|
||||
idp, "grant_type=refresh_token&refresh_token=" + urlEncode(current.refreshToken()));
|
||||
}
|
||||
|
||||
@Override
|
||||
public void injectIntoBrowser(final BrowserContext context, final TokenSet tokens) {
|
||||
AppStateInjection.seed(context, tokens.idToken());
|
||||
}
|
||||
|
||||
private TokenSet requestTokens(final MockOidcServer idp, final String formBody) {
|
||||
final URI tokenUri = URI.create(idp.issuerUrl(profile.issuerId()) + "/token");
|
||||
final String body =
|
||||
formBody
|
||||
+ "&client_id="
|
||||
+ urlEncode(CLIENT_ID)
|
||||
+ "&client_secret="
|
||||
+ urlEncode(CLIENT_SECRET);
|
||||
final HttpRequest request =
|
||||
HttpRequest.newBuilder(tokenUri)
|
||||
.header("Content-Type", "application/x-www-form-urlencoded")
|
||||
.header("Accept", "application/json")
|
||||
.timeout(HTTP_TIMEOUT)
|
||||
.POST(BodyPublishers.ofString(body, StandardCharsets.UTF_8))
|
||||
.build();
|
||||
final HttpResponse<String> response = send(request);
|
||||
if (response.statusCode() != 200) {
|
||||
throw new IllegalStateException(
|
||||
"Mock IdP rejected token request (" + response.statusCode() + "): " + response.body());
|
||||
}
|
||||
return parse(response.body());
|
||||
}
|
||||
|
||||
private HttpResponse<String> send(final HttpRequest request) {
|
||||
try {
|
||||
return http.send(request, BodyHandlers.ofString());
|
||||
} catch (java.io.IOException e) {
|
||||
throw new IllegalStateException("Mock IdP token request failed", e);
|
||||
} catch (InterruptedException e) {
|
||||
Thread.currentThread().interrupt();
|
||||
throw new IllegalStateException("Mock IdP token request interrupted", e);
|
||||
}
|
||||
}
|
||||
|
||||
private static TokenSet parse(final String body) {
|
||||
final String accessToken = requireMatch(ACCESS_TOKEN_PATTERN, body, "access_token");
|
||||
final String idToken = requireMatch(ID_TOKEN_PATTERN, body, "id_token");
|
||||
final String refreshToken = optionalMatch(REFRESH_TOKEN_PATTERN, body);
|
||||
final long expiresIn = Long.parseLong(requireMatch(EXPIRES_IN_PATTERN, body, "expires_in"));
|
||||
return new TokenSet(accessToken, refreshToken, idToken, Instant.now().plusSeconds(expiresIn));
|
||||
}
|
||||
|
||||
private static String requireMatch(final Pattern pattern, final String body, final String field) {
|
||||
final Matcher matcher = pattern.matcher(body);
|
||||
if (!matcher.find()) {
|
||||
throw new IllegalStateException("Mock IdP response missing '" + field + "': " + body);
|
||||
}
|
||||
return matcher.group(1);
|
||||
}
|
||||
|
||||
private static String optionalMatch(final Pattern pattern, final String body) {
|
||||
final Matcher matcher = pattern.matcher(body);
|
||||
return matcher.find() ? matcher.group(1) : null;
|
||||
}
|
||||
|
||||
private static String urlEncode(final String s) {
|
||||
return URLEncoder.encode(s, StandardCharsets.UTF_8);
|
||||
}
|
||||
}
|
||||
+83
@@ -0,0 +1,83 @@
|
||||
package org.openmetadata.it.auth;
|
||||
|
||||
import java.time.Duration;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.ScheduledExecutorService;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import org.openmetadata.it.server.ServerHandle;
|
||||
import org.openmetadata.it.server.sso.MockOidcServer;
|
||||
import org.openmetadata.it.util.SdkClients;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
/**
|
||||
* Background daemon that keeps {@link AuthSession#current()} valid for the lifetime of a
|
||||
* test JVM. Wakes every {@link #TICK_INTERVAL} and refreshes if the token is within
|
||||
* {@link #REFRESH_THRESHOLD} of its expiry, then publishes the new token to
|
||||
* {@link AuthSession} and re-points {@link SdkClients} so factories pick it up.
|
||||
*
|
||||
* <p>UI tests pull the latest token at {@code beforeEach} via {@link AuthSession#current()},
|
||||
* so a refresh between tests is transparent. Tests that hold a long-lived browser context
|
||||
* across many minutes are the only case where refresh wouldn't take effect mid-flight —
|
||||
* acceptable as long as the per-test token is fresh on entry.
|
||||
*/
|
||||
public final class TokenRefresher implements AutoCloseable {
|
||||
|
||||
private static final Logger LOG = LoggerFactory.getLogger(TokenRefresher.class);
|
||||
private static final Duration TICK_INTERVAL = Duration.ofSeconds(30);
|
||||
private static final Duration REFRESH_THRESHOLD = Duration.ofMinutes(2);
|
||||
|
||||
private final ScheduledExecutorService scheduler;
|
||||
private final AuthBackend backend;
|
||||
private final ServerHandle server;
|
||||
private final MockOidcServer idp;
|
||||
|
||||
private TokenRefresher(
|
||||
final ScheduledExecutorService scheduler,
|
||||
final AuthBackend backend,
|
||||
final ServerHandle server,
|
||||
final MockOidcServer idp) {
|
||||
this.scheduler = scheduler;
|
||||
this.backend = backend;
|
||||
this.server = server;
|
||||
this.idp = idp;
|
||||
}
|
||||
|
||||
public static TokenRefresher start(
|
||||
final AuthBackend backend, final ServerHandle server, final MockOidcServer idp) {
|
||||
final ScheduledExecutorService scheduler =
|
||||
Executors.newSingleThreadScheduledExecutor(
|
||||
r -> {
|
||||
Thread t = new Thread(r, "jpw-token-refresher");
|
||||
t.setDaemon(true);
|
||||
return t;
|
||||
});
|
||||
final TokenRefresher refresher = new TokenRefresher(scheduler, backend, server, idp);
|
||||
scheduler.scheduleAtFixedRate(
|
||||
refresher::tick, TICK_INTERVAL.toSeconds(), TICK_INTERVAL.toSeconds(), TimeUnit.SECONDS);
|
||||
return refresher;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() {
|
||||
scheduler.shutdownNow();
|
||||
}
|
||||
|
||||
private void tick() {
|
||||
try {
|
||||
final TokenSet current = AuthSession.current();
|
||||
if (!current.expiresWithin(REFRESH_THRESHOLD)) {
|
||||
return;
|
||||
}
|
||||
LOG.info(
|
||||
"Refreshing {} token (expires in {}s)",
|
||||
backend.name(),
|
||||
current.timeUntilExpiry().toSeconds());
|
||||
final TokenSet refreshed = backend.refresh(current, server, idp);
|
||||
AuthSession.update(refreshed);
|
||||
SdkClients.overrideAdminToken(refreshed.accessToken());
|
||||
} catch (RuntimeException e) {
|
||||
LOG.warn("Token refresh failed; will retry on next tick", e);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
package org.openmetadata.it.auth;
|
||||
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
|
||||
/**
|
||||
* A snapshot of credentials issued by an {@link AuthBackend}.
|
||||
*
|
||||
* <p>Refresh-incapable backends (e.g. basic JWT) leave {@code refreshToken} {@code null}
|
||||
* and rely on {@link AuthBackend#refresh} to mint a fresh access token from scratch.
|
||||
* The {@code idToken} is populated for OIDC backends and used as the value the OM UI
|
||||
* expects in {@code localStorage.app_state.primary}.
|
||||
*/
|
||||
public record TokenSet(String accessToken, String refreshToken, String idToken, Instant expiresAt) {
|
||||
|
||||
public Duration timeUntilExpiry() {
|
||||
return Duration.between(Instant.now(), expiresAt);
|
||||
}
|
||||
|
||||
public boolean expiresWithin(final Duration window) {
|
||||
return timeUntilExpiry().compareTo(window) <= 0;
|
||||
}
|
||||
}
|
||||
+149
@@ -0,0 +1,149 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.bootstrap;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.fasterxml.jackson.databind.node.ArrayNode;
|
||||
import com.fasterxml.jackson.databind.node.ObjectNode;
|
||||
import com.sun.net.httpserver.HttpExchange;
|
||||
import com.sun.net.httpserver.HttpServer;
|
||||
import java.io.IOException;
|
||||
import java.io.OutputStream;
|
||||
import java.net.InetSocketAddress;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.List;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
/**
|
||||
* In-JVM stub of an OpenAI-compatible {@code /v1/chat/completions} endpoint, used by the embedded
|
||||
* integration-test suite to drive the Company Context knowledge-pill pipeline deterministically
|
||||
* without a real LLM provider or network egress. {@code OpenAICompletionClient} parses the stub's
|
||||
* reply exactly as it would a real OpenAI response, so the upload → text-extraction →
|
||||
* completion → pill-persistence → linking path is exercised end to end.
|
||||
*
|
||||
* <p>To keep other suite tests that process Context Center files unaffected, the stub returns the
|
||||
* canned {@link #EXPECTED_PILLS} only when the request body contains {@link #PILL_TRIGGER}; every
|
||||
* other completion request gets an empty array and therefore creates no memories.
|
||||
*/
|
||||
public final class LlmStubServer {
|
||||
private static final Logger LOG = LoggerFactory.getLogger(LlmStubServer.class);
|
||||
private static final ObjectMapper MAPPER = new ObjectMapper();
|
||||
private static final String CHAT_COMPLETIONS_PATH = "/v1/chat/completions";
|
||||
private static final String CONTENT_TYPE = "Content-Type";
|
||||
private static final String APPLICATION_JSON = "application/json";
|
||||
|
||||
/** Sentinel a test embeds in its uploaded file content to opt into pill generation. */
|
||||
public static final String PILL_TRIGGER = "OM-PILL-STUB-TRIGGER";
|
||||
|
||||
/** A knowledge pill the stub returns, matching the server-side {@code KnowledgePill} contract. */
|
||||
public record ExpectedPill(
|
||||
String title, String question, String answer, String summary, String memoryType) {}
|
||||
|
||||
public static final List<ExpectedPill> EXPECTED_PILLS =
|
||||
List.of(
|
||||
new ExpectedPill(
|
||||
"Refund Policy",
|
||||
"What is the refund policy?",
|
||||
"Customers can return any product within 30 days for a full refund.",
|
||||
"30-day full-refund window.",
|
||||
"Faq"),
|
||||
new ExpectedPill(
|
||||
"Primary Data Warehouse",
|
||||
"What is the primary data warehouse?",
|
||||
"Google BigQuery is the primary data warehouse.",
|
||||
"Primary warehouse is BigQuery.",
|
||||
"Note"),
|
||||
new ExpectedPill(
|
||||
"Support SLA",
|
||||
"What is the support team SLA?",
|
||||
"The support team responds to all tickets within 24 hours.",
|
||||
"24-hour ticket response SLA.",
|
||||
"Runbook"));
|
||||
|
||||
private final HttpServer server;
|
||||
private final int port;
|
||||
|
||||
private LlmStubServer(HttpServer server, int port) {
|
||||
this.server = server;
|
||||
this.port = port;
|
||||
}
|
||||
|
||||
public static LlmStubServer start() {
|
||||
LlmStubServer result;
|
||||
try {
|
||||
HttpServer server = HttpServer.create(new InetSocketAddress(0), 0);
|
||||
LlmStubServer stub = new LlmStubServer(server, server.getAddress().getPort());
|
||||
server.createContext(CHAT_COMPLETIONS_PATH, stub::handleCompletion);
|
||||
server.start();
|
||||
LOG.info("LLM stub server started on {}", stub.baseUrl());
|
||||
result = stub;
|
||||
} catch (IOException e) {
|
||||
throw new IllegalStateException("Failed to start LLM stub server", e);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
/** OpenAI-compatible base URL; {@code OpenAICompletionClient} appends {@code /chat/completions}. */
|
||||
public String baseUrl() {
|
||||
return "http://localhost:" + port + "/v1";
|
||||
}
|
||||
|
||||
public void stop() {
|
||||
server.stop(0);
|
||||
LOG.info("LLM stub server stopped");
|
||||
}
|
||||
|
||||
private void handleCompletion(HttpExchange exchange) throws IOException {
|
||||
boolean includePills = readRequest(exchange).contains(PILL_TRIGGER);
|
||||
byte[] body = completionResponse(includePills).getBytes(StandardCharsets.UTF_8);
|
||||
exchange.getResponseHeaders().add(CONTENT_TYPE, APPLICATION_JSON);
|
||||
exchange.sendResponseHeaders(200, body.length);
|
||||
try (OutputStream os = exchange.getResponseBody()) {
|
||||
os.write(body);
|
||||
}
|
||||
}
|
||||
|
||||
private String readRequest(HttpExchange exchange) throws IOException {
|
||||
return new String(exchange.getRequestBody().readAllBytes(), StandardCharsets.UTF_8);
|
||||
}
|
||||
|
||||
private String completionResponse(boolean includePills) {
|
||||
String result;
|
||||
try {
|
||||
ArrayNode pills = includePills ? pillsArray() : MAPPER.createArrayNode();
|
||||
ObjectNode message = MAPPER.createObjectNode();
|
||||
message.put("role", "assistant");
|
||||
message.put("content", MAPPER.writeValueAsString(pills));
|
||||
ObjectNode envelope = MAPPER.createObjectNode();
|
||||
envelope.putArray("choices").addObject().set("message", message);
|
||||
result = MAPPER.writeValueAsString(envelope);
|
||||
} catch (IOException e) {
|
||||
throw new IllegalStateException("Failed to build LLM stub response", e);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
private ArrayNode pillsArray() {
|
||||
ArrayNode pills = MAPPER.createArrayNode();
|
||||
for (ExpectedPill pill : EXPECTED_PILLS) {
|
||||
ObjectNode node = pills.addObject();
|
||||
node.put("title", pill.title());
|
||||
node.put("question", pill.question());
|
||||
node.put("answer", pill.answer());
|
||||
node.put("summary", pill.summary());
|
||||
node.put("memoryType", pill.memoryType());
|
||||
}
|
||||
return pills;
|
||||
}
|
||||
}
|
||||
+85
@@ -0,0 +1,85 @@
|
||||
package org.openmetadata.it.bootstrap;
|
||||
|
||||
import io.dropwizard.core.server.DefaultServerFactory;
|
||||
import io.dropwizard.jetty.ConnectorFactory;
|
||||
import io.dropwizard.jetty.HttpConnectorFactory;
|
||||
import io.dropwizard.jetty.HttpsConnectorFactory;
|
||||
import io.dropwizard.testing.junit5.DropwizardAppExtension;
|
||||
import java.util.concurrent.atomic.AtomicReference;
|
||||
import org.openmetadata.service.OpenMetadataApplication;
|
||||
import org.openmetadata.service.OpenMetadataApplicationConfig;
|
||||
|
||||
public final class SessionMultiNodeCluster {
|
||||
private static final AtomicReference<SessionMultiNodeCluster> INSTANCE = new AtomicReference<>();
|
||||
|
||||
private final DropwizardAppExtension<OpenMetadataApplicationConfig> nodeA;
|
||||
private final DropwizardAppExtension<OpenMetadataApplicationConfig> nodeB;
|
||||
|
||||
private SessionMultiNodeCluster(
|
||||
DropwizardAppExtension<OpenMetadataApplicationConfig> nodeA,
|
||||
DropwizardAppExtension<OpenMetadataApplicationConfig> nodeB) {
|
||||
this.nodeA = nodeA;
|
||||
this.nodeB = nodeB;
|
||||
}
|
||||
|
||||
public static SessionMultiNodeCluster getInstance() {
|
||||
SessionMultiNodeCluster existing = INSTANCE.get();
|
||||
if (existing != null) {
|
||||
return existing;
|
||||
}
|
||||
|
||||
synchronized (SessionMultiNodeCluster.class) {
|
||||
existing = INSTANCE.get();
|
||||
if (existing != null) {
|
||||
return existing;
|
||||
}
|
||||
|
||||
DropwizardAppExtension<OpenMetadataApplicationConfig> nodeA = startNode();
|
||||
DropwizardAppExtension<OpenMetadataApplicationConfig> nodeB = startNode();
|
||||
SessionMultiNodeCluster cluster = new SessionMultiNodeCluster(nodeA, nodeB);
|
||||
INSTANCE.set(cluster);
|
||||
return cluster;
|
||||
}
|
||||
}
|
||||
|
||||
public String nodeABaseUrl() {
|
||||
return "http://localhost:" + nodeA.getLocalPort();
|
||||
}
|
||||
|
||||
public String nodeBBaseUrl() {
|
||||
return "http://localhost:" + nodeB.getLocalPort();
|
||||
}
|
||||
|
||||
private static DropwizardAppExtension<OpenMetadataApplicationConfig> startNode() {
|
||||
OpenMetadataApplicationConfig config = TestSuiteBootstrap.createApplicationConfigCopy();
|
||||
resetPorts(config);
|
||||
DropwizardAppExtension<OpenMetadataApplicationConfig> app =
|
||||
new DropwizardAppExtension<>(OpenMetadataApplication.class, config);
|
||||
try {
|
||||
app.before();
|
||||
} catch (Exception e) {
|
||||
throw new IllegalStateException("Failed to start additional OpenMetadata node", e);
|
||||
}
|
||||
TestSuiteBootstrap.registerAdditionalApp(app);
|
||||
return app;
|
||||
}
|
||||
|
||||
private static void resetPorts(OpenMetadataApplicationConfig config) {
|
||||
if (config.getServerFactory() instanceof DefaultServerFactory serverFactory) {
|
||||
for (ConnectorFactory connectorFactory : serverFactory.getApplicationConnectors()) {
|
||||
if (connectorFactory instanceof HttpConnectorFactory httpConnectorFactory) {
|
||||
httpConnectorFactory.setPort(0);
|
||||
} else if (connectorFactory instanceof HttpsConnectorFactory httpsConnectorFactory) {
|
||||
httpsConnectorFactory.setPort(0);
|
||||
}
|
||||
}
|
||||
for (ConnectorFactory connectorFactory : serverFactory.getAdminConnectors()) {
|
||||
if (connectorFactory instanceof HttpConnectorFactory httpConnectorFactory) {
|
||||
httpConnectorFactory.setPort(0);
|
||||
} else if (connectorFactory instanceof HttpsConnectorFactory httpsConnectorFactory) {
|
||||
httpsConnectorFactory.setPort(0);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
+443
@@ -0,0 +1,443 @@
|
||||
/*
|
||||
* Copyright 2021 Collate
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.openmetadata.it.bootstrap;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
import org.openmetadata.schema.api.classification.CreateClassification;
|
||||
import org.openmetadata.schema.api.classification.CreateTag;
|
||||
import org.openmetadata.schema.api.data.CreateGlossary;
|
||||
import org.openmetadata.schema.api.data.CreateGlossaryTerm;
|
||||
import org.openmetadata.schema.api.domains.CreateDomain;
|
||||
import org.openmetadata.schema.api.domains.CreateDomain.DomainType;
|
||||
import org.openmetadata.schema.api.policies.CreatePolicy;
|
||||
import org.openmetadata.schema.api.services.CreateDatabaseService;
|
||||
import org.openmetadata.schema.api.services.DatabaseConnection;
|
||||
import org.openmetadata.schema.api.teams.CreateTeam;
|
||||
import org.openmetadata.schema.api.teams.CreateTeam.TeamType;
|
||||
import org.openmetadata.schema.api.teams.CreateUser;
|
||||
import org.openmetadata.schema.entity.classification.Classification;
|
||||
import org.openmetadata.schema.entity.classification.Tag;
|
||||
import org.openmetadata.schema.entity.data.Glossary;
|
||||
import org.openmetadata.schema.entity.data.GlossaryTerm;
|
||||
import org.openmetadata.schema.entity.domains.Domain;
|
||||
import org.openmetadata.schema.entity.policies.Policy;
|
||||
import org.openmetadata.schema.entity.policies.accessControl.Rule;
|
||||
import org.openmetadata.schema.entity.services.DatabaseService;
|
||||
import org.openmetadata.schema.entity.teams.Role;
|
||||
import org.openmetadata.schema.entity.teams.Team;
|
||||
import org.openmetadata.schema.entity.teams.User;
|
||||
import org.openmetadata.schema.services.connections.database.MysqlConnection;
|
||||
import org.openmetadata.schema.services.connections.database.common.basicAuth;
|
||||
import org.openmetadata.schema.type.EntityReference;
|
||||
import org.openmetadata.schema.type.MetadataOperation;
|
||||
import org.openmetadata.schema.type.TagLabel;
|
||||
import org.openmetadata.schema.type.TagLabel.TagSource;
|
||||
import org.openmetadata.sdk.client.OpenMetadataClient;
|
||||
import org.openmetadata.sdk.services.classification.ClassificationService;
|
||||
import org.openmetadata.sdk.services.classification.TagService;
|
||||
import org.openmetadata.sdk.services.domains.DomainService;
|
||||
import org.openmetadata.sdk.services.glossary.GlossaryService;
|
||||
import org.openmetadata.sdk.services.glossary.GlossaryTermService;
|
||||
import org.openmetadata.sdk.services.policies.PolicyService;
|
||||
import org.openmetadata.sdk.services.services.DatabaseServiceService;
|
||||
import org.openmetadata.sdk.services.teams.RoleService;
|
||||
import org.openmetadata.sdk.services.teams.TeamService;
|
||||
import org.openmetadata.sdk.services.teams.UserService;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
/**
|
||||
* Session-scoped shared entities used across all test classes.
|
||||
* Created ONCE at session start via TestSuiteBootstrap, deleted at session end.
|
||||
*
|
||||
* <p>These entities are NEVER modified by individual tests - they are
|
||||
* read-only reference entities for ownership, team membership, access control, etc.
|
||||
*
|
||||
* <p>Individual tests create their own namespaced entities for testing, but
|
||||
* can reference these shared entities as owners, domains, tags, etc.
|
||||
*/
|
||||
public final class SharedEntities {
|
||||
|
||||
private static final Logger LOG = LoggerFactory.getLogger(SharedEntities.class);
|
||||
private static SharedEntities INSTANCE;
|
||||
|
||||
// Users
|
||||
public final User USER1;
|
||||
public final User USER2;
|
||||
public final User USER3;
|
||||
public final EntityReference USER1_REF;
|
||||
public final EntityReference USER2_REF;
|
||||
public final EntityReference USER3_REF;
|
||||
|
||||
// Teams
|
||||
public final Team ORG_TEAM;
|
||||
public final Team TEAM1;
|
||||
public final Team TEAM11;
|
||||
public final Team TEAM2;
|
||||
public final Team TEAM21;
|
||||
|
||||
// Roles
|
||||
public final Role DATA_STEWARD_ROLE;
|
||||
public final Role DATA_CONSUMER_ROLE;
|
||||
public final EntityReference DATA_STEWARD_ROLE_REF;
|
||||
public final EntityReference DATA_CONSUMER_ROLE_REF;
|
||||
|
||||
// Policies
|
||||
public final Policy POLICY1;
|
||||
public final Policy POLICY2;
|
||||
|
||||
// Domains
|
||||
public final Domain DOMAIN;
|
||||
public final Domain SUB_DOMAIN;
|
||||
|
||||
// Classifications & Tags
|
||||
public final Classification PII_CLASSIFICATION;
|
||||
public final Tag PERSONAL_DATA_TAG;
|
||||
public final Tag SENSITIVE_TAG;
|
||||
public final TagLabel PERSONAL_DATA_TAG_LABEL;
|
||||
public final TagLabel PII_SENSITIVE_TAG_LABEL;
|
||||
|
||||
// Glossaries
|
||||
public final Glossary GLOSSARY1;
|
||||
public final Glossary GLOSSARY2;
|
||||
public final GlossaryTerm GLOSSARY1_TERM1;
|
||||
public final TagLabel GLOSSARY1_TERM1_LABEL;
|
||||
|
||||
// Services - DatabaseService as example (others can be added as needed)
|
||||
public final DatabaseService MYSQL_SERVICE;
|
||||
public final EntityReference MYSQL_REFERENCE;
|
||||
|
||||
private SharedEntities(
|
||||
User user1,
|
||||
User user2,
|
||||
User user3,
|
||||
Team orgTeam,
|
||||
Team team1,
|
||||
Team team11,
|
||||
Team team2,
|
||||
Team team21,
|
||||
Role dataStewardRole,
|
||||
Role dataConsumerRole,
|
||||
Policy policy1,
|
||||
Policy policy2,
|
||||
Domain domain,
|
||||
Domain subDomain,
|
||||
Classification piiClassification,
|
||||
Tag personalDataTag,
|
||||
Tag sensitiveTag,
|
||||
Glossary glossary1,
|
||||
Glossary glossary2,
|
||||
GlossaryTerm glossary1Term1,
|
||||
DatabaseService mysqlService) {
|
||||
|
||||
this.USER1 = user1;
|
||||
this.USER2 = user2;
|
||||
this.USER3 = user3;
|
||||
this.USER1_REF = user1.getEntityReference();
|
||||
this.USER2_REF = user2.getEntityReference();
|
||||
this.USER3_REF = user3.getEntityReference();
|
||||
|
||||
this.ORG_TEAM = orgTeam;
|
||||
this.TEAM1 = team1;
|
||||
this.TEAM11 = team11;
|
||||
this.TEAM2 = team2;
|
||||
this.TEAM21 = team21;
|
||||
|
||||
this.DATA_STEWARD_ROLE = dataStewardRole;
|
||||
this.DATA_CONSUMER_ROLE = dataConsumerRole;
|
||||
this.DATA_STEWARD_ROLE_REF = dataStewardRole.getEntityReference();
|
||||
this.DATA_CONSUMER_ROLE_REF = dataConsumerRole.getEntityReference();
|
||||
|
||||
this.POLICY1 = policy1;
|
||||
this.POLICY2 = policy2;
|
||||
|
||||
this.DOMAIN = domain;
|
||||
this.SUB_DOMAIN = subDomain;
|
||||
|
||||
this.PII_CLASSIFICATION = piiClassification;
|
||||
this.PERSONAL_DATA_TAG = personalDataTag;
|
||||
this.SENSITIVE_TAG = sensitiveTag;
|
||||
this.PERSONAL_DATA_TAG_LABEL =
|
||||
new TagLabel()
|
||||
.withTagFQN(personalDataTag.getFullyQualifiedName())
|
||||
.withSource(TagSource.CLASSIFICATION)
|
||||
.withLabelType(TagLabel.LabelType.MANUAL);
|
||||
this.PII_SENSITIVE_TAG_LABEL =
|
||||
new TagLabel()
|
||||
.withTagFQN(sensitiveTag.getFullyQualifiedName())
|
||||
.withSource(TagSource.CLASSIFICATION)
|
||||
.withLabelType(TagLabel.LabelType.MANUAL);
|
||||
|
||||
this.GLOSSARY1 = glossary1;
|
||||
this.GLOSSARY2 = glossary2;
|
||||
this.GLOSSARY1_TERM1 = glossary1Term1;
|
||||
this.GLOSSARY1_TERM1_LABEL =
|
||||
new TagLabel()
|
||||
.withTagFQN(glossary1Term1.getFullyQualifiedName())
|
||||
.withSource(TagSource.GLOSSARY)
|
||||
.withLabelType(TagLabel.LabelType.MANUAL);
|
||||
|
||||
this.MYSQL_SERVICE = mysqlService;
|
||||
this.MYSQL_REFERENCE = mysqlService.getEntityReference();
|
||||
}
|
||||
|
||||
public static SharedEntities get() {
|
||||
if (INSTANCE == null) {
|
||||
throw new IllegalStateException(
|
||||
"SharedEntities not initialized. Ensure TestSuiteBootstrap has run.");
|
||||
}
|
||||
return INSTANCE;
|
||||
}
|
||||
|
||||
public static boolean isInitialized() {
|
||||
return INSTANCE != null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Initialize all shared entities. Called by TestSuiteBootstrap.open().
|
||||
*/
|
||||
public static void initialize(OpenMetadataClient adminClient) {
|
||||
if (INSTANCE != null) {
|
||||
LOG.info("SharedEntities already initialized, skipping");
|
||||
return;
|
||||
}
|
||||
|
||||
LOG.info("=== SharedEntities: Creating shared test entities ===");
|
||||
long startTime = System.currentTimeMillis();
|
||||
|
||||
try {
|
||||
// Get existing system roles (created by seed data)
|
||||
RoleService roleService = new RoleService(adminClient.getHttpClient());
|
||||
Role dataStewardRole = roleService.getByName("DataSteward", "policies");
|
||||
Role dataConsumerRole = roleService.getByName("DataConsumer", "policies");
|
||||
|
||||
// Create policies
|
||||
PolicyService policyService = new PolicyService(adminClient.getHttpClient());
|
||||
Policy policy1 = createPolicy(policyService, "shared_policy1");
|
||||
Policy policy2 = createPolicy(policyService, "shared_policy2");
|
||||
|
||||
// Create a test role with AllowAll policy for permission tests
|
||||
org.openmetadata.schema.api.teams.CreateRole createTestRole =
|
||||
new org.openmetadata.schema.api.teams.CreateRole()
|
||||
.withName("shared_test_admin_role")
|
||||
.withDescription("Test role with AllowAll permissions for integration tests")
|
||||
.withPolicies(List.of(policy1.getName()));
|
||||
Role testAdminRole = roleService.create(createTestRole);
|
||||
|
||||
// Get org team (system entity)
|
||||
TeamService teamService = new TeamService(adminClient.getHttpClient());
|
||||
Team orgTeam = teamService.getByName("Organization", null);
|
||||
|
||||
// Create teams
|
||||
Team team1 = createTeam(teamService, "shared_team1", TeamType.DEPARTMENT, null);
|
||||
Team team11 = createTeam(teamService, "shared_team11", TeamType.GROUP, team1.getId());
|
||||
Team team2 = createTeam(teamService, "shared_team2", TeamType.DEPARTMENT, null);
|
||||
Team team21 = createTeam(teamService, "shared_team21", TeamType.GROUP, team2.getId());
|
||||
|
||||
// Create users
|
||||
// USER1 has test admin role (AllowAll) for permission tests
|
||||
UserService userService = new UserService(adminClient.getHttpClient());
|
||||
User user1 =
|
||||
createUser(
|
||||
userService, "shared_user1", List.of(team1.getId()), List.of(testAdminRole.getId()));
|
||||
User user2 = createUser(userService, "shared_user2", List.of(team2.getId()), List.of());
|
||||
User user3 = createUser(userService, "shared_user3", List.of(), List.of());
|
||||
|
||||
// Create domains
|
||||
DomainService domainService = new DomainService(adminClient.getHttpClient());
|
||||
Domain domain = createDomain(domainService, "shared_domain");
|
||||
Domain subDomain =
|
||||
createSubDomain(domainService, "shared_subdomain", domain.getFullyQualifiedName());
|
||||
|
||||
// Create classification and tags
|
||||
ClassificationService classificationService =
|
||||
new ClassificationService(adminClient.getHttpClient());
|
||||
TagService tagService = new TagService(adminClient.getHttpClient());
|
||||
Classification piiClassification =
|
||||
getOrCreateClassification(classificationService, "SharedPII");
|
||||
Tag personalDataTag = createTag(tagService, piiClassification.getName(), "PersonalData");
|
||||
Tag sensitiveTag = createTag(tagService, piiClassification.getName(), "Sensitive");
|
||||
|
||||
// Create glossaries
|
||||
GlossaryService glossaryService = new GlossaryService(adminClient.getHttpClient());
|
||||
GlossaryTermService glossaryTermService =
|
||||
new GlossaryTermService(adminClient.getHttpClient());
|
||||
Glossary glossary1 = createGlossary(glossaryService, "shared_glossary1");
|
||||
Glossary glossary2 = createGlossary(glossaryService, "shared_glossary2");
|
||||
GlossaryTerm glossary1Term1 =
|
||||
createGlossaryTerm(glossaryTermService, glossary1, "shared_term1");
|
||||
|
||||
// Create database service
|
||||
DatabaseServiceService dbServiceService =
|
||||
new DatabaseServiceService(adminClient.getHttpClient());
|
||||
DatabaseService mysqlService = createMySqlService(dbServiceService, "shared_mysql");
|
||||
|
||||
INSTANCE =
|
||||
new SharedEntities(
|
||||
user1,
|
||||
user2,
|
||||
user3,
|
||||
orgTeam,
|
||||
team1,
|
||||
team11,
|
||||
team2,
|
||||
team21,
|
||||
dataStewardRole,
|
||||
dataConsumerRole,
|
||||
policy1,
|
||||
policy2,
|
||||
domain,
|
||||
subDomain,
|
||||
piiClassification,
|
||||
personalDataTag,
|
||||
sensitiveTag,
|
||||
glossary1,
|
||||
glossary2,
|
||||
glossary1Term1,
|
||||
mysqlService);
|
||||
|
||||
long duration = System.currentTimeMillis() - startTime;
|
||||
LOG.info("=== SharedEntities: Created shared entities in {}ms ===", duration);
|
||||
|
||||
} catch (Exception e) {
|
||||
LOG.error("Failed to create shared entities", e);
|
||||
throw new RuntimeException("SharedEntities initialization failed", e);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Cleanup all shared entities. Called by TestSuiteBootstrap.close().
|
||||
*/
|
||||
public static void cleanup(OpenMetadataClient adminClient) {
|
||||
if (INSTANCE == null) {
|
||||
LOG.info("SharedEntities not initialized, skipping cleanup");
|
||||
return;
|
||||
}
|
||||
|
||||
LOG.info("=== SharedEntities: Cleaning up shared test entities ===");
|
||||
// Skip cleanup since containers are destroyed anyway
|
||||
INSTANCE = null;
|
||||
LOG.info("=== SharedEntities: Cleanup complete ===");
|
||||
}
|
||||
|
||||
// === Entity Creation Helpers ===
|
||||
|
||||
private static User createUser(
|
||||
UserService userService, String name, List<UUID> teamIds, List<UUID> roleIds) {
|
||||
CreateUser createUser =
|
||||
new CreateUser()
|
||||
.withName(name)
|
||||
.withEmail(name + "@test.openmetadata.org")
|
||||
.withTeams(teamIds)
|
||||
.withRoles(roleIds);
|
||||
return userService.create(createUser);
|
||||
}
|
||||
|
||||
private static Team createTeam(
|
||||
TeamService teamService, String name, TeamType type, UUID parentId) {
|
||||
CreateTeam createTeam =
|
||||
new CreateTeam().withName(name).withDisplayName(name).withTeamType(type);
|
||||
if (parentId != null) {
|
||||
createTeam.withParents(List.of(parentId));
|
||||
}
|
||||
return teamService.create(createTeam);
|
||||
}
|
||||
|
||||
private static Policy createPolicy(PolicyService policyService, String name) {
|
||||
Rule allowAllRule =
|
||||
new Rule()
|
||||
.withName("AllowAll")
|
||||
.withResources(List.of("All"))
|
||||
.withOperations(List.of(MetadataOperation.ALL))
|
||||
.withEffect(Rule.Effect.ALLOW);
|
||||
|
||||
CreatePolicy createPolicy = new CreatePolicy().withName(name).withRules(List.of(allowAllRule));
|
||||
return policyService.create(createPolicy);
|
||||
}
|
||||
|
||||
private static Domain createDomain(DomainService domainService, String name) {
|
||||
CreateDomain createDomain =
|
||||
new CreateDomain()
|
||||
.withName(name)
|
||||
.withDomainType(DomainType.AGGREGATE)
|
||||
.withDescription("Shared test domain");
|
||||
return domainService.create(createDomain);
|
||||
}
|
||||
|
||||
private static Domain createSubDomain(
|
||||
DomainService domainService, String name, String parentFqn) {
|
||||
CreateDomain createDomain =
|
||||
new CreateDomain()
|
||||
.withName(name)
|
||||
.withDomainType(DomainType.AGGREGATE)
|
||||
.withDescription("Shared test subdomain")
|
||||
.withParent(parentFqn);
|
||||
return domainService.create(createDomain);
|
||||
}
|
||||
|
||||
private static Classification getOrCreateClassification(
|
||||
ClassificationService classificationService, String name) {
|
||||
try {
|
||||
return classificationService.getByName(name, null);
|
||||
} catch (Exception e) {
|
||||
CreateClassification create =
|
||||
new CreateClassification().withName(name).withDescription("Shared test classification");
|
||||
return classificationService.create(create);
|
||||
}
|
||||
}
|
||||
|
||||
private static Tag createTag(TagService tagService, String classificationName, String tagName) {
|
||||
CreateTag createTag =
|
||||
new CreateTag()
|
||||
.withName(tagName)
|
||||
.withClassification(classificationName)
|
||||
.withDescription("Shared test tag");
|
||||
return tagService.create(createTag);
|
||||
}
|
||||
|
||||
private static Glossary createGlossary(GlossaryService glossaryService, String name) {
|
||||
CreateGlossary createGlossary =
|
||||
new CreateGlossary().withName(name).withDescription("Shared test glossary");
|
||||
return glossaryService.create(createGlossary);
|
||||
}
|
||||
|
||||
private static GlossaryTerm createGlossaryTerm(
|
||||
GlossaryTermService glossaryTermService, Glossary glossary, String name) {
|
||||
CreateGlossaryTerm createTerm =
|
||||
new CreateGlossaryTerm()
|
||||
.withName(name)
|
||||
.withGlossary(glossary.getFullyQualifiedName())
|
||||
.withDescription("Shared test glossary term");
|
||||
return glossaryTermService.create(createTerm);
|
||||
}
|
||||
|
||||
private static DatabaseService createMySqlService(
|
||||
DatabaseServiceService serviceService, String name) {
|
||||
MysqlConnection connection =
|
||||
new MysqlConnection()
|
||||
.withHostPort("localhost:3306")
|
||||
.withUsername("test")
|
||||
.withAuthType(new basicAuth().withPassword("test"));
|
||||
|
||||
DatabaseConnection dbConn = new DatabaseConnection().withConfig(connection);
|
||||
|
||||
CreateDatabaseService create =
|
||||
new CreateDatabaseService()
|
||||
.withName(name)
|
||||
.withServiceType(CreateDatabaseService.DatabaseServiceType.Mysql)
|
||||
.withConnection(dbConn);
|
||||
return serviceService.create(create);
|
||||
}
|
||||
}
|
||||
+1326
File diff suppressed because it is too large
Load Diff
+235
@@ -0,0 +1,235 @@
|
||||
package org.openmetadata.it.drive;
|
||||
|
||||
import static jakarta.ws.rs.core.Response.Status.CREATED;
|
||||
import static org.awaitility.Awaitility.await;
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertNotNull;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
import static org.junit.jupiter.api.Assumptions.assumeTrue;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import io.dropwizard.jackson.Jackson;
|
||||
import io.dropwizard.jersey.jackson.JacksonFeature;
|
||||
import jakarta.ws.rs.client.Client;
|
||||
import jakarta.ws.rs.client.ClientBuilder;
|
||||
import jakarta.ws.rs.client.Entity;
|
||||
import jakarta.ws.rs.client.WebTarget;
|
||||
import jakarta.ws.rs.core.MediaType;
|
||||
import jakarta.ws.rs.core.MultivaluedHashMap;
|
||||
import jakarta.ws.rs.core.MultivaluedMap;
|
||||
import jakarta.ws.rs.core.Response;
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.Duration;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.UUID;
|
||||
import org.glassfish.jersey.client.ClientProperties;
|
||||
import org.glassfish.jersey.media.multipart.FormDataMultiPart;
|
||||
import org.glassfish.jersey.media.multipart.MultiPartFeature;
|
||||
import org.glassfish.jersey.media.multipart.file.StreamDataBodyPart;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.extension.ExtendWith;
|
||||
import org.openmetadata.it.bootstrap.LlmStubServer;
|
||||
import org.openmetadata.it.bootstrap.TestSuiteBootstrap;
|
||||
import org.openmetadata.schema.entity.context.ContextMemory;
|
||||
import org.openmetadata.schema.entity.context.ContextMemorySourceType;
|
||||
import org.openmetadata.schema.entity.data.ContextFile;
|
||||
import org.openmetadata.schema.entity.data.ProcessingStatus;
|
||||
import org.openmetadata.schema.utils.JsonUtils;
|
||||
import org.openmetadata.sdk.test.util.RestClient;
|
||||
import org.openmetadata.sdk.test.util.SdkClients;
|
||||
import org.openmetadata.sdk.test.util.TestNamespace;
|
||||
import org.openmetadata.sdk.test.util.TestNamespaceExtension;
|
||||
|
||||
/**
|
||||
* End-to-end test for the Company Context pipeline: upload a Context Center file, the async
|
||||
* pipeline extracts text then (when an LLM provider is configured) knowledge pills, stored as
|
||||
* {@link ContextMemory} rows linked back to the file via {@code sourceFile}.
|
||||
*
|
||||
* <p>Pill assertions are skipped (via {@link org.junit.jupiter.api.Assumptions}) when the server
|
||||
* has {@code llmConfiguration.enabled=false} (the default), since no pills are produced then. The
|
||||
* text-extraction path to {@code Processed} is always asserted.
|
||||
*/
|
||||
@ExtendWith(TestNamespaceExtension.class)
|
||||
class CompanyContextPipelineIT {
|
||||
|
||||
private static String serverBaseUrl;
|
||||
private static Client multipartClient;
|
||||
private static WebTarget uploadTarget;
|
||||
|
||||
@BeforeAll
|
||||
static void setup() {
|
||||
String itBaseUrl =
|
||||
System.getProperty(
|
||||
"IT_BASE_URL",
|
||||
System.getenv().getOrDefault("IT_BASE_URL", "http://localhost:8585/api"));
|
||||
serverBaseUrl =
|
||||
itBaseUrl.endsWith("/api") ? itBaseUrl.substring(0, itBaseUrl.length() - 4) : itBaseUrl;
|
||||
|
||||
multipartClient = ClientBuilder.newClient();
|
||||
multipartClient.register(MultiPartFeature.class);
|
||||
multipartClient.register(new JacksonFeature(Jackson.newObjectMapper()));
|
||||
uploadTarget =
|
||||
multipartClient
|
||||
.target(serverBaseUrl + "/api/v1/contextCenter/drive/files/upload")
|
||||
.property(ClientProperties.CONNECT_TIMEOUT, 30000)
|
||||
.property(ClientProperties.READ_TIMEOUT, 30000);
|
||||
}
|
||||
|
||||
@AfterAll
|
||||
static void tearDown() {
|
||||
if (multipartClient != null) {
|
||||
multipartClient.close();
|
||||
multipartClient = null;
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void fileUploadProducesLinkedKnowledgePills(TestNamespace ns) throws Exception {
|
||||
String text =
|
||||
"Our refund policy allows customers to return any product within 30 days for a full "
|
||||
+ "refund. The primary data warehouse is Google BigQuery. The support team SLA is to "
|
||||
+ "respond to all tickets within 24 hours. "
|
||||
+ LlmStubServer.PILL_TRIGGER;
|
||||
|
||||
ContextFile file = upload(ns.prefix("company-policy") + ".txt", text);
|
||||
|
||||
await()
|
||||
.pollInterval(Duration.ofMillis(500))
|
||||
.atMost(Duration.ofSeconds(45))
|
||||
.untilAsserted(
|
||||
() ->
|
||||
assertEquals(
|
||||
ProcessingStatus.Processed, fetchFile(file.getId()).getProcessingStatus()));
|
||||
|
||||
boolean stubLlm = TestSuiteBootstrap.isLlmStubEnabled();
|
||||
int memoryCount = awaitMemoryCount(file.getId(), stubLlm);
|
||||
|
||||
List<ContextMemory> pills = pillsForFile(file.getId());
|
||||
assertEquals(memoryCount, pills.size(), "memoryCount should match the linked pills");
|
||||
for (ContextMemory pill : pills) {
|
||||
assertEquals(ContextMemorySourceType.FILE_EXTRACTION, pill.getSourceType());
|
||||
assertNotNull(pill.getSourceFile(), "pill must reference its source file");
|
||||
assertEquals(file.getId(), pill.getSourceFile().getId());
|
||||
assertTrue(pill.getQuestion() != null && !pill.getQuestion().isBlank());
|
||||
assertTrue(pill.getAnswer() != null && !pill.getAnswer().isBlank());
|
||||
}
|
||||
|
||||
if (stubLlm) {
|
||||
assertStubPillsPresent(pills);
|
||||
assertExtractionStats(file.getId(), memoryCount);
|
||||
}
|
||||
}
|
||||
|
||||
private void assertExtractionStats(UUID fileId, int expectedPills) throws Exception {
|
||||
ContextFile processed = fetchFile(fileId);
|
||||
assertNotNull(processed.getExtractionStats(), "extraction stats must be recorded");
|
||||
assertEquals(expectedPills, processed.getExtractionStats().getPillsCreated());
|
||||
assertEquals(
|
||||
processed.getExtractionStats().getChunksTotal(),
|
||||
processed.getExtractionStats().getChunksProcessed(),
|
||||
"stub extraction should cover every chunk");
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves the file's derived {@code memoryCount}. With the embedded LLM stub active the pipeline
|
||||
* is deterministic, so this hard-asserts the exact canned pill count (polling defensively for the
|
||||
* derived field); otherwise it preserves the legacy lenient skip for servers without an LLM.
|
||||
*/
|
||||
private int awaitMemoryCount(UUID fileId, boolean stubLlm) throws Exception {
|
||||
int result;
|
||||
if (stubLlm) {
|
||||
int expected = LlmStubServer.EXPECTED_PILLS.size();
|
||||
await()
|
||||
.pollInterval(Duration.ofMillis(500))
|
||||
.atMost(Duration.ofSeconds(20))
|
||||
.untilAsserted(
|
||||
() ->
|
||||
assertEquals(
|
||||
expected,
|
||||
memoryCountOf(fetchFile(fileId)),
|
||||
"embedded LLM stub should produce exactly the canned knowledge pills"));
|
||||
result = expected;
|
||||
} else {
|
||||
int memoryCount = memoryCountOf(fetchFile(fileId));
|
||||
assumeTrue(
|
||||
memoryCount > 0,
|
||||
"Server has no LLM provider configured (llmConfiguration.enabled=false); "
|
||||
+ "skipping knowledge-pill assertions.");
|
||||
result = memoryCount;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
private static int memoryCountOf(ContextFile file) {
|
||||
return file.getMemoryCount() == null ? 0 : file.getMemoryCount();
|
||||
}
|
||||
|
||||
private void assertStubPillsPresent(List<ContextMemory> pills) {
|
||||
Map<String, String> answersByQuestion = new HashMap<>();
|
||||
for (ContextMemory pill : pills) {
|
||||
answersByQuestion.put(pill.getQuestion(), pill.getAnswer());
|
||||
}
|
||||
for (LlmStubServer.ExpectedPill expected : LlmStubServer.EXPECTED_PILLS) {
|
||||
assertEquals(
|
||||
expected.answer(),
|
||||
answersByQuestion.get(expected.question()),
|
||||
"stub pill answer mismatch for question: " + expected.question());
|
||||
}
|
||||
}
|
||||
|
||||
private ContextFile upload(String fileName, String content) throws Exception {
|
||||
try (FormDataMultiPart multipart = new FormDataMultiPart()) {
|
||||
multipart.field("displayName", fileName);
|
||||
multipart.bodyPart(
|
||||
new StreamDataBodyPart(
|
||||
"file",
|
||||
new ByteArrayInputStream(content.getBytes(StandardCharsets.UTF_8)),
|
||||
fileName,
|
||||
MediaType.APPLICATION_OCTET_STREAM_TYPE));
|
||||
try (Response response =
|
||||
uploadTarget
|
||||
.request()
|
||||
.headers(adminAuthHeaders())
|
||||
.post(Entity.entity(multipart, multipart.getMediaType()))) {
|
||||
String body = response.readEntity(String.class);
|
||||
assertEquals(CREATED.getStatusCode(), response.getStatus(), "Upload failed: " + body);
|
||||
return JsonUtils.readValue(body, ContextFile.class);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private ContextFile fetchFile(UUID fileId) throws Exception {
|
||||
return RestClient.admin()
|
||||
.getById("v1/contextCenter/drive/files", fileId, "memoryCount", ContextFile.class);
|
||||
}
|
||||
|
||||
/** Lists pills through the server-side {@code sourceFileId} filter the file panel UI uses. */
|
||||
private List<ContextMemory> pillsForFile(UUID fileId) throws Exception {
|
||||
List<ContextMemory> pills = new ArrayList<>();
|
||||
try (Response response =
|
||||
RestClient.admin()
|
||||
.rawGet(
|
||||
"v1/contextCenter/memories?fields=sourceFile&limit=1000&sourceFileId=" + fileId)) {
|
||||
JsonNode data =
|
||||
JsonUtils.getObjectMapper().readTree(response.readEntity(String.class)).get("data");
|
||||
if (data != null) {
|
||||
for (JsonNode node : data) {
|
||||
pills.add(JsonUtils.getObjectMapper().convertValue(node, ContextMemory.class));
|
||||
}
|
||||
}
|
||||
}
|
||||
return pills;
|
||||
}
|
||||
|
||||
private static MultivaluedMap<String, Object> adminAuthHeaders() {
|
||||
MultivaluedMap<String, Object> headers = new MultivaluedHashMap<>();
|
||||
headers.add("Authorization", "Bearer " + SdkClients.getAdminToken());
|
||||
return headers;
|
||||
}
|
||||
}
|
||||
+745
@@ -0,0 +1,745 @@
|
||||
package org.openmetadata.it.drive;
|
||||
|
||||
import static org.awaitility.Awaitility.await;
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertFalse;
|
||||
import static org.junit.jupiter.api.Assertions.assertNotNull;
|
||||
import static org.junit.jupiter.api.Assertions.assertNull;
|
||||
import static org.junit.jupiter.api.Assertions.assertThrows;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import jakarta.ws.rs.core.Response;
|
||||
import java.net.URLEncoder;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.Duration;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.UUID;
|
||||
import org.apache.http.client.HttpResponseException;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.extension.ExtendWith;
|
||||
import org.openmetadata.schema.api.data.CreateContextFile;
|
||||
import org.openmetadata.schema.api.data.CreateFolder;
|
||||
import org.openmetadata.schema.api.data.MoveContextFileRequest;
|
||||
import org.openmetadata.schema.entity.data.ContextFile;
|
||||
import org.openmetadata.schema.entity.data.ContextFileSourceType;
|
||||
import org.openmetadata.schema.entity.data.ContextFileType;
|
||||
import org.openmetadata.schema.entity.data.Folder;
|
||||
import org.openmetadata.schema.entity.data.ProcessingStatus;
|
||||
import org.openmetadata.schema.entity.teams.User;
|
||||
import org.openmetadata.schema.type.EntityReference;
|
||||
import org.openmetadata.schema.utils.JsonUtils;
|
||||
import org.openmetadata.sdk.test.util.RestClient;
|
||||
import org.openmetadata.sdk.test.util.TestNamespace;
|
||||
import org.openmetadata.sdk.test.util.TestNamespaceExtension;
|
||||
|
||||
@ExtendWith(TestNamespaceExtension.class)
|
||||
class ContextFileIT {
|
||||
|
||||
private static final String FILE_PATH = "v1/contextCenter/drive/files";
|
||||
private static final String FOLDER_PATH = "v1/contextCenter/drive/folders";
|
||||
|
||||
private ContextFile createFile(RestClient rest, CreateContextFile request)
|
||||
throws HttpResponseException {
|
||||
return rest.create(FILE_PATH, request, ContextFile.class);
|
||||
}
|
||||
|
||||
private ContextFile getFile(RestClient rest, UUID id, String fields)
|
||||
throws HttpResponseException {
|
||||
return rest.getById(FILE_PATH, id, fields, ContextFile.class);
|
||||
}
|
||||
|
||||
private Folder createFolder(RestClient rest, CreateFolder request) throws HttpResponseException {
|
||||
return rest.create(FOLDER_PATH, request, Folder.class);
|
||||
}
|
||||
|
||||
private List<String> listFileIds(RestClient rest, String path) {
|
||||
try (Response response = rest.rawGet(path)) {
|
||||
assertEquals(200, response.getStatus());
|
||||
JsonNode root = JsonUtils.readTree(response.readEntity(String.class));
|
||||
List<String> ids = new ArrayList<>();
|
||||
root.get("data").forEach(node -> ids.add(node.get("id").asText()));
|
||||
return ids;
|
||||
}
|
||||
}
|
||||
|
||||
// --- CRUD ---
|
||||
|
||||
@Test
|
||||
void testCreateContextFile(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
|
||||
CreateContextFile create =
|
||||
new CreateContextFile()
|
||||
.withName(ns.prefix("report-pdf"))
|
||||
.withDisplayName("Annual Report 2023")
|
||||
.withFileType(ContextFileType.PDF)
|
||||
.withFileSize(4200000)
|
||||
.withContentType("application/pdf")
|
||||
.withFileExtension("pdf")
|
||||
.withProcessingStatus(ProcessingStatus.Uploaded);
|
||||
|
||||
ContextFile file = createFile(rest, create);
|
||||
assertNotNull(file.getId());
|
||||
assertEquals("Annual Report 2023", file.getDisplayName());
|
||||
assertEquals(ContextFileType.PDF, file.getFileType());
|
||||
assertEquals(4200000, file.getFileSize().intValue());
|
||||
}
|
||||
|
||||
@Test
|
||||
void testCreateSpreadsheet(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
|
||||
CreateContextFile create =
|
||||
new CreateContextFile()
|
||||
.withName(ns.prefix("pricing-xlsx"))
|
||||
.withDisplayName("Product Pricing")
|
||||
.withFileType(ContextFileType.Spreadsheet)
|
||||
.withFileSize(128000)
|
||||
.withContentType("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet")
|
||||
.withFileExtension("xlsx")
|
||||
.withProcessingStatus(ProcessingStatus.Uploaded);
|
||||
|
||||
ContextFile file = createFile(rest, create);
|
||||
assertEquals(ContextFileType.Spreadsheet, file.getFileType());
|
||||
}
|
||||
|
||||
@Test
|
||||
void testGetFileById(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
|
||||
ContextFile created =
|
||||
createFile(
|
||||
rest,
|
||||
new CreateContextFile()
|
||||
.withName(ns.prefix("get-test"))
|
||||
.withFileType(ContextFileType.CSV)
|
||||
.withProcessingStatus(ProcessingStatus.Uploaded));
|
||||
|
||||
ContextFile fetched = getFile(rest, created.getId(), "");
|
||||
assertEquals(created.getId(), fetched.getId());
|
||||
assertEquals(ContextFileType.CSV, fetched.getFileType());
|
||||
}
|
||||
|
||||
@Test
|
||||
void testDeleteFile(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
|
||||
ContextFile file =
|
||||
createFile(
|
||||
rest,
|
||||
new CreateContextFile()
|
||||
.withName(ns.prefix("delete-test"))
|
||||
.withProcessingStatus(ProcessingStatus.Uploaded));
|
||||
|
||||
rest.delete(FILE_PATH, file.getId());
|
||||
|
||||
HttpResponseException ex =
|
||||
assertThrows(HttpResponseException.class, () -> getFile(rest, file.getId(), ""));
|
||||
assertEquals(404, ex.getStatusCode());
|
||||
|
||||
try (Response deletedResponse = rest.rawGet(FILE_PATH + "/" + file.getId() + "?include=all")) {
|
||||
assertEquals(200, deletedResponse.getStatus());
|
||||
assertTrue(deletedResponse.readEntity(String.class).contains("\"deleted\":true"));
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void testRestoreSoftDeletedFile(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
|
||||
ContextFile file =
|
||||
createFile(
|
||||
rest,
|
||||
new CreateContextFile()
|
||||
.withName(ns.prefix("restore-test"))
|
||||
.withProcessingStatus(ProcessingStatus.Uploaded));
|
||||
|
||||
rest.delete(FILE_PATH, file.getId());
|
||||
ContextFile restored = rest.restore(FILE_PATH, file.getId(), ContextFile.class);
|
||||
|
||||
assertEquals(file.getId(), restored.getId());
|
||||
assertTrue(!Boolean.TRUE.equals(restored.getDeleted()));
|
||||
assertEquals(file.getId(), getFile(rest, file.getId(), "").getId());
|
||||
}
|
||||
|
||||
@Test
|
||||
void testListFilesOrderByUpdatedAtDesc(TestNamespace ns) throws Exception {
|
||||
RestClient rest = RestClient.admin();
|
||||
|
||||
ContextFile older =
|
||||
createFile(
|
||||
rest,
|
||||
new CreateContextFile()
|
||||
.withName(ns.prefix("ordered-older"))
|
||||
.withProcessingStatus(ProcessingStatus.Uploaded));
|
||||
Thread.sleep(5);
|
||||
ContextFile newer =
|
||||
createFile(
|
||||
rest,
|
||||
new CreateContextFile()
|
||||
.withName(ns.prefix("ordered-newer"))
|
||||
.withProcessingStatus(ProcessingStatus.Uploaded));
|
||||
|
||||
List<String> ids = listFileIds(rest, FILE_PATH + "?limit=1000&orderBy=DESC");
|
||||
|
||||
int olderIndex = ids.indexOf(older.getId().toString());
|
||||
int newerIndex = ids.indexOf(newer.getId().toString());
|
||||
assertTrue(olderIndex >= 0, "Expected ordered older file in list response");
|
||||
assertTrue(newerIndex >= 0, "Expected ordered newer file in list response");
|
||||
assertTrue(newerIndex < olderIndex, "Newer file should be listed before older file");
|
||||
}
|
||||
|
||||
@Test
|
||||
void testListFilesOrderByRejectsDefaultCursor(TestNamespace ns) throws Exception {
|
||||
RestClient rest = RestClient.admin();
|
||||
|
||||
createFile(
|
||||
rest,
|
||||
new CreateContextFile()
|
||||
.withName(ns.prefix("default-cursor-first"))
|
||||
.withProcessingStatus(ProcessingStatus.Uploaded));
|
||||
createFile(
|
||||
rest,
|
||||
new CreateContextFile()
|
||||
.withName(ns.prefix("default-cursor-second"))
|
||||
.withProcessingStatus(ProcessingStatus.Uploaded));
|
||||
|
||||
try (Response response = rest.rawGet(FILE_PATH + "?limit=1")) {
|
||||
assertEquals(200, response.getStatus());
|
||||
JsonNode root = JsonUtils.readTree(response.readEntity(String.class));
|
||||
JsonNode after = root.get("paging").get("after");
|
||||
assertNotNull(after, "Default list response should include an after cursor");
|
||||
|
||||
String encodedCursor = URLEncoder.encode(after.asText(), StandardCharsets.UTF_8);
|
||||
try (Response orderByResponse =
|
||||
rest.rawGet(FILE_PATH + "?limit=1&orderBy=DESC&after=" + encodedCursor)) {
|
||||
String body = orderByResponse.readEntity(String.class);
|
||||
assertEquals(
|
||||
Response.Status.BAD_REQUEST.getStatusCode(), orderByResponse.getStatus(), body);
|
||||
assertTrue(body.contains("Invalid cursor for orderBy pagination"));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void testListArchivedFilesFilteredByUpdatedBy(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient adminRest = RestClient.admin();
|
||||
|
||||
ContextFile file =
|
||||
createFile(
|
||||
adminRest,
|
||||
new CreateContextFile()
|
||||
.withName(ns.prefix("archived"))
|
||||
.withProcessingStatus(ProcessingStatus.Uploaded));
|
||||
// Archiving is a soft-delete; the archiver is recorded in updatedBy (admin here, who deletes).
|
||||
adminRest.delete(FILE_PATH, file.getId());
|
||||
String archiver = getFileIncludeAll(adminRest, file.getId()).getUpdatedBy();
|
||||
|
||||
// Scoped to the archiver: the archived file is returned.
|
||||
List<String> byArchiver =
|
||||
listFileIds(
|
||||
adminRest, FILE_PATH + "?include=deleted&limit=1000&updatedBy=" + encode(archiver));
|
||||
assertTrue(
|
||||
byArchiver.contains(file.getId().toString()),
|
||||
"updatedBy filter must include files archived by that user");
|
||||
|
||||
// Scoped to a different user: the archived file is excluded.
|
||||
List<String> byOther =
|
||||
listFileIds(
|
||||
adminRest,
|
||||
FILE_PATH + "?include=deleted&limit=1000&updatedBy=" + encode(ns.prefix("nobody")));
|
||||
assertFalse(
|
||||
byOther.contains(file.getId().toString()),
|
||||
"updatedBy filter must exclude files archived by a different user");
|
||||
|
||||
// Without the filter the archived file is still listed (proves the filter, not the delete,
|
||||
// scopes).
|
||||
List<String> unfiltered = listFileIds(adminRest, FILE_PATH + "?include=deleted&limit=1000");
|
||||
assertTrue(
|
||||
unfiltered.contains(file.getId().toString()),
|
||||
"Unfiltered archive list must still contain the archived file");
|
||||
}
|
||||
|
||||
private String encode(String value) {
|
||||
return URLEncoder.encode(value, StandardCharsets.UTF_8);
|
||||
}
|
||||
|
||||
private ContextFile getFileIncludeAll(RestClient rest, UUID id) {
|
||||
try (Response response = rest.rawGet(FILE_PATH + "/" + id + "?include=all")) {
|
||||
assertEquals(200, response.getStatus());
|
||||
return JsonUtils.readValue(response.readEntity(String.class), ContextFile.class);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void testHardDeleteFileIsAsync(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
|
||||
ContextFile file =
|
||||
createFile(
|
||||
rest,
|
||||
new CreateContextFile()
|
||||
.withName(ns.prefix("perm-delete-test"))
|
||||
.withProcessingStatus(ProcessingStatus.Uploaded));
|
||||
|
||||
try (Response deleteResponse =
|
||||
rest.rawDelete(FILE_PATH + "/" + file.getId() + "?hardDelete=true")) {
|
||||
assertEquals(202, deleteResponse.getStatus());
|
||||
assertTrue(deleteResponse.readEntity(String.class).contains("\"hardDelete\":true"));
|
||||
}
|
||||
|
||||
await()
|
||||
.atMost(Duration.ofSeconds(10))
|
||||
.untilAsserted(
|
||||
() -> {
|
||||
try (Response deletedResponse =
|
||||
rest.rawGet(FILE_PATH + "/" + file.getId() + "?include=all")) {
|
||||
assertEquals(404, deletedResponse.getStatus());
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// --- File in Folder ---
|
||||
|
||||
@Test
|
||||
void testFileInFolder(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
|
||||
Folder folder = createFolder(rest, new CreateFolder().withName(ns.prefix("docs-folder")));
|
||||
|
||||
ContextFile file =
|
||||
createFile(
|
||||
rest,
|
||||
new CreateContextFile()
|
||||
.withName(ns.prefix("file-in-folder"))
|
||||
.withDisplayName("Report in Folder")
|
||||
.withFileType(ContextFileType.PDF)
|
||||
.withFolder(folder.getFullyQualifiedName())
|
||||
.withProcessingStatus(ProcessingStatus.Uploaded));
|
||||
|
||||
ContextFile fetched = getFile(rest, file.getId(), "folder");
|
||||
assertNotNull(fetched.getFolder());
|
||||
assertEquals(folder.getId(), fetched.getFolder().getId());
|
||||
|
||||
// FQN should include folder name
|
||||
assertTrue(
|
||||
fetched.getFullyQualifiedName().contains(folder.getName()),
|
||||
"File FQN should include folder name");
|
||||
}
|
||||
|
||||
@Test
|
||||
void testFileInNestedFolder(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
|
||||
Folder root = createFolder(rest, new CreateFolder().withName(ns.prefix("root")));
|
||||
Folder child =
|
||||
createFolder(
|
||||
rest,
|
||||
new CreateFolder()
|
||||
.withName(ns.prefix("child"))
|
||||
.withParent(root.getFullyQualifiedName()));
|
||||
|
||||
ContextFile file =
|
||||
createFile(
|
||||
rest,
|
||||
new CreateContextFile()
|
||||
.withName(ns.prefix("deep-file"))
|
||||
.withFolder(child.getFullyQualifiedName())
|
||||
.withProcessingStatus(ProcessingStatus.Uploaded));
|
||||
|
||||
ContextFile fetched = getFile(rest, file.getId(), "folder");
|
||||
assertTrue(
|
||||
fetched.getFullyQualifiedName().contains(root.getName()),
|
||||
"File FQN should contain root folder");
|
||||
assertTrue(
|
||||
fetched.getFullyQualifiedName().contains(child.getName()),
|
||||
"File FQN should contain child folder");
|
||||
}
|
||||
|
||||
// --- Source Provenance ---
|
||||
|
||||
@Test
|
||||
void testFileSourceProvenance(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
|
||||
CreateContextFile create =
|
||||
new CreateContextFile()
|
||||
.withName(ns.prefix("synced-file"))
|
||||
.withFileType(ContextFileType.Document)
|
||||
.withSourceType(ContextFileSourceType.Confluence)
|
||||
.withSourceId("page-12345")
|
||||
.withSourceUrl(java.net.URI.create("https://wiki.example.com/page/12345"))
|
||||
.withProcessingStatus(ProcessingStatus.Processed);
|
||||
|
||||
ContextFile file = createFile(rest, create);
|
||||
assertEquals(ContextFileSourceType.Confluence, file.getSourceType());
|
||||
assertEquals("page-12345", file.getSourceId());
|
||||
}
|
||||
|
||||
// --- Processing Status Update ---
|
||||
|
||||
@Test
|
||||
void testUpdateProcessingStatus(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
|
||||
ContextFile file =
|
||||
createFile(
|
||||
rest,
|
||||
new CreateContextFile()
|
||||
.withName(ns.prefix("status-test"))
|
||||
.withFileType(ContextFileType.PDF)
|
||||
.withProcessingStatus(ProcessingStatus.Uploaded));
|
||||
|
||||
assertEquals(ProcessingStatus.Uploaded, file.getProcessingStatus());
|
||||
|
||||
// Patch to Processed
|
||||
String original = JsonUtils.pojoToJson(file);
|
||||
file.setProcessingStatus(ProcessingStatus.Processed);
|
||||
ContextFile updated = rest.patch(FILE_PATH, file.getId(), original, file, ContextFile.class);
|
||||
|
||||
assertEquals(ProcessingStatus.Processed, updated.getProcessingStatus());
|
||||
}
|
||||
|
||||
// --- Permissions ---
|
||||
|
||||
@Test
|
||||
void testUnprivilegedUserCannotDeleteFile(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient adminRest = RestClient.admin();
|
||||
User owner = DriveTestUsers.createUser(ns, "file-owner");
|
||||
|
||||
ContextFile file =
|
||||
createFile(
|
||||
adminRest,
|
||||
new CreateContextFile()
|
||||
.withName(ns.prefix("perm-delete"))
|
||||
.withFileType(ContextFileType.PDF)
|
||||
.withProcessingStatus(ProcessingStatus.Uploaded)
|
||||
.withOwners(List.of(owner.getEntityReference())));
|
||||
|
||||
RestClient consumerRest = RestClient.forUser("test@open-metadata.org", new String[] {});
|
||||
|
||||
HttpResponseException ex =
|
||||
assertThrows(
|
||||
HttpResponseException.class, () -> consumerRest.hardDelete(FILE_PATH, file.getId()));
|
||||
|
||||
assertTrue(
|
||||
ex.getStatusCode() == 403 || ex.getStatusCode() == 401,
|
||||
"Expected 403/401, got " + ex.getStatusCode());
|
||||
}
|
||||
|
||||
@Test
|
||||
void testUnprivilegedUserCannotUpdateOthersFile(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient adminRest = RestClient.admin();
|
||||
User owner = DriveTestUsers.createUser(ns, "file-editor");
|
||||
|
||||
ContextFile file =
|
||||
createFile(
|
||||
adminRest,
|
||||
new CreateContextFile()
|
||||
.withName(ns.prefix("perm-update"))
|
||||
.withDisplayName("Admin's File")
|
||||
.withFileType(ContextFileType.PDF)
|
||||
.withProcessingStatus(ProcessingStatus.Uploaded)
|
||||
.withOwners(List.of(owner.getEntityReference())));
|
||||
|
||||
RestClient consumerRest = RestClient.forUser("test@open-metadata.org", new String[] {});
|
||||
|
||||
String original = JsonUtils.pojoToJson(file);
|
||||
file.setDisplayName("Hacked");
|
||||
|
||||
HttpResponseException ex =
|
||||
assertThrows(
|
||||
HttpResponseException.class,
|
||||
() -> consumerRest.patch(FILE_PATH, file.getId(), original, file, ContextFile.class));
|
||||
|
||||
assertTrue(
|
||||
ex.getStatusCode() == 403 || ex.getStatusCode() == 401,
|
||||
"Expected 403/401, got " + ex.getStatusCode());
|
||||
}
|
||||
|
||||
@Test
|
||||
void testOwnerCanUpdateOwnFile(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient adminRest = RestClient.admin();
|
||||
User owner = DriveTestUsers.createUser(ns, "file-self-owner");
|
||||
|
||||
ContextFile file =
|
||||
createFile(
|
||||
adminRest,
|
||||
new CreateContextFile()
|
||||
.withName(ns.prefix("owner-update"))
|
||||
.withDisplayName("Owner's File")
|
||||
.withFileType(ContextFileType.PDF)
|
||||
.withProcessingStatus(ProcessingStatus.Uploaded)
|
||||
.withOwners(List.of(owner.getEntityReference())));
|
||||
|
||||
// The explicit owner should be able to update the file.
|
||||
RestClient ownerRest = RestClient.forUser(owner.getEmail(), new String[] {});
|
||||
|
||||
String original = JsonUtils.pojoToJson(file);
|
||||
file.setDisplayName("Updated by Owner");
|
||||
|
||||
ContextFile updated =
|
||||
ownerRest.patch(FILE_PATH, file.getId(), original, file, ContextFile.class);
|
||||
assertEquals("Updated by Owner", updated.getDisplayName());
|
||||
}
|
||||
|
||||
// --- Search ---
|
||||
|
||||
// --- Move ---
|
||||
|
||||
private ContextFile moveFile(RestClient rest, UUID id, EntityReference newFolder)
|
||||
throws HttpResponseException {
|
||||
MoveContextFileRequest body = new MoveContextFileRequest().withFolder(newFolder);
|
||||
try (Response response = rest.rawPut(FILE_PATH + "/" + id + "/move", body)) {
|
||||
if (response.getStatus() >= 400) {
|
||||
throw new HttpResponseException(response.getStatus(), response.readEntity(String.class));
|
||||
}
|
||||
return JsonUtils.readValue(response.readEntity(String.class), ContextFile.class);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void testMoveFileBetweenFolders(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
Folder folderA = createFolder(rest, new CreateFolder().withName(ns.prefix("folder-a")));
|
||||
Folder folderB = createFolder(rest, new CreateFolder().withName(ns.prefix("folder-b")));
|
||||
|
||||
ContextFile file =
|
||||
createFile(
|
||||
rest,
|
||||
new CreateContextFile()
|
||||
.withName(ns.prefix("move-between"))
|
||||
.withFileType(ContextFileType.PDF)
|
||||
.withFolder(folderA.getFullyQualifiedName())
|
||||
.withProcessingStatus(ProcessingStatus.Uploaded));
|
||||
assertEquals(folderA.getId(), file.getFolder().getId());
|
||||
|
||||
ContextFile moved = moveFile(rest, file.getId(), folderB.getEntityReference());
|
||||
|
||||
assertEquals(folderB.getId(), moved.getFolder().getId());
|
||||
assertTrue(
|
||||
moved.getFullyQualifiedName().contains(folderB.getName()),
|
||||
"Moved file FQN should reflect new folder, got " + moved.getFullyQualifiedName());
|
||||
|
||||
ContextFile reloaded = getFile(rest, file.getId(), "folder");
|
||||
assertEquals(folderB.getId(), reloaded.getFolder().getId());
|
||||
}
|
||||
|
||||
@Test
|
||||
void testMoveFileToRoot(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
Folder folder = createFolder(rest, new CreateFolder().withName(ns.prefix("folder-root-test")));
|
||||
|
||||
ContextFile file =
|
||||
createFile(
|
||||
rest,
|
||||
new CreateContextFile()
|
||||
.withName(ns.prefix("move-to-root"))
|
||||
.withFileType(ContextFileType.PDF)
|
||||
.withFolder(folder.getFullyQualifiedName())
|
||||
.withProcessingStatus(ProcessingStatus.Uploaded));
|
||||
|
||||
ContextFile moved = moveFile(rest, file.getId(), null);
|
||||
|
||||
assertNull(moved.getFolder(), "File moved to root should have no folder reference");
|
||||
assertEquals(
|
||||
file.getName(), moved.getFullyQualifiedName(), "Root-level FQN should equal the file name");
|
||||
}
|
||||
|
||||
@Test
|
||||
void testMoveFileNonExistentFolder(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
ContextFile file =
|
||||
createFile(
|
||||
rest,
|
||||
new CreateContextFile()
|
||||
.withName(ns.prefix("move-bad-folder"))
|
||||
.withFileType(ContextFileType.PDF)
|
||||
.withProcessingStatus(ProcessingStatus.Uploaded));
|
||||
|
||||
EntityReference bogus = new EntityReference().withId(UUID.randomUUID()).withType("folder");
|
||||
|
||||
HttpResponseException ex =
|
||||
assertThrows(HttpResponseException.class, () -> moveFile(rest, file.getId(), bogus));
|
||||
assertEquals(404, ex.getStatusCode());
|
||||
}
|
||||
|
||||
@Test
|
||||
void testMoveFilePermissions(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient adminRest = RestClient.admin();
|
||||
User owner = DriveTestUsers.createUser(ns, "file-mover");
|
||||
|
||||
Folder folderA = createFolder(adminRest, new CreateFolder().withName(ns.prefix("perm-a")));
|
||||
Folder folderB = createFolder(adminRest, new CreateFolder().withName(ns.prefix("perm-b")));
|
||||
|
||||
ContextFile file =
|
||||
createFile(
|
||||
adminRest,
|
||||
new CreateContextFile()
|
||||
.withName(ns.prefix("perm-move"))
|
||||
.withFileType(ContextFileType.PDF)
|
||||
.withFolder(folderA.getFullyQualifiedName())
|
||||
.withOwners(List.of(owner.getEntityReference()))
|
||||
.withProcessingStatus(ProcessingStatus.Uploaded));
|
||||
|
||||
RestClient consumerRest = RestClient.forUser("test@open-metadata.org", new String[] {});
|
||||
|
||||
HttpResponseException ex =
|
||||
assertThrows(
|
||||
HttpResponseException.class,
|
||||
() -> moveFile(consumerRest, file.getId(), folderB.getEntityReference()));
|
||||
assertTrue(
|
||||
ex.getStatusCode() == 403 || ex.getStatusCode() == 401,
|
||||
"Expected 403/401, got " + ex.getStatusCode());
|
||||
}
|
||||
|
||||
@Test
|
||||
void testBulkMoveAndDeleteFiles(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
Folder target = createFolder(rest, new CreateFolder().withName(ns.prefix("bulk-target")));
|
||||
ContextFile first =
|
||||
createFile(
|
||||
rest,
|
||||
new CreateContextFile()
|
||||
.withName(ns.prefix("bulk-first"))
|
||||
.withDisplayName("Bulk First")
|
||||
.withProcessingStatus(ProcessingStatus.Uploaded));
|
||||
ContextFile second =
|
||||
createFile(
|
||||
rest,
|
||||
new CreateContextFile()
|
||||
.withName(ns.prefix("bulk-second"))
|
||||
.withDisplayName("Bulk Second")
|
||||
.withProcessingStatus(ProcessingStatus.Uploaded));
|
||||
List<String> ids = List.of(first.getId().toString(), second.getId().toString());
|
||||
|
||||
try (Response response =
|
||||
rest.rawPut(
|
||||
FILE_PATH + "/bulk/move", Map.of("ids", ids, "folder", target.getEntityReference()))) {
|
||||
String body = response.readEntity(String.class);
|
||||
assertEquals(200, response.getStatus(), body);
|
||||
JsonNode result = JsonUtils.readTree(body);
|
||||
assertEquals("success", result.get("status").asText());
|
||||
assertEquals(2, result.get("numberOfRowsPassed").asInt());
|
||||
}
|
||||
|
||||
assertEquals(target.getId(), getFile(rest, first.getId(), "folder").getFolder().getId());
|
||||
assertEquals(target.getId(), getFile(rest, second.getId(), "folder").getFolder().getId());
|
||||
|
||||
try (Response response =
|
||||
rest.rawPost(FILE_PATH + "/bulk/delete", Map.of("ids", ids, "hardDelete", false))) {
|
||||
String body = response.readEntity(String.class);
|
||||
assertEquals(200, response.getStatus(), body);
|
||||
JsonNode result = JsonUtils.readTree(body);
|
||||
assertEquals("success", result.get("status").asText());
|
||||
assertEquals(2, result.get("numberOfRowsPassed").asInt());
|
||||
}
|
||||
|
||||
HttpResponseException firstEx =
|
||||
assertThrows(HttpResponseException.class, () -> getFile(rest, first.getId(), ""));
|
||||
HttpResponseException secondEx =
|
||||
assertThrows(HttpResponseException.class, () -> getFile(rest, second.getId(), ""));
|
||||
assertEquals(404, firstEx.getStatusCode());
|
||||
assertEquals(404, secondEx.getStatusCode());
|
||||
|
||||
rest.delete(FOLDER_PATH, target.getId());
|
||||
|
||||
HttpResponseException folderEx =
|
||||
assertThrows(
|
||||
HttpResponseException.class,
|
||||
() -> rest.getById(FOLDER_PATH, target.getId(), "", Folder.class));
|
||||
assertEquals(404, folderEx.getStatusCode());
|
||||
}
|
||||
|
||||
@Test
|
||||
void testDeleteFolderCascadesMovedFiles(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
Folder target =
|
||||
createFolder(rest, new CreateFolder().withName(ns.prefix("delete-cascade-target")));
|
||||
ContextFile first =
|
||||
createFile(
|
||||
rest,
|
||||
new CreateContextFile()
|
||||
.withName(ns.prefix("cascade-first"))
|
||||
.withDisplayName("Cascade First")
|
||||
.withProcessingStatus(ProcessingStatus.Uploaded));
|
||||
ContextFile second =
|
||||
createFile(
|
||||
rest,
|
||||
new CreateContextFile()
|
||||
.withName(ns.prefix("cascade-second"))
|
||||
.withDisplayName("Cascade Second")
|
||||
.withProcessingStatus(ProcessingStatus.Uploaded));
|
||||
List<String> ids = List.of(first.getId().toString(), second.getId().toString());
|
||||
|
||||
try (Response response =
|
||||
rest.rawPut(
|
||||
FILE_PATH + "/bulk/move", Map.of("ids", ids, "folder", target.getEntityReference()))) {
|
||||
String body = response.readEntity(String.class);
|
||||
assertEquals(200, response.getStatus(), body);
|
||||
}
|
||||
|
||||
rest.delete(FOLDER_PATH, target.getId());
|
||||
|
||||
HttpResponseException folderEx =
|
||||
assertThrows(
|
||||
HttpResponseException.class,
|
||||
() -> rest.getById(FOLDER_PATH, target.getId(), "", Folder.class));
|
||||
HttpResponseException firstEx =
|
||||
assertThrows(HttpResponseException.class, () -> getFile(rest, first.getId(), ""));
|
||||
HttpResponseException secondEx =
|
||||
assertThrows(HttpResponseException.class, () -> getFile(rest, second.getId(), ""));
|
||||
assertEquals(404, folderEx.getStatusCode());
|
||||
assertEquals(404, firstEx.getStatusCode());
|
||||
assertEquals(404, secondEx.getStatusCode());
|
||||
}
|
||||
|
||||
@Test
|
||||
void testFileAppearsInSearch(TestNamespace ns) throws Exception {
|
||||
RestClient rest = RestClient.admin();
|
||||
|
||||
String uniqueName = ns.prefix("searchable-file");
|
||||
ContextFile file =
|
||||
createFile(
|
||||
rest,
|
||||
new CreateContextFile()
|
||||
.withName(uniqueName)
|
||||
.withDisplayName("Searchable PDF")
|
||||
.withFileType(ContextFileType.PDF)
|
||||
.withProcessingStatus(ProcessingStatus.Processed));
|
||||
|
||||
// ES indexing is async. Poll the direct get-by-id endpoint, which performs a real-time
|
||||
// ES GET (no query_string parsing, no analyzer involvement) and is the most reliable
|
||||
// signal that the document was indexed. The previous version of this test issued a
|
||||
// free-text q= search using the namespaced unique name, but the prefix contains '-'
|
||||
// which the query_string parser treats as a NOT operator and can produce a 500 on
|
||||
// ES 9.x — yielding a flaky 30s-timeout failure even when the document is indexed.
|
||||
await()
|
||||
.pollDelay(Duration.ZERO)
|
||||
.pollInterval(Duration.ofMillis(200))
|
||||
.atMost(Duration.ofSeconds(60))
|
||||
.untilAsserted(
|
||||
() -> {
|
||||
try (Response getResp =
|
||||
rest.rawGet("v1/search/get/context_file_search_index/doc/" + file.getId())) {
|
||||
int status = getResp.getStatus();
|
||||
String body = getResp.readEntity(String.class);
|
||||
if (status != 200) {
|
||||
throw new AssertionError(
|
||||
"Expected 200 from search-by-id for file "
|
||||
+ file.getId()
|
||||
+ " but got "
|
||||
+ status
|
||||
+ " body="
|
||||
+ body);
|
||||
}
|
||||
assertTrue(
|
||||
body.contains(file.getId().toString()),
|
||||
"Expected file " + file.getId() + " in search-by-id response: " + body);
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
+950
@@ -0,0 +1,950 @@
|
||||
package org.openmetadata.it.drive;
|
||||
|
||||
import static jakarta.ws.rs.core.Response.Status.CREATED;
|
||||
import static org.awaitility.Awaitility.await;
|
||||
import static org.junit.jupiter.api.Assertions.assertArrayEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertNotNull;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
|
||||
import io.dropwizard.jackson.Jackson;
|
||||
import io.dropwizard.jersey.jackson.JacksonFeature;
|
||||
import jakarta.ws.rs.client.Client;
|
||||
import jakarta.ws.rs.client.ClientBuilder;
|
||||
import jakarta.ws.rs.client.Entity;
|
||||
import jakarta.ws.rs.client.WebTarget;
|
||||
import jakarta.ws.rs.core.MediaType;
|
||||
import jakarta.ws.rs.core.MultivaluedHashMap;
|
||||
import jakarta.ws.rs.core.MultivaluedMap;
|
||||
import jakarta.ws.rs.core.Response;
|
||||
import java.awt.Color;
|
||||
import java.awt.Font;
|
||||
import java.awt.Graphics2D;
|
||||
import java.awt.RenderingHints;
|
||||
import java.awt.image.BufferedImage;
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.ByteArrayOutputStream;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.net.URI;
|
||||
import java.net.URLEncoder;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.time.Duration;
|
||||
import java.util.Comparator;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.UUID;
|
||||
import java.util.zip.ZipEntry;
|
||||
import java.util.zip.ZipInputStream;
|
||||
import javax.imageio.ImageIO;
|
||||
import org.apache.pdfbox.pdmodel.PDDocument;
|
||||
import org.apache.pdfbox.pdmodel.PDPage;
|
||||
import org.apache.pdfbox.pdmodel.PDPageContentStream;
|
||||
import org.apache.pdfbox.pdmodel.font.PDType1Font;
|
||||
import org.apache.poi.ss.usermodel.Workbook;
|
||||
import org.apache.poi.xssf.usermodel.XSSFWorkbook;
|
||||
import org.glassfish.jersey.client.ClientProperties;
|
||||
import org.glassfish.jersey.media.multipart.FormDataMultiPart;
|
||||
import org.glassfish.jersey.media.multipart.MultiPartFeature;
|
||||
import org.glassfish.jersey.media.multipart.file.StreamDataBodyPart;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.extension.ExtendWith;
|
||||
import org.openmetadata.schema.api.data.CreateFolder;
|
||||
import org.openmetadata.schema.entity.data.ContextFile;
|
||||
import org.openmetadata.schema.entity.data.ContextFileType;
|
||||
import org.openmetadata.schema.entity.data.Folder;
|
||||
import org.openmetadata.schema.entity.data.ProcessingStatus;
|
||||
import org.openmetadata.schema.utils.JsonUtils;
|
||||
import org.openmetadata.sdk.test.util.RestClient;
|
||||
import org.openmetadata.sdk.test.util.SdkClients;
|
||||
import org.openmetadata.sdk.test.util.TestNamespace;
|
||||
import org.openmetadata.sdk.test.util.TestNamespaceExtension;
|
||||
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
|
||||
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
|
||||
import software.amazon.awssdk.core.ResponseInputStream;
|
||||
import software.amazon.awssdk.regions.Region;
|
||||
import software.amazon.awssdk.services.s3.S3Client;
|
||||
import software.amazon.awssdk.services.s3.S3Configuration;
|
||||
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
|
||||
import software.amazon.awssdk.services.s3.model.GetObjectResponse;
|
||||
import software.amazon.awssdk.services.s3.model.ListObjectsV2Request;
|
||||
import software.amazon.awssdk.services.s3.model.S3Object;
|
||||
|
||||
/**
|
||||
* Integration test for Context Center Drive file upload with MinIO-backed S3 storage using
|
||||
* fixture files from src/test/resources.
|
||||
*/
|
||||
@ExtendWith(TestNamespaceExtension.class)
|
||||
class DriveFileUploadIT {
|
||||
|
||||
private static final String MINIO_BUCKET = "test-bucket";
|
||||
private static final String TIKA_TESSERACT_PATH_PROPERTY = "collate.tika.tesseract.path";
|
||||
private static String serverBaseUrl;
|
||||
private static Client multipartClient;
|
||||
private static WebTarget uploadTarget;
|
||||
|
||||
@BeforeAll
|
||||
static void setup() {
|
||||
String itBaseUrl =
|
||||
System.getProperty(
|
||||
"IT_BASE_URL",
|
||||
System.getenv().getOrDefault("IT_BASE_URL", "http://localhost:8585/api"));
|
||||
if (itBaseUrl.endsWith("/api")) {
|
||||
serverBaseUrl = itBaseUrl.substring(0, itBaseUrl.length() - 4);
|
||||
} else {
|
||||
serverBaseUrl = itBaseUrl;
|
||||
}
|
||||
|
||||
multipartClient = ClientBuilder.newClient();
|
||||
multipartClient.register(MultiPartFeature.class);
|
||||
multipartClient.register(new JacksonFeature(Jackson.newObjectMapper()));
|
||||
|
||||
uploadTarget =
|
||||
multipartClient
|
||||
.target(serverBaseUrl + "/api/v1/contextCenter/drive/files/upload")
|
||||
.property(ClientProperties.CONNECT_TIMEOUT, 30000)
|
||||
.property(ClientProperties.READ_TIMEOUT, 30000);
|
||||
}
|
||||
|
||||
@AfterAll
|
||||
static void tearDown() {
|
||||
if (multipartClient != null) {
|
||||
multipartClient.close();
|
||||
multipartClient = null;
|
||||
}
|
||||
}
|
||||
|
||||
private static MultivaluedMap<String, Object> adminAuthHeaders() {
|
||||
String token = SdkClients.getAdminToken();
|
||||
MultivaluedMap<String, Object> headers = new MultivaluedHashMap<>();
|
||||
headers.add("Authorization", "Bearer " + token);
|
||||
return headers;
|
||||
}
|
||||
|
||||
private byte[] readFixture(String resourcePath) throws IOException {
|
||||
try (InputStream inputStream = getClass().getResourceAsStream(resourcePath)) {
|
||||
assertNotNull(inputStream, "Missing drive fixture: " + resourcePath);
|
||||
return inputStream.readAllBytes();
|
||||
}
|
||||
}
|
||||
|
||||
private Response uploadFile(String fileName, byte[] content, String displayName, String folderFqn)
|
||||
throws IOException {
|
||||
try (FormDataMultiPart multipart = new FormDataMultiPart()) {
|
||||
if (displayName != null) {
|
||||
multipart.field("displayName", displayName);
|
||||
}
|
||||
if (folderFqn != null) {
|
||||
multipart.field("folder", folderFqn);
|
||||
}
|
||||
multipart.bodyPart(
|
||||
new StreamDataBodyPart(
|
||||
"file",
|
||||
new ByteArrayInputStream(content),
|
||||
fileName,
|
||||
MediaType.APPLICATION_OCTET_STREAM_TYPE));
|
||||
|
||||
return uploadTarget
|
||||
.request()
|
||||
.headers(adminAuthHeaders())
|
||||
.post(Entity.entity(multipart, multipart.getMediaType()));
|
||||
}
|
||||
}
|
||||
|
||||
private Response uploadFixture(String resourcePath, String displayName) throws IOException {
|
||||
String fileName = resourcePath.substring(resourcePath.lastIndexOf('/') + 1);
|
||||
return uploadFixture(resourcePath, fileName, displayName, null);
|
||||
}
|
||||
|
||||
private Response uploadUniqueFixture(TestNamespace ns, String resourcePath, String displayName)
|
||||
throws IOException {
|
||||
return uploadFixture(resourcePath, uniqueUploadedFileName(ns, resourcePath), displayName, null);
|
||||
}
|
||||
|
||||
private Response uploadFixture(
|
||||
String resourcePath, String uploadedFileName, String displayName, String folderFqn)
|
||||
throws IOException {
|
||||
return uploadFile(uploadedFileName, readFixture(resourcePath), displayName, folderFqn);
|
||||
}
|
||||
|
||||
private String uniqueUploadedFileName(TestNamespace ns, String resourcePath) {
|
||||
String fileName = resourcePath.substring(resourcePath.lastIndexOf('/') + 1);
|
||||
int dotIndex = fileName.lastIndexOf('.');
|
||||
if (dotIndex <= 0) {
|
||||
return ns.prefix(fileName);
|
||||
}
|
||||
return ns.prefix(fileName.substring(0, dotIndex)) + fileName.substring(dotIndex);
|
||||
}
|
||||
|
||||
private ContextFile readCreatedContextFile(Response response, String failurePrefix) {
|
||||
String body = response.readEntity(String.class);
|
||||
assertEquals(CREATED.getStatusCode(), response.getStatus(), failurePrefix + ": " + body);
|
||||
return JsonUtils.readValue(body, ContextFile.class);
|
||||
}
|
||||
|
||||
private String resolveStoredObjectKey(S3Client s3Client, String assetId) {
|
||||
return s3Client
|
||||
.listObjectsV2Paginator(ListObjectsV2Request.builder().bucket(MINIO_BUCKET).build())
|
||||
.contents()
|
||||
.stream()
|
||||
.map(S3Object::key)
|
||||
.filter(key -> key.equals(assetId) || key.endsWith(assetId) || key.contains(assetId))
|
||||
.findFirst()
|
||||
.orElse(null);
|
||||
}
|
||||
|
||||
private S3Client buildMinioClient() {
|
||||
return S3Client.builder()
|
||||
.region(Region.US_EAST_1)
|
||||
.credentialsProvider(
|
||||
StaticCredentialsProvider.create(AwsBasicCredentials.create("minio", "minio123")))
|
||||
.endpointOverride(
|
||||
URI.create(
|
||||
System.getProperty(
|
||||
"IT_MINIO_ENDPOINT",
|
||||
System.getenv().getOrDefault("IT_MINIO_ENDPOINT", "http://localhost:9000"))))
|
||||
.serviceConfiguration(S3Configuration.builder().pathStyleAccessEnabled(true).build())
|
||||
.build();
|
||||
}
|
||||
|
||||
private void assertStoredInMinIO(String assetId, byte[] expectedBytes) {
|
||||
try (S3Client s3Client = buildMinioClient()) {
|
||||
// atMost must stay above the global Awaitility pollInterval that
|
||||
// K8sOMJobOperatorIT raises to 5s; otherwise Awaitility rejects with
|
||||
// "Timeout must be greater than the poll delay".
|
||||
await()
|
||||
.pollDelay(Duration.ZERO)
|
||||
.pollInterval(Duration.ofMillis(200))
|
||||
.atMost(Duration.ofSeconds(20))
|
||||
.untilAsserted(
|
||||
() -> {
|
||||
String objectKey = resolveStoredObjectKey(s3Client, assetId);
|
||||
assertNotNull(objectKey, "Expected uploaded object for asset " + assetId);
|
||||
try (ResponseInputStream<GetObjectResponse> objectStream =
|
||||
s3Client.getObject(
|
||||
GetObjectRequest.builder().bucket(MINIO_BUCKET).key(objectKey).build())) {
|
||||
assertArrayEquals(expectedBytes, objectStream.readAllBytes());
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
private void assertRemovedFromMinIO(String assetId) {
|
||||
try (S3Client s3Client = buildMinioClient()) {
|
||||
await()
|
||||
.atMost(Duration.ofSeconds(10))
|
||||
.untilAsserted(() -> assertTrue(resolveStoredObjectKey(s3Client, assetId) == null));
|
||||
}
|
||||
}
|
||||
|
||||
private ContextFile fetchFile(UUID fileId) {
|
||||
try {
|
||||
return RestClient.admin()
|
||||
.getById("v1/contextCenter/drive/files", fileId, "", ContextFile.class);
|
||||
} catch (Exception e) {
|
||||
throw new AssertionError("Failed to fetch uploaded file " + fileId, e);
|
||||
}
|
||||
}
|
||||
|
||||
private Folder createFolder(TestNamespace ns, String name) throws Exception {
|
||||
return RestClient.admin()
|
||||
.create(
|
||||
"v1/contextCenter/drive/folders",
|
||||
new CreateFolder().withName(ns.prefix(name)),
|
||||
Folder.class);
|
||||
}
|
||||
|
||||
private void assertSearchContainsFile(String query, UUID fileId) {
|
||||
RestClient rest = RestClient.admin();
|
||||
String encodedQuery = URLEncoder.encode(query, StandardCharsets.UTF_8);
|
||||
|
||||
await()
|
||||
.atMost(Duration.ofSeconds(20))
|
||||
.untilAsserted(
|
||||
() -> {
|
||||
try (Response searchResponse =
|
||||
rest.rawGet(
|
||||
"v1/search/query?q="
|
||||
+ encodedQuery
|
||||
+ "&index=context_file_search_index&from=0&size=10")) {
|
||||
assertEquals(200, searchResponse.getStatus());
|
||||
assertTrue(searchResponse.readEntity(String.class).contains(fileId.toString()));
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
private byte[] createPdf(String text) throws IOException {
|
||||
try (PDDocument document = new PDDocument();
|
||||
ByteArrayOutputStream outputStream = new ByteArrayOutputStream()) {
|
||||
PDPage page = new PDPage();
|
||||
document.addPage(page);
|
||||
try (PDPageContentStream contentStream = new PDPageContentStream(document, page)) {
|
||||
contentStream.beginText();
|
||||
contentStream.setFont(PDType1Font.HELVETICA_BOLD, 12);
|
||||
contentStream.newLineAtOffset(72, 720);
|
||||
contentStream.showText(text);
|
||||
contentStream.endText();
|
||||
}
|
||||
document.save(outputStream);
|
||||
return outputStream.toByteArray();
|
||||
}
|
||||
}
|
||||
|
||||
private byte[] createWorkbook(String sheetName, String key, String value) throws IOException {
|
||||
try (Workbook workbook = new XSSFWorkbook();
|
||||
ByteArrayOutputStream outputStream = new ByteArrayOutputStream()) {
|
||||
var sheet = workbook.createSheet(sheetName);
|
||||
var header = sheet.createRow(0);
|
||||
header.createCell(0).setCellValue("Key");
|
||||
header.createCell(1).setCellValue("Value");
|
||||
var row = sheet.createRow(1);
|
||||
row.createCell(0).setCellValue(key);
|
||||
row.createCell(1).setCellValue(value);
|
||||
workbook.write(outputStream);
|
||||
return outputStream.toByteArray();
|
||||
}
|
||||
}
|
||||
|
||||
private byte[] createPngWithText(String text) throws IOException {
|
||||
BufferedImage image = new BufferedImage(1400, 240, BufferedImage.TYPE_INT_RGB);
|
||||
Graphics2D graphics = image.createGraphics();
|
||||
try {
|
||||
graphics.setColor(Color.WHITE);
|
||||
graphics.fillRect(0, 0, image.getWidth(), image.getHeight());
|
||||
graphics.setColor(Color.BLACK);
|
||||
graphics.setRenderingHint(
|
||||
RenderingHints.KEY_TEXT_ANTIALIASING, RenderingHints.VALUE_TEXT_ANTIALIAS_ON);
|
||||
graphics.setFont(new Font("Monospaced", Font.BOLD, 56));
|
||||
graphics.drawString(text, 40, 140);
|
||||
} finally {
|
||||
graphics.dispose();
|
||||
}
|
||||
|
||||
try (ByteArrayOutputStream outputStream = new ByteArrayOutputStream()) {
|
||||
ImageIO.write(image, "png", outputStream);
|
||||
return outputStream.toByteArray();
|
||||
}
|
||||
}
|
||||
|
||||
private Path createFakeTesseractHome(String extractedText) throws IOException {
|
||||
Path home = Files.createTempDirectory("fake-tesseract-home-");
|
||||
Path executable = home.resolve("tesseract");
|
||||
Files.writeString(
|
||||
executable,
|
||||
"#!/bin/sh\n"
|
||||
+ "if [ $# -eq 0 ] || [ \"$1\" = \"--version\" ]; then\n"
|
||||
+ " echo \"tesseract 5.0.0\"\n"
|
||||
+ " exit 0\n"
|
||||
+ "fi\n"
|
||||
+ "output_base=\"$2\"\n"
|
||||
+ "printf '%s\\n' \""
|
||||
+ extractedText
|
||||
+ "\" > \"${output_base}.txt\"\n",
|
||||
StandardCharsets.UTF_8);
|
||||
executable.toFile().setExecutable(true);
|
||||
return home;
|
||||
}
|
||||
|
||||
private void deleteRecursively(Path root) throws IOException {
|
||||
if (root == null || Files.notExists(root)) {
|
||||
return;
|
||||
}
|
||||
try (var paths = Files.walk(root)) {
|
||||
paths.sorted(Comparator.reverseOrder()).forEach(path -> path.toFile().delete());
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void testUploadPdfToMinIO(TestNamespace ns) throws Exception {
|
||||
byte[] content = readFixture("/drive/sample-report.pdf");
|
||||
ContextFile file;
|
||||
try (Response response = uploadUniqueFixture(ns, "/drive/sample-report.pdf", "Annual Report")) {
|
||||
file = readCreatedContextFile(response, "Upload to MinIO failed");
|
||||
assertNotNull(file.getId());
|
||||
assertNotNull(file.getAssetId(), "File should have assetId from S3 upload");
|
||||
assertNotNull(file.getHeadContentId(), "File should point at a current content snapshot");
|
||||
assertEquals("Annual Report", file.getDisplayName());
|
||||
assertEquals(content.length, file.getFileSize().intValue());
|
||||
assertStoredInMinIO(file.getAssetId(), content);
|
||||
}
|
||||
|
||||
await()
|
||||
.atMost(Duration.ofSeconds(20))
|
||||
.untilAsserted(
|
||||
() -> {
|
||||
ContextFile refreshed = fetchFile(file.getId());
|
||||
assertEquals(ProcessingStatus.Processed, refreshed.getProcessingStatus());
|
||||
assertTrue(refreshed.getExtractedText().contains("Context Center PDF Fixture"));
|
||||
assertEquals(1, refreshed.getPageCount());
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
void testUploadSpreadsheetToMinIO(TestNamespace ns) throws Exception {
|
||||
byte[] content = readFixture("/drive/sample-pricing.xlsx");
|
||||
Response response = uploadUniqueFixture(ns, "/drive/sample-pricing.xlsx", "Pricing Sheet");
|
||||
|
||||
ContextFile file = readCreatedContextFile(response, "Upload failed");
|
||||
assertNotNull(file.getAssetId());
|
||||
assertNotNull(file.getHeadContentId());
|
||||
assertEquals("Pricing Sheet", file.getDisplayName());
|
||||
assertEquals(content.length, file.getFileSize().intValue());
|
||||
}
|
||||
|
||||
@Test
|
||||
void testUploadCsvToMinIO(TestNamespace ns) throws Exception {
|
||||
byte[] content = readFixture("/drive/sample-data.csv");
|
||||
String uploadedFileName = uniqueUploadedFileName(ns, "/drive/sample-data.csv");
|
||||
Response response = uploadFixture("/drive/sample-data.csv", uploadedFileName, null, null);
|
||||
|
||||
ContextFile file = readCreatedContextFile(response, "Upload failed");
|
||||
assertNotNull(file.getAssetId());
|
||||
assertNotNull(file.getHeadContentId());
|
||||
assertEquals(uploadedFileName, file.getDisplayName());
|
||||
assertEquals(content.length, file.getFileSize().intValue());
|
||||
}
|
||||
|
||||
@Test
|
||||
void testUploadVerifyFileSize(TestNamespace ns) throws Exception {
|
||||
byte[] contentBytes = readFixture("/drive/sample-notes.txt");
|
||||
try (Response response = uploadUniqueFixture(ns, "/drive/sample-notes.txt", "Sized File")) {
|
||||
ContextFile file = readCreatedContextFile(response, "Upload failed");
|
||||
assertEquals(
|
||||
contentBytes.length,
|
||||
file.getFileSize().intValue(),
|
||||
"File size should match uploaded bytes");
|
||||
assertEquals("txt", file.getFileExtension());
|
||||
assertEquals(ProcessingStatus.Uploaded, file.getProcessingStatus());
|
||||
assertNotNull(file.getHeadContentId());
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void testUploadedTextFileIsSearchableByExtractedText(TestNamespace ns) throws Exception {
|
||||
String uniqueToken = "contextneedle" + UUID.randomUUID().toString().replace("-", "");
|
||||
byte[] content =
|
||||
("User supplied context that should be searchable " + uniqueToken)
|
||||
.getBytes(StandardCharsets.UTF_8);
|
||||
|
||||
ContextFile file;
|
||||
try (Response response = uploadFile("search-fixture.txt", content, "Search Fixture", null)) {
|
||||
String body = response.readEntity(String.class);
|
||||
assertEquals(CREATED.getStatusCode(), response.getStatus(), "Upload failed: " + body);
|
||||
file = JsonUtils.readValue(body, ContextFile.class);
|
||||
}
|
||||
|
||||
await()
|
||||
.atMost(Duration.ofSeconds(20))
|
||||
.untilAsserted(
|
||||
() -> {
|
||||
ContextFile refreshed = fetchFile(file.getId());
|
||||
assertEquals(ProcessingStatus.Processed, refreshed.getProcessingStatus());
|
||||
assertTrue(refreshed.getExtractedText().contains(uniqueToken));
|
||||
});
|
||||
|
||||
assertSearchContainsFile(uniqueToken, file.getId());
|
||||
}
|
||||
|
||||
@Test
|
||||
void testUploadedPdfIsSearchableByExtractedText(TestNamespace ns) throws Exception {
|
||||
String uniqueToken = "pdfneedle" + UUID.randomUUID().toString().replace("-", "");
|
||||
byte[] content = createPdf("Quarterly context for " + uniqueToken);
|
||||
|
||||
ContextFile file;
|
||||
try (Response response =
|
||||
uploadFile("search-fixture.pdf", content, ns.shortPrefix("PDF Search"), null)) {
|
||||
String body = response.readEntity(String.class);
|
||||
assertEquals(CREATED.getStatusCode(), response.getStatus(), "Upload failed: " + body);
|
||||
file = JsonUtils.readValue(body, ContextFile.class);
|
||||
}
|
||||
|
||||
await()
|
||||
.atMost(Duration.ofSeconds(20))
|
||||
.untilAsserted(
|
||||
() -> {
|
||||
ContextFile refreshed = fetchFile(file.getId());
|
||||
assertEquals(ProcessingStatus.Processed, refreshed.getProcessingStatus());
|
||||
assertTrue(refreshed.getExtractedText().contains(uniqueToken));
|
||||
});
|
||||
|
||||
assertSearchContainsFile(uniqueToken, file.getId());
|
||||
}
|
||||
|
||||
@Test
|
||||
void testUploadedSpreadsheetIsSearchableByExtractedText(TestNamespace ns) throws Exception {
|
||||
String uniqueToken = "sheetneedle" + UUID.randomUUID().toString().replace("-", "");
|
||||
byte[] content = createWorkbook("Pricing", "SearchToken", uniqueToken);
|
||||
|
||||
ContextFile file;
|
||||
try (Response response =
|
||||
uploadFile("search-fixture.xlsx", content, ns.shortPrefix("Spreadsheet Search"), null)) {
|
||||
String body = response.readEntity(String.class);
|
||||
assertEquals(CREATED.getStatusCode(), response.getStatus(), "Upload failed: " + body);
|
||||
file = JsonUtils.readValue(body, ContextFile.class);
|
||||
}
|
||||
|
||||
await()
|
||||
.atMost(Duration.ofSeconds(20))
|
||||
.untilAsserted(
|
||||
() -> {
|
||||
ContextFile refreshed = fetchFile(file.getId());
|
||||
assertEquals(ProcessingStatus.Processed, refreshed.getProcessingStatus());
|
||||
assertTrue(refreshed.getExtractedText().contains(uniqueToken));
|
||||
});
|
||||
|
||||
assertSearchContainsFile(uniqueToken, file.getId());
|
||||
}
|
||||
|
||||
@Test
|
||||
void testUploadedImageIsSearchableByOcrExtractedText(TestNamespace ns) throws Exception {
|
||||
String uniqueToken =
|
||||
"IMAGENEEDLE"
|
||||
+ UUID.randomUUID().toString().replace("-", "").substring(0, 10).toUpperCase();
|
||||
Path fakeTesseractHome = createFakeTesseractHome("Revenue chart " + uniqueToken);
|
||||
String originalPath = System.getProperty(TIKA_TESSERACT_PATH_PROPERTY);
|
||||
|
||||
try {
|
||||
System.setProperty(TIKA_TESSERACT_PATH_PROPERTY, fakeTesseractHome.toString());
|
||||
byte[] content = createPngWithText(uniqueToken);
|
||||
|
||||
ContextFile file;
|
||||
try (Response response =
|
||||
uploadFile("search-fixture.png", content, ns.shortPrefix("Image Search"), null)) {
|
||||
String body = response.readEntity(String.class);
|
||||
assertEquals(CREATED.getStatusCode(), response.getStatus(), "Upload failed: " + body);
|
||||
file = JsonUtils.readValue(body, ContextFile.class);
|
||||
}
|
||||
|
||||
await()
|
||||
.atMost(Duration.ofSeconds(20))
|
||||
.untilAsserted(
|
||||
() -> {
|
||||
ContextFile refreshed = fetchFile(file.getId());
|
||||
assertEquals(ProcessingStatus.Processed, refreshed.getProcessingStatus());
|
||||
assertTrue(refreshed.getExtractedText().contains(uniqueToken));
|
||||
});
|
||||
|
||||
assertSearchContainsFile(uniqueToken, file.getId());
|
||||
} finally {
|
||||
if (originalPath == null) {
|
||||
System.clearProperty(TIKA_TESSERACT_PATH_PROPERTY);
|
||||
} else {
|
||||
System.setProperty(TIKA_TESSERACT_PATH_PROPERTY, originalPath);
|
||||
}
|
||||
deleteRecursively(fakeTesseractHome);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void testUploadFileIntoFolder(TestNamespace ns) throws Exception {
|
||||
RestClient rest = RestClient.admin();
|
||||
Folder folder =
|
||||
rest.create(
|
||||
"v1/contextCenter/drive/folders",
|
||||
new CreateFolder().withName(ns.prefix("upload-target-folder")),
|
||||
Folder.class);
|
||||
|
||||
Response response =
|
||||
uploadFixture(
|
||||
"/drive/sample-report.pdf",
|
||||
"nested.pdf",
|
||||
"File In Folder",
|
||||
folder.getFullyQualifiedName());
|
||||
|
||||
String body = response.readEntity(String.class);
|
||||
assertEquals(CREATED.getStatusCode(), response.getStatus(), "Upload failed: " + body);
|
||||
|
||||
ContextFile file = JsonUtils.readValue(body, ContextFile.class);
|
||||
assertNotNull(file.getAssetId());
|
||||
assertNotNull(file.getHeadContentId());
|
||||
|
||||
ContextFile fetched =
|
||||
rest.getById("v1/contextCenter/drive/files", file.getId(), "folder", ContextFile.class);
|
||||
assertNotNull(fetched.getFolder(), "File should be in folder");
|
||||
assertEquals(folder.getId(), fetched.getFolder().getId());
|
||||
}
|
||||
|
||||
@Test
|
||||
void testUploadRejectsDuplicateFileNameCaseInsensitiveInRoot(TestNamespace ns) throws Exception {
|
||||
byte[] content = readFixture("/drive/sample-report.pdf");
|
||||
|
||||
try (Response resp1 = uploadFile("Duplicate.PDF", content, "First Title", null);
|
||||
Response resp2 = uploadFile("duplicate.pdf", content, "Different Title", null)) {
|
||||
String body1 = resp1.readEntity(String.class);
|
||||
String body2 = resp2.readEntity(String.class);
|
||||
|
||||
assertEquals(CREATED.getStatusCode(), resp1.getStatus(), "First upload failed: " + body1);
|
||||
ContextFile file = JsonUtils.readValue(body1, ContextFile.class);
|
||||
assertEquals("First Title", file.getDisplayName());
|
||||
assertStoredInMinIO(file.getAssetId(), content);
|
||||
|
||||
assertEquals(Response.Status.BAD_REQUEST.getStatusCode(), resp2.getStatus(), body2);
|
||||
assertTrue(body2.contains("duplicate.pdf"));
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void testUploadRejectsDuplicateFileNameCaseInsensitiveInSameFolder(TestNamespace ns)
|
||||
throws Exception {
|
||||
byte[] content = "nested duplicate content".getBytes(StandardCharsets.UTF_8);
|
||||
Folder folder = createFolder(ns, "duplicate-upload-folder");
|
||||
|
||||
try (Response resp1 =
|
||||
uploadFile(
|
||||
"NestedDuplicate.txt", content, "Nested First", folder.getFullyQualifiedName());
|
||||
Response resp2 =
|
||||
uploadFile(
|
||||
"nestedduplicate.TXT",
|
||||
content,
|
||||
"Nested Different",
|
||||
folder.getFullyQualifiedName())) {
|
||||
String body1 = resp1.readEntity(String.class);
|
||||
String body2 = resp2.readEntity(String.class);
|
||||
|
||||
assertEquals(CREATED.getStatusCode(), resp1.getStatus(), "First upload failed: " + body1);
|
||||
ContextFile file = JsonUtils.readValue(body1, ContextFile.class);
|
||||
assertEquals("Nested First", file.getDisplayName());
|
||||
assertEquals(folder.getId(), file.getFolder().getId());
|
||||
assertStoredInMinIO(file.getAssetId(), content);
|
||||
|
||||
assertEquals(Response.Status.BAD_REQUEST.getStatusCode(), resp2.getStatus(), body2);
|
||||
assertTrue(body2.contains("nestedduplicate.txt"));
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void testUploadRejectsDuplicateSanitizedFileName() throws Exception {
|
||||
byte[] content = "duplicate sanitized content".getBytes(StandardCharsets.UTF_8);
|
||||
String fileName = "Quarterly Report (" + UUID.randomUUID() + ").pdf";
|
||||
String expectedName =
|
||||
fileName.replaceAll("[^a-zA-Z0-9._-]", "_").replaceAll("_+", "_").toLowerCase();
|
||||
|
||||
try (Response resp1 = uploadFile(fileName, content, "First Title", null);
|
||||
Response resp2 = uploadFile(fileName, content, "Different Title", null)) {
|
||||
String body1 = resp1.readEntity(String.class);
|
||||
String body2 = resp2.readEntity(String.class);
|
||||
|
||||
assertEquals(CREATED.getStatusCode(), resp1.getStatus(), "First upload failed: " + body1);
|
||||
ContextFile file = JsonUtils.readValue(body1, ContextFile.class);
|
||||
assertEquals(expectedName, file.getName());
|
||||
assertStoredInMinIO(file.getAssetId(), content);
|
||||
|
||||
assertEquals(Response.Status.BAD_REQUEST.getStatusCode(), resp2.getStatus(), body2);
|
||||
assertTrue(body2.contains(expectedName));
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void testUploadAllowsSameFileNameInDifferentFolders(TestNamespace ns) throws Exception {
|
||||
byte[] content = "shared name different folders".getBytes(StandardCharsets.UTF_8);
|
||||
Folder firstFolder = createFolder(ns, "shared-name-first-folder");
|
||||
Folder secondFolder = createFolder(ns, "shared-name-second-folder");
|
||||
|
||||
try (Response resp1 =
|
||||
uploadFile(
|
||||
"shared-name.txt",
|
||||
content,
|
||||
"First Folder Title",
|
||||
firstFolder.getFullyQualifiedName());
|
||||
Response resp2 =
|
||||
uploadFile(
|
||||
"SHARED-NAME.txt",
|
||||
content,
|
||||
"Second Folder Different Title",
|
||||
secondFolder.getFullyQualifiedName())) {
|
||||
String body1 = resp1.readEntity(String.class);
|
||||
String body2 = resp2.readEntity(String.class);
|
||||
|
||||
assertEquals(CREATED.getStatusCode(), resp1.getStatus(), "First upload failed: " + body1);
|
||||
assertEquals(CREATED.getStatusCode(), resp2.getStatus(), "Second upload failed: " + body2);
|
||||
|
||||
ContextFile firstFile = JsonUtils.readValue(body1, ContextFile.class);
|
||||
ContextFile secondFile = JsonUtils.readValue(body2, ContextFile.class);
|
||||
assertEquals(firstFolder.getId(), firstFile.getFolder().getId());
|
||||
assertEquals(secondFolder.getId(), secondFile.getFolder().getId());
|
||||
assertEquals("First Folder Title", firstFile.getDisplayName());
|
||||
assertEquals("Second Folder Different Title", secondFile.getDisplayName());
|
||||
assertStoredInMinIO(firstFile.getAssetId(), content);
|
||||
assertStoredInMinIO(secondFile.getAssetId(), content);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void testUploadLargeFileRejected(TestNamespace ns) throws Exception {
|
||||
Response response =
|
||||
uploadFile("too_large.jpg", readFixture("/2mb-jpg-example-file.jpg"), "Too Large", null);
|
||||
|
||||
assertTrue(
|
||||
response.getStatus() >= 400,
|
||||
"Oversized upload should be rejected, got " + response.getStatus());
|
||||
}
|
||||
|
||||
@Test
|
||||
void testUploadDetectsFileType(TestNamespace ns) throws Exception {
|
||||
ContextFile pdf;
|
||||
ContextFile csv;
|
||||
ContextFile spreadsheet;
|
||||
ContextFile text;
|
||||
try (Response pdfResp = uploadUniqueFixture(ns, "/drive/sample-report.pdf", "PDF Test");
|
||||
Response csvResp = uploadUniqueFixture(ns, "/drive/sample-data.csv", "CSV Test");
|
||||
Response spreadsheetResp =
|
||||
uploadUniqueFixture(ns, "/drive/sample-pricing.xlsx", "Spreadsheet Test");
|
||||
Response textResp = uploadUniqueFixture(ns, "/drive/sample-notes.txt", "Text Test")) {
|
||||
pdf = readCreatedContextFile(pdfResp, "PDF upload failed");
|
||||
csv = readCreatedContextFile(csvResp, "CSV upload failed");
|
||||
spreadsheet = readCreatedContextFile(spreadsheetResp, "Spreadsheet upload failed");
|
||||
text = readCreatedContextFile(textResp, "Text upload failed");
|
||||
}
|
||||
|
||||
assertEquals(ContextFileType.PDF, pdf.getFileType());
|
||||
assertEquals(ContextFileType.CSV, csv.getFileType());
|
||||
assertEquals(ContextFileType.Spreadsheet, spreadsheet.getFileType());
|
||||
assertEquals(ContextFileType.Text, text.getFileType());
|
||||
}
|
||||
|
||||
@Test
|
||||
void testDownloadUploadedFileThroughContextFileEndpoint(TestNamespace ns) throws Exception {
|
||||
byte[] content = readFixture("/drive/sample-notes.txt");
|
||||
|
||||
Response uploadResponse = uploadUniqueFixture(ns, "/drive/sample-notes.txt", "Download Test");
|
||||
ContextFile file = readCreatedContextFile(uploadResponse, "Upload failed");
|
||||
|
||||
await()
|
||||
.pollDelay(Duration.ZERO)
|
||||
.pollInterval(Duration.ofMillis(200))
|
||||
.atMost(Duration.ofSeconds(20))
|
||||
.untilAsserted(
|
||||
() -> {
|
||||
try (Response downloadResponse =
|
||||
multipartClient
|
||||
.target(
|
||||
serverBaseUrl
|
||||
+ "/api/v1/contextCenter/drive/files/"
|
||||
+ file.getId()
|
||||
+ "/download?redirect=false")
|
||||
.request()
|
||||
.headers(adminAuthHeaders())
|
||||
.get();
|
||||
InputStream downloaded = downloadResponse.readEntity(InputStream.class)) {
|
||||
assertEquals(200, downloadResponse.getStatus());
|
||||
assertArrayEquals(content, downloaded.readAllBytes());
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
void testBulkDownloadUploadedFilesAsZip(TestNamespace ns) throws Exception {
|
||||
byte[] firstContent = "first bulk download".getBytes(StandardCharsets.UTF_8);
|
||||
byte[] secondContent = "second bulk download".getBytes(StandardCharsets.UTF_8);
|
||||
|
||||
ContextFile firstFile;
|
||||
try (Response response = uploadFile("bulk-one.txt", firstContent, "Bulk One", null)) {
|
||||
String body = response.readEntity(String.class);
|
||||
assertEquals(CREATED.getStatusCode(), response.getStatus(), "Upload failed: " + body);
|
||||
firstFile = JsonUtils.readValue(body, ContextFile.class);
|
||||
}
|
||||
|
||||
ContextFile secondFile;
|
||||
try (Response response = uploadFile("bulk-two.txt", secondContent, "Bulk Two", null)) {
|
||||
String body = response.readEntity(String.class);
|
||||
assertEquals(CREATED.getStatusCode(), response.getStatus(), "Upload failed: " + body);
|
||||
secondFile = JsonUtils.readValue(body, ContextFile.class);
|
||||
}
|
||||
|
||||
Map<String, Object> request =
|
||||
Map.of("ids", List.of(firstFile.getId().toString(), secondFile.getId().toString()));
|
||||
try (Response downloadResponse =
|
||||
multipartClient
|
||||
.target(serverBaseUrl + "/api/v1/contextCenter/drive/files/bulk/download")
|
||||
.request()
|
||||
.headers(adminAuthHeaders())
|
||||
.post(Entity.entity(JsonUtils.pojoToJson(request), MediaType.APPLICATION_JSON));
|
||||
ZipInputStream zipInputStream =
|
||||
new ZipInputStream(downloadResponse.readEntity(InputStream.class))) {
|
||||
assertEquals(200, downloadResponse.getStatus());
|
||||
|
||||
Map<String, byte[]> entries = new HashMap<>();
|
||||
ZipEntry entry;
|
||||
while ((entry = zipInputStream.getNextEntry()) != null) {
|
||||
entries.put(entry.getName(), zipInputStream.readAllBytes());
|
||||
zipInputStream.closeEntry();
|
||||
}
|
||||
|
||||
assertArrayEquals(firstContent, entries.get("bulk-one.txt"));
|
||||
assertArrayEquals(secondContent, entries.get("bulk-two.txt"));
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void testBulkDownloadDisambiguatesGeneratedZipEntryCollisions(TestNamespace ns) throws Exception {
|
||||
byte[] firstContent = "first collision download".getBytes(StandardCharsets.UTF_8);
|
||||
byte[] explicitContent = "explicit collision download".getBytes(StandardCharsets.UTF_8);
|
||||
byte[] secondContent = "second collision download".getBytes(StandardCharsets.UTF_8);
|
||||
Folder firstFolder = createFolder(ns, "zip-collision-first-folder");
|
||||
Folder explicitFolder = createFolder(ns, "zip-collision-explicit-folder");
|
||||
Folder secondFolder = createFolder(ns, "zip-collision-second-folder");
|
||||
|
||||
ContextFile firstFile;
|
||||
try (Response response =
|
||||
uploadFile(
|
||||
"collision.txt",
|
||||
firstContent,
|
||||
"Collision First",
|
||||
firstFolder.getFullyQualifiedName())) {
|
||||
String body = response.readEntity(String.class);
|
||||
assertEquals(CREATED.getStatusCode(), response.getStatus(), "Upload failed: " + body);
|
||||
firstFile = JsonUtils.readValue(body, ContextFile.class);
|
||||
}
|
||||
|
||||
ContextFile explicitFile;
|
||||
try (Response response =
|
||||
uploadFile(
|
||||
"collision (2).txt",
|
||||
explicitContent,
|
||||
"Collision Explicit",
|
||||
explicitFolder.getFullyQualifiedName())) {
|
||||
String body = response.readEntity(String.class);
|
||||
assertEquals(CREATED.getStatusCode(), response.getStatus(), "Upload failed: " + body);
|
||||
explicitFile = JsonUtils.readValue(body, ContextFile.class);
|
||||
}
|
||||
|
||||
ContextFile secondFile;
|
||||
try (Response response =
|
||||
uploadFile(
|
||||
"collision.txt",
|
||||
secondContent,
|
||||
"Collision Second",
|
||||
secondFolder.getFullyQualifiedName())) {
|
||||
String body = response.readEntity(String.class);
|
||||
assertEquals(CREATED.getStatusCode(), response.getStatus(), "Upload failed: " + body);
|
||||
secondFile = JsonUtils.readValue(body, ContextFile.class);
|
||||
}
|
||||
|
||||
Map<String, Object> request =
|
||||
Map.of(
|
||||
"ids",
|
||||
List.of(
|
||||
firstFile.getId().toString(),
|
||||
explicitFile.getId().toString(),
|
||||
secondFile.getId().toString()));
|
||||
try (Response downloadResponse =
|
||||
multipartClient
|
||||
.target(serverBaseUrl + "/api/v1/contextCenter/drive/files/bulk/download")
|
||||
.request()
|
||||
.headers(adminAuthHeaders())
|
||||
.post(Entity.entity(JsonUtils.pojoToJson(request), MediaType.APPLICATION_JSON));
|
||||
ZipInputStream zipInputStream =
|
||||
new ZipInputStream(downloadResponse.readEntity(InputStream.class))) {
|
||||
assertEquals(200, downloadResponse.getStatus());
|
||||
|
||||
Map<String, byte[]> entries = new HashMap<>();
|
||||
ZipEntry entry;
|
||||
while ((entry = zipInputStream.getNextEntry()) != null) {
|
||||
entries.put(entry.getName(), zipInputStream.readAllBytes());
|
||||
zipInputStream.closeEntry();
|
||||
}
|
||||
|
||||
assertEquals(3, entries.size());
|
||||
assertArrayEquals(firstContent, entries.get("collision.txt"));
|
||||
assertArrayEquals(explicitContent, entries.get("collision (2).txt"));
|
||||
assertArrayEquals(secondContent, entries.get("collision (3).txt"));
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void testDownloadUploadedFileThroughSignedRedirect(TestNamespace ns) throws Exception {
|
||||
byte[] content = readFixture("/drive/sample-notes.txt");
|
||||
|
||||
Response uploadResponse =
|
||||
uploadUniqueFixture(ns, "/drive/sample-notes.txt", "Redirect Download");
|
||||
ContextFile file = readCreatedContextFile(uploadResponse, "Upload failed");
|
||||
|
||||
await()
|
||||
.pollDelay(Duration.ZERO)
|
||||
.pollInterval(Duration.ofMillis(200))
|
||||
.atMost(Duration.ofSeconds(20))
|
||||
.untilAsserted(
|
||||
() -> {
|
||||
try (Response redirectResponse =
|
||||
multipartClient
|
||||
.target(
|
||||
serverBaseUrl
|
||||
+ "/api/v1/contextCenter/drive/files/"
|
||||
+ file.getId()
|
||||
+ "/download")
|
||||
.property(ClientProperties.FOLLOW_REDIRECTS, false)
|
||||
.request()
|
||||
.headers(adminAuthHeaders())
|
||||
.get();
|
||||
Client signedUrlClient = ClientBuilder.newClient()) {
|
||||
assertEquals(307, redirectResponse.getStatus());
|
||||
String signedUrl = redirectResponse.getHeaderString("Location");
|
||||
assertNotNull(signedUrl);
|
||||
|
||||
try (Response signedDownload = signedUrlClient.target(signedUrl).request().get();
|
||||
InputStream downloaded = signedDownload.readEntity(InputStream.class)) {
|
||||
assertEquals(200, signedDownload.getStatus());
|
||||
assertArrayEquals(content, downloaded.readAllBytes());
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
void testSoftDeletedFileCanDownloadFromTrash(TestNamespace ns) throws Exception {
|
||||
byte[] content = readFixture("/drive/sample-notes.txt");
|
||||
RestClient rest = RestClient.admin();
|
||||
|
||||
Response uploadResponse = uploadUniqueFixture(ns, "/drive/sample-notes.txt", "Trash Download");
|
||||
ContextFile file = readCreatedContextFile(uploadResponse, "Upload failed");
|
||||
rest.delete("v1/contextCenter/drive/files", file.getId());
|
||||
|
||||
try (Response downloadResponse =
|
||||
multipartClient
|
||||
.target(
|
||||
serverBaseUrl
|
||||
+ "/api/v1/contextCenter/drive/files/"
|
||||
+ file.getId()
|
||||
+ "/download?include=all&redirect=false")
|
||||
.request()
|
||||
.headers(adminAuthHeaders())
|
||||
.get();
|
||||
InputStream downloaded = downloadResponse.readEntity(InputStream.class)) {
|
||||
assertEquals(200, downloadResponse.getStatus());
|
||||
assertArrayEquals(content, downloaded.readAllBytes());
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void testHardDeleteRemovesObjectFromMinIO(TestNamespace ns) throws Exception {
|
||||
byte[] content = readFixture("/drive/sample-notes.txt");
|
||||
RestClient rest = RestClient.admin();
|
||||
|
||||
Response uploadResponse = uploadUniqueFixture(ns, "/drive/sample-notes.txt", "Hard Delete");
|
||||
ContextFile file = readCreatedContextFile(uploadResponse, "Upload failed");
|
||||
assertStoredInMinIO(file.getAssetId(), content);
|
||||
|
||||
rest.hardDelete("v1/contextCenter/drive/files", file.getId());
|
||||
|
||||
// Hard delete is asynchronous: the server returns 200 immediately, then a background
|
||||
// worker soft-deletes (if needed), removes search/relationship state, drops the row,
|
||||
// and unlinks the object from MinIO. Under CI load this chain can take well over 10s,
|
||||
// so poll with a generous ceiling rather than gambling on a tight window.
|
||||
await()
|
||||
.pollInterval(Duration.ofMillis(200))
|
||||
.atMost(Duration.ofSeconds(30))
|
||||
.untilAsserted(
|
||||
() -> {
|
||||
try (Response deletedResponse =
|
||||
rest.rawGet("v1/contextCenter/drive/files/" + file.getId() + "?include=all")) {
|
||||
assertEquals(404, deletedResponse.getStatus());
|
||||
}
|
||||
});
|
||||
assertRemovedFromMinIO(file.getAssetId());
|
||||
}
|
||||
}
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
package org.openmetadata.it.drive;
|
||||
|
||||
import org.openmetadata.schema.api.teams.CreateUser;
|
||||
import org.openmetadata.schema.entity.teams.User;
|
||||
import org.openmetadata.sdk.services.teams.UserService;
|
||||
import org.openmetadata.sdk.test.util.SdkClients;
|
||||
import org.openmetadata.sdk.test.util.TestNamespace;
|
||||
|
||||
final class DriveTestUsers {
|
||||
|
||||
private DriveTestUsers() {}
|
||||
|
||||
static User createUser(TestNamespace ns, String suffix) {
|
||||
String base = (ns.shortPrefix("drive") + suffix).replaceAll("[^a-zA-Z0-9]", "").toLowerCase();
|
||||
String name = base.substring(0, Math.min(base.length(), 48));
|
||||
CreateUser createUser =
|
||||
new CreateUser()
|
||||
.withName(name)
|
||||
.withDisplayName(name)
|
||||
.withEmail(name + "@test.openmetadata.org");
|
||||
return new UserService(SdkClients.adminClient().getHttpClient()).create(createUser);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,384 @@
|
||||
package org.openmetadata.it.drive;
|
||||
|
||||
import static org.awaitility.Awaitility.await;
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertNotNull;
|
||||
import static org.junit.jupiter.api.Assertions.assertThrows;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
|
||||
import jakarta.ws.rs.core.Response;
|
||||
import java.time.Duration;
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
import org.apache.http.client.HttpResponseException;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.extension.ExtendWith;
|
||||
import org.junit.jupiter.api.parallel.Execution;
|
||||
import org.junit.jupiter.api.parallel.ExecutionMode;
|
||||
import org.openmetadata.schema.api.data.CreateContextFile;
|
||||
import org.openmetadata.schema.api.data.CreateFolder;
|
||||
import org.openmetadata.schema.entity.data.ContextFile;
|
||||
import org.openmetadata.schema.entity.data.ContextFileType;
|
||||
import org.openmetadata.schema.entity.data.Folder;
|
||||
import org.openmetadata.schema.entity.data.ProcessingStatus;
|
||||
import org.openmetadata.schema.entity.teams.User;
|
||||
import org.openmetadata.schema.utils.JsonUtils;
|
||||
import org.openmetadata.sdk.client.OpenMetadataClient;
|
||||
import org.openmetadata.sdk.services.teams.UserService;
|
||||
import org.openmetadata.sdk.test.util.RestClient;
|
||||
import org.openmetadata.sdk.test.util.SdkClients;
|
||||
import org.openmetadata.sdk.test.util.TestNamespace;
|
||||
import org.openmetadata.sdk.test.util.TestNamespaceExtension;
|
||||
|
||||
@ExtendWith(TestNamespaceExtension.class)
|
||||
class FolderIT {
|
||||
|
||||
private static final String PATH = "v1/contextCenter/drive/folders";
|
||||
|
||||
private Folder createFolder(RestClient rest, CreateFolder request) throws HttpResponseException {
|
||||
return rest.create(PATH, request, Folder.class);
|
||||
}
|
||||
|
||||
private Folder getFolder(RestClient rest, UUID id, String fields) throws HttpResponseException {
|
||||
return rest.getById(PATH, id, fields, Folder.class);
|
||||
}
|
||||
|
||||
private Folder patchFolder(RestClient rest, UUID id, String origJson, Folder updated)
|
||||
throws HttpResponseException {
|
||||
return rest.patch(PATH, id, origJson, updated, Folder.class);
|
||||
}
|
||||
|
||||
// --- CRUD ---
|
||||
|
||||
@Test
|
||||
void testCreateFolder(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
|
||||
CreateFolder create =
|
||||
new CreateFolder().withName(ns.prefix("my-folder")).withDisplayName("My Folder");
|
||||
|
||||
Folder folder = createFolder(rest, create);
|
||||
assertNotNull(folder.getId());
|
||||
assertEquals("My Folder", folder.getDisplayName());
|
||||
}
|
||||
|
||||
@Test
|
||||
void testGetFolderById(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
|
||||
Folder created = createFolder(rest, new CreateFolder().withName(ns.prefix("get-test")));
|
||||
|
||||
Folder fetched = getFolder(rest, created.getId(), "");
|
||||
assertEquals(created.getId(), fetched.getId());
|
||||
assertEquals(created.getName(), fetched.getName());
|
||||
}
|
||||
|
||||
@Test
|
||||
void testUpdateFolderDisplayName(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
|
||||
Folder folder =
|
||||
createFolder(
|
||||
rest,
|
||||
new CreateFolder().withName(ns.prefix("update-test")).withDisplayName("Original Name"));
|
||||
|
||||
String original = JsonUtils.pojoToJson(folder);
|
||||
folder.setDisplayName("Updated Name");
|
||||
Folder updated = patchFolder(rest, folder.getId(), original, folder);
|
||||
|
||||
assertEquals("Updated Name", updated.getDisplayName());
|
||||
}
|
||||
|
||||
@Test
|
||||
void testDeleteFolder(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
|
||||
Folder folder = createFolder(rest, new CreateFolder().withName(ns.prefix("delete-test")));
|
||||
|
||||
rest.delete(PATH, folder.getId());
|
||||
|
||||
HttpResponseException ex =
|
||||
assertThrows(HttpResponseException.class, () -> getFolder(rest, folder.getId(), ""));
|
||||
assertEquals(404, ex.getStatusCode());
|
||||
|
||||
try (Response deletedResponse = rest.rawGet(PATH + "/" + folder.getId() + "?include=all")) {
|
||||
assertEquals(200, deletedResponse.getStatus());
|
||||
assertTrue(deletedResponse.readEntity(String.class).contains("\"deleted\":true"));
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void testRestoreSoftDeletedFolder(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
|
||||
Folder folder = createFolder(rest, new CreateFolder().withName(ns.prefix("restore-folder")));
|
||||
rest.delete(PATH, folder.getId());
|
||||
|
||||
Folder restored = rest.restore(PATH, folder.getId(), Folder.class);
|
||||
assertEquals(folder.getId(), restored.getId());
|
||||
assertTrue(!Boolean.TRUE.equals(restored.getDeleted()));
|
||||
}
|
||||
|
||||
@Test
|
||||
void testHardDeleteFolderIsAsync(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
|
||||
Folder folder =
|
||||
createFolder(rest, new CreateFolder().withName(ns.prefix("hard-delete-folder")));
|
||||
|
||||
try (Response deleteResponse =
|
||||
rest.rawDelete(PATH + "/" + folder.getId() + "?hardDelete=true&recursive=true")) {
|
||||
assertEquals(202, deleteResponse.getStatus());
|
||||
assertTrue(deleteResponse.readEntity(String.class).contains("\"hardDelete\":true"));
|
||||
}
|
||||
|
||||
await()
|
||||
.atMost(Duration.ofSeconds(20))
|
||||
.untilAsserted(
|
||||
() -> {
|
||||
try (Response deletedResponse =
|
||||
rest.rawGet(PATH + "/" + folder.getId() + "?include=all")) {
|
||||
assertEquals(404, deletedResponse.getStatus());
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// --- Nested Folder Hierarchy ---
|
||||
|
||||
@Test
|
||||
void testNestedFolders(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
|
||||
Folder root =
|
||||
createFolder(
|
||||
rest, new CreateFolder().withName(ns.prefix("root")).withDisplayName("Root Folder"));
|
||||
|
||||
Folder child =
|
||||
createFolder(
|
||||
rest,
|
||||
new CreateFolder()
|
||||
.withName(ns.prefix("child"))
|
||||
.withDisplayName("Child Folder")
|
||||
.withParent(root.getFullyQualifiedName()));
|
||||
|
||||
Folder grandchild =
|
||||
createFolder(
|
||||
rest,
|
||||
new CreateFolder()
|
||||
.withName(ns.prefix("grandchild"))
|
||||
.withDisplayName("Grandchild Folder")
|
||||
.withParent(child.getFullyQualifiedName()));
|
||||
|
||||
// Verify parent-child
|
||||
Folder fetchedChild = getFolder(rest, child.getId(), "parent");
|
||||
assertNotNull(fetchedChild.getParent());
|
||||
assertEquals(root.getId(), fetchedChild.getParent().getId());
|
||||
|
||||
// Verify FQN includes full path
|
||||
Folder fetchedGrandchild = getFolder(rest, grandchild.getId(), "parent");
|
||||
assertTrue(
|
||||
fetchedGrandchild.getFullyQualifiedName().contains(root.getName()),
|
||||
"Grandchild FQN should contain root folder name");
|
||||
assertTrue(
|
||||
fetchedGrandchild.getFullyQualifiedName().contains(child.getName()),
|
||||
"Grandchild FQN should contain child folder name");
|
||||
}
|
||||
|
||||
@Test
|
||||
void testFolderWithChildren(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
|
||||
Folder parent = createFolder(rest, new CreateFolder().withName(ns.prefix("parent-list")));
|
||||
|
||||
createFolder(
|
||||
rest,
|
||||
new CreateFolder()
|
||||
.withName(ns.prefix("child-1"))
|
||||
.withParent(parent.getFullyQualifiedName()));
|
||||
createFolder(
|
||||
rest,
|
||||
new CreateFolder()
|
||||
.withName(ns.prefix("child-2"))
|
||||
.withParent(parent.getFullyQualifiedName()));
|
||||
|
||||
Folder fetched = getFolder(rest, parent.getId(), "children");
|
||||
assertNotNull(fetched.getChildren());
|
||||
assertEquals(2, fetched.getChildren().size());
|
||||
}
|
||||
|
||||
// --- Ownership (personal vs team folder) ---
|
||||
|
||||
@Test
|
||||
void testFolderWithUserOwner(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
OpenMetadataClient adminClient = SdkClients.adminClient();
|
||||
UserService userSvc = new UserService(adminClient.getHttpClient());
|
||||
User admin = userSvc.getByName("admin", null);
|
||||
|
||||
Folder folder =
|
||||
createFolder(
|
||||
rest,
|
||||
new CreateFolder()
|
||||
.withName(ns.prefix("personal"))
|
||||
.withDisplayName("My Personal Docs")
|
||||
.withOwners(List.of(admin.getEntityReference())));
|
||||
|
||||
Folder fetched = getFolder(rest, folder.getId(), "owners");
|
||||
assertNotNull(fetched.getOwners());
|
||||
assertEquals(1, fetched.getOwners().size());
|
||||
assertEquals(admin.getId(), fetched.getOwners().get(0).getId());
|
||||
}
|
||||
|
||||
// --- Permissions ---
|
||||
|
||||
@Test
|
||||
void testUnprivilegedUserCannotDeleteFolder(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient adminRest = RestClient.admin();
|
||||
User owner = DriveTestUsers.createUser(ns, "folder-owner");
|
||||
|
||||
Folder folder =
|
||||
createFolder(
|
||||
adminRest,
|
||||
new CreateFolder()
|
||||
.withName(ns.prefix("perm-delete"))
|
||||
.withOwners(List.of(owner.getEntityReference())));
|
||||
|
||||
RestClient consumerRest = RestClient.forUser("test@open-metadata.org", new String[] {});
|
||||
|
||||
HttpResponseException ex =
|
||||
assertThrows(
|
||||
HttpResponseException.class, () -> consumerRest.hardDelete(PATH, folder.getId()));
|
||||
|
||||
assertTrue(
|
||||
ex.getStatusCode() == 403 || ex.getStatusCode() == 401,
|
||||
"Expected 403 or 401, got " + ex.getStatusCode());
|
||||
}
|
||||
|
||||
@Test
|
||||
void testUnprivilegedUserCannotUpdateOthersFolder(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient adminRest = RestClient.admin();
|
||||
User owner = DriveTestUsers.createUser(ns, "folder-editor");
|
||||
|
||||
Folder folder =
|
||||
createFolder(
|
||||
adminRest,
|
||||
new CreateFolder()
|
||||
.withName(ns.prefix("perm-update"))
|
||||
.withDisplayName("Original")
|
||||
.withOwners(List.of(owner.getEntityReference())));
|
||||
|
||||
RestClient consumerRest = RestClient.forUser("test@open-metadata.org", new String[] {});
|
||||
|
||||
String original = JsonUtils.pojoToJson(folder);
|
||||
folder.setDisplayName("Hacked Name");
|
||||
|
||||
HttpResponseException ex =
|
||||
assertThrows(
|
||||
HttpResponseException.class,
|
||||
() -> consumerRest.patch(PATH, folder.getId(), original, folder, Folder.class));
|
||||
|
||||
assertTrue(
|
||||
ex.getStatusCode() == 403 || ex.getStatusCode() == 401,
|
||||
"Expected 403 or 401, got " + ex.getStatusCode());
|
||||
}
|
||||
|
||||
@Test
|
||||
@Execution(ExecutionMode.SAME_THREAD)
|
||||
void testDeleteFolderRecursive(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
|
||||
Folder parent = createFolder(rest, new CreateFolder().withName(ns.prefix("recursive-parent")));
|
||||
|
||||
Folder child =
|
||||
createFolder(
|
||||
rest,
|
||||
new CreateFolder()
|
||||
.withName(ns.prefix("recursive-child"))
|
||||
.withParent(parent.getFullyQualifiedName()));
|
||||
|
||||
// Delete parent recursively
|
||||
try (Response deleteResponse =
|
||||
rest.rawDelete(PATH + "/" + parent.getId() + "?recursive=true&hardDelete=true")) {
|
||||
assertEquals(202, deleteResponse.getStatus());
|
||||
String responseBody = deleteResponse.readEntity(String.class);
|
||||
assertTrue(responseBody.contains("\"hardDelete\":true"));
|
||||
assertTrue(responseBody.contains("\"recursive\":true"));
|
||||
}
|
||||
|
||||
// Both should be gone. Close each Response before opening the next so the Apache HTTP
|
||||
// client's connection pool doesn't hold two concurrent requests — under parallel-test load
|
||||
// the second GET can otherwise block waiting for a free connection.
|
||||
await()
|
||||
.atMost(Duration.ofMinutes(2))
|
||||
.pollInterval(Duration.ofSeconds(1))
|
||||
.untilAsserted(
|
||||
() -> {
|
||||
int parentStatus;
|
||||
try (Response parentResponse =
|
||||
rest.rawGet(PATH + "/" + parent.getId() + "?include=all")) {
|
||||
parentStatus = parentResponse.getStatus();
|
||||
}
|
||||
int childStatus;
|
||||
try (Response childResponse =
|
||||
rest.rawGet(PATH + "/" + child.getId() + "?include=all")) {
|
||||
childStatus = childResponse.getStatus();
|
||||
}
|
||||
assertEquals(404, parentStatus);
|
||||
assertEquals(404, childStatus);
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
void testFolderContentsIncludesFoldersAndFiles(TestNamespace ns) throws Exception {
|
||||
RestClient rest = RestClient.admin();
|
||||
OpenMetadataClient adminClient = SdkClients.adminClient();
|
||||
UserService userSvc = new UserService(adminClient.getHttpClient());
|
||||
User admin = userSvc.getByName("admin", null);
|
||||
|
||||
Folder parent = createFolder(rest, new CreateFolder().withName(ns.prefix("contents-parent")));
|
||||
Folder child =
|
||||
createFolder(
|
||||
rest,
|
||||
new CreateFolder()
|
||||
.withName(ns.prefix("child-folder"))
|
||||
.withParent(parent.getFullyQualifiedName())
|
||||
.withOwners(List.of(admin.getEntityReference())));
|
||||
|
||||
ContextFile file =
|
||||
rest.create(
|
||||
"v1/contextCenter/drive/files",
|
||||
new CreateContextFile()
|
||||
.withName(ns.prefix("contents-file"))
|
||||
.withDisplayName("Contents File")
|
||||
.withFileType(ContextFileType.PDF)
|
||||
.withFolder(parent.getFullyQualifiedName())
|
||||
.withOwners(List.of(admin.getEntityReference()))
|
||||
.withProcessingStatus(ProcessingStatus.Uploaded),
|
||||
ContextFile.class);
|
||||
|
||||
String json;
|
||||
try (Response response = rest.rawGet(PATH + "/" + parent.getId() + "/contents")) {
|
||||
assertEquals(200, response.getStatus());
|
||||
json = response.readEntity(String.class);
|
||||
}
|
||||
jakarta.json.JsonObject contents =
|
||||
jakarta.json.Json.createReader(new java.io.StringReader(json)).readObject();
|
||||
jakarta.json.JsonObject folderJson = contents.getJsonArray("folders").getJsonObject(0);
|
||||
jakarta.json.JsonObject fileJson = contents.getJsonArray("files").getJsonObject(0);
|
||||
|
||||
assertEquals(1, contents.getInt("childrenFolderCount"));
|
||||
assertEquals(1, contents.getInt("childrenFileCount"));
|
||||
assertEquals(2, contents.getInt("itemCount"));
|
||||
assertEquals(1, contents.getJsonArray("folders").size());
|
||||
assertEquals(1, contents.getJsonArray("files").size());
|
||||
assertEquals(child.getName(), folderJson.getString("name"));
|
||||
assertEquals(parent.getId().toString(), folderJson.getJsonObject("parent").getString("id"));
|
||||
assertEquals(
|
||||
admin.getId().toString(),
|
||||
folderJson.getJsonArray("owners").getJsonObject(0).getString("id"));
|
||||
assertEquals(file.getName(), fileJson.getString("name"));
|
||||
assertEquals(parent.getId().toString(), fileJson.getJsonObject("folder").getString("id"));
|
||||
assertEquals(
|
||||
admin.getId().toString(), fileJson.getJsonArray("owners").getJsonObject(0).getString("id"));
|
||||
}
|
||||
}
|
||||
+52
@@ -0,0 +1,52 @@
|
||||
package org.openmetadata.it.factories;
|
||||
|
||||
import java.net.URI;
|
||||
import java.util.UUID;
|
||||
import org.openmetadata.it.util.SdkClients;
|
||||
import org.openmetadata.it.util.TestNamespace;
|
||||
import org.openmetadata.schema.api.services.CreateApiService;
|
||||
import org.openmetadata.schema.api.services.CreateApiService.ApiServiceType;
|
||||
import org.openmetadata.schema.entity.services.ApiService;
|
||||
import org.openmetadata.schema.services.connections.api.OpenAPISchemaURL;
|
||||
import org.openmetadata.schema.services.connections.api.RestConnection;
|
||||
import org.openmetadata.schema.type.ApiConnection;
|
||||
import org.openmetadata.service.Entity;
|
||||
|
||||
/**
|
||||
* Factory for creating ApiService entities in integration tests.
|
||||
*
|
||||
* <p>Provides namespace-isolated entity creation with consistent patterns.
|
||||
*/
|
||||
public class APIServiceTestFactory {
|
||||
|
||||
/**
|
||||
* Create a REST API service with default settings. Each call creates a unique service to avoid
|
||||
* conflicts in parallel test execution.
|
||||
*/
|
||||
public static ApiService createRest(TestNamespace ns) {
|
||||
String uniqueId = UUID.randomUUID().toString().substring(0, 8);
|
||||
String name = ns.prefix("restApiService_" + uniqueId);
|
||||
|
||||
RestConnection restConn =
|
||||
new RestConnection()
|
||||
.withOpenAPISchemaConnection(
|
||||
new OpenAPISchemaURL()
|
||||
.withOpenAPISchemaURL(URI.create("http://localhost:8585/swagger.json")));
|
||||
|
||||
ApiConnection conn = new ApiConnection().withConfig(restConn);
|
||||
|
||||
CreateApiService request =
|
||||
new CreateApiService()
|
||||
.withName(name)
|
||||
.withServiceType(ApiServiceType.Rest)
|
||||
.withConnection(conn)
|
||||
.withDescription("Test REST API service");
|
||||
|
||||
return ns.trackRoot(Entity.API_SERVICE, SdkClients.adminClient().apiServices().create(request));
|
||||
}
|
||||
|
||||
/** Get API service by ID. */
|
||||
public static ApiService getById(String id) {
|
||||
return SdkClients.adminClient().apiServices().get(id);
|
||||
}
|
||||
}
|
||||
+37
@@ -0,0 +1,37 @@
|
||||
package org.openmetadata.it.factories;
|
||||
|
||||
import java.util.UUID;
|
||||
import org.openmetadata.it.util.SdkClients;
|
||||
import org.openmetadata.it.util.TestNamespace;
|
||||
import org.openmetadata.schema.api.services.CreateStorageService;
|
||||
import org.openmetadata.schema.api.services.CreateStorageService.StorageServiceType;
|
||||
import org.openmetadata.schema.entity.services.StorageService;
|
||||
import org.openmetadata.schema.services.connections.storage.S3Connection;
|
||||
import org.openmetadata.schema.type.StorageConnection;
|
||||
import org.openmetadata.service.Entity;
|
||||
|
||||
public class ContainerServiceTestFactory {
|
||||
|
||||
public static StorageService createS3(TestNamespace ns) {
|
||||
String uniqueId = UUID.randomUUID().toString().substring(0, 8);
|
||||
String name = ns.prefix("containerS3Service_" + uniqueId);
|
||||
|
||||
S3Connection s3Conn = new S3Connection();
|
||||
|
||||
StorageConnection conn = new StorageConnection().withConfig(s3Conn);
|
||||
|
||||
CreateStorageService request =
|
||||
new CreateStorageService()
|
||||
.withName(name)
|
||||
.withServiceType(StorageServiceType.S3)
|
||||
.withConnection(conn)
|
||||
.withDescription("Test container S3 service");
|
||||
|
||||
return ns.trackRoot(
|
||||
Entity.STORAGE_SERVICE, SdkClients.adminClient().storageServices().create(request));
|
||||
}
|
||||
|
||||
public static StorageService getById(String id) {
|
||||
return SdkClients.adminClient().storageServices().get(id);
|
||||
}
|
||||
}
|
||||
+78
@@ -0,0 +1,78 @@
|
||||
package org.openmetadata.it.factories;
|
||||
|
||||
import java.net.URI;
|
||||
import java.util.UUID;
|
||||
import org.openmetadata.it.util.SdkClients;
|
||||
import org.openmetadata.it.util.TestNamespace;
|
||||
import org.openmetadata.schema.api.data.CreateDashboardDataModel.DashboardServiceType;
|
||||
import org.openmetadata.schema.api.services.CreateDashboardService;
|
||||
import org.openmetadata.schema.entity.services.DashboardService;
|
||||
import org.openmetadata.schema.services.connections.dashboard.LookerConnection;
|
||||
import org.openmetadata.schema.services.connections.dashboard.MetabaseConnection;
|
||||
import org.openmetadata.schema.type.DashboardConnection;
|
||||
import org.openmetadata.service.Entity;
|
||||
|
||||
/**
|
||||
* Factory for creating DashboardService entities in integration tests.
|
||||
*
|
||||
* <p>Provides namespace-isolated entity creation with consistent patterns.
|
||||
*/
|
||||
public class DashboardServiceTestFactory {
|
||||
|
||||
/**
|
||||
* Create a Metabase dashboard service with default settings. Each call creates a unique service
|
||||
* to avoid conflicts in parallel test execution.
|
||||
*/
|
||||
public static DashboardService createMetabase(TestNamespace ns) {
|
||||
String uniqueId = UUID.randomUUID().toString().substring(0, 8);
|
||||
String name = ns.prefix("metabaseService_" + uniqueId);
|
||||
|
||||
MetabaseConnection metabaseConn =
|
||||
new MetabaseConnection()
|
||||
.withHostPort(URI.create("http://localhost:3000"))
|
||||
.withUsername("admin");
|
||||
|
||||
DashboardConnection conn = new DashboardConnection().withConfig(metabaseConn);
|
||||
|
||||
CreateDashboardService request =
|
||||
new CreateDashboardService()
|
||||
.withName(name)
|
||||
.withServiceType(DashboardServiceType.Metabase)
|
||||
.withConnection(conn)
|
||||
.withDescription("Test Metabase service");
|
||||
|
||||
return ns.trackRoot(
|
||||
Entity.DASHBOARD_SERVICE, SdkClients.adminClient().dashboardServices().create(request));
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a Looker dashboard service with default settings. Each call creates a unique service to
|
||||
* avoid conflicts in parallel test execution.
|
||||
*/
|
||||
public static DashboardService createLooker(TestNamespace ns) {
|
||||
String uniqueId = UUID.randomUUID().toString().substring(0, 8);
|
||||
String name = ns.prefix("lookerService_" + uniqueId);
|
||||
|
||||
LookerConnection lookerConn =
|
||||
new LookerConnection()
|
||||
.withHostPort(URI.create("http://localhost:9999"))
|
||||
.withClientId("test-client");
|
||||
|
||||
DashboardConnection conn = new DashboardConnection().withConfig(lookerConn);
|
||||
|
||||
CreateDashboardService request =
|
||||
new CreateDashboardService()
|
||||
.withName(name)
|
||||
.withServiceType(DashboardServiceType.Looker)
|
||||
.withConnection(conn)
|
||||
.withDescription("Test Looker service");
|
||||
|
||||
return ns.trackRoot(
|
||||
Entity.DASHBOARD_SERVICE, SdkClients.adminClient().dashboardServices().create(request));
|
||||
}
|
||||
|
||||
/** Get dashboard service by ID. */
|
||||
public static DashboardService getById(String id) {
|
||||
return SdkClients.adminClient().dashboardServices().get(id);
|
||||
}
|
||||
}
|
||||
+66
@@ -0,0 +1,66 @@
|
||||
package org.openmetadata.it.factories;
|
||||
|
||||
import org.openmetadata.it.util.TestNamespace;
|
||||
import org.openmetadata.schema.entity.data.Database;
|
||||
import org.openmetadata.schema.entity.data.DatabaseSchema;
|
||||
import org.openmetadata.schema.entity.services.DatabaseService;
|
||||
import org.openmetadata.sdk.fluent.DatabaseSchemas;
|
||||
import org.openmetadata.sdk.fluent.Databases;
|
||||
|
||||
/**
|
||||
* Factory for creating DatabaseSchema entities in integration tests using fluent API.
|
||||
*
|
||||
* <p>Uses the static fluent API from {@link DatabaseSchemas}. Ensure
|
||||
* fluent APIs are initialized before using these methods.
|
||||
*/
|
||||
public class DatabaseSchemaTestFactory {
|
||||
|
||||
/**
|
||||
* Create a schema with database FQN using fluent API.
|
||||
*/
|
||||
public static DatabaseSchema create(TestNamespace ns, String databaseFqn) {
|
||||
return DatabaseSchemas.create().name(ns.prefix("schema")).in(databaseFqn).execute();
|
||||
}
|
||||
|
||||
public static DatabaseSchema create(String databaseFqn, String schemaName) {
|
||||
return DatabaseSchemas.create().name(schemaName).in(databaseFqn).execute();
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a schema with its parent database using fluent API.
|
||||
*/
|
||||
public static DatabaseSchema createSimple(TestNamespace ns, DatabaseService service) {
|
||||
// Create database first using fluent API
|
||||
Database database =
|
||||
Databases.create().name(ns.prefix("db")).in(service.getFullyQualifiedName()).execute();
|
||||
|
||||
// Then create schema using fluent API
|
||||
return DatabaseSchemas.create()
|
||||
.name(ns.prefix("schema"))
|
||||
.in(database.getFullyQualifiedName())
|
||||
.execute();
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a schema with a newly created database service.
|
||||
*/
|
||||
public static DatabaseSchema createSimple(TestNamespace ns) {
|
||||
DatabaseService service = DatabaseServiceTestFactory.createPostgres(ns);
|
||||
return createSimple(ns, service);
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a schema with a custom name using fluent API.
|
||||
* Useful for tests that need short names to avoid FQN length limits.
|
||||
*/
|
||||
public static DatabaseSchema createSimpleWithName(
|
||||
String schemaName, TestNamespace ns, DatabaseService service) {
|
||||
// Create database first using short name
|
||||
String shortDbName = "db" + schemaName.substring(2); // Use same short ID
|
||||
Database database =
|
||||
Databases.create().name(shortDbName).in(service.getFullyQualifiedName()).execute();
|
||||
|
||||
// Then create schema using the specified name
|
||||
return DatabaseSchemas.create().name(schemaName).in(database.getFullyQualifiedName()).execute();
|
||||
}
|
||||
}
|
||||
+90
@@ -0,0 +1,90 @@
|
||||
package org.openmetadata.it.factories;
|
||||
|
||||
import java.util.UUID;
|
||||
import org.openmetadata.it.util.TestNamespace;
|
||||
import org.openmetadata.schema.entity.services.DatabaseService;
|
||||
import org.openmetadata.schema.services.connections.database.PostgresConnection;
|
||||
import org.openmetadata.schema.services.connections.database.SnowflakeConnection;
|
||||
import org.openmetadata.sdk.fluent.DatabaseServices;
|
||||
import org.openmetadata.service.Entity;
|
||||
|
||||
/**
|
||||
* Factory for creating DatabaseService entities in integration tests.
|
||||
*
|
||||
* <p>Uses the static fluent API from {@link DatabaseServices}. Ensure
|
||||
* fluent APIs are initialized before using these methods.
|
||||
*/
|
||||
public class DatabaseServiceTestFactory {
|
||||
|
||||
/**
|
||||
* Create a Postgres database service with default settings.
|
||||
*/
|
||||
public static DatabaseService createPostgres(TestNamespace ns) {
|
||||
PostgresConnection conn =
|
||||
DatabaseServices.postgresConnection().hostPort("localhost:5432").username("test").build();
|
||||
|
||||
String uniqueId = UUID.randomUUID().toString().substring(0, 8);
|
||||
String name = ns.prefix("postgresService_" + uniqueId);
|
||||
|
||||
return ns.trackRoot(
|
||||
Entity.DATABASE_SERVICE,
|
||||
DatabaseServices.builder()
|
||||
.name(name)
|
||||
.connection(conn)
|
||||
.description("Test Postgres service")
|
||||
.create());
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a Snowflake database service with default settings.
|
||||
*/
|
||||
public static DatabaseService createSnowflake(TestNamespace ns) {
|
||||
SnowflakeConnection conn =
|
||||
DatabaseServices.snowflakeConnection()
|
||||
.account("test-account")
|
||||
.username("test")
|
||||
.warehouse("test-warehouse")
|
||||
.build();
|
||||
|
||||
String uniqueId = UUID.randomUUID().toString().substring(0, 8);
|
||||
String name = ns.prefix("snowflakeService_" + uniqueId);
|
||||
|
||||
return ns.trackRoot(
|
||||
Entity.DATABASE_SERVICE,
|
||||
DatabaseServices.builder()
|
||||
.name(name)
|
||||
.connection(conn)
|
||||
.description("Test Snowflake service")
|
||||
.create());
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a database service with specified type.
|
||||
*/
|
||||
public static DatabaseService create(TestNamespace ns, String serviceType) {
|
||||
if ("Postgres".equalsIgnoreCase(serviceType)) {
|
||||
return createPostgres(ns);
|
||||
} else if ("Snowflake".equalsIgnoreCase(serviceType)) {
|
||||
return createSnowflake(ns);
|
||||
} else {
|
||||
return createPostgres(ns);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a Postgres database service with a custom name.
|
||||
* Useful for tests that need short names to avoid FQN length limits.
|
||||
*/
|
||||
public static DatabaseService createPostgresWithName(String name, TestNamespace ns) {
|
||||
PostgresConnection conn =
|
||||
DatabaseServices.postgresConnection().hostPort("localhost:5432").username("test").build();
|
||||
|
||||
return ns.trackRoot(
|
||||
Entity.DATABASE_SERVICE,
|
||||
DatabaseServices.builder()
|
||||
.name(name)
|
||||
.connection(conn)
|
||||
.description("Test Postgres service")
|
||||
.create());
|
||||
}
|
||||
}
|
||||
+32
@@ -0,0 +1,32 @@
|
||||
package org.openmetadata.it.factories;
|
||||
|
||||
import org.openmetadata.it.util.TestNamespace;
|
||||
import org.openmetadata.schema.entity.data.Database;
|
||||
import org.openmetadata.sdk.fluent.Databases;
|
||||
|
||||
/**
|
||||
* Factory for creating Database entities in integration tests using fluent API.
|
||||
*
|
||||
* <p>Uses the static fluent API from {@link Databases}. Ensure
|
||||
* {@code Databases.setDefaultClient(client)} is called before using these methods.
|
||||
*/
|
||||
public class DatabaseTestFactory {
|
||||
|
||||
/**
|
||||
* Create a database with default settings using fluent API.
|
||||
*/
|
||||
public static Database create(TestNamespace ns, String serviceFqn) {
|
||||
return Databases.create()
|
||||
.name(ns.prefix("db"))
|
||||
.in(serviceFqn)
|
||||
.withDescription("Test database created by integration test")
|
||||
.execute();
|
||||
}
|
||||
|
||||
/**
|
||||
* Create database with custom name using fluent API.
|
||||
*/
|
||||
public static Database createWithName(String serviceFqn, String name) {
|
||||
return Databases.create().name(name).in(serviceFqn).execute();
|
||||
}
|
||||
}
|
||||
+53
@@ -0,0 +1,53 @@
|
||||
package org.openmetadata.it.factories;
|
||||
|
||||
import java.util.UUID;
|
||||
import org.openmetadata.it.util.SdkClients;
|
||||
import org.openmetadata.it.util.TestNamespace;
|
||||
import org.openmetadata.schema.api.services.CreateDriveService;
|
||||
import org.openmetadata.schema.api.services.CreateDriveService.DriveServiceType;
|
||||
import org.openmetadata.schema.entity.services.DriveService;
|
||||
import org.openmetadata.schema.services.connections.drive.GoogleDriveConnection;
|
||||
import org.openmetadata.schema.type.DriveConnection;
|
||||
import org.openmetadata.sdk.exceptions.OpenMetadataException;
|
||||
import org.openmetadata.sdk.network.HttpMethod;
|
||||
|
||||
public class DriveServiceTestFactory {
|
||||
|
||||
public static DriveService createGoogleDrive(TestNamespace ns) {
|
||||
String uniqueId = UUID.randomUUID().toString().substring(0, 8);
|
||||
return createGoogleDrive(ns, "googleDrive_" + uniqueId);
|
||||
}
|
||||
|
||||
public static DriveService createGoogleDrive(TestNamespace ns, String suffix) {
|
||||
String name = ns.prefix(suffix);
|
||||
|
||||
GoogleDriveConnection googleConn = new GoogleDriveConnection();
|
||||
|
||||
DriveConnection conn = new DriveConnection().withConfig(googleConn);
|
||||
|
||||
CreateDriveService request =
|
||||
new CreateDriveService()
|
||||
.withName(name)
|
||||
.withServiceType(DriveServiceType.GoogleDrive)
|
||||
.withConnection(conn)
|
||||
.withDescription("Test GoogleDrive service");
|
||||
|
||||
try {
|
||||
return SdkClients.adminClient()
|
||||
.getHttpClient()
|
||||
.execute(HttpMethod.POST, "/v1/services/driveServices", request, DriveService.class);
|
||||
} catch (OpenMetadataException e) {
|
||||
throw new RuntimeException("Failed to create GoogleDrive service", e);
|
||||
}
|
||||
}
|
||||
|
||||
public static DriveService getById(String id) {
|
||||
try {
|
||||
return SdkClients.adminClient()
|
||||
.getHttpClient()
|
||||
.execute(HttpMethod.GET, "/v1/services/driveServices/" + id, null, DriveService.class);
|
||||
} catch (OpenMetadataException e) {
|
||||
throw new RuntimeException("Failed to get DriveService by id: " + id, e);
|
||||
}
|
||||
}
|
||||
}
|
||||
+143
@@ -0,0 +1,143 @@
|
||||
package org.openmetadata.it.factories;
|
||||
|
||||
import java.util.EnumMap;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* Declarative spec for how many entities of each type {@link EntityLoader} should create.
|
||||
*
|
||||
* <p>Tests own the absolute counts (typically as {@code private static final int} constants
|
||||
* in the test class) so a single line change scales the load up or down. {@link EntityLoader}
|
||||
* just executes whatever the spec asks for.
|
||||
*/
|
||||
public record EntityLoadSpec(
|
||||
int parallelWorkers,
|
||||
Map<EntityKind, Integer> counts,
|
||||
int columnsPerTable,
|
||||
int chartsPerDashboard,
|
||||
int endpointsPerCollection,
|
||||
int columnsPerDataModel,
|
||||
int testCasesPerSuite,
|
||||
int resultsPerTestCase) {
|
||||
|
||||
/**
|
||||
* Entity types {@link EntityLoader} can create. Mirrors {@code perf-test.sh}'s coverage so
|
||||
* Java UIITs and the bash perf script can drive equivalent loads. Order is not significant;
|
||||
* actual dispatch order honours prerequisites (services before children, glossaries before
|
||||
* terms, tables before testCases, etc.).
|
||||
*/
|
||||
public enum EntityKind {
|
||||
// Asset entities (each has its own search index)
|
||||
TABLE,
|
||||
TOPIC,
|
||||
DASHBOARD,
|
||||
PIPELINE,
|
||||
CHART,
|
||||
ML_MODEL,
|
||||
CONTAINER,
|
||||
SEARCH_INDEX,
|
||||
API_COLLECTION,
|
||||
API_ENDPOINT,
|
||||
STORED_PROCEDURE,
|
||||
QUERY,
|
||||
DASHBOARD_DATA_MODEL,
|
||||
// Taxonomy
|
||||
GLOSSARY,
|
||||
GLOSSARY_TERM,
|
||||
CLASSIFICATION,
|
||||
TAG,
|
||||
// Org
|
||||
USER,
|
||||
TEAM,
|
||||
// Governance
|
||||
DOMAIN,
|
||||
DATA_PRODUCT,
|
||||
// Quality
|
||||
TEST_SUITE,
|
||||
TEST_CASE,
|
||||
// Graph
|
||||
LINEAGE_EDGE,
|
||||
// Time-series telemetry
|
||||
TEST_CASE_RESULT,
|
||||
ENTITY_REPORT_DATA,
|
||||
WEB_ANALYTIC_VIEW,
|
||||
WEB_ANALYTIC_ACTIVITY,
|
||||
RAW_COST_ANALYSIS,
|
||||
AGG_COST_ANALYSIS
|
||||
}
|
||||
|
||||
public int countOf(EntityKind kind) {
|
||||
return counts.getOrDefault(kind, 0);
|
||||
}
|
||||
|
||||
public int total() {
|
||||
return counts.values().stream().mapToInt(Integer::intValue).sum();
|
||||
}
|
||||
|
||||
public static Builder builder() {
|
||||
return new Builder();
|
||||
}
|
||||
|
||||
public static final class Builder {
|
||||
private int parallelWorkers = 16;
|
||||
private int columnsPerTable = 5;
|
||||
private int chartsPerDashboard = 3;
|
||||
private int endpointsPerCollection = 3;
|
||||
private int columnsPerDataModel = 5;
|
||||
private int testCasesPerSuite = 3;
|
||||
private int resultsPerTestCase = 5;
|
||||
private final Map<EntityKind, Integer> counts = new EnumMap<>(EntityKind.class);
|
||||
|
||||
public Builder parallelWorkers(int n) {
|
||||
this.parallelWorkers = n;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder columnsPerTable(int n) {
|
||||
this.columnsPerTable = n;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder chartsPerDashboard(int n) {
|
||||
this.chartsPerDashboard = n;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder endpointsPerCollection(int n) {
|
||||
this.endpointsPerCollection = n;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder columnsPerDataModel(int n) {
|
||||
this.columnsPerDataModel = n;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder testCasesPerSuite(int n) {
|
||||
this.testCasesPerSuite = n;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder resultsPerTestCase(int n) {
|
||||
this.resultsPerTestCase = n;
|
||||
return this;
|
||||
}
|
||||
|
||||
public Builder count(EntityKind kind, int n) {
|
||||
this.counts.put(kind, n);
|
||||
return this;
|
||||
}
|
||||
|
||||
public EntityLoadSpec build() {
|
||||
return new EntityLoadSpec(
|
||||
parallelWorkers,
|
||||
Map.copyOf(counts),
|
||||
columnsPerTable,
|
||||
chartsPerDashboard,
|
||||
endpointsPerCollection,
|
||||
columnsPerDataModel,
|
||||
testCasesPerSuite,
|
||||
resultsPerTestCase);
|
||||
}
|
||||
}
|
||||
}
|
||||
+51
@@ -0,0 +1,51 @@
|
||||
package org.openmetadata.it.factories;
|
||||
|
||||
import java.time.Duration;
|
||||
import java.util.EnumMap;
|
||||
import java.util.Map;
|
||||
import org.openmetadata.it.factories.EntityLoadSpec.EntityKind;
|
||||
|
||||
/**
|
||||
* Result of an {@link EntityLoader} run. Tests use it to drive ES-side count assertions
|
||||
* after triggering reindex without re-querying OM for what they just created.
|
||||
*/
|
||||
public record EntityLoadSummary(
|
||||
Map<EntityKind, Integer> created,
|
||||
int totalColumns,
|
||||
Duration totalDuration,
|
||||
Map<EntityKind, Duration> perKindDuration) {
|
||||
|
||||
public int countOf(EntityKind kind) {
|
||||
return created.getOrDefault(kind, 0);
|
||||
}
|
||||
|
||||
public int totalEntities() {
|
||||
return created.values().stream().mapToInt(Integer::intValue).sum();
|
||||
}
|
||||
|
||||
static final class Builder {
|
||||
private final Map<EntityKind, Integer> created = new EnumMap<>(EntityKind.class);
|
||||
private final Map<EntityKind, Duration> perKindDuration = new EnumMap<>(EntityKind.class);
|
||||
private int totalColumns;
|
||||
|
||||
Builder recordCreated(EntityKind kind, int n) {
|
||||
created.merge(kind, n, Integer::sum);
|
||||
return this;
|
||||
}
|
||||
|
||||
Builder recordColumns(int n) {
|
||||
totalColumns += n;
|
||||
return this;
|
||||
}
|
||||
|
||||
Builder recordKindDuration(EntityKind kind, Duration d) {
|
||||
perKindDuration.put(kind, d);
|
||||
return this;
|
||||
}
|
||||
|
||||
EntityLoadSummary build(Duration total) {
|
||||
return new EntityLoadSummary(
|
||||
Map.copyOf(created), totalColumns, total, Map.copyOf(perKindDuration));
|
||||
}
|
||||
}
|
||||
}
|
||||
+1603
File diff suppressed because it is too large
Load Diff
+112
@@ -0,0 +1,112 @@
|
||||
package org.openmetadata.it.factories;
|
||||
|
||||
import java.util.List;
|
||||
import org.openmetadata.it.util.TestNamespace;
|
||||
import org.openmetadata.schema.entity.data.Glossary;
|
||||
import org.openmetadata.schema.entity.data.GlossaryTerm;
|
||||
import org.openmetadata.sdk.fluent.GlossaryTerms;
|
||||
|
||||
/**
|
||||
* Factory for creating GlossaryTerm entities in integration tests using fluent API.
|
||||
*
|
||||
* <p>Uses the static fluent API from {@link GlossaryTerms}. Ensure {@code
|
||||
* GlossaryTerms.setDefaultClient(client)} is called before using these methods (handled by
|
||||
* SdkClients.initializeFluentAPIs).
|
||||
*/
|
||||
public class GlossaryTermTestFactory {
|
||||
|
||||
/**
|
||||
* Create a glossary term with default settings using fluent API.
|
||||
*/
|
||||
public static GlossaryTerm createSimple(TestNamespace ns, Glossary glossary) {
|
||||
return GlossaryTerms.create()
|
||||
.name(ns.prefix("term"))
|
||||
.in(glossary.getFullyQualifiedName())
|
||||
.withDescription("Test term")
|
||||
.execute();
|
||||
}
|
||||
|
||||
/**
|
||||
* Create glossary term with custom name using fluent API.
|
||||
*/
|
||||
public static GlossaryTerm createWithName(TestNamespace ns, Glossary glossary, String baseName) {
|
||||
return GlossaryTerms.create()
|
||||
.name(ns.prefix(baseName))
|
||||
.in(glossary.getFullyQualifiedName())
|
||||
.withDescription("Test term: " + baseName)
|
||||
.execute();
|
||||
}
|
||||
|
||||
/**
|
||||
* Create glossary term with description using fluent API.
|
||||
*/
|
||||
public static GlossaryTerm createWithDescription(
|
||||
TestNamespace ns, Glossary glossary, String description) {
|
||||
return GlossaryTerms.create()
|
||||
.name(ns.prefix("term"))
|
||||
.in(glossary.getFullyQualifiedName())
|
||||
.withDescription(description)
|
||||
.execute();
|
||||
}
|
||||
|
||||
/**
|
||||
* Create glossary term under a parent term using fluent API.
|
||||
*/
|
||||
public static GlossaryTerm createChild(
|
||||
TestNamespace ns, Glossary glossary, GlossaryTerm parent, String baseName) {
|
||||
return GlossaryTerms.create()
|
||||
.name(ns.prefix(baseName))
|
||||
.in(glossary.getFullyQualifiedName())
|
||||
.under(parent.getFullyQualifiedName())
|
||||
.withDescription("Child term of " + parent.getName())
|
||||
.execute();
|
||||
}
|
||||
|
||||
/**
|
||||
* Create glossary term with synonyms using fluent API.
|
||||
*/
|
||||
public static GlossaryTerm createWithSynonyms(
|
||||
TestNamespace ns, Glossary glossary, String baseName, List<String> synonyms) {
|
||||
return GlossaryTerms.create()
|
||||
.name(ns.prefix(baseName))
|
||||
.in(glossary.getFullyQualifiedName())
|
||||
.withDescription("Term with synonyms")
|
||||
.withSynonyms(synonyms)
|
||||
.execute();
|
||||
}
|
||||
|
||||
/**
|
||||
* Create glossary term with related terms using fluent API.
|
||||
*
|
||||
* @param relatedTermFqns List of fully qualified names of related terms
|
||||
*/
|
||||
public static GlossaryTerm createWithRelatedTerms(
|
||||
TestNamespace ns, Glossary glossary, String baseName, List<String> relatedTermFqns) {
|
||||
return GlossaryTerms.create()
|
||||
.name(ns.prefix(baseName))
|
||||
.in(glossary.getFullyQualifiedName())
|
||||
.withDescription("Term with related terms")
|
||||
.withRelatedTerms(relatedTermFqns)
|
||||
.execute();
|
||||
}
|
||||
|
||||
/**
|
||||
* Create glossary term with display name using fluent API.
|
||||
*/
|
||||
public static GlossaryTerm createWithDisplayName(
|
||||
TestNamespace ns, Glossary glossary, String baseName, String displayName) {
|
||||
return GlossaryTerms.create()
|
||||
.name(ns.prefix(baseName))
|
||||
.in(glossary.getFullyQualifiedName())
|
||||
.withDisplayName(displayName)
|
||||
.withDescription("Term with display name")
|
||||
.execute();
|
||||
}
|
||||
|
||||
/**
|
||||
* Delete a glossary term by ID (hard delete with recursive).
|
||||
*/
|
||||
public static void delete(GlossaryTerm term) {
|
||||
GlossaryTerms.find(term.getId()).delete().recursively().permanently().confirm();
|
||||
}
|
||||
}
|
||||
+67
@@ -0,0 +1,67 @@
|
||||
package org.openmetadata.it.factories;
|
||||
|
||||
import org.openmetadata.it.util.TestNamespace;
|
||||
import org.openmetadata.schema.entity.data.Glossary;
|
||||
import org.openmetadata.sdk.fluent.Glossaries;
|
||||
import org.openmetadata.service.Entity;
|
||||
|
||||
/**
|
||||
* Factory for creating Glossary entities in integration tests using fluent API.
|
||||
*
|
||||
* <p>Uses the static fluent API from {@link Glossaries}. Ensure {@code
|
||||
* Glossaries.setDefaultClient(client)} is called before using these methods (handled by
|
||||
* SdkClients.initializeFluentAPIs).
|
||||
*/
|
||||
public class GlossaryTestFactory {
|
||||
|
||||
/**
|
||||
* Create a glossary with default settings using fluent API.
|
||||
*/
|
||||
public static Glossary createSimple(TestNamespace ns) {
|
||||
return ns.trackRoot(
|
||||
Entity.GLOSSARY,
|
||||
Glossaries.create().name(ns.prefix("glossary")).withDescription("Test glossary").execute());
|
||||
}
|
||||
|
||||
/**
|
||||
* Create glossary with custom name using fluent API.
|
||||
*/
|
||||
public static Glossary createWithName(TestNamespace ns, String baseName) {
|
||||
return ns.trackRoot(
|
||||
Entity.GLOSSARY,
|
||||
Glossaries.create()
|
||||
.name(ns.prefix(baseName))
|
||||
.withDescription("Test glossary: " + baseName)
|
||||
.execute());
|
||||
}
|
||||
|
||||
/**
|
||||
* Create glossary with description using fluent API.
|
||||
*/
|
||||
public static Glossary createWithDescription(TestNamespace ns, String description) {
|
||||
return ns.trackRoot(
|
||||
Entity.GLOSSARY,
|
||||
Glossaries.create().name(ns.prefix("glossary")).withDescription(description).execute());
|
||||
}
|
||||
|
||||
/**
|
||||
* Create glossary with display name using fluent API.
|
||||
*/
|
||||
public static Glossary createWithDisplayName(
|
||||
TestNamespace ns, String baseName, String displayName) {
|
||||
return ns.trackRoot(
|
||||
Entity.GLOSSARY,
|
||||
Glossaries.create()
|
||||
.name(ns.prefix(baseName))
|
||||
.withDisplayName(displayName)
|
||||
.withDescription("Test glossary with display name")
|
||||
.execute());
|
||||
}
|
||||
|
||||
/**
|
||||
* Delete a glossary by ID (hard delete with recursive).
|
||||
*/
|
||||
public static void delete(Glossary glossary) {
|
||||
Glossaries.find(glossary.getId()).delete().recursively().permanently().confirm();
|
||||
}
|
||||
}
|
||||
+47
@@ -0,0 +1,47 @@
|
||||
package org.openmetadata.it.factories;
|
||||
|
||||
import java.util.UUID;
|
||||
import org.openmetadata.it.util.SdkClients;
|
||||
import org.openmetadata.it.util.TestNamespace;
|
||||
import org.openmetadata.schema.api.ai.CreateLLMModel;
|
||||
import org.openmetadata.schema.entity.ai.LLMModel;
|
||||
import org.openmetadata.schema.entity.services.LLMService;
|
||||
|
||||
public class LLMModelTestFactory {
|
||||
|
||||
public static LLMModel createGPT(TestNamespace ns) {
|
||||
String uniqueId = UUID.randomUUID().toString().substring(0, 8);
|
||||
String name = ns.prefix("llmModel_" + uniqueId);
|
||||
|
||||
LLMService service = LLMServiceTestFactory.createOpenAI(ns);
|
||||
|
||||
CreateLLMModel request =
|
||||
new CreateLLMModel()
|
||||
.withName(name)
|
||||
.withService(service.getFullyQualifiedName())
|
||||
.withBaseModel("gpt-4")
|
||||
.withModelProvider("OpenAI")
|
||||
.withDescription("Test GPT model");
|
||||
|
||||
return SdkClients.adminClient().llmModels().create(request);
|
||||
}
|
||||
|
||||
public static LLMModel createWithService(TestNamespace ns, LLMService service) {
|
||||
String uniqueId = UUID.randomUUID().toString().substring(0, 8);
|
||||
String name = ns.prefix("llmModel_" + uniqueId);
|
||||
|
||||
CreateLLMModel request =
|
||||
new CreateLLMModel()
|
||||
.withName(name)
|
||||
.withService(service.getFullyQualifiedName())
|
||||
.withBaseModel("gpt-3.5-turbo")
|
||||
.withModelProvider("OpenAI")
|
||||
.withDescription("Test LLM model");
|
||||
|
||||
return SdkClients.adminClient().llmModels().create(request);
|
||||
}
|
||||
|
||||
public static LLMModel getById(String id) {
|
||||
return SdkClients.adminClient().llmModels().get(id);
|
||||
}
|
||||
}
|
||||
+36
@@ -0,0 +1,36 @@
|
||||
package org.openmetadata.it.factories;
|
||||
|
||||
import java.util.UUID;
|
||||
import org.openmetadata.it.util.SdkClients;
|
||||
import org.openmetadata.it.util.TestNamespace;
|
||||
import org.openmetadata.schema.api.services.CreateLLMService;
|
||||
import org.openmetadata.schema.api.services.CreateLLMService.LlmServiceType;
|
||||
import org.openmetadata.schema.entity.services.LLMService;
|
||||
import org.openmetadata.schema.services.connections.llm.OpenAIConnection;
|
||||
import org.openmetadata.schema.type.LLMConnection;
|
||||
|
||||
public class LLMServiceTestFactory {
|
||||
|
||||
public static LLMService createOpenAI(TestNamespace ns) {
|
||||
String uniqueId = UUID.randomUUID().toString().substring(0, 8);
|
||||
String name = ns.prefix("llmService_" + uniqueId);
|
||||
|
||||
OpenAIConnection openAIConn =
|
||||
new OpenAIConnection().withApiKey("test-key").withBaseURL("https://api.openai.com/v1");
|
||||
|
||||
LLMConnection conn = new LLMConnection().withConfig(openAIConn);
|
||||
|
||||
CreateLLMService request =
|
||||
new CreateLLMService()
|
||||
.withName(name)
|
||||
.withServiceType(LlmServiceType.OpenAI)
|
||||
.withConnection(conn)
|
||||
.withDescription("Test OpenAI service");
|
||||
|
||||
return SdkClients.adminClient().llmServices().create(request);
|
||||
}
|
||||
|
||||
public static LLMService getById(String id) {
|
||||
return SdkClients.adminClient().llmServices().get(id);
|
||||
}
|
||||
}
|
||||
+73
@@ -0,0 +1,73 @@
|
||||
package org.openmetadata.it.factories;
|
||||
|
||||
import java.util.UUID;
|
||||
import org.openmetadata.it.util.SdkClients;
|
||||
import org.openmetadata.it.util.TestNamespace;
|
||||
import org.openmetadata.schema.api.services.CreateMessagingService;
|
||||
import org.openmetadata.schema.api.services.CreateMessagingService.MessagingServiceType;
|
||||
import org.openmetadata.schema.entity.services.MessagingService;
|
||||
import org.openmetadata.schema.services.connections.messaging.KafkaConnection;
|
||||
import org.openmetadata.schema.services.connections.messaging.RedpandaConnection;
|
||||
import org.openmetadata.schema.type.MessagingConnection;
|
||||
import org.openmetadata.service.Entity;
|
||||
|
||||
/**
|
||||
* Factory for creating MessagingService entities in integration tests.
|
||||
*
|
||||
* <p>Provides namespace-isolated entity creation with consistent patterns.
|
||||
*/
|
||||
public class MessagingServiceTestFactory {
|
||||
|
||||
/**
|
||||
* Create a Kafka messaging service with default settings. Each call creates a unique service to
|
||||
* avoid conflicts in parallel test execution.
|
||||
*/
|
||||
public static MessagingService createKafka(TestNamespace ns) {
|
||||
String uniqueId = UUID.randomUUID().toString().substring(0, 8);
|
||||
String name = ns.prefix("kafkaService_" + uniqueId);
|
||||
|
||||
KafkaConnection kafkaConn = new KafkaConnection().withBootstrapServers("localhost:9092");
|
||||
|
||||
MessagingConnection conn = new MessagingConnection().withConfig(kafkaConn);
|
||||
|
||||
CreateMessagingService request =
|
||||
new CreateMessagingService()
|
||||
.withName(name)
|
||||
.withServiceType(MessagingServiceType.Kafka)
|
||||
.withConnection(conn)
|
||||
.withDescription("Test Kafka service");
|
||||
|
||||
return ns.trackRoot(
|
||||
Entity.MESSAGING_SERVICE, SdkClients.adminClient().messagingServices().create(request));
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a Redpanda messaging service with default settings. Each call creates a unique service
|
||||
* to avoid conflicts in parallel test execution.
|
||||
*/
|
||||
public static MessagingService createRedpanda(TestNamespace ns) {
|
||||
String uniqueId = UUID.randomUUID().toString().substring(0, 8);
|
||||
String name = ns.prefix("redpandaService_" + uniqueId);
|
||||
|
||||
// Use RedpandaConnection for Redpanda service type
|
||||
RedpandaConnection redpandaConn =
|
||||
new RedpandaConnection().withBootstrapServers("localhost:9092");
|
||||
|
||||
MessagingConnection conn = new MessagingConnection().withConfig(redpandaConn);
|
||||
|
||||
CreateMessagingService request =
|
||||
new CreateMessagingService()
|
||||
.withName(name)
|
||||
.withServiceType(MessagingServiceType.Redpanda)
|
||||
.withConnection(conn)
|
||||
.withDescription("Test Redpanda service");
|
||||
|
||||
return ns.trackRoot(
|
||||
Entity.MESSAGING_SERVICE, SdkClients.adminClient().messagingServices().create(request));
|
||||
}
|
||||
|
||||
/** Get messaging service by ID. */
|
||||
public static MessagingService getById(String id) {
|
||||
return SdkClients.adminClient().messagingServices().get(id);
|
||||
}
|
||||
}
|
||||
+50
@@ -0,0 +1,50 @@
|
||||
package org.openmetadata.it.factories;
|
||||
|
||||
import java.util.UUID;
|
||||
import org.openmetadata.it.util.SdkClients;
|
||||
import org.openmetadata.it.util.TestNamespace;
|
||||
import org.openmetadata.schema.api.services.CreateMlModelService;
|
||||
import org.openmetadata.schema.api.services.CreateMlModelService.MlModelServiceType;
|
||||
import org.openmetadata.schema.entity.services.MlModelService;
|
||||
import org.openmetadata.schema.services.connections.mlmodel.MlflowConnection;
|
||||
import org.openmetadata.schema.type.MlModelConnection;
|
||||
import org.openmetadata.service.Entity;
|
||||
|
||||
/**
|
||||
* Factory for creating MlModelService entities in integration tests.
|
||||
*
|
||||
* <p>Provides namespace-isolated entity creation with consistent patterns.
|
||||
*/
|
||||
public class MlModelServiceTestFactory {
|
||||
|
||||
/**
|
||||
* Create an MLflow ML model service with default settings. Each call creates a unique service to
|
||||
* avoid conflicts in parallel test execution.
|
||||
*/
|
||||
public static MlModelService createMlflow(TestNamespace ns) {
|
||||
String uniqueId = UUID.randomUUID().toString().substring(0, 8);
|
||||
String name = ns.prefix("mlflowService_" + uniqueId);
|
||||
|
||||
MlflowConnection mlflowConn =
|
||||
new MlflowConnection()
|
||||
.withTrackingUri("http://localhost:5000")
|
||||
.withRegistryUri("http://localhost:5000");
|
||||
|
||||
MlModelConnection conn = new MlModelConnection().withConfig(mlflowConn);
|
||||
|
||||
CreateMlModelService request =
|
||||
new CreateMlModelService()
|
||||
.withName(name)
|
||||
.withServiceType(MlModelServiceType.Mlflow)
|
||||
.withConnection(conn)
|
||||
.withDescription("Test MLflow service");
|
||||
|
||||
return ns.trackRoot(
|
||||
Entity.MLMODEL_SERVICE, SdkClients.adminClient().mlModelServices().create(request));
|
||||
}
|
||||
|
||||
/** Get ML model service by ID. */
|
||||
public static MlModelService getById(String id) {
|
||||
return SdkClients.adminClient().mlModelServices().get(id);
|
||||
}
|
||||
}
|
||||
+77
@@ -0,0 +1,77 @@
|
||||
package org.openmetadata.it.factories;
|
||||
|
||||
import java.net.URI;
|
||||
import java.util.UUID;
|
||||
import org.openmetadata.it.util.SdkClients;
|
||||
import org.openmetadata.it.util.TestNamespace;
|
||||
import org.openmetadata.schema.api.services.CreatePipelineService;
|
||||
import org.openmetadata.schema.api.services.CreatePipelineService.PipelineServiceType;
|
||||
import org.openmetadata.schema.entity.services.PipelineService;
|
||||
import org.openmetadata.schema.services.connections.pipeline.AirflowConnection;
|
||||
import org.openmetadata.schema.services.connections.pipeline.GluePipelineConnection;
|
||||
import org.openmetadata.schema.type.PipelineConnection;
|
||||
import org.openmetadata.service.Entity;
|
||||
|
||||
/**
|
||||
* Factory for creating PipelineService entities in integration tests.
|
||||
*
|
||||
* <p>Provides namespace-isolated entity creation with consistent patterns.
|
||||
*/
|
||||
public class PipelineServiceTestFactory {
|
||||
|
||||
/**
|
||||
* Create an Airflow pipeline service with default settings. Each call creates a unique service to
|
||||
* avoid conflicts in parallel test execution.
|
||||
*/
|
||||
public static PipelineService createAirflow(TestNamespace ns) {
|
||||
String uniqueId = UUID.randomUUID().toString().substring(0, 8);
|
||||
String name = ns.prefix("airflowService_" + uniqueId);
|
||||
|
||||
AirflowConnection airflowConn =
|
||||
new AirflowConnection().withHostPort(URI.create("http://localhost:8080"));
|
||||
|
||||
PipelineConnection conn = new PipelineConnection().withConfig(airflowConn);
|
||||
|
||||
CreatePipelineService request =
|
||||
new CreatePipelineService()
|
||||
.withName(name)
|
||||
.withServiceType(PipelineServiceType.Airflow)
|
||||
.withConnection(conn)
|
||||
.withDescription("Test Airflow service");
|
||||
|
||||
return ns.trackRoot(
|
||||
Entity.PIPELINE_SERVICE, SdkClients.adminClient().pipelineServices().create(request));
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a Glue pipeline service with default settings. Each call creates a unique service to
|
||||
* avoid conflicts in parallel test execution.
|
||||
*/
|
||||
public static PipelineService createGlue(TestNamespace ns) {
|
||||
String uniqueId = UUID.randomUUID().toString().substring(0, 8);
|
||||
String name = ns.prefix("glueService_" + uniqueId);
|
||||
|
||||
GluePipelineConnection glueConn =
|
||||
new GluePipelineConnection()
|
||||
.withAwsConfig(
|
||||
new org.openmetadata.schema.security.credentials.AWSCredentials()
|
||||
.withAwsRegion("us-west-2"));
|
||||
|
||||
PipelineConnection conn = new PipelineConnection().withConfig(glueConn);
|
||||
|
||||
CreatePipelineService request =
|
||||
new CreatePipelineService()
|
||||
.withName(name)
|
||||
.withServiceType(PipelineServiceType.GluePipeline)
|
||||
.withConnection(conn)
|
||||
.withDescription("Test Glue service");
|
||||
|
||||
return ns.trackRoot(
|
||||
Entity.PIPELINE_SERVICE, SdkClients.adminClient().pipelineServices().create(request));
|
||||
}
|
||||
|
||||
/** Get pipeline service by ID. */
|
||||
public static PipelineService getById(String id) {
|
||||
return SdkClients.adminClient().pipelineServices().get(id);
|
||||
}
|
||||
}
|
||||
+48
@@ -0,0 +1,48 @@
|
||||
package org.openmetadata.it.factories;
|
||||
|
||||
import org.openmetadata.it.util.SdkClients;
|
||||
import org.openmetadata.it.util.TestNamespace;
|
||||
import org.openmetadata.schema.entity.policies.Policy;
|
||||
import org.openmetadata.sdk.client.OpenMetadataClient;
|
||||
import org.openmetadata.sdk.exceptions.OpenMetadataException;
|
||||
|
||||
/**
|
||||
* Factory for accessing predefined Policy entities in integration tests.
|
||||
*
|
||||
* <p>Uses the SDK client to access policies that are predefined in the OpenMetadata system.
|
||||
*/
|
||||
public class PolicyTestFactory {
|
||||
|
||||
/**
|
||||
* Get the DataConsumerPolicy (predefined in the system).
|
||||
*/
|
||||
public static Policy getDataConsumerPolicy(TestNamespace ns) {
|
||||
return getPolicyByName("DataConsumerPolicy");
|
||||
}
|
||||
|
||||
/**
|
||||
* Get the DataStewardPolicy (predefined in the system).
|
||||
*/
|
||||
public static Policy getDataStewardPolicy(TestNamespace ns) {
|
||||
return getPolicyByName("DataStewardPolicy");
|
||||
}
|
||||
|
||||
/**
|
||||
* Get the OrganizationPolicy (predefined in the system).
|
||||
*/
|
||||
public static Policy getOrganizationPolicy(TestNamespace ns) {
|
||||
return getPolicyByName("OrganizationPolicy");
|
||||
}
|
||||
|
||||
/**
|
||||
* Get a policy by name.
|
||||
*/
|
||||
public static Policy getPolicyByName(String policyName) {
|
||||
try {
|
||||
OpenMetadataClient client = SdkClients.adminClient();
|
||||
return client.policies().getByName(policyName);
|
||||
} catch (OpenMetadataException e) {
|
||||
throw new RuntimeException("Policy not found: " + policyName, e);
|
||||
}
|
||||
}
|
||||
}
|
||||
+75
@@ -0,0 +1,75 @@
|
||||
package org.openmetadata.it.factories;
|
||||
|
||||
import java.net.URI;
|
||||
import java.util.UUID;
|
||||
import org.openmetadata.it.util.SdkClients;
|
||||
import org.openmetadata.it.util.TestNamespace;
|
||||
import org.openmetadata.schema.api.services.CreateSearchService;
|
||||
import org.openmetadata.schema.api.services.CreateSearchService.SearchServiceType;
|
||||
import org.openmetadata.schema.entity.services.SearchService;
|
||||
import org.openmetadata.schema.services.connections.search.ElasticSearchConnection;
|
||||
import org.openmetadata.schema.services.connections.search.OpenSearchConnection;
|
||||
import org.openmetadata.schema.type.SearchConnection;
|
||||
import org.openmetadata.service.Entity;
|
||||
|
||||
/**
|
||||
* Factory for creating SearchService entities in integration tests.
|
||||
*
|
||||
* <p>Provides namespace-isolated entity creation with consistent patterns.
|
||||
*/
|
||||
public class SearchServiceTestFactory {
|
||||
|
||||
/**
|
||||
* Create an ElasticSearch service with default settings. Each call creates a unique service to
|
||||
* avoid conflicts in parallel test execution.
|
||||
*/
|
||||
public static SearchService createElasticSearch(TestNamespace ns) {
|
||||
String uniqueId = UUID.randomUUID().toString().substring(0, 8);
|
||||
String name = ns.prefix("elasticService_" + uniqueId);
|
||||
|
||||
ElasticSearchConnection esConn =
|
||||
new ElasticSearchConnection().withHostPort(URI.create("http://localhost:9200"));
|
||||
|
||||
SearchConnection conn = new SearchConnection().withConfig(esConn);
|
||||
|
||||
CreateSearchService request =
|
||||
new CreateSearchService()
|
||||
.withName(name)
|
||||
.withServiceType(SearchServiceType.ElasticSearch)
|
||||
.withConnection(conn)
|
||||
.withDescription("Test ElasticSearch service");
|
||||
|
||||
return ns.trackRoot(
|
||||
Entity.SEARCH_SERVICE, SdkClients.adminClient().searchServices().create(request));
|
||||
}
|
||||
|
||||
/**
|
||||
* Create an OpenSearch service with default settings. Each call creates a unique service to
|
||||
* avoid conflicts in parallel test execution.
|
||||
*/
|
||||
public static SearchService createOpenSearch(TestNamespace ns) {
|
||||
String uniqueId = UUID.randomUUID().toString().substring(0, 8);
|
||||
String name = ns.prefix("openSearchService_" + uniqueId);
|
||||
|
||||
// Use OpenSearchConnection for OpenSearch service type
|
||||
OpenSearchConnection osConn =
|
||||
new OpenSearchConnection().withHostPort(URI.create("http://localhost:9200"));
|
||||
|
||||
SearchConnection conn = new SearchConnection().withConfig(osConn);
|
||||
|
||||
CreateSearchService request =
|
||||
new CreateSearchService()
|
||||
.withName(name)
|
||||
.withServiceType(SearchServiceType.OpenSearch)
|
||||
.withConnection(conn)
|
||||
.withDescription("Test OpenSearch service");
|
||||
|
||||
return ns.trackRoot(
|
||||
Entity.SEARCH_SERVICE, SdkClients.adminClient().searchServices().create(request));
|
||||
}
|
||||
|
||||
/** Get search service by ID. */
|
||||
public static SearchService getById(String id) {
|
||||
return SdkClients.adminClient().searchServices().get(id);
|
||||
}
|
||||
}
|
||||
+187
@@ -0,0 +1,187 @@
|
||||
package org.openmetadata.it.factories;
|
||||
|
||||
import java.time.Duration;
|
||||
import java.util.EnumMap;
|
||||
import java.util.Locale;
|
||||
import java.util.Map;
|
||||
import org.openmetadata.it.factories.EntityLoadSpec.EntityKind;
|
||||
import org.openmetadata.it.search.DbCountQuerier;
|
||||
import org.openmetadata.it.server.ServerHandle;
|
||||
import org.openmetadata.it.util.TestNamespace;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
/**
|
||||
* Configurable entry point for provisioning the entity cohort a search/reindex IT needs. Tests call
|
||||
* {@link #provision} instead of {@link EntityLoader#load} directly so the same scenario can either
|
||||
* create its data on the fly or reuse a dataset loaded once on a shared external cluster.
|
||||
*
|
||||
* <p>The mode is chosen by {@code -Djpw.data.mode} (env fallback {@code OM_DATA_MODE}):
|
||||
*
|
||||
* <ul>
|
||||
* <li>{@code ingest} (default) — always create the full cohort fresh, exactly as before. Roots are
|
||||
* tracked on the namespace so {@code TestNamespaceExtension} hard-deletes them after the test.
|
||||
* <li>{@code ensure} — create only the shortfall: for each requested kind, top up to the requested
|
||||
* count from whatever already exists. Idempotent; the top-up is tracked for cleanup.
|
||||
* <li>{@code static} — create nothing and track nothing. Asserts the cluster already holds at least
|
||||
* the requested count for each countable kind so a misconfigured run fails loudly rather than
|
||||
* passing trivially on empty indices. Pre-seed once with {@code StaticDatasetSeedIT}.
|
||||
* </ul>
|
||||
*
|
||||
* <p>{@code ensure}/{@code static} measure existing data with {@link DbCountQuerier} (DB-side
|
||||
* {@code paging.total}). Kinds it cannot count (lineage edges, time-series telemetry) are cheap
|
||||
* relative to bulk assets, so {@code ensure} always creates them and {@code static} skips them from
|
||||
* the guard. In practice the slow suites seed only {@link EntityKind#TABLE}, so the common path is
|
||||
* exact.
|
||||
*/
|
||||
public final class SeedData {
|
||||
|
||||
private static final Logger LOG = LoggerFactory.getLogger(SeedData.class);
|
||||
private static final String MODE_PROPERTY = "jpw.data.mode";
|
||||
private static final String MODE_ENV = "OM_DATA_MODE";
|
||||
|
||||
/** EntityKind → OM entity type, for the kinds {@link DbCountQuerier} can count. */
|
||||
private static final Map<EntityKind, String> COUNTABLE_TYPES = countableTypes();
|
||||
|
||||
private SeedData() {}
|
||||
|
||||
public enum Mode {
|
||||
INGEST,
|
||||
ENSURE,
|
||||
STATIC
|
||||
}
|
||||
|
||||
public static Mode mode() {
|
||||
final String raw = lookup();
|
||||
final Mode mode;
|
||||
if (raw == null || raw.isBlank()) {
|
||||
mode = Mode.INGEST;
|
||||
} else {
|
||||
mode = Mode.valueOf(raw.trim().toUpperCase(Locale.ROOT));
|
||||
}
|
||||
return mode;
|
||||
}
|
||||
|
||||
public static EntityLoadSummary provision(
|
||||
final EntityLoadSpec spec, final TestNamespace ns, final ServerHandle server) {
|
||||
final Mode mode = mode();
|
||||
LOG.info("SeedData provisioning {} entities in mode={}", spec.total(), mode);
|
||||
final EntityLoadSummary summary =
|
||||
switch (mode) {
|
||||
case INGEST -> EntityLoader.load(spec, ns);
|
||||
case ENSURE -> ensure(spec, ns, server);
|
||||
case STATIC -> useStatic(spec, server);
|
||||
};
|
||||
return summary;
|
||||
}
|
||||
|
||||
private static EntityLoadSummary ensure(
|
||||
final EntityLoadSpec spec, final TestNamespace ns, final ServerHandle server) {
|
||||
final DbCountQuerier db = new DbCountQuerier(server);
|
||||
final EntityLoadSpec shortfall = shortfallSpec(spec, db);
|
||||
final EntityLoadSummary summary;
|
||||
if (shortfall.total() > 0) {
|
||||
LOG.info("SeedData ensure: creating shortfall of {} entities", shortfall.total());
|
||||
summary = EntityLoader.load(shortfall, ns);
|
||||
} else {
|
||||
LOG.info("SeedData ensure: cohort already satisfied, nothing to create");
|
||||
summary = new EntityLoadSummary.Builder().build(Duration.ZERO);
|
||||
}
|
||||
assertPresent(spec, db);
|
||||
return summary;
|
||||
}
|
||||
|
||||
private static EntityLoadSummary useStatic(final EntityLoadSpec spec, final ServerHandle server) {
|
||||
final DbCountQuerier db = new DbCountQuerier(server);
|
||||
assertPresent(spec, db);
|
||||
final EntityLoadSummary.Builder summary = new EntityLoadSummary.Builder();
|
||||
for (final Map.Entry<EntityKind, Integer> requested : spec.counts().entrySet()) {
|
||||
final String type = COUNTABLE_TYPES.get(requested.getKey());
|
||||
if (type != null && db.canCount(type)) {
|
||||
summary.recordCreated(
|
||||
requested.getKey(), (int) Math.min(db.count(type), Integer.MAX_VALUE));
|
||||
}
|
||||
}
|
||||
return summary.build(Duration.ZERO);
|
||||
}
|
||||
|
||||
private static EntityLoadSpec shortfallSpec(final EntityLoadSpec spec, final DbCountQuerier db) {
|
||||
final EntityLoadSpec.Builder builder = copyKnobs(spec);
|
||||
for (final Map.Entry<EntityKind, Integer> requested : spec.counts().entrySet()) {
|
||||
final EntityKind kind = requested.getKey();
|
||||
final int want = requested.getValue();
|
||||
final String type = COUNTABLE_TYPES.get(kind);
|
||||
final int existing = (type != null && db.canCount(type)) ? (int) db.count(type) : 0;
|
||||
final int missing = Math.max(0, want - existing);
|
||||
if (missing > 0) {
|
||||
builder.count(kind, missing);
|
||||
}
|
||||
}
|
||||
return builder.build();
|
||||
}
|
||||
|
||||
private static void assertPresent(final EntityLoadSpec spec, final DbCountQuerier db) {
|
||||
for (final Map.Entry<EntityKind, Integer> requested : spec.counts().entrySet()) {
|
||||
final EntityKind kind = requested.getKey();
|
||||
final int want = requested.getValue();
|
||||
final String type = COUNTABLE_TYPES.get(kind);
|
||||
if (type != null && db.canCount(type)) {
|
||||
final long existing = db.count(type);
|
||||
if (existing < want) {
|
||||
throw new IllegalStateException(
|
||||
String.format(
|
||||
"%s mode requires pre-seeded data; found %d < expected %d for %s — run "
|
||||
+ "StaticDatasetSeedIT first (or use -Djpw.data.mode=ensure)",
|
||||
mode(), existing, want, kind));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private static EntityLoadSpec.Builder copyKnobs(final EntityLoadSpec spec) {
|
||||
return EntityLoadSpec.builder()
|
||||
.parallelWorkers(spec.parallelWorkers())
|
||||
.columnsPerTable(spec.columnsPerTable())
|
||||
.chartsPerDashboard(spec.chartsPerDashboard())
|
||||
.endpointsPerCollection(spec.endpointsPerCollection())
|
||||
.columnsPerDataModel(spec.columnsPerDataModel())
|
||||
.testCasesPerSuite(spec.testCasesPerSuite())
|
||||
.resultsPerTestCase(spec.resultsPerTestCase());
|
||||
}
|
||||
|
||||
private static Map<EntityKind, String> countableTypes() {
|
||||
final Map<EntityKind, String> map = new EnumMap<>(EntityKind.class);
|
||||
map.put(EntityKind.TABLE, "table");
|
||||
map.put(EntityKind.TOPIC, "topic");
|
||||
map.put(EntityKind.DASHBOARD, "dashboard");
|
||||
map.put(EntityKind.PIPELINE, "pipeline");
|
||||
map.put(EntityKind.ML_MODEL, "mlmodel");
|
||||
map.put(EntityKind.CONTAINER, "container");
|
||||
map.put(EntityKind.SEARCH_INDEX, "searchIndex");
|
||||
map.put(EntityKind.API_COLLECTION, "apiCollection");
|
||||
map.put(EntityKind.API_ENDPOINT, "apiEndpoint");
|
||||
map.put(EntityKind.QUERY, "query");
|
||||
map.put(EntityKind.GLOSSARY, "glossary");
|
||||
map.put(EntityKind.GLOSSARY_TERM, "glossaryTerm");
|
||||
map.put(EntityKind.CLASSIFICATION, "classification");
|
||||
map.put(EntityKind.TAG, "tag");
|
||||
map.put(EntityKind.USER, "user");
|
||||
map.put(EntityKind.TEAM, "team");
|
||||
map.put(EntityKind.DOMAIN, "domain");
|
||||
map.put(EntityKind.DATA_PRODUCT, "dataProduct");
|
||||
map.put(EntityKind.TEST_SUITE, "testSuite");
|
||||
map.put(EntityKind.TEST_CASE, "testCase");
|
||||
return Map.copyOf(map);
|
||||
}
|
||||
|
||||
private static String lookup() {
|
||||
final String env = System.getenv(MODE_ENV);
|
||||
final String value;
|
||||
if (env != null && !env.isBlank()) {
|
||||
value = env;
|
||||
} else {
|
||||
value = System.getProperty(MODE_PROPERTY);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
}
|
||||
+75
@@ -0,0 +1,75 @@
|
||||
package org.openmetadata.it.factories;
|
||||
|
||||
import java.util.List;
|
||||
import org.openmetadata.it.util.TestNamespace;
|
||||
import org.openmetadata.schema.entity.data.Database;
|
||||
import org.openmetadata.schema.entity.data.DatabaseSchema;
|
||||
import org.openmetadata.schema.entity.data.Table;
|
||||
import org.openmetadata.schema.entity.services.DatabaseService;
|
||||
import org.openmetadata.schema.services.connections.database.PostgresConnection;
|
||||
import org.openmetadata.schema.type.Column;
|
||||
import org.openmetadata.schema.type.ColumnDataType;
|
||||
import org.openmetadata.sdk.fluent.DatabaseSchemas;
|
||||
import org.openmetadata.sdk.fluent.DatabaseServices;
|
||||
import org.openmetadata.sdk.fluent.Databases;
|
||||
import org.openmetadata.sdk.fluent.Tables;
|
||||
import org.openmetadata.service.Entity;
|
||||
|
||||
/**
|
||||
* Builds a service → database → schema → table chain whose total FQN length is
|
||||
* bounded — every level uses {@link TestNamespace#uniqueShortId()} (~16 chars)
|
||||
* instead of {@link TestNamespace#prefix(String)} (~80–120 chars).
|
||||
*
|
||||
* <p>Use this when a test creates entities the system will downstream-name
|
||||
* (e.g., implicit executable test suites named {@code <table.fqn>.testSuite})
|
||||
* — those derived names hit MySQL column length limits when each level
|
||||
* carries the full namespace prefix.
|
||||
*/
|
||||
public final class ShortStackFactory {
|
||||
|
||||
private ShortStackFactory() {}
|
||||
|
||||
/**
|
||||
* Build the full chain and return the table. Each level shares the same
|
||||
* {@code shortPrefix} so the components are still globally unique within the
|
||||
* test run, just much shorter.
|
||||
*/
|
||||
public static Table table(final TestNamespace ns) {
|
||||
final String tag = ns.uniqueShortId();
|
||||
final DatabaseService service = ns.trackRoot(Entity.DATABASE_SERVICE, service(tag));
|
||||
final Database database = database(service, tag);
|
||||
final DatabaseSchema schema = schema(database, tag);
|
||||
return Tables.create()
|
||||
.name("tbl_" + tag)
|
||||
.inSchema(schema.getFullyQualifiedName())
|
||||
.withColumns(defaultColumns())
|
||||
.execute();
|
||||
}
|
||||
|
||||
private static DatabaseService service(final String tag) {
|
||||
final PostgresConnection conn =
|
||||
DatabaseServices.postgresConnection().hostPort("localhost:5432").username("test").build();
|
||||
return DatabaseServices.builder()
|
||||
.name("svc_" + tag)
|
||||
.connection(conn)
|
||||
.description("short stack")
|
||||
.create();
|
||||
}
|
||||
|
||||
private static Database database(final DatabaseService service, final String tag) {
|
||||
return Databases.create().name("db_" + tag).in(service.getFullyQualifiedName()).execute();
|
||||
}
|
||||
|
||||
private static DatabaseSchema schema(final Database database, final String tag) {
|
||||
return DatabaseSchemas.create()
|
||||
.name("sch_" + tag)
|
||||
.in(database.getFullyQualifiedName())
|
||||
.execute();
|
||||
}
|
||||
|
||||
private static List<Column> defaultColumns() {
|
||||
return List.of(
|
||||
new Column().withName("id").withDataType(ColumnDataType.STRING),
|
||||
new Column().withName("v").withDataType(ColumnDataType.STRING));
|
||||
}
|
||||
}
|
||||
+47
@@ -0,0 +1,47 @@
|
||||
package org.openmetadata.it.factories;
|
||||
|
||||
import java.util.UUID;
|
||||
import org.openmetadata.it.util.SdkClients;
|
||||
import org.openmetadata.it.util.TestNamespace;
|
||||
import org.openmetadata.schema.api.services.CreateStorageService;
|
||||
import org.openmetadata.schema.api.services.CreateStorageService.StorageServiceType;
|
||||
import org.openmetadata.schema.entity.services.StorageService;
|
||||
import org.openmetadata.schema.services.connections.storage.S3Connection;
|
||||
import org.openmetadata.schema.type.StorageConnection;
|
||||
import org.openmetadata.service.Entity;
|
||||
|
||||
/**
|
||||
* Factory for creating StorageService entities in integration tests.
|
||||
*
|
||||
* <p>Provides namespace-isolated entity creation with consistent patterns.
|
||||
*/
|
||||
public class StorageServiceTestFactory {
|
||||
|
||||
/**
|
||||
* Create an S3 storage service with default settings. Each call creates a unique service to
|
||||
* avoid conflicts in parallel test execution.
|
||||
*/
|
||||
public static StorageService createS3(TestNamespace ns) {
|
||||
String uniqueId = UUID.randomUUID().toString().substring(0, 8);
|
||||
String name = ns.prefix("s3Service_" + uniqueId);
|
||||
|
||||
S3Connection s3Conn = new S3Connection();
|
||||
|
||||
StorageConnection conn = new StorageConnection().withConfig(s3Conn);
|
||||
|
||||
CreateStorageService request =
|
||||
new CreateStorageService()
|
||||
.withName(name)
|
||||
.withServiceType(StorageServiceType.S3)
|
||||
.withConnection(conn)
|
||||
.withDescription("Test S3 service");
|
||||
|
||||
return ns.trackRoot(
|
||||
Entity.STORAGE_SERVICE, SdkClients.adminClient().storageServices().create(request));
|
||||
}
|
||||
|
||||
/** Get storage service by ID. */
|
||||
public static StorageService getById(String id) {
|
||||
return SdkClients.adminClient().storageServices().get(id);
|
||||
}
|
||||
}
|
||||
+93
@@ -0,0 +1,93 @@
|
||||
package org.openmetadata.it.factories;
|
||||
|
||||
import java.util.List;
|
||||
import org.openmetadata.it.util.TestNamespace;
|
||||
import org.openmetadata.schema.entity.data.Table;
|
||||
import org.openmetadata.schema.type.Column;
|
||||
import org.openmetadata.schema.type.ColumnDataType;
|
||||
import org.openmetadata.sdk.fluent.Tables;
|
||||
|
||||
/**
|
||||
* Factory for creating Table entities in integration tests using fluent API.
|
||||
*
|
||||
* <p>Uses the static fluent API from {@link Tables}. Ensure
|
||||
* {@code Tables.setDefaultClient(client)} is called before using these methods.
|
||||
*/
|
||||
public class TableTestFactory {
|
||||
|
||||
private static final List<Column> DEFAULT_COLUMNS =
|
||||
List.of(
|
||||
new Column().withName("id").withDataType(ColumnDataType.BIGINT),
|
||||
new Column().withName("name").withDataType(ColumnDataType.VARCHAR).withDataLength(255));
|
||||
|
||||
/**
|
||||
* Create a table with default settings using fluent API.
|
||||
*/
|
||||
public static Table createSimple(TestNamespace ns, String schemaFqn) {
|
||||
return Tables.create()
|
||||
.name(ns.prefix("table"))
|
||||
.inSchema(schemaFqn)
|
||||
.withColumns(DEFAULT_COLUMNS)
|
||||
.execute();
|
||||
}
|
||||
|
||||
/**
|
||||
* Create table with custom name using fluent API.
|
||||
*/
|
||||
public static Table createWithName(TestNamespace ns, String schemaFqn, String baseName) {
|
||||
return Tables.create()
|
||||
.name(ns.prefix(baseName))
|
||||
.inSchema(schemaFqn)
|
||||
.withColumns(DEFAULT_COLUMNS)
|
||||
.execute();
|
||||
}
|
||||
|
||||
/**
|
||||
* Create table with description using fluent API.
|
||||
*/
|
||||
public static Table createWithDescription(
|
||||
TestNamespace ns, String schemaFqn, String description) {
|
||||
return Tables.create()
|
||||
.name(ns.prefix("table"))
|
||||
.inSchema(schemaFqn)
|
||||
.withDescription(description)
|
||||
.withColumns(DEFAULT_COLUMNS)
|
||||
.execute();
|
||||
}
|
||||
|
||||
/**
|
||||
* Create table with a direct name (no namespace prefix) using fluent API.
|
||||
* Useful for tests that need short names to avoid FQN length limits.
|
||||
*/
|
||||
public static Table createSimpleWithName(String name, TestNamespace ns, String schemaFqn) {
|
||||
return Tables.create().name(name).inSchema(schemaFqn).withColumns(DEFAULT_COLUMNS).execute();
|
||||
}
|
||||
|
||||
/**
|
||||
* Create table with columns that have descriptions using fluent API. Useful for testing
|
||||
* column-level description updates.
|
||||
*/
|
||||
public static Table createWithColumns(TestNamespace ns, String schemaFqn) {
|
||||
List<Column> columnsWithDescriptions =
|
||||
List.of(
|
||||
new Column()
|
||||
.withName("id")
|
||||
.withDataType(ColumnDataType.BIGINT)
|
||||
.withDescription("Primary key identifier"),
|
||||
new Column()
|
||||
.withName("name")
|
||||
.withDataType(ColumnDataType.VARCHAR)
|
||||
.withDataLength(255)
|
||||
.withDescription("Entity name field"),
|
||||
new Column()
|
||||
.withName("created_at")
|
||||
.withDataType(ColumnDataType.TIMESTAMP)
|
||||
.withDescription("Record creation timestamp"));
|
||||
|
||||
return Tables.create()
|
||||
.name(ns.prefix("table_with_cols"))
|
||||
.inSchema(schemaFqn)
|
||||
.withColumns(columnsWithDescriptions)
|
||||
.execute();
|
||||
}
|
||||
}
|
||||
+79
@@ -0,0 +1,79 @@
|
||||
package org.openmetadata.it.factories;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
import org.openmetadata.it.util.SdkClients;
|
||||
import org.openmetadata.it.util.TestNamespace;
|
||||
import org.openmetadata.schema.api.data.CreateTopic;
|
||||
import org.openmetadata.schema.entity.data.Topic;
|
||||
import org.openmetadata.schema.entity.services.MessagingService;
|
||||
import org.openmetadata.schema.type.Field;
|
||||
import org.openmetadata.schema.type.FieldDataType;
|
||||
import org.openmetadata.schema.type.MessageSchema;
|
||||
import org.openmetadata.schema.type.SchemaType;
|
||||
|
||||
/**
|
||||
* Factory for creating Topic entities under a Kafka MessagingService.
|
||||
*
|
||||
* <p>The default topic ships with the schema shape consumed by UI scenarios: two top-level
|
||||
* Record fields, the first containing nested children (a nested Record with first/last
|
||||
* name, plus int and string fields) and the second empty. This mirrors {@code TopicClass}
|
||||
* in the TS Playwright suite so UI assertions about field counts and nested expansion
|
||||
* port over unchanged.
|
||||
*/
|
||||
public class TopicTestFactory {
|
||||
|
||||
private static final int DEFAULT_PARTITIONS = 128;
|
||||
|
||||
/** Creates a fresh Kafka service in the namespace and a topic under it with default schema. */
|
||||
public static Topic createSimple(TestNamespace ns) {
|
||||
MessagingService service = MessagingServiceTestFactory.createKafka(ns);
|
||||
return createSimple(ns, service.getFullyQualifiedName());
|
||||
}
|
||||
|
||||
/** Creates a topic under an existing messaging service with default schema. */
|
||||
public static Topic createSimple(TestNamespace ns, String messagingServiceFqn) {
|
||||
String uniqueId = UUID.randomUUID().toString().substring(0, 8);
|
||||
String name = ns.prefix("topic_" + uniqueId);
|
||||
|
||||
CreateTopic request =
|
||||
new CreateTopic()
|
||||
.withName(name)
|
||||
.withService(messagingServiceFqn)
|
||||
.withPartitions(DEFAULT_PARTITIONS)
|
||||
.withMessageSchema(defaultSchema());
|
||||
|
||||
return SdkClients.adminClient().topics().create(request);
|
||||
}
|
||||
|
||||
private static MessageSchema defaultSchema() {
|
||||
Field firstName =
|
||||
new Field()
|
||||
.withName("first_name")
|
||||
.withDataType(FieldDataType.STRING)
|
||||
.withDescription("Description for schema field first_name");
|
||||
Field lastName = new Field().withName("last_name").withDataType(FieldDataType.STRING);
|
||||
Field nestedName =
|
||||
new Field()
|
||||
.withName("name")
|
||||
.withDataType(FieldDataType.RECORD)
|
||||
.withChildren(List.of(firstName, lastName));
|
||||
Field age = new Field().withName("age").withDataType(FieldDataType.INT);
|
||||
Field clubName = new Field().withName("club_name").withDataType(FieldDataType.STRING);
|
||||
|
||||
Field defaultRecord =
|
||||
new Field()
|
||||
.withName("default")
|
||||
.withDataType(FieldDataType.RECORD)
|
||||
.withChildren(List.of(nestedName, age, clubName));
|
||||
Field secondaryRecord =
|
||||
new Field()
|
||||
.withName("secondary")
|
||||
.withDataType(FieldDataType.RECORD)
|
||||
.withChildren(List.of());
|
||||
|
||||
return new MessageSchema()
|
||||
.withSchemaType(SchemaType.JSON)
|
||||
.withSchemaFields(List.of(defaultRecord, secondaryRecord));
|
||||
}
|
||||
}
|
||||
+85
@@ -0,0 +1,85 @@
|
||||
package org.openmetadata.it.factories;
|
||||
|
||||
import java.util.UUID;
|
||||
import org.openmetadata.it.util.TestNamespace;
|
||||
import org.openmetadata.schema.entity.teams.User;
|
||||
import org.openmetadata.sdk.exceptions.OpenMetadataException;
|
||||
import org.openmetadata.sdk.fluent.Users;
|
||||
import org.openmetadata.service.Entity;
|
||||
|
||||
/**
|
||||
* Factory for creating User entities in integration tests using fluent API.
|
||||
*
|
||||
* <p>Uses the static fluent API from {@link Users}. Ensure
|
||||
* fluent APIs are initialized before using these methods.
|
||||
*/
|
||||
public class UserTestFactory {
|
||||
|
||||
private static final int HTTP_NOT_FOUND = 404;
|
||||
private static final int HTTP_CONFLICT = 409;
|
||||
|
||||
/**
|
||||
* Create a user with unique name using fluent API.
|
||||
*/
|
||||
public static User createUser(TestNamespace ns, String baseName) {
|
||||
// Use shorter UUID to avoid name length issues
|
||||
String uniqueSuffix = UUID.randomUUID().toString().substring(0, 8);
|
||||
String name = baseName + "_" + uniqueSuffix;
|
||||
String email = name + "@test.om.org";
|
||||
|
||||
// Check if user already exists
|
||||
try {
|
||||
return Users.findByName(email).fetch().get();
|
||||
} catch (OpenMetadataException e) {
|
||||
// User doesn't exist, create it
|
||||
}
|
||||
|
||||
try {
|
||||
return ns.trackRoot(
|
||||
Entity.USER,
|
||||
Users.create().name(name).withEmail(email).withDescription("Test user").execute());
|
||||
} catch (OpenMetadataException e) {
|
||||
throw new RuntimeException("Failed to create user: " + email, e);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Get or create a user by email using fluent API. The User entity's FQN is its name (the
|
||||
* email's local part), not the email itself — so we look up by the derived name. The create
|
||||
* path tolerates a 409 conflict from a parallel caller by re-fetching, since multiple test
|
||||
* classes may invoke this concurrently from their {@code @BeforeAll}. Any other status code
|
||||
* propagates immediately so misconfigured setups fail fast rather than masking later 404s.
|
||||
*/
|
||||
public static User getOrCreateUser(String email) {
|
||||
String name = email.split("@")[0];
|
||||
try {
|
||||
return Users.findByName(name).fetch().get();
|
||||
} catch (OpenMetadataException notFound) {
|
||||
if (notFound.getStatusCode() != HTTP_NOT_FOUND) {
|
||||
throw notFound;
|
||||
}
|
||||
try {
|
||||
return Users.create().name(name).withEmail(email).withDescription("Test user").execute();
|
||||
} catch (OpenMetadataException conflict) {
|
||||
if (conflict.getStatusCode() != HTTP_CONFLICT) {
|
||||
throw conflict;
|
||||
}
|
||||
return Users.findByName(name).fetch().get();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Get the DataConsumer user (predefined in the system).
|
||||
*/
|
||||
public static User getDataConsumer(TestNamespace ns) {
|
||||
return getOrCreateUser("data-consumer@open-metadata.org");
|
||||
}
|
||||
|
||||
/**
|
||||
* Get the DataSteward user (predefined in the system).
|
||||
*/
|
||||
public static User getDataSteward(TestNamespace ns) {
|
||||
return getOrCreateUser("data-steward@open-metadata.org");
|
||||
}
|
||||
}
|
||||
+376
@@ -0,0 +1,376 @@
|
||||
/*
|
||||
* Copyright 2021 Collate
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.openmetadata.it.k8s;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
|
||||
import io.fabric8.kubernetes.api.model.ConfigMapKeySelectorBuilder;
|
||||
import io.fabric8.kubernetes.api.model.EnvVar;
|
||||
import io.fabric8.kubernetes.api.model.EnvVarBuilder;
|
||||
import io.fabric8.kubernetes.api.model.EnvVarSource;
|
||||
import io.fabric8.kubernetes.api.model.EnvVarSourceBuilder;
|
||||
import io.kubernetes.client.openapi.models.V1ConfigMapKeySelector;
|
||||
import io.kubernetes.client.openapi.models.V1EnvVar;
|
||||
import io.kubernetes.client.openapi.models.V1EnvVarSource;
|
||||
import java.util.Arrays;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.openmetadata.service.clients.pipeline.k8s.CronOMJob;
|
||||
import org.openmetadata.service.clients.pipeline.k8s.K8sJobUtils;
|
||||
import org.openmetadata.service.clients.pipeline.k8s.OMJob;
|
||||
|
||||
/**
|
||||
* Integration test for K8s environment variable serialization.
|
||||
* Validates that ConfigMapKeyRef and other valueFrom sources are preserved
|
||||
* through the full serialization/deserialization cycle.
|
||||
*/
|
||||
class K8sEnvVarSerializationIT {
|
||||
|
||||
private static final String PIPELINE_NAME = "test-pipeline";
|
||||
private static final String NAMESPACE = "openmetadata-pipelines-test";
|
||||
private static final String CONFIG_MAP_NAME = "om-config-" + PIPELINE_NAME;
|
||||
|
||||
private final ObjectMapper yamlMapper = new ObjectMapper(new YAMLFactory());
|
||||
private final ObjectMapper jsonMapper = new ObjectMapper();
|
||||
|
||||
@Test
|
||||
void testOMJobSerializationPreservesConfigMapKeyRef() {
|
||||
// Create env vars with ConfigMapKeyRef
|
||||
V1EnvVar configEnvVar =
|
||||
new V1EnvVar()
|
||||
.name("config")
|
||||
.valueFrom(
|
||||
new V1EnvVarSource()
|
||||
.configMapKeyRef(
|
||||
new V1ConfigMapKeySelector().name(CONFIG_MAP_NAME).key("config")));
|
||||
|
||||
List<V1EnvVar> envVars =
|
||||
List.of(new V1EnvVar().name("pipelineType").value("metadata"), configEnvVar);
|
||||
|
||||
OMJob.OMJobPodSpec podSpec =
|
||||
OMJob.OMJobPodSpec.builder()
|
||||
.image("openmetadata/ingestion:latest")
|
||||
.command(List.of("python", "main.py"))
|
||||
.env(envVars)
|
||||
.build();
|
||||
|
||||
OMJob omJob =
|
||||
OMJob.builder()
|
||||
.apiVersion("pipelines.openmetadata.org/v1")
|
||||
.kind("OMJob")
|
||||
.metadata(OMJob.OMJobMetadata.builder().name("test-omjob").namespace(NAMESPACE).build())
|
||||
.spec(OMJob.OMJobSpec.builder().mainPodSpec(podSpec).exitHandlerSpec(podSpec).build())
|
||||
.build();
|
||||
|
||||
// Convert to Map
|
||||
Map<String, Object> omJobMap = omJob.toMap();
|
||||
|
||||
// Verify ConfigMapKeyRef is preserved
|
||||
verifyConfigMapKeyRefInMap(omJobMap, "spec.mainPodSpec");
|
||||
verifyConfigMapKeyRefInMap(omJobMap, "spec.exitHandlerSpec");
|
||||
}
|
||||
|
||||
@Test
|
||||
void testCronOMJobSerializationPreservesConfigMapKeyRef() {
|
||||
// Create env vars with ConfigMapKeyRef
|
||||
V1EnvVar configEnvVar =
|
||||
new V1EnvVar()
|
||||
.name("config")
|
||||
.valueFrom(
|
||||
new V1EnvVarSource()
|
||||
.configMapKeyRef(
|
||||
new V1ConfigMapKeySelector().name(CONFIG_MAP_NAME).key("config")));
|
||||
|
||||
List<V1EnvVar> envVars =
|
||||
List.of(
|
||||
new V1EnvVar().name("pipelineType").value("metadata"),
|
||||
new V1EnvVar().name("pipelineRunId").value("scheduled"),
|
||||
configEnvVar);
|
||||
|
||||
OMJob.OMJobPodSpec podSpec =
|
||||
OMJob.OMJobPodSpec.builder()
|
||||
.image("openmetadata/ingestion:latest")
|
||||
.command(List.of("python", "main.py"))
|
||||
.env(envVars)
|
||||
.build();
|
||||
|
||||
CronOMJob cronOMJob =
|
||||
CronOMJob.builder()
|
||||
.apiVersion("pipelines.openmetadata.org/v1")
|
||||
.kind("CronOMJob")
|
||||
.metadata(
|
||||
CronOMJob.CronOMJobMetadata.builder()
|
||||
.name("om-cronomjob-" + PIPELINE_NAME)
|
||||
.namespace(NAMESPACE)
|
||||
.build())
|
||||
.spec(
|
||||
CronOMJob.CronOMJobSpec.builder()
|
||||
.schedule("0 * * * *")
|
||||
.omJobSpec(
|
||||
OMJob.OMJobSpec.builder()
|
||||
.mainPodSpec(podSpec)
|
||||
.exitHandlerSpec(podSpec)
|
||||
.build())
|
||||
.build())
|
||||
.build();
|
||||
|
||||
// Convert to Map
|
||||
Map<String, Object> cronOMJobMap = cronOMJob.toMap();
|
||||
|
||||
// Verify ConfigMapKeyRef is preserved
|
||||
verifyConfigMapKeyRefInMap(cronOMJobMap, "spec.omJobSpec.mainPodSpec");
|
||||
verifyConfigMapKeyRefInMap(cronOMJobMap, "spec.omJobSpec.exitHandlerSpec");
|
||||
}
|
||||
|
||||
@Test
|
||||
void testK8sJobUtilsPreservesAllValueFromTypes() {
|
||||
// Test all types of valueFrom sources
|
||||
List<V1EnvVar> envVars =
|
||||
List.of(
|
||||
new V1EnvVar().name("SIMPLE").value("simple_value"),
|
||||
new V1EnvVar()
|
||||
.name("CONFIG_MAP")
|
||||
.valueFrom(
|
||||
new V1EnvVarSource()
|
||||
.configMapKeyRef(new V1ConfigMapKeySelector().name("config").key("data"))),
|
||||
new V1EnvVar()
|
||||
.name("SECRET")
|
||||
.valueFrom(
|
||||
new V1EnvVarSource()
|
||||
.secretKeyRef(
|
||||
new io.kubernetes.client.openapi.models.V1SecretKeySelector()
|
||||
.name("secret")
|
||||
.key("password"))),
|
||||
new V1EnvVar()
|
||||
.name("FIELD")
|
||||
.valueFrom(
|
||||
new V1EnvVarSource()
|
||||
.fieldRef(
|
||||
new io.kubernetes.client.openapi.models.V1ObjectFieldSelector()
|
||||
.fieldPath("metadata.name"))));
|
||||
|
||||
List<Map<String, Object>> envMapList = K8sJobUtils.convertEnvVarsToMap(envVars);
|
||||
|
||||
assertEquals(4, envMapList.size());
|
||||
|
||||
// Simple value
|
||||
Map<String, Object> simpleEnv = envMapList.get(0);
|
||||
assertEquals("SIMPLE", simpleEnv.get("name"));
|
||||
assertEquals("simple_value", simpleEnv.get("value"));
|
||||
assertFalse(simpleEnv.containsKey("valueFrom"));
|
||||
|
||||
// ConfigMapKeyRef
|
||||
Map<String, Object> configMapEnv = envMapList.get(1);
|
||||
assertEquals("CONFIG_MAP", configMapEnv.get("name"));
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, Object> configMapValueFrom = (Map<String, Object>) configMapEnv.get("valueFrom");
|
||||
assertNotNull(configMapValueFrom);
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, Object> configMapRef =
|
||||
(Map<String, Object>) configMapValueFrom.get("configMapKeyRef");
|
||||
assertNotNull(configMapRef);
|
||||
assertEquals("config", configMapRef.get("name"));
|
||||
assertEquals("data", configMapRef.get("key"));
|
||||
|
||||
// SecretKeyRef
|
||||
Map<String, Object> secretEnv = envMapList.get(2);
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, Object> secretValueFrom = (Map<String, Object>) secretEnv.get("valueFrom");
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, Object> secretRef = (Map<String, Object>) secretValueFrom.get("secretKeyRef");
|
||||
assertNotNull(secretRef);
|
||||
assertEquals("secret", secretRef.get("name"));
|
||||
assertEquals("password", secretRef.get("key"));
|
||||
|
||||
// FieldRef
|
||||
Map<String, Object> fieldEnv = envMapList.get(3);
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, Object> fieldValueFrom = (Map<String, Object>) fieldEnv.get("valueFrom");
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, Object> fieldRef = (Map<String, Object>) fieldValueFrom.get("fieldRef");
|
||||
assertNotNull(fieldRef);
|
||||
assertEquals("metadata.name", fieldRef.get("fieldPath"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void testEmptyValueFromHandling() {
|
||||
// Test that we can detect empty valueFrom fields that would cause K8s API errors
|
||||
List<EnvVar> envVars =
|
||||
Arrays.asList(
|
||||
new EnvVarBuilder().withName("GOOD").withValue("value").build(),
|
||||
new EnvVarBuilder()
|
||||
.withName("EMPTY_VALUE_FROM")
|
||||
.withValueFrom(new EnvVarSource()) // Empty - would cause K8s API error
|
||||
.build(),
|
||||
new EnvVarBuilder()
|
||||
.withName("CONFIG_REF")
|
||||
.withValueFrom(
|
||||
new EnvVarSourceBuilder()
|
||||
.withConfigMapKeyRef(
|
||||
new ConfigMapKeySelectorBuilder()
|
||||
.withName(CONFIG_MAP_NAME)
|
||||
.withKey("config")
|
||||
.build())
|
||||
.build())
|
||||
.build());
|
||||
|
||||
// Check which env vars have empty valueFrom
|
||||
for (EnvVar env : envVars) {
|
||||
if ("EMPTY_VALUE_FROM".equals(env.getName())) {
|
||||
assertNotNull(env.getValueFrom());
|
||||
// This valueFrom is empty (no valid references)
|
||||
assertNull(env.getValueFrom().getConfigMapKeyRef());
|
||||
assertNull(env.getValueFrom().getSecretKeyRef());
|
||||
assertNull(env.getValueFrom().getFieldRef());
|
||||
assertNull(env.getValueFrom().getResourceFieldRef());
|
||||
} else if ("CONFIG_REF".equals(env.getName())) {
|
||||
assertNotNull(env.getValueFrom());
|
||||
assertNotNull(env.getValueFrom().getConfigMapKeyRef());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void testYamlSerializationRoundTrip() throws Exception {
|
||||
// Test YAML serialization/deserialization preserves ConfigMapKeyRef
|
||||
CronOMJob cronOMJob = buildCronOMJobWithConfigMapRef();
|
||||
Map<String, Object> cronOMJobMap = cronOMJob.toMap();
|
||||
|
||||
// Serialize to YAML
|
||||
String yaml = yamlMapper.writeValueAsString(cronOMJobMap);
|
||||
assertNotNull(yaml);
|
||||
assertTrue(yaml.contains("configMapKeyRef"));
|
||||
assertTrue(yaml.contains(CONFIG_MAP_NAME));
|
||||
|
||||
// Parse back from YAML
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, Object> parsedMap = yamlMapper.readValue(yaml, Map.class);
|
||||
|
||||
// Verify ConfigMapKeyRef survived round trip
|
||||
verifyConfigMapKeyRefInMap(parsedMap, "spec.omJobSpec.mainPodSpec");
|
||||
}
|
||||
|
||||
@Test
|
||||
void testJsonSerializationRoundTrip() throws Exception {
|
||||
// Test JSON serialization/deserialization preserves ConfigMapKeyRef
|
||||
OMJob omJob = buildOMJobWithConfigMapRef();
|
||||
Map<String, Object> omJobMap = omJob.toMap();
|
||||
|
||||
// Serialize to JSON
|
||||
String json = jsonMapper.writeValueAsString(omJobMap);
|
||||
assertNotNull(json);
|
||||
assertTrue(json.contains("configMapKeyRef"));
|
||||
assertTrue(json.contains(CONFIG_MAP_NAME));
|
||||
|
||||
// Parse back from JSON
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, Object> parsedMap = jsonMapper.readValue(json, Map.class);
|
||||
|
||||
// Verify ConfigMapKeyRef survived round trip
|
||||
verifyConfigMapKeyRefInMap(parsedMap, "spec.mainPodSpec");
|
||||
}
|
||||
|
||||
private void verifyConfigMapKeyRefInMap(Map<String, Object> map, String path) {
|
||||
String[] pathParts = path.split("\\.");
|
||||
Map<String, Object> current = map;
|
||||
|
||||
for (String part : pathParts) {
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, Object> next = (Map<String, Object>) current.get(part);
|
||||
assertNotNull(next, "Path " + part + " should exist in " + path);
|
||||
current = next;
|
||||
}
|
||||
|
||||
@SuppressWarnings("unchecked")
|
||||
List<Map<String, Object>> envList = (List<Map<String, Object>>) current.get("env");
|
||||
assertNotNull(envList, "env should exist at " + path);
|
||||
|
||||
// Find config env var
|
||||
Map<String, Object> configEnv =
|
||||
envList.stream()
|
||||
.filter(env -> "config".equals(env.get("name")))
|
||||
.findFirst()
|
||||
.orElseThrow(() -> new AssertionError("config env var not found at " + path));
|
||||
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, Object> valueFrom = (Map<String, Object>) configEnv.get("valueFrom");
|
||||
assertNotNull(valueFrom, "valueFrom should exist for config env at " + path);
|
||||
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, Object> configMapKeyRef = (Map<String, Object>) valueFrom.get("configMapKeyRef");
|
||||
assertNotNull(configMapKeyRef, "configMapKeyRef must be preserved at " + path);
|
||||
assertEquals(CONFIG_MAP_NAME, configMapKeyRef.get("name"));
|
||||
assertEquals("config", configMapKeyRef.get("key"));
|
||||
}
|
||||
|
||||
private CronOMJob buildCronOMJobWithConfigMapRef() {
|
||||
V1EnvVar configEnvVar =
|
||||
new V1EnvVar()
|
||||
.name("config")
|
||||
.valueFrom(
|
||||
new V1EnvVarSource()
|
||||
.configMapKeyRef(
|
||||
new V1ConfigMapKeySelector().name(CONFIG_MAP_NAME).key("config")));
|
||||
|
||||
OMJob.OMJobPodSpec podSpec =
|
||||
OMJob.OMJobPodSpec.builder()
|
||||
.image("openmetadata/ingestion:latest")
|
||||
.command(List.of("python", "main.py"))
|
||||
.env(List.of(new V1EnvVar().name("pipelineType").value("metadata"), configEnvVar))
|
||||
.build();
|
||||
|
||||
return CronOMJob.builder()
|
||||
.apiVersion("pipelines.openmetadata.org/v1")
|
||||
.kind("CronOMJob")
|
||||
.metadata(
|
||||
CronOMJob.CronOMJobMetadata.builder()
|
||||
.name("om-cronomjob-" + PIPELINE_NAME)
|
||||
.namespace(NAMESPACE)
|
||||
.build())
|
||||
.spec(
|
||||
CronOMJob.CronOMJobSpec.builder()
|
||||
.schedule("0 * * * *")
|
||||
.omJobSpec(
|
||||
OMJob.OMJobSpec.builder().mainPodSpec(podSpec).exitHandlerSpec(podSpec).build())
|
||||
.build())
|
||||
.build();
|
||||
}
|
||||
|
||||
private OMJob buildOMJobWithConfigMapRef() {
|
||||
V1EnvVar configEnvVar =
|
||||
new V1EnvVar()
|
||||
.name("config")
|
||||
.valueFrom(
|
||||
new V1EnvVarSource()
|
||||
.configMapKeyRef(
|
||||
new V1ConfigMapKeySelector().name(CONFIG_MAP_NAME).key("config")));
|
||||
|
||||
OMJob.OMJobPodSpec podSpec =
|
||||
OMJob.OMJobPodSpec.builder()
|
||||
.image("openmetadata/ingestion:latest")
|
||||
.command(List.of("python", "main.py"))
|
||||
.env(List.of(new V1EnvVar().name("pipelineType").value("metadata"), configEnvVar))
|
||||
.build();
|
||||
|
||||
return OMJob.builder()
|
||||
.apiVersion("pipelines.openmetadata.org/v1")
|
||||
.kind("OMJob")
|
||||
.metadata(OMJob.OMJobMetadata.builder().name("test-omjob").namespace(NAMESPACE).build())
|
||||
.spec(OMJob.OMJobSpec.builder().mainPodSpec(podSpec).exitHandlerSpec(podSpec).build())
|
||||
.build();
|
||||
}
|
||||
}
|
||||
+281
@@ -0,0 +1,281 @@
|
||||
/*
|
||||
* Copyright 2021 Collate
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.openmetadata.it.k8s;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.*;
|
||||
|
||||
import io.kubernetes.client.openapi.models.V1ConfigMapKeySelector;
|
||||
import io.kubernetes.client.openapi.models.V1EnvVar;
|
||||
import io.kubernetes.client.openapi.models.V1EnvVarSource;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.openmetadata.service.clients.pipeline.k8s.CronOMJob;
|
||||
import org.openmetadata.service.clients.pipeline.k8s.OMJob;
|
||||
|
||||
/**
|
||||
* Integration test for K8s Pipeline environment variable flow.
|
||||
* Validates that ConfigMapKeyRef and other valueFrom sources are preserved
|
||||
* from K8sPipelineClient through CronOMJob/OMJob to pod creation.
|
||||
*/
|
||||
class K8sPipelineEnvVarIT {
|
||||
|
||||
private static final String PIPELINE_NAME = "test-pipeline-envvar";
|
||||
private static final String NAMESPACE = "openmetadata-pipelines-test";
|
||||
private static final String CONFIG_MAP_NAME = "om-config-" + PIPELINE_NAME;
|
||||
|
||||
@BeforeEach
|
||||
void setUp() {
|
||||
// Tests will validate serialization logic only, not actual K8s interaction
|
||||
}
|
||||
|
||||
@Test
|
||||
void testCronOMJobPreservesConfigMapKeyRef() {
|
||||
// Create env vars with ConfigMapKeyRef as K8sPipelineClient does
|
||||
V1EnvVar configEnvVar =
|
||||
new V1EnvVar()
|
||||
.name("config")
|
||||
.valueFrom(
|
||||
new V1EnvVarSource()
|
||||
.configMapKeyRef(
|
||||
new V1ConfigMapKeySelector().name(CONFIG_MAP_NAME).key("config")));
|
||||
|
||||
List<V1EnvVar> envVars =
|
||||
List.of(
|
||||
new V1EnvVar().name("pipelineType").value("metadata"),
|
||||
new V1EnvVar().name("pipelineRunId").value("scheduled"),
|
||||
new V1EnvVar().name("ingestionPipelineFQN").value("service." + PIPELINE_NAME),
|
||||
configEnvVar,
|
||||
new V1EnvVar().name("jobName").value("om-cronomjob-" + PIPELINE_NAME),
|
||||
new V1EnvVar().name("namespace").value(NAMESPACE));
|
||||
|
||||
// Build CronOMJob as K8sPipelineClient would
|
||||
OMJob.OMJobPodSpec mainPodSpec =
|
||||
OMJob.OMJobPodSpec.builder()
|
||||
.image("openmetadata/ingestion:latest")
|
||||
.imagePullPolicy("IfNotPresent")
|
||||
.command(List.of("python", "main.py"))
|
||||
.env(envVars)
|
||||
.serviceAccountName("openmetadata-ingestion")
|
||||
.build();
|
||||
|
||||
OMJob.OMJobPodSpec exitHandlerSpec =
|
||||
OMJob.OMJobPodSpec.builder()
|
||||
.image("openmetadata/ingestion:latest")
|
||||
.imagePullPolicy("IfNotPresent")
|
||||
.command(List.of("python", "exit_handler.py"))
|
||||
.env(envVars)
|
||||
.serviceAccountName("openmetadata-ingestion")
|
||||
.build();
|
||||
|
||||
OMJob.OMJobSpec omJobSpec =
|
||||
OMJob.OMJobSpec.builder()
|
||||
.mainPodSpec(mainPodSpec)
|
||||
.exitHandlerSpec(exitHandlerSpec)
|
||||
.ttlSecondsAfterFinished(3600)
|
||||
.build();
|
||||
|
||||
CronOMJob cronOMJob =
|
||||
CronOMJob.builder()
|
||||
.apiVersion("pipelines.openmetadata.org/v1")
|
||||
.kind("CronOMJob")
|
||||
.metadata(
|
||||
CronOMJob.CronOMJobMetadata.builder()
|
||||
.name("om-cronomjob-" + PIPELINE_NAME)
|
||||
.namespace(NAMESPACE)
|
||||
.build())
|
||||
.spec(
|
||||
CronOMJob.CronOMJobSpec.builder()
|
||||
.schedule("0 * * * *")
|
||||
.omJobSpec(omJobSpec)
|
||||
.build())
|
||||
.build();
|
||||
|
||||
// Convert to Map (what gets sent to K8s API)
|
||||
Map<String, Object> cronOMJobMap = cronOMJob.toMap();
|
||||
|
||||
// Verify the structure
|
||||
assertNotNull(cronOMJobMap);
|
||||
assertEquals("CronOMJob", cronOMJobMap.get("kind"));
|
||||
|
||||
// Navigate to env vars and verify ConfigMapKeyRef is preserved
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, Object> spec = (Map<String, Object>) cronOMJobMap.get("spec");
|
||||
assertNotNull(spec);
|
||||
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, Object> omJobSpecMap = (Map<String, Object>) spec.get("omJobSpec");
|
||||
assertNotNull(omJobSpecMap);
|
||||
|
||||
// Check mainPodSpec
|
||||
verifyPodSpecEnvVars(omJobSpecMap, "mainPodSpec");
|
||||
|
||||
// Check exitHandlerSpec
|
||||
verifyPodSpecEnvVars(omJobSpecMap, "exitHandlerSpec");
|
||||
}
|
||||
|
||||
@Test
|
||||
void testOMJobPreservesConfigMapKeyRef() {
|
||||
// Create env vars with various valueFrom sources
|
||||
List<V1EnvVar> envVars =
|
||||
List.of(
|
||||
new V1EnvVar().name("SIMPLE").value("simple_value"),
|
||||
new V1EnvVar()
|
||||
.name("CONFIG_MAP")
|
||||
.valueFrom(
|
||||
new V1EnvVarSource()
|
||||
.configMapKeyRef(new V1ConfigMapKeySelector().name("config").key("data"))),
|
||||
new V1EnvVar()
|
||||
.name("SECRET")
|
||||
.valueFrom(
|
||||
new V1EnvVarSource()
|
||||
.secretKeyRef(
|
||||
new io.kubernetes.client.openapi.models.V1SecretKeySelector()
|
||||
.name("secret")
|
||||
.key("password"))),
|
||||
new V1EnvVar()
|
||||
.name("FIELD")
|
||||
.valueFrom(
|
||||
new V1EnvVarSource()
|
||||
.fieldRef(
|
||||
new io.kubernetes.client.openapi.models.V1ObjectFieldSelector()
|
||||
.fieldPath("metadata.name"))));
|
||||
|
||||
OMJob.OMJobPodSpec podSpec =
|
||||
OMJob.OMJobPodSpec.builder()
|
||||
.image("test:latest")
|
||||
.command(List.of("echo", "test"))
|
||||
.env(envVars)
|
||||
.build();
|
||||
|
||||
OMJob omJob =
|
||||
OMJob.builder()
|
||||
.apiVersion("pipelines.openmetadata.org/v1")
|
||||
.kind("OMJob")
|
||||
.metadata(OMJob.OMJobMetadata.builder().name("test-omjob").namespace(NAMESPACE).build())
|
||||
.spec(OMJob.OMJobSpec.builder().mainPodSpec(podSpec).exitHandlerSpec(podSpec).build())
|
||||
.build();
|
||||
|
||||
Map<String, Object> omJobMap = omJob.toMap();
|
||||
|
||||
// Verify all env var types are preserved
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, Object> spec = (Map<String, Object>) omJobMap.get("spec");
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, Object> mainPodSpecMap = (Map<String, Object>) spec.get("mainPodSpec");
|
||||
@SuppressWarnings("unchecked")
|
||||
List<Map<String, Object>> envList = (List<Map<String, Object>>) mainPodSpecMap.get("env");
|
||||
|
||||
assertEquals(4, envList.size());
|
||||
|
||||
// Check simple value
|
||||
Map<String, Object> simpleEnv = envList.get(0);
|
||||
assertEquals("SIMPLE", simpleEnv.get("name"));
|
||||
assertEquals("simple_value", simpleEnv.get("value"));
|
||||
assertFalse(simpleEnv.containsKey("valueFrom"));
|
||||
|
||||
// Check ConfigMapKeyRef
|
||||
Map<String, Object> configMapEnv = envList.get(1);
|
||||
assertEquals("CONFIG_MAP", configMapEnv.get("name"));
|
||||
assertFalse(configMapEnv.containsKey("value"));
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, Object> configMapValueFrom = (Map<String, Object>) configMapEnv.get("valueFrom");
|
||||
assertNotNull(configMapValueFrom);
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, Object> configMapRef =
|
||||
(Map<String, Object>) configMapValueFrom.get("configMapKeyRef");
|
||||
assertNotNull(configMapRef, "ConfigMapKeyRef must be preserved");
|
||||
assertEquals("config", configMapRef.get("name"));
|
||||
assertEquals("data", configMapRef.get("key"));
|
||||
|
||||
// Check SecretKeyRef
|
||||
Map<String, Object> secretEnv = envList.get(2);
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, Object> secretValueFrom = (Map<String, Object>) secretEnv.get("valueFrom");
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, Object> secretRef = (Map<String, Object>) secretValueFrom.get("secretKeyRef");
|
||||
assertNotNull(secretRef, "SecretKeyRef must be preserved");
|
||||
assertEquals("secret", secretRef.get("name"));
|
||||
assertEquals("password", secretRef.get("key"));
|
||||
|
||||
// Check FieldRef
|
||||
Map<String, Object> fieldEnv = envList.get(3);
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, Object> fieldValueFrom = (Map<String, Object>) fieldEnv.get("valueFrom");
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, Object> fieldRef = (Map<String, Object>) fieldValueFrom.get("fieldRef");
|
||||
assertNotNull(fieldRef, "FieldRef must be preserved");
|
||||
assertEquals("metadata.name", fieldRef.get("fieldPath"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void testK8sPipelineClientCreatesConfigMapReference() {
|
||||
// This test validates that the ConfigMapKeyRef is properly created
|
||||
// in the env var structure when building a CronOMJob
|
||||
|
||||
V1EnvVar configEnvVar =
|
||||
new V1EnvVar()
|
||||
.name("config")
|
||||
.valueFrom(
|
||||
new V1EnvVarSource()
|
||||
.configMapKeyRef(
|
||||
new V1ConfigMapKeySelector().name(CONFIG_MAP_NAME).key("config")));
|
||||
|
||||
assertNotNull(configEnvVar.getValueFrom());
|
||||
assertNotNull(configEnvVar.getValueFrom().getConfigMapKeyRef());
|
||||
assertEquals(CONFIG_MAP_NAME, configEnvVar.getValueFrom().getConfigMapKeyRef().getName());
|
||||
assertEquals("config", configEnvVar.getValueFrom().getConfigMapKeyRef().getKey());
|
||||
}
|
||||
|
||||
private void verifyPodSpecEnvVars(Map<String, Object> omJobSpecMap, String podSpecKey) {
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, Object> podSpecMap = (Map<String, Object>) omJobSpecMap.get(podSpecKey);
|
||||
assertNotNull(podSpecMap);
|
||||
|
||||
@SuppressWarnings("unchecked")
|
||||
List<Map<String, Object>> envList = (List<Map<String, Object>>) podSpecMap.get("env");
|
||||
assertNotNull(envList);
|
||||
assertEquals(6, envList.size());
|
||||
|
||||
// Find the config env var
|
||||
Map<String, Object> configEnv =
|
||||
envList.stream()
|
||||
.filter(env -> "config".equals(env.get("name")))
|
||||
.findFirst()
|
||||
.orElseThrow(() -> new AssertionError("config env var not found in " + podSpecKey));
|
||||
|
||||
// Verify ConfigMapKeyRef is preserved
|
||||
assertFalse(configEnv.containsKey("value"), "config env should not have 'value'");
|
||||
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, Object> valueFrom = (Map<String, Object>) configEnv.get("valueFrom");
|
||||
assertNotNull(valueFrom, "valueFrom must be present for config env in " + podSpecKey);
|
||||
|
||||
@SuppressWarnings("unchecked")
|
||||
Map<String, Object> configMapKeyRef = (Map<String, Object>) valueFrom.get("configMapKeyRef");
|
||||
assertNotNull(configMapKeyRef, "configMapKeyRef must be preserved in " + podSpecKey);
|
||||
assertEquals(CONFIG_MAP_NAME, configMapKeyRef.get("name"));
|
||||
assertEquals("config", configMapKeyRef.get("key"));
|
||||
|
||||
// Verify other env vars
|
||||
Map<String, Object> pipelineTypeEnv =
|
||||
envList.stream()
|
||||
.filter(env -> "pipelineType".equals(env.get("name")))
|
||||
.findFirst()
|
||||
.orElseThrow();
|
||||
assertEquals("metadata", pipelineTypeEnv.get("value"));
|
||||
assertFalse(pipelineTypeEnv.containsKey("valueFrom"));
|
||||
}
|
||||
}
|
||||
+520
@@ -0,0 +1,520 @@
|
||||
package org.openmetadata.it.knowledge;
|
||||
|
||||
import static org.awaitility.Awaitility.await;
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertNotNull;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
|
||||
import com.fasterxml.jackson.core.type.TypeReference;
|
||||
import jakarta.ws.rs.core.Response;
|
||||
import java.time.Duration;
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
import org.apache.http.client.HttpResponseException;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.extension.ExtendWith;
|
||||
import org.openmetadata.schema.api.context.CreateContextMemory;
|
||||
import org.openmetadata.schema.api.data.CreatePage;
|
||||
import org.openmetadata.schema.api.domains.CreateDataProduct;
|
||||
import org.openmetadata.schema.api.domains.CreateDomain;
|
||||
import org.openmetadata.schema.entity.context.ContextMemory;
|
||||
import org.openmetadata.schema.entity.context.MemoryStats;
|
||||
import org.openmetadata.schema.entity.data.Article;
|
||||
import org.openmetadata.schema.entity.data.ExtractionStats;
|
||||
import org.openmetadata.schema.entity.data.Page;
|
||||
import org.openmetadata.schema.entity.data.PageProcessingStatus;
|
||||
import org.openmetadata.schema.entity.data.PageType;
|
||||
import org.openmetadata.schema.entity.domains.DataProduct;
|
||||
import org.openmetadata.schema.entity.domains.Domain;
|
||||
import org.openmetadata.schema.entity.teams.Team;
|
||||
import org.openmetadata.schema.entity.teams.User;
|
||||
import org.openmetadata.schema.type.EntityReference;
|
||||
import org.openmetadata.schema.utils.JsonUtils;
|
||||
import org.openmetadata.schema.utils.ResultList;
|
||||
import org.openmetadata.sdk.client.OpenMetadataClient;
|
||||
import org.openmetadata.sdk.services.domains.DataProductService;
|
||||
import org.openmetadata.sdk.services.domains.DomainService;
|
||||
import org.openmetadata.sdk.services.teams.TeamService;
|
||||
import org.openmetadata.sdk.services.teams.UserService;
|
||||
import org.openmetadata.sdk.test.util.RestClient;
|
||||
import org.openmetadata.sdk.test.util.SdkClients;
|
||||
import org.openmetadata.sdk.test.util.TestNamespace;
|
||||
import org.openmetadata.sdk.test.util.TestNamespaceExtension;
|
||||
import org.openmetadata.service.Entity;
|
||||
import org.openmetadata.service.jdbi3.ContextMemoryRepository;
|
||||
|
||||
@ExtendWith(TestNamespaceExtension.class)
|
||||
public class KnowledgeCenterIT {
|
||||
|
||||
private static final String KC_PATH = "v1/contextCenter/pages";
|
||||
|
||||
private Page createPage(RestClient rest, CreatePage request) throws HttpResponseException {
|
||||
return rest.create(KC_PATH, request, Page.class);
|
||||
}
|
||||
|
||||
private Page getPage(RestClient rest, UUID id, String fields) throws HttpResponseException {
|
||||
return rest.getById(KC_PATH, id, fields, Page.class);
|
||||
}
|
||||
|
||||
private Page patchPage(RestClient rest, UUID id, String originalJson, Page updated)
|
||||
throws HttpResponseException {
|
||||
return rest.patch(KC_PATH, id, originalJson, updated, Page.class);
|
||||
}
|
||||
|
||||
private CreatePage buildCreateRequest(String name, EntityReference relatedEntity) {
|
||||
return new CreatePage()
|
||||
.withName(name)
|
||||
.withPageType(PageType.ARTICLE)
|
||||
.withDescription("This is a test Description.")
|
||||
.withPage(new Article())
|
||||
.withRelatedEntities(List.of(relatedEntity));
|
||||
}
|
||||
|
||||
private EntityReference getOrganizationRef() {
|
||||
OpenMetadataClient adminClient = SdkClients.adminClient();
|
||||
TeamService teamService = new TeamService(adminClient.getHttpClient());
|
||||
Team org = teamService.getByName("Organization", null);
|
||||
return org.getEntityReference();
|
||||
}
|
||||
|
||||
/**
|
||||
* processingStatus, processingError and extractionStats are machine-managed fields the extraction
|
||||
* engine stamps. They must round-trip through Postgres and the REST layer, and — recorded with
|
||||
* updateVersion=false — must NOT churn the article's version history when stamped.
|
||||
*/
|
||||
@Test
|
||||
void articleProcessingStatusRoundTripsWithoutVersionBump(TestNamespace ns)
|
||||
throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
EntityReference orgRef = getOrganizationRef();
|
||||
|
||||
Page created = createPage(rest, buildCreateRequest(ns.prefix("pageProcessingStatus"), orgRef));
|
||||
Double versionBeforeStamp = created.getVersion();
|
||||
|
||||
Page updated = JsonUtils.deepCopy(created, Page.class);
|
||||
updated.setProcessingStatus(PageProcessingStatus.Failed);
|
||||
updated.setProcessingError("extraction failed: provider timeout");
|
||||
updated.setExtractionStats(new ExtractionStats().withChunksTotal(1).withPillsCreated(0));
|
||||
Page patched = patchPage(rest, created.getId(), JsonUtils.pojoToJson(created), updated);
|
||||
|
||||
assertEquals(PageProcessingStatus.Failed, patched.getProcessingStatus());
|
||||
assertEquals("extraction failed: provider timeout", patched.getProcessingError());
|
||||
assertNotNull(patched.getExtractionStats());
|
||||
assertEquals(
|
||||
versionBeforeStamp,
|
||||
patched.getVersion(),
|
||||
"machine-managed status fields must not churn the article version");
|
||||
|
||||
Page fetched = getPage(rest, created.getId(), "");
|
||||
assertEquals(PageProcessingStatus.Failed, fetched.getProcessingStatus());
|
||||
assertEquals("extraction failed: provider timeout", fetched.getProcessingError());
|
||||
}
|
||||
|
||||
/**
|
||||
* memoryCount is computed from the {@code page --MENTIONED_IN--> contextMemory} relationship a
|
||||
* memory creates from its sourceEntity. Linking a memory to a page via sourceEntity must make the
|
||||
* page's memoryCount reflect it (and the relationship-backed sourceEntityId listing return it),
|
||||
* proving the edge is stored with the page's entity type so the typed {@code findTo} count matches.
|
||||
*/
|
||||
@Test
|
||||
void articleMemoryCountReflectsLinkedMemories(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
EntityReference orgRef = getOrganizationRef();
|
||||
|
||||
Page page = createPage(rest, buildCreateRequest(ns.prefix("pageMemoryCount"), orgRef));
|
||||
|
||||
CreateContextMemory memoryRequest =
|
||||
new CreateContextMemory()
|
||||
.withName(ns.prefix("memCountPill"))
|
||||
.withQuestion("What does the engagement-weighted CLV metric measure?")
|
||||
.withAnswer("Customer lifetime value adjusted by an engagement-tier multiplier.")
|
||||
.withSourceEntity(new EntityReference().withId(page.getId()).withType("page"));
|
||||
rest.create("v1/contextCenter/memories", memoryRequest, ContextMemory.class);
|
||||
|
||||
Page withCount = getPage(rest, page.getId(), "memoryCount");
|
||||
assertEquals(
|
||||
Integer.valueOf(1),
|
||||
withCount.getMemoryCount(),
|
||||
"memoryCount must reflect the MENTIONED_IN-linked memory");
|
||||
}
|
||||
|
||||
/**
|
||||
* The Memory Agent stamps each memory's memoryStats after deriving, loading it via {@code
|
||||
* getFields("")} (no relationship fields). That machine update must NOT delete the memory's
|
||||
* sourceEntity MENTIONED_IN edge — a regression that orphaned the pill from its page, zeroing
|
||||
* memoryCount and the article's derived ontologies. Reproduces the exact stamp path in-process.
|
||||
*/
|
||||
@Test
|
||||
void ontologyStampPreservesMemorySourceLink(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
EntityReference orgRef = getOrganizationRef();
|
||||
|
||||
Page page = createPage(rest, buildCreateRequest(ns.prefix("pageOntologyStamp"), orgRef));
|
||||
|
||||
CreateContextMemory memoryRequest =
|
||||
new CreateContextMemory()
|
||||
.withName(ns.prefix("ontologyStampPill"))
|
||||
.withQuestion("What engagement multiplier does the CLV metric apply?")
|
||||
.withAnswer("1.2x engaged, 0.8x occasional, 0.5x dormant.")
|
||||
.withSourceEntity(new EntityReference().withId(page.getId()).withType("page"));
|
||||
ContextMemory memory =
|
||||
rest.create("v1/contextCenter/memories", memoryRequest, ContextMemory.class);
|
||||
|
||||
assertEquals(
|
||||
Integer.valueOf(1),
|
||||
getPage(rest, page.getId(), "memoryCount").getMemoryCount(),
|
||||
"precondition: the memory is linked to its source page");
|
||||
|
||||
ContextMemoryRepository memoryRepo =
|
||||
(ContextMemoryRepository) Entity.getEntityRepository(Entity.CONTEXT_MEMORY);
|
||||
ContextMemory partial = memoryRepo.get(null, memory.getId(), memoryRepo.getFields(""));
|
||||
memoryRepo.stampMemoryStats(
|
||||
partial, new MemoryStats().withSourceHash("test-hash").withLastRunAt(1L));
|
||||
|
||||
assertEquals(
|
||||
Integer.valueOf(1),
|
||||
getPage(rest, page.getId(), "memoryCount").getMemoryCount(),
|
||||
"ontology stamp must preserve the memory's source link (MENTIONED_IN edge)");
|
||||
}
|
||||
|
||||
@Test
|
||||
void testRelatedEntitiesExcludesDomainsAndDataProducts(TestNamespace ns)
|
||||
throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
OpenMetadataClient adminClient = SdkClients.adminClient();
|
||||
DomainService domainSvc = new DomainService(adminClient.getHttpClient());
|
||||
|
||||
EntityReference orgRef = getOrganizationRef();
|
||||
CreatePage createPageReq = buildCreateRequest(ns.prefix("pageExcludesDomains"), orgRef);
|
||||
Page page = createPage(rest, createPageReq);
|
||||
|
||||
CreateDomain createDomain =
|
||||
new CreateDomain()
|
||||
.withName(ns.prefix("testDomain"))
|
||||
.withDomainType(CreateDomain.DomainType.AGGREGATE)
|
||||
.withDescription("Test domain");
|
||||
Domain domain = domainSvc.create(createDomain);
|
||||
|
||||
String original = JsonUtils.pojoToJson(page);
|
||||
page.withDomains(List.of(domain.getEntityReference()));
|
||||
page = patchPage(rest, page.getId(), original, page);
|
||||
|
||||
Page fetchedPage = getPage(rest, page.getId(), "relatedEntities,domains,dataProducts");
|
||||
|
||||
assertEquals(1, fetchedPage.getDomains().size());
|
||||
assertEquals(domain.getName(), fetchedPage.getDomains().get(0).getName());
|
||||
|
||||
boolean domainInRelatedEntities =
|
||||
fetchedPage.getRelatedEntities().stream().anyMatch(ref -> "domain".equals(ref.getType()));
|
||||
assertEquals(false, domainInRelatedEntities, "Domains should not appear in relatedEntities");
|
||||
|
||||
boolean dataProductInRelatedEntities =
|
||||
fetchedPage.getRelatedEntities().stream()
|
||||
.anyMatch(ref -> "dataProduct".equals(ref.getType()));
|
||||
assertEquals(
|
||||
false, dataProductInRelatedEntities, "DataProducts should not appear in relatedEntities");
|
||||
}
|
||||
|
||||
@Test
|
||||
void testDomainAddUpdateRemove(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
OpenMetadataClient adminClient = SdkClients.adminClient();
|
||||
DomainService domainSvc = new DomainService(adminClient.getHttpClient());
|
||||
|
||||
EntityReference orgRef = getOrganizationRef();
|
||||
CreatePage createPageReq = buildCreateRequest(ns.prefix("pageDomainCrud"), orgRef);
|
||||
Page page = createPage(rest, createPageReq);
|
||||
|
||||
CreateDomain createDomain1 =
|
||||
new CreateDomain()
|
||||
.withName(ns.prefix("testDomain1"))
|
||||
.withDomainType(CreateDomain.DomainType.AGGREGATE)
|
||||
.withDescription("Test domain 1");
|
||||
Domain domain1 = domainSvc.create(createDomain1);
|
||||
|
||||
CreateDomain createDomain2 =
|
||||
new CreateDomain()
|
||||
.withName(ns.prefix("testDomain2"))
|
||||
.withDomainType(CreateDomain.DomainType.AGGREGATE)
|
||||
.withDescription("Test domain 2");
|
||||
Domain domain2 = domainSvc.create(createDomain2);
|
||||
|
||||
String original = JsonUtils.pojoToJson(page);
|
||||
page.withDomains(List.of(domain1.getEntityReference()));
|
||||
page = patchPage(rest, page.getId(), original, page);
|
||||
|
||||
Page fetchedPage = getPage(rest, page.getId(), "domains,relatedEntities");
|
||||
assertEquals(1, fetchedPage.getDomains().size());
|
||||
assertEquals(domain1.getName(), fetchedPage.getDomains().get(0).getName());
|
||||
|
||||
original = JsonUtils.pojoToJson(page);
|
||||
page.withDomains(List.of(domain2.getEntityReference()));
|
||||
page = patchPage(rest, page.getId(), original, page);
|
||||
|
||||
fetchedPage = getPage(rest, page.getId(), "domains,relatedEntities");
|
||||
assertEquals(1, fetchedPage.getDomains().size());
|
||||
assertEquals(domain2.getName(), fetchedPage.getDomains().get(0).getName());
|
||||
|
||||
boolean domain1InDomains =
|
||||
fetchedPage.getDomains().stream().anyMatch(ref -> domain1.getName().equals(ref.getName()));
|
||||
assertEquals(false, domain1InDomains, "Old domain should be removed after update");
|
||||
|
||||
original = JsonUtils.pojoToJson(page);
|
||||
page.withDomains(null);
|
||||
page = patchPage(rest, page.getId(), original, page);
|
||||
|
||||
fetchedPage = getPage(rest, page.getId(), "domains,relatedEntities");
|
||||
int domainCount = fetchedPage.getDomains() == null ? 0 : fetchedPage.getDomains().size();
|
||||
assertEquals(0, domainCount, "Domain should be removed");
|
||||
|
||||
boolean anyDomainInRelatedEntities =
|
||||
fetchedPage.getRelatedEntities().stream().anyMatch(ref -> "domain".equals(ref.getType()));
|
||||
assertEquals(
|
||||
false, anyDomainInRelatedEntities, "No domains should ever appear in relatedEntities");
|
||||
}
|
||||
|
||||
@Test
|
||||
void testDataProductAddUpdateRemove(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
OpenMetadataClient adminClient = SdkClients.adminClient();
|
||||
DomainService domainSvc = new DomainService(adminClient.getHttpClient());
|
||||
DataProductService dpSvc = new DataProductService(adminClient.getHttpClient());
|
||||
|
||||
EntityReference orgRef = getOrganizationRef();
|
||||
CreatePage createPageReq = buildCreateRequest(ns.prefix("pageDpCrud"), orgRef);
|
||||
Page page = createPage(rest, createPageReq);
|
||||
|
||||
CreateDomain createDomain =
|
||||
new CreateDomain()
|
||||
.withName(ns.prefix("testDomainDP"))
|
||||
.withDomainType(CreateDomain.DomainType.AGGREGATE)
|
||||
.withDescription("Test domain for data products");
|
||||
Domain domain = domainSvc.create(createDomain);
|
||||
|
||||
String original = JsonUtils.pojoToJson(page);
|
||||
page.withDomains(List.of(domain.getEntityReference()));
|
||||
page = patchPage(rest, page.getId(), original, page);
|
||||
|
||||
page = getPage(rest, page.getId(), "domains,relatedEntities");
|
||||
|
||||
CreateDataProduct createDataProduct1 =
|
||||
new CreateDataProduct()
|
||||
.withName(ns.prefix("testDP1"))
|
||||
.withDomains(List.of(domain.getFullyQualifiedName()))
|
||||
.withDescription("Test data product 1");
|
||||
DataProduct dataProduct1 = dpSvc.create(createDataProduct1);
|
||||
|
||||
CreateDataProduct createDataProduct2 =
|
||||
new CreateDataProduct()
|
||||
.withName(ns.prefix("testDP2"))
|
||||
.withDomains(List.of(domain.getFullyQualifiedName()))
|
||||
.withDescription("Test data product 2");
|
||||
DataProduct dataProduct2 = dpSvc.create(createDataProduct2);
|
||||
|
||||
original = JsonUtils.pojoToJson(page);
|
||||
page.withDataProducts(List.of(dataProduct1.getEntityReference()));
|
||||
page = patchPage(rest, page.getId(), original, page);
|
||||
|
||||
Page fetchedPage = getPage(rest, page.getId(), "dataProducts,relatedEntities,domains");
|
||||
assertEquals(1, fetchedPage.getDataProducts().size());
|
||||
assertEquals(dataProduct1.getName(), fetchedPage.getDataProducts().get(0).getName());
|
||||
|
||||
original = JsonUtils.pojoToJson(page);
|
||||
page.withDataProducts(List.of(dataProduct2.getEntityReference()));
|
||||
page = patchPage(rest, page.getId(), original, page);
|
||||
|
||||
fetchedPage = getPage(rest, page.getId(), "dataProducts,relatedEntities,domains");
|
||||
assertEquals(1, fetchedPage.getDataProducts().size());
|
||||
assertEquals(dataProduct2.getName(), fetchedPage.getDataProducts().get(0).getName());
|
||||
|
||||
boolean dataProduct1InDataProducts =
|
||||
fetchedPage.getDataProducts().stream()
|
||||
.anyMatch(ref -> dataProduct1.getName().equals(ref.getName()));
|
||||
assertEquals(
|
||||
false, dataProduct1InDataProducts, "Old dataProduct should be removed after update");
|
||||
|
||||
original = JsonUtils.pojoToJson(page);
|
||||
page.withDataProducts(null);
|
||||
page = patchPage(rest, page.getId(), original, page);
|
||||
|
||||
fetchedPage = getPage(rest, page.getId(), "dataProducts,relatedEntities,domains");
|
||||
int dataProductCount =
|
||||
fetchedPage.getDataProducts() == null ? 0 : fetchedPage.getDataProducts().size();
|
||||
assertEquals(0, dataProductCount, "DataProduct should be removed");
|
||||
|
||||
boolean anyDataProductInRelatedEntities =
|
||||
fetchedPage.getRelatedEntities().stream()
|
||||
.anyMatch(ref -> "dataProduct".equals(ref.getType()));
|
||||
assertEquals(
|
||||
false,
|
||||
anyDataProductInRelatedEntities,
|
||||
"No dataProducts should ever appear in relatedEntities");
|
||||
}
|
||||
|
||||
// --- SortBy ---
|
||||
|
||||
private ResultList<Page> listPagesSorted(
|
||||
RestClient rest, String sortBy, String sortOrder, int limit) {
|
||||
String path = KC_PATH + "?sortBy=" + sortBy + "&sortOrder=" + sortOrder + "&limit=" + limit;
|
||||
try (Response response = rest.rawGet(path)) {
|
||||
assertEquals(200, response.getStatus(), "List call failed: " + response.getStatus());
|
||||
String body = response.readEntity(String.class);
|
||||
return JsonUtils.readValue(body, new TypeReference<ResultList<Page>>() {});
|
||||
}
|
||||
}
|
||||
|
||||
private static void awaitClockPast(long timestamp) {
|
||||
await()
|
||||
.pollInterval(Duration.ofMillis(2))
|
||||
.atMost(Duration.ofSeconds(2))
|
||||
.until(() -> System.currentTimeMillis() > timestamp);
|
||||
}
|
||||
|
||||
private void awaitPageIndexed(RestClient rest, UUID id) {
|
||||
await()
|
||||
.pollDelay(Duration.ZERO)
|
||||
.pollInterval(Duration.ofMillis(200))
|
||||
.atMost(Duration.ofSeconds(60))
|
||||
.untilAsserted(
|
||||
() -> {
|
||||
try (Response getResp =
|
||||
rest.rawGet("v1/search/get/knowledge_page_search_index/doc/" + id)) {
|
||||
assertEquals(
|
||||
200,
|
||||
getResp.getStatus(),
|
||||
"Page " + id + " not yet indexed: " + getResp.readEntity(String.class));
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
void testListPagesSortByUpdatedAtDesc(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
EntityReference orgRef = getOrganizationRef();
|
||||
|
||||
Page older = createPage(rest, buildCreateRequest(ns.prefix("sort-older"), orgRef));
|
||||
awaitClockPast(older.getUpdatedAt());
|
||||
Page middle = createPage(rest, buildCreateRequest(ns.prefix("sort-middle"), orgRef));
|
||||
awaitClockPast(middle.getUpdatedAt());
|
||||
Page newer = createPage(rest, buildCreateRequest(ns.prefix("sort-newer"), orgRef));
|
||||
|
||||
awaitPageIndexed(rest, older.getId());
|
||||
awaitPageIndexed(rest, middle.getId());
|
||||
awaitPageIndexed(rest, newer.getId());
|
||||
|
||||
List<UUID> ourIds = List.of(older.getId(), middle.getId(), newer.getId());
|
||||
await()
|
||||
.pollInterval(Duration.ofMillis(250))
|
||||
.atMost(Duration.ofSeconds(30))
|
||||
.untilAsserted(
|
||||
() -> {
|
||||
ResultList<Page> result = listPagesSorted(rest, "updatedAt", "desc", 1000);
|
||||
List<UUID> ordered =
|
||||
result.getData().stream().map(Page::getId).filter(ourIds::contains).toList();
|
||||
assertEquals(
|
||||
List.of(newer.getId(), middle.getId(), older.getId()),
|
||||
ordered,
|
||||
"Expected newest-first ordering for our test pages");
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
void testListPagesSortByNameAsc(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
EntityReference orgRef = getOrganizationRef();
|
||||
|
||||
Page zebra = createPage(rest, buildCreateRequest(ns.prefix("zzz-name"), orgRef));
|
||||
Page apple = createPage(rest, buildCreateRequest(ns.prefix("aaa-name"), orgRef));
|
||||
|
||||
awaitPageIndexed(rest, zebra.getId());
|
||||
awaitPageIndexed(rest, apple.getId());
|
||||
|
||||
List<UUID> ourIds = List.of(zebra.getId(), apple.getId());
|
||||
await()
|
||||
.pollInterval(Duration.ofMillis(250))
|
||||
.atMost(Duration.ofSeconds(30))
|
||||
.untilAsserted(
|
||||
() -> {
|
||||
ResultList<Page> result = listPagesSorted(rest, "name", "asc", 1000);
|
||||
List<UUID> ordered =
|
||||
result.getData().stream().map(Page::getId).filter(ourIds::contains).toList();
|
||||
assertEquals(
|
||||
List.of(apple.getId(), zebra.getId()),
|
||||
ordered,
|
||||
"Expected ascending name ordering, apple before zebra");
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
void testListPagesSortByCreatedAtAliasesUpdatedAt(TestNamespace ns) throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
EntityReference orgRef = getOrganizationRef();
|
||||
|
||||
Page first = createPage(rest, buildCreateRequest(ns.prefix("created-first"), orgRef));
|
||||
awaitClockPast(first.getUpdatedAt());
|
||||
Page second = createPage(rest, buildCreateRequest(ns.prefix("created-second"), orgRef));
|
||||
|
||||
awaitPageIndexed(rest, first.getId());
|
||||
awaitPageIndexed(rest, second.getId());
|
||||
|
||||
List<UUID> ourIds = List.of(first.getId(), second.getId());
|
||||
await()
|
||||
.pollInterval(Duration.ofMillis(250))
|
||||
.atMost(Duration.ofSeconds(30))
|
||||
.untilAsserted(
|
||||
() -> {
|
||||
ResultList<Page> result = listPagesSorted(rest, "createdAt", "desc", 1000);
|
||||
List<UUID> ordered =
|
||||
result.getData().stream().map(Page::getId).filter(ourIds::contains).toList();
|
||||
assertEquals(
|
||||
List.of(second.getId(), first.getId()),
|
||||
ordered,
|
||||
"createdAt sort should return newest first (currently aliased to updatedAt)");
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
void testListPagesSortByRejectsCursorCombo() {
|
||||
RestClient rest = RestClient.admin();
|
||||
try (Response response =
|
||||
rest.rawGet(KC_PATH + "?sortBy=updatedAt&sortOrder=desc&after=anything")) {
|
||||
assertEquals(
|
||||
400,
|
||||
response.getStatus(),
|
||||
"sortBy combined with cursor should be 400, got " + response.getStatus());
|
||||
}
|
||||
}
|
||||
|
||||
// Regression: the search index stores `followers` as a flat UUID-string list (see
|
||||
// SearchIndexUtils.parseFollowers), while the Page schema types it as
|
||||
// List<EntityReference>. Without excluding the field from `_source`, listing pages
|
||||
// through the sort-by-search path 400s on Jackson deserialization.
|
||||
@Test
|
||||
void testListPagesSortByDoesNotFailWhenPageHasFollowers(TestNamespace ns)
|
||||
throws HttpResponseException {
|
||||
RestClient rest = RestClient.admin();
|
||||
OpenMetadataClient adminClient = SdkClients.adminClient();
|
||||
UserService userSvc = new UserService(adminClient.getHttpClient());
|
||||
User admin = userSvc.getByName("admin", null);
|
||||
EntityReference orgRef = getOrganizationRef();
|
||||
|
||||
Page page = createPage(rest, buildCreateRequest(ns.prefix("followed-sort"), orgRef));
|
||||
try (Response addResp =
|
||||
rest.rawPut(KC_PATH + "/" + page.getId() + "/followers", admin.getId())) {
|
||||
assertEquals(200, addResp.getStatus(), "Adding follower failed: " + addResp.getStatus());
|
||||
}
|
||||
|
||||
awaitPageIndexed(rest, page.getId());
|
||||
|
||||
await()
|
||||
.pollInterval(Duration.ofMillis(250))
|
||||
.atMost(Duration.ofSeconds(30))
|
||||
.untilAsserted(
|
||||
() -> {
|
||||
ResultList<Page> result = listPagesSorted(rest, "updatedAt", "desc", 1000);
|
||||
boolean found =
|
||||
result.getData().stream().anyMatch(p -> page.getId().equals(p.getId()));
|
||||
assertTrue(found, "Followed page should appear in sorted list");
|
||||
});
|
||||
}
|
||||
}
|
||||
+120
@@ -0,0 +1,120 @@
|
||||
package org.openmetadata.it.search;
|
||||
|
||||
import java.lang.management.ManagementFactory;
|
||||
import java.lang.management.OperatingSystemMXBean;
|
||||
import java.time.Duration;
|
||||
import java.util.Arrays;
|
||||
import java.util.concurrent.ExecutorService;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
import java.util.concurrent.atomic.AtomicLong;
|
||||
import java.util.concurrent.atomic.AtomicReferenceArray;
|
||||
|
||||
/**
|
||||
* Samples the embedded OM JVM's process CPU load at a configurable interval and
|
||||
* exposes percentiles + max over the captured window. Used by scale tests
|
||||
* (#14 ReindexCpuHeadroomIT, #12 Scale100kEntitiesIT) to assert reindex doesn't
|
||||
* starve the Jetty request handler.
|
||||
*
|
||||
* <p>Reads {@code com.sun.management.OperatingSystemMXBean.getProcessCpuLoad()},
|
||||
* which returns CPU usage of THIS JVM normalized to the number of available CPUs
|
||||
* (0..1). For containerized scale runs that need the OM container's CPU rather
|
||||
* than the test JVM's, swap the impl for the testcontainer's stats stream.
|
||||
*/
|
||||
public final class CpuSampler {
|
||||
|
||||
private static final int DEFAULT_CAPACITY = 1024;
|
||||
|
||||
private final OperatingSystemMXBean osBean;
|
||||
private final Duration interval;
|
||||
private final AtomicReferenceArray<Double> samples;
|
||||
private final AtomicInteger index = new AtomicInteger();
|
||||
private final AtomicLong startMillis = new AtomicLong();
|
||||
private volatile ExecutorService executor;
|
||||
private volatile boolean running;
|
||||
|
||||
public CpuSampler() {
|
||||
this(Duration.ofSeconds(1));
|
||||
}
|
||||
|
||||
public CpuSampler(final Duration interval) {
|
||||
this.osBean = ManagementFactory.getOperatingSystemMXBean();
|
||||
this.interval = interval;
|
||||
this.samples = new AtomicReferenceArray<>(DEFAULT_CAPACITY);
|
||||
}
|
||||
|
||||
public void start() {
|
||||
if (running) {
|
||||
return;
|
||||
}
|
||||
running = true;
|
||||
startMillis.set(System.currentTimeMillis());
|
||||
executor =
|
||||
Executors.newSingleThreadExecutor(
|
||||
r -> {
|
||||
final Thread t = new Thread(r, "cpu-sampler");
|
||||
t.setDaemon(true);
|
||||
return t;
|
||||
});
|
||||
executor.submit(this::sampleLoop);
|
||||
}
|
||||
|
||||
public void stop() {
|
||||
running = false;
|
||||
if (executor != null) {
|
||||
executor.shutdownNow();
|
||||
executor = null;
|
||||
}
|
||||
}
|
||||
|
||||
public Stats stats() {
|
||||
final int collected = Math.min(index.get(), samples.length());
|
||||
final double[] copy = new double[collected];
|
||||
for (int i = 0; i < collected; i++) {
|
||||
final Double sample = samples.get(i);
|
||||
copy[i] = sample == null ? 0.0 : sample;
|
||||
}
|
||||
Arrays.sort(copy);
|
||||
if (copy.length == 0) {
|
||||
return new Stats(0, 0, 0, 0, 0);
|
||||
}
|
||||
final double sum = Arrays.stream(copy).sum();
|
||||
return new Stats(
|
||||
copy.length, copy[0], copy[copy.length - 1], sum / copy.length, percentile(copy, 0.95));
|
||||
}
|
||||
|
||||
private void sampleLoop() {
|
||||
while (running) {
|
||||
try {
|
||||
final double load = readCpuLoad();
|
||||
final int slot = index.getAndIncrement() % samples.length();
|
||||
samples.set(slot, load);
|
||||
Thread.sleep(interval.toMillis());
|
||||
} catch (final InterruptedException e) {
|
||||
Thread.currentThread().interrupt();
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private double readCpuLoad() {
|
||||
if (osBean instanceof com.sun.management.OperatingSystemMXBean sunBean) {
|
||||
return Math.max(0.0, sunBean.getProcessCpuLoad());
|
||||
}
|
||||
return Math.max(0.0, osBean.getSystemLoadAverage())
|
||||
/ Math.max(1, osBean.getAvailableProcessors());
|
||||
}
|
||||
|
||||
private static double percentile(final double[] sorted, final double p) {
|
||||
if (sorted.length == 0) {
|
||||
return 0.0;
|
||||
}
|
||||
// Index off (length - 1), not length: floor(p * length) returns the max for many small
|
||||
// n (e.g. n=20, p=0.95 -> idx 19), overstating p95 and making headroom asserts flaky.
|
||||
final int idx = (int) Math.min(sorted.length - 1L, Math.round(p * (sorted.length - 1)));
|
||||
return sorted[idx];
|
||||
}
|
||||
|
||||
/** Immutable snapshot of CPU usage over the sample window. Values are 0..1. */
|
||||
public record Stats(int samples, double min, double max, double avg, double p95) {}
|
||||
}
|
||||
+105
@@ -0,0 +1,105 @@
|
||||
package org.openmetadata.it.search;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import java.util.Map;
|
||||
import org.openmetadata.it.server.ServerHandle;
|
||||
import org.openmetadata.sdk.network.HttpMethod;
|
||||
|
||||
/**
|
||||
* Counts entities in the application database via REST list endpoints with
|
||||
* {@code limit=0}. The {@code paging.total} field on a {@code ResultList}
|
||||
* response is the authoritative DB-side count for an entity type and is the
|
||||
* reference value for the reindex {@code db_count == es_count} invariant.
|
||||
*
|
||||
* <p>Mapping entity type → REST collection path is deliberately explicit so
|
||||
* that adding new types is an obvious code change. Aliases declared in
|
||||
* {@code indexMapping.json} that don't have a list endpoint (system/derived
|
||||
* types like {@code dataAsset}, {@code all}) are skipped by the caller.
|
||||
*/
|
||||
public final class DbCountQuerier {
|
||||
|
||||
private static final ObjectMapper MAPPER = new ObjectMapper();
|
||||
|
||||
/**
|
||||
* Entity type → REST collection path. Covers user-creatable assets that the
|
||||
* default reindex run handles. Add entries when new entity types come under
|
||||
* test.
|
||||
*/
|
||||
private static final Map<String, String> COLLECTION_PATHS =
|
||||
Map.ofEntries(
|
||||
Map.entry("table", "/v1/tables"),
|
||||
Map.entry("databaseSchema", "/v1/databaseSchemas"),
|
||||
Map.entry("database", "/v1/databases"),
|
||||
Map.entry("databaseService", "/v1/services/databaseServices"),
|
||||
Map.entry("topic", "/v1/topics"),
|
||||
Map.entry("messagingService", "/v1/services/messagingServices"),
|
||||
Map.entry("dashboard", "/v1/dashboards"),
|
||||
Map.entry("dashboardService", "/v1/services/dashboardServices"),
|
||||
Map.entry("pipeline", "/v1/pipelines"),
|
||||
Map.entry("pipelineService", "/v1/services/pipelineServices"),
|
||||
Map.entry("mlmodel", "/v1/mlmodels"),
|
||||
Map.entry("mlmodelService", "/v1/services/mlmodelServices"),
|
||||
Map.entry("container", "/v1/containers"),
|
||||
Map.entry("storageService", "/v1/services/storageServices"),
|
||||
Map.entry("searchIndex", "/v1/searchIndexes"),
|
||||
Map.entry("searchService", "/v1/services/searchServices"),
|
||||
Map.entry("apiCollection", "/v1/apiCollections"),
|
||||
Map.entry("apiEndpoint", "/v1/apiEndpoints"),
|
||||
Map.entry("apiService", "/v1/services/apiServices"),
|
||||
Map.entry("glossary", "/v1/glossaries"),
|
||||
Map.entry("glossaryTerm", "/v1/glossaryTerms"),
|
||||
Map.entry("tag", "/v1/tags"),
|
||||
Map.entry("classification", "/v1/classifications"),
|
||||
Map.entry("user", "/v1/users"),
|
||||
Map.entry("team", "/v1/teams"),
|
||||
Map.entry("domain", "/v1/domains"),
|
||||
Map.entry("dataProduct", "/v1/dataProducts"),
|
||||
Map.entry("query", "/v1/queries"),
|
||||
Map.entry("testCase", "/v1/dataQuality/testCases"),
|
||||
Map.entry("testSuite", "/v1/dataQuality/testSuites"));
|
||||
|
||||
private final ServerHandle server;
|
||||
|
||||
public DbCountQuerier(final ServerHandle server) {
|
||||
this.server = server;
|
||||
}
|
||||
|
||||
/** Whether this querier knows how to count the given entity type. */
|
||||
public boolean canCount(final String entityType) {
|
||||
return COLLECTION_PATHS.containsKey(entityType);
|
||||
}
|
||||
|
||||
/**
|
||||
* DB count for the entity type. Includes soft-deleted entities iff {@code
|
||||
* includeDeleted} is true.
|
||||
*/
|
||||
public long count(final String entityType, final boolean includeDeleted) {
|
||||
final String path = COLLECTION_PATHS.get(entityType);
|
||||
if (path == null) {
|
||||
throw new IllegalArgumentException(
|
||||
"No collection path mapped for entity type: " + entityType);
|
||||
}
|
||||
final String url = path + "?limit=0" + (includeDeleted ? "&include=all" : "");
|
||||
// The list endpoints return a JSON object ({paging, data}); the SDK deserializes the response
|
||||
// into the requested type, so request a Map (String.class makes Jackson fail with
|
||||
// "Cannot deserialize String from Object").
|
||||
final Map<?, ?> response =
|
||||
server.sdk().getHttpClient().execute(HttpMethod.GET, url, null, Map.class);
|
||||
return parsePagingTotal(response);
|
||||
}
|
||||
|
||||
/** Convenience: count of non-deleted entities. */
|
||||
public long count(final String entityType) {
|
||||
return count(entityType, false);
|
||||
}
|
||||
|
||||
private long parsePagingTotal(final Map<?, ?> response) {
|
||||
try {
|
||||
final JsonNode node = MAPPER.valueToTree(response);
|
||||
return node.path("paging").path("total").asLong();
|
||||
} catch (final Exception e) {
|
||||
throw new IllegalStateException("Failed to parse paging.total from list response", e);
|
||||
}
|
||||
}
|
||||
}
|
||||
+68
@@ -0,0 +1,68 @@
|
||||
package org.openmetadata.it.search;
|
||||
|
||||
import com.github.dockerjava.api.DockerClient;
|
||||
import java.time.Duration;
|
||||
import org.openmetadata.it.bootstrap.TestSuiteBootstrap;
|
||||
import org.testcontainers.DockerClientFactory;
|
||||
import org.testcontainers.containers.GenericContainer;
|
||||
|
||||
/**
|
||||
* Toggles the embedded search engine container's runtime state to validate
|
||||
* retry / failure semantics in the indexing pipeline.
|
||||
*
|
||||
* <p>Pause uses Docker's {@code pauseContainer} (SIGSTOP-equivalent), which freezes
|
||||
* the engine without tearing down its TCP connections — the OM live-index path sees
|
||||
* timeouts rather than connection refused. Unpause resumes the engine in-place; no
|
||||
* state is lost.
|
||||
*
|
||||
* <p>Only meaningful for embedded backend ITs (the bootstrap exposes the testcontainer);
|
||||
* UIIT scenarios that boot the OM Docker image use a separate ContainerizedServer
|
||||
* helper not covered here.
|
||||
*/
|
||||
public final class EsOutageInjector {
|
||||
|
||||
private EsOutageInjector() {}
|
||||
|
||||
/**
|
||||
* Pauses the search engine for {@code duration}, then unpauses on a daemon thread.
|
||||
* Returns immediately so the caller can run the workload that should observe the
|
||||
* outage in parallel.
|
||||
*/
|
||||
public static void pauseFor(final Duration duration) {
|
||||
final String containerId = requireContainerId();
|
||||
final DockerClient docker = DockerClientFactory.lazyClient();
|
||||
docker.pauseContainerCmd(containerId).exec();
|
||||
final Thread resumer =
|
||||
new Thread(
|
||||
() -> {
|
||||
try {
|
||||
Thread.sleep(duration.toMillis());
|
||||
} catch (final InterruptedException e) {
|
||||
Thread.currentThread().interrupt();
|
||||
} finally {
|
||||
try {
|
||||
docker.unpauseContainerCmd(containerId).exec();
|
||||
} catch (final RuntimeException ignored) {
|
||||
// container already unpaused — best-effort cleanup
|
||||
}
|
||||
}
|
||||
},
|
||||
"es-outage-resumer");
|
||||
resumer.setDaemon(true);
|
||||
resumer.start();
|
||||
}
|
||||
|
||||
/** Synchronous unpause for tests that want to control the resume themselves. */
|
||||
public static void unpause() {
|
||||
DockerClientFactory.lazyClient().unpauseContainerCmd(requireContainerId()).exec();
|
||||
}
|
||||
|
||||
private static String requireContainerId() {
|
||||
final GenericContainer<?> container = TestSuiteBootstrap.getSearchContainer();
|
||||
if (container == null || container.getContainerId() == null) {
|
||||
throw new IllegalStateException(
|
||||
"Search container is not running — EsOutageInjector requires the embedded bootstrap");
|
||||
}
|
||||
return container.getContainerId();
|
||||
}
|
||||
}
|
||||
+150
@@ -0,0 +1,150 @@
|
||||
package org.openmetadata.it.search;
|
||||
|
||||
import java.net.URI;
|
||||
import java.net.http.HttpClient;
|
||||
import java.net.http.HttpRequest;
|
||||
import java.net.http.HttpResponse;
|
||||
import java.time.Duration;
|
||||
import java.util.Arrays;
|
||||
import java.util.concurrent.ExecutorService;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
import java.util.concurrent.atomic.AtomicLongArray;
|
||||
import org.openmetadata.it.server.ServerHandle;
|
||||
|
||||
/**
|
||||
* Hammers an OM endpoint (default {@code /api/v1/system/version}) at a fixed QPS and
|
||||
* records success rate + latency histogram. Used by #14 ReindexCpuHeadroomIT to prove
|
||||
* the Jetty request handler stays responsive while reindex is hot.
|
||||
*
|
||||
* <p>k8s liveness probes have ~1 s budget; if any probe in the window exceeds it the
|
||||
* pod gets restarted in prod. The test asserts {@link Stats#latencyP99Ms} stays under
|
||||
* a configured ceiling and {@link Stats#successRatio} stays at 1.0.
|
||||
*/
|
||||
public final class HealthProbe {
|
||||
|
||||
private static final int DEFAULT_CAPACITY = 16_384;
|
||||
|
||||
private final HttpClient http;
|
||||
private final URI url;
|
||||
private final int qps;
|
||||
private final AtomicLongArray latenciesMicros;
|
||||
private final AtomicInteger writeIndex = new AtomicInteger();
|
||||
private final AtomicInteger successes = new AtomicInteger();
|
||||
private final AtomicInteger failures = new AtomicInteger();
|
||||
private volatile ExecutorService executor;
|
||||
private volatile boolean running;
|
||||
|
||||
public HealthProbe(final ServerHandle server, final int qps) {
|
||||
this(server, qps, "/v1/system/version");
|
||||
}
|
||||
|
||||
public HealthProbe(final ServerHandle server, final int qps, final String path) {
|
||||
this.qps = qps;
|
||||
this.http = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(2)).build();
|
||||
final String base = server.baseUrl().toString();
|
||||
this.url = URI.create(base.replaceAll("/api$", "") + "/api" + path);
|
||||
this.latenciesMicros = new AtomicLongArray(DEFAULT_CAPACITY);
|
||||
}
|
||||
|
||||
public void start() {
|
||||
if (running) {
|
||||
return;
|
||||
}
|
||||
running = true;
|
||||
executor =
|
||||
Executors.newFixedThreadPool(
|
||||
Math.max(2, qps / 2),
|
||||
r -> {
|
||||
final Thread t = new Thread(r, "health-probe");
|
||||
t.setDaemon(true);
|
||||
return t;
|
||||
});
|
||||
final long intervalNanos = 1_000_000_000L / Math.max(1, qps);
|
||||
executor.submit(() -> probeLoop(intervalNanos));
|
||||
}
|
||||
|
||||
public void stop() {
|
||||
running = false;
|
||||
if (executor != null) {
|
||||
executor.shutdownNow();
|
||||
executor = null;
|
||||
}
|
||||
}
|
||||
|
||||
public Stats stats() {
|
||||
final int collected = Math.min(writeIndex.get(), latenciesMicros.length());
|
||||
final long[] copy = new long[collected];
|
||||
for (int i = 0; i < collected; i++) {
|
||||
copy[i] = latenciesMicros.get(i);
|
||||
}
|
||||
Arrays.sort(copy);
|
||||
final long total = (long) successes.get() + failures.get();
|
||||
final double ratio = total == 0 ? 1.0 : (double) successes.get() / total;
|
||||
return new Stats(
|
||||
successes.get(),
|
||||
failures.get(),
|
||||
ratio,
|
||||
copy.length == 0 ? 0 : copy[copy.length - 1] / 1000.0,
|
||||
copy.length == 0 ? 0 : percentile(copy, 0.99) / 1000.0,
|
||||
copy.length == 0 ? 0 : percentile(copy, 0.50) / 1000.0);
|
||||
}
|
||||
|
||||
private void probeLoop(final long intervalNanos) {
|
||||
long next = System.nanoTime();
|
||||
while (running) {
|
||||
executor.submit(this::issueOne);
|
||||
next += intervalNanos;
|
||||
final long sleep = next - System.nanoTime();
|
||||
if (sleep > 0) {
|
||||
try {
|
||||
Thread.sleep(sleep / 1_000_000L, (int) (sleep % 1_000_000L));
|
||||
} catch (final InterruptedException e) {
|
||||
Thread.currentThread().interrupt();
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private void issueOne() {
|
||||
final long start = System.nanoTime();
|
||||
try {
|
||||
final HttpResponse<Void> resp =
|
||||
http.send(
|
||||
HttpRequest.newBuilder(url).timeout(Duration.ofSeconds(5)).GET().build(),
|
||||
HttpResponse.BodyHandlers.discarding());
|
||||
if (resp.statusCode() >= 200 && resp.statusCode() < 300) {
|
||||
recordSuccess(start);
|
||||
} else {
|
||||
failures.incrementAndGet();
|
||||
}
|
||||
} catch (final Exception e) {
|
||||
failures.incrementAndGet();
|
||||
}
|
||||
}
|
||||
|
||||
private void recordSuccess(final long startNanos) {
|
||||
successes.incrementAndGet();
|
||||
final long elapsedMicros = (System.nanoTime() - startNanos) / 1000L;
|
||||
final int slot = writeIndex.getAndIncrement() % latenciesMicros.length();
|
||||
latenciesMicros.set(slot, elapsedMicros);
|
||||
}
|
||||
|
||||
private static long percentile(final long[] sorted, final double p) {
|
||||
if (sorted.length == 0) {
|
||||
return 0L;
|
||||
}
|
||||
final int idx = (int) Math.min(sorted.length - 1L, Math.floor(p * sorted.length));
|
||||
return sorted[idx];
|
||||
}
|
||||
|
||||
/** Immutable snapshot of probe results. Latencies are milliseconds. */
|
||||
public record Stats(
|
||||
int successes,
|
||||
int failures,
|
||||
double successRatio,
|
||||
double latencyMaxMs,
|
||||
double latencyP99Ms,
|
||||
double latencyP50Ms) {}
|
||||
}
|
||||
+198
@@ -0,0 +1,198 @@
|
||||
package org.openmetadata.it.search;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import java.io.IOException;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Iterator;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import org.openmetadata.it.server.ServerHandle;
|
||||
import org.openmetadata.sdk.network.HttpMethod;
|
||||
import org.openmetadata.search.IndexMapping;
|
||||
import org.openmetadata.search.IndexMappingLoader;
|
||||
import org.openmetadata.service.Entity;
|
||||
import org.openmetadata.service.search.SearchRepository;
|
||||
|
||||
/**
|
||||
* Read-only view over the live search engine for reindex assertions:
|
||||
* resolves declared aliases (merged OM + Collate), looks up the backing
|
||||
* index for each alias, fetches the live mapping JSON, and counts mapping
|
||||
* fields for field-explosion checks.
|
||||
*
|
||||
* <p>Backed by {@link IndexMappingLoader} for the canonical list of aliases
|
||||
* and by {@link SearchClient} for live engine state. Works against both
|
||||
* Elasticsearch and OpenSearch.
|
||||
*/
|
||||
public final class IndexAliasInspector {
|
||||
|
||||
private final SearchClient client;
|
||||
private final String clusterAlias;
|
||||
|
||||
public IndexAliasInspector(final ServerHandle server) {
|
||||
this.client = new SearchClient(server);
|
||||
ensureMappingLoaderInitialized();
|
||||
this.clusterAlias = resolveClusterAlias(server);
|
||||
}
|
||||
|
||||
/**
|
||||
* Entity types declared in the merged indexMapping.json. The Collate test
|
||||
* suite picks up Collate-only entries because Collate's indexMapping.json
|
||||
* is on the same classpath.
|
||||
*/
|
||||
public Set<String> declaredEntityTypes() {
|
||||
return IndexMappingLoader.getInstance().getIndexMapping().keySet();
|
||||
}
|
||||
|
||||
/**
|
||||
* Embedded backend ITs already init the loader via TestSuiteBootstrap. UIITs run in
|
||||
* a separate JVM from the OM service, so the test JVM's loader is uninitialized —
|
||||
* lazily initialize it here so callers don't have to care which boot mode they're in.
|
||||
*/
|
||||
private static void ensureMappingLoaderInitialized() {
|
||||
try {
|
||||
IndexMappingLoader.getInstance();
|
||||
} catch (final IllegalStateException uninitialized) {
|
||||
try {
|
||||
IndexMappingLoader.init();
|
||||
} catch (final java.io.IOException ioe) {
|
||||
throw new IllegalStateException("Failed to lazily init IndexMappingLoader", ioe);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Alias name for the given entity type, cluster-alias-aware. When the server runs with a cluster
|
||||
* alias (the test stacks use {@code "openmetadata"}), the live read alias is prefixed
|
||||
* ({@code "table" -> "openmetadata_table_search_index"}); without one it's the bare
|
||||
* {@code "table_search_index"}. Resolving via {@link SearchRepository#getClusterAlias()} (the same
|
||||
* value the server names indices with) keeps these assertions querying the alias that actually
|
||||
* exists in the engine instead of a name that 404s under a cluster alias.
|
||||
*/
|
||||
public String aliasFor(final String entityType) {
|
||||
return mappingFor(entityType).getAlias(clusterAlias);
|
||||
}
|
||||
|
||||
/**
|
||||
* The entity's own canonical index name, cluster-alias-aware ({@code table} ->
|
||||
* {@code openmetadata_table_search_index}). Unlike {@link #aliasFor}, this resolves to the 1:1
|
||||
* read alias the server attaches to each entity's single backing index — not the short grouping
|
||||
* alias ({@code openmetadata_table}) that also spans child/sibling indices (e.g. columns under
|
||||
* {@code table}, time-series under {@code testCase}). Use this whenever an assertion must target
|
||||
* exactly one entity type's index rather than an alias that fans out across several.
|
||||
*/
|
||||
public String indexNameFor(final String entityType) {
|
||||
return mappingFor(entityType).getIndexName(clusterAlias);
|
||||
}
|
||||
|
||||
private static IndexMapping mappingFor(final String entityType) {
|
||||
final IndexMapping mapping = IndexMappingLoader.getInstance().getIndexMapping().get(entityType);
|
||||
if (mapping == null) {
|
||||
throw new IllegalArgumentException(
|
||||
"No index mapping declared for entity type: " + entityType);
|
||||
}
|
||||
return mapping;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves the server's configured cluster alias (the index-name prefix). Embedded mode reads it
|
||||
* from the in-JVM {@link SearchRepository}. External mode (where the SearchRepository isn't
|
||||
* initialized in the test JVM) fetches it from the server's {@code /v1/test-support/search}
|
||||
* endpoints, so assertions query the alias that actually exists on the remote cluster rather
|
||||
* than a bare name that would 404 under a cluster alias.
|
||||
*/
|
||||
private static String resolveClusterAlias(final ServerHandle server) {
|
||||
final String result;
|
||||
if (server.isExternal()) {
|
||||
result = fetchClusterAliasFromServer(server);
|
||||
} else {
|
||||
final SearchRepository searchRepository = Entity.getSearchRepository();
|
||||
result = searchRepository != null ? searchRepository.getClusterAlias() : null;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
private static String fetchClusterAliasFromServer(final ServerHandle server) {
|
||||
final String body =
|
||||
server
|
||||
.sdk()
|
||||
.getHttpClient()
|
||||
.executeForString(HttpMethod.GET, "/v1/test-support/search/cluster-alias", null);
|
||||
try {
|
||||
return new ObjectMapper().readTree(body).path("clusterAlias").asText("");
|
||||
} catch (final IOException e) {
|
||||
throw new IllegalStateException("Failed to resolve cluster alias from server: " + body, e);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Live 1:1 alias -> backing index map (alphabetical for stable diffs). Shared grouping aliases
|
||||
* that resolve to multiple indices (e.g. {@code testSuite} spans test_case / test_suite /
|
||||
* test_case_result) are skipped: they aren't a single entity type's primary read alias, so they
|
||||
* don't belong in a 1:1 alias→index snapshot used for swap assertions.
|
||||
*/
|
||||
public Map<String, String> aliasToIndex() {
|
||||
final Map<String, String> result = new LinkedHashMap<>();
|
||||
for (final String entityType : declaredEntityTypes()) {
|
||||
final String alias = aliasFor(entityType);
|
||||
final List<String> indices = indicesForAlias(alias);
|
||||
if (indices.size() == 1) {
|
||||
result.put(alias, indices.get(0));
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
/** List of backing indices for an alias (typically one; empty if the alias does not exist). */
|
||||
public List<String> indicesForAlias(final String alias) {
|
||||
final List<String> indices = new ArrayList<>();
|
||||
if (!client.aliasExists(alias)) {
|
||||
return indices;
|
||||
}
|
||||
final JsonNode body = client.alias(alias);
|
||||
final Iterator<Map.Entry<String, JsonNode>> fields = body.fields();
|
||||
while (fields.hasNext()) {
|
||||
indices.add(fields.next().getKey());
|
||||
}
|
||||
return indices;
|
||||
}
|
||||
|
||||
/** Live mapping JSON for the alias (the {@code properties} node). */
|
||||
public JsonNode mapping(final String alias) {
|
||||
final JsonNode body = client.mapping(alias);
|
||||
final Iterator<Map.Entry<String, JsonNode>> fields = body.fields();
|
||||
if (!fields.hasNext()) {
|
||||
throw new IllegalStateException("No mapping returned for alias: " + alias);
|
||||
}
|
||||
return fields.next().getValue().path("mappings");
|
||||
}
|
||||
|
||||
/**
|
||||
* Count of leaf fields in the alias mapping — flattens nested objects so a deeply nested
|
||||
* doc shape still produces a comparable single number. Used to detect mapping explosions.
|
||||
*/
|
||||
public long fieldCount(final String alias) {
|
||||
return countLeaves(mapping(alias).path("properties"), 0);
|
||||
}
|
||||
|
||||
private long countLeaves(final JsonNode properties, final int depth) {
|
||||
if (properties == null || properties.isMissingNode() || !properties.isObject()) {
|
||||
return 0;
|
||||
}
|
||||
long total = 0;
|
||||
final Iterator<Map.Entry<String, JsonNode>> entries = properties.fields();
|
||||
while (entries.hasNext()) {
|
||||
final Map.Entry<String, JsonNode> entry = entries.next();
|
||||
final JsonNode value = entry.getValue();
|
||||
final JsonNode nested = value.path("properties");
|
||||
if (nested.isObject()) {
|
||||
total += countLeaves(nested, depth + 1);
|
||||
} else {
|
||||
total++;
|
||||
}
|
||||
}
|
||||
return total;
|
||||
}
|
||||
}
|
||||
+156
@@ -0,0 +1,156 @@
|
||||
package org.openmetadata.it.search;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import java.time.Duration;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.stream.Collectors;
|
||||
import org.awaitility.Awaitility;
|
||||
import org.openmetadata.it.server.ServerHandle;
|
||||
import org.openmetadata.schema.EntityInterface;
|
||||
import org.openmetadata.schema.type.EntityReference;
|
||||
import org.openmetadata.sdk.network.HttpMethod;
|
||||
|
||||
/**
|
||||
* Wraps {@code POST /v1/search/reindexEntities} — the per-entity refresh path
|
||||
* that observability tests use to validate "does the doc shape survive an
|
||||
* in-place rebuild?"
|
||||
*
|
||||
* <p>The endpoint is async (returns 202 immediately, runs the reindex on
|
||||
* {@code AsyncService.executorService}). Callers need a deterministic wait so
|
||||
* the post-reindex UI render reflects fresh state, not stale. This helper
|
||||
* polls the search engine until every targeted entity id is present in its
|
||||
* alias and gives up after 30 s.
|
||||
*
|
||||
* <p>Always uses {@code recreate=true} — that's the harder path (delete-then-add
|
||||
* exposes the brief gap where the doc is absent) and the one the user wants
|
||||
* exercised by every observability UIIT.
|
||||
*/
|
||||
public final class ReindexEntitiesClient {
|
||||
|
||||
public static final Duration DEFAULT_WAIT = Duration.ofSeconds(30);
|
||||
private static final Duration POLL_INTERVAL = Duration.ofMillis(500);
|
||||
|
||||
private final ServerHandle server;
|
||||
private final SearchClient search;
|
||||
private final IndexAliasInspector inspector;
|
||||
|
||||
public ReindexEntitiesClient(final ServerHandle server) {
|
||||
this.server = server;
|
||||
this.search = new SearchClient(server);
|
||||
this.inspector = new IndexAliasInspector(server);
|
||||
}
|
||||
|
||||
/**
|
||||
* Triggers a recreate reindex for every entity in {@code refs} and blocks
|
||||
* until each one is present again in its alias (or 30 s elapses).
|
||||
*/
|
||||
public void recreateAndAwait(final List<EntityReference> refs) {
|
||||
recreateAndAwait(refs, DEFAULT_WAIT);
|
||||
}
|
||||
|
||||
public void recreateAndAwait(final List<EntityReference> refs, final Duration maxWait) {
|
||||
if (refs == null || refs.isEmpty()) {
|
||||
throw new IllegalArgumentException("entity reference list must not be empty");
|
||||
}
|
||||
trigger(refs);
|
||||
awaitAllPresent(refs, maxWait);
|
||||
}
|
||||
|
||||
/**
|
||||
* Convenience overload — builds {@link EntityReference}s from the entities with an
|
||||
* explicit {@code entityType}. Avoids the gotcha where {@code Entity.getEntityReference()}
|
||||
* may return a reference whose {@code type} field is null (the API doesn't always
|
||||
* populate it on the response), which makes the endpoint reject the body with
|
||||
* "type must not be null".
|
||||
*/
|
||||
public void recreateAndAwait(
|
||||
final String entityType, final List<? extends EntityInterface> entities) {
|
||||
if (entities == null || entities.isEmpty()) {
|
||||
throw new IllegalArgumentException("entity list must not be empty");
|
||||
}
|
||||
final List<EntityReference> refs =
|
||||
entities.stream()
|
||||
.map(
|
||||
e ->
|
||||
new EntityReference()
|
||||
.withId(e.getId())
|
||||
.withType(entityType)
|
||||
.withFullyQualifiedName(e.getFullyQualifiedName()))
|
||||
.toList();
|
||||
recreateAndAwait(refs, DEFAULT_WAIT);
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait (without triggering anything) for the given entities to be present in their
|
||||
* alias. Useful as a pre-test gate so the "before" UI snapshot doesn't race the
|
||||
* live-index path that runs after entity creation.
|
||||
*/
|
||||
public void awaitIndexed(
|
||||
final String entityType, final List<? extends EntityInterface> entities) {
|
||||
awaitIndexed(entityType, entities, DEFAULT_WAIT);
|
||||
}
|
||||
|
||||
public void awaitIndexed(
|
||||
final String entityType,
|
||||
final List<? extends EntityInterface> entities,
|
||||
final Duration maxWait) {
|
||||
final List<EntityReference> refs =
|
||||
entities.stream()
|
||||
.map(
|
||||
e ->
|
||||
new EntityReference()
|
||||
.withId(e.getId())
|
||||
.withType(entityType)
|
||||
.withFullyQualifiedName(e.getFullyQualifiedName()))
|
||||
.toList();
|
||||
awaitAllPresent(refs, maxWait);
|
||||
}
|
||||
|
||||
private void trigger(final List<EntityReference> refs) {
|
||||
// The endpoint returns a plain-text "Reindex process started for N..." message,
|
||||
// not JSON — use executeForString so the SDK doesn't try to parse it as JSON.
|
||||
server
|
||||
.sdk()
|
||||
.getHttpClient()
|
||||
.executeForString(
|
||||
HttpMethod.POST, "/v1/search/reindexEntities?recreate=true&timeoutMinutes=5", refs);
|
||||
}
|
||||
|
||||
/**
|
||||
* For each entity, poll its alias until {@code id} matches one doc. Recreate
|
||||
* deletes then re-adds, so a doc may briefly be absent — we wait for it to
|
||||
* reappear rather than checking once.
|
||||
*/
|
||||
private void awaitAllPresent(final List<EntityReference> refs, final Duration maxWait) {
|
||||
final Map<String, List<String>> byAlias =
|
||||
refs.stream()
|
||||
.collect(
|
||||
Collectors.groupingBy(
|
||||
r -> inspector.aliasFor(r.getType()),
|
||||
Collectors.mapping(r -> r.getId().toString(), Collectors.toList())));
|
||||
|
||||
Awaitility.await("reindexEntities to land for " + refs.size() + " entities")
|
||||
.atMost(maxWait)
|
||||
.pollInterval(POLL_INTERVAL)
|
||||
.ignoreExceptions()
|
||||
.until(
|
||||
() -> {
|
||||
for (final Map.Entry<String, List<String>> entry : byAlias.entrySet()) {
|
||||
final String alias = entry.getKey();
|
||||
for (final String id : entry.getValue()) {
|
||||
if (!docExists(alias, id)) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
return true;
|
||||
});
|
||||
}
|
||||
|
||||
private boolean docExists(final String alias, final String id) {
|
||||
final String body = "{\"size\":0,\"query\":{\"term\":{\"id.keyword\":\"" + id + "\"}}}";
|
||||
final JsonNode response = search.search(alias, body);
|
||||
return response.path("hits").path("total").path("value").asLong() == 1;
|
||||
}
|
||||
}
|
||||
+382
@@ -0,0 +1,382 @@
|
||||
package org.openmetadata.it.search;
|
||||
|
||||
import java.time.Duration;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import org.awaitility.Awaitility;
|
||||
import org.awaitility.core.ConditionTimeoutException;
|
||||
import org.openmetadata.it.server.ServerHandle;
|
||||
import org.openmetadata.it.util.OssTestServer;
|
||||
import org.openmetadata.schema.entity.app.AppRunRecord;
|
||||
import org.openmetadata.sdk.network.HttpClient;
|
||||
import org.openmetadata.sdk.network.HttpMethod;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
/**
|
||||
* Drives the SearchIndexingApplication via the SDK and waits for run completion.
|
||||
*
|
||||
* <p>Triggering the app exercises the full reindex pipeline — {@code SearchIndexApp},
|
||||
* {@code ReindexingOrchestrator}, {@code BulkSink} — which is the surface most of the
|
||||
* EPIC #3731 fixes touch. Use this helper instead of the entity-level
|
||||
* {@code /v1/search/reindex} endpoint when a scenario must validate orchestrator behaviour.
|
||||
*/
|
||||
public final class ReindexHelpers {
|
||||
|
||||
public static final String SEARCH_INDEX_APP = "SearchIndexingApplication";
|
||||
|
||||
private static final Logger LOG = LoggerFactory.getLogger(ReindexHelpers.class);
|
||||
private static final Set<String> TERMINAL_STATUSES =
|
||||
Set.of("success", "failed", "completed", "stopped", "activeError");
|
||||
private static final Set<String> SUCCESS_STATUSES = Set.of("success", "completed");
|
||||
private static final String REINDEX_TIMEOUT_MIN_PROP = "jpw.reindex.timeoutMin";
|
||||
private static final int EXTERNAL_TIMEOUT_MINUTES = 60;
|
||||
private static final int EMBEDDED_TIMEOUT_MINUTES = 15;
|
||||
private static final String PROPAGATION_TIMEOUT_MIN_PROP = "jpw.search.propagationTimeoutMin";
|
||||
private static final int DEFAULT_PROPAGATION_MINUTES = 5;
|
||||
// A stopped run leaves the server in a post-stop window (reindex lock still held by the
|
||||
// winding-down job) where fresh runs are ACCEPTED but fail within seconds with an empty
|
||||
// failureContext. The window self-heals within minutes, so baseline-recreate retries must
|
||||
// be spaced across a minutes-scale budget — back-to-back attempts all land inside it.
|
||||
private static final Duration BASELINE_RECREATE_MAX_WAIT = Duration.ofMinutes(10);
|
||||
private static final Duration BASELINE_RECREATE_BACKOFF = Duration.ofSeconds(30);
|
||||
private static final Duration POLL_INTERVAL = Duration.ofSeconds(2);
|
||||
|
||||
private ReindexHelpers() {}
|
||||
|
||||
/**
|
||||
* Max wait for a reindex run to reach a terminal status — and for the singleton SearchIndexApp
|
||||
* lock to free so a fresh trigger is accepted. Reindex duration scales with catalog size and
|
||||
* cluster load: on a large shared/external cluster a single run can take ~20 minutes, and a
|
||||
* concurrent test's run holds the lock for that whole window, so external mode defaults to 60
|
||||
* minutes. Embedded/PR runs have a small catalog and a private cluster, so they default to 15 —
|
||||
* a wedged run fails the build in minutes rather than burning the external cap. Override either
|
||||
* with {@code -Djpw.reindex.timeoutMin}. The waits poll run status and return the instant it goes
|
||||
* terminal, so a large cap never slows a fast run — it only bounds the failure.
|
||||
*/
|
||||
public static Duration reindexTimeout() {
|
||||
final int fallback =
|
||||
OssTestServer.isExternalMode() ? EXTERNAL_TIMEOUT_MINUTES : EMBEDDED_TIMEOUT_MINUTES;
|
||||
return Duration.ofMinutes(Integer.getInteger(REINDEX_TIMEOUT_MIN_PROP, fallback));
|
||||
}
|
||||
|
||||
/**
|
||||
* Max wait for a single document/field change to become query-visible — live-indexing lag or the
|
||||
* tail of a reindex. On a clean cluster this is seconds, so the poll returns almost immediately
|
||||
* and the cap never slows a passing test; it only bounds the failure. On a shared/external cluster
|
||||
* under concurrent reindex load it can run minutes, so this defaults high and is overridable via
|
||||
* {@code -Djpw.search.propagationTimeoutMin}.
|
||||
*/
|
||||
public static Duration searchPropagationTimeout() {
|
||||
return Duration.ofMinutes(
|
||||
Integer.getInteger(PROPAGATION_TIMEOUT_MIN_PROP, DEFAULT_PROPAGATION_MINUTES));
|
||||
}
|
||||
|
||||
/** Triggers the named app via {@code POST /v1/apps/trigger/{name}}. */
|
||||
public static void triggerApp(final ServerHandle server, final String appName) {
|
||||
final HttpClient http = server.sdk().getHttpClient();
|
||||
http.execute(HttpMethod.POST, "/v1/apps/trigger/" + appName, null, Void.class);
|
||||
}
|
||||
|
||||
/**
|
||||
* Triggers the named app with an inline config payload — overrides the app's persisted
|
||||
* configuration for this run only. Useful for slowing reindex (small batch + few
|
||||
* threads) so concurrent assertions have time to observe mid-flight state.
|
||||
*/
|
||||
public static void triggerAppWithConfig(
|
||||
final ServerHandle server, final String appName, final Map<String, Object> config) {
|
||||
final HttpClient http = server.sdk().getHttpClient();
|
||||
http.execute(HttpMethod.POST, "/v1/apps/trigger/" + appName, config, Void.class);
|
||||
}
|
||||
|
||||
/**
|
||||
* Waits for the previous run to reach a terminal status, then triggers a fresh run with
|
||||
* the given inline config — retrying the trigger until the server accepts it. OM's
|
||||
* internal app-execution lock can linger briefly after {@code AppRunRecord} flips to
|
||||
* {@code success}; a naive immediate trigger races with that release and gets
|
||||
* "Job is already running" errors.
|
||||
*/
|
||||
public static void triggerSearchIndexWithConfigWhenIdle(
|
||||
final ServerHandle server,
|
||||
final Map<String, Object> config,
|
||||
final Duration acceptanceTimeout) {
|
||||
waitForLatestRunTerminal(server, SEARCH_INDEX_APP, reindexTimeout());
|
||||
Awaitility.await("trigger SearchIndexApp with config")
|
||||
.atMost(acceptanceTimeout)
|
||||
.pollInterval(Duration.ofSeconds(2))
|
||||
.ignoreExceptions()
|
||||
.until(
|
||||
() -> {
|
||||
triggerAppWithConfig(server, SEARCH_INDEX_APP, config);
|
||||
return true;
|
||||
});
|
||||
}
|
||||
|
||||
/** Status of the latest run for the named app, or {@code null} if no runs yet. */
|
||||
public static String latestRunStatus(final ServerHandle server, final String appName) {
|
||||
final AppRunRecord run = fetchLatestRun(server, appName);
|
||||
return (run == null || run.getStatus() == null) ? null : run.getStatus().value();
|
||||
}
|
||||
|
||||
/** Whether the latest run for the named app is in a terminal status. */
|
||||
public static boolean latestRunIsTerminal(final ServerHandle server, final String appName) {
|
||||
return isTerminal(fetchLatestRun(server, appName));
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether the latest run started at or after {@code sinceMillis} AND is in a terminal
|
||||
* status. Lets callers distinguish "the run I just triggered finished" from "an older
|
||||
* run is still marked Success" — which {@link #latestRunIsTerminal} cannot.
|
||||
*/
|
||||
public static boolean freshRunIsTerminal(
|
||||
final ServerHandle server, final String appName, final long sinceMillis) {
|
||||
final AppRunRecord run = fetchLatestRun(server, appName);
|
||||
if (run == null || run.getTimestamp() == null || run.getTimestamp() < sinceMillis) {
|
||||
return false;
|
||||
}
|
||||
return isTerminal(run);
|
||||
}
|
||||
|
||||
/**
|
||||
* Blocks until a run started at or after {@code sinceMillis} appears for the app. Use
|
||||
* after a trigger to avoid racing the next probe loop against a stale {@code Success}
|
||||
* status from the previous run.
|
||||
*/
|
||||
public static void waitForRunStartedSince(
|
||||
final ServerHandle server,
|
||||
final String appName,
|
||||
final long sinceMillis,
|
||||
final Duration timeout) {
|
||||
Awaitility.await("new run for " + appName + " to register")
|
||||
.atMost(timeout)
|
||||
.pollInterval(Duration.ofSeconds(1))
|
||||
.ignoreExceptions()
|
||||
.until(
|
||||
() -> {
|
||||
final AppRunRecord run = fetchLatestRun(server, appName);
|
||||
return run != null && run.getTimestamp() != null && run.getTimestamp() >= sinceMillis;
|
||||
});
|
||||
}
|
||||
|
||||
/** Trigger SearchIndexingApplication and block until the latest run reaches a terminal state. */
|
||||
public static AppRunRecord triggerSearchIndexAndWait(final ServerHandle server) {
|
||||
return triggerSearchIndexAndWait(server, reindexTimeout());
|
||||
}
|
||||
|
||||
public static AppRunRecord triggerSearchIndexAndWait(
|
||||
final ServerHandle server, final Duration timeout) {
|
||||
waitForLatestRunTerminal(server, SEARCH_INDEX_APP, reindexTimeout());
|
||||
final long triggeredAtMillis = System.currentTimeMillis();
|
||||
triggerWhenAccepted(server, reindexTimeout());
|
||||
final AppRunRecord run = waitForRunAfter(server, SEARCH_INDEX_APP, triggeredAtMillis, timeout);
|
||||
logIfNotSuccess(run);
|
||||
return run;
|
||||
}
|
||||
|
||||
/**
|
||||
* Fires the plain trigger, retrying until the server accepts it. OM's app-execution lock can
|
||||
* linger briefly after the previous run flips to terminal (notably the {@code beforeEach}
|
||||
* recreate at class transitions in the serial search-it suite), so a one-shot trigger races
|
||||
* the release and gets a 500 "Job is already running". The first accepting call starts exactly
|
||||
* one run, so the retry never double-triggers.
|
||||
*/
|
||||
private static void triggerWhenAccepted(
|
||||
final ServerHandle server, final Duration acceptanceTimeout) {
|
||||
Awaitility.await("trigger SearchIndexApp")
|
||||
.atMost(acceptanceTimeout)
|
||||
.pollInterval(Duration.ofSeconds(2))
|
||||
.ignoreExceptions()
|
||||
.until(
|
||||
() -> {
|
||||
triggerApp(server, SEARCH_INDEX_APP);
|
||||
return true;
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Rebuilds every index from scratch and blocks until the fresh run reaches a terminal status.
|
||||
* Unlike {@link #triggerSearchIndexAndWait}, this passes {@code recreateIndex=true} so dropped
|
||||
* indices are recreated and read aliases are re-promoted — the only reliable way to restore a
|
||||
* baseline after a test has dropped indices or left an alias unswapped (e.g. a stopped recreate
|
||||
* run). Used by {@code SearchClusterResetExtension}.
|
||||
*
|
||||
* <p>Failed attempts retry with {@link #BASELINE_RECREATE_BACKOFF} spacing for up to {@link
|
||||
* #BASELINE_RECREATE_MAX_WAIT}: after a stop, fresh runs fast-fail until the stopped job's
|
||||
* wind-down releases the server-side reindex lock, so only spaced retries can outlast that
|
||||
* window. The budget only bounds the failure — the first successful run returns immediately.
|
||||
*
|
||||
* <p>Throws {@link IllegalStateException} if the baseline never succeeds within the budget, so no
|
||||
* caller silently proceeds against a cluster whose indices are still half-dropped or unswapped.
|
||||
*/
|
||||
public static AppRunRecord recreateAllAndWait(final ServerHandle server, final Duration timeout) {
|
||||
final long deadlineMillis = System.currentTimeMillis() + BASELINE_RECREATE_MAX_WAIT.toMillis();
|
||||
AppRunRecord run = null;
|
||||
for (int attempt = 1; shouldRetryBaselineRecreate(run, attempt, deadlineMillis); attempt++) {
|
||||
if (attempt > 1) {
|
||||
backOff(BASELINE_RECREATE_BACKOFF);
|
||||
}
|
||||
final long triggeredAtMillis = System.currentTimeMillis();
|
||||
triggerSearchIndexWithConfigWhenIdle(server, Map.of("recreateIndex", true), reindexTimeout());
|
||||
run = waitForRunAfter(server, SEARCH_INDEX_APP, triggeredAtMillis, timeout);
|
||||
logBaselineRecreateAttempt(run, attempt);
|
||||
}
|
||||
if (!isSuccess(run)) {
|
||||
throw new IllegalStateException(
|
||||
"Baseline recreate never succeeded after retrying for "
|
||||
+ BASELINE_RECREATE_MAX_WAIT
|
||||
+ "; final status '"
|
||||
+ statusOf(run)
|
||||
+ "'"
|
||||
+ failureSummary(run));
|
||||
}
|
||||
return run;
|
||||
}
|
||||
|
||||
private static boolean shouldRetryBaselineRecreate(
|
||||
final AppRunRecord run, final int attempt, final long deadlineMillis) {
|
||||
return !isSuccess(run) && (attempt == 1 || System.currentTimeMillis() < deadlineMillis);
|
||||
}
|
||||
|
||||
private static void logBaselineRecreateAttempt(final AppRunRecord run, final int attempt) {
|
||||
if (!isSuccess(run)) {
|
||||
LOG.warn(
|
||||
"Baseline recreate attempt {} ended in status '{}'{} — a stopped reindex leaves a"
|
||||
+ " minutes-long post-stop window where fresh runs are accepted but fail within"
|
||||
+ " seconds with an empty failureContext; backing off {} and retrying for up to {}.",
|
||||
attempt,
|
||||
statusOf(run),
|
||||
failureSummary(run),
|
||||
BASELINE_RECREATE_BACKOFF,
|
||||
BASELINE_RECREATE_MAX_WAIT);
|
||||
}
|
||||
}
|
||||
|
||||
private static void backOff(final Duration duration) {
|
||||
try {
|
||||
Thread.sleep(duration.toMillis());
|
||||
} catch (final InterruptedException e) {
|
||||
Thread.currentThread().interrupt();
|
||||
}
|
||||
}
|
||||
|
||||
/** Triggers a per-entity reindex via {@code POST /v1/search/reindex?entityType=...}. */
|
||||
public static void reindexEntityType(final ServerHandle server, final String entityType) {
|
||||
server.sdk().search().reindex(entityType);
|
||||
}
|
||||
|
||||
/**
|
||||
* Sends a stop request to the named app and blocks until the latest run reaches a
|
||||
* terminal status (typically {@code stopped}). Used by stop-under-load tests.
|
||||
*/
|
||||
public static AppRunRecord stopAppAndWait(
|
||||
final ServerHandle server, final String appName, final Duration timeout) {
|
||||
server
|
||||
.sdk()
|
||||
.getHttpClient()
|
||||
.execute(HttpMethod.POST, "/v1/apps/stop/" + appName, null, Void.class);
|
||||
// sinceMillis=0: we want the run we just stopped to reach terminal, not a fresh run.
|
||||
return waitForRunAfter(server, appName, 0L, timeout);
|
||||
}
|
||||
|
||||
/** Records the wall-clock instant just before sending a stop request. */
|
||||
public static long sendStop(final ServerHandle server, final String appName) {
|
||||
final long ts = System.currentTimeMillis();
|
||||
server
|
||||
.sdk()
|
||||
.getHttpClient()
|
||||
.execute(HttpMethod.POST, "/v1/apps/stop/" + appName, null, Void.class);
|
||||
return ts;
|
||||
}
|
||||
|
||||
/** Triggers reindex of all entity types via {@code POST /v1/search/reindex/all}. */
|
||||
public static void reindexAll(final ServerHandle server) {
|
||||
server.sdk().search().reindexAll();
|
||||
}
|
||||
|
||||
/** Latest {@link AppRunRecord} for the named app, or {@code null} if no runs yet. */
|
||||
public static AppRunRecord fetchLatestRun(final ServerHandle server, final String appName) {
|
||||
return server
|
||||
.sdk()
|
||||
.getHttpClient()
|
||||
.execute(
|
||||
HttpMethod.GET, "/v1/apps/name/" + appName + "/runs/latest", null, AppRunRecord.class);
|
||||
}
|
||||
|
||||
/** Whether the run ended in a status the suite treats as success. */
|
||||
public static boolean isSuccess(final AppRunRecord run) {
|
||||
return run != null && run.getStatus() != null && SUCCESS_STATUSES.contains(statusOf(run));
|
||||
}
|
||||
|
||||
/** The run's status value, or {@code "none"} when the run or its status is absent. */
|
||||
public static String statusOf(final AppRunRecord run) {
|
||||
return (run == null || run.getStatus() == null) ? "none" : run.getStatus().value();
|
||||
}
|
||||
|
||||
private static void logIfNotSuccess(final AppRunRecord run) {
|
||||
if (!isSuccess(run)) {
|
||||
LOG.warn(
|
||||
"SearchIndexApp run did not succeed: status='{}'{}", statusOf(run), failureSummary(run));
|
||||
}
|
||||
}
|
||||
|
||||
/** A short, log-safe rendering of the run's failure context, or empty when there is none. */
|
||||
private static String failureSummary(final AppRunRecord run) {
|
||||
final String summary;
|
||||
if (run == null || run.getFailureContext() == null) {
|
||||
summary = "";
|
||||
} else {
|
||||
summary = " failureContext=" + run.getFailureContext();
|
||||
}
|
||||
return summary;
|
||||
}
|
||||
|
||||
private static boolean isTerminal(final AppRunRecord run) {
|
||||
if (run == null || run.getStatus() == null) {
|
||||
return true;
|
||||
}
|
||||
return TERMINAL_STATUSES.contains(run.getStatus().value());
|
||||
}
|
||||
|
||||
private static void waitForLatestRunTerminal(
|
||||
final ServerHandle server, final String appName, final Duration timeout) {
|
||||
try {
|
||||
Awaitility.await("waitForLatestRunTerminal " + appName)
|
||||
.atMost(timeout)
|
||||
.pollInterval(POLL_INTERVAL)
|
||||
.ignoreExceptions()
|
||||
.until(() -> isTerminal(fetchLatestRun(server, appName)));
|
||||
} catch (ConditionTimeoutException ignored) {
|
||||
// Best-effort: trigger will retry if the previous run is still active.
|
||||
}
|
||||
}
|
||||
|
||||
private static AppRunRecord waitForRunAfter(
|
||||
final ServerHandle server,
|
||||
final String appName,
|
||||
final long minStartMillis,
|
||||
final Duration timeout) {
|
||||
final Map<String, AppRunRecord> ref = new java.util.concurrent.ConcurrentHashMap<>();
|
||||
Awaitility.await("waitForRunAfter " + appName)
|
||||
.atMost(timeout)
|
||||
.pollInterval(POLL_INTERVAL)
|
||||
.ignoreExceptions()
|
||||
.until(() -> isFreshTerminalRun(server, appName, minStartMillis, ref));
|
||||
return ref.get("latest");
|
||||
}
|
||||
|
||||
private static boolean isFreshTerminalRun(
|
||||
final ServerHandle server,
|
||||
final String appName,
|
||||
final long minStartMillis,
|
||||
final Map<String, AppRunRecord> ref) {
|
||||
final AppRunRecord run = fetchLatestRun(server, appName);
|
||||
if (run == null || run.getTimestamp() == null || run.getTimestamp() < minStartMillis) {
|
||||
return false;
|
||||
}
|
||||
if (!isTerminal(run)) {
|
||||
return false;
|
||||
}
|
||||
ref.put("latest", run);
|
||||
return true;
|
||||
}
|
||||
}
|
||||
+88
@@ -0,0 +1,88 @@
|
||||
package org.openmetadata.it.search;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
import java.time.Duration;
|
||||
import java.time.LocalDate;
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
import org.awaitility.Awaitility;
|
||||
import org.openmetadata.it.util.SdkClients;
|
||||
import org.openmetadata.schema.api.data.CreateTable;
|
||||
import org.openmetadata.schema.entity.data.DatabaseSchema;
|
||||
import org.openmetadata.schema.entity.data.Table;
|
||||
import org.openmetadata.schema.type.Column;
|
||||
import org.openmetadata.schema.type.ColumnDataType;
|
||||
import org.openmetadata.schema.type.DailyCount;
|
||||
import org.openmetadata.schema.type.EntityUsage;
|
||||
import org.openmetadata.schema.type.TagLabel;
|
||||
import org.openmetadata.sdk.network.HttpMethod;
|
||||
|
||||
/**
|
||||
* Shared seed + wait helpers for the search-relevancy ITs: a controlled cohort of tables that share
|
||||
* a searchable marker (so one query matches them all) and differ only in the field under test (e.g.
|
||||
* tier), plus a poll that blocks until the cohort is live-indexed and countable.
|
||||
*/
|
||||
public final class RelevancyFixtures {
|
||||
|
||||
private static final String TABLE_INDEX = "table";
|
||||
|
||||
private RelevancyFixtures() {}
|
||||
|
||||
/**
|
||||
* Creates a table whose {@code description} carries {@code marker} (so a {@code q=marker} search
|
||||
* matches it), optionally tagged with a tier. The parent schema is tracked for cleanup by its
|
||||
* factory.
|
||||
*/
|
||||
public static Table createTable(
|
||||
final DatabaseSchema schema, final String name, final String marker, final String tierFqn) {
|
||||
final CreateTable request =
|
||||
new CreateTable()
|
||||
.withName(name)
|
||||
.withDatabaseSchema(schema.getFullyQualifiedName())
|
||||
.withDescription(marker)
|
||||
.withColumns(List.of(new Column().withName("c1").withDataType(ColumnDataType.STRING)));
|
||||
if (tierFqn != null) {
|
||||
request.withTags(List.of(tierLabel(tierFqn)));
|
||||
}
|
||||
return SdkClients.adminClient().tables().create(request);
|
||||
}
|
||||
|
||||
/** Reports a daily usage count so the table's {@code usageSummary.weeklyStats.count} becomes nonzero. */
|
||||
public static void reportUsage(final Table table, final int count) {
|
||||
final DailyCount usage = new DailyCount().withDate(LocalDate.now().toString()).withCount(count);
|
||||
SdkClients.adminClient()
|
||||
.getHttpClient()
|
||||
.execute(HttpMethod.POST, "/v1/usage/table/" + table.getId(), usage, EntityUsage.class);
|
||||
}
|
||||
|
||||
/** A short token with no shared substring across calls — safe to use as a search query marker. */
|
||||
public static String uniqueToken(final String prefix) {
|
||||
return prefix + UUID.randomUUID().toString().replace("-", "").substring(0, 12);
|
||||
}
|
||||
|
||||
public static TagLabel tierLabel(final String tierFqn) {
|
||||
return new TagLabel()
|
||||
.withTagFQN(tierFqn)
|
||||
.withSource(TagLabel.TagSource.CLASSIFICATION)
|
||||
.withLabelType(TagLabel.LabelType.MANUAL)
|
||||
.withState(TagLabel.State.CONFIRMED);
|
||||
}
|
||||
|
||||
/** Blocks until exactly {@code expected} tables whose name starts with {@code namePrefix} index. */
|
||||
public static void awaitTablesIndexed(
|
||||
final IndexAliasInspector indices,
|
||||
final SearchAssertions search,
|
||||
final String namePrefix,
|
||||
final int expected,
|
||||
final Duration timeout) {
|
||||
final String index = indices.indexNameFor(TABLE_INDEX);
|
||||
Awaitility.await("tables with prefix '" + namePrefix + "' indexed")
|
||||
.atMost(timeout)
|
||||
.pollInterval(Duration.ofSeconds(2))
|
||||
.pollDelay(Duration.ZERO)
|
||||
.ignoreExceptions()
|
||||
.untilAsserted(
|
||||
() -> assertThat(search.countByNamePrefix(index, namePrefix)).isEqualTo(expected));
|
||||
}
|
||||
}
|
||||
+148
@@ -0,0 +1,148 @@
|
||||
package org.openmetadata.it.search;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Iterator;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import org.openmetadata.it.server.ServerHandle;
|
||||
|
||||
/**
|
||||
* Assertions over the live search engine — index existence, doc counts, alias resolution,
|
||||
* doc shape, presence of fields. All assertions hit the engine directly via {@link SearchClient}
|
||||
* to validate what reindex actually wrote (rather than what the SDK reports).
|
||||
*/
|
||||
public final class SearchAssertions {
|
||||
|
||||
private final SearchClient search;
|
||||
|
||||
public SearchAssertions(final ServerHandle server) {
|
||||
this.search = new SearchClient(server);
|
||||
}
|
||||
|
||||
public boolean indexExists(final String indexOrAlias) {
|
||||
return search.indexExists(indexOrAlias);
|
||||
}
|
||||
|
||||
public long count(final String indexOrAlias) {
|
||||
final JsonNode body = search.count(indexOrAlias);
|
||||
return body.path("count").asLong();
|
||||
}
|
||||
|
||||
public long countByEntityType(final String alias, final String entityType) {
|
||||
final String body = "{\"query\":{\"term\":{\"entityType\":\"" + entityType + "\"}}}";
|
||||
final JsonNode response = search.count(alias, body);
|
||||
return response.path("count").asLong();
|
||||
}
|
||||
|
||||
/**
|
||||
* Exact count of docs whose {@code name} starts with {@code namePrefix}. Scopes a count to a
|
||||
* single test run's entities (named {@code <unique-prefix>_<n>}) regardless of what else lives on
|
||||
* the cluster — unlike the relevance-ranked Explore search, which fuzzy-matches across the whole
|
||||
* index. {@code name.keyword} carries a {@code lowercase_normalizer}, so the prefix is lowercased
|
||||
* to match the indexed (lowercased) value.
|
||||
*/
|
||||
public long countByNamePrefix(final String indexOrAlias, final String namePrefix) {
|
||||
final String prefix = escape(namePrefix.toLowerCase(java.util.Locale.ROOT));
|
||||
final String body = "{\"query\":{\"prefix\":{\"name.keyword\":\"" + prefix + "\"}}}";
|
||||
final JsonNode response = search.count(indexOrAlias, body);
|
||||
return response.path("count").asLong();
|
||||
}
|
||||
|
||||
public List<String> indicesForAlias(final String alias) {
|
||||
final JsonNode body = search.alias(alias);
|
||||
final List<String> indices = new ArrayList<>();
|
||||
final Iterator<Map.Entry<String, JsonNode>> fields = body.fields();
|
||||
while (fields.hasNext()) {
|
||||
indices.add(fields.next().getKey());
|
||||
}
|
||||
return indices;
|
||||
}
|
||||
|
||||
public List<String> listIndices(final String pattern) {
|
||||
final JsonNode body = search.indices(pattern);
|
||||
final List<String> result = new ArrayList<>();
|
||||
body.forEach(node -> result.add(node.path("index").asText()));
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether any OM entity id appears in more than one document under the alias — the exact
|
||||
* duplicate check behind the no-duplicates-during-reindex invariant. A non-atomic alias swap
|
||||
* leaves the alias transiently spanning both the old and new backing index, so the same
|
||||
* logical entity is returned twice; a {@code terms} aggregation on {@code id.keyword} with
|
||||
* {@code min_doc_count=2} surfaces exactly that.
|
||||
*
|
||||
* <p>Aggregating on {@code id.keyword} rather than {@code _id} works without enabling
|
||||
* fielddata; OM sets the doc {@code _id} equal to the entity id so the two values match. This
|
||||
* is exact on a single-shard index (the OM test indices are created single-shard from the
|
||||
* static mapping), unlike an approximate {@code cardinality} aggregation, which can differ
|
||||
* from the doc count by a handful purely from HyperLogLog error and produce false positives.
|
||||
*/
|
||||
public boolean hasDuplicateIds(final String alias) {
|
||||
final String body =
|
||||
"{\"size\":0,\"aggs\":{\"dupes\":{\"terms\":{\"field\":\"id.keyword\","
|
||||
+ "\"min_doc_count\":2,\"size\":1}}}}";
|
||||
final JsonNode response = search.search(alias, body);
|
||||
return !response.path("aggregations").path("dupes").path("buckets").isEmpty();
|
||||
}
|
||||
|
||||
public void assertCountAtLeast(final String indexOrAlias, final long minimum) {
|
||||
final long actual = count(indexOrAlias);
|
||||
assertThat(actual)
|
||||
.as("Document count for %s should be >= %d", indexOrAlias, minimum)
|
||||
.isGreaterThanOrEqualTo(minimum);
|
||||
}
|
||||
|
||||
public void assertCountEquals(final String indexOrAlias, final long expected) {
|
||||
final long actual = count(indexOrAlias);
|
||||
assertThat(actual)
|
||||
.as("Document count for %s should equal %d", indexOrAlias, expected)
|
||||
.isEqualTo(expected);
|
||||
}
|
||||
|
||||
public void assertEntityIndexed(final String alias, final String entityType, final String fqn) {
|
||||
final String body =
|
||||
"{\"query\":{\"bool\":{\"must\":["
|
||||
+ "{\"term\":{\"entityType\":\""
|
||||
+ entityType
|
||||
+ "\"}},"
|
||||
+ "{\"term\":{\"fullyQualifiedName\":\""
|
||||
+ escape(fqn)
|
||||
+ "\"}}"
|
||||
+ "]}}}";
|
||||
final JsonNode response = search.count(alias, body);
|
||||
final long count = response.path("count").asLong();
|
||||
assertThat(count).as("%s '%s' should be indexed in %s", entityType, fqn, alias).isEqualTo(1);
|
||||
}
|
||||
|
||||
public void assertEntityNotIndexed(
|
||||
final String alias, final String entityType, final String fqn) {
|
||||
final String body =
|
||||
"{\"query\":{\"bool\":{\"must\":["
|
||||
+ "{\"term\":{\"entityType\":\""
|
||||
+ entityType
|
||||
+ "\"}},"
|
||||
+ "{\"term\":{\"fullyQualifiedName\":\""
|
||||
+ escape(fqn)
|
||||
+ "\"}}"
|
||||
+ "]}}}";
|
||||
final JsonNode response = search.count(alias, body);
|
||||
final long count = response.path("count").asLong();
|
||||
assertThat(count).as("%s '%s' should NOT be indexed in %s", entityType, fqn, alias).isZero();
|
||||
}
|
||||
|
||||
public void assertNoOrphanedIndices(final String entityIndexBaseName) {
|
||||
final List<String> indices = listIndices(entityIndexBaseName + "*");
|
||||
final List<String> aliasOwned = indicesForAlias(entityIndexBaseName);
|
||||
assertThat(indices)
|
||||
.as("No prefixed/orphaned indices should remain for %s", entityIndexBaseName)
|
||||
.containsExactlyInAnyOrderElementsOf(aliasOwned);
|
||||
}
|
||||
|
||||
private static String escape(final String value) {
|
||||
return value.replace("\\", "\\\\").replace("\"", "\\\"");
|
||||
}
|
||||
}
|
||||
+274
@@ -0,0 +1,274 @@
|
||||
package org.openmetadata.it.search;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import java.io.IOException;
|
||||
import java.net.URI;
|
||||
import java.net.URLEncoder;
|
||||
import java.net.http.HttpClient;
|
||||
import java.net.http.HttpRequest;
|
||||
import java.net.http.HttpResponse;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.Duration;
|
||||
import org.openmetadata.it.server.ServerHandle;
|
||||
import org.openmetadata.sdk.network.HttpMethod;
|
||||
|
||||
/**
|
||||
* Thin JSON client for the search engine's read-only introspection surface (works against ES and
|
||||
* OpenSearch). Exposes only the typed operations assertions need: {@code count}, {@code search},
|
||||
* {@code alias}, {@code mapping}, {@code indices}, {@code exists}.
|
||||
*
|
||||
* <p><b>Embedded mode</b> talks directly to {@code searchHost:searchPort} via {@link
|
||||
* java.net.http.HttpClient}, so the harness has no dependency on a specific ES/OpenSearch client
|
||||
* version. <b>External mode</b> (a remote cluster where {@code :9200} isn't reachable) routes the
|
||||
* same operations through the server's authenticated, typed {@code /v1/test-support/search}
|
||||
* endpoints — there is no raw-path passthrough, so the server only ever runs these read-only ops.
|
||||
*/
|
||||
public final class SearchClient {
|
||||
|
||||
private static final ObjectMapper MAPPER = new ObjectMapper();
|
||||
private static final Duration TIMEOUT = Duration.ofSeconds(30);
|
||||
private static final String TEST_SUPPORT = "/v1/test-support/search";
|
||||
|
||||
private final ServerHandle server;
|
||||
private final HttpClient http;
|
||||
private final URI base;
|
||||
|
||||
public SearchClient(final ServerHandle server) {
|
||||
this.server = server;
|
||||
if (server.isExternal()) {
|
||||
this.http = null;
|
||||
this.base = null;
|
||||
} else {
|
||||
this.http = HttpClient.newBuilder().connectTimeout(TIMEOUT).build();
|
||||
this.base =
|
||||
URI.create(
|
||||
server.searchScheme() + "://" + server.searchHost() + ":" + server.searchPort());
|
||||
}
|
||||
}
|
||||
|
||||
/** Doc count for an index/alias. */
|
||||
public JsonNode count(final String index) {
|
||||
final JsonNode result;
|
||||
if (server.isExternal()) {
|
||||
result = parse(proxyGet(TEST_SUPPORT + "/count?index=" + encode(index)));
|
||||
} else {
|
||||
result = engineGet("/" + index + "/_count");
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
/** Doc count for an index/alias matching the given query body. */
|
||||
public JsonNode count(final String index, final String query) {
|
||||
final JsonNode result;
|
||||
if (server.isExternal()) {
|
||||
result = parse(proxyPost(TEST_SUPPORT + "/count?index=" + encode(index), query));
|
||||
} else {
|
||||
result = enginePost("/" + index + "/_count", query);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
/** Search an index/alias with the given query body. */
|
||||
public JsonNode search(final String index, final String query) {
|
||||
final JsonNode result;
|
||||
if (server.isExternal()) {
|
||||
result = parse(proxyPost(TEST_SUPPORT + "/search?index=" + encode(index), query));
|
||||
} else {
|
||||
result = enginePost("/" + index + "/_search", query);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
/** Backing-index map for an alias (each top-level key is a backing index). */
|
||||
public JsonNode alias(final String name) {
|
||||
final JsonNode result;
|
||||
if (server.isExternal()) {
|
||||
result = parse(proxyGet(TEST_SUPPORT + "/alias?name=" + encode(name)));
|
||||
} else {
|
||||
result = engineGet("/_alias/" + name);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
public JsonNode get(final String path) {
|
||||
requireEmbedded("GET " + path);
|
||||
return execute(HttpRequest.newBuilder(base.resolve(path)).GET().build());
|
||||
}
|
||||
|
||||
public JsonNode post(final String path, final String jsonBody) {
|
||||
requireEmbedded("POST " + path);
|
||||
return execute(
|
||||
HttpRequest.newBuilder(base.resolve(path))
|
||||
.header("Content-Type", "application/json")
|
||||
.POST(HttpRequest.BodyPublishers.ofString(jsonBody))
|
||||
.build());
|
||||
}
|
||||
|
||||
public JsonNode put(final String path, final String jsonBody) {
|
||||
requireEmbedded("PUT " + path);
|
||||
return execute(
|
||||
HttpRequest.newBuilder(base.resolve(path))
|
||||
.header("Content-Type", "application/json")
|
||||
.PUT(HttpRequest.BodyPublishers.ofString(jsonBody))
|
||||
.build());
|
||||
}
|
||||
|
||||
public void delete(final String path) {
|
||||
requireEmbedded("DELETE " + path);
|
||||
try {
|
||||
final HttpResponse<String> response =
|
||||
http.send(
|
||||
HttpRequest.newBuilder(base.resolve(path)).DELETE().build(),
|
||||
HttpResponse.BodyHandlers.ofString());
|
||||
final int status = response.statusCode();
|
||||
final boolean success = (status >= 200 && status < 300) || status == 404;
|
||||
if (!success) {
|
||||
throw new SearchClientException(
|
||||
"HTTP " + status + " from DELETE " + path + ": " + response.body());
|
||||
}
|
||||
} catch (final InterruptedException e) {
|
||||
Thread.currentThread().interrupt();
|
||||
throw new SearchClientException("DELETE " + path + " interrupted", e);
|
||||
} catch (final IOException e) {
|
||||
throw new SearchClientException("DELETE " + path + " failed", e);
|
||||
}
|
||||
}
|
||||
|
||||
/** Mapping JSON for an index/alias. */
|
||||
public JsonNode mapping(final String index) {
|
||||
final JsonNode result;
|
||||
if (server.isExternal()) {
|
||||
result = parse(proxyGet(TEST_SUPPORT + "/mapping?index=" + encode(index)));
|
||||
} else {
|
||||
result = engineGet("/" + index + "/_mapping");
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
/** Index names matching the pattern (a JSON array from {@code _cat/indices}). */
|
||||
public JsonNode indices(final String pattern) {
|
||||
final JsonNode result;
|
||||
if (server.isExternal()) {
|
||||
result = parse(proxyGet(TEST_SUPPORT + "/indices?pattern=" + encode(pattern)));
|
||||
} else {
|
||||
result = engineGet("/_cat/indices/" + pattern + "?format=json&h=index");
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
/** Whether an index (or alias) with the given name exists. */
|
||||
public boolean indexExists(final String name) {
|
||||
return server.isExternal() ? existsViaServer("index", name) : headExists("/" + name);
|
||||
}
|
||||
|
||||
/** Whether an alias with the given name exists. */
|
||||
public boolean aliasExists(final String name) {
|
||||
return server.isExternal() ? existsViaServer("alias", name) : headExists("/_alias/" + name);
|
||||
}
|
||||
|
||||
private boolean existsViaServer(final String kind, final String name) {
|
||||
return parse(proxyGet(TEST_SUPPORT + "/exists?" + kind + "=" + encode(name)))
|
||||
.path("exists")
|
||||
.asBoolean();
|
||||
}
|
||||
|
||||
private String proxyGet(final String path) {
|
||||
return proxy(HttpMethod.GET, path, null);
|
||||
}
|
||||
|
||||
private String proxyPost(final String path, final String body) {
|
||||
return proxy(HttpMethod.POST, path, body);
|
||||
}
|
||||
|
||||
private String proxy(final HttpMethod method, final String path, final String body) {
|
||||
try {
|
||||
return server.sdk().getHttpClient().executeForString(method, path, body);
|
||||
} catch (final RuntimeException e) {
|
||||
throw new SearchClientException(method + " " + path + " via test-support endpoint failed", e);
|
||||
}
|
||||
}
|
||||
|
||||
private static String encode(final String value) {
|
||||
return URLEncoder.encode(value, StandardCharsets.UTF_8);
|
||||
}
|
||||
|
||||
private static JsonNode parse(final String body) {
|
||||
try {
|
||||
return MAPPER.readTree(body);
|
||||
} catch (final IOException e) {
|
||||
throw new SearchClientException("Failed to parse search response: " + body, e);
|
||||
}
|
||||
}
|
||||
|
||||
private JsonNode engineGet(final String path) {
|
||||
return execute(HttpRequest.newBuilder(base.resolve(path)).GET().build());
|
||||
}
|
||||
|
||||
private JsonNode enginePost(final String path, final String jsonBody) {
|
||||
return execute(
|
||||
HttpRequest.newBuilder(base.resolve(path))
|
||||
.header("Content-Type", "application/json")
|
||||
.POST(HttpRequest.BodyPublishers.ofString(jsonBody))
|
||||
.build());
|
||||
}
|
||||
|
||||
private boolean headExists(final String path) {
|
||||
try {
|
||||
final HttpResponse<Void> response =
|
||||
http.send(
|
||||
HttpRequest.newBuilder(base.resolve(path))
|
||||
.method("HEAD", HttpRequest.BodyPublishers.noBody())
|
||||
.build(),
|
||||
HttpResponse.BodyHandlers.discarding());
|
||||
return response.statusCode() >= 200 && response.statusCode() < 300;
|
||||
} catch (final InterruptedException e) {
|
||||
Thread.currentThread().interrupt();
|
||||
throw new SearchClientException("HEAD " + path + " interrupted", e);
|
||||
} catch (final IOException e) {
|
||||
throw new SearchClientException("HEAD " + path + " failed", e);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The raw {@link #get}/{@link #post}/{@link #put}/{@link #delete} helpers talk straight to the
|
||||
* engine over {@code base}/{@code http}, which are null in external mode (there is no raw-path
|
||||
* passthrough — only the typed read-only proxy). Fail fast with a clear message instead of a bare
|
||||
* NPE so an external-mode run is self-explanatory.
|
||||
*/
|
||||
private void requireEmbedded(final String operation) {
|
||||
if (server.isExternal()) {
|
||||
throw new SearchClientException(
|
||||
operation
|
||||
+ " requires direct engine access, which is unavailable in external mode "
|
||||
+ "(the test-support proxy exposes only read-only introspection).");
|
||||
}
|
||||
}
|
||||
|
||||
private JsonNode execute(final HttpRequest request) {
|
||||
try {
|
||||
final HttpResponse<String> response =
|
||||
http.send(request, HttpResponse.BodyHandlers.ofString());
|
||||
if (response.statusCode() >= 400) {
|
||||
throw new SearchClientException(
|
||||
"HTTP " + response.statusCode() + " from " + request.uri() + ": " + response.body());
|
||||
}
|
||||
return MAPPER.readTree(response.body());
|
||||
} catch (final InterruptedException e) {
|
||||
Thread.currentThread().interrupt();
|
||||
throw new SearchClientException(request.method() + " " + request.uri() + " interrupted", e);
|
||||
} catch (final IOException e) {
|
||||
throw new SearchClientException(request.method() + " " + request.uri() + " failed", e);
|
||||
}
|
||||
}
|
||||
|
||||
public static final class SearchClientException extends RuntimeException {
|
||||
public SearchClientException(final String message) {
|
||||
super(message);
|
||||
}
|
||||
|
||||
public SearchClientException(final String message, final Throwable cause) {
|
||||
super(message, cause);
|
||||
}
|
||||
}
|
||||
}
|
||||
+57
@@ -0,0 +1,57 @@
|
||||
package org.openmetadata.it.search;
|
||||
|
||||
import java.time.Duration;
|
||||
import org.junit.jupiter.api.extension.BeforeEachCallback;
|
||||
import org.junit.jupiter.api.extension.ExtensionContext;
|
||||
import org.openmetadata.it.server.ServerHandle;
|
||||
import org.openmetadata.it.util.OssTestServer;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
/**
|
||||
* Restores the shared embedded search cluster to a known-good baseline before each
|
||||
* server-global search IT.
|
||||
*
|
||||
* <p>The {@code tests/search/*IT} classes mutate cluster-wide state — full reindex with
|
||||
* {@code recreateIndex=true}, alias swaps, and pausing the engine container via {@link
|
||||
* EsOutageInjector}. They run serially in the {@code search-it} profile against one reused
|
||||
* server, so a test that drops indices or leaves the engine paused would otherwise cascade
|
||||
* {@code index_not_found} / read-timeout failures across every test that follows. This
|
||||
* extension makes each test order-independent by re-establishing the baseline up front:
|
||||
* resume the engine if it was paused, and rebuild all indices if they are missing.
|
||||
*/
|
||||
public final class SearchClusterResetExtension implements BeforeEachCallback {
|
||||
|
||||
private static final Logger LOG = LoggerFactory.getLogger(SearchClusterResetExtension.class);
|
||||
private static final Duration REBUILD_TIMEOUT = ReindexHelpers.reindexTimeout();
|
||||
|
||||
@Override
|
||||
public void beforeEach(final ExtensionContext context) {
|
||||
final ServerHandle server = OssTestServer.defaultHandle();
|
||||
resumeEngineIfPaused();
|
||||
rebuildBaseline(server);
|
||||
}
|
||||
|
||||
private void resumeEngineIfPaused() {
|
||||
try {
|
||||
EsOutageInjector.unpause();
|
||||
} catch (final RuntimeException e) {
|
||||
LOG.debug("Search engine resume was a no-op (not paused / not embedded): {}", e.toString());
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Always recreate every index up front. A presence probe is unreliable here — a prior test can
|
||||
* leave an alias pointing at a dropped/staged index (still "present" but unqueryable), and a
|
||||
* non-recreate reindex won't rebuild a missing index or re-promote a swapped alias. A full
|
||||
* recreate restores a clean, queryable baseline regardless of the prior test's end state.
|
||||
*
|
||||
* <p>{@link ReindexHelpers#recreateAllAndWait} throws if the baseline never succeeds, so a
|
||||
* still-broken cluster fails the test fast with a clear cause rather than as a misleading
|
||||
* per-test assertion failure.
|
||||
*/
|
||||
private void rebuildBaseline(final ServerHandle server) {
|
||||
LOG.info("Recreating all search indices to restore a clean baseline before test");
|
||||
ReindexHelpers.recreateAllAndWait(server, REBUILD_TIMEOUT);
|
||||
}
|
||||
}
|
||||
+145
@@ -0,0 +1,145 @@
|
||||
package org.openmetadata.it.search;
|
||||
|
||||
import java.time.Duration;
|
||||
import java.util.ArrayList;
|
||||
import java.util.HashSet;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import org.openmetadata.it.server.ServerHandle;
|
||||
import org.openmetadata.sdk.exceptions.OpenMetadataException;
|
||||
import org.openmetadata.sdk.network.HttpClient;
|
||||
import org.openmetadata.sdk.network.HttpMethod;
|
||||
|
||||
/**
|
||||
* Hits OM's {@code /v1/search/query} endpoint — the same path the Explore page uses —
|
||||
* and returns the entity IDs in the result set. Designed for tests that need to probe
|
||||
* search availability mid-reindex (zero-downtime, no-duplicates) without going through a
|
||||
* browser for each query.
|
||||
*
|
||||
* <p>Response parsing reads {@code hits.hits[]._id} via {@link Map} navigation rather than
|
||||
* binding to a typed schema — search responses are loosely typed and we only need ID
|
||||
* counts and uniqueness for these probes.
|
||||
*/
|
||||
public final class SearchQueryHelper {
|
||||
|
||||
private SearchQueryHelper() {}
|
||||
|
||||
/**
|
||||
* Queries the given index alias and returns every {@code _id} in the response. Uses a
|
||||
* page size large enough to fetch the full result set in one call — callers should
|
||||
* ensure {@code size} exceeds the expected entity count.
|
||||
*/
|
||||
public static SearchProbe probeIndex(
|
||||
final ServerHandle server, final String indexAlias, final int size) {
|
||||
return probeIndex(server, indexAlias, "*", size);
|
||||
}
|
||||
|
||||
@SuppressWarnings("unchecked")
|
||||
public static SearchProbe probeIndex(
|
||||
final ServerHandle server, final String indexAlias, final String query, final int size) {
|
||||
final HttpClient http = server.sdk().getHttpClient();
|
||||
final String path =
|
||||
String.format(
|
||||
"/v1/search/query?q=%s&index=%s&from=0&size=%d&deleted=false&track_total_hits=true",
|
||||
urlEncode(query), urlEncode(indexAlias), size);
|
||||
final Map<String, Object> response = http.execute(HttpMethod.GET, path, null, Map.class);
|
||||
return parse(response);
|
||||
}
|
||||
|
||||
/**
|
||||
* Like {@link #probeIndex} but rides out transient post-swap unavailability. Right after
|
||||
* a recreate-mode reindex swaps an alias onto a freshly-created index, that index's shards
|
||||
* may still be initialising, so OpenSearch answers {@code 503 "all shards failed"} (empty
|
||||
* {@code failed_shards}) for a beat. That's shard-allocation lag, not a search blackout —
|
||||
* retry within {@code budget} before surfacing it. A genuine sustained outage still
|
||||
* propagates once the budget is exhausted.
|
||||
*/
|
||||
public static SearchProbe probeIndexToleratingShardLag(
|
||||
final ServerHandle server, final String indexAlias, final int size, final Duration budget) {
|
||||
final long deadlineNanos = System.nanoTime() + budget.toNanos();
|
||||
while (true) {
|
||||
try {
|
||||
return probeIndex(server, indexAlias, size);
|
||||
} catch (final OpenMetadataException e) {
|
||||
if (e.getStatusCode() < 500 || System.nanoTime() >= deadlineNanos) {
|
||||
throw e;
|
||||
}
|
||||
sleepBriefly();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private static void sleepBriefly() {
|
||||
try {
|
||||
Thread.sleep(500);
|
||||
} catch (final InterruptedException ie) {
|
||||
Thread.currentThread().interrupt();
|
||||
throw new IllegalStateException("Interrupted while waiting out transient search 503", ie);
|
||||
}
|
||||
}
|
||||
|
||||
private static String urlEncode(final String value) {
|
||||
return java.net.URLEncoder.encode(value, java.nio.charset.StandardCharsets.UTF_8);
|
||||
}
|
||||
|
||||
@SuppressWarnings("unchecked")
|
||||
private static SearchProbe parse(final Map<String, Object> response) {
|
||||
final List<String> ids = new ArrayList<>();
|
||||
final Set<String> seen = new HashSet<>();
|
||||
long totalHits = 0;
|
||||
final Map<String, Object> outerHits = (Map<String, Object>) response.get("hits");
|
||||
if (outerHits == null) {
|
||||
return new SearchProbe(ids, seen, 0);
|
||||
}
|
||||
final Object total = outerHits.get("total");
|
||||
if (total instanceof Map) {
|
||||
final Object value = ((Map<String, Object>) total).get("value");
|
||||
if (value instanceof Number) {
|
||||
totalHits = ((Number) value).longValue();
|
||||
}
|
||||
} else if (total instanceof Number) {
|
||||
totalHits = ((Number) total).longValue();
|
||||
}
|
||||
final List<Map<String, Object>> innerHits = (List<Map<String, Object>>) outerHits.get("hits");
|
||||
if (innerHits != null) {
|
||||
for (Map<String, Object> hit : innerHits) {
|
||||
final Object idObj = hit.get("_id");
|
||||
if (idObj != null) {
|
||||
final String id = idObj.toString();
|
||||
ids.add(id);
|
||||
seen.add(id);
|
||||
}
|
||||
}
|
||||
}
|
||||
return new SearchProbe(ids, seen, totalHits);
|
||||
}
|
||||
|
||||
/**
|
||||
* Result of a single search probe: every hit ID, the deduplicated set, and the true
|
||||
* total from the response's {@code hits.total.value} (which can exceed {@code ids.size}
|
||||
* when the response is paginated by the {@code size} param).
|
||||
*
|
||||
* <p>Defensive copies on construction and accessors keep the internal collections
|
||||
* immutable so callers can't mutate one probe's state and pollute another's assertions.
|
||||
*/
|
||||
public record SearchProbe(List<String> ids, Set<String> uniqueIds, long totalHits) {
|
||||
|
||||
public SearchProbe {
|
||||
ids = List.copyOf(ids);
|
||||
uniqueIds = Set.copyOf(uniqueIds);
|
||||
}
|
||||
|
||||
public int total() {
|
||||
return ids.size();
|
||||
}
|
||||
|
||||
public int unique() {
|
||||
return uniqueIds.size();
|
||||
}
|
||||
|
||||
public boolean hasDuplicates() {
|
||||
return total() != unique();
|
||||
}
|
||||
}
|
||||
}
|
||||
+299
@@ -0,0 +1,299 @@
|
||||
package org.openmetadata.it.search;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.node.ObjectNode;
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import org.openmetadata.it.server.ServerHandle;
|
||||
import org.openmetadata.schema.api.search.AssetTypeConfiguration;
|
||||
import org.openmetadata.schema.api.search.FieldBoost;
|
||||
import org.openmetadata.schema.api.search.FieldValueBoost;
|
||||
import org.openmetadata.schema.api.search.GlobalSettings;
|
||||
import org.openmetadata.schema.api.search.SearchSettings;
|
||||
import org.openmetadata.schema.api.search.TermBoost;
|
||||
import org.openmetadata.schema.search.PreviewSearchRequest;
|
||||
import org.openmetadata.schema.settings.Settings;
|
||||
import org.openmetadata.schema.settings.SettingsType;
|
||||
import org.openmetadata.schema.utils.JsonUtils;
|
||||
import org.openmetadata.sdk.network.HttpMethod;
|
||||
import org.openmetadata.service.resources.settings.SettingsCache;
|
||||
|
||||
/**
|
||||
* Reads, mutates, previews, and restores the global SearchSettings (the Settings > Search config)
|
||||
* so relevancy tests can assert how a settings change reshapes ranking.
|
||||
*
|
||||
* <p>Two paths, mirroring the product:
|
||||
*
|
||||
* <ul>
|
||||
* <li><b>Preview</b> ({@link #previewIds}/{@link #previewScores}) — POSTs inline settings to
|
||||
* {@code /v1/search/preview}; runs the exact production query builder but persists nothing and
|
||||
* bypasses the settings cache, so a test can score the same indexed cohort under two settings
|
||||
* with no global mutation. This is the default for ranking assertions.
|
||||
* <li><b>Live</b> ({@link #putSettings}/{@link #resetSettings}) — the real Settings > Search
|
||||
* save path; subsequent {@code /v1/search/query} calls observe the change. Always restore with
|
||||
* {@link #resetSettings} in a finally block.
|
||||
* </ul>
|
||||
*
|
||||
* <p>Preview/search take the <em>logical</em> index name (e.g. {@code "table"}); the server resolves
|
||||
* the cluster-aliased physical index, exactly as the UI search box does.
|
||||
*/
|
||||
public final class SearchSettingsTestHelper {
|
||||
|
||||
private static final String SETTINGS_PATH = "/v1/system/settings";
|
||||
private static final String SEARCH_SETTINGS_PATH = SETTINGS_PATH + "/searchSettings";
|
||||
private static final String RESET_SEARCH_SETTINGS_PATH = SETTINGS_PATH + "/reset/searchSettings";
|
||||
private static final String PREVIEW_PATH = "/v1/search/preview";
|
||||
|
||||
private SearchSettingsTestHelper() {}
|
||||
|
||||
/** The currently persisted SearchSettings (Settings > Search). */
|
||||
public static SearchSettings currentSettings(final ServerHandle server) {
|
||||
final String json =
|
||||
server.sdk().getHttpClient().executeForString(HttpMethod.GET, SEARCH_SETTINGS_PATH, null);
|
||||
final Settings envelope = JsonUtils.readValue(json, Settings.class);
|
||||
return JsonUtils.convertValue(envelope.getConfigValue(), SearchSettings.class);
|
||||
}
|
||||
|
||||
/** A deep copy, so a test can build a variant without mutating the shared baseline. */
|
||||
public static SearchSettings copyOf(final SearchSettings settings) {
|
||||
return JsonUtils.readValue(JsonUtils.pojoToJson(settings), SearchSettings.class);
|
||||
}
|
||||
|
||||
/** Persists settings via the real save path and invalidates the in-JVM cache (embedded mode). */
|
||||
public static void putSettings(final ServerHandle server, final SearchSettings settings) {
|
||||
final Settings envelope =
|
||||
new Settings().withConfigType(SettingsType.SEARCH_SETTINGS).withConfigValue(settings);
|
||||
server.sdk().getHttpClient().executeForString(HttpMethod.PUT, SETTINGS_PATH, envelope);
|
||||
invalidateEmbeddedCache(server);
|
||||
}
|
||||
|
||||
/** Restores the seeded defaults — call in a finally block after any {@link #putSettings}. */
|
||||
public static void resetSettings(final ServerHandle server) {
|
||||
server.sdk().getHttpClient().executeForString(HttpMethod.PUT, RESET_SEARCH_SETTINGS_PATH, null);
|
||||
invalidateEmbeddedCache(server);
|
||||
}
|
||||
|
||||
private static void invalidateEmbeddedCache(final ServerHandle server) {
|
||||
if (!server.isExternal()) {
|
||||
SettingsCache.invalidateSettings(SettingsType.SEARCH_SETTINGS.value());
|
||||
}
|
||||
}
|
||||
|
||||
/** Hit IDs in ranked order for a preview run with the given inline settings. */
|
||||
public static List<String> previewIds(
|
||||
final ServerHandle server,
|
||||
final String query,
|
||||
final String index,
|
||||
final SearchSettings settings,
|
||||
final int size) {
|
||||
final List<String> ids = new ArrayList<>();
|
||||
preview(server, query, index, settings, size, false)
|
||||
.path("hits")
|
||||
.path("hits")
|
||||
.forEach(hit -> ids.add(hit.path("_id").asText()));
|
||||
return ids;
|
||||
}
|
||||
|
||||
/** Per-hit {@code _score} keyed by id, preserving ranked order. */
|
||||
public static Map<String, Double> previewScores(
|
||||
final ServerHandle server,
|
||||
final String query,
|
||||
final String index,
|
||||
final SearchSettings settings,
|
||||
final int size) {
|
||||
final Map<String, Double> scores = new LinkedHashMap<>();
|
||||
preview(server, query, index, settings, size, false)
|
||||
.path("hits")
|
||||
.path("hits")
|
||||
.forEach(hit -> scores.put(hit.path("_id").asText(), hit.path("_score").asDouble()));
|
||||
return scores;
|
||||
}
|
||||
|
||||
/** Raw engine response for a preview run; pass {@code explain=true} to inspect scoring. */
|
||||
public static JsonNode preview(
|
||||
final ServerHandle server,
|
||||
final String query,
|
||||
final String index,
|
||||
final SearchSettings settings,
|
||||
final int size,
|
||||
final boolean explain) {
|
||||
final PreviewSearchRequest request =
|
||||
new PreviewSearchRequest()
|
||||
.withQuery(query)
|
||||
.withIndex(index)
|
||||
.withSearchSettings(settings)
|
||||
.withFrom(0)
|
||||
.withSize(size)
|
||||
.withTrackTotalHits(true)
|
||||
.withFetchSource(false)
|
||||
.withExplain(explain);
|
||||
final String response =
|
||||
server.sdk().getHttpClient().executeForString(HttpMethod.POST, PREVIEW_PATH, request);
|
||||
return JsonUtils.readTree(response);
|
||||
}
|
||||
|
||||
/** The asset-type block within settings (e.g. {@code "table"}); throws if absent. */
|
||||
public static AssetTypeConfiguration assetConfig(
|
||||
final SearchSettings settings, final String assetType) {
|
||||
return settings.getAssetTypeConfigurations().stream()
|
||||
.filter(config -> assetType.equalsIgnoreCase(config.getAssetType()))
|
||||
.findFirst()
|
||||
.orElseThrow(
|
||||
() -> new IllegalArgumentException("No asset config for assetType: " + assetType));
|
||||
}
|
||||
|
||||
/** Adds a global field=value term boost (applies to every index). */
|
||||
public static void addGlobalTermBoost(
|
||||
final SearchSettings settings, final String field, final String value, final double boost) {
|
||||
termBoosts(settings.getGlobalSettings()).add(termBoost(field, value, boost));
|
||||
}
|
||||
|
||||
/** Adds a per-asset field=value term boost (stacks on top of any global term boosts). */
|
||||
public static void addAssetTermBoost(
|
||||
final SearchSettings settings,
|
||||
final String assetType,
|
||||
final String field,
|
||||
final String value,
|
||||
final double boost) {
|
||||
final AssetTypeConfiguration config = assetConfig(settings, assetType);
|
||||
if (config.getTermBoosts() == null) {
|
||||
config.setTermBoosts(new ArrayList<>());
|
||||
}
|
||||
config.getTermBoosts().add(termBoost(field, value, boost));
|
||||
}
|
||||
|
||||
/** Adds a global numeric field-value boost (applies to every index). */
|
||||
public static void addGlobalFieldValueBoost(
|
||||
final SearchSettings settings, final String field, final double factor) {
|
||||
fieldValueBoosts(settings.getGlobalSettings()).add(fieldValueBoost(field, factor));
|
||||
}
|
||||
|
||||
/** Adds a per-asset numeric field-value boost, optionally with a modifier and missing default. */
|
||||
public static void addAssetFieldValueBoost(
|
||||
final SearchSettings settings,
|
||||
final String assetType,
|
||||
final String field,
|
||||
final double factor,
|
||||
final FieldValueBoost.Modifier modifier,
|
||||
final Double missing) {
|
||||
final FieldValueBoost boost = new FieldValueBoost().withField(field).withFactor(factor);
|
||||
if (modifier != null) {
|
||||
boost.withModifier(modifier);
|
||||
}
|
||||
if (missing != null) {
|
||||
boost.withMissing(missing);
|
||||
}
|
||||
final AssetTypeConfiguration config = assetConfig(settings, assetType);
|
||||
if (config.getFieldValueBoosts() == null) {
|
||||
config.setFieldValueBoosts(new ArrayList<>());
|
||||
}
|
||||
config.getFieldValueBoosts().add(boost);
|
||||
}
|
||||
|
||||
/** Sets how an asset type combines multiple boost functions (sum, max, …). */
|
||||
public static void setScoreMode(
|
||||
final SearchSettings settings,
|
||||
final String assetType,
|
||||
final AssetTypeConfiguration.ScoreMode scoreMode) {
|
||||
assetConfig(settings, assetType).setScoreMode(scoreMode);
|
||||
}
|
||||
|
||||
/** Sets how an asset type merges the boost score with the text-match score (multiply, replace…). */
|
||||
public static void setBoostMode(
|
||||
final SearchSettings settings,
|
||||
final String assetType,
|
||||
final AssetTypeConfiguration.BoostMode boostMode) {
|
||||
assetConfig(settings, assetType).setBoostMode(boostMode);
|
||||
}
|
||||
|
||||
public static SearchSettings withRankingDisabled(
|
||||
final SearchSettings settings, final String assetType) {
|
||||
final JsonNode root = JsonUtils.readTree(JsonUtils.pojoToJson(settings));
|
||||
disableRankingNode(root.path("defaultConfiguration"));
|
||||
|
||||
for (JsonNode config : root.path("assetTypeConfigurations")) {
|
||||
if (assetType.equalsIgnoreCase(config.path("assetType").asText())) {
|
||||
disableRankingNode(config);
|
||||
}
|
||||
}
|
||||
|
||||
return JsonUtils.treeToValue(root, SearchSettings.class);
|
||||
}
|
||||
|
||||
private static void disableRankingNode(final JsonNode config) {
|
||||
if (config instanceof ObjectNode configNode
|
||||
&& configNode.path("ranking") instanceof ObjectNode rankingNode) {
|
||||
rankingNode.put("enabled", false);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Clears all global and per-asset term/field-value boosts so a scoring test starts from a clean
|
||||
* function-score baseline and only the boosts it adds will fire.
|
||||
*/
|
||||
public static void clearBoosts(final SearchSettings settings, final String assetType) {
|
||||
settings.getGlobalSettings().setTermBoosts(new ArrayList<>());
|
||||
settings.getGlobalSettings().setFieldValueBoosts(new ArrayList<>());
|
||||
final AssetTypeConfiguration config = assetConfig(settings, assetType);
|
||||
config.setTermBoosts(new ArrayList<>());
|
||||
config.setFieldValueBoosts(new ArrayList<>());
|
||||
}
|
||||
|
||||
/** Replaces an asset type's searchable fields with a single boosted, standard-match field. */
|
||||
public static void setOnlySearchField(
|
||||
final SearchSettings settings,
|
||||
final String assetType,
|
||||
final String field,
|
||||
final double boost) {
|
||||
setOnlySearchField(settings, assetType, field, boost, FieldBoost.MatchType.STANDARD);
|
||||
}
|
||||
|
||||
/** Replaces an asset type's searchable fields with a single field using the given match type. */
|
||||
public static void setOnlySearchField(
|
||||
final SearchSettings settings,
|
||||
final String assetType,
|
||||
final String field,
|
||||
final double boost,
|
||||
final FieldBoost.MatchType matchType) {
|
||||
assetConfig(settings, assetType)
|
||||
.setSearchFields(
|
||||
new ArrayList<>(
|
||||
List.of(
|
||||
new FieldBoost().withField(field).withBoost(boost).withMatchType(matchType))));
|
||||
}
|
||||
|
||||
/** Sets the fields an asset type highlights (overrides the global highlight list for that type). */
|
||||
public static void setAssetHighlightFields(
|
||||
final SearchSettings settings, final String assetType, final String... fields) {
|
||||
assetConfig(settings, assetType).setHighlightFields(new ArrayList<>(List.of(fields)));
|
||||
}
|
||||
|
||||
/** Caps how many hits any search returns, regardless of the requested size. */
|
||||
public static void setMaxResultHits(final SearchSettings settings, final int maxResultHits) {
|
||||
settings.getGlobalSettings().setMaxResultHits(maxResultHits);
|
||||
}
|
||||
|
||||
private static List<TermBoost> termBoosts(final GlobalSettings global) {
|
||||
if (global.getTermBoosts() == null) {
|
||||
global.setTermBoosts(new ArrayList<>());
|
||||
}
|
||||
return global.getTermBoosts();
|
||||
}
|
||||
|
||||
private static List<FieldValueBoost> fieldValueBoosts(final GlobalSettings global) {
|
||||
if (global.getFieldValueBoosts() == null) {
|
||||
global.setFieldValueBoosts(new ArrayList<>());
|
||||
}
|
||||
return global.getFieldValueBoosts();
|
||||
}
|
||||
|
||||
private static TermBoost termBoost(final String field, final String value, final double boost) {
|
||||
return new TermBoost().withField(field).withValue(value).withBoost(boost);
|
||||
}
|
||||
|
||||
private static FieldValueBoost fieldValueBoost(final String field, final double factor) {
|
||||
return new FieldValueBoost().withField(field).withFactor(factor);
|
||||
}
|
||||
}
|
||||
+91
@@ -0,0 +1,91 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
import org.openmetadata.schema.service.configuration.elasticsearch.ElasticSearchConfiguration.SearchType;
|
||||
|
||||
/**
|
||||
* Opt-in list of search-index shapes whose non-OK outcome is consciously accepted. A case listed
|
||||
* here is tolerated (the canary stays green and logs the reason); ANY case NOT listed must index
|
||||
* and be queryable (Outcome.OK) or {@link org.openmetadata.it.tests.EntityShapeIT} fails red.
|
||||
*
|
||||
* <p>Add an entry to accept a limit (e.g. "1M columns may fail"); remove it (or fix the root cause)
|
||||
* to make the case red again. Granularity is per (engine, entityType, dimension, rung). Use {@link
|
||||
* #ALL_ENTITIES} as the entityType for a limit that is inherent to the engine (not one entity's
|
||||
* mapping) and therefore universal across every indexed type, and {@link #ANY_ENGINE} unless the
|
||||
* limit only trips on one engine.
|
||||
*/
|
||||
public final class AcceptedLimits {
|
||||
|
||||
/**
|
||||
* Wildcard entityType. An {@link Accepted} declared with this matches the given dimension+rung for
|
||||
* every entity type — use it for inherent ES/OS limits rather than repeating the same entry once
|
||||
* per entity.
|
||||
*/
|
||||
public static final String ALL_ENTITIES = "*";
|
||||
|
||||
/**
|
||||
* Sentinel engine: an {@link Accepted} declared with this applies regardless of the active search
|
||||
* engine. Use a concrete {@link SearchType} only when a limit is engine-specific — e.g. the 10MB
|
||||
* {@code http.max_content_length} on AWS-managed OpenSearch that rejects the 16MB size cases,
|
||||
* which self-hosted Elasticsearch (100MB default) indexes fine.
|
||||
*/
|
||||
public static final SearchType ANY_ENGINE = null;
|
||||
|
||||
public record Accepted(
|
||||
SearchType engine,
|
||||
String entityType,
|
||||
String dimension,
|
||||
String rung,
|
||||
Outcome outcome,
|
||||
String reason) {}
|
||||
|
||||
private static final List<Accepted> ACCEPTED =
|
||||
List.of(
|
||||
new Accepted(
|
||||
ANY_ENGINE,
|
||||
ALL_ENTITIES,
|
||||
"owners.count",
|
||||
"aboveNestedLimit",
|
||||
Outcome.REJECTED,
|
||||
"owners is a nested field; ramping past the server-controlled "
|
||||
+ "index.mapping.nested_objects.limit (SearchFieldLimits.getNestedObjectsLimit) is "
|
||||
+ "rejected by the engine. The rung magnitude is derived from that limit, so this "
|
||||
+ "tracks the server config. No real entity has that many owners."),
|
||||
new Accepted(
|
||||
ANY_ENGINE,
|
||||
ALL_ENTITIES,
|
||||
"keyword.overIgnoreAbove",
|
||||
"300chars",
|
||||
Outcome.DEGRADED_UNSEARCHABLE,
|
||||
"displayName.keyword sets ignore_above:256 in the served index mapping (a per-field "
|
||||
+ "cap harden() preserves — not one of the tunable SearchFieldLimits). A 300-char "
|
||||
+ "value stays in _source and full-text searchable, but is dropped from the keyword "
|
||||
+ "term index so exact-match on displayName.keyword misses."));
|
||||
|
||||
private AcceptedLimits() {}
|
||||
|
||||
public static Optional<Accepted> find(
|
||||
final SearchType engine, final String entityType, final String dimension, final String rung) {
|
||||
return ACCEPTED.stream()
|
||||
.filter(
|
||||
a ->
|
||||
(a.engine() == ANY_ENGINE || a.engine() == engine)
|
||||
&& (a.entityType().equals(ALL_ENTITIES) || a.entityType().equals(entityType))
|
||||
&& a.dimension().equals(dimension)
|
||||
&& a.rung().equals(rung))
|
||||
.findFirst();
|
||||
}
|
||||
}
|
||||
+49
@@ -0,0 +1,49 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.function.BiFunction;
|
||||
import java.util.function.Function;
|
||||
import org.openmetadata.schema.EntityInterface;
|
||||
|
||||
public final class EntityCases {
|
||||
private final String entityType;
|
||||
private final Function<ShapeContext, EntityInterface> minimal;
|
||||
private final ShapeContext ctx;
|
||||
private final List<PlannedCase> cases = new ArrayList<>();
|
||||
|
||||
public EntityCases(
|
||||
final String entityType,
|
||||
final Function<ShapeContext, EntityInterface> minimal,
|
||||
final ShapeContext ctx) {
|
||||
this.entityType = entityType;
|
||||
this.minimal = minimal;
|
||||
this.ctx = ctx;
|
||||
}
|
||||
|
||||
public EntityCases add(
|
||||
final String dimension,
|
||||
final Rung rung,
|
||||
final BiFunction<EntityInterface, Rung, EntityInterface> apply) {
|
||||
cases.add(
|
||||
new PlannedCase(
|
||||
entityType, dimension, rung, () -> apply.apply(minimal.apply(ctx), rung), null));
|
||||
return this;
|
||||
}
|
||||
|
||||
public List<PlannedCase> build() {
|
||||
return List.copyOf(cases);
|
||||
}
|
||||
}
|
||||
+26
@@ -0,0 +1,26 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape;
|
||||
|
||||
import java.util.List;
|
||||
import org.openmetadata.schema.EntityInterface;
|
||||
|
||||
public interface EntityShapeProfile {
|
||||
String entityType();
|
||||
|
||||
EntityInterface minimal(ShapeContext ctx);
|
||||
|
||||
default List<PlannedCase> entitySpecificCases(ShapeContext ctx) {
|
||||
return List.of();
|
||||
}
|
||||
}
|
||||
+111
@@ -0,0 +1,111 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import org.openmetadata.it.search.shape.mutations.CustomPropertiesBreadthMutation;
|
||||
import org.openmetadata.it.search.shape.mutations.DescriptionSizeMutation;
|
||||
import org.openmetadata.it.search.shape.mutations.FollowersCountMutation;
|
||||
import org.openmetadata.it.search.shape.mutations.KeywordIgnoreAboveMutation;
|
||||
import org.openmetadata.it.search.shape.mutations.OwnersCountMutation;
|
||||
import org.openmetadata.it.search.shape.mutations.TagsCountMutation;
|
||||
import org.openmetadata.it.search.shape.profiles.ApiCollectionShapeProfile;
|
||||
import org.openmetadata.it.search.shape.profiles.ApiEndpointShapeProfile;
|
||||
import org.openmetadata.it.search.shape.profiles.ChartShapeProfile;
|
||||
import org.openmetadata.it.search.shape.profiles.ContainerShapeProfile;
|
||||
import org.openmetadata.it.search.shape.profiles.DashboardDataModelShapeProfile;
|
||||
import org.openmetadata.it.search.shape.profiles.DashboardShapeProfile;
|
||||
import org.openmetadata.it.search.shape.profiles.DataProductShapeProfile;
|
||||
import org.openmetadata.it.search.shape.profiles.DatabaseSchemaShapeProfile;
|
||||
import org.openmetadata.it.search.shape.profiles.DatabaseShapeProfile;
|
||||
import org.openmetadata.it.search.shape.profiles.DomainShapeProfile;
|
||||
import org.openmetadata.it.search.shape.profiles.GlossaryShapeProfile;
|
||||
import org.openmetadata.it.search.shape.profiles.GlossaryTermShapeProfile;
|
||||
import org.openmetadata.it.search.shape.profiles.MetricShapeProfile;
|
||||
import org.openmetadata.it.search.shape.profiles.MlModelShapeProfile;
|
||||
import org.openmetadata.it.search.shape.profiles.PipelineShapeProfile;
|
||||
import org.openmetadata.it.search.shape.profiles.QueryShapeProfile;
|
||||
import org.openmetadata.it.search.shape.profiles.SearchIndexShapeProfile;
|
||||
import org.openmetadata.it.search.shape.profiles.StoredProcedureShapeProfile;
|
||||
import org.openmetadata.it.search.shape.profiles.TableShapeProfile;
|
||||
import org.openmetadata.it.search.shape.profiles.TopicShapeProfile;
|
||||
|
||||
public final class EntityShapeRegistry {
|
||||
private final List<EntityShapeProfile> profiles =
|
||||
List.of(
|
||||
new TableShapeProfile(),
|
||||
new ContainerShapeProfile(),
|
||||
new DashboardShapeProfile(),
|
||||
new TopicShapeProfile(),
|
||||
new GlossaryTermShapeProfile(),
|
||||
new QueryShapeProfile(),
|
||||
new StoredProcedureShapeProfile(),
|
||||
new MetricShapeProfile(),
|
||||
new DashboardDataModelShapeProfile(),
|
||||
new PipelineShapeProfile(),
|
||||
new MlModelShapeProfile(),
|
||||
new SearchIndexShapeProfile(),
|
||||
new ApiEndpointShapeProfile(),
|
||||
new DatabaseShapeProfile(),
|
||||
new DatabaseSchemaShapeProfile(),
|
||||
new ChartShapeProfile(),
|
||||
new DataProductShapeProfile(),
|
||||
new DomainShapeProfile(),
|
||||
new GlossaryShapeProfile(),
|
||||
new ApiCollectionShapeProfile());
|
||||
|
||||
private final List<ShapeMutation> sharedMutations =
|
||||
List.of(
|
||||
new DescriptionSizeMutation(),
|
||||
new TagsCountMutation(),
|
||||
new OwnersCountMutation(),
|
||||
new FollowersCountMutation(),
|
||||
new CustomPropertiesBreadthMutation(),
|
||||
new KeywordIgnoreAboveMutation());
|
||||
|
||||
public List<PlannedCase> plannedCases() {
|
||||
final List<PlannedCase> cases = new ArrayList<>();
|
||||
final ShapeContext ctx = new ShapeContext();
|
||||
for (final EntityShapeProfile profile : profiles) {
|
||||
addSharedCases(cases, ctx, profile);
|
||||
cases.addAll(profile.entitySpecificCases(ctx));
|
||||
}
|
||||
return List.copyOf(cases);
|
||||
}
|
||||
|
||||
private void addSharedCases(
|
||||
final List<PlannedCase> cases, final ShapeContext ctx, final EntityShapeProfile profile) {
|
||||
for (final ShapeMutation mutation : sharedMutations) {
|
||||
if (mutation.appliesTo(profile.minimal(ctx))) {
|
||||
addLadder(cases, ctx, profile, mutation);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private void addLadder(
|
||||
final List<PlannedCase> cases,
|
||||
final ShapeContext ctx,
|
||||
final EntityShapeProfile profile,
|
||||
final ShapeMutation mutation) {
|
||||
for (final Rung rung : mutation.ladder()) {
|
||||
cases.add(
|
||||
new PlannedCase(
|
||||
profile.entityType(),
|
||||
mutation.dimension(),
|
||||
rung,
|
||||
() -> mutation.apply(profile.minimal(ctx), rung),
|
||||
mutation.probe(rung)));
|
||||
}
|
||||
}
|
||||
}
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape;
|
||||
|
||||
import org.openmetadata.it.search.SearchClient;
|
||||
|
||||
@FunctionalInterface
|
||||
public interface FieldProbe {
|
||||
boolean searchable(SearchClient httpSearch, String indexName);
|
||||
}
|
||||
+27
@@ -0,0 +1,27 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape;
|
||||
|
||||
public enum Outcome {
|
||||
/** Indexed, retrievable, and (when probed) the ramped value is searchable. */
|
||||
OK,
|
||||
/** Indexed and present in _source, but the value was dropped from the term index (e.g. a keyword
|
||||
* field's {@code ignore_above}), so it cannot be found by an exact-match query on that field. */
|
||||
DEGRADED_UNSEARCHABLE,
|
||||
/** The index refused the document: the write (PUT) failed and nothing was indexed. The cause is
|
||||
* NOT classified here — see {@link ShapeResult#detail()} for the raw engine error, which may be a
|
||||
* size / total_fields / nested_objects / depth / parse limit or anything else. */
|
||||
REJECTED,
|
||||
/** Unexpected: the PUT succeeded but the document was not retrievable by id afterwards. */
|
||||
ERROR_OTHER
|
||||
}
|
||||
+33
@@ -0,0 +1,33 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape;
|
||||
|
||||
import java.util.function.Supplier;
|
||||
import org.openmetadata.schema.EntityInterface;
|
||||
|
||||
public record PlannedCase(
|
||||
String entityType,
|
||||
String dimension,
|
||||
Rung rung,
|
||||
Supplier<EntityInterface> entity,
|
||||
FieldProbe probe) {
|
||||
|
||||
public String label() {
|
||||
return entityType + "/" + dimension + "/" + rung.label();
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return label();
|
||||
}
|
||||
}
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape;
|
||||
|
||||
public record Rung(String label, int magnitude) {
|
||||
public static Rung of(final String label, final int magnitude) {
|
||||
return new Rung(label, magnitude);
|
||||
}
|
||||
}
|
||||
+117
@@ -0,0 +1,117 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.node.ObjectNode;
|
||||
import java.util.Iterator;
|
||||
import java.util.Locale;
|
||||
import java.util.UUID;
|
||||
import org.openmetadata.it.search.SearchClient;
|
||||
import org.openmetadata.schema.utils.JsonUtils;
|
||||
import org.openmetadata.service.search.SearchRepository;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
/**
|
||||
* Provisions a throwaway "shadow" index cloned from an entity's real mapping so each canary case
|
||||
* runs in isolation.
|
||||
*
|
||||
* <p>The shared {@code <entity>_search_index} accumulates dynamic mapping fields (e.g. from
|
||||
* {@code customProperties.breadth}) that a doc-delete cleanup cannot remove, which pollutes the
|
||||
* index and makes outcomes order-dependent across ITs. Cloning the real mapping into a uniquely
|
||||
* named index per case, then dropping it, keeps each measurement deterministic.
|
||||
*/
|
||||
public final class ShadowIndex {
|
||||
private static final Logger LOG = LoggerFactory.getLogger(ShadowIndex.class);
|
||||
|
||||
// mirrors the value in *_index_mapping.json
|
||||
private static final int DEFAULT_MAX_NGRAM_DIFF = 17;
|
||||
|
||||
private static final String SHADOW_SUFFIX = "_sc_";
|
||||
private static final String SETTINGS = "settings";
|
||||
private static final String INDEX = "index";
|
||||
private static final String MAPPINGS = "mappings";
|
||||
private static final String ANALYSIS = "analysis";
|
||||
private static final String MAPPING = "mapping";
|
||||
private static final String MAX_NGRAM_DIFF = "max_ngram_diff";
|
||||
private static final String NUMBER_OF_REPLICAS = "number_of_replicas";
|
||||
|
||||
private final SearchRepository searchRepository;
|
||||
private final SearchClient httpSearch;
|
||||
|
||||
public ShadowIndex(final SearchRepository searchRepository, final SearchClient httpSearch) {
|
||||
this.searchRepository = searchRepository;
|
||||
this.httpSearch = httpSearch;
|
||||
}
|
||||
|
||||
public String create(final String entityType) {
|
||||
final String realIndex =
|
||||
searchRepository
|
||||
.getIndexMapping(entityType)
|
||||
.getIndexName(searchRepository.getClusterAlias());
|
||||
final String freshIndex =
|
||||
(realIndex + SHADOW_SUFFIX + UUID.randomUUID().toString().substring(0, 8))
|
||||
.toLowerCase(Locale.ROOT);
|
||||
final JsonNode source = innerSource(httpSearch.get("/" + realIndex), realIndex);
|
||||
httpSearch.put("/" + freshIndex, JsonUtils.pojoToJson(buildCreateBody(source)));
|
||||
return freshIndex;
|
||||
}
|
||||
|
||||
public void drop(final String freshIndex) {
|
||||
try {
|
||||
httpSearch.delete("/" + freshIndex);
|
||||
} catch (final Exception e) {
|
||||
// Best-effort cleanup on a finally path: never let a drop failure mask the case outcome.
|
||||
LOG.warn("Failed to drop shadow index {}", freshIndex, e);
|
||||
}
|
||||
}
|
||||
|
||||
private JsonNode innerSource(final JsonNode response, final String realIndex) {
|
||||
final JsonNode inner;
|
||||
if (response.has(realIndex)) {
|
||||
inner = response.get(realIndex);
|
||||
} else {
|
||||
final Iterator<JsonNode> elements = response.elements();
|
||||
if (!elements.hasNext()) {
|
||||
throw new IllegalStateException("Empty mapping response for index " + realIndex);
|
||||
}
|
||||
inner = elements.next();
|
||||
}
|
||||
return inner;
|
||||
}
|
||||
|
||||
private ObjectNode buildCreateBody(final JsonNode source) {
|
||||
final JsonNode srcSettingsIndex = source.path(SETTINGS).path(INDEX);
|
||||
final ObjectNode indexSettings = JsonUtils.getObjectNode();
|
||||
indexSettings.put(
|
||||
MAX_NGRAM_DIFF, srcSettingsIndex.path(MAX_NGRAM_DIFF).asInt(DEFAULT_MAX_NGRAM_DIFF));
|
||||
indexSettings.put(NUMBER_OF_REPLICAS, 0);
|
||||
if (srcSettingsIndex.has(ANALYSIS)) {
|
||||
indexSettings.set(ANALYSIS, srcSettingsIndex.get(ANALYSIS));
|
||||
}
|
||||
// Carry over any index.mapping.* limit overrides (total_fields/nested_objects/depth) so the
|
||||
// shadow index enforces exactly the real index's limits — otherwise an entity that raised a
|
||||
// limit would falsely report REJECTED against the default.
|
||||
if (srcSettingsIndex.has(MAPPING)) {
|
||||
indexSettings.set(MAPPING, srcSettingsIndex.get(MAPPING));
|
||||
}
|
||||
final ObjectNode settings = JsonUtils.getObjectNode();
|
||||
settings.set(INDEX, indexSettings);
|
||||
|
||||
final ObjectNode body = JsonUtils.getObjectNode();
|
||||
body.set(SETTINGS, settings);
|
||||
body.set(MAPPINGS, source.get(MAPPINGS));
|
||||
return body;
|
||||
}
|
||||
}
|
||||
+153
@@ -0,0 +1,153 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import org.openmetadata.it.search.SearchClient;
|
||||
import org.openmetadata.schema.EntityInterface;
|
||||
import org.openmetadata.schema.utils.JsonUtils;
|
||||
import org.openmetadata.service.search.SearchRepository;
|
||||
import org.openmetadata.service.search.indexes.SearchIndex;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
public final class ShapeCanary {
|
||||
private static final Logger LOG = LoggerFactory.getLogger(ShapeCanary.class);
|
||||
private static final String FOUND = "found";
|
||||
private static final int MAX_CAUSE_DEPTH = 10;
|
||||
|
||||
private final SearchRepository searchRepository;
|
||||
private final SearchClient httpSearch;
|
||||
private final ShadowIndex shadowIndex;
|
||||
|
||||
public ShapeCanary(final SearchRepository searchRepository, final SearchClient httpSearch) {
|
||||
this.searchRepository = searchRepository;
|
||||
this.httpSearch = httpSearch;
|
||||
this.shadowIndex = new ShadowIndex(searchRepository, httpSearch);
|
||||
}
|
||||
|
||||
public ShapeResult index(
|
||||
final String entityType, final EntityInterface entity, final FieldProbe probe) {
|
||||
final String docId = entity.getId().toString();
|
||||
final String freshIndex = shadowIndex.create(entityType);
|
||||
ShapeResult result;
|
||||
try {
|
||||
final String doc = buildDoc(entityType, entity);
|
||||
final String rejection = putDoc(freshIndex, docId, doc);
|
||||
result =
|
||||
rejection != null
|
||||
? new ShapeResult(Outcome.REJECTED, rejection)
|
||||
: verify(freshIndex, docId, probe);
|
||||
} finally {
|
||||
shadowIndex.drop(freshIndex);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* Builds the search document from the entity. Deliberately NOT wrapped in the REJECTED path: a
|
||||
* failure here is a doc-build/serialization bug in the harness or index code, not the engine
|
||||
* refusing an unindexable shape, so it must surface as an error rather than masquerade as
|
||||
* REJECTED (which is reserved for the write below).
|
||||
*/
|
||||
private String buildDoc(final String entityType, final EntityInterface entity) {
|
||||
final SearchIndex index =
|
||||
searchRepository.getSearchIndexFactory().buildIndex(entityType, entity);
|
||||
return JsonUtils.pojoToJson(index.buildSearchIndexDoc());
|
||||
}
|
||||
|
||||
/**
|
||||
* PUTs the built doc into the shadow index. Returns the engine's rejection detail when the write
|
||||
* fails, or {@code null} when it succeeds. Only this network write is caught — the engine refuses
|
||||
* an unindexable doc with varied exception types, and REJECTED plus the raw error chain is the
|
||||
* honest signal. We do NOT classify the cause.
|
||||
*/
|
||||
private String putDoc(final String freshIndex, final String docId, final String doc) {
|
||||
String rejection = null;
|
||||
try {
|
||||
searchRepository.getSearchClient().createEntity(freshIndex, docId, doc);
|
||||
} catch (final Exception e) {
|
||||
rejection = describe(e);
|
||||
LOG.warn("REJECTED: PUT to {} failed for doc {}: {}", freshIndex, docId, rejection, e);
|
||||
}
|
||||
return rejection;
|
||||
}
|
||||
|
||||
private ShapeResult verify(final String indexName, final String docId, final FieldProbe probe) {
|
||||
final ShapeResult result;
|
||||
final JsonNode response = httpSearch.get("/" + indexName + "/_doc/" + docId);
|
||||
if (!response.path(FOUND).asBoolean(false)) {
|
||||
result =
|
||||
new ShapeResult(Outcome.ERROR_OTHER, notRetrievableDetail(indexName, docId, response));
|
||||
} else if (probe == null || isSearchable(indexName, probe)) {
|
||||
result = new ShapeResult(Outcome.OK, "");
|
||||
} else {
|
||||
result =
|
||||
new ShapeResult(
|
||||
Outcome.DEGRADED_UNSEARCHABLE,
|
||||
"doc indexed and present in _source, but a term query on the ramped field returned no"
|
||||
+ " hits (value dropped from the term index, e.g. keyword ignore_above)");
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs the probe's term query, refreshing first. The get-by-id above is realtime, but {@code
|
||||
* _search} reads the near-real-time view — without a refresh a just-indexed searchable value can
|
||||
* return zero hits and be misreported as DEGRADED_UNSEARCHABLE.
|
||||
*/
|
||||
private boolean isSearchable(final String indexName, final FieldProbe probe) {
|
||||
httpSearch.post("/" + indexName + "/_refresh", "");
|
||||
return probe.searchable(httpSearch, indexName);
|
||||
}
|
||||
|
||||
/**
|
||||
* The PUT returned without throwing, yet a get-by-id finds nothing. The index {@code _count}
|
||||
* disambiguates "nothing was written" (silent no-op, count 0) from "written elsewhere" (count
|
||||
* > 0, e.g. wrong id/index), and the raw get response is included verbatim.
|
||||
*/
|
||||
private String notRetrievableDetail(
|
||||
final String indexName, final String docId, final JsonNode getResponse) {
|
||||
// _count reads the near-real-time search view (unlike the realtime get-by-id), so refresh
|
||||
// first — otherwise a just-written doc could read 0 and make the hint below wrong.
|
||||
httpSearch.post("/" + indexName + "/_refresh", "");
|
||||
final long count = httpSearch.get("/" + indexName + "/_count").path("count").asLong(-1);
|
||||
final String detail =
|
||||
"PUT returned without error but GET /"
|
||||
+ indexName
|
||||
+ "/_doc/"
|
||||
+ docId
|
||||
+ " -> found=false; post-refresh index _count="
|
||||
+ count
|
||||
+ (count == 0 ? " (nothing written — silent no-op PUT)" : " (doc written elsewhere?)")
|
||||
+ "; raw get: "
|
||||
+ getResponse;
|
||||
LOG.warn("ERROR_OTHER: {}", detail);
|
||||
return detail;
|
||||
}
|
||||
|
||||
private static String describe(final Throwable error) {
|
||||
final StringBuilder chain = new StringBuilder();
|
||||
Throwable cursor = error;
|
||||
int depth = 0;
|
||||
while (cursor != null && depth < MAX_CAUSE_DEPTH) {
|
||||
if (chain.length() > 0) {
|
||||
chain.append(" | caused by: ");
|
||||
}
|
||||
chain.append(cursor.getClass().getSimpleName()).append(": ").append(cursor.getMessage());
|
||||
cursor = cursor.getCause();
|
||||
depth++;
|
||||
}
|
||||
return chain.toString();
|
||||
}
|
||||
}
|
||||
+25
@@ -0,0 +1,25 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape;
|
||||
|
||||
import java.util.UUID;
|
||||
|
||||
public final class ShapeContext {
|
||||
public String unique(final String base) {
|
||||
return base + "_" + UUID.randomUUID().toString().substring(0, 8);
|
||||
}
|
||||
|
||||
public UUID id() {
|
||||
return UUID.randomUUID();
|
||||
}
|
||||
}
|
||||
+30
@@ -0,0 +1,30 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape;
|
||||
|
||||
import java.util.List;
|
||||
import org.openmetadata.schema.EntityInterface;
|
||||
|
||||
public interface ShapeMutation {
|
||||
String dimension();
|
||||
|
||||
boolean appliesTo(EntityInterface entity);
|
||||
|
||||
List<Rung> ladder();
|
||||
|
||||
EntityInterface apply(EntityInterface entity, Rung rung);
|
||||
|
||||
default FieldProbe probe(Rung rung) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape;
|
||||
|
||||
/**
|
||||
* Result of indexing one shape through {@link ShapeCanary}. {@code detail} carries the raw engine
|
||||
* error verbatim when {@code outcome} is {@link Outcome#REJECTED} (empty otherwise) — the canary
|
||||
* does not interpret WHY the write failed, it reports the engine's own words.
|
||||
*/
|
||||
public record ShapeResult(Outcome outcome, String detail) {}
|
||||
+63
@@ -0,0 +1,63 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape.mutations;
|
||||
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import org.openmetadata.it.search.shape.Rung;
|
||||
import org.openmetadata.it.search.shape.ShapeMutation;
|
||||
import org.openmetadata.schema.EntityInterface;
|
||||
|
||||
public final class CustomPropertiesBreadthMutation implements ShapeMutation {
|
||||
|
||||
@Override
|
||||
public String dimension() {
|
||||
return "customProperties.breadth";
|
||||
}
|
||||
|
||||
/**
|
||||
* Only entity types whose generated class overrides {@link EntityInterface#getExtension()} put the
|
||||
* {@code extension} map into their search document. For the rest, {@code setExtension} is the
|
||||
* no-op interface default, so ramping custom properties would index nothing and the case would
|
||||
* pass trivially — masking the very gap this dimension exists to catch. Checking the declaring
|
||||
* class (not {@code getExtension() != null}, which is always null on the un-mutated entity) keeps
|
||||
* the filter type-aware.
|
||||
*/
|
||||
@Override
|
||||
public boolean appliesTo(final EntityInterface entity) {
|
||||
boolean serializesExtension;
|
||||
try {
|
||||
serializesExtension =
|
||||
entity.getClass().getMethod("getExtension").getDeclaringClass() != EntityInterface.class;
|
||||
} catch (final NoSuchMethodException impossible) {
|
||||
serializesExtension = false;
|
||||
}
|
||||
return serializesExtension;
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<Rung> ladder() {
|
||||
return List.of(Rung.of("100", 100), Rung.of("2k", 2_000));
|
||||
}
|
||||
|
||||
@Override
|
||||
public EntityInterface apply(final EntityInterface entity, final Rung rung) {
|
||||
final Map<String, Object> props = new LinkedHashMap<>();
|
||||
for (int i = 0; i < rung.magnitude(); i++) {
|
||||
props.put("prop_" + i, "value_" + i);
|
||||
}
|
||||
entity.setExtension(props);
|
||||
return entity;
|
||||
}
|
||||
}
|
||||
+42
@@ -0,0 +1,42 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape.mutations;
|
||||
|
||||
import java.util.List;
|
||||
import org.openmetadata.it.search.shape.Rung;
|
||||
import org.openmetadata.it.search.shape.ShapeMutation;
|
||||
import org.openmetadata.schema.EntityInterface;
|
||||
|
||||
public final class DescriptionSizeMutation implements ShapeMutation {
|
||||
|
||||
@Override
|
||||
public String dimension() {
|
||||
return "description.size";
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean appliesTo(final EntityInterface entity) {
|
||||
return true;
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<Rung> ladder() {
|
||||
return List.of(Rung.of("1KB", 1_000), Rung.of("1MB", 1_000_000), Rung.of("16MB", 16_000_000));
|
||||
}
|
||||
|
||||
@Override
|
||||
public EntityInterface apply(final EntityInterface entity, final Rung rung) {
|
||||
entity.setDescription("x".repeat(rung.magnitude()));
|
||||
return entity;
|
||||
}
|
||||
}
|
||||
+48
@@ -0,0 +1,48 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape.mutations;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
import org.openmetadata.it.search.shape.Rung;
|
||||
import org.openmetadata.it.search.shape.ShapeMutation;
|
||||
import org.openmetadata.schema.EntityInterface;
|
||||
import org.openmetadata.schema.type.EntityReference;
|
||||
|
||||
public final class FollowersCountMutation implements ShapeMutation {
|
||||
@Override
|
||||
public String dimension() {
|
||||
return "followers.count";
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean appliesTo(final EntityInterface entity) {
|
||||
return true;
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<Rung> ladder() {
|
||||
return List.of(Rung.of("100", 100), Rung.of("50k", 50_000));
|
||||
}
|
||||
|
||||
@Override
|
||||
public EntityInterface apply(final EntityInterface entity, final Rung rung) {
|
||||
final List<EntityReference> followers = new ArrayList<>(rung.magnitude());
|
||||
for (int i = 0; i < rung.magnitude(); i++) {
|
||||
followers.add(new EntityReference().withId(UUID.randomUUID()).withType("user"));
|
||||
}
|
||||
entity.setFollowers(followers);
|
||||
return entity;
|
||||
}
|
||||
}
|
||||
+59
@@ -0,0 +1,59 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape.mutations;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import java.util.List;
|
||||
import org.openmetadata.it.search.shape.FieldProbe;
|
||||
import org.openmetadata.it.search.shape.Rung;
|
||||
import org.openmetadata.it.search.shape.ShapeMutation;
|
||||
import org.openmetadata.schema.EntityInterface;
|
||||
|
||||
public final class KeywordIgnoreAboveMutation implements ShapeMutation {
|
||||
private static final int OVER_IGNORE_ABOVE = 300;
|
||||
|
||||
@Override
|
||||
public String dimension() {
|
||||
return "keyword.overIgnoreAbove";
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean appliesTo(final EntityInterface entity) {
|
||||
return true;
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<Rung> ladder() {
|
||||
return List.of(Rung.of("300chars", OVER_IGNORE_ABOVE));
|
||||
}
|
||||
|
||||
@Override
|
||||
public EntityInterface apply(final EntityInterface entity, final Rung rung) {
|
||||
entity.setDisplayName(value(rung));
|
||||
return entity;
|
||||
}
|
||||
|
||||
@Override
|
||||
public FieldProbe probe(final Rung rung) {
|
||||
final String value = value(rung);
|
||||
return (httpSearch, indexName) -> {
|
||||
final String body = "{\"query\":{\"term\":{\"displayName.keyword\":\"" + value + "\"}}}";
|
||||
final JsonNode response = httpSearch.post("/" + indexName + "/_search", body);
|
||||
return response.path("hits").path("total").path("value").asLong(0) > 0;
|
||||
};
|
||||
}
|
||||
|
||||
private static String value(final Rung rung) {
|
||||
return "d".repeat(rung.magnitude());
|
||||
}
|
||||
}
|
||||
+66
@@ -0,0 +1,66 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape.mutations;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
import org.openmetadata.it.search.shape.Rung;
|
||||
import org.openmetadata.it.search.shape.ShapeMutation;
|
||||
import org.openmetadata.schema.EntityInterface;
|
||||
import org.openmetadata.schema.type.EntityReference;
|
||||
import org.openmetadata.service.search.SearchFieldLimits;
|
||||
|
||||
public final class OwnersCountMutation implements ShapeMutation {
|
||||
|
||||
@Override
|
||||
public String dimension() {
|
||||
return "owners.count";
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean appliesTo(final EntityInterface entity) {
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* {@code owners} is a nested field, so the engine rejects a document once its owner array exceeds
|
||||
* {@code index.mapping.nested_objects.limit}. That limit is server-controlled — its effective
|
||||
* value is {@link SearchFieldLimits#getNestedObjectsLimit()} — so the boundary rungs are derived
|
||||
* from it (not hardcoded) and stay correct if an operator retunes {@code searchIndexingLimits}.
|
||||
* The {@code aboveNestedLimit} rung is the one {@code AcceptedLimits} tolerates as REJECTED.
|
||||
*/
|
||||
@Override
|
||||
public List<Rung> ladder() {
|
||||
final int nestedLimit = SearchFieldLimits.active().getNestedObjectsLimit();
|
||||
return List.of(
|
||||
Rung.of("50", 50),
|
||||
Rung.of("belowNestedLimit", nestedLimit - Math.max(1, nestedLimit / 10)),
|
||||
Rung.of("aboveNestedLimit", nestedLimit + Math.max(1, nestedLimit / 5)));
|
||||
}
|
||||
|
||||
@Override
|
||||
public EntityInterface apply(final EntityInterface entity, final Rung rung) {
|
||||
final List<EntityReference> owners = new ArrayList<>(rung.magnitude());
|
||||
for (int i = 0; i < rung.magnitude(); i++) {
|
||||
owners.add(
|
||||
new EntityReference()
|
||||
.withId(UUID.randomUUID())
|
||||
.withType("user")
|
||||
.withName("owner_" + i)
|
||||
.withDisplayName("Owner " + i));
|
||||
}
|
||||
entity.setOwners(owners);
|
||||
return entity;
|
||||
}
|
||||
}
|
||||
+53
@@ -0,0 +1,53 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape.mutations;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import org.openmetadata.it.search.shape.Rung;
|
||||
import org.openmetadata.it.search.shape.ShapeMutation;
|
||||
import org.openmetadata.schema.EntityInterface;
|
||||
import org.openmetadata.schema.type.TagLabel;
|
||||
|
||||
public final class TagsCountMutation implements ShapeMutation {
|
||||
@Override
|
||||
public String dimension() {
|
||||
return "tags.count";
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean appliesTo(final EntityInterface entity) {
|
||||
return true;
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<Rung> ladder() {
|
||||
return List.of(Rung.of("10", 10), Rung.of("1k", 1_000), Rung.of("50k", 50_000));
|
||||
}
|
||||
|
||||
@Override
|
||||
public EntityInterface apply(final EntityInterface entity, final Rung rung) {
|
||||
final List<TagLabel> tags = new ArrayList<>(rung.magnitude());
|
||||
for (int i = 0; i < rung.magnitude(); i++) {
|
||||
tags.add(
|
||||
new TagLabel()
|
||||
.withTagFQN("ShapeCanary.tag_" + i)
|
||||
.withName("tag_" + i)
|
||||
.withSource(TagLabel.TagSource.CLASSIFICATION)
|
||||
.withLabelType(TagLabel.LabelType.MANUAL)
|
||||
.withState(TagLabel.State.CONFIRMED));
|
||||
}
|
||||
entity.setTags(tags);
|
||||
return entity;
|
||||
}
|
||||
}
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape.profiles;
|
||||
|
||||
import org.openmetadata.it.search.shape.EntityShapeProfile;
|
||||
import org.openmetadata.it.search.shape.ShapeContext;
|
||||
import org.openmetadata.schema.EntityInterface;
|
||||
import org.openmetadata.schema.entity.data.APICollection;
|
||||
import org.openmetadata.service.Entity;
|
||||
|
||||
public final class ApiCollectionShapeProfile implements EntityShapeProfile {
|
||||
|
||||
@Override
|
||||
public String entityType() {
|
||||
return Entity.API_COLLECTION;
|
||||
}
|
||||
|
||||
@Override
|
||||
public EntityInterface minimal(final ShapeContext ctx) {
|
||||
return new APICollection()
|
||||
.withId(ctx.id())
|
||||
.withName("apiCollection")
|
||||
.withFullyQualifiedName("shapeCanary.api." + ctx.unique("apiCollection"));
|
||||
}
|
||||
}
|
||||
+79
@@ -0,0 +1,79 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape.profiles;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import org.openmetadata.it.search.shape.EntityCases;
|
||||
import org.openmetadata.it.search.shape.EntityShapeProfile;
|
||||
import org.openmetadata.it.search.shape.PlannedCase;
|
||||
import org.openmetadata.it.search.shape.Rung;
|
||||
import org.openmetadata.it.search.shape.ShapeContext;
|
||||
import org.openmetadata.schema.EntityInterface;
|
||||
import org.openmetadata.schema.entity.data.APIEndpoint;
|
||||
import org.openmetadata.schema.type.APISchema;
|
||||
import org.openmetadata.schema.type.Field;
|
||||
import org.openmetadata.schema.type.FieldDataType;
|
||||
import org.openmetadata.service.Entity;
|
||||
|
||||
public final class ApiEndpointShapeProfile implements EntityShapeProfile {
|
||||
|
||||
@Override
|
||||
public String entityType() {
|
||||
return Entity.API_ENDPOINT;
|
||||
}
|
||||
|
||||
@Override
|
||||
public EntityInterface minimal(final ShapeContext ctx) {
|
||||
return new APIEndpoint()
|
||||
.withId(ctx.id())
|
||||
.withName("endpoint")
|
||||
.withFullyQualifiedName("shapeCanary.api." + ctx.unique("endpoint"))
|
||||
.withRequestSchema(new APISchema().withSchemaFields(List.of(field("req0"))))
|
||||
.withResponseSchema(new APISchema().withSchemaFields(List.of(field("resp0"))));
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<PlannedCase> entitySpecificCases(final ShapeContext ctx) {
|
||||
return new EntityCases(entityType(), this::minimal, ctx)
|
||||
.add("requestSchemaFields.count", Rung.of("1k", 1_000), this::requestFields)
|
||||
.add("requestSchemaFields.count", Rung.of("50k", 50_000), this::requestFields)
|
||||
.add("responseSchemaFields.count", Rung.of("1k", 1_000), this::responseFields)
|
||||
.add("responseSchemaFields.count", Rung.of("50k", 50_000), this::responseFields)
|
||||
.build();
|
||||
}
|
||||
|
||||
private EntityInterface requestFields(final EntityInterface entity, final Rung rung) {
|
||||
final APIEndpoint endpoint = (APIEndpoint) entity;
|
||||
endpoint.setRequestSchema(new APISchema().withSchemaFields(fields("req_", rung.magnitude())));
|
||||
return endpoint;
|
||||
}
|
||||
|
||||
private EntityInterface responseFields(final EntityInterface entity, final Rung rung) {
|
||||
final APIEndpoint endpoint = (APIEndpoint) entity;
|
||||
endpoint.setResponseSchema(new APISchema().withSchemaFields(fields("resp_", rung.magnitude())));
|
||||
return endpoint;
|
||||
}
|
||||
|
||||
private static List<Field> fields(final String prefix, final int count) {
|
||||
final List<Field> fields = new ArrayList<>(count);
|
||||
for (int i = 0; i < count; i++) {
|
||||
fields.add(field(prefix + i));
|
||||
}
|
||||
return fields;
|
||||
}
|
||||
|
||||
private static Field field(final String name) {
|
||||
return new Field().withName(name).withDataType(FieldDataType.STRING);
|
||||
}
|
||||
}
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape.profiles;
|
||||
|
||||
import org.openmetadata.it.search.shape.EntityShapeProfile;
|
||||
import org.openmetadata.it.search.shape.ShapeContext;
|
||||
import org.openmetadata.schema.EntityInterface;
|
||||
import org.openmetadata.schema.entity.data.Chart;
|
||||
import org.openmetadata.service.Entity;
|
||||
|
||||
public final class ChartShapeProfile implements EntityShapeProfile {
|
||||
|
||||
@Override
|
||||
public String entityType() {
|
||||
return Entity.CHART;
|
||||
}
|
||||
|
||||
@Override
|
||||
public EntityInterface minimal(final ShapeContext ctx) {
|
||||
return new Chart()
|
||||
.withId(ctx.id())
|
||||
.withName("chart")
|
||||
.withFullyQualifiedName("shapeCanary.bi.chart." + ctx.unique("chart"));
|
||||
}
|
||||
}
|
||||
+61
@@ -0,0 +1,61 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape.profiles;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import org.openmetadata.it.search.shape.EntityCases;
|
||||
import org.openmetadata.it.search.shape.EntityShapeProfile;
|
||||
import org.openmetadata.it.search.shape.PlannedCase;
|
||||
import org.openmetadata.it.search.shape.Rung;
|
||||
import org.openmetadata.it.search.shape.ShapeContext;
|
||||
import org.openmetadata.schema.EntityInterface;
|
||||
import org.openmetadata.schema.entity.data.Container;
|
||||
import org.openmetadata.schema.type.Column;
|
||||
import org.openmetadata.schema.type.ColumnDataType;
|
||||
import org.openmetadata.schema.type.ContainerDataModel;
|
||||
import org.openmetadata.service.Entity;
|
||||
|
||||
public final class ContainerShapeProfile implements EntityShapeProfile {
|
||||
|
||||
@Override
|
||||
public String entityType() {
|
||||
return Entity.CONTAINER;
|
||||
}
|
||||
|
||||
@Override
|
||||
public EntityInterface minimal(final ShapeContext ctx) {
|
||||
return new Container()
|
||||
.withId(ctx.id())
|
||||
.withName("container")
|
||||
.withFullyQualifiedName("shapeCanary.storage." + ctx.unique("container"));
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<PlannedCase> entitySpecificCases(final ShapeContext ctx) {
|
||||
return new EntityCases(entityType(), this::minimal, ctx)
|
||||
.add("dataModelColumns.count", Rung.of("1k", 1_000), this::dataModelColumns)
|
||||
.add("dataModelColumns.count", Rung.of("50k", 50_000), this::dataModelColumns)
|
||||
.build();
|
||||
}
|
||||
|
||||
private EntityInterface dataModelColumns(final EntityInterface entity, final Rung rung) {
|
||||
final Container container = (Container) entity;
|
||||
final List<Column> columns = new ArrayList<>(rung.magnitude());
|
||||
for (int i = 0; i < rung.magnitude(); i++) {
|
||||
columns.add(new Column().withName("c_" + i).withDataType(ColumnDataType.STRING));
|
||||
}
|
||||
container.setDataModel(new ContainerDataModel().withColumns(columns));
|
||||
return container;
|
||||
}
|
||||
}
|
||||
+62
@@ -0,0 +1,62 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape.profiles;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import org.openmetadata.it.search.shape.EntityCases;
|
||||
import org.openmetadata.it.search.shape.EntityShapeProfile;
|
||||
import org.openmetadata.it.search.shape.PlannedCase;
|
||||
import org.openmetadata.it.search.shape.Rung;
|
||||
import org.openmetadata.it.search.shape.ShapeContext;
|
||||
import org.openmetadata.schema.EntityInterface;
|
||||
import org.openmetadata.schema.entity.data.DashboardDataModel;
|
||||
import org.openmetadata.schema.type.Column;
|
||||
import org.openmetadata.schema.type.ColumnDataType;
|
||||
import org.openmetadata.service.Entity;
|
||||
|
||||
public final class DashboardDataModelShapeProfile implements EntityShapeProfile {
|
||||
|
||||
@Override
|
||||
public String entityType() {
|
||||
return Entity.DASHBOARD_DATA_MODEL;
|
||||
}
|
||||
|
||||
@Override
|
||||
public EntityInterface minimal(final ShapeContext ctx) {
|
||||
return new DashboardDataModel()
|
||||
.withId(ctx.id())
|
||||
.withName("dataModel")
|
||||
.withFullyQualifiedName("shapeCanary.bi.model." + ctx.unique("dataModel"))
|
||||
.withColumns(List.of(new Column().withName("c0").withDataType(ColumnDataType.STRING)));
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<PlannedCase> entitySpecificCases(final ShapeContext ctx) {
|
||||
return new EntityCases(entityType(), this::minimal, ctx)
|
||||
.add("dataModelColumns.count", Rung.of("1k", 1_000), this::columns)
|
||||
.add("dataModelColumns.count", Rung.of("10k", 10_000), this::columns)
|
||||
.add("dataModelColumns.count", Rung.of("50k", 50_000), this::columns)
|
||||
.build();
|
||||
}
|
||||
|
||||
private EntityInterface columns(final EntityInterface entity, final Rung rung) {
|
||||
final DashboardDataModel model = (DashboardDataModel) entity;
|
||||
final List<Column> columns = new ArrayList<>(rung.magnitude());
|
||||
for (int i = 0; i < rung.magnitude(); i++) {
|
||||
columns.add(new Column().withName("c_" + i).withDataType(ColumnDataType.STRING));
|
||||
}
|
||||
model.setColumns(columns);
|
||||
return model;
|
||||
}
|
||||
}
|
||||
+61
@@ -0,0 +1,61 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape.profiles;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
import org.openmetadata.it.search.shape.EntityCases;
|
||||
import org.openmetadata.it.search.shape.EntityShapeProfile;
|
||||
import org.openmetadata.it.search.shape.PlannedCase;
|
||||
import org.openmetadata.it.search.shape.Rung;
|
||||
import org.openmetadata.it.search.shape.ShapeContext;
|
||||
import org.openmetadata.schema.EntityInterface;
|
||||
import org.openmetadata.schema.entity.data.Dashboard;
|
||||
import org.openmetadata.schema.type.EntityReference;
|
||||
import org.openmetadata.service.Entity;
|
||||
|
||||
public final class DashboardShapeProfile implements EntityShapeProfile {
|
||||
|
||||
@Override
|
||||
public String entityType() {
|
||||
return Entity.DASHBOARD;
|
||||
}
|
||||
|
||||
@Override
|
||||
public EntityInterface minimal(final ShapeContext ctx) {
|
||||
return new Dashboard()
|
||||
.withId(ctx.id())
|
||||
.withName("dashboard")
|
||||
.withFullyQualifiedName("shapeCanary.bi." + ctx.unique("dashboard"));
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<PlannedCase> entitySpecificCases(final ShapeContext ctx) {
|
||||
return new EntityCases(entityType(), this::minimal, ctx)
|
||||
.add("charts.count", Rung.of("1k", 1_000), this::charts)
|
||||
.add("charts.count", Rung.of("50k", 50_000), this::charts)
|
||||
.build();
|
||||
}
|
||||
|
||||
private EntityInterface charts(final EntityInterface entity, final Rung rung) {
|
||||
final Dashboard dashboard = (Dashboard) entity;
|
||||
final List<EntityReference> charts = new ArrayList<>(rung.magnitude());
|
||||
for (int i = 0; i < rung.magnitude(); i++) {
|
||||
charts.add(
|
||||
new EntityReference().withId(UUID.randomUUID()).withType("chart").withName("chart_" + i));
|
||||
}
|
||||
dashboard.setCharts(charts);
|
||||
return dashboard;
|
||||
}
|
||||
}
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape.profiles;
|
||||
|
||||
import org.openmetadata.it.search.shape.EntityShapeProfile;
|
||||
import org.openmetadata.it.search.shape.ShapeContext;
|
||||
import org.openmetadata.schema.EntityInterface;
|
||||
import org.openmetadata.schema.entity.domains.DataProduct;
|
||||
import org.openmetadata.service.Entity;
|
||||
|
||||
public final class DataProductShapeProfile implements EntityShapeProfile {
|
||||
|
||||
@Override
|
||||
public String entityType() {
|
||||
return Entity.DATA_PRODUCT;
|
||||
}
|
||||
|
||||
@Override
|
||||
public EntityInterface minimal(final ShapeContext ctx) {
|
||||
return new DataProduct()
|
||||
.withId(ctx.id())
|
||||
.withName("dataProduct")
|
||||
.withFullyQualifiedName("shapeCanaryDomain." + ctx.unique("dataProduct"));
|
||||
}
|
||||
}
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape.profiles;
|
||||
|
||||
import org.openmetadata.it.search.shape.EntityShapeProfile;
|
||||
import org.openmetadata.it.search.shape.ShapeContext;
|
||||
import org.openmetadata.schema.EntityInterface;
|
||||
import org.openmetadata.schema.entity.data.DatabaseSchema;
|
||||
import org.openmetadata.service.Entity;
|
||||
|
||||
public final class DatabaseSchemaShapeProfile implements EntityShapeProfile {
|
||||
|
||||
@Override
|
||||
public String entityType() {
|
||||
return Entity.DATABASE_SCHEMA;
|
||||
}
|
||||
|
||||
@Override
|
||||
public EntityInterface minimal(final ShapeContext ctx) {
|
||||
return new DatabaseSchema()
|
||||
.withId(ctx.id())
|
||||
.withName("schema")
|
||||
.withFullyQualifiedName("shapeCanary.db.schema." + ctx.unique("schema"));
|
||||
}
|
||||
}
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape.profiles;
|
||||
|
||||
import org.openmetadata.it.search.shape.EntityShapeProfile;
|
||||
import org.openmetadata.it.search.shape.ShapeContext;
|
||||
import org.openmetadata.schema.EntityInterface;
|
||||
import org.openmetadata.schema.entity.data.Database;
|
||||
import org.openmetadata.service.Entity;
|
||||
|
||||
public final class DatabaseShapeProfile implements EntityShapeProfile {
|
||||
|
||||
@Override
|
||||
public String entityType() {
|
||||
return Entity.DATABASE;
|
||||
}
|
||||
|
||||
@Override
|
||||
public EntityInterface minimal(final ShapeContext ctx) {
|
||||
return new Database()
|
||||
.withId(ctx.id())
|
||||
.withName("database")
|
||||
.withFullyQualifiedName("shapeCanary.db." + ctx.unique("database"));
|
||||
}
|
||||
}
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape.profiles;
|
||||
|
||||
import org.openmetadata.it.search.shape.EntityShapeProfile;
|
||||
import org.openmetadata.it.search.shape.ShapeContext;
|
||||
import org.openmetadata.schema.EntityInterface;
|
||||
import org.openmetadata.schema.entity.domains.Domain;
|
||||
import org.openmetadata.service.Entity;
|
||||
|
||||
public final class DomainShapeProfile implements EntityShapeProfile {
|
||||
|
||||
@Override
|
||||
public String entityType() {
|
||||
return Entity.DOMAIN;
|
||||
}
|
||||
|
||||
@Override
|
||||
public EntityInterface minimal(final ShapeContext ctx) {
|
||||
return new Domain()
|
||||
.withId(ctx.id())
|
||||
.withName("domain")
|
||||
.withFullyQualifiedName("shapeCanaryDomain." + ctx.unique("domain"));
|
||||
}
|
||||
}
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape.profiles;
|
||||
|
||||
import org.openmetadata.it.search.shape.EntityShapeProfile;
|
||||
import org.openmetadata.it.search.shape.ShapeContext;
|
||||
import org.openmetadata.schema.EntityInterface;
|
||||
import org.openmetadata.schema.entity.data.Glossary;
|
||||
import org.openmetadata.service.Entity;
|
||||
|
||||
public final class GlossaryShapeProfile implements EntityShapeProfile {
|
||||
|
||||
@Override
|
||||
public String entityType() {
|
||||
return Entity.GLOSSARY;
|
||||
}
|
||||
|
||||
@Override
|
||||
public EntityInterface minimal(final ShapeContext ctx) {
|
||||
return new Glossary()
|
||||
.withId(ctx.id())
|
||||
.withName("glossary")
|
||||
.withFullyQualifiedName("shapeCanary." + ctx.unique("glossary"));
|
||||
}
|
||||
}
|
||||
+62
@@ -0,0 +1,62 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape.profiles;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
import org.openmetadata.it.search.shape.EntityCases;
|
||||
import org.openmetadata.it.search.shape.EntityShapeProfile;
|
||||
import org.openmetadata.it.search.shape.PlannedCase;
|
||||
import org.openmetadata.it.search.shape.Rung;
|
||||
import org.openmetadata.it.search.shape.ShapeContext;
|
||||
import org.openmetadata.schema.EntityInterface;
|
||||
import org.openmetadata.schema.entity.data.GlossaryTerm;
|
||||
import org.openmetadata.schema.type.EntityReference;
|
||||
import org.openmetadata.service.Entity;
|
||||
|
||||
public final class GlossaryTermShapeProfile implements EntityShapeProfile {
|
||||
|
||||
@Override
|
||||
public String entityType() {
|
||||
return Entity.GLOSSARY_TERM;
|
||||
}
|
||||
|
||||
@Override
|
||||
public EntityInterface minimal(final ShapeContext ctx) {
|
||||
return new GlossaryTerm()
|
||||
.withId(ctx.id())
|
||||
.withName("term")
|
||||
.withFullyQualifiedName("shapeCanaryGlossary." + ctx.unique("term"))
|
||||
.withGlossary(
|
||||
new EntityReference().withId(UUID.randomUUID()).withType("glossary").withName("g"));
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<PlannedCase> entitySpecificCases(final ShapeContext ctx) {
|
||||
return new EntityCases(entityType(), this::minimal, ctx)
|
||||
.add("synonyms.count", Rung.of("1k", 1_000), this::synonyms)
|
||||
.add("synonyms.count", Rung.of("100k", 100_000), this::synonyms)
|
||||
.build();
|
||||
}
|
||||
|
||||
private EntityInterface synonyms(final EntityInterface entity, final Rung rung) {
|
||||
final GlossaryTerm term = (GlossaryTerm) entity;
|
||||
final List<String> synonyms = new ArrayList<>(rung.magnitude());
|
||||
for (int i = 0; i < rung.magnitude(); i++) {
|
||||
synonyms.add("synonym_" + i);
|
||||
}
|
||||
term.setSynonyms(synonyms);
|
||||
return term;
|
||||
}
|
||||
}
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape.profiles;
|
||||
|
||||
import org.openmetadata.it.search.shape.EntityShapeProfile;
|
||||
import org.openmetadata.it.search.shape.ShapeContext;
|
||||
import org.openmetadata.schema.EntityInterface;
|
||||
import org.openmetadata.schema.entity.data.Metric;
|
||||
import org.openmetadata.service.Entity;
|
||||
|
||||
public final class MetricShapeProfile implements EntityShapeProfile {
|
||||
|
||||
@Override
|
||||
public String entityType() {
|
||||
return Entity.METRIC;
|
||||
}
|
||||
|
||||
@Override
|
||||
public EntityInterface minimal(final ShapeContext ctx) {
|
||||
return new Metric()
|
||||
.withId(ctx.id())
|
||||
.withName("metric")
|
||||
.withFullyQualifiedName("shapeCanaryMetric." + ctx.unique("metric"));
|
||||
}
|
||||
}
|
||||
+61
@@ -0,0 +1,61 @@
|
||||
/*
|
||||
* Copyright 2024 Collate.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.openmetadata.it.search.shape.profiles;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import org.openmetadata.it.search.shape.EntityCases;
|
||||
import org.openmetadata.it.search.shape.EntityShapeProfile;
|
||||
import org.openmetadata.it.search.shape.PlannedCase;
|
||||
import org.openmetadata.it.search.shape.Rung;
|
||||
import org.openmetadata.it.search.shape.ShapeContext;
|
||||
import org.openmetadata.schema.EntityInterface;
|
||||
import org.openmetadata.schema.entity.data.MlModel;
|
||||
import org.openmetadata.schema.type.MlFeature;
|
||||
import org.openmetadata.service.Entity;
|
||||
|
||||
public final class MlModelShapeProfile implements EntityShapeProfile {
|
||||
|
||||
@Override
|
||||
public String entityType() {
|
||||
return Entity.MLMODEL;
|
||||
}
|
||||
|
||||
@Override
|
||||
public EntityInterface minimal(final ShapeContext ctx) {
|
||||
return new MlModel()
|
||||
.withId(ctx.id())
|
||||
.withName("model")
|
||||
.withFullyQualifiedName("shapeCanary.ml." + ctx.unique("model"))
|
||||
.withMlFeatures(List.of(new MlFeature().withName("f0")));
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<PlannedCase> entitySpecificCases(final ShapeContext ctx) {
|
||||
return new EntityCases(entityType(), this::minimal, ctx)
|
||||
.add("mlFeatures.count", Rung.of("1k", 1_000), this::features)
|
||||
.add("mlFeatures.count", Rung.of("10k", 10_000), this::features)
|
||||
.add("mlFeatures.count", Rung.of("50k", 50_000), this::features)
|
||||
.build();
|
||||
}
|
||||
|
||||
private EntityInterface features(final EntityInterface entity, final Rung rung) {
|
||||
final MlModel model = (MlModel) entity;
|
||||
final List<MlFeature> features = new ArrayList<>(rung.magnitude());
|
||||
for (int i = 0; i < rung.magnitude(); i++) {
|
||||
features.add(new MlFeature().withName("f_" + i));
|
||||
}
|
||||
model.setMlFeatures(features);
|
||||
return model;
|
||||
}
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user