chore: import upstream snapshot with attribution

docker/development/distributed-test/README.md

@@ -0,0 +1,253 @@
+# Distributed Search Indexing Test Environment
+
+This directory contains scripts and configurations to test the distributed search indexing feature with multiple OpenMetadata servers sharing a common database.
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│                     Test Environment                            │
+├─────────────────────────────────────────────────────────────────┤
+│                                                                 │
+│  ┌──────────┐  ┌──────────┐  ┌──────────┐                      │
+│  │ OM Server│  │ OM Server│  │ OM Server│                      │
+│  │  :8585   │  │  :8587   │  │  :8589   │                      │
+│  │ SERVER-1 │  │ SERVER-2 │  │ SERVER-3 │                      │
+│  └────┬─────┘  └────┬─────┘  └────┬─────┘                      │
+│       │             │             │                             │
+│       └─────────────┼─────────────┘                             │
+│                     │                                           │
+│              ┌──────┴──────┐                                    │
+│              │   Polling   │  (DB-based coordination)           │
+│              └──────┬──────┘                                    │
+│                     │                                           │
+│       ┌─────────────┴─────────────┐                             │
+│       ▼                           ▼                             │
+│  ┌─────────┐                ┌───────────┐                       │
+│  │  MySQL  │                │ OpenSearch│                       │
+│  │  :3306  │                │   :9200   │                       │
+│  └─────────┘                └───────────┘                       │
+│                                                                 │
+└─────────────────────────────────────────────────────────────────┘
+```
+
+## Quick Start (Docker Compose)
+
+### 1. Start the Environment
+
+```bash
+cd docker/development/distributed-test
+
+# Start all services (builds images on first run)
+./scripts/start.sh
+
+# Or force rebuild
+./scripts/start.sh --build
+```
+
+### 2. Load Test Data
+
+```bash
+# Load 10,000 tables (default)
+./scripts/perf-test.sh
+
+# Or specify the number
+./scripts/perf-test.sh --tables 50000 --databases 50
+```
+
+### 3. Trigger Reindexing
+
+```bash
+# Trigger reindex on server 1
+./scripts/trigger-reindex.sh
+
+# With index recreation
+./scripts/trigger-reindex.sh --recreate
+
+# Specific entities only
+./scripts/trigger-reindex.sh --entities table,dashboard
+```
+
+### 4. Monitor Progress
+
+```bash
+# Follow logs from all servers
+./scripts/logs.sh -f
+
+# Filter by pattern
+./scripts/logs.sh -f --grep "partition"
+
+# Single server logs
+./scripts/logs.sh -f --server 1
+```
+
+### 5. Stop the Environment
+
+```bash
+# Stop containers (preserve data)
+./scripts/stop.sh
+
+# Stop and clean up volumes
+./scripts/stop.sh --clean
+```
+
+## Local Development (IDE Debugging)
+
+For debugging with breakpoints, run OM servers locally while using Docker for MySQL and OpenSearch.
+
+### 1. Start Dependencies Only
+
+```bash
+cd docker/development/distributed-test
+docker compose -f local/docker-compose-deps.yml up -d
+```
+
+### 2. Run Migrations (First Time)
+
+```bash
+cd /path/to/openmetadata
+./bootstrap/openmetadata-ops.sh -d migrate --force
+```
+
+### 3. Option A: Run from Terminal
+
+```bash
+# Start all 3 servers in separate terminals
+./local/run-local-servers.sh
+
+# Or specific servers
+./local/run-local-servers.sh 1 2
+```
+
+### 3. Option B: Run from IDE
+
+Create run configurations in IntelliJ IDEA:
+
+**Server 1:**
+- Main class: `org.openmetadata.service.OpenMetadataApplication`
+- Program arguments: `server docker/development/distributed-test/local/server1.yaml`
+- VM options: `-Xmx1G -Xms512M`
+- Working directory: Project root
+
+**Server 2:**
+- Same as above but with `local/server2.yaml`
+
+**Server 3:**
+- Same as above but with `local/server3.yaml`
+
+## Server Ports
+
+| Server | API Port | Admin Port |
+|--------|----------|------------|
+| Server 1 | 8585 | 8586 |
+| Server 2 | 8587 | 8588 |
+| Server 3 | 8589 | 8590 |
+
+## Configuration
+
+Edit `.env` to customize:
+
+```bash
+# Number of tables for test data
+TEST_DATA_TABLES=10000
+
+# Log level
+LOG_LEVEL=INFO
+
+# Heap size per server
+OPENMETADATA_HEAP_OPTS=-Xmx1G -Xms1G
+```
+
+## Testing Distributed Indexing
+
+### Verify Partition Distribution
+
+1. Start all 3 servers
+2. Load test data: `./scripts/perf-test.sh --tables 10000`
+3. Trigger reindex: `./scripts/trigger-reindex.sh --recreate`
+4. Watch logs: `./scripts/logs.sh -f --grep "partition"`
+
+You should see output like:
+```
+[SERVER-1] INFO  Claimed partition: table_0-999 (1000 records)
+[SERVER-2] INFO  Claimed partition: table_1000-1999 (1000 records)
+[SERVER-3] INFO  Claimed partition: table_2000-2999 (1000 records)
+[SERVER-1] INFO  Completed partition: table_0-999
+...
+```
+
+### Test Server Failure Recovery
+
+1. Start reindexing with many partitions
+2. Stop one server mid-process: `docker stop distributed_test_om_server_2`
+3. Watch remaining servers pick up orphaned partitions
+4. Verify job completes successfully
+
+### Check Job Status
+
+```bash
+# Via API
+curl -s http://localhost:8585/api/v1/apps/name/SearchIndexingApplication/status | jq
+
+# Check partition table directly
+docker exec -it distributed_test_mysql mysql -uopenmetadata_user -popenmetadata_password openmetadata_db \
+  -e "SELECT status, COUNT(*) FROM search_index_partition GROUP BY status"
+```
+
+## Troubleshooting
+
+### Servers Not Starting
+
+Check if ports are in use:
+```bash
+lsof -i :8585
+lsof -i :8587
+lsof -i :8589
+```
+
+### Database Connection Issues
+
+Verify MySQL is accessible:
+```bash
+docker exec -it distributed_test_mysql mysql -uopenmetadata_user -popenmetadata_password -e "SELECT 1"
+```
+
+### OpenSearch Not Ready
+
+Check health:
+```bash
+curl http://localhost:9200/_cluster/health?pretty
+```
+
+### View Full Logs
+
+```bash
+# All container logs
+docker compose logs -f
+
+# Specific container
+docker logs -f distributed_test_om_server_1
+```
+
+## File Structure
+
+```
+distributed-test/
+├── docker-compose.yml          # Full environment (3 OM servers + deps)
+├── .env                        # Configuration variables
+├── config/
+│   └── mysql-init.sql          # Database initialization
+├── scripts/
+│   ├── start.sh                # Start full environment
+│   ├── stop.sh                 # Stop environment
+│   ├── logs.sh                 # View aggregated logs
+│   ├── trigger-reindex.sh      # Trigger reindexing
+│   └── perf-test.sh       # Load test data
+├── local/
+│   ├── docker-compose-deps.yml # Dependencies only (for IDE debugging)
+│   ├── server1.yaml            # Server 1 config (port 8585)
+│   ├── server2.yaml            # Server 2 config (port 8587)
+│   ├── server3.yaml            # Server 3 config (port 8589)
+│   └── run-local-servers.sh    # Start servers locally
+└── README.md                   # This file
+```

docker/development/distributed-test/config/mysql-init.sql

@@ -0,0 +1,13 @@
+-- MySQL initialization script for distributed test environment
+
+-- Create the OpenMetadata database
+CREATE DATABASE IF NOT EXISTS openmetadata_db;
+
+-- Create the OpenMetadata user
+CREATE USER IF NOT EXISTS 'openmetadata_user'@'%' IDENTIFIED BY 'openmetadata_password';
+
+-- Grant privileges
+GRANT ALL PRIVILEGES ON openmetadata_db.* TO 'openmetadata_user'@'%';
+GRANT ALL PRIVILEGES ON *.* TO 'openmetadata_user'@'%';
+
+FLUSH PRIVILEGES;

docker/development/distributed-test/docker-compose.yml

@@ -0,0 +1,270 @@
+#  Copyright 2021 Collate
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#  http://www.apache.org/licenses/LICENSE-2.0
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+# Distributed Search Indexing Test Environment
+# This compose file sets up multiple OM servers sharing a common MySQL and OpenSearch
+
+version: "3.9"
+
+volumes:
+  mysql-data:
+  opensearch-data:
+
+networks:
+  distributed-test-net:
+    name: distributed_test_network
+    driver: bridge
+
+services:
+  # Shared MySQL Database
+  mysql:
+    image: mysql:8.0
+    container_name: distributed_test_mysql
+    restart: unless-stopped
+    environment:
+      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD:-password}
+    command: >
+      --sort_buffer_size=10M
+      --max_connections=500
+    ports:
+      - "${MYSQL_PORT:-3306}:3306"
+    networks:
+      - distributed-test-net
+    healthcheck:
+      test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-uroot", "-p${MYSQL_ROOT_PASSWORD:-password}"]
+      interval: 10s
+      timeout: 5s
+      retries: 10
+    volumes:
+      - mysql-data:/var/lib/mysql
+      - ./config/mysql-init.sql:/docker-entrypoint-initdb.d/init.sql:ro
+
+  # Shared OpenSearch
+  opensearch:
+    image: opensearchproject/opensearch:3.4.0
+    container_name: distributed_test_opensearch
+    restart: unless-stopped
+    environment:
+      - discovery.type=single-node
+      - plugins.security.disabled=true
+      - "OPENSEARCH_JAVA_OPTS=${OPENSEARCH_JAVA_OPTS:--Xms512m -Xmx512m}"
+    ports:
+      - "${OPENSEARCH_PORT:-9200}:9200"
+      - "9600:9600"
+    networks:
+      - distributed-test-net
+    healthcheck:
+      test: ["CMD-SHELL", "curl -s http://localhost:9200/_cluster/health | grep -qE '\"status\":\"(green|yellow)\"'"]
+      interval: 10s
+      timeout: 5s
+      retries: 10
+    volumes:
+      - opensearch-data:/usr/share/opensearch/data
+
+  # Database migration (runs once)
+  migrate:
+    build:
+      context: ../../../.
+      dockerfile: docker/development/Dockerfile
+    container_name: distributed_test_migrate
+    command: "./bootstrap/openmetadata-ops.sh -d migrate --force"
+    environment:
+      LOG_LEVEL: INFO
+      DB_DRIVER_CLASS: com.mysql.cj.jdbc.Driver
+      DB_SCHEME: mysql
+      DB_HOST: mysql
+      DB_PORT: 3306
+      DB_USER: ${DB_USER:-openmetadata_user}
+      DB_USER_PASSWORD: ${DB_USER_PASSWORD:-openmetadata_password}
+      OM_DATABASE: ${OM_DATABASE:-openmetadata_db}
+      DB_PARAMS: allowPublicKeyRetrieval=true&useSSL=false&serverTimezone=UTC
+      SEARCH_TYPE: opensearch
+      ELASTICSEARCH_HOST: opensearch
+      ELASTICSEARCH_PORT: 9200
+      ELASTICSEARCH_SCHEME: http
+    depends_on:
+      mysql:
+        condition: service_healthy
+      opensearch:
+        condition: service_healthy
+    networks:
+      - distributed-test-net
+
+  # OpenMetadata Server 1 (Primary - can trigger reindex)
+  openmetadata-server-1:
+    build:
+      context: ../../../.
+      dockerfile: docker/development/Dockerfile
+    container_name: distributed_test_om_server_1
+    hostname: om-server-1
+    restart: unless-stopped
+    environment:
+      SERVER_PORT: 8585
+      SERVER_ADMIN_PORT: 8586
+      LOG_LEVEL: ${LOG_LEVEL:-INFO}
+      OPENMETADATA_CLUSTER_NAME: distributed-test
+
+      # Unique server identifier for distributed indexing
+      OM_SERVER_ID: server-1
+
+      # Database
+      DB_DRIVER_CLASS: com.mysql.cj.jdbc.Driver
+      DB_SCHEME: mysql
+      DB_HOST: mysql
+      DB_PORT: 3306
+      DB_USER: ${DB_USER:-openmetadata_user}
+      DB_USER_PASSWORD: ${DB_USER_PASSWORD:-openmetadata_password}
+      OM_DATABASE: ${OM_DATABASE:-openmetadata_db}
+      DB_PARAMS: allowPublicKeyRetrieval=true&useSSL=false&serverTimezone=UTC
+
+      # Search
+      SEARCH_TYPE: opensearch
+      ELASTICSEARCH_HOST: opensearch
+      ELASTICSEARCH_PORT: 9200
+      ELASTICSEARCH_SCHEME: http
+
+      # Auth
+      AUTHENTICATION_PROVIDER: basic
+      AUTHORIZER_ADMIN_PRINCIPALS: "[admin]"
+      AUTHENTICATION_PUBLIC_KEYS: "[http://localhost:8585/api/v1/system/config/jwks]"
+
+      # Pipeline service disabled for testing
+      PIPELINE_SERVICE_CLIENT_ENABLED: "false"
+
+      # Heap
+      OPENMETADATA_HEAP_OPTS: ${OPENMETADATA_HEAP_OPTS:--Xmx1G -Xms1G}
+    ports:
+      - "8585:8585"
+      - "8586:8586"
+    depends_on:
+      migrate:
+        condition: service_completed_successfully
+    networks:
+      - distributed-test-net
+    healthcheck:
+      test: ["CMD", "wget", "-q", "--spider", "http://localhost:8586/healthcheck"]
+      interval: 15s
+      timeout: 10s
+      retries: 10
+
+  # OpenMetadata Server 2
+  openmetadata-server-2:
+    build:
+      context: ../../../.
+      dockerfile: docker/development/Dockerfile
+    container_name: distributed_test_om_server_2
+    hostname: om-server-2
+    restart: unless-stopped
+    environment:
+      SERVER_PORT: 8585
+      SERVER_ADMIN_PORT: 8586
+      LOG_LEVEL: ${LOG_LEVEL:-INFO}
+      OPENMETADATA_CLUSTER_NAME: distributed-test
+
+      # Unique server identifier for distributed indexing
+      OM_SERVER_ID: server-2
+
+      # Database
+      DB_DRIVER_CLASS: com.mysql.cj.jdbc.Driver
+      DB_SCHEME: mysql
+      DB_HOST: mysql
+      DB_PORT: 3306
+      DB_USER: ${DB_USER:-openmetadata_user}
+      DB_USER_PASSWORD: ${DB_USER_PASSWORD:-openmetadata_password}
+      OM_DATABASE: ${OM_DATABASE:-openmetadata_db}
+      DB_PARAMS: allowPublicKeyRetrieval=true&useSSL=false&serverTimezone=UTC
+
+      # Search
+      SEARCH_TYPE: opensearch
+      ELASTICSEARCH_HOST: opensearch
+      ELASTICSEARCH_PORT: 9200
+      ELASTICSEARCH_SCHEME: http
+
+      # Auth
+      AUTHENTICATION_PROVIDER: basic
+      AUTHORIZER_ADMIN_PRINCIPALS: "[admin]"
+      AUTHENTICATION_PUBLIC_KEYS: "[http://localhost:8587/api/v1/system/config/jwks]"
+
+      # Pipeline service disabled for testing
+      PIPELINE_SERVICE_CLIENT_ENABLED: "false"
+
+      # Heap
+      OPENMETADATA_HEAP_OPTS: ${OPENMETADATA_HEAP_OPTS:--Xmx1G -Xms1G}
+    ports:
+      - "8587:8585"
+      - "8588:8586"
+    depends_on:
+      migrate:
+        condition: service_completed_successfully
+    networks:
+      - distributed-test-net
+    healthcheck:
+      test: ["CMD", "wget", "-q", "--spider", "http://localhost:8586/healthcheck"]
+      interval: 15s
+      timeout: 10s
+      retries: 10
+
+  # OpenMetadata Server 3
+  openmetadata-server-3:
+    build:
+      context: ../../../.
+      dockerfile: docker/development/Dockerfile
+    container_name: distributed_test_om_server_3
+    hostname: om-server-3
+    restart: unless-stopped
+    environment:
+      SERVER_PORT: 8585
+      SERVER_ADMIN_PORT: 8586
+      LOG_LEVEL: ${LOG_LEVEL:-INFO}
+      OPENMETADATA_CLUSTER_NAME: distributed-test
+
+      # Unique server identifier for distributed indexing
+      OM_SERVER_ID: server-3
+
+      # Database
+      DB_DRIVER_CLASS: com.mysql.cj.jdbc.Driver
+      DB_SCHEME: mysql
+      DB_HOST: mysql
+      DB_PORT: 3306
+      DB_USER: ${DB_USER:-openmetadata_user}
+      DB_USER_PASSWORD: ${DB_USER_PASSWORD:-openmetadata_password}
+      OM_DATABASE: ${OM_DATABASE:-openmetadata_db}
+      DB_PARAMS: allowPublicKeyRetrieval=true&useSSL=false&serverTimezone=UTC
+
+      # Search
+      SEARCH_TYPE: opensearch
+      ELASTICSEARCH_HOST: opensearch
+      ELASTICSEARCH_PORT: 9200
+      ELASTICSEARCH_SCHEME: http
+
+      # Auth
+      AUTHENTICATION_PROVIDER: basic
+      AUTHORIZER_ADMIN_PRINCIPALS: "[admin]"
+      AUTHENTICATION_PUBLIC_KEYS: "[http://localhost:8589/api/v1/system/config/jwks]"
+
+      # Pipeline service disabled for testing
+      PIPELINE_SERVICE_CLIENT_ENABLED: "false"
+
+      # Heap
+      OPENMETADATA_HEAP_OPTS: ${OPENMETADATA_HEAP_OPTS:--Xmx1G -Xms1G}
+    ports:
+      - "8589:8585"
+      - "8590:8586"
+    depends_on:
+      migrate:
+        condition: service_completed_successfully
+    networks:
+      - distributed-test-net
+    healthcheck:
+      test: ["CMD", "wget", "-q", "--spider", "http://localhost:8586/healthcheck"]
+      interval: 15s
+      timeout: 10s
+      retries: 10

docker/development/distributed-test/local/docker-compose-deps.yml

@@ -0,0 +1,70 @@
+#  Copyright 2021 Collate
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#  http://www.apache.org/licenses/LICENSE-2.0
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+# Dependencies-only compose file for local JVM debugging
+# Use this when running OM servers from your IDE
+
+version: "3.9"
+
+volumes:
+  mysql-data:
+  opensearch-data:
+
+networks:
+  distributed-test-net:
+    name: distributed_test_network
+    driver: bridge
+
+services:
+  # MySQL Database
+  mysql:
+    image: mysql:8.0
+    container_name: distributed_test_mysql
+    restart: unless-stopped
+    environment:
+      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD:-password}
+    command: >
+      --sort_buffer_size=10M
+      --max_connections=500
+    ports:
+      - "${MYSQL_PORT:-3306}:3306"
+    networks:
+      - distributed-test-net
+    healthcheck:
+      test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-uroot", "-p${MYSQL_ROOT_PASSWORD:-password}"]
+      interval: 10s
+      timeout: 5s
+      retries: 10
+    volumes:
+      - mysql-data:/var/lib/mysql
+      - ../config/mysql-init.sql:/docker-entrypoint-initdb.d/init.sql:ro
+
+  # OpenSearch
+  opensearch:
+    image: opensearchproject/opensearch:3.4.0
+    container_name: distributed_test_opensearch
+    restart: unless-stopped
+    environment:
+      - discovery.type=single-node
+      - plugins.security.disabled=true
+      - "OPENSEARCH_JAVA_OPTS=${OPENSEARCH_JAVA_OPTS:--Xms512m -Xmx512m}"
+    ports:
+      - "${OPENSEARCH_PORT:-9200}:9200"
+      - "9600:9600"
+    networks:
+      - distributed-test-net
+    healthcheck:
+      test: ["CMD-SHELL", "curl -s http://localhost:9200/_cluster/health | grep -qE '\"status\":\"(green|yellow)\"'"]
+      interval: 10s
+      timeout: 5s
+      retries: 10
+    volumes:
+      - opensearch-data:/usr/share/opensearch/data

docker/development/distributed-test/local/run-local-servers.sh

@@ -0,0 +1,140 @@
+#!/bin/bash
+# Run multiple OpenMetadata servers locally for debugging
+
+set -e
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_DIR="$(cd "$SCRIPT_DIR/../../../.." && pwd)"
+
+# Default: run all 3 servers
+SERVERS="${@:-1 2 3}"
+
+# Check if we should run migrations first
+RUN_MIGRATE=false
+if [ "$1" == "--migrate" ]; then
+    RUN_MIGRATE=true
+    shift
+    SERVERS="${@:-1 2 3}"
+fi
+
+echo "======================================"
+echo "Local Multi-Server Development Setup"
+echo "======================================"
+echo "Project dir: $PROJECT_DIR"
+echo "Servers to start: $SERVERS"
+echo ""
+
+# Check if JAR exists
+JAR_PATH=$(find "$PROJECT_DIR/openmetadata-service/target" -name "openmetadata-service-*.jar" -not -name "*sources*" -not -name "*javadoc*" 2>/dev/null | head -1)
+
+if [ -z "$JAR_PATH" ]; then
+    echo "OpenMetadata JAR not found. Building..."
+    cd "$PROJECT_DIR"
+    mvn clean package -DskipTests -pl openmetadata-service -am
+    JAR_PATH=$(find "$PROJECT_DIR/openmetadata-service/target" -name "openmetadata-service-*.jar" -not -name "*sources*" -not -name "*javadoc*" 2>/dev/null | head -1)
+fi
+
+if [ -z "$JAR_PATH" ]; then
+    echo "ERROR: Could not find or build openmetadata-service JAR"
+    exit 1
+fi
+
+echo "Using JAR: $JAR_PATH"
+echo ""
+
+# Check if dependencies are running
+echo "Checking dependencies..."
+if ! curl -s http://localhost:3306 >/dev/null 2>&1; then
+    if ! docker ps | grep -q distributed_test_mysql; then
+        echo "MySQL not running. Starting dependencies..."
+        cd "$SCRIPT_DIR"
+        docker compose -f docker-compose-deps.yml up -d
+        echo "Waiting for MySQL..."
+        until docker compose -f docker-compose-deps.yml exec -T mysql mysqladmin ping -h localhost -uroot -ppassword --silent 2>/dev/null; do
+            sleep 2
+        done
+        echo "MySQL ready."
+    fi
+fi
+
+if ! curl -s http://localhost:9200 >/dev/null 2>&1; then
+    echo "OpenSearch not running. Please start dependencies first:"
+    echo "  cd $SCRIPT_DIR && docker compose -f docker-compose-deps.yml up -d"
+    exit 1
+fi
+
+echo "Dependencies are running."
+echo ""
+
+# Run migrations if requested
+if [ "$RUN_MIGRATE" == "true" ]; then
+    echo "Running database migrations..."
+    cd "$PROJECT_DIR"
+    java -cp "$JAR_PATH" \
+        -Dloader.main=org.openmetadata.service.util.OpenMetadataSetup \
+        org.springframework.boot.loader.launch.PropertiesLauncher \
+        migrate --force
+    echo "Migrations complete."
+    echo ""
+fi
+
+# Function to start a server
+start_server() {
+    local server_num=$1
+    local config_file="$SCRIPT_DIR/server${server_num}.yaml"
+
+    if [ ! -f "$config_file" ]; then
+        echo "Config file not found: $config_file"
+        return 1
+    fi
+
+    local port=$((8583 + server_num * 2))
+    echo "Starting Server $server_num on port $port..."
+
+    cd "$PROJECT_DIR"
+
+    # Start in a new terminal window (macOS)
+    if [[ "$OSTYPE" == "darwin"* ]]; then
+        osascript -e "tell application \"Terminal\" to do script \"cd '$PROJECT_DIR' && java -Xmx1G -Xms512M -Ddw.logging.appenders[0].logFormat='[SERVER-$server_num] %level [%d{ISO8601,UTC}] [%t] %logger{5} - %msg%n' -jar '$JAR_PATH' server '$config_file'\""
+    # Linux with gnome-terminal
+    elif command -v gnome-terminal &> /dev/null; then
+        gnome-terminal -- bash -c "cd '$PROJECT_DIR' && java -Xmx1G -Xms512M -Ddw.logging.appenders[0].logFormat='[SERVER-$server_num] %level [%d{ISO8601,UTC}] [%t] %logger{5} - %msg%n' -jar '$JAR_PATH' server '$config_file'; exec bash"
+    # Linux with xterm
+    elif command -v xterm &> /dev/null; then
+        xterm -e "cd '$PROJECT_DIR' && java -Xmx1G -Xms512M -jar '$JAR_PATH' server '$config_file'" &
+    else
+        # Fallback: run in background
+        echo "No terminal emulator found. Running in background..."
+        java -Xmx1G -Xms512M \
+            -Ddw.logging.appenders[0].logFormat="[SERVER-$server_num] %level [%d{ISO8601,UTC}] [%t] %logger{5} - %msg%n" \
+            -jar "$JAR_PATH" server "$config_file" \
+            > "/tmp/om-server-$server_num.log" 2>&1 &
+        echo "Server $server_num started in background. Log: /tmp/om-server-$server_num.log"
+    fi
+}
+
+# Start requested servers
+for server in $SERVERS; do
+    start_server "$server"
+    sleep 2  # Stagger startup
+done
+
+echo ""
+echo "======================================"
+echo "Servers starting..."
+echo "======================================"
+echo ""
+echo "Server endpoints:"
+for server in $SERVERS; do
+    port=$((8583 + server * 2))
+    echo "  - Server $server: http://localhost:$port"
+done
+echo ""
+echo "For IDE debugging:"
+echo "  1. Open your IDE"
+echo "  2. Create a new Run Configuration"
+echo "  3. Main class: org.openmetadata.service.OpenMetadataApplication"
+echo "  4. Program arguments: server local/server1.yaml"
+echo "  5. VM options: -Xmx1G -Xms512M"
+echo "  6. Working directory: $PROJECT_DIR"
+echo ""

docker/development/distributed-test/local/server1.yaml

@@ -0,0 +1,639 @@
+# OpenMetadata Server 1 Configuration for Local Development
+# Run with: java -Dconfig=local/server1.yaml -jar openmetadata-service/target/openmetadata-service-*.jar server
+
+clusterName: distributed-test
+
+swagger:
+  resourcePackage: org.openmetadata.service.resources
+
+assets:
+  resourcePath: /assets/
+  uriPath: /
+
+basePath: ${BASE_PATH:-/}
+
+server:
+  applicationContextPath: /
+  rootPath: /api/*
+  applicationConnectors:
+    - type: http
+      bindHost: ${SERVER_HOST:-0.0.0.0}
+      port: ${SERVER_PORT:-8585}
+
+      # Jetty 12 URI Compliance - UNSAFE allows all special characters in entity names
+      # Required for backward compatibility with entity names containing /, ", etc.
+      uriCompliance: UNSAFE
+
+      # Jetty Acceptor and Selector threads for high concurrency
+      acceptorThreads: ${SERVER_ACCEPTOR_THREADS:-2}  # 1-2 per CPU core
+      selectorThreads: ${SERVER_SELECTOR_THREADS:-8}  # 2-4 per CPU core
+
+      # Connection settings - relaxed for Docker/local development
+      acceptQueueSize: ${SERVER_ACCEPT_QUEUE_SIZE:-256}  # OS-level connection backlog
+      idleTimeout: ${SERVER_IDLE_TIMEOUT:-60 seconds}  # Close idle connections (increased from 30s)
+
+      # Buffer sizes for better throughput
+      outputBufferSize: ${SERVER_OUTPUT_BUFFER_SIZE:-32KiB}
+      inputBufferSize: ${SERVER_INPUT_BUFFER_SIZE:-8KiB}
+      maxRequestHeaderSize: ${SERVER_MAX_REQUEST_HEADER_SIZE:-8KiB}
+      maxResponseHeaderSize: ${SERVER_MAX_RESPONSE_HEADER_SIZE:-8KiB}
+      headerCacheSize: ${SERVER_HEADER_CACHE_SIZE:-512B}  # Cache parsed headers (in bytes)
+
+      # Performance settings
+      useServerHeader: false  # Don't send server version header
+      useDateHeader: true
+      useForwardedHeaders: ${SERVER_USE_FORWARDED_HEADERS:-false}  # Enable if behind proxy
+
+      # Data rate limits (prevent slow loris attacks)
+      minRequestDataPerSecond: ${SERVER_MIN_REQUEST_DATA_RATE:-0B}  # 0B = disabled
+      minResponseDataPerSecond: ${SERVER_MIN_RESPONSE_DATA_RATE:-0B}  # 0B = disabled
+
+  adminConnectors:
+    - type: http
+      bindHost: ${SERVER_HOST:-0.0.0.0}
+      port: ${SERVER_ADMIN_PORT:-8586}
+      acceptorThreads: 1  # Admin endpoint needs minimal resources
+      selectorThreads: 1
+
+  # Response compression disabled for maximum throughput
+  gzip:
+    enabled: false
+
+  # Thread pool configuration
+  # With virtual threads enabled (Java 21+), these settings become less critical
+  # as blocking operations are handled efficiently by the JVM
+  maxThreads: ${SERVER_MAX_THREADS:-150}
+  minThreads: ${SERVER_MIN_THREADS:-100}
+  idleThreadTimeout: ${SERVER_IDLE_THREAD_TIMEOUT:-1 minute}
+
+  # Virtual Threads (Project Loom) - Recommended for Java 21+
+  # Jetty 12 uses AdaptiveExecutionStrategy to route blocking tasks to virtual threads
+  # while keeping non-blocking I/O on platform threads for optimal cache locality
+  enableVirtualThreads: ${SERVER_ENABLE_VIRTUAL_THREAD:-false}
+  # Note: maxQueuedRequests removed in Dropwizard 5.0/Jetty 12
+
+  # Request/Response logging (disable in production for performance)
+  # Set LOG_FORMAT=json for structured logs. The default text format preserves legacy output.
+  requestLog:
+    appenders:
+      - type: console
+        threshold: ${REQUEST_LOG_LEVEL:-ERROR}  # Only log errors by default
+        layout:
+          type: om-access-layout
+          format: ${LOG_FORMAT:-text}
+          appendLineSeparator: true
+          additionalFields:
+            server: server-1
+  
+  # Above configuration for running http is fine for dev and testing.
+  # For production setup, where UI app will hit apis through DPS it
+  # is strongly recommended to run https instead. Note that only
+  # keyStorePath and keyStorePassword are mandatory properties. Values
+  # for other properties are defaults
+  #server:
+  #applicationConnectors:
+  #  - type: https
+  #    port: 8585
+  #    keyStorePath: ./conf/keystore.jks
+  #    keyStorePassword: changeit
+  #    keyStoreType: JKS
+  #    keyStoreProvider:
+  #    trustStorePath: /path/to/file
+  #    trustStorePassword: changeit
+  #    trustStoreType: JKS
+  #    trustStoreProvider:
+  #    keyManagerPassword: changeit
+  #    needClientAuth: false
+  #    wantClientAuth:
+  #    certAlias: <alias>
+  #    crlPath: /path/to/file
+  #    enableCRLDP: false
+  #    enableOCSP: false
+  #    maxCertPathLength: (unlimited)
+  #    ocspResponderUrl: (none)
+  #    jceProvider: (none)
+  #    validateCerts: true
+  #    validatePeers: true
+  #    supportedProtocols: SSLv3
+  #    supportedCipherSuites: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
+  #    allowRenegotiation: true
+  #    endpointIdentificationAlgorithm: (none)
+
+  #adminConnectors:
+  #  - type: https
+  #    port: 8586
+  #    keyStorePath: ./conf/keystore.jks
+  #    keyStorePassword: changeit
+  #    keyStoreType: JKS
+  #    keyStoreProvider:
+  #    trustStorePath: /path/to/file
+  #    trustStorePassword: changeit
+  #    trustStoreType: JKS
+  #    trustStoreProvider:
+  #    keyManagerPassword: changeit
+  #    needClientAuth: false
+  #    wantClientAuth:
+  #    certAlias: <alias>
+  #    crlPath: /path/to/file
+  #    enableCRLDP: false
+  #    enableOCSP: false
+  #    maxCertPathLength: (unlimited)
+  #    ocspResponderUrl: (none)
+  #    jceProvider: (none)
+  #    validateCerts: true
+  #    validatePeers: true
+  #    supportedProtocols: SSLv3
+  #    supportedCipherSuites: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
+  #    allowRenegotiation: true
+  #    endpointIdentificationAlgorithm: (none)
+
+# Logging settings.
+# https://logback.qos.ch/manual/layouts.html#conversionWord
+# Set LOG_FORMAT=json for structured logs. The default text format preserves legacy output.
+logging:
+  level: ${LOG_LEVEL:-INFO}
+  appenders:
+    - type: console
+      threshold: INFO
+      layout:
+        type: om-event-layout
+        format: ${LOG_FORMAT:-text}
+        pattern: "[server-1] %level [%d{ISO8601,UTC}] [%t] %logger{5} - %msg%n"
+        appendLineSeparator: true
+        additionalFields:
+          server: server-1
+
+database:
+  # the name of the JDBC driver, mysql in our case
+  driverClass: ${DB_DRIVER_CLASS:-com.mysql.cj.jdbc.Driver}
+  # the username and password
+  user: ${DB_USER:-openmetadata_user}
+  password: ${DB_USER_PASSWORD:-openmetadata_password}
+  # the JDBC URL; the database is called openmetadata_db
+  url: jdbc:${DB_SCHEME:-mysql}://${DB_HOST:-localhost}:${DB_PORT:-3306}/${OM_DATABASE:-openmetadata_db}?${DB_PARAMS:-allowPublicKeyRetrieval=true&useSSL=false&serverTimezone=UTC}
+
+  # HikariCP Connection Pool Settings - Optimized for Performance
+  maxSize: ${DB_CONNECTION_POOL_MAX_SIZE:-100}  # Increased from 50 for better concurrency
+  minSize: ${DB_CONNECTION_POOL_MIN_SIZE:-20}   # Increased from 10 to reduce connection creation overhead
+  minimumIdle: ${DB_CONNECTION_POOL_MIN_IDLE:-20}  # HikariCP specific minimum idle connections
+  initialSize: ${DB_CONNECTION_POOL_INITIAL_SIZE:-20}  # Start with more connections ready
+  checkConnectionWhileIdle: ${DB_CONNECTION_CHECK_CONNECTION_WHILE_IDLE:-true}
+  checkConnectionOnBorrow: ${DB_CONNECTION_CHECK_CONNECTION_ON_BORROW:-false}  # Disable for performance
+  evictionInterval: ${DB_CONNECTION_EVICTION_INTERVAL:-5 minutes}
+  minIdleTime: ${DB_CONNECTION_MIN_IDLE_TIME:-5 minute}
+
+
+  # JDBC Driver Properties - Critical for Performance
+  # These work across both PostgreSQL and MySQL drivers
+  properties:
+    # HikariCP connection pool settings
+    connectionTimeout: ${DB_CONNECTION_TIMEOUT:-300000}  # 300 seconds
+    idleTimeout: ${DB_IDLE_TIMEOUT:-600000}  # 10 minutes
+    maxLifetime: ${DB_MAX_LIFETIME:-1800000}  # 30 minutes
+    leakDetectionThreshold: ${DB_LEAK_DETECTION_THRESHOLD:-600000}  # 10 minute
+    keepaliveTime: ${DB_KEEPALIVE_TIME:-0}  # 0 = disabled (set to 30000 for Aurora)
+    validationTimeout: ${DB_VALIDATION_TIMEOUT:-300000}  # 300 seconds
+    # PostgreSQL specific - these are ignored by MySQL driver
+    prepareThreshold: ${DB_PG_PREPARE_THRESHOLD:-1}  # Use prepared statements immediately
+    preparedStatementCacheQueries: ${DB_PG_PREP_STMT_CACHE_QUERIES:-500}  # Cache more statements
+    preparedStatementCacheSizeMiB: ${DB_PG_PREP_STMT_CACHE_SIZE_MB:-10}  # Larger cache
+    reWriteBatchedInserts: ${DB_PG_REWRITE_BATCHED_INSERTS:-true}  # Critical for batch performance
+    defaultRowFetchSize: ${DB_PG_DEFAULT_ROW_FETCH_SIZE:-1000}  # Fetch more rows at once
+    assumeMinServerVersion: ${DB_PG_ASSUME_MIN_SERVER_VERSION:-12}  # Skip version checks
+    ApplicationName: ${DB_PG_APPLICATION_NAME:-OpenMetadata}
+    loginTimeout: ${DB_PG_LOGIN_TIMEOUT:-300}  # Login timeout in seconds
+    postgresqlConnectTimeout: ${DB_POSTGRESQL_CONNECT_TIMEOUT:-60}  # Connection timeout in seconds
+    postgresqlSocketTimeout: ${DB_POSTGRESQL_SOCKET_TIMEOUT:-30000}  # Socket timeout in seconds (0 = infinite)
+
+    # Aurora PostgreSQL specific optimizations
+    loadBalanceHosts: ${DB_PG_LOAD_BALANCE_HOSTS:-false}  # Set to true for Aurora reader endpoints
+    hostRecheckSeconds: ${DB_PG_HOST_RECHECK_SECONDS:-10}  # How often to check host status
+    targetServerType: ${DB_PG_TARGET_SERVER_TYPE:-primary}  # primary, secondary, any, preferSecondary
+
+    # MySQL specific - these are ignored by PostgreSQL driver
+    rewriteBatchedStatements: ${DB_MYSQL_REWRITE_BATCHED_STATEMENTS:-true}  # Critical for MySQL batch
+    cachePrepStmts: ${DB_MYSQL_CACHE_PREP_STMTS:-true}
+    prepStmtCacheSize: ${DB_MYSQL_PREP_STMT_CACHE_SIZE:-500}
+    prepStmtCacheSqlLimit: ${DB_MYSQL_PREP_STMT_CACHE_SQL_LIMIT:-2048}
+    useServerPrepStmts: ${DB_MYSQL_USE_SERVER_PREP_STMTS:-true}
+    useLocalSessionState: ${DB_MYSQL_USE_LOCAL_SESSION_STATE:-true}
+    useLocalTransactionState: ${DB_MYSQL_USE_LOCAL_TRANSACTION_STATE:-true}
+    elideSetAutoCommits: ${DB_MYSQL_ELIDE_SET_AUTO_COMMITS:-true}
+    maintainTimeStats: ${DB_MYSQL_MAINTAIN_TIME_STATS:-false}
+    cacheResultSetMetadata: ${DB_MYSQL_CACHE_RESULT_SET_METADATA:-true}
+    cacheServerConfiguration: ${DB_MYSQL_CACHE_SERVER_CONFIG:-true}
+    tcpKeepAlive: ${DB_MYSQL_TCP_KEEP_ALIVE:-true}
+    tcpNoDelay: ${DB_MYSQL_TCP_NO_DELAY:-true}
+    mysqlConnectTimeout: ${DB_MYSQL_CONNECT_TIMEOUT:-60000}  # Connection timeout in milliseconds
+    mysqlSocketTimeout: ${DB_MYSQL_SOCKET_TIMEOUT:-30000000}  # Socket timeout in milliseconds (0 = infinite)
+
+objectStorage:
+  enabled: false
+  provider: NOOP
+  maxFileSize: 5242880
+
+migrationConfiguration:
+  flywayPath: "./bootstrap/sql/migrations/flyway"
+  nativePath: "./bootstrap/sql/migrations/native"
+  extensionPath: ""
+
+# Authorizer Configuration
+authorizerConfiguration:
+  className: ${AUTHORIZER_CLASS_NAME:-org.openmetadata.service.security.DefaultAuthorizer}
+  containerRequestFilter: ${AUTHORIZER_REQUEST_FILTER:-org.openmetadata.service.security.JwtFilter}
+  adminPrincipals: ${AUTHORIZER_ADMIN_PRINCIPALS:-[admin]}
+  allowedEmailRegistrationDomains: ${AUTHORIZER_ALLOWED_REGISTRATION_DOMAIN:-["all"]}
+  principalDomain: ${AUTHORIZER_PRINCIPAL_DOMAIN:-"open-metadata.org"}
+  allowedDomains: ${AUTHORIZER_ALLOWED_DOMAINS:-[]}
+  enforcePrincipalDomain: ${AUTHORIZER_ENFORCE_PRINCIPAL_DOMAIN:-false}
+  enableSecureSocketConnection : ${AUTHORIZER_ENABLE_SECURE_SOCKET:-false}
+  useRolesFromProvider: ${AUTHORIZER_USE_ROLES_FROM_PROVIDER:-false}
+
+authenticationConfiguration:
+  clientType: ${AUTHENTICATION_CLIENT_TYPE:-public}
+  provider: ${AUTHENTICATION_PROVIDER:-basic}
+  # This is used by auth provider provide response as either id_token or code
+  responseType: ${AUTHENTICATION_RESPONSE_TYPE:-id_token}
+  # This will only be valid when provider type specified is customOidc
+  providerName: ${CUSTOM_OIDC_AUTHENTICATION_PROVIDER_NAME:-""}
+  publicKeyUrls: ${AUTHENTICATION_PUBLIC_KEYS:-[http://localhost:8585/api/v1/system/config/jwks]}
+  tokenValidationAlgorithm: ${AUTHENTICATION_TOKEN_VALIDATION_ALGORITHM:-"RS256"}
+  authority: ${AUTHENTICATION_AUTHORITY:-https://accounts.google.com}
+  clientId: ${AUTHENTICATION_CLIENT_ID:-""}
+  callbackUrl: ${AUTHENTICATION_CALLBACK_URL:-""}
+  jwtPrincipalClaims: ${AUTHENTICATION_JWT_PRINCIPAL_CLAIMS:-[email,preferred_username,sub]}
+  jwtPrincipalClaimsMapping: ${AUTHENTICATION_JWT_PRINCIPAL_CLAIMS_MAPPING:-[]}
+  enableSelfSignup : ${AUTHENTICATION_ENABLE_SELF_SIGNUP:-true}
+  enableAutoRedirect: ${AUTHENTICATION_ENABLE_AUTO_REDIRECT:-false}
+  # Force secure flag on session cookies even when not using HTTPS directly.
+  # Enable this when running behind a proxy/load balancer that handles SSL termination.
+  # Default: false (secure flag only set when HTTPS is detected)
+  forceSecureSessionCookie: ${FORCE_SECURE_SESSION_COOKIE:-false}
+  sessionExpiry: ${AUTHENTICATION_SESSION_EXPIRY:-"604800"} # 7 days; applies to all auth providers
+  maxActiveSessionsPerUser: ${AUTHENTICATION_MAX_ACTIVE_SESSIONS_PER_USER:-5}
+  oidcConfiguration:
+    id: ${OIDC_CLIENT_ID:-""}
+    type: ${OIDC_TYPE:-""} # google, azure etc.
+    secret: ${OIDC_CLIENT_SECRET:-""}
+    scope: ${OIDC_SCOPE:-"openid email profile"}
+    discoveryUri: ${OIDC_DISCOVERY_URI:-""}
+    useNonce: ${OIDC_USE_NONCE:-true}
+    preferredJwsAlgorithm: ${OIDC_PREFERRED_JWS:-"RS256"}
+    responseType: ${OIDC_RESPONSE_TYPE:-"code"}
+    disablePkce: ${OIDC_DISABLE_PKCE:-true}
+    callbackUrl: ${OIDC_CALLBACK:-"http://localhost:8585/callback"}
+    serverUrl: ${OIDC_SERVER_URL:-"http://localhost:8585"}
+    clientAuthenticationMethod: ${OIDC_CLIENT_AUTH_METHOD:-"client_secret_post"}
+    tenant: ${OIDC_TENANT:-""}
+    maxClockSkew: ${OIDC_MAX_CLOCK_SKEW:-""}
+    tokenValidity: ${OIDC_OM_REFRESH_TOKEN_VALIDITY:-"3600"} # in seconds
+    customParams: ${OIDC_CUSTOM_PARAMS:-}
+    maxAge: ${OIDC_MAX_AGE:-"0"}
+    prompt: ${OIDC_PROMPT_TYPE:-"consent"}
+    sessionExpiry: ${OIDC_SESSION_EXPIRY:-"604800"} #7 days
+  samlConfiguration:
+    debugMode: ${SAML_DEBUG_MODE:-false}
+    idp:
+      entityId: ${SAML_IDP_ENTITY_ID:-""}
+      ssoLoginUrl: ${SAML_IDP_SSO_LOGIN_URL:-""}
+      idpX509Certificate: ${SAML_IDP_CERTIFICATE:-""}
+      nameId: ${SAML_IDP_NAME_ID:-"urn:oasis:names:tc:SAML:2.0:nameid-format:emailAddress"}
+    sp:
+      entityId: ${SAML_SP_ENTITY_ID:-"http://localhost:8585/api/v1/saml/metadata"}
+      acs: ${SAML_SP_ACS:-"http://localhost:8585/api/v1/saml/acs"}
+      spX509Certificate: ${SAML_SP_CERTIFICATE:-""}
+      spPrivateKey: ${SAML_SP_PRIVATE_KEY:-""}
+      callback: ${SAML_SP_CALLBACK:-"http://localhost:8585/saml/callback"}
+    security:
+      strictMode: ${SAML_STRICT_MODE:-false}
+      validateXml: ${SAML_VALIDATE_XML:-false}
+      tokenValidity: ${SAML_SP_TOKEN_VALIDITY:-"3600"}
+      sendEncryptedNameId: ${SAML_SEND_ENCRYPTED_NAME_ID:-false}
+      sendSignedAuthRequest: ${SAML_SEND_SIGNED_AUTH_REQUEST:-false}
+      signSpMetadata: ${SAML_SIGNED_SP_METADATA:-false}
+      wantMessagesSigned: ${SAML_WANT_MESSAGE_SIGNED:-false}
+      wantAssertionsSigned: ${SAML_WANT_ASSERTION_SIGNED:-false}
+      wantAssertionEncrypted: ${SAML_WANT_ASSERTION_ENCRYPTED:-false}
+      keyStoreFilePath: ${SAML_KEYSTORE_FILE_PATH:-""}
+      keyStoreAlias: ${SAML_KEYSTORE_ALIAS:-""}
+      keyStorePassword: ${SAML_KEYSTORE_PASSWORD:-""}
+  ldapConfiguration:
+    host: ${AUTHENTICATION_LDAP_HOST:-}
+    port: ${AUTHENTICATION_LDAP_PORT:-}
+    dnAdminPrincipal: ${AUTHENTICATION_LOOKUP_ADMIN_DN:-""}
+    dnAdminPassword: ${AUTHENTICATION_LOOKUP_ADMIN_PWD:-""}
+    userBaseDN: ${AUTHENTICATION_USER_LOOKUP_BASEDN:-""}
+    groupBaseDN: ${AUTHENTICATION_GROUP_LOOKUP_BASEDN:-""}
+    roleAdminName: ${AUTHENTICATION_USER_ROLE_ADMIN_NAME:-}
+    allAttributeName: ${AUTHENTICATION_USER_ALL_ATTR:-}
+    mailAttributeName: ${AUTHENTICATION_USER_MAIL_ATTR:-}
+    usernameAttributeName: ${AUTHENTICATION_USER_NAME_ATTR:-}
+    groupAttributeName: ${AUTHENTICATION_USER_GROUP_ATTR:-}
+    groupAttributeValue: ${AUTHENTICATION_USER_GROUP_ATTR_VALUE:-}
+    groupMemberAttributeName: ${AUTHENTICATION_USER_GROUP_MEMBER_ATTR:-}
+    #the mapping of roles to LDAP groups
+    authRolesMapping: ${AUTH_ROLES_MAPPING:-""}
+    authReassignRoles: ${AUTH_REASSIGN_ROLES:-[]}
+    #optional
+    maxPoolSize: ${AUTHENTICATION_LDAP_POOL_SIZE:-3}
+    sslEnabled: ${AUTHENTICATION_LDAP_SSL_ENABLED:-}
+    truststoreConfigType: ${AUTHENTICATION_LDAP_TRUSTSTORE_TYPE:-TrustAll}
+    trustStoreConfig:
+      customTrustManagerConfig:
+        trustStoreFilePath: ${AUTHENTICATION_LDAP_TRUSTSTORE_PATH:-}
+        trustStoreFilePassword: ${AUTHENTICATION_LDAP_KEYSTORE_PASSWORD:-}
+        trustStoreFileFormat: ${AUTHENTICATION_LDAP_SSL_KEY_FORMAT:-}
+        verifyHostname: ${AUTHENTICATION_LDAP_SSL_VERIFY_CERT_HOST:-}
+        examineValidityDates: ${AUTHENTICATION_LDAP_EXAMINE_VALIDITY_DATES:-}
+      hostNameConfig:
+        allowWildCards: ${AUTHENTICATION_LDAP_ALLOW_WILDCARDS:-}
+        acceptableHostNames: ${AUTHENTICATION_LDAP_ALLOWED_HOSTNAMES:-[]}
+      jvmDefaultConfig:
+        verifyHostname: ${AUTHENTICATION_LDAP_SSL_VERIFY_CERT_HOST:-}
+      trustAllConfig:
+        examineValidityDates: ${AUTHENTICATION_LDAP_EXAMINE_VALIDITY_DATES:-true}
+
+jwtTokenConfiguration:
+  rsapublicKeyFilePath: ${RSA_PUBLIC_KEY_FILE_PATH:-"./conf/public_key.der"}
+  rsaprivateKeyFilePath: ${RSA_PRIVATE_KEY_FILE_PATH:-"./conf/private_key.der"}
+  jwtissuer: ${JWT_ISSUER:-"open-metadata.org"}
+  keyId: ${JWT_KEY_ID:-"Gb389a-9f76-gdjs-a92j-0242bk94356"}
+
+elasticsearch:
+  searchType: ${SEARCH_TYPE:- "opensearch"}
+  # Single host or comma-separated list for multiple hosts
+  # Examples: "localhost" or "es-node1:9200,es-node2:9200,es-node3:9200"
+  host: ${ELASTICSEARCH_HOST:-localhost}
+  port: ${ELASTICSEARCH_PORT:-9200}
+  scheme: ${ELASTICSEARCH_SCHEME:-http}
+  username: ${ELASTICSEARCH_USER:-""}
+  password: ${ELASTICSEARCH_PASSWORD:-""}
+  clusterAlias: ${ELASTICSEARCH_CLUSTER_ALIAS:-""}
+  truststorePath: ${ELASTICSEARCH_TRUST_STORE_PATH:-""}
+  truststorePassword: ${ELASTICSEARCH_TRUST_STORE_PASSWORD:-""}
+  connectionTimeoutSecs: ${ELASTICSEARCH_CONNECTION_TIMEOUT_SECS:-10}  # Increased from 5s for Docker networks
+  socketTimeoutSecs: ${ELASTICSEARCH_SOCKET_TIMEOUT_SECS:-120}  # Increased from 60s for slow queries
+  keepAliveTimeoutSecs: ${ELASTICSEARCH_KEEP_ALIVE_TIMEOUT_SECS:-600}
+  # Connection pool settings for better load balancing and performance
+  maxConnTotal: ${ELASTICSEARCH_MAX_CONN_TOTAL:-30}  # Total connections across all hosts
+  maxConnPerRoute: ${ELASTICSEARCH_MAX_CONN_PER_ROUTE:-10}  # Max connections per host
+  batchSize: ${ELASTICSEARCH_BATCH_SIZE:-100}
+  payLoadSize: ${ELASTICSEARCH_PAYLOAD_BYTES_SIZE:-10485760}
+  searchIndexMappingLanguage : ${ELASTICSEARCH_INDEX_MAPPING_LANG:-EN}
+  searchIndexFactoryClassName : org.openmetadata.service.search.SearchIndexFactory
+  # AWS IAM Authentication for OpenSearch (only applicable when searchType is "opensearch")
+  # Uses standard AWS environment variables: https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-envvars.html
+  # IAM auth is automatically enabled when AWS_DEFAULT_REGION is set
+  # Credentials: Use AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY or rely on AWS SDK default credential provider chain
+  aws:
+    enabled: ${SEARCH_AWS_IAM_AUTH_ENABLED:-false}
+    region: ${AWS_DEFAULT_REGION:-""}
+    accessKeyId: ${AWS_ACCESS_KEY_ID:-""}
+    secretAccessKey: ${AWS_SECRET_ACCESS_KEY:-""}
+    sessionToken: ${AWS_SESSION_TOKEN:-""}
+    serviceName: ${SEARCH_AWS_SERVICE_NAME:-"es"}  # Use "es" for OpenSearch, "aoss" for OpenSearch Serverless
+  naturalLanguageSearch:
+    enabled: false
+    embeddingProvider: ${EMBEDDING_PROVIDER:-bedrock}
+    providerClass: ${NATURAL_LANGUAGE_SEARCH_PROVIDER_CLASS:-org.openmetadata.service.search.nlq.NoOpNLQService}
+    bedrock:
+      awsConfig:
+        enabled: false
+        region: ${AWS_DEFAULT_REGION:-""}
+        accessKeyId: ${AWS_ACCESS_KEY_ID:-""}
+        secretAccessKey: ${AWS_SECRET_ACCESS_KEY:-""}
+        sessionToken: ${AWS_SESSION_TOKEN:-""}
+      modelId: ${AWS_BEDROCK_MODEL_ID:-""}
+      embeddingModelId: ${AWS_BEDROCK_EMBED_MODEL_ID:-""}
+      embeddingDimension: ${AWS_BEDROCK_EMBEDDING_DIMENSION:-""}
+
+eventMonitoringConfiguration:
+  eventMonitor: ${EVENT_MONITOR:-prometheus}  # Possible values are "prometheus", "cloudwatch"
+  batchSize: ${EVENT_MONITOR_BATCH_SIZE:-10}
+  pathPattern: ${EVENT_MONITOR_PATH_PATTERN:-["/api/v1/tables/*", "/api/v1/health-check"]}
+  latency: ${EVENT_MONITOR_LATENCY:-[0.99, 0.90]} # For value p99=0.99, p90=0.90, p50=0.50 etc.
+  servicesHealthCheckInterval: ${EVENT_MONITOR_SERVICES_HEALTH_CHECK_INTERVAL:-300}
+  # it will use the default auth provider for AWS services if parameters are not set
+  # parameters:
+  #   region: ${OM_MONITOR_REGION:-""}
+  #   accessKeyId: ${OM_MONITOR_ACCESS_KEY_ID:-""}
+  #   secretAccessKey: ${OM_MONITOR_ACCESS_KEY:-""}
+
+eventHandlerConfiguration:
+  eventHandlerClassNames:
+    - "org.openmetadata.service.events.AuditEventHandler"
+    - "org.openmetadata.service.events.ChangeEventHandler"
+
+pipelineServiceClientConfiguration:
+  enabled: ${PIPELINE_SERVICE_CLIENT_ENABLED:-true}
+  # If we don't need this, set "org.openmetadata.service.clients.pipeline.noop.NoopClient"
+  className: ${PIPELINE_SERVICE_CLIENT_CLASS_NAME:-"org.openmetadata.service.clients.pipeline.airflow.AirflowRESTClient"}
+  apiEndpoint: ${PIPELINE_SERVICE_CLIENT_ENDPOINT:-http://localhost:8080}
+  metadataApiEndpoint: ${SERVER_HOST_API_URL:-http://localhost:8585/api}
+  ingestionIpInfoEnabled: ${PIPELINE_SERVICE_IP_INFO_ENABLED:-false}
+  hostIp: ${PIPELINE_SERVICE_CLIENT_HOST_IP:-""}
+  healthCheckInterval: ${PIPELINE_SERVICE_CLIENT_HEALTH_CHECK_INTERVAL:-300}
+  # This SSL information is about the OpenMetadata server.
+  # It will be picked up from the pipelineServiceClient to use/ignore SSL when connecting to the OpenMetadata server.
+  verifySSL: ${PIPELINE_SERVICE_CLIENT_VERIFY_SSL:-"no-ssl"} # Possible values are "no-ssl", "ignore", "validate"
+  sslConfig:
+    certificatePath: ${PIPELINE_SERVICE_CLIENT_SSL_CERT_PATH:-""} # Local path for the Pipeline Service Client
+  logStorageConfiguration:
+    type: ${PIPELINE_SERVICE_CLIENT_LOG_TYPE:-"default"} # Possible values are "default", "s3"
+    enabled: ${PIPELINE_SERVICE_CLIENT_LOG_ENABLED:-false} # Enable it for pipelines deployed in the server
+    # if type is s3, provide the following configuration
+    bucketName: ${PIPELINE_SERVICE_CLIENT_LOG_BUCKET_NAME:-""}
+    # optional path within the bucket to store the logs
+    prefix: ${PIPELINE_SERVICE_CLIENT_LOG_PREFIX:-""}
+    enableServerSideEncryption: ${PIPELINE_SERVICE_CLIENT_LOG_SSE_ENABLED:-false}
+    sseAlgorithm: ${PIPELINE_SERVICE_CLIENT_LOG_SSE_ALGORITHM:-"AES256"} # Allowed values: "AES256" or "aws:kms"
+    kmsKeyId: ${PIPELINE_SERVICE_CLIENT_LOG_KMS_KEY_ID:-""} # Required only if sseAlgorithm is "aws:kms"
+    awsConfig:
+      enabled: ${PIPELINE_SERVICE_CLIENT_AWS_IAM_AUTH_ENABLED:-false}
+      awsAccessKeyId: ${PIPELINE_SERVICE_CLIENT_LOG_AWS_ACCESS_KEY_ID:-""}
+      awsSecretAccessKey: ${PIPELINE_SERVICE_CLIENT_LOG_AWS_SECRET_ACCESS_KEY:-""}
+      awsRegion: ${PIPELINE_SERVICE_CLIENT_LOG_REGION:-""}
+      awsSessionToken: ${PIPELINE_SERVICE_CLIENT_LOG_AWS_SESSION_TOKEN:-""}
+      endPointURL: ${PIPELINE_SERVICE_CLIENT_LOG_AWS_ENDPOINT_URL:-""} # port forward localhost:9000 for minio
+
+  # Secrets Manager Loader: specify to the Ingestion Framework how to load the SM credentials from its env
+  # Supported: noop, airflow, env
+  secretsManagerLoader: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-"noop"}
+
+  # Default required parameters for Airflow as Pipeline Service Client
+  parameters:
+    ## Airflow parameters
+    username: ${AIRFLOW_USERNAME:-admin}
+    password: ${AIRFLOW_PASSWORD:-admin}
+    timeout: ${AIRFLOW_TIMEOUT:-10}
+    # If we need to use SSL to reach Airflow
+    truststorePath: ${AIRFLOW_TRUST_STORE_PATH:-""}
+    truststorePassword: ${AIRFLOW_TRUST_STORE_PASSWORD:-""}
+    ## Kubernetes client parameters
+    namespace: ${K8S_NAMESPACE:-"openmetadata-pipelines"}
+    ingestionImage: ${K8S_INGESTION_IMAGE:-"docker.getcollate.io/openmetadata/ingestion-base:latest"}
+    imagePullPolicy: ${K8S_IMAGE_PULL_POLICY:-"IfNotPresent"}
+    imagePullSecrets: ${K8S_IMAGE_PULL_SECRETS:-""}
+    serviceAccountName: ${K8S_SERVICE_ACCOUNT_NAME:-"openmetadata-ingestion"}
+    # Resources configuration
+    resources:
+      limits:
+        cpu: ${K8S_LIMITS_CPU:-"2"}
+        memory: ${K8S_LIMITS_MEMORY:-"4Gi"}
+      requests:
+        cpu: ${K8S_REQUESTS_CPU:-"500m"}
+        memory: ${K8S_REQUESTS_MEMORY:-"1Gi"}
+    ttlSecondsAfterFinished: ${K8S_TTL_SECONDS_AFTER_FINISHED:-604800}
+    activeDeadlineSeconds: ${K8S_ACTIVE_DEADLINE_SECONDS:-604800}
+    backoffLimit: ${K8S_BACKOFF_LIMIT:-3}
+    successfulJobsHistoryLimit: ${K8S_SUCCESSFUL_JOBS_HISTORY_LIMIT:-3}
+    failedJobsHistoryLimit: ${K8S_FAILED_JOBS_HISTORY_LIMIT:-1}
+    nodeSelector: ${K8S_NODE_SELECTOR:-""}
+    runAsUser: ${K8S_RUN_AS_USER:-1000}
+    runAsGroup: ${K8S_RUN_AS_GROUP:-1000}
+    fsGroup: ${K8S_FS_GROUP:-1000}
+    runAsNonRoot: ${K8S_RUN_AS_NON_ROOT:-"true"}
+    extraEnvVars: ${K8S_EXTRA_ENV_VARS:-[]}
+    podAnnotations: ${K8S_POD_ANNOTATIONS:-""}
+    useOMJobOperator: ${USE_OMJOB_OPERATOR:-"true"}
+
+
+# no_encryption_at_rest is the default value, and it does what it says. Please read the manual on how
+# to secure your instance of OpenMetadata with TLS and encryption at rest.
+fernetConfiguration:
+  fernetKey: ${FERNET_KEY:-jJ/9sz0g0OHxsfxOoSfdFdmk3ysNmPRnH3TUAbz3IHA=}
+
+secretsManagerConfiguration:
+  secretsManager: ${SECRET_MANAGER:-db} # Possible values are "db", "managed-aws","aws", "managed-aws-ssm", "aws-ssm", "managed-azure-kv", "azure-kv", "in-memory", "gcp", "kubernetes"
+  prefix: ${SECRET_MANAGER_PREFIX:-""} # Define the secret key ID as /<prefix>/<clusterName>/<key>
+  tags: ${SECRET_MANAGER_TAGS:-[]} # Add tags to the created resource. Format is `[key1:value1,key2:value2,...]`
+  # it will use the default auth provider for the secrets' manager service if parameters are not set
+  parameters:
+    ## For AWS
+    accessKeyId: ${OM_SM_ACCESS_KEY_ID:-""}
+    secretAccessKey: ${OM_SM_ACCESS_KEY:-""}
+    ## For Azure Key Vault
+    clientId: ${OM_SM_CLIENT_ID:-""}
+    clientSecret: ${OM_SM_CLIENT_SECRET:-""}
+    tenantId: ${OM_SM_TENANT_ID:-""}
+    vaultName: ${OM_SM_VAULT_NAME:-""}
+    ## For GCP
+    projectId: ${OM_SM_PROJECT_ID:-""}
+    ## For Kubernetes
+    namespace: ${OM_SM_NAMESPACE:-"default"}
+    kubeconfigPath: ${OM_SM_KUBECONFIG_PATH:-""}
+    inCluster: ${OM_SM_IN_CLUSTER:-"false"}
+
+health:
+  delayedShutdownHandlerEnabled: true
+  shutdownWaitPeriod: 1s
+  healthChecks:
+    - name: OpenMetadataServerHealthCheck
+      critical: true
+      schedule:
+        checkInterval: 2500ms
+        downtimeInterval: 10s
+        failureAttempts: 2
+        successAttempts: 1
+
+limits:
+  enable: ${LIMITS_ENABLED:-false}
+  className: ${LIMITS_CLASS_NAME:-"org.openmetadata.service.limits.DefaultLimits"}
+  limitsConfigFile: ${LIMITS_CONFIG_FILE:-""}
+
+# Bulk Operation Configuration
+# Controls parallelism and resource usage for bulk API operations (e.g., bulk import/export)
+# Uses a bounded thread pool to prevent connection pool exhaustion
+bulkOperation:
+  # Max threads for bulk operations (recommendations: 2 vCore=5-8, 4 vCore=8-15, 8 vCore=15-25)
+  maxThreads: ${BULK_OPERATION_MAX_THREADS:-10}
+  # Max queued operations before rejection (returns 503)
+  queueSize: ${BULK_OPERATION_QUEUE_SIZE:-1000}
+  # Timeout in seconds for entire bulk operation
+  timeoutSeconds: ${BULK_OPERATION_TIMEOUT_SECONDS:-300}
+
+web:
+  uriPath: ${WEB_CONF_URI_PATH:-"/api"}
+  hsts:
+    enabled: ${WEB_CONF_HSTS_ENABLED:-false}
+    maxAge: ${WEB_CONF_HSTS_MAX_AGE:-"365 days"}
+    includeSubDomains: ${WEB_CONF_HSTS_INCLUDE_SUBDOMAINS:-"true"}
+    preload: ${WEB_CONF_HSTS_PRELOAD:-"true"}
+  frame-options:
+    enabled: ${WEB_CONF_FRAME_OPTION_ENABLED:-false}
+    option: ${WEB_CONF_FRAME_OPTION:-"SAMEORIGIN"}
+    origin: ${WEB_CONF_FRAME_ORIGIN:-""}
+  content-type-options:
+    enabled: ${WEB_CONF_CONTENT_TYPE_OPTIONS_ENABLED:-false}
+  xss-protection:
+    enabled: ${WEB_CONF_XSS_PROTECTION_ENABLED:-false}
+    on: ${WEB_CONF_XSS_PROTECTION_ON:-true}
+    block: ${WEB_CONF_XSS_PROTECTION_BLOCK:-true}
+  csp:
+    enabled: ${WEB_CONF_XSS_CSP_ENABLED:-false}
+    policy: ${WEB_CONF_XSS_CSP_POLICY:-"default-src 'self'"}
+    reportOnlyPolicy: ${WEB_CONF_XSS_CSP_REPORT_ONLY_POLICY:-""}
+  referrer-policy:
+    enabled: ${WEB_CONF_REFERRER_POLICY_ENABLED:-false}
+    option: ${WEB_CONF_REFERRER_POLICY_OPTION:-"SAME_ORIGIN"}
+  permission-policy:
+    enabled: ${WEB_CONF_PERMISSION_POLICY_ENABLED:-false}
+    option: ${WEB_CONF_PERMISSION_POLICY_OPTION:-""}
+  cache-control: ${WEB_CONF_CACHE_CONTROL:-""}
+  pragma: ${WEB_CONF_PRAGMA:-""}
+
+operationalConfig:
+  enable: ${OPERATIONAL_CONFIG_ENABLED:-true}
+  operationsConfigFile: ${OPERATIONAL_CONFIG_FILE:-"./conf/operations.yaml"}
+
+rdf:
+  enabled: ${RDF_ENABLED:-false}
+  baseUri: ${RDF_BASE_URI:-"https://open-metadata.org/"}
+  storageType: ${RDF_STORAGE_TYPE:-"FUSEKI"}
+  remoteEndpoint: ${RDF_ENDPOINT:-"http://localhost:3030/openmetadata"}
+  username: ${RDF_REMOTE_USERNAME:-"admin"}
+  password: ${RDF_REMOTE_PASSWORD:-"admin"}
+  dataset: ${RDF_DATASET:-"openmetadata"}
+
+# Cache Configuration
+# Caching layer for entity metadata, relationships, and tag usage to reduce database load
+# Default: Disabled (uses NoopCacheProvider)
+cache:
+  # Cache provider: none (default) or redis
+  provider: ${CACHE_PROVIDER:-none}
+
+  # TTL (Time To Live) settings in seconds
+  entityTtlSeconds: ${CACHE_ENTITY_TTL:-172800}             # 48 hour for entities
+  relationshipTtlSeconds: ${CACHE_RELATIONSHIP_TTL:-172800} # 48 hour for relationships
+  tagTtlSeconds: ${CACHE_TAG_TTL:-172800}                   # 48 hour for tags
+
+  # Redis configuration
+  redis:
+    # Redis connection URL
+    # Standalone: redis://localhost:6379
+    # AWS ElastiCache: redis://my-cluster.abc123.cache.amazonaws.com:6379
+    url: ${CACHE_REDIS_URL:-redis://localhost:6379}
+    authType: ${CACHE_REDIS_AUTH_TYPE:-NONE}
+    database: ${CACHE_REDIS_DATABASE:-0}  # Redis database index (0-15)
+
+    # Authentication for standalone Redis
+    username: ${CACHE_REDIS_USERNAME:-}
+    passwordRef: ${CACHE_REDIS_PASSWORD:-}  # Reference to password in secrets manager
+    useSSL: ${CACHE_REDIS_USE_SSL:-false}
+
+    # Key namespace prefix (useful for multi-tenant deployments)
+    keyspace: ${CACHE_REDIS_KEYSPACE:-"om:prod"}
+
+    # Connection pool settings
+    poolSize: ${CACHE_REDIS_POOL_SIZE:-64}
+    connectTimeoutMs: ${CACHE_REDIS_CONNECT_TIMEOUT:-2000}
+
+    # AWS ElastiCache IAM Authentication (only if using ElastiCache)
+    aws:
+      enabled: ${CACHE_REDIS_AWS_IAM_AUTH_ENABLED:-false}
+      region: ${CACHE_REDIS_AWS_REGION:-""}
+      useInstanceProfile: ${CACHE_REDIS_AWS_INSTANCE_PROFILE:-true}
+      # If not using instance profile, provide credentials:
+      accessKeyId: ${AWS_ACCESS_KEY_ID:-""}
+      secretAccessKey: ${AWS_SECRET_ACCESS_KEY:-""}
+      tokenRefreshIntervalSeconds: ${CACHE_REDIS_TOKEN_REFRESH:-900}  # 15 minutes

docker/development/distributed-test/local/server2.yaml

@@ -0,0 +1,639 @@
+# OpenMetadata Server 2 Configuration for Local Development
+# Run with: java -Dconfig=local/server2.yaml -jar openmetadata-service/target/openmetadata-service-*.jar server
+
+clusterName: distributed-test
+
+swagger:
+  resourcePackage: org.openmetadata.service.resources
+
+assets:
+  resourcePath: /assets/
+  uriPath: /
+
+basePath: ${BASE_PATH:-/}
+
+server:
+  applicationContextPath: /
+  rootPath: /api/*
+  applicationConnectors:
+    - type: http
+      bindHost: ${SERVER_HOST:-0.0.0.0}
+      port: ${SERVER_PORT:-8587}
+
+      # Jetty 12 URI Compliance - UNSAFE allows all special characters in entity names
+      # Required for backward compatibility with entity names containing /, ", etc.
+      uriCompliance: UNSAFE
+
+      # Jetty Acceptor and Selector threads for high concurrency
+      acceptorThreads: ${SERVER_ACCEPTOR_THREADS:-2}  # 1-2 per CPU core
+      selectorThreads: ${SERVER_SELECTOR_THREADS:-8}  # 2-4 per CPU core
+
+      # Connection settings - relaxed for Docker/local development
+      acceptQueueSize: ${SERVER_ACCEPT_QUEUE_SIZE:-256}  # OS-level connection backlog
+      idleTimeout: ${SERVER_IDLE_TIMEOUT:-60 seconds}  # Close idle connections (increased from 30s)
+
+      # Buffer sizes for better throughput
+      outputBufferSize: ${SERVER_OUTPUT_BUFFER_SIZE:-32KiB}
+      inputBufferSize: ${SERVER_INPUT_BUFFER_SIZE:-8KiB}
+      maxRequestHeaderSize: ${SERVER_MAX_REQUEST_HEADER_SIZE:-8KiB}
+      maxResponseHeaderSize: ${SERVER_MAX_RESPONSE_HEADER_SIZE:-8KiB}
+      headerCacheSize: ${SERVER_HEADER_CACHE_SIZE:-512B}  # Cache parsed headers (in bytes)
+
+      # Performance settings
+      useServerHeader: false  # Don't send server version header
+      useDateHeader: true
+      useForwardedHeaders: ${SERVER_USE_FORWARDED_HEADERS:-false}  # Enable if behind proxy
+
+      # Data rate limits (prevent slow loris attacks)
+      minRequestDataPerSecond: ${SERVER_MIN_REQUEST_DATA_RATE:-0B}  # 0B = disabled
+      minResponseDataPerSecond: ${SERVER_MIN_RESPONSE_DATA_RATE:-0B}  # 0B = disabled
+
+  adminConnectors:
+    - type: http
+      bindHost: ${SERVER_HOST:-0.0.0.0}
+      port: ${SERVER_ADMIN_PORT:-8588}
+      acceptorThreads: 1  # Admin endpoint needs minimal resources
+      selectorThreads: 1
+
+  # Response compression disabled for maximum throughput
+  gzip:
+    enabled: false
+
+  # Thread pool configuration
+  # With virtual threads enabled (Java 21+), these settings become less critical
+  # as blocking operations are handled efficiently by the JVM
+  maxThreads: ${SERVER_MAX_THREADS:-150}
+  minThreads: ${SERVER_MIN_THREADS:-100}
+  idleThreadTimeout: ${SERVER_IDLE_THREAD_TIMEOUT:-1 minute}
+
+  # Virtual Threads (Project Loom) - Recommended for Java 21+
+  # Jetty 12 uses AdaptiveExecutionStrategy to route blocking tasks to virtual threads
+  # while keeping non-blocking I/O on platform threads for optimal cache locality
+  enableVirtualThreads: ${SERVER_ENABLE_VIRTUAL_THREAD:-false}
+  # Note: maxQueuedRequests removed in Dropwizard 5.0/Jetty 12
+
+  # Request/Response logging (disable in production for performance)
+  # Set LOG_FORMAT=json for structured logs. The default text format preserves legacy output.
+  requestLog:
+    appenders:
+      - type: console
+        threshold: ${REQUEST_LOG_LEVEL:-ERROR}  # Only log errors by default
+        layout:
+          type: om-access-layout
+          format: ${LOG_FORMAT:-text}
+          appendLineSeparator: true
+          additionalFields:
+            server: server-2
+  
+  # Above configuration for running http is fine for dev and testing.
+  # For production setup, where UI app will hit apis through DPS it
+  # is strongly recommended to run https instead. Note that only
+  # keyStorePath and keyStorePassword are mandatory properties. Values
+  # for other properties are defaults
+  #server:
+  #applicationConnectors:
+  #  - type: https
+  #    port: 8585
+  #    keyStorePath: ./conf/keystore.jks
+  #    keyStorePassword: changeit
+  #    keyStoreType: JKS
+  #    keyStoreProvider:
+  #    trustStorePath: /path/to/file
+  #    trustStorePassword: changeit
+  #    trustStoreType: JKS
+  #    trustStoreProvider:
+  #    keyManagerPassword: changeit
+  #    needClientAuth: false
+  #    wantClientAuth:
+  #    certAlias: <alias>
+  #    crlPath: /path/to/file
+  #    enableCRLDP: false
+  #    enableOCSP: false
+  #    maxCertPathLength: (unlimited)
+  #    ocspResponderUrl: (none)
+  #    jceProvider: (none)
+  #    validateCerts: true
+  #    validatePeers: true
+  #    supportedProtocols: SSLv3
+  #    supportedCipherSuites: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
+  #    allowRenegotiation: true
+  #    endpointIdentificationAlgorithm: (none)
+
+  #adminConnectors:
+  #  - type: https
+  #    port: 8586
+  #    keyStorePath: ./conf/keystore.jks
+  #    keyStorePassword: changeit
+  #    keyStoreType: JKS
+  #    keyStoreProvider:
+  #    trustStorePath: /path/to/file
+  #    trustStorePassword: changeit
+  #    trustStoreType: JKS
+  #    trustStoreProvider:
+  #    keyManagerPassword: changeit
+  #    needClientAuth: false
+  #    wantClientAuth:
+  #    certAlias: <alias>
+  #    crlPath: /path/to/file
+  #    enableCRLDP: false
+  #    enableOCSP: false
+  #    maxCertPathLength: (unlimited)
+  #    ocspResponderUrl: (none)
+  #    jceProvider: (none)
+  #    validateCerts: true
+  #    validatePeers: true
+  #    supportedProtocols: SSLv3
+  #    supportedCipherSuites: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
+  #    allowRenegotiation: true
+  #    endpointIdentificationAlgorithm: (none)
+
+# Logging settings.
+# https://logback.qos.ch/manual/layouts.html#conversionWord
+# Set LOG_FORMAT=json for structured logs. The default text format preserves legacy output.
+logging:
+  level: ${LOG_LEVEL:-INFO}
+  appenders:
+    - type: console
+      threshold: INFO
+      layout:
+        type: om-event-layout
+        format: ${LOG_FORMAT:-text}
+        pattern: "[server-2] %level [%d{ISO8601,UTC}] [%t] %logger{5} - %msg%n"
+        appendLineSeparator: true
+        additionalFields:
+          server: server-2
+
+database:
+  # the name of the JDBC driver, mysql in our case
+  driverClass: ${DB_DRIVER_CLASS:-com.mysql.cj.jdbc.Driver}
+  # the username and password
+  user: ${DB_USER:-openmetadata_user}
+  password: ${DB_USER_PASSWORD:-openmetadata_password}
+  # the JDBC URL; the database is called openmetadata_db
+  url: jdbc:${DB_SCHEME:-mysql}://${DB_HOST:-localhost}:${DB_PORT:-3306}/${OM_DATABASE:-openmetadata_db}?${DB_PARAMS:-allowPublicKeyRetrieval=true&useSSL=false&serverTimezone=UTC}
+
+  # HikariCP Connection Pool Settings - Optimized for Performance
+  maxSize: ${DB_CONNECTION_POOL_MAX_SIZE:-100}  # Increased from 50 for better concurrency
+  minSize: ${DB_CONNECTION_POOL_MIN_SIZE:-20}   # Increased from 10 to reduce connection creation overhead
+  minimumIdle: ${DB_CONNECTION_POOL_MIN_IDLE:-20}  # HikariCP specific minimum idle connections
+  initialSize: ${DB_CONNECTION_POOL_INITIAL_SIZE:-20}  # Start with more connections ready
+  checkConnectionWhileIdle: ${DB_CONNECTION_CHECK_CONNECTION_WHILE_IDLE:-true}
+  checkConnectionOnBorrow: ${DB_CONNECTION_CHECK_CONNECTION_ON_BORROW:-false}  # Disable for performance
+  evictionInterval: ${DB_CONNECTION_EVICTION_INTERVAL:-5 minutes}
+  minIdleTime: ${DB_CONNECTION_MIN_IDLE_TIME:-5 minute}
+
+
+  # JDBC Driver Properties - Critical for Performance
+  # These work across both PostgreSQL and MySQL drivers
+  properties:
+    # HikariCP connection pool settings
+    connectionTimeout: ${DB_CONNECTION_TIMEOUT:-300000}  # 300 seconds
+    idleTimeout: ${DB_IDLE_TIMEOUT:-600000}  # 10 minutes
+    maxLifetime: ${DB_MAX_LIFETIME:-1800000}  # 30 minutes
+    leakDetectionThreshold: ${DB_LEAK_DETECTION_THRESHOLD:-600000}  # 10 minute
+    keepaliveTime: ${DB_KEEPALIVE_TIME:-0}  # 0 = disabled (set to 30000 for Aurora)
+    validationTimeout: ${DB_VALIDATION_TIMEOUT:-300000}  # 300 seconds
+    # PostgreSQL specific - these are ignored by MySQL driver
+    prepareThreshold: ${DB_PG_PREPARE_THRESHOLD:-1}  # Use prepared statements immediately
+    preparedStatementCacheQueries: ${DB_PG_PREP_STMT_CACHE_QUERIES:-500}  # Cache more statements
+    preparedStatementCacheSizeMiB: ${DB_PG_PREP_STMT_CACHE_SIZE_MB:-10}  # Larger cache
+    reWriteBatchedInserts: ${DB_PG_REWRITE_BATCHED_INSERTS:-true}  # Critical for batch performance
+    defaultRowFetchSize: ${DB_PG_DEFAULT_ROW_FETCH_SIZE:-1000}  # Fetch more rows at once
+    assumeMinServerVersion: ${DB_PG_ASSUME_MIN_SERVER_VERSION:-12}  # Skip version checks
+    ApplicationName: ${DB_PG_APPLICATION_NAME:-OpenMetadata}
+    loginTimeout: ${DB_PG_LOGIN_TIMEOUT:-300}  # Login timeout in seconds
+    postgresqlConnectTimeout: ${DB_POSTGRESQL_CONNECT_TIMEOUT:-60}  # Connection timeout in seconds
+    postgresqlSocketTimeout: ${DB_POSTGRESQL_SOCKET_TIMEOUT:-30000}  # Socket timeout in seconds (0 = infinite)
+
+    # Aurora PostgreSQL specific optimizations
+    loadBalanceHosts: ${DB_PG_LOAD_BALANCE_HOSTS:-false}  # Set to true for Aurora reader endpoints
+    hostRecheckSeconds: ${DB_PG_HOST_RECHECK_SECONDS:-10}  # How often to check host status
+    targetServerType: ${DB_PG_TARGET_SERVER_TYPE:-primary}  # primary, secondary, any, preferSecondary
+
+    # MySQL specific - these are ignored by PostgreSQL driver
+    rewriteBatchedStatements: ${DB_MYSQL_REWRITE_BATCHED_STATEMENTS:-true}  # Critical for MySQL batch
+    cachePrepStmts: ${DB_MYSQL_CACHE_PREP_STMTS:-true}
+    prepStmtCacheSize: ${DB_MYSQL_PREP_STMT_CACHE_SIZE:-500}
+    prepStmtCacheSqlLimit: ${DB_MYSQL_PREP_STMT_CACHE_SQL_LIMIT:-2048}
+    useServerPrepStmts: ${DB_MYSQL_USE_SERVER_PREP_STMTS:-true}
+    useLocalSessionState: ${DB_MYSQL_USE_LOCAL_SESSION_STATE:-true}
+    useLocalTransactionState: ${DB_MYSQL_USE_LOCAL_TRANSACTION_STATE:-true}
+    elideSetAutoCommits: ${DB_MYSQL_ELIDE_SET_AUTO_COMMITS:-true}
+    maintainTimeStats: ${DB_MYSQL_MAINTAIN_TIME_STATS:-false}
+    cacheResultSetMetadata: ${DB_MYSQL_CACHE_RESULT_SET_METADATA:-true}
+    cacheServerConfiguration: ${DB_MYSQL_CACHE_SERVER_CONFIG:-true}
+    tcpKeepAlive: ${DB_MYSQL_TCP_KEEP_ALIVE:-true}
+    tcpNoDelay: ${DB_MYSQL_TCP_NO_DELAY:-true}
+    mysqlConnectTimeout: ${DB_MYSQL_CONNECT_TIMEOUT:-60000}  # Connection timeout in milliseconds
+    mysqlSocketTimeout: ${DB_MYSQL_SOCKET_TIMEOUT:-30000000}  # Socket timeout in milliseconds (0 = infinite)
+
+objectStorage:
+  enabled: false
+  provider: NOOP
+  maxFileSize: 5242880
+
+migrationConfiguration:
+  flywayPath: "./bootstrap/sql/migrations/flyway"
+  nativePath: "./bootstrap/sql/migrations/native"
+  extensionPath: ""
+
+# Authorizer Configuration
+authorizerConfiguration:
+  className: ${AUTHORIZER_CLASS_NAME:-org.openmetadata.service.security.DefaultAuthorizer}
+  containerRequestFilter: ${AUTHORIZER_REQUEST_FILTER:-org.openmetadata.service.security.JwtFilter}
+  adminPrincipals: ${AUTHORIZER_ADMIN_PRINCIPALS:-[admin]}
+  allowedEmailRegistrationDomains: ${AUTHORIZER_ALLOWED_REGISTRATION_DOMAIN:-["all"]}
+  principalDomain: ${AUTHORIZER_PRINCIPAL_DOMAIN:-"open-metadata.org"}
+  allowedDomains: ${AUTHORIZER_ALLOWED_DOMAINS:-[]}
+  enforcePrincipalDomain: ${AUTHORIZER_ENFORCE_PRINCIPAL_DOMAIN:-false}
+  enableSecureSocketConnection : ${AUTHORIZER_ENABLE_SECURE_SOCKET:-false}
+  useRolesFromProvider: ${AUTHORIZER_USE_ROLES_FROM_PROVIDER:-false}
+
+authenticationConfiguration:
+  clientType: ${AUTHENTICATION_CLIENT_TYPE:-public}
+  provider: ${AUTHENTICATION_PROVIDER:-basic}
+  # This is used by auth provider provide response as either id_token or code
+  responseType: ${AUTHENTICATION_RESPONSE_TYPE:-id_token}
+  # This will only be valid when provider type specified is customOidc
+  providerName: ${CUSTOM_OIDC_AUTHENTICATION_PROVIDER_NAME:-""}
+  publicKeyUrls: ${AUTHENTICATION_PUBLIC_KEYS:-[http://localhost:8585/api/v1/system/config/jwks]}
+  tokenValidationAlgorithm: ${AUTHENTICATION_TOKEN_VALIDATION_ALGORITHM:-"RS256"}
+  authority: ${AUTHENTICATION_AUTHORITY:-https://accounts.google.com}
+  clientId: ${AUTHENTICATION_CLIENT_ID:-""}
+  callbackUrl: ${AUTHENTICATION_CALLBACK_URL:-""}
+  jwtPrincipalClaims: ${AUTHENTICATION_JWT_PRINCIPAL_CLAIMS:-[email,preferred_username,sub]}
+  jwtPrincipalClaimsMapping: ${AUTHENTICATION_JWT_PRINCIPAL_CLAIMS_MAPPING:-[]}
+  enableSelfSignup : ${AUTHENTICATION_ENABLE_SELF_SIGNUP:-true}
+  enableAutoRedirect: ${AUTHENTICATION_ENABLE_AUTO_REDIRECT:-false}
+  # Force secure flag on session cookies even when not using HTTPS directly.
+  # Enable this when running behind a proxy/load balancer that handles SSL termination.
+  # Default: false (secure flag only set when HTTPS is detected)
+  forceSecureSessionCookie: ${FORCE_SECURE_SESSION_COOKIE:-false}
+  sessionExpiry: ${AUTHENTICATION_SESSION_EXPIRY:-"604800"} # 7 days; applies to all auth providers
+  maxActiveSessionsPerUser: ${AUTHENTICATION_MAX_ACTIVE_SESSIONS_PER_USER:-5}
+  oidcConfiguration:
+    id: ${OIDC_CLIENT_ID:-""}
+    type: ${OIDC_TYPE:-""} # google, azure etc.
+    secret: ${OIDC_CLIENT_SECRET:-""}
+    scope: ${OIDC_SCOPE:-"openid email profile"}
+    discoveryUri: ${OIDC_DISCOVERY_URI:-""}
+    useNonce: ${OIDC_USE_NONCE:-true}
+    preferredJwsAlgorithm: ${OIDC_PREFERRED_JWS:-"RS256"}
+    responseType: ${OIDC_RESPONSE_TYPE:-"code"}
+    disablePkce: ${OIDC_DISABLE_PKCE:-true}
+    callbackUrl: ${OIDC_CALLBACK:-"http://localhost:8585/callback"}
+    serverUrl: ${OIDC_SERVER_URL:-"http://localhost:8585"}
+    clientAuthenticationMethod: ${OIDC_CLIENT_AUTH_METHOD:-"client_secret_post"}
+    tenant: ${OIDC_TENANT:-""}
+    maxClockSkew: ${OIDC_MAX_CLOCK_SKEW:-""}
+    tokenValidity: ${OIDC_OM_REFRESH_TOKEN_VALIDITY:-"3600"} # in seconds
+    customParams: ${OIDC_CUSTOM_PARAMS:-}
+    maxAge: ${OIDC_MAX_AGE:-"0"}
+    prompt: ${OIDC_PROMPT_TYPE:-"consent"}
+    sessionExpiry: ${OIDC_SESSION_EXPIRY:-"604800"} #7 days
+  samlConfiguration:
+    debugMode: ${SAML_DEBUG_MODE:-false}
+    idp:
+      entityId: ${SAML_IDP_ENTITY_ID:-""}
+      ssoLoginUrl: ${SAML_IDP_SSO_LOGIN_URL:-""}
+      idpX509Certificate: ${SAML_IDP_CERTIFICATE:-""}
+      nameId: ${SAML_IDP_NAME_ID:-"urn:oasis:names:tc:SAML:2.0:nameid-format:emailAddress"}
+    sp:
+      entityId: ${SAML_SP_ENTITY_ID:-"http://localhost:8585/api/v1/saml/metadata"}
+      acs: ${SAML_SP_ACS:-"http://localhost:8585/api/v1/saml/acs"}
+      spX509Certificate: ${SAML_SP_CERTIFICATE:-""}
+      spPrivateKey: ${SAML_SP_PRIVATE_KEY:-""}
+      callback: ${SAML_SP_CALLBACK:-"http://localhost:8585/saml/callback"}
+    security:
+      strictMode: ${SAML_STRICT_MODE:-false}
+      validateXml: ${SAML_VALIDATE_XML:-false}
+      tokenValidity: ${SAML_SP_TOKEN_VALIDITY:-"3600"}
+      sendEncryptedNameId: ${SAML_SEND_ENCRYPTED_NAME_ID:-false}
+      sendSignedAuthRequest: ${SAML_SEND_SIGNED_AUTH_REQUEST:-false}
+      signSpMetadata: ${SAML_SIGNED_SP_METADATA:-false}
+      wantMessagesSigned: ${SAML_WANT_MESSAGE_SIGNED:-false}
+      wantAssertionsSigned: ${SAML_WANT_ASSERTION_SIGNED:-false}
+      wantAssertionEncrypted: ${SAML_WANT_ASSERTION_ENCRYPTED:-false}
+      keyStoreFilePath: ${SAML_KEYSTORE_FILE_PATH:-""}
+      keyStoreAlias: ${SAML_KEYSTORE_ALIAS:-""}
+      keyStorePassword: ${SAML_KEYSTORE_PASSWORD:-""}
+  ldapConfiguration:
+    host: ${AUTHENTICATION_LDAP_HOST:-}
+    port: ${AUTHENTICATION_LDAP_PORT:-}
+    dnAdminPrincipal: ${AUTHENTICATION_LOOKUP_ADMIN_DN:-""}
+    dnAdminPassword: ${AUTHENTICATION_LOOKUP_ADMIN_PWD:-""}
+    userBaseDN: ${AUTHENTICATION_USER_LOOKUP_BASEDN:-""}
+    groupBaseDN: ${AUTHENTICATION_GROUP_LOOKUP_BASEDN:-""}
+    roleAdminName: ${AUTHENTICATION_USER_ROLE_ADMIN_NAME:-}
+    allAttributeName: ${AUTHENTICATION_USER_ALL_ATTR:-}
+    mailAttributeName: ${AUTHENTICATION_USER_MAIL_ATTR:-}
+    usernameAttributeName: ${AUTHENTICATION_USER_NAME_ATTR:-}
+    groupAttributeName: ${AUTHENTICATION_USER_GROUP_ATTR:-}
+    groupAttributeValue: ${AUTHENTICATION_USER_GROUP_ATTR_VALUE:-}
+    groupMemberAttributeName: ${AUTHENTICATION_USER_GROUP_MEMBER_ATTR:-}
+    #the mapping of roles to LDAP groups
+    authRolesMapping: ${AUTH_ROLES_MAPPING:-""}
+    authReassignRoles: ${AUTH_REASSIGN_ROLES:-[]}
+    #optional
+    maxPoolSize: ${AUTHENTICATION_LDAP_POOL_SIZE:-3}
+    sslEnabled: ${AUTHENTICATION_LDAP_SSL_ENABLED:-}
+    truststoreConfigType: ${AUTHENTICATION_LDAP_TRUSTSTORE_TYPE:-TrustAll}
+    trustStoreConfig:
+      customTrustManagerConfig:
+        trustStoreFilePath: ${AUTHENTICATION_LDAP_TRUSTSTORE_PATH:-}
+        trustStoreFilePassword: ${AUTHENTICATION_LDAP_KEYSTORE_PASSWORD:-}
+        trustStoreFileFormat: ${AUTHENTICATION_LDAP_SSL_KEY_FORMAT:-}
+        verifyHostname: ${AUTHENTICATION_LDAP_SSL_VERIFY_CERT_HOST:-}
+        examineValidityDates: ${AUTHENTICATION_LDAP_EXAMINE_VALIDITY_DATES:-}
+      hostNameConfig:
+        allowWildCards: ${AUTHENTICATION_LDAP_ALLOW_WILDCARDS:-}
+        acceptableHostNames: ${AUTHENTICATION_LDAP_ALLOWED_HOSTNAMES:-[]}
+      jvmDefaultConfig:
+        verifyHostname: ${AUTHENTICATION_LDAP_SSL_VERIFY_CERT_HOST:-}
+      trustAllConfig:
+        examineValidityDates: ${AUTHENTICATION_LDAP_EXAMINE_VALIDITY_DATES:-true}
+
+jwtTokenConfiguration:
+  rsapublicKeyFilePath: ${RSA_PUBLIC_KEY_FILE_PATH:-"./conf/public_key.der"}
+  rsaprivateKeyFilePath: ${RSA_PRIVATE_KEY_FILE_PATH:-"./conf/private_key.der"}
+  jwtissuer: ${JWT_ISSUER:-"open-metadata.org"}
+  keyId: ${JWT_KEY_ID:-"Gb389a-9f76-gdjs-a92j-0242bk94356"}
+
+elasticsearch:
+  searchType: ${SEARCH_TYPE:- "opensearch"}
+  # Single host or comma-separated list for multiple hosts
+  # Examples: "localhost" or "es-node1:9200,es-node2:9200,es-node3:9200"
+  host: ${ELASTICSEARCH_HOST:-localhost}
+  port: ${ELASTICSEARCH_PORT:-9200}
+  scheme: ${ELASTICSEARCH_SCHEME:-http}
+  username: ${ELASTICSEARCH_USER:-""}
+  password: ${ELASTICSEARCH_PASSWORD:-""}
+  clusterAlias: ${ELASTICSEARCH_CLUSTER_ALIAS:-""}
+  truststorePath: ${ELASTICSEARCH_TRUST_STORE_PATH:-""}
+  truststorePassword: ${ELASTICSEARCH_TRUST_STORE_PASSWORD:-""}
+  connectionTimeoutSecs: ${ELASTICSEARCH_CONNECTION_TIMEOUT_SECS:-10}  # Increased from 5s for Docker networks
+  socketTimeoutSecs: ${ELASTICSEARCH_SOCKET_TIMEOUT_SECS:-120}  # Increased from 60s for slow queries
+  keepAliveTimeoutSecs: ${ELASTICSEARCH_KEEP_ALIVE_TIMEOUT_SECS:-600}
+  # Connection pool settings for better load balancing and performance
+  maxConnTotal: ${ELASTICSEARCH_MAX_CONN_TOTAL:-30}  # Total connections across all hosts
+  maxConnPerRoute: ${ELASTICSEARCH_MAX_CONN_PER_ROUTE:-10}  # Max connections per host
+  batchSize: ${ELASTICSEARCH_BATCH_SIZE:-100}
+  payLoadSize: ${ELASTICSEARCH_PAYLOAD_BYTES_SIZE:-10485760}
+  searchIndexMappingLanguage : ${ELASTICSEARCH_INDEX_MAPPING_LANG:-EN}
+  searchIndexFactoryClassName : org.openmetadata.service.search.SearchIndexFactory
+  # AWS IAM Authentication for OpenSearch (only applicable when searchType is "opensearch")
+  # Uses standard AWS environment variables: https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-envvars.html
+  # IAM auth is automatically enabled when AWS_DEFAULT_REGION is set
+  # Credentials: Use AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY or rely on AWS SDK default credential provider chain
+  aws:
+    enabled: ${SEARCH_AWS_IAM_AUTH_ENABLED:-false}
+    region: ${AWS_DEFAULT_REGION:-""}
+    accessKeyId: ${AWS_ACCESS_KEY_ID:-""}
+    secretAccessKey: ${AWS_SECRET_ACCESS_KEY:-""}
+    sessionToken: ${AWS_SESSION_TOKEN:-""}
+    serviceName: ${SEARCH_AWS_SERVICE_NAME:-"es"}  # Use "es" for OpenSearch, "aoss" for OpenSearch Serverless
+  naturalLanguageSearch:
+    enabled: ${NATURAL_LANGUAGE_SEARCH_ENABLED:-false}
+    embeddingProvider: ${EMBEDDING_PROVIDER:-bedrock}
+    providerClass: ${NATURAL_LANGUAGE_SEARCH_PROVIDER_CLASS:-org.openmetadata.service.search.nlq.NoOpNLQService}
+    bedrock:
+      awsConfig:
+        enabled: ${BEDROCK_AWS_IAM_AUTH_ENABLED:-false}
+        region: ${AWS_DEFAULT_REGION:-""}
+        accessKeyId: ${AWS_ACCESS_KEY_ID:-""}
+        secretAccessKey: ${AWS_SECRET_ACCESS_KEY:-""}
+        sessionToken: ${AWS_SESSION_TOKEN:-""}
+      modelId: ${AWS_BEDROCK_MODEL_ID:-""}
+      embeddingModelId: ${AWS_BEDROCK_EMBED_MODEL_ID:-""}
+      embeddingDimension: ${AWS_BEDROCK_EMBEDDING_DIMENSION:-""}
+
+eventMonitoringConfiguration:
+  eventMonitor: ${EVENT_MONITOR:-prometheus}  # Possible values are "prometheus", "cloudwatch"
+  batchSize: ${EVENT_MONITOR_BATCH_SIZE:-10}
+  pathPattern: ${EVENT_MONITOR_PATH_PATTERN:-["/api/v1/tables/*", "/api/v1/health-check"]}
+  latency: ${EVENT_MONITOR_LATENCY:-[0.99, 0.90]} # For value p99=0.99, p90=0.90, p50=0.50 etc.
+  servicesHealthCheckInterval: ${EVENT_MONITOR_SERVICES_HEALTH_CHECK_INTERVAL:-300}
+  # it will use the default auth provider for AWS services if parameters are not set
+  # parameters:
+  #   region: ${OM_MONITOR_REGION:-""}
+  #   accessKeyId: ${OM_MONITOR_ACCESS_KEY_ID:-""}
+  #   secretAccessKey: ${OM_MONITOR_ACCESS_KEY:-""}
+
+eventHandlerConfiguration:
+  eventHandlerClassNames:
+    - "org.openmetadata.service.events.AuditEventHandler"
+    - "org.openmetadata.service.events.ChangeEventHandler"
+
+pipelineServiceClientConfiguration:
+  enabled: ${PIPELINE_SERVICE_CLIENT_ENABLED:-true}
+  # If we don't need this, set "org.openmetadata.service.clients.pipeline.noop.NoopClient"
+  className: ${PIPELINE_SERVICE_CLIENT_CLASS_NAME:-"org.openmetadata.service.clients.pipeline.airflow.AirflowRESTClient"}
+  apiEndpoint: ${PIPELINE_SERVICE_CLIENT_ENDPOINT:-http://localhost:8080}
+  metadataApiEndpoint: ${SERVER_HOST_API_URL:-http://localhost:8585/api}
+  ingestionIpInfoEnabled: ${PIPELINE_SERVICE_IP_INFO_ENABLED:-false}
+  hostIp: ${PIPELINE_SERVICE_CLIENT_HOST_IP:-""}
+  healthCheckInterval: ${PIPELINE_SERVICE_CLIENT_HEALTH_CHECK_INTERVAL:-300}
+  # This SSL information is about the OpenMetadata server.
+  # It will be picked up from the pipelineServiceClient to use/ignore SSL when connecting to the OpenMetadata server.
+  verifySSL: ${PIPELINE_SERVICE_CLIENT_VERIFY_SSL:-"no-ssl"} # Possible values are "no-ssl", "ignore", "validate"
+  sslConfig:
+    certificatePath: ${PIPELINE_SERVICE_CLIENT_SSL_CERT_PATH:-""} # Local path for the Pipeline Service Client
+  logStorageConfiguration:
+    type: ${PIPELINE_SERVICE_CLIENT_LOG_TYPE:-"default"} # Possible values are "default", "s3"
+    enabled: ${PIPELINE_SERVICE_CLIENT_LOG_ENABLED:-false} # Enable it for pipelines deployed in the server
+    # if type is s3, provide the following configuration
+    bucketName: ${PIPELINE_SERVICE_CLIENT_LOG_BUCKET_NAME:-""}
+    # optional path within the bucket to store the logs
+    prefix: ${PIPELINE_SERVICE_CLIENT_LOG_PREFIX:-""}
+    enableServerSideEncryption: ${PIPELINE_SERVICE_CLIENT_LOG_SSE_ENABLED:-false}
+    sseAlgorithm: ${PIPELINE_SERVICE_CLIENT_LOG_SSE_ALGORITHM:-"AES256"} # Allowed values: "AES256" or "aws:kms"
+    kmsKeyId: ${PIPELINE_SERVICE_CLIENT_LOG_KMS_KEY_ID:-""} # Required only if sseAlgorithm is "aws:kms"
+    awsConfig:
+      enabled: ${PIPELINE_SERVICE_CLIENT_AWS_IAM_AUTH_ENABLED:-false}
+      awsAccessKeyId: ${PIPELINE_SERVICE_CLIENT_LOG_AWS_ACCESS_KEY_ID:-""}
+      awsSecretAccessKey: ${PIPELINE_SERVICE_CLIENT_LOG_AWS_SECRET_ACCESS_KEY:-""}
+      awsRegion: ${PIPELINE_SERVICE_CLIENT_LOG_REGION:-""}
+      awsSessionToken: ${PIPELINE_SERVICE_CLIENT_LOG_AWS_SESSION_TOKEN:-""}
+      endPointURL: ${PIPELINE_SERVICE_CLIENT_LOG_AWS_ENDPOINT_URL:-""} # port forward localhost:9000 for minio
+
+  # Secrets Manager Loader: specify to the Ingestion Framework how to load the SM credentials from its env
+  # Supported: noop, airflow, env
+  secretsManagerLoader: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-"noop"}
+
+  # Default required parameters for Airflow as Pipeline Service Client
+  parameters:
+    ## Airflow parameters
+    username: ${AIRFLOW_USERNAME:-admin}
+    password: ${AIRFLOW_PASSWORD:-admin}
+    timeout: ${AIRFLOW_TIMEOUT:-10}
+    # If we need to use SSL to reach Airflow
+    truststorePath: ${AIRFLOW_TRUST_STORE_PATH:-""}
+    truststorePassword: ${AIRFLOW_TRUST_STORE_PASSWORD:-""}
+    ## Kubernetes client parameters
+    namespace: ${K8S_NAMESPACE:-"openmetadata-pipelines"}
+    ingestionImage: ${K8S_INGESTION_IMAGE:-"docker.getcollate.io/openmetadata/ingestion-base:latest"}
+    imagePullPolicy: ${K8S_IMAGE_PULL_POLICY:-"IfNotPresent"}
+    imagePullSecrets: ${K8S_IMAGE_PULL_SECRETS:-""}
+    serviceAccountName: ${K8S_SERVICE_ACCOUNT_NAME:-"openmetadata-ingestion"}
+    # Resources configuration
+    resources:
+      limits:
+        cpu: ${K8S_LIMITS_CPU:-"2"}
+        memory: ${K8S_LIMITS_MEMORY:-"4Gi"}
+      requests:
+        cpu: ${K8S_REQUESTS_CPU:-"500m"}
+        memory: ${K8S_REQUESTS_MEMORY:-"1Gi"}
+    ttlSecondsAfterFinished: ${K8S_TTL_SECONDS_AFTER_FINISHED:-604800}
+    activeDeadlineSeconds: ${K8S_ACTIVE_DEADLINE_SECONDS:-604800}
+    backoffLimit: ${K8S_BACKOFF_LIMIT:-3}
+    successfulJobsHistoryLimit: ${K8S_SUCCESSFUL_JOBS_HISTORY_LIMIT:-3}
+    failedJobsHistoryLimit: ${K8S_FAILED_JOBS_HISTORY_LIMIT:-1}
+    nodeSelector: ${K8S_NODE_SELECTOR:-""}
+    runAsUser: ${K8S_RUN_AS_USER:-1000}
+    runAsGroup: ${K8S_RUN_AS_GROUP:-1000}
+    fsGroup: ${K8S_FS_GROUP:-1000}
+    runAsNonRoot: ${K8S_RUN_AS_NON_ROOT:-"true"}
+    extraEnvVars: ${K8S_EXTRA_ENV_VARS:-[]}
+    podAnnotations: ${K8S_POD_ANNOTATIONS:-""}
+    useOMJobOperator: ${USE_OMJOB_OPERATOR:-"true"}
+
+
+# no_encryption_at_rest is the default value, and it does what it says. Please read the manual on how
+# to secure your instance of OpenMetadata with TLS and encryption at rest.
+fernetConfiguration:
+  fernetKey: ${FERNET_KEY:-jJ/9sz0g0OHxsfxOoSfdFdmk3ysNmPRnH3TUAbz3IHA=}
+
+secretsManagerConfiguration:
+  secretsManager: ${SECRET_MANAGER:-db} # Possible values are "db", "managed-aws","aws", "managed-aws-ssm", "aws-ssm", "managed-azure-kv", "azure-kv", "in-memory", "gcp", "kubernetes"
+  prefix: ${SECRET_MANAGER_PREFIX:-""} # Define the secret key ID as /<prefix>/<clusterName>/<key>
+  tags: ${SECRET_MANAGER_TAGS:-[]} # Add tags to the created resource. Format is `[key1:value1,key2:value2,...]`
+  # it will use the default auth provider for the secrets' manager service if parameters are not set
+  parameters:
+    ## For AWS
+    accessKeyId: ${OM_SM_ACCESS_KEY_ID:-""}
+    secretAccessKey: ${OM_SM_ACCESS_KEY:-""}
+    ## For Azure Key Vault
+    clientId: ${OM_SM_CLIENT_ID:-""}
+    clientSecret: ${OM_SM_CLIENT_SECRET:-""}
+    tenantId: ${OM_SM_TENANT_ID:-""}
+    vaultName: ${OM_SM_VAULT_NAME:-""}
+    ## For GCP
+    projectId: ${OM_SM_PROJECT_ID:-""}
+    ## For Kubernetes
+    namespace: ${OM_SM_NAMESPACE:-"default"}
+    kubeconfigPath: ${OM_SM_KUBECONFIG_PATH:-""}
+    inCluster: ${OM_SM_IN_CLUSTER:-"false"}
+
+health:
+  delayedShutdownHandlerEnabled: true
+  shutdownWaitPeriod: 1s
+  healthChecks:
+    - name: OpenMetadataServerHealthCheck
+      critical: true
+      schedule:
+        checkInterval: 2500ms
+        downtimeInterval: 10s
+        failureAttempts: 2
+        successAttempts: 1
+
+limits:
+  enable: ${LIMITS_ENABLED:-false}
+  className: ${LIMITS_CLASS_NAME:-"org.openmetadata.service.limits.DefaultLimits"}
+  limitsConfigFile: ${LIMITS_CONFIG_FILE:-""}
+
+# Bulk Operation Configuration
+# Controls parallelism and resource usage for bulk API operations (e.g., bulk import/export)
+# Uses a bounded thread pool to prevent connection pool exhaustion
+bulkOperation:
+  # Max threads for bulk operations (recommendations: 2 vCore=5-8, 4 vCore=8-15, 8 vCore=15-25)
+  maxThreads: ${BULK_OPERATION_MAX_THREADS:-10}
+  # Max queued operations before rejection (returns 503)
+  queueSize: ${BULK_OPERATION_QUEUE_SIZE:-1000}
+  # Timeout in seconds for entire bulk operation
+  timeoutSeconds: ${BULK_OPERATION_TIMEOUT_SECONDS:-300}
+
+web:
+  uriPath: ${WEB_CONF_URI_PATH:-"/api"}
+  hsts:
+    enabled: ${WEB_CONF_HSTS_ENABLED:-false}
+    maxAge: ${WEB_CONF_HSTS_MAX_AGE:-"365 days"}
+    includeSubDomains: ${WEB_CONF_HSTS_INCLUDE_SUBDOMAINS:-"true"}
+    preload: ${WEB_CONF_HSTS_PRELOAD:-"true"}
+  frame-options:
+    enabled: ${WEB_CONF_FRAME_OPTION_ENABLED:-false}
+    option: ${WEB_CONF_FRAME_OPTION:-"SAMEORIGIN"}
+    origin: ${WEB_CONF_FRAME_ORIGIN:-""}
+  content-type-options:
+    enabled: ${WEB_CONF_CONTENT_TYPE_OPTIONS_ENABLED:-false}
+  xss-protection:
+    enabled: ${WEB_CONF_XSS_PROTECTION_ENABLED:-false}
+    on: ${WEB_CONF_XSS_PROTECTION_ON:-true}
+    block: ${WEB_CONF_XSS_PROTECTION_BLOCK:-true}
+  csp:
+    enabled: ${WEB_CONF_XSS_CSP_ENABLED:-false}
+    policy: ${WEB_CONF_XSS_CSP_POLICY:-"default-src 'self'"}
+    reportOnlyPolicy: ${WEB_CONF_XSS_CSP_REPORT_ONLY_POLICY:-""}
+  referrer-policy:
+    enabled: ${WEB_CONF_REFERRER_POLICY_ENABLED:-false}
+    option: ${WEB_CONF_REFERRER_POLICY_OPTION:-"SAME_ORIGIN"}
+  permission-policy:
+    enabled: ${WEB_CONF_PERMISSION_POLICY_ENABLED:-false}
+    option: ${WEB_CONF_PERMISSION_POLICY_OPTION:-""}
+  cache-control: ${WEB_CONF_CACHE_CONTROL:-""}
+  pragma: ${WEB_CONF_PRAGMA:-""}
+
+operationalConfig:
+  enable: ${OPERATIONAL_CONFIG_ENABLED:-true}
+  operationsConfigFile: ${OPERATIONAL_CONFIG_FILE:-"./conf/operations.yaml"}
+
+rdf:
+  enabled: ${RDF_ENABLED:-false}
+  baseUri: ${RDF_BASE_URI:-"https://open-metadata.org/"}
+  storageType: ${RDF_STORAGE_TYPE:-"FUSEKI"}
+  remoteEndpoint: ${RDF_ENDPOINT:-"http://localhost:3030/openmetadata"}
+  username: ${RDF_REMOTE_USERNAME:-"admin"}
+  password: ${RDF_REMOTE_PASSWORD:-"admin"}
+  dataset: ${RDF_DATASET:-"openmetadata"}
+
+# Cache Configuration
+# Caching layer for entity metadata, relationships, and tag usage to reduce database load
+# Default: Disabled (uses NoopCacheProvider)
+cache:
+  # Cache provider: none (default) or redis
+  provider: ${CACHE_PROVIDER:-none}
+
+  # TTL (Time To Live) settings in seconds
+  entityTtlSeconds: ${CACHE_ENTITY_TTL:-172800}             # 48 hour for entities
+  relationshipTtlSeconds: ${CACHE_RELATIONSHIP_TTL:-172800} # 48 hour for relationships
+  tagTtlSeconds: ${CACHE_TAG_TTL:-172800}                   # 48 hour for tags
+
+  # Redis configuration
+  redis:
+    # Redis connection URL
+    # Standalone: redis://localhost:6379
+    # AWS ElastiCache: redis://my-cluster.abc123.cache.amazonaws.com:6379
+    url: ${CACHE_REDIS_URL:-redis://localhost:6379}
+    authType: ${CACHE_REDIS_AUTH_TYPE:-NONE}
+    database: ${CACHE_REDIS_DATABASE:-0}  # Redis database index (0-15)
+
+    # Authentication for standalone Redis
+    username: ${CACHE_REDIS_USERNAME:-}
+    passwordRef: ${CACHE_REDIS_PASSWORD:-}  # Reference to password in secrets manager
+    useSSL: ${CACHE_REDIS_USE_SSL:-false}
+
+    # Key namespace prefix (useful for multi-tenant deployments)
+    keyspace: ${CACHE_REDIS_KEYSPACE:-"om:prod"}
+
+    # Connection pool settings
+    poolSize: ${CACHE_REDIS_POOL_SIZE:-64}
+    connectTimeoutMs: ${CACHE_REDIS_CONNECT_TIMEOUT:-2000}
+
+    # AWS ElastiCache IAM Authentication (only if using ElastiCache)
+    aws:
+      enabled: ${CACHE_REDIS_AWS_IAM_AUTH_ENABLED:-false}
+      region: ${CACHE_REDIS_AWS_REGION:-""}
+      useInstanceProfile: ${CACHE_REDIS_AWS_INSTANCE_PROFILE:-true}
+      # If not using instance profile, provide credentials:
+      accessKeyId: ${AWS_ACCESS_KEY_ID:-""}
+      secretAccessKey: ${AWS_SECRET_ACCESS_KEY:-""}
+      tokenRefreshIntervalSeconds: ${CACHE_REDIS_TOKEN_REFRESH:-900}  # 15 minutes

docker/development/distributed-test/local/server3.yaml

@@ -0,0 +1,640 @@
+# OpenMetadata Server 3 Configuration for Local Development
+# Run with: java -Dconfig=local/server3.yaml -jar openmetadata-service/target/openmetadata-service-*.jar server
+
+clusterName: distributed-test
+
+swagger:
+  resourcePackage: org.openmetadata.service.resources
+
+assets:
+  resourcePath: /assets/
+  uriPath: /
+
+basePath: ${BASE_PATH:-/}
+
+server:
+  applicationContextPath: /
+  rootPath: /api/*
+  applicationConnectors:
+    - type: http
+      bindHost: ${SERVER_HOST:-0.0.0.0}
+      port: ${SERVER_PORT:-8589}
+
+      # Jetty 12 URI Compliance - UNSAFE allows all special characters in entity names
+      # Required for backward compatibility with entity names containing /, ", etc.
+      uriCompliance: UNSAFE
+
+      # Jetty Acceptor and Selector threads for high concurrency
+      acceptorThreads: ${SERVER_ACCEPTOR_THREADS:-2}  # 1-2 per CPU core
+      selectorThreads: ${SERVER_SELECTOR_THREADS:-8}  # 2-4 per CPU core
+
+      # Connection settings - relaxed for Docker/local development
+      acceptQueueSize: ${SERVER_ACCEPT_QUEUE_SIZE:-256}  # OS-level connection backlog
+      idleTimeout: ${SERVER_IDLE_TIMEOUT:-60 seconds}  # Close idle connections (increased from 30s)
+
+      # Buffer sizes for better throughput
+      outputBufferSize: ${SERVER_OUTPUT_BUFFER_SIZE:-32KiB}
+      inputBufferSize: ${SERVER_INPUT_BUFFER_SIZE:-8KiB}
+      maxRequestHeaderSize: ${SERVER_MAX_REQUEST_HEADER_SIZE:-8KiB}
+      maxResponseHeaderSize: ${SERVER_MAX_RESPONSE_HEADER_SIZE:-8KiB}
+      headerCacheSize: ${SERVER_HEADER_CACHE_SIZE:-512B}  # Cache parsed headers (in bytes)
+
+      # Performance settings
+      useServerHeader: false  # Don't send server version header
+      useDateHeader: true
+      useForwardedHeaders: ${SERVER_USE_FORWARDED_HEADERS:-false}  # Enable if behind proxy
+
+      # Data rate limits (prevent slow loris attacks)
+      minRequestDataPerSecond: ${SERVER_MIN_REQUEST_DATA_RATE:-0B}  # 0B = disabled
+      minResponseDataPerSecond: ${SERVER_MIN_RESPONSE_DATA_RATE:-0B}  # 0B = disabled
+
+  adminConnectors:
+    - type: http
+      bindHost: ${SERVER_HOST:-0.0.0.0}
+      port: ${SERVER_ADMIN_PORT:-8590}
+      acceptorThreads: 1  # Admin endpoint needs minimal resources
+      selectorThreads: 1
+
+  # Response compression disabled for maximum throughput
+  gzip:
+    enabled: false
+
+  # Thread pool configuration
+  # With virtual threads enabled (Java 21+), these settings become less critical
+  # as blocking operations are handled efficiently by the JVM
+  maxThreads: ${SERVER_MAX_THREADS:-150}
+  minThreads: ${SERVER_MIN_THREADS:-100}
+  idleThreadTimeout: ${SERVER_IDLE_THREAD_TIMEOUT:-1 minute}
+
+  # Virtual Threads (Project Loom) - Recommended for Java 21+
+  # Jetty 12 uses AdaptiveExecutionStrategy to route blocking tasks to virtual threads
+  # while keeping non-blocking I/O on platform threads for optimal cache locality
+  enableVirtualThreads: ${SERVER_ENABLE_VIRTUAL_THREAD:-false}
+  # Note: maxQueuedRequests removed in Dropwizard 5.0/Jetty 12
+
+  # Request/Response logging (disable in production for performance)
+  # Set LOG_FORMAT=json for structured logs. The default text format preserves legacy output.
+  requestLog:
+    appenders:
+      - type: console
+        threshold: ${REQUEST_LOG_LEVEL:-ERROR}  # Only log errors by default
+        layout:
+          type: om-access-layout
+          format: ${LOG_FORMAT:-text}
+          appendLineSeparator: true
+          additionalFields:
+            server: server-3
+  
+  # Above configuration for running http is fine for dev and testing.
+  # For production setup, where UI app will hit apis through DPS it
+  # is strongly recommended to run https instead. Note that only
+  # keyStorePath and keyStorePassword are mandatory properties. Values
+  # for other properties are defaults
+  #server:
+  #applicationConnectors:
+  #  - type: https
+  #    port: 8585
+  #    keyStorePath: ./conf/keystore.jks
+  #    keyStorePassword: changeit
+  #    keyStoreType: JKS
+  #    keyStoreProvider:
+  #    trustStorePath: /path/to/file
+  #    trustStorePassword: changeit
+  #    trustStoreType: JKS
+  #    trustStoreProvider:
+  #    keyManagerPassword: changeit
+  #    needClientAuth: false
+  #    wantClientAuth:
+  #    certAlias: <alias>
+  #    crlPath: /path/to/file
+  #    enableCRLDP: false
+  #    enableOCSP: false
+  #    maxCertPathLength: (unlimited)
+  #    ocspResponderUrl: (none)
+  #    jceProvider: (none)
+  #    validateCerts: true
+  #    validatePeers: true
+  #    supportedProtocols: SSLv3
+  #    supportedCipherSuites: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
+  #    allowRenegotiation: true
+  #    endpointIdentificationAlgorithm: (none)
+
+  #adminConnectors:
+  #  - type: https
+  #    port: 8586
+  #    keyStorePath: ./conf/keystore.jks
+  #    keyStorePassword: changeit
+  #    keyStoreType: JKS
+  #    keyStoreProvider:
+  #    trustStorePath: /path/to/file
+  #    trustStorePassword: changeit
+  #    trustStoreType: JKS
+  #    trustStoreProvider:
+  #    keyManagerPassword: changeit
+  #    needClientAuth: false
+  #    wantClientAuth:
+  #    certAlias: <alias>
+  #    crlPath: /path/to/file
+  #    enableCRLDP: false
+  #    enableOCSP: false
+  #    maxCertPathLength: (unlimited)
+  #    ocspResponderUrl: (none)
+  #    jceProvider: (none)
+  #    validateCerts: true
+  #    validatePeers: true
+  #    supportedProtocols: SSLv3
+  #    supportedCipherSuites: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
+  #    allowRenegotiation: true
+  #    endpointIdentificationAlgorithm: (none)
+
+# Logging settings.
+# https://logback.qos.ch/manual/layouts.html#conversionWord
+# Set LOG_FORMAT=json for structured logs. The default text format preserves legacy output.
+logging:
+  level: ${LOG_LEVEL:-INFO}
+  appenders:
+    - type: console
+      threshold: INFO
+      layout:
+        type: om-event-layout
+        format: ${LOG_FORMAT:-text}
+        pattern: "[server-3] %level [%d{ISO8601,UTC}] [%t] %logger{5} - %msg%n"
+        appendLineSeparator: true
+        additionalFields:
+          server: server-3
+
+
+database:
+  # the name of the JDBC driver, mysql in our case
+  driverClass: ${DB_DRIVER_CLASS:-com.mysql.cj.jdbc.Driver}
+  # the username and password
+  user: ${DB_USER:-openmetadata_user}
+  password: ${DB_USER_PASSWORD:-openmetadata_password}
+  # the JDBC URL; the database is called openmetadata_db
+  url: jdbc:${DB_SCHEME:-mysql}://${DB_HOST:-localhost}:${DB_PORT:-3306}/${OM_DATABASE:-openmetadata_db}?${DB_PARAMS:-allowPublicKeyRetrieval=true&useSSL=false&serverTimezone=UTC}
+
+  # HikariCP Connection Pool Settings - Optimized for Performance
+  maxSize: ${DB_CONNECTION_POOL_MAX_SIZE:-100}  # Increased from 50 for better concurrency
+  minSize: ${DB_CONNECTION_POOL_MIN_SIZE:-20}   # Increased from 10 to reduce connection creation overhead
+  minimumIdle: ${DB_CONNECTION_POOL_MIN_IDLE:-20}  # HikariCP specific minimum idle connections
+  initialSize: ${DB_CONNECTION_POOL_INITIAL_SIZE:-20}  # Start with more connections ready
+  checkConnectionWhileIdle: ${DB_CONNECTION_CHECK_CONNECTION_WHILE_IDLE:-true}
+  checkConnectionOnBorrow: ${DB_CONNECTION_CHECK_CONNECTION_ON_BORROW:-false}  # Disable for performance
+  evictionInterval: ${DB_CONNECTION_EVICTION_INTERVAL:-5 minutes}
+  minIdleTime: ${DB_CONNECTION_MIN_IDLE_TIME:-5 minute}
+
+
+  # JDBC Driver Properties - Critical for Performance
+  # These work across both PostgreSQL and MySQL drivers
+  properties:
+    # HikariCP connection pool settings
+    connectionTimeout: ${DB_CONNECTION_TIMEOUT:-300000}  # 300 seconds
+    idleTimeout: ${DB_IDLE_TIMEOUT:-600000}  # 10 minutes
+    maxLifetime: ${DB_MAX_LIFETIME:-1800000}  # 30 minutes
+    leakDetectionThreshold: ${DB_LEAK_DETECTION_THRESHOLD:-600000}  # 10 minute
+    keepaliveTime: ${DB_KEEPALIVE_TIME:-0}  # 0 = disabled (set to 30000 for Aurora)
+    validationTimeout: ${DB_VALIDATION_TIMEOUT:-300000}  # 300 seconds
+    # PostgreSQL specific - these are ignored by MySQL driver
+    prepareThreshold: ${DB_PG_PREPARE_THRESHOLD:-1}  # Use prepared statements immediately
+    preparedStatementCacheQueries: ${DB_PG_PREP_STMT_CACHE_QUERIES:-500}  # Cache more statements
+    preparedStatementCacheSizeMiB: ${DB_PG_PREP_STMT_CACHE_SIZE_MB:-10}  # Larger cache
+    reWriteBatchedInserts: ${DB_PG_REWRITE_BATCHED_INSERTS:-true}  # Critical for batch performance
+    defaultRowFetchSize: ${DB_PG_DEFAULT_ROW_FETCH_SIZE:-1000}  # Fetch more rows at once
+    assumeMinServerVersion: ${DB_PG_ASSUME_MIN_SERVER_VERSION:-12}  # Skip version checks
+    ApplicationName: ${DB_PG_APPLICATION_NAME:-OpenMetadata}
+    loginTimeout: ${DB_PG_LOGIN_TIMEOUT:-300}  # Login timeout in seconds
+    postgresqlConnectTimeout: ${DB_POSTGRESQL_CONNECT_TIMEOUT:-60}  # Connection timeout in seconds
+    postgresqlSocketTimeout: ${DB_POSTGRESQL_SOCKET_TIMEOUT:-30000}  # Socket timeout in seconds (0 = infinite)
+
+    # Aurora PostgreSQL specific optimizations
+    loadBalanceHosts: ${DB_PG_LOAD_BALANCE_HOSTS:-false}  # Set to true for Aurora reader endpoints
+    hostRecheckSeconds: ${DB_PG_HOST_RECHECK_SECONDS:-10}  # How often to check host status
+    targetServerType: ${DB_PG_TARGET_SERVER_TYPE:-primary}  # primary, secondary, any, preferSecondary
+
+    # MySQL specific - these are ignored by PostgreSQL driver
+    rewriteBatchedStatements: ${DB_MYSQL_REWRITE_BATCHED_STATEMENTS:-true}  # Critical for MySQL batch
+    cachePrepStmts: ${DB_MYSQL_CACHE_PREP_STMTS:-true}
+    prepStmtCacheSize: ${DB_MYSQL_PREP_STMT_CACHE_SIZE:-500}
+    prepStmtCacheSqlLimit: ${DB_MYSQL_PREP_STMT_CACHE_SQL_LIMIT:-2048}
+    useServerPrepStmts: ${DB_MYSQL_USE_SERVER_PREP_STMTS:-true}
+    useLocalSessionState: ${DB_MYSQL_USE_LOCAL_SESSION_STATE:-true}
+    useLocalTransactionState: ${DB_MYSQL_USE_LOCAL_TRANSACTION_STATE:-true}
+    elideSetAutoCommits: ${DB_MYSQL_ELIDE_SET_AUTO_COMMITS:-true}
+    maintainTimeStats: ${DB_MYSQL_MAINTAIN_TIME_STATS:-false}
+    cacheResultSetMetadata: ${DB_MYSQL_CACHE_RESULT_SET_METADATA:-true}
+    cacheServerConfiguration: ${DB_MYSQL_CACHE_SERVER_CONFIG:-true}
+    tcpKeepAlive: ${DB_MYSQL_TCP_KEEP_ALIVE:-true}
+    tcpNoDelay: ${DB_MYSQL_TCP_NO_DELAY:-true}
+    mysqlConnectTimeout: ${DB_MYSQL_CONNECT_TIMEOUT:-60000}  # Connection timeout in milliseconds
+    mysqlSocketTimeout: ${DB_MYSQL_SOCKET_TIMEOUT:-30000000}  # Socket timeout in milliseconds (0 = infinite)
+
+objectStorage:
+  enabled: false
+  provider: NOOP
+  maxFileSize: 5242880
+
+migrationConfiguration:
+  flywayPath: "./bootstrap/sql/migrations/flyway"
+  nativePath: "./bootstrap/sql/migrations/native"
+  extensionPath: ""
+
+# Authorizer Configuration
+authorizerConfiguration:
+  className: ${AUTHORIZER_CLASS_NAME:-org.openmetadata.service.security.DefaultAuthorizer}
+  containerRequestFilter: ${AUTHORIZER_REQUEST_FILTER:-org.openmetadata.service.security.JwtFilter}
+  adminPrincipals: ${AUTHORIZER_ADMIN_PRINCIPALS:-[admin]}
+  allowedEmailRegistrationDomains: ${AUTHORIZER_ALLOWED_REGISTRATION_DOMAIN:-["all"]}
+  principalDomain: ${AUTHORIZER_PRINCIPAL_DOMAIN:-"open-metadata.org"}
+  allowedDomains: ${AUTHORIZER_ALLOWED_DOMAINS:-[]}
+  enforcePrincipalDomain: ${AUTHORIZER_ENFORCE_PRINCIPAL_DOMAIN:-false}
+  enableSecureSocketConnection : ${AUTHORIZER_ENABLE_SECURE_SOCKET:-false}
+  useRolesFromProvider: ${AUTHORIZER_USE_ROLES_FROM_PROVIDER:-false}
+
+authenticationConfiguration:
+  clientType: ${AUTHENTICATION_CLIENT_TYPE:-public}
+  provider: ${AUTHENTICATION_PROVIDER:-basic}
+  # This is used by auth provider provide response as either id_token or code
+  responseType: ${AUTHENTICATION_RESPONSE_TYPE:-id_token}
+  # This will only be valid when provider type specified is customOidc
+  providerName: ${CUSTOM_OIDC_AUTHENTICATION_PROVIDER_NAME:-""}
+  publicKeyUrls: ${AUTHENTICATION_PUBLIC_KEYS:-[http://localhost:8585/api/v1/system/config/jwks]}
+  tokenValidationAlgorithm: ${AUTHENTICATION_TOKEN_VALIDATION_ALGORITHM:-"RS256"}
+  authority: ${AUTHENTICATION_AUTHORITY:-https://accounts.google.com}
+  clientId: ${AUTHENTICATION_CLIENT_ID:-""}
+  callbackUrl: ${AUTHENTICATION_CALLBACK_URL:-""}
+  jwtPrincipalClaims: ${AUTHENTICATION_JWT_PRINCIPAL_CLAIMS:-[email,preferred_username,sub]}
+  jwtPrincipalClaimsMapping: ${AUTHENTICATION_JWT_PRINCIPAL_CLAIMS_MAPPING:-[]}
+  enableSelfSignup : ${AUTHENTICATION_ENABLE_SELF_SIGNUP:-true}
+  enableAutoRedirect: ${AUTHENTICATION_ENABLE_AUTO_REDIRECT:-false}
+  # Force secure flag on session cookies even when not using HTTPS directly.
+  # Enable this when running behind a proxy/load balancer that handles SSL termination.
+  # Default: false (secure flag only set when HTTPS is detected)
+  forceSecureSessionCookie: ${FORCE_SECURE_SESSION_COOKIE:-false}
+  sessionExpiry: ${AUTHENTICATION_SESSION_EXPIRY:-"604800"} # 7 days; applies to all auth providers
+  maxActiveSessionsPerUser: ${AUTHENTICATION_MAX_ACTIVE_SESSIONS_PER_USER:-5}
+  oidcConfiguration:
+    id: ${OIDC_CLIENT_ID:-""}
+    type: ${OIDC_TYPE:-""} # google, azure etc.
+    secret: ${OIDC_CLIENT_SECRET:-""}
+    scope: ${OIDC_SCOPE:-"openid email profile"}
+    discoveryUri: ${OIDC_DISCOVERY_URI:-""}
+    useNonce: ${OIDC_USE_NONCE:-true}
+    preferredJwsAlgorithm: ${OIDC_PREFERRED_JWS:-"RS256"}
+    responseType: ${OIDC_RESPONSE_TYPE:-"code"}
+    disablePkce: ${OIDC_DISABLE_PKCE:-true}
+    callbackUrl: ${OIDC_CALLBACK:-"http://localhost:8585/callback"}
+    serverUrl: ${OIDC_SERVER_URL:-"http://localhost:8585"}
+    clientAuthenticationMethod: ${OIDC_CLIENT_AUTH_METHOD:-"client_secret_post"}
+    tenant: ${OIDC_TENANT:-""}
+    maxClockSkew: ${OIDC_MAX_CLOCK_SKEW:-""}
+    tokenValidity: ${OIDC_OM_REFRESH_TOKEN_VALIDITY:-"3600"} # in seconds
+    customParams: ${OIDC_CUSTOM_PARAMS:-}
+    maxAge: ${OIDC_MAX_AGE:-"0"}
+    prompt: ${OIDC_PROMPT_TYPE:-"consent"}
+    sessionExpiry: ${OIDC_SESSION_EXPIRY:-"604800"} #7 days
+  samlConfiguration:
+    debugMode: ${SAML_DEBUG_MODE:-false}
+    idp:
+      entityId: ${SAML_IDP_ENTITY_ID:-""}
+      ssoLoginUrl: ${SAML_IDP_SSO_LOGIN_URL:-""}
+      idpX509Certificate: ${SAML_IDP_CERTIFICATE:-""}
+      nameId: ${SAML_IDP_NAME_ID:-"urn:oasis:names:tc:SAML:2.0:nameid-format:emailAddress"}
+    sp:
+      entityId: ${SAML_SP_ENTITY_ID:-"http://localhost:8585/api/v1/saml/metadata"}
+      acs: ${SAML_SP_ACS:-"http://localhost:8585/api/v1/saml/acs"}
+      spX509Certificate: ${SAML_SP_CERTIFICATE:-""}
+      spPrivateKey: ${SAML_SP_PRIVATE_KEY:-""}
+      callback: ${SAML_SP_CALLBACK:-"http://localhost:8585/saml/callback"}
+    security:
+      strictMode: ${SAML_STRICT_MODE:-false}
+      validateXml: ${SAML_VALIDATE_XML:-false}
+      tokenValidity: ${SAML_SP_TOKEN_VALIDITY:-"3600"}
+      sendEncryptedNameId: ${SAML_SEND_ENCRYPTED_NAME_ID:-false}
+      sendSignedAuthRequest: ${SAML_SEND_SIGNED_AUTH_REQUEST:-false}
+      signSpMetadata: ${SAML_SIGNED_SP_METADATA:-false}
+      wantMessagesSigned: ${SAML_WANT_MESSAGE_SIGNED:-false}
+      wantAssertionsSigned: ${SAML_WANT_ASSERTION_SIGNED:-false}
+      wantAssertionEncrypted: ${SAML_WANT_ASSERTION_ENCRYPTED:-false}
+      keyStoreFilePath: ${SAML_KEYSTORE_FILE_PATH:-""}
+      keyStoreAlias: ${SAML_KEYSTORE_ALIAS:-""}
+      keyStorePassword: ${SAML_KEYSTORE_PASSWORD:-""}
+  ldapConfiguration:
+    host: ${AUTHENTICATION_LDAP_HOST:-}
+    port: ${AUTHENTICATION_LDAP_PORT:-}
+    dnAdminPrincipal: ${AUTHENTICATION_LOOKUP_ADMIN_DN:-""}
+    dnAdminPassword: ${AUTHENTICATION_LOOKUP_ADMIN_PWD:-""}
+    userBaseDN: ${AUTHENTICATION_USER_LOOKUP_BASEDN:-""}
+    groupBaseDN: ${AUTHENTICATION_GROUP_LOOKUP_BASEDN:-""}
+    roleAdminName: ${AUTHENTICATION_USER_ROLE_ADMIN_NAME:-}
+    allAttributeName: ${AUTHENTICATION_USER_ALL_ATTR:-}
+    mailAttributeName: ${AUTHENTICATION_USER_MAIL_ATTR:-}
+    usernameAttributeName: ${AUTHENTICATION_USER_NAME_ATTR:-}
+    groupAttributeName: ${AUTHENTICATION_USER_GROUP_ATTR:-}
+    groupAttributeValue: ${AUTHENTICATION_USER_GROUP_ATTR_VALUE:-}
+    groupMemberAttributeName: ${AUTHENTICATION_USER_GROUP_MEMBER_ATTR:-}
+    #the mapping of roles to LDAP groups
+    authRolesMapping: ${AUTH_ROLES_MAPPING:-""}
+    authReassignRoles: ${AUTH_REASSIGN_ROLES:-[]}
+    #optional
+    maxPoolSize: ${AUTHENTICATION_LDAP_POOL_SIZE:-3}
+    sslEnabled: ${AUTHENTICATION_LDAP_SSL_ENABLED:-}
+    truststoreConfigType: ${AUTHENTICATION_LDAP_TRUSTSTORE_TYPE:-TrustAll}
+    trustStoreConfig:
+      customTrustManagerConfig:
+        trustStoreFilePath: ${AUTHENTICATION_LDAP_TRUSTSTORE_PATH:-}
+        trustStoreFilePassword: ${AUTHENTICATION_LDAP_KEYSTORE_PASSWORD:-}
+        trustStoreFileFormat: ${AUTHENTICATION_LDAP_SSL_KEY_FORMAT:-}
+        verifyHostname: ${AUTHENTICATION_LDAP_SSL_VERIFY_CERT_HOST:-}
+        examineValidityDates: ${AUTHENTICATION_LDAP_EXAMINE_VALIDITY_DATES:-}
+      hostNameConfig:
+        allowWildCards: ${AUTHENTICATION_LDAP_ALLOW_WILDCARDS:-}
+        acceptableHostNames: ${AUTHENTICATION_LDAP_ALLOWED_HOSTNAMES:-[]}
+      jvmDefaultConfig:
+        verifyHostname: ${AUTHENTICATION_LDAP_SSL_VERIFY_CERT_HOST:-}
+      trustAllConfig:
+        examineValidityDates: ${AUTHENTICATION_LDAP_EXAMINE_VALIDITY_DATES:-true}
+
+jwtTokenConfiguration:
+  rsapublicKeyFilePath: ${RSA_PUBLIC_KEY_FILE_PATH:-"./conf/public_key.der"}
+  rsaprivateKeyFilePath: ${RSA_PRIVATE_KEY_FILE_PATH:-"./conf/private_key.der"}
+  jwtissuer: ${JWT_ISSUER:-"open-metadata.org"}
+  keyId: ${JWT_KEY_ID:-"Gb389a-9f76-gdjs-a92j-0242bk94356"}
+
+elasticsearch:
+  searchType: ${SEARCH_TYPE:- "opensearch"}
+  # Single host or comma-separated list for multiple hosts
+  # Examples: "localhost" or "es-node1:9200,es-node2:9200,es-node3:9200"
+  host: ${ELASTICSEARCH_HOST:-localhost}
+  port: ${ELASTICSEARCH_PORT:-9200}
+  scheme: ${ELASTICSEARCH_SCHEME:-http}
+  username: ${ELASTICSEARCH_USER:-""}
+  password: ${ELASTICSEARCH_PASSWORD:-""}
+  clusterAlias: ${ELASTICSEARCH_CLUSTER_ALIAS:-""}
+  truststorePath: ${ELASTICSEARCH_TRUST_STORE_PATH:-""}
+  truststorePassword: ${ELASTICSEARCH_TRUST_STORE_PASSWORD:-""}
+  connectionTimeoutSecs: ${ELASTICSEARCH_CONNECTION_TIMEOUT_SECS:-10}  # Increased from 5s for Docker networks
+  socketTimeoutSecs: ${ELASTICSEARCH_SOCKET_TIMEOUT_SECS:-120}  # Increased from 60s for slow queries
+  keepAliveTimeoutSecs: ${ELASTICSEARCH_KEEP_ALIVE_TIMEOUT_SECS:-600}
+  # Connection pool settings for better load balancing and performance
+  maxConnTotal: ${ELASTICSEARCH_MAX_CONN_TOTAL:-30}  # Total connections across all hosts
+  maxConnPerRoute: ${ELASTICSEARCH_MAX_CONN_PER_ROUTE:-10}  # Max connections per host
+  batchSize: ${ELASTICSEARCH_BATCH_SIZE:-100}
+  payLoadSize: ${ELASTICSEARCH_PAYLOAD_BYTES_SIZE:-10485760}
+  searchIndexMappingLanguage : ${ELASTICSEARCH_INDEX_MAPPING_LANG:-EN}
+  searchIndexFactoryClassName : org.openmetadata.service.search.SearchIndexFactory
+  # AWS IAM Authentication for OpenSearch (only applicable when searchType is "opensearch")
+  # Uses standard AWS environment variables: https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-envvars.html
+  # IAM auth is automatically enabled when AWS_DEFAULT_REGION is set
+  # Credentials: Use AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY or rely on AWS SDK default credential provider chain
+  aws:
+    enabled: ${SEARCH_AWS_IAM_AUTH_ENABLED:-false}
+    region: ${AWS_DEFAULT_REGION:-""}
+    accessKeyId: ${AWS_ACCESS_KEY_ID:-""}
+    secretAccessKey: ${AWS_SECRET_ACCESS_KEY:-""}
+    sessionToken: ${AWS_SESSION_TOKEN:-""}
+    serviceName: ${SEARCH_AWS_SERVICE_NAME:-"es"}  # Use "es" for OpenSearch, "aoss" for OpenSearch Serverless
+  naturalLanguageSearch:
+    enabled: ${NATURAL_LANGUAGE_SEARCH_ENABLED:-false}
+    embeddingProvider: ${EMBEDDING_PROVIDER:-bedrock}
+    providerClass: ${NATURAL_LANGUAGE_SEARCH_PROVIDER_CLASS:-org.openmetadata.service.search.nlq.NoOpNLQService}
+    bedrock:
+      awsConfig:
+        enabled: ${BEDROCK_AWS_IAM_AUTH_ENABLED:-false}
+        region: ${AWS_DEFAULT_REGION:-""}
+        accessKeyId: ${AWS_ACCESS_KEY_ID:-""}
+        secretAccessKey: ${AWS_SECRET_ACCESS_KEY:-""}
+        sessionToken: ${AWS_SESSION_TOKEN:-""}
+      modelId: ${AWS_BEDROCK_MODEL_ID:-""}
+      embeddingModelId: ${AWS_BEDROCK_EMBED_MODEL_ID:-""}
+      embeddingDimension: ${AWS_BEDROCK_EMBEDDING_DIMENSION:-""}
+
+eventMonitoringConfiguration:
+  eventMonitor: ${EVENT_MONITOR:-prometheus}  # Possible values are "prometheus", "cloudwatch"
+  batchSize: ${EVENT_MONITOR_BATCH_SIZE:-10}
+  pathPattern: ${EVENT_MONITOR_PATH_PATTERN:-["/api/v1/tables/*", "/api/v1/health-check"]}
+  latency: ${EVENT_MONITOR_LATENCY:-[0.99, 0.90]} # For value p99=0.99, p90=0.90, p50=0.50 etc.
+  servicesHealthCheckInterval: ${EVENT_MONITOR_SERVICES_HEALTH_CHECK_INTERVAL:-300}
+  # it will use the default auth provider for AWS services if parameters are not set
+  # parameters:
+  #   region: ${OM_MONITOR_REGION:-""}
+  #   accessKeyId: ${OM_MONITOR_ACCESS_KEY_ID:-""}
+  #   secretAccessKey: ${OM_MONITOR_ACCESS_KEY:-""}
+
+eventHandlerConfiguration:
+  eventHandlerClassNames:
+    - "org.openmetadata.service.events.AuditEventHandler"
+    - "org.openmetadata.service.events.ChangeEventHandler"
+
+pipelineServiceClientConfiguration:
+  enabled: ${PIPELINE_SERVICE_CLIENT_ENABLED:-true}
+  # If we don't need this, set "org.openmetadata.service.clients.pipeline.noop.NoopClient"
+  className: ${PIPELINE_SERVICE_CLIENT_CLASS_NAME:-"org.openmetadata.service.clients.pipeline.airflow.AirflowRESTClient"}
+  apiEndpoint: ${PIPELINE_SERVICE_CLIENT_ENDPOINT:-http://localhost:8080}
+  metadataApiEndpoint: ${SERVER_HOST_API_URL:-http://localhost:8585/api}
+  ingestionIpInfoEnabled: ${PIPELINE_SERVICE_IP_INFO_ENABLED:-false}
+  hostIp: ${PIPELINE_SERVICE_CLIENT_HOST_IP:-""}
+  healthCheckInterval: ${PIPELINE_SERVICE_CLIENT_HEALTH_CHECK_INTERVAL:-300}
+  # This SSL information is about the OpenMetadata server.
+  # It will be picked up from the pipelineServiceClient to use/ignore SSL when connecting to the OpenMetadata server.
+  verifySSL: ${PIPELINE_SERVICE_CLIENT_VERIFY_SSL:-"no-ssl"} # Possible values are "no-ssl", "ignore", "validate"
+  sslConfig:
+    certificatePath: ${PIPELINE_SERVICE_CLIENT_SSL_CERT_PATH:-""} # Local path for the Pipeline Service Client
+  logStorageConfiguration:
+    type: ${PIPELINE_SERVICE_CLIENT_LOG_TYPE:-"default"} # Possible values are "default", "s3"
+    enabled: ${PIPELINE_SERVICE_CLIENT_LOG_ENABLED:-false} # Enable it for pipelines deployed in the server
+    # if type is s3, provide the following configuration
+    bucketName: ${PIPELINE_SERVICE_CLIENT_LOG_BUCKET_NAME:-""}
+    # optional path within the bucket to store the logs
+    prefix: ${PIPELINE_SERVICE_CLIENT_LOG_PREFIX:-""}
+    enableServerSideEncryption: ${PIPELINE_SERVICE_CLIENT_LOG_SSE_ENABLED:-false}
+    sseAlgorithm: ${PIPELINE_SERVICE_CLIENT_LOG_SSE_ALGORITHM:-"AES256"} # Allowed values: "AES256" or "aws:kms"
+    kmsKeyId: ${PIPELINE_SERVICE_CLIENT_LOG_KMS_KEY_ID:-""} # Required only if sseAlgorithm is "aws:kms"
+    awsConfig:
+      enabled: ${PIPELINE_SERVICE_CLIENT_AWS_IAM_AUTH_ENABLED:-false}
+      awsAccessKeyId: ${PIPELINE_SERVICE_CLIENT_LOG_AWS_ACCESS_KEY_ID:-""}
+      awsSecretAccessKey: ${PIPELINE_SERVICE_CLIENT_LOG_AWS_SECRET_ACCESS_KEY:-""}
+      awsRegion: ${PIPELINE_SERVICE_CLIENT_LOG_REGION:-""}
+      awsSessionToken: ${PIPELINE_SERVICE_CLIENT_LOG_AWS_SESSION_TOKEN:-""}
+      endPointURL: ${PIPELINE_SERVICE_CLIENT_LOG_AWS_ENDPOINT_URL:-""} # port forward localhost:9000 for minio
+
+  # Secrets Manager Loader: specify to the Ingestion Framework how to load the SM credentials from its env
+  # Supported: noop, airflow, env
+  secretsManagerLoader: ${PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER:-"noop"}
+
+  # Default required parameters for Airflow as Pipeline Service Client
+  parameters:
+    ## Airflow parameters
+    username: ${AIRFLOW_USERNAME:-admin}
+    password: ${AIRFLOW_PASSWORD:-admin}
+    timeout: ${AIRFLOW_TIMEOUT:-10}
+    # If we need to use SSL to reach Airflow
+    truststorePath: ${AIRFLOW_TRUST_STORE_PATH:-""}
+    truststorePassword: ${AIRFLOW_TRUST_STORE_PASSWORD:-""}
+    ## Kubernetes client parameters
+    namespace: ${K8S_NAMESPACE:-"openmetadata-pipelines"}
+    ingestionImage: ${K8S_INGESTION_IMAGE:-"docker.getcollate.io/openmetadata/ingestion-base:latest"}
+    imagePullPolicy: ${K8S_IMAGE_PULL_POLICY:-"IfNotPresent"}
+    imagePullSecrets: ${K8S_IMAGE_PULL_SECRETS:-""}
+    serviceAccountName: ${K8S_SERVICE_ACCOUNT_NAME:-"openmetadata-ingestion"}
+    # Resources configuration
+    resources:
+      limits:
+        cpu: ${K8S_LIMITS_CPU:-"2"}
+        memory: ${K8S_LIMITS_MEMORY:-"4Gi"}
+      requests:
+        cpu: ${K8S_REQUESTS_CPU:-"500m"}
+        memory: ${K8S_REQUESTS_MEMORY:-"1Gi"}
+    ttlSecondsAfterFinished: ${K8S_TTL_SECONDS_AFTER_FINISHED:-604800}
+    activeDeadlineSeconds: ${K8S_ACTIVE_DEADLINE_SECONDS:-604800}
+    backoffLimit: ${K8S_BACKOFF_LIMIT:-3}
+    successfulJobsHistoryLimit: ${K8S_SUCCESSFUL_JOBS_HISTORY_LIMIT:-3}
+    failedJobsHistoryLimit: ${K8S_FAILED_JOBS_HISTORY_LIMIT:-1}
+    nodeSelector: ${K8S_NODE_SELECTOR:-""}
+    runAsUser: ${K8S_RUN_AS_USER:-1000}
+    runAsGroup: ${K8S_RUN_AS_GROUP:-1000}
+    fsGroup: ${K8S_FS_GROUP:-1000}
+    runAsNonRoot: ${K8S_RUN_AS_NON_ROOT:-"true"}
+    extraEnvVars: ${K8S_EXTRA_ENV_VARS:-[]}
+    podAnnotations: ${K8S_POD_ANNOTATIONS:-""}
+    useOMJobOperator: ${USE_OMJOB_OPERATOR:-"true"}
+
+
+# no_encryption_at_rest is the default value, and it does what it says. Please read the manual on how
+# to secure your instance of OpenMetadata with TLS and encryption at rest.
+fernetConfiguration:
+  fernetKey: ${FERNET_KEY:-jJ/9sz0g0OHxsfxOoSfdFdmk3ysNmPRnH3TUAbz3IHA=}
+
+secretsManagerConfiguration:
+  secretsManager: ${SECRET_MANAGER:-db} # Possible values are "db", "managed-aws","aws", "managed-aws-ssm", "aws-ssm", "managed-azure-kv", "azure-kv", "in-memory", "gcp", "kubernetes"
+  prefix: ${SECRET_MANAGER_PREFIX:-""} # Define the secret key ID as /<prefix>/<clusterName>/<key>
+  tags: ${SECRET_MANAGER_TAGS:-[]} # Add tags to the created resource. Format is `[key1:value1,key2:value2,...]`
+  # it will use the default auth provider for the secrets' manager service if parameters are not set
+  parameters:
+    ## For AWS
+    accessKeyId: ${OM_SM_ACCESS_KEY_ID:-""}
+    secretAccessKey: ${OM_SM_ACCESS_KEY:-""}
+    ## For Azure Key Vault
+    clientId: ${OM_SM_CLIENT_ID:-""}
+    clientSecret: ${OM_SM_CLIENT_SECRET:-""}
+    tenantId: ${OM_SM_TENANT_ID:-""}
+    vaultName: ${OM_SM_VAULT_NAME:-""}
+    ## For GCP
+    projectId: ${OM_SM_PROJECT_ID:-""}
+    ## For Kubernetes
+    namespace: ${OM_SM_NAMESPACE:-"default"}
+    kubeconfigPath: ${OM_SM_KUBECONFIG_PATH:-""}
+    inCluster: ${OM_SM_IN_CLUSTER:-"false"}
+
+health:
+  delayedShutdownHandlerEnabled: true
+  shutdownWaitPeriod: 1s
+  healthChecks:
+    - name: OpenMetadataServerHealthCheck
+      critical: true
+      schedule:
+        checkInterval: 2500ms
+        downtimeInterval: 10s
+        failureAttempts: 2
+        successAttempts: 1
+
+limits:
+  enable: ${LIMITS_ENABLED:-false}
+  className: ${LIMITS_CLASS_NAME:-"org.openmetadata.service.limits.DefaultLimits"}
+  limitsConfigFile: ${LIMITS_CONFIG_FILE:-""}
+
+# Bulk Operation Configuration
+# Controls parallelism and resource usage for bulk API operations (e.g., bulk import/export)
+# Uses a bounded thread pool to prevent connection pool exhaustion
+bulkOperation:
+  # Max threads for bulk operations (recommendations: 2 vCore=5-8, 4 vCore=8-15, 8 vCore=15-25)
+  maxThreads: ${BULK_OPERATION_MAX_THREADS:-10}
+  # Max queued operations before rejection (returns 503)
+  queueSize: ${BULK_OPERATION_QUEUE_SIZE:-1000}
+  # Timeout in seconds for entire bulk operation
+  timeoutSeconds: ${BULK_OPERATION_TIMEOUT_SECONDS:-300}
+
+web:
+  uriPath: ${WEB_CONF_URI_PATH:-"/api"}
+  hsts:
+    enabled: ${WEB_CONF_HSTS_ENABLED:-false}
+    maxAge: ${WEB_CONF_HSTS_MAX_AGE:-"365 days"}
+    includeSubDomains: ${WEB_CONF_HSTS_INCLUDE_SUBDOMAINS:-"true"}
+    preload: ${WEB_CONF_HSTS_PRELOAD:-"true"}
+  frame-options:
+    enabled: ${WEB_CONF_FRAME_OPTION_ENABLED:-false}
+    option: ${WEB_CONF_FRAME_OPTION:-"SAMEORIGIN"}
+    origin: ${WEB_CONF_FRAME_ORIGIN:-""}
+  content-type-options:
+    enabled: ${WEB_CONF_CONTENT_TYPE_OPTIONS_ENABLED:-false}
+  xss-protection:
+    enabled: ${WEB_CONF_XSS_PROTECTION_ENABLED:-false}
+    on: ${WEB_CONF_XSS_PROTECTION_ON:-true}
+    block: ${WEB_CONF_XSS_PROTECTION_BLOCK:-true}
+  csp:
+    enabled: ${WEB_CONF_XSS_CSP_ENABLED:-false}
+    policy: ${WEB_CONF_XSS_CSP_POLICY:-"default-src 'self'"}
+    reportOnlyPolicy: ${WEB_CONF_XSS_CSP_REPORT_ONLY_POLICY:-""}
+  referrer-policy:
+    enabled: ${WEB_CONF_REFERRER_POLICY_ENABLED:-false}
+    option: ${WEB_CONF_REFERRER_POLICY_OPTION:-"SAME_ORIGIN"}
+  permission-policy:
+    enabled: ${WEB_CONF_PERMISSION_POLICY_ENABLED:-false}
+    option: ${WEB_CONF_PERMISSION_POLICY_OPTION:-""}
+  cache-control: ${WEB_CONF_CACHE_CONTROL:-""}
+  pragma: ${WEB_CONF_PRAGMA:-""}
+
+operationalConfig:
+  enable: ${OPERATIONAL_CONFIG_ENABLED:-true}
+  operationsConfigFile: ${OPERATIONAL_CONFIG_FILE:-"./conf/operations.yaml"}
+
+rdf:
+  enabled: ${RDF_ENABLED:-false}
+  baseUri: ${RDF_BASE_URI:-"https://open-metadata.org/"}
+  storageType: ${RDF_STORAGE_TYPE:-"FUSEKI"}
+  remoteEndpoint: ${RDF_ENDPOINT:-"http://localhost:3030/openmetadata"}
+  username: ${RDF_REMOTE_USERNAME:-"admin"}
+  password: ${RDF_REMOTE_PASSWORD:-"admin"}
+  dataset: ${RDF_DATASET:-"openmetadata"}
+
+# Cache Configuration
+# Caching layer for entity metadata, relationships, and tag usage to reduce database load
+# Default: Disabled (uses NoopCacheProvider)
+cache:
+  # Cache provider: none (default) or redis
+  provider: ${CACHE_PROVIDER:-none}
+
+  # TTL (Time To Live) settings in seconds
+  entityTtlSeconds: ${CACHE_ENTITY_TTL:-172800}             # 48 hour for entities
+  relationshipTtlSeconds: ${CACHE_RELATIONSHIP_TTL:-172800} # 48 hour for relationships
+  tagTtlSeconds: ${CACHE_TAG_TTL:-172800}                   # 48 hour for tags
+
+  # Redis configuration
+  redis:
+    # Redis connection URL
+    # Standalone: redis://localhost:6379
+    # AWS ElastiCache: redis://my-cluster.abc123.cache.amazonaws.com:6379
+    url: ${CACHE_REDIS_URL:-redis://localhost:6379}
+    authType: ${CACHE_REDIS_AUTH_TYPE:-NONE}
+    database: ${CACHE_REDIS_DATABASE:-0}  # Redis database index (0-15)
+
+    # Authentication for standalone Redis
+    username: ${CACHE_REDIS_USERNAME:-}
+    passwordRef: ${CACHE_REDIS_PASSWORD:-}  # Reference to password in secrets manager
+    useSSL: ${CACHE_REDIS_USE_SSL:-false}
+
+    # Key namespace prefix (useful for multi-tenant deployments)
+    keyspace: ${CACHE_REDIS_KEYSPACE:-"om:prod"}
+
+    # Connection pool settings
+    poolSize: ${CACHE_REDIS_POOL_SIZE:-64}
+    connectTimeoutMs: ${CACHE_REDIS_CONNECT_TIMEOUT:-2000}
+
+    # AWS ElastiCache IAM Authentication (only if using ElastiCache)
+    aws:
+      enabled: ${CACHE_REDIS_AWS_IAM_AUTH_ENABLED:-false}
+      region: ${CACHE_REDIS_AWS_REGION:-""}
+      useInstanceProfile: ${CACHE_REDIS_AWS_INSTANCE_PROFILE:-true}
+      # If not using instance profile, provide credentials:
+      accessKeyId: ${AWS_ACCESS_KEY_ID:-""}
+      secretAccessKey: ${AWS_SECRET_ACCESS_KEY:-""}
+      tokenRefreshIntervalSeconds: ${CACHE_REDIS_TOKEN_REFRESH:-900}  # 15 minutes