chore: import upstream snapshot with attribution

docker/development/helm/README.md

@@ -0,0 +1,295 @@
+# OpenMetadata Helm Chart Local Testing
+
+Helper to test changes from https://github.com/open-metadata/openmetadata-helm-charts with local images while developing.
+
+## Prerequisites
+
+### Required Tools
+```bash
+# Install required tools
+brew install helm kubectl minikube docker-compose
+
+# Or on Ubuntu/Debian:
+# sudo snap install helm kubectl minikube
+# sudo apt-get install docker-compose
+```
+
+### Required Resources
+- **CPU**: 4+ cores recommended
+- **Memory**: 8GB+ RAM recommended  
+- **Storage**: 20GB+ free space
+
+## Quick Start
+
+### 1. Start Local Dependencies
+
+First, start the required database and search services:
+
+```bash
+cd docker/development/helm
+
+# Start PostgreSQL and OpenSearch
+docker-compose -f docker-compose-deps.yml up -d
+
+# Wait for services to be ready (2-3 minutes)
+docker-compose -f docker-compose-deps.yml logs -f
+
+# Verify services are running
+curl http://localhost:9200/_cluster/health
+docker exec openmetadata_postgres_test psql -U openmetadata_user -d openmetadata_db -c "SELECT 1"
+```
+
+### 2. Start Kubernetes Cluster
+
+```bash
+# Start minikube with sufficient resources
+minikube start --cpus 4 --memory 8192 --driver docker
+
+# Enable ingress addon (optional)
+minikube addons enable ingress
+
+# Verify cluster is ready
+kubectl cluster-info
+kubectl get nodes
+```
+
+### 3. Create Required Secrets
+
+```bash
+# Create secrets that the chart expects
+kubectl create secret generic postgres-secrets \
+  --from-literal=openmetadata-postgres-password=openmetadata_password
+
+kubectl create secret generic airflow-secrets \
+  --from-literal=openmetadata-airflow-password=admin
+```
+
+## Test Scenarios
+
+### Scenario A: Test Kubernetes Native Pipeline Client (NEW)
+
+Test the new K8s native pipeline execution without Airflow dependencies.
+
+```bash
+# Use local chart directly, e.g.,
+CHART_PATH="/Users/pmbrull/github/openmetadata-helm-charts/charts/openmetadata"
+
+# Install with K8s native configuration
+helm install openmetadata-k8s-test $CHART_PATH --values values-k8s-test.yaml
+
+# Check deployment status
+kubectl get pods -A
+kubectl get jobs -n openmetadata-pipelines-test
+kubectl get serviceaccounts,roles,rolebindings -n openmetadata-pipelines-test
+
+# Check logs
+kubectl logs -l app.kubernetes.io/name=openmetadata -f
+
+# Test access
+minikube tunnel  # In separate terminal
+curl http://localhost:8585/api/v1/system/health
+```
+
+### Scenario B: Test Migrated Airflow Configuration
+
+Test that the new nested Airflow configuration still works.
+
+```bash
+# Install with migrated Airflow configuration  
+helm install openmetadata-airflow-test $CHART_PATH \
+  --values values-airflow-test.yaml \
+  --timeout 10m \
+  --wait
+
+# Check deployment
+kubectl get pods -A
+kubectl logs -l app.kubernetes.io/name=openmetadata -f
+```
+
+## Validation Checklist
+
+### ✅ Basic Functionality
+
+1. **Pod Health**
+```bash
+# Check all pods are running
+kubectl get pods -A
+
+# Check OpenMetadata pod logs
+kubectl logs deployment/openmetadata-k8s-test -f
+
+# Check database connectivity
+kubectl exec deployment/openmetadata-k8s-test -- curl -f http://postgres:5432 || echo "DB connection test"
+```
+
+2. **API Health**
+```bash
+# Access health endpoint
+curl http://localhost:8585/api/v1/system/health
+
+# Check API response
+curl http://localhost:8585/api/v1/system/config
+```
+
+### ✅ K8s Native Pipeline Features
+
+1. **RBAC Resources**
+```bash
+# Check namespace creation
+kubectl get namespace openmetadata-pipelines-test
+
+# Check service account
+kubectl get serviceaccount -n openmetadata-pipelines-test openmetadata-ingestion-test
+
+# Check permissions
+kubectl auth can-i create jobs \
+  --as=system:serviceaccount:openmetadata-pipelines-test:openmetadata-ingestion-test \
+  -n openmetadata-pipelines-test
+```
+
+2. **Configuration Validation**
+```bash
+# Check environment variables in pod
+kubectl exec deployment/openmetadata-k8s-test -- env | grep K8S_
+
+# Check secrets
+kubectl get secret openmetadata-k8s-test-pipeline-secret -o yaml
+kubectl get secret openmetadata-k8s-test-pipeline-secret -o jsonpath='{.data}' | base64 -d
+```
+
+3. **Pipeline Job Testing**
+```bash
+# Create a test pipeline job (manual)
+cat <<EOF | kubectl apply -f -
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: test-ingestion-job
+  namespace: openmetadata-pipelines-test
+spec:
+  template:
+    spec:
+      serviceAccountName: openmetadata-ingestion-test
+      containers:
+      - name: test
+        image: docker.getcollate.io/openmetadata/ingestion:latest
+        command: ["echo", "Test pipeline job works"]
+      restartPolicy: Never
+EOF
+
+# Check job execution
+kubectl get jobs -n openmetadata-pipelines-test
+kubectl logs job/test-ingestion-job -n openmetadata-pipelines-test
+```
+
+### ✅ Failure Diagnostics Testing
+
+1. **Create Failing Job**
+```bash
+# Create a job that will fail
+cat <<EOF | kubectl apply -f -
+apiVersion: batch/v1
+kind: Job  
+metadata:
+  name: test-failing-job
+  namespace: openmetadata-pipelines-test
+  labels:
+    app.kubernetes.io/pipeline: test-pipeline
+    app.kubernetes.io/run-id: test-123
+spec:
+  template:
+    spec:
+      serviceAccountName: openmetadata-ingestion-test
+      containers:
+      - name: main
+        image: docker.getcollate.io/openmetadata/ingestion:latest
+        command: ["sh", "-c", "echo 'Starting ingestion...'; sleep 10; echo 'Something went wrong!'; exit 1"]
+      restartPolicy: Never
+EOF
+
+# Watch for diagnostic job creation
+kubectl get jobs -n openmetadata-pipelines-test -w
+
+# Check diagnostic job logs
+kubectl logs -n openmetadata-pipelines-test -l app.kubernetes.io/component=diagnostics
+```
+
+### ✅ Configuration Migration Testing
+
+1. **Test Both Configurations**
+```bash
+# Test that both airflow and k8s configs are valid
+helm template test-airflow $CHART_PATH --values values-airflow-test.yaml > /tmp/airflow-manifest.yaml
+helm template test-k8s $CHART_PATH --values values-k8s-test.yaml > /tmp/k8s-manifest.yaml
+
+# Validate manifests
+kubectl apply --dry-run=client -f /tmp/airflow-manifest.yaml
+kubectl apply --dry-run=client -f /tmp/k8s-manifest.yaml
+```
+
+2. **Test Breaking Changes**
+```bash
+# Test old configuration format (should fail validation or show warnings)
+cat > /tmp/old-values.yaml << EOF
+openmetadata:
+  config:
+    pipelineServiceClientConfig:
+      enabled: true
+      className: "org.openmetadata.service.clients.pipeline.airflow.AirflowRESTClient"
+      apiEndpoint: http://test
+EOF
+
+helm template test-old $CHART_PATH --values /tmp/old-values.yaml
+```
+
+### Debug Commands
+
+```bash
+# Get all resources
+kubectl get all -A
+
+# Describe OpenMetadata deployment
+kubectl describe deployment openmetadata-k8s-test
+
+# Get events
+kubectl get events --sort-by='.lastTimestamp' -A
+
+# Check helm release
+helm list
+helm status openmetadata-k8s-test
+helm get values openmetadata-k8s-test
+```
+
+## Cleanup
+
+### Remove Test Deployments
+```bash
+# Remove helm releases
+helm uninstall openmetadata-k8s-test
+helm uninstall openmetadata-airflow-test
+
+# Clean up namespaces
+kubectl delete namespace openmetadata-pipelines-test
+
+# Clean up secrets
+kubectl delete secret postgres-secrets airflow-secrets
+```
+
+## Expected Results
+
+### Successful Deployment Indicators
+
+1. ✅ **All pods running**: `kubectl get pods -A` shows all pods in `Running` state
+2. ✅ **Health check passing**: `curl http://localhost:8585/api/v1/system/health` returns 200
+3. ✅ **RBAC created**: K8s resources created in `openmetadata-pipelines-test` namespace
+4. ✅ **Configuration loaded**: Environment variables properly set in pods
+5. ✅ **No errors in logs**: `kubectl logs deployment/openmetadata-k8s-test` shows successful startup
+
+### Performance Benchmarks
+
+- **Startup time**: < 5 minutes for full deployment
+- **Memory usage**: < 4GB total (including dependencies)
+- **CPU usage**: < 2 cores under normal load
+- **Job creation**: < 30 seconds for pipeline job creation and execution
+
+This comprehensive testing suite validates both the new K8s native functionality and ensures backward compatibility with existing Airflow configurations.

docker/development/helm/docker-compose-deps.yml

@@ -0,0 +1,82 @@
+# Docker compose for OpenMetadata dependencies (PostgreSQL + OpenSearch)
+# Use this for local Helm chart testing
+
+version: "3.9"
+services:
+
+  # PostgreSQL Database
+  postgres:
+    container_name: openmetadata_postgres_test
+    build:
+      context: ../../../
+      dockerfile: docker/postgresql/Dockerfile_postgres
+    restart: always
+    environment:
+      POSTGRES_DB: openmetadata_db
+      POSTGRES_USER: openmetadata_user
+      POSTGRES_PASSWORD: openmetadata_password
+      POSTGRES_ROOT_PASSWORD: password
+      PGDATA: /var/lib/postgresql/data/pgdata
+    expose:
+      - 5432
+    ports:
+      - "5433:5432"  # Use different port to avoid conflicts
+    networks:
+      - openmetadata_network
+    volumes:
+      - ./docker-volume/db-data-postgres:/var/lib/postgresql/data
+    command: >
+      postgres
+      -c shared_preload_libraries=pg_stat_statements
+      -c max_connections=200
+      -c shared_buffers=256MB
+      -c effective_cache_size=1GB
+      -c maintenance_work_mem=64MB
+      -c checkpoint_completion_target=0.9
+      -c wal_buffers=16MB
+      -c default_statistics_target=100
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U openmetadata_user -d openmetadata_db"]
+      timeout: 20s
+      retries: 10
+      interval: 30s
+
+  # OpenSearch for metadata search
+  opensearch:
+    container_name: openmetadata_opensearch_test
+    image: opensearchproject/opensearch:3.4.0
+    restart: always
+    environment:
+      - discovery.type=single-node
+      - node.name=opensearch
+      - cluster.name=opensearch-cluster
+      - bootstrap.memory_lock=true
+      - "OPENSEARCH_JAVA_OPTS=-Xms1024m -Xmx1024m"
+      - "DISABLE_INSTALL_DEMO_CONFIG=true"
+      - "DISABLE_SECURITY_PLUGIN=true"
+    ulimits:
+      memlock:
+        soft: -1
+        hard: -1
+      nofile:
+        soft: 65536
+        hard: 65536
+    ports:
+      - "9200:9200"
+      - "9300:9300"
+    networks:
+      - openmetadata_network
+    volumes:
+      - opensearch_data:/usr/share/opensearch/data
+    healthcheck:
+      test: ["CMD-SHELL", "curl -f http://localhost:9200/_cluster/health || exit 1"]
+      interval: 30s
+      timeout: 10s
+      retries: 5
+
+volumes:
+  opensearch_data:
+
+networks:
+  openmetadata_network:
+    driver: bridge

docker/development/helm/values-airflow-test.yaml

@@ -0,0 +1,67 @@
+# Test values for migrated Airflow pipeline client validation  
+# Only contains essential overrides for testing the new nested structure
+
+openmetadata:
+  config:
+    # Enable debug logging for testing
+    logLevel: DEBUG
+    
+    # Database configuration (using local PostgreSQL via host machine)
+    database:
+      host: host.docker.internal
+      port: 5433
+      driverClass: org.postgresql.Driver
+      dbScheme: postgresql
+      auth:
+        password:
+          secretRef: postgres-secrets
+          secretKey: openmetadata-postgres-password
+    
+    # Search configuration (using local OpenSearch via host machine)
+    elasticsearch:
+      host: host.docker.internal
+      port: 9200
+      searchType: opensearch
+      scheme: http
+    
+    # Airflow Pipeline Client Configuration (updated structure with common configs)
+    pipelineServiceClientConfig:
+      type: "airflow"  # Use traditional Airflow approach
+      # Common configuration for all pipeline service clients
+      metadataApiEndpoint: http://openmetadata:8585/api
+      airflow:
+        # Airflow-specific connection settings
+        apiEndpoint: http://openmetadata-dependencies-api-server:8080
+        verifySsl: "no-ssl"
+        auth:
+          username: admin
+          password:
+            secretRef: airflow-secrets
+            secretKey: openmetadata-airflow-password
+
+# Local image configuration
+image:
+  repository: openmetadata/server
+  tag: "20260107"
+  pullPolicy: IfNotPresent
+
+# Service configuration for local access
+service:
+  type: LoadBalancer  # For minikube tunnel access
+
+# Reduced resources for local testing
+resources:
+  limits:
+    cpu: 2
+    memory: 4Gi
+  requests:
+    cpu: 500m
+    memory: 1Gi
+
+# More relaxed health checks for local testing
+livenessProbe:
+  initialDelaySeconds: 120
+readinessProbe:
+  initialDelaySeconds: 120
+startupProbe:
+  failureThreshold: 10

docker/development/helm/values-k8s-test.yaml

@@ -0,0 +1,112 @@
+# Test values for K8s native pipeline client validation
+# Only contains essential overrides for testing
+
+openmetadata:
+  config:
+    # Enable debug logging for testing
+    logLevel: DEBUG
+    
+    # Database configuration (using local PostgreSQL via host machine)
+    database:
+      host: host.docker.internal
+      port: 5432
+      driverClass: org.postgresql.Driver
+      dbScheme: postgresql
+      auth:
+        password:
+          secretRef: postgres-secrets
+          secretKey: openmetadata-postgres-password
+    
+    # Search configuration (using local OpenSearch via host machine)
+    elasticsearch:
+      host: host.docker.internal
+      port: 9200
+      searchType: opensearch
+      scheme: http
+    
+    # K8s Pipeline Client Configuration - Updated structure with common configs
+    pipelineServiceClientConfig:
+      type: "k8s"  # Use Kubernetes native instead of Airflow
+      # Common configuration for all pipeline service clients
+      # Override metadataApiEndpoint for cross-namespace access (default uses service name only)
+      metadataApiEndpoint: "http://openmetadata.openmetadata-ingestion.svc.cluster.local:8585/api"
+      k8s:
+        # K8s-specific configuration
+        # Use release namespace (empty = defaults to release namespace)
+        namespace: ""
+        serviceAccountName: "openmetadata-ingestion"
+        ttlSecondsAfterFinished: 10
+        
+        # Local ingestion image
+        ingestionImage: "openmetadata/collate-base:1.12.0-20260402"
+        imagePullPolicy: "IfNotPresent"
+        
+        # Reduced resources for local testing
+        resources:
+          limits:
+            cpu: "1"
+            memory: "2Gi"
+          requests:
+            cpu: "100m"
+            memory: "512Mi"
+
+        # Test tolerations for ingestion pods
+        tolerations:
+          - key: "dedicated"
+            operator: "Equal"
+            value: "ingestion"
+            effect: "NoSchedule"
+
+        # Enable OMJob operator for guaranteed exit handler execution
+        useOMJobOperator: true
+
+# Local image configuration
+image:
+  repository: openmetadata/server
+  tag: "1.12.0"
+  pullPolicy: IfNotPresent
+
+# Service configuration for local access
+service:
+  type: LoadBalancer  # For minikube tunnel access
+
+# OMJob Operator configuration
+omjobOperator:
+  enabled: true
+  # Use locally built fixed image
+  image:
+    repository: docker.getcollate.io/openmetadata/omjob-operator
+    tag: "1.12.0-SNAPSHOT"
+    pullPolicy: IfNotPresent
+  # Adjusted resources for Java application (was OOMKilled at 128Mi)
+  resources:
+    requests:
+      cpu: "500m"
+      memory: "512Mi"
+    limits:
+      cpu: "1"
+      memory: "1Gi"
+  # Debug logging for testing
+  env:
+    logLevel: "DEBUG"
+    watchNamespaces: ""
+  # Enable health checks now that operator implements proper endpoints
+  healthCheck:
+    enabled: true
+
+# Reduced resources for local testing
+resources:
+  limits:
+    cpu: 2
+    memory: 4Gi
+  requests:
+    cpu: 500m
+    memory: 1Gi
+
+# More relaxed health checks for local testing
+livenessProbe:
+  initialDelaySeconds: 120
+readinessProbe:
+  initialDelaySeconds: 120
+startupProbe:
+  failureThreshold: 10