open-metadata--openmetadata/skills/standards/main.md

OpenMetadata Connector Standards

Architecture: Schema-First

OpenMetadata connectors follow a schema-first architecture. One JSON Schema definition cascades through 6 layers:

JSON Schema (single source of truth)
    ├── Python Pydantic models     (make generate)
    ├── Java models                (mvn install -pl openmetadata-spec)
    ├── TypeScript types           (yarn parse-schema)
    ├── UI config forms            (RJSF auto-renders from schema)
    ├── API request validation     (server uses Java models)
    └── Test fixtures              (tests import Pydantic models)

Never hand-write config classes. Define the JSON Schema; everything else is generated.

Connector Anatomy

Every connector lives at ingestion/src/metadata/ingestion/source/{service_type}/{name}/ and has:

File	Purpose	Required
__init__.py	Module marker	Always
connection.py	Create and test connections	Always
metadata.py	Extract metadata from the source	Always
service_spec.py	Register connector with the framework	Always
client.py	REST/SDK client wrapper	Non-database
queries.py	SQL query templates	Database
lineage.py	Lineage extraction	If lineage capability
usage.py	Usage extraction	If usage capability
query_parser.py	Query log parsing	If lineage or usage
CONNECTOR_CONTEXT.md	AI implementation brief	Generated by scaffold

Service Types

Service Type	Base Class	Reference
database	CommonDbSourceService	mysql/
dashboard	DashboardServiceSource	metabase/
pipeline	PipelineServiceSource	airflow/
messaging	MessagingServiceSource	kafka/
mlmodel	MlModelServiceSource	mlflow/
storage	StorageServiceSource	s3/
search	SearchServiceSource	elasticsearch/
api	ApiServiceSource	rest/

Connection Types (Database Only)

Type	Base Class	Pattern
sqlalchemy	BaseConnection[Config, Engine]	SQLAlchemy dialect + engine
rest_api	get_connection() / test_connection()	Custom REST client
sdk_client	get_connection() / test_connection()	Vendor SDK wrapper

Non-database connectors always use get_connection() / test_connection() functions.

ServiceSpec System

Every connector declares a ServiceSpec in service_spec.py:

• Database: DefaultDatabaseSpec(metadata_source_class=..., connection_class=..., lineage_source_class=..., usage_source_class=...)
• All others: BaseSpec(metadata_source_class=...)

The framework resolves specs dynamically via: metadata.ingestion.source.{service_type}.{name}.service_spec.ServiceSpec

Registration Checklist

To register a new connector, modify these files:

1. Service enum: openmetadata-spec/.../entity/services/{serviceType}Service.json — add type to enum + connection oneOf
2. Test connection: openmetadata-service/.../testConnections/{serviceType}/{name}.json — create file
3. UI utils: openmetadata-ui/.../utils/{ServiceType}ServiceUtils.tsx — import schema + add switch case
4. Localization: openmetadata-ui/.../locale/languages/ — add i18n display name keys

Code Generation Commands

source env/bin/activate
make generate                                # Python Pydantic models
mvn clean install -pl openmetadata-spec      # Java models
cd openmetadata-ui/src/main/resources/ui && yarn parse-schema  # UI schemas
make py_format                               # Format Python
mvn spotless:apply                           # Format Java