open-metadata--openmetadata/AGENTS.md

# AGENTS.md

This file provides guidance to Codex (Codex.ai/code) when working with code in this repository.

## About OpenMetadata

OpenMetadata is a unified metadata platform for data discovery, data observability, and data governance. This is a multi-module project with Java backend services, React frontend, Python ingestion framework, and comprehensive Docker infrastructure.

## Architecture Overview

- **Backend**: Java 21 + Dropwizard REST API framework, multi-module Maven project
- **Frontend**: React + TypeScript + Ant Design, built with Webpack and Yarn
- **Ingestion**: Python 3.10-3.12 with Pydantic 2.x, 75+ data source connectors
- **Database**: MySQL (default) or PostgreSQL with Flyway migrations
- **Search**: Elasticsearch 7.17+ or OpenSearch 2.6+ for metadata discovery
- **Infrastructure**: Apache Airflow for workflow orchestration

## Essential Development Commands

### Prerequisites and Setup
```bash
make prerequisites              # Check system requirements
make install_dev_env           # Install all development dependencies
make yarn_install_cache        # Install UI dependencies
```

### Frontend Development
```bash
cd openmetadata-ui/src/main/resources/ui
yarn start                     # Start development server on localhost:3000
yarn test                      # Run Jest unit tests
yarn test path/to/test.spec.ts # Run a specific test file
yarn test:watch               # Run tests in watch mode
yarn playwright:run            # Run E2E tests
yarn lint                      # ESLint check
yarn lint:fix                  # ESLint with auto-fix
yarn build                     # Production build
```

### Backend Development
```bash
mvn clean package -DskipTests  # Build without tests
mvn clean package -DonlyBackend -pl !openmetadata-ui  # Backend only
mvn test                       # Run unit tests
mvn verify                     # Run integration tests
mvn spotless:apply             # Format Java code
```

### Python Ingestion Development
```bash
cd ingestion
make install_dev_env           # Install in development mode
make generate                  # Generate Pydantic models from JSON schemas
make unit_ingestion_dev_env    # Run unit tests
make py_format                 # Apply ruff lint-fix + format
make py_format_check           # Verify lint + format (matches CI; catches non-auto-fixable issues)
make static-checks             # Run type checking with basedpyright
```

### Full Local Environment
```bash
./docker/run_local_docker.sh -m ui -d mysql        # Complete local setup with UI
./docker/run_local_docker.sh -m no-ui -d postgresql # Backend only with PostgreSQL
./docker/run_local_docker.sh -s true               # Skip Maven build step
```

### Testing
```bash
make run_e2e_tests             # Full E2E test suite
make unit_ingestion            # Python unit tests with coverage
yarn test:coverage             # Frontend test coverage
```

## Code Generation and Schemas

OpenMetadata uses a schema-first approach with JSON Schema definitions driving code generation:

```bash
make generate                  # Generate all models from schemas
make py_antlr                  # Generate Python ANTLR parsers
make js_antlr                  # Generate JavaScript ANTLR parsers
yarn parse-schema              # Parse JSON schemas for frontend (connection and ingestion schemas)
```

### Schema Architecture
- **Source schemas** in `openmetadata-spec/` define the canonical data models
- **Connection schemas** are pre-processed at build time via `parseSchemas.js` to resolve all `$ref` references
- **Application schemas** in `openmetadata-ui/.../ApplicationSchemas/` are resolved at runtime using `schemaResolver.ts`
- JSON schemas with `$ref` references to external files require resolution before use in forms

## Key Directories

- `openmetadata-service/` - Core Java backend services and REST APIs
- `openmetadata-ui/src/main/resources/ui/` - React frontend application
- `ingestion/` - Python ingestion framework with connectors
- `openmetadata-spec/` - JSON Schema specifications for all entities
- `bootstrap/sql/` - Database schema migrations and sample data
- `conf/` - Configuration files for different environments
- `docker/` - Docker configurations for local and production deployment

## Development Workflow

1. **Schema Changes**: Modify JSON schemas in `openmetadata-spec/`, then run `mvn clean install` on openmetadata-spec to update models
2. **Backend**: Develop in Java using Dropwizard patterns, test with `mvn test`, format with `mvn spotless:apply`
3. **Frontend**: Use React/TypeScript with Ant Design components, test with Jest/Playwright
4. **Ingestion**: Python connectors follow plugin pattern, use `make install_dev_env` for development
5. **Full Testing**: Use `make run_e2e_tests` before major changes

## Frontend Architecture Patterns

### React Component Patterns
- **File Naming**: Components use `ComponentName.component.tsx`, interfaces use `ComponentName.interface.ts`
- **State Management**: Use `useState` with proper typing, avoid `any`
- **Side Effects**: Use `useEffect` with proper dependency arrays
- **Performance**: Use `useCallback` for event handlers, `useMemo` for expensive computations
- **Custom Hooks**: Prefix with `use`, place in `src/hooks/`, return typed objects
- **Internationalization**: Use `useTranslation` hook from react-i18next, access with `t('key')`
- **Component Structure**: Functional components only, no class components
- **Props**: Define interfaces for all component props, place in `.interface.ts` files
- **Loading States**: Use object state for multiple loading states: `useState<Record<string, boolean>>({})`
- **Error Handling**: Use `showErrorToast` and `showSuccessToast` utilities from ToastUtils
- **Navigation**: Use `useNavigate` from react-router-dom, not direct history manipulation
- **Data Fetching**: Async functions with try-catch blocks, update loading states appropriately

### State Management
- Use Zustand stores for global state (e.g., `useLimitStore`, `useWelcomeStore`)
- Keep component state local when possible with `useState`
- Use context providers for feature-specific shared state (e.g., `ApplicationsProvider`)

### Styling

- **MUI Migration**: The project is gradually migrating from Ant Design to Material-UI (MUI) v7.3.1
- **Preferred Approach**: Use MUI components v7.3.1 and styles wherever possible for new features
- **Theme and Styles**: MUI theme data and styles are defined in `openmetadata-ui-core-components`
- **Colors and Design Tokens**: Always reference theme colors and design tokens from the MUI theme, not hardcoded values
- **Legacy Components**: Ant Design components remain in existing code but should be replaced with MUI equivalents when refactoring
- Do not add unnecessary spacing between logs and code.
- In Java, avoid wildcards imports (e.g., use `import java.util.List;` instead of `import java.util.*;`)
- Custom styles in `.less` files with component-specific naming (legacy pattern)
- Follow BEM naming convention for custom CSS classes
- Use CSS modules where appropriate

### UI considerations

- Do not use string literals at any place. You should use useTranslation hook and use it like const {t} = useTranslation(). And for example if you want to have "Run" as string, you should be using { t('label.run') }, this label is defined in locales.


### Application Configuration
- Applications use `ApplicationsClassBase` for schema loading and configuration
- Dynamic imports handle application-specific schemas and assets
- Form schemas use React JSON Schema Form (RJSF) with custom UI widgets

### Service Utilities
- Each service type has dedicated utility files (e.g., `DatabaseServiceUtils.tsx`)
- Connection schemas are imported statically and pre-resolved
- Service configurations use switch statements to map types to schemas

### Type Safety
- All API responses have generated TypeScript interfaces in `generated/`
- Custom types extend base interfaces when needed
- Avoid type assertions unless absolutely necessary
- Use discriminated unions for action types and state variants

## Database and Migrations

- Flyway handles schema migrations in `bootstrap/sql/migrations/`
- Use Docker containers for local database setup
- Default MySQL, PostgreSQL supported as alternative
- Sample data loaded automatically in development environment

## Security and Authentication

- JWT-based authentication with OAuth2/SAML support
- Role-based access control defined in Java entities
- Security configurations in `conf/openmetadata.yaml`
- Never commit secrets - use environment variables or secure vaults

## Code Generation Standards

### Comments Policy
- **Do NOT add unnecessary comments** - write self-documenting code
- **NEVER add single-line comments that describe what the code obviously does**
- Only include comments for:
    - Complex business logic that isn't obvious
    - Non-obvious algorithms or workarounds
    - Public API JavaDoc documentation
    - TODO/FIXME with ticket references
- Bad examples (NEVER do this):
    - `// Create user` before `createUser()`
    - `// Get client` before `SdkClients.adminClient()`
    - `// Verify domain is set` before `assertNotNull(entity.getDomain())`
    - `// User names are lowercased` when the code `toLowerCase()` makes it obvious
- If the code needs a comment to be understood, refactor the code to be clearer instead

### Java Code Requirements
- **Always run `mvn spotless:apply`** before finishing any task that touched
  `.java` files. CI runs `mvn spotless:check` and will fail the PR otherwise
  (bot's exact phrasing: "Please run `mvn spotless:apply` in the root of your
  repository and commit the changes to this PR"). Scope with `-pl <module>`
  for speed if only one module changed. A reusable procedure is written up at
  `.agents/skills/java-checkstyle/SKILL.md`.
- Use clear, descriptive variable and method names instead of comments
- Follow existing project patterns and conventions
- Generate production-ready code, not tutorial code
- Create integration tests in openmetadata-integration-tests
- Do not use Fully Qualified Names in the code such as org.openmetadata.schema.type.Status instead import the class name
- Do not import wild-card packages instead import exactly required packages

### TypeScript/Frontend Code Requirements
- **Always run the UI checkstyle sequence** before finishing any task that
  touched `.ts`/`.tsx`/`.js`/`.jsx`/`.json` under
  `openmetadata-ui/src/main/resources/ui/src/`, `.../playwright/`, or
  `openmetadata-ui-core-components/src/main/resources/ui/src/`. CI's
  `UI Checkstyle / lint-src|lint-playwright|lint-core-components` jobs fail
  the PR otherwise. Order matters: `organize-imports-cli` → `eslint --fix` →
  `prettier --write`. A reusable procedure lives at
  `.agents/skills/ui-checkstyle/SKILL.md`.
- **NEVER use `any` type** in TypeScript code - always use proper types
- Use `unknown` when the type is truly unknown and add type guards
- Import types from existing type definitions (e.g., `RJSFSchema` from `@rjsf/utils`)
- Follow ESLint rules strictly - the project enforces no-console, proper formatting
- Add `// eslint-disable-next-line` comments only when absolutely necessary
- **Import Organization** (in order):
  1. External libraries (React, Ant Design, etc.)
  2. Internal absolute imports from `generated/`, `constants/`, `hooks/`, etc.
  3. Relative imports for utilities and components
  4. Asset imports (SVGs, styles)
  5. Type imports grouped separately when needed

### Python Code Requirements
- **Use pytest, not unittest** - write tests using pytest style with plain `assert` statements
- Use pytest fixtures for test setup instead of `setUp`/`tearDown` methods
- Use `unittest.mock` for mocking (MagicMock, patch) - this is compatible with pytest
- Test classes should not inherit from `TestCase` - use plain classes prefixed with `Test`
- Use `assert x == y` instead of `self.assertEqual(x, y)`
- Use `assert x is None` instead of `self.assertIsNone(x)`
- Use `assert "text" in string` instead of `self.assertIn("text", string)`

### Python Ingestion Connector Guidelines
- **Keep connector-specific logic in connector-specific files**, not in generic/shared files like `builders.py`
- Example: Redshift IAM auth should be in `ingestion/src/metadata/ingestion/source/database/redshift/connection.py`, not in `ingestion/src/metadata/ingestion/connections/builders.py`
- This keeps the codebase modular and prevents generic utilities from becoming cluttered with connector-specific edge cases

### Testing Philosophy
- **Test real behavior, not mock wiring** - if a test requires mocking 3+ classes just to verify a method call, it's testing the wrong thing
- **Prefer integration tests** over heavily-mocked unit tests. This project has full integration test infrastructure (OpenMetadataApplicationTest, Docker containers, real OpenSearch). Use it.
- **Mocks are for boundaries, not internals** - mock external services (HTTP clients, third-party APIs), not your own classes. If you're mocking static methods left and right to test internal plumbing, write an integration test instead.
- **A test that mocks everything proves nothing** - it only verifies that your mocks are wired correctly, not that the system works
- **Ask "what breaks if this test passes but the code is wrong?"** - if the answer is "nothing, because everything real is mocked out", delete the test and write a better one
- **Test the outcome, not the implementation** - assert on observable results (API responses, database state, stats values) rather than verifying internal method calls with `verify()`

### Response Format
- Provide clean code blocks without unnecessary explanations
- Assume readers are experienced developers
- Focus on functionality over education