open-metadata--openmetadata/docs/session-management-multi-node-design.md

# Multi-Node Session and WebSocket Session Management Design

## 1. Status

This document describes the current server-side session and websocket session design for
OpenMetadata issue [#21971](https://github.com/open-metadata/OpenMetadata/issues/21971).

The implementation is centered on these files:

- `openmetadata-service/src/main/java/org/openmetadata/service/security/session/SessionService.java`
- `openmetadata-service/src/main/java/org/openmetadata/service/security/session/SessionStore.java`
- `openmetadata-service/src/main/java/org/openmetadata/service/security/session/JdbcSessionStore.java`
- `openmetadata-service/src/main/java/org/openmetadata/service/security/session/RedisSessionStore.java`
- `openmetadata-service/src/main/java/org/openmetadata/service/security/session/SessionStoreFactory.java`
- `openmetadata-service/src/main/java/org/openmetadata/service/security/JwtFilter.java`
- `openmetadata-service/src/main/java/org/openmetadata/service/security/jwt/JWTTokenGenerator.java`
- `openmetadata-service/src/main/java/org/openmetadata/service/socket/SocketAddressFilter.java`
- `openmetadata-service/src/main/java/org/openmetadata/service/socket/WebSocketManager.java`
- `openmetadata-service/src/main/java/org/openmetadata/service/OpenMetadataApplication.java`
- `openmetadata-service/src/main/java/org/openmetadata/service/cache/CacheBundle.java`

## 2. Problem

Server-managed login state used to depend on pod-local servlet state. That breaks in a multi-node
deployment because login, callback, refresh, logout, and websocket reconnects can land on different
pods.

The key failure modes are:

- A login or OIDC/SAML callback starts on one node and completes on another.
- Refresh state is unavailable when the request is routed to a different node.
- Logout or session revocation is not visible to all nodes.
- A websocket can remain connected after the browser session is revoked.
- A secure websocket handshake can be spoofed if the server trusts a client-supplied `userId`.

## 3. Goals

1. Store server-managed user sessions in a shared, authoritative backend.
2. Support both JDBC-backed sessions and Redis-backed sessions.
3. Bind newly issued browser access JWTs to the server-side session that issued them.
4. Keep provider refresh tokens and OpenMetadata refresh tokens server-side.
5. Make refresh safe under cross-node concurrency.
6. Make logout and revocation visible to API and websocket paths.
7. Reject websocket handshakes whose token principal, cookie session, or requested socket user do not match.
8. Close websocket connections for revoked or expired sessions.

## 4. Non-Goals

1. Changing personal access token or bot token semantics.
2. Replacing the browser-managed public OIDC flow.
3. Making legacy JWTs without a `sessionId` claim retroactively session-bound.
4. Guaranteeing instant cross-node websocket disconnect without pub/sub. Without pub/sub, remote
   sockets are closed by periodic validation.

## 5. Architecture

### 5.1 Component Model

| Component | Responsibility |
| --- | --- |
| `SessionService` | Creates, activates, refreshes, revokes, expires, and prunes sessions. Owns the Caffeine near-cache and revocation listeners. |
| `SessionStore` | Shared persistence contract used by both JDBC and Redis stores. |
| `JdbcSessionStore` | Default store backed by the `user_session` table through `SessionRepository`. |
| `RedisSessionStore` | Optional Redis store with key TTLs, per-user status indexes, and Lua compare-and-set on `version` plus index maintenance. |
| `SessionStoreFactory` | Selects Redis when Redis cache is configured and available; otherwise uses JDBC. Refuses Redis-to-JDBC fallback when Redis is configured but unavailable. |
| `SessionCookieUtil` | Reads, writes, validates, and clears the opaque `OM_SESSION` cookie. |
| `JWTTokenGenerator` | Issues OpenMetadata JWTs and can include the `sessionId` claim for session-backed auth flows. |
| `JwtFilter` | Validates JWTs. If the JWT has `sessionId`, it reloads the session from the store and requires an active, unexpired, username-matching session. |
| `SocketAddressFilter` | Validates websocket handshake identity and session state before Socket.IO sees the connection. |
| `WebSocketManager` | Tracks sockets by user and by session, sends events, and disconnects revoked or inactive session sockets. |
| `OpenMetadataApplication` | Wires `SessionService` into auth handlers, websockets, revocation listeners, and the websocket session validator. |
| `CacheBundle` | Handles cache invalidation pub/sub. Session invalidation messages also disconnect sockets on remote pods. |

### 5.2 Storage Selection

`SessionStoreFactory` chooses the store at application startup:

- If `cache.provider = redis` and Redis is available, sessions use `RedisSessionStore`.
- If Redis is configured but unavailable, startup fails closed.
- If Redis is not configured, sessions use `JdbcSessionStore`.

The system does not fail over live from Redis to JDBC. Mixing stores would split active sessions
across backends and make revocation unpredictable.

### 5.3 Session Cache

Each pod keeps a local Caffeine near-cache:

- maximum size: `10_000`
- expire after access: `10s`

The cache is a performance optimization, not a correctness boundary. Security-sensitive checks use
fresh reloads where revocation must be observed immediately.

## 6. User Session Management

### 6.1 Session ID

`UserSession.id` is an opaque bearer secret carried in the `OM_SESSION` cookie.

It is generated by `SessionIdGenerator` from secure random bytes and base64url encoded without
padding. It is not a UUID.

### 6.2 Session Types

Current session types:

- `AUTH`: browser or interactive user auth session.
- `MCP`: reserved for future interactive MCP session support.

### 6.3 Session Status

`SessionStatus` values:

- `PENDING`: login started, callback not completed.
- `ACTIVE`: usable session.
- `REFRESHING`: one node holds the refresh lease.
- `REVOKED`: logout or session-limit revocation.
- `EXPIRED`: timeout reached.

### 6.4 Session Fields

The important logical fields are:

```json
{
  "id": "opaque-session-id",
  "type": "AUTH",
  "provider": "openmetadata",
  "status": "ACTIVE",
  "userId": "uuid",
  "username": "alice",
  "email": "alice@example.com",
  "omRefreshToken": "fernet:encrypted-token",
  "providerRefreshToken": "fernet:encrypted-provider-token",
  "redirectUri": "https://ui.example.com/callback",
  "state": "oidc-state",
  "nonce": "oidc-nonce",
  "pkceVerifier": "pkce-verifier",
  "version": 7,
  "refreshLeaseUntil": 1741300000000,
  "createdAt": 1741200000000,
  "updatedAt": 1741200005000,
  "lastAccessedAt": 1741200005000,
  "expiresAt": 1743792000000,
  "idleExpiresAt": 1741804800000
}
```

Refresh tokens are encrypted before persistence with `Fernet.encryptIfApplies(...)`. If the Fernet
key is not configured, session creation fails instead of writing plaintext refresh tokens.

### 6.5 Session Creation

Basic, LDAP, and OpenMetadata login create an `ACTIVE` session directly:

1. Validate credentials.
2. Resolve the provisioned OpenMetadata user.
3. Persist or receive the OpenMetadata refresh token.
4. Create an `ACTIVE` `AUTH` session.
5. Encrypt and store the refresh token in the session.
6. Write the `OM_SESSION` cookie.
7. Return an OpenMetadata-signed JWT with a `sessionId` claim.

If user lookup or session creation fails after a refresh token is created, the refresh token is
deleted.

### 6.6 Pending Session Activation

SAML and confidential OIDC use pending sessions:

1. Login creates a `PENDING` `AUTH` session containing redirect state, OIDC state, nonce, and PKCE
   verifier when applicable.
2. The callback loads the pending session from the shared store.
3. The user is created or updated.
4. The OpenMetadata refresh token is inserted.
5. `activatePendingSession` expires the pending session.
6. A brand-new active session ID is generated and stored.
7. The active session cookie replaces the pending cookie.
8. The browser receives an OpenMetadata-signed JWT with the active session ID.

Issuing a new active session ID during activation is the session fixation defense. The pre-auth
cookie value is never reused for the authenticated session.

If activation fails, the newly inserted refresh token is deleted and no JWT is issued.

### 6.7 Refresh

Refresh is guarded by an optimistic lease:

1. Load the session from `OM_SESSION`.
2. Reject missing, expired, pending, revoked, or already expired sessions.
3. If another node holds a non-stale `REFRESHING` lease, return retry guidance through
   `SessionRefreshInProgressException`.
4. Acquire the lease by writing `REFRESHING`, setting `refreshLeaseUntil`, and incrementing
   `version` with compare-and-set.
5. The winning node decrypts the stored refresh token.
6. The provider or OpenMetadata refresh token is rotated as needed.
7. `completeRefresh` writes the refreshed session back to `ACTIVE`, clears the lease, updates idle
   expiry without extending beyond the absolute session expiry, and increments `version`.
8. The response contains a new OpenMetadata-signed JWT bound to the same session ID.

Lease duration is currently `15s`.

### 6.8 Logout and Revocation

Logout calls `SessionService.revokeSession(request, response)`:

1. Read `OM_SESSION`.
2. Reload the session from the authoritative store.
3. Write `REVOKED` with compare-and-set.
4. Clear `refreshLeaseUntil`.
5. Clear the `OM_SESSION` cookie.
6. Notify local revocation listeners.

Session limit enforcement also uses `revokeSession` for least-recently-used active sessions.

The limit is configured by `authenticationConfiguration.maxActiveSessionsPerUser`, exposed through
`AUTHENTICATION_MAX_ACTIVE_SESSIONS_PER_USER` in `openmetadata.yaml`. The default is `5`; values
below `1` fall back to the default.

### 6.9 Expiration and Cleanup

`SessionService` runs cleanup every `15m`:

- mark expired sessions as `EXPIRED`
- prune `REVOKED` and `EXPIRED` rows after `7d`
- process in bounded batches

For Redis, primary keys have TTLs and cleanup methods are no-ops. Session correctness still relies
on in-process status and expiry checks.

Default timeouts:

- pending session timeout: `10m`
- authenticated session expiry: `authenticationConfiguration.sessionExpiry`, default `7d`
- refresh lease: `15s`
- cleanup retention: `7d`

The `OM_SESSION` cookie max age is rewritten during refresh lease acquisition and is capped at the
remaining effective session lifetime.

## 7. Session-Bound JWTs

Server-managed auth flows return OpenMetadata-signed access JWTs with:

```json
{
  "sub": "alice",
  "tokenType": "OM_USER",
  "sessionId": "opaque-session-id"
}
```

`JwtFilter` handles the claim as follows:

1. Validate the JWT signature, expiry, token type, principal, and token-specific rules.
2. If there is no `sessionId` claim, preserve existing stateless behavior.
3. If `sessionId` exists, call `SessionService.getFreshSessionById(sessionId)`.
4. Require:
   - session exists
   - status is `ACTIVE`
   - session is not expired
   - session username matches the JWT principal
5. Reject the token when any check fails.

This means session-backed browser API requests now consult the shared session store. That is an
intentional tradeoff in the current implementation: revocation is observed on the next request
instead of waiting for access-token expiry. PATs, bot tokens, and legacy JWTs without `sessionId`
remain stateless.

## 8. WebSocket Session Management

### 8.1 Handshake Validation

`SocketAddressFilter` runs before the Socket.IO server receives the connection.

When secure websocket connections are enabled:

1. Extract and validate the `Authorization` header.
2. Resolve the token principal from JWT claims.
3. Resolve the principal's user UUID server-side.
4. Reject the request if the query `userId` is present and does not match the resolved user UUID.
5. Inject the server-resolved `UserId` header for `WebSocketManager`.
6. If the JWT has `sessionId`, inject a `SessionId` header.
7. Validate `OM_SESSION` when present:
   - reload the session fresh
   - require `ACTIVE`
   - require not expired
   - require session username to match the token principal
   - require cookie session ID to match token `sessionId` when both are present

If no `OM_SESSION` cookie is present:

- session-bound JWTs are accepted because `JwtFilter` has already validated the session ID
  against `SessionService`
- legacy secure JWTs without `sessionId` are rejected with `401 Session is required`
- non-secure websocket mode remains compatible with existing query-based behavior

The filter no longer forwards trust from the user-supplied `userId` query parameter when secure
mode is enabled.

### 8.2 Socket Tracking

`WebSocketManager` maintains two local maps per pod:

- `activityFeedEndpoints`: `userId -> socketId -> SocketIoSocket`
- `socketSessionIds`: `socketId -> sessionId`

On connection:

1. Read `UserId` from the injected header, falling back to query only for legacy/non-secure paths.
2. Read `SessionId` from the injected header, falling back to query `sessionId` only for legacy
   paths.
3. Store the socket in the user's local socket map.
4. Store the socket-to-session mapping when a session ID is available.

On disconnect, both maps are cleaned up.

The connection log records user and remote address only. It does not log initial headers, so bearer
tokens are not written to logs.

### 8.3 Revocation-Driven Disconnect

`SessionService` exposes revocation listeners. `OpenMetadataApplication` registers a listener that:

1. Converts the revoked session's `userId` to UUID.
2. Calls `WebSocketManager.disconnectForSession(userId, sessionId)` on the local pod.
3. Publishes a `"session"` invalidation message through cache invalidation pub/sub when available.

`CacheBundle` handles remote `"session"` invalidation messages:

- if the message has a session ID, call `disconnectForSession(userId, sessionId)`
- if no session ID is present, fall back to `disconnectAllForUser(userId)` for backward
  compatibility

This gives targeted disconnects. Logging out one browser session does not force-close other
sessions for the same user.

### 8.4 Periodic WebSocket Validation

`OpenMetadataApplication.WebSocketSessionValidator` runs every `60s` by default. Operators can tune
the interval with the `openmetadata.websocketSessionValidationIntervalSeconds` system property or
the `WEBSOCKET_SESSION_VALIDATION_INTERVAL_SECONDS` environment variable. Values below `15s` are
clamped to `15s`.

Each run calls `WebSocketManager.disconnectInactiveSessions(sessionService, intervalMillis)`, which:

1. Iterates local sockets with known `sessionId`.
2. Reloads a socket's session fresh through `SessionService.getFreshSessionById` only when that
   socket's revalidation interval is due.
3. Disconnects sockets whose session is missing, not `ACTIVE`, expired, or owned by a different
   user.

This is the fallback when there is no cross-pod pub/sub. With JDBC and no pub/sub, a socket on a
remote node is closed within the validator interval instead of immediately.

## 9. End-to-End Flow

```mermaid
flowchart TD
  Browser["Browser / UI"]
  NodeA["OpenMetadata Pod A"]
  NodeB["OpenMetadata Pod B"]
  AuthHandlers["Auth handlers<br/>Basic / LDAP / SAML / OIDC"]
  SessionServiceA["SessionService A"]
  SessionServiceB["SessionService B"]
  Store[("Shared SessionStore<br/>JDBC user_session or Redis")]
  JwtFilter["JwtFilter"]
  SocketFilter["SocketAddressFilter"]
  WebSocketManagerA["WebSocketManager A"]
  WebSocketManagerB["WebSocketManager B"]
  PubSub["Cache invalidation pub/sub<br/>optional"]
  Validator["WebSocketSessionValidator<br/>default 60s, min 15s"]

  Browser -->|"login / callback"| NodeA
  NodeA --> AuthHandlers
  AuthHandlers --> SessionServiceA
  SessionServiceA -->|"create PENDING or ACTIVE<br/>activate pending<br/>encrypt refresh tokens"| Store
  SessionServiceA -->|"Set-Cookie: OM_SESSION"| Browser
  AuthHandlers -->|"JWT with sessionId"| Browser

  Browser -->|"API request<br/>Bearer JWT(sessionId)"| NodeB
  NodeB --> JwtFilter
  JwtFilter --> SessionServiceB
  SessionServiceB -->|"fresh session lookup"| Store
  JwtFilter -->|"allow only ACTIVE + unexpired + matching username"| NodeB

  Browser -->|"websocket handshake<br/>Authorization + OM_SESSION"| NodeB
  NodeB --> SocketFilter
  SocketFilter -->|"validate JWT principal<br/>resolve user UUID<br/>validate session"| SessionServiceB
  SocketFilter -->|"inject UserId + SessionId"| WebSocketManagerB
  WebSocketManagerB -->|"track user -> sockets<br/>track socket -> session"| WebSocketManagerB

  Browser -->|"logout / revoke"| NodeA
  NodeA --> SessionServiceA
  SessionServiceA -->|"mark REVOKED"| Store
  SessionServiceA -->|"local disconnectForSession"| WebSocketManagerA
  SessionServiceA -->|"publish session invalidation"| PubSub
  PubSub -->|"remote disconnectForSession"| WebSocketManagerB

  Validator --> WebSocketManagerB
  WebSocketManagerB -->|"fresh validate socket sessions"| SessionServiceB
  SessionServiceB --> Store
  WebSocketManagerB -->|"disconnect inactive sockets"| Browser
```

## 10. Consistency Model

### 10.1 API Requests

For tokens with `sessionId`, the session store is authoritative. A revoked or expired session is
rejected on the next API request that uses that token.

For tokens without `sessionId`, existing stateless behavior is preserved.

### 10.2 Refresh

Refresh uses optimistic compare-and-set on `version`, so only one node can hold the refresh lease
for a session at a time.

JDBC implements this through the session repository update path. Redis implements it with a Lua CAS
script over the stored session JSON. The Redis script also removes the session ID from all
non-terminal per-user status indexes and adds it to the target non-terminal index before returning,
so the JSON write and index movement succeed or fail together.

### 10.3 WebSockets

Websocket consistency has two layers:

- event-driven disconnect through local revocation listeners and optional cache invalidation pub/sub
- polling-based validation with a `60s` default interval and `15s` minimum

The event path is immediate when revocation occurs on the same pod or pub/sub delivers the remote
event. The polling path bounds staleness when pub/sub is unavailable, and each socket is fresh
loaded at most once per validation interval.

## 11. Operational Characteristics

| Path | Store behavior |
| --- | --- |
| Login | create active or pending session |
| OIDC/SAML callback | fresh load pending session, expire pending session, create active session |
| Session-bound API request | fresh load session by `sessionId` |
| Refresh | load session, acquire CAS lease, complete CAS update |
| Logout | fresh load session, CAS revoke, clear cookie |
| WebSocket handshake | validate JWT, optionally fresh load cookie session |
| WebSocket validator | throttled fresh load for each tracked socket with `sessionId` |

Redis deployments should monitor Redis availability as auth-critical infrastructure. When Redis is
configured for sessions, the service refuses to start without it.

## 12. Security Properties

1. `OM_SESSION` is opaque and high entropy.
2. `OM_SESSION` is written as an HTTP-only cookie.
3. Provider refresh tokens and OpenMetadata refresh tokens are encrypted at rest.
4. Refresh tokens are not returned to the browser by server-managed auth flows.
5. Pending-session activation issues a brand-new active session ID.
6. Session-bound JWTs are invalid once the backing session is revoked, expired, deleted, or owned by
   a different user.
7. Secure websocket mode derives socket user identity from the JWT principal, not from query params.
8. Websocket logs do not include initial headers or bearer tokens.
9. Revocation targets the revoked session instead of disconnecting every socket for the user.

## 13. Test Coverage

Relevant unit coverage includes:

- `SessionServiceTest`
- `SessionCookieUtilTest`
- `SessionTimeoutResolverTest`
- `SessionStoreContractTest`
- `RedisSessionStoreTest`
- `JwtFilterTest`
- `BasicAuthServletHandlerTest`
- `LdapAuthServletHandlerTest`
- `SamlAuthServletHandlerTest`
- `AuthenticationCodeFlowHandlerTest`
- `SocketAddressFilterTest`
- `WebSocketManagerTest`

Relevant integration coverage includes:

- `SessionMultiNodeIT`
- `SessionRedisMultiNodeIT`
- `SessionMultiNodeCluster`

Important scenarios covered or expected from this suite:

- login on one node and refresh/logout on another
- pending OIDC/SAML callback state loaded from shared session storage
- refresh lease contention
- stale cache behavior after revocation
- Redis-backed cross-node sessions
- websocket principal binding
- per-session websocket disconnect
- session-bound JWT rejection for revoked sessions

## 14. Tradeoff Resolutions

1. Session-bound browser API requests intentionally reload session state on JWT validation. This is
   the chosen correctness boundary: logout and revocation are observed on the next browser API
   request instead of waiting for access-token expiry.
2. Tokens without `sessionId` remain on existing JWT semantics. This preserves PAT, bot, public OIDC,
   and rolling-upgrade compatibility. New server-managed auth responses include `sessionId`.
3. Non-secure websocket mode remains query-param based only for backward compatibility. Production
   deployments should keep secure websocket connections enabled so `SocketAddressFilter` derives the
   socket user from the JWT principal and records the session ID.
4. The active-session cap is now configurable with
   `authenticationConfiguration.maxActiveSessionsPerUser` and
   `AUTHENTICATION_MAX_ACTIVE_SESSIONS_PER_USER`; the default remains `5`.
5. Secure, session-managed websocket handshakes now record a session ID from the JWT claim or
   `OM_SESSION` cookie. The validator checks those sockets on a configurable interval with a `60s`
   default and `15s` minimum; sockets without session IDs are legacy/non-secure compatibility cases.
6. Cross-pod websocket revocation has two paths: cache invalidation pub/sub for immediate targeted
   disconnects when available, and the configurable websocket validator as the bounded-staleness
   fallback for JDBC-only deployments.