open-metadata--openmetadata/docs/perf/cdn-deployment-guide.md

Shipping Collate / OpenMetadata releases through CloudFront

Each customer gets a Collate deployment at their own host — acme.getcolate.io, widgets.getcolate.io, globex.getcolate.io — and each customer can be on a different release. This is the AWS-only design for serving the UI bundle from CloudFront in that model, and the coordination story when a request lands at one of those hosts.

What we want

• One CloudFront distribution for every customer (not one per customer).
• One S3 bucket for every release. Releases are immutable; promotion is a separate step from upload.
• Per-customer version pinning that updates atomically — no DNS change, no CloudFront redeploy.
• Customer's own ALB continues to serve /api/*; CloudFront only handles the UI bundle.

What we explicitly do NOT want

• A new external data store to maintain (DynamoDB, an extra RDS, a separate Redis). The customer-version mapping is small (a few hundred entries, two tiny strings each) and changes rarely (a few writes per week, even at peak). Standing up a data store for that buys nothing and adds backup, monitoring, IAM, and cost surface.
• Per-customer CloudFront distributions. They give clean isolation but at N customers we have N distributions to manage, N caches that share no edge state across customers, and hit the AWS 200-distributions-per-account cap by default. The savings from edge cache sharing (a thousand customers on v1.12.0 hit the same cached chunk) are the entire reason the shared model is worth using.
• A lookup that requires Lambda@Edge. The cold start and per-request cost is real ($1+/M, plus 30-60 ms when cold) and we don't need the SDK access Lambda@Edge gives.

The architecture

                                ┌──────────────────────────────────────┐
acme.getcolate.io     ───┐      │  CloudFront distribution             │
widgets.getcolate.io  ───┼─────►│  d1234abc.cloudfront.net              │
globex.getcolate.io   ───┘      │                                      │
                                │  ┌─ behavior: /* ──────────────────┐ │
                                │  │  origin: S3                      │ │     ┌─────────────────────────────┐
                                │  │  viewer-request: host_router.js  │─┼────►│  S3: collate-cdn            │
                                │  │   rewrites /foo →                │ │     │   release/v1.11.2/index.html │
                                │  │   /release/<version>/foo         │ │     │   release/v1.12.0/index.html │
                                │  └──────────────────────────────────┘ │     │   release/v1.13.0-beta/...   │
                                │                                      │     └─────────────────────────────┘
                                │  ┌─ behavior: /api/* ──────────────┐ │
                                │  │  bypass: same host's per-       │ │     ┌─────────────────────────────┐
                                │  │  customer ALB (Option A below)  │─┼────►│  Each customer's own ALB     │
                                │  └──────────────────────────────────┘ │     └─────────────────────────────┘
                                └──────────────────────────────────────┘

The CloudFront Function holds the customer→version routing table as JavaScript object literal. Source of truth is the Function's source code in our git repo. Promotion is a Function code update.

The Function (no external lookup)

// host_router.js — CloudFront Function v2.0 (no Lambda@Edge, no KVS, no DynamoDB)
//
// Source of truth for which release each customer is pinned to. Edit, commit, deploy.
// CI propagates a change to every edge POP in ~60 s.

const CUSTOMER_VERSIONS = {
  acme:    'v1.12.0',
  widgets: 'v1.11.2',
  globex:  'v1.13.0-beta',
  // … N customers
};

// Hosts that don't match a customer slug (apex, www., staging) fall back to the latest
// stable release. Bump this in lockstep with every GA release so new customers that
// haven't been added to CUSTOMER_VERSIONS yet still get a current build.
const DEFAULT_VERSION = 'v1.12.0';

function handler(event) {
    const request = event.request;

    // /api/* lives on a separate behavior with the customer's own ALB as origin.
    // The Function should never see these requests under the current behavior config,
    // but guard anyway.
    if (request.uri.startsWith('/api/')) {
        return request;
    }

    const host = (request.headers.host && request.headers.host.value) || '';
    // Convention: customer slug is the first label of the host.
    // acme.getcolate.io -> 'acme'
    const slug = host.split('.')[0];
    const version = CUSTOMER_VERSIONS[slug] || DEFAULT_VERSION;

    // /assets/foo.js -> /release/v1.12.0/assets/foo.js
    request.uri = '/release/' + version + request.uri;
    return request;
}

Function v2.0 has a 10 KB code limit. At ~30 bytes per entry that's ~300 customers comfortably; well beyond that the design needs revisiting — but if you ever reach 300+ customers on this product, the operational economics of standing up KVS or DynamoDB will have shifted significantly anyway.

Promotion flow

1. Edit CUSTOMER_VERSIONS in the Function source.
2. Commit, push, open PR. The PR diff IS the promotion record — reviewable, auditable, git-blame'd.
3. CI runs on merge: pushes the new Function code via aws cloudfront update-function and publish-function.
4. ~60 s of edge propagation. Every POP picks up the new code.

A typical promotion PR looks like one line changed:

 const CUSTOMER_VERSIONS = {
-  acme:    'v1.12.0',
+  acme:    'v1.12.1',
   widgets: 'v1.11.2',
   globex:  'v1.13.0-beta',
 };

That's the entire surface area of a promotion. No DynamoDB write. No KVS API call. No extra IAM role. No backup story. Just a code change reviewed like any other.

Rollback is symmetric: revert the commit. Canary is "promote one slug first, watch error metrics, then PR the next batch." Roll-forward on a regression is the same revert.

Release upload (independent of promotion)

The bundle bytes go to S3 separately, on every release tag, regardless of which customer ends up using them:

VERSION="v1.12.0"
aws s3 sync openmetadata-ui/src/main/resources/ui/dist/assets/ \
    s3://collate-cdn/release/${VERSION}/assets/ \
    --cache-control "public, max-age=31536000, immutable"

aws s3 cp openmetadata-ui/src/main/resources/ui/dist/index.html \
    s3://collate-cdn/release/${VERSION}/index.html \
    --cache-control "no-cache, must-revalidate" \
    --content-type "text/html; charset=utf-8"

After this, the release exists in S3 but no customer is using it. Promotion (the PR above) is what flips customers to it. The decoupling matters: you can sit on a release in S3 for a week, watching it on staging, before promoting any customer to it.

Why the Function code is a fine routing table

Honest comparison of the three approaches:

	Function-embedded (this design)	CloudFront KeyValueStore	DynamoDB + Lambda@Edge
New AWS service to monitor / back up	none	KVS	DynamoDB + Lambda
Read latency at edge	~0 (in-function)	~1 ms	~10 ms (warm Lambda)
Cold start	none	none	30-60 ms
Per-request cost	$0.10/M Function	$0.10/M Function + $0.04/M KVS	$0.10/M + $1+/M Lambda + DynamoDB reads
Promotion surface	git PR	API call (put-key)	API call (update-item)
Audit trail	git history	CloudWatch + KVS audit logs	CloudWatch + DDB streams
Capacity ceiling	~300 customers (10 KB code limit)	millions	millions
Concurrent promotion safety	git merge serializes	IfMatch ETag	conditional writes
Operational ownership	"this is in the repo"	"who paged on this last quarter?"	"who paged on this last quarter?"

For a product that ships per-customer clusters and reaches dozens-to-low-hundreds of customers, "the routing table is a file in the repo" wins on every operational axis that matters. It only loses on capacity ceiling, and the day that becomes a problem we already have a clear migration target (KVS) without changing anything else in the design.

API routing — two options, pick one

The Function above only handles UI bundle requests. /api/* still has to reach the customer's own ALB.

Option A — Separate API host (recommended)

acme.getcolate.io      → CNAME → CloudFront distribution (this design)
api-acme.getcolate.io  → CNAME → acme's ALB

SPA's API base URL is derived from the page host at runtime: https://api-{slug}.getcolate.io/api.

Pros: CloudFront does one thing well (static delivery). No Lambda@Edge anywhere. Failure modes are easy to reason about. Cons: SPA has a cookie/CORS story that knows about two hosts; we already handle this for various integrations.

Option B — Same host, Lambda@Edge for /api/*

CloudFront's /api/* behavior runs a Lambda@Edge on origin-request that reads the host header and rewrites the origin to the right ALB.

Pros: single host per customer. Cons: now we DO have Lambda@Edge (which we explicitly chose to avoid for routing), and the operational cost is per-customer-API-request, not just per-promotion. We strongly prefer Option A.

S3 bucket layout

collate-cdn/
└── release/
    ├── v1.11.5/
    │   ├── index.html                         no-cache, must-revalidate
    │   ├── assets/index-Z3O_FBkA.js           immutable
    │   ├── assets/index-Z3O_FBkA.js.br        immutable
    │   ├── assets/index-Z3O_FBkA.js.gz        immutable
    │   ├── assets/vendor-antd-BgrjOjhB.js     immutable
    │   └── ...
    ├── v1.12.0/        ← acme + widgets currently here
    │   └── ...
    └── v1.13.0-beta/   ← globex currently here (canary)
        └── ...

Releases are immutable once uploaded. The promotion step never modifies S3 contents — only the Function code that maps slug → /release/<v>/.

Disk cost is small: a typical OM bundle is ~12 MB on disk after content-hash dedup, Brotli+gzip siblings add ~25%, call it 15 MB per release. 100 releases × 15 MB = 1.5 GB. S3 standard rates put that at a few cents per month — keep many releases live for instant rollback and don't bother with aggressive lifecycle pruning.

CloudFront cache behaviors

Path pattern (after Function rewrite)	Edge TTL	Notes
/release/<v>/assets/*	1 year	Content-addressed; bytes can't change
/release/<v>/index.html, /release/<v>/	30 s	Concurrent users in one region share one origin hit; ETag layer takes over after 30 s
/api/*	bypass	Separate behavior to customer ALB (Option A: not via CloudFront at all)

30 s on the shell is the sweet spot: long enough to dedupe a thousand concurrent reloads to one origin fetch, short enough that a promotion lands at all customers within ~90 s end-to-end (60 s Function propagation + 30 s residual edge cache).

Per-customer branding (without per-customer bundles)

If a customer needs a different logo or accent colour, the right move is to keep one universal bundle and overlay branding assets at request time:

• Universal default: /release/v1.12.0/images/logo.png in S3.
• Per-customer override (optional, only when needed): the Function checks for s3://collate-cdn/customer-overrides/<slug>/logo.png first and rewrites if it exists.

Branding stays out of the build artifact, which means one bundle still serves every customer and the cache-sharing argument holds.

Verification after promotion

Two synthetic checks worth running automatically after a promotion PR merges:

SLUG=acme
EXPECTED_VERSION=v1.12.0

# 1. CloudFront serves the right release for this slug
RESPONSE=$(curl -s "https://${SLUG}.getcolate.io/?nocache=$(uuidgen)")
echo "$RESPONSE" | grep -oE 'index-[A-Za-z0-9_-]+\.js' | sort -u
# Should match the hash from the v1.12.0 build manifest

# 2. The HTML shell is being served fresh from the right S3 prefix
curl -sI "https://${SLUG}.getcolate.io/" \
  | grep -i 'x-amz-cf-pop\|via\|x-cache'
# Should show an edge POP near the test runner, and either "Miss from cloudfront"
# (first request after promotion) or "Hit from cloudfront" (within the 30 s edge TTL)

CI runs this on every promotion PR after the Function deploys, and fails loud if the served bundle doesn't match the version we just pinned.

What's not in this design

• Per-customer API origin selection inside CloudFront. Option A keeps /api/* off the CloudFront path entirely. If a customer ever needs single-host behavior, that's the moment to revisit Option B and accept Lambda@Edge.
• Multi-region S3 origin failover. Single bucket in one region; CloudFront's edge caching handles regional reach. If you want CRR + origin groups, add them; the cost is straightforward but rarely justified for a UI bundle.
• WAF / Shield Advanced. Add separately if your security posture requires them.

What this design is good for and what would push it elsewhere

• Good for: dozens to low-hundreds of customers, infrequent promotion (a few per week), engineering ownership over the routing table.
• Push toward KVS when: customer count grows past a few hundred (function size pressure) OR promotions happen via a non-engineering UI (a customer-success dashboard that flips slugs without a git PR).
• Push toward Lambda@Edge when: routing decisions stop being a slug→version map and start needing per-request information not available in the host header (e.g. A/B testing by user ID, geo-routing, header-derived feature flags).

When those days come, the migration path from this design is small — the Function code becomes a kvs.get(slug) instead of a hash lookup, and the rest of the architecture (S3 layout, distribution behaviors, ALB routing) is identical.