chainlit--chainlit/docs/research/ci-cache-optimization.md

CI Cache Optimization Research

Last CI run analyzed: #5989 on feat/parallel-cypress (2026-04-06, success, 13m 36s wall clock)

Timing Findings

Critical Path

The pipeline bottleneck is Windows E2E shards (13m 12s worst case). 60% of that time is setup, not tests.

e2e-tests / windows-latest-2 (13m 12s) ← CRITICAL PATH

  ┌─ Install Node/pnpm ──── 1m 57s ─┐
  │  Install Cypress ─────── 1m 33s  │
  │  Install Python/uv ──── 2m 50s  │  7m 49s setup (60%)
  │  Build UI ────────────── 1m 29s  │
  └──────────────────────────────────┘
  ┌─ Run tests ───────────── 4m 19s ─┐  4m 19s actual work
  └──────────────────────────────────┘

All Jobs (wall clock)

Job	Total	pnpm install	Cypress	uv sync	Build UI	Tests
lint-backend	1m 21s	—	—	11s	—	mypy 58s
lint-ui	1m 52s	30s	—	—	1m 07s	lint 10s
pytest (3.10)	3m 09s	33s	—	58s	1m 07s	24s
pytest (3.11)	3m 04s	33s	—	56s	1m 06s	23s
pytest (3.12)	3m 03s	30s	—	53s	1m 07s	29s
pytest (3.13)	5m 45s	30s	—	3m 40s	1m 06s	25s
e2e ubuntu (avg of 5)	~5m 50s	~31s	~11s	~55s	~1m 07s	~3m
e2e windows (avg of 5)	~11m 56s	~2m 02s	~1m 30s	~2m 28s	~1m 22s	~3m 50s

Anomalies

• pytest (3.13) uv sync: 3m 40s — vs ~55s for 3.10–3.12. Verified from logs: uv cache save failed during this run due to a transient GitHub cache service outage ("Our services aren't available right now"). The 3.13 slowness is likely source-building wheels that lack pre-built binaries for Python 3.13. This is a one-time cost that gets cached naturally once the service recovers — not a config issue.
• Windows pnpm install: ~2m — vs ~30s on Ubuntu. Disk I/O and process spawning overhead.
• uv cache save failed across all jobs in this run — transient outage, not a config problem.

Redundant Work

pnpm run buildUi (builds libs/react-client → libs/copilot → frontend) runs independently in 7 jobs:

• 2 E2E jobs (ubuntu + windows) × ~1m 10s = ~2m 20s
• 4 pytest jobs × ~1m 07s = ~4m 28s
• 1 lint-ui job × ~1m 07s = ~1m 07s
• Total: ~8 min of cumulative runner time rebuilding identical, OS-independent output

Note: On feat/refactor-scripts, the Hatch build hook is already bypassed — uv-python-install uses --no-install-project --no-editable, and pnpm run buildUi is an explicit step. The lint-backend workflow does not need the build.

---

Current Cache Configuration

Tool	Action	Cache mechanism	Status
pnpm store	actions/setup-node@v6.3.0	cache: 'pnpm', cache-dependency-path: '**/pnpm-lock.yaml'	✅ Working (30s installs on Ubuntu)
uv download cache	astral-sh/setup-uv@v8.0.0	enable-cache: true	✅ Working (55s on Ubuntu), ⚠️ cache miss on 3.13
Cypress binary	cypress-io/github-action@v6 (run used v6 despite local file referencing v7.1.9)	Internal (built-in to the action)	✅ Ubuntu (11s), ⚠️ Windows (90s)
Python installation	actions/setup-python@v6.2.0	Built-in (pre-installed on runners)	✅ Fast on Ubuntu
UI build output	—	Not cached	❌ Rebuilt 14× per run

---

Research: Recommended Approaches per Tool

1. pnpm — actions/setup-node and pnpm/action-setup

Source: actions/setup-node README, pnpm/action-setup README

What's recommended:

• actions/setup-node with cache: 'pnpm' — caches pnpm's content-addressable store. This is already in use and is the recommended approach.
• pnpm/action-setup also offers cache: true but using both is redundant.
• Docs explicitly recommend committing pnpm-lock.yaml and using --frozen-lockfile in CI.

What's NOT recommended:

• Caching node_modules/ directly. pnpm uses hard-linked content-addressable storage; caching node_modules breaks this model and can cause integrity issues.

Assessment: Current config is correct. No change needed for pnpm caching itself.

2. Cypress binary — cypress-io/github-action

Source: cypress-io/github-action README — "Install Cypress only"

Recommended pattern (from the README, section "Install Cypress only"):

- uses: actions/cache@v5
  with:
    path: |
      ~/.cache/Cypress
      node_modules
    key: my-cache-${{ runner.os }}-${{ hashFiles('package-lock.json') }}
- run: npm i cypress
- uses: cypress-io/github-action@v7
  with:
    install: false

Key observations:

1. The README shows explicit actions/cache for ~/.cache/Cypress as the recommended approach when splitting install from test execution.
2. The cypress-io/github-action has a cache-key input for custom cache keys.
3. For pnpm projects, the README shows: install pnpm → setup-node with cache: 'pnpm' → let the Cypress action handle installation.

Verified from CI logs (Ubuntu shard 1):

1. pnpm install (03:46:41–03:47:00) runs Cypress's postinstall hook, which fully installs the binary in ~9s:

   cypress@14.5.3 postinstall: Installing Cypress (version: 14.5.3)
   cypress@14.5.3 postinstall: Done
   

2. cypress-io/github-action@v6 with runTests: false (03:47:00–03:47:11, ~11s) then runs again. Its only log output: "Skipping running tests: runTests parameter is false". It re-runs the install/verify (redundant) and saves the Cypress binary cache in its post-step.

Conclusion: The action is redundant for installation. Its only value is the post-step cache save of ~/.cache/Cypress. This can be replaced with a plain actions/cache step that:

1. Restores ~/.cache/Cypress BEFORE pnpm install (so postinstall skips the download)
2. Saves it after the job completes

This follows the "Install Cypress only" pattern from the action's own README.

Paths for Windows: The ~/.cache/Cypress path is Linux-only. Windows uses ~/AppData/Local/Cypress/Cache. Both must be listed in the actions/cache path, OR use the CYPRESS_CACHE_FOLDER env var to standardize.

3. uv — astral-sh/setup-uv

Source: setup-uv README, uv GitHub Actions guide, uv caching concepts

What's recommended:

• enable-cache: true — caches uv's download cache (wheels, sdists). Already in use.
• cache-python: true — caches Python installations managed by uv. Not currently used. This would help when uv installs Python (vs actions/setup-python).
• cache-dependency-glob — controls cache key. Defaults to **/uv.lock etc. Current config uses defaults.
• Manual approach: actions/cache with UV_CACHE_DIR + uv cache prune --ci for more control.

What's NOT recommended:

• Caching .venv/ directly. The uv docs don't mention or recommend this. uv is designed to be fast at resolving+linking from its cache.
• Caching pre-built wheels. From the uv docs: "it's often faster to omit pre-built wheels from the cache (and instead re-download them from the registry on each run)." Only source-built wheels benefit from caching.

Verified from CI logs: The 3.13 outlier coincided with a transient GitHub cache service outage ("Failed to save: Our services aren't available right now" in the post-step). All pytest jobs in this run failed to save their uv cache. The 3m40s uv sync is likely source-building wheels that don't have pre-built binaries for Python 3.13 yet, combined with the cache miss.

Assessment: cache-python: true would NOT help here — it caches the Python installation binary, not package wheels. The uv download cache (enable-cache: true, already in use) is the correct mechanism; it just failed to save in this run due to the outage. No config change needed — this resolves on the next successful cache save.

Important from uv docs: "it's often faster to omit pre-built wheels from the cache (and instead re-download them from the registry on each run)." Only source-built wheels benefit from caching. The current enable-cache: true already handles this correctly.

4. UI build output — no standard tool

Source: No specific recommendation from any action. This is a custom optimization.

Standard GitHub Actions patterns:

1. actions/cache — Cache build output keyed on source hash. Pros: simple. Cons: 10GB cache limit shared across all branches; large build outputs can evict other caches.
2. actions/upload-artifact + actions/download-artifact — Build once in a dedicated job, share artifacts with downstream jobs. Pros: no cache eviction issues, guaranteed fresh builds. Cons: adds a sequential dependency (build job must complete before shards start), artifact upload/download overhead.

Assessment: A dedicated build job with artifact sharing is the cleaner pattern for this case (14 consumers, deterministic output). But it changes the dependency graph — E2E and pytest jobs would need to wait for the build job.

---

Proposed Changes

P1: Replace cypress-io/github-action with actions/cache for Cypress binary

Status: Verified from logs — the action is redundant for installation.

Why install: false + runTests: false doesn't help: The action's cache restore happens in its main step, which runs AFTER pnpm-node-install. By then, pnpm install has already triggered Cypress's postinstall hook and downloaded the binary. The cache restore is too late. With both flags off, the action is effectively a no-op during the main step — it only saves the cache in its post-step for future runs.

The README's install: false pattern assumes the action is still used to run tests. We don't — we use pnpm test:e2e with cypress-split.

Approach: Place actions/cache before pnpm install so the Cypress binary is restored before the postinstall hook runs:

1. Add actions/cache for Cypress binary dir BEFORE pnpm-node-install
2. Remove the cypress-io/github-action step entirely
3. Cypress's postinstall hook skips download when it finds the binary pre-cached

# In e2e-tests.yaml, BEFORE pnpm-node-install:
- name: Cache Cypress binary
  uses: actions/cache@v4
  with:
    path: |
      ~/.cache/Cypress
      ~/AppData/Local/Cypress/Cache
    key: cypress-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml') }}
# Then remove the cypress-io/github-action step entirely.

Cypress version: package.json specifies ^14.5.3, pnpm-lock.yaml resolves to 14.5.4. Cypress is installed by pnpm install (postinstall hook) in both the current and proposed flow — no version change.

Expected critical-path saving: ~1m 30s per Windows shard (eliminates the redundant action step AND speeds up postinstall via cache hit on warm cache).

P2: Build UI once at CI startup, share via cache

Status: Primary optimization — eliminates ~1m build per downstream job (×7 jobs).

Problem: pnpm run buildUi runs independently in 7 jobs, producing identical OS-independent output each time. Without a dedicated build job, all parallel jobs miss the cache simultaneously on the first run and all rebuild.

Build command: pnpm run buildUi = pnpm build:libs && cd frontend && pnpm run build

Build output directories:

• libs/react-client/dist/
• libs/copilot/dist/
• frontend/dist/

Approach: Add a build-ui job in ci.yaml that builds once and saves to actions/cache. Downstream workflows restore from cache and skip the build.

Cache key: ui-build-${{ github.sha }}

Using the commit SHA is the simplest correct key. Enumerating source files in hashFiles() would be fragile — the build depends on package.json, pnpm-lock.yaml, vite.config.*, tsconfig.*, index.html, plus all source in frontend/src/, libs/react-client/src/, libs/copilot/src/. Too many inputs to track reliably. The trade-off: github.sha rebuilds on every new commit even if only backend files changed. But on cache hit the build-ui job is ~15s (restore only), so the wasted work is minimal. github.run_id would NOT work — it's unique per run, so the cache would never hit across runs (e.g. retries).

# ci.yaml
jobs:
  build-ui:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/pnpm-node-install
      - uses: actions/cache@v4
        id: ui-cache
        with:
          path: |
            frontend/dist
            libs/react-client/dist
            libs/copilot/dist
          key: ui-build-${{ github.sha }}
      - name: Build UI
        if: steps.ui-cache.outputs.cache-hit != 'true'
        run: pnpm run buildUi

  pytest:
    needs: build-ui # wait for build
    uses: ./.github/workflows/pytest.yaml
  e2e-tests:
    needs: build-ui # wait for build
    uses: ./.github/workflows/e2e-tests.yaml
  lint-ui:
    needs: build-ui # wait for build
    uses: ./.github/workflows/lint-ui.yaml
  lint-backend: # no build needed — runs in parallel with build-ui
    uses: ./.github/workflows/lint-backend.yaml

Downstream workflows restore from cache and skip pnpm run buildUi:

# In pytest.yaml, e2e-tests.yaml, lint-ui.yaml — replace the buildUi step:
- uses: actions/cache@v4
  with:
    path: |
      frontend/dist
      libs/react-client/dist
      libs/copilot/dist
    key: ui-build-${{ github.sha }}
    fail-on-cache-miss: true # build-ui already ran, cache must exist

Why cache over artifacts?:

• Artifacts: free for public repos (no storage cap), but re-upload every run even when nothing changed. Each run pays upload+download overhead.
• Cache: 10 GB per repo (shared with pnpm/uv/Cypress caches), but on repeat runs with the same SHA (retries, re-triggered workflows) the build-ui job itself becomes ~15s (cache hit, skip build). Build output is ~50-100 MB, well within limits alongside other caches.

Why build-ui is still needed: Without it, all parallel downstream jobs would miss the cache simultaneously on the first run and all build independently. needs: build-ui guarantees the cache is populated before they start.

Trade-off: build-ui adds a sequential dependency. On cache miss: ~1m 30s (pnpm install ~30s + build ~1m). On cache hit: ~15s. lint-backend runs in parallel and takes ~1m 20s, so build-ui doesn't extend the critical path.

Expected saving: ~1m per downstream job × 7 jobs = ~7 min cumulative runner time. Critical path saving: ~1m per Windows E2E job (build eliminated, ~5s cache restore).

P3: Merge validate + prepare e2e jobs

Approach: Combine the two trivial ubuntu-slim jobs into one to eliminate a job startup cycle.

Expected saving: ~10-15s.

P4: Add cache-python: true to setup-uv (dropped)

Reason: Verified from logs that the Python 3.13 outlier was caused by a transient GitHub cache service outage, not a config issue. cache-python: true caches the Python installation binary, which is irrelevant — the slowness came from source-building wheels. The existing enable-cache: true already caches built wheels correctly. No change needed.

---

Estimated Impact on Critical Path

Change	Current (win E2E)	After	Saving
P1: Cypress cache + remove action	1m 33s (action) + ~30s (postinstall)	~5s	~2m
P2: Build UI once + cache	~1m 22s (build per job)	~5s (cache restore)	~1m 17s
P3: merge validate+prepare	~15s overhead	0s	~15s
Cumulative (critical path)	13m 12s	~9m 40s	~3m 32s

Notes:

• P1 and P2 are the most reliable wins. P2 eliminates redundant builds across all 7 consuming jobs (E2E, pytest, lint-ui) with a single approach.
• P2's build-ui job adds a sequential dependency (~1m 30s on Ubuntu) but lint-backend runs in parallel and takes ~1m 20s, so build-ui doesn't extend the critical path.
• Cumulative runner time saved (not just critical path): ~9m across all 7 jobs that currently build UI.

---

Discarded Ideas (with reasons)

Idea	Why discarded	Source
Cache node_modules/ directly	pnpm uses hard-linked content-addressable storage; caching node_modules breaks this model	pnpm docs, actions/setup-node
Cache .venv/ directly	uv docs don't recommend this; uv is designed to resolve+link fast from its cache	uv caching docs
cache-python: true in setup-uv	Wouldn't help — 3.13 outlier was source-building wheels (not Python install), and was caused by a transient cache service outage	Verified from CI logs
Cache pre-built wheels	uv docs explicitly say "it's often faster to omit pre-built wheels from the cache" — only source-built wheels benefit	uv caching docs