wehub-resource-sync
b7f52be4c9
CI / Run CI (push) Has been cancelled
CI / check-backend (push) Has been cancelled
CI / check-frontend (push) Has been cancelled
CI / tests (push) Has been cancelled
CI / e2e-tests (push) Has been cancelled
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
310 lines
20 KiB
Markdown
310 lines
20 KiB
Markdown
# CI Cache Optimization Research
|
||
|
||
> Last CI run analyzed: [#5989](https://github.com/Chainlit/chainlit/actions/runs/24017813922) on `feat/parallel-cypress` (2026-04-06, success, 13m 36s wall clock)
|
||
|
||
## Timing Findings
|
||
|
||
### Critical Path
|
||
|
||
The pipeline bottleneck is **Windows E2E shards** (13m 12s worst case). 60% of that time is setup, not tests.
|
||
|
||
```
|
||
e2e-tests / windows-latest-2 (13m 12s) ← CRITICAL PATH
|
||
|
||
┌─ Install Node/pnpm ──── 1m 57s ─┐
|
||
│ Install Cypress ─────── 1m 33s │
|
||
│ Install Python/uv ──── 2m 50s │ 7m 49s setup (60%)
|
||
│ Build UI ────────────── 1m 29s │
|
||
└──────────────────────────────────┘
|
||
┌─ Run tests ───────────── 4m 19s ─┐ 4m 19s actual work
|
||
└──────────────────────────────────┘
|
||
```
|
||
|
||
### All Jobs (wall clock)
|
||
|
||
| Job | Total | pnpm install | Cypress | uv sync | Build UI | Tests |
|
||
| ---------------------- | ---------- | ------------ | ------- | ---------- | -------- | -------- |
|
||
| lint-backend | 1m 21s | — | — | 11s | — | mypy 58s |
|
||
| lint-ui | 1m 52s | 30s | — | — | 1m 07s | lint 10s |
|
||
| pytest (3.10) | 3m 09s | 33s | — | 58s | 1m 07s | 24s |
|
||
| pytest (3.11) | 3m 04s | 33s | — | 56s | 1m 06s | 23s |
|
||
| pytest (3.12) | 3m 03s | 30s | — | 53s | 1m 07s | 29s |
|
||
| **pytest (3.13)** | **5m 45s** | 30s | — | **3m 40s** | 1m 06s | 25s |
|
||
| e2e ubuntu (avg of 5) | ~5m 50s | ~31s | ~11s | ~55s | ~1m 07s | ~3m |
|
||
| e2e windows (avg of 5) | ~11m 56s | ~2m 02s | ~1m 30s | ~2m 28s | ~1m 22s | ~3m 50s |
|
||
|
||
### Anomalies
|
||
|
||
- **pytest (3.13) uv sync: 3m 40s** — vs ~55s for 3.10–3.12. Verified from logs: uv cache save **failed** during this run due to a transient GitHub cache service outage (`"Our services aren't available right now"`). The 3.13 slowness is likely source-building wheels that lack pre-built binaries for Python 3.13. This is a one-time cost that gets cached naturally once the service recovers — not a config issue.
|
||
- **Windows pnpm install: ~2m** — vs ~30s on Ubuntu. Disk I/O and process spawning overhead.
|
||
- **uv cache save failed across all jobs** in this run — transient outage, not a config problem.
|
||
|
||
### Redundant Work
|
||
|
||
`pnpm run buildUi` (builds `libs/react-client` → `libs/copilot` → `frontend`) runs independently in **7 jobs**:
|
||
|
||
- 2 E2E jobs (ubuntu + windows) × ~1m 10s = ~2m 20s
|
||
- 4 pytest jobs × ~1m 07s = ~4m 28s
|
||
- 1 lint-ui job × ~1m 07s = ~1m 07s
|
||
- **Total: ~8 min of cumulative runner time** rebuilding identical, OS-independent output
|
||
|
||
Note: On `feat/refactor-scripts`, the Hatch build hook is already bypassed — `uv-python-install` uses `--no-install-project --no-editable`, and `pnpm run buildUi` is an explicit step. The `lint-backend` workflow does **not** need the build.
|
||
|
||
---
|
||
|
||
## Current Cache Configuration
|
||
|
||
| Tool | Action | Cache mechanism | Status |
|
||
| ------------------- | --------------------------------------------------------------------------------- | ------------------------------------------------------------- | ------------------------------------------------- |
|
||
| pnpm store | `actions/setup-node@v6.3.0` | `cache: 'pnpm'`, `cache-dependency-path: '**/pnpm-lock.yaml'` | ✅ Working (30s installs on Ubuntu) |
|
||
| uv download cache | `astral-sh/setup-uv@v8.0.0` | `enable-cache: true` | ✅ Working (55s on Ubuntu), ⚠️ cache miss on 3.13 |
|
||
| Cypress binary | `cypress-io/github-action@v6` (run used v6 despite local file referencing v7.1.9) | Internal (built-in to the action) | ✅ Ubuntu (11s), ⚠️ Windows (90s) |
|
||
| Python installation | `actions/setup-python@v6.2.0` | Built-in (pre-installed on runners) | ✅ Fast on Ubuntu |
|
||
| UI build output | — | **Not cached** | ❌ Rebuilt 14× per run |
|
||
|
||
---
|
||
|
||
## Research: Recommended Approaches per Tool
|
||
|
||
### 1. pnpm — `actions/setup-node` and `pnpm/action-setup`
|
||
|
||
**Source**: [actions/setup-node README](https://github.com/actions/setup-node), [pnpm/action-setup README](https://github.com/pnpm/action-setup)
|
||
|
||
**What's recommended**:
|
||
|
||
- `actions/setup-node` with `cache: 'pnpm'` — caches pnpm's content-addressable store. This is **already in use** and is the recommended approach.
|
||
- `pnpm/action-setup` also offers `cache: true` but using both is redundant.
|
||
- Docs explicitly recommend committing `pnpm-lock.yaml` and using `--frozen-lockfile` in CI.
|
||
|
||
**What's NOT recommended**:
|
||
|
||
- Caching `node_modules/` directly. pnpm uses hard-linked content-addressable storage; caching `node_modules` breaks this model and can cause integrity issues.
|
||
|
||
**Assessment**: Current config is correct. No change needed for pnpm caching itself.
|
||
|
||
### 2. Cypress binary — `cypress-io/github-action`
|
||
|
||
**Source**: [cypress-io/github-action README — "Install Cypress only"](https://github.com/cypress-io/github-action#install-cypress-only)
|
||
|
||
**Recommended pattern** (from the README, section "Install Cypress only"):
|
||
|
||
```yaml
|
||
- uses: actions/cache@v5
|
||
with:
|
||
path: |
|
||
~/.cache/Cypress
|
||
node_modules
|
||
key: my-cache-${{ runner.os }}-${{ hashFiles('package-lock.json') }}
|
||
- run: npm i cypress
|
||
- uses: cypress-io/github-action@v7
|
||
with:
|
||
install: false
|
||
```
|
||
|
||
**Key observations**:
|
||
|
||
1. The README shows explicit `actions/cache` for `~/.cache/Cypress` as the recommended approach when splitting install from test execution.
|
||
2. The `cypress-io/github-action` has a `cache-key` input for custom cache keys.
|
||
3. For pnpm projects, the README shows: install pnpm → setup-node with `cache: 'pnpm'` → let the Cypress action handle installation.
|
||
|
||
**Verified from CI logs (Ubuntu shard 1)**:
|
||
|
||
1. `pnpm install` (03:46:41–03:47:00) runs Cypress's `postinstall` hook, which **fully installs the binary** in ~9s:
|
||
```
|
||
cypress@14.5.3 postinstall: Installing Cypress (version: 14.5.3)
|
||
cypress@14.5.3 postinstall: Done
|
||
```
|
||
2. `cypress-io/github-action@v6` with `runTests: false` (03:47:00–03:47:11, ~11s) then runs again. Its only log output: `"Skipping running tests: runTests parameter is false"`. It re-runs the install/verify (redundant) and saves the Cypress binary cache in its post-step.
|
||
|
||
**Conclusion**: The action is **redundant for installation**. Its only value is the **post-step cache save** of `~/.cache/Cypress`. This can be replaced with a plain `actions/cache` step that:
|
||
|
||
1. Restores `~/.cache/Cypress` BEFORE `pnpm install` (so postinstall skips the download)
|
||
2. Saves it after the job completes
|
||
|
||
This follows the "Install Cypress only" pattern from the action's own README.
|
||
|
||
**Paths for Windows**: The `~/.cache/Cypress` path is Linux-only. Windows uses `~/AppData/Local/Cypress/Cache`. Both must be listed in the `actions/cache` path, OR use the `CYPRESS_CACHE_FOLDER` env var to standardize.
|
||
|
||
### 3. uv — `astral-sh/setup-uv`
|
||
|
||
**Source**: [setup-uv README](https://github.com/astral-sh/setup-uv), [uv GitHub Actions guide](https://github.com/astral-sh/uv/blob/main/docs/guides/integration/github.md), [uv caching concepts](https://github.com/astral-sh/uv/blob/main/docs/concepts/cache.md)
|
||
|
||
**What's recommended**:
|
||
|
||
- `enable-cache: true` — caches uv's download cache (wheels, sdists). **Already in use.**
|
||
- `cache-python: true` — caches Python installations managed by uv. **Not currently used.** This would help when uv installs Python (vs `actions/setup-python`).
|
||
- `cache-dependency-glob` — controls cache key. Defaults to `**/uv.lock` etc. Current config uses defaults.
|
||
- Manual approach: `actions/cache` with `UV_CACHE_DIR` + `uv cache prune --ci` for more control.
|
||
|
||
**What's NOT recommended**:
|
||
|
||
- Caching `.venv/` directly. The uv docs don't mention or recommend this. uv is designed to be fast at resolving+linking from its cache.
|
||
- Caching pre-built wheels. From the uv docs: _"it's often faster to omit pre-built wheels from the cache (and instead re-download them from the registry on each run)."_ Only source-built wheels benefit from caching.
|
||
|
||
**Verified from CI logs**: The 3.13 outlier coincided with a **transient GitHub cache service outage** (`"Failed to save: Our services aren't available right now"` in the post-step). All pytest jobs in this run failed to save their uv cache. The 3m40s `uv sync` is likely source-building wheels that don't have pre-built binaries for Python 3.13 yet, combined with the cache miss.
|
||
|
||
**Assessment**: `cache-python: true` would NOT help here — it caches the Python installation binary, not package wheels. The uv download cache (`enable-cache: true`, already in use) is the correct mechanism; it just failed to save in this run due to the outage. No config change needed — this resolves on the next successful cache save.
|
||
|
||
**Important from uv docs**: _"it's often faster to omit pre-built wheels from the cache (and instead re-download them from the registry on each run)."_ Only source-built wheels benefit from caching. The current `enable-cache: true` already handles this correctly.
|
||
|
||
### 4. UI build output — no standard tool
|
||
|
||
**Source**: No specific recommendation from any action. This is a custom optimization.
|
||
|
||
**Standard GitHub Actions patterns**:
|
||
|
||
1. **`actions/cache`** — Cache build output keyed on source hash. Pros: simple. Cons: 10GB cache limit shared across all branches; large build outputs can evict other caches.
|
||
2. **`actions/upload-artifact` + `actions/download-artifact`** — Build once in a dedicated job, share artifacts with downstream jobs. Pros: no cache eviction issues, guaranteed fresh builds. Cons: adds a sequential dependency (build job must complete before shards start), artifact upload/download overhead.
|
||
|
||
**Assessment**: A dedicated build job with artifact sharing is the cleaner pattern for this case (14 consumers, deterministic output). But it changes the dependency graph — E2E and pytest jobs would need to wait for the build job.
|
||
|
||
---
|
||
|
||
## Proposed Changes
|
||
|
||
### P1: Replace `cypress-io/github-action` with `actions/cache` for Cypress binary
|
||
|
||
**Status**: Verified from logs — the action is redundant for installation.
|
||
|
||
**Why `install: false` + `runTests: false` doesn't help**: The action's cache restore happens in its **main step**, which runs AFTER `pnpm-node-install`. By then, `pnpm install` has already triggered Cypress's `postinstall` hook and downloaded the binary. The cache restore is too late. With both flags off, the action is effectively a no-op during the main step — it only saves the cache in its post-step for future runs.
|
||
|
||
The README's [`install: false` pattern](https://github.com/cypress-io/github-action#install-cypress-only) assumes the action is still used to **run tests**. We don't — we use `pnpm test:e2e` with `cypress-split`.
|
||
|
||
**Approach**: Place `actions/cache` **before** `pnpm install` so the Cypress binary is restored before the postinstall hook runs:
|
||
|
||
1. Add `actions/cache` for Cypress binary dir BEFORE `pnpm-node-install`
|
||
2. Remove the `cypress-io/github-action` step entirely
|
||
3. Cypress's `postinstall` hook skips download when it finds the binary pre-cached
|
||
|
||
```yaml
|
||
# In e2e-tests.yaml, BEFORE pnpm-node-install:
|
||
- name: Cache Cypress binary
|
||
uses: actions/cache@v4
|
||
with:
|
||
path: |
|
||
~/.cache/Cypress
|
||
~/AppData/Local/Cypress/Cache
|
||
key: cypress-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml') }}
|
||
# Then remove the cypress-io/github-action step entirely.
|
||
```
|
||
|
||
**Cypress version**: `package.json` specifies `^14.5.3`, `pnpm-lock.yaml` resolves to `14.5.4`. Cypress is installed by `pnpm install` (postinstall hook) in both the current and proposed flow — no version change.
|
||
|
||
**Expected critical-path saving**: ~1m 30s per Windows shard (eliminates the redundant action step AND speeds up postinstall via cache hit on warm cache).
|
||
|
||
### P2: Build UI once at CI startup, share via cache
|
||
|
||
**Status**: Primary optimization — eliminates ~1m build per downstream job (×7 jobs).
|
||
|
||
**Problem**: `pnpm run buildUi` runs independently in 7 jobs, producing identical OS-independent output each time. Without a dedicated build job, all parallel jobs miss the cache simultaneously on the first run and all rebuild.
|
||
|
||
**Build command**: `pnpm run buildUi` = `pnpm build:libs && cd frontend && pnpm run build`
|
||
|
||
**Build output directories**:
|
||
|
||
- `libs/react-client/dist/`
|
||
- `libs/copilot/dist/`
|
||
- `frontend/dist/`
|
||
|
||
**Approach**: Add a `build-ui` job in `ci.yaml` that builds once and saves to `actions/cache`. Downstream workflows restore from cache and skip the build.
|
||
|
||
**Cache key**: `ui-build-${{ github.sha }}`
|
||
|
||
Using the commit SHA is the simplest correct key. Enumerating source files in `hashFiles()` would be fragile — the build depends on `package.json`, `pnpm-lock.yaml`, `vite.config.*`, `tsconfig.*`, `index.html`, plus all source in `frontend/src/`, `libs/react-client/src/`, `libs/copilot/src/`. Too many inputs to track reliably. The trade-off: `github.sha` rebuilds on every new commit even if only backend files changed. But on cache hit the `build-ui` job is ~15s (restore only), so the wasted work is minimal. `github.run_id` would NOT work — it's unique per run, so the cache would never hit across runs (e.g. retries).
|
||
|
||
```yaml
|
||
# ci.yaml
|
||
jobs:
|
||
build-ui:
|
||
runs-on: ubuntu-latest
|
||
steps:
|
||
- uses: actions/checkout@v4
|
||
- uses: ./.github/actions/pnpm-node-install
|
||
- uses: actions/cache@v4
|
||
id: ui-cache
|
||
with:
|
||
path: |
|
||
frontend/dist
|
||
libs/react-client/dist
|
||
libs/copilot/dist
|
||
key: ui-build-${{ github.sha }}
|
||
- name: Build UI
|
||
if: steps.ui-cache.outputs.cache-hit != 'true'
|
||
run: pnpm run buildUi
|
||
|
||
pytest:
|
||
needs: build-ui # wait for build
|
||
uses: ./.github/workflows/pytest.yaml
|
||
e2e-tests:
|
||
needs: build-ui # wait for build
|
||
uses: ./.github/workflows/e2e-tests.yaml
|
||
lint-ui:
|
||
needs: build-ui # wait for build
|
||
uses: ./.github/workflows/lint-ui.yaml
|
||
lint-backend: # no build needed — runs in parallel with build-ui
|
||
uses: ./.github/workflows/lint-backend.yaml
|
||
```
|
||
|
||
Downstream workflows restore from cache and skip `pnpm run buildUi`:
|
||
|
||
```yaml
|
||
# In pytest.yaml, e2e-tests.yaml, lint-ui.yaml — replace the buildUi step:
|
||
- uses: actions/cache@v4
|
||
with:
|
||
path: |
|
||
frontend/dist
|
||
libs/react-client/dist
|
||
libs/copilot/dist
|
||
key: ui-build-${{ github.sha }}
|
||
fail-on-cache-miss: true # build-ui already ran, cache must exist
|
||
```
|
||
|
||
**Why cache over artifacts?**:
|
||
|
||
- **Artifacts**: free for public repos (no storage cap), but re-upload every run even when nothing changed. Each run pays upload+download overhead.
|
||
- **Cache**: 10 GB per repo (shared with pnpm/uv/Cypress caches), but on repeat runs with the same SHA (retries, re-triggered workflows) the `build-ui` job itself becomes ~15s (cache hit, skip build). Build output is ~50-100 MB, well within limits alongside other caches.
|
||
|
||
**Why `build-ui` is still needed**: Without it, all parallel downstream jobs would miss the cache simultaneously on the first run and all build independently. `needs: build-ui` guarantees the cache is populated before they start.
|
||
|
||
**Trade-off**: `build-ui` adds a sequential dependency. On cache miss: ~1m 30s (pnpm install ~30s + build ~1m). On cache hit: ~15s. `lint-backend` runs in parallel and takes ~1m 20s, so `build-ui` doesn't extend the critical path.
|
||
|
||
**Expected saving**: ~1m per downstream job × 7 jobs = **~7 min cumulative runner time**. Critical path saving: ~1m per Windows E2E job (build eliminated, ~5s cache restore).
|
||
|
||
### P3: Merge `validate` + `prepare` e2e jobs
|
||
|
||
**Approach**: Combine the two trivial `ubuntu-slim` jobs into one to eliminate a job startup cycle.
|
||
|
||
**Expected saving**: ~10-15s.
|
||
|
||
### ~~P4: Add `cache-python: true` to setup-uv~~ (dropped)
|
||
|
||
**Reason**: Verified from logs that the Python 3.13 outlier was caused by a transient GitHub cache service outage, not a config issue. `cache-python: true` caches the Python installation binary, which is irrelevant — the slowness came from source-building wheels. The existing `enable-cache: true` already caches built wheels correctly. No change needed.
|
||
|
||
---
|
||
|
||
## Estimated Impact on Critical Path
|
||
|
||
| Change | Current (win E2E) | After | Saving |
|
||
| --------------------------------- | ------------------------------------ | ------------------- | ----------- |
|
||
| P1: Cypress cache + remove action | 1m 33s (action) + ~30s (postinstall) | ~5s | ~2m |
|
||
| P2: Build UI once + cache | ~1m 22s (build per job) | ~5s (cache restore) | ~1m 17s |
|
||
| P3: merge validate+prepare | ~15s overhead | 0s | ~15s |
|
||
| **Cumulative (critical path)** | **13m 12s** | **~9m 40s** | **~3m 32s** |
|
||
|
||
Notes:
|
||
|
||
- P1 and P2 are the most reliable wins. P2 eliminates redundant builds across all 7 consuming jobs (E2E, pytest, lint-ui) with a single approach.
|
||
- P2's `build-ui` job adds a sequential dependency (~1m 30s on Ubuntu) but `lint-backend` runs in parallel and takes ~1m 20s, so `build-ui` doesn't extend the critical path.
|
||
- Cumulative runner time saved (not just critical path): ~9m across all 7 jobs that currently build UI.
|
||
|
||
---
|
||
|
||
## Discarded Ideas (with reasons)
|
||
|
||
| Idea | Why discarded | Source |
|
||
| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
|
||
| Cache `node_modules/` directly | pnpm uses hard-linked content-addressable storage; caching `node_modules` breaks this model | [pnpm docs](https://pnpm.io/faq#why-is-it-recommended-to-not-use-npm-ci-for-pnpm), [actions/setup-node](https://github.com/actions/setup-node) |
|
||
| Cache `.venv/` directly | uv docs don't recommend this; uv is designed to resolve+link fast from its cache | [uv caching docs](https://github.com/astral-sh/uv/blob/main/docs/concepts/cache.md) |
|
||
| `cache-python: true` in setup-uv | Wouldn't help — 3.13 outlier was source-building wheels (not Python install), and was caused by a transient cache service outage | Verified from CI logs |
|
||
| Cache pre-built wheels | uv docs explicitly say _"it's often faster to omit pre-built wheels from the cache"_ — only source-built wheels benefit | [uv caching docs](https://github.com/astral-sh/uv/blob/main/docs/concepts/cache.md) |
|