wehub-resource-sync
b7f52be4c9
CI / Run CI (push) Has been cancelled
CI / check-backend (push) Has been cancelled
CI / check-frontend (push) Has been cancelled
CI / tests (push) Has been cancelled
CI / e2e-tests (push) Has been cancelled
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
111 lines
5.7 KiB
Markdown
111 lines
5.7 KiB
Markdown
# E2E Test Parallel Execution Research
|
|
|
|
## Current Architecture: Strictly Sequential
|
|
|
|
The e2e tests **cannot run in parallel** with the current setup. There are four structural reasons.
|
|
|
|
### 1. Single Hardcoded Port
|
|
|
|
Every test backend spawns on port `8000` (`backend/chainlit/config.py`):
|
|
|
|
```python
|
|
DEFAULT_PORT = 8000
|
|
```
|
|
|
|
`runChainlit` in `cypress/support/run.ts` invokes `uv run chainlit run <entryPoint> -h --ci` without passing `--port`, so every backend binds to the default.
|
|
|
|
### 2. Kill-Restart Lifecycle Per Spec File
|
|
|
|
`cypress.config.ts` orchestrates a strict **kill → start → test → kill** cycle:
|
|
|
|
```ts
|
|
on('before:spec', async (spec) => {
|
|
await killChainlit();
|
|
await runChainlit(spec);
|
|
});
|
|
|
|
on('after:spec', async () => {
|
|
await killChainlit();
|
|
});
|
|
```
|
|
|
|
`killChainlit()` uses `fkill(:8000)` — a port-based kill. If two test runners were active, one would kill the other's backend.
|
|
|
|
### 3. Each Spec Has Its Own Chainlit App
|
|
|
|
Each test directory (`cypress/e2e/<test_name>/`) contains:
|
|
|
|
- `main.py` — the Chainlit application for that test
|
|
- `.chainlit/config.toml` — per-test configuration
|
|
|
|
`runChainlit` uses `spec.absolute` to find the test directory and spawns Chainlit with `CHAINLIT_APP_ROOT` set to that directory. Since each test has a completely different backend app, the server must be restarted between specs.
|
|
|
|
### 4. Cypress Runs Specs Sequentially
|
|
|
|
Standard `cypress run` processes spec files one at a time in a single browser. Cypress Cloud offers parallelization across multiple CI machines, but the project doesn't use it (no `--parallel` or `--record` flags in `pnpm test:e2e`).
|
|
|
|
### Special Case: Mid-Test Restarts
|
|
|
|
The `data_layer` spec calls `cy.task('restartChainlit', Cypress.spec)` to kill and re-launch the backend within a single test to verify thread persistence across server restarts.
|
|
|
|
---
|
|
|
|
## Cypress Parallelization Capabilities
|
|
|
|
### Built-in `--parallel` Flag Requires Cypress Cloud
|
|
|
|
Cypress does **not** support parallelization out of the box for free.
|
|
|
|
```bash
|
|
cypress run --record --parallel
|
|
```
|
|
|
|
- `--parallel` **must** be used with `--record`, which sends results to Cypress Cloud (paid).
|
|
- Cypress Cloud acts as the orchestrator: it dynamically assigns spec files to available CI machines using a load-balancing strategy based on historical run durations.
|
|
- There is no local multi-process parallelism — it is **multi-machine** parallelism coordinated by the cloud service.
|
|
|
|
### Cypress vs Playwright
|
|
|
|
| Feature | Cypress | Playwright |
|
|
| ----------------- | ------------------------- | ------------------------ |
|
|
| Local parallelism | Not supported | Built-in (`--workers=4`) |
|
|
| CI sharding | Via Cypress Cloud (paid) | Built-in (`--shard=1/4`) |
|
|
| Orchestration | Dynamic, based on history | Manual split |
|
|
|
|
### Free Alternatives for Cypress Parallelization
|
|
|
|
| Tool | Description |
|
|
| -------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
|
|
| **Manual CI matrix sharding** | Split spec files across GitHub Actions matrix jobs using `--spec` |
|
|
| **[sorry-cypress](https://sorry-cypress.dev/)** | Open-source, self-hosted drop-in replacement for Cypress Cloud; supports `--parallel` |
|
|
| **[cypress-split](https://github.com/bahmutov/cypress-split)** | Plugin that splits specs across CI machines using `SPLIT`/`SPLIT_INDEX` env vars |
|
|
| **[currents.dev](https://currents.dev/)** | Cypress Cloud alternative with a free tier |
|
|
|
|
---
|
|
|
|
## What Would It Take to Parallelize
|
|
|
|
### Bottlenecks and Solutions
|
|
|
|
| Bottleneck | Solution |
|
|
| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| Hardcoded port `8000` | Assign a unique port per worker (e.g. `8000 + workerIndex`). Pass `--port` to `chainlit run` and configure `baseUrl` dynamically. |
|
|
| Port-based kill (`fkill(:8000)`) | Switch to PID-based process management — store the child PID from `spawn()` and kill it directly. |
|
|
| Single Cypress browser | Use Cypress Cloud `--parallel`, sorry-cypress, cypress-split, or shard tests across CI matrix jobs. |
|
|
| Shared filesystem (`.chainlit/` dirs) | Already isolated per test. Some tests (e.g. `data_layer`) write temp files like `thread_history.pickle` that must remain isolated per worker. |
|
|
|
|
### Strategy A: CI Matrix Sharding (Simplest)
|
|
|
|
Split the ~50 spec files into N groups across separate GitHub Actions runners:
|
|
|
|
1. Add a matrix dimension to the workflow that assigns each group to a separate runner.
|
|
2. Each runner uses port `8000` (since they are on separate machines).
|
|
3. No changes to Cypress config or `run.ts` — pass `--spec` with the subset.
|
|
|
|
### Strategy B: Same-Machine Parallelism (More Work)
|
|
|
|
1. Make `CHAINLIT_APP_PORT` dynamic — read from an env var per worker.
|
|
2. Replace `fkill(:port)` with PID tracking (`child.pid` from `spawn`).
|
|
3. Use `cypress-split` or `sorry-cypress` to distribute specs across multiple Cypress processes, each with its own `baseUrl`.
|
|
4. Each Cypress process gets its own `CYPRESS_BASE_URL` pointing to its dedicated backend port.
|