chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,121 @@
|
||||
# Test-selection — implementation state / handoff
|
||||
|
||||
Working notes for the diff-based CI test-selection system, so the work can be
|
||||
resumed without re-deriving context. For *how the system works / how to use it*,
|
||||
see [`TEST_SELECTION.md`](TEST_SELECTION.md); this file is only about the **state
|
||||
of the implementation**.
|
||||
|
||||
_Last updated: 2026-06-18._
|
||||
|
||||
|
||||
## Status: complete & locally verified; in review on PR #8077
|
||||
|
||||
The system is implemented end-to-end and passes local checks. Opened as
|
||||
[deepspeedai/DeepSpeed#8077](https://github.com/deepspeedai/DeepSpeed/pull/8077)
|
||||
(scoped to `modal-torch-latest` first; replicate to other workflows if accepted).
|
||||
It has **not** yet run a live `subset`/`deploy` on modal, so the first real runs
|
||||
should be watched.
|
||||
|
||||
### Review fixes applied (Codex bot, PR #8077)
|
||||
|
||||
- **P1 — run the selector from trusted code.** `collect-tests` now restores `ci/`
|
||||
from the base branch before running the selector + self-tests, so a PR can't
|
||||
rewrite `tests_fetcher.py` to emit `mode=none` and skip the Required check. (A
|
||||
PR's `ci/` changes still show in the diff and force a full run via `ci/**`.)
|
||||
Includes a bootstrap fallback: if base has no `tests_fetcher.py` yet, run all.
|
||||
- **P1 — hidden-file artifact.** `actions/upload-artifact@v4` skips dot-prefixed
|
||||
paths by default; `ci/.test_selection/` is one. Added `include-hidden-files:
|
||||
true` so `deploy`'s download doesn't fail.
|
||||
|
||||
> Reminder: under `pull_request_target`, `deploy` restores `ci/` from the base
|
||||
> branch, so changes under `ci/*` (the fetcher included) only take effect once
|
||||
> **merged**. To validate the live behavior before merge, trigger via a
|
||||
> `pull_request`-based run or the `modal` CLI.
|
||||
|
||||
|
||||
## What was built
|
||||
|
||||
A diff-driven selector that, on a PR, runs only the `tests/unit/v1` tests impacted
|
||||
by the changed files, with a fail-safe fallback to the full suite. Driven by the
|
||||
`modal-torch-latest` workflow.
|
||||
|
||||
### Files
|
||||
|
||||
| File | State | Notes |
|
||||
| --- | --- | --- |
|
||||
| `ci/tests_fetcher.py` | new/rewritten | Config-driven `TestSelector` class; import-graph selector. |
|
||||
| `ci/test_tests_fetcher.py` | new | 12 stdlib self-tests over synthetic git repos. |
|
||||
| `ci/torch_latest.py` | modified | Reads `DS_TEST_LIST_FILE`, runs pytest on the selected targets (falls back to full `tests/unit/v1`). |
|
||||
| `.github/workflows/modal-torch-latest.yml` | rewritten | `collect-tests` (fetcher) gates `deploy` (modal). |
|
||||
| `.github/workflows/TEST_SELECTION.md` | new | Full design + usage doc. |
|
||||
| `.github/workflows/TEST_SELECTION_STATE.md` | new | This file. |
|
||||
| `CONTRIBUTING.md` | modified | "Diff-based CI test selection" section + link. |
|
||||
| `.gitignore` | modified | Ignores `ci/.test_selection/`. |
|
||||
|
||||
|
||||
## Key design decisions (and why)
|
||||
|
||||
- **Always-runs Required job, selects instead of skips.** `modal-torch-latest` is a
|
||||
Required check, so path filters can't skip it. `collect-tests` decides
|
||||
`all|subset|none`; `none` skips `deploy` while still satisfying the Required
|
||||
status.
|
||||
- **`deploy` tests PR code but restores `ci/` from base.** Selection must reflect
|
||||
PR code, but the orchestration that holds modal/HF secrets must not be
|
||||
PR-controlled. (Security trade-off accepted earlier in the work.)
|
||||
- **Opaque hub modules + curated run-all globs.** `deepspeed`, `deepspeed.comm`,
|
||||
`deepspeed.accelerator` are opaque in the graph to stop universal fan-out; core
|
||||
hubs (`deepspeed/__init__.py`, `runtime/engine.py`, `comm/**`, `accelerator/**`,
|
||||
…) instead live in the run-all globs. (Coarseness fix accepted earlier.)
|
||||
- **Fail-safe to `all`, never `none`.** Every uncertain case (no base, no
|
||||
merge-base, parse error, unexpected exception) widens to the full suite.
|
||||
|
||||
### Improvements implemented in the latest pass (items #3–#11)
|
||||
|
||||
- **#3** Degrade to `all` on any selector error (top-level guard in `main()`).
|
||||
- **#4** `DYNAMIC_EDGES` map for runtime/dynamic deps the AST can't see
|
||||
(seeded: `deepspeed/module_inject/** → tests/unit/v1/moe/**`).
|
||||
- **#5** Deleted `.py` only forces `all` when a *surviving* file still imports it
|
||||
(dangling import); clean deletions no longer over-trigger.
|
||||
- **#6** GitHub job summary (`$GITHUB_STEP_SUMMARY`) with mode/reason/file list;
|
||||
`reason` also on `$GITHUB_OUTPUT`.
|
||||
- **#7** `--explain` prints import chains from changed files to selected tests.
|
||||
- **#8** Escape hatches documented in `CONTRIBUTING.md` + `TEST_SELECTION.md`.
|
||||
- **#9** Self-tests in `ci/test_tests_fetcher.py`, also run in `collect-tests`.
|
||||
- **#10** Robust merge-base: explicit check, `all` if missing; workflow keeps
|
||||
`fetch-depth: 0` with rationale.
|
||||
- **#11** Multi-workflow config: `WORKFLOWS` registry + `WorkflowConfig` +
|
||||
`--workflow` flag.
|
||||
|
||||
|
||||
## Verification done locally
|
||||
|
||||
- `python ci/tests_fetcher.py` self-tests: **12/12 pass**
|
||||
(narrowing, shared importers, core/comm/fixture run-all, commit tag, docs-only →
|
||||
none, new test file, clean vs. dangling delete, dynamic edge, missing base).
|
||||
- Real-repo smoke: `--base`, `--explain`, `--workflow` validation all behave.
|
||||
- Confirmed `$GITHUB_OUTPUT` + `$GITHUB_STEP_SUMMARY` emission and the #3
|
||||
error-fallback path directly.
|
||||
- No linter errors.
|
||||
|
||||
Re-run the core check any time:
|
||||
|
||||
```bash
|
||||
python ci/test_tests_fetcher.py
|
||||
python ci/tests_fetcher.py --base origin/master --explain
|
||||
```
|
||||
|
||||
|
||||
## Open items / possible follow-ups
|
||||
|
||||
- **Live validation:** watch the first real PR; confirm `collect-tests` artifact
|
||||
hand-off and `deploy` gating behave on Actions + modal.
|
||||
- **`DYNAMIC_EDGES` is minimal.** Only one seeded edge. Expand as blind spots are
|
||||
found (tests that fail on `master`'s full run but weren't selected on their PR
|
||||
are the signal).
|
||||
- **Single workflow wired.** The engine is config-driven (#11) but only
|
||||
`modal-torch-latest` is registered/wired. Other GPU workflows could adopt it by
|
||||
adding a `WorkflowConfig` and mirroring the `collect-tests` job.
|
||||
- **Run-all glob tuning.** The core-hub list is a conservative first cut; revisit
|
||||
if it proves too coarse (full runs on minor edits) or too narrow (missed impact).
|
||||
- **No telemetry.** Consider logging selection stats over time to tune
|
||||
globs/opaque-modules against real false-positive/negative rates.
|
||||
Reference in New Issue
Block a user