chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,306 @@
|
||||
# Diff-based CI test selection
|
||||
|
||||
This document explains the diff-driven test-selection system that lets some GPU CI
|
||||
workflows run only the tests a PR could affect, instead of the whole suite, while
|
||||
still satisfying Required status checks.
|
||||
|
||||
It currently drives **`modal-torch-latest`** (which runs `tests/unit/v1/` on
|
||||
[modal.com](https://modal.com) GPUs), but is built to drive more workflows from
|
||||
one config — see [Adding a workflow](#adding-a-new-workflow).
|
||||
|
||||
- [TL;DR](#tldr)
|
||||
- [Why](#why)
|
||||
- [Moving parts](#moving-parts)
|
||||
- [How a decision is made](#how-a-decision-is-made)
|
||||
- [Driving it (as a contributor)](#driving-it-as-a-contributor)
|
||||
- [Changing it (as a maintainer)](#changing-it-as-a-maintainer)
|
||||
- [Security model](#security-model)
|
||||
- [Failure modes & guarantees](#failure-modes--guarantees)
|
||||
- [FAQ / troubleshooting](#faq--troubleshooting)
|
||||
|
||||
|
||||
## TL;DR
|
||||
|
||||
- On a PR, `ci/tests_fetcher.py` diffs your branch against the base branch, traces
|
||||
the import graph from your changed files to the impacted tests, and writes the
|
||||
list to `ci/.test_selection/test_list.txt`.
|
||||
- The workflow runs pytest on just that list. `push` to `master` and manual runs
|
||||
always run everything.
|
||||
- It is **fail-safe**: anything it can't reason about safely → run the *full* suite.
|
||||
It never silently runs *fewer* tests than reality.
|
||||
- Preview locally:
|
||||
```bash
|
||||
python ci/tests_fetcher.py --base origin/master # what would CI run?
|
||||
python ci/tests_fetcher.py --base origin/master --explain # ...and why?
|
||||
```
|
||||
- Force a full run: put `[test all]` (or `[no filter]`) in a commit message.
|
||||
|
||||
|
||||
## Why
|
||||
|
||||
`modal-torch-latest` is a **Required** check, so it must report a status on every
|
||||
PR — which means we can't use GitHub's path filters (`on.<event>.paths`) to skip
|
||||
it, because a skipped Required job blocks merges. Instead, the job always runs but
|
||||
*selects* what to test: a docs-only PR resolves to "no impacted tests" and exits
|
||||
fast (still green), while a PR touching a leaf module runs only the tests that
|
||||
import it.
|
||||
|
||||
The design is a small, self-contained take on HuggingFace `transformers`'
|
||||
`utils/tests_fetcher.py`.
|
||||
|
||||
|
||||
## Moving parts
|
||||
|
||||
| File | Role |
|
||||
| --- | --- |
|
||||
| `.github/workflows/modal-torch-latest.yml` | The workflow: a `collect-tests` job (runs the fetcher) gating a `deploy` job (runs pytest on modal). |
|
||||
| `ci/tests_fetcher.py` | The selector. AST-parses the repo, builds an import graph, decides `all` / `subset` / `none`, writes the test-list file, emits a job summary. |
|
||||
| `ci/test_tests_fetcher.py` | Self-tests for the selector (pure stdlib; run in `collect-tests`). |
|
||||
| `ci/torch_latest.py` | The modal runner. Reads the test-list file and feeds it to pytest. |
|
||||
| `ci/.test_selection/test_list.txt` | The hand-off artifact (one pytest target per line). Git-ignored. |
|
||||
|
||||
### Job flow
|
||||
|
||||
```
|
||||
pull_request_target / push / workflow_dispatch
|
||||
│
|
||||
┌─────────────┴─────────────┐
|
||||
│ collect-tests │ (no secrets; AST-only)
|
||||
│ checkout PR head │
|
||||
│ restore ci/ from base │
|
||||
│ self-test the fetcher │
|
||||
│ run tests_fetcher.py │
|
||||
│ → mode (all|subset|none) │
|
||||
│ → upload test_list.txt │
|
||||
└─────────────┬──────────────┘
|
||||
│ needs:
|
||||
▼
|
||||
┌────────────────────────────┐
|
||||
│ deploy │ (has modal/HF secrets)
|
||||
│ if mode != none (or manual │
|
||||
│ / collector failed) │
|
||||
│ checkout PR head │
|
||||
│ restore ci/ from base │
|
||||
│ download test_list.txt │
|
||||
│ modal run ci.torch_latest │
|
||||
└────────────────────────────┘
|
||||
```
|
||||
|
||||
`mode` controls `deploy`:
|
||||
|
||||
- **`none`** → `deploy` is skipped. The Required status is still satisfied (a
|
||||
skipped dependent job counts as success here).
|
||||
- **`subset`** → `deploy` runs pytest on exactly the impacted files.
|
||||
- **`all`** → `deploy` runs the whole scope (`tests/unit/v1`).
|
||||
|
||||
|
||||
## How a decision is made
|
||||
|
||||
`TestSelector.select()` in `ci/tests_fetcher.py` runs these checks in order; the
|
||||
first that matches wins:
|
||||
|
||||
1. **No base ref** (push / manual) → `all`.
|
||||
2. **Base ref unresolvable** → `all`.
|
||||
3. **No merge-base** with the base (e.g. shallow clone, unrelated history) → `all`.
|
||||
A diff here would be wrong, so we never narrow on it.
|
||||
4. **Commit message tag** `[test all]` / `[no filter]` anywhere on the branch → `all`.
|
||||
5. **A changed file matches a run-all glob** (`COMMON_RUN_ALL_GLOBS` +
|
||||
the workflow's `extra_run_all_globs`) → `all`. These are files too central or
|
||||
too dynamic to narrow safely: CI scripts, build system, `csrc/`, `op_builder/`,
|
||||
`accelerator/`, shared fixtures (`tests/unit/common.py`, `tests/conftest.py`,
|
||||
`pytest.ini`), and core runtime hubs (`deepspeed/__init__.py`,
|
||||
`deepspeed/runtime/engine.py`, `deepspeed/comm/**`, `deepspeed/accelerator/**`, …).
|
||||
6. **A deleted module is still imported** by a surviving file (a dangling import
|
||||
the graph can't follow) → `all`. A *clean* deletion (importers removed/updated
|
||||
in the same PR) does **not** trigger this.
|
||||
7. Otherwise, **narrow via the import graph** (below). If nothing is impacted →
|
||||
`none`; if everything is → `all`; else → `subset`.
|
||||
|
||||
### The import graph
|
||||
|
||||
- Nodes are Python files under the package roots: `deepspeed/**` and the `unit`
|
||||
test-helper package at `tests/unit/**`.
|
||||
- An edge `A → B` means "B imports A". The selector walks **backwards** from each
|
||||
changed file to every test that (transitively) imports it.
|
||||
- **Opaque hub modules** (`OPAQUE_MODULES`: `deepspeed`, `deepspeed.comm`,
|
||||
`deepspeed.accelerator`): almost every test imports `unit.common`, which imports
|
||||
these. Their `__init__.py` files eagerly pull in huge subtrees, so if treated as
|
||||
normal nodes *any* `deepspeed/**` change would fan out to the whole suite. We
|
||||
therefore don't expand their `__init__` imports; instead, changes to the hubs
|
||||
themselves are caught by the run-all globs in step 5.
|
||||
- **`conftest.py`** changes select every test under that conftest's directory.
|
||||
- **New test files** are selected directly (they have no importers yet).
|
||||
|
||||
### Dynamic edges (`DYNAMIC_EDGES`)
|
||||
|
||||
Some dependencies are wired at runtime (monkey-patching, plugin/registry lookup,
|
||||
JIT-loaded ops, `deepspeed.initialize()`-time `replace_module` injection), so a
|
||||
test can depend on code it never `import`s. `DYNAMIC_EDGES` is a curated map of
|
||||
`changed-file glob → extra test-path globs` that patches these blind spots. It is
|
||||
additive on top of the static graph (and is only consulted if step 5 didn't
|
||||
already short-circuit to `all`).
|
||||
|
||||
|
||||
## Driving it (as a contributor)
|
||||
|
||||
### Preview what CI will run
|
||||
|
||||
The fetcher is pure stdlib — no DeepSpeed/torch install needed, just `git`:
|
||||
|
||||
```bash
|
||||
# Selection for your branch vs. the base branch
|
||||
python ci/tests_fetcher.py --base origin/master
|
||||
cat ci/.test_selection/test_list.txt
|
||||
|
||||
# Explain *why* each test was selected (prints import chains)
|
||||
python ci/tests_fetcher.py --base origin/master --explain
|
||||
```
|
||||
|
||||
`--explain` output looks like:
|
||||
|
||||
```
|
||||
deepspeed/shared.py impacts:
|
||||
tests/unit/v1/test_shared.py <- deepspeed/shared.py
|
||||
tests/unit/v1/moe/test_moe.py <- tests/unit/v1/moe/test_moe.py <- deepspeed/shared.py
|
||||
```
|
||||
|
||||
### Escape hatches
|
||||
|
||||
- **Force the full suite for a push:** include `[test all]` (or `[no filter]`)
|
||||
anywhere in a commit message on the branch.
|
||||
- **Touch an infra file:** any change to a run-all glob runs everything.
|
||||
- **Found a missed test?** It's likely a runtime/dynamic dependency the static
|
||||
graph can't see — add a `DYNAMIC_EDGES` entry (see below) and/or report it.
|
||||
|
||||
|
||||
## Changing it (as a maintainer)
|
||||
|
||||
All knobs live in `ci/tests_fetcher.py`. After any change, run the self-tests:
|
||||
|
||||
```bash
|
||||
python ci/test_tests_fetcher.py # standalone
|
||||
# or: pytest ci/test_tests_fetcher.py
|
||||
```
|
||||
|
||||
### Add a run-all trigger
|
||||
|
||||
Add a glob to `TestSelector.COMMON_RUN_ALL_GLOBS` (shared across workflows) or to
|
||||
a specific workflow's `extra_run_all_globs`. Globs match repo-root-relative POSIX
|
||||
paths; `dir/**` matches everything under `dir/`.
|
||||
|
||||
### Add a dynamic edge
|
||||
|
||||
Add to `TestSelector.DYNAMIC_EDGES`:
|
||||
|
||||
```python
|
||||
DYNAMIC_EDGES = {
|
||||
# changed-file glob : test-path globs to pull in when it changes
|
||||
"deepspeed/module_inject/**": ("tests/unit/v1/moe/**",),
|
||||
}
|
||||
```
|
||||
|
||||
Keep entries conservative: too broad just wastes GPU time; too narrow misses
|
||||
coverage. Prefer fixing it here over widening a run-all glob when only a slice of
|
||||
tests is truly affected.
|
||||
|
||||
### Mark a module opaque
|
||||
|
||||
If a new universal hub starts fanning every change out to the whole suite, add it
|
||||
to `OPAQUE_MODULES` and (usually) add the hub file itself to the run-all globs so
|
||||
real changes to it still run everything.
|
||||
|
||||
### Add a new workflow
|
||||
|
||||
The engine is config-driven (`WORKFLOWS` registry of `WorkflowConfig`). To drive
|
||||
another workflow:
|
||||
|
||||
1. Add an entry:
|
||||
```python
|
||||
WORKFLOWS["my-workflow"] = WorkflowConfig(
|
||||
name="my-workflow",
|
||||
test_scopes=("tests/unit/v2",), # dirs this workflow runs
|
||||
extra_run_all_globs=(".github/workflows/my-workflow.yml",),
|
||||
)
|
||||
```
|
||||
2. In that workflow's YAML, mirror `modal-torch-latest.yml`'s `collect-tests` job
|
||||
and call the fetcher with `--workflow my-workflow`.
|
||||
3. Point the runner at `ci/.test_selection/test_list.txt`.
|
||||
4. Add coverage to `ci/test_tests_fetcher.py` if the scope/behavior differs.
|
||||
|
||||
### Add a self-test
|
||||
|
||||
`ci/test_tests_fetcher.py` builds throwaway git repos (`TmpRepo`) from a synthetic
|
||||
`BASELINE` tree and asserts the resulting `mode`/`tests`. Add a `test_*` function
|
||||
for any new behavior; it runs both standalone and under pytest, and executes in
|
||||
the `collect-tests` CI job, so a broken selector is caught before it mis-picks
|
||||
tests.
|
||||
|
||||
|
||||
## Security model
|
||||
|
||||
The workflow triggers on **`pull_request_target`**, so it runs in the base repo's
|
||||
context and the `deploy` job has the modal/HF **secrets** in scope. To keep PRs
|
||||
from abusing that:
|
||||
|
||||
- `collect-tests` holds **no secrets** and only **AST-parses** (never executes)
|
||||
the PR tree.
|
||||
- **Both jobs restore `ci/` from the base branch** before using it:
|
||||
- `collect-tests` runs the **base** selector + self-tests. This job decides
|
||||
whether the Required `deploy` runs, so it must not trust the PR's own
|
||||
`ci/tests_fetcher.py` — otherwise a PR could rewrite it to emit `mode=none`
|
||||
and skip CI while still going green. The diff is computed from git history, so
|
||||
a PR's `ci/` changes still appear in the diff and (via the base selector's
|
||||
`ci/**` run-all glob) force a full run.
|
||||
- `deploy` runs the **base** orchestration (which drives `modal run` with the
|
||||
secrets), so a PR can't repoint it at the secrets to exfiltrate them.
|
||||
In both jobs the PR's `deepspeed/` + `tests/` are what gets parsed/tested.
|
||||
- The `pull_request_target` trigger types are `review_requested`,
|
||||
`ready_for_review`, and `synchronize`. Because `synchronize` re-runs on every
|
||||
push to an open PR (not just on a maintainer action), the maintainer review is a
|
||||
mitigation, **not** an absolute barrier — the base-`ci/` restore above is the
|
||||
primary protection for the secrets.
|
||||
|
||||
> **Consequence:** changes to `ci/*` (including `tests_fetcher.py` itself) take
|
||||
> effect under `pull_request_target` only after they're **merged**. Validate
|
||||
> `ci/*` changes via a `pull_request`-triggered run or the `modal` CLI locally.
|
||||
>
|
||||
> **Bootstrap:** when the base branch has no selector yet (the PR that introduces
|
||||
> it), the restored base `ci/` won't contain `tests_fetcher.py`; `collect-tests`
|
||||
> detects this and falls back to running the full suite.
|
||||
|
||||
|
||||
## Failure modes & guarantees
|
||||
|
||||
The selector is built to **fail safe — to `all`, never to `none`**:
|
||||
|
||||
- Missing/unresolvable base, no merge-base, a parse error on a file, or **any
|
||||
unexpected exception** in the selector → it falls back to the full suite (the
|
||||
top-level handler in `main()` logs the traceback and sets `mode=all`).
|
||||
- If `collect-tests` fails entirely, `deploy` still runs (and the workflow has an
|
||||
explicit "fail if selection failed" guard) so a broken collector can't let a PR
|
||||
pass without testing.
|
||||
- The only way to run *fewer* tests is a clean, well-understood narrow decision;
|
||||
every uncertain case widens to everything.
|
||||
|
||||
Every run writes a summary to the GitHub job summary (mode, reason, and the
|
||||
selected files) so the decision is auditable from the Actions UI.
|
||||
|
||||
|
||||
## FAQ / troubleshooting
|
||||
|
||||
**My PR shows `mode=none` but I changed code.**
|
||||
Either the change is non-Python / out of the workflow's scope, or your edits
|
||||
aren't committed (the fetcher diffs *committed* history, `merge-base..HEAD`).
|
||||
|
||||
**A relevant test wasn't selected.**
|
||||
Likely a runtime/dynamic dependency. Confirm with `--explain`, then add a
|
||||
`DYNAMIC_EDGES` entry. As a stop-gap, push with `[test all]`.
|
||||
|
||||
**Everything runs even for a tiny change.**
|
||||
You touched a run-all glob (CI/build/shared fixture/core runtime), or a hub module
|
||||
fanned out. Check the job summary's `reason`. If a hub over-fans, consider
|
||||
`OPAQUE_MODULES`.
|
||||
|
||||
**It ran the full suite and the summary says "shallow clone?" / "no merge-base".**
|
||||
The runner didn't have enough history to compute the diff. `collect-tests` uses
|
||||
`fetch-depth: 0`; if you changed that, restore full history for the base.
|
||||
@@ -0,0 +1,121 @@
|
||||
# Test-selection — implementation state / handoff
|
||||
|
||||
Working notes for the diff-based CI test-selection system, so the work can be
|
||||
resumed without re-deriving context. For *how the system works / how to use it*,
|
||||
see [`TEST_SELECTION.md`](TEST_SELECTION.md); this file is only about the **state
|
||||
of the implementation**.
|
||||
|
||||
_Last updated: 2026-06-18._
|
||||
|
||||
|
||||
## Status: complete & locally verified; in review on PR #8077
|
||||
|
||||
The system is implemented end-to-end and passes local checks. Opened as
|
||||
[deepspeedai/DeepSpeed#8077](https://github.com/deepspeedai/DeepSpeed/pull/8077)
|
||||
(scoped to `modal-torch-latest` first; replicate to other workflows if accepted).
|
||||
It has **not** yet run a live `subset`/`deploy` on modal, so the first real runs
|
||||
should be watched.
|
||||
|
||||
### Review fixes applied (Codex bot, PR #8077)
|
||||
|
||||
- **P1 — run the selector from trusted code.** `collect-tests` now restores `ci/`
|
||||
from the base branch before running the selector + self-tests, so a PR can't
|
||||
rewrite `tests_fetcher.py` to emit `mode=none` and skip the Required check. (A
|
||||
PR's `ci/` changes still show in the diff and force a full run via `ci/**`.)
|
||||
Includes a bootstrap fallback: if base has no `tests_fetcher.py` yet, run all.
|
||||
- **P1 — hidden-file artifact.** `actions/upload-artifact@v4` skips dot-prefixed
|
||||
paths by default; `ci/.test_selection/` is one. Added `include-hidden-files:
|
||||
true` so `deploy`'s download doesn't fail.
|
||||
|
||||
> Reminder: under `pull_request_target`, `deploy` restores `ci/` from the base
|
||||
> branch, so changes under `ci/*` (the fetcher included) only take effect once
|
||||
> **merged**. To validate the live behavior before merge, trigger via a
|
||||
> `pull_request`-based run or the `modal` CLI.
|
||||
|
||||
|
||||
## What was built
|
||||
|
||||
A diff-driven selector that, on a PR, runs only the `tests/unit/v1` tests impacted
|
||||
by the changed files, with a fail-safe fallback to the full suite. Driven by the
|
||||
`modal-torch-latest` workflow.
|
||||
|
||||
### Files
|
||||
|
||||
| File | State | Notes |
|
||||
| --- | --- | --- |
|
||||
| `ci/tests_fetcher.py` | new/rewritten | Config-driven `TestSelector` class; import-graph selector. |
|
||||
| `ci/test_tests_fetcher.py` | new | 12 stdlib self-tests over synthetic git repos. |
|
||||
| `ci/torch_latest.py` | modified | Reads `DS_TEST_LIST_FILE`, runs pytest on the selected targets (falls back to full `tests/unit/v1`). |
|
||||
| `.github/workflows/modal-torch-latest.yml` | rewritten | `collect-tests` (fetcher) gates `deploy` (modal). |
|
||||
| `.github/workflows/TEST_SELECTION.md` | new | Full design + usage doc. |
|
||||
| `.github/workflows/TEST_SELECTION_STATE.md` | new | This file. |
|
||||
| `CONTRIBUTING.md` | modified | "Diff-based CI test selection" section + link. |
|
||||
| `.gitignore` | modified | Ignores `ci/.test_selection/`. |
|
||||
|
||||
|
||||
## Key design decisions (and why)
|
||||
|
||||
- **Always-runs Required job, selects instead of skips.** `modal-torch-latest` is a
|
||||
Required check, so path filters can't skip it. `collect-tests` decides
|
||||
`all|subset|none`; `none` skips `deploy` while still satisfying the Required
|
||||
status.
|
||||
- **`deploy` tests PR code but restores `ci/` from base.** Selection must reflect
|
||||
PR code, but the orchestration that holds modal/HF secrets must not be
|
||||
PR-controlled. (Security trade-off accepted earlier in the work.)
|
||||
- **Opaque hub modules + curated run-all globs.** `deepspeed`, `deepspeed.comm`,
|
||||
`deepspeed.accelerator` are opaque in the graph to stop universal fan-out; core
|
||||
hubs (`deepspeed/__init__.py`, `runtime/engine.py`, `comm/**`, `accelerator/**`,
|
||||
…) instead live in the run-all globs. (Coarseness fix accepted earlier.)
|
||||
- **Fail-safe to `all`, never `none`.** Every uncertain case (no base, no
|
||||
merge-base, parse error, unexpected exception) widens to the full suite.
|
||||
|
||||
### Improvements implemented in the latest pass (items #3–#11)
|
||||
|
||||
- **#3** Degrade to `all` on any selector error (top-level guard in `main()`).
|
||||
- **#4** `DYNAMIC_EDGES` map for runtime/dynamic deps the AST can't see
|
||||
(seeded: `deepspeed/module_inject/** → tests/unit/v1/moe/**`).
|
||||
- **#5** Deleted `.py` only forces `all` when a *surviving* file still imports it
|
||||
(dangling import); clean deletions no longer over-trigger.
|
||||
- **#6** GitHub job summary (`$GITHUB_STEP_SUMMARY`) with mode/reason/file list;
|
||||
`reason` also on `$GITHUB_OUTPUT`.
|
||||
- **#7** `--explain` prints import chains from changed files to selected tests.
|
||||
- **#8** Escape hatches documented in `CONTRIBUTING.md` + `TEST_SELECTION.md`.
|
||||
- **#9** Self-tests in `ci/test_tests_fetcher.py`, also run in `collect-tests`.
|
||||
- **#10** Robust merge-base: explicit check, `all` if missing; workflow keeps
|
||||
`fetch-depth: 0` with rationale.
|
||||
- **#11** Multi-workflow config: `WORKFLOWS` registry + `WorkflowConfig` +
|
||||
`--workflow` flag.
|
||||
|
||||
|
||||
## Verification done locally
|
||||
|
||||
- `python ci/tests_fetcher.py` self-tests: **12/12 pass**
|
||||
(narrowing, shared importers, core/comm/fixture run-all, commit tag, docs-only →
|
||||
none, new test file, clean vs. dangling delete, dynamic edge, missing base).
|
||||
- Real-repo smoke: `--base`, `--explain`, `--workflow` validation all behave.
|
||||
- Confirmed `$GITHUB_OUTPUT` + `$GITHUB_STEP_SUMMARY` emission and the #3
|
||||
error-fallback path directly.
|
||||
- No linter errors.
|
||||
|
||||
Re-run the core check any time:
|
||||
|
||||
```bash
|
||||
python ci/test_tests_fetcher.py
|
||||
python ci/tests_fetcher.py --base origin/master --explain
|
||||
```
|
||||
|
||||
|
||||
## Open items / possible follow-ups
|
||||
|
||||
- **Live validation:** watch the first real PR; confirm `collect-tests` artifact
|
||||
hand-off and `deploy` gating behave on Actions + modal.
|
||||
- **`DYNAMIC_EDGES` is minimal.** Only one seeded edge. Expand as blind spots are
|
||||
found (tests that fail on `master`'s full run but weren't selected on their PR
|
||||
are the signal).
|
||||
- **Single workflow wired.** The engine is config-driven (#11) but only
|
||||
`modal-torch-latest` is registered/wired. Other GPU workflows could adopt it by
|
||||
adding a `WorkflowConfig` and mirroring the `collect-tests` job.
|
||||
- **Run-all glob tuning.** The core-hub list is a conservative first cut; revisit
|
||||
if it proves too coarse (full runs on minor edits) or too narrow (missed impact).
|
||||
- **No telemetry.** Consider logging selection stats over time to tune
|
||||
globs/opaque-modules against real false-positive/negative rates.
|
||||
@@ -0,0 +1,85 @@
|
||||
name: amd-mi200
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
pull_request:
|
||||
paths:
|
||||
- '.github/workflows/amd-mi200.yml'
|
||||
- 'requirements/**'
|
||||
schedule:
|
||||
- cron: "0 0 * * *"
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
|
||||
jobs:
|
||||
amd-tests:
|
||||
name: amd-mi200 / AMD MI200 tests
|
||||
# The type of runner that the job will run on
|
||||
runs-on: [self-hosted, amd, mi200]
|
||||
|
||||
# Steps represent a sequence of tasks that will be executed as part of the job
|
||||
steps:
|
||||
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- id: setup-venv
|
||||
uses: ./.github/workflows/setup-venv
|
||||
|
||||
- name: Install pytorch
|
||||
run: |
|
||||
pip install -U --cache-dir $TORCH_CACHE torch torchvision --index-url https://download.pytorch.org/whl/rocm6.0
|
||||
python -c "import torch; print('torch:', torch.__version__, torch)"
|
||||
python -c "import torch; print('CUDA available:', torch.cuda.is_available())"
|
||||
|
||||
- name: Install transformers
|
||||
run: |
|
||||
git clone https://github.com/huggingface/transformers
|
||||
cd transformers
|
||||
# if needed switch to the last known good SHA until transformers@master is fixed
|
||||
# git checkout 1cc453d33
|
||||
git rev-parse --short HEAD
|
||||
pip install .
|
||||
|
||||
- name: Install (ROCm) apex
|
||||
run: |
|
||||
git clone https://github.com/ROCmSoftwarePlatform/apex.git
|
||||
CURRENT_VER=$(git rev-parse HEAD)
|
||||
INSTALLED_VER=$(cat /blob/amd-apex/.venv_installed_version)
|
||||
if [[ "$CURRENT_VER" != "$INSTALLED_VER" ]]; then
|
||||
pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings="--global-option=--cpp_ext" --config-settings="--global-option=--cuda_ext" --target=/blob/amd-apex/ --upgrade .
|
||||
git rev-parse HEAD > /blob/amd-apex/.venv_installed_version
|
||||
fi
|
||||
echo PYTHONPATH=$PYTHONPATH:/blob/amd-apex/ >> $GITHUB_ENV
|
||||
# Runs a set of commands using the runners shell
|
||||
- name: Install deepspeed
|
||||
run: |
|
||||
pip install .[dev,1bit,autotuning]
|
||||
#python -c "from deepspeed.env_report import cli_main; cli_main()"
|
||||
ds_report
|
||||
|
||||
- name: Python environment
|
||||
run: |
|
||||
pip list
|
||||
|
||||
# Runs a set of commands using the runners shell
|
||||
- name: Unit tests
|
||||
run: |
|
||||
unset TORCH_CUDA_ARCH_LIST # only jit compile for current arch
|
||||
cd tests
|
||||
pytest $PYTEST_OPTS -n 4 --verbose unit/
|
||||
pytest $PYTEST_OPTS -m 'sequential' unit/
|
||||
|
||||
- name: Open GitHub issue if nightly CI fails
|
||||
if: ${{ failure() && (github.event_name == 'schedule') }}
|
||||
uses: JasonEtco/create-an-issue@v2
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
with:
|
||||
filename: .github/ISSUE_TEMPLATE/ci_failure_report.md
|
||||
update_existing: true
|
||||
@@ -0,0 +1,115 @@
|
||||
################################################################################
|
||||
# DeepSpeed CI - AWS L40S GPU Tests (HuggingFace Accelerate Integration)
|
||||
#
|
||||
# Runs the same tests as modal-accelerate.yml but on AWS self-hosted runners.
|
||||
# Tests DeepSpeed integration with HuggingFace Accelerate library.
|
||||
# Uses 4x NVIDIA L40S GPUs on g6e.12xlarge instances.
|
||||
################################################################################
|
||||
|
||||
name: aws-accelerate
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
|
||||
push:
|
||||
branches:
|
||||
- master
|
||||
|
||||
pull_request:
|
||||
branches:
|
||||
- master
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
check-paths:
|
||||
name: aws-accelerate / check paths
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
should_run: ${{ steps.filter.outputs.run_tests }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: dorny/paths-filter@v3
|
||||
id: filter
|
||||
with:
|
||||
filters: |
|
||||
run_tests:
|
||||
- '**'
|
||||
- '!docs/**'
|
||||
- '!blogs/**'
|
||||
- '!deepspeed/inference/v2/**'
|
||||
- '!tests/unit/inference/v2/**'
|
||||
|
||||
accelerate-tests:
|
||||
name: aws-accelerate / accelerate integration tests
|
||||
needs: check-paths
|
||||
if: needs.check-paths.outputs.should_run == 'true'
|
||||
runs-on: [self-hosted, gpu-ci, gpu-l40s, l40s-1gpu, aws]
|
||||
timeout-minutes: 60
|
||||
|
||||
container:
|
||||
image: nvidia/cuda:12.6.3-devel-ubuntu22.04
|
||||
options: --gpus all --shm-size "32G"
|
||||
|
||||
env:
|
||||
TORCH_VER: "2.7"
|
||||
CUDA_VER: "12.6"
|
||||
|
||||
steps:
|
||||
- name: Install system dependencies
|
||||
run: |
|
||||
apt-get update && apt-get install -y git git-lfs libaio-dev python3 python3-pip
|
||||
git lfs install
|
||||
ln -sf /usr/bin/python3 /usr/bin/python
|
||||
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
lfs: true
|
||||
|
||||
- name: Install PyTorch
|
||||
run: |
|
||||
pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu126
|
||||
|
||||
- name: Install Python dependencies
|
||||
run: |
|
||||
pip install --upgrade pip
|
||||
pip install -r requirements/requirements.txt
|
||||
pip install -r requirements/requirements-dev.txt
|
||||
pip install datasets
|
||||
|
||||
- name: Check environment
|
||||
run: |
|
||||
echo "=== GPU Information ==="
|
||||
nvidia-smi
|
||||
echo ""
|
||||
echo "=== CUDA Version ==="
|
||||
nvcc --version
|
||||
echo ""
|
||||
echo "=== Python/PyTorch Info ==="
|
||||
python --version
|
||||
python -c "import torch; print(f'PyTorch: {torch.__version__}')"
|
||||
python -c "import torch; print(f'CUDA available: {torch.cuda.is_available()}')"
|
||||
python -c "import torch; print(f'CUDA devices: {torch.cuda.device_count()}')"
|
||||
python -c "import torch; print(f'BF16 support: {torch.cuda.is_bf16_supported()}')"
|
||||
|
||||
- name: Install DeepSpeed
|
||||
run: |
|
||||
# Initialize CUDA before install so setup.py can detect NCCL version
|
||||
python -c "import torch; torch.cuda.init(); print(f'NCCL version: {torch.cuda.nccl.version()}')"
|
||||
# Use --no-build-isolation so setup.py can access pre-installed PyTorch
|
||||
pip install --no-build-isolation .
|
||||
ds_report
|
||||
# Debug: Check captured torch_info values
|
||||
python -c "from deepspeed.git_version_info import torch_info; print(f'torch_info: {torch_info}')"
|
||||
|
||||
- name: Clone and install Accelerate
|
||||
run: |
|
||||
git clone https://github.com/huggingface/accelerate
|
||||
pip install "./accelerate[testing]"
|
||||
|
||||
- name: Run Accelerate DeepSpeed tests
|
||||
run: |
|
||||
pytest --verbose ./accelerate/tests/deepspeed
|
||||
@@ -0,0 +1,380 @@
|
||||
################################################################################
|
||||
# DeepSpeed CI - AWS L40S GPU Full Tests (PyTorch Latest)
|
||||
#
|
||||
# Runs the full DeepSpeed unit test suite on AWS self-hosted runners.
|
||||
# Prefers 4x NVIDIA L40S GPUs on g6e.12xlarge instances, with AWS-side
|
||||
# fallback to 8x A100 nodes when L40S capacity is unavailable.
|
||||
#
|
||||
# This workflow runs:
|
||||
# - Parallel tests with pytest-xdist (-n 8)
|
||||
# - Sequential tests marked with @pytest.mark.sequential
|
||||
# - Nightly schedule: skips if no new commits since last successful run
|
||||
################################################################################
|
||||
|
||||
name: aws-torch-latest-full
|
||||
|
||||
on:
|
||||
schedule:
|
||||
- cron: '0 8 * * *' # Daily at 08:00 UTC (midnight PST)
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
torch_preset:
|
||||
description: PyTorch preset to install for manual runs
|
||||
required: false
|
||||
default: '2.10.0-cu126'
|
||||
type: choice
|
||||
options:
|
||||
- '2.7.1-cu126'
|
||||
- '2.8.0-cu126'
|
||||
- '2.9.1-cu126'
|
||||
- '2.10.0-cu126'
|
||||
- '2.11.0-cu126'
|
||||
transformers_version:
|
||||
description: Hugging Face Transformers PyPI package version to install
|
||||
required: false
|
||||
default: '4.50.0'
|
||||
type: string
|
||||
transformers_source:
|
||||
description: Hugging Face Transformers source for manual runs
|
||||
required: false
|
||||
default: 'git'
|
||||
type: choice
|
||||
options:
|
||||
- 'pypi'
|
||||
- 'git'
|
||||
transformers_ref:
|
||||
description: Hugging Face Transformers git ref to install when source is git
|
||||
required: false
|
||||
default: 'main'
|
||||
type: string
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
check-changes:
|
||||
name: Check for new commits
|
||||
runs-on: ubuntu-latest
|
||||
if: github.event_name == 'schedule'
|
||||
outputs:
|
||||
has_changes: ${{ steps.check.outputs.has_changes }}
|
||||
steps:
|
||||
- name: Check for commits since last successful run
|
||||
id: check
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
default_branch="${{ github.event.repository.default_branch }}"
|
||||
|
||||
last_sha=$(gh api \
|
||||
"repos/${{ github.repository }}/actions/workflows/aws-torch-latest-full.yml/runs?status=success&event=schedule&branch=${default_branch}&per_page=1" \
|
||||
--jq '.workflow_runs[0].head_sha // empty')
|
||||
|
||||
current_sha="${{ github.sha }}"
|
||||
|
||||
if [ -z "$last_sha" ]; then
|
||||
echo "No previous successful run found - running tests"
|
||||
echo "has_changes=true" >> "$GITHUB_OUTPUT"
|
||||
elif [ "$last_sha" = "$current_sha" ]; then
|
||||
echo "No new commits since last successful run ($last_sha) - skipping"
|
||||
echo "has_changes=false" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "New commits detected: $last_sha -> $current_sha - running tests"
|
||||
echo "has_changes=true" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
unit-tests:
|
||||
name: Unit Tests (Full)
|
||||
needs: [check-changes]
|
||||
if: |
|
||||
!cancelled() &&
|
||||
(github.event_name == 'workflow_dispatch' || needs.check-changes.outputs.has_changes == 'true')
|
||||
runs-on: [self-hosted, gpu-ci, gpu-l40s, l40s-4gpu, aws]
|
||||
timeout-minutes: 180
|
||||
|
||||
container:
|
||||
image: nvidia/cuda:12.6.3-devel-ubuntu22.04
|
||||
# Mount /mnt/aio for async I/O tests (O_DIRECT requires native filesystem, not overlayfs)
|
||||
options: --gpus all --shm-size "32G" -v /mnt/aio:/mnt/aio
|
||||
|
||||
env:
|
||||
DEFAULT_TORCH_PRESET: '2.10.0-cu126'
|
||||
DEFAULT_TRANSFORMERS_SOURCE: 'git'
|
||||
DEFAULT_TRANSFORMERS_VERSION: '4.50.0'
|
||||
DEFAULT_TRANSFORMERS_REF: 'main'
|
||||
CUTLASS_PATH: /opt/cutlass
|
||||
# Disable reuse_dist_env to prevent pool worker cleanup hangs in full test runs
|
||||
DS_DISABLE_REUSE_DIST_ENV: '1'
|
||||
|
||||
steps:
|
||||
- name: Install system dependencies
|
||||
run: |
|
||||
apt-get update && apt-get install -y git git-lfs libaio-dev pdsh python3 python3-pip
|
||||
git lfs install
|
||||
ln -sf /usr/bin/python3 /usr/bin/python
|
||||
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
lfs: true
|
||||
|
||||
- name: Resolve dependency inputs
|
||||
env:
|
||||
GITHUB_EVENT_NAME: ${{ github.event_name }}
|
||||
MANUAL_TORCH_PRESET: ${{ github.event.inputs.torch_preset || '' }}
|
||||
MANUAL_TRANSFORMERS_SOURCE: ${{ github.event.inputs.transformers_source || '' }}
|
||||
MANUAL_TRANSFORMERS_VERSION: ${{ github.event.inputs.transformers_version || '' }}
|
||||
MANUAL_TRANSFORMERS_REF: ${{ github.event.inputs.transformers_ref || '' }}
|
||||
run: |
|
||||
if [ "$GITHUB_EVENT_NAME" = 'workflow_dispatch' ] && [ -n "$MANUAL_TORCH_PRESET" ]; then
|
||||
selected_preset="$MANUAL_TORCH_PRESET"
|
||||
else
|
||||
selected_preset="$DEFAULT_TORCH_PRESET"
|
||||
fi
|
||||
|
||||
if [ "$GITHUB_EVENT_NAME" = 'workflow_dispatch' ] && [ -n "$MANUAL_TRANSFORMERS_SOURCE" ]; then
|
||||
transformers_source="$MANUAL_TRANSFORMERS_SOURCE"
|
||||
else
|
||||
transformers_source="$DEFAULT_TRANSFORMERS_SOURCE"
|
||||
fi
|
||||
|
||||
if [ "$GITHUB_EVENT_NAME" = 'workflow_dispatch' ] && [ -n "$MANUAL_TRANSFORMERS_VERSION" ]; then
|
||||
transformers_version="$MANUAL_TRANSFORMERS_VERSION"
|
||||
else
|
||||
transformers_version="$DEFAULT_TRANSFORMERS_VERSION"
|
||||
fi
|
||||
|
||||
if [ "$GITHUB_EVENT_NAME" = 'workflow_dispatch' ] && [ -n "$MANUAL_TRANSFORMERS_REF" ]; then
|
||||
transformers_ref="$MANUAL_TRANSFORMERS_REF"
|
||||
else
|
||||
transformers_ref="$DEFAULT_TRANSFORMERS_REF"
|
||||
fi
|
||||
|
||||
if [ "$transformers_source" = 'git' ] && [ -z "$transformers_ref" ]; then
|
||||
transformers_ref='main'
|
||||
fi
|
||||
|
||||
case "$selected_preset" in
|
||||
'2.7.1-cu126')
|
||||
torch_install_version='2.7.1'
|
||||
torchvision_install_version='0.22.1'
|
||||
torchaudio_install_version='2.7.1'
|
||||
torch_test_version='2.7'
|
||||
cuda_test_version='12.6'
|
||||
pytorch_index_url='https://download.pytorch.org/whl/cu126'
|
||||
;;
|
||||
'2.8.0-cu126')
|
||||
torch_install_version='2.8.0'
|
||||
torchvision_install_version='0.23.0'
|
||||
torchaudio_install_version='2.8.0'
|
||||
torch_test_version='2.8'
|
||||
cuda_test_version='12.6'
|
||||
pytorch_index_url='https://download.pytorch.org/whl/cu126'
|
||||
;;
|
||||
'2.9.1-cu126')
|
||||
torch_install_version='2.9.1'
|
||||
torchvision_install_version='0.24.1'
|
||||
torchaudio_install_version='2.9.1'
|
||||
torch_test_version='2.9'
|
||||
cuda_test_version='12.6'
|
||||
pytorch_index_url='https://download.pytorch.org/whl/cu126'
|
||||
;;
|
||||
'2.10.0-cu126')
|
||||
torch_install_version='2.10.0'
|
||||
torchvision_install_version='0.25.0'
|
||||
torchaudio_install_version='2.10.0'
|
||||
torch_test_version='2.10'
|
||||
cuda_test_version='12.6'
|
||||
pytorch_index_url='https://download.pytorch.org/whl/cu126'
|
||||
;;
|
||||
'2.11.0-cu126')
|
||||
torch_install_version='2.11.0'
|
||||
torchvision_install_version='0.26.0'
|
||||
torchaudio_install_version='2.11.0'
|
||||
torch_test_version='2.11'
|
||||
cuda_test_version='12.6'
|
||||
pytorch_index_url='https://download.pytorch.org/whl/cu126'
|
||||
;;
|
||||
*)
|
||||
echo "Unsupported torch_preset: $selected_preset" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
{
|
||||
echo "SELECTED_TORCH_PRESET=$selected_preset"
|
||||
echo "TORCH_INSTALL_VERSION=$torch_install_version"
|
||||
echo "TORCHVISION_INSTALL_VERSION=$torchvision_install_version"
|
||||
echo "TORCHAUDIO_INSTALL_VERSION=$torchaudio_install_version"
|
||||
echo "TORCH_TEST_VERSION=$torch_test_version"
|
||||
echo "CUDA_TEST_VERSION=$cuda_test_version"
|
||||
echo "PYTORCH_INDEX_URL=$pytorch_index_url"
|
||||
echo "TRANSFORMERS_SOURCE=$transformers_source"
|
||||
echo "TRANSFORMERS_VERSION=$transformers_version"
|
||||
echo "TRANSFORMERS_REF=$transformers_ref"
|
||||
} >> "$GITHUB_ENV"
|
||||
|
||||
echo "Selected PyTorch preset: $selected_preset"
|
||||
echo "Resolved install tuple: torch==$torch_install_version torchvision==$torchvision_install_version torchaudio==$torchaudio_install_version"
|
||||
echo "Resolved test expectations: torch=$torch_test_version cuda=$cuda_test_version"
|
||||
echo "Resolved PyTorch index: $pytorch_index_url"
|
||||
echo "Resolved Transformers source: $transformers_source"
|
||||
echo "Resolved Transformers version: $transformers_version"
|
||||
echo "Resolved Transformers ref: $transformers_ref"
|
||||
|
||||
- name: Install CUTLASS
|
||||
run: |
|
||||
git clone --depth 1 --branch v3.5.1 https://github.com/NVIDIA/cutlass.git /opt/cutlass
|
||||
echo "CUTLASS installed at /opt/cutlass"
|
||||
ls -la /opt/cutlass/include/ | head -10
|
||||
|
||||
- name: Install PyTorch
|
||||
run: |
|
||||
pip install \
|
||||
torch=="$TORCH_INSTALL_VERSION" \
|
||||
torchvision=="$TORCHVISION_INSTALL_VERSION" \
|
||||
torchaudio=="$TORCHAUDIO_INSTALL_VERSION" \
|
||||
--index-url "$PYTORCH_INDEX_URL"
|
||||
|
||||
- name: Install Transformers
|
||||
run: |
|
||||
case "$TRANSFORMERS_SOURCE" in
|
||||
'pypi')
|
||||
pip install "transformers==$TRANSFORMERS_VERSION"
|
||||
;;
|
||||
'git')
|
||||
git clone --filter=blob:none https://github.com/huggingface/transformers /tmp/transformers
|
||||
cd /tmp/transformers
|
||||
git checkout "$TRANSFORMERS_REF"
|
||||
resolved_ref="$(git rev-parse HEAD)"
|
||||
echo "TRANSFORMERS_RESOLVED_REF=$resolved_ref" >> "$GITHUB_ENV"
|
||||
echo "Resolved Transformers git ref: $resolved_ref"
|
||||
pip install .
|
||||
;;
|
||||
*)
|
||||
echo "Unsupported TRANSFORMERS_SOURCE: $TRANSFORMERS_SOURCE" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
python -c "import transformers; print('transformers:', transformers.__version__, transformers)"
|
||||
|
||||
- name: Install Python dependencies
|
||||
run: |
|
||||
pip install --upgrade pip
|
||||
pip install -r requirements/requirements.txt
|
||||
pip install -r requirements/requirements-dev.txt
|
||||
pip install -r requirements/requirements-deepcompile.txt
|
||||
pip install pytest-timeout pytest-instafail
|
||||
|
||||
- name: Check environment
|
||||
run: |
|
||||
echo "=== Selected PyTorch Preset ==="
|
||||
echo "Preset: $SELECTED_TORCH_PRESET"
|
||||
echo "Install tuple: torch==$TORCH_INSTALL_VERSION torchvision==$TORCHVISION_INSTALL_VERSION torchaudio==$TORCHAUDIO_INSTALL_VERSION"
|
||||
echo "PyTorch index URL: $PYTORCH_INDEX_URL"
|
||||
echo "Expected test versions: torch=$TORCH_TEST_VERSION cuda=$CUDA_TEST_VERSION"
|
||||
echo "Transformers source: $TRANSFORMERS_SOURCE"
|
||||
echo "Transformers version: $TRANSFORMERS_VERSION"
|
||||
echo "Transformers ref: $TRANSFORMERS_REF"
|
||||
echo "Transformers resolved ref: ${TRANSFORMERS_RESOLVED_REF:-}"
|
||||
echo ""
|
||||
echo "=== GPU Information ==="
|
||||
nvidia-smi
|
||||
echo ""
|
||||
echo "=== CUDA Version ==="
|
||||
nvcc --version
|
||||
echo ""
|
||||
echo "=== Python/PyTorch Info ==="
|
||||
python --version
|
||||
python -c "import torch; print(f'PyTorch: {torch.__version__}')"
|
||||
python -c "import torch; print(f'CUDA available: {torch.cuda.is_available()}')"
|
||||
python -c "import torch; print(f'CUDA devices: {torch.cuda.device_count()}')"
|
||||
python -c "import torch; print(f'BF16 support: {torch.cuda.is_bf16_supported()}')"
|
||||
echo ""
|
||||
echo "=== CUTLASS ==="
|
||||
echo "CUTLASS_PATH: $CUTLASS_PATH"
|
||||
ls -la "$CUTLASS_PATH"/include/ | head -5
|
||||
|
||||
- name: Detect GPU architecture
|
||||
run: |
|
||||
python - <<'PY'
|
||||
import os
|
||||
import torch
|
||||
|
||||
torch.cuda.init()
|
||||
major, minor = torch.cuda.get_device_capability(0)
|
||||
arch = f"{major}.{minor}"
|
||||
gpu_count = torch.cuda.device_count()
|
||||
gpu_name = torch.cuda.get_device_name(0)
|
||||
|
||||
with open(os.environ["GITHUB_ENV"], "a", encoding="utf-8") as env_file:
|
||||
env_file.write(f"TORCH_CUDA_ARCH_LIST={arch}\n")
|
||||
env_file.write(f"GPU_COUNT={gpu_count}\n")
|
||||
|
||||
print(f"Detected GPU: {gpu_name}")
|
||||
print(f"Detected compute capability: {arch}")
|
||||
print(f"Detected GPU count: {gpu_count}")
|
||||
PY
|
||||
|
||||
- name: Install DeepSpeed
|
||||
run: |
|
||||
echo "Using TORCH_CUDA_ARCH_LIST=$TORCH_CUDA_ARCH_LIST"
|
||||
# Initialize CUDA before install so setup.py can detect NCCL version
|
||||
python -c "import torch; torch.cuda.init(); print(f'NCCL version: {torch.cuda.nccl.version()}')"
|
||||
# Use --no-build-isolation so setup.py can access pre-installed PyTorch
|
||||
pip install --no-build-isolation .[dev,1bit,autotuning,deepcompile]
|
||||
ds_report
|
||||
|
||||
- name: Reinstall selected Transformers
|
||||
run: |
|
||||
case "$TRANSFORMERS_SOURCE" in
|
||||
'pypi')
|
||||
pip install --no-deps --force-reinstall "transformers==$TRANSFORMERS_VERSION"
|
||||
;;
|
||||
'git')
|
||||
cd /tmp/transformers
|
||||
pip install --no-deps --force-reinstall .
|
||||
;;
|
||||
*)
|
||||
echo "Unsupported TRANSFORMERS_SOURCE: $TRANSFORMERS_SOURCE" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
python -c "import transformers; print('transformers:', transformers.__version__, transformers)"
|
||||
|
||||
- name: Python environment
|
||||
run: |
|
||||
pip list
|
||||
|
||||
- name: Unit tests (parallel)
|
||||
run: |
|
||||
echo "Running parallel tests with TORCH_CUDA_ARCH_LIST=$TORCH_CUDA_ARCH_LIST on $GPU_COUNT GPUs"
|
||||
cd tests
|
||||
# Skip tests requiring unavailable hardware or known issues:
|
||||
# - nvme checkpointing: no nvme device
|
||||
# - GDS tests: no GPUDirect Storage support
|
||||
# - launcher user_args: pdsh requires SSH server
|
||||
# - zenflow: Stage 3 tests have pre-existing bugs + CUDA/fork issues
|
||||
rm -rf /mnt/aio/pytest
|
||||
pytest --instafail --timeout 600 --forked -n 8 --basetemp=/mnt/aio/pytest unit/ \
|
||||
--ignore=unit/runtime/zero/test_nvme_checkpointing.py \
|
||||
--ignore=unit/ops/aio/test_gds.py \
|
||||
--ignore=unit/launcher/test_user_args.py \
|
||||
--ignore=unit/runtime/zenflow \
|
||||
--ignore=unit/ops/adam/test_zf_torch_adam.py \
|
||||
--torch_ver="$TORCH_TEST_VERSION" --cuda_ver="$CUDA_TEST_VERSION"
|
||||
|
||||
- name: Unit tests (sequential)
|
||||
run: |
|
||||
echo "Running sequential tests with TORCH_CUDA_ARCH_LIST=$TORCH_CUDA_ARCH_LIST on $GPU_COUNT GPUs"
|
||||
cd tests
|
||||
rm -rf /mnt/aio/pytest
|
||||
pytest --instafail --timeout 600 -m 'sequential' --basetemp=/mnt/aio/pytest unit/ \
|
||||
--ignore=unit/runtime/zero/test_nvme_checkpointing.py \
|
||||
--ignore=unit/ops/aio/test_gds.py \
|
||||
--ignore=unit/launcher/test_user_args.py \
|
||||
--ignore=unit/runtime/zenflow \
|
||||
--ignore=unit/ops/adam/test_zf_torch_adam.py \
|
||||
--ignore=unit/ops/deepspeed4science/test_DS4Sci_EvoformerAttention.py \
|
||||
--torch_ver="$TORCH_TEST_VERSION" --cuda_ver="$CUDA_TEST_VERSION"
|
||||
@@ -0,0 +1,113 @@
|
||||
################################################################################
|
||||
# DeepSpeed CI - AWS L40S GPU Tests (PyTorch Latest)
|
||||
#
|
||||
# Runs the same tests as modal-torch-latest.yml but on AWS self-hosted runners.
|
||||
# Uses 4x NVIDIA L40S GPUs on g6e.12xlarge instances.
|
||||
################################################################################
|
||||
|
||||
name: aws-torch-latest
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
|
||||
push:
|
||||
branches:
|
||||
- master
|
||||
|
||||
pull_request:
|
||||
branches:
|
||||
- master
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
check-paths:
|
||||
name: aws-torch-latest / check paths
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
should_run: ${{ steps.filter.outputs.run_tests }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: dorny/paths-filter@v3
|
||||
id: filter
|
||||
with:
|
||||
filters: |
|
||||
run_tests:
|
||||
- '**'
|
||||
- '!docs/**'
|
||||
- '!blogs/**'
|
||||
- '!deepspeed/inference/v2/**'
|
||||
- '!tests/unit/inference/v2/**'
|
||||
|
||||
unit-tests:
|
||||
name: aws-torch-latest / unit tests (v1)
|
||||
needs: check-paths
|
||||
if: needs.check-paths.outputs.should_run == 'true'
|
||||
runs-on: [self-hosted, gpu-ci, gpu-l40s, l40s-4gpu, aws]
|
||||
timeout-minutes: 60
|
||||
|
||||
container:
|
||||
image: nvidia/cuda:12.6.3-devel-ubuntu22.04
|
||||
options: --gpus all --shm-size "32G"
|
||||
|
||||
env:
|
||||
TORCH_VER: "2.7"
|
||||
CUDA_VER: "12.6"
|
||||
|
||||
steps:
|
||||
- name: Install system dependencies
|
||||
run: |
|
||||
apt-get update && apt-get install -y git git-lfs libaio-dev python3 python3-pip
|
||||
git lfs install
|
||||
ln -sf /usr/bin/python3 /usr/bin/python
|
||||
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
lfs: true
|
||||
|
||||
- name: Install PyTorch
|
||||
run: |
|
||||
pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu126
|
||||
|
||||
- name: Install Python dependencies
|
||||
run: |
|
||||
pip install --upgrade pip
|
||||
pip install -r requirements/requirements.txt
|
||||
pip install -r requirements/requirements-dev.txt
|
||||
pip install -r requirements/requirements-deepcompile.txt
|
||||
|
||||
- name: Check environment
|
||||
run: |
|
||||
echo "=== GPU Information ==="
|
||||
nvidia-smi
|
||||
echo ""
|
||||
echo "=== CUDA Version ==="
|
||||
nvcc --version
|
||||
echo ""
|
||||
echo "=== Python/PyTorch Info ==="
|
||||
python --version
|
||||
python -c "import torch; print(f'PyTorch: {torch.__version__}')"
|
||||
python -c "import torch; print(f'CUDA available: {torch.cuda.is_available()}')"
|
||||
python -c "import torch; print(f'CUDA devices: {torch.cuda.device_count()}')"
|
||||
python -c "import torch; print(f'BF16 support: {torch.cuda.is_bf16_supported()}')"
|
||||
|
||||
- name: Install DeepSpeed
|
||||
run: |
|
||||
# Initialize CUDA before install so setup.py can detect NCCL version
|
||||
python -c "import torch; torch.cuda.init(); print(f'NCCL version: {torch.cuda.nccl.version()}')"
|
||||
# Use --no-build-isolation so setup.py can access pre-installed PyTorch
|
||||
pip install --no-build-isolation .
|
||||
ds_report
|
||||
# Debug: Check captured torch_info values
|
||||
python -c "from deepspeed.git_version_info import torch_info; print(f'torch_info: {torch_info}')"
|
||||
|
||||
- name: Run unit tests
|
||||
run: |
|
||||
pytest -n 4 --forked --verbose tests/unit/v1/ --torch_ver=${{ env.TORCH_VER }} --cuda_ver=${{ env.CUDA_VER }}
|
||||
|
||||
- name: Run sequential unit tests
|
||||
run: |
|
||||
pytest --verbose -m 'sequential' tests/unit/v1/ --torch_ver=${{ env.TORCH_VER }} --cuda_ver=${{ env.CUDA_VER }}
|
||||
@@ -0,0 +1,243 @@
|
||||
name: cpu-torch-latest
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
torch_preset:
|
||||
description: PyTorch CPU preset to install for manual runs
|
||||
required: false
|
||||
default: '2.10.0-cpu'
|
||||
type: choice
|
||||
options:
|
||||
- '2.7.1-cpu'
|
||||
- '2.8.0-cpu'
|
||||
- '2.9.1-cpu'
|
||||
- '2.10.0-cpu'
|
||||
transformers_version:
|
||||
description: Hugging Face Transformers PyPI package version to install
|
||||
required: false
|
||||
default: '4.50.0'
|
||||
type: string
|
||||
transformers_source:
|
||||
description: Hugging Face Transformers source for manual runs
|
||||
required: false
|
||||
default: 'git'
|
||||
type: choice
|
||||
options:
|
||||
- 'pypi'
|
||||
- 'git'
|
||||
transformers_ref:
|
||||
description: Hugging Face Transformers git ref to install when source is git
|
||||
required: false
|
||||
default: 'main'
|
||||
type: string
|
||||
pull_request:
|
||||
merge_group:
|
||||
branches: [ master ]
|
||||
schedule:
|
||||
- cron: "0 0 * * *"
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
check-paths:
|
||||
name: cpu-torch-latest / check paths
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: read
|
||||
outputs:
|
||||
should_run: ${{ steps.non_pr.outputs.should_run || steps.filter.outputs.run_tests }}
|
||||
steps:
|
||||
- id: non_pr
|
||||
if: github.event_name != 'pull_request'
|
||||
run: echo "should_run=true" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- uses: actions/checkout@v4
|
||||
if: github.event_name == 'pull_request'
|
||||
|
||||
- uses: dorny/paths-filter@v3
|
||||
id: filter
|
||||
if: github.event_name == 'pull_request'
|
||||
with:
|
||||
predicate-quantifier: every
|
||||
filters: |
|
||||
run_tests:
|
||||
- '**'
|
||||
- '!docs/**'
|
||||
- '!blogs/**'
|
||||
- '!deepspeed/inference/v2/**'
|
||||
- '!tests/unit/inference/v2/**'
|
||||
|
||||
unit-tests:
|
||||
name: cpu-torch-latest / unit tests
|
||||
needs: check-paths
|
||||
if: ${{ !cancelled() && (needs.check-paths.result != 'success' || needs.check-paths.outputs.should_run == 'true') }}
|
||||
runs-on: ubuntu-24.04
|
||||
|
||||
env:
|
||||
DEFAULT_TORCH_PRESET: '2.10.0-cpu'
|
||||
DEFAULT_TRANSFORMERS_SOURCE: 'git'
|
||||
DEFAULT_TRANSFORMERS_VERSION: '4.50.0'
|
||||
DEFAULT_TRANSFORMERS_REF: 'main'
|
||||
|
||||
steps:
|
||||
- name: Fail if path filter failed
|
||||
if: needs.check-paths.result != 'success'
|
||||
run: exit 1
|
||||
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- id: setup-venv
|
||||
uses: ./.github/workflows/setup-venv
|
||||
|
||||
- name: Install system packages
|
||||
run: |
|
||||
sudo apt-get install -y numactl pdsh
|
||||
|
||||
- name: Resolve dependency inputs
|
||||
env:
|
||||
GITHUB_EVENT_NAME: ${{ github.event_name }}
|
||||
MANUAL_TORCH_PRESET: ${{ github.event.inputs.torch_preset || '' }}
|
||||
MANUAL_TRANSFORMERS_SOURCE: ${{ github.event.inputs.transformers_source || '' }}
|
||||
MANUAL_TRANSFORMERS_VERSION: ${{ github.event.inputs.transformers_version || '' }}
|
||||
MANUAL_TRANSFORMERS_REF: ${{ github.event.inputs.transformers_ref || '' }}
|
||||
run: |
|
||||
if [ "$GITHUB_EVENT_NAME" = 'workflow_dispatch' ] && [ -n "$MANUAL_TORCH_PRESET" ]; then
|
||||
selected_preset="$MANUAL_TORCH_PRESET"
|
||||
else
|
||||
selected_preset="$DEFAULT_TORCH_PRESET"
|
||||
fi
|
||||
|
||||
if [ "$GITHUB_EVENT_NAME" = 'workflow_dispatch' ] && [ -n "$MANUAL_TRANSFORMERS_SOURCE" ]; then
|
||||
transformers_source="$MANUAL_TRANSFORMERS_SOURCE"
|
||||
else
|
||||
transformers_source="$DEFAULT_TRANSFORMERS_SOURCE"
|
||||
fi
|
||||
|
||||
if [ "$GITHUB_EVENT_NAME" = 'workflow_dispatch' ] && [ -n "$MANUAL_TRANSFORMERS_VERSION" ]; then
|
||||
transformers_version="$MANUAL_TRANSFORMERS_VERSION"
|
||||
else
|
||||
transformers_version="$DEFAULT_TRANSFORMERS_VERSION"
|
||||
fi
|
||||
|
||||
if [ "$GITHUB_EVENT_NAME" = 'workflow_dispatch' ] && [ -n "$MANUAL_TRANSFORMERS_REF" ]; then
|
||||
transformers_ref="$MANUAL_TRANSFORMERS_REF"
|
||||
else
|
||||
transformers_ref="$DEFAULT_TRANSFORMERS_REF"
|
||||
fi
|
||||
|
||||
if [ "$transformers_source" = 'git' ] && [ -z "$transformers_ref" ]; then
|
||||
transformers_ref='main'
|
||||
fi
|
||||
|
||||
case "$selected_preset" in
|
||||
'2.7.1-cpu')
|
||||
torch_install_version='2.7.1'
|
||||
torchvision_install_version='0.22.1'
|
||||
torch_test_version='2.7'
|
||||
;;
|
||||
'2.8.0-cpu')
|
||||
torch_install_version='2.8.0'
|
||||
torchvision_install_version='0.23.0'
|
||||
torch_test_version='2.8'
|
||||
;;
|
||||
'2.9.1-cpu')
|
||||
torch_install_version='2.9.1'
|
||||
torchvision_install_version='0.24.1'
|
||||
torch_test_version='2.9'
|
||||
;;
|
||||
'2.10.0-cpu')
|
||||
torch_install_version='2.10.0'
|
||||
torchvision_install_version='0.25.0'
|
||||
torch_test_version='2.10'
|
||||
;;
|
||||
*)
|
||||
echo "Unsupported torch_preset: $selected_preset" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
{
|
||||
echo "SELECTED_TORCH_PRESET=$selected_preset"
|
||||
echo "TORCH_INSTALL_VERSION=$torch_install_version"
|
||||
echo "TORCHVISION_INSTALL_VERSION=$torchvision_install_version"
|
||||
echo "TORCH_TEST_VERSION=$torch_test_version"
|
||||
echo "PYTORCH_INDEX_URL=https://download.pytorch.org/whl/cpu"
|
||||
echo "TRANSFORMERS_SOURCE=$transformers_source"
|
||||
echo "TRANSFORMERS_VERSION=$transformers_version"
|
||||
echo "TRANSFORMERS_REF=$transformers_ref"
|
||||
} >> "$GITHUB_ENV"
|
||||
|
||||
echo "Selected PyTorch preset: $selected_preset"
|
||||
echo "Resolved install tuple: torch==$torch_install_version torchvision==$torchvision_install_version"
|
||||
echo "Resolved test expectation: torch=$torch_test_version"
|
||||
echo "Resolved Transformers source: $transformers_source"
|
||||
echo "Resolved Transformers version: $transformers_version"
|
||||
echo "Resolved Transformers ref: $transformers_ref"
|
||||
|
||||
- name: Install PyTorch
|
||||
run: |
|
||||
pip install \
|
||||
torch=="$TORCH_INSTALL_VERSION" \
|
||||
torchvision=="$TORCHVISION_INSTALL_VERSION" \
|
||||
--index-url "$PYTORCH_INDEX_URL"
|
||||
python -c "import torch; print('torch:', torch.__version__, torch)"
|
||||
python -c "import torch; print('CUDA available:', torch.cuda.is_available())"
|
||||
|
||||
- name: Install Transformers
|
||||
run: |
|
||||
case "$TRANSFORMERS_SOURCE" in
|
||||
'pypi')
|
||||
pip install "transformers==$TRANSFORMERS_VERSION"
|
||||
;;
|
||||
'git')
|
||||
git clone --filter=blob:none https://github.com/huggingface/transformers /tmp/transformers
|
||||
cd /tmp/transformers
|
||||
git checkout "$TRANSFORMERS_REF"
|
||||
resolved_ref="$(git rev-parse HEAD)"
|
||||
echo "TRANSFORMERS_RESOLVED_REF=$resolved_ref" >> "$GITHUB_ENV"
|
||||
echo "Resolved Transformers git ref: $resolved_ref"
|
||||
pip install .
|
||||
;;
|
||||
*)
|
||||
echo "Unsupported TRANSFORMERS_SOURCE: $TRANSFORMERS_SOURCE" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
python -c "import transformers; print('transformers:', transformers.__version__, transformers)"
|
||||
|
||||
- name: Install deepspeed
|
||||
run: |
|
||||
pip install .[dev,autotuning]
|
||||
ds_report
|
||||
|
||||
- name: Reinstall selected Transformers
|
||||
run: |
|
||||
case "$TRANSFORMERS_SOURCE" in
|
||||
'pypi')
|
||||
pip install --no-deps --force-reinstall "transformers==$TRANSFORMERS_VERSION"
|
||||
;;
|
||||
'git')
|
||||
cd /tmp/transformers
|
||||
pip install --no-deps --force-reinstall .
|
||||
;;
|
||||
*)
|
||||
echo "Unsupported TRANSFORMERS_SOURCE: $TRANSFORMERS_SOURCE" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
python -c "import transformers; print('transformers:', transformers.__version__, transformers)"
|
||||
|
||||
- name: Python environment
|
||||
run: |
|
||||
pip list
|
||||
|
||||
- name: Unit tests
|
||||
run: |
|
||||
unset TORCH_CUDA_ARCH_LIST # only jit compile for current arch
|
||||
cd tests
|
||||
HF_HOME=/tmp/hf_home/ pytest $PYTEST_OPTS --forked -n 4 unit/ --torch_ver="$TORCH_TEST_VERSION"
|
||||
HF_HOME=/tmp/hf_home/ pytest $PYTEST_OPTS --forked -m 'sequential' unit/ --torch_ver="$TORCH_TEST_VERSION"
|
||||
@@ -0,0 +1,585 @@
|
||||
# Copyright (c) DeepSpeed Team.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
# DeepSpeed Team
|
||||
|
||||
name: DCO / required
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
branches:
|
||||
- master
|
||||
merge_group:
|
||||
branches:
|
||||
- master
|
||||
|
||||
permissions:
|
||||
checks: read
|
||||
contents: read
|
||||
pull-requests: read
|
||||
|
||||
jobs:
|
||||
dco_required:
|
||||
name: DCO / required
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Validate commit signoffs
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
python - <<'PY'
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
|
||||
# GitHub App ID for https://github.com/apps/dco.
|
||||
PROBOT_DCO_APP_ID = 1861
|
||||
|
||||
|
||||
def load_event(path):
|
||||
with open(path, encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
|
||||
|
||||
def fail(message):
|
||||
print(f"::error::{message}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def extract_pr_numbers_from_refs(*refs):
|
||||
numbers = []
|
||||
text = "\n".join(value for value in refs if value)
|
||||
for match in re.finditer(r"(?:^|[/-])pr-(\d+)(?=$|[/-])", text):
|
||||
number = int(match.group(1))
|
||||
if number not in numbers:
|
||||
numbers.append(number)
|
||||
return numbers
|
||||
|
||||
|
||||
def extract_pull_request_number_from_refs(*refs):
|
||||
text = "\n".join(value for value in refs if value)
|
||||
match = re.search(r"(?:^|/)pull/(\d+)(?=/|$)", text)
|
||||
if match:
|
||||
return int(match.group(1))
|
||||
return None
|
||||
|
||||
|
||||
def discover_pr_numbers(event_name, event, github_ref):
|
||||
if event_name == "pull_request":
|
||||
number = event.get("number")
|
||||
if number is not None:
|
||||
try:
|
||||
return [int(number)]
|
||||
except (TypeError, ValueError):
|
||||
fail(f"pull_request event had non-integer number: {number!r}")
|
||||
|
||||
try:
|
||||
return [int(event["pull_request"]["number"])]
|
||||
except (KeyError, TypeError, ValueError):
|
||||
ref_number = extract_pull_request_number_from_refs(
|
||||
os.environ.get("GITHUB_REF"),
|
||||
github_ref,
|
||||
event.get("ref"),
|
||||
)
|
||||
if ref_number is not None:
|
||||
return [ref_number]
|
||||
fail("pull_request event did not include a pull request number")
|
||||
|
||||
if event_name == "merge_group":
|
||||
merge_group = event.get("merge_group", {})
|
||||
numbers = extract_pr_numbers_from_refs(
|
||||
merge_group.get("head_ref"),
|
||||
os.environ.get("GITHUB_REF_NAME"),
|
||||
github_ref,
|
||||
event.get("ref"),
|
||||
)
|
||||
if numbers:
|
||||
return numbers
|
||||
|
||||
fail(
|
||||
"merge_group event did not include a parseable PR number. "
|
||||
f"head_ref={merge_group.get('head_ref')!r} "
|
||||
f"GITHUB_REF_NAME={os.environ.get('GITHUB_REF_NAME')!r} "
|
||||
f"GITHUB_REF={github_ref!r}"
|
||||
)
|
||||
|
||||
fail(f"Unsupported event for DCO check: {event_name}")
|
||||
|
||||
|
||||
def graphql_request(query, variables, token):
|
||||
body = json.dumps({"query": query, "variables": variables}).encode("utf-8")
|
||||
req = urllib.request.Request(
|
||||
"https://api.github.com/graphql",
|
||||
data=body,
|
||||
headers={
|
||||
"Accept": "application/vnd.github+json",
|
||||
"Authorization": f"Bearer {token}",
|
||||
"Content-Type": "application/json",
|
||||
"X-GitHub-Api-Version": "2022-11-28",
|
||||
"User-Agent": "deepspeed-dco-check",
|
||||
},
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=30) as response:
|
||||
payload = json.loads(response.read().decode("utf-8"))
|
||||
except urllib.error.HTTPError as exc:
|
||||
detail = exc.read().decode("utf-8", errors="replace")
|
||||
fail(f"GitHub GraphQL request failed: HTTP {exc.code} {detail}")
|
||||
except urllib.error.URLError as exc:
|
||||
fail(f"GitHub GraphQL request failed: {exc}")
|
||||
|
||||
if payload.get("errors"):
|
||||
fail(f"GitHub GraphQL returned errors: {payload['errors']}")
|
||||
return payload["data"]
|
||||
|
||||
|
||||
def rest_request(path, token, fatal=True):
|
||||
url = f"https://api.github.com/repos/{os.environ['GITHUB_REPOSITORY']}{path}"
|
||||
items = []
|
||||
while url:
|
||||
req = urllib.request.Request(
|
||||
url,
|
||||
headers={
|
||||
"Accept": "application/vnd.github+json",
|
||||
"Authorization": f"Bearer {token}",
|
||||
"X-GitHub-Api-Version": "2022-11-28",
|
||||
"User-Agent": "deepspeed-dco-check",
|
||||
},
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=30) as response:
|
||||
data = json.loads(response.read().decode("utf-8"))
|
||||
link = response.headers.get("Link", "")
|
||||
except urllib.error.HTTPError as exc:
|
||||
detail = exc.read().decode("utf-8", errors="replace")
|
||||
message = (
|
||||
f"GitHub REST request failed for {path}: "
|
||||
f"HTTP {exc.code} {detail}"
|
||||
)
|
||||
if fatal:
|
||||
fail(message)
|
||||
print(f"::warning::{message}")
|
||||
return None
|
||||
except urllib.error.URLError as exc:
|
||||
message = f"GitHub REST request failed for {path}: {exc}"
|
||||
if fatal:
|
||||
fail(message)
|
||||
print(f"::warning::{message}")
|
||||
return None
|
||||
|
||||
if isinstance(data, list):
|
||||
items.extend(data)
|
||||
else:
|
||||
commits = data.get("commits")
|
||||
if isinstance(commits, list):
|
||||
items.extend(commits)
|
||||
else:
|
||||
return data
|
||||
|
||||
next_url = None
|
||||
for part in link.split(","):
|
||||
if 'rel="next"' in part:
|
||||
next_url = part[part.find("<") + 1:part.find(">")]
|
||||
break
|
||||
url = next_url
|
||||
return items
|
||||
|
||||
|
||||
def fetch_compare_commits(base_sha, head_sha, token):
|
||||
base = urllib.parse.quote(base_sha, safe="")
|
||||
head = urllib.parse.quote(head_sha, safe="")
|
||||
return rest_request(f"/compare/{base}...{head}?per_page=100", token)
|
||||
|
||||
|
||||
def fetch_commit(sha, token):
|
||||
ref = urllib.parse.quote(sha, safe="")
|
||||
return rest_request(f"/commits/{ref}", token, fatal=False)
|
||||
|
||||
|
||||
def has_successful_probot_dco(head_sha, token):
|
||||
if not head_sha:
|
||||
return False
|
||||
|
||||
ref = urllib.parse.quote(head_sha, safe="")
|
||||
payload = rest_request(
|
||||
f"/commits/{ref}/check-runs?check_name=DCO&filter=latest",
|
||||
token,
|
||||
fatal=False,
|
||||
)
|
||||
if not payload:
|
||||
return False
|
||||
|
||||
for check_run in payload.get("check_runs", []):
|
||||
app = check_run.get("app") or {}
|
||||
is_probot_dco = app.get("slug") == "dco" or app.get("id") == PROBOT_DCO_APP_ID
|
||||
if (
|
||||
check_run.get("name") == "DCO"
|
||||
and is_probot_dco
|
||||
and check_run.get("status") == "completed"
|
||||
and check_run.get("conclusion") == "success"
|
||||
):
|
||||
print(
|
||||
"Found successful Probot DCO check for PR head "
|
||||
f"{head_sha}; accepting Probot result."
|
||||
)
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def fetch_pr_commits(owner, repo, number, token):
|
||||
query = """
|
||||
query($owner: String!, $repo: String!, $number: Int!, $cursor: String) {
|
||||
repository(owner: $owner, name: $repo) {
|
||||
PULL_REQUEST_FIELD(number: $number) {
|
||||
baseRefName
|
||||
baseRepository {
|
||||
nameWithOwner
|
||||
}
|
||||
headRefOid
|
||||
commits(first: 100, after: $cursor) {
|
||||
pageInfo {
|
||||
hasNextPage
|
||||
endCursor
|
||||
}
|
||||
nodes {
|
||||
commit {
|
||||
oid
|
||||
message
|
||||
author {
|
||||
name
|
||||
email
|
||||
user {
|
||||
login
|
||||
}
|
||||
}
|
||||
committer {
|
||||
name
|
||||
email
|
||||
user {
|
||||
login
|
||||
}
|
||||
}
|
||||
parents(first: 2) {
|
||||
totalCount
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
""".replace("PULL_REQUEST_FIELD", "pull" + "Request")
|
||||
cursor = None
|
||||
commits = []
|
||||
base_ref = None
|
||||
base_repo = None
|
||||
head_sha = None
|
||||
|
||||
while True:
|
||||
data = graphql_request(
|
||||
query,
|
||||
{"owner": owner, "repo": repo, "number": number, "cursor": cursor},
|
||||
token,
|
||||
)
|
||||
pull_request = data["repository"]["pull" + "Request"]
|
||||
if pull_request is None:
|
||||
fail(f"PR #{number} was not found")
|
||||
|
||||
base_ref = pull_request["baseRefName"]
|
||||
base_repo = pull_request["baseRepository"]["nameWithOwner"]
|
||||
head_sha = pull_request["headRefOid"]
|
||||
connection = pull_request["commits"]
|
||||
commits.extend(connection["nodes"])
|
||||
|
||||
page_info = connection["pageInfo"]
|
||||
if not page_info["hasNextPage"]:
|
||||
break
|
||||
cursor = page_info["endCursor"]
|
||||
if not cursor:
|
||||
fail(f"PR #{number} pagination did not return an end cursor")
|
||||
|
||||
return {
|
||||
"base_ref": base_ref,
|
||||
"base_repo": base_repo,
|
||||
"head_sha": head_sha,
|
||||
"commits": commits,
|
||||
}
|
||||
|
||||
|
||||
def has_signed_off_by(message):
|
||||
return bool(
|
||||
re.search(
|
||||
r"^Signed-off-by:\s+\S.*$",
|
||||
message,
|
||||
flags=re.MULTILINE,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def commit_subject(message):
|
||||
return message.splitlines()[0] if message.splitlines() else "(empty subject)"
|
||||
|
||||
|
||||
def actor_from_graphql(git_actor):
|
||||
git_actor = git_actor or {}
|
||||
user = git_actor.get("user") or {}
|
||||
return {
|
||||
"login": user.get("login") or "",
|
||||
"type": "",
|
||||
"name": git_actor.get("name") or "",
|
||||
"email": git_actor.get("email") or "",
|
||||
}
|
||||
|
||||
|
||||
def actor_from_rest(api_actor, git_actor):
|
||||
api_actor = api_actor or {}
|
||||
git_actor = git_actor or {}
|
||||
return {
|
||||
"login": api_actor.get("login") or "",
|
||||
"type": api_actor.get("type") or "",
|
||||
"name": git_actor.get("name") or "",
|
||||
"email": git_actor.get("email") or "",
|
||||
}
|
||||
|
||||
|
||||
def is_trusted_bot_actor(actor):
|
||||
actor = actor or {}
|
||||
return str(actor.get("type", "")).lower() == "bot"
|
||||
|
||||
|
||||
def has_bot_marker(actor):
|
||||
actor = actor or {}
|
||||
for key in ("login", "name", "email"):
|
||||
value = str(actor.get(key, "")).lower()
|
||||
if "[bot]" in value:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def actor_label(actor):
|
||||
actor = actor or {}
|
||||
return (
|
||||
actor.get("login")
|
||||
or actor.get("name")
|
||||
or actor.get("email")
|
||||
or "unknown actor"
|
||||
)
|
||||
|
||||
|
||||
def is_verified_bot_authored(record, token):
|
||||
author = record.get("author") or {}
|
||||
if is_trusted_bot_actor(author):
|
||||
return True
|
||||
|
||||
if not token or not has_bot_marker(author):
|
||||
return False
|
||||
|
||||
commit = fetch_commit(record["sha"], token)
|
||||
if not commit:
|
||||
return False
|
||||
|
||||
api_author = commit.get("author") or {}
|
||||
if str(api_author.get("type", "")).lower() != "bot":
|
||||
return False
|
||||
|
||||
author["login"] = api_author.get("login") or author.get("login") or ""
|
||||
author["type"] = api_author.get("type") or author.get("type") or ""
|
||||
return True
|
||||
|
||||
|
||||
def validate_records(records, seen, token=None, skip_sha=None):
|
||||
failures = []
|
||||
checked = []
|
||||
skipped = []
|
||||
accepted = []
|
||||
for record in records:
|
||||
oid = record["sha"]
|
||||
if oid in seen:
|
||||
continue
|
||||
seen.add(oid)
|
||||
|
||||
if skip_sha and oid == skip_sha:
|
||||
skipped.append(oid)
|
||||
print(f"Skipping merge group head commit {oid}")
|
||||
continue
|
||||
|
||||
if record["parent_count"] > 1:
|
||||
skipped.append(oid)
|
||||
print(f"Skipping merge commit {oid}")
|
||||
continue
|
||||
|
||||
if is_verified_bot_authored(record, token):
|
||||
skipped.append(oid)
|
||||
print(
|
||||
"Skipping bot-authored commit "
|
||||
f"{oid} ({actor_label(record.get('author'))})"
|
||||
)
|
||||
continue
|
||||
|
||||
checked.append(oid)
|
||||
message = record.get("message") or ""
|
||||
if not has_signed_off_by(message):
|
||||
failures.append({"sha": oid, "subject": commit_subject(message)})
|
||||
|
||||
return {
|
||||
"checked": checked,
|
||||
"skipped": skipped,
|
||||
"accepted": accepted,
|
||||
"failures": failures,
|
||||
}
|
||||
|
||||
|
||||
def validate_pr(owner, repo, number, token, seen):
|
||||
print(f"Validating DCO trailers for PR #{number}")
|
||||
pull_request = fetch_pr_commits(owner, repo, number, token)
|
||||
expected_base = f"{owner}/{repo}"
|
||||
if (
|
||||
pull_request["base_repo"] != expected_base
|
||||
or pull_request["base_ref"] != "master"
|
||||
):
|
||||
fail(
|
||||
f"PR #{number} targets "
|
||||
f"{pull_request['base_repo']}:{pull_request['base_ref']}, "
|
||||
f"expected {expected_base}:master"
|
||||
)
|
||||
|
||||
records = []
|
||||
for node in pull_request["commits"]:
|
||||
commit = node["commit"]
|
||||
records.append(
|
||||
{
|
||||
"sha": commit["oid"],
|
||||
"message": commit.get("message") or "",
|
||||
"author": actor_from_graphql(commit.get("author")),
|
||||
"committer": actor_from_graphql(commit.get("committer")),
|
||||
"parent_count": commit["parents"]["totalCount"],
|
||||
}
|
||||
)
|
||||
|
||||
if not records:
|
||||
return {
|
||||
"checked": [],
|
||||
"skipped": [],
|
||||
"accepted": [],
|
||||
"failures": [
|
||||
{
|
||||
"sha": f"PR #{number}",
|
||||
"subject": "no commits returned by pull request commits API",
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
if has_successful_probot_dco(pull_request["head_sha"], token):
|
||||
accepted = []
|
||||
for record in records:
|
||||
oid = record["sha"]
|
||||
if oid not in seen:
|
||||
seen.add(oid)
|
||||
accepted.append(oid)
|
||||
return {
|
||||
"checked": [],
|
||||
"skipped": [],
|
||||
"accepted": accepted,
|
||||
"failures": [],
|
||||
}
|
||||
|
||||
return validate_records(records, seen, token=token)
|
||||
|
||||
|
||||
def verify_merge_group_range_coverage(event, token, seen):
|
||||
merge_group = event.get("merge_group", {})
|
||||
base_sha = merge_group.get("base_sha")
|
||||
head_sha = merge_group.get("head_sha") or os.environ.get("GITHUB_SHA")
|
||||
if not base_sha or not head_sha:
|
||||
fail(
|
||||
"merge_group event did not include base_sha and head_sha. "
|
||||
f"base_sha={base_sha!r} head_sha={head_sha!r} "
|
||||
f"GITHUB_SHA={os.environ.get('GITHUB_SHA')!r}"
|
||||
)
|
||||
|
||||
print(f"Checking merge group range coverage {base_sha}...{head_sha}")
|
||||
commits = fetch_compare_commits(base_sha, head_sha, token)
|
||||
records = []
|
||||
for commit in commits:
|
||||
git_commit = commit.get("commit", {}) or {}
|
||||
records.append(
|
||||
{
|
||||
"sha": commit.get("sha", ""),
|
||||
"message": git_commit.get("message", "") or "",
|
||||
"author": actor_from_rest(
|
||||
commit.get("author"),
|
||||
git_commit.get("author"),
|
||||
),
|
||||
"committer": actor_from_rest(
|
||||
commit.get("committer"),
|
||||
git_commit.get("committer"),
|
||||
),
|
||||
"parent_count": len(commit.get("parents", [])),
|
||||
}
|
||||
)
|
||||
|
||||
if not commits:
|
||||
fail(f"merge_group compare range {base_sha}...{head_sha} returned no commits")
|
||||
|
||||
return validate_records(records, seen, token=token, skip_sha=head_sha)
|
||||
|
||||
|
||||
def main():
|
||||
repository = os.environ["GITHUB_REPOSITORY"]
|
||||
owner, repo = repository.split("/", 1)
|
||||
event_name = os.environ["GITHUB_EVENT_NAME"]
|
||||
github_ref = os.environ.get("GITHUB_REF")
|
||||
token = os.environ["GITHUB_TOKEN"]
|
||||
event = load_event(os.environ["GITHUB_EVENT_PATH"])
|
||||
|
||||
pull_numbers = discover_pr_numbers(event_name, event, github_ref)
|
||||
failures = []
|
||||
checked = set()
|
||||
skipped = set()
|
||||
accepted = set()
|
||||
seen = set()
|
||||
|
||||
for number in sorted(pull_numbers):
|
||||
result = validate_pr(owner, repo, number, token, seen)
|
||||
failures.extend(result["failures"])
|
||||
checked.update(result["checked"])
|
||||
skipped.update(result["skipped"])
|
||||
accepted.update(result.get("accepted", []))
|
||||
|
||||
if event_name == "merge_group":
|
||||
result = verify_merge_group_range_coverage(
|
||||
event,
|
||||
token,
|
||||
seen,
|
||||
)
|
||||
failures.extend(result["failures"])
|
||||
checked.update(result["checked"])
|
||||
skipped.update(result["skipped"])
|
||||
accepted.update(result.get("accepted", []))
|
||||
|
||||
if failures:
|
||||
for failure in failures:
|
||||
print(f"::error::{failure['sha']}: {failure['subject']}")
|
||||
fail(
|
||||
f"{len(failures)} commit(s) are missing a Signed-off-by trailer."
|
||||
)
|
||||
|
||||
print(
|
||||
"DCO validation passed for "
|
||||
f"{len(pull_numbers)} pull request(s): "
|
||||
f"checked {len(checked)} commit(s), "
|
||||
f"accepted {len(accepted)} commit(s) via Probot DCO, "
|
||||
f"skipped {len(skipped)} merge, bot, or synthetic commit(s)."
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
PY
|
||||
@@ -0,0 +1,41 @@
|
||||
name: Formatting
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
pull_request:
|
||||
branches:
|
||||
'**'
|
||||
merge_group:
|
||||
branches: [ master ]
|
||||
schedule:
|
||||
- cron: "0 0 * * *"
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
|
||||
# formatting and basic install on cpu-only machine
|
||||
unit-tests:
|
||||
name: formatting / formatting checks
|
||||
runs-on: ubuntu-22.04
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: environment
|
||||
run: |
|
||||
which python
|
||||
python --version
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
# Previously we would do pip install .[dev] but this is causing out of
|
||||
# space errors start with torch 2.1.0 release
|
||||
grep -E "clang-format|pre-commit" requirements/requirements-dev.txt | xargs pip install
|
||||
|
||||
- name: Formatting checks
|
||||
run: |
|
||||
pip show pre-commit clang-format
|
||||
pre-commit run --all-files
|
||||
@@ -0,0 +1,88 @@
|
||||
name: hpu-gaudi2-nightly
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
schedule:
|
||||
- cron: "0 0 * * *"
|
||||
pull_request:
|
||||
paths:
|
||||
- ".github/workflows/hpu-gaudi2-nightly.yml"
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
|
||||
jobs:
|
||||
unit-tests:
|
||||
name: hpu-gaudi2-nightly / unit tests
|
||||
# The type of runner that the job will run on
|
||||
runs-on: [self-hosted, intel, gaudi2]
|
||||
container:
|
||||
image: vault.habana.ai/gaudi-docker/1.21.0/ubuntu22.04/habanalabs/pytorch-installer-2.6.0:latest
|
||||
ports:
|
||||
- 80
|
||||
options: --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice
|
||||
|
||||
env:
|
||||
PT_HPU_LAZY_MODE: 0
|
||||
TORCHINDUCTOR_COMPILE_THREADS: 1
|
||||
TEST_LIST: |
|
||||
test_adamw.py
|
||||
test_bf16.py
|
||||
test_ds_config_dict.py
|
||||
test_dynamic_loss_scale.py
|
||||
test_latest_checkpoint.py
|
||||
test_moe_checkpoint.py
|
||||
test_multi_output_model.py
|
||||
test_other_optimizer.py
|
||||
test_pipe.py
|
||||
test_pipeline.py
|
||||
test_universal_checkpoint.py
|
||||
test_zero_context_return.py
|
||||
test_zero_leaf_module.py
|
||||
test_zero_offloadpp.py
|
||||
test_zero_tiled.py
|
||||
test_autotp_training.py
|
||||
test_ulysses.py
|
||||
|
||||
# Steps represent a sequence of tasks that will be executed as part of the job
|
||||
steps:
|
||||
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Check container state
|
||||
run: |
|
||||
ldd --version
|
||||
hl-smi -L
|
||||
python -c "import torch; print('torch:', torch.__version__, torch)"
|
||||
python -c "import torch; print('CUDA available:', torch.cuda.is_available())"
|
||||
|
||||
- name: Install transformers
|
||||
run: |
|
||||
git clone https://github.com/huggingface/transformers
|
||||
cd transformers
|
||||
git rev-parse --short HEAD
|
||||
pip install .
|
||||
|
||||
- name: Install deepspeed
|
||||
run: |
|
||||
pip install .[dev,autotuning]
|
||||
ds_report
|
||||
|
||||
- name: Python environment
|
||||
run: |
|
||||
pip list
|
||||
|
||||
- name: Unit tests
|
||||
run: |
|
||||
unset TORCH_CUDA_ARCH_LIST # only jit compile for current arch
|
||||
cd tests
|
||||
export PT_HPU_LAZY_MODE=${PT_HPU_LAZY_MODE}
|
||||
export TORCHINDUCTOR_COMPILE_THREADS=${TORCHINDUCTOR_COMPILE_THREADS}
|
||||
TEST_LIST=$(echo "$TEST_LIST" | awk 'NF{printf "%s%s", (NR>1 ? " or " : ""), $0} END{if (NR>1) print ""}')
|
||||
echo "TEST_LIST ${TEST_LIST}"
|
||||
pytest --verbose unit/ -k "${TEST_LIST}"
|
||||
@@ -0,0 +1,139 @@
|
||||
name: hpu-gaudi2
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
schedule:
|
||||
- cron: "0 0 * * *"
|
||||
pull_request:
|
||||
paths:
|
||||
- ".github/workflows/hpu-gaudi2.yml"
|
||||
- "accelerator/hpu_accelerator.py"
|
||||
- "op_builder/hpu/**"
|
||||
- "deepspeed/runtime/engine.py"
|
||||
- "deepspeed/runtime/bf16_optimizer.py"
|
||||
- "deepspeed/runtime/zero/stage_1_and_2.py"
|
||||
- "deepspeed/runtime/zero/stage3.py"
|
||||
- "deepspeed/runtime/zero/partition_parameters.py"
|
||||
- "deepspeed/runtime/zero/partitioned_param_coordinator.py"
|
||||
- "deepspeed/runtime/zero/parameter_offload.py"
|
||||
- "deepspeed/runtime/pipe/engine.py"
|
||||
- "deepspeed/runtime/utils.py"
|
||||
- "deepspeed/inference/engine.py"
|
||||
- "deepspeed/module_inject/auto_tp.py"
|
||||
- "deepspeed/module_inject/replace_module.py"
|
||||
- "deepspeed/module_inject/load_checkpoint.py"
|
||||
- "deepspeed/module_inject/inject.py"
|
||||
- "deepspeed/ops/transformer/**"
|
||||
- "deepspeed/ops/adam/**"
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
|
||||
jobs:
|
||||
unit-tests:
|
||||
name: hpu-gaudi2 / unit tests
|
||||
# The type of runner that the job will run on
|
||||
runs-on: [self-hosted, intel, gaudi2]
|
||||
container:
|
||||
image: vault.habana.ai/gaudi-docker/1.21.0/ubuntu22.04/habanalabs/pytorch-installer-2.6.0:latest
|
||||
ports:
|
||||
- 80
|
||||
options: --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice
|
||||
|
||||
env:
|
||||
PT_HPU_LAZY_MODE: 0
|
||||
TORCHINDUCTOR_COMPILE_THREADS: 1
|
||||
TEST_LIST: |
|
||||
test_accelerator.py
|
||||
test_autotuning.py
|
||||
test_compression.py
|
||||
test_dist.py
|
||||
test_elastic.py
|
||||
test_ds_arguments.py
|
||||
test_run.py
|
||||
test_multinode_runner.py
|
||||
test_moe_tp.py
|
||||
test_monitor.py
|
||||
(test_zero_optimizer.py and (TestSaveTensorClone or TestZeRONonDistributed))
|
||||
(test_latest_checkpoint.py and test_missing_latest)
|
||||
test_reshape_checkpoint.py
|
||||
test_shared_weights.py
|
||||
test_sparse.py
|
||||
test_tag_validation.py
|
||||
test_pipe_module.py
|
||||
(test_flops_profiler.py and test_flops_profiler_in_inference)
|
||||
test_get_optim_files.py
|
||||
test_groups.py
|
||||
test_partition_balanced.py
|
||||
(test_adamw.py and TestAdamConfigs)
|
||||
test_coalesced_collectives.py
|
||||
test_activation_checkpointing_non_reentrant.py
|
||||
test_activation_checkpointing.py
|
||||
test_data.py
|
||||
(test_ds_config_dict.py and (TestBasicConfig or TestBatchConfig))
|
||||
test_ds_config_model.py
|
||||
test_mup_optimizers.py
|
||||
(test_pld.py and test_pld_schedule)
|
||||
test_runtime_utils.py
|
||||
test_pipe_schedule.py
|
||||
test_topology.py
|
||||
(test_ds_initialize.py and (TestClientOptimizer or TestClientLrScheduler))
|
||||
test_csr.py
|
||||
(test_fp16.py and (TestZeroEmptyGrad or TestZeroAllowUntestedOptimizer))
|
||||
(test_bf16.py and TestZeroDtypeCocktail)
|
||||
test_partition.py
|
||||
test_ignore_unused_parameters.py
|
||||
test_zero_config.py
|
||||
test_zero_context_ancestry.py
|
||||
(test_zero_context.py and not TestSerialContext)
|
||||
test_zero_dynamic_class.py
|
||||
test_zero_nesting_init.py
|
||||
test_zeropp.py
|
||||
(test_zero.py and (TestZero3ParamPartitioningLargeParam or TestZero3ParamPartitioningLargeParam))
|
||||
(test_linear.py and (TestLoRALinear or TestBasicLinear))
|
||||
(test_ctx.py and TestEngine)
|
||||
|
||||
# Steps represent a sequence of tasks that will be executed as part of the job
|
||||
steps:
|
||||
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Check container state
|
||||
run: |
|
||||
ldd --version
|
||||
hl-smi -L
|
||||
python -c "import torch; print('torch:', torch.__version__, torch)"
|
||||
python -c "import torch; print('CUDA available:', torch.cuda.is_available())"
|
||||
|
||||
- name: Install transformers
|
||||
run: |
|
||||
git clone https://github.com/huggingface/transformers
|
||||
cd transformers
|
||||
# if needed switch to the last known good SHA until transformers@master is fixed
|
||||
# git checkout 981c276
|
||||
git rev-parse --short HEAD
|
||||
pip install .
|
||||
|
||||
- name: Install deepspeed
|
||||
run: |
|
||||
pip install .[dev,autotuning]
|
||||
ds_report
|
||||
|
||||
- name: Python environment
|
||||
run: |
|
||||
pip list
|
||||
|
||||
- name: Unit tests
|
||||
run: |
|
||||
unset TORCH_CUDA_ARCH_LIST # only jit compile for current arch
|
||||
cd tests
|
||||
export PT_HPU_LAZY_MODE=${PT_HPU_LAZY_MODE}
|
||||
export TORCHINDUCTOR_COMPILE_THREADS=${TORCHINDUCTOR_COMPILE_THREADS}
|
||||
TEST_LIST=$(echo "$TEST_LIST" | awk 'NF{printf "%s%s", (NR>1 ? " or " : ""), $0} END{if (NR>1) print ""}')
|
||||
echo "TEST_LIST ${TEST_LIST}"
|
||||
pytest --verbose unit/ -k "${TEST_LIST}"
|
||||
@@ -0,0 +1,104 @@
|
||||
name: modal-accelerate
|
||||
|
||||
# This CI is running on modal.com's GPUs.
|
||||
#
|
||||
# It's set up here on github actions and then the cloned repo is sent to modal and everything
|
||||
# happens on their hw - see ci/accelerate.py for where the actual vm is loaded, updated and the tests are
|
||||
# run.
|
||||
#
|
||||
# Both files are annotated to what's important and how one might change or update things if needed.
|
||||
#
|
||||
# Note that since this is a Required job we can't use `on.push.path` file filter - we are using
|
||||
# collect-tests job to do the filtering for us so that the job can be skipped and satisfy the
|
||||
# Required status for PRs to pass.
|
||||
#
|
||||
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
push:
|
||||
branches:
|
||||
- master
|
||||
|
||||
# you have to switch to `pull_request` if you need to change the CI job's python script,
|
||||
# otherwise GH will use a master version of the CI files, ignoring the modifications in the PR -
|
||||
# the other way is to use modal cli to test this job from one's host - it'd require setting up
|
||||
# modal secrets
|
||||
# pull_request:
|
||||
pull_request_target:
|
||||
paths-ignore:
|
||||
- 'docs/**'
|
||||
- 'blogs/**'
|
||||
- 'deepspeed/inference/v2/**'
|
||||
- 'tests/unit/inference/v2/**'
|
||||
types: [review_requested, ready_for_review, synchronize]
|
||||
branches:
|
||||
- master
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
collect-tests:
|
||||
name: modal-accelerate / collect tests
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: read
|
||||
outputs:
|
||||
deepspeed: ${{ steps.filter.outputs.deepspeed }}
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
lfs: true
|
||||
|
||||
- name: Filter changed files
|
||||
uses: dorny/paths-filter@v2
|
||||
id: filter
|
||||
with:
|
||||
token: ${{ secrets.GITHUB_TOKEN }}
|
||||
filters: |
|
||||
deepspeed:
|
||||
- 'deepspeed/**'
|
||||
- '.github/workflows/modal*.yml'
|
||||
- 'ci/**'
|
||||
- 'tests/unit/**'
|
||||
- 'csrc/**'
|
||||
|
||||
deploy:
|
||||
name: modal-accelerate / DeepSpeedAI CI
|
||||
runs-on: ubuntu-latest
|
||||
needs: collect-tests
|
||||
env:
|
||||
# these are created at https://modal.com/settings/deepspeedai/tokens
|
||||
# they are then added to the repo's secrets at https://github.com/deepspeedai/deepspeed/settings/secrets/actions
|
||||
MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}
|
||||
MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }}
|
||||
# this one comes from https://huggingface.co/settings/profile of the bot user
|
||||
# and it too is then updated at https://github.com/deepspeedai/deepspeed/settings/secrets/actions
|
||||
HF_TOKEN: ${{ secrets.HF_TOKEN }}
|
||||
|
||||
if: needs.collect-tests.outputs.deepspeed == 'true'
|
||||
steps:
|
||||
- name: Checkout Repository
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
lfs: true
|
||||
|
||||
- name: Install Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.10"
|
||||
cache: 'pip' # caching pip dependencies
|
||||
|
||||
- name: Install build dependencies
|
||||
run: |
|
||||
pip install uv # much faster than pip
|
||||
uv pip install --system modal
|
||||
|
||||
- name: Run tests
|
||||
run: |
|
||||
modal run -m ci.accelerate
|
||||
@@ -0,0 +1,241 @@
|
||||
name: modal-torch-latest
|
||||
|
||||
# This CI is running on modal.com's GPUs.
|
||||
#
|
||||
# It's set up here on github actions and then the cloned repo is sent to modal and everything
|
||||
# happens on their hw - see ci/torch_latest.py for where the actual vm is loaded, updated and the tests are
|
||||
# run.
|
||||
#
|
||||
# Both files are annotated to what's important and how one might change or update things if needed.
|
||||
#
|
||||
# Note that since this is a Required job we can't use `on.push.path` file filter - we are using
|
||||
# the collect-tests job to do the filtering for us so that the job can be skipped and satisfy the
|
||||
# Required status for PRs to pass.
|
||||
#
|
||||
# Test selection: collect-tests runs ci/tests_fetcher.py, which traces the import
|
||||
# graph from the PR's changed files to the impacted tests/unit/v1 tests. It emits a
|
||||
# `mode` (all | subset | none) and a test-list file:
|
||||
# - none -> deploy is skipped (Required status is still satisfied by the skip)
|
||||
# - subset -> deploy runs pytest on only the impacted test files
|
||||
# - all -> deploy runs the whole tests/unit/v1 suite
|
||||
#
|
||||
# Full design / how to drive & change it: .github/workflows/TEST_SELECTION.md
|
||||
#
|
||||
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
torch_preset:
|
||||
description: Modal PyTorch/CUDA image preset for manual runs
|
||||
required: false
|
||||
default: '2.10.0-cuda12.8'
|
||||
type: choice
|
||||
options:
|
||||
- '2.7.1-cuda12.8'
|
||||
- '2.8.0-cuda12.8'
|
||||
- '2.9.1-cuda12.8'
|
||||
- '2.10.0-cuda12.8'
|
||||
- '2.11.0-cuda12.8'
|
||||
transformers_source:
|
||||
description: Hugging Face Transformers source for manual runs
|
||||
required: false
|
||||
default: 'git'
|
||||
type: choice
|
||||
options:
|
||||
- 'requirements'
|
||||
- 'git'
|
||||
transformers_ref:
|
||||
description: Hugging Face Transformers git ref to install when source is git
|
||||
required: false
|
||||
default: 'main'
|
||||
type: string
|
||||
|
||||
push:
|
||||
branches:
|
||||
- master
|
||||
|
||||
pull_request_target:
|
||||
types: [review_requested, ready_for_review, synchronize]
|
||||
branches:
|
||||
- master
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
collect-tests:
|
||||
name: modal-torch-latest / collect tests
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: read
|
||||
outputs:
|
||||
mode: ${{ steps.fetch.outputs.mode }}
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
# For PRs, check out the PR head so the fetcher sees the proposed
|
||||
# deepspeed/ + tests/ code. This job holds NO secrets. The selection
|
||||
# *logic* (ci/) is restored from the base branch in a later step so a PR
|
||||
# can't tamper with the decision (see "Use base-branch CI scripts").
|
||||
# Falls back to the pushed/manual commit for non-PR events.
|
||||
ref: ${{ github.event.pull_request.head.sha || github.sha }}
|
||||
# Full history so the merge-base with the base branch is always present.
|
||||
# This is the robust (if slower) choice: a shallow clone that can't reach
|
||||
# the fork point would produce a wrong diff. The fetcher additionally
|
||||
# guards against a missing merge-base by falling back to the full suite,
|
||||
# so we never run a narrow (false-negative) selection on a bad base.
|
||||
fetch-depth: 0
|
||||
lfs: true
|
||||
|
||||
- name: Determine base ref
|
||||
id: base
|
||||
env:
|
||||
EVENT_NAME: ${{ github.event_name }}
|
||||
BASE_REF: ${{ github.base_ref }}
|
||||
run: |
|
||||
git config --global --add safe.directory '*'
|
||||
if [ "$EVENT_NAME" = "pull_request_target" ]; then
|
||||
# Fetch the base branch's full history too, so a merge-base always exists.
|
||||
git fetch --no-tags origin "$BASE_REF"
|
||||
echo "ref=origin/$BASE_REF" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
# push to master / manual dispatch -> no diff base -> run the full suite
|
||||
echo "ref=" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
- name: Use base-branch CI scripts
|
||||
# SECURITY: this job decides whether the Required `deploy` job runs. Under
|
||||
# pull_request_target we must NOT trust the PR's own ci/ for that decision:
|
||||
# a PR could rewrite ci/tests_fetcher.py to emit `mode=none` (or its self
|
||||
# tests) and skip CI entirely while still satisfying the Required check.
|
||||
# So run the selector + self-tests from the *base* branch's ci/. The diff is
|
||||
# computed from git history, so a PR's ci/ changes still show up in the diff
|
||||
# and (via the base selector's `ci/**` run-all glob) force a full run.
|
||||
if: github.event_name == 'pull_request_target'
|
||||
env:
|
||||
BASE_REF: ${{ github.base_ref }}
|
||||
run: |
|
||||
git config --global --add safe.directory '*'
|
||||
git checkout "origin/$BASE_REF" -- ci/
|
||||
|
||||
- name: Self-test the fetcher
|
||||
# Pure-stdlib self-tests for the selector (only needs git + python). Runs
|
||||
# here so a broken selector is caught before it (mis)picks tests. Skipped
|
||||
# when the base branch has no selector yet (bootstrap; see below).
|
||||
run: |
|
||||
if [ -f ci/test_tests_fetcher.py ]; then
|
||||
python3 ci/test_tests_fetcher.py
|
||||
else
|
||||
echo "No base-branch ci/test_tests_fetcher.py (bootstrap) -> skipping self-tests."
|
||||
fi
|
||||
|
||||
- name: Select impacted tests
|
||||
id: fetch
|
||||
# Pure-stdlib script; the runner's system python is fine (no deps needed).
|
||||
# If the base branch doesn't have the selector yet (e.g. the PR that
|
||||
# introduces it), the restored base ci/ won't contain tests_fetcher.py, so
|
||||
# fall back to the full suite -- safe, and unblocks the bootstrap PR.
|
||||
run: |
|
||||
if [ -f ci/tests_fetcher.py ]; then
|
||||
python3 ci/tests_fetcher.py --workflow modal-torch-latest --base "${{ steps.base.outputs.ref }}"
|
||||
else
|
||||
echo "No base-branch ci/tests_fetcher.py (bootstrap) -> running the full suite."
|
||||
echo "mode=all" >> "$GITHUB_OUTPUT"
|
||||
mkdir -p ci/.test_selection
|
||||
printf 'tests/unit/v1\n' > ci/.test_selection/test_list.txt
|
||||
fi
|
||||
|
||||
- name: Upload test selection
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: modal-torch-latest-test-selection
|
||||
path: ci/.test_selection/test_list.txt
|
||||
# The path lives under a dot-folder; upload-artifact treats dot-prefixed
|
||||
# paths as hidden and skips them by default (only warns), which would make
|
||||
# deploy's download fail. Opt in explicitly.
|
||||
include-hidden-files: true
|
||||
retention-days: 3
|
||||
|
||||
deploy:
|
||||
name: modal-torch-latest / DeepSpeedAI CI
|
||||
runs-on: ubuntu-latest
|
||||
needs: collect-tests
|
||||
|
||||
# Run when: manual dispatch, OR the selector failed (fail closed so the Required
|
||||
# check doesn't silently pass), OR the diff impacts at least one test (mode != none).
|
||||
# (!cancelled() adopted from upstream so a cancelled run doesn't trigger deploy.)
|
||||
if: ${{ !cancelled() && (github.event_name == 'workflow_dispatch' || needs.collect-tests.result != 'success' || needs.collect-tests.outputs.mode != 'none') }}
|
||||
steps:
|
||||
- name: Fail if test selection failed
|
||||
if: github.event_name != 'workflow_dispatch' && needs.collect-tests.result != 'success'
|
||||
run: exit 1
|
||||
|
||||
- name: Checkout Repository
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
# Test the PR's code (not the base) so diff-driven selection is meaningful.
|
||||
# SECURITY NOTE: under pull_request_target this runs PR-authored code while
|
||||
# the modal/HF secrets are on this runner. The trigger types include
|
||||
# `synchronize`, so this re-runs on every push to an open PR (not only on a
|
||||
# maintainer's review_requested / ready_for_review action) -- do not treat
|
||||
# the maintainer gate as an absolute barrier. The primary protection is that
|
||||
# the CI orchestration (which actually receives the secrets) is restored
|
||||
# from the base branch in the next step, so a PR can't repoint it at the
|
||||
# secrets; only the PR's deepspeed/ + tests/ run, inside modal.
|
||||
ref: ${{ github.event.pull_request.head.sha || github.sha }}
|
||||
lfs: true
|
||||
|
||||
- name: Use base-branch CI scripts
|
||||
# Run the *base* version of ci/ (which drives `modal run` with the secrets in
|
||||
# scope) rather than the PR's, so a PR can't modify the orchestration to
|
||||
# exfiltrate secrets. The PR's deepspeed/ + tests/ are still what gets tested.
|
||||
# (As noted in the header, changes to ci/* must be validated via `pull_request`
|
||||
# or the modal CLI, since pull_request_target always uses the base ci/ here.)
|
||||
if: github.event_name == 'pull_request_target'
|
||||
env:
|
||||
BASE_REF: ${{ github.base_ref }}
|
||||
run: |
|
||||
git config --global --add safe.directory '*'
|
||||
git fetch --no-tags --depth=1 origin "$BASE_REF"
|
||||
git checkout "origin/$BASE_REF" -- ci/
|
||||
|
||||
- name: Install Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.10"
|
||||
cache: 'pip' # caching pip dependencies
|
||||
|
||||
- name: Download test selection
|
||||
if: needs.collect-tests.result == 'success'
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
name: modal-torch-latest-test-selection
|
||||
path: ci/.test_selection
|
||||
|
||||
- name: Install build dependencies
|
||||
run: |
|
||||
pip install uv # much faster than pip
|
||||
uv pip install --system modal
|
||||
|
||||
- name: Run tests
|
||||
env:
|
||||
MODAL_TORCH_PRESET: ${{ github.event.inputs.torch_preset || '2.10.0-cuda12.8' }}
|
||||
MODAL_TRANSFORMERS_SOURCE: ${{ github.event.inputs.transformers_source || 'git' }}
|
||||
MODAL_TRANSFORMERS_REF: ${{ github.event.inputs.transformers_ref || 'main' }}
|
||||
# Diff-selected pytest targets produced by collect-tests (falls back to the
|
||||
# full tests/unit/v1 suite if the file is missing/empty).
|
||||
DS_TEST_LIST_FILE: ci/.test_selection/test_list.txt
|
||||
# these are created at https://modal.com/settings/deepspeedai/tokens
|
||||
# they are then added to the repo's secrets at https://github.com/deepspeedai/deepspeed/settings/secrets/actions
|
||||
MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}
|
||||
MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }}
|
||||
# this one comes from https://huggingface.co/settings/profile of the bot user
|
||||
# and it too is then updated at https://github.com/deepspeedai/deepspeed/settings/secrets/actions
|
||||
HF_TOKEN: ${{ secrets.HF_TOKEN }}
|
||||
run: |
|
||||
modal run -m ci.torch_latest
|
||||
@@ -0,0 +1,50 @@
|
||||
name: no-torch
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
pull_request:
|
||||
paths:
|
||||
- 'accelerator/**'
|
||||
- '.github/workflows/no-torch.yml'
|
||||
- 'op_builder/**'
|
||||
schedule:
|
||||
- cron: "0 0 * * *"
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
|
||||
jobs:
|
||||
unit-tests:
|
||||
name: no-torch / source distribution build
|
||||
runs-on: ubuntu-24.04
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- id: setup-venv
|
||||
uses: ./.github/workflows/setup-venv
|
||||
|
||||
- name: Python environment
|
||||
run: |
|
||||
pip uninstall torch --yes
|
||||
pip install setuptools
|
||||
pip install build
|
||||
pip list
|
||||
|
||||
- name: Build deepspeed
|
||||
run: |
|
||||
DS_BUILD_STRING=" " python -m build --sdist
|
||||
|
||||
- name: Open GitHub issue if nightly CI fails
|
||||
if: ${{ failure() && (github.event_name == 'schedule') }}
|
||||
uses: JasonEtco/create-an-issue@v2
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
with:
|
||||
filename: .github/ISSUE_TEMPLATE/ci_failure_report.md
|
||||
update_existing: true
|
||||
@@ -0,0 +1,75 @@
|
||||
name: nv-a6000
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- 'accelerator/cuda_accelerator.py'
|
||||
- 'deepspeed/inference/v2/**'
|
||||
- 'tests/unit/inference/v2/**'
|
||||
- '.github/workflows/nv-a6000.yml'
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
mii_branch:
|
||||
description: 'DeepSpeed-MII Branch'
|
||||
required: false
|
||||
default: 'main'
|
||||
type: string
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
unit-tests:
|
||||
name: nv-a6000 / inference and MII tests
|
||||
runs-on: [self-hosted, nvidia, a6000]
|
||||
container:
|
||||
image: nvcr.io/nvidia/pytorch:25.01-py3
|
||||
ports:
|
||||
- 80
|
||||
options: --gpus all --shm-size "8G"
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Check container state
|
||||
run: |
|
||||
ldd --version
|
||||
nvcc --version
|
||||
nvidia-smi
|
||||
python -c "import torch; print('torch:', torch.__version__, torch)"
|
||||
python -c "import torch; print('CUDA available:', torch.cuda.is_available())"
|
||||
- name: Install transformers
|
||||
run: |
|
||||
git clone https://github.com/huggingface/transformers
|
||||
cd transformers
|
||||
# if you need to use an older transformers version temporarily in case of breakage
|
||||
# git checkout 981c276
|
||||
git rev-parse --short HEAD
|
||||
python -m pip install .
|
||||
- name: Install deepspeed
|
||||
run: |
|
||||
python -m pip install docutils==0.18.1 jinja2==3.0 urllib3==1.26.11 ninja
|
||||
python -m pip install .[dev,1bit,autotuning,inf]
|
||||
ds_report
|
||||
- name: Python environment
|
||||
run: |
|
||||
python -m pip list
|
||||
- name: Unit tests
|
||||
run: |
|
||||
unset TORCH_CUDA_ARCH_LIST # only jit compile for current arch
|
||||
cd tests
|
||||
python -m pytest --color=yes --durations=0 --verbose -rF -m 'inference_v2' unit/ --torch_ver="2.6" --cuda_ver="12"
|
||||
python -m pytest --color=yes --durations=0 --verbose -rF -m 'inference_v2_ops' unit/ --torch_ver="2.6" --cuda_ver="12"
|
||||
- name: MII unit tests
|
||||
run: |
|
||||
BRANCH="main"
|
||||
if [[ ! -z "${{ github.event.inputs.mii_branch }}" ]]; then
|
||||
BRANCH="${{ github.event.inputs.mii_branch }}"
|
||||
fi
|
||||
echo "Cloning DeepSpeed-MII branch: $BRANCH"
|
||||
git clone -b $BRANCH --depth=1 https://github.com/deepspeedai/DeepSpeed-MII.git
|
||||
cd DeepSpeed-MII
|
||||
pip install .[dev]
|
||||
cd tests
|
||||
python -m pytest --color=yes --durations=0 --verbose -rF ./
|
||||
@@ -0,0 +1,47 @@
|
||||
name: nv-accelerate-v100
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
unit-tests:
|
||||
runs-on: [self-hosted, nvidia, cu124, v100]
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- id: setup-venv
|
||||
uses: ./.github/workflows/setup-venv
|
||||
|
||||
- name: Install pytorch
|
||||
run: |
|
||||
pip install -U --cache-dir $TORCH_CACHE torch torchvision --index-url https://download.pytorch.org/whl/cu124
|
||||
python -c "import torch; print('torch:', torch.__version__, torch)"
|
||||
python -c "import torch; print('CUDA available:', torch.cuda.is_available())"
|
||||
|
||||
- name: Install deepspeed
|
||||
run: |
|
||||
pip install .[dev,autotuning]
|
||||
ds_report
|
||||
|
||||
- name: Python environment
|
||||
run: |
|
||||
pip list
|
||||
|
||||
- name: HF Accelerate tests
|
||||
run: |
|
||||
unset TORCH_CUDA_ARCH_LIST # only jit compile for current arch
|
||||
git clone https://github.com/huggingface/accelerate
|
||||
cd accelerate
|
||||
git rev-parse --short HEAD
|
||||
|
||||
# temp workaround until this is resolved https://github.com/huggingface/accelerate/issues/3676
|
||||
pip install datasets==3.6.0
|
||||
|
||||
# installing dependencies
|
||||
pip install .[testing]
|
||||
# force protobuf version due to issues
|
||||
pip install "protobuf<4.21.0"
|
||||
pip list
|
||||
pytest $PYTEST_OPTS --color=yes --durations=0 --verbose tests/deepspeed
|
||||
@@ -0,0 +1,80 @@
|
||||
name: nv-ds-chat
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
dse_branch:
|
||||
description: 'DeepSpeedExamples Branch'
|
||||
required: false
|
||||
default: 'master'
|
||||
type: string
|
||||
pull_request:
|
||||
paths:
|
||||
- ".github/workflows/nv-ds-chat.yml"
|
||||
- "deepspeed/runtime/zero/stage_1_and_2.py"
|
||||
- "deepspeed/runtime/zero/stage3.py"
|
||||
- "deepspeed/runtime/hybrid_engine.py"
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
|
||||
jobs:
|
||||
unit-tests:
|
||||
name: nv-ds-chat / DeepSpeed-Chat tests
|
||||
runs-on: [self-hosted, nvidia, cu124, v100]
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- id: setup-venv
|
||||
uses: ./.github/workflows/setup-venv
|
||||
|
||||
- name: Install pytorch
|
||||
run: |
|
||||
pip install -U --cache-dir $TORCH_CACHE torch torchvision --index-url https://download.pytorch.org/whl/cu124
|
||||
python -c "import torch; print('torch:', torch.__version__, torch)"
|
||||
python -c "import torch; print('CUDA available:', torch.cuda.is_available())"
|
||||
|
||||
- name: Install deepspeed
|
||||
run: |
|
||||
pip install .[dev]
|
||||
pip install transformers==4.48.3
|
||||
ds_report
|
||||
|
||||
- name: Install deepspeed-chat
|
||||
run: |
|
||||
BRANCH="master"
|
||||
if [[ ! -z "${{ github.event.inputs.dse_branch }}" ]]; then
|
||||
BRANCH="${{ github.event.inputs.dse_branch }}"
|
||||
fi
|
||||
echo "DeepSpeedExamples Branch: $BRANCH"
|
||||
git clone -b $BRANCH https://github.com/deepspeedai/DeepSpeedExamples.git
|
||||
cd DeepSpeedExamples/applications/DeepSpeed-Chat
|
||||
pip install -r requirements.txt
|
||||
pip install -e .
|
||||
|
||||
- name: Python environment
|
||||
run: |
|
||||
pip list
|
||||
|
||||
- name: DS-Chat unit tests
|
||||
run: |
|
||||
cd DeepSpeedExamples/applications/DeepSpeed-Chat
|
||||
unset TORCH_CUDA_ARCH_LIST # only jit compile for current arch
|
||||
unset NCCL_DEBUG
|
||||
cd tests
|
||||
pytest $PYTEST_OPTS ./
|
||||
|
||||
- name: Open GitHub issue if nightly CI fails
|
||||
if: ${{ failure() && (github.event_name == 'schedule') }}
|
||||
uses: JasonEtco/create-an-issue@v2
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
with:
|
||||
filename: .github/ISSUE_TEMPLATE/ci_failure_report.md
|
||||
update_existing: true
|
||||
@@ -0,0 +1,68 @@
|
||||
name: nv-flash-attn
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
pull_request:
|
||||
paths:
|
||||
- 'deepspeed/sequence/**'
|
||||
- 'tests/unit/sequence_parallelism/**'
|
||||
- '.github/workflows/nv-flash-attn.yml'
|
||||
schedule:
|
||||
- cron: "0 0 * * *"
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
unit-tests:
|
||||
name: nv-flash-attn / sequence parallelism tests
|
||||
runs-on: [self-hosted, nvidia, a6000]
|
||||
container:
|
||||
image: nvcr.io/nvidia/pytorch:24.12-py3
|
||||
ports:
|
||||
- 80
|
||||
options: --gpus all --shm-size "8G"
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Check container state
|
||||
run: |
|
||||
ldd --version
|
||||
nvcc --version
|
||||
nvidia-smi
|
||||
python -c "import torch; print('torch:', torch.__version__, torch)"
|
||||
python -c "import torch; print('CUDA available:', torch.cuda.is_available())"
|
||||
|
||||
|
||||
|
||||
- name: Install deepspeed
|
||||
run: |
|
||||
python -m pip install .[dev]
|
||||
ds_report
|
||||
|
||||
# install transformers after deepspeed so that the right version of transformers is installed
|
||||
- name: Install transformers
|
||||
run: |
|
||||
python -m pip install transformers==4.50.0
|
||||
|
||||
- name: Install FlashAttention
|
||||
run: |
|
||||
python -m pip install flash-attn
|
||||
- name: Python environment
|
||||
run: |
|
||||
python -m pip list
|
||||
- name: Unit tests
|
||||
run: |
|
||||
unset TORCH_CUDA_ARCH_LIST # only jit compile for current arch
|
||||
cd tests
|
||||
python -m pytest --color=yes --durations=0 --verbose -rF unit/sequence_parallelism/test_ulysses.py --torch_ver="2.6" --cuda_ver="12"
|
||||
- name: Open GitHub issue if nightly CI fails
|
||||
if: ${{ failure() && (github.event_name == 'schedule') }}
|
||||
uses: JasonEtco/create-an-issue@v2
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
with:
|
||||
filename: .github/ISSUE_TEMPLATE/ci_failure_report.md
|
||||
update_existing: true
|
||||
@@ -0,0 +1,66 @@
|
||||
name: nv-inference
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
pull_request:
|
||||
paths:
|
||||
- '.github/workflows/nv-inference.yml'
|
||||
- 'requirements/**'
|
||||
- 'deepspeed/__init__.py'
|
||||
- 'deepspeed/inference/**'
|
||||
- '!deepspeed/inference/v2/**' # exclude v2 dir
|
||||
- 'tests/unit/inference/**'
|
||||
- '!tests/unit/inference/v2/**' # exclude v2 tests dir
|
||||
merge_group:
|
||||
branches: [ master ]
|
||||
schedule:
|
||||
- cron: "0 0 * * *"
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
unit-tests:
|
||||
name: nv-inference / inference tests
|
||||
runs-on: [self-hosted, nvidia, cu124, v100]
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- id: setup-venv
|
||||
uses: ./.github/workflows/setup-venv
|
||||
|
||||
- name: Install pytorch
|
||||
run: |
|
||||
pip install -U --cache-dir $TORCH_CACHE torch==2.1.2 torchvision==0.16.2 --index-url https://download.pytorch.org/whl/cu124
|
||||
python -c "import torch; print('torch:', torch.__version__, torch)"
|
||||
python -c "import torch; print('CUDA available:', torch.cuda.is_available())"
|
||||
|
||||
- name: Install transformers
|
||||
run: |
|
||||
git clone https://github.com/huggingface/transformers
|
||||
cd transformers
|
||||
#git checkout f370bebdc
|
||||
git rev-parse --short HEAD
|
||||
pip install .
|
||||
|
||||
- name: Install deepspeed
|
||||
run: |
|
||||
DS_ACCELERATOR=cpu pip install .[dev,1bit,autotuning,inf]
|
||||
#pip install .[dev,1bit,autotuning,inf,triton]
|
||||
ds_report
|
||||
|
||||
- name: Python environment
|
||||
run: |
|
||||
pip list
|
||||
|
||||
- name: Unit tests
|
||||
run: |
|
||||
unset TORCH_CUDA_ARCH_LIST # only jit compile for current arch
|
||||
cd tests
|
||||
#pytest $PYTEST_OPTS -m 'seq_inference' unit/ --torch_ver="2.1" --cuda_ver="12.4"
|
||||
pytest $PYTEST_OPTS -m 'inference_ops' unit/ --torch_ver="2.1" --cuda_ver="12.4"
|
||||
pytest $PYTEST_OPTS --forked -n 4 -m 'inference' unit/ --torch_ver="2.1" --cuda_ver="12.4"
|
||||
# run ds_report again to check updated op list
|
||||
ds_report
|
||||
@@ -0,0 +1,53 @@
|
||||
# name: nv-lightning-v100
|
||||
|
||||
# disabled as the v100s are no more - need to port to modal while removing v100
|
||||
|
||||
# on:
|
||||
# workflow_dispatch:
|
||||
# pull_request:
|
||||
# paths-ignore:
|
||||
# - 'docs/**'
|
||||
# - 'blogs/**'
|
||||
# - 'deepspeed/inference/v2/**'
|
||||
# - 'tests/unit/inference/v2/**'
|
||||
# merge_group:
|
||||
# branches: [ master ]
|
||||
# schedule:
|
||||
# - cron: "0 0 * * *"
|
||||
|
||||
# concurrency:
|
||||
# group: ${{ github.workflow }}-${{ github.ref }}
|
||||
# cancel-in-progress: true
|
||||
|
||||
# jobs:
|
||||
# unit-tests:
|
||||
# runs-on: [self-hosted, nvidia, cu124, v100]
|
||||
|
||||
# steps:
|
||||
# - uses: actions/checkout@v4
|
||||
|
||||
# - id: setup-venv
|
||||
# uses: ./.github/workflows/setup-venv
|
||||
|
||||
# - name: Install pytorch
|
||||
# run: |
|
||||
# pip install -U --cache-dir $TORCH_CACHE torch torchvision --index-url https://download.pytorch.org/whl/cu124
|
||||
# python -c "import torch; print('torch:', torch.__version__, torch)"
|
||||
# python -c "import torch; print('CUDA available:', torch.cuda.is_available())"
|
||||
|
||||
# - name: Install deepspeed
|
||||
# run: |
|
||||
# pip install .[dev,autotuning]
|
||||
# ds_report
|
||||
|
||||
# - name: Python environment
|
||||
# run: |
|
||||
# pip list
|
||||
|
||||
# - name: PyTorch Lightning Tests
|
||||
# run: |
|
||||
# unset TORCH_CUDA_ARCH_LIST # only jit compile for current arch
|
||||
# pip install pytorch-lightning
|
||||
# pip install "protobuf<4.21.0"
|
||||
# cd tests
|
||||
# pytest $PYTEST_OPTS lightning/
|
||||
@@ -0,0 +1,54 @@
|
||||
name: nv-mii
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
unit-tests:
|
||||
runs-on: [self-hosted, nvidia, cu124, v100]
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- id: setup-venv
|
||||
uses: ./.github/workflows/setup-venv
|
||||
|
||||
- name: Install pytorch
|
||||
run: |
|
||||
pip3 install -U --cache-dir $TORCH_CACHE torch torchvision --index-url https://download.pytorch.org/whl/cu124
|
||||
python -c "import torch; print('torch:', torch.__version__, torch)"
|
||||
python -c "import torch; print('CUDA available:', torch.cuda.is_available())"
|
||||
|
||||
- name: Install deepspeed
|
||||
run: |
|
||||
pip install .[dev]
|
||||
ds_report
|
||||
|
||||
# install transformers after deepspeed so that the right version of transformers is installed
|
||||
- name: Install transformers
|
||||
run: |
|
||||
git clone https://github.com/huggingface/transformers
|
||||
cd transformers
|
||||
# if needed switch to the last known good SHA until transformers@master is fixed
|
||||
git checkout v4.42.4
|
||||
git rev-parse --short HEAD
|
||||
pip install .
|
||||
|
||||
- name: Python environment
|
||||
run: |
|
||||
pip list
|
||||
|
||||
- name: MII unit tests
|
||||
run: |
|
||||
BRANCH="main"
|
||||
if [[ ! -z "${{ github.event.inputs.mii_branch }}" ]]; then
|
||||
BRANCH="${{ github.event.inputs.mii_branch }}"
|
||||
fi
|
||||
echo "Cloning DeepSpeed-MII branch: $BRANCH"
|
||||
git clone -b $BRANCH --depth=1 https://github.com/deepspeedai/DeepSpeed-MII.git
|
||||
cd DeepSpeed-MII
|
||||
pip install .[dev]
|
||||
unset TORCH_CUDA_ARCH_LIST # only jit compile for current arch
|
||||
cd tests/legacy
|
||||
pytest $PYTEST_OPTS --forked -m "deepspeed" ./
|
||||
@@ -0,0 +1,62 @@
|
||||
name: nv-nightly
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
|
||||
jobs:
|
||||
unit-tests:
|
||||
runs-on: [self-hosted, nvidia, cu124, v100]
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- id: setup-venv
|
||||
uses: ./.github/workflows/setup-venv
|
||||
|
||||
- name: Install pytorch
|
||||
run: |
|
||||
pip install -U --cache-dir $TORCH_CACHE torch torchvision --index-url https://download.pytorch.org/whl/cu124
|
||||
python -c "import torch; print('torch:', torch.__version__, torch)"
|
||||
python -c "import torch; print('CUDA available:', torch.cuda.is_available())"
|
||||
|
||||
- name: Install transformers
|
||||
run: |
|
||||
git clone https://github.com/huggingface/transformers
|
||||
cd transformers
|
||||
# if needed switch to the last known good SHA until transformers@master is fixed
|
||||
git checkout v4.42.4
|
||||
git rev-parse --short HEAD
|
||||
pip install .
|
||||
|
||||
- name: Install datasets
|
||||
run: |
|
||||
pip install datasets
|
||||
|
||||
- name: Install deepspeed
|
||||
run: |
|
||||
pip install .[dev,1bit,autotuning,inf]
|
||||
ds_report
|
||||
|
||||
- name: Python environment
|
||||
run: |
|
||||
pip list
|
||||
|
||||
- name: Unit tests
|
||||
run: |
|
||||
unset TORCH_CUDA_ARCH_LIST # only jit compile for current arch
|
||||
cd tests
|
||||
pytest $PYTEST_OPTS --forked -m 'nightly' unit/ --torch_ver="2.6" --cuda_ver="12.4"
|
||||
|
||||
- name: Open GitHub issue if nightly CI fails
|
||||
if: ${{ failure() && (github.event_name == 'schedule') }}
|
||||
uses: JasonEtco/create-an-issue@v2
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
with:
|
||||
filename: .github/ISSUE_TEMPLATE/ci_failure_report.md
|
||||
update_existing: true
|
||||
@@ -0,0 +1,82 @@
|
||||
name: nv-pre-compile-ops
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
pull_request:
|
||||
branches:
|
||||
'**'
|
||||
merge_group:
|
||||
branches: [ master ]
|
||||
schedule:
|
||||
- cron: "0 0 * * *"
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
check-paths:
|
||||
name: nv-pre-compile-ops / check paths
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: read
|
||||
outputs:
|
||||
should_run: ${{ steps.non_pr.outputs.should_run || steps.filter.outputs.run_tests }}
|
||||
steps:
|
||||
- id: non_pr
|
||||
if: github.event_name != 'pull_request'
|
||||
run: echo "should_run=true" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- uses: actions/checkout@v4
|
||||
if: github.event_name == 'pull_request'
|
||||
|
||||
- uses: dorny/paths-filter@v3
|
||||
id: filter
|
||||
if: github.event_name == 'pull_request'
|
||||
with:
|
||||
predicate-quantifier: every
|
||||
filters: |
|
||||
run_tests:
|
||||
- '**'
|
||||
- '!docs/**'
|
||||
- '!blogs/**'
|
||||
- '!deepspeed/inference/v2/**'
|
||||
- '!tests/unit/inference/v2/**'
|
||||
|
||||
unit-tests:
|
||||
name: nv-pre-compile-ops / precompile ops
|
||||
needs: check-paths
|
||||
if: ${{ !cancelled() && (needs.check-paths.result != 'success' || needs.check-paths.outputs.should_run == 'true') }}
|
||||
runs-on: ubuntu-24.04
|
||||
container:
|
||||
image: nvidia/cuda:12.6.3-devel-ubuntu22.04
|
||||
|
||||
steps:
|
||||
- name: Fail if path filter failed
|
||||
if: needs.check-paths.result != 'success'
|
||||
run: exit 1
|
||||
|
||||
- name: Install system dependencies
|
||||
run: |
|
||||
apt-get update && apt-get install -y git python3 python3-pip libaio-dev ninja-build
|
||||
ln -sf /usr/bin/python3 /usr/bin/python
|
||||
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Install PyTorch
|
||||
run: |
|
||||
pip install torch==2.10.0 --index-url https://download.pytorch.org/whl/cu126
|
||||
|
||||
- name: environment
|
||||
run: |
|
||||
which python
|
||||
python --version
|
||||
python -c "import torch; print('torch:', torch.__version__, torch)"
|
||||
#python -c "import torch; print('CUDA available:', torch.cuda.is_available())"
|
||||
- name: Compile DeepSpeed Ops
|
||||
run: |
|
||||
DS_ACCELERATOR=cuda DS_ENABLE_NINJA=1 TORCH_CUDA_ARCH_LIST="7.0;7.5;8.0;8.6;8.9;9.0" DS_BUILD_OPS=1 DS_BUILD_SPARSE_ATTN=0 DS_BUILD_FP_QUANTIZER=0 DS_BUILD_CUTLASS_OPS=0 DS_BUILD_GDS=0 DS_BUILD_RAGGED_DEVICE_OPS=0 DS_BUILD_EVOFORMER_ATTN=0 DS_BUILD_DEEP_COMPILE=0 pip3 install .
|
||||
- name: DS Report
|
||||
run: |
|
||||
DS_ACCELERATOR=cuda ds_report
|
||||
@@ -0,0 +1,73 @@
|
||||
name: nv-sd
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
pull_request:
|
||||
paths:
|
||||
- "deepspeed/ops/transformer/inference/diffusers_**"
|
||||
- "tests/unit/inference/test_stable_diffusion.py"
|
||||
- "deepspeed/model_implementations/diffusers/unet.py"
|
||||
- "deepspeed/model_implementations/diffusers/vae.py"
|
||||
- "deepspeed/module_inject/containers/vae.py"
|
||||
- "deepspeed/module_inject/containers/unet.py"
|
||||
- ".github/workflows/nv-sd.yml"
|
||||
- "requirements/requirements-sd.txt"
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
|
||||
jobs:
|
||||
sd-tests:
|
||||
name: nv-sd / stable diffusion tests
|
||||
runs-on: [self-hosted, nvidia, a6000]
|
||||
container:
|
||||
image: nvcr.io/nvidia/pytorch:24.03-py3
|
||||
ports:
|
||||
- 80
|
||||
options: --gpus all --shm-size "8G"
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Check container state
|
||||
run: |
|
||||
ldd --version
|
||||
nvcc --version
|
||||
nvidia-smi
|
||||
python -c "import torch; print('torch:', torch.__version__, torch)"
|
||||
python -c "import torch; print('CUDA available:', torch.cuda.is_available())"
|
||||
- name: Install transformers
|
||||
run: |
|
||||
git clone https://github.com/huggingface/transformers
|
||||
cd transformers
|
||||
git rev-parse --short HEAD
|
||||
python -m pip install .
|
||||
- name: Install deepspeed
|
||||
run: |
|
||||
pip install image-similarity-measures
|
||||
python -m pip install opencv-python==4.6.* --force-reinstall
|
||||
python -m pip install docutils==0.18.1 jinja2==3.0 urllib3==1.26.11 ninja
|
||||
python -m pip install .[dev,1bit,autotuning,sd]
|
||||
ds_report
|
||||
- name: Python environment
|
||||
run: |
|
||||
python -m pip list
|
||||
- name: Unit tests
|
||||
run: |
|
||||
unset TORCH_CUDA_ARCH_LIST # only jit compile for current arch
|
||||
cd tests
|
||||
python -m pytest --color=yes --durations=0 --verbose -rF -m 'stable_diffusion' -k "TestStableDiffusion" unit/ --torch_ver="2.3" --cuda_ver="12"
|
||||
|
||||
- name: Open GitHub issue if weekly CI fails
|
||||
if: ${{ failure() && (github.event_name == 'schedule') }}
|
||||
uses: JasonEtco/create-an-issue@v2
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
with:
|
||||
filename: .github/ISSUE_TEMPLATE/ci_failure_report.md
|
||||
update_existing: true
|
||||
@@ -0,0 +1,47 @@
|
||||
name: nv-torch-latest-v100
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
unit-tests:
|
||||
runs-on: [self-hosted, nvidia, cu124, v100]
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- id: setup-venv
|
||||
uses: ./.github/workflows/setup-venv
|
||||
|
||||
- name: Install pytorch
|
||||
run: |
|
||||
pip install -U --cache-dir $TORCH_CACHE torch torchvision --index-url https://download.pytorch.org/whl/cu124
|
||||
python -c "import torch; print('torch:', torch.__version__, torch)"
|
||||
python -c "import torch; print('CUDA available:', torch.cuda.is_available())"
|
||||
|
||||
- name: Install transformers
|
||||
run: |
|
||||
git clone https://github.com/huggingface/transformers
|
||||
cd transformers
|
||||
# if needed switch to the last known good SHA until transformers@master is fixed
|
||||
git checkout 981c276
|
||||
git rev-parse --short HEAD
|
||||
pip install .
|
||||
|
||||
- name: Install deepspeed
|
||||
run: |
|
||||
pip install .[dev,1bit,autotuning,deepcompile]
|
||||
pip install pytest-timeout pytest-instafail
|
||||
ds_report
|
||||
|
||||
- name: Python environment
|
||||
run: |
|
||||
pip list
|
||||
|
||||
- name: Unit tests
|
||||
run: |
|
||||
unset TORCH_CUDA_ARCH_LIST # only jit compile for current arch
|
||||
cd tests
|
||||
pytest -x $PYTEST_OPTS --instafail --timeout 600 --forked -n 8 unit/ --torch_ver="2.6" --cuda_ver="12.4"
|
||||
pytest $PYTEST_OPTS --instafail --timeout 600 --forked -m 'sequential' unit/ --torch_ver="2.6" --cuda_ver="12.4"
|
||||
@@ -0,0 +1,59 @@
|
||||
name: nv-torch-nightly-v100
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
|
||||
jobs:
|
||||
unit-tests:
|
||||
runs-on: [self-hosted, nvidia, cu124, v100]
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- id: setup-venv
|
||||
uses: ./.github/workflows/setup-venv
|
||||
|
||||
- name: Install pytorch
|
||||
run: |
|
||||
pip install --pre torch torchvision --index-url https://download.pytorch.org/whl/nightly/cu124
|
||||
python -c "import torch; print('torch:', torch.__version__, torch)"
|
||||
python -c "import torch; print('CUDA available:', torch.cuda.is_available())"
|
||||
|
||||
- name: Install transformers
|
||||
run: |
|
||||
git clone https://github.com/huggingface/transformers
|
||||
cd transformers
|
||||
# if needed switch to the last known good SHA until transformers@master is fixed
|
||||
# git checkout 981c276
|
||||
git rev-parse --short HEAD
|
||||
pip install .
|
||||
|
||||
- name: Install deepspeed
|
||||
run: |
|
||||
pip install .[dev,1bit,autotuning]
|
||||
ds_report
|
||||
|
||||
- name: Python environment
|
||||
run: |
|
||||
pip list
|
||||
|
||||
- name: Unit tests
|
||||
run: |
|
||||
unset TORCH_CUDA_ARCH_LIST # only jit compile for current arch
|
||||
cd tests
|
||||
pytest $PYTEST_OPTS --forked -n 8 unit/
|
||||
pytest $PYTEST_OPTS --forked -m 'sequential' unit/
|
||||
|
||||
- name: Open GitHub issue if nightly CI fails
|
||||
if: ${{ failure() && (github.event_name == 'schedule') }}
|
||||
uses: JasonEtco/create-an-issue@v2
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
with:
|
||||
filename: .github/ISSUE_TEMPLATE/ci_failure_report.md
|
||||
update_existing: true
|
||||
@@ -0,0 +1,52 @@
|
||||
name: nv-transformers-v100
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
unit-tests:
|
||||
runs-on: [self-hosted, nvidia, cu124, v100]
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- id: setup-venv
|
||||
uses: ./.github/workflows/setup-venv
|
||||
|
||||
- name: Install pytorch
|
||||
run: |
|
||||
# use the same pytorch version as transformers CI
|
||||
pip install -U --cache-dir $TORCH_CACHE torch==2.0.1+cu124 --index-url https://download.pytorch.org/whl/cu124
|
||||
python -c "import torch; print('torch:', torch.__version__, torch)"
|
||||
python -c "import torch; print('CUDA available:', torch.cuda.is_available())"
|
||||
|
||||
- name: Install transformers
|
||||
run: |
|
||||
git clone https://github.com/huggingface/transformers
|
||||
cd transformers
|
||||
# if needed switch to the last known good SHA until transformers@master is fixed
|
||||
git checkout e7e9261a2
|
||||
git rev-parse --short HEAD
|
||||
pip install .
|
||||
|
||||
- name: Install deepspeed
|
||||
run: |
|
||||
pip install .[dev,autotuning]
|
||||
ds_report
|
||||
|
||||
- name: Python environment
|
||||
run: |
|
||||
pip list
|
||||
|
||||
- name: HF transformers tests
|
||||
run: |
|
||||
unset TORCH_CUDA_ARCH_LIST # only jit compile for current arch
|
||||
cd transformers
|
||||
pip install .[testing]
|
||||
# find reqs used in ds integration tests
|
||||
find examples/pytorch -regextype posix-egrep -regex '.*(language-modeling|question-answering|summarization|image-classification|text-classification|translation).*/requirements.txt' -exec grep -v 'torch' {} \; | xargs -I {} pip install --upgrade {}
|
||||
# force protobuf version due to issues
|
||||
pip install "protobuf<4.21.0"
|
||||
pip list
|
||||
WANDB_DISABLED=true RUN_SLOW=1 pytest $PYTEST_OPTS tests/deepspeed
|
||||
@@ -0,0 +1,90 @@
|
||||
name: python
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
pull_request:
|
||||
branches:
|
||||
'**'
|
||||
merge_group:
|
||||
branches: [ master ]
|
||||
schedule:
|
||||
- cron: "0 0 * * *"
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
check-paths:
|
||||
name: python / check paths
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: read
|
||||
outputs:
|
||||
should_run: ${{ steps.non_pr.outputs.should_run || steps.filter.outputs.run_tests }}
|
||||
steps:
|
||||
- id: non_pr
|
||||
if: github.event_name != 'pull_request'
|
||||
run: echo "should_run=true" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- uses: actions/checkout@v4
|
||||
if: github.event_name == 'pull_request'
|
||||
|
||||
- uses: dorny/paths-filter@v3
|
||||
id: filter
|
||||
if: github.event_name == 'pull_request'
|
||||
with:
|
||||
predicate-quantifier: every
|
||||
filters: |
|
||||
run_tests:
|
||||
- '**'
|
||||
- '!docs/**'
|
||||
- '!blogs/**'
|
||||
|
||||
unit-tests:
|
||||
name: python / install smoke (Python ${{ matrix.pyVersion }})
|
||||
needs: check-paths
|
||||
if: ${{ !cancelled() }}
|
||||
strategy:
|
||||
matrix:
|
||||
pyVersion: ["3.10", "3.11", "3.12"]
|
||||
fail-fast: false
|
||||
|
||||
runs-on: ubuntu-24.04
|
||||
container:
|
||||
image: python:${{ matrix.pyVersion }}-slim
|
||||
|
||||
steps:
|
||||
- name: Fail if path filter failed
|
||||
if: needs.check-paths.result != 'success'
|
||||
run: exit 1
|
||||
|
||||
- name: Skip ignored-path install smoke
|
||||
if: needs.check-paths.outputs.should_run != 'true'
|
||||
run: echo "Only ignored paths changed; install smoke intentionally skipped."
|
||||
|
||||
- uses: actions/checkout@v4
|
||||
if: needs.check-paths.outputs.should_run == 'true'
|
||||
|
||||
- name: Install build dependencies
|
||||
if: needs.check-paths.outputs.should_run == 'true'
|
||||
run: |
|
||||
apt-get update && apt-get install -y build-essential ninja-build
|
||||
- name: environment
|
||||
if: needs.check-paths.outputs.should_run == 'true'
|
||||
run: |
|
||||
which python
|
||||
python --version
|
||||
- name: Install PyTorch (CPU)
|
||||
if: needs.check-paths.outputs.should_run == 'true'
|
||||
run: |
|
||||
pip install torch --index-url https://download.pytorch.org/whl/cpu
|
||||
- name: Install deepspeed
|
||||
if: needs.check-paths.outputs.should_run == 'true'
|
||||
run: |
|
||||
pip install .
|
||||
- name: DS Report
|
||||
if: needs.check-paths.outputs.should_run == 'true'
|
||||
run: |
|
||||
ds_report
|
||||
@@ -0,0 +1,52 @@
|
||||
name: Build and publish DeepSpeed release
|
||||
|
||||
on:
|
||||
push:
|
||||
tags:
|
||||
- 'v*.*.*'
|
||||
|
||||
jobs:
|
||||
deploy:
|
||||
runs-on: ubuntu-24.04
|
||||
environment: release-env
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
ref: "master"
|
||||
- id: setup-venv
|
||||
uses: ./.github/workflows/setup-venv
|
||||
- name: Get release version from tag
|
||||
run: |
|
||||
echo "RELEASE_VERSION=${GITHUB_REF#refs/*/v}" >> $GITHUB_ENV
|
||||
- name: Check release version
|
||||
run: |
|
||||
pip install packaging
|
||||
python release/check_release_version.py --release_version ${{ env.RELEASE_VERSION }}
|
||||
- name: Build DeepSpeed
|
||||
run: |
|
||||
pip install setuptools
|
||||
pip install build
|
||||
DS_BUILD_STRING=" " python -m build --sdist
|
||||
- name: Publish to PyPI
|
||||
uses: pypa/gh-action-pypi-publish@release/v1
|
||||
with:
|
||||
password: ${{ secrets.PYPI_API_TOKEN }}
|
||||
repository-url: https://upload.pypi.org/legacy/
|
||||
- name: Bump version
|
||||
run: |
|
||||
python release/bump_patch_version.py --current_version ${{ env.RELEASE_VERSION }}
|
||||
- name: Create Pull Request
|
||||
uses: peter-evans/create-pull-request@v6
|
||||
with:
|
||||
token: ${{ secrets.GH_PAT }}
|
||||
add-paths: |
|
||||
version.txt
|
||||
body: |
|
||||
**Auto-generated PR to update version.txt after a DeepSpeed release**
|
||||
Released version - ${{ env.RELEASE_VERSION }}
|
||||
Author - @${{ github.actor }}
|
||||
branch: AutoPR/${{ env.RELEASE_VERSION }}
|
||||
assignees: ${{ github.actor }}
|
||||
title: "Update version.txt after ${{ env.RELEASE_VERSION }} release"
|
||||
author: ${{ github.actor }} <${{ github.actor }}@users.noreply.github.com>
|
||||
@@ -0,0 +1,53 @@
|
||||
name: Create Virtual Environment
|
||||
|
||||
runs:
|
||||
using: "composite"
|
||||
steps:
|
||||
- id: update-env
|
||||
run: |
|
||||
sudo apt-get update
|
||||
# Temporary disable nvme UTs
|
||||
# sudo apt-get install -y libaio-dev
|
||||
sudo apt remove -y libaio-dev
|
||||
python -m pip install --user --upgrade pip
|
||||
python -m pip install --user --upgrade virtualenv
|
||||
shell: bash
|
||||
- id: create-venv
|
||||
run: |
|
||||
rm -rf ./unit-test-venv
|
||||
python -m venv unit-test-venv
|
||||
source ./unit-test-venv/bin/activate
|
||||
python -m pip install --upgrade pip
|
||||
pip install wheel # required after pip>=23.1
|
||||
echo PATH=$PATH >> $GITHUB_ENV # Make it so venv is inherited for other steps
|
||||
shell: bash
|
||||
- id: set-env-vars
|
||||
run: |
|
||||
echo TEST_DATA_DIR=/blob/ >> $GITHUB_ENV
|
||||
echo HF_HOME=/blob/hf_home/ >> $GITHUB_ENV
|
||||
echo TORCH_EXTENSIONS_DIR=./torch-extensions/ >> $GITHUB_ENV
|
||||
echo TORCH_CACHE=/blob/torch_cache/ >> $GITHUB_ENV
|
||||
echo HF_DATASETS_CACHE=/blob/datasets_cache/ >> $GITHUB_ENV
|
||||
echo MEGATRON_CKPT_DIR=/blob/megatron_ckpt/ >> $GITHUB_ENV
|
||||
echo CRITIC_CKPT_DIR=/blob/step2_opt_125m_ckpt/ >> $GITHUB_ENV
|
||||
echo PYTEST_OPTS="--maxfail=100 --color=yes --durations=0 --verbose -rF" >> $GITHUB_ENV
|
||||
shell: bash
|
||||
- id: print-env
|
||||
run: |
|
||||
which python
|
||||
python --version
|
||||
if [[ -z "${AISC_NODE_INSTANCE_ID}" ]]; then
|
||||
echo "Not on self-hosted node"
|
||||
else
|
||||
echo "JobID: ${AISC_NODE_INSTANCE_ID}"
|
||||
fi
|
||||
if command -v nvidia-smi; then
|
||||
nvidia-smi
|
||||
which nvcc
|
||||
nvcc --version
|
||||
elif command -v rocm-smi; then
|
||||
rocm-smi --showhw
|
||||
which hipcc
|
||||
hipcc --version
|
||||
fi
|
||||
shell: bash
|
||||
@@ -0,0 +1,62 @@
|
||||
name: xpu-compile
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
schedule:
|
||||
- cron: "0 0 * * *"
|
||||
pull_request:
|
||||
paths:
|
||||
- ".github/workflows/xpu-compile.yml"
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
|
||||
jobs:
|
||||
compile-tests:
|
||||
name: xpu-compile / compile tests
|
||||
runs-on: [self-hosted, intel, xpu]
|
||||
container:
|
||||
image: intel/oneapi-basekit:2025.0.2-0-devel-ubuntu22.04
|
||||
ports:
|
||||
- 80
|
||||
options: --privileged -it --rm --device /dev/dri:/dev/dri -v /dev/dri/by-path:/dev/dri/by-path --ipc=host --cap-add=ALL
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Install prerequisite
|
||||
run: |
|
||||
apt-get update
|
||||
apt-get install clinfo libaio-dev python3-pip -y
|
||||
pip install torch==2.10.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/xpu
|
||||
pip install py-cpuinfo numpy
|
||||
pip install .[dev,autotuning]
|
||||
|
||||
- name: Check container state
|
||||
run: |
|
||||
ldd --version
|
||||
ds_report
|
||||
python3 -c "import torch; print('torch:', torch.__version__, torch)"
|
||||
python3 -c "import torch; print('XPU available:', torch.xpu.is_available())"
|
||||
python3 -c "from deepspeed.accelerator import get_accelerator; print('accelerator:', get_accelerator()._name)"
|
||||
pip list
|
||||
|
||||
- name: Compile Status
|
||||
shell: bash
|
||||
run: |
|
||||
echo "# torch.compile graph breaks" >> $GITHUB_STEP_SUMMARY
|
||||
export FI_HMEM=system
|
||||
ulimit -n 1048575
|
||||
cd tests/torch_compile
|
||||
export ZE_AFFINITY_MASK=0,1
|
||||
echo "## ZeRO stage 3" >> $GITHUB_STEP_SUMMARY
|
||||
deepspeed test_compile.py --deepspeed_config ds_config_z3.json 2>&1 | tee log_z3.txt
|
||||
# for each line start with 'dynamo_output', extract the second field and following fields and append to GITHUB_STEP_SUMMARY using awk
|
||||
cat log_z3.txt | awk '/^dynamo_output/ {$1=""; print $0}' >> $GITHUB_STEP_SUMMARY
|
||||
echo "## ZeRO stage 2" >> $GITHUB_STEP_SUMMARY
|
||||
deepspeed test_compile.py --deepspeed_config ds_config_z2.json 2>&1 | tee log_z2.txt
|
||||
cat log_z2.txt | awk '/^dynamo_output/ {$1=""; print $0}' >> $GITHUB_STEP_SUMMARY
|
||||
@@ -0,0 +1,85 @@
|
||||
name: xpu-max1100
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
schedule:
|
||||
- cron: "0 0 * * *"
|
||||
pull_request:
|
||||
paths:
|
||||
- ".github/workflows/xpu-max1100.yml"
|
||||
- "accelerator/xpu_accelerator.py"
|
||||
- "accelerator/abstract_accelerator.py"
|
||||
- "accelerator/cpu_accelerator.py"
|
||||
- "accelerator/real_accelerator.py"
|
||||
- "csrc/xpu/**"
|
||||
- "deepspeed/runtime/engine.py"
|
||||
- "deepspeed/runtime/bf16_optimizer.py"
|
||||
- "deepspeed/runtime/zero/stage_1_and_2.py"
|
||||
- "deepspeed/runtime/zero/stage3.py"
|
||||
- "deepspeed/runtime/zero/partition_parameters.py"
|
||||
- "deepspeed/runtime/zero/partitioned_param_coordinator.py"
|
||||
- "deepspeed/runtime/zero/parameter_offload.py"
|
||||
- "deepspeed/runtime/pipe/engine.py"
|
||||
- "deepspeed/runtime/utils.py"
|
||||
- "op_builder/xpu/**"
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
|
||||
|
||||
jobs:
|
||||
unit-tests:
|
||||
name: xpu-max1100 / unit tests
|
||||
runs-on: [self-hosted, intel, xpu]
|
||||
container:
|
||||
image: intel/oneapi-basekit:2025.0.2-0-devel-ubuntu22.04
|
||||
ports:
|
||||
- 80
|
||||
options: --privileged -it --rm --device /dev/dri:/dev/dri -v /dev/dri/by-path:/dev/dri/by-path --ipc=host --cap-add=ALL
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Install prerequisite
|
||||
shell: bash
|
||||
run: |
|
||||
apt-get update
|
||||
apt-get install -y python3.11 python3.11-dev python3-pip clinfo libaio-dev
|
||||
pip install --upgrade pip
|
||||
pip install py-cpuinfo
|
||||
pip install torch==2.10.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/xpu
|
||||
pip install .[dev,autotuning]
|
||||
|
||||
- name: Check container state
|
||||
shell: bash
|
||||
run: |
|
||||
ldd --version
|
||||
ds_report
|
||||
python3 -c "import torch; print('torch:', torch.__version__, torch)"
|
||||
python3 -c "import torch; print('XPU available:', torch.xpu.is_available())"
|
||||
python3 -c "from deepspeed.accelerator import get_accelerator; print('accelerator:', get_accelerator()._name)"
|
||||
pip list
|
||||
|
||||
- name: Unit tests
|
||||
shell: bash
|
||||
run: |
|
||||
cd tests/unit
|
||||
export FI_PROVIDER="tcp"
|
||||
export I_MPI_SHM=off
|
||||
pytest --verbose accelerator/*
|
||||
pytest --verbose autotuning/*
|
||||
pytest --verbose model_parallelism/*
|
||||
pytest --verbose monitor/*
|
||||
pytest --verbose utils/*
|
||||
pytest --verbose runtime/test_ds_config_model.py
|
||||
pytest --verbose runtime/pipe/test_pipe_schedule.py
|
||||
pytest --verbose runtime/zero/test_zero_config.py
|
||||
pytest --verbose runtime/zero/test_zero_tiled.py
|
||||
pytest --verbose runtime/zero/test_zeropp.py
|
||||
pytest --verbose runtime/test_autocast.py
|
||||
pytest --verbose runtime/test_data.py
|
||||
pytest --verbose runtime/zero/test_zero_dynamic_class.py
|
||||
Reference in New Issue
Block a user