wehub-resource-sync
255 lines
9.4 KiB
Markdown
255 lines
9.4 KiB
Markdown
# raydepsets
|
|
|
|
A dependency lock file management tool for Ray CI pipelines. It
|
|
maintains consistency relationships among lock files — ensuring that
|
|
when a dependency is updated, all related lock files are regenerated
|
|
together in the correct order. Built on top of
|
|
[`uv pip compile`](https://docs.astral.sh/uv/pip/compile/), it
|
|
generates reproducible, hash-verified lock files across multiple
|
|
Python versions, platforms, and CUDA variants.
|
|
|
|
## Why
|
|
|
|
Ray's CI builds containers and test environments for many
|
|
combinations of Python version, platform (Linux x86_64, macOS ARM),
|
|
and GPU support (CPU, e.g. CUDA 12.8). Each combination needs a
|
|
locked, reproducible set of dependencies, and many of these lock
|
|
files have consistency relationships with each other — for example,
|
|
a test environment's lock file must be a strict superset of the base
|
|
image it runs on, and all CUDA variants for a given Python version
|
|
must agree on common package versions. A single `uv pip compile`
|
|
call can produce one lock file, but it has no awareness of these
|
|
cross-file constraints. When a dependency is updated, all downstream
|
|
lock files need to be regenerated in the right order to stay
|
|
consistent.
|
|
|
|
raydepsets solves this by:
|
|
- Modeling the relationships between lock files as a dependency
|
|
graph, so that updating one file automatically propagates to all
|
|
dependents
|
|
- Supporting four composable operations (**compile**, **subset**,
|
|
**expand**, **relax**) to express how lock files derive from each other
|
|
- Defining dependency sets declaratively in YAML with template
|
|
variables for matrix builds
|
|
- Automatically resolving execution order via topological sort
|
|
- Validating that committed lock files are up-to-date and mutually
|
|
consistent in CI (`--check` mode)
|
|
|
|
## Directory Structure
|
|
|
|
```
|
|
ci/raydepsets/
|
|
├── raydepsets.py # Entry point
|
|
├── cli.py # CLI and DependencySetManager
|
|
├── workspace.py # Config parsing and data models
|
|
├── BUILD.bazel # Bazel build targets
|
|
├── configs/ # Production YAML configs
|
|
│ ├── rayimg.depsets.yaml
|
|
│ ├── rayllm.depsets.yaml
|
|
│ ├── data_test.depsets.yaml
|
|
│ ├── docs.depsets.yaml
|
|
│ └── ...
|
|
├── pre_hooks/ # Shell scripts run before compilation
|
|
│ ├── build-placeholder-wheel.sh
|
|
│ └── remove-compiled-headers.sh
|
|
└── tests/
|
|
├── test_cli.py
|
|
├── test_workspace.py
|
|
├── utils.py
|
|
└── test_data/
|
|
```
|
|
|
|
## Usage
|
|
|
|
raydepsets is built and run via Bazel:
|
|
|
|
```bash
|
|
# Build all depsets in a config
|
|
bazelisk run //ci/raydepsets:raydepsets -- build ci/raydepsets/configs/rayimg.depsets.yaml
|
|
|
|
# Build a single named depset (and its dependencies)
|
|
bazelisk run //ci/raydepsets:raydepsets -- build ci/raydepsets/configs/rayimg.depsets.yaml --name ray_img_depset_313
|
|
|
|
# Build all configs at once
|
|
bazelisk run //ci/raydepsets:raydepsets -- build --all-configs
|
|
|
|
# Validate that lock files are up-to-date (used in CI)
|
|
bazelisk run //ci/raydepsets:raydepsets -- build ci/raydepsets/configs/rayimg.depsets.yaml --check
|
|
```
|
|
|
|
### CLI Options
|
|
|
|
| Option | Description |
|
|
|---|---|
|
|
| `CONFIG_PATH` | Path to a `.depsets.yaml` config file (default: `ci/raydepsets/configs/*.depsets.yaml`) |
|
|
| `--workspace-dir` | Workspace root directory (default: `$BUILD_WORKSPACE_DIRECTORY`) |
|
|
| `--name` | Build only this depset and its dependencies |
|
|
| `--uv-cache-dir` | Cache directory for uv |
|
|
| `--check` | Validate lock files match what would be generated; exit non-zero on diff |
|
|
| `--all-configs` | Build depsets from all config files, not just the specified one |
|
|
|
|
## Configuration Format
|
|
|
|
Config files use the `.depsets.yaml` extension and contain two top-level keys:
|
|
|
|
### `build_arg_sets` (optional)
|
|
|
|
Defines template variable sets for matrix expansion. Each key maps to a dictionary of variable substitutions:
|
|
|
|
```yaml
|
|
build_arg_sets:
|
|
py311:
|
|
PYTHON_VERSION: "3.11"
|
|
PYTHON_SHORT: "311"
|
|
py312:
|
|
PYTHON_VERSION: "3.12"
|
|
PYTHON_SHORT: "312"
|
|
```
|
|
|
|
Variables are referenced in depset fields using `${VARIABLE_NAME}` syntax. When a depset lists multiple `build_arg_sets`, it is expanded into one depset per set.
|
|
|
|
### `depsets`
|
|
|
|
A list of dependency set definitions. Each depset has these common fields:
|
|
|
|
| Field | Type | Description |
|
|
|---|---|---|
|
|
| `name` | string | Unique identifier (supports `${VAR}` substitution) |
|
|
| `operation` | string | One of `compile`, `subset`, `expand`, `relax` |
|
|
| `output` | string | Output lock file path relative to workspace root |
|
|
| `build_arg_sets` | list | Which build arg sets to expand this depset with |
|
|
| `append_flags` | list | Additional flags passed to `uv pip compile` |
|
|
| `override_flags` | list | Flags that replace matching defaults |
|
|
| `pre_hooks` | list | Shell commands to run before this depset executes |
|
|
| `include_setuptools` | bool | Allow setuptools in output (default: `false`) |
|
|
|
|
#### Operation: `compile`
|
|
|
|
Runs `uv pip compile` to resolve and lock dependencies from requirements files.
|
|
|
|
```yaml
|
|
- name: ray_img_depset_${PYTHON_SHORT}
|
|
operation: compile
|
|
requirements:
|
|
- python/deplocks/ray_img/ray_dev.in
|
|
constraints:
|
|
- /tmp/ray-deps/requirements_compiled_py${PYTHON_VERSION}.txt
|
|
output: python/deplocks/ray_img/ray_img_py${PYTHON_SHORT}.lock
|
|
append_flags:
|
|
- --python-version=${PYTHON_VERSION}
|
|
build_arg_sets:
|
|
- py310
|
|
- py311
|
|
- py312
|
|
- py313
|
|
```
|
|
|
|
Additional fields: `requirements` (input requirement files), `constraints` (version constraint files), `packages` (inline package specs passed via stdin).
|
|
|
|
#### Operation: `subset`
|
|
|
|
Extracts a subset of already-resolved dependencies from another depset's lock file. Validates that all requested requirements exist in the source.
|
|
|
|
```yaml
|
|
- name: ray_base_deps_${PYTHON_SHORT}
|
|
operation: subset
|
|
source_depset: ray_base_extra_testdeps_${PYTHON_SHORT}
|
|
requirements:
|
|
- docker/base-deps/requirements.in
|
|
output: python/deplocks/base_deps/ray_base_deps_py${PYTHON_VERSION}.lock
|
|
```
|
|
|
|
Additional fields: `source_depset` (name of the depset to subset from).
|
|
|
|
#### Operation: `expand`
|
|
|
|
Combines multiple depsets into one, optionally adding new requirements. Recursively collects all transitive requirements from referenced depsets.
|
|
|
|
```yaml
|
|
- name: compiled_ray_llm_test_depset_${PYTHON_VERSION}_${CUDA_CODE}
|
|
operation: expand
|
|
depsets:
|
|
- ray_base_test_depset_${PYTHON_VERSION}_${CUDA_CODE}
|
|
requirements:
|
|
- python/requirements/llm/llm-requirements.txt
|
|
- python/requirements/llm/llm-test-requirements.txt
|
|
constraints:
|
|
- python/deplocks/llm/ray_test_${PYTHON_VERSION}_${CUDA_CODE}.lock
|
|
output: python/deplocks/llm/rayllm_test_${PYTHON_VERSION}_${CUDA_CODE}.lock
|
|
```
|
|
|
|
Additional fields: `depsets` (list of depset names to combine), `requirements` (extra requirements to include), `constraints` (constraint files).
|
|
|
|
#### Operation: `relax`
|
|
|
|
Removes specified packages from another depset's lock file. Validates that all specified packages exist in the source before removing them.
|
|
|
|
```yaml
|
|
- name: relaxed_depset_${PYTHON_SHORT}
|
|
operation: relax
|
|
source_depset: ray_img_depset_${PYTHON_SHORT}
|
|
packages:
|
|
- some-unwanted-package
|
|
- another-package
|
|
output: python/deplocks/ray_img/ray_img_relaxed_py${PYTHON_SHORT}.lock
|
|
```
|
|
|
|
Additional fields: `source_depset` (name of the depset to relax from), `packages` (list of package names to remove from the lock file).
|
|
|
|
> **Warning:** This operation performs a simple removal of packages from the lock file and does not re-evaluate the dependency graph. Removing a package that is required by another package in the lock file may result in an inconsistent environment. Use with caution.
|
|
|
|
### YAML Anchors
|
|
|
|
Configs support standard YAML anchors for DRY definitions:
|
|
|
|
```yaml
|
|
.common_settings: &common_settings
|
|
append_flags:
|
|
- --python-version=3.11
|
|
- --unsafe-package ray
|
|
build_arg_sets:
|
|
- cpu
|
|
- cu128
|
|
|
|
depsets:
|
|
- name: my_depset
|
|
<<: *common_settings
|
|
operation: compile
|
|
requirements:
|
|
- requirements.txt
|
|
output: output.lock
|
|
```
|
|
|
|
## Pre-Hooks
|
|
|
|
Pre-hooks are shell scripts that run before a depset is executed. They are useful for preparing the build environment (e.g., building placeholder wheels, stripping GPU index URLs from constraint files).
|
|
|
|
```yaml
|
|
pre_hooks:
|
|
- ci/raydepsets/pre_hooks/build-placeholder-wheel.sh
|
|
- ci/raydepsets/pre_hooks/remove-compiled-headers.sh ${PYTHON_VERSION}
|
|
```
|
|
|
|
Pre-hooks support template variable substitution and are modeled as nodes in the dependency graph, so they execute in the correct order.
|
|
|
|
## How It Works
|
|
|
|
1. **Config loading** -- YAML configs are parsed into `Depset` dataclasses. Template variables from `build_arg_sets` are substituted, expanding one depset definition into N concrete depsets.
|
|
2. **Graph construction** -- A directed acyclic graph (NetworkX `DiGraph`) is built from depset dependencies and pre-hooks.
|
|
3. **Topological execution** -- Depsets are executed in topological order so dependencies are resolved before dependents.
|
|
4. **Lock file generation** -- Each depset calls `uv pip compile` with the appropriate flags, constraints, and requirements.
|
|
5. **Validation** (`--check` mode) -- Lock files are generated to a temp directory and compared against committed versions. Any diff causes a non-zero exit.
|
|
|
|
### Default `uv pip compile` Flags
|
|
|
|
```
|
|
--no-header
|
|
--generate-hashes
|
|
--index-strategy unsafe-best-match
|
|
--no-strip-markers
|
|
--emit-index-url
|
|
--emit-find-links
|
|
--quiet
|
|
--unsafe-package setuptools (unless include_setuptools: true)
|
|
```
|