chore: import upstream snapshot with attribution
Lockfile supply-chain audit / lockfile supply-chain audit (push) Has been cancelled
Windows Studio GGUF CI / GPU prebuilt resolves without Visual Studio (push) Has been cancelled
Windows Studio GGUF CI / setup.ps1 unit tests (VS 2026 / CMake guard) (push) Has been cancelled
Windows Studio GGUF CI / real-VS detection (VS 2022) (push) Has been cancelled
Windows Studio GGUF CI / real-VS detection (VS 2026) (push) Has been cancelled
Windows Studio GGUF CI / VC++ runtime detect + install round-trip (windows-2025-vs2026) (push) Has been cancelled
Windows Studio GGUF CI / VC++ runtime detect + install round-trip (windows-latest) (push) Has been cancelled
Windows Studio Update CI / Studio Updating Tests (push) Has been cancelled
Wheel CI / Wheel build + content sanity + import smoke (push) Has been cancelled
Lint CI / Source lint (Python + shell + YAML + JSON + safety nets) (push) Has been cancelled
MLX CI on Mac M1 / dispatch (push) Has been cancelled
Security audit / advisory audit (pip + npm + cargo) (push) Has been cancelled
Security audit / pip scan-packages :: extras (push) Has been cancelled
Security audit / pip scan-packages :: studio (push) Has been cancelled
Security audit / pip scan-packages :: hf-stack (push) Has been cancelled
Security audit / npm scan-packages (Studio frontend tarballs) (push) Has been cancelled
Security audit / workflow-trigger lint (pull_request_target / cache-poisoning) (push) Has been cancelled
Security audit / pytest tests/security (push) Has been cancelled
Security audit / npm provenance + new install-script diff (push) Has been cancelled
Studio API CI / Studio API & Auth Tests (push) Has been cancelled
Backend CI / (Python 3.10) (push) Has been cancelled
Backend CI / (Python 3.11) (push) Has been cancelled
Backend CI / (Python 3.12) (push) Has been cancelled
Backend CI / (Python 3.13) (push) Has been cancelled
Backend CI / Repo tests (CPU) (push) Has been cancelled
Frontend CI / Frontend build + bundle sanity (push) Has been cancelled
Studio GGUF CI / OpenAI, Anthropic API tests (push) Has been cancelled
Studio GGUF CI / Tool calling Tests (push) Has been cancelled
Studio GGUF CI / JSON, images (push) Has been cancelled
Mac Studio GGUF CI / OpenAI, Anthropic API tests (push) Has been cancelled
Mac Studio GGUF CI / Tool calling Tests (push) Has been cancelled
Mac Studio GGUF CI / JSON, images (push) Has been cancelled
Mac Studio Install Matrix CI / Install + load (macos-14) (push) Has been cancelled
Mac Studio Install Matrix CI / Install + load (macos-15) (push) Has been cancelled
Mac Studio Install Matrix CI / Install + load (macos-26) (push) Has been cancelled
Mac Studio Install Matrix CI / Install + load (macos-15-intel) (push) Has been cancelled
Mac Studio API CI / Studio API & Auth Tests (push) Has been cancelled
Mac Studio Install Matrix CI / Install + load (macos-26-intel) (push) Has been cancelled
Mac Studio UI CI / Chat UI Tests (push) Has been cancelled
Studio Tauri CI / Tauri Linux debug build (no codesign) (push) Has been cancelled
Mac Studio Update CI / Studio Updating Tests (push) Has been cancelled
Studio UI CI / Chat UI Tests (push) Has been cancelled
Windows Studio API CI / Studio API & Auth Tests (push) Has been cancelled
Windows Studio UI CI / Chat UI Tests (push) Has been cancelled
Studio Update CI / Studio Updating Tests (push) Has been cancelled
Core / Core (HF=default + TRL=default) (push) Has been cancelled
Core / Core (HF=4.57.6 + TRL<1) (push) Has been cancelled
Core / Core (HF=latest + TRL=latest) (push) Has been cancelled
Core / llama.cpp build + smoke (push) Has been cancelled
Windows Studio GGUF CI / OpenAI, Anthropic API tests (push) Has been cancelled
Windows Studio GGUF CI / Tool calling Tests (push) Has been cancelled
Windows Studio GGUF CI / JSON, images (push) Has been cancelled
Windows Studio GGUF CI / Studio install + inference without Visual Studio (push) Has been cancelled
Studio export capability / capability (macos-latest) (push) Has been cancelled
Studio export capability / capability (ubuntu-latest) (push) Has been cancelled
Studio export capability / capability (windows-latest) (push) Has been cancelled
Cross-platform parity / parity (macos-latest) (push) Has been cancelled
Cross-platform parity / parity (windows-latest) (push) Has been cancelled
Scorecard supply-chain security / Scorecard analysis (push) Has been cancelled
Studio load-orchestrator CI / test (push) Has been cancelled
Lockfile supply-chain audit / lockfile supply-chain audit (push) Has been cancelled
Windows Studio GGUF CI / GPU prebuilt resolves without Visual Studio (push) Has been cancelled
Windows Studio GGUF CI / setup.ps1 unit tests (VS 2026 / CMake guard) (push) Has been cancelled
Windows Studio GGUF CI / real-VS detection (VS 2022) (push) Has been cancelled
Windows Studio GGUF CI / real-VS detection (VS 2026) (push) Has been cancelled
Windows Studio GGUF CI / VC++ runtime detect + install round-trip (windows-2025-vs2026) (push) Has been cancelled
Windows Studio GGUF CI / VC++ runtime detect + install round-trip (windows-latest) (push) Has been cancelled
Windows Studio Update CI / Studio Updating Tests (push) Has been cancelled
Wheel CI / Wheel build + content sanity + import smoke (push) Has been cancelled
Lint CI / Source lint (Python + shell + YAML + JSON + safety nets) (push) Has been cancelled
MLX CI on Mac M1 / dispatch (push) Has been cancelled
Security audit / advisory audit (pip + npm + cargo) (push) Has been cancelled
Security audit / pip scan-packages :: extras (push) Has been cancelled
Security audit / pip scan-packages :: studio (push) Has been cancelled
Security audit / pip scan-packages :: hf-stack (push) Has been cancelled
Security audit / npm scan-packages (Studio frontend tarballs) (push) Has been cancelled
Security audit / workflow-trigger lint (pull_request_target / cache-poisoning) (push) Has been cancelled
Security audit / pytest tests/security (push) Has been cancelled
Security audit / npm provenance + new install-script diff (push) Has been cancelled
Studio API CI / Studio API & Auth Tests (push) Has been cancelled
Backend CI / (Python 3.10) (push) Has been cancelled
Backend CI / (Python 3.11) (push) Has been cancelled
Backend CI / (Python 3.12) (push) Has been cancelled
Backend CI / (Python 3.13) (push) Has been cancelled
Backend CI / Repo tests (CPU) (push) Has been cancelled
Frontend CI / Frontend build + bundle sanity (push) Has been cancelled
Studio GGUF CI / OpenAI, Anthropic API tests (push) Has been cancelled
Studio GGUF CI / Tool calling Tests (push) Has been cancelled
Studio GGUF CI / JSON, images (push) Has been cancelled
Mac Studio GGUF CI / OpenAI, Anthropic API tests (push) Has been cancelled
Mac Studio GGUF CI / Tool calling Tests (push) Has been cancelled
Mac Studio GGUF CI / JSON, images (push) Has been cancelled
Mac Studio Install Matrix CI / Install + load (macos-14) (push) Has been cancelled
Mac Studio Install Matrix CI / Install + load (macos-15) (push) Has been cancelled
Mac Studio Install Matrix CI / Install + load (macos-26) (push) Has been cancelled
Mac Studio Install Matrix CI / Install + load (macos-15-intel) (push) Has been cancelled
Mac Studio API CI / Studio API & Auth Tests (push) Has been cancelled
Mac Studio Install Matrix CI / Install + load (macos-26-intel) (push) Has been cancelled
Mac Studio UI CI / Chat UI Tests (push) Has been cancelled
Studio Tauri CI / Tauri Linux debug build (no codesign) (push) Has been cancelled
Mac Studio Update CI / Studio Updating Tests (push) Has been cancelled
Studio UI CI / Chat UI Tests (push) Has been cancelled
Windows Studio API CI / Studio API & Auth Tests (push) Has been cancelled
Windows Studio UI CI / Chat UI Tests (push) Has been cancelled
Studio Update CI / Studio Updating Tests (push) Has been cancelled
Core / Core (HF=default + TRL=default) (push) Has been cancelled
Core / Core (HF=4.57.6 + TRL<1) (push) Has been cancelled
Core / Core (HF=latest + TRL=latest) (push) Has been cancelled
Core / llama.cpp build + smoke (push) Has been cancelled
Windows Studio GGUF CI / OpenAI, Anthropic API tests (push) Has been cancelled
Windows Studio GGUF CI / Tool calling Tests (push) Has been cancelled
Windows Studio GGUF CI / JSON, images (push) Has been cancelled
Windows Studio GGUF CI / Studio install + inference without Visual Studio (push) Has been cancelled
Studio export capability / capability (macos-latest) (push) Has been cancelled
Studio export capability / capability (ubuntu-latest) (push) Has been cancelled
Studio export capability / capability (windows-latest) (push) Has been cancelled
Cross-platform parity / parity (macos-latest) (push) Has been cancelled
Cross-platform parity / parity (windows-latest) (push) Has been cancelled
Scorecard supply-chain security / Scorecard analysis (push) Has been cancelled
Studio load-orchestrator CI / test (push) Has been cancelled
This commit is contained in:
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,242 @@
|
||||
#!/usr/bin/env python3
|
||||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
|
||||
|
||||
"""Diff two `package-lock.json` files and flag NEW install-script deps.
|
||||
|
||||
A `"hasInstallScript": true` package runs preinstall/install/postinstall
|
||||
hooks on every `npm ci` -- the lever behind recent npm supply-chain
|
||||
compromises (attacker publishes a malicious version of a trusted dep).
|
||||
This refuses to land a newly-introduced install-script dep without a
|
||||
maintainer eyeball; pre-existing ones are not re-flagged.
|
||||
|
||||
Supports lockfileVersion 1 (recursive `dependencies`) and 2/3 (flat
|
||||
`packages` with `node_modules/.../node_modules/...` nesting). For each
|
||||
new entry we best-effort fetch the registry metadata to recover the
|
||||
postinstall command body; the finding is still emitted if unreachable.
|
||||
|
||||
Exit codes: 0 = none; 1 = one or more (on stderr); 2 = internal error.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
REGISTRY_BASE = "https://registry.npmjs.org/"
|
||||
REGISTRY_TIMEOUT_SECS = 5
|
||||
|
||||
CRITICAL = "CRITICAL"
|
||||
HIGH = "HIGH"
|
||||
|
||||
|
||||
class Finding:
|
||||
__slots__ = ("severity", "name", "version", "kind", "detail")
|
||||
|
||||
def __init__(self, severity: str, name: str, version: str, kind: str, detail: str) -> None:
|
||||
self.severity = severity
|
||||
self.name = name
|
||||
self.version = version
|
||||
self.kind = kind
|
||||
self.detail = detail
|
||||
|
||||
def __str__(self) -> str:
|
||||
return (
|
||||
f" [{self.severity}] {self.name}@{self.version}\n"
|
||||
f" kind: {self.kind}\n"
|
||||
f" detail: {self.detail}"
|
||||
)
|
||||
|
||||
|
||||
# Lockfile parsing.
|
||||
|
||||
|
||||
def _strip_nm_prefix(key: str) -> str:
|
||||
"""Convert a v2/v3 `packages` key into a bare package name (leaf after last `node_modules/`)."""
|
||||
if not key:
|
||||
return ""
|
||||
# LAST node_modules/ segment so transitives map to their leaf name.
|
||||
marker = "node_modules/"
|
||||
idx = key.rfind(marker)
|
||||
if idx == -1:
|
||||
return key
|
||||
return key[idx + len(marker) :]
|
||||
|
||||
|
||||
def _collect_install_script_entries(lock: dict) -> dict[str, str]:
|
||||
"""Return {name@version: name} for entries with hasInstallScript (v2/v3) or a lifecycle script (v1).
|
||||
|
||||
Keyed by name@version so dup copies at different versions aren't lost.
|
||||
"""
|
||||
seen: dict[str, str] = {}
|
||||
version = lock.get("lockfileVersion")
|
||||
|
||||
# v2 / v3: flat `packages` map.
|
||||
packages = lock.get("packages") or {}
|
||||
for key, entry in packages.items():
|
||||
if key == "" or not isinstance(entry, dict):
|
||||
continue
|
||||
if entry.get("link"):
|
||||
continue
|
||||
if not entry.get("hasInstallScript"):
|
||||
continue
|
||||
name = _strip_nm_prefix(key)
|
||||
if not name:
|
||||
continue
|
||||
ver = entry.get("version") or "<unversioned>"
|
||||
seen[f"{name}@{ver}"] = name
|
||||
|
||||
# v1 has no hasInstallScript flag; detect lifecycle scripts directly.
|
||||
def _walk_v1(deps: dict, depth: int = 0) -> None:
|
||||
if depth > 64 or not isinstance(deps, dict):
|
||||
return
|
||||
for name, entry in deps.items():
|
||||
if not isinstance(entry, dict):
|
||||
continue
|
||||
scripts = entry.get("scripts") or {}
|
||||
lifecycle = any(
|
||||
isinstance(scripts, dict) and scripts.get(hook)
|
||||
for hook in ("preinstall", "install", "postinstall")
|
||||
)
|
||||
if lifecycle:
|
||||
ver = entry.get("version") or "<unversioned>"
|
||||
seen[f"{name}@{ver}"] = name
|
||||
_walk_v1(entry.get("dependencies"), depth = depth + 1)
|
||||
|
||||
if version == 1 or "dependencies" in lock:
|
||||
_walk_v1(lock.get("dependencies") or {})
|
||||
|
||||
return seen
|
||||
|
||||
|
||||
def _load_lockfile(path: Path) -> dict:
|
||||
if not path.exists():
|
||||
raise FileNotFoundError(f"lockfile not found: {path}")
|
||||
try:
|
||||
return json.loads(path.read_text(encoding = "utf-8"))
|
||||
except json.JSONDecodeError as exc:
|
||||
raise ValueError(f"{path}: not valid JSON: {exc}") from exc
|
||||
|
||||
|
||||
# Registry lookup for the postinstall command body (best-effort).
|
||||
|
||||
|
||||
def _fetch_registry_scripts(name: str, version: str) -> dict[str, str] | None:
|
||||
"""Return {hook: command} for lifecycle hooks in registry metadata; None on any error (never raises)."""
|
||||
safe_name = urllib.parse.quote(name, safe = "@/")
|
||||
url = f"{REGISTRY_BASE}{safe_name}/{urllib.parse.quote(version)}"
|
||||
try:
|
||||
with urllib.request.urlopen(url, timeout = REGISTRY_TIMEOUT_SECS) as resp:
|
||||
body = resp.read()
|
||||
except (urllib.error.URLError, OSError, ValueError, TimeoutError):
|
||||
return None
|
||||
try:
|
||||
meta = json.loads(body)
|
||||
except json.JSONDecodeError:
|
||||
return None
|
||||
scripts = meta.get("scripts") or {}
|
||||
if not isinstance(scripts, dict):
|
||||
return None
|
||||
keep = {}
|
||||
for hook in ("preinstall", "install", "postinstall"):
|
||||
cmd = scripts.get(hook)
|
||||
if isinstance(cmd, str) and cmd.strip():
|
||||
keep[hook] = cmd
|
||||
return keep or None
|
||||
|
||||
|
||||
# Diff.
|
||||
|
||||
|
||||
def diff_new_install_scripts(base_lock: dict, head_lock: dict) -> list[Finding]:
|
||||
base = _collect_install_script_entries(base_lock)
|
||||
head = _collect_install_script_entries(head_lock)
|
||||
findings: list[Finding] = []
|
||||
for key in sorted(head):
|
||||
if key in base:
|
||||
continue # pre-existing install-script dep; not in scope
|
||||
name = head[key]
|
||||
version = key[len(name) + 1 :] if key.startswith(name + "@") else "<unversioned>"
|
||||
scripts = _fetch_registry_scripts(name, version)
|
||||
if scripts:
|
||||
detail = "; ".join(f"{h}={cmd!r}" for h, cmd in scripts.items())
|
||||
else:
|
||||
detail = (
|
||||
"newly added with hasInstallScript=true; registry "
|
||||
"metadata unreachable -- inspect the package's "
|
||||
"scripts.{preinstall,install,postinstall} manually"
|
||||
)
|
||||
findings.append(
|
||||
Finding(
|
||||
severity = CRITICAL,
|
||||
name = name,
|
||||
version = version,
|
||||
kind = "new-install-script",
|
||||
detail = detail,
|
||||
)
|
||||
)
|
||||
return findings
|
||||
|
||||
|
||||
# CLI.
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description = (
|
||||
"Diff two package-lock.json files and refuse any newly-added install-script dep."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--base",
|
||||
required = True,
|
||||
help = "Path to the BASE package-lock.json (e.g. main branch).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--head",
|
||||
required = True,
|
||||
help = "Path to the HEAD package-lock.json (this PR).",
|
||||
)
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
try:
|
||||
base_lock = _load_lockfile(Path(args.base))
|
||||
head_lock = _load_lockfile(Path(args.head))
|
||||
except (FileNotFoundError, ValueError) as exc:
|
||||
print(f"[install-script-diff] ERROR: {exc}", file = sys.stderr)
|
||||
return 2
|
||||
|
||||
findings = diff_new_install_scripts(base_lock, head_lock)
|
||||
if not findings:
|
||||
print(
|
||||
"[install-script-diff] OK: no newly-added install-script "
|
||||
"dependencies between base and head",
|
||||
flush = True,
|
||||
)
|
||||
return 0
|
||||
|
||||
print(
|
||||
f"\n[install-script-diff] FAIL: {len(findings)} newly-added "
|
||||
f"install-script dependency(ies):\n",
|
||||
file = sys.stderr,
|
||||
)
|
||||
for f in findings:
|
||||
print(str(f), file = sys.stderr)
|
||||
print(file = sys.stderr)
|
||||
print(
|
||||
"[install-script-diff] Refusing to proceed. Every new "
|
||||
"install-script dep is a postinstall lifecycle hook that "
|
||||
"would run on the next `npm ci`. Review each finding above, "
|
||||
"confirm the maintainer + version, and re-run.",
|
||||
file = sys.stderr,
|
||||
)
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,9 @@
|
||||
# Do not modify this file directly; it is generated by extract_colabx_testing_tarballs.sh via
|
||||
# $ (lsb_release -ds;python --version;) > os-info-gpu.txt
|
||||
# Be aware that this list does not necessarily reflect the current state of the
|
||||
# staging or production container, but rather the state as of the most recent
|
||||
# submitted CL where extract_colabx_testing_tarballs.sh was run.
|
||||
Ubuntu 22.04.5 LTS
|
||||
Python 3.12.13
|
||||
R version 4.5.3 (2026-03-11) -- "Reassured Reassurer"
|
||||
julia version 1.12.6
|
||||
@@ -0,0 +1,731 @@
|
||||
# Do not modify this file directly; it is generated by extract_colabx_testing_tarballs.sh via
|
||||
# $ python3 -m pip freeze
|
||||
# Be aware that this list does not necessarily reflect the current state of the
|
||||
# staging or production container, but rather the state as of the most recent
|
||||
# submitted CL where extract_colabx_testing_tarballs.sh was run.
|
||||
absl-py==1.4.0
|
||||
accelerate==1.13.0
|
||||
access==1.1.10.post3
|
||||
affine==2.4.0
|
||||
aiofiles==24.1.0
|
||||
aiohappyeyeballs==2.6.1
|
||||
aiohttp==3.13.5
|
||||
aiosignal==1.4.0
|
||||
aiosqlite==0.22.1
|
||||
alabaster==1.0.0
|
||||
albucore==0.0.24
|
||||
albumentations==2.0.8
|
||||
ale-py==0.11.2
|
||||
alembic==1.18.4
|
||||
altair==5.5.0
|
||||
annotated-doc==0.0.4
|
||||
annotated-types==0.7.0
|
||||
antlr4-python3-runtime==4.9.3
|
||||
anyio==4.13.0
|
||||
anywidget==0.9.21
|
||||
apsw==3.53.0.0
|
||||
apswutils==0.1.2
|
||||
argon2-cffi==25.1.0
|
||||
argon2-cffi-bindings==25.1.0
|
||||
array_record==0.8.3
|
||||
arrow==1.4.0
|
||||
arviz==0.22.0
|
||||
astropy==7.2.0
|
||||
astropy-iers-data==0.2026.4.20.0.58.15
|
||||
astunparse==1.6.3
|
||||
atpublic==5.1
|
||||
attrs==26.1.0
|
||||
audioread==3.1.0
|
||||
Authlib==1.6.11
|
||||
autograd==1.8.0
|
||||
babel==2.18.0
|
||||
backcall==0.2.0
|
||||
beartype==0.22.9
|
||||
beautifulsoup4==4.13.5
|
||||
betterproto==2.0.0b6
|
||||
bigframes==2.39.0
|
||||
bigquery-magics==0.14.0
|
||||
bleach==6.3.0
|
||||
blinker==1.9.0
|
||||
blis==1.3.3
|
||||
blobfile==3.2.0
|
||||
blosc2==4.1.2
|
||||
bokeh==3.8.2
|
||||
Bottleneck==1.4.2
|
||||
bqplot==0.12.45
|
||||
branca==0.8.2
|
||||
brotli==1.2.0
|
||||
CacheControl==0.14.4
|
||||
cachetools==6.2.6
|
||||
catalogue==2.0.10
|
||||
certifi==2026.4.22
|
||||
cffi==2.0.0
|
||||
chardet==5.2.0
|
||||
charset-normalizer==3.4.7
|
||||
clarabel==0.11.1
|
||||
click==8.3.3
|
||||
click-plugins==1.1.1.2
|
||||
cligj==0.7.2
|
||||
cloudpathlib==0.23.0
|
||||
cloudpickle==3.1.2
|
||||
cmake==3.31.10
|
||||
cmdstanpy==1.3.0
|
||||
colorcet==3.1.0
|
||||
colorlover==0.3.0
|
||||
community==1.0.0b1
|
||||
confection==1.3.3
|
||||
cons==0.4.7
|
||||
contourpy==1.3.3
|
||||
cramjam==2.11.0
|
||||
cryptography==43.0.3
|
||||
cucim-cu12 @ https://pypi.nvidia.com/cucim-cu12/cucim_cu12-26.2.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
|
||||
cuda-bindings==12.9.4
|
||||
cuda-core==0.3.2
|
||||
cuda-pathfinder==1.5.3
|
||||
cuda-python==12.9.4
|
||||
cuda-toolkit==12.8.1
|
||||
cudf-cu12==26.2.1
|
||||
cudf-polars-cu12==26.2.1
|
||||
cufflinks==0.17.3
|
||||
cuml-cu12==26.2.0
|
||||
cupy-cuda12x==14.0.1
|
||||
curl_cffi==0.15.0
|
||||
cuvs-cu12 @ https://pypi.nvidia.com/cuvs-cu12/cuvs_cu12-26.2.0-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
|
||||
cvxopt==1.3.2
|
||||
cvxpy==1.6.7
|
||||
cycler==0.12.1
|
||||
cyipopt==1.5.0
|
||||
cymem==2.0.13
|
||||
Cython==3.0.12
|
||||
dask==2026.1.1
|
||||
dask-cuda==26.2.0
|
||||
dask-cudf-cu12==26.2.1
|
||||
dataproc-spark-connect==1.1.0
|
||||
datasets==4.0.0
|
||||
db-dtypes==1.5.1
|
||||
dbus-python==1.2.18
|
||||
debugpy==1.8.15
|
||||
decorator==4.4.2
|
||||
defusedxml==0.7.1
|
||||
deprecation==2.1.0
|
||||
diffusers==0.37.1
|
||||
dill==0.3.8
|
||||
distributed==2026.1.1
|
||||
distributed-ucxx-cu12==0.48.0
|
||||
distro==1.9.0
|
||||
dlib==19.24.6
|
||||
dm-tree==0.1.10
|
||||
docstring_parser==0.18.0
|
||||
docutils==0.21.2
|
||||
dopamine_rl==4.1.2
|
||||
duckdb==1.3.2
|
||||
earthengine-api==1.7.22
|
||||
easydict==1.13
|
||||
editdistance==0.8.1
|
||||
eerepr==0.1.2
|
||||
einops==0.8.2
|
||||
en_core_web_sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl#sha256=1932429db727d4bff3deed6b34cfc05df17794f4a52eeb26cf8928f7c1a0fb85
|
||||
entrypoints==0.4
|
||||
esda==2.9.0
|
||||
et_xmlfile==2.0.0
|
||||
etils==1.14.0
|
||||
etuples==0.3.10
|
||||
Farama-Notifications==0.0.4
|
||||
fastai==2.8.7
|
||||
fastapi==0.136.1
|
||||
fastcore==1.12.42
|
||||
fastdownload==0.0.7
|
||||
fastjsonschema==2.21.2
|
||||
fastlite==0.2.4
|
||||
fastprogress==1.1.5
|
||||
fasttransform==0.0.2
|
||||
ffmpy==1.0.0
|
||||
filelock==3.29.0
|
||||
fiona==1.10.1
|
||||
firebase-admin==6.9.0
|
||||
Flask==3.1.3
|
||||
flatbuffers==25.12.19
|
||||
flax==0.11.2
|
||||
folium==0.20.0
|
||||
fonttools==4.62.1
|
||||
fqdn==1.5.1
|
||||
frozendict==2.4.7
|
||||
frozenlist==1.8.0
|
||||
fsspec==2025.3.0
|
||||
future==1.0.0
|
||||
gast==0.7.0
|
||||
gcsfs==2025.3.0
|
||||
GDAL==3.8.4
|
||||
gdown==5.2.2
|
||||
geemap==0.37.2
|
||||
geocoder==1.38.1
|
||||
geographiclib==2.1
|
||||
geopandas==1.1.3
|
||||
geopy==2.4.1
|
||||
giddy==2.3.6
|
||||
gin-config==0.5.0
|
||||
gitdb==4.0.12
|
||||
GitPython==3.1.47
|
||||
glob2==0.7
|
||||
google==3.0.0
|
||||
google-adk==1.29.0
|
||||
google-ai-generativelanguage==0.6.15
|
||||
google-api-core==2.30.3
|
||||
google-api-python-client==2.194.0
|
||||
google-auth==2.47.0
|
||||
google-auth-httplib2==0.3.1
|
||||
google-auth-oauthlib==1.3.1
|
||||
google-cloud-aiplatform==1.148.1
|
||||
google-cloud-appengine-logging==1.9.0
|
||||
google-cloud-audit-log==0.5.0
|
||||
google-cloud-bigquery==3.41.0
|
||||
google-cloud-bigquery-connection==1.21.0
|
||||
google-cloud-bigquery-storage==2.37.0
|
||||
google-cloud-bigtable==2.36.0
|
||||
google-cloud-core==2.5.1
|
||||
google-cloud-dataplex==2.18.0
|
||||
google-cloud-dataproc==5.27.0
|
||||
google-cloud-datastore==2.24.0
|
||||
google-cloud-discoveryengine==0.13.12
|
||||
google-cloud-firestore==2.27.0
|
||||
google-cloud-functions==1.23.0
|
||||
google-cloud-iam==2.22.0
|
||||
google-cloud-language==2.20.0
|
||||
google-cloud-logging==3.15.0
|
||||
google-cloud-monitoring==2.30.0
|
||||
google-cloud-pubsub==2.37.0
|
||||
google-cloud-resource-manager==1.17.0
|
||||
google-cloud-secret-manager==2.27.0
|
||||
google-cloud-spanner==3.65.0
|
||||
google-cloud-speech==2.38.0
|
||||
google-cloud-storage==3.10.1
|
||||
google-cloud-trace==1.19.0
|
||||
google-cloud-translate==3.26.0
|
||||
google-colab @ file:///colabtools/dist/google_colab-1.0.0.tar.gz
|
||||
google-crc32c==1.8.0
|
||||
google-genai==1.68.0
|
||||
google-generativeai==0.8.6
|
||||
google-pasta==0.2.0
|
||||
google-resumable-media==2.8.2
|
||||
googleapis-common-protos==1.74.0
|
||||
googledrivedownloader==1.1.0
|
||||
gradio==5.50.0
|
||||
gradio_client==1.14.0
|
||||
grain==0.2.16
|
||||
graphviz==0.21
|
||||
greenlet==3.4.0
|
||||
groovy==0.1.2
|
||||
grpc-google-iam-v1==0.14.4
|
||||
grpc-interceptor==0.15.4
|
||||
grpcio==1.80.0
|
||||
grpcio-status==1.71.2
|
||||
grpclib==0.4.9
|
||||
gspread==6.2.1
|
||||
gspread-dataframe==4.0.0
|
||||
gym==0.25.2
|
||||
gym-notices==0.1.0
|
||||
gymnasium==1.3.0
|
||||
h11==0.16.0
|
||||
h2==4.3.0
|
||||
h5netcdf==1.8.1
|
||||
h5py==3.16.0
|
||||
hdbscan==0.8.42
|
||||
hf-xet==1.4.3
|
||||
highspy==1.14.0
|
||||
holidays==0.95
|
||||
holoviews==1.22.1
|
||||
hpack==4.1.0
|
||||
html5lib==1.1
|
||||
httpcore==1.0.9
|
||||
httpimport==1.4.1
|
||||
httplib2==0.31.2
|
||||
httptools==0.7.1
|
||||
httpx==0.28.1
|
||||
httpx-sse==0.4.3
|
||||
huggingface_hub==1.11.0
|
||||
humanize==4.15.0
|
||||
hyperframe==6.1.0
|
||||
hyperopt==0.2.7
|
||||
ibis-framework==9.5.0
|
||||
idna==3.13
|
||||
ImageIO==2.37.3
|
||||
imageio-ffmpeg==0.6.0
|
||||
imagesize==2.0.0
|
||||
imbalanced-learn==0.14.1
|
||||
immutabledict==4.3.1
|
||||
importlib_metadata==8.7.1
|
||||
importlib_resources==7.1.0
|
||||
imutils==0.5.4
|
||||
inequality==1.1.2
|
||||
inflect==7.5.0
|
||||
iniconfig==2.3.0
|
||||
intel-cmplr-lib-ur==2025.3.3
|
||||
intel-openmp==2025.3.3
|
||||
ipyevents==2.0.4
|
||||
ipyfilechooser==0.6.0
|
||||
ipykernel==6.17.1
|
||||
ipyleaflet==0.20.0
|
||||
ipyparallel==8.8.0
|
||||
ipython==7.34.0
|
||||
ipython-genutils==0.2.0
|
||||
ipython-sql==0.5.0
|
||||
ipywidgets==7.7.1
|
||||
isoduration==20.11.0
|
||||
itsdangerous==2.2.0
|
||||
jaraco.classes==3.4.0
|
||||
jaraco.context==6.1.2
|
||||
jaraco.functools==4.4.0
|
||||
jax==0.7.2
|
||||
jax-cuda12-pjrt==0.7.2
|
||||
jax-cuda12-plugin==0.7.2
|
||||
jaxlib==0.7.2
|
||||
jeepney==0.9.0
|
||||
jieba==0.42.1
|
||||
Jinja2==3.1.6
|
||||
jiter==0.14.0
|
||||
joblib==1.5.3
|
||||
jsonpatch==1.33
|
||||
jsonpickle==4.1.1
|
||||
jsonpointer==3.1.1
|
||||
jsonschema==4.26.0
|
||||
jsonschema-specifications==2025.9.1
|
||||
jupyter-console==6.6.3
|
||||
jupyter-events==0.12.1
|
||||
jupyter-leaflet==0.20.0
|
||||
jupyter_client==7.4.9
|
||||
jupyter_core==5.9.1
|
||||
jupyter_kernel_gateway @ git+https://github.com/googlecolab/kernel_gateway@b134e9945df25c2dcb98ade9129399be10788671
|
||||
jupyter_server==2.14.0
|
||||
jupyter_server_terminals==0.5.4
|
||||
jupyterlab_pygments==0.3.0
|
||||
jupyterlab_widgets==3.0.16
|
||||
jupytext==1.19.1
|
||||
kaggle==2.0.2
|
||||
kagglehub==1.0.0
|
||||
kagglesdk==0.1.20
|
||||
keras==3.13.2
|
||||
keras-hub==0.26.0
|
||||
keras-nlp==0.26.0
|
||||
keyring==25.7.0
|
||||
keyrings.google-artifactregistry-auth==1.1.2
|
||||
kiwisolver==1.5.0
|
||||
langchain==1.2.15
|
||||
langchain-core==1.3.1
|
||||
langgraph==1.1.9
|
||||
langgraph-checkpoint==4.0.2
|
||||
langgraph-prebuilt==1.0.10
|
||||
langgraph-sdk==0.3.13
|
||||
langsmith==0.7.34
|
||||
lark==1.3.1
|
||||
launchpadlib==1.10.16
|
||||
lazr.restfulclient==0.14.4
|
||||
lazr.uri==1.0.6
|
||||
lazy-loader==0.5
|
||||
libclang==18.1.1
|
||||
libcudf-cu12==26.2.1
|
||||
libcugraph-cu12==26.2.0
|
||||
libcuml-cu12==26.2.0
|
||||
libcuvs-cu12==26.2.0
|
||||
libkvikio-cu12==26.2.0
|
||||
libpysal==4.14.1
|
||||
libraft-cu12==26.2.0
|
||||
librmm-cu12==26.2.0
|
||||
librosa==0.11.0
|
||||
libucx-cu12==1.19.0
|
||||
libucxx-cu12==0.48.0
|
||||
lightgbm==4.6.0
|
||||
linkify-it-py==2.1.0
|
||||
llvmlite==0.43.0
|
||||
locket==1.0.0
|
||||
logical-unification==0.4.7
|
||||
lxml==6.1.0
|
||||
Mako==1.3.11
|
||||
mapclassify==2.10.0
|
||||
Markdown==3.10.2
|
||||
markdown-it-py==4.0.0
|
||||
MarkupSafe==3.0.3
|
||||
matplotlib==3.10.0
|
||||
matplotlib-inline==0.2.1
|
||||
matplotlib-venn==1.1.2
|
||||
mcp==1.27.0
|
||||
mdit-py-plugins==0.5.0
|
||||
mdurl==0.1.2
|
||||
mgwr==2.2.1
|
||||
miniKanren==1.0.5
|
||||
missingno==0.5.2
|
||||
mistune==3.2.0
|
||||
mizani==0.13.5
|
||||
mkl==2025.3.1
|
||||
ml_dtypes==0.5.4
|
||||
mlxtend==0.23.4
|
||||
mmh3==5.2.1
|
||||
momepy==0.11.0
|
||||
more-itertools==10.8.0
|
||||
moviepy==1.0.3
|
||||
mpmath==1.3.0
|
||||
msgpack==1.1.2
|
||||
multidict==6.7.1
|
||||
multipledispatch==1.0.0
|
||||
multiprocess==0.70.16
|
||||
multitasking==0.0.13
|
||||
murmurhash==1.0.15
|
||||
music21==9.9.1
|
||||
namex==0.1.0
|
||||
narwhals==2.20.0
|
||||
natsort==8.4.0
|
||||
nbclassic==1.3.3
|
||||
nbclient==0.10.4
|
||||
nbconvert==7.17.1
|
||||
nbformat==5.10.4
|
||||
ndindex==1.10.1
|
||||
nest-asyncio==1.6.0
|
||||
networkx==3.6.1
|
||||
nibabel==5.4.2
|
||||
nltk==3.9.1
|
||||
notebook==6.5.7
|
||||
notebook_shim==0.2.4
|
||||
numba==0.60.0
|
||||
numba-cuda==0.22.2
|
||||
numexpr==2.14.1
|
||||
numpy==2.0.2
|
||||
nvidia-cublas-cu12==12.8.4.1
|
||||
nvidia-cuda-cccl-cu12==12.9.27
|
||||
nvidia-cuda-cupti-cu12==12.8.90
|
||||
nvidia-cuda-nvcc-cu12==12.8.93
|
||||
nvidia-cuda-nvrtc-cu12==12.8.93
|
||||
nvidia-cuda-runtime-cu12==12.8.90
|
||||
nvidia-cudnn-cu12==9.10.2.21
|
||||
nvidia-cufft-cu12==11.3.3.83
|
||||
nvidia-cufile-cu12==1.13.1.3
|
||||
nvidia-curand-cu12==10.3.9.90
|
||||
nvidia-cusolver-cu12==11.7.3.90
|
||||
nvidia-cusparse-cu12==12.5.8.93
|
||||
nvidia-cusparselt-cu12==0.7.1
|
||||
nvidia-libnvcomp-cu12==5.1.0.21
|
||||
nvidia-ml-py==13.595.45
|
||||
nvidia-nccl-cu12==2.27.5
|
||||
nvidia-nvimgcodec-cu12==0.7.0.11
|
||||
nvidia-nvjitlink-cu12==12.8.93
|
||||
nvidia-nvshmem-cu12==3.4.5
|
||||
nvidia-nvtx-cu12==12.8.90
|
||||
nvtx==0.2.15
|
||||
nx-cugraph-cu12 @ https://pypi.nvidia.com/nx-cugraph-cu12/nx_cugraph_cu12-26.2.0-py3-none-any.whl
|
||||
oauth2client==4.1.3
|
||||
oauthlib==3.3.1
|
||||
omegaconf==2.3.0
|
||||
onemkl-license==2025.3.1
|
||||
openai==2.32.0
|
||||
opencv-contrib-python==4.13.0.92
|
||||
opencv-python==4.13.0.92
|
||||
opencv-python-headless==4.13.0.92
|
||||
openpyxl==3.1.5
|
||||
opentelemetry-api==1.38.0
|
||||
opentelemetry-exporter-gcp-logging==1.11.0a0
|
||||
opentelemetry-exporter-gcp-monitoring==1.11.0a0
|
||||
opentelemetry-exporter-gcp-trace==1.11.0
|
||||
opentelemetry-exporter-otlp-proto-common==1.38.0
|
||||
opentelemetry-exporter-otlp-proto-http==1.38.0
|
||||
opentelemetry-proto==1.38.0
|
||||
opentelemetry-resourcedetector-gcp==1.11.0a0
|
||||
opentelemetry-sdk==1.38.0
|
||||
opentelemetry-semantic-conventions==0.59b0
|
||||
opt_einsum==3.4.0
|
||||
optax==0.2.8
|
||||
optree==0.19.0
|
||||
orbax-checkpoint==0.11.36
|
||||
orjson==3.11.8
|
||||
ormsgpack==1.12.2
|
||||
osqp==1.1.1
|
||||
overrides==7.7.0
|
||||
packaging==26.1
|
||||
pandas==2.2.2
|
||||
pandas-datareader==0.10.0
|
||||
pandas-gbq==0.30.0
|
||||
pandas-stubs==2.2.2.240909
|
||||
pandocfilters==1.5.1
|
||||
panel==1.8.10
|
||||
param==2.3.3
|
||||
parso==0.8.6
|
||||
parsy==2.2
|
||||
partd==1.4.2
|
||||
patsy==1.0.2
|
||||
peewee==4.0.5
|
||||
peft==0.19.1
|
||||
pexpect==4.9.0
|
||||
pickleshare==0.7.5
|
||||
pillow==11.3.0
|
||||
pip==24.1.2
|
||||
platformdirs==4.9.6
|
||||
plotly==5.24.1
|
||||
plotnine==0.14.5
|
||||
pluggy==1.6.0
|
||||
plum-dispatch==2.8.0
|
||||
pointpats==2.5.5
|
||||
polars==1.35.2
|
||||
polars-runtime-32==1.35.2
|
||||
pooch==1.9.0
|
||||
portpicker==1.5.2
|
||||
preshed==3.0.13
|
||||
prettytable==3.17.0
|
||||
proglog==0.1.12
|
||||
progressbar2==4.5.0
|
||||
prometheus_client==0.25.0
|
||||
promise==2.3
|
||||
prompt_toolkit==3.0.52
|
||||
propcache==0.4.1
|
||||
prophet==1.3.0
|
||||
proto-plus==1.27.2
|
||||
protobuf==5.29.6
|
||||
psutil==5.9.5
|
||||
psycopg2==2.9.12
|
||||
psygnal==0.15.1
|
||||
ptyprocess==0.7.0
|
||||
PuLP==3.3.0
|
||||
py-cpuinfo==9.0.0
|
||||
py4j==0.10.9.9
|
||||
pyarrow==18.1.0
|
||||
pyasn1==0.6.3
|
||||
pyasn1_modules==0.4.2
|
||||
pycairo==1.29.0
|
||||
pycocotools==2.0.11
|
||||
pycparser==3.0
|
||||
pycryptodomex==3.23.0
|
||||
pydantic==2.12.3
|
||||
pydantic-settings==2.14.0
|
||||
pydantic_core==2.41.4
|
||||
pydata-google-auth==1.9.1
|
||||
pydot==4.0.1
|
||||
pydotplus==2.0.2
|
||||
PyDrive2==1.21.3
|
||||
pydub==0.25.1
|
||||
pyerfa==2.0.1.5
|
||||
pygame==2.6.1
|
||||
pygit2==1.19.2
|
||||
Pygments==2.20.0
|
||||
PyGObject==3.48.2
|
||||
pyiceberg==0.11.1
|
||||
PyJWT==2.12.1
|
||||
pylibcudf-cu12==26.2.1
|
||||
pylibcugraph-cu12==26.2.0
|
||||
pylibraft-cu12==26.2.0
|
||||
pymc==5.28.4
|
||||
pynndescent==0.6.0
|
||||
pyogrio==0.12.1
|
||||
pyomo==6.10.0
|
||||
PyOpenGL==3.1.10
|
||||
pyOpenSSL==24.2.1
|
||||
pyparsing==3.3.2
|
||||
pyperclip==1.11.0
|
||||
pyproj==3.7.2
|
||||
pyroaring==1.0.4
|
||||
pysal==25.7
|
||||
pyshp==3.0.3
|
||||
PySocks==1.7.1
|
||||
pyspark==4.0.2
|
||||
pytensor==2.38.2
|
||||
pytest==8.4.2
|
||||
python-apt==0.0.0
|
||||
python-box==7.4.1
|
||||
python-dateutil==2.9.0.post0
|
||||
python-dotenv==1.2.2
|
||||
python-fasthtml==0.12.50
|
||||
python-json-logger==4.1.0
|
||||
python-louvain==0.16
|
||||
python-multipart==0.0.26
|
||||
python-slugify==8.0.4
|
||||
python-snappy==0.7.3
|
||||
python-utils==3.9.1
|
||||
pytz==2025.2
|
||||
pyviz_comms==3.0.6
|
||||
PyWavelets==1.9.0
|
||||
PyYAML==6.0.3
|
||||
pyzmq==26.2.1
|
||||
quantecon==0.11.2
|
||||
raft-dask-cu12==26.2.0
|
||||
rapids-dask-dependency==26.2.0
|
||||
rapids-logger==0.2.3
|
||||
rasterio==1.5.0
|
||||
rasterstats==0.20.0
|
||||
ratelim==0.1.6
|
||||
referencing==0.37.0
|
||||
regex==2025.11.3
|
||||
requests==2.32.4
|
||||
requests-oauthlib==2.0.0
|
||||
requests-toolbelt==1.0.0
|
||||
requirements-parser==0.9.0
|
||||
rfc3339-validator==0.1.4
|
||||
rfc3986-validator==0.1.1
|
||||
rfc3987-syntax==1.1.0
|
||||
rich==13.9.4
|
||||
rmm-cu12==26.2.0
|
||||
roman-numerals==4.1.0
|
||||
roman-numerals-py==4.1.0
|
||||
rpds-py==0.30.0
|
||||
rpy2==3.5.17
|
||||
rsa==4.9.1
|
||||
rtree==1.4.1
|
||||
ruff==0.15.11
|
||||
safehttpx==0.1.7
|
||||
safetensors==0.7.0
|
||||
scikit-image==0.25.2
|
||||
scikit-learn==1.6.1
|
||||
scipy==1.16.3
|
||||
scooby==0.11.2
|
||||
scs==3.2.11
|
||||
seaborn==0.13.2
|
||||
SecretStorage==3.5.0
|
||||
segregation==2.5.4
|
||||
semantic-version==2.10.0
|
||||
Send2Trash==2.1.0
|
||||
sentence-transformers==5.4.1
|
||||
sentencepiece==0.2.1
|
||||
sentry-sdk==2.58.0
|
||||
setuptools==75.2.0
|
||||
shap==0.51.0
|
||||
shapely==2.1.2
|
||||
shellingham==1.5.4
|
||||
simple-parsing==0.1.8
|
||||
simplejson==4.1.0
|
||||
simsimd==6.5.16
|
||||
six==1.17.0
|
||||
sklearn-compat==0.1.5
|
||||
sklearn-pandas==2.2.0
|
||||
slicer==0.0.8
|
||||
smart_open==7.6.0
|
||||
smmap==5.0.3
|
||||
sniffio==1.3.1
|
||||
snowballstemmer==3.0.1
|
||||
sortedcontainers==2.4.0
|
||||
soundfile==0.13.1
|
||||
soupsieve==2.8.3
|
||||
soxr==1.0.0
|
||||
spacy==3.8.14
|
||||
spacy-legacy==3.0.12
|
||||
spacy-loggers==1.0.5
|
||||
spaghetti==1.7.6
|
||||
spanner-graph-notebook==1.1.10
|
||||
spglm==1.1.0
|
||||
Sphinx==8.2.3
|
||||
sphinxcontrib-applehelp==2.0.0
|
||||
sphinxcontrib-devhelp==2.0.0
|
||||
sphinxcontrib-htmlhelp==2.1.0
|
||||
sphinxcontrib-jsmath==1.0.1
|
||||
sphinxcontrib-qthelp==2.0.0
|
||||
sphinxcontrib-serializinghtml==2.0.0
|
||||
spint==1.0.7
|
||||
splot==1.1.7
|
||||
spopt==0.7.0
|
||||
spreg==1.9.0
|
||||
SQLAlchemy==2.0.49
|
||||
sqlalchemy-spanner==1.17.3
|
||||
sqlglot==25.20.2
|
||||
sqlparse==0.5.5
|
||||
srsly==2.5.3
|
||||
sse-starlette==3.3.4
|
||||
stanio==0.5.1
|
||||
starlette==0.52.1
|
||||
statsmodels==0.14.6
|
||||
strictyaml==1.7.3
|
||||
stringzilla==4.6.0
|
||||
stumpy==1.13.0
|
||||
sympy==1.14.0
|
||||
tables==3.10.2
|
||||
tabulate==0.9.0
|
||||
tbb==2022.3.1
|
||||
tblib==3.2.2
|
||||
tcmlib==1.4.1
|
||||
tenacity==9.1.4
|
||||
tensorboard==2.20.0
|
||||
tensorboard-data-server==0.7.2
|
||||
tensorflow==2.20.0
|
||||
tensorflow-datasets==4.9.9
|
||||
tensorflow-hub==0.16.1
|
||||
tensorflow-metadata==1.17.3
|
||||
tensorflow-probability==0.25.0
|
||||
tensorflow-text==2.20.1
|
||||
tensorstore==0.1.82
|
||||
termcolor==3.3.0
|
||||
terminado==0.18.1
|
||||
text-unidecode==1.3
|
||||
textblob==0.19.0
|
||||
tf-slim==1.1.0
|
||||
tf_keras==2.20.0
|
||||
thinc==8.3.13
|
||||
threadpoolctl==3.6.0
|
||||
tifffile==2026.4.11
|
||||
tiktoken==0.12.0
|
||||
timm==1.0.26
|
||||
tinycss2==1.4.0
|
||||
tobler==0.14.0
|
||||
tokenizers==0.22.2
|
||||
toml==0.10.2
|
||||
tomlkit==0.13.3
|
||||
toolz==0.12.1
|
||||
torch==2.10.0+cu128
|
||||
torchao==0.10.0
|
||||
torchaudio==2.10.0+cu128
|
||||
torchcodec==0.10.0+cu128
|
||||
torchdata==0.11.0
|
||||
torchsummary==1.5.1
|
||||
torchtune==0.6.1
|
||||
torchvision==0.25.0+cu128
|
||||
tornado==6.5.1
|
||||
tqdm==4.67.3
|
||||
traitlets==5.7.1
|
||||
traittypes==0.2.3
|
||||
transformers==5.0.0
|
||||
treelite==4.7.0
|
||||
treescope==0.1.10
|
||||
triton==3.6.0
|
||||
tsfresh==0.21.1
|
||||
tweepy==4.16.0
|
||||
typeguard==4.5.1
|
||||
typer==0.24.2
|
||||
typer-slim==0.24.0
|
||||
types-pytz==2026.1.1.20260408
|
||||
types-setuptools==82.0.0.20260408
|
||||
typing-inspection==0.4.2
|
||||
typing_extensions==4.15.0
|
||||
tzdata==2026.1
|
||||
tzlocal==5.3.1
|
||||
uc-micro-py==2.0.0
|
||||
ucxx-cu12==0.48.0
|
||||
umap-learn==0.5.12
|
||||
umf==1.0.3
|
||||
uri-template==1.3.0
|
||||
uritemplate==4.2.0
|
||||
urllib3==2.5.0
|
||||
uuid_utils==0.14.1
|
||||
uvicorn==0.46.0
|
||||
uvloop==0.22.1
|
||||
vega-datasets==0.9.0
|
||||
wadllib==1.3.6
|
||||
wandb==0.26.1
|
||||
wasabi==1.1.3
|
||||
watchdog==6.0.0
|
||||
watchfiles==1.1.1
|
||||
wcwidth==0.6.0
|
||||
weasel==1.0.0
|
||||
webcolors==25.10.0
|
||||
webencodings==0.5.1
|
||||
websocket-client==1.9.0
|
||||
websockets==15.0.1
|
||||
Werkzeug==3.1.8
|
||||
wheel==0.47.0
|
||||
widgetsnbextension==3.6.10
|
||||
wordcloud==1.9.6
|
||||
wrapt==2.1.2
|
||||
xarray==2025.12.0
|
||||
xarray-einstats==0.10.0
|
||||
xgboost==3.2.0
|
||||
xlrd==2.0.2
|
||||
xxhash==3.6.0
|
||||
xyzservices==2026.3.0
|
||||
yarl==1.23.0
|
||||
ydf==0.15.0
|
||||
ydf_tf==2.20.0
|
||||
yellowbrick==1.5
|
||||
yfinance==0.2.66
|
||||
zict==3.0.0
|
||||
zipp==3.23.1
|
||||
zstandard==0.25.0
|
||||
@@ -0,0 +1,36 @@
|
||||
{
|
||||
"_comment": "Maps Colab GPU runtime pinned wheels to CPU equivalents for ubuntu-latest CI smoke jobs. The Colab GPU image ships +cu128 builds that won't install on a CPU-only runner; this map either rewrites the spec to a CPU wheel from https://download.pytorch.org/whl/cpu or falls back to module-spoof for packages with no CPU build.",
|
||||
"rewrite": {
|
||||
"torch": {
|
||||
"from_local_version": "+cu128",
|
||||
"to_index_url": "https://download.pytorch.org/whl/cpu"
|
||||
},
|
||||
"torchvision": {
|
||||
"from_local_version": "+cu128",
|
||||
"to_index_url": "https://download.pytorch.org/whl/cpu"
|
||||
},
|
||||
"torchaudio": {
|
||||
"from_local_version": "+cu128",
|
||||
"to_index_url": "https://download.pytorch.org/whl/cpu"
|
||||
}
|
||||
},
|
||||
"module_spoof": {
|
||||
"torchcodec": "no CPU wheel published; smoke job sys.modules-stubs torchcodec before importing unsloth"
|
||||
},
|
||||
"skip": [
|
||||
"nvidia-cublas-cu12",
|
||||
"nvidia-cuda-cupti-cu12",
|
||||
"nvidia-cuda-nvrtc-cu12",
|
||||
"nvidia-cuda-runtime-cu12",
|
||||
"nvidia-cudnn-cu12",
|
||||
"nvidia-cufft-cu12",
|
||||
"nvidia-curand-cu12",
|
||||
"nvidia-cusolver-cu12",
|
||||
"nvidia-cusparse-cu12",
|
||||
"nvidia-cusparselt-cu12",
|
||||
"nvidia-nccl-cu12",
|
||||
"nvidia-nvjitlink-cu12",
|
||||
"nvidia-nvtx-cu12",
|
||||
"triton"
|
||||
]
|
||||
}
|
||||
Executable
+656
@@ -0,0 +1,656 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Ensure keyword arguments use spaces around '=', prune redundant pass statements,
|
||||
drop the blank line after a short indented import block, merge adjacent same-line
|
||||
string literals, normalize def-signature magic commas (pre-ruff) so a def with
|
||||
>= 3 params and a default goes one-per-line while everything else stays
|
||||
collapsible, and collapse a short multi-line assert onto one line (pre-ruff) by
|
||||
stripping the magic trailing comma that holds it open."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
import argparse
|
||||
import io
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import tokenize
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def _atomic_write_text(path: Path, data: str, encoding: str) -> None:
|
||||
"""Write ``data`` to ``path`` atomically via same-dir tmp + fsync + os.replace,
|
||||
so a crash mid-write leaves either the old or full new content, never a truncation."""
|
||||
dirpath = str(path.parent) or "."
|
||||
fd, tmp_path = tempfile.mkstemp(prefix=".kwargs_fix.", dir=dirpath)
|
||||
try:
|
||||
with os.fdopen(fd, "w", encoding=encoding) as handle:
|
||||
handle.write(data)
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
os.replace(tmp_path, path)
|
||||
except Exception:
|
||||
try:
|
||||
os.unlink(tmp_path)
|
||||
except OSError:
|
||||
pass
|
||||
raise
|
||||
|
||||
|
||||
def enforce_spacing(text: str) -> tuple[str, bool]:
|
||||
"""Return updated text with keyword '=' padded by spaces, plus change flag."""
|
||||
lines = text.splitlines(keepends=True)
|
||||
if not lines:
|
||||
return text, False
|
||||
|
||||
offsets: dict[int, int] = defaultdict(int)
|
||||
changed = False
|
||||
|
||||
reader = io.StringIO(text).readline
|
||||
for token in tokenize.generate_tokens(reader):
|
||||
if token.type != tokenize.OP or token.string != "=":
|
||||
continue
|
||||
|
||||
line_index = token.start[0] - 1
|
||||
col = token.start[1] + offsets[line_index]
|
||||
|
||||
if line_index < 0 or line_index >= len(lines):
|
||||
continue
|
||||
|
||||
line = lines[line_index]
|
||||
if col >= len(line) or line[col] != "=":
|
||||
continue
|
||||
|
||||
line_changed = False
|
||||
|
||||
# Insert a space before '=' when missing and not preceded by whitespace.
|
||||
if col > 0 and line[col - 1] not in {" ", "\t"}:
|
||||
line = f"{line[:col]} {line[col:]}"
|
||||
offsets[line_index] += 1
|
||||
col += 1
|
||||
line_changed = True
|
||||
changed = True
|
||||
|
||||
# Insert a space after '=' when missing and not followed by whitespace or newline.
|
||||
next_index = col + 1
|
||||
if next_index < len(line) and line[next_index] not in {" ", "\t", "\n", "\r"}:
|
||||
line = f"{line[:next_index]} {line[next_index:]}"
|
||||
offsets[line_index] += 1
|
||||
line_changed = True
|
||||
changed = True
|
||||
|
||||
if line_changed:
|
||||
lines[line_index] = line
|
||||
|
||||
if not changed:
|
||||
return text, False
|
||||
|
||||
return "".join(lines), True
|
||||
|
||||
|
||||
def remove_redundant_passes(text: str) -> tuple[str, bool]:
|
||||
"""Drop pass statements that share a block with other executable code."""
|
||||
|
||||
try:
|
||||
tree = ast.parse(text)
|
||||
except SyntaxError:
|
||||
return text, False
|
||||
|
||||
redundant: list[ast.Pass] = []
|
||||
|
||||
def visit(node: ast.AST) -> None:
|
||||
for attr in ("body", "orelse", "finalbody"):
|
||||
value = getattr(node, attr, None)
|
||||
if not isinstance(value, list) or len(value) <= 1:
|
||||
continue
|
||||
for stmt in value:
|
||||
if isinstance(stmt, ast.Pass):
|
||||
redundant.append(stmt)
|
||||
for stmt in value:
|
||||
if isinstance(stmt, ast.AST):
|
||||
visit(stmt)
|
||||
handlers = getattr(node, "handlers", None)
|
||||
if handlers:
|
||||
for handler in handlers:
|
||||
visit(handler)
|
||||
|
||||
visit(tree)
|
||||
|
||||
if not redundant:
|
||||
return text, False
|
||||
|
||||
lines = text.splitlines(keepends=True)
|
||||
changed = False
|
||||
|
||||
for node in sorted(redundant, key=lambda item: (item.lineno, item.col_offset), reverse=True):
|
||||
start = node.lineno - 1
|
||||
end = (node.end_lineno or node.lineno) - 1
|
||||
if start >= len(lines):
|
||||
continue
|
||||
changed = True
|
||||
if start == end:
|
||||
line = lines[start]
|
||||
col_start = node.col_offset
|
||||
col_end = node.end_col_offset or (col_start + 4)
|
||||
segment = line[:col_start] + line[col_end:]
|
||||
lines[start] = segment if segment.strip() else ""
|
||||
continue
|
||||
|
||||
# Fall-back for unexpected multi-line 'pass'.
|
||||
prefix = lines[start][: node.col_offset]
|
||||
lines[start] = prefix if prefix.strip() else ""
|
||||
for idx in range(start + 1, end):
|
||||
lines[idx] = ""
|
||||
suffix = lines[end][(node.end_col_offset or 0) :]
|
||||
lines[end] = suffix
|
||||
|
||||
# Normalise to ensure lines end with newlines except at EOF.
|
||||
result_lines: list[str] = []
|
||||
for index, line in enumerate(lines):
|
||||
if not line:
|
||||
continue
|
||||
if index < len(lines) - 1 and not line.endswith("\n"):
|
||||
result_lines.append(f"{line}\n")
|
||||
else:
|
||||
result_lines.append(line)
|
||||
|
||||
return "".join(result_lines), changed
|
||||
|
||||
|
||||
def remove_blank_after_short_import(text: str) -> tuple[str, bool]:
|
||||
"""Drop blank line(s) after an import block in a small nested suite.
|
||||
|
||||
In an indented suite of <= 3 statements (never module level), when consecutive
|
||||
imports are followed across blank lines (nothing else) by another statement,
|
||||
remove those blanks. A comment in the gap blocks the rule. Removing blank lines
|
||||
never changes the AST.
|
||||
"""
|
||||
try:
|
||||
tree = ast.parse(text)
|
||||
except SyntaxError:
|
||||
return text, False
|
||||
|
||||
lines = text.splitlines(keepends=True)
|
||||
import_types = (ast.Import, ast.ImportFrom)
|
||||
drop: set[int] = set() # 1-based physical line numbers to delete
|
||||
|
||||
def suites_of(node: ast.AST) -> list[list[ast.stmt]]:
|
||||
if isinstance(node, ast.Module):
|
||||
return [] # module-level import spacing is left alone
|
||||
out: list[list[ast.stmt]] = []
|
||||
for attr in ("body", "orelse", "finalbody"):
|
||||
val = getattr(node, attr, None)
|
||||
if isinstance(val, list) and val and all(isinstance(s, ast.stmt) for s in val):
|
||||
out.append(val)
|
||||
return out
|
||||
|
||||
for node in ast.walk(tree):
|
||||
for suite in suites_of(node):
|
||||
if len(suite) > 3: # only small blocks
|
||||
continue
|
||||
i = 0
|
||||
while i < len(suite):
|
||||
if not isinstance(suite[i], import_types):
|
||||
i += 1
|
||||
continue
|
||||
j = i
|
||||
while j + 1 < len(suite) and isinstance(suite[j + 1], import_types):
|
||||
j += 1
|
||||
if j + 1 < len(suite): # an import block followed by another statement
|
||||
last_imp, nxt = suite[j], suite[j + 1]
|
||||
gap = range((last_imp.end_lineno or last_imp.lineno) + 1, nxt.lineno)
|
||||
nums = [n for n in gap if 1 <= n <= len(lines)]
|
||||
if nums and all(lines[n - 1].strip() == "" for n in nums):
|
||||
drop.update(nums)
|
||||
i = j + 1
|
||||
|
||||
if not drop:
|
||||
return text, False
|
||||
kept = [ln for idx, ln in enumerate(lines, start=1) if idx not in drop]
|
||||
return "".join(kept), True
|
||||
|
||||
|
||||
_STRING_TRIVIA = (tokenize.NL, tokenize.NEWLINE, tokenize.COMMENT, tokenize.INDENT, tokenize.DEDENT)
|
||||
|
||||
|
||||
_DEF_MIN_PARAMS_FOR_MULTILINE = 3 # signatures with < this many params stay one line
|
||||
|
||||
|
||||
def _def_specs_by_line(tree: ast.AST) -> dict[int, tuple[int, bool]]:
|
||||
"""Map each def keyword line to (param count, has-any-default).
|
||||
|
||||
``*`` / ``/`` markers aren't counted. A default exists if any positional default
|
||||
is present or any keyword-only default is not ``None`` (``None`` in ``kw_defaults``
|
||||
means a required keyword-only arg).
|
||||
"""
|
||||
out: dict[int, tuple[int, bool]] = {}
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
|
||||
a = node.args
|
||||
count = (
|
||||
len(a.posonlyargs)
|
||||
+ len(a.args)
|
||||
+ len(a.kwonlyargs)
|
||||
+ (1 if a.vararg else 0)
|
||||
+ (1 if a.kwarg else 0)
|
||||
)
|
||||
has_default = bool(a.defaults) or any(d is not None for d in a.kw_defaults)
|
||||
out[node.lineno] = (count, has_default)
|
||||
return out
|
||||
|
||||
|
||||
def normalize_def_trailing_comma(text: str) -> tuple[str, bool]:
|
||||
"""Force a def signature one-per-line iff >= 3 params AND a default; else collapsible.
|
||||
|
||||
A qualifying signature gets a magic trailing comma added (ruff wraps it
|
||||
one-per-line); every other signature has its trailing comma stripped so ruff
|
||||
collapses it when it fits. Def parameter lists only, never call sites or
|
||||
collection literals. Run BEFORE ruff format. Never changes the AST (re-checked).
|
||||
"""
|
||||
try:
|
||||
tree = ast.parse(text)
|
||||
toks = list(tokenize.generate_tokens(io.StringIO(text).readline))
|
||||
except (tokenize.TokenError, IndentationError, SyntaxError):
|
||||
return text, False
|
||||
|
||||
specs = _def_specs_by_line(tree)
|
||||
n = len(toks)
|
||||
edits: list[tuple[int, int, str]] = [] # (row, col, "del" | "ins")
|
||||
i = 0
|
||||
while i < n:
|
||||
t = toks[i]
|
||||
if t.type == tokenize.NAME and t.string == "def" and t.start[0] in specs:
|
||||
cnt, has_default = specs[t.start[0]]
|
||||
force_multiline = cnt >= _DEF_MIN_PARAMS_FOR_MULTILINE and has_default
|
||||
j = i + 1
|
||||
while j < n and not (toks[j].type == tokenize.OP and toks[j].string == "("):
|
||||
if toks[j].type == tokenize.NEWLINE:
|
||||
break
|
||||
j += 1
|
||||
if j < n and toks[j].type == tokenize.OP and toks[j].string == "(":
|
||||
depth = 0
|
||||
k = j
|
||||
while k < n:
|
||||
tk = toks[k]
|
||||
if tk.type == tokenize.OP and tk.string == "(":
|
||||
depth += 1
|
||||
elif tk.type == tokenize.OP and tk.string == ")":
|
||||
depth -= 1
|
||||
if depth == 0:
|
||||
m = k - 1
|
||||
while m > j and toks[m].type in _STRING_TRIVIA:
|
||||
m -= 1
|
||||
last = toks[m]
|
||||
has_comma = last.type == tokenize.OP and last.string == ","
|
||||
empty = m == j # nothing between ( and )
|
||||
if force_multiline and not has_comma and not empty:
|
||||
edits.append((last.end[0], last.end[1], "ins"))
|
||||
elif not force_multiline and has_comma:
|
||||
edits.append((last.start[0], last.start[1], "del"))
|
||||
break
|
||||
k += 1
|
||||
i = k + 1
|
||||
continue
|
||||
i += 1
|
||||
|
||||
if not edits:
|
||||
return text, False
|
||||
|
||||
lines = text.splitlines(keepends=True)
|
||||
for row, col, kind in sorted(edits, reverse=True):
|
||||
ln = lines[row - 1]
|
||||
if kind == "del":
|
||||
if col < len(ln) and ln[col] == ",":
|
||||
lines[row - 1] = ln[:col] + ln[col + 1 :]
|
||||
else: # ins
|
||||
lines[row - 1] = ln[:col] + "," + ln[col:]
|
||||
out = "".join(lines)
|
||||
try:
|
||||
if ast.dump(ast.parse(out)) != ast.dump(ast.parse(text)):
|
||||
return text, False
|
||||
except SyntaxError:
|
||||
return text, False
|
||||
return out, True
|
||||
|
||||
|
||||
def _split_string_token(s: str) -> tuple[str, str, str] | None:
|
||||
"""Split a string literal source into (prefix, quote, body).
|
||||
|
||||
``prefix`` is the letters before the opening quote, ``quote`` the delimiter,
|
||||
``body`` everything between. ``None`` if not a recognizable string literal.
|
||||
"""
|
||||
i = 0
|
||||
while i < len(s) and s[i] not in ("'", '"'):
|
||||
i += 1
|
||||
if i >= len(s):
|
||||
return None
|
||||
prefix, rest = s[:i], s[i:]
|
||||
for q in ('"""', "'''", '"', "'"):
|
||||
if rest.startswith(q) and rest.endswith(q) and len(rest) >= 2 * len(q):
|
||||
return prefix, q, rest[len(q) : len(rest) - len(q)]
|
||||
return None
|
||||
|
||||
|
||||
# A "piece" is one string literal in source: a plain STRING token, or a whole
|
||||
# f-string spanning FSTRING_START..FSTRING_END. (kind, (row, col0), (row, col1), raw)
|
||||
def _string_pieces(
|
||||
toks: list[tokenize.TokenInfo], lines: list[str]
|
||||
) -> list[tuple[str, tuple[int, int], tuple[int, int], str | None]]:
|
||||
pieces: list[tuple[str, tuple[int, int], tuple[int, int], str | None]] = []
|
||||
n = len(toks)
|
||||
|
||||
def raw_of(start: tuple[int, int], end: tuple[int, int]) -> str | None:
|
||||
if start[0] != end[0]: # only single-physical-line pieces are mergeable
|
||||
return None
|
||||
return lines[start[0] - 1][start[1] : end[1]]
|
||||
|
||||
i = 0
|
||||
while i < n:
|
||||
t = toks[i]
|
||||
if t.type == tokenize.STRING:
|
||||
pieces.append(("str", t.start, t.end, raw_of(t.start, t.end)))
|
||||
i += 1
|
||||
elif t.type == tokenize.FSTRING_START:
|
||||
depth = 0
|
||||
j = i
|
||||
while j < n: # walk to the matching FSTRING_END (f-strings can nest)
|
||||
if toks[j].type == tokenize.FSTRING_START:
|
||||
depth += 1
|
||||
elif toks[j].type == tokenize.FSTRING_END:
|
||||
depth -= 1
|
||||
if depth == 0:
|
||||
break
|
||||
j += 1
|
||||
end = toks[j].end
|
||||
pieces.append(("f", t.start, end, raw_of(t.start, end)))
|
||||
i = j + 1
|
||||
else:
|
||||
pieces.append(("other", t.start, t.end, None))
|
||||
i += 1
|
||||
return pieces
|
||||
|
||||
|
||||
def _merge_string_run(pieces: list[tuple[str, str]]) -> str | None:
|
||||
"""Merge a run of adjacent string pieces into one literal's source text.
|
||||
|
||||
``pieces`` is ``(kind, raw_source)`` with kind ``"str"`` or ``"f"``. Bytes are
|
||||
left side-by-side (``None``); a run with no f-string merges plain/raw/unicode
|
||||
sharing one prefix+quote by body concatenation; a run mixing an f-string with a
|
||||
plain string (no bytes, no raw) folds into one f-string with plain braces escaped.
|
||||
Runs of only f-strings are left alone. Caller re-checks the AST and drops a
|
||||
differing change, so subtle cases are caught.
|
||||
"""
|
||||
parsed = []
|
||||
for kind, raw in pieces:
|
||||
pqb = _split_string_token(raw)
|
||||
if pqb is None:
|
||||
return None
|
||||
prefix, quote, body = pqb
|
||||
if "b" in prefix.lower():
|
||||
return None # bytes: leave side-by-side
|
||||
parsed.append((kind, prefix, quote, body))
|
||||
if len({p[2] for p in parsed}) != 1:
|
||||
return None # mixed quote style: not a safe textual merge
|
||||
quote = parsed[0][2]
|
||||
if not any(p[0] == "f" for p in parsed):
|
||||
# No f-string: merge plain/raw/unicode sharing one prefix by concatenation.
|
||||
if len({p[1].lower() for p in parsed}) != 1:
|
||||
return None
|
||||
return f"{parsed[0][1]}{quote}{''.join(p[3] for p in parsed)}{quote}"
|
||||
# f-string fold only when a plain string is glued onto an f-string; a run of
|
||||
# only f-strings is left side-by-side (folding long ones would force ruff to
|
||||
# re-wrap the surrounding statement).
|
||||
if all(p[0] == "f" for p in parsed):
|
||||
return None
|
||||
# raw mixed with f is too subtle (backslash + brace escaping) -> skip.
|
||||
if any("r" in p[1].lower() for p in parsed):
|
||||
return None
|
||||
body = "".join(
|
||||
b if kind == "f" else b.replace("{", "{{").replace("}", "}}")
|
||||
for kind, _pfx, _q, b in parsed
|
||||
)
|
||||
return f"f{quote}{body}{quote}"
|
||||
|
||||
|
||||
_LINE_LENGTH = 100 # ruff line-length; an f-fold must not push a statement past it
|
||||
|
||||
|
||||
def _enclosing_stmt(tree: ast.AST, row: int) -> ast.stmt | None:
|
||||
"""The innermost statement whose physical-line span contains ``row``."""
|
||||
best: tuple[ast.stmt, int] | None = None
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.stmt):
|
||||
lo = node.lineno
|
||||
hi = node.end_lineno or lo
|
||||
if lo <= row <= hi and (best is None or hi - lo < best[1]):
|
||||
best = (node, hi - lo)
|
||||
return best[0] if best else None
|
||||
|
||||
|
||||
def _fold_collapses(
|
||||
tree: ast.AST, lines: list[str], row: int, c0: int, c1: int, merged: str
|
||||
) -> bool:
|
||||
"""Whether an f-string fold at ``row[c0:c1]`` -> ``merged`` is safe to apply.
|
||||
|
||||
Only ``assert`` wraps awkwardly when a message folds (ruff parenthesizes the
|
||||
condition once it no longer fits one line); every other construct wraps
|
||||
acceptably so is always allowed. An ``assert`` fold is allowed only if already
|
||||
one line, or its estimated folded one-line length fits the line length.
|
||||
"""
|
||||
stmt = _enclosing_stmt(tree, row)
|
||||
if not isinstance(stmt, ast.Assert):
|
||||
return True
|
||||
lo, hi = stmt.lineno, stmt.end_lineno or stmt.lineno
|
||||
if lo == hi:
|
||||
return True
|
||||
seg = []
|
||||
for k in range(lo, hi + 1):
|
||||
ln = lines[k - 1].rstrip("\n")
|
||||
if k == row:
|
||||
ln = ln[:c0] + merged + ln[c1:]
|
||||
seg.append(ln)
|
||||
indent = len(seg[0]) - len(seg[0].lstrip())
|
||||
# Conservative over-estimate: join continuation lines with a single space
|
||||
# (ruff joins bracketed wraps with none), so borderline cases skip the fold.
|
||||
joined = " ".join(s.strip() for s in seg)
|
||||
return indent + len(joined) <= _LINE_LENGTH
|
||||
|
||||
|
||||
def merge_adjacent_string_literals(text: str) -> tuple[str, bool]:
|
||||
"""Merge adjacent string literals on ONE physical line into a single literal.
|
||||
|
||||
Plain/raw/unicode runs merge by concatenation; an f-string + plain string folds
|
||||
into one f-string (plain braces escaped) only while the statement still fits one
|
||||
line. Runs of only f-strings, and bytes, are left side-by-side. The file AST is
|
||||
re-checked and a differing change dropped, so meaning never changes.
|
||||
"""
|
||||
try:
|
||||
toks = list(tokenize.generate_tokens(io.StringIO(text).readline))
|
||||
tree = ast.parse(text)
|
||||
except (tokenize.TokenError, IndentationError, SyntaxError):
|
||||
return text, False
|
||||
|
||||
lines = text.splitlines(keepends=True)
|
||||
pieces = _string_pieces(toks, lines)
|
||||
|
||||
# Group consecutive mergeable pieces (str/f, single line, same physical line).
|
||||
runs: list[list[tuple[str, tuple[int, int], tuple[int, int], str]]] = []
|
||||
cur: list[tuple[str, tuple[int, int], tuple[int, int], str]] = []
|
||||
for kind, start, end, raw in pieces:
|
||||
if kind in ("str", "f") and raw is not None:
|
||||
if cur and cur[-1][2][0] != start[0]:
|
||||
if len(cur) >= 2:
|
||||
runs.append(cur)
|
||||
cur = []
|
||||
cur.append((kind, start, end, raw))
|
||||
else:
|
||||
if len(cur) >= 2:
|
||||
runs.append(cur)
|
||||
cur = []
|
||||
if len(cur) >= 2:
|
||||
runs.append(cur)
|
||||
if not runs:
|
||||
return text, False
|
||||
|
||||
edits = []
|
||||
for run in runs:
|
||||
merged = _merge_string_run([(kind, raw) for kind, _s, _e, raw in run])
|
||||
if merged is None:
|
||||
continue
|
||||
row, c0, c1 = run[0][1][0], run[0][1][1], run[-1][2][1]
|
||||
# An f-string fold must not push its statement onto extra lines; a plain
|
||||
# concatenation always collapses cleanly so it skips this check.
|
||||
if any(kind == "f" for kind, _s, _e, _r in run) and not _fold_collapses(
|
||||
tree, lines, row, c0, c1, merged
|
||||
):
|
||||
continue
|
||||
edits.append((row, c0, c1, merged))
|
||||
if not edits:
|
||||
return text, False
|
||||
|
||||
for row, c0, c1, repl in sorted(edits, key=lambda e: (e[0], e[1]), reverse=True):
|
||||
ln = lines[row - 1]
|
||||
lines[row - 1] = ln[:c0] + repl + ln[c1:]
|
||||
out = "".join(lines)
|
||||
try:
|
||||
if ast.dump(ast.parse(text)) != ast.dump(ast.parse(out)):
|
||||
return text, False
|
||||
except SyntaxError:
|
||||
return text, False
|
||||
return out, True
|
||||
|
||||
|
||||
def collapse_short_asserts(text: str) -> tuple[str, bool]:
|
||||
"""Collapse a multi-line ``assert`` onto one line when it would fit.
|
||||
|
||||
When the statement's estimated one-line length fits, strip the magic trailing
|
||||
commas (comma before a closer) holding it open so ruff rejoins it. Run BEFORE
|
||||
ruff format. Skips asserts with a comment (would oscillate). Stripping is
|
||||
non-semantic except for a one-element tuple; AST is re-checked and changing
|
||||
asserts left alone.
|
||||
"""
|
||||
try:
|
||||
tree = ast.parse(text)
|
||||
toks = list(tokenize.generate_tokens(io.StringIO(text).readline))
|
||||
except (tokenize.TokenError, IndentationError, SyntaxError):
|
||||
return text, False
|
||||
|
||||
lines = text.splitlines(keepends=True)
|
||||
multiline = [
|
||||
(n.lineno, n.end_lineno)
|
||||
for n in ast.walk(tree)
|
||||
if isinstance(n, ast.Assert) and (n.end_lineno or n.lineno) > n.lineno
|
||||
]
|
||||
if not multiline:
|
||||
return text, False
|
||||
|
||||
comment_rows = {t.start[0] for t in toks if t.type == tokenize.COMMENT}
|
||||
|
||||
targets = [] # (lo, hi) spans whose one-line form fits and have no comment
|
||||
for lo, hi in multiline:
|
||||
if any(lo <= r <= hi for r in comment_rows):
|
||||
continue # a comment would keep ruff multi-line -> never collapses
|
||||
seg = [lines[k].rstrip("\n") for k in range(lo - 1, hi)]
|
||||
indent = len(seg[0]) - len(seg[0].lstrip())
|
||||
# Over-estimate (join with a space; keep the comma) so a "fits" verdict
|
||||
# is always at least as long as ruff's real one-line output -> no fight.
|
||||
if indent + len(" ".join(s.strip() for s in seg)) <= _LINE_LENGTH:
|
||||
targets.append((lo, hi))
|
||||
if not targets:
|
||||
return text, False
|
||||
|
||||
# Trailing commas (a ',' whose next significant token is a closer), grouped
|
||||
# by the target assert they belong to.
|
||||
sig = [t for t in toks if t.type not in _STRING_TRIVIA]
|
||||
by_target: dict[tuple[int, int], list[tuple[int, int]]] = defaultdict(list)
|
||||
for i, t in enumerate(sig):
|
||||
if t.type == tokenize.OP and t.string == ",":
|
||||
nxt = sig[i + 1] if i + 1 < len(sig) else None
|
||||
if nxt and nxt.type == tokenize.OP and nxt.string in (")", "]", "}"):
|
||||
for lo, hi in targets:
|
||||
if lo <= t.start[0] <= hi:
|
||||
by_target[(lo, hi)].append(t.start)
|
||||
break
|
||||
if not by_target:
|
||||
return text, False
|
||||
|
||||
base_dump = ast.dump(tree)
|
||||
working = lines[:]
|
||||
changed = False
|
||||
for positions in by_target.values(): # apply per assert; skip any that break AST
|
||||
trial = working[:]
|
||||
for row, col in sorted(positions, reverse=True):
|
||||
ln = trial[row - 1]
|
||||
if col < len(ln) and ln[col] == ",":
|
||||
trial[row - 1] = ln[:col] + ln[col + 1 :]
|
||||
try:
|
||||
if ast.dump(ast.parse("".join(trial))) == base_dump:
|
||||
working, changed = trial, True
|
||||
except SyntaxError:
|
||||
pass
|
||||
return ("".join(working), True) if changed else (text, False)
|
||||
|
||||
|
||||
def process_file(path: Path, pre: bool = False) -> bool:
|
||||
try:
|
||||
with tokenize.open(path) as handle:
|
||||
original = handle.read()
|
||||
encoding = handle.encoding
|
||||
except (OSError, SyntaxError) as exc: # SyntaxError from tokenize on invalid python
|
||||
print(f"Failed to read {path}: {exc}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
if pre:
|
||||
# Pre-ruff: normalize def-signature magic commas (>=3 params + a default
|
||||
# add so ruff forces one-per-line; everything else strips so ruff
|
||||
# collapses), and strip the magic trailing comma from a short multi-line
|
||||
# assert so ruff joins it onto one line. Everything else runs post-ruff.
|
||||
updated, normalized = normalize_def_trailing_comma(original)
|
||||
updated, collapsed = collapse_short_asserts(updated)
|
||||
if normalized or collapsed:
|
||||
_atomic_write_text(path, updated, encoding)
|
||||
return True
|
||||
return False
|
||||
|
||||
updated, changed = enforce_spacing(original)
|
||||
updated, blanked = remove_blank_after_short_import(updated)
|
||||
updated, merged = merge_adjacent_string_literals(updated)
|
||||
updated, removed = remove_redundant_passes(updated)
|
||||
if changed or blanked or merged or removed:
|
||||
_atomic_write_text(path, updated, encoding)
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("files", nargs="+", help="Python files to fix")
|
||||
parser.add_argument(
|
||||
"--pre",
|
||||
action="store_true",
|
||||
help="pre-ruff pass: normalize def-signature commas + collapse short multi-line asserts",
|
||||
)
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
touched: list[Path] = []
|
||||
self_path = Path(__file__).resolve()
|
||||
|
||||
for entry in args.files:
|
||||
path = Path(entry)
|
||||
# Skip modifying this script to avoid self-edit loops.
|
||||
if path.resolve() == self_path:
|
||||
continue
|
||||
if not path.exists() or path.is_dir():
|
||||
continue
|
||||
if process_file(path, pre=args.pre):
|
||||
touched.append(path)
|
||||
|
||||
if touched:
|
||||
for path in touched:
|
||||
print(f"Adjusted kwarg spacing in {path}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
Executable
+184
@@ -0,0 +1,184 @@
|
||||
#!/bin/bash
|
||||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
set -euo pipefail
|
||||
|
||||
# ============================================================
|
||||
# Gemma 4 MLX — One-command setup + inference
|
||||
#
|
||||
# Supply-chain hardening: the uv installer payload is pinned by
|
||||
# SHA-256. Rotate by running:
|
||||
# curl -sSLf https://astral.sh/uv/install.sh | shasum -a 256
|
||||
# and updating _UV_INSTALLER_SHA256 below.
|
||||
# ============================================================
|
||||
#
|
||||
# Usage:
|
||||
# bash install_gemma4_mlx.sh [--venv-dir DIR]
|
||||
#
|
||||
# This script:
|
||||
# 1. Creates a Python virtual environment
|
||||
# 2. Installs uv, mlx-vlm, transformers
|
||||
# ============================================================
|
||||
|
||||
# ── Output style (inspired by unsloth/install.sh) ─────────────
|
||||
RULE=""
|
||||
_rule_i=0
|
||||
while [ "$_rule_i" -lt 52 ]; do
|
||||
RULE="${RULE}─"
|
||||
_rule_i=$((_rule_i + 1))
|
||||
done
|
||||
|
||||
if [ -n "${NO_COLOR:-}" ]; then
|
||||
C_TITLE= C_DIM= C_OK= C_WARN= C_ERR= C_RST=
|
||||
elif [ -t 1 ] || [ -n "${FORCE_COLOR:-}" ]; then
|
||||
_ESC="$(printf '\033')"
|
||||
C_TITLE="${_ESC}[38;5;117m"
|
||||
C_DIM="${_ESC}[38;5;245m"
|
||||
C_OK="${_ESC}[38;5;108m"
|
||||
C_WARN="${_ESC}[38;5;136m"
|
||||
C_ERR="${_ESC}[91m"
|
||||
C_RST="${_ESC}[0m"
|
||||
else
|
||||
C_TITLE= C_DIM= C_OK= C_WARN= C_ERR= C_RST=
|
||||
fi
|
||||
|
||||
step() { printf " ${C_DIM}%-18.18s${C_RST}${3:-$C_OK}%s${C_RST}\n" "$1" "$2"; }
|
||||
substep() { printf " ${C_DIM}%-18s${2:-$C_DIM}%s${C_RST}\n" "" "$1"; }
|
||||
fail() { step "error" "$1" "$C_ERR"; exit 1; }
|
||||
|
||||
# ── Parse flags ───────────────────────────────────────────────
|
||||
VENV_DIR=""
|
||||
_next_is_venv=false
|
||||
|
||||
for arg in "$@"; do
|
||||
if [ "$_next_is_venv" = true ]; then
|
||||
VENV_DIR="$arg"
|
||||
_next_is_venv=false
|
||||
continue
|
||||
fi
|
||||
case "$arg" in
|
||||
--venv-dir) _next_is_venv=true ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Default venv location
|
||||
if [ -z "$VENV_DIR" ]; then
|
||||
VENV_DIR="$HOME/.unsloth/unsloth_gemma4_mlx"
|
||||
fi
|
||||
|
||||
# ── Banner ────────────────────────────────────────────────────
|
||||
echo ""
|
||||
printf " ${C_TITLE}%s${C_RST}\n" "💎 Gemma 4 MLX Installer"
|
||||
printf " ${C_DIM}%s${C_RST}\n" "$RULE"
|
||||
echo ""
|
||||
|
||||
# ── Platform check ────────────────────────────────────────────
|
||||
if [ "$(uname)" != "Darwin" ]; then
|
||||
fail "MLX requires macOS with Apple Silicon. Detected: $(uname)"
|
||||
fi
|
||||
|
||||
_ARCH=$(uname -m)
|
||||
if [ "$_ARCH" != "arm64" ]; then
|
||||
step "warning" "Apple Silicon recommended (detected: $_ARCH)" "$C_WARN"
|
||||
fi
|
||||
|
||||
step "platform" "macOS ($_ARCH)"
|
||||
|
||||
# ── Detect Python ─────────────────────────────────────────────
|
||||
PYTHON=""
|
||||
for _candidate in python3.12 python3.11 python3.13 python3; do
|
||||
if command -v "$_candidate" >/dev/null 2>&1; then
|
||||
PYTHON="$_candidate"
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
if [ -z "$PYTHON" ]; then
|
||||
fail "Python 3 not found. Install via: brew install python@3.12"
|
||||
fi
|
||||
|
||||
_PY_VERSION=$("$PYTHON" -c "import sys; print(f'{sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro}')")
|
||||
step "python" "$PYTHON ($_PY_VERSION)"
|
||||
|
||||
# ── Create virtual environment ────────────────────────────────
|
||||
if [ -x "$VENV_DIR/bin/python" ]; then
|
||||
step "venv" "using existing environment"
|
||||
substep "$VENV_DIR"
|
||||
else
|
||||
step "venv" "creating virtual environment"
|
||||
substep "$VENV_DIR"
|
||||
mkdir -p "$(dirname "$VENV_DIR")"
|
||||
"$PYTHON" -m venv "$VENV_DIR"
|
||||
fi
|
||||
|
||||
# ── Install uv ───────────────────────────────────────────────
|
||||
_UV_INSTALLER_SHA256="48cd5aca5d5671a3b3d5f61538cc8622e4434af63319115159990d8b0dd02416"
|
||||
|
||||
if ! command -v uv >/dev/null 2>&1; then
|
||||
step "uv" "installing uv package manager..."
|
||||
_uv_tmp=$(mktemp)
|
||||
curl -LsSf "https://astral.sh/uv/install.sh" -o "$_uv_tmp"
|
||||
_uv_actual=$(shasum -a 256 "$_uv_tmp" | awk '{print $1}')
|
||||
if [ "$_uv_actual" != "$_UV_INSTALLER_SHA256" ]; then
|
||||
rm -f "$_uv_tmp"
|
||||
fail "uv installer SHA-256 mismatch: got $_uv_actual expected $_UV_INSTALLER_SHA256 (refusing to execute)"
|
||||
fi
|
||||
sh "$_uv_tmp" </dev/null >/dev/null 2>&1
|
||||
rm -f "$_uv_tmp"
|
||||
if [ -f "$HOME/.local/bin/env" ]; then
|
||||
. "$HOME/.local/bin/env"
|
||||
fi
|
||||
export PATH="$HOME/.local/bin:$PATH"
|
||||
substep "done"
|
||||
else
|
||||
step "uv" "found $(uv --version 2>/dev/null || echo 'uv')"
|
||||
fi
|
||||
|
||||
_VENV_PY="$VENV_DIR/bin/python"
|
||||
|
||||
# ── Install dependencies ──────────────────────────────────────
|
||||
step "install" "installing mlx-vlm..."
|
||||
uv pip install --python "$_VENV_PY" -q mlx-vlm
|
||||
substep "done"
|
||||
|
||||
step "install" "installing transformers>=5.5.0..."
|
||||
if uv pip install --python "$_VENV_PY" -q "transformers>=5.5.0" 2>/dev/null; then
|
||||
substep "installed from PyPI"
|
||||
else
|
||||
substep "PyPI install failed (Python <3.10?), trying GitHub..."
|
||||
if uv pip install --python "$_VENV_PY" -q "git+https://github.com/huggingface/transformers.git@v5.5-release" 2>/dev/null; then
|
||||
substep "installed from huggingface/transformers v5.5-release"
|
||||
else
|
||||
step "warning" "could not install transformers>=5.5.0" "$C_WARN"
|
||||
substep "tried: PyPI, huggingface/transformers v5.5-release"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Verify installation ──────────────────────────────────────
|
||||
if "$_VENV_PY" -c "import mlx_vlm"; then
|
||||
substep "mlx-vlm verified"
|
||||
else
|
||||
fail "Installation verification failed."
|
||||
fi
|
||||
|
||||
# ── Done ──────────────────────────────────────────────────────
|
||||
echo ""
|
||||
printf " ${C_TITLE}%s${C_RST}\n" "Gemma 4 MLX installed!"
|
||||
printf " ${C_DIM}%s${C_RST}\n" "$RULE"
|
||||
echo ""
|
||||
step "available models" "unsloth/gemma-4-E2B-it-UD-MLX-4bit"
|
||||
substep "unsloth/gemma-4-E4B-it-UD-MLX-4bit"
|
||||
substep "unsloth/gemma-4-26b-a4b-it-UD-MLX-4bit"
|
||||
substep "unsloth/gemma-4-31b-it-UD-MLX-4bit"
|
||||
echo ""
|
||||
step "venv activate" "source ${VENV_DIR}/bin/activate"
|
||||
echo ""
|
||||
step "text chat" "python -m mlx_vlm.chat --model unsloth/gemma-4-E2B-it-UD-MLX-4bit"
|
||||
echo ""
|
||||
step "vision chat" "python -m mlx_vlm.chat --model unsloth/gemma-4-31b-it-UD-MLX-4bit"
|
||||
substep "Use /image path/to/image.jpg to load an image"
|
||||
echo ""
|
||||
step "gradio UI" "python -m mlx_vlm.chat_ui --model unsloth/gemma-4-31b-it-UD-MLX-4bit"
|
||||
echo ""
|
||||
printf " ${C_DIM}%s${C_RST}\n" "$RULE"
|
||||
echo ""
|
||||
@@ -0,0 +1,247 @@
|
||||
#!/bin/bash
|
||||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
set -euo pipefail
|
||||
|
||||
# ============================================================
|
||||
# Qwen3.6 MLX — One-command setup + inference
|
||||
#
|
||||
# Supply-chain hardening:
|
||||
# - All third-party downloads (uv installer, mlx_vlm qwen3_5
|
||||
# patches) are pinned to an immutable git commit SHA and verified
|
||||
# against a hardcoded SHA-256. Any mismatch aborts the install
|
||||
# before the bytes are copied into site-packages.
|
||||
# - To rotate any pin, fetch the new file with `curl`, run
|
||||
# `shasum -a 256`, and update the corresponding constant below.
|
||||
# ============================================================
|
||||
#
|
||||
# Usage:
|
||||
# bash install_qwen3_6_mlx.sh [--venv-dir DIR]
|
||||
#
|
||||
# This script:
|
||||
# 1. Creates a Python virtual environment
|
||||
# 2. Installs uv, mlx-vlm, transformers, torch, torchvision
|
||||
# ============================================================
|
||||
|
||||
# ── Output style (inspired by unsloth/install.sh) ─────────────
|
||||
RULE=""
|
||||
_rule_i=0
|
||||
while [ "$_rule_i" -lt 52 ]; do
|
||||
RULE="${RULE}─"
|
||||
_rule_i=$((_rule_i + 1))
|
||||
done
|
||||
|
||||
if [ -n "${NO_COLOR:-}" ]; then
|
||||
C_TITLE= C_DIM= C_OK= C_WARN= C_ERR= C_RST=
|
||||
elif [ -t 1 ] || [ -n "${FORCE_COLOR:-}" ]; then
|
||||
_ESC="$(printf '\033')"
|
||||
C_TITLE="${_ESC}[38;5;117m"
|
||||
C_DIM="${_ESC}[38;5;245m"
|
||||
C_OK="${_ESC}[38;5;108m"
|
||||
C_WARN="${_ESC}[38;5;136m"
|
||||
C_ERR="${_ESC}[91m"
|
||||
C_RST="${_ESC}[0m"
|
||||
else
|
||||
C_TITLE= C_DIM= C_OK= C_WARN= C_ERR= C_RST=
|
||||
fi
|
||||
|
||||
step() { printf " ${C_DIM}%-18.18s${C_RST}${3:-$C_OK}%s${C_RST}\n" "$1" "$2"; }
|
||||
substep() { printf " ${C_DIM}%-18s${2:-$C_DIM}%s${C_RST}\n" "" "$1"; }
|
||||
fail() { step "error" "$1" "$C_ERR"; exit 1; }
|
||||
|
||||
# ── Parse flags ───────────────────────────────────────────────
|
||||
VENV_DIR=""
|
||||
_next_is_venv=false
|
||||
|
||||
for arg in "$@"; do
|
||||
if [ "$_next_is_venv" = true ]; then
|
||||
VENV_DIR="$arg"
|
||||
_next_is_venv=false
|
||||
continue
|
||||
fi
|
||||
case "$arg" in
|
||||
--venv-dir) _next_is_venv=true ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Default venv location
|
||||
if [ -z "$VENV_DIR" ]; then
|
||||
VENV_DIR="$HOME/.unsloth/unsloth_qwen3_6_mlx"
|
||||
fi
|
||||
|
||||
# ── Banner ────────────────────────────────────────────────────
|
||||
echo ""
|
||||
printf " ${C_TITLE}%s${C_RST}\n" "Qwen3.6 MLX Installer"
|
||||
printf " ${C_DIM}%s${C_RST}\n" "$RULE"
|
||||
echo ""
|
||||
|
||||
# ── Platform check ────────────────────────────────────────────
|
||||
if [ "$(uname)" != "Darwin" ]; then
|
||||
fail "MLX requires macOS with Apple Silicon. Detected: $(uname)"
|
||||
fi
|
||||
|
||||
_ARCH=$(uname -m)
|
||||
if [ "$_ARCH" != "arm64" ]; then
|
||||
step "warning" "Apple Silicon recommended (detected: $_ARCH)" "$C_WARN"
|
||||
fi
|
||||
|
||||
step "platform" "macOS ($_ARCH)"
|
||||
|
||||
# ── Detect Python ─────────────────────────────────────────────
|
||||
PYTHON=""
|
||||
for _candidate in python3.12 python3.11 python3.13 python3; do
|
||||
if command -v "$_candidate" >/dev/null 2>&1; then
|
||||
PYTHON="$_candidate"
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
if [ -z "$PYTHON" ]; then
|
||||
fail "Python 3 not found. Install via: brew install python@3.12"
|
||||
fi
|
||||
|
||||
_PY_VERSION=$("$PYTHON" -c "import sys; print(f'{sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro}')")
|
||||
step "python" "$PYTHON ($_PY_VERSION)"
|
||||
|
||||
# ── Create virtual environment ────────────────────────────────
|
||||
if [ -x "$VENV_DIR/bin/python" ]; then
|
||||
step "venv" "using existing environment"
|
||||
substep "$VENV_DIR"
|
||||
else
|
||||
step "venv" "creating virtual environment"
|
||||
substep "$VENV_DIR"
|
||||
mkdir -p "$(dirname "$VENV_DIR")"
|
||||
"$PYTHON" -m venv "$VENV_DIR"
|
||||
fi
|
||||
|
||||
# ── Install uv ───────────────────────────────────────────────
|
||||
# Pin the uv installer payload by SHA-256. Rotate by running:
|
||||
# curl -sSLf https://astral.sh/uv/install.sh | shasum -a 256
|
||||
# and updating the constant below. We fetch into a temp file, verify
|
||||
# the digest, and only then execute. Mismatch aborts.
|
||||
_UV_INSTALLER_SHA256="48cd5aca5d5671a3b3d5f61538cc8622e4434af63319115159990d8b0dd02416"
|
||||
|
||||
if ! command -v uv >/dev/null 2>&1; then
|
||||
step "uv" "installing uv package manager..."
|
||||
_uv_tmp=$(mktemp)
|
||||
curl -LsSf "https://astral.sh/uv/install.sh" -o "$_uv_tmp"
|
||||
_uv_actual=$(shasum -a 256 "$_uv_tmp" | awk '{print $1}')
|
||||
if [ "$_uv_actual" != "$_UV_INSTALLER_SHA256" ]; then
|
||||
rm -f "$_uv_tmp"
|
||||
fail "uv installer SHA-256 mismatch: got $_uv_actual expected $_UV_INSTALLER_SHA256 (refusing to execute)"
|
||||
fi
|
||||
sh "$_uv_tmp" </dev/null
|
||||
rm -f "$_uv_tmp"
|
||||
if [ -f "$HOME/.local/bin/env" ]; then
|
||||
. "$HOME/.local/bin/env"
|
||||
fi
|
||||
export PATH="$HOME/.local/bin:$PATH"
|
||||
substep "done"
|
||||
else
|
||||
step "uv" "found $(uv --version 2>/dev/null || echo 'uv')"
|
||||
fi
|
||||
|
||||
_VENV_PY="$VENV_DIR/bin/python"
|
||||
|
||||
# ── Install dependencies ──────────────────────────────────────
|
||||
step "install" "installing mlx-vlm..."
|
||||
uv pip install --python "$_VENV_PY" -q mlx-vlm
|
||||
substep "done"
|
||||
|
||||
step "install" "installing transformers>=5.2.0..."
|
||||
if uv pip install --python "$_VENV_PY" -q "transformers>=5.2.0"; then
|
||||
substep "installed from PyPI"
|
||||
else
|
||||
substep "PyPI install failed, trying GitHub..."
|
||||
if uv pip install --python "$_VENV_PY" -q "git+https://github.com/huggingface/transformers.git"; then
|
||||
substep "installed from huggingface/transformers main"
|
||||
else
|
||||
fail "Could not install transformers>=5.2.0 (required for Qwen3.5/3.6 model support). Please check your Python version (>=3.10 required) and network connection, then try again."
|
||||
fi
|
||||
fi
|
||||
|
||||
step "install" "installing torch + torchvision (needed for Qwen3 VL processor)..."
|
||||
uv pip install --python "$_VENV_PY" -q torch torchvision
|
||||
substep "done"
|
||||
|
||||
# ── Verify installation ──────────────────────────────────────
|
||||
if "$_VENV_PY" -c "import mlx_vlm; import torch; import torchvision; import transformers"; then
|
||||
substep "mlx-vlm + torch + transformers verified"
|
||||
else
|
||||
fail "Installation verification failed. Please ensure Python >=3.10 and try again."
|
||||
fi
|
||||
|
||||
# ── Apply patches for multi-turn image chat ──────────────────
|
||||
#
|
||||
# Pin every patch to an immutable commit SHA and verify the body
|
||||
# against a hardcoded SHA-256. The mlx_vlm_qwen3_5 patch tree
|
||||
# currently only exists on the upstream `fix/ui-fix` branch; we pin
|
||||
# to the branch HEAD commit, NOT the floating ref, so a forced push
|
||||
# on `fix/ui-fix` cannot swap the bytes under us.
|
||||
#
|
||||
# Rotate by:
|
||||
# _PATCH_COMMIT=<new SHA>
|
||||
# curl -sSLf "https://raw.githubusercontent.com/unslothai/unsloth/$_PATCH_COMMIT/unsloth/models/patches/mlx_vlm_qwen3_5/qwen3_5.py" | shasum -a 256
|
||||
# curl -sSLf "https://raw.githubusercontent.com/unslothai/unsloth/$_PATCH_COMMIT/unsloth/models/patches/mlx_vlm_qwen3_5/generate.py" | shasum -a 256
|
||||
_PATCH_COMMIT="013c99e51bbb8c4b83d88f3b150a1e53251a19d2"
|
||||
_PATCH_BASE="https://raw.githubusercontent.com/unslothai/unsloth/${_PATCH_COMMIT}/unsloth/models/patches/mlx_vlm_qwen3_5"
|
||||
_PATCH_SHA_QWEN35="4b6fbbcc59b1d6b935e7204351aae1476836d25542a11c7885402b672d2efa64"
|
||||
_PATCH_SHA_GENERATE="50c4cbb8c3d94c0c74a4d209db6d2b23b102944c147c6421f2eded427b8edaf7"
|
||||
|
||||
_SITE_PKGS=$("$_VENV_PY" -c "import site; print(site.getsitepackages()[0])")
|
||||
|
||||
step "patch" "fixing multi-turn image chat..."
|
||||
|
||||
# Stage all downloads in an isolated tmpdir; we only copy into
|
||||
# site-packages after every checksum has matched.
|
||||
_PATCH_TMP=$(mktemp -d)
|
||||
trap 'rm -rf "$_PATCH_TMP"' EXIT
|
||||
|
||||
apply_pinned_patch() {
|
||||
# apply_pinned_patch <remote_basename> <expected_sha256> <dest_abspath>
|
||||
_name="$1"; _expected="$2"; _dest="$3"
|
||||
_staged="$_PATCH_TMP/$_name"
|
||||
if ! curl -sSLf "${_PATCH_BASE}/${_name}" -o "$_staged"; then
|
||||
step "warning" "failed to download ${_name} patch — multi-turn image chat may not work" "$C_WARN"
|
||||
return 1
|
||||
fi
|
||||
_actual=$(shasum -a 256 "$_staged" | awk '{print $1}')
|
||||
if [ "$_actual" != "$_expected" ]; then
|
||||
step "warning" "${_name} SHA-256 mismatch (got $_actual expected $_expected) — refusing to install patch" "$C_WARN"
|
||||
return 1
|
||||
fi
|
||||
mkdir -p "$(dirname "$_dest")"
|
||||
cp "$_staged" "$_dest"
|
||||
return 0
|
||||
}
|
||||
|
||||
if apply_pinned_patch "qwen3_5.py" "$_PATCH_SHA_QWEN35" "${_SITE_PKGS}/mlx_vlm/models/qwen3_5/qwen3_5.py"; then
|
||||
substep "patched qwen3_5.py (MRoPE position reset)"
|
||||
fi
|
||||
|
||||
if apply_pinned_patch "generate.py" "$_PATCH_SHA_GENERATE" "${_SITE_PKGS}/mlx_vlm/generate.py"; then
|
||||
substep "patched generate.py (mask trim on cache reuse)"
|
||||
fi
|
||||
|
||||
# Clear pycache so patches take effect
|
||||
find "${_SITE_PKGS}/mlx_vlm" -name "__pycache__" -type d -exec rm -rf {} + 2>/dev/null || true
|
||||
substep "cleared bytecode cache"
|
||||
|
||||
# ── Done ──────────────────────────────────────────────────────
|
||||
echo ""
|
||||
printf " ${C_TITLE}%s${C_RST}\n" "Qwen3.6 MLX installed!"
|
||||
printf " ${C_DIM}%s${C_RST}\n" "$RULE"
|
||||
echo ""
|
||||
step "available models" "unsloth/Qwen3.6-35B-A3B-UD-MLX-3bit"
|
||||
substep "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit"
|
||||
substep "unsloth/Qwen3.6-35B-A3B-MLX-8bit"
|
||||
echo ""
|
||||
step "venv activate" "source ${VENV_DIR}/bin/activate"
|
||||
echo ""
|
||||
step "vision chat" "python -m mlx_vlm.chat --model unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit"
|
||||
substep "Use /image path/to/image.jpg to load an image"
|
||||
echo ""
|
||||
step "gradio UI" "python -m mlx_vlm.chat_ui --model unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit"
|
||||
echo ""
|
||||
printf " ${C_DIM}%s${C_RST}\n" "$RULE"
|
||||
echo ""
|
||||
@@ -0,0 +1,310 @@
|
||||
#!/usr/bin/env bash
|
||||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
|
||||
#
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
# Enable ROCm-on-WSL for AMD GPUs (Strix Halo/Point APUs AND discrete Radeon RX
|
||||
# 7000/9000). Verified on gfx1151 (Radeon 8060S) and gfx1200 (Radeon RX 9060 XT).
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
# install.sh routes the detected arch to the right ROCm wheels once a runtime exists;
|
||||
# what it does NOT do is install AMD's ROCm userspace + the WSL DXG bridge (librocdxg).
|
||||
# This helper does that Linux-side prerequisite on Ubuntu 24.04 WSL2, invoked by
|
||||
# install.sh when it sees an AMD GPU via /dev/dxg but no ROCm yet. Arch-agnostic: the
|
||||
# arch is auto-detected from rocminfo (override UNSLOTH_WSL_GFX=gfx1200). Idempotent.
|
||||
#
|
||||
# Manual, admin-gated Windows prerequisite: an AMD Adrenalin driver with
|
||||
# production ROCDXG/WSL support (26.2.2+). install.ps1 offers to update it. Once
|
||||
# installed + rebooted, /dev/dxg is exposed to WSL and this script builds the rest.
|
||||
#
|
||||
# HOW ROCDXG WORKS (and why older /usr/lib/wsl/lib notes are wrong): librocdxg.so
|
||||
# is AMD's user-mode bridge between the Linux HSA runtime and the Windows driver
|
||||
# over /dev/dxg. The STANDARD hsa-rocr runtime (NOT the gone "roc4wsl" package)
|
||||
# loads it when HSA_ENABLE_DXG_DETECTION=1. No hsa/rocm libs need injecting into
|
||||
# /usr/lib/wsl/lib (it holds only d3d12/dxcore), yet rocminfo enumerates gfx1151
|
||||
# fine -- so we gate on /dev/dxg, not on WSL lib injection.
|
||||
#
|
||||
# KNOWN CAVEAT (ROCm/ROCm#6022): librocdxg can cap usable ROCm VRAM at the WSL
|
||||
# VM's RAM (.wslconfig [wsl2] memory=) on some BIOS UMA layouts, and amd-smi
|
||||
# doesn't work in WSL. On OOM below capacity, raise memory= (then wsl --shutdown)
|
||||
# and watch GPU use from Windows. Large-UMA BIOS exposes the full pool regardless.
|
||||
#
|
||||
# Verified on Ryzen AI Max+ PRO 395 / Radeon 8060S (gfx1151) with ROCm 7.2.1 +
|
||||
# Ubuntu 24.04 + WSL2 + Adrenalin. These pins MOVE; bump + re-verify on newer ROCm.
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
set -euo pipefail
|
||||
|
||||
# ── Tunables (override via env) ──────────────────────────────────────────────
|
||||
ROCM_VER="${UNSLOTH_WSL_ROCM_VER:-7.2.1}" # ROCm release to install
|
||||
# GPU arch: empty = auto-detect from rocminfo after install (override UNSLOTH_WSL_GFX=gfx1200).
|
||||
# The ROCm + librocdxg setup is arch-agnostic; only verify + the smoke test need the arch.
|
||||
GFX="${UNSLOTH_WSL_GFX:-}"
|
||||
LIBROCDXG_REF="${UNSLOTH_LIBROCDXG_REF:-develop}" # ROCm/librocdxg git ref to build
|
||||
# AMD's wheel index for the (optional) smoke test; resolved after arch detection.
|
||||
TORCH_INDEX=""
|
||||
# Optional torch smoke test (throwaway venv). OFF by default: install.sh installs
|
||||
# torch itself into the real venv right after, so a duplicate download is wasteful.
|
||||
SMOKE_TEST="${UNSLOTH_WSL_SMOKE_TEST:-0}"
|
||||
# REQUIRED constraint -- without it pip prefers PyPI's newer CUDA torch over the
|
||||
# gfx1151 ROCm wheel. 2.11 carries AMD's real gfx1151 fix (matches install.sh).
|
||||
TORCH_CONSTRAINT="${UNSLOTH_WSL_TORCH_CONSTRAINT:-torch>=2.11.0,<2.12.0}"
|
||||
ROCM_DIR="" # resolved after install
|
||||
|
||||
say() { printf '\n\033[1;36m== %s\033[0m\n' "$*"; }
|
||||
note() { printf ' %s\n' "$*"; }
|
||||
die() { printf '\n\033[1;31m[BLOCKED] %s\033[0m\n' "$*" >&2; exit 1; }
|
||||
|
||||
# sudo only if not already root (WSL distros often run as root)
|
||||
SUDO=""
|
||||
if [ "$(id -u)" -ne 0 ]; then
|
||||
command -v sudo >/dev/null 2>&1 || die "Need root or sudo to install ROCm."
|
||||
SUDO="sudo"
|
||||
fi
|
||||
|
||||
# ── Windows 11 SDK (headers for the librocdxg build) ─────────────────────────
|
||||
# librocdxg's cmake build needs the Windows SDK 'shared' headers, which live on
|
||||
# the Windows HOST under C:\Program Files (x86)\Windows Kits\10\Include\<ver>\.
|
||||
_WIN_SDK_INC_BASE="/mnt/c/Program Files (x86)/Windows Kits/10/Include"
|
||||
|
||||
# Print the newest installed SDK include dir with 'shared' headers, or nothing.
|
||||
# find + read loop (not `for ... in $(ls)`) since the base path has a space.
|
||||
_find_win_sdk() {
|
||||
[ -d "$_WIN_SDK_INC_BASE" ] || return 0
|
||||
while IFS= read -r _inc; do
|
||||
[ -n "$_inc" ] || continue
|
||||
if [ -d "$_inc/shared" ]; then printf '%s' "$_inc"; return 0; fi
|
||||
done < <(find "$_WIN_SDK_INC_BASE" -mindepth 1 -maxdepth 1 -type d 2>/dev/null | sort -Vr)
|
||||
return 0
|
||||
}
|
||||
|
||||
# Best-effort: install the Windows 11 SDK on the Windows HOST via winget so the
|
||||
# build has its headers with no manual step. Elevates -> ONE UAC prompt; headers
|
||||
# appear under /mnt/c immediately (no reboot). Never fatal -- failure falls
|
||||
# through to a manual-install message. Opt out: UNSLOTH_SKIP_WIN_SDK_INSTALL=1.
|
||||
_install_windows_sdk_via_winget() {
|
||||
[ "${UNSLOTH_SKIP_WIN_SDK_INSTALL:-0}" = "1" ] && { note "Skipping Windows SDK auto-install (UNSLOTH_SKIP_WIN_SDK_INSTALL=1)."; return 0; }
|
||||
command -v powershell.exe >/dev/null 2>&1 || return 0
|
||||
# `command -v` succeeds even with WSL interop OFF (.exe on PATH but fails
|
||||
# with "Exec format error"); verify it actually executes.
|
||||
powershell.exe -NoProfile -Command "exit 0" >/dev/null 2>&1 || return 0
|
||||
if ! powershell.exe -NoProfile -Command "if (Get-Command winget -ErrorAction SilentlyContinue) { exit 0 } else { exit 1 }" >/dev/null 2>&1; then
|
||||
note "winget not available on the Windows host -- cannot auto-install the Windows SDK."
|
||||
return 0
|
||||
fi
|
||||
say "Installing the Windows 11 SDK on the Windows host via winget"
|
||||
note "librocdxg needs its headers. Approve the UAC prompt on the Windows desktop."
|
||||
note "One-time (~1-3 GB download); opt out with UNSLOTH_SKIP_WIN_SDK_INSTALL=1."
|
||||
# Newest SDK first, then a fallback. Header presence is the source of truth
|
||||
# (re-check each attempt), not winget's exit code. </dev/null so winget never
|
||||
# consumes a piped `curl | sh` stdin.
|
||||
for _sdk_id in Microsoft.WindowsSDK.10.0.26100 Microsoft.WindowsSDK.10.0.22621; do
|
||||
note "winget install ${_sdk_id} ..."
|
||||
# --source winget: pin the community source so a broken default msstore
|
||||
# source (the cert failure this PR fixes) can't abort SDK resolution.
|
||||
powershell.exe -NoProfile -Command "winget install --id ${_sdk_id} -e --source winget --accept-source-agreements --accept-package-agreements --disable-interactivity" </dev/null || true
|
||||
if [ -n "$(_find_win_sdk)" ]; then
|
||||
note "Windows SDK headers present after install."
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
note "Automatic Windows SDK install did not complete."
|
||||
return 0
|
||||
}
|
||||
|
||||
# ── PREFLIGHT ────────────────────────────────────────────────────────────────
|
||||
say "Preflight checks"
|
||||
|
||||
# shellcheck disable=SC1091
|
||||
. /etc/os-release 2>/dev/null || true
|
||||
if [ "${VERSION_ID:-}" != "24.04" ]; then
|
||||
die "This targets Ubuntu 24.04 (found '${VERSION_ID:-unknown}'). AMD's ROCm-on-WSL supports 24.04; create a dedicated distro: wsl --install Ubuntu-24.04 (do not run on 26.04 -- ROCm 7.2 does not target it yet)."
|
||||
fi
|
||||
|
||||
if [ ! -e /dev/dxg ]; then
|
||||
die "/dev/dxg missing -- WSL GPU paravirtualization not present. Ensure this is WSL2 (not WSL1) on a recent Windows build, and that an AMD GPU + ROCDXG-capable Adrenalin driver is installed on the Windows host (then reboot)."
|
||||
fi
|
||||
note "Ubuntu 24.04 + /dev/dxg present."
|
||||
# Don't block on hsa/rocm libs in /usr/lib/wsl/lib: a working ROCDXG setup
|
||||
# doesn't need them (only d3d12/dxcore). Real readiness is checked via rocminfo.
|
||||
|
||||
# ── Step 1: build/runtime prerequisites ──────────────────────────────────────
|
||||
say "Installing build prerequisites"
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
$SUDO apt-get update -y
|
||||
# `make` is explicit: cmake shells out to it but Ubuntu only *recommends* it, so
|
||||
# minimal images lack it and the librocdxg `make -j` build would fail.
|
||||
$SUDO apt-get install -y cmake make gcc g++ git wget gpg ca-certificates python3-venv python3-pip
|
||||
|
||||
# ── Step 2: ROCm ${ROCM_VER} userspace (no DKMS -- WSL uses the Windows driver) ─
|
||||
say "Installing ROCm ${ROCM_VER} userspace"
|
||||
if ! command -v rocminfo >/dev/null 2>&1 && [ ! -x /opt/rocm/bin/rocminfo ]; then
|
||||
# Direct apt-repo install (leaner than amdgpu-install; repo is indexed by
|
||||
# ROCm version, e.g. .../apt/7.2.1).
|
||||
$SUDO mkdir -p /etc/apt/keyrings
|
||||
wget -qO- https://repo.radeon.com/rocm/rocm.gpg.key \
|
||||
| gpg --dearmor | $SUDO tee /etc/apt/keyrings/rocm.gpg >/dev/null
|
||||
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/${ROCM_VER} noble main" \
|
||||
| $SUDO tee /etc/apt/sources.list.d/rocm.list >/dev/null
|
||||
printf 'Package: *\nPin: release o=repo.radeon.com\nPin-Priority: 600\n' \
|
||||
| $SUDO tee /etc/apt/preferences.d/rocm-pin-600 >/dev/null
|
||||
$SUDO apt-get update -y
|
||||
# rocm-libs pulls everything torch links at runtime (rocblas, hipblas,
|
||||
# miopen-hip, rccl, ...); hsa-rocr + rocminfo come as deps. Large (~5 GB
|
||||
# download / ~23 GB installed).
|
||||
$SUDO apt-get install -y rocm-libs rocminfo hip-runtime-amd
|
||||
else
|
||||
note "ROCm already present -- skipping apt install."
|
||||
fi
|
||||
|
||||
# Resolve the real ROCm dir and ensure the canonical /opt/rocm symlink. apt lays
|
||||
# ROCm under /opt/rocm-<ver> and rocm-core symlinks /opt/rocm -> that; repair if
|
||||
# an earlier partial run left /opt/rocm as a real dir blocking the symlink.
|
||||
_real="$(ls -d /opt/rocm-* 2>/dev/null | sort -V | tail -1 || true)"
|
||||
if [ -n "$_real" ] && [ ! -L /opt/rocm ] && [ -d /opt/rocm ]; then
|
||||
# /opt/rocm is a real dir blocking the symlink. Only treat it as a removable
|
||||
# stray stub if it's NOT a real ROCm install (a real one has bin/rocminfo /
|
||||
# bin/hipcc / .info/version) -- this protects a user's pre-existing ROCm. Even
|
||||
# then we MOVE IT ASIDE, never rm -rf, so a wrong guess can't lose data.
|
||||
if [ -e /opt/rocm/bin/rocminfo ] || [ -e /opt/rocm/bin/hipcc ] || [ -e /opt/rocm/.info/version ]; then
|
||||
note "/opt/rocm is a real ROCm install -- leaving it untouched (will install librocdxg into it)."
|
||||
else
|
||||
note "Moving stray /opt/rocm stub aside -> $_real (not deleting it)"
|
||||
$SUDO cp -an /opt/rocm/. "$_real"/ 2>/dev/null || true
|
||||
$SUDO mv /opt/rocm "/opt/rocm.unsloth-stub-bak.$(date +%s)" 2>/dev/null || true
|
||||
[ -e /opt/rocm ] || $SUDO ln -s "$_real" /opt/rocm
|
||||
fi
|
||||
elif [ -n "$_real" ] && [ ! -e /opt/rocm ]; then
|
||||
$SUDO ln -s "$_real" /opt/rocm
|
||||
fi
|
||||
if [ -L /opt/rocm ] || [ -d /opt/rocm ]; then ROCM_DIR="/opt/rocm"; else ROCM_DIR="$_real"; fi
|
||||
{ [ -n "$ROCM_DIR" ] && [ -d "$ROCM_DIR" ]; } || die "ROCm not found under /opt after install."
|
||||
note "ROCm at ${ROCM_DIR}"
|
||||
|
||||
# ── Step 3: build librocdxg (DXG <-> HSA bridge; not yet an apt package) ──────
|
||||
say "Building librocdxg (${LIBROCDXG_REF})"
|
||||
if [ -e "${ROCM_DIR}/lib/librocdxg.so" ]; then
|
||||
note "librocdxg already installed -- skipping build."
|
||||
else
|
||||
# Discover the newest installed Win11 SDK (version differs per machine). If
|
||||
# absent, auto-install via winget (one UAC prompt) and re-discover; only if
|
||||
# that ALSO fails do we stop with manual instructions.
|
||||
_win_sdk="$(_find_win_sdk)"
|
||||
if [ -z "$_win_sdk" ]; then
|
||||
note "Windows 11 SDK headers not found -- attempting automatic install..."
|
||||
_install_windows_sdk_via_winget
|
||||
_win_sdk="$(_find_win_sdk)"
|
||||
fi
|
||||
[ -n "$_win_sdk" ] || die "Windows 11 SDK headers not found under 'C:\\Program Files (x86)\\Windows Kits\\10\\Include\\*\\shared', and the automatic winget install did not complete. Install it on the Windows host (e.g. 'winget install Microsoft.WindowsSDK.10.0.26100') and re-run."
|
||||
note "Windows SDK: ${_win_sdk}"
|
||||
_src="${HOME}/.unsloth/librocdxg"
|
||||
rm -rf "$_src"
|
||||
git clone --depth 1 --branch "$LIBROCDXG_REF" https://github.com/ROCm/librocdxg.git "$_src" \
|
||||
|| git clone "https://github.com/ROCm/librocdxg.git" "$_src"
|
||||
(
|
||||
cd "$_src"
|
||||
git checkout "$LIBROCDXG_REF" 2>/dev/null || true
|
||||
mkdir -p build && cd build
|
||||
cmake .. -DWIN_SDK="${_win_sdk}/shared"
|
||||
make -j"$(nproc)"
|
||||
$SUDO make install
|
||||
)
|
||||
fi
|
||||
# Ensure soname symlinks resolve to whatever version was built (e.g. 1.2.0).
|
||||
_dxg_real="$(ls -1 "${ROCM_DIR}"/lib/librocdxg.so.*.* 2>/dev/null | sort -V | tail -1 || true)"
|
||||
if [ -n "$_dxg_real" ]; then
|
||||
_dxg_base="$(basename "$_dxg_real")" # librocdxg.so.1.2.0
|
||||
_dxg_major="$(printf '%s' "$_dxg_base" | sed -E 's/librocdxg\.so\.([0-9]+).*/\1/')"
|
||||
$SUDO ln -sf "$_dxg_base" "${ROCM_DIR}/lib/librocdxg.so.${_dxg_major}"
|
||||
$SUDO ln -sf "librocdxg.so.${_dxg_major}" "${ROCM_DIR}/lib/librocdxg.so"
|
||||
fi
|
||||
echo "${ROCM_DIR}/lib" | $SUDO tee /etc/ld.so.conf.d/rocm.conf >/dev/null
|
||||
$SUDO ldconfig
|
||||
|
||||
# ── Step 4: persist environment (system-wide so Studio's worker inherits it) ──
|
||||
say "Persisting ROCm-on-WSL environment"
|
||||
_envfile="/etc/profile.d/unsloth-rocm-wsl.sh"
|
||||
$SUDO tee "$_envfile" >/dev/null <<EOF
|
||||
# >>> Unsloth ROCm-on-WSL >>>
|
||||
export HSA_ENABLE_DXG_DETECTION=1
|
||||
export TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1
|
||||
export PATH="${ROCM_DIR}/bin:\${PATH}"
|
||||
export LD_LIBRARY_PATH="${ROCM_DIR}/lib:\${LD_LIBRARY_PATH:-}"
|
||||
# <<< Unsloth ROCm-on-WSL <<<
|
||||
EOF
|
||||
# also drop into ~/.bashrc for interactive shells
|
||||
if [ -n "${HOME:-}" ] && ! grep -q "Unsloth ROCm-on-WSL" "${HOME}/.bashrc" 2>/dev/null; then
|
||||
cat "$_envfile" >> "${HOME}/.bashrc"
|
||||
fi
|
||||
# export into the current process so verification below works immediately
|
||||
export HSA_ENABLE_DXG_DETECTION=1
|
||||
export PATH="${ROCM_DIR}/bin:${PATH}"
|
||||
export LD_LIBRARY_PATH="${ROCM_DIR}/lib:${LD_LIBRARY_PATH:-}"
|
||||
|
||||
# ── Step 5: verify the runtime enumerates the GPU ────────────────────────────
|
||||
say "Verifying rocminfo enumerates the GPU over DXG"
|
||||
# Capture rocminfo into a var BEFORE grepping: piping into `grep -q` SIGPIPEs
|
||||
# rocminfo on first match, which under `set -o pipefail` turns a successful match
|
||||
# into a pipeline failure.
|
||||
_rocminfo_out="$(rocminfo 2>/dev/null || true)"
|
||||
# GPU agents advertise an ISA "Name: gfxNNNN". Match gfx[1-9] (excludes gfx000, the CPU
|
||||
# agent), drop the "gfx*-generic" fallback ISA, and take the first real GPU arch.
|
||||
_detected_gfx="$(printf '%s\n' "$_rocminfo_out" | grep -E 'Name:[[:space:]]*gfx[1-9]' | grep -v 'generic' | grep -oE 'gfx[1-9][0-9a-z]*' | head -1 || true)"
|
||||
if [ -z "$_detected_gfx" ]; then
|
||||
printf '%s\n' "$_rocminfo_out" | head -25 >&2 || true
|
||||
die "rocminfo did not enumerate any GPU agent. Most common cause: the Windows AMD driver predates production ROCDXG -- update Adrenalin (install.ps1 offers this), reboot, and re-run."
|
||||
fi
|
||||
# Honour a caller-pinned arch (sanity-check via a consuming grep, not grep -q: under
|
||||
# pipefail -q would SIGPIPE printf on large output and misreport the arch); else adopt.
|
||||
if [ -n "$GFX" ] && ! printf '%s\n' "$_rocminfo_out" | grep -E "Name:[[:space:]]*${GFX}([^0-9]|$)" >/dev/null; then
|
||||
die "rocminfo enumerated '${_detected_gfx}' but not the requested UNSLOTH_WSL_GFX='${GFX}'."
|
||||
fi
|
||||
GFX="${GFX:-$_detected_gfx}"
|
||||
# Display-only summary: best-effort (|| true) so head's early pipe-close under
|
||||
# `set -o pipefail` can't fail the bootstrap after verification already passed.
|
||||
printf '%s\n' "$_rocminfo_out" | grep -E 'Marketing Name|Device Type|Compute Unit' | grep -iE "Radeon|GPU|Compute" | head -3 || true
|
||||
note "ROCm-on-WSL runtime is live for ${GFX}."
|
||||
|
||||
# ── Step 6 (optional): torch smoke test from AMD's per-arch wheel index ───────
|
||||
if [ "$SMOKE_TEST" = "1" ]; then
|
||||
say "Smoke-testing PyTorch on ${GFX} (throwaway venv)"
|
||||
# Map the detected arch to AMD's repo.amd.com wheel family index.
|
||||
case "$GFX" in
|
||||
gfx1200|gfx1201) _fam="gfx120X-all" ;;
|
||||
gfx1100|gfx1101|gfx1102|gfx1103) _fam="gfx110X-all" ;;
|
||||
*) _fam="$GFX" ;; # gfx1150/gfx1151/gfx90a: own index
|
||||
esac
|
||||
TORCH_INDEX="${UNSLOTH_AMD_ROCM_MIRROR:-https://repo.amd.com/rocm/whl}/${_fam}/"
|
||||
_venv="${HOME}/.unsloth/rocm-smoketest"
|
||||
rm -rf "$_venv"; python3 -m venv "$_venv"
|
||||
"$_venv/bin/pip" install --quiet --upgrade pip
|
||||
# AMD arch index is primary (torch + triton); PyPI only an extra for pure-py
|
||||
# deps. The constraint keeps pip on the ROCm wheel, not a newer PyPI CUDA torch.
|
||||
"$_venv/bin/pip" install --index-url "$TORCH_INDEX" \
|
||||
--extra-index-url https://pypi.org/simple "$TORCH_CONSTRAINT" || \
|
||||
die "torch install from ${TORCH_INDEX} failed."
|
||||
# WSL: torch's bundled ROCr must load the DXG bridge -- drop librocdxg into torch/lib.
|
||||
_tlib="$("$_venv/bin/python" -c 'import torch,os;print(os.path.join(os.path.dirname(torch.__file__),"lib"))' 2>/dev/null || true)"
|
||||
[ -d "$_tlib" ] && cp -f "${ROCM_DIR}"/lib/librocdxg.so* "$_tlib"/ 2>/dev/null || true
|
||||
"$_venv/bin/python" - <<'PY'
|
||||
import torch
|
||||
ok = torch.cuda.is_available()
|
||||
print("torch:", torch.__version__, "| cuda(rocm) available:", ok)
|
||||
if ok:
|
||||
print("device:", torch.cuda.get_device_name(0))
|
||||
free, total = torch.cuda.mem_get_info(0)
|
||||
print(f"mem: free={free/1e9:.1f} GB total={total/1e9:.1f} GB")
|
||||
import time
|
||||
a = torch.randn(4096, 4096, device="cuda", dtype=torch.float16)
|
||||
b = torch.randn(4096, 4096, device="cuda", dtype=torch.float16)
|
||||
torch.cuda.synchronize(); t0 = time.time()
|
||||
for _ in range(10): c = a @ b
|
||||
torch.cuda.synchronize()
|
||||
print(f"matmul ok ({(time.time()-t0)/10*1e3:.1f} ms/iter)")
|
||||
raise SystemExit(0 if ok else 1)
|
||||
PY
|
||||
rm -rf "$_venv"
|
||||
fi
|
||||
|
||||
say "Done."
|
||||
note "ROCm-on-WSL is ready for ${GFX}. If you ran this standalone, install Unsloth"
|
||||
note "in THIS distro and it will detect the GPU automatically:"
|
||||
note " curl -fsSL https://unsloth.ai/install.sh | sh"
|
||||
@@ -0,0 +1,150 @@
|
||||
#!/usr/bin/env python3
|
||||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
|
||||
|
||||
"""Refuse dangerous GitHub Actions trigger patterns at PR time.
|
||||
|
||||
Bans patterns behind the TanStack GHSA-g7cv-rxg3-hmpx compromise:
|
||||
|
||||
1. `pull_request_target` -- runs a fork's workflow against the base
|
||||
repo's secrets/permissions; use `pull_request` instead.
|
||||
2. `workflow_run` chained to a PR-triggered workflow -- same trust
|
||||
boundary problem one hop later (poisoned artifacts/caches run with
|
||||
elevated permissions).
|
||||
3. Cache keys shared between PR-triggered and publish/release/push
|
||||
workflows -- a fork PR could poison a cache the publish workflow
|
||||
restores. Partition the key namespaces.
|
||||
|
||||
Exit codes: 0 = no findings, 1 = findings (listed on stderr).
|
||||
Run from repo root: python3 scripts/lint_workflow_triggers.py
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError:
|
||||
print("ERROR: PyYAML is required. Install with 'pip install pyyaml'", file = sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
DEFAULT_WORKFLOWS_DIR = REPO_ROOT / ".github" / "workflows"
|
||||
|
||||
BANNED_TRIGGERS: tuple[str, ...] = ("pull_request_target",)
|
||||
RESTRICTED_TRIGGERS: tuple[str, ...] = ("workflow_run",)
|
||||
PUBLISH_WORKFLOW_NAMES: tuple[str, ...] = ("release-desktop.yml",)
|
||||
|
||||
|
||||
def _normalise_on(on_field):
|
||||
if isinstance(on_field, str):
|
||||
return {on_field}
|
||||
if isinstance(on_field, list):
|
||||
return set(on_field)
|
||||
if isinstance(on_field, dict):
|
||||
return set(on_field.keys())
|
||||
return set()
|
||||
|
||||
|
||||
def _load_workflow(path: Path):
|
||||
try:
|
||||
return yaml.safe_load(path.read_text())
|
||||
except Exception as exc:
|
||||
print(f"ERROR: failed to parse {path}: {exc}", file = sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
|
||||
def _extract_cache_keys(path: Path) -> list[str]:
|
||||
text = path.read_text()
|
||||
keys: list[str] = []
|
||||
for m in re.finditer(r"(?:^|\n)\s*key:\s*([^\n]+)", text):
|
||||
keys.append(m.group(1).strip())
|
||||
return keys
|
||||
|
||||
|
||||
def _trigger_set(yaml_doc) -> set[str]:
|
||||
on = yaml_doc.get(True)
|
||||
if on is None:
|
||||
on = yaml_doc.get("on")
|
||||
return _normalise_on(on)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description = __doc__)
|
||||
parser.add_argument(
|
||||
"--workflows-dir",
|
||||
type = Path,
|
||||
default = DEFAULT_WORKFLOWS_DIR,
|
||||
help = "Override the workflows directory (used by tests).",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
workflows_dir = args.workflows_dir
|
||||
|
||||
findings: list[str] = []
|
||||
workflows = sorted(workflows_dir.glob("*.yml"))
|
||||
pr_triggered: list[tuple[Path, list[str]]] = []
|
||||
publish_triggered: list[tuple[Path, list[str]]] = []
|
||||
|
||||
for path in workflows:
|
||||
doc = _load_workflow(path)
|
||||
triggers = _trigger_set(doc)
|
||||
|
||||
for t in BANNED_TRIGGERS:
|
||||
if t in triggers:
|
||||
findings.append(
|
||||
f"{path.name}: BANNED trigger '{t}' (GHSA-g7cv-rxg3-hmpx "
|
||||
"pattern: fork PRs run in base-repo context). Switch to "
|
||||
"'pull_request' and use a deploy-on-merge workflow for "
|
||||
"any privileged step."
|
||||
)
|
||||
|
||||
for t in RESTRICTED_TRIGGERS:
|
||||
if t in triggers:
|
||||
text = path.read_text()
|
||||
if "lint:workflow_triggers-allow-workflow_run" not in text:
|
||||
findings.append(
|
||||
f"{path.name}: RESTRICTED trigger '{t}' requires an "
|
||||
"explicit `# lint:workflow_triggers-allow-workflow_run` "
|
||||
"comment somewhere in the file, with a justification."
|
||||
)
|
||||
|
||||
if "pull_request" in triggers:
|
||||
pr_triggered.append((path, _extract_cache_keys(path)))
|
||||
is_dispatch_only = "workflow_dispatch" in triggers and not (
|
||||
"push" in triggers or "pull_request" in triggers
|
||||
)
|
||||
if path.name in PUBLISH_WORKFLOW_NAMES or is_dispatch_only:
|
||||
publish_triggered.append((path, _extract_cache_keys(path)))
|
||||
|
||||
pr_keys = {key for _, keys in pr_triggered for key in keys}
|
||||
for pub_path, pub_keys in publish_triggered:
|
||||
for k in pub_keys:
|
||||
if k in pr_keys:
|
||||
findings.append(
|
||||
f"{pub_path.name}: cache key {k!r} is also declared in a "
|
||||
"PR-triggered workflow. A fork PR could poison this cache "
|
||||
"and the publish workflow would restore it on next run. "
|
||||
"Add a unique suffix (e.g. '-publish-only') to partition "
|
||||
"the namespaces."
|
||||
)
|
||||
|
||||
if findings:
|
||||
print("Workflow trigger lint failed with the following issues:", file = sys.stderr)
|
||||
for f in findings:
|
||||
print(f" - {f}", file = sys.stderr)
|
||||
return 1
|
||||
|
||||
print(
|
||||
f"OK: scanned {len(workflows)} workflow file(s); "
|
||||
f"no pull_request_target, no unjustified workflow_run, "
|
||||
f"no PR/publish cache-key collision."
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,780 @@
|
||||
#!/usr/bin/env python3
|
||||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
|
||||
|
||||
"""Lockfile supply-chain audit for the Studio frontend and Tauri shell.
|
||||
|
||||
Runs BEFORE `npm ci` / `cargo fetch` in CI. Refuses to proceed when a
|
||||
lockfile contains patterns indicating supply-chain injection (npm
|
||||
Shai-Hulud waves, cargo crates.io brand-squats).
|
||||
|
||||
Checks package-lock.json (lockfileVersion 2/3): `resolved` URL must be
|
||||
the npm registry (direct git/github/file refs are the injection vector);
|
||||
`integrity` SHA must be present; known IOC substrings grepped from the
|
||||
body. Checks Cargo.lock: `source` must be the crates.io registry index;
|
||||
known cargo IOC substrings.
|
||||
|
||||
Exit codes: 0 = clean (or skip env var set to a justification >=5 chars,
|
||||
not '1'/'true'); 1 = findings; 2 = internal error.
|
||||
|
||||
Only PARSES the lockfiles, never executes or networks. Complements (not
|
||||
replaces) `npm audit` / OSV-Scanner / the advisory-DB pipeline. Fires
|
||||
before any third-party install script runs on the runner.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
# Known IOC strings (case-sensitive substring match). Each is tied to a
|
||||
# public advisory; speculative/generic patterns would false-positive on
|
||||
# upgrades.
|
||||
NPM_IOC_STRINGS: tuple[str, ...] = (
|
||||
# Shai-Hulud TanStack wave -- May 11, 2026 (GHSA-g7cv-rxg3-hmpx).
|
||||
"router_init.js",
|
||||
"tanstack_runner.js",
|
||||
"router_runtime.js",
|
||||
"@tanstack/setup",
|
||||
"github:tanstack/router#79ac49eedf774dd4b0cfa308722bc463cfe5885c",
|
||||
# Exfiltration endpoints observed across both Shai-Hulud waves.
|
||||
"filev2.getsession.org",
|
||||
"getsession.org/file/",
|
||||
# Campaign markers; the worm tarballs print this to stdout on run.
|
||||
"A Mini Shai-Hulud has Appeared",
|
||||
# Mini Shai-Hulud May-12 2026 wave.
|
||||
"git-tanstack.com",
|
||||
"transformers.pyz",
|
||||
"/tmp/transformers.pyz",
|
||||
"With Love TeamPCP",
|
||||
# Aikido (May-12 wave): payload SHA-256 hashes + Bun marker.
|
||||
"ab4fcadaec49c03278063dd269ea5eef82d24f2124a8e15d7b90f2fa8601266c",
|
||||
"2ec78d556d696e208927cc503d48e4b5eb56b31abc2870c2ed2e98d6be27fc96",
|
||||
"bun run tanstack_runner.js",
|
||||
"We've been online over 2 hours",
|
||||
)
|
||||
|
||||
# Hard pin-blocks for publicly confirmed malicious versions.
|
||||
# keep in sync with scripts/scan_npm_packages.py
|
||||
BLOCKED_NPM_VERSIONS: dict[str, set[str]] = {
|
||||
# GHSA-g7cv-rxg3-hmpx -- TanStack May-11 2026 (84 versions).
|
||||
"@tanstack/arktype-adapter": {"1.166.12", "1.166.15"},
|
||||
"@tanstack/eslint-plugin-router": {"1.161.9", "1.161.12"},
|
||||
"@tanstack/eslint-plugin-start": {"0.0.4", "0.0.7"},
|
||||
"@tanstack/history": {"1.161.9", "1.161.12"},
|
||||
"@tanstack/nitro-v2-vite-plugin": {"1.154.12", "1.154.15"},
|
||||
"@tanstack/react-router": {"1.169.5", "1.169.8"},
|
||||
"@tanstack/react-router-devtools": {"1.166.16", "1.166.19"},
|
||||
"@tanstack/react-router-ssr-query": {"1.166.15", "1.166.18"},
|
||||
"@tanstack/react-start": {"1.167.68", "1.167.71"},
|
||||
"@tanstack/react-start-client": {"1.166.51", "1.166.54"},
|
||||
"@tanstack/react-start-rsc": {"0.0.47", "0.0.50"},
|
||||
"@tanstack/react-start-server": {"1.166.55", "1.166.58"},
|
||||
"@tanstack/router-cli": {"1.166.46", "1.166.49"},
|
||||
"@tanstack/router-core": {"1.169.5", "1.169.8"},
|
||||
"@tanstack/router-devtools": {"1.166.16", "1.166.19"},
|
||||
"@tanstack/router-devtools-core": {"1.167.6", "1.167.9"},
|
||||
"@tanstack/router-generator": {"1.166.45", "1.166.48"},
|
||||
"@tanstack/router-plugin": {"1.167.38", "1.167.41"},
|
||||
"@tanstack/router-ssr-query-core": {"1.168.3", "1.168.6"},
|
||||
"@tanstack/router-utils": {"1.161.11", "1.161.14"},
|
||||
"@tanstack/router-vite-plugin": {"1.166.53", "1.166.56"},
|
||||
"@tanstack/solid-router": {"1.169.5", "1.169.8"},
|
||||
"@tanstack/solid-router-devtools": {"1.166.16", "1.166.19"},
|
||||
"@tanstack/solid-router-ssr-query": {"1.166.15", "1.166.18"},
|
||||
"@tanstack/solid-start": {"1.167.65", "1.167.68"},
|
||||
"@tanstack/solid-start-client": {"1.166.50", "1.166.53"},
|
||||
"@tanstack/solid-start-server": {"1.166.54", "1.166.57"},
|
||||
"@tanstack/start-client-core": {"1.168.5", "1.168.8"},
|
||||
"@tanstack/start-fn-stubs": {"1.161.9", "1.161.12"},
|
||||
"@tanstack/start-plugin-core": {"1.169.23", "1.169.26"},
|
||||
"@tanstack/start-server-core": {"1.167.33", "1.167.36"},
|
||||
"@tanstack/start-static-server-functions": {"1.166.44", "1.166.47"},
|
||||
"@tanstack/start-storage-context": {"1.166.38", "1.166.41"},
|
||||
"@tanstack/valibot-adapter": {"1.166.12", "1.166.15"},
|
||||
"@tanstack/virtual-file-routes": {"1.161.10", "1.161.13"},
|
||||
"@tanstack/vue-router": {"1.169.5", "1.169.8"},
|
||||
"@tanstack/vue-router-devtools": {"1.166.16", "1.166.19"},
|
||||
"@tanstack/vue-router-ssr-query": {"1.166.15", "1.166.18"},
|
||||
"@tanstack/vue-start": {"1.167.61", "1.167.64"},
|
||||
"@tanstack/vue-start-client": {"1.166.46", "1.166.49"},
|
||||
"@tanstack/vue-start-server": {"1.166.50", "1.166.53"},
|
||||
"@tanstack/zod-adapter": {"1.166.12", "1.166.15"},
|
||||
# Mini Shai-Hulud May-12 wave: OpenSearch JS client.
|
||||
"@opensearch-project/opensearch": {"3.5.3", "3.6.2", "3.7.0", "3.8.0"},
|
||||
# Mini Shai-Hulud May-12 wave: @squawk/* (22 packages, 5 versions each;
|
||||
# https://safedep.io/mass-npm-supply-chain-attack-tanstack-mistral/).
|
||||
"@squawk/airport-data": {"0.7.4", "0.7.5", "0.7.6", "0.7.7", "0.7.8"},
|
||||
"@squawk/airports": {"0.6.2", "0.6.3", "0.6.4", "0.6.5", "0.6.6"},
|
||||
"@squawk/airspace": {"0.8.1", "0.8.2", "0.8.3", "0.8.4", "0.8.5"},
|
||||
"@squawk/airspace-data": {"0.5.3", "0.5.4", "0.5.5", "0.5.6", "0.5.7"},
|
||||
"@squawk/airway-data": {"0.5.4", "0.5.5", "0.5.6", "0.5.7", "0.5.8"},
|
||||
"@squawk/airways": {"0.4.2", "0.4.3", "0.4.4", "0.4.5", "0.4.6"},
|
||||
"@squawk/fix-data": {"0.6.4", "0.6.5", "0.6.6", "0.6.7", "0.6.8"},
|
||||
"@squawk/fixes": {"0.3.2", "0.3.3", "0.3.4", "0.3.5", "0.3.6"},
|
||||
"@squawk/flight-math": {"0.5.4", "0.5.5", "0.5.6", "0.5.7", "0.5.8"},
|
||||
"@squawk/flightplan": {"0.5.2", "0.5.3", "0.5.4", "0.5.5", "0.5.6"},
|
||||
"@squawk/geo": {"0.4.4", "0.4.5", "0.4.6", "0.4.7", "0.4.8"},
|
||||
"@squawk/icao-registry": {"0.5.2", "0.5.3", "0.5.4", "0.5.5", "0.5.6"},
|
||||
"@squawk/icao-registry-data": {"0.8.4", "0.8.5", "0.8.6", "0.8.7", "0.8.8"},
|
||||
"@squawk/mcp": {"0.9.1", "0.9.2", "0.9.3", "0.9.4", "0.9.5"},
|
||||
"@squawk/navaid-data": {"0.6.4", "0.6.5", "0.6.6", "0.6.7", "0.6.8"},
|
||||
"@squawk/navaids": {"0.4.2", "0.4.3", "0.4.4", "0.4.5", "0.4.6"},
|
||||
"@squawk/notams": {"0.3.6", "0.3.7", "0.3.8", "0.3.9", "0.3.10"},
|
||||
"@squawk/procedure-data": {"0.7.3", "0.7.4", "0.7.5", "0.7.6", "0.7.7"},
|
||||
"@squawk/procedures": {"0.5.2", "0.5.3", "0.5.4", "0.5.5", "0.5.6"},
|
||||
"@squawk/types": {"0.8.1", "0.8.2", "0.8.3", "0.8.4", "0.8.5"},
|
||||
"@squawk/units": {"0.4.3", "0.4.4", "0.4.5", "0.4.6", "0.4.7"},
|
||||
"@squawk/weather": {"0.5.6", "0.5.7", "0.5.8", "0.5.9", "0.5.10"},
|
||||
# Mini Shai-Hulud May-12 wave: @uipath/* (64 packages, single version each;
|
||||
# https://www.aikido.dev/blog/mini-shai-hulud-is-back-tanstack-compromised).
|
||||
"@uipath/apollo-react": {"4.24.5"},
|
||||
"@uipath/apollo-wind": {"2.16.2"},
|
||||
"@uipath/cli": {"1.0.1"},
|
||||
"@uipath/rpa-tool": {"0.9.5"},
|
||||
"@uipath/apollo-core": {"5.9.2"},
|
||||
"@uipath/filesystem": {"1.0.1"},
|
||||
"@uipath/solutionpackager-tool-core": {"0.0.34"},
|
||||
"@uipath/solution-tool": {"1.0.1"},
|
||||
"@uipath/maestro-tool": {"1.0.1"},
|
||||
"@uipath/codedapp-tool": {"1.0.1"},
|
||||
"@uipath/agent-tool": {"1.0.1"},
|
||||
"@uipath/orchestrator-tool": {"1.0.1"},
|
||||
"@uipath/integrationservice-tool": {"1.0.2"},
|
||||
"@uipath/rpa-legacy-tool": {"1.0.1"},
|
||||
"@uipath/vertical-solutions-tool": {"1.0.1"},
|
||||
"@uipath/flow-tool": {"1.0.2"},
|
||||
"@uipath/codedagent-tool": {"1.0.1"},
|
||||
"@uipath/common": {"1.0.1"},
|
||||
"@uipath/resource-tool": {"1.0.1"},
|
||||
"@uipath/auth": {"1.0.1"},
|
||||
"@uipath/docsai-tool": {"1.0.1"},
|
||||
"@uipath/case-tool": {"1.0.1"},
|
||||
"@uipath/api-workflow-tool": {"1.0.1"},
|
||||
"@uipath/test-manager-tool": {"1.0.2"},
|
||||
"@uipath/robot": {"1.3.4"},
|
||||
"@uipath/traces-tool": {"1.0.1"},
|
||||
"@uipath/agent-sdk": {"1.0.2"},
|
||||
"@uipath/integrationservice-sdk": {"1.0.2"},
|
||||
"@uipath/maestro-sdk": {"1.0.1"},
|
||||
"@uipath/data-fabric-tool": {"1.0.2"},
|
||||
"@uipath/tasks-tool": {"1.0.1"},
|
||||
"@uipath/insights-tool": {"1.0.1"},
|
||||
"@uipath/insights-sdk": {"1.0.1"},
|
||||
"@uipath/uipath-python-bridge": {"1.0.1"},
|
||||
"@uipath/ap-chat": {"1.5.7"},
|
||||
"@uipath/project-packager": {"1.1.16"},
|
||||
"@uipath/packager-tool-case": {"0.0.9"},
|
||||
"@uipath/packager-tool-workflowcompiler-browser": {"0.0.34"},
|
||||
"@uipath/packager-tool-connector": {"0.0.19"},
|
||||
"@uipath/packager-tool-workflowcompiler": {"0.0.16"},
|
||||
"@uipath/packager-tool-webapp": {"1.0.6"},
|
||||
"@uipath/packager-tool-apiworkflow": {"0.0.19"},
|
||||
"@uipath/packager-tool-functions": {"0.1.1"},
|
||||
"@uipath/widget.sdk": {"1.2.3"},
|
||||
"@uipath/resources-tool": {"0.1.11"},
|
||||
"@uipath/agent.sdk": {"0.0.18"},
|
||||
"@uipath/codedagents-tool": {"0.1.12"},
|
||||
"@uipath/aops-policy-tool": {"0.3.1"},
|
||||
"@uipath/solution-packager": {"0.0.35"},
|
||||
"@uipath/packager-tool-bpmn": {"0.0.9"},
|
||||
"@uipath/packager-tool-flow": {"0.0.19"},
|
||||
"@uipath/telemetry": {"0.0.7"},
|
||||
"@uipath/tool-workflowcompiler": {"0.0.12"},
|
||||
"@uipath/vss": {"0.1.6"},
|
||||
"@uipath/solutionpackager-sdk": {"1.0.11"},
|
||||
"@uipath/ui-widgets-multi-file-upload": {"1.0.1"},
|
||||
"@uipath/access-policy-tool": {"0.3.1"},
|
||||
"@uipath/context-grounding-tool": {"0.1.1"},
|
||||
"@uipath/gov-tool": {"0.3.1"},
|
||||
"@uipath/admin-tool": {"0.1.1"},
|
||||
"@uipath/identity-tool": {"0.1.1"},
|
||||
"@uipath/llmgw-tool": {"1.0.1"},
|
||||
"@uipath/resourcecatalog-tool": {"0.1.1"},
|
||||
"@uipath/functions-tool": {"1.0.1"},
|
||||
"@uipath/access-policy-sdk": {"0.3.1"},
|
||||
"@uipath/platform-tool": {"1.0.1"},
|
||||
# Mini Shai-Hulud May-12 wave: @mistralai/* (npm) — separate from PyPI mistralai
|
||||
# (https://www.aikido.dev/blog/mini-shai-hulud-is-back-tanstack-compromised).
|
||||
"@mistralai/mistralai": {"2.2.2", "2.2.3", "2.2.4"},
|
||||
"@mistralai/mistralai-gcp": {"1.7.1", "1.7.2", "1.7.3"},
|
||||
"@mistralai/mistralai-azure": {"1.7.1", "1.7.2", "1.7.3"},
|
||||
# Mini Shai-Hulud May-12 wave: @tallyui/* (30 entries, 10 packages)
|
||||
# (Aikido enumeration).
|
||||
"@tallyui/components": {"1.0.1", "1.0.2", "1.0.3"},
|
||||
"@tallyui/connector-medusa": {"1.0.1", "1.0.2", "1.0.3"},
|
||||
"@tallyui/connector-shopify": {"1.0.1", "1.0.2", "1.0.3"},
|
||||
"@tallyui/connector-vendure": {"1.0.1", "1.0.2", "1.0.3"},
|
||||
"@tallyui/connector-woocommerce": {"1.0.1", "1.0.2", "1.0.3"},
|
||||
"@tallyui/core": {"0.2.1", "0.2.2", "0.2.3"},
|
||||
"@tallyui/database": {"1.0.1", "1.0.2", "1.0.3"},
|
||||
"@tallyui/pos": {"0.1.1", "0.1.2", "0.1.3"},
|
||||
"@tallyui/storage-sqlite": {"0.2.1", "0.2.2", "0.2.3"},
|
||||
"@tallyui/theme": {"0.2.1", "0.2.2", "0.2.3"},
|
||||
# Mini Shai-Hulud May-12 wave: @beproduct/nestjs-auth (18 versions)
|
||||
# (Aikido enumeration).
|
||||
"@beproduct/nestjs-auth": {
|
||||
"0.1.2",
|
||||
"0.1.3",
|
||||
"0.1.4",
|
||||
"0.1.5",
|
||||
"0.1.6",
|
||||
"0.1.7",
|
||||
"0.1.8",
|
||||
"0.1.9",
|
||||
"0.1.10",
|
||||
"0.1.11",
|
||||
"0.1.12",
|
||||
"0.1.13",
|
||||
"0.1.14",
|
||||
"0.1.15",
|
||||
"0.1.16",
|
||||
"0.1.17",
|
||||
"0.1.18",
|
||||
"0.1.19",
|
||||
},
|
||||
# Mini Shai-Hulud May-12 wave: @draftlab/* + @draftauth/*
|
||||
# (Aikido enumeration).
|
||||
"@draftauth/client": {"0.2.1", "0.2.2"},
|
||||
"@draftauth/core": {"0.13.1", "0.13.2"},
|
||||
"@draftlab/auth": {"0.24.1", "0.24.2"},
|
||||
"@draftlab/auth-router": {"0.5.1", "0.5.2"},
|
||||
"@draftlab/db": {"0.16.1"},
|
||||
# Mini Shai-Hulud May-12 wave: @taskflow-corp/cli + @tolka/cli
|
||||
# (Aikido enumeration).
|
||||
"@taskflow-corp/cli": {"0.1.24", "0.1.25", "0.1.26", "0.1.27", "0.1.28", "0.1.29"},
|
||||
"@tolka/cli": {"1.0.2", "1.0.3", "1.0.4", "1.0.5", "1.0.6"},
|
||||
# Mini Shai-Hulud May-12 wave: @ml-toolkit-ts/* + @mesadev/* + @dirigible-ai/sdk + @supersurkhet/*
|
||||
# (Aikido enumeration).
|
||||
"@dirigible-ai/sdk": {"0.6.2", "0.6.3"},
|
||||
"@mesadev/rest": {"0.28.3"},
|
||||
"@mesadev/saguaro": {"0.4.22"},
|
||||
"@mesadev/sdk": {"0.28.3"},
|
||||
"@ml-toolkit-ts/preprocessing": {"1.0.2", "1.0.3"},
|
||||
"@ml-toolkit-ts/xgboost": {"1.0.3", "1.0.4"},
|
||||
"@supersurkhet/cli": {"0.0.2", "0.0.3", "0.0.4", "0.0.5", "0.0.6", "0.0.7"},
|
||||
"@supersurkhet/sdk": {"0.0.2", "0.0.3", "0.0.4", "0.0.5", "0.0.6", "0.0.7"},
|
||||
# Mini Shai-Hulud May-12 wave: Unscoped packages (10 entries)
|
||||
# (Aikido enumeration).
|
||||
"safe-action": {"0.8.3", "0.8.4"},
|
||||
"ts-dna": {"3.0.1", "3.0.2", "3.0.3", "3.0.4"},
|
||||
"cross-stitch": {"1.1.3", "1.1.4", "1.1.5", "1.1.6"},
|
||||
"cmux-agent-mcp": {"0.1.3", "0.1.4", "0.1.5", "0.1.6", "0.1.7", "0.1.8"},
|
||||
"agentwork-cli": {"0.1.4", "0.1.5"},
|
||||
"git-branch-selector": {"1.3.3", "1.3.4", "1.3.5", "1.3.6", "1.3.7"},
|
||||
"wot-api": {"0.8.1", "0.8.2", "0.8.3", "0.8.4"},
|
||||
"git-git-git": {"1.0.8", "1.0.9", "1.0.10", "1.0.11", "1.0.12"},
|
||||
"nextmove-mcp": {"0.1.3", "0.1.4", "0.1.5", "0.1.6", "0.1.7"},
|
||||
"ml-toolkit-ts": {"1.0.4", "1.0.5"},
|
||||
# Cross-ecosystem Mini Shai-Hulud (Apr-30 wave): npm counterpart of
|
||||
# PyPI lightning 2.6.2/2.6.3. Same threat actor (TeamPCP) per Semgrep,
|
||||
# Aikido, OX Security, Resecurity. Safe version: 7.0.3 and earlier.
|
||||
"intercom-client": {"7.0.4"},
|
||||
}
|
||||
|
||||
CARGO_IOC_STRINGS: tuple[str, ...] = (
|
||||
# Empty by default; the `source` origin check catches the structural
|
||||
# pattern. Reserved for future cargo-side incidents.
|
||||
)
|
||||
|
||||
|
||||
# Allowed lockfile origins.
|
||||
NPM_REGISTRY_PREFIX = "https://registry.npmjs.org/"
|
||||
NPM_REGISTRY_PREFIXES_ALLOWED: tuple[str, ...] = (NPM_REGISTRY_PREFIX,)
|
||||
|
||||
CARGO_REGISTRY_SOURCE = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
|
||||
|
||||
# Cargo non-registry source allowlist: `(crate_name, exact_source_string)`.
|
||||
# Both must match verbatim; bumping the pinned SHA forces a re-review.
|
||||
# Studio's Tauri shell pulls `fix-path-env` from git because it is not
|
||||
# published to crates.io; commit c4c45d5 was reviewed when it landed.
|
||||
CARGO_SOURCE_ALLOWLIST: tuple[tuple[str, str], ...] = (
|
||||
(
|
||||
"fix-path-env",
|
||||
"git+https://github.com/tauri-apps/fix-path-env-rs#"
|
||||
"c4c45d503ea115a839aae718d02f79e7c7f0f673",
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class Finding:
|
||||
__slots__ = ("path", "package", "kind", "detail")
|
||||
|
||||
def __init__(self, path: str, package: str, kind: str, detail: str) -> None:
|
||||
self.path = path
|
||||
self.package = package
|
||||
self.kind = kind
|
||||
self.detail = detail
|
||||
|
||||
def __str__(self) -> str:
|
||||
return (
|
||||
f" [{self.kind}] {self.path}\n"
|
||||
f" package: {self.package}\n"
|
||||
f" detail: {self.detail}"
|
||||
)
|
||||
|
||||
|
||||
def _gha_escape(text: str) -> str:
|
||||
"""Escape a string for a GH Actions `::warning::`/`::error::` message.
|
||||
|
||||
GH Actions truncates at the first newline unless `\\n`/`\\r` are
|
||||
escaped as `%0A`/`%0D`. `%` must be replaced first to avoid
|
||||
double-encoding the subsequent escapes.
|
||||
"""
|
||||
return text.replace("%", "%25").replace("\r", "%0D").replace("\n", "%0A")
|
||||
|
||||
|
||||
def audit_npm_lockfile(path: Path) -> list[Finding]:
|
||||
findings: list[Finding] = []
|
||||
if not path.exists():
|
||||
# Missing lockfile is a config error, not a clean audit.
|
||||
findings.append(
|
||||
Finding(
|
||||
path = str(path),
|
||||
package = "<root>",
|
||||
kind = "missing-lockfile",
|
||||
detail = (
|
||||
"expected lockfile not found; refusing to silently "
|
||||
"report a clean audit for a path that was not scanned"
|
||||
),
|
||||
)
|
||||
)
|
||||
return findings
|
||||
|
||||
try:
|
||||
raw = path.read_text(encoding = "utf-8")
|
||||
except OSError as exc:
|
||||
# Surface as a finding instead of crashing CI with a traceback.
|
||||
findings.append(
|
||||
Finding(
|
||||
path = str(path),
|
||||
package = "<root>",
|
||||
kind = "unreadable-lockfile",
|
||||
detail = f"could not read file: {exc}",
|
||||
)
|
||||
)
|
||||
return findings
|
||||
try:
|
||||
lock = json.loads(raw)
|
||||
except json.JSONDecodeError as exc:
|
||||
findings.append(
|
||||
Finding(
|
||||
path = str(path),
|
||||
package = "<root>",
|
||||
kind = "malformed-lockfile",
|
||||
detail = f"could not parse as JSON: {exc}",
|
||||
)
|
||||
)
|
||||
return findings
|
||||
|
||||
lockfile_version = lock.get("lockfileVersion")
|
||||
if lockfile_version not in (2, 3):
|
||||
findings.append(
|
||||
Finding(
|
||||
path = str(path),
|
||||
package = "<root>",
|
||||
kind = "unsupported-lockfile-version",
|
||||
detail = (f"only lockfileVersion 2 or 3 audited; got {lockfile_version}"),
|
||||
)
|
||||
)
|
||||
|
||||
packages = lock.get("packages") or {}
|
||||
for key, entry in packages.items():
|
||||
# Empty key "" is the project root (no `resolved`); skip it.
|
||||
if key == "":
|
||||
continue
|
||||
if entry.get("link"):
|
||||
# Workspace symlink; no tarball to resolve.
|
||||
continue
|
||||
|
||||
resolved = entry.get("resolved")
|
||||
# Entries nested in another package's node_modules are bundled
|
||||
# fold-ins covered by the parent's integrity; treat as transparent.
|
||||
nested = key.count("/node_modules/") >= 1
|
||||
|
||||
# 1. resolved-URL origin.
|
||||
if resolved is None:
|
||||
if nested or entry.get("bundled"):
|
||||
# Bundled / fold-in entry; covered by parent integrity.
|
||||
pass
|
||||
elif entry.get("version"):
|
||||
# Top-level entry without a resolved URL is suspicious.
|
||||
findings.append(
|
||||
Finding(
|
||||
path = str(path),
|
||||
package = key,
|
||||
kind = "missing-resolved-url",
|
||||
detail = (
|
||||
f"version={entry['version']!r} but no `resolved` "
|
||||
"field; lockfile is incomplete"
|
||||
),
|
||||
)
|
||||
)
|
||||
else:
|
||||
if not any(resolved.startswith(p) for p in NPM_REGISTRY_PREFIXES_ALLOWED):
|
||||
findings.append(
|
||||
Finding(
|
||||
path = str(path),
|
||||
package = key,
|
||||
kind = "non-registry-resolved-url",
|
||||
detail = (
|
||||
f"resolved={resolved!r}; only "
|
||||
f"{NPM_REGISTRY_PREFIX} is permitted. Direct "
|
||||
"GitHub / git / file references are the "
|
||||
"Shai-Hulud injection vector."
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
# 2. integrity-hash presence.
|
||||
if resolved is not None and not entry.get("integrity"):
|
||||
findings.append(
|
||||
Finding(
|
||||
path = str(path),
|
||||
package = key,
|
||||
kind = "missing-integrity-hash",
|
||||
detail = (
|
||||
"no `integrity` field; npm cannot verify the "
|
||||
"tarball SHA against the registry-published hash"
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
# 3. Blocked malicious version list.
|
||||
nm_prefix = "node_modules/"
|
||||
pkg_name = key[len(nm_prefix) :] if key.startswith(nm_prefix) else key
|
||||
version = entry.get("version")
|
||||
blocked = BLOCKED_NPM_VERSIONS.get(pkg_name, set())
|
||||
if version and version in blocked:
|
||||
findings.append(
|
||||
Finding(
|
||||
path = str(path),
|
||||
package = key,
|
||||
kind = "blocked-known-malicious",
|
||||
detail = (f"{pkg_name}@{version} is on the BLOCKED_NPM_VERSIONS list"),
|
||||
)
|
||||
)
|
||||
|
||||
# 4. Known IOC strings: scan the raw body to catch fields the
|
||||
# structural pass doesn't enumerate (scripts, optional deps, etc.).
|
||||
for ioc in NPM_IOC_STRINGS:
|
||||
if ioc in raw:
|
||||
line_no = _first_line_containing(raw, ioc)
|
||||
findings.append(
|
||||
Finding(
|
||||
path = f"{path}:{line_no}" if line_no else str(path),
|
||||
package = "<ioc-match>",
|
||||
kind = "known-ioc-string",
|
||||
detail = (
|
||||
f"matched known IOC substring {ioc!r}; this is "
|
||||
"a public indicator of a recent supply-chain "
|
||||
"compromise. Refuse to install."
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
return findings
|
||||
|
||||
|
||||
def _first_line_containing(text: str, needle: str) -> int | None:
|
||||
for i, line in enumerate(text.splitlines(), start = 1):
|
||||
if needle in line:
|
||||
return i
|
||||
return None
|
||||
|
||||
|
||||
# Cargo.lock is TOML; parsed with stdlib tomllib (Python 3.11+).
|
||||
_PACKAGE_HEADER = re.compile(r"^\[\[package\]\]\s*$")
|
||||
|
||||
|
||||
def audit_cargo_lockfile(path: Path) -> list[Finding]:
|
||||
findings: list[Finding] = []
|
||||
if not path.exists():
|
||||
# See audit_npm_lockfile: missing lockfile is a finding.
|
||||
findings.append(
|
||||
Finding(
|
||||
path = str(path),
|
||||
package = "<root>",
|
||||
kind = "missing-lockfile",
|
||||
detail = (
|
||||
"expected lockfile not found; refusing to silently "
|
||||
"report a clean audit for a path that was not scanned"
|
||||
),
|
||||
)
|
||||
)
|
||||
return findings
|
||||
|
||||
try:
|
||||
raw = path.read_text(encoding = "utf-8")
|
||||
except OSError as exc:
|
||||
findings.append(
|
||||
Finding(
|
||||
path = str(path),
|
||||
package = "<root>",
|
||||
kind = "unreadable-lockfile",
|
||||
detail = f"could not read file: {exc}",
|
||||
)
|
||||
)
|
||||
return findings
|
||||
try:
|
||||
import tomllib # type: ignore[import-not-found]
|
||||
except ImportError:
|
||||
# Python <3.11; fall back to a tomli shim if importable.
|
||||
try:
|
||||
import tomli as tomllib # type: ignore[no-redef]
|
||||
except ImportError:
|
||||
findings.append(
|
||||
Finding(
|
||||
path = str(path),
|
||||
package = "<root>",
|
||||
kind = "missing-toml-parser",
|
||||
detail = (
|
||||
"Python 3.11+ tomllib or tomli is required to "
|
||||
"parse Cargo.lock; install tomli or upgrade "
|
||||
"Python before re-running this audit"
|
||||
),
|
||||
)
|
||||
)
|
||||
return findings
|
||||
|
||||
try:
|
||||
lock = tomllib.loads(raw)
|
||||
except Exception as exc:
|
||||
findings.append(
|
||||
Finding(
|
||||
path = str(path),
|
||||
package = "<root>",
|
||||
kind = "malformed-lockfile",
|
||||
detail = f"could not parse as TOML: {exc}",
|
||||
)
|
||||
)
|
||||
return findings
|
||||
|
||||
for entry in lock.get("package", []):
|
||||
name = entry.get("name") or "<unnamed>"
|
||||
version = entry.get("version") or "<unversioned>"
|
||||
source = entry.get("source")
|
||||
# Workspace-local crates have no `source` field; skip them.
|
||||
if source is None:
|
||||
continue
|
||||
if source != CARGO_REGISTRY_SOURCE:
|
||||
if (name, source) in CARGO_SOURCE_ALLOWLIST:
|
||||
# Pre-approved non-registry source pinned by SHA.
|
||||
pass
|
||||
else:
|
||||
findings.append(
|
||||
Finding(
|
||||
path = str(path),
|
||||
package = f"{name}@{version}",
|
||||
kind = "non-registry-cargo-source",
|
||||
detail = (
|
||||
f"source={source!r}; only "
|
||||
f"{CARGO_REGISTRY_SOURCE!r} is permitted "
|
||||
"by default, and no allowlist entry covers "
|
||||
"this crate. If the source is legitimate, "
|
||||
"add `(name, source)` to "
|
||||
"CARGO_SOURCE_ALLOWLIST after reviewing the "
|
||||
"pinned commit."
|
||||
),
|
||||
)
|
||||
)
|
||||
if not entry.get("checksum") and source == CARGO_REGISTRY_SOURCE:
|
||||
findings.append(
|
||||
Finding(
|
||||
path = str(path),
|
||||
package = f"{name}@{version}",
|
||||
kind = "missing-cargo-checksum",
|
||||
detail = (
|
||||
"registry crate without checksum; cargo cannot "
|
||||
"verify the downloaded source against the "
|
||||
"registry-published SHA"
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
for ioc in CARGO_IOC_STRINGS:
|
||||
if ioc in raw:
|
||||
line_no = _first_line_containing(raw, ioc)
|
||||
findings.append(
|
||||
Finding(
|
||||
path = f"{path}:{line_no}" if line_no else str(path),
|
||||
package = "<ioc-match>",
|
||||
kind = "known-ioc-string",
|
||||
detail = f"matched known IOC substring {ioc!r}",
|
||||
)
|
||||
)
|
||||
|
||||
return findings
|
||||
|
||||
|
||||
# Finding kinds split into BLOCKING vs ADVISORY for the default run mode.
|
||||
# Blocking = public attack indicators (known-malicious version, IOC
|
||||
# string). Advisory = structural anomalies that warn but don't block.
|
||||
# --strict makes every finding blocking.
|
||||
BLOCKING_KINDS: frozenset[str] = frozenset(
|
||||
{
|
||||
"blocked-known-malicious",
|
||||
"known-ioc-string",
|
||||
# A structurally broken lockfile might hide a real attack.
|
||||
"malformed-lockfile",
|
||||
"missing-lockfile",
|
||||
"unreadable-lockfile",
|
||||
"missing-toml-parser",
|
||||
}
|
||||
)
|
||||
|
||||
DEFAULT_NPM_LOCKFILES = (
|
||||
"studio/frontend/package-lock.json",
|
||||
"studio/backend/core/data_recipe/oxc-validator/package-lock.json",
|
||||
"studio/package-lock.json",
|
||||
)
|
||||
DEFAULT_CARGO_LOCKFILES = ("studio/src-tauri/Cargo.lock",)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description = "Pre-install lockfile supply-chain audit.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--root",
|
||||
default = str(REPO_ROOT),
|
||||
help = "Repo root (default: parent of this script).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--npm-lockfile",
|
||||
action = "append",
|
||||
default = None,
|
||||
help = (
|
||||
"Path to a package-lock.json (repeatable). "
|
||||
"Default: studio/frontend/package-lock.json, "
|
||||
"studio/backend/core/data_recipe/oxc-validator/package-lock.json, "
|
||||
"and studio/package-lock.json (Tauri CLI for desktop release)."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--cargo-lockfile",
|
||||
action = "append",
|
||||
default = None,
|
||||
help = ("Path to a Cargo.lock (repeatable). Default: studio/src-tauri/Cargo.lock."),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--strict",
|
||||
action = "store_true",
|
||||
help = (
|
||||
"Treat every finding as blocking (exit 1). "
|
||||
"Default mode only blocks on known-malicious versions, "
|
||||
"indicator-of-compromise strings, or structurally broken "
|
||||
"lockfiles; everything else is printed as an advisory "
|
||||
"warning with exit 0. CI should use the default; local "
|
||||
"audits aiming for zero noise can opt in via --strict."
|
||||
),
|
||||
)
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
# Require a real justification (>=5 chars, not a boolean-shaped token)
|
||||
# for the skip env var. An invalid value warns and falls through to
|
||||
# run the audit (fail-safe); a valid one warns and skips with rc=0.
|
||||
_skip_raw = os.environ.get("UNSLOTH_LOCKFILE_AUDIT_SKIP")
|
||||
if _skip_raw is not None:
|
||||
_skip = _skip_raw.strip()
|
||||
_invalid_tokens = {"", "1", "0", "true", "false", "yes", "no", "on", "off"}
|
||||
if _skip.lower() in _invalid_tokens or len(_skip) < 5:
|
||||
print(
|
||||
"::warning::Lockfile audit skip REQUIRES a justification "
|
||||
f"value (>=5 chars, not '{_skip_raw}'). Proceeding with "
|
||||
"audit. Use e.g. UNSLOTH_LOCKFILE_AUDIT_SKIP=ticket-1234.",
|
||||
file = sys.stderr,
|
||||
flush = True,
|
||||
)
|
||||
else:
|
||||
print(
|
||||
f"::warning::Lockfile audit skipped: reason='{_skip}'",
|
||||
file = sys.stderr,
|
||||
flush = True,
|
||||
)
|
||||
return 0
|
||||
|
||||
root = Path(args.root).resolve()
|
||||
# Explicit flags scope the scan; defaults apply only to no-args CI.
|
||||
_user_explicit = args.npm_lockfile is not None or args.cargo_lockfile is not None
|
||||
if _user_explicit:
|
||||
npm_paths = [root / p for p in (args.npm_lockfile or ())]
|
||||
cargo_paths = [root / p for p in (args.cargo_lockfile or ())]
|
||||
else:
|
||||
npm_paths = [root / p for p in DEFAULT_NPM_LOCKFILES]
|
||||
cargo_paths = [root / p for p in DEFAULT_CARGO_LOCKFILES]
|
||||
|
||||
all_findings: list[Finding] = []
|
||||
for p in npm_paths:
|
||||
print(f"[lockfile-audit] npm: {p}", flush = True)
|
||||
all_findings.extend(audit_npm_lockfile(p))
|
||||
for p in cargo_paths:
|
||||
print(f"[lockfile-audit] cargo: {p}", flush = True)
|
||||
all_findings.extend(audit_cargo_lockfile(p))
|
||||
|
||||
if not all_findings:
|
||||
print(
|
||||
f"[lockfile-audit] OK: 0 findings across "
|
||||
f"{len(npm_paths)} npm + {len(cargo_paths)} cargo lockfile(s)",
|
||||
flush = True,
|
||||
)
|
||||
return 0
|
||||
|
||||
# Split into blocking (known-malicious / IOC / structurally broken)
|
||||
# and advisory (everything else). Default mode prints advisories
|
||||
# without changing the exit code; --strict makes all blocking.
|
||||
blocking = [f for f in all_findings if f.kind in BLOCKING_KINDS]
|
||||
advisory = [f for f in all_findings if f.kind not in BLOCKING_KINDS]
|
||||
|
||||
if args.strict:
|
||||
blocking = list(all_findings)
|
||||
advisory = []
|
||||
|
||||
if advisory:
|
||||
print(
|
||||
f"\n[lockfile-audit] {len(advisory)} advisory finding(s) "
|
||||
"(non-blocking; pass --strict to fail the build on these):\n",
|
||||
file = sys.stderr,
|
||||
)
|
||||
for f in advisory:
|
||||
# GH Actions warning annotation; _gha_escape collapses the
|
||||
# multi-line Finding onto one line so it renders fully in the UI.
|
||||
print(f"::warning::{_gha_escape(str(f))}", file = sys.stderr)
|
||||
print(file = sys.stderr)
|
||||
|
||||
if not blocking:
|
||||
print(
|
||||
f"[lockfile-audit] OK: {len(advisory)} advisory finding(s), "
|
||||
"0 blocking. Run with --strict to escalate advisory findings.",
|
||||
flush = True,
|
||||
)
|
||||
return 0
|
||||
|
||||
print(
|
||||
f"\n[lockfile-audit] FAIL: {len(blocking)} blocking finding(s):\n",
|
||||
file = sys.stderr,
|
||||
)
|
||||
for f in blocking:
|
||||
# Same %-encoding rationale as the advisory branch above.
|
||||
print(f"::error::{_gha_escape(str(f))}", file = sys.stderr)
|
||||
print(file = sys.stderr)
|
||||
print(
|
||||
"[lockfile-audit] Refusing to proceed. Each blocking finding "
|
||||
"above is either a public indicator-of-compromise, a known-"
|
||||
"malicious pinned version, or a structurally broken lockfile. "
|
||||
"Investigate before running `npm ci` or `cargo fetch`.",
|
||||
file = sys.stderr,
|
||||
)
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,360 @@
|
||||
#!/usr/bin/env python
|
||||
# coding: utf-8
|
||||
"""
|
||||
Convert Jupyter notebooks (.ipynb) to executable Python scripts (.py).
|
||||
|
||||
Converts IPython magics to plain Python:
|
||||
!command -> subprocess.run('command', shell=True)
|
||||
%cd path -> os.chdir('path')
|
||||
%env VAR=value -> os.environ['VAR'] = 'value'
|
||||
%%file filename -> with open('filename', 'w') as f: f.write(...)
|
||||
%%capture -> (skipped)
|
||||
/content/... -> _WORKING_DIR + /...
|
||||
"""
|
||||
|
||||
import nbformat
|
||||
import re
|
||||
import shlex
|
||||
import sys
|
||||
import os
|
||||
import urllib.request
|
||||
import urllib.parse
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
# Allowlist of hosts for raw notebook fetches; anything else rejected before urlopen.
|
||||
_ALLOWED_NOTEBOOK_HOSTS = {
|
||||
"raw.githubusercontent.com",
|
||||
"gist.githubusercontent.com",
|
||||
}
|
||||
|
||||
|
||||
# Metacharacters that mean a `!cmd` line can't be a flat argv -> keep shell=True + review marker.
|
||||
_SHELL_METACHARS_RE = re.compile(r"\$\(|`|\|\||\||&&|>>?|<<?|\*|\?|;")
|
||||
|
||||
|
||||
def needs_fstring(cmd: str) -> bool:
|
||||
"""Check if command has Python variable interpolation like {var_name}."""
|
||||
pattern = r"(?<!\$)\{([a-zA-Z_][a-zA-Z0-9_]*)\}"
|
||||
return bool(re.search(pattern, cmd))
|
||||
|
||||
|
||||
def github_blob_to_raw(url: str) -> str:
|
||||
"""Convert GitHub blob URL to raw URL."""
|
||||
# github.com/user/repo/blob/branch/path -> raw.githubusercontent.com/user/repo/branch/path
|
||||
# Exact host match (not substring) so attacker.example.com/github.com/blob/... is not rewritten.
|
||||
parsed = urllib.parse.urlparse(url)
|
||||
if parsed.netloc != "github.com" or "/blob/" not in parsed.path:
|
||||
return url
|
||||
new_path = parsed.path.replace("/blob/", "/", 1)
|
||||
return urllib.parse.urlunparse(
|
||||
parsed._replace(netloc = "raw.githubusercontent.com", path = new_path)
|
||||
)
|
||||
|
||||
|
||||
def download_notebook(url: str) -> tuple[str, str]:
|
||||
"""Download notebook from URL. Returns (content, filename)."""
|
||||
raw_url = github_blob_to_raw(url)
|
||||
|
||||
parsed = urllib.parse.urlparse(raw_url)
|
||||
filename = os.path.basename(urllib.parse.unquote(parsed.path))
|
||||
|
||||
# Host allowlist: refuse to fetch from anything we don't recognise.
|
||||
host = parsed.hostname
|
||||
if host not in _ALLOWED_NOTEBOOK_HOSTS:
|
||||
raise ValueError(
|
||||
f"Refused notebook fetch from {host!r}: not in allowlist "
|
||||
f"{sorted(_ALLOWED_NOTEBOOK_HOSTS)}"
|
||||
)
|
||||
|
||||
print(f"Downloading {url}...")
|
||||
with urllib.request.urlopen(raw_url, timeout = 60) as response:
|
||||
content = response.read().decode("utf-8")
|
||||
|
||||
return content, filename
|
||||
|
||||
|
||||
def is_url(path: str) -> bool:
|
||||
"""Check if path is a URL."""
|
||||
return path.startswith("http://") or path.startswith("https://")
|
||||
|
||||
|
||||
def replace_colab_paths(source: str) -> str:
|
||||
"""Replace Colab-specific /content/ paths with current working directory."""
|
||||
source = source.replace('"/content/', 'f"{_WORKING_DIR}/')
|
||||
source = source.replace("'/content/", "f'{_WORKING_DIR}/")
|
||||
return source
|
||||
|
||||
|
||||
def _emit_shell_command(indent: str, full_cmd: str, *, allow_shell: bool) -> list[str]:
|
||||
"""Render a `!cmd` notebook line as Python statements.
|
||||
|
||||
f-string interpolation, shell metacharacters, or multiline force
|
||||
shell=True (shlex.split would drop operators), flagged with a
|
||||
WARNING comment. Otherwise emit shell=False argv form. allow_shell
|
||||
False makes shell=True emission a hard error.
|
||||
"""
|
||||
needs_f = needs_fstring(full_cmd)
|
||||
has_meta = bool(_SHELL_METACHARS_RE.search(full_cmd))
|
||||
multiline = "\n" in full_cmd
|
||||
|
||||
must_use_shell = needs_f or has_meta or multiline
|
||||
|
||||
if must_use_shell:
|
||||
if not allow_shell:
|
||||
raise ValueError(
|
||||
"Cell uses shell metacharacters / interpolation but "
|
||||
"--no-allow-shell was set; refusing to emit shell=True"
|
||||
)
|
||||
warn = f"{indent}# WARNING: shell=True; reviewed for hostile input"
|
||||
f_prefix = "f" if needs_f else ""
|
||||
if multiline:
|
||||
escaped_cmd = full_cmd.replace('"""', r"\"\"\"")
|
||||
if escaped_cmd.rstrip().endswith('"'):
|
||||
escaped_cmd = escaped_cmd.rstrip() + " "
|
||||
stmt = f'{indent}subprocess.run({f_prefix}"""{escaped_cmd}""", shell=True)'
|
||||
else:
|
||||
stmt = f"{indent}subprocess.run({f_prefix}{full_cmd!r}, shell=True)"
|
||||
return [warn, stmt]
|
||||
|
||||
return [f"{indent}subprocess.run(shlex.split({full_cmd!r}), shell=False)"]
|
||||
|
||||
|
||||
def convert_cell_to_python(source: str, *, allow_shell: bool = True) -> str:
|
||||
"""Convert a cell's IPython magics to plain Python."""
|
||||
lines = source.split("\n")
|
||||
result = []
|
||||
i = 0
|
||||
|
||||
while i < len(lines):
|
||||
line = lines[i]
|
||||
stripped = line.strip()
|
||||
indent = line[: len(line) - len(line.lstrip())]
|
||||
|
||||
if stripped.startswith("%%capture"):
|
||||
i += 1
|
||||
continue
|
||||
|
||||
if stripped.startswith("%%file "):
|
||||
filename = stripped[7:].strip()
|
||||
file_lines = []
|
||||
i += 1
|
||||
while i < len(lines):
|
||||
file_lines.append(lines[i])
|
||||
i += 1
|
||||
file_content = "\n".join(file_lines)
|
||||
file_content = file_content.replace('"""', r"\"\"\"")
|
||||
result.append(f'{indent}with open({filename!r}, "w") as _f:')
|
||||
result.append(f'{indent} _f.write("""{file_content}""")')
|
||||
continue
|
||||
|
||||
if stripped.startswith("!"):
|
||||
cmd_lines = [stripped[1:]]
|
||||
while cmd_lines[-1].rstrip().endswith("\\") and i + 1 < len(lines):
|
||||
i += 1
|
||||
cmd_lines.append(lines[i].strip())
|
||||
full_cmd = "\n".join(cmd_lines)
|
||||
|
||||
result.extend(_emit_shell_command(indent, full_cmd, allow_shell = allow_shell))
|
||||
|
||||
# %cd path -> os.chdir(path)
|
||||
elif stripped.startswith("%cd "):
|
||||
path = stripped[4:].strip()
|
||||
result.append(f"{indent}os.chdir({path!r})")
|
||||
|
||||
# %env VAR=value
|
||||
elif stripped.startswith("%env ") and "=" in stripped:
|
||||
match = re.match(r"%env\s+(\w+)=(.+)", stripped)
|
||||
if match:
|
||||
var, val = match.groups()
|
||||
result.append(f"{indent}os.environ[{var!r}] = {val!r}")
|
||||
|
||||
# %env VAR
|
||||
elif stripped.startswith("%env "):
|
||||
var = stripped[5:].strip()
|
||||
result.append(f"{indent}os.environ.get({var!r})")
|
||||
|
||||
# %pwd
|
||||
elif stripped == "%pwd":
|
||||
result.append(f"{indent}os.getcwd()")
|
||||
|
||||
else:
|
||||
result.append(line)
|
||||
|
||||
i += 1
|
||||
|
||||
return "\n".join(result)
|
||||
|
||||
|
||||
def convert_notebook(
|
||||
notebook_content: str,
|
||||
source_name: str = "notebook",
|
||||
*,
|
||||
allow_shell: bool = True,
|
||||
) -> str:
|
||||
"""Convert notebook JSON content to Python script."""
|
||||
# Parse notebook
|
||||
if isinstance(notebook_content, str):
|
||||
notebook = nbformat.reads(notebook_content, as_version = 4)
|
||||
else:
|
||||
notebook = notebook_content
|
||||
|
||||
lines = [
|
||||
"#!/usr/bin/env python",
|
||||
"# coding: utf-8",
|
||||
f"# Converted from: {source_name}",
|
||||
"",
|
||||
"import shlex",
|
||||
"import subprocess",
|
||||
"import os",
|
||||
"import sys",
|
||||
"import re",
|
||||
"",
|
||||
"# Capture original packages before any installs",
|
||||
"_original_packages = subprocess.run(",
|
||||
" [sys.executable, '-m', 'pip', 'freeze'],",
|
||||
" capture_output=True, text=True",
|
||||
").stdout",
|
||||
"",
|
||||
"# Working directory (replaces Colab's /content/)",
|
||||
"_WORKING_DIR = os.getcwd()",
|
||||
"",
|
||||
]
|
||||
|
||||
for cell in notebook.cells:
|
||||
source = cell.source.strip()
|
||||
if not source:
|
||||
continue
|
||||
|
||||
if cell.cell_type == "code":
|
||||
converted = convert_cell_to_python(source, allow_shell = allow_shell)
|
||||
converted = replace_colab_paths(converted)
|
||||
lines.append(converted)
|
||||
lines.append("")
|
||||
|
||||
elif cell.cell_type == "markdown":
|
||||
for line in source.split("\n"):
|
||||
lines.append(f"# {line}")
|
||||
lines.append("")
|
||||
|
||||
# Add package restoration at the end
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
"# Restore original packages (install one by one, skip failures)",
|
||||
"for _pkg in _original_packages.strip().split('\\n'):",
|
||||
" if _pkg:",
|
||||
" subprocess.run([sys.executable, '-m', 'pip', 'install', _pkg, '-q'],",
|
||||
" stderr=subprocess.DEVNULL)",
|
||||
"",
|
||||
]
|
||||
)
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def convert_notebook_to_script(
|
||||
source: str,
|
||||
output_dir: str | None = None,
|
||||
*,
|
||||
allow_shell: bool = True,
|
||||
):
|
||||
"""
|
||||
Convert a notebook to Python script.
|
||||
|
||||
Args:
|
||||
source: Local file path or URL to notebook
|
||||
output_dir: Output directory (optional, defaults to current directory)
|
||||
allow_shell: When False, refuse to emit `shell=True` for any
|
||||
`!cmd` cell that uses metacharacters / interpolation.
|
||||
"""
|
||||
if is_url(source):
|
||||
content, filename = download_notebook(source)
|
||||
source_name = source
|
||||
else:
|
||||
filename = os.path.basename(source)
|
||||
with open(source, "r", encoding = "utf-8") as f:
|
||||
content = f.read()
|
||||
source_name = source
|
||||
|
||||
output_filename = filename.replace(".ipynb", ".py")
|
||||
output_filename = output_filename.replace("(", "").replace(")", "").replace("-", "_")
|
||||
|
||||
if output_dir:
|
||||
output_path = os.path.join(output_dir, output_filename)
|
||||
else:
|
||||
output_path = output_filename
|
||||
|
||||
script = convert_notebook(content, source_name, allow_shell = allow_shell)
|
||||
|
||||
with open(output_path, "w", encoding = "utf-8") as f:
|
||||
f.write(script)
|
||||
|
||||
print(f"Converted {source} -> {output_path}")
|
||||
return output_path
|
||||
|
||||
|
||||
def main():
|
||||
import argparse
|
||||
|
||||
class Formatter(argparse.ArgumentDefaultsHelpFormatter, argparse.RawDescriptionHelpFormatter):
|
||||
pass
|
||||
|
||||
parser = argparse.ArgumentParser(
|
||||
description = __doc__,
|
||||
formatter_class = Formatter,
|
||||
epilog = """
|
||||
Examples:
|
||||
python notebook_to_python.py notebook.ipynb
|
||||
python notebook_to_python.py -o scripts/ notebook1.ipynb notebook2.ipynb
|
||||
python notebook_to_python.py --output ./converted https://github.com/user/repo/blob/main/notebook.ipynb
|
||||
python notebook_to_python.py https://github.com/unslothai/notebooks/blob/main/nb/Oute_TTS_(1B).ipynb
|
||||
""",
|
||||
)
|
||||
parser.add_argument("notebooks", nargs = "+", help = "Notebook files or URLs to convert.")
|
||||
parser.add_argument("-o", "--output", dest = "output_dir", default = ".", help = "Output directory.")
|
||||
# Default True for backwards compat; pass --no-allow-shell for untrusted notebooks.
|
||||
parser.add_argument(
|
||||
"--allow-shell",
|
||||
dest = "allow_shell",
|
||||
action = "store_true",
|
||||
default = True,
|
||||
help = "Allow emitting subprocess.run(..., shell=True) for cells "
|
||||
"that use shell metacharacters or interpolation (default).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-allow-shell",
|
||||
dest = "allow_shell",
|
||||
action = "store_false",
|
||||
help = "Refuse to emit shell=True; cells with metacharacters error out.",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
os.makedirs(args.output_dir, exist_ok = True)
|
||||
|
||||
# Track per-notebook failures; continue the loop and exit 1 if any failed.
|
||||
failures: list[tuple[str, str]] = []
|
||||
ok = 0
|
||||
total = len(args.notebooks)
|
||||
for source in args.notebooks:
|
||||
try:
|
||||
convert_notebook_to_script(
|
||||
source,
|
||||
output_dir = args.output_dir if args.output_dir != "." else None,
|
||||
allow_shell = args.allow_shell,
|
||||
)
|
||||
ok += 1
|
||||
except Exception as e:
|
||||
print(f"ERROR converting {source}: {e}")
|
||||
failures.append((source, f"{type(e).__name__}: {e}"))
|
||||
|
||||
print(
|
||||
f"converted {ok}/{total}, {len(failures)} failed",
|
||||
file = sys.stderr if failures else sys.stdout,
|
||||
)
|
||||
sys.exit(1 if failures else 0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
File diff suppressed because it is too large
Load Diff
Executable
+40
@@ -0,0 +1,40 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run a pre-pass (normalize def-signature magic commas + collapse short
|
||||
multi-line asserts), then `ruff format`, then the kwarg-spacing / import /
|
||||
string-merge post-pass."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
files = [arg for arg in argv if Path(arg).exists()]
|
||||
if not files:
|
||||
return 0
|
||||
|
||||
spacing_script = HERE / "enforce_kwargs_spacing.py"
|
||||
|
||||
# Pre-ruff: normalize def-signature magic commas and strip the magic comma
|
||||
# from short multi-line asserts so ruff wraps/joins accordingly.
|
||||
pre_cmd = [sys.executable, str(spacing_script), "--pre", *files]
|
||||
pre_proc = subprocess.run(pre_cmd)
|
||||
if pre_proc.returncode != 0:
|
||||
return pre_proc.returncode
|
||||
|
||||
ruff_cmd = [sys.executable, "-m", "ruff", "format", *files]
|
||||
ruff_proc = subprocess.run(ruff_cmd)
|
||||
if ruff_proc.returncode != 0:
|
||||
return ruff_proc.returncode
|
||||
|
||||
spacing_cmd = [sys.executable, str(spacing_script), *files]
|
||||
spacing_proc = subprocess.run(spacing_cmd)
|
||||
return spacing_proc.returncode
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main(sys.argv[1:]))
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"_comment": "scan_npm_packages.py allowlist. Each entry is a HIGH/CRITICAL finding manually judged benign. Matched on (package, package-relative path, pattern, evidence hash); a new payload under an already-listed package/path/pattern reopens instead of riding the entry. severity is for review only. Regenerate with --write-baseline AFTER reviewing every line. EMPTY by design: a full scan of studio/frontend/package-lock.json (915 packages) produced 0 findings, so nothing needs suppressing and the CI gate can run enforcing (SCAN_ENFORCE=1) as-is. If a future dependency adds a reviewed-benign HIGH/CRITICAL, add it here rather than weakening a pattern.",
|
||||
"version": 3,
|
||||
"entries": []
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because one or more lines are too long
@@ -0,0 +1,281 @@
|
||||
#!/usr/bin/env python3
|
||||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
"""Stamp and verify display-only Studio release metadata for builds."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
import tempfile
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def _atomic_write_text(
|
||||
path: Path,
|
||||
data: str,
|
||||
encoding: str = "utf-8",
|
||||
) -> None:
|
||||
"""Atomic ``Path.write_text``: a crash mid-write leaves the prior file
|
||||
intact, so the build never reads a partial ``_studio_release_build.py``."""
|
||||
dirpath = str(path.parent) or "."
|
||||
path.parent.mkdir(parents = True, exist_ok = True)
|
||||
fd, tmp_path = tempfile.mkstemp(prefix = ".stamp_studio.", dir = dirpath)
|
||||
try:
|
||||
with os.fdopen(fd, "w", encoding = encoding) as handle:
|
||||
handle.write(data)
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
os.replace(tmp_path, path)
|
||||
except Exception:
|
||||
try:
|
||||
os.unlink(tmp_path)
|
||||
except OSError:
|
||||
pass
|
||||
raise
|
||||
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
BUILD_INFO_PATH = REPO_ROOT / "studio" / "backend" / "utils" / "_studio_release_build.py"
|
||||
BUILD_INFO_SUFFIX = "studio/backend/utils/_studio_release_build.py"
|
||||
VERSION_RE = re.compile(r"^v\d+\.\d+\.\d+(?:-[0-9A-Za-z.][0-9A-Za-z.-]*)?$")
|
||||
GIT_DESCRIBE_SUFFIX_RE = re.compile(r"-\d+-g[0-9A-Fa-f]+(?:-dirty)?$")
|
||||
MAX_VERSION_LENGTH = 64
|
||||
PLACEHOLDER = """# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
\"\"\"Build-stamped Studio release metadata.
|
||||
|
||||
Release builds may rewrite this module in the build workspace before creating
|
||||
Python artifacts. Keep the committed value neutral so source checkouts do not
|
||||
accidentally report a stale release tag.
|
||||
\"\"\"
|
||||
|
||||
STUDIO_RELEASE_VERSION = None
|
||||
"""
|
||||
|
||||
|
||||
def is_valid_version(value: object) -> bool:
|
||||
if not isinstance(value, str):
|
||||
return False
|
||||
version = value.strip()
|
||||
if not version or len(version) > MAX_VERSION_LENGTH:
|
||||
return False
|
||||
if version.endswith("-dirty") or GIT_DESCRIBE_SUFFIX_RE.search(version):
|
||||
return False
|
||||
return VERSION_RE.fullmatch(version) is not None
|
||||
|
||||
|
||||
def _exact_git_tag() -> str | None:
|
||||
try:
|
||||
result = subprocess.run(
|
||||
[
|
||||
"git",
|
||||
"describe",
|
||||
"--tags",
|
||||
"--exact-match",
|
||||
"--match",
|
||||
"v[0-9]*",
|
||||
"HEAD",
|
||||
],
|
||||
cwd = REPO_ROOT,
|
||||
check = False,
|
||||
stdout = subprocess.PIPE,
|
||||
stderr = subprocess.DEVNULL,
|
||||
text = True,
|
||||
timeout = 2.0,
|
||||
)
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
return None
|
||||
if result.returncode != 0:
|
||||
return None
|
||||
tag = result.stdout.strip()
|
||||
return tag if is_valid_version(tag) else None
|
||||
|
||||
|
||||
def _git_worktree_is_dirty() -> bool:
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["git", "status", "--porcelain"],
|
||||
cwd = REPO_ROOT,
|
||||
check = False,
|
||||
stdout = subprocess.PIPE,
|
||||
stderr = subprocess.DEVNULL,
|
||||
text = True,
|
||||
timeout = 2.0,
|
||||
)
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
return True
|
||||
if result.returncode != 0:
|
||||
return True
|
||||
return bool(result.stdout.strip())
|
||||
|
||||
|
||||
def _github_tag() -> str | None:
|
||||
if os.environ.get("GITHUB_REF_TYPE") != "tag":
|
||||
return None
|
||||
github_ref = os.environ.get("GITHUB_REF_NAME", "").strip()
|
||||
return github_ref or None
|
||||
|
||||
|
||||
def resolve_version() -> tuple[str | None, str]:
|
||||
env_version = os.environ.get("UNSLOTH_STUDIO_RELEASE_VERSION", "").strip()
|
||||
if env_version:
|
||||
return (env_version, "UNSLOTH_STUDIO_RELEASE_VERSION")
|
||||
|
||||
github_ref = _github_tag()
|
||||
if github_ref:
|
||||
return (github_ref, "GITHUB_REF_NAME")
|
||||
|
||||
git_tag = _exact_git_tag()
|
||||
if git_tag:
|
||||
return (git_tag, "exact git tag")
|
||||
|
||||
return (None, "none")
|
||||
|
||||
|
||||
def build_info_source(version: str | None) -> str:
|
||||
literal = repr(version) if version is not None else "None"
|
||||
return f'''# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
"""Build-stamped Studio release metadata."""
|
||||
|
||||
STUDIO_RELEASE_VERSION = {literal}
|
||||
'''
|
||||
|
||||
|
||||
def _env_version_conflicts(version: str) -> list[tuple[str, str]]:
|
||||
conflicts: list[tuple[str, str]] = []
|
||||
github_ref = _github_tag()
|
||||
if github_ref and is_valid_version(github_ref) and github_ref != version:
|
||||
conflicts.append(("GITHUB_REF_NAME", github_ref))
|
||||
|
||||
git_tag = _exact_git_tag()
|
||||
if git_tag and git_tag != version:
|
||||
conflicts.append(("exact git tag", git_tag))
|
||||
|
||||
return conflicts
|
||||
|
||||
|
||||
def stamp(require_release: bool) -> int:
|
||||
version, source = resolve_version()
|
||||
if version is not None and not is_valid_version(version):
|
||||
print(
|
||||
f"Invalid Studio release version from {source}: {version!r}",
|
||||
file = sys.stderr,
|
||||
)
|
||||
return 2
|
||||
|
||||
if version is not None and source == "UNSLOTH_STUDIO_RELEASE_VERSION":
|
||||
conflicts = _env_version_conflicts(version)
|
||||
if conflicts:
|
||||
details = ", ".join(f"{name}={value!r}" for name, value in conflicts)
|
||||
print(
|
||||
"UNSLOTH_STUDIO_RELEASE_VERSION does not match available "
|
||||
f"release tag metadata: {details}",
|
||||
file = sys.stderr,
|
||||
)
|
||||
return 2
|
||||
|
||||
if require_release and source == "exact git tag" and _git_worktree_is_dirty():
|
||||
print(
|
||||
"Refusing to publish from a dirty exact-tag checkout. Set "
|
||||
"UNSLOTH_STUDIO_RELEASE_VERSION explicitly from release automation "
|
||||
"or publish from a clean tag checkout.",
|
||||
file = sys.stderr,
|
||||
)
|
||||
return 2
|
||||
|
||||
if version is None:
|
||||
if require_release:
|
||||
print(
|
||||
"No Studio release version available. Set "
|
||||
"UNSLOTH_STUDIO_RELEASE_VERSION, build from a GitHub tag, "
|
||||
"or run from an exact local Studio release tag.",
|
||||
file = sys.stderr,
|
||||
)
|
||||
return 2
|
||||
_atomic_write_text(BUILD_INFO_PATH, PLACEHOLDER, encoding = "utf-8")
|
||||
print("dev")
|
||||
return 0
|
||||
|
||||
_atomic_write_text(BUILD_INFO_PATH, build_info_source(version), encoding = "utf-8")
|
||||
print(f"Stamping Studio release version {version} from {source}", file = sys.stderr)
|
||||
print(version)
|
||||
return 0
|
||||
|
||||
|
||||
def _read_wheel_member(path: Path) -> str | None:
|
||||
with zipfile.ZipFile(path) as archive:
|
||||
for name in archive.namelist():
|
||||
if name.endswith(BUILD_INFO_SUFFIX):
|
||||
return archive.read(name).decode("utf-8")
|
||||
return None
|
||||
|
||||
|
||||
def _read_sdist_member(path: Path) -> str | None:
|
||||
with tarfile.open(path) as archive:
|
||||
for member in archive.getmembers():
|
||||
if member.name.endswith(BUILD_INFO_SUFFIX):
|
||||
extracted = archive.extractfile(member)
|
||||
if extracted is None:
|
||||
return None
|
||||
return extracted.read().decode("utf-8")
|
||||
return None
|
||||
|
||||
|
||||
def verify_dist(expected: str, dist_dir: Path) -> int:
|
||||
if not is_valid_version(expected):
|
||||
print(f"Invalid expected Studio release version: {expected!r}", file = sys.stderr)
|
||||
return 2
|
||||
|
||||
artifacts = list(dist_dir.glob("*.whl")) + list(dist_dir.glob("*.tar.gz"))
|
||||
if not artifacts:
|
||||
print(f"No wheel or sdist artifacts found in {dist_dir}", file = sys.stderr)
|
||||
return 2
|
||||
|
||||
expected_line = f"STUDIO_RELEASE_VERSION = {expected!r}"
|
||||
failures: list[str] = []
|
||||
for artifact in artifacts:
|
||||
if artifact.suffix == ".whl":
|
||||
content = _read_wheel_member(artifact)
|
||||
else:
|
||||
content = _read_sdist_member(artifact)
|
||||
if content is None:
|
||||
failures.append(f"{artifact.name}: missing {BUILD_INFO_SUFFIX}")
|
||||
elif expected_line not in content:
|
||||
failures.append(f"{artifact.name}: Studio release version mismatch")
|
||||
|
||||
if failures:
|
||||
for failure in failures:
|
||||
print(failure, file = sys.stderr)
|
||||
return 2
|
||||
|
||||
print(f"Verified Studio release version {expected} in {len(artifacts)} artifact(s)")
|
||||
return 0
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description = __doc__)
|
||||
parser.add_argument("--require-release", action = "store_true")
|
||||
parser.add_argument("--verify-dist", type = Path)
|
||||
parser.add_argument("--expected")
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.verify_dist is not None:
|
||||
if not args.expected:
|
||||
parser.error("--verify-dist requires --expected")
|
||||
return verify_dist(args.expected, args.verify_dist)
|
||||
|
||||
return stamp(require_release = args.require_release)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,143 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Keep `allowScripts` pins in studio/frontend/package.json in sync with
|
||||
package-lock.json.
|
||||
|
||||
`npm approve-scripts` writes version-pinned entries ("pkg@1.2.3": true).
|
||||
A dependency bump strands the pin, so the approval (or denial) silently
|
||||
stops matching and the package's install scripts fall back to
|
||||
"unreviewed". This tool re-pins existing entries to the versions the
|
||||
lockfile actually resolves; it never adds or removes entries, so
|
||||
approving a brand-new script-bearing package stays a human decision.
|
||||
|
||||
Usage:
|
||||
python scripts/sync_allow_scripts_pins.py --check # CI: exit 1 on drift
|
||||
python scripts/sync_allow_scripts_pins.py --fix # rewrite package.json
|
||||
|
||||
Pinned keys follow npm's allowScripts grammar: "name@1.2.3" or
|
||||
"name@1.2.3 || 1.2.4". Bare names (no version) match every version and
|
||||
are left alone. Entries whose range is not an exact-version disjunction
|
||||
(wildcards, tags) are left alone too.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
DEFAULT_DIR = REPO_ROOT / "studio" / "frontend"
|
||||
|
||||
EXACT_VERSION_RE = re.compile(r"^\d+\.\d+\.\d+(?:[-+][0-9A-Za-z.+-]+)?$")
|
||||
|
||||
|
||||
def split_spec(key: str) -> tuple[str, str | None]:
|
||||
"""'@scope/name@1.2.3' -> ('@scope/name', '1.2.3'); bare names -> (key, None)."""
|
||||
if key.startswith("@"):
|
||||
rest = key[1:]
|
||||
if "@" not in rest:
|
||||
return key, None
|
||||
name, rng = rest.split("@", 1)
|
||||
return "@" + name, rng
|
||||
if "@" not in key:
|
||||
return key, None
|
||||
name, rng = key.split("@", 1)
|
||||
return name, rng
|
||||
|
||||
|
||||
def is_exact_disjunction(rng: str) -> bool:
|
||||
parts = [p.strip() for p in rng.split("||")]
|
||||
return all(EXACT_VERSION_RE.match(p) for p in parts) and bool(parts)
|
||||
|
||||
|
||||
def version_sort_key(version: str) -> tuple:
|
||||
release = version.split("-", 1)[0].split("+", 1)[0]
|
||||
return tuple(int(x) for x in release.split(".")), version
|
||||
|
||||
|
||||
def script_versions_from_lock(lock: dict) -> dict[str, list[str]]:
|
||||
"""Map package name -> sorted versions that carry install scripts."""
|
||||
out: dict[str, set[str]] = {}
|
||||
for path, meta in (lock.get("packages") or {}).items():
|
||||
if not path or not meta.get("hasInstallScript"):
|
||||
continue
|
||||
name = path.rsplit("node_modules/", 1)[-1]
|
||||
version = meta.get("version")
|
||||
if name and version:
|
||||
out.setdefault(name, set()).add(version)
|
||||
return {n: sorted(vs, key = version_sort_key) for n, vs in out.items()}
|
||||
|
||||
|
||||
def desired_key(name: str, versions: list[str]) -> str:
|
||||
return f"{name}@{' || '.join(versions)}"
|
||||
|
||||
|
||||
def compute_renames(policy: dict, lock_versions: dict[str, list[str]]) -> dict[str, str]:
|
||||
renames: dict[str, str] = {}
|
||||
for key in policy:
|
||||
name, rng = split_spec(key)
|
||||
if rng is None or not is_exact_disjunction(rng):
|
||||
continue # bare name or non-exact spec: matches by name, never stale
|
||||
versions = lock_versions.get(name)
|
||||
if not versions:
|
||||
continue # package gone or script-free now: stale pin is inert
|
||||
want = desired_key(name, versions)
|
||||
if key != want:
|
||||
renames[key] = want
|
||||
return renames
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
ap = argparse.ArgumentParser(description = __doc__)
|
||||
mode = ap.add_mutually_exclusive_group(required = True)
|
||||
mode.add_argument("--check", action = "store_true", help = "exit 1 if pins are stale")
|
||||
mode.add_argument("--fix", action = "store_true", help = "rewrite package.json in place")
|
||||
ap.add_argument(
|
||||
"--dir",
|
||||
type = Path,
|
||||
default = DEFAULT_DIR,
|
||||
help = "directory holding package.json + package-lock.json",
|
||||
)
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
pkg_path = args.dir / "package.json"
|
||||
lock_path = args.dir / "package-lock.json"
|
||||
if not pkg_path.exists() or not lock_path.exists():
|
||||
print(f"sync-allow-scripts: nothing to do ({args.dir} has no package.json + lockfile)")
|
||||
return 0
|
||||
|
||||
pkg = json.loads(pkg_path.read_text(encoding = "utf-8"))
|
||||
policy = pkg.get("allowScripts")
|
||||
if not isinstance(policy, dict) or not policy:
|
||||
print("sync-allow-scripts: no allowScripts policy in package.json, nothing to do")
|
||||
return 0
|
||||
|
||||
lock = json.loads(lock_path.read_text(encoding = "utf-8"))
|
||||
renames = compute_renames(policy, script_versions_from_lock(lock))
|
||||
|
||||
if not renames:
|
||||
print(f"sync-allow-scripts: {len(policy)} allowScripts entries in sync with the lockfile")
|
||||
return 0
|
||||
|
||||
for old, new in renames.items():
|
||||
print(f' stale pin: "{old}" -> "{new}"')
|
||||
|
||||
if args.check:
|
||||
print(
|
||||
"sync-allow-scripts: pins are stale; run "
|
||||
"`python scripts/sync_allow_scripts_pins.py --fix` and commit the result"
|
||||
)
|
||||
return 1
|
||||
|
||||
pkg["allowScripts"] = {renames.get(k, k): v for k, v in policy.items()}
|
||||
pkg_path.write_text(json.dumps(pkg, indent = 2, ensure_ascii = False) + "\n", encoding = "utf-8")
|
||||
print(
|
||||
f"sync-allow-scripts: re-pinned {len(renames)} entr{'y' if len(renames) == 1 else 'ies'} in {pkg_path}"
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,496 @@
|
||||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
#
|
||||
# Unsloth Studio uninstaller for Windows PowerShell.
|
||||
# Stops running servers and removes install dir, launcher data, CLI shim,
|
||||
# desktop and Start Menu shortcuts, the user PATH entry, and the PathBackup
|
||||
# registry key. Honors custom roots set via UNSLOTH_STUDIO_HOME / STUDIO_HOME
|
||||
# at install time (read back from share\studio.conf).
|
||||
#
|
||||
# Usage: irm https://raw.githubusercontent.com/unslothai/unsloth/main/scripts/uninstall.ps1 | iex
|
||||
# Local: Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass; .\scripts\uninstall.ps1
|
||||
|
||||
function Uninstall-UnslothStudio {
|
||||
$ErrorActionPreference = "Continue"
|
||||
|
||||
function _Step { param([string]$Msg) Write-Host $Msg }
|
||||
function _Substep { param([string]$Msg, [string]$Color = "Gray") Write-Host " $Msg" -ForegroundColor $Color }
|
||||
|
||||
# Remove a file/dir/symlink if present. Idempotent; retries since a just-killed
|
||||
# process can briefly hold a handle (Windows refuses the delete until released).
|
||||
function _RemovePath {
|
||||
param([string]$Path)
|
||||
if ([string]::IsNullOrWhiteSpace($Path)) { return }
|
||||
if (-not (Test-Path -LiteralPath $Path)) { return }
|
||||
for ($attempt = 1; $attempt -le 4; $attempt++) {
|
||||
try {
|
||||
Remove-Item -LiteralPath $Path -Recurse -Force -ErrorAction Stop
|
||||
} catch {
|
||||
if ($attempt -lt 4) { Start-Sleep -Milliseconds 700; continue }
|
||||
_Substep "could not remove: $Path ($($_.Exception.Message))" "Yellow"
|
||||
return
|
||||
}
|
||||
# Remove-Item -Recurse can report success yet leave a transiently-locked
|
||||
# child (e.g. unsloth.ico in Explorer's icon cache); verify + retry so we
|
||||
# never falsely claim "removed" or orphan the dir.
|
||||
if (-not (Test-Path -LiteralPath $Path)) {
|
||||
_Substep "removed: $Path" "Green"
|
||||
return
|
||||
}
|
||||
if ($attempt -lt 4) { Start-Sleep -Milliseconds 700; continue }
|
||||
_Substep "still present (files held open): $Path" "Yellow"
|
||||
}
|
||||
}
|
||||
|
||||
# Remove the shared data dir, but keep unsloth.ico if a WSL shortcut still points
|
||||
# at it (else that shortcut blanks); uninstall.sh drops it when WSL is removed.
|
||||
function _RemoveDataDirKeepingWslIcon {
|
||||
param(
|
||||
[string]$DataDir,
|
||||
# WSL-shortcut search dirs; default Start Menu + Desktop, overridable for tests.
|
||||
[string[]]$ShortcutDirs = $null
|
||||
)
|
||||
if ([string]::IsNullOrWhiteSpace($DataDir)) { return }
|
||||
if (-not (Test-Path -LiteralPath $DataDir)) { return }
|
||||
# $null = not passed (use defaults); test $null not truthiness so an explicit
|
||||
# @() is honored (-not @() is $true).
|
||||
if ($null -eq $ShortcutDirs) {
|
||||
# Guard $env:APPDATA: it can be unset in service/CI Windows contexts, where
|
||||
# an unguarded Join-Path emits a noisy parameter-binding error.
|
||||
$ShortcutDirs = @()
|
||||
if (-not [string]::IsNullOrWhiteSpace($env:APPDATA)) {
|
||||
$ShortcutDirs += Join-Path $env:APPDATA "Microsoft\Windows\Start Menu\Programs"
|
||||
}
|
||||
try {
|
||||
$desktop = [Environment]::GetFolderPath("Desktop")
|
||||
if (-not [string]::IsNullOrWhiteSpace($desktop)) { $ShortcutDirs += $desktop }
|
||||
} catch {}
|
||||
}
|
||||
$wslShortcuts = @()
|
||||
foreach ($d in $ShortcutDirs) {
|
||||
if ($d -and (Test-Path -LiteralPath $d)) {
|
||||
$wslShortcuts += Get-ChildItem -LiteralPath $d -Filter "Unsloth Studio (WSL*.lnk" -ErrorAction SilentlyContinue
|
||||
}
|
||||
}
|
||||
if (@($wslShortcuts).Count -eq 0) {
|
||||
_RemovePath $DataDir
|
||||
return
|
||||
}
|
||||
# A WSL shortcut survives: drop everything except its shared icon.
|
||||
_Substep "keeping $(Join-Path $DataDir 'unsloth.ico') for the WSL shortcut" "Gray"
|
||||
Get-ChildItem -LiteralPath $DataDir -Force -ErrorAction SilentlyContinue | ForEach-Object {
|
||||
if ($_.Name -ne "unsloth.ico") { _RemovePath $_.FullName }
|
||||
}
|
||||
}
|
||||
|
||||
# A path is a Studio-owned root iff one of install.ps1's sentinels exists:
|
||||
# <root>\share\studio.conf, <root>\unsloth_studio\.unsloth-studio-owned,
|
||||
# or <root>\bin\unsloth.exe.
|
||||
function _IsStudioRoot {
|
||||
param([string]$Path)
|
||||
if ([string]::IsNullOrWhiteSpace($Path)) { return $false }
|
||||
if (Test-Path -LiteralPath (Join-Path $Path "share\studio.conf") -PathType Leaf) { return $true }
|
||||
if (Test-Path -LiteralPath (Join-Path $Path "unsloth_studio\.unsloth-studio-owned") -PathType Leaf) { return $true }
|
||||
if (Test-Path -LiteralPath (Join-Path $Path "bin\unsloth.exe") -PathType Leaf) { return $true }
|
||||
return $false
|
||||
}
|
||||
|
||||
# Hard deny list. Refuse to recursively delete drive roots, USERPROFILE
|
||||
# itself, parent of USERPROFILE, or system directories.
|
||||
function _IsUnsafeRoot {
|
||||
param([string]$Path)
|
||||
if ([string]::IsNullOrWhiteSpace($Path)) { return $true }
|
||||
$norm = $null
|
||||
try { $norm = [System.IO.Path]::GetFullPath($Path).TrimEnd('\','/') } catch { return $true }
|
||||
if ([string]::IsNullOrWhiteSpace($norm)) { return $true }
|
||||
# Drive root, e.g. C:\
|
||||
if ($norm -match '^[A-Za-z]:[\\/]?$') { return $true }
|
||||
$userProfile = $env:USERPROFILE
|
||||
if ($userProfile) {
|
||||
$userProfile = $userProfile.TrimEnd('\','/')
|
||||
if ($norm -ieq $userProfile) { return $true }
|
||||
try {
|
||||
$parent = Split-Path -LiteralPath $userProfile -Parent
|
||||
if ($parent -and ($norm -ieq $parent.TrimEnd('\','/'))) { return $true }
|
||||
} catch { }
|
||||
}
|
||||
$systemRoots = @(
|
||||
$env:SystemRoot, $env:windir, $env:ProgramFiles, ${env:ProgramFiles(x86)},
|
||||
$env:ProgramData, $env:APPDATA, $env:LOCALAPPDATA
|
||||
)
|
||||
foreach ($s in $systemRoots) {
|
||||
if (-not [string]::IsNullOrWhiteSpace($s)) {
|
||||
$s2 = $s.TrimEnd('\','/')
|
||||
if ($norm -ieq $s2) { return $true }
|
||||
}
|
||||
}
|
||||
return $false
|
||||
}
|
||||
|
||||
# Parse UNSLOTH_EXE='<path>' out of a share\studio.conf and return the
|
||||
# implied install root (three dirnames up from the venv exe).
|
||||
function _RootFromConf {
|
||||
param([string]$ConfFile)
|
||||
if (-not (Test-Path -LiteralPath $ConfFile -PathType Leaf)) { return $null }
|
||||
$line = Get-Content -LiteralPath $ConfFile -ErrorAction SilentlyContinue |
|
||||
Where-Object { $_ -match "^UNSLOTH_EXE\s*=" } | Select-Object -First 1
|
||||
if (-not $line) { return $null }
|
||||
# Tolerate ' value ' single-quoted with '' -> ' apostrophe escape.
|
||||
if ($line -match "^UNSLOTH_EXE\s*=\s*'(.*)'\s*$") {
|
||||
$exe = $Matches[1] -replace "''", "'"
|
||||
try {
|
||||
$bin = Split-Path -LiteralPath $exe -Parent
|
||||
$studio = Split-Path -LiteralPath $bin -Parent
|
||||
$root = Split-Path -LiteralPath $studio -Parent
|
||||
if ($root) { return $root }
|
||||
} catch { }
|
||||
}
|
||||
return $null
|
||||
}
|
||||
|
||||
# Expand a leading ~ or ~/ ~\ to $env:USERPROFILE so env-mode roots
|
||||
# written with the tilde shape install.ps1 supports (lines 152-154) are
|
||||
# found here too.
|
||||
function _ExpandTilde {
|
||||
param([string]$Path)
|
||||
if ([string]::IsNullOrWhiteSpace($Path)) { return $Path }
|
||||
$p = $Path.Trim()
|
||||
if ($p -eq '~') { return $env:USERPROFILE }
|
||||
if ($p.StartsWith('~/') -or $p.StartsWith('~\')) {
|
||||
if ($env:USERPROFILE) {
|
||||
return (Join-Path $env:USERPROFILE $p.Substring(2).TrimStart('/','\'))
|
||||
}
|
||||
}
|
||||
return $p
|
||||
}
|
||||
|
||||
# Discover non-default Studio roots from env vars + studio.conf files.
|
||||
# Mirrors install.ps1's precedence: UNSLOTH_STUDIO_HOME wins, STUDIO_HOME
|
||||
# is ignored when both are set, so uninstalling install A doesn't also
|
||||
# delete install B if the user has a stale STUDIO_HOME pointing at B.
|
||||
function _CustomStudioRoots {
|
||||
$seen = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::OrdinalIgnoreCase)
|
||||
$defaultRoot = $null
|
||||
if ($env:USERPROFILE) {
|
||||
$defaultRoot = (Join-Path $env:USERPROFILE ".unsloth\studio")
|
||||
}
|
||||
|
||||
$emit = {
|
||||
param($Path)
|
||||
if ([string]::IsNullOrWhiteSpace($Path)) { return }
|
||||
$expanded = _ExpandTilde $Path
|
||||
$norm = $null
|
||||
try { $norm = [System.IO.Path]::GetFullPath($expanded).TrimEnd('\','/') } catch { return }
|
||||
if (-not $norm) { return }
|
||||
if ($defaultRoot -and ($norm -ieq $defaultRoot.TrimEnd('\','/'))) { return }
|
||||
if ($seen.Add($norm)) { Write-Output $norm }
|
||||
}
|
||||
|
||||
$envRoot = $null
|
||||
if ($env:UNSLOTH_STUDIO_HOME) {
|
||||
$envRoot = $env:UNSLOTH_STUDIO_HOME
|
||||
} elseif ($env:STUDIO_HOME) {
|
||||
$envRoot = $env:STUDIO_HOME
|
||||
}
|
||||
if ($envRoot) {
|
||||
$expandedEnv = _ExpandTilde $envRoot
|
||||
& $emit $expandedEnv
|
||||
$confRoot = _RootFromConf (Join-Path $expandedEnv "share\studio.conf")
|
||||
if ($confRoot) { & $emit $confRoot }
|
||||
}
|
||||
# Default-mode conf at LOCALAPPDATA\Unsloth Studio.
|
||||
if ($env:LOCALAPPDATA) {
|
||||
$confRoot = _RootFromConf (Join-Path $env:LOCALAPPDATA "Unsloth Studio\studio.conf")
|
||||
if ($confRoot) { & $emit $confRoot }
|
||||
}
|
||||
}
|
||||
|
||||
# Return $true iff the PID's image path lives under one of $KnownRoots.
|
||||
# Prevents killing an unrelated process that happens to listen on a stale
|
||||
# Studio port.
|
||||
function _PidUnderKnownRoot {
|
||||
param([int]$Pid_, [string[]]$KnownRoots)
|
||||
if (-not $KnownRoots -or $KnownRoots.Count -eq 0) { return $false }
|
||||
try {
|
||||
$proc = Get-CimInstance Win32_Process -Filter "ProcessId=$Pid_" -ErrorAction SilentlyContinue
|
||||
if (-not $proc) { return $false }
|
||||
$exe = $proc.ExecutablePath
|
||||
if (-not $exe) { return $false }
|
||||
foreach ($r in $KnownRoots) {
|
||||
if ($r -and ($exe -ilike "$r\*")) { return $true }
|
||||
}
|
||||
} catch { }
|
||||
return $false
|
||||
}
|
||||
|
||||
# Stop a Studio backend whose port is recorded in <DataDir>\studio.port.
|
||||
# Only kills if the listening PID's exe path is under a known Studio root.
|
||||
function _StopByPortFile {
|
||||
param([string]$PortFile, [string[]]$KnownRoots)
|
||||
if (-not (Test-Path -LiteralPath $PortFile -PathType Leaf)) { return }
|
||||
$port = Get-Content -LiteralPath $PortFile -ErrorAction SilentlyContinue | Select-Object -First 1
|
||||
if ($port) { $port = $port.Trim() }
|
||||
if (-not ($port -match '^[0-9]+$')) {
|
||||
Remove-Item -LiteralPath $PortFile -Force -ErrorAction SilentlyContinue
|
||||
return
|
||||
}
|
||||
try {
|
||||
$conns = Get-NetTCPConnection -State Listen -LocalPort ([int]$port) -ErrorAction SilentlyContinue
|
||||
foreach ($c in $conns) {
|
||||
if (-not (_PidUnderKnownRoot -Pid_ ([int]$c.OwningProcess) -KnownRoots $KnownRoots)) { continue }
|
||||
try {
|
||||
Stop-Process -Id $c.OwningProcess -Force -ErrorAction SilentlyContinue
|
||||
} catch { }
|
||||
}
|
||||
} catch {
|
||||
# netstat fallback for older PowerShell. Require LISTENING state so
|
||||
# we never kill a process whose remote endpoint just happens to be
|
||||
# the cached port (browser -> :443 etc.).
|
||||
try {
|
||||
$lines = & netstat.exe -ano 2>$null |
|
||||
Select-String -Pattern "LISTENING" |
|
||||
Select-String -Pattern ":$port\s"
|
||||
foreach ($l in $lines) {
|
||||
$parts = ($l.ToString() -split '\s+') | Where-Object { $_ }
|
||||
$pid_ = $parts[-1]
|
||||
if ($pid_ -match '^\d+$') {
|
||||
if (-not (_PidUnderKnownRoot -Pid_ ([int]$pid_) -KnownRoots $KnownRoots)) { continue }
|
||||
try { Stop-Process -Id ([int]$pid_) -Force -ErrorAction SilentlyContinue } catch { }
|
||||
}
|
||||
}
|
||||
} catch { }
|
||||
}
|
||||
Remove-Item -LiteralPath $PortFile -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
|
||||
# Stop processes whose ExecutablePath lives under an unsloth_studio venv.
|
||||
# Anchoring on the venv path avoids matching unrelated python.exe / studio.exe.
|
||||
function _StopStudioProcesses {
|
||||
param([string[]]$KnownRoots)
|
||||
try {
|
||||
$procs = Get-CimInstance Win32_Process -ErrorAction SilentlyContinue |
|
||||
Where-Object {
|
||||
$_.ExecutablePath -and ($_.ExecutablePath -match '\\unsloth_studio\\.*\\(unsloth|python|studio)\.exe$') -and
|
||||
$_.CommandLine -and ($_.CommandLine -match 'studio')
|
||||
}
|
||||
foreach ($p in $procs) {
|
||||
# Optional scope: only kill if the exe is under a known root.
|
||||
if ($KnownRoots) {
|
||||
$match = $false
|
||||
foreach ($r in $KnownRoots) {
|
||||
if ($p.ExecutablePath -and ($p.ExecutablePath -ilike "$r\*")) { $match = $true; break }
|
||||
}
|
||||
if (-not $match) { continue }
|
||||
}
|
||||
try {
|
||||
Stop-Process -Id $p.ProcessId -Force -ErrorAction SilentlyContinue
|
||||
} catch { }
|
||||
}
|
||||
} catch { }
|
||||
}
|
||||
|
||||
# Stop processes that would block deleting the paths we remove. Unlike
|
||||
# _StopStudioProcesses (venv exe only), this also catches llama-server/llama-cli,
|
||||
# the unsloth.exe shim, and orphaned mp workers under SYSTEM python holding a
|
||||
# venv DLL (an open DLL handle blocks the dir delete) -- found by scanning each
|
||||
# candidate's loaded modules, not just its image path.
|
||||
function _StopProcessesLockingRoots {
|
||||
param([string[]]$Roots)
|
||||
$clean = @($Roots | Where-Object { $_ } | ForEach-Object { $_.TrimEnd('\','/') })
|
||||
if ($clean.Count -eq 0) { return }
|
||||
$underRoot = {
|
||||
param($p)
|
||||
if (-not $p) { return $false }
|
||||
foreach ($r in $clean) { if ($p -ieq $r -or $p -ilike "$r\*") { return $true } }
|
||||
return $false
|
||||
}
|
||||
# 1. Image path under a target root (venv python, shim, llama-server).
|
||||
try {
|
||||
foreach ($proc in (Get-CimInstance Win32_Process -ErrorAction SilentlyContinue)) {
|
||||
if ((& $underRoot $proc.ExecutablePath)) {
|
||||
try { Stop-Process -Id $proc.ProcessId -Force -ErrorAction SilentlyContinue } catch { }
|
||||
}
|
||||
}
|
||||
} catch { }
|
||||
# 2. A loaded module under a target root (orphaned mp-fork python holding a
|
||||
# venv DLL). Scoped to names that load our DLLs to keep the scan fast.
|
||||
try {
|
||||
$cands = Get-Process -Name python, pythonw, unsloth, llama-server, llama-cli -ErrorAction SilentlyContinue
|
||||
foreach ($proc in $cands) {
|
||||
$hit = $false
|
||||
try {
|
||||
foreach ($m in $proc.Modules) { if ((& $underRoot $m.FileName)) { $hit = $true; break } }
|
||||
} catch { } # access denied enumerating modules -> skip
|
||||
if ($hit) { try { Stop-Process -Id $proc.Id -Force -ErrorAction SilentlyContinue } catch { } }
|
||||
}
|
||||
} catch { }
|
||||
}
|
||||
|
||||
# Default install root + default data dir.
|
||||
$defaultStudioHome = if ($env:USERPROFILE) { Join-Path $env:USERPROFILE ".unsloth\studio" } else { $null }
|
||||
$defaultDataDir = if ($env:LOCALAPPDATA) { Join-Path $env:LOCALAPPDATA "Unsloth Studio" } else { $null }
|
||||
# Default-mode ~/.unsloth holds a SHARED llama.cpp build + .cache that are
|
||||
# siblings of studio (not under it), so deleting <studio> misses them -- handle
|
||||
# explicitly. No-op in env/custom mode (nested under the custom root, removed
|
||||
# with it). A user-set UNSLOTH_LLAMA_CPP_PATH is left alone.
|
||||
$defaultUnslothHome = if ($env:USERPROFILE) { Join-Path $env:USERPROFILE ".unsloth" } else { $null }
|
||||
$defaultLlamaCpp = if ($defaultUnslothHome) { Join-Path $defaultUnslothHome "llama.cpp" } else { $null }
|
||||
$defaultCache = if ($defaultUnslothHome) { Join-Path $defaultUnslothHome ".cache" } else { $null }
|
||||
# Isolated Node.js runtime (install_node_prebuilt.py), a sibling of studio in
|
||||
# default mode. No-op in env/custom mode (nested under the custom root) and absent.
|
||||
$defaultNode = if ($defaultUnslothHome) { Join-Path $defaultUnslothHome "node" } else { $null }
|
||||
# llama.cpp atomic-install staging root (install_llama_prebuilt.py .staging,
|
||||
# sibling of the install dir). Usually pruned after activate, but an interrupted
|
||||
# build can leave a "<name>.staging-XXXX" tree; removing it lets the empty-dir
|
||||
# cleanup of ~/.unsloth below succeed. No-op in env/custom mode and when absent.
|
||||
$defaultStaging = if ($defaultUnslothHome) { Join-Path $defaultUnslothHome ".staging" } else { $null }
|
||||
|
||||
# Build known-root list FIRST so the port-file kill can verify ownership.
|
||||
$customRoots = @(_CustomStudioRoots)
|
||||
$knownRoots = @()
|
||||
if ($defaultStudioHome) { $knownRoots += $defaultStudioHome }
|
||||
$knownRoots += $customRoots
|
||||
|
||||
# ── Stop running servers ──
|
||||
_Step "Stopping any running Unsloth Studio servers..."
|
||||
if ($defaultDataDir) {
|
||||
_StopByPortFile -PortFile (Join-Path $defaultDataDir "studio.port") -KnownRoots $knownRoots
|
||||
}
|
||||
foreach ($r in $customRoots) {
|
||||
_StopByPortFile -PortFile (Join-Path $r "share\studio.port") -KnownRoots $knownRoots
|
||||
}
|
||||
_StopStudioProcesses -KnownRoots $knownRoots
|
||||
# Also stop anything holding a handle on the exact paths we delete (llama-server,
|
||||
# the CLI shim, an mp-fork python with a venv DLL) so the dir delete isn't refused.
|
||||
_StopProcessesLockingRoots -Roots (@($knownRoots) + @($defaultDataDir, $defaultLlamaCpp, $defaultCache, $defaultNode))
|
||||
|
||||
# ── Remove custom-root install trees ──
|
||||
_Step "Removing data and install directories..."
|
||||
foreach ($r in $customRoots) {
|
||||
if (_IsUnsafeRoot $r) {
|
||||
_Substep "refusing to remove unsafe path: $r" "Yellow"
|
||||
continue
|
||||
}
|
||||
if (-not (_IsStudioRoot $r)) {
|
||||
_Substep "refusing to remove non-Studio path: $r" "Yellow"
|
||||
continue
|
||||
}
|
||||
_RemovePath $r
|
||||
}
|
||||
# Default install dir (always at %USERPROFILE%\.unsloth\studio when present).
|
||||
if ($defaultStudioHome) { _RemovePath $defaultStudioHome }
|
||||
# Default data dir.
|
||||
if ($defaultDataDir) { _RemoveDataDirKeepingWslIcon $defaultDataDir }
|
||||
# Default-mode shared llama.cpp build + cache (siblings of studio under
|
||||
# ~/.unsloth). No-op in env/custom mode and when absent.
|
||||
if ($defaultLlamaCpp) { _RemovePath $defaultLlamaCpp }
|
||||
if ($defaultCache) { _RemovePath $defaultCache }
|
||||
# Isolated Node.js runtime (sibling of studio under ~/.unsloth). No-op in env/
|
||||
# custom mode (nested under the custom root, removed with it) and when absent.
|
||||
if ($defaultNode) { _RemovePath $defaultNode }
|
||||
if ($defaultStaging) { _RemovePath $defaultStaging }
|
||||
# llama.cpp install lock (serializes the shared build); a stray lock keeps
|
||||
# ~/.unsloth from being pruned below. No-op in env/custom mode and when absent.
|
||||
if ($defaultUnslothHome) { _RemovePath (Join-Path $defaultUnslothHome ".llama.cpp.install.lock") }
|
||||
# Drop ~/.unsloth itself, but ONLY if now empty -- never nuke unrelated content.
|
||||
if ($defaultUnslothHome -and (Test-Path -LiteralPath $defaultUnslothHome) -and
|
||||
-not (Get-ChildItem -LiteralPath $defaultUnslothHome -Force -ErrorAction SilentlyContinue)) {
|
||||
_RemovePath $defaultUnslothHome
|
||||
}
|
||||
|
||||
# ── Remove desktop and Start Menu shortcuts ──
|
||||
_Step "Removing desktop and Start Menu shortcuts..."
|
||||
try {
|
||||
$desktop = [Environment]::GetFolderPath("Desktop")
|
||||
if ($desktop) { _RemovePath (Join-Path $desktop "Unsloth Studio.lnk") }
|
||||
} catch { }
|
||||
if ($env:APPDATA) {
|
||||
_RemovePath (Join-Path $env:APPDATA "Microsoft\Windows\Start Menu\Programs\Unsloth Studio.lnk")
|
||||
}
|
||||
# Invalidate the Win11 Start Menu tile cache so the removed shortcut's tile
|
||||
# disappears promptly instead of lingering stale (mirrors install.ps1's
|
||||
# New-StudioShortcuts). Preserves start2.bin (the pin layout).
|
||||
try {
|
||||
$smehTemp = Join-Path $env:LOCALAPPDATA "Packages\Microsoft.Windows.StartMenuExperienceHost_cw5n1h2txyewy\TempState"
|
||||
if (Test-Path -LiteralPath $smehTemp) {
|
||||
Get-ChildItem -LiteralPath $smehTemp -Filter "TileCache_*" -ErrorAction SilentlyContinue |
|
||||
Remove-Item -Force -ErrorAction SilentlyContinue
|
||||
Remove-Item -LiteralPath (Join-Path $smehTemp "StartUnifiedTileModelCache.dat") -Force -ErrorAction SilentlyContinue
|
||||
Stop-Process -Name StartMenuExperienceHost -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
} catch { }
|
||||
|
||||
# Re-sweep: the first pass may have left unsloth.ico locked by Explorer/SMEH for
|
||||
# the native shortcut; that handle is now freed. (A surviving WSL shortcut still
|
||||
# keeps the icon -- see the helper.)
|
||||
if ($defaultDataDir -and (Test-Path -LiteralPath $defaultDataDir)) { _RemoveDataDirKeepingWslIcon $defaultDataDir }
|
||||
|
||||
# ── Clean user PATH and registry backup ──
|
||||
_Step "Cleaning user PATH and registry..."
|
||||
try {
|
||||
$regKey = [Microsoft.Win32.Registry]::CurrentUser.OpenSubKey('Environment', $true)
|
||||
if ($regKey) {
|
||||
try {
|
||||
$rawPath = $regKey.GetValue('Path', '', [Microsoft.Win32.RegistryValueOptions]::DoNotExpandEnvironmentNames)
|
||||
if ($rawPath) {
|
||||
$entries = $rawPath -split ';'
|
||||
$kept = New-Object System.Collections.ArrayList
|
||||
$removedAny = $false
|
||||
# Only remove PATH entries that live inside a Studio root we
|
||||
# actually own (default or env-mode). A literal substring
|
||||
# match on `unsloth_studio` would clobber unrelated user
|
||||
# virtualenvs that happen to share the name.
|
||||
foreach ($e in $entries) {
|
||||
if ([string]::IsNullOrWhiteSpace($e)) { continue }
|
||||
$expanded = [Environment]::ExpandEnvironmentVariables($e).TrimEnd('\','/')
|
||||
$isStudio = $false
|
||||
foreach ($r in $knownRoots) {
|
||||
if (-not $r) { continue }
|
||||
$rNorm = $r.TrimEnd('\','/')
|
||||
if ($expanded -ieq $rNorm -or $expanded -ilike "$rNorm\*") {
|
||||
$isStudio = $true; break
|
||||
}
|
||||
}
|
||||
if ($isStudio) {
|
||||
_Substep "removed PATH entry: $e" "Green"
|
||||
$removedAny = $true
|
||||
continue
|
||||
}
|
||||
[void]$kept.Add($e)
|
||||
}
|
||||
if ($removedAny) {
|
||||
$newPath = ($kept -join ';')
|
||||
$regKey.SetValue('Path', $newPath, [Microsoft.Win32.RegistryValueKind]::ExpandString)
|
||||
try {
|
||||
$d = "UnslothPathRefresh_" + ([guid]::NewGuid().ToString('N').Substring(0, 8))
|
||||
[Environment]::SetEnvironmentVariable($d, '1', 'User')
|
||||
[Environment]::SetEnvironmentVariable($d, [NullString]::Value, 'User')
|
||||
} catch { }
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
$regKey.Close()
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
_Substep "could not update user PATH: $($_.Exception.Message)" "Yellow"
|
||||
}
|
||||
# Remove HKCU\Software\Unsloth (PathBackup lives here; install.ps1 owns it).
|
||||
try {
|
||||
Remove-Item -LiteralPath 'HKCU:\Software\Unsloth' -Recurse -Force -ErrorAction SilentlyContinue
|
||||
} catch { }
|
||||
|
||||
Write-Host ""
|
||||
Write-Host "Unsloth Studio uninstalled."
|
||||
Write-Host "Note: Hugging Face model cache at %USERPROFILE%\.cache\huggingface was left in place."
|
||||
Write-Host "Remove it manually with 'Remove-Item -Recurse -Force `"$env:USERPROFILE\.cache\huggingface\hub`"' if desired."
|
||||
if (-not $env:UNSLOTH_STUDIO_HOME -and -not $env:STUDIO_HOME) {
|
||||
Write-Host ""
|
||||
Write-Host "If you installed Unsloth Studio with UNSLOTH_STUDIO_HOME or STUDIO_HOME"
|
||||
Write-Host "pointing at a custom directory, re-run this script with the same variable"
|
||||
Write-Host "set to also remove that install tree, e.g.:"
|
||||
Write-Host " `$env:UNSLOTH_STUDIO_HOME = 'C:\your\path'; irm https://raw.githubusercontent.com/unslothai/unsloth/main/scripts/uninstall.ps1 | iex"
|
||||
}
|
||||
}
|
||||
|
||||
Uninstall-UnslothStudio @args
|
||||
Executable
+427
@@ -0,0 +1,427 @@
|
||||
#!/usr/bin/env sh
|
||||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
#
|
||||
# Unsloth Studio uninstaller (macOS / Linux / WSL).
|
||||
# Stops running servers and removes install dir, launcher data,
|
||||
# CLI shim, desktop shortcut, .app bundle, and Launch Services entry.
|
||||
# Honors custom roots set via UNSLOTH_STUDIO_HOME / STUDIO_HOME at
|
||||
# install time (read back from studio.conf).
|
||||
#
|
||||
# Usage: curl -fsSL https://raw.githubusercontent.com/unslothai/unsloth/main/scripts/uninstall.sh | sh
|
||||
|
||||
set -e
|
||||
|
||||
# Stop a Studio server via its PID file (written by install.sh's _spawn_terminal).
|
||||
_kill_pid_file() {
|
||||
_pid_file="$1"
|
||||
[ -f "$_pid_file" ] || return 0
|
||||
_pid=$(sed -n '1s/[^0-9].*//p' "$_pid_file" 2>/dev/null || true)
|
||||
if [ -n "$_pid" ] && kill -0 "$_pid" 2>/dev/null; then
|
||||
kill -TERM "$_pid" 2>/dev/null || true
|
||||
# Wait up to 10s for graceful shutdown.
|
||||
_i=0
|
||||
while kill -0 "$_pid" 2>/dev/null && [ "$_i" -lt 20 ]; do
|
||||
sleep 0.5
|
||||
_i=$((_i + 1))
|
||||
done
|
||||
kill -0 "$_pid" 2>/dev/null && kill -KILL "$_pid" 2>/dev/null || true
|
||||
fi
|
||||
rm -f "$_pid_file" 2>/dev/null || true
|
||||
}
|
||||
|
||||
# BRE-escape a path so it can be embedded in a pkill -f regex.
|
||||
_pkill_escape() {
|
||||
printf '%s' "$1" | sed -e 's:[][\\.^$*+?{|}()/]:\\&:g'
|
||||
}
|
||||
|
||||
_pkill_studio() {
|
||||
# Prefer PID files written by _spawn_terminal so we only touch our own installs.
|
||||
for _data_dir in "$HOME/.local/share/unsloth" $(_custom_studio_data_dirs); do
|
||||
[ -d "$_data_dir" ] || continue
|
||||
for _pf in "$_data_dir"/studio-*.pid; do
|
||||
[ -f "$_pf" ] && _kill_pid_file "$_pf"
|
||||
done
|
||||
done
|
||||
|
||||
command -v pkill >/dev/null 2>&1 || return 0
|
||||
|
||||
# Scope fallback patterns to the install roots we are removing so a
|
||||
# different Studio install (different UNSLOTH_STUDIO_HOME) is not touched.
|
||||
_kill_roots="$HOME/.unsloth/studio"
|
||||
_roots_from_conf=$(_custom_studio_roots 2>/dev/null || true)
|
||||
[ -n "$_roots_from_conf" ] && _kill_roots="$_kill_roots
|
||||
$_roots_from_conf"
|
||||
|
||||
printf '%s\n' "$_kill_roots" | while IFS= read -r _root; do
|
||||
[ -n "$_root" ] || continue
|
||||
[ -d "$_root" ] || continue
|
||||
_re=$(_pkill_escape "$_root")
|
||||
# `unsloth studio` (default port) + `-p N` + `--port N` forms, all
|
||||
# anchored on the install root's venv path.
|
||||
for _pat in \
|
||||
"${_re}/unsloth_studio/bin/[^ ]* studio( |\$|.*-p[ =][0-9])" \
|
||||
"${_re}/unsloth_studio/bin/[^ ]* studio.*--port[ =][0-9]" \
|
||||
"${_re}/.*studio/backend/run\.py"
|
||||
do
|
||||
pkill -TERM -f "$_pat" 2>/dev/null || true
|
||||
done
|
||||
done
|
||||
sleep 0.5
|
||||
printf '%s\n' "$_kill_roots" | while IFS= read -r _root; do
|
||||
[ -n "$_root" ] || continue
|
||||
[ -d "$_root" ] || continue
|
||||
_re=$(_pkill_escape "$_root")
|
||||
for _pat in \
|
||||
"${_re}/unsloth_studio/bin/[^ ]* studio( |\$|.*-p[ =][0-9])" \
|
||||
"${_re}/unsloth_studio/bin/[^ ]* studio.*--port[ =][0-9]" \
|
||||
"${_re}/.*studio/backend/run\.py"
|
||||
do
|
||||
pkill -KILL -f "$_pat" 2>/dev/null || true
|
||||
done
|
||||
done
|
||||
}
|
||||
|
||||
_remove_path() {
|
||||
_p="$1"
|
||||
if [ -e "$_p" ] || [ -L "$_p" ]; then
|
||||
rm -rf "$_p" 2>/dev/null && echo " removed: $_p" || echo " could not remove: $_p" >&2
|
||||
fi
|
||||
}
|
||||
|
||||
# Accept as Studio root only if Studio sentinels exist (matches install.sh's
|
||||
# env-mode ownership guard at install.sh:1358-1361). A bare unsloth_studio/
|
||||
# directory is NOT enough -- require the install-time owner marker so a user
|
||||
# directory that happens to contain a folder named "unsloth_studio" is safe.
|
||||
_is_studio_root() {
|
||||
_r="$1"
|
||||
[ -n "$_r" ] || return 1
|
||||
[ -f "$_r/share/studio.conf" ] && return 0
|
||||
[ -f "$_r/unsloth_studio/.unsloth-studio-owned" ] && return 0
|
||||
if [ -L "$_r/bin/unsloth" ]; then
|
||||
_t=$(readlink "$_r/bin/unsloth" 2>/dev/null || true)
|
||||
case "$_t" in *unsloth_studio/bin/unsloth) return 0 ;; esac
|
||||
fi
|
||||
return 1
|
||||
}
|
||||
|
||||
# Hard deny list: never delete /, $HOME, $HOME's parent, or system paths.
|
||||
_is_unsafe_root() {
|
||||
_r="$1"
|
||||
[ -z "$_r" ] && return 0
|
||||
case "$_r" in /|""|"$HOME"|"$HOME/") return 0 ;; esac
|
||||
case "$_r" in /bin|/sbin|/etc|/usr|/usr/*|/var|/var/*|/opt|/opt/*|/Library|/Library/*|/System|/System/*|/Applications|/Applications/*) return 0 ;; esac
|
||||
_parent=$(dirname "$HOME" 2>/dev/null || echo "")
|
||||
[ -n "$_parent" ] && [ "$_r" = "$_parent" ] && return 0
|
||||
return 1
|
||||
}
|
||||
|
||||
# Print share/ dirs of known custom roots (where PID files live).
|
||||
_custom_studio_data_dirs() {
|
||||
_custom_studio_roots 2>/dev/null | while IFS= read -r _r; do
|
||||
[ -d "$_r/share" ] && printf '%s\n' "$_r/share"
|
||||
done
|
||||
}
|
||||
|
||||
# Resolve a custom install root from any of:
|
||||
# 1. UNSLOTH_STUDIO_HOME / STUDIO_HOME env vars at uninstall time
|
||||
# 2. Default-mode studio.conf at $HOME/.local/share/unsloth/studio.conf
|
||||
# 3. Env-mode studio.conf at $<root>/share/studio.conf (discovered via 1)
|
||||
# install.sh writes UNSLOTH_EXE='<root>/unsloth_studio/bin/unsloth', so
|
||||
# the install root is three dirnames up. Prints each discovered non-default
|
||||
# root on its own line; the caller iterates and de-duplicates.
|
||||
_custom_studio_roots() {
|
||||
_seen=""
|
||||
_emit() {
|
||||
_r="$1"
|
||||
[ -z "$_r" ] && return 0
|
||||
# Tilde expansion (env vars are not subject to it on quoted assignment),
|
||||
# matches install.sh's _resolve_studio_destinations. The literal "~/"
|
||||
# pattern is intentional; SC2088 is a false positive here.
|
||||
# shellcheck disable=SC2088
|
||||
case "$_r" in
|
||||
"~") _r="$HOME" ;;
|
||||
"~/"*) _r="$HOME/${_r#'~/'}" ;;
|
||||
esac
|
||||
# Canonicalize so syntactic variants ($HOME/../$USER, trailing slash)
|
||||
# resolve to the same path and hit the _is_unsafe_root deny list.
|
||||
# shellcheck disable=SC1007
|
||||
_canon=$(CDPATH= cd -P -- "$_r" 2>/dev/null && pwd -P)
|
||||
[ -n "$_canon" ] && _r="$_canon"
|
||||
case "$_r" in "$HOME/.unsloth/studio"|/|"") return 0 ;; esac
|
||||
case ":$_seen:" in *":$_r:"*) return 0 ;; esac
|
||||
_seen="$_seen:$_r"
|
||||
printf '%s\n' "$_r"
|
||||
}
|
||||
_from_conf() {
|
||||
[ -f "$1" ] || return 0
|
||||
# Tolerate paths containing apostrophes (install.sh emits '\'' for them).
|
||||
_exe=$(sed -n "s/^UNSLOTH_EXE='\(.*\)'\$/\1/p" "$1" | head -n1)
|
||||
_exe=$(printf '%s' "$_exe" | sed "s/'\\\\''/'/g")
|
||||
[ -n "$_exe" ] || return 0
|
||||
_emit "$(dirname "$(dirname "$(dirname "$_exe")")")"
|
||||
}
|
||||
# Mirror install.sh's precedence: UNSLOTH_STUDIO_HOME wins, STUDIO_HOME is
|
||||
# ignored when both are set. Otherwise uninstalling install A could also
|
||||
# delete install B if the user has STUDIO_HOME left over from B.
|
||||
if [ -n "${UNSLOTH_STUDIO_HOME:-}" ]; then
|
||||
_emit "$UNSLOTH_STUDIO_HOME"
|
||||
_from_conf "$UNSLOTH_STUDIO_HOME/share/studio.conf"
|
||||
elif [ -n "${STUDIO_HOME:-}" ]; then
|
||||
_emit "$STUDIO_HOME"
|
||||
_from_conf "$STUDIO_HOME/share/studio.conf"
|
||||
fi
|
||||
# Default-mode conf.
|
||||
_from_conf "$HOME/.local/share/unsloth/studio.conf"
|
||||
}
|
||||
|
||||
# Remove $HOME/.local/bin/unsloth only if it's a Studio-managed symlink.
|
||||
# Studio's install.sh writes this as a symlink into the studio venv
|
||||
# (install.sh: `ln -sfn "$VENV_DIR/bin/unsloth" "$_shim_path"`). A
|
||||
# pip-installed `unsloth` CLI is a regular file — leave it alone to avoid
|
||||
# wiping an unrelated install.
|
||||
_remove_cli_shim() {
|
||||
_shim="$HOME/.local/bin/unsloth"
|
||||
[ -L "$_shim" ] || return 0
|
||||
_target=$(readlink "$_shim" 2>/dev/null || true)
|
||||
case "$_target" in
|
||||
*/unsloth_studio/bin/unsloth) _remove_path "$_shim" ;;
|
||||
*) ;;
|
||||
esac
|
||||
}
|
||||
|
||||
_uid=$(id -u 2>/dev/null || echo 0)
|
||||
_os=$(uname 2>/dev/null || echo unknown)
|
||||
_is_wsl=0
|
||||
[ "$_os" = "Linux" ] && grep -qi microsoft /proc/version 2>/dev/null && _is_wsl=1
|
||||
|
||||
echo "Stopping any running Unsloth Studio servers..."
|
||||
_pkill_studio
|
||||
|
||||
echo "Removing data and install directories..."
|
||||
_custom_studio_roots | while IFS= read -r _custom_root; do
|
||||
[ -n "$_custom_root" ] || continue
|
||||
if _is_unsafe_root "$_custom_root"; then
|
||||
echo " refusing to remove unsafe path: $_custom_root" >&2
|
||||
continue
|
||||
fi
|
||||
if ! _is_studio_root "$_custom_root"; then
|
||||
echo " refusing to remove non-Studio path: $_custom_root" >&2
|
||||
continue
|
||||
fi
|
||||
_remove_path "$_custom_root"
|
||||
done
|
||||
_remove_path "$HOME/.unsloth/studio"
|
||||
# Default-mode shared llama.cpp build + cache are siblings of studio (not removed
|
||||
# by deleting it). No-op in env/custom mode (they nest under the custom root) and
|
||||
# when absent. A user-set UNSLOTH_LLAMA_CPP_PATH is intentionally kept.
|
||||
_remove_path "$HOME/.unsloth/llama.cpp"
|
||||
_remove_path "$HOME/.unsloth/.cache"
|
||||
# Isolated Node.js runtime (install_node_prebuilt.py), a sibling of studio in
|
||||
# default mode. No-op in env/custom mode (nested under the custom root) and absent.
|
||||
_remove_path "$HOME/.unsloth/node"
|
||||
# llama.cpp atomic-install staging root (install_llama_prebuilt.py .staging).
|
||||
# Normally pruned after activate, but an interrupted build can leave it behind;
|
||||
# removing it lets the rmdir below succeed. No-op in env/custom mode and absent.
|
||||
_remove_path "$HOME/.unsloth/.staging"
|
||||
# llama.cpp install lock (serializes the shared build); a stray one keeps ~/.unsloth
|
||||
# from being pruned below. No-op in env/custom mode and when absent.
|
||||
_remove_path "$HOME/.unsloth/.llama.cpp.install.lock"
|
||||
# ROCm-on-WSL helper artifacts (librocdxg build clone + smoke-test venv). No-op
|
||||
# where they don't exist; removing them lets the rmdir below succeed.
|
||||
_remove_path "$HOME/.unsloth/librocdxg"
|
||||
_remove_path "$HOME/.unsloth/rocm-smoketest"
|
||||
# Drop ~/.unsloth only if now empty (rmdir refuses non-empty, so user content is kept).
|
||||
rmdir "$HOME/.unsloth" 2>/dev/null || true
|
||||
_remove_path "$HOME/.local/share/unsloth"
|
||||
# CLI shim: only the symlink Studio created, never a pip-installed file.
|
||||
_remove_cli_shim
|
||||
|
||||
echo "Removing desktop shortcut and launcher lock..."
|
||||
# install.sh creates Desktop/Unsloth Studio as a symlink. If the user has an
|
||||
# unrelated regular directory by that name, leave it alone.
|
||||
_desktop_link="$HOME/Desktop/Unsloth Studio"
|
||||
if [ -L "$_desktop_link" ] || [ ! -e "$_desktop_link" ]; then
|
||||
_remove_path "$_desktop_link"
|
||||
else
|
||||
echo " refusing to remove non-symlink Desktop path: $_desktop_link" >&2
|
||||
fi
|
||||
_remove_path "$HOME/Desktop/unsloth-studio.desktop"
|
||||
# Locks are namespaced per-uid; env-mode adds an extra suffix.
|
||||
_lock_glob="${XDG_RUNTIME_DIR:-/tmp}/unsloth-studio-launcher-${_uid}"
|
||||
for _lock in "$_lock_glob".lock "$_lock_glob"-*.lock; do
|
||||
[ -e "$_lock" ] && _remove_path "$_lock"
|
||||
done
|
||||
|
||||
case "$_os" in
|
||||
Darwin)
|
||||
echo "Removing macOS .app bundle and Launch Services entry..."
|
||||
_remove_path "$HOME/Applications/Unsloth Studio.app"
|
||||
_lsr="/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/Support/lsregister"
|
||||
if [ -x "$_lsr" ]; then
|
||||
"$_lsr" -u "$HOME/Applications/Unsloth Studio.app" 2>/dev/null || true
|
||||
fi
|
||||
;;
|
||||
Linux)
|
||||
if [ "$_is_wsl" = "1" ]; then
|
||||
echo "Removing WSL Windows-side shortcuts..."
|
||||
# install.sh creates per-distro 'Unsloth Studio (WSL - <distro>).lnk'
|
||||
# on the Windows Desktop + Start Menu via powershell.exe. Scope removal
|
||||
# to THIS distro (passed as $args[0]) so a multi-distro install keeps the
|
||||
# other distros' launchers; the TARGET=wsl.exe check still spares a
|
||||
# native install's "Unsloth Studio.lnk". Prefer powershell.exe; test it
|
||||
# can EXECUTE (`command -v` succeeds even with interop OFF -- .exe then
|
||||
# fails "Exec format error", common on systemd-enabled distros).
|
||||
_wsl_distro="${WSL_DISTRO_NAME:-}"
|
||||
_ps_ran=0
|
||||
if command -v powershell.exe >/dev/null 2>&1 && \
|
||||
powershell.exe -NoProfile -Command "exit 0" >/dev/null 2>&1; then
|
||||
_ps_ran=1
|
||||
# Inject the distro into the command: a -Command string does not
|
||||
# receive trailing tokens as $args. WSL distro names are safe to
|
||||
# embed (no quotes/$/backtick).
|
||||
# shellcheck disable=SC2016
|
||||
powershell.exe -NoProfile -Command '$distro = "'"$_wsl_distro"'";
|
||||
$dirs = @(
|
||||
[Environment]::GetFolderPath("Desktop"),
|
||||
(Join-Path $env:APPDATA "Microsoft\Windows\Start Menu\Programs")
|
||||
);
|
||||
$ws = New-Object -ComObject WScript.Shell;
|
||||
foreach ($d in $dirs) {
|
||||
if (-not $d -or -not (Test-Path -LiteralPath $d)) { continue }
|
||||
Get-ChildItem -LiteralPath $d -Filter "Unsloth Studio*.lnk" -ErrorAction SilentlyContinue | ForEach-Object {
|
||||
try {
|
||||
$sc = $ws.CreateShortcut($_.FullName);
|
||||
if ("$($sc.TargetPath) $($sc.Arguments)" -notmatch "wsl\.exe") { return }
|
||||
# When the distro is known, require the per-distro
|
||||
# name for this distro or its -d "<distro>" argument
|
||||
# so launchers for other distros are not removed.
|
||||
if ($distro) {
|
||||
$nameMatch = ($_.Name -eq "Unsloth Studio (WSL - $distro).lnk");
|
||||
$argMatch = ($sc.Arguments -match ("-d\s+`"?" + [regex]::Escape($distro) + "`"?"));
|
||||
if (-not ($nameMatch -or $argMatch)) { return }
|
||||
}
|
||||
Remove-Item -LiteralPath $_.FullName -Force -ErrorAction SilentlyContinue
|
||||
} catch { }
|
||||
}
|
||||
}
|
||||
# Keep the shared icon while any Unsloth shortcut still uses it (native
|
||||
# install or another WSL distro); drop it only with the last one.
|
||||
$iconInUse = $false;
|
||||
foreach ($d in $dirs) {
|
||||
if (-not $d -or -not (Test-Path -LiteralPath $d)) { continue }
|
||||
if (Get-ChildItem -LiteralPath $d -Filter "Unsloth Studio*.lnk" -ErrorAction SilentlyContinue) { $iconInUse = $true; break }
|
||||
}
|
||||
# Guard LOCALAPPDATA: empty on a service/SYSTEM account makes
|
||||
# Join-Path throw, aborting the icon cleanup (mirror uninstall.ps1).
|
||||
if (-not [string]::IsNullOrWhiteSpace($env:LOCALAPPDATA)) {
|
||||
$iconDir = Join-Path $env:LOCALAPPDATA "Unsloth Studio";
|
||||
$ico = Join-Path $iconDir "unsloth.ico";
|
||||
if ((-not $iconInUse) -and (Test-Path -LiteralPath $ico)) { Remove-Item -LiteralPath $ico -Force -ErrorAction SilentlyContinue }
|
||||
if ((Test-Path -LiteralPath $iconDir) -and -not (Get-ChildItem -LiteralPath $iconDir -Force -ErrorAction SilentlyContinue)) { Remove-Item -LiteralPath $iconDir -Recurse -Force -ErrorAction SilentlyContinue }
|
||||
}' >/dev/null 2>&1 || true
|
||||
fi
|
||||
# Remove $1's shared unsloth.ico only if no Unsloth shortcut (native install
|
||||
# or another WSL distro) still uses it, then drop the dir if empty. Reciprocal
|
||||
# of uninstall.ps1's _RemoveDataDirKeepingWslIcon (keeps the icon for a
|
||||
# surviving WSL shortcut when the native side is removed).
|
||||
_drop_shared_icon_if_unused() {
|
||||
_du="$1"
|
||||
_icodir="$_du/AppData/Local/Unsloth Studio"
|
||||
_icon_in_use=0
|
||||
for _sd in \
|
||||
"$_du/Desktop" \
|
||||
"$_du/OneDrive/Desktop" \
|
||||
"$_du"/OneDrive*/Desktop \
|
||||
"$_du/AppData/Roaming/Microsoft/Windows/Start Menu/Programs"; do
|
||||
[ -d "$_sd" ] || continue
|
||||
for _any in "$_sd"/"Unsloth Studio"*.lnk; do
|
||||
[ -e "$_any" ] && { _icon_in_use=1; break; }
|
||||
done
|
||||
[ "$_icon_in_use" = "1" ] && break
|
||||
done
|
||||
if [ "$_icon_in_use" = "0" ]; then
|
||||
[ -f "$_icodir/unsloth.ico" ] && rm -f "$_icodir/unsloth.ico" 2>/dev/null || true
|
||||
fi
|
||||
[ -d "$_icodir" ] && rmdir "$_icodir" 2>/dev/null || true
|
||||
}
|
||||
# Fallback when powershell.exe can't run (interop disabled): remove WSL .lnk
|
||||
# files via drvfs. The "Unsloth Studio (WSL..." name is WSL-specific, so a
|
||||
# native install's "Unsloth Studio.lnk" never matches.
|
||||
if [ "$_ps_ran" = "0" ]; then
|
||||
for _drive in /mnt/c /mnt/d /mnt/e; do
|
||||
[ -d "$_drive/Users" ] || continue
|
||||
for _udir in "$_drive"/Users/*; do
|
||||
[ -d "$_udir" ] || continue
|
||||
for _scdir in \
|
||||
"$_udir/Desktop" \
|
||||
"$_udir/OneDrive/Desktop" \
|
||||
"$_udir"/OneDrive*/Desktop \
|
||||
"$_udir/AppData/Roaming/Microsoft/Windows/Start Menu/Programs"; do
|
||||
[ -d "$_scdir" ] || continue
|
||||
if [ -n "$_wsl_distro" ]; then
|
||||
# Exact per-distro name (no glob) so other distros survive.
|
||||
_lnk="$_scdir/Unsloth Studio (WSL - ${_wsl_distro}).lnk"
|
||||
[ -e "$_lnk" ] && rm -f "$_lnk" 2>/dev/null && echo " removed: $_lnk" || true
|
||||
else
|
||||
# Distro unknown: fall back to the broad WSL prefix.
|
||||
for _lnk in "$_scdir"/"Unsloth Studio (WSL"*.lnk; do
|
||||
[ -e "$_lnk" ] && rm -f "$_lnk" 2>/dev/null && echo " removed: $_lnk" || true
|
||||
done
|
||||
fi
|
||||
done
|
||||
# Drop the shared icon only when no shortcut still needs it.
|
||||
_drop_shared_icon_if_unused "$_udir"
|
||||
done
|
||||
done
|
||||
fi
|
||||
# ── ROCm-on-WSL config (install_rocm_wsl_strixhalo.sh) ──
|
||||
# Remove Unsloth's own ROCDXG config (the env it persisted). The system
|
||||
# ROCm userspace is a shared prereq (like CUDA) and is LEFT IN PLACE by
|
||||
# default; set UNSLOTH_UNINSTALL_ROCM=1 to remove it too.
|
||||
echo "Removing ROCm-on-WSL config..."
|
||||
_sudo=""
|
||||
if [ "$_uid" != "0" ] && command -v sudo >/dev/null 2>&1; then _sudo="sudo"; fi
|
||||
$_sudo rm -f /etc/profile.d/unsloth-rocm-wsl.sh 2>/dev/null || true
|
||||
if [ -f "$HOME/.bashrc" ] && grep -q "Unsloth ROCm-on-WSL" "$HOME/.bashrc" 2>/dev/null; then
|
||||
_bk=$(mktemp 2>/dev/null || echo "$HOME/.bashrc.unsloth.tmp")
|
||||
if sed '/# >>> Unsloth ROCm-on-WSL/,/# <<< Unsloth ROCm-on-WSL/d' "$HOME/.bashrc" > "$_bk" 2>/dev/null; then
|
||||
cat "$_bk" > "$HOME/.bashrc" 2>/dev/null || true
|
||||
echo " cleaned ROCm-on-WSL block from ~/.bashrc"
|
||||
fi
|
||||
rm -f "$_bk" 2>/dev/null || true
|
||||
fi
|
||||
if [ "${UNSLOTH_UNINSTALL_ROCM:-0}" = "1" ]; then
|
||||
echo " removing system ROCm (UNSLOTH_UNINSTALL_ROCM=1)..."
|
||||
$_sudo rm -f /etc/apt/sources.list.d/rocm.list /etc/apt/preferences.d/rocm-pin-600 \
|
||||
/etc/apt/keyrings/rocm.gpg /etc/ld.so.conf.d/rocm.conf 2>/dev/null || true
|
||||
$_sudo sh -c 'rm -rf /opt/rocm /opt/rocm-*' 2>/dev/null || true
|
||||
if command -v ldconfig >/dev/null 2>&1; then $_sudo ldconfig 2>/dev/null || true; fi
|
||||
elif [ -d /opt/rocm ]; then
|
||||
echo " Note: ROCm userspace (/opt/rocm*) left in place (shared prereq)."
|
||||
echo " Remove it by re-running with UNSLOTH_UNINSTALL_ROCM=1, or manually:"
|
||||
echo " sudo rm -rf /opt/rocm /opt/rocm-* && sudo ldconfig"
|
||||
fi
|
||||
fi
|
||||
echo "Removing Linux .desktop entry..."
|
||||
_remove_path "$HOME/.local/share/applications/unsloth-studio.desktop"
|
||||
if command -v update-desktop-database >/dev/null 2>&1; then
|
||||
update-desktop-database "$HOME/.local/share/applications" 2>/dev/null || true
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
|
||||
echo ""
|
||||
echo "Unsloth Studio uninstalled."
|
||||
echo "Note: Hugging Face model cache at ~/.cache/huggingface was left in place."
|
||||
echo "Remove it manually with 'rm -rf ~/.cache/huggingface/hub' if desired."
|
||||
# Env-mode installs leave no breadcrumb in $HOME, so a custom root can
|
||||
# only be located if the user re-exports the variable. Print a hint when
|
||||
# neither var is set so the bare `curl | sh` flow doesn't silently miss.
|
||||
if [ -z "${UNSLOTH_STUDIO_HOME:-}" ] && [ -z "${STUDIO_HOME:-}" ]; then
|
||||
echo ""
|
||||
echo "If you installed Unsloth Studio with UNSLOTH_STUDIO_HOME or STUDIO_HOME"
|
||||
echo "pointing at a custom directory, re-run this script with the same variable"
|
||||
echo "set to also remove that install tree, e.g.:"
|
||||
echo " UNSLOTH_STUDIO_HOME=/your/path sh -c \"\$(curl -fsSL https://raw.githubusercontent.com/unslothai/unsloth/main/scripts/uninstall.sh)\""
|
||||
fi
|
||||
@@ -0,0 +1,245 @@
|
||||
# Unsloth - 2x faster, 60% less VRAM LLM training and finetuning
|
||||
# Copyright 2023-present Daniel Han-Chen, Michael Han-Chen & the Unsloth team. All rights reserved.
|
||||
#
|
||||
# This program is free software: you can redistribute it and/or modify
|
||||
# it under the terms of the GNU Lesser General Public License as published by
|
||||
# the Free Software Foundation, either version 3 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU Lesser General Public License for more details.
|
||||
|
||||
"""Deterministic comment / docstring-only verifier.
|
||||
|
||||
Compares a list of changed files between two git refs and reports whether
|
||||
each diff is strictly comments / docstrings (Python) or comments
|
||||
(YAML / GitHub Actions). Useful for gating a "comment trim" /
|
||||
"docstring refactor" PR against accidental code drift.
|
||||
|
||||
Per .py file: parse both revs into AST, strip module / class / function
|
||||
docstrings, then compare ast.unparse output. Pure Python comments are
|
||||
discarded by the parser by construction, so any post-strip diff is real
|
||||
code. Per .yml file: yaml.safe_load both sides and compare the parsed
|
||||
Python object; if scalar values differ, also strip shell comments inside
|
||||
``run: |`` block bodies before comparing. Exit code 0 = all OK, 1 = at
|
||||
least one file has a real (non-comment) diff or an error.
|
||||
|
||||
Usage:
|
||||
python scripts/verify_comment_only_diff.py [--base REF] [--head REF] path ...
|
||||
|
||||
Defaults: --base origin/main, --head HEAD. Paths are repo-relative.
|
||||
|
||||
Example:
|
||||
git diff --name-only origin/main..HEAD \\
|
||||
| xargs python scripts/verify_comment_only_diff.py --base origin/main
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import ast
|
||||
import difflib
|
||||
import subprocess
|
||||
import sys
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
|
||||
def _git_show(rev: str, path: str) -> str:
|
||||
return subprocess.check_output(
|
||||
["git", "show", f"{rev}:{path}"],
|
||||
text = True,
|
||||
stderr = subprocess.DEVNULL,
|
||||
)
|
||||
|
||||
|
||||
def _strip_docstrings(tree: ast.AST) -> ast.AST:
|
||||
"""Remove docstrings; empty bodies become ``pass`` so unparse stays valid."""
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(
|
||||
node,
|
||||
(ast.Module, ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef),
|
||||
):
|
||||
body = getattr(node, "body", None)
|
||||
if not body:
|
||||
continue
|
||||
first = body[0]
|
||||
if (
|
||||
isinstance(first, ast.Expr)
|
||||
and isinstance(first.value, ast.Constant)
|
||||
and isinstance(first.value.value, str)
|
||||
):
|
||||
node.body = body[1:]
|
||||
if not node.body:
|
||||
node.body = [ast.Pass()]
|
||||
return tree
|
||||
|
||||
|
||||
def _normalize_py(src: str) -> str:
|
||||
tree = ast.parse(src)
|
||||
tree = _strip_docstrings(tree)
|
||||
return ast.unparse(tree)
|
||||
|
||||
|
||||
def _strip_shell_comments(s: str) -> str:
|
||||
"""Strip shell comments and collapse blank lines. Heuristic: skips lines
|
||||
with an odd quote count (open string)."""
|
||||
out = []
|
||||
for line in s.splitlines():
|
||||
stripped = line.lstrip()
|
||||
if stripped.startswith("#"):
|
||||
continue
|
||||
has_single = line.count("'") % 2 == 0
|
||||
has_double = line.count('"') % 2 == 0
|
||||
if has_single and has_double:
|
||||
idx = line.find(" #")
|
||||
if idx >= 0:
|
||||
line = line[:idx].rstrip()
|
||||
out.append(line)
|
||||
norm = []
|
||||
prev_blank = False
|
||||
for line in out:
|
||||
if line.strip() == "":
|
||||
if prev_blank:
|
||||
continue
|
||||
prev_blank = True
|
||||
else:
|
||||
prev_blank = False
|
||||
norm.append(line)
|
||||
return "\n".join(norm).strip()
|
||||
|
||||
|
||||
def _normalize_yaml_run_strings(obj: Any) -> Any:
|
||||
"""Strip shell comments from any multi-line string (``run: |`` body)."""
|
||||
if isinstance(obj, dict):
|
||||
return {k: _normalize_yaml_run_strings(v) for k, v in obj.items()}
|
||||
if isinstance(obj, list):
|
||||
return [_normalize_yaml_run_strings(x) for x in obj]
|
||||
if isinstance(obj, str) and "\n" in obj:
|
||||
return _strip_shell_comments(obj)
|
||||
return obj
|
||||
|
||||
|
||||
def _walk_yaml_diff(
|
||||
b: Any,
|
||||
a: Any,
|
||||
prefix: str = "",
|
||||
) -> None:
|
||||
"""Print a path-keyed summary of the first structural / scalar diff."""
|
||||
if type(b) is not type(a):
|
||||
print(
|
||||
f" type-diff at {prefix or '/'}: " f"{type(b).__name__} -> {type(a).__name__}",
|
||||
)
|
||||
return
|
||||
if isinstance(b, dict):
|
||||
keys = sorted((set(b.keys()) | set(a.keys())), key = lambda x: str(x))
|
||||
for k in keys:
|
||||
if k not in b:
|
||||
print(f" added key {prefix}/{k}")
|
||||
elif k not in a:
|
||||
print(f" removed key {prefix}/{k}")
|
||||
else:
|
||||
_walk_yaml_diff(b[k], a[k], f"{prefix}/{k}")
|
||||
elif isinstance(b, list):
|
||||
if len(b) != len(a):
|
||||
print(
|
||||
f" list len at {prefix or '/'}: " f"{len(b)} -> {len(a)}",
|
||||
)
|
||||
for i, (bi, ai) in enumerate(zip(b, a)):
|
||||
_walk_yaml_diff(bi, ai, f"{prefix}[{i}]")
|
||||
elif b != a:
|
||||
bs = repr(b)[:300]
|
||||
as_ = repr(a)[:300]
|
||||
print(f" scalar at {prefix or '/'}:")
|
||||
print(f" before: {bs}")
|
||||
print(f" after: {as_}")
|
||||
|
||||
|
||||
def _verify_python(path: str, before: str, after: str) -> bool:
|
||||
try:
|
||||
norm_before = _normalize_py(before)
|
||||
norm_after = _normalize_py(after)
|
||||
except SyntaxError as exc:
|
||||
print(f"FAIL {path}: SyntaxError parsing -- {exc}")
|
||||
return False
|
||||
if norm_before == norm_after:
|
||||
print(f"OK {path} (AST identical after docstring strip)")
|
||||
return True
|
||||
diff = list(
|
||||
difflib.unified_diff(
|
||||
norm_before.splitlines(),
|
||||
norm_after.splitlines(),
|
||||
fromfile = f"{path}@before",
|
||||
tofile = f"{path}@after",
|
||||
n = 2,
|
||||
)
|
||||
)
|
||||
print(f"FAIL {path}: AST differs after docstring strip:")
|
||||
for line in diff[:40]:
|
||||
print(f" {line}")
|
||||
return False
|
||||
|
||||
|
||||
def _verify_yaml(path: str, before: str, after: str) -> bool:
|
||||
try:
|
||||
raw_before = yaml.safe_load(before)
|
||||
raw_after = yaml.safe_load(after)
|
||||
except yaml.YAMLError as exc:
|
||||
print(f"FAIL {path}: YAML parse error -- {exc}")
|
||||
return False
|
||||
if raw_before == raw_after:
|
||||
print(f"OK {path} (YAML parsed object identical)")
|
||||
return True
|
||||
norm_before = _normalize_yaml_run_strings(raw_before)
|
||||
norm_after = _normalize_yaml_run_strings(raw_after)
|
||||
if norm_before == norm_after:
|
||||
print(
|
||||
f"OK {path} (YAML parsed object identical after "
|
||||
f"stripping shell comments from run: bodies)",
|
||||
)
|
||||
return True
|
||||
print(
|
||||
f"FAIL {path}: YAML parsed objects still differ after stripping "
|
||||
f"shell comments from `run:` bodies.",
|
||||
)
|
||||
_walk_yaml_diff(norm_before, norm_after)
|
||||
return False
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description = "Verify each path's diff between BASE and HEAD is "
|
||||
"strictly comments / docstrings.",
|
||||
)
|
||||
parser.add_argument("--base", default = "origin/main", help = "base git ref")
|
||||
parser.add_argument("--head", default = "HEAD", help = "head git ref")
|
||||
parser.add_argument("paths", nargs = "+", help = "repo-relative paths")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
rc = 0
|
||||
print(f"Comparing {len(args.paths)} files: {args.base} vs {args.head}\n")
|
||||
for path in args.paths:
|
||||
try:
|
||||
before = _git_show(args.base, path)
|
||||
after = _git_show(args.head, path)
|
||||
except subprocess.CalledProcessError as exc:
|
||||
print(f"SKIP {path}: {exc}")
|
||||
continue
|
||||
|
||||
if path.endswith(".py"):
|
||||
if not _verify_python(path, before, after):
|
||||
rc = 1
|
||||
elif path.endswith((".yml", ".yaml")):
|
||||
if not _verify_yaml(path, before, after):
|
||||
rc = 1
|
||||
else:
|
||||
print(f"NOTE {path}: not .py or .yaml -- skipped automated check.")
|
||||
|
||||
return rc
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,839 @@
|
||||
#!/usr/bin/env python3
|
||||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
|
||||
"""Deterministic, scope-aware verifier for import-hoisting / alias-rename refactors.
|
||||
|
||||
The risk when moving `from a import b as _b` (or `import b as _b`) to module top
|
||||
and normalizing `_b` -> `b` is twofold:
|
||||
|
||||
1. DANGLING ALIAS - a `_b` reference is left un-normalized; it now resolves to
|
||||
nothing (NameError) or, worse, to some *other* module-level `_b`.
|
||||
2. RENAME CLASH - `_b` was an alias on purpose because `b` already meant
|
||||
something else in that scope; normalizing `_b` -> `b` silently re-points the
|
||||
reference at the wrong object (no NameError, no pyflakes warning).
|
||||
|
||||
This tool parses BEFORE (a git ref, default origin/main) and AFTER (default HEAD)
|
||||
for each file, builds a real LEGB scope model (functions, classes, lambdas,
|
||||
comprehensions, global/nonlocal, args, walrus, star-imports), and resolves every
|
||||
Name load to its binding. It then compares, PER SCOPE:
|
||||
|
||||
* UNRESOLVED-NEW : loads that resolve to nothing in AFTER but did in BEFORE
|
||||
(or are newly present) -> catches dangling aliases.
|
||||
* TARGET-MISSING : an import *target* (e.g. module `glob`, or
|
||||
`importlib.metadata.version`) that a function resolved to
|
||||
in BEFORE but no longer resolves to in AFTER -> catches a
|
||||
function that lost access to a module it still uses.
|
||||
Robust to alias renames because it compares the *target*,
|
||||
not the local name.
|
||||
* TARGET-CHANGED : a load whose resolved import target differs BEFORE vs
|
||||
AFTER -> catches a rename that re-points to a different
|
||||
module (the clash case).
|
||||
* AMBIGUOUS-BIND : a name bound by BOTH an import and a non-import in the same
|
||||
scope in AFTER (and not in BEFORE) -> the "alias was on
|
||||
purpose / now collides" smell.
|
||||
* MODULE-DUP-IMPORT: a module-level name imported and also defined/assigned at
|
||||
module level (introduced by the change).
|
||||
* NEW-UNUSED-IMPORT: a module-level import added in AFTER that nothing resolves
|
||||
to (informational; re-exports are a known false positive).
|
||||
|
||||
Usage:
|
||||
verify_import_hoist.py [--before REF] [--after REF] <file>... # compare
|
||||
verify_import_hoist.py --self-test # prove it catches bugs
|
||||
Exit code 1 if any non-informational finding.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import ast
|
||||
import builtins
|
||||
import re as _re_mod
|
||||
import subprocess
|
||||
import sys
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
_BUILTINS = set(dir(builtins)) | {
|
||||
"__file__",
|
||||
"__name__",
|
||||
"__doc__",
|
||||
"__package__",
|
||||
"__spec__",
|
||||
"__loader__",
|
||||
"__builtins__",
|
||||
"__class__",
|
||||
"__annotations__",
|
||||
"__dict__",
|
||||
"__qualname__",
|
||||
"__module__",
|
||||
"__path__",
|
||||
"__debug__",
|
||||
"__import__",
|
||||
"NotImplemented",
|
||||
"Ellipsis",
|
||||
"copyright",
|
||||
"credits",
|
||||
"license",
|
||||
"help",
|
||||
"exit",
|
||||
"quit",
|
||||
"__build_class__",
|
||||
"__cached__",
|
||||
"reveal_type",
|
||||
"reveal_locals",
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- scope model
|
||||
|
||||
|
||||
@dataclass
|
||||
class Binding:
|
||||
kind: str # 'import' | 'importfrom' | 'def' | 'class' | 'other'
|
||||
target: str | None = None # canonical import target id, else None
|
||||
|
||||
|
||||
@dataclass
|
||||
class Scope:
|
||||
kind: str # 'module' | 'function' | 'class' | 'lambda' | 'comp'
|
||||
qualname: str
|
||||
parent: "Scope | None"
|
||||
bindings: dict[str, list[Binding]] = field(default_factory = dict)
|
||||
globals: set[str] = field(default_factory = set)
|
||||
nonlocals: set[str] = field(default_factory = set)
|
||||
star_import: bool = False
|
||||
|
||||
def add(self, name: str, b: Binding) -> None:
|
||||
self.bindings.setdefault(name, []).append(b)
|
||||
|
||||
|
||||
def _import_target(node: ast.AST, alias: ast.alias) -> tuple[str, str]:
|
||||
"""Return (bound_name, canonical_target_id) for one import alias."""
|
||||
if isinstance(node, ast.Import):
|
||||
bound = alias.asname or alias.name.split(".")[0]
|
||||
return bound, f"import:{alias.name}"
|
||||
# ImportFrom
|
||||
bound = alias.asname or alias.name
|
||||
mod = ("." * (node.level or 0)) + (node.module or "")
|
||||
return bound, f"from:{mod}:{alias.name}"
|
||||
|
||||
|
||||
class _Builder(ast.NodeVisitor):
|
||||
"""Builds the scope tree + bindings, and records every (scope, Name-load)."""
|
||||
|
||||
def __init__(self):
|
||||
self.module = Scope("module", "<module>", None)
|
||||
self.uses: list[tuple[Scope, str, int]] = [] # hard loads
|
||||
# annotations: count as "used" but never as "unresolved" (forward refs)
|
||||
self.soft_uses: list[tuple[Scope, str, int]] = []
|
||||
|
||||
def _visit_annotation(self, node, scope: Scope) -> None:
|
||||
"""Record annotation names as SOFT uses: an import used only in an annotation
|
||||
counts as used, but a forward-ref name is never 'unresolved'."""
|
||||
if node is None:
|
||||
return
|
||||
for n in ast.walk(node):
|
||||
if isinstance(n, ast.Name) and isinstance(n.ctx, ast.Load):
|
||||
self.soft_uses.append((scope, n.id, n.lineno))
|
||||
|
||||
# -- binding helpers --
|
||||
def _bind_targets(self, scope: Scope, target: ast.AST) -> None:
|
||||
for n in ast.walk(target):
|
||||
if isinstance(n, ast.Name) and isinstance(n.ctx, (ast.Store, ast.Del)):
|
||||
self._bind_name(scope, n.id, Binding("other"))
|
||||
elif isinstance(n, ast.Starred):
|
||||
pass
|
||||
|
||||
def _bind_name(self, scope: Scope, name: str, b: Binding) -> None:
|
||||
if name in scope.globals:
|
||||
self.module.add(name, b)
|
||||
elif name in scope.nonlocals:
|
||||
p = scope.parent
|
||||
while p is not None and p.kind not in ("function", "lambda"):
|
||||
p = p.parent
|
||||
(p or self.module).add(name, b)
|
||||
else:
|
||||
scope.add(name, b)
|
||||
|
||||
# -- generic dispatch within a scope --
|
||||
def _visit_body(self, stmts, scope: Scope) -> None:
|
||||
for s in stmts:
|
||||
self._visit_stmt(s, scope)
|
||||
|
||||
def _visit_stmt(self, node: ast.AST, scope: Scope) -> None:
|
||||
if isinstance(node, (ast.Import, ast.ImportFrom)):
|
||||
star = isinstance(node, ast.ImportFrom) and any(a.name == "*" for a in node.names)
|
||||
if star:
|
||||
scope.star_import = True
|
||||
for alias in node.names:
|
||||
if alias.name == "*":
|
||||
continue
|
||||
bound, target = _import_target(node, alias)
|
||||
kind = "import" if isinstance(node, ast.Import) else "importfrom"
|
||||
self._bind_name(scope, bound, Binding(kind, target))
|
||||
return
|
||||
if isinstance(node, ast.Global):
|
||||
scope.globals.update(node.names)
|
||||
return
|
||||
if isinstance(node, ast.Nonlocal):
|
||||
scope.nonlocals.update(node.names)
|
||||
return
|
||||
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
|
||||
self._bind_name(scope, node.name, Binding("def"))
|
||||
# decorators / defaults evaluate in the ENCLOSING scope
|
||||
for d in node.decorator_list:
|
||||
self._visit_expr(d, scope)
|
||||
self._visit_arg_defaults(node.args, scope)
|
||||
child = Scope("function", f"{scope.qualname}.{node.name}", scope)
|
||||
self._bind_type_params(node, child)
|
||||
self._bind_args(node.args, child)
|
||||
# arg + return annotations: soft uses
|
||||
for a in self._all_args(node.args):
|
||||
self._visit_annotation(a.annotation, child)
|
||||
self._visit_annotation(getattr(node, "returns", None), child)
|
||||
self._visit_body(node.body, child)
|
||||
return
|
||||
if isinstance(node, ast.ClassDef):
|
||||
self._bind_name(scope, node.name, Binding("class"))
|
||||
for d in node.decorator_list:
|
||||
self._visit_expr(d, scope)
|
||||
for b in node.bases:
|
||||
self._visit_expr(b, scope)
|
||||
for kw in node.keywords:
|
||||
self._visit_expr(kw.value, scope)
|
||||
child = Scope("class", f"{scope.qualname}.{node.name}", scope)
|
||||
self._bind_type_params(node, child)
|
||||
self._visit_body(node.body, child)
|
||||
return
|
||||
if isinstance(node, ast.Match):
|
||||
self._visit_expr(node.subject, scope)
|
||||
for case in node.cases:
|
||||
self._bind_pattern(case.pattern, scope)
|
||||
if case.guard is not None:
|
||||
self._visit_expr(case.guard, scope)
|
||||
self._visit_body(case.body, scope)
|
||||
return
|
||||
if isinstance(node, getattr(ast, "TryStar", ())): # py3.11 except*
|
||||
self._visit_body(node.body, scope)
|
||||
for h in node.handlers:
|
||||
if h.type is not None:
|
||||
self._visit_expr(h.type, scope)
|
||||
if h.name:
|
||||
self._bind_name(scope, h.name, Binding("other"))
|
||||
self._visit_body(h.body, scope)
|
||||
self._visit_body(node.orelse, scope)
|
||||
self._visit_body(node.finalbody, scope)
|
||||
return
|
||||
if isinstance(node, getattr(ast, "TypeAlias", ())): # py3.12 `type X = ...`
|
||||
if isinstance(node.name, ast.Name):
|
||||
self._bind_name(scope, node.name.id, Binding("other"))
|
||||
self._visit_annotation(node.value, scope)
|
||||
return
|
||||
if isinstance(node, (ast.Assign, ast.AnnAssign, ast.AugAssign)):
|
||||
targets = node.targets if isinstance(node, ast.Assign) else [node.target]
|
||||
val = node.value
|
||||
if val is not None:
|
||||
self._visit_expr(val, scope)
|
||||
if isinstance(node, ast.AnnAssign) and node.annotation is not None:
|
||||
self._visit_annotation(node.annotation, scope)
|
||||
for t in targets:
|
||||
self._bind_targets(scope, t)
|
||||
# AugAssign target is also a load
|
||||
if isinstance(node, ast.AugAssign):
|
||||
self._record_loads(t, scope)
|
||||
return
|
||||
if isinstance(node, (ast.For, ast.AsyncFor)):
|
||||
self._visit_expr(node.iter, scope)
|
||||
self._bind_targets(scope, node.target)
|
||||
self._visit_body(node.body, scope)
|
||||
self._visit_body(node.orelse, scope)
|
||||
return
|
||||
if isinstance(node, (ast.With, ast.AsyncWith)):
|
||||
for item in node.items:
|
||||
self._visit_expr(item.context_expr, scope)
|
||||
if item.optional_vars is not None:
|
||||
self._bind_targets(scope, item.optional_vars)
|
||||
self._visit_body(node.body, scope)
|
||||
return
|
||||
if isinstance(node, ast.Try):
|
||||
self._visit_body(node.body, scope)
|
||||
for h in node.handlers:
|
||||
if h.type is not None:
|
||||
self._visit_expr(h.type, scope)
|
||||
if h.name:
|
||||
self._bind_name(scope, h.name, Binding("other"))
|
||||
self._visit_body(h.body, scope)
|
||||
self._visit_body(node.orelse, scope)
|
||||
self._visit_body(node.finalbody, scope)
|
||||
return
|
||||
# generic statement: visit all child expressions/stmts in same scope
|
||||
for child in ast.iter_child_nodes(node):
|
||||
if isinstance(child, ast.stmt):
|
||||
self._visit_stmt(child, scope)
|
||||
else:
|
||||
self._visit_expr(child, scope)
|
||||
|
||||
# -- expressions --
|
||||
def _visit_arg_defaults(self, args: ast.arguments, scope: Scope) -> None:
|
||||
for d in list(args.defaults) + [d for d in args.kw_defaults if d is not None]:
|
||||
self._visit_expr(d, scope)
|
||||
|
||||
def _all_args(self, args: ast.arguments) -> list[ast.arg]:
|
||||
out = list(args.posonlyargs) + list(args.args) + list(args.kwonlyargs)
|
||||
if args.vararg:
|
||||
out.append(args.vararg)
|
||||
if args.kwarg:
|
||||
out.append(args.kwarg)
|
||||
return out
|
||||
|
||||
def _bind_args(self, args: ast.arguments, scope: Scope) -> None:
|
||||
for a in self._all_args(args):
|
||||
scope.add(a.arg, Binding("other"))
|
||||
|
||||
def _bind_type_params(self, node, scope: Scope) -> None:
|
||||
for tp in getattr(node, "type_params", []) or []:
|
||||
name = getattr(tp, "name", None)
|
||||
if isinstance(name, str):
|
||||
scope.add(name, Binding("other"))
|
||||
self._visit_annotation(getattr(tp, "bound", None), scope)
|
||||
self._visit_annotation(getattr(tp, "default_value", None), scope)
|
||||
|
||||
def _bind_pattern(self, pat, scope: Scope) -> None:
|
||||
if pat is None:
|
||||
return
|
||||
if isinstance(pat, ast.MatchValue):
|
||||
self._visit_expr(pat.value, scope)
|
||||
elif isinstance(pat, ast.MatchSingleton):
|
||||
pass
|
||||
elif isinstance(pat, ast.MatchSequence):
|
||||
for p in pat.patterns:
|
||||
self._bind_pattern(p, scope)
|
||||
elif isinstance(pat, ast.MatchStar):
|
||||
if pat.name:
|
||||
self._bind_name(scope, pat.name, Binding("other"))
|
||||
elif isinstance(pat, ast.MatchMapping):
|
||||
for k in pat.keys:
|
||||
self._visit_expr(k, scope)
|
||||
for p in pat.patterns:
|
||||
self._bind_pattern(p, scope)
|
||||
if pat.rest:
|
||||
self._bind_name(scope, pat.rest, Binding("other"))
|
||||
elif isinstance(pat, ast.MatchClass):
|
||||
self._visit_expr(pat.cls, scope)
|
||||
for p in pat.patterns:
|
||||
self._bind_pattern(p, scope)
|
||||
for p in pat.kwd_patterns:
|
||||
self._bind_pattern(p, scope)
|
||||
elif isinstance(pat, ast.MatchAs):
|
||||
self._bind_pattern(pat.pattern, scope)
|
||||
if pat.name:
|
||||
self._bind_name(scope, pat.name, Binding("other"))
|
||||
elif isinstance(pat, ast.MatchOr):
|
||||
for p in pat.patterns:
|
||||
self._bind_pattern(p, scope)
|
||||
|
||||
def _record_loads(self, node: ast.AST, scope: Scope) -> None:
|
||||
for n in ast.walk(node):
|
||||
if isinstance(n, ast.Name) and isinstance(n.ctx, ast.Load):
|
||||
self.uses.append((scope, n.id, n.lineno))
|
||||
|
||||
def _visit_expr(self, node: ast.AST, scope: Scope) -> None:
|
||||
if isinstance(node, ast.Name):
|
||||
if isinstance(node.ctx, ast.Load):
|
||||
self.uses.append((scope, node.id, node.lineno))
|
||||
elif isinstance(node.ctx, (ast.Store, ast.Del)):
|
||||
self._bind_name(scope, node.id, Binding("other"))
|
||||
return
|
||||
if isinstance(node, ast.Lambda):
|
||||
self._visit_arg_defaults(node.args, scope)
|
||||
child = Scope("lambda", f"{scope.qualname}.<lambda>", scope)
|
||||
self._bind_args(node.args, child)
|
||||
self._visit_expr(node.body, child)
|
||||
return
|
||||
if isinstance(node, (ast.ListComp, ast.SetComp, ast.GeneratorExp, ast.DictComp)):
|
||||
child = Scope("comp", f"{scope.qualname}.<comp>", scope)
|
||||
for i, gen in enumerate(node.generators):
|
||||
# first iterable evaluates in the enclosing scope
|
||||
self._visit_expr(gen.iter, scope if i == 0 else child)
|
||||
self._bind_targets(child, gen.target)
|
||||
for cond in gen.ifs:
|
||||
self._visit_expr(cond, child)
|
||||
if isinstance(node, ast.DictComp):
|
||||
self._visit_expr(node.key, child)
|
||||
self._visit_expr(node.value, child)
|
||||
else:
|
||||
self._visit_expr(node.elt, child)
|
||||
return
|
||||
if isinstance(node, ast.NamedExpr): # walrus binds in enclosing scope
|
||||
self._visit_expr(node.value, scope)
|
||||
if isinstance(node.target, ast.Name):
|
||||
self._bind_name(scope, node.target.id, Binding("other"))
|
||||
return
|
||||
for child in ast.iter_child_nodes(node):
|
||||
if isinstance(child, ast.stmt):
|
||||
self._visit_stmt(child, scope)
|
||||
else:
|
||||
self._visit_expr(child, scope)
|
||||
|
||||
def run(self, tree: ast.Module) -> None:
|
||||
self._visit_body(tree.body, self.module)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- resolution
|
||||
|
||||
|
||||
def _any_star(scope: Scope) -> bool:
|
||||
c = scope
|
||||
while c is not None:
|
||||
if c.star_import:
|
||||
return True
|
||||
c = c.parent
|
||||
return False
|
||||
|
||||
|
||||
def _resolve(scope: Scope, name: str):
|
||||
"""LEGB resolution. Returns (status, bindings); status in
|
||||
{'local','import','other','builtin','star','unresolved'}."""
|
||||
start = scope
|
||||
if name in scope.globals:
|
||||
chain = [_module_of(scope)]
|
||||
elif name in scope.nonlocals:
|
||||
chain = _enclosing_functions(scope)
|
||||
else:
|
||||
chain = _legb_chain(scope)
|
||||
for i, sc in enumerate(chain):
|
||||
if sc is None:
|
||||
continue
|
||||
if name in sc.bindings:
|
||||
binds = sc.bindings[name]
|
||||
if any(b.kind in ("import", "importfrom") for b in binds):
|
||||
return "import", binds
|
||||
return "other", binds
|
||||
if name in _BUILTINS:
|
||||
return "builtin", []
|
||||
if _any_star(start):
|
||||
return "star", []
|
||||
return "unresolved", []
|
||||
|
||||
|
||||
def _module_of(scope: Scope) -> Scope:
|
||||
while scope.parent is not None:
|
||||
scope = scope.parent
|
||||
return scope
|
||||
|
||||
|
||||
def _enclosing_functions(scope: Scope) -> list[Scope]:
|
||||
out = []
|
||||
p = scope.parent
|
||||
while p is not None:
|
||||
if p.kind in ("function", "lambda"):
|
||||
out.append(p)
|
||||
p = p.parent
|
||||
out.append(_module_of(scope))
|
||||
return out
|
||||
|
||||
|
||||
def _legb_chain(scope: Scope) -> list[Scope]:
|
||||
"""Immediate scope, then enclosing scopes skipping class scopes, then module."""
|
||||
chain = [scope]
|
||||
p = scope.parent
|
||||
while p is not None:
|
||||
if p.kind != "class" or p.parent is None: # skip class scopes, keep module
|
||||
if p.kind != "class":
|
||||
chain.append(p)
|
||||
p = p.parent
|
||||
return chain
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- analysis
|
||||
|
||||
|
||||
def _analyze(src: str):
|
||||
tree = ast.parse(src)
|
||||
b = _Builder()
|
||||
b.run(tree)
|
||||
# Per-scope: unresolved load names + import targets it resolves to.
|
||||
unresolved: dict[str, set[str]] = {}
|
||||
targets_by_scope: dict[str, set[str]] = {}
|
||||
target_by_use: dict[tuple[str, str], set[str]] = {}
|
||||
for scope, name, _ln in b.uses:
|
||||
status, binds = _resolve(scope, name)
|
||||
if status == "unresolved":
|
||||
unresolved.setdefault(scope.qualname, set()).add(name)
|
||||
elif status == "import":
|
||||
tids = {bd.target for bd in binds if bd.target}
|
||||
targets_by_scope.setdefault(scope.qualname, set()).update(tids)
|
||||
target_by_use.setdefault((scope.qualname, name), set()).update(tids)
|
||||
# soft uses (annotations): contribute to "used" only, never "unresolved"
|
||||
for scope, name, _ln in b.soft_uses:
|
||||
status, binds = _resolve(scope, name)
|
||||
if status == "import":
|
||||
tids = {bd.target for bd in binds if bd.target}
|
||||
targets_by_scope.setdefault(scope.qualname, set()).update(tids)
|
||||
# module-level binding info for clash checks
|
||||
module = b.module
|
||||
module_imports = {
|
||||
n: bs
|
||||
for n, bs in module.bindings.items()
|
||||
if any(x.kind in ("import", "importfrom") for x in bs)
|
||||
}
|
||||
module_dup = {
|
||||
n
|
||||
for n, bs in module.bindings.items()
|
||||
if any(x.kind in ("import", "importfrom") for x in bs)
|
||||
and any(x.kind not in ("import", "importfrom") for x in bs)
|
||||
}
|
||||
# ambiguous: any scope where a name is bound by import AND non-import
|
||||
ambiguous: dict[str, set[str]] = {}
|
||||
|
||||
def walk_scopes(scope: Scope):
|
||||
for n, bs in scope.bindings.items():
|
||||
if any(x.kind in ("import", "importfrom") for x in bs) and any(
|
||||
x.kind not in ("import", "importfrom") for x in bs
|
||||
):
|
||||
ambiguous.setdefault(scope.qualname, set()).add(n)
|
||||
# scope tree isn't stored; approximate with module only.
|
||||
|
||||
walk_scopes(module)
|
||||
return {
|
||||
"unresolved": unresolved,
|
||||
"targets_by_scope": targets_by_scope,
|
||||
"target_by_use": target_by_use,
|
||||
"module_import_targets": {
|
||||
n: {x.target for x in bs if x.target} for n, bs in module_imports.items()
|
||||
},
|
||||
"module_dup": module_dup,
|
||||
"ambiguous": ambiguous,
|
||||
}
|
||||
|
||||
|
||||
def _git_show(ref: str, path: str) -> str | None:
|
||||
try:
|
||||
return subprocess.run(
|
||||
["git", "show", f"{ref}:{path}"], capture_output = True, text = True, check = True
|
||||
).stdout
|
||||
except subprocess.CalledProcessError:
|
||||
return None
|
||||
|
||||
|
||||
def compare(before_src: str, after_src: str, path: str) -> list[tuple[str, str]]:
|
||||
"""Return list of (severity, message). severity in BLOCKER/WARN/INFO.
|
||||
|
||||
Blocker signals (precise, no relocation false-positives):
|
||||
UNRESOLVED-NEW - a load became undefined (dangling alias / removed import).
|
||||
NEW-UNUSED-HOIST - a module-level import added by this change is resolved by
|
||||
NO load (un-normalized alias or wrong rename target).
|
||||
TARGET-CHANGED - same (scope, name) load resolves to a different import
|
||||
target before vs after (a same-name re-point).
|
||||
"""
|
||||
a = _analyze(before_src)
|
||||
b = _analyze(after_src)
|
||||
findings: list[tuple[str, str]] = []
|
||||
|
||||
def used_targets(analysis) -> set[str]:
|
||||
out: set[str] = set()
|
||||
for tids in analysis["targets_by_scope"].values():
|
||||
out |= tids
|
||||
return out
|
||||
|
||||
before_used = used_targets(a)
|
||||
after_used = used_targets(b)
|
||||
before_module_targets: set[str] = set()
|
||||
for tids in a["module_import_targets"].values():
|
||||
before_module_targets |= tids
|
||||
after_module_targets: set[str] = set()
|
||||
for tids in b["module_import_targets"].values():
|
||||
after_module_targets |= tids
|
||||
added_module_targets = after_module_targets - before_module_targets
|
||||
|
||||
# 1. UNRESOLVED-NEW
|
||||
for scope, names in b["unresolved"].items():
|
||||
new = names - a["unresolved"].get(scope, set())
|
||||
for n in sorted(new):
|
||||
findings.append(
|
||||
(
|
||||
"BLOCKER",
|
||||
f"{path}: UNRESOLVED-NEW '{n}' in scope {scope} "
|
||||
f"(undefined after change -> dangling alias / removed import)",
|
||||
)
|
||||
)
|
||||
|
||||
# 2. HOISTED-IMPORT-UNUSED (core botched-hoist / wrong-rename signal)
|
||||
# A module-level import in AFTER that NO load resolves to, that was either
|
||||
# newly added by this change OR actually used before. Excludes relocation
|
||||
# (import removed) and stable pre-existing re-exports.
|
||||
for n, tids in b["module_import_targets"].items():
|
||||
if tids & after_used:
|
||||
continue # resolved -> fine
|
||||
# `from __future__ import ...` is a compiler directive, not a runtime
|
||||
# binding: the name (`annotations`, ...) is never loaded, so it can never
|
||||
# "resolve" to a use. Skip it so a legitimately-added future import
|
||||
# (e.g. `annotations` for lazy PEP 604 `X | None` on py3.9) is not flagged.
|
||||
if all(t.startswith("from:__future__:") for t in tids):
|
||||
continue
|
||||
newly_added = bool(tids - before_module_targets)
|
||||
was_used_before = bool(tids & before_used)
|
||||
if newly_added or was_used_before:
|
||||
why = (
|
||||
"added but unused"
|
||||
if newly_added
|
||||
else "was used before, now unused (references re-pointed)"
|
||||
)
|
||||
findings.append(
|
||||
(
|
||||
"BLOCKER",
|
||||
f"{path}: HOISTED-IMPORT-UNUSED '{n}' ({sorted(tids)}) "
|
||||
f"{why} -> un-normalized alias or wrong rename target?",
|
||||
)
|
||||
)
|
||||
|
||||
# 3. TARGET-CHANGED (same scope+name resolves to a different import target)
|
||||
# Only a *swap* is dangerous: a BEFORE target that is no longer reachable in
|
||||
# AFTER means a reference was silently re-pointed. A pure superset growth
|
||||
# (tbefore <= tafter) is the benign `import pkg.subA` + `import pkg.subB`
|
||||
# case: both statements bind the same top-level name `pkg` to the same
|
||||
# package object and only *add* submodule attributes (e.g. adding
|
||||
# `import urllib.error` next to `import urllib.request`). Nothing the name
|
||||
# resolved to before is lost, so no reference is re-pointed -- skip it.
|
||||
#
|
||||
# A deliberate *relocation* is also benign and must not block: when a name
|
||||
# keeps its spelling but its import source is moved A -> B in THIS diff (the
|
||||
# old `from A import x` is removed at module level and a new `from B import x`
|
||||
# is added), the swap is intentional, not a silent re-point to a pre-existing
|
||||
# different object. This mirrors the relocation tolerance already applied to
|
||||
# TARGET-MISSING. The dangerous case -- the name now resolving to a target
|
||||
# that already existed before (shadow/clash) -- is NOT exempted.
|
||||
removed_module_targets = before_module_targets - after_module_targets
|
||||
for key, tafter in b["target_by_use"].items():
|
||||
tbefore = a["target_by_use"].get(key)
|
||||
if tbefore and tbefore != tafter and (tbefore - tafter):
|
||||
lost = tbefore - tafter
|
||||
gained = tafter - tbefore
|
||||
relocated = lost <= removed_module_targets and gained <= added_module_targets
|
||||
if relocated:
|
||||
continue
|
||||
findings.append(
|
||||
(
|
||||
"BLOCKER",
|
||||
f"{path}: TARGET-CHANGED name '{key[1]}' in {key[0]} "
|
||||
f"{sorted(tbefore)} -> {sorted(tafter)} (rename re-points module)",
|
||||
)
|
||||
)
|
||||
|
||||
# 4. MODULE-DUP-IMPORT introduced
|
||||
for n in sorted(b["module_dup"] - a["module_dup"]):
|
||||
findings.append(
|
||||
(
|
||||
"WARN",
|
||||
f"{path}: MODULE-DUP-IMPORT '{n}' bound by import AND non-import "
|
||||
f"at module level (possible clash)",
|
||||
)
|
||||
)
|
||||
|
||||
# 5. AMBIGUOUS-BIND introduced (module scope)
|
||||
for scope, names in b["ambiguous"].items():
|
||||
new = names - a["ambiguous"].get(scope, set())
|
||||
for n in sorted(new):
|
||||
findings.append(("WARN", f"{path}: AMBIGUOUS-BIND '{n}' import+non-import in {scope}"))
|
||||
|
||||
# 6. TARGET-MISSING (informational): a scope stopped resolving to an import
|
||||
# target. Real bugs are covered above; remaining cases are relocated code.
|
||||
for scope, tbefore in a["targets_by_scope"].items():
|
||||
tafter = b["targets_by_scope"].get(scope, set())
|
||||
for t in sorted(tbefore - tafter):
|
||||
relocated = (
|
||||
""
|
||||
if t in added_module_targets
|
||||
else " [target not re-added here -> likely relocated/deleted]"
|
||||
)
|
||||
findings.append(("INFO", f"{path}: TARGET-MISSING {t} in scope {scope}{relocated}"))
|
||||
return findings
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- self-test
|
||||
|
||||
_SELF_TESTS = {
|
||||
"dangling_alias": (
|
||||
# before: inline aliased import, used as _b
|
||||
"import os\ndef f():\n import glob as _b\n return _b.glob('*')\n",
|
||||
# after: hoisted to canonical, but reference NOT normalized -> _b dangles
|
||||
"import os\nimport glob\ndef f():\n return _b.glob('*')\n",
|
||||
"BLOCKER",
|
||||
),
|
||||
"rename_clash": (
|
||||
# before: _b is a deliberate alias; `b` already means something else
|
||||
"import re as _b\nb = 123\ndef f():\n return _b.compile('x'), b\n",
|
||||
# after: someone normalized _b -> b ; now f().b is the int, re is lost
|
||||
"import re\nb = 123\ndef f():\n return b.compile('x'), b\n",
|
||||
"BLOCKER", # TARGET-MISSING from:.. or import:re in f
|
||||
),
|
||||
"clean_rename": (
|
||||
"def f():\n import glob as _g\n return _g.glob('*')\n",
|
||||
"import glob\ndef f():\n return glob.glob('*')\n",
|
||||
None, # expect NO blocker
|
||||
),
|
||||
"clean_dedup_redundant": (
|
||||
"import sys\ndef f():\n import sys\n return sys.argv\n",
|
||||
"import sys\ndef f():\n return sys.argv\n",
|
||||
None,
|
||||
),
|
||||
"from_import_dangling": (
|
||||
# from-import alias left un-normalized
|
||||
"def f():\n from importlib.metadata import version as _v\n return _v('x')\n",
|
||||
"from importlib.metadata import version\ndef f():\n return _v('x')\n",
|
||||
"BLOCKER",
|
||||
),
|
||||
"local_var_clash": (
|
||||
# _b renamed to b, but b is a LOCAL var in f -> import silently unused
|
||||
"def f(b):\n import re as _b\n return _b.compile(b)\n",
|
||||
"import re\ndef f(b):\n return b.compile(b)\n", # 'b' is the param, not the module
|
||||
"BLOCKER",
|
||||
),
|
||||
"substring_safe": (
|
||||
# correct _copy->copy rename while config_copy var exists: NO false positive
|
||||
"def f(config):\n"
|
||||
" import copy as _copy\n"
|
||||
" config_copy = _copy.deepcopy(config)\n"
|
||||
" return config_copy\n",
|
||||
"import copy\n"
|
||||
"def f(config):\n"
|
||||
" config_copy = copy.deepcopy(config)\n"
|
||||
" return config_copy\n",
|
||||
None,
|
||||
),
|
||||
"attr_access_not_a_use": (
|
||||
# x._b is attribute access, not a use of name _b; removing import _b is fine
|
||||
"import os\ndef f(x):\n import sys as _b\n return x._b + _b.argv[0]\n",
|
||||
"import os\nimport sys\ndef f(x):\n return x._b + sys.argv[0]\n",
|
||||
None,
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _self_test() -> int:
|
||||
ok = True
|
||||
for name, (before, after, expect) in _SELF_TESTS.items():
|
||||
findings = compare(before, after, f"<{name}>")
|
||||
blockers = [m for sev, m in findings if sev == "BLOCKER"]
|
||||
got = "BLOCKER" if blockers else None
|
||||
passed = got == expect
|
||||
ok = ok and passed
|
||||
print(f"[{'PASS' if passed else 'FAIL'}] {name}: expect={expect} got={got}")
|
||||
for sev, m in findings:
|
||||
print(f" ({sev}) {m}")
|
||||
print("\nSELF-TEST:", "ALL PASS" if ok else "FAILURES")
|
||||
return 0 if ok else 1
|
||||
|
||||
|
||||
def _pyflakes_undefined(path: str) -> set[str] | None:
|
||||
"""Return the set of names pyflakes reports as 'undefined name' for `path`,
|
||||
or None if pyflakes failed to run/parse the file."""
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
[sys.executable, "-m", "pyflakes", path], capture_output = True, text = True
|
||||
)
|
||||
except Exception:
|
||||
return None
|
||||
if "syntax error" in (proc.stdout + proc.stderr).lower():
|
||||
return None
|
||||
names = set()
|
||||
for line in proc.stdout.splitlines():
|
||||
m = _re_mod.search(r"undefined name '([^']+)'", line)
|
||||
if m:
|
||||
names.add(m.group(1))
|
||||
return names
|
||||
|
||||
|
||||
def audit_files(paths: list[str]) -> int:
|
||||
"""Single-version robustness audit: confirm the analyzer doesn't crash, then
|
||||
cross-check its 'unresolved' names against pyflakes. A name the resolver flags
|
||||
that pyflakes accepts is a tool false positive."""
|
||||
n_files = n_err = n_fp = n_syntax = 0
|
||||
fp_detail: dict[str, set[str]] = {}
|
||||
err_detail: dict[str, str] = {}
|
||||
for path in paths:
|
||||
n_files += 1
|
||||
try:
|
||||
src = open(path, encoding = "utf-8").read()
|
||||
except Exception as e: # unreadable
|
||||
n_err += 1
|
||||
err_detail[path] = f"read: {e}"
|
||||
continue
|
||||
try:
|
||||
res = _analyze(src)
|
||||
except SyntaxError:
|
||||
n_syntax += 1
|
||||
continue
|
||||
except Exception as e: # analyzer crash -> robustness bug
|
||||
n_err += 1
|
||||
err_detail[path] = f"{type(e).__name__}: {e}"
|
||||
continue
|
||||
tool_unresolved = set()
|
||||
for names in res["unresolved"].values():
|
||||
tool_unresolved |= names
|
||||
if not tool_unresolved:
|
||||
continue
|
||||
pf = _pyflakes_undefined(path)
|
||||
if pf is None:
|
||||
continue # pyflakes couldn't adjudicate; skip cross-check
|
||||
false_pos = tool_unresolved - pf
|
||||
if false_pos:
|
||||
n_fp += 1
|
||||
fp_detail[path] = false_pos
|
||||
print(f"audited files : {n_files}")
|
||||
print(f"syntax-skipped : {n_syntax}")
|
||||
print(f"analyzer errors : {n_err}")
|
||||
for p, e in sorted(err_detail.items()):
|
||||
print(f" ERROR {p}: {e}")
|
||||
print(f"false-positive files: {n_fp} (resolver flagged a name pyflakes accepts)")
|
||||
for p, names in sorted(fp_detail.items()):
|
||||
print(f" FP {p}: {sorted(names)}")
|
||||
ok = n_err == 0 and n_fp == 0
|
||||
print(
|
||||
"\nAUDIT:",
|
||||
"ROBUST (no crashes, no false positives vs pyflakes)" if ok else "NEEDS WORK (see above)",
|
||||
)
|
||||
return 0 if ok else 1
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--before", default = "origin/main")
|
||||
ap.add_argument("--after", default = "HEAD")
|
||||
ap.add_argument("--self-test", action = "store_true")
|
||||
ap.add_argument(
|
||||
"--audit",
|
||||
action = "store_true",
|
||||
help = "single-version robustness audit on filesystem paths",
|
||||
)
|
||||
ap.add_argument("files", nargs = "*")
|
||||
args = ap.parse_args()
|
||||
|
||||
if args.self_test:
|
||||
return _self_test()
|
||||
if args.audit:
|
||||
return audit_files(args.files)
|
||||
|
||||
any_blocker = False
|
||||
for path in args.files:
|
||||
before = _git_show(args.before, path)
|
||||
after = _git_show(args.after, path)
|
||||
if after is None:
|
||||
print(f"SKIP {path}: not found at {args.after}")
|
||||
continue
|
||||
if before is None:
|
||||
before = "" # new file
|
||||
findings = compare(before, after, path)
|
||||
blockers = [f for f in findings if f[0] == "BLOCKER"]
|
||||
warns = [f for f in findings if f[0] == "WARN"]
|
||||
infos = [f for f in findings if f[0] == "INFO"]
|
||||
status = "CLEAN" if not blockers and not warns else ("BLOCKERS" if blockers else "WARNINGS")
|
||||
print(f"\n=== {path}: {status} ===")
|
||||
for sev, m in blockers + warns + infos:
|
||||
print(f" [{sev}] {m}")
|
||||
any_blocker = any_blocker or bool(blockers)
|
||||
print("\nOVERALL:", "FAIL (blockers found)" if any_blocker else "PASS (no blockers)")
|
||||
return 1 if any_blocker else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user