chore: import upstream snapshot with attribution
Lockfile supply-chain audit / lockfile supply-chain audit (push) Has been cancelled
Windows Studio GGUF CI / GPU prebuilt resolves without Visual Studio (push) Has been cancelled
Windows Studio GGUF CI / setup.ps1 unit tests (VS 2026 / CMake guard) (push) Has been cancelled
Windows Studio GGUF CI / real-VS detection (VS 2022) (push) Has been cancelled
Windows Studio GGUF CI / real-VS detection (VS 2026) (push) Has been cancelled
Windows Studio GGUF CI / VC++ runtime detect + install round-trip (windows-2025-vs2026) (push) Has been cancelled
Windows Studio GGUF CI / VC++ runtime detect + install round-trip (windows-latest) (push) Has been cancelled
Windows Studio Update CI / Studio Updating Tests (push) Has been cancelled
Wheel CI / Wheel build + content sanity + import smoke (push) Has been cancelled
Lint CI / Source lint (Python + shell + YAML + JSON + safety nets) (push) Has been cancelled
MLX CI on Mac M1 / dispatch (push) Has been cancelled
Security audit / advisory audit (pip + npm + cargo) (push) Has been cancelled
Security audit / pip scan-packages :: extras (push) Has been cancelled
Security audit / pip scan-packages :: studio (push) Has been cancelled
Security audit / pip scan-packages :: hf-stack (push) Has been cancelled
Security audit / npm scan-packages (Studio frontend tarballs) (push) Has been cancelled
Security audit / workflow-trigger lint (pull_request_target / cache-poisoning) (push) Has been cancelled
Security audit / pytest tests/security (push) Has been cancelled
Security audit / npm provenance + new install-script diff (push) Has been cancelled
Studio API CI / Studio API & Auth Tests (push) Has been cancelled
Backend CI / (Python 3.10) (push) Has been cancelled
Backend CI / (Python 3.11) (push) Has been cancelled
Backend CI / (Python 3.12) (push) Has been cancelled
Backend CI / (Python 3.13) (push) Has been cancelled
Backend CI / Repo tests (CPU) (push) Has been cancelled
Frontend CI / Frontend build + bundle sanity (push) Has been cancelled
Studio GGUF CI / OpenAI, Anthropic API tests (push) Has been cancelled
Studio GGUF CI / Tool calling Tests (push) Has been cancelled
Studio GGUF CI / JSON, images (push) Has been cancelled
Mac Studio GGUF CI / OpenAI, Anthropic API tests (push) Has been cancelled
Mac Studio GGUF CI / Tool calling Tests (push) Has been cancelled
Mac Studio GGUF CI / JSON, images (push) Has been cancelled
Mac Studio Install Matrix CI / Install + load (macos-14) (push) Has been cancelled
Mac Studio Install Matrix CI / Install + load (macos-15) (push) Has been cancelled
Mac Studio Install Matrix CI / Install + load (macos-26) (push) Has been cancelled
Mac Studio Install Matrix CI / Install + load (macos-15-intel) (push) Has been cancelled
Mac Studio API CI / Studio API & Auth Tests (push) Has been cancelled
Mac Studio Install Matrix CI / Install + load (macos-26-intel) (push) Has been cancelled
Mac Studio UI CI / Chat UI Tests (push) Has been cancelled
Studio Tauri CI / Tauri Linux debug build (no codesign) (push) Has been cancelled
Mac Studio Update CI / Studio Updating Tests (push) Has been cancelled
Studio UI CI / Chat UI Tests (push) Has been cancelled
Windows Studio API CI / Studio API & Auth Tests (push) Has been cancelled
Windows Studio UI CI / Chat UI Tests (push) Has been cancelled
Studio Update CI / Studio Updating Tests (push) Has been cancelled
Core / Core (HF=default + TRL=default) (push) Has been cancelled
Core / Core (HF=4.57.6 + TRL<1) (push) Has been cancelled
Core / Core (HF=latest + TRL=latest) (push) Has been cancelled
Core / llama.cpp build + smoke (push) Has been cancelled
Windows Studio GGUF CI / OpenAI, Anthropic API tests (push) Has been cancelled
Windows Studio GGUF CI / Tool calling Tests (push) Has been cancelled
Windows Studio GGUF CI / JSON, images (push) Has been cancelled
Windows Studio GGUF CI / Studio install + inference without Visual Studio (push) Has been cancelled
Studio export capability / capability (macos-latest) (push) Has been cancelled
Studio export capability / capability (ubuntu-latest) (push) Has been cancelled
Studio export capability / capability (windows-latest) (push) Has been cancelled
Cross-platform parity / parity (macos-latest) (push) Has been cancelled
Cross-platform parity / parity (windows-latest) (push) Has been cancelled
Scorecard supply-chain security / Scorecard analysis (push) Has been cancelled
Studio load-orchestrator CI / test (push) Has been cancelled

This commit is contained in:
wehub-resource-sync
2026-07-13 12:59:56 +08:00
commit e93507a09c
2027 changed files with 674044 additions and 0 deletions
File diff suppressed because it is too large Load Diff
+242
View File
@@ -0,0 +1,242 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
"""Diff two `package-lock.json` files and flag NEW install-script deps.
A `"hasInstallScript": true` package runs preinstall/install/postinstall
hooks on every `npm ci` -- the lever behind recent npm supply-chain
compromises (attacker publishes a malicious version of a trusted dep).
This refuses to land a newly-introduced install-script dep without a
maintainer eyeball; pre-existing ones are not re-flagged.
Supports lockfileVersion 1 (recursive `dependencies`) and 2/3 (flat
`packages` with `node_modules/.../node_modules/...` nesting). For each
new entry we best-effort fetch the registry metadata to recover the
postinstall command body; the finding is still emitted if unreachable.
Exit codes: 0 = none; 1 = one or more (on stderr); 2 = internal error.
"""
from __future__ import annotations
import argparse
import json
import sys
import urllib.error
import urllib.parse
import urllib.request
from pathlib import Path
REGISTRY_BASE = "https://registry.npmjs.org/"
REGISTRY_TIMEOUT_SECS = 5
CRITICAL = "CRITICAL"
HIGH = "HIGH"
class Finding:
__slots__ = ("severity", "name", "version", "kind", "detail")
def __init__(self, severity: str, name: str, version: str, kind: str, detail: str) -> None:
self.severity = severity
self.name = name
self.version = version
self.kind = kind
self.detail = detail
def __str__(self) -> str:
return (
f" [{self.severity}] {self.name}@{self.version}\n"
f" kind: {self.kind}\n"
f" detail: {self.detail}"
)
# Lockfile parsing.
def _strip_nm_prefix(key: str) -> str:
"""Convert a v2/v3 `packages` key into a bare package name (leaf after last `node_modules/`)."""
if not key:
return ""
# LAST node_modules/ segment so transitives map to their leaf name.
marker = "node_modules/"
idx = key.rfind(marker)
if idx == -1:
return key
return key[idx + len(marker) :]
def _collect_install_script_entries(lock: dict) -> dict[str, str]:
"""Return {name@version: name} for entries with hasInstallScript (v2/v3) or a lifecycle script (v1).
Keyed by name@version so dup copies at different versions aren't lost.
"""
seen: dict[str, str] = {}
version = lock.get("lockfileVersion")
# v2 / v3: flat `packages` map.
packages = lock.get("packages") or {}
for key, entry in packages.items():
if key == "" or not isinstance(entry, dict):
continue
if entry.get("link"):
continue
if not entry.get("hasInstallScript"):
continue
name = _strip_nm_prefix(key)
if not name:
continue
ver = entry.get("version") or "<unversioned>"
seen[f"{name}@{ver}"] = name
# v1 has no hasInstallScript flag; detect lifecycle scripts directly.
def _walk_v1(deps: dict, depth: int = 0) -> None:
if depth > 64 or not isinstance(deps, dict):
return
for name, entry in deps.items():
if not isinstance(entry, dict):
continue
scripts = entry.get("scripts") or {}
lifecycle = any(
isinstance(scripts, dict) and scripts.get(hook)
for hook in ("preinstall", "install", "postinstall")
)
if lifecycle:
ver = entry.get("version") or "<unversioned>"
seen[f"{name}@{ver}"] = name
_walk_v1(entry.get("dependencies"), depth = depth + 1)
if version == 1 or "dependencies" in lock:
_walk_v1(lock.get("dependencies") or {})
return seen
def _load_lockfile(path: Path) -> dict:
if not path.exists():
raise FileNotFoundError(f"lockfile not found: {path}")
try:
return json.loads(path.read_text(encoding = "utf-8"))
except json.JSONDecodeError as exc:
raise ValueError(f"{path}: not valid JSON: {exc}") from exc
# Registry lookup for the postinstall command body (best-effort).
def _fetch_registry_scripts(name: str, version: str) -> dict[str, str] | None:
"""Return {hook: command} for lifecycle hooks in registry metadata; None on any error (never raises)."""
safe_name = urllib.parse.quote(name, safe = "@/")
url = f"{REGISTRY_BASE}{safe_name}/{urllib.parse.quote(version)}"
try:
with urllib.request.urlopen(url, timeout = REGISTRY_TIMEOUT_SECS) as resp:
body = resp.read()
except (urllib.error.URLError, OSError, ValueError, TimeoutError):
return None
try:
meta = json.loads(body)
except json.JSONDecodeError:
return None
scripts = meta.get("scripts") or {}
if not isinstance(scripts, dict):
return None
keep = {}
for hook in ("preinstall", "install", "postinstall"):
cmd = scripts.get(hook)
if isinstance(cmd, str) and cmd.strip():
keep[hook] = cmd
return keep or None
# Diff.
def diff_new_install_scripts(base_lock: dict, head_lock: dict) -> list[Finding]:
base = _collect_install_script_entries(base_lock)
head = _collect_install_script_entries(head_lock)
findings: list[Finding] = []
for key in sorted(head):
if key in base:
continue # pre-existing install-script dep; not in scope
name = head[key]
version = key[len(name) + 1 :] if key.startswith(name + "@") else "<unversioned>"
scripts = _fetch_registry_scripts(name, version)
if scripts:
detail = "; ".join(f"{h}={cmd!r}" for h, cmd in scripts.items())
else:
detail = (
"newly added with hasInstallScript=true; registry "
"metadata unreachable -- inspect the package's "
"scripts.{preinstall,install,postinstall} manually"
)
findings.append(
Finding(
severity = CRITICAL,
name = name,
version = version,
kind = "new-install-script",
detail = detail,
)
)
return findings
# CLI.
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
description = (
"Diff two package-lock.json files and refuse any newly-added install-script dep."
),
)
parser.add_argument(
"--base",
required = True,
help = "Path to the BASE package-lock.json (e.g. main branch).",
)
parser.add_argument(
"--head",
required = True,
help = "Path to the HEAD package-lock.json (this PR).",
)
args = parser.parse_args(argv)
try:
base_lock = _load_lockfile(Path(args.base))
head_lock = _load_lockfile(Path(args.head))
except (FileNotFoundError, ValueError) as exc:
print(f"[install-script-diff] ERROR: {exc}", file = sys.stderr)
return 2
findings = diff_new_install_scripts(base_lock, head_lock)
if not findings:
print(
"[install-script-diff] OK: no newly-added install-script "
"dependencies between base and head",
flush = True,
)
return 0
print(
f"\n[install-script-diff] FAIL: {len(findings)} newly-added "
f"install-script dependency(ies):\n",
file = sys.stderr,
)
for f in findings:
print(str(f), file = sys.stderr)
print(file = sys.stderr)
print(
"[install-script-diff] Refusing to proceed. Every new "
"install-script dep is a postinstall lifecycle hook that "
"would run on the next `npm ci`. Review each finding above, "
"confirm the maintainer + version, and re-run.",
file = sys.stderr,
)
return 1
if __name__ == "__main__":
sys.exit(main())
File diff suppressed because it is too large Load Diff
+9
View File
@@ -0,0 +1,9 @@
# Do not modify this file directly; it is generated by extract_colabx_testing_tarballs.sh via
# $ (lsb_release -ds;python --version;) > os-info-gpu.txt
# Be aware that this list does not necessarily reflect the current state of the
# staging or production container, but rather the state as of the most recent
# submitted CL where extract_colabx_testing_tarballs.sh was run.
Ubuntu 22.04.5 LTS
Python 3.12.13
R version 4.5.3 (2026-03-11) -- "Reassured Reassurer"
julia version 1.12.6
+731
View File
@@ -0,0 +1,731 @@
# Do not modify this file directly; it is generated by extract_colabx_testing_tarballs.sh via
# $ python3 -m pip freeze
# Be aware that this list does not necessarily reflect the current state of the
# staging or production container, but rather the state as of the most recent
# submitted CL where extract_colabx_testing_tarballs.sh was run.
absl-py==1.4.0
accelerate==1.13.0
access==1.1.10.post3
affine==2.4.0
aiofiles==24.1.0
aiohappyeyeballs==2.6.1
aiohttp==3.13.5
aiosignal==1.4.0
aiosqlite==0.22.1
alabaster==1.0.0
albucore==0.0.24
albumentations==2.0.8
ale-py==0.11.2
alembic==1.18.4
altair==5.5.0
annotated-doc==0.0.4
annotated-types==0.7.0
antlr4-python3-runtime==4.9.3
anyio==4.13.0
anywidget==0.9.21
apsw==3.53.0.0
apswutils==0.1.2
argon2-cffi==25.1.0
argon2-cffi-bindings==25.1.0
array_record==0.8.3
arrow==1.4.0
arviz==0.22.0
astropy==7.2.0
astropy-iers-data==0.2026.4.20.0.58.15
astunparse==1.6.3
atpublic==5.1
attrs==26.1.0
audioread==3.1.0
Authlib==1.6.11
autograd==1.8.0
babel==2.18.0
backcall==0.2.0
beartype==0.22.9
beautifulsoup4==4.13.5
betterproto==2.0.0b6
bigframes==2.39.0
bigquery-magics==0.14.0
bleach==6.3.0
blinker==1.9.0
blis==1.3.3
blobfile==3.2.0
blosc2==4.1.2
bokeh==3.8.2
Bottleneck==1.4.2
bqplot==0.12.45
branca==0.8.2
brotli==1.2.0
CacheControl==0.14.4
cachetools==6.2.6
catalogue==2.0.10
certifi==2026.4.22
cffi==2.0.0
chardet==5.2.0
charset-normalizer==3.4.7
clarabel==0.11.1
click==8.3.3
click-plugins==1.1.1.2
cligj==0.7.2
cloudpathlib==0.23.0
cloudpickle==3.1.2
cmake==3.31.10
cmdstanpy==1.3.0
colorcet==3.1.0
colorlover==0.3.0
community==1.0.0b1
confection==1.3.3
cons==0.4.7
contourpy==1.3.3
cramjam==2.11.0
cryptography==43.0.3
cucim-cu12 @ https://pypi.nvidia.com/cucim-cu12/cucim_cu12-26.2.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
cuda-bindings==12.9.4
cuda-core==0.3.2
cuda-pathfinder==1.5.3
cuda-python==12.9.4
cuda-toolkit==12.8.1
cudf-cu12==26.2.1
cudf-polars-cu12==26.2.1
cufflinks==0.17.3
cuml-cu12==26.2.0
cupy-cuda12x==14.0.1
curl_cffi==0.15.0
cuvs-cu12 @ https://pypi.nvidia.com/cuvs-cu12/cuvs_cu12-26.2.0-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
cvxopt==1.3.2
cvxpy==1.6.7
cycler==0.12.1
cyipopt==1.5.0
cymem==2.0.13
Cython==3.0.12
dask==2026.1.1
dask-cuda==26.2.0
dask-cudf-cu12==26.2.1
dataproc-spark-connect==1.1.0
datasets==4.0.0
db-dtypes==1.5.1
dbus-python==1.2.18
debugpy==1.8.15
decorator==4.4.2
defusedxml==0.7.1
deprecation==2.1.0
diffusers==0.37.1
dill==0.3.8
distributed==2026.1.1
distributed-ucxx-cu12==0.48.0
distro==1.9.0
dlib==19.24.6
dm-tree==0.1.10
docstring_parser==0.18.0
docutils==0.21.2
dopamine_rl==4.1.2
duckdb==1.3.2
earthengine-api==1.7.22
easydict==1.13
editdistance==0.8.1
eerepr==0.1.2
einops==0.8.2
en_core_web_sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl#sha256=1932429db727d4bff3deed6b34cfc05df17794f4a52eeb26cf8928f7c1a0fb85
entrypoints==0.4
esda==2.9.0
et_xmlfile==2.0.0
etils==1.14.0
etuples==0.3.10
Farama-Notifications==0.0.4
fastai==2.8.7
fastapi==0.136.1
fastcore==1.12.42
fastdownload==0.0.7
fastjsonschema==2.21.2
fastlite==0.2.4
fastprogress==1.1.5
fasttransform==0.0.2
ffmpy==1.0.0
filelock==3.29.0
fiona==1.10.1
firebase-admin==6.9.0
Flask==3.1.3
flatbuffers==25.12.19
flax==0.11.2
folium==0.20.0
fonttools==4.62.1
fqdn==1.5.1
frozendict==2.4.7
frozenlist==1.8.0
fsspec==2025.3.0
future==1.0.0
gast==0.7.0
gcsfs==2025.3.0
GDAL==3.8.4
gdown==5.2.2
geemap==0.37.2
geocoder==1.38.1
geographiclib==2.1
geopandas==1.1.3
geopy==2.4.1
giddy==2.3.6
gin-config==0.5.0
gitdb==4.0.12
GitPython==3.1.47
glob2==0.7
google==3.0.0
google-adk==1.29.0
google-ai-generativelanguage==0.6.15
google-api-core==2.30.3
google-api-python-client==2.194.0
google-auth==2.47.0
google-auth-httplib2==0.3.1
google-auth-oauthlib==1.3.1
google-cloud-aiplatform==1.148.1
google-cloud-appengine-logging==1.9.0
google-cloud-audit-log==0.5.0
google-cloud-bigquery==3.41.0
google-cloud-bigquery-connection==1.21.0
google-cloud-bigquery-storage==2.37.0
google-cloud-bigtable==2.36.0
google-cloud-core==2.5.1
google-cloud-dataplex==2.18.0
google-cloud-dataproc==5.27.0
google-cloud-datastore==2.24.0
google-cloud-discoveryengine==0.13.12
google-cloud-firestore==2.27.0
google-cloud-functions==1.23.0
google-cloud-iam==2.22.0
google-cloud-language==2.20.0
google-cloud-logging==3.15.0
google-cloud-monitoring==2.30.0
google-cloud-pubsub==2.37.0
google-cloud-resource-manager==1.17.0
google-cloud-secret-manager==2.27.0
google-cloud-spanner==3.65.0
google-cloud-speech==2.38.0
google-cloud-storage==3.10.1
google-cloud-trace==1.19.0
google-cloud-translate==3.26.0
google-colab @ file:///colabtools/dist/google_colab-1.0.0.tar.gz
google-crc32c==1.8.0
google-genai==1.68.0
google-generativeai==0.8.6
google-pasta==0.2.0
google-resumable-media==2.8.2
googleapis-common-protos==1.74.0
googledrivedownloader==1.1.0
gradio==5.50.0
gradio_client==1.14.0
grain==0.2.16
graphviz==0.21
greenlet==3.4.0
groovy==0.1.2
grpc-google-iam-v1==0.14.4
grpc-interceptor==0.15.4
grpcio==1.80.0
grpcio-status==1.71.2
grpclib==0.4.9
gspread==6.2.1
gspread-dataframe==4.0.0
gym==0.25.2
gym-notices==0.1.0
gymnasium==1.3.0
h11==0.16.0
h2==4.3.0
h5netcdf==1.8.1
h5py==3.16.0
hdbscan==0.8.42
hf-xet==1.4.3
highspy==1.14.0
holidays==0.95
holoviews==1.22.1
hpack==4.1.0
html5lib==1.1
httpcore==1.0.9
httpimport==1.4.1
httplib2==0.31.2
httptools==0.7.1
httpx==0.28.1
httpx-sse==0.4.3
huggingface_hub==1.11.0
humanize==4.15.0
hyperframe==6.1.0
hyperopt==0.2.7
ibis-framework==9.5.0
idna==3.13
ImageIO==2.37.3
imageio-ffmpeg==0.6.0
imagesize==2.0.0
imbalanced-learn==0.14.1
immutabledict==4.3.1
importlib_metadata==8.7.1
importlib_resources==7.1.0
imutils==0.5.4
inequality==1.1.2
inflect==7.5.0
iniconfig==2.3.0
intel-cmplr-lib-ur==2025.3.3
intel-openmp==2025.3.3
ipyevents==2.0.4
ipyfilechooser==0.6.0
ipykernel==6.17.1
ipyleaflet==0.20.0
ipyparallel==8.8.0
ipython==7.34.0
ipython-genutils==0.2.0
ipython-sql==0.5.0
ipywidgets==7.7.1
isoduration==20.11.0
itsdangerous==2.2.0
jaraco.classes==3.4.0
jaraco.context==6.1.2
jaraco.functools==4.4.0
jax==0.7.2
jax-cuda12-pjrt==0.7.2
jax-cuda12-plugin==0.7.2
jaxlib==0.7.2
jeepney==0.9.0
jieba==0.42.1
Jinja2==3.1.6
jiter==0.14.0
joblib==1.5.3
jsonpatch==1.33
jsonpickle==4.1.1
jsonpointer==3.1.1
jsonschema==4.26.0
jsonschema-specifications==2025.9.1
jupyter-console==6.6.3
jupyter-events==0.12.1
jupyter-leaflet==0.20.0
jupyter_client==7.4.9
jupyter_core==5.9.1
jupyter_kernel_gateway @ git+https://github.com/googlecolab/kernel_gateway@b134e9945df25c2dcb98ade9129399be10788671
jupyter_server==2.14.0
jupyter_server_terminals==0.5.4
jupyterlab_pygments==0.3.0
jupyterlab_widgets==3.0.16
jupytext==1.19.1
kaggle==2.0.2
kagglehub==1.0.0
kagglesdk==0.1.20
keras==3.13.2
keras-hub==0.26.0
keras-nlp==0.26.0
keyring==25.7.0
keyrings.google-artifactregistry-auth==1.1.2
kiwisolver==1.5.0
langchain==1.2.15
langchain-core==1.3.1
langgraph==1.1.9
langgraph-checkpoint==4.0.2
langgraph-prebuilt==1.0.10
langgraph-sdk==0.3.13
langsmith==0.7.34
lark==1.3.1
launchpadlib==1.10.16
lazr.restfulclient==0.14.4
lazr.uri==1.0.6
lazy-loader==0.5
libclang==18.1.1
libcudf-cu12==26.2.1
libcugraph-cu12==26.2.0
libcuml-cu12==26.2.0
libcuvs-cu12==26.2.0
libkvikio-cu12==26.2.0
libpysal==4.14.1
libraft-cu12==26.2.0
librmm-cu12==26.2.0
librosa==0.11.0
libucx-cu12==1.19.0
libucxx-cu12==0.48.0
lightgbm==4.6.0
linkify-it-py==2.1.0
llvmlite==0.43.0
locket==1.0.0
logical-unification==0.4.7
lxml==6.1.0
Mako==1.3.11
mapclassify==2.10.0
Markdown==3.10.2
markdown-it-py==4.0.0
MarkupSafe==3.0.3
matplotlib==3.10.0
matplotlib-inline==0.2.1
matplotlib-venn==1.1.2
mcp==1.27.0
mdit-py-plugins==0.5.0
mdurl==0.1.2
mgwr==2.2.1
miniKanren==1.0.5
missingno==0.5.2
mistune==3.2.0
mizani==0.13.5
mkl==2025.3.1
ml_dtypes==0.5.4
mlxtend==0.23.4
mmh3==5.2.1
momepy==0.11.0
more-itertools==10.8.0
moviepy==1.0.3
mpmath==1.3.0
msgpack==1.1.2
multidict==6.7.1
multipledispatch==1.0.0
multiprocess==0.70.16
multitasking==0.0.13
murmurhash==1.0.15
music21==9.9.1
namex==0.1.0
narwhals==2.20.0
natsort==8.4.0
nbclassic==1.3.3
nbclient==0.10.4
nbconvert==7.17.1
nbformat==5.10.4
ndindex==1.10.1
nest-asyncio==1.6.0
networkx==3.6.1
nibabel==5.4.2
nltk==3.9.1
notebook==6.5.7
notebook_shim==0.2.4
numba==0.60.0
numba-cuda==0.22.2
numexpr==2.14.1
numpy==2.0.2
nvidia-cublas-cu12==12.8.4.1
nvidia-cuda-cccl-cu12==12.9.27
nvidia-cuda-cupti-cu12==12.8.90
nvidia-cuda-nvcc-cu12==12.8.93
nvidia-cuda-nvrtc-cu12==12.8.93
nvidia-cuda-runtime-cu12==12.8.90
nvidia-cudnn-cu12==9.10.2.21
nvidia-cufft-cu12==11.3.3.83
nvidia-cufile-cu12==1.13.1.3
nvidia-curand-cu12==10.3.9.90
nvidia-cusolver-cu12==11.7.3.90
nvidia-cusparse-cu12==12.5.8.93
nvidia-cusparselt-cu12==0.7.1
nvidia-libnvcomp-cu12==5.1.0.21
nvidia-ml-py==13.595.45
nvidia-nccl-cu12==2.27.5
nvidia-nvimgcodec-cu12==0.7.0.11
nvidia-nvjitlink-cu12==12.8.93
nvidia-nvshmem-cu12==3.4.5
nvidia-nvtx-cu12==12.8.90
nvtx==0.2.15
nx-cugraph-cu12 @ https://pypi.nvidia.com/nx-cugraph-cu12/nx_cugraph_cu12-26.2.0-py3-none-any.whl
oauth2client==4.1.3
oauthlib==3.3.1
omegaconf==2.3.0
onemkl-license==2025.3.1
openai==2.32.0
opencv-contrib-python==4.13.0.92
opencv-python==4.13.0.92
opencv-python-headless==4.13.0.92
openpyxl==3.1.5
opentelemetry-api==1.38.0
opentelemetry-exporter-gcp-logging==1.11.0a0
opentelemetry-exporter-gcp-monitoring==1.11.0a0
opentelemetry-exporter-gcp-trace==1.11.0
opentelemetry-exporter-otlp-proto-common==1.38.0
opentelemetry-exporter-otlp-proto-http==1.38.0
opentelemetry-proto==1.38.0
opentelemetry-resourcedetector-gcp==1.11.0a0
opentelemetry-sdk==1.38.0
opentelemetry-semantic-conventions==0.59b0
opt_einsum==3.4.0
optax==0.2.8
optree==0.19.0
orbax-checkpoint==0.11.36
orjson==3.11.8
ormsgpack==1.12.2
osqp==1.1.1
overrides==7.7.0
packaging==26.1
pandas==2.2.2
pandas-datareader==0.10.0
pandas-gbq==0.30.0
pandas-stubs==2.2.2.240909
pandocfilters==1.5.1
panel==1.8.10
param==2.3.3
parso==0.8.6
parsy==2.2
partd==1.4.2
patsy==1.0.2
peewee==4.0.5
peft==0.19.1
pexpect==4.9.0
pickleshare==0.7.5
pillow==11.3.0
pip==24.1.2
platformdirs==4.9.6
plotly==5.24.1
plotnine==0.14.5
pluggy==1.6.0
plum-dispatch==2.8.0
pointpats==2.5.5
polars==1.35.2
polars-runtime-32==1.35.2
pooch==1.9.0
portpicker==1.5.2
preshed==3.0.13
prettytable==3.17.0
proglog==0.1.12
progressbar2==4.5.0
prometheus_client==0.25.0
promise==2.3
prompt_toolkit==3.0.52
propcache==0.4.1
prophet==1.3.0
proto-plus==1.27.2
protobuf==5.29.6
psutil==5.9.5
psycopg2==2.9.12
psygnal==0.15.1
ptyprocess==0.7.0
PuLP==3.3.0
py-cpuinfo==9.0.0
py4j==0.10.9.9
pyarrow==18.1.0
pyasn1==0.6.3
pyasn1_modules==0.4.2
pycairo==1.29.0
pycocotools==2.0.11
pycparser==3.0
pycryptodomex==3.23.0
pydantic==2.12.3
pydantic-settings==2.14.0
pydantic_core==2.41.4
pydata-google-auth==1.9.1
pydot==4.0.1
pydotplus==2.0.2
PyDrive2==1.21.3
pydub==0.25.1
pyerfa==2.0.1.5
pygame==2.6.1
pygit2==1.19.2
Pygments==2.20.0
PyGObject==3.48.2
pyiceberg==0.11.1
PyJWT==2.12.1
pylibcudf-cu12==26.2.1
pylibcugraph-cu12==26.2.0
pylibraft-cu12==26.2.0
pymc==5.28.4
pynndescent==0.6.0
pyogrio==0.12.1
pyomo==6.10.0
PyOpenGL==3.1.10
pyOpenSSL==24.2.1
pyparsing==3.3.2
pyperclip==1.11.0
pyproj==3.7.2
pyroaring==1.0.4
pysal==25.7
pyshp==3.0.3
PySocks==1.7.1
pyspark==4.0.2
pytensor==2.38.2
pytest==8.4.2
python-apt==0.0.0
python-box==7.4.1
python-dateutil==2.9.0.post0
python-dotenv==1.2.2
python-fasthtml==0.12.50
python-json-logger==4.1.0
python-louvain==0.16
python-multipart==0.0.26
python-slugify==8.0.4
python-snappy==0.7.3
python-utils==3.9.1
pytz==2025.2
pyviz_comms==3.0.6
PyWavelets==1.9.0
PyYAML==6.0.3
pyzmq==26.2.1
quantecon==0.11.2
raft-dask-cu12==26.2.0
rapids-dask-dependency==26.2.0
rapids-logger==0.2.3
rasterio==1.5.0
rasterstats==0.20.0
ratelim==0.1.6
referencing==0.37.0
regex==2025.11.3
requests==2.32.4
requests-oauthlib==2.0.0
requests-toolbelt==1.0.0
requirements-parser==0.9.0
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rfc3987-syntax==1.1.0
rich==13.9.4
rmm-cu12==26.2.0
roman-numerals==4.1.0
roman-numerals-py==4.1.0
rpds-py==0.30.0
rpy2==3.5.17
rsa==4.9.1
rtree==1.4.1
ruff==0.15.11
safehttpx==0.1.7
safetensors==0.7.0
scikit-image==0.25.2
scikit-learn==1.6.1
scipy==1.16.3
scooby==0.11.2
scs==3.2.11
seaborn==0.13.2
SecretStorage==3.5.0
segregation==2.5.4
semantic-version==2.10.0
Send2Trash==2.1.0
sentence-transformers==5.4.1
sentencepiece==0.2.1
sentry-sdk==2.58.0
setuptools==75.2.0
shap==0.51.0
shapely==2.1.2
shellingham==1.5.4
simple-parsing==0.1.8
simplejson==4.1.0
simsimd==6.5.16
six==1.17.0
sklearn-compat==0.1.5
sklearn-pandas==2.2.0
slicer==0.0.8
smart_open==7.6.0
smmap==5.0.3
sniffio==1.3.1
snowballstemmer==3.0.1
sortedcontainers==2.4.0
soundfile==0.13.1
soupsieve==2.8.3
soxr==1.0.0
spacy==3.8.14
spacy-legacy==3.0.12
spacy-loggers==1.0.5
spaghetti==1.7.6
spanner-graph-notebook==1.1.10
spglm==1.1.0
Sphinx==8.2.3
sphinxcontrib-applehelp==2.0.0
sphinxcontrib-devhelp==2.0.0
sphinxcontrib-htmlhelp==2.1.0
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==2.0.0
sphinxcontrib-serializinghtml==2.0.0
spint==1.0.7
splot==1.1.7
spopt==0.7.0
spreg==1.9.0
SQLAlchemy==2.0.49
sqlalchemy-spanner==1.17.3
sqlglot==25.20.2
sqlparse==0.5.5
srsly==2.5.3
sse-starlette==3.3.4
stanio==0.5.1
starlette==0.52.1
statsmodels==0.14.6
strictyaml==1.7.3
stringzilla==4.6.0
stumpy==1.13.0
sympy==1.14.0
tables==3.10.2
tabulate==0.9.0
tbb==2022.3.1
tblib==3.2.2
tcmlib==1.4.1
tenacity==9.1.4
tensorboard==2.20.0
tensorboard-data-server==0.7.2
tensorflow==2.20.0
tensorflow-datasets==4.9.9
tensorflow-hub==0.16.1
tensorflow-metadata==1.17.3
tensorflow-probability==0.25.0
tensorflow-text==2.20.1
tensorstore==0.1.82
termcolor==3.3.0
terminado==0.18.1
text-unidecode==1.3
textblob==0.19.0
tf-slim==1.1.0
tf_keras==2.20.0
thinc==8.3.13
threadpoolctl==3.6.0
tifffile==2026.4.11
tiktoken==0.12.0
timm==1.0.26
tinycss2==1.4.0
tobler==0.14.0
tokenizers==0.22.2
toml==0.10.2
tomlkit==0.13.3
toolz==0.12.1
torch==2.10.0+cu128
torchao==0.10.0
torchaudio==2.10.0+cu128
torchcodec==0.10.0+cu128
torchdata==0.11.0
torchsummary==1.5.1
torchtune==0.6.1
torchvision==0.25.0+cu128
tornado==6.5.1
tqdm==4.67.3
traitlets==5.7.1
traittypes==0.2.3
transformers==5.0.0
treelite==4.7.0
treescope==0.1.10
triton==3.6.0
tsfresh==0.21.1
tweepy==4.16.0
typeguard==4.5.1
typer==0.24.2
typer-slim==0.24.0
types-pytz==2026.1.1.20260408
types-setuptools==82.0.0.20260408
typing-inspection==0.4.2
typing_extensions==4.15.0
tzdata==2026.1
tzlocal==5.3.1
uc-micro-py==2.0.0
ucxx-cu12==0.48.0
umap-learn==0.5.12
umf==1.0.3
uri-template==1.3.0
uritemplate==4.2.0
urllib3==2.5.0
uuid_utils==0.14.1
uvicorn==0.46.0
uvloop==0.22.1
vega-datasets==0.9.0
wadllib==1.3.6
wandb==0.26.1
wasabi==1.1.3
watchdog==6.0.0
watchfiles==1.1.1
wcwidth==0.6.0
weasel==1.0.0
webcolors==25.10.0
webencodings==0.5.1
websocket-client==1.9.0
websockets==15.0.1
Werkzeug==3.1.8
wheel==0.47.0
widgetsnbextension==3.6.10
wordcloud==1.9.6
wrapt==2.1.2
xarray==2025.12.0
xarray-einstats==0.10.0
xgboost==3.2.0
xlrd==2.0.2
xxhash==3.6.0
xyzservices==2026.3.0
yarl==1.23.0
ydf==0.15.0
ydf_tf==2.20.0
yellowbrick==1.5
yfinance==0.2.66
zict==3.0.0
zipp==3.23.1
zstandard==0.25.0
+36
View File
@@ -0,0 +1,36 @@
{
"_comment": "Maps Colab GPU runtime pinned wheels to CPU equivalents for ubuntu-latest CI smoke jobs. The Colab GPU image ships +cu128 builds that won't install on a CPU-only runner; this map either rewrites the spec to a CPU wheel from https://download.pytorch.org/whl/cpu or falls back to module-spoof for packages with no CPU build.",
"rewrite": {
"torch": {
"from_local_version": "+cu128",
"to_index_url": "https://download.pytorch.org/whl/cpu"
},
"torchvision": {
"from_local_version": "+cu128",
"to_index_url": "https://download.pytorch.org/whl/cpu"
},
"torchaudio": {
"from_local_version": "+cu128",
"to_index_url": "https://download.pytorch.org/whl/cpu"
}
},
"module_spoof": {
"torchcodec": "no CPU wheel published; smoke job sys.modules-stubs torchcodec before importing unsloth"
},
"skip": [
"nvidia-cublas-cu12",
"nvidia-cuda-cupti-cu12",
"nvidia-cuda-nvrtc-cu12",
"nvidia-cuda-runtime-cu12",
"nvidia-cudnn-cu12",
"nvidia-cufft-cu12",
"nvidia-curand-cu12",
"nvidia-cusolver-cu12",
"nvidia-cusparse-cu12",
"nvidia-cusparselt-cu12",
"nvidia-nccl-cu12",
"nvidia-nvjitlink-cu12",
"nvidia-nvtx-cu12",
"triton"
]
}
+656
View File
@@ -0,0 +1,656 @@
#!/usr/bin/env python3
"""Ensure keyword arguments use spaces around '=', prune redundant pass statements,
drop the blank line after a short indented import block, merge adjacent same-line
string literals, normalize def-signature magic commas (pre-ruff) so a def with
>= 3 params and a default goes one-per-line while everything else stays
collapsible, and collapse a short multi-line assert onto one line (pre-ruff) by
stripping the magic trailing comma that holds it open."""
from __future__ import annotations
import ast
import argparse
import io
import os
import sys
import tempfile
import tokenize
from collections import defaultdict
from pathlib import Path
def _atomic_write_text(path: Path, data: str, encoding: str) -> None:
"""Write ``data`` to ``path`` atomically via same-dir tmp + fsync + os.replace,
so a crash mid-write leaves either the old or full new content, never a truncation."""
dirpath = str(path.parent) or "."
fd, tmp_path = tempfile.mkstemp(prefix=".kwargs_fix.", dir=dirpath)
try:
with os.fdopen(fd, "w", encoding=encoding) as handle:
handle.write(data)
handle.flush()
os.fsync(handle.fileno())
os.replace(tmp_path, path)
except Exception:
try:
os.unlink(tmp_path)
except OSError:
pass
raise
def enforce_spacing(text: str) -> tuple[str, bool]:
"""Return updated text with keyword '=' padded by spaces, plus change flag."""
lines = text.splitlines(keepends=True)
if not lines:
return text, False
offsets: dict[int, int] = defaultdict(int)
changed = False
reader = io.StringIO(text).readline
for token in tokenize.generate_tokens(reader):
if token.type != tokenize.OP or token.string != "=":
continue
line_index = token.start[0] - 1
col = token.start[1] + offsets[line_index]
if line_index < 0 or line_index >= len(lines):
continue
line = lines[line_index]
if col >= len(line) or line[col] != "=":
continue
line_changed = False
# Insert a space before '=' when missing and not preceded by whitespace.
if col > 0 and line[col - 1] not in {" ", "\t"}:
line = f"{line[:col]} {line[col:]}"
offsets[line_index] += 1
col += 1
line_changed = True
changed = True
# Insert a space after '=' when missing and not followed by whitespace or newline.
next_index = col + 1
if next_index < len(line) and line[next_index] not in {" ", "\t", "\n", "\r"}:
line = f"{line[:next_index]} {line[next_index:]}"
offsets[line_index] += 1
line_changed = True
changed = True
if line_changed:
lines[line_index] = line
if not changed:
return text, False
return "".join(lines), True
def remove_redundant_passes(text: str) -> tuple[str, bool]:
"""Drop pass statements that share a block with other executable code."""
try:
tree = ast.parse(text)
except SyntaxError:
return text, False
redundant: list[ast.Pass] = []
def visit(node: ast.AST) -> None:
for attr in ("body", "orelse", "finalbody"):
value = getattr(node, attr, None)
if not isinstance(value, list) or len(value) <= 1:
continue
for stmt in value:
if isinstance(stmt, ast.Pass):
redundant.append(stmt)
for stmt in value:
if isinstance(stmt, ast.AST):
visit(stmt)
handlers = getattr(node, "handlers", None)
if handlers:
for handler in handlers:
visit(handler)
visit(tree)
if not redundant:
return text, False
lines = text.splitlines(keepends=True)
changed = False
for node in sorted(redundant, key=lambda item: (item.lineno, item.col_offset), reverse=True):
start = node.lineno - 1
end = (node.end_lineno or node.lineno) - 1
if start >= len(lines):
continue
changed = True
if start == end:
line = lines[start]
col_start = node.col_offset
col_end = node.end_col_offset or (col_start + 4)
segment = line[:col_start] + line[col_end:]
lines[start] = segment if segment.strip() else ""
continue
# Fall-back for unexpected multi-line 'pass'.
prefix = lines[start][: node.col_offset]
lines[start] = prefix if prefix.strip() else ""
for idx in range(start + 1, end):
lines[idx] = ""
suffix = lines[end][(node.end_col_offset or 0) :]
lines[end] = suffix
# Normalise to ensure lines end with newlines except at EOF.
result_lines: list[str] = []
for index, line in enumerate(lines):
if not line:
continue
if index < len(lines) - 1 and not line.endswith("\n"):
result_lines.append(f"{line}\n")
else:
result_lines.append(line)
return "".join(result_lines), changed
def remove_blank_after_short_import(text: str) -> tuple[str, bool]:
"""Drop blank line(s) after an import block in a small nested suite.
In an indented suite of <= 3 statements (never module level), when consecutive
imports are followed across blank lines (nothing else) by another statement,
remove those blanks. A comment in the gap blocks the rule. Removing blank lines
never changes the AST.
"""
try:
tree = ast.parse(text)
except SyntaxError:
return text, False
lines = text.splitlines(keepends=True)
import_types = (ast.Import, ast.ImportFrom)
drop: set[int] = set() # 1-based physical line numbers to delete
def suites_of(node: ast.AST) -> list[list[ast.stmt]]:
if isinstance(node, ast.Module):
return [] # module-level import spacing is left alone
out: list[list[ast.stmt]] = []
for attr in ("body", "orelse", "finalbody"):
val = getattr(node, attr, None)
if isinstance(val, list) and val and all(isinstance(s, ast.stmt) for s in val):
out.append(val)
return out
for node in ast.walk(tree):
for suite in suites_of(node):
if len(suite) > 3: # only small blocks
continue
i = 0
while i < len(suite):
if not isinstance(suite[i], import_types):
i += 1
continue
j = i
while j + 1 < len(suite) and isinstance(suite[j + 1], import_types):
j += 1
if j + 1 < len(suite): # an import block followed by another statement
last_imp, nxt = suite[j], suite[j + 1]
gap = range((last_imp.end_lineno or last_imp.lineno) + 1, nxt.lineno)
nums = [n for n in gap if 1 <= n <= len(lines)]
if nums and all(lines[n - 1].strip() == "" for n in nums):
drop.update(nums)
i = j + 1
if not drop:
return text, False
kept = [ln for idx, ln in enumerate(lines, start=1) if idx not in drop]
return "".join(kept), True
_STRING_TRIVIA = (tokenize.NL, tokenize.NEWLINE, tokenize.COMMENT, tokenize.INDENT, tokenize.DEDENT)
_DEF_MIN_PARAMS_FOR_MULTILINE = 3 # signatures with < this many params stay one line
def _def_specs_by_line(tree: ast.AST) -> dict[int, tuple[int, bool]]:
"""Map each def keyword line to (param count, has-any-default).
``*`` / ``/`` markers aren't counted. A default exists if any positional default
is present or any keyword-only default is not ``None`` (``None`` in ``kw_defaults``
means a required keyword-only arg).
"""
out: dict[int, tuple[int, bool]] = {}
for node in ast.walk(tree):
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
a = node.args
count = (
len(a.posonlyargs)
+ len(a.args)
+ len(a.kwonlyargs)
+ (1 if a.vararg else 0)
+ (1 if a.kwarg else 0)
)
has_default = bool(a.defaults) or any(d is not None for d in a.kw_defaults)
out[node.lineno] = (count, has_default)
return out
def normalize_def_trailing_comma(text: str) -> tuple[str, bool]:
"""Force a def signature one-per-line iff >= 3 params AND a default; else collapsible.
A qualifying signature gets a magic trailing comma added (ruff wraps it
one-per-line); every other signature has its trailing comma stripped so ruff
collapses it when it fits. Def parameter lists only, never call sites or
collection literals. Run BEFORE ruff format. Never changes the AST (re-checked).
"""
try:
tree = ast.parse(text)
toks = list(tokenize.generate_tokens(io.StringIO(text).readline))
except (tokenize.TokenError, IndentationError, SyntaxError):
return text, False
specs = _def_specs_by_line(tree)
n = len(toks)
edits: list[tuple[int, int, str]] = [] # (row, col, "del" | "ins")
i = 0
while i < n:
t = toks[i]
if t.type == tokenize.NAME and t.string == "def" and t.start[0] in specs:
cnt, has_default = specs[t.start[0]]
force_multiline = cnt >= _DEF_MIN_PARAMS_FOR_MULTILINE and has_default
j = i + 1
while j < n and not (toks[j].type == tokenize.OP and toks[j].string == "("):
if toks[j].type == tokenize.NEWLINE:
break
j += 1
if j < n and toks[j].type == tokenize.OP and toks[j].string == "(":
depth = 0
k = j
while k < n:
tk = toks[k]
if tk.type == tokenize.OP and tk.string == "(":
depth += 1
elif tk.type == tokenize.OP and tk.string == ")":
depth -= 1
if depth == 0:
m = k - 1
while m > j and toks[m].type in _STRING_TRIVIA:
m -= 1
last = toks[m]
has_comma = last.type == tokenize.OP and last.string == ","
empty = m == j # nothing between ( and )
if force_multiline and not has_comma and not empty:
edits.append((last.end[0], last.end[1], "ins"))
elif not force_multiline and has_comma:
edits.append((last.start[0], last.start[1], "del"))
break
k += 1
i = k + 1
continue
i += 1
if not edits:
return text, False
lines = text.splitlines(keepends=True)
for row, col, kind in sorted(edits, reverse=True):
ln = lines[row - 1]
if kind == "del":
if col < len(ln) and ln[col] == ",":
lines[row - 1] = ln[:col] + ln[col + 1 :]
else: # ins
lines[row - 1] = ln[:col] + "," + ln[col:]
out = "".join(lines)
try:
if ast.dump(ast.parse(out)) != ast.dump(ast.parse(text)):
return text, False
except SyntaxError:
return text, False
return out, True
def _split_string_token(s: str) -> tuple[str, str, str] | None:
"""Split a string literal source into (prefix, quote, body).
``prefix`` is the letters before the opening quote, ``quote`` the delimiter,
``body`` everything between. ``None`` if not a recognizable string literal.
"""
i = 0
while i < len(s) and s[i] not in ("'", '"'):
i += 1
if i >= len(s):
return None
prefix, rest = s[:i], s[i:]
for q in ('"""', "'''", '"', "'"):
if rest.startswith(q) and rest.endswith(q) and len(rest) >= 2 * len(q):
return prefix, q, rest[len(q) : len(rest) - len(q)]
return None
# A "piece" is one string literal in source: a plain STRING token, or a whole
# f-string spanning FSTRING_START..FSTRING_END. (kind, (row, col0), (row, col1), raw)
def _string_pieces(
toks: list[tokenize.TokenInfo], lines: list[str]
) -> list[tuple[str, tuple[int, int], tuple[int, int], str | None]]:
pieces: list[tuple[str, tuple[int, int], tuple[int, int], str | None]] = []
n = len(toks)
def raw_of(start: tuple[int, int], end: tuple[int, int]) -> str | None:
if start[0] != end[0]: # only single-physical-line pieces are mergeable
return None
return lines[start[0] - 1][start[1] : end[1]]
i = 0
while i < n:
t = toks[i]
if t.type == tokenize.STRING:
pieces.append(("str", t.start, t.end, raw_of(t.start, t.end)))
i += 1
elif t.type == tokenize.FSTRING_START:
depth = 0
j = i
while j < n: # walk to the matching FSTRING_END (f-strings can nest)
if toks[j].type == tokenize.FSTRING_START:
depth += 1
elif toks[j].type == tokenize.FSTRING_END:
depth -= 1
if depth == 0:
break
j += 1
end = toks[j].end
pieces.append(("f", t.start, end, raw_of(t.start, end)))
i = j + 1
else:
pieces.append(("other", t.start, t.end, None))
i += 1
return pieces
def _merge_string_run(pieces: list[tuple[str, str]]) -> str | None:
"""Merge a run of adjacent string pieces into one literal's source text.
``pieces`` is ``(kind, raw_source)`` with kind ``"str"`` or ``"f"``. Bytes are
left side-by-side (``None``); a run with no f-string merges plain/raw/unicode
sharing one prefix+quote by body concatenation; a run mixing an f-string with a
plain string (no bytes, no raw) folds into one f-string with plain braces escaped.
Runs of only f-strings are left alone. Caller re-checks the AST and drops a
differing change, so subtle cases are caught.
"""
parsed = []
for kind, raw in pieces:
pqb = _split_string_token(raw)
if pqb is None:
return None
prefix, quote, body = pqb
if "b" in prefix.lower():
return None # bytes: leave side-by-side
parsed.append((kind, prefix, quote, body))
if len({p[2] for p in parsed}) != 1:
return None # mixed quote style: not a safe textual merge
quote = parsed[0][2]
if not any(p[0] == "f" for p in parsed):
# No f-string: merge plain/raw/unicode sharing one prefix by concatenation.
if len({p[1].lower() for p in parsed}) != 1:
return None
return f"{parsed[0][1]}{quote}{''.join(p[3] for p in parsed)}{quote}"
# f-string fold only when a plain string is glued onto an f-string; a run of
# only f-strings is left side-by-side (folding long ones would force ruff to
# re-wrap the surrounding statement).
if all(p[0] == "f" for p in parsed):
return None
# raw mixed with f is too subtle (backslash + brace escaping) -> skip.
if any("r" in p[1].lower() for p in parsed):
return None
body = "".join(
b if kind == "f" else b.replace("{", "{{").replace("}", "}}")
for kind, _pfx, _q, b in parsed
)
return f"f{quote}{body}{quote}"
_LINE_LENGTH = 100 # ruff line-length; an f-fold must not push a statement past it
def _enclosing_stmt(tree: ast.AST, row: int) -> ast.stmt | None:
"""The innermost statement whose physical-line span contains ``row``."""
best: tuple[ast.stmt, int] | None = None
for node in ast.walk(tree):
if isinstance(node, ast.stmt):
lo = node.lineno
hi = node.end_lineno or lo
if lo <= row <= hi and (best is None or hi - lo < best[1]):
best = (node, hi - lo)
return best[0] if best else None
def _fold_collapses(
tree: ast.AST, lines: list[str], row: int, c0: int, c1: int, merged: str
) -> bool:
"""Whether an f-string fold at ``row[c0:c1]`` -> ``merged`` is safe to apply.
Only ``assert`` wraps awkwardly when a message folds (ruff parenthesizes the
condition once it no longer fits one line); every other construct wraps
acceptably so is always allowed. An ``assert`` fold is allowed only if already
one line, or its estimated folded one-line length fits the line length.
"""
stmt = _enclosing_stmt(tree, row)
if not isinstance(stmt, ast.Assert):
return True
lo, hi = stmt.lineno, stmt.end_lineno or stmt.lineno
if lo == hi:
return True
seg = []
for k in range(lo, hi + 1):
ln = lines[k - 1].rstrip("\n")
if k == row:
ln = ln[:c0] + merged + ln[c1:]
seg.append(ln)
indent = len(seg[0]) - len(seg[0].lstrip())
# Conservative over-estimate: join continuation lines with a single space
# (ruff joins bracketed wraps with none), so borderline cases skip the fold.
joined = " ".join(s.strip() for s in seg)
return indent + len(joined) <= _LINE_LENGTH
def merge_adjacent_string_literals(text: str) -> tuple[str, bool]:
"""Merge adjacent string literals on ONE physical line into a single literal.
Plain/raw/unicode runs merge by concatenation; an f-string + plain string folds
into one f-string (plain braces escaped) only while the statement still fits one
line. Runs of only f-strings, and bytes, are left side-by-side. The file AST is
re-checked and a differing change dropped, so meaning never changes.
"""
try:
toks = list(tokenize.generate_tokens(io.StringIO(text).readline))
tree = ast.parse(text)
except (tokenize.TokenError, IndentationError, SyntaxError):
return text, False
lines = text.splitlines(keepends=True)
pieces = _string_pieces(toks, lines)
# Group consecutive mergeable pieces (str/f, single line, same physical line).
runs: list[list[tuple[str, tuple[int, int], tuple[int, int], str]]] = []
cur: list[tuple[str, tuple[int, int], tuple[int, int], str]] = []
for kind, start, end, raw in pieces:
if kind in ("str", "f") and raw is not None:
if cur and cur[-1][2][0] != start[0]:
if len(cur) >= 2:
runs.append(cur)
cur = []
cur.append((kind, start, end, raw))
else:
if len(cur) >= 2:
runs.append(cur)
cur = []
if len(cur) >= 2:
runs.append(cur)
if not runs:
return text, False
edits = []
for run in runs:
merged = _merge_string_run([(kind, raw) for kind, _s, _e, raw in run])
if merged is None:
continue
row, c0, c1 = run[0][1][0], run[0][1][1], run[-1][2][1]
# An f-string fold must not push its statement onto extra lines; a plain
# concatenation always collapses cleanly so it skips this check.
if any(kind == "f" for kind, _s, _e, _r in run) and not _fold_collapses(
tree, lines, row, c0, c1, merged
):
continue
edits.append((row, c0, c1, merged))
if not edits:
return text, False
for row, c0, c1, repl in sorted(edits, key=lambda e: (e[0], e[1]), reverse=True):
ln = lines[row - 1]
lines[row - 1] = ln[:c0] + repl + ln[c1:]
out = "".join(lines)
try:
if ast.dump(ast.parse(text)) != ast.dump(ast.parse(out)):
return text, False
except SyntaxError:
return text, False
return out, True
def collapse_short_asserts(text: str) -> tuple[str, bool]:
"""Collapse a multi-line ``assert`` onto one line when it would fit.
When the statement's estimated one-line length fits, strip the magic trailing
commas (comma before a closer) holding it open so ruff rejoins it. Run BEFORE
ruff format. Skips asserts with a comment (would oscillate). Stripping is
non-semantic except for a one-element tuple; AST is re-checked and changing
asserts left alone.
"""
try:
tree = ast.parse(text)
toks = list(tokenize.generate_tokens(io.StringIO(text).readline))
except (tokenize.TokenError, IndentationError, SyntaxError):
return text, False
lines = text.splitlines(keepends=True)
multiline = [
(n.lineno, n.end_lineno)
for n in ast.walk(tree)
if isinstance(n, ast.Assert) and (n.end_lineno or n.lineno) > n.lineno
]
if not multiline:
return text, False
comment_rows = {t.start[0] for t in toks if t.type == tokenize.COMMENT}
targets = [] # (lo, hi) spans whose one-line form fits and have no comment
for lo, hi in multiline:
if any(lo <= r <= hi for r in comment_rows):
continue # a comment would keep ruff multi-line -> never collapses
seg = [lines[k].rstrip("\n") for k in range(lo - 1, hi)]
indent = len(seg[0]) - len(seg[0].lstrip())
# Over-estimate (join with a space; keep the comma) so a "fits" verdict
# is always at least as long as ruff's real one-line output -> no fight.
if indent + len(" ".join(s.strip() for s in seg)) <= _LINE_LENGTH:
targets.append((lo, hi))
if not targets:
return text, False
# Trailing commas (a ',' whose next significant token is a closer), grouped
# by the target assert they belong to.
sig = [t for t in toks if t.type not in _STRING_TRIVIA]
by_target: dict[tuple[int, int], list[tuple[int, int]]] = defaultdict(list)
for i, t in enumerate(sig):
if t.type == tokenize.OP and t.string == ",":
nxt = sig[i + 1] if i + 1 < len(sig) else None
if nxt and nxt.type == tokenize.OP and nxt.string in (")", "]", "}"):
for lo, hi in targets:
if lo <= t.start[0] <= hi:
by_target[(lo, hi)].append(t.start)
break
if not by_target:
return text, False
base_dump = ast.dump(tree)
working = lines[:]
changed = False
for positions in by_target.values(): # apply per assert; skip any that break AST
trial = working[:]
for row, col in sorted(positions, reverse=True):
ln = trial[row - 1]
if col < len(ln) and ln[col] == ",":
trial[row - 1] = ln[:col] + ln[col + 1 :]
try:
if ast.dump(ast.parse("".join(trial))) == base_dump:
working, changed = trial, True
except SyntaxError:
pass
return ("".join(working), True) if changed else (text, False)
def process_file(path: Path, pre: bool = False) -> bool:
try:
with tokenize.open(path) as handle:
original = handle.read()
encoding = handle.encoding
except (OSError, SyntaxError) as exc: # SyntaxError from tokenize on invalid python
print(f"Failed to read {path}: {exc}", file=sys.stderr)
return False
if pre:
# Pre-ruff: normalize def-signature magic commas (>=3 params + a default
# add so ruff forces one-per-line; everything else strips so ruff
# collapses), and strip the magic trailing comma from a short multi-line
# assert so ruff joins it onto one line. Everything else runs post-ruff.
updated, normalized = normalize_def_trailing_comma(original)
updated, collapsed = collapse_short_asserts(updated)
if normalized or collapsed:
_atomic_write_text(path, updated, encoding)
return True
return False
updated, changed = enforce_spacing(original)
updated, blanked = remove_blank_after_short_import(updated)
updated, merged = merge_adjacent_string_literals(updated)
updated, removed = remove_redundant_passes(updated)
if changed or blanked or merged or removed:
_atomic_write_text(path, updated, encoding)
return True
return False
def main(argv: list[str]) -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("files", nargs="+", help="Python files to fix")
parser.add_argument(
"--pre",
action="store_true",
help="pre-ruff pass: normalize def-signature commas + collapse short multi-line asserts",
)
args = parser.parse_args(argv)
touched: list[Path] = []
self_path = Path(__file__).resolve()
for entry in args.files:
path = Path(entry)
# Skip modifying this script to avoid self-edit loops.
if path.resolve() == self_path:
continue
if not path.exists() or path.is_dir():
continue
if process_file(path, pre=args.pre):
touched.append(path)
if touched:
for path in touched:
print(f"Adjusted kwarg spacing in {path}")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+184
View File
@@ -0,0 +1,184 @@
#!/bin/bash
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
set -euo pipefail
# ============================================================
# Gemma 4 MLX — One-command setup + inference
#
# Supply-chain hardening: the uv installer payload is pinned by
# SHA-256. Rotate by running:
# curl -sSLf https://astral.sh/uv/install.sh | shasum -a 256
# and updating _UV_INSTALLER_SHA256 below.
# ============================================================
#
# Usage:
# bash install_gemma4_mlx.sh [--venv-dir DIR]
#
# This script:
# 1. Creates a Python virtual environment
# 2. Installs uv, mlx-vlm, transformers
# ============================================================
# ── Output style (inspired by unsloth/install.sh) ─────────────
RULE=""
_rule_i=0
while [ "$_rule_i" -lt 52 ]; do
RULE="${RULE}"
_rule_i=$((_rule_i + 1))
done
if [ -n "${NO_COLOR:-}" ]; then
C_TITLE= C_DIM= C_OK= C_WARN= C_ERR= C_RST=
elif [ -t 1 ] || [ -n "${FORCE_COLOR:-}" ]; then
_ESC="$(printf '\033')"
C_TITLE="${_ESC}[38;5;117m"
C_DIM="${_ESC}[38;5;245m"
C_OK="${_ESC}[38;5;108m"
C_WARN="${_ESC}[38;5;136m"
C_ERR="${_ESC}[91m"
C_RST="${_ESC}[0m"
else
C_TITLE= C_DIM= C_OK= C_WARN= C_ERR= C_RST=
fi
step() { printf " ${C_DIM}%-18.18s${C_RST}${3:-$C_OK}%s${C_RST}\n" "$1" "$2"; }
substep() { printf " ${C_DIM}%-18s${2:-$C_DIM}%s${C_RST}\n" "" "$1"; }
fail() { step "error" "$1" "$C_ERR"; exit 1; }
# ── Parse flags ───────────────────────────────────────────────
VENV_DIR=""
_next_is_venv=false
for arg in "$@"; do
if [ "$_next_is_venv" = true ]; then
VENV_DIR="$arg"
_next_is_venv=false
continue
fi
case "$arg" in
--venv-dir) _next_is_venv=true ;;
esac
done
# Default venv location
if [ -z "$VENV_DIR" ]; then
VENV_DIR="$HOME/.unsloth/unsloth_gemma4_mlx"
fi
# ── Banner ────────────────────────────────────────────────────
echo ""
printf " ${C_TITLE}%s${C_RST}\n" "💎 Gemma 4 MLX Installer"
printf " ${C_DIM}%s${C_RST}\n" "$RULE"
echo ""
# ── Platform check ────────────────────────────────────────────
if [ "$(uname)" != "Darwin" ]; then
fail "MLX requires macOS with Apple Silicon. Detected: $(uname)"
fi
_ARCH=$(uname -m)
if [ "$_ARCH" != "arm64" ]; then
step "warning" "Apple Silicon recommended (detected: $_ARCH)" "$C_WARN"
fi
step "platform" "macOS ($_ARCH)"
# ── Detect Python ─────────────────────────────────────────────
PYTHON=""
for _candidate in python3.12 python3.11 python3.13 python3; do
if command -v "$_candidate" >/dev/null 2>&1; then
PYTHON="$_candidate"
break
fi
done
if [ -z "$PYTHON" ]; then
fail "Python 3 not found. Install via: brew install python@3.12"
fi
_PY_VERSION=$("$PYTHON" -c "import sys; print(f'{sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro}')")
step "python" "$PYTHON ($_PY_VERSION)"
# ── Create virtual environment ────────────────────────────────
if [ -x "$VENV_DIR/bin/python" ]; then
step "venv" "using existing environment"
substep "$VENV_DIR"
else
step "venv" "creating virtual environment"
substep "$VENV_DIR"
mkdir -p "$(dirname "$VENV_DIR")"
"$PYTHON" -m venv "$VENV_DIR"
fi
# ── Install uv ───────────────────────────────────────────────
_UV_INSTALLER_SHA256="48cd5aca5d5671a3b3d5f61538cc8622e4434af63319115159990d8b0dd02416"
if ! command -v uv >/dev/null 2>&1; then
step "uv" "installing uv package manager..."
_uv_tmp=$(mktemp)
curl -LsSf "https://astral.sh/uv/install.sh" -o "$_uv_tmp"
_uv_actual=$(shasum -a 256 "$_uv_tmp" | awk '{print $1}')
if [ "$_uv_actual" != "$_UV_INSTALLER_SHA256" ]; then
rm -f "$_uv_tmp"
fail "uv installer SHA-256 mismatch: got $_uv_actual expected $_UV_INSTALLER_SHA256 (refusing to execute)"
fi
sh "$_uv_tmp" </dev/null >/dev/null 2>&1
rm -f "$_uv_tmp"
if [ -f "$HOME/.local/bin/env" ]; then
. "$HOME/.local/bin/env"
fi
export PATH="$HOME/.local/bin:$PATH"
substep "done"
else
step "uv" "found $(uv --version 2>/dev/null || echo 'uv')"
fi
_VENV_PY="$VENV_DIR/bin/python"
# ── Install dependencies ──────────────────────────────────────
step "install" "installing mlx-vlm..."
uv pip install --python "$_VENV_PY" -q mlx-vlm
substep "done"
step "install" "installing transformers>=5.5.0..."
if uv pip install --python "$_VENV_PY" -q "transformers>=5.5.0" 2>/dev/null; then
substep "installed from PyPI"
else
substep "PyPI install failed (Python <3.10?), trying GitHub..."
if uv pip install --python "$_VENV_PY" -q "git+https://github.com/huggingface/transformers.git@v5.5-release" 2>/dev/null; then
substep "installed from huggingface/transformers v5.5-release"
else
step "warning" "could not install transformers>=5.5.0" "$C_WARN"
substep "tried: PyPI, huggingface/transformers v5.5-release"
fi
fi
# ── Verify installation ──────────────────────────────────────
if "$_VENV_PY" -c "import mlx_vlm"; then
substep "mlx-vlm verified"
else
fail "Installation verification failed."
fi
# ── Done ──────────────────────────────────────────────────────
echo ""
printf " ${C_TITLE}%s${C_RST}\n" "Gemma 4 MLX installed!"
printf " ${C_DIM}%s${C_RST}\n" "$RULE"
echo ""
step "available models" "unsloth/gemma-4-E2B-it-UD-MLX-4bit"
substep "unsloth/gemma-4-E4B-it-UD-MLX-4bit"
substep "unsloth/gemma-4-26b-a4b-it-UD-MLX-4bit"
substep "unsloth/gemma-4-31b-it-UD-MLX-4bit"
echo ""
step "venv activate" "source ${VENV_DIR}/bin/activate"
echo ""
step "text chat" "python -m mlx_vlm.chat --model unsloth/gemma-4-E2B-it-UD-MLX-4bit"
echo ""
step "vision chat" "python -m mlx_vlm.chat --model unsloth/gemma-4-31b-it-UD-MLX-4bit"
substep "Use /image path/to/image.jpg to load an image"
echo ""
step "gradio UI" "python -m mlx_vlm.chat_ui --model unsloth/gemma-4-31b-it-UD-MLX-4bit"
echo ""
printf " ${C_DIM}%s${C_RST}\n" "$RULE"
echo ""
+247
View File
@@ -0,0 +1,247 @@
#!/bin/bash
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
set -euo pipefail
# ============================================================
# Qwen3.6 MLX — One-command setup + inference
#
# Supply-chain hardening:
# - All third-party downloads (uv installer, mlx_vlm qwen3_5
# patches) are pinned to an immutable git commit SHA and verified
# against a hardcoded SHA-256. Any mismatch aborts the install
# before the bytes are copied into site-packages.
# - To rotate any pin, fetch the new file with `curl`, run
# `shasum -a 256`, and update the corresponding constant below.
# ============================================================
#
# Usage:
# bash install_qwen3_6_mlx.sh [--venv-dir DIR]
#
# This script:
# 1. Creates a Python virtual environment
# 2. Installs uv, mlx-vlm, transformers, torch, torchvision
# ============================================================
# ── Output style (inspired by unsloth/install.sh) ─────────────
RULE=""
_rule_i=0
while [ "$_rule_i" -lt 52 ]; do
RULE="${RULE}"
_rule_i=$((_rule_i + 1))
done
if [ -n "${NO_COLOR:-}" ]; then
C_TITLE= C_DIM= C_OK= C_WARN= C_ERR= C_RST=
elif [ -t 1 ] || [ -n "${FORCE_COLOR:-}" ]; then
_ESC="$(printf '\033')"
C_TITLE="${_ESC}[38;5;117m"
C_DIM="${_ESC}[38;5;245m"
C_OK="${_ESC}[38;5;108m"
C_WARN="${_ESC}[38;5;136m"
C_ERR="${_ESC}[91m"
C_RST="${_ESC}[0m"
else
C_TITLE= C_DIM= C_OK= C_WARN= C_ERR= C_RST=
fi
step() { printf " ${C_DIM}%-18.18s${C_RST}${3:-$C_OK}%s${C_RST}\n" "$1" "$2"; }
substep() { printf " ${C_DIM}%-18s${2:-$C_DIM}%s${C_RST}\n" "" "$1"; }
fail() { step "error" "$1" "$C_ERR"; exit 1; }
# ── Parse flags ───────────────────────────────────────────────
VENV_DIR=""
_next_is_venv=false
for arg in "$@"; do
if [ "$_next_is_venv" = true ]; then
VENV_DIR="$arg"
_next_is_venv=false
continue
fi
case "$arg" in
--venv-dir) _next_is_venv=true ;;
esac
done
# Default venv location
if [ -z "$VENV_DIR" ]; then
VENV_DIR="$HOME/.unsloth/unsloth_qwen3_6_mlx"
fi
# ── Banner ────────────────────────────────────────────────────
echo ""
printf " ${C_TITLE}%s${C_RST}\n" "Qwen3.6 MLX Installer"
printf " ${C_DIM}%s${C_RST}\n" "$RULE"
echo ""
# ── Platform check ────────────────────────────────────────────
if [ "$(uname)" != "Darwin" ]; then
fail "MLX requires macOS with Apple Silicon. Detected: $(uname)"
fi
_ARCH=$(uname -m)
if [ "$_ARCH" != "arm64" ]; then
step "warning" "Apple Silicon recommended (detected: $_ARCH)" "$C_WARN"
fi
step "platform" "macOS ($_ARCH)"
# ── Detect Python ─────────────────────────────────────────────
PYTHON=""
for _candidate in python3.12 python3.11 python3.13 python3; do
if command -v "$_candidate" >/dev/null 2>&1; then
PYTHON="$_candidate"
break
fi
done
if [ -z "$PYTHON" ]; then
fail "Python 3 not found. Install via: brew install python@3.12"
fi
_PY_VERSION=$("$PYTHON" -c "import sys; print(f'{sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro}')")
step "python" "$PYTHON ($_PY_VERSION)"
# ── Create virtual environment ────────────────────────────────
if [ -x "$VENV_DIR/bin/python" ]; then
step "venv" "using existing environment"
substep "$VENV_DIR"
else
step "venv" "creating virtual environment"
substep "$VENV_DIR"
mkdir -p "$(dirname "$VENV_DIR")"
"$PYTHON" -m venv "$VENV_DIR"
fi
# ── Install uv ───────────────────────────────────────────────
# Pin the uv installer payload by SHA-256. Rotate by running:
# curl -sSLf https://astral.sh/uv/install.sh | shasum -a 256
# and updating the constant below. We fetch into a temp file, verify
# the digest, and only then execute. Mismatch aborts.
_UV_INSTALLER_SHA256="48cd5aca5d5671a3b3d5f61538cc8622e4434af63319115159990d8b0dd02416"
if ! command -v uv >/dev/null 2>&1; then
step "uv" "installing uv package manager..."
_uv_tmp=$(mktemp)
curl -LsSf "https://astral.sh/uv/install.sh" -o "$_uv_tmp"
_uv_actual=$(shasum -a 256 "$_uv_tmp" | awk '{print $1}')
if [ "$_uv_actual" != "$_UV_INSTALLER_SHA256" ]; then
rm -f "$_uv_tmp"
fail "uv installer SHA-256 mismatch: got $_uv_actual expected $_UV_INSTALLER_SHA256 (refusing to execute)"
fi
sh "$_uv_tmp" </dev/null
rm -f "$_uv_tmp"
if [ -f "$HOME/.local/bin/env" ]; then
. "$HOME/.local/bin/env"
fi
export PATH="$HOME/.local/bin:$PATH"
substep "done"
else
step "uv" "found $(uv --version 2>/dev/null || echo 'uv')"
fi
_VENV_PY="$VENV_DIR/bin/python"
# ── Install dependencies ──────────────────────────────────────
step "install" "installing mlx-vlm..."
uv pip install --python "$_VENV_PY" -q mlx-vlm
substep "done"
step "install" "installing transformers>=5.2.0..."
if uv pip install --python "$_VENV_PY" -q "transformers>=5.2.0"; then
substep "installed from PyPI"
else
substep "PyPI install failed, trying GitHub..."
if uv pip install --python "$_VENV_PY" -q "git+https://github.com/huggingface/transformers.git"; then
substep "installed from huggingface/transformers main"
else
fail "Could not install transformers>=5.2.0 (required for Qwen3.5/3.6 model support). Please check your Python version (>=3.10 required) and network connection, then try again."
fi
fi
step "install" "installing torch + torchvision (needed for Qwen3 VL processor)..."
uv pip install --python "$_VENV_PY" -q torch torchvision
substep "done"
# ── Verify installation ──────────────────────────────────────
if "$_VENV_PY" -c "import mlx_vlm; import torch; import torchvision; import transformers"; then
substep "mlx-vlm + torch + transformers verified"
else
fail "Installation verification failed. Please ensure Python >=3.10 and try again."
fi
# ── Apply patches for multi-turn image chat ──────────────────
#
# Pin every patch to an immutable commit SHA and verify the body
# against a hardcoded SHA-256. The mlx_vlm_qwen3_5 patch tree
# currently only exists on the upstream `fix/ui-fix` branch; we pin
# to the branch HEAD commit, NOT the floating ref, so a forced push
# on `fix/ui-fix` cannot swap the bytes under us.
#
# Rotate by:
# _PATCH_COMMIT=<new SHA>
# curl -sSLf "https://raw.githubusercontent.com/unslothai/unsloth/$_PATCH_COMMIT/unsloth/models/patches/mlx_vlm_qwen3_5/qwen3_5.py" | shasum -a 256
# curl -sSLf "https://raw.githubusercontent.com/unslothai/unsloth/$_PATCH_COMMIT/unsloth/models/patches/mlx_vlm_qwen3_5/generate.py" | shasum -a 256
_PATCH_COMMIT="013c99e51bbb8c4b83d88f3b150a1e53251a19d2"
_PATCH_BASE="https://raw.githubusercontent.com/unslothai/unsloth/${_PATCH_COMMIT}/unsloth/models/patches/mlx_vlm_qwen3_5"
_PATCH_SHA_QWEN35="4b6fbbcc59b1d6b935e7204351aae1476836d25542a11c7885402b672d2efa64"
_PATCH_SHA_GENERATE="50c4cbb8c3d94c0c74a4d209db6d2b23b102944c147c6421f2eded427b8edaf7"
_SITE_PKGS=$("$_VENV_PY" -c "import site; print(site.getsitepackages()[0])")
step "patch" "fixing multi-turn image chat..."
# Stage all downloads in an isolated tmpdir; we only copy into
# site-packages after every checksum has matched.
_PATCH_TMP=$(mktemp -d)
trap 'rm -rf "$_PATCH_TMP"' EXIT
apply_pinned_patch() {
# apply_pinned_patch <remote_basename> <expected_sha256> <dest_abspath>
_name="$1"; _expected="$2"; _dest="$3"
_staged="$_PATCH_TMP/$_name"
if ! curl -sSLf "${_PATCH_BASE}/${_name}" -o "$_staged"; then
step "warning" "failed to download ${_name} patch — multi-turn image chat may not work" "$C_WARN"
return 1
fi
_actual=$(shasum -a 256 "$_staged" | awk '{print $1}')
if [ "$_actual" != "$_expected" ]; then
step "warning" "${_name} SHA-256 mismatch (got $_actual expected $_expected) — refusing to install patch" "$C_WARN"
return 1
fi
mkdir -p "$(dirname "$_dest")"
cp "$_staged" "$_dest"
return 0
}
if apply_pinned_patch "qwen3_5.py" "$_PATCH_SHA_QWEN35" "${_SITE_PKGS}/mlx_vlm/models/qwen3_5/qwen3_5.py"; then
substep "patched qwen3_5.py (MRoPE position reset)"
fi
if apply_pinned_patch "generate.py" "$_PATCH_SHA_GENERATE" "${_SITE_PKGS}/mlx_vlm/generate.py"; then
substep "patched generate.py (mask trim on cache reuse)"
fi
# Clear pycache so patches take effect
find "${_SITE_PKGS}/mlx_vlm" -name "__pycache__" -type d -exec rm -rf {} + 2>/dev/null || true
substep "cleared bytecode cache"
# ── Done ──────────────────────────────────────────────────────
echo ""
printf " ${C_TITLE}%s${C_RST}\n" "Qwen3.6 MLX installed!"
printf " ${C_DIM}%s${C_RST}\n" "$RULE"
echo ""
step "available models" "unsloth/Qwen3.6-35B-A3B-UD-MLX-3bit"
substep "unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit"
substep "unsloth/Qwen3.6-35B-A3B-MLX-8bit"
echo ""
step "venv activate" "source ${VENV_DIR}/bin/activate"
echo ""
step "vision chat" "python -m mlx_vlm.chat --model unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit"
substep "Use /image path/to/image.jpg to load an image"
echo ""
step "gradio UI" "python -m mlx_vlm.chat_ui --model unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit"
echo ""
printf " ${C_DIM}%s${C_RST}\n" "$RULE"
echo ""
+310
View File
@@ -0,0 +1,310 @@
#!/usr/bin/env bash
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
#
# ──────────────────────────────────────────────────────────────────────────────
# Enable ROCm-on-WSL for AMD GPUs (Strix Halo/Point APUs AND discrete Radeon RX
# 7000/9000). Verified on gfx1151 (Radeon 8060S) and gfx1200 (Radeon RX 9060 XT).
# ──────────────────────────────────────────────────────────────────────────────
# install.sh routes the detected arch to the right ROCm wheels once a runtime exists;
# what it does NOT do is install AMD's ROCm userspace + the WSL DXG bridge (librocdxg).
# This helper does that Linux-side prerequisite on Ubuntu 24.04 WSL2, invoked by
# install.sh when it sees an AMD GPU via /dev/dxg but no ROCm yet. Arch-agnostic: the
# arch is auto-detected from rocminfo (override UNSLOTH_WSL_GFX=gfx1200). Idempotent.
#
# Manual, admin-gated Windows prerequisite: an AMD Adrenalin driver with
# production ROCDXG/WSL support (26.2.2+). install.ps1 offers to update it. Once
# installed + rebooted, /dev/dxg is exposed to WSL and this script builds the rest.
#
# HOW ROCDXG WORKS (and why older /usr/lib/wsl/lib notes are wrong): librocdxg.so
# is AMD's user-mode bridge between the Linux HSA runtime and the Windows driver
# over /dev/dxg. The STANDARD hsa-rocr runtime (NOT the gone "roc4wsl" package)
# loads it when HSA_ENABLE_DXG_DETECTION=1. No hsa/rocm libs need injecting into
# /usr/lib/wsl/lib (it holds only d3d12/dxcore), yet rocminfo enumerates gfx1151
# fine -- so we gate on /dev/dxg, not on WSL lib injection.
#
# KNOWN CAVEAT (ROCm/ROCm#6022): librocdxg can cap usable ROCm VRAM at the WSL
# VM's RAM (.wslconfig [wsl2] memory=) on some BIOS UMA layouts, and amd-smi
# doesn't work in WSL. On OOM below capacity, raise memory= (then wsl --shutdown)
# and watch GPU use from Windows. Large-UMA BIOS exposes the full pool regardless.
#
# Verified on Ryzen AI Max+ PRO 395 / Radeon 8060S (gfx1151) with ROCm 7.2.1 +
# Ubuntu 24.04 + WSL2 + Adrenalin. These pins MOVE; bump + re-verify on newer ROCm.
# ──────────────────────────────────────────────────────────────────────────────
set -euo pipefail
# ── Tunables (override via env) ──────────────────────────────────────────────
ROCM_VER="${UNSLOTH_WSL_ROCM_VER:-7.2.1}" # ROCm release to install
# GPU arch: empty = auto-detect from rocminfo after install (override UNSLOTH_WSL_GFX=gfx1200).
# The ROCm + librocdxg setup is arch-agnostic; only verify + the smoke test need the arch.
GFX="${UNSLOTH_WSL_GFX:-}"
LIBROCDXG_REF="${UNSLOTH_LIBROCDXG_REF:-develop}" # ROCm/librocdxg git ref to build
# AMD's wheel index for the (optional) smoke test; resolved after arch detection.
TORCH_INDEX=""
# Optional torch smoke test (throwaway venv). OFF by default: install.sh installs
# torch itself into the real venv right after, so a duplicate download is wasteful.
SMOKE_TEST="${UNSLOTH_WSL_SMOKE_TEST:-0}"
# REQUIRED constraint -- without it pip prefers PyPI's newer CUDA torch over the
# gfx1151 ROCm wheel. 2.11 carries AMD's real gfx1151 fix (matches install.sh).
TORCH_CONSTRAINT="${UNSLOTH_WSL_TORCH_CONSTRAINT:-torch>=2.11.0,<2.12.0}"
ROCM_DIR="" # resolved after install
say() { printf '\n\033[1;36m== %s\033[0m\n' "$*"; }
note() { printf ' %s\n' "$*"; }
die() { printf '\n\033[1;31m[BLOCKED] %s\033[0m\n' "$*" >&2; exit 1; }
# sudo only if not already root (WSL distros often run as root)
SUDO=""
if [ "$(id -u)" -ne 0 ]; then
command -v sudo >/dev/null 2>&1 || die "Need root or sudo to install ROCm."
SUDO="sudo"
fi
# ── Windows 11 SDK (headers for the librocdxg build) ─────────────────────────
# librocdxg's cmake build needs the Windows SDK 'shared' headers, which live on
# the Windows HOST under C:\Program Files (x86)\Windows Kits\10\Include\<ver>\.
_WIN_SDK_INC_BASE="/mnt/c/Program Files (x86)/Windows Kits/10/Include"
# Print the newest installed SDK include dir with 'shared' headers, or nothing.
# find + read loop (not `for ... in $(ls)`) since the base path has a space.
_find_win_sdk() {
[ -d "$_WIN_SDK_INC_BASE" ] || return 0
while IFS= read -r _inc; do
[ -n "$_inc" ] || continue
if [ -d "$_inc/shared" ]; then printf '%s' "$_inc"; return 0; fi
done < <(find "$_WIN_SDK_INC_BASE" -mindepth 1 -maxdepth 1 -type d 2>/dev/null | sort -Vr)
return 0
}
# Best-effort: install the Windows 11 SDK on the Windows HOST via winget so the
# build has its headers with no manual step. Elevates -> ONE UAC prompt; headers
# appear under /mnt/c immediately (no reboot). Never fatal -- failure falls
# through to a manual-install message. Opt out: UNSLOTH_SKIP_WIN_SDK_INSTALL=1.
_install_windows_sdk_via_winget() {
[ "${UNSLOTH_SKIP_WIN_SDK_INSTALL:-0}" = "1" ] && { note "Skipping Windows SDK auto-install (UNSLOTH_SKIP_WIN_SDK_INSTALL=1)."; return 0; }
command -v powershell.exe >/dev/null 2>&1 || return 0
# `command -v` succeeds even with WSL interop OFF (.exe on PATH but fails
# with "Exec format error"); verify it actually executes.
powershell.exe -NoProfile -Command "exit 0" >/dev/null 2>&1 || return 0
if ! powershell.exe -NoProfile -Command "if (Get-Command winget -ErrorAction SilentlyContinue) { exit 0 } else { exit 1 }" >/dev/null 2>&1; then
note "winget not available on the Windows host -- cannot auto-install the Windows SDK."
return 0
fi
say "Installing the Windows 11 SDK on the Windows host via winget"
note "librocdxg needs its headers. Approve the UAC prompt on the Windows desktop."
note "One-time (~1-3 GB download); opt out with UNSLOTH_SKIP_WIN_SDK_INSTALL=1."
# Newest SDK first, then a fallback. Header presence is the source of truth
# (re-check each attempt), not winget's exit code. </dev/null so winget never
# consumes a piped `curl | sh` stdin.
for _sdk_id in Microsoft.WindowsSDK.10.0.26100 Microsoft.WindowsSDK.10.0.22621; do
note "winget install ${_sdk_id} ..."
# --source winget: pin the community source so a broken default msstore
# source (the cert failure this PR fixes) can't abort SDK resolution.
powershell.exe -NoProfile -Command "winget install --id ${_sdk_id} -e --source winget --accept-source-agreements --accept-package-agreements --disable-interactivity" </dev/null || true
if [ -n "$(_find_win_sdk)" ]; then
note "Windows SDK headers present after install."
return 0
fi
done
note "Automatic Windows SDK install did not complete."
return 0
}
# ── PREFLIGHT ────────────────────────────────────────────────────────────────
say "Preflight checks"
# shellcheck disable=SC1091
. /etc/os-release 2>/dev/null || true
if [ "${VERSION_ID:-}" != "24.04" ]; then
die "This targets Ubuntu 24.04 (found '${VERSION_ID:-unknown}'). AMD's ROCm-on-WSL supports 24.04; create a dedicated distro: wsl --install Ubuntu-24.04 (do not run on 26.04 -- ROCm 7.2 does not target it yet)."
fi
if [ ! -e /dev/dxg ]; then
die "/dev/dxg missing -- WSL GPU paravirtualization not present. Ensure this is WSL2 (not WSL1) on a recent Windows build, and that an AMD GPU + ROCDXG-capable Adrenalin driver is installed on the Windows host (then reboot)."
fi
note "Ubuntu 24.04 + /dev/dxg present."
# Don't block on hsa/rocm libs in /usr/lib/wsl/lib: a working ROCDXG setup
# doesn't need them (only d3d12/dxcore). Real readiness is checked via rocminfo.
# ── Step 1: build/runtime prerequisites ──────────────────────────────────────
say "Installing build prerequisites"
export DEBIAN_FRONTEND=noninteractive
$SUDO apt-get update -y
# `make` is explicit: cmake shells out to it but Ubuntu only *recommends* it, so
# minimal images lack it and the librocdxg `make -j` build would fail.
$SUDO apt-get install -y cmake make gcc g++ git wget gpg ca-certificates python3-venv python3-pip
# ── Step 2: ROCm ${ROCM_VER} userspace (no DKMS -- WSL uses the Windows driver) ─
say "Installing ROCm ${ROCM_VER} userspace"
if ! command -v rocminfo >/dev/null 2>&1 && [ ! -x /opt/rocm/bin/rocminfo ]; then
# Direct apt-repo install (leaner than amdgpu-install; repo is indexed by
# ROCm version, e.g. .../apt/7.2.1).
$SUDO mkdir -p /etc/apt/keyrings
wget -qO- https://repo.radeon.com/rocm/rocm.gpg.key \
| gpg --dearmor | $SUDO tee /etc/apt/keyrings/rocm.gpg >/dev/null
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/${ROCM_VER} noble main" \
| $SUDO tee /etc/apt/sources.list.d/rocm.list >/dev/null
printf 'Package: *\nPin: release o=repo.radeon.com\nPin-Priority: 600\n' \
| $SUDO tee /etc/apt/preferences.d/rocm-pin-600 >/dev/null
$SUDO apt-get update -y
# rocm-libs pulls everything torch links at runtime (rocblas, hipblas,
# miopen-hip, rccl, ...); hsa-rocr + rocminfo come as deps. Large (~5 GB
# download / ~23 GB installed).
$SUDO apt-get install -y rocm-libs rocminfo hip-runtime-amd
else
note "ROCm already present -- skipping apt install."
fi
# Resolve the real ROCm dir and ensure the canonical /opt/rocm symlink. apt lays
# ROCm under /opt/rocm-<ver> and rocm-core symlinks /opt/rocm -> that; repair if
# an earlier partial run left /opt/rocm as a real dir blocking the symlink.
_real="$(ls -d /opt/rocm-* 2>/dev/null | sort -V | tail -1 || true)"
if [ -n "$_real" ] && [ ! -L /opt/rocm ] && [ -d /opt/rocm ]; then
# /opt/rocm is a real dir blocking the symlink. Only treat it as a removable
# stray stub if it's NOT a real ROCm install (a real one has bin/rocminfo /
# bin/hipcc / .info/version) -- this protects a user's pre-existing ROCm. Even
# then we MOVE IT ASIDE, never rm -rf, so a wrong guess can't lose data.
if [ -e /opt/rocm/bin/rocminfo ] || [ -e /opt/rocm/bin/hipcc ] || [ -e /opt/rocm/.info/version ]; then
note "/opt/rocm is a real ROCm install -- leaving it untouched (will install librocdxg into it)."
else
note "Moving stray /opt/rocm stub aside -> $_real (not deleting it)"
$SUDO cp -an /opt/rocm/. "$_real"/ 2>/dev/null || true
$SUDO mv /opt/rocm "/opt/rocm.unsloth-stub-bak.$(date +%s)" 2>/dev/null || true
[ -e /opt/rocm ] || $SUDO ln -s "$_real" /opt/rocm
fi
elif [ -n "$_real" ] && [ ! -e /opt/rocm ]; then
$SUDO ln -s "$_real" /opt/rocm
fi
if [ -L /opt/rocm ] || [ -d /opt/rocm ]; then ROCM_DIR="/opt/rocm"; else ROCM_DIR="$_real"; fi
{ [ -n "$ROCM_DIR" ] && [ -d "$ROCM_DIR" ]; } || die "ROCm not found under /opt after install."
note "ROCm at ${ROCM_DIR}"
# ── Step 3: build librocdxg (DXG <-> HSA bridge; not yet an apt package) ──────
say "Building librocdxg (${LIBROCDXG_REF})"
if [ -e "${ROCM_DIR}/lib/librocdxg.so" ]; then
note "librocdxg already installed -- skipping build."
else
# Discover the newest installed Win11 SDK (version differs per machine). If
# absent, auto-install via winget (one UAC prompt) and re-discover; only if
# that ALSO fails do we stop with manual instructions.
_win_sdk="$(_find_win_sdk)"
if [ -z "$_win_sdk" ]; then
note "Windows 11 SDK headers not found -- attempting automatic install..."
_install_windows_sdk_via_winget
_win_sdk="$(_find_win_sdk)"
fi
[ -n "$_win_sdk" ] || die "Windows 11 SDK headers not found under 'C:\\Program Files (x86)\\Windows Kits\\10\\Include\\*\\shared', and the automatic winget install did not complete. Install it on the Windows host (e.g. 'winget install Microsoft.WindowsSDK.10.0.26100') and re-run."
note "Windows SDK: ${_win_sdk}"
_src="${HOME}/.unsloth/librocdxg"
rm -rf "$_src"
git clone --depth 1 --branch "$LIBROCDXG_REF" https://github.com/ROCm/librocdxg.git "$_src" \
|| git clone "https://github.com/ROCm/librocdxg.git" "$_src"
(
cd "$_src"
git checkout "$LIBROCDXG_REF" 2>/dev/null || true
mkdir -p build && cd build
cmake .. -DWIN_SDK="${_win_sdk}/shared"
make -j"$(nproc)"
$SUDO make install
)
fi
# Ensure soname symlinks resolve to whatever version was built (e.g. 1.2.0).
_dxg_real="$(ls -1 "${ROCM_DIR}"/lib/librocdxg.so.*.* 2>/dev/null | sort -V | tail -1 || true)"
if [ -n "$_dxg_real" ]; then
_dxg_base="$(basename "$_dxg_real")" # librocdxg.so.1.2.0
_dxg_major="$(printf '%s' "$_dxg_base" | sed -E 's/librocdxg\.so\.([0-9]+).*/\1/')"
$SUDO ln -sf "$_dxg_base" "${ROCM_DIR}/lib/librocdxg.so.${_dxg_major}"
$SUDO ln -sf "librocdxg.so.${_dxg_major}" "${ROCM_DIR}/lib/librocdxg.so"
fi
echo "${ROCM_DIR}/lib" | $SUDO tee /etc/ld.so.conf.d/rocm.conf >/dev/null
$SUDO ldconfig
# ── Step 4: persist environment (system-wide so Studio's worker inherits it) ──
say "Persisting ROCm-on-WSL environment"
_envfile="/etc/profile.d/unsloth-rocm-wsl.sh"
$SUDO tee "$_envfile" >/dev/null <<EOF
# >>> Unsloth ROCm-on-WSL >>>
export HSA_ENABLE_DXG_DETECTION=1
export TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1
export PATH="${ROCM_DIR}/bin:\${PATH}"
export LD_LIBRARY_PATH="${ROCM_DIR}/lib:\${LD_LIBRARY_PATH:-}"
# <<< Unsloth ROCm-on-WSL <<<
EOF
# also drop into ~/.bashrc for interactive shells
if [ -n "${HOME:-}" ] && ! grep -q "Unsloth ROCm-on-WSL" "${HOME}/.bashrc" 2>/dev/null; then
cat "$_envfile" >> "${HOME}/.bashrc"
fi
# export into the current process so verification below works immediately
export HSA_ENABLE_DXG_DETECTION=1
export PATH="${ROCM_DIR}/bin:${PATH}"
export LD_LIBRARY_PATH="${ROCM_DIR}/lib:${LD_LIBRARY_PATH:-}"
# ── Step 5: verify the runtime enumerates the GPU ────────────────────────────
say "Verifying rocminfo enumerates the GPU over DXG"
# Capture rocminfo into a var BEFORE grepping: piping into `grep -q` SIGPIPEs
# rocminfo on first match, which under `set -o pipefail` turns a successful match
# into a pipeline failure.
_rocminfo_out="$(rocminfo 2>/dev/null || true)"
# GPU agents advertise an ISA "Name: gfxNNNN". Match gfx[1-9] (excludes gfx000, the CPU
# agent), drop the "gfx*-generic" fallback ISA, and take the first real GPU arch.
_detected_gfx="$(printf '%s\n' "$_rocminfo_out" | grep -E 'Name:[[:space:]]*gfx[1-9]' | grep -v 'generic' | grep -oE 'gfx[1-9][0-9a-z]*' | head -1 || true)"
if [ -z "$_detected_gfx" ]; then
printf '%s\n' "$_rocminfo_out" | head -25 >&2 || true
die "rocminfo did not enumerate any GPU agent. Most common cause: the Windows AMD driver predates production ROCDXG -- update Adrenalin (install.ps1 offers this), reboot, and re-run."
fi
# Honour a caller-pinned arch (sanity-check via a consuming grep, not grep -q: under
# pipefail -q would SIGPIPE printf on large output and misreport the arch); else adopt.
if [ -n "$GFX" ] && ! printf '%s\n' "$_rocminfo_out" | grep -E "Name:[[:space:]]*${GFX}([^0-9]|$)" >/dev/null; then
die "rocminfo enumerated '${_detected_gfx}' but not the requested UNSLOTH_WSL_GFX='${GFX}'."
fi
GFX="${GFX:-$_detected_gfx}"
# Display-only summary: best-effort (|| true) so head's early pipe-close under
# `set -o pipefail` can't fail the bootstrap after verification already passed.
printf '%s\n' "$_rocminfo_out" | grep -E 'Marketing Name|Device Type|Compute Unit' | grep -iE "Radeon|GPU|Compute" | head -3 || true
note "ROCm-on-WSL runtime is live for ${GFX}."
# ── Step 6 (optional): torch smoke test from AMD's per-arch wheel index ───────
if [ "$SMOKE_TEST" = "1" ]; then
say "Smoke-testing PyTorch on ${GFX} (throwaway venv)"
# Map the detected arch to AMD's repo.amd.com wheel family index.
case "$GFX" in
gfx1200|gfx1201) _fam="gfx120X-all" ;;
gfx1100|gfx1101|gfx1102|gfx1103) _fam="gfx110X-all" ;;
*) _fam="$GFX" ;; # gfx1150/gfx1151/gfx90a: own index
esac
TORCH_INDEX="${UNSLOTH_AMD_ROCM_MIRROR:-https://repo.amd.com/rocm/whl}/${_fam}/"
_venv="${HOME}/.unsloth/rocm-smoketest"
rm -rf "$_venv"; python3 -m venv "$_venv"
"$_venv/bin/pip" install --quiet --upgrade pip
# AMD arch index is primary (torch + triton); PyPI only an extra for pure-py
# deps. The constraint keeps pip on the ROCm wheel, not a newer PyPI CUDA torch.
"$_venv/bin/pip" install --index-url "$TORCH_INDEX" \
--extra-index-url https://pypi.org/simple "$TORCH_CONSTRAINT" || \
die "torch install from ${TORCH_INDEX} failed."
# WSL: torch's bundled ROCr must load the DXG bridge -- drop librocdxg into torch/lib.
_tlib="$("$_venv/bin/python" -c 'import torch,os;print(os.path.join(os.path.dirname(torch.__file__),"lib"))' 2>/dev/null || true)"
[ -d "$_tlib" ] && cp -f "${ROCM_DIR}"/lib/librocdxg.so* "$_tlib"/ 2>/dev/null || true
"$_venv/bin/python" - <<'PY'
import torch
ok = torch.cuda.is_available()
print("torch:", torch.__version__, "| cuda(rocm) available:", ok)
if ok:
print("device:", torch.cuda.get_device_name(0))
free, total = torch.cuda.mem_get_info(0)
print(f"mem: free={free/1e9:.1f} GB total={total/1e9:.1f} GB")
import time
a = torch.randn(4096, 4096, device="cuda", dtype=torch.float16)
b = torch.randn(4096, 4096, device="cuda", dtype=torch.float16)
torch.cuda.synchronize(); t0 = time.time()
for _ in range(10): c = a @ b
torch.cuda.synchronize()
print(f"matmul ok ({(time.time()-t0)/10*1e3:.1f} ms/iter)")
raise SystemExit(0 if ok else 1)
PY
rm -rf "$_venv"
fi
say "Done."
note "ROCm-on-WSL is ready for ${GFX}. If you ran this standalone, install Unsloth"
note "in THIS distro and it will detect the GPU automatically:"
note " curl -fsSL https://unsloth.ai/install.sh | sh"
+150
View File
@@ -0,0 +1,150 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
"""Refuse dangerous GitHub Actions trigger patterns at PR time.
Bans patterns behind the TanStack GHSA-g7cv-rxg3-hmpx compromise:
1. `pull_request_target` -- runs a fork's workflow against the base
repo's secrets/permissions; use `pull_request` instead.
2. `workflow_run` chained to a PR-triggered workflow -- same trust
boundary problem one hop later (poisoned artifacts/caches run with
elevated permissions).
3. Cache keys shared between PR-triggered and publish/release/push
workflows -- a fork PR could poison a cache the publish workflow
restores. Partition the key namespaces.
Exit codes: 0 = no findings, 1 = findings (listed on stderr).
Run from repo root: python3 scripts/lint_workflow_triggers.py
"""
from __future__ import annotations
import argparse
import re
import sys
from pathlib import Path
try:
import yaml
except ImportError:
print("ERROR: PyYAML is required. Install with 'pip install pyyaml'", file = sys.stderr)
sys.exit(2)
REPO_ROOT = Path(__file__).resolve().parents[1]
DEFAULT_WORKFLOWS_DIR = REPO_ROOT / ".github" / "workflows"
BANNED_TRIGGERS: tuple[str, ...] = ("pull_request_target",)
RESTRICTED_TRIGGERS: tuple[str, ...] = ("workflow_run",)
PUBLISH_WORKFLOW_NAMES: tuple[str, ...] = ("release-desktop.yml",)
def _normalise_on(on_field):
if isinstance(on_field, str):
return {on_field}
if isinstance(on_field, list):
return set(on_field)
if isinstance(on_field, dict):
return set(on_field.keys())
return set()
def _load_workflow(path: Path):
try:
return yaml.safe_load(path.read_text())
except Exception as exc:
print(f"ERROR: failed to parse {path}: {exc}", file = sys.stderr)
sys.exit(2)
def _extract_cache_keys(path: Path) -> list[str]:
text = path.read_text()
keys: list[str] = []
for m in re.finditer(r"(?:^|\n)\s*key:\s*([^\n]+)", text):
keys.append(m.group(1).strip())
return keys
def _trigger_set(yaml_doc) -> set[str]:
on = yaml_doc.get(True)
if on is None:
on = yaml_doc.get("on")
return _normalise_on(on)
def main() -> int:
parser = argparse.ArgumentParser(description = __doc__)
parser.add_argument(
"--workflows-dir",
type = Path,
default = DEFAULT_WORKFLOWS_DIR,
help = "Override the workflows directory (used by tests).",
)
args = parser.parse_args()
workflows_dir = args.workflows_dir
findings: list[str] = []
workflows = sorted(workflows_dir.glob("*.yml"))
pr_triggered: list[tuple[Path, list[str]]] = []
publish_triggered: list[tuple[Path, list[str]]] = []
for path in workflows:
doc = _load_workflow(path)
triggers = _trigger_set(doc)
for t in BANNED_TRIGGERS:
if t in triggers:
findings.append(
f"{path.name}: BANNED trigger '{t}' (GHSA-g7cv-rxg3-hmpx "
"pattern: fork PRs run in base-repo context). Switch to "
"'pull_request' and use a deploy-on-merge workflow for "
"any privileged step."
)
for t in RESTRICTED_TRIGGERS:
if t in triggers:
text = path.read_text()
if "lint:workflow_triggers-allow-workflow_run" not in text:
findings.append(
f"{path.name}: RESTRICTED trigger '{t}' requires an "
"explicit `# lint:workflow_triggers-allow-workflow_run` "
"comment somewhere in the file, with a justification."
)
if "pull_request" in triggers:
pr_triggered.append((path, _extract_cache_keys(path)))
is_dispatch_only = "workflow_dispatch" in triggers and not (
"push" in triggers or "pull_request" in triggers
)
if path.name in PUBLISH_WORKFLOW_NAMES or is_dispatch_only:
publish_triggered.append((path, _extract_cache_keys(path)))
pr_keys = {key for _, keys in pr_triggered for key in keys}
for pub_path, pub_keys in publish_triggered:
for k in pub_keys:
if k in pr_keys:
findings.append(
f"{pub_path.name}: cache key {k!r} is also declared in a "
"PR-triggered workflow. A fork PR could poison this cache "
"and the publish workflow would restore it on next run. "
"Add a unique suffix (e.g. '-publish-only') to partition "
"the namespaces."
)
if findings:
print("Workflow trigger lint failed with the following issues:", file = sys.stderr)
for f in findings:
print(f" - {f}", file = sys.stderr)
return 1
print(
f"OK: scanned {len(workflows)} workflow file(s); "
f"no pull_request_target, no unjustified workflow_run, "
f"no PR/publish cache-key collision."
)
return 0
if __name__ == "__main__":
sys.exit(main())
+780
View File
@@ -0,0 +1,780 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
"""Lockfile supply-chain audit for the Studio frontend and Tauri shell.
Runs BEFORE `npm ci` / `cargo fetch` in CI. Refuses to proceed when a
lockfile contains patterns indicating supply-chain injection (npm
Shai-Hulud waves, cargo crates.io brand-squats).
Checks package-lock.json (lockfileVersion 2/3): `resolved` URL must be
the npm registry (direct git/github/file refs are the injection vector);
`integrity` SHA must be present; known IOC substrings grepped from the
body. Checks Cargo.lock: `source` must be the crates.io registry index;
known cargo IOC substrings.
Exit codes: 0 = clean (or skip env var set to a justification >=5 chars,
not '1'/'true'); 1 = findings; 2 = internal error.
Only PARSES the lockfiles, never executes or networks. Complements (not
replaces) `npm audit` / OSV-Scanner / the advisory-DB pipeline. Fires
before any third-party install script runs on the runner.
"""
from __future__ import annotations
import argparse
import json
import os
import re
import sys
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[1]
# Known IOC strings (case-sensitive substring match). Each is tied to a
# public advisory; speculative/generic patterns would false-positive on
# upgrades.
NPM_IOC_STRINGS: tuple[str, ...] = (
# Shai-Hulud TanStack wave -- May 11, 2026 (GHSA-g7cv-rxg3-hmpx).
"router_init.js",
"tanstack_runner.js",
"router_runtime.js",
"@tanstack/setup",
"github:tanstack/router#79ac49eedf774dd4b0cfa308722bc463cfe5885c",
# Exfiltration endpoints observed across both Shai-Hulud waves.
"filev2.getsession.org",
"getsession.org/file/",
# Campaign markers; the worm tarballs print this to stdout on run.
"A Mini Shai-Hulud has Appeared",
# Mini Shai-Hulud May-12 2026 wave.
"git-tanstack.com",
"transformers.pyz",
"/tmp/transformers.pyz",
"With Love TeamPCP",
# Aikido (May-12 wave): payload SHA-256 hashes + Bun marker.
"ab4fcadaec49c03278063dd269ea5eef82d24f2124a8e15d7b90f2fa8601266c",
"2ec78d556d696e208927cc503d48e4b5eb56b31abc2870c2ed2e98d6be27fc96",
"bun run tanstack_runner.js",
"We've been online over 2 hours",
)
# Hard pin-blocks for publicly confirmed malicious versions.
# keep in sync with scripts/scan_npm_packages.py
BLOCKED_NPM_VERSIONS: dict[str, set[str]] = {
# GHSA-g7cv-rxg3-hmpx -- TanStack May-11 2026 (84 versions).
"@tanstack/arktype-adapter": {"1.166.12", "1.166.15"},
"@tanstack/eslint-plugin-router": {"1.161.9", "1.161.12"},
"@tanstack/eslint-plugin-start": {"0.0.4", "0.0.7"},
"@tanstack/history": {"1.161.9", "1.161.12"},
"@tanstack/nitro-v2-vite-plugin": {"1.154.12", "1.154.15"},
"@tanstack/react-router": {"1.169.5", "1.169.8"},
"@tanstack/react-router-devtools": {"1.166.16", "1.166.19"},
"@tanstack/react-router-ssr-query": {"1.166.15", "1.166.18"},
"@tanstack/react-start": {"1.167.68", "1.167.71"},
"@tanstack/react-start-client": {"1.166.51", "1.166.54"},
"@tanstack/react-start-rsc": {"0.0.47", "0.0.50"},
"@tanstack/react-start-server": {"1.166.55", "1.166.58"},
"@tanstack/router-cli": {"1.166.46", "1.166.49"},
"@tanstack/router-core": {"1.169.5", "1.169.8"},
"@tanstack/router-devtools": {"1.166.16", "1.166.19"},
"@tanstack/router-devtools-core": {"1.167.6", "1.167.9"},
"@tanstack/router-generator": {"1.166.45", "1.166.48"},
"@tanstack/router-plugin": {"1.167.38", "1.167.41"},
"@tanstack/router-ssr-query-core": {"1.168.3", "1.168.6"},
"@tanstack/router-utils": {"1.161.11", "1.161.14"},
"@tanstack/router-vite-plugin": {"1.166.53", "1.166.56"},
"@tanstack/solid-router": {"1.169.5", "1.169.8"},
"@tanstack/solid-router-devtools": {"1.166.16", "1.166.19"},
"@tanstack/solid-router-ssr-query": {"1.166.15", "1.166.18"},
"@tanstack/solid-start": {"1.167.65", "1.167.68"},
"@tanstack/solid-start-client": {"1.166.50", "1.166.53"},
"@tanstack/solid-start-server": {"1.166.54", "1.166.57"},
"@tanstack/start-client-core": {"1.168.5", "1.168.8"},
"@tanstack/start-fn-stubs": {"1.161.9", "1.161.12"},
"@tanstack/start-plugin-core": {"1.169.23", "1.169.26"},
"@tanstack/start-server-core": {"1.167.33", "1.167.36"},
"@tanstack/start-static-server-functions": {"1.166.44", "1.166.47"},
"@tanstack/start-storage-context": {"1.166.38", "1.166.41"},
"@tanstack/valibot-adapter": {"1.166.12", "1.166.15"},
"@tanstack/virtual-file-routes": {"1.161.10", "1.161.13"},
"@tanstack/vue-router": {"1.169.5", "1.169.8"},
"@tanstack/vue-router-devtools": {"1.166.16", "1.166.19"},
"@tanstack/vue-router-ssr-query": {"1.166.15", "1.166.18"},
"@tanstack/vue-start": {"1.167.61", "1.167.64"},
"@tanstack/vue-start-client": {"1.166.46", "1.166.49"},
"@tanstack/vue-start-server": {"1.166.50", "1.166.53"},
"@tanstack/zod-adapter": {"1.166.12", "1.166.15"},
# Mini Shai-Hulud May-12 wave: OpenSearch JS client.
"@opensearch-project/opensearch": {"3.5.3", "3.6.2", "3.7.0", "3.8.0"},
# Mini Shai-Hulud May-12 wave: @squawk/* (22 packages, 5 versions each;
# https://safedep.io/mass-npm-supply-chain-attack-tanstack-mistral/).
"@squawk/airport-data": {"0.7.4", "0.7.5", "0.7.6", "0.7.7", "0.7.8"},
"@squawk/airports": {"0.6.2", "0.6.3", "0.6.4", "0.6.5", "0.6.6"},
"@squawk/airspace": {"0.8.1", "0.8.2", "0.8.3", "0.8.4", "0.8.5"},
"@squawk/airspace-data": {"0.5.3", "0.5.4", "0.5.5", "0.5.6", "0.5.7"},
"@squawk/airway-data": {"0.5.4", "0.5.5", "0.5.6", "0.5.7", "0.5.8"},
"@squawk/airways": {"0.4.2", "0.4.3", "0.4.4", "0.4.5", "0.4.6"},
"@squawk/fix-data": {"0.6.4", "0.6.5", "0.6.6", "0.6.7", "0.6.8"},
"@squawk/fixes": {"0.3.2", "0.3.3", "0.3.4", "0.3.5", "0.3.6"},
"@squawk/flight-math": {"0.5.4", "0.5.5", "0.5.6", "0.5.7", "0.5.8"},
"@squawk/flightplan": {"0.5.2", "0.5.3", "0.5.4", "0.5.5", "0.5.6"},
"@squawk/geo": {"0.4.4", "0.4.5", "0.4.6", "0.4.7", "0.4.8"},
"@squawk/icao-registry": {"0.5.2", "0.5.3", "0.5.4", "0.5.5", "0.5.6"},
"@squawk/icao-registry-data": {"0.8.4", "0.8.5", "0.8.6", "0.8.7", "0.8.8"},
"@squawk/mcp": {"0.9.1", "0.9.2", "0.9.3", "0.9.4", "0.9.5"},
"@squawk/navaid-data": {"0.6.4", "0.6.5", "0.6.6", "0.6.7", "0.6.8"},
"@squawk/navaids": {"0.4.2", "0.4.3", "0.4.4", "0.4.5", "0.4.6"},
"@squawk/notams": {"0.3.6", "0.3.7", "0.3.8", "0.3.9", "0.3.10"},
"@squawk/procedure-data": {"0.7.3", "0.7.4", "0.7.5", "0.7.6", "0.7.7"},
"@squawk/procedures": {"0.5.2", "0.5.3", "0.5.4", "0.5.5", "0.5.6"},
"@squawk/types": {"0.8.1", "0.8.2", "0.8.3", "0.8.4", "0.8.5"},
"@squawk/units": {"0.4.3", "0.4.4", "0.4.5", "0.4.6", "0.4.7"},
"@squawk/weather": {"0.5.6", "0.5.7", "0.5.8", "0.5.9", "0.5.10"},
# Mini Shai-Hulud May-12 wave: @uipath/* (64 packages, single version each;
# https://www.aikido.dev/blog/mini-shai-hulud-is-back-tanstack-compromised).
"@uipath/apollo-react": {"4.24.5"},
"@uipath/apollo-wind": {"2.16.2"},
"@uipath/cli": {"1.0.1"},
"@uipath/rpa-tool": {"0.9.5"},
"@uipath/apollo-core": {"5.9.2"},
"@uipath/filesystem": {"1.0.1"},
"@uipath/solutionpackager-tool-core": {"0.0.34"},
"@uipath/solution-tool": {"1.0.1"},
"@uipath/maestro-tool": {"1.0.1"},
"@uipath/codedapp-tool": {"1.0.1"},
"@uipath/agent-tool": {"1.0.1"},
"@uipath/orchestrator-tool": {"1.0.1"},
"@uipath/integrationservice-tool": {"1.0.2"},
"@uipath/rpa-legacy-tool": {"1.0.1"},
"@uipath/vertical-solutions-tool": {"1.0.1"},
"@uipath/flow-tool": {"1.0.2"},
"@uipath/codedagent-tool": {"1.0.1"},
"@uipath/common": {"1.0.1"},
"@uipath/resource-tool": {"1.0.1"},
"@uipath/auth": {"1.0.1"},
"@uipath/docsai-tool": {"1.0.1"},
"@uipath/case-tool": {"1.0.1"},
"@uipath/api-workflow-tool": {"1.0.1"},
"@uipath/test-manager-tool": {"1.0.2"},
"@uipath/robot": {"1.3.4"},
"@uipath/traces-tool": {"1.0.1"},
"@uipath/agent-sdk": {"1.0.2"},
"@uipath/integrationservice-sdk": {"1.0.2"},
"@uipath/maestro-sdk": {"1.0.1"},
"@uipath/data-fabric-tool": {"1.0.2"},
"@uipath/tasks-tool": {"1.0.1"},
"@uipath/insights-tool": {"1.0.1"},
"@uipath/insights-sdk": {"1.0.1"},
"@uipath/uipath-python-bridge": {"1.0.1"},
"@uipath/ap-chat": {"1.5.7"},
"@uipath/project-packager": {"1.1.16"},
"@uipath/packager-tool-case": {"0.0.9"},
"@uipath/packager-tool-workflowcompiler-browser": {"0.0.34"},
"@uipath/packager-tool-connector": {"0.0.19"},
"@uipath/packager-tool-workflowcompiler": {"0.0.16"},
"@uipath/packager-tool-webapp": {"1.0.6"},
"@uipath/packager-tool-apiworkflow": {"0.0.19"},
"@uipath/packager-tool-functions": {"0.1.1"},
"@uipath/widget.sdk": {"1.2.3"},
"@uipath/resources-tool": {"0.1.11"},
"@uipath/agent.sdk": {"0.0.18"},
"@uipath/codedagents-tool": {"0.1.12"},
"@uipath/aops-policy-tool": {"0.3.1"},
"@uipath/solution-packager": {"0.0.35"},
"@uipath/packager-tool-bpmn": {"0.0.9"},
"@uipath/packager-tool-flow": {"0.0.19"},
"@uipath/telemetry": {"0.0.7"},
"@uipath/tool-workflowcompiler": {"0.0.12"},
"@uipath/vss": {"0.1.6"},
"@uipath/solutionpackager-sdk": {"1.0.11"},
"@uipath/ui-widgets-multi-file-upload": {"1.0.1"},
"@uipath/access-policy-tool": {"0.3.1"},
"@uipath/context-grounding-tool": {"0.1.1"},
"@uipath/gov-tool": {"0.3.1"},
"@uipath/admin-tool": {"0.1.1"},
"@uipath/identity-tool": {"0.1.1"},
"@uipath/llmgw-tool": {"1.0.1"},
"@uipath/resourcecatalog-tool": {"0.1.1"},
"@uipath/functions-tool": {"1.0.1"},
"@uipath/access-policy-sdk": {"0.3.1"},
"@uipath/platform-tool": {"1.0.1"},
# Mini Shai-Hulud May-12 wave: @mistralai/* (npm) — separate from PyPI mistralai
# (https://www.aikido.dev/blog/mini-shai-hulud-is-back-tanstack-compromised).
"@mistralai/mistralai": {"2.2.2", "2.2.3", "2.2.4"},
"@mistralai/mistralai-gcp": {"1.7.1", "1.7.2", "1.7.3"},
"@mistralai/mistralai-azure": {"1.7.1", "1.7.2", "1.7.3"},
# Mini Shai-Hulud May-12 wave: @tallyui/* (30 entries, 10 packages)
# (Aikido enumeration).
"@tallyui/components": {"1.0.1", "1.0.2", "1.0.3"},
"@tallyui/connector-medusa": {"1.0.1", "1.0.2", "1.0.3"},
"@tallyui/connector-shopify": {"1.0.1", "1.0.2", "1.0.3"},
"@tallyui/connector-vendure": {"1.0.1", "1.0.2", "1.0.3"},
"@tallyui/connector-woocommerce": {"1.0.1", "1.0.2", "1.0.3"},
"@tallyui/core": {"0.2.1", "0.2.2", "0.2.3"},
"@tallyui/database": {"1.0.1", "1.0.2", "1.0.3"},
"@tallyui/pos": {"0.1.1", "0.1.2", "0.1.3"},
"@tallyui/storage-sqlite": {"0.2.1", "0.2.2", "0.2.3"},
"@tallyui/theme": {"0.2.1", "0.2.2", "0.2.3"},
# Mini Shai-Hulud May-12 wave: @beproduct/nestjs-auth (18 versions)
# (Aikido enumeration).
"@beproduct/nestjs-auth": {
"0.1.2",
"0.1.3",
"0.1.4",
"0.1.5",
"0.1.6",
"0.1.7",
"0.1.8",
"0.1.9",
"0.1.10",
"0.1.11",
"0.1.12",
"0.1.13",
"0.1.14",
"0.1.15",
"0.1.16",
"0.1.17",
"0.1.18",
"0.1.19",
},
# Mini Shai-Hulud May-12 wave: @draftlab/* + @draftauth/*
# (Aikido enumeration).
"@draftauth/client": {"0.2.1", "0.2.2"},
"@draftauth/core": {"0.13.1", "0.13.2"},
"@draftlab/auth": {"0.24.1", "0.24.2"},
"@draftlab/auth-router": {"0.5.1", "0.5.2"},
"@draftlab/db": {"0.16.1"},
# Mini Shai-Hulud May-12 wave: @taskflow-corp/cli + @tolka/cli
# (Aikido enumeration).
"@taskflow-corp/cli": {"0.1.24", "0.1.25", "0.1.26", "0.1.27", "0.1.28", "0.1.29"},
"@tolka/cli": {"1.0.2", "1.0.3", "1.0.4", "1.0.5", "1.0.6"},
# Mini Shai-Hulud May-12 wave: @ml-toolkit-ts/* + @mesadev/* + @dirigible-ai/sdk + @supersurkhet/*
# (Aikido enumeration).
"@dirigible-ai/sdk": {"0.6.2", "0.6.3"},
"@mesadev/rest": {"0.28.3"},
"@mesadev/saguaro": {"0.4.22"},
"@mesadev/sdk": {"0.28.3"},
"@ml-toolkit-ts/preprocessing": {"1.0.2", "1.0.3"},
"@ml-toolkit-ts/xgboost": {"1.0.3", "1.0.4"},
"@supersurkhet/cli": {"0.0.2", "0.0.3", "0.0.4", "0.0.5", "0.0.6", "0.0.7"},
"@supersurkhet/sdk": {"0.0.2", "0.0.3", "0.0.4", "0.0.5", "0.0.6", "0.0.7"},
# Mini Shai-Hulud May-12 wave: Unscoped packages (10 entries)
# (Aikido enumeration).
"safe-action": {"0.8.3", "0.8.4"},
"ts-dna": {"3.0.1", "3.0.2", "3.0.3", "3.0.4"},
"cross-stitch": {"1.1.3", "1.1.4", "1.1.5", "1.1.6"},
"cmux-agent-mcp": {"0.1.3", "0.1.4", "0.1.5", "0.1.6", "0.1.7", "0.1.8"},
"agentwork-cli": {"0.1.4", "0.1.5"},
"git-branch-selector": {"1.3.3", "1.3.4", "1.3.5", "1.3.6", "1.3.7"},
"wot-api": {"0.8.1", "0.8.2", "0.8.3", "0.8.4"},
"git-git-git": {"1.0.8", "1.0.9", "1.0.10", "1.0.11", "1.0.12"},
"nextmove-mcp": {"0.1.3", "0.1.4", "0.1.5", "0.1.6", "0.1.7"},
"ml-toolkit-ts": {"1.0.4", "1.0.5"},
# Cross-ecosystem Mini Shai-Hulud (Apr-30 wave): npm counterpart of
# PyPI lightning 2.6.2/2.6.3. Same threat actor (TeamPCP) per Semgrep,
# Aikido, OX Security, Resecurity. Safe version: 7.0.3 and earlier.
"intercom-client": {"7.0.4"},
}
CARGO_IOC_STRINGS: tuple[str, ...] = (
# Empty by default; the `source` origin check catches the structural
# pattern. Reserved for future cargo-side incidents.
)
# Allowed lockfile origins.
NPM_REGISTRY_PREFIX = "https://registry.npmjs.org/"
NPM_REGISTRY_PREFIXES_ALLOWED: tuple[str, ...] = (NPM_REGISTRY_PREFIX,)
CARGO_REGISTRY_SOURCE = "registry+https://github.com/rust-lang/crates.io-index"
# Cargo non-registry source allowlist: `(crate_name, exact_source_string)`.
# Both must match verbatim; bumping the pinned SHA forces a re-review.
# Studio's Tauri shell pulls `fix-path-env` from git because it is not
# published to crates.io; commit c4c45d5 was reviewed when it landed.
CARGO_SOURCE_ALLOWLIST: tuple[tuple[str, str], ...] = (
(
"fix-path-env",
"git+https://github.com/tauri-apps/fix-path-env-rs#"
"c4c45d503ea115a839aae718d02f79e7c7f0f673",
),
)
class Finding:
__slots__ = ("path", "package", "kind", "detail")
def __init__(self, path: str, package: str, kind: str, detail: str) -> None:
self.path = path
self.package = package
self.kind = kind
self.detail = detail
def __str__(self) -> str:
return (
f" [{self.kind}] {self.path}\n"
f" package: {self.package}\n"
f" detail: {self.detail}"
)
def _gha_escape(text: str) -> str:
"""Escape a string for a GH Actions `::warning::`/`::error::` message.
GH Actions truncates at the first newline unless `\\n`/`\\r` are
escaped as `%0A`/`%0D`. `%` must be replaced first to avoid
double-encoding the subsequent escapes.
"""
return text.replace("%", "%25").replace("\r", "%0D").replace("\n", "%0A")
def audit_npm_lockfile(path: Path) -> list[Finding]:
findings: list[Finding] = []
if not path.exists():
# Missing lockfile is a config error, not a clean audit.
findings.append(
Finding(
path = str(path),
package = "<root>",
kind = "missing-lockfile",
detail = (
"expected lockfile not found; refusing to silently "
"report a clean audit for a path that was not scanned"
),
)
)
return findings
try:
raw = path.read_text(encoding = "utf-8")
except OSError as exc:
# Surface as a finding instead of crashing CI with a traceback.
findings.append(
Finding(
path = str(path),
package = "<root>",
kind = "unreadable-lockfile",
detail = f"could not read file: {exc}",
)
)
return findings
try:
lock = json.loads(raw)
except json.JSONDecodeError as exc:
findings.append(
Finding(
path = str(path),
package = "<root>",
kind = "malformed-lockfile",
detail = f"could not parse as JSON: {exc}",
)
)
return findings
lockfile_version = lock.get("lockfileVersion")
if lockfile_version not in (2, 3):
findings.append(
Finding(
path = str(path),
package = "<root>",
kind = "unsupported-lockfile-version",
detail = (f"only lockfileVersion 2 or 3 audited; got {lockfile_version}"),
)
)
packages = lock.get("packages") or {}
for key, entry in packages.items():
# Empty key "" is the project root (no `resolved`); skip it.
if key == "":
continue
if entry.get("link"):
# Workspace symlink; no tarball to resolve.
continue
resolved = entry.get("resolved")
# Entries nested in another package's node_modules are bundled
# fold-ins covered by the parent's integrity; treat as transparent.
nested = key.count("/node_modules/") >= 1
# 1. resolved-URL origin.
if resolved is None:
if nested or entry.get("bundled"):
# Bundled / fold-in entry; covered by parent integrity.
pass
elif entry.get("version"):
# Top-level entry without a resolved URL is suspicious.
findings.append(
Finding(
path = str(path),
package = key,
kind = "missing-resolved-url",
detail = (
f"version={entry['version']!r} but no `resolved` "
"field; lockfile is incomplete"
),
)
)
else:
if not any(resolved.startswith(p) for p in NPM_REGISTRY_PREFIXES_ALLOWED):
findings.append(
Finding(
path = str(path),
package = key,
kind = "non-registry-resolved-url",
detail = (
f"resolved={resolved!r}; only "
f"{NPM_REGISTRY_PREFIX} is permitted. Direct "
"GitHub / git / file references are the "
"Shai-Hulud injection vector."
),
)
)
# 2. integrity-hash presence.
if resolved is not None and not entry.get("integrity"):
findings.append(
Finding(
path = str(path),
package = key,
kind = "missing-integrity-hash",
detail = (
"no `integrity` field; npm cannot verify the "
"tarball SHA against the registry-published hash"
),
)
)
# 3. Blocked malicious version list.
nm_prefix = "node_modules/"
pkg_name = key[len(nm_prefix) :] if key.startswith(nm_prefix) else key
version = entry.get("version")
blocked = BLOCKED_NPM_VERSIONS.get(pkg_name, set())
if version and version in blocked:
findings.append(
Finding(
path = str(path),
package = key,
kind = "blocked-known-malicious",
detail = (f"{pkg_name}@{version} is on the BLOCKED_NPM_VERSIONS list"),
)
)
# 4. Known IOC strings: scan the raw body to catch fields the
# structural pass doesn't enumerate (scripts, optional deps, etc.).
for ioc in NPM_IOC_STRINGS:
if ioc in raw:
line_no = _first_line_containing(raw, ioc)
findings.append(
Finding(
path = f"{path}:{line_no}" if line_no else str(path),
package = "<ioc-match>",
kind = "known-ioc-string",
detail = (
f"matched known IOC substring {ioc!r}; this is "
"a public indicator of a recent supply-chain "
"compromise. Refuse to install."
),
)
)
return findings
def _first_line_containing(text: str, needle: str) -> int | None:
for i, line in enumerate(text.splitlines(), start = 1):
if needle in line:
return i
return None
# Cargo.lock is TOML; parsed with stdlib tomllib (Python 3.11+).
_PACKAGE_HEADER = re.compile(r"^\[\[package\]\]\s*$")
def audit_cargo_lockfile(path: Path) -> list[Finding]:
findings: list[Finding] = []
if not path.exists():
# See audit_npm_lockfile: missing lockfile is a finding.
findings.append(
Finding(
path = str(path),
package = "<root>",
kind = "missing-lockfile",
detail = (
"expected lockfile not found; refusing to silently "
"report a clean audit for a path that was not scanned"
),
)
)
return findings
try:
raw = path.read_text(encoding = "utf-8")
except OSError as exc:
findings.append(
Finding(
path = str(path),
package = "<root>",
kind = "unreadable-lockfile",
detail = f"could not read file: {exc}",
)
)
return findings
try:
import tomllib # type: ignore[import-not-found]
except ImportError:
# Python <3.11; fall back to a tomli shim if importable.
try:
import tomli as tomllib # type: ignore[no-redef]
except ImportError:
findings.append(
Finding(
path = str(path),
package = "<root>",
kind = "missing-toml-parser",
detail = (
"Python 3.11+ tomllib or tomli is required to "
"parse Cargo.lock; install tomli or upgrade "
"Python before re-running this audit"
),
)
)
return findings
try:
lock = tomllib.loads(raw)
except Exception as exc:
findings.append(
Finding(
path = str(path),
package = "<root>",
kind = "malformed-lockfile",
detail = f"could not parse as TOML: {exc}",
)
)
return findings
for entry in lock.get("package", []):
name = entry.get("name") or "<unnamed>"
version = entry.get("version") or "<unversioned>"
source = entry.get("source")
# Workspace-local crates have no `source` field; skip them.
if source is None:
continue
if source != CARGO_REGISTRY_SOURCE:
if (name, source) in CARGO_SOURCE_ALLOWLIST:
# Pre-approved non-registry source pinned by SHA.
pass
else:
findings.append(
Finding(
path = str(path),
package = f"{name}@{version}",
kind = "non-registry-cargo-source",
detail = (
f"source={source!r}; only "
f"{CARGO_REGISTRY_SOURCE!r} is permitted "
"by default, and no allowlist entry covers "
"this crate. If the source is legitimate, "
"add `(name, source)` to "
"CARGO_SOURCE_ALLOWLIST after reviewing the "
"pinned commit."
),
)
)
if not entry.get("checksum") and source == CARGO_REGISTRY_SOURCE:
findings.append(
Finding(
path = str(path),
package = f"{name}@{version}",
kind = "missing-cargo-checksum",
detail = (
"registry crate without checksum; cargo cannot "
"verify the downloaded source against the "
"registry-published SHA"
),
)
)
for ioc in CARGO_IOC_STRINGS:
if ioc in raw:
line_no = _first_line_containing(raw, ioc)
findings.append(
Finding(
path = f"{path}:{line_no}" if line_no else str(path),
package = "<ioc-match>",
kind = "known-ioc-string",
detail = f"matched known IOC substring {ioc!r}",
)
)
return findings
# Finding kinds split into BLOCKING vs ADVISORY for the default run mode.
# Blocking = public attack indicators (known-malicious version, IOC
# string). Advisory = structural anomalies that warn but don't block.
# --strict makes every finding blocking.
BLOCKING_KINDS: frozenset[str] = frozenset(
{
"blocked-known-malicious",
"known-ioc-string",
# A structurally broken lockfile might hide a real attack.
"malformed-lockfile",
"missing-lockfile",
"unreadable-lockfile",
"missing-toml-parser",
}
)
DEFAULT_NPM_LOCKFILES = (
"studio/frontend/package-lock.json",
"studio/backend/core/data_recipe/oxc-validator/package-lock.json",
"studio/package-lock.json",
)
DEFAULT_CARGO_LOCKFILES = ("studio/src-tauri/Cargo.lock",)
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
description = "Pre-install lockfile supply-chain audit.",
)
parser.add_argument(
"--root",
default = str(REPO_ROOT),
help = "Repo root (default: parent of this script).",
)
parser.add_argument(
"--npm-lockfile",
action = "append",
default = None,
help = (
"Path to a package-lock.json (repeatable). "
"Default: studio/frontend/package-lock.json, "
"studio/backend/core/data_recipe/oxc-validator/package-lock.json, "
"and studio/package-lock.json (Tauri CLI for desktop release)."
),
)
parser.add_argument(
"--cargo-lockfile",
action = "append",
default = None,
help = ("Path to a Cargo.lock (repeatable). Default: studio/src-tauri/Cargo.lock."),
)
parser.add_argument(
"--strict",
action = "store_true",
help = (
"Treat every finding as blocking (exit 1). "
"Default mode only blocks on known-malicious versions, "
"indicator-of-compromise strings, or structurally broken "
"lockfiles; everything else is printed as an advisory "
"warning with exit 0. CI should use the default; local "
"audits aiming for zero noise can opt in via --strict."
),
)
args = parser.parse_args(argv)
# Require a real justification (>=5 chars, not a boolean-shaped token)
# for the skip env var. An invalid value warns and falls through to
# run the audit (fail-safe); a valid one warns and skips with rc=0.
_skip_raw = os.environ.get("UNSLOTH_LOCKFILE_AUDIT_SKIP")
if _skip_raw is not None:
_skip = _skip_raw.strip()
_invalid_tokens = {"", "1", "0", "true", "false", "yes", "no", "on", "off"}
if _skip.lower() in _invalid_tokens or len(_skip) < 5:
print(
"::warning::Lockfile audit skip REQUIRES a justification "
f"value (>=5 chars, not '{_skip_raw}'). Proceeding with "
"audit. Use e.g. UNSLOTH_LOCKFILE_AUDIT_SKIP=ticket-1234.",
file = sys.stderr,
flush = True,
)
else:
print(
f"::warning::Lockfile audit skipped: reason='{_skip}'",
file = sys.stderr,
flush = True,
)
return 0
root = Path(args.root).resolve()
# Explicit flags scope the scan; defaults apply only to no-args CI.
_user_explicit = args.npm_lockfile is not None or args.cargo_lockfile is not None
if _user_explicit:
npm_paths = [root / p for p in (args.npm_lockfile or ())]
cargo_paths = [root / p for p in (args.cargo_lockfile or ())]
else:
npm_paths = [root / p for p in DEFAULT_NPM_LOCKFILES]
cargo_paths = [root / p for p in DEFAULT_CARGO_LOCKFILES]
all_findings: list[Finding] = []
for p in npm_paths:
print(f"[lockfile-audit] npm: {p}", flush = True)
all_findings.extend(audit_npm_lockfile(p))
for p in cargo_paths:
print(f"[lockfile-audit] cargo: {p}", flush = True)
all_findings.extend(audit_cargo_lockfile(p))
if not all_findings:
print(
f"[lockfile-audit] OK: 0 findings across "
f"{len(npm_paths)} npm + {len(cargo_paths)} cargo lockfile(s)",
flush = True,
)
return 0
# Split into blocking (known-malicious / IOC / structurally broken)
# and advisory (everything else). Default mode prints advisories
# without changing the exit code; --strict makes all blocking.
blocking = [f for f in all_findings if f.kind in BLOCKING_KINDS]
advisory = [f for f in all_findings if f.kind not in BLOCKING_KINDS]
if args.strict:
blocking = list(all_findings)
advisory = []
if advisory:
print(
f"\n[lockfile-audit] {len(advisory)} advisory finding(s) "
"(non-blocking; pass --strict to fail the build on these):\n",
file = sys.stderr,
)
for f in advisory:
# GH Actions warning annotation; _gha_escape collapses the
# multi-line Finding onto one line so it renders fully in the UI.
print(f"::warning::{_gha_escape(str(f))}", file = sys.stderr)
print(file = sys.stderr)
if not blocking:
print(
f"[lockfile-audit] OK: {len(advisory)} advisory finding(s), "
"0 blocking. Run with --strict to escalate advisory findings.",
flush = True,
)
return 0
print(
f"\n[lockfile-audit] FAIL: {len(blocking)} blocking finding(s):\n",
file = sys.stderr,
)
for f in blocking:
# Same %-encoding rationale as the advisory branch above.
print(f"::error::{_gha_escape(str(f))}", file = sys.stderr)
print(file = sys.stderr)
print(
"[lockfile-audit] Refusing to proceed. Each blocking finding "
"above is either a public indicator-of-compromise, a known-"
"malicious pinned version, or a structurally broken lockfile. "
"Investigate before running `npm ci` or `cargo fetch`.",
file = sys.stderr,
)
return 1
if __name__ == "__main__":
sys.exit(main())
+360
View File
@@ -0,0 +1,360 @@
#!/usr/bin/env python
# coding: utf-8
"""
Convert Jupyter notebooks (.ipynb) to executable Python scripts (.py).
Converts IPython magics to plain Python:
!command -> subprocess.run('command', shell=True)
%cd path -> os.chdir('path')
%env VAR=value -> os.environ['VAR'] = 'value'
%%file filename -> with open('filename', 'w') as f: f.write(...)
%%capture -> (skipped)
/content/... -> _WORKING_DIR + /...
"""
import nbformat
import re
import shlex
import sys
import os
import urllib.request
import urllib.parse
from pathlib import Path
# Allowlist of hosts for raw notebook fetches; anything else rejected before urlopen.
_ALLOWED_NOTEBOOK_HOSTS = {
"raw.githubusercontent.com",
"gist.githubusercontent.com",
}
# Metacharacters that mean a `!cmd` line can't be a flat argv -> keep shell=True + review marker.
_SHELL_METACHARS_RE = re.compile(r"\$\(|`|\|\||\||&&|>>?|<<?|\*|\?|;")
def needs_fstring(cmd: str) -> bool:
"""Check if command has Python variable interpolation like {var_name}."""
pattern = r"(?<!\$)\{([a-zA-Z_][a-zA-Z0-9_]*)\}"
return bool(re.search(pattern, cmd))
def github_blob_to_raw(url: str) -> str:
"""Convert GitHub blob URL to raw URL."""
# github.com/user/repo/blob/branch/path -> raw.githubusercontent.com/user/repo/branch/path
# Exact host match (not substring) so attacker.example.com/github.com/blob/... is not rewritten.
parsed = urllib.parse.urlparse(url)
if parsed.netloc != "github.com" or "/blob/" not in parsed.path:
return url
new_path = parsed.path.replace("/blob/", "/", 1)
return urllib.parse.urlunparse(
parsed._replace(netloc = "raw.githubusercontent.com", path = new_path)
)
def download_notebook(url: str) -> tuple[str, str]:
"""Download notebook from URL. Returns (content, filename)."""
raw_url = github_blob_to_raw(url)
parsed = urllib.parse.urlparse(raw_url)
filename = os.path.basename(urllib.parse.unquote(parsed.path))
# Host allowlist: refuse to fetch from anything we don't recognise.
host = parsed.hostname
if host not in _ALLOWED_NOTEBOOK_HOSTS:
raise ValueError(
f"Refused notebook fetch from {host!r}: not in allowlist "
f"{sorted(_ALLOWED_NOTEBOOK_HOSTS)}"
)
print(f"Downloading {url}...")
with urllib.request.urlopen(raw_url, timeout = 60) as response:
content = response.read().decode("utf-8")
return content, filename
def is_url(path: str) -> bool:
"""Check if path is a URL."""
return path.startswith("http://") or path.startswith("https://")
def replace_colab_paths(source: str) -> str:
"""Replace Colab-specific /content/ paths with current working directory."""
source = source.replace('"/content/', 'f"{_WORKING_DIR}/')
source = source.replace("'/content/", "f'{_WORKING_DIR}/")
return source
def _emit_shell_command(indent: str, full_cmd: str, *, allow_shell: bool) -> list[str]:
"""Render a `!cmd` notebook line as Python statements.
f-string interpolation, shell metacharacters, or multiline force
shell=True (shlex.split would drop operators), flagged with a
WARNING comment. Otherwise emit shell=False argv form. allow_shell
False makes shell=True emission a hard error.
"""
needs_f = needs_fstring(full_cmd)
has_meta = bool(_SHELL_METACHARS_RE.search(full_cmd))
multiline = "\n" in full_cmd
must_use_shell = needs_f or has_meta or multiline
if must_use_shell:
if not allow_shell:
raise ValueError(
"Cell uses shell metacharacters / interpolation but "
"--no-allow-shell was set; refusing to emit shell=True"
)
warn = f"{indent}# WARNING: shell=True; reviewed for hostile input"
f_prefix = "f" if needs_f else ""
if multiline:
escaped_cmd = full_cmd.replace('"""', r"\"\"\"")
if escaped_cmd.rstrip().endswith('"'):
escaped_cmd = escaped_cmd.rstrip() + " "
stmt = f'{indent}subprocess.run({f_prefix}"""{escaped_cmd}""", shell=True)'
else:
stmt = f"{indent}subprocess.run({f_prefix}{full_cmd!r}, shell=True)"
return [warn, stmt]
return [f"{indent}subprocess.run(shlex.split({full_cmd!r}), shell=False)"]
def convert_cell_to_python(source: str, *, allow_shell: bool = True) -> str:
"""Convert a cell's IPython magics to plain Python."""
lines = source.split("\n")
result = []
i = 0
while i < len(lines):
line = lines[i]
stripped = line.strip()
indent = line[: len(line) - len(line.lstrip())]
if stripped.startswith("%%capture"):
i += 1
continue
if stripped.startswith("%%file "):
filename = stripped[7:].strip()
file_lines = []
i += 1
while i < len(lines):
file_lines.append(lines[i])
i += 1
file_content = "\n".join(file_lines)
file_content = file_content.replace('"""', r"\"\"\"")
result.append(f'{indent}with open({filename!r}, "w") as _f:')
result.append(f'{indent} _f.write("""{file_content}""")')
continue
if stripped.startswith("!"):
cmd_lines = [stripped[1:]]
while cmd_lines[-1].rstrip().endswith("\\") and i + 1 < len(lines):
i += 1
cmd_lines.append(lines[i].strip())
full_cmd = "\n".join(cmd_lines)
result.extend(_emit_shell_command(indent, full_cmd, allow_shell = allow_shell))
# %cd path -> os.chdir(path)
elif stripped.startswith("%cd "):
path = stripped[4:].strip()
result.append(f"{indent}os.chdir({path!r})")
# %env VAR=value
elif stripped.startswith("%env ") and "=" in stripped:
match = re.match(r"%env\s+(\w+)=(.+)", stripped)
if match:
var, val = match.groups()
result.append(f"{indent}os.environ[{var!r}] = {val!r}")
# %env VAR
elif stripped.startswith("%env "):
var = stripped[5:].strip()
result.append(f"{indent}os.environ.get({var!r})")
# %pwd
elif stripped == "%pwd":
result.append(f"{indent}os.getcwd()")
else:
result.append(line)
i += 1
return "\n".join(result)
def convert_notebook(
notebook_content: str,
source_name: str = "notebook",
*,
allow_shell: bool = True,
) -> str:
"""Convert notebook JSON content to Python script."""
# Parse notebook
if isinstance(notebook_content, str):
notebook = nbformat.reads(notebook_content, as_version = 4)
else:
notebook = notebook_content
lines = [
"#!/usr/bin/env python",
"# coding: utf-8",
f"# Converted from: {source_name}",
"",
"import shlex",
"import subprocess",
"import os",
"import sys",
"import re",
"",
"# Capture original packages before any installs",
"_original_packages = subprocess.run(",
" [sys.executable, '-m', 'pip', 'freeze'],",
" capture_output=True, text=True",
").stdout",
"",
"# Working directory (replaces Colab's /content/)",
"_WORKING_DIR = os.getcwd()",
"",
]
for cell in notebook.cells:
source = cell.source.strip()
if not source:
continue
if cell.cell_type == "code":
converted = convert_cell_to_python(source, allow_shell = allow_shell)
converted = replace_colab_paths(converted)
lines.append(converted)
lines.append("")
elif cell.cell_type == "markdown":
for line in source.split("\n"):
lines.append(f"# {line}")
lines.append("")
# Add package restoration at the end
lines.extend(
[
"",
"# Restore original packages (install one by one, skip failures)",
"for _pkg in _original_packages.strip().split('\\n'):",
" if _pkg:",
" subprocess.run([sys.executable, '-m', 'pip', 'install', _pkg, '-q'],",
" stderr=subprocess.DEVNULL)",
"",
]
)
return "\n".join(lines)
def convert_notebook_to_script(
source: str,
output_dir: str | None = None,
*,
allow_shell: bool = True,
):
"""
Convert a notebook to Python script.
Args:
source: Local file path or URL to notebook
output_dir: Output directory (optional, defaults to current directory)
allow_shell: When False, refuse to emit `shell=True` for any
`!cmd` cell that uses metacharacters / interpolation.
"""
if is_url(source):
content, filename = download_notebook(source)
source_name = source
else:
filename = os.path.basename(source)
with open(source, "r", encoding = "utf-8") as f:
content = f.read()
source_name = source
output_filename = filename.replace(".ipynb", ".py")
output_filename = output_filename.replace("(", "").replace(")", "").replace("-", "_")
if output_dir:
output_path = os.path.join(output_dir, output_filename)
else:
output_path = output_filename
script = convert_notebook(content, source_name, allow_shell = allow_shell)
with open(output_path, "w", encoding = "utf-8") as f:
f.write(script)
print(f"Converted {source} -> {output_path}")
return output_path
def main():
import argparse
class Formatter(argparse.ArgumentDefaultsHelpFormatter, argparse.RawDescriptionHelpFormatter):
pass
parser = argparse.ArgumentParser(
description = __doc__,
formatter_class = Formatter,
epilog = """
Examples:
python notebook_to_python.py notebook.ipynb
python notebook_to_python.py -o scripts/ notebook1.ipynb notebook2.ipynb
python notebook_to_python.py --output ./converted https://github.com/user/repo/blob/main/notebook.ipynb
python notebook_to_python.py https://github.com/unslothai/notebooks/blob/main/nb/Oute_TTS_(1B).ipynb
""",
)
parser.add_argument("notebooks", nargs = "+", help = "Notebook files or URLs to convert.")
parser.add_argument("-o", "--output", dest = "output_dir", default = ".", help = "Output directory.")
# Default True for backwards compat; pass --no-allow-shell for untrusted notebooks.
parser.add_argument(
"--allow-shell",
dest = "allow_shell",
action = "store_true",
default = True,
help = "Allow emitting subprocess.run(..., shell=True) for cells "
"that use shell metacharacters or interpolation (default).",
)
parser.add_argument(
"--no-allow-shell",
dest = "allow_shell",
action = "store_false",
help = "Refuse to emit shell=True; cells with metacharacters error out.",
)
args = parser.parse_args()
os.makedirs(args.output_dir, exist_ok = True)
# Track per-notebook failures; continue the loop and exit 1 if any failed.
failures: list[tuple[str, str]] = []
ok = 0
total = len(args.notebooks)
for source in args.notebooks:
try:
convert_notebook_to_script(
source,
output_dir = args.output_dir if args.output_dir != "." else None,
allow_shell = args.allow_shell,
)
ok += 1
except Exception as e:
print(f"ERROR converting {source}: {e}")
failures.append((source, f"{type(e).__name__}: {e}"))
print(
f"converted {ok}/{total}, {len(failures)} failed",
file = sys.stderr if failures else sys.stdout,
)
sys.exit(1 if failures else 0)
if __name__ == "__main__":
main()
File diff suppressed because it is too large Load Diff
+40
View File
@@ -0,0 +1,40 @@
#!/usr/bin/env python3
"""Run a pre-pass (normalize def-signature magic commas + collapse short
multi-line asserts), then `ruff format`, then the kwarg-spacing / import /
string-merge post-pass."""
from __future__ import annotations
import subprocess
import sys
from pathlib import Path
HERE = Path(__file__).resolve().parent
def main(argv: list[str]) -> int:
files = [arg for arg in argv if Path(arg).exists()]
if not files:
return 0
spacing_script = HERE / "enforce_kwargs_spacing.py"
# Pre-ruff: normalize def-signature magic commas and strip the magic comma
# from short multi-line asserts so ruff wraps/joins accordingly.
pre_cmd = [sys.executable, str(spacing_script), "--pre", *files]
pre_proc = subprocess.run(pre_cmd)
if pre_proc.returncode != 0:
return pre_proc.returncode
ruff_cmd = [sys.executable, "-m", "ruff", "format", *files]
ruff_proc = subprocess.run(ruff_cmd)
if ruff_proc.returncode != 0:
return ruff_proc.returncode
spacing_cmd = [sys.executable, str(spacing_script), *files]
spacing_proc = subprocess.run(spacing_cmd)
return spacing_proc.returncode
if __name__ == "__main__":
raise SystemExit(main(sys.argv[1:]))
File diff suppressed because it is too large Load Diff
+5
View File
@@ -0,0 +1,5 @@
{
"_comment": "scan_npm_packages.py allowlist. Each entry is a HIGH/CRITICAL finding manually judged benign. Matched on (package, package-relative path, pattern, evidence hash); a new payload under an already-listed package/path/pattern reopens instead of riding the entry. severity is for review only. Regenerate with --write-baseline AFTER reviewing every line. EMPTY by design: a full scan of studio/frontend/package-lock.json (915 packages) produced 0 findings, so nothing needs suppressing and the CI gate can run enforcing (SCAN_ENFORCE=1) as-is. If a future dependency adds a reviewed-benign HIGH/CRITICAL, add it here rather than weakening a pattern.",
"version": 3,
"entries": []
}
File diff suppressed because it is too large Load Diff
File diff suppressed because one or more lines are too long
+281
View File
@@ -0,0 +1,281 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
"""Stamp and verify display-only Studio release metadata for builds."""
from __future__ import annotations
import argparse
import os
import re
import subprocess
import sys
import tarfile
import tempfile
import zipfile
from pathlib import Path
def _atomic_write_text(
path: Path,
data: str,
encoding: str = "utf-8",
) -> None:
"""Atomic ``Path.write_text``: a crash mid-write leaves the prior file
intact, so the build never reads a partial ``_studio_release_build.py``."""
dirpath = str(path.parent) or "."
path.parent.mkdir(parents = True, exist_ok = True)
fd, tmp_path = tempfile.mkstemp(prefix = ".stamp_studio.", dir = dirpath)
try:
with os.fdopen(fd, "w", encoding = encoding) as handle:
handle.write(data)
handle.flush()
os.fsync(handle.fileno())
os.replace(tmp_path, path)
except Exception:
try:
os.unlink(tmp_path)
except OSError:
pass
raise
REPO_ROOT = Path(__file__).resolve().parents[1]
BUILD_INFO_PATH = REPO_ROOT / "studio" / "backend" / "utils" / "_studio_release_build.py"
BUILD_INFO_SUFFIX = "studio/backend/utils/_studio_release_build.py"
VERSION_RE = re.compile(r"^v\d+\.\d+\.\d+(?:-[0-9A-Za-z.][0-9A-Za-z.-]*)?$")
GIT_DESCRIBE_SUFFIX_RE = re.compile(r"-\d+-g[0-9A-Fa-f]+(?:-dirty)?$")
MAX_VERSION_LENGTH = 64
PLACEHOLDER = """# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
\"\"\"Build-stamped Studio release metadata.
Release builds may rewrite this module in the build workspace before creating
Python artifacts. Keep the committed value neutral so source checkouts do not
accidentally report a stale release tag.
\"\"\"
STUDIO_RELEASE_VERSION = None
"""
def is_valid_version(value: object) -> bool:
if not isinstance(value, str):
return False
version = value.strip()
if not version or len(version) > MAX_VERSION_LENGTH:
return False
if version.endswith("-dirty") or GIT_DESCRIBE_SUFFIX_RE.search(version):
return False
return VERSION_RE.fullmatch(version) is not None
def _exact_git_tag() -> str | None:
try:
result = subprocess.run(
[
"git",
"describe",
"--tags",
"--exact-match",
"--match",
"v[0-9]*",
"HEAD",
],
cwd = REPO_ROOT,
check = False,
stdout = subprocess.PIPE,
stderr = subprocess.DEVNULL,
text = True,
timeout = 2.0,
)
except (OSError, subprocess.TimeoutExpired):
return None
if result.returncode != 0:
return None
tag = result.stdout.strip()
return tag if is_valid_version(tag) else None
def _git_worktree_is_dirty() -> bool:
try:
result = subprocess.run(
["git", "status", "--porcelain"],
cwd = REPO_ROOT,
check = False,
stdout = subprocess.PIPE,
stderr = subprocess.DEVNULL,
text = True,
timeout = 2.0,
)
except (OSError, subprocess.TimeoutExpired):
return True
if result.returncode != 0:
return True
return bool(result.stdout.strip())
def _github_tag() -> str | None:
if os.environ.get("GITHUB_REF_TYPE") != "tag":
return None
github_ref = os.environ.get("GITHUB_REF_NAME", "").strip()
return github_ref or None
def resolve_version() -> tuple[str | None, str]:
env_version = os.environ.get("UNSLOTH_STUDIO_RELEASE_VERSION", "").strip()
if env_version:
return (env_version, "UNSLOTH_STUDIO_RELEASE_VERSION")
github_ref = _github_tag()
if github_ref:
return (github_ref, "GITHUB_REF_NAME")
git_tag = _exact_git_tag()
if git_tag:
return (git_tag, "exact git tag")
return (None, "none")
def build_info_source(version: str | None) -> str:
literal = repr(version) if version is not None else "None"
return f'''# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
"""Build-stamped Studio release metadata."""
STUDIO_RELEASE_VERSION = {literal}
'''
def _env_version_conflicts(version: str) -> list[tuple[str, str]]:
conflicts: list[tuple[str, str]] = []
github_ref = _github_tag()
if github_ref and is_valid_version(github_ref) and github_ref != version:
conflicts.append(("GITHUB_REF_NAME", github_ref))
git_tag = _exact_git_tag()
if git_tag and git_tag != version:
conflicts.append(("exact git tag", git_tag))
return conflicts
def stamp(require_release: bool) -> int:
version, source = resolve_version()
if version is not None and not is_valid_version(version):
print(
f"Invalid Studio release version from {source}: {version!r}",
file = sys.stderr,
)
return 2
if version is not None and source == "UNSLOTH_STUDIO_RELEASE_VERSION":
conflicts = _env_version_conflicts(version)
if conflicts:
details = ", ".join(f"{name}={value!r}" for name, value in conflicts)
print(
"UNSLOTH_STUDIO_RELEASE_VERSION does not match available "
f"release tag metadata: {details}",
file = sys.stderr,
)
return 2
if require_release and source == "exact git tag" and _git_worktree_is_dirty():
print(
"Refusing to publish from a dirty exact-tag checkout. Set "
"UNSLOTH_STUDIO_RELEASE_VERSION explicitly from release automation "
"or publish from a clean tag checkout.",
file = sys.stderr,
)
return 2
if version is None:
if require_release:
print(
"No Studio release version available. Set "
"UNSLOTH_STUDIO_RELEASE_VERSION, build from a GitHub tag, "
"or run from an exact local Studio release tag.",
file = sys.stderr,
)
return 2
_atomic_write_text(BUILD_INFO_PATH, PLACEHOLDER, encoding = "utf-8")
print("dev")
return 0
_atomic_write_text(BUILD_INFO_PATH, build_info_source(version), encoding = "utf-8")
print(f"Stamping Studio release version {version} from {source}", file = sys.stderr)
print(version)
return 0
def _read_wheel_member(path: Path) -> str | None:
with zipfile.ZipFile(path) as archive:
for name in archive.namelist():
if name.endswith(BUILD_INFO_SUFFIX):
return archive.read(name).decode("utf-8")
return None
def _read_sdist_member(path: Path) -> str | None:
with tarfile.open(path) as archive:
for member in archive.getmembers():
if member.name.endswith(BUILD_INFO_SUFFIX):
extracted = archive.extractfile(member)
if extracted is None:
return None
return extracted.read().decode("utf-8")
return None
def verify_dist(expected: str, dist_dir: Path) -> int:
if not is_valid_version(expected):
print(f"Invalid expected Studio release version: {expected!r}", file = sys.stderr)
return 2
artifacts = list(dist_dir.glob("*.whl")) + list(dist_dir.glob("*.tar.gz"))
if not artifacts:
print(f"No wheel or sdist artifacts found in {dist_dir}", file = sys.stderr)
return 2
expected_line = f"STUDIO_RELEASE_VERSION = {expected!r}"
failures: list[str] = []
for artifact in artifacts:
if artifact.suffix == ".whl":
content = _read_wheel_member(artifact)
else:
content = _read_sdist_member(artifact)
if content is None:
failures.append(f"{artifact.name}: missing {BUILD_INFO_SUFFIX}")
elif expected_line not in content:
failures.append(f"{artifact.name}: Studio release version mismatch")
if failures:
for failure in failures:
print(failure, file = sys.stderr)
return 2
print(f"Verified Studio release version {expected} in {len(artifacts)} artifact(s)")
return 0
def main() -> int:
parser = argparse.ArgumentParser(description = __doc__)
parser.add_argument("--require-release", action = "store_true")
parser.add_argument("--verify-dist", type = Path)
parser.add_argument("--expected")
args = parser.parse_args()
if args.verify_dist is not None:
if not args.expected:
parser.error("--verify-dist requires --expected")
return verify_dist(args.expected, args.verify_dist)
return stamp(require_release = args.require_release)
if __name__ == "__main__":
raise SystemExit(main())
+143
View File
@@ -0,0 +1,143 @@
#!/usr/bin/env python3
"""Keep `allowScripts` pins in studio/frontend/package.json in sync with
package-lock.json.
`npm approve-scripts` writes version-pinned entries ("pkg@1.2.3": true).
A dependency bump strands the pin, so the approval (or denial) silently
stops matching and the package's install scripts fall back to
"unreviewed". This tool re-pins existing entries to the versions the
lockfile actually resolves; it never adds or removes entries, so
approving a brand-new script-bearing package stays a human decision.
Usage:
python scripts/sync_allow_scripts_pins.py --check # CI: exit 1 on drift
python scripts/sync_allow_scripts_pins.py --fix # rewrite package.json
Pinned keys follow npm's allowScripts grammar: "name@1.2.3" or
"name@1.2.3 || 1.2.4". Bare names (no version) match every version and
are left alone. Entries whose range is not an exact-version disjunction
(wildcards, tags) are left alone too.
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[1]
DEFAULT_DIR = REPO_ROOT / "studio" / "frontend"
EXACT_VERSION_RE = re.compile(r"^\d+\.\d+\.\d+(?:[-+][0-9A-Za-z.+-]+)?$")
def split_spec(key: str) -> tuple[str, str | None]:
"""'@scope/name@1.2.3' -> ('@scope/name', '1.2.3'); bare names -> (key, None)."""
if key.startswith("@"):
rest = key[1:]
if "@" not in rest:
return key, None
name, rng = rest.split("@", 1)
return "@" + name, rng
if "@" not in key:
return key, None
name, rng = key.split("@", 1)
return name, rng
def is_exact_disjunction(rng: str) -> bool:
parts = [p.strip() for p in rng.split("||")]
return all(EXACT_VERSION_RE.match(p) for p in parts) and bool(parts)
def version_sort_key(version: str) -> tuple:
release = version.split("-", 1)[0].split("+", 1)[0]
return tuple(int(x) for x in release.split(".")), version
def script_versions_from_lock(lock: dict) -> dict[str, list[str]]:
"""Map package name -> sorted versions that carry install scripts."""
out: dict[str, set[str]] = {}
for path, meta in (lock.get("packages") or {}).items():
if not path or not meta.get("hasInstallScript"):
continue
name = path.rsplit("node_modules/", 1)[-1]
version = meta.get("version")
if name and version:
out.setdefault(name, set()).add(version)
return {n: sorted(vs, key = version_sort_key) for n, vs in out.items()}
def desired_key(name: str, versions: list[str]) -> str:
return f"{name}@{' || '.join(versions)}"
def compute_renames(policy: dict, lock_versions: dict[str, list[str]]) -> dict[str, str]:
renames: dict[str, str] = {}
for key in policy:
name, rng = split_spec(key)
if rng is None or not is_exact_disjunction(rng):
continue # bare name or non-exact spec: matches by name, never stale
versions = lock_versions.get(name)
if not versions:
continue # package gone or script-free now: stale pin is inert
want = desired_key(name, versions)
if key != want:
renames[key] = want
return renames
def main(argv: list[str] | None = None) -> int:
ap = argparse.ArgumentParser(description = __doc__)
mode = ap.add_mutually_exclusive_group(required = True)
mode.add_argument("--check", action = "store_true", help = "exit 1 if pins are stale")
mode.add_argument("--fix", action = "store_true", help = "rewrite package.json in place")
ap.add_argument(
"--dir",
type = Path,
default = DEFAULT_DIR,
help = "directory holding package.json + package-lock.json",
)
args = ap.parse_args(argv)
pkg_path = args.dir / "package.json"
lock_path = args.dir / "package-lock.json"
if not pkg_path.exists() or not lock_path.exists():
print(f"sync-allow-scripts: nothing to do ({args.dir} has no package.json + lockfile)")
return 0
pkg = json.loads(pkg_path.read_text(encoding = "utf-8"))
policy = pkg.get("allowScripts")
if not isinstance(policy, dict) or not policy:
print("sync-allow-scripts: no allowScripts policy in package.json, nothing to do")
return 0
lock = json.loads(lock_path.read_text(encoding = "utf-8"))
renames = compute_renames(policy, script_versions_from_lock(lock))
if not renames:
print(f"sync-allow-scripts: {len(policy)} allowScripts entries in sync with the lockfile")
return 0
for old, new in renames.items():
print(f' stale pin: "{old}" -> "{new}"')
if args.check:
print(
"sync-allow-scripts: pins are stale; run "
"`python scripts/sync_allow_scripts_pins.py --fix` and commit the result"
)
return 1
pkg["allowScripts"] = {renames.get(k, k): v for k, v in policy.items()}
pkg_path.write_text(json.dumps(pkg, indent = 2, ensure_ascii = False) + "\n", encoding = "utf-8")
print(
f"sync-allow-scripts: re-pinned {len(renames)} entr{'y' if len(renames) == 1 else 'ies'} in {pkg_path}"
)
return 0
if __name__ == "__main__":
sys.exit(main())
+496
View File
@@ -0,0 +1,496 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
#
# Unsloth Studio uninstaller for Windows PowerShell.
# Stops running servers and removes install dir, launcher data, CLI shim,
# desktop and Start Menu shortcuts, the user PATH entry, and the PathBackup
# registry key. Honors custom roots set via UNSLOTH_STUDIO_HOME / STUDIO_HOME
# at install time (read back from share\studio.conf).
#
# Usage: irm https://raw.githubusercontent.com/unslothai/unsloth/main/scripts/uninstall.ps1 | iex
# Local: Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass; .\scripts\uninstall.ps1
function Uninstall-UnslothStudio {
$ErrorActionPreference = "Continue"
function _Step { param([string]$Msg) Write-Host $Msg }
function _Substep { param([string]$Msg, [string]$Color = "Gray") Write-Host " $Msg" -ForegroundColor $Color }
# Remove a file/dir/symlink if present. Idempotent; retries since a just-killed
# process can briefly hold a handle (Windows refuses the delete until released).
function _RemovePath {
param([string]$Path)
if ([string]::IsNullOrWhiteSpace($Path)) { return }
if (-not (Test-Path -LiteralPath $Path)) { return }
for ($attempt = 1; $attempt -le 4; $attempt++) {
try {
Remove-Item -LiteralPath $Path -Recurse -Force -ErrorAction Stop
} catch {
if ($attempt -lt 4) { Start-Sleep -Milliseconds 700; continue }
_Substep "could not remove: $Path ($($_.Exception.Message))" "Yellow"
return
}
# Remove-Item -Recurse can report success yet leave a transiently-locked
# child (e.g. unsloth.ico in Explorer's icon cache); verify + retry so we
# never falsely claim "removed" or orphan the dir.
if (-not (Test-Path -LiteralPath $Path)) {
_Substep "removed: $Path" "Green"
return
}
if ($attempt -lt 4) { Start-Sleep -Milliseconds 700; continue }
_Substep "still present (files held open): $Path" "Yellow"
}
}
# Remove the shared data dir, but keep unsloth.ico if a WSL shortcut still points
# at it (else that shortcut blanks); uninstall.sh drops it when WSL is removed.
function _RemoveDataDirKeepingWslIcon {
param(
[string]$DataDir,
# WSL-shortcut search dirs; default Start Menu + Desktop, overridable for tests.
[string[]]$ShortcutDirs = $null
)
if ([string]::IsNullOrWhiteSpace($DataDir)) { return }
if (-not (Test-Path -LiteralPath $DataDir)) { return }
# $null = not passed (use defaults); test $null not truthiness so an explicit
# @() is honored (-not @() is $true).
if ($null -eq $ShortcutDirs) {
# Guard $env:APPDATA: it can be unset in service/CI Windows contexts, where
# an unguarded Join-Path emits a noisy parameter-binding error.
$ShortcutDirs = @()
if (-not [string]::IsNullOrWhiteSpace($env:APPDATA)) {
$ShortcutDirs += Join-Path $env:APPDATA "Microsoft\Windows\Start Menu\Programs"
}
try {
$desktop = [Environment]::GetFolderPath("Desktop")
if (-not [string]::IsNullOrWhiteSpace($desktop)) { $ShortcutDirs += $desktop }
} catch {}
}
$wslShortcuts = @()
foreach ($d in $ShortcutDirs) {
if ($d -and (Test-Path -LiteralPath $d)) {
$wslShortcuts += Get-ChildItem -LiteralPath $d -Filter "Unsloth Studio (WSL*.lnk" -ErrorAction SilentlyContinue
}
}
if (@($wslShortcuts).Count -eq 0) {
_RemovePath $DataDir
return
}
# A WSL shortcut survives: drop everything except its shared icon.
_Substep "keeping $(Join-Path $DataDir 'unsloth.ico') for the WSL shortcut" "Gray"
Get-ChildItem -LiteralPath $DataDir -Force -ErrorAction SilentlyContinue | ForEach-Object {
if ($_.Name -ne "unsloth.ico") { _RemovePath $_.FullName }
}
}
# A path is a Studio-owned root iff one of install.ps1's sentinels exists:
# <root>\share\studio.conf, <root>\unsloth_studio\.unsloth-studio-owned,
# or <root>\bin\unsloth.exe.
function _IsStudioRoot {
param([string]$Path)
if ([string]::IsNullOrWhiteSpace($Path)) { return $false }
if (Test-Path -LiteralPath (Join-Path $Path "share\studio.conf") -PathType Leaf) { return $true }
if (Test-Path -LiteralPath (Join-Path $Path "unsloth_studio\.unsloth-studio-owned") -PathType Leaf) { return $true }
if (Test-Path -LiteralPath (Join-Path $Path "bin\unsloth.exe") -PathType Leaf) { return $true }
return $false
}
# Hard deny list. Refuse to recursively delete drive roots, USERPROFILE
# itself, parent of USERPROFILE, or system directories.
function _IsUnsafeRoot {
param([string]$Path)
if ([string]::IsNullOrWhiteSpace($Path)) { return $true }
$norm = $null
try { $norm = [System.IO.Path]::GetFullPath($Path).TrimEnd('\','/') } catch { return $true }
if ([string]::IsNullOrWhiteSpace($norm)) { return $true }
# Drive root, e.g. C:\
if ($norm -match '^[A-Za-z]:[\\/]?$') { return $true }
$userProfile = $env:USERPROFILE
if ($userProfile) {
$userProfile = $userProfile.TrimEnd('\','/')
if ($norm -ieq $userProfile) { return $true }
try {
$parent = Split-Path -LiteralPath $userProfile -Parent
if ($parent -and ($norm -ieq $parent.TrimEnd('\','/'))) { return $true }
} catch { }
}
$systemRoots = @(
$env:SystemRoot, $env:windir, $env:ProgramFiles, ${env:ProgramFiles(x86)},
$env:ProgramData, $env:APPDATA, $env:LOCALAPPDATA
)
foreach ($s in $systemRoots) {
if (-not [string]::IsNullOrWhiteSpace($s)) {
$s2 = $s.TrimEnd('\','/')
if ($norm -ieq $s2) { return $true }
}
}
return $false
}
# Parse UNSLOTH_EXE='<path>' out of a share\studio.conf and return the
# implied install root (three dirnames up from the venv exe).
function _RootFromConf {
param([string]$ConfFile)
if (-not (Test-Path -LiteralPath $ConfFile -PathType Leaf)) { return $null }
$line = Get-Content -LiteralPath $ConfFile -ErrorAction SilentlyContinue |
Where-Object { $_ -match "^UNSLOTH_EXE\s*=" } | Select-Object -First 1
if (-not $line) { return $null }
# Tolerate ' value ' single-quoted with '' -> ' apostrophe escape.
if ($line -match "^UNSLOTH_EXE\s*=\s*'(.*)'\s*$") {
$exe = $Matches[1] -replace "''", "'"
try {
$bin = Split-Path -LiteralPath $exe -Parent
$studio = Split-Path -LiteralPath $bin -Parent
$root = Split-Path -LiteralPath $studio -Parent
if ($root) { return $root }
} catch { }
}
return $null
}
# Expand a leading ~ or ~/ ~\ to $env:USERPROFILE so env-mode roots
# written with the tilde shape install.ps1 supports (lines 152-154) are
# found here too.
function _ExpandTilde {
param([string]$Path)
if ([string]::IsNullOrWhiteSpace($Path)) { return $Path }
$p = $Path.Trim()
if ($p -eq '~') { return $env:USERPROFILE }
if ($p.StartsWith('~/') -or $p.StartsWith('~\')) {
if ($env:USERPROFILE) {
return (Join-Path $env:USERPROFILE $p.Substring(2).TrimStart('/','\'))
}
}
return $p
}
# Discover non-default Studio roots from env vars + studio.conf files.
# Mirrors install.ps1's precedence: UNSLOTH_STUDIO_HOME wins, STUDIO_HOME
# is ignored when both are set, so uninstalling install A doesn't also
# delete install B if the user has a stale STUDIO_HOME pointing at B.
function _CustomStudioRoots {
$seen = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::OrdinalIgnoreCase)
$defaultRoot = $null
if ($env:USERPROFILE) {
$defaultRoot = (Join-Path $env:USERPROFILE ".unsloth\studio")
}
$emit = {
param($Path)
if ([string]::IsNullOrWhiteSpace($Path)) { return }
$expanded = _ExpandTilde $Path
$norm = $null
try { $norm = [System.IO.Path]::GetFullPath($expanded).TrimEnd('\','/') } catch { return }
if (-not $norm) { return }
if ($defaultRoot -and ($norm -ieq $defaultRoot.TrimEnd('\','/'))) { return }
if ($seen.Add($norm)) { Write-Output $norm }
}
$envRoot = $null
if ($env:UNSLOTH_STUDIO_HOME) {
$envRoot = $env:UNSLOTH_STUDIO_HOME
} elseif ($env:STUDIO_HOME) {
$envRoot = $env:STUDIO_HOME
}
if ($envRoot) {
$expandedEnv = _ExpandTilde $envRoot
& $emit $expandedEnv
$confRoot = _RootFromConf (Join-Path $expandedEnv "share\studio.conf")
if ($confRoot) { & $emit $confRoot }
}
# Default-mode conf at LOCALAPPDATA\Unsloth Studio.
if ($env:LOCALAPPDATA) {
$confRoot = _RootFromConf (Join-Path $env:LOCALAPPDATA "Unsloth Studio\studio.conf")
if ($confRoot) { & $emit $confRoot }
}
}
# Return $true iff the PID's image path lives under one of $KnownRoots.
# Prevents killing an unrelated process that happens to listen on a stale
# Studio port.
function _PidUnderKnownRoot {
param([int]$Pid_, [string[]]$KnownRoots)
if (-not $KnownRoots -or $KnownRoots.Count -eq 0) { return $false }
try {
$proc = Get-CimInstance Win32_Process -Filter "ProcessId=$Pid_" -ErrorAction SilentlyContinue
if (-not $proc) { return $false }
$exe = $proc.ExecutablePath
if (-not $exe) { return $false }
foreach ($r in $KnownRoots) {
if ($r -and ($exe -ilike "$r\*")) { return $true }
}
} catch { }
return $false
}
# Stop a Studio backend whose port is recorded in <DataDir>\studio.port.
# Only kills if the listening PID's exe path is under a known Studio root.
function _StopByPortFile {
param([string]$PortFile, [string[]]$KnownRoots)
if (-not (Test-Path -LiteralPath $PortFile -PathType Leaf)) { return }
$port = Get-Content -LiteralPath $PortFile -ErrorAction SilentlyContinue | Select-Object -First 1
if ($port) { $port = $port.Trim() }
if (-not ($port -match '^[0-9]+$')) {
Remove-Item -LiteralPath $PortFile -Force -ErrorAction SilentlyContinue
return
}
try {
$conns = Get-NetTCPConnection -State Listen -LocalPort ([int]$port) -ErrorAction SilentlyContinue
foreach ($c in $conns) {
if (-not (_PidUnderKnownRoot -Pid_ ([int]$c.OwningProcess) -KnownRoots $KnownRoots)) { continue }
try {
Stop-Process -Id $c.OwningProcess -Force -ErrorAction SilentlyContinue
} catch { }
}
} catch {
# netstat fallback for older PowerShell. Require LISTENING state so
# we never kill a process whose remote endpoint just happens to be
# the cached port (browser -> :443 etc.).
try {
$lines = & netstat.exe -ano 2>$null |
Select-String -Pattern "LISTENING" |
Select-String -Pattern ":$port\s"
foreach ($l in $lines) {
$parts = ($l.ToString() -split '\s+') | Where-Object { $_ }
$pid_ = $parts[-1]
if ($pid_ -match '^\d+$') {
if (-not (_PidUnderKnownRoot -Pid_ ([int]$pid_) -KnownRoots $KnownRoots)) { continue }
try { Stop-Process -Id ([int]$pid_) -Force -ErrorAction SilentlyContinue } catch { }
}
}
} catch { }
}
Remove-Item -LiteralPath $PortFile -Force -ErrorAction SilentlyContinue
}
# Stop processes whose ExecutablePath lives under an unsloth_studio venv.
# Anchoring on the venv path avoids matching unrelated python.exe / studio.exe.
function _StopStudioProcesses {
param([string[]]$KnownRoots)
try {
$procs = Get-CimInstance Win32_Process -ErrorAction SilentlyContinue |
Where-Object {
$_.ExecutablePath -and ($_.ExecutablePath -match '\\unsloth_studio\\.*\\(unsloth|python|studio)\.exe$') -and
$_.CommandLine -and ($_.CommandLine -match 'studio')
}
foreach ($p in $procs) {
# Optional scope: only kill if the exe is under a known root.
if ($KnownRoots) {
$match = $false
foreach ($r in $KnownRoots) {
if ($p.ExecutablePath -and ($p.ExecutablePath -ilike "$r\*")) { $match = $true; break }
}
if (-not $match) { continue }
}
try {
Stop-Process -Id $p.ProcessId -Force -ErrorAction SilentlyContinue
} catch { }
}
} catch { }
}
# Stop processes that would block deleting the paths we remove. Unlike
# _StopStudioProcesses (venv exe only), this also catches llama-server/llama-cli,
# the unsloth.exe shim, and orphaned mp workers under SYSTEM python holding a
# venv DLL (an open DLL handle blocks the dir delete) -- found by scanning each
# candidate's loaded modules, not just its image path.
function _StopProcessesLockingRoots {
param([string[]]$Roots)
$clean = @($Roots | Where-Object { $_ } | ForEach-Object { $_.TrimEnd('\','/') })
if ($clean.Count -eq 0) { return }
$underRoot = {
param($p)
if (-not $p) { return $false }
foreach ($r in $clean) { if ($p -ieq $r -or $p -ilike "$r\*") { return $true } }
return $false
}
# 1. Image path under a target root (venv python, shim, llama-server).
try {
foreach ($proc in (Get-CimInstance Win32_Process -ErrorAction SilentlyContinue)) {
if ((& $underRoot $proc.ExecutablePath)) {
try { Stop-Process -Id $proc.ProcessId -Force -ErrorAction SilentlyContinue } catch { }
}
}
} catch { }
# 2. A loaded module under a target root (orphaned mp-fork python holding a
# venv DLL). Scoped to names that load our DLLs to keep the scan fast.
try {
$cands = Get-Process -Name python, pythonw, unsloth, llama-server, llama-cli -ErrorAction SilentlyContinue
foreach ($proc in $cands) {
$hit = $false
try {
foreach ($m in $proc.Modules) { if ((& $underRoot $m.FileName)) { $hit = $true; break } }
} catch { } # access denied enumerating modules -> skip
if ($hit) { try { Stop-Process -Id $proc.Id -Force -ErrorAction SilentlyContinue } catch { } }
}
} catch { }
}
# Default install root + default data dir.
$defaultStudioHome = if ($env:USERPROFILE) { Join-Path $env:USERPROFILE ".unsloth\studio" } else { $null }
$defaultDataDir = if ($env:LOCALAPPDATA) { Join-Path $env:LOCALAPPDATA "Unsloth Studio" } else { $null }
# Default-mode ~/.unsloth holds a SHARED llama.cpp build + .cache that are
# siblings of studio (not under it), so deleting <studio> misses them -- handle
# explicitly. No-op in env/custom mode (nested under the custom root, removed
# with it). A user-set UNSLOTH_LLAMA_CPP_PATH is left alone.
$defaultUnslothHome = if ($env:USERPROFILE) { Join-Path $env:USERPROFILE ".unsloth" } else { $null }
$defaultLlamaCpp = if ($defaultUnslothHome) { Join-Path $defaultUnslothHome "llama.cpp" } else { $null }
$defaultCache = if ($defaultUnslothHome) { Join-Path $defaultUnslothHome ".cache" } else { $null }
# Isolated Node.js runtime (install_node_prebuilt.py), a sibling of studio in
# default mode. No-op in env/custom mode (nested under the custom root) and absent.
$defaultNode = if ($defaultUnslothHome) { Join-Path $defaultUnslothHome "node" } else { $null }
# llama.cpp atomic-install staging root (install_llama_prebuilt.py .staging,
# sibling of the install dir). Usually pruned after activate, but an interrupted
# build can leave a "<name>.staging-XXXX" tree; removing it lets the empty-dir
# cleanup of ~/.unsloth below succeed. No-op in env/custom mode and when absent.
$defaultStaging = if ($defaultUnslothHome) { Join-Path $defaultUnslothHome ".staging" } else { $null }
# Build known-root list FIRST so the port-file kill can verify ownership.
$customRoots = @(_CustomStudioRoots)
$knownRoots = @()
if ($defaultStudioHome) { $knownRoots += $defaultStudioHome }
$knownRoots += $customRoots
# ── Stop running servers ──
_Step "Stopping any running Unsloth Studio servers..."
if ($defaultDataDir) {
_StopByPortFile -PortFile (Join-Path $defaultDataDir "studio.port") -KnownRoots $knownRoots
}
foreach ($r in $customRoots) {
_StopByPortFile -PortFile (Join-Path $r "share\studio.port") -KnownRoots $knownRoots
}
_StopStudioProcesses -KnownRoots $knownRoots
# Also stop anything holding a handle on the exact paths we delete (llama-server,
# the CLI shim, an mp-fork python with a venv DLL) so the dir delete isn't refused.
_StopProcessesLockingRoots -Roots (@($knownRoots) + @($defaultDataDir, $defaultLlamaCpp, $defaultCache, $defaultNode))
# ── Remove custom-root install trees ──
_Step "Removing data and install directories..."
foreach ($r in $customRoots) {
if (_IsUnsafeRoot $r) {
_Substep "refusing to remove unsafe path: $r" "Yellow"
continue
}
if (-not (_IsStudioRoot $r)) {
_Substep "refusing to remove non-Studio path: $r" "Yellow"
continue
}
_RemovePath $r
}
# Default install dir (always at %USERPROFILE%\.unsloth\studio when present).
if ($defaultStudioHome) { _RemovePath $defaultStudioHome }
# Default data dir.
if ($defaultDataDir) { _RemoveDataDirKeepingWslIcon $defaultDataDir }
# Default-mode shared llama.cpp build + cache (siblings of studio under
# ~/.unsloth). No-op in env/custom mode and when absent.
if ($defaultLlamaCpp) { _RemovePath $defaultLlamaCpp }
if ($defaultCache) { _RemovePath $defaultCache }
# Isolated Node.js runtime (sibling of studio under ~/.unsloth). No-op in env/
# custom mode (nested under the custom root, removed with it) and when absent.
if ($defaultNode) { _RemovePath $defaultNode }
if ($defaultStaging) { _RemovePath $defaultStaging }
# llama.cpp install lock (serializes the shared build); a stray lock keeps
# ~/.unsloth from being pruned below. No-op in env/custom mode and when absent.
if ($defaultUnslothHome) { _RemovePath (Join-Path $defaultUnslothHome ".llama.cpp.install.lock") }
# Drop ~/.unsloth itself, but ONLY if now empty -- never nuke unrelated content.
if ($defaultUnslothHome -and (Test-Path -LiteralPath $defaultUnslothHome) -and
-not (Get-ChildItem -LiteralPath $defaultUnslothHome -Force -ErrorAction SilentlyContinue)) {
_RemovePath $defaultUnslothHome
}
# ── Remove desktop and Start Menu shortcuts ──
_Step "Removing desktop and Start Menu shortcuts..."
try {
$desktop = [Environment]::GetFolderPath("Desktop")
if ($desktop) { _RemovePath (Join-Path $desktop "Unsloth Studio.lnk") }
} catch { }
if ($env:APPDATA) {
_RemovePath (Join-Path $env:APPDATA "Microsoft\Windows\Start Menu\Programs\Unsloth Studio.lnk")
}
# Invalidate the Win11 Start Menu tile cache so the removed shortcut's tile
# disappears promptly instead of lingering stale (mirrors install.ps1's
# New-StudioShortcuts). Preserves start2.bin (the pin layout).
try {
$smehTemp = Join-Path $env:LOCALAPPDATA "Packages\Microsoft.Windows.StartMenuExperienceHost_cw5n1h2txyewy\TempState"
if (Test-Path -LiteralPath $smehTemp) {
Get-ChildItem -LiteralPath $smehTemp -Filter "TileCache_*" -ErrorAction SilentlyContinue |
Remove-Item -Force -ErrorAction SilentlyContinue
Remove-Item -LiteralPath (Join-Path $smehTemp "StartUnifiedTileModelCache.dat") -Force -ErrorAction SilentlyContinue
Stop-Process -Name StartMenuExperienceHost -Force -ErrorAction SilentlyContinue
}
} catch { }
# Re-sweep: the first pass may have left unsloth.ico locked by Explorer/SMEH for
# the native shortcut; that handle is now freed. (A surviving WSL shortcut still
# keeps the icon -- see the helper.)
if ($defaultDataDir -and (Test-Path -LiteralPath $defaultDataDir)) { _RemoveDataDirKeepingWslIcon $defaultDataDir }
# ── Clean user PATH and registry backup ──
_Step "Cleaning user PATH and registry..."
try {
$regKey = [Microsoft.Win32.Registry]::CurrentUser.OpenSubKey('Environment', $true)
if ($regKey) {
try {
$rawPath = $regKey.GetValue('Path', '', [Microsoft.Win32.RegistryValueOptions]::DoNotExpandEnvironmentNames)
if ($rawPath) {
$entries = $rawPath -split ';'
$kept = New-Object System.Collections.ArrayList
$removedAny = $false
# Only remove PATH entries that live inside a Studio root we
# actually own (default or env-mode). A literal substring
# match on `unsloth_studio` would clobber unrelated user
# virtualenvs that happen to share the name.
foreach ($e in $entries) {
if ([string]::IsNullOrWhiteSpace($e)) { continue }
$expanded = [Environment]::ExpandEnvironmentVariables($e).TrimEnd('\','/')
$isStudio = $false
foreach ($r in $knownRoots) {
if (-not $r) { continue }
$rNorm = $r.TrimEnd('\','/')
if ($expanded -ieq $rNorm -or $expanded -ilike "$rNorm\*") {
$isStudio = $true; break
}
}
if ($isStudio) {
_Substep "removed PATH entry: $e" "Green"
$removedAny = $true
continue
}
[void]$kept.Add($e)
}
if ($removedAny) {
$newPath = ($kept -join ';')
$regKey.SetValue('Path', $newPath, [Microsoft.Win32.RegistryValueKind]::ExpandString)
try {
$d = "UnslothPathRefresh_" + ([guid]::NewGuid().ToString('N').Substring(0, 8))
[Environment]::SetEnvironmentVariable($d, '1', 'User')
[Environment]::SetEnvironmentVariable($d, [NullString]::Value, 'User')
} catch { }
}
}
} finally {
$regKey.Close()
}
}
} catch {
_Substep "could not update user PATH: $($_.Exception.Message)" "Yellow"
}
# Remove HKCU\Software\Unsloth (PathBackup lives here; install.ps1 owns it).
try {
Remove-Item -LiteralPath 'HKCU:\Software\Unsloth' -Recurse -Force -ErrorAction SilentlyContinue
} catch { }
Write-Host ""
Write-Host "Unsloth Studio uninstalled."
Write-Host "Note: Hugging Face model cache at %USERPROFILE%\.cache\huggingface was left in place."
Write-Host "Remove it manually with 'Remove-Item -Recurse -Force `"$env:USERPROFILE\.cache\huggingface\hub`"' if desired."
if (-not $env:UNSLOTH_STUDIO_HOME -and -not $env:STUDIO_HOME) {
Write-Host ""
Write-Host "If you installed Unsloth Studio with UNSLOTH_STUDIO_HOME or STUDIO_HOME"
Write-Host "pointing at a custom directory, re-run this script with the same variable"
Write-Host "set to also remove that install tree, e.g.:"
Write-Host " `$env:UNSLOTH_STUDIO_HOME = 'C:\your\path'; irm https://raw.githubusercontent.com/unslothai/unsloth/main/scripts/uninstall.ps1 | iex"
}
}
Uninstall-UnslothStudio @args
+427
View File
@@ -0,0 +1,427 @@
#!/usr/bin/env sh
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
#
# Unsloth Studio uninstaller (macOS / Linux / WSL).
# Stops running servers and removes install dir, launcher data,
# CLI shim, desktop shortcut, .app bundle, and Launch Services entry.
# Honors custom roots set via UNSLOTH_STUDIO_HOME / STUDIO_HOME at
# install time (read back from studio.conf).
#
# Usage: curl -fsSL https://raw.githubusercontent.com/unslothai/unsloth/main/scripts/uninstall.sh | sh
set -e
# Stop a Studio server via its PID file (written by install.sh's _spawn_terminal).
_kill_pid_file() {
_pid_file="$1"
[ -f "$_pid_file" ] || return 0
_pid=$(sed -n '1s/[^0-9].*//p' "$_pid_file" 2>/dev/null || true)
if [ -n "$_pid" ] && kill -0 "$_pid" 2>/dev/null; then
kill -TERM "$_pid" 2>/dev/null || true
# Wait up to 10s for graceful shutdown.
_i=0
while kill -0 "$_pid" 2>/dev/null && [ "$_i" -lt 20 ]; do
sleep 0.5
_i=$((_i + 1))
done
kill -0 "$_pid" 2>/dev/null && kill -KILL "$_pid" 2>/dev/null || true
fi
rm -f "$_pid_file" 2>/dev/null || true
}
# BRE-escape a path so it can be embedded in a pkill -f regex.
_pkill_escape() {
printf '%s' "$1" | sed -e 's:[][\\.^$*+?{|}()/]:\\&:g'
}
_pkill_studio() {
# Prefer PID files written by _spawn_terminal so we only touch our own installs.
for _data_dir in "$HOME/.local/share/unsloth" $(_custom_studio_data_dirs); do
[ -d "$_data_dir" ] || continue
for _pf in "$_data_dir"/studio-*.pid; do
[ -f "$_pf" ] && _kill_pid_file "$_pf"
done
done
command -v pkill >/dev/null 2>&1 || return 0
# Scope fallback patterns to the install roots we are removing so a
# different Studio install (different UNSLOTH_STUDIO_HOME) is not touched.
_kill_roots="$HOME/.unsloth/studio"
_roots_from_conf=$(_custom_studio_roots 2>/dev/null || true)
[ -n "$_roots_from_conf" ] && _kill_roots="$_kill_roots
$_roots_from_conf"
printf '%s\n' "$_kill_roots" | while IFS= read -r _root; do
[ -n "$_root" ] || continue
[ -d "$_root" ] || continue
_re=$(_pkill_escape "$_root")
# `unsloth studio` (default port) + `-p N` + `--port N` forms, all
# anchored on the install root's venv path.
for _pat in \
"${_re}/unsloth_studio/bin/[^ ]* studio( |\$|.*-p[ =][0-9])" \
"${_re}/unsloth_studio/bin/[^ ]* studio.*--port[ =][0-9]" \
"${_re}/.*studio/backend/run\.py"
do
pkill -TERM -f "$_pat" 2>/dev/null || true
done
done
sleep 0.5
printf '%s\n' "$_kill_roots" | while IFS= read -r _root; do
[ -n "$_root" ] || continue
[ -d "$_root" ] || continue
_re=$(_pkill_escape "$_root")
for _pat in \
"${_re}/unsloth_studio/bin/[^ ]* studio( |\$|.*-p[ =][0-9])" \
"${_re}/unsloth_studio/bin/[^ ]* studio.*--port[ =][0-9]" \
"${_re}/.*studio/backend/run\.py"
do
pkill -KILL -f "$_pat" 2>/dev/null || true
done
done
}
_remove_path() {
_p="$1"
if [ -e "$_p" ] || [ -L "$_p" ]; then
rm -rf "$_p" 2>/dev/null && echo " removed: $_p" || echo " could not remove: $_p" >&2
fi
}
# Accept as Studio root only if Studio sentinels exist (matches install.sh's
# env-mode ownership guard at install.sh:1358-1361). A bare unsloth_studio/
# directory is NOT enough -- require the install-time owner marker so a user
# directory that happens to contain a folder named "unsloth_studio" is safe.
_is_studio_root() {
_r="$1"
[ -n "$_r" ] || return 1
[ -f "$_r/share/studio.conf" ] && return 0
[ -f "$_r/unsloth_studio/.unsloth-studio-owned" ] && return 0
if [ -L "$_r/bin/unsloth" ]; then
_t=$(readlink "$_r/bin/unsloth" 2>/dev/null || true)
case "$_t" in *unsloth_studio/bin/unsloth) return 0 ;; esac
fi
return 1
}
# Hard deny list: never delete /, $HOME, $HOME's parent, or system paths.
_is_unsafe_root() {
_r="$1"
[ -z "$_r" ] && return 0
case "$_r" in /|""|"$HOME"|"$HOME/") return 0 ;; esac
case "$_r" in /bin|/sbin|/etc|/usr|/usr/*|/var|/var/*|/opt|/opt/*|/Library|/Library/*|/System|/System/*|/Applications|/Applications/*) return 0 ;; esac
_parent=$(dirname "$HOME" 2>/dev/null || echo "")
[ -n "$_parent" ] && [ "$_r" = "$_parent" ] && return 0
return 1
}
# Print share/ dirs of known custom roots (where PID files live).
_custom_studio_data_dirs() {
_custom_studio_roots 2>/dev/null | while IFS= read -r _r; do
[ -d "$_r/share" ] && printf '%s\n' "$_r/share"
done
}
# Resolve a custom install root from any of:
# 1. UNSLOTH_STUDIO_HOME / STUDIO_HOME env vars at uninstall time
# 2. Default-mode studio.conf at $HOME/.local/share/unsloth/studio.conf
# 3. Env-mode studio.conf at $<root>/share/studio.conf (discovered via 1)
# install.sh writes UNSLOTH_EXE='<root>/unsloth_studio/bin/unsloth', so
# the install root is three dirnames up. Prints each discovered non-default
# root on its own line; the caller iterates and de-duplicates.
_custom_studio_roots() {
_seen=""
_emit() {
_r="$1"
[ -z "$_r" ] && return 0
# Tilde expansion (env vars are not subject to it on quoted assignment),
# matches install.sh's _resolve_studio_destinations. The literal "~/"
# pattern is intentional; SC2088 is a false positive here.
# shellcheck disable=SC2088
case "$_r" in
"~") _r="$HOME" ;;
"~/"*) _r="$HOME/${_r#'~/'}" ;;
esac
# Canonicalize so syntactic variants ($HOME/../$USER, trailing slash)
# resolve to the same path and hit the _is_unsafe_root deny list.
# shellcheck disable=SC1007
_canon=$(CDPATH= cd -P -- "$_r" 2>/dev/null && pwd -P)
[ -n "$_canon" ] && _r="$_canon"
case "$_r" in "$HOME/.unsloth/studio"|/|"") return 0 ;; esac
case ":$_seen:" in *":$_r:"*) return 0 ;; esac
_seen="$_seen:$_r"
printf '%s\n' "$_r"
}
_from_conf() {
[ -f "$1" ] || return 0
# Tolerate paths containing apostrophes (install.sh emits '\'' for them).
_exe=$(sed -n "s/^UNSLOTH_EXE='\(.*\)'\$/\1/p" "$1" | head -n1)
_exe=$(printf '%s' "$_exe" | sed "s/'\\\\''/'/g")
[ -n "$_exe" ] || return 0
_emit "$(dirname "$(dirname "$(dirname "$_exe")")")"
}
# Mirror install.sh's precedence: UNSLOTH_STUDIO_HOME wins, STUDIO_HOME is
# ignored when both are set. Otherwise uninstalling install A could also
# delete install B if the user has STUDIO_HOME left over from B.
if [ -n "${UNSLOTH_STUDIO_HOME:-}" ]; then
_emit "$UNSLOTH_STUDIO_HOME"
_from_conf "$UNSLOTH_STUDIO_HOME/share/studio.conf"
elif [ -n "${STUDIO_HOME:-}" ]; then
_emit "$STUDIO_HOME"
_from_conf "$STUDIO_HOME/share/studio.conf"
fi
# Default-mode conf.
_from_conf "$HOME/.local/share/unsloth/studio.conf"
}
# Remove $HOME/.local/bin/unsloth only if it's a Studio-managed symlink.
# Studio's install.sh writes this as a symlink into the studio venv
# (install.sh: `ln -sfn "$VENV_DIR/bin/unsloth" "$_shim_path"`). A
# pip-installed `unsloth` CLI is a regular file — leave it alone to avoid
# wiping an unrelated install.
_remove_cli_shim() {
_shim="$HOME/.local/bin/unsloth"
[ -L "$_shim" ] || return 0
_target=$(readlink "$_shim" 2>/dev/null || true)
case "$_target" in
*/unsloth_studio/bin/unsloth) _remove_path "$_shim" ;;
*) ;;
esac
}
_uid=$(id -u 2>/dev/null || echo 0)
_os=$(uname 2>/dev/null || echo unknown)
_is_wsl=0
[ "$_os" = "Linux" ] && grep -qi microsoft /proc/version 2>/dev/null && _is_wsl=1
echo "Stopping any running Unsloth Studio servers..."
_pkill_studio
echo "Removing data and install directories..."
_custom_studio_roots | while IFS= read -r _custom_root; do
[ -n "$_custom_root" ] || continue
if _is_unsafe_root "$_custom_root"; then
echo " refusing to remove unsafe path: $_custom_root" >&2
continue
fi
if ! _is_studio_root "$_custom_root"; then
echo " refusing to remove non-Studio path: $_custom_root" >&2
continue
fi
_remove_path "$_custom_root"
done
_remove_path "$HOME/.unsloth/studio"
# Default-mode shared llama.cpp build + cache are siblings of studio (not removed
# by deleting it). No-op in env/custom mode (they nest under the custom root) and
# when absent. A user-set UNSLOTH_LLAMA_CPP_PATH is intentionally kept.
_remove_path "$HOME/.unsloth/llama.cpp"
_remove_path "$HOME/.unsloth/.cache"
# Isolated Node.js runtime (install_node_prebuilt.py), a sibling of studio in
# default mode. No-op in env/custom mode (nested under the custom root) and absent.
_remove_path "$HOME/.unsloth/node"
# llama.cpp atomic-install staging root (install_llama_prebuilt.py .staging).
# Normally pruned after activate, but an interrupted build can leave it behind;
# removing it lets the rmdir below succeed. No-op in env/custom mode and absent.
_remove_path "$HOME/.unsloth/.staging"
# llama.cpp install lock (serializes the shared build); a stray one keeps ~/.unsloth
# from being pruned below. No-op in env/custom mode and when absent.
_remove_path "$HOME/.unsloth/.llama.cpp.install.lock"
# ROCm-on-WSL helper artifacts (librocdxg build clone + smoke-test venv). No-op
# where they don't exist; removing them lets the rmdir below succeed.
_remove_path "$HOME/.unsloth/librocdxg"
_remove_path "$HOME/.unsloth/rocm-smoketest"
# Drop ~/.unsloth only if now empty (rmdir refuses non-empty, so user content is kept).
rmdir "$HOME/.unsloth" 2>/dev/null || true
_remove_path "$HOME/.local/share/unsloth"
# CLI shim: only the symlink Studio created, never a pip-installed file.
_remove_cli_shim
echo "Removing desktop shortcut and launcher lock..."
# install.sh creates Desktop/Unsloth Studio as a symlink. If the user has an
# unrelated regular directory by that name, leave it alone.
_desktop_link="$HOME/Desktop/Unsloth Studio"
if [ -L "$_desktop_link" ] || [ ! -e "$_desktop_link" ]; then
_remove_path "$_desktop_link"
else
echo " refusing to remove non-symlink Desktop path: $_desktop_link" >&2
fi
_remove_path "$HOME/Desktop/unsloth-studio.desktop"
# Locks are namespaced per-uid; env-mode adds an extra suffix.
_lock_glob="${XDG_RUNTIME_DIR:-/tmp}/unsloth-studio-launcher-${_uid}"
for _lock in "$_lock_glob".lock "$_lock_glob"-*.lock; do
[ -e "$_lock" ] && _remove_path "$_lock"
done
case "$_os" in
Darwin)
echo "Removing macOS .app bundle and Launch Services entry..."
_remove_path "$HOME/Applications/Unsloth Studio.app"
_lsr="/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/Support/lsregister"
if [ -x "$_lsr" ]; then
"$_lsr" -u "$HOME/Applications/Unsloth Studio.app" 2>/dev/null || true
fi
;;
Linux)
if [ "$_is_wsl" = "1" ]; then
echo "Removing WSL Windows-side shortcuts..."
# install.sh creates per-distro 'Unsloth Studio (WSL - <distro>).lnk'
# on the Windows Desktop + Start Menu via powershell.exe. Scope removal
# to THIS distro (passed as $args[0]) so a multi-distro install keeps the
# other distros' launchers; the TARGET=wsl.exe check still spares a
# native install's "Unsloth Studio.lnk". Prefer powershell.exe; test it
# can EXECUTE (`command -v` succeeds even with interop OFF -- .exe then
# fails "Exec format error", common on systemd-enabled distros).
_wsl_distro="${WSL_DISTRO_NAME:-}"
_ps_ran=0
if command -v powershell.exe >/dev/null 2>&1 && \
powershell.exe -NoProfile -Command "exit 0" >/dev/null 2>&1; then
_ps_ran=1
# Inject the distro into the command: a -Command string does not
# receive trailing tokens as $args. WSL distro names are safe to
# embed (no quotes/$/backtick).
# shellcheck disable=SC2016
powershell.exe -NoProfile -Command '$distro = "'"$_wsl_distro"'";
$dirs = @(
[Environment]::GetFolderPath("Desktop"),
(Join-Path $env:APPDATA "Microsoft\Windows\Start Menu\Programs")
);
$ws = New-Object -ComObject WScript.Shell;
foreach ($d in $dirs) {
if (-not $d -or -not (Test-Path -LiteralPath $d)) { continue }
Get-ChildItem -LiteralPath $d -Filter "Unsloth Studio*.lnk" -ErrorAction SilentlyContinue | ForEach-Object {
try {
$sc = $ws.CreateShortcut($_.FullName);
if ("$($sc.TargetPath) $($sc.Arguments)" -notmatch "wsl\.exe") { return }
# When the distro is known, require the per-distro
# name for this distro or its -d "<distro>" argument
# so launchers for other distros are not removed.
if ($distro) {
$nameMatch = ($_.Name -eq "Unsloth Studio (WSL - $distro).lnk");
$argMatch = ($sc.Arguments -match ("-d\s+`"?" + [regex]::Escape($distro) + "`"?"));
if (-not ($nameMatch -or $argMatch)) { return }
}
Remove-Item -LiteralPath $_.FullName -Force -ErrorAction SilentlyContinue
} catch { }
}
}
# Keep the shared icon while any Unsloth shortcut still uses it (native
# install or another WSL distro); drop it only with the last one.
$iconInUse = $false;
foreach ($d in $dirs) {
if (-not $d -or -not (Test-Path -LiteralPath $d)) { continue }
if (Get-ChildItem -LiteralPath $d -Filter "Unsloth Studio*.lnk" -ErrorAction SilentlyContinue) { $iconInUse = $true; break }
}
# Guard LOCALAPPDATA: empty on a service/SYSTEM account makes
# Join-Path throw, aborting the icon cleanup (mirror uninstall.ps1).
if (-not [string]::IsNullOrWhiteSpace($env:LOCALAPPDATA)) {
$iconDir = Join-Path $env:LOCALAPPDATA "Unsloth Studio";
$ico = Join-Path $iconDir "unsloth.ico";
if ((-not $iconInUse) -and (Test-Path -LiteralPath $ico)) { Remove-Item -LiteralPath $ico -Force -ErrorAction SilentlyContinue }
if ((Test-Path -LiteralPath $iconDir) -and -not (Get-ChildItem -LiteralPath $iconDir -Force -ErrorAction SilentlyContinue)) { Remove-Item -LiteralPath $iconDir -Recurse -Force -ErrorAction SilentlyContinue }
}' >/dev/null 2>&1 || true
fi
# Remove $1's shared unsloth.ico only if no Unsloth shortcut (native install
# or another WSL distro) still uses it, then drop the dir if empty. Reciprocal
# of uninstall.ps1's _RemoveDataDirKeepingWslIcon (keeps the icon for a
# surviving WSL shortcut when the native side is removed).
_drop_shared_icon_if_unused() {
_du="$1"
_icodir="$_du/AppData/Local/Unsloth Studio"
_icon_in_use=0
for _sd in \
"$_du/Desktop" \
"$_du/OneDrive/Desktop" \
"$_du"/OneDrive*/Desktop \
"$_du/AppData/Roaming/Microsoft/Windows/Start Menu/Programs"; do
[ -d "$_sd" ] || continue
for _any in "$_sd"/"Unsloth Studio"*.lnk; do
[ -e "$_any" ] && { _icon_in_use=1; break; }
done
[ "$_icon_in_use" = "1" ] && break
done
if [ "$_icon_in_use" = "0" ]; then
[ -f "$_icodir/unsloth.ico" ] && rm -f "$_icodir/unsloth.ico" 2>/dev/null || true
fi
[ -d "$_icodir" ] && rmdir "$_icodir" 2>/dev/null || true
}
# Fallback when powershell.exe can't run (interop disabled): remove WSL .lnk
# files via drvfs. The "Unsloth Studio (WSL..." name is WSL-specific, so a
# native install's "Unsloth Studio.lnk" never matches.
if [ "$_ps_ran" = "0" ]; then
for _drive in /mnt/c /mnt/d /mnt/e; do
[ -d "$_drive/Users" ] || continue
for _udir in "$_drive"/Users/*; do
[ -d "$_udir" ] || continue
for _scdir in \
"$_udir/Desktop" \
"$_udir/OneDrive/Desktop" \
"$_udir"/OneDrive*/Desktop \
"$_udir/AppData/Roaming/Microsoft/Windows/Start Menu/Programs"; do
[ -d "$_scdir" ] || continue
if [ -n "$_wsl_distro" ]; then
# Exact per-distro name (no glob) so other distros survive.
_lnk="$_scdir/Unsloth Studio (WSL - ${_wsl_distro}).lnk"
[ -e "$_lnk" ] && rm -f "$_lnk" 2>/dev/null && echo " removed: $_lnk" || true
else
# Distro unknown: fall back to the broad WSL prefix.
for _lnk in "$_scdir"/"Unsloth Studio (WSL"*.lnk; do
[ -e "$_lnk" ] && rm -f "$_lnk" 2>/dev/null && echo " removed: $_lnk" || true
done
fi
done
# Drop the shared icon only when no shortcut still needs it.
_drop_shared_icon_if_unused "$_udir"
done
done
fi
# ── ROCm-on-WSL config (install_rocm_wsl_strixhalo.sh) ──
# Remove Unsloth's own ROCDXG config (the env it persisted). The system
# ROCm userspace is a shared prereq (like CUDA) and is LEFT IN PLACE by
# default; set UNSLOTH_UNINSTALL_ROCM=1 to remove it too.
echo "Removing ROCm-on-WSL config..."
_sudo=""
if [ "$_uid" != "0" ] && command -v sudo >/dev/null 2>&1; then _sudo="sudo"; fi
$_sudo rm -f /etc/profile.d/unsloth-rocm-wsl.sh 2>/dev/null || true
if [ -f "$HOME/.bashrc" ] && grep -q "Unsloth ROCm-on-WSL" "$HOME/.bashrc" 2>/dev/null; then
_bk=$(mktemp 2>/dev/null || echo "$HOME/.bashrc.unsloth.tmp")
if sed '/# >>> Unsloth ROCm-on-WSL/,/# <<< Unsloth ROCm-on-WSL/d' "$HOME/.bashrc" > "$_bk" 2>/dev/null; then
cat "$_bk" > "$HOME/.bashrc" 2>/dev/null || true
echo " cleaned ROCm-on-WSL block from ~/.bashrc"
fi
rm -f "$_bk" 2>/dev/null || true
fi
if [ "${UNSLOTH_UNINSTALL_ROCM:-0}" = "1" ]; then
echo " removing system ROCm (UNSLOTH_UNINSTALL_ROCM=1)..."
$_sudo rm -f /etc/apt/sources.list.d/rocm.list /etc/apt/preferences.d/rocm-pin-600 \
/etc/apt/keyrings/rocm.gpg /etc/ld.so.conf.d/rocm.conf 2>/dev/null || true
$_sudo sh -c 'rm -rf /opt/rocm /opt/rocm-*' 2>/dev/null || true
if command -v ldconfig >/dev/null 2>&1; then $_sudo ldconfig 2>/dev/null || true; fi
elif [ -d /opt/rocm ]; then
echo " Note: ROCm userspace (/opt/rocm*) left in place (shared prereq)."
echo " Remove it by re-running with UNSLOTH_UNINSTALL_ROCM=1, or manually:"
echo " sudo rm -rf /opt/rocm /opt/rocm-* && sudo ldconfig"
fi
fi
echo "Removing Linux .desktop entry..."
_remove_path "$HOME/.local/share/applications/unsloth-studio.desktop"
if command -v update-desktop-database >/dev/null 2>&1; then
update-desktop-database "$HOME/.local/share/applications" 2>/dev/null || true
fi
;;
esac
echo ""
echo "Unsloth Studio uninstalled."
echo "Note: Hugging Face model cache at ~/.cache/huggingface was left in place."
echo "Remove it manually with 'rm -rf ~/.cache/huggingface/hub' if desired."
# Env-mode installs leave no breadcrumb in $HOME, so a custom root can
# only be located if the user re-exports the variable. Print a hint when
# neither var is set so the bare `curl | sh` flow doesn't silently miss.
if [ -z "${UNSLOTH_STUDIO_HOME:-}" ] && [ -z "${STUDIO_HOME:-}" ]; then
echo ""
echo "If you installed Unsloth Studio with UNSLOTH_STUDIO_HOME or STUDIO_HOME"
echo "pointing at a custom directory, re-run this script with the same variable"
echo "set to also remove that install tree, e.g.:"
echo " UNSLOTH_STUDIO_HOME=/your/path sh -c \"\$(curl -fsSL https://raw.githubusercontent.com/unslothai/unsloth/main/scripts/uninstall.sh)\""
fi
+245
View File
@@ -0,0 +1,245 @@
# Unsloth - 2x faster, 60% less VRAM LLM training and finetuning
# Copyright 2023-present Daniel Han-Chen, Michael Han-Chen & the Unsloth team. All rights reserved.
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Lesser General Public License for more details.
"""Deterministic comment / docstring-only verifier.
Compares a list of changed files between two git refs and reports whether
each diff is strictly comments / docstrings (Python) or comments
(YAML / GitHub Actions). Useful for gating a "comment trim" /
"docstring refactor" PR against accidental code drift.
Per .py file: parse both revs into AST, strip module / class / function
docstrings, then compare ast.unparse output. Pure Python comments are
discarded by the parser by construction, so any post-strip diff is real
code. Per .yml file: yaml.safe_load both sides and compare the parsed
Python object; if scalar values differ, also strip shell comments inside
``run: |`` block bodies before comparing. Exit code 0 = all OK, 1 = at
least one file has a real (non-comment) diff or an error.
Usage:
python scripts/verify_comment_only_diff.py [--base REF] [--head REF] path ...
Defaults: --base origin/main, --head HEAD. Paths are repo-relative.
Example:
git diff --name-only origin/main..HEAD \\
| xargs python scripts/verify_comment_only_diff.py --base origin/main
"""
from __future__ import annotations
import argparse
import ast
import difflib
import subprocess
import sys
from typing import Any
import yaml
def _git_show(rev: str, path: str) -> str:
return subprocess.check_output(
["git", "show", f"{rev}:{path}"],
text = True,
stderr = subprocess.DEVNULL,
)
def _strip_docstrings(tree: ast.AST) -> ast.AST:
"""Remove docstrings; empty bodies become ``pass`` so unparse stays valid."""
for node in ast.walk(tree):
if isinstance(
node,
(ast.Module, ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef),
):
body = getattr(node, "body", None)
if not body:
continue
first = body[0]
if (
isinstance(first, ast.Expr)
and isinstance(first.value, ast.Constant)
and isinstance(first.value.value, str)
):
node.body = body[1:]
if not node.body:
node.body = [ast.Pass()]
return tree
def _normalize_py(src: str) -> str:
tree = ast.parse(src)
tree = _strip_docstrings(tree)
return ast.unparse(tree)
def _strip_shell_comments(s: str) -> str:
"""Strip shell comments and collapse blank lines. Heuristic: skips lines
with an odd quote count (open string)."""
out = []
for line in s.splitlines():
stripped = line.lstrip()
if stripped.startswith("#"):
continue
has_single = line.count("'") % 2 == 0
has_double = line.count('"') % 2 == 0
if has_single and has_double:
idx = line.find(" #")
if idx >= 0:
line = line[:idx].rstrip()
out.append(line)
norm = []
prev_blank = False
for line in out:
if line.strip() == "":
if prev_blank:
continue
prev_blank = True
else:
prev_blank = False
norm.append(line)
return "\n".join(norm).strip()
def _normalize_yaml_run_strings(obj: Any) -> Any:
"""Strip shell comments from any multi-line string (``run: |`` body)."""
if isinstance(obj, dict):
return {k: _normalize_yaml_run_strings(v) for k, v in obj.items()}
if isinstance(obj, list):
return [_normalize_yaml_run_strings(x) for x in obj]
if isinstance(obj, str) and "\n" in obj:
return _strip_shell_comments(obj)
return obj
def _walk_yaml_diff(
b: Any,
a: Any,
prefix: str = "",
) -> None:
"""Print a path-keyed summary of the first structural / scalar diff."""
if type(b) is not type(a):
print(
f" type-diff at {prefix or '/'}: " f"{type(b).__name__} -> {type(a).__name__}",
)
return
if isinstance(b, dict):
keys = sorted((set(b.keys()) | set(a.keys())), key = lambda x: str(x))
for k in keys:
if k not in b:
print(f" added key {prefix}/{k}")
elif k not in a:
print(f" removed key {prefix}/{k}")
else:
_walk_yaml_diff(b[k], a[k], f"{prefix}/{k}")
elif isinstance(b, list):
if len(b) != len(a):
print(
f" list len at {prefix or '/'}: " f"{len(b)} -> {len(a)}",
)
for i, (bi, ai) in enumerate(zip(b, a)):
_walk_yaml_diff(bi, ai, f"{prefix}[{i}]")
elif b != a:
bs = repr(b)[:300]
as_ = repr(a)[:300]
print(f" scalar at {prefix or '/'}:")
print(f" before: {bs}")
print(f" after: {as_}")
def _verify_python(path: str, before: str, after: str) -> bool:
try:
norm_before = _normalize_py(before)
norm_after = _normalize_py(after)
except SyntaxError as exc:
print(f"FAIL {path}: SyntaxError parsing -- {exc}")
return False
if norm_before == norm_after:
print(f"OK {path} (AST identical after docstring strip)")
return True
diff = list(
difflib.unified_diff(
norm_before.splitlines(),
norm_after.splitlines(),
fromfile = f"{path}@before",
tofile = f"{path}@after",
n = 2,
)
)
print(f"FAIL {path}: AST differs after docstring strip:")
for line in diff[:40]:
print(f" {line}")
return False
def _verify_yaml(path: str, before: str, after: str) -> bool:
try:
raw_before = yaml.safe_load(before)
raw_after = yaml.safe_load(after)
except yaml.YAMLError as exc:
print(f"FAIL {path}: YAML parse error -- {exc}")
return False
if raw_before == raw_after:
print(f"OK {path} (YAML parsed object identical)")
return True
norm_before = _normalize_yaml_run_strings(raw_before)
norm_after = _normalize_yaml_run_strings(raw_after)
if norm_before == norm_after:
print(
f"OK {path} (YAML parsed object identical after "
f"stripping shell comments from run: bodies)",
)
return True
print(
f"FAIL {path}: YAML parsed objects still differ after stripping "
f"shell comments from `run:` bodies.",
)
_walk_yaml_diff(norm_before, norm_after)
return False
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
description = "Verify each path's diff between BASE and HEAD is "
"strictly comments / docstrings.",
)
parser.add_argument("--base", default = "origin/main", help = "base git ref")
parser.add_argument("--head", default = "HEAD", help = "head git ref")
parser.add_argument("paths", nargs = "+", help = "repo-relative paths")
args = parser.parse_args(argv)
rc = 0
print(f"Comparing {len(args.paths)} files: {args.base} vs {args.head}\n")
for path in args.paths:
try:
before = _git_show(args.base, path)
after = _git_show(args.head, path)
except subprocess.CalledProcessError as exc:
print(f"SKIP {path}: {exc}")
continue
if path.endswith(".py"):
if not _verify_python(path, before, after):
rc = 1
elif path.endswith((".yml", ".yaml")):
if not _verify_yaml(path, before, after):
rc = 1
else:
print(f"NOTE {path}: not .py or .yaml -- skipped automated check.")
return rc
if __name__ == "__main__":
sys.exit(main())
+839
View File
@@ -0,0 +1,839 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
"""Deterministic, scope-aware verifier for import-hoisting / alias-rename refactors.
The risk when moving `from a import b as _b` (or `import b as _b`) to module top
and normalizing `_b` -> `b` is twofold:
1. DANGLING ALIAS - a `_b` reference is left un-normalized; it now resolves to
nothing (NameError) or, worse, to some *other* module-level `_b`.
2. RENAME CLASH - `_b` was an alias on purpose because `b` already meant
something else in that scope; normalizing `_b` -> `b` silently re-points the
reference at the wrong object (no NameError, no pyflakes warning).
This tool parses BEFORE (a git ref, default origin/main) and AFTER (default HEAD)
for each file, builds a real LEGB scope model (functions, classes, lambdas,
comprehensions, global/nonlocal, args, walrus, star-imports), and resolves every
Name load to its binding. It then compares, PER SCOPE:
* UNRESOLVED-NEW : loads that resolve to nothing in AFTER but did in BEFORE
(or are newly present) -> catches dangling aliases.
* TARGET-MISSING : an import *target* (e.g. module `glob`, or
`importlib.metadata.version`) that a function resolved to
in BEFORE but no longer resolves to in AFTER -> catches a
function that lost access to a module it still uses.
Robust to alias renames because it compares the *target*,
not the local name.
* TARGET-CHANGED : a load whose resolved import target differs BEFORE vs
AFTER -> catches a rename that re-points to a different
module (the clash case).
* AMBIGUOUS-BIND : a name bound by BOTH an import and a non-import in the same
scope in AFTER (and not in BEFORE) -> the "alias was on
purpose / now collides" smell.
* MODULE-DUP-IMPORT: a module-level name imported and also defined/assigned at
module level (introduced by the change).
* NEW-UNUSED-IMPORT: a module-level import added in AFTER that nothing resolves
to (informational; re-exports are a known false positive).
Usage:
verify_import_hoist.py [--before REF] [--after REF] <file>... # compare
verify_import_hoist.py --self-test # prove it catches bugs
Exit code 1 if any non-informational finding.
"""
from __future__ import annotations
import argparse
import ast
import builtins
import re as _re_mod
import subprocess
import sys
from dataclasses import dataclass, field
_BUILTINS = set(dir(builtins)) | {
"__file__",
"__name__",
"__doc__",
"__package__",
"__spec__",
"__loader__",
"__builtins__",
"__class__",
"__annotations__",
"__dict__",
"__qualname__",
"__module__",
"__path__",
"__debug__",
"__import__",
"NotImplemented",
"Ellipsis",
"copyright",
"credits",
"license",
"help",
"exit",
"quit",
"__build_class__",
"__cached__",
"reveal_type",
"reveal_locals",
}
# ---------------------------------------------------------------- scope model
@dataclass
class Binding:
kind: str # 'import' | 'importfrom' | 'def' | 'class' | 'other'
target: str | None = None # canonical import target id, else None
@dataclass
class Scope:
kind: str # 'module' | 'function' | 'class' | 'lambda' | 'comp'
qualname: str
parent: "Scope | None"
bindings: dict[str, list[Binding]] = field(default_factory = dict)
globals: set[str] = field(default_factory = set)
nonlocals: set[str] = field(default_factory = set)
star_import: bool = False
def add(self, name: str, b: Binding) -> None:
self.bindings.setdefault(name, []).append(b)
def _import_target(node: ast.AST, alias: ast.alias) -> tuple[str, str]:
"""Return (bound_name, canonical_target_id) for one import alias."""
if isinstance(node, ast.Import):
bound = alias.asname or alias.name.split(".")[0]
return bound, f"import:{alias.name}"
# ImportFrom
bound = alias.asname or alias.name
mod = ("." * (node.level or 0)) + (node.module or "")
return bound, f"from:{mod}:{alias.name}"
class _Builder(ast.NodeVisitor):
"""Builds the scope tree + bindings, and records every (scope, Name-load)."""
def __init__(self):
self.module = Scope("module", "<module>", None)
self.uses: list[tuple[Scope, str, int]] = [] # hard loads
# annotations: count as "used" but never as "unresolved" (forward refs)
self.soft_uses: list[tuple[Scope, str, int]] = []
def _visit_annotation(self, node, scope: Scope) -> None:
"""Record annotation names as SOFT uses: an import used only in an annotation
counts as used, but a forward-ref name is never 'unresolved'."""
if node is None:
return
for n in ast.walk(node):
if isinstance(n, ast.Name) and isinstance(n.ctx, ast.Load):
self.soft_uses.append((scope, n.id, n.lineno))
# -- binding helpers --
def _bind_targets(self, scope: Scope, target: ast.AST) -> None:
for n in ast.walk(target):
if isinstance(n, ast.Name) and isinstance(n.ctx, (ast.Store, ast.Del)):
self._bind_name(scope, n.id, Binding("other"))
elif isinstance(n, ast.Starred):
pass
def _bind_name(self, scope: Scope, name: str, b: Binding) -> None:
if name in scope.globals:
self.module.add(name, b)
elif name in scope.nonlocals:
p = scope.parent
while p is not None and p.kind not in ("function", "lambda"):
p = p.parent
(p or self.module).add(name, b)
else:
scope.add(name, b)
# -- generic dispatch within a scope --
def _visit_body(self, stmts, scope: Scope) -> None:
for s in stmts:
self._visit_stmt(s, scope)
def _visit_stmt(self, node: ast.AST, scope: Scope) -> None:
if isinstance(node, (ast.Import, ast.ImportFrom)):
star = isinstance(node, ast.ImportFrom) and any(a.name == "*" for a in node.names)
if star:
scope.star_import = True
for alias in node.names:
if alias.name == "*":
continue
bound, target = _import_target(node, alias)
kind = "import" if isinstance(node, ast.Import) else "importfrom"
self._bind_name(scope, bound, Binding(kind, target))
return
if isinstance(node, ast.Global):
scope.globals.update(node.names)
return
if isinstance(node, ast.Nonlocal):
scope.nonlocals.update(node.names)
return
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
self._bind_name(scope, node.name, Binding("def"))
# decorators / defaults evaluate in the ENCLOSING scope
for d in node.decorator_list:
self._visit_expr(d, scope)
self._visit_arg_defaults(node.args, scope)
child = Scope("function", f"{scope.qualname}.{node.name}", scope)
self._bind_type_params(node, child)
self._bind_args(node.args, child)
# arg + return annotations: soft uses
for a in self._all_args(node.args):
self._visit_annotation(a.annotation, child)
self._visit_annotation(getattr(node, "returns", None), child)
self._visit_body(node.body, child)
return
if isinstance(node, ast.ClassDef):
self._bind_name(scope, node.name, Binding("class"))
for d in node.decorator_list:
self._visit_expr(d, scope)
for b in node.bases:
self._visit_expr(b, scope)
for kw in node.keywords:
self._visit_expr(kw.value, scope)
child = Scope("class", f"{scope.qualname}.{node.name}", scope)
self._bind_type_params(node, child)
self._visit_body(node.body, child)
return
if isinstance(node, ast.Match):
self._visit_expr(node.subject, scope)
for case in node.cases:
self._bind_pattern(case.pattern, scope)
if case.guard is not None:
self._visit_expr(case.guard, scope)
self._visit_body(case.body, scope)
return
if isinstance(node, getattr(ast, "TryStar", ())): # py3.11 except*
self._visit_body(node.body, scope)
for h in node.handlers:
if h.type is not None:
self._visit_expr(h.type, scope)
if h.name:
self._bind_name(scope, h.name, Binding("other"))
self._visit_body(h.body, scope)
self._visit_body(node.orelse, scope)
self._visit_body(node.finalbody, scope)
return
if isinstance(node, getattr(ast, "TypeAlias", ())): # py3.12 `type X = ...`
if isinstance(node.name, ast.Name):
self._bind_name(scope, node.name.id, Binding("other"))
self._visit_annotation(node.value, scope)
return
if isinstance(node, (ast.Assign, ast.AnnAssign, ast.AugAssign)):
targets = node.targets if isinstance(node, ast.Assign) else [node.target]
val = node.value
if val is not None:
self._visit_expr(val, scope)
if isinstance(node, ast.AnnAssign) and node.annotation is not None:
self._visit_annotation(node.annotation, scope)
for t in targets:
self._bind_targets(scope, t)
# AugAssign target is also a load
if isinstance(node, ast.AugAssign):
self._record_loads(t, scope)
return
if isinstance(node, (ast.For, ast.AsyncFor)):
self._visit_expr(node.iter, scope)
self._bind_targets(scope, node.target)
self._visit_body(node.body, scope)
self._visit_body(node.orelse, scope)
return
if isinstance(node, (ast.With, ast.AsyncWith)):
for item in node.items:
self._visit_expr(item.context_expr, scope)
if item.optional_vars is not None:
self._bind_targets(scope, item.optional_vars)
self._visit_body(node.body, scope)
return
if isinstance(node, ast.Try):
self._visit_body(node.body, scope)
for h in node.handlers:
if h.type is not None:
self._visit_expr(h.type, scope)
if h.name:
self._bind_name(scope, h.name, Binding("other"))
self._visit_body(h.body, scope)
self._visit_body(node.orelse, scope)
self._visit_body(node.finalbody, scope)
return
# generic statement: visit all child expressions/stmts in same scope
for child in ast.iter_child_nodes(node):
if isinstance(child, ast.stmt):
self._visit_stmt(child, scope)
else:
self._visit_expr(child, scope)
# -- expressions --
def _visit_arg_defaults(self, args: ast.arguments, scope: Scope) -> None:
for d in list(args.defaults) + [d for d in args.kw_defaults if d is not None]:
self._visit_expr(d, scope)
def _all_args(self, args: ast.arguments) -> list[ast.arg]:
out = list(args.posonlyargs) + list(args.args) + list(args.kwonlyargs)
if args.vararg:
out.append(args.vararg)
if args.kwarg:
out.append(args.kwarg)
return out
def _bind_args(self, args: ast.arguments, scope: Scope) -> None:
for a in self._all_args(args):
scope.add(a.arg, Binding("other"))
def _bind_type_params(self, node, scope: Scope) -> None:
for tp in getattr(node, "type_params", []) or []:
name = getattr(tp, "name", None)
if isinstance(name, str):
scope.add(name, Binding("other"))
self._visit_annotation(getattr(tp, "bound", None), scope)
self._visit_annotation(getattr(tp, "default_value", None), scope)
def _bind_pattern(self, pat, scope: Scope) -> None:
if pat is None:
return
if isinstance(pat, ast.MatchValue):
self._visit_expr(pat.value, scope)
elif isinstance(pat, ast.MatchSingleton):
pass
elif isinstance(pat, ast.MatchSequence):
for p in pat.patterns:
self._bind_pattern(p, scope)
elif isinstance(pat, ast.MatchStar):
if pat.name:
self._bind_name(scope, pat.name, Binding("other"))
elif isinstance(pat, ast.MatchMapping):
for k in pat.keys:
self._visit_expr(k, scope)
for p in pat.patterns:
self._bind_pattern(p, scope)
if pat.rest:
self._bind_name(scope, pat.rest, Binding("other"))
elif isinstance(pat, ast.MatchClass):
self._visit_expr(pat.cls, scope)
for p in pat.patterns:
self._bind_pattern(p, scope)
for p in pat.kwd_patterns:
self._bind_pattern(p, scope)
elif isinstance(pat, ast.MatchAs):
self._bind_pattern(pat.pattern, scope)
if pat.name:
self._bind_name(scope, pat.name, Binding("other"))
elif isinstance(pat, ast.MatchOr):
for p in pat.patterns:
self._bind_pattern(p, scope)
def _record_loads(self, node: ast.AST, scope: Scope) -> None:
for n in ast.walk(node):
if isinstance(n, ast.Name) and isinstance(n.ctx, ast.Load):
self.uses.append((scope, n.id, n.lineno))
def _visit_expr(self, node: ast.AST, scope: Scope) -> None:
if isinstance(node, ast.Name):
if isinstance(node.ctx, ast.Load):
self.uses.append((scope, node.id, node.lineno))
elif isinstance(node.ctx, (ast.Store, ast.Del)):
self._bind_name(scope, node.id, Binding("other"))
return
if isinstance(node, ast.Lambda):
self._visit_arg_defaults(node.args, scope)
child = Scope("lambda", f"{scope.qualname}.<lambda>", scope)
self._bind_args(node.args, child)
self._visit_expr(node.body, child)
return
if isinstance(node, (ast.ListComp, ast.SetComp, ast.GeneratorExp, ast.DictComp)):
child = Scope("comp", f"{scope.qualname}.<comp>", scope)
for i, gen in enumerate(node.generators):
# first iterable evaluates in the enclosing scope
self._visit_expr(gen.iter, scope if i == 0 else child)
self._bind_targets(child, gen.target)
for cond in gen.ifs:
self._visit_expr(cond, child)
if isinstance(node, ast.DictComp):
self._visit_expr(node.key, child)
self._visit_expr(node.value, child)
else:
self._visit_expr(node.elt, child)
return
if isinstance(node, ast.NamedExpr): # walrus binds in enclosing scope
self._visit_expr(node.value, scope)
if isinstance(node.target, ast.Name):
self._bind_name(scope, node.target.id, Binding("other"))
return
for child in ast.iter_child_nodes(node):
if isinstance(child, ast.stmt):
self._visit_stmt(child, scope)
else:
self._visit_expr(child, scope)
def run(self, tree: ast.Module) -> None:
self._visit_body(tree.body, self.module)
# ---------------------------------------------------------------- resolution
def _any_star(scope: Scope) -> bool:
c = scope
while c is not None:
if c.star_import:
return True
c = c.parent
return False
def _resolve(scope: Scope, name: str):
"""LEGB resolution. Returns (status, bindings); status in
{'local','import','other','builtin','star','unresolved'}."""
start = scope
if name in scope.globals:
chain = [_module_of(scope)]
elif name in scope.nonlocals:
chain = _enclosing_functions(scope)
else:
chain = _legb_chain(scope)
for i, sc in enumerate(chain):
if sc is None:
continue
if name in sc.bindings:
binds = sc.bindings[name]
if any(b.kind in ("import", "importfrom") for b in binds):
return "import", binds
return "other", binds
if name in _BUILTINS:
return "builtin", []
if _any_star(start):
return "star", []
return "unresolved", []
def _module_of(scope: Scope) -> Scope:
while scope.parent is not None:
scope = scope.parent
return scope
def _enclosing_functions(scope: Scope) -> list[Scope]:
out = []
p = scope.parent
while p is not None:
if p.kind in ("function", "lambda"):
out.append(p)
p = p.parent
out.append(_module_of(scope))
return out
def _legb_chain(scope: Scope) -> list[Scope]:
"""Immediate scope, then enclosing scopes skipping class scopes, then module."""
chain = [scope]
p = scope.parent
while p is not None:
if p.kind != "class" or p.parent is None: # skip class scopes, keep module
if p.kind != "class":
chain.append(p)
p = p.parent
return chain
# ---------------------------------------------------------------- analysis
def _analyze(src: str):
tree = ast.parse(src)
b = _Builder()
b.run(tree)
# Per-scope: unresolved load names + import targets it resolves to.
unresolved: dict[str, set[str]] = {}
targets_by_scope: dict[str, set[str]] = {}
target_by_use: dict[tuple[str, str], set[str]] = {}
for scope, name, _ln in b.uses:
status, binds = _resolve(scope, name)
if status == "unresolved":
unresolved.setdefault(scope.qualname, set()).add(name)
elif status == "import":
tids = {bd.target for bd in binds if bd.target}
targets_by_scope.setdefault(scope.qualname, set()).update(tids)
target_by_use.setdefault((scope.qualname, name), set()).update(tids)
# soft uses (annotations): contribute to "used" only, never "unresolved"
for scope, name, _ln in b.soft_uses:
status, binds = _resolve(scope, name)
if status == "import":
tids = {bd.target for bd in binds if bd.target}
targets_by_scope.setdefault(scope.qualname, set()).update(tids)
# module-level binding info for clash checks
module = b.module
module_imports = {
n: bs
for n, bs in module.bindings.items()
if any(x.kind in ("import", "importfrom") for x in bs)
}
module_dup = {
n
for n, bs in module.bindings.items()
if any(x.kind in ("import", "importfrom") for x in bs)
and any(x.kind not in ("import", "importfrom") for x in bs)
}
# ambiguous: any scope where a name is bound by import AND non-import
ambiguous: dict[str, set[str]] = {}
def walk_scopes(scope: Scope):
for n, bs in scope.bindings.items():
if any(x.kind in ("import", "importfrom") for x in bs) and any(
x.kind not in ("import", "importfrom") for x in bs
):
ambiguous.setdefault(scope.qualname, set()).add(n)
# scope tree isn't stored; approximate with module only.
walk_scopes(module)
return {
"unresolved": unresolved,
"targets_by_scope": targets_by_scope,
"target_by_use": target_by_use,
"module_import_targets": {
n: {x.target for x in bs if x.target} for n, bs in module_imports.items()
},
"module_dup": module_dup,
"ambiguous": ambiguous,
}
def _git_show(ref: str, path: str) -> str | None:
try:
return subprocess.run(
["git", "show", f"{ref}:{path}"], capture_output = True, text = True, check = True
).stdout
except subprocess.CalledProcessError:
return None
def compare(before_src: str, after_src: str, path: str) -> list[tuple[str, str]]:
"""Return list of (severity, message). severity in BLOCKER/WARN/INFO.
Blocker signals (precise, no relocation false-positives):
UNRESOLVED-NEW - a load became undefined (dangling alias / removed import).
NEW-UNUSED-HOIST - a module-level import added by this change is resolved by
NO load (un-normalized alias or wrong rename target).
TARGET-CHANGED - same (scope, name) load resolves to a different import
target before vs after (a same-name re-point).
"""
a = _analyze(before_src)
b = _analyze(after_src)
findings: list[tuple[str, str]] = []
def used_targets(analysis) -> set[str]:
out: set[str] = set()
for tids in analysis["targets_by_scope"].values():
out |= tids
return out
before_used = used_targets(a)
after_used = used_targets(b)
before_module_targets: set[str] = set()
for tids in a["module_import_targets"].values():
before_module_targets |= tids
after_module_targets: set[str] = set()
for tids in b["module_import_targets"].values():
after_module_targets |= tids
added_module_targets = after_module_targets - before_module_targets
# 1. UNRESOLVED-NEW
for scope, names in b["unresolved"].items():
new = names - a["unresolved"].get(scope, set())
for n in sorted(new):
findings.append(
(
"BLOCKER",
f"{path}: UNRESOLVED-NEW '{n}' in scope {scope} "
f"(undefined after change -> dangling alias / removed import)",
)
)
# 2. HOISTED-IMPORT-UNUSED (core botched-hoist / wrong-rename signal)
# A module-level import in AFTER that NO load resolves to, that was either
# newly added by this change OR actually used before. Excludes relocation
# (import removed) and stable pre-existing re-exports.
for n, tids in b["module_import_targets"].items():
if tids & after_used:
continue # resolved -> fine
# `from __future__ import ...` is a compiler directive, not a runtime
# binding: the name (`annotations`, ...) is never loaded, so it can never
# "resolve" to a use. Skip it so a legitimately-added future import
# (e.g. `annotations` for lazy PEP 604 `X | None` on py3.9) is not flagged.
if all(t.startswith("from:__future__:") for t in tids):
continue
newly_added = bool(tids - before_module_targets)
was_used_before = bool(tids & before_used)
if newly_added or was_used_before:
why = (
"added but unused"
if newly_added
else "was used before, now unused (references re-pointed)"
)
findings.append(
(
"BLOCKER",
f"{path}: HOISTED-IMPORT-UNUSED '{n}' ({sorted(tids)}) "
f"{why} -> un-normalized alias or wrong rename target?",
)
)
# 3. TARGET-CHANGED (same scope+name resolves to a different import target)
# Only a *swap* is dangerous: a BEFORE target that is no longer reachable in
# AFTER means a reference was silently re-pointed. A pure superset growth
# (tbefore <= tafter) is the benign `import pkg.subA` + `import pkg.subB`
# case: both statements bind the same top-level name `pkg` to the same
# package object and only *add* submodule attributes (e.g. adding
# `import urllib.error` next to `import urllib.request`). Nothing the name
# resolved to before is lost, so no reference is re-pointed -- skip it.
#
# A deliberate *relocation* is also benign and must not block: when a name
# keeps its spelling but its import source is moved A -> B in THIS diff (the
# old `from A import x` is removed at module level and a new `from B import x`
# is added), the swap is intentional, not a silent re-point to a pre-existing
# different object. This mirrors the relocation tolerance already applied to
# TARGET-MISSING. The dangerous case -- the name now resolving to a target
# that already existed before (shadow/clash) -- is NOT exempted.
removed_module_targets = before_module_targets - after_module_targets
for key, tafter in b["target_by_use"].items():
tbefore = a["target_by_use"].get(key)
if tbefore and tbefore != tafter and (tbefore - tafter):
lost = tbefore - tafter
gained = tafter - tbefore
relocated = lost <= removed_module_targets and gained <= added_module_targets
if relocated:
continue
findings.append(
(
"BLOCKER",
f"{path}: TARGET-CHANGED name '{key[1]}' in {key[0]} "
f"{sorted(tbefore)} -> {sorted(tafter)} (rename re-points module)",
)
)
# 4. MODULE-DUP-IMPORT introduced
for n in sorted(b["module_dup"] - a["module_dup"]):
findings.append(
(
"WARN",
f"{path}: MODULE-DUP-IMPORT '{n}' bound by import AND non-import "
f"at module level (possible clash)",
)
)
# 5. AMBIGUOUS-BIND introduced (module scope)
for scope, names in b["ambiguous"].items():
new = names - a["ambiguous"].get(scope, set())
for n in sorted(new):
findings.append(("WARN", f"{path}: AMBIGUOUS-BIND '{n}' import+non-import in {scope}"))
# 6. TARGET-MISSING (informational): a scope stopped resolving to an import
# target. Real bugs are covered above; remaining cases are relocated code.
for scope, tbefore in a["targets_by_scope"].items():
tafter = b["targets_by_scope"].get(scope, set())
for t in sorted(tbefore - tafter):
relocated = (
""
if t in added_module_targets
else " [target not re-added here -> likely relocated/deleted]"
)
findings.append(("INFO", f"{path}: TARGET-MISSING {t} in scope {scope}{relocated}"))
return findings
# ---------------------------------------------------------------- self-test
_SELF_TESTS = {
"dangling_alias": (
# before: inline aliased import, used as _b
"import os\ndef f():\n import glob as _b\n return _b.glob('*')\n",
# after: hoisted to canonical, but reference NOT normalized -> _b dangles
"import os\nimport glob\ndef f():\n return _b.glob('*')\n",
"BLOCKER",
),
"rename_clash": (
# before: _b is a deliberate alias; `b` already means something else
"import re as _b\nb = 123\ndef f():\n return _b.compile('x'), b\n",
# after: someone normalized _b -> b ; now f().b is the int, re is lost
"import re\nb = 123\ndef f():\n return b.compile('x'), b\n",
"BLOCKER", # TARGET-MISSING from:.. or import:re in f
),
"clean_rename": (
"def f():\n import glob as _g\n return _g.glob('*')\n",
"import glob\ndef f():\n return glob.glob('*')\n",
None, # expect NO blocker
),
"clean_dedup_redundant": (
"import sys\ndef f():\n import sys\n return sys.argv\n",
"import sys\ndef f():\n return sys.argv\n",
None,
),
"from_import_dangling": (
# from-import alias left un-normalized
"def f():\n from importlib.metadata import version as _v\n return _v('x')\n",
"from importlib.metadata import version\ndef f():\n return _v('x')\n",
"BLOCKER",
),
"local_var_clash": (
# _b renamed to b, but b is a LOCAL var in f -> import silently unused
"def f(b):\n import re as _b\n return _b.compile(b)\n",
"import re\ndef f(b):\n return b.compile(b)\n", # 'b' is the param, not the module
"BLOCKER",
),
"substring_safe": (
# correct _copy->copy rename while config_copy var exists: NO false positive
"def f(config):\n"
" import copy as _copy\n"
" config_copy = _copy.deepcopy(config)\n"
" return config_copy\n",
"import copy\n"
"def f(config):\n"
" config_copy = copy.deepcopy(config)\n"
" return config_copy\n",
None,
),
"attr_access_not_a_use": (
# x._b is attribute access, not a use of name _b; removing import _b is fine
"import os\ndef f(x):\n import sys as _b\n return x._b + _b.argv[0]\n",
"import os\nimport sys\ndef f(x):\n return x._b + sys.argv[0]\n",
None,
),
}
def _self_test() -> int:
ok = True
for name, (before, after, expect) in _SELF_TESTS.items():
findings = compare(before, after, f"<{name}>")
blockers = [m for sev, m in findings if sev == "BLOCKER"]
got = "BLOCKER" if blockers else None
passed = got == expect
ok = ok and passed
print(f"[{'PASS' if passed else 'FAIL'}] {name}: expect={expect} got={got}")
for sev, m in findings:
print(f" ({sev}) {m}")
print("\nSELF-TEST:", "ALL PASS" if ok else "FAILURES")
return 0 if ok else 1
def _pyflakes_undefined(path: str) -> set[str] | None:
"""Return the set of names pyflakes reports as 'undefined name' for `path`,
or None if pyflakes failed to run/parse the file."""
try:
proc = subprocess.run(
[sys.executable, "-m", "pyflakes", path], capture_output = True, text = True
)
except Exception:
return None
if "syntax error" in (proc.stdout + proc.stderr).lower():
return None
names = set()
for line in proc.stdout.splitlines():
m = _re_mod.search(r"undefined name '([^']+)'", line)
if m:
names.add(m.group(1))
return names
def audit_files(paths: list[str]) -> int:
"""Single-version robustness audit: confirm the analyzer doesn't crash, then
cross-check its 'unresolved' names against pyflakes. A name the resolver flags
that pyflakes accepts is a tool false positive."""
n_files = n_err = n_fp = n_syntax = 0
fp_detail: dict[str, set[str]] = {}
err_detail: dict[str, str] = {}
for path in paths:
n_files += 1
try:
src = open(path, encoding = "utf-8").read()
except Exception as e: # unreadable
n_err += 1
err_detail[path] = f"read: {e}"
continue
try:
res = _analyze(src)
except SyntaxError:
n_syntax += 1
continue
except Exception as e: # analyzer crash -> robustness bug
n_err += 1
err_detail[path] = f"{type(e).__name__}: {e}"
continue
tool_unresolved = set()
for names in res["unresolved"].values():
tool_unresolved |= names
if not tool_unresolved:
continue
pf = _pyflakes_undefined(path)
if pf is None:
continue # pyflakes couldn't adjudicate; skip cross-check
false_pos = tool_unresolved - pf
if false_pos:
n_fp += 1
fp_detail[path] = false_pos
print(f"audited files : {n_files}")
print(f"syntax-skipped : {n_syntax}")
print(f"analyzer errors : {n_err}")
for p, e in sorted(err_detail.items()):
print(f" ERROR {p}: {e}")
print(f"false-positive files: {n_fp} (resolver flagged a name pyflakes accepts)")
for p, names in sorted(fp_detail.items()):
print(f" FP {p}: {sorted(names)}")
ok = n_err == 0 and n_fp == 0
print(
"\nAUDIT:",
"ROBUST (no crashes, no false positives vs pyflakes)" if ok else "NEEDS WORK (see above)",
)
return 0 if ok else 1
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--before", default = "origin/main")
ap.add_argument("--after", default = "HEAD")
ap.add_argument("--self-test", action = "store_true")
ap.add_argument(
"--audit",
action = "store_true",
help = "single-version robustness audit on filesystem paths",
)
ap.add_argument("files", nargs = "*")
args = ap.parse_args()
if args.self_test:
return _self_test()
if args.audit:
return audit_files(args.files)
any_blocker = False
for path in args.files:
before = _git_show(args.before, path)
after = _git_show(args.after, path)
if after is None:
print(f"SKIP {path}: not found at {args.after}")
continue
if before is None:
before = "" # new file
findings = compare(before, after, path)
blockers = [f for f in findings if f[0] == "BLOCKER"]
warns = [f for f in findings if f[0] == "WARN"]
infos = [f for f in findings if f[0] == "INFO"]
status = "CLEAN" if not blockers and not warns else ("BLOCKERS" if blockers else "WARNINGS")
print(f"\n=== {path}: {status} ===")
for sev, m in blockers + warns + infos:
print(f" [{sev}] {m}")
any_blocker = any_blocker or bool(blockers)
print("\nOVERALL:", "FAIL (blockers found)" if any_blocker else "PASS (no blockers)")
return 1 if any_blocker else 0
if __name__ == "__main__":
sys.exit(main())