chore: import upstream snapshot with attribution
Lint / lint (push) Has been cancelled
CI / MacOS (push) Has been cancelled
CI / Windows (push) Has been cancelled

This commit is contained in:
wehub-resource-sync
2026-07-13 13:36:25 +08:00
commit 26446540fa
3151 changed files with 974126 additions and 0 deletions
+4
View File
@@ -0,0 +1,4 @@
*.md
!README.md
*.csv
*.pkl
+64
View File
@@ -0,0 +1,64 @@
<!--- Licensed to the Apache Software Foundation (ASF) under one -->
<!--- or more contributor license agreements. See the NOTICE file -->
<!--- distributed with this work for additional information -->
<!--- regarding copyright ownership. The ASF licenses this file -->
<!--- to you under the Apache License, Version 2.0 (the -->
<!--- "License"); you may not use this file except in compliance -->
<!--- with the License. You may obtain a copy of the License at -->
<!--- http://www.apache.org/licenses/LICENSE-2.0 -->
<!--- Unless required by applicable law or agreed to in writing, -->
<!--- software distributed under the License is distributed on an -->
<!--- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -->
<!--- KIND, either express or implied. See the License for the -->
<!--- specific language governing permissions and limitations -->
<!--- under the License. -->
These scripts can be helpful when creating release notes and testing release packages.
# Create release notes
```bash
# example: create a csv file of all PRs since the v0.8 and v0.9.0 releases
# the result will be in 2 CSV files based on the --threshold arg (small PRs vs large PRs)
export GITHUB_TOKEN=<github oauth token>
python release/gather_prs.py --from-commit $(git rev-parse v0.9.0) --to-commit $(git merge-base origin/main v0.8.0)
```
After running the commands above, you will get a csv file named `out_pr_gathered.csv`. You can then import this CSV into a collaborative spreadsheet editor to distribute the work of categorizing PRs for the notes, **especially check the `pr_title_tags` column for each row and correct it if it's wrong**.
Once done, you can download the csv file assuming with name `out_pr_gathered_corrected.csv` and convert it to readable release notes using commands below:
```bash
# example: use a csv of tags-corrected PRs to create a markdown file
# Export monthly report on forum:
python make_notes.py --notes out_pr_gathered_corrected.csv --is-pr-with-link true > monthly_report.md
# Export release report on GitHub:
python make_notes.py --notes out_pr_gathered_corrected.csv --is-pr-with-link true > release_report.md
# If release report exported but forget set `--is-pr-with-link true`,
# you can append arg `--convert-with-link true`.
python3 make_notes.py --notes ./release_report.md --convert-with-link true
```
You can also create a list of RFCs
```bash
git clone https://github.com/apache/tvm-rfcs.git
# example: list RFCs since a specific commit in the tvm-rfcs repo
python list_rfcs.py --since-commit <hash> --rfcs-repo ./tvm-rfcs > rfc.md
```
Finally, combine `rfc.md` and `out.md` along with some prose to create the final release notes.
# Test release packages
After uploading release (candidate) packages to apache.org or GitHub release page, you can validate packages step-by-step from downloading, verification and compiling use script below, but don't forget edit the `version` and `rc` number in script.
```bash
test_release_package.sh
```
+221
View File
@@ -0,0 +1,221 @@
#!/usr/bin/env python3
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
# ruff: noqa: E402, F401
import argparse
import csv
import os
import pickle
import sys
from collections.abc import Callable
from pathlib import Path
from typing import Any
REPO_ROOT = Path(__file__).resolve().parent.parent.parent.parent
sys.path.append(str(REPO_ROOT / "ci" / "scripts" / "jenkins"))
from cmd_utils import tags_from_title
from git_utils import GitHubRepo, git
GITHUB_TOKEN = os.environ["GITHUB_TOKEN"]
PRS_QUERY = """
query ($owner: String!, $name: String!, $after: String, $pageSize: Int!) {
repository(owner: $owner, name: $name) {
defaultBranchRef {
name
target {
... on Commit {
oid
history(after: $after, first: $pageSize) {
pageInfo {
hasNextPage
endCursor
}
nodes {
oid
committedDate
associatedPullRequests(first: 1) {
nodes {
number
additions
changedFiles
deletions
author {
login
}
title
body
}
}
}
}
}
}
}
}
}
"""
def append_and_save(items, file):
if not file.exists():
data = []
else:
with open(file, "rb") as f:
data = pickle.load(f)
data += items
with open(file, "wb") as f:
pickle.dump(data, f)
def fetch_pr_data(args, cache):
github = GitHubRepo(user=user, repo=repo, token=GITHUB_TOKEN)
if args.from_commit is None or args.to_commit is None:
print("--from-commit and --to-commit must be specified if --skip-query is not used")
exit(1)
i = 0
page_size = 80
cursor = f"{args.from_commit} {i}"
while True:
try:
r = github.graphql(
query=PRS_QUERY,
variables={
"owner": user,
"name": repo,
"after": cursor,
"pageSize": page_size,
},
)
except RuntimeError as e:
print(f"{e}\nPlease check enviroment variable GITHUB_TOKEN whether is valid.")
exit(1)
data = r["data"]["repository"]["defaultBranchRef"]["target"]["history"]
if not data["pageInfo"]["hasNextPage"]:
break
cursor = data["pageInfo"]["endCursor"]
results = data["nodes"]
to_add = []
stop = False
for r in results:
if r["oid"] == args.to_commit:
print(f"Found {r['oid']}, stopping")
stop = True
break
else:
to_add.append(r)
oids = [r["oid"] for r in to_add]
print(oids)
append_and_save(to_add, cache)
if stop:
break
print(i)
i += page_size
def write_csv(
filename: str, data: list[dict[str, Any]], threshold_filter: Callable[[dict[str, Any]], bool]
) -> None:
with open(filename, "w", newline="") as csvfile:
writer = csv.writer(csvfile, quotechar='"')
writer.writerow(
(
"category",
"subject",
"date",
"url",
"author",
"pr_title_tags",
"pr_title",
"additions",
"deletions",
"additions+deletions>threshold",
"changed_files",
)
)
for item in data:
nodes = item["associatedPullRequests"]["nodes"]
if len(nodes) == 0:
continue
pr = nodes[0]
tags = tags_from_title(pr["title"])
actual_tags = []
for t in tags:
items = [x.strip() for x in t.split(",")]
actual_tags += items
tags = actual_tags
tags = [t.lower() for t in tags]
category = tags[0] if len(tags) > 0 else ""
author = pr["author"] if pr["author"] else "ghost"
author = author.get("login", "") if isinstance(author, dict) else author
writer.writerow(
(
category,
"n/a",
item["committedDate"],
f"https://github.com/apache/tvm/pull/{pr['number']}",
author,
"/".join(tags),
pr["title"].replace(",", " "),
pr["additions"],
pr["deletions"],
1 if threshold_filter(pr) else 0,
pr["changedFiles"],
)
)
print(f"{filename} generated!")
if __name__ == "__main__":
help = "List out commits with attached PRs since a certain commit"
parser = argparse.ArgumentParser(description=help)
parser.add_argument("--from-commit", help="commit to start checking PRs from")
parser.add_argument("--to-commit", help="commit to stop checking PRs from")
parser.add_argument(
"--threshold", default=0, help="sum of additions + deletions to consider large, such as 150"
)
parser.add_argument(
"--skip-query", action="store_true", help="don't query GitHub and instead use cache file"
)
args = parser.parse_args()
user = "apache"
repo = "tvm"
threshold = int(args.threshold)
cache = Path("out.pkl")
if not args.skip_query:
fetch_pr_data(args, cache)
with open(cache, "rb") as f:
data = pickle.load(f)
print(f"Found {len(data)} PRs")
write_csv(
filename="out_pr_gathered.csv",
data=data,
threshold_filter=lambda pr: pr["additions"] + pr["deletions"] > threshold,
)
+75
View File
@@ -0,0 +1,75 @@
#!/usr/bin/env python3
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
import argparse
import subprocess
import sys
LINK_BASE = "https://github.com/apache/tvm-rfcs/blob/main/"
COMMIT_BASE = "https://github.com/apache/tvm-rfcs/commit/"
def sprint(*args):
print(*args, file=sys.stderr)
if __name__ == "__main__":
help = "List out RFCs since a commit"
parser = argparse.ArgumentParser(description=help)
parser.add_argument("--since-commit", required=True, help="last commit to include")
parser.add_argument("--rfcs-repo", required=True, help="path to checkout of apache/tvm-rfcs")
args = parser.parse_args()
user = "apache"
repo = "tvm"
rfc_repo = args.rfcs_repo
subprocess.run("git fetch origin main", cwd=rfc_repo, shell=True)
subprocess.run("git checkout main", cwd=rfc_repo, shell=True)
subprocess.run("git reset --hard origin/main", cwd=rfc_repo, shell=True)
r = subprocess.run(
f"git log {args.since_commit}..HEAD --format='%H %s'",
cwd=rfc_repo,
shell=True,
stdout=subprocess.PIPE,
encoding="utf-8",
)
commits = r.stdout.strip().split("\n")
for commit in commits:
parts = commit.split()
commit = parts[0]
subject = " ".join(parts[1:])
r2 = subprocess.run(
f"git diff-tree --no-commit-id --name-only -r {commit}",
cwd=rfc_repo,
shell=True,
stdout=subprocess.PIPE,
encoding="utf-8",
)
files = r2.stdout.strip().split("\n")
rfc_file = None
for file in files:
if file.startswith("rfcs/") and file.endswith(".md"):
if rfc_file is not None:
sprint(f"error on {commit} {subject}")
rfc_file = file
if rfc_file is None:
sprint(f"error on {commit} {subject}")
continue
print(f" * [{subject}]({LINK_BASE + rfc_file}) ([`{commit[:7]}`]({COMMIT_BASE + commit}))")
+255
View File
@@ -0,0 +1,255 @@
#!/usr/bin/env python3
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
import argparse
import csv
import pickle
import re
import sys
from collections import defaultdict
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parent.parent.parent.parent
sys.path.append(str(REPO_ROOT / "tests" / "scripts"))
sys.path.append(str(REPO_ROOT / "tests" / "scripts" / "github"))
sys.path.append(str(REPO_ROOT / "tests" / "scripts" / "jenkins"))
# Tag dictionary used to create a mapping relation to categorize PRs owning same tag.
TAG_DICT = {
"metaschedule": "MetaSchedule",
"cuda": "cuda & cutlass & tensorrt",
"cutlass": "cuda & cutlass & tensorrt",
"tensorrt": "cuda & cutlass & tensorrt",
"hexagon": "Hexagon",
"metal": "Metal",
"vulkan": "Vulkan",
"clml": "OpenCL & CLML",
"opencl": "OpenCL & CLML",
"openclml": "OpenCL & CLML",
"adreno": "Adreno",
"acl": "ArmComputeLibrary",
"rocm": "ROCm",
"crt": "CRT",
"web": "web",
"wasm": "web",
"runtime": "Runtime",
"aot": "AOT",
"arith": "Arith",
"byoc": "BYOC",
"community": "Community",
"tensorir": "TIR",
"tirx": "TIR",
"tensorflow": "Frontend",
"tflite": "Frontend",
"pytorch": "Frontend",
"torch": "Frontend",
"keras": "Frontend",
"frontend": "Frontend",
"onnx": "Frontend",
"roofline": "Misc",
"rpc": "Misc",
"tophub": "Misc",
"ux": "Misc",
"APP": "Misc",
"docker": "Docker",
"doc": "Docs",
"docs": "Docs",
"llvm": "LLVM",
"sve": "LLVM",
"ci": "CI",
"test": "CI",
"tests": "CI",
"testing": "CI",
"unittest": "CI",
"bugfix": "BugFix",
"fix": "BugFix",
"bug": "BugFix",
"hotfix": "BugFix",
"qnn": "Relay",
"quantization": "Relay",
"relax": "Relax",
"unity": "Relax",
"transform": "Relax",
"kvcache": "Relax",
"s_tir": "S-TIR",
"disco": "Disco",
"tvmscript": "TVMScript",
"tvmscripts": "TVMScript",
"tvmc": "TVMC",
"topi": "TOPI",
}
def strip_header(title: str, header: str) -> str:
pos = title.lower().find(header.lower())
if pos == -1:
return title
return title[0:pos] + title[pos + len(header) :].strip()
def sprint(*args):
print(*args, file=sys.stderr)
def create_pr_dict(cache: Path):
with open(cache, "rb") as f:
data = pickle.load(f)
sprint(data[1])
pr_dict = {}
for item in data:
prs = item["associatedPullRequests"]["nodes"]
if len(prs) != 1:
continue
pr = prs[0]
pr_dict[pr["number"]] = pr
return pr_dict
def categorize_csv_file(csv_path: str):
headings = defaultdict(lambda: defaultdict(list))
sprint("Opening CSV")
with open(csv_path) as f:
input_file = csv.DictReader(f)
i = 0
blank_cate_set = {"Misc"}
for row in input_file:
# print(row)
tags = row["pr_title_tags"].split("/")
tags = ["misc"] if len(tags) == 0 else tags
categories = map(lambda t: TAG_DICT.get(t.lower(), "Misc"), tags)
categories = list(categories)
categories = list(set(categories) - blank_cate_set)
category = "Misc" if len(categories) == 0 else categories[0]
subject = row["subject"].strip()
pr_number = row["url"].split("/")[-1]
if category == "" or subject == "":
sprint(f"Skipping {i}th pr with number: {pr_number}, row: {row}")
continue
headings[category][subject].append(pr_number)
i += 1
# if i > 30:
# break
return headings
if __name__ == "__main__":
help = "List out commits with attached PRs since a certain commit"
parser = argparse.ArgumentParser(description=help)
parser.add_argument(
"--notes", required=True, help="csv or markdown file of categorized PRs in order"
)
parser.add_argument(
"--is-pr-with-link",
required=False,
help="exported pr number with hyper-link for forum format",
)
parser.add_argument(
"--convert-with-link",
required=False,
help="make PR number in markdown file owning hyper-link",
)
args = parser.parse_args()
user = "apache"
repo = "tvm"
if args.convert_with_link:
with open(args.notes) as f:
lines = f.readlines()
formated = []
for line in lines:
match = re.search(r"#\d+", line)
if match:
pr_num_str = match.group()
pr_num_int = pr_num_str.replace("#", "")
pr_number_str = f"[#{pr_num_int}](https://github.com/apache/tvm/pull/{pr_num_int})"
line = line.replace(pr_num_str, pr_number_str)
formated.append(line)
result = "".join(formated)
print(result)
exit(0)
# 1. Create PR dict from cache file
cache = Path("out.pkl")
if not cache.exists():
sprint("run gather_prs.py first to generate out.pkl")
exit(1)
pr_dict = create_pr_dict(cache)
# 2. Categorize csv file as dict by category and subject (sub-category)
headings = categorize_csv_file(args.notes)
# 3. Summarize and sort all categories
def sorter(x):
if x == "Misc":
return 10
return 0
keys = list(headings.keys())
keys = list(sorted(keys))
keys = list(sorted(keys, key=sorter))
# 4. Generate markdown by loop categorized csv file dict
def pr_title(number, heading):
# print(f"number:{number}, heading:{heading}, len(pr_dict):{len(pr_dict)}")
try:
title = pr_dict[int(number)]["title"]
title = strip_header(title, heading)
except Exception:
sprint("The out.pkl file is not match with csv file.")
exit(1)
return title
output = ""
for key in keys:
value = headings[key]
if key == "DO NOT INCLUDE":
continue
value = dict(value)
output += f"### {key}\n"
misc = []
misc += value.get("n/a", [])
misc += value.get("Misc", [])
for pr_number in misc:
if args.is_pr_with_link:
pr_number_str = f"[#{pr_number}](https://github.com/apache/tvm/pull/{pr_number})"
else:
pr_number_str = f"#{pr_number}"
pr_str = f" * {pr_number_str} - {pr_title(pr_number, '[' + key + ']')}\n"
output += pr_str
for subheading, pr_numbers in value.items():
if subheading == "DO NOT INCLUDE":
continue
if subheading == "n/a" or subheading == "Misc":
continue
else:
output += f" * {subheading} - " + ", ".join([f"#{n}" for n in pr_numbers]) + "\n"
# print(value)
output += "\n"
# 5. Print markdown-format output
print(output)
@@ -0,0 +1,123 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
set -exu
######################################################
# Write current test version and rc number here
######################################################
# NOTE about "rc":
# 1. Required for test candidate, such as "rc0"
# 2. Not required for release, leave blank ""
######################################################
version="v0.16.0"
rc="rc0"
######################################################
# This script is used test release (cancdidate)
# packages uploading to apache.org, github.com
# before release vote starts or release.
#
# The release (candidate) package contains files
# below:
# 1. apache-tvm-src-${version_rc}.tar.gz.asc
# 2. apache-tvm-src-${version_rc}.tar.gz.sha512
# 3. apache-tvm-src-${version_rc}.tar.gz
######################################################
version_rc="${version}"
apache_prefix="${version}"
if [ "$rc" != "" ]; then
apache_prefix="${version_rc}-${rc}"
version_rc="${version_rc}.${rc}"
fi
mkdir test_tvm_${version_rc}
cd test_tvm_${version_rc}
echo "[setup] Create an isolated virtualenv ..."
# Used for the build, dependency install, and import below so this script never
# pollutes the caller's environment and works on PEP 668 "externally-managed"
# system pythons. Requires the venv module (e.g. the python3-venv package on Debian).
python3 -m venv .venv
export PATH="$PWD/.venv/bin:$PATH"
python3 -m pip install --upgrade pip
echo "[1/10] Downloading from apache.org ..."
mkdir apache
cd apache
wget -c https://dist.apache.org/repos/dist/dev/tvm/tvm-${apache_prefix}/apache-tvm-src-${version_rc}.tar.gz.sha512
wget -c https://dist.apache.org/repos/dist/dev/tvm/tvm-${apache_prefix}/apache-tvm-src-${version_rc}.tar.gz.asc
wget -c https://dist.apache.org/repos/dist/dev/tvm/tvm-${apache_prefix}/apache-tvm-src-${version_rc}.tar.gz
md5sum ./* > ./md5sum.txt
cd -
echo "[2/10] Downloading from github.com ..."
mkdir github
cd github
wget -c https://github.com/apache/tvm/releases/download/${version_rc}/apache-tvm-src-${version_rc}.tar.gz.sha512
wget -c https://github.com/apache/tvm/releases/download/${version_rc}/apache-tvm-src-${version_rc}.tar.gz.asc
wget -c https://github.com/apache/tvm/releases/download/${version_rc}/apache-tvm-src-${version_rc}.tar.gz
md5sum ./* > ./md5sum.txt
cd -
echo "[3/10] Check difference between github.com and apache.org ..."
diff github/md5sum.txt ./apache/md5sum.txt
echo "[4/10] Checking asc ..."
cd github
gpg --verify ./apache-tvm-src-${version_rc}.tar.gz.asc ./apache-tvm-src-${version_rc}.tar.gz
echo "[5/10] Checking sha512 ..."
sha512sum -c ./apache-tvm-src-${version_rc}.tar.gz.sha512
echo "[6/10] Unzip ..."
tar -zxf apache-tvm-src-${version_rc}.tar.gz
echo "[7/10] Checking whether binary in source code ..."
output=`find apache-tvm-src-${version} -type f -exec file {} + | grep -w "ELF\|shared object" || true`
if [[ -n "$output" ]]; then
echo "Error: ELF or shared object files found:"
echo "$output"
exit 1
fi
echo "[8/10] Compile and Python Import on Linux ..."
cd apache-tvm-src-${version}
mkdir build
cd build
cp ../cmake/config.cmake .
cmake ..
make -j16
cd ..
echo "[9/10] Install Python runtime dependencies (declared in the source pyproject.toml) ..."
# Single source of truth: install exactly the runtime deps the release declares, so
# this never needs version/list maintenance when the project's dependencies change.
deps=$(python3 - <<'PY'
import re
text = open("pyproject.toml").read()
match = re.search(r"(?ms)^dependencies\s*=\s*\[(.*?)\]", text)
print(" ".join(re.findall(r'"([^"]+)"', match.group(1))))
PY
)
echo "Installing declared runtime deps: ${deps}"
python3 -m pip install ${deps}
echo "[10/10] Import TVM and print path ..."
export TVM_HOME=$(pwd)
export PYTHONPATH=$TVM_HOME/python:${PYTHONPATH:-}
python3 -c "import tvm; print(tvm.__path__)"