chore: import upstream snapshot with attribution
CICD NeMo / cicd-main-unit-tests (push) Blocked by required conditions
CICD NeMo / cicd-main-speech (push) Blocked by required conditions
CICD NeMo / cicd-test-container-build (push) Blocked by required conditions
CICD NeMo / cicd-import-tests (push) Blocked by required conditions
CICD NeMo / L0_Setup_Test_Data_And_Models (push) Blocked by required conditions
CICD NeMo / Nemo_CICD_Test (push) Blocked by required conditions
CICD NeMo / Coverage (e2e) (push) Blocked by required conditions
CICD NeMo / Coverage (unit-test) (push) Blocked by required conditions
CodeQL / Analyze (python) (push) Waiting to run
Create PR to main with cherry-pick from release / cherry-pick (push) Failing after 0s
CICD NeMo / pre-flight (push) Failing after 0s
CICD NeMo / configure (push) Has been skipped
Build, validate, and release Neural Modules / pre-flight (push) Failing after 1s
CICD NeMo / code-linting (push) Has been skipped
CICD NeMo / cicd-wait-in-queue (push) Waiting to run
Build, validate, and release Neural Modules / release (push) Has been skipped
Build, validate, and release Neural Modules / release-summary (push) Has been cancelled
CICD NeMo / cicd-main-unit-tests (push) Blocked by required conditions
CICD NeMo / cicd-main-speech (push) Blocked by required conditions
CICD NeMo / cicd-test-container-build (push) Blocked by required conditions
CICD NeMo / cicd-import-tests (push) Blocked by required conditions
CICD NeMo / L0_Setup_Test_Data_And_Models (push) Blocked by required conditions
CICD NeMo / Nemo_CICD_Test (push) Blocked by required conditions
CICD NeMo / Coverage (e2e) (push) Blocked by required conditions
CICD NeMo / Coverage (unit-test) (push) Blocked by required conditions
CodeQL / Analyze (python) (push) Waiting to run
Create PR to main with cherry-pick from release / cherry-pick (push) Failing after 0s
CICD NeMo / pre-flight (push) Failing after 0s
CICD NeMo / configure (push) Has been skipped
Build, validate, and release Neural Modules / pre-flight (push) Failing after 1s
CICD NeMo / code-linting (push) Has been skipped
CICD NeMo / cicd-wait-in-queue (push) Waiting to run
Build, validate, and release Neural Modules / release (push) Has been skipped
Build, validate, and release Neural Modules / release-summary (push) Has been cancelled
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,6 @@
|
||||
.github/ @nvidia-nemo/automation
|
||||
docker/ @nvidia-nemo/automation
|
||||
requirements/ @nvidia-nemo/automation
|
||||
pyproject.toml @nvidia-nemo/automation
|
||||
setup.py @nvidia-nemo/automation
|
||||
uv.lock @nvidia-nemo/automation
|
||||
@@ -0,0 +1,42 @@
|
||||
---
|
||||
name: Bug report
|
||||
about: Create a report to help us improve
|
||||
title: ''
|
||||
labels: bug
|
||||
assignees: ''
|
||||
|
||||
---
|
||||
|
||||
**Describe the bug**
|
||||
|
||||
A clear and concise description of what the bug is.
|
||||
|
||||
**Steps/Code to reproduce bug**
|
||||
|
||||
Please list *minimal* steps or code snippet for us to be able to reproduce the bug.
|
||||
|
||||
A helpful guide on on how to craft a minimal bug report http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports.
|
||||
|
||||
|
||||
**Expected behavior**
|
||||
|
||||
A clear and concise description of what you expected to happen.
|
||||
|
||||
**Environment overview (please complete the following information)**
|
||||
|
||||
- Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
|
||||
- Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
|
||||
- If method of install is [Docker], provide `docker pull` & `docker run` commands used
|
||||
|
||||
**Environment details**
|
||||
|
||||
If NVIDIA docker image is used you don't need to specify these.
|
||||
Otherwise, please provide:
|
||||
- OS version
|
||||
- PyTorch version
|
||||
- Python version
|
||||
|
||||
**Additional context**
|
||||
|
||||
Add any other context about the problem here.
|
||||
Example: GPU model
|
||||
@@ -0,0 +1,2 @@
|
||||
blank_issues_enabled: false
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
---
|
||||
container pulled on date: mm/dd/yyyy
|
||||
name: Dev container - Bug report
|
||||
about: Create a report to help us improve
|
||||
title: ''
|
||||
labels: bug
|
||||
assignees: ''
|
||||
|
||||
---
|
||||
|
||||
**Describe the bug**
|
||||
|
||||
A clear and concise description of what the bug is.
|
||||
|
||||
**Steps/Code to reproduce bug**
|
||||
|
||||
Please list *minimal* steps or code snippet for us to be able to reproduce the bug.
|
||||
|
||||
A helpful guide on on how to craft a minimal bug report http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports.
|
||||
|
||||
|
||||
**Expected behavior**
|
||||
|
||||
A clear and concise description of what you expected to happen.
|
||||
|
||||
**Environment overview (please complete the following information)**
|
||||
|
||||
- Environment location: Docker
|
||||
- Method of install: Please specify exact commands you used to install.
|
||||
- If method of install is [Docker], provide `docker pull` & `docker run` commands used
|
||||
|
||||
**Additional context**
|
||||
|
||||
Add any other context about the problem here.
|
||||
Example: GPU model
|
||||
@@ -0,0 +1,25 @@
|
||||
---
|
||||
name: Feature request
|
||||
about: Suggest an idea for this project
|
||||
title: ''
|
||||
labels: feature request
|
||||
assignees: ''
|
||||
|
||||
---
|
||||
|
||||
**Is your feature request related to a problem? Please describe.**
|
||||
|
||||
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
|
||||
|
||||
**Describe the solution you'd like**
|
||||
|
||||
A clear and concise description of what you want to happen.
|
||||
Provide a code snippet on how new APIs/changes would be used by others.
|
||||
|
||||
**Describe alternatives you've considered**
|
||||
|
||||
A clear and concise description of any alternative solutions or features you've considered.
|
||||
|
||||
**Additional context**
|
||||
|
||||
Add any other context or screenshots about the feature request here.
|
||||
@@ -0,0 +1,57 @@
|
||||
> [!IMPORTANT]
|
||||
> The `Update branch` button must only be pressed in very rare occassions.
|
||||
> An outdated branch is never blocking the merge of a PR.
|
||||
> Please reach out to the automation team before pressing that button.
|
||||
|
||||
# What does this PR do ?
|
||||
|
||||
Add a one line overview of what this PR aims to accomplish.
|
||||
|
||||
**Collection**: [Note which collection this PR will affect]
|
||||
|
||||
# Changelog
|
||||
|
||||
- Add specific line by line info of high level changes in this PR.
|
||||
|
||||
# Usage
|
||||
|
||||
- You can potentially add a usage example below
|
||||
|
||||
```python
|
||||
# Add a code snippet demonstrating how to use this
|
||||
```
|
||||
|
||||
# GitHub Actions CI
|
||||
|
||||
The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.
|
||||
|
||||
The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
|
||||
To re-run CI remove and add the label again.
|
||||
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".
|
||||
|
||||
# Before your PR is "Ready for review"
|
||||
|
||||
**Pre checks**:
|
||||
|
||||
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
|
||||
- [ ] Did you write any new necessary tests?
|
||||
- [ ] Did you add or update any necessary documentation?
|
||||
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
|
||||
- [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
|
||||
|
||||
**PR Type**:
|
||||
|
||||
- [ ] New Feature
|
||||
- [ ] Bugfix
|
||||
- [ ] Documentation
|
||||
|
||||
If you haven't finished some of the above items you can still open "Draft" PR.
|
||||
|
||||
## Who can review?
|
||||
|
||||
Anyone in the community is free to review the PR once the checks have passed.
|
||||
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.
|
||||
|
||||
# Additional Information
|
||||
|
||||
- Related to # (issue)
|
||||
@@ -0,0 +1,25 @@
|
||||
name: Cancel Workflow
|
||||
description: >
|
||||
Cancels the current workflow run, i.e. all jobs. Useful if you want to cancel the rest of the workflow when one job
|
||||
fails. Note that this will cause the workflow to appear cancelled, not failed.
|
||||
|
||||
# Cancelling the workflow in a post-script (like this:
|
||||
# https://docs.github.com/en/actions/creating-actions/metadata-syntax-for-github-actions#runspost; can also be done with
|
||||
# this action: https://github.com/webiny/action-post-run, see Git history of this file) wouldn't help the status, it
|
||||
# would still be cancelled. It actually indeed is, but it would be nicer to set it to failed, but there seems to be no
|
||||
# way to do this.
|
||||
|
||||
runs:
|
||||
using: "composite"
|
||||
steps:
|
||||
- name: Cancel Workflow
|
||||
# # Fork PRs won't have a token with write access to Actions, thus won't be able to cancel the workflow.
|
||||
# if: github.event.pull_request == '' || github.event.pull_request.head.repo.fork == false
|
||||
shell: bash
|
||||
run: |
|
||||
curl --verbose \
|
||||
-X POST \
|
||||
-H "Accept: application/vnd.github+json" \
|
||||
-H "Authorization: Bearer ${{ github.token }}" \
|
||||
-H "X-GitHub-Api-Version: 2022-11-28" \
|
||||
https://api.github.com/repos/${{ github.repository }}/actions/runs/${{ github.run_id }}/cancel
|
||||
@@ -0,0 +1,234 @@
|
||||
# Copyright (c) 2025, NVIDIA CORPORATION.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
name: "Test Template"
|
||||
description: "Template for running NeMo tests in a containerized environment"
|
||||
|
||||
inputs:
|
||||
runner:
|
||||
description: "Runner to use for test"
|
||||
required: true
|
||||
timeout:
|
||||
description: "Max runtime of test in minutes"
|
||||
required: false
|
||||
default: "10"
|
||||
script:
|
||||
description: "Test script to execute"
|
||||
required: true
|
||||
after_script:
|
||||
description: "Script to run after main test"
|
||||
required: false
|
||||
default: ":"
|
||||
is_optional:
|
||||
description: "Failure will cancel all other tests if set to true"
|
||||
required: false
|
||||
default: "false"
|
||||
is_unit_test:
|
||||
description: "Upload coverage as unit test"
|
||||
required: false
|
||||
default: "false"
|
||||
tests_to_run:
|
||||
description: "Tests to run"
|
||||
required: false
|
||||
default: '["all"]'
|
||||
image:
|
||||
description: "Full image URL with tag (e.g., 766267172432.dkr.ecr.us-east-1.amazonaws.com/nemo-speech:nemo_container-12345)"
|
||||
required: true
|
||||
test-data-path:
|
||||
description: "Host path to mount as /home/TestData inside the container"
|
||||
required: false
|
||||
default: /mnt/datadrive/TestData
|
||||
cpu-only:
|
||||
description: "Run tests on CPU only"
|
||||
required: false
|
||||
default: "false"
|
||||
test_dir:
|
||||
description: "Directory under tests/ containing the test scripts"
|
||||
required: false
|
||||
default: "functional_tests"
|
||||
runs:
|
||||
using: "composite"
|
||||
steps:
|
||||
- name: Noop
|
||||
shell: bash
|
||||
run: |
|
||||
chmod -R u+rwX ${{ github.run_id }}
|
||||
echo "noop"
|
||||
|
||||
- name: Docker system cleanup
|
||||
shell: bash
|
||||
run: |
|
||||
docker system prune -af --filter "until=24h" --filter "label!=nemo.pr_number=${{ github.event.pull_request.number || 0 }}" --force || true
|
||||
|
||||
- name: Docker pull image
|
||||
shell: bash
|
||||
run: |
|
||||
docker pull ${{ inputs.image }}
|
||||
|
||||
- name: Clean repos
|
||||
shell: bash
|
||||
run: |
|
||||
|
||||
- name: Create UUID
|
||||
id: uuid
|
||||
shell: bash
|
||||
run: |
|
||||
echo "id=$(uuidgen)" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Checkout NeMo
|
||||
uses: actions/checkout@v6
|
||||
env:
|
||||
DIR: ${{ github.run_id }}
|
||||
with:
|
||||
path: ${{ github.run_id }}/${{steps.uuid.outputs.id }}/NeMo
|
||||
|
||||
- name: Start container
|
||||
shell: bash
|
||||
env:
|
||||
DIR: ${{ github.run_id }}
|
||||
run: |
|
||||
mkdir -p $DIR
|
||||
|
||||
# Map of runner names to GPU device configurations
|
||||
declare -A GPU_CONFIGS=(
|
||||
["myVm-01"]="0,1"
|
||||
["myVm-02"]="2,3"
|
||||
["myVm-03"]="4,5"
|
||||
["myVm-04"]="6,7"
|
||||
)
|
||||
|
||||
ARG=("")
|
||||
if [[ "${{ inputs.cpu-only }}" == "false" ]]; then
|
||||
ARG=("--runtime=nvidia --gpus all")
|
||||
fi
|
||||
|
||||
cmd=$(cat <<RUN_TEST_EOF
|
||||
#!/bin/bash
|
||||
docker container rm -f nemo_container_${{ github.run_id }}_${{ inputs.runner }} || true
|
||||
docker run \
|
||||
--rm \
|
||||
-d \
|
||||
--name nemo_container_${{ github.run_id }}_${{ inputs.runner }} ${ARG[@]} \
|
||||
--shm-size=64g \
|
||||
--env TRANSFORMERS_OFFLINE=0 \
|
||||
--env NEMO_HOME="/home/TestData/nemo_home" \
|
||||
--env HYDRA_FULL_ERROR=1 \
|
||||
--env HF_HOME=/home/TestData/HF_HOME \
|
||||
--env RUN_ID=${{ github.run_id }} \
|
||||
--volume $(pwd)/${{ github.run_id }}/${{steps.uuid.outputs.id }}/NeMo:/workspace \
|
||||
--volume ${{ inputs.test-data-path }}:/home/TestData ${{ inputs.image }} \
|
||||
bash -c "sleep $(( ${{ inputs.timeout }} * 60 + 60 ))"
|
||||
RUN_TEST_EOF
|
||||
)
|
||||
|
||||
echo "$cmd" | tee "$DIR/retry_job.sh"
|
||||
bash $DIR/retry_job.sh
|
||||
|
||||
- name: Create run-script
|
||||
id: create
|
||||
env:
|
||||
DIR: ${{ github.run_id }}
|
||||
shell: bash
|
||||
run: |
|
||||
COVERAGE_PREFIX=$([[ "${{ inputs.is_unit_test }}" == "true" ]] && echo "unit-test" || echo "e2e")
|
||||
echo "coverage-prefix=$COVERAGE_PREFIX" | tee -a "$GITHUB_OUTPUT"
|
||||
|
||||
mkdir -p $DIR
|
||||
rm $DIR/.coverage || true
|
||||
rm $DIR/err.log || true
|
||||
|
||||
cmd=$(cat <<RUN_TEST_EOF
|
||||
#!/bin/bash
|
||||
|
||||
(
|
||||
set -e
|
||||
|
||||
docker exec -t nemo_container_${{ github.run_id }}_${{ inputs.runner }} bash -c '\
|
||||
bash tests/${{ inputs.test_dir }}/${{ inputs.script }}.sh && \
|
||||
echo "Finished successfully." || echo "Did not finish."'
|
||||
) 2>&1 | tee $DIR/err.log
|
||||
|
||||
RUN_TEST_EOF
|
||||
)
|
||||
|
||||
echo "timeout_in_seconds=$(( ${{ inputs.timeout }} * 60 ))" | tee -a "$GITHUB_OUTPUT"
|
||||
echo "$cmd" | tee "$DIR/job.sh"
|
||||
|
||||
- name: Run main script
|
||||
uses: nick-fields/retry@v3
|
||||
with:
|
||||
timeout_seconds: ${{ steps.create.outputs.timeout_in_seconds }}
|
||||
max_attempts: 3
|
||||
shell: bash
|
||||
retry_on: timeout
|
||||
command: /bin/bash ${{ github.run_id }}/job.sh
|
||||
on_retry_command: /bin/bash ${{ github.run_id }}/retry_job.sh
|
||||
|
||||
- name: Check result
|
||||
id: check
|
||||
shell: bash
|
||||
env:
|
||||
DIR: ${{ github.run_id }}
|
||||
run: |
|
||||
cat $DIR/err.log
|
||||
|
||||
log=$(tail -c 2000 $DIR/err.log | base64 -w 0)
|
||||
echo "log=$log" >> "$GITHUB_OUTPUT"
|
||||
|
||||
potential_infra_failure=$(cat $DIR/err.log | grep -Eqiw "device" && echo true || echo false)
|
||||
echo "potential_infra_failure=$potential_infra_failure" >> "$GITHUB_OUTPUT"
|
||||
|
||||
docker exec nemo_container_${{ github.run_id }}_${{ inputs.runner }} coverage combine
|
||||
docker exec nemo_container_${{ github.run_id }}_${{ inputs.runner }} coverage xml
|
||||
docker cp nemo_container_${{ github.run_id }}_${{ inputs.runner }}:/workspace/.coverage $DIR/.coverage
|
||||
docker cp nemo_container_${{ github.run_id }}_${{ inputs.runner }}:/workspace/coverage.xml $DIR/coverage.xml
|
||||
|
||||
coverage_report=coverage-${{ steps.create.outputs.coverage-prefix }}-${{ github.run_id }}-$(python3 -c "import uuid; print(uuid.uuid4())")
|
||||
echo "coverage_report=$coverage_report" >> "$GITHUB_OUTPUT"
|
||||
|
||||
IS_SUCCESS=$(tail -n 1 $DIR/err.log | grep -q "Finished successfully." && echo "true" || echo "false")
|
||||
|
||||
if [[ "$IS_SUCCESS" == "false" && "${{ inputs.is_optional }}" == "true" ]]; then
|
||||
echo "::warning:: Test failed, but displayed as successful because it is marked as optional."
|
||||
IS_SUCCESS=true
|
||||
fi
|
||||
|
||||
if [[ "$IS_SUCCESS" == "false" ]]; then
|
||||
echo Test did not finish successfully.
|
||||
exit 1
|
||||
fi
|
||||
|
||||
exit $EXIT_CODE
|
||||
|
||||
- name: Test coverage
|
||||
shell: bash -x -e -u -o pipefail {0}
|
||||
run: |
|
||||
docker exec -t nemo_container_${{ github.run_id }}_${{ inputs.runner }} coverage report -i
|
||||
|
||||
- name: Upload artifacts
|
||||
uses: actions/upload-artifact@v6
|
||||
if: ${{ steps.check.outputs.coverage_report != 'none' }}
|
||||
with:
|
||||
name: ${{ steps.check.outputs.coverage_report }}
|
||||
path: |
|
||||
${{ github.run_id }}/coverage.xml
|
||||
${{ github.run_id }}/.coverage
|
||||
include-hidden-files: true
|
||||
|
||||
- name: Container shutdown
|
||||
if: always()
|
||||
shell: bash
|
||||
run: |
|
||||
docker exec nemo_container_${{ github.run_id }}_${{ inputs.runner }} bash -c "chown -R $(id -u):$(id -g) /workspace"
|
||||
rm -rf $(pwd)/${{ github.run_id }}/${{steps.uuid.outputs.id }} || true
|
||||
docker container rm -f nemo_container_${{ github.run_id }}_${{ inputs.runner }} || true
|
||||
@@ -0,0 +1,3 @@
|
||||
enabled: true
|
||||
auto_sync_draft: false
|
||||
auto_sync_ready: true
|
||||
@@ -0,0 +1,41 @@
|
||||
ASR:
|
||||
- nemo/collections/asr/**/*
|
||||
- examples/asr/**/*
|
||||
- tutorials/asr/**/*
|
||||
- docs/source/asr/**/*
|
||||
- tests/collections/asr/**
|
||||
|
||||
Speaker Tasks:
|
||||
- examples/speaker_tasks/**/*
|
||||
- tutorials/speaker_tasks/**/*
|
||||
|
||||
TTS:
|
||||
- nemo/collections/tts/**/*
|
||||
- nemo/collections/common/tokenizers/text_to_speech/**
|
||||
- examples/tts/**/*
|
||||
- tutorials/tts/**/*
|
||||
- docs/source/tts/**/*
|
||||
- scripts/dataset_processing/tts/**
|
||||
- scripts/tts_dataset_files/**
|
||||
- tests/collections/tts/**
|
||||
- tests/collections/common/tokenizers/text_to_speech/**
|
||||
|
||||
Audio:
|
||||
- nemo/collections/audio/**/*
|
||||
- examples/audio/**/*
|
||||
- tutorials/audio/**/*
|
||||
- docs/source/audio/**/*
|
||||
- tests/collections/audio/**
|
||||
|
||||
core:
|
||||
- nemo/core/**/*
|
||||
- tests/core/**
|
||||
|
||||
common:
|
||||
- nemo/collections/common/**/*
|
||||
|
||||
CI:
|
||||
- .github/**/*
|
||||
- Jenkinsfile
|
||||
- Dockerfile
|
||||
- ci.groovy
|
||||
@@ -0,0 +1,79 @@
|
||||
# Copyright (c) 2025, NVIDIA CORPORATION.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
import os
|
||||
|
||||
import requests
|
||||
from github import Github
|
||||
|
||||
|
||||
def send_slack_notification():
|
||||
# Get environment variables
|
||||
gh_token = os.environ.get('GH_TOKEN')
|
||||
webhook_url = os.environ.get('SLACK_WEBHOOK')
|
||||
repository = os.environ.get('REPOSITORY')
|
||||
run_id = os.environ.get('RUN_ID')
|
||||
server_url = os.environ.get('SERVER_URL', 'https://github.com')
|
||||
pr_number = int(os.environ.get('PR_NUMBER', 0))
|
||||
branch_name = os.environ.get('BRANCH_NAME')
|
||||
|
||||
# Get failure info from GitHub API
|
||||
gh = Github(gh_token)
|
||||
repo = gh.get_repo(repository)
|
||||
|
||||
# Get failed jobs
|
||||
failed_jobs = [job.name for job in repo.get_workflow_run(int(run_id)).jobs() if job.conclusion == 'failure']
|
||||
|
||||
if pr_number != 0:
|
||||
pr = repo.get_pull(pr_number)
|
||||
|
||||
title = f"*<{server_url}/{repository}/pull/{pr_number}|PR#{pr_number}>: {pr.title.replace('`', '')}*"
|
||||
author = f"<{server_url}/{pr.user.login}|{pr.user.login}>"
|
||||
branch = f"<{server_url}/{pr.head.repo.full_name}/tree/{pr.head.ref}|{pr.head.ref}>"
|
||||
else:
|
||||
title = f"*Run on <{server_url}/{repository}/tree/{branch_name}|{branch_name}>*"
|
||||
author = "No author"
|
||||
branch = f"<{server_url}/{repository}/tree/{branch_name}|{branch_name}>"
|
||||
|
||||
blocks = [
|
||||
{
|
||||
"type": "section",
|
||||
"text": {
|
||||
"type": "mrkdwn",
|
||||
"text": (
|
||||
f"{title}\n"
|
||||
f"• Author: {author}\n"
|
||||
f"• Branch: {branch}\n"
|
||||
f"• Pipeline: <{server_url}/{repository}/actions/runs/{run_id}|View Run>\n"
|
||||
f"• Failed Jobs:\n"
|
||||
+ "\n".join(
|
||||
[
|
||||
f" • <{server_url}/{repository}/actions/runs/{run_id}|{job.split('/')[-1]}>"
|
||||
for job in failed_jobs
|
||||
if job.split('/')[-1] != 'Nemo_CICD_Test'
|
||||
]
|
||||
)
|
||||
),
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
print({"blocks": blocks})
|
||||
|
||||
# Send to Slack
|
||||
response = requests.post(webhook_url, json={"blocks": blocks})
|
||||
response.raise_for_status()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
send_slack_notification()
|
||||
@@ -0,0 +1,111 @@
|
||||
name: ~Build container template
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
image-name:
|
||||
required: true
|
||||
type: string
|
||||
description: "The name of the image to build"
|
||||
dockerfile:
|
||||
required: true
|
||||
type: string
|
||||
runner:
|
||||
required: false
|
||||
default: nemo-ci-aws-gpu-x2
|
||||
type: string
|
||||
description: "The runner to use for the build"
|
||||
registry:
|
||||
required: false
|
||||
default: 766267172432.dkr.ecr.us-east-1.amazonaws.com
|
||||
type: string
|
||||
description: "Container registry"
|
||||
outputs:
|
||||
image:
|
||||
description: "Full image URL (registry/nemo-speech:image-name-run_id)"
|
||||
value: ${{ jobs.build.outputs.image }}
|
||||
|
||||
permissions:
|
||||
id-token: write
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
pre-flight:
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
build_args: ${{ steps.manifest.outputs.BUILD_ARGS }}
|
||||
cache-from: ${{ steps.cache_from.outputs.LAST_PRS }}
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Get last merged PR
|
||||
id: cache_from
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
LAST_PRS=$(gh api graphql -f query='
|
||||
query {
|
||||
repository(owner: "NVIDIA", name: "NeMo") {
|
||||
pullRequests(states: MERGED, first: 100, orderBy: {field: UPDATED_AT, direction: DESC}) {
|
||||
nodes {
|
||||
number
|
||||
}
|
||||
}
|
||||
}
|
||||
}' | jq -r '.data.repository.pullRequests.nodes[].number' | while read -r number; do
|
||||
echo "type=registry,ref=${{ inputs.registry }}/nemo-speech:${{ inputs.image-name }}-buildcache-$number,mode=max"
|
||||
done)
|
||||
|
||||
echo "LAST_PRS<<EOF" | tee -a $GITHUB_OUTPUT
|
||||
echo "$LAST_PRS" | tee -a $GITHUB_OUTPUT
|
||||
echo "EOF" | tee -a $GITHUB_OUTPUT
|
||||
|
||||
build:
|
||||
runs-on: ${{ inputs.runner }}
|
||||
needs: [pre-flight]
|
||||
outputs:
|
||||
image: ${{ steps.image-tag.outputs.image }}
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Compute image tag
|
||||
id: image-tag
|
||||
run: |
|
||||
echo "image=${{ inputs.registry }}/nemo-speech:${{ inputs.image-name }}-${{ github.run_id }}" | tee -a $GITHUB_OUTPUT
|
||||
|
||||
- name: Compute cache keys
|
||||
id: cache-keys
|
||||
run: |
|
||||
PR_NUMBER="${{ github.event.pull_request.number }}"
|
||||
if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
|
||||
KEY="main"
|
||||
elif [[ -n "$PR_NUMBER" ]]; then
|
||||
KEY="$PR_NUMBER"
|
||||
else
|
||||
KEY=$(echo "${{ github.ref_name }}" | tr '/' '-' | tr -cd '[:alnum:]._-')
|
||||
fi
|
||||
echo "cache-to=type=registry,ref=${{ inputs.registry }}/nemo-speech:${{ inputs.image-name }}-buildcache-${KEY},mode=max" | tee -a $GITHUB_OUTPUT
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
- name: Build and push
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
file: ${{ inputs.dockerfile }}
|
||||
push: true
|
||||
context: .
|
||||
build-args: |
|
||||
IMAGE_LABEL=nemo-core
|
||||
NEMO_TAG=${{ github.sha }}
|
||||
NEMO_REPO=https://github.com/NVIDIA/NeMo
|
||||
PR_NUMBER=${{ github.event.pull_request.number || 0 }}
|
||||
cache-from: |
|
||||
type=registry,ref=${{ inputs.registry }}/nemo-speech:${{ inputs.image-name }}-buildcache-main,mode=max
|
||||
type=registry,ref=${{ inputs.registry }}/nemo-speech:${{ inputs.image-name }}-buildcache-${{ github.event.pull_request.number || 0 }},mode=max
|
||||
${{ needs.pre-flight.outputs.cache-from }}
|
||||
cache-to: ${{ steps.cache-keys.outputs.cache-to }}
|
||||
tags: |
|
||||
${{ inputs.registry }}/nemo-speech:${{ inputs.image-name }}-${{ github.run_id }}
|
||||
${{ inputs.registry }}/nemo-speech:${{ inputs.image-name }}-${{ github.sha }}
|
||||
@@ -0,0 +1,56 @@
|
||||
name: ~Bump Megatron Tag template
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
nemo-target-branch:
|
||||
required: true
|
||||
type: string
|
||||
description: "The target branch to bump"
|
||||
mcore-target-branch:
|
||||
required: true
|
||||
type: string
|
||||
description: "The target branch to bump"
|
||||
secrets:
|
||||
PAT:
|
||||
required: true
|
||||
|
||||
jobs:
|
||||
update-branch:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.nemo-target-branch }}
|
||||
|
||||
- name: Set Git config
|
||||
run: |
|
||||
git config --local user.email "actions@github.com"
|
||||
git config --local user.name "Github Actions"
|
||||
- name: Merge weekly-bump-${{ inputs.nemo-target-branch }} back to base branch
|
||||
env:
|
||||
SOURCE_BRANCH: weekly-bump-${{ inputs.nemo-target-branch }}
|
||||
TARGET_BRANCH: ${{ inputs.nemo-target-branch }}
|
||||
run: |
|
||||
if git ls-remote --exit-code origin $SOURCE_BRANCH; then
|
||||
git fetch --unshallow
|
||||
git checkout $SOURCE_BRANCH
|
||||
git pull
|
||||
git merge --no-ff $TARGET_BRANCH -m "chore: Auto-merge $TARGET_BRANCH into $SOURCE_BRANCH"
|
||||
else
|
||||
git checkout -b $SOURCE_BRANCH $TARGET_BRANCH
|
||||
fi
|
||||
git push -u origin $SOURCE_BRANCH
|
||||
|
||||
mcore:
|
||||
uses: NVIDIA-NeMo/FW-CI-templates/.github/workflows/_bump_yamlfile.yml@v0.27.1
|
||||
needs: [update-branch]
|
||||
with:
|
||||
source-repository: NVIDIA/Megatron-LM
|
||||
source-ref: ${{ inputs.mcore-target-branch }}
|
||||
yaml-path: '."vcs-dependencies"."megatron-lm".ref'
|
||||
file: requirements/manifest.json
|
||||
base-branch: weekly-bump-${{ inputs.nemo-target-branch }}
|
||||
cicd-labels: Run CICD,no-fail-fast
|
||||
pr-reviewers: ${{ inputs.pr-reviewers }}
|
||||
secrets:
|
||||
PAT: ${{ secrets.PAT }}
|
||||
@@ -0,0 +1,14 @@
|
||||
name: Create PR to main with cherry-pick from release
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
|
||||
jobs:
|
||||
cherry-pick:
|
||||
uses: NVIDIA-NeMo/FW-CI-templates/.github/workflows/_cherry_pick.yml@v0.63.0
|
||||
secrets:
|
||||
PAT: ${{ secrets.PAT }}
|
||||
SLACK_WEBHOOK_ADMIN: ${{ secrets.SLACK_WEBHOOK_ADMIN }}
|
||||
SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
|
||||
@@ -0,0 +1,245 @@
|
||||
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
name: Approve Test Queue
|
||||
|
||||
on:
|
||||
schedule:
|
||||
- cron: '*/5 * * * *' # Runs every 5 minutes
|
||||
workflow_dispatch: # Allows manual triggering
|
||||
|
||||
jobs:
|
||||
approve-queue:
|
||||
runs-on: ubuntu-latest
|
||||
environment: main
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.12"
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install requests
|
||||
|
||||
- name: Approve waiting deployments
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.PAT }}
|
||||
MAX_CONCURRENCY: ${{ vars.MAX_CONCURRENCY || 1 }}
|
||||
run: |
|
||||
python - <<EOF
|
||||
import os
|
||||
import requests
|
||||
|
||||
|
||||
# GitHub API configuration
|
||||
GITHUB_TOKEN = os.environ["GITHUB_TOKEN"]
|
||||
REPO = os.environ["GITHUB_REPOSITORY"]
|
||||
MAX_CONCURRENCY = int(os.environ["MAX_CONCURRENCY"])
|
||||
API_BASE = f"https://api.github.com/repos/{REPO}"
|
||||
|
||||
# Headers for GitHub API
|
||||
headers = {
|
||||
"Authorization": f"token {GITHUB_TOKEN}",
|
||||
"Accept": "application/vnd.github.v3+json",
|
||||
"X-GitHub-Api-Version": "2022-11-28",
|
||||
}
|
||||
|
||||
def make_request(endpoint, method="GET", data=None, allow_no_pending_deployments=False):
|
||||
"""Make a request to the GitHub API with error handling."""
|
||||
url = f"{API_BASE}/{endpoint}"
|
||||
try:
|
||||
if method == "GET":
|
||||
response = requests.get(url, headers=headers)
|
||||
else:
|
||||
response = requests.post(url, headers=headers, json=data)
|
||||
response.raise_for_status()
|
||||
response_json = response.json()
|
||||
if hasattr(response, "links") and "actions/runs?status" in endpoint:
|
||||
response_json["next"] = response.links.get("next", {}).get("url")
|
||||
|
||||
return response_json
|
||||
except requests.exceptions.HTTPError as e:
|
||||
if (
|
||||
allow_no_pending_deployments
|
||||
and e.response is not None
|
||||
and e.response.status_code == 422
|
||||
):
|
||||
try:
|
||||
response_json = e.response.json()
|
||||
except ValueError:
|
||||
response_json = {}
|
||||
if "No pending deployment requests to approve or reject" in str(response_json.get("errors", "")):
|
||||
print(f"No pending deployment requests remain for {endpoint}; skipping")
|
||||
return {"skipped": True, "reason": "no_pending_deployments"}
|
||||
|
||||
print(f"Error making request to {endpoint}: {str(e)}")
|
||||
if e.response is not None:
|
||||
print(f"Response: {e.response.text}")
|
||||
return None
|
||||
except requests.exceptions.RequestException as e:
|
||||
print(f"Error making request to {endpoint}: {str(e)}")
|
||||
if e.response is not None:
|
||||
print(f"Response: {e.response.text}")
|
||||
return None
|
||||
|
||||
|
||||
def get_workflow_runs(status):
|
||||
"""Get all workflow runs for a given status."""
|
||||
all_results = []
|
||||
endpoint = f"actions/runs?status={status}"
|
||||
while endpoint:
|
||||
response = make_request(endpoint)
|
||||
if not response:
|
||||
break
|
||||
|
||||
all_results.extend(response.get("workflow_runs", []))
|
||||
endpoint = None
|
||||
next_url = response.get("next")
|
||||
if next_url:
|
||||
endpoint = f"actions/runs?{next_url.split('?')[1]}"
|
||||
|
||||
return all_results
|
||||
|
||||
|
||||
def filter_cicd_runs(workflow_runs):
|
||||
"""Keep only CICD workflow runs."""
|
||||
return [run for run in workflow_runs if run.get("name") == "CICD NeMo"]
|
||||
|
||||
|
||||
def print_workflow_run_details(label, workflow_runs):
|
||||
"""Print the runs that are counted against concurrency."""
|
||||
if not workflow_runs:
|
||||
print(f"{label}: none")
|
||||
return
|
||||
|
||||
print(f"{label}:")
|
||||
for run in workflow_runs:
|
||||
print(
|
||||
" "
|
||||
f"id={run.get('id')} "
|
||||
f"status={run.get('status')} "
|
||||
f"branch={run.get('head_branch')} "
|
||||
f"title={run.get('display_title')}"
|
||||
)
|
||||
|
||||
|
||||
# Get current running and queued workflows
|
||||
print("Fetching workflow runs...")
|
||||
queued_workflow_runs = filter_cicd_runs(get_workflow_runs("queued"))
|
||||
in_progress_workflow_runs = filter_cicd_runs(get_workflow_runs("in_progress"))
|
||||
print_workflow_run_details("Queued CICD workflows counted against concurrency", queued_workflow_runs)
|
||||
print_workflow_run_details("Running CICD workflows counted against concurrency", in_progress_workflow_runs)
|
||||
|
||||
# Count running and queued workflows
|
||||
queued_workflows = len(queued_workflow_runs)
|
||||
in_progress_workflows = len(in_progress_workflow_runs)
|
||||
|
||||
total_workflows = queued_workflows + in_progress_workflows
|
||||
print(f"Current queued workflows: {queued_workflows}")
|
||||
print(f"Current running workflows: {in_progress_workflows}")
|
||||
print(f"Total workflows: {total_workflows}")
|
||||
print(f"Max concurrency: {MAX_CONCURRENCY}")
|
||||
|
||||
if total_workflows >= MAX_CONCURRENCY:
|
||||
print("Maximum concurrency reached, no new approvals will be made")
|
||||
exit(0)
|
||||
|
||||
# Get waiting CI workflows for test environment
|
||||
print("Fetching deployments...")
|
||||
pending_workflows = filter_cicd_runs(get_workflow_runs("waiting"))
|
||||
|
||||
# Sort deployments by creation date (oldest first)
|
||||
print("Sorting workflows...")
|
||||
pending_workflows = sorted(pending_workflows, key=lambda x: x.get("created_at", ""))
|
||||
|
||||
# Process each deployment
|
||||
print("Processing ...")
|
||||
for workflow in pending_workflows:
|
||||
if total_workflows >= MAX_CONCURRENCY:
|
||||
print("Maximum concurrency reached, stopping approvals")
|
||||
break
|
||||
|
||||
workflow_id = workflow.get("id")
|
||||
workflow_name = workflow.get("display_title") or workflow.get("name") or "<unknown>"
|
||||
if not workflow_id:
|
||||
print(f"Skipping workflow without a run id: {workflow_name}")
|
||||
continue
|
||||
print(f"Approving workflow {workflow_name} with Run Id: {workflow_id}")
|
||||
|
||||
deployment_url = f"actions/runs/{workflow_id}/pending_deployments"
|
||||
pending_deployments = make_request(deployment_url)
|
||||
if not pending_deployments:
|
||||
print(f"No pending deployments found for workflow {workflow_name}; skipping")
|
||||
continue
|
||||
|
||||
environment_ids = []
|
||||
environment_names = []
|
||||
for deployment in pending_deployments:
|
||||
environment = deployment.get("environment") or {}
|
||||
environment_id = environment.get("id")
|
||||
environment_name = environment.get("name") or "<unknown>"
|
||||
if not environment_id:
|
||||
print(f"Skipping deployment without an environment id for workflow {workflow_name}")
|
||||
continue
|
||||
|
||||
environment_ids.append(environment_id)
|
||||
environment_names.append(environment_name)
|
||||
|
||||
if not environment_ids:
|
||||
print(f"No pending deployments with environment ids found for workflow {workflow_name}")
|
||||
exit(1)
|
||||
|
||||
# Approve the deployment
|
||||
status_data = {
|
||||
"environment_ids": environment_ids,
|
||||
"state": "approved",
|
||||
"comment": "Automatically approved by queue manager"
|
||||
}
|
||||
result = make_request(
|
||||
deployment_url,
|
||||
method="POST",
|
||||
data=status_data,
|
||||
allow_no_pending_deployments=True,
|
||||
)
|
||||
|
||||
if result is None:
|
||||
print(f"Failed to approve environments {environment_names} for workflow {workflow_name}")
|
||||
exit(1)
|
||||
if isinstance(result, dict) and result.get("skipped"):
|
||||
continue
|
||||
|
||||
total_workflows += 1
|
||||
|
||||
EOF
|
||||
notify:
|
||||
if: failure()
|
||||
runs-on: ubuntu-latest
|
||||
needs: [approve-queue]
|
||||
steps:
|
||||
- name: Notify
|
||||
env:
|
||||
SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
|
||||
SLACK_WEBHOOK_ADMIN: <!subteam^${{ secrets.SLACK_WEBHOOK_ADMIN }}>
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
GITHUB_REPOSITORY: ${{ github.repository }}
|
||||
run: |
|
||||
curl -X POST \
|
||||
-H 'Content-type: application/json' \
|
||||
--data "{\"text\":\":robot_joy: <https://github.com/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}|Test-queue-approval-bot workflow> failed. Please review manually.\n\ncc ${SLACK_WEBHOOK_ADMIN}\"}" \
|
||||
$SLACK_WEBHOOK
|
||||
@@ -0,0 +1,578 @@
|
||||
# Copyright (c) 2025, NVIDIA CORPORATION.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
name: NeMo E2E Speech Tests
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
test_to_run:
|
||||
required: true
|
||||
type: string
|
||||
image-name:
|
||||
required: false
|
||||
default: nemo_container_speech
|
||||
type: string
|
||||
runner:
|
||||
required: false
|
||||
default: nemo-ci-aws-gpu-x2
|
||||
type: string
|
||||
description: "Runner to use for GPU jobs"
|
||||
run_nightly:
|
||||
required: false
|
||||
default: 'false'
|
||||
type: string
|
||||
description: "Set to 'true' to run the nightly e2e suite"
|
||||
|
||||
jobs:
|
||||
build:
|
||||
uses: ./.github/workflows/_build_container.yml
|
||||
with:
|
||||
image-name: ${{ inputs.image-name }}
|
||||
dockerfile: docker/Dockerfile
|
||||
runner: ${{ inputs.runner }}
|
||||
|
||||
unit-tests:
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- script: L0_Unit_Tests_GPU_ASR_1
|
||||
runner: ${{ inputs.runner }}
|
||||
timeout: 15
|
||||
- script: L0_Unit_Tests_GPU_ASR_2
|
||||
runner: ${{ inputs.runner }}
|
||||
timeout: 15
|
||||
- script: L0_Unit_Tests_GPU_ASR_3
|
||||
runner: ${{ inputs.runner }}
|
||||
timeout: 15
|
||||
- script: L0_Unit_Tests_GPU_ASR_4
|
||||
runner: ${{ inputs.runner }}
|
||||
timeout: 15
|
||||
- script: L0_Unit_Tests_GPU_ASR_5
|
||||
runner: ${{ inputs.runner }}
|
||||
timeout: 15
|
||||
- script: L0_Unit_Tests_GPU_ASR_6
|
||||
runner: ${{ inputs.runner }}
|
||||
timeout: 15
|
||||
- script: L0_Unit_Tests_CPU_ASR_1
|
||||
runner: ${{ inputs.runner }}
|
||||
cpu-only: true
|
||||
timeout: 12
|
||||
- script: L0_Unit_Tests_CPU_ASR_2
|
||||
runner: ${{ inputs.runner }}
|
||||
cpu-only: true
|
||||
timeout: 12
|
||||
- script: L0_Unit_Tests_CPU_ASR_3
|
||||
runner: ${{ inputs.runner }}
|
||||
cpu-only: true
|
||||
timeout: 12
|
||||
- script: L0_Unit_Tests_CPU_ASR_4
|
||||
runner: ${{ inputs.runner }}
|
||||
cpu-only: true
|
||||
timeout: 12
|
||||
- script: L0_Unit_Tests_CPU_ASR_5
|
||||
runner: ${{ inputs.runner }}
|
||||
cpu-only: true
|
||||
timeout: 12
|
||||
- script: L0_Unit_Tests_CPU_ASR_6
|
||||
runner: ${{ inputs.runner }}
|
||||
cpu-only: true
|
||||
timeout: 12
|
||||
- script: L0_Unit_Tests_GPU_TTS
|
||||
runner: ${{ inputs.runner }}
|
||||
timeout: 30
|
||||
- script: L0_Unit_Tests_CPU_TTS
|
||||
runner: ${{ inputs.runner }}
|
||||
cpu-only: true
|
||||
- script: L0_Unit_Tests_GPU_Audio_1
|
||||
runner: ${{ inputs.runner }}
|
||||
timeout: 15
|
||||
- script: L0_Unit_Tests_GPU_Audio_2
|
||||
runner: ${{ inputs.runner }}
|
||||
timeout: 15
|
||||
- script: L0_Unit_Tests_CPU_Audio
|
||||
runner: ${{ inputs.runner }}
|
||||
cpu-only: true
|
||||
timeout: 20
|
||||
- script: L0_Unit_Tests_GPU_SpeechLM2
|
||||
runner: ${{ inputs.runner }}
|
||||
timeout: 20
|
||||
- script: L0_Unit_Tests_CPU_SpeechLM2_1
|
||||
runner: ${{ inputs.runner }}
|
||||
cpu-only: true
|
||||
timeout: 12
|
||||
- script: L0_Unit_Tests_CPU_SpeechLM2_2
|
||||
runner: ${{ inputs.runner }}
|
||||
cpu-only: true
|
||||
timeout: 12
|
||||
- script: L0_Unit_Tests_CPU_SpeechLM2_3
|
||||
runner: ${{ inputs.runner }}
|
||||
cpu-only: true
|
||||
timeout: 12
|
||||
- script: L0_Unit_Tests_CPU_SpeechLM2_4
|
||||
runner: ${{ inputs.runner }}
|
||||
cpu-only: true
|
||||
timeout: 12
|
||||
needs: [build]
|
||||
runs-on: ${{ matrix.runner }}
|
||||
name: ${{ matrix.script }}
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
- name: Checkout NeMo
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
path: ${{ github.run_id }}
|
||||
- name: main
|
||||
uses: ./.github/actions/test-template
|
||||
with:
|
||||
runner: ${{ runner.name }}
|
||||
script: ${{ matrix.script }}
|
||||
is_unit_test: true
|
||||
tests_to_run: ${{ inputs.test_to_run }}
|
||||
image: ${{ needs.build.outputs.image }}
|
||||
test-data-path: ${{ vars.DEFAULT_TEST_DATA_PATH || '/mnt/datadrive/TestData' }}
|
||||
timeout: ${{ matrix.timeout || 10 }}
|
||||
cpu-only: ${{ matrix.cpu-only || false }}
|
||||
is_optional: ${{ matrix.is-optional || false }}
|
||||
|
||||
e2e-tests:
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: ASR_dev_run_Speech_to_Text
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: Optional_ASR_dev_run_Speech_To_Text_Finetuning
|
||||
is-optional: true
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: Optional_ASR_dev_run_Speech_To_Text_HF_Finetuning
|
||||
is-optional: true
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: ASR_dev_run_Speech_to_Text_WPE_-_Conformer
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: ASR_dev_run_Speech_to_Text_Hybrid_RNNT_CTC_Prompt
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: ASR_dev_run_Speech_to_Text_RNNT_Prompt
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_ASR_Multi-dataloader_dev_run_Speech_to_Text_multi-dataloader
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_ASR_Multi-dataloader_dev_run_Speech_to_Label_multi-dataloader
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_ASR_Adapters_Linear_Adapters
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_ASR_Adapters_RelPos_MHA_Adapters
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Speech_to_Text_EMA
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Speech_to_Text_AED
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Speaker_dev_run_Speech_to_Label
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Speech_Estimate_Duration_Bins
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Speech_Estimate_Token_Bins
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Speech_Batch_Size_OOMptimizer
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Speech_Distributed_Batch_Size_OOMptimizer
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: Optional_L2_Speech_Batch_Size_OOMptimizer_Canary
|
||||
is-optional: true
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Speech_Transcription_Speech_to_Text_Transcribe
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Speech_Transcription_Speech_to_Text_Streaming_Infer
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Speech_Transcription_Speech_to_Text_Cache_Aware_Infer
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Speech_Transcription_Speech_to_Text_Cache_Aware_with_Prompt_Infer
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Speech_Transcription_Streaming_Inference
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Speech_Transcription_Speech_to_Text_Inference_Boost_GT
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Speech_Transcription_Speech_to_Text_Transcribe_Boost_GT
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Speech_Transcription_Canary_Transcribe_Full_Manifest
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Speech_Transcription_Canary_Transcribe_With_Prompt
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Speech_Transcription_Canary_Transcribe_Audio_Dir
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Speech_Transcription_Canary_Streaming_Full_Manifest
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Longform_Speech_Transcription_Canary_Chunked_Infer_from_Audio_Dir
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Longform_Speech_Transcription_with_TimeStamps_Canary_Chunked_Infer_from_Audio_Dir
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Longform_Speech_Transcription_with_TimeStamps_Canary_Chunked_Infer_from_Manifest
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: Speech_Checkpoints_tests
|
||||
timeout: 20
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Speaker_dev_run_Speaker_Recognition
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Speaker_dev_run_EndtoEnd_Speaker_Diarization_Sortformer
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Speaker_dev_run_EndtoEnd_Diarizer_Inference
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Speaker_dev_run_Speaker_Diarization_with_ASR_Inference
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Speaker_dev_run_Clustering_Diarizer_Inference
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Speaker_dev_run_Multispeaker_ASR_Data_Simulation
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_TTS_Fast_dev_runs_1_FastPitch
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_TTS_Fast_dev_runs_1_Hifigan
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_G2P_Models_G2P_Conformer_training_evaluation_and_inference
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: SPEECHLM_HF_Training_DuplexS2S
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: SPEECHLM_HF_Training_DuplexS2SSpeechDecoder
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: SPEECHLM_HF_Training_SALM
|
||||
timeout: 20
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_TTS_Fast_dev_runs_Magpietts_DecoderContext
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_TTS_Fast_dev_runs_Magpietts_MultiEncoder
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_TTS_Fast_dev_runs_Magpietts_MoE
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_TTS_Fast_dev_runs_Magpietts_OnlinePO
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_TTS_InferEvaluate_Magpietts_ZeroShot
|
||||
timeout: 20
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_TTS_InferEvaluate_Magpietts_SeenSpeakers
|
||||
timeout: 20
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_TTS_InferEvaluatelongform_Magpietts_ZeroShot
|
||||
timeout: 20
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_TTS_InferEvaluate_Magpietts_MoE_ZeroShot
|
||||
timeout: 20
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_TTS_InferEvaluatelongform_Magpietts_MoE_ZeroShot
|
||||
timeout: 20
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_TTS_InferEvaluate_Magpietts_FrameStacking
|
||||
timeout: 20
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_TTS_Fast_dev_runs_Magpietts_OnlineCFGDistillation
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_TTS_Fast_dev_runs_Magpietts_MoE_OnlineCFGDistillation
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_TTS_Fast_dev_runs_Magpietts_FrameStacking_OnlineCFGDistillation
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_TTS_Fast_dev_runs_EasyMagpietts_Qwen
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_TTS_Fast_dev_runs_EasyMagpietts_Nemotron
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_TTS_Fast_dev_runs_EasyMagpietts_OnlinePO
|
||||
timeout: 20
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_TTS_InferEvaluate_EasyMagpietts
|
||||
timeout: 20
|
||||
needs: [build, unit-tests]
|
||||
runs-on: ${{ matrix.runner }}
|
||||
name: ${{ matrix.is-optional && 'PLEASEFIXME_' || '' }}${{ matrix.script }}
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
- name: Checkout NeMo
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
path: ${{ github.run_id }}
|
||||
- name: main
|
||||
uses: ./.github/actions/test-template
|
||||
with:
|
||||
runner: ${{ runner.name }}
|
||||
script: ${{ matrix.script }}
|
||||
tests_to_run: ${{ inputs.test_to_run }}
|
||||
image: ${{ needs.build.outputs.image }}
|
||||
test-data-path: ${{ vars.DEFAULT_TEST_DATA_PATH || '/mnt/datadrive/TestData' }}
|
||||
timeout: ${{ matrix.timeout || 10 }}
|
||||
is_optional: ${{ matrix.is-optional || false }}
|
||||
|
||||
e2e-nightly:
|
||||
if: ${{ github.event_name == 'schedule' || inputs.run_nightly == 'true' }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_de_fastconformer_hybrid_large_pc
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_en_fastconformer_hybrid_large_pc
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_es_fastconformer_hybrid_large_pc
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_it_fastconformer_hybrid_large_pc
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_ua_fastconformer_hybrid_large_pc
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_pl_fastconformer_hybrid_large_pc
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_hr_fastconformer_hybrid_large_pc
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_be_fastconformer_hybrid_large_pc
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_fr_fastconformer_hybrid_large_pc
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_ru_fastconformer_hybrid_large_pc
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_nl_fastconformer_hybrid_large_pc
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_fa_fastconformer_hybrid_large
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_ka_fastconformer_hybrid_large_pc
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_kk_ru_fastconformer_hybrid_large
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_ka_fastconformer_hybrid_transducer_ctc_large_streaming_80ms_pc
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_uz_fastconformer_hybrid_large_pc
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_ar_fastconformer_hybrid_large_pc_v1_0
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_hy_fastconformer_hybrid_large_pc
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_en_fastconformer_hybrid_medium_streaming_80ms_pc
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_en_fastconformer_hybrid_medium_streaming_80ms
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_pt_fastconformer_hybrid_large_pc
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_es_fastconformer_hybrid_large_pc_nc
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_ar_fastconformer_hybrid_large_pcd_v1_0
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_en_fastconformer_hybrid_large_streaming_multi
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_en_fastconformer_ctc_large
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_en_fastconformer_transducer_large
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_en_fastconformer_ctc_xlarge
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_en_fastconformer_transducer_xlarge
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_en_fastconformer_transducer_xxlarge
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_en_fastconformer_ctc_xxlarge
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__stt_en_fastconformer_tdt_large
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_stt_en_fastconformer_hybrid_large_streaming_1040ms
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_stt_multilingual_fastconformer_hybrid_large_pc_blend_eu
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__parakeet_rnnt_1_1b
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__parakeet_ctc_1_1b
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__parakeet_rnnt_0_6b
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__parakeet_ctc_0_6b
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__parakeet_tdt_1_1b
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__parakeet_tdt_ctc_1_1b
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__parakeet_tdt_ctc_0_6b_ja
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__parakeet_tdt_ctc_110m
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__parakeet_tdt_0_6b_v2
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__parakeet_rnnt_110m_da_dk
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__parakeet_tdt_0_6b_v3
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__parakeet_ctc_0_6b_Vietnamese
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__canary_1b
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__canary_1b_flash
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__canary_180m_flash
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__canary_1b_v2
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__parakeet_realtime_eou_120m_v1
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__multitalker_parakeet_streaming_0_6b_v1
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__nemotron_speech_streaming_en_0_6b
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__canary_qwen_2_5b
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__diar_sortformer_4spk_v1
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__diar_streaming_sortformer_4spk_v2
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__diar_streaming_sortformer_4spk_v2_1
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_titanet_large
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__speakerverification_en_titanet_large
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__ssl_en_nest_large_v1_0
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__ssl_en_nest_xlarge_v1_0
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_vad_multilingual_marblenet
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_vad_multilingual_frame_marblenet
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__Frame_VAD_Multilingual_MarbleNet_v2_0
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__se_den_sb_16k_small
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__se_der_sb_16k_small
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__sr_ssl_flowmatching_16k_430m
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_mel_codec_44khz_medium
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_mel_codec_22khz_fullband_medium
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__low_frame_rate_speech_codec_22khz
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__audio_codec_22khz
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__audio_codec_44khz
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__mel_codec_22khz
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__mel_codec_44khz
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__nemo_nano_codec_22khz_1_78kbps_12_5fps
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__nemo_nano_codec_22khz_1_89kbps_21_5fps
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__nemo_nano_codec_22khz_0_6kbps_12_5fps
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__tts_en_fastpitch
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__tts_hifigan
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_nvidia__magpie_tts_multilingual_357m
|
||||
timeout: 15
|
||||
- runner: ${{ inputs.runner }}
|
||||
script: L2_Model_Support_tts_en_e2e_fastspeech2hifigan
|
||||
timeout: 15
|
||||
needs: [build, unit-tests]
|
||||
runs-on: ${{ matrix.runner }}
|
||||
name: ${{ matrix.script }}
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
- name: Checkout NeMo
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
path: ${{ github.run_id }}
|
||||
- name: main
|
||||
uses: ./.github/actions/test-template
|
||||
with:
|
||||
runner: ${{ runner.name }}
|
||||
script: ${{ matrix.script }}
|
||||
tests_to_run: ${{ inputs.test_to_run }}
|
||||
image: ${{ needs.build.outputs.image }}
|
||||
test-data-path: ${{ vars.DEFAULT_TEST_DATA_PATH || '/mnt/datadrive/TestData' }}
|
||||
timeout: ${{ matrix.timeout || 10 }}
|
||||
test_dir: e2e_nightly
|
||||
@@ -0,0 +1,126 @@
|
||||
# Copyright (c) 2025, NVIDIA CORPORATION.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
name: NeMo Unit Tests
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
test_to_run:
|
||||
required: true
|
||||
type: string
|
||||
runner:
|
||||
required: false
|
||||
default: nemo-ci-aws-gpu-x2
|
||||
type: string
|
||||
description: "Runner to use for GPU jobs"
|
||||
container-image:
|
||||
required: true
|
||||
type: string
|
||||
description: "Full image URL with tag for the NeMo container"
|
||||
|
||||
jobs:
|
||||
collections-common-tests:
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- script: L0_Unit_Tests_GPU_Common
|
||||
runner: ${{ inputs.runner }}
|
||||
- script: L0_Unit_Tests_CPU_Common
|
||||
runner: ${{ inputs.runner }}
|
||||
cpu-only: true
|
||||
runs-on: ${{ matrix.runner }}
|
||||
name: ${{ matrix.script }}
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
- name: Checkout NeMo
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
path: ${{ github.run_id }}
|
||||
- name: main
|
||||
uses: ./.github/actions/test-template
|
||||
with:
|
||||
runner: ${{ runner.name }}
|
||||
script: ${{ matrix.script }}
|
||||
is_unit_test: true
|
||||
tests_to_run: ${{ inputs.test_to_run }}
|
||||
image: ${{ inputs.container-image }}
|
||||
test-data-path: ${{ vars.DEFAULT_TEST_DATA_PATH || '/mnt/datadrive/TestData' }}
|
||||
cpu-only: ${{ matrix.cpu-only || false }}
|
||||
|
||||
core-tests:
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- script: L0_Unit_Tests_GPU_Core
|
||||
runner: ${{ inputs.runner }}
|
||||
- script: L0_Unit_Tests_CPU_Core
|
||||
runner: ${{ inputs.runner }}
|
||||
cpu-only: true
|
||||
- script: L0_Unit_Tests_GPU_Hydra
|
||||
runner: ${{ inputs.runner }}
|
||||
- script: L0_Unit_Tests_CPU_Hydra
|
||||
runner: ${{ inputs.runner }}
|
||||
cpu-only: true
|
||||
runs-on: ${{ matrix.runner }}
|
||||
name: ${{ matrix.script }}
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
- name: Checkout NeMo
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
path: ${{ github.run_id }}
|
||||
- name: main
|
||||
uses: ./.github/actions/test-template
|
||||
with:
|
||||
runner: ${{ runner.name }}
|
||||
script: ${{ matrix.script }}
|
||||
is_unit_test: true
|
||||
tests_to_run: ${{ inputs.test_to_run }}
|
||||
image: ${{ inputs.container-image }}
|
||||
test-data-path: ${{ vars.DEFAULT_TEST_DATA_PATH || '/mnt/datadrive/TestData' }}
|
||||
cpu-only: ${{ matrix.cpu-only || false }}
|
||||
|
||||
other-tests:
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- script: L0_Unit_Tests_GPU_Others
|
||||
runner: ${{ inputs.runner }}
|
||||
- script: L0_Unit_Tests_CPU_Others
|
||||
runner: ${{ inputs.runner }}
|
||||
cpu-only: true
|
||||
runs-on: ${{ matrix.runner }}
|
||||
name: ${{ matrix.script }}
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
- name: Checkout NeMo
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
path: ${{ github.run_id }}
|
||||
- name: main
|
||||
uses: ./.github/actions/test-template
|
||||
with:
|
||||
runner: ${{ runner.name }}
|
||||
script: ${{ matrix.script }}
|
||||
is_unit_test: true
|
||||
tests_to_run: ${{ inputs.test_to_run }}
|
||||
image: ${{ inputs.container-image }}
|
||||
test-data-path: ${{ vars.DEFAULT_TEST_DATA_PATH || '/mnt/datadrive/TestData' }}
|
||||
cpu-only: ${{ matrix.cpu-only || false }}
|
||||
is_optional: ${{ matrix.is-optional || false }}
|
||||
@@ -0,0 +1,416 @@
|
||||
# Copyright (c) 2025, NVIDIA CORPORATION.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
name: CICD NeMo
|
||||
on:
|
||||
schedule:
|
||||
- cron: 0 0 * * *
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
- r**
|
||||
- weekly-bump*
|
||||
- "pull-request/[0-9]+"
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
test_to_run:
|
||||
required: false
|
||||
default: all
|
||||
type: string
|
||||
description: Comma-separated list of tests to run. Use "all" to run the full test suite.
|
||||
|
||||
concurrency:
|
||||
# group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/main' && github.run_id || github.event.pull_request.number || github.ref }}-${{ github.event_name }}
|
||||
group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event_name }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
pre-flight:
|
||||
uses: NVIDIA-NeMo/FW-CI-templates/.github/workflows/_cicd_preflight.yml@v0.89.0
|
||||
with:
|
||||
default_runner_prefix: ${{ vars.DEFAULT_RUNNER_PREFIX }}
|
||||
non_nvidia_runner_prefix: ${{ vars.NON_NVIDIA_RUNNER_PREFIX }}
|
||||
default_test_data_path: ${{ vars.DEFAULT_TEST_DATA_PATH }}
|
||||
non_nvidia_test_data_path: ${{ vars.NON_NVIDIA_TEST_DATA_PATH }}
|
||||
default_registry: ${{ vars.DEFAULT_CONTAINER_REGISTRY }}
|
||||
non_nvidia_registry: ${{ vars.NON_NVIDIA_CONTAINER_REGISTRY }}
|
||||
sso_users_filename: ${{ vars.SSO_USERS_FILENAME }}
|
||||
secrets:
|
||||
NVIDIA_MANAGEMENT_ORG_PAT: ${{ secrets.NVIDIA_MANAGEMENT_ORG_PAT }}
|
||||
|
||||
configure:
|
||||
runs-on: ubuntu-latest
|
||||
needs: [pre-flight]
|
||||
outputs:
|
||||
test_to_run: ${{ steps.test_to_run.outputs.main }}
|
||||
is_ci_workload: ${{ steps.is_ci_workload.outputs.main }}
|
||||
no_fail_fast: ${{ steps.no_fail_fast.outputs.main }}
|
||||
components_to_run: ${{ steps.components_to_run.outputs.main }}
|
||||
run_nightly: ${{ steps.pr_labels.outputs.run_nightly }}
|
||||
skip_docs: ${{ steps.pr_labels.outputs.skip_docs }}
|
||||
skip_linting: ${{ steps.pr_labels.outputs.skip_linting }}
|
||||
env:
|
||||
TESTS_TO_RUN: ${{ inputs.test_to_run }}
|
||||
EVENT_NAME: ${{ github.event_name }}
|
||||
steps:
|
||||
- name: Checkout branch
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Select components to run
|
||||
id: components_to_run
|
||||
run: |
|
||||
echo "main=[\"speech\"]" | tee -a "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Select tests to run
|
||||
id: test_to_run
|
||||
run: |
|
||||
# For manual dispatch, use the provided input
|
||||
if [[ "$EVENT_NAME" == "workflow_dispatch" ]]; then
|
||||
TESTS_TO_RUN=$TESTS_TO_RUN
|
||||
|
||||
# For push (PR branches via copy-pr-bot, main, release) and schedule, run all tests
|
||||
elif [[ "$EVENT_NAME" == "push" || "$EVENT_NAME" == "schedule" ]]; then
|
||||
TESTS_TO_RUN=all
|
||||
|
||||
else
|
||||
echo "Unsupported event_name $EVENT_NAME provided".
|
||||
exit 1
|
||||
fi
|
||||
|
||||
parsed_string=$(echo "$TESTS_TO_RUN" | jq -c --raw-input 'split(",")')
|
||||
echo "main=${parsed_string}" | tee -a "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Check if this is a CI workload
|
||||
shell: bash
|
||||
id: is_ci_workload
|
||||
run: |
|
||||
branch_name=${GITHUB_HEAD_REF:-${GITHUB_REF#refs/heads/}}
|
||||
|
||||
if [[ "$branch_name" =~ ^bump-ci-container || "$EVENT_NAME" == "schedule" ]]; then
|
||||
is_ci_workload=true
|
||||
echo "main=true" | tee -a "$GITHUB_OUTPUT"
|
||||
else
|
||||
is_ci_workload=false
|
||||
fi
|
||||
|
||||
echo "main=$is_ci_workload" | tee -a "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Fetch PR labels
|
||||
id: pr_labels
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
REPO: ${{ github.repository }}
|
||||
run: |
|
||||
REF="${GITHUB_REF#refs/heads/}"
|
||||
if [[ "$REF" =~ ^pull-request/([0-9]+)$ ]]; then
|
||||
PR_NUMBER="${BASH_REMATCH[1]}"
|
||||
LABELS=$(gh pr view "$PR_NUMBER" --repo "$REPO" --json labels -q '[.labels[].name]')
|
||||
else
|
||||
LABELS='[]'
|
||||
fi
|
||||
echo "run_nightly=$(echo "$LABELS" | jq 'any(. == "Run e2e nightly")')" | tee -a "$GITHUB_OUTPUT"
|
||||
echo "skip_docs=$(echo "$LABELS" | jq 'any(. == "skip-docs")')" | tee -a "$GITHUB_OUTPUT"
|
||||
echo "skip_linting=$(echo "$LABELS" | jq 'any(. == "skip-linting")')" | tee -a "$GITHUB_OUTPUT"
|
||||
echo "no_fail_fast=$(echo "$LABELS" | jq 'any(. == "no-fail-fast")')" | tee -a "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Check if no-fail-fast is set
|
||||
shell: bash
|
||||
id: no_fail_fast
|
||||
env:
|
||||
HAS_FAIL_FAST_LABEL: ${{ steps.pr_labels.outputs.no_fail_fast }}
|
||||
run: |
|
||||
if [[ "$HAS_FAIL_FAST_LABEL" == "true" || "$EVENT_NAME" == "schedule" ]]; then
|
||||
no_fail_fast=true
|
||||
else
|
||||
no_fail_fast=false
|
||||
fi
|
||||
|
||||
echo "main=$no_fail_fast" | tee -a "$GITHUB_OUTPUT"
|
||||
|
||||
code-linting:
|
||||
if: needs.configure.outputs.test_to_run != '[]'
|
||||
needs: [configure]
|
||||
uses: ./.github/workflows/code-linting.yml
|
||||
with:
|
||||
skip_docs: ${{ needs.configure.outputs.skip_docs }}
|
||||
skip_linting: ${{ needs.configure.outputs.skip_linting }}
|
||||
|
||||
cicd-wait-in-queue:
|
||||
needs: [configure, code-linting]
|
||||
runs-on: ubuntu-latest
|
||||
environment: test
|
||||
if: |
|
||||
needs.configure.outputs.test_to_run != '[]'
|
||||
&& needs.configure.outputs.components_to_run != '[]'
|
||||
&& needs.configure.outputs.is_ci_workload == 'false'
|
||||
steps:
|
||||
- name: Running CI tests
|
||||
run: |
|
||||
echo "Running CI tests"
|
||||
|
||||
cicd-test-container-build:
|
||||
uses: ./.github/workflows/_build_container.yml
|
||||
needs: [pre-flight, configure, code-linting, cicd-wait-in-queue]
|
||||
if: |
|
||||
needs.configure.outputs.test_to_run != '[]'
|
||||
&& needs.configure.outputs.components_to_run != '[]'
|
||||
&& (
|
||||
success()
|
||||
|| (
|
||||
needs.cicd-wait-in-queue.result == 'skipped'
|
||||
&& needs.configure.outputs.is_ci_workload == 'true'
|
||||
)
|
||||
)
|
||||
&& !cancelled()
|
||||
with:
|
||||
image-name: nemo_container
|
||||
dockerfile: docker/Dockerfile
|
||||
runner: ${{ needs.pre-flight.outputs.runner_prefix }}
|
||||
registry: ${{ needs.pre-flight.outputs.registry }}
|
||||
|
||||
cicd-import-tests:
|
||||
if: |
|
||||
needs.configure.outputs.test_to_run != '[]'
|
||||
&& needs.configure.outputs.components_to_run != '[]'
|
||||
&& (
|
||||
success()
|
||||
|| (
|
||||
needs.cicd-wait-in-queue.result == 'skipped'
|
||||
&& needs.configure.outputs.is_ci_workload == 'true'
|
||||
)
|
||||
)
|
||||
&& !cancelled()
|
||||
needs: [cicd-test-container-build, configure, pre-flight]
|
||||
runs-on: ${{ needs.pre-flight.outputs.runner_prefix }}
|
||||
steps:
|
||||
- name: Create UUID
|
||||
id: uuid
|
||||
run: |
|
||||
echo "id=$(uuidgen)" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Checkout NeMo
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
path: ${{ github.run_id }}/${{steps.uuid.outputs.id }}/NeMo
|
||||
|
||||
- name: Run some checks
|
||||
run: |
|
||||
docker run \
|
||||
--rm \
|
||||
--device=/dev/nvidia0 \
|
||||
--gpus all \
|
||||
--shm-size=8g \
|
||||
--volume $(pwd)/${{ github.run_id }}/${{steps.uuid.outputs.id }}/NeMo:/workspace \
|
||||
--env TRANSFORMERS_OFFLINE=0 \
|
||||
--env HYDRA_FULL_ERROR=1 --env PYTHONUNBUFFERED=1 ${{ needs.cicd-test-container-build.outputs.image }} bash -c '\
|
||||
# PyTorch Lightning version
|
||||
python -c "import lightning.pytorch; print(lightning.pytorch.__version__)"
|
||||
|
||||
# Basic Import Checks
|
||||
python tests/core_ptl/check_imports.py --domain asr
|
||||
python tests/core_ptl/check_imports.py --domain tts
|
||||
'
|
||||
|
||||
L0_Setup_Test_Data_And_Models:
|
||||
needs: [configure, cicd-test-container-build, cicd-wait-in-queue, pre-flight]
|
||||
runs-on: ${{ needs.pre-flight.outputs.runner_prefix }}
|
||||
if: |
|
||||
needs.configure.outputs.test_to_run != '[]'
|
||||
&& needs.configure.outputs.components_to_run != '[]'
|
||||
&& (
|
||||
success()
|
||||
|| (
|
||||
needs.cicd-wait-in-queue.result == 'skipped'
|
||||
&& needs.configure.outputs.is_ci_workload == 'true'
|
||||
)
|
||||
)
|
||||
&& !cancelled()
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
- name: Checkout NeMo
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
path: ${{ github.run_id }}
|
||||
|
||||
- name: main
|
||||
uses: ./.github/actions/test-template
|
||||
with:
|
||||
runner: ${{ runner.name }}
|
||||
script: L0_Setup_Test_Data_And_Models
|
||||
tests_to_run: '["L0_Setup_Test_Data_And_Models"]'
|
||||
image: ${{ needs.cicd-test-container-build.outputs.image }}
|
||||
test-data-path: ${{ vars.DEFAULT_TEST_DATA_PATH || '/mnt/datadrive/TestData' }}
|
||||
|
||||
cicd-main-unit-tests:
|
||||
needs: [pre-flight, configure, cicd-test-container-build]
|
||||
uses: ./.github/workflows/cicd-main-unit-tests.yml
|
||||
if: |
|
||||
needs.configure.outputs.test_to_run != '[]'
|
||||
&& needs.configure.outputs.components_to_run != '[]'
|
||||
&& (
|
||||
success()
|
||||
|| (
|
||||
needs.cicd-wait-in-queue.result == 'skipped'
|
||||
&& needs.configure.outputs.is_ci_workload == 'true'
|
||||
)
|
||||
)
|
||||
&& !cancelled()
|
||||
with:
|
||||
test_to_run: ${{ needs.configure.outputs.test_to_run }}
|
||||
runner: ${{ needs.pre-flight.outputs.runner_prefix }}
|
||||
container-image: ${{ needs.cicd-test-container-build.outputs.image }}
|
||||
|
||||
cicd-main-speech:
|
||||
needs: [pre-flight, configure, cicd-test-container-build, cicd-main-unit-tests]
|
||||
uses: ./.github/workflows/cicd-main-speech.yml
|
||||
if: |
|
||||
(
|
||||
needs.configure.outputs.test_to_run != '[]'
|
||||
&& (
|
||||
contains(fromJson(needs.configure.outputs.components_to_run), 'speech')
|
||||
)
|
||||
)
|
||||
&& (
|
||||
success()
|
||||
|| (
|
||||
needs.cicd-wait-in-queue.result == 'skipped'
|
||||
&& needs.configure.outputs.is_ci_workload == 'true'
|
||||
)
|
||||
)
|
||||
&& !cancelled()
|
||||
with:
|
||||
test_to_run: ${{ needs.configure.outputs.test_to_run }}
|
||||
runner: ${{ needs.pre-flight.outputs.runner_prefix }}
|
||||
run_nightly: ${{ needs.configure.outputs.run_nightly }}
|
||||
|
||||
Nemo_CICD_Test:
|
||||
needs:
|
||||
- configure
|
||||
- cicd-test-container-build
|
||||
- cicd-import-tests
|
||||
- L0_Setup_Test_Data_And_Models
|
||||
- cicd-main-unit-tests
|
||||
- cicd-main-speech
|
||||
if: always()
|
||||
runs-on: ubuntu-latest
|
||||
permissions: write-all
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Get workflow result
|
||||
id: result
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
RUN_ID: ${{ github.run_id }}
|
||||
run: |
|
||||
# Get workflow run details and check job conclusions
|
||||
LATEST_ATTEMPT=$(gh run view $RUN_ID --json jobs -q '[.jobs[] | select(.conclusion != null) | .conclusion] | last')
|
||||
NUM_FAILED=$(gh run view $RUN_ID --json jobs -q '[.jobs[] | select(.conclusion == "failure") | .name] | length')
|
||||
NUM_CANCELLED=$(gh run view $RUN_ID --json jobs -q '[.jobs[] | select(.conclusion == "cancelled") | .name] | length')
|
||||
|
||||
if [[ $NUM_FAILED -eq 0 && $NUM_CANCELLED -eq 0 ]]; then
|
||||
RESULT="success"
|
||||
elif [[ $NUM_CANCELLED -gt 0 ]]; then
|
||||
RESULT="cancelled"
|
||||
else
|
||||
RESULT="failure"
|
||||
fi
|
||||
|
||||
# Output the final status
|
||||
echo "code=$RESULT" | tee -a $GITHUB_OUTPUT
|
||||
|
||||
- name: Pipeline successful, add PR comment
|
||||
if: |
|
||||
steps.result.outputs.code == 'success'
|
||||
&& github.event_name == 'push'
|
||||
&& startsWith(github.ref, 'refs/heads/pull-request/')
|
||||
&& env.SLACK_WEBHOOK != ''
|
||||
env:
|
||||
SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
|
||||
REPOSITORY: ${{ github.repository }}
|
||||
RUN_ID: ${{ github.run_id }}
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
PR_NUMBER="${GITHUB_REF#refs/heads/pull-request/}"
|
||||
PR_AUTHOR=$(gh pr view "$PR_NUMBER" --repo "$REPOSITORY" --json author -q '.author.login')
|
||||
gh pr comment "$PR_NUMBER" --repo "$REPOSITORY" --body "[🤖]: Hi @${PR_AUTHOR} 👋,
|
||||
|
||||
We wanted to let you know that a [CICD pipeline](https://github.com/${REPOSITORY}/actions/runs/${RUN_ID}) for this PR just finished successfully.
|
||||
|
||||
So it might be time to merge this PR or get some approvals."
|
||||
|
||||
- name: Exit
|
||||
if: ${{ always() }}
|
||||
env:
|
||||
RESULT: ${{ steps.result.outputs.code }}
|
||||
run: |
|
||||
if [ $RESULT == "success" ]; then
|
||||
exit 0
|
||||
else
|
||||
exit 1
|
||||
fi
|
||||
|
||||
Coverage:
|
||||
runs-on: ubuntu-latest
|
||||
needs: [configure, Nemo_CICD_Test]
|
||||
if: |
|
||||
needs.configure.outputs.test_to_run != '[]'
|
||||
&& needs.configure.outputs.components_to_run != '[]'
|
||||
&& (
|
||||
success()
|
||||
|| needs.Nemo_CICD_Test.result == 'success'
|
||||
)
|
||||
&& !cancelled()
|
||||
strategy:
|
||||
matrix:
|
||||
flag: [unit-test, e2e]
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Download coverage reports of current branch
|
||||
uses: actions/download-artifact@v7
|
||||
with:
|
||||
pattern: coverage-${{ matrix.flag }}-*
|
||||
|
||||
- name: Get total coverage of current branch
|
||||
shell: bash -x -e -u -o pipefail {0}
|
||||
if: always()
|
||||
run: |
|
||||
pip install coverage
|
||||
|
||||
ls -al .
|
||||
ls -al coverage-*/
|
||||
coverage combine --keep $(ls coverage-*/.coverage)
|
||||
coverage report -i
|
||||
rm -rf coverage-*
|
||||
ls -al
|
||||
|
||||
- name: Upload coverage reports to Codecov
|
||||
uses: codecov/codecov-action@v5
|
||||
with:
|
||||
token: ${{ secrets.CODECOV_TOKEN }}
|
||||
verbose: true
|
||||
flags: ${{ matrix.flag }}
|
||||
|
||||
- name: Upload artifacts
|
||||
uses: actions/upload-artifact@v6
|
||||
with:
|
||||
name: coverage-${{ matrix.flag }}-aggregated
|
||||
path: |
|
||||
.coverage
|
||||
include-hidden-files: true
|
||||
@@ -0,0 +1,70 @@
|
||||
name: Claude Answer Issue
|
||||
|
||||
on:
|
||||
issue_comment:
|
||||
types: [created]
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
id-token: write
|
||||
|
||||
jobs:
|
||||
authorize:
|
||||
if: >-
|
||||
!github.event.issue.pull_request &&
|
||||
contains(github.event.comment.body, '/claude answer')
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Check team membership
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
github-token: ${{ secrets.ORG_TEAM_READ_TOKEN }}
|
||||
script: |
|
||||
const username = context.payload.comment.user.login;
|
||||
try {
|
||||
const res = await github.rest.teams.getMembershipForUserInOrg({
|
||||
org: 'NVIDIA-NeMo',
|
||||
team_slug: 'speech_team',
|
||||
username,
|
||||
});
|
||||
if (res.data.state !== 'active') {
|
||||
core.setFailed(`${username} is not an active member of NVIDIA Speech Team`);
|
||||
}
|
||||
} catch (e) {
|
||||
core.setFailed(`${username} is not a member of NVIDIA Speech Team`);
|
||||
}
|
||||
|
||||
acknowledge:
|
||||
needs: authorize
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Add eyes reaction to comment
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
script: |
|
||||
await github.rest.reactions.createForIssueComment({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
comment_id: context.payload.comment.id,
|
||||
content: 'eyes'
|
||||
});
|
||||
|
||||
claude-answer:
|
||||
needs: acknowledge
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- uses: anthropics/claude-code-action@v1
|
||||
env:
|
||||
ANTHROPIC_BASE_URL: ${{ secrets.NVIDIA_INFERENCE_URL }}
|
||||
CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS: "1"
|
||||
DISABLE_PROMPT_CACHING: "1"
|
||||
with:
|
||||
prompt: |
|
||||
You are a helpful assistant for the NeMo repository.
|
||||
Answer the user's question based on the issue description, comments, and the codebase.
|
||||
Be concise and provide code references where relevant.
|
||||
Do NOT make any code changes or create PRs — only answer the question.
|
||||
anthropic_api_key: ${{ secrets.NVIDIA_INFERENCE_KEY }}
|
||||
claude_args: --model "${{ vars.CLAUDE_MODEL }}"
|
||||
@@ -0,0 +1,648 @@
|
||||
# PR Babysitter — half-autonomous CI fix loop, speech_team-only.
|
||||
#
|
||||
# Master switch: the "Has Babysitter" label. Adding it activates, removing
|
||||
# it deactivates and cancels any in-flight work.
|
||||
#
|
||||
# Authorization model:
|
||||
# - The entire workflow is restricted to the NVIDIA-NeMo/speech_team team.
|
||||
# Every job that mutates state starts with a preflight that verifies the
|
||||
# acting user (label sender / comment author / review author) is an active
|
||||
# speech_team member. Non-members cause the job to post a refusal comment
|
||||
# and revert the triggering label, leaving no misleading state behind.
|
||||
# - The workflow never runs for PRs from forks. Every PR-scoped job
|
||||
# fork-guards on head.repo.full_name == github.repository.
|
||||
#
|
||||
# Lifecycle (all participants must be speech_team members):
|
||||
# 1. "Has Babysitter" added -> activate posts a takeover comment and adds
|
||||
# "Run CICD" to start the pipeline.
|
||||
# 2. CI runs (CICD NeMo workflow).
|
||||
# 3. If all checks pass -> babysitter stays silent, done.
|
||||
# If a CI workflow fails -> workflow_run completion event triggers
|
||||
# investigate-and-propose. The Isort+Black
|
||||
# workflow isn't in the trigger list — it
|
||||
# auto-pushes its own fixes.
|
||||
# 4. investigate-and-propose: Claude investigates root cause, posts a *plan
|
||||
# comment* on the PR (tagged with `<!-- babysit-plan -->`), and adds
|
||||
# "Agent Plan Awaiting Approval". It does NOT push code.
|
||||
# 5. The PR author (who must also be speech_team) approves by replying with a
|
||||
# comment that mentions `@claude` affirmatively (e.g. `@claude go ahead`).
|
||||
# evaluate-comment-approval classifies the reply via an LLM and, on APPROVE,
|
||||
# swaps "Agent Plan Awaiting Approval" for "Agent Plan Approved".
|
||||
# 6. "Agent Plan Approved" added -> execute-fix verifies the sender is
|
||||
# speech_team, verifies a bot-authored plan comment exists, then reads the
|
||||
# plan, pushes the fix, re-adds "Run CICD", and goes back to #2.
|
||||
#
|
||||
# Termination:
|
||||
# - "Has Babysitter" removed at any time -> deactivate cancels in-progress
|
||||
# work via a shared concurrency group and clears plan-state labels.
|
||||
# - Claude can't investigate or can't execute -> ping-author-on-failure pings
|
||||
# the author and clears labels.
|
||||
#
|
||||
# Additionally, speech_team members can @claude in a PR review comment on a
|
||||
# babysitter-enabled PR to ask questions or request changes.
|
||||
#
|
||||
# Required secrets: NVIDIA_INFERENCE_URL, NVIDIA_INFERENCE_KEY, ORG_TEAM_READ_TOKEN, NEMO_RELABEL_TOKEN
|
||||
|
||||
name: Claude Code — PR Babysitter
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [labeled, unlabeled]
|
||||
pull_request_review_comment:
|
||||
types: [created]
|
||||
pull_request_review:
|
||||
types: [submitted]
|
||||
# `workflow_run` rather than `check_run`: check-runs produced by GHA jobs
|
||||
# authenticated with GITHUB_TOKEN do not fire `check_run` events on
|
||||
# downstream workflows (GitHub's recursion guard), so the babysitter never
|
||||
# saw real CI failures. `workflow_run` bypasses that restriction.
|
||||
# Omitted intentionally: "Isort and Black Formatting" (auto-pushes fixes),
|
||||
# this workflow itself, and labeler/relabel bots.
|
||||
workflow_run:
|
||||
workflows:
|
||||
- "CICD NeMo"
|
||||
- "PyLint and flake8 linting"
|
||||
- "Build, test, and publish a PyPi wheel (to testpypi)."
|
||||
- "Check __init__ files"
|
||||
- "Copyright check"
|
||||
- "CI-Install-Check"
|
||||
- "CodeQL"
|
||||
- "Secrets detector"
|
||||
types: [completed]
|
||||
issue_comment:
|
||||
types: [created]
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
pull-requests: write
|
||||
issues: write
|
||||
id-token: write
|
||||
|
||||
jobs:
|
||||
# ---- Activation: one-time setup when "Has Babysitter" is added ----
|
||||
|
||||
activate:
|
||||
if: >-
|
||||
github.event_name == 'pull_request' &&
|
||||
github.event.action == 'labeled' &&
|
||||
github.event.label.name == 'Has Babysitter' &&
|
||||
github.event.pull_request.head.repo.full_name == github.repository
|
||||
runs-on: ubuntu-latest
|
||||
concurrency:
|
||||
group: babysit-activate-${{ github.event.pull_request.number }}
|
||||
cancel-in-progress: true
|
||||
steps:
|
||||
# Preflight: only speech_team members may enable the babysitter. If a
|
||||
# non-member adds the label, revert it and explain.
|
||||
- name: Verify label sender is speech_team
|
||||
id: authz
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
github-token: ${{ secrets.ORG_TEAM_READ_TOKEN }}
|
||||
script: |
|
||||
const username = context.payload.sender.login;
|
||||
try {
|
||||
const res = await github.rest.teams.getMembershipForUserInOrg({
|
||||
org: 'NVIDIA-NeMo',
|
||||
team_slug: 'speech_team',
|
||||
username,
|
||||
});
|
||||
if (res.data.state !== 'active') {
|
||||
core.setOutput('authorized', 'false');
|
||||
return;
|
||||
}
|
||||
core.setOutput('authorized', 'true');
|
||||
} catch (e) {
|
||||
core.setOutput('authorized', 'false');
|
||||
}
|
||||
|
||||
- name: Checkout for GH CLI
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Refuse activation from non-speech_team sender
|
||||
if: steps.authz.outputs.authorized != 'true'
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.NEMO_RELABEL_TOKEN }}
|
||||
PR_NUMBER: ${{ github.event.pull_request.number }}
|
||||
SENDER: ${{ github.event.sender.login }}
|
||||
run: |
|
||||
gh pr comment "$PR_NUMBER" --body \
|
||||
"@${SENDER} the PR Babysitter is restricted to the NVIDIA-NeMo \`speech_team\`. Removing the \`Has Babysitter\` label."
|
||||
gh pr edit "$PR_NUMBER" --remove-label "Has Babysitter"
|
||||
exit 0
|
||||
|
||||
- name: Post takeover comment
|
||||
if: steps.authz.outputs.authorized == 'true'
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
script: |
|
||||
await github.rest.issues.createComment({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number: context.payload.pull_request.number,
|
||||
body: "I'll monitor this PR until CI is green. I'll post a plan for any fix and wait for your approval before pushing anything. Ping me by removing 'Has Babysitter' to cancel."
|
||||
});
|
||||
|
||||
- name: Trigger CI
|
||||
if: steps.authz.outputs.authorized == 'true'
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.NEMO_RELABEL_TOKEN }}
|
||||
PR_NUMBER: ${{ github.event.pull_request.number }}
|
||||
run: gh pr edit "$PR_NUMBER" --add-label "Run CICD"
|
||||
|
||||
# ---- @claude mentions in PR reviews (speech_team only) ----
|
||||
|
||||
authorize-review:
|
||||
if: >-
|
||||
github.event.pull_request.head.repo.full_name == github.repository &&
|
||||
((github.event_name == 'pull_request_review_comment' &&
|
||||
contains(github.event.comment.body, '@claude') &&
|
||||
contains(github.event.pull_request.labels.*.name, 'Has Babysitter')) ||
|
||||
(github.event_name == 'pull_request_review' &&
|
||||
contains(github.event.review.body, '@claude') &&
|
||||
contains(github.event.pull_request.labels.*.name, 'Has Babysitter')))
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Check team membership
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
github-token: ${{ secrets.ORG_TEAM_READ_TOKEN }}
|
||||
script: |
|
||||
const username =
|
||||
context.payload.comment?.user?.login ||
|
||||
context.payload.review?.user?.login;
|
||||
try {
|
||||
const res = await github.rest.teams.getMembershipForUserInOrg({
|
||||
org: 'NVIDIA-NeMo',
|
||||
team_slug: 'speech_team',
|
||||
username,
|
||||
});
|
||||
if (res.data.state !== 'active') {
|
||||
core.setFailed(`${username} is not an active member of NVIDIA Speech Team`);
|
||||
}
|
||||
} catch (e) {
|
||||
core.setFailed(`${username} is not a member of NVIDIA Speech Team`);
|
||||
}
|
||||
|
||||
acknowledge-review:
|
||||
needs: authorize-review
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Add eyes reaction
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
script: |
|
||||
if (context.payload.comment) {
|
||||
await github.rest.reactions.createForPullRequestReviewComment({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
comment_id: context.payload.comment.id,
|
||||
content: 'eyes'
|
||||
});
|
||||
}
|
||||
|
||||
respond-to-review:
|
||||
needs: acknowledge-review
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
|
||||
- uses: anthropics/claude-code-action@v1
|
||||
env:
|
||||
ANTHROPIC_BASE_URL: ${{ secrets.NVIDIA_INFERENCE_URL }}
|
||||
CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS: "1"
|
||||
DISABLE_PROMPT_CACHING: "1"
|
||||
with:
|
||||
anthropic_api_key: ${{ secrets.NVIDIA_INFERENCE_KEY }}
|
||||
# No prompt — Claude reads the @claude mention and responds in context
|
||||
claude_args: "--max-turns 15 --model ${{ vars.CLAUDE_MODEL }}"
|
||||
|
||||
# ---- CI failure: investigate, propose a plan, DO NOT push ----
|
||||
|
||||
check-label-for-ci:
|
||||
# Fires on CI workflow completion. The `workflow_run.pull_requests` array is
|
||||
# populated for same-repo PRs whose head SHA matches the completed run; for
|
||||
# fork PRs it is empty, which doubles as an implicit fork-guard here.
|
||||
if: >-
|
||||
github.event_name == 'workflow_run' &&
|
||||
github.event.workflow_run.conclusion == 'failure' &&
|
||||
github.event.workflow_run.pull_requests[0] != null
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
pr_number: ${{ steps.lookup.outputs.pr_number }}
|
||||
should_propose: ${{ steps.lookup.outputs.should_propose }}
|
||||
head_ref: ${{ steps.lookup.outputs.head_ref }}
|
||||
head_sha: ${{ steps.lookup.outputs.head_sha }}
|
||||
author_login: ${{ steps.lookup.outputs.author_login }}
|
||||
steps:
|
||||
- name: Look up PR and assess state
|
||||
id: lookup
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
script: |
|
||||
const prNumber = context.payload.workflow_run.pull_requests[0].number;
|
||||
const { data: pr } = await github.rest.pulls.get({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
pull_number: prNumber,
|
||||
});
|
||||
const labels = pr.labels.map(l => l.name);
|
||||
const hasBabysitter = labels.includes('Has Babysitter');
|
||||
const hasPending = labels.includes('Agent Plan Awaiting Approval');
|
||||
const hasApproved = labels.includes('Agent Plan Approved');
|
||||
const failedSha = context.payload.workflow_run.head_sha;
|
||||
const isStale = pr.head.sha !== failedSha;
|
||||
// Fork-guard: the babysitter never runs on PRs from forks.
|
||||
const isFork = pr.head.repo.full_name !== `${context.repo.owner}/${context.repo.repo}`;
|
||||
// Propose a new plan only if: babysitter is on, no plan is already
|
||||
// in flight, the failure isn't stale, and the PR is not from a fork.
|
||||
const shouldPropose = hasBabysitter && !hasPending && !hasApproved && !isStale && !isFork;
|
||||
core.setOutput('pr_number', prNumber);
|
||||
core.setOutput('should_propose', shouldPropose ? 'true' : 'false');
|
||||
core.setOutput('head_ref', pr.head.ref);
|
||||
core.setOutput('head_sha', pr.head.sha);
|
||||
core.setOutput('author_login', pr.user.login);
|
||||
if (!shouldPropose) {
|
||||
core.info(`Skipping investigate-and-propose: babysitter=${hasBabysitter}, pending=${hasPending}, approved=${hasApproved}, stale=${isStale}, fork=${isFork}`);
|
||||
}
|
||||
|
||||
investigate-and-propose:
|
||||
needs: check-label-for-ci
|
||||
if: needs.check-label-for-ci.outputs.should_propose == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
concurrency:
|
||||
group: babysit-fix-${{ needs.check-label-for-ci.outputs.pr_number }}
|
||||
cancel-in-progress: true
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ needs.check-label-for-ci.outputs.head_ref }}
|
||||
|
||||
- uses: anthropics/claude-code-action@v1
|
||||
env:
|
||||
ANTHROPIC_BASE_URL: ${{ secrets.NVIDIA_INFERENCE_URL }}
|
||||
CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS: "1"
|
||||
DISABLE_PROMPT_CACHING: "1"
|
||||
with:
|
||||
anthropic_api_key: ${{ secrets.NVIDIA_INFERENCE_KEY }}
|
||||
prompt: |
|
||||
The CI workflow "${{ github.event.workflow_run.name }}" just failed
|
||||
on PR #${{ needs.check-label-for-ci.outputs.pr_number }}.
|
||||
|
||||
You are in HALF-AUTONOMOUS mode. You MUST NOT edit files, commit,
|
||||
or push code in this job. Investigation and planning only.
|
||||
|
||||
Instructions:
|
||||
1. Inspect the failure logs for ALL currently failing CI checks on
|
||||
this PR, not just the one named above.
|
||||
2. Check `git log` for recent commits — if a prior babysitter plan
|
||||
already tried the same approach and it's still failing, call
|
||||
that out in your plan so the author can reconsider.
|
||||
3. Identify the root cause and draft a concrete fix plan covering:
|
||||
- What's broken and why
|
||||
- Which files/lines you'd change
|
||||
- The minimal diff you'd apply
|
||||
4. Post a SINGLE PR comment with this exact structure:
|
||||
|
||||
**CI Fix Plan** — awaiting approval from @${{ needs.check-label-for-ci.outputs.author_login }}
|
||||
|
||||
<your analysis and proposed change>
|
||||
|
||||
---
|
||||
To approve, **reply with an affirmative comment that mentions `@claude`**
|
||||
(e.g., `@claude go ahead`). Approval is restricted to the PR author.
|
||||
Remove the `Has Babysitter` label to cancel.
|
||||
|
||||
<!-- babysit-plan -->
|
||||
|
||||
The `<!-- babysit-plan -->` marker MUST be the last line — later
|
||||
jobs use it to find the approved plan.
|
||||
5. If you conclude the issue cannot be fixed from this PR alone,
|
||||
DO NOT include the `<!-- babysit-plan -->` marker. Instead post
|
||||
a comment that mentions @${{ needs.check-label-for-ci.outputs.author_login }}
|
||||
describing what you found and asking for help.
|
||||
claude_args: "--max-turns 10 --model ${{ vars.CLAUDE_MODEL }}"
|
||||
|
||||
- name: Mark plan awaiting approval (or stop if Claude gave up)
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.NEMO_RELABEL_TOKEN }}
|
||||
PR_NUMBER: ${{ needs.check-label-for-ci.outputs.pr_number }}
|
||||
REPO: ${{ github.repository }}
|
||||
run: |
|
||||
# Detect whether Claude actually posted a plan comment.
|
||||
PLAN_COUNT=$(gh api "repos/$REPO/issues/$PR_NUMBER/comments" \
|
||||
--jq '[.[] | select(.body | contains("<!-- babysit-plan -->"))] | length')
|
||||
if [ "$PLAN_COUNT" -gt 0 ]; then
|
||||
echo "Plan comment posted; adding 'Agent Plan Awaiting Approval'"
|
||||
gh pr edit "$PR_NUMBER" --add-label "Agent Plan Awaiting Approval"
|
||||
else
|
||||
echo "No plan posted; Claude decided it couldn't fix. Disabling babysitter."
|
||||
gh pr edit "$PR_NUMBER" --remove-label "Has Babysitter"
|
||||
fi
|
||||
|
||||
# ---- Approval Path A: affirmative comment from PR author mentioning @claude ----
|
||||
|
||||
evaluate-comment-approval:
|
||||
if: >-
|
||||
github.event_name == 'issue_comment' &&
|
||||
github.event.action == 'created' &&
|
||||
github.event.issue.pull_request != null &&
|
||||
contains(github.event.issue.labels.*.name, 'Has Babysitter') &&
|
||||
contains(github.event.issue.labels.*.name, 'Agent Plan Awaiting Approval') &&
|
||||
github.event.comment.user.type != 'Bot' &&
|
||||
github.event.comment.user.login == github.event.issue.user.login &&
|
||||
contains(github.event.comment.body, '@claude')
|
||||
runs-on: ubuntu-latest
|
||||
concurrency:
|
||||
group: babysit-fix-${{ github.event.issue.number }}
|
||||
cancel-in-progress: false
|
||||
steps:
|
||||
# Preflight: the commenter is the PR author (enforced by `if:`) AND must
|
||||
# be an active speech_team member. Also fork-guard by fetching the PR.
|
||||
- name: Verify commenter is speech_team and PR is not a fork
|
||||
id: authz
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
github-token: ${{ secrets.ORG_TEAM_READ_TOKEN }}
|
||||
script: |
|
||||
const prNumber = context.payload.issue.number;
|
||||
const { data: pr } = await github.rest.pulls.get({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
pull_number: prNumber,
|
||||
});
|
||||
const isFork = pr.head.repo.full_name !== `${context.repo.owner}/${context.repo.repo}`;
|
||||
if (isFork) {
|
||||
core.info(`PR #${prNumber} is a fork; babysitter does not operate on forks.`);
|
||||
core.setOutput('authorized', 'false');
|
||||
core.setOutput('reason', 'fork');
|
||||
return;
|
||||
}
|
||||
const username = context.payload.comment.user.login;
|
||||
try {
|
||||
const res = await github.rest.teams.getMembershipForUserInOrg({
|
||||
org: 'NVIDIA-NeMo',
|
||||
team_slug: 'speech_team',
|
||||
username,
|
||||
});
|
||||
if (res.data.state !== 'active') {
|
||||
core.setOutput('authorized', 'false');
|
||||
core.setOutput('reason', 'not-speech-team');
|
||||
return;
|
||||
}
|
||||
core.setOutput('authorized', 'true');
|
||||
} catch (e) {
|
||||
core.setOutput('authorized', 'false');
|
||||
core.setOutput('reason', 'not-speech-team');
|
||||
}
|
||||
|
||||
- name: Checkout for GH CLI
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Refuse non-speech_team approval
|
||||
if: steps.authz.outputs.authorized != 'true' && steps.authz.outputs.reason == 'not-speech-team'
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
PR_NUMBER: ${{ github.event.issue.number }}
|
||||
SENDER: ${{ github.event.comment.user.login }}
|
||||
run: |
|
||||
gh pr comment "$PR_NUMBER" --body \
|
||||
"@${SENDER} only NVIDIA-NeMo \`speech_team\` members can approve the babysitter's plan. Ignoring this reply."
|
||||
|
||||
- uses: anthropics/claude-code-action@v1
|
||||
if: steps.authz.outputs.authorized == 'true'
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.NEMO_RELABEL_TOKEN }}
|
||||
PR_NUMBER: ${{ github.event.issue.number }}
|
||||
ANTHROPIC_BASE_URL: ${{ secrets.NVIDIA_INFERENCE_URL }}
|
||||
CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS: "1"
|
||||
DISABLE_PROMPT_CACHING: "1"
|
||||
with:
|
||||
anthropic_api_key: ${{ secrets.NVIDIA_INFERENCE_KEY }}
|
||||
prompt: |
|
||||
Decide whether the PR author has approved the fix plan, then act.
|
||||
|
||||
Context:
|
||||
- PR #${{ github.event.issue.number }} has label "Agent Plan Awaiting Approval".
|
||||
- The PR author (@${{ github.event.issue.user.login }}) just posted a reply.
|
||||
- The approved plan is a previous PR comment containing `<!-- babysit-plan -->`.
|
||||
|
||||
Steps:
|
||||
1. Read the latest `<!-- babysit-plan -->` comment on the PR for the plan.
|
||||
2. Read the PR author's reply (use `gh api repos/${{ github.repository }}/issues/comments/${{ github.event.comment.id }}`).
|
||||
3. Classify the reply as exactly one of:
|
||||
- APPROVE: the author clearly agrees to proceed with the plan.
|
||||
- REJECT: the author clearly disagrees or wants changes.
|
||||
- NEITHER: the reply is a question, side remark, or ambiguous.
|
||||
4. Take action:
|
||||
- APPROVE: run exactly
|
||||
`gh pr edit $PR_NUMBER --remove-label "Agent Plan Awaiting Approval" --add-label "Agent Plan Approved"`
|
||||
and do nothing else.
|
||||
- REJECT: post a short PR comment acknowledging the rejection
|
||||
and suggesting they either remove `Has Babysitter` to cancel
|
||||
or reply with a new direction. Do NOT change labels.
|
||||
- NEITHER: post a short PR comment asking for an explicit
|
||||
approve/reject. Do NOT change labels.
|
||||
5. Do not push any code. Do not edit files in the repository.
|
||||
claude_args: "--max-turns 8 --model ${{ vars.CLAUDE_MODEL }}"
|
||||
|
||||
# ---- Execute the approved plan (the only job allowed to push code) ----
|
||||
|
||||
execute-fix:
|
||||
if: >-
|
||||
github.event_name == 'pull_request' &&
|
||||
github.event.action == 'labeled' &&
|
||||
github.event.label.name == 'Agent Plan Approved' &&
|
||||
contains(github.event.pull_request.labels.*.name, 'Has Babysitter') &&
|
||||
github.event.pull_request.head.repo.full_name == github.repository
|
||||
runs-on: ubuntu-latest
|
||||
concurrency:
|
||||
group: babysit-fix-${{ github.event.pull_request.number }}
|
||||
cancel-in-progress: true
|
||||
steps:
|
||||
# Preflight 1: the sender (who added `Agent Plan Approved`) must be
|
||||
# speech_team. Defense-in-depth against a non-member adding the label.
|
||||
- name: Verify label sender is speech_team
|
||||
id: authz
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
github-token: ${{ secrets.ORG_TEAM_READ_TOKEN }}
|
||||
script: |
|
||||
const username = context.payload.sender.login;
|
||||
try {
|
||||
const res = await github.rest.teams.getMembershipForUserInOrg({
|
||||
org: 'NVIDIA-NeMo',
|
||||
team_slug: 'speech_team',
|
||||
username,
|
||||
});
|
||||
core.setOutput('authorized', res.data.state === 'active' ? 'true' : 'false');
|
||||
} catch (e) {
|
||||
core.setOutput('authorized', 'false');
|
||||
}
|
||||
|
||||
# Preflight 2: there must exist a bot-authored `<!-- babysit-plan -->`
|
||||
# comment on this PR. Prevents execution when someone adds the approved
|
||||
# label manually without an investigator-posted plan.
|
||||
- name: Verify a bot-authored plan comment exists
|
||||
id: plan
|
||||
if: steps.authz.outputs.authorized == 'true'
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
script: |
|
||||
const comments = await github.paginate(github.rest.issues.listComments, {
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number: context.payload.pull_request.number,
|
||||
per_page: 100,
|
||||
});
|
||||
const planComment = [...comments].reverse().find(c =>
|
||||
c.body.includes('<!-- babysit-plan -->') && c.user && c.user.type === 'Bot'
|
||||
);
|
||||
core.setOutput('found', planComment ? 'true' : 'false');
|
||||
|
||||
- name: Checkout for GH CLI (refusal paths)
|
||||
if: steps.authz.outputs.authorized != 'true' || steps.plan.outputs.found != 'true'
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Refuse execution from non-speech_team sender
|
||||
if: steps.authz.outputs.authorized != 'true'
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.NEMO_RELABEL_TOKEN }}
|
||||
PR_NUMBER: ${{ github.event.pull_request.number }}
|
||||
SENDER: ${{ github.event.sender.login }}
|
||||
run: |
|
||||
gh pr comment "$PR_NUMBER" --body \
|
||||
"@${SENDER} only NVIDIA-NeMo \`speech_team\` members can approve the babysitter's plan. Removing the \`Agent Plan Approved\` label."
|
||||
gh pr edit "$PR_NUMBER" --remove-label "Agent Plan Approved" || true
|
||||
exit 0
|
||||
|
||||
- name: Refuse execution without a plan comment
|
||||
if: steps.authz.outputs.authorized == 'true' && steps.plan.outputs.found != 'true'
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.NEMO_RELABEL_TOKEN }}
|
||||
PR_NUMBER: ${{ github.event.pull_request.number }}
|
||||
run: |
|
||||
gh pr comment "$PR_NUMBER" --body \
|
||||
"No approved plan found for execution. Remove \`Agent Plan Approved\` and wait for the investigator to post a plan first."
|
||||
gh pr edit "$PR_NUMBER" --remove-label "Agent Plan Approved" || true
|
||||
exit 0
|
||||
|
||||
- uses: actions/checkout@v6
|
||||
if: steps.authz.outputs.authorized == 'true' && steps.plan.outputs.found == 'true'
|
||||
with:
|
||||
ref: ${{ github.event.pull_request.head.ref }}
|
||||
|
||||
- name: Record starting SHA
|
||||
id: start
|
||||
if: steps.authz.outputs.authorized == 'true' && steps.plan.outputs.found == 'true'
|
||||
run: echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- uses: anthropics/claude-code-action@v1
|
||||
if: steps.authz.outputs.authorized == 'true' && steps.plan.outputs.found == 'true'
|
||||
env:
|
||||
ANTHROPIC_BASE_URL: ${{ secrets.NVIDIA_INFERENCE_URL }}
|
||||
CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS: "1"
|
||||
DISABLE_PROMPT_CACHING: "1"
|
||||
with:
|
||||
anthropic_api_key: ${{ secrets.NVIDIA_INFERENCE_KEY }}
|
||||
prompt: |
|
||||
The PR author has approved your previously posted fix plan. Execute it now.
|
||||
|
||||
Instructions:
|
||||
1. Read the PR comment thread and find the most recent comment
|
||||
containing the marker `<!-- babysit-plan -->`. That is the
|
||||
approved plan.
|
||||
2. Implement exactly what the plan described. Do not expand scope.
|
||||
Do not refactor code outside the plan.
|
||||
3. Commit with DCO sign-off and push to the same branch.
|
||||
4. If the plan is no longer applicable (e.g., merge conflicts, the
|
||||
code has moved on, the reproduction no longer triggers the bug):
|
||||
post a PR comment mentioning @${{ github.event.pull_request.user.login }}
|
||||
explaining what changed and DO NOT push.
|
||||
claude_args: "--max-turns 10 --model ${{ vars.CLAUDE_MODEL }}"
|
||||
|
||||
- name: Re-trigger CI or stop loop
|
||||
if: steps.authz.outputs.authorized == 'true' && steps.plan.outputs.found == 'true'
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.NEMO_RELABEL_TOKEN }}
|
||||
PR_NUMBER: ${{ github.event.pull_request.number }}
|
||||
ORIGINAL_SHA: ${{ steps.start.outputs.sha }}
|
||||
run: |
|
||||
# Clear "Agent Plan Approved" so the next iteration can re-use it.
|
||||
gh pr edit "$PR_NUMBER" --remove-label "Agent Plan Approved" || true
|
||||
NEW_SHA=$(gh pr view "$PR_NUMBER" --json headRefOid -q '.headRefOid')
|
||||
if [ "$NEW_SHA" != "$ORIGINAL_SHA" ]; then
|
||||
echo "Fix pushed (SHA changed); re-triggering CI"
|
||||
gh pr edit "$PR_NUMBER" --add-label "Run CICD"
|
||||
else
|
||||
echo "No fix pushed; disabling babysitter"
|
||||
gh pr edit "$PR_NUMBER" --remove-label "Has Babysitter"
|
||||
fi
|
||||
|
||||
# ---- Safety net: ping author and clear state if investigate-and-propose crashes ----
|
||||
|
||||
ping-author-on-failure:
|
||||
needs: [check-label-for-ci, investigate-and-propose]
|
||||
if: failure() && needs.check-label-for-ci.outputs.should_propose == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Ping PR author
|
||||
uses: actions/github-script@v8
|
||||
env:
|
||||
PR_NUMBER: ${{ needs.check-label-for-ci.outputs.pr_number }}
|
||||
CHECK_NAME: ${{ github.event.workflow_run.name }}
|
||||
AUTHOR_LOGIN: ${{ needs.check-label-for-ci.outputs.author_login }}
|
||||
with:
|
||||
script: |
|
||||
await github.rest.issues.createComment({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number: parseInt(process.env.PR_NUMBER, 10),
|
||||
body: `I wasn't able to investigate the CI failure in \`${process.env.CHECK_NAME}\`. @${process.env.AUTHOR_LOGIN}, could you take a look?`
|
||||
});
|
||||
|
||||
- name: Checkout for GH CLI
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Clear all babysitter labels
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.NEMO_RELABEL_TOKEN }}
|
||||
PR_NUMBER: ${{ needs.check-label-for-ci.outputs.pr_number }}
|
||||
run: |
|
||||
gh pr edit "$PR_NUMBER" --remove-label "Has Babysitter" || true
|
||||
gh pr edit "$PR_NUMBER" --remove-label "Agent Plan Awaiting Approval" || true
|
||||
gh pr edit "$PR_NUMBER" --remove-label "Agent Plan Approved" || true
|
||||
|
||||
# ---- Deactivation: cancel in-progress fixes and clear state ----
|
||||
|
||||
deactivate:
|
||||
if: >-
|
||||
github.event_name == 'pull_request' &&
|
||||
github.event.action == 'unlabeled' &&
|
||||
github.event.label.name == 'Has Babysitter'
|
||||
runs-on: ubuntu-latest
|
||||
concurrency:
|
||||
group: babysit-fix-${{ github.event.pull_request.number }}
|
||||
cancel-in-progress: true
|
||||
steps:
|
||||
- name: Post deactivation comment
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
script: |
|
||||
await github.rest.issues.createComment({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number: context.payload.pull_request.number,
|
||||
body: "Babysitter deactivated for this PR."
|
||||
});
|
||||
|
||||
- name: Checkout for GH CLI
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Clear plan-state labels
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.NEMO_RELABEL_TOKEN }}
|
||||
PR_NUMBER: ${{ github.event.pull_request.number }}
|
||||
run: |
|
||||
gh pr edit "$PR_NUMBER" --remove-label "Agent Plan Awaiting Approval" || true
|
||||
gh pr edit "$PR_NUMBER" --remove-label "Agent Plan Approved" || true
|
||||
@@ -0,0 +1,119 @@
|
||||
name: Claude Fix Issue
|
||||
|
||||
on:
|
||||
issue_comment:
|
||||
types: [created]
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
pull-requests: write
|
||||
issues: write
|
||||
id-token: write
|
||||
|
||||
jobs:
|
||||
authorize:
|
||||
if: >-
|
||||
!github.event.issue.pull_request &&
|
||||
contains(github.event.comment.body, '/claude fix')
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Check team membership
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
github-token: ${{ secrets.ORG_TEAM_READ_TOKEN }}
|
||||
script: |
|
||||
const username = context.payload.comment.user.login;
|
||||
try {
|
||||
const res = await github.rest.teams.getMembershipForUserInOrg({
|
||||
org: 'NVIDIA-NeMo',
|
||||
team_slug: 'speech_team',
|
||||
username,
|
||||
});
|
||||
if (res.data.state !== 'active') {
|
||||
core.setFailed(`${username} is not an active member of NVIDIA Speech Team`);
|
||||
}
|
||||
} catch (e) {
|
||||
core.setFailed(`${username} is not a member of NVIDIA Speech Team`);
|
||||
}
|
||||
|
||||
acknowledge:
|
||||
needs: authorize
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Add eyes reaction to comment
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
script: |
|
||||
await github.rest.reactions.createForIssueComment({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
comment_id: context.payload.comment.id,
|
||||
content: 'eyes'
|
||||
});
|
||||
|
||||
claude-fix:
|
||||
needs: acknowledge
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- uses: anthropics/claude-code-action@v1
|
||||
id: claude
|
||||
env:
|
||||
ANTHROPIC_BASE_URL: ${{ secrets.NVIDIA_INFERENCE_URL }}
|
||||
CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS: "1"
|
||||
DISABLE_PROMPT_CACHING: "1"
|
||||
with:
|
||||
prompt: |
|
||||
You are a developer working on the NeMo repository.
|
||||
Implement a focused fix for the issue based on the issue description and comments.
|
||||
|
||||
Requirements:
|
||||
- Always use `git commit -s` to sign off all commits (DCO requirement).
|
||||
- Prioritize correctness and minimal scope; avoid unrelated refactors.
|
||||
- Reproduce or reason about the failure first, then implement the smallest robust fix.
|
||||
- If required, add or update tests for the changed behavior. If tests are not feasible, explain why.
|
||||
- If required, update related docs or comments when behavior or usage changes.
|
||||
|
||||
PR expectations:
|
||||
- Create a new branch and open a pull request with your changes.
|
||||
- Include a concise summary of root cause and fix based on the following PR template:
|
||||
```
|
||||
# What does this PR do ?
|
||||
Add a one line overview of what this PR aims to accomplish.
|
||||
|
||||
# Changelog
|
||||
Add specific line by line info of high level changes in this PR.
|
||||
|
||||
# Usage
|
||||
Add a usage example of the changed functionality.
|
||||
|
||||
Related to #${{ github.event.issue.number }}
|
||||
|
||||
This PR is created by Claude.
|
||||
```
|
||||
|
||||
|
||||
anthropic_api_key: ${{ secrets.NVIDIA_INFERENCE_KEY }}
|
||||
claude_args: --model "${{ vars.CLAUDE_MODEL }}"
|
||||
|
||||
- name: Label PR with agent-contribution
|
||||
if: steps.claude.outputs.branch_name
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
script: |
|
||||
const prs = await github.rest.pulls.list({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
head: `${context.repo.owner}:${process.env.BRANCH_NAME}`,
|
||||
state: 'open'
|
||||
});
|
||||
if (prs.data.length > 0) {
|
||||
await github.rest.issues.addLabels({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number: prs.data[0].number,
|
||||
labels: ['agent-contribution']
|
||||
});
|
||||
}
|
||||
env:
|
||||
BRANCH_NAME: ${{ steps.claude.outputs.branch_name }}
|
||||
@@ -0,0 +1,60 @@
|
||||
name: Claude Code Review
|
||||
|
||||
on:
|
||||
issue_comment:
|
||||
types: [created]
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
issues: write
|
||||
id-token: write
|
||||
|
||||
jobs:
|
||||
acknowledge:
|
||||
if: >-
|
||||
github.event.issue.pull_request &&
|
||||
contains(github.event.comment.body, '/claude review')
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Add eyes reaction to comment
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
script: |
|
||||
await github.rest.reactions.createForIssueComment({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
comment_id: context.payload.comment.id,
|
||||
content: 'eyes'
|
||||
});
|
||||
|
||||
claude-review:
|
||||
needs: acknowledge
|
||||
uses: NVIDIA-NeMo/FW-CI-templates/.github/workflows/_claude_review.yml@v1.7.0
|
||||
with:
|
||||
model: ${{ vars.CLAUDE_MODEL }}
|
||||
prompt: |
|
||||
You are doing a light code review. Keep it concise and actionable.
|
||||
|
||||
Focus ONLY on:
|
||||
- Critical bugs or logic errors
|
||||
- Typos in code, comments, or strings
|
||||
- Missing or insufficient test coverage for changed code
|
||||
- Outdated or inaccurate documentation affected by the changes
|
||||
|
||||
Do NOT comment on:
|
||||
- Style preferences or formatting
|
||||
- Minor naming suggestions
|
||||
- Architectural opinions or refactoring ideas
|
||||
- Performance unless there is a clear, measurable issue
|
||||
|
||||
Provide feedback using inline comments for specific code suggestions.
|
||||
Use top-level comments for general observations.
|
||||
|
||||
IMPORTANT: Do NOT approve the pull request. Only leave comments.
|
||||
|
||||
It's perfectly acceptable to not have anything to comment on.
|
||||
If you do not have anything to comment on, post "LGTM".
|
||||
secrets:
|
||||
NVIDIA_INFERENCE_URL: ${{ secrets.NVIDIA_INFERENCE_URL }}
|
||||
NVIDIA_INFERENCE_KEY: ${{ secrets.NVIDIA_INFERENCE_KEY }}
|
||||
@@ -0,0 +1,25 @@
|
||||
name: Stale-Close-Inactive-Issues-PRs
|
||||
on:
|
||||
schedule:
|
||||
- cron: "30 1 * * *"
|
||||
|
||||
jobs:
|
||||
close-issues:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
issues: write
|
||||
pull-requests: write
|
||||
steps:
|
||||
- uses: actions/stale@v6
|
||||
with:
|
||||
operations-per-run: 100
|
||||
days-before-issue-stale: 30
|
||||
days-before-issue-close: 7
|
||||
stale-issue-label: "stale"
|
||||
stale-issue-message: "This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days."
|
||||
close-issue-message: "This issue was closed because it has been inactive for 7 days since being marked as stale."
|
||||
days-before-pr-stale: 14
|
||||
days-before-pr-close: 7
|
||||
stale-pr-message: "This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days."
|
||||
close-pr-message: "This PR was closed because it has been inactive for 7 days since being marked as stale."
|
||||
repo-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
@@ -0,0 +1,50 @@
|
||||
name: Isort and Black Formatting Check
|
||||
# Check that changed files are formatted with black and isort.
|
||||
# Fails if any file needs reformatting — run `pre-commit run --all-files` locally to fix.
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- "**.py"
|
||||
types: [opened, synchronize, reopened]
|
||||
|
||||
defaults:
|
||||
run:
|
||||
shell: bash -x -e -u -o pipefail {0}
|
||||
|
||||
jobs:
|
||||
check_isort_and_black:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
steps:
|
||||
- name: Checkout branch
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Get changed files
|
||||
id: changed-files
|
||||
uses: step-security/changed-files@v45.0.1
|
||||
with:
|
||||
files: |
|
||||
**.py
|
||||
|
||||
- name: Setup Python env
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.10"
|
||||
|
||||
- name: black
|
||||
uses: psf/black@stable
|
||||
if: ${{ steps.changed-files.outputs.any_changed == 'true' }}
|
||||
with:
|
||||
options: "--check --verbose"
|
||||
# check only changed files (pass explicitly the files)
|
||||
src: "${{ steps.changed-files.outputs.all_changed_files }}"
|
||||
version: "~= 24.3"
|
||||
|
||||
- name: isort
|
||||
uses: isort/isort-action@v1
|
||||
if: ${{ steps.changed-files.outputs.any_changed == 'true' }}
|
||||
with:
|
||||
isort-version: "5.13.2"
|
||||
configuration: "--check-only --diff"
|
||||
@@ -0,0 +1,23 @@
|
||||
name: Check __init__ files
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened]
|
||||
|
||||
jobs:
|
||||
check-init-files:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.11"
|
||||
|
||||
- name: Install init-file-checker
|
||||
run: pip install init-file-checker
|
||||
|
||||
- name: Run init-file-checker
|
||||
run: init-file-checker nemo/
|
||||
@@ -0,0 +1,178 @@
|
||||
name: PyLint and flake8 linting
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened, labeled, unlabeled]
|
||||
workflow_call:
|
||||
inputs:
|
||||
skip_docs:
|
||||
required: false
|
||||
default: 'false'
|
||||
type: string
|
||||
skip_linting:
|
||||
required: false
|
||||
default: 'false'
|
||||
type: string
|
||||
|
||||
jobs:
|
||||
linting:
|
||||
name: "Domain: ${{ matrix.domain }}"
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
domain: [speech, other]
|
||||
env:
|
||||
DOMAIN: ${{ matrix.domain }}
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Get PR info
|
||||
id: get-pr-info
|
||||
if: startsWith(github.ref, 'refs/heads/pull-request/')
|
||||
uses: nv-gha-runners/get-pr-info@main
|
||||
|
||||
- name: Select filter
|
||||
id: filter
|
||||
run: |
|
||||
if [[ "$DOMAIN" == "speech" ]]; then
|
||||
FILTER=$(jq -crn '[
|
||||
"nemo/collections/common/data/lhotse/*.py",
|
||||
"nemo/collections/asr/**/*.py",
|
||||
"nemo/collections/tts/**/*.py",
|
||||
"nemo/collections/audio/**/*.py",
|
||||
"nemo/collections/multimodal/speech_llm/**/*.py",
|
||||
"nemo/collections/speechlm/**/*.py",
|
||||
"nemo/collections/speechlm2/**/*.py"
|
||||
] | join(",")')
|
||||
|
||||
else
|
||||
FILTER=$(jq -crn '[
|
||||
"nemo/**/*.py",
|
||||
"!nemo/collections/common/data/lhotse/*.py",
|
||||
"!nemo/collections/asr/**/*.py",
|
||||
"!nemo/collections/tts/**/*.py",
|
||||
"!nemo/collections/audio/**/*.py",
|
||||
"!nemo/collections/multimodal/speech_llm/**/*.py",
|
||||
"!nemo/collections/speechlm/**/*.py",
|
||||
"!nemo/collections/speechlm2/**/*.py",
|
||||
"!nemo/export/**/*.py"
|
||||
] | join(",")')
|
||||
fi
|
||||
|
||||
echo "main=$FILTER" | tee -a "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Get changed files
|
||||
id: changed-files
|
||||
uses: step-security/changed-files@v45.0.1
|
||||
with:
|
||||
sha: ${{ github.sha }}
|
||||
base_sha: ${{ fromJSON(steps.get-pr-info.outputs.pr-info || '{}').base.sha || github.event.pull_request.base.sha }}
|
||||
files: ${{ steps.filter.outputs.main }}
|
||||
files_separator: ","
|
||||
separator: " "
|
||||
|
||||
- name: Run PyLint
|
||||
id: pylint
|
||||
env:
|
||||
CHANGED_FILES: ${{ steps.changed-files.outputs.all_changed_files }}
|
||||
SKIP_DOCS: ${{ inputs.skip_docs == 'true' || contains(github.event.pull_request.labels.*.name, 'skip-docs') }}
|
||||
SKIP_LINTING: ${{ inputs.skip_linting == 'true' || contains(github.event.pull_request.labels.*.name, 'skip-linting') }}
|
||||
run: |
|
||||
if [[ -z "$CHANGED_FILES" ]]; then
|
||||
echo Nothing to lint.
|
||||
echo "exit-code=0" | tee -a "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ $SKIP_DOCS == true ]]; then
|
||||
ADDITIONAL_PYLINT_ARGS="--disable=C0115,C0116"
|
||||
else
|
||||
ADDITIONAL_PYLINT_ARGS=""
|
||||
fi
|
||||
|
||||
if [[ $SKIP_LINTING == true ]]; then
|
||||
ADDITIONAL_PYLINT_ARGS="--exit-zero"
|
||||
fi
|
||||
|
||||
pip install pylint
|
||||
set +e
|
||||
pylint $ADDITIONAL_PYLINT_ARGS --output "pylintrc.$DOMAIN.txt" --rcfile ".pylintrc.$DOMAIN" ${CHANGED_FILES[@]}
|
||||
echo "exit-code=$?" | tee -a "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Run flake8
|
||||
id: flake8
|
||||
env:
|
||||
CHANGED_FILES: ${{ steps.changed-files.outputs.all_changed_files }}
|
||||
SKIP_LINTING: ${{ inputs.skip_linting == 'true' || contains(github.event.pull_request.labels.*.name, 'skip-linting') }}
|
||||
run: |
|
||||
if [[ -z "$CHANGED_FILES" ]]; then
|
||||
echo Nothing to lint.
|
||||
echo "exit-code=0" | tee -a "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ $SKIP_LINTING == true ]]; then
|
||||
ADDITIONAL_FLAKE8_ARGS="--exit-zero"
|
||||
else
|
||||
ADDITIONAL_FLAKE8_ARGS=""
|
||||
fi
|
||||
|
||||
pip install flake8
|
||||
set +e
|
||||
flake8 $ADDITIONAL_FLAKE8_ARGS --output "flake8.$DOMAIN.txt" --config ".flake8.$DOMAIN" ${CHANGED_FILES[@]}
|
||||
echo "exit-code=$?" | tee -a "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Summary
|
||||
env:
|
||||
PYLINT: ${{ steps.pylint.outputs.exit-code == 0 }}
|
||||
FLAKE8: ${{ steps.flake8.outputs.exit-code == 0 }}
|
||||
run: |
|
||||
|
||||
if [[ "$PYLINT" != "true" ]]; then
|
||||
echo "Pylint output:" | tee -a $GITHUB_STEP_SUMMARY
|
||||
|
||||
echo '```' | tee -a $GITHUB_STEP_SUMMARY
|
||||
cat pylintrc.$DOMAIN.txt | tee -a $GITHUB_STEP_SUMMARY
|
||||
echo '```' | tee -a $GITHUB_STEP_SUMMARY
|
||||
fi
|
||||
|
||||
if [[ "$FLAKE8" != "true" ]]; then
|
||||
echo "Flake8 output:" | tee -a $GITHUB_STEP_SUMMARY
|
||||
|
||||
echo '```' | tee -a $GITHUB_STEP_SUMMARY
|
||||
cat flake8.$DOMAIN.txt | tee -a $GITHUB_STEP_SUMMARY
|
||||
echo '```' | tee -a $GITHUB_STEP_SUMMARY
|
||||
fi
|
||||
|
||||
if [[ "$PYLINT" != "true" || "$FLAKE8" != "true" ]]; then
|
||||
echo "The following directories got scanned:" | tee -a $GITHUB_STEP_SUMMARY
|
||||
|
||||
echo '```' | tee -a $GITHUB_STEP_SUMMARY
|
||||
echo ${{ steps.filter.outputs.main }} | tee -a $GITHUB_STEP_SUMMARY
|
||||
echo '```' | tee -a $GITHUB_STEP_SUMMARY
|
||||
|
||||
exit 1
|
||||
fi
|
||||
|
||||
Nemo_Linting_Test:
|
||||
needs: linting
|
||||
runs-on: ubuntu-latest
|
||||
if: always()
|
||||
steps:
|
||||
- name: Main
|
||||
env:
|
||||
RESULTS: ${{ toJson(needs.linting) }}
|
||||
run: |
|
||||
RESULT=$(echo "$RESULTS" | jq -r '.result')
|
||||
|
||||
if [[ "$RESULT" == "success" ]]; then
|
||||
echo "All passed."
|
||||
exit 0
|
||||
else
|
||||
echo "Some linting domains failed."
|
||||
exit 1
|
||||
fi
|
||||
@@ -0,0 +1,75 @@
|
||||
# For most projects, this workflow file will not need changing; you simply need
|
||||
# to commit it to your repository.
|
||||
#
|
||||
# You may wish to alter this file to override the set of languages analyzed,
|
||||
# or to provide custom queries or build logic.
|
||||
#
|
||||
# ******** NOTE ********
|
||||
# We have attempted to detect the languages in your repository. Please check
|
||||
# the `language` matrix defined below to confirm you have the correct set of
|
||||
# supported CodeQL languages.
|
||||
#
|
||||
name: "CodeQL"
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ "main", "[rv][0-9]*", "gh-pages-src" ]
|
||||
pull_request:
|
||||
# The branches below must be a subset of the branches above
|
||||
branches: [ "main" ]
|
||||
schedule:
|
||||
- cron: '19 1 * * 4'
|
||||
|
||||
jobs:
|
||||
analyze:
|
||||
name: Analyze
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
actions: read
|
||||
contents: read
|
||||
security-events: write
|
||||
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
language: [ 'python' ]
|
||||
# CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby' ]
|
||||
# Learn more about CodeQL language support at https://aka.ms/codeql-docs/language-support
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v6
|
||||
|
||||
# Initializes the CodeQL tools for scanning.
|
||||
- name: Initialize CodeQL
|
||||
uses: github/codeql-action/init@v2
|
||||
with:
|
||||
languages: ${{ matrix.language }}
|
||||
# If you wish to specify custom queries, you can do so here or in a config file.
|
||||
# By default, queries listed here will override any specified in a config file.
|
||||
# Prefix the list here with "+" to use these queries and those in the config file.
|
||||
|
||||
# Details on CodeQL's query packs refer to : https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
|
||||
queries: security-and-quality # security-extended,
|
||||
config-file: ./.github/workflows/config/codeql.yml
|
||||
|
||||
|
||||
# Autobuild attempts to build any compiled languages (C/C++, C#, Go, or Java).
|
||||
# If this step fails, then you should remove it and run the build manually (see below)
|
||||
- name: Autobuild
|
||||
uses: github/codeql-action/autobuild@v2
|
||||
|
||||
# ℹ️ Command-line programs to run using the OS shell.
|
||||
# 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
|
||||
|
||||
# If the Autobuild fails above, remove it and uncomment the following three lines.
|
||||
# modify them (or add more) to build your code if your project, please refer to the EXAMPLE below for guidance.
|
||||
|
||||
# - run: |
|
||||
# echo "Run, Build Application using script"
|
||||
# ./location_of_script_within_repo/buildscript.sh
|
||||
|
||||
- name: Perform CodeQL Analysis
|
||||
uses: github/codeql-action/analyze@v2
|
||||
with:
|
||||
category: "/language:${{matrix.language}}"
|
||||
@@ -0,0 +1,15 @@
|
||||
name: Community Bot
|
||||
|
||||
on:
|
||||
issues:
|
||||
types: [opened, edited, reopened, closed, deleted]
|
||||
issue_comment:
|
||||
types: [created, edited, deleted]
|
||||
|
||||
jobs:
|
||||
community-bot:
|
||||
uses: NVIDIA-NeMo/FW-CI-templates/.github/workflows/_community_bot.yml@v0.62.0
|
||||
with:
|
||||
community_project_id: ${{ vars.COMMUNITY_PROJECT_ID }}
|
||||
secrets:
|
||||
GH_TOKEN: ${{ secrets.PAT }}
|
||||
@@ -0,0 +1,205 @@
|
||||
{
|
||||
"categories": [
|
||||
{
|
||||
"title": "## ASR\n\n<details><summary>Changelog</summary>",
|
||||
"labels": [
|
||||
"asr"
|
||||
],
|
||||
"exclude_labels": [
|
||||
"cherry-pick"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "</details>\n\n## TTS\n\n<details><summary>Changelog</summary>",
|
||||
"labels": [
|
||||
"tts"
|
||||
],
|
||||
"exclude_labels": [
|
||||
"cherry-pick"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "</details>\n\n## NLP / NMT\n\n<details><summary>Changelog</summary>",
|
||||
"labels": [
|
||||
"nlp",
|
||||
"nmt",
|
||||
"megatron"
|
||||
],
|
||||
"exclude_labels": [
|
||||
"cherry-pick"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "</details>\n\n## Text Normalization / Inverse Text Normalization\n\n<details><summary>Changelog</summary>",
|
||||
"labels": [
|
||||
"tn",
|
||||
"itn"
|
||||
],
|
||||
"exclude_labels": [
|
||||
"cherry-pick"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "</details>\n\n## NeMo Tools\n\n<details><summary>Changelog</summary>",
|
||||
"labels": [
|
||||
"tools"
|
||||
],
|
||||
"exclude_labels": [
|
||||
"cherry-pick"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "</details>\n\n## Export\n\n<details><summary>Changelog</summary>",
|
||||
"labels": [
|
||||
"export"
|
||||
],
|
||||
"exclude_labels": [
|
||||
"cherry-pick"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "</details>\n\n## Documentation\n\n<details><summary>Changelog</summary>",
|
||||
"labels": [
|
||||
"docs"
|
||||
],
|
||||
"exclude_labels": [
|
||||
"cherry-pick"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "</details>\n\n## Bugfixes\n\n<details><summary>Changelog</summary>",
|
||||
"labels": [
|
||||
"bug"
|
||||
],
|
||||
"exclude_labels": [
|
||||
"cherry-pick"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "</details>\n\n## Cherrypick\n\n<details><summary>Changelog</summary>",
|
||||
"labels": [
|
||||
"cherry-pick"
|
||||
],
|
||||
"exclude_labels": [
|
||||
"cherry-pick"
|
||||
]
|
||||
}
|
||||
],
|
||||
"ignore_labels": [
|
||||
"ignore"
|
||||
],
|
||||
"sort": "ASC",
|
||||
"template": "\n${{CHANGELOG}}</details>\n\n## Uncategorized:\n\n<details><summary>Changelog</summary>\n\n${{UNCATEGORIZED}}\n</details>\n",
|
||||
"pr_template": "- ${{TITLE}} by @${{AUTHOR}} :: PR: #${{NUMBER}}",
|
||||
"empty_template": "${{OWNER}}\n${{REPO}}\n${{FROM_TAG}}\n${{TO_TAG}}",
|
||||
"label_extractor": [
|
||||
{
|
||||
"pattern": "(.*tts.*)|(.*g2p.*)",
|
||||
"target": "tts",
|
||||
"flags": "gimu",
|
||||
"on_property": [
|
||||
"title",
|
||||
"body"
|
||||
]
|
||||
},
|
||||
{
|
||||
"pattern": "(.*asr.*)|(.*ctc.*)|(.*rnnt.*)|(.*transducer.*)|(.*dali.*)|(.*k2.*)",
|
||||
"target": "asr",
|
||||
"flags": "gimu",
|
||||
"on_property": [
|
||||
"title",
|
||||
"body"
|
||||
]
|
||||
},
|
||||
{
|
||||
"pattern": "(.*nlp.*)|(.*punctuation.*)|(.*capitalization.*)|(.*entity.*)|(.*glue.*)|(.*entity.*)|(.*retrieval.*)|(.*entity.*)|(.*intent.*)|(.*slot.*)|(.*entity.*)|(.*language.*)|(.*qa.*)|(.*token class.*)|(.*text class.*)",
|
||||
"target": "nlp",
|
||||
"flags": "gimu",
|
||||
"on_property": [
|
||||
"title",
|
||||
"body"
|
||||
]
|
||||
},
|
||||
{
|
||||
"pattern": "(.*nmt.*)|(.*bignlp.*)|(.*megatron.*)|(.*machine.*)|(.*translation.*)|(.*gpt.*)",
|
||||
"target": "nmt",
|
||||
"flags": "gimu",
|
||||
"on_property": [
|
||||
"title",
|
||||
"body"
|
||||
]
|
||||
},
|
||||
{
|
||||
"pattern": "(.*tn.*)|(.*itn.*)|(.*text norm.*)",
|
||||
"target": "tn",
|
||||
"flags": "gimu",
|
||||
"on_property": [
|
||||
"title",
|
||||
"body"
|
||||
]
|
||||
},
|
||||
{
|
||||
"pattern": "(.*sde.*)|(.*ctc segment.*)",
|
||||
"target": "tools",
|
||||
"flags": "gimu",
|
||||
"on_property": [
|
||||
"title",
|
||||
"body"
|
||||
]
|
||||
},
|
||||
{
|
||||
"pattern": "(.*trt.*)|(.*onnx.*)|(.*export.*)",
|
||||
"target": "export",
|
||||
"flags": "gimu",
|
||||
"on_property": [
|
||||
"title",
|
||||
"body"
|
||||
]
|
||||
},
|
||||
{
|
||||
"pattern": "(.*\\[x\\] Documentation.*)",
|
||||
"target": "docs",
|
||||
"flags": "gmu",
|
||||
"on_property": [
|
||||
"title",
|
||||
"body"
|
||||
]
|
||||
},
|
||||
{
|
||||
"pattern": "(.*\\[x\\] Bugfix.*)|(.*patch.*)",
|
||||
"target": "bug",
|
||||
"flags": "gmu",
|
||||
"on_property": [
|
||||
"title",
|
||||
"body"
|
||||
]
|
||||
},
|
||||
{
|
||||
"pattern": "(.*cherry-pick.*)|(.*cherrypick.*)",
|
||||
"target": "cherrypick",
|
||||
"flags": "gimu",
|
||||
"on_property": [
|
||||
"title",
|
||||
"body"
|
||||
]
|
||||
}
|
||||
],
|
||||
"duplicate_filter": {
|
||||
"pattern": ".+",
|
||||
"on_property": "title",
|
||||
"method": "match"
|
||||
},
|
||||
"transformers": [
|
||||
{
|
||||
"pattern": "^cp:\\s*`(.+?)`\\s*into\\s*\\S+",
|
||||
"target": "$1"
|
||||
}
|
||||
],
|
||||
"max_tags_to_fetch": 100,
|
||||
"max_pull_requests": 500,
|
||||
"max_back_track_time_days": 365,
|
||||
"exclude_merge_branches": [],
|
||||
"tag_resolver": {
|
||||
"method": "semver"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
name: "CodeQL config"
|
||||
|
||||
paths:
|
||||
- nemo/
|
||||
- tests/
|
||||
- tools/
|
||||
- scripts/
|
||||
- examples/
|
||||
- .github/
|
||||
@@ -0,0 +1,22 @@
|
||||
# Copyright (c) 2020-2021, NVIDIA CORPORATION.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
name: Copyright check
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
|
||||
jobs:
|
||||
copyright-check:
|
||||
uses: NVIDIA-NeMo/FW-CI-templates/.github/workflows/_copyright_check.yml@v0.2.0
|
||||
@@ -0,0 +1,283 @@
|
||||
name: CI-Install-Check
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- "**"
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
test-installs-macos:
|
||||
name: ${{ matrix.os }}-py${{ matrix.python }}-${{ matrix.installer }}
|
||||
runs-on: ${{ matrix.os }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
os: [macos-latest]
|
||||
python: ["3.10", "3.11", "3.12"]
|
||||
installer: ["pip-install", "nemo-install"]
|
||||
steps:
|
||||
- name: Checkout repo
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Check disk space before cleanup
|
||||
run: df -h
|
||||
|
||||
- name: Free up disk space
|
||||
run: |
|
||||
# Remove unnecessary files on macOS
|
||||
sudo rm -rf /usr/local/lib/android || true
|
||||
sudo rm -rf /usr/local/.ghcup || true
|
||||
sudo rm -rf /usr/local/lib/node_modules || true
|
||||
brew cleanup || true
|
||||
# Clear pip cache
|
||||
pip cache purge || true
|
||||
|
||||
- name: Check disk space after cleanup
|
||||
run: df -h
|
||||
|
||||
- uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "${{ matrix.python }}"
|
||||
|
||||
- name: Install NeMo
|
||||
env:
|
||||
INSTALLER: ${{ matrix.installer }}
|
||||
NEMO_TAG: ${{ github.sha }}
|
||||
NEMO_REPO: ${{ github.server_url }}/${{ github.repository }}
|
||||
run: |
|
||||
if [[ "$INSTALLER" == "pip-install" ]]; then
|
||||
pip install --no-cache-dir -U pip
|
||||
pip install --no-cache-dir ".[all]"
|
||||
else
|
||||
export NEMO_TAG
|
||||
export NEMO_REPO
|
||||
export INSTALL_DIR=$(pwd)
|
||||
|
||||
pip install --no-cache-dir ".[all]"
|
||||
fi
|
||||
|
||||
- name: Check disk space after installation
|
||||
run: df -h
|
||||
|
||||
- name: Run import checks
|
||||
run: |
|
||||
# Run import checks
|
||||
for collection in "asr" "tts" "lightning" "core"; do
|
||||
python tests/core_ptl/check_imports.py --domain "$collection"
|
||||
done
|
||||
|
||||
test-installs-linux-amd:
|
||||
name: ubuntu-22.04-amd-py${{ matrix.python }}-${{ matrix.installer }}
|
||||
runs-on: ubuntu-22.04
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
python: ["3.10", "3.11", "3.12"]
|
||||
installer: ["pip-install", "nemo-install"]
|
||||
steps:
|
||||
- name: Checkout repo
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Check disk space before cleanup
|
||||
run: df -h
|
||||
|
||||
- name: Free up disk space
|
||||
run: |
|
||||
# Remove unnecessary packages and files on Ubuntu
|
||||
sudo apt-get clean
|
||||
sudo rm -rf /usr/local/lib/android || true
|
||||
sudo rm -rf /opt/ghc || true
|
||||
sudo rm -rf /usr/local/.ghcup || true
|
||||
sudo rm -rf /usr/share/dotnet || true
|
||||
sudo rm -rf /opt/az || true
|
||||
# Clear pip and npm caches
|
||||
pip cache purge || true
|
||||
sudo npm cache clean --force || true
|
||||
|
||||
- name: Check disk space after cleanup
|
||||
run: df -h
|
||||
|
||||
- name: Install Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: ${{ matrix.python }}
|
||||
|
||||
- name: Install NeMo
|
||||
env:
|
||||
INSTALLER: ${{ matrix.installer }}
|
||||
run: |
|
||||
if [ "$INSTALLER" = "pip-install" ]; then
|
||||
pip install --no-cache-dir --upgrade pip
|
||||
pip install --no-cache-dir ".[all]"
|
||||
else
|
||||
export INSTALL_DIR=$(pwd)
|
||||
pip install --no-cache-dir ".[all]"
|
||||
fi
|
||||
|
||||
- name: Check disk space after installation
|
||||
run: df -h
|
||||
|
||||
- name: Run import checks
|
||||
run: |
|
||||
# Run import checks
|
||||
for collection in "asr" "tts" "lightning" "core"; do
|
||||
python tests/core_ptl/check_imports.py --domain "$collection"
|
||||
done
|
||||
|
||||
test-asr-install-linux-amd:
|
||||
name: ubuntu-22.04-amd-py${{ matrix.python }}-asr
|
||||
runs-on: ubuntu-22.04
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
python: ["3.10", "3.11", "3.12"]
|
||||
steps:
|
||||
- name: Checkout repo
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Check disk space before cleanup
|
||||
run: df -h
|
||||
|
||||
- name: Free up disk space
|
||||
run: |
|
||||
# Remove unnecessary packages and files on Ubuntu
|
||||
sudo apt-get clean
|
||||
sudo rm -rf /usr/local/lib/android || true
|
||||
sudo rm -rf /opt/ghc || true
|
||||
sudo rm -rf /usr/local/.ghcup || true
|
||||
sudo rm -rf /usr/share/dotnet || true
|
||||
sudo rm -rf /opt/az || true
|
||||
# Clear pip and npm caches
|
||||
pip cache purge || true
|
||||
sudo npm cache clean --force || true
|
||||
|
||||
- name: Check disk space after cleanup
|
||||
run: df -h
|
||||
|
||||
- name: Install Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: ${{ matrix.python }}
|
||||
|
||||
- name: Install NeMo
|
||||
run: |
|
||||
pip install --no-cache-dir --upgrade pip
|
||||
pip install --no-cache-dir ".[asr]"
|
||||
|
||||
- name: Check disk space after installation
|
||||
run: df -h
|
||||
|
||||
- name: Run import checks
|
||||
run: |
|
||||
# Run import checks
|
||||
python tests/core_ptl/check_imports.py --domain asr
|
||||
|
||||
test-installs-linux-arm:
|
||||
name: ubuntu-22.04-arm-py${{ matrix.python }}-${{ matrix.installer }}
|
||||
runs-on: ubuntu-22.04-arm
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
python: ["3.10", "3.11", "3.12"]
|
||||
installer: ["pip-install", "nemo-install"]
|
||||
steps:
|
||||
- name: Checkout repo
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Check disk space before cleanup
|
||||
run: df -h
|
||||
|
||||
- name: Free up disk space
|
||||
run: |
|
||||
# Remove unnecessary packages and files on Ubuntu ARM
|
||||
sudo apt-get clean
|
||||
sudo rm -rf /usr/local/lib/android || true
|
||||
sudo rm -rf /opt/ghc || true
|
||||
sudo rm -rf /usr/local/.ghcup || true
|
||||
sudo rm -rf /usr/share/dotnet || true
|
||||
sudo rm -rf /opt/az || true
|
||||
# Clear pip and npm caches
|
||||
pip cache purge || true
|
||||
sudo npm cache clean --force || true
|
||||
|
||||
- name: Check disk space after cleanup
|
||||
run: df -h
|
||||
|
||||
- name: Install Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: ${{ matrix.python }}
|
||||
|
||||
- name: Install NeMo
|
||||
env:
|
||||
INSTALLER: ${{ matrix.installer }}
|
||||
run: |
|
||||
if [ "$INSTALLER" = "pip-install" ]; then
|
||||
pip install --no-cache-dir --upgrade pip
|
||||
pip install --no-cache-dir ".[all]"
|
||||
else
|
||||
export INSTALL_DIR=$(pwd)
|
||||
pip install --no-cache-dir ".[all]"
|
||||
fi
|
||||
|
||||
- name: Check disk space after installation
|
||||
run: df -h
|
||||
|
||||
- name: Run import checks
|
||||
run: |
|
||||
# Run import checks
|
||||
for collection in "asr" "tts" "lightning" "core"; do
|
||||
python tests/core_ptl/check_imports.py --domain "$collection"
|
||||
done
|
||||
|
||||
test-asr-installs-linux-arm:
|
||||
name: ubuntu-22.04-arm-py${{ matrix.python }}-asr
|
||||
runs-on: ubuntu-22.04-arm
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
python: ["3.10", "3.11", "3.12"]
|
||||
steps:
|
||||
- name: Checkout repo
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Check disk space before cleanup
|
||||
run: df -h
|
||||
|
||||
- name: Free up disk space
|
||||
run: |
|
||||
# Remove unnecessary packages and files on Ubuntu ARM
|
||||
sudo apt-get clean
|
||||
sudo rm -rf /usr/local/lib/android || true
|
||||
sudo rm -rf /opt/ghc || true
|
||||
sudo rm -rf /usr/local/.ghcup || true
|
||||
sudo rm -rf /usr/share/dotnet || true
|
||||
sudo rm -rf /opt/az || true
|
||||
# Clear pip and npm caches
|
||||
pip cache purge || true
|
||||
sudo npm cache clean --force || true
|
||||
|
||||
- name: Check disk space after cleanup
|
||||
run: df -h
|
||||
|
||||
- name: Install Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: ${{ matrix.python }}
|
||||
|
||||
- name: Install NeMo
|
||||
run: |
|
||||
pip install --no-cache-dir --upgrade pip
|
||||
pip install --no-cache-dir ".[asr]"
|
||||
|
||||
- name: Check disk space after installation
|
||||
run: df -h
|
||||
|
||||
- name: Run import checks
|
||||
run: |
|
||||
# Run import checks
|
||||
python tests/core_ptl/check_imports.py --domain asr
|
||||
@@ -0,0 +1,14 @@
|
||||
name: "Pull Request Labeler"
|
||||
on:
|
||||
- pull_request_target
|
||||
|
||||
jobs:
|
||||
triage:
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/labeler@v4
|
||||
with:
|
||||
repo-token: "${{ secrets.GITHUB_TOKEN }}"
|
||||
@@ -0,0 +1,62 @@
|
||||
# Regularly updates the CI container
|
||||
name: Megatron Tag Bump Bot
|
||||
on:
|
||||
workflow_dispatch:
|
||||
schedule:
|
||||
- cron: 0 0 * * *
|
||||
|
||||
jobs:
|
||||
get-release-branch-names:
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
mcore: ${{ steps.get-branch.outputs.mcore_release_branch }}
|
||||
nemo: ${{ steps.get-branch.outputs.nemo_release_branch }}
|
||||
steps:
|
||||
- name: Get release branch names
|
||||
id: get-branch
|
||||
run: |
|
||||
latest_branch=$(git ls-remote --heads https://github.com/NVIDIA/Megatron-LM.git 'refs/heads/core_r*' |
|
||||
grep -o 'core_r[0-9]\+\.[0-9]\+\.[0-9]\+' |
|
||||
sort -V |
|
||||
tail -n1)
|
||||
echo "mcore_release_branch=$latest_branch" >> $GITHUB_OUTPUT
|
||||
|
||||
latest_branch=$(git ls-remote --heads https://github.com/NVIDIA/NeMo.git 'refs/heads/r*' |
|
||||
grep -o 'r[0-9]\+\.[0-9]\+\.[0-9]\+' |
|
||||
sort -V |
|
||||
tail -n1)
|
||||
echo "nemo_release_branch=$latest_branch" >> $GITHUB_OUTPUT
|
||||
|
||||
bump-tags:
|
||||
needs: [get-release-branch-names]
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- nemo-target-branch: ${{ needs.get-release-branch-names.outputs.nemo }}
|
||||
mcore-target-branch: ${{ needs.get-release-branch-names.outputs.mcore }}
|
||||
- nemo-target-branch: main
|
||||
mcore-target-branch: main
|
||||
uses: ./.github/workflows/_bump_mcore_tag.yml
|
||||
with:
|
||||
nemo-target-branch: ${{ matrix.nemo-target-branch }}
|
||||
mcore-target-branch: ${{ matrix.mcore-target-branch }}
|
||||
secrets:
|
||||
PAT: ${{ secrets.PAT }}
|
||||
|
||||
notify:
|
||||
if: failure()
|
||||
runs-on: ubuntu-latest
|
||||
needs: [bump-tags]
|
||||
steps:
|
||||
- name: Notify
|
||||
env:
|
||||
SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
|
||||
SLACK_WEBHOOK_ADMIN: <!subteam^${{ secrets.SLACK_WEBHOOK_ADMIN }}>
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
GITHUB_REPOSITORY: ${{ github.repository }}
|
||||
run: |
|
||||
curl -X POST \
|
||||
-H 'Content-type: application/json' \
|
||||
--data "{\"text\":\":robot_joy: <https://github.com/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}|Mcore-bump-bot workflow> failed. Please fix manually.\n\ncc ${SLACK_WEBHOOK_ADMIN}\"}" \
|
||||
$SLACK_WEBHOOK
|
||||
@@ -0,0 +1,54 @@
|
||||
name: ~shut down a single VM
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
vm:
|
||||
type: string
|
||||
description: Name of VM
|
||||
required: true
|
||||
n_gpus:
|
||||
type: string
|
||||
description: Number of GPUs this VM has
|
||||
required: true
|
||||
|
||||
jobs:
|
||||
check-status-and-maybe-shutdown:
|
||||
environment: main
|
||||
runs-on: ${{ inputs.vm }}
|
||||
outputs:
|
||||
status: ${{ steps.status.outputs.main }}
|
||||
steps:
|
||||
- name: Check status
|
||||
id: status
|
||||
run: |
|
||||
docker run --rm --runtime=nvidia --gpus ${{ inputs.n_gpus }} ubuntu nvidia-smi
|
||||
|
||||
NUM_GPUS=$(nvidia-smi --query-gpu=name --format=csv,noheader | wc -l)
|
||||
|
||||
if [[ $NUM_GPUS -ne ${{ inputs.n_gpus }} ]]; then
|
||||
echo "Issues with GPU detected, will take this runner offline."
|
||||
echo "main=degraded" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "main=healthy" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
- name: Send Slack message & Disconnect runner from GitHub
|
||||
if: ${{ steps.status.outputs.main == 'degraded' || failure() }}
|
||||
run: |
|
||||
MESSAGE='{
|
||||
"blocks": [
|
||||
{
|
||||
"type": "section",
|
||||
"text": {
|
||||
"type": "mrkdwn",
|
||||
"text": ":alert: VM bot 🤖: Hey <!subteam^${{ secrets.SLACK_WEBHOOK_ADMIN }}>: VM `${{ inputs.vm }}` is having not the best day of their life, maybe bring them an apple or so."
|
||||
}
|
||||
}
|
||||
]
|
||||
}'
|
||||
|
||||
curl -X POST -H "Content-type: application/json" --data "$MESSAGE" ${{ secrets.SLACK_WEBHOOK }}
|
||||
|
||||
cd /home/azureuser/actions-runner
|
||||
echo ${{ secrets.VM_KEY }} | sudo -S ./svc.sh stop
|
||||
@@ -0,0 +1,54 @@
|
||||
# Regularly updates the CI container
|
||||
name: Reboots VMs in a controlled way
|
||||
on:
|
||||
schedule:
|
||||
- cron: 0/15 * * * *
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
pre-flight:
|
||||
runs-on: ubuntu-latest
|
||||
if: github.repository_owner == 'NVIDIA'
|
||||
outputs:
|
||||
list-of-vms: ${{ steps.main.outputs.main }}
|
||||
environment: main
|
||||
steps:
|
||||
- name: Get list of VMs
|
||||
id: main
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.PAT }}
|
||||
run: |
|
||||
RUNNERS=$(curl -L \
|
||||
-H "Accept: application/vnd.github+json" \
|
||||
-H "Authorization: Bearer $GITHUB_TOKEN" \
|
||||
-H "X-GitHub-Api-Version: 2022-11-28" \
|
||||
https://api.github.com/repos/NVIDIA/NeMo/actions/runners)
|
||||
|
||||
MATRIX=$(echo $RUNNERS \
|
||||
| jq -c '[
|
||||
.runners[]
|
||||
| select(.status == "online")
|
||||
| select(.name | contains("cpu") | not)
|
||||
| {
|
||||
"vm": .name,
|
||||
"n_gpus": [
|
||||
.labels[]
|
||||
| select(.name | endswith("gpu")) | .name
|
||||
][0][:1]
|
||||
}
|
||||
]
|
||||
'
|
||||
)
|
||||
echo main=$MATRIX | tee -a "$GITHUB_OUTPUT"
|
||||
|
||||
maintenance:
|
||||
needs: pre-flight
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include: ${{ fromJSON(needs.pre-flight.outputs.list-of-vms )}}
|
||||
uses: ./.github/workflows/monitor-single-vm.yml
|
||||
with:
|
||||
vm: ${{ matrix.vm }}
|
||||
n_gpus: ${{ matrix.n_gpus }}
|
||||
secrets: inherit # pragma: allowlist secret
|
||||
@@ -0,0 +1,123 @@
|
||||
# Copyright (c) 2026, NVIDIA CORPORATION.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
name: Release docs
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
dry-run:
|
||||
description: Whether to run the workflow in dry-run mode
|
||||
required: true
|
||||
type: boolean
|
||||
default: true
|
||||
publish-as-latest:
|
||||
description: Publish as Latest stable version.
|
||||
required: false
|
||||
type: boolean
|
||||
default: true
|
||||
docs-version-override:
|
||||
description: Docs version if commit is not tagged
|
||||
required: false
|
||||
type: string
|
||||
default: ""
|
||||
update-version-picker:
|
||||
description: Update version picker.
|
||||
required: false
|
||||
type: boolean
|
||||
default: true
|
||||
notify-emails:
|
||||
description: Email addresses to send the notification to. Format as "me@me.com,you@you.com".
|
||||
required: false
|
||||
type: string
|
||||
github-ref:
|
||||
description: Github ref to checkout
|
||||
required: false
|
||||
type: string
|
||||
default: ""
|
||||
workflow_call:
|
||||
inputs:
|
||||
dry-run:
|
||||
description: Whether to run the workflow in dry-run mode
|
||||
required: false
|
||||
type: boolean
|
||||
default: true
|
||||
publish-as-latest:
|
||||
description: Publish as Latest stable version.
|
||||
required: false
|
||||
type: boolean
|
||||
default: true
|
||||
docs-version-override:
|
||||
description: Docs version if commit is not tagged
|
||||
required: false
|
||||
type: string
|
||||
default: ""
|
||||
update-version-picker:
|
||||
description: Update version picker.
|
||||
required: false
|
||||
type: boolean
|
||||
default: true
|
||||
notify-emails:
|
||||
description: Email addresses to send the notification to. Format as "me@me.com,you@you.com".
|
||||
required: false
|
||||
type: string
|
||||
github-ref:
|
||||
description: Github ref to checkout
|
||||
required: false
|
||||
type: string
|
||||
default: ""
|
||||
|
||||
jobs:
|
||||
build-docs:
|
||||
uses: NVIDIA-NeMo/FW-CI-templates/.github/workflows/_build_docs.yml@v0.83.0
|
||||
with:
|
||||
ref: ${{ inputs.github-ref }}
|
||||
docs-directory: docs/source
|
||||
sync-all: true
|
||||
no-extras: "--no-extra cu12 --no-extra compiled --no-extra compiled-a100"
|
||||
|
||||
publish-docs:
|
||||
runs-on: ubuntu-latest
|
||||
needs: [build-docs]
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
repository: NVIDIA-NeMo/FW-CI-templates
|
||||
ref: v0.74.0
|
||||
path: FW-CI-templates
|
||||
|
||||
- uses: ./FW-CI-templates/.github/actions/publish-docs
|
||||
# This workflow runs either on main, or on a version tag. Any other git ref will lead
|
||||
# to an error.
|
||||
# If its on main, it will publish to "latest" directory in Akamai.
|
||||
# If its on a versioned tag, it will extract the version number from the tag (strip `v` prefix)
|
||||
# and publish to the versioned directory in Akamai.
|
||||
with:
|
||||
dry-run: ${{ inputs.dry-run }}
|
||||
artifacts-name: docs-html
|
||||
artifacts-path: _build/html
|
||||
emails-csv: ${{ inputs.notify-emails && format('{0},{1}', vars.docs_release_emails, inputs.notify-emails) || vars.docs_release_emails }}
|
||||
overwrite-latest-on-tag: ${{ inputs.publish-as-latest }}
|
||||
docs-version-override: ${{ inputs.docs-version-override }}
|
||||
update-version-picker: ${{ inputs.update-version-picker }}
|
||||
run-on-version-tag-only: ${{ github.ref_name != 'main' }}
|
||||
request-name: nemo-speech-publish-docs-${{ github.run_id }}
|
||||
aws-region: ${{ vars.DOCS_AWS_REGION }}
|
||||
aws-role-to-assume: ${{ secrets.AWS_ASSUME_ROLE_ARN }}
|
||||
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
||||
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
||||
akamai-host: ${{ secrets.AKAMAI_HOST }}
|
||||
akamai-client-token: ${{ secrets.AKAMAI_CLIENT_TOKEN }}
|
||||
akamai-client-secret: ${{ secrets.AKAMAI_CLIENT_SECRET }}
|
||||
akamai-access-token: ${{ secrets.AKAMAI_ACCESS_TOKEN }}
|
||||
s3-target-root: ${{ secrets.S3_BUCKET_NAME }}
|
||||
s3-target-path: nemo/speech
|
||||
@@ -0,0 +1,80 @@
|
||||
name: "Code freeze"
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
type_of_release:
|
||||
type: choice
|
||||
description: Type of release
|
||||
options:
|
||||
- major
|
||||
- minor
|
||||
freeze-commit:
|
||||
type: string
|
||||
description: Commit SHA to use for cut-off
|
||||
required: false
|
||||
default: main
|
||||
mcore_version:
|
||||
description: "Version of MCore to use (must be a valid git ref)"
|
||||
required: true
|
||||
type: string
|
||||
dry-run:
|
||||
type: boolean
|
||||
description: Dry-run of code-freeze
|
||||
required: false
|
||||
default: true
|
||||
|
||||
jobs:
|
||||
code-freeze:
|
||||
uses: NVIDIA-NeMo/FW-CI-templates/.github/workflows/_code_freeze.yml@v0.86.0
|
||||
with:
|
||||
library-name: NeMo-Toolkit
|
||||
python-package: nemo
|
||||
release-type: ${{ inputs.type_of_release }}
|
||||
freeze-commit: ${{ inputs.freeze-commit }}
|
||||
dry-run: ${{ inputs.dry-run }}
|
||||
use-pat: true
|
||||
secrets:
|
||||
SLACK_WEBHOOK: ${{ secrets.SLACK_RELEASE_ENDPOINT }}
|
||||
SLACK_WEBHOOK_ADMIN: ${{ secrets.SLACK_WEBHOOK_ADMIN }}
|
||||
PAT: ${{ secrets.PAT }}
|
||||
|
||||
freeze-tags:
|
||||
runs-on: ubuntu-latest
|
||||
needs: [code-freeze]
|
||||
environment: main
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
path: ${{ github.run_id }}
|
||||
token: ${{ secrets.PAT }}
|
||||
fetch-depth: 0
|
||||
fetch-tags: true
|
||||
ref: ${{ inputs.dry-run == true && inputs.freeze-commit || needs.code-freeze.outputs.release-branch }}
|
||||
|
||||
- name: Pin branch name in Notebooks
|
||||
run: |
|
||||
cd ${{ github.run_id }}
|
||||
find tutorials -type f -name "*.ipynb" -exec sed -i "s/BRANCH = 'main'/BRANCH = '${{ needs.code-freeze.outputs.release-branch }}'/g" {} +
|
||||
|
||||
- name: Show status
|
||||
run: |
|
||||
cd ${{ github.run_id }}
|
||||
git status
|
||||
|
||||
- name: Create PR
|
||||
uses: peter-evans/create-pull-request@v6
|
||||
id: create-pull-request
|
||||
if: ${{ inputs.dry-run != true }}
|
||||
with:
|
||||
path: ${{ github.run_id }}
|
||||
base: ${{ needs.code-freeze.outputs.release-branch }}
|
||||
branch: ci/freeze-tags-${{ needs.code-freeze.outputs.release-branch }}
|
||||
title: "Freeze tags in in `${{ needs.code-freeze.outputs.release-branch }}`"
|
||||
body: |
|
||||
🚀 PR to freeze tags in `${{ needs.code-freeze.outputs.release-branch }}`.
|
||||
|
||||
commit-message: "[🤠]: Howdy folks, let's release NeMo `${{ needs.code-freeze.outputs.release-branch }}` !"
|
||||
signoff: true
|
||||
assignees: okoenig
|
||||
@@ -0,0 +1,29 @@
|
||||
# Copyright (c) 2026, NVIDIA CORPORATION.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
name: Release Nightly Docs
|
||||
|
||||
on:
|
||||
schedule:
|
||||
- cron: "0 10 * * *"
|
||||
|
||||
jobs:
|
||||
call-release-docs:
|
||||
uses: ./.github/workflows/release-docs.yml
|
||||
with:
|
||||
dry-run: false
|
||||
publish-as-latest: false
|
||||
docs-version-override: "nightly"
|
||||
update-version-picker: false
|
||||
secrets: inherit
|
||||
@@ -0,0 +1,127 @@
|
||||
# Copyright (c) 2020-2026, NVIDIA CORPORATION.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
name: "Build, validate, and release Neural Modules"
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
- "r**"
|
||||
- "pull-request/**"
|
||||
- "deploy-release/*"
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
release-ref:
|
||||
description: Ref (SHA or branch name) to release
|
||||
required: true
|
||||
type: string
|
||||
version-bump-branch:
|
||||
description: Branch for version bump
|
||||
required: true
|
||||
type: string
|
||||
dry-run:
|
||||
description: Compute the release but do not publish wheel, GH release, or docs.
|
||||
required: true
|
||||
default: true
|
||||
type: boolean
|
||||
create-gh-release:
|
||||
description: Create a GitHub release
|
||||
required: true
|
||||
default: true
|
||||
type: boolean
|
||||
generate-changelog:
|
||||
description: Generate changelog
|
||||
required: false
|
||||
default: true
|
||||
type: boolean
|
||||
publish-docs:
|
||||
description: Publish docs
|
||||
required: false
|
||||
default: true
|
||||
type: boolean
|
||||
gh-release-from-tag:
|
||||
description: Tag of previous release for changelog builder
|
||||
required: false
|
||||
type: string
|
||||
default: ""
|
||||
|
||||
defaults:
|
||||
run:
|
||||
shell: bash -x -e -u -o pipefail {0}
|
||||
|
||||
permissions:
|
||||
id-token: write
|
||||
contents: write
|
||||
pull-requests: write
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}-${{ github.event_name }}
|
||||
cancel-in-progress: ${{ github.event_name == 'push' }}
|
||||
|
||||
jobs:
|
||||
pre-flight:
|
||||
if: github.event_name != 'workflow_dispatch'
|
||||
uses: NVIDIA-NeMo/FW-CI-templates/.github/workflows/_cicd_preflight.yml@v0.94.1
|
||||
|
||||
release:
|
||||
needs: [pre-flight]
|
||||
if: |
|
||||
!cancelled() && !failure()
|
||||
&& !(needs.pre-flight.outputs.docs_only == 'true'
|
||||
|| needs.pre-flight.outputs.is_deployment_workflow == 'true')
|
||||
uses: NVIDIA-NeMo/FW-CI-templates/.github/workflows/_release_library.yml@v1.4.3
|
||||
with:
|
||||
release-ref: ${{ inputs.release-ref || github.sha }}
|
||||
python-package: nemo
|
||||
python-version: "3.10"
|
||||
library-name: Neural Modules
|
||||
validate-only: ${{ github.event_name != 'workflow_dispatch' }}
|
||||
dry-run: ${{ inputs.dry-run || false }}
|
||||
wheel-content-ignore: "W002,W004,W005,W009"
|
||||
skip-check-manifest: true
|
||||
version-bump-branch: ${{ inputs.version-bump-branch || github.ref_name }}
|
||||
create-gh-release: ${{ inputs.create-gh-release || true }}
|
||||
app-id: ${{ vars.BOT_ID }}
|
||||
gh-release-use-changelog-builder: ${{ inputs.generate-changelog || false }}
|
||||
publish-docs: ${{ inputs.publish-docs || true }}
|
||||
docs-target-path: nemo
|
||||
docs-directory: docs/source
|
||||
docs-requirements-file: requirements/requirements_docs.txt
|
||||
docs-sync-all: true
|
||||
docs-no-extras: "--no-extra cu12 --no-extra compiled --no-extra compiled-a100"
|
||||
publish-as-latest: true
|
||||
run-on-version-tag-only: ${{ github.ref_name != 'main' }}
|
||||
gh-release-from-tag: ${{ inputs.gh-release-from-tag || '' }}
|
||||
restrict-to-admins: true
|
||||
secrets: inherit # pragma: allowlist secret
|
||||
|
||||
release-summary:
|
||||
needs: [pre-flight, release]
|
||||
if: ${{ !cancelled() }}
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Result
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: |
|
||||
FAILED_JOBS=$(gh run view $GITHUB_RUN_ID --repo ${{ github.repository }} --json jobs --jq '[.jobs[] | select(.conclusion == "failure" or .conclusion == "timed_out" or .conclusion == "action_required")] | length')
|
||||
|
||||
if [ "${FAILED_JOBS:-0}" -eq 0 ]; then
|
||||
echo "✅ All previous jobs completed successfully"
|
||||
exit 0
|
||||
else
|
||||
echo "❌ Found $FAILED_JOBS failed job(s)"
|
||||
gh run view $GITHUB_RUN_ID --repo ${{ github.repository }} --json jobs --jq '.jobs[] | select(.conclusion == "failure" or .conclusion == "timed_out" or .conclusion == "action_required") | .name'
|
||||
exit 1
|
||||
fi
|
||||
@@ -0,0 +1,43 @@
|
||||
# Copyright (c) 2020-2021, NVIDIA CORPORATION.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
name: Secrets detector
|
||||
|
||||
on:
|
||||
pull_request_target:
|
||||
branches:
|
||||
- 'main'
|
||||
|
||||
jobs:
|
||||
main:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
fetch-depth: 0
|
||||
token: ${{ secrets.NEMO_REFORMAT_TOKEN }}
|
||||
|
||||
- name: Install secrets detector
|
||||
run: pip install detect-secrets
|
||||
|
||||
- name: Run on change-set
|
||||
run: |
|
||||
git diff --name-only --diff-filter=d --merge-base origin/main -z | xargs -0 detect-secrets-hook --disable-plugin HexHighEntropyString --baseline .secrets.baseline
|
||||
|
||||
- uses: EndBug/add-and-commit@v9
|
||||
# Commit changes. Nothing is committed if no changes.
|
||||
if: always()
|
||||
with:
|
||||
message: Update baseline
|
||||
commit: --signoff
|
||||
Reference in New Issue
Block a user