chore: import upstream snapshot with attribution
Codestyle Check / Lint (push) Has been cancelled
Codestyle Check / Check bypass (push) Has been cancelled
Pipelines-Test / Pipelines-Test (push) Has been cancelled

This commit is contained in:
wehub-resource-sync
2026-07-13 13:37:14 +08:00
commit 2aaeece67c
5662 changed files with 1035705 additions and 0 deletions
+89
View File
@@ -0,0 +1,89 @@
**简体中文**🀄 | [English🌎](./CODE_OF_CONDUCT_en.md)
# 贡献者公约
## 我们的承诺
身为社区成员、贡献者和领袖,我们承诺使社区参与者不受骚扰,无论其年龄、体型、可见或不可见的缺陷、族裔、性征、性别认同和表达、经验水平、教育程度、社会与经济地位、国籍、相貌、种族、种姓、肤色、宗教信仰、性倾向或性取向如何。
我们承诺以有助于建立开放、友善、多样化、包容、健康社区的方式行事和互动。
## 我们的准则
有助于为我们的社区创造积极环境的行为例子包括但不限于:
* 表现出对他人的同情和善意
* 尊重不同的主张、观点和感受
* 提出和大方接受建设性意见
* 承担责任并向受我们错误影响的人道歉
* 注重社区共同诉求,而非个人得失
不当行为例子包括:
* 使用情色化的语言或图像,及性引诱或挑逗
* 嘲弄、侮辱或诋毁性评论,以及人身或政治攻击
* 公开或私下的骚扰行为
* 未经他人明确许可,公布他人的私人信息,如物理或电子邮件地址
* 其他有理由认定为违反职业操守的不当行为
## 责任和权力
社区领袖有责任解释和落实我们所认可的行为准则,并妥善公正地对他们认为不当、威胁、冒犯或有害的任何行为采取纠正措施。
社区领导有权力和责任删除、编辑或拒绝或拒绝与本行为准则不相符的评论(comment)、提交(commits)、代码、维基(wiki)编辑、议题(issues)或其他贡献,并在适当时机知采取措施的理由。
## 适用范围
本行为准则适用于所有社区场合,也适用于在公共场所代表社区时的个人。
代表社区的情形包括使用官方电子邮件地址、通过官方社交媒体帐户发帖或在线上或线下活动中担任指定代表。
## 监督
辱骂、骚扰或其他不可接受的行为可通过 paddlenlp@baidu.com 向负责监督的社区领袖报告。
所有投诉都将得到及时和公平的审查和调查。
所有社区领袖都有义务尊重任何事件报告者的隐私和安全。
## 处理方针
社区领袖将遵循下列社区处理方针来明确他们所认定违反本行为准则的行为的处理方式:
### 1. 纠正
**社区影响**:使用不恰当的语言或其他在社区中被认定为不符合职业道德或不受欢迎的行为。
**处理意见**:由社区领袖发出非公开的书面警告,明确说明违规行为的性质,并解释举止如何不妥。或将要求公开道歉。
### 2. 警告
**社区影响**:单个或一系列违规行为。
**处理意见**:警告并对连续性行为进行处理。在指定时间内,不得与相关人员互动,包括主动与行为准则执行者互动。这包括避免在社区场所和外部渠道中的互动。违反这些条款可能会导致临时或永久封禁。
### 3. 临时封禁
**社区影响**: 严重违反社区准则,包括持续的不当行为。
**处理意见**: 在指定时间内,暂时禁止与社区进行任何形式的互动或公开交流。在此期间,不得与相关人员进行公开或私下互动,包括主动与行为准则执行者互动。违反这些条款可能会导致永久封禁。
### 4. 永久封禁
**社区影响**:行为模式表现出违反社区准则,包括持续的不当行为、骚扰个人或攻击或贬低某个类别的个体。
**处理意见**:永久禁止在社区内进行任何形式的公开互动。
## 参见
本行为准则改编自 [Contributor Covenant][homepage] 2.1 版, 参见 [https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1]。
社区处理方针灵感来源于 [Mozilla's code of conduct enforcement ladder][Mozilla CoC]。
有关本行为准则的常见问题的答案,参见 [https://www.contributor-covenant.org/faq][FAQ]。
其他语言翻译参见 [https://www.contributor-covenant.org/translations][translations]。
[homepage]: https://www.contributor-covenant.org
[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
[Mozilla CoC]: https://github.com/mozilla/diversity
[FAQ]: https://www.contributor-covenant.org/faq
[translations]: https://www.contributor-covenant.org/translations
+134
View File
@@ -0,0 +1,134 @@
[简体中文🀄](./CODE_OF_CONDUCT.md) | **English**🌎
# Contributor Covenant Code of Conduct
## Our Pledge
We as members, contributors, and leaders pledge to make participation in our
community a harassment-free experience for everyone, regardless of age, body
size, visible or invisible disability, ethnicity, sex characteristics, gender
identity and expression, level of experience, education, socio-economic status,
nationality, personal appearance, race, caste, color, religion, or sexual
identity and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.
## Our Standards
Examples of behavior that contributes to a positive environment for our
community include:
* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes,
and learning from the experience
* Focusing on what is best not just for us as individuals, but for the overall
community
Examples of unacceptable behavior include:
* The use of sexualized language or imagery, and sexual attention or advances of
any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email address,
without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Enforcement Responsibilities
Community leaders are responsible for clarifying and enforcing our standards of
acceptable behavior and will take appropriate and fair corrective action in
response to any behavior that they deem inappropriate, threatening, offensive,
or harmful.
Community leaders have the right and responsibility to remove, edit, or reject
comments, commits, code, wiki edits, issues, and other contributions that are
not aligned to this Code of Conduct, and will communicate reasons for moderation
decisions when appropriate.
## Scope
This Code of Conduct applies within all community spaces, and also applies when
an individual is officially representing the community in public spaces.
Examples of representing our community include using an official e-mail address,
posting via an official social media account, or acting as an appointed
representative at an online or offline event.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement at
paddlenlp@baidu.com.
All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the
reporter of any incident.
## Enforcement Guidelines
Community leaders will follow these Community Impact Guidelines in determining
the consequences for any action they deem in violation of this Code of Conduct:
### 1. Correction
**Community Impact**: Use of inappropriate language or other behavior deemed
unprofessional or unwelcome in the community.
**Consequence**: A private, written warning from community leaders, providing
clarity around the nature of the violation and an explanation of why the
behavior was inappropriate. A public apology may be requested.
### 2. Warning
**Community Impact**: A violation through a single incident or series of
actions.
**Consequence**: A warning with consequences for continued behavior. No
interaction with the people involved, including unsolicited interaction with
those enforcing the Code of Conduct, for a specified period of time. This
includes avoiding interactions in community spaces as well as external channels
like social media. Violating these terms may lead to a temporary or permanent
ban.
### 3. Temporary Ban
**Community Impact**: A serious violation of community standards, including
sustained inappropriate behavior.
**Consequence**: A temporary ban from any sort of interaction or public
communication with the community for a specified period of time. No public or
private interaction with the people involved, including unsolicited interaction
with those enforcing the Code of Conduct, is allowed during this period.
Violating these terms may lead to a permanent ban.
### 4. Permanent Ban
**Community Impact**: Demonstrating a pattern of violation of community
standards, including sustained inappropriate behavior, harassment of an
individual, or aggression toward or disparagement of classes of individuals.
**Consequence**: A permanent ban from any sort of public interaction within the
community.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
version 2.1, available at
[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
Community Impact Guidelines were inspired by
[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
For answers to common questions about this code of conduct, see the FAQ at
[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
[https://www.contributor-covenant.org/translations][translations].
[homepage]: https://www.contributor-covenant.org
[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
[Mozilla CoC]: https://github.com/mozilla/diversity
[FAQ]: https://www.contributor-covenant.org/faq
[translations]: https://www.contributor-covenant.org/translations
+154
View File
@@ -0,0 +1,154 @@
[简体中文🀄](../CONTRIBUTING.md) | **English**🌎
# Contributing to PaddleNLP
We highly welcome and value your contributions to `PaddleNLP`. The first step to start your contribution is to sign the [PaddlePaddle Contributor License Agreement](https://cla-assistant.io/PaddlePaddle/PaddleNLP).
This document explains our workflow and work style:
## Finding out what to work on Workflow
## Development Workflow
PaddleNLP uses the [Git branching model](http://nvie.com/posts/a-successful-git-branching-model/). The following steps guide usual contributions.
#### 1. Fork
Our development community has been growing fastly; it doesn't make sense for everyone to write into the official repo. So, please file Pull Requests from your fork. To make a fork, just head over to the GitHub page and click the ["Fork" button](https://help.github.com/articles/fork-a-repo/).
#### 2. Clone
To make a copy of your fork to your local computers, please run
```bash
git clone https://github.com/<your-github-account>/PaddleNLP
cd PaddleNLP
```
#### 3. Create the local feature branch
For daily works like adding a new feature or fixing a bug, please open your feature branch before coding:
```bash
git checkout -b my-cool-feature
```
#### 4. Set up the development environment
Before you start coding, you need to setup the development environment. We highly recommend doing all your development in a virtual environment such as
[venv](https://docs.python.org/3/library/venv.html) or [conda](https://docs.conda.io/en/latest/). After you setup and activated your virtual environment,
run the following command:
```bash
make install
```
This will setup all the dependencies of `PaddleNLP` as well as the [`pre-commit`](http://pre-commit.com/) tool.
If you are working on the `examples` or `applications` module and require importing from `PaddleNLP`, make sure you install `PaddleNLP` in editable mode.
If `PaddleNLP` is already installed in the virtual environment, remove it with `pip uninstall paddlenlp` before reinstalling it in editable mode with
`pip install -e .`
#### 5. Develop
As you develop your new exciting feature, keep in mind that it should be covered by unit tests. All of our unit tests can be found under the `tests` directory.
You can either modify existing unit test to cover the new feature, or create a new test from scratch.
As you finish up the your code, you should make sure the test suite passes. You can run the tests impacted by your changes like this:
```bash
pytest tests/<test_to_run>.py
```
#### 6. Commit
We utilizes [`pre-commit`](http://pre-commit.com/) (with [black](https://black.readthedocs.io/en/stable/), [isort](https://pycqa.github.io/isort/) and
[flake8](https://flake8.pycqa.org/en/latest/) under the hood) to check the style of code and documentation in every commit. When you run run `git commit`, you will see
something like the following:
```
➜ (my-virtual-env) git commit -m "commiting my cool feature"
black....................................................................Passed
isort....................................................................Passed
flake8...................................................................Passed
check for merge conflicts................................................Passed
check for broken symlinks............................(no files to check)Skipped
detect private key.......................................................Passed
fix end of files.....................................(no files to check)Skipped
trim trailing whitespace.............................(no files to check)Skipped
CRLF end-lines checker...............................(no files to check)Skipped
CRLF end-lines remover...............................(no files to check)Skipped
No-tabs checker......................................(no files to check)Skipped
Tabs remover.........................................(no files to check)Skipped
copyright_checker........................................................Passed
```
But most of the time things don't go so smoothly. When your code or documentation doesn't meet the standard, the `pre-commit` check will fail.
```
➜ (my-virtual-env) git commit -m "commiting my cool feature"
black....................................................................Passed
isort....................................................................Failed
- hook id: isort
- files were modified by this hook
Fixing examples/information_extraction/waybill_ie/run_ernie_crf.py
flake8...................................................................Passed
check for merge conflicts................................................Passed
check for broken symlinks............................(no files to check)Skipped
detect private key.......................................................Passed
fix end of files.....................................(no files to check)Skipped
trim trailing whitespace.............................(no files to check)Skipped
CRLF end-lines checker...............................(no files to check)Skipped
CRLF end-lines remover...............................(no files to check)Skipped
No-tabs checker......................................(no files to check)Skipped
Tabs remover.........................................(no files to check)Skipped
copyright_checker........................................................Passed
```
But **don't panic**!
Our tooling will fix most of the style errors automatically. Some errors will need to be addressed manually. Fortunately, the error messages are straight forward and
the errors are usually simple to fix. After addressing the errors, you can run `git add <files>` and `git commit` again, which will trigger `pre-commit` again.
Once the `pre-commit` checks pass, you are ready to push the code.
[Google][http://google.com/] or [StackOverflow](https://stackoverflow.com/) are great tools to help you understand the code style errors.
Don't worry if you still can't figure it out. You can commit with `git commit -m "style error" --no-verify` and we are happy to help you once you create a Pull Request.
#### 7. Keep pulling
An experienced Git user pulls from the official repo often -- daily or even hourly, so they notice conflicts with others work early, and it's easier to resolve smaller conflicts.
```bash
git remote add upstream https://github.com/PaddlePaddle/PaddleNLP
git pull upstream develop
```
#### 8. Push and file a pull request
You can "push" your local work into your forked repo:
```bash
git push origin my-cool-stuff
```
The push allows you to create a pull request, requesting owners of this [official repo](https://github.com/PaddlePaddle/PaddleNLP) to pull your change into the official one.
To create a pull request, please follow [these steps](https://help.github.com/articles/creating-a-pull-request/).
#### 9. Delete local and remote branches
To keep your local workspace and your fork clean, you might want to remove merged branches:
```bash
git push origin my-cool-stuff
git checkout develop
git pull upstream develop
git branch -d my-cool-stuff
```
## Code Review
- Please feel free to ping your reviewers by @-mentioning the in the Pull Request. Please do this after your pull request passes the CI.
- Please answer reviewers' every comment. If you are to follow the comment, please write "Done"; Otherwise, please start a discussion under the comment.
- If you don't want your reviewers to get overwhelmed by email notifications, you might reply their comments by [in a batch](https://help.github.com/articles/reviewing-proposed-changes-in-a-pull-request/).
+23
View File
@@ -0,0 +1,23 @@
name: 🐛 Ask Question
description: 请描述您使用PaddleNLP时遇到的问题
title: "[Question]: "
labels:
- question
body:
- type: markdown
attributes:
value: >
#### 你可以在这里提出一个使用/咨询问题,提问之前请确保:
- 1)已经百度/谷歌搜索过你的问题,但是没有找到解答;
- 2)已经在官网查询过[API文档](https://www.paddlepaddle.org.cn/documentation/docs/zh/api/index_cn.html)与[FAQ](https://www.paddlepaddle.org.cn/documentation/docs/zh/faq/index_cn.html),但是没有找到解答;
- 3)已经在[历史issue](https://github.com/PaddlePaddle/Paddle/issues)中搜索过,没有找到同类issue或issue未被解答。
- type: textarea
id: question
attributes:
label: 请提出你的问题
validations:
required: true
+45
View File
@@ -0,0 +1,45 @@
name: 🐛 Bug Report
description: PaddleNLP问题反馈
title: "[Bug]: "
labels: bug
body:
- type: textarea
id: environment
attributes:
label: 软件环境
description: |
请使用以下命令给出您本地Paddle相关包信息
```sh
pip list | grep paddle
```
value: |
- paddlepaddle:
- paddlepaddle-gpu:
- paddlenlp:
render: Markdown
validations:
required: true
- type: checkboxes
id: dumplicated-problem
attributes:
label: 重复问题
description: 是否已在issues中搜索相关问题
options:
- label: I have searched the existing issues
required: true
- type: textarea
id: descripton
attributes:
label: 错误描述
description: 给出错误详细描述,以便能够更好的追踪相关问题
render: Markdown
validations:
required: true
- type: textarea
id: mvp-code
attributes:
label: 稳定复现步骤 & 代码
description: 请给出稳定复现该问题的步骤 & 代码,以便相关人员能够快速定位到具体问题。
validations:
required: true
+32
View File
@@ -0,0 +1,32 @@
name: 🐛 Docs Report
description: PaddleNLP文档反馈
title: "[Docs]: "
labels:
- documentation
body:
- type: textarea
id: environment
attributes:
label: 软件环境
description: |
请使用以下命令给出您本地Paddle相关包信息
```sh
pip list | grep paddle
```
value: |
- paddlepaddle:
- paddlepaddle-gpu:
- paddlenlp:
render: Markdown
validations:
required: true
- type: textarea
id: description
attributes:
label: 详细描述
description: 请详细描述您想要反馈的具体问题
render: Markdown
validations:
required: true
@@ -0,0 +1,30 @@
name: "\U0001F680 Feature request"
description: 请详细描述您所需功能
labels: [ "feature" ]
body:
- type: textarea
id: feature-request
validations:
required: true
attributes:
label: Feature request
description: |
对特性提案的清晰而简明的描述。如果论文和代码存在,请提供链接。
- type: textarea
id: motivation
validations:
required: true
attributes:
label: Motivation
description: |
请概述这项建议的动机。您的特性要求与问题有关吗?
- type: textarea
id: contribution
validations:
required: true
attributes:
label: Your contribution
description: |
Is there any way that you could help, e.g. by submitting a PR? Make sure to read the CONTRIBUTING.MD [readme](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md)
+28
View File
@@ -0,0 +1,28 @@
name: "\U0001F31F 添加新模型"
description: 请为新模型提交一份说明
labels: [ "New model" ]
body:
- type: textarea
id: description-request
validations:
required: true
attributes:
label: 简要描述
description: |
请简要描述模型的类型、解决的问题等。
- type: checkboxes
id: information-tasks
attributes:
label: 是否已开源
options:
- label: 已开源
- label: 未开源
- type: textarea
id: additional-info
attributes:
label: 模型详细信息
description: |
请给出新模型相关信息,如论文地址、现存代码地址等。
+23
View File
@@ -0,0 +1,23 @@
name: 🧩 其他 Others
description: 提出其他问题。
labels: [others]
body:
- type: markdown
attributes:
value: >
#### 你可以在这里提出任何前面几类模板不适用的问题,包括但不限于:优化性建议、框架使用体验反馈、版本兼容性问题、报错信息不清楚等。
- type: textarea
id: others
attributes:
label: 问题描述
validations:
required: true
- type: markdown
attributes:
value: >
感谢你的贡献 🎉!
+23
View File
@@ -0,0 +1,23 @@
<!-- Demo: https://github.com/PaddlePaddle/PaddleNLP/pull/26 -->
#### Before submitting
- [ ] Lint code. If there are lint issues, please format the code first.
```shell
# Install and register `pre-commit` in the project folder
pip install pre-commit && pre-commit install
# Process previous code files separately
pre-commit run --file XXXX.py
```
- [ ] Add test cases into `tests` folder. If there are codecov issues, please add tests cases first.
### PR types
<!-- One of [ New features | Bug fixes | Function optimization | Performance optimization | Breaking changes | Others ] -->
### PR changes
<!-- One of [ Models | APIs | Docs | Others ] -->
### Description
<!-- Describe what this PR does -->
+41
View File
@@ -0,0 +1,41 @@
name: "Check bypass"
description: "A custom action to encapsulate PFCCLab/ci-bypass"
inputs:
github-token:
description: "GitHub token"
required: true
workflow-name:
description: "Workflow name"
required: true
outputs:
can-skip:
description: "Whether the workflow can be skipped."
value: ${{ steps.check-bypass.outputs.can-skip }}
runs:
using: "composite"
steps:
- id: check-bypass
name: Check Bypass
env:
CI_TEAM_MEMBERS: '["tianshuo78520a", "swgu98", "risemeup1", "XieYunshen","luotao1","From00"]'
uses: PFCCLab/ci-bypass@v1
with:
github-token: ${{ inputs.github-token }}
non-pull-request-event-strategy: 'never-skipped'
type: 'composite'
composite-rule: |
{
"any": [
{
"type": "labeled",
"label": ["skip-ci: ${{ inputs.workflow-name }}", "skip-ci: all"],
"username": ${{ env.CI_TEAM_MEMBERS }}
},
{
"type": "commented",
"comment-pattern": [".*/skip-ci ${{ inputs.workflow-name }}.*", ".*/skip-ci all.*"],
"username": ${{ env.CI_TEAM_MEMBERS }}
}
]
}
+30
View File
@@ -0,0 +1,30 @@
name: 'Rerun Workflow'
description: 'Re-run GitHub Actions workflow for a given Pull Request'
inputs:
GITHUB_TOKEN:
description: 'GitHub token with repo scope'
required: true
OWNER:
description: 'Repository owner'
required: true
REPO:
description: 'Repository name'
required: true
PR_ID:
description: 'Pull Request ID'
required: true
JOB_NAME:
description: 'Job name to rerun'
required: true
runs:
using: 'composite'
steps:
- run: bash ./.github/actions/rerun-workflow/rerun.sh
shell: bash
env:
GITHUB_TOKEN: ${{ inputs.GITHUB_TOKEN }}
OWNER: ${{ inputs.OWNER }}
REPO: ${{ inputs.REPO }}
PR_ID: ${{ inputs.PR_ID }}
JOB_NAME: ${{ inputs.JOB_NAME }}
+77
View File
@@ -0,0 +1,77 @@
# Copyright (c) 2025 PaddleNLP Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -e
COMMIT_SHA=$(curl -s -H "Authorization: token $GITHUB_TOKEN" \
"https://api.github.com/repos/$OWNER/$REPO/pulls/$PR_ID" | jq -r '.head.sha')
echo "Commit SHA: $COMMIT_SHA"
response=$(curl -s -H "Authorization: token $GITHUB_TOKEN" \
"https://api.github.com/repos/$OWNER/$REPO/actions/runs?head_sha=$COMMIT_SHA&per_page=100")
echo "Response: $response"
run_ids=$(echo "$response" | jq -r '.workflow_runs[].id')
if [ -n "$run_ids" ]; then
echo "Found run_ids for commit $COMMIT_SHA: $run_ids"
for run_id in $run_ids; do
if [ "$JOB_NAME" = "all-failed" ]; then
echo "Rerunning all failed jobs for run_id: $run_id"
rerun_response=$(curl -X POST -s -w "%{http_code}" -o /dev/null \
-H "Accept: application/vnd.github.v3+json" \
-H "Authorization: Bearer $GITHUB_TOKEN" \
"https://api.github.com/repos/$OWNER/$REPO/actions/runs/$run_id/rerun-failed-jobs")
if [ "$rerun_response" -eq 201 ]; then
echo "Successfully requested rerun for all blocked jobs in run_id: $run_id"
else
echo "Failed to request rerun for run_id: $run_id with status code $rerun_response"
fi
else
jobs_response=$(curl -s -H "Authorization: token $GITHUB_TOKEN" \
"https://api.github.com/repos/$OWNER/$REPO/actions/runs/$run_id/jobs")
echo "Jobs Response for run_id $run_id: $jobs_response"
# if [[ "$JOB_NAME" == *"bypass"* ]]; then
block_jobs=$(echo "$jobs_response" | jq -r --arg job_name "$JOB_NAME" \
'.jobs[] | select(.name == $job_name) | .id')
# else
# block_jobs=$(echo "$jobs_response" | jq -r --arg job_name "$JOB_NAME" \
# '.jobs[] | select(.name == $job_name and .conclusion != "success") | .id')
# fi
if [ -n "$block_jobs" ]; then
echo "Found block jobs for run_id $run_id: $block_jobs"
for job_id in $block_jobs; do
echo "Rerunning job_id: $job_id"
curl -X POST -H "Accept: application/vnd.github.v3+json" \
-H "Authorization: token $GITHUB_TOKEN" \
"https://api.github.com/repos/$OWNER/$REPO/actions/jobs/$job_id/rerun"
done
else
echo "No block jobs found for run_id $run_id with name $JOB_NAME."
fi
fi
done
else
echo "No matching workflow runs found for commit $COMMIT_SHA."
exit 1
fi
+13
View File
@@ -0,0 +1,13 @@
codecov:
notify:
wait_for_ci: false
coverage:
status:
project:
default:
target: 58% # overall project Coverage
threshold: 1% # Allow the coverage to drop by 1%, and posting a success status.
patch:
default:
target: 80% # lines adjusted Coverage < 80% CI will fail
@@ -0,0 +1,47 @@
name: PaddleFormers PR Check
on:
pull_request:
types: [opened, synchronize]
env:
BRANCH: ${{ github.base_ref }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
PR_ID: ${{ github.event.pull_request.number }}
COMMIT_ID: ${{ github.event.pull_request.head.sha }}
token: ${{ vars.ACTION_GITHUB_TOKEN }}
jobs:
check-approvers:
name: Check approval
runs-on:
group: APPROVAL
steps:
- name: Cleanup
run: |
rm -rf * .[^.]*
- name: Update paddle
run: |
wget -q --no-proxy https://xly-devops.bj.bcebos.com/PaddleTest/PaddleNLP/PaddleNLP-develop.tar.gz --no-check-certificate
tar zxf PaddleNLP-develop.tar.gz --strip-components=1 >/dev/null
rm -rf PaddleNLP-develop.tar.gz >/dev/null
git fetch origin pull/${PR_ID}/head
git checkout -b origin_pr FETCH_HEAD
git remote add upstream https://github.com/PaddlePaddle/PaddleNLP.git
git fetch upstream
git checkout -b test_pr upstream/${BRANCH}
git merge --no-edit origin_pr
git log --pretty=oneline -10
- name: Check bypass
id: check-bypass
uses: ./.github/actions/check-bypass
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
workflow-name: approval
- name: Display Required Approvers
if: steps.check-bypass.outputs.can-skip != 'true'
run: |
cd scripts/ci_approval
bash run_ci_approval_cherry-pick.sh
+47
View File
@@ -0,0 +1,47 @@
name: Approval
on:
pull_request:
types: [opened, synchronize]
env:
BRANCH: ${{ github.base_ref }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
PR_ID: ${{ github.event.pull_request.number }}
COMMIT_ID: ${{ github.event.pull_request.head.sha }}
token: ${{ vars.ACTION_GITHUB_TOKEN }}
jobs:
check-approvers:
name: Check approval
runs-on:
group: APPROVAL
steps:
- name: Cleanup
run: |
rm -rf * .[^.]*
- name: Update paddle
run: |
wget -q --no-proxy https://xly-devops.bj.bcebos.com/PaddleTest/PaddleNLP/PaddleNLP-develop.tar.gz --no-check-certificate
tar zxf PaddleNLP-develop.tar.gz --strip-components=1 >/dev/null
rm -rf PaddleNLP-develop.tar.gz >/dev/null
git fetch origin pull/${PR_ID}/head
git checkout -b origin_pr FETCH_HEAD
git remote add upstream https://github.com/PaddlePaddle/PaddleNLP.git
git fetch upstream
git checkout -b test_pr upstream/${BRANCH}
git merge --no-edit origin_pr
git log --pretty=oneline -10
- name: Check bypass
id: check-bypass
uses: ./.github/actions/check-bypass
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
workflow-name: approval
- name: Display Required Approvers
if: steps.check-bypass.outputs.can-skip != 'true'
run: |
cd scripts/ci_approval
bash -x run_ci_approval.sh
+52
View File
@@ -0,0 +1,52 @@
on:
workflow_call:
inputs:
workflow-name:
required: true
type: string
secrets:
github-token:
required: true
outputs:
can-skip:
description: "Whether the workflow can be skipped."
value: ${{ jobs.check-bypass.outputs.can-skip }}
jobs:
check-bypass:
name: Check bypass
runs-on:
group: APPROVAL
permissions:
contents: read
env:
CI_TEAM_MEMBERS: '["tianshuo78520a", "swgu98", "risemeup1" , "XieYunshen","luotao1","From00"]'
outputs:
can-skip: ${{ steps.check-bypass.outputs.can-skip }}
steps:
- name: Cleanup
run: |
rm -rf * .[^.]*
- id: check-bypass
name: Check Bypass
uses: PFCCLab/ci-bypass@v1
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
non-pull-request-event-strategy: 'never-skipped'
type: 'composite'
composite-rule: |
{
"any": [
{
"type": "labeled",
"label": ["skip-ci: ${{ inputs.workflow-name }}", "skip-ci: all"],
"username": ${{ env.CI_TEAM_MEMBERS }}
},
{
"type": "commented",
"comment-pattern": [".*/skip-ci ${{ inputs.workflow-name }}.*", ".*/skip-ci all.*"],
"username": ${{ env.CI_TEAM_MEMBERS }}
}
]
}
+113
View File
@@ -0,0 +1,113 @@
name: Codestyle Check
on: [push, pull_request]
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
cancel-in-progress: true
env:
PR_ID: ${{ github.event.pull_request.number }}
COMMIT_ID: ${{ github.event.pull_request.head.sha }}
BRANCH: ${{ github.event.pull_request.base.ref }}
TASK: PaddleNLP-CI-Lint-${{ github.event.pull_request.number }}
jobs:
check-bypass:
name: Check bypass
uses: ./.github/workflows/check-bypass.yml
with:
workflow-name: 'lint'
secrets:
github-token: ${{ secrets.GITHUB_TOKEN }}
Lint:
name: Lint
needs: check-bypass
if: ${{ needs.check-bypass.outputs.can-skip != 'true' }}
runs-on: [self-hosted, ernie-cpu]
steps:
- name: Run Container
env:
work_dir: ${{ github.workspace }}
python_version: "3.10"
run: |
container_name=${TASK}-$(date +%Y%m%d-%H%M%S)
echo "container_name=${container_name}" >> ${{ github.env }}
docker_image="iregistry.baidu-int.com/paddlecloud/base-images:paddlecloud-ubuntu20.04-gcc12.2-cuda12.3-cudnn9.0-nccl2.20.3.1-openmpi4.1.5-latest"
docker run -d -t --name ${container_name} --net=host -v /dev/shm:/dev/shm --shm-size=32G \
-v $work_dir/../../..:$work_dir/../../.. \
-v $work_dir:/workspace \
-v /home/.cache/pip:/home/.cache/pip \
-e BRANCH \
-e PR_ID \
-e COMMIT_ID \
-e work_dir \
-e no_proxy \
-e python_version \
-w /workspace ${docker_image}
- name: Download Code
env:
work_dir: ${{ github.workspace }}
run: |
docker exec -t ${container_name} /bin/bash -c '
rm -rf * .[^.]*
echo "Downloading PaddleNLP.tar"
wget -q --no-proxy https://paddle-qa.bj.bcebos.com/CodeSync/develop/PaddleNLP.tar --no-check-certificate
echo "Extracting PaddleNLP.tar"
tar xf PaddleNLP.tar && rm -rf PaddleNLP.tar
source $work_dir/../../../proxy
cd PaddleNLP
git config --global user.name "PaddleCI"
git config --global user.email "paddle_ci@example.com"
git pull
git submodule update --init --recursive --force
if [ -n "${PR_ID}" ]; then
git fetch origin pull/${PR_ID}/head
git checkout -b PR_${PR_ID} FETCH_HEAD
git remote add upstream https://github.com/PaddlePaddle/PaddleNLP.git
git fetch upstream ${BRANCH}
git merge ${BRANCH} --no-edit
git diff --numstat ${BRANCH} -- | awk "{print \$NF}"
else
echo "Not in a pull_request event. Skipping PR-specific operations."
fi
git log --pretty=oneline -10
if ! git show-ref --quiet refs/heads/develop; then \
echo "local develop branch is missing, creating local develop branch that tracks remote develop branch"
git fetch origin develop
git branch develop --track origin/develop
else
echo "local develop branch exist, skipping"
fi
'
- name: Setup Environment
run: |
docker exec -t $container_name /bin/bash -c '
unlink /usr/local/bin/python
ln -sf $(which python${python_version}) /usr/local/bin/python
set -e
python -m pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
python -m pip config set global.cache-dir "/home/.cache/pip"
source $work_dir/../../../proxy
python -m pip install --upgrade pip
cd /workspace/PaddleNLP && git config --global --add safe.directory $PWD
make install
'
- name: Test
run: |
docker exec -t $container_name /bin/bash -c '
set -e
source $work_dir/../../../proxy
cd /workspace/PaddleNLP
make lint
'
- name: Terminate And Delete the Container
if: always()
run: |
docker rm -f $container_name 2>/dev/null || true
+207
View File
@@ -0,0 +1,207 @@
# Disabled LLM CI workflow - manually enabled when needed
name: LLM CI (DISABLED)
#on:
# pull_request:
# types: [opened, synchronize, reopened]
# branches: [develop]
# schedule:
# - cron: "2 0 * * *"
# workflow_call:
# inputs:
# run_downstream:
# required: true
# type: string
# image_name:
# required: true
# type: string
#concurrency:
# group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
# cancel-in-progress: true
#env:
# PR_ID: ${{ github.event.pull_request.number }}
COMMIT_ID: ${{ github.event.pull_request.head.sha }}
TASK: paddlenlp-CI-${{ github.event.pull_request.number }}-llm
ci_scripts: /workspace/PaddleNLP/scripts/regression
BRANCH: ${{ github.event.pull_request.base.ref }}
AGILE_COMPILE_BRANCH: ${{ github.event.pull_request.base.ref }}
CI_name: llm-ci
no_proxy: "localhost,bj.bcebos.com,su.bcebos.com,bcebos.com,apiin.im.baidu.com,gitee.com,aliyun.com,.baidu.com,.tuna.tsinghua.edu.cn"
HF_ENDPOINT: https://hf-mirror.com
STUDIO_GIT_HOST: http://git.prod.idc-to-cloud.aistudio.baidu-int.com
PPNLP_HOME: /ssd1/paddlenlp
HF_DATASETS_CACHE: /ssd1/paddlenlp/huggingface/datasets
TRANSFORMERS_CACHE: /ssd1/paddlenlp/huggingface
CCACHE_DIR: /home/data/gzcfs/.ccache/gpubox
RUN_DOWNSTREAM: ${{ inputs.run_downstream }}
#defaults:
# run:
# shell: bash
#jobs:
# llm-ci:
# name: llm-ci (DISABLED)
runs-on: [self-hosted, ernie-8gpu]
steps:
- name: Determine Image Name
env:
IMAGE_NAME: ${{ inputs.image_name }}
run: |
if [[ -n "${IMAGE_NAME}" ]]; then
echo "IMAGE_NAME=${IMAGE_NAME}" >> "$GITHUB_ENV"
else
echo "IMAGE_NAME=iregistry.baidu-int.com/paddlecloud/base-images:paddlecloud-ubuntu18.04-gcc8.2-cuda11.8-cudnn8.6-nccl2.15.5-paddlenlp-latest" >> "$GITHUB_ENV"
fi
- name: Run Container
env:
work_dir: ${{ github.workspace }}
CACHE_DIR: /home/data/cfs/.cache
FLAGS_dynamic_static_unified_comm: "True"
python_version: "3.10"
paddle_whl: https://paddle-qa.bj.bcebos.com/paddle-pipeline/Develop-GpuSome-LinuxCentos-Gcc82-Cuda118-Cudnn86-Trt85-Py310-CINN-Compile/latest/paddlepaddle_gpu-0.0.0-cp310-cp310-linux_x86_64.whl
run: |
container_name=${TASK}-$(date +%Y%m%d-%H%M%S)
echo "container_name=${container_name}" >> "$GITHUB_ENV"
if [[ "$RUN_DOWNSTREAM" == "false" ]]; then
echo "Not in a pull_request or test_build event. Skipping..."
else
docker run -d -t --name ${container_name} --net=host -v /dev/shm:/dev/shm --shm-size=32G \
-v $work_dir/../../..:$work_dir/../../.. \
-v $work_dir:/workspace \
-v /home/.cache/pip:/home/.cache/pip \
-v /ssd1/paddlenlp:/ssd1/paddlenlp \
-v /home/data/gzcfs/.ccache/gpubox:/home/data/gzcfs/.ccache/gpubox \
-e BRANCH \
-e AGILE_COMPILE_BRANCH \
-e PR_ID \
-e COMMIT_ID \
-e work_dir \
-e ci_scripts \
-e no_proxy \
-e CI_name \
-e paddle_whl \
-e HF_ENDPOINT \
-e STUDIO_GIT_HOST \
-e PPNLP_HOME \
-e HF_DATASETS_CACHE \
-e TRANSFORMERS_CACHE \
-e CACHE_DIR \
-e FLAGS_dynamic_static_unified_comm \
-e python_version \
-w /workspace --runtime=nvidia $IMAGE_NAME
fi
- name: Download Code
env:
work_dir: ${{ github.workspace }}
run: |
if [[ "$RUN_DOWNSTREAM" == "false" ]]; then
echo "Not in a pull_request or test_build event. Skipping.."
else
docker exec -t $container_name /bin/bash -c '
rm -rf * .[^.]*
echo "Downloading PaddleNLP.tar.gz"
wget -q --no-proxy https://paddle-qa.bj.bcebos.com/CodeSync/develop/PaddleNLP.tar --no-check-certificate
echo "Extracting PaddleNLP.tar.gz"
tar xf PaddleNLP.tar && rm -rf PaddleNLP.tar
source $work_dir/../../../proxy
cd PaddleNLP
git config --global user.name "PaddleCI"
git config --global user.email "paddle_ci@example.com"
git pull
git submodule update --init --recursive --force
if [ -n "${PR_ID}" ]; then
git fetch origin pull/${PR_ID}/head
git checkout -b PR_${PR_ID} FETCH_HEAD
git remote add upstream https://github.com/PaddlePaddle/PaddleNLP.git
git fetch upstream ${BRANCH}
git merge ${BRANCH} --no-edit
git diff --numstat ${BRANCH} -- | awk "{print \$NF}"
else
echo "Not in a pull_request event. Skipping PR-specific operations."
fi
git log --pretty=oneline -10
'
fi
- name: Skip For Bug
run: |
if [[ "$RUN_DOWNSTREAM" == "false" ]]; then
echo "Not in a pull_request or test_build event. Skipping..."
else
docker exec -t $container_name /bin/bash -c '
cd /workspace/PaddleNLP
git revert f2477c07272d04244cd3287d1f21c70482a4a85f --no-edit # 套件PR#10413引入bug-待修复
git revert d74c950e15a35b7a100d1688c89318195cc83bca --no-edit # 框架升级后配合修改,当前因框架bug固定commit适配
'
fi
- name: Test
run: |
if [[ "$RUN_DOWNSTREAM" == "false" ]]; then
echo "Not in a pull_request or test_build event. Skipping..."
else
docker exec -t $container_name /bin/bash -c '
ldconfig
unlink /usr/bin/python3
ln -sf $(which python${python_version}) /usr/bin/python3
pip config set global.cache-dir "/home/.cache/pip"
set -e
source $work_dir/../../../proxy
cd /workspace/PaddleNLP && git config --global --add safe.directory $PWD
echo "Paddle PR#73283 PR#74484 import bug, set paddle_commit=8ae742 to skip "
export paddle_whl=https://paddle-qa.bj.bcebos.com/paddle-pipeline/Develop-GpuSome-LinuxCentos-Gcc82-Cuda118-Cudnn86-Trt85-Py310-CINN-Compile/8ae7423e99b2ea96e410968a0ebb3f1795e37205/paddlepaddle_gpu-0.0.0-cp310-cp310-linux_x86_64.whl
timeout 2h bash scripts/regression/run_ci.sh python${python_version} ${paddle_whl}
'
fi
- name: Upload Allure-reports & Logs
if: always()
env:
home_path: ${{ github.workspace }}/../../..
bos_file: ${{ github.workspace }}/../../../bos/BosClient.py
allure_file: ${{ github.workspace }}/../../../allure-2.19.0/bin/allure
run: |
if [[ "$RUN_DOWNSTREAM" == "false" ]]; then
echo "Not in a pull_request or test_build event. Skipping..."
else
docker exec -t $container_name /bin/bash -c '
unset http_proxy && unset https_proxy
if [ ! -f "${{ env.bos_file }}" ]; then
wget -q --no-proxy -O ${{ env.home_path }}/bos_new.tar.gz https://xly-devops.bj.bcebos.com/home/bos_new.tar.gz --no-check-certificate
mkdir ${{ env.home_path }}/bos
tar xf ${{ env.home_path }}/bos_new.tar.gz -C ${{ env.home_path }}/bos
fi
if [ ! -f "${{ env.allure_file }}" ]; then
wget -q --no-proxy -O ${{ env.home_path }}/allure-2.19.0.zip https://xly-devops.bj.bcebos.com/tools/allure-2.19.0.zip --no-check-certificate
unzip -q ${{ env.home_path }}/allure-2.19.0.zip -d ${{ env.home_path }}/
fi
if [[ "${{ env.RUN_DOWNSTREAM }}" == "" && -n "${PR_ID}" ]]; then
bos_prefix="${PR_ID}/${COMMIT_ID}"
elif [[ "${{ env.RUN_DOWNSTREAM }}" == "true" && -n "${PR_ID}" ]]; then
bos_prefix="${PR_ID}/${COMMIT_ID}/test_build"
else
bos_prefix="schedule/$(date +%Y%m%d)"
fi
cd /workspace/PaddleNLP/model_logs
for FILE in /workspace/PaddleNLP/model_logs/*; do
file=$(basename "$FILE")
python ${{ env.bos_file }} $file paddle-github-action/PR/PaddleNLP/llm/${bos_prefix}/logs
echo "$file: https://paddle-github-action.bj.bcebos.com/PR/PaddleNLP/llm/${bos_prefix}/logs/$file"
done
cd /workspace/PaddleNLP/
${{ env.allure_file }} generate result -o report
tar -czf products.tar.gz report model_logs
python ${{ env.bos_file }} products.tar.gz paddle-github-action/PR/PaddleNLP/llm/${bos_prefix}/logs
echo "products: https://paddle-github-action.bj.bcebos.com/PR/PaddleNLP/llm/${bos_prefix}/logs/products.tar.gz"
'
fi
- name: Terminate And Delete the Container
if: always()
run: |
docker rm -f ${{ env.container_name }} 2>/dev/null || true
+30
View File
@@ -0,0 +1,30 @@
name: Pipelines-Test
on:
push:
paths:
- 'slm/pipelines/*'
pull_request:
paths:
- 'slm/pipelines/*'
jobs:
Pipelines-Test:
name: Pipelines-Test
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: '3.10'
cache: 'pip' # caching pip dependencies
- name: Install dependencies
working-directory: ./slm/pipelines
run: |
python -m pip install --upgrade pip
make install
pip install -r tests/requirements.txt
- name: run the command
working-directory: ./slm/pipelines
run: make test
+58
View File
@@ -0,0 +1,58 @@
name: Release
on:
workflow_dispatch:
inputs:
package:
description: 'Package to be deployed'
required: true
default: 'paddlenlp'
type: choice
options:
- paddlenlp
- paddle-pipelines
- ppdiffusers
jobs:
Release:
name: Release
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v1
with:
python-version: '3.10'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install setuptools wheel twine
- name: Should Deploy
id: should-deploy
env:
PACKAGE: ${{ inputs.package }}
run: python scripts/should_deploy.py --name $PACKAGE >> $GITHUB_OUTPUT
- name: Check Branch
id: check-branch
run: |
if [[ ${{ github.ref }} =~ ^refs/tags/v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
echo "match=true" >> $GITHUB_OUTPUT
fi # See: https://stackoverflow.com/a/58869470/1123955
- name: Is A Publish Branch
if: steps.check-branch.outputs.match == 'true' && steps.should-deploy.outputs.should_deploy == 'true'
env:
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.paddlenlp }}
PACKAGE: ${{ inputs.package }}
PADDLENLP_STABLE_VERSION: "true"
run: |
make deploy-$PACKAGE
- name: Should Not Deploy To Pypi Server
if: steps.should-deploy.outputs.should_deploy != 'true'
run: echo 'should not deploy pypi server'
- name: Is Not A Publish Branch
if: steps.check-branch.outputs.match != 'true'
run: echo 'Not A Publish Branch'
+73
View File
@@ -0,0 +1,73 @@
name: Re-run
on:
issue_comment:
types: [created]
jobs:
re-run:
if: ${{ github.event.issue.pull_request && contains(github.event.comment.body, '/re-run') && github.event.comment.user.login == github.event.issue.user.login }}
runs-on: [self-hosted, ernie-cpu]
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Rerun all failed jobs
if: ${{ contains(github.event.comment.body, 'all-failed') }}
uses: ./.github/actions/rerun-workflow
with:
PR_ID: ${{ github.event.issue.number }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
OWNER: ${{ github.repository_owner }}
REPO: ${{ github.event.repository.name }}
JOB_NAME: 'all-failed'
- name: Rerun LLM
if: ${{ contains(github.event.comment.body, 'LLM') }}
uses: ./.github/actions/rerun-workflow
with:
PR_ID: ${{ github.event.issue.number }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
OWNER: ${{ github.repository_owner }}
REPO: ${{ github.event.repository.name }}
JOB_NAME: 'LLM CI / llm-ci'
- name: Unittest GPU
if: ${{ contains(github.event.comment.body, 'Unittest GPU') }}
uses: ./.github/actions/rerun-workflow
with:
PR_ID: ${{ github.event.issue.number }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
OWNER: ${{ github.repository_owner }}
REPO: ${{ github.event.repository.name }}
JOB_NAME: 'Unittest GPU CI / unittest-gpu-ci'
- name: Rerun Unittest CPU
if: ${{ contains(github.event.comment.body, 'Unittest CPU') }}
uses: ./.github/actions/rerun-workflow
with:
PR_ID: ${{ github.event.issue.number }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
OWNER: ${{ github.repository_owner }}
REPO: ${{ github.event.repository.name }}
JOB_NAME: 'Unittest CPU CI / unit-cpu-ci'
- name: Rerun Distribute (V100)
if: ${{ contains(github.event.comment.body, 'Distribute (V100)') }}
uses: ./.github/actions/rerun-workflow
with:
PR_ID: ${{ github.event.issue.number }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
OWNER: ${{ github.repository_owner }}
REPO: ${{ github.event.repository.name }}
JOB_NAME: 'Distribute CI (V100)/ distribute-v100-ci'
- name: Rerun Codestyle Check
if: ${{ contains(github.event.comment.body, 'Codestyle Check') }}
uses: ./.github/actions/rerun-workflow
with:
PR_ID: ${{ github.event.issue.number }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
OWNER: ${{ github.repository_owner }}
REPO: ${{ github.event.repository.name }}
JOB_NAME: 'Codestyle Check / Lint'
+35
View File
@@ -0,0 +1,35 @@
name: Stale
on:
# Allow manual run via GitHub web or CLI
workflow_dispatch:
schedule:
# Run daily at midnight UTC
- cron: 0 0 * * *
permissions:
issues: write
pull-requests: write
jobs:
stale:
runs-on: ubuntu-24.04
permissions:
issues: write
pull-requests: write
steps:
- uses: actions/stale@v6.0.1
with:
days-before-issue-stale: 60
days-before-issue-close: 14
stale-issue-label: "stale"
stale-issue-message: "This issue is stale because it has been open for 60 days with no activity. 当前issue 60天内无活动,被标记为stale。"
close-issue-message: "This issue was closed because it has been inactive for 14 days since being marked as stale. 当前issue 被标记为stale已有14天,即将关闭。"
exempt-issue-labels: 'triage,keep'
days-before-pr-stale: 60
days-before-pr-close: -1
stale-pr-label: "stale"
stale-pr-message: "This Pull Request is stale because it has been open for 60 days with no activity. 当前Pull Request 60天内无活动,被标记为stale。"
operations-per-run: 400
+120
View File
@@ -0,0 +1,120 @@
# Disabled unittest CPU workflow - manually enabled when needed
name: Unittest CPU CI (DISABLED)
#on: [pull_request]
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
cancel-in-progress: true
env:
PR_ID: ${{ github.event.pull_request.number }}
COMMIT_ID: ${{ github.event.pull_request.head.sha }}
BRANCH: ${{ github.event.pull_request.base.ref }}
TASK: PaddleNLP-CI-${{ github.event.pull_request.number }}-unittest-cpu
HF_ENDPOINT: https://hf-mirror.com
STUDIO_GIT_HOST: http://git.prod.idc-to-cloud.aistudio.baidu-int.com
PPNLP_HOME: /home/disk1/cache
HF_DATASETS_CACHE: /home/disk1/cache/huggingface/datasets
TRANSFORMERS_CACHE: /home/disk1/cache/huggingface
#jobs:
# unittest-cpu-ci:
# name: unittest-cpu-ci (DISABLED)
runs-on: [self-hosted, ernie-cpu]
permissions:
pull-requests: write
contents: read
id-token: write
steps:
- name: Run Container
env:
work_dir: ${{ github.workspace }}
python_version: "3.10"
run: |
container_name=${TASK}-$(date +%Y%m%d-%H%M%S)
echo "container_name=${container_name}" >> ${{ github.env }}
docker_image="iregistry.baidu-int.com/paddlecloud/base-images:paddlecloud-ubuntu20.04-gcc12.2-cuda12.3-cudnn9.0-nccl2.20.3.1-openmpi4.1.5-latest"
docker run -d -t --name ${container_name} --net=host -v /dev/shm:/dev/shm --shm-size=32G \
-v $work_dir/../../..:$work_dir/../../.. \
-v $work_dir:/workspace \
-v /home/.cache/pip:/home/.cache/pip \
-v /home/disk1/cache:/home/disk1/cache \
-e BRANCH \
-e PR_ID \
-e COMMIT_ID \
-e work_dir \
-e no_proxy \
-e python_version \
-e HF_ENDPOINT \
-e STUDIO_GIT_HOST \
-e PPNLP_HOME \
-e HF_DATASETS_CACHE \
-e TRANSFORMERS_CACHE \
-w /workspace ${docker_image}
- name: Download Code
env:
work_dir: ${{ github.workspace }}
run: |
docker exec -t ${container_name} /bin/bash -c '
rm -rf * .[^.]*
echo "Downloading PaddleNLP.tar"
wget -q --no-proxy https://paddle-qa.bj.bcebos.com/CodeSync/develop/PaddleNLP.tar --no-check-certificate
echo "Extracting PaddleNLP.tar"
tar xf PaddleNLP.tar && rm -rf PaddleNLP.tar
source $work_dir/../../../proxy
cd PaddleNLP
git config --global user.name "PaddleCI"
git config --global user.email "paddle_ci@example.com"
git pull
git submodule update --init --recursive --force
if [ -n "${PR_ID}" ]; then
git fetch origin pull/${PR_ID}/head
git checkout -b PR_${PR_ID} FETCH_HEAD
git remote add upstream https://github.com/PaddlePaddle/PaddleNLP.git
git fetch upstream ${BRANCH}
git merge ${BRANCH} --no-edit
git diff --numstat ${BRANCH} -- | awk "{print \$NF}"
else
echo "Not in a pull_request event. Skipping PR-specific operations."
fi
git log --pretty=oneline -10
'
- name: Setup Environment
run: |
docker exec -t $container_name /bin/bash -c '
unlink /usr/bin/python3
ln -sf $(which python${python_version}) /usr/bin/python3
set -e
python -m pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
python -m pip config set global.cache-dir "/home/.cache/pip"
source $work_dir/../../../proxy
python -m pip install --upgrade pip
cd /workspace/PaddleNLP && git config --global --add safe.directory $PWD
pip install -r tests/requirements.txt
make install
'
- name: Test
run: |
docker exec -t $container_name /bin/bash -c '
source $work_dir/../../../proxy
cd /workspace/PaddleNLP
set -e
make test
'
# - name: Upload Coverage To Codecov
# uses: codecov/codecov-action@v3
# env:
# CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
# with:
# token: ${{ secrets.CODECOV_TOKEN }}
# files: ./PaddleNLP/coverage.xml
- name: Terminate And Delete the Container
if: always()
run: |
docker rm -f $container_name 2>/dev/null || true
+204
View File
@@ -0,0 +1,204 @@
# Disabled unittest GPU workflow - manually enabled when needed
name: Unittest GPU CI (DISABLED)
#on:
# pull_request:
# types: [opened, synchronize, reopened]
# branches: [develop]
# schedule:
# - cron: "3 0 * * *"
# workflow_call:
# inputs:
# run_downstream:
# required: true
# type: string
# image_name:
# required: true
# type: string
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
cancel-in-progress: true
env:
PR_ID: ${{ github.event.pull_request.number }}
COMMIT_ID: ${{ github.event.pull_request.head.sha }}
TASK: paddlenlp-CI-${{ github.event.pull_request.number }}-unit-gpu
ci_scripts: /workspace/PaddleNLP/scripts/unit_test
BRANCH: ${{ github.event.pull_request.base.ref }}
AGILE_COMPILE_BRANCH: ${{ github.event.pull_request.base.ref }}
CI_name: unittest-gpu-ci
no_proxy: "localhost,bj.bcebos.com,su.bcebos.com,bcebos.com,apiin.im.baidu.com,gitee.com,aliyun.com,.baidu.com,.tuna.tsinghua.edu.cn"
HF_ENDPOINT: https://hf-mirror.com
STUDIO_GIT_HOST: http://git.prod.idc-to-cloud.aistudio.baidu-int.com
PPNLP_HOME: /ssd1/paddlenlp
HF_DATASETS_CACHE: /ssd1/paddlenlp/huggingface/datasets
TRANSFORMERS_CACHE: /ssd1/paddlenlp/huggingface
CCACHE_DIR: /home/data/gzcfs/.ccache/gpubox
RUN_DOWNSTREAM: ${{ inputs.run_downstream }}
defaults:
run:
shell: bash
#jobs:
# unittest-gpu-ci:
# name: unittest-gpu-ci (DISABLED)
runs-on: [self-hosted, ernie-8gpu]
steps:
- name: Determine Image Name
env:
IMAGE_NAME: ${{ inputs.image_name }}
run: |
if [[ -n "${IMAGE_NAME}" ]]; then
echo "IMAGE_NAME=${IMAGE_NAME}" >> "$GITHUB_ENV"
else
echo "IMAGE_NAME=iregistry.baidu-int.com/paddlecloud/base-images:paddlecloud-ubuntu18.04-gcc8.2-cuda11.8-cudnn8.6-nccl2.15.5-paddlenlp-latest" >> "$GITHUB_ENV"
fi
- name: Run Container
env:
work_dir: ${{ github.workspace }}
CACHE_DIR: /home/data/cfs/.cache
FLAGS_dynamic_static_unified_comm: "True"
python_version: "3.10"
paddle_whl: https://paddle-qa.bj.bcebos.com/paddle-pipeline/Develop-GpuSome-LinuxCentos-Gcc82-Cuda118-Cudnn86-Trt85-Py310-CINN-Compile/latest/paddlepaddle_gpu-0.0.0-cp310-cp310-linux_x86_64.whl
run: |
container_name=${TASK}-$(date +%Y%m%d-%H%M%S)
echo "container_name=${container_name}" >> "$GITHUB_ENV"
if [[ "$RUN_DOWNSTREAM" == "false" ]]; then
echo "Not in a pull_request or test_build event. Skipping..."
else
docker run -d -t --name ${container_name} --net=host -v /dev/shm:/dev/shm --shm-size=32G \
-v $work_dir/../../..:$work_dir/../../.. \
-v $work_dir:/workspace \
-v /home/.cache/pip:/home/.cache/pip \
-v /ssd1/paddlenlp:/ssd1/paddlenlp \
-v /home/data/gzcfs/.ccache/gpubox:/home/data/gzcfs/.ccache/gpubox \
-e BRANCH \
-e AGILE_COMPILE_BRANCH \
-e PR_ID \
-e COMMIT_ID \
-e work_dir \
-e ci_scripts \
-e no_proxy \
-e CI_name \
-e paddle_whl \
-e HF_ENDPOINT \
-e STUDIO_GIT_HOST \
-e PPNLP_HOME \
-e HF_DATASETS_CACHE \
-e TRANSFORMERS_CACHE \
-e CACHE_DIR \
-e FLAGS_dynamic_static_unified_comm \
-e python_version \
-w /workspace --runtime=nvidia $IMAGE_NAME
fi
- name: Download Code
env:
work_dir: ${{ github.workspace }}
run: |
if [[ "$RUN_DOWNSTREAM" == "false" ]]; then
echo "Not in a pull_request or test_build event. Skipping.."
else
docker exec -t $container_name /bin/bash -c '
rm -rf * .[^.]*
echo "Downloading PaddleNLP.tar.gz"
wget -q --no-proxy https://paddle-qa.bj.bcebos.com/CodeSync/develop/PaddleNLP.tar --no-check-certificate
echo "Extracting PaddleNLP.tar.gz"
tar xf PaddleNLP.tar && rm -rf PaddleNLP.tar
source $work_dir/../../../proxy
cd PaddleNLP
git config --global user.name "PaddleCI"
git config --global user.email "paddle_ci@example.com"
git pull
git submodule update --init --recursive --force
if [ -n "${PR_ID}" ]; then
git fetch origin pull/${PR_ID}/head
git checkout -b PR_${PR_ID} FETCH_HEAD
git remote add upstream https://github.com/PaddlePaddle/PaddleNLP.git
git fetch upstream ${BRANCH}
git merge ${BRANCH} --no-edit
git diff --numstat ${BRANCH} -- | awk "{print \$NF}"
else
echo "Not in a pull_request event. Skipping PR-specific operations."
fi
git log --pretty=oneline -10
'
fi
- name: Skip For Bug
run: |
if [[ "$RUN_DOWNSTREAM" == "false" ]]; then
echo "Not in a pull_request or test_build event. Skipping..."
else
docker exec -t $container_name /bin/bash -c '
cd /workspace/PaddleNLP
echo "no skip for bug"
'
fi
- name: Test
run: |
if [[ "$RUN_DOWNSTREAM" == "false" ]]; then
echo "Not in a pull_request or test_build event. Skipping..."
else
docker exec -t $container_name /bin/bash -c '
ldconfig
unlink /usr/bin/python3
ln -sf $(which python${python_version}) /usr/bin/python3
pip config set global.cache-dir "/home/.cache/pip"
source $work_dir/../../../proxy
set -e
cd /workspace/PaddleNLP && git config --global --add safe.directory $PWD
timeout 50m bash scripts/unit_test/ci_unit.sh ${paddle_whl}
'
fi
- name: Upload Allure-reports & Logs
if: always()
env:
home_path: ${{ github.workspace }}/../../..
bos_file: ${{ github.workspace }}/../../../bos/BosClient.py
allure_file: ${{ github.workspace }}/../../../allure-2.19.0/bin/allure
run: |
if [[ "$RUN_DOWNSTREAM" == "false" ]]; then
echo "Not in a pull_request or test_build event. Skipping..."
else
docker exec -t $container_name /bin/bash -c '
unset http_proxy && unset https_proxy
if [ ! -f "${{ env.bos_file }}" ]; then
wget -q --no-proxy -O ${{ env.home_path }}/bos_new.tar.gz https://xly-devops.bj.bcebos.com/home/bos_new.tar.gz --no-check-certificate
mkdir ${{ env.home_path }}/bos
tar xf ${{ env.home_path }}/bos_new.tar.gz -C ${{ env.home_path }}/bos
fi
if [ ! -f "${{ env.allure_file }}" ]; then
wget -q --no-proxy -O ${{ env.home_path }}/allure-2.19.0.zip https://xly-devops.bj.bcebos.com/tools/allure-2.19.0.zip --no-check-certificate
unzip -q ${{ env.home_path }}/allure-2.19.0.zip -d ${{ env.home_path }}/
fi
if [[ "${{ env.RUN_DOWNSTREAM }}" == "" && -n "${PR_ID}" ]]; then
bos_prefix="${PR_ID}/${COMMIT_ID}"
elif [[ "${{ env.RUN_DOWNSTREAM }}" == "true" && -n "${PR_ID}" ]]; then
bos_prefix="${PR_ID}/${COMMIT_ID}/test_build"
else
bos_prefix="schedule/$(date +%Y%m%d)"
fi
cd /workspace/PaddleNLP/unittest_logs
for FILE in /workspace/PaddleNLP/unittest_logs/*; do
file=$(basename "$FILE")
python ${{ env.bos_file }} $file paddle-github-action/PR/PaddleNLP/unittest-gpu/${bos_prefix}/logs
echo "$file: https://paddle-github-action.bj.bcebos.com/PR/PaddleNLP/unittest-gpu/${bos_prefix}/logs/$file"
done
cd /workspace/PaddleNLP/
${{ env.allure_file }} generate result -o report
tar -czf products.tar.gz report unittest_logs
python ${{ env.bos_file }} products.tar.gz paddle-github-action/PR/PaddleNLP/unittest-gpu/${bos_prefix}/logs
echo "report: https://paddle-github-action.bj.bcebos.com/PR/PaddleNLP/unittest-gpu/${bos_prefix}/logs/products.tar.gz"
'
fi
- name: Terminate And Delete the Container
if: always()
run: |
docker rm -f $container_name 2>/dev/null || true