chore: import upstream snapshot with attribution
This commit is contained in:
@@ -0,0 +1,89 @@
|
||||
**简体中文**🀄 | [English🌎](./CODE_OF_CONDUCT_en.md)
|
||||
|
||||
# 贡献者公约
|
||||
|
||||
## 我们的承诺
|
||||
|
||||
身为社区成员、贡献者和领袖,我们承诺使社区参与者不受骚扰,无论其年龄、体型、可见或不可见的缺陷、族裔、性征、性别认同和表达、经验水平、教育程度、社会与经济地位、国籍、相貌、种族、种姓、肤色、宗教信仰、性倾向或性取向如何。
|
||||
|
||||
我们承诺以有助于建立开放、友善、多样化、包容、健康社区的方式行事和互动。
|
||||
|
||||
## 我们的准则
|
||||
|
||||
有助于为我们的社区创造积极环境的行为例子包括但不限于:
|
||||
|
||||
* 表现出对他人的同情和善意
|
||||
* 尊重不同的主张、观点和感受
|
||||
* 提出和大方接受建设性意见
|
||||
* 承担责任并向受我们错误影响的人道歉
|
||||
* 注重社区共同诉求,而非个人得失
|
||||
|
||||
不当行为例子包括:
|
||||
|
||||
* 使用情色化的语言或图像,及性引诱或挑逗
|
||||
* 嘲弄、侮辱或诋毁性评论,以及人身或政治攻击
|
||||
* 公开或私下的骚扰行为
|
||||
* 未经他人明确许可,公布他人的私人信息,如物理或电子邮件地址
|
||||
* 其他有理由认定为违反职业操守的不当行为
|
||||
|
||||
## 责任和权力
|
||||
|
||||
社区领袖有责任解释和落实我们所认可的行为准则,并妥善公正地对他们认为不当、威胁、冒犯或有害的任何行为采取纠正措施。
|
||||
|
||||
社区领导有权力和责任删除、编辑或拒绝或拒绝与本行为准则不相符的评论(comment)、提交(commits)、代码、维基(wiki)编辑、议题(issues)或其他贡献,并在适当时机知采取措施的理由。
|
||||
|
||||
## 适用范围
|
||||
|
||||
本行为准则适用于所有社区场合,也适用于在公共场所代表社区时的个人。
|
||||
|
||||
代表社区的情形包括使用官方电子邮件地址、通过官方社交媒体帐户发帖或在线上或线下活动中担任指定代表。
|
||||
|
||||
## 监督
|
||||
|
||||
辱骂、骚扰或其他不可接受的行为可通过 paddlenlp@baidu.com 向负责监督的社区领袖报告。
|
||||
所有投诉都将得到及时和公平的审查和调查。
|
||||
|
||||
所有社区领袖都有义务尊重任何事件报告者的隐私和安全。
|
||||
|
||||
## 处理方针
|
||||
|
||||
社区领袖将遵循下列社区处理方针来明确他们所认定违反本行为准则的行为的处理方式:
|
||||
|
||||
### 1. 纠正
|
||||
|
||||
**社区影响**:使用不恰当的语言或其他在社区中被认定为不符合职业道德或不受欢迎的行为。
|
||||
|
||||
**处理意见**:由社区领袖发出非公开的书面警告,明确说明违规行为的性质,并解释举止如何不妥。或将要求公开道歉。
|
||||
|
||||
### 2. 警告
|
||||
|
||||
**社区影响**:单个或一系列违规行为。
|
||||
|
||||
**处理意见**:警告并对连续性行为进行处理。在指定时间内,不得与相关人员互动,包括主动与行为准则执行者互动。这包括避免在社区场所和外部渠道中的互动。违反这些条款可能会导致临时或永久封禁。
|
||||
|
||||
### 3. 临时封禁
|
||||
|
||||
**社区影响**: 严重违反社区准则,包括持续的不当行为。
|
||||
|
||||
**处理意见**: 在指定时间内,暂时禁止与社区进行任何形式的互动或公开交流。在此期间,不得与相关人员进行公开或私下互动,包括主动与行为准则执行者互动。违反这些条款可能会导致永久封禁。
|
||||
|
||||
### 4. 永久封禁
|
||||
|
||||
**社区影响**:行为模式表现出违反社区准则,包括持续的不当行为、骚扰个人或攻击或贬低某个类别的个体。
|
||||
|
||||
**处理意见**:永久禁止在社区内进行任何形式的公开互动。
|
||||
|
||||
## 参见
|
||||
|
||||
本行为准则改编自 [Contributor Covenant][homepage] 2.1 版, 参见 [https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1]。
|
||||
|
||||
社区处理方针灵感来源于 [Mozilla's code of conduct enforcement ladder][Mozilla CoC]。
|
||||
|
||||
有关本行为准则的常见问题的答案,参见 [https://www.contributor-covenant.org/faq][FAQ]。
|
||||
其他语言翻译参见 [https://www.contributor-covenant.org/translations][translations]。
|
||||
|
||||
[homepage]: https://www.contributor-covenant.org
|
||||
[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
|
||||
[Mozilla CoC]: https://github.com/mozilla/diversity
|
||||
[FAQ]: https://www.contributor-covenant.org/faq
|
||||
[translations]: https://www.contributor-covenant.org/translations
|
||||
@@ -0,0 +1,134 @@
|
||||
[简体中文🀄](./CODE_OF_CONDUCT.md) | **English**🌎
|
||||
|
||||
# Contributor Covenant Code of Conduct
|
||||
|
||||
## Our Pledge
|
||||
|
||||
We as members, contributors, and leaders pledge to make participation in our
|
||||
community a harassment-free experience for everyone, regardless of age, body
|
||||
size, visible or invisible disability, ethnicity, sex characteristics, gender
|
||||
identity and expression, level of experience, education, socio-economic status,
|
||||
nationality, personal appearance, race, caste, color, religion, or sexual
|
||||
identity and orientation.
|
||||
|
||||
We pledge to act and interact in ways that contribute to an open, welcoming,
|
||||
diverse, inclusive, and healthy community.
|
||||
|
||||
## Our Standards
|
||||
|
||||
Examples of behavior that contributes to a positive environment for our
|
||||
community include:
|
||||
|
||||
* Demonstrating empathy and kindness toward other people
|
||||
* Being respectful of differing opinions, viewpoints, and experiences
|
||||
* Giving and gracefully accepting constructive feedback
|
||||
* Accepting responsibility and apologizing to those affected by our mistakes,
|
||||
and learning from the experience
|
||||
* Focusing on what is best not just for us as individuals, but for the overall
|
||||
community
|
||||
|
||||
Examples of unacceptable behavior include:
|
||||
|
||||
* The use of sexualized language or imagery, and sexual attention or advances of
|
||||
any kind
|
||||
* Trolling, insulting or derogatory comments, and personal or political attacks
|
||||
* Public or private harassment
|
||||
* Publishing others' private information, such as a physical or email address,
|
||||
without their explicit permission
|
||||
* Other conduct which could reasonably be considered inappropriate in a
|
||||
professional setting
|
||||
|
||||
## Enforcement Responsibilities
|
||||
|
||||
Community leaders are responsible for clarifying and enforcing our standards of
|
||||
acceptable behavior and will take appropriate and fair corrective action in
|
||||
response to any behavior that they deem inappropriate, threatening, offensive,
|
||||
or harmful.
|
||||
|
||||
Community leaders have the right and responsibility to remove, edit, or reject
|
||||
comments, commits, code, wiki edits, issues, and other contributions that are
|
||||
not aligned to this Code of Conduct, and will communicate reasons for moderation
|
||||
decisions when appropriate.
|
||||
|
||||
## Scope
|
||||
|
||||
This Code of Conduct applies within all community spaces, and also applies when
|
||||
an individual is officially representing the community in public spaces.
|
||||
Examples of representing our community include using an official e-mail address,
|
||||
posting via an official social media account, or acting as an appointed
|
||||
representative at an online or offline event.
|
||||
|
||||
## Enforcement
|
||||
|
||||
Instances of abusive, harassing, or otherwise unacceptable behavior may be
|
||||
reported to the community leaders responsible for enforcement at
|
||||
paddlenlp@baidu.com.
|
||||
All complaints will be reviewed and investigated promptly and fairly.
|
||||
|
||||
All community leaders are obligated to respect the privacy and security of the
|
||||
reporter of any incident.
|
||||
|
||||
## Enforcement Guidelines
|
||||
|
||||
Community leaders will follow these Community Impact Guidelines in determining
|
||||
the consequences for any action they deem in violation of this Code of Conduct:
|
||||
|
||||
### 1. Correction
|
||||
|
||||
**Community Impact**: Use of inappropriate language or other behavior deemed
|
||||
unprofessional or unwelcome in the community.
|
||||
|
||||
**Consequence**: A private, written warning from community leaders, providing
|
||||
clarity around the nature of the violation and an explanation of why the
|
||||
behavior was inappropriate. A public apology may be requested.
|
||||
|
||||
### 2. Warning
|
||||
|
||||
**Community Impact**: A violation through a single incident or series of
|
||||
actions.
|
||||
|
||||
**Consequence**: A warning with consequences for continued behavior. No
|
||||
interaction with the people involved, including unsolicited interaction with
|
||||
those enforcing the Code of Conduct, for a specified period of time. This
|
||||
includes avoiding interactions in community spaces as well as external channels
|
||||
like social media. Violating these terms may lead to a temporary or permanent
|
||||
ban.
|
||||
|
||||
### 3. Temporary Ban
|
||||
|
||||
**Community Impact**: A serious violation of community standards, including
|
||||
sustained inappropriate behavior.
|
||||
|
||||
**Consequence**: A temporary ban from any sort of interaction or public
|
||||
communication with the community for a specified period of time. No public or
|
||||
private interaction with the people involved, including unsolicited interaction
|
||||
with those enforcing the Code of Conduct, is allowed during this period.
|
||||
Violating these terms may lead to a permanent ban.
|
||||
|
||||
### 4. Permanent Ban
|
||||
|
||||
**Community Impact**: Demonstrating a pattern of violation of community
|
||||
standards, including sustained inappropriate behavior, harassment of an
|
||||
individual, or aggression toward or disparagement of classes of individuals.
|
||||
|
||||
**Consequence**: A permanent ban from any sort of public interaction within the
|
||||
community.
|
||||
|
||||
## Attribution
|
||||
|
||||
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
|
||||
version 2.1, available at
|
||||
[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
|
||||
|
||||
Community Impact Guidelines were inspired by
|
||||
[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
|
||||
|
||||
For answers to common questions about this code of conduct, see the FAQ at
|
||||
[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
|
||||
[https://www.contributor-covenant.org/translations][translations].
|
||||
|
||||
[homepage]: https://www.contributor-covenant.org
|
||||
[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
|
||||
[Mozilla CoC]: https://github.com/mozilla/diversity
|
||||
[FAQ]: https://www.contributor-covenant.org/faq
|
||||
[translations]: https://www.contributor-covenant.org/translations
|
||||
@@ -0,0 +1,154 @@
|
||||
[简体中文🀄](../CONTRIBUTING.md) | **English**🌎
|
||||
|
||||
# Contributing to PaddleNLP
|
||||
|
||||
We highly welcome and value your contributions to `PaddleNLP`. The first step to start your contribution is to sign the [PaddlePaddle Contributor License Agreement](https://cla-assistant.io/PaddlePaddle/PaddleNLP).
|
||||
|
||||
This document explains our workflow and work style:
|
||||
|
||||
## Finding out what to work on Workflow
|
||||
|
||||
## Development Workflow
|
||||
|
||||
PaddleNLP uses the [Git branching model](http://nvie.com/posts/a-successful-git-branching-model/). The following steps guide usual contributions.
|
||||
|
||||
#### 1. Fork
|
||||
|
||||
Our development community has been growing fastly; it doesn't make sense for everyone to write into the official repo. So, please file Pull Requests from your fork. To make a fork, just head over to the GitHub page and click the ["Fork" button](https://help.github.com/articles/fork-a-repo/).
|
||||
|
||||
#### 2. Clone
|
||||
|
||||
To make a copy of your fork to your local computers, please run
|
||||
|
||||
```bash
|
||||
git clone https://github.com/<your-github-account>/PaddleNLP
|
||||
cd PaddleNLP
|
||||
```
|
||||
|
||||
#### 3. Create the local feature branch
|
||||
|
||||
For daily works like adding a new feature or fixing a bug, please open your feature branch before coding:
|
||||
|
||||
```bash
|
||||
git checkout -b my-cool-feature
|
||||
```
|
||||
|
||||
#### 4. Set up the development environment
|
||||
|
||||
Before you start coding, you need to setup the development environment. We highly recommend doing all your development in a virtual environment such as
|
||||
[venv](https://docs.python.org/3/library/venv.html) or [conda](https://docs.conda.io/en/latest/). After you setup and activated your virtual environment,
|
||||
run the following command:
|
||||
|
||||
```bash
|
||||
make install
|
||||
```
|
||||
|
||||
This will setup all the dependencies of `PaddleNLP` as well as the [`pre-commit`](http://pre-commit.com/) tool.
|
||||
|
||||
If you are working on the `examples` or `applications` module and require importing from `PaddleNLP`, make sure you install `PaddleNLP` in editable mode.
|
||||
If `PaddleNLP` is already installed in the virtual environment, remove it with `pip uninstall paddlenlp` before reinstalling it in editable mode with
|
||||
`pip install -e .`
|
||||
|
||||
#### 5. Develop
|
||||
|
||||
As you develop your new exciting feature, keep in mind that it should be covered by unit tests. All of our unit tests can be found under the `tests` directory.
|
||||
You can either modify existing unit test to cover the new feature, or create a new test from scratch.
|
||||
As you finish up the your code, you should make sure the test suite passes. You can run the tests impacted by your changes like this:
|
||||
|
||||
```bash
|
||||
pytest tests/<test_to_run>.py
|
||||
```
|
||||
|
||||
#### 6. Commit
|
||||
|
||||
We utilizes [`pre-commit`](http://pre-commit.com/) (with [black](https://black.readthedocs.io/en/stable/), [isort](https://pycqa.github.io/isort/) and
|
||||
[flake8](https://flake8.pycqa.org/en/latest/) under the hood) to check the style of code and documentation in every commit. When you run run `git commit`, you will see
|
||||
something like the following:
|
||||
|
||||
```
|
||||
➜ (my-virtual-env) git commit -m "commiting my cool feature"
|
||||
black....................................................................Passed
|
||||
isort....................................................................Passed
|
||||
flake8...................................................................Passed
|
||||
check for merge conflicts................................................Passed
|
||||
check for broken symlinks............................(no files to check)Skipped
|
||||
detect private key.......................................................Passed
|
||||
fix end of files.....................................(no files to check)Skipped
|
||||
trim trailing whitespace.............................(no files to check)Skipped
|
||||
CRLF end-lines checker...............................(no files to check)Skipped
|
||||
CRLF end-lines remover...............................(no files to check)Skipped
|
||||
No-tabs checker......................................(no files to check)Skipped
|
||||
Tabs remover.........................................(no files to check)Skipped
|
||||
copyright_checker........................................................Passed
|
||||
```
|
||||
|
||||
But most of the time things don't go so smoothly. When your code or documentation doesn't meet the standard, the `pre-commit` check will fail.
|
||||
```
|
||||
➜ (my-virtual-env) git commit -m "commiting my cool feature"
|
||||
black....................................................................Passed
|
||||
isort....................................................................Failed
|
||||
- hook id: isort
|
||||
- files were modified by this hook
|
||||
|
||||
Fixing examples/information_extraction/waybill_ie/run_ernie_crf.py
|
||||
|
||||
flake8...................................................................Passed
|
||||
check for merge conflicts................................................Passed
|
||||
check for broken symlinks............................(no files to check)Skipped
|
||||
detect private key.......................................................Passed
|
||||
fix end of files.....................................(no files to check)Skipped
|
||||
trim trailing whitespace.............................(no files to check)Skipped
|
||||
CRLF end-lines checker...............................(no files to check)Skipped
|
||||
CRLF end-lines remover...............................(no files to check)Skipped
|
||||
No-tabs checker......................................(no files to check)Skipped
|
||||
Tabs remover.........................................(no files to check)Skipped
|
||||
copyright_checker........................................................Passed
|
||||
```
|
||||
|
||||
But **don't panic**!
|
||||
Our tooling will fix most of the style errors automatically. Some errors will need to be addressed manually. Fortunately, the error messages are straight forward and
|
||||
the errors are usually simple to fix. After addressing the errors, you can run `git add <files>` and `git commit` again, which will trigger `pre-commit` again.
|
||||
Once the `pre-commit` checks pass, you are ready to push the code.
|
||||
|
||||
[Google][http://google.com/] or [StackOverflow](https://stackoverflow.com/) are great tools to help you understand the code style errors.
|
||||
Don't worry if you still can't figure it out. You can commit with `git commit -m "style error" --no-verify` and we are happy to help you once you create a Pull Request.
|
||||
|
||||
#### 7. Keep pulling
|
||||
|
||||
An experienced Git user pulls from the official repo often -- daily or even hourly, so they notice conflicts with others work early, and it's easier to resolve smaller conflicts.
|
||||
|
||||
```bash
|
||||
git remote add upstream https://github.com/PaddlePaddle/PaddleNLP
|
||||
git pull upstream develop
|
||||
```
|
||||
|
||||
#### 8. Push and file a pull request
|
||||
|
||||
You can "push" your local work into your forked repo:
|
||||
|
||||
```bash
|
||||
git push origin my-cool-stuff
|
||||
```
|
||||
|
||||
The push allows you to create a pull request, requesting owners of this [official repo](https://github.com/PaddlePaddle/PaddleNLP) to pull your change into the official one.
|
||||
|
||||
To create a pull request, please follow [these steps](https://help.github.com/articles/creating-a-pull-request/).
|
||||
|
||||
#### 9. Delete local and remote branches
|
||||
|
||||
To keep your local workspace and your fork clean, you might want to remove merged branches:
|
||||
|
||||
```bash
|
||||
git push origin my-cool-stuff
|
||||
git checkout develop
|
||||
git pull upstream develop
|
||||
git branch -d my-cool-stuff
|
||||
```
|
||||
|
||||
## Code Review
|
||||
|
||||
- Please feel free to ping your reviewers by @-mentioning the in the Pull Request. Please do this after your pull request passes the CI.
|
||||
|
||||
- Please answer reviewers' every comment. If you are to follow the comment, please write "Done"; Otherwise, please start a discussion under the comment.
|
||||
|
||||
- If you don't want your reviewers to get overwhelmed by email notifications, you might reply their comments by [in a batch](https://help.github.com/articles/reviewing-proposed-changes-in-a-pull-request/).
|
||||
@@ -0,0 +1,23 @@
|
||||
name: 🐛 Ask Question
|
||||
description: 请描述您使用PaddleNLP时遇到的问题
|
||||
title: "[Question]: "
|
||||
labels:
|
||||
- question
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: >
|
||||
#### 你可以在这里提出一个使用/咨询问题,提问之前请确保:
|
||||
|
||||
- 1)已经百度/谷歌搜索过你的问题,但是没有找到解答;
|
||||
|
||||
- 2)已经在官网查询过[API文档](https://www.paddlepaddle.org.cn/documentation/docs/zh/api/index_cn.html)与[FAQ](https://www.paddlepaddle.org.cn/documentation/docs/zh/faq/index_cn.html),但是没有找到解答;
|
||||
|
||||
- 3)已经在[历史issue](https://github.com/PaddlePaddle/Paddle/issues)中搜索过,没有找到同类issue或issue未被解答。
|
||||
|
||||
- type: textarea
|
||||
id: question
|
||||
attributes:
|
||||
label: 请提出你的问题
|
||||
validations:
|
||||
required: true
|
||||
@@ -0,0 +1,45 @@
|
||||
name: 🐛 Bug Report
|
||||
description: PaddleNLP问题反馈
|
||||
title: "[Bug]: "
|
||||
labels: bug
|
||||
body:
|
||||
- type: textarea
|
||||
id: environment
|
||||
attributes:
|
||||
label: 软件环境
|
||||
description: |
|
||||
请使用以下命令给出您本地Paddle相关包信息
|
||||
```sh
|
||||
pip list | grep paddle
|
||||
|
||||
```
|
||||
value: |
|
||||
- paddlepaddle:
|
||||
- paddlepaddle-gpu:
|
||||
- paddlenlp:
|
||||
render: Markdown
|
||||
validations:
|
||||
required: true
|
||||
- type: checkboxes
|
||||
id: dumplicated-problem
|
||||
attributes:
|
||||
label: 重复问题
|
||||
description: 是否已在issues中搜索相关问题
|
||||
options:
|
||||
- label: I have searched the existing issues
|
||||
required: true
|
||||
- type: textarea
|
||||
id: descripton
|
||||
attributes:
|
||||
label: 错误描述
|
||||
description: 给出错误详细描述,以便能够更好的追踪相关问题
|
||||
render: Markdown
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: mvp-code
|
||||
attributes:
|
||||
label: 稳定复现步骤 & 代码
|
||||
description: 请给出稳定复现该问题的步骤 & 代码,以便相关人员能够快速定位到具体问题。
|
||||
validations:
|
||||
required: true
|
||||
@@ -0,0 +1,32 @@
|
||||
name: 🐛 Docs Report
|
||||
description: PaddleNLP文档反馈
|
||||
title: "[Docs]: "
|
||||
labels:
|
||||
- documentation
|
||||
|
||||
body:
|
||||
- type: textarea
|
||||
id: environment
|
||||
attributes:
|
||||
label: 软件环境
|
||||
description: |
|
||||
请使用以下命令给出您本地Paddle相关包信息
|
||||
```sh
|
||||
pip list | grep paddle
|
||||
|
||||
```
|
||||
value: |
|
||||
- paddlepaddle:
|
||||
- paddlepaddle-gpu:
|
||||
- paddlenlp:
|
||||
render: Markdown
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: description
|
||||
attributes:
|
||||
label: 详细描述
|
||||
description: 请详细描述您想要反馈的具体问题
|
||||
render: Markdown
|
||||
validations:
|
||||
required: true
|
||||
@@ -0,0 +1,30 @@
|
||||
name: "\U0001F680 Feature request"
|
||||
description: 请详细描述您所需功能
|
||||
labels: [ "feature" ]
|
||||
body:
|
||||
- type: textarea
|
||||
id: feature-request
|
||||
validations:
|
||||
required: true
|
||||
attributes:
|
||||
label: Feature request
|
||||
description: |
|
||||
对特性提案的清晰而简明的描述。如果论文和代码存在,请提供链接。
|
||||
|
||||
- type: textarea
|
||||
id: motivation
|
||||
validations:
|
||||
required: true
|
||||
attributes:
|
||||
label: Motivation
|
||||
description: |
|
||||
请概述这项建议的动机。您的特性要求与问题有关吗?
|
||||
|
||||
- type: textarea
|
||||
id: contribution
|
||||
validations:
|
||||
required: true
|
||||
attributes:
|
||||
label: Your contribution
|
||||
description: |
|
||||
Is there any way that you could help, e.g. by submitting a PR? Make sure to read the CONTRIBUTING.MD [readme](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md)
|
||||
@@ -0,0 +1,28 @@
|
||||
name: "\U0001F31F 添加新模型"
|
||||
description: 请为新模型提交一份说明
|
||||
labels: [ "New model" ]
|
||||
|
||||
body:
|
||||
- type: textarea
|
||||
id: description-request
|
||||
validations:
|
||||
required: true
|
||||
attributes:
|
||||
label: 简要描述
|
||||
description: |
|
||||
请简要描述模型的类型、解决的问题等。
|
||||
|
||||
- type: checkboxes
|
||||
id: information-tasks
|
||||
attributes:
|
||||
label: 是否已开源
|
||||
options:
|
||||
- label: 已开源
|
||||
- label: 未开源
|
||||
|
||||
- type: textarea
|
||||
id: additional-info
|
||||
attributes:
|
||||
label: 模型详细信息
|
||||
description: |
|
||||
请给出新模型相关信息,如论文地址、现存代码地址等。
|
||||
@@ -0,0 +1,23 @@
|
||||
name: 🧩 其他 Others
|
||||
description: 提出其他问题。
|
||||
labels: [others]
|
||||
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: >
|
||||
#### 你可以在这里提出任何前面几类模板不适用的问题,包括但不限于:优化性建议、框架使用体验反馈、版本兼容性问题、报错信息不清楚等。
|
||||
|
||||
- type: textarea
|
||||
id: others
|
||||
attributes:
|
||||
label: 问题描述
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: >
|
||||
感谢你的贡献 🎉!
|
||||
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
<!-- Demo: https://github.com/PaddlePaddle/PaddleNLP/pull/26 -->
|
||||
#### Before submitting
|
||||
|
||||
- [ ] Lint code. If there are lint issues, please format the code first.
|
||||
|
||||
```shell
|
||||
# Install and register `pre-commit` in the project folder
|
||||
pip install pre-commit && pre-commit install
|
||||
|
||||
# Process previous code files separately
|
||||
pre-commit run --file XXXX.py
|
||||
```
|
||||
|
||||
- [ ] Add test cases into `tests` folder. If there are codecov issues, please add tests cases first.
|
||||
|
||||
### PR types
|
||||
<!-- One of [ New features | Bug fixes | Function optimization | Performance optimization | Breaking changes | Others ] -->
|
||||
|
||||
### PR changes
|
||||
<!-- One of [ Models | APIs | Docs | Others ] -->
|
||||
|
||||
### Description
|
||||
<!-- Describe what this PR does -->
|
||||
@@ -0,0 +1,41 @@
|
||||
name: "Check bypass"
|
||||
description: "A custom action to encapsulate PFCCLab/ci-bypass"
|
||||
inputs:
|
||||
github-token:
|
||||
description: "GitHub token"
|
||||
required: true
|
||||
workflow-name:
|
||||
description: "Workflow name"
|
||||
required: true
|
||||
outputs:
|
||||
can-skip:
|
||||
description: "Whether the workflow can be skipped."
|
||||
value: ${{ steps.check-bypass.outputs.can-skip }}
|
||||
|
||||
runs:
|
||||
using: "composite"
|
||||
steps:
|
||||
- id: check-bypass
|
||||
name: Check Bypass
|
||||
env:
|
||||
CI_TEAM_MEMBERS: '["tianshuo78520a", "swgu98", "risemeup1", "XieYunshen","luotao1","From00"]'
|
||||
uses: PFCCLab/ci-bypass@v1
|
||||
with:
|
||||
github-token: ${{ inputs.github-token }}
|
||||
non-pull-request-event-strategy: 'never-skipped'
|
||||
type: 'composite'
|
||||
composite-rule: |
|
||||
{
|
||||
"any": [
|
||||
{
|
||||
"type": "labeled",
|
||||
"label": ["skip-ci: ${{ inputs.workflow-name }}", "skip-ci: all"],
|
||||
"username": ${{ env.CI_TEAM_MEMBERS }}
|
||||
},
|
||||
{
|
||||
"type": "commented",
|
||||
"comment-pattern": [".*/skip-ci ${{ inputs.workflow-name }}.*", ".*/skip-ci all.*"],
|
||||
"username": ${{ env.CI_TEAM_MEMBERS }}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
name: 'Rerun Workflow'
|
||||
description: 'Re-run GitHub Actions workflow for a given Pull Request'
|
||||
inputs:
|
||||
GITHUB_TOKEN:
|
||||
description: 'GitHub token with repo scope'
|
||||
required: true
|
||||
OWNER:
|
||||
description: 'Repository owner'
|
||||
required: true
|
||||
REPO:
|
||||
description: 'Repository name'
|
||||
required: true
|
||||
PR_ID:
|
||||
description: 'Pull Request ID'
|
||||
required: true
|
||||
JOB_NAME:
|
||||
description: 'Job name to rerun'
|
||||
required: true
|
||||
|
||||
runs:
|
||||
using: 'composite'
|
||||
steps:
|
||||
- run: bash ./.github/actions/rerun-workflow/rerun.sh
|
||||
shell: bash
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ inputs.GITHUB_TOKEN }}
|
||||
OWNER: ${{ inputs.OWNER }}
|
||||
REPO: ${{ inputs.REPO }}
|
||||
PR_ID: ${{ inputs.PR_ID }}
|
||||
JOB_NAME: ${{ inputs.JOB_NAME }}
|
||||
@@ -0,0 +1,77 @@
|
||||
# Copyright (c) 2025 PaddleNLP Authors. All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
set -e
|
||||
|
||||
COMMIT_SHA=$(curl -s -H "Authorization: token $GITHUB_TOKEN" \
|
||||
"https://api.github.com/repos/$OWNER/$REPO/pulls/$PR_ID" | jq -r '.head.sha')
|
||||
|
||||
echo "Commit SHA: $COMMIT_SHA"
|
||||
|
||||
response=$(curl -s -H "Authorization: token $GITHUB_TOKEN" \
|
||||
"https://api.github.com/repos/$OWNER/$REPO/actions/runs?head_sha=$COMMIT_SHA&per_page=100")
|
||||
|
||||
echo "Response: $response"
|
||||
|
||||
run_ids=$(echo "$response" | jq -r '.workflow_runs[].id')
|
||||
|
||||
if [ -n "$run_ids" ]; then
|
||||
echo "Found run_ids for commit $COMMIT_SHA: $run_ids"
|
||||
|
||||
for run_id in $run_ids; do
|
||||
if [ "$JOB_NAME" = "all-failed" ]; then
|
||||
echo "Rerunning all failed jobs for run_id: $run_id"
|
||||
|
||||
rerun_response=$(curl -X POST -s -w "%{http_code}" -o /dev/null \
|
||||
-H "Accept: application/vnd.github.v3+json" \
|
||||
-H "Authorization: Bearer $GITHUB_TOKEN" \
|
||||
"https://api.github.com/repos/$OWNER/$REPO/actions/runs/$run_id/rerun-failed-jobs")
|
||||
if [ "$rerun_response" -eq 201 ]; then
|
||||
echo "Successfully requested rerun for all blocked jobs in run_id: $run_id"
|
||||
else
|
||||
echo "Failed to request rerun for run_id: $run_id with status code $rerun_response"
|
||||
fi
|
||||
|
||||
else
|
||||
jobs_response=$(curl -s -H "Authorization: token $GITHUB_TOKEN" \
|
||||
"https://api.github.com/repos/$OWNER/$REPO/actions/runs/$run_id/jobs")
|
||||
|
||||
echo "Jobs Response for run_id $run_id: $jobs_response"
|
||||
|
||||
# if [[ "$JOB_NAME" == *"bypass"* ]]; then
|
||||
block_jobs=$(echo "$jobs_response" | jq -r --arg job_name "$JOB_NAME" \
|
||||
'.jobs[] | select(.name == $job_name) | .id')
|
||||
# else
|
||||
# block_jobs=$(echo "$jobs_response" | jq -r --arg job_name "$JOB_NAME" \
|
||||
# '.jobs[] | select(.name == $job_name and .conclusion != "success") | .id')
|
||||
# fi
|
||||
|
||||
if [ -n "$block_jobs" ]; then
|
||||
echo "Found block jobs for run_id $run_id: $block_jobs"
|
||||
|
||||
for job_id in $block_jobs; do
|
||||
echo "Rerunning job_id: $job_id"
|
||||
curl -X POST -H "Accept: application/vnd.github.v3+json" \
|
||||
-H "Authorization: token $GITHUB_TOKEN" \
|
||||
"https://api.github.com/repos/$OWNER/$REPO/actions/jobs/$job_id/rerun"
|
||||
done
|
||||
else
|
||||
echo "No block jobs found for run_id $run_id with name $JOB_NAME."
|
||||
fi
|
||||
fi
|
||||
done
|
||||
else
|
||||
echo "No matching workflow runs found for commit $COMMIT_SHA."
|
||||
exit 1
|
||||
fi
|
||||
@@ -0,0 +1,13 @@
|
||||
codecov:
|
||||
notify:
|
||||
wait_for_ci: false
|
||||
|
||||
coverage:
|
||||
status:
|
||||
project:
|
||||
default:
|
||||
target: 58% # overall project Coverage
|
||||
threshold: 1% # Allow the coverage to drop by 1%, and posting a success status.
|
||||
patch:
|
||||
default:
|
||||
target: 80% # lines adjusted Coverage < 80% CI will fail
|
||||
@@ -0,0 +1,47 @@
|
||||
name: PaddleFormers PR Check
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [opened, synchronize]
|
||||
|
||||
env:
|
||||
BRANCH: ${{ github.base_ref }}
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
PR_ID: ${{ github.event.pull_request.number }}
|
||||
COMMIT_ID: ${{ github.event.pull_request.head.sha }}
|
||||
token: ${{ vars.ACTION_GITHUB_TOKEN }}
|
||||
|
||||
jobs:
|
||||
check-approvers:
|
||||
name: Check approval
|
||||
runs-on:
|
||||
group: APPROVAL
|
||||
steps:
|
||||
- name: Cleanup
|
||||
run: |
|
||||
rm -rf * .[^.]*
|
||||
- name: Update paddle
|
||||
run: |
|
||||
wget -q --no-proxy https://xly-devops.bj.bcebos.com/PaddleTest/PaddleNLP/PaddleNLP-develop.tar.gz --no-check-certificate
|
||||
tar zxf PaddleNLP-develop.tar.gz --strip-components=1 >/dev/null
|
||||
rm -rf PaddleNLP-develop.tar.gz >/dev/null
|
||||
git fetch origin pull/${PR_ID}/head
|
||||
git checkout -b origin_pr FETCH_HEAD
|
||||
git remote add upstream https://github.com/PaddlePaddle/PaddleNLP.git
|
||||
git fetch upstream
|
||||
git checkout -b test_pr upstream/${BRANCH}
|
||||
git merge --no-edit origin_pr
|
||||
git log --pretty=oneline -10
|
||||
|
||||
- name: Check bypass
|
||||
id: check-bypass
|
||||
uses: ./.github/actions/check-bypass
|
||||
with:
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
workflow-name: approval
|
||||
|
||||
- name: Display Required Approvers
|
||||
if: steps.check-bypass.outputs.can-skip != 'true'
|
||||
run: |
|
||||
cd scripts/ci_approval
|
||||
bash run_ci_approval_cherry-pick.sh
|
||||
@@ -0,0 +1,47 @@
|
||||
name: Approval
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [opened, synchronize]
|
||||
|
||||
env:
|
||||
BRANCH: ${{ github.base_ref }}
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
PR_ID: ${{ github.event.pull_request.number }}
|
||||
COMMIT_ID: ${{ github.event.pull_request.head.sha }}
|
||||
token: ${{ vars.ACTION_GITHUB_TOKEN }}
|
||||
|
||||
jobs:
|
||||
check-approvers:
|
||||
name: Check approval
|
||||
runs-on:
|
||||
group: APPROVAL
|
||||
steps:
|
||||
- name: Cleanup
|
||||
run: |
|
||||
rm -rf * .[^.]*
|
||||
- name: Update paddle
|
||||
run: |
|
||||
wget -q --no-proxy https://xly-devops.bj.bcebos.com/PaddleTest/PaddleNLP/PaddleNLP-develop.tar.gz --no-check-certificate
|
||||
tar zxf PaddleNLP-develop.tar.gz --strip-components=1 >/dev/null
|
||||
rm -rf PaddleNLP-develop.tar.gz >/dev/null
|
||||
git fetch origin pull/${PR_ID}/head
|
||||
git checkout -b origin_pr FETCH_HEAD
|
||||
git remote add upstream https://github.com/PaddlePaddle/PaddleNLP.git
|
||||
git fetch upstream
|
||||
git checkout -b test_pr upstream/${BRANCH}
|
||||
git merge --no-edit origin_pr
|
||||
git log --pretty=oneline -10
|
||||
|
||||
- name: Check bypass
|
||||
id: check-bypass
|
||||
uses: ./.github/actions/check-bypass
|
||||
with:
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
workflow-name: approval
|
||||
|
||||
- name: Display Required Approvers
|
||||
if: steps.check-bypass.outputs.can-skip != 'true'
|
||||
run: |
|
||||
cd scripts/ci_approval
|
||||
bash -x run_ci_approval.sh
|
||||
@@ -0,0 +1,52 @@
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
workflow-name:
|
||||
required: true
|
||||
type: string
|
||||
secrets:
|
||||
github-token:
|
||||
required: true
|
||||
outputs:
|
||||
can-skip:
|
||||
description: "Whether the workflow can be skipped."
|
||||
value: ${{ jobs.check-bypass.outputs.can-skip }}
|
||||
|
||||
jobs:
|
||||
check-bypass:
|
||||
name: Check bypass
|
||||
runs-on:
|
||||
group: APPROVAL
|
||||
permissions:
|
||||
contents: read
|
||||
env:
|
||||
CI_TEAM_MEMBERS: '["tianshuo78520a", "swgu98", "risemeup1" , "XieYunshen","luotao1","From00"]'
|
||||
outputs:
|
||||
can-skip: ${{ steps.check-bypass.outputs.can-skip }}
|
||||
steps:
|
||||
- name: Cleanup
|
||||
run: |
|
||||
rm -rf * .[^.]*
|
||||
|
||||
- id: check-bypass
|
||||
name: Check Bypass
|
||||
uses: PFCCLab/ci-bypass@v1
|
||||
with:
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
non-pull-request-event-strategy: 'never-skipped'
|
||||
type: 'composite'
|
||||
composite-rule: |
|
||||
{
|
||||
"any": [
|
||||
{
|
||||
"type": "labeled",
|
||||
"label": ["skip-ci: ${{ inputs.workflow-name }}", "skip-ci: all"],
|
||||
"username": ${{ env.CI_TEAM_MEMBERS }}
|
||||
},
|
||||
{
|
||||
"type": "commented",
|
||||
"comment-pattern": [".*/skip-ci ${{ inputs.workflow-name }}.*", ".*/skip-ci all.*"],
|
||||
"username": ${{ env.CI_TEAM_MEMBERS }}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
name: Codestyle Check
|
||||
|
||||
on: [push, pull_request]
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
|
||||
cancel-in-progress: true
|
||||
|
||||
env:
|
||||
PR_ID: ${{ github.event.pull_request.number }}
|
||||
COMMIT_ID: ${{ github.event.pull_request.head.sha }}
|
||||
BRANCH: ${{ github.event.pull_request.base.ref }}
|
||||
TASK: PaddleNLP-CI-Lint-${{ github.event.pull_request.number }}
|
||||
|
||||
jobs:
|
||||
check-bypass:
|
||||
name: Check bypass
|
||||
uses: ./.github/workflows/check-bypass.yml
|
||||
with:
|
||||
workflow-name: 'lint'
|
||||
secrets:
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
Lint:
|
||||
name: Lint
|
||||
needs: check-bypass
|
||||
if: ${{ needs.check-bypass.outputs.can-skip != 'true' }}
|
||||
runs-on: [self-hosted, ernie-cpu]
|
||||
steps:
|
||||
- name: Run Container
|
||||
env:
|
||||
work_dir: ${{ github.workspace }}
|
||||
python_version: "3.10"
|
||||
run: |
|
||||
container_name=${TASK}-$(date +%Y%m%d-%H%M%S)
|
||||
echo "container_name=${container_name}" >> ${{ github.env }}
|
||||
docker_image="iregistry.baidu-int.com/paddlecloud/base-images:paddlecloud-ubuntu20.04-gcc12.2-cuda12.3-cudnn9.0-nccl2.20.3.1-openmpi4.1.5-latest"
|
||||
docker run -d -t --name ${container_name} --net=host -v /dev/shm:/dev/shm --shm-size=32G \
|
||||
-v $work_dir/../../..:$work_dir/../../.. \
|
||||
-v $work_dir:/workspace \
|
||||
-v /home/.cache/pip:/home/.cache/pip \
|
||||
-e BRANCH \
|
||||
-e PR_ID \
|
||||
-e COMMIT_ID \
|
||||
-e work_dir \
|
||||
-e no_proxy \
|
||||
-e python_version \
|
||||
-w /workspace ${docker_image}
|
||||
|
||||
- name: Download Code
|
||||
env:
|
||||
work_dir: ${{ github.workspace }}
|
||||
run: |
|
||||
docker exec -t ${container_name} /bin/bash -c '
|
||||
rm -rf * .[^.]*
|
||||
echo "Downloading PaddleNLP.tar"
|
||||
wget -q --no-proxy https://paddle-qa.bj.bcebos.com/CodeSync/develop/PaddleNLP.tar --no-check-certificate
|
||||
echo "Extracting PaddleNLP.tar"
|
||||
tar xf PaddleNLP.tar && rm -rf PaddleNLP.tar
|
||||
source $work_dir/../../../proxy
|
||||
cd PaddleNLP
|
||||
git config --global user.name "PaddleCI"
|
||||
git config --global user.email "paddle_ci@example.com"
|
||||
git pull
|
||||
git submodule update --init --recursive --force
|
||||
if [ -n "${PR_ID}" ]; then
|
||||
git fetch origin pull/${PR_ID}/head
|
||||
git checkout -b PR_${PR_ID} FETCH_HEAD
|
||||
git remote add upstream https://github.com/PaddlePaddle/PaddleNLP.git
|
||||
git fetch upstream ${BRANCH}
|
||||
git merge ${BRANCH} --no-edit
|
||||
git diff --numstat ${BRANCH} -- | awk "{print \$NF}"
|
||||
else
|
||||
echo "Not in a pull_request event. Skipping PR-specific operations."
|
||||
fi
|
||||
git log --pretty=oneline -10
|
||||
|
||||
if ! git show-ref --quiet refs/heads/develop; then \
|
||||
echo "local develop branch is missing, creating local develop branch that tracks remote develop branch"
|
||||
git fetch origin develop
|
||||
git branch develop --track origin/develop
|
||||
else
|
||||
echo "local develop branch exist, skipping"
|
||||
fi
|
||||
'
|
||||
|
||||
- name: Setup Environment
|
||||
run: |
|
||||
docker exec -t $container_name /bin/bash -c '
|
||||
unlink /usr/local/bin/python
|
||||
ln -sf $(which python${python_version}) /usr/local/bin/python
|
||||
set -e
|
||||
python -m pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
|
||||
python -m pip config set global.cache-dir "/home/.cache/pip"
|
||||
source $work_dir/../../../proxy
|
||||
python -m pip install --upgrade pip
|
||||
cd /workspace/PaddleNLP && git config --global --add safe.directory $PWD
|
||||
make install
|
||||
'
|
||||
|
||||
- name: Test
|
||||
run: |
|
||||
docker exec -t $container_name /bin/bash -c '
|
||||
set -e
|
||||
source $work_dir/../../../proxy
|
||||
cd /workspace/PaddleNLP
|
||||
make lint
|
||||
'
|
||||
|
||||
- name: Terminate And Delete the Container
|
||||
if: always()
|
||||
run: |
|
||||
docker rm -f $container_name 2>/dev/null || true
|
||||
@@ -0,0 +1,207 @@
|
||||
# Disabled LLM CI workflow - manually enabled when needed
|
||||
name: LLM CI (DISABLED)
|
||||
|
||||
#on:
|
||||
# pull_request:
|
||||
# types: [opened, synchronize, reopened]
|
||||
# branches: [develop]
|
||||
# schedule:
|
||||
# - cron: "2 0 * * *"
|
||||
# workflow_call:
|
||||
# inputs:
|
||||
# run_downstream:
|
||||
# required: true
|
||||
# type: string
|
||||
# image_name:
|
||||
# required: true
|
||||
# type: string
|
||||
|
||||
#concurrency:
|
||||
# group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
|
||||
# cancel-in-progress: true
|
||||
|
||||
#env:
|
||||
# PR_ID: ${{ github.event.pull_request.number }}
|
||||
COMMIT_ID: ${{ github.event.pull_request.head.sha }}
|
||||
TASK: paddlenlp-CI-${{ github.event.pull_request.number }}-llm
|
||||
ci_scripts: /workspace/PaddleNLP/scripts/regression
|
||||
BRANCH: ${{ github.event.pull_request.base.ref }}
|
||||
AGILE_COMPILE_BRANCH: ${{ github.event.pull_request.base.ref }}
|
||||
CI_name: llm-ci
|
||||
no_proxy: "localhost,bj.bcebos.com,su.bcebos.com,bcebos.com,apiin.im.baidu.com,gitee.com,aliyun.com,.baidu.com,.tuna.tsinghua.edu.cn"
|
||||
HF_ENDPOINT: https://hf-mirror.com
|
||||
STUDIO_GIT_HOST: http://git.prod.idc-to-cloud.aistudio.baidu-int.com
|
||||
PPNLP_HOME: /ssd1/paddlenlp
|
||||
HF_DATASETS_CACHE: /ssd1/paddlenlp/huggingface/datasets
|
||||
TRANSFORMERS_CACHE: /ssd1/paddlenlp/huggingface
|
||||
CCACHE_DIR: /home/data/gzcfs/.ccache/gpubox
|
||||
RUN_DOWNSTREAM: ${{ inputs.run_downstream }}
|
||||
|
||||
#defaults:
|
||||
# run:
|
||||
# shell: bash
|
||||
|
||||
#jobs:
|
||||
# llm-ci:
|
||||
# name: llm-ci (DISABLED)
|
||||
runs-on: [self-hosted, ernie-8gpu]
|
||||
steps:
|
||||
- name: Determine Image Name
|
||||
env:
|
||||
IMAGE_NAME: ${{ inputs.image_name }}
|
||||
run: |
|
||||
if [[ -n "${IMAGE_NAME}" ]]; then
|
||||
echo "IMAGE_NAME=${IMAGE_NAME}" >> "$GITHUB_ENV"
|
||||
else
|
||||
echo "IMAGE_NAME=iregistry.baidu-int.com/paddlecloud/base-images:paddlecloud-ubuntu18.04-gcc8.2-cuda11.8-cudnn8.6-nccl2.15.5-paddlenlp-latest" >> "$GITHUB_ENV"
|
||||
fi
|
||||
|
||||
- name: Run Container
|
||||
env:
|
||||
work_dir: ${{ github.workspace }}
|
||||
CACHE_DIR: /home/data/cfs/.cache
|
||||
FLAGS_dynamic_static_unified_comm: "True"
|
||||
python_version: "3.10"
|
||||
paddle_whl: https://paddle-qa.bj.bcebos.com/paddle-pipeline/Develop-GpuSome-LinuxCentos-Gcc82-Cuda118-Cudnn86-Trt85-Py310-CINN-Compile/latest/paddlepaddle_gpu-0.0.0-cp310-cp310-linux_x86_64.whl
|
||||
run: |
|
||||
container_name=${TASK}-$(date +%Y%m%d-%H%M%S)
|
||||
echo "container_name=${container_name}" >> "$GITHUB_ENV"
|
||||
if [[ "$RUN_DOWNSTREAM" == "false" ]]; then
|
||||
echo "Not in a pull_request or test_build event. Skipping..."
|
||||
else
|
||||
docker run -d -t --name ${container_name} --net=host -v /dev/shm:/dev/shm --shm-size=32G \
|
||||
-v $work_dir/../../..:$work_dir/../../.. \
|
||||
-v $work_dir:/workspace \
|
||||
-v /home/.cache/pip:/home/.cache/pip \
|
||||
-v /ssd1/paddlenlp:/ssd1/paddlenlp \
|
||||
-v /home/data/gzcfs/.ccache/gpubox:/home/data/gzcfs/.ccache/gpubox \
|
||||
-e BRANCH \
|
||||
-e AGILE_COMPILE_BRANCH \
|
||||
-e PR_ID \
|
||||
-e COMMIT_ID \
|
||||
-e work_dir \
|
||||
-e ci_scripts \
|
||||
-e no_proxy \
|
||||
-e CI_name \
|
||||
-e paddle_whl \
|
||||
-e HF_ENDPOINT \
|
||||
-e STUDIO_GIT_HOST \
|
||||
-e PPNLP_HOME \
|
||||
-e HF_DATASETS_CACHE \
|
||||
-e TRANSFORMERS_CACHE \
|
||||
-e CACHE_DIR \
|
||||
-e FLAGS_dynamic_static_unified_comm \
|
||||
-e python_version \
|
||||
-w /workspace --runtime=nvidia $IMAGE_NAME
|
||||
fi
|
||||
|
||||
- name: Download Code
|
||||
env:
|
||||
work_dir: ${{ github.workspace }}
|
||||
run: |
|
||||
if [[ "$RUN_DOWNSTREAM" == "false" ]]; then
|
||||
echo "Not in a pull_request or test_build event. Skipping.."
|
||||
else
|
||||
docker exec -t $container_name /bin/bash -c '
|
||||
rm -rf * .[^.]*
|
||||
echo "Downloading PaddleNLP.tar.gz"
|
||||
wget -q --no-proxy https://paddle-qa.bj.bcebos.com/CodeSync/develop/PaddleNLP.tar --no-check-certificate
|
||||
echo "Extracting PaddleNLP.tar.gz"
|
||||
tar xf PaddleNLP.tar && rm -rf PaddleNLP.tar
|
||||
source $work_dir/../../../proxy
|
||||
cd PaddleNLP
|
||||
git config --global user.name "PaddleCI"
|
||||
git config --global user.email "paddle_ci@example.com"
|
||||
git pull
|
||||
git submodule update --init --recursive --force
|
||||
if [ -n "${PR_ID}" ]; then
|
||||
git fetch origin pull/${PR_ID}/head
|
||||
git checkout -b PR_${PR_ID} FETCH_HEAD
|
||||
git remote add upstream https://github.com/PaddlePaddle/PaddleNLP.git
|
||||
git fetch upstream ${BRANCH}
|
||||
git merge ${BRANCH} --no-edit
|
||||
git diff --numstat ${BRANCH} -- | awk "{print \$NF}"
|
||||
else
|
||||
echo "Not in a pull_request event. Skipping PR-specific operations."
|
||||
fi
|
||||
git log --pretty=oneline -10
|
||||
'
|
||||
fi
|
||||
|
||||
- name: Skip For Bug
|
||||
run: |
|
||||
if [[ "$RUN_DOWNSTREAM" == "false" ]]; then
|
||||
echo "Not in a pull_request or test_build event. Skipping..."
|
||||
else
|
||||
docker exec -t $container_name /bin/bash -c '
|
||||
cd /workspace/PaddleNLP
|
||||
git revert f2477c07272d04244cd3287d1f21c70482a4a85f --no-edit # 套件PR#10413引入bug-待修复
|
||||
git revert d74c950e15a35b7a100d1688c89318195cc83bca --no-edit # 框架升级后配合修改,当前因框架bug固定commit适配
|
||||
'
|
||||
fi
|
||||
|
||||
- name: Test
|
||||
run: |
|
||||
if [[ "$RUN_DOWNSTREAM" == "false" ]]; then
|
||||
echo "Not in a pull_request or test_build event. Skipping..."
|
||||
else
|
||||
docker exec -t $container_name /bin/bash -c '
|
||||
ldconfig
|
||||
unlink /usr/bin/python3
|
||||
ln -sf $(which python${python_version}) /usr/bin/python3
|
||||
pip config set global.cache-dir "/home/.cache/pip"
|
||||
set -e
|
||||
source $work_dir/../../../proxy
|
||||
cd /workspace/PaddleNLP && git config --global --add safe.directory $PWD
|
||||
echo "Paddle PR#73283 PR#74484 import bug, set paddle_commit=8ae742 to skip "
|
||||
export paddle_whl=https://paddle-qa.bj.bcebos.com/paddle-pipeline/Develop-GpuSome-LinuxCentos-Gcc82-Cuda118-Cudnn86-Trt85-Py310-CINN-Compile/8ae7423e99b2ea96e410968a0ebb3f1795e37205/paddlepaddle_gpu-0.0.0-cp310-cp310-linux_x86_64.whl
|
||||
timeout 2h bash scripts/regression/run_ci.sh python${python_version} ${paddle_whl}
|
||||
'
|
||||
fi
|
||||
|
||||
- name: Upload Allure-reports & Logs
|
||||
if: always()
|
||||
env:
|
||||
home_path: ${{ github.workspace }}/../../..
|
||||
bos_file: ${{ github.workspace }}/../../../bos/BosClient.py
|
||||
allure_file: ${{ github.workspace }}/../../../allure-2.19.0/bin/allure
|
||||
run: |
|
||||
if [[ "$RUN_DOWNSTREAM" == "false" ]]; then
|
||||
echo "Not in a pull_request or test_build event. Skipping..."
|
||||
else
|
||||
docker exec -t $container_name /bin/bash -c '
|
||||
unset http_proxy && unset https_proxy
|
||||
if [ ! -f "${{ env.bos_file }}" ]; then
|
||||
wget -q --no-proxy -O ${{ env.home_path }}/bos_new.tar.gz https://xly-devops.bj.bcebos.com/home/bos_new.tar.gz --no-check-certificate
|
||||
mkdir ${{ env.home_path }}/bos
|
||||
tar xf ${{ env.home_path }}/bos_new.tar.gz -C ${{ env.home_path }}/bos
|
||||
fi
|
||||
if [ ! -f "${{ env.allure_file }}" ]; then
|
||||
wget -q --no-proxy -O ${{ env.home_path }}/allure-2.19.0.zip https://xly-devops.bj.bcebos.com/tools/allure-2.19.0.zip --no-check-certificate
|
||||
unzip -q ${{ env.home_path }}/allure-2.19.0.zip -d ${{ env.home_path }}/
|
||||
fi
|
||||
if [[ "${{ env.RUN_DOWNSTREAM }}" == "" && -n "${PR_ID}" ]]; then
|
||||
bos_prefix="${PR_ID}/${COMMIT_ID}"
|
||||
elif [[ "${{ env.RUN_DOWNSTREAM }}" == "true" && -n "${PR_ID}" ]]; then
|
||||
bos_prefix="${PR_ID}/${COMMIT_ID}/test_build"
|
||||
else
|
||||
bos_prefix="schedule/$(date +%Y%m%d)"
|
||||
fi
|
||||
cd /workspace/PaddleNLP/model_logs
|
||||
for FILE in /workspace/PaddleNLP/model_logs/*; do
|
||||
file=$(basename "$FILE")
|
||||
python ${{ env.bos_file }} $file paddle-github-action/PR/PaddleNLP/llm/${bos_prefix}/logs
|
||||
echo "$file: https://paddle-github-action.bj.bcebos.com/PR/PaddleNLP/llm/${bos_prefix}/logs/$file"
|
||||
done
|
||||
cd /workspace/PaddleNLP/
|
||||
${{ env.allure_file }} generate result -o report
|
||||
tar -czf products.tar.gz report model_logs
|
||||
python ${{ env.bos_file }} products.tar.gz paddle-github-action/PR/PaddleNLP/llm/${bos_prefix}/logs
|
||||
echo "products: https://paddle-github-action.bj.bcebos.com/PR/PaddleNLP/llm/${bos_prefix}/logs/products.tar.gz"
|
||||
'
|
||||
fi
|
||||
|
||||
- name: Terminate And Delete the Container
|
||||
if: always()
|
||||
run: |
|
||||
docker rm -f ${{ env.container_name }} 2>/dev/null || true
|
||||
@@ -0,0 +1,30 @@
|
||||
name: Pipelines-Test
|
||||
|
||||
on:
|
||||
push:
|
||||
paths:
|
||||
- 'slm/pipelines/*'
|
||||
pull_request:
|
||||
paths:
|
||||
- 'slm/pipelines/*'
|
||||
|
||||
|
||||
jobs:
|
||||
Pipelines-Test:
|
||||
name: Pipelines-Test
|
||||
runs-on: ubuntu-24.04
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: '3.10'
|
||||
cache: 'pip' # caching pip dependencies
|
||||
- name: Install dependencies
|
||||
working-directory: ./slm/pipelines
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
make install
|
||||
pip install -r tests/requirements.txt
|
||||
- name: run the command
|
||||
working-directory: ./slm/pipelines
|
||||
run: make test
|
||||
@@ -0,0 +1,58 @@
|
||||
name: Release
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
package:
|
||||
description: 'Package to be deployed'
|
||||
required: true
|
||||
default: 'paddlenlp'
|
||||
type: choice
|
||||
options:
|
||||
- paddlenlp
|
||||
- paddle-pipelines
|
||||
- ppdiffusers
|
||||
|
||||
jobs:
|
||||
Release:
|
||||
name: Release
|
||||
runs-on: ubuntu-24.04
|
||||
steps:
|
||||
- uses: actions/checkout@v2
|
||||
- uses: actions/setup-python@v1
|
||||
with:
|
||||
python-version: '3.10'
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install setuptools wheel twine
|
||||
- name: Should Deploy
|
||||
id: should-deploy
|
||||
env:
|
||||
PACKAGE: ${{ inputs.package }}
|
||||
run: python scripts/should_deploy.py --name $PACKAGE >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Check Branch
|
||||
id: check-branch
|
||||
run: |
|
||||
if [[ ${{ github.ref }} =~ ^refs/tags/v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
|
||||
echo "match=true" >> $GITHUB_OUTPUT
|
||||
fi # See: https://stackoverflow.com/a/58869470/1123955
|
||||
|
||||
- name: Is A Publish Branch
|
||||
if: steps.check-branch.outputs.match == 'true' && steps.should-deploy.outputs.should_deploy == 'true'
|
||||
env:
|
||||
TWINE_USERNAME: __token__
|
||||
TWINE_PASSWORD: ${{ secrets.paddlenlp }}
|
||||
PACKAGE: ${{ inputs.package }}
|
||||
PADDLENLP_STABLE_VERSION: "true"
|
||||
run: |
|
||||
make deploy-$PACKAGE
|
||||
|
||||
- name: Should Not Deploy To Pypi Server
|
||||
if: steps.should-deploy.outputs.should_deploy != 'true'
|
||||
run: echo 'should not deploy pypi server'
|
||||
|
||||
- name: Is Not A Publish Branch
|
||||
if: steps.check-branch.outputs.match != 'true'
|
||||
run: echo 'Not A Publish Branch'
|
||||
@@ -0,0 +1,73 @@
|
||||
name: Re-run
|
||||
|
||||
on:
|
||||
issue_comment:
|
||||
types: [created]
|
||||
|
||||
jobs:
|
||||
re-run:
|
||||
if: ${{ github.event.issue.pull_request && contains(github.event.comment.body, '/re-run') && github.event.comment.user.login == github.event.issue.user.login }}
|
||||
runs-on: [self-hosted, ernie-cpu]
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v3
|
||||
|
||||
- name: Rerun all failed jobs
|
||||
if: ${{ contains(github.event.comment.body, 'all-failed') }}
|
||||
uses: ./.github/actions/rerun-workflow
|
||||
with:
|
||||
PR_ID: ${{ github.event.issue.number }}
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
OWNER: ${{ github.repository_owner }}
|
||||
REPO: ${{ github.event.repository.name }}
|
||||
JOB_NAME: 'all-failed'
|
||||
|
||||
- name: Rerun LLM
|
||||
if: ${{ contains(github.event.comment.body, 'LLM') }}
|
||||
uses: ./.github/actions/rerun-workflow
|
||||
with:
|
||||
PR_ID: ${{ github.event.issue.number }}
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
OWNER: ${{ github.repository_owner }}
|
||||
REPO: ${{ github.event.repository.name }}
|
||||
JOB_NAME: 'LLM CI / llm-ci'
|
||||
|
||||
- name: Unittest GPU
|
||||
if: ${{ contains(github.event.comment.body, 'Unittest GPU') }}
|
||||
uses: ./.github/actions/rerun-workflow
|
||||
with:
|
||||
PR_ID: ${{ github.event.issue.number }}
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
OWNER: ${{ github.repository_owner }}
|
||||
REPO: ${{ github.event.repository.name }}
|
||||
JOB_NAME: 'Unittest GPU CI / unittest-gpu-ci'
|
||||
|
||||
- name: Rerun Unittest CPU
|
||||
if: ${{ contains(github.event.comment.body, 'Unittest CPU') }}
|
||||
uses: ./.github/actions/rerun-workflow
|
||||
with:
|
||||
PR_ID: ${{ github.event.issue.number }}
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
OWNER: ${{ github.repository_owner }}
|
||||
REPO: ${{ github.event.repository.name }}
|
||||
JOB_NAME: 'Unittest CPU CI / unit-cpu-ci'
|
||||
|
||||
- name: Rerun Distribute (V100)
|
||||
if: ${{ contains(github.event.comment.body, 'Distribute (V100)') }}
|
||||
uses: ./.github/actions/rerun-workflow
|
||||
with:
|
||||
PR_ID: ${{ github.event.issue.number }}
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
OWNER: ${{ github.repository_owner }}
|
||||
REPO: ${{ github.event.repository.name }}
|
||||
JOB_NAME: 'Distribute CI (V100)/ distribute-v100-ci'
|
||||
|
||||
- name: Rerun Codestyle Check
|
||||
if: ${{ contains(github.event.comment.body, 'Codestyle Check') }}
|
||||
uses: ./.github/actions/rerun-workflow
|
||||
with:
|
||||
PR_ID: ${{ github.event.issue.number }}
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
OWNER: ${{ github.repository_owner }}
|
||||
REPO: ${{ github.event.repository.name }}
|
||||
JOB_NAME: 'Codestyle Check / Lint'
|
||||
@@ -0,0 +1,35 @@
|
||||
name: Stale
|
||||
|
||||
on:
|
||||
# Allow manual run via GitHub web or CLI
|
||||
workflow_dispatch:
|
||||
schedule:
|
||||
# Run daily at midnight UTC
|
||||
- cron: 0 0 * * *
|
||||
|
||||
permissions:
|
||||
issues: write
|
||||
pull-requests: write
|
||||
|
||||
jobs:
|
||||
stale:
|
||||
runs-on: ubuntu-24.04
|
||||
|
||||
permissions:
|
||||
issues: write
|
||||
pull-requests: write
|
||||
|
||||
steps:
|
||||
- uses: actions/stale@v6.0.1
|
||||
with:
|
||||
days-before-issue-stale: 60
|
||||
days-before-issue-close: 14
|
||||
stale-issue-label: "stale"
|
||||
stale-issue-message: "This issue is stale because it has been open for 60 days with no activity. 当前issue 60天内无活动,被标记为stale。"
|
||||
close-issue-message: "This issue was closed because it has been inactive for 14 days since being marked as stale. 当前issue 被标记为stale已有14天,即将关闭。"
|
||||
exempt-issue-labels: 'triage,keep'
|
||||
days-before-pr-stale: 60
|
||||
days-before-pr-close: -1
|
||||
stale-pr-label: "stale"
|
||||
stale-pr-message: "This Pull Request is stale because it has been open for 60 days with no activity. 当前Pull Request 60天内无活动,被标记为stale。"
|
||||
operations-per-run: 400
|
||||
@@ -0,0 +1,120 @@
|
||||
# Disabled unittest CPU workflow - manually enabled when needed
|
||||
name: Unittest CPU CI (DISABLED)
|
||||
|
||||
#on: [pull_request]
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
|
||||
cancel-in-progress: true
|
||||
|
||||
env:
|
||||
PR_ID: ${{ github.event.pull_request.number }}
|
||||
COMMIT_ID: ${{ github.event.pull_request.head.sha }}
|
||||
BRANCH: ${{ github.event.pull_request.base.ref }}
|
||||
TASK: PaddleNLP-CI-${{ github.event.pull_request.number }}-unittest-cpu
|
||||
HF_ENDPOINT: https://hf-mirror.com
|
||||
STUDIO_GIT_HOST: http://git.prod.idc-to-cloud.aistudio.baidu-int.com
|
||||
PPNLP_HOME: /home/disk1/cache
|
||||
HF_DATASETS_CACHE: /home/disk1/cache/huggingface/datasets
|
||||
TRANSFORMERS_CACHE: /home/disk1/cache/huggingface
|
||||
|
||||
#jobs:
|
||||
# unittest-cpu-ci:
|
||||
# name: unittest-cpu-ci (DISABLED)
|
||||
runs-on: [self-hosted, ernie-cpu]
|
||||
permissions:
|
||||
pull-requests: write
|
||||
contents: read
|
||||
id-token: write
|
||||
steps:
|
||||
- name: Run Container
|
||||
env:
|
||||
work_dir: ${{ github.workspace }}
|
||||
python_version: "3.10"
|
||||
run: |
|
||||
container_name=${TASK}-$(date +%Y%m%d-%H%M%S)
|
||||
echo "container_name=${container_name}" >> ${{ github.env }}
|
||||
docker_image="iregistry.baidu-int.com/paddlecloud/base-images:paddlecloud-ubuntu20.04-gcc12.2-cuda12.3-cudnn9.0-nccl2.20.3.1-openmpi4.1.5-latest"
|
||||
docker run -d -t --name ${container_name} --net=host -v /dev/shm:/dev/shm --shm-size=32G \
|
||||
-v $work_dir/../../..:$work_dir/../../.. \
|
||||
-v $work_dir:/workspace \
|
||||
-v /home/.cache/pip:/home/.cache/pip \
|
||||
-v /home/disk1/cache:/home/disk1/cache \
|
||||
-e BRANCH \
|
||||
-e PR_ID \
|
||||
-e COMMIT_ID \
|
||||
-e work_dir \
|
||||
-e no_proxy \
|
||||
-e python_version \
|
||||
-e HF_ENDPOINT \
|
||||
-e STUDIO_GIT_HOST \
|
||||
-e PPNLP_HOME \
|
||||
-e HF_DATASETS_CACHE \
|
||||
-e TRANSFORMERS_CACHE \
|
||||
-w /workspace ${docker_image}
|
||||
|
||||
- name: Download Code
|
||||
env:
|
||||
work_dir: ${{ github.workspace }}
|
||||
run: |
|
||||
docker exec -t ${container_name} /bin/bash -c '
|
||||
rm -rf * .[^.]*
|
||||
echo "Downloading PaddleNLP.tar"
|
||||
wget -q --no-proxy https://paddle-qa.bj.bcebos.com/CodeSync/develop/PaddleNLP.tar --no-check-certificate
|
||||
echo "Extracting PaddleNLP.tar"
|
||||
tar xf PaddleNLP.tar && rm -rf PaddleNLP.tar
|
||||
source $work_dir/../../../proxy
|
||||
cd PaddleNLP
|
||||
git config --global user.name "PaddleCI"
|
||||
git config --global user.email "paddle_ci@example.com"
|
||||
git pull
|
||||
git submodule update --init --recursive --force
|
||||
if [ -n "${PR_ID}" ]; then
|
||||
git fetch origin pull/${PR_ID}/head
|
||||
git checkout -b PR_${PR_ID} FETCH_HEAD
|
||||
git remote add upstream https://github.com/PaddlePaddle/PaddleNLP.git
|
||||
git fetch upstream ${BRANCH}
|
||||
git merge ${BRANCH} --no-edit
|
||||
git diff --numstat ${BRANCH} -- | awk "{print \$NF}"
|
||||
else
|
||||
echo "Not in a pull_request event. Skipping PR-specific operations."
|
||||
fi
|
||||
git log --pretty=oneline -10
|
||||
'
|
||||
|
||||
- name: Setup Environment
|
||||
run: |
|
||||
docker exec -t $container_name /bin/bash -c '
|
||||
unlink /usr/bin/python3
|
||||
ln -sf $(which python${python_version}) /usr/bin/python3
|
||||
set -e
|
||||
python -m pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
|
||||
python -m pip config set global.cache-dir "/home/.cache/pip"
|
||||
source $work_dir/../../../proxy
|
||||
python -m pip install --upgrade pip
|
||||
cd /workspace/PaddleNLP && git config --global --add safe.directory $PWD
|
||||
pip install -r tests/requirements.txt
|
||||
make install
|
||||
'
|
||||
|
||||
- name: Test
|
||||
run: |
|
||||
docker exec -t $container_name /bin/bash -c '
|
||||
source $work_dir/../../../proxy
|
||||
cd /workspace/PaddleNLP
|
||||
set -e
|
||||
make test
|
||||
'
|
||||
|
||||
# - name: Upload Coverage To Codecov
|
||||
# uses: codecov/codecov-action@v3
|
||||
# env:
|
||||
# CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
|
||||
# with:
|
||||
# token: ${{ secrets.CODECOV_TOKEN }}
|
||||
# files: ./PaddleNLP/coverage.xml
|
||||
|
||||
- name: Terminate And Delete the Container
|
||||
if: always()
|
||||
run: |
|
||||
docker rm -f $container_name 2>/dev/null || true
|
||||
@@ -0,0 +1,204 @@
|
||||
# Disabled unittest GPU workflow - manually enabled when needed
|
||||
name: Unittest GPU CI (DISABLED)
|
||||
|
||||
#on:
|
||||
# pull_request:
|
||||
# types: [opened, synchronize, reopened]
|
||||
# branches: [develop]
|
||||
# schedule:
|
||||
# - cron: "3 0 * * *"
|
||||
# workflow_call:
|
||||
# inputs:
|
||||
# run_downstream:
|
||||
# required: true
|
||||
# type: string
|
||||
# image_name:
|
||||
# required: true
|
||||
# type: string
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
|
||||
cancel-in-progress: true
|
||||
|
||||
env:
|
||||
PR_ID: ${{ github.event.pull_request.number }}
|
||||
COMMIT_ID: ${{ github.event.pull_request.head.sha }}
|
||||
TASK: paddlenlp-CI-${{ github.event.pull_request.number }}-unit-gpu
|
||||
ci_scripts: /workspace/PaddleNLP/scripts/unit_test
|
||||
BRANCH: ${{ github.event.pull_request.base.ref }}
|
||||
AGILE_COMPILE_BRANCH: ${{ github.event.pull_request.base.ref }}
|
||||
CI_name: unittest-gpu-ci
|
||||
no_proxy: "localhost,bj.bcebos.com,su.bcebos.com,bcebos.com,apiin.im.baidu.com,gitee.com,aliyun.com,.baidu.com,.tuna.tsinghua.edu.cn"
|
||||
HF_ENDPOINT: https://hf-mirror.com
|
||||
STUDIO_GIT_HOST: http://git.prod.idc-to-cloud.aistudio.baidu-int.com
|
||||
PPNLP_HOME: /ssd1/paddlenlp
|
||||
HF_DATASETS_CACHE: /ssd1/paddlenlp/huggingface/datasets
|
||||
TRANSFORMERS_CACHE: /ssd1/paddlenlp/huggingface
|
||||
CCACHE_DIR: /home/data/gzcfs/.ccache/gpubox
|
||||
RUN_DOWNSTREAM: ${{ inputs.run_downstream }}
|
||||
|
||||
defaults:
|
||||
run:
|
||||
shell: bash
|
||||
|
||||
#jobs:
|
||||
# unittest-gpu-ci:
|
||||
# name: unittest-gpu-ci (DISABLED)
|
||||
runs-on: [self-hosted, ernie-8gpu]
|
||||
steps:
|
||||
- name: Determine Image Name
|
||||
env:
|
||||
IMAGE_NAME: ${{ inputs.image_name }}
|
||||
run: |
|
||||
if [[ -n "${IMAGE_NAME}" ]]; then
|
||||
echo "IMAGE_NAME=${IMAGE_NAME}" >> "$GITHUB_ENV"
|
||||
else
|
||||
echo "IMAGE_NAME=iregistry.baidu-int.com/paddlecloud/base-images:paddlecloud-ubuntu18.04-gcc8.2-cuda11.8-cudnn8.6-nccl2.15.5-paddlenlp-latest" >> "$GITHUB_ENV"
|
||||
fi
|
||||
|
||||
- name: Run Container
|
||||
env:
|
||||
work_dir: ${{ github.workspace }}
|
||||
CACHE_DIR: /home/data/cfs/.cache
|
||||
FLAGS_dynamic_static_unified_comm: "True"
|
||||
python_version: "3.10"
|
||||
paddle_whl: https://paddle-qa.bj.bcebos.com/paddle-pipeline/Develop-GpuSome-LinuxCentos-Gcc82-Cuda118-Cudnn86-Trt85-Py310-CINN-Compile/latest/paddlepaddle_gpu-0.0.0-cp310-cp310-linux_x86_64.whl
|
||||
run: |
|
||||
container_name=${TASK}-$(date +%Y%m%d-%H%M%S)
|
||||
echo "container_name=${container_name}" >> "$GITHUB_ENV"
|
||||
if [[ "$RUN_DOWNSTREAM" == "false" ]]; then
|
||||
echo "Not in a pull_request or test_build event. Skipping..."
|
||||
else
|
||||
docker run -d -t --name ${container_name} --net=host -v /dev/shm:/dev/shm --shm-size=32G \
|
||||
-v $work_dir/../../..:$work_dir/../../.. \
|
||||
-v $work_dir:/workspace \
|
||||
-v /home/.cache/pip:/home/.cache/pip \
|
||||
-v /ssd1/paddlenlp:/ssd1/paddlenlp \
|
||||
-v /home/data/gzcfs/.ccache/gpubox:/home/data/gzcfs/.ccache/gpubox \
|
||||
-e BRANCH \
|
||||
-e AGILE_COMPILE_BRANCH \
|
||||
-e PR_ID \
|
||||
-e COMMIT_ID \
|
||||
-e work_dir \
|
||||
-e ci_scripts \
|
||||
-e no_proxy \
|
||||
-e CI_name \
|
||||
-e paddle_whl \
|
||||
-e HF_ENDPOINT \
|
||||
-e STUDIO_GIT_HOST \
|
||||
-e PPNLP_HOME \
|
||||
-e HF_DATASETS_CACHE \
|
||||
-e TRANSFORMERS_CACHE \
|
||||
-e CACHE_DIR \
|
||||
-e FLAGS_dynamic_static_unified_comm \
|
||||
-e python_version \
|
||||
-w /workspace --runtime=nvidia $IMAGE_NAME
|
||||
fi
|
||||
|
||||
- name: Download Code
|
||||
env:
|
||||
work_dir: ${{ github.workspace }}
|
||||
run: |
|
||||
if [[ "$RUN_DOWNSTREAM" == "false" ]]; then
|
||||
echo "Not in a pull_request or test_build event. Skipping.."
|
||||
else
|
||||
docker exec -t $container_name /bin/bash -c '
|
||||
rm -rf * .[^.]*
|
||||
echo "Downloading PaddleNLP.tar.gz"
|
||||
wget -q --no-proxy https://paddle-qa.bj.bcebos.com/CodeSync/develop/PaddleNLP.tar --no-check-certificate
|
||||
echo "Extracting PaddleNLP.tar.gz"
|
||||
tar xf PaddleNLP.tar && rm -rf PaddleNLP.tar
|
||||
source $work_dir/../../../proxy
|
||||
cd PaddleNLP
|
||||
git config --global user.name "PaddleCI"
|
||||
git config --global user.email "paddle_ci@example.com"
|
||||
git pull
|
||||
git submodule update --init --recursive --force
|
||||
if [ -n "${PR_ID}" ]; then
|
||||
git fetch origin pull/${PR_ID}/head
|
||||
git checkout -b PR_${PR_ID} FETCH_HEAD
|
||||
git remote add upstream https://github.com/PaddlePaddle/PaddleNLP.git
|
||||
git fetch upstream ${BRANCH}
|
||||
git merge ${BRANCH} --no-edit
|
||||
git diff --numstat ${BRANCH} -- | awk "{print \$NF}"
|
||||
else
|
||||
echo "Not in a pull_request event. Skipping PR-specific operations."
|
||||
fi
|
||||
git log --pretty=oneline -10
|
||||
'
|
||||
fi
|
||||
|
||||
- name: Skip For Bug
|
||||
run: |
|
||||
if [[ "$RUN_DOWNSTREAM" == "false" ]]; then
|
||||
echo "Not in a pull_request or test_build event. Skipping..."
|
||||
else
|
||||
docker exec -t $container_name /bin/bash -c '
|
||||
cd /workspace/PaddleNLP
|
||||
echo "no skip for bug"
|
||||
'
|
||||
fi
|
||||
|
||||
- name: Test
|
||||
run: |
|
||||
if [[ "$RUN_DOWNSTREAM" == "false" ]]; then
|
||||
echo "Not in a pull_request or test_build event. Skipping..."
|
||||
else
|
||||
docker exec -t $container_name /bin/bash -c '
|
||||
ldconfig
|
||||
unlink /usr/bin/python3
|
||||
ln -sf $(which python${python_version}) /usr/bin/python3
|
||||
pip config set global.cache-dir "/home/.cache/pip"
|
||||
source $work_dir/../../../proxy
|
||||
set -e
|
||||
cd /workspace/PaddleNLP && git config --global --add safe.directory $PWD
|
||||
timeout 50m bash scripts/unit_test/ci_unit.sh ${paddle_whl}
|
||||
'
|
||||
fi
|
||||
|
||||
- name: Upload Allure-reports & Logs
|
||||
if: always()
|
||||
env:
|
||||
home_path: ${{ github.workspace }}/../../..
|
||||
bos_file: ${{ github.workspace }}/../../../bos/BosClient.py
|
||||
allure_file: ${{ github.workspace }}/../../../allure-2.19.0/bin/allure
|
||||
run: |
|
||||
if [[ "$RUN_DOWNSTREAM" == "false" ]]; then
|
||||
echo "Not in a pull_request or test_build event. Skipping..."
|
||||
else
|
||||
docker exec -t $container_name /bin/bash -c '
|
||||
unset http_proxy && unset https_proxy
|
||||
if [ ! -f "${{ env.bos_file }}" ]; then
|
||||
wget -q --no-proxy -O ${{ env.home_path }}/bos_new.tar.gz https://xly-devops.bj.bcebos.com/home/bos_new.tar.gz --no-check-certificate
|
||||
mkdir ${{ env.home_path }}/bos
|
||||
tar xf ${{ env.home_path }}/bos_new.tar.gz -C ${{ env.home_path }}/bos
|
||||
fi
|
||||
if [ ! -f "${{ env.allure_file }}" ]; then
|
||||
wget -q --no-proxy -O ${{ env.home_path }}/allure-2.19.0.zip https://xly-devops.bj.bcebos.com/tools/allure-2.19.0.zip --no-check-certificate
|
||||
unzip -q ${{ env.home_path }}/allure-2.19.0.zip -d ${{ env.home_path }}/
|
||||
fi
|
||||
if [[ "${{ env.RUN_DOWNSTREAM }}" == "" && -n "${PR_ID}" ]]; then
|
||||
bos_prefix="${PR_ID}/${COMMIT_ID}"
|
||||
elif [[ "${{ env.RUN_DOWNSTREAM }}" == "true" && -n "${PR_ID}" ]]; then
|
||||
bos_prefix="${PR_ID}/${COMMIT_ID}/test_build"
|
||||
else
|
||||
bos_prefix="schedule/$(date +%Y%m%d)"
|
||||
fi
|
||||
cd /workspace/PaddleNLP/unittest_logs
|
||||
for FILE in /workspace/PaddleNLP/unittest_logs/*; do
|
||||
file=$(basename "$FILE")
|
||||
python ${{ env.bos_file }} $file paddle-github-action/PR/PaddleNLP/unittest-gpu/${bos_prefix}/logs
|
||||
echo "$file: https://paddle-github-action.bj.bcebos.com/PR/PaddleNLP/unittest-gpu/${bos_prefix}/logs/$file"
|
||||
done
|
||||
cd /workspace/PaddleNLP/
|
||||
${{ env.allure_file }} generate result -o report
|
||||
tar -czf products.tar.gz report unittest_logs
|
||||
python ${{ env.bos_file }} products.tar.gz paddle-github-action/PR/PaddleNLP/unittest-gpu/${bos_prefix}/logs
|
||||
echo "report: https://paddle-github-action.bj.bcebos.com/PR/PaddleNLP/unittest-gpu/${bos_prefix}/logs/products.tar.gz"
|
||||
'
|
||||
fi
|
||||
|
||||
- name: Terminate And Delete the Container
|
||||
if: always()
|
||||
run: |
|
||||
docker rm -f $container_name 2>/dev/null || true
|
||||
Reference in New Issue
Block a user